SMO Wiki

A pretty snapshot of the Wiki brought to you by the Social Media Observatory at HBI

Twitter Tools

This list provides an overview of useful data collection tools that can be used for research on Twitter. If you face problems or issues with one of the applications on the list, feel free to post an Issue. It helps us to maintain this list.

Table of contents

Overview

Most of these Twitter tools connect to official Twitter APIs and therefore need an API key from Twitter. You can retrieve an API key from Twitter easily, just follow the documentation. You are bound to the restrictions given by Twitter. You can read about the rate limits here. Version 2 of the API will be more restrictive (at least it looks like that at time of writing). As an academic, you can apply for access to the new academic track (Twitter Academic API Track Application) though to elevate your access levels to 10 million tweets per month and access to the ‘historic’ archive, i.e., tweets older than 7 days.

Some of the tools are scrapers, which do not use the official APIs. Please be aware that the use of these tools might violate Twitter’s Terms of Service. Despite being public, Twitter data can be very personal. Ensure to inform yourself thoroughly in order to follow data protection laws and ethical guidelines that apply to your research before starting your data collection.

Data collectors

Twitter API? Last Tested Language Interfaces Comments
Facepager V1/V2 2/2/2021 - GUI No programming needed
twacapic V2 7/4/2021 Python CLI early development
twarc V1/V2 2/2/2021 Python CLI / Python Module Programming possible
TwitterAPI V1/V2 2/2/2021 Python Python Module Programming needed
Twint Scraper 2/2/2021 Python Python Module Programming needed
Twitterscraper V1 2/2/2021 Python Python Module Programming needed
tweepy V1 2/2/2021 Python Python Module Programming needed
rtweet V1 Not Tested R R Module Programming needed
twitter-explorer V1 Not Tested Python GUI No Programming needed

Tools with extended analysis/refinement capabilities and/or dashboards

DMI-TCAT

The Digital Methods Initiative Twitter Capture and Analysis Toolset [DMI-TCAT] is a set of tools to retrieve and collect tweets from Twitter and to analyze them in various ways. It is written mostly in PHP and runs in a webserver (LAMP) environment.

DMI-TCAT provides robust and reproducible data capture and analysis, and interlinks with existing analytical software. Captured data sets can be refined in different ways (search queries, exclusions, date range, etc.) and the resulting selections of tweets can be analyzed in various ways, mainly by outputting files in standard formats (CSV for tabular files and GEXF for network files).

The big plus of DMI-TCAT is that it is organized around a MySQL database, which can run 24/7 robustly over months to years. However, setting up DMI-TCAT on a server requires some command line skills.

Installation Instructions

SMO-TMAS

SMO-TMAS allows users to pull tweets of specified Twitter handles and tweets containing specified keywords by querying Twitter’s REST API GET search/tweets endpoint and statuses/user_timeline endpoint as well as Twitter’s STREAM API. The collected tweets can be downloaded as .csv file and SMO-TMAS also provides data analysis components that can be used to analyze and visualize the collected data right away.

known issues and limitations:

Notable Features:

Installation via: Accessible through a Web Application, no local installation needed.

Documentation and Development

Access

TAGS

TAGS is a free Google Sheet template which lets you setup and run automated collection of search results from Twitter.

known issues and limitations:

Notable Features:

Installation via: An Google account is needed to install this sheets

Download and installation instructions

Support forum for beginners and advanced users

Twitter Explorer

Twitter explorer is an openly accessible interface that can collect, transform and visualize Twitter data. It can create a complex network without prior programming skills. twitter explorer consists of mainly three parts which are collector, visualizer, and explorer.

Notable Features:

Installation via: An installation package is available for Windows, Linux and MacOS

Requires python 3.6 or above .

# replace XXX by release number
 cd ~/Downloads/twitter-explorer-vXXX
pip install -r requirements.txt

After installation we can collect data using streamlit

streamlit run collector.py

Documentation and Usage

VOSONDash

VOSONDash is an interactive R Shiny web application for the visualisation and analysis of social network data. The app has a dashboard layout with sections for visualising and manipulating network graphs, performing text analysis, displaying network metrics and the collection of network data using the vosonSML R package.

known issues and limitations:

Notable Features:

Installation via: CRAN

Download

Installation and Usage

Helpers

Hydrator

Hydrator is an Electron based desktop application for hydrating Twitter ID datasets. Twitter’s Terms of Service do not allow the full JSON for datasets of tweets to be distributed to third parties. However they do allow datasets of tweet IDs to be shared. Hydrator helps you turn these tweet IDs back into JSON and also CSV from the comfort of your desktop.

Notable Features:

Installation via: An installation package is available for Windows, Linux and MacOS

Downloads

Documentation and Usage

Single Twitter ID Lookup webtools