If you need to scrape Twitter followers, we’ll discuss the right way to do that using the official Twitter API V2, which will allow you to scrape the complete list of Twitter followers and following lists into several formats as CSV & Excel files.
Exporting Twitter follower lists can be an extremely useful social media marketing tactic for marketing research, Twitter analytics & advertising, helping you better understand what these audiences have in common and why they follow relevant accounts in your niche or industry.
You can also export your Twitter followers list for safekeeping, as well as scrape emails from Twitter followers bios (perhaps to have a backup email list). You will not be able to use the API as a Twitter email scraper, as there is no structured data field in the user objects for the email. Instead, you can find one of many free email extractors online to pull out publicly shared emails from the user’s bio, which is accessible via API.
And unlike other social media platforms, Twitter actually encourages you, the public, to make use of their public data through the official Twitter API, making it extremely easy to download complete follower lists and user details for any public Twitter account via the Followers API Endpoint.
The rest of this article will discuss how to use the Stevesie Data Platform to collect the followers, Tweets & details from the Twitter API using the endpoint listed above, which will return the results CSV files you can easily analyze and use in other programs without writing any code. Stevesie Data is a paid platform and the rest of this article assumes you have the plus plan for running workflows.
Let’s say we want to scrape all of the followers for the @YouTube Twitter handle, all 73.6 million of them at the time of this writing. We will refer to this as the “target” account (because we’re targeting this account to get the follower accounts of it).
To use the followers API, we’ll need to first know the User ID
of the target account to continue, which you can obtain through the User Details Integration. Simply provide the Twitter Username and you’ll get back the ID.
Once you execute the endpoint, you’ll get back the User ID
or 10228272
in this example (wow, that’s a low number, meaning they joined Twitter relatively early):
Before we get the full 73.6 million followers, we should start small to make sure we’re getting back all the data we need from the Twitter API. Head to the Twitter User Followers Integration and we’ll get started by scraping the first 1,000 followers.
Simply paste in the User ID
we obtained in the previous step and provide your Twitter API key:
We’ll assume that you also want to scrape details about the followers (such as each followers follower / Tweet count, pinned Tweet, description, bio, etc…), so you’ll want to provide values for Expansions & Fields. Just copy-paste the “Example” values provided in the integration to get as much data back as possible:
Hit execute, and you’ll see the response data organized into downloadable collections as CSV files that you can open in Excel or similar programs. You should first see the data
collection which contains all of the user data of the followers, complete with metadata about the followers including their own follower counts:
Below is a summary of the helpful fields you will get back for each follower:
description
Their Twitter Bio, which sometimes contains contact information such as emailusername
Twitter username of the followerurl
Website URL the user displays on their profileprofile_image_url
URL to the user’s profile imageverified
If the user is verified or notcreated_at
When the Twitter account was createdid
The User ID you can use in other endpointsname
The user’s display name, sometimes this may be an email addresspublic_metrics.followers_count
How many followers the user haspublic_metrics.following_count
How many Twitter accounts the user is followingpublic_metrics.tweet_count
Number of Tweets from the accountpublic_metrics.listed_count
How many lists include the userpinned_tweet_id
The ID of a Tweet “pinned” to their account (cross reference with the includes > tweets
collection)pinned_tweet_id
Where the user is located, in text formFor users that have a value for their pinned_tweet_id
column, you’ll find a corresponding entry in the includes > tweets
collection. Simply look for the matching value for the id
column and you’ll get back details about the user’s pinned Tweet (which can reveal even more details about the user):
You’ll see the following useful fields for the pinned Tweets (note, not every user has a pinned Tweet, but if they do it’s pretty good odds they’re not a bot!):
lang
Language of the Tweetcreated_at
When the Tweet was publishedpublic_metrics.retweet_count
The number of times the pinned Tweet was Retweetedpublic_metrics.reply_count
The number of replies to the Tweetpublic_metrics.like_count
Number of likes of the Tweetpublic_metrics.quote_count
Number of times the Tweet was quotedsource
Which client posted the Tweet (e.g. iPhone, Web, Android, etc…)text
The content of the TweetThe Twitter API is kind enough to return very useful “metadata” about the followers extracted from the texts of the user bios and pinned Tweets. E.g. if a user mentions hashtags, other users, cashtags, URLs, or even topics written in free text, the API will return them as structured data parsed out into additional collections you can download.
These will help you significantly if you’re trying to understand the relationship between users, concepts, hashtags, etc… without having to write your own free text parser. To download these, click on “All Collections” and scroll down until you start seeing “entities” collections:
Download these as “Expaned CSV” format and you’ll get a reference to the parent Tweets & Users on the right side of the downloaded CSV files.
So the data about the first 1,000 followers is great (note, these are just the first 1,000 the API returns, and not necessarily the account’s first chronological followers), but how do we get the next 1,000 followers and so on, until we scrape them all?
The answer lies in the “Pagination Token,” which is a special value that the Twitter API returns in the <root>
collection’s meta.next_token
field when there’s another page of results (e.g. you won’t see this if your target account has under 1,000 followers). This value tells the Twitter API to get new followers beyond the pagination “marker” provided.
You need to copy and paste this value into the “Pagination Token” input on the User Followers Endpoint Integration and keep doing this until you get to the end and have gone through all the followers.
This is obviously a pain to do, which is why we can use the Twitter User Followers & Details - Pagination workflow to do all of this for us. Just visit the link here and then click “Import” to add the workflow to your account (which will automatically go through all the pages and combine results for you).
After importing the workflow, simply provide the inputs like you did earlier with the individual endpoint version:
Make sure you also provide the necessary values for the Expansions and Fields (just copy-paste the example values) to ensure you get all of the data back. When collapsed, these inputs should look as follows:
On the right side of the screen you’ll notice a list of “Extractors,” which are pre-defined collections that the workflow will collect and combine for you at the end. You can add to them by going back to the Twitter Followers Endpoint and then select the “New Extractor…” option next to the collection you want to capture:
Once created, you’ll be able to select it and add to your workflow’s extractor list:
You can also remove any extractors you don’t need by clicking the “X” on any extractor.
Once you’re done setting up your workflow, hit the “Execute” button to run the workflow. It will query Twitter and make requests, page-by-page and automatically extract and pass down the pagination token for you. When done, it will combine all the results together for you from all of the pages into CSV files (one CSV file per extractor):
You’ll probably be interested in the Twitter_Followers.csv
file, which will contain one row per follower and you can also download the Twitter_Followers_Pinned_Tweets.csv
file which will contain details about the individual Tweets that the followers pinned to their accounts at the time of the scraping. For time-sake, we only ran this demonstration workflow for 5 requests to get 5,000 followers, but there’s no limit on how many followers you can get back.
Twitter enforces a rate limit of 15 requests every 15 minutes for scraping followers. The workflow automatically takes this into account and will make an initial batch of 15 requests, then wait 15 minutes, then keep going until told to stop or until it reaches the end of the list. This means though that you’ll be limited to the following throughput when scraping followers:
So this means that to get all 73.6 million of YouTube’s followers, I would need to let the workflow run for over a month, which is not the most practical solution.
In practice, you’ll probably find it easier to work with “samples” from very large and popular accounts (e.g. just scrape 10,000 followers from each account). The workflow makes it easy to do this, e.g. if you enter in a list of User IDs one per line, you can set a “Pagination Limit” under the “Pagination Token” as shown below:
This tells the workflow to only scrape an additional 9 pages PER User ID when scraping followers, so if we enter in 100 User IDs to scrape the followers for, the workflow will only get up to 10,000 followers PER user ID (10 total pages per User ID). This can be very helpful when you’re trying to scrape followers in a niche and don’t want to go “too deep” on any one account, but rather just get a general overview of the follower data.
/2/tweets/search/{{recent_or_all}}
/2/users/{{user_id}}/followers
/2/users/{{user_id}}/{{tweets_or_mentions}}
/2/users/by/username/{{username}}
/2/users/{{user_id}}/following
/1.1/tweets/search/fullarchive/{{environment_label}}.json
/1.1/search/tweets.json
/2/users/{{user_id}}/liked_tweets
/2/spaces/search
/1.1/lists/members.json
/1.1/statuses/retweets/{{tweet_id}}.json
/2/tweets/{{tweet_id}}/liking_users
/1.1/trends/place.json
/2/tweets/counts/{{recent_or_all}}
/1.1/users/lookup.json
/1.1/geo/search.json
/2/spaces/{{space_id}}
/1.1/trends/closest.json
/2/tweets/{{tweet_id}}
/2/tweets/{{tweet_id}}/retweeted_by
/2/users/{{user_id}}
/1.1/followers/list.json
/1.1/friends/list.json
/1.1/lists/memberships.json
/1.1/lists/list.json
/1.1/lists/ownerships.json
/1.1/statuses/user_timeline.json