Scrape Twitter Search Results
If you need to scrape Tweets from Twitter search results, we’ll show you how to legally do this using the Official Twitter API with our Twitter Search Scraper, allowing you to download up to 10,000 Tweets from the past 7 days on the entry-level basic Twitter API tier ($100/mo.).
While you may be tempted to look for a free solution, we can assure you that using the official API will save you hours of headache with Twitter’s recent changes in late 2023 to block web scrapers from scraping their site and violating their Terms of Service. Web scraping Twitter data via API also offers many advantages you don’t get elsewhere, e.g. location filtering and assurance your data will not be sampled as it is on their website.
Querying the API
We’ll detail how to use the Twitter API V2 Tweet Search Endpoint with step by step examples on how to use advanced operators like Location Search, Time Ranges, etc… with our scraping tools to download a list of Tweets instantly.
You can follow along with all of the examples on this page with our Twitter scraper! Just scroll up and you’ll see a green box allowing you to enter any Twitter search query and download data from Twitter to CSV files with just one click from our Twitter scraping tool.
Keyword Search
You’ll find that nearly all of the search functionality is wrapped into a single query parameter that accepts a string of keywords and search operators. We’ll walk through these examples here so you can better understand what’s possible with the API, or you can browse the official Twitter Query Operator Documentation for more information.
The easiest search to perform is for a keyword, just enter one in like beer and you’ll get back results. You can try on this page, just scroll up and enter any free text term with your Twitter API to scrape some initial results & start extracting data.
Broad Match
By default, entering multiple keywords will use AND logic and only return Tweets containing both keywords or semantically similar words. E.g. a search for craft beer will return Tweets containing both of these words or similar (e.g. “breweries”):
If you want to change the behavior to return Tweets containing either word, then you can put an OR in between your search, e.g. a search for beer OR vodka will return Tweets containing either of these terms.
Exact Match
If you need to guarantee that the search terms in an AND clause appear next to each other, you can simply quote them. This will automatically group them too, so we can use our example above and change it to "free beer" OR "free vodka" and the results will only show Tweets with either of these exact matches. Below is what happens when we search for "craft beer":
Negation
You can also specify words to not include in your results with the - sign. Simply prepend it to a clause, e.g. beer -free will exclude any results containing the word free. This also works with groups or quotes, e.g. -(free beer) or -"free beer".
A common use for these negations is to exclude retweets, quotes & replies from your search results. You can do this by adding -is:retweet -is:reply -is:quote to the end of your query.
Grouping
If you want to combine this logic, you can use parenthesis to make the logic explicit. E.g. (free beer) OR (free vodka) will return Tweets that either contain both free and beer OR free and vodka.
Hashtag Tweets
To find all Tweets containing a hashtag, simply provide it as your query with the # sign in front of the hashtag. E.g. #beer will return all Tweets containing this hashtag. You can also use this in conjunction with keyword searches, e.g. #beer -free will exclude any Tweets with the word “free.”
User Tweets
To find all Tweets from a certain profile page, simply prepend the username with from: and use it as a search operator just like above. To find Tweets from multiple accounts in a single search, you can combine the logic with the OR operator: from:twitterdev OR from:twitterapi.
Historical Timeline
Since the Twitter API search endpoint will only return the past 7 days of Tweets, it’s not a great option if you need to scrape the historical Tweets for a specific user. Instead, you can see our User Tweets Scraper which uses a different endpoint and allows you to scrape up to 3,200 Tweets from any public user without having to upgrade your Twitter API tier to Pro.
Response Data
If you use all of the default options when querying the Twitter API, the response data will be extremely thin and only contain the Tweet ID and the Tweet content as shown below in the scraped data:
But don’t worry, the Twitter API offers much more details in its response for both Tweets (e.g. metadata & popularity metrics) as well as for Tweet authors (names, bios, follower counts). All we have to do is ask the Twitter API to return these fields & objects.
Tweet Details
You’ll likely want to know who wrote each Tweet and how much engagement each Tweet is receiving. To get this data back, you need to add author_id,public_metrics to the tweet.fields parameter and the Twitter API will return these as additional fields.
You can enter as many fields as you want, so we suggest just entering in all of them to get as much data back as possible since it doesn’t count against your Twitter API usage quota.
Author Details
The most common request we get is to know more about the Twitter accounts behind the Tweets in the search results. Since the Twitter profiles are separate objects from the Tweets, we need to use “Expansions” to tell the Twitter API to return these in addition to the Tweets. To do this, just query the expansions parameter with author_id and you’ll get back basic information about the user profiles of each Tweet.
To get back more fields about each author (like follower counts), you can use the user.fields parameter just like we did with the tweet.fields option and supply a list of additional fields we’d like back, such as public_metrics.
Advanced Filters
If you need to refine your search results further than just what the Tweet contains, you can see the following options. However, be warned that some require “Pro” level access to the Twitter API, which starts at $5,000/mo.
Language Filtering
You can filter Tweets by language using the lang: operator. Just add it to your search term and specify a BCP 47 language. E.g. lang:en to filter for only English Tweets. You can see the full list of supported languages in building a query (scroll to the bottom).
Location Filtering
You can restrict Tweet search results to only those that are tagged with a geolocation if you have Twitter Pro level API access only. There are four main options to find Tweets by location we’ll walk through below:
Place
This will match Tweets for a specific place that Twitter has indexed and knows about. You can often get Place IDs from other Tweets you searched for, or you can specify by name like “new york city”. Some examples are: place:"new york city" OR place:seattle OR place:fd70c22040963ac7
Country
You can also specify Tweet location by country with the place_country operator. Use it with any valid ISO Country Code like this: place_country:US OR place_country:MX OR place_country:CA
Coordinates
If you have a specific latitude and longitude in mind, you can use the point_radius operator with a radius of up to 25 miles. The syntax is as follows: point_radius:[longitude latitude radius] (including the brackets) and an example is: point_radius:[2.355128 48.861118 16km] OR point_radius:[-41.287336 174.761070 20mi]
Bounding Box
If you have a larger area to search for, you can specify 2 coordinates to form a bounding rectangle (called “box” for some reason) to restrict the Tweet search to. The format is: bounding_box:[west_long south_lat east_long north_lat] where the width and height of the box can be up to 25 miles. An example is: bounding_box:[-105.301758 39.964069 -105.178505 40.09455]
Time Range Filtering
While this option is available to all access levels, in reality it’s probably only useful for Pro Twitter API users querying the full archive, as you’ll otherwise be limited to only the past week of Twitter data.
To specify a time range to get Tweets back, you need to use the start_time and end_time fields, providing exact datetimes in ISO 8601 format. E.g. YYYY-MM-DDTHH:mm:ssZ. Note, this is not part of the query field as was everything described earlier.