⚙️ Twitter Tweet Search Full Archive - Pagination Workflow Formula By steve


Automatically paginate through the Twitter Archive search, aggregating Tweets, hashtags used in all Tweets, user mentions, photos, extended entities, photos, videos & more.

Getting Started

Import this formula to add the workflow to your account and follow the instructions for the required inputs. You’ll most likely want to specify a time range under optional inputs, so be sure to enter those!

IMPORTANT This is a premium Twitter API endpoint (per Twitter) and you need to pay close attention to your quota in your Twitter’s developer account. They provide you with a small amount of free requests you can make each month, but if you go over your quota you will need to pay Twitter (a separate charge from the Stevesie Data Platform) to make more requests.

⚙️  Twitter Tweet Search Full Archive - Pagination

Twitter Tweet Search Full Archive - Pagination Cursor (Self-Loop) Pagination Limit: Unlimited ➡️ Pagination Cursor

Pagination cursor. Run the first query with this blank, then look for the “next” token returned in the response. Copy and paste that here to get the 2nd page, and repeat for the 3rd page etc…

  • Environment Label

    Your environment label that you set up here. Pick any name that you want, such as dev.

  • Query

    Search query to run on Twitter, see their docs for examples on how to search. Some quick tips:

    Check out the list of search operators for more advanced searching techniques. Here are a few you may be interested in:

    • Hashtag Search: Just enter the search term with a hashtag in front of it, e.g. #beer If multiple hashtags are entered, the API will return tweets containing both hashtags.
    • User Mention Search: Enter the username prepended with the @ sign to get all tweets mentioning the user, e.g. @taylorswift
    • Tweets From Account: Find tweets sent from a specific account by prefixing with from:, e.g. from:interior (note do not use the @ here)
    • Free Text: Find tweets containing free text words, e.g. watching now will return tweets containing both words. Use the OR operator if you’d like, e.g. love OR hate.
    • Geolocation: Use point_radius: lat lon radius to only return Tweets geo-tagged at the location of interest
  • Bearer Token

    Your Twitter app’s Bearer Token for collecting data with. If you don’t have one, see How to Get a Twitter API Key Without Approval or check out Twitter’s docs on Bearer Tokens for more information.

To Date Example: 201712220000

Up until when to search for Tweets from.

From Date Example: 201512220000

Date to search Tweets from (back to 2006).

Limit Example: 100

Limit of how many results to return (100 for the free sandbox).

Pagination Cursor

Pagination cursor. Run the first query with this blank, then look for the “next” token returned in the response. Copy and paste that here to get the 2nd page, and repeat for the 3rd page etc…

Extractors

  • Twitter Tweet Search Full Archive URLs .results.entities.urls
  • Twitter Tweet Search Full Archive Extended Entities .results.extended_entities
  • Twitter Tweet Search Full Archive User Mentions .results.entities.user_mentions
  • Twitter Tweet Search Full Archive Hashtags .results.entities.hashtags
  • Twitter Tweet Search Full Archive Tweets .results
Posted by steve on May 9, 2020, 5:57 p.m. 🚩  Report