โš™๏ธ Twitter Tweets & Archive Search - Pagination Workflow Formula By steve


Use this to access the New & Improved Twitter API V2 for scraping recent or historical Tweets since 2006: https://developer.twitter.com/en/docs/twitter-api/tweets/search/introduction

If you are on the standard track you can scrape 500,000 Tweets per month, and if academic 10,000,000 - YES TEN MILLION - Tweets per month: https://developer.twitter.com/en/docs/projects/overview#tweet-cap

To get back the most data possible (e.g. info about users posting content that match your query), copy and paste the “Example” values in the inputs for all of the fields, otherwise you will only get the basic Tweet text with an ID.

You may want to play around with the endpoint first: https://stevesie.com/apps/twitter-api/tweets-archive-search-v2 before using this workflow. But when you’re ready just click “Import” here to get started.

โš™๏ธ  Twitter Tweets & Archive Search (V2) - Pagination

Twitter Tweets & Archive Search (V2) Pagination Cursor (Self-Loop) Pagination Limit: Unlimited โžก๏ธ Pagination Token

Used for pagination. Provide the next_token you see in a response here to get the next page of results.

  • Bearer Token

    Your Twitter app’s Bearer Token for collecting data with. If you don’t have one, see How to Get a Twitter API Key Without Approval or check out Twitter’s docs on Bearer Tokens for more information.

  • Query

    Search query to run on Twitter, see their docs for examples on how to search.

    Also see their quick start guide for more information: https://developer.twitter.com/en/docs/twitter-api/tweets/search/quick-start/recent-search

    Some common example:

    • from:twitterdev Gets tweets from the @twitterdev account (leave out the @ in your query)
    • #beer Gets tweets containing the hashtag #beer
    • @twitterdev Gets tweets mentioning @twitterdev

    These require the Academic Research Project:

    • $TWTR Gets tweets mentioning the cashtag $TWTR

    You may also want to add -is:retweet -is:reply -is:quote to your query to avoid these other types.

Fields (Users) Example: created_at,description,entities,location,pinned_tweet_id,profile_image_url,protected,public_metrics,url,verified,withheld

A comma-separated list of fields you would like back from the User Model: https://developer.twitter.com/en/docs/twitter-api/data-dictionary/object-model/user

Recent or All Example: recent

Leave this as the default recent to search for Tweets published in the last 7 days. Change this to all to access the V2 Full Archive Search (only available to users on the Academic Research product track). Learn more here: https://developer.twitter.com/en/docs/twitter-api/tweets/search/api-reference/get-tweets-search-all

Fields (Places) Example: contained_within,country,country_code,geo,name,place_type

A comma-separated list of fields you would like back from the Place Model: https://developer.twitter.com/en/docs/twitter-api/data-dictionary/object-model/place

Time Range (Ending Time) Example: 2021-01-01T00:00:01Z

End time range of the Tweets to fetch. This is exclusive, e.g. 2021-01-01T00:00:01Z will exclude the first second of 2021. Format is in YYYY-MM-DDTHH:mm:ssZ.

Fields (Polls) Example: duration_minutes,end_datetime,voting_status

A comma-separated list of fields you would like back from the Poll Model: https://developer.twitter.com/en/docs/twitter-api/data-dictionary/object-model/poll

Fields (Media) Example: duration_ms,height,preview_image_url,public_metrics,width

A comma-separated list of fields you would like back from the Media Model: https://developer.twitter.com/en/docs/twitter-api/data-dictionary/object-model/media

Limit Example: 100

How many results to return on each request.

Fields (Tweets) Example: attachments,author_id,context_annotations,conversation_id,created_at,entities,geo,in_reply_to_user_id,lang,possibly_sensitive,public_metrics,referenced_tweets,reply_settings,source,withheld

A comma-separated list of fields you would like back from the Tweet Model: https://developer.twitter.com/en/docs/twitter-api/data-dictionary/object-model/tweet

Pagination Token

Used for pagination. Provide the next_token you see in a response here to get the next page of results.

Expansions Example: author_id,referenced_tweets.id,in_reply_to_user_id,attachments.media_keys,attachments.poll_ids,geo.place_id,entities.mentions.username,referenced_tweets.id.author_id

A comma-separates list of expanded object data attached to the Tweet. See https://developer.twitter.com/en/docs/twitter-api/expansions for more and possible values.

Time Range (Starting Time) Example: 2020-01-01T00:00:00Z

The starting timestamp of Tweets to return in the query. This is inclusive, so 2020-01-01T00:00:00 will include the first second of the new year.

Extractors

  • Tweet Place Coordinates .data.geo.coordinates.coordinates
  • Tweet Places .data.geo
  • Tweet Attached Media .data.attachments.media_keys
  • Tweet Context Annotations .data.context_annotations
  • Tweet Referenced Tweets .data.referenced_tweets
  • Tweet Annotations .data.entities.annotations
  • Tweet URL Images .data.entities.urls.images
  • Tweet URLs .data.entities.urls
  • Tweet Hashtags .data.entities.hashtags
  • Tweet Mentions .data.entities.mentions
  • Expanded Tweet Places .includes.tweets.geo
  • Expanded Tweet Media .includes.tweets.attachments.media_keys
  • Expanded Tweet Mentions .includes.tweets.entities.mentions
  • Expanded Tweet References .includes.tweets.referenced_tweets
  • Expanded Tweets .includes.tweets
  • Expanded Media .includes.media
  • Expanded Places .includes.places
  • Expanded Users .includes.users
  • Tweets .data
Posted by steve on April 29, 2021, 10:11 p.m. ๐Ÿšฉ  Report