Instagram Hashtag Scraper
We’ll walk through how to scrape Instagram hashtags legally (without violating the Instagram Terms of Service) using a little-known technique involving HAR files that records your web traffic as you browse Instagram. This means it’s 100% safe to use this scraping approach while logged in to Instagram!
You’ll be able to scrape the public posts for any hashtag, allowing you to download captions, engagement metrics, image URLs and the usernames of the post authors.
You can also use this HAR scraping method for anything else on Instagram, such as follower lists, comments, or user posts. Just see the video below to learn more.
Hashtag List Scraping
If you’re trying to find influencers or generate content ideas within a specific niche, you may want to take a closer look at the top & recent content for hashtags within your niche.
To generate a list of hashtags relevant to your business, you can see the Hootsuite Hashtag Generator, which allows you to provide a few keywords and descriptions about your niche to generate a list of hashtags.
Below is the list of hashtags that the tool generated, which we’ll scrape the top posts & recent posts for:
1. #beerlovers
2. #craftbeer
3. #drinkbeer
4. #beerporn
5. #brewery
6. #beertime
7. #beersofinstagram
8. #beerstagram
9. #beeroclock
10. #drinklocalbeer
Once you have a list of hashtags, we can browse the top posts and first page of recent results while logged out of Instagram by visiting the following URL format:
https://www.instagram.com/explore/tags/HASHTAG_NAME_LOWERCASE/
So for example, if the Hashtag Generator gave us the hashtag #BeerLovers, we would remove the # and convert it to lowercase, then substitute it in the URL format above to get this final URL: https://www.instagram.com/explore/tags/beerlovers/ which we can visit to get the top & recent hashtag posts.
1. Browse Hashtags
To start scraping your list of hashtags, head to the first hashtag URL as explained above (you don’t need to be logged in to Instagram), then right click on the page and hit “Inspect” to open developer tools, which will automatically begin recording your web traffic.
Under developer tools, select the “Network” tab and type in logged_out_web_info under the filter and make sure the “Preserve Log” box is checked so your browser will capture web traffic as you browse through multiple URLs.
Now that we’re all set up, refresh the web page to force the Instagram API to send its data to our browser now that we’re recording. Under the network tab, you should see a single entry show up for logged_out_web_info under the Network tab filter. Click on that entry to ensure it’s loaded into the web browser and you can see the raw JSON data as shown in the screenshot above.
Top & Recent Posts
Although the Instagram web page only shows top posts, if you inspect the JSON you’ll see in the edge_hashtag_to_media collection an edges array with the first 50 or so most recent posts using the hashtag. So we’ll still be able to scrape these recent posts (in addition to the top posts), even though they are not shown visually on the Instagram web page.
Multiple Hashtag Pages
Now if you have a 2nd URL you want to scrape, enter it in the same browser tab you currently have open. Wait for it to load and you should see a 2nd entry for logged_out_web_info in the Network tab. Click on that entry to ensure it loads. If you skip this step (clicking on the entry in the Network tab), your browser may lose this data when it goes to the next URL.
Repeat this process until you go through all of your URLs. You should be able to easily process 10 - 100 hashtags in one sitting. Please see the first video on this page if you need more detailed instructions or get stuck.
2. Export a HAR File
Once you’re done browsing through the hashtags you want to scrape, click on the down arrow labeled “Export HAR…” under the Network tab in developer tools to download a HAR file containing all of the raw JSON data that the Instagram API sent to our browser.
Upload that HAR file to the HAR File Web Scraper and we’ll automatically combine the hashtag data requests together, so you can download the raw JSON to your downloads folder for free, just click on the blue links on the group ending in /logged_out_web_info/.
If you want to use our service to combine all of this data into downloadable CSV files, then click on the “Parse Group” button as shown above and we’ll automatically combine the data together so you can download it instantly.
3. Download Hashtag Data
After clicking “Parse Group” you’ll see separate collections for the “Recent Posts” and “Top Posts” for all of the hashtag posts.
To download the recent hashtag posts, look for the data › hashtag › edge_hashtag_to_media › edges collection as shown below. In this example, we averaged around 70 posts per hashtag for a total of scraping 695 posts for 10 hashtags.
To download the top hashtag posts, look for the data › hashtag › edge_hashtag_to_top_posts › edges collection. This will have up to 9 posts per hashtag, but in this example some hashtags had fewer than 9 top posts, so we only scraped a total of 80 posts, or an average of 8 posts per hashtag.
Hashtag Post Fields
The main fields you’ll be interested in are outlined below:
- ID of the Post
- Post Type (Image, Video or Carousel)
- Shortcode (Use for the Post URL, e.g.
https://www.instagram.com/p/SHORTCODE/) - Post Timestamp (Unix Epoch)
- Full Image URL
- Thumbnail Image URL
- Is Video?
- Accessibility Caption of Photo
- Caption of Post
- Comment Count
- Like Count
Root Hashtag Reference
To see the “root” hashtag used to find each post, download the CSV file for the collection and look for the column labeled data.hashtag.name towards the right hand side of the CSV file. This will repeat over many rows for posts that were found for each provided hashtags.
If you scrape multiple hashtags that are related, you may want to de-duplicate the dataset by the shortcode column to ensure you remove any duplicate posts, for example if you’re trying to analyze the total number of likes for the entire dataset.
Single Hashtag Scraping
If you have a specific hashtag in mind you’d like to scrape a lot of posts for, you’ll need to be logged in to Instagram so you can scroll through all of the available hashtag posts.
While scrolling through a hashtag’s posts, Instagram will eventually cut you off from scrolling through data, so you will typically be limited to the few thousand or so posts that Instagram returns per hashtag, so please don’t expect to scrape the full results for any given hashtag if it’s popular.
1. Browse Hashtag Posts
Open up Instagram in your web browser, then navigate to the hashtag page you want to scrape the data of the posts for. You can use a search engine to find this page, or simply find a post on Instagram with the hashtag you’re interested in, then click through that hashtag to access the Instagram Hashtag Posts page.
On your web browser (we’ll assume you’re using Google Chrome, but these instructions should be similar for any browser), right click anywhere on the page and hit Inspect to open up developer tools (this will allow you to record your web traffic as Instagram serves back the hashtag posts). You can then click on the “Network” tab to see your web traffic with the data in it.
You may now want to refresh the page (now that you’re recording your web traffic), so you get the initial set of posts loaded into your browser. This will also load the “Top Posts” of the page into your browser so you’ll be able to extract this data later on.
Now with developer tools open, simply scroll through the posts (just keep scrolling down the page) and your web browser will record the raw data that Instagram is sending back to your browser about the posts.
You should be able to scroll through at least a few thousand or so… just try to take a break or don’t scroll too quickly so Instagram doesn’t think you’re a bot (even though you’re not since you’re just using the website normally).
2. Export HAR File
Once you’re done scrolling, look for a download arrow in your browser’s developer tools (Network Tab) that’s labeled “Export HAR…” Click this and save the HAR file somewhere you can access quickly, like your Desktop:
Now that we have the raw Instagram data inside this large file, the trick is to get the data out of this file. For this, we use the HAR File Web Scraper where you can upload the file to, and the software will parse everything out for you and group together all the different requests with the hashtag posts in it:
Look for the parse group labeled with graphql - this will contain the useful data Instagram sends to your browser to show the hashtag posts.
3. Download Instagram Data
You can click the small disk icon next to each of the requests to download the raw data individually, or scroll up and click the “Parse Group” button (above all the individual responses) to download a combined CSV file with the hashtag posts in them.
You can see that the HAR File Scraper combined all of the individual requests into a combined list that you can now download as a single CSV file. The file will contain details about each post, including links to the images, videos and basic information about the author, such as user ID.
Scrape Related Hashtags
If you don’t care too much about the actual posts using the hashtag, but just want to find related hashtags, say to fill your post’s maximum of 30 unique hashtags requirement, then you can use the same HAR File Web Scraping approach outlined here.
Simply go to the Instagram website, begin recording your web traffic, and instead of browsing posts for a hashtag, just type the hashtag into the search bar and you’ll see related results. Keep typing new hashtags if you need more, and when you’re done export the HAR file and follow the steps above to extract the search results (and hashtag suggestions).
Why Scrape Instagram Hashtags?
Instagram hashtags can be very helpful for finding social media content or tracking branded content for hashtags that you or your clients promote. While it’s great to mindlessly scroll through these Instagram posts for hashtags of interest, it can be difficult to make any actual decisions without the raw data behind the hashtag posts. For example, how many people are posting with the hashtag daily, weekly, or monthly? What’s the engagement for the posts - is it increasing or decreasing?
Legal Concerns
Regardless of how useful this raw data for Instagram hashtag posts can be for influencers, marketers & researchers, Instagram prohibits the automated scraping of their data, making collecting the raw hashtag post data tricky (and we wonder why some people call it a dying platform).
Although the Instagram graph API offers an “Instagram basic” type of hashtag API for Instagram business account, you’ll find it to be extremely limiting and only allows scraping a very narrow set of data each month.
While there are automated browser extensions & hashtag scrapers that promise to scrape Instagram data for you, these nearly all violate the Instagram Terms of Service, which prohibit automated access to Instagram or the scraping of its data through the website. Using these unofficial tools will often leave footprints behind that Instagram can detect and then ban and block your account.