Scraping the Instagram API
If you need Instagram data fast and don’t want to get blocked or banned for using an automated IG scraper, read on to learn how to legally scrape anything on Instagram without being detected or violating Instagram’s Terms of Service.
Our approach relies on “recording” your web traffic from your browser as you interact with Instagram and then performing the scraping on that recording instead of the actual Instagram website.
1. Browse Instagram
In order to record Instagram data, you’ll simply need to visit the target profile or post on Instagram you’d like to get data from. One trick is to use Google and simply type site:instagram.com followed by your search term to pull up the Instagram URL.
Once you have the content loaded that you want to scrape, right click on the page and hit “Inspect” to open up developer tools. This will now begin recording your web traffic as you interact with Instagram. You’ll then want to reload the page to get a fresh batch of data in, then click on or scroll through the elements you want to scrape.
In the above example, we visited The Rock’s Instagram page and are scrolling down through his posts to scrape them. Under the network tab you can see the raw data is sending to your browser if you type in graphql under the filter, but this is optional and not required by our Instagram data scraper.
2. Export a HAR File
Once you’re done scrolling through the data you need, look for the down arrow under the Network tab in developer tools labeled “Export HAR…” and click that to download a HAR file recording of the raw Instagram data.
You’ll then want to upload that file to the HAR File Web Scraper and look for the following group with /graphql/query/ in the header:
3. Download Instagram Data
After clicking Parse Group, you’ll see downloadable collections you can save to your computer containing the Instagram data from your HAR file.
This will typically work best with lists of items on Instagram, such as posts (by user, explore page, or hashtag), reels, follower lists, comments, etc… You can try for yourself for free using the HAR File Web Scraper to see exactly what’s possible.
The tool will not work with extracting personal details from profiles, such as contact information or email. If you need to do lead generation, we suggest seeing our Google Maps Data Scraper instead.
Popular Examples
We’ll walk through a few of the most popular examples we see when scraping publicly available data from Instagram. While you can also use the HAR file scraper to extract data from private profiles (as long as you have access to them), we suggest being extremely cautious here to never share this private data since it was never intended to be public data!
Instagram Posts
Our HAR File scraping approach is great for scraping Instagram posts from hashtag search results pages, explore, profile pages and more. Basically anywhere you see a list of posts on the Instagram website, you can scrape the basic details like post captions, post URL, post author ID, Instagram username (and therefore profile URL) and the link to the actual image or video posted.
In order to use this post scraper, you’ll need to navigate to an Instagram URL that contains a list of posts to scrape, such as a profile page, hashtag page or search results page.
By Specific Hashtags
Instagram post data for hashtags is very easy to collect, simply navigate to any hashtag page, begin recording your web traffic, and scroll through all the results. See our detailed article on Instagram Hashtag Post Scraping for an in-depth guide.
By Instagram Profiles
You can also scrape all the Instagram posts from any Instagram account you have access to. Simply browse to the user you want to scrape the posts of, record your web traffic and scroll through the posts to capture the data. See our article on our Instagram User Post Scraper for more information.
By Search Queries
Believe it or not, Instagram can also (barely) function as a search engine! If you’re looking for posts about a very specific topic, you can simply try it as a search term and browse through the results while recording your web traffic. Export this to a HAR file and see what the HAR file web scraper picks up.
Instagram Followers
You can scrape the basic Instagram usernames of followers from any profile you have access to. However, since you’re scraping a basic list of users (not their full profiles), don’t expect to download rich data about each user. You’ll need to visit a profile URL and then scroll through their list of followers to collect the list of names, then export & parse the HAR file to download the username list.
To scrape the followers list, simply visit any profile and click on the text containing their follower count to open up the list. While recording your web traffic, scroll down through the list to capture their information. See our article on Instagram Follower Scraping for details on how to do this.
Instagram Comments
If you have a specific Instagram post that you want to analyze, the comments can be a great way to find interesting Instagram profiles or gauge the sentiment and feedback of your own content (or competitors). Simply navigate to the post, record your web traffic and then scroll through the comments to capture all comments & replies. See our Instagram Comments Scraping article for more details and walkthrough.
Instagram Accounts
Another common use for our service is to find Instagram profiles you may want to reach out to for sponsorship deals.
By Popular Instagram Hashtags
If you’re looking for Instagram profiles in a certain niche, a good way is to start with a list of hashtags that represents your target niche and then find users and posts that are popular & authoritative containing those hashtags.
By Comments
Another good way to spot real influencers is finding Instagram profiles with high levels of engagement in the comments section of a very popular Instagram post.
Why Scrape Instagram?
Publicly available data from Instagram is a valuable resource many businesses & top influencers use for unlocking new market opportunities, finding their target audience and generating more ideas for creative assets.
With the right Instagram data, you can find the most relevant hashtags for upcoming posts or perform sentiment analysis on Instagram comments from your previous posts. All of this requires collecting as much data relevant to your business from Instagram as possible, such as from an Instagram profile scraper or web scraping APIs.
In this article, we’ll provide a brief overview of the data scraping options you have for profiting from scraped data. Disclaimer: we do offer our own Instagram scraping solution, so we will definitely be a bit biased!
Legal Concerns
Believe it or not, Instagram does not want you scraping its data and explicitly forbids doing so in its Terms of Service, prohibiting automated access and collection of Instagram data such as user profiles through web scraping techniques.
But this hasn’t stopped countless developers & companies from collecting data from Instagram over the years, despite it being against their Terms of Service. Although violating Terms of Service is not illegal if only done between you and Instagram (without using a third party), once a third party helps you violate Instagram’s Terms, they are breaking the law. See Tortious Interference of Contract for more information.
Most of these Instagram scrapers get shut down within a few years thanks to lawsuits against scraping for hire and explains why most of the Instagram scrapers you see are “brand new” to the market. This also explains why the remaining Instagram scraping companies are based outside of the United States - it’s much more difficult to extradite these companies for trial.
If you want to scrape user data, profiles, post comments or other pieces of data from Instagram or similar social media platforms, it’s important you not only have the right tool, but that the tool you use doesn’t involve yourself in illegal scraping activities that could lead to your Instagram account being banned.
HAR File Scraping
While you’re free to attempt to automatically extract Instagram data yourself without using an illegal Instagram scraper tool (through your own code), we legally can’t recommend doing so! However, we can suggest a totally safe & legal way to manually export Instagram data (their lawyers even challenged us on this and had to back down) using HAR Files that we explained above.
Web scraping Instagram with the HAR File Web Scraper is perfectly legal and cannot be challenged since it does not scrape data from the Instagram website, but rather scrapes it from a recording of your usage of Instagram. Your usage while generating the HAR file is governed under the Instagram Terms of Service, but the act of extracting specific data out of your recording cannot be considered under the scope of Instagram’s Terms.
Scraping Instagram via HAR Files is similar to using an Instagram web scraper, but will extract data from a recording of your completely legal usage of the Instagram website. So instead of using an automated Instagram scraping tool, web scraping software or proxy provider, you just use Instagram normally and then extract data from that browsing session.
All major web browsers allow you to record the raw data that Instagram sends to you while using their site, and a HAR file acts as a “recording” of this data (similar to VHS tapes if you’re old enough) - we then simply pull the data from the HAR file instead of through automated access to Instagram. The freemium tool we provide to do this is our HAR File Web Scraper you can learn more about in the video on this page.
You can use the HAR file scraper free of charge to download the raw JSON format from the Instagram API. This can be great if you have your own custom code that can work with this data; just be warned that it can change in structure and format without any notice.
If you need to download Instagram data to CSV format however, you can create a free account with our service and download up to 10 rows daily. Give it a try to see how it works, or see above for a step-by-step guide.
Scrape Responsibly
Please keep in mind that most Instagram profiles do not want to be scraped, even if they are public! While you’re typically safe to do any level of private analysis on the data you scrape, re-publishing data (even if publicly scraped Instagram data) can be risky, especially if it contains personal details.
For example, republishing scraped data without permission from the original creator may result in copyright infringement and you must check the Terms of Use for Instagram before doing anything with any collected data. Always get consent before publishing scraped data, even if you think it’s “fair use” - always be a little extra cautious since the data was shared without the expectation of it being re-purposed outside of Instagram.