Scraping Instagram Data
Publicly available data from Instagram is a valuable resource many businesses & top influencers use for unlocking new market opportunities, finding their target audience and generating more ideas for creative assets.
With the right Instagram data, you can find the most relevant hashtags for upcoming posts or perform sentiment analysis on Instagram comments from your previous posts. All of this requires collecting as much data relevant to your business from Instagram as possible, such as from an Instagram profile scraper or web scraping APIs.
In this article, we’ll provide a brief overview of the data scraping options you have for profiting from scraped data. Disclaimer: we do offer our own Instagram scraping solution, so we will definitely be a bit biased!
Illegal & Unofficial Instagram Scraping Tools
Believe it or not, Instagram does not want you scraping its data and explicitly forbids doing so in its Terms of Service, prohibiting automated access and collection of Instagram data such as user profiles through web scraping techniques.
But this hasn’t stopped countless developers & companies from scraping Instagram over the years, despite it being against their Terms of Service. Although violating Terms of Service is not illegal if only done between you and Instagram (without using a third party), once a third party helps you violate Instagram’s Terms, they are breaking the law. See Tortious Interference of Contract for more information.
Most of these Instagram scrapers get shut down within a few years thanks to lawsuits against scraping for hire and explains why most of the Instagram scrapers you see are “brand new” to the market. This also explains why the remaining Instagram scraping companies are based outside of the United States - it’s much more difficult to extradite these companies for trial.
If you want to scrape Instagram profiles or other pieces of data from Instagram, it’s important you not only have the right tool, but that the tool you use doesn’t involve yourself in illegal scraping activities that could lead to your Instagram account being banned.
100% Legal HAR File Instagram Scraper
While you’re free to attempt to scrape Instagram yourself without using an illegal scraper (through your own code), we legally can’t recommend doing so! However, we can suggest a totally safe & legal way to scrape Instagram (their lawyers even challenged us on this and had to back down) using HAR Files.
Scraping Instagram via HAR Files is similar to using an Instagram web scraper, but will extract data from a recording of your completely legal usage of the Instagram website. So instead of using an automated Instagram scraping tool, web scraping software or proxy provider, you just use Instagram normally and then extract data from that browsing session.
All major web browsers allow you to record the raw data that Instagram sends to you while using their site, and a HAR file acts as a “recording” of this data (similar to VHS tapes if you’re old enough) - we then simply pull the data from the HAR file instead of through automated access to Instagram. The freemium tool we provide to do this is our HAR File Web Scraper you can learn more about in the following video, or the one at the top of this page.
You can use the HAR file scraper free of charge to download the raw JSON format from the Instagram API. This can be great if you have your own custom code that can work with this data; just be warned that it can change in structure and format without any notice.
If you need to download Instagram data to CSV format however, you can create a free account with our service and download up to 10 rows daily. Give it a try to see how it works, or keep reading for a step by step guide.
HAR File Scraping Benefits
Before you dive into scraping with HAR files, it’s important to understand pros and cons that we’ll outline here. Aside from our service not being illegal, there are 3 main benefits you should care about.
Easy to Use
Unlike screen scrapers, you don’t need to configure any tools for how to access Instagram or provide technical instructions like CSS selectors to specify what public data you want scraped. Instead, you “drive” the scraping by simply using Instagram as you otherwise normally would. The HAR file will capture ALL data you browse and our HAR File Web Scraper allows you to download the subset you need easily.
Nothing to Install
Since you’re just recording your usage of the Instagram website, you don’t need to install any Chrome plugins, download any software nor connect to any proxies. HAR file recording is built into all four major browsers: Chrome, Firefox, Edge & Safari - so there’s absolutely nothing special you need to install!
No Session IDs or Proxies
When using an Instagram data scraper or unofficial Instagram API, many will ask users to provide their Instagram account login details or session ID. This is a huge red flag and means your Instagram account could bet blocked or permanently banned when you let an automated tool “hijack” your account credentials.
With our HAR file tool, you simply use the Instagram website normally (without violating their Terms of Service) and then follow the instructions above to generate a HAR file and upload it here for processing to collect data.
Downsides to HAR File Instagram Scraping
The largest constraint you have when scraping with HAR files is that you must access every piece of information manually to not violate the Instagram Terms of Service. So if you’re trying to scrape data at scale (e.g. millions of Instagram posts), then don’t waste your time with the HAR file approach.
Instagram User Contact Details
Since you need to manually visit the content you want to scrape using the HAR file approach, it’s not practical to collect contact details, which are only available on individual Instagram users pages, such as phone numbers and emails.
If you need to do this, you can look into using an Instagram profile scraper that will automatically visit and scrape data from individual Instagram users, however these are very risky as they often require you to provide your Instagram credentials to scrape the data from a logged-in Instagram account. We do not recommend going down this route!
Automated Instagram Scraping
If you need to monitor follower counts, engagement stats or perform any other types of regular scraping (that you don’t want to do manually), then the HAR file approach will not work since “automated” access to Instagram is a direct violation of their Terms of Service.
If you absolutely must scrape data from Instagram automatically, you would need to use an illegal web scraper, Instagram profile scraper, post scraper or similar tool that can automatically collect this content on your behalf, and for legal reasons we cannot recommend doing so!
Instagram Scraping Examples
We’ll walk through a few of the most popular examples we see when scraping publicly available data from Instagram. While you can also use the HAR file scraper to extract data from private profiles (as long as you have access to them), we suggest being extremely cautious here to never share this private data since it was never intended to be public data!
Instagram Posts
Our HAR File scraping approach is great for scraping Instagram posts from hashtag search results pages, explore, profile pages and more. Basically anywhere you see a list of posts on the Instagram website, you can scrape the basic details like post captions, post URL, post author ID, Instagram username (and therefore profile URL) and the link to the actual image or video posted.
In order to use this post scraper, you’ll need to navigate to an Instagram URL that contains a list of posts to scrape, such as a profile page, hashtag page or search results page.
By Specific Hashtags
Instagram post data for hashtags is very easy to collect, simply navigate to any hashtag page, begin recording your web traffic, and scroll through all the results. See our detailed article on Instagram Hashtag Post Scraping for an in-depth guide.
By Instagram Profiles
You can also scrape all the Instagram posts from any Instagram account you have access to. Simply browse to the user you want to scrape the posts of, record your web traffic and scroll through the posts to capture the data. See our article on Scraping Instagram User Posts for more information.
By Search Queries
Believe it or not, Instagram can also (barely) function as a search engine! If you’re looking for posts about a very specific topic, you can simply try it as a search term and browse through the results while recording your web traffic. Export this to a HAR file and see what the HAR file web scraper picks up.
Instagram Followers
You can scrape the basic Instagram usernames of followers from any profile you have access to. However, since you’re scraping a basic list of users (not their full profiles), don’t expect to download rich data about each user. You’ll need to visit a profile URL and then scroll through their list of followers to collect the list of names, then export & parse the HAR file to download the username list.
To scrape the followers list, simply visit any profile and click on the text containing their follower count to open up the list. While recording your web traffic, scroll down through the list to capture their information. See our article on Instagram Follower Scraping for details on how to do this.
Instagram Comments
If you have a specific Instagram post that you want to analyze, the comments can be a great way to find interesting Instagram profiles or gauge the sentiment and feedback of your own content (or competitors). Simply navigate to the post, record your web traffic and then scroll through the comments to capture all comments & replies. See our Instagram Comments Scraping article for more details and walkthrough.
Instagram Accounts
Another common use for our service is to find Instagram profiles you may want to reach out to for sponsorship deals.
By Popular Instagram Hashtags
If you’re looking for Instagram profiles in a certain niche, a good way is to start with a list of hashtags that represents your target niche and then find users and posts that are popular & authoritative containing those hashtags.
By Comments
Another good way to spot real influencers is finding Instagram profiles with high levels of engagement in the comments section of a very popular Instagram post.
Scrape Instagram Responsibly
Please keep in mind that most Instagram profiles do not want to be scraped, even if they are public! While you’re typically safe to do any level of private analysis on the data you scrape, re-publishing data (even if publicly scraped Instagram data) can be risky, especially if it contains personal details.
For example, republishing scraped data without permission from the original creator may result in copyright infringement and you must check the Terms of Use for Instagram before doing anything with any collected data. Always get consent before publishing scraped data, even if you think it’s “fair use” - always be a little extra cautious since the data was shared without the expectation of it being re-purposed outside of Instagram.