Instagram User Posts Scraping
If you’re worried about the next 8-hour Instagram outage (or maybe it will just permanently go offline), you may have considered “backing up” your own Instagram profile and downloading all of your content, captions, comments, etc… Alternatively, you may also be interested in doing this to any other Instagram account you have access to - for example to analyze a competing influencers content & calculate engagement stats.
Whatever your need may be for scraping all of an Instagram user’s posts, stats, comments, etc… Instagram likes to make life difficult for everyone (in more ways than one) by prohibiting automated scraping of anything on Instagram. This means that a lot of the Chrome extensions and unofficial “scraper bots” simply will not work, or will risk getting your Instagram account banned if you try using them with your credentials.
HAR File Web Scraping
In this article, we’ll discuss how to use a completely safe approach that uses a recording of your network traffic to Instagram for collecting the raw data behind any Instagram account’s posts. Check out the video or read on to learn more:
Go to the Instagram Profile Page
Open up Instagram in your web browser, then navigate to the profile page you want to scrape the data of the posts for. You can use a search engine to find this page, or simply use the (very limited) Instagram search feature to go to the user’s page.
Open Developer Tools
On your web browser (we’ll assume you’re using Google Chrome, but these instructions should be similar for any browser), right click anywhere on the page and hit Inspect to open up developer tools (this will allow you to record your web traffic as Instagram serves back the user’s posts). You can then click on the “Network” tab to see your web traffic with the data in it.
Click Through Posts
Now with developer tools open, click on the first page to load it into your browser (along with some initial comments, engagement stats, etc…). It will appear with a greyed-out background as well as arrows on the left and right you can use to go through the rest of the user’s posts:
Now click the “right” arrow to keep loading more and more posts from the user. Don’t go too fast so Instagram doesn’t think you’re a bot.
Export HAR File
Once you’re done clicking, look for a download arrow in your browser’s developer tools (Network Tab) that’s labeled something along the lines of “Export HAR…” Click this and save the HAR file somewhere you can access quickly, like your Desktop:
Upload to HAR File Web Scraper
Now that we have the data inside this large file, the trick is to get the data out of this file. For this, we suggest using our HAR File Web Scraper where you can upload the file here, and the software will parse everything out for you and group together all the different requests with the user’s posts in it:
You can click the disk icon next to each of the requests to download the data for free, or scroll up and click the “Parse Group” button (above all the individual responses) to download a combined CSV file with the user’s posts in it (note this is a paid feature):
You can see that the HAR File Scraper combined all of the individual requests into a combined list that you can now download as a single CSV file.