Scraping the Facebook API
If you need to export data from a Facebook group, page or post then we’ll walk you through what options you have for legally doing this without violating the Facebook Terms of Service. You’ll be able to download basic information into CSV files for general lists of items, either from any public resource or private resource that you have access to with our Facebook account scraper.
We do not cover nor support scraping individual contact details (either from profiles or businesses). If you need to do this, there are plenty of better methods to collect business contact details such as through our Google Maps Data Scraper.
Getting Started
To scrape any Facebook group, post or page’s data, simply visit the page in your web browser (we suggest Google Chrome) and navigate to the list you want to scrape (like comments, events, members, etc…) and then right click on the page and hit “Inspect” to open up developer tools. This will begin recording your web traffic, so we suggest refreshing the page to get a fresh load of data in while recording:
Then scroll through the data you need (e.g. the list of members, a post’s comments, etc…) and when you’re done, click the Network tab in developer tools and select “Export HAR…” to download a HAR file containing the raw JSON network data Facebook sent to your browser (which you have a legal right to inspect and download).
Then upload that file to the HAR File Web Scraper and look for any section labeled with /api/graphql/ - these are the network responses containing the structured data Facebook sends to your browser:
We suggest clicking the smaller “Parse JSON” button on the lower section when getting started to try with a small set of data first. From there, you’ll see downloadable collections. Try to look for the collections with more than a few rows in them, as those are probably what you need:
Caveats
We’ve found the Facebook JSON data to be extremely cumbersome and difficult for certain use cases with our Facebook scraping tools. For example, when scraping a group’s posts it tends to have a lot of trouble, but when scraping announcements it seems to have an easier time. You may need to try several times depending on what you’re doing, or try accessing the data you want differently. If you need help, just reach out to us in your Stevesie Account through the chat widget.
You may also want to see our Instagram Data Scraper as it works much better and will give you a smoother experience for legally extracting social media data from your own web traffic.
Groups
Facebook groups are a great way to stay connected and meet new people. They can also be a good source of information for market research as Facebook group members all share a common interest. You can scrape group posts for some lightweight information from a single group, such as announcements, post comments, events and basic user information with our Facebook group scraper.
You’ll need to have access to the Facebook group if it’s private and you’ll be able to scrape the list of basic usernames, URLs and IDs of who is in the group. This is done through our HAR File Web Scraper, which extracts data from a recording of your web traffic (instead of the Facebook website) so it’s 100% undetectable and not governed under Facebook’s Terms of Service.
It’s important to note that since you’re scraping a list of basic information, you will only be able to collect this limited data back (e.g. it’s typically just the name, link to profile photo, and user ID) with our Facebook URL scraper. You will NOT be able to scrape emails from Facebook, contact details, or specific information like this using our service.
Pages
If you want to scrape posts from popular pages, you want to scrape them one page at a time with our Facebook page scraper. This will allow you to more cleanly scroll and extract data from Facebook by organizing your HAR files based on the Facebook pages you target.
For example, if you want to extract Facebook posts for 2 Facebook pages, you would start with the first page, record a new HAR file, scroll through the posts, members, etc… and then export your HAR file.
From there, you can hit the “Clear” icon in the “Network” tab to reset the recording or just open a new tab before moving on to the second or more Facebook pages to ensure you have a clean HAR file for each page.
As mentioned earlier, you’ll only extract basic information about each post using our Facebook post scraper, such as caption, media URL, like count, etc… If you need more information on the posts, you can create the Post URL by simply adding the Post ID to this format (you’ll need to substitute the page name): facebook.com/PAGE_NAME/posts/POST_ID
Why Scrape Facebook?
Most online businesses have some relationship with Facebook, either directly through their own Facebook pages and/or indirectly through word-of-mouth on Facebook posts. Tracking this data from Facebook & other social media platforms helps businesses gain valuable insights into their marketing efforts and finding new customers.
In this article, we’ll walk through a few of the most popular reasons for how our customers are using Facebook data and provide an overview on what your scraping options are to get this data - using our service, a similar one, or your own code.
Market Research
Whether planning a new business or scaling an existing one, having as many data points as possible on who your customers are and what other solutions they’re using is critical for success.
There are countless Facebook Pages about any topic you can imagine, and analyzing the content of these posts and members can help guide your marketing efforts.
Competitor Analysis
If you plan on being on Facebook, then odds are your competitors are on Facebook too. You’ll likely want to gain some competitive intelligence from their Facebook pages to get a feel for the content they post, who their followers are, and the types of people who engage with their posts through comments.
Another popular method of gaining competitive intelligence is to analyze the data behind the Facebook ads your competitors are running. This way you can see which ads they’ve been running and for how long. Ads that have been running the longest are likely to be the most successful, so you can “borrow” ideas from these creatives.
You can see the public data for any Facebook advertiser on the Facebook Ads Library - just enter in your competitor names (or even an entire industry) to see the ads that are running. To collect this data for your own use, see our Facebook Ads Scraper.
Creative Ideas
If you create websites, apps or other creative assets, then you may want to draw inspiration from similar Facebook pages in your industry or from related pages & posts. For example, you may want to collect the images or videos from popular posts with positive comments across a variety of Facebook pages to use as examples or inspiration for future work.
You do need to be careful though when working with a media URL or file captured from Facebook, as they retain copyright from the original owner. While it’s probably fine to access & extract these files for private use and research, never blindly re-publish them without first consulting competent legal counsel.
E-Commerce
If you’re selling products online, you may be interested in collecting data in Facebook marketplace to see what products are available, their pricing, descriptions, images and videos. Since Facebook offers this data to the public, it’s easy to gather this information for some preliminary research (especially in your local market) to see what’s already trading and selling.
Legal Concerns
Facebook is notorious for shutting down other startups (not us) that attempt to work with its public data through legal threats & lawsuits. This is because all other Facebook Scrapers rely on interacting with Facebook either through your browser or through web scraping the backend, which is technically a violation of Facebook’s Terms of Service.
Although the Facebook Terms don’t allow us to extract data using “automated” means, they still need to send data to our web browsers in order to function, and we can “record” the data they send to us and then scrape the data from a recording after we’re done using Facebook where their Terms of Use do not apply. This means scraping from a recording is not only totally legal, but also 100% undetectable and therefore unblockable.
Scraping from a Recording
All modern web browser have a little-known feature that allows anyone to record their web traffic while using any website, such as Facebook. So if we simply use Facebook “normally” while recording our web traffic, then we can legally scrape any Facebook URL from the recording instead of from their website.
This means the scraped data will originate from the Facebook website, then as we interact with Facebook’s website the data will be copied to a local file on our computer that we can then use the following freemium service we provide to get the data out.
HAR File Web Scraper
To legally collect Facebook data, see the instructions on this page under the “HAR File Web Scraper” and follow step 1: open a new browser tab, then right click anywhere on the new tab and hit Inspect to begin recording your web traffic or see the below above for a more in-depth tutorial.
Now head to the Facebook URL or page you want to scrape in your new browser tab (that’s recording the web traffic), and browse through the data you want to scrape. This type of data collection will only work on interactive elements on the Facebook website such as scrolling through lists of Facebook pages in search results, page members or comments.
Each request you make to Facebook for more data (as you scroll through a list of items) will be captured in the recording as structured JSON data, and our HAR file web scraper will pull the data from this file instead of from Facebook directly.
See our detailed video on Legally Scraping Facebook for a full step-by-step guide. Once you’re done browsing through the content you want to scrape, go to the “Network Tab” in the developer tools panel and click the down arrow labeled “Export HAR…” to download a file containing Facebook data.
Upload that file to the HAR File Web Scraper and look for the data you want. You may want to start with the group ending in “/api/graphql/” and click the “Parse Group” button. That will open up a new page allowing you to download the Facebook content as a CSV file.
You’ll be able to easily collect public data such as profile name, profile URL, profile image URLs, comments, posts & more with our Facebook profile scraper. You will not be able to scrape contact information like scrape Facebook emails, phone, etc… using this tool, even if available on public pages!
Drawbacks of HAR File Scraping
The key thing to know about this form of web scraping is that since it requires you to manually browse through the data you need, it will not work well for large-scale Facebook scraping. For example, if you need to scrape 1,000s of Facebook pages, companies, links or URLs, you’re going to have trouble.
Instead, we suggest this method of Facebook scraping for more targeted web scraping endeavors, such as if you want to scrape Facebook comments from page posts of your direct competitors or for public data in a very narrow domain.
If you’re interested in leveraging Facebook data for your own business, you have a number of options for collecting data from Facebook - whether you gather data manually or decide to automate Facebook data collection, we’ll walk through the major concerns you may want to consider.
Illegal Facebook Scrapers
Can using an automated Facebook scraper tool get you banned? Yes! It may even get your sued for violating Facebook’s Terms of Service, which explicitly forbids scraping. Furthermore, Facebook employs an army of engineers whose sole purpose is to prevent unauthorized scraping!
But if web scraping Facebook pages is illegal, then how are there so many Facebook scrapers on the market today (e.g. Facebook phone number scrapers)? To clarify, if you automatically collect Facebook data on your own (e.g. via your own code) then you are simply violating the Facebook Terms and nothing is technically illegal - Facebook could just ban you.
However, the moment you involve a third party, such as a proxy provider or Facebook scraping service, then that 3rd party is guilty of Tortious Interference with Contract and Facebook as the legal basis to sue these scrapers. And they have, filing many lawsuits against scraping for hire & similar tools that ultimately shut down many operations and explain why so many new ones pop up every year.
Unauthorized Scraping
Since we don’t enjoy being sued, we can’t legally suggest that you engage in scraping data from Facebook via unauthorized methods (e.g. anything automated). However, we can walk you through the most common issues you’ll face if you go down this route.
Getting Blocked & Banned
Since you need a Facebook account to access information hosted on Facebook, most Facebook scrapers need to log in as a Facebook user to access the site and extract data. If you try to access Facebook with a tool that asks you to provide your username and password, then uninstall it immediately!
If you provide your username & password to these tools, they will start accessing Facebook using your account - and once detected Facebook will likely give you a warning or straight up ban your account and/or IP address for attempted data scraping & exceeding rate limits. This means saying goodbye to all your saved photos, friends & communities!
Proxy Server Costs
More advanced tools will attempt to circumvent these account and IP blocks using proxies for data collection purposes, logging in through multiple fake “puppet” Facebook accounts solely used for web scraping.
Not only does this approach require nearly unlimited time & programming skills, but it will also get very expensive, as you’ll need to pay quite a lot for premium proxy services to try and outsmart Facebook’s request blocking.
Although many large-scale operations have gotten away with this in the past (we don’t want to name anyone specifically), they are now facing legal consequences for reselling extracted data from Facebook.