Facebook Data
Most online businesses have some relationship with Facebook, either directly through their own Facebook pages and/or indirectly through word-of-mouth on Facebook posts. Tracking this data from Facebook & other social media platforms helps businesses gain valuable insights into their marketing efforts and finding new customers.
In this article, we’ll walk through a few of the most popular reasons for how our customers are using Facebook data and provide an overview on what your scraping options are to get this data - using our service, a similar one, or your own code.
Market Research
Whether planning a new business or scaling an existing one, having as many data points as possible on who your customers are and what other solutions they’re using is critical for success.
There are countless Facebook Pages about any topic you can imagine, and analyzing the content of these posts and members can help guide your marketing efforts.
Competitor Analysis
If you plan on being on Facebook, then odds are your competitors are on Facebook too. You’ll likely want to gain some competitive intelligence from their Facebook pages to get a feel for the content they post, who their followers are, and the types of people who engage with their posts through comments.
Another popular method of gaining competitive intelligence is to analyze the data behind the Facebook ads your competitors are running. This way you can see which ads they’ve been running and for how long. Ads that have been running the longest are likely to be the most successful, so you can “borrow” ideas from these creatives.
You can see the public data for any Facebook advertiser on the Facebook Ads Library - just enter in your competitor names (or even an entire industry) to see the ads that are running. To collect this data for your own use, see our Facebook Ads Scraper.
Creative Ideas
If you create websites, apps or other creative assets, then you may want to draw inspiration from similar Facebook pages in your industry or from related pages & posts. For example, you may want to collect the images or videos from popular posts with positive comments across a variety of Facebook pages to use as examples or inspiration for future work.
You do need to be careful though when working with a media URL or file captured from Facebook, as they retain copyright from the original owner. While it’s probably fine to access & extract these files for private use and research, never blindly re-publish them without first consulting competent legal counsel.
E-Commerce
If you’re selling products online, you may be interested in collecting data in Facebook marketplace to see what products are available, their pricing, descriptions, images and videos. Since Facebook offers this data to the public, it’s easy to gather this information for some preliminary research (especially in your local market) to see what’s already trading and selling.
Using a Facebook Scraper
If you’re interested in leveraging Facebook data for your own business, you have a number of options for collecting data from Facebook - whether you gather data manually or decide to scrape Facebook data automatically, we’ll walk through the major concerns you may want to consider.
Illegal & Automated Scrapers
Can using an automated Facebook scraper get you banned? Yes! It may even get your sued for violating Facebook’s Terms of Service, which explicitly forbids scraping. Furthermore, Facebook employs an army of engineers whose sole purpose is to prevent unauthorized scraping!
But if scraping Facebook is illegal, then how are there so many Facebook scrapers on the market today? To clarify, if you directly scrape Facebook data on your own (e.g. via your own code) then you are simply violating the Facebook terms and nothing is technically illegal - Facebook could just ban you.
However, the moment you involve a third party, such as a proxy provider or Facebook scraping service, then that 3rd party is guilty of Tortious Interference with Contract and Facebook as the legal basis to sue these scrapers. And they have, filing many lawsuits against scraping for hire & similar tools that ultimately shut down many operations and explain why so many new ones pop up every year.
Unauthorized Scraping Concerns
Since we don’t enjoy being sued, we can’t legally suggest that you engage in scraping Facebook via unauthorized methods (e.g. anything automated). However, we can walk you through the most common issues you’ll face if you go down this route.
Getting Blocked & Banned
Since you need a Facebook account to access information hosted on Facebook, most Facebook scrapers need to log in as a Facebook user to access the site and extract data. If you try to scrape Facebook with a tool that asks you to provide your username and password, then uninstall it immediately!
If you provide your username & password to these tools, they will start scraping Facebook using your account - and once detected Facebook will likely give you a warning or straight up ban your account and/or IP address for attempted data scraping & exceeding rate limits. This means saying goodbye to all your saved photos, friends & communities!
Proxy Server Costs
More advanced tools will attempt to circumvent these account and IP blocks using proxies for data collection purposes, logging in through multiple fake “puppet” Facebook accounts solely used for web scraping.
Not only does this approach require nearly unlimited time & programming skills, but it will also get very expensive, as you’ll need to pay quite a lot for premium proxy services to try and outsmart Facebook’s request blocking.
Although many large scale operations have gotten away with this in the past (we don’t want to name anyone specifically), they are now facing legal consequences for reselling extracted data from Facebook.
Collect Data from Facebook Legally
So what’s the point of this article if we keep warning you against scraping Facebook data? Well there is a legal way to scrape Facebook that we can help you with - the key is that as long as you don’t violate the Facebook Terms of Service, there’s nothing Facebook can do to block us from collecting data.
Since the Facbeook terms don’t allow us to extract data using “automated” means, it simply means that we can scrape Facebook by just using the site normally and then extracting the data after the fact.
Recording Facebook Web Traffic
All modern web browser have a little-known feature that allows anyone to record their web traffic while using any website, such as Facebook. So if we simply use Facebook “normally” while recording our web traffic, then we can legally scrape any Facebook URL from the recording instead of from their website.
This means the scraped data will originate from the Facebook website, then as we interact with Facebook’s website the data will be copied to a local file on our computer that we can then use the following freemium service we provide to get the data out.
Scraping Facebook From a HAR File
To start scraping Facebook legally, see the instructions on this page under the “HAR File Web Scraper” and follow step 1: open a new browser tab, then right click anywhere on the new tab and hit Inspect to begin recording your web traffic.
Now head to the Facebook URL or page you want to scrape in your new browser tab (that’s recording the web traffic), and browse through the data you want to scrape. This type of data collection will only work on interactive elements on the Facebook website such as scrolling through lists of Facebook pages in search results, page members or comments.
Each request you make to Facebook for more data (as you scroll through a list of items) will be captured in the recording as structured JSON data, and our HAR file web scraper will pull the data from this file instead of from Facebook directly.
See our detailed video on Legally Scraping Facebook for a full step-by-step guide. Once you’re done browsing through the content you want to scrape, go to the “Network Tab” in the developer tools panel and click the down arrow labeled “Export HAR…” to download a file containing Facebook data.
Upload that file to the HAR File Web Scraper and look for the data you want. You may want to start with the group ending in “/api/graphql/” and click the “Parse Group” button. That will open up a new page allowing you to download the Facebook content as a CSV file.
You’ll be able to easily collect public data such as profile name, profile URL, profile image URLs, comments, posts & more. You will not be able to scrape contact information like phone, email, etc… using this tool, even if available on public pages!
Drawbacks of HAR File Scraping
The key thing to know about this form of web scraping is that since it requires you to manually browse through the data you need, it will not work well for large-scale Facebook scraping. For example, if you need to scrape 1,000s of Facebook pages, companies, links or URLs, you’re going to have trouble.
Instead, we suggest this method of Facebook scraping for more targeted web scraping endeavors, such as if you want to scrape Facebook pages of your direct competitors or scrape Facebook for public data in a very narrow domain.
Scraping Multiple Facebook Pages
If you want to scrape Facebook posts from popular pages, you want to scrape them one page at a time using HAR file web scraping. This will allow you to more cleanly scroll and extract data from Facebook by organizing your HAR files based on the Facebook pages you target.
For example, if you want to scrape Facebook posts for 2 Facebook pages, you would start with the first page, record a new HAR file, scroll through the posts, members, etc… and then export your HAR file.
From there, you can hit the “Clear” icon in the “Network” tab to reset the recording or just open a new tab before moving on to the second or more Facebook pages to ensure you have a clean HAR file for each page.
As mentioned earlier, you’ll only extract basic information about each post using our Facebook scraping tool, such as caption, media URL, like count, etc… If you need more information on the posts, you can create the Post URL by simply adding the ID to this format (you’ll need to substitute the page name): https://www.facebook.com/PAGE_NAME/posts/POST_ID
Scraping Facebook Group Members
Another common use case we see is to extract the users from a Facebook group URL. Using our HAR file tool, you’ll need to first request access to the Facebook group you want to join and then while recording your web traffic, simply interact with the group member list and scroll through the list.
It’s important to note that since you’re scraping a list of basic information, you will only be able to collect this limited data back (e.g. it’s typically just the name, link to profile photo, and user ID). You will NOT be able to scrape emails, contact details, or specific information like this using our service.