Scraping the BigSpy API
Scrape advertising results from your BigSpy subscription into downloadable CSV files legally using a little-known technique involving HAR files. The approach below will let you record the raw data BigSpy sends to your browser as you use their service in full compliance with their Terms of Service. The scraping then occurs after the fact on the recording instead of on the BigSpy website, so it is perfectly legal.
1. Browse BigSpy Data
Head to any tool on BigSpy you’d like to scrape the data from, such as from the BigSpy Facebook Ads Directory and apply any relevant filters you may need. Now right click on the page and hit “Inspect” to open up developer tools and begin recording your web traffic.
Now refresh the page to force BigSpy to resend its data to your browser (since you’re now recording) and browse through the data you want by scrolling and going through all the pages you’d like to scrape.
2. Export a HAR File
Once you’ve browsed through the data you want to scrape, look for the “Network” tab under developer tools and click the down arrow labeled “Export HAR…” to download a HAR file containing a recording of all the raw data that BigSpy sent to your browser.
Upload that file to the HAR File Web Scraper and we’ll parse the data in the file into groups you can easily extract the data out of. Look for the group containing relevant advertising fields and click “Parse Group” to extract it out.
3. Download BigSpy Data
Once you parse a group, you’ll be able to download it into CSV files. Just browse through the collections and look for the ones with the relevant advertising data in it, then you can download the file and analyze the ad data using any spreadsheet program like Excel.