Watch on YouTube
Scraping the SpyFu API
Scrape data from SpyFu’s research tool (while logged in to your account, or from their publicly available trial data) into downloadable CSV files using HAR files. This approach is completely legal as you are simply recording your web traffic from SpyFu into a HAR file and then scraping your recording (instead of the actual SpyFu website).
1. Browse SpyFu Data
Head to SpyFu and enter any search term or domain you want to scrape the data for. You’ll then want to select a section of data you want (e.g. SERP Analysis) to open up a table with the full results that you want to scrape.
Now right click on the page and hit “Inspect” to open up developer tools and record your web traffic. This will capture the raw network data SpyFu sends to your browser. No refresh the page to force SpyFu to resend the data to your browser while you’re recording.
Paginate through the results if needed or click around to load some more data. Whatever SpyFu interactively sends to your browser, we’ll be able to scrape.
2. Export a HAR File
After you’re done browsing, go to the “Network” tab under developer tools and click the down arrow labeled “Export HAR…” to download a HAR file containing all of the raw data from SpyFu in it. Then upload that file to the HAR File Web Scraper, which will parse out the data into groups so you can download all of the related data from SpyFu together.
3. Download SpyFu Data
Click “Parse Group” on the relevant group from the previous step and you’ll see downloadable collections. Select the ones you want and click download to save them to your computer and you can then open them in any spreadsheet program you’d like, such as Excel.