Helpful Product Data for E-Commerce Businesses
When researching which products to sell, it can be very helpful to see what your competitors or the rest of the market is up to. This is where scraping data from Wayfair comes into play, which for some reason doesn’t seem to make it easy to collect data for market research from its site (possibly to protect their own products & market share).
Fortunately though, we have a totally legal way you can extract data out from Wayfair Search Results while complying with the Terms of Service by loading your browsing session data into a HAR file recording and then extracting the data from the recording instead of the Wayfair website as to not violate Wayfair’s Terms of Service.
1. Browse Wayfair Products
Head to the Wayfair Search Results Feature and run a search for any product or category you want to scrape the data for. You can apply any special filters you’d like, just use the website as you normally would. Once happy with the results, right click on the page and hit “Inspect” to open developer tools and begin recording your network traffic.
Now refresh the page to reload the initial results and then scroll through the results, hitting the next button as needed. This will load more and more Wayfair data into your browser now that you’re recording.
2. Export & Download
Once you’ve scrolled through the data, select the “Network” tab under developer tools and click the down arrow labeled “Export HAR…” to download a HAR file containing the network data from Wayfair. Upload that file to the HAR File Web Scraper and look for the parse groups with Wayfair product data in them. Click “Parse Group” and you’ll then see downloadable collections you can save to your computer in analyze in Excel or your favorite spreadsheet program.