Scraping the Mercari API
Scrape product data directly from the Mercari website without violating their Terms of Service or getting blocked by recording the data that the official Mercari API sends to your web browser.
We’ll show you how to do this using a little-known technique involving HAR files to scrape search results or categories from Mercari.
1. Browse Mercari Products
Head to the Mercari Website and navigate to any category or enter any search term you want to scrape the product data for. You can also apply any filters or sort options you want, just select them on the website as you’d normally do.
Once you have the search results loaded, right click on the page and hit “Inspect” to open up developer tools and record your network web traffic. This will allow you to scrape the data from this recording, which will be 100% undetectable.
Now refresh the page (to force Mercari to send the data again now that you’re recording) and then scroll down through the results to get Mercari to keep loading more and more pages of data. You may need to click the “Show More” button as you scroll down.
2. Export a HAR File
Once you’ve scrolled through all the products in the search results, find the “Network” tab under developer tools and look for the down arrow labeled “Export HAR…” and click it to download a HAR file containing the Mercari data.
Upload that file here to the HAR File Web Scraper and look for the group labeled /v1/api in the header. If you see more than one, look for the group with originalPrice (or similar) in the field list as shown above.
3. Download Mercari Data
Once the group is parsed, you’ll be able to download the combined paginated product data into a CSV file in one click in the data › search › itemsList collection.
The data will contain one row per product and contain the following fields you may find useful for your e-commerce research:
- Product Title
- Description
- Original Price
- Sale Price
- Category Name & ID
- Shipping Policy
- Mercari Seller ID
- Product Brand
- Product Size
- Item Condition
- Photo URL
These fields may change depending exactly what type of product you’re scraping (e.g. you may get storage capacity when scraping electronics instead of T-Shirt sizes as in this example). The best way to see what data you can get is to try for yourself!