Scraping the Redfin API
You can legally download Redfin housing market data directly from their public website (without using their limited Redfin data export tool, which was recently restricted due to MLS demands) using a little-known technique involving HAR files to record the data that the Redfin API sends to your web browser.
Our Redfin scraper will let you export Redfin data to Excel spreadsheets using the following methods which are relatively painless and most importantly do not violate the Redfin Terms of Service since we’ll be web scraping a recording of your web traffic instead of the actual Redfin website to extract real estate data.
1. Browse Redfin Listings
To download Redfin’s public data, simply head to Redfin’s Website and enter a search term for the city or area you want to download the data from or browse to an existing city page, like Redfin New York or Redfin Miami listings.
Next, right click somewhere on the page and hit “Inspect” to open up developer tools to begin recording your web traffic. This will allow us to legally scrape the recording (as opposed to the website) so we do not violate Redfin’s Terms of Service or get blocked for making sketchy requests. Once Developer Tools are open, click on the “Network” tab and you should see something like the following:
Now that we’re recording web traffic, you just need to browse through the real estate listings you want to scrape on the map so they’re loaded into the browser. You typically just need to move the map and wait for the site to reload the listings. To verify that they loaded, type in /api/gis? under the “Filter” field in the developer tools Network tab and you should see something like this:
2. Export a HAR File
Now we need to get the data into the HAR File Web Scraper via a HAR File your browser will export. Look for the down arrow under the Network tab in developer tools labeled “Export HAR…” and click that to download Redfin’s data in the form of a HAR file.
Upload the HAR file to this page under the HAR File Web Scraper section and look for the group ending in /api/gis which will hold the network requests containing the downloadable Redfin data:
3. Download Redfin Data
On the next screen you’ll want to look for a collection named payload > homes for the Redfin listings. Simply click “Download CSV” to get all of the Redfin data from your web browsing session.
Listing Data
You’ll see the following information in your downloadable spreadsheet (basically anything that Redfin publicly shows on its website will be included):
- Redfin Listing ID
- MLS Status (Active, Coming Soon, Sold)
- MLS ID
- Number of Bedrooms
- Number of Bathrooms
- Street Address
- Unit Number
- City
- State
- Zip Code
- Date Sold
- URL to Property Pages
- Location Name
- Description
- HOA Fee
- Square Footage
- Price per Square Foot
- Latitude & Longitude Coordinates
- Year Built
- Time Listed on Redfin
- Number of Photos
- Virtual Tour Link
- Real Estate Agent Name
- Real Estate Agent Phone Number
- Broker Phone Number
Removing Duplicates
If you browse around the Redfin map a bunch while recording your web traffic in the earlier steps, you will likely get some overlap and duplicate results when you extract data. Therefore, it’s very important to deduplicate your Redfin data using the listingId column in the downloadable file.
Otherwise, you may end up running an analysis on duplicated rows which will inflate or impact calculations like average & median sales pricing, time on market, square footage statistics, etc…
Redfin’s Downloadable Data
Before you jump too much into data scraping Redfin, you may want to check out Redfin’s Publicly Downloadable Data section on its website. While it will not show individual listing data as we were able to scrape in the previous section, it may be useful if you just need to perform an overall market analysis and look at trends over time.
You can also download this aggregate data, just look for the little arrow on the lower right part of the visualization which will contain export options.