Scraping the Realtor.com API
If you need to scrape real estate data from Realtor.com, we’ll show you how to legally do this by recording your web traffic to the Realtor.com API as you browse and use their website in full accordance with their Terms of Service.
Our Realtor.com scraper lets you export housing market data to CSV & Excel spreadsheets using only your web browser so there’s nothing to install. Simply use the Realtor website while recording your web traffic, and we then scrape the data out from the recording instead of the Realtor website so it is completely legal & undetectable.
1. Browse Realtor Listings
To download public Realtor data, head to the Realtor Website and enter a search for whichever city or area you want to scrape the listings data for (e.g. the Miami Listings Map).
Make sure you click on the “Map” option on the upper right part of the page. You must browse through the listings in map mode, otherwise the recording will not pick up the correct data!
Once you have the map loaded, right click on the page and hit “Inspect” to open up developer tools, which will record your web traffic. Your browser should look something like the following:
Now that you’re recording web traffic, simply pan around the map and/or zoom in to the specific areas you want to scrape the listings for. You can keep doing this and sweeping through different areas, you’re just trying to get the Realtor website to send you as many datapoints as you need to scrape.
To verify the data is loading correctly, you can type in search_ in the filter under the “Network” tab in developer tools (as shown in the above screenshot) and you’ll see the raw network data used to populate the listings on the map.
2. Export a HAR File
When you’re done browsing the listings (you should be able to scrape a few thousand), click on the down arrow in the “Network” tab in developer tools labeled “Export HAR…” to download a HAR file containing the raw network data we were recording.
Upload that file to the HAR File Web Scraper and look for the group ending in rdc_search_srp (this may be subject to change if Realtor changes this) and click “Parse Group” to extract out the data from those requests.
3. Download Realtor Data
Once the parse finishes, you’ll want to look for a collection named data › home_search › properties for the Realtor housing listings. Simply click “Download CSV” to download the Realtor data into CSV format which you can then import into Excel.
Listing Data
The downloadable file will contain all of the fields you can see on the public search results page, with a few extra fields that may be of interest.
- Property ID
- Asking Price
- Listing ID
- Matterport Available?
- Status (For Sale, Ready to Build)
- Permalink to Realtor.com
- Price Reduction Amount
- Listing Date
- Primary Photo URL
- Community Permalink
- Community Name
- Community Office Hours
- Community Phone Number
- Bed Count
- Bathroom Count
- Square Footage
- Lot Square Footage
- Listing Type (Single Family, Condo, Townhouse, Mobile, Other)
- Sold Price
- Date Sold
- Google Street View URL
- Street Address
- City
- Latitude & Longitude Coordinates
- County Name
- Is New Listing?
- Is Price Reduced?
- Is Foreclosure?
- Is New Construction?
- Is Pending?
- Realty Group Name
- Realty Group Photo URL
- Realty Group Website
- Virtual Tour URL
- Open House Start & End Dates
Removing Duplicates
If you have more than one request in your HAR file parse group (meaning you panned around the map a bit), then you’re likely to have duplicates in your data set. You’ll want to remove them by using the permalink column in the downloadable CSV file, which serves as a unique identifier for each property listing as it references the URL for the property.
You can also see the property_id and listing_id columns, however these can get tricky because the Listing ID value can be null for some listings, so this is why we suggest using permalink as an identifier, but this is all of course subject to change.