Scraping the Compass Real Estate API
If you need to scrape real estate data from Compass, you may be upset to learn they don’t offer a public real estate API. But don’t worry, as we can legally scrape their search result listings for any market to quickly download listings for rentals & for sale properties from Compass.
The process involves simply recording your web traffic as you browse through the Compass website normally, in full accordance with their Terms of Service, and we then perform the scraping on the recording of the web traffic instead of the actual Compass website. This means our scraping is not only 100% undetectable, but also completely safe & legal so you’ll never need to worry about getting blocked or banned.
1. Browse Compass Listings
To get started, head to the Compass Website and run a search for any market you’re interested in collecting data from. In this case we’ll use Miami, Fl as our example, but you can use any area that Compass supports.
Make sure you see a map on web page as shown in the screenshot above. Then apply any filters you’d like to narrow down your search to exactly the types of homes, condos, etc… you’d like to scrape. Once you’re happy with the results you see on the page, right click on the browser and hit “Inspect” to open up developer tools.
This will begin automatically recording your web traffic to capture the data that Compass sends to you. Now move the map ever so slightly to force the Compass Real Estate API to send new results to your browser. Keep panning the map around (zooming in and out as needed) to sweep through all of the listings you’d like to scrape.
2. Export a HAR File
Once you’ve panned through enough listings, look for the “Network” tab in Developer Tools and click the down arrow labeled “Export HAR…” to download a HAR file containing the combined JSON data that the Compass API sent to your browser, containing the real estate information.
Upload that file to the HAR File Web Scraper and look for the group beginning with /homes-for-sale/ similar to the group shown above. If you’re not sure or see multiple groups, search the parse page for lolResults which will appear in the Fields section of the group containing the real estate data. Click “Parse Group” on this group to get the data out.
3. Download Real Estate Data
Once the HAR file parse finishes, you’ll see the extract real estate listings under the lolResults › data collection, where you’ll see a lot of extra details not presented in the search results on Compass.
Below are some columns you may find helpful, but this is only a subset and there’s much more information extracted from the HAR file in the parse you may find useful, so be sure to try for yourself! If you pan through overlapping pages on the map while scraping, be sure to de-duplicate the results by Listing ID to remove the overlapping listings!
- Listing ID (Use for de-duplication)
- Parcel Number
- External ID (e.g. MLS ID)
- Compass Listing URL
- Property Street Address
- Neighborhood
- City
- State
- ZIP Code
- Latitude & Longitude Coordinates
- County FIPS Code
- MLS Neighborhood
- Square Footage Size
- Last Known Price
- Price Listed
- Price per Square Foot
- Monthly Sales Charges
- Monthly Sales Charges with Taxes
- Price Negotiability
- Date Listed
- Date Last Updated
- Days on Market
- Number of Units in Building
- Building Construction Year
- Total Parking Spaces
- Is Waterfront?
- Property Type
- Property Tax
- Property Tax Assessment Year
- Land Value
- Land Value Assessment Year
- Amenities
- Listing Image URLs
- Unit / Apartment Number
- Number of Bedrooms
- Number of Full Bathrooms
- Number of Total Bathrooms
- Building Service Level (Doorman, Concierge, Full-Time Doorman)
- Outdoor Space