FB Pixel

🏑 Scraping ALL Airbnb Listings from a City By steve


βš–οΈ Disclaimer: This article may reference endpoints that are not part of official APIs endorsed by airbnb.com and were found while using airbnb.com's official website and/or mobile app. They are documented here for informational purposes, such as to cross reference with HAR Files after using airbnb.com's official website and/or mobile app in accordance with airbnb.com's Terms of Service. Stevesie has no affiliation with airbnb.com.

If you access any of these endpoints with Stevesie or any other tool outside of an official airbnb.com client, you must check airbnb.com's Terms of Service to ensure said access is not prohibited. If you are not sure whether or not your use of Stevesie or any other tool in a specific instance violates airbnb.com's Terms of Service or applicable law, you should consult with competent legal counsel before proceeding. Learn more here: Is Data Scraping Legal?

If you’re looking to collect ALL the Airbnb listings for a location, not just the first 300 that Airbnb returns by default, you’ve found the right article.

If you go to Airbnb’s Official Website and search for listings, you’ll notice that you can only paginate up to about 300 before you get cutoff. If you’re searching within a large city, you know there’s way more than 300 listings! So how to we get them all?

The trick here is to note that Airbnb returns 300 listings per search and offers pretty powerful search tools. For example, we can search for listings that are within a specific price range and/or of a specific property type (e.g. apartment, house, or even a boat! ⛡️).

We can combine these 2 filters (price and property type) we can perform many “micro-targeted” searches and get back 300 results per search, e.g. we can run a search for apartments between $100 and $200, then another search for apartments $201 - $300, then one for full houses $100 - $200, then again for $201 - $300 and so on.

These micro searches will return up to 300 results per search, and because they are mutually exclusive from one another (non-overlapping), we won’t get duplicates either.

So let’s get started!

Step 1 - Determine Property Type IDs & Price Ranges

You’ll first want to get a “lay of the land” to figure out exactly how to build your “micro-targeted” searches, as these will be different in each city based on price ranges and property types.

You can import the Airbnb Paginated Listings Workflow Formula to see how to do this, just import the workflow and enter your target city in the input collection (where it will prompt you):

Set Your Target City

You can either execute the workflow (please note the disclaimer) or follow the URLs in the execution plan to manually collect the listings data. Either way, you’ll get back about 300 Airbnb listings for your target city you can use as a broad sample for subsequent searches.

Property Type IDs

Within the response data, you’ll want to look for the explore_tabs.sections.listings.listing.property_type_id field, which will show you which property types are popular in the city you’re interested in. Copy this list of IDs, these will be the property types you’re interested in and we’ll use them later!

Price Ranges

You can see the average nightly price in the explore_tabs.sections.listings.pricing_quote.rate.amount field and get an idea of what the distribution looks like. Another helpful tool is the histogram Airbnb provides on its website when you search for a location, you can open it up and get an idea of the pricing distributions and which segments you’ll want to target:

Airbnb Website Price Segments

You can try to define specific price ranges if you’d like, or just use a generic definition like this JSON list that will go through 10 price ranges:

[{"max_price": 100}, {"max_price": 200, "min_price": 101}, {"max_price": 300, "min_price": 201}, {"max_price": 400, "min_price": 301}, {"max_price": 500, "min_price": 401}, {"max_price": 600, "min_price": 501}, {"max_price": 700, "min_price": 601}, {"max_price": 800, "min_price": 701}, {"max_price": 900, "min_price": 801}, {"min_price": 901}]

Step 2 - Run Micro-Targeted Searches & Combine Results

Once you have your list of Property Type IDs and price ranges, you can import the Micro-Targeting Workflow to have Stevesie construct the endpoints one could use to perform these micro-searches.

Simply enter your location name in the Airbnb Location Names input, then paste in your list of Property Type IDs (from the last step) into the Airbnb Micro-Targeted Property Type IDs collection (we will remove the duplicates for you). Lastly, paste in the JSON array of price ranges into the Airbnb Micro-Targeted Listings Price Segments collection so it looks like this when you’re done:

Saved Price Ranges

Once you’re done, your workflow inputs should look something like this:

Airbnb Workflow Inputs

At this point, you can run the searches again or follow the URLs in the execution plan to manually collect the listings data for every combination of these price and property type segments.

Step 3 - Refine Searches

If you notice you end up hitting 300 listings for a given segment (if your offset gets to 250), then you’ll want to split these segments up even more into finer-grained definitions to get more data. Just revise your price ranges and try again. Also take note of any new Property Type IDs you may encounter, so you can add them back to your input list if you’d like a more exhaustive list.

Posted by steve on July 11, 2019, 1:29 p.m. 🚩  Report