Discover APIs

There are thousands of APIs out there waiting to be uncovered and put to use for extracting out data relevant to your projects. We'll talk about a few ways to go about uncovering new APIs or adding new functionality to existing APIs on the Stevesie platform.

Official APIs explicitly grant you persmission to use them, typically in the form of official API documentation that you'll find on the provider's website. If you have an existing source you want to scrape data from, you should first check to see if they offer an official API.

Check the official website you want to scrape data from and look for any links (typically in the footer) to "developer resources" or an official API. Sometimes these APIs may be paid (which is needed to support the work that goes into them) and you'll information about pricing, authentication, trial quotas, etc...

If you have trouble finding official API documentation on your favorite website, you may need to turn to a search engine. E.g. you can just type in "twitter api" into Google and you'll immediately get the link to their developer site. Always be sure the site your using is the official website for the API and not some third party trying to impersonate the source you're interested in.

Finding unofficial APIs from websites and mobile apps is a little trickier and requires some minor technical aptitude. You'll need to inspect your network traffic either on your web browser of mobile device to see what URLs are being called as APIs and then replicate these calls to be performed outside of official clients.

Note that accessing these unofficial APIs outside of the official websites and clients may or may not violate the API providers' terms of service. Learn more here: Is Data Scraping Legal?

If you're trying to find an API from a website you can access on your computer, it's pretty easy! Just go to the website in your browser (typically Google Chrome is easiest) and open up the developer tools.

Under the developer tools, select the network tab and make sure it preserves the log of network activity. Then just click around the site you want to get data from until you see it displayed on the screen. Check back on the network tab and go to the "XHR" part to look at the network requests. If you see a request that returns nice, structured data for what you're after - then congratulations - you've found an unofficial API endpoint!

If you can't find the API you're after using the website with developer tools, you may want to try the mobile app version if available. You'll need to use an interceptable proxy like mitmproxy to intercept your network traffic.

Setting up a man-in-the-middle proxy just to intercept your own traffic is a bit of a pain, so we added it as part of the Stevesie platform. To use this feature, just launch a new proxy and check the "Interceptable" checkbox before launching. You'll then see instructions for how you can connect your phone to the proxy to view your own network traffic and use it to uncover API endpoints.

If you still can't uncover the API endpoints used, then as a last resort you can search for open source software that other people have made and see if they were able to uncover the endpoints. Typically you can just add the word "github" to a Google search for an API, e.g. "twitter api github" to pull up GitHub repositories containing client code to access the API you're interested in.

Next: Adding APIs »