Scraping the Patreon API
If you work in the creator economy or content marketing industry, you may need to scrape data from Patreon. Although there’s no official Patreon API to do this, we can legally record the network traffic that Patreon sends to our browser as we use their site normally and perform the scraping on the recording as to not violate the Patreon Terms of Service.
We’ll walk through an example of scraping all of the posts from any popular creator, such as Cold Ones and you can follow the steps below to scrape all the posts for any creator. This method will also work for scraping search results from Patreon, but will only include basic information about each creator.
1. Browse Patreon Posts
Head to any Patreon profile you want to scrape the posts for, such as Cold Ones. You don’t need to be logged in, though if you are then you may be able to scrape additional fields not shown in the following steps (we perform the scraping as a logged out user).
Once you’re on the web page, right click on the page and hit “Inspect” to open up developer tools, which will begin recording your web traffic. Now reload the page and scroll down through all the posts, clicking the “Show More” button as needed.
2. Export a HAR File
Once you’ve scrolled through all the posts, select the “Network” tab under developer tools and click the down arrow labeled “Export HAR…” to download a HAR file containing all of the raw JSON data we recorded from interacting with the Patreon API.
Upload that file to the HAR File Web Scraper and look for the group ending in /api/posts and click the “Parse Group” button to parse and combine together all of the intercepted JSON data.
3. Download Patreon Data
Once the parse completes, look for the data collection and you’ll be able to download the data as a combined CSV file. You’ll also see some other collections in the page that may be of interest depending on your use case, such as if you need to collect all of the tagged users, photo thumbnails, etc…
Post Data Fields
You’ll see the following columns available for each post in the scraped data Excel or CSV file. Note that some columns will be blank when not applicable (e.g. content duration will only appear for audio or video posts).
- Post ID
- Title
- Comments Count
- Commenter Count
- Like Count
- Thumbnail URL
- Patreon URL
- Post Type (Video, Audio, Text, etc…)
- Published Timestamp
- Teaser Preview Text
- Image URL
- Content Duration
Since this web scraping method is completely undetectable, you can safely do this as a logged in user to Patreon. And if logged in as a paying user of the creator you’re scraping the posts of, you may see more fields show up, as you should be able to scrape whatever is visible on the web page in addition to any additional fields the Patreon API sends to your browser.