Executing Workflow Runs

Executing

Many workflows you import from a public formula will have an aggregator already defined for you. If you want to add a new aggregator, be sure your account has the appropriate "extractor" for the task you're interested in. You can build an extractor by executing a task and finding the appropriate collection in the response.

Proxy Type

If you expect your workflow to take more than 750 requests, you're better off selecting a dedicated proxy instead of the default shared proxies.

Dedicated Proxy Management

If you select a dedicated proxy (and don't have any free ones currently running), the system will automatically launch a new proxy in your desired location, use that proxy for requests, and then terminate the proxy when done.

Rate Limiting

Requests are timed to be 2 seconds apart for each workflow run. We've found this to be the sweet spot for most services to not rate limit requests. If the service does issue an HTTP 429 response to rate limit, the workflow will respect this request and wait 60 seconds before trying again.

Data Delivery

If you have aggregators set for a workflow, the results will be temporarily written to a URL you can access for 24 hours after the workflow completes. Upon completion, you'll be emailed the URLs of where to access your data from. You can also use our API to poll the workflow and get the URL programatically once it's done.