Collect Search Console, GA4 or Shopify data with a job
A job collects data on a schedule with a built-in job type. You choose the type, its connections and its settings; you write no code. This guide collects Search Console performance. GA4 and Shopify work the same way with another source connection.
You need:
- an Owner or Admin role (to add connections),
- a runner group whose image provides the job type. The Runners page lists each runner’s Job types; the standard runner image provides
synconly, so collection types need an image that includes them, - an S3 bucket and an access key that may write under one prefix of it.
1. The source connection
Section titled “1. The source connection”Search Console or GA4: a Google service account
Section titled “Search Console or GA4: a Google service account”- In Google Cloud, create a service account and a JSON key for it. Enable the Search Console API (or the Google Analytics Data API).
- Give the service account’s address access: in Search Console, add it as a user of the property; in GA4, as a Viewer of the property.
- Connections → Google service account → Connect, and paste the JSON key.
One service account can serve many sites: add it to each property. In an organization with clients, leave it shared so every client’s jobs can use it.
Shopify: a Shopify Admin connection
Section titled “Shopify: a Shopify Admin connection”Create a custom app in the store’s admin with read access to what the job reads (orders, customers, products), install it, and add a Shopify Admin connection with the store’s *.myshopify.com address and the Admin API access token.
2. The destination connection
Section titled “2. The destination connection”Every collecting job writes to a destination, and only there.
- In AWS, create a key that may write under one prefix only, for example
orgs/<org id>/or, for a client,orgs/<org id>/clients/<client id>/. - Connections → S3 destination → Connect: bucket, prefix, region, access key, secret key (and a session token for temporary credentials).
In an organization with clients, give each client its own destination: a client’s job may only write to that client’s destination.
3. Create the job
Section titled “3. Create the job”Jobs → New job, choose Search Console performance, and fill in:
| Field | Example |
|---|---|
| Name | Search Console - example.com |
| Client | The client it is for, or empty |
| Settings | site (the key the data is stored under, such as example.com) and the property |
| Connections | The service account as source, the S3 destination as destination |
| Schedule (cron), Time zone | 0 5 * * *, once a day |
| Runner group | A group whose runners list search_console |
| Time limit (minutes) | A run that takes longer is stopped. A Lambda function stops at 15 minutes. |
| Alert when a run fails | On |
| Alert when no new data for (hours) | 26 for a daily job |
The exact settings of each type are in Job types. Save.
4. Run it and backfill
Section titled “4. Run it and backfill”- Run now collects from where the job left off (its cursor). The first run collects the type’s recent days.
- Backfill collects named days, such as the last 16 months, up to the type’s maximum (480 days for Search Console, 400 for GA4). A backfill that can’t finish in one run continues in parts, each a run of its own (“Backfill, part 2”), until it is done.
While a run is queued or running, the job’s next slot is skipped, and Run now and Backfill answer a run is already active (409 JOB_RUN_ACTIVE).
Open the job to see each run: its status (completed, partial or failed), rows written, the days it covered, and rows per table.
5. Get told when it stops
Section titled “5. Get told when it stops”The job raises two alerts, routed like any other:
- Job “Search Console - example.com” failed, when the latest run failed, errored, timed out or found no runner. The next run that writes data resolves it.
- Job “…” has no new data, when no run has written data for 26 hours.
Route them to your channel (Send alerts to your team), and add a Job freshness item to a status page.
Common problems
Section titled “Common problems”| What you see | Fix |
|---|---|
The run is skipped with no_runner |
No runner in the group provides the type at that version. Check Job types on the Runners page. |
| The run fails with a permission error | The service account isn’t a user of the property, or the destination key can’t write under its prefix. |
| Saving is refused on a hosted group | Hosted jobs need a destination and can’t use connections whose secrets are on your runners. Your plan also limits hosted jobs. |
To collect with your own code instead, see the job type contract.