Skip to content

Collect Search Console, GA4 or Shopify data with a job

A job collects data on a schedule with a built-in job type. You choose the type, its connections and its settings; you write no code. This guide collects Search Console performance. GA4 and Shopify work the same way with another source connection.

You need:

  • an Owner or Admin role (to add connections),
  • a runner group whose image provides the job type. The Runners page lists each runner’s Job types; the standard runner image provides sync only, so collection types need an image that includes them,
  • an S3 bucket and an access key that may write under one prefix of it.

Search Console or GA4: a Google service account

Section titled “Search Console or GA4: a Google service account”
  1. In Google Cloud, create a service account and a JSON key for it. Enable the Search Console API (or the Google Analytics Data API).
  2. Give the service account’s address access: in Search Console, add it as a user of the property; in GA4, as a Viewer of the property.
  3. Connections → Google service account → Connect, and paste the JSON key.

One service account can serve many sites: add it to each property. In an organization with clients, leave it shared so every client’s jobs can use it.

Create a custom app in the store’s admin with read access to what the job reads (orders, customers, products), install it, and add a Shopify Admin connection with the store’s *.myshopify.com address and the Admin API access token.

Every collecting job writes to a destination, and only there.

  1. In AWS, create a key that may write under one prefix only, for example orgs/<org id>/ or, for a client, orgs/<org id>/clients/<client id>/.
  2. Connections → S3 destination → Connect: bucket, prefix, region, access key, secret key (and a session token for temporary credentials).

In an organization with clients, give each client its own destination: a client’s job may only write to that client’s destination.

Jobs → New job, choose Search Console performance, and fill in:

Field Example
Name Search Console - example.com
Client The client it is for, or empty
Settings site (the key the data is stored under, such as example.com) and the property
Connections The service account as source, the S3 destination as destination
Schedule (cron), Time zone 0 5 * * *, once a day
Runner group A group whose runners list search_console
Time limit (minutes) A run that takes longer is stopped. A Lambda function stops at 15 minutes.
Alert when a run fails On
Alert when no new data for (hours) 26 for a daily job

The exact settings of each type are in Job types. Save.

  • Run now collects from where the job left off (its cursor). The first run collects the type’s recent days.
  • Backfill collects named days, such as the last 16 months, up to the type’s maximum (480 days for Search Console, 400 for GA4). A backfill that can’t finish in one run continues in parts, each a run of its own (“Backfill, part 2”), until it is done.

While a run is queued or running, the job’s next slot is skipped, and Run now and Backfill answer a run is already active (409 JOB_RUN_ACTIVE).

Open the job to see each run: its status (completed, partial or failed), rows written, the days it covered, and rows per table.

The job raises two alerts, routed like any other:

  • Job “Search Console - example.com” failed, when the latest run failed, errored, timed out or found no runner. The next run that writes data resolves it.
  • Job “…” has no new data, when no run has written data for 26 hours.

Route them to your channel (Send alerts to your team), and add a Job freshness item to a status page.

What you see Fix
The run is skipped with no_runner No runner in the group provides the type at that version. Check Job types on the Runners page.
The run fails with a permission error The service account isn’t a user of the property, or the destination key can’t write under its prefix.
Saving is refused on a hosted group Hosted jobs need a destination and can’t use connections whose secrets are on your runners. Your plan also limits hosted jobs.

To collect with your own code instead, see the job type contract.