Skip to content

Jobs and job types

A job collects data on a schedule: one site’s Search Console performance, a store’s orders, a GA4 property. You make one by choosing a job type, the connections it reads from and writes to, and its settings. You write no code: the job type’s code ships inside a runner image, and SREInsights knows a type only by its name, version, settings and connection roles.

Jobs sit next to monitors, not inside them. They have their own page (Jobs), their own run history and their own alerts. A job’s run is an ordinary run on the queue, so scheduling, Run now, leases, sealed connections and Lambda runners work exactly as for checks.

Type Collects
search_console Search Console performance by day, query and page
ga4 GA4 sessions, users, new users and revenue by day
url_inspection The index status of the URLs in a site’s sitemaps
google_ads Campaign, keyword and search term performance
shopify_orders Orders created in the window (totals, not line items)
shopify_order_lines One row per order line: product, variant, SKU, quantity, amount
shopify_customers Every customer with orders and spend
shopify_catalog Every product with variants, prices, images and SEO fields, and the collections
onpage_audit SEO titles, descriptions, word counts and images of a Shopify store’s pages, with findings
etsy_catalog, etsy_orders An Etsy shop’s listings, and the items it sold
ebay_catalog, ebay_orders An eBay seller’s active listings, and the items sold
backlink_check Whether pages that should link to a site still do
anomaly_scan Nothing new: reports unusual changes in data already collected, as findings
lake_ingest, lake_upkeep Nothing new: merge what other jobs wrote into a lake, and keep the lake tidy. They run on the runner’s own credentials.
sync Rows from a Google Sheet into an API, a table or the lake. See Syncs.

Settings, versions and limits of each are in Job types. A job type can be used only on a runner group whose image provides it; the Runners page lists each runner’s job types.

A job type names connection roles, such as source (a Google service account, a Shopify Admin connection) and destination. Every collecting type writes to a destination, which is required: an s3_destination connection with a bucket, a prefix, a region and a key that may write under that prefix only. A run writes nowhere else.

By convention a job writes under orgs/<org id>/clients/<client id>/, or orgs/<org id>/ when it belongs to no client.

  • Scheduled runs and Run now carry on from the job’s cursor: where its last run with data stopped.
  • Backfill collects named days instead: a start and an end date, up to the type’s maximum (for example 480 days for Search Console). It leaves the cursor alone. A backfill that can’t finish in one run continues in parts (“Backfill, part 2”), each a run of its own, until it completes.
  • One run at a time. While a run is queued or running, the next scheduled slot is skipped, and Run now and Backfill answer 409 JOB_RUN_ACTIVE.
  • Results. A run reports completed, partial (some data written; it says what is missing) or failed, with the rows written, the days covered and rows per table. The job’s page shows each run.

Every job can raise two alerts. They are ordinary alerts: routed, silenced and notified like any other.

  • Job “…” failed when its latest run failed, errored, timed out or found no runner. The next run that writes data resolves it.
  • Job “…” has no new data when no run has written data for the job’s freshness (in hours), if you set one.

Pausing or deleting the job resolves them.

Some job types report findings: things worth someone’s attention, such as an unusual drop in clicks. Each finding is an alert of its own, with source job_finding and labels such as job, job_type, detector and finding, so routes can send them somewhere different from failures. A completed run resolves the findings it no longer reports. The job’s page lists each run’s findings.

A job runs on the job type version it was saved with. When a newer version exists, the job’s page says so and offers to move it; SREInsights checks the job’s settings against the new version first. A runner image may provide both versions while jobs move.

Built-in job types may run on hosted runner groups whose image provides them, because their code is the operator’s, not yours. A hosted job must name a destination, can’t use connections whose secrets are kept on your runners, is held to your plan’s shortest interval, and counts against the plan’s hosted job limit.

To run your own collection code the same way, build a runner image that provides a job type: a manifest that names a command, and a command that reads an input file and writes a result. See the job type contract and Move a scheduled job into SREInsights.