Jobs and job types
A job collects data on a schedule: one site’s Search Console performance, a store’s orders, a GA4 property. You make one by choosing a job type, the connections it reads from and writes to, and its settings. You write no code: the job type’s code ships inside a runner image, and SREInsights knows a type only by its name, version, settings and connection roles.
Jobs sit next to monitors, not inside them. They have their own page (Jobs), their own run history and their own alerts. A job’s run is an ordinary run on the queue, so scheduling, Run now, leases, sealed connections and Lambda runners work exactly as for checks.
Job types
Section titled “Job types”| Type | Collects |
|---|---|
search_console |
Search Console performance by day, query and page |
ga4 |
GA4 sessions, users, new users and revenue by day |
url_inspection |
The index status of the URLs in a site’s sitemaps |
google_ads |
Campaign, keyword and search term performance |
shopify_orders |
Orders created in the window (totals, not line items) |
shopify_order_lines |
One row per order line: product, variant, SKU, quantity, amount |
shopify_customers |
Every customer with orders and spend |
shopify_catalog |
Every product with variants, prices, images and SEO fields, and the collections |
onpage_audit |
SEO titles, descriptions, word counts and images of a Shopify store’s pages, with findings |
etsy_catalog, etsy_orders |
An Etsy shop’s listings, and the items it sold |
ebay_catalog, ebay_orders |
An eBay seller’s active listings, and the items sold |
backlink_check |
Whether pages that should link to a site still do |
anomaly_scan |
Nothing new: reports unusual changes in data already collected, as findings |
lake_ingest, lake_upkeep |
Nothing new: merge what other jobs wrote into a lake, and keep the lake tidy. They run on the runner’s own credentials. |
sync |
Rows from a Google Sheet into an API, a table or the lake. See Syncs. |
Settings, versions and limits of each are in Job types. A job type can be used only on a runner group whose image provides it; the Runners page lists each runner’s job types.
Connections and the destination
Section titled “Connections and the destination”A job type names connection roles, such as source (a Google service account, a Shopify Admin connection) and destination. Every collecting type writes to a destination, which is required: an s3_destination connection with a bucket, a prefix, a region and a key that may write under that prefix only. A run writes nowhere else.
By convention a job writes under orgs/<org id>/clients/<client id>/, or orgs/<org id>/ when it belongs to no client.
- Scheduled runs and Run now carry on from the job’s cursor: where its last run with data stopped.
- Backfill collects named days instead: a start and an end date, up to the type’s maximum (for example 480 days for Search Console). It leaves the cursor alone. A backfill that can’t finish in one run continues in parts (“Backfill, part 2”), each a run of its own, until it completes.
- One run at a time. While a run is queued or running, the next scheduled slot is skipped, and Run now and Backfill answer
409 JOB_RUN_ACTIVE. - Results. A run reports
completed,partial(some data written; it says what is missing) orfailed, with the rows written, the days covered and rows per table. The job’s page shows each run.
Alerts from jobs
Section titled “Alerts from jobs”Every job can raise two alerts. They are ordinary alerts: routed, silenced and notified like any other.
- Job “…” failed when its latest run failed, errored, timed out or found no runner. The next run that writes data resolves it.
- Job “…” has no new data when no run has written data for the job’s freshness (in hours), if you set one.
Pausing or deleting the job resolves them.
Findings
Section titled “Findings”Some job types report findings: things worth someone’s attention, such as an unusual drop in clicks. Each finding is an alert of its own, with source job_finding and labels such as job, job_type, detector and finding, so routes can send them somewhere different from failures. A completed run resolves the findings it no longer reports. The job’s page lists each run’s findings.
Versions
Section titled “Versions”A job runs on the job type version it was saved with. When a newer version exists, the job’s page says so and offers to move it; SREInsights checks the job’s settings against the new version first. A runner image may provide both versions while jobs move.
Jobs on hosted runners
Section titled “Jobs on hosted runners”Built-in job types may run on hosted runner groups whose image provides them, because their code is the operator’s, not yours. A hosted job must name a destination, can’t use connections whose secrets are kept on your runners, is held to your plan’s shortest interval, and counts against the plan’s hosted job limit.
Your own job types
Section titled “Your own job types”To run your own collection code the same way, build a runner image that provides a job type: a manifest that names a command, and a command that reads an input file and writes a result. See the job type contract and Move a scheduled job into SREInsights.