Skip to content

Job types

A job collects data on a schedule. You make one by choosing a job type, its connections and its settings. You never write code for it: the type’s code ships inside a runner image. This page lists every type SREInsights knows, at every version a saved job can still run on, and then the fields, runs and alerts that every job shares.

For what a job is and when to use one, see Jobs. For a walkthrough, see Collect data with a job.

SREInsights knows a job type only as data: its name, version, settings and connection roles. The code that collects the data is in a runner image, and the image lists the types it provides in a manifest. See Job type contract.

  • Each type at each version is a capability, job:<name>@<version>, for example job:search_console@1.
  • A job’s run goes only to a runner whose image provides that type at that version. Choose a runner group whose runners run such an image.
  • In the job form, a runner group with no runner that provides the type shows · no runner with this type yet next to its name.
  • GET /api/v1/job-types lists the latest version of each type, with capability and providedBy: the runner groups (yours and hosted) that have a runner with the type.
  • sync is different: every runner provides it, whatever its image, from the runner version that added syncs.
Type Version Label Backfill Egress Credentials Findings Hosted runners
search_console 1 Search Console performance up to 480 days provider connections no yes
ga4 1 Google Analytics 4 up to 400 days provider connections no yes
url_inspection 2 Index status no public connections no yes
url_inspection 1 (superseded) Index status no public connections no yes
google_ads 2 Google Ads up to 400 days provider connections no yes
google_ads 1 (superseded) Google Ads up to 400 days provider connections no yes
shopify_orders 1 Shopify orders up to 730 days provider connections no yes
shopify_customers 1 Shopify customers no provider connections no yes
shopify_catalog 1 Shopify catalog no provider connections no yes
shopify_order_lines 1 Shopify order lines up to 1,100 days provider connections no yes
onpage_audit 1 On-page audit no provider connections no yes
etsy_catalog 1 Etsy listings no provider connections no yes
etsy_orders 1 Etsy orders up to 1,100 days provider connections no yes
ebay_catalog 1 eBay listings no provider connections no yes
ebay_orders 1 eBay orders up to 730 days provider connections no yes
backlink_check 1 Backlink check no public connections no yes
anomaly_scan 1 Anomaly scan no provider connections yes yes
sync 1 Sync no public connections no yes
lake_ingest 1 Lake ingest no provider runner no no
lake_upkeep 1 Lake upkeep no provider runner no no

“Hosted runners: yes” means the type may be saved on a group hosted by SREInsights. It runs there only if that group’s image provides it, and the hosted rules apply.

  • Backfill. The longest range of days one Backfill may ask for. “No” means the type collects from its own cursor only, and Backfill answers 400 BACKFILL_NOT_SUPPORTED.
  • Egress. Where a run connects. provider: only the provider’s own API hosts. public: also public URLs that the job’s settings lead to, such as a site’s sitemaps on a CDN. On a hosted runner a run never connects to a private, loopback, link-local or metadata address.
  • Credentials. connections: the run works only with the job’s connections and writes to its destination. runner: the run works with the runner’s own credentials (see Types on the runner’s credentials).
  • Findings. Whether runs may report findings, which become alerts. See Findings.

Every type with connections credentials, except sync, takes a Destination connection (role destination), and it is required:

Role Label Connection types Required
destination Destination s3_destination yes

An s3_destination connection holds a bucket, a prefix, a region and a key for that prefix only (plus a session token for temporary credentials). A run writes nowhere else. By convention the prefix is orgs/<orgId>/clients/<clientId>/ for a client’s job, or orgs/<orgId>/ for the organization’s own. See Connection types and Clients.

Every collecting type also has the setting below. It is listed again under each type.

Setting Label Type Required Default What it does
site Site key text yes A short name for the site, which keys its data, for example acme-shop.

Search Console performance, version 1. Clicks, impressions, CTR and position by day, query and page from Google Search Console. Each run reads the last few days again, because Google keeps revising them.

Role Label Connection types Required
google Google service account google_service_account yes
destination Destination s3_destination yes

The service account must be a user of the Search Console property.

Setting Label Type Required Default What it does
site Site key text yes Keys the site’s data.
siteUrl Search Console property text yes sc-domain:example.com, or a URL-prefix property such as https://www.example.com/.

Backfill: up to 480 days. Default schedule 0 8 * * *, time limit 10 minutes.

Google Analytics 4, version 1. Sessions, users, new users and revenue by day, channel and landing page from a GA4 property.

Role Label Connection types Required
google Google service account google_service_account yes
destination Destination s3_destination yes

The service account needs the Viewer role on the GA4 property.

Setting Label Type Required Default What it does
site Site key text yes Keys the site’s data.
propertyId GA4 property id text yes The numeric property id, for example 123456789.

Backfill: up to 400 days. Default schedule 15 8 * * *, time limit 10 minutes.

Index status. Whether Google has indexed each URL of the site’s sitemaps, through Search Console’s URL Inspection API, oldest checks first. It reads the sitemaps at the property’s own host and the child sitemaps they list, wherever those are served from, so its egress is public.

Role Label Connection types Required
google Google service account google_service_account yes
destination Destination s3_destination yes

The service account must be a user of the Search Console property.

Version 2 (current):

Setting Label Type Required Default What it does
site Site key text yes Keys the site’s data.
siteUrl Search Console property text yes sc-domain:example.com, or a URL-prefix property.
maxUrls URLs per run number, 1 to 2,000 no 1800 Google allows 2,000 inspections a property a day. A Run now on the same day shares them.

Version 1 (superseded): the same, except maxUrls has a default of 300 and no bounds.

Backfill: no. Default schedule 20 9 * * *, time limit 10 minutes.

Google Ads. Campaign, keyword and search term performance by day from one Google Ads account.

Role Label Connection types Required
google_ads Google Ads google_ads yes
destination Destination s3_destination yes

The google_ads connection holds a developer token and an OAuth refresh token for a user with access to the account.

Version 2 (current):

Setting Label Type Required Default What it does
site Site key text yes Keys the site’s data.
customerId Customer id text yes The Ads account, for example 123-456-7890.
campaignPrefix Campaign name prefix text no Only campaigns whose name starts with this, for example [JC] . Empty: every campaign of the account. Use it when several sites share one Ads account.

Version 1 (superseded): site and customerId only.

Backfill: up to 400 days. Default schedule 35 8 * * *, time limit 10 minutes.

Every Shopify type reads through the same role:

Role Label Connection types Required
shopify Shopify store shopify_admin yes
destination Destination s3_destination yes

The shopify_admin connection holds an Admin API access token of a custom app with read access to what the job collects. Its store domain must be the store’s *.myshopify.com address.

Shopify orders, version 1. Orders created in the window, read again over the following 7 days to pick up refunds and edits. Order totals, not their line items.

Setting Label Type Required Default What it does
site Site key text yes Keys the site’s data.

Backfill: up to 730 days. Default schedule 50 8 * * *, time limit 10 minutes.

Shopify customers, version 1. Every customer of the store with their order count and spend.

Setting Label Type Required Default What it does
site Site key text yes Keys the site’s data.

Backfill: no. Default schedule 5 9 * * *, time limit 10 minutes.

Shopify catalog, version 1. Every product of the store with its variants and SKUs, prices, images and the fields people edit (title, description, tags, SEO), and its collections with their products. Each run reads everything. A product that is no longer there is marked deleted.

Setting Label Type Required Default What it does
site Site key text yes Keys the site’s data.
publicDomain Storefront domain text no The store’s own domain, for product and collection links, for example shop.example.com. Empty: its myshopify.com address.

Backfill: no. Default schedule 15 9 * * *, time limit 14 minutes.

Shopify order lines, version 1. What sold and when, one row per order line (product, variant, SKU, quantity, amount, cancelled or not), for the orders changed in the window, so later edits and cancellations are read again. Nothing about the customer.

Setting Label Type Required Default What it does
site Site key text yes Keys the site’s data.

Backfill: up to 1,100 days. Default schedule 40 9 * * *, time limit 10 minutes.

On-page audit, version 1. The SEO title, description, word count and images with their alt text of every product, collection, page and article of the store, and the SEO findings built from them. (These findings are data the job writes, not alerts.)

Setting Label Type Required Default What it does
site Site key text yes Keys the site’s data.

Backfill: no. Default schedule 50 9 * * 0 (Sundays), time limit 10 minutes.

These types read a store’s tokens from AWS Secrets Manager, at <secret prefix>store/<key>, through an aws_secrets_manager connection. The Stores page puts the tokens there when you sign in to the store, and then offers the store’s jobs with its key filled in.

Every Etsy and eBay type has these settings:

Setting Label Type Required Default What it does
site Site key text yes Keys the site’s data.
store Store key text yes The store’s key, for example acme. Its secret is <secret prefix>store/<key>.

Etsy listings, version 1. Every listing of an Etsy shop in every state, with its variations and SKUs, prices, images and the fields people edit, and its shop sections, read with the shop owner’s approval. Each run reads everything. A listing that is no longer there is marked removed.

Role Label Connection types Required
etsy Etsy store secrets aws_secrets_manager yes
destination Destination s3_destination yes

The connection names the store secrets’ prefix and the token lock (Lock bucket). Its key must be able to read and update the store’s secret and its lock: Etsy replaces its refresh token on every use.

Backfill: no. Default schedule 20 9 * * *, time limit 14 minutes.

Etsy orders, version 1. What sold and when, one row per item sold (listing, SKU, quantity, amount, cancelled or not), for the orders changed in the window. Nothing about the buyer. The store must be connected with permission to read orders.

Roles as for etsy_catalog.

Backfill: up to 1,100 days. Default schedule 25 9 * * *, time limit 10 minutes.

eBay listings, version 1. Every active listing of an eBay seller, with its variations and SKUs, prices, pictures and the fields people edit, and the seller’s Store categories and the eBay categories its listings sit in. Each run reads everything. A listing that is no longer active is marked ended.

Role Label Connection types Required
ebay eBay store secrets aws_secrets_manager yes
destination Destination s3_destination yes

The connection’s key must be able to read the eBay app’s secret and the store’s secret.

Backfill: no. Default schedule 30 9 * * *, time limit 14 minutes.

eBay orders, version 1. What sold and when, one row per item sold (listing, SKU, quantity, amount, cancelled or not), for the orders changed in the window, from the two years eBay keeps. Nothing about the buyer.

Roles as for ebay_catalog.

Backfill: up to 730 days. Default schedule 35 9 * * *, time limit 10 minutes.

These types collect nothing from a provider. They read read-outs that the lake ingest writes into the client’s own folder, under the destination prefix at <prefix>_lake/. Their only connection is the destination, and its key also needs s3:GetObject on <prefix>_lake/*.

Role Label Connection types Required
destination Destination s3_destination yes

Backlink check, version 1. Visits the pages where links to the client were placed (guest posts, profiles, bookmarks and the like) and records whether each still links to the client: live, mention, not found, dead, blocked or error, with where it points, nofollow or not, and its anchor text. Links checked longest ago come first. The list of links comes from the read-out <prefix>_lake/backlink_check/<site>.json. It visits third-party pages, so its egress is public.

Setting Label Type Required Default What it does
site Site key text yes Keys the site’s data.
maxLinks Links per run number, 1 to 2,000 no 300 The most pages one run visits.
staleDays Check again after (days) number, 0 to 365 no 7 A link checked more recently than this is skipped. 0 checks every link again.

Backfill: no. Default schedule 20 10 * * *, time limit 14 minutes.

Anomaly scan, version 1. Looks for unusual changes in a site’s data in the lake (search clicks, impressions and positions, traffic, orders and the like) and reports each one as a finding. A finding stays an alert until a later run no longer finds it. Small pages and queries are left out.

Setting Label Type Required Default What it does
site Site key text yes Keys the site’s data.
sensitivity Sensitivity select: low, normal, high no normal high reports smaller changes; low only the largest.
detectors Detectors text no Which checks to run, separated by commas. Empty: every one that applies to the site’s data.
minWeeklyClicks Least weekly clicks number, 0 to 1,000,000 no 50 A page or query with fewer clicks a week than this is left out.
topN Findings per run number, 1 to 50 no 20 The most findings one run reports, largest first.

Backfill: no. Default schedule 0 14 * * *, time limit 14 minutes.

Sync, version 1. Reads the rows of a source, such as the tabs of a Google Sheet, maps their columns to fields, and writes them to a destination: an HTTP API, a Postgres table or the lake. It updates each row by its key, so running it again never adds a row twice.

A sync has no settings. Its configuration is its spec, saved on the job as sync (in the form: Spec (JSON)). See Sync spec.

Role Label Connection types Required
source Source google_service_account only when the sheet is read as a service account
destination Destination http_destination, postgres_destination, s3_destination yes
  • The runner provides sync itself, in its own process, not as a command from the image’s manifest. Every runner from the version that added syncs reports job:sync@1.
  • A sync is saved paused. Preview it, then switch it on. See Preview, then switch on.
  • Backfill: no. Default schedule 15 * * * * (hourly), time limit 10 minutes.

lake_ingest and lake_upkeep look after the lake that the destinations feed, instead of collecting into it. They work with their runner’s own credentials (credentials: "runner"): the variables the runner’s operator passes on to commands in SREINSIGHTS_SCRIPT_ENV_PASSTHROUGH, such as a Lambda function’s role. See Runner settings.

  • They take no settings, no connections and no destination. The form says: “None: this type works with its runner’s own credentials, the variables the runner group passes on to commands. It runs on your own runner groups only.”
  • They can’t be saved on a hosted group: 400 RUNNER_CREDENTIALS_NOT_HOSTED. A hosted runner holds no credentials and passes none on. The form offers only your own groups for them.
  • They usually belong to no client.

Lake ingest, version 1. Merges what the organization’s jobs wrote to their destinations into the DuckLake lake beside them, as the lake’s one writer, and builds what is computed from those rows. A file it can’t take is left where it is, or moved aside, and reported.

Default schedule */30 10-13 * * *, time limit 10 minutes.

Lake upkeep, version 1. Weekly care of the DuckLake lake: expires snapshots older than 30 days, merges small data files, deletes files nothing refers to any more, and applies rows that other writers left staged.

Default schedule 10 9 * * 0 (Sundays), time limit 14 minutes.

These are the fields of POST /api/v1/jobs and PATCH /api/v1/jobs/:id, with their labels in the job form (Jobs → New job, or New sync).

Field Form label Type Required Default What it does
name Name text, 1 to 120 characters yes The job’s name in lists, alerts and notifications.
type (chosen before the form) job type name yes The type. A new job is saved on the type’s latest version. It can’t be changed later.
clientId Client client id or null no null The client it belongs to. Empty: the organization’s own.
params Settings object per type the type’s defaults The type’s settings, converted to their declared types. A name the type doesn’t declare is refused.
connections Connections object: role → connection id per type At most 10. Every role must be one the type has.
sync Spec (JSON) object for sync only A sync’s spec. Any other type refuses it.
schedule.cron Schedule (cron) cron yes When it runs. The form suggests the type’s default schedule.
schedule.timezone Time zone IANA time zone no UTC The cron’s time zone.
runsOn.group Runner group group name, or a hosted group’s id yes Where it runs. A runner in the group needs an image that provides the type.
runsOn.labels list of labels no [] Only runners with these labels take its runs. At most 20.
timeoutMs Time limit (minutes) 10,000 to 900,000 ms no the type’s default A run that takes longer is stopped. The form takes 1 to 15 minutes. A Lambda function stops at 15 minutes.
alerting.onFailure Alert when a run fails boolean no true Raises Job “…” failed.
alerting.freshnessHours Alert when no new data for (hours) 1 to 8,760, or null no null Raises Job “…” has no new data. Empty: no freshness alert. A daily job usually allows 26 hours.
alerting.severity Severity critical, high, medium, low, info no high The severity of the job’s two alerts.
description Description text, up to 2,000 characters no
enabled Enabled boolean no true (a sync: false) Paused jobs don’t run on their schedule, and their open alerts resolve.
typeVersion Use version N (button) number PATCH only Moves the job to another version of its type. See Versions.

A client’s job uses the client’s own connections and shared ones, except its destination, which must be the client’s own. A job with no client uses shared connections only. See Clients.

Terminal window
curl -X POST "https://console.sreinsights.com/api/v1/jobs" \
-H "Authorization: Bearer $SREINSIGHTS_TOKEN" \
-H "Content-Type: application/json" \
-d '{
"name": "Search Console · acme-shop",
"type": "search_console",
"clientId": "0190c3c4-0000-7000-8000-00000000c001",
"params": { "site": "acme-shop", "siteUrl": "sc-domain:example.com" },
"connections": {
"google": "0190c3c4-0000-7000-8000-00000000a001",
"destination": "0190c3c4-0000-7000-8000-00000000a002"
},
"runsOn": { "group": "jobs" },
"schedule": { "cron": "0 8 * * *", "timezone": "UTC" },
"alerting": { "onFailure": true, "freshnessHours": 26 }
}'

For authentication and the response envelope, see REST API.

A type’s version goes up when its input or output changes in a way older code can’t handle, such as a new setting or a changed meaning. SREInsights keeps every version that saved jobs may still run on. The older one is superseded: new jobs are saved on the latest.

  • A job runs on the version it was saved with. Its run goes only to a runner whose image lists that version.

  • A job on an older version shows a banner on its page: “Label has a newer version (1 → 2). A runner runs this job only while its image provides version 1.” Press Use version 2 to move it.

  • Or move it with the API:

    Terminal window
    curl -X PATCH "https://console.sreinsights.com/api/v1/jobs/<job id>" \
    -H "Authorization: Bearer $SREINSIGHTS_TOKEN" \
    -H "Content-Type: application/json" \
    -d '{ "typeVersion": 2 }'
  • Moving checks the job’s settings against the new version. A new setting gets its default, and a changed bound applies. Runs use the new version from the next one.

  • A runner image may list both versions while jobs move, so no job is left without a runner.

A run of a job is an ordinary run on the queue, so scheduling, leases, sealed connections and retries work as they do for checks. A run whose lease is lost runs again, up to three attempts in all.

Run now on the job’s page queues one run straight away, whether or not the job is enabled. The dialog shows the type’s settings “for this run only. An empty field uses the job’s value.”

Terminal window
curl -X POST "https://console.sreinsights.com/api/v1/jobs/<job id>/run" \
-H "Authorization: Bearer $SREINSIGHTS_TOKEN" \
-H "Content-Type: application/json" \
-d '{ "params": { "maxUrls": 100 } }'

The answer is 202 with runId, status, queueExpiresAt, the params the run uses and window (null).

A sync must have a preview of its current version first, or Run now answers 409 SYNC_NOT_PREVIEWED.

Scheduled runs and Run now carry on from the job’s cursor: what its last run that wrote data reported, such as the last updated_at seen. The job’s page shows it under Data → Cursor.

  • A run that completed or was partial moves the cursor on, if it reported one.
  • A failed run leaves the cursor where it was, so the next run tries the same data again.
  • A Backfill leaves the cursor alone.

Backfill collects a range of days you name, for a type that takes a range (see the Backfill column above). In the dialog, choose Days to collect: the first and last day, both included.

Terminal window
curl -X POST "https://console.sreinsights.com/api/v1/jobs/<job id>/backfill" \
-H "Authorization: Bearer $SREINSIGHTS_TOKEN" \
-H "Content-Type: application/json" \
-d '{ "start": "2026-01-01", "end": "2026-06-30" }'
  • start and end are YYYY-MM-DD. params is optional, as for Run now.
  • The range may cover at most the type’s maximum days. A longer one is refused: split it into several backfills.
  • end may not be in the future.
  • A type without a range answers 400 BACKFILL_NOT_SUPPORTED.

A run can’t always cover the whole range in its time limit. A part that stops early reports partial with nextDay, the first day it didn’t collect. SREInsights then queues the next part, from nextDay to the Backfill’s end, with the same settings.

  • Parts run one after another, each a run of its own. The Runs table shows them as “Backfill, part 2”, “Backfill, part 3” and so on.
  • It stops when a part completes, fails, makes no progress, reaches past the end, after 100 parts, or when the job is deleted.
  • While parts run, the job’s scheduled slots are skipped.

A job runs one run at a time, because two runs at once would write the same days twice. While a run is queued or running:

  • the job’s next scheduled slot is skipped;
  • Run now and Backfill answer 409 JOB_RUN_ACTIVE, with “This job is running. Wait for the run to finish.” or “A run of this job is already waiting for a runner.”

Each run reports one of these statuses:

Status Meaning
completed Everything in the window was collected and written.
partial Some of it was written. error says what is missing. Counts as data arriving.
failed Nothing usable was written.

A run also reports rows written, the days covered, rows per table and a cursor. If a run can’t report at all, SREInsights records error (it could not run), timeout (it ran out of time) or skipped (no runner took it before its queue window closed).

The Runs table on the job’s page lists runs newest first, with Status, Started by (Schedule, Run now or Backfill), Data covered, Rows, Took, When and Attempt. GET /api/v1/jobs/:id/runs?limit=50&before=<ISO time> returns the same, up to 200 a page.

A job has no check rules. It raises two alerts of its own, judged from its latest run. Both are ordinary alerts with source = job: routed, silenced and notified like any other. See Alerts.

Alert Fires when Resolves when
Job “…” failed Alert when a run fails is on, and the latest run failed, errored, timed out or found no runner. The next run that writes data (completed or partial).
Job “…” has no new data Alert when no new data for (hours) is set, and no run has written data for that many hours. Before the first data, it counts from when the job was made. A run writes data.
  • Both have the job’s Severity and its client.
  • Labels: job (the job’s name), job_id, job_type and alert (failed or stale).
  • Pausing or deleting the job resolves them. A paused job raises neither.

A type with findings (today anomaly_scan) may report, in its result, things worth someone’s attention. A run of any other type that reports findings is an error.

Each finding is an alert of its own:

  • source = job_finding, so that notification routes can pick findings out.
  • Its title, severity, detail and value are the finding’s. Its client is the job’s.
  • Labels: job, job_id, job_type, detector, finding (the finding’s key) and client (when the job has one). The detector is data.detector when the finding sets it, else its key up to the first :.

How findings open and resolve:

  • A completed or partial run opens an alert for each new key and updates the ones still reported (its severity raised or lowered).
  • A completed run resolves the job’s open findings that it no longer reports. A partial or failed run resolves nothing, because it may not have looked.
  • Pausing or deleting the job resolves them. A paused job’s runs (Run now) raise none.

The job’s page lists each run’s findings (the Findings column of Runs). For the result format, see Job type contract.

You can run jobs on a group hosted by SREInsights, including a hosted Lambda group, when its image provides the type. Saving a job on a hosted group checks:

  1. No connection keeps its secrets on your own runners: 400 HOSTED_RUNNER_LOCAL_SECRETS.
  2. A destination is chosen. In the form: “Required on hosted runners: they hold no credentials of their own, so the job writes only where this connection allows.”
  3. The schedule is no shorter than your plan’s shortest interval: 403 PLAN_LIMIT.
  4. Your organization is under its plan’s count of hosted jobs, which is separate from hosted monitors: 403 PLAN_LIMIT.
  5. The type doesn’t work on its runner’s credentials: 400 RUNNER_CREDENTIALS_NOT_HOSTED.

A sync on a hosted group also needs an https:// HTTP destination, or a Postgres destination with TLS.

Each run receives only its own organization’s connections, sealed to that runner for that lease. See Runners.

Code When
400 UNKNOWN_JOB_TYPE The type, or the job’s saved version, isn’t one SREInsights knows.
400 BACKFILL_NOT_SUPPORTED Backfill on a type without a range.
400 RUNNER_CREDENTIALS_NOT_HOSTED A type on its runner’s credentials, saved on a hosted group.
400 HOSTED_RUNNER_LOCAL_SECRETS A connection whose secrets stay on your runners, on a hosted group.
403 PLAN_LIMIT A hosted job over the plan’s interval or count.
409 JOB_RUN_ACTIVE Run now or Backfill while a run is queued or running.
409 SYNC_NOT_PREVIEWED Switching on or running a sync whose current version has no preview.