Runner protocol
A runner talks to SREInsights over the runner protocol, under /runner/v1. This page describes every call, what it sends and what it gets back, and what happens to a run along the way. Read it to understand what a runner does, to debug one, or to build your own.
How it works
Section titled “How it works”Every call goes out from the runner over HTTPS. SREInsights never connects to a runner, so a runner needs no open ports.
- Register once with a one-time registration token. The runner makes its own keys and sends only the public halves.
- Sign in by signing a challenge with its private key. It gets a 15-minute access token.
- Say hello on every start: its version, what it can run, and the key secrets are sealed to.
- Lease runs with a long poll. Each lease comes with the run’s check and its connection secrets, sealed to this runner.
- Send heartbeats every 20 seconds while a run is going, to keep its lease.
- Post the result when the run finishes.
Base URL: https://console.sreinsights.com/runner/v1, or <your SREInsights URL>/runner/v1 on a self-hosted instance. Every call is a POST with a JSON body.
Responses use the same envelope and error shape as the REST API: { "status": "success", "data": … } on success, and { "status": "fail", "code", "message", "details"?, "requestId" } on failure.
Timings and limits
Section titled “Timings and limits”| Value | Setting |
|---|---|
| Registration token | One-time by default (up to 50 uses), valid 24 hours by default (up to 7 days). Starts with prt_. |
| Challenge | Valid 60 seconds. |
| Access token | 15 minutes. |
| Lease poll | Held open up to 25 seconds. |
| Heartbeat | Every 20 seconds. |
| Lease | Expires 60 seconds after it was taken or last extended. |
| Attempts | A run whose lease expires is queued again, up to 3 attempts in all. |
| Result output | At most 65,536 bytes of JSON. |
| Batch sizes | Up to 50 runs per lease call, 100 lease ids per extend or release, 50 results per post. |
Keys and identity
Section titled “Keys and identity”A runner has two key pairs. Both are made on the runner, and the private keys never leave it.
| Key | Algorithm | Used for |
|---|---|---|
| Auth key | Ed25519 | Signing the sign-in challenge. |
| Seal key | X25519 | Opening the connection secrets sealed into each lease. |
Public keys are sent as SPKI DER, base64-encoded.
The official runner keeps its identity (runner id, group, both private keys) in runner.json in its state directory, /var/lib/sreinsights-runner by default. Keep that directory on a volume. If it is lost, the runner can’t sign in and must be registered again. See Runner settings.
A Lambda function has no disk that lasts, so it is registered once from another machine with sreinsights-runner identity. That command prints the identity as one line, sri1.…. The line is sri1. followed by the identity file’s JSON, base64url-encoded. Whoever holds it can act as that runner. Store it in a secret, and give it to the function as SREINSIGHTS_RUNNER_IDENTITY, or name the secret in SREINSIGHTS_RUNNER_IDENTITY_SECRET.
Register
Section titled “Register”POST /runner/v1/register. No authentication: the registration token in the body is the key. Make the token on the Runners page, or with POST /api/v1/runner-groups/{id}/registration-tokens.
| Field | Type | Required | What it is |
|---|---|---|---|
token |
string | Yes | The registration token, prt_…. |
name |
string | Yes | The runner’s name, up to 100 characters. |
version |
string | Yes | The runner’s version, up to 40 characters. |
os |
string | Yes | For example linux. Added to the labels. |
arch |
string | Yes | For example x64 or arm64. Added to the labels. |
labels |
string[] | No | Up to 20. Lower-case letters, digits, ., _ or -, up to 63 characters. |
capabilities |
object | Yes | What it can run. See Capabilities. |
authPublicKey |
string | Yes | Ed25519 public key, SPKI DER, base64. |
sealPublicKey |
string | No | X25519 public key, SPKI DER, base64. Send it here or in hello. |
os and arch become labels next to yours, lower-cased, so a monitor can ask for labels: ["linux"] without any setup. Labels that don’t fit the pattern are dropped.
Answers 201:
{ "status": "success", "data": { "runnerId": "0193a1c2-…", "orgId": "0192f0aa-…", "groupId": "0193a0b1-…", "groupName": "eu-prod" }}orgId is null for a hosted runner, which takes work from every organization that uses its group.
| Status | Code | When |
|---|---|---|
| 400 | INVALID_PUBLIC_KEY |
authPublicKey is not an Ed25519 public key. |
| 400 | INVALID_SEAL_KEY |
sealPublicKey is not an X25519 public key. |
| 401 | REGISTRATION_TOKEN_INVALID |
The token is unknown, used up or expired. Make a new one. |
Two runners racing for a single-use token can’t both register with it.
Sign in
Section titled “Sign in”Sign-in takes two calls and needs no secret stored on the runner’s disk other than its private key.
Get a challenge
Section titled “Get a challenge”POST /runner/v1/token/challenge
{ "runnerId": "0193a1c2-…" }Answers { "challenge": "<string>" }. The challenge is valid for 60 seconds. SREInsights answers the same way for any runner id; a challenge is worth nothing without the private key.
Trade it for a token
Section titled “Trade it for a token”Sign the challenge string, as UTF-8 bytes, with the Ed25519 private key. Then POST /runner/v1/token:
| Field | Type | What it is |
|---|---|---|
runnerId |
uuid | The runner’s id from registration. |
challenge |
string | The challenge, exactly as received. |
signature |
string | The Ed25519 signature, base64. |
Answers:
{ "status": "success", "data": { "accessToken": "eyJhbGciOiJIUzI1NiIs…", "expiresAt": "2026-11-22T10:15:00.000Z" } }Send the access token on every other call as Authorization: Bearer <access token>. The official runner gets a new one a minute before expiresAt, and at once when a call answers RUNNER_TOKEN_EXPIRED or RUNNER_TOKEN_INVALID.
A runner’s access token is not a user’s: it works only on /runner/v1, and a user’s token doesn’t work here.
Authentication errors
Section titled “Authentication errors”| Status | Code | What to do |
|---|---|---|
| 401 | RUNNER_UNAUTHENTICATED |
No bearer token. Sign in. |
| 401 | RUNNER_TOKEN_EXPIRED |
Get a new token and retry. |
| 401 | RUNNER_TOKEN_INVALID |
Get a new token and retry. |
| 401 | RUNNER_AUTH_FAILED |
The signature, challenge or runner id didn’t check out. If the state directory was lost, register again. Stop. |
| 401 | RUNNER_UNKNOWN |
The runner no longer exists in SREInsights. Register again. Stop. |
| 401 | RUNNER_DISABLED |
Someone disabled the runner. Register a new runner to continue. Stop. |
The official runner stops when it gets RUNNER_AUTH_FAILED, RUNNER_UNKNOWN or RUNNER_DISABLED: retrying can’t help.
POST /runner/v1/hello, on every start, before leasing. It tells SREInsights what this build of the runner can do and which key to seal secrets to.
| Field | Type | Required | What it is |
|---|---|---|---|
version |
string | Yes | The runner’s version. |
capabilities |
object | Yes | See below. |
sealPublicKey |
string | Yes | X25519 public key, SPKI DER, base64. |
Answers { "ok": true }. A bad seal key answers 400 INVALID_SEAL_KEY.
Capabilities
Section titled “Capabilities”{ "engineApiVersion": 2, "checks": ["http", "postgres", "script", "flow"], "jobTypes": [{ "name": "sync", "version": 1 }]}| Field | Type | What it is |
|---|---|---|
engineApiVersion |
integer | The version of the run payload the runner understands. A runner whose version differs from the control plane’s gets no work. |
checks |
string[] | Check and step types the runner can execute, up to 50. See Check types. |
jobTypes |
{ name, version }[] |
Job types the runner’s image provides, up to 100. A job’s run goes only to a runner that reported its type at its version. See Job types. |
GET /api/v1/runner-groups lists each runner’s job types as <name>@<version>.
Lease runs
Section titled “Lease runs”POST /runner/v1/leases asks for work. It is a long poll: SREInsights answers as soon as there is a run for this runner, or with an empty list when the wait is over.
| Field | Type | Required | Default | What it is |
|---|---|---|---|---|
capacity |
integer, 1–50 | Yes | How many runs the runner can take now. | |
waitMs |
integer, 0–25000 | No | 25000 | How long to wait for work. |
jobId |
uuid | No | Only this run. A Lambda function woken for one run asks for it by id. |
Answers { "jobs": [ … ] }, with up to capacity leased runs.
The control plane may hold the poll for less than waitMs. An instance that runs as functions answers at once by default (it may hold polls for up to 20 seconds there). When the answer is empty and came back early, wait out the rest of waitMs before asking again.
A runner gets a run only when all of these hold:
- The run is for the runner’s group, and, for a group of your own, for the runner’s organization.
- The run’s labels are all among the runner’s labels.
- Every check and step type the run needs is in the runner’s
checks, and its job type, if any, is injobTypes. - The runner’s
engineApiVersionmatches. - The run is still queued and its queue window hasn’t closed.
- The runner is not disabled.
The oldest scheduled run goes first. A hosted runner never gets runs that use scripts, whatever it reports.
If the runner hangs up before the answer arrives, runs claimed for it are handed straight back to the queue.
A leased run
Section titled “A leased run”{ "jobId": "0193b7d0-…", "leaseId": "0193b7d1-…", "leaseExpiresAt": "2026-11-22T10:01:00.000Z", "attempt": 1, "engineApiVersion": 2, "payload": { "monitorId": "0193a1c2-…", "monitorName": "Checkout API", "monitorVersion": 4, "check": { "type": "http", "…": "…" }, "requires": ["http"], "scheduledFor": "2026-11-22T10:00:00.000Z", "params": { "region": "eu" }, "trigger": { "type": "schedule" } }, "sealed": { "v": 1, "epk": "…", "iv": "…", "tag": "…", "ct": "…" }}| Field | What it is |
|---|---|
jobId |
The run’s id. |
leaseId |
This lease. Extend, release and report by it. |
leaseExpiresAt |
When the lease ends unless extended. |
attempt |
1 for the first try, higher when the run was queued again after a lost lease. |
engineApiVersion |
The payload version. |
payload |
What to run. See below. |
sealed |
The run’s connection secrets, sealed to this runner. null when the check uses no connections. |
sealError |
Present when the secrets couldn’t be sent, for example the runner never sent a seal key. Report the run as error with this message. |
payload fields:
| Field | What it is |
|---|---|
monitorId, monitorName, monitorVersion |
The monitor and the version of its definition. A data job’s run has the nil UUID as monitorId. |
check |
The check definition to run. |
requires |
Every check and step type the check needs, a flow’s steps included. |
scheduledFor |
The time slot this run is for. |
params |
The run’s parameter values, already checked by SREInsights. |
trigger |
How the run started: { "type": "schedule" | "manual" | "webhook", "name"?, "body"?, "receivedAt"? }. A webhook’s JSON body is body. |
dataJob |
For a data job’s run: { "id", "name", "orgId"?, "client"?, "backfill"?, "preview"? }. |
Sealed secrets
Section titled “Sealed secrets”Connection secrets are never stored in the queue. They are opened when a run is leased and sealed to the leasing runner’s X25519 key, bound to that lease. Only that runner can open them, and a sealed box can’t be replayed into another lease.
sealed is { "v": 1, "epk", "iv", "tag", "ct" }, all base64. To open it:
epkis an ephemeral X25519 public key (SPKI DER). Do an X25519 key agreement between it and the runner’s seal private key.- Derive a 32-byte key with HKDF-SHA256: the shared secret as input,
epkDER followed by the runner’s own seal public key DER as salt, andsreinsights/sealed-box/v1as info. - Decrypt
ctwith AES-256-GCM, usingiv(12 bytes),tag, and the associated datasreinsights/job-secrets/v1:<leaseId>.
The plaintext is JSON:
{ "connections": { "<connection id>": { "id": "<connection id>", "type": "postgres", "name": "Orders DB", "config": { "host": "db.example.com", "port": 5432 }, "secrets": { "password": "…" }, "secretsOnRunner": false } }, "missing": []}missing lists connections the check names that no longer exist. When secretsOnRunner is true, secrets is empty: read them from the runner’s environment instead, as SREINSIGHTS_SECRET_<CONNECTION NAME>_<FIELD>, for example SREINSIGHTS_SECRET_ORDERS_DB_PASSWORD. See Connections and secrets.
Keep secrets in memory only, for the length of the run.
Heartbeats
Section titled “Heartbeats”POST /runner/v1/leases/extend keeps leases alive. Send it every 20 seconds with every lease the runner is working on.
{ "leaseIds": ["0193b7d1-…", "0193b7e4-…"] }Answers { "extended": [ … ], "lost": [ … ] }. Each extended lease now expires 60 seconds from now.
A lease in lost is no longer this runner’s: it expired, or the run was handed back to the queue. Stop that run and drop its result.
Release
Section titled “Release”POST /runner/v1/leases/release hands runs back without a result, for example when the runner shuts down before they finish.
{ "leaseIds": ["0193b7d1-…"] }Answers { "released": 1 }. Each run goes back to the queue with at least 15 more seconds to find a runner. Its attempt count doesn’t change.
The official runner, when stopped, stops leasing, lets running checks finish for up to 30 seconds, then releases the rest.
Results
Section titled “Results”POST /runner/v1/results reports finished runs, up to 50 at a time.
{ "results": [ { "leaseId": "0193b7d1-…", "status": "ok", "durationMs": 412, "startedAt": "2026-11-22T10:00:01.120Z", "finishedAt": "2026-11-22T10:00:01.532Z", "output": { "statusCode": 200 } } ]}| Field | Type | Required | What it is |
|---|---|---|---|
leaseId |
uuid | Yes | The lease the run was taken under. |
status |
ok, failed, timeout, error |
Yes | See Run statuses. |
durationMs |
integer | Yes | How long the run took. |
startedAt, finishedAt |
ISO date-time | Yes | |
output |
object | No | What the check produced. At most 65,536 bytes as JSON. |
error |
string | No | Why it failed, up to 2,000 characters. |
Answers one acknowledgement per result:
{ "results": [{ "leaseId": "0193b7d1-…", "accepted": true }] }accepted |
reason |
What it means |
|---|---|---|
true |
Stored. | |
true |
duplicate |
Already stored by an earlier post of the same result. |
false |
lease_lost |
The lease was queued again or handed back. Another runner owns the run. Drop the result. |
Posting is idempotent per lease, so retry freely after a network error. A batch that fails validation, for example an output over 65,536 bytes, answers 400 VALIDATION_FAILED for the whole batch; don’t retry it unchanged. The official runner replaces an oversized output with { "truncated": true, "note": "output exceeded 65536 bytes" }.
SREInsights orders results by its own clock: it records a result as finished when it arrives, and started durationMs before that.
Cancellation
Section titled “Cancellation”SREInsights doesn’t call a runner to cancel a run. A runner learns that a run is no longer its own in two ways:
- A heartbeat lists the lease in
lost. - A result post answers
lease_lost.
In both cases, stop the run and drop its result. Disabling a runner on the Runners page hands its leased runs back to the group at once, and its next call answers RUNNER_DISABLED.
Lambda runner groups
Section titled “Lambda runner groups”A Lambda runner group has no runner waiting between runs. When a run is queued for the group, SREInsights invokes the group’s function asynchronously with this event:
{ "source": "sreinsights", "version": 1, "groupId": "0193a0b1-…", "jobId": "0193b7d0-…" }The event carries no check, no secret and nothing a user typed. The function signs in as its group’s runner, says hello, leases that one run with jobId, runs it with heartbeats, posts the result and returns. Someone who invokes the function with a forged event gets nothing a registered runner couldn’t already ask for.
- If the invoke fails, SREInsights tries again after 2 and then 6 seconds, 3 tries in all. A failure that can’t succeed on retry (no permission, no such function) ends the run at once. Either way the run ends as
errorwith reasondispatch_failed. - After the invoke, the run stays queued for at least 30 seconds, so a cold function has time to start and sign in.
- A run that comes back to the queue (a lost lease) is invoked again.
- The official runner refuses an event for another group. Invoked without an SREInsights event, for example a test event from the Lambda console, it takes the group’s oldest queued run, if any.
A Lambda function has no Docker, so it can’t run scripts in a container. See Set up a Lambda runner.
Run statuses
Section titled “Run statuses”A run moves through these states:
| State | What it means |
|---|---|
queued |
Waiting for a runner. The UI shows “Waiting for a runner”. |
leased |
A runner took it and is running it. |
completed |
The runner posted a result. |
skipped |
No runner leased it before its queue window closed. |
error |
SREInsights gave up on it without a result. |
A scheduled run’s queue window is its schedule’s interval, at most 5 minutes and at least 15 seconds. Run now and webhook runs wait up to 5 minutes.
The result a monitor shows is one of:
| Status | Reason | Who records it | What it means |
|---|---|---|---|
ok |
Runner | The check ran and every assertion passed. | |
failed |
Runner | The check ran and an assertion failed: the target is unhealthy. | |
timeout |
Runner | No answer within the check’s timeout. | |
error |
Runner | The check couldn’t run: DNS failure, refused connection, bad configuration. | |
skipped |
no_runner |
SREInsights | Nobody leased the run in its queue window. On a Lambda group: the function was invoked and didn’t take the run. |
error |
runner_lost |
SREInsights | The lease expired on every attempt. |
error |
dispatch_failed |
SREInsights | A Lambda group’s function couldn’t be invoked. |
When a lease expires because its runner stopped sending heartbeats, the run goes back to the group with its attempt count raised and a fresh queue window. After the third attempt it ends as error / runner_lost. A late result from the old runner is refused with lease_lost.
Skipped runs and runner_lost runs say nothing about the target, so alert rules don’t count them. See Runners and Troubleshooting.