Skip to content

Runners and runner groups

A runner runs checks and jobs and reports the results. SREInsights itself never runs a check: it only queues work and hands it to runners. Runners connect out to SREInsights over HTTPS, so nothing has to connect into your network.

A runner group is a set of runners that can reach the same things, such as prod-eu-dc or staging. Every monitor and job names one group in its Runner group setting (runsOn in the API). There is no fallback: a monitor whose group has no suitable runner is skipped.

Runners you host An AWS Lambda function Hosted
What runs Containers you keep running Your function, invoked for each run Runners operated for every organization
Reaches Whatever its network reaches Whatever the function reaches Public addresses only
Scripts Yes Yes, runtime: local only No
Secrets kept on the runner Yes Yes No
Shown as Online / offline per runner On demand Location and runners online
Set up in Install a Docker runner Set up a Lambda runner Nothing to set up

The runner is one container image, ghcr.io/bintree-softwares/sreinsights-runner. It is small (about 50 MB), runs as a non-root user with no shell, and works with a read-only file system.

  • Registration. On Runners, Add runner on a group creates a one-time registration token and a ready docker run command. The runner uses the token once to register a key pair. After that it signs in with its private key, kept in its state volume. No shared secret stays on the runner.
  • Work. A runner long-polls SREInsights for runs, sends a heartbeat while it works, and posts the result. If it stops sending heartbeats, its lease expires and the run goes back to the queue.
  • Liveness. The Runners page shows each runner’s status, labels, job types, version and when it was last seen.
  • Disabling a runner cuts it off at its next call and hands its runs back to the group.
  • Two or more runners per group, on different hosts, keep checks running when one host goes down.

In a Lambda group nothing runs between runs. When a run is queued for the group, SREInsights invokes your AWS Lambda function once. The function is a runner for that one run: it signs in, takes the run it was woken for, runs it, reports, and exits. It suits work that runs now and then, such as nightly data collection or a script that used to be a cron job.

  • A run can take up to 15 minutes, AWS’s limit for a function. A run still going 20 seconds before the function’s timeout is stopped and reported as timeout.
  • There is no Docker in a function, so script checks must use runtime: local.
  • One run per invocation. The group’s concurrency is the function’s concurrency.
  • A Lambda group never raises a runner group offline alert. Problems show on the run: error/dispatch_failed when the function could not be invoked, skipped/no_runner when it was invoked but did not take the run in time.
  • The instance operator must allow Lambda groups. On your own instance, see Self-hosting. On a shared instance such as the hosted product, organizations can’t create their own Lambda groups.

Hosted runner groups are run by the SREInsights operator for every organization, in named locations. Use them for websites and public APIs when you don’t want to install anything. They:

  • reach public addresses only; private, loopback, link-local (cloud metadata) and other reserved addresses are refused, even through DNS,
  • don’t run scripts (also inside flows),
  • can’t use connections whose secrets are kept on your runners,
  • count against your plan’s limits for hosted monitors and hosted jobs, and its shortest interval.

Each run receives only its own organization’s connections, sealed for that run. The monitor form lists hosted groups under their own heading, and the Runners page shows them read-only with how many runners are online.

A run goes only to a runner that can do it.

  • Labels. A runner reports labels: os and arch automatically, plus any you set in SREINSIGHTS_RUNNER_LABELS, such as elastic-prod,eu. A monitor’s Runner labels must all be on the runner for it to take the run.
  • Capabilities. A runner reports which check types, integration actions and job types it provides. A flow is leased only to a runner that provides every step it uses, and a job only to a runner whose image provides that job type at that version.

When every enabled runner in a group of runners you host has been quiet for a minute, SREInsights raises one critical runner group offline alert. New runners get a minute’s grace, and a group whose runners are all disabled stays quiet. The alert resolves when a runner returns or the group is deleted.

A run that no runner leases within the monitor’s interval (at most 5 minutes) is recorded as skipped with reason no_runner. It never fires the monitor’s own alert, so a missing runner doesn’t page you for every monitor at once. The runner group offline alert covers that.

See also: Runner settings, Runner protocol.