Runners and runner groups
A runner runs checks and jobs and reports the results. SREInsights itself never runs a check: it only queues work and hands it to runners. Runners connect out to SREInsights over HTTPS, so nothing has to connect into your network.
A runner group is a set of runners that can reach the same things, such as prod-eu-dc or staging. Every monitor and job names one group in its Runner group setting (runsOn in the API). There is no fallback: a monitor whose group has no suitable runner is skipped.
Three kinds of group
Section titled “Three kinds of group”| Runners you host | An AWS Lambda function | Hosted | |
|---|---|---|---|
| What runs | Containers you keep running | Your function, invoked for each run | Runners operated for every organization |
| Reaches | Whatever its network reaches | Whatever the function reaches | Public addresses only |
| Scripts | Yes | Yes, runtime: local only |
No |
| Secrets kept on the runner | Yes | Yes | No |
| Shown as | Online / offline per runner | On demand | Location and runners online |
| Set up in | Install a Docker runner | Set up a Lambda runner | Nothing to set up |
Runners you host
Section titled “Runners you host”The runner is one container image, ghcr.io/bintree-softwares/sreinsights-runner. It is small (about 50 MB), runs as a non-root user with no shell, and works with a read-only file system.
- Registration. On Runners, Add runner on a group creates a one-time registration token and a ready
docker runcommand. The runner uses the token once to register a key pair. After that it signs in with its private key, kept in its state volume. No shared secret stays on the runner. - Work. A runner long-polls SREInsights for runs, sends a heartbeat while it works, and posts the result. If it stops sending heartbeats, its lease expires and the run goes back to the queue.
- Liveness. The Runners page shows each runner’s status, labels, job types, version and when it was last seen.
- Disabling a runner cuts it off at its next call and hands its runs back to the group.
- Two or more runners per group, on different hosts, keep checks running when one host goes down.
Lambda runner groups
Section titled “Lambda runner groups”In a Lambda group nothing runs between runs. When a run is queued for the group, SREInsights invokes your AWS Lambda function once. The function is a runner for that one run: it signs in, takes the run it was woken for, runs it, reports, and exits. It suits work that runs now and then, such as nightly data collection or a script that used to be a cron job.
- A run can take up to 15 minutes, AWS’s limit for a function. A run still going 20 seconds before the function’s timeout is stopped and reported as
timeout. - There is no Docker in a function, so script checks must use
runtime: local. - One run per invocation. The group’s concurrency is the function’s concurrency.
- A Lambda group never raises a runner group offline alert. Problems show on the run:
error/dispatch_failedwhen the function could not be invoked,skipped/no_runnerwhen it was invoked but did not take the run in time. - The instance operator must allow Lambda groups. On your own instance, see Self-hosting. On a shared instance such as the hosted product, organizations can’t create their own Lambda groups.
Hosted runners
Section titled “Hosted runners”Hosted runner groups are run by the SREInsights operator for every organization, in named locations. Use them for websites and public APIs when you don’t want to install anything. They:
- reach public addresses only; private, loopback, link-local (cloud metadata) and other reserved addresses are refused, even through DNS,
- don’t run scripts (also inside flows),
- can’t use connections whose secrets are kept on your runners,
- count against your plan’s limits for hosted monitors and hosted jobs, and its shortest interval.
Each run receives only its own organization’s connections, sealed for that run. The monitor form lists hosted groups under their own heading, and the Runners page shows them read-only with how many runners are online.
Labels and capabilities
Section titled “Labels and capabilities”A run goes only to a runner that can do it.
- Labels. A runner reports labels:
osandarchautomatically, plus any you set inSREINSIGHTS_RUNNER_LABELS, such aselastic-prod,eu. A monitor’s Runner labels must all be on the runner for it to take the run. - Capabilities. A runner reports which check types, integration actions and job types it provides. A flow is leased only to a runner that provides every step it uses, and a job only to a runner whose image provides that job type at that version.
When a group goes offline
Section titled “When a group goes offline”When every enabled runner in a group of runners you host has been quiet for a minute, SREInsights raises one critical runner group offline alert. New runners get a minute’s grace, and a group whose runners are all disabled stays quiet. The alert resolves when a runner returns or the group is deleted.
Runs that no runner takes
Section titled “Runs that no runner takes”A run that no runner leases within the monitor’s interval (at most 5 minutes) is recorded as skipped with reason no_runner. It never fires the monitor’s own alert, so a missing runner doesn’t page you for every monitor at once. The runner group offline alert covers that.
See also: Runner settings, Runner protocol.