Skip to content

How SREInsights works

SREInsights has two halves:

  • The control plane is the console and API at your SREInsights URL. It stores your monitors, jobs, rules and routes, keeps the run queue, evaluates results and sends notifications. It never runs a check and never connects into your network.
  • Runners do the work. A runner is a container you run where it can reach what you check, an AWS Lambda function in your account, or a hosted runner for public endpoints. Runners connect out to the control plane over HTTPS.

This works like GitHub Actions with self-hosted runners.

  1. The scheduler queues the run. At each time the monitor’s schedule names, SREInsights puts one run in the queue for the monitor’s runner group. Run now and a webhook trigger queue a run too. A monitor never has two runs at once.
  2. A runner leases it. Runners in the group ask for work. SREInsights gives the run to one runner whose labels and capabilities match. For a Lambda runner group, SREInsights invokes the group’s function, and the function leases the run.
  3. Secrets arrive sealed. The connections the check uses are decrypted and sealed to that runner’s own key, bound to this one lease. Another runner can’t open them, and they can’t be replayed.
  4. The runner runs the check and reports the result: passed or failed, how long it took, and what it found (rows, a status code, a value).
  5. The processor evaluates the rules. It compares the result with the monitor’s alert rules and opens, updates or resolves alerts.
  6. The delivery worker notifies. Each change that should notify someone goes through the routes to channels: PagerDuty, Slack, Telegram, email and others. A resolve never overtakes its firing, and failed sends are retried with back-off.

Jobs follow the same path. A job run is leased, sealed and reported the same way. Its outcome is recorded on the job, and it raises its own alerts when it fails or when no new data arrives.

What you see What it means
Waiting for a runner The run is queued and no runner has taken it yet.
skipped / no_runner No runner took the run in time (the monitor’s interval, at most 5 minutes). It does not fire the monitor’s alert. Check that the group has an online runner with the labels the monitor asks for.
error / runner_lost A runner took the run and went quiet three times in a row.
error / dispatch_failed A Lambda group’s function could not be invoked. The result says why. It counts as an error and fires the alert.
timeout The check ran past its time limit.

Troubleshooting has more.

Thing Where
Monitors, rules, routes, results, alerts The control plane’s Postgres database, separated per organization by row-level security
Connection secrets Encrypted in that database under your organization’s own key, or kept only on your runners as environment variables SREInsights never sees
Check code (HTTP, SQL, scripts, flows) Runs on the runner
Job type code Ships inside the runner image; SREInsights knows a job type only by its name, version and settings
Data a job collects Its destination (for example your own S3 bucket); never the control plane

Everything belongs to an organization. Members have a role: Owner, Admin, Operator, Responder or Viewer. An organization can keep clients apart inside it, which suits agencies: each client owns its connections, jobs and monitors, and a member can be limited to some clients.