Troubleshooting
Waiting for a runner, then skipped (no_runner)
Section titled “Waiting for a runner, then skipped (no_runner)”No runner took the run within the monitor’s interval (at most 5 minutes). It fires no alert. Check, on Runners:
- Does the monitor’s group have an online runner? The monitor form says so under Runner group.
- Does the runner have every label in the monitor’s Runner labels?
- Does it provide what the run needs? A job needs a runner that lists its job type at that version. A flow needs every step’s action. A script with
runtime: dockerneeds a runner with Docker. - On a Lambda group: the function was invoked but didn’t take the run. It may have no identity, may not reach SREInsights, or may lack the labels. Look at the function’s CloudWatch logs.
error (dispatch_failed)
Section titled “error (dispatch_failed)”A Lambda group’s function could not be invoked. The run’s result carries AWS’s reason. Usual causes: the control plane has no lambda:InvokeFunction on the function, the function name or region is wrong, or the name doesn’t start with the instance’s allowed prefix. It counts as a failure and fires the monitor’s alert.
error (runner_lost)
Section titled “error (runner_lost)”A runner took the run and stopped sending heartbeats, three times. The runner may have been killed mid-run (for example a container stopped without --stop-timeout 40), run out of memory, or lost its network.
timeout
Section titled “timeout”The check ran past its time limit. Raise its timeout (scripts allow up to 10 minutes), or for a job its Time limit (minutes). On a Lambda runner a run is stopped 20 seconds before the function’s own timeout, at most 15 minutes.
The run fails with “No complete response within … ms”
Section titled “The run fails with “No complete response within … ms””The HTTP request didn’t finish in its timeout. Check the address from the runner’s network, not from your laptop: the runner is what connects.
A script fails with “exited with code 127” or “not found”
Section titled “A script fails with “exited with code 127” or “not found””The interpreter isn’t in the runner image. The standard images have Node and a shell. For Python, extend the image, or use runtime: docker on a runner with Docker.
A script can’t see a variable
Section titled “A script can’t see a variable”Scripts get a fixed environment, their own env, and only the runner variables listed in SREINSIGHTS_SCRIPT_ENV_PASSTHROUGH (for runtime: local). SREINSIGHTS_SECRET_*, the registration token and the runner identity are never passed. See Move a scheduled job in.
Runners
Section titled “Runners”A restarted runner asks for a new token
Section titled “A restarted runner asks for a new token”Its state volume was lost. Mount a volume at /var/lib/sreinsights-runner and keep it. Then create a new token with Add runner.
A runner shows offline
Section titled “A runner shows offline”The runner hasn’t checked in for a minute. Check the container is running, its logs, and that it reaches your SREInsights URL over HTTPS. When every runner of a group is quiet, SREInsights raises one runner group offline alert.
A hosted runner refuses a monitor
Section titled “A hosted runner refuses a monitor”| Error | Means |
|---|---|
400 HOSTED_SCRIPT_NOT_ALLOWED |
Hosted runners don’t run scripts, also inside flows. |
400 HOSTED_RUNNER_LOCAL_SECRETS |
The monitor uses a connection whose secrets are kept on your runners. |
403 PLAN_LIMIT |
The schedule is shorter than your plan allows on hosted runners, or you have reached the plan’s count. |
A hosted runner also can’t reach private addresses: a check of an internal host fails to connect.
Alerts and notifications
Section titled “Alerts and notifications”A run failed but no alert opened
Section titled “A run failed but no alert opened”- The run was
skippedorrunner_lost: those count neither way. - The monitor has rules and none of them fired. Open the rule and look at its preview.
- The alert opened but is silenced: it shows on Alerts, and its notices wait.
An alert opened but nobody was told
Section titled “An alert opened but nobody was told”Open the alert and look at Notifications:
- Nothing listed: no route matched. Use Try an alert on Notifications with the alert’s severity and labels.
- Failed with a reason: fix the connection. Bad credentials fail at once; other errors are retried with back-off.
- Held: the alert is silenced, flapping, or inside an alert storm.
Telegram, Slack or email fails
Section titled “Telegram, Slack or email fails”| Message | Fix |
|---|---|
| Telegram: chat not found | Wrong Chat id, or the bot isn’t in the chat. Group ids start with -100. |
| Slack: “Invite the bot to …” | Invite the bot to the channel, or use a webhook connection. |
| SNS: no AWS credentials, or access denied | SREInsights publishes with its own role; it needs sns:Publish on the topic. |
An incoming webhook answers 401
Section titled “An incoming webhook answers 401”INGEST_UNAUTHORIZED: the token is wrong or the connection is gone. Create a new token on the connection and update the tool.
Jobs and syncs
Section titled “Jobs and syncs”| Error | Means |
|---|---|
409 JOB_RUN_ACTIVE |
A run is already queued or running. Wait for it. |
409 SYNC_NOT_PREVIEWED |
Preview the sync’s current version before switching it on. |
400 RUNNER_CREDENTIALS_NOT_HOSTED |
A type that runs on the runner’s own credentials can’t run on a hosted group. |
| Job “…” has no new data | No run wrote data for the job’s freshness. Look at the latest runs: partial runs say what was missing. |
Signing in
Section titled “Signing in”- Forgot a password: SREInsights sends no email. Ask an Owner or Admin for a reset link from Team.
- An invitation link doesn’t work: it works once. Ask for a new one.
- “Lambda runner groups aren’t switched on” (
LAMBDA_RUNNERS_DISABLED): the instance doesn’t allow Lambda groups. See Set up a Lambda runner.
Still stuck? Check the FAQ, or the error codes in the REST API reference.