Skip to content

Troubleshooting

Waiting for a runner, then skipped (no_runner)

Section titled “Waiting for a runner, then skipped (no_runner)”

No runner took the run within the monitor’s interval (at most 5 minutes). It fires no alert. Check, on Runners:

  • Does the monitor’s group have an online runner? The monitor form says so under Runner group.
  • Does the runner have every label in the monitor’s Runner labels?
  • Does it provide what the run needs? A job needs a runner that lists its job type at that version. A flow needs every step’s action. A script with runtime: docker needs a runner with Docker.
  • On a Lambda group: the function was invoked but didn’t take the run. It may have no identity, may not reach SREInsights, or may lack the labels. Look at the function’s CloudWatch logs.

A Lambda group’s function could not be invoked. The run’s result carries AWS’s reason. Usual causes: the control plane has no lambda:InvokeFunction on the function, the function name or region is wrong, or the name doesn’t start with the instance’s allowed prefix. It counts as a failure and fires the monitor’s alert.

A runner took the run and stopped sending heartbeats, three times. The runner may have been killed mid-run (for example a container stopped without --stop-timeout 40), run out of memory, or lost its network.

The check ran past its time limit. Raise its timeout (scripts allow up to 10 minutes), or for a job its Time limit (minutes). On a Lambda runner a run is stopped 20 seconds before the function’s own timeout, at most 15 minutes.

The run fails with “No complete response within … ms”

Section titled “The run fails with “No complete response within … ms””

The HTTP request didn’t finish in its timeout. Check the address from the runner’s network, not from your laptop: the runner is what connects.

A script fails with “exited with code 127” or “not found”

Section titled “A script fails with “exited with code 127” or “not found””

The interpreter isn’t in the runner image. The standard images have Node and a shell. For Python, extend the image, or use runtime: docker on a runner with Docker.

Scripts get a fixed environment, their own env, and only the runner variables listed in SREINSIGHTS_SCRIPT_ENV_PASSTHROUGH (for runtime: local). SREINSIGHTS_SECRET_*, the registration token and the runner identity are never passed. See Move a scheduled job in.

Its state volume was lost. Mount a volume at /var/lib/sreinsights-runner and keep it. Then create a new token with Add runner.

The runner hasn’t checked in for a minute. Check the container is running, its logs, and that it reaches your SREInsights URL over HTTPS. When every runner of a group is quiet, SREInsights raises one runner group offline alert.

Error Means
400 HOSTED_SCRIPT_NOT_ALLOWED Hosted runners don’t run scripts, also inside flows.
400 HOSTED_RUNNER_LOCAL_SECRETS The monitor uses a connection whose secrets are kept on your runners.
403 PLAN_LIMIT The schedule is shorter than your plan allows on hosted runners, or you have reached the plan’s count.

A hosted runner also can’t reach private addresses: a check of an internal host fails to connect.

  • The run was skipped or runner_lost: those count neither way.
  • The monitor has rules and none of them fired. Open the rule and look at its preview.
  • The alert opened but is silenced: it shows on Alerts, and its notices wait.

Open the alert and look at Notifications:

  • Nothing listed: no route matched. Use Try an alert on Notifications with the alert’s severity and labels.
  • Failed with a reason: fix the connection. Bad credentials fail at once; other errors are retried with back-off.
  • Held: the alert is silenced, flapping, or inside an alert storm.
Message Fix
Telegram: chat not found Wrong Chat id, or the bot isn’t in the chat. Group ids start with -100.
Slack: “Invite the bot to …” Invite the bot to the channel, or use a webhook connection.
SNS: no AWS credentials, or access denied SREInsights publishes with its own role; it needs sns:Publish on the topic.

INGEST_UNAUTHORIZED: the token is wrong or the connection is gone. Create a new token on the connection and update the tool.

Error Means
409 JOB_RUN_ACTIVE A run is already queued or running. Wait for it.
409 SYNC_NOT_PREVIEWED Preview the sync’s current version before switching it on.
400 RUNNER_CREDENTIALS_NOT_HOSTED A type that runs on the runner’s own credentials can’t run on a hosted group.
Job “…” has no new data No run wrote data for the job’s freshness. Look at the latest runs: partial runs say what was missing.
  • Forgot a password: SREInsights sends no email. Ask an Owner or Admin for a reset link from Team.
  • An invitation link doesn’t work: it works once. Ask for a new one.
  • “Lambda runner groups aren’t switched on” (LAMBDA_RUNNERS_DISABLED): the instance doesn’t allow Lambda groups. See Set up a Lambda runner.

Still stuck? Check the FAQ, or the error codes in the REST API reference.