Skip to content

Alert rules

An alert rule belongs to one monitor. After every run, SREInsights evaluates the monitor’s rules over its recent results and opens, updates or resolves an alert for each.

A monitor without rules still alerts: a failed run fires a high alert called Check failing, and the next successful run resolves it.

Add rules on the monitor’s page under Alert rules, with Add rule. The form previews what the rule would have said about the stored runs before you save.

Fires when enough runs in a window count as breaches.

  • Looking at the last N runs or the last N minutes.
  • A run counts when it fails (pick Failed, Timed out, Errored) or an expression is true, such as durationMs > 2000.
  • Fire at at least N runs, or a share of the runs in the window.

Examples: “3 of the last 5 runs failed”, “4 timeouts in 10 minutes”, “at least half the runs in the last hour were slower than 2 seconds”.

Fires when a number from the runs crosses a limit.

  • Value: which number, such as durationMs or output.value.
  • Taking the last, avg, min, max, sum or p95 over the window.
  • Is >, >=, <, <=, == or != the Limit.
  • Min runs: don’t judge until the window has this many runs.
  • Resolve only past: an optional second line the value must cross back over before the alert resolves. It stops an alert flickering at the limit.

Example: “average response time over 10 minutes above 800 ms, from at least 5 runs; resolve when it drops below 500 ms”.

Option What it does
Name The alert’s title.
Severity critical, high, medium, low or info. Routes match on it.
Severity tiers Stricter levels of the same rule, such as medium above 800 ms and critical above 2000 ms. See Severity tiers.
Clean checks before resolving How many good evaluations in a row resolve the alert.
  • A skipped run (no runner took it) and a runner lost run count neither for nor against a rule. A missing runner is reported by the runner group’s own alert instead.
  • A run whose Lambda function could not be invoked (dispatch_failed) counts as an error, like any other.
  • For a PromQL check with one alert per series, rules are evaluated per series, so each pod or container has its own alert. A run that errors changes no series.
{ "name": "Failing repeatedly", "severity": "critical", "resolveAfter": 2,
"condition": { "type": "count", "window": { "type": "evaluations", "count": 5 }, "atLeast": 3 } }

Every field, with more examples, is in the alert rule reference.