Alert rules
An alert rule decides when a monitor’s results raise an alert, how severe it is, and when it resolves. This page is the full schema, with examples you can paste. It ends with the syntax of notification routes and silences, which decide who hears about the alert.
For the ideas behind rules, read Alert rules and Severity tiers.
Where you set rules
Section titled “Where you set rules”In the web app, open a monitor and select Add rule under Alert rules. The drawer shows Right now, against past runs: what the rule would say about the monitor’s stored results, before you save.
In the API:
| Method and path | What it does |
|---|---|
GET /api/v1/monitors/:id/rules |
The monitor’s rules, each with its lastEvaluation (at, matched, value, detail, error). |
POST /api/v1/monitors/:id/rules |
Adds a rule. Returns 201. |
POST /api/v1/monitors/:id/rules/preview |
What a rule would say now. Saves nothing. See Preview a rule. |
PATCH /api/v1/alert-rules/:id |
Changes some fields. The merged rule is checked as a whole. |
DELETE /api/v1/alert-rules/:id |
Deletes a rule. Its open alert resolves on the next evaluation. |
Saving a rule evaluates it again at once.
A monitor without rules
Section titled “A monitor without rules”A monitor with no rules alerts when a run fails. It behaves like this rule:
{ "name": "Check failing", "severity": "high", "resolveAfter": 1, "condition": { "type": "count", "window": { "type": "evaluations", "count": 1 }, "atLeast": 1 } }A failed, timed-out or errored run fires a high alert, and the next successful run resolves it. Once you add a rule, this default alert resolves and only your rules apply.
The rule
Section titled “The rule”{ "name": "Latency high", "enabled": true, "severity": "high", "condition": { "type": "threshold", "value": "durationMs", "comparator": ">", "threshold": 800 }, "resolveAfter": 2, "reopenWindowSeconds": 1800, "labels": { "team": "payments" }, "annotations": { "summary": "Checkout is slow", "runbook_url": "https://wiki.example.com/runbooks/checkout" }}| Field | Type | Required | Default | What it does |
|---|---|---|---|---|
name |
string, 1–120 | Yes | Shown in the alert’s title, <monitor name>: <rule name>. |
|
enabled |
boolean | No | true |
A disabled rule is not evaluated, and its open alert resolves. |
severity |
severity | Yes | critical, high, medium, low or info. |
|
condition |
object | Yes | A count or threshold condition. | |
resolveAfter |
integer, 1–20 | No | 2 |
Consecutive evaluations that must not match before the alert resolves. Clean checks before resolving in the drawer. |
reopenWindowSeconds |
integer, 0–86400 | No | 1800 |
An alert that fires again within this many seconds of resolving reopens, keeping its incident and ticket. Reopen if it fires again within (min) in the drawer. |
labels |
object | No | {} |
Added to the alert’s labels, for routes and silences. Names are lower-case letters, digits and _ (up to 63 characters, not starting with a digit); values up to 255 characters. |
annotations |
object | No | {} |
summary (≤ 500, Summary for responders), description (≤ 4000), runbook_url (a URL, Runbook URL). Notices show the summary and link the runbook. |
An alert raised by a rule carries the label monitor=<monitor name>, the monitor’s own labels, and the rule’s labels, which win on a clash.
Invalid fields return 400 VALIDATION_FAILED with the path of each field to fix. An expression that doesn’t parse returns 400 INVALID_EXPRESSION with its path, condition.value or condition.breach.expression.
Windows
Section titled “Windows”Both kinds of rule look at a window of results.
window |
Fields | What it holds | Drawer |
|---|---|---|---|
{ "type": "evaluations", "count": 5 } |
count: 1–100 |
The last N results. | The last N runs |
{ "type": "duration", "seconds": 600 } |
seconds: 60–86400 |
Results that finished within the last N seconds, ending at the newest result. | The last N minutes |
A rule fires when the window that ends at the newest result matches. It resolves when the windows ending at each of the newest resolveAfter results are all clear.
Count rules
Section titled “Count rules”“3 of the last 5 runs failed”, “at least 4 timeouts in 10 minutes”, “at least 50 % of runs in 15 minutes”. Runs keep failing in the drawer.
| Field | Type | Required | Default | What it does |
|---|---|---|---|---|
type |
"count" |
Yes | ||
window |
window | Yes | See Windows. | |
breach |
object | No | every failed, timed-out or errored run | What makes one result count. A run counts when in the drawer. |
atLeast |
integer, 1–10000 | One of the two | Fire when at least this many results in the window breached. With an evaluations window, it can’t be more than count. |
|
atLeastPercent |
number, > 0 to 100 | One of the two | Fire when at least this percentage of the results in the window breached (at least one). | |
escalations |
array, up to 4 | No | [] |
Severity tiers. See Escalations. |
Set exactly one of atLeast and atLeastPercent.
breach takes one of two forms:
breach |
What counts | Drawer |
|---|---|---|
{ "type": "status", "statuses": ["failed", "timeout", "error"] } |
Results with one of these statuses. statuses defaults to all three. |
It fails, with Failed, Timed out, Errored |
{ "type": "expression", "expression": "durationMs > 2000" } |
Results where the expression is true, even if the run succeeded. Up to 500 characters. | An expression is true |
Threshold rules
Section titled “Threshold rules”“Average durationMs over 10 minutes above 800”, “p95 over the last 20 runs above 1500”, “output.count above 50”. A value crosses a limit in the drawer.
| Field | Type | Required | Default | What it does |
|---|---|---|---|---|
type |
"threshold" |
Yes | ||
value |
expression | Yes | The number to watch, e.g. durationMs, output.durationMs or output.count. Up to 500 characters. Value in the drawer. |
|
aggregate |
last, avg, min, max, sum, p95 |
No | last |
How the window’s values become one. p95 is the nearest-rank 95th percentile. Taking the in the drawer, where last reads “latest”. |
window |
window | No | { "type": "evaluations", "count": 1 } |
See Windows. |
comparator |
>, >=, <, <=, ==, != |
Yes | Is in the drawer. | |
threshold |
number | Yes | The limit. Limit in the drawer. | |
resolveThreshold |
number | No | Once firing, the alert holds until the value is past this limit. With > or >= it must be at or below threshold; with < or <=, at or above. Not with == or !=. Resolve only past in the drawer. |
|
minSamples |
integer, 1–100 | No | 1 |
Values the window needs before it can fire, so a 10-minute average doesn’t fire on one run. Min runs in the drawer. |
escalations |
array, up to 4 | No | [] |
Severity tiers. See Escalations. |
A result whose value isn’t a number is left out of the window. Booleans count as 1 and 0, and numeric strings as numbers.
With "comparator": ">", "threshold": 0.9, "resolveThreshold": 0.8, a value of 0.85 keeps a firing alert firing, so it doesn’t flap at the line.
Expressions in rules
Section titled “Expressions in rules”breach.expression and value are JEXL expressions over one result:
| Name | What it is |
|---|---|
status |
ok, failed, timeout or error. |
durationMs |
How long the run took. |
output |
What the check returned, e.g. output.statusCode, output.count, output.certificate.daysLeft. |
If an expression fails on a result, the rule’s lastEvaluation.error says so, and the rule never resolves an alert while that is set.
Escalations (severity tiers)
Section titled “Escalations (severity tiers)”Stricter levels of the same rule that raise the alert’s severity. It stays one alert that moves up and down between levels. Severity tiers in the drawer.
| Rule kind | Each tier | Rules |
|---|---|---|
| Count | { "severity": "critical", "atLeast": 4 } or { "severity": "critical", "atLeastPercent": 80 } |
Use the same form as the rule (atLeast or atLeastPercent). Each must be higher than the level before it, and an atLeast within the window’s count. |
| Threshold | { "severity": "critical", "threshold": 2000 } |
Only with >, >=, < or <=. Each limit is further in the comparator’s direction than the one before it: higher for >, lower for <. |
Each tier’s severity must be more severe than the rule’s and the tier’s before it. The alert takes the highest tier the newest window reaches. Rising a tier notifies as escalated; falling back is recorded without a notice. See Severity tiers.
What counts as a result
Section titled “What counts as a result”Rules judge the check’s own results. Some runs say nothing about the target:
| Run outcome | Counts? |
|---|---|
ok, failed, timeout, error |
Yes. |
skipped with reason no_runner (nobody leased the run in time) |
Neither for nor against a rule. |
error with reason runner_lost (leases kept expiring) |
Neither for nor against a rule. |
error with reason dispatch_failed (a Lambda runner group’s function could not be invoked) |
Yes, as an error, like any other. |
When every runner in a group goes quiet, the group raises its own alert instead. See Runners.
PromQL checks: one alert per series
Section titled “PromQL checks: one alert per series”A promql check with perSeries (the default, true) judges every series its query returns on its own, and each raises its own alert: one per pod or container.
- Each series is one result per run. Its
statusisfailedwhen the series is failing andokotherwise, and itsoutputis{ "value": <number>, "labels": { … } }. So a count rule with the defaultbreachcounts failing series, and a threshold rule watchesoutput.value. - Without an expression on the check, every series the query returns is failing, so alert-style queries such as
… > 0.9work as written. - Only
okandfailedruns count. A run that timed out or errored says nothing about any series, so it neither fires nor resolves them. - A series missing from a run counts as healthy in that run. When the query no longer returns a series, its alert resolves.
- The alert’s title is
<monitor>: <rule> · <namespace>/<pod>/<container>(or other labels that tell series apart). The series’ labels become alert labels, lower-cased, with__name__asmetric. - Up to 500 series per rule.
Set "perSeries": false on the check to judge the query’s result as a whole.
Preview a rule
Section titled “Preview a rule”POST /api/v1/monitors/:id/rules/preview evaluates a condition against the monitor’s stored results (up to the latest 3000, without skipped and runner-lost runs) and saves nothing.
POST /api/v1/monitors/:id/rules/previewContent-Type: application/json
{ "condition": { "type": "count", "window": { "type": "evaluations", "count": 5 }, "atLeast": 3 }, "resolveAfter": 2}{ "status": "success", "data": { "firing": false, "clear": true, "latest": { "matched": false, "value": 1, "samples": 5, "tier": null, "detail": "1 of 5 runs in the last 5 runs breached (fires at 3)" }, "error": null, "results": 240 }}| Field | What it says |
|---|---|
firing |
The newest window matches: the alert would be firing. The drawer shows Would fire. |
clear |
The newest resolveAfter windows are all clear: a firing alert would resolve. The drawer shows Clear, or Not firing, not yet clear when neither holds. |
latest |
The newest window: matched, value (breaching results, or the aggregated value), samples, tier (index into escalations, or null) and a readable detail. |
error |
An expression that failed on a result. |
results |
How many stored results were judged. |
resolveAfter defaults to 2. The preview judges the monitor’s results as a whole, not per series.
Examples
Section titled “Examples”POST /api/v1/monitors/:id/rules with any of these bodies.
// 3 of the last 5 runs failed, timed out or errored; resolve after 2 clean evaluations{ "name": "Failing repeatedly", "severity": "critical", "resolveAfter": 2, "condition": { "type": "count", "window": { "type": "evaluations", "count": 5 }, "atLeast": 3 } }
// At least 4 slow runs in 10 minutes, even if each one succeeded{ "name": "Slow responses", "severity": "medium", "condition": { "type": "count", "breach": { "type": "expression", "expression": "durationMs > 2000" }, "window": { "type": "duration", "seconds": 600 }, "atLeast": 4 } }
// Latency with tiers: medium above 800 ms, high above 1200 ms, critical above 2000 ms, as one alert{ "name": "Latency", "severity": "medium", "condition": { "type": "threshold", "value": "durationMs", "comparator": ">", "threshold": 800, "escalations": [ { "severity": "high", "threshold": 1200 }, { "severity": "critical", "threshold": 2000 } ] } }
// Average latency over 10 minutes above 800 ms, from at least 5 runs; stays firing until it drops below 500 ms{ "name": "Latency high", "severity": "high", "condition": { "type": "threshold", "value": "output.durationMs", "aggregate": "avg", "window": { "type": "duration", "seconds": 600 }, "minSamples": 5, "comparator": ">", "threshold": 800, "resolveThreshold": 500 } }
// TLS certificate: medium under 14 days left, high under 7, critical under 2{ "name": "Certificate expiring", "severity": "medium", "resolveAfter": 1, "condition": { "type": "threshold", "value": "output.certificate.daysLeft", "comparator": "<", "threshold": 14, "escalations": [ { "severity": "high", "threshold": 7 }, { "severity": "critical", "threshold": 2 } ] } }The certificate rule needs an HTTP check with "certificate": true on its request, which adds the certificate’s issuer, expiry and daysLeft to the result. A result without a certificate has no value, so it neither fires nor counts toward minSamples. See Check types.
Notification routes
Section titled “Notification routes”Routes decide which channels hear about an alert. They are one ordered list per organization, on the Notifications page under Routes, or in the API:
PUT /api/v1/notification-policyContent-Type: application/json
{ "routes": [ { "name": "Critical pages", "match": { "severity": ["critical"] }, "channels": ["<pagerduty channel id>"], "continue": true }, { "name": "Payments team", "match": { "labels": { "team": "payments" } }, "channels": ["<slack channel id>"] }, { "name": "Acme's alerts", "match": { "clientIds": ["<client id>"] }, "channels": ["<email channel id>"] }, { "name": "Everything else", "match": {}, "channels": ["<jira channel id>"] } ]}GET /api/v1/notification-policy returns the list. PUT replaces it.
| Route field | Type | Default | What it does |
|---|---|---|---|
name |
string, ≤ 120 | Shown in the routes editor. | |
match |
matchers | {} |
Which alerts the route takes. An empty match takes every alert: use it last, as a catch-all. |
channels |
array of channel ids, up to 20 | [] |
Where the alert goes. Send to in the editor. An empty list records the alert and tells nobody. |
continue |
boolean | false |
Keep matching the routes below. Keep matching routes below this one in the editor. |
Up to 100 routes. Each notice walks the list top to bottom. The first route that matches decides, unless it says continue: then matching goes on and the channels add up. An alert that matches no route is recorded, but nobody is notified.
Firing, firing again and escalated notices go to the routes’ channels plus every channel the alert already reached, so an escalation can add PagerDuty to an alert that went only to Jira. Acknowledged and resolved notices go only to the channels the alert already reached, even if the routes changed since.
Matchers
Section titled “Matchers”Routes and silences use the same matchers. Every matcher you set must hold. Within one matcher, any listed value will do.
| Matcher | Type | Matches alerts |
|---|---|---|
severity |
array of severities | with one of these severities. |
source |
array of strings | from one of these sources: check (monitors), job, runner (runner groups), sreinsights (alert storms), kibana, dynatrace, alertmanager. |
monitorIds |
array of monitor ids, up to 500 | of one of these monitors. |
clientIds |
array of client ids, up to 100 | of one of these clients, by id, so renaming a client changes nothing. |
labels |
object | with every listed label, set to the given value or one of a list of values (up to 50): { "team": "payments", "env": ["prod", "staging"] }. |
Useful labels: monitor on every rule alert, connection on every alert from Kibana, Dynatrace or Alertmanager, and whatever labels your rules, monitors and sources add.
Silences
Section titled “Silences”A silence holds back notifications for alerts that match it, for a while. Alerts are still raised and recorded while silenced; only telling people waits. If an alert is still firing when its silence ends, it is notified then. See Silences.
POST /api/v1/silencesContent-Type: application/json
{ "matchers": { "labels": { "cluster": "prod-eu" }, "severity": ["medium", "low", "info"] }, "durationMinutes": 120, "comment": "Postgres 17 upgrade on the payments cluster"}| Field | Type | Required | What it does |
|---|---|---|---|
matchers |
matchers | Yes | The same matchers as routes. At least one: a silence can’t match every alert. |
startsAt |
ISO 8601 time with offset | No | Defaults to now. A future time makes an upcoming silence. |
endsAt |
ISO 8601 time with offset | One of the two | When it ends. |
durationMinutes |
integer, 1–43200 | One of the two | How long it lasts from startsAt. |
comment |
string, 1–1000 | Yes | Why: shown to everyone who wonders why they weren’t paged. Why in the form. |
Set exactly one of endsAt and durationMinutes. A silence lasts at most 30 days. One that ends before it starts, or is already over, returns 400 INVALID_SILENCE.
GET /api/v1/silences?state=currentlists silences.stateiscurrent(active or upcoming, the default),active,upcoming,expiredorall.POST /api/v1/silences/:id/expireends a silence now (End now on the Silences page).
On the Silences page, New silence offers Labels, Severity and Source, a duration under For (30 minutes to 7 days; the API allows up to 30 days), and Why.