Alerts
An alert is one problem: a rule that fired for a monitor, a job that failed, a finding, a problem reported by Dynatrace, or an alert received from Alertmanager or Kibana. It lives from the moment it fires until it resolves, and everything that happens to it is on its timeline.
Where alerts come from
Section titled “Where alerts come from”| Source | Raised by |
|---|---|
check |
A monitor’s alert rules, or a failed run when it has none |
job |
A job whose last run failed, or which has had no new data for its freshness |
job_finding |
A finding a job reported |
| Incoming | Alertmanager, Kibana or Dynatrace, through their connections’ webhooks |
| SREInsights itself | Runner group offline, Alert storm |
States
Section titled “States”| State | Meaning |
|---|---|
| Firing | The problem is happening. |
| Acknowledged | Someone is on it. It stays acknowledged until it resolves. |
| Resolved | The problem stopped, or someone resolved it by hand. |
- Reopening. An alert that fires again soon after it resolved (within its rule’s reopen window, 30 minutes by default) reopens instead of starting over. It keeps its history, and a Jira issue or PagerDuty incident it made stays the same.
- Escalating. A severity raised while firing notifies again. See Severity tiers.
The Alerts page
Section titled “The Alerts page”Alerts lists Open, Resolved or All alerts, filtered by severity and searched by title and labels (for example cluster=prod-eu). Each row shows severity, title, state, source, duration and when it was last seen. From the list or the alert’s page:
- Acknowledge tells everyone someone is on it. The acknowledgement goes to every channel the alert already reached.
- Resolve closes it by hand.
- Silence alerts like this opens a silence prefilled with the alert’s labels.
The alert’s page shows:
- Timeline: every change, and who or what made it.
- Notifications: what each channel was sent, every attempt, and whether it went through.
- Details: why it fired, its value, its client, and links to the monitor or job.
- Labels: the labels routes and silences match on. They come from the monitor’s alert labels, the rule and the source. A job’s findings, for example, carry
job,job_id,job_type,detectorandfinding.
What notifies
Section titled “What notifies”Each of these changes sends a notice through the routes: firing, reopened, escalated, acknowledged and resolved.
Notices for one alert reach a channel strictly in order, so a resolve never overtakes its firing. Acknowledgements and resolves go to every channel the alert already reached. Silences, flapping and alert storms hold some notices back; the alert itself is always recorded.