Skip to content

Alerts

An alert is one problem: a rule that fired for a monitor, a job that failed, a finding, a problem reported by Dynatrace, or an alert received from Alertmanager or Kibana. It lives from the moment it fires until it resolves, and everything that happens to it is on its timeline.

Source Raised by
check A monitor’s alert rules, or a failed run when it has none
job A job whose last run failed, or which has had no new data for its freshness
job_finding A finding a job reported
Incoming Alertmanager, Kibana or Dynatrace, through their connections’ webhooks
SREInsights itself Runner group offline, Alert storm
State Meaning
Firing The problem is happening.
Acknowledged Someone is on it. It stays acknowledged until it resolves.
Resolved The problem stopped, or someone resolved it by hand.
  • Reopening. An alert that fires again soon after it resolved (within its rule’s reopen window, 30 minutes by default) reopens instead of starting over. It keeps its history, and a Jira issue or PagerDuty incident it made stays the same.
  • Escalating. A severity raised while firing notifies again. See Severity tiers.

Alerts lists Open, Resolved or All alerts, filtered by severity and searched by title and labels (for example cluster=prod-eu). Each row shows severity, title, state, source, duration and when it was last seen. From the list or the alert’s page:

  • Acknowledge tells everyone someone is on it. The acknowledgement goes to every channel the alert already reached.
  • Resolve closes it by hand.
  • Silence alerts like this opens a silence prefilled with the alert’s labels.

The alert’s page shows:

  • Timeline: every change, and who or what made it.
  • Notifications: what each channel was sent, every attempt, and whether it went through.
  • Details: why it fired, its value, its client, and links to the monitor or job.
  • Labels: the labels routes and silences match on. They come from the monitor’s alert labels, the rule and the source. A job’s findings, for example, carry job, job_id, job_type, detector and finding.

Each of these changes sends a notice through the routes: firing, reopened, escalated, acknowledged and resolved.

Notices for one alert reach a channel strictly in order, so a resolve never overtakes its firing. Acknowledgements and resolves go to every channel the alert already reached. Silences, flapping and alert storms hold some notices back; the alert itself is always recorded.