Skip to content

Flapping and alert storms

SREInsights handles two kinds of noise for you. Both hold back notices only: every alert is still raised and recorded.

An alert flaps when it keeps changing between firing and resolved, for example a check that fails every other run.

  • After 4 changes between firing and resolved in 30 minutes, the alert is marked flapping.
  • While it flaps, its reopen and resolve notices are held back. PagerDuty keeps one incident open instead of opening and closing it each time.
  • After 30 quiet minutes it settles. If it ended resolved, the held resolve goes out then.

To stop an alert flapping at its source, give a threshold rule a Resolve only past line, raise Clean checks before resolving, or count over more runs (“3 of the last 5”).

An alert storm is a burst of new alerts, usually one cause seen by many monitors: a network outage, a DNS failure.

  • When more than 100 new alerts arrive for one organization within 5 minutes, individual new-alert notices are held back, and one critical Alert storm alert is raised instead.
  • It resolves once the rate drops below half the threshold.

The threshold can be changed for an instance with SREINSIGHTS_ALERT_STORM_THRESHOLD, or for one organization with alertStormThreshold in its settings.