Flapping and alert storms
SREInsights handles two kinds of noise for you. Both hold back notices only: every alert is still raised and recorded.
Flapping
Section titled “Flapping”An alert flaps when it keeps changing between firing and resolved, for example a check that fails every other run.
- After 4 changes between firing and resolved in 30 minutes, the alert is marked flapping.
- While it flaps, its reopen and resolve notices are held back. PagerDuty keeps one incident open instead of opening and closing it each time.
- After 30 quiet minutes it settles. If it ended resolved, the held resolve goes out then.
To stop an alert flapping at its source, give a threshold rule a Resolve only past line, raise Clean checks before resolving, or count over more runs (“3 of the last 5”).
Alert storms
Section titled “Alert storms”An alert storm is a burst of new alerts, usually one cause seen by many monitors: a network outage, a DNS failure.
- When more than 100 new alerts arrive for one organization within 5 minutes, individual new-alert notices are held back, and one critical Alert storm alert is raised instead.
- It resolves once the rate drops below half the threshold.
The threshold can be changed for an instance with SREINSIGHTS_ALERT_STORM_THRESHOLD, or for one organization with alertStormThreshold in its settings.