AIOps in Practice: How to Reduce Alert Fatigue, From Alertmanager to Machine Learning

A pager that goes off dozens of times a night does not make a team more careful. It teaches people to acknowledge first and read later, until the alert that mattered gets the same treatment as all the rest.

AIOps is sold as the cure: put machine learning on the alert stream and let it separate signal from noise. It can help, but a model trained on alerts with missing labels, no owners and no record of what was acted on learns your noise very accurately. The order that works: measure the flood, use what Alertmanager already does, fix labels and dependencies, then put machine learning where a static rule cannot do the job.

This post is about the flood you already have. For which alerts to create in the first place, see our guide to Kubernetes alerting that stays actionable.

Measure the Flood Before You Buy Anything

You cannot reduce noise you have not counted, and the ranking is often lopsided: a handful of rules produce most of the pages.

Which alerts fire most

Prometheus writes an ALERTS series for every alert its rules produce, labeled with alertname, alertstate (pending or firing) and the alert's own labels. Two queries over it rank your rules:

# Time spent firing: rule evaluations in which each alert was firing,
# summed over all its label sets (pods, instances, ...)
topk(20, sum by (alertname) (
  count_over_time(ALERTS{alertstate="firing"}[7d])
))

# Breadth: how many distinct label sets each alert fired for this week
topk(20, count by (alertname) (
  count_over_time(ALERTS{alertstate="firing"}[7d])
))

The first query counts samples, not firings: multiply by your rule evaluation interval to get firing time. An alert that stays on for days without anyone fixing it is either unneeded or ignored; both call for a decision. The second surfaces alerts that fire separately for every pod or instance in a fleet, candidates for aggregation or grouping.

To find flapping rules, count how often each alert starts firing:

# Roughly how many times each alert went from not firing to firing in the last day
topk(20, sum by (alertname) (
  count_over_time(
    (ALERTS{alertstate="firing"} unless ALERTS{alertstate="firing"} offset 1m)[1d:1m]
  )
))

A rule that starts many times a day and stays on for minutes each time needs a longer for: duration, or keep_firing_for in recent Prometheus versions, not a human.

Two limits: ALERTS covers only rules Prometheus evaluates, and it ignores silences and inhibition, so it shows what fired, not what reached a person. A 7-day range also needs a week of retention.

Which alerts get acknowledged and ignored

Alertmanager has no concept of acknowledgment. That data lives in your paging tool, so export the last 30 days of pages and, for each alertname, record how often it paged, how often someone acted on it (a fix, a rollback, a ticket, a runbook step), and how often it was acknowledged and closed with no action or resolved before anyone looked.

If the export cannot show whether action was taken, have on-call tag each page in a weekly review as actioned, acknowledged only or duplicate. Also list long-lived silences with amtool silence query: an alert silenced for months is one nobody trusts.

Each rule then gets a verdict: delete, demote to a ticket or dashboard, fix (threshold, duration, labels), or keep as a page.

What Alertmanager Already Does Before Any Machine Learning

Much of the correlation that AIOps platforms advertise starts with four mechanisms Alertmanager has had for years:

• Grouping. group_by decides which alerts share one notification. group_wait delays the first notification of a new group so related alerts, and any alert that should inhibit them, arrive together: temporal correlation in its simplest form. group_interval paces notifications about alerts added to an existing group, and repeat_interval sets when still-firing alerts are sent again.
• Routing. A tree of label-matched routes sends critical alerts to the pager and the rest to team queues. The first matching route wins unless it sets continue: true.
• Inhibition. While a source alert fires, matching target alerts are muted if the labels listed in equal match on both.
• Silences. Time-boxed mutes for planned work.

A realistic starting point that passes amtool check-config:

route:
  receiver: team-tickets
  group_by: ['env', 'system']
  group_wait: 30s
  group_interval: 5m
  repeat_interval: 4h
  routes:
    - receiver: oncall-pager
      matchers:
        - severity="critical"
      group_wait: 10s
      repeat_interval: 1h
    - receiver: payments-queue
      matchers:
        - team="payments"
    - receiver: dashboard-only
      matchers:
        - severity="info"

inhibit_rules:
  # A node that is down explains every other alert on that node
  - source_matchers:
      - alertname="KubeNodeNotReady"
    target_matchers:
      - alertname!="KubeNodeNotReady"
    equal: ['cluster', 'node']
  # A database that is down explains the errors of the apps that use it
  - source_matchers:
      - alertname="PostgresDown"
      - system!=""
    target_matchers:
      - tier="app"
    equal: ['env', 'system']

receivers:
  - name: oncall-pager
    webhook_configs:
      - url: 'http://pager-bridge.monitoring.svc:8080/alerts'
  - name: payments-queue
    webhook_configs:
      - url: 'http://ticket-bridge.monitoring.svc:8080/payments'
  - name: team-tickets
    webhook_configs:
      - url: 'http://ticket-bridge.monitoring.svc:8080/default'
  - name: dashboard-only

system and tier are your own labels (see below), and PostgresDown stands for your database alert. A receiver with no configuration drops what reaches it, which suits informational alerts.

The node rule hides a trap. Pod alerts built on kube-state-metrics carry namespace and pod but no node label, and their instance label points at kube-state-metrics itself. Until you add node, that inhibition rule silently does nothing. Join it in the pod rule's expression:

(
  max by (cluster, namespace, pod) (
    increase(kube_pod_container_status_restarts_total[15m])
  ) > 3
)
* on (cluster, namespace, pod) group_left(node)
  max by (cluster, namespace, pod, node) (kube_pod_info)

For planned work, have the pipeline create a narrow silence with an expiry, rather than someone muting a whole channel:

amtool silence add env=prod system=orders \
  --duration=2h \
  --comment="Planned PostgreSQL failover, change 4812" \
  --alertmanager.url=http://alertmanager.monitoring.svc:9093

Want the inventory without building it yourself? Our free alert-noise assessment runs these queries against your Prometheus, reviews your routing and inhibition rules, and hands back a ranked list of rules to delete, demote, group or fix, before you spend anything on tooling.

Correlation Starts With Labels and a Dependency Map

Grouping and inhibition work only on labels, as does every correlation engine, with or without machine learning. Agree on a set that every alert must carry:

• service: what is broken, named the way the owning team names it
• team: who gets notified; an alert without one has no owner
• env and cluster (or region): where it is happening
• severity: critical, warning or info, each with one agreed meaning

Put cluster and env in Prometheus' global.external_labels, which Prometheus attaches to every alert it sends to Alertmanager. Put team and severity in each rule's labels: block, and fail CI on any rule that misses one: promtool check rules covers the syntax, a short script covers the labels. Making this consistent across clusters is most of the work in an observability cleanup.

Then write down the dependencies, which topological correlation needs. A file in Git listing what each service calls and runs on is enough to start, and from it you derive a system label, required like the others, that ties a database to the applications that use it.

A worked example. Say orders-db, a PostgreSQL primary, fails over. Within a couple of minutes you could receive the database alert, two replica-lag alerts, three alerts (error rate, latency, failing readiness probe) from each of the six services that use it, four queue-backlog alerts, two SLO burn-rate alerts and one failed synthetic checkout check. That is 28 notifications for one cause.

With the map turned into labels, the same event produces one page. Everything in the orders system carries system="orders", so the route groups it into a single notification, and the database inhibition rule mutes the application alerts while PostgresDown fires. Give your SLO alerts their own tier, such as tier="slo", so that rule never mutes them: they tell you customers are affected, which the database alert cannot.

Where Machine Learning Helps, and Where It Does Not

Once alerts are labeled, owned and grouped, a smaller set of problems remains where a learned model beats a fixed rule.

Where it helps

Metrics with no good static threshold. Order volume or queue depth can be healthy at 03:00 and alarming at the same level at 11:00. Try a seasonal comparison before reaching for a model (orders_total stands for any business counter you export):

sum(rate(orders_total[30m]))
  < 0.5 * sum(rate(orders_total[30m] offset 1w))

This fires when the last half hour runs at less than half the rate of the same half hour a week earlier. It breaks on public holidays and after launches, which is where a model that learns several seasonal patterns earns its place. The same applies to the alert stream: a service that normally raises a couple of alerts a day and suddenly raises fifty is telling you something, even if each one is only a warning.

Correlating alerts with changes. The first question on any incident call is what changed. Have your pipeline record deploys, config pushes, feature flag flips and certificate rotations as events with the same service and env labels as your alerts, and show recent changes to the affected service and its dependencies on every incident. A model can then learn which changes tend to precede which alerts and rank the suspects.

Recurring patterns. The same alerts every Sunday at 02:00 while the backup runs. Clustering past incidents by alert fingerprint lets the system say "this looks like last month's incident, and this is what fixed it". That needs history with consistent labels and recorded outcomes, which the measurement step starts to build.

Where it does not

• It cannot correlate alerts that share no labels. If service is spelled three ways, the model sees three services.
• It cannot route an alert nobody owns, or learn from feedback nobody gives.
• It cannot fix a bad alert. A CPU-above-80 rule that never mattered gets learned as noise and suppressed, as your engineers already do by hand. Delete it.
• It should not be a black box. If on-call cannot see why alerts were grouped or suppressed, they stop trusting it and read every alert again.

Symptom-Based SLO Alerts Are the Long-Term Fix

Everything above reduces noise from cause-based alerts. The durable fix is to page on fewer of them: page when users are affected, measured against a service level objective, and send causes such as CPU, memory and pod restarts to tickets and dashboards.

Multi-window burn-rate alerts make this practical. With a 99.9% availability objective over 30 days, your error budget is 0.1% of requests. A burn rate of 1 spends that budget in exactly 30 days, or 720 hours. Burning at 14.4 times that rate for one hour spends 14.4 / 720 = 2% of the month's budget, which is worth waking someone for:

(
  sum(rate(http_requests_total{service="checkout", code=~"5.."}[1h]))
  /
  sum(rate(http_requests_total{service="checkout"}[1h]))
) > (14.4 * 0.001)
and
(
  sum(rate(http_requests_total{service="checkout", code=~"5.."}[5m]))
  /
  sum(rate(http_requests_total{service="checkout"}[5m]))
) > (14.4 * 0.001)

The one-hour window proves the burn is sustained; the five-minute window lets the alert clear soon after recovery. A slower pair, such as a burn rate of 6 over six hours (6 × 6 / 720 = 5% of the budget), catches slow leaks and can open a ticket instead of paging. Adjust the metric names to what your services export, and set objectives with the people who own the service, not for them; that conversation is the core of our SRE consulting work.

Automate Runbooks, With Guardrails

Some alerts have one known, safe response. Automating it removes a page; automating the wrong thing causes an outage at machine speed.

Safe to automate: idempotent, reversible actions on one target with a well-understood cause, such as restarting a worker with a known, ticketed memory leak, scaling out within a preset maximum, rotating logs on a filling volume, renewing a certificate or cordoning a node that reports hardware errors.

Keep these human: anything that deletes data, database promotion or failover, schema rollbacks, changes to firewall rules or access policies, actions that span more than one failure domain, and any case where the diagnosis is ambiguous.

The guardrails matter more than the scripts:

• Dry run first. For the first weeks the automation only posts what it would have done, to compare with what the engineer did.
• Rate limits. Cap runs per target, say three an hour, then stop and page. An auto-fix that runs every day is hiding a bug, so each run also opens a ticket.
• Audit and kill switch. Every action is logged against its alert, and one flag disables all automation.
• Labels as a gate. Automation acts only on alerts that carry service, team and a runbook_url annotation.

Wire it as an Alertmanager webhook receiver: Alertmanager posts JSON with each alert's labels and annotations, and a small service maps known alertname values to approved actions.

A 90-Day Plan

Start with your noisiest service or team, prove it there, then roll it out.

Month 1 (days 1-30): measure and clean

• An alert inventory: every rule with owner, severity, runbook link, firing time and action rate
• The label standard written down and enforced in CI
• Unowned or never-actioned rules deleted or demoted, with the owning team's agreement
• Flapping rules fixed with for: durations and keep_firing_for

Month 2 (days 31-60): group, inhibit, route

• A routing tree by team and severity, grouped by system
• Inhibition rules for your three biggest cascades, such as node to pods and database to applications
• Silences created by the pipeline, not by hand
• The dependency map in Git for the services covered
• Burn-rate alerts for two or three customer-facing journeys, with the matching cause-based alerts demoted
• The month-one queries rerun against the baseline, plus pages per on-call shift

Month 3 (days 61-90): add intelligence where it pays

• Seasonal baselines on the metrics that have no usable static threshold
• Change events from the pipeline attached to every incident
• One to three automated runbooks, in dry-run mode before they act
• A written buy, build or skip decision on a machine learning correlation layer, based on what is still noisy and on the labeled history you now have

Where to Start

Grouping, inhibition, labels and SLO alerts are not glamorous, but they work with the tools you already run and decide whether any model you add later has anything useful to learn from. Our NOC operations team triages alerts around the clock and builds this setup with the engineers who own the services.

The free alert-noise assessment reviews your last 30 days of alerts and pages, your Alertmanager routing and inhibition rules and your label consistency across clusters, and returns the month-one inventory plus the three cascades worth fixing first.