Key takeaways

• Treating on-call schedules and escalation policies as version-controlled config instead of editing them only in a vendor UI gets you code review, a diff for every change, an audit trail, and the ability to rebuild the entire on-call setup from a git checkout.

• The building blocks exist today (Terraform providers for the major platforms, or a sync script against a REST API), but almost nothing lets you simulate a policy change against real traffic before it's live, which is the actual risk this pattern is trying to manage.

Why on-call config ends up as tribal knowledge

Ask most teams who's allowed to change an escalation policy, and the honest answer is "whoever has admin access and remembers to." The policy that decides who gets woken up at 3 a.m., and how fast, typically lives entirely inside a vendor's web UI: click into a policy, adjust a timeout, save. No review, no diff, no record of why the timeout went from five minutes to ten, and no way to see what the policy looked like six months ago short of asking whoever made the change, if they still remember.

This is the same problem infrastructure had before Terraform: configuration that only exists as the current state of a system, changeable by anyone with access, with history that lives in people's memory instead of a repository. The fix there was treating infrastructure as code. The same fix applies to on-call config, and for the same reason: escalation policies and schedules are operational logic, not administrative settings, and operational logic that can silently change without review is a real risk during an actual incident, and a quiet contributor to alert fatigue months later when nobody remembers why a timeout is set the way it is.

The pattern

The shape is the same as any other infrastructure-as-code setup: a directory of declarative files, a tool that reconciles them against the live system, and a CI pipeline that runs that tool on merge.

oncall/
  schedules/
    payments-primary.yaml
    payments-secondary.yaml
    infra-oncall.yaml
  escalation-policies/
    payments-outage.yaml
    infra-warning.yaml
  integrations/
    prometheus-payments.yaml
    cloudwatch-infra.yaml

A schedule file might look like:

# schedules/payments-primary.yaml
name: payments-primary
timezone: America/New_York
rotation:
  type: weekly
  handoff: monday 09:00
participants:
  - alice@company.com
  - bob@company.com
  - carla@company.com
overrides:
  - date: 2026-12-24
    user: dave@company.com
    reason: holiday coverage swap

And an escalation policy:

# escalation-policies/payments-outage.yaml
name: payments-outage
severity: critical
steps:
  - after: 0m
    notify: schedule:payments-primary
  - after: 5m
    notify: schedule:payments-secondary
  - after: 15m
    notify: user:eng-manager@company.com
    channel: phone

Nothing here is exotic. It's the same declarative-config idea applied to a domain that mostly hasn't gotten it yet.

What you actually gain

Review before it's live: A change to a critical escalation policy goes through a pull request like any other production change. A reviewer can catch "this removes the secondary on-call from the payments policy" before it ships, not after an incident reveals it.

A real audit trail: git blame on an escalation policy answers "who changed this and why" in one command, with the linked PR and its discussion attached, instead of a support ticket to the vendor asking for change history.

Rollback: Revert the commit, re-apply, done. No hunting through a UI trying to remember what the settings used to be.

Disaster recovery for the on-call config itself: If the on-call platform loses data or an account gets locked out, the entire schedule and escalation setup can be rebuilt from a git checkout, instead of reconstructed from memory during an outage, which is a genuinely bad time to be reconstructing your escalation policies.

What's still missing

The gap in this pattern today isn't tooling to declare the config, it's tooling to validate it before it's live. Terraform can plan a diff and show you what will change, but it can't tell you that a proposed escalation policy would have caused an on-call engineer to get skipped during last month's actual incident. Policy simulation, replaying a set of historical alerts against a proposed policy change and showing who would have been paged and when, is close to nonexistent across the ecosystem right now.

The other open problem is drift. If someone edits a schedule directly in the vendor UI during an incident (which will happen, and should be allowed to happen, because the incident doesn't wait for a PR to merge), the git state and the live state disagree until someone reconciles them. Few setups have automated drift detection for on-call config the way they do for infrastructure; most rely on someone noticing during the next terraform plan equivalent.

Existing building blocks

You don't have to build this from scratch. Terraform has official or community providers for several on-call and incident platforms, letting you declare schedules and escalation policies as Terraform resources and manage them through a normal plan/apply workflow. Where a provider doesn't exist or doesn't cover something you need, the fallback is a sync script: read the YAML, diff it against the platform's REST or GraphQL API, apply the difference, run it in CI on merge to main. It's less polished than a proper provider but gets you the review-and-audit-trail benefits, which are most of the point.

If you're evaluating on-call tools with this pattern in mind, the thing to check isn't whether a Terraform provider already exists for it; those get built or abandoned constantly. Check whether the platform exposes full API coverage over schedules, escalation policies, and integrations, and whether you can inspect the underlying data model directly if you're self-hosting it. An API-complete, self-hostable platform is one you can always build the sync layer for yourself, provider or not. That's part of why we designed FluidifyAI Regen as open-source and API-first, with the source and API reference on GitHub: the config-as-code layer on top is something a team can build, even before a dedicated integration exists.

If you haven't settled on a platform yet, our rundown of open-source on-call tools is worth a read before you pick one to build this pattern on top of.

FAQ

Do we need a custom sync script if a Terraform provider already exists? No, use the provider. It's more mature, handles state tracking, and you get terraform plan for free. Reach for a sync script only when no provider covers what you need.

What about changes made during an actual incident? Let them happen directly in the tool. Reconcile afterward: either update the git source to match what was done live, or revert the live change back to the declared state once the incident is over, whichever is correct for that specific change.

Is this worth it for a five-person team? The audit trail and rollback value scale with team size and policy complexity, but even a small team benefits from "why does this escalation policy skip Tuesdays" having an answer in git history instead of living in one person's head.

Where this is heading

Policy simulation is the missing piece that would make this pattern complete: replay real alert history against a proposed change and see the difference before merging, the same way a terraform plan shows infrastructure diff before apply. Nobody has shipped this well yet across the ecosystem. It's the natural next step for whichever platform builds it first.

Every few years, a team gets tired of drowning in pages and migrates to a new on-call tool. The rollout goes fine. Schedules import cleanly, integrations reconnect, everyone gets a shiny new mobile app. Then, somewhere between four and twelve weeks later, the same on-call engineer who pushed for the migration is back to muting notifications at 2 a.m. and triaging the same noisy alert they were triaging on the old platform.

This isn't a failure of the new tool. It's what happens when a team changes where the alerts land without changing what causes them to fire in the first place. The rules that produced fatigue on the old system, which alerts get created, how they escalate, who they wake up, come across in the migration as faithfully as the schedules do. A new UI on top of the same rules produces the same outcome with a different logo.

What actually causes alert fatigue

Three root causes show up in almost every team that reports fatigue, regardless of which tool they're on:

Noisy sources with no classification layer. Most alerting pipelines treat every signal from Prometheus, CloudWatch, or a health check as equally worth a human's attention. A flapping check that recovers in ninety seconds pages the same way a full outage does. Without deduplication and severity classification at the point alerts are generated, the on-call engineer becomes the classification layer, by hand, every single time.

Escalation policies that don't differentiate urgency. A five-minute escalation timeout makes sense for a payment outage. It does not make sense for a disk-usage warning that can wait until morning. When every alert type shares one escalation policy, either the urgent stuff waits too long or the routine stuff interrupts sleep, and teams routinely get this backwards in both directions on the same policy.

No owner for alert hygiene. Someone has to look at the alerts that fired last week, decide which ones were useful, and either fix, downgrade, or delete the ones that weren't. In most teams, nobody owns this. It falls to whoever happens to be on-call when they get frustrated enough, which means the noisy alert survives until someone has a bad week and finally kills it out of spite. That's not a process, and it doesn't scale.

Why a new tool doesn't fix any of the three

None of those three causes live in the vendor. Classification, escalation policy design, and hygiene ownership are decisions a team makes and re-makes, and a tool migration is exactly the moment those decisions get carried over unexamined, because reproducing the existing setup quickly is the whole point of a smooth migration. The team optimizes for "nothing broke" and, in doing so, optimizes for "nothing changed," including the parts that were broken.

The tools also look similar enough on this specific axis that switching between them rarely forces the issue. Every mainstream on-call platform supports severity levels, deduplication rules, and tiered escalation. The gap isn't feature availability. It's that configuring those features well takes an audit most teams skip during a migration.

If a migration is already on the table, it's worth doing the audit below before comparing vendors, not after.

What actually works

The interventions that measurably reduce fatigue touch the three root causes directly, independent of vendor:

Classify and deduplicate at the source, not at the human. Group alerts from the same root cause into one notification. Suppress known-flapping checks instead of paging on every transition. This is configuration work on the monitoring side (Alertmanager routing rules, Grafana notification policies) as much as the on-call tool.

Match escalation speed to actual urgency, per alert type, not per team. A payment-path outage and a disk-warning should never share an escalation policy. Once that logic is right, managing it as version-controlled config instead of a vendor UI setting is what keeps it from drifting back.

Name an owner for alert hygiene, and give them a recurring slot. Someone reviews the last week or month of pages, tags what was noise, and has the authority to retune or delete rules without three approvals.

Close the loop after every incident. If a postmortem doesn't produce a concrete change to an alert rule or escalation policy when the alert was part of the problem, the same failure mode reappears with the label "known issue" attached to it.

A 30-day exercise that costs nothing

Before any team migrates on-call tools to solve fatigue, it's worth running this against the current one:

1. Pull every page from the last 30 days, grouped by alert rule.

2. For each rule, tag it: actionable, informational (nice to know, didn't need a wake-up), or noise (should never have paged).

3. For anything tagged noise, delete the rule, add deduplication, or move it off the paging path entirely.

4. For anything tagged informational, move it to a lower urgency tier with a slower escalation.

5. For actionable pages, check that the escalation timeout matches how much time the team actually has before impact grows.

Most teams find that a third to half of their pages fall into noise or informational once they actually look, on the tool they already have. If fatigue is still there after that exercise, a new tool might genuinely help.

FAQ

Does a better on-call tool ever fix alert fatigue? It can remove friction around fixing it, but the tool doesn't do the classification or ownership work for you.

How do we know if our escalation policies are the problem? Check how many pages resolved themselves or were ignored until auto-resolve, versus how many required real action within the escalation window. A high self-resolving ratio usually means the timeout is too aggressive.

Who should own alert hygiene without a dedicated SRE team? Rotate it with the on-call schedule: whoever was on-call last week reviews last week's pages before handing off.