Escalation Policies as Code: On-Call Rotations in Git

Cross-posted from the Fluidify blog.

Key takeaways

• Treating on-call schedules and escalation policies as version-controlled config instead of editing them only in a vendor UI gets you code review, a diff for every change, an audit trail, and the ability to rebuild the entire on-call setup from a git checkout.

• The building blocks exist today (Terraform providers for the major platforms, or a sync script against a REST API), but almost nothing lets you simulate a policy change against real traffic before it's live, which is the actual risk this pattern is trying to manage.

Why on-call config ends up as tribal knowledge

Ask most teams who's allowed to change an escalation policy, and the honest answer is "whoever has admin access and remembers to." The policy that decides who gets woken up at 3 a.m., and how fast, typically lives entirely inside a vendor's web UI: click into a policy, adjust a timeout, save. No review, no diff, no record of why the timeout went from five minutes to ten, and no way to see what the policy looked like six months ago short of asking whoever made the change, if they still remember.

This is the same problem infrastructure had before Terraform: configuration that only exists as the current state of a system, changeable by anyone with access, with history that lives in people's memory instead of a repository. The fix there was treating infrastructure as code. The same fix applies to on-call config, and for the same reason: escalation policies and schedules are operational logic, not administrative settings, and operational logic that can silently change without review is a real risk during an actual incident, and a quiet contributor to alert fatigue months later when nobody remembers why a timeout is set the way it is.

The pattern

The shape is the same as any other infrastructure-as-code setup: a directory of declarative files, a tool that reconciles them against the live system, and a CI pipeline that runs that tool on merge.

oncall/
  schedules/
    payments-primary.yaml
    payments-secondary.yaml
    infra-oncall.yaml
  escalation-policies/
    payments-outage.yaml
    infra-warning.yaml
  integrations/
    prometheus-payments.yaml
    cloudwatch-infra.yaml

A schedule file might look like:

# schedules/payments-primary.yaml
name: payments-primary
timezone: America/New_York
rotation:
  type: weekly
  handoff: monday 09:00
participants:
  - alice@company.com
  - bob@company.com
  - carla@company.com
overrides:
  - date: 2026-12-24
    user: dave@company.com
    reason: holiday coverage swap

And an escalation policy:

# escalation-policies/payments-outage.yaml
name: payments-outage
severity: critical
steps:
  - after: 0m
    notify: schedule:payments-primary
  - after: 5m
    notify: schedule:payments-secondary
  - after: 15m
    notify: user:eng-manager@company.com
    channel: phone

Nothing here is exotic. It's the same declarative-config idea applied to a domain that mostly hasn't gotten it yet.

What you actually gain

Review before it's live: A change to a critical escalation policy goes through a pull request like any other production change. A reviewer can catch "this removes the secondary on-call from the payments policy" before it ships, not after an incident reveals it.

A real audit trail: git blame on an escalation policy answers "who changed this and why" in one command, with the linked PR and its discussion attached, instead of a support ticket to the vendor asking for change history.

Rollback: Revert the commit, re-apply, done. No hunting through a UI trying to remember what the settings used to be.

Disaster recovery for the on-call config itself: If the on-call platform loses data or an account gets locked out, the entire schedule and escalation setup can be rebuilt from a git checkout, instead of reconstructed from memory during an outage, which is a genuinely bad time to be reconstructing your escalation policies.

What's still missing

The gap in this pattern today isn't tooling to declare the config, it's tooling to validate it before it's live. Terraform can plan a diff and show you what will change, but it can't tell you that a proposed escalation policy would have caused an on-call engineer to get skipped during last month's actual incident. Policy simulation, replaying a set of historical alerts against a proposed policy change and showing who would have been paged and when, is close to nonexistent across the ecosystem right now.

The other open problem is drift. If someone edits a schedule directly in the vendor UI during an incident (which will happen, and should be allowed to happen, because the incident doesn't wait for a PR to merge), the git state and the live state disagree until someone reconciles them. Few setups have automated drift detection for on-call config the way they do for infrastructure; most rely on someone noticing during the next terraform plan equivalent.

Existing building blocks

You don't have to build this from scratch. Terraform has official or community providers for several on-call and incident platforms, letting you declare schedules and escalation policies as Terraform resources and manage them through a normal plan/apply workflow. Where a provider doesn't exist or doesn't cover something you need, the fallback is a sync script: read the YAML, diff it against the platform's REST or GraphQL API, apply the difference, run it in CI on merge to main. It's less polished than a proper provider but gets you the review-and-audit-trail benefits, which are most of the point.

If you're evaluating on-call tools with this pattern in mind, the thing to check isn't whether a Terraform provider already exists for it; those get built or abandoned constantly. Check whether the platform exposes full API coverage over schedules, escalation policies, and integrations, and whether you can inspect the underlying data model directly if you're self-hosting it. An API-complete, self-hostable platform is one you can always build the sync layer for yourself, provider or not. That's part of why we designed FluidifyAI Regen as open-source and API-first, with the source and API reference on GitHub: the config-as-code layer on top is something a team can build, even before a dedicated integration exists.

If you haven't settled on a platform yet, our rundown of open-source on-call tools is worth a read before you pick one to build this pattern on top of.

FAQ

Do we need a custom sync script if a Terraform provider already exists? No, use the provider. It's more mature, handles state tracking, and you get terraform plan for free. Reach for a sync script only when no provider covers what you need.

What about changes made during an actual incident? Let them happen directly in the tool. Reconcile afterward: either update the git source to match what was done live, or revert the live change back to the declared state once the incident is over, whichever is correct for that specific change.

Is this worth it for a five-person team? The audit trail and rollback value scale with team size and policy complexity, but even a small team benefits from "why does this escalation policy skip Tuesdays" having an answer in git history instead of living in one person's head.

Where this is heading

Policy simulation is the missing piece that would make this pattern complete: replay real alert history against a proposed change and see the difference before merging, the same way a terraform plan shows infrastructure diff before apply. Nobody has shipped this well yet across the ecosystem. It's the natural next step for whichever platform builds it first.