Most runbooks contain at least one step that reads, in effect, check with the DBA. Not a command, not a condition — a person. That single line is the reason two decades of automation investment has left incident recovery roughly where it was in 2010: faster to type, still gated on somebody's memory of how the estate is wired together.

So when Stripe's engineering team describes modelling their global database infrastructure as a graph, then using graph search alongside state machines to — in InfoQ's summary — "compute and execute remediation plans automatically", the algorithm is not what I find interesting. Graph traversal is a solved undergraduate problem. What's interesting is that they got their topology into a form precise enough for a search algorithm to trust. That's the hard part, and it's the part nobody puts on a conference slide.

Topology as data, not as tribal knowledge

A remediation plan is a path. You are at a bad state, you want a good state, and there are a finite number of legal moves between them: promote a replica, drain a shard, fail over to the secondary region, reroute writes, rebuild from a snapshot. Humans do this with a mental model built from scar tissue. The mental model works right up until the person holding it is on annual leave, or the incident touches a corner of the estate they've never seen.

Turning that into a graph forces a specific discipline: every node must be an addressable thing with a known state, and every edge must be a move with known preconditions and known effects. You cannot fudge it. The moment your model contains a node whose real state you can only determine by asking someone, the search returns nonsense — and worse, it returns nonsense confidently.

This is where most organisations fall over, and it has nothing to do with graph theory. It's that our record of the estate is a story we tell ourselves.

Your CMDB is fiction, and everyone knows it

Configuration management databases are rarely accurate on relationships. Servers, yes — you can discover those. Dependencies, no. The SQL instance that a finance reporting app quietly points at. The listener that was moved during a maintenance window in 2022 and never documented. The witness that lives on a host nobody has patched because it was labelled non-production.

Automated remediation is unforgiving of this. A human reading a stale runbook notices something is off and stops. A state machine executing a computed plan does not notice; it does the next legal move on a graph that no longer matches reality. The Stripe approach only works because the graph is derived from live infrastructure rather than curated by hand, and I'd bet a good chunk of their effort went into that derivation rather than the search.

If you want the operational lesson in one sentence: the constraint on automating recovery is the fidelity of your inventory, not the sophistication of your orchestration tool.

Why state machines beat scripts, at any scale

This part transfers immediately, even if you run four SQL Server availability groups rather than a global sharded estate. Most remediation automation I encounter is written as a script: a linear sequence of steps with a bit of error handling. Scripts have a fatal property in incident conditions — they don't know where they are. Kill one halfway through a failover and you're left with a partially promoted cluster and no record of intent.

A state machine records intent and position separately from execution. It can be resumed, it can be inspected mid-flight, and it can be made idempotent step by step. When an on-call engineer takes over at 03:00, the question changes from "what did the script get through?" to "what state are we in and what's the next transition?" That's a materially better question to answer under pressure.

The same pattern retrofits onto ordinary Microsoft-stack work: Entra Connect staging-server cutover, Exchange database failover across DAG members, and file-service migrations where the rollback path matters more than the forward path. Nothing exotic — a durable record of steps, each one safe to re-run, each one with an explicit precondition check. The gain isn't speed. It's that a second engineer can pick up a half-finished operation without a handover call.

The obvious failure mode: automation as an incident amplifier

Automated remediation that computes its own plan is a machine with a strong opinion and no fear. The classic outage shape is not "automation failed to act"; it's "automation acted, correctly, on a bad reading, at scale, faster than anyone could intervene". If your graph mislabels a healthy primary as unreachable because a monitoring path broke, a plan that promotes the replica is entirely rational and entirely catastrophic.

Anything of this sort needs the same guardrails you'd put on any privileged automation:

  • Blast-radius caps — a hard limit on how many nodes a single plan may touch before it requires approval.
  • Dry-run by default — the plan is computed and logged even when it isn't executed, so you can review its judgement against real incidents before you trust it.
  • Independent health signals — never let the same telemetry path decide both "is it broken" and "did the fix work".
  • A stop button that works — one that halts the state machine cleanly at a known state rather than killing a process mid-transition.
  • Full audit of transitions — because your auditor will ask who authorised a production failover, and "the graph did" is not an answer.

What's actually worth copying

Very few of us need a global remediation planner. Almost all of us would benefit from three cheaper things drawn from the same thinking.

First, write down the legal moves. Not the runbook prose — the discrete state transitions your estate supports, with preconditions. This exercise alone surfaces the steps that only exist in one person's head.

Second, derive dependencies rather than documenting them. Query the config, read the connection strings, walk the listeners. A dependency map generated weekly and 85% right beats a hand-maintained diagram that was 100% right eighteen months ago.

Third, convert your two or three highest-frequency recovery procedures from scripts into resumable state machines with explicit checks. Leave the plan computation to humans for now. You'll get most of the resilience benefit and none of the amplification risk.

The pattern here is a familiar one: the clever algorithm is available to everyone, and the boring data quality work is what actually separates the organisations that can use it from the ones that just read about it.

Source: Stripe Uses Graph Search and State Machines to Automate Database Remediation