Try to reproduce an incident that involves a language model and the whole ritual breaks in about ninety seconds. Same prompt, same endpoint, same user context, different answer. No stack trace. No 500. Nothing in the error logs because nothing errored — the system did exactly what it was built to do, and the output was still wrong. The engineer's question is the real one: what do I put in the ticket? An InfoQ Live panel on production operations in the age of AI, held on 1 October with Michael Hausenblas, Sujana Sooreddy and Noam Levi, circles that problem from several directions. Hausenblas questions whether software was ever fully deterministic; Levi describes an era of pragmatism, with small steps and fast feedback because model updates can change behaviour; Sooreddy argues for machine-readable runbooks tagged with autonomy levels.

The assumption nobody wrote down

Most operational practice rests on determinism. Runbooks assume a fault has a reproducible trigger. Regression suites assume a correct answer exists and stays correct. Root cause analysis assumes a cause. Change control assumes that if you don't change the code, the behaviour holds. Monitoring thresholds assume a signal distribution that only moves when something is wrong.

Put a model in the request path — or worse, somebody else's model behind an API you don't version-pin — and every one of those assumptions degrades. The vendor retrains. The system prompt gets tuned by a product manager through a console you don't audit. A retrieval index picks up a document that quietly poisons a class of answers. None of that shows up as a deployment. Your change management record is clean and your behaviour has drifted.

That's the part most operations teams haven't internalised. This isn't an observability tooling gap. It's a category change in what "working" means.

Observability was already a cost problem

Before we add a new telemetry class, be honest about the one we have. Most organisations already ingest far more than they query. Log Analytics charges per gigabyte ingested, and at pay-as-you-go rates verbose IIS logs, firewall accept traffic and debug-level application output nobody turned off after go-live add up quickly. Check the current rate for your region and commitment tier, then check how much of that data any human or alert rule has ever read.

Now add prompt and completion logging. Full request and response capture on a chatty internal assistant is not small, and unlike a web log it's largely incompressible text you may be contractually or legally obliged to retain — or prohibited from retaining, depending on what users typed into it. It is easy for telemetry cost on a pilot to rival the inference cost. Decide your sampling strategy and your redaction strategy before the pilot, not after the invoice.

What actionable actually means at 2am

Turning operational data into actionable insights is the kind of phrase that survives a conference abstract and dies on contact with a Sev 1. The useful version is narrower: at 2am, the on-call engineer needs to answer three questions quickly — is it us or upstream, is it everyone or a subset, and is there a safe lever to pull.

For AI-fronted systems that means instrumenting things traditional APM doesn't give you. Token counts and latency per call, yes, but also which model version and which system prompt revision served the request, which retrieval documents were pulled in, whether a guardrail or content filter fired, and whether the user accepted or abandoned the output. OpenTelemetry's semantic conventions for generative AI are the right place to standardise this rather than inventing per-team custom fields (the conventions are still evolving) — I'd rather have imperfect agreed attribute names than six teams each calling the same thing something different.

The safe lever matters most. If your only remediation for a misbehaving model is "raise a ticket with the vendor," you have no incident response. Pin model versions. Keep the previous system prompt deployable. Build a feature flag that falls back to the deterministic path — the old keyword search, the old form, the human queue. That fallback is the single highest-value piece of engineering in an AI deployment and it is almost always the thing that gets cut for time.

Evaluations are the new regression suite

Here is where delivery practice has to change rather than just tooling. A deterministic system gets a pass/fail test. A probabilistic one gets an evaluation set — a corpus of representative inputs with graded expected behaviour, run on every change to model, prompt, retrieval index or chunking strategy, and reported as a distribution rather than a binary.

That lands awkwardly in most pipelines. It's slower, it costs money per run, and it produces a score that somebody has to set a threshold on. But the alternative is shipping prompt changes with no gate at all, which is what most teams are doing right now. If you wouldn't let a developer push a schema change without a test, don't let a product owner push a system prompt change without an eval.

What I'd change first

  • Version-pin everything. Model, prompt, index, embedding model. Record all four as attributes on every trace.
  • Build the deterministic fallback before the launch, and test it quarterly like a DR failover.
  • Define the sampling and redaction policy for prompt logging before the first production call, with Legal and your DPO in the room.
  • Add an eval gate to the delivery pipeline, even a crude one. Twenty graded cases beats zero.
  • Rewrite the RCA template so "cause" can legitimately be "input distribution shifted" without the reviewer sending it back.

The limit of the argument

None of this is a reason to avoid putting models into production. It's a reason to stop pretending the existing operational model stretches to cover them. The failure mode I expect over the next two years isn't a dramatic AI outage — it's a long tail of quietly wrong outputs that nobody noticed because the dashboards were all green and the error rate was zero. Availability monitoring cannot detect a confidently incorrect answer.

Panels like this one are useful mainly as permission to say the uncomfortable thing out loud to your own leadership: the deterministic guarantees you've been selling the business for a decade no longer apply to part of the estate, and the operational investment has to change to match. That conversation is cheaper now than after the first incident where the post-mortem has no cause in it.

Source: Beyond Observability: Evolving Production Operations in the Age of AI