Picture the usual shape of a major incident: hours of archaeology establishing what changed, then minutes of remediation once someone finds it. Four people on a bridge call, three of them reading dashboards aloud, one scrolling a change calendar that turns out to be a week out of date; then a firewall rule nobody logged turns out to be the cause and the fix takes four minutes. That ratio of archaeology to remediation is the actual shape of most major incidents, and it's why an AI assistant that reads the telemetry for you is genuinely interesting rather than just another demo.

What Instacart has actually built

Blueberry, per the InfoQ write-up, is an incident response assistant that sits in Slack and helps on-call engineers investigate production issues. It runs agents in parallel against operational data and past incidents, pulls context through MCP integrations, and produces root cause hypotheses — grounded ones, meaning tied back to observable evidence rather than confidently improvised. The engineer stays in the driving seat. It suggests; it does not act.

That last design choice is the one I'd defend hardest if I were on their architecture review board. An agent that proposes "the p99 latency spike correlates with a deploy to service X at 14:02, and a similar pattern appears in incident 4471" is doing exactly what a good senior engineer does in the first ten minutes, only faster and without the tunnel vision. An agent that restarts service X on its own initiative is an availability risk wearing a helpful face.

Per InfoQ's report, Instacart's CTO Anirban Kundu says an alert launches about ten subagents in parallel, and a grounded hypothesis lands in the Slack thread, typically within about three minutes. Instacart reports diagnostic accuracy rising from the mid-60% range to the high-90% range once the agents were grounded in more than 14 years of incident history, which is the clearest evidence in the piece for the argument below.

The prerequisite nobody puts in the architecture diagram

Here's the part that should give most IT organisations pause. This system is grounded in historical incident knowledge. That phrase does a colossal amount of work. It assumes that when something broke eighteen months ago, somebody wrote down what broke, why, what the signal looked like, and what fixed it — in a structured, searchable, consistently formatted way.

Go and look at your own incident record. Genuinely, go and look. In most environments the honest state of it is some combination of: a ticket closed with the resolution field reading "resolved", a Teams thread that's been auto-purged by retention policy, a post-incident review document that exists for the three incidents severe enough to have reached an executive, and a great deal of institutional memory living in the head of one engineer who is now doing the same job at a competitor.

You cannot ground an agent in that. Retrieval over a corpus of "restarted the service, monitoring" produces an assistant that suggests restarting the service. The model isn't the constraint here. The corpus is. Instacart didn't build an AI capability so much as they built a data asset and then put a language interface on it — and the data asset is the part that took the discipline.

MCP is the quietly important detail

The Model Context Protocol integrations get a passing mention in the summary and deserve more attention than that. MCP is how the agent reaches your monitoring, your deployment pipeline, your logs, your ticketing. Which means MCP is also your new, poorly-understood, highly privileged integration surface.

Think about what an incident-investigation agent needs to be useful: broad read access across observability, change management, source control and configuration. That's close to a full-fidelity view of your estate. If that's brokered through a service principal with standing permissions and a token sitting in an environment variable, you have built an extremely convenient reconnaissance tool for anyone who compromises the bot. I'd want that identity in Entra with Conditional Access, scoped to read-only, subject to Privileged Identity Management if it touches anything sensitive, and logged the same way a human account would be. The agent's actions should be attributable and reviewable. Nobody is going to ask you for this in the first sprint. Ask for it anyway.

Parallel subagents and the cost of being confidently wrong

Running subagents in parallel — one chasing deploys, one chasing infrastructure metrics, one chasing similar past incidents — is a sensible pattern. It mirrors how you'd split a bridge call between three people. It also multiplies the number of plausible-sounding narratives arriving at the incident commander at once.

During an outage, cognitive load is the enemy. A wrong hypothesis delivered with fluency is worse than no hypothesis, because someone will chase it, and chasing it costs twenty minutes you don't have. The mitigation is that every hypothesis must arrive with its evidence attached and its confidence stated plainly. If the assistant can't show you the metric, the log line or the change record it built the theory on, it should stay quiet. "Grounded" has to mean citation, not vibes.

What this means if you're not Instacart

Most of the people reading this run an estate that's some mix of on-prem, Azure, a few SaaS platforms and one legacy system nobody will name in writing. You are not building Blueberry. But the underlying move — turn operational history into a queryable asset, then let a model reason over it — is available to you at a much smaller scale, and the prep work is identical.

  • Make post-incident write-ups mandatory and structured: symptom, signal, root cause, fix, detection gap. Five fields. Enforce them at ticket closure.
  • Get change records accurate and timestamped. Correlation with deploys is the single highest-value signal any of these systems use, and it's worthless if your CAB record lags reality by a week.
  • Decide now how a non-human identity gets read access to your telemetry, and put it under the same identity governance as a person.
  • Start with retrieval over your own incident history before you go anywhere near autonomous action. The boring version delivers most of the value.

The genuinely useful lesson here isn't about AI at all. It's that Instacart could build this because their operational data was good enough to build on. That's a decade of unglamorous discipline showing up as a capability. If your first reaction to this news is "we should get one of those", the honest second reaction should be "we should first be able to answer what changed in the last hour". Fix that, and the assistant becomes worth having. Skip it, and you've bought an expensive way to generate confident nonsense at three in the morning.

Source: Instacart Builds Blueberry, an AI-Powered Assistant to Help On-Call Engineers Investigate Incidents