Many organisations have the same undocumented asset: a set of model API keys on somebody's corporate card, spread across three or four teams, with no idea of the aggregate spend and no idea what data is going out through them. Nobody planned it. A developer needed a model in a sprint, procurement was going to take six weeks, and a personal key cost nothing to create. Multiply that by every squad that read the same blog post and you have a real egress problem hiding inside a rounding error on the expenses.

So when Microsoft puts a dedicated AI Gateway tier of Azure API Management into public preview — a control plane whose primary objects are models, MCP servers and tools rather than APIs — my first reaction is not scepticism. It is that somebody finally built the thing the shape of the problem demanded. The second reaction is the important one: a gateway governs the traffic that goes through it, and nothing else.

The consolidation is the genuinely useful part

Fronting Foundry, Bedrock, Vertex AI and OpenAI behind a single endpoint is not an architectural flourish. It is the difference between one set of managed identities, one audit log and one place to enforce a token budget, versus four sets of credentials living in four different key vaults with four different billing relationships. If you have ever tried to reconcile model spend across providers at month end, you know that the reconciliation problem alone justifies a chokepoint.

APIM has had genAI-flavoured policies for a while — token rate limiting, token metric emission, semantic caching, backend load balancing with circuit breakers. Those were bolted onto a product whose mental model was HTTP operations and XML policy documents. Making models and tools first-class objects in the control plane is the overdue correction. What I want to see in the preview is whether the token accounting is actually accurate against provider billing, because approximate token counts are worse than none: they give finance a number to argue with rather than a number to plan on. Per InfoQ's report, the new tier is a separate experience rather than a policy layer on the existing gateway, and the classic and v2 tiers keep their current AI gateway capabilities.

MCP tools are not APIs, and that is the whole problem

Here is where I would slow down. An API call in the classic APIM sense is something a developer wrote deliberately, into code you can review, with parameters you can validate against a schema you control. An MCP tool invocation is something a model decided to do, at runtime, based on text it was given — and some of that text came from outside your organisation.

That changes the governance question from who is allowed to call this endpoint to under what circumstances is a non-deterministic agent allowed to take this action on behalf of this user. Those are not the same control. Rate limits and subscription keys answer the first. Nothing in a traditional gateway answers the second. If a tool can write to a ticketing system, send mail, or read a SharePoint site, then indirect prompt injection turns your beautifully governed gateway into a very well-audited confused deputy. The log will tell you exactly which policy allowed the damage.

So the questions I would put to a Microsoft account team, in order: does the tool call carry the end user's identity through to the downstream system, or does it run as the gateway's own identity? Can I scope tool availability per caller rather than per server? Can I require human confirmation on a defined subset of tools? If the answer to the identity question is "the gateway's managed identity", you have consolidated your permissions into a single over-privileged principal and called it governance.

Policy cards versus XML, and what that costs you

Replacing XML policy documents with policy cards will be popular, and I understand why. The APIM policy language is powerful and genuinely unpleasant — inline C# expressions inside XML, evaluated in an order you have to hold in your head, with a debugging story that has improved but never delighted.

The thing XML had going for it was that it was a file. Files go into Git, get reviewed in a pull request, deploy through Bicep, and produce a diff you can hand to an auditor when they ask who changed the rate limit in March. Before I move a production estate onto a card-based model, I want to know that the cards export to something declarative and importable, and that the export is the source of truth rather than a screenshot of one. A portal-only configuration surface is a regression in change control no matter how good it looks, and preview is exactly when to ask, because the answer is easier to change now than at GA.

The portability illusion

One endpoint across four providers implies you can swap models freely. The request shape may be portable. Behaviour is not. Prompts tuned against one model family degrade against another in ways that unit tests do not catch, tool-calling reliability varies significantly between providers, and content filters differ enough to change your compliance posture without changing a line of your code. Treat the abstraction as an operational convenience — failover, cost routing, credential consolidation — and not as a licence to skip evaluation when you change backends.

Where the boundary actually sits

The gateway covers code your teams write and point at it. It does not cover Copilot, models embedded in SaaS products you already licence, browser extensions, agent frameworks that speak to MCP servers directly over stdio on a developer laptop, or anything running locally. That is not a criticism of the product; it is the definition of a gateway. But if the internal narrative becomes "we have an AI gateway, therefore AI is governed", you have bought a false sense of coverage at tier pricing.

What I would do while it is in preview

  • Run the discovery first: firewall and proxy logs for calls to known model endpoints, plus expenses for card-funded subscriptions. You cannot route what you have not found.
  • Put one non-critical internal workload behind it and measure added latency, especially on streamed responses.
  • Test the identity flow on a tool that writes something, not just one that reads.
  • Block direct model egress at the network layer only after the gateway path demonstrably works — the reverse order creates shadow workarounds.
  • Keep production off it until GA. Public preview typically means no SLA, and a single chokepoint without an SLA is a single point of failure with better logging.

The direction is right. Models and tools deserve a control plane of their own rather than a set of policies pretending they are REST endpoints. Just do not let the existence of the tier substitute for the harder work of deciding what your agents are actually permitted to do.

Source: Azure API Management Adds Dedicated AI Gateway Tier, Governing Models and MCP Tools