Note: This article reflects my personal opinion. It is in no way a recommendation or endorsement of any company or solution.

Microsoft’s AI Gateway: The Security and Infrastructure Layer Between Applications and AI

Generative AI applications often begin with a deceptively simple architecture: an application sends an HTTPS request directly to a model endpoint and receives a response.

That works well enough for a proof of concept. It becomes problematic when several applications, teams and agents begin sharing the same models.

Who is allowed to use which model? How much capacity may each application consume? Which identity made a request? Can an application bypass centrally defined safety controls? Where are prompts and responses logged? What happens when a model deployment reaches its quota or becomes unavailable? How do security teams govern MCP tools that an agent can invoke?

Microsoft’s answer is the AI Gateway in Azure API Management.

Microsoft AI Gateway is not a separate Azure service or appliance. It is a collection of AI-specific capabilities in Azure API Management, or APIM. Microsoft Foundry can create or associate an APIM instance and expose governance controls through the Foundry portal.

The gateway sits between applications and the AI services they consume:

flowchart LR
    A["Applications and agents"] --> B["APIM AI Gateway"]
    B --> C["Foundry models"]
    B --> D["MCP and A2A services"]
    B --> E["Other model providers"]

Applications no longer need direct access to model deployments or credentials. The gateway becomes the enforcement point for identity, authorization, consumption, routing, inspection and audit logging. But APIM is not a complete AI security solution, and poor placement can create a bottleneck or single point of failure.

This article examines the AI Gateway from a security and infrastructure perspective: where it belongs, which controls should be enabled, which threats it does not solve, and when another product may be a better fit.

This article reflects product capabilities and documentation available on 20 July 2026. Some Foundry and APIM AI capabilities, including the unified model API and parts of the Foundry integration, remain in preview.

What the AI Gateway actually is

Azure API Management has traditionally been used to publish, secure and observe conventional APIs. It provides a managed gateway data plane, a management plane, a developer portal, API products and a policy engine that can inspect and transform requests and responses.

The AI Gateway extends this existing platform with policies and interfaces that understand AI-specific concepts such as:

• Prompts and completions
• Input and output tokens
• Tokens per minute
• Model deployments
• Semantic caching
• Content-safety categories
• Streaming model responses
• Model backend pools
• MCP servers
• Agent-to-agent APIs

Microsoft is explicit that the AI Gateway is an extension of APIM, not a separate offering. It can currently manage OpenAI Chat Completions and Responses APIs, Anthropic Messages APIs, Google Vertex AI models, self-hosted models, remote MCP servers and A2A agent APIs. Support for Anthropic and Vertex AI reached general availability in mid-2026. Models may reside in Microsoft Foundry, other Azure resources or non-Microsoft environments such as Amazon Bedrock. Microsoft Learn: AI Gateway capabilities, Microsoft: New AI Gateway capabilities

The gateway also pairs naturally with Foundry Agent Service. Its generally available Bring Your Own Model capability lets Foundry agents route model calls through enterprise gateway infrastructure, giving agent workloads the same governance as direct traffic. Microsoft Learn: Bring your own model

There are two ways to implement the gateway.

Foundry-integrated AI Gateway

The Foundry portal can create or associate an APIM instance from the Foundry Admin console. Projects within the Foundry resource can then be added to the gateway and assigned their own token limits and quotas.

However, the native integration has important boundaries:

• One gateway integration is associated with one Foundry resource.
• Multiple projects within that resource share the underlying APIM instance.
• A reused APIM instance must currently use a v2 tier.
• APIM and Foundry must be in the same Microsoft Entra tenant and Azure subscription.
• Projects that require separate gateways must be placed in separate Foundry resources.

New projects created in the Foundry resource have the gateway enabled by default; existing projects must be added to the gateway manually.

These constraints may conflict with an enterprise landing zone in which central APIM services reside in a dedicated connectivity or shared-platform subscription. Microsoft Learn: Configure AI Gateway in Foundry

Standalone APIM AI Gateway

APIM can also be configured independently of the Foundry-native integration. Administrators import model endpoints, create backend pools and manage policies directly in APIM.

This approach is more flexible across Foundry resources, subscriptions, providers, self-hosted models and network zones. The trade-off is that the platform team owns more configuration and cannot rely solely on the simplified Foundry experience.

Why direct model access becomes a security problem

A model endpoint already supports authentication and quota management. It is reasonable to ask why another hop is necessary.

The model endpoint protects itself; the gateway enforces policy across consumers. With direct connections, each application manages credentials, retries, telemetry and safety controls. One client can exhaust shared capacity, model changes propagate into code, and security teams cannot readily prove consistent enforcement.

A gateway introduces a stable endpoint and a shared enforcement plane. It allows the organization to separate the identity presented by the application from the identity used against the model backend.

That separation is the foundation of the recommended design.

The recommended production architecture

For a private, enterprise Foundry environment, APIM should be treated as part of the AI platform—not as an application component deployed separately by every development team.

A sensible baseline architecture looks like this:

flowchart TD
    A["Applications and agents"] --> B["Private DNS"]
    B --> C["APIM private endpoint"]
    C --> D["Authentication and policy engine"]
    D --> E["Foundry private endpoint"]
    D --> F["Private MCP and tool APIs"]
    D --> G["Approved external AI APIs"]
    D --> H["Azure AI Content Safety"]
    C --> I["Azure Monitor and Log Analytics"]

The design establishes three distinct boundaries:

1. Applications authenticate to APIM.
2. APIM authenticates to Foundry and other backends.
3. Network controls prevent applications from bypassing APIM.

APIM should normally be owned by the central AI platform or API platform team. Workload teams receive access to approved APIs, model aliases and quotas rather than administrative access to the gateway.

Shared gateway or one gateway per application?

Neither extreme is usually correct.

One APIM instance per application creates cost and policy duplication; one global instance creates a large blast radius, shared change domain and potential bottleneck.

A more useful boundary is one gateway per:

• Security or regulatory zone
• Major data-residency boundary
• Environment class, such as production versus non-production
• AI platform with common ownership and availability requirements

Several applications may share a gateway, but unrelated regulatory zones should not be forced through one instance. Production and development should use separate APIs, policies, credentials and quotas; higher-assurance environments should use separate instances.

Choosing an APIM tier

The Foundry portal creates a Basic v2 instance when asked to create a new gateway. Microsoft describes this as suitable for development and testing. It should not automatically become the production design.

For production, the practical choices are generally Standard v2 and Premium v2.

Standard v2 can combine an inbound private endpoint with outbound VNet integration. This allows clients to reach APIM privately while APIM reaches a Foundry private endpoint through the connected or peered network.

Premium v2 supports full VNet injection. Its gateway obtains a private address in a delegated subnet and can isolate both inbound and outbound traffic. Microsoft recommends this option when the gateway and its backends must be fully isolated. Microsoft Learn: APIM virtual networking

There are two important availability caveats. First, multi-region deployment is currently unavailable in the v2 tiers. An architecture requiring regional failover must therefore evaluate separate APIM instances, external global routing, or a standalone design using a tier and topology that support the required resilience. Second, within the v2 family only Premium v2 supports availability zone redundancy, and it can be enabled only when the instance is created—it cannot be added or changed later. Both decisions must be made during design rather than discovered during production onboarding. Microsoft Learn: APIM v2 tiers, Microsoft Learn: Reliability in API Management

Private networking in both directions

An APIM private endpoint only protects traffic entering APIM. It does not automatically provide private connectivity from APIM to Foundry.

A complete private path requires an inbound APIM private endpoint, private DNS, outbound VNet integration or Premium v2 injection, a Foundry private endpoint, connectivity between them, blocked direct public access and explicit egress rules for Content Safety, monitoring and approved providers.

The objective is to make APIM the only route to protected model APIs. If applications retain direct network access and Foundry permissions, the gateway becomes an optional control that is difficult to defend in an audit.

Public consumers

Some AI APIs must be exposed to external customers or partners. In that case, APIM should not necessarily be the internet edge.

A common architecture is:

flowchart LR
    A["External client"] --> B["Azure Front Door Premium and WAF"]
    B -->|Private Link| C["APIM AI Gateway"]
    C -->|Private access| D["Microsoft Foundry"]

Azure Front Door or Application Gateway supplies web application firewall capabilities and external edge protection. APIM provides API authentication, authorization, AI policies and backend mediation.

APIM is not a WAF. Azure Firewall is not an AI Gateway. AI Content Safety is neither of those. Each protects a different layer.

Identity, authentication and authorization

AI Gateway security is easiest to understand when identity is divided into separate planes.

flowchart LR
    A["Application identity"] -->|OAuth token| B["APIM"]
    B -->|Managed identity token| C["Foundry"]
    B -->|OAuth or managed identity| D["Tool API"]
    E["Platform administrators"] -->|Azure RBAC| B

Management-plane authorization

Azure RBAC controls who may configure APIM and Foundry.

The Foundry-native setup currently requires roles such as:

• Contributor or Owner to create APIM
• API Management Service Contributor or Owner to reuse APIM
• Foundry Owner or another appropriate Foundry administrative role

Note that the Foundry RBAC roles were recently renamed: Foundry Owner and Foundry Account Owner were previously called Azure AI Owner and Azure AI Account Owner, and the old names may still appear in some places while the rename rolls out.

Grant these roles through privileged groups and just-in-time elevation. Policy changes can alter authentication, logging, routing and data inspection, so they should pass through infrastructure-as-code, peer review and controlled deployment. Application developers should not receive service-level administration merely because they consume an API.

Client-to-gateway authentication

Applications should authenticate to APIM using Microsoft Entra workload identities whenever possible.

APIM can validate Entra tokens with the validate-azure-ad-token policy. Validation should include more than checking that a token came from the tenant. At minimum, verify:

• Expected tenant and issuer
• API audience
• Approved client application ID
• Required application role or delegated scope
• Relevant custom claims where used

A simplified example looks like this:

<validate-azure-ad-token
    tenant-id="{{tenant-id}}"
    header-name="Authorization"
    failed-validation-httpcode="401"
    failed-validation-error-message="Unauthorized">
    <client-application-ids>
        <application-id>{{approved-client-id}}</application-id>
    </client-application-ids>
    <audiences>
        <audience>api://{{ai-gateway-app-id}}</audience>
    </audiences>
    <required-claims>
        <claim name="roles" match="any">
            <value>AI.Model.Invoke</value>
        </claim>
    </required-claims>
</validate-azure-ad-token>

The example is deliberately restrictive. Accepting any valid tenant token would authenticate the caller without proving that it is authorized to consume this API.

APIM subscription keys can still be useful as a secondary consumer identifier for products, metering or migration scenarios. They should not normally replace Entra authentication for internal enterprise applications.

For selected partner integrations, APIM also supports other OpenID Connect providers, mutual TLS, client certificates and IP filtering.

Gateway-to-Foundry authentication

APIM should use managed identity to call Foundry.

Enable a system-assigned or user-assigned identity on APIM, assign the minimum Foundry data-plane role at the narrowest suitable scope, and have APIM obtain a token for https://cognitiveservices.azure.com/. APIM then replaces the incoming authorization header before forwarding the request.

A simplified policy is:

<authentication-managed-identity
    resource="https://cognitiveservices.azure.com"
    output-token-variable-name="foundry-token"
    ignore-error="false" />

<set-header name="Authorization" exists-action="override">
    <value>@("Bearer " + (string)context.Variables["foundry-token"])</value>
</set-header>

APIM can configure managed-identity authentication automatically when a Foundry API is imported. Microsoft Learn: Authenticate and authorize LLM APIs

This pattern removes model API keys from applications. It also means a compromised client credential grants access to the gateway policy assigned to that client—not automatically to the entire Foundry resource.

API keys remain supported for backends that do not accept Entra ID. Store them in Key Vault and reference them through protected APIM named values. Do not place provider keys directly in policy XML.

Authorization belongs at more than one layer

A valid identity is not enough.

The gateway should determine which APIs, model aliases and tools the caller may use; its rate and periodic quota; the applicable logging and safety profile; and whether it may access high-cost models.

The backend must still enforce its own authorization. APIM policy is a security layer, not permission to weaken Foundry or tool-level RBAC.

Security controls to enable

Not every available policy should be enabled blindly. The controls should respond to a defined threat or governance requirement.

Token limits and quotas

Token governance should be enabled for every shared gateway.

APIM’s llm-token-limit policy can enforce tokens per minute and a quota over an hourly, daily, weekly, monthly or yearly period. The counter can be based on a subscription, application identity, IP address, project or arbitrary policy expression.

<llm-token-limit
    counter-key="@(context.Subscription.Id)"
    tokens-per-minute="50000"
    token-quota="10000000"
    token-quota-period="Monthly"
    estimate-prompt-tokens="true"
    remaining-tokens-header-name="x-ai-tokens-remaining"
    remaining-quota-tokens-header-name="x-ai-quota-remaining" />

When the rate is exceeded, APIM returns 429 Too Many Requests. When the periodic quota is exhausted, it returns 403 Forbidden. Prompt-token estimation allows APIM to reject an oversized request before it consumes backend capacity. Note that the remaining-quota value exposed in headers or variables is an estimate and may be larger than actual consumption suggests, so consumers should not build hard logic on it. Microsoft Learn: LLM token limits

Recommended practice is to set limits at more than one logical level:

• Per application, to contain defective or compromised clients
• Per project or team, to provide fair allocation
• At the API or platform level, to protect aggregate backend capacity

Token quotas are cost controls, but they are not exact monetary budgets. Model prices differ, and reasoning, cached, image and audio tokens can have different rates. Financial showback requires model-aware cost calculations outside the basic token-limit policy.

Content Safety and Prompt Shields

APIM’s llm-content-safety policy sends prompts and, optionally, completions to Azure AI Content Safety.

It supports harm categories and severity thresholds, custom blocklists, Prompt Shields, completion inspection, MCP and A2A payloads, and streaming responses.

A representative configuration is:

<llm-content-safety
    backend-id="azure-ai-content-safety"
    shield-prompt="true"
    enforce-on-completions="true">
    <categories output-type="EightSeverityLevels">
        <category name="Hate" threshold="4" />
        <category name="SelfHarm" threshold="4" />
        <category name="Sexual" threshold="4" />
        <category name="Violence" threshold="4" />
    </categories>
    <blocklists>
        <id>company-restricted-content</id>
    </blocklists>
</llm-content-safety>

The APIM managed identity should authenticate to the Content Safety backend. If a policy violation is detected, APIM blocks the request or response and returns a 403 error. Streaming completions behave differently: once a violation is found mid-stream, APIM simply stops forwarding subsequent events without returning an error code. Clients and agents that consume streams must therefore handle an abrupt stop gracefully rather than expect an explicit 403. Microsoft Learn: LLM Content Safety policy

Enable Prompt Shields for untrusted user input or retrieved documents. Select completion inspection according to data classification, latency tolerance and the consequences of unsafe output. Filtering adds cost, latency and another dependency, so an internal summarizer, public chatbot and autonomous agent may need different profiles.

Backend routing, load balancing and circuit breakers

The gateway can create backend pools and distribute traffic using:

• Round-robin routing
• Weighted routing
• Priority-based routing
• Session-aware routing

Priority routing is especially valuable when provisioned throughput has been purchased. Traffic can use the provisioned deployment first and fall back to pay-as-you-go capacity when necessary.

Circuit breakers temporarily remove unhealthy or throttled backends. APIM can use the backend’s Retry-After response to determine when it should be retried.

These are security-adjacent controls because availability is part of security. They also prevent uncontrolled retry storms from amplifying a backend failure.

Session-aware routing matters for stateful APIs such as the Responses API. A generic stateless load-balancing configuration may break conversations that depend on backend affinity.

Input constraints and policy enforcement

APIM’s ordinary API policies remain useful for AI workloads.

They can:

• Enforce maximum request sizes
• Validate required headers
• Restrict HTTP methods and API versions
• Remove untrusted headers
• Override forbidden parameters
• Reject excessive max_tokens values
• Restrict model names to approved aliases
• Normalize client input
• Prevent callers from choosing arbitrary backend URLs
• Apply IP restrictions
• Validate JSON structure

The platform should expose stable aliases such as standard-chat, advanced-reasoning or approved-embedding, rather than the provider’s raw deployment names. Aliases decouple application contracts from specific providers and model versions.

Microsoft’s unified model API can expose several backends through one endpoint: clients use the OpenAI Chat Completions format, and APIM translates requests to backends that use either the OpenAI Chat Completions or Anthropic Messages API. It also provides a /models discovery endpoint for published aliases. The unified model API remains in preview and is rolling out gradually—in the classic tiers it is available through the AI Gateway early release channel—and it should not be treated as a universal abstraction for every provider-specific feature. Microsoft Learn: Unified model API

Semantic caching

APIM supports semantic caching through Azure Managed Redis or another compatible external cache with RediSearch.

Instead of matching the exact request, semantic caching compares prompt embeddings and may reuse a completion for a sufficiently similar prompt. This can reduce inference cost and latency.

It should not be enabled indiscriminately.

Cache keys and lookup scopes must preserve:

• Tenant boundaries
• User authorization
• Data classification
• System prompt and policy version
• Model and model version
• Tool availability
• Retrieval context
• Relevant conversation state

Semantic caching is a poor fit for current information, personalized responses, security decisions or agent tool execution. Treat cached completions as stored application data subject to encryption, retention, access control and deletion requirements.

Logging, compliance and auditability

One of the strongest reasons to introduce a gateway is the ability to produce a common audit trail.

APIM can log the caller, correlation ID, API, model, backend, response, token consumption, latency and policy outcomes. It can optionally record prompts and completions. Token metrics are provider-dependent and can include prompt, completion and total counts. Additional categories such as cached, reasoning and thinking tokens remain marked as preview in the current policy reference. Microsoft Learn: Emit token metrics

AI gateway logs can be sent to Azure Monitor and Log Analytics. APIM provides the ApiManagementGatewayLlmLog table and an AI analytics dashboard. Prompts and responses can be correlated by CorrelationId, including messages split across several log records. Microsoft Learn: Log LLM requests and responses

This supports investigations, usage attribution, showback, evaluation, capacity planning, abuse detection and evidence that policy was consistently applied.

However, full prompt logging should not be the default.

Prompts can contain personal data, health information, credentials, source code, contractual information and data retrieved from internal systems. Completions may reproduce the same material. Logging both can turn Log Analytics into a new repository of highly sensitive data.

A defensible logging strategy separates operational telemetry from content:

Where content logging is justified, define its lawful purpose, access roles, retention, regional storage, redaction, incident-response access, deletion handling and whether it may be reused for evaluation.

An audit trail is valuable only if its integrity and interpretation are clear. APIM can prove that a request passed through a policy, but it cannot by itself prove that the model response was factually correct or that an agent’s resulting business action was appropriate.

What the AI Gateway does not solve

APIM is a powerful control point, but placing it in front of a model does not make the AI system secure.

Complete prompt-injection prevention

Prompt Shields can detect attack patterns, but prompt injection is not solely a text-filtering problem.

An agent can encounter malicious instructions in:

• Retrieved documents
• Websites
• Emails
• Tool output
• MCP resources
• Images
• Code repositories
• Previous conversation state

A secure agent must treat external content as untrusted data, separate instructions from evidence and validate actions before execution. High-impact actions may require deterministic policy checks or human approval.

Use Prompt Shields as one layer, not as proof that indirect prompt injection has been eliminated.

Data-loss prevention

Content Safety is designed primarily around harmful content and user attack patterns. It is not a complete enterprise DLP system.

Organizations that need to detect customer identifiers, secrets, source code, regulated data or intellectual property should consider:

• Microsoft Purview data classification and DLP capabilities
• Application-level redaction and tokenization
• Secret scanning
• Prisma AIRS runtime security
• Another specialized AI DLP or guardrail service
• Strict minimization before content reaches the model

The safest sensitive field is often the one never added to the prompt.

Tool authorization

APIM can authenticate and govern calls to MCP and A2A services, but it does not understand the business legitimacy of every action.

A model deciding to invoke delete_user, approve_payment or send_email still requires:

• A narrowly scoped tool identity
• Operation-level authorization
• Input validation
• Business policy
• Transaction limits
• Idempotency
• Human confirmation where appropriate
• An audit record at the target system

The tool remains responsible for authorization. A gateway cannot safely convert an overprivileged backend into a least-privileged one.

Model and output correctness

The gateway does not prevent hallucinations, bias, poor retrieval, model drift, unsafe business logic or excessive autonomy. Those require evaluation, grounding, testing, business rules and human oversight.

Endpoint and workload security

APIM does not replace WAF and DDoS controls, network egress enforcement, workload protection, vulnerability management, Key Vault, Defender services, secure development or SIEM-based detection and response. The AI Gateway is one component in a layered architecture.

Resilience, performance and operational risk

A gateway removes complexity from applications but concentrates it in the platform.

This creates a shared failure domain. A malformed policy, failed certificate renewal, exhausted APIM capacity or networking error can affect every connected AI workload.

Production design should include realistic load tests, scaling thresholds, backend health monitoring, staged policy deployments, rollback, synthetic requests, certificate and DNS monitoring, failure-mode analysis and runbooks for safety-service or provider failure.

Most security controls should fail closed, but decide per dependency. A low-risk summarizer may continue without optional evaluation; a public agent should normally stop if mandatory DLP is unavailable. Document the choice rather than inherit it accidentally from an APIM error path.

APIM adds latency. JWT validation is usually minor beside inference, but Content Safety, DLP, cache lookups and transformations can become visible. Test streaming because the gateway must handle long-lived connections without premature timeouts or buffering.

Cost and economic justification

There is no separate Microsoft AI Gateway license. The primary platform cost is APIM.

At public list prices in July 2026, the approximate monthly base prices are:

Regional pricing, enterprise agreements, currency and taxes change the actual amount. Basic v2 is currently free for up to 100,000 requests when created as an AI Gateway through Microsoft Foundry. Production architects should still verify eligibility and the billing transition in the target subscription. Azure API Management pricing

The total architecture also includes model inference, Content Safety, caching, telemetry, private networking, edge security, extra gateway instances and operational ownership.

A gateway can still be economically positive: limits contain runaway applications, caching reduces repeated inference, priority routing improves PTU utilization, and central policies reduce duplicate engineering. One small application may not need a gateway; ten teams sharing capacity and compliance requirements probably do.

Competitors and complementary products

Distinguish model brokers from AI-traffic inspection products; they solve different problems.

Prisma AIRS: the security-led gateway competitor

Prisma AIRS AI Gateway became generally available on 16 July 2026 after Palo Alto Networks acquired Portkey. It combines gateway controls with DLP, prompt defence, agent identity, budgets, model scanning and red teaming. APIM remains stronger in Azure identity, networking, observability and general API management.

APIM can remain the boundary while Prisma AIRS supplies inspection, adding latency, licensing and another prompt-data processor. Palo Alto documents Foundry-integration limitations for latency, multimodal content and tool calls. Palo Alto Networks: Prisma AIRS Foundry integration

Check Point: complementary enforcement rather than gateway replacement

Check Point’s closest product is Check Point WAF with built-in GenAI Security, incorporating Lakera technology. It detects prompt attacks, leakage, harmful content and misuse, alongside WAF, API-schema, bot and DDoS controls. It supports SaaS, reverse-proxy and agent deployments. Check Point WAF, API deployment guide

It does not replace APIM authentication, managed identity, API products, routing, token quotas or Foundry integration:

flowchart TD
    A["Applications and users"] --> B["Check Point WAF: attack and leakage prevention"]
    B --> C["APIM AI Gateway: identity, quota and routing"]
    C --> D["Foundry and approved models"]

Workforce AI Security governs employee AI use; AI Cloud Protect secures BlueField-based private infrastructure. Neither brokers models. Workforce AI Security, AI Cloud Protect

Start with APIM and Content Safety. Add Prisma AIRS or Check Point when DLP, prompt defence or vendor standardization justifies another inline processor.

A recommended security baseline

A production APIM AI Gateway should, at minimum, implement the following baseline.

Infrastructure

• Separate production and non-production and choose Standard v2 or Premium v2 according to isolation requirements.
• Use private inbound and outbound paths and prevent applications from reaching Foundry directly.
• Control external egress, monitor private DNS and define regional and zonal resilience explicitly at design time.

Identity

• Require Entra workload identity and validate audience, client, roles and scopes.
• Use APIM managed identity toward Azure backends and store unavoidable secrets in Key Vault.
• Separate gateway administration from consumption and protect changes with privileged access.

AI policies

• Apply per-application, project and aggregate token limits and expose only approved model aliases.
• Bound requests, enable Prompt Shields for untrusted input and configure content categories by use case.
• Use backend pools and circuit breakers, and test streaming, stateful APIs and safety-terminated streams.

Monitoring and compliance

• Log identity, model, tokens, latency, policy result and correlation ID; avoid full prompt logging by default.
• Define retention and access before collecting content and export security events to the SIEM.
• Alert on abnormal consumption, safety blocks and failover, and manage policies through APIOps and source control.

Controls outside APIM

• Use edge, DDoS, workload and DLP controls appropriate to the exposure and data.
• Authorize every tool and require confirmation for high-impact actions.
• Minimize input data, evaluate models and agents, and red-team high-risk systems.

Conclusion

Microsoft’s AI Gateway is best understood as a control point, not an AI firewall and not a model platform.

Its principal value is that it moves common responsibility away from individual applications. Identity validation, model authentication, token allocation, routing, safety policy and audit telemetry can be implemented once and applied consistently across many consumers.

From a security and infrastructure perspective, the most important design decision is not which APIM policy to enable. It is whether the organization is willing to make the gateway authoritative.

If applications can bypass it, retain model keys or reach Foundry directly, the gateway provides convenience but not a reliable security boundary. If network access, identity and backend permissions force traffic through APIM, it becomes a meaningful enforcement point with strong compliance and audit value.

The recommended Azure pattern is therefore:

1. Authenticate applications to APIM using Entra ID.
2. Authorize them using explicit application roles and policy.
3. Apply per-consumer token limits and approved model aliases.
4. Inspect untrusted prompts with Azure AI Content Safety and Prompt Shields.
5. Authenticate APIM to Foundry using managed identity.
6. Keep both gateway and model traffic private.
7. Record operational and security events without indiscriminately storing sensitive prompts.
8. Add DLP, agent authorization, evaluation and runtime AI security outside APIM where the risk requires it.

APIM combines AI-aware policy with mature Azure identity, networking and API management, turning scattered model endpoints into a governed platform service.

The gateway does not make AI safe by itself. It provides one defensible place to enforce the controls that make safer AI possible.