AI Gateway in an AI Workspace: What It Controls

An AI gateway is a layer that applications call instead of calling AI models directly, so that one place decides which models are reached, with which credentials, at what volume and with what record. Insulin’s AI Gateway does part of that job for one path, the AI calls Insulin itself makes: which providers may serve your organization, in what order, and whether only your own keys may be used.

---

An IT or platform evaluator with “AI gateway” on a requirements list will find a tab of the same name in Insulin: Settings → Organization → AI Gateway, formerly AI Model Policy. So the question follows quickly: does it replace an AI gateway like Azure API Management?

No, and it isn’t built to. The honest answer has two halves. Insulin’s AI Gateway is a provider policy for one workspace: it governs which providers and whose keys serve Insulin’s own agents, chat and other AI features, and Insulin fails over and says so when a provider breaks. It isn’t a proxy that your other applications call, and several jobs on a gateway’s usual list aren’t described in Fours’ documentation at all. This post sets Microsoft’s list beside Insulin’s documented controls and grades each row Covered, Partly or Not described.

What is an AI gateway?

An AI gateway is a proxy that sits between applications and the AI models they call, so that one layer can authenticate, route, limit, cache, filter and log that traffic. Microsoft describes the AI gateway in Azure API Management as “a set of capabilities that help you manage your AI backends effectively”, used to “secure, scale, monitor, and govern AI models, agents, and tools”. Its reasons to use one are to:

• “Authenticate and authorize access to AI services”
• “Load balance across multiple AI endpoints”
• “Monitor and log AI interactions”
• “Manage token usage and quotas across multiple applications”

AWS’s Guidance for Multi-Provider Generative AI Gateway has the same shape: tenants and client applications access a LiteLLM gateway proxy API through one endpoint, which tracks usage and applies governance across many model providers.

The name is also used at a narrower scope. Microsoft’s architecture guide to model gateways notes that the AI Gateway built into Microsoft Foundry “is designed to support ingress to a single Foundry resource and doesn’t span multiple resources.” Insulin’s AI Gateway is scoped to one product as well.

What does Insulin’s AI Gateway control?

Insulin’s AI Gateway is the organization setting that decides which AI providers may serve your organization, in what order, and whether Fours’ own keys may be used at all. It sits under Settings → Organization → AI Gateway, only an org ADMIN can change it, and the AI Gateway reference describes the tab in full. It holds two controls:

• Allowed AI integrations, an ordered allow-list. AI features may use only the integrations on it, and the top row, badged Default, is the organization default for any feature without a model picker of its own. An empty list means no restriction.
• Allow Fours platform key, a switch that is on by default. Turned off, every AI call must use one of your own connected keys, and an allowed provider with no connected key fails closed instead of falling back to Fours.

Beside it, every agent fails over on its own: if a model call fails, the turn retries on your next connected provider, and the reply names the model that took over. On your own keys, Fours’ metering documentation says the platform “still routes and secures every request and meters the token usage the provider reports.”

One limit of scope matters to a gateway review: each row admits a provider, not a model, which restricting AI models, not just their providers works through. The rows themselves, the Default badge, the Fours Hosted (DeepInfra) entry and the checks before turning Fours’ key off are covered in how Insulin’s AI model allowlist decides which vendors see your prompts, so this post doesn’t repeat them.

How does Insulin’s AI Gateway compare with an AI gateway?

Two jobs on Microsoft’s list are covered, three are partly covered, and the rest, including caching, content safety and routing through your own gateway, aren’t described in Fours’ documentation. The grades mean:

• Covered: Fours’ documentation describes a control that does the job for the AI calls Insulin makes.
• Partly: a documented control does part of the job, and the cell says which part is missing.
• Not described: Fours’ documentation, as of October 23, 2026, describes no such control. That is a statement about the documentation, so ask Fours before you plan around it either way.

What an AI gateway does, in Microsoft’s terms	Insulin’s documented counterpart	Grade
Govern models from several providers: manage models deployed in Microsoft Foundry “or providers such as Amazon Bedrock”, and apply governance policies “once across all models”	Allowed AI integrations: AI features use only the providers on the ordered list, and the top row is the organization default. On a non-empty list, the Fours-hosted models stay available only if Fours Hosted (DeepInfra) is one of the rows	Covered
Authenticate and authorize access to AI services	Allow Fours platform key off: every AI call uses one of your own connected keys, and an allowed provider without one fails closed. Whose keys a call can reach follows ownership: organization agents use the organization’s connections; the built-in assistant and personal agents use each person’s own	Covered
Resiliency: “circuit breaker rules to stop forwarding requests to AI service backends if they’re not responsive”	Failover: a model call that fails on a provider outage, a usage limit or a rejected key retries on your next connected provider, and the reply ends with a note naming both models. Failover is per turn: the docs describe no rule that stops sending to a failing provider	Partly
Load balancing: round-robin, weighted or session-aware distribution across backends	Not described. Insulin fails over to the next provider; spreading traffic across several isn’t documented	Not described
Token rate limiting and quotas: “a TPM limit or a token quota over a specified period, such as hourly, daily, weekly, monthly, or yearly”, per API consumer	One monthly spending limit for the whole organization, in US dollars, on Settings → Billing. No per-person, per-app or per-agent token quota is described	Partly
Observability: “Track token metrics per consumer” and “Log prompts and completions”	The Usage panel: each month’s spend grouped by resource, user, app, model or provider. It’s a billing view, with no per-request drill-down, not a record of what was sent	Partly
Semantic caching: reusing completions of similar prior prompts to cut backend calls	Not described	Not described
Content safety: policies “to automatically moderate LLM prompts by using Azure AI Content Safety”	Not described	Not described
Prompt-log export: prompts and completions logged to Azure Monitor and analyzed in Application Insights	Not described. The Usage panel itself has no CSV export	Not described
Sit in front of every caller: a reverse proxy that decouples “the intelligent application” from the model backends	Routing Insulin’s calls through your own gateway or endpoint: not described	Not described

The left column keeps to what Microsoft’s pages state. Microsoft’s list also includes jobs outside this comparison, such as governing MCP servers and A2A agent APIs, and self-service developer portals.

Why are token quotas and logging only partly covered?

Because the controls beside the AI Gateway answer a billing question, who spent what, rather than a gateway’s question of how much each consumer may send and what was sent.

Token quotas. Microsoft’s token limit policy sets limits “per API consumer”, on “any counter key, such as subscription key, originating IP address, or an arbitrary key defined through a policy expression”. Insulin’s monthly spending limit is one ceiling on what the whole organization may spend in a calendar month, from 0 to $100,000. Once the month’s spend reaches it, AI requests pause for the rest of the month. It counts dollars, not tokens, and it has no per-consumer key. One other documented cap counts job runs, not tokens: runs Insulin starts on a schedule or an event are limited to 60 an hour per job and 300 per organization. On your own key, the provider bills the tokens directly, so its own limits are the per-key control; AI API spend limits when you bring your own key covers how those and Insulin’s limit fit together. A turn that hits a provider’s usage limit fails over.

Logging. Microsoft’s gateway can “Log prompts and completions to Azure Monitor” and “Track token metrics per consumer in Application Insights”. Insulin’s Usage panel covers the second half in billing terms: it re-pivots the period’s consumption by Resource, User, App, Model and Provider, and every pivot totals the same amount. It doesn’t cover the first half. The panel is re-aggregated every couple of hours, and Fours’ documentation is plain about its grain: “There is no CSV export and no per-request drill-down.” It tells you who spent how much on which model, not what anyone sent.

Can Insulin’s model calls go through your own AI gateway?

Fours’ documentation doesn’t describe it. The provider connections it documents ask for credentials. The OpenAI connection, for example, takes an API key and an optional Organization ID, and no documented connection takes a custom endpoint or base URL. So putting Insulin’s model traffic behind Azure API Management, a LiteLLM proxy or another gateway is a question to ask Fours, not a configuration step to plan on.

What you can do from the documentation is narrow which providers and keys Insulin uses, and set each provider’s own limits on the keys you connect.

When do you still need your own AI gateway?

When your own applications call model APIs directly, or when you need a job Insulin doesn’t describe: per-consumer token quotas, semantic caching, content safety, or prompt logs in your monitoring stack. Insulin’s AI Gateway governs the AI features inside Insulin. A call your own software makes to a model provider never reaches it.

Microsoft’s architecture guide treats adding a gateway as a design decision with costs. It says to “make the decision to add a gateway as part of the application design”, and warns about the cost: a gateway “is in a unique position to see raw request data and formulated responses from the language model”, and it “can introduce a single point of failure.”

Before you mark the requirement met or unmet, check five things:

1. Which calls must the gateway see? If only Insulin’s, the AI Gateway governs them. If your own applications’ too, they need a gateway of their own.
2. Must every call use your own keys? Turn Allow Fours platform key off, and match every row on the list to a connected key first.
3. Do you need quotas per person, app or team? Insulin documents one organization-wide monthly limit in dollars. Set per-key caps at the provider.
4. Do you need prompt and completion logs? Not described. Ask Fours what’s retained and how it can be exported before you count it as met.
5. Do you need caching or content moderation? Not described for Insulin’s calls.

Frequently asked questions

What is an AI gateway?

A layer that applications call instead of calling AI models directly, so that one place can authenticate, route, limit, cache, filter and log that traffic. Microsoft describes the AI gateway in Azure API Management as capabilities to secure, scale, monitor and govern AI models, agents and tools.

Does Insulin’s AI Gateway replace Azure API Management?

No. Insulin’s AI Gateway governs the AI calls Insulin makes: which providers may serve your organization, in what order, and whether only your own keys may be used. It isn’t a proxy your other applications call, so their model traffic needs a gateway of its own.

Can Insulin set token quotas per user or per app?

Not as Fours documents it. The control is one monthly spending limit for the whole organization, in US dollars, on Settings → Billing; reaching it pauses AI requests for the rest of the month. A provider’s own limits on the key you connect still apply.

Does Insulin’s AI Gateway log prompts and completions?

Fours’ documentation describes no gateway log of prompts and completions, and no way to export one. The Usage panel on Settings → Billing groups each month’s spend by resource, user, app, model or provider, with no CSV export and no per-request drill-down.

Can we route Insulin’s model calls through our own AI gateway?

Fours’ documentation doesn’t describe it. The provider connections it documents ask for credentials, such as an API key, and none takes a custom endpoint or base URL. Ask Fours before you plan to put Insulin behind Azure API Management or another gateway.

Is the AI Gateway the same setting as the AI Model Policy?

Yes. Settings → Organization → AI Gateway was formerly named AI Model Policy. It holds the same two controls: the ordered Allowed AI integrations list, whose top row is the default, and the Allow Fours platform key switch. Only an org ADMIN can change it.

Takeaways

• Insulin’s AI Gateway is a provider policy for one workspace, not a proxy for every app. It governs which providers and whose keys serve Insulin’s own AI calls.
• Two gateway jobs are covered: an ordered provider allow-list with a default, and your own keys only, failing closed.
• Three are partly covered: resiliency maps to per-turn failover that names the model that took over, token quotas to one organization-wide monthly spending limit in dollars, and logging to a usage breakdown that is a billing view, not a log.
• The rest aren’t described: load balancing, semantic caching, content safety, prompt-log export and routing Insulin’s calls through your own gateway.
• Your own applications that call models directly still need their own gateway. Insulin’s setting never sees their traffic.

To see the AI Gateway beside the agents it governs, start with Insulin’s AI agents and the AI Gateway your admins control.