You Know Money Is Being Spent. Not How Much or Why.
Each team calls the LLM API directly with its own key. The total cost appears at end of month. Attributing it to a team, project, or specific agent requires manual work nobody has time for.
AI GOVERNANCE
Govern Every AI Call, Beyond Your Vendor's View.
Many teams, many LLM providers, no central visibility of who is calling what, how much it costs, or what data leaves the perimeter. Your AI provider's dashboard only shows its own traffic, not an audit trail that's yours, reviewable by your security team.
The problem isn't adopting AI. It's governing it.
The AI Governance Gap
The Three Symptoms
You know AI adoption is happening across the organization. What you don't have is a shared control layer across it.
The Same Prompts Run Multiple Times, Across Teams.
Two teams send semantically identical questions to the same model. They pay twice. Nobody knows, because there is no shared layer between them and the provider.
Security Cannot Audit What It Cannot See.
Prompt injection, PII in payloads, API keys hardcoded into prompts. Without inspection at the entry point, these surface only when something goes wrong.
Definition
What Is AI Governance?
AI governance is the set of policies, controls, and observability mechanisms that determine who in your organization can call which AI model, under what conditions, at what cost, and with what data leaving your perimeter.
It is not a compliance framework alone. It is not a dashboard. At the technical level, it is the enforcement layer between your teams and the LLM providers they call, applied at the point where the traffic actually flows: the API layer.
Without AI governance enforced at that layer, the only visibility you have into your AI traffic is the bill at end of month and the logs inside the vendor's portal, which is not your portal. With it, every LLM call is authenticated, authorized, rate-limited, logged, and attributable to a team, a user, or a cost center, before the request leaves your infrastructure.
Why the API Layer
Governance Belongs at the API Layer
Every LLM call is an API call: an endpoint, a payload, a cost. The same infrastructure that governs REST traffic can govern LLM traffic, plus AI-specific additions: token budgets, prompt inspection, semantic caching.
The alternative is governance built into each application, every team with its own auth, logging, cost controls. That's shadow AI: a dozen teams calling a provider with a dozen keys, no central visibility, no way to answer who sent a prompt last Tuesday.
Governance at the API layer means one enforcement point, one audit trail, one config file, versioned in Git.
Every organization also writes a governance policy, usually owned by legal and compliance under frameworks like the EU AI Act. Necessary, but a wiki page can't stop a request or produce a log for your CISO. KrakenD does not replace the policy, it enforces it.
Categories of Control
What AI Governance Enforces
Categories of control, applied at the gateway before the request reaches the model, each backed by a specific component, all defined in a single JSON config file that deploys as a Docker image and runs on any infrastructure. See the full AI Gateway »
Identity and Access
Not every team should call every model. Not every user should access every agent. JWT, OIDC, mTLS, and API key validation are enforced at the gateway before the request reaches the model, the same auth stack already used for REST APIs.
How KrakenD Solves This: auth/validator points at your existing IdP, Entra ID, Cognito, Keycloak, Auth0. The same validator you already use for REST endpoints applies automatically to your LLM and MCP tool-call endpoints, no separate auth stack to build or maintain.
Token Budgets and Cost Attribution
Which team is spending how much, on which model, per day, per month. Budget enforcement at the gateway means a runaway process or a misconfigured agent cannot generate unbounded LLM spend. Quotas are enforced before the request leaves the perimeter.
How KrakenD Solves This: governance/quota, backed by Redis for persistent, cross-instance counting, enforces hourly, daily, and monthly limits per user, team, or cost center, weighted by token cost per model. The remaining budget is returned to the client in response headers, so applications can react before they hit the ceiling.
Prompt Inspection and Data Protection
PII in prompts, malformed or oversized payloads, prompt structures that let an end user inject arbitrary instructions. Inspection at the API layer happens before the payload reaches the model, so data that should not leave your infrastructure does not.
How KrakenD Solves This: JSON Schema validation rejects malformed or oversized payloads before they reach the backend. Request body modifiers can remove or replace specific fields, including sensitive data patterns, and prompt templates constrain what an end user actually controls in the payload sent to the model.
Audit Trail Independent of the Vendor
Your LLM vendor keeps logs inside their portal. Those logs are not yours. AI governance enforced at the API layer emits traces to your own observability stack, in the formats you already use, so the audit trail lives in your infrastructure, not in a vendor dashboard you lose access to when you change providers.
How KrakenD Solves This: native, vendor-neutral OpenTelemetry exports metrics and traces to Grafana, Datadog, New Relic, your SIEM, whatever you already use, enabled once at the service level. No separate agent, no vendor lock-in on where the logs live.
Real Architectures, No Names
How Organizations Like Yours Are Solving This
Scenarios based on real customer architectures, each grounded in a verified, solvable technical problem.
The Industrial Company With a Heterogeneous Agent Fleet
One Visibility Layer Across Every Agent
Dynamics, Copilot Studio, and internally built integrations running in parallel. The goal, a single visibility layer across all AI traffic, with token budgets per team and semantic caching so the same question from two different business units does not get billed twice. KrakenD sits in front of the pro-code and custom-built layer. The office suite manages its own server-side calls. The two layers are complementary, neither replaces the other.
One thing worth being clear about: KrakenD governs traffic that passes through a client you control. SaaS platforms that route LLM calls server-side (Copilot Studio, ServiceNow agents, Dynamics) go through the vendor's infrastructure, not yours. It complements those layers, it does not replace vendor-specific consoles.
One thing worth being clear about: KrakenD governs traffic that passes through a client you control. SaaS platforms that route LLM calls server-side (Copilot Studio, ServiceNow agents, Dynamics) go through the vendor's infrastructure, not yours. It complements those layers, it does not replace vendor-specific consoles.
The Team Building Role-Based Agent Access
Same Endpoint, Same Model, Different Access by Role
A platform where executives consolidate data from three regional agents in parallel, while regional users see only their own region. Same endpoint. Same model. Behavior determined by the JWT claim. KrakenD reads the claim, routes to the correct backends, aggregates responses in parallel for the executive role, and keeps a per-user trace, who called which agent, how many times.
The Regulated Environment With On-Premise GPUs
No Database, No PVC, Full Auditability
On-site inference hardware, no PVCs, no database in the critical path. Two non-negotiable requirements, attribute token cost per model and per team, and fail over automatically when a model does not respond. KrakenD stateless, no PVC, no node coordination. Configuration lives in Git. Any auditor can review it without going through the LLM vendor.
The Platform Team Exposing APIs to AI Agents
One Spec, REST and MCP From the Same Definition
A backend team with 200+ REST endpoints already in production. The ask: expose them as tools for LLM agents without duplicating the OpenAPI spec, without duplicating the auth model, and without touching the backends. KrakenD reads the existing endpoint config and exposes each operation as an MCP tool. The JWT validation, rate limiting, and schema validation already in place for REST apply automatically to every agent tool call. The backend never knows an AI agent called it. The governance cost of adding a new surface: one annotation per operation.
One Source of Truth, Two Delivery Surfaces
Hivebrite runs the community engagement platform behind 2,000+ alumni networks, professional associations, and corporate programs worldwide. When MCP became a practical way to expose APIs to LLM agents, the question was whether the same OpenAPI contract that already drove their REST API could generate MCP tools without duplicating configuration.
The answer was a single annotation per operation. Their build-time resolver reads it and emits both the REST endpoint and the corresponding MCP tool from the same definition. No second spec to maintain. No second security model to enforce. The JWT validation, schema validation, and rate limiting already built for REST come along automatically on the agent surface.
Today, more than 200 of Hivebrite's REST operations are wired as MCP tools from those same definitions using KrakenD's built-in MCP Server (see how the MCP Server works »). The cost of exposing an existing operation to an LLM agent is one annotation line. Three MCP-specific Go plugins handle the edge cases specific to their stack: argument forwarding to Rails bracket notation, error passthrough for agent-readable validation detail, and header-to-body promotion for pagination metadata. Everything else runs on the stock configuration.
One source of truth, two delivery surfaces. We expect this pattern to become fairly standard as more teams start exposing their REST surface to LLMs.
Cyril Schmitt
Backend Engineer, Hivebrite
Read the Case Study »No Vendor Lock-In
Works With Your Existing Stack
No IdP migration, no new observability pipeline, no rip-and-replace of the gateway you already run. If you are currently on Kong, Apigee, Tyk, or AWS API Gateway, AI governance is one more reason to consider the switch.
Auth You Already Have
JWT, OIDC, mTLS. No IdP migration required.
Observability Where You Already Look
Native OpenTelemetry. Grafana, Datadog, New Relic, your SIEM. The trace goes where you already have tooling.
No Database in the Critical Path
Stateless by design. No PVC, no coordinator node. Configuration is immutable and lives in Git.
Multi-Provider by Default
OpenAI, Anthropic, Bedrock, Azure OpenAI, Gemini, on-prem models. Your application calls one endpoint. KrakenD decides the backend based on cost, latency, role, or compliance rules.
Your Existing REST APIs, Exposed as Agent Tools
KrakenD's built-in MCP Server exposes your existing REST APIs as tools for LLM agents, from the same OpenAPI contract, with the same auth and rate limiting already in place.
Runs Where Your Infrastructure Already Is
KrakenD deploys as a standard Docker image on Kubernetes, any cloud provider, bare metal, or air-gapped on-premise environments. No managed service to provision, no cloud account required, no vendor dependency in the runtime. The same binary that governs your LLM traffic in GKE today runs in an air-gapped data center tomorrow, with no changes to the config.
Independently Verified
Verified by G2 Users
G2 Fall 2026 API Management, Mid-Market segment: KrakenD Quality of Support scores 98%, the highest of any vendor in the table. Mid-Market customers recover their investment in 3 months (category average, 11.65 months).
Source: G2 Fall 2026 API Management reports, Mid-Market segment. Data collected through July 28, 2026. Third-party verified peer reviews, not vendor claims.
98%
Quality of Support, Mid-Market, the highest of any vendor in the category
100%
of reviewers rate
KrakenD 4 or 5 stars
KrakenD 4 or 5 stars
High Performer
in every G2 quadrant measured, Mid-Market Europe and Overall
AI Governance FAQ
Frequently Asked Questions
1. Does KrakenD replace the AI features already built into Copilot Studio, ServiceNow, or Dynamics?
No. KrakenD governs traffic that passes through a client you control. SaaS platforms that route LLM calls server-side go through the vendor's own infrastructure, not yours. It complements those layers by governing the pro-code and custom-built traffic sitting alongside them, it does not replace vendor-specific consoles.
2. How does per-team token budget enforcement actually work?
KrakenD reads the JWT claim on each request, routes it to the correct backend, and enforces a token ceiling per team, project, or agent at the gateway level. No application code has to track or report its own usage. The same claim-based logic is what lets a single endpoint serve different roles differently, for example an executive view that aggregates multiple regional agents in parallel while a regional user only sees their own region.
3. We already log LLM calls at the application level. Why do we need a gateway layer for this?
Application-level logging only sees what that one application sends. When many teams call several providers directly with their own keys, there is no shared layer to attribute cost, catch duplicate prompts across teams, or produce one audit trail your security team can review without going application by application. A gateway sits in front of all of it, independent of which team or app is calling.
4. We run inference on-prem with no PVCs and no database allowed in the critical path. Does KrakenD still work?
Yes, this is a common deployment pattern. KrakenD is stateless: no PVC, no node coordination, no database in the critical path. Configuration lives in Git, so any auditor can review exactly what is deployed without going through the LLM vendor. Token cost attribution per model and per team, and automatic failover when a model does not respond, both work the same way in this setup as anywhere else.
5. Is this only for OpenAI and Anthropic, or does it work with our internal or on-prem model?
Multi-provider is the default, not an add-on. OpenAI, Anthropic, Bedrock, Azure OpenAI, Gemini, and on-prem models all sit behind the same endpoint. Your application calls one URL. KrakenD decides which backend actually serves the request based on cost, latency, user role, or compliance rules.
6. Do we need to change our application code to adopt this?
No. Your application keeps calling one endpoint. Routing decisions, token budgets, semantic caching, and prompt guarding are all configured at the gateway, not in application code. Exposing an existing REST operation as an MCP tool for agents follows the same pattern: one annotation per operation, not a rewrite.
7. What is the difference between an AI gateway and AI governance?
An AI gateway is the infrastructure: the proxy that sits between your applications and LLM providers. AI governance is what that gateway enforces: access control, cost limits, audit trails, data protection. You need both. A gateway without governance policies is a passthrough. Governance policies without a gateway are a document. KrakenD is the gateway; the governance is the config you write for it. See the full AI Gateway »
8. Does KrakenD replace my LLM provider's dashboard?
No. Your provider's dashboard shows you their view of your traffic. KrakenD gives you your view, attributable to users, teams, and cost centers, stored in your own observability stack, independent of which provider you use. The two are complementary. You keep both.
9. Does AI governance require a separate service?
No. AI governance enforcement runs in the same binary as your API gateway. There is no separate AI governance service to deploy, no additional database for the governance layer, no second control plane. The governance config lives alongside your existing gateway config in the same file.
10. What happens if a model goes down?
KrakenD's conditional backend routing supports a fallback strategy that routes to a secondary model when the primary does not respond. Failover is declared in config and requires no application changes. The circuit breaker is complementary: it stops routing to a model that is consistently failing before fallback even triggers.
11. Does KrakenD govern agent-to-agent traffic?
KrakenD governs LLM traffic and MCP tool calls through its built-in MCP Server. Agent-to-agent (A2A) traffic is a different protocol and not in scope for the current release. The MCP layer covers the tool execution surface, your existing REST APIs exposed as MCP tools, with auth, rate limiting, and observability applied at the tool level. Learn more about the MCP Server »
12. Is this relevant if I only use one LLM provider?
Yes. Single-provider setups benefit from governance in the same ways: cost attribution by team, prompt inspection, quota enforcement, and an audit trail that lives in your infrastructure. The multi-provider routing features become relevant when you evaluate a second provider or when pricing changes make switching attractive.
Request a Demo
Walk through real-world scenarios: LLM routing across multiple providers, token budget enforcement per team, prompt security policies, and AI agent governance, all from a single config file on infrastructure you already trust.
Request a Demo See How the AI Gateway Works