AI platform · governance · security · LLMOpsShared control plane

AI Gateway— governed model access as shared infrastructure.

A shared edge gateway between applications and model providers. Clients never receive provider credentials and do not bind product logic to a specific upstream model. Identity, request contracts, routing, spend controls, policy and telemetry are enforced at the platform boundary.

Cloudflare Workers
Durable Objects
D1
App Attest
Turnstile
Cloudflare Access
TypeScript
OpenAI / Kilo

Stack

Edge runtimeCloudflare Workers · TypeScript

Rate limitingsharded Durable Objects

TelemetryD1 · route / latency / token / cost metadata

Native trustApple App Attest · short-lived gateway tokens

Browser trustorigin policy · CORS · Turnstile

Operator accessCloudflare Access / Zero Trust

Engineering evolution

Reconstructed from commit history. The sequence reflects technical changes visible in the repository; related commits are grouped into platform milestones.

Centralize credentials and narrow the proxy.

The first control was deliberately simple: provider credentials remain server-side and applications call allowlisted capability routes instead of an unrestricted upstream proxy.

Add rate limits, usage accounting and budgets.

Once multiple applications share infrastructure, abuse and spend become architectural concerns. Durable Objects, usage accounting and daily/monthly budgets moved those controls into the request path.

Support different client trust models.

Native apps gained App Attest and replay-protected challenges; browser clients can use origin policy and Turnstile. The platform supports heterogeneous clients without reducing every client to the weakest trust mechanism.

Decouple product code from model names.

Logical routes were refactored into tiers and capability-specific endpoints. Provider or model migration becomes a platform concern instead of an application release concern.

Make routing decisions observable.

Cost-aware and capability-aware routing, route decisions and feedback telemetry allow model selection to be evaluated rather than treated as opaque infrastructure.

Harden operator access and fail closed.

Zero Trust around operational surfaces and fail-closed budget checks moved the gateway from a shared proxy toward infrastructure that can be operated across multiple products.

Developer contract

Stable logical routeslow, mid, high, coding and typed capability routes express product intent while isolating applications from provider churn.

Safe defaults over arbitrary configurationClients cannot freely override providers, methods or output limits. A narrower contract is easier to secure, observe, migrate and support.

Operational telemetry, not prompt surveillanceUseful platform telemetry is route, selected model, latency, token usage, estimated cost, status and quality signals. Prompt bodies do not need to become an analytics dependency.

Budgets are executable policyA budget that only reports overspend is accounting. A preflight budget check that blocks before provider spend is enforceable platform behavior.

Governance & compliance model

Identity before inferenceClient identity and request validity are established before paid or sensitive upstream processing begins. Native, browser and operator access can use different assurance mechanisms.

Provider policy is centralizedApplications request capabilities. The control plane determines which providers, models and limits are allowed, making policy changes possible without redeploying every client.

Cost controls fail closedRate limits and spend policy are part of authorization for the request. This prevents an availability or abuse incident from automatically becoming a cost incident.

Auditability without unnecessary payload captureRouting and operational metadata can support traceability while avoiding a design that requires sensitive prompt content to be retained by default.

Trust domains stay explicitNative clients, browsers, operators and model providers cross different trust boundaries. Those boundaries remain visible in architecture and policy instead of being collapsed into one API key.

Governance is executableIdentity, request schemas, route allowlists, limits and budgets are implemented as code paths. Documentation explains the policy; runtime controls enforce it.

Request path

Client→Calls a logical tier or typed capability, not an arbitrary provider URL.

Policy→Identity, request schema, rate limits, output caps and budget policy execute before upstream work.

Router→Server selects provider/model from capability, cost and route policy.

Provider→Request streams through server-held credentials and normalized error handling.

Telemetry→Route, status, latency, token usage, estimated cost and routing decision can be written to D1.

Platform outcome

The application boundary is intentionally uneventful: a product asks for a capability; the platform authenticates, constrains, routes, meters and observes the request. That separation allows multiple AI features to operate under one policy model instead of becoming independent provider integrations.

Sharedone control plane serves native and web products

Abstractedapplications target capabilities / tiers instead of model names

Fail closedconfigured identity, rate and cost gates execute before provider spend