Unified AI Gateway for App Builders

Providers diverge in four places: the shape of the message list, the names of the parameters, the event protocol on the stream, and the path where the error message sits. A module per vendor encodes those four differences once per vendor, and the modules drift apart as they are maintained — a fix applied to one is a fix missing from the others.

So a provider is not code here. It is a declarative spec the engine executes. The spec names the auth headers with a placeholder where the secret goes, a message style and a stream style by name, a map of parameter rules, and dot paths into the provider's error body.

Parameter rules cover four cases. Rename a field. Wrap a scalar the provider only accepts as an array. Fill a required field from the catalog row when the client omits it. Or hand the value to a named translator when the shape has to change rather than the name.

The engine reads that and does the work. The entire Anthropic provider is nineteen lines and contains no logic.

Config stops where the behavior is genuinely new. A provider whose message shape and stream protocol match an existing style is a data change: add the spec, add the catalog rows, ship. A provider speaking a protocol nothing else speaks needs a new stream style, which is code in the engine.

So the second provider on a known protocol is close to free and the first on a new one is not. That is the intended property — the cost sits where the novelty is.

A trace is not one record. A single turn in an app fans out into several model calls — a plan, a tool call, a summarization — and the useful question is almost always about the group rather than the call. So the SDK carries three identifiers: a session id that lives as long as the conversation, a trace id for one logical operation, and a request id per model call. The gateway validates them, bounds their length, and stamps them onto the event it records.

The record itself is split by access pattern. Metadata — the three ids, model, provider, status, finish reason, token counts, latency and cost — is a narrow row of roughly two hundred bytes in ClickHouse. The bodies, meaning the prompt, the response and any tool arguments, go to R2 under a key derived from the request id. The row stores the key and nothing else about the content.

That split follows from how the data gets read. Aggregations scan millions of rows and never open a body. A trace view reads one body at a time, and only when someone expands a call. Keeping bodies in the columnar store would make every scan carry bytes nobody asked for, and object storage costs roughly an order of magnitude less per gigabyte than analytic storage does.

The table is ordered by organization, then session, then time, so every call in a session is contiguous on disk. Rendering a trace is one range scan rather than a search, and it stays one range scan as the table grows.

None of this sits on the request path. Both writes happen after the response has been handed back — the row through the staging table and the minute drain, the body straight to R2 — so a tracing failure costs a log line rather than a call. For a streamed response the row is written when the stream drains, which is why latency measures generation rather than time to first byte, and why a stream that breaks mid-flight still produces a trace instead of a gap.

Cost is stamped on each row at request-time prices, so average cost per session is a sum over a contiguous range. That is the question I could not answer before any of this existed.

Model choice in most apps is a decision made once, on the day the feature shipped, and never revisited. New models arrive every few weeks, and the public benchmarks measure general capability, which is not the question anyone actually has. The question is whether a particular model is better at the particular job one app gives it.

The compare screen answers that with traffic that already happened. Pick a trace, or a saved set of them, pick two or three models, and the playground replays the recorded request against each. Output, token counts, latency and cost land side by side.

This is only cheap because of the storage split. The full request body is already in R2, keyed by request id, so a replay reconstructs nothing — it reads the recorded payload and sends it again. It sends it through the same adapter engine the gateway uses in production, so a comparison exercises the identical translation path rather than a separate harness that can drift from it.

Replays are marked at write time and carry a flag through to ClickHouse, so experiment traffic never lands in production cost rollups or latency percentiles. They still bill against the same key and the same balance. A model comparison that hides its own cost is not much of a comparison.

A group of traces can be promoted into a fixed set — twenty representative conversations, say — and rerun whenever a new model ships. The set stays constant, the model changes, and the diff is the only thing left to read.

What the screen does not do is grade. There is no score and no automatic winner: outputs sit next to each other and a person decides. An automated judge is the obvious next step and is not built.