LLM Observability & FinOps Service

The problem

Three AI services in production, each calling a model provider, each logging its own usage somewhere, each computing its own cost. The finance question — what did AI cost us last month, and for which customers — had three partial answers that did not reconcile.

Worse, the three services disagreed about vocabulary. One called them prompt and candidate tokens, another input and output, a third counted reasoning tokens separately. Summing them naively double-counted reasoning, because in one provider’s accounting output tokens already include it.

The approach

One service that owns the canonical usage table. Producers push raw events, as a single object or as a batch; this service validates, resolves the price, computes the cost and persists it. Reads go through Metabase — there is deliberately no query API of its own.

Decisions worth defending

The cost sent by a client is discarded. Producers may include a computed cost; the ingestion whitelist drops it. Every figure is recalculated here.

Trusting the client would have been less code and would have let each service optimise its own accounting. It also would have meant that a pricing bug in any one producer becomes a permanent wrong number in the financial record, discoverable only by someone recomputing by hand. A single place to be wrong is worth far more than three places to be right.

Price is versioned in time, and the version is frozen onto the row. The pricing table carries an effective-from date; resolution picks the latest entry at or before the event time, and the resulting pricing_version is written alongside the cost. A provider changing its tariff therefore never rewrites history. The alternative — computing cost at read time from current prices — would have made last quarter’s reported spend change when a vendor updated a price list, which is not a report, it is a rumour.

A model with no price recorded produces zero cost and a version of unknown. Not a rejected event, and not a silent zero. Rejecting would lose the usage data, which is still true and still useful; a silent zero would understate spend with no trace. unknown is queryable, so the gap shows up in the dashboard as a gap.

Idempotency by client-supplied event id, with a unique constraint. A retry is a no-op at the database level rather than at the application level, because the failure mode being defended against is the producer retrying after a response it never received.

Batches accept partial success. A batch returns 202 with the accepted count and a list of rejections by index and reason. All-or-nothing would mean one malformed event in a batch of two hundred discards a hundred and ninety-nine valid ones, and the producer’s retry resends the same bad event forever.

The emitters are best-effort and degrade to no-ops. An unconfigured or unreachable metrics service must not slow down, fail, or block the extraction being measured. The rule written into every producer: the chat never depends on this.

Backfill runs with a dry run that reports parity before writing anything. Importing history from legacy databases through read-only credentials, the default mode reads, maps, reports whether the numbers agree, and writes nothing. A one-way migration of financial data should have to be asked for twice.

Scale

• ~1,560 lines, 4 modules, 5 migrations, 2 entities
• ~20 analysis dimensions with 7 composite indexes designed against the dashboard queries
• 3 producing services normalised into one canonical token model
• 2 providers in the pricing catalogue