The recorded requests are read from recorded/runs.json; when that file is not there yet, or JavaScript is off, this sentence is all there is.
Applied AI · LLMOps · evaluation in CI
Pay for an answer once. Prove it is still right.
Three things keep generative features out of production: the model bill grows with every request, tail latency is whatever the provider's queue is today, and nobody can show that last week's prompt change did not make the answers worse. This gateway sits in front of every model call and deals with all three. A question that was already answered comes back from a Redis semantic cache in milliseconds, with no model call. A new one is scored for complexity and sent to a light or a heavy chain of models, and a failing model hands over to the next one without dropping the request. Every step is a span of one trace, with tokens and dollars, mirrored to Langfuse.
And nothing merges on a hunch. A GitHub Actions workflow answers a golden set of questions with the real pipeline, has DeepEval and Ragas judge every answer, and blocks the pull request when the hallucination rate passes 5% or faithfulness drops below 0.95.
FastAPI · Pydantic v2 · Server-Sent Events · Redis 8 vector search (HNSW, cosine) · bge-small embeddings on the CPU · Langfuse v5 · DeepEval · Ragas · GitHub Actions · free OpenRouter models
Open the live demo
Code on GitHub
The live demo runs on free hosting and sleeps between visits. Opening it wakes it, which takes about a minute.
Runs at zero marginal cost: free models, a local embedding model, Redis and Langfuse on free plans. Every call is still priced at a paid model's list price, so the savings stay visible.
Recorded requests
Requests sent through the real application, saved as the API returned them.
What happens to one request
One request, one trace. The cache answers what it has seen; everything else is retrieved, routed, answered by the first model that can, checked, and stored for next time.
- 1
Guard
Unicode folded, invisible characters dropped, chat-template tokens and spoofed tags neutralized, known injection phrasings flagged. The question then travels inside tags with a random suffix, under a static system prompt that says what they mean. Input and output token ceilings are applied here.
- 2
Semantic cache
An exact repeat is a key lookup. Otherwise the question is embedded (bge-small, on the CPU) and Redis returns the three nearest cached questions of the same namespace. The first one above 0.96 that the contrast guard accepts is served.
- 3
Retrieval
Dense similarity and BM25 each rank the 45 chunks of a seven-document policy handbook; reciprocal rank fusion merges them and the top four become the prompt's sources.
- 4
Router
No model decides: a score built from named signals (classify, extract, compare, plan, several questions, how many documents the strongest passages span) picks the light or the heavy chain in microseconds, and every point is in the trace.
- 5
Cascade
A 429, a timeout, a 5xx, a withdrawn model, a model closed to this account (a 403), a refusal or an answer that fails validation moves the request to the next model, then to the other tier. A bad key (a 401) stops it. Streaming can fail over until the first token is sent.
- 6
Telemetry
Every step is a span; every model attempt a generation with tokens, the actual cost and the list-price cost. The service keeps p50, p95 and p99 per outcome, and mirrors each trace to Langfuse as it happens.
The cache threshold, measured
Pairs of questions embedded with the real model; no answer generated.
A semantic cache is only as good as the line it draws between "the same question" and "a different question that looks the same". Seventy-eight pairs below are rephrasings of one question; seventy-eight are look-alikes with another answer (domestic against international meal limits, a P1 against a P2 ticket). Move the threshold and see what each would cost: a missed saving, or a wrong answer served from the cache.
The measurements are read from recorded/cache-threshold.json.
- -of rephrasings would be served from the cache
- -of look-alikes would get the wrong answer, with the contrast guard
- -without the guard
The look-alikes closest to the line
| Similarity | Cached question | New question | Contrast guard |
|---|
What the cache saved on a replay
A synthetic help-desk day, replayed through the real pipeline: questions drawn from a list of help-desk intents with Zipf-like popularity (a few asked often, most rarely), each time in one of three or four phrasings. Misses are answered by the offline provider and priced at the list price of the tier's reference model, so the saving is measured in dollars a paid deployment would have spent.
- -of requests answered by the cache
- -exact repeats / semantic matches
- -wrong answers served (false hits)
- -of the list-price model spend avoided
- -exact lookup, p50
- -semantic lookup (embedding included), p50 / p99
Read from recorded/replay.json, written by python -m evals.replay.
Routing by complexity
Most questions a help desk gets are lookups: one passage, one number. They go to the light chain. Comparing, planning, a scenario, or passages from several documents push the score past 0.5 and the question goes to the heavy chain. The router is a few regular expressions and one number from retrieval, so it costs nothing and its reasons are visible. Here is what it decided for every question of the golden set:
| Question | Tier | Score | Signals |
|---|
Token economics
What a month of traffic costs at list price, and what each lever removes. The prices are Anthropic's per million tokens; the defaults are this gateway's measured prompt size and the replay's hit rate. Change them.
| Setup | Monthly cost | Against all-frontier |
|---|
Static prefix first
Providers cache prompts by prefix. The system prompt here has no date, no id and no nonce: it is byte for byte the same on every request, and the varying parts (passages, question) come after it. With Claude it is marked cache_control: ephemeral, and repeated prefixes are read at a tenth of the input price or less.
Ceilings on output
350 tokens for the light tier, 900 for the heavy one, whatever the client asks for. A model that rambles, or thinks out loud, is cut off and the next model is tried, so one bad response cannot run up the bill.
Priced twice
Every call carries two costs: what it cost (zero on free models, OpenRouter's own figure otherwise) and what the same tokens would cost at the list price of Claude Haiku 5.5 or Claude Opus 5.5. The second is what the cache and the router are measured against.
The quality gate
Telemetry tells you what happened in production; it cannot tell you whether a change is safe to merge. That is the job of an offline evaluation against a fixed set of questions, on every pull request. The golden set has 37 cases: 20 typical questions, 7 edge cases (boundaries, multi-step, two documents), 5 prompt-injection attempts and 5 questions the handbook does not answer.
Context precision
Are the passages that hold the answer ranked first? Scored twice: from labels, with no model, on every pull request; and by DeepEval's and Ragas' judges against the reference answer.
Faithfulness
Every claim of the answer is extracted and checked against the retrieved passages. A claim the passages do not support counts against the answer, not only one they contradict. The gate wants 0.95 or more.
Answer relevance
How much of the answer addresses the question. An answer can be faithful and useless; this catches the drift.
Hallucination rate
A case hallucinates when its answer contradicts the reference passages, when fewer than 70% of its claims are supported, or when the system answers a question the handbook does not answer. The gate allows 5% at most.
Injection resistance
Every adversarial case names strings that must not appear in its answer (a canary, a system prompt line). One leak fails the build: no judge needed.
Refusal accuracy
Answered when it should, declined when it should. Measured exactly, from the refusal token.
How a pull request is blocked
- Teststhe hermetic suite and ruff; no key, no network
- Retrieval checkthe real embedding model on the golden set; no key
- Generate answersthe real pipeline answers all 37 cases on free models
- DeepEvalRAG triad and hallucination, LLM-as-a-judge
- Ragasa second judge of the same answers, in its own environment
- Gatefloors, plus no metric more than 0.03 below the accepted baseline
| Rule | Threshold | Last measured |
|---|
The thresholds are read from the repository's evals/thresholds.json; the measurements from recorded/retrieval.json, recorded/deepeval.json (26 October 2026, free judge models: the gate failed by one case, a real incomplete answer and one false alarm from the judge, both named in the report) and, once Ragas has run with a key, recorded/ragas.json.
What it runs on
| Part | This demo (free) | Drop-in alternative |
|---|---|---|
| Gateway | FastAPI, Pydantic v2 bodies, SSE streaming, an OpenAI-compatible /v1/chat/completions | Any ASGI host; any Chat Completions client points at it unchanged |
| Semantic cache | Redis 8 with vector search on Redis Cloud's free plan (HNSW, cosine), or in memory | Any Redis 8 or Redis Stack (docker compose up -d redis) |
| Embeddings | BAAI/bge-small-en-v1.5 through fastembed, on the CPU, baked into the image | Any fastembed model (FASTEMBED_MODEL) |
| Models | Two chains of free OpenRouter models, light and heavy | Claude Haiku 5.5 and Claude Opus 5.5 through the Anthropic SDK, with prompt caching; OpenAI or Gemini |
| Tracing | In-process traces and percentiles; Langfuse Cloud's free plan when its keys are set | A self-hosted Langfuse (LANGFUSE_BASE_URL) |
| Evaluation | DeepEval and Ragas with free judge models, on GitHub Actions | A stronger judge: EVAL_JUDGE_PROVIDER=anthropic, a few cents a run |
| Hosting | Render free web service (Docker), GitHub Pages for this page | Docker Compose, any container host |
Read the README How it is deployed Phase 1: Invoice Extraction Service Phase 2: Compliance Retrieval Engine Phase 3: Multi-Agent Operations Pipeline