Applied AI · structured extraction
Messy text in. Validated invoice out.
This web service takes the text of an email, a till receipt or a scanned page and asks a free language model to fill in a fixed invoice form. If it gets no answer that fits the form, it asks a second model.
FastAPI · Pydantic v2 · OpenRouter · strict JSON schema · temperature 0
The live demo runs on free hosting and sleeps between visits. Opening it wakes it, which takes about a minute.
The live demo is not online yet. This page shows recorded runs instead.
Measured on
- First model 21 of 21 answers fully correct no answer: 0 of 21 median answer 7.1 s
- Second model 16 of 19 answers fully correct no answer: 2 of 21 median answer 2.8 s
3 passes of 7 documents per model show whether a model can do the job, not how often it is right. See all four models
This email went in. This invoice came back.
Recorded run ·
A recording, not liveFrom: billing@northwind-traders.example To: sam@customer.example Subject: RE: RE: Your order - invoice attached Hi Sam, Sorry for the delay! Here are the details you asked for. Let me know if anything looks off. NORTHWIND TRADERS INC. Invoice #: NW-2024-0187 Date: March 14, 2024 Payment terms: Net 30 2 x Widget Pro (SKU 4471) @ $49.99 each ............ $99.98 1 x Annual support plan ............................ $250.00 Subtotal: $349.98 Sales tax (8%): $28.00 TOTAL DUE: $377.98 Thanks, Priya Northwind Traders | Accounts Receivable This email and any attachments are confidential.
Returned 200 Invoice extracted
NORTHWIND TRADERS INC.
| Description | Qty | Unit price | Line total |
|---|---|---|---|
| Widget Pro (SKU 4471) | 2 | $49.99 | $99.98 |
| Annual support plan | 1 | $250.00 | $250.00 |
- Line items
- $349.98
- Tax
- $28.00
- Total
- $377.98
- Answered by
- nemotron-3-super-120b-a12b the first of the two models
- Response time
- 9.15 s measured by the service
- Recorded
Checks this run had to pass 7 of 7 passed
- Returns HTTP 200
- Invoice number is NW-2024-0187
- Date is normalised to 2024-03-14
- Total is 377.98
- Tax is 28.00
- Currency is USD
- Two line items
The first model was made to fail. The second one answered.
Recorded run · · simulated failure
The same email was sent again. This time the first model's call was replaced by a simulated timeout, after 1 second instead of the usual 30. The rest was real: the service gave up on the first model, called the second one, and returned the same invoice.
-
1
1.01 snemotron-3-super-120b-a12bTimed out simulated
-
2
1.95 slfm-2.5-2.6bAnswered with a valid invoice real call
200 Invoice extracted Fallback used
- Answered by
- lfm-2.5-2.6b the second model
- Response time
- 2.96 s both attempts, the first one simulated
- Invoice
- NW-2024-0187 the same as in the first run, field for field
- Checks
- 4 of 4 passed
How the two models were chosen
Measured on , 17:54 to 19:02 UTC
Free models come and go, so this is a dated snapshot, not a live reading. Each of the four free models on OpenRouter that advertised structured output was sent 7 documents in each of 3 passes: six invoices and receipts, one of them an invoice for services, and one meeting email that must be refused. Every call went through this service's own prompt, schema and validation, with a time limit of 30 s.
| Model | No answer | Fully correct | Median answer |
|---|---|---|---|
nemotron-3-super-120b-a12bFirst model |
0 of 21 | 21 of 21 | 7.1 sslowest 14.4 s |
lfm-2.5-2.6bSecond model · Wrong tax on the service invoice in every pass |
2 of 21 | 16 of 19 | 2.8 sslowest 9.8 s |
dots-3-note-previewLeft out · Usual answer takes most of the time limit; the model ends on 31 December 2026 |
1 of 21 | 20 of 20 | 21.6 sslowest 27.5 s |
apodex-1.1-miniLeft out · Rejects the strict response format |
21 of 21 | – | – |
- No answer: the call ended without one. A rate limit, a timeout, a rejected format or cut-off output.
- Fully correct: of the calls that answered, those with every checked field right, or the meeting email refused.
- Order of the models: a model that rejects the format, or whose usual answer takes more than half of the time limit, is left out. Of the rest, fewest wrong answers first, then fewest calls with no answer. A fallback replaces a failed call, never a wrong answer.
- Records: every call behind this table is saved in
benchmarks/2026-10-07/175417Z.jsonl.
The code and the guides
- Code on GitHub The service, its tests, and this page with the test that checks its figures.
- Using the service Covers installing, calling, configuring and troubleshooting the service.
- The ideas behind it Explains the Applied AI concepts and why the code is shaped this way.
- The Python, explained Explains the actual code with runnable examples and exercises.