Blog ·

Receipt and invoice extraction: own model vs GPT, field by field

Extraction looks easy until you parse the output. A model that gets the total right but writes "12,50 €" when your system expects 12.50 has failed, even though a person reading it would say it was correct. We score extraction field by field, as exact matches, and count format errors separately.

The setup

Data. CORD v2 (CC BY 4.0) from NAVER Clova: about 1,000 receipts with ground-truth JSON for menu items, quantities, prices, subtotal, tax and total. We used the text of each receipt as input and the JSON as the answer.

Split. We uploaded 800 receipts. Honed trained on 649 and held out 114.

Models. With the "lowest cost" goal, Honed trained gpt-oss 20B and Nemotron 3.5 Lightning 30B in parallel and kept gpt-oss 20B. It compared the winner with Claude Haiku 5.5, GPT-6 Luna, Gemini 3.8 Flash and Claude Sonnet 5.5, each prompted with the field list and the target format, and with the untrained gpt-oss 20B.

Scoring. Field accuracy over the top-level fields menu, sub_total and total. A field counts only if its whole value matches the ground truth exactly, including nested items.

Results

Model Field accuracy Median latency Cost per 1,000 receipts
Fine-tuned gpt-oss 20B 71.1% 701 ms $0.12
Claude Sonnet 5.5 51.6% 1,474 ms $3.22
GPT-6 Luna 49.7% 3,746 ms $0.21
Gemini 3.8 Flash 37.0% 4,678 ms $2.56
gpt-oss 20B, untrained 14.3% 1,374 ms $0.20
Claude Haiku 5.5 3.9% 3,452 ms $0.42

The fine-tuned model was 19.5 points more accurate than Claude Sonnet 5.5, at about 1/26 of its cost per receipt and half its median latency. The whole run, from upload to this table, took 13 minutes and cost us $1.38 in compute.

By field

A small sample: these are the 10 examples Honed shows on the result page.

Model menu sub_total total
Fine-tuned gpt-oss 20B 6/10 7/8 10/10
Claude Sonnet 5.5 3/10 7/8 9/10
GPT-6 Luna 2/10 8/8 9/10
Gemini 3.8 Flash 1/10 8/8 8/10
Claude Haiku 5.5 0/10 2/8 0/10

The frontier models get the totals right. Almost the whole gap is in menu, the line items and their shape.

What the misses look like

  • A receipt with one item, SATE T1 40,000. The CORD format writes a single item as an object, {"nm": "SATE T1", "price": "40,000"}, and several items as a list. The fine-tuned model returned the object. Luna, Sonnet and Gemini returned a list with one object in it. The values were right and the field was wrong.
  • Claude Haiku 5.5 filled every key it was told about. It added "unitprice": null, "service_price": null and more, so no object matched, and 3 of 10 answers were not a valid JSON object.
  • The untrained gpt-oss 20B put the price in the count field, "cnt": "40,000", and listed the tax line as a menu item.

None of these are hard for a person. They are conventions of one schema, and a prompt describes them less well than 649 examples do.

Why fine-tuning suits extraction

Extraction is a format problem more than a knowledge problem. The model already knows what a total is. It has to learn your schema: which keys, which number format, what to do when a field is missing. That is the kind of behaviour fine-tuning teaches well. Research on structure-heavy tasks points the same way. Anyscale fine-tuned Llama-2-13B on a structured-output task and accuracy rose from 58% to 98% (Anyscale), and in LoRA Land the fine-tuned models beat GPT-4 on named-entity extraction, 0.989 against 0.742 (arXiv 2405.00732).

Both studies tested against frontier models from 2023 and 2024.

On your own invoices

Honed has an "Invoice fields" template that scores exact JSON field matches. Upload a CSV with the document text in one column and the fields your finance team entered in the others, or a JSONL with the target JSON as the answer. Card numbers, IBANs and tax IDs are masked on upload. If your fields contain them, decide first whether the model needs to see them.

Sources