Model results

Qwen3.5-4B fine-tunedReal data · $0
Reads receipt photos and checks that the numbers add up · finds the page in a real SEC filing that answers an analyst's question · one 8 GB laptop GPU

Receipt field F1

0.79→0.94

100 real test receipts

Receipts fully correct

6→62/100

every field right

Evidence in top 5

0.53→0.71

Recall@5, 150 analyst questions

pgvector query

26.5 ms→2.18 ms

HNSW index vs exact scan (median)

Receipt reader results

10 sent to review
Qwen3.5-4BF1NumbersCorrects/rcpt
Zero-shot0.7910.8036/10014.9
+ checker (best-of-4)0.7990.8127/10037.0
+ QLoRA fine-tune0.9350.96259/10011.0
+ fine-tune + checker0.9400.96562/10028.1

Fine-tuning is the big gain; the accounting checker adds a little at 2.5× the time. Only 88 of the 100 human labels pass every rule, because some real receipts are printed inconsistently.

Retrieval ladder

0.71Recall@5

BM25 struggles on 10-K pages full of repeated wording and number tables, so the common hybrid default drops below dense search. Routing by company and year, then reranking, gives the best result.

QLoRA training

30.5M trainable params · 7.2 GB peak
0.000.250.50end of epoch 1 · checkpoint used150100150optimizer step (8 receipts each)
Validation field F1 (30 receipts):epoch 0: 0.83epoch 1: 0.98

Vector stores

identical top-10 · 150/150
Index · 12,013 × 1024-dMedian latencySame top-10
FAISS IndexFlatIP (exact)4.8 ms100%
FAISS HNSW, efSearch 640.32 ms100%
PostgreSQL + pgvector, exact26.5 ms150/150 = FAISS
PostgreSQL + pgvector, HNSW2.18 ms99.6%

In pgvector, filing routing is a SQL WHERE doc_name = ANY(...) clause next to the vector search.

Answer generation (RAG)

judge agrees with exact rule 92.5%
Context given to Qwen3.5-4BCorrectWrongRefusedPagesPrompt tokens
No pages (closed book) · what the model knows8.0%58.7%33.3%0.0113
Top 5 reranked pages · standard RAG35.3%59.3%5.3%5.05,089
Confidence-gated pages · 1 to 5 pages32.7%56.7%10.0%3.53,689
The analysts' evidence pages · ceiling44.0%51.3%4.7%1.31,320

RAG raises correct answers from 8% to 35%. Keeping pages until they hold 90% of the reranker's confidence uses 28% fewer tokens with no significant accuracy change (McNemar p = 0.48). Even with the analysts' own pages a 4B model reaches 44%, so the next gain is in reasoning, not retrieval. For scale, the FinanceBench paper (2023, graded by hand) reports GPT-4-Turbo answering 19% of these 150 questions correctly with one search index over all filings and 50% with one index per filing; this pipeline's routing is closer to the per-filing setting.

Thinking budget (budget forcing)

38% to 68%, McNemar p = 0.0001
Reasoning before answeringCorrectWrongRefusedSeconds / question
No thinking38%60%2%4
1,024 tokens50%34%16%18
2,048 tokens54%28%18%33
4,096 tokens68%14%18%54

The 50 calculation questions, the same top-5 pages, Qwen3.5-4B in Ollama. The model may reason up to the budget, then must answer (s1 budget forcing). Reasoning cuts wrong answers from 60% to 14%: it more often notices a missing figure and says so instead of guessing. Filing search shows these reasoned answers for the calculation questions.

Explore the results