Receipt field F1
0.79→0.94
100 real test receipts
Receipts fully correct
6→62/100
every field right
Evidence in top 5
0.53→0.71
Recall@5, 150 analyst questions
pgvector query
26.5 ms→2.18 ms
HNSW index vs exact scan (median)
Receipt reader results
| Qwen3.5-4B | F1 | Numbers | Correct | s/rcpt |
|---|---|---|---|---|
| Zero-shot | 0.791 | 0.803 | 6/100 | 14.9 |
| + checker (best-of-4) | 0.799 | 0.812 | 7/100 | 37.0 |
| + QLoRA fine-tune | 0.935 | 0.962 | 59/100 | 11.0 |
| + fine-tune + checker | 0.940 | 0.965 | 62/100 | 28.1 |
Fine-tuning is the big gain; the accounting checker adds a little at 2.5× the time. Only 88 of the 100 human labels pass every rule, because some real receipts are printed inconsistently.
Retrieval ladder
BM25 struggles on 10-K pages full of repeated wording and number tables, so the common hybrid default drops below dense search. Routing by company and year, then reranking, gives the best result.
QLoRA training
Vector stores
| Index · 12,013 × 1024-d | Median latency | Same top-10 |
|---|---|---|
| FAISS IndexFlatIP (exact) | 4.8 ms | 100% |
| FAISS HNSW, efSearch 64 | 0.32 ms | 100% |
| PostgreSQL + pgvector, exact | 26.5 ms | 150/150 = FAISS |
| PostgreSQL + pgvector, HNSW | 2.18 ms | 99.6% |
In pgvector, filing routing is a SQL WHERE doc_name = ANY(...) clause next to the vector search.
Answer generation (RAG)
| Context given to Qwen3.5-4B | Correct | Wrong | Refused | Pages | Prompt tokens |
|---|---|---|---|---|---|
| No pages (closed book) · what the model knows | 8.0% | 58.7% | 33.3% | 0.0 | 113 |
| Top 5 reranked pages · standard RAG | 35.3% | 59.3% | 5.3% | 5.0 | 5,089 |
| Confidence-gated pages · 1 to 5 pages | 32.7% | 56.7% | 10.0% | 3.5 | 3,689 |
| The analysts' evidence pages · ceiling | 44.0% | 51.3% | 4.7% | 1.3 | 1,320 |
RAG raises correct answers from 8% to 35%. Keeping pages until they hold 90% of the reranker's confidence uses 28% fewer tokens with no significant accuracy change (McNemar p = 0.48). Even with the analysts' own pages a 4B model reaches 44%, so the next gain is in reasoning, not retrieval. For scale, the FinanceBench paper (2023, graded by hand) reports GPT-4-Turbo answering 19% of these 150 questions correctly with one search index over all filings and 50% with one index per filing; this pipeline's routing is closer to the per-filing setting.
Thinking budget (budget forcing)
| Reasoning before answering | Correct | Wrong | Refused | Seconds / question |
|---|---|---|---|---|
| No thinking | 38% | 60% | 2% | 4 |
| 1,024 tokens | 50% | 34% | 16% | 18 |
| 2,048 tokens | 54% | 28% | 18% | 33 |
| 4,096 tokens | 68% | 14% | 18% | 54 |
The 50 calculation questions, the same top-5 pages, Qwen3.5-4B in Ollama. The model may reason up to the budget, then must answer (s1 budget forcing). Reasoning cuts wrong answers from 60% to 14%: it more often notices a missing figure and says so instead of guessing. Filing search shows these reasoned answers for the calculation questions.
Explore the results
Before vs after
Every test receipt read before and after fine-tuning, against the human label.
Receipt inbox
All 100 test receipts: which add up, which need a person, and export to CSV.
Filing search
150 analyst questions: SumProof's cited answer and the evidence pages from real SEC filings.
Compare search methods
The same question under BM25, dense, routing and reranking, side by side.