Architecture

two pipelines
Solid boxes run for every request; dashed boxes are offline steps that run once.
SumProof architecture: receipt reader and financial filing search pipelines

A. Receipt reader

  1. 1

    Read

    The photo goes straight into Qwen3.5-4B, a vision-language model. Its vision encoder turns the image into up to ~800 image tokens and the language model writes the receipt as JSON. There is no separate OCR step.

    receipts/vlm.py
  2. 2

    Sample

    Besides the normal (greedy) reading, the model samples four more, giving five candidates.

    vlm.generate
  3. 3

    Check

    Each candidate is tested against the receipt's arithmetic. The checker keeps the one that passes the most rules; if even that one breaks a rule, the receipt goes to human review.

    receipts/consistency.py
  4. 4

    Fine-tune (offline)

    QLoRA trains a small LoRA adapter (0.67% of the weights) on 800 real CORD receipts while the 4-bit base model stays frozen; the adapter plugs into the language model at inference.

    receipts/train_qlora.py

B. Financial filing search

  1. 1

    Index (offline)

    The 84 filings are split into 12,013 pages and each page is embedded with Qwen3-Embedding into a 1024-number vector.

    rag/download_financebench.py
  2. 2

    Route

    The question's company (name, ticker or nickname) and fiscal year select the matching filings.

    financebench.build_router
  3. 3

    Search

    The question is embedded the same way, and exact vector search finds the closest pages among the routed filings, in FAISS or in PostgreSQL with pgvector, where routing is a SQL WHERE clause.

    rag/index.py · rag/pgstore.py
  4. 4

    Rerank

    Qwen3-Reranker reads the question and each of the top 30 pages together and reorders them; the output is the evidence pages with their page numbers.

    financebench.py