Skip to content
Johnu Marattil

← Lab

Receipts to JSON with GPT

Johnu MarattilresearchWahCoins4 min

Losing a grocery receipt means losing track of what I spent. I like knowing where every dollar goes, and I kept missing individual items after a shopping trip. Try remembering what the tomatoes cost at Walmart last time.

So I went looking for a fix. Receipt scanning later became one of the features in WahCoins.

OCR plus GPT-3.5 Turbo

The idea came from a use of ChatGPT I came across: taking the messy output of OCR and pulling structured data out of it. Some commercial apps already do this with OpenAI and beat tools like Google Document AI, but they cost too much for personal use. I wanted a mix of open-source and commercial pieces that fit my needs.

Why DocTR

I spent days trying OCR tools: Google Vision, Document AI and Tesseract. Mindee DocTR, built on TensorFlow 2 and PyTorch, was the best fit for receipts:

  • It is built for document analysis, especially printed invoices and receipts.
  • It reads clear printed text well, from scans or photos with good contrast.
  • It handles structured documents that follow a consistent layout.
  • It reads screenshot receipts accurately when the quality is good.
  • It does better with simple preprocessing, like alignment and enhancement.

DocTR suited my case. A different language or document type might suit a different tool, so test more than one.

How it is built

Three parts:

  1. OCR. A Flask service runs DocTR on each uploaded image in real time and returns the text as blocks, lines and words.
  2. Structuring. Receipts put an item and its price on the same visual line, so lines that sit close together vertically are merged. That keeps each item paired with its price.
  3. API. One endpoint takes an image and returns the result.

What DocTR returns

Raw output for a receipt, with the card and reference numbers removed:

srveywalmart.com
SEETET
FR our, Walmart *
863-299-5527 Mgr:RAYMOND
355 CYPRESS GARDENS BLVD
WINTER HAVEN FL. 33880
00968 OP# 009038 TE# 38 TR# 06610
STH CCKTLS 004450005190 F 5.98 N
HF2 007874235186 F 3.54 0
GV WHOLE WHIP TOP 007874237173 F 1.72 N
REG CRM CHSE 2PK 007874203270 F 2.56 N
GRD BISCUIT 001800000188 F 1.24 N
...
JEGGING 088496824373 12.96 X
LEGGING 880979376907 5.96 X
SUBTOTAL 58.72
TAX 1 7.000 % 1.32
TOTAL 60.04
DEBIT TEND 60.04
CHANGE DUE 0.00
11/11/21 09:42:12
#I ITEMS SOLD 19

What GPT-3.5 Turbo makes of it

Fed that text, GPT-3.5 Turbo returns clean JSON:

{
  "store_information": {
    "name": "Walmart",
    "address": "355 CYPRESS GARDENS BLVD, WINTER HAVEN FL. 33880",
    "phone": "863-299-5527",
    "manager": "RAYMOND",
    "store_website": "srveywalmart.com"
  },
  "transaction_details": {
    "operator": "00968",
    "terminal": "38",
    "transaction": "06610"
  },
  "items": [
    { "description": "STH CCKTLS", "code": "004450005190", "price": "5.98" },
    { "description": "HF2", "code": "007874235186", "price": "3.54" },
    { "description": "GV WHOLE WHIP TOP", "code": "007874237173", "price": "1.72" },
    { "description": "REG CRM CHSE 2PK", "code": "007874203270", "price": "2.56" },
    { "description": "GRD BISCUIT", "code": "001800000188", "price": "1.24" },
    { "description": "JEGGING", "code": "088496824373", "price": "12.96" },
    { "description": "LEGGING", "code": "880979376907", "price": "5.96" }
  ],
  "summary": {
    "subtotal": "58.72",
    "tax_percentage": "7.000%",
    "tax_amount": "1.32",
    "total": "60.04",
    "payment_method": "DEBIT TEND",
    "change_due": "0.00"
  },
  "timestamp": "11/11/21 09:42:12",
  "items_sold_count": 19
}

What it does for personal finance

Grocery receipts, purchase orders, insurance receipts: I can now pull the details out of any of them, down to the tax, and save them.

Less data entry. You upload the document instead of typing each line. The system captures every line item, plus the tax and any discounts. Each document is kept, so a lost paper receipt no longer matters.

Sorting into categories.

  • Items. It picks out individual items, down to a single tomato, so you can see where the money went.
  • Context. Tomatoes from a grocery store go under Groceries. From a plant nursery, they might go under Gardening.
  • Automatic. Each item is filed under groceries, dining, entertainment, utilities and so on, with no manual tagging.
  • Custom categories. You can add your own on top of the defaults.

What could come next

Once you have item-level history, other features become possible:

  • Forecasting. Predict future spending from past receipts, and warn before you overspend.
  • Budgets. Set a budget per category and get feedback as you spend against it.
  • Shared costs. Split and track shared expenses for roommates, trips or an office.

Combining OCR with a language model only scratches the surface of what’s possible for managing money. The real potential is in how these pieces keep improving together.

Filed under research