At Yad Vashem I replaced a slow, manual Microsoft Access process with a Python pipeline that turns piles of mixed documents into clean, usable datasets. Batches of 5,000 to 100,000 records that once took months now take about a week.
The same approach works on the paperwork most businesses are still retyping by hand.
The problem
The records arrived as Excel files, CSVs, PDFs, Word documents and handwritten pages, in many languages, dating from the early 1800s to the mid-1900s. Someone had to read each one and key it in.
How the pipeline works
- Extract. Pattern matching (regex) handles the predictable parts, and a local RAG setup running on Ollama handles the messy ones.
- Translate. English translations are added alongside the original text.
- Clean and check. Results are normalised into one dataset, and anything doubtful is flagged for a person to review.
The biggest gain came from separating extraction from checking, so people only look at what needs a human eye.
Why local models
Sensitive records stayed on local machines rather than being sent to an outside service, which mattered for this kind of material.
Beyond archives
Accounting firms retyping client statements, brokers copying policy schedules, logistics teams keying in delivery notes: it is the same problem with different paper.
What I would tell anyone with a pile like this
- Start with a small sample of your real documents, not a perfect one.
- Measure the manual time first so you know what you saved.
- Keep a review step. Automation should surface doubt, not hide it.
If you have a pile like that, I can usually show what is possible in a short call, using a few of your own sample documents.