How to Test OCR Software on Your Own Legal Documents
By Schema LawPublished Updated
To test OCR software properly, run every candidate on one fixed set of your own pages, chosen to cover each kind of document you handle, under the same conditions. Score the exported files against a checked answer for text, tables and layout, time the corrections, and decide the pass mark before you look at any result.
A test built this way replaces opinion with a count you can show a partner, and the same set can be run again whenever a product or your intake changes.
Pick the sample with the people who will live with the result. Include a deposition transcript, a pleading with footnotes, and a contract schedule. Ask a paralegal to time the corrections and an attorney to say whether the Word file is fit to edit. If litigation support will run the tool at volume, have them score the same set so the pass mark reflects their queue.
How many pages does a fair OCR software test need?
A fair OCR software test needs enough pages that every document class you rely on has several, so one odd page cannot decide a class. For a small office that is 20 to 40 pages: at least three per class, more for the classes that make up the bulk of your work, plus one long file.
Why one page per class is too few
A single faxed page might be the cleanest fax you ever received or the worst. If it is your only fax in the test, the tool's score on faxes is really its score on that page. Three pages from three different senders show whether a result repeats. Five show it more firmly. Beyond that, the extra pages cost checking time and add little for a buying decision.
A sizing table
| Your situation | Classes | Pages per class | Test set |
|---|---|---|---|
| Solo, two or three document types | 3 | 4 to 5 | About 15 pages |
| Small office, mixed intake | 5 to 6 | 3 to 5 | 20 to 30 pages |
| Team with tables and poor copies in volume | 6 to 8 | 4 to 6 | 30 to 45 pages |
Add one complete long document on top, at the length you really handle, to test limits and not reading. You do not score every page of it.
What limits the size
The limit is the answer key. Every test page needs a checked list of what it truly says, and someone has to prepare that by reading the paper. Time yourself preparing the key for two pages, then multiply by the size of the set you have in mind. Decide how much checking the office can give before you settle the number of pages.
This is a buying test, not a statistic
Thirty pages will not give you an accuracy percentage you could publish. It will show you which candidate needs fewer corrections on your kinds of page, which is the question you are paying to answer. If you need formal error rates, the section on measuring OCR accuracy on a sample of your own pages explains how to count character and word errors.
Choosing OCR test pages that represent a firm's most common and worst scans
Choose OCR test pages in two groups. The first mirrors your ordinary intake in proportion, so the score reflects daily work. The second holds your worst scans on purpose, so you learn where each tool breaks. Label every page with its group and class, because the two groups are scored apart.
The representative group
Pull pages by source, not by how they look. Selecting by eye favors pages that are easy to read.
| Source | How to pick | What it stands for |
|---|---|---|
| Your own copier | Every tenth page of last week's scans | First-generation scans at your settings |
| Fax intake | One page from each of three senders | Low resolution, headers, banding |
| Productions received | First page of three documents from different custodians | Bates stamps, legends, mixed quality |
| Signed agreements returned | Signature page and one clause page | Print with ink over it |
| Records from providers or agencies | Three pages with tables | Ledgers, billing, logs |
| Phone photos from clients | Two photos as they arrived | Angle, shadow, uneven focus |
The stress group
Add four to eight pages that have caused trouble before. Ask the people who do the conversion now. They will name them without looking: the ledger with no lines between columns, the order with a stamp across the date, the third-generation copy of a deed, the intake form filled in by hand. If you have nothing on file, the post on why OCR output is full of errors lists the scan faults worth including.
One control page
Include a single clean, laser-printed page. It is the page every candidate has the best chance with. If one stumbles on it, suspect the upload or the settings first, and sort that out before trusting the rest of its results.
Freeze the set
Save the chosen pages as files in one folder, number them, and do not change them once testing starts. Record for each: number, group, class, source and page count. A candidate tested on a different set has not been compared with anything.
Prepare the answer key
For each page, a person reads the original and writes down:
- Ten to fifteen items where an error would hurt: names, dates, amounts, section and account numbers.
- For a table, the number of rows and columns and three cell values with their positions.
- For layout, what should stay together: a heading with its number, a footnote apart from the body.
Do this before any tool has seen the page. A key written after looking at output can inherit the output's mistakes.
Which documents are safe to use in an OCR software test?
The safest documents for an OCR software test are ones already public, such as filings on an open court docket. Next come the firm's own non-client papers. Pages from client matters, open or closed, are a disclosure to a vendor, and whether you may upload them depends on your duties, firm policy and the tool's terms.
This section describes how to pick test material. It is not advice on your professional obligations, which vary by jurisdiction. Check your own rules and firm policy.
Four sources, in order of caution
| Source | Why it suits a test | What to check first |
|---|---|---|
| Public filings | Already open to anyone, and they look like your real work | That the document is truly public: not sealed, not restricted, no protective order covering it |
| Firm's own papers | No client involved | That they contain no client or staff personal details |
| Closed-matter pages | Realistic poor scans and tables | Firm policy on closed files, any agreement or order that still binds, and the tool's terms |
| Open-matter pages with client consent | The exact documents you will convert | The engagement terms, whether consent is needed and has been given, and the tool's terms |
Public filings
Briefs, orders and exhibits from a public docket give you transcripts, tables and stamped pages with no confidentiality question. Print a clean one and re-scan it at poor settings, or fax it to yourself, to make a degraded copy for the stress group. Confirm the filing is not under seal before you use it.
Closed matters
A closed file is still a client's file. Do not assume that closing the matter freed its documents for other uses: look up what the professional conduct rules where you practice say about a former client's information, and read your firm's policy on closed files, before treating them as test material. Check, too, whether a protective order or a confidentiality agreement from the matter still restricts the documents. The post on OCR for documents under a protective order covers that question.
Client consent
If the only way to test on realistic pages is to use a live matter, the question of consent comes first. The post on client consent to use AI tools discusses when it is asked for and how engagement letters word it, and attorney-client privilege and AI tools covers the duties behind it.
Read the trial terms
A free trial is governed by the vendor's terms like any paid plan, and a trial tier can carry different terms from a paid one. Before any upload, find the vendor's statements on training, retention and deletion, and note them. When the test ends, delete the test documents from every candidate you are not buying, and record that you did.
Stripping details is not a cure-all
Blacking out names on a scan before a test changes the page the tool sees, which changes the result. It can also fail to remove what it hides if done carelessly. If you take this route, the post on OCR before or after redaction explains the safe sequence.
Running the same test pages through each OCR candidate under equal conditions
Equal conditions mean each OCR candidate receives identical files, in the same order, at default settings, handled by the same person, with the same exports requested and the same things recorded. Anything that differs between runs becomes a rival explanation for a difference in the scores.
The rules
- Same files. Upload the frozen files byte for byte. Do not re-scan, crop or rotate for one candidate and not another.
- Defaults first. Run each tool as it comes. If you then adjust a setting, such as a language or a table option, record it as a second run and score it apart.
- Same operator. One person runs every candidate. A second person who knows one product well will get more from it.
- Same exports. Ask each tool for the same formats: the ones on your requirements list. Where a tool lacks a format, record that and move on.
- Same clock. Time the same steps for each: from starting the upload to having the export open.
- Same plan tier you would buy. A trial may differ from the paid plan in limits or formats. Note which tier you tested.
- No help from the vendor. A sales engineer running the demo is not the same as your paralegal on a Tuesday.
A run log
Keep one line per candidate per run.
| Field | Entry |
|---|---|
| Candidate and plan tier | |
| Date and operator | |
| Settings | Default, or what was changed |
| Files uploaded | Numbers from the frozen set |
| Files refused, and the message shown | |
| Formats exported | |
| Formats not available | |
| Time for the long document, upload to export | |
| Anything odd | Pages out of order, pages missing, a stall |
Where the test tier limits you
A free tier may not take your whole set in one sitting. Schema Law's Free plan, for instance, allows 1 document a day of up to 3 pages, with every export format included, according to the pricing page. A 30-page set would take at least ten days at that rate, so running it in one sitting there means a paid month or a call with the vendor. Work out the same thing for each candidate before you start, so that no product is scored on fewer pages or formats without the log saying so.
Save everything
Keep each export in a folder named for the candidate and the run. You will open these files again when scoring, and a colleague may want to see the page that decided the matter.
A scoring sheet for OCR test results: text, tables, layout and time to correct
Score OCR test results on four measures per page: key items read correctly, tables rebuilt correctly, layout kept, and minutes needed to make the export usable. Record each as a number. Score from the exported file opened in the program you will use it in, and score the stress group on its own sheet.
The four measures
| Measure | How to score it | Unit |
|---|---|---|
| Text | Compare the export with the answer key. Count key items that are wrong or missing. | Errors out of items checked |
| Tables | Check row count, column count and the three known cells. A table with the right digits in the wrong cells fails. | Pass or fail per table |
| Layout | Check each "should stay together" note in the key: numbering with headings, footnotes apart, reading order. | Faults per page |
| Time to correct | Fix the export until you would send it to a colleague. Stop the clock. | Minutes per page |
Time to correct is the measure that connects the others to money. Two tools with the same error count can differ widely in how long the errors take to find.
The sheet
One sheet per candidate, one row per test page.
| Page | Group | Class | Key items | Text errors | Table | Layout faults | Minutes to correct | Note |
|---|---|---|---|---|---|---|---|---|
| 01 | Rep. | Control | 12 | n/a | ||||
| 02 | Rep. | Fax | 14 | n/a | ||||
| 03 | Rep. | Ledger | 10 | |||||
| 04 | Stress | Stamped order | 12 | n/a | ||||
| ... | ||||||||
| Totals, representative | ||||||||
| Totals, stress |
A worked example
Suppose a fictional probate practice, Halloran Estates Law, tests two candidates on 24 representative pages carrying 300 key items and 8 tables.
| Candidate 1 | Candidate 2 | |
|---|---|---|
| Text errors, of 300 items | 11 | 19 |
| Tables passed, of 8 | 5 | 8 |
| Layout faults | 6 | 9 |
| Minutes to correct, all 24 pages | 95 | 70 |
Candidate 1 reads text better. Candidate 2 gets every table right and takes 25 fewer minutes to correct overall, because a failed table costs far more time to rebuild than a misread word costs to fix. For a practice whose work is estate inventories and account statements, Candidate 2 is ahead despite more text errors. The figures are invented. The point is that the four measures can disagree, and the time column settles it.
Things to look for that a count can miss
- Text that is fluent but wrong. A tool that produces a sensible sentence the page does not contain is more dangerous than one that produces garbage, because nothing draws the eye. The post on OCR hallucination explains how to catch it.
- Missing pages. Count the pages in every export of the long document.
- Silent uncertainty. Note whether the tool tells you where it struggled. Where it does not, as Schema Law's homepage says of its own output, every name, date and amount needs checking against the original.
When do OCR test results justify a purchase, and when should you retest?
OCR test results justify a purchase when one candidate meets a pass mark you wrote down in advance on the representative group and its failures on the stress group are ones you can live with. Retest when no candidate passes, when the leaders are too close to call, or when the test itself was flawed.
Set the pass mark before scoring
Write it down while the answer key is being prepared, in terms of your four measures. For example:
- Text: no more than 5 errors per 100 key items on representative pages.
- Tables: every table in your most common class passes.
- Layout: no fault that would corrupt a quotation.
- Time: correction takes less time than your current method.
These numbers are an illustration. Yours come from what an error costs in your work. A pass mark chosen after seeing the results will fit whichever tool you already liked.
Reading the outcome
| Outcome | What it means | Next step |
|---|---|---|
| One candidate passes, the others do not | A clear result | Buy, on the shortest term offered, and keep the test set |
| Two pass | Both are adequate readers of your pages | Decide on cost, formats and terms, using your comparison table |
| None passes on representative pages | Either the tools or the scans are the problem | Check the control page, then your scan settings, then widen the field |
| A candidate passes on representative pages and fails badly on stress pages | It suits daily work and not your worst intake | Buy if stress pages are rare, and plan how those pages get handled |
| Results differ by a few items | Within the noise of a small set | Add pages to the deciding class and rerun both |
When the test was at fault
Rerun instead of deciding if any of these happened:
- The control page scored poorly on every candidate. The files or the scanner settings are suspect.
- Candidates were run on different files, by different people, or on different tiers.
- The answer key was written after seeing output.
- A class that drives the decision had fewer than three pages.
When to run it again after buying
Keep the frozen set, the key and the sheets. Run them again when the vendor announces a significant change, when your intake changes, before a renewal, and when staff report that results seem worse. A stored test turns "it seems worse" into a count.
Where the trial sits in the wider decision
A passing test shows that a tool reads your pages. It says nothing about its terms, its limits or its price, which are settled on paper before the test begins. The section on running your own test on real documents places the trial among the other buying criteria, and the questions section of the Schema Law homepage shows the kind of plain statement about limits to look for from any vendor.