How Is OCR Accuracy Measured? Character and Word Rates

OCR accuracy is measured by comparing the software's text with a transcript a person has checked, then counting the differences. The count is divided by the length of the checked transcript, in characters or in words. So an accuracy percentage always depends on which unit was counted and which pages were tested.

A figure such as "99 percent accurate" therefore leaves out most of what you need to know. Ninety-nine percent of what, measured on which documents, against whose transcript? This post explains the arithmetic so you can read a quoted figure critically and produce one of your own.

What is OCR accuracy a percentage of?

OCR accuracy is a percentage of some counted unit, and the unit is a choice. Most often it is characters. It can also be words, or fields such as dates and amounts, or whole pages. The same output scores very differently depending on which one is counted.

Each step down the table is stricter. A word is wrong if a single one of its characters is wrong. A field is wrong if any word in it is wrong. A page is wrong if anything on it is wrong. A tool can score in the high nineties by character and far lower by page, on the same output.

Accuracy and error rate are two views of one count. An error rate of 2 percent is commonly restated as an accuracy of 98 percent. Error rates have the tighter definition, set out in the next section. When you see either figure, ask which unit it refers to.

One thing no accuracy figure includes is the software's own opinion of its work. That is a separate number, explained in the post on what an OCR confidence score does and does not say. Confidence is an estimate made without knowing the right answer. Accuracy is a measurement made against it.

Character error rate and word error rate in plain terms

Character error rate is the number of single-character edits needed to turn the OCR text into the correct text, divided by the number of characters in the correct text. Word error rate is the same idea counted in whole words. Both are usually written as percentages, and lower is better.

The edits come in three kinds. The evaluation specification published by the OCR-D project sets out definitions of character and word error rates in precise terms. In its terms:

• A substitution is a character replaced by another character in the output.
• A deletion is a character present in the text that is missing from the output.
• An insertion is a new character introduced in the output.

The smallest number of such edits that turns one string into the other is called the edit distance, or Levenshtein distance. The specification gives the character error rate as insertions plus substitutions plus deletions, divided by the number of characters in the checked text.

A small example

Take a four-word phrase from a fictional lease.

One character differs: the second "l" of "shall" was read as the digit 1. That is one substitution.

• Characters in the correct text, counting spaces: 23
• Character errors: 1
• Character error rate: 1 ÷ 23, about 4.3 percent
• Words in the correct text: 4
• Words with an error: 1
• Word error rate: 1 ÷ 4, or 25 percent

The same single mistake is 4.3 percent by one measure and 25 percent by the other. Neither is wrong. Word error rate is always the harsher of the two on ordinary text, because one bad character spoils a whole word.

Include one invoice in the sample and count mistakes in the total, the due date, and the party named on the claim. An error in the total changes the obligation printed on that page. An error in the party name hides the invoice when you search the folder. Write both figures down beside the page you checked.

Now a second output: Lessee shall assign. Every character that appears is correct, and the word "not" is gone. Counted by character, that is four deletions out of 23, about 17 percent. Counted by meaning, the clause has been reversed. No error rate weighs a dropped "not" more heavily than a dropped "the".

Details that change the number

Several choices inside the calculation move the result, and a quoted figure rarely states them.

• Spaces and punctuation. Whether they count as characters, and whether "end.Then" for "end. Then" is an error.
• Case. Whether "Section" and "section" are treated as the same.
• Character forms. Curly and straight quotation marks, or a ligature and its separate letters, may be stored as different characters. The OCR-D specification says both texts must be normalized in the same way before they are compared.
• What a word is. The same specification defines a word as a sequence of characters between white space, with leading and trailing punctuation removed. Another test may split on hyphens or keep the punctuation.
• Order. If the software reads two columns in the wrong sequence, every letter can be right and the character error rate still very high, because the comparison runs in order. The specification also describes a "bag of words" measure, which treats a text as a collection of words regardless of their order.
• Rates above 100 percent. Output with many inserted characters can need more edits than the correct text has characters. The specification notes this and gives a normalized form of the rate that stays between 0 and 100 percent.

What does 99 percent accuracy mean on a page of legal text?

On a page of legal text, 99 percent accuracy by character means roughly one wrong character in every hundred. A double-spaced pleading page of about 1,800 characters would carry about 18 errors. If each fell in a different word, around 18 of its 300 words would be wrong.

That page is an illustration, not a measurement of any product. The arithmetic for other rates on the same page, assuming the worst case in which each error lands in a separate word:

Real errors cluster, so the true count of damaged words is usually lower than the worst case. A smudge ruins five characters in one word, not one character in five words. But clustering cuts the other way too. An average over a page hides a clean top half and a bottom half under a coffee ring where nothing can be trusted.

Scale the same sum up. A 400-page production of similar pages holds about 720,000 characters. At 99 percent that is about 7,200 character errors. At 99.9 percent it is still about 720. Whether that matters depends on where they fall and what you are doing with the text.

Three limits of the percentage are worth stating plainly:

1. It does not say where the errors are. Eighteen errors in "the" and "of" are harmless. One in a settlement figure is not.
2. It does not say what kind they are. A garbled word draws the eye. A 3 read as an 8 does not.
3. It is an average. A set that scores 99 percent overall may contain pages at 100 and pages at 85.

How much accuracy is enough is a question about the task, and the post on OCR errors sets out how much accuracy a legal task needs, from search through to quotation.

Ground truth: the checked transcript every accuracy figure depends on

Ground truth is the correct text of the test pages, typed or corrected by people and checked against the page images. Every accuracy figure is a comparison with it. If the ground truth is wrong, incomplete or built to different rules, the accuracy figure is wrong in the same way.

The OCR-D specification describes ground truth as realistic images with manual transcriptions. The word "manual" carries the weight. Somebody has to read each test page and settle what it says.

Making ground truth forces decisions that have no single right answer:

• Line-end hyphens. Is "indem-" at the end of a line and "nify" at the start of the next transcribed as printed, or joined into "indemnify"?
• Headers, footers and page numbers. Are they part of the text or left out?
• Stamps and handwriting. Is a "RECEIVED" stamp transcribed? A handwritten initial?
• Struck-through text. Included, omitted or marked?
• Illegible passages. What does the transcript say where the person could not read the page either?
• Tables. Read across rows or down columns?
• Reading order. In what sequence are sidebars, footnotes and captions placed?

Whatever the transcriber chose becomes "correct". If the ground truth leaves out page headers and a tool faithfully reads them, each header counts as a run of insertion errors. If the ground truth joins hyphenated words and the tool keeps them as printed, each one is an error. A tool can lose points for being more literal than the transcript.

Ground truth can also be wrong. People mistype, and a tired checker passes the same misreadings software makes. Guard against that in your own test by having one person key the text and a second verify it against the image, or by making two independent transcripts and reconciling the differences.

A shortcut to be wary of is ground truth made by running OCR and having someone correct the output. It is faster, and it tends to inherit whatever the corrector did not notice. It also favors the tool that produced the first draft.

Why does a vendor's accuracy figure not transfer to your documents?

A vendor's accuracy figure does not transfer because it describes the vendor's test pages, and yours are different. Recognition depends heavily on print quality, layout, typeface and language. A figure earned on clean, modern, single-column pages says little about a faxed ledger or a stamped medical chart.

When you read a quoted figure, the missing facts are usually these:

None of this means a published figure is dishonest. It means the figure answers a narrower question than "how will this do on my files". Two figures from two vendors cannot be compared with each other either, unless both were measured on the same pages against the same ground truth by the same rules.

For that reason this blog gives no accuracy percentage for any product, Schema Law included. A number measured on someone else's pages would not tell you what you need to know. The useful figure is one you measure on your own documents.

Measuring OCR accuracy on a sample of your own pages

Measure OCR accuracy on your own pages by choosing a small sample, making a checked transcript of it, running the tool, and counting differences. For most law office decisions a field-level count on twenty pages is enough. A full character error rate is possible with free comparison tools, but it is rarely necessary.

Before uploading test pages anywhere, confirm that you may. Use public filings, closed matters your policy permits, or redacted copies.

1. Pick the sample. Take fifteen to twenty-five pages that reflect your real intake in its real proportions. If a third of your paper arrives by fax, a third of the sample should be faxes. Do not choose only the hard pages or only the clean ones.
2. Decide what you are counting. For search, count words. For data entry, count fields. For quotation, count characters in the passages you would quote.
3. Write the rules. Decide in advance how you treat headers, hyphens at line ends, stamps and handwriting. Write the rules down so a second test uses the same ones.
4. Make the ground truth. For a field-level test, list the values that matter on each page: names, dates, amounts, account and case numbers, citations. For a character-level test, type or correct the full text of a few pages, and have a second person check it against the image.
5. Run the tool at its default settings and export plain text, so formatting does not interfere with the comparison.
6. Compare. For fields, tick each value right or wrong. For full text, use the compare feature in a word processor or a text comparison tool, and count the differing characters or words.
7. Calculate. Errors divided by the total in your ground truth is the error rate. Subtract from 100 percent for accuracy.
8. Record the spread. Note the rate for each page as well as the overall figure, and note what kind of errors they were.

A worked tally

Suppose a paralegal at a fictional firm, Okafor and Lindqvist, tests twenty pages and lists fifteen critical values on each, for 300 values in all. The tool returns 288 exactly right.

• Field accuracy: 288 ÷ 300, or 96 percent
• Errors: 12
• Where they fell: 9 on the four faxed pages, 2 on a stamped page, 1 on a clean page
• What they were: 7 wrong digits, 3 misspelled surnames, 2 values missing

The overall figure, 96 percent, is the least useful line of that tally. The spread says more: clean pages are close to reliable, and faxed pages lose more than two values a page. The firm now knows which intake needs a full check against the image. Those numbers are invented to show the method.

How far a small sample goes

Twenty pages will show a large difference between two tools or two scan settings. It will not show a small one. If one tool misses 12 values in 300 and another misses 10, the gap is within what chance would produce, and a different twenty pages could reverse it. Treat close results as a tie and decide on other grounds.

Keep the sample and the ground truth. They let you repeat the measurement when a tool changes or when you change scanners. The post on choosing software describes running your own test on real documents as part of a wider trial that also covers limits, exports and cleanup time.

Schema Law's free plan reads one document a day of up to three pages and exports plain text, which is enough to try the procedure on a few pages before you set up a full sample. The pricing page lists what the paid plans allow.