How Does OCR Work? From Page Image to Characters

OCR works by narrowing a picture of a page down in stages. The software separates ink from paper, finds the lines of text, splits them into words and characters, decides which letter each shape is, weighs that guess against the letters around it, and writes the result into a file you can search and copy.

Knowing the stages is useful for one practical reason. Each stage makes its own kind of mistake, and a wrong word in your converted lease looks different depending on where it went wrong.

How does OCR work, in one paragraph?

OCR takes a page image, cleans it, maps where the text sits, cuts the text into lines, words and character shapes, classifies each shape as a letter, uses the surrounding letters and words to settle doubtful ones, and then assembles the characters into an output file. Every stage works on what the stage before handed over.

The stages, with what each one receives and produces:

This post covers the middle of that table in depth: segmentation and classification. Three neighbors own the rest. Cleanup is explained in the post on OCR preprocessing. The mapping step is covered under how document OCR divides a page into regions. The context stage has its own post on whether OCR uses a dictionary.

Two designs you will meet

Not every reader is built as a chain of separate stages. Ray Smith's paper An Overview of the Tesseract OCR Engine describes the staged design, saying of that engine that "processing follows a traditional step-by-step pipeline". A later design recognizes a whole line of text in one step. The Tesseract project's overview of the neural network system in Tesseract 4.00 says that version "includes a new neural network subsystem configured as a textline recognizer."

The staged design is the easier one to learn from, because you can see each decision being made. This post uses it as the worked case and notes where a line recognizer differs.

From scanned page to lines, words and single characters

Segmentation is the stage that cuts a block of text into lines, the lines into words, and the words into single character shapes. It works from geometry alone: which dark pixels touch, how tall the marks are, where they sit relative to a baseline, and how wide the gaps between them are.

Blobs: marks that touch

The first unit is a connected patch of ink. The Tesseract paper calls the first step "a connected component analysis in which outlines of the components are stored", after which outlines are "gathered together, purely by nesting, into Blobs."

Take one line from a fictional promissory note between Aldous Reyne and Corbin Mill Supply:

Borrower shall pay $11,900.00 on or before 1 June 2027.

On a clean scan, the B is one blob. The i in "Corbin" would be two: the stem and the dot do not touch. A lowercase r and n printed so close that their ink meets form a single blob. An h whose shaft has faded in the middle forms two. So a blob is not yet a letter. It is a letter, or a piece of one, or several stuck together.

Lines and baselines

Blobs are then sorted into text lines. The paper says the line finding algorithm was "designed so that a skewed page can be recognized without having to de-skew, thus saving loss of image quality." It works with the blobs of ordinary height first. Blobs much smaller than the median are set aside for the moment, "being most likely punctuation, diacritical marks and noise", and are fitted back into their lines once the lines are known.

That detail matters on a legal page. The decimal point in $11,900.00 and a speck of toner are the same kind of object at this stage: a very small blob near a line.

Each line then gets a fitted baseline, the invisible rule the letters sit on. The paper notes that the engine fits curved baselines, which it calls "a common artifact in scanning, and not just at book bindings."

Words: definite spaces and doubtful ones

Lines are cut into words by measuring gaps. How that is done depends on the type. The paper puts it this way: "Fixed pitch text is chopped immediately by character cells. Proportional text is broken into words using definite spaces and fuzzy spaces."

Fixed pitch means typewriter-style type, where every character occupies the same width. A typed affidavit from an older file is the easy case: the software can cut on a regular grid.

Proportional type is harder, and the paper says so with an example. In a sample it shows, the gap between the tens and units of "11.9%" is "a similar size to the general space". A reader that cuts on gap width alone would write 1 1.9%. For a figure in a loan document, that is the difference between one number and two.

Characters: chopping and joining

Within a word, the engine must decide which blobs are whole letters. Two repairs are described:

• Chopping joined characters. When a word reads poorly, the engine tries cutting the least convincing blob at likely points. The paper's figure shows a cut "where the 'r' touches the 'm'".
• Associating broken characters. If cutting does not help, the pieces are tried in different combinations until some grouping reads as letters.

Both repairs are trial and error steered by the classifier. Segmentation and classification are not strictly one after the other. The engine proposes a cut, asks what the pieces look like, and keeps or undoes the cut depending on the answer.

A line recognizer skips the character cutting. It still needs the page cut into lines, which is why a crooked or crowded page can hurt either design.

How does software decide which letter a shape is?

Software decides which letter a shape is by comparing it with what it knows about each character and picking the closest match. There are three broad ways to make the comparison: laying the shape over stored patterns, measuring features of its outline, or passing it through a network trained on a large set of examples.

Pattern matching

The simplest method stores a picture of each character and compares the unknown shape with every one. Amazon's explainer on intelligent document processing describes it in one line: "Simple OCR software uses matching algorithms to compare text images to text and font image pattern templates". The method suits a single, known typeface. A change of font, size or print quality moves the shape away from every stored pattern at once.

Features

A feature-based classifier does not compare whole pictures. It breaks the outline of the shape into small measurable parts and asks which character has parts like these. In the Tesseract paper, short segments of the unknown outline are "matched many-to-one against the clustered prototype features of the training data."

The paper gives a reason for working this way, with a broken h as the example. The shaft of the letter is in two pieces, yet nearly every small feature still finds a matching prototype, so the letter is recognized. The process, it says, "is easily able to cope with recognition of damaged images."

Classification in that engine happens in two steps. First, "a class pruner creates a shortlist of character classes that the unknown might match." Then the shape is measured closely against each character on the shortlist. The result is not one letter. It is a ranked set of candidates, each with a number attached. The post on the OCR confidence score explains what such numbers do and do not mean.

Learning from the page itself

A general classifier has to cope with any font, which makes it less sharp on any particular one. The Tesseract paper describes a second, adaptive classifier that learns during the job. In the first pass over a page, "each word that is satisfactory is passed to an adaptive classifier as training data", and that classifier "then gets a chance to more accurately recognize text lower down the page." A second pass then revisits words that were not recognized well enough the first time.

One consequence for a law office: an engine built this way does not read a page in isolation from itself. A clean first paragraph can help it with a smudged signature block in the same typeface.

Trained networks

A network-based reader replaces hand-designed features with a neural network trained on images paired with their correct text. Training, not a programmer, settles which aspects of a shape matter. Tesseract's documentation says its network is "integrated into Tesseract as a line recognizer", so the unit it classifies is a strip of text, not one blob.

Whatever the method, some pairs of shapes are close in any typeface: lowercase l, capital I and the digit 1; capital O and zero; rn and m. The typeface decides how close. That subject belongs to the post on the best font for OCR.

Why OCR reads whole words and lines instead of lone letters

OCR reads whole words and lines because a single shape can be ambiguous in a way its neighbors resolve. A vertical stroke could be l, I or 1. Next to "ega" it is an l. Between a dollar sign and a comma it is a 1. The surrounding characters carry information the shape alone does not.

Context enters at three points.

During word cutting. In the Tesseract paper, a gap that is neither clearly a space nor clearly not one is marked as fuzzy "so that a final decision can be made after word recognition." The decision about the gap waits until the text on either side of it has been read.

During character cutting. A proposed chop is tested by reading the pieces. The paper says: "Any chop that fails to improve the confidence of the result is undone".

During word choice. The engine compares several candidate strings for the same patch of ink. The paper lists the categories its language module considers, starting with "Top frequent word, Top dictionary word, Top numeric word". The winner is the candidate with the best combined score across shape and category.

A line recognizer takes this further. It sees the whole strip, so every character can be read in the light of those on both sides.

What context cannot help with

Context is evidence about language. It helps most where the page holds ordinary sentences. Tesseract's guide to improving the quality of the output says so directly: "By default Tesseract is optimized to recognize sentences of words." The same guide lists receipts, price lists and codes as text for which that default may need changing.

Legal documents are full of strings that are not sentences of words:

On these strings the reader falls back on shape alone. That is why the proofreading advice for any OCR result is the same: read the names and numbers against the image first.

When the engine has emitted words, try a pleading. Look up the allegation and any citation to a statute. Letter cuts can glue the allegation to a line number, and language context can rewrite a party name missing from the word list. Match those strings to the shapes still visible on the page.

What happens after the characters are recognized?

After recognition, the software cleans up its own result and builds a file. It settles the last doubtful spaces, joins words into lines and paragraphs, attaches a position to each piece of text where the format needs one, and writes everything out in the format you asked for.

The finishing steps differ by product, but they fall into four groups.

1. Late corrections. In the Tesseract paper, "a final phase resolves fuzzy spaces, and checks alternative hypotheses for the x-height to locate smallcap text." Some applications also run their own spelling or pattern checks at this point.
2. Rebuilding structure. Lines are grouped into paragraphs. Words hyphenated across a line break may be rejoined. Text regions are placed in reading order, and table cells are assigned to rows and columns.
3. Recording positions. For a searchable PDF, each word needs the coordinates of the spot it came from, so the invisible text can sit under the picture of the word.
4. Writing the file. The same recognized page can be written as a searchable PDF, a Word document, plain text or structured data.

Some of this work belongs to the recognition core and some to the application around it, which is the split described in the post on what an OCR engine is. It also explains why two exports of the same recognized page can look quite different. A searchable PDF shows you the original picture, so a wrong letter is invisible until you search or copy. A Word export shows you only the recognized text, so every wrong letter is on display. The post on what an OCR file is sets out the forms recognized text can take.

In Schema Law the choice of file comes last. The page is read once, you see the result on screen, and you can export it in any of the formats listed on the homepage, all of which are included in every plan. If the tool you use runs in a browser, the post on OCR text recognition online follows the same pipeline from upload to result.

Where in the OCR pipeline each kind of mistake is made

Each stage of OCR leaves a recognizable kind of mistake. Wrong letters inside a correct word shape point to classification. Wrong word boundaries point to segmentation. Correct words in the wrong place point to layout. A real word that is not on the page points to the context stage. Missing or extra marks point to cleanup.

Use this table when you find an error and want to know whether more like it are waiting.

Tracing one error

Suppose the converted promissory note says the borrower shall pay $1 1,900.00.

1. Open the page image at high zoom and look at the figure. Are the two ones farther apart than the other digits?
2. If so, the fault is word segmentation, and it comes from the spacing of that typeface. Check every other figure with repeated or narrow digits in the document, because the same spacing will recur.
3. If the gap looks normal, look for a mark between the digits: a fold line, a speck, the tail of a signature from the line overhead. That is a cleanup or line-finding fault, local to that spot.
4. Correct the figure against the image, and note which kind of fault it was. The note tells you how much of the remaining document to recheck.

The distinction between a systematic fault and a local one is the point of the exercise. A classification error tied to the typeface will repeat on every page set in that typeface. A speck is a one-off.

No tool removes the need to look. Schema Law's own answer to how bad a scan can be, under questions partners ask first, says that nothing marks a passage as uncertain and that you should check names, dates and amounts against the original. The broader patterns, and a method for proofreading, are in the post on why OCR output is full of errors.