OCR Extract Text From PDF: What Happens Inside

When you use OCR to extract text from a PDF, the tool works through six stages. It checks whether each page already holds text, draws the page as an image, maps the text regions and their order, recognizes the characters, rebuilds lines and pages, and writes a file. Each stage can fail in its own way.

Most explanations of OCR stop at the letters. For a lawyer deciding how far to rely on a text file, the letters are only one handoff out of six, and several of the others produce mistakes that are spelled perfectly. This post walks through each handoff and shows what its failure looks like in the text of a legal document.

What does OCR do to a PDF before any text is extracted?

Before any text is extracted, an OCR tool makes three decisions about your PDF that you never see. It decides whether a page needs reading at all, it draws the page as a picture at a resolution it picks, and it marks out where the text sits and in which sequence. Recognition starts only after that.

Those early decisions belong mostly to the program wrapped around the recognizer, not to the recognizer. The post on what an OCR engine is explains that split. It matters here because two products can share a recognizer and still hand you different text, purely from what they did to the PDF first.

Here are the six stages in order, with the failure each one tends to produce and how that failure reaches you.

Only stage 4 produces what people picture when they think of an OCR error. The rest fail at the scale of a block or a page, and their output passes a spell check.

Not every tool keeps the middle stages separate. The post on AI OCR describes what model-based systems do differently. Stages 1, 2 and 6 remain in any design, because a tool that reads pictures of pages still has to make those pictures and still has to write a file.

The first decision: use the stored text or read the picture

A PDF page can already contain characters, so the first thing a tool settles is whether to recognize the page at all. It has three choices: take the stored characters as they are, leave the page alone, or ignore what is stored and read the picture. You are rarely told which choice was made.

The open-source tool OCRmyPDF shows how deliberate this decision is. Its documentation on when OCR is skipped says the default behavior exists so that files recognized earlier, or created digitally and never scanned, are not processed a second time. Other products choose differently, and many do not document the choice.

The decision goes wrong in three ways.

• A stamp passes for a full page of text. A scanned exhibit that later received a typed e-filing header or Bates number contains a few real characters. A tool that asks only "is there text here?" answers yes. In the extracted text, that page is one line long.
• Bad stored text is trusted. Characters left by an earlier, poor recognition pass, or by a font with broken character codes, are copied straight through. You get pages of garbage sitting between clean pages. The post on converting a PDF to readable text covers how to tell those two causes apart.
• Good stored text is thrown away. A typed brief is read as a picture anyway, and exact characters are replaced by recognized ones. The post on OCR for PDF documents explains when OCR makes a PDF document worse.

The mark of a stage 1 failure is its size. The problem covers a whole page and stops cleanly at the page's edge. To find one, compare the amount of text per page with what the page plainly holds. A dense contract page that yields twelve words was never read.

What picture of your page does the recognizer receive?

The recognizer never opens your PDF. It receives a picture of each page that the tool has drawn for it, at a size and resolution the tool selected. If that picture is coarser than the scan inside the file, turned sideways or scaled wrongly, the characters are damaged before anything tries to read them.

A PDF page is a set of drawing instructions, not a single photograph. The introduction in the OCRmyPDF documentation notes that "A PDF page may contain multiple images, even if it appears to have only one image." So the tool has to flatten whatever is on the page into one image. The same page describes that step: the program "analyzes each page of a PDF to determine the required colorspace and resolution (DPI) for capturing all the information on that page without losing content", then uses a rasterizer "to convert each page to an image".

That sentence describes a careful tool. Nothing forces every product to be careful. A scan saved at a high resolution can be drawn at a lower one to save time, and the detail the scanner captured is then gone as far as the recognizer is concerned.

Resolution can also be wrong without being low. A file may record the wrong figure for its images, and the tool then misjudges how large the type is. The OCRmyPDF cookbook puts the consequence this way: "OCR quality will suffer if the resolution of input images is not correct (since the range of pixel sizes that will be checked for possible fonts will also be incorrect)." The post on the best DPI for OCR has a section on scanned dpi versus the dpi a PDF was saved or downsampled at.

Rendering failures have a recognizable pattern on legal pages.

• Small type fails first. The body of a lease reads well. The footnotes, the fine print in a rider and the reference line under a signature come out as fragments. Large type survives a coarse picture and small type does not.
• One page fails completely. A landscape schedule bound into a portrait document is drawn on its side. Its text is nonsense or absent, while the pages around it are fine.
• Everything is slightly off. When the scale is misjudged for the whole file, errors are spread evenly and do not cluster around stamps or smudges.

If the weak passages in your text are the small ones, suspect this stage before you blame the scan.

Finding the text regions and numbering them in order

Before reading a single letter, the tool marks out blocks on the page picture and decides the order in which to read them. A person does this at a glance. Software has to infer it from white space and alignment, and when it infers wrongly, every word is right and the passage is still false.

Engines start from an assumption about what a page looks like. Tesseract's guide to improving the quality of the output states its own: "By default Tesseract expects a page of text when it segments an image." Plenty of legal pages are not that. A caption page, a form with labeled boxes, a ledger and a page of initials are all arrangements, not running text.

The same guide is frank about one arrangement, saying "It is known tesseract has a problem to recognize text/data from tables" unless custom layout analysis is added. The OCRmyPDF introduction lists reading order among that engine's limits: it "might fail to recognize two columns in a document and attempt to join text across columns."

Those are statements about one engine, and others do better or worse. The point is the kind of failure, which looks like this in practice:

• Two columns of a commercial policy read straight across, so each line of text is half of one clause and half of another.
• The line numbers down the margin of pleading paper land inside the sentences.
• A footnote is read where it sits on the page and appears between two halves of a body sentence.
• A shaded notice box is treated as a picture and its words are left out.
• A rent schedule is read as a paragraph, with dates and amounts strung together in one line.

Nothing in that list contains a misspelling. A search still finds the words, which makes the damage easy to miss until someone quotes the passage. How a page is cut into blocks is the subject of the post on how document OCR divides a page into regions, and the two-column post explains what reading order is and why it breaks.

How do recognized characters become lines, paragraphs and pages?

Recognition returns small pieces, typically words or lines with their positions on the page. A separate step joins those pieces into paragraphs, decides what to do with headers and footers, and puts the pages in sequence. That joining step guesses at structure the engine never reported.

Recognition itself, meaning how a shape is judged to be a particular letter, is covered in the post on how OCR works. Its failures are the familiar ones: a wrong digit in an amount, a wrong letter in a surname.

What comes back from recognition is flatter than most people expect. Amazon's documentation on lines and words of text in Textract defines the unit it returns: "A LINE is a string of tab-delimited and contiguous words." The OCRmyPDF introduction says of the engine it uses: "Tesseract does not divide text into paragraphs or headings. It only provides the text and its bounding box."

So a paragraph in your text file is usually a reconstruction. Something looked at line spacing, indents and line lengths and decided where one paragraph ended. The same goes for a heading, and for the judgment that a line at the foot of a page is a footer and not part of a sentence.

Assembly errors on a legal page:

• A sentence that crosses a page is interrupted. The footer, the page number and the next page's header sit in the middle of an indemnity clause.
• Paragraphs fuse or shatter. Two numbered subsections become one block, or a signature block is run into the paragraph before it.
• A page drops out. If one page fails and the rest are stitched together, every later page is one position early. Count the pages in the output against the pages in the PDF.

The post on scanned PDF to text has the practical checks for line breaks, hyphens and page boundaries in extracted text. One further risk belongs to model-based readers, which can write fluent words that are not on the page. The post on OCR hallucination deals with that.

Writing the output: what each file format can carry

The last stage writes what was recognized into the format you asked for, and a format can only hold what it has room for. Plain text has no place for a table grid or bold type. A searchable PDF keeps the page picture and hides the text. The same recognized page therefore looks different in each file.

Engines themselves offer several outputs. Tesseract's command-line usage guide says "The default output format is text", and that its PDF option "creates a pdf with the image and a separate searchable text layer with the recognized text." In the first case you see every recognized character. In the second you see the scan, and a wrong character stays out of sight until you search or copy.

For checking a result, the useful question about any format is how it marks the start of a new page. Without that mark you cannot trace a passage to the page it came from. Five of Schema Law's exports mark a page boundary like this:

The searchable PDF is described on the homepage's list of export formats as "Your original file with a hidden text layer, so search finds the page". All eight formats are available on every plan, the free one included.

Output failures are losses, not misreadings. A table that was recognized correctly turns into rows of loose words in a format with no cells. Emphasis vanishes. A section sign displays as stray symbols when another program opens the file under the wrong encoding. If a passage looks wrong in one export, open another export of the same document before you conclude that the page was misread.

One lease page followed through all six stages

Take page 14 of a fictional lease between Quillfeather Mews LLC and Orvanne Dental Group, scanned and later filed in court. The page carries a typed e-filing header, a rent clause, a footnote in small type, a RECEIVED stamp across one line, and a footer reading "Lease, page 14".

1. Text check. The filing header was added by software, so the page holds one line of real characters. A tool that treats any text as enough returns only the header for this page. A tool that reads the picture goes on to stage 2.
2. Rendering. The page is drawn as an image. If the picture is coarse, the rent clause survives and the footnote, which says the escalation applies "only after the third Lease Year", comes out as fragments.
3. Regions and order. The page should yield four blocks in sequence: header, clause, footnote, footer. If the stamp is merged into the clause block, the line beneath the stamp is at risk. If the footnote is numbered ahead of the clause's final lines, it lands inside the clause.
4. Recognition. The monthly rent of $8,450.00 sits partly under the stamp's border. This is where a digit can change.
5. Assembly. The clause continues on page 15. In the joined text, "Tenant shall pay" is followed by the footer, the page break and the next header before the sentence resumes.
6. Output. In plain text the footnote is an ordinary line with nothing to show it is a footnote. In the searchable PDF the page looks exactly as it did, whatever was recognized.

Six stages give six different defects on one page, and only the fourth is a wrong character. A proofreader hunting for typos would catch that one and could miss the other five.

Which stage should you suspect when extracted text looks wrong?

Judge by the size and shape of the problem. A fault that covers a whole page points to the text check or rendering. Correct words in a wrong sequence point to regions. A wrong character points to recognition. Misplaced breaks point to assembly. A difference between two exports points to output.

Work from the largest unit down:

1. Count pages. The output should have as many pages as the PDF, in the same order.
2. Scan for thin pages. A page with far less text than its neighbors was skipped or drawn badly.
3. Compare small type with body type. If only the small type is poor, rendering is the likely cause, and a better source file may fix it.
4. Read one complex page for order. Pick a page with columns, a table or footnotes and follow the text against the image.
5. Check names, dates and amounts. Wrong characters do the most harm in those three places.
6. Look at page joins. Read the last line of one page and the first of the next.

The stage tells you how far the fault spreads. A text-check or rendering fault will repeat on every page of the same kind, so one bad exhibit suggests the other exhibits need a look. A stamp over one figure is local. The full method for comparing output with the source is in the post on proofreading OCR output against the original.

No tool removes the need for this. Schema Law says as much under the questions partners ask first: "Nothing marks a passage as uncertain, so check names, dates, and amounts against the original." The free plan reads one document a day of up to three pages, which is enough to put your hardest page through and run the six checks. The pricing page lists the allowances for longer files.