Table OCR: How Software Reads Rows, Columns and Cells

Table OCR is recognition that reads a table on a scanned page as a table. It finds the table, works out which rows and columns divide it, and assigns each piece of text to one cell. The software learns where one cell ends from ruled lines, from white space, and from how the text lines up.

Plain OCR would return the same characters as a run of words and numbers. Table OCR returns them with their places in a grid, which is what a spreadsheet, a sum or a comparison needs.

What is table OCR?

Table OCR is the part of document recognition that recovers a table's structure as well as its text. It involves two separate jobs. Detection decides that a region of the page is a table. Structure recognition decides how that region divides into rows, columns and cells. Reading the characters inside each cell is ordinary OCR.

The character-reading step is the same one described in how OCR works; this post is about the structure around it.

The two jobs fail differently, so it helps to keep them apart.

Detection is covered in the section on how OCR finds a table on a scanned page. This post is about the second job and the words used to describe its result.

Not every OCR tool does it

Table structure is an extra capability, and a tool that reads text well may not attempt it. The Tesseract project's guide to improving the quality of the output says plainly: "It is known tesseract has a problem to recognize text/data from tables" without custom segmentation or layout analysis. Google's help page on how to convert PDF and photo files to text in Drive says of the converted file that "Lists, tables, columns, footnotes, and endnotes are not likely to be detected."

So the first question about any tool is whether it claims to return tables at all. The second is what it means by a table, which is where the vocabulary comes in.

Cells, header rows and spans: the vocabulary of a recognized table

A recognized table is described as a grid of cells. Each cell has a row position, a column position and its text. Some cells are marked as headers. A cell that covers more than one row or column has a span. Those few terms are enough to read any tool's description of its table output.

The HTML standard gives the underlying model in its chapter on tabular data: "Tables have rows, columns, and cells given by their descendants. The rows and columns form a grid; a table's cells must completely cover that grid without overlap."

The terms

The HTML standard defines the two kinds of cell in one line each: "The td element represents a data cell in a table" and "The th element represents a header cell in a table." It defines spans as attributes of a cell: colspan is the "Number of columns that the cell is to span", and rowspan the number of rows.

Recognition services use the same ideas under their own names. Microsoft's documentation for its Document Intelligence layout model says: "Extracted table information includes the number of columns and rows, row span, and column span", and that each cell is output "along with information whether the area is recognized as columnHeader or not." Amazon's developer guide page on tables in Amazon Textract lists what that service returns as "cells, merged cells, column headers, titles, section titles, footers, table type (structured or semistructured), and summary cells within a table."

The terms on one table

Here is a rent schedule from a fictional lease between Pruett Holdings LLC and the Lark and Vane Bakery.

It has four columns and four rows. The first row is the header row. The cells under "Lease year" act as row headers. The "Notes" cell for year 2 is an empty cell, and it still occupies a grid position. On the printed lease, "Base rent" might be a single merged cell spanning the two middle columns, with "Monthly" and "Annual" in a second header row beneath it. That one design choice changes the recognized structure from a simple grid to a grid with a span and a two-level header.

When a tool returns this table, check it in those terms. Is the row count right? Is the column count right? Did the header row come back as headers or as the first data row? Is the empty cell still there, or did "Option year" slide up into its place?

Ruled grids versus tables held together by white space

A ruled grid is a table whose cells are boxed by printed lines. A borderless table has no lines, and its columns exist only because the text is aligned and separated by gaps. In the first kind the page states where each cell ends. In the second the software has to infer it.

What a ruled grid tells the software

Lines are direct evidence. A horizontal rule is a row boundary. A vertical rule is a column boundary. Where two rules cross, a cell corner exists. Where a rule stops short, two cells have been merged. An empty box is still a box, so an empty cell is unambiguous.

Rules bring their own hazards on a scanned page. A faint or broken rule may be missed, which joins two cells. A rule printed close to the text may be read as part of it: an underscore, a lowercase l or the digit 1 at the edge of a cell. A skewed scan turns straight rules into shallow stair-steps.

What a borderless table leaves to inference

Without rules, the evidence is alignment and space. Text that starts at the same horizontal position line after line suggests a column edge. A band of white running down the page suggests a gap between columns. Each line of text suggests a row.

Every one of those inferences can go wrong:

• Empty cells have no marker. In a ruled grid a blank is a box. In a borderless table a blank is more white space, and the entry from the next column can be assigned to the wrong one.
• Alignment differs by column. Names are aligned left and amounts are aligned right. The software has to accept both as column evidence.
• Gaps vary. A long entry narrows the gap beside it. If the gap closes entirely on one row, two columns may be joined for that row or for the whole table.
• Rows are not marked. With no horizontal rules, an entry that runs onto a second line looks like a new row.

The kinds in between

Printed tables do not fall neatly into two groups. Some have horizontal rules only, as in a financial statement with a line under the headings and a double line over the total. Some replace rules with alternating shaded bands. Some box the header and leave the body open. Amazon's documentation reflects the range by giving each detected table a type, either STRUCTURED_TABLE or SEMI_STRUCTURED_TABLE.

For a reviewer, the practical point is to know which kind you are looking at before you read the output. On a fully ruled table, check for missed rules and stray characters at cell edges. On a borderless one, check empty cells and long entries first.

Why does a wrapped or merged cell break table recognition?

Wrapped and merged cells break the simple rule that one line of text is one row and one column position is one cell. A wrapped cell holds two or more lines of text that belong to a single entry. A merged cell covers several grid positions at once. Software that assumes a regular grid mislabels both.

Wrapped cells

Take a privilege log from a fictional matter, Hallam v. Brightwater Marine. The description column is narrow, so entries wrap:

Doc   Date         Author      Description
14    03/02/2023   R. Osei     Email to counsel re draft
                               indemnity language
15    03/05/2023   T. Lindqvist Memo re insurance

A person sees two documents. Software that treats each text line as a row may return three rows:

Every character is correct. The row count is wrong, and anything built on the row count is wrong with it: a tally of withheld documents, a sort by date, a filter on author. The stranded second line also has no document number, so after a sort it may end up nowhere near its first half.

Treat a medical bill and an invoice as grids you will recount. After the cells come back, look up the amount, the service date, and whichever party the claim names. A wrapped line that splits one invoice row in two leaves a total short of the obligation printed on the sheet. Count the rows on the paper.

In a ruled grid the horizontal rules can rescue the software, because both lines sit inside one box. In a borderless table nothing on the page says the second line is a continuation except the empty cells to its left.

Merged cells

A merged cell breaks the count in the other direction. Suppose a lease's rent schedule has four columns, and the heading "Base rent" is printed once over the two middle ones, "Monthly" and "Annual". The first header row then has three cells and the second has four. Failures to look for:

• The merged heading is assigned to one column only, and the other column loses its label.
• The two header rows are collapsed into one, giving headings like "Base rent Monthly" over one column and a bare "Annual" over the next.
• The merged heading is repeated in every column it covers. That is tidy for a spreadsheet and is not what the page shows.

Vertical merges cause a quieter problem. A party name printed once beside five rows of payments is a cell with a row span of five. Flattened into a plain grid, the name appears on the first row and the other four are blank in that column.

Services differ in how they report merges, which is worth knowing before you compare outputs. Amazon's documentation says that in its response "A merged cell is broken down into all the individual cells that are combined to make one merged cell", and that an ordinary cell block "will always have row span of 1 and column span of 1", with the merge recorded separately. Microsoft's layout documentation shows a header cell carrying its own columnSpan value. Both describe the same page. They store it differently.

What to check

1. Count the rows on the page, counting each wrapped entry once. Compare with the output.
2. Look at the row after any long entry for a line with mostly empty cells.
3. For every heading that sits over more than one column, confirm each column underneath still has a label.
4. For every label that sits beside more than one row, decide whether the blanks under it need filling before you sort.

How is a recognized table stored: cells, markup or delimited text?

A recognized table is stored in one of three ways. As a list of cells, each with its row, column, span and text. As markup, such as an HTML table. Or as delimited text, such as CSV, where separators mark the columns and line breaks mark the rows. Each keeps a different amount of the structure.

As a list of cells

A recognition service may return something like this for a rent schedule whose heading "Base rent" covers a "Monthly" and an "Annual" column. The field names here are illustrative and are not any product's format:

{ "row": 0, "column": 1, "columnSpan": 2, "kind": "header", "text": "Base rent" }
{ "row": 1, "column": 1, "kind": "header", "text": "Monthly" }
{ "row": 1, "column": 2, "kind": "header", "text": "Annual" }
{ "row": 2, "column": 1, "text": "$4,850.00" }

This form keeps everything: position in the grid, spans and header status. Some services add where the cell sits on the page. It is the form software reads, not people.

As markup

HTML writes the header of that rent schedule like this:

<tr><th rowspan="2">Lease year</th><th colspan="2">Base rent</th><th rowspan="2">Notes</th></tr>
<tr><th>Monthly</th><th>Annual</th></tr>

Markup keeps headers and spans and can be displayed directly. Microsoft's layout documentation notes that for one version of its service "the representation of tables is changed to HTML tables to enable rendering of items like merged cells and multirow headers." Simpler markup, such as a Markdown pipe table, keeps rows, columns and one header row, and has no way to express a span.

As delimited text

CSV writes one row per line with commas between fields. The rent schedule's headings and first year become:

Lease year,"Base rent, monthly","Base rent, annual",Notes
1,"$4,850.00","$58,200.00",Two months abated

The format's common definition, RFC 4180, says "Each line should contain the same number of fields throughout the file." There is no field for a span and no marker for a header beyond its position on the first line. The same document says "Fields containing line breaks (CRLF), double quotes, and commas should be enclosed in double-quotes", which is why amounts such as "$58,200.00" are quoted in the example. A CSV written without that quoting splits $58,200.00 into two fields.

What each form keeps

A spreadsheet file sits alongside these. It stores cells in a grid and can hold merged cells, though whether an export uses them is up to the tool.

Schema Law's homepage describes its table-bearing exports in the list of export formats. Excel gives "Each table on its own sheet, with the rest of the text on another". CSV gives "The document's tables, or its text line by line with page numbers". JSON gives "Each page's text and tables", and Markdown gives "Headings, lists, and tables as plain text". If spans or two-level headers matter in your table, export one page in the format you plan to use and look at how they came through.

Pages a lawyer calls a table that software reads as plain text

Lawyers use "table" for things that are not grids: a table of contents, a table of authorities, a caption, a pair of signature blocks. Software looks for rows and columns of cells. Where the page has aligned text with no repeating grid, it may return plain text, or force a grid that was never there.

None of these outcomes is a malfunction. A table of authorities is a list with a page reference on each line. Its dot leaders are decoration, and a row of periods is not a column boundary.

The opposite mistake

The reverse also happens, and it can do more damage. Ordinary text that happens to be aligned may be boxed into cells: a letterhead with the address on the left and the date on the right, or a closing with "Very truly yours" set beside a list of enclosures. A sentence chopped across three cells is harder to repair than a table that arrived as text.

What to do with each

• For a contents page, index or table of authorities, accept plain text. If you need the entries in a spreadsheet, the post on getting OCR into Excel explains how to turn recognized lines into columns.
• For a caption or signature block, read the output against the page and fix the order by hand. These are a few lines each.
• For a true table inside an exhibit, such as a ledger, invoice or schedule, see the post on OCR for exhibits with tables, which covers ledgers, invoices and schedules and how to reconcile an extracted table with the exhibit.
• For any page where you are unsure, ask one question: does every row have the same kinds of entry in the same positions? If yes, it is a table and should come back as one. If no, it is a list or a layout, and text is the right result.

Whatever the tool returns, the test of a recognized table is arithmetic and counting. Rows on the page against rows in the output, columns against columns, and a total recomputed from the cells against the total printed on the page.