Handwritten Text Recognition: How HTR Reads a Page

Handwritten text recognition, or HTR, is software that turns handwriting into typed characters by reading a whole line of writing at once, using a model trained on examples of handwriting with their transcriptions. OCR in its older sense matches separate printed characters. HTR was developed for script, where letters join and vary from writer to writer.

The term comes up in vendor material and in archive transcription projects. This post explains what happens between the page and the text, so that you can ask a vendor better questions and know which errors to look for. It uses the documentation of two systems as worked examples: Transkribus, a platform for transcribing historical documents, and Google's developer tools.

What is handwritten text recognition?

Handwritten text recognition is the conversion of handwriting into machine-readable text by a trained model. The model is shown many images of handwriting together with accurate transcriptions, and learns to output the most probable sequence of characters for writing it has not seen before.

That description follows how one HTR vendor puts it. Transkribus's help page on training text recognition models defines a text recognition model as an algorithm trained on images and transcriptions, "able to detect the most probable sequence of characters for each segmented text line."

Three words in that definition carry the meaning.

• Probable. The output is the model's best estimate. It is a ranking of possibilities with one chosen, not a lookup.
• Sequence. The model reads a run of writing, not one letter cut out from the rest.
• Line. The unit handed to the model is a line of text that some earlier step has found on the page.

The label itself is used loosely. Transkribus's page on choosing a model calls its engine, PyLaia, a "Handwritten Text Recognition engine". Google's Cloud Vision documentation describes detecting handwriting in images under the heading "Handwriting detection with Optical Character Recognition (OCR)". Both read handwriting from an image. One vendor calls it HTR and the other calls it OCR. So the name on a product page tells you little. What the software was trained on tells you more.

Online recognition of pen strokes versus offline recognition of a page image

Handwriting reaches software in two forms. Online recognition receives the pen's movements as they are made on a screen or tablet. Offline recognition receives a finished picture of a page, such as a scan. Scanned paper in a legal file is the second kind, and it gives software less to work with because the movement is gone.

Strokes

Google's developer documentation for ML Kit digital ink recognition shows what the stroke form of input looks like. It says the recognizer takes an "Ink" object, which it describes as "a sequence of strokes, each being a list of coordinates with time information". A stroke starts when the stylus or finger goes down and ends when it lifts. The page adds that digital ink recognition "works with the strokes the user draws on the screen", and that for text in images taken with a camera you should use a different tool, its Text Recognition API.

With strokes, the software knows the order in which the marks were made. It knows that the bar of a "t" was added after the word was finished. It knows where the pen lifted. A photograph of the same word records none of that.

A trap in the vocabulary

"Online" and "offline" here have nothing to do with the internet. The ML Kit page says its stroke-based recognition "works fully offline", meaning with no network connection. In the sense this section uses the words, that same tool is an online recognizer, because it reads pen movement. When a vendor says a handwriting feature is "offline", find out which meaning is intended. One is about the kind of input. The other is about whether your writing leaves the device.

What this means for a file

The last row catches people out. Notes that converted well inside the app where you wrote them can do worse once they have been exported as pictures and sent through a different tool, because the stroke information did not travel with them. If you want typed text from tablet notes, convert in the app that holds the strokes.

How does an HTR model learn one writer's hand or a whole period's?

An HTR model learns from pairs: an image of a line of handwriting and the correct text for that line. Given enough pairs from one writer, it learns that writer. Given a large and varied set from many writers of one time and place, it can read hands it has never seen, with less certainty.

Transkribus documents the quantities it advises, which makes the idea concrete. Its page on data preparation calls the paired images and transcriptions Ground Truth. It asks for between 5,000 and 15,000 words to start, and advises training on at least 10,000 words for each hand in handwritten material. It says models trained on more than 100,000 words "comprising many hands from the same period and region should be capable of recognising hands not seen in any way during the training", and adds that the results will probably be somewhat worse than the measured error rate.

Those figures are one vendor's guidance for its own engine. They are not a law of HTR. But the shape of the advice carries over to any model that learns from examples.

What the model learns from, it repeats

The same page warns that "any mistake in the Ground Truth will train the model to learn something wrong." It also explains that the model picks up the transcriber's conventions. If the training transcripts expand abbreviations or modernise spelling, and do so consistently, the model can learn to do the same.

That has a consequence for legal use. A model can be trained to write out "said" where the page has "sd", or a modern spelling where the page has an old one. The output then differs from the page by design. Before you quote from a transcript made this way, find out which conventions the model was trained with. Transkribus recommends that whoever trains a model record them in the model's details.

How performance is measured

Transkribus reports a Character Error Rate for each model. Its page on model setup and training explains that part of the Ground Truth is set aside as validation data, kept out of training and used to assess the model. The error rate is counted on those pages.

So a published error rate answers a narrow question: how did the model do on held-back pages from the same collection it learned from? Your documents are a different collection. The honest test is the same one the vendor uses, done on your own material: transcribe a page by hand, run the model on it, and compare.

One hand, or a period

• One writer. A single clerk's register, a physician's notes over twenty years, a diarist. A model can be fitted closely, and it will be poor on anyone else.
• A period and region. Nineteenth-century English court hands, for example. A broad model reads many writers passably and none perfectly.
• General-purpose models. Large models trained on print and handwriting of many kinds. You cannot inspect what they were trained on, so a test on your pages is the only evidence you have.

Line detection: why HTR finds the lines before it reads them

HTR systems that read line by line must first find the lines. A step called layout analysis marks where each line of writing runs across the page, and the recognizer then reads one line at a time. If that step misses a line or joins two, the text is wrong however good the reading model is.

Transkribus's page on automatic layout recognition describes the step as "the segmentation of the image into text regions and baselines". It defines the baseline as a line "running along the bottom of the handwritten text line", and calls it "the most important reference point for text recognition." On that platform layout recognition runs automatically when you start text recognition. The page recommends running it as a separate step when the page layout is complex, which gives you the chance to look at the detected lines before any text is read.

Why lines and not letters

Printed characters are separated by white space, so older engines could cut a line into letters and classify each one. Joined handwriting gives no dependable place to cut. Reading the whole line as a sequence avoids the cut. The post on whether OCR can read cursive looks at joined letters as a problem in its own right.

Where line detection goes wrong on legal pages

• Marginal notes. A note written sideways or squeezed beside a paragraph may be attached to the wrong line, or read in the middle of a sentence it does not belong to.
• Interlineations. Words inserted above a line with a caret can be read as a separate line, dropped, or merged into the line above.
• Sloping and curving lines. Writing on unruled paper that climbs toward the right edge can be split in two or joined to its neighbour.
• Ledgers and forms. Entries in columns may be read straight across as one line, mixing a date, a name and an amount from different entries.
• Writing in two directions. If a page has writing running across it and more written at a right angle over the top, do not expect a line finder to separate the two layers.

A check that takes one minute

Count the handwritten lines on the page, including insertions and marginal notes. Count the lines in the output. If the numbers differ, a line was lost, split or merged, and you know to look before you read a single word. Where a tool shows the detected lines over the image, as the Transkribus editor does, look at that overlay first.

How is HTR different from OCR and ICR in practice?

As the terms are commonly used, the three labels belong to three kinds of writing: OCR to machine print, ICR to hand-printed characters kept apart in form fields, and HTR to running handwriting read a line at a time. Current products blur the labels, so use them to ask what a tool was built and trained for.

The ICR column is a summary only. Where that term came from, and what it covers, is explained in the post on the difference between ICR and OCR.

The table describes origins, not product boundaries. As the Google and Transkribus pages show, one company files handwriting under OCR and another calls its engine HTR. Some systems no longer work line by line at all. They take in the whole page and write out the text, an approach covered in the section on what model-based systems do differently.

Questions that get past the label

1. Was the software trained on handwriting, or only on print?
2. On what kind of handwriting: modern notes, historical documents, form fields?
3. Does it need a language or script setting, and is yours offered?
4. Can you see which lines it found?
5. Does it mark doubtful words, or does everything come back looking equally certain?
6. Can you try it on your own pages before you pay?

On the fifth question, Schema Law's own answer is on its homepage: recognition is strongest on clean print and weaker on handwriting, and nothing marks a passage as uncertain. It reads uploaded pages into text and tables and offers them in the export formats listed on the homepage. On a handwritten page, treat what comes back as a draft and read it against the image.

Fluent but wrong: how a language model changes handwriting errors

A language model makes a recognizer prefer output that reads like real language. It can repair misread letters. It also changes the errors that remain: you get a real word that was not written, where you would once have got a visibly broken one. Those errors are harder to see.

Transkribus documents both sides. Among the advanced settings on its page on automatically transcribing documents is a language model option. The page says the impact "should be tested on a case-by-case basis" and that language models have been shown to enhance recognition in most instances, "but there may be cases where they do not." Its page on Super Models describes newer models that combine an optical part with "an extensive language model that tries to make sense of and improve the extracted text information."

An illustration

This example is made up to show the mechanism. It does not report a real case.

A handwritten note of a telephone call reads, in a hurried hand, "Mr Hale will not sign." The "n" of "not" is a single shallow curve.

• A recognizer with no language help might return "Mr Hale will uot sign". The broken word tells you where to look.
• A recognizer leaning on a language model might return "Mr Hale will now sign". Every word is English. The sentence is grammatical. It says the opposite.

Nothing in the second output invites a second look. The same mechanism can turn an unusual surname into a common one, or a rare drug name into a familiar word.

Compare the name, the date, any amount, and the words about a signature on a telephone note with the image. A fluent line can reverse whether someone will sign. Keep the scan as documentary evidence of the call, and record who checked it for authentication. Store it where you can state the chain of custody. Judge probative value after the small words match the ink.

Where language help is weakest

A language model helps most where the text is predictable and least where it matters to a lawyer:

• Names. A surname is not predictable from the words around it.
• Numbers. No language model knows whether the sum was 1,500 or 7,500.
• Dates. "3 March" and "8 March" are equally good English.
• Negatives and small words. "Not", "no", "now", "nor" are short, similar in script, and change the meaning.
• Abbreviations and terms of art. A term the model has seldom met may be swapped for a common word that resembles it.

Reading for this kind of error

Proofreading for sense will not find a fluent error, because the sentence makes sense. Read against the image instead, and read the unpredictable items one at a time: every name, every figure, every date, every negative. The section on new failure modes to watch for covers the same risk in model-based reading of printed documents.

Fluent output is a good reason to keep the scan beside any transcript you rely on, and to record who checked it and against what. The transcript is a reading. The page is the evidence.