Text in a Photograph: Can OCR Read Signs and Labels?
By Schema LawPublished Updated
Software can read text in a photograph of a sign, a label or a vehicle, but with poorer odds than on a scanned page. The words are small within the frame, tilted, unevenly lit and set against clutter. Crop a copy to the words, run recognition, and confirm every character against the picture.
A case file can hold photographs that were never meant as documents: an intersection after a collision, a shelf in a store, the side of a delivery truck, a notice taped to a door. The words in them can matter a great deal. This post is about what recognition can do with those words and how far to trust it.
Crop a working copy when the photograph may be demonstrative evidence, and leave the wide frame alone. You pull text from the tight crop. You keep the original for authentication of where the sign sat. If the words are on a notice you will keep as documentary evidence, save both with the date so the chain of custody covers the pair.
Can OCR read text in a photograph that is not a document?
OCR can read text in a photograph that is not a document when the letters are large enough in pixels, reasonably square to the camera and clear of their background. A bold street sign filling a third of the frame is an easy target. A serial number in the corner of a wide shot may be beyond any software.
OCR was designed around pages: dark type, light paper, straight lines, one direction. Text in a scene breaks those assumptions. Recognition tools differ in how they cope with that, and no description of a tool settles the question for your picture. The honest way to find out is to try your own photograph and read the result against it.
A quick sort before you spend time:
| The words in the photo are | Expect |
|---|---|
| Large, printed, facing the camera | A usable reading with a few errors to fix |
| Printed but small, angled or partly shaded | Fragments; some words right, some missing |
| Stylized lettering, logos, neon, hand-painted | Guesses; treat every character as unverified |
| Too small to read when you zoom in yourself | Nothing reliable; no software adds detail the camera did not record |
For a handful of words, the phone that holds the picture may be all you need, and the post on copying text in a photo covers the built-in tools. Uploading suits a batch of photographs or a need to keep the recognized text in a file.
How does text on a sign differ from text on a page?
Text on a sign differs from text on a page in four ways that all work against recognition: it is seen at an angle, from a distance, in display lettering, and against a busy background. A page scan controls all four. A scene photograph controls none of them.
- Angle. A page is photographed from straight over it. A sign is wherever it was hung, and the photographer stood wherever the pavement allowed. Letters on a surface turned away from the lens are squeezed on one side and stretched on the other.
- Distance. The words on a sign thirty feet away may occupy a strip a few dozen pixels tall. What counts is the height of each letter in pixels, and the section on how large the letters in an image must be for OCR gives the measure.
- Lettering. Signs use bold display faces, all capitals, wide spacing, outlines, shadows and vertical stacking. Words may curve around a logo. There is no baseline for software to follow from one line to the next.
- Background. Paper is blank. A sign sits among brick, foliage, sky, wires and other signs. Software has to find where the text is before it can read it, and a railing or a row of windows can be mistaken for letters.
Light adds a fifth problem outdoors. Reflective sign faces flare under flash or headlights. Lit signs at night blow out to solid white with a halo. Shade from a tree puts half a word in the dark.
Labels, license plates and screens in scene photographs
Labels, plates and screens are three kinds of scene text a case file can hold, and each fails in its own way. Labels are small and curved. Plates are short strings with no context to catch a wrong character. Screens add stripes and glare of their own.
Labels
Product labels, medication bottles, warning stickers, equipment rating plates and shipping labels carry the text a file turns on: a model, a lot or serial number, a date, a warning. They also sit on curved, glossy or stamped metal surfaces. A label wrapped around a bottle shows only part of its text in one view, and the visible part bends away at both sides. Stamped or engraved characters have no ink at all, only shadow, so they read differently as the light moves.
If the object still exists and you have access to it, a new close photograph of the label, square on and filling the frame, will do more than any processing of the old wide shot.
License plates and vehicle markings
A plate is a short string of characters with no dictionary behind it. On a page, software and reader alike can tell that "c0ntract" is wrong. Nothing tells either of them that a plate ending in 8 should have ended in B. Plate frames, dirt, motion blur and the angle of a parked car all cost characters. The same goes for unit numbers and registration numbers painted on commercial vehicles.
Screens inside a scene
A dashboard display, a point-of-sale terminal, a phone held up in a photo: each is a small picture of a screen inside a larger picture. All the faults described in the post on photographing a computer screen apply, at a fraction of the size.
Notices and posted paper
A notice taped to a door or a placard on a wall is the friendliest case, because it is a page that happens to be in a scene. Crop to the sheet and it behaves much like any phone photo of a document.
Cropping a photograph to the words before recognition
Cropping is the one preparation step worth doing every time. Make a copy of the photograph, cut it down to the sign or label with a small margin, and run recognition on the crop. The software then starts on the text instead of searching a whole street for it.
Do it this way:
- Duplicate first. Work on a copy. The original photograph stays exactly as received, with its file name and its embedded details intact.
- Crop loosely. Leave a border about one letter-height wide all round the words. A crop that touches the letters can cost the first or last character.
- One block of text per crop. If the frame holds a street sign, a shop front and a truck, make three crops. Mixed blocks come back as one jumbled paragraph.
- Rotate if the text is sideways or upside down. Turn the crop so the words read left to right.
- Do not enlarge. Blowing a crop up to twice its size adds no information. It only makes the same pixels bigger and softer.
- Go easy on enhancement. A modest lift in brightness on a shaded sign is reasonable. Heavy sharpening and contrast can turn a 3 into an 8 as easily as the reverse. Record anything you did.
- Save without recompressing hard. Export the crop as PNG, or as JPEG at the highest quality.
- Name the crop after its parent. Something like the original file name plus "crop-sign-1", so anyone can find the full picture it came from.
A crop is small, so upload size is not a concern. Schema Law accepts PNG and JPEG images as well as PDF and reads each one as a page of text you can view and export. The how it works section shows the steps. Whether a given sign or label comes through is something to test on the photograph in front of you.
Why should recognized scene text be treated as a lead, not a reading?
Recognized scene text is a lead because recognition returns its best guess with nothing to mark the guess, and a scene gives it more to guess about. A converted plate, serial number or street name tells you where to look. Only a person comparing each character with the image, and where possible with another source, establishes what the photograph shows.
Recognition is strongest on clean print and nothing marks a passage as uncertain, so a wrong character in a scene reading looks exactly like a right one. Software can also return a tidy, plausible string where the image holds only a smear. A partial plate "completed" by a tool is more dangerous than a blank.
A routine that keeps the reading honest:
- Read it yourself first. Zoom in on the original and write down what you can see, using a placeholder for any character you cannot make out.
- Compare. Set your reading beside the software's. Where they differ, look again. Where neither is sure, say so.
- Record ambiguity as ambiguity. "7 or 1", "O or 0", "last character not legible" are accurate statements. A clean string is not, if the image does not support it.
- Look for a second source. A registration record, a purchase invoice, the object itself, another photograph from a different angle. The recognized text is what tells you which record to ask for.
- Keep the chain visible. Original photograph, crop, recognized text and your note on how they relate should sit together in the matter file.
How a photograph and the words in it may be used in a proceeding is a question for the rules that apply to your matter, and this post does not answer it. The handling point is narrower: text lifted from a scene is a pointer into the picture, and the picture is what you go back to. The questions section of the homepage says the same about any converted page, where names, dates and amounts should be checked against the original.