Custom Load File OCR: Matching a Production Specification
By Schema LawPublished Updated
A custom load file is one built to a recipient's own specification: its field names, column order, delimiters, text format and folder layout. To match it from scanned documents, read the specification line by line, map what OCR and the page can supply to each requested field, and settle in writing what happens to fields paper cannot fill.
Nothing about the task is technically deep. It is exacting, because an importer on the other end compares your file with a template and rejects what does not match. The examples below come from one agency's published standards and two review platforms' published upload requirements. Yours will differ, which is the point of the word custom.
What makes a load file custom?
A load file is custom when the recipient, not the producer, decides its layout. The usual pair of data file and image cross-reference file is still there. What changes is everything inside: which fields appear, what they are called, how they are ordered and separated, and how text and images are referenced.
There is no single standard to depart from, only a common habit. The habit is a delimited data file with one row per document plus a page-level image file, described in the post on the Concordance load file format. A specification then pins down the points that habit leaves open.
| Point | A common default | What a specification may ask instead |
|---|---|---|
| Field names | BEGBATES, ENDBATES | FIRSTBATES, LASTBATES, or a control number |
| Field order | Whatever the producing tool writes | A fixed sequence |
| Delimiters | Control characters | Commas, pipes, or the recipient's own choice |
| Text | A path column to text files | Text files found by name alone, with no path column |
| Text encoding | Left to the producer | One encoding for the whole production |
| Dates | Any consistent format | One format, with time in a separate field |
| File and folder names | Free | Restricted characters, a cap on files per folder |
| Images | Single-page TIFF | PDF, JPEG for color, or multi-page files |
Every row of that table is a place where a production built to "the standard format" can fail to load at one particular recipient.
Why would a recipient bother? Because a fixed template lets its technician map the columns once for every producer. A production in some other layout means manual work on the recipient's side, and you may be asked to resend it.
Where do custom load file requirements come from?
They come from three places: an ESI protocol or discovery order in the case, a government agency's published production standards, and the technical requirements of the platform that will receive the data. Sometimes two apply at once, and they do not always agree.
The case itself
In federal civil litigation, the rules let the requesting side name the form. Rule 34(b)(1)(C) of the Federal Rules of Civil Procedure says a request "may specify the form or forms in which electronically stored information is to be produced." Under Rule 34(b)(2)(D) the response "may state an objection to a requested form", and if it objects, or if no form was named, the responding party "must state the form or forms it intends to use." How those provisions apply to a set that began as paper is discussed in the section on whether you can require a searchable production.
The form is also on the list for the parties' early planning conference. Rule 26(f)(3)(C) requires the discovery plan to give the parties' views on issues about electronically stored information, "including the form or forms in which it should be produced". The result may be a negotiated ESI protocol with a field table attached. Once a court enters it, that table is your specification. State courts and arbitration bodies have their own rules, so check which apply to you.
An agency
An agency may publish its own standards. The SEC's Data Delivery Standards open with the instruction that "any questions or proposed file formats other than those described below must be discussed with the legal and technical staff of the SEC Division of Enforcement prior to submission." The document then sets image formats, a delimited data file, text rules and an addendum of metadata fields.
A platform
When the recipient gives you no document but names its review platform, the platform's import requirements become the working specification. Vendors publish these. Everlaw's standard format for processed data says "the load file should not contain the document text embedded as a field" and accepts one text file per document. The file is found through a text path in the load file or, where no path is given, by a file name that exactly matches the document's beginning Bates number or ID. Logikcull's production upload requirements say the metadata load file "must have a header row with unique/clear field names" and list the encodings it accepts. Those pages describe what each vendor's importer takes. They are not an agreement between you and the other side, so confirm with the recipient that the platform's defaults are what it wants.
Reading a production specification field by field
Read the specification with a pen, and turn it into a checklist with one line per requirement. Go through it in a fixed order: unit of production, images, data file, each field, text, naming, delivery. Mark each line as met, not applicable to paper, or a question for the recipient.
A specification is written for every kind of data the recipient might receive. Much of it may concern email and native files. Your job with a paper set is to find the lines that still bind you.
The seven passes
- Unit. What counts as a document, and how are families kept together? Paper has staples and folders instead of attachments.
- Images. Format, resolution, color handling, single-page or multi-page, endorsement position. The choice between page images and PDFs has its own post on TIFF versus PDF for document production.
- Data file. Extension, delimiters, encoding, header row, how many files.
- Fields. Name, order, format and whether each is required. This pass takes longest.
- Text. In the data file or in separate files, per document or per page, encoding, naming, what to supply for redacted pages.
- Naming and folders. Permitted characters, files per folder, folder names.
- Delivery. Media, encryption, cover letter contents, where passwords go.
An example reading
Here is how a few lines of the SEC's standards turn into checklist entries for a production of scanned paper.
| The standard says | What it means for a paper set |
|---|---|
| "All load-ready collections should include only one data load file and one image pointer file." | Do not send a data file per box. Combine them. |
| Text files "may be in either ANSI or Unicode format, however, ALL text files must be in the same format within the same production." | Check the encoding of every OCR text file, not a sample. |
| "For redacted documents, provide the full text for the redacted version." | Run OCR again on the redacted image and send that text. |
| "The time zone must be included in all time fields" | Paper has no time fields. Mark as not applicable, and confirm. |
| A cover letter must include "a list of custodians, identifying the Bates range for each custodian". | Track the custodian while scanning, or you cannot write this. |
Note what the last row implies. Requirements hide outside the field table. A specification's general instructions can contain duties, such as a cover letter or a naming rule, that are as binding as any column.
Read the specification as the instruction sheet for this discovery production. A protective order may name a confidentiality legend, and that legend has to sit in the field the recipient listed. Copy the field name character for character. Scanned paper becomes electronically stored information when you export it, and the other side's tool expects the columns you filled.
When a line is unclear, ask. A short email, "Your specification lists a FILE_NAME field. Our documents are scanned paper with no original file names. Should the field be empty or omitted?", costs nothing and produces a written answer you can keep.
Mapping OCR output to a requested field list
Go down the requested fields and assign each a source: your numbering log, the scan, the OCR text, a person reading the page, or nothing. OCR itself fills few columns directly. It supplies the text file every row points to, and it gives a reader fast access to the facts a coded field needs.
| Requested field | Source for scanned paper |
|---|---|
| First and last Bates number | Your stamping log, checked against the stamp read on the page |
| Page count | The PDF, or a count of the pages in a JSON export |
| Custodian | Collection notes: whose office or file the paper came from |
| Text path | The folder you build from OCR text exports |
| Attachment range | Physical grouping noted at scanning |
| Document date | A person reading the page, if the parties agreed to coding |
| Author, recipient, title | A person reading the page, if agreed |
| Confidentiality designation | The reviewing lawyer |
| Redaction flag | The redaction log |
What OCR contributes
The text file. One per document, named as the specification says. Schema Law's plain text export gives one text file for each document you upload, and you rename it to the identifier.
A check on numbering. Schema Law's CSV export, for a document without tables, lists the text line by line with a page number. Filter for your Bates prefix and you can see which stamp was read on which page, and compare that with the log.
Speed for coding. If the parties agreed that someone will type in a date and author for each document, searchable text lets that person jump to the letterhead and signature block. The typing is still human work, and the typed value is a judgment about the document, not something read from a file system.
Schema Law's part ends there. Its exports are Word, Excel, CSV, JSON, searchable PDF, plain text, Markdown and HTML, shown under export formats on the home page. None is a load file in any recipient's layout. You upload a PDF, PNG or JPEG of up to 20 MB, and what comes back is text and tables for you to arrange.
Keep a mapping sheet
Write the mapping down as a table with the specification's field names in the first column, in the specification's order. Add the source, the format and a sample value. Send it to the recipient with your first test delivery. A recipient who has seen and accepted the mapping has less reason to reject the full production over it.
Which requested fields can a scanned document never fill?
A scanned document cannot fill any field that describes a computer file or an email message, because the paper never was one. File names, file paths, sizes, hash values of an original, created and modified dates, senders and recipients taken from message headers: none of these exist for a page that came out of a filing cabinet.
The SEC's Addendum A makes the pattern visible, because it defines each field separately for email and for native files. Paper appears in neither column.
| Field in the SEC's list | Defined there as | For scanned paper |
|---|---|---|
| FILE_NAME | "Name of the original native file, including extension" | Nothing to enter |
| PATH | Where the native file was stored | Nothing to enter |
| DATE_CREATED, DATE_MOD | When the native file was created and last modified | Nothing to enter |
| LAST_AUTHOR | Last author of the native file | Nothing to enter |
| INTMSGID | An email's unique message ID | Nothing to enter |
| DATE_SENT | Date the email was sent | Nothing to enter, though the page may show a date |
| FROM, TO, CC | Email sender and recipients, or a native file's author | Only if a person codes them from the page |
| FILE_SIZE | Size of the native file or email | Nothing to enter |
| MD5HASH | Hash value of the document | No original file to hash |
Three mistakes to avoid here.
Using the scan's own properties. The PDF your scanner made has a creation date, a file name and a size. They describe the scanning, last Tuesday, by a paralegal. Putting them in DATE_CREATED tells the recipient that a 2014 letter was created last Tuesday.
Promoting a guess to metadata. A date typed on a letter is evidence of when it was written. It is not a system-recorded sent date. If the recipient wants page dates, agree a separately named coded field for them, so nobody mistakes them for metadata.
Dropping the column. An importer that expects 30 columns may fail on 22. Unless the recipient says otherwise, keep every requested column in place and leave the value empty.
Empty is an honest answer. What matters is that the recipient knows in advance which fields will be empty and why, which leads to the last section.
Agreeing a simpler layout for a paper-only production
When a production is all scanned paper, ask the recipient to accept a short field list in place of the full one, and get the answer in writing before you build anything. Much of a specification written for email and native files does not apply to paper, and a recipient may prefer a reduced layout to thirty blank columns.
The federal rules leave room for the conversation: the form of production is a subject for the parties' planning conference, and a responding party may object to a requested form and state the one it intends to use. Whether and how to do either is a decision for the lawyer on the matter. Courts and agencies vary in how much they will bend.
What to propose
A clear proposal has five parts.
- The reason. "The production consists of approximately 2,400 pages of scanned paper from two custodians. No email or native files are included."
- The fields you will provide. First Bates, last Bates, page count, custodian, text path, and any attachment range.
- The fields you will leave empty or omit, listed by the specification's own names.
- The text. Document-level OCR text files, made from the images as produced.
- A sample. Ten documents in the proposed layout, sent first so their technician can confirm it loads.
A worked example
Calloway Tile is producing three boxes of paper invoices to a state agency whose standards list 34 metadata fields. Its lawyers write to the agency's contact, attach the mapping sheet, and propose six fields plus OCR text. The agency replies that it wants all 34 columns present, in order, with the 28 inapplicable ones blank, and asks that a document date be coded from each invoice into a new field called DOCDATE. That is more typing than Calloway hoped for and far less than a rejected production. The email chain goes in the file as the agreed specification.
If the answer is no
Then build what was asked. For a few hundred documents, the hand method in the post on creating a load file from scanned PDFs still works with a longer header row. Beyond that, a vendor or a tool that writes the layout for you is safer, and the section on having the load file produced for you sets out the routes. Schema Law is not one of them: it does not produce load files or custom layouts on any plan, and the pricing page shows what each plan includes. Bring the written specification and the mapping sheet to whoever builds the file, because the layout is defined by them.