Types of Load Files: DAT, OPT, LFP and DII Compared

The types of load files a production request can name include the DAT, the OPT, the LFP and the DII. A DAT carries one row of fields per document. An OPT and an LFP each list page images and mark document breaks. A DII can carry fields, image references and a text flag in a single file.

All four are plain text, and each is tied by name to a review product: the DAT to Concordance, the OPT to Opticon, the LFP to IPRO and the DII to Summation. This post compares them from documents that can be opened and read: a federal agency's production standard and the published help pages of two software makers that write these files. Neither maker is the owner of the LFP or DII format, and both tell the reader to consult the vendor for the full specification. Your recipient's own specification controls wherever it differs from anything here.

What types of load files will a production request name?

A request can name a data load file, an image load file, or one of each. The names that appear in the sources read for this post are DAT, OPT (also written LOG), LFP, DII, CSV and EDRM XML. Which of them your request names depends on the review software the recipient uses.

Here is where each name appears.

The ZyLAB page on defining a load file gives a plain definition of the whole category: "A load file organizes and allows access to the data that has been produced. It is used by the receiving party to import the produced data into their own database." It adds that "There are various load file formats depending on the software application. Some of the common load files include .DII (Summation) and .OPT/.DAT (Opticon/Concordance)."

Two cautions about names.

The extension is a label, not a guarantee. The SEC accepts its image cross-reference file as ".LOG or .OPT". Those are two extensions for one layout. In the other direction, .dat and .log are generic extensions that many unrelated programs use. Identify a file by its contents.

A request may describe the file without naming it. "Image cross-reference file", "image pointer file", "data file" and "metadata load file" are descriptions of roles. The SEC standard uses three of those phrases for two files. Match the description to the role before you match it to an extension.

If the request says "Concordance load file format" and nothing more, treat that as a DAT with the Concordance delimiters and ask whether an image file is wanted with it. The anatomy of that pair is in the post on what is inside a DAT file.

Data load files versus image load files: which does which job?

A data load file tells the review database what each document is. An image load file tells the viewer which picture files make up each document and in what order. The DAT is the data file in a Concordance-style delivery and the OPT is its image file. The DII is described as doing both jobs in one file.

The SEC standard states the two roles in one sentence each:

• "The data file (.DAT) contains all of the fielded information that will be loaded into the database."
• "The image cross-reference file (.LOG or .OPT) links the images to the database records."

It then requires exactly one of each: "All load-ready collections should include only one data load file and one image pointer file."

The unit of each file is different, and that is the quickest way to keep them straight.

ZyLAB's page describes the Opticon file in the same terms: "The Opticon load file is a page level load file, with each line representing one image." It says the DAT is produced separately for the database side.

Where the combined file fits

The DII does not divide the work this way. ZyLAB's Summation page says: "To load images and data into a Summation database, a .DII (Document Image Information) file is used." Nuix's appendix describes its own Summation export as "a single DII, containing metadata, file and full-text references." One file, read by one program, carries what a DAT and an OPT carry between them.

Pick the load file type the review tool expects for this production. A DAT holds one row per document, and an OPT holds one row per page image. A DII can carry both jobs for a Summation load. The OCR text travels as a path or as a field, and that text makes the electronically stored information searchable after import. Send the file the specification names.

Why the boundary markers matter

Both kinds of file record where one document stops and the next starts. The data file does it with a first and last page number on each row. The image file does it with a flag on the first page of each document. If the two disagree, nobody loading the set can tell which one is right. Deciding those boundaries for a paper set is its own task, covered in the post on document unitization.

IPRO LFP and Summation DII files beside the Opticon OPT

The LFP is an image load file that does the Opticon OPT's job in a different syntax, with a two-letter command at the start of each line. The DII is a token file in which lines beginning with @ create a record, fill its fields and name its image files. Both are plain text.

The descriptions in this section come from ZyLAB's and Nuix's help pages for their own products, which write these formats. They are not IPRO's or Summation's specifications. ZyLAB closes both of its pages by telling the reader to consult the vendor for more.

The IPRO LFP

ZyLAB's page says "An IPRO load file (LFP) instructs IPRO View how to display a project's images." It describes the structure this way: "Each record (or line) in the LFP file begins with a 2-letter code that determines the action that line will perform. Commas separate each part of the record."

Three lines from the example file on that page:

IM,MSC-00007,D,0,@MDEMO;IMAGES\00\00;007.TIF;2,0
IM,MSC-00008,,0,@MDEMO;IMAGES\00\00;008.TIF;2,0
IM,MSC-00010,D,0,@MDEMO;IMAGES\00\00;010.TIF;2,0

The page breaks a line of this kind into parts:

So MSC-00007 carries the document boundary flag, and MSC-00008, whose boundary position is empty, does not. The example file on the page also shows S and C in the boundary position. The page does not define them, and it does not explain the numbers after the file name. Ask the recipient what it expects in those positions before you write an LFP by hand.

The page lists other things an LFP can hold besides page lines, including "The position, size, and color of redactions or highlights" and "Any embedded text or sticky notes". It also says "The sequence of the records in the file does not matter." A recipient may still want the lines in Bates order, so ask.

The Summation DII

ZyLAB's Summation page shows an example file. The first record in it reads:

@FULLTEXT DOC
;Record 1
@T AA000001
@C DOCTYPE Fax
@C PGCOUNT 3
@D @I
AA000001.TIF

The page explains each token:

The page adds that "The last line(s) of each DII record is always the image filename(s)." Nuix's appendix lists more tokens that its export writes, among them @DOCID, @PARENTID, @O, @L and @ATTACH, and it labels its file "A Class I DII load file". Neither page gives the complete token list. ZyLAB says so directly: "For more information and a complete, up-to-date list of DII tokens, consult your vendor."

Side by side

How does each load file type point to the OCR text?

The DAT points to the text with a column that holds a file path. The OPT does not refer to text at all. The DII signals text with a @FULLTEXT line. For the LFP, the pages read for this post do not say how a document-level text file is referenced, so ask the recipient.

For scanned paper the text in question is OCR text. A production of scans has no other words to offer, so every load file question about "extracted text" or "full text" is a question about your OCR output.

What the DAT route requires of the text files

The SEC standard is specific, and its instructions show what "pointing to the text" means in practice:

• "Text must be produced as separate document-level text files, not as fields within the .DAT file."
• "The text file must be named after the FIRSTBATES."
• Text files "may be in either ANSI or Unicode format, however, ALL text files must be in the same format within the same production."
• "Text files must be in a separate folder, and the number of text files per folder should not exceed 2,000 files."

Other recipients set different rules, including whether text may sit inside the DAT itself. Read yours.

What the DII route leaves open

The two pages tie the @FULLTEXT line to text that belongs to each record. Neither page says what that file must be named or which folder it must sit in. The ZyLAB example has no path on the line. Before you deliver a DII with OCR text, ask the recipient two questions: what name must each text file carry, and where must it be relative to the images?

What this means for "OCR for Concordance"

A request for OCR for Concordance is, in file terms, a request for one text file per document and a DAT column that points to each one. The OCR step produces the text files. The load file step names them, places them and writes the paths. If you convert a scanned set in Schema Law, the plain text export gives you the words of each document you uploaded as a text file. Schema Law does not write load files on any plan, as the pricing page shows, so the DAT, OPT, LFP or DII is something you or a vendor build around the text.

Whichever route applies, the text file and the load file meet at the document identifier. A text file named for a Bates number that is not the first page of a document will not attach to anything.

Which load file type should a scanned paper production use?

Use the type the request, the ESI protocol or the court's order names. If nothing names one, ask the recipient which review software will load the production and deliver what that software reads. Do not choose a format because your own tools write it most easily.

The decision runs in this order.

1. Read the request and any protocol for a named format. A phrase such as "Concordance DAT and Opticon OPT", "IPRO LFP" or "Summation DII" settles it. The post on custom specifications shows how to go about reading a production specification field by field.
2. If no format is named, ask in writing. One email asking what platform the production will be loaded into, and what load files that platform's operator wants, costs less than a rejected delivery.
3. If the recipient has no preference, propose the pair an agency standard describes. The SEC standard states a preference that "scanned paper and electronic file collections" be produced with "industry standard load files, Bates numbered image files, native files and searchable document-level text files", and the load files it then specifies are a DAT and an OPT or LOG. That is one agency's standard. It is a reasonable starting proposal, and the other side can decline it.
4. For a small paper set, ask whether load files are needed at all. The SEC standard has a separate section under which, "With prior approval, Adobe PDF files may be produced in native file format", on conditions that include "All PDFs must be unitized at the document level, i.e., each PDF must represent a discrete document" and that the PDFs "contain embedded text". Your recipient may agree to the same. The comparison of TIFF and PDF for document production covers that choice.

Matching the type to what a paper set can fill

A scanned document has few fields. It has a Bates range, a page count, a custodian or source, and OCR text. It has no file name, no author metadata and no sent date unless someone reads those off the page and types them.

A worked example

Fictional facts: in Castellan Foods v. Mirren Logistics, the defendant will produce 1,900 scanned pages of driver logs and bills of lading, about 310 documents. The plaintiff's request says "produce with standard load files" and names no format.

Defense counsel's paralegal writes to ask which platform will receive the set. The answer is that the plaintiff's vendor loads Concordance-format deliveries. The paralegal confirms in a short letter: single-page TIFF images named by Bates number, a DAT with first Bates, last Bates, page count, custodian and text path, an OPT, and one OCR text file per document named for its first Bates number. Had the answer been "our reviewers use Summation", the same letter would have specified a DII and asked how the text files should be named.

The point of the exchange is that the format was fixed before any file was built.

Telling load file types apart by opening them in a text editor

Open the file in a plain text editor and read the first three lines. A DAT starts with a row of field names wrapped in an unusual character. An OPT is short comma-separated lines ending in mostly empty positions. An LFP line starts with a two-letter code such as IM. A DII has lines that start with @.

Work on a copy. Some editors change the character encoding or line endings when they save, and a load file altered that way may no longer import. Open, look and close without saving.

The sample lines in the table are illustrations built from the layouts described in the SEC, ZyLAB and Nuix documents, with fictional Bates numbers.

When the file looks like none of these

• It shows as blocks, boxes or question marks. The editor may be guessing the wrong encoding. Try opening it as UTF-8 and then as a Western single-byte encoding. The SEC standard says a Unicode DAT "must contain the byte order marker", which is how software tells the two apart.
• Everything is on one line. The line endings may be a kind your editor does not show. Open the file in a different editor before you conclude it is broken.
• The extension is .txt. Read the contents. A DII or an OPT saved with a .txt extension is still a DII or an OPT.
• There are two files with the same base name. That is the data and image pair. Identify each separately.
• It is not text at all. A file that opens as unreadable binary data is not one of the four types compared in this post. Stop and ask the sender what it is and which program writes it.

Three checks once you know the type

1. Count. In an image file, the number of page lines should equal the number of image files delivered. In a data file, the number of rows less the header should equal the number of documents.
2. Follow one path. Take the image path from one line and find that file in the delivery folders. Then do the same for one text path.
3. Match one boundary. Pick a document, read its first and last page from the data file, and confirm the image file marks a new document on that first page and on the page after the last.

A load file that passes those three checks on a sample of documents is at least internally consistent. Whether it loads is for the recipient's platform to say, so send a small test delivery first when the schedule allows. If you are still at the stage of turning scans into text, the formats section of the homepage lists what Schema Law exports for each uploaded document.