Concordance Load File Format: DAT and OPT Explained

A Concordance load file is a pair of plain text files that tells a review platform how to assemble a production. The DAT file lists one document per row with its fields, such as the first and last Bates number. The OPT file lists one page image per line and marks where each document begins.

The format carries the name of a review database product, but it is asked for well beyond that product. The SEC's Data Delivery Standards require a DAT that uses the Concordance default delimiters, and Relativity's load file specifications use the same characters as defaults. Those two documents are the sources for the details below. Your own recipient's specification always wins where it differs.

What is inside a DAT file?

A DAT file is a delimited text file with one row per document and one column per field. The first row normally names the fields. Every later row gives one document's values: its first and last Bates number, page count, custodian, dates, and usually the path to a text file holding its words.

Here is a two-document DAT. The field separator is an invisible control character, so it is drawn as ¶ to make the structure readable:

þFIRSTBATESþ¶þLASTBATESþ¶þPGCOUNTþ¶þCUSTODIANþ¶þOCRPATHþ
þABC0000001þ¶þABC0000003þ¶þ3þ¶þSmith, Johnþ¶þTEXT/001/ABC0000001.txtþ
þABC0000004þ¶þABC0000005þ¶þ2þ¶þSmith, Johnþ¶þTEXT/001/ABC0000004.txtþ

Three points about that sample trip people up.

• The header row is expected. The SEC says the first line "must be a header row identifying the field names". Relativity does not require one but calls header rows "strongly recommended to ensure accuracy".
• Field names are not standardized. The sample uses the names from the SEC's field list. Another specification may call the same column BEGBATES or BEGDOC. Use the names your recipient publishes, in the order it asks for.
• The words of the document usually live outside the DAT. Relativity accepts either the text itself in a field or a path to a document-level text file. The SEC allows only the second: an OCRPATH column pointing at a text file named after the first Bates number, with the instruction "Do not include the text in the .DAT file."

One column must identify each document uniquely. Relativity states that "the identifier field is required for each load", and in a production that identifier is normally the first Bates number.

What does the OPT file add?

The OPT file ties page images to the documents listed in the DAT. It is a comma-separated text file with one line for every page image. Each line names the page, gives the path to its image file, and carries a Y when that page starts a new document.

The SEC describes it as "a comma-delimited file consisting of seven fields per line with a line in the cross-reference file for every image". The OPT for the two documents in the DAT sample looks like this:

ABC0000001,VOL001,IMAGES\001\ABC0000001.TIF,Y,,,3
ABC0000002,VOL001,IMAGES\001\ABC0000002.TIF,,,,
ABC0000003,VOL001,IMAGES\001\ABC0000003.TIF,,,,
ABC0000004,VOL001,IMAGES\001\ABC0000004.TIF,Y,,,2
ABC0000005,VOL001,IMAGES\001\ABC0000005.TIF,,,,

Read the two files together and the logic is plain. The DAT says a document runs from ABC0000001 to ABC0000003. The OPT says where the three page images are and that a new document starts at ABC0000004. The image ID on a Y line should match a first Bates number in the DAT.

Treat the first Bates number as the control number for that item of documentary evidence in the discovery production. The DAT row is what you search. The OPT row is the page image you open. When those two identifiers disagree, the text hit and the image are different documents. Fix a sample of the load before you import the rest.

The OPT assumes one image file per page. Relativity's specification lists single-page Group IV TIFs and single-page JPGs as the supported image formats. A set of whole-document PDFs is a different kind of delivery, so ask the recipient how it wants those referenced before you build anything.

Concordance delimiters and why they look strange

Concordance delimiters are characters picked from outside the range anyone types. By default, character 020 separates fields and character 254, which displays as þ, wraps each value. The likely reason is practical: commas and quotation marks fill names, addresses and document titles, so they make unreliable separators.

Look at the custodian value in the DAT sample, Smith, John. In an ordinary comma-separated file that comma would need special handling, and one mistake would push every later value a column to the right. A character that never appears in the text removes the problem.

The codes come from Relativity's list of defaults, and the SEC names the first two. Each record ends at a real line break, so a line break inside a value, such as a multi-line address, needs a stand-in. The SEC's field list also tells producers to separate several recipients in one field with a semicolon.

To see the characters, open the DAT in a plain text editor. Do not double-click it. The .dat extension is shared by many unrelated programs, and a spreadsheet will either refuse the file or show each row as one long cell.

Common mistakes that make a load file fail to import

Most failed imports come from a mismatch between the two text files and the folders they describe. The platform reads a path or a Bates number, looks for the matching file or record, and stops or skips when it is missing. Check these before you send or load anything.

1. No header row, or the wrong field names. The importer cannot map columns it cannot name.
2. Delimiters lost in editing. Saving the DAT from a spreadsheet as ordinary CSV replaces the 020 and 254 characters with commas and quotation marks.
3. Wrong encoding. The SEC requires a byte order mark when a DAT is produced in Unicode. Relativity's specification requires ANSI/Western European encoding in the Opticon file for image imports and says Unicode OPT files are not supported.
4. Paths that do not resolve. A path written for the drive the files were built on fails on another machine. Relativity accepts relative and absolute paths, but the file must exist at whichever path is written.
5. Text in the wrong place. Full text pasted into the DAT when the recipient wants separate document-level text files, or the reverse.
6. Counts that disagree. The number of OPT lines should equal the number of image files. The number of Y markers should equal the number of DAT records below the header.
7. Dates and times combined. The SEC wants dates as mm/dd/yyyy and requires that "date and time fields must be two separate fields".
8. Illegal file names. The SEC bars spaces and special characters, commas included, in image file and folder names. A comma in a path also breaks a comma-separated OPT line.

Schema Law does not write DAT or OPT files on any plan. It reads scanned PDFs into text and tables that you export as plain text, CSV, searchable PDF and other formats, and those exports supply the text files and page counts a load file points to. The post on how to create a load file from scanned PDFs walks through that build. Every plan exports the same eight formats, and the pricing page lists the page allowance for each. For where load files sit in the wider review process, see the overview of eDiscovery OCR.