oais-structure-kaitai notes

This module bridges Kaitai Struct onto the common StructureNode model. It uses the Kaitai Struct Java runtime io.kaitai:kaitai-struct-runtime:0.11 (MIT licence, from Maven Central), and was checked against classes generated by the Kaitai Struct compiler 0.11.

How a format is described

A Kaitai Struct description is a .ksy file (YAML): a seq of fields, each with an id and a type (s4, u1, f8, str, a user-defined type, ...), plus size, encoding, terminator, repeat (eos, expr, until), if, switch-on for variant records, computed instances, and endian at the top level. It is built for reverse-engineering arbitrary binary formats; the public format gallery (https://formats.kaitai.io) has hundreds of real specifications. See src/main/ksy for this module's two examples: a binary "point" record and a CSV file.

Unlike DFDL and DRB, a .ksy is not read at run time. The Kaitai Struct compiler (ksc) translates it ahead of time into source code - here, a Java class - and that class is what parses the bytes. So:

• KaitaiFormatSpecification names a generated class (new KaitaiFormatSpecification(Point2d.class)), not a .ksy file.
• The generated classes must be compiled into the application. This module's are checked in under src/main/java/.../generated/, so its build needs no compiler.
• archive-manager's RepInfo Tools generates a .ksy from its engine-neutral description (one type per record and choice branch, repeat: expr/eos, if, switch-on, text fields with terminator; and, when Kaitai is among the chosen languages, bit fields bN, size: records, process: zlib and instances with pos: for elements at an offset). It can test it against a sample file: it bundles the Kaitai Struct compiler, runs it as a separate Java process, compiles the generated Java in-process and loads it (KaitaiSampleRunner; see archive-manager's README).

What Kaitai Struct is used for

Beyond this project's two small examples, Kaitai Struct is mostly used to describe and reverse-engineer existing binary formats. The gallery's specifications include:

• media containers: MP4, AVI, WAV, MIDI;
• executables and other binary formats: ELF, PE/EXE, Mach-O, Java .class;
• filesystems and forensics: NTFS, ext2, Windows registry hives, prefetch files, event logs (popular in malware analysis and CTF challenges);
• archive and compression formats, game asset formats, and network protocol and packet capture formats (PCAP).

Any of the gallery's .ksy files can be compiled and used with this adapter in the same way as the examples here.

Galleries of examples

• Kaitai Struct format gallery -- hundreds of .ksy descriptions of real formats, by category, each with its generated parsers and documentation.
• kaitai_struct_formats -- the same descriptions as source, on GitHub.
• Kaitai Struct Web IDE -- try a description against a file in the browser.

Generating the classes

After editing a .ksy, regenerate with the Kaitai Struct compiler (0.11, https://kaitai.io), from src/main/ksy:

kaitai-struct-compiler -t java --java-package info.oais.infomodel.structure.kaitai.generated --debug --outdir ../java point2d.ksy csv_points.ksy

--debug is what makes the generated classes record where each field was read from, which this adapter turns into byte positions (see below). Leave it out and everything else still works, just without positions.

How the adapter works

• Generic, by reflection. KaitaiReflectiveStructureNode walks any generated class without per-format code, using the conventions of Kaitai's Java target: each field is a public no-argument accessor named after the field in camelCase with no get prefix (label_len becomes labelLen()), a nested type is another KaitaiStruct, and a repeat field is a List. Kaitai's own bookkeeping accessors (_io, _parent, _root, _read, ...) are skipped. Children come in file order: from a --debug class's _seqFields list, otherwise from the order of the class's declared fields (Java reflection doesn't promise method order).
• Leftover bytes. The root node's trailingBytes attribute (StructureNode.TRAILING_BYTES) says how many bytes the parse didn't reach (the stream's final position against its size).
• Typed values. Values come back as the generated accessors' Java types (int, long, double, String, byte[], ...).
• Repetition is an ARRAY node. A repeat field becomes one node of kind ARRAY whose children are named 0, 1, ... (the DFDL and DRB adapters instead use same-named siblings). Table views select these rows with <rows select="array">.
• Byte positions from --debug builds. A class compiled with --debug has public _attrStart/_attrEnd maps (each field's start and end offset, keyed by accessor name) and _arrStart/_arrEnd (one entry per element of a repeated field). The adapter reads them, so every field - and every element of a repeated field - reports its byte range. A --debug class also doesn't parse in its constructor; KaitaiStructureRepInfo detects that and calls _read() itself (call it yourself if you construct such a class directly).
• Switch types. For a switch-on field, the node's type name is the concrete type actually parsed, not the declared common type.
• In memory. The Digital Object is read fully into a byte buffer before parsing.

Writing data back

KaitaiStructureRepInfo is a WritableStructureRepInfo, for classes compiled with Kaitai Struct's read-write mode (kaitai-struct-compiler -w, Java and Python only, 0.11 and later); a read-only class is refused with that explanation. write(dataObject, changes) reads the data, fetches the lazily read instances, sets the changed values through the generated setters (path steps name fields as in the .ksy, sample_count, or as the accessor does, sampleCount; [n] picks an element of a repeated field), has each changed object check itself (_check(), which refuses e.g. a repeat count that no longer matches its list, or a string that no longer fits its size), and writes everything with _write. Kaitai writes into a stream of a fixed size, and an element repeated or sized to the end of the data must end exactly where the stream does, so the writer starts from the data's own size and adjusts it by what Kaitai reports. Unused bytes inside a type of stated size are written as zeros.

Known limitations

• Descriptions can't be loaded at run time: every format needs its class generated and compiled first (archive-manager does both on demand for its sample test, which takes a few seconds).
• The Kaitai Struct compiler doesn't rename fields whose names are Java reserved words (class, int, null, ...), so their Java doesn't compile; archive-manager's editor rejects such names when Kaitai is a target. Names YAML reads as booleans or null (on, no, null) must be quoted in a .ksy, as archive-manager's generator does.
• Parsing is eager and in memory; for very large files a RandomAccessFileKaitaiStream variant of KaitaiStructureRepInfo would be the natural extension.