Help

What this application is

This is a prototype archive. It takes in information and records — accession in records-management terms, ingest in OAIS terms — and brings together two disciplines that usually live in separate systems:

A bridging ontology links the two, so a record described with RiC-O can point to the OAIS package that preserves it, and the other way round. Both are held as linked data (RDF) in one store, so everything can be browsed, graphed and queried together.

This page follows the life of the information: first how to take it in, describe it, create its Representation Information, see how it relates, and check that each Archival Information Package has every component OAIS requires; then the three preservation techniques; then how to view and analyse the data.

Why astronomical applications? To show that preserved data can actually be used, this prototype opens it in astronomical tools: TOPCAT and SPLAT for tables and spectra, DS9 and Aladin for images. They were chosen because they are designed to fetch data from remote archives, then analyse, compare and combine information from many different sources (they talk to each other, and to archives, using Virtual Observatory standards such as VOTable and SAMP). That is exactly what the data in an archive needs. Nothing depends on them, though: the archive serves the data in standard formats, described by its Representation Information, so other applications could be used in their place.

Working with the archive

Browsing — no login needed

Everything can be looked at without an account. Catalogue and Accession Register are public search and browse views. A search finds records by any word of their title, description, subjects, type, record number or identifiers, or by the start of one: “harb” finds “harbour”. From any record, Records, Agents, Provenance events and Mandates link to each other (an agent's page lists what it created, and so on), and Entities lists everything, of any kind. It can be searched like the catalogue, and its type filter offers Representation Information you can reuse: descriptions and groups, with how many Data Objects use each and whether it is preserved. From a Data Object's page, Find Representation Information to use opens that list with a button to use one for the Data Object, so a format described once serves every object in it.

Logging in to take in or change anything

Browsing, viewing and downloading are open to everyone. To take information in, edit, preserve or look after the records, log in with your own name and password: an administrator gives you an account. The first time, and after an administrator gives you a new password, you are asked to choose your own (at least 12 characters); you can change it at any time from your name in the top bar. Your role (Visitor, Manager, Expert, Administrator, or one your archive has made) decides what you may do, such as taking in, editing the catalogue, Representation Information, fixity or backups, and what you see, such as closed material, a record's OAIS detail or every property as stored. The menu shows only what your role allows, in groups: Catalogue, Taking in, Preservation, Access and privacy, and Administration, each a list in alphabetical order; a page your role doesn't allow says so. After 10 wrong passwords from one address, or for one account, within 15 minutes, logging in from there, or to that account, waits until that time is up.

Once logged in, Edit history shows who changed what, and Backups lists the archive's backups: one is made every night while the archive runs, and “Back up now” makes another.

Where your role allows it, the Security log lists logins, failed logins, account and role changes, releases of closed records, destruction approved and privacy decisions, newest first, with who, when and from where; passwords are never in it.

Jobs lists the long tasks of the kinds your role works with -- harvests, AIPs of emails, imports and exports, fixity checks, automatic transformations, gap analysis and scans -- running and finished, with their progress and problems. They are kept when the archive restarts: one that was running then is marked interrupted, and Resume carries it on where the task allows.

Your role also decides how much detail you see. Lower roles get short forms in plain words with suggestions already chosen, such as who an AIP is kept for or how often a mailbox is harvested; Experts see every field, in OAIS terms; Administrators see every field with nothing filled in for them. Settings a short form doesn't show stay as they are.

From your name in the top bar you can also pick your own theme, light or dark, instead of the archive's.

Taking information in (accession and ingest)

Describing records, and editing properties and relationships

Records are described with RiC-O: a Record, Record Part or Record Set, its title and description, who created it (an Agent), the activities it came from, the mandates that govern it, its identifiers and dates, and the records it is part of.

Any individual's page has Edit, which opens the generic editor: add or remove its types, its properties (text, numbers, dates, links) and its relationships to other individuals, one at a time, choosing from the properties RiC-O and OAIS define for it. Adding a relationship offers "+ Create new …" in the target picker, so what it points to doesn't need to exist beforehand — it's created and linked in the same step.

Creating Representation Information

Representation Information is what's needed to understand a Data Object's bits: their structure (how the bits make numbers, text, records and arrays), their semantics (what each value means, in what units), and other Representation Information such as the software that reads them.

On a Data Object's page, Representation Information Network draws everything needed to understand it, for a Designated Community: what it is interpreted using, what each of those needs in turn, and where each chain ends, coloured by why: understood by the community, a specification held, a specification listed but not yet held, or a gap. Click a node to open it, or save the drawing as SVG.

Viewing relationships

Records and their OAIS packages. RiC-O and OAIS ask different questions about the same material: RiC-O who created something, in what context, and how to find it again; OAIS what has to be kept alongside the bits for it to still make sense decades from now. Their classes don't line up one-to-one, so the bridging ontology connects them two ways: class-level correspondences using SKOS closeMatch/broadMatch/relatedMatch (deliberately weaker than owl:equivalentClass where the concepts don't coincide), and individual-level bridge:hasOAISCounterpart links from a record to the OAIS structure that preserves it. Open any record and follow its "OAIS mapping" link to see both: which OAIS classes correspond to its RiC-O type(s), and its actual linked OAIS structure.

The graph. Graph shows the whole archive as a network; "View in graph" on any page shows a focused view around that one item, with an adjustable depth (1–4 hops). Every listing page, and the SPARQL console's results, also has "View this page as a graph", which graphs exactly the items on screen.

Querying. The SPARQL console runs any SELECT query over the data and the ontologies together — for questions no page answers, such as which packages lack Fixity Information.

The components of an OAIS Archival Information Package

An Archival Information Package (AIP) is what OAIS preserves: everything needed for its information to stay understandable, kept together. By the OAIS Information Model, an AIP has:

Making an AIP from a Data Object. A Data Object's page has Create the AIP, which makes an AIP around it with everything the archive knows or can work out: its Content Information (the Data Object and the Representation Information it's interpreted using); Fixity Information, the SHA-256 and size of its bits, fetched from their storage location then; Reference Information (its identifiers and where its bits are); Provenance Information (when, and by whom, the AIP was made from it); Context Information (what it's interpreted using and what links to it, and anything you add); and a Package Description. What only the archive can decide is asked for on the same form — the Designated Community (an existing one, or a new one) and its Preservation Objective, the Access Rights — and left out if not given, never made up. Packaging Information comes with the BagIt bag. The AIP's page then shows what's still missing. A Data Object already in an AIP links to it instead.

Preserving descriptions and other data the archive holds. Representation Information is itself information that may need preserving. Anything in the archive that holds data of its own — a Kaitai Struct, DFDL, DRB or EAST description, a drb-python driver, a view specification, a Transformation mapping — has Create the AIP on its page too. Its text is stored in the archive as a file and becomes a Data Object, interpreted using a description of the language it is written in, whose IRI is its identifier; the AIP is made around it as above. The AIP of any data interpreted using it then refers to that AIP — its Context Information names the AIP and the Data Object's identifier, and links to it — instead of packaging the description again, and the data's page lists such Representation Information. If the description is edited later, its text as it is then becomes a new Data Object, with a new identifier, and the earlier one stays in its AIP.

Sharing Representation Information among many AIPs. Many Data Objects — every FITS file, say — need the same Structure and Semantic Representation Information. Any Representation Information has Create the AIP on its page: its description in RDF (Turtle), with the Representation Information it needs, becomes a Data Object of its own, preserved once. From then on the AIP of each FITS file refers to that AIP, by the Data Object's identifier, instead of holding a copy — both in the archive and when written out as a BagIt bag, whose aip.ttl names the AIP it is in and whose components file says so. Representation Information changed since it was preserved is included in full again until it is preserved anew. In the Bulk Upload Spreadsheet, rows with the same FormatInfo, Semantics and OtherRI share one Representation Information, and URI: and an AIP's identifier there points to Representation Information already preserved.

Build an AIP with the editor: create an Archival Information Package and add these relationships in turn, creating each component as you go. As OAIS requires, also say who the information must remain understandable to and by what tests: a Designated Community, linked to the Preservation Objective(s) it's tested against, and any Transformational Information Properties whose values must survive a Transformation. These are defined by an Agent (normally the archive), consulting the Designated Community, not by the Community itself.

Checking it. An AIP's page shows a checklist of every required component: present, missing, or (for Packaging Information) provided when the AIP is written out as a BagIt bag. Add what's missing with the editor. The page also lists everything linked to the AIP, and "View in graph" shows the whole package.

An AIP stored as a package. Often the components aren't described in the archive itself but are inside the package the AIP is stored as — for instance an AIP made by Eternal, kept as a .7z or .zip file and named by the AIP's storage location. Then the AIP's page offers Map its package to the AIP components: the archive fetches the package, opens it, checks its BagIt bag (every file's checksum, and Payload-Oxum), and shows, for each component, where in the package it is and what it says there — the upload spreadsheet carried in it (metadata.csv, read with the column definitions in metadata_definition.xml), the METS file's PREMIS (each file's format with its PRONOM identifier, its checksum, and events such as ingestion and virus checks), the checksum audit and the Preservation Description Information file. Each component also lists the upload spreadsheet's entries expected to hold it — Semantics (often the identifier of a separate AIP holding the Semantic Representation Information, linked when that AIP is described in this archive) and Language; FixityHashSHA256; Provenance, Creator, Publisher and Contributor; Relation for Context Information; Rights for Access Rights Information; Format and FormatInfo; OtherRI — and says which aren't filled in, and why: left blank, or no such column in the spreadsheet used. Other Representation Information is optional. The table can be downloaded as CSV.

Preserving digital information: the three OAIS techniques

Keeping the bits safe isn't enough. Software, hardware and what people know all change, so information has to stay independently understandable to its Designated Community — the people who must be able to use it — in the ways its Preservation Objectives set out. OAIS names three fundamental ways of achieving that. Each is described below, with how to do it in this application.

1. Add Representation Information

Keep the Data Object exactly as it is, and add the Representation Information needed to understand it. Representation Information needs its own Representation Information in turn, until what's left is something the Designated Community already understands. This is the technique to use first: nothing about the data changes, so nothing can be lost.

2. Transformation

When adding Representation Information isn't enough, or a format is becoming obsolete, rewrite the Data Object in another format. This changes the Content Information, so it has to be checked and recorded: decide which values must survive (its Transformational Information Properties), check them after the Transformation, and add Provenance Information saying what was done. The result is an AIP Version, a candidate to replace the original. A Transformation that can't be shown to be reversible is a Non-Reversible Transformation, where those checks are the evidence that nothing that matters was lost.

3. Hand over complete AIPs to another archive

If this archive can no longer look after some information, another archive can take responsibility for it. That only works if each AIP is complete — every component listed in The components of an OAIS Archival Information Package — so the receiving archive can carry on without asking anything.

Viewing and analysing the data

A Data Object whose bits have a storage location, and whose Representation Information says how to read them, can be opened from its page or its right-click menu in the graph:

As explained at the start, these astronomical applications are used because they are made to fetch data from remote archives and to analyse, compare and combine information from many sources; any application that reads VOTable or FITS could be used instead. If you don't have them: TOPCAT (topcat-full.jar) or SPLAT-VO (splat-vo-<version>.jar; see About SPLAT), each run with java -jar; for images, SAOImage DS9, Aladin Desktop or Fiji. RepInfo Tools' preview page also downloads a bundle for a local data file: a manifest naming your file and each description, with a generated table view and, for an image, an image view.

Table, spectrum or image: which viewers are offered

RepInfo Tools doesn't ask what kind of data you have: it works it out from the shape of the byte layout you describe, and saves the answer as view specifications (im:ViewSpecification, with im:viewKind "table" or "image") in the data's Semantic Representation Information. The archive reads those back to decide what to offer.

One description can be both a table and an image. Every viewer needs a DFDL or DRB SDF description, which the server decodes the data with, and a storage location for the bits. Since views are ordinary view specifications, Representation Information written by hand can give its own. Decoding builds every pixel as an element, so very large images are slow: a 128 × 128 image takes a few seconds.

Data description languages

Notes on each of the data description languages RepInfo Tools works with — how a format is described, how this application runs the descriptions, what each engine reports (typed values, byte positions), and its known limitations:

Looking after the records: access, personal data and integrity

These pages are for logged-in editors; each is in the menu once you've logged in. Most of them are also worked from a record's own page: below its description, editors see a panel with each of the sections below for that record. What is set on a group applies to everything in it.

Closed records

Material taken in from harvests is closed until an editor releases it: visitors don't see it anywhere, not in lists, search, the graph or SPARQL. Release a record, or a group with everything in it, from its page.

Adding many Gmail mailboxes at once

On the Harvesting page, Add mailboxes from Google Workspace lists the domain's users, alphabetically by name, saying which accounts are suspended and which mailboxes are harvested already. Tick the ones you want, or Select all, and choose one schedule, retention rule, legal constraint and labels to leave out for them all: each becomes a Gmail mailbox of its own, and one harvested already is left as it is. Scheduled harvests run one after another, never all at once. If the page says the archive can't list the users, an administrator needs to finish setting it up (see the administrator's manual).

Harvested emails: threads and AIPs

On a harvested mailbox's page, Emails in threads, and their AIPs lists its emails in threads: those whose subjects are the same once Re:, Fwd:, AW: and the like are taken off, whichever folders or labels they are in. Each thread shows how many emails it has, their dates and who wrote them; open one to read its emails in order.

You can make Archival Information Packages while the emails are still closed: of one email, from its page or its thread; of a whole thread; or of the whole mailbox. Each email becomes an AIP of its own, and a thread or mailbox also gets an Archival Information Collection listing them. An email already in an AIP isn't packaged again, so after a new harvest, run it again to add the new emails. The AIPs stay closed until the emails are released.

Or choose One AIP holding them all: the thread's or mailbox's emails go into a single AIP, each email's message a Data Object of its Content Information with its own Fixity Information, and its Reference, Provenance and Context Information and Package Description saying it holds that thread's or mailbox's emails. Run it again after a new harvest and a new version of that AIP is made, holding the earlier emails and the new ones; the earlier version is kept. It runs in the background, and stays closed while the emails are.

Retention

Retention rules say how long records are kept, counted from their date, from when the archive took them in, or from an event, and what happens then: destroy (after an editor approves), review, or keep. A legal hold stops anything under it being destroyed. The Retention page lists what is due.

Legal constraints

A legal constraint (data protection, copyright, a statutory closure, a deposit agreement, ...) gives what it covers an access level: open; redacted; content withheld (the description shown, the files not); description partly closed; registered readers only; or closed. Each has a statement visitors see, a legal basis, and an end date or period; when it ends it comes up for review. Where several apply, the most restrictive wins. Apply one to a record or group from its page; the Legal constraints page shows what is covered and what is due for review.

Identifying information

Names, identity numbers, email addresses, faces and other details that identify people can be marked, in a description or in a file, and each given a safeguard: withhold, redact, generalise, pseudonymise or release. "Scan" on a record's page suggests what to mark (by identifier patterns, people the archive knows, and, where set up, name and face recognition); a suggestion changes nothing until an editor confirms it. "Make a redacted copy" makes a new file with the confirmed parts blacked out, which visitors get instead of the original. Before releasing a group, "Check before releasing" lists what could identify someone in combination.

Personal data and data subject requests

On a record's page, say whether it holds personal data (special category data, such as health or beliefs, closes it), whom it concerns, and which data protection laws it falls under. A record under several laws gets every right any of them gives and the shortest deadline. Data subject requests records each request: who asked, what for (access, rectification, erasure, restriction, objection), and when the reply is due. On a request's page, find the person; the records linked to them are listed, closed ones too, and can be exported for an access request. Rectification adds a correction beside the record; the record itself is never changed. Erasure destroys the records' content and writes it in the erasure ledger, so restoring an older backup can't bring them back. The register of processing lists each harvest and accession for the data protection officer.

Fixity

Fixity shows whether files are exactly as they were. Each file has a digest, a value computed from its contents; a check computes it again and compares. A check that finds a difference never changes the digest: it is recorded, and listed on the Fixity page until an editor says how it was resolved (for example, the file restored from a copy). A fixity policy says how often files are checked, with which algorithms, and what happens when a change is found: report it, or withhold the file from visitors until it is resolved. Set a policy on a record or group from its page; without one, the archive's own policy applies. Checks that are due run each night; "Check now" on a record's page checks it at once.

Where the archive keeps further copies of the bits on other disks, Fixity on a record's or collection's page also has Copies of the bits: tick the stores its Data Objects are also kept in. Each check then makes the missing copies, checks every copy, and repairs a damaged one from a good one, listing each repair.

Representation Information: making sure files can still be understood

A file is only useful if someone can understand it. Representation Information is what that takes: the specification of its format, the meaning of the values in it (names, units), and whatever those need in turn. An archive keeps its material for a Designated Community, the people it is meant for, and a chain of Representation Information can stop at what that community already knows: its Knowledge Base. One community comes set up: astronomy and planetary science users, who are taken to know the FITS standard, the PDS4 standard and its common dictionaries, VOTable and XML, but not each mission's own dictionary.

Finding gaps. "Analyse now" on a record's page follows each of its files' chains and says whether the community could understand them. A gap is where a chain stops short: a specification the community doesn't know and the archive doesn't hold; a file inside a container that nothing explains; a format identified with too little confidence; or software as the only way in (software ages fastest, so a written specification must be there too). The record's panel shows each gap and where its chain stops; the Representation Information page lists every open gap. To close one, attach the missing Representation Information ("Find Representation Information to use" on the file's page, or make it with RepInfo Tools), or collect the specification.

The required specifications. The archive doesn't keep a copy of every specification it relies on. The Representation Information page lists each one it needs but doesn't hold, with its identifier or web address, the formats that need it and how many files; those where a chain actually stops come first. Download the list as CSV to work through it. When you have collected a copy, press "We have a copy" on its row, saying where it is kept: it leaves the list.

Containers. A ZIP file, a Word document or a PDF with attachments holds other files. "Scan containers" on a record's page looks inside, and inside containers within containers, lists each item with its format, and proposes what explains each format it finds; confirm the proposals on the record's page. Parts that belong to the container's own format, such as a Word file's styles or a PDF's fonts, need nothing more. Word documents and PDF/A-3 files are already described, so they need no setting up.

What changes the answer. Representation Information set on a collection reaches the files of the same format in it; the meanings of terms add up, and a term a file defines differently is flagged for review. An editor can change what a community is taken to know on the Representation Information page: each change makes a new version of its Knowledge Base, and the files it affects are analysed again, as they are when a file's format is identified differently.

Preservation copies

Images, PDFs, office documents, audio and video can be given a preservation copy: the same content in a format kept for the long term, made by another application such as Ghostscript, LibreOffice or FFmpeg. On a collection's page, under Preservation copies, pick recipes from the library (for example, PDF to PDF/A, or video to lossless FFV1), adjust their settings, and everything in the collection gets copies: on request with "Make preservation copies now", and each night for what is new. A collection can have its own selection, different from the one above it.

Each copy is kept in the archive beside its original, which is never changed. The record's page lists its copies, the recipe and command line that made each, and checks of what the copy should keep (for an image, its size and colour depth; for a PDF, its pages; for video, its length and frame size), each preserved, changed or not checked. A copy is closed if its original is, and follows the same fixity policy. The Preservation copies page shows the tools installed, the library and each run, and, where the administrator allows it, lets you add a command line of your own.

HTML pages can be given a PDF/A copy too: the recipe HTML page to PDF/A has headless Chrome print the page as a browser shows it, then Ghostscript make that PDF a PDF/A. The page lists the tools with their versions and documentation; Check runs a tool there and then and shows what it said. Where the administrator allows it, you can also add or change a tool (its name, its program on the server, how it says its version, its documentation) and add recipes of your own, including ones of several steps, for input formats given as media types or PRONOM identifiers. Where it isn't allowed, those forms are shown greyed out, saying why. By default Chrome draws the page from its own file only, with no network, so images and styles from elsewhere are left out; on a collection, set the recipe's network to on for pages that need them, and javascript to off to draw a page without running its scripts.

The collections a record is in

What is set on a collection reaches what is in it: its provenance while the record was in it, its context, its legal constraints, its fixity policy and its Representation Information. The panel on a record's page lists the collections it is in and since when. There you can correct when it joined, record when it left, or stop it inheriting one kind of information from one collection, saying why (for example, an item from another donor). A group can be made an Access Collection, a grouping for finding things, which passes on only what you choose.

Developer documentation

Javadoc API reference → — generated from this codebase's own /** ... */ source comments (mvn javadoc:javadoc, which writes it to archive-manager/target/reports/apidocs). It isn't in the jar, to keep it smaller: each build on GitHub publishes it as the download api-docs, beside the jar. It covers this application, the OAIS core model (oaiscore), and the OAIS structure adapters (oais-structure-*: DFDL, Kaitai Struct, DRB, TOPCAT and the demo), grouped under separate tabs.

API reference (Swagger) → — every JSON and download address under /api, described in OpenAPI and callable from the page; the description itself is at /v3/api-docs. Reading needs no login; to try a change, log in as an editor in this browser first.

The ontology source files are downloadable from the home page, byte-for-byte as loaded at startup. For the full picture — storage, the login model, scale limits, internationalisation, how to update the ontologies — see the project's README.md.