Export and Import

An export writes every event of a database as a plain CloudEvent on a line of its own. An import takes exactly these lines and writes them into a database without events, with their IDs and their times, and computes the hashes itself. The reference pages of export-events and import-events describe both routes in every detail; this guide is about getting the whole task done.

Export or Backup?#

Both put all events of a database into a file, and both bring them back only into a database without events. They differ in what the file is for:

Aspect	Backup and restore	Export and import
Purpose	An exact copy of a database	The events themselves, to change them or to bring them in from elsewhere
Each event carries	Its hash and its signature	Neither, the import computes the hashes
Event schemas	Part of the file	Not part of the file, you register them beforehand
Changing an event	Breaks the chain of hashes, which a restore refuses	Change its line
Increments	Available	None, an export always holds every event
Where the events come from	EventSourcingDB	EventSourcingDB or any other system

Use a backup to protect a database, or to copy it as it is, as Backup and Restore and Migrating Between Instances describe. Use an export when the events have to change on the way, such as to remove personal data, and use an import to bring in events from another system.

Moving Events to Another Database#

A move takes the same steps every time: stop the writes, export, change what has to change, prepare the target, import, and compare the result.

Stop the Writes#

An export holds the events written up to its start, and writes go on while it runs, so whatever is written afterwards would stay behind. Put the source into read-only mode before you export. Reading and exporting go on as usual, and writes are refused with store-read-only:

curl \
  -X POST \
  -H "Authorization: Bearer <API_TOKEN>" \
  http://localhost:3000/api/v2/management/enable-read-only-mode

Switched on like this, read-only mode lasts until the server is next started. Where the source has to stay read-only across restarts, start it with the --read-only flag instead.

Export the Events#

Send a QUERY request, which takes no body, and keep the answer in a file:

curl \
  -X QUERY \
  -H "Authorization: Bearer <API_TOKEN>" \
  http://localhost:3000/api/v2/management/export-events \
  -o events.ndjson

Then check that the export is complete. Since every line is an event, the answer has no item that marks its end: an export that breaks off is only recognizable by the connection, and by comparing it with the database, as Checking That an Export Is Complete describes.

Change the Events, if You Need To#

Each line is a JSON object of its own, so a script can work through the file line by line. What it may do follows from what an import checks:

• Changing the data of an event needs nothing else, as long as the event still matches the schema of its type.
• Removing an event means removing its line and numbering the later events again, since the IDs begin at 0 and follow each other without gaps. Otherwise, the import stops with event-chain-broken.
• Keep the time of every event, since no event may be older than the one before it.

To remove personal data, follow GDPR Compliance, which goes through these steps for that case.

Prepare the Target#

Start a new instance on an empty data directory, since an import goes only into a database without events. An export holds no event schemas, so register them first: read-event-types lists them on the source, and register-event-schema registers each of them on the target. The import then checks every event against the schema of its type.

Import the Events#

Send the file as the body of a POST request, all of it in one request. Use --data-binary, since -d removes the line breaks:

curl \
  -X POST \
  -H "Authorization: Bearer <API_TOKEN>" \
  -H "Content-Type: application/x-ndjson" \
  --data-binary @events.ndjson \
  http://localhost:3000/api/v2/management/import-events

The answer is a stream that ends with an end item. Without an error item before it, the events were imported. Until the import is over, the database refuses reads and writes.

Compare the Result#

Read the latest event of the target with read-events, on the subject / with the options recursive set to true and order set to antichronological, and compare its ID with that of the last line of the file. The target then takes writes as usual, and its next event gets the next ID, so you can point your applications at it.

What Stays the Same and What Changes#

The IDs, times, subjects, types, and data of the events are what the lines say. Unless you removed a line, read models and the positions of observers therefore stay valid. After a removal, the IDs of the later events differ, so read models, the positions of observers, and anything else that stored them have to be rebuilt.

The hashes are computed anew, in the current version. An event written before version 1.3 gets a new hash even where nothing about it changed, and a changed or removed line changes the hash of every event after it, since each hash covers the one before it. So the hashes that anything outside the database stored no longer match, and the result of a previous audit no longer applies.

Bringing Events From Another System#

An import takes events from any system that can write them as lines of this shape. You convert the events yourself, and EventSourcingDB computes the hashes, so a converter needs to know nothing about them. Stop the writes in the other system first, as you would for a move, so that the stock you convert is the whole of it.

Each line is a plain CloudEvent with exactly the fields that import-events lists, and no others. What most often needs thought when you convert:

• The IDs begin at 0 and follow each other without gaps, so the order of the lines is the order of the events, and the IDs of the other system cannot be kept.
• No time may lie before that of the event before it, nor more than an hour beyond the clock of the server.
• The source is a URI, the type a reverse domain name, and the data a JSON object.

An import does not check that the stock is complete: a file that was cut at the end of a line imports as a smaller stock. So compare the number of lines with the number of events in the other system before you import. Then register your schemas, import the file, and compare the result, as for a move. An import goes only into a database without events, so bring in the whole stock at once, and write later events as usual.

When an Import Fails#

An import is all or nothing. If it fails, it removes what it has imported so far and keeps the schemas, so you can correct the lines and import them again. The error item says why it failed, and names an event by its ID and a line by its number.

If the server crashes during an import, it refuses to start on the data directory, since what is in there is only partly imported. Empty the data directory, start EventSourcingDB, register the schemas, and import the events again.

A database that took events before today's checks can hold events that an import refuses, such as one whose source is no URI, so that its export does not import unchanged. The import reports the first such event, and Events That Do Not Import Unchanged lists what to look for. Correct the line, and import again.