Backup and Restore

Overview#

Backups in EventSourcingDB are newline-delimited JSON files that contain both events and schemas. Snapshots are implicitly included because they are stored as regular events.

Changed in 1.3A backup is either a full backup, which holds the whole database, or an increment, which holds only the events written after a previous backup. A restore takes a full backup followed by its increments, so you can back up a large database in full once in a while, and in small increments in between.

A backup keeps a database exactly as it is. To change events on the way, or to bring in events from another system, use Export and Import instead.

Performing a Backup#

Changed in 1.3To create a backup, send a POST request to /api/v2/management/create-backup. Its body says which backup you want, through lastBackedUpEvent: null asks for a full backup, with all events and schemas in the database.

curl \
  -X POST \
  -H "Authorization: Bearer <API_TOKEN>" \
  -H "Content-Type: application/json" \
  -d '{ "lastBackedUpEvent": null }' \
  http://localhost:3000/api/v2/management/create-backup \
  -o full-backup.ndjson

The server responds with a stream of newline-delimited JSON, which is written to standard output by default. In the example above, the -o option redirects it to a file named full-backup.ndjson. The file names on this page are used for demonstration purposes only – you can choose any name or storage path that fits your backup strategy.

lastBackedUpEvent is required: a body that leaves it out, such as {}, is answered with last-backed-up-event-missing rather than with a full backup, so that a client that has lost track of its previous backup notices.

A complete backup ends with an end item. If the backup cannot be completed, for example because the database shuts down while it runs, an error item comes before the end, such as server-shutting-down, or file-unreadable for a damaged file of the store, which its details name. If the backup fails unexpectedly otherwise, say because reading from the disk fails, the stream breaks off without an end. The status code is already 200 by the time the first item goes out, so the end of the stream is what tells you whether the backup is complete. Keep a backup only if it ends with end and holds no error item; a restore refuses any other.

Changed in 1.3The backup can be run while the database is in normal operation or in read-only mode. Writes go on while the backup runs. The backup holds the state at its start – the events written and the schemas registered up to then – and leaves out whatever is written after it began.

The Format of a BackupAdded in 1.3#

A backup is a stream of items, one per line, each an envelope with a type and its data, in this order:

1. A backup item, which names the version of its format as formatVersion, currently 2, and the event the backup follows on as lastBackedUpEvent, which is null for a full backup.
2. An eventSchema item for each event type with a schema, with the event type as type and its schema as schema.
3. An event item for each event, in the order in which the events were written.
4. A lastBackedUpEvent item, which names the event the backup ended at, by its id and hash. A full backup of a database without events has none.
5. The end item.

While the backup runs, the stream also sends heartbeat items, which a restore skips, so you can keep the file exactly as you received it. A full backup of a database with one schema and one event looks like this:

{"type":"backup","data":{"formatVersion":2,"lastBackedUpEvent":null}}
{"type":"eventSchema","data":{"type":"io.eventsourcingdb.library.book-acquired","schema":{"additionalProperties":false,"properties":{"title":{"type":"string"}},"required":["title"],"type":"object"}}}
{"type":"event","data":{"specversion":"1.0","id":"0","source":"https://library.eventsourcingdb.io","subject":"/books/42","type":"io.eventsourcingdb.library.book-acquired","time":"2026-10-28T09:37:59.474457Z","predecessorhash":"0000000000000000000000000000000000000000000000000000000000000000","datacontenttype":"application/json","data":{"title":"2001 – A Space Odyssey"},"traceparent":null,"tracestate":null,"hash":"esdb:hash:v2:b12136d2fc9cad9c083b8ef89f98c76f069c5d80fe9b75165b733ced287e43a0","signature":null}}
{"type":"lastBackedUpEvent","data":{"id":"0","hash":"esdb:hash:v2:b12136d2fc9cad9c083b8ef89f98c76f069c5d80fe9b75165b733ced287e43a0"}}
{"type":"end","data":{}}

Incremental Backups#

An increment holds the events written after a previous backup, so that you do not have to back up the whole database every time. To create one, send the id and hash from the lastBackedUpEvent item of the previous backup as lastBackedUpEvent, whether that backup was a full backup or an increment itself:

curl \
  -X POST \
  -H "Authorization: Bearer <API_TOKEN>" \
  -H "Content-Type: application/json" \
  -d '{
        "lastBackedUpEvent": {
          "id": "0",
          "hash": "esdb:hash:v2:108513ef6826326fadf15a2e18c32c0955fd42b6f0a0d84591c16bf7d116388f"
        }
      }' \
  http://localhost:3000/api/v2/management/create-backup \
  -o increment-1.ndjson

An increment has the same format as a full backup. Its backup item names the event it follows on, it holds every event schema again, since a backup cannot tell which schemas were registered after the previous one, and only the events after that event. Its own lastBackedUpEvent item names the event it ended at, which the next increment follows on; if nothing was written since the previous backup, that is the same event again.

Before it sends anything, EventSourcingDB checks that the database holds that event with that hash. If it holds no event with that ID, the backup fails with last-backed-up-event-unknown, and if the event has another hash, with last-backed-up-event-mismatched. Both usually mean that the request goes to another database than the one the previous backup was created from, or that the database was restored from another backup since. In that case, start anew with a full backup.

Restoring a Backup#

Changed in 1.3To restore a backup, send it as the body of a POST request to /api/v2/management/restore-backup, with the content type application/x-ndjson. A restore takes a full backup followed by its increments, in the order in which you created them, one after the other in one request body, each exactly as creating it sent it:

cat full-backup.ndjson increment-1.ndjson increment-2.ndjson \
  | curl \
      -i \
      -X POST \
      -H "Authorization: Bearer <API_TOKEN>" \
      -H "Content-Type: application/x-ndjson" \
      --data-binary @- \
      http://localhost:3000/api/v2/management/restore-backup

A full backup without increments is restored the same way, and so is a backup that the v1 API created, which is always a full backup and is restored on its own. A body with another content type is rejected with 400 Bad Request.

The server responds with a stream that ends with end once the backup is restored. If the backup cannot be restored, an error item comes before the end, and its code says why, for example full-backup-missing for a body that begins with an increment, backup-increment-disconnected for an increment that does not follow on the part before it, as when one is missing or the parts are out of order, or event-chain-broken for events that were changed. Send the full backup and all of its increments in one request: once the full backup is restored, the database is no longer empty, and a second restore is refused with store-not-empty.

The restore puts the database into lockdown by itself. Until the restore is over, the database refuses reads and writes with restore-in-progress, and ends observations that are running with it. An audit, and a backup that began before the restore, wait until it is over instead. The lockdown is lifted once the restore is over, without a restart.

When a Restore FailsAdded in 1.3#

If a restore fails, for example because the backup is invalid, exceeds the licensed capacity, or EventSourcingDB shuts down meanwhile, EventSourcingDB removes what it has restored so far, so that the database is empty again and the restore can be retried.

If removing it fails as well, EventSourcingDB refuses reads and writes with store-failed. It does the same, without removing anything, if the disk does not confirm the mark with which a restore flags the data directory before it writes anything, since removing what was restored under a mark that may not be stored could leave a data directory that EventSourcingDB takes for a valid database after a crash. The health endpoint reports this in its check store:operationMode as locked with the status fail, and the log names the cause. If EventSourcingDB stops without shutting down gracefully while a restore is running, for example because it is killed or the machine loses power, it refuses to start on the data directory afterwards. In both cases, empty the data directory, start EventSourcingDB, and restore the backup again.

Compatibility#

Changed in 1.3There are no hard limits on the size of the backup file itself, apart from operating system constraints. What does apply is the licensed event capacity: a restore whose backup holds more events than the license permits is rejected with capacity-exceeded. A single line of the backup, though, may hold at most 601 KiB, which is more than any item of a backup takes, and a restore refuses a longer one without reading it to its end, with backup-malformed, see Limits.

This version restores the backups that its v2 API creates, and those that the v1 API of this or an earlier version created. It checks every event of a backup, though, much as writing events checks it, and refuses a backup with an event that lacks its source, subject, type, or data, whose source is not a URI as RFC 3986 defines it, or that takes more than 64 KiB as it is stored. Earlier versions did not check all of this, so a backup that one of them created can hold such events, and this version refuses it with backup-malformed or event-too-large. Like the earlier versions, it also refuses a backup with an event that is older than the event before it, which an earlier version could write when the clock of the server went back, with event-chain-broken.

It also refuses a backup with an event dated in the future, such as 9999-12-31, once that event lies more than an hour beyond the clock of the server, since every event written after the restore would take the time of that event until the clock had caught up with it. It refuses such a backup with backup-ahead-of-clock, and restoring it through the v1 API with a 400. The same backup restores once the clock has caught up. An hour covers the difference between the clocks of two servers, so a backup from a server whose clock ran slightly ahead still restores, and new events take the time of its latest event until the clock has caught up, see Watching the Clock.

It checks every schema of a backup the same way as registering one, and refuses one that does not describe an object, such as {"type": "array"}, with event-schema-not-object, also in a backup that the v1 API created. Restoring that backup through the v1 API refuses such a schema as well, with a 400. Only a backup that was changed by hand can hold one, since no version of EventSourcingDB registers it, and it is of no use: it refuses every write of its event type with event-schema-violated, and verifying the store reports it as an error.

A schema that refers to anything outside of itself with $ref, such as a file, is refused with event-schema-malformed. Earlier versions registered such a schema, so a backup that one of them created can hold it.

Such a backup can also hold a schema that is too complex to validate events against: one that refers back to a part of itself, such as one that describes a tree, one that uses $dynamicRef or $recursiveRef to refer to an anchor, or one that reaches more than 10,000 subschemas, counting a referenced one each time it is referenced. This version refuses such a schema with event-schema-too-complex, and restoring through the v1 API refuses it with a 400.

It also refuses a schema with a number of more than 1,000 digits before or after its decimal point, with event-schema-malformed as well, and a schema that takes more than 600 KiB as a backup writes it, which is more than any schema that registering one takes, with backup-malformed. Only a backup that was changed by hand can bring either into a store: earlier versions converted every number of a schema to a 64-bit floating-point number, and a restore takes a schema as the backup holds it, which a backup made by hand may hold far shorter than the next backup writes it. A schema that an earlier version took from such a backup stays in the store, but no backup that holds it can be restored again.

It reads every key of a backup exactly as it is documented, such as type, payload, or eventType. A key spelled any other way, such as Type or EVENTTYPE, is one that the line does not have: a line that holds one next to the documented key restores as the documented key says, and a backup spelled otherwise throughout is refused with backup-malformed, and restoring it through the v1 API with a 400. Earlier versions read the keys without regard to case, and took the last of several spellings of one key. No version of EventSourcingDB writes a key otherwise than documented, so only a backup that another tool wrote or that was changed by hand can hold one.

The other way round, a backup that this version creates cannot be restored in an earlier version: earlier versions know neither the format of the v2 API nor version 2 of the hash that the events written by this version carry. In general, a version restores a backup of a newer one only as long as it knows its format and the versions of its hashes, and refuses it otherwise with backup-format-unknown or event-hash-version-unknown.

Before you upgrade, restore a backup of your database into an empty instance of the new version on trial. Your backups are only of use if the version you run can restore them.

Best Practices#

• Store backups in a secure, version-controlled location
• Use read-only mode for zero-downtime migrations
• Changed in 1.3Monitor backup and restore processes via logs, HTTP status codes, and the end of each stream
• Restore only into freshly initialized databases
• Use version-pinned builds to ensure compatibility
• Added in 1.3Restore a backup on trial into an empty instance of a new version before you upgrade to it

Backups give you confidence that your event history is safe, portable, and restorable – across environments and over time.