Upgrading EventSourcingDB

Overview#

Changed in 1.3Upgrading EventSourcingDB is designed to be simple and safe. In most cases, you can stop the running instance and start the new version using the same data directory. A version that needs your attention before the upgrade says so in its release notes, as version 1.3 does: go through Before Upgrading From 1.2 or Earlier first.

Upgrade Procedure#

The general steps are:

1. Stop the current EventSourcingDB process.
2. Replace the binary or container image with the new version.
3. Start EventSourcingDB again, using the same persistent data directory.

Changed in 1.3This process is the same whether you are running EventSourcingDB as a pre-built binary, in Docker, or in Kubernetes. Wherever the Docker image runs on a data directory that an earlier version wrote, though, mind that the image of version 1.3 runs EventSourcingDB as another user, as described in Upgrading a Docker Deployment From 1.2 or Earlier and Upgrading a Kubernetes Deployment From 1.2 or Earlier.

Before Upgrading From 1.2 or EarlierAdded in 1.3#

Version 1.3 needs your attention before the upgrade wherever one of the following applies to your deployment. Go through the list, and take care of every point that applies before you start version 1.3:

• The Docker image runs on a data directory that an earlier version wrote. The image runs EventSourcingDB as another user from version 1.3 on, so give the data directory to that user first, see Upgrading a Docker Deployment From 1.2 or Earlier and Upgrading a Kubernetes Deployment From 1.2 or Earlier.
• An entry of the data directory, such as events or wal, is a link or lies on another file system, or the data directory lies on a file system that cannot lock. Version 1.3 refuses to start on such a data directory, see Data Directory and One Server per Data Directory.
• A rollout starts the new server before the old one has stopped, such as a Kubernetes Deployment with the strategy RollingUpdate. Two servers must never run on one data directory, and since versions up to 1.2 do not lock it, nothing keeps version 1.3 from starting next to one of them, see Downtime Considerations.
• A reverse proxy or a pool of connections keeps idle connections open for longer than 120 seconds. EventSourcingDB closes an idle connection after that time, so keep the idle timeout of the proxy or the pool below it, see Operational Limits.
• The API token is shorter than 32 characters, holds anything but printable ASCII, or has whitespace at its start or end. Version 1.3 refuses to start with such a token, with a shorter one only on a persistent data directory, so change it first, see Compatibility.
• The server is started with --ai-model or --ai-option. Version 1.3 removes EventAI together with both flags and refuses to start with either of them, saying unknown flag: --ai-model, so remove them from its command line.
• Clients verify event hashes. Update them to an SDK that knows version 2 of the hash before you upgrade the server, see Compatibility.
• The source of events is not a URI as RFC 3986 defines it. Version 1.3 refuses such writes, see Sources.
• Event schemas refer to anything outside of themselves, refer back to a part of themselves, use $dynamicRef or $recursiveRef, or reach too many subschemas. Version 1.3 refuses the writes of their event types afterwards, see Compatibility.
• Subjects have more than 32 segments or more than 1,024 characters. Events can no longer be written or read by such a subject, see Size Limits.
• EventQL queries or preconditions rely on behavior that version 1.3 changes, such as how values of different types compare, that the ID of an event is a string, or that dates and times are taken in UTC. The EventQL reference marks every change, and the changelog lists them.

At its first start, version 1.3 rebuilds its indexes, as described in First Start After Upgrading, and an earlier version cannot open the data directory again afterwards, see Downgrading.

Upgrading a Docker Deployment From 1.2 or EarlierAdded in 1.3#

Up to version 1.2, the Docker image runs EventSourcingDB as the user esdb, with the UID 100 and the GID 101. From version 1.3 on, it runs it as the user nonroot, with the UID and the GID 65532. A data directory that an earlier version wrote belongs to esdb, so version 1.3 cannot open it: it exits right away, and its log says failed to lock the data directory: open lock.esdl: permission denied. Before you start version 1.3 on such a data directory, take one of two ways:

• Give the data directory to the new user, once, with a container of the new image that runs chown as root. The image knows no user named root, so give its UID and GID, 0:0:

  docker run --rm \
    --user 0:0 \
    --entrypoint chown \
    -v <HOST_DATA>:/var/lib/esdb \
    thenativeweb/eventsourcingdb:<VERSION> \
    -R 65532:65532 /var/lib/esdb
  

• Keep running EventSourcingDB as the old user, by adding --user 100:101 to the docker run command that starts version 1.3.

Replace <HOST_DATA> with the volume or the directory on your host that you mount at /var/lib/esdb, and <VERSION> with the version you upgrade to.

The same holds for the files that EventSourcingDB reads, such as a TLS certificate, its key, or a license file: they have to be readable by the UID 65532. If you restricted them to esdb with chmod 600, as Managing Configuration and Secrets recommends, give them to the new user as well, or keep running EventSourcingDB as the old user, which covers both.

Upgrading a Kubernetes Deployment From 1.2 or EarlierAdded in 1.3#

On Kubernetes, version 1.3 meets the same data directory that belongs to esdb, but the fsGroup of the Pod can take care of it. Set it to 65532, as the reference manifest does, in the same rollout that brings in version 1.3. When the kubelet mounts the volume, it gives every file and directory in it to that group and lets the group read and write them, so version 1.3 opens the data directory, although its files still belong to esdb.

Since they belong to another user, though, version 1.3 cannot keep other users of the machine out of these files, as it does with the files it creates itself, see Permissions of the Data Directory. It works with them all the same, and logs a warning whenever it starts, could not restrict the permissions of entries of the data directory, so other users of the machine may read them, with their number, the first of them, and its error, since it tries again at every start, which takes the longer the more events the data directory holds. To keep other users out of them as well, and to keep the start short, give the data directory to the new user with an init container, as described below.

This takes a volume whose driver applies the fsGroup. By default, a CSI driver does so for a volume with the access mode ReadWriteOnce and a file system type, such as a block volume formatted with ext4; kubectl get csidriver <DRIVER> -o jsonpath='{.spec.fsGroupPolicy}' shows how yours handles it, with <DRIVER> being the provisioner of your storage class. A hostPath volume never gets the fsGroup. Where it does not apply, or where your Pod sets none, take one of two ways:

• Keep running EventSourcingDB as the old user, by setting runAsUser: 100 and runAsGroup: 101 in the securityContext of the container.
• Give the data directory to the new user, with an init container that runs chown -R 65532:65532 /var/lib/esdb as root on the same volume, such as one of the new image with runAsUser: 0 and chown as its command. A namespace that enforces the Pod Security Standard restricted does not admit a container that runs as root.

First Start After UpgradingAdded in 1.3#

A new version may convert data in the data directory once, at its first start. It starts right away and serves reads and writes meanwhile. Where the conversion rebuilds the indexes, though, the requests that need them complete are refused with indexes-rebuilding, an error item at status 200, until the conversion is done, which can take several minutes:

• Listing subjects and listing event types
• Reading an event type
• Running EventQL queries
• Writing events with an isEventQlQueryTrue precondition, whatever its query reads
• Verifying the store
• Restoring a backup

Everything else goes on: reading and observing events, writing them without such a precondition, auditing the event store, and creating backups. Registering an event schema works meanwhile, but checks all existing events even for an event type without events, which takes the longer the more events the database holds. This needs no manual steps. The health endpoint reports this as rebuilding in its check store:indexes, and as ready once the conversion is done. If the conversion fails, for example because the disk is full, it is tried again until it succeeds, and the check reports the status fail meanwhile, so the health endpoint answers with a 503.

Some conversions refuse nothing: until they are done, every request is answered from the data as it was, and the old data is removed in the background afterwards, while the database works as usual. Until then, the data directory holds both, so the disk needs room for the converted data next to the old – which is usually far less than the old data takes.

A browser that holds the files of the management UI of version 1.2 or earlier keeps running that version for up to a day after the upgrade to version 1.3, unless you reload the UI once bypassing the cache. From version 1.3 on, a browser checks with the server whether the files of the UI are still current whenever it loads them.

Compatibility#

Changed in 1.3EventSourcingDB aims for long-term compatibility across versions. No manual adjustment of data or configuration is usually required. Backups, though, are restored only within limits:

• Newer versions restore a backup only if they accept every event it holds. Version 1.3 refuses a backup with an event that lacks its source, subject, type, or data, whose source is not a URI as RFC 3986 defines it, that takes more than 64 KiB as it is stored, or that lies more than an hour beyond the clock of the server, such as one dated 9999-12-31. Earlier versions did not check all of this, so a backup that one of them created can hold such events. Like the earlier versions, version 1.3 also refuses a backup with an event that is older than the event before it, which an earlier version could write when the clock of the server went back.
• Older versions restore a backup only as long as they can read everything it contains. A backup that the v2 API creates cannot be restored in a version before 1.3: such a version knows neither its format nor version 2 of the hash that the events written by version 1.3 or later carry.

Version 1.3 also refuses an event schema that refers to anything outside of itself with $ref, such as a file or a URL; a schema may only refer to its own parts, such as #/$defs/address. Earlier versions accepted such a schema. If one of them registered it, the schema no longer compiles after the upgrade: every write of its event type fails with 500 Internal Server Error, and verifying the store reports the schema. Since a schema cannot be replaced, use a new event type instead. A backup that holds such a schema is refused as well.

Version 1.3 also refuses an event schema that is too complex to validate events against: one that refers back to a part of itself, such as one that describes a tree, one that uses $dynamicRef or $recursiveRef to refer to an anchor, and one that reaches more than 10,000 subschemas, counting a referenced one each time it is referenced. Earlier versions accepted such a schema. If one of them registered it, the schema stays in the store, but after the upgrade every write of its event type is refused with event-schema-too-complex rather than validated, which the v1 API answers with a 409, and verifying the store reports the schema. Since a schema cannot be replaced, use a new event type instead. A backup that holds such a schema is refused as well.

So before you upgrade, look for event schemas with a $ref that does not begin with #, and for those that refer back to a part of themselves or use $dynamicRef or $recursiveRef, and restore a backup of your database into an empty instance of the new version on trial, as described in Backup and Restore. After the upgrade, your backups are only of use if the new version can restore them.

Added in 1.3Clients that verify event hashes need an SDK that knows the version of the hash the server writes. Version 1.3 writes version 2: update such clients first, to an SDK version that supports it, and upgrade the server afterwards. A client with an older SDK reports every event written after the upgrade as tampered with.

Added in 1.3A server on a data directory requires an API token of at least 32 characters from version 1.3 on, and refuses to start with a shorter one, saying you must specify an API token of at least 32 characters. Every server, also one with a temporary data directory, requires a token of printable ASCII characters only, without whitespace at its start or end, and refuses to start with any other, saying you must specify an API token of printable ASCII characters only, without whitespace at its start or end. Earlier versions took any token that was not empty, so if yours is shorter or has another character, change it before you upgrade, and give your clients the new one.

The following versions of other products, client SDKs, and tools run with this version of EventSourcingDB, as their tests show:

Product, SDK, or Tool	Versions that run with EventSourcingDB 1.3
.NET SDK	1.0 and later
ArchitectureKit	1.0 and later
Elixir SDK	1.0 and later
Go SDK	1.0 and later
Java SDK	1.0 and later
JavaScript SDK	1.0 and later
MCP Server	1.0 and later
PHP SDK	1.0 and later
Python SDK	1.0 and later
Rust SDK	1.0 and later

DowngradingAdded in 1.3#

EventSourcingDB does not support downgrading a data directory. Once a newer version has used a data directory, do not start an older version on it. Where an older version cannot read the data directory safely, it refuses to start.

To go back to an older version, restore a backup into it. Create the backup before upgrading, as recommended below. A backup created with the newer version can be restored in an older one only as described under Compatibility.

Downtime Considerations#

Currently, EventSourcingDB operates as a single-instance service. As a result, upgrades require downtime. Zero-downtime upgrades will be supported in future releases once clustering and replication become available.

Added in 1.3A server locks its data directory, so the new version starts only once the old one has stopped: a rollout that starts the new version first, such as one of a Kubernetes Deployment with the strategy RollingUpdate, never gets it ready, see One Server per Data Directory. Versions up to 1.2 do not take the lock, though, so it does not keep version 1.3 from starting next to one of them: when you upgrade from such a version, make sure that it has stopped before you start version 1.3, as the upgrade procedure does.

Safety Recommendations#

• Always create a backup before upgrading
• Added in 1.3Restore that backup on trial into an empty instance of the new version
• Use version-pinned binaries or image tags
• Verify the upgrade in a staging environment before applying it in production
• Monitor the logs after startup for any warnings or errors

Upgrading EventSourcingDB is typically fast and reliable – and with proper preparation, it can be performed with minimal risk.