Verifying Store Integrity

Overview#

EventSourcingDB provides a verification endpoint that performs a complete integrity check of the event store. It recalculates all event hashes from scratch, validates the consistency of stored data, checks that every stored event schema compiles and describes an object, and verifies all internal indexes.

Added in 1.3Every stored event schema also carries a checksum of its content, which the verification checks as well. It also reports a schema that is too complex to validate against, as one that an earlier version registered can be, since every write of its event type is refused.

This operation can help detect data corruption, storage issues, or index inconsistencies.

Running a Store Verification#

Changed in 1.3To start a store verification, send an HTTP POST request to /api/v2/management/verify-store, which takes no body:

curl -i \
  -X POST \
  -H "Authorization: Bearer <API_TOKEN>" \
  http://localhost:3000/api/v2/management/verify-store

The server responds with a stream of newline-delimited JSON. While the verification is still running, the server sends heartbeat items to keep the connection open. Once it is done, it sends what it found as finding items, each with a severity and a description, and then the end item. A store in which the verification finds nothing gets the end item alone:

{"type":"end","data":{}}

A finding with the severity error is an inconsistency of the store, and one with the severity warning an additional notice, which does not necessarily indicate a failure. A broken hash chain, for example, is reported like this:

{"type":"finding","data":{"severity":"error","description":"structural verification of \"events\" failed: failed to verify event chain: verification failed at event with id 17: wrong hash encountered; the chain was not checked after this event"}}
{"type":"end","data":{}}

Each check reports every inconsistency it finds, up to 100; beyond that, a further finding says how many more it found. Only the hash chain cannot be followed past a break: the finding names the event where it breaks, and the events after it are not checked against the chain. A damaged events file does not break it: the finding names the file, and the chain is checked on from the file after it. Neither do events that are missing between two events files: the finding names them and the files around them, and the chain is checked on from the next file:

{"type":"finding","data":{"severity":"error","description":"structural verification of \"events\" failed: the events from 6 to 11 are missing: \"events/0.esdb\" ends with event 5, and the next events file, \"events/12.esdb\", begins with event 12"}}

A single missing event reads the event 132 is missing: "events/0.esdb" ends with event 131, and the next events file, "events/133.esdb", begins with event 133. Events missing before the first events file are named by where that file begins, as in the events from 0 to 11 are missing: the first events file, "events/12.esdb", begins with event 12.

A gap between two files has one of two causes, which the verification cannot tell apart: one or more events files are missing, here events/6.esdb and possibly further files up to event 11, or the file before, here events/0.esdb, lost its last events. A file that lost events from its middle or its beginning, or that holds none, is reported as a break of the chain instead.

Added in 1.3Reads and writes go on while the store is verified. The verification checks the store as of the latest event at its start; events written meanwhile are checked by the next verification.

Changed in 1.3If the verification itself cannot run to completion, the stream holds no findings, but an error item followed by the end item. If the database shuts down while the verification runs, its code is server-shutting-down; verify the store again once it is back.

If the verification cannot read a file of the store, for example because the disk returns input/output errors or the database lacks the permission to read it, it stops at that file and reports file-unreadable, with the file in its details, rather than reporting it as an inconsistency, since a file that cannot be read says nothing about whether the store is damaged:

{"type":"error","data":{"code":"https://docs.eventsourcingdb.io/errors/file-unreadable","description":"file is unreadable: 'events/278863.esdb' could not be read, so the store was not verified: input/output error","details":{"file":"events/278863.esdb"}}}
{"type":"end","data":{}}

Check the disk, and verify the store again once the file can be read. A file that is missing, or whose content is wrong, is damage of the store, and is reported as a finding.

A verification is also refused with an error item if it cannot start: with indexes-rebuilding while the indexes are being rebuilt after an upgrade, since indexes that are not complete yet would differ from the events, with restore-in-progress if a backup is being restored when it starts, and with store-failed once the database refuses reads after a failed restore. If the verification fails unexpectedly, the stream breaks off without its end item.

The status code is already 200 by the time the first item goes out, so the end of the stream is what tells you whether the verification finished. Treat an error item, or a stream without end, as an aborted check, not as a clean one.

Typical Use Cases#

Store verification is useful in the following scenarios:

• After recovering from disk failures or system crashes.
• As part of regular maintenance procedures.
• Before migrating data to a new instance.
• To investigate unexpected read or write issues.

A successful store verification provides assurance that the event store is internally consistent and fully operational.