Modeling
SynthEvents
Remembering
EventSourcingDB
Building
ArchitectureKit
Version
1.3 · latest1.21.11.0
Docs / EventSourcingDB / Operations / Logging and Monitoring

Logging and Monitoring

This guide explains how to observe and monitor EventSourcingDB in production environments. It describes the built-in log output, available health and license endpoints, and how to integrate them into existing monitoring setups.

Logging#

EventSourcingDB logs to standard output using structured JSON. It uses the Go slog standard library, which produces logs that are machine-readable and well-suited for centralized log aggregation.

Each log entry includes a timestamp, log level, message, and optional structured fields such as additional data. The following log levels are used:

  • INFO: Normal operational messages
  • WARN: Unusual behavior or predictions of errors
  • ERROR: Non-recoverable problems

If EventSourcingDB cannot start, for example because a required flag is missing, it logs the reason as ERROR and exits with code 1. In that case, it also writes the error and its usage as plain text to standard error.

Log messages can be consumed by any system capable of parsing and/or forwarding JSON logs.

Health and Monitoring EndpointsChanged in 1.3#

EventSourcingDB provides two routes for status checks and monitoring, both in the system category of the HTTP API: ping and health. Both answer the HTTP method GET, so that load balancers, probes, and scrapers such as Prometheus can ask them.

/api/v2/system/ping – Liveness Check#

This route returns the HTTP status code 200 OK as soon as the database process is running, and 503 Service Unavailable while it is shutting down. It requires no authentication and is ideal for liveness and readiness probes or for use with load balancers. Besides GET, it answers POST:

curl -i \
  http://localhost:3000/api/v2/system/ping

/api/v2/system/health – Health and Metrics#

This route reports the health of the server as a set of checks, each with the status pass, warn, or fail. It requires a valid API token, and answers QUERY as well as GET. By default, it answers in the Health Check Response Format for HTTP APIs, with the content type application/health+json:

curl -i \
  -H "Authorization: Bearer <API_TOKEN>" \
  http://localhost:3000/api/v2/system/health

The response holds the overall status, which is the worst status of all checks, and the checks, each as a list of entries with its status, its observedValue, and, where it has one, its observedUnit. Shortened to three of its checks, a response looks like this:

{
  "status": "pass",
  "checks": {
    "license:remaining": [
      {
        "status": "pass",
        "observedValue": 28771199.403,
        "observedUnit": "seconds"
      }
    ],
    "store:eventCount": [
      {
        "status": "pass",
        "observedValue": 18240
      }
    ],
    "store:operationMode": [
      {
        "status": "pass",
        "observedValue": "read-write"
      }
    ]
  }
}

The overall status decides the status code: pass and warn are answered with 200 OK, and fail with 503 Service Unavailable, so that a load balancer can take an instance whose health fails out of service. While the server is shutting down, the route answers with 503 as well, with the message database is shutting down: try again later rather than a health report.

OpenMetrics for Prometheus#

To get the same values in the OpenMetrics format, which Prometheus reads, ask for it in the Accept header, as application/openmetrics-text:

curl -i \
  -H "Authorization: Bearer <API_TOKEN>" \
  -H "Accept: application/openmetrics-text" \
  http://localhost:3000/api/v2/system/health

The route chooses OpenMetrics only where the Accept header prefers it over the health-check format: a request without an Accept header, with */*, or with a type that the route does not offer gets the health-check format. Prometheus asks for OpenMetrics by itself, with its default Accept header, so a scrape configuration needs only the route and the API token:

scrape_configs:
  - job_name: eventsourcingdb
    metrics_path: /api/v2/system/health
    authorization:
      credentials: <API_TOKEN>
    static_configs:
      - targets: [ "localhost:3000" ]

In OpenMetrics, the route always answers with 200 OK, whatever the status of its checks, since Prometheus drops every sample of a scrape that answers otherwise, which would hide the values exactly when something is wrong. So alert on the values rather than on the status code of the scrape.

The values of the process carry the standard names that the client libraries of Prometheus give them, such as process_cpu_seconds_total, so that generic dashboards and alerts find them, and every other value carries the prefix eventsourcingdb_. If you move from the v1 API, its metrics had other names, so queries, dashboards, and alerts built on them need rewriting, see Migrating to the v2 API.

Checks#

The health route reports the following checks. The first column names a check as the health-check format does, the second as OpenMetrics does, and the third says when it warns or fails:

Check OpenMetrics Status
license:remaining eventsourcingdb_license_remaining_seconds warn from 30 days before the license expires, fail once it has expired
process:cpuUsage process_cpu_seconds_total warn while the process uses more than three quarters of the cores it may use, see below
process:fileDescriptorsMaximum process_max_fds fail if it cannot be measured
process:fileDescriptorsOpen process_open_fds warn from 80 % of process:fileDescriptorsMaximum, fail from 95 %, or if it cannot be measured
process:memoryResident process_resident_memory_bytes fail if it cannot be measured
process:memorySys eventsourcingdb_process_memory_sys_bytes always pass
process:uptime process_start_time_seconds always pass
store:operationMode eventsourcingdb_store_operation_mode pass for read-write; warn for read-only and locked, or fail where the database got there by a failure
store:indexes eventsourcingdb_store_indexes pass for ready; warn for rebuilding, or fail once an attempt to rebuild them has failed, until one succeeds
store:clockLag eventsourcingdb_store_clock_lag_seconds warn from a minute on, see Watching the Clock
store:eventCapacity eventsourcingdb_store_event_capacity always pass
store:eventCount eventsourcingdb_store_event_count warn above 80 % of the event capacity, fail once it is reached
store:eventDataSizeAverage eventsourcingdb_store_event_data_size_average_bytes always pass
store:eventDataSizeMaximum eventsourcingdb_store_event_data_size_maximum_bytes always pass
store:eventWriteTimeAverage eventsourcingdb_store_event_write_time_average_seconds always pass
store:eventWriteTimeMaximum eventsourcingdb_store_event_write_time_maximum_seconds always pass
store:sizeData eventsourcingdb_store_size_data_bytes fail if it cannot be measured
store:sizeIndex eventsourcingdb_store_size_index_bytes fail if it cannot be measured
store:sizeJournal eventsourcingdb_store_size_journal_bytes fail if it cannot be measured
store:sizeSchemas eventsourcingdb_store_size_schemas_bytes fail if it cannot be measured
store:sizeTotal eventsourcingdb_store_size_total_bytes fail if one of the sizes it adds up cannot be measured
store:writes eventsourcingdb_store_writes_total always pass, see Counting Writes
store:writeRejections eventsourcingdb_store_write_rejections_total always pass, see Counting Writes
system:cpuCount eventsourcingdb_system_cpu_count_cores always pass
system:diskRemaining eventsourcingdb_system_disk_remaining_bytes warn once 80 % of the space above the reserve of 1 GiB is used, but not before less than 100 GiB are left above it; fail while writes are refused for lack of space, or if it cannot be measured

A few of them need a word more:

  • store:operationMode is read-only while read-only mode is on, whether you switched it on or the disk is almost full, and locked while a restore runs, which refuses reads as well. After a storage failure, the database puts itself into one of these modes until it is restarted, and the check fails meanwhile.
  • process:cpuUsage gives the cores that the process uses, and warns once these are more than three quarters of the cores it may use: the CPUs it may run on, or the value of the environment variable GOMAXPROCS where it is set, or the CPU limit of its container where that is lower, as it is set, so that a container limited to half a CPU warns above 0.375 cores. The limit of a container is read on Linux, and one that changes while the database runs counts from the next second on. The first value after the start covers the time since the start.
  • The resident memory and the file descriptors are left out where the platform does not provide them: macOS has no resident memory, and Windows no file descriptors. They are missing from both formats then.
  • The memory, the file descriptors, and the sizes of the data directory are measured in the background, about every 10 seconds, so they can be that old. The sizes of the events and of the indexes are each measured again after at least 10 seconds, or after nine times as long as their last measurement took, whichever is longer, so in a large store, they can be older. The CPU usage is sampled every second.
  • store:sizeJournal covers the journal file and the write-ahead log, which holds the writes that are not yet written to the data files, so it is larger than in version 1.2.
  • The sizes of the events and of the indexes, and with them the total size, have no value until they have been measured once, which happens in the background after the start and can take a while for a store with many events or many subjects. Until then, the check passes without a value, and OpenMetrics leaves out its sample. A size that cannot be measured has no value either, rather than reading as 0. After an upgrade from version 1.2 or earlier, the old indexes of the subjects stay on disk until EventSourcingDB has removed them in the background, as described in First Start After Upgrading, and neither the size of the indexes nor the total size counts them meanwhile.
  • The CPU and the uptime are reported differently in OpenMetrics: as the CPU time that the process has spent since it started, from which Prometheus derives the usage over any window, such as rate(process_cpu_seconds_total[5m]), and as the time at which the process started, in seconds since the Unix epoch, from which time() - process_start_time_seconds derives the uptime.
  • The operation mode and the indexes are state sets in OpenMetrics, with a sample for each state, which is 1 for the current state and 0 for the others, as in eventsourcingdb_store_operation_mode{eventsourcingdb_store_operation_mode="read-write"} 1.

Counting Writes#

The health endpoint of the v2 API, /api/v2/system/health, also counts the writes since the server started, by their outcome: how many were written, how many were rejected because they asked for something that the database refuses, and how many failed because the database could not store a write that was in order. A write counts once, however many events it holds, and writes through the v1 API count as well. A write that the API refuses before it reaches the database does not count, such as one whose request is malformed, whose event is invalid, or that comes in while the license has expired.

Rejected writes are counted by their reason as well:

  • precondition: a precondition did not hold, including an EventQL precondition whose query returned false.
  • query: the query of an EventQL precondition could not be evaluated, or did not return a single boolean.
  • schema: an event did not match the schema of its type, its type had no schema although schemas are required, or the stored schema of its type is too complex to validate against or malformed, such as one with a number that has more than 1,000 digits before or after its decimal point.
  • size: an event was too large.
  • operation-mode: the database took no writes at the time, since it was read-only, shutting down, or restoring a backup, or it took no writes with an EventQL precondition, since it was rebuilding its indexes after an upgrade.
  • capacity: the events would have exceeded the event capacity of the license.
  • events: the write held no events.

A write that the database refuses after a storage failure or because the disk is full counts as failed, not as rejected: the write itself was in order, and someone needs to act.

To get the counters in the OpenMetrics format, ask for it in the Accept header:

curl -i \
  -H "Authorization: Bearer <API_TOKEN>" \
  -H "Accept: application/openmetrics-text" \
  http://localhost:3000/api/v2/system/health

Among its other values, the response holds two counters, with a sample for every outcome and every reason from the start, at zero until it first occurs:

eventsourcingdb_store_writes_total{outcome="written"} 1024
eventsourcingdb_store_writes_total{outcome="rejected"} 12
eventsourcingdb_store_writes_total{outcome="failed"} 0
eventsourcingdb_store_write_rejections_total{reason="precondition"} 11
eventsourcingdb_store_write_rejections_total{reason="query"} 0
eventsourcingdb_store_write_rejections_total{reason="schema"} 1
eventsourcingdb_store_write_rejections_total{reason="size"} 0
eventsourcingdb_store_write_rejections_total{reason="operation-mode"} 0
eventsourcingdb_store_write_rejections_total{reason="capacity"} 0
eventsourcingdb_store_write_rejections_total{reason="events"} 0

In the default health-check format, the same counters appear as the checks store:writes and store:writeRejections, with an entry for each outcome and each reason, named by its componentId.

The counters start at zero whenever the server starts, and they always report the status pass, since how many writes failed since the start says nothing about the health now. So alert on how fast they grow, rather than on their value, for example with this Prometheus expression:

rate(eventsourcingdb_store_writes_total{outcome="failed"}[5m]) > 0

Watching the Clock#

The time of an event is never earlier than the time of the event before it, since verifying the store, auditing it, and restoring a backup all rely on that. If the clock of the server goes back, for example when NTP corrects it, new events take the time of the latest event until the clock has caught up, and the database takes writes meanwhile as usual. A step of a fraction of a second, as NTP makes it, passes without notice.

If the clock is far behind the latest event, new events keep the time of the latest event for as long as the clock takes to catch up. This happens after a reboot with a wrong clock, or once an event carries a time in the future, because the clock was set ahead when it was written or because it came with a restored backup. A restore refuses an event that lies more than an hour beyond the clock, with backup-ahead-of-clock, so that after a restore, new events keep the time of its latest event for an hour at most. The health endpoint of the v2 API reports how far the clock is behind as the check store:clockLag, in seconds, and in the OpenMetrics format as the gauge eventsourcingdb_store_clock_lag_seconds. It is 0 while the clock is not behind, and its status turns to warn once the clock is behind by a minute or more, which the status tile of the management UI names as well. The database also logs a warning with the first write that finds the clock that far behind, and a note with the first write that finds it within the minute again.

So that someone looks at the clock in that case, alert on the status of the check, or for example with this Prometheus expression:

eventsourcingdb_store_clock_lag_seconds >= 60

Management UI#

The built-in management UI provides a visual dashboard to display status and metrics from the above endpoints. It is most useful in local or small-scale environments that lack external observability tooling.

In larger setups, health and metrics data should be pulled directly via the API. In such cases, you may want to disable or restrict access to the management UI.

Best Practices#

Changed in 1.3Use /api/v2/system/ping for lightweight availability checks, and /api/v2/system/health with Accept: application/openmetrics-text for Prometheus-based monitoring. Always capture structured logs from standard output and forward them using a centralized logging pipeline. Restrict access to sensitive endpoints using API tokens and firewall rules, and avoid exposing metrics to the public internet.

This setup provides robust, standards-based observability – suitable for both local development and production-scale deployments.