Operations

Health checks, logs, queues, scaling and retention.

Health

Service	Readiness	Ready when	Liveness
`api`	`/health/ready`	PostgreSQL and ClickHouse	`/health/live`
`api-ingest`	`/healthz`	PostgreSQL, ClickHouse, Redis	`/health/live`
`mcp`	`/healthz`	PostgreSQL, ClickHouse, Redis	`/health/live`
`worker-errors`	`/healthz` on `:4400`	PostgreSQL and Redis	`/health/live`
`worker-jobs`	`/healthz` on `:4200`	PostgreSQL	`/health/live`
`web`	—	—	`/health/live`

/healthz of api-ingest and worker-errors also reports the queue: pending, retrying, in-flight and dead entries, the lag of the oldest entry and how long draining has been paused.

api-ingest and mcp answer /healthz in full only to a caller whose address is in TRUSTED_PROXIES. Any other caller gets the status code and status, without the dependencies or the queue.

Logs

Each service writes pino JSON to stdout, one record per line. LOG_LEVEL sets the lowest level. Records carry no service name: take it from the container.

{
  "level": "info",
  "time": "2026-11-05T14:02:15.871Z",
  "requestId": "01JC9X4M2T8Q6R0B5N7V3W1YZK",
  "organizationId": "…",
  "userId": "…",
  "module": "http",
  "method": "POST",
  "path": "/api/v1/alert-rules",
  "status": 201,
  "durationMs": 18,
  "msg": "POST /api/v1/alert-rules 201"
}

Fields. level, time and msg are always set. module names the component. A failure carries err with type, message and stack, including its causes.
Requests. api, api-ingest and mcp write one record per request when it ends. api-ingest and mcp write the route template as route instead of path. Health probes and preflights are not logged. api-ingest writes successful intake requests at debug.
Errors. A 5xx is logged once at error with its err. So is a failed background job, with its queue and jobId.
Request IDs. Each request gets an ID, logged as requestId. A service keeps an incoming x-request-id instead only from one of the TRUSTED_PROXIES, and only when it holds 1 to 64 letters, digits, _ or -. api returns it in error responses.
Redaction. Records hold no headers, query strings or bodies. The params of a failed SQL query and the detail of a database error are written as [REDACTED].

Queues

Telemetry waits in the state Redis, in ingest:queue:<table> lists, until it’s inserted into ClickHouse. A batch inserted twice is deduplicated.

Failure	What happens
ClickHouse down	Draining of the failing table pauses with a growing backoff; other tables keep draining. Intake accepts until a queue reaches `INGEST_QUEUE_LIMIT`, then answers 429.
Rejected batch	Retried with a growing backoff, then moved to `ingest:dead:<table>` after `INGEST_DEAD_LETTER_AFTER_MS`. A batch that keeps failing while other batches reach the same table counts as rejected.
Redis full	Intake answers 429. Run the state Redis with `maxmemory-policy noeviction` so nothing queued is evicted.

Background jobs live in PostgreSQL, in the pgboss schema, and survive restarts.

Scaling

Service	Scale with	Notes
`api-ingest`	Incoming telemetry	Each instance also drains logs and spans.
`worker-errors`	Error volume	Each instance keeps its own source map cache of `SOURCE_MAP_CACHE_BYTES`.
`worker-jobs`	Alert rules and notifications	One or two instances. Concurrency settings apply per instance.
`api`, `web`	Dashboard users	Stateless.

Each ClickHouse table sets its retention as a TTL: 180 days for logs, spans and error events. Change it with a new ALTER TABLE … MODIFY TTL migration. Fired alert events are kept for ALERT_EVENT_RETENTION_DAYS.

Deleting a project removes its ClickHouse rows and its source maps through a project-cleanup job.

Operations

Health

Logs

Queues

Scaling

Retention