---
title: Operations
description: Health checks, logs, queues, scaling and retention.
---

## Health

| Service         | Readiness             | Ready when                    | Liveness       |
| --------------- | --------------------- | ----------------------------- | -------------- |
| `api`           | `/health/ready`       | PostgreSQL and ClickHouse     | `/health/live` |
| `api-ingest`    | `/healthz`            | PostgreSQL, ClickHouse, Redis | `/health/live` |
| `mcp`           | `/healthz`            | PostgreSQL, ClickHouse, Redis | `/health/live` |
| `worker-errors` | `/healthz` on `:4400` | PostgreSQL and Redis          | `/health/live` |
| `worker-jobs`   | `/healthz` on `:4200` | PostgreSQL                    | `/health/live` |
| `web`           | —                     | —                             | `/health/live` |

`/healthz` of `api-ingest` and `worker-errors` also reports the queue: pending, retrying, in-flight and dead entries, the lag of the oldest entry and how long draining has been paused.

`api-ingest` and `mcp` answer `/healthz` in full only to a caller whose address is in `TRUSTED_PROXIES`. Any other caller gets the status code and `status`, without the dependencies or the queue.

## Logs

Each service writes [pino](https://getpino.io) JSON to stdout, one record per line. [`LOG_LEVEL`](/self-hosting/configuration#logging) sets the lowest level. Records carry no service name: take it from the container.

```json
{
  "level": "info",
  "time": "2026-11-05T14:02:15.871Z",
  "requestId": "01JC9X4M2T8Q6R0B5N7V3W1YZK",
  "organizationId": "…",
  "userId": "…",
  "module": "http",
  "method": "POST",
  "path": "/api/v1/alert-rules",
  "status": 201,
  "durationMs": 18,
  "msg": "POST /api/v1/alert-rules 201"
}
```

- **Fields.** `level`, `time` and `msg` are always set. `module` names the component. A failure carries `err` with `type`, `message` and `stack`, including its causes.
- **Requests.** `api`, `api-ingest` and `mcp` write one record per request when it ends. `api-ingest` and `mcp` write the route template as `route` instead of `path`. Health probes and preflights are not logged. `api-ingest` writes successful intake requests at `debug`.
- **Errors.** A 5xx is logged once at `error` with its `err`. So is a failed background job, with its `queue` and `jobId`.
- **Request IDs.** Each request gets an ID, logged as `requestId`. A service keeps an incoming `x-request-id` instead only from one of the `TRUSTED_PROXIES`, and only when it holds 1 to 64 letters, digits, `_` or `-`. `api` returns it in error responses.
- **Redaction.** Records hold no headers, query strings or bodies. The `params` of a failed SQL query and the `detail` of a database error are written as `[REDACTED]`.

## Queues

Telemetry waits in the state Redis, in `ingest:queue:<table>` lists, until it's inserted into ClickHouse. A batch inserted twice is deduplicated.

| Failure         | What happens                                                                                                                                                                                     |
| --------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| ClickHouse down | Draining of the failing table pauses with a growing backoff; other tables keep draining. Intake accepts until a queue reaches `INGEST_QUEUE_LIMIT`, then answers 429.                            |
| Rejected batch  | Retried with a growing backoff, then moved to `ingest:dead:<table>` after `INGEST_DEAD_LETTER_AFTER_MS`. A batch that keeps failing while other batches reach the same table counts as rejected. |
| Redis full      | Intake answers 429. Run the state Redis with `maxmemory-policy noeviction` so nothing queued is evicted.                                                                                         |

Background jobs live in PostgreSQL, in the `pgboss` schema, and survive restarts.

## Scaling

| Service         | Scale with                    | Notes                                                                     |
| --------------- | ----------------------------- | ------------------------------------------------------------------------- |
| `api-ingest`    | Incoming telemetry            | Each instance also drains logs and spans.                                 |
| `worker-errors` | Error volume                  | Each instance keeps its own source map cache of `SOURCE_MAP_CACHE_BYTES`. |
| `worker-jobs`   | Alert rules and notifications | One or two instances. Concurrency settings apply per instance.            |
| `api`, `web`    | Dashboard users               | Stateless.                                                                |

## Retention

Each ClickHouse table sets its retention as a `TTL`: 180 days for logs, spans and error events. Change it with a new `ALTER TABLE … MODIFY TTL` migration. Fired alert events are kept for `ALERT_EVENT_RETENTION_DAYS`.

Deleting a project removes its ClickHouse rows and its source maps through a `project-cleanup` job.