Observability

Logging

Logger (apps/api/src/middlewares/createLogger.middleware.ts), mounted globally in apps/api/src/app.ts, creates one request-scoped logger per request and runs the rest of the middleware chain inside runWithLogger (@thenetaji/packages, backed by AsyncLocalStorage) so getLogger() returns it from anywhere downstream — including UpstreamHandler, which has no ctx parameter by design.

It emits at most two structured events per request:

• http_request_started — debug level, method/path/site/operation.
• http_request_completed — the terminal event, emitted either from the success path or from the catch block if something threw. Level is error for 5xx, warn for 4xx, info otherwise. Carries status, duration, the registered route template, resolved site/operation, and a recursively redacted query object. Failures also carry their classification and upstream diagnostics.

The shared logger normalizes metadata keys, bounds oversized values, redacts credential-shaped fields and text, and attaches error_kind plus a stable error_fingerprint. The fingerprint excludes the error message so record IDs and other dynamic values do not split one failure into many groups.

/metrics, /health, /ready, /openapi.json, and /llm.txt are excluded from both events (SKIP_PATHS) — they're not customer traffic worth logging per-request.

Metrics: ClickHouse api_requests

metricsMiddleware (apps/api/src/middlewares/metrics.middleware.ts), also mounted globally, records HTTP duration, traffic, in-flight work, and classified failures. Every path label is the registered route template; api_errors_total adds bounded error code/kind labels for alerting while fingerprints remain in logs. It also writes per-request insight for /v{n}/{site}/* traffic to ClickHouse without blocking the response.

The insert only happens if CLICKHOUSE_PASSWORD is set (shouldRecordClickHouseInsight) — ClickHouse is optional infrastructure. Table DDL lives in apps/api/src/db/clickhouse.ts.

Background jobs

The in-process scheduler runs each job behind its own local guard and cross-replica advisory lock, so a long catalogue sweep cannot suppress canaries or Vantage work. api_background_job_runs_total uses bounded scheduled_job and outcome labels; api_background_job_last_completion_timestamp measures runner completion rather than business success, and api_background_job_enabled keeps disabled jobs out of stale-runner alerts. Canary result health remains a separate signal.

api_requests is a short-lived transaction log: ClickHouse deletes rows seven days after ts, and /hub/logs applies the same seven-day boundary at read time because TTL removal happens during background merges. Before that TTL is enabled, the versioned boot migration in apps/api/src/db/apiRequestRetention.ts backfills api_request_hourly and installs a materialized view for new traffic. The rollup has no TTL and preserves request totals, credit totals, latency distributions, status/upstream attribution, endpoint/source dimensions, and last-seen timestamps without retaining request IDs or individual request times.

All Hub usage and admin-health aggregates read api_request_hourly; only /hub/logs reads api_requests. Postgres credit-ledger rows are independent and are not deleted by this retention policy.

After a deployment logs ClickHouse request retention ready, clear one workspace's current transaction log without resetting its lifetime totals by running this against ClickHouse (substitute the workspace UUID):

ALTER TABLE core_analytics.api_requests
DELETE WHERE workspace_id = '00000000-0000-0000-0000-000000000000'
SETTINGS mutations_sync = 1;

Do not delete the matching rows from api_request_hourly; that table is the retained usage history.

If a single request makes more than one upstream call, only the most recent call's numbers survive in upstream_* — recordUpstreamInsight overwrites, it doesn't accumulate (apps/api/src/observability/upstreamInsight.ts).

There is no local /metrics scrape endpoint any more (per the comment at the top of apps/api/src/metrics.ts) — all metric emission goes through OpenTelemetry.

Tracing / OTel export path

apps/api/src/observability/tracing.ts is imported before anything else (apps/api/src/main.ts's first import, and the dev script's tsx watch --import). If OTEL_EXPORTER_OTLP_ENDPOINT is set and OTEL_SDK_DISABLED isn't "true", it starts a NodeSDK (@opentelemetry/sdk-node) that:

• Exports traces via OTLPTraceExporter.
• Exports metrics via PeriodicExportingMetricReader + OTLPMetricExporter — this is what actually ships the api.ts-defined instruments (histograms/counters from apps/api/src/metrics.ts) to the collector.
• Exports log records via BatchLogRecordProcessor + OTLPLogExporter, and wires @loglayer/transport-opentelemetry as a transport on the shared log instance — without this wiring, structured logs never leave the process even though they're generated.
• Applies Node auto-instrumentation (getNodeAutoInstrumentations) with the filesystem instrumentation explicitly disabled.

The collector on the receiving end re-exposes the metrics for Prometheus to scrape and the traces for whatever backend is configured — this API talks OTLP only, it never speaks the Prometheus scrape protocol itself.

Shutdown is explicit: main.ts's SIGTERM/SIGINT handler calls shutdownTracing() only after the HTTP server has finished draining in-flight requests, so spans for requests still in flight during shutdown aren't dropped.

See env.md for the full list of OTel-related variables.

Runbooks

Operational runbooks — what an alert means, first queries to run, and the response procedure — live outside this folder, in infrastructure/runbooks/observability/. Relevant to this service:

• api-high-5xx.md — elevated 5xx ratio on /v1.
• worker-high-5xx.md, worker-redis-errors.md — the Python worker side of the app/worker split (see architecture.md).
• upstream-degradation.md — a third-party/upstream source degrading.
• clickhouse-event-insert.md, collector-scrape-loss.md — pipeline/infrastructure failures in the observability stack itself.

This folder documents what the code does and where the data comes from; the runbooks document what to do when a dashboard alerts. Don't duplicate one into the other.