Infrastructure runbook

How to stand the VPS side of core up from scratch, and how it behaves once it is running.

Written to be followed top-to-bottom on an empty machine. If you only need to know how a change reaches production, read deployment.md instead.

What gets deployed

Three independent stacks, brought up in this order. Each is a separate Dokploy service pointing at a different compose file in this repo.

Order matters: the app stack expects Redis to exist, and reports to the OTel Collector. It will boot without ClickHouse, Postgres, or the collector — those degrade rather than fail — but Redis is required for the worker.

All three join the dokploy-network overlay, which is how they reach each other by service name (redis, clickhouse, otel-collector, worker).

The application stack

The app stack is deployed as a Docker Swarm stack, not a Compose app. That is what makes rollouts zero-downtime — see How zero-downtime works.

Swarm cannot build images. CI builds and pushes them to GHCR, and the compose file only references them:

GitHub repo (main)                         GHCR
  └── docker-compose.yml                     ├── core-api:main
        image: ghcr.io/.../core-api:main     └── core-worker:main
              │                                     │
              │ 1. Dokploy clones repo              │
              │ 2. writes .env from Env tab         │
              ▼                                     │
        docker stack deploy ────── 3. pulls ────────┘
              │                    (Registry tab credentials)
              ▼
        Swarm starts api ×2 + worker ×2  =  4 containers

So the git repo supplies the deployment description; GHCR supplies the code. Dokploy still needs repo/branch/compose-path settings — it just never builds anything for this service.

Prerequisites

• A VPS with Docker installed and Swarm initialised (docker swarm init). Dokploy's installer does both.
• Dokploy installed, which also creates the dokploy-network overlay network.
• DNS for api.toolzerhub.com pointed at the new VPS IP.
• A GitHub PAT with read:packages scope, for pulling private GHCR images.

Confirm before continuing:

docker info --format '{{.Swarm.LocalNodeState}}'      # -> active
docker network ls --filter name=dokploy-network       # -> exists, overlay

Step 1 — Secrets

Every stack reads its config from a .env file that Dokploy writes from the service's Environment tab. Set these before deploying anything.

Dokploy's env editor writes every value quoted (LOG_LEVEL="info"), and env_file values reach the container verbatim, quotes attached. Confirmed as the root cause of a 2026-08-16 outage where the API exited in ~6s with no log output. Strip quotes on any value pasted into the Environment tab.

Required

Optional

Hostnames (CLICKHOUSE_URL, POSTGRES_HOST, WORKER_HOST) default to the compose service names and should be left alone in production.

Step 2 — Data services

Create a Dokploy Compose service pointing at infrastructure/data-services/docker-compose.yml. Set CLICKHOUSE_PASSWORD and POSTGRES_PASSWORD in its Environment tab. Deploy.

docker ps --filter name=clickhouse --filter name=postgres --filter name=redis

Migrating data from the old VPS

Named volumes (postgres_data, clickhouse_data, redis_data) start empty on a new host. Redis holds only fetch sessions with a 20-minute TTL — let it start cold. The other two need moving.

On the old host:

docker exec <postgres-container> pg_dump -U core core_data \
  | gzip > core_data.sql.gz

docker exec <clickhouse-container> clickhouse-client \
  --password "$CLICKHOUSE_PASSWORD" \
  --query "BACKUP DATABASE core_analytics TO Disk('backups', 'migration.zip')"

On the new host, after the data services stack is up:

gunzip -c core_data.sql.gz \
  | docker exec -i <postgres-container> psql -U core -d core_data

The API runs its own Postgres migrations at boot under an advisory lock, so schema drift self-corrects — you only need the data.

Step 3 — Observability

Create a Dokploy Compose service pointing at infrastructure/docker-compose.yml. Set CLICKHOUSE_PASSWORD (Grafana's ClickHouse datasource uses it) and POSTGRES_PASSWORD, the same value as the data services (below). Create Grafana's database first, then deploy.

Publishes on the host: Grafana 3010, Prometheus 9090, Loki 3100. Firewall these or put them behind Dokploy auth — they are not protected by default.

Grafana's database

Grafana keeps users, service-account tokens, alert state and history, and silences in its own grafana database on the data-services Postgres (host postgres over dokploy-network). Dashboards, alert rules, contact points, policies and datasources are files in infrastructure/grafana/provisioning and come back on every start. The grafana-data volume holds only plugins and cache, so wiping it (Dokploy's "clean volume") loses nothing. This is why: on 2026-10-10 that volume still held Grafana's SQLite database, a clean wiped it, and coto watch went blind because its token went with it.

Grafana logs in as its own grafana role with the data-services POSTGRES_PASSWORD, so there is no extra secret. Without it the stack refuses to deploy; Grafana never falls back to SQLite.

Create the role and database (once, and again to rotate the password). infrastructure/data-services/grafana-database.sql is idempotent. It creates role grafana (login only, no superuser, createdb or createrole rights) and database grafana owned by it. It grants nothing in core_data, so Grafana cannot read app schemas, and it ends with a check that must print no rows. Run it on the VPS from the repo checkout:

sudo docker ps --filter name=postgres   # the data-services one, not dokploy-postgres
sudo docker exec -i <postgres-container> psql -U core -d core_data \
  < infrastructure/data-services/grafana-database.sql

psql runs inside the container and reads the container's own POSTGRES_PASSWORD, so the password never appears on a command line. Postgres's /docker-entrypoint-initdb.d does not help here: init scripts run only against an empty data directory, and postgres_data is not empty.

Cut over (or recover after Grafana's database is lost):

1. Run the SQL above.
2. Set POSTGRES_PASSWORD (the data-services value) in the observability stack's Dokploy Environment tab.
3. Redeploy the stack so grafana is recreated. Grafana creates its tables on first start and re-provisions everything from files. Its old SQLite data needs no migration (on 2026-10-10 it was already gone).
4. Log in as admin / admin and change the password right away. A fresh database starts with that default, and port 3010 is public.
5. Administration → Users and access → Service accounts: create coto with role Viewer, add a token. Viewer is enough for everything coto does with that token: the rules API, the Loki/Prometheus/Tempo datasource proxy, /api/ds/query for ClickHouse, dashboards and annotations. Silences use the separate GRAFANA_SILENCE_TOKEN.
6. Put the token in tools/coretower/coto.env as GRAFANA_TOKEN=, then run coto doctor. Its grafana line shows the rule count and checks every rule in core-alert-rules.yml is loaded.

Grafana cannot create users from files or environment variables (only the first admin), so a backup basic-auth login for coto is not provisioned. Instead, when the token is rejected, coto watch --send posts a 🔴 "Alert pipeline is blind" incident and pages through ntfy (coretower SETUP).

Step 4 — Registry credentials

The GHCR images are private. In Dokploy → Registry, add:

• Registry URL: ghcr.io
• Username: your GitHub username
• Password: a PAT with read:packages

Without this, Swarm tasks sit in Rejected with no such image and no containers start.

Step 5 — Application stack

Create a Dokploy Compose service pointing at docker-compose.yml (repo root).

Set Compose Type to Stack, not Docker Compose. This is the setting that makes the whole thing zero-downtime. In Compose mode, Docker silently ignores deploy.replicas, update_config, and restart_policy — you get 1 replica per service and a stop-then-start gap on every deploy.

Then:

1. Fill in the Environment tab from Step 1.
2. Add a Domain:
   • Host: api.toolzerhub.com
   • Service name: api (must match the compose service key)
   • Container port: 2626
3. Deploy.

Adding a domain does nothing until you redeploy. Traefik labels live in the Swarm service spec, which is only rewritten by docker stack deploy. Saving the domain in the UI updates Dokploy's database, not Swarm. Symptom is a bare 404 on every request with no incoming requests in the API logs, while docker service ls still shows a healthy 2/2 — the containers are fine, nothing is routing to them. This has caused one production outage.

Step 6 — CI wiring

In the GitHub repo settings, add secret DOKPLOY_DEPLOY_WEBHOOK — the deploy webhook URL from the application stack's Dokploy settings.

After that, a main push affecting backend or image inputs runs tests and contract checks alongside the image build, then promotes the images and triggers deployment only after every applicable check passes. A change limited to the root docker-compose.yml redeploys the application stack without rebuilding its images. Unrelated pushes skip the backend release jobs.

Verification

docker service ls

Both services must read 2/2. 1/2 means a task is failing its healthcheck; 0/2 usually means the image pull failed (check Step 4).

curl -s https://api.toolzerhub.com/health          # process liveness + build identity
curl -s https://api.toolzerhub.com/ready           # required dependency readiness

/health is the Swarm probe and returns 503 only while draining. /ready is monitored separately and returns 503 when Postgres, the worker, or the browser minter is unavailable. It exposes their three image identities only after the API itself has a baked identity; drift: true means they differ, which is normal mid-rollout. coto's deploy report warns when drift outlasts a rollout. Using it as Docker health would restart healthy API processes during a shared dependency outage. ClickHouse is reported as optional degradation.

Confirm Traefik is actually routing — this is the check that would have caught the outage above:

docker service inspect <stack>_api --format '{{json .Spec.Labels}}'

You want traefik.* entries here, in .Spec.Labels (service labels). If they appear instead under .Spec.TaskTemplate.ContainerSpec.Labels (container labels), Swarm ignores them and nothing will route.

Finally, watch an actual rollout overlap:

docker service ps <stack>_api

During a deploy a new task should reach Running while the old one is still Running, then the old one goes Shutdown. If the old task shuts down before a new one appears, start-first is not being applied — recheck that Compose Type is Stack.

Routine deploys

Merge to main. CI does the rest. Nothing manual.

The compose file pins the :main tag; docker stack deploy resolves that tag to a digest at deploy time and records it in the service spec, so a moved tag is what triggers the rollout.

Rollback

Undo the last deploy — works even though both specs say :main, because the previous digest is recorded in the service spec:

docker service rollback <stack>_api
docker service rollback <stack>_worker

Go back to a specific older build. Every CI run also pushes an immutable sha-<full-sha> tag:

docker service update --image ghcr.io/thenetaji/core-api:sha-<full-sha> <stack>_api

Swarm also rolls back automatically when a new task fails its healthcheck (failure_action: rollback). Note this only catches unhealthy — a build that boots fine but returns wrong data will not trigger it, and is exactly the case where you need the commands above.

How zero-downtime works

Two mechanisms, both required. Either alone still drops requests.

1. Overlap — order: start-first. Swarm starts the replacement task and waits for its healthcheck before signalling the old one. With parallelism: 1 and 2 replicas, half the capacity is always serving a known-good build.

2. Draining. Traefik learns a container is going away by watching Docker, which lags several seconds. If the process stopped accepting the instant SIGTERM landed, everything Traefik routed in that gap would 502. So both services invert the order:

t=0     SIGTERM arrives
t=0     API and worker /health flip to 503 — sockets stay open, still serving
t=0..20 Traefik notices, pulls this replica out of rotation
        requests already in flight finish normally
t=20    nobody is routing here anymore -> close the listener
t=20+   flush ClickHouse, Postgres, traces -> exit

That is all apps/api/src/lifecycle.ts and apps/worker/src/lifecycle.py do: hold a boolean that /health reads. The worker additionally installs its SIGTERM handler ahead of uvicorn's and hands the signal back once the drain finishes.

Tuning the drain

SHUTDOWN_DRAIN_MS (api) and SHUTDOWN_DRAIN_SECONDS (worker) must stay below stop_grace_period (60s), which also has to cover in-flight requests and the final flush. The drain needs to exceed however long Traefik takes to notice a failing healthcheck. 20s is conservative; measure before lowering.

Gotchas

Swarm silently ignores these compose keys. They are removed from docker-compose.yml on purpose:

Service discovery. Service names resolve to a Swarm VIP that load-balances across replicas at the TCP layer — per connection, not per request. With keep-alive, one connection pins to one replica forever and the other idles. packages/fetch/index.ts sends Connection: close on the worker client specifically to defeat this. Do not "optimise" that back to keep-alive without re-checking replica balance.

Cross-stack hostnames. Within one stack, the short service name (worker) resolves. Across stacks on the shared dokploy-network, you need the fully-qualified <stack-name>_worker. This is why api and worker live in one stack rather than two.

@thenetaji/packages publishing. The publish workflow fires on any push to main touching packages/** and publishes unconditionally. If the version in packages/package.json was already published, it fails with a 409 Conflict. Bump the version in the same commit as any packages/ change.