Auth and credits

API keys (/v1, /v2)

Machine auth for the metered data API. Checked by apiKeyAuth (apps/api/src/middlewares/apiKeyAuth.middleware.ts) on every /v1 and /v2 request via the x-api-key header.

Format. tz_hub_<64 hex chars> or tz_vantage_<64 hex chars> — the issuing product followed by 32 cryptographically random bytes encoded as hex (apps/api/src/routes/hub/hub.keys.ts).

POST /hub/keys accepts product: "hub" | "vantage", defaulting to "hub" for existing callers. This is issuance metadata, independent of signup product, channel, and API permissions. The database stores it as issued_product and responses expose it as product. Existing keys are classified as Hub without changing their hashes or masked values; they remain valid until rotated or revoked. Product cannot be changed through key updates.

The full plaintext key is shown to the caller exactly once, at creation (POST /hub/keys), and never stored or made recoverable again.

Hashing. Only the sha256 hash of the key lands in platform.api_keys.key_hash (apps/api/src/db/schema/platform.ts). A database leak alone doesn't hand over usable keys.

Why sha256, not a KDF (scrypt/bcrypt/argon2). A deliberately slow KDF is for hashing low-entropy human passwords, where an attacker can afford to guess. A newly issued API key here is 32 bytes of CSPRNG output — brute-forcing the hash is infeasible regardless of hash speed — and this hash is looked up on every single API request, so a slow KDF would dominate request latency for no security benefit. (Session tokens and password-reset tokens follow the same reasoning — see platform.sessions.token_hash and platform.password_reset_tokens.token_hash, both plain-hashed high-entropy secrets. User passwords are the one place apps/api does use a slow KDF: platform.users.password_hash is a scrypt digest.)

Deletion. DELETE /hub/keys/{id} permanently removes that workspace-scoped key, so it immediately fails the same "Invalid or missing API key" check as a key that never existed. A key id from another workspace returns the same 404 as a missing id. Existing billing rows survive with api_key_id = NULL; ClickHouse request history is retained separately.

Rotation. POST /hub/keys/{id}/rotate atomically replaces the secret and returns the new plaintext exactly once, using the stored issuing product and fresh 32-byte randomness. Older tzhub_live_ keys rotate to tz_hub_. The database id, creator, name, channel, metering, budget period, and usage attribution stay unchanged. Preserving the budget counters is security-relevant: rotating a key must not refill its monthly limit. The old plaintext stops authenticating as soon as the update commits.

last_used_at throttling. Updated fire-and-forget, and only when stale by more than 5 minutes (LAST_USED_STALE_MS in apiKeyAuth.middleware.ts). A write on every request would be write amplification on the hottest path in the API for a value that only needs minute-scale freshness.

The legacy shared key

TOOLZERHUB_API_KEY is a single shared secret, checked first (via timingSafeEqual, constant-time) before the per-workspace lookup. A match resolves to a fixed reserved workspace (00000000-0000-4000-8000-000000000001, seeded by migration 0021_toolzerhub_seed.sql) and is marked metered: false.

This is deprecated but not removable yet. Every published Apify Actor currently uses this key. It gets a much higher rate-limit ceiling than a normal key (LEGACY_KEY_TIER, 6000/min vs. API_KEY_TIER's 300/min — see apps/api/src/middlewares/rateLimit.middleware.ts) because it covers the whole fleet. Its compatibility value remains committed and first-checked until every Actor migrates. During a future rotation, override the primary value and accept the outgoing value through TOOLZERHUB_API_PREVIOUS_KEY; clear the latter after every client moves. Keep JOBS_ID_SECRET stable so API-key rotation does not invalidate issued job IDs. See env.md.

Unmetered callers never touch the credit wallet (see below), but their usage cost is still recorded for ClickHouse attribution via creditsCharged on the request context — see observability.md.

How the legacy key sits inside the per-key model below. It is channel = 'legacy', not 'apify', even though every published Actor authenticates through it. Two reasons, both about not corrupting the numbers the channel column exists to produce: it is unmetered, so it writes no credit_ledger rows at all and folding it into 'apify' would put a large request volume against zero credits and make that channel look free; and it is one shared secret fronting the whole fleet rather than a key issued to a workspace, so its traffic is not attributable to a customer. Keeping it separate also makes the migration legible — 'legacy' is the number that should fall to zero as Actors move onto per-workspace keys, which is invisible if it is already counted as 'apify'.

Its monthly_credit_limit is NULL and must stay that way: a ceiling on this one key is a ceiling on every Actor at once. It is doubly protected — creditMeter returns on !caller.metered before the budget path is reached at all.

Actor telemetry accepts active API keys whose creator is currently a platform admin. Their database identity is retained for attribution, but telemetry is always unmetered. Ordinary workspace keys cannot forge Actor events or dashboards. ACTOR_TELEMETRY_API_KEY is a deprecated compatibility boundary for older Actor builds: while set it is accepted alongside admin-created keys; when unset, the current and previous legacy shared keys are accepted instead. This compatibility path is intended for removal after the published fleet rotates. The dedicated compatibility credential is not accepted by ordinary /v1 data routes.

Consequently GET /hub/keys/{id}/usage would report real traffic and credits: 0 for this key. That is accurate rather than broken: nothing was ever charged for it. (The endpoint is unreachable for it in practice — see the reserved workspace note under the workspace model below.)

Channels and per-key budgets

A key carries two optional columns beyond its identity (apps/api/src/db/schema/platform.ts):

• channel — a slug for the distribution channel it was issued to ("apify", "rapidapi", "direct"). Free text, not an enum, because onboarding a marketplace is a sales event and must not be a migration. Attribution only; nothing authorises on it.
• monthly_credit_limit — a ceiling on what this one key may spend per 30-day period. NULL means uncapped, and that is the default for every key, including a workspace's first key and the legacy shared Actor key. A cap is something a channel key is deliberately opted into.

Why a column rather than a member user per marketplace: a marketplace is not a person. It has no password to rotate, no session, and nothing to sign in to. Every mechanism that makes per-channel tracking work — credit_ledger.api_key_id and ClickHouse's api_key_id — is already keyed on the key row, so a user row would add an identity to maintain and answer no question the column doesn't.

The budget exists for isolation, not billing. Keys in one workspace share one wallet by design (they are one customer with one bill), but without a ceiling a runaway loop on one channel silently consumes the credits every other channel needs, and the first symptom is another channel's customers getting 402s.

Enforcement happens in the same transaction as the wallet reservation. The transaction locks and reads the current key row rather than trusting the earlier authentication snapshot, so a cap added during a request cannot be bypassed. A capped key is booked before the wallet; if the wallet refuses, the whole transaction rolls both changes back. An uncapped key is still read and locked to make that decision authoritative, but its budget counters are not written.

Reads are split by store, deliberately: credits comes from the Postgres ledger, traffic from ClickHouse (GET /hub/keys/{id}/usage). ClickHouse is optional infrastructure here — every read in hub.usage.ts degrades to empty when it is unconfigured or unreachable — and a credit figure a customer is billed against cannot silently read zero because an analytics node is down. Volume, latency and error rates are cheap to lose; money is not.

Email verification (OTP)

A Toolzer Hub signup on a deployment with outbound email configured gets no session and no API key until a 6-digit code sent to the address is entered back (apps/api/src/routes/app/app.verification.ts). POST /auth/signup returns { verification_required: true }; POST /auth/verify-email returns the session and the workspace's first key.

The gate is before the session rather than after it because every new workspace is seeded with a free credit allowance, and an allowance claimable by anyone typing an address they don't own is an allowance that gets claimed in bulk. Verifying afterwards hands over the credits first and asks questions later.

A 6-digit code is ~20 bits — low enough that an attacker can afford to guess, unlike the 32-byte secrets everywhere else in this document. Three defences, none sufficient alone: an attempt cap counted in Postgres (not in memory — the API runs several replicas, and an in-process counter would give a guesser the full cap per replica), a 10-minute expiry, and authRateLimit on the route, which is what stops a guesser simply requesting a fresh code each time they burn one. Unknown address, wrong code and expired code are deliberately the same 400, for the same anti-enumeration reason as /auth/password-reset.

isVerificationRequired() is tied to isEmailConfigured() rather than a separate flag: a deployment that cannot deliver a code cannot demand one, and two facts that can disagree is how the gate ends up silently off in an environment where email works fine. Login checks email_verified_at unconditionally, though — turning email off must not promote every half-finished signup into a working login. Rows predating this flow were backfilled to their own created_at by migration 0026; a NULL now genuinely means "signed up and never verified".

Vantage signups are deliberately not gated: apps/web/vantage has no code-entry screen, so returning verification_required to a client that only reads token would make signup appear to hang. That is a scope boundary, not a judgement that Vantage's trial credits matter less.

Sessions (/hub, /app)

Human auth for the two dashboards, checked by sessionAuth (apps/api/src/middlewares/sessionAuth.middleware.ts). Deliberately distinct from apiKeyAuth: this answers "which human is this, and are they a member of the workspace they're asking about", not "is this a paying machine".

• The token comes from either the Authorization: Bearer header (server-side Next.js rendering) or the vantage_session cookie (browser fetches) — same session mechanism serves both, so a Toolzer Hub session and a Vantage session are interchangeable.
• Only the token's hash is stored (platform.sessions.token_hash), same rationale as API keys.
• Expiry and revocation are enforced in the SQL query itself (expires_at > now(), revoked_at IS NULL), not in application code — a clock-skewed app server can't accidentally honor a dead session.
• Workspace membership is re-verified on every request (not trusted from the token), so revoking a member takes effect immediately rather than at token expiry.
• A requested workspace (x-workspace-id header) the caller isn't a member of 403s identically to one that doesn't exist, for the same anti-probing reason as revoked API keys above.
• sessionAuth resolves identity and tenancy only — no product entitlement. That's requireCapability (apps/api/src/middlewares/requireCapability.middleware.ts), applied per-router: /hub requires capability "api", most of /app requires capability "rankings".

Workspace, subscription, and allowance model

• Workspace (platform.workspaces) is the customer/billing boundary. It holds no product configuration of its own. Lifecycle — create, list, rename, soft-delete, leave, and role changes — is apps/api/src/routes/hub/hub.workspace.ts (/hub/workspaces, /hub/workspace). Until that module existed a workspace could only be created by signing up, one per account, permanently, which is why the multi-workspace session model (x-workspace-id) already existed with nothing to point at.
• A Toolzer Hub signup creates exactly one workspace, named "Default" — there is no naming step, and the name is no longer derived from the person or their email. A derived name reads like something the customer chose and therefore like something they shouldn't touch; "Default" reads as what it is and invites renaming. The slug still derives from the email's local part, because that one has to be unique: slugify("Default") would collide for every account on the platform and push all but the first into a 4-character random suffix, which starts failing signups on the unique index in the low hundreds.
• The reserved workspace (00000000-0000-4000-8000-000000000001, "Toolzer Hub Internal") is seeded by migration 0021 with no members, no subscription and no credit balance — it exists only to own the legacy shared key. Because sessionAuth only ever selects a workspace the caller is a member of, no session can name it, so every /hub/workspace* endpoint is unreachable for it. That is one INSERT away from being false and the blast radius is the whole Actor fleet, so deleteWorkspace refuses it explicitly on both the customer surface (hub.workspace.ts) and the admin surface (admin.service.ts), both importing the one LEGACY_WORKSPACE_ID constant rather than re-typing the uuid.
• Two guards exist because their absence is silent until it strands someone: a workspace is never left with no owner, and a user is never left with no workspace — a state sessionAuth reads as 403 on every request, with no screen offering a way out. Changing roles is owner-only, stricter than the other team operations, because an admin who could promote themselves to owner could then delete the workspace.
• Subscription (platform.subscriptions) is one row per (workspace_id, capability) — a workspace can hold "api", "rankings", and "influencers" independently, each with its own plan, status, and quantity. status values are trialing | active | past_due | canceled | paused; trialing, active, and past_due all still grant access (ENTITLED in requireCapability.middleware.ts) — a failed card doesn't cut off a paying customer.
• Credit balance (platform.credit_balances) is one row per workspace, not per capability — a single shared credit wallet spent by both /v1//v2 API metering and Vantage's on-demand actions (scans, creator intel).
• Allowance is credit_balances.allowance, a materialized sum of ALLOWANCE[capability][plan] (apps/api/src/routes/app/app.allowance.ts) over every entitled subscription the workspace holds. It's recomputed (recomputeAllowance) whenever a subscription changes, not read live on every spend.
• Refill is lazy, not cron-driven: the shared ensureCreditPeriod routine checks reads and spends, locks the wallet row, and resets an elapsed period only when no unresolved reservation belongs to it. That includes executing work: it remains attached to its original period until settlement, so an ambiguous upstream call cannot receive a fresh-period refill before it is charged. The lock and fresh reservation check prevent a concurrent request from losing its deduction during rollover.
• A workspace's very first spend, before any subscription has ever computed a real allowance, falls back to 25 credits (FALLBACK_NEW_WORKSPACE_ALLOWANCE).

Credit reservation lifecycle (/v1, /v2)

creditMeter (apps/api/src/middlewares/creditMeter.middleware.ts), mounted after apiKeyAuth and apiKeyRateLimit, prices the request by longest-matching path prefix in CREDIT_COST_API (default 1 credit; /v1/etsy costs 3, /v1/ad-library costs 4, /v1/linkedin-pro costs 2 — shipping a new route is never blocked on a pricing decision).

For a metered caller (any per-workspace API key):

1. Reserve, before the handler runs. reserveMeteredCredits atomically decrements the wallet, books any key ceiling, and inserts platform.credit_reservations. It persists both original period identities, so a delayed release cannot credit a later allowance.
2. Begin execution durably, before the handler runs. The record changes from pending to executing; if that write fails, no upstream work starts.
3. On success (handler returned without throwing, and the final response status is < 400): the record is marked succeeded, then committed with one ledger row. A commit outage preserves succeeded; the reconciler safely retries it rather than losing the charge. If both the success marker and its indeterminate fallback are unavailable, the original 2xx still returns: the durable executing record is later settled charge-side rather than inviting a client retry of work that may already have run.
4. On an explicit failure (the handler threw, or the response status is >= 400): the record synchronously releases wallet and key budget only when their saved periods still match. An expired pending record becomes expired, never free credits in the new period. If that explicit release cannot be recorded, the durable executing state is conservative ambiguity and is charged on expiry.

The >= 400 check is why 404 is not charged: it's not special-cased on its own, it's just one instance of the general "any non-2xx/3xx response releases rather than commits" rule. (Separately, apps/api/src/routes/hub/hub.usage.ts's dashboard analytics do special-case 404 — excluding it from error-rate figures shown to customers, since a 404 from a data route, e.g. "no videos for this TikTok user", is a valid answer, not a failure. That's a display concern in the usage dashboard, not part of the credit lifecycle itself.)

For an unmetered caller (the legacy shared key): the wallet is never touched. creditsCharged is still set on the request context so the cost is visible in ClickHouse attribution, but nothing is reserved, committed, or released.

credit_reservation_reconcile runs hourly behind its own advisory lock, claims at most 100 rows with FOR UPDATE SKIP LOCKED, and settles succeeded/indeterminate records as charges. It releases only expired pending records (where execution never began); an expired executing record is commit-side ambiguity and is charged once. Its backlog separates safe expired-pending releases from charge-side work, excludes normal in-flight requests, and alerts only after five records persist for 15 minutes.

Vantage's own credit spends

Outside the /v1//v2 meter, apps/api/src/routes/app/app.credits.ts also exposes spendCredits/refundCredits for Vantage's own synchronous actions (scan_now, keyword_first_scan, creator_intel, ad_intel — priced in CREDIT_COST). These always write a ledger row up front and compensate with a second row on failure, rather than reserve/commit/release — a different shape from the API meter because Vantage's spends are already-decided at the point they're made, not provisional.

Marketplace ingress evidence

Admin-created RapidAPI-channel keys cannot access platform account, credits, or usage routes. Both free and paid consumers must use a published service route.

Free consumers have fixed per-consumer/service allowances defined in MARKETPLACE_FREE_LIMITS, shared across equivalent listings. Counters reset at UTC calendar month and hour boundaries; these local windows are independent of RapidAPI billing resets. Paid consumers bypass these allowances. The former monthly and per-minute quota env overrides are removed.

MARKETPLACE_TRUSTED_PROXY_CIDRS lists verified ingress proxy IPs/CIDRs. It defaults to empty: quotas still apply, but request headers contribute no IP evidence. The resolver verifies the Node socket peer, then walks X-Forwarded-For right-to-left, stopping at the first untrusted address. Only list proxy networks whose forwarding behavior has been verified to append or overwrite addresses from the actual peer; include any gateway hops needed to reach the consumer. Do not list end-user networks or assume a shared proxy/gateway egress address identifies a consumer. Fetch-only runtimes without socket metadata and malformed chains contribute no IP evidence.

Verified IP fingerprints and restrictions carry a new evidence marker so historical unverified IPs cannot support new decisions or continue enforcing old restrictions. Already verified consumer restrictions remain enforceable if a later request omits forwarding headers. Changing proxy trust requires validating the actual ingress chain; no deployment CIDRs are inferred or enabled by this change.

RapidAPI free traffic also shares a temporary daily estimated-cost budget in MARKETPLACE_FREE_LIMITS, across consumers, services, listings, keys, and replicas. The store reserves integer microdollars in the admission transaction after quota and risk checks; budget exhaustion rolls that transaction back and returns 429 with free_access_daily_budget and Retry-After until the next UTC midnight. The budget applies in both observation and enforcement modes. Paid RapidAPI, ordinary API keys, and Apify traffic bypass it. No additional env is required.

estimateMarketplaceCostMicrousd uses route credit weights with a buffer. This is a trial estimate, not vendor billing or a guaranteed ceiling on actual spend; retries, model usage, and provider price changes can differ. Once admitted, a request retains its reservation even if it fails, because execution may already have incurred cost. Restarting a replica does not reset the daily total. Budget rows are retained for seven days.