Architecture

The app/worker split

This repo ships two separate services that together answer a /v1 request:

• apps/api (this directory) — a Node/Hono service. It handles auth, credits, rate limiting, request/response shaping, and talks to two kinds of upstream: third-party APIs (RapidAPI, TikHub) called directly, and this repo's own Python worker for sites that need a stateful, proxied browser session.
• apps/worker — a Python/FastAPI service (apps/worker/src/app.py). It owns proxy pools, session persistence, and anti-bot handling for sites that can't be scraped with a bare HTTP request — e.g. Etsy runs DataDome, so a real proxied browser has to mint a validated session before fast HTTP fetches can use it (see the Etsy tag description in apps/api/src/app.ts).

A site's service file picks one path. Sites that go through the worker call buildUrl(path, params) (apps/api/src/utils/buildUrl.ts), which resolves against WORKER_HOST — see apps/api/src/routes/walmart/walmart.service.ts, ebay.service.ts, etsy.service.ts, pinterest.service.ts, meta.service.ts, instagram.service.ts, threads.service.ts, linkedin.service.ts, rednote.service.ts for examples. Sites backed by RapidAPI or TikHub call out directly via UpstreamHandler subclasses (below), never touching WORKER_HOST.

Vantage's background scans (apps/api/src/jobs/) also go through the worker rather than a metered vendor — see the policy comment at the top of apps/api/src/routes/app/app.credits.ts.

Why packages/ exists

packages/ (@thenetaji/packages) is shared TypeScript code used by apps/api only — not apps/web, and not apps/worker (which is Python). It holds three things apps/api needs everywhere in its codebase and would otherwise duplicate per-route:

• errors/ — the AppError hierarchy (BadRequestError, UnauthorizedError, ForbiddenError, PaymentRequiredError, NotFoundError, RateLimitError, ServiceUnavailableError, SourceError, TimeoutError, ...). One error thrown anywhere in a route handler is caught once, centrally, and turned into a consistent HTTP response — see handleGlobalError below.
• fetch/ — HttpClient, the shared HTTP client with retry/timeout behavior, used by every UpstreamHandler subclass to call third parties.
• logger/ — structured logging built on AsyncLocalStorage, so getLogger() returns the request-scoped logger from anywhere without threading a logger argument through every function call.

Request flow: /v1

Global middleware (mounted once, in apps/api/src/app.ts, in this order) runs on every request regardless of surface:

1. contextStorage() (hono/context-storage) — makes the active request's context reachable via getContext()/tryGetContext() from code with no ctx parameter (e.g. UpstreamHandler). Must wrap everything else, so it's mounted first.
2. sentry(app) — only if SENTRY_DSN is set.
3. cors()
4. requestId() (hono/request-id) — generates the per-request id everything else correlates on.
5. mountEnv — validates env once per process (apps/api/src/configs/index.ts) and puts it on the context.
6. Logger (apps/api/src/middlewares/createLogger.middleware.ts) — creates a request-scoped logger, logs http_request_started, and wraps the rest of the chain in runWithLogger so getLogger() works downstream. Logs the terminal http_request_completed event on the way out (or on a thrown error).
7. metricsMiddleware (apps/api/src/middlewares/metrics.middleware.ts) — records Prometheus/OTel HTTP metrics and fires the ClickHouse api_requests insert. See observability.md.
8. trimTrailingSlash(), prettyJSON()

Then the request reaches whichever router matched. For /v1 and /v2 specifically (apps/api/src/routes/index.ts):

9. apiKeyAuth (apps/api/src/middlewares/apiKeyAuth.middleware.ts) — resolves the caller from x-api-key and sets apiCaller on the context.
10. apiKeyRateLimit (apps/api/src/middlewares/rateLimit.middleware.ts) — mounted before creditMeter deliberately: a throttled request must never spend a credit.
11. creditMeter (apps/api/src/middlewares/creditMeter.middleware.ts) — reserves credits for metered callers before the handler runs. See auth-and-credits.md.
12. The route handler.

Inside a route: route → controller → service → normalizer

Each platform under apps/api/src/routes/<site>/ follows the same file split (see apps/api/src/routes/walmart/ for a concrete example):

• <site>.route.ts — the OpenAPI route definition (createRoute): path, method, tags, request/response schemas.
• <site>.controller.ts — the Hono handler: validates input (via the route's Zod schema), calls the service, wraps the result with ok(ctx, data) (apps/api/src/utils/response.utils.ts).
• <site>.service.ts — the actual upstream call, either buildUrl(...) to the worker or an UpstreamHandler subclass to a third party.
• <site>.normalizer.ts — reshapes the upstream's raw response into this API's stable response shape.
• <site>.schema.ts — Zod request/response schemas shared by the route and controller.

Out to the upstream and back

Third-party calls go through UpstreamHandler (apps/api/src/helpers/upstream/upstream-handler.ts), an abstract base every direct-to-third-party service subclasses. It:

• Records duration/status/bytes for every call, win or lose, via recordUpstreamInsight (apps/api/src/observability/upstreamInsight.ts) — stashed on the request context so metricsMiddleware can attach it to the api_requests row without threading it through every function signature.
• Translates failures into the public vocabulary: a transport failure (timeout/connection refused, no HTTP status at all) or a retryable HTTP status becomes ServiceUnavailableError (503); a non-retryable failure becomes SourceError (500). The specific upstream status/provider code is logged for diagnostics but never echoed to the client.
• Archives the raw response body (apps/api/src/ingestion/response-archiver.ts) unless the subclass opts out — see observability.md for what's optional there.

Back out through the response

• Success: the controller calls ok(ctx, data), which sets the X-Toolzer-Request-Id header and returns { data }.
• Failure: anything thrown (from validation, apiKeyAuth, creditMeter, the service, or the handler) is caught once by handleGlobalError (apps/api/src/helpers/globalErrorHandler.ts), which normalizes AppError subclasses into { error: { code, status, message, retryable, details } } via fail(ctx, ...) — same request-id header, same shape, regardless of where in the chain it was thrown.
• Zod validation failures on the request itself are caught earlier by handleZodValidationErrors (apps/api/src/helpers/zodError.ts, wired as defaultHook in apps/api/src/utils/createHonoInstance.utils.ts) and thrown as BadRequestError before the handler ever runs.

creditMeter's finally block runs after the handler settles (see auth-and-credits.md for exactly which outcomes commit vs. release), then metricsMiddleware's finally records the request, then Logger logs the terminal event.

Other surfaces, briefly

• /hub and /app use sessionAuth (apps/api/src/middlewares/sessionAuth.middleware.ts) instead of apiKeyAuth — cookie or bearer session token, resolved against platform.sessions. /hub additionally requires requireCapability("api"); most of /app requires requireCapability("rankings") (apps/api/src/middlewares/requireCapability.middleware.ts).
• /admin applies its own adminAuth inside the router rather than at the mount point, so the requirement travels with the routes.
• /internal keeps public Apify Store GET catalogue views separate from adminAuth-protected POST controls. The route module owns that split so alternate mounts preserve it.

OpenAPI generation

GET /openapi.json (apps/api/src/app.ts) builds the spec from every mounted route's createRoute schema, then strips operations tagged admin, internal, vantage*, hub, or disapproved* (PRIVATE_TAG_PATTERNS) before returning it — so the published spec, and the Fumadocs site built from it, never lists internal-only operations. GET /llm.txt renders the same spec (OpenAPI 3.1, via @scalar/openapi-to-markdown) as markdown for LLM consumption.