Worker Architecture

Overview

Self-hosted HTTP fetch worker with dual-path architecture: fast stateless requests and Redis-backed stateful sessions. All traffic routed through residential/datacenter proxies via Scrapling with TLS fingerprint impersonation.

Request Flow

   POST /fetch  ·  GET /health  ·  DELETE /sessions/{id}  ·  POST /pool/{site}/*  ·  site routes (11 sites)
                           │
                ┌──────────▼──────────┐
                │     FastAPI           │
                │  middleware/logging   │
                └──────┬───────┬───────┘
                       │       │
          session_id? NO     session_id? YES / site route
                       │       │
           ┌───────────▼┐  ┌──▼──────────────┐
           │ STATELESS  │  │    STATEFUL      │
           │ SimpleFetch│  │  StatefulFetcher │
           │            │  │                  │
           │ no Redis   │  │ SessionPool      │
           │ fresh proxy│  │ · allocate       │
           │ retry on   │  │ · release        │
           │ blocked    │  │ · quarantine     │
           └─────┬──────┘  └────────┬─────────┘
                 │                  │
                 └────────┬─────────┘
                          │
                 ┌────────▼────────┐
                 │   HttpFetcher    │
                 │   (curl_cffi)    │
                 │   impersonate:   │
                 │     chrome       │
                 │   stealthy_hdrs  │
                 └────────┬────────┘
                          │
                 ┌────────▼────────┐
                 │  Proxy Manager   │
                 │  (Evomi/Decodo)  │
                 │  + Health track  │
                 └────────┬────────┘
                          │
                 ┌────────▼────────┐
                 │  Redis (shared)  │
                 │  · sessions      │
                 │  · pool state    │
                 │  · rate limits   │
                 │  · locks         │
                 └─────────────────┘

Stateless Path

No Redis involvement beyond the global rate limiter. Fresh proxy per request. Proxy rotates on block. Implemented in core/simple_fetcher.py, which has its own inline retry loop — it does not go through core/pipeline.py.

Stateful Path

The caller provides a session_id — the worker persists cookies + sticky proxy identity in Redis. Subsequent requests with the same session_id reuse the cookie jar and exit IP. Cookie merge rule: stored ∪ caller_cookies ∪ response_cookies (response wins). Implemented in core/stateful_fetcher.py, which uses core/pipeline.py for the retry loop and block detection.

Site Routes (11 sites)

Thin convenience wrappers that build FetchRequest objects and delegate to SiteRouteEngine (sites/stateful.py). Sessions are managed by a per-site SessionPool (sites/pool.py) — the client never sees session mechanics. Pool size (min_idle/max_idle/max_total) is per-site, set in sites.yaml; only instagram, facebook, and meta have a docs page today (see "Sites without a doc page" below for the rest).

Most routes let SiteRouteEngine allocate a pool session, use it, and return it. tiktok_shop is the exception: a single API request there is a multi-leg chain that must run on one identity across legs, so its caller leases a session explicitly via POST /pool/{site}/lease and holds it across the chain — see api/routes/pool.py.

Module Map

src/
├── main.py                    # uvicorn entry point
├── app.py                     # FastAPI factory + lifespan (create_app())
├── settings.py                # pydantic-settings (core env vars)
├── cli.py                     # `uv run serve|dev|serve-fast|dev-fast` entry points
├── lifecycle.py                # Process-level drain flag, set on SIGTERM/SIGINT so
│                                # /health fails before uvicorn stops accepting
│
├── api/                       # HTTP interface
│   ├── schemas.py             # FetchRequest / FetchData / BrowserMintRequest models
│   ├── responses.py           # ok() / fail() — raw payload, no envelope wrapper (see below)
│   ├── errors.py              # AppError hierarchy
│   ├── middleware.py           # request-id, timing, error handlers
│   └── routes/
│       ├── fetch.py           # POST /fetch
│       ├── health.py          # GET /health
│       ├── pool.py            # POST /pool/{site}/lease|release|quarantine
│       ├── sessions.py        # DELETE /sessions/{id}, POST /sessions/{id}/refresh
│       └── browser.py         # DEAD — not imported/routed by app.py, see fetchers/browser.py
│
├── core/                      # Orchestration
│   ├── fetch_service.py       # Routes to stateless/stateful fetchers
│   ├── simple_fetcher.py      # Stateless path — own inline retry loop, no pipeline.py
│   ├── stateful_fetcher.py    # Stateful path — uses pipeline.py for retry/block detection
│   ├── pipeline.py            # Retry loop + block detection (stateful path only)
│   └── session/
│       ├── loader.py          # SessionBundle create/load/reset
│       ├── persister.py       # Cookie + metadata persistence
│       └── lock_manager.py    # Distributed Redis locks
│
├── fetchers/                  # HTTP backends
│   ├── base.py                # Fetcher Protocol + RawResponse
│   ├── http.py                # HttpFetcher (Scrapling/curl_cffi)
│   ├── browser.py             # DEAD — Camoufox browser mint; camoufox not installed,
│   │                          # module not importable (pyproject.toml dependency comment)
│   └── registry.py            # Singleton fetcher access
│
├── proxy/                     # Proxy providers
│   ├── base.py                # ProxyProvider Protocol
│   ├── manager.py             # ProxyManager
│   ├── health.py              # ProxyHealthTracker (cooldown on block)
│   ├── usage.py               # EvomiUsageReconciler — what Evomi billed, against
│   │                          # what attribution.py could name a route for
│   └── providers/
│       ├── evomi.py           # Evomi residential + core-residential + datacenter
│       └── decodo.py          # Decodo residential + datacenter
│
├── sessions/                  # Redis-backed storage
│   ├── bundle.py              # SessionBundle dataclass
│   ├── store.py               # Redis-backed persistence
│   └── locks.py               # SessionLock (Lua scripts)
│
├── ratelimit/                 # Concurrency control
│   └── limiter.py             # GlobalLimiter (Redis token permits)
│                              # Per-site limits are NOT here — they are enforced
│                              # by SessionPool.allocate() (sites/pool.py), as
│                              # both a concurrency cap and, where the site's
│                              # ceiling is a rate, a per-identity cooldown.
│
├── blocking/                  # Anti-bot detection
│   └── detector.py            # is_blocked() signature matching
│
├── docext/                    # Invoice/receipt extraction — CPU OCR, no site, no proxy.
│   │                          # Optional deps: `uv sync --group docext` (pyproject.toml)
│   ├── pages.py               # bytes -> page rasters, size/page/edge caps enforced first
│   ├── ocr.py                 # PaddleOCR engines, one per process, 2 concurrent docs
│   ├── rules.py               # Engine A: layout rules -> fields, no LLM
│   ├── prompt.py              # Engine B: OCR lines -> strict-JSON instruction, and back
│   ├── scoring.py             # Field-level grading against dataset ground truth
│   └── schemas.py             # InvoiceExtraction / ReceiptExtraction + per-field confidence
│
├── observability/             # Logs, metrics, traces
│   ├── logging.py             # Structured JSON logging
│   ├── attribution.py         # Which site/route spent proxy bandwidth; the
│   │                          # route ContextVar every fetch and mint reads
│   ├── browser_bandwidth.py   # Per-mint byte totals off Playwright request sizes
│   ├── metrics.py             # OTel Metrics API instruments — no local /metrics endpoint,
│   │                          # emission is 100% OpenTelemetry (see otel.py)
│   └── otel.py                # OTLP bootstrap; no-op unless OTEL_EXPORTER_OTLP_ENDPOINT is set
│
└── sites/                     # Site adapter framework
    ├── stateful.py            # StatefulSiteAdapter protocol + SiteRouteEngine
    ├── pool.py                # SessionPool (allocate/release/quarantine, Redis-backed)
    ├── pool_manager.py        # SitePoolManager (per-site pools, warmup metadata extractors)
    ├── config_loader.py       # sites.yaml loader
    ├── registry.py            # Auto-discovery from sites.yaml
    ├── _template/             # Scaffold for new site adapters
    ├── instagram/              # documented: docs/sites/instagram.md
    │   ├── adapter.py           # Request builder + response transformer
    │   ├── routes.py            # Worker-internal routes
    │   ├── state.py             # IgFingerprint dataclass
    │   ├── config.py            # InstagramSettings (env vars)
    │   └── schemas.py            # Request schemas
    ├── facebook/                # documented: docs/sites/facebook.md — public logged-out
    │   │                        # Facebook, NOT the Ad Library
    │   ├── adapter.py             # Request builder + response transformer
    │   ├── routes.py              # Worker-internal routes
    │   ├── state.py               # FacebookFingerprint + guest-token extraction
    │   ├── parsers.py             # SSR blob + streamed-Relay decoding
    │   ├── operations.py          # Observed Relay operation contracts
    │   ├── failures.py            # Failure-kind taxonomy (stale_doc_id, transient, ...)
    │   ├── config.py              # FacebookSettings (env vars)
    │   └── schemas.py              # Request schemas
    ├── meta/                       # documented: docs/sites/meta.md — Ad Library only
    │   ├── adapter.py               # Request builder + response transformer
    │   ├── routes.py                # Worker-internal routes
    │   ├── state.py                 # MetaFingerprint dataclass
    │   ├── config.py                # MetaSettings (env vars)
    │   └── schemas.py                # Request schemas
    │
    │                                 # Same shape (adapter/routes/config/schemas, plus
    │                                 # state.py/parsers.py where the site needs them) —
    │                                 # no doc page yet, see "Sites without a doc page":
    ├── ads_transparency/             # Google Ads Transparency Center
    ├── ebay/
    ├── linkedin/
    ├── linktree/
    ├── threads/
    ├── tiktok_ads/                    # TikTok Ads Library + Creative Center Top Ads
    ├── tiktok_shop/                    # POST-based, session-chained (page -> api legs)
    └── walmart/

Sites without a doc page

ads_transparency, ebay, linkedin, linktree, threads, tiktok_ads, tiktok_shop, and walmart are registered, enabled (sites.yaml), and mounted in a full-mode worker, but have no page under docs/sites/. Route behavior for these lives only in each site's routes.py + schemas.py — read those directly. tiktok_shop is the one most worth documenting first if this list gets picked up: its two routes are POST (every other undocumented site is GET) and require the caller to drive a multi-leg session chain via session_id, which isn't obvious from the route signature alone.

Bootstrap Sequence

create_app() (src/app.py) — configure logging, validate proxy credentials (settings.check_proxy_credentials(), raises if misconfigured), load sites.yaml, and — in full mode only — discover site adapters into the registry
lifespan() startup:
- Create the Redis client (5s socket/connect timeout, 30s health-check interval)
- Create GlobalLimiter, SessionStore, ProxyManager, ProxyHealthTracker
- Create SimpleFetcher (stateless path)
- Create SessionLoader, SessionPersister, LockManager, Pipeline, then StatefulFetcher (stateful path)
- Create FetchService(simple_fetcher, stateful_fetcher)
- In full mode only: create SitePoolManager, build a pool per enabled site from sites.yaml, and await pool_manager.warmup_all()
- Create SiteRouteEngine (wraps fetch service, session store, persister, lock manager, pool manager)
- Mount everything on app.state
- Install SIGTERM/SIGINT drain handlers (lifecycle.py) if SHUTDOWN_DRAIN_SECONDS > 0
lifespan() shutdown — restore signal handlers, close fetchers, close Redis, shut down OTel providers

Routers mounted unconditionally: health, fetch, sessions. Routers mounted only in full mode: pool and one router per registered site adapter — a stateless-mode worker exposes none of the site routes or the pool lease API. docs_url is /swagger; a separate /docs route serves a Scalar-rendered API reference over the same OpenAPI schema.

Adding a New Site

Copy src/sites/_template/ to src/sites/yoursite/
Implement build_fetch_request() and transform_response() in adapter.py
Define request schemas in schemas.py
Add routes in routes.py
Add config in config.py (site-specific env vars)
Add entry to sites.yaml and src/sites/registry.py's _SITE_ADAPTER_MAP

The adapter class name is derived mechanically from the site name in registry.py (tiktok_shop → TikTokShopAdapter, facebook → FacebookAdapter), and a site missing from _SITE_ADAPTER_MAP is skipped silently rather than raising — so a typo in either place looks like a site that simply never registered.

If the site's POST operations need tokens that live in the page body rather than in cookies (lsd, jazoest, spin/haste values), register a warmup extractor in pool_manager.py::_load_metadata_extractor — the pool hands it the warmup response body and merges the result into session metadata, so sessions arrive pre-warmed instead of bootstrapping per request.

Response Envelope

api/responses.py does not wrap responses in a {success, data, error} object. ok(data, request_id) returns data itself as the JSON body (object, list, or null) with x-request-id/x-trace-id response headers carrying the request id — the HTTP status line carries success/failure. fail(message, code, type, request_id, context) returns {"type": ..., "message": ..., "context": ...} with code as the HTTP status. There is no success or metadata field in either case. AppError subclasses in api/errors.py map to fixed (code, type) pairs: ValidationError (400, VALIDATION_ERROR), FetchError (502, FETCH_ERROR), ContractError (502, CONTRACT_ERROR), UpstreamBlockedError (503, UPSTREAM_BLOCKED), ProxyExhaustedError (503, PROXY_EXHAUSTED), RateLimitedError (429, RATE_LIMITED); an uncaught exception falls through to AppError's own default, 500 INTERNAL_ERROR.

Source of Truth

What	Where
API request/response schema	`apps/api/src/routes//.schema.ts` or `/openapi.json`
Worker envelope	`apps/worker/src/api/responses.py`
Worker route behavior	`apps/worker/src/sites/*/routes.py` + `schemas.py`
Architecture	This file (update on module boundary changes)

On this page