Environment Variables

Every variable the worker reads, with its default from source. Verified against src/settings.py, each src/sites/*/config.py, and src/observability/otel.py (2026-08-13). Copy .env.example to .env and fill in proxy credentials — the example file only covers a subset of what's below; see "Not in .env.example" note at the end.

None of these variables individually fail startup via a missing-value check — Pydantic gives every one of them a default. The one thing that does fail startup is proxy credentials: see "Proxy credentials" below.

Core (`src/settings.py`)

Variable	Default	Description
`REDIS_URL`	`redis://127.0.0.1:6379`	Redis connection string
`PORT`	`8000`	HTTP listen port
`LOG_LEVEL`	`info`	`debug` / `info` / `warning` / `error` / `critical` — validated, invalid values raise at load
`BASE_URL`	`""`	OpenAPI server URL override (only added to the OpenAPI `servers` list if non-empty)
`HOST`	`0.0.0.0`	Bind address — set to `127.0.0.1` behind a reverse proxy
`WORKER_MODE`	`full`	`full` (both paths, plus `/pool` and all site routes) / `stateless` (only `/fetch`, `/health`, `/sessions`) — validated
`MAX_CONCURRENCY`	`100`	Global Redis-backed in-flight fetch cap, shared across every replica (1–1000, validated)
`SHUTDOWN_DRAIN_SECONDS`	`20`	Seconds `/health` keeps returning 503 after SIGTERM before the process actually stops accepting, so a load balancer can drop the replica first. Set to `0` to disable draining.
`SESSION_TTL`	`1200`	Default session bundle TTL in seconds for stateless-path sessions (60–7200, validated)

Fetcher defaults (`src/settings.py`)

Variable	Default	Description
`DEFAULT_IMPERSONATE`	`chrome`	Default TLS/browser fingerprint (`chrome`, `firefox`, `safari`, `chrome_android`, ...)
`DEFAULT_STEALTHY_HEADERS`	`true`	Default stealthy-header strategy

Media analysis (`src/settings.py`)

Read by /media/transcribe and /media/scenes. The two caps are derived from measured CPU seconds per minute of video — research/ad-creative-intel/01-media-cost.md.

Variable	Default	Description
`MEDIA_MAX_DURATION_S`	`120`	Longest media analysed in one request. Beyond it the route answers 400 `media_too_large` rather than spending the caller's whole edge budget
`MEDIA_MAX_BYTES`	`104857600`	Largest file downloaded, enforced against `Content-Length` and again against bytes written
`MEDIA_DOWNLOAD_TIMEOUT_S`	`45`	Wall-clock budget for pulling the file, separate from the decode budget
`MEDIA_FFMPEG_TIMEOUT_S`	`60`	Hard kill for one ffmpeg/ffprobe invocation
`MEDIA_MAX_CONCURRENCY`	`2`	Concurrent analysis jobs. A RAM ceiling before a CPU one (~200MB resident per job)
`MEDIA_QUEUE_WAIT_S`	`5.0`	How long a caller waits for a slot before getting 429 `capacity_limited`
`MEDIA_WHISPER_MODEL`	`base`	Whisper model. `base` is baked into the image; anything else downloads on first use, inside a caller's request
`MEDIA_WHISPER_COMPUTE_TYPE`	`int8`	CTranslate2 compute type
`MEDIA_WHISPER_CPU_THREADS`	`2`	Threads per job
`MEDIA_MODEL_CACHE_DIR`	`""`	Weights cache. Empty uses `HF_HOME`, which the image sets to `/opt/hf-cache`
`MEDIA_FFMPEG_BIN`	`ffmpeg`	Overridable so a dev box can point at a static build
`MEDIA_FFPROBE_BIN`	`ffprobe`	Same, for the probe

Browser mint (`src/settings.py`) — currently dead code

Variable	Default	Description
`BROWSER_MAX_CONCURRENCY`	`2`	In-process cap on concurrent Camoufox browser mints (1–20, validated)
`BROWSER_MINT_TIMEOUT_MS`	`90000`	Browser mint timeout

These two settings exist and are validated, but have no effect: the browser mint path (src/fetchers/browser.py, src/api/routes/browser.py) is not imported or routed from src/app.py — see pyproject.toml's dependency comment. POST /browser/mint is not a live endpoint.

Proxy (`src/settings.py`)

Variable	Default	Description
`PREMIUM_PROXY_PROVIDER`	`decodo`	`decodo` / `evomi`; shared premium provider switch for API, worker and minter
`EVOMI_RESI_USER`	`""`	Evomi residential proxy username
`EVOMI_RESI_PASS`	`""`	Evomi residential proxy password
`EVOMI_CORE_RESI_USER`	`""`	Evomi Core Residential proxy username
`EVOMI_CORE_RESI_PASS`	`""`	Evomi Core Residential proxy password
`EVOMI_DC_USER`	`""`	Evomi datacenter proxy username
`EVOMI_DC_PASS`	`""`	Evomi datacenter proxy password
`EVOMI_API_KEY`	`""`	Read-only account API key, feeding both the balance watchdog and the usage reconciler. Optional: without it bytes are still attributed per route and scraping is unaffected — only the balance gauge and the comparison against the bill go missing.
`DECODO_API_KEY`	`""`	Read-only Decodo public API key. Optional; enables the same periodic balance gauge and low-balance alerts for metered Decodo subscriptions.
`PROXY_BALANCE_CHECK_INTERVAL_S`	`3600`	How often to read remaining Evomi and Decodo balances. `0` disables both watchdogs.
`PROXY_BALANCE_WARNING_MB`	`10240`	Balance below which a tier is reported as running down.
`PROXY_BALANCE_CRITICAL_MB`	`2048`	Balance below which a tier is critical. Must not exceed the warning threshold.
`PROXY_USAGE_CHECK_INTERVAL_S`	`3600`	How often to read Evomi's last-24h usage. `0` disables the reconciler.
`DECODO_RESI_USER`	`""`	Decodo residential proxy username
`DECODO_RESI_PASS`	`""`	Decodo residential proxy password
`DECODO_DC_USER`	`""`	Decodo datacenter proxy username
`DECODO_DC_PASS`	`""`	Decodo datacenter proxy password

Proxy credentials — the one thing that fails startup

Set PREMIUM_PROXY_PROVIDER=decodo (default) or evomi in the shared .env, then restart API, worker and minter together. This resolves premium.residential in sites.yaml, the API residential tier, and escalation defaults. Explicit core and datacenter specs stay pinned. Keep the selected provider’s credentials configured. Premium session and browser credential caches are separated by provider. Existing per-site escalation overrides take precedence. The outage failover table is separate and remains unchanged.

Settings.check_proxy_credentials() runs at create_app() and raises ValueError (crashing startup) if no complete username/password pair is configured. Enabled tiers are checked at boot too, so blank Core/Premium credentials cannot first surface on customer traffic.

A vendor with no credentials is absent, not fatal — a single-vendor setup is normal. What turns that into a failure is a site naming the absent vendor: validate_specs() runs right after sites.yaml loads and fails the boot, naming every offending site.

Proxies

A spec is <provider>.<tier>, e.g. evomi.core_residential. What each vendor sells lives in one table, PROVIDER_TIERS in src/proxy/selector.py:

Provider	Tiers
`evomi`	`datacenter`, `residential`, `core_residential`
`decodo`	`datacenter`, `residential`

Rules the code enforces, all of them loud:

Every site declares one. proxy is a required field on a site in sites.yaml; there is no default.
A spec is never substituted for a healthy provider. The single exception is a provider marked down, which is an outage, and it logs at ERROR (proxy_failover_engaged) every time. One hop only — the failover target is never in the provider that just failed.
A browser mint requires a residential tier. A datacenter exit cannot mint a usable credential, so BrowserSiteSpec and BrowserMintRequest both refuse one.
/fetch callers are held to the same table. FetchRequest.proxy runs the same validator, so no caller can reach a tier a site could not declare.

Observability (`src/observability/otel.py`)

Read directly via os.getenv, not through Settings — standard OpenTelemetry variable names, not worker-prefixed.

Variable	Default	Description
`OTEL_EXPORTER_OTLP_ENDPOINT`	unset	OTLP collector endpoint for traces/metrics/logs. Unset disables OTel entirely — `setup_otel()` becomes a no-op, so local dev without a collector runs unchanged. No local `/metrics` endpoint exists; metrics only ship if this is set.
`OTEL_SERVICE_NAME`	unset	Read by the OTel SDK itself, not by worker code, when the endpoint above is set
`OTEL_RESOURCE_ATTRIBUTES`	unset	Same — SDK-level, only consulted when OTLP export is active

The image build also accepts GIT_COMMIT and BUILD_TIME Docker build args. CI writes them to a root-owned read-only /app/.build-info.json; /health returns its validated values under build so an operator can confirm which image is serving traffic. Task environment cannot override this identity.

Site configuration variables

None of the variables below can fail startup — every site Settings subclass gives every field a default, and none has a field validator. Each site is its own pydantic-settings class with its own env_prefix.

Instagram — `IG_` (`src/sites/instagram/config.py`)

Variable	Default	Description
`IG_APP_ID`	`936619743392459`	Instagram app ID header
`IG_ASBD_ID`	`359341`	ASBD ID fallback (worker prefers a value extracted from Instagram itself)
`IG_POST_DOC_ID`	`25951076901259180`	GraphQL doc ID — `post_simple` operation (kept, not routed)
`IG_COMMENTS_DOC_ID`	`27261273046856309`	GraphQL doc ID — comments
`IG_HIGHLIGHTS_QUERY_ID`	`9957820854288654`	GraphQL query ID — story/live/highlight status
`IG_TIMELINE_QUERY_ID`	`27538170685818452`	GraphQL query ID — `timeline`, backs `/user/posts` and `/user/reels?detail=full`
`IG_PROFILE_ROOT_DOC_ID`	`27981003384861049`	GraphQL doc ID — logged-out profile root, backs `user_id`
`IG_REELS_TAB_DOC_ID`	`27838951732404191`	GraphQL doc ID — logged-out reels tab (`play_count`)
`IG_POST_ROOT_DOC_ID`	`27130156389949648`	GraphQL doc ID — `post` (post + first-page comments)
`IG_POPULAR_SEARCH_DOC_ID`	`26300727659606513`	GraphQL doc ID — popular/hashtag search
`IG_PROFILE_CONTENT_DOC_ID`	`28036671149327607`	GraphQL doc ID — condensed profile (`profile_graphql`, backs `/user/summary` and the `/user/info?detail=summary` fast path)
`IG_SESSION_TTL`	`1800`	Session TTL (seconds)
`IG_WARMUP_TIMEOUT_MS`	`15000`	Warmup fetch timeout
`IG_MAX_SESSIONS`	`70`	Max concurrent Instagram pool sessions

Facebook — `FACEBOOK_` (`src/sites/facebook/config.py`)

Public logged-out Facebook. Distinct from META_* below, which is the Ad Library only — same host, different everything else.

Variable	Default	Description
`FACEBOOK_BASE_URL`	`https://www.facebook.com/`	Base URL
`FACEBOOK_GRAPHQL_URL`	`https://www.facebook.com/api/graphql/`	GraphQL endpoint
`FACEBOOK_ASBD_ID`	`359341`	Fallback ASBD ID
`FACEBOOK_ESCALATED_PROXY`	selected premium provider	Spec used during the post-block cooldown window. The steady-state spec comes from `sites.yaml`, not from here.
`FACEBOOK_FEED_DOC_ID`	`28728774723376759`	Relay doc ID — page feed pagination
`FACEBOOK_COLLECTION_DOC_ID`	`28160238627000313`	Relay doc ID — followers/following pagination
`FACEBOOK_VIDEOS_DOC_ID`	`36237204515925186`	Relay doc ID — page video list pagination
`FACEBOOK_ALBUMS_DOC_ID`	`27733032642961647`	Relay doc ID — page albums pagination
`FACEBOOK_SINGLE_POST_DOC_ID`	`37591875023793055`	Relay doc ID — single post + initial comments
`FACEBOOK_FEED_PAGE_SIZE`	`8`	Requested feed page size (upstream returns fewer regardless)
`FACEBOOK_COLLECTION_PAGE_SIZE`	`8`	Requested collection page size

Meta Ad Library — `META_` (`src/sites/meta/config.py`)

Variable	Default	Description
`META_BASE_URL`	`https://www.facebook.com/ads/library/`	Meta Ads Library base URL
`META_GRAPHQL_URL`	`https://www.facebook.com/api/graphql/`	GraphQL endpoint
`META_ASBD_ID`	`359341`	Fallback ASBD ID
`META_SEARCH_DOC_ID`	`24922295957467452`	Search / page ads doc ID
`META_AD_DETAILS_DOC_ID`	`25068828942793558`	Ad details doc ID
`META_FILTER_OPTIONS_DOC_ID`	`29650582277919185`	Filter options doc ID
`META_SEARCH_PAGES_DOC_ID`	`9755915494515334`	Page typeahead doc ID

Ads Transparency (Google) — `ADS_TRANSPARENCY_` (`src/sites/ads_transparency/config.py`)

Variable	Default	Description
`ADS_TRANSPARENCY_BASE_URL`	`https://adstransparency.google.com`	Base URL
`ADS_TRANSPARENCY_RPC_BASE_PATH`	`/anji/_/rpc`	RPC base path

eBay — `EBAY_` (`src/sites/ebay/config.py`)

Variable	Default	Description
`EBAY_BASE_URL`	`https://www.ebay.com`	Base URL
`EBAY_AUTOSUG_URL`	`https://autosug.ebay.com`	Autocomplete host (separate, open endpoint — no Akamai gate)

LinkedIn — `LI_` (`src/sites/linkedin/config.py`)

Variable	Default	Description
`LI_BASE_URL`	`https://www.linkedin.com`	Base URL
`LI_ADAPTIVE_STORAGE_FILE`	`""`	Path for Scrapling's adaptive selector storage. Empty falls back to `<WORKER_DATA_DIR>/linkedin_selectors.db` at runtime. Must be a durable mounted path — the scrapling package directory default is wiped on every redeploy.
`LI_JOBS_SEARCH_PATH`	`/jobs-guest/jobs/api/seeMoreJobPostings/search`	Guest job search endpoint path
`LI_JOB_DETAIL_PATH`	`/jobs-guest/jobs/api/jobPosting`	Guest job detail endpoint path
`LI_GEO_TYPEAHEAD_PATH`	`/jobs-guest/api/typeaheadHits`	Guest geo typeahead endpoint path
`LI_PROFILE_PATH`	`/in`	Guest profile page path
`LI_COMPANY_PATH`	`/company`	Guest company page path
`LI_SESSION_TTL`	`1800`	Session TTL (seconds)
`LI_WARMUP_TIMEOUT_MS`	`15000`	Warmup fetch timeout
`LI_MAX_SESSIONS`	`40`	Max concurrent LinkedIn pool sessions

WORKER_DATA_DIR (no LI_ prefix, read via plain os.getenv, default .data) is the fallback parent directory for LI_ADAPTIVE_STORAGE_FILE above.

Linktree — `LINKTREE_` (`src/sites/linktree/config.py`)

Variable	Default	Description
`LINKTREE_BASE_URL`	`https://linktr.ee`	Base URL

Threads — `THREADS_` (`src/sites/threads/config.py`)

Variable	Default	Description
`THREADS_BASE_URL`	`https://www.threads.com`	Base URL
`THREADS_APP_ID`	`238260118697367`	Threads app ID header
`THREADS_ASBD_ID`	`359341`	Fallback ASBD ID
`THREADS_PROFILE_POSTS_DOC_ID`	`27591776493787818`	GraphQL doc ID — profile posts pagination

TikTok Ads Library — `TIKTOK_ADS_` (`src/sites/tiktok_ads/config.py`)

All path-only settings (no credentials). Full list, since these are the kind of thing that silently breaks a route if edited wrong:

Variable	Default
`TIKTOK_ADS_BASE_URL`	`https://library.tiktok.com`
`TIKTOK_ADS_SUPPORT_REGIONS_PATH`	`/api/v1/support-regions`
`TIKTOK_ADS_SUGGESTION_PATH`	`/api/v1/suggestion`
`TIKTOK_ADS_SEARCH_PATH`	`/api/v1/search`
`TIKTOK_ADS_REPORTS_PATH`	`/api/v1/reports`
`TIKTOK_ADS_CREATIVE_CENTER_BASE_URL`	`https://ads.tiktok.com`
`TIKTOK_ADS_TOP_ADS_FILTERS_PATH`	`/creative_radar_api/v1/top_ads/v2/filters`
`TIKTOK_ADS_TOP_ADS_COUNTRIES_PATH`	`/creative_radar_api/v1/initial/location_list`
`TIKTOK_ADS_TOP_ADS_SUGGESTIONS_PATH`	`/creative_radar_api/v1/top_ads/query_suggestion`
`TIKTOK_ADS_TOP_ADS_LIST_PATH`	`/creative_radar_api/v1/top_ads/v2/list`
`TIKTOK_ADS_TOP_ADS_DETAIL_PATH`	`/creative_radar_api/v1/top_ads/v2/detail`
`TIKTOK_ADS_TOP_ADS_RECOMMENDATIONS_PATH`	`/creative_radar_api/v1/top_ads/v2/recommend`
`TIKTOK_ADS_TOP_ADS_KEYFRAMES_PATH`	`/creative_radar_api/v1/top_ads/keyframe`
`TIKTOK_ADS_TOP_ADS_PERCENTILE_PATH`	`/creative_radar_api/v1/top_ads/percentile`
`TIKTOK_ADS_TOP_ADS_SPOTLIGHT_PATH`	`/creative_radar_api/v1/education/top_ads/list`

TikTok Shop — `TIKTOK_SHOP_` (`src/sites/tiktok_shop/config.py`)

Variable	Default	Description
`TIKTOK_SHOP_BASE_URL`	`https://shop.tiktok.com`	Base URL

Region table, user agent, client-hint header, and block signatures are hardcoded module-level constants in the same file, not environment variables.

Walmart — `WALMART_` (`src/sites/walmart/config.py`)

Variable	Default	Description
`WALMART_BASE_URL`	`https://www.walmart.com`	Base URL

Not in `.env.example`

.env.example documents REDIS_URL, the six Evomi vars, the four Decodo vars, DEFAULT_IMPERSONATE, DEFAULT_STEALTHY_HEADERS, MAX_CONCURRENCY (example value 20 — the code default is 100), SESSION_TTL, WORKER_MODE, LOG_LEVEL, PORT, BASE_URL, and THREADS_PROFILE_POSTS_DOC_ID. Every other variable above — HOST, SHUTDOWN_DRAIN_SECONDS, the browser-mint vars, the OTel vars, and every other site's config — is absent from the example file and falls back to its code default unless set.

On this page