fabi 91d058c426 fix(crawler): send a realistic browser User-Agent (Cloudflare evasion) ...

The crawler's headless Chromium advertised the default HeadlessChrome UA,
which Cloudflare's bot detection challenges on sight — over Tor that meant an
unsolvable 'Just a moment' page on every fetch, so session verification kept
classifying pages as transient and the metadata pass failed at
'acquire browser lease'. (Confirmed empirically: same Tor exit + a normal
Chrome UA returns the real page with the #logo sentinel; the HeadlessChrome
UA returns the challenge.) The existing user_agent setting only fed reqwest,
never the browser. Override the launcher's UA with a realistic non-headless
default (CRAWLER_USER_AGENT to customize); the UA may contain spaces, unlike
CRAWLER_BROWSER_ARGS.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

2026-06-14 21:41:00 +02:00

862 B

Raw Blame History

View Raw