fix(crawler): send a realistic browser User-Agent (Cloudflare evasion)
The crawler's headless Chromium advertised the default HeadlessChrome UA, which Cloudflare's bot detection challenges on sight — over Tor that meant an unsolvable 'Just a moment' page on every fetch, so session verification kept classifying pages as transient and the metadata pass failed at 'acquire browser lease'. (Confirmed empirically: same Tor exit + a normal Chrome UA returns the real page with the #logo sentinel; the HeadlessChrome UA returns the challenge.) The existing user_agent setting only fed reqwest, never the browser. Override the launcher's UA with a realistic non-headless default (CRAWLER_USER_AGENT to customize); the UA may contain spaces, unlike CRAWLER_BROWSER_ARGS. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
@@ -1,6 +1,6 @@
|
||||
{
|
||||
"name": "mangalord-frontend",
|
||||
"version": "0.81.1",
|
||||
"version": "0.81.2",
|
||||
"private": true,
|
||||
"type": "module",
|
||||
"scripts": {
|
||||
|
||||
Reference in New Issue
Block a user