fix(crawler): send a realistic browser User-Agent (Cloudflare evasion)
Some checks failed
deploy / build-and-push (push) Has been cancelled
deploy / deploy (push) Has been cancelled
deploy / test-frontend (push) Has been cancelled
deploy / test-backend (push) Failing after 6m10s

The crawler's headless Chromium advertised the default HeadlessChrome UA,
which Cloudflare's bot detection challenges on sight — over Tor that meant an
unsolvable 'Just a moment' page on every fetch, so session verification kept
classifying pages as transient and the metadata pass failed at
'acquire browser lease'. (Confirmed empirically: same Tor exit + a normal
Chrome UA returns the real page with the #logo sentinel; the HeadlessChrome
UA returns the challenge.) The existing user_agent setting only fed reqwest,
never the browser. Override the launcher's UA with a realistic non-headless
default (CRAWLER_USER_AGENT to customize); the UA may contain spaces, unlike
CRAWLER_BROWSER_ARGS.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
2026-06-14 21:41:00 +02:00
parent 4262efeff7
commit 91d058c426
4 changed files with 61 additions and 8 deletions

View File

@@ -1,6 +1,6 @@
{
"name": "mangalord-frontend",
"version": "0.81.1",
"version": "0.81.2",
"private": true,
"type": "module",
"scripts": {