MechaCat02 e4d087682d Retry transient upstream failures; fix schulcloud --help ...

A full crawl made the instance answer 4 of 26 course pages with an nginx
503 "temporarily unavailable" front-page — it pushes back when several
hundred requests arrive quickly. Nothing retried, so the index was
quietly incomplete: 22 courses and 153 files rather than 26 and 205,
with the failures recorded per course rather than surfaced as a problem.

The client now retries 429/500/502/503/504 and transient network errors
with exponential backoff plus jitter (so parallel crawl workers do not
retry in lockstep), honouring Retry-After when sent. Every call here is
an idempotent GET, so retrying is safe. 401 and 404 are deliberately not
retried: an expired token will not recover, and neither will a bad id.

Re-crawled after the fix: 26 courses, 205 files, zero failures.

Also: `schulcloud --help` printed 'Unknown command "--help"' because a
leading flag was parsed as the command name.

63 tests.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

2026-09-12 22:07:02 +02:00

2.6 KiB

Raw Blame History

View Raw