Merge branch 'fix/deploy-unattended-blockers' into main
Two independent lines of production hardening diverged at7d0334band attacked overlapping problems. Neither was a superset, so this is a merge of substance rather than a fast-forward: every conflict was resolved on the merits, and the losing side's intent was re-checked against the winner rather than assumed. MIGRATIONS. The branch's 021/022/023 collided with main's already-DEPLOYED 021_hashtag_counts_respect_bans and 022_client_upload_idempotency. Renumbered to 023/024/025 in a prior commit — main's versions are applied in production, so their version numbers are immutable and the branch's had to move. Verified by running the full sqlx::test suite, which applies the whole chain from scratch. RESOLVED IN MAIN'S FAVOUR (the branch would have regressed these): * upload-queue.ts wholesale — the branch's copy has ZERO client_upload_id references, so taking it would have silently destroyed end-to-end upload idempotency, the one thing standing between a lost response and a duplicate photo charged twice against the guest's quota. * maintenance.rs supervisor — the branch replaced it with a bare tokio::spawn, where one panic silently stops session pruning, media reclaim, the temp sweep and both HashMap prunes, permanently and with no log line. * The decode-budget probe on spawn_blocking, not inline on the async runtime. * feed/+page.svelte's 8s debounce + jitter + max-wait + hidden-tab deferral, against the branch's naive 800ms — at 100 guests the branch's version walks straight into the per-user feed rate limit. * db.rs pool tuning, /uploaders, and the docker-compose deployment story. * ONE /health, still DB-backed. The branch's split (dependency-free liveness + DB-backed readiness) is defensible, but a constant-"ok" /health is the exact defectfaea555fixed and verified live, its motive (Caddy's boot gate) is already covered by app depends_on db: service_healthy, and the two handlers were the same SELECT 1 under two names. TAKEN FROM THE BRANCH: * The large-PNG OOM guard and its bounded-retry counter (023). Together these turn a single upload that can OOM-kill a 1G container into a bounded failure instead of an infinite restart loop under `restart: unless-stopped`. * 024_feed_scalar_counts — the feed no longer aggregates the whole event per page. Pure SQL; column names, order and types are unchanged by design. * The admin-lockout fix: look the admin up BY ROLE, never by name. 025 also frees any guest already squatting on a reserved name. * PIN lockout tier ordering, bounded caption/hashtag reads, SSE ticket caps, PoolTimedOut -> 503 + Retry-After, and the ffmpeg stderr drain. * backfill_video_posters, which main lacked entirely. * TempFileGuard, plus sweep_orphan_originals wired into main's SUPERVISED loop (not the branch's bare one) — it reclaims final-named originals whose commit never happened, a class main's .tmp-only sweep structurally cannot see. * shouldAbortForStall, hand-ported into main's upload-queue.ts since that file was resolved to main. Widens the watchdog at loadend instead of disarming it, bounding a half-open socket at 2 minutes rather than handing the window to xhr.timeout (5-60 min) with the whole queue's `processing` latch held. ALSO: RUST_LOG and EXPORT_PATH pinned in compose. The code fallback was `debug` (a line per request, all night) and EXPORT_PATH was the one path with a mount-shaped default that nothing validated. Verified: cargo check --all-targets, cargo clippy (clean), 144/144 backend tests against a live Postgres including upload_idempotency and upload_concurrency, 51/51 vitest, svelte-check 0 errors, eslint clean, vite build. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
@@ -46,6 +46,13 @@ async function request<T>(method: string, path: string, body?: unknown): Promise
|
||||
|
||||
// Abort hung requests so a dead connection surfaces as a friendly error
|
||||
// instead of a spinner that never resolves.
|
||||
//
|
||||
// The timer must stay armed until the BODY has been read, not just the headers.
|
||||
// `fetch` resolves as soon as the response head arrives, so clearing it in a `finally`
|
||||
// around the fetch left `res.text()` below completely uncovered — and no longer
|
||||
// abortable, since the controller had already been disarmed. An upstream that sends
|
||||
// headers and then stalls the body (the shape of a half-dead proxy, or of the pool
|
||||
// saturation this same release adds shedding for) hung that call forever.
|
||||
const controller = new AbortController();
|
||||
const timer = setTimeout(() => controller.abort(), TIMEOUT_MS);
|
||||
|
||||
@@ -57,6 +64,26 @@ async function request<T>(method: string, path: string, body?: unknown): Promise
|
||||
body: body !== undefined ? JSON.stringify(body) : undefined,
|
||||
signal: controller.signal
|
||||
});
|
||||
} catch (e) {
|
||||
clearTimeout(timer);
|
||||
if (e instanceof DOMException && e.name === 'AbortError') {
|
||||
throw new ApiError(0, 'timeout', 'Zeitüberschreitung – bitte erneut versuchen.');
|
||||
}
|
||||
throw new ApiError(0, 'network', 'Netzwerkfehler – bitte Verbindung prüfen.');
|
||||
}
|
||||
|
||||
if (res.status === 204) {
|
||||
// Must clear on this path too, or every no-content request (logout, delete, like)
|
||||
// leaks a live 20 s timer.
|
||||
clearTimeout(timer);
|
||||
return undefined as T;
|
||||
}
|
||||
|
||||
// A 5xx behind a proxy (or a crash page) can return HTML, not JSON — parsing
|
||||
// it directly would throw an opaque SyntaxError. Read text, parse defensively.
|
||||
let raw: string;
|
||||
try {
|
||||
raw = await res.text();
|
||||
} catch (e) {
|
||||
if (e instanceof DOMException && e.name === 'AbortError') {
|
||||
throw new ApiError(0, 'timeout', 'Zeitüberschreitung – bitte erneut versuchen.');
|
||||
@@ -66,13 +93,6 @@ async function request<T>(method: string, path: string, body?: unknown): Promise
|
||||
clearTimeout(timer);
|
||||
}
|
||||
|
||||
if (res.status === 204) {
|
||||
return undefined as T;
|
||||
}
|
||||
|
||||
// A 5xx behind a proxy (or a crash page) can return HTML, not JSON — parsing
|
||||
// it directly would throw an opaque SyntaxError. Read text, parse defensively.
|
||||
const raw = await res.text();
|
||||
let data: { error?: string; message?: string } | unknown = null;
|
||||
if (raw) {
|
||||
try {
|
||||
|
||||
@@ -1,5 +1,10 @@
|
||||
import { describe, it, expect } from 'vitest';
|
||||
import { classifyUploadStatus, isReversibleLock, entryToQueueItem } from './upload-queue';
|
||||
import {
|
||||
classifyUploadStatus,
|
||||
isReversibleLock,
|
||||
entryToQueueItem,
|
||||
shouldAbortForStall
|
||||
} from './upload-queue';
|
||||
|
||||
/**
|
||||
* Regression guard for the upload-queue retry policy (H2 + M1). The bug being locked out:
|
||||
@@ -107,3 +112,32 @@ describe('entryToQueueItem', () => {
|
||||
expect(item.hashtags).toBe('');
|
||||
});
|
||||
});
|
||||
|
||||
/**
|
||||
* The upload XHR had no timeout of any kind while `processQueue` held the `isProcessing`
|
||||
* latch across it. On a half-open socket neither `error` nor `abort` ever fires, so the
|
||||
* latch was pinned forever and the whole queue wedged with no recovery but a reload.
|
||||
*
|
||||
* The policy that matters: bound SILENCE, not total duration. A 500 MB video over a venue
|
||||
* uplink legitimately runs 30+ minutes while making steady progress, and a flat total cap
|
||||
* would kill exactly the uploads worth keeping.
|
||||
*/
|
||||
describe('shouldAbortForStall', () => {
|
||||
const now = 1_000_000;
|
||||
|
||||
it('lets a long upload run as long as progress keeps arriving', () => {
|
||||
// Two hours in, but progress landed a second ago.
|
||||
expect(shouldAbortForStall(now - 1_000, now, false)).toBe(false);
|
||||
});
|
||||
|
||||
it('aborts once the body stalls past the no-progress ceiling', () => {
|
||||
expect(shouldAbortForStall(now - 89_000, now, false)).toBe(false);
|
||||
expect(shouldAbortForStall(now - 91_000, now, false)).toBe(true);
|
||||
});
|
||||
|
||||
it('applies the wider ceiling once the body is sent and progress goes quiet', () => {
|
||||
// Silence that would abort mid-body is normal while waiting for the response.
|
||||
expect(shouldAbortForStall(now - 91_000, now, true)).toBe(false);
|
||||
expect(shouldAbortForStall(now - 121_000, now, true)).toBe(true);
|
||||
});
|
||||
});
|
||||
|
||||
@@ -65,6 +65,35 @@ const MAX_RETRY_DELAY_MS = 5 * 60_000;
|
||||
const STALL_TIMEOUT_MS = 90_000;
|
||||
const STALL_CHECK_INTERVAL_MS = 5_000;
|
||||
|
||||
/**
|
||||
* Ceiling for the window AFTER the last byte is sent.
|
||||
*
|
||||
* `upload.progress` is silent there BY DEFINITION — the server is sniffing the magic bytes,
|
||||
* committing the row and writing the response — so the bytes-moved signal above has nothing
|
||||
* to measure and the watchdog used to simply switch off at `loadend`. That left the wall-clock
|
||||
* `xhr.timeout` as the only remaining bound: 5 minutes for a photo, up to 60 for a video. A
|
||||
* half-open socket in that window (phone roams wifi→LTE while the server is still writing a
|
||||
* 500 MB video) pinned `processing` for that entire time, so NOTHING else in the guest's queue
|
||||
* drained either.
|
||||
*
|
||||
* Sized above the backend's worst-case commit path, not above compression — compression is
|
||||
* spawned after the response and does not hold it open.
|
||||
*/
|
||||
const RESPONSE_TIMEOUT_MS = 120_000;
|
||||
|
||||
/**
|
||||
* Pure predicate behind the watchdog, extracted so the policy is unit-testable without
|
||||
* standing up an XHR harness.
|
||||
*/
|
||||
export function shouldAbortForStall(
|
||||
lastActivityAt: number,
|
||||
now: number,
|
||||
bodySent: boolean
|
||||
): boolean {
|
||||
const ceiling = bodySent ? RESPONSE_TIMEOUT_MS : STALL_TIMEOUT_MS;
|
||||
return now - lastActivityAt > ceiling;
|
||||
}
|
||||
|
||||
/**
|
||||
* Wall-clock cap for one attempt, scaled by file size assuming a floor of ~8 kB/s — a
|
||||
* deliberately pessimistic rate, because killing a slow-but-progressing upload would lose
|
||||
@@ -865,9 +894,10 @@ async function uploadItem(id: string): Promise<void> {
|
||||
// connection that never errors and never completes. Only "no bytes moved" catches
|
||||
// that without also punishing a healthy slow link.
|
||||
let lastProgressAt = Date.now();
|
||||
let bodySent = false;
|
||||
let stalled = false;
|
||||
const stallTimer = setInterval(() => {
|
||||
if (Date.now() - lastProgressAt < STALL_TIMEOUT_MS) return;
|
||||
if (!shouldAbortForStall(lastProgressAt, Date.now(), bodySent)) return;
|
||||
stalled = true;
|
||||
xhr.abort();
|
||||
}, STALL_CHECK_INTERVAL_MS);
|
||||
@@ -889,9 +919,19 @@ async function uploadItem(id: string): Promise<void> {
|
||||
});
|
||||
// Once the last byte is out the watchdog has nothing left to measure: the server may
|
||||
// legitimately sit on the request while it validates and stores the file, and no
|
||||
// progress event fires in that window. Aborting there would re-send a whole video
|
||||
// the server had already accepted, so hand over to `xhr.timeout` instead.
|
||||
xhr.upload.addEventListener('loadend', () => clearInterval(stallTimer));
|
||||
// progress event fires in that window. Aborting on the 90s no-progress ceiling there
|
||||
// would re-send a whole video the server had already accepted.
|
||||
//
|
||||
// But switching the watchdog OFF here (which is what this used to do) handed the
|
||||
// window to `xhr.timeout` alone — 5 to 60 minutes, during which a half-open socket
|
||||
// holds `processing` and the guest's whole queue stops draining. So instead of
|
||||
// disarming, widen: `shouldAbortForStall` switches to RESPONSE_TIMEOUT_MS once
|
||||
// `bodySent` flips, which bounds the wedge at 2 minutes without ever firing on a
|
||||
// server that is legitimately still working.
|
||||
xhr.upload.addEventListener('loadend', () => {
|
||||
bodySent = true;
|
||||
lastProgressAt = Date.now();
|
||||
});
|
||||
|
||||
xhr.addEventListener('load', () => {
|
||||
const body = (() => {
|
||||
|
||||
Reference in New Issue
Block a user