Some checks failed
Audit / cargo audit (backend) (push) Failing after 9m29s
Audit / npm audit (frontend) (push) Successful in 59s
Checks / Backend — cargo test + clippy + fmt (push) Failing after 59s
Checks / Frontend — vitest + svelte-check (push) Failing after 5m40s
Checks / Keepsake viewer — builds, self-contained, committed artifact in sync (push) Failing after 5m18s
Checks / E2E — typecheck + lint (push) Failing after 37s
E2E / Playwright E2E (chromium + webkit) (push) Failing after 8m55s
E2E / Cross-UA smoke matrix (push) Failing after 4m2s
v0.18.3 added a 400 branch to keep a guest's photo when the request body arrives
truncated, gated on `body.error !== 'bad_request'`. Its premise was that axum's
multipart rejection is a plain-text 400 with no envelope, so an envelope with
`bad_request` in it must be a considered verdict on the file.
That is false for the path that actually fails, and the branch was inert against
the exact incident it was written for. A guest on the live event lost a photo at
07:22; replaying the same truncation against the running backend at 07:50
produced a byte-identical log line:
WARN request rejected status=400 code="bad_request"
detail="Error parsing `multipart/form-data` request"
and this response body:
400 application/json
{"error":"bad_request","message":"Error parsing `multipart/form-data` request"}
A `bad_request` envelope. The guard evaluates false, the item still goes
terminal, and the blob is still purged.
The handler never reaches axum's extractor rejection: it pulls the fields itself
and wraps `MultipartError` in `AppError::BadRequest` -- bare from the field loop,
prefixed with "Datei konnte nicht gelesen werden: " from the chunk loop. Both
arrive indistinguishable BY CODE from "file too large". Axum's own plain-text
rejection does exist (no boundary in Content-Type -> "Invalid `boundary` ...")
but is a different error and not one a webview produces.
So the rule keys on the MESSAGE, and moves into an exported `isIncompleteBody`
beside `isReversibleLock`, matching how the other data-loss-critical rules in
this file are made testable. Tests transcribe the four responses captured from
the running backend, so reverting to an envelope check fails them.
Matching an upstream Display string is the weakness here and is called out in
the doc comment: an axum upgrade could reword it and silently re-open the data
loss. The durable fix is a distinct backend code (`body_incomplete`) this can
prefer once it exists -- deliberately not done now, because it means an app image
release and the backend has not needed one since v0.18.0.
311 lines
12 KiB
TypeScript
311 lines
12 KiB
TypeScript
import { describe, it, expect } from 'vitest';
|
|
import {
|
|
classifyUploadStatus,
|
|
isIncompleteBody,
|
|
isReversibleLock,
|
|
entryToQueueItem,
|
|
shouldAbortForStall,
|
|
suspendedSinceLastTick,
|
|
MAX_SUSPEND_CREDIT_MS
|
|
} from './upload-queue';
|
|
|
|
/**
|
|
* Regression guard for the upload-queue retry policy (H2 + M1). The bug being locked out:
|
|
* every 4xx except 429 was classified terminal, and a terminal item has its blob PURGED
|
|
* from IndexedDB. A 401 (a sliding session that lapsed, or a host PIN-reset) is a 4xx, so a
|
|
* guest's queued photos were irrecoverably destroyed the moment the `online` auto-resume
|
|
* fired against a dead session. 401 must be `auth` (blob kept, re-auth), NEVER `terminal`.
|
|
*/
|
|
describe('classifyUploadStatus', () => {
|
|
it('2xx → success', () => {
|
|
expect(classifyUploadStatus(200)).toBe('success');
|
|
expect(classifyUploadStatus(201)).toBe('success');
|
|
expect(classifyUploadStatus(299)).toBe('success');
|
|
});
|
|
|
|
it('401 → auth, NOT terminal (never purge the blob on a dead session)', () => {
|
|
expect(classifyUploadStatus(401)).toBe('auth');
|
|
});
|
|
|
|
it('429 → rate_limit (back off, auto-resume)', () => {
|
|
expect(classifyUploadStatus(429)).toBe('rate_limit');
|
|
});
|
|
|
|
it('408 → transient (request timeout is retryable, not terminal)', () => {
|
|
expect(classifyUploadStatus(408)).toBe('transient');
|
|
});
|
|
|
|
it('genuinely permanent 4xx → terminal (locked / banned / released / quota)', () => {
|
|
expect(classifyUploadStatus(403)).toBe('terminal'); // banned / locked / released
|
|
expect(classifyUploadStatus(413)).toBe('terminal'); // quota exhausted
|
|
expect(classifyUploadStatus(400)).toBe('terminal');
|
|
expect(classifyUploadStatus(404)).toBe('terminal');
|
|
});
|
|
|
|
it('5xx / unexpected → transient (retryable, blob kept)', () => {
|
|
expect(classifyUploadStatus(500)).toBe('transient');
|
|
expect(classifyUploadStatus(502)).toBe('transient');
|
|
expect(classifyUploadStatus(503)).toBe('transient');
|
|
});
|
|
});
|
|
|
|
/**
|
|
* Regression guard for the reversible-lock discrimination inside the `terminal` bucket — the
|
|
* branch that decides whether a 4xx KEEPS the blob (event closed / gallery released: a host can
|
|
* reopen and the photo resumes) or PURGES it (permanent ban / quota). Getting this wrong either
|
|
* loses a photo the guest expected to survive a reopen, or lets a banned device retry forever.
|
|
*/
|
|
/**
|
|
* Regression guard for the data loss this was written for: an iPhone guest on the live event
|
|
* got "Error parsing `multipart/form-data` request", the item went terminal, and the ONLY copy
|
|
* of the photo was purged from IndexedDB with no retry offered.
|
|
*
|
|
* The first attempt at the fix keyed on the envelope (`body.error !== 'bad_request'`) and was
|
|
* inert, because the backend wraps the multipart error in its own `bad_request` envelope. These
|
|
* cases are transcribed from real responses captured against the running backend, so they fail
|
|
* if that reasoning is ever reverted.
|
|
*/
|
|
describe('isIncompleteBody', () => {
|
|
const parseError = 'Error parsing `multipart/form-data` request';
|
|
|
|
it('the exact live failure: bad_request envelope carrying the parse error → incomplete', () => {
|
|
expect(isIncompleteBody(400, { error: 'bad_request', message: parseError, status: 400 })).toBe(
|
|
true
|
|
);
|
|
});
|
|
|
|
it('the same error raised mid-file, with the German prefix → incomplete', () => {
|
|
expect(
|
|
isIncompleteBody(400, {
|
|
error: 'bad_request',
|
|
message: `Datei konnte nicht gelesen werden: ${parseError}`,
|
|
status: 400
|
|
})
|
|
).toBe(true);
|
|
});
|
|
|
|
it('an unparseable body (plain-text rejection, proxy, WAF) → incomplete', () => {
|
|
expect(isIncompleteBody(400, null)).toBe(true);
|
|
expect(isIncompleteBody(400, undefined)).toBe(true);
|
|
});
|
|
|
|
it('a real verdict on the file → NOT incomplete, so it still purges', () => {
|
|
expect(
|
|
isIncompleteBody(400, { error: 'bad_request', message: 'Datei ist zu groß. Maximum: 500 MB.' })
|
|
).toBe(false);
|
|
expect(
|
|
isIncompleteBody(400, { error: 'bad_request', message: 'Keine Datei hochgeladen.' })
|
|
).toBe(false);
|
|
expect(
|
|
isIncompleteBody(400, { error: 'bad_request', message: 'Dateityp wird nicht unterstützt.' })
|
|
).toBe(false);
|
|
});
|
|
|
|
it('only applies to 400 — other statuses keep their own rules', () => {
|
|
expect(isIncompleteBody(413, { error: 'quota_exceeded' })).toBe(false);
|
|
expect(isIncompleteBody(403, null)).toBe(false);
|
|
expect(isIncompleteBody(500, null)).toBe(false);
|
|
});
|
|
});
|
|
|
|
describe('isReversibleLock', () => {
|
|
it('an `uploads_locked` code is reversible at any status (event closed / released)', () => {
|
|
expect(isReversibleLock(403, 'uploads_locked')).toBe(true);
|
|
expect(isReversibleLock(409, 'uploads_locked')).toBe(true);
|
|
});
|
|
|
|
it('a `forbidden` 403 (banned) is PERMANENT — purge, never resume', () => {
|
|
expect(isReversibleLock(403, 'forbidden')).toBe(false);
|
|
});
|
|
|
|
it('an unidentifiable 403 (unparseable proxy/WAF/captive-portal body) is treated reversible', () => {
|
|
// Losing a photo is the worst outcome; 403 is the reversible-lock status here.
|
|
expect(isReversibleLock(403, undefined)).toBe(true);
|
|
expect(isReversibleLock(403, null)).toBe(true);
|
|
expect(isReversibleLock(403, '')).toBe(true);
|
|
});
|
|
|
|
it('a non-403 permanent 4xx (e.g. 413 quota) is NOT reversible unless explicitly locked', () => {
|
|
expect(isReversibleLock(413, undefined)).toBe(false);
|
|
expect(isReversibleLock(400, 'bad_request')).toBe(false);
|
|
expect(isReversibleLock(413, 'uploads_locked')).toBe(true); // explicit tag still wins
|
|
});
|
|
});
|
|
|
|
/**
|
|
* Regression guard for the queue-rehydration mapping. The bug this locks out: `loadQueue`
|
|
* rebuilt items from IndexedDB WITHOUT copying `lastModified`, so a reloaded item had
|
|
* `lastModified === undefined`. addToQueue's dedup keys on (name, size, lastModified), so
|
|
* re-selecting the same file after a reload would MISS the duplicate and queue it twice.
|
|
*/
|
|
describe('entryToQueueItem', () => {
|
|
const base = {
|
|
id: 'e1',
|
|
userId: 'u1',
|
|
fileName: 'photo.jpg',
|
|
fileSize: 1234,
|
|
lastModified: 1_700_000_000_000,
|
|
mimeType: 'image/jpeg',
|
|
status: 'pending' as const
|
|
};
|
|
|
|
it('carries lastModified across rehydration (dedup depends on it)', () => {
|
|
expect(entryToQueueItem(base).lastModified).toBe(1_700_000_000_000);
|
|
});
|
|
|
|
it('downgrades an interrupted `uploading` entry to `pending` so it resumes', () => {
|
|
expect(entryToQueueItem({ ...base, status: 'uploading' }).status).toBe('pending');
|
|
});
|
|
|
|
it('a `done` entry reports 100% progress; others start at 0', () => {
|
|
expect(entryToQueueItem({ ...base, status: 'done' }).progress).toBe(100);
|
|
expect(entryToQueueItem(base).progress).toBe(0);
|
|
});
|
|
|
|
it('defaults caption/hashtags to empty strings', () => {
|
|
const item = entryToQueueItem(base);
|
|
expect(item.caption).toBe('');
|
|
expect(item.hashtags).toBe('');
|
|
});
|
|
});
|
|
|
|
/**
|
|
* The upload XHR had no timeout of any kind while `processQueue` held the `isProcessing`
|
|
* latch across it. On a half-open socket neither `error` nor `abort` ever fires, so the
|
|
* latch was pinned forever and the whole queue wedged with no recovery but a reload.
|
|
*
|
|
* The policy that matters: bound SILENCE, not total duration. A 500 MB video over a venue
|
|
* uplink legitimately runs 30+ minutes while making steady progress, and a flat total cap
|
|
* would kill exactly the uploads worth keeping.
|
|
*/
|
|
describe('shouldAbortForStall', () => {
|
|
const now = 1_000_000;
|
|
|
|
it('lets a long upload run as long as progress keeps arriving', () => {
|
|
// Two hours in, but progress landed a second ago.
|
|
expect(shouldAbortForStall(now - 1_000, now, false)).toBe(false);
|
|
});
|
|
|
|
it('aborts once the body stalls past the no-progress ceiling', () => {
|
|
expect(shouldAbortForStall(now - 89_000, now, false)).toBe(false);
|
|
expect(shouldAbortForStall(now - 91_000, now, false)).toBe(true);
|
|
});
|
|
|
|
it('applies the wider ceiling once the body is sent and progress goes quiet', () => {
|
|
// Silence that would abort mid-body is normal while waiting for the response.
|
|
expect(shouldAbortForStall(now - 91_000, now, true)).toBe(false);
|
|
expect(shouldAbortForStall(now - 121_000, now, true)).toBe(true);
|
|
});
|
|
});
|
|
|
|
/**
|
|
* The watchdog measures SILENCE via `Date.now()`, but a backgrounded phone freezes the
|
|
* interval while the clock keeps running. Without crediting the un-run window back, the first
|
|
* tick after a screen lock reads the whole sleep as a stall and aborts a healthy upload —
|
|
* re-sending from byte zero and spending one of five permanent auto-attempts. That is what
|
|
* every phone does between shots at a party.
|
|
*
|
|
* Detecting the freeze from the tick gap (rather than from `visibilitychange`) also covers the
|
|
* causes that fire no visibility event at all: a throttled-but-visible tab, a closed lid, an
|
|
* occluded window.
|
|
*/
|
|
describe('suspendedSinceLastTick', () => {
|
|
const now = 1_000_000;
|
|
|
|
it('credits nothing for a tick that arrived on schedule', () => {
|
|
expect(suspendedSinceLastTick(now - 5_000, now, 5_000)).toBe(0);
|
|
});
|
|
|
|
it('credits nothing for ordinary timer jitter or throttling', () => {
|
|
expect(suspendedSinceLastTick(now - 6_900, now, 5_000)).toBe(0);
|
|
});
|
|
|
|
/**
|
|
* The bound that matters: however the page is throttled or frozen, a genuinely dead socket
|
|
* must be detected in a time a guest would tolerate — not left to `xhr.timeout` (5-60 min).
|
|
*/
|
|
it('detects a dead socket in bounded time even under 1-tick-per-minute throttling', () => {
|
|
// Background tabs are throttled to ~1 tick/min WITHOUT the network stack pausing, so a
|
|
// socket can be dead while ticks keep arriving. Unbounded crediting made this take ~18
|
|
// minutes, holding the queue's latch the whole time.
|
|
const CAP = MAX_SUSPEND_CREDIT_MS;
|
|
let lastProgressAt = 0;
|
|
let lastTickAt = 0;
|
|
let creditSpent = 0;
|
|
let clock = 0;
|
|
let ticks = 0;
|
|
while (!shouldAbortForStall(lastProgressAt, clock, false) && ticks < 100) {
|
|
clock += 60_000; // throttled tick
|
|
const credit = Math.min(
|
|
suspendedSinceLastTick(lastTickAt, clock, 5_000),
|
|
Math.max(0, CAP - creditSpent)
|
|
);
|
|
creditSpent += credit;
|
|
lastProgressAt = Math.min(clock, lastProgressAt + credit);
|
|
lastTickAt = clock;
|
|
ticks += 1;
|
|
}
|
|
expect(clock).toBeLessThanOrEqual(240_000);
|
|
});
|
|
|
|
it('credits the whole frozen window when the interval did not run', () => {
|
|
// Screen locked ~2 minutes: a 5s interval arriving 130s late.
|
|
expect(suspendedSinceLastTick(now - 130_000, now, 5_000)).toBe(125_000);
|
|
});
|
|
|
|
it('credits nothing when the clock jumps backwards (NTP correction)', () => {
|
|
expect(suspendedSinceLastTick(now + 60_000, now, 5_000)).toBe(0);
|
|
});
|
|
|
|
/**
|
|
* Replays the production watchdog tick faithfully — including the MAX_SUSPEND_CREDIT_MS clamp.
|
|
*
|
|
* The previous version of the test below omitted that clamp, so it asserted a property the
|
|
* shipped code does not have and could not fail. Anything checking the suspension behaviour
|
|
* must go through here.
|
|
*/
|
|
function runTicks(lockMs: number, tickMs = 5_000, ticks = 3): boolean {
|
|
const CAP = MAX_SUSPEND_CREDIT_MS;
|
|
let lastProgressAt = 0;
|
|
let lastTickAt = 0;
|
|
let creditSpent = 0;
|
|
let clock = lockMs; // first tick lands when the page resumes
|
|
for (let i = 0; i < ticks; i++) {
|
|
const credit = Math.min(
|
|
suspendedSinceLastTick(lastTickAt, clock, tickMs),
|
|
Math.max(0, CAP - creditSpent)
|
|
);
|
|
creditSpent += credit;
|
|
lastProgressAt = Math.min(clock, lastProgressAt + credit);
|
|
lastTickAt = clock;
|
|
if (shouldAbortForStall(lastProgressAt, clock, false)) return true;
|
|
clock += tickMs;
|
|
}
|
|
return false;
|
|
}
|
|
|
|
it('a pocket-length screen lock does not abort a healthy upload', () => {
|
|
// 60s locked, then the interval resumes: fully credited, nothing aborted.
|
|
expect(runTicks(60_000)).toBe(false);
|
|
});
|
|
|
|
it('a suspension beyond the credit cap DOES abort — the cap is the deliberate bound', () => {
|
|
// 3 minutes locked. The cap forgives 90s, so the first tick after resume survives and the
|
|
// next one aborts. This is intended: after a lock that long the socket is almost certainly
|
|
// reaped (iOS does so without firing `error`), and re-sending beats hanging on `xhr.timeout`
|
|
// for 5-60 minutes while the queue's latch is held.
|
|
//
|
|
// It is asserted rather than merely tolerated because the cost lands on the retry budget —
|
|
// see RETRY_BUDGET_WINDOW_MS, which is what keeps this from parking the photo for good.
|
|
expect(runTicks(180_000)).toBe(true);
|
|
});
|
|
|
|
it('but a socket still silent 91s AFTER resume is aborted, never left to xhr.timeout', () => {
|
|
// iOS reaps backgrounded sockets without firing `error`. The credit buys one fresh
|
|
// window, not immunity — otherwise a dead upload would hang for 5-60 minutes holding
|
|
// the queue's `processing` latch.
|
|
const resumedAt = now - 91_000;
|
|
expect(shouldAbortForStall(resumedAt, now, false)).toBe(true);
|
|
});
|
|
});
|