fabi/EventSnap
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

fix(ops): bound logs, drain ffmpeg's stderr, and stop three silent hangs

Browse Source 
Six independent operational defects, none of which needed a new feature to fix.

Log rotation. Docker's json-file driver is unbounded by default, and those files
land on the HOST filesystem — outside every deploy.resources.limits in the compose
file, and on the same disk as postgres_data and media_data. A full disk stops
Postgres writing WAL, which takes the event down. Capped at 10m x 3 per service.

Log level. RUST_LOG was set in neither .env.example nor docker-compose.yml, so the
code fallback WAS the production level — and it was `debug`, with tower_http=debug
emitting a line per request and per response into that unrotated file. Now info,
with tower_http=warn to state that those spans are diagnostics, not an access log.

ffmpeg pipe deadlock. run_ffmpeg piped stdout and stderr and then called wait(),
which drains neither. Once the ~64 KiB pipe buffer filled, ffmpeg blocked writing
and wait() never returned — burning the full 120s timeout, twice per seek position,
three times per compression attempt. And the timeout is an Err, so the end state was
a soft-deleted upload: a guest's playable video destroyed by a poster-frame failure.
Now stdout is null (nothing ever read it) and stderr is drained by wait_with_output,
whose tail is logged on a non-zero exit. Note wait_with_output consumes the child, so
the old kill-on-timeout is gone; kill_on_drop(true) already covers it.

Readiness probe. /health never touched the pool, so the disk-full endgame above
stayed green all the way down. Adds /health/ready (SELECT 1 under 2s) as a SECOND
route — the compose healthcheck deliberately keeps pointing at /health, because
caddy gates its startup on it and a DB-dependent probe would turn a Postgres blip
into the reverse proxy refusing to start.

api.ts request timeout. The abort timer was cleared in a finally around fetch(),
which resolves on the response HEAD — leaving res.text() uncovered and no longer
abortable. An upstream that sends headers then stalls the body hung the call
forever. The timer now lives until the body is read, including the 204 path (which
otherwise leaked a live 20s timer per no-content request).

Upload XHR watchdog. The XHR had no timeout while processQueue held the
isProcessing latch across it; on a half-open socket neither error nor abort ever
fires, so the latch pinned and the queue wedged. Bounds SILENCE rather than total
duration — a 500 MB video over a venue uplink legitimately runs 30+ minutes while
making steady progress. Rejects as NetworkError, which is already the retryable
branch, so a stalled upload now recovers like any network blip.

IndexedDB failures. addToQueue called getDb() unguarded and handleSubmit had no
catch, so a private-mode refusal or a QuotaExceededError on a large blob left a
permanent "Wird hochgeladen…" spinner, no toast, and — for an in-app camera
capture — the only copy of the photo gone. Now reported as 'failed', which keeps
the staged files on screen and stays on the page.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
MechaCat02
2026-07-31 22:25:23 +02:00
parent 7d0334bf22
commit 6fd75adb27
11 changed files with 394 additions and 32 deletions
Download Patch File Download Diff File Expand all files Collapse all files

122
frontend/src/lib/upload-queue.ts
View File

@@ -37,6 +37,37 @@ const STORE_NAME = 'queue';
/** Hard cap on queued items per device — bounds IndexedDB growth from stuck blobs. */
const MAX_QUEUE_ITEMS = 100;
// ── Upload watchdog ───────────────────────────────────────────────────────────
// The upload XHR had no timeout of any kind while `processQueue` held the `processing` /
// `isProcessing` latch across it. A half-open socket — the classic phone-leaves-wifi case,
// where no `error` and no `abort` event ever fires — pinned that latch forever and wedged
// the whole queue with no recovery short of a reload.
//
// Deliberately NOT a flat total timeout: a legitimate 500 MB video over a venue uplink can
// run 30+ minutes while making perfectly steady progress, and a total cap would kill exactly
// the uploads that matter most. What we bound is SILENCE.
/** No-progress ceiling while the request body is still being sent. */
const UPLOAD_STALL_MS = 90_000;
/** Ceiling for the window AFTER the last byte is sent. `upload.progress` is silent there by
* definition (the server is sniffing, committing and answering), so the stall detector has
* no signal and this coarser bound takes over. Sized above the backend's own worst-case
* commit path, not above compression — compression is async and does not hold the response. */
const UPLOAD_RESPONSE_TIMEOUT_MS = 120_000;
/** How often the watchdog re-checks. Coarse on purpose; it only needs to bound the wedge. */
const UPLOAD_WATCHDOG_INTERVAL_MS = 5_000;
/** Pure predicate behind the watchdog, extracted so the policy is unit-testable without
* standing up an XHR harness. */
export function shouldAbortForStall(
lastActivityAt: number,
now: number,
bodySent: boolean
): boolean {
const ceiling = bodySent ? UPLOAD_RESPONSE_TIMEOUT_MS : UPLOAD_STALL_MS;
return now - lastActivityAt > ceiling;
}
let db: IDBPDatabase | null = null;
// Resume the queue as soon as connectivity returns. Registered once, guarded for SSR.
@@ -139,8 +170,10 @@ async function getDb(): Promise<IDBPDatabase> {
* blamed for) the previous guest's pending uploads.
*/
export async function clearQueue(): Promise<void> {
const database = await getDb();
await database.clear(STORE_NAME);
// Always clear the in-memory view, even if the store is unreachable: this runs on logout,
// and leaving the previous guest's items on screen for the next one is the worse failure.
const database = await getDbSafe();
if (database) await database.clear(STORE_NAME);
queueItems.set([]);
rateLimitRetryAt.set(null);
}
@@ -287,15 +320,27 @@ export async function loadQueue(): Promise<void> {
}
/** Outcome of an `addToQueue` call, so the caller can tell the user when a file was NOT
* actually queued (deduped, or the queue is full of un-evictable in-flight items). */
export type EnqueueResult = 'queued' | 'duplicate' | 'full';
* actually queued (deduped, the queue is full of un-evictable in-flight items, or the
* local store itself is unusable). */
export type EnqueueResult = 'queued' | 'duplicate' | 'full' | 'failed';
export async function addToQueue(
file: File,
caption: string,
hashtags: string
): Promise<EnqueueResult> {
const database = await getDb();
// IndexedDB is not guaranteed available: Safari private mode refuses to open a DB, an
// upgrade can be blocked by another tab, and `put` of a 500 MB blob can hit a
// QuotaExceededError. Every one of those used to reject out of here into a `handleSubmit`
// with no catch — leaving a permanent "Wird hochgeladen…" spinner, no toast, and (for an
// in-app camera capture) the only copy of the photo gone. Report it instead.
let database: IDBPDatabase;
try {
database = await getDb();
} catch (e) {
console.warn('upload queue unavailable (IndexedDB)', e);
return 'failed';
}
const userId = getUserId();
// Not authenticated — nothing to queue. Return the silent 'duplicate' outcome rather than
// 'full' so the caller doesn't show a misleading "queue full" toast. Practically
@@ -348,7 +393,14 @@ export async function addToQueue(
status: 'pending',
blob: file
};
await database.put(STORE_NAME, entry);
// Persist BEFORE touching the store: a failure here (quota exceeded on a large blob) must
// not leave a phantom item the UI shows as queued but nothing can ever upload.
try {
await database.put(STORE_NAME, entry);
} catch (e) {
console.warn('upload queue write failed (IndexedDB)', e);
return 'failed';
}
queueItems.update((items) => [
...items,
@@ -370,8 +422,21 @@ export async function addToQueue(
return 'queued';
}
/** `getDb` that reports unavailability instead of rejecting. For the UI-driven queue
* operations, where an unusable IndexedDB must degrade to "nothing happened" rather than
* leave a caller awaiting a rejected promise it never catches. */
async function getDbSafe(): Promise<IDBPDatabase | null> {
try {
return await getDb();
} catch (e) {
console.warn('upload queue unavailable (IndexedDB)', e);
return null;
}
}
export async function retryItem(id: string): Promise<void> {
const database = await getDb();
const database = await getDbSafe();
if (!database) return;
const entry = await database.get(STORE_NAME, id);
if (!entry) return;
@@ -389,13 +454,15 @@ export async function retryItem(id: string): Promise<void> {
}
export async function removeItem(id: string): Promise<void> {
const database = await getDb();
const database = await getDbSafe();
if (!database) return;
await database.delete(STORE_NAME, id);
queueItems.update((items) => items.filter((item) => item.id !== id));
}
export async function clearCompleted(): Promise<void> {
const database = await getDb();
const database = await getDbSafe();
if (!database) return;
const items = get(queueItems);
for (const item of items) {
if (item.status === 'done') {
@@ -495,7 +562,24 @@ async function uploadItem(id: string): Promise<void> {
xhr.open('POST', '/api/v1/upload');
xhr.setRequestHeader('Authorization', `Bearer ${token}`);
// Watchdog state — see UPLOAD_STALL_MS. `timedOut` distinguishes our own abort from
// a user-initiated one so the surfaced message stays honest.
let lastActivityAt = Date.now();
let bodySent = false;
let timedOut = false;
let watchdog: ReturnType<typeof setInterval> | undefined;
const stopWatchdog = () => {
if (watchdog !== undefined) {
clearInterval(watchdog);
watchdog = undefined;
}
};
// `loadend` on the XHR itself fires on success, error, abort and timeout alike — the
// one hook that guarantees the interval is released on every exit path.
xhr.addEventListener('loadend', stopWatchdog);
xhr.upload.addEventListener('progress', (e) => {
lastActivityAt = Date.now();
if (e.lengthComputable) {
const pct = Math.round((e.loaded / e.total) * 100);
queueItems.update((items) =>
@@ -557,9 +641,27 @@ async function uploadItem(id: string): Promise<void> {
}
});
// Body fully handed to the socket: `upload.progress` goes quiet from here, so switch
// the watchdog to the response ceiling rather than let it fire on normal waiting.
xhr.upload.addEventListener('loadend', () => {
bodySent = true;
lastActivityAt = Date.now();
});
xhr.addEventListener('error', () => reject(new NetworkError('Netzwerkfehler')));
xhr.addEventListener('abort', () => reject(new NetworkError('Abgebrochen')));
xhr.addEventListener('abort', () =>
reject(new NetworkError(timedOut ? 'Zeitüberschreitung' : 'Abgebrochen'))
);
xhr.send(formData);
// Armed only after send(), so the clock starts with the request. NetworkError is
// already the retryable branch (blob kept, "Erneut" offered), so a stalled upload
// now recovers exactly like a network blip instead of wedging the queue.
watchdog = setInterval(() => {
if (!shouldAbortForStall(lastActivityAt, Date.now(), bodySent)) return;
stopWatchdog();
timedOut = true;
xhr.abort();
}, UPLOAD_WATCHDOG_INTERVAL_MS);
});
// Success — remove blob from IndexedDB, mark done