test: report which corpora a run actually had (#16)

`cargo test --workspace` reports the same tally whether the disc corpus was
exercised or entirely absent. Measured: the disc suites RAN on a developer
desktop (1936 s, mesh_consistency_disc alone 1220 s) and SKIPPED on CI
(2.4 s total) -- and both reported 207 passed / 0 failed / 14 ignored across
30 suites.

Two mechanisms compound:

  * a skip is a PASSING test. The gated suites `eprintln!("SKIP: ...")` and
    return early from a test that still passes, so a skipped suite and a
    fully exercised one both score 1 passed. The totals are invariant.
  * the message is invisible. `cargo test` captures a passing test's output,
    so NEITHER log contains a `SKIP:` line. The absence of one proves
    nothing, which makes the obvious check useless too.

And `14 ignored` cannot help: `#[ignore]` is static, so that column is the
literal count of attributes in the source and cannot move at runtime. Ask
what this check would still report if the corpus were entirely absent, and
the answer is 207/0/14.

This is #16's remedy (2) -- the only one that touches the REPORT, which is
the defect. Remedies (1) and (3) improve the control and are left open.

Adds `tests/corpus_report.rs`: always runs, never fails, resolves all three
corpora exactly as the per-suite helpers do, and records what was available.
It writes to a FILE rather than relying on stdout, because a passing test's
stdout is captured and would be invisible in exactly the CI log that needs
it; the workflow then prints that file. It also appends to
GITHUB_STEP_SUMMARY when set.

Run here, it immediately shows the thing the issue is about -- all three
corpora resolve through the HARDCODED fallback, so SYLPHEED_DISC is not
controlling anything on this machine:

  SYLPHEED_DISC   PRESENT via the HARDCODED fallback, NOT $SYLPHEED_DISC
  SYLPHEED_RES3D  PRESENT via the HARDCODED fallback, NOT $SYLPHEED_RES3D
  SYLPHEED_ISO    PRESENT via the HARDCODED fallback, NOT $SYLPHEED_ISO

The ABSENT branch is the one CI takes and cannot be reached on a machine
that has the corpora, so `resolve_renders_every_branch` exercises it
directly rather than shipping it unrun -- along with "set but does not
resolve", which is what a typo in the env var produces and which is
deliberately reported as a DIFFERENT state from absent, since the two want
different fixes.

Verified: `cargo fmt --all -- --check` clean. Clippy is unchanged by this
(a test target; CI's `cargo clippy --workspace` does not build test cfg) --
it fails identically on unmodified main here with
`only_used_in_recursion` at vfs.rs:85, which is the rustc 1.90.0 vs the
runner's 1.98.1 divergence, i.e. #15, not this.

Refs #16

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
2026-09-09 07:39:49 +02:00
parent 1bae6c6bad
commit 420e84c90f
2 changed files with 168 additions and 0 deletions

View File

@@ -77,6 +77,17 @@ jobs:
- name: Run tests
run: cargo test --workspace
# The tally above cannot tell you what it verified. `cargo test` reports
# the same 207/0/14 whether the disc corpus was exercised or entirely
# absent -- a gated suite that skips still counts as passed, and the
# `ignored` column is a static count of `#[ignore]` attributes that cannot
# move at runtime. Issue #16. This prints what the run ACTUALLY had, from
# a file, because a passing test's stdout is captured and would be
# invisible in exactly this log.
- name: Report which corpora the tests actually had
if: always()
run: cat target/sylpheed-corpus-report.txt || echo "(no corpus report produced -- did corpus_report run?)"
# This step has never once executed on this codebase: the toolchain above
# shipped without the component, so every run died on "not installed"
# before clippy saw a line of source. Its result was never pass or fail,