test: report which corpora a run actually had (#16 remedy 2) #19

Merged
fabi merged 2 commits from fix/corpus-report into main 2026-09-09 16:50:53 +00:00

2 Commits

Author SHA1 Message Date
f78e308ca7 test: build the fallback message with format!, matching the commit before it
All checks were successful
CI / Native — linux (pull_request) Successful in 33m26s
CI / WASM — Web (pull_request) Successful in 29m10s
CI / Formatting (pull_request) Successful in 53s
`4057bee` claimed the report prints

  SYLPHEED_DISC   PRESENT via the HARDCODED fallback, NOT $SYLPHEED_DISC

and it did not. The string was built by `"...(not ${env})".replace("${env}",
env)` -- a placeholder substituted at runtime, which printed
`(not SYLPHEED_DISC)` and dropped the `$`. I had cleaned this up before
committing, but the edit silently no-op'd (rustfmt had already rewrapped the
line, so the pattern no longer matched) and the commit went ahead with the
message quoting output the code does not produce.

Now a plain `format!`, which is what it should have been: shorter, no
placeholder to keep in sync, and the `$` makes it read as the env var it is.
The quoted output in `4057bee` is accurate as of this commit.

Not squashed into it because force-push is denied on this remote, and a
visible correction is worth more than a rewritten history anyway.

Verified: `cargo fmt --all -- --check` clean, both tests pass, output matches
the text above exactly.

Refs #16

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-09-09 07:40:20 +02:00
4057bee897 test: report which corpora a run actually had (#16)
`cargo test --workspace` reports the same tally whether the disc corpus was
exercised or entirely absent. Measured: the disc suites RAN on a developer
desktop (1936 s, mesh_consistency_disc alone 1220 s) and SKIPPED on CI
(2.4 s total) -- and both reported 207 passed / 0 failed / 14 ignored across
30 suites.

Two mechanisms compound:

  * a skip is a PASSING test. The gated suites `eprintln!("SKIP: ...")` and
    return early from a test that still passes, so a skipped suite and a
    fully exercised one both score 1 passed. The totals are invariant.
  * the message is invisible. `cargo test` captures a passing test's output,
    so NEITHER log contains a `SKIP:` line. The absence of one proves
    nothing, which makes the obvious check useless too.

And `14 ignored` cannot help: `#[ignore]` is static, so that column is the
literal count of attributes in the source and cannot move at runtime. Ask
what this check would still report if the corpus were entirely absent, and
the answer is 207/0/14.

This is #16's remedy (2) -- the only one that touches the REPORT, which is
the defect. Remedies (1) and (3) improve the control and are left open.

Adds `tests/corpus_report.rs`: always runs, never fails, resolves all three
corpora exactly as the per-suite helpers do, and records what was available.
It writes to a FILE rather than relying on stdout, because a passing test's
stdout is captured and would be invisible in exactly the CI log that needs
it; the workflow then prints that file. It also appends to
GITHUB_STEP_SUMMARY when set.

Run here, it immediately shows the thing the issue is about -- all three
corpora resolve through the HARDCODED fallback, so SYLPHEED_DISC is not
controlling anything on this machine:

  SYLPHEED_DISC   PRESENT via the HARDCODED fallback, NOT $SYLPHEED_DISC
  SYLPHEED_RES3D  PRESENT via the HARDCODED fallback, NOT $SYLPHEED_RES3D
  SYLPHEED_ISO    PRESENT via the HARDCODED fallback, NOT $SYLPHEED_ISO

The ABSENT branch is the one CI takes and cannot be reached on a machine
that has the corpora, so `resolve_renders_every_branch` exercises it
directly rather than shipping it unrun -- along with "set but does not
resolve", which is what a typo in the env var produces and which is
deliberately reported as a DIFFERENT state from absent, since the two want
different fixes.

Verified: `cargo fmt --all -- --check` clean. Clippy is unchanged by this
(a test target; CI's `cargo clippy --workspace` does not build test cfg) --
it fails identically on unmodified main here with
`only_used_in_recursion` at vfs.rs:85, which is the rustc 1.90.0 vs the
runner's 1.98.1 divergence, i.e. #15, not this.

Refs #16

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-09-09 07:39:49 +02:00