Skip to content

Run smoke tests against a built quarto (binary mode)#14706

Draft
cderv wants to merge 37 commits into
mainfrom
test/smoke-tests-built-version
Draft

Run smoke tests against a built quarto (binary mode)#14706
cderv wants to merge 37 commits into
mainfrom
test/smoke-tests-built-version

Conversation

@cderv

@cderv cderv commented Jul 17, 2026

Copy link
Copy Markdown
Member

Note

Stacked on #14705 (base branch test/verify-vacuous-pass). Review/merge that first; this branch includes its commit and will rebase cleanly once it lands.

Smoke tests only ever run against the in-process dev TypeScript sources (version 99.9.9). Bugs that live in the shipped product — bundling, the packaged Lua filter set, the launcher, signing/notarization side effects — have no CI coverage, because the code path that exhibits them is never executed by the harness.

This adds a binary mode so the same smoke suite can run against a built quarto distribution, with no change to how the verifiers work.

Harness

Every testQuartoCmd()-based test now invokes quarto through one dispatch point, runQuarto() in tests/quarto-cmd.ts. In dev mode (default) it calls the in-process quarto() entry point exactly as before. When QUARTO_TEST_BIN points at a built quarto extracted outside the checkout, it spawns that binary as a subprocess with --log <file> --log-format json-stream, merges the child log into the test log, and the verifiers run unchanged — they only ever see log records and rendered output.

Dev callers are untouched; binary mode is opt-in via one env var. Around 30 smoke tests that spawned quarto directly or imported src/quarto.ts were migrated onto the dispatch helper so they work in both modes; a requiresDevQuarto escape hatch skips the few that exercise in-process internals.

CI

test-smokes.yml gains a quarto-install: dev | release | artifact input (dev callers unchanged). A new test-smokes-built.yml orchestrates three sources for the binary under test:

  • nightly — reuses the signed artifacts of a nightly create-release build, triggered automatically via workflow_run. Tests exactly what the release pipeline produces, on all three OSes, before anything is published.
  • build — builds a fresh linux dist from the current ref; works on forks and PR branches; manual dispatch.
  • release — installs a published (pre-)release at its tag, for post-publish verification.

workflow_run was chosen over modifying the release pipeline so smoke failures can never redden — or block — a real publish. The shared build step is extracted into a build-dist-tarball composite action, and create-release.yml gains a smoke-artifacts-only fast path for cheap branch builds.

Design decisions and their rationale are recorded in llm-docs/built-version-testing-architecture.md.

Test hardening

Migrating tests onto the dispatch seam exposed several that were silently vacuous — assertions the harness swallowed, or fixtures that never existed. Fixed here: a writer.releaseLock()-before-close() bug in the editor-support helper, and two missing fixtures (convert/issue-12318, editor-support/all.qmd). The shared verify.ts vacuous-pass fixes these rely on are in #14705, which this stacks on.

Testing

Test-infrastructure only — no user-facing change, no changelog entry. Dev-mode CI exercises the refactored dispatch path on this PR. Binary mode is verified by dispatching test-smokes-built.yml with source=build, and after merge by the automatic nightly workflow_run.

@posit-snyk-bot

posit-snyk-bot commented Jul 17, 2026 •
edited
Loading

Copy link
Copy Markdown
Collaborator

✅ Snyk checks have passed. No issues have been found so far.

Status Scan Engine Critical High Medium Low Total (0)
✅ Open Source Security 0 0 0 0 0 issues
✅ Licenses 0 0 0 0 0 issues

💻 Catch issues earlier using the plugins for VS Code, JetBrains IDEs, Visual Studio, and Eclipse.

@cderv
cderv force-pushed the test/verify-vacuous-pass branch from b9b54a6 to f3ea9c9 Compare July 17, 2026 18:44
@cderv
cderv force-pushed the test/smoke-tests-built-version branch from f1ca881 to 64a0f89 Compare July 17, 2026 18:54
Base automatically changed from test/verify-vacuous-pass to main July 17, 2026 19:31
claude and others added 26 commits July 17, 2026 21:39
@claude @cderv
Add plan for running smoke tests against a built Quarto version
311103c 
Analysis of the current test harness (in-process quarto() invocation,
json-stream log capture, smoke-all/_quarto.tests dispatch, ff-matrix
bucket routing, release CI artifacts) plus a proposal:

- Add a subprocess 'binary mode' seam (QUARTO_TEST_BIN) in tests/test.ts
  reusing the existing --log/--log-format json-stream contract
- Pair built binaries with a repo checkout at the matching v<version> tag
  to avoid harness/binary version skew
- Parameterize test-smokes.yml (dev | release | artifact install modes)
- Mode A: scheduled workflow testing the GitHub prerelease
- Mode B: smoke a built artifact inside create-release.yml
- Phased roadmap starting with the feature-format-matrix bucket

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@claude @cderv
Revise built-version test plan per maintainer decisions
dd0d0f0 
- Weekly schedule plus manual workflow_dispatch trigger
- Scope is full smoke-all (ff-matrix bucket as optional extra)
- Primary mode is preventive build-then-test: the workflow builds the
  linux-amd64 dist itself (same configure.sh + quarto-bld prepare-dist
  steps as create-release.yml make-tarball) and tests that artifact in
  the same run, so harness and binary share one commit
- Published-(pre)release testing becomes a secondary dispatch input for
  Windows coverage and version backfill
- create-release.yml gating deferred until the mode has a track record

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@claude @cderv
Add binary-mode classification results for all 133 smoke test files
fd89884 
Full sweep of tests/smoke/**/*.test.ts: 107 compatible as-is, 24
adaptable through four mechanical patterns (shared runQuarto dispatch
helper, quartoDevCmd honoring QUARTO_TEST_BIN, consolidating hardcoded
package/dist/bin spawns, two semantic one-offs), and only 2 genuinely
dev-only yaml-intelligence unitTest files that should move to
tests/unit/ instead of carrying a requiresDevQuarto flag.

Also audits smoke-all documents: docs are binary-clean except ten
fixture extensions pinned to quarto-required '>=99.9.0', which
hard-error against a real-version binary; recommend relaxing to
'>=1.9'. Records a no-reorganization verdict with two greppable lint
rules to keep future tests binary-compatible.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@claude @cderv
Rewrite built-version test plan as grounded v2 design
18a9c4d 
Incorporates four verification passes with file:line evidence:

- Git archaeology: the ten quarto-required '>=99.9.0' fixtures are
  scaffolding artifacts (create-extension template computes the value
  from the running dev version); no test exercises version gating;
  all ten are safe to relax to '>=1.9'. QUARTO_FORCE_VERSION=99.9.9
  is rejected (masks version gates, flips quarto check into dev mode).
- Log pipeline: --log/--log-format json-stream confirmed global and
  per-record flushed; render failures write ERROR before exit 1;
  pandoc/typst/run passthroughs are the documented exception.
- Env hygiene: full strip/allow/overlay contract with clearEnv;
  DENO_DIR/QUARTO_DENO added to strip list; dev-mode trap documented
  (in-repo package/dist/bin/quarto runs TS sources — dist must be
  extracted outside the checkout, guarded by a 99.9.9 version check).
- CI: quarto-dev action must run in every mode (harness runtime);
  built binary is a PATH override on top, not a replacement;
  prepare-dist writes only to package/pkg-working so build+test can
  share a checkout. Windows built layout ships both quarto.cmd
  (prepare-dist) and the Rust launcher quarto.exe (installer job);
  release testing targets quarto.exe.

Includes concrete seam spec (runQuarto helper), parameterized
test-smokes.yml step diff, test-smokes-built.yml sketch, and phased
roadmap with acceptance criteria.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@claude @cderv
Fold adversarial review findings into built-version test plan (v3)
1c30286 
Five-lens adversarial review (19 agents, every critical/major finding
independently verification-checked); 32 findings stood, none refuted.
Design changes:

- Version marker must use semver BUILD metadata (+test.YYYYMMDD): the
  vendored semver throws on the 4-component string the v2 sketch
  produced, and any '-prerelease' marker fails all >=X.Y ranges
  (verified by executing deno.land/x/semver@v1.4.0), which would have
  broken every quarto-required gate including the freshly relaxed
  fixtures.
- Silent-green invariant: a child failing before logger init (deno
  startup, bundle load, missing share) or via commandFailed paths
  exits non-zero with an empty log that vacuously passes
  noErrorsOrWarnings (~23% of the 1424-doc corpus is log-only);
  runQuarto must synthesize an ERROR record for record-free non-zero
  exits.
- Keep QUARTO_SHARE_PATH exported for the harness process in all modes
  (getenv throws when unset, killing smoke-all at module load); child
  isolation via spawn-time strip only.
- Env mechanism flipped from clearEnv+allowlist to inherit+strip
  (Windows system-var surface too large to enumerate safely);
  QUARTO_DENO_DOM corrected as the real leak vector.
- runQuarto gains throwOnFailure (direct/setup call sites keep throw
  semantics; test.execute uses no-throw), process-tree timeout kill
  (launchers spawn-and-wait, so killing the child orphans deno and
  corrupts the two-writer log), piped+drained stdout/stderr with
  stderr tail in failure reports, and logger-lifecycle branching in
  test().
- CI: runners input to restrict artifact mode to linux (Windows leg
  would download a Linux tarball), buckets becomes optional, resolve
  job for release mode, fork guard, Windows setup-step gates extended
  to non-dev full runs, release mode scoped to post-Phase-1 tags
  (older tags lack harness plumbing and local composite actions).
- Corrected citations: nightly create-release runs are build-only
  (publishing comes from dispatch), prepare-dist also regenerates
  src/ artifacts (separate build/test jobs are a requirement),
  dev-symlink location, local configure-test-env PATH-quarto nuance.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@claude @cderv
Relax dev-only quarto-required fixtures and fix drafts-env env mutation
a3aef4e 
The ten fixture extensions declaring quarto-required '>=99.9.0' got
that value from create-extension scaffolding run on a dev build (the
template computes major.minor.0 from the running version, which is the
99.9.9 dev sentinel). No test exercises version gating, and sibling
fixture extensions use realistic constraints, so relax them to '>=1.9'
to make the fixtures loadable by a real-version built quarto. No
behavior change under dev (99.9.9 satisfies >=1.9).

Also convert drafts-env.test.ts from a process-global Deno.env.set
(flagged in .claude/rules/testing/test-anti-patterns.md as a parallel
test race) to the sanctioned per-test TestContext.env channel.

Part of the built-version smoke testing plan
(dev-docs/smoke-tests-built-version-plan.md).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@claude @cderv
Add binary mode (QUARTO_TEST_BIN) to the smoke test harness
8612a6f 
Introduces tests/quarto-cmd.ts as the single dispatch point for
invoking the quarto under test. Without QUARTO_TEST_BIN nothing
changes: quarto() runs in-process exactly as before. With it, quarto
is spawned as a subprocess with --log/--log-format json-stream so the
existing log-record verifiers work unchanged against a built
distribution.

Seam design (see dev-docs/smoke-tests-built-version-plan.md):

- runQuarto() dev branch preserves the historical in-process call and
  timeout semantics byte-for-byte; the binary branch spawns with
  inherit+strip env (dev-tree identity vars like QUARTO_SHARE_PATH,
  DENO_DIR, QUARTO_DEBUG never reach the child; per-test env overlays
  last), pipes and drains stdout/stderr, and kills the whole process
  tree on timeout (the launcher spawn-and-waits on deno, so killing
  only the direct child would orphan the renderer).
- Silent-green guard: a child can exit non-zero without writing any
  ERROR record (failures before logger init, quarto add/remove
  commandFailed paths). runQuarto appends a synthetic ERROR record
  (exit code + stderr tail) whenever a non-zero exit leaves the log
  record-free, so log-only verifiers like the default
  noErrorsOrWarnings cannot pass vacuously.
- throwOnFailure defaults to true so direct call sites (module-level
  project pre-render in smoke-all, context.setup pre-renders) keep
  fail-loudly semantics; testQuartoCmd passes false and lets verifiers
  consume the log records.
- test() skips initializeLogger/cleanupLogger/flushLoggers in binary
  mode (the child owns the log file; cleanupLogger would tear down the
  default handlers for subsequent tests) and appends harness-side
  failures to the log file directly.
- TestContext.requiresDevQuarto marks in-process tests to ignore in
  binary mode (currently unused; escape hatch).
- quartoDevCmd() returns QUARTO_TEST_BIN when set, migrating the
  subprocess-based tests (run/, lua-unit, logging, create) wholesale.
- run-tests.sh/.ps1 verify the binary before running (fail on the
  99.9.9 dev sentinel, which means the launcher resolved to dev mode)
  and default to smoke/ in binary mode (unit/ and integration/ are
  dev-only).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@claude @cderv
Migrate direct quarto() call sites to the runQuarto dispatch helper
72bc843 
Replaces every 'import { quarto } from src/quarto.ts' in smoke tests
with runQuarto from tests/quarto-cmd.ts so these tests work in both
dev mode (unchanged in-process behavior) and binary mode
(QUARTO_TEST_BIN subprocess):

- Setup pre-renders keep fail-loudly semantics (throwOnFailure
  defaults to true): render-freeze, render-format-extension,
  render-output-file-collision, extension-render-journals,
  extension-render-typst-templates, self-contained/stdout,
  issues/9133 (parallel renders; no shared log file since two
  concurrent children would interleave one json-stream).
- Hand-rolled execute() bodies now accept the harness log file and
  pass throwOnFailure:false, mirroring testQuartoCmd: crossref/syntax,
  convert/issue-12318, jupyter/cache.
- jupyter/issue-10097 and issue-12374 rewritten as plain testQuartoCmd
  calls (log-only verifiers).
- engine/invalid-engine-in-project: the previous assertRejects was
  never awaited, so the test passed regardless of outcome. Rewritten
  as testQuartoCmd + printsMessage on the specific engine error, which
  actually asserts the failure and works in both modes.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@claude @cderv
Add built-version smoke testing CI: parameterize test-smokes.yml, add…
b59fadf 
… weekly workflow

test-smokes.yml gains workflow_call inputs (quarto-install:
dev|release|artifact, quarto-version, quarto-artifact-name, ref,
runners; buckets becomes optional) while keeping every default-path
behavior identical — the quarto-dev action still runs unconditionally
in all modes because run-tests.sh hardcodes the harness Deno runtime
it provisions. Non-dev modes add: release install via
quarto-actions/setup, artifact download + extraction OUTSIDE the
checkout (the installed launcher enters dev mode when a sibling
src/quarto.ts exists), and a pin-and-verify step that fails on the
99.9.9 dev sentinel, validates the version marker shape (build
metadata only — prerelease suffixes fail every semver range in the
vendored library), checks the checkout has binary-mode support, and
exports QUARTO_TEST_BIN. Windows playwright/node setup gates also
fire for full non-dev runs.

test-smokes-built.yml (weekly Monday + workflow_dispatch) implements
the preventive build-then-test mode: build the linux-amd64 dist from
the current commit with the same configure.sh + quarto-bld
prepare-dist recipe as create-release.yml make-tarball, versioned
with semver build metadata (+test.YYYYMMDD), then run the full smoke
suite against it at the same SHA on ubuntu only. A dispatch-only
source=release path resolves pre-release/release/explicit versions
from quarto.org and tests the published binary at its matching
v<version> tag (scoped to releases that contain binary-mode support;
preflighted via the contents API).

Validated with actionlint 1.7.7 (zero findings) and yaml parse; not
yet exercised on CI.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@claude @cderv
Apply binary-mode env hygiene to subprocess-spawning tests
a213fa9 
Tests that spawn quarto themselves (via execProcess or Deno.Command)
previously inherited the harness's full environment — including
QUARTO_SHARE_PATH, QUARTO_DEBUG and DENO_DIR exported by run-tests.sh,
which the installed launcher inherits when set, so in binary mode the
spawned binary would silently use dev-tree resources.

Adds quartoSpawnEnvOptions() to tests/quarto-cmd.ts: in binary mode it
returns the ambient env with the dev-tree strip list applied (per-test
overlay merged last) plus clearEnv, and in dev mode today's
inherit-and-merge behavior exactly. execProcess forwards clearEnv to
Deno.Command untouched, so no src/ change is needed. Applied at every
quarto spawn site in run/, lua-unit, logging, create, filters/
editor-support and typst-gather; the hardcoded 'quarto' and
package/dist/bin paths in stdlib-run-version, editor-support and
typst-gather now resolve through quartoDevCmd() (which honors
QUARTO_TEST_BIN).

Semantic one-offs: env/check.test.ts asserts a real semver version in
binary mode instead of the 99.9.9 dev sentinel;
inspect-standalone-rstudio passes RSTUDIO=1 via per-test env in binary
mode (the in-process _setIsRStudioForTest hook cannot cross the
process boundary and remains the dev-mode mechanism).

The two in-process yaml-intelligence tests were moved to tests/unit/
in the previous commit; with that, zero smoke tests need the
requiresDevQuarto escape hatch today.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@claude @cderv
Fix binary-mode issues found by code review
5520f69 
Correctness fixes from an 8-angle review of the binary-mode diff:

- Shared-logfile truncation: the built quarto opens --log in truncate
  mode, so a test running two runQuarto calls against one log file
  (crossref/syntax, convert/issue-12318, jupyter/cache) would have the
  second invocation erase the first's records, including the synthetic
  ERROR guard. Each child now writes its own temp log which runQuarto
  merges into the caller's log file after exit; the record-free check
  now inspects only that invocation's output.
- drafts-env: restore the module-load QUARTO_PROFILE set alongside
  context.env. src/project/project-profile.ts caches the base profile
  from the env on the first render in the process, so a per-render
  override is ignored whenever another test rendered first — the
  process-global set (before any test runs) is what makes the drafts
  profile apply in dev mode; context.env is what the spawned binary
  sees in binary mode. Documented in llm-docs/testing-patterns.md.
- Timeout kill portability: process-tree walk now uses pgrep -P
  (Linux + macOS/BSD) instead of the procps-only ps --ppid, which on
  macOS silently killed only the launcher and orphaned the renderer.
- Dev-branch timer leak: runQuarto's in-process timeout timer is now
  cleared after the race; migrated direct call sites previously had no
  timer and must not gain a dangling 10-minute one.
- Windows path resolution in CI pin-and-verify: 'command -v quarto'
  under git-bash yields an extensionless /c/... path that Deno.Command
  cannot spawn; resolve quarto.exe/quarto.cmd explicitly and convert
  with cygpath -w. Also export QUARTO_TEST_EXPECTED_VERSION when the
  caller pinned a concrete version so every spawn is verified against
  it (previously an unwired guard).
- editor-support and typst-gather were pinned to the locally built
  package/dist/bin/quarto before the migration; quartoDevCmd() would
  have silently retargeted them to PATH quarto in local dev runs. New
  quartoDevBinCmd() preserves the dev-tree pinning (QUARTO_BIN_PATH)
  while honoring QUARTO_TEST_BIN.
- Reuse: dev sentinel now imported from src/core/quarto.ts
  kLocalDevelopment instead of a private literal copy.
- Docs: testing-patterns.md env-var section updated to reflect the
  drafts-env resolution and the TestContext.env channel; testing rules
  core-file tables list tests/quarto-cmd.ts.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@claude @cderv
Allow partial runs of the built-version smoke workflow via a buckets …
1dd50eb 
…input

An empty buckets input (the default, and what schedule runs use) keeps
the full smoke run; a dispatch can pass a JSON list of bucket globs
(e.g. the feature-format-matrix qmd glob) for a cheaper validation run
before committing to the full 1400-document suite.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@claude @cderv
Extract shared build-dist-tarball composite action
3f55c88 
The configure.sh + prepare-dist + tarball recipe existed three times:
create-release.yml make-tarball, make-arm64-tarball, and the new
test-smokes-built.yml build job. Extract it into
.github/actions/build-dist-tarball (inputs: version, arch,
tarball-name, artifact-name) and use it from all three call sites.

Behavior-preserving for create-release.yml: same commands, same
--owner/--group root ownership, same tarball top-level directory
(quarto-<version>), same artifact names; the only mechanical change is
tar -czf instead of the equivalent tar cvf + gzip two-step. The
prevent-rerun and version_commit checkout steps stay in the workflow.
This also guarantees the weekly built-version smoke run exercises
exactly the release build recipe by construction.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@claude @cderv
Add nightly mode: test the signed artifacts of a no-publish create-re…
8ca870f 
…lease run

The signing steps in make-installer-win run unconditionally, so every
create-release run - including the nightly no-publish schedule -
already produces a signed Windows Zip (the real quarto.exe) and the
linux Deb Zip tarball as workflow artifacts. source=nightly reuses
them instead of building anything: resolve a create-release run (input
run-id, or latest successful), check out its exact commit for the
harness, download its artifacts cross-run (test-smokes.yml gains a
quarto-artifact-run-id input backed by download-artifact's
run-id/github-token support), and run the smokes on both OSes.

This is the preventive Windows coverage path: signed shipped binaries,
zero extra build or signing infrastructure, harness at the same commit
by construction. The artifact install step now handles both layouts
(linux tarball with a quarto-<version>/ top dir, Windows zip with
bin/share at the root, extracted via bsdtar as test-zip-win does).

Only works once the commit create-release built from contains the
binary-mode harness (post-merge); the pin-and-verify preflight fails
with a clear message otherwise.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@claude @cderv
Document nightly mode and shared build action in the plan
4839cec 
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@claude @cderv
Add isBinaryMode()/quartoTestBin() predicates for readable mode checks
6adf54b 
binaryMode() returned the binary path, which made boolean call sites
read awkwardly (binaryMode() !== undefined). Split the API:
quartoTestBin() returns the path for spawn sites; isBinaryMode() is
the boolean predicate for mode checks in test.ts and the two
mode-aware tests.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@claude @cderv
Doc audit for binary-mode testing (part 1)
42ec794 
First tranche of the documentation audit for the QUARTO_TEST_BIN
binary mode: debugging-flaky-tests gains a binary-mode isolation tip
(a flake vanishing in binary mode implicates in-process harness state),
the prerelease checklist gains an optional post-publish validation
dispatch of Smoke Tests (Built Version), llm-docs/testing-patterns.md
gains staleness frontmatter and a Dev Mode vs Binary Mode section with
the authoring rules, and the testing rules overview points at binary
mode. Remaining audit edits (tests/README.md et al.) follow.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@claude @cderv
Add architecture diagrams for the test harness and CI modes
d68cd21 
Four mermaid diagrams in tests/README.md (GitHub renders them
natively): the runQuarto dispatch seam (dev vs binary mode), the
lifecycle of one binary-mode test (per-invocation child log, merge,
synthetic-ERROR guard, tree kill), the CI workflow map (which workflow
tests what, and how the modes relate to create-release and the shared
build action), and the QUARTO_TEST_BIN handoff chain from workflow
input to per-test dispatch. All four validated with mermaid-cli.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@claude @cderv
Simplify built-version workflow: schedule uses nightly artifacts, gh …
b4d0db6 
…CLI resolution

The weekly schedule previously rebuilt a dist that create-release had
already built hours earlier from nearly the same commit - unsigned and
linux-only, where the nightly artifacts are signed and include the
Windows quarto.exe. The schedule now resolves to source=nightly;
source=build becomes dispatch-only, keeping its unique job: testing an
arbitrary ref (PR branches, forks, pre-merge validation) without
needing signing secrets or existing create-release runs.

Also replaces the curl/jq run-resolution with the preinstalled gh CLI
(one readable line per lookup), and documents the on-demand
signed-build composition: dispatch create-release with
publish-release=false, then source=nightly with that run id.

Mode resolution is one expression repeated in job conditions:
schedule => nightly; dispatch => the source input (default build).
CI diagram and plan updated to match.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@claude @cderv
Plan: record that the weekly schedule uses nightly mode
3617271 
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@claude @cderv
Built-version testing goes daily with three-OS coverage
3f7da72 
Implements the conclusions of two studies (create-release branch
dispatch; daily dev-vs-built migration):

- test-smokes-built.yml now triggers via workflow_run after every
  completed nightly create-release build instead of a weekly cron:
  once per build, right after it, no stale-artifact risk (the previous
  cron resolved 'latest successful run', silently testing yesterday's
  commit when a build failed or was slow). A failed nightly build
  means a visibly skipped day, not a false green.
- New macOS smoke leg from the nightly signed+notarized Mac Zip - the
  only macOS smoke coverage in CI (previously the Mac build only ever
  got quarto check). test-smokes.yml gains a macOS system-deps step
  (brew poppler/librsvg) and the artifact install step now detects
  layout instead of hardcoding strip-components (the linux tarball has
  a version top dir; the Mac tarball and Windows Zip are flat).
- create-release.yml gains a smoke-artifacts-only dispatch input that
  skips source/arm64 tarballs, deb/rpm installers and the macOS build,
  making 'signed build of a branch tip' cheap (verified safe: all
  publish/tag/docker/cloudsmith paths are input-gated, no protected
  environments restrict secrets on branches, version_commit resolves
  empty so jobs build the dispatched ref); publish-release is guarded
  against partial builds.
- The daily dev-mode test-smokes.yml schedule is deliberately left
  untouched: the plan records the remaining migration decisions
  (dedicated daily unit/integration job, then downgrade daily dev to
  weekly) as maintainer calls after a ~2-week green track record.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@claude @cderv
Document the macOS-runner policy on the runners input
95b3847 
macOS smoke runs are reserved for the scheduled built-version path
(test-smokes-built.yml nightly mode, once per nightly build); it must
not be added to per-commit callers, which need to stay fast. Encodes
maintainer intent so the constraint survives as policy rather than
just current wiring.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@claude @cderv
docs: purge stale weekly-cron wording after workflow_run migration
05233c4 
The trigger for test-smokes-built.yml changed from a weekly Monday cron
to workflow_run on each nightly Build Installers run, and the nightly
mode gained a macOS leg. The mermaid diagrams were updated at the time
but three prose spots still described the old design:

- tests/README.md CI section (weekly on Monday; nightly = Linux+Windows)
- test-smokes-built.yml source input description and nightly job comment
- plan doc §5.2 trigger sketch, §5.2b framing, and Phase 2 acceptance

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@cderv
Use gh api for release-tag preflight check
4be0328 
gh (2.96+ on the runner) auto-reads GH_TOKEN and hits the same contents
endpoint, so the manual Authorization/Accept headers and HTTP-status
string compare go away in favor of a plain exit-code check.
@claude @cderv
docs: add built-version testing architecture doc with design decision…
a88598c 
… records

New llm-docs/built-version-testing-architecture.md following the
llm-docs conventions (staleness frontmatter, *-architecture.md naming):
architecture summary, mode-selection guidance, and ten decision records
(D1-D10) capturing settled trade-offs - notably D1, the nightly wiring
choice (workflow_run over create-release dispatching/calling the tests,
and over the inversion where test-smokes-built would workflow_call
create-release with smoke-artifacts-only).

Also records the D1 analysis compactly in the plan doc (new §5.2c) and
cross-references the new doc from tests/README.md, testing-patterns.md,
and .claude/rules/testing/overview.md.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@claude @cderv
docs: add practical when-to-trigger guidance for built-version modes
4e03512 
The architecture doc had the mode table but not the operational guidance
(nightly = automatic workhorse; build = on-demand for packaging/harness
branches, with the smoke-artifacts-only recipe for signed branch builds;
release = curative post-publish check). Add it there, and mirror a
compact trigger/purpose table in tests/README.md next to the
test-smokes-built.yml description where dispatchers will look.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
claude and others added 9 commits July 17, 2026 21:39
@claude @cderv
ci: harden built-version workflows against review findings
e263310 
Focused review of every if:/expression across test-smokes-built.yml,
test-smokes.yml, and create-release.yml (three independent truth-table
reviews). All mode-resolution expressions, needs-chain skip propagation,
input defaults, and reusable-workflow contexts verified correct. Three
real issues found and fixed:

1. workflow_run fires on EVERY completed create-release run, not only
   nightly schedules - a smoke-artifacts-only branch build (no Mac Zip)
   would fail the mac leg red at artifact download. resolve-nightly now
   probes the run's artifacts and each OS leg gates on its artifact
   existing. Side benefit: a smoke-artifacts-only dispatch now gets its
   linux+windows smokes automatically - one dispatch instead of two.

2. publish-release defaults to TRUE, so dispatching create-release with
   only smoke-artifacts-only checked reached the version commit+tag step
   and pushed an orphan tag while the publish job was guarded off.
   configure now fails fast on the publish+smoke combination.

3. Such runs also joined the shared 'building-releases-prerelease'
   concurrency group; the group expression now sends them to per-run
   groups.

Header comments and docs (README trigger table, architecture doc D1/D7)
updated to describe the actual trigger scope and the one-dispatch flow.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@claude @cderv
tests: sanitize env for QUARTO_TEST_BIN --version probes
c881e92 
The binary-mode guards in run-tests.[sh|ps1] and assertTestBinary()
probed the built quarto with the harness env intact. The installed
launcher keeps an inherited QUARTO_SHARE_PATH (exported by the runner
for the harness), and its --version fast path is 'cat
$QUARTO_SHARE_PATH/version' - pointing at the dev tree, where
src/resources/version does not exist. A healthy built quarto therefore
reported an EMPTY version (not the 99.9.9 sentinel a review suggested -
that file doesn't exist in the dev tree): cosmetically wrong banners,
and a false-red QUARTO_TEST_EXPECTED_VERSION mismatch in release-mode
CI.

Probe with the same strip-list as test spawns (env -u chain in bash,
save/clear/restore in pwsh, buildBinaryEnv()+clearEnv in the harness)
and fail loudly on an empty version (incomplete distribution).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@cderv
docs: drop the historical plan doc, point to the ADR instead
4a360f1 
The 744-line dev-docs/smoke-tests-built-version-plan.md was the working
plan/spec produced while building this. The settled design and its
rationale now live in llm-docs/built-version-testing-architecture.md
(the decision record); the plan added nothing durable and duplicated the
ADR. Repoint the eight in-tree references at the ADR.
@claude @cderv
tests: surface swallowed failures in custom-execute tests
12f932a 
The harness converts execute() errors into log records and only fails a
test through its verifiers. Three migrated custom-execute tests relied
on that contract without a log-reading verifier:

- convert/issue-12318: assertion lived inside execute() with verify: []
  - a convert failure or failed assertion passed SILENTLY green. Moved
  the assertion into a verifier and prepended noErrors. Also clean up
  the roundtrip output file the teardown previously leaked.
- crossref/syntax: only verifier was the body comparison (ignores log
  records) - render failures surfaced as an unhelpful ENOENT. Prepended
  noErrors.
- jupyter/cache: not broken (printsMessage reads the log) but noErrors
  now reports the actual render error instead of 'expected message not
  found'.

New anti-pattern rule: assertions belong in verify, and
throwOnFailure:false requires a log-reading verifier.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@claude @cderv
tests: make editor-support and logging json-stream tests failable
64f3fe3 
Audit follow-up to the custom-execute silent-green class:

- filters/editor-support: all four tests had verify: [] with the spawn
  and assertions inside execute() - every failure mode (quarto missing,
  non-zero exit, non-JSON output, wrong crossref entries) was swallowed
  by the harness and the tests could never fail. The work now runs
  inside verifiers.
- logging/log-level-and-formats: the json-stream branch wrapped its
  level assertions in an empty catch {} - any violation (e.g. DEBUG
  records leaking into an --log-level error log) was silently ignored.
  Also the shouldContainLevel check never asserted presence (vacuous on
  zero matching records) and JSON.parsed each record's plain-text msg
  (guaranteed to fail - which the empty catch then hid). Rewritten:
  narrow parse-only try/catch, real absence assertions with the
  offending message, real presence assertions per expected level.

These 11 test registrations were previously unfailable; they may
surface genuine regressions on their first live CI run.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@claude @cderv
tests: close silent-green paths found by testQuartoCmd call-site audit
f2ce758 
Harness (tests/test.ts):
- verifiers now read logTarget (logConfig.log support was desynced: the
  logger and binary-mode child wrote to logTarget while verification
  read the temp file - any test using logConfig.log would hand every
  log verifier an empty array)
- a missing log file now FAILS the test instead of silently skipping
  every verifier
- the failure-report path guards its log re-read so a corrupt log
  cannot clobber the assembled report with a secondary parse error

Call sites:
- render-freeze: projectOutputExists constructed outputCreated's Verify
  and DISCARDED it (guaranteed no-op); combined with the purely-negative
  useFrozen check, a completely failed freeze render passed two tests.
  Now delegates properly + noErrors on all three tests.
- verify/pdf-metadata + pdf-text-position: assertions lived in teardown,
  which skips cleanup on failure - leaving a stale fixture.pdf that
  satisfies the next run even if its render fails (self-perpetuating).
  Assertions moved into verifiers; setup+teardown both remove the pdf.
- project-stdout: verify was [] - the test asserted nothing at all.
- typst-gather: cached typst/ dirs are gitignored and survive local
  runs, keeping existence verifiers green against stale caches; each
  test now cleans its cache in setup, plus noErrors (incl. the
  negative-only no-packages-staged test).
- project-prepost: teardown assertions moved into verifiers (extension
  and issue-10828 tests), noErrors added.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@cderv
tests: fix latent bugs the hardening commits surfaced
583933c 
The preceding hardening commits made three tests fail-capable that had
been silently green, exposing pre-existing defects (all present on main):

- editor-support: runEditorSupportCrossref called writer.releaseLock()
  before writer.close(), so close() threw "A writable stream is not
  associated with the writer". close() releases the lock itself - drop
  the releaseLock() call.
- convert/issue-12318: the test converted docs/convert/issue-12318.qmd,
  which never existed. Add the fixture from GH #12318 (qmd->ipynb->qmd
  roundtrip repro) so the test exercises the round trip it claims to.
- editor-support all.qmd: referenced since #6136 but never created. Add
  a combined-crossref fixture (sec/fig/tbl/eq/thm).
@cderv
tests: stop teardown ENOENT from masking render-failure reports
2d99991 
The figure-syntax comparison test in crossref/syntax.test.ts runs
runQuarto with throwOnFailure:false and relies on the noErrors verifier
to surface a failed render. Its teardown removed the output/support paths
with raw Deno.removeSync, so when a render fails (outputs never created)
the finally-phase teardown threw NotFound and replaced the in-flight
noErrors assertion — the surfaced failure became an unhelpful ENOENT
instead of the actual render errors, defeating the reason noErrors was
added. Switch to safeRemoveSync, which tolerates already-absent paths.

Also correct the editor-support stdin comment: writer.close() closes the
stream but does not release the writer's lock (releaseLock is separate);
the held lock is inert here and leaks no resource (verified under Deno's
resource/op sanitizers).
@cderv
tests: clean prepost artifacts in setup so verifyPath proves creation
f8368cf 
project-prepost.test.ts checks that i-was-created.txt (extension) and
input-files.txt/output-files.txt (issue-10828) exist after render, but
only removed them in teardown. Those are fixed repo paths, so a run that
crashes between creating them and teardown leaves a stale copy — after
which a regression that stops the render creating them still passes
verifyPath. Add setup cleanup (matching the safeRemoveIfExists the file
already uses in teardown) so each verifyPath proves this render wrote the
file, not a leftover. Also clears i-exist.txt up front so pre-render's
"must not exist" guard starts clean after a prior crashed run.
@cderv
cderv force-pushed the test/smoke-tests-built-version branch from 12d6e7e to f8368cf Compare July 17, 2026 19:39
cderv added 2 commits July 17, 2026 22:10
@cderv
tests: split runQuarto by mode and add withCwd for chdir safety
c1c6d21 
Cleanups surfaced while reviewing the binary-mode test harness.

runQuarto had grown into one 125-line function interleaving the
in-process dev path with the spawn/timeout/log-merge binary path. Split
into runDevQuarto, runBinaryQuarto, and a mergeChildLog helper so each
mode's mechanics read on their own; behavior is unchanged.

Three extension tests chdir'd into a working dir inside setup() and
restored cwd only on the success path. When "use/update template" throws
(the network step), cwd leaked into later tests, which then ran in — and
whose teardown deleted — a directory they never meant to be in. withCwd()
restores cwd in a finally, matching the existing withTempDir idiom.

Also corrected a run-tests.sh comment that claimed its --version env
strip list matched quarto-cmd.ts's kStripEnvVars; it is deliberately the
subset that affects version resolution.
@cderv
ci: TEMP push trigger + small bucket for branch shakedown
ada4305 
Temporary so the built-version smoke workflow runs on this PR branch
before it lands on the default branch (workflow_dispatch is not
available until then). Push trigger + a single-file bucket keep the
run cheap. Both are reverted before merge.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@cderv @posit-snyk-bot @claude