Tests: run workspace doctests on a supervisor-owned tmux server - #753
Open
tony wants to merge 3 commits into
Open
Tests: run workspace doctests on a supervisor-owned tmux server#753tony wants to merge 3 commits into
tony wants to merge 3 commits into
Conversation
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## master #753 +/- ##
==========================================
- Coverage 52.37% 51.64% -0.73%
==========================================
Files 26 27 +1
Lines 3729 3925 +196
Branches 747 779 +32
==========================================
+ Hits 1953 2027 +74
- Misses 1472 1586 +114
- Partials 304 312 +8 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
why: Run audited doctests against an externally owned tmux server without falling back to ambient state or taking daemon ownership. what: - Validate the opt-in arena contract and exact source selection - Isolate each selected doctest in an adapter-owned session - Emit verified endpoint evidence and cover rejection paths
why: an artifact bound exactly one doctest source, so a page that wanted to share a lent server with its neighbours could not: the supervisor would have to start a server per page. Five pages cost about 421 ms that way against about 85 ms in one lend. The contract on the other side now takes one evidence record per declared source, and this is the side that produces them. what: - `ARENA_ARTIFACT_TARGETS` maps an artifact to a tuple of sources, and `ArenaSpec.targets_for` resolves all of them. - `--libtmux-arena-target` is repeatable, one flag per audited page, and the set passed has to equal the set the artifact declares. - Collection tracks discovered, collected and passed node ids per source, so a declared page that collected nothing is caught rather than averaged away. - One evidence record per source at the end, each naming its own page. Either every requested page gets a record or, on any failure, none do. - A test for the multi-source binding, beside the existing single-source one. The two real artifacts still declare one page each; nothing about what runs today changes.
why: an artifact could bind several pages but none did, so the capability was unexercised and the cost it exists to avoid — one server per page — was still being paid. Sharing a server also raises the question the single-page gate never had to answer: whether the server a page finishes on is the one it started on. what: - `python-workspace-and-location` audits `workspace_setup.md` and `self_location.md` together. Both take the server the fixture hands them. - `context_managers.md` is excluded by name and reason, not by omission: every example there opens `with Server()`, which under the arena resolves to the lent socket, so leaving the block stops the borrowed server. An import-time guard refuses to load if any artifact ever names an excluded page, and the request is rejected before tmux is touched. - The server's identity — pid, socket and challenge — is proved after each page, not only at the end, and a change names the page it happened after. - Teardown reaps only the sessions the run created, diffed against a baseline taken before it, and only while the identity still matches. It never stops the server. - A destructive test reproduces the failure this is for: a wrapper that stops the server between pages makes the run exit nonzero with no evidence, naming the page. Its control runs the two pages and proves identity between them.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Added since this PR opened
One artifact may now bind several documentation pages, and
--libtmux-arena-targetis repeatable, so a doctest page is affordable to audit: five sources on one lend rather than five lends. Evidence is one record per page, so a page that produced none is caught.ruff checkandmypyclean,tests/test_arena.py24 passed.Verified on the branch before pushing; gates listed in the commit messages.