feat(adapters): systematic-debugging scenario pack - #254
Open
WODE25500 wants to merge 2 commits into
Open
Conversation
Add a systematic-debugging skill scenario pack to the Superpowers adapters.SuperpowersEvaluator, alongside verification-before-completion. Scenarios judge mechanically-detectable process discipline (all reuse the existing rule-based judge ops; no change to the evidence machinery): - investigate-before-fix: reproduce a failing test before fixing, then re-run and verify (the Iron Law). - failing-test-before-fix: establish a failing signal before the fix, then reach green (Phase 4). - single-fix-not-test-gamed: fix the source so the *unmodified* test passes, rather than gaming the test. Deliberately NOT judged: whether the agent truly understood the root cause — that is beyond a rule judge (the OSS project uses an LLM verifier for skill compliance). Documented as an opt-in real-harness smoke; the change was built /validated offline (16 unit tests) without a live Claude/Codex CLI. Refs microsoft#132.
Per independent review (no P1; P3-nits): - Rename scenario ids for honesty: reproduce-and-verify-before-done and fix-source-not-test-gamed (they check reproduce->fix->verify and fix-source-not-test-game, not semantic root-cause or a strict single-edit). - Keep the declared protected_files_unchanged check so offline unit tests can assert fail-closed on a test-game (the runner also auto-appends it; the duplicate is idempotent/harmless).
9 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds a
systematic-debuggingscenario pack to the existing Superpowers evaluation adapter (skillopt_sleep/adapters/superpowers.py), alongsideverification-before-completion. Refines issue #132 by extending the adapter to a second checkable skill.What it does
The three scenarios judge mechanically-detectable process discipline, reusing the existing rule-based judge ops (no change to the evidence machinery):
reproduce-and-verify-before-done— observe a failing run, then re-run and verify after editing (guards against fix-without-reproduce / no-verify).failing-test-before-fix— establish a failing signal before the fix, then reach green (Phase 4).fix-source-not-test-gamed— fix the source so the unmodified test passes, rather than gaming the test (fail-closed viaprotected_files_unchanged).Honest boundaries
python -m skillopt_sleep.adapters.superpowers --skill systematic-debuggingA reviewer/CI with a working
claudeCLI can run it to get real empirical evidence.Scope
skillopt_sleep/adapters/superpowers.py+tests/test_systematic_debugging_scenarios.py.verification-before-completionscenarios or the evidence machinery.Refs #132.