feat(plugin): add optimize-skill's execution track and the outcome-suite template by uipreliga · Pull Request #130 · UiPath/coder_eval

uipreliga · 2026-08-21T03:50:12Z

PR 2 of 3 extracted from #109. Based on pr/split-mvp (#129), not main, so the diff
above is exactly this PR's own slice rather than a cumulative one. It merges after #129.

⚠️ This PR has no CI until it is rebased

Every gating workflow filters on the base branch (branches: [main, develop], and
develop does not exist here), so a PR based on another PR's branch triggers no CI at
all. That is deliberate — the alternative was basing on main and showing a ~4.7k-line
cumulative diff.

What makes it safe to review now: this exact tree was verified locally against the full
gate — 376 files already formatted, ruff check clean, pyright 0 errors, 0 warnings,
and 4759 passed, 0 failed.

Before merging, this branch gets rebased onto main and force-pushed. The repo's
strict_required_status_checks_policy requires being up to date anyway, and that
force-push is the synchronize event that finally runs all the required checks. Do not
merge until they are green.

Blast radius

One product change. criteria/skill_triggered.py: an errored Skill call is no longer
engagement (3b2802b). A refused or failed call loaded no skill body, so counting it inflates
the activation metric on precisely the rows where the measurement itself failed. This matters for
ci, which sets disable-model-invocation: true and therefore makes the Skill tool refuse.

Everything else is skill prose, a bundled template, a sample task suite, a fixture and two
tutorials:


`optimize-skill/SKILL.md`	the execution track
`reference/templates/outcome.yaml` + `outcome-rows.jsonl`	the bundled suite users copy
`tasks/skills/ci-outcome.yaml` + `-rows.jsonl`, `templates/ci-outcome-fixture/**`	the worked example, run against our own `ci` skill
`docs/tutorials/09-optimizing-a-skill-body.md`	the round, reported as a null result
`orchestration/task_loader.py`	the `tune`/`holdout` → `train`/`test` split-value rename
`.gitignore`	ignores `.optimize-skill/`, the skill's multi-megabyte per-arm working tree

The split-value rename (6e9c1b4) landed before any of this shipped — tune/holdout were
never in a release, so reviewers seeing those values disappear are seeing a rename, not a
breaking change.

The correction — two lint rules were squatting on `main`'s numbers

The branch numbered its two new dataset rules CE035 and CE036. main already owns both:
TestCE035WorkflowOutputParity (tests/lint/workflow_outputs.py) and
TestCE036LiveVerdictContract (tests/lint/live_verdict_contract.py). This PR renumbers the new
pair to CE060 and CE061 — the numbers they carry at the branch tip, so no later PR in the
stack has to move an id.

Nothing in the build could have caught this, which is the more important half. From
CLAUDE.md:

Claiming a number. tests/lint/runner.py's id-uniqueness assert covers ALL_RULES ONLY, so
a class-wired id can collide with a BaseRule's — or with a number that is RESERVED (CE056) or
RETIRED (CE044) — without failing anything.

So this PR also adds test_rule_ids_are_unique_across_baserules_and_test_classes, which reads
the TestCE<NNN> class names — the one place every rule surfaces, since a BaseRule has a test
class and a class-wired rule is one. It was observed failing on ['CE035', 'CE036'] before the
renumber and passing after.

Why it matters beyond tidiness: a # noqa keys on the id string, so while two rules shared a
number one suppression would have disarmed both — including main's live-verdict contract.

The renumber reaches five files, because two of them are user-facing: tutorial 09 tells the
reader CE035 / CE036 fails the build on a partly-labelled dataset and on a verbatim leak, and
tasks/skills/ci-outcome.yaml names the rules that load tasks/**/*.yaml. Left at the old
numbers, each of those sentences would point a reader at an unrelated rule on main.

The renumber was applied by explicit occurrence, never by pattern. A global s/CE036/CE061/g
would have renamed main's TestCE036LiveVerdictContract, orphaning its contract file and
silently retiring a live rule. main's six CE034/CE035/CE036 mentions in CLAUDE.md sit on one
line; exactly three moved. Of the six mentions in .claude/harness-candidates.md, exactly one —
the "Semantic answer-leak" bullet — refers to this branch's rule, and only that one moved.

What this PR does not do

Tutorial 09 is a null result, and says so. The execution-track A/B did not produce a
significant winner. It is published as a documented method with an honest outcome, not as a
success story.
No gate. There is still no significance test and no promotion rule behind either track —
Stage B remains a human reading runs. The gate is not in this wave.
No coder_eval.optimize package.

Verification

ruff format --check (375 files), ruff check, pyright 0 errors, and the full suite green:
4753 passed, 104 skipped — the checkpoint's 4752 plus exactly one, the new guard.

The 90-skip rise versus PR1 is benign and expected: CE061 parametrises per task file and skips
every task that carries no dataset: block.

…ite template Adds the second half of `/coder-eval:optimize-skill`: the EXECUTION track, which asks whether a skill's BODY produces better outcomes, next to the activation track's question of whether its DESCRIPTION gets the skill engaged. Like PR1 this is mostly prose — the skill's execution track, the bundled `outcome.yaml` template, the `ci-outcome` sample suite and its fixture, and tutorial 09. Tutorial 09 reports a NULL result and says so. One product change: an errored `Skill` call no longer counts as engagement (3b2802b). A refused or failed call loaded no skill body, so scoring it as engagement inflates the activation metric on exactly the rows where the measurement failed. Corrections applied during the split: * The two lint rules this range introduces were numbered CE035 and CE036 on the branch, and `main` already owns both — `TestCE035WorkflowOutputParity` and `TestCE036LiveVerdictContract`. They are renumbered CE060 and CE061 here, the numbers they carry at the branch tip, so no later PR in the stack has to move an id. The rename reaches five files, two of them user-facing: tutorial 09 and the ci-outcome sample both name the rules by number. * `test_rule_ids_are_unique_across_baserules_and_test_classes` is pulled forward into this PR — the one that introduces the two ids. `runner.py`'s import-time uniqueness assert covers `ALL_RULES` only, and roughly a third of the CE rules are `@pytest.mark.lint` classes whose id lives in a class NAME, so nothing in the build could see this collision. The guard was observed failing on ['CE035', 'CE036'] before the renumber and passing after. Squashed from feat/plugin-optimize-skill: 35e7714 feat(plugin): add an execution track to optimize-skill — optimize the body, not just the description 9526d54 fix(plugin): repairs from live-testing optimize-skill with cold agents 2a0d72d docs(tutorial): conform 08 to the tutorial front-matter and title convention 6e9c1b4 refactor(dataset): rename the split values tune/holdout to train/test, and label them for the user 6ebe2d9 docs(tutorial): use blockquotes, not mkdocs admonitions, in tutorial 08 a50a356 fix(plugin): 1/8 — correct optimize-skill's execution track and cost table a90f849 feat(plugin): 2/8 — ship the outcome-suite template for the execution track 17aa999 fix(docs): 3/8 — repair the stale skill counts, and the sensor that missed them b910adc feat(tasks): 4/8 — add the ci-outcome sample, the execution track's worked example f252e6e feat(lint): 8/8 — CE035 catches the partly-labelled split dataset b375fb8 fix(plugin): 5/8 — execution-track A/B returns a NULL result, and corrects the slash-form claim it disproved c0d6959 docs(tutorial): 6/8 — add tutorial 09, the content track, reported as a null eb758e4 docs(tutorial): 7/8 — make tutorial 08's A/B reproducible, and untangle it d5b4fb0 fix: code review fixes for the optimize-skill plan c2315ed test(lint): CE036 — a row's prompt must not contain what its criteria grade 95553ec docs(harness): defer three gaps found reviewing the optimize-skill plan 3b2802b fix(criteria): an errored Skill call is not engagement 2ddb737 docs: correct tutorial 09 — the round's null had a different cause than reported 5f14f73 docs(tutorial): 09 gets the real baseline — a verified ceiling, 1.000 on 6/6 c40cdd4 fix(tasks): gate engagement, in the artifacts that call it a gate bc9125b docs(tasks): make the ci-outcome sample runnable in two lines, and record the dead end 3ce116b chore: close the last loose ends from the optimize-skill plan Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

github-actions · 2026-08-21T03:50:40Z

Claude finished @uipreliga's task in 2m 8s —— View job

Code Review for PR #130

Review Checklist

Read .github/code_review.md for review standards
Review full diff against origin/pr/split-mvp
Check product change: errored Skill calls no longer count as engagement
Verify lint rule renumbering: CE035/CE036 → CE060/CE061
Review optimize-skill implementation and execution track
Check outcome-suite template and related files
Review tutorials and documentation updates
Verify cross-file consistency
Check for missing components or edge cases

Reviewing changes now...

uipreliga

Review: coder_eval — pr:130

Scope: pr:130 · branch pr/split-execution-track · 884fe56 · 2026-08-21T05:25Z · workflow variant

Change class: complex — changes dataset split-filter semantics, adds a Skill-tool error-gating branch in engagement detection, and ships a new outcome-eval suite template plus two new CE lint rules; correctness requires reasoning about label conventions and measurement validity

Security, type safety and architecture are excellent (10/9.9/9.9) and the harness's own lint/provenance machinery is unusually disciplined, but the measurement path is where the risk sits: three confirmed defects can change a row's score or a run's final status for identical agent output — a denylist engagement gate that counts unresolved and "unknown" Skill calls as engagement, an ungated file-path signal that counts failed reads, and a mistyped --split that exits 0 with zero rows measured — so the bottom line is that this is a healthy codebase whose weakest axis (Error Handling 7.4) is precisely the half that decides whether a reported number is real, and those fixes should land before the next optimize round spends money.

Summary

Axis	Score	🟠	🟡	🔵	Top Issue
1. Code Quality & Style	8.8 / 10	0	2	2	The slash-form prompt rule in optimize-skill/SKILL.md contradicts the shipped artifacts: neither outcome.yaml, ci-outcome.yaml nor the exemplars' initial_prompt blocks use the form the docs mandate, and the 6/6 figure is attributed three ways
2. Type Safety	9.9 / 10	0	0	1	Test helper widens the result_status Literal to a bare str and silences the resulting error with a self-inflicted, unjustified `# type: ignore[arg-type]`
3. Test Health	7.9 / 10	1	2	1	Fixture eval YAMLs under templates/ci-outcome-fixture/evals/ escape every tasks/** validation scan, and activation.yaml is an invalid TaskDefinition (suite_thresholds with no dataset:)
4. Security	10 / 10	0	0	0	—
5. Architecture & Design	9.9 / 10	0	0	1	tests/test_custom_lint.py grows to 4385 lines / 59 marked rule classes in one module
6. Error Handling & Resilience	7.4 / 10	2	1	1	skill_triggered's new engagement gate is a denylist over a 4-valued result_status: only "error" is refused, so "unknown" and None still count as skill engagement (sibling command_executed allowlists "success")
7. API Surface & Maintainability	8.8 / 10	0	2	2	CE061's locator-exemption list is declared twice (inline rule + CLAUDE.md prose) with nothing binding them: the copies already disagree, one entry is dead, and skill_name/expected_skill are omitted although the same PR mandates them
8. Evaluation Harness Quality	8.5 / 10	1	1	0	skill_triggered now ignores errored Skill calls, but three definition surfaces still document the old contract (one of them generated) and no score-comparability marker was added

Overall Score: 8.9 / 10 · Weakest Axis: Error Handling & Resilience at 7.4 / 10
Totals: 🔴 0 · 🟠 4 · 🟡 8 · 🔵 8 across 8 axes.

Blockers

[Axis 3] Fixture eval YAMLs under templates/ci-outcome-fixture/evals/ escape every tasks/ validation scan, and activation.yaml is an invalid TaskDefinition (suite_thresholds with no dataset:)** (templates/ci-outcome-fixture/evals/activation.yaml:29) — tasks/skills/ci-outcome.yaml states the placement rationale explicitly: "The fixture lives under templates/, NOT beside this file, and that is deliberate: anything matching tasks/**/*.yaml is loaded as a real TaskDefinition by CE034, CE060, test_tags and test_yaml_migration". Every repo-wide task scan is rooted at tasks/ (tests/test_tags.py:154 for task_file in sorted(Path("tasks").rglob("*.yaml")):; CE060/CE061 at tests/test_custom_lint.py:4228 and :4336 use (Path(__file__).parent.parent / "tasks").rglob("*.yaml")), so the four new files under templates/ci-outcome-fixture/evals/ are validated by nothing. tests/test_custom_lint.py:1732 test_checked_in_outcome_fixture_lets_the_skill_act reads the fixture but only globs .github/workflows/*.yml and counts evals/ depths — it never calls load_task on the eval files. The gap already hid a defect: load_task(templates/ci-outcome-fixture/evals/activation.yaml) raises at PR HEAD — ValueError: Invalid task definition: ... success_criteria['skill_triggered'].suite_thresholds requires a dataset: block (thresholds are evaluated on aggregated across-row metrics) — because lines 29-30

suite_thresholds:
recall.yes: 0.7

sit on a task with no dataset: block (validator check_suite_thresholds_require_dataset, src/coder_eval/models/tasks.py:665-679). This is the one fixture file tasks/skills/ci-outcome.yaml calls load-bearing ("one task interpolating $SKILL_SOURCE_PATH"). Fix: either drop suite_thresholds from the fixture task (or give it a dataset: block), and add an assertion inside test_checked_in_outcome_fixture_lets_the_skill_act that every non-experiments/ YAML under the mounted fixture round-trips through load_task — that is the replacement for the tasks/** scans the placement deliberately escapes.
2. [Axis 6] skill_triggered's new engagement gate is a denylist over a 4-valued result_status: only "error" is refused, so "unknown" and None still count as skill engagement (sibling command_executed allowlists "success") (src/coder_eval/criteria/skill_triggered.py:74) — The new guard tests one value of a four-valued field. models/telemetry.py:372 declares result_status: Literal["success", "error", "unknown"] | None, and the guard is if cmd.result_status == "error": (line 74) with everything else falling to the else: at line 80 that adds the skill. Two non-"error" states remain fail-open:

(a) None — the live/early-stop path. claude_code_agent.py:364 constructs the tool's CommandTelemetry(... result_status=None ...) at dispatch and hands that exact object to ToolStartEvent(task_id=..., turn_id=..., tool=telemetry) (claude_code_agent.py:380). EarlyStopWatcher._on_event_impl evaluates on the CALL: elif isinstance(event, ToolStartEvent): / self._evaluate_impl(in_flight=event.tool) (early_stop.py:451,454), and _collect_verdicts appends it to the trajectory (record.commands = sorted([*record.commands, in_flight], ...), early_stop.py:538). So live_verdict (line 197) sees a Skill command with result_status=None, counts it as engagement, and returns "pass" for a positive criterion — for a Skill call that has not returned and may return the disable-model-invocation refusal this PR exists to reject. With stop_early: {on_pass: stop} the run is then cut before the refusal ever arrives, and the row is scored observed=yes. Arming skill_triggered is the documented canonical example (docs/TASK_DEFINITION_GUIDE.md:366-370).

(b) "unknown" — the finalized path. claude_code_agent._finalize_commands sets cmd.result_status = "unknown" for any tool that never got a result (claude_code_agent.py:1355, logged as a WARNING), so a Skill call cut short by a turn timeout / crash / early stop is scored as engagement too.

This also falsifies the docstring immediately above: "Because engagement is monotonic (a skill, once engaged, stays engaged), a latched verdict never flips, so it agrees with _check_impl on the frozen trajectory by construction" (lines 186-188) — engagement is no longer monotonic across the in-flight → resolved transition once an errored call is subtracted.

Fix: make it an allowlist per the project's own Review Criteria #6 ("status classification uses explicit allowlists, not denylists") — if cmd.result_status == "success": add the skill, else log and skip. For Claude a resolved Skill call is always success/error, so nothing legitimate is lost; the pending call then stays undecided until its ToolEndEvent, which the watcher already re-evaluates ("The matching ToolEndEvent still evaluates, which covers a verdict that only becomes decidable once the result is known", early_stop.py:425-427). Add a test for result_status=None and "unknown"; the PR's new tests cover only "error" (tests/test_skill_triggered.py, test_errored_skill_call_is_not_engagement).
3. [Axis 6] A --split value matching no row is a global invocation error, but it is demoted to skipped_tasks and the run exits 0 with zero rows measured (src/coder_eval/orchestration/task_loader.py:498) — The split miss is raised as a bare ValueError:

            if not rows:
                raise ValueError(
                    f"Dataset for task '{task.task_id}' has no rows in split {split!r} "

(task_loader.py:497-499). resolve_all_tasks catches it in the load/expand handler — except (FileNotFoundError, OSError, ValueError, yaml.YAMLError) as exc: … skipped.append(SkippedTask(path=str(task_file), reason=reason)) … continue (experiment.py:638-642) — and the CLI's exit gate ignores skipped entirely: if summary.tasks_failed > 0 or summary.tasks_error > 0 or failed_suite_gates > 0: (cli/run_command.py:560); skipped only prints a yellow line at run_command.py:690-693. Net effect for the single-suite invocations this PR's workflow issues: coder-eval run tasks/skills/ci-outcome.yaml --split trian skips the only task, resolves zero tasks, and exits 0.

That contradicts the policy resolve_all_tasks states for the other error class in the same function: "If EVERY task that reaches resolution fails, the cause is a global invocation error (a bad --type / -D value, repeats over the cap) rather than a per-task incompatibility, so it is re-raised and aborts the run" (experiment.py:562-567). --split is exactly that shape — one CLI value applied to every task — but takes the demote path.

The PR makes this the primary invocation path (--split train / --split test on nearly every command in plugins/coder-eval/skills/optimize-skill/SKILL.md, e.g. lines 414, 657, 686-688) and mitigates it only with prose, three times: "A mistyped split name (--split holdou) is reported as a skipped task and the run still exits 0 — a green run of zero rows" (SKILL.md:417-419; also 402-403 and 649).

Fix: raise a dedicated exception type (e.g. SplitSelectorError) that resolve_all_tasks does NOT demote, so a bad --split aborts at plan/run time; or make the CLI exit non-zero when zero tasks resolve. Related silent-degradation sibling one line up — rows = [r for r, label in labelled if label == split] (line 496) drops unlabelled rows from a partly-labelled dataset with no runtime warning; CE060 (new in tests/test_custom_lint.py:4189) covers only tasks/** in this repo, not the suites users build from the shipped template, so a runtime warning is worth adding there too.
4. [Axis 8] skill_triggered now ignores errored Skill calls, but three definition surfaces still document the old contract (one of them generated) and no score-comparability marker was added (docs/TASK_DEFINITION_GUIDE.md:1275) — src/coder_eval/criteria/skill_triggered.py:74 now adds if cmd.result_status == "error": ... else: names.add(skill.split(":")[-1]), so a Skill tool call that the tool refused no longer counts as engagement. The reference section that defines this criterion was not updated and is now false. It still reads, verbatim at docs/TASK_DEFINITION_GUIDE.md:1275:

"Agent-agnostic — scans the run's turn_records for either signal: Claude's explicit Skill tool call whose skill parameter matches skill_name (namespace prefixes like plugin:skill are stripped, ...)"

No mention of result_status. CLAUDE.md:64 ("Binary: did the agent engage the target skill (Skill tool / file read)?") and CLAUDE.md:168 are equally silent.

Second half: this is a scoring-semantics change on a persisted metric. A run.json / task.json written before this PR scores 1.0 / observed_label: "yes" on a trajectory that now scores 0.0 / "no", and nothing in the artifact distinguishes the two definitions beyond RunSummary.framework_version (src/coder_eval/models/results.py:1088). The nightly pipeline and its dashboard live in a separate repo (coder-eval-uipath / eval-runner), so any skill_triggered trend line there spans the change silently — recall/F1 can step down for identical agent behaviour and read as a skill regression.

This repo already has the convention for exactly this, in this same file: the reference: migration block carries a bold Score-comparability warning that names the external consumer ("Every task suite outside this repo (the coder-eval-uipath / eval-runner suites among them) must migrate before its next scheduled run" and "judge scores are not comparable across this migration").

Fix: (a) amend line 1275 to say the Claude signal is a successful Skill tool call and that a refused call (e.g. disable-model-invocation: true) scores no; (b) add a score-comparability note in the same shape as the reference: one, naming eval-runner and the version at which the definition changed; (c) mirror the one-line summary in CLAUDE.md:168.

Non-blocking, but please consider before merge

[Axis 1] The slash-form prompt rule in optimize-skill/SKILL.md contradicts the shipped artifacts: neither outcome.yaml, ci-outcome.yaml nor the exemplars' initial_prompt blocks use the form the docs mandate, and the 6/6 figure is attributed three ways (plugins/coder-eval/skills/optimize-skill/SKILL.md:222) — Line 222 reads **Use the slash form, not a description of it.** Open \initial_prompt` with /:`, and the YAML example at lines 225-232 shows the slash form ALONE:

initial_prompt: |
  /coder-eval:ci

Twenty lines later (242-245) the same file reports that this exact form is the weakest of the three measured: "measured at 3/6 rows, against 5/6 for a plain prose instruction and 6/6 for an explicit imperative. Pair the two". A reader (or a model) following the bolded instruction and its worked example copies the 3/6 form. The two shipped exemplars silently do a third thing — neither carries a slash at all:

plugins/coder-eval/reference/templates/outcome.yaml:121 says # Invoke the skill EXPLICITLY, and pair the slash form with an imperative. while its prompt at 149-152 is Use the \my-plugin:my-skill` skill to handle this request. Invoke it with the Skill tool...` — imperative only.
tasks/skills/ci-outcome.yaml:6-7 claims so the rows MUST use the slash form — dogfooding the mechanism the template documents, and its comment table at 131-133 lists slash + explicit imperative 6/6 engaged, yet the prompt at 153-157 is Use the \coder-eval:ci` skill...` with no slash.

Pick one form and make the instruction, the example and both exemplars agree. Either put /coder-eval:ci back on the first line of both prompts (matching the 6/6 claim), or demote line 222 to "open with an explicit imperative; the slash form alone is a hint, not a mechanism" and drop the slash-only example. Also correct ci-outcome.yaml:6-7, which asserts a property of the rows that the rows do not have. _assert_outcome_suite_shape in tests/test_custom_lint.py only checks that the bare plugin:skill substring appears, so nothing currently catches the drift.
2. [Axis 1] Two competing missing-value conventions kept side by side: _stratified_sample still buckets an explicit null into a separate "None" stratum (src/coder_eval/orchestration/task_loader.py:342) — The PR extracts row_split_label (line 349) as "the single definition of the split-filter convention" and then spends ~10 lines of prose in two places explaining why a SECOND convention stays. task_loader.py:342 is unchanged:

groups.setdefault(str(row.get(field, "")), []).append(row)

and the new note at 333-337 states the defect outright: "it also turns an explicit None into the string \"None\"". So two rows that are both unlabelled land in different strata depending on whether the key is absent ("") or explicitly null ("None"), and the stratified draw silently treats them as separate populations. Nothing forces the divergence — the stated goal ("folding a missing key to "" groups it with the genuine-empty stratum") is exactly row_split_label(row, field) or "", which also preserves the 0-is-a-label rule the docstring is careful about:

groups.setdefault(row_split_label(row, field) or "", []).append(row)

Make that one-line change, delete both prose notes (task_loader.py:333-337 and 363-366), and the codebase has one convention instead of a documented trap. If the divergence really must stay, add a test pinning the "None" stratum so the behaviour is asserted rather than only described.
3. [Axis 3] CE061 has no synthetic-offender test: every weakening of its inlined detection (threshold 12 -> 120, dropping .lower(), adding locator exemptions) leaves the suite green (tests/test_custom_lint.py:4339) — TestCE061RowPromptsDoNotLeakWhatTheyGrade (class at line 4318) ships exactly one test, test_repo_task_prompts_do_not_contain_the_graded_string (line 4339), and it only asserts assert not offenders over the live tasks/ tree. The entire detection logic is inlined in that test body and is never exercised on a known leak: the locator pops for locator in ("path", "agent_file", "file_path", "command"): dumped.pop(locator, None) (line 4361-4362), the description exclusion (line 4357), and above all the threshold at line 4366 — if len(value) >= 12 and value.lower() in prompt:. Raise 12 to 120, drop the .lower(), or add a field to the locator tuple, and the test still passes green forever. Its sibling CE060 (line 4189) is the house standard and got seven cases including test_detects_a_partly_labelled_dataset (line 4258). Fix: extract the per-row scan into a module-level _prompt_leaks(row) -> list[str] helper (mirroring CE060's _offenders) and add positive cases — a prompt containing a >=12-char file_check includes value must be flagged; a prompt containing only the criterion path and only a <12-char value must not be.
4. [Axis 3] Two outcome-suite contracts (threshold metric vocabulary, per-row cost cap) are asserted only against the bundled template, never against the checked-in suite that spends the money (tests/test_custom_lint.py:1702) — test_checked_in_outcome_sample_matches_the_shipped_template_shape (line 1702) says the worked example is "Asserted through the SAME helper as the bundled template, so the shipped shape and the worked example cannot drift" — but two of the four template sensors are outside that shared helper and run against reference/templates/outcome.yaml only. _assert_outcome_suite_shape (line 1433) checks dataset-backedness, row/split counts, ${row. substitution, prompt invocation and the engagement criterion — and nothing else. Not shared: test_outcome_template_thresholds_use_real_metric_keys (line 1633), whose own comment says "A threshold naming a metric nothing emits is not a loose gate — _attach_row_accounting records it with actual_value=None and passed=False, so the suite fails forever", and test_outcome_template_caps_cost (line 1651), "without it a single runaway row can consume a whole stage's budget". tasks/skills/ci-outcome.yaml is the artifact that actually spends the money (its own header: "~~$4.30" a pass, "~~$40 and several hours" an A/B round) and it carries suite_thresholds: {recall.yes: 1.0}, {mean: 0.7, completion_rate: 1.0} and run_limits: {max_turns: 20, max_usd: 2.00} — all valid today, all unasserted. Fix: move the metric-vocabulary loop and the run_limits cost-cap assertions into _assert_outcome_suite_shape so both call sites get them.
5. [Axis 6] The file-read engagement signal is ungated on result_status, so a FAILED or merely incidental skills/<name>/ path reference counts as engagement (skill_triggered.py:88-90) (src/coder_eval/criteria/skill_triggered.py:88) — The scan runs over every string parameter of every command, whatever the tool and whatever its result:

    for value in cmd.parameters.values():
        if isinstance(value, str):
            names.update(_SKILL_PATH_RE.findall(value))

(lines 88-90). The comment the PR adds to justify exempting it is the part that does not hold: "This is deliberately NOT gated on result_status above — a failed Skill tool call loaded nothing, whereas a path reference means the file was opened" (lines 86-87). A path reference means a path was mentioned in a tool's inputs, not that a file was opened — Bash: cat /host/plugin/skills/ci/SKILL.md that fails with "No such file", a Grep whose pattern/path contains skills/ci/, or a Write to skills/ci/SKILL.md all set result_status="error" (or never read anything) and still add ci to the engaged set. The failed-lookup case is real in this PR's own environment: "the plugin sits at a host path the sandbox cannot discover. Tested: 0 of 2 rows found the file" (plugins/coder-eval/skills/optimize-skill/SKILL.md:273-275).

That produces a false observed=yes on exactly the criterion the execution track uses as its hard engagement gate (tasks/skills/ci-outcome.yaml:162-165), i.e. the round reports "the body was measured" when it was not — the failure class this PR set out to close, surviving in the other half of the same function. The skill body already concedes the hole ("Treat the criterion as necessary, not sufficient", SKILL.md:299-301), which is why this is Medium rather than High.

Fix: gate the path scan on cmd.result_status != "error" as well (a failed read loaded nothing either), and narrow it to tools that actually read files (Read/Bash/Grep inputs) rather than every parameter of every tool.
6. [Axis 7] CE061's locator-exemption list is declared twice (inline rule + CLAUDE.md prose) with nothing binding them: the copies already disagree, one entry is dead, and skill_name/expected_skill are omitted although the same PR mandates them (tests/test_custom_lint.py:4361) — The list is written inline as for locator in ("path", "agent_file", "file_path", "command"): (tests/test_custom_lint.py:4361) and again in prose in CLAUDE.md:221 as "Location fields (path, agent_file, command) are exempt". Three concrete defects: (1) the two copies disagree — CLAUDE.md omits file_path, and no test binds them, so make lint stays green on the drift; (2) file_path is dead — git show pr-130:src/coder_eval/models/criteria.py | grep -n file_path returns nothing, no criterion in the SuccessCriterion union has that field; (3) two real locators are missing — SkillTriggeredCriterion.skill_name (models/criteria.py:1203) and CliCalledCriterion.log (models/criteria.py:579). (3) is the live one: the new execution track REQUIRES the prompt to name the skill (SKILL.md:236 initial_prompt: | / /coder-eval:ci, and outcome.yaml:158 "Use the my-plugin:my-skill skill to handle this request") while the criterion sets skill_name: to that same name. For any skill whose bare name is >= 12 chars — e.g. optimize-skill (14) — skill_name and the substituted expected_skill both appear verbatim in the prompt and CE061 flags a correctly-authored suite as an answer leak. It is masked in-repo only because ci is 2 chars and lint-tasks is 10, both under the len(value) >= 12 floor on line 4366. Fix: hoist the tuple to ONE named constant, add skill_name and log, drop file_path (or keep it with a comment saying which criterion is expected to grow it), and add a test asserting the constant and the CLAUDE.md sentence list the same fields.
7. [Axis 7] The check-skill activation template (plugins/coder-eval/reference/templates/activation.yaml:41-44) omits type: "claude-code" + setting_sources: [], unlike every sibling activation/outcome surface in this PR — and it is the one such surface copied verbatim into user repos (plugins/coder-eval/reference/templates/activation.yaml:41) — The template's whole agent block is agent: / plugins: / - type: "local" / path: "$SKILL_SOURCE_PATH" (lines 41-44) — no type: "claude-code" and no setting_sources: []. Every sibling activation surface this PR touches or adds has both, with the reason spelled out: tasks/skills/lint-tasks-activation.yaml:24-32 ("this measures the skill listing, and injecting a 20 KB project guide into every call is both expensive and a confound"), the new templates/ci-outcome-fixture/evals/activation.yaml:11-15, the new plugins/coder-eval/reference/templates/outcome.yaml:41, and the rewritten docs/tutorials/08-optimizing-a-skill.md:127. CLAUDE.md's own sandbox-isolation guidance says the same. check-skill copies this file verbatim (plugins/coder-eval/skills/check-skill/SKILL.md:144-158 lists only task_id / skill_name / expected_skill / prompt / split as substitutions) and never mentions setting_sources, so every user-generated activation suite pays host-CLAUDE.md injection on every probe row and carries the confound. Add type: "claude-code" + setting_sources: [] to the template's agent block with the same one-line rationale the outcome template uses.
8. [Axis 8] The outcome template ships 2 train rows and the execution track states no minimum-n, while its gate is a paired t-test that can promote on n=2 with p=0.000 (plugins/coder-eval/reference/templates/outcome-rows.jsonl:1) — outcome-rows.jsonl ships four placeholder rows — core-1 and regression-1 on split: "train", core-2 and regression-2 on split: "test". Stage B runs --split train (SKILL.md:738-739), so the paired comparison that decides promotion pairs two rows.

src/coder_eval/reports_stats.py:381-389 only short-circuits below 2: if len(common_tasks) < 2: return PairedComparison(..., None, None, None, None, None). At exactly 2 it computes a real CI with df=1 (t* ≈ 12.7), so Stage B's promotion rule — "The paired mean difference favours the candidate and its 95% CI excludes zero" (SKILL.md:764) — is evaluated on an interval that is almost never informative, with no warning anywhere in the output.

The activation twin has the guidance the execution track lacks. plugins/coder-eval/reference/templates/activation.yaml (unchanged block): "A split HALVES each side, so it multiplies the sizing rule: a suite you intend to optimize wants 16-24 of each polarity, not the 8-12 a one-shot check aims for." And SKILL.md:141-143 — "Below check-skill's un-doubled minimum, hand back rather than gate — a metric over three or four rows moves in 25–33 point jumps" — sits inside ### Activation track — the activation suite (lines 125-146) and so does not govern ### Execution track — an outcome suite (147-368), whose "Five requirements specific to this track" list contains no row-count floor.

Fix: state a minimum train-row count for an outcome suite in the execution-track section and in outcome.yaml's header (the derived tasks/skills/ci-outcome.yaml uses 6 train / 4 test, which is a usable anchor), and either ship more placeholder rows or make the template's header say plainly that four rows is a shape, not a sample size.

Nits

[Axis 1] The artifact-scoring criterion-type set is a hardcoded literal duplicated in two tests in the same file (tests/test_custom_lint.py:1621) — The identical literal appears at line 1621 (test_outcome_template_scores_artifacts_not_prose) and line 1726 (test_checked_in_outcome_sample_matches_the_shipped_template_shape):

artifact_scoring = {"file_check", "json_check", "run_command", "cli_called", "command_executed"}

The same file goes to real trouble elsewhere to derive its vocabularies from live code (_outcome_metric_vocabulary calls BaseCriterion.aggregate and reports._attach_row_accounting rather than writing metric names down), so the hand-copied set is inconsistent with the file's own standard and will drift when a new artifact-scoring criterion type lands. Lift it to one module-level constant beside _outcome_metric_vocabulary and reference it from both tests.
2. [Axis 1] Tutorial titles 08 and 09 use Title Case while 01-07 use sentence case (mkdocs.yml:108) — mkdocs.yml:107-109 now reads 07 · Driving Coder Eval from Claude Code, 08 · Optimizing a Skill Description, 09 · Optimizing a Skill Body — the PR changed 08 from the previous sentence-case Optimizing a skill description and added 09 in Title Case, so the nav mixes two conventions (Comparing two models, Docker isolation vs Optimizing a Skill Body). The same casing propagates through the generated surfaces: docs/tutorials/README.md:23-24 and docs/llms.txt:53-54. Restore sentence case (08 · Optimizing a skill description, 09 · Optimizing a skill body) in mkdocs.yml and re-run make docs-indexes so the three derived tables follow.
3. [Axis 2] Test helper widens the result_status Literal to a bare str and silences the resulting error with a self-inflicted, unjustified # type: ignore[arg-type] (tests/test_skill_triggered.py:24) — The helper declares the parameter as a free str and then suppresses the mismatch at the construction site:

    result_status: str = "success",          # line 24
...
        result_status=result_status,  # type: ignore[arg-type]   # line 32

The field it feeds is Literal["success", "error", "unknown"] | None (src/coder_eval/models/telemetry.py:372). Two problems: the ignore carries no justifying comment (the axis-2 Medium anchor's exact shape), and it is inert — pyproject.toml's [tool.pyright] sets exclude = [..., "tests", ...], and the second make typecheck pass only includes tests/lint/live_verdict_contract.py and tests/_fixtures/live_criteria.py (tests/lint/pyright_config.py::INCLUDE), so nothing type-checks this file at all. Impact is bounded to Low because pydantic still rejects a bad literal at runtime, so a typo fails the test loudly rather than silently.

Fix: annotate the parameter with the real type and delete the ignore —

    result_status: Literal["success", "error", "unknown"] | None = "success",

This also documents the closed set at the criterion/agent seam, which is the contract finding #1 is about.
4. [Axis 3] CE061 scans rows through expand_dataset, so an unseeded stratified sample would silently reduce it to a random subset (tests/test_custom_lint.py:4348) — Line 4348 is for row in expand_dataset(task, path.parent):. expand_dataset applies dataset.sample_per_stratum seeded by dataset.sample_seed, and src/coder_eval/orchestration/task_loader.py:518-522 states "When that is None the sample is deliberately nondeterministic — re-drawn every run". No tasks/** suite sets sample_per_stratum today (git grep -n 'sample_per_stratum' -- tasks/ is empty), so the rule is deterministic right now; the moment a suite adds one without a seed, CE061 checks a different random subset of rows each run and a leaking row passes CI intermittently. Its sibling CE060 avoids this by reading the unsampled rows — line 4211-4214 uses _load_dataset_rows(task.dataset, task_file_dir). Fix: have CE061 read _load_dataset_rows too and substitute ${row.*} itself, or assert task.dataset.sample_per_stratum is None or task.dataset.sample_seed is not None before scanning.
5. [Axis 5] tests/test_custom_lint.py grows to 4385 lines / 59 marked rule classes in one module (tests/test_custom_lint.py:4188) — git show pr-130:tests/test_custom_lint.py | wc -l is 4385, up from 3425 at the base (origin/pr/split-mvp) — a +1060 line, ~28% growth in one commit, and grep -c "^class \|^@pytest.mark.lint" counts 59 entries. The two new rules land as @pytest.mark.lint class TestCE060SplitLabelsAllOrNothing (line 4188) and class TestCE061RowPromptsDoNotLeakWhatTheyGrade (line 4318), plus the module-level helper _string_leaves at the very bottom. This follows the documented convention (CLAUDE.md line 221: Markdown/YAML-reasoning rules are @pytest.mark.lint classes rather than BaseRules), so it is not a pattern violation — but the convention says nothing about WHICH file those classes live in, and the module is now a single grab-bag spanning doc parity, plugin artifacts, workflow outputs, dataset labelling and prompt leakage. Split it by subject into a tests/lint_tests/ package (one module per family, e.g. test_dataset_rules.py for CE060/CE061) and keep make lint's marker selection as-is so coverage is unchanged by the move; also give _string_leaves a home beside the rule that uses it rather than at module scope where any of the other 58 classes can pick it up.
6. [Axis 6] The refused Skill call is recorded only as a DEBUG line and never reaches the criterion's details, so the report shows observed='no' with no attribution (src/coder_eval/criteria/skill_triggered.py:75) — The drop is traced with logger.debug("Skill call for %r errored (%s); not counting it as engagement", ...) (lines 75-79) and nothing else: the returned result carries only details=f"observed={observed!r}, expected={expected!r}{filt}", (line 165), so task.json and every report render a plain observed='no' — indistinguishable from "the model never called the skill", which needs the opposite remedy (fix the description vs. remove disable-model-invocation: from the frontmatter). The DEBUG line does land in task.log (task_log_handler lowers the app logger to DEBUG, logging_config.py:329-330, and criteria run inside that context, orchestrator.py:518/781), so the evidence exists — but only per-task, and the operator reading an aggregate recall of 0.0 across 24 rows has no pointer to it.

Compare the repo precedent for the analogous abnormal-flow case: "Permission-blocked tool use is abnormal flow — warn so it surfaces in runs that don't have DEBUG enabled" → self._log.warning(...) (src/coder_eval/agents/claude_code_agent.py:1799-1806); that phrase list ("permission", "not allowed", "requires approval", "denied", "blocked") does not match "cannot be used with Skill tool due to disable-model-invocation", so no warning fires for this case today.

Fix: append the refusal to the criterion's details (e.g. refused Skill call(s): <skill> — <first 120 chars>) so the attribution travels with the score, and raise the log to warning for a measurement-invalidating refusal.
7. [Axis 7] optimize-skill's only worked max_usd figure is the value both shipped templates document as a hazard (plugins/coder-eval/skills/optimize-skill/SKILL.md:348) — SKILL.md:347-349 reads: "Set run_limits.max_usd on the suite. It is a per-row cap ... so a 12-row suite at 0.50 bounds one arm at about $6 per replicate". 0.50 is the only concrete value in the skill body, and both artifacts this same PR ships set 2.00 and call 0.50 a trap: plugins/coder-eval/reference/templates/outcome.yaml:80-83 — "A real outcome row here cost $0.43, and a draft cap of 0.50 sat 15% above that: a slightly longer row would have aborted as COST_BUDGET_EXCEEDED and scored as a body failure that never happened — a fabricated result of exactly the kind this file warns about" — and tasks/skills/ci-outcome.yaml:83-89 repeats it. Change the illustration to 2.00 (or restate the arithmetic on a value the templates endorse) and carry the "set it from a measured row, generously" sentence into the skill body so the agent following it does not reproduce the failure the templates were edited to prevent.
8. [Axis 7] The new capability preflight claims to verify --split exists, but coder-eval plan has no such flag (plugins/coder-eval/skills/optimize-skill/SKILL.md:45) — SKILL.md:44-48 says "run coder-eval plan <suite> and require it to exit 0 before spending. Two binaries can report the same version and differ in whether --split and dataset.split_field exist at all". plan only takes --experiment — git show pr-130:src/coder_eval/cli/plan_command.py | grep typer.Option yields exactly one option, at line 20-22 — and it does not expand datasets or accept any row selector. It therefore proves only the dataset.split_field half (via Dataset's model_config = ConfigDict(extra="forbid") in models/tasks.py); it says nothing about whether coder-eval run accepts --split. Either narrow the claim to dataset.split_field, or add a second explicit probe the sentence can stand behind (e.g. coder-eval run --help | grep -q -- --split).

What's Missing

Parallel paths:

🟠 🟠 skill_triggered.py — the sibling that already solved "did the tool actually succeed?" was not reused: criteria/command_executed.py:173 gates on cmd.result_status != "success" behind an author-facing require_success flag, while the new Skill gate hardcodes a == "error" denylist with no opt-out. The same question now has two answers in criteria/, and only one of them is an allowlist. (trigger: src/coder_eval/criteria/skill_triggered.py) (restates: Axis 6: skill_triggered's new engagement gate is a denylist over a 4-valued result_status)
🟡 🟡 skill_triggered.py — the gate landed on one of the two engagement signals inside the same function. The _SKILL_PATH_RE scan over every parameter of every command (lines 88-90) is left ungated, so the half of the criterion that Codex and Antigravity actually exercise never sees the new rule. (trigger: src/coder_eval/criteria/skill_triggered.py) (restates: Axis 6: The file-read engagement signal is ungated on result_status)
🟡 🟡 skill_triggered.py — the cross-harness parity surfaces were not touched. docs/agents/CODEX.md:228 documents this criterion's per-harness divergence verbatim ("Codex has no distinct Skill tool … the file-read signal is weaker") and docs/agents/HARNESS_PARITY.md does not mention the criterion at all; the Claude-only refusal gate widens that divergence (a refused Claude call now scores no, a failed Codex cat of the same SKILL.md still scores yes) with neither doc updated. (trigger: src/coder_eval/criteria/skill_triggered.py) (restates: Axis 8: skill_triggered now ignores errored Skill calls, but three definition surfaces still document the old contract)
🟡 🟡 task_loader.py — expand_dataset was migrated to the new row_split_label, _stratified_sample (line 342) was not. The PR documents the divergence in two prose blocks instead of applying the one-line row_split_label(row, field) or "" that satisfies both conventions. (trigger: src/coder_eval/orchestration/task_loader.py) (restates: Axis 1: Two competing missing-value conventions kept side by side)
🟡 🟡 run_command.py — only run learned about splits. cli/plan_command.py still exposes exactly one option (--experiment, line 20) and expands no dataset, so the preview command the skill body makes its mandatory preflight cannot show per-split row counts, cannot detect a mistyped split, and cannot prove the binary supports --split at all. (trigger: src/coder_eval/cli/run_command.py) (restates: Axis 7: The new capability preflight claims to verify --split exists, but coder-eval plan has no such flag)
🟡 🟡 reference/templates/activation.yaml — three of the four activation/outcome surfaces this PR touches gained (or already had) type: "claude-code" + setting_sources: [] + a run_limits: cap block; the one file that is copied verbatim into user repos was left without any of them, and setting_sources defaults to ["project"] (pinned by tests/test_agent.py::test_setting_sources_default_is_project). (trigger: plugins/coder-eval/reference/templates/activation.yaml) (restates: Axis 7: The check-skill activation template omits type: "claude-code" + setting_sources: [])

Tests:

🟠 🟠 skill_triggered.py — CE036's live-verdict contract table was not extended for the new branch, although CLAUDE.md states that adding a live-criterion behaviour means adding ContractCases in the same change. tests/lint/live_verdict_contract.py:146 _skill() builds only success-status Skill calls, and both fixture builders type result_status as Literal["success","error","unknown"] with no | None (tests/_fixtures/live_criteria.py:33, live_verdict_contract.py:141) — so the in-flight result_status=None shape the EarlyStopWatcher actually feeds live_verdict is not merely untested, it is unrepresentable in the harness that exists to prove monotonicity. (trigger: src/coder_eval/criteria/skill_triggered.py) (restates: Axis 6: skill_triggered's new engagement gate is a denylist over a 4-valued result_status)
🟠 🟠 templates/ci-outcome-fixture/evals/*.yaml — four new task YAMLs ship with no load_task round-trip anywhere, while the bundled plugin templates get exactly that check two hundred lines earlier in the same file (tests/test_custom_lint.py:1548, :1565). The missing assertion is why an invalid suite_thresholds-without-dataset task shipped. (trigger: templates/ci-outcome-fixture/evals/activation.yaml) (restates: Axis 3: Fixture eval YAMLs under templates/ci-outcome-fixture/evals/ escape every tasks/** validation scan)
🟡 🟡 tests/test_custom_lint.py — CE061 ships one repo-sweep assertion and no positive case, so every weakening of its inlined detection (threshold, .lower(), locator tuple) stays green; its sibling CE060, added in the same commit, has seven synthetic cases behind an extracted _offenders helper. (trigger: tests/test_custom_lint.py) (restates: Axis 3: CE061 has no synthetic-offender test)
🟡 🟡 CE060/CE061 scan tasks/ only, so neither rule sees the artifacts this PR exists to ship: plugins/coder-eval/reference/templates/outcome-rows.jsonl + activation-rows.jsonl (the rows every user's suite is copied from) and templates/ci-outcome-fixture/evals/. A shipped template could carry a partly-labelled dataset or a verbatim answer leak and the new rules would be silent — the first thing a user inherits is the un-linted copy. (trigger: tests/test_custom_lint.py)
🟡 🟡 tasks/skills/ci-outcome.yaml — the metric-vocabulary check and the per-row max_usd cap check run against the bundled template only, never against the checked-in suite that actually spends ~$4.30 a pass; both sensors sit outside the helper the sample test claims to share. (trigger: tasks/skills/ci-outcome.yaml) (restates: Axis 3: Two outcome-suite contracts are asserted only against the bundled template)
🔵 🔵 task_loader.py — row_split_label is a new PUBLIC function (three lint tests already import it directly) with no direct unit test. Its contract — absent/null/"" are unlabelled but a falsy 0 is a real label — is asserted only through expand_dataset in tests/test_dataset_expansion.py, so a regression in the helper is caught only where a caller happens to exercise it. (trigger: src/coder_eval/orchestration/task_loader.py)

Downstream consumers:

🟠 🟠 run_command.py / models/results.py — --split is now the primary invocation shape (Stage A/B on --split train, confirmation on --split test), but it is recorded nowhere. BatchRunConfig.split (orchestration/config.py:89) is the request only; RunSummary has no split/row-selection field, and reports.py / reports_html.py / reports_junit.py contain no split reference at all. So a run.json from the train arm is byte-indistinguishable from the test arm or from an unfiltered run, and nothing — not the skill, not an external consumer — can verify which population produced a number it is comparing. (trigger: src/coder_eval/cli/run_command.py)
🟠 🟠 skill_triggered.py — the repo already has a worked pattern for a scoring-semantics change (docs/REPORT_SCHEMA.md:143-151: a marker field in the run record plus an explicit "not score-comparable, consumers should segment on it" note, as done for system_prompt_semantics). The criterion's definition changed and neither half shipped: no marker, no REPORT_SCHEMA entry, so every stored skill_triggered accuracy/recall/F1 silently spans two definitions. (trigger: src/coder_eval/criteria/skill_triggered.py) (restates: Axis 8: skill_triggered now ignores errored Skill calls, but three definition surfaces still document the old contract)
🟡 🟡 skill_triggered.py — thresholds tuned against the old definition were not revisited. Every existing suite_thresholds: {recall.yes: …} on a skill_triggered criterion (tasks/skills/lint-tasks-activation.yaml, both shipped templates, the fixture suite) is now evaluated by a stricter classifier, so a suite that passed at 0.7 can fail on unchanged agent behaviour with no note saying why. (trigger: src/coder_eval/criteria/skill_triggered.py) (restates: Axis 8: skill_triggered now ignores errored Skill calls, but three definition surfaces still document the old contract)

Display & mapping dicts:

🟡 🟡 models/criteria.py:1204 still renders "Only count Skill invocations whose 'skill' parameter matches this name" into the GENERATED plugins/coder-eval/reference/criteria.md:260 — the criteria reference bundled into the published plugin and read by the skills. make plugin-reference was not re-run, and CE033 forbids hand-editing the output, so this one must be fixed at the model. (trigger: src/coder_eval/criteria/skill_triggered.py) (restates: Axis 8: skill_triggered now ignores errored Skill calls, but three definition surfaces still document the old contract)
🟡 🟡 No rendering surface gained a slot for the new outcome. The refusal is dropped into logger.debug and the criterion's details string (line 165) still renders only observed=… expected=…, so task.json, the markdown report and the HTML report all show a plain observed='no' — identical to "the model never called the skill", which needs the opposite remedy. (trigger: src/coder_eval/criteria/skill_triggered.py) (restates: Axis 6: The refused Skill call is recorded only as a DEBUG line and never reaches the criterion's details)
🟡 🟡 A selector-emptied run has no report representation. reports_junit.py:328 renders skipped_tasks as a synthetic skipped testsuite (green in every CI reader), and reports.py does not render skipped tasks at all — so the only surface distinguishing "zero rows because the split name was mistyped" from a completed run is one yellow console line. (trigger: src/coder_eval/orchestration/task_loader.py) (restates: Axis 6: A --split value matching no row is a global invocation error but is demoted to skipped_tasks)

Daily/nightly:

🟠 🟠 The split LABEL VALUES were renamed tune/holdout → train/test in the in-repo suite and in every doc example, and the PR states no blast radius for it. Any external caller that pins the old value — a saved eval-runner / coder-eval-uipath nightly command, a scheduled --split tune invocation, a dashboard keyed on the label — now lands exactly on the exit-0 hole: ValueError → skipped_tasks → green run of zero rows. This is a data-value break dressed as a terminology cleanup. (trigger: tasks/skills/lint-tasks-activation-rows.jsonl) (restates: Axis 6: A --split value matching no row is a global invocation error but is demoted to skipped_tasks)
🟠 🟠 The PR does not say what happens to any recurring run that already scores skill_triggered. Recall/F1 can step down for byte-identical agent behaviour the first night after this merges, and the only discriminator in the artifact is RunSummary.framework_version — which no trend line segments on. State the version at which the definition changed and name the consumers that must re-baseline. (trigger: src/coder_eval/criteria/skill_triggered.py) (restates: Axis 8: skill_triggered now ignores errored Skill calls, but three definition surfaces still document the old contract)
🟡 🟡 Nothing executes the new suite. No workflow references tasks/skills/** (CI's coder-eval run tasks/*.yaml is depth-1 by design, per the comment at pr-checks.yml:548), so the checked-in worked example, its 10 rows and the mounted templates/ci-outcome-fixture/ are exercised by shape assertions only — which is precisely why the fixture's invalid activation.yaml shipped. The PR should say whether a nightly adopts this suite (and who pays the ~$4.30/pass, ~$40/A-B-round) or state plainly that it is a manually-run reference artifact. (trigger: tasks/skills/ci-outcome.yaml)

Harness & Lint Improvements

Static checks (lint / type):

[ce-lint] CE067 — a lint rule that enumerates dataset rows must read the UNSAMPLED rows. AST check over tests/lint/** and tests/lint_tests/**: a call to orchestration.task_loader.expand_dataset(...) inside a @pytest.mark.lint class body (or a shared reader it calls) is a violation unless the same function first asserts task.dataset.sample_per_stratum is None or task.dataset.sample_seed is not None. The sanctioned reader is _load_dataset_rows + local ${row.*} substitution, which is what CE060 already does. Live offender at HEAD: tests/lint_tests/test_lint_task_surfaces.py:194 (for row in expand_dataset(task, task_file_dir)) — CE061's whole row scan. Wire as a BaseRule in tests/lint/rules/ce067_lint_rules_read_unsampled_rows.py + tests/lint/runner.py (its subject is one .py AST, so it belongs in the runner, unlike CE060/CE061 themselves). Prevents: Finding A3-low "CE061 scans rows through expand_dataset": the moment any tasks/** suite sets sample_per_stratum without a sample_seed, CE061 silently checks a different random subset each run and a leaking row passes CI intermittently. Also protects any future row-reading rule from the same nondeterminism.
[ce-lint] CE068 — the CLI exit gate must consult every failure bucket RunSummary declares. Model-driven check (the CE058 mirrored_result_fields shape): collect the fields of models.results.RunSummary whose name matches tasks_failed|tasks_error|skipped*|*_gates, then AST-walk cli/run_command.py's exit-code expression and require each to be referenced or carry an explicit # noqa: CE068 — <reason> naming why that bucket is not a failure. Today the gate is if summary.tasks_failed > 0 or summary.tasks_error > 0 or failed_suite_gates > 0: (run_command.py:560) and skipped_tasks is referenced nowhere in it. Wire as a @pytest.mark.lint class (it reads model metadata + one AST) in tests/lint_tests/test_lint_wiring_pointers.py. Prevents: Finding A6-high "a --split value matching no row … the run exits 0 with zero rows measured" — reproduced end-to-end at pr-130. The per-finding fix (SplitSelectorError, now in task_loader.py:39) closes ONE demoted error; this closes the class, i.e. any future demoted-to-skipped cause producing a green run of zero rows.
[ce-lint] CE069 — every coder-eval <command> … --flag written in plugins/** or docs/** must resolve to a flag that command declares. The inverse direction of CE046, over the same reader tests/lint/cli_flags.py::long_flags (which resolves Typer OptionInfo.param_decls), so both directions share one declaration source. Scan fenced/inline coder-eval <cmd> invocations, extract --long flags, and require each in long_flags(<cmd fn>). Declare the existing blind spot in the docstring: Typer-derived flags carry no param_decls, so the checker must union in the parameter-name-derived form or exempt it explicitly. Prevents: Finding A7-low "the new capability preflight claims to verify --split exists, but coder-eval plan has no such flag" (SKILL.md:44-48). Same rule catches the recurring class: a skill or tutorial telling a user to run a flag combination the binary rejects, which costs a paid round to discover.
[ce-lint] CE070 — a shipped measurement template's per-split row count must meet a single declared minimum, and the estimator must enforce the same number. Declare MIN_PAIRED_N once in reports_stats.py (replacing the bare if len(common_tasks) < 2: at reports_stats.py:549), then have the rule (a) read every plugins/coder-eval/reference/templates/*-rows.jsonl, group by split, and assert each split's count ≥ MIN_PAIRED_N, and (b) assert the execution-track section of optimize-skill/SKILL.md states that number. At HEAD outcome-rows.jsonl is still 2 train / 2 test. Same family as CE040 (bootstrap_p_floor seam): the constant is derived, never respelled. Prevents: Finding A8-medium "the outcome template ships 2 train rows and the execution track states no minimum-n, while its gate is a paired t-test that can promote on n=2 with p=0.000". Also closes the asymmetry the verifier found: activation.yaml:13-16 states the sizing rule, outcome.yaml has no counterpart.
[ce-lint] CE071 — no # type: ignore in a file that no pyright pass analyzes. Derive the analyzed set from [tool.pyright].include/exclude in pyproject.toml plus tests/lint/pyright_config.py::INCLUDE (the second-pass generator, which already reads the same TOML), then flag any # type: ignore comment in a file outside it. An inert suppression is worse than none: it reads as "a reviewer decided this" while suppressing nothing. There are 158 such comments under tests/ today, so land it with a one-time sweep or a checked-in baseline. Pair with the ruff item below. Prevents: Finding A1/A2-low "test helper widens the result_status Literal to a bare str and silences the resulting error with a self-inflicted, unjustified # type: ignore[arg-type]" (tests/test_skill_triggered.py:24,32). The ignore is unjustified AND inert; this rule makes the second half mechanical, and deleting it forces the honest annotation Literal["success", "error", "unknown"] | None.
[ruff] Add "PGH" to [tool.ruff.lint].select in pyproject.toml (currently E,F,I,N,W,UP,B,SIM,RUF,PLR0915,PLR0912,C90). PGH003 forbids a blanket # type: ignore with no rule code and PGH004 a blanket # noqa; both are zero-config and complement CE071, which handles the coded-but-inert case. Expect a small sweep across the 189 existing ignore comments; run ruff check --select PGH --statistics first to size it. Prevents: Same finding as CE071 (tests/test_skill_triggered.py:24). Ruff reaches it today with no new rule module, so it is the cheapest half of that fix.
[pyright] Set reportUnnecessaryTypeIgnoreComment = "error" in [tool.pyright]. tests/lint/pyright_config.py::build_config copies every rule setting verbatim into the generated second-pass config, so one edit governs both passes and the two cannot drift. This turns a suppression that no longer suppresses anything into a hard failure instead of a fossil. Prevents: Finding A1/A2-low (# type: ignore[arg-type] at tests/test_skill_triggered.py:32). Once the parameter is annotated as the real Literal, this setting is what deletes the stale comment rather than leaving it to the next reader.
[ce-lint] CE072 — under src/coder_eval/criteria/, a comparison against a proper subset of a closed status Literal must be an allowlist. Widen CE018's existing engine (tests/lint/rules/ce018_no_final_status_name_denylist.py, which already detects ==/!=/in against a member-name set) from a hardcoded FinalStatus name set to a small registry that also derives CommandTelemetry.result_status's members from models/telemetry.py. Flag if <expr>.result_status == "error": where the ACCEPTING branch is the else — i.e. the comparison names the rejected values rather than the accepted ones. State the boundary in the docstring: the rule reads branch polarity syntactically, so # noqa: CE072 is the escape for a genuine reject-side test. .claude/harness-candidates.md records this denylist mirror as deliberately deferred under CE054's rationale; the pr-130 finding is the second occurrence, which is the promotion trigger. Prevents: Finding A6-high (cross-axis 1/2/6/7/8) "skill_triggered's new engagement gate is a denylist over a 4-valued result_status: only "error" is refused, so "unknown" and None still count as engagement" (skill_triggered.py:74) and the sibling A6-medium ungated file-read scan at :88-90. The fix has since landed as _delivered (an explicit allowlist), so this rule guards the NEXT criterion rather than the current one — which is exactly the CLAUDE.md standing rule.
[ce-lint] CE073 — one declaration of the artifact-scoring criterion-type set, derived from the SuccessCriterion union. Forbid a set/tuple/frozenset display of ≥2 string literals in tests/** whose members are all criterion type tags (read the tags from models.criteria.SuccessCriterion). The sanctioned form is one module-level constant beside _outcome_metric_vocabulary, itself derived from the union. Same seam family as CE062 (single f1 implementation) and CE040. Prevents: Finding A1-low "the artifact-scoring criterion-type set is a hardcoded literal duplicated in two tests in the same file" ({"file_check", "json_check", "run_command", "cli_called", "command_executed"} at the pr-130 lines 1621 and 1726). A new artifact-scoring criterion type would leave both copies stale and both tests green.
[ce-lint] CE074 — mkdocs nav: titles are sentence case. Add a check to the CE028 doc-index family (tests/lint/doc_indexes.py, which already owns nav: + extra.docs_index as the SSOT for the three generated tables): after stripping a NN · prefix, a title whose second-or-later word starts uppercase and is not in a small proper-noun allowlist (Docker, Claude, Bedrock, GitHub, …) fails. CE028 already regenerates README.md / docs/index.md / docs/llms.txt from the same nav, so one fix propagates. Prevents: Finding A1-low "tutorial titles 08 and 09 use Title Case while 01-07 use sentence case" (mkdocs.yml:107-109, propagated into docs/tutorials/README.md:23-24 and docs/llms.txt:53-54 by the generator). Casing drift in a generated surface is invisible to review and permanent once shipped.
[ce-lint] CE075 — every tree a skill copies into a user's repo has its task YAML round-tripped through load_task, discovered from ONE registry. tests/lint/task_yaml_discovery.py already parses YAML correctly for two consumers (CE052 over templates/, the plugin-template test over plugins/coder-eval/reference/templates/), but the ROOTS are hardcoded per consumer, so a new fixture tree is covered by nobody until someone remembers. Derive the root set instead: every sandbox.template_sources[*].path referenced by any in-repo task, plus templates/ and plugins/coder-eval/reference/, minus experiments/ subdirs (which are experiments, not tasks). Fail on a task YAML under a registered root that load_task rejects. Prevents: Finding A3-high "fixture eval YAMLs under templates/ci-outcome-fixture/evals/ escape every tasks/** validation scan, and activation.yaml is an invalid TaskDefinition (suite_thresholds with no dataset:)" — a shipped-invalid artifact a README told users to run. CE052 now covers that specific tree; deriving the roots is what stops the NEXT deliberately-placed-outside-tasks/ fixture from being unvalidated.
[ce-lint] CE076 — a measured figure quoted on more than one plugin surface has ONE declaration. Extend CE064's claim registry (tests/lint/computed_claims.py, which already enforces "an arithmetic-bearing table that no registered claim names is a failure") with a MEASURED-FIGURE claim class: a span matching \b\d+/\d+ (rows|engaged)\b anywhere in plugins/coder-eval/** or an in-repo suite's comment block must map to a registry entry that records the figure AND the form it measures; two surfaces quoting the same ratio with different attributions fail. Second half, same rule: parse the initial_prompt out of the SKILL.md YAML fence and require every shipped exemplar's prompt opening to match its SHAPE (slash-prefixed or not) — today tests/lint_tests/shared.py:405-406 asserts only invocation in opening, a bare substring the backticked imperative already satisfies. Prevents: Finding A1/A8-medium "the slash-form prompt rule in optimize-skill/SKILL.md contradicts the shipped artifacts … and the 6/6 figure is attributed three ways". Verified still live at HEAD: shared.py:369 still says "by slash form", shared.py:406 still substring-matches, and outcome.yaml:186 / SKILL.md:344 still carry the imperative-only form. A doc rule no shipped artifact obeys is a rule the next author copies the wrong side of.
[ce-lint] CE077 — module-size ratchet for tests/lint_tests/** and a cap on rule classes per module. Same shape as the existing CLAUDE.md size gate (tests/lint_tests/test_lint_claude_md_size.py): a per-module line ceiling set one step above today's worst, plus "no module holds rule classes from more than one subject family" enforced via a declared <module> -> family map that must cover every @pytest.mark.lint class (a class in no family is a failure — the CE064 coverage shape). Prevents: Finding A5-low "tests/test_custom_lint.py grows to 4385 lines / 59 marked rule classes in one module". The split into tests/lint_tests/ has since happened; without a ratchet the grab-bag reforms one module at a time, and the coverage half is what makes the split a rule instead of a one-time cleanup.

Harness improvements (not statically reachable):

A score-comparability ledger, diff-gated like the estimator protocol. Add a second gate beside tests/lint/estimator_ledger.py (the pull_request-only estimator-protocol job): when a PR's diff touches a decision predicate in src/coder_eval/criteria/** — the expression that maps a trajectory to observed_label / a score — require a row in a new ## Scoring changes table in docs/REPORT_SCHEMA.md giving the criterion, the version, the direction of the change, and the named external consumers (coder-eval-uipath / eval-runner). Model it verbatim on the reference: migration block's existing bold Score-comparability warning (docs/TASK_DEFINITION_GUIDE.md:1332), which is this repo's own precedent for exactly this. Why not static: It is a DIFF property: the gate must compare a criterion's scoring predicate against a base ref, and a working tree has no base ref — the same reason estimator_ledger.py carries no CE number and cannot run inside make verify. Whether a change alters the meaning of a persisted metric is also a judgement no AST walk can make; the gate can only force the judgement to be recorded. Prevents: Finding A8-high "skill_triggered now ignores errored Skill calls, but three definition surfaces still document the old contract (one of them generated) and no score-comparability marker was added". A run.json written before the change scores 1.0 on a trajectory that now scores 0.0, and only RunSummary.framework_version distinguishes them — so a trend line in the separate nightly repo steps down for identical agent behaviour and reads as a skill regression.
Every lint rule class must fire on a synthetic offender — asserted by a meta-test, not by convention. Extend tests/lint_tests/test_lint_harness_meta.py with a sweep over every @pytest.mark.lint class: it must expose its detection as a callable (a module-level reader under tests/lint/ or a classmethod, the CE060::_offenders / outcome_prompt_leak.leaks shape), and must contain at least one test that feeds that callable a tmp_path fixture built to violate the rule and asserts a NON-EMPTY result. Detection logic inlined in the body of an assert not offenders sweep is a failure, because that shape cannot be exercised in the firing direction at all. Why not static: The property is "this detector returns findings on a known-bad input", which requires constructing the input and executing the rule. A static check can see that a test function exists; it cannot see that the assertion is assert offenders rather than assert not offenders over a fixture that is actually an offender. Prevents: Finding A3-medium "CE061 has no synthetic-offender test: every weakening of its inlined detection (threshold 12 -> 120, dropping .lower(), adding locator exemptions) leaves the suite green" — empirically confirmed by the verifier: four separate weakenings each produced 0 offenders and a green suite. CE060 next door is the house standard with seven synthetic cases; this makes the standard mechanical.
A degenerate-certainty guard on the paired estimator, with a pinned fixture. paired_comparison must return a result that cannot promote when the evidence is degenerate: n below MIN_PAIRED_N, or stddev(diffs) == 0 (which today makes paired_t_ci's half-width 0, so the CI is [d, d] and "excludes zero", while paired_t_test short-circuits to p = 0.0 at reports_stats.py:459-461). Add a fixture asserting the concrete false-promotion case — incumbent 0.5/0.5 vs candidate 1.0/1.0 on two train rows yields mean_diff +0.500, CI [+0.500, +0.500], p = 0.000 — and assert the guarded result is NOT promotable. Register the change in the estimator ledger, since it moves a rendered number. Why not static: It is numerical behaviour of an estimator on specific inputs; no lint rule can evaluate a t-distribution. The static half (CE070's row-count floor) prevents the shipped template from producing the input, but a user-authored two-row suite still reaches this code path. Prevents: Finding A8-medium (escalated by the verifier from "almost never informative" to a live false-promotion path): the template's own shape — one file_check plus the engagement criterion — is enough to produce identical per-row diffs and promote a skill on two rows with zero evidence about between-row variance.
A CLI exit-code smoke for invocation-scope selector errors. A test that actually invokes coder-eval run <suite> --split <typo> and asserts a non-zero exit, plus a companion asserting a run that resolves ZERO tasks exits non-zero with a named reason. Extend it to the other one-value-applied-to-every-task selectors (--sample, --sample-per-stratum, -D typos) so the policy resolve_all_tasks states in prose — "a global invocation error … is re-raised and aborts the run" (experiment.py:563-567) — is asserted per selector rather than per exception type. Why not static: Exit code is a property of the assembled process: the demote path runs in resolve_all_tasks, the gate in cli/run_command.py, and only running the CLI joins them. The verifier reproduced the pr-130 bug this way (Running 0 task(s), Results: 0/0 succeeded, exit 0) after static reading alone had only suggested it. Prevents: Finding A6-high "a --split value matching no row … the run exits 0 with zero rows measured", and it is the behavioural counterpart to CE068 above: CE068 pins the gate's SHAPE, this pins the OUTCOME.
One shape-assert helper per artifact family, with a coverage assert that names every sensor. Today _assert_outcome_suite_shape carries five checks and is shared by the bundled template and the checked-in suite, while three further template sensors (metric vocabulary, per-row cost cap, artifact-scoring) sit outside it — one of them copy-pasted inline for the second call site. Invert it: enumerate the sensor functions for the family (by naming convention or a decorator), assert every shipped artifact of that family goes through ALL of them, and fail when a sensor exists that some artifact does not run — the same "an artifact no claim names is a failure" rule CE064 already applies to arithmetic tables. Why not static: The subject is which assertions execute against which artifact, which is a pytest-collection property; a source-level check can see the call sites but not that the two lists are the same SET, and the copy-pasted inline duplicate reads as a call site while being a divergent copy. Prevents: Finding A3-medium "two outcome-suite contracts (threshold metric vocabulary, per-row cost cap) are asserted only against the bundled template, never against the checked-in suite that spends the money" — tasks/skills/ci-outcome.yaml costs ~$4.30 a pass and ~$40 an A/B round, and its suite_thresholds and run_limits are unasserted. Also prevents the A1-low duplicated criterion-type literal from re-appearing as the shape that motivates it.
A criterion evidence-contract test: a suppression that changes the score must reach details. Extend the tests/lint/live_verdict_contract.py engine (already the place where criteria are executed against synthetic trajectories and type-checked by the second pyright pass) with a per-criterion case: run the checker over a trajectory containing a suppressed signal, and assert the returned CriterionResult.details names it. A DEBUG log line is not enough — it lands in task.log per task, while the operator reads an aggregate recall of 0.0 across 24 rows and has no pointer to the cause. Why not static: It needs a constructed trajectory pushed through the real checker and the produced CriterionResult inspected; a static rule can see that logger.debug is called but not what the returned details string contains. Prevents: Finding A6-low "the refused Skill call is recorded only as a DEBUG line and never reaches the criterion's details, so the report shows observed='no' with no attribution" — a no from "the model never called the skill" needs the opposite remedy from a no from disable-model-invocation, and the report cannot tell them apart. (The _SUPPRESSED_RENDER_LIMIT note now in skill_triggered.py is the fix; this is what keeps the next criterion honest.)
Bind every prose-declared exemption list to its single constant, both directions. The pattern that worked for CE061 — LEAK_LOCATOR_FIELDS in src/coder_eval/leak_detection.py plus test_ce061_exemption_list_matches_claude_md — should become the standing shape: any allowlist/denylist a CLAUDE.md sentence or a skill surface enumerates must exist as ONE named constant in src/, and a test must fail when either side is edited alone. Sweep the remaining prose-enumerated lists (the merge-strategy defaults, the CE030 EXEMPT set, the LEAK_LOCATOR_FIELDS prose sentence's own membership) and add the binding test per list. Why not static: Matching a natural-language sentence's enumeration against a Python tuple is only mechanical once someone has declared WHICH sentence corresponds to WHICH constant; that pairing is authored, not inferable. Once paired, the assertion is trivial — which is precisely why the pairing should be a required artifact rather than a remembered convention. Prevents: Finding A7-medium (cross-axis 1/5/7/8) "CE061's locator-exemption list is declared twice (inline rule + CLAUDE.md prose) with nothing binding them: the copies already disagree, one entry is dead, and skill_name/expected_skill are omitted although the same PR mandates them". Note the verifier's open residue: expected_skill carries the same value as skill_name and is exempt in NEITHER copy, so a skill whose bare name is ≥12 chars (e.g. optimize-skill) still trips CE061 on a correctly-authored suite — the sweep should settle that while the binding is added.
Two conventions for one missing-value question must not coexist; assert the survivor. stratum_key / row_split_label now share a definition of "unlabelled", but the general lesson is the reviewable one: when a PR's own docstring documents a divergence ("it also turns an explicit None into the string \"None\"") instead of removing it, the review should require either the convergence or a test that PINS the divergent behaviour. Add the missing pin: a TestStratifiedSample case for an explicit null stratify value, so the grouping behaviour is asserted rather than described. Why not static: Whether two spellings of "absent" are meant to be the same population is a semantic judgement about the data model; no rule can distinguish a deliberate two-bucket design from an accident. What IS enforceable is the follow-up — that the chosen behaviour has a test — and that only exists once the behaviour is executed. Prevents: Finding A1-medium "two competing missing-value conventions kept side by side: _stratified_sample still buckets an explicit null into a separate \"None\" stratum". Unreachable with any in-repo dataset today, which is exactly why only a test — not a green make lint — would catch a user-authored JSONL hitting it.

Top 5 Priority Actions

Replace the denylist at src/coder_eval/criteria/skill_triggered.py:74 with an allowlist on result_status == "success", because today an in-flight Skill call (result_status=None, seen live via ToolStartEvent) can pass-stop an armed run and every early-stopped or timed-out call finalizes as "unknown" and still scores observed='yes' — add tests for None and "unknown", which the PR's "error"-only tests do not cover.
Fix the silent no-op run: make the split miss at src/coder_eval/orchestration/task_loader.py:498 raise a dedicated exception that resolve_all_tasks does NOT demote to skipped_tasks (or make cli/run_command.py:560 exit non-zero when zero tasks resolve), since --split trian was reproduced exiting 0 with "Running 0 task(s)" on the very invocation shape optimize-skill now makes primary.
Gate the file-read engagement scan at src/coder_eval/criteria/skill_triggered.py:88-90 on result_status != "error" and narrow it to tools that actually read file bodies, so a failed cat .../skills/ci/SKILL.md or an incidental Grep path no longer produces a false observed='yes' on the execution track's hard engagement gate (tasks/skills/ci-outcome.yaml:171-172, recall.yes: 1.0).
State a minimum train-row count for an outcome suite in plugins/coder-eval/skills/optimize-skill/SKILL.md's execution-track section and in the outcome.yaml header, because plugins/coder-eval/reference/templates/outcome-rows.jsonl ships 2 train rows and identical per-row diffs make reports_stats give a degenerate CI with p=0.000, satisfying the promotion rule at SKILL.md:765 on zero evidence about between-row variance.
Repair templates/ci-outcome-fixture/evals/activation.yaml:29 (suite_thresholds with no dataset: block — load_task raises today) and add a load_task round-trip assertion for every non-experiments YAML under the mounted fixture in tests/test_custom_lint.py:1732, since placing the fixture under templates/ deliberately escapes every tasks/** scan and that gap is what hid the defect.

Stats: 0 🔴 · 4 🟠 · 8 🟡 · 8 🔵 across 8 axes reviewed.

uipreliga requested review from akshaylive, bai-uipath and tmatup as code owners August 21, 2026 03:50

uipreliga mentioned this pull request Aug 21, 2026

feat(cli): plan expands datasets and takes --split, plus four correctness fixes #131

Open

uipreliga commented Aug 21, 2026

View reviewed changes

Navigation Menu

Uh oh!

feat(plugin): add optimize-skill's execution track and the outcome-suite template - #130

feat(plugin): add optimize-skill's execution track and the outcome-suite template#130
uipreliga wants to merge 1 commit into
pr/split-mvpfrom
pr/split-execution-track

uipreliga commented Aug 21, 2026

Uh oh!

github-actions Bot commented Aug 21, 2026 •

edited

Loading

Uh oh!

uipreliga left a comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Uh oh!

Conversation

uipreliga commented Aug 21, 2026

⚠️ This PR has no CI until it is rebased

Blast radius

The correction — two lint rules were squatting on main's numbers

What this PR does not do

Verification

Uh oh!

github-actions Bot commented Aug 21, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Code Review for PR #130

Review Checklist

Uh oh!

uipreliga left a comment

Choose a reason for hiding this comment

Review: coder_eval — pr:130

Summary

Blockers

Non-blocking, but please consider before merge

Nits

What's Missing

Harness & Lint Improvements

Top 5 Priority Actions

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

The correction — two lint rules were squatting on `main`'s numbers

github-actions Bot commented Aug 21, 2026 •

edited

Loading