Skip to content

simplify(search-eval): embed the canonical corpus - #4530

Open
ScriptedAlchemy wants to merge 1 commit into
fix/retire-unused-sync-controlsfrom
fix/search-eval-canonical-corpus
Open

ScriptedAlchemy wants to merge 1 commit into
fix/retire-unused-sync-controlsfrom
fix/search-eval-canonical-corpus

Conversation

@ScriptedAlchemy

@ScriptedAlchemy ScriptedAlchemy commented Oct 25, 2026 •

Copy link
Copy Markdown
Owner

Merge #4478 first, followed by #4526, #4510, #4524, #4489, #4485, #4527, #4528, #4530, #4531, #4529, #4534, and #4541.

This PR is number 9 of 13, based on #4528. Merge #4531 next.

Use merge commits. After merging a parent PR, change its child's base to master before merging the child. Auto-merge is disabled.

Fixes #4449.

Why

The packaged search evaluator embedded a separately maintained copy of its runtime corpus under crates/tracedecay-query/assets/runtime-root. Fifteen files duplicated existing canonical fixtures; the workload JSON existed only in the mirror. This made source and packaged qualification depend on keeping two editable trees synchronized.

Changes

  • Embed all 16 inputs directly from their canonical repository paths. Keep the logical paths and ordering unchanged.
  • Move the workload JSON to tests/fixtures/search_quality/query-lexical-graph-workload-v1.json without rewriting it, and delete the 15 redundant corpus copies.
  • Regenerate query compile inputs and public fixture groups with the maintained build generator. Correct its package-owner lookup to respect existing nested Bazel packages: the Cargo.lock fixture belongs to evals/agent_adoption, not a new parent evals package. Remove the obsolete generated parent BUILD and preserve handwritten rules.
  • Retain canonical LF policies, remove the mirror-specific policy, and point the receipt diagnostic at the canonical workload.
  • Add a production-path regression that enrolls an owned temporary fixture through the existing repository identity authority and compares source-backed and packaged evaluation identities, provenance, and pinned fallback receipts.

The runtime corpus remains embedded. This change makes no binary-size reduction claim.

Exact corpus inventory

Every removed path below had the prefix crates/tracedecay-query/assets/runtime-root/. The retained canonical path is the listed path at the repository root. All 15 duplicate pairs had zero byte differences, verified with SHA-256 and original Git blob identity; together they remove 149,745 redundant source bytes. The workload relocation also retains its exact original blob, b7e73cd31925006254c4aa7f020d3eab5bbb6337.

Logical/canonical path Bytes Operation
evals/agent_adoption/fixture/Cargo.lock 158 Delete identical mirror
tests/fixtures/context_eval_project/src/auth/login.rs 749 Delete identical mirror
tests/fixtures/context_eval_project/src/auth/session.rs 956 Delete identical mirror
tests/fixtures/context_eval_project/src/net/retry.rs 1,342 Delete identical mirror
tests/fixtures/context_eval_project/src/storage/config_store.rs 1,297 Delete identical mirror
tests/fixtures/sample.dockerfile 754 Delete identical mirror
tests/fixtures/search_quality/corpus/cargo-slot/src/main.rust.fixture 182 Delete identical mirror
tests/fixtures/search_quality/corpus/crates/tracedecay-domain/src/integration.rs 22,360 Delete identical mirror
tests/fixtures/search_quality/corpus/crates/tracedecay-domain/src/repository.rs 20,408 Delete identical mirror
tests/fixtures/search_quality/corpus/crates/tracedecay-domain/src/research/canonical.rs 5,338 Delete identical mirror
tests/fixtures/search_quality/corpus/crates/tracedecay-domain/src/research/coverage.rs 21,786 Delete identical mirror
tests/fixtures/search_quality/corpus/crates/tracedecay-domain/src/research/error.rs 1,645 Delete identical mirror
tests/fixtures/search_quality/corpus/crates/tracedecay-domain/src/research/time.rs 728 Delete identical mirror
tests/fixtures/search_quality/corpus/crates/tracedecay-domain/src/research/watermark.rs 1,641 Delete identical mirror
tests/fixtures/search_quality/corpus/crates/tracedecay-domain/src/session.rs 70,401 Delete identical mirror
tests/fixtures/search_quality/query-lexical-graph-workload-v1.json 49,842 Move unchanged

The embedding list is the runtime consumer; source workload/corpus loading uses the same logical paths. Generated compile inputs, LF handling, and the report diagnostic are updated together. Historical Git fixture packs and distinct/versioned fixtures retain their existing ownership.

Preserved semantic pins

  • Workload: sha256:b40c4c470dc2afae55debd2930d39f9f5d56494afd8a010672fb53bceb0295ec
  • Source commit: 8312618fee8109b16be09e65f45118b4e550fa14
  • Source tree: 587aed48f162f74765da3a35fc41850523ea377d
  • Train fallback receipt: sha256:7f52a7213e6caab00e6a2bfca4a24ccdfe3f37fcd2312fd86da34807a397b846
  • Validation fallback receipt: sha256:f8f6e5973a2c07c2e3dc46ec270c75e819f6dc333030f8acabdf1580311577eb

All 67 query definitions, labels, quoted provenance, and train/validation boundaries retain their original bytes. No receipt or semantic identity was re-pinned.

Validation

Canonical Bazel validation on the integrated parent tree:

  • BEFORE: evaluator analysis failed because //evals:agent_adoption_fixture used a glob that could not cross the existing evals/agent_adoption package boundary. Invocation 173f64e8-65b9-4ef4-8f07-4063fda24401; no tests executed.
  • AFTER: 28 evaluator tests passed, including full source-backed/packaged corpus and receipt parity; affected Rust formatting passed. Invocation 67158e9e-29e7-4c4b-95c0-2ef4b74f06f5.
  • Query search-quality prefix: 15 passed, nonzero helper passed. Invocation 07b62bf5-4f1c-4ff4-b920-1908eb857cf2.
  • Generator behavioral tests: 8 passed, including nested package ownership. Maintained generator --check: all 61 generated BUILD files current, invocation 21e4261d-c459-4203-ac18-daec4236a130.
  • Scoped query/evaluator Clippy passed, invocation 1aa066dc-10d5-4d1b-a39a-e17e16e1b8c6.
  • Independently verified all 15 deleted mirrors are byte-identical to their canonical inputs and the workload relocation preserves its exact bytes. No semantic identity or receipt was re-pinned.
  • The evaluator deliberately retains its existing qualification Fail result for conceptual misses. Passing these tests proves preservation and truthful reporting, not a qualification Pass.

Stack

Based on #4528 (fix/retire-unused-sync-controls). This layer contains only the corpus consolidation, generated compile inputs, and focused regression coverage.

@changeset-bot

changeset-bot Bot commented Oct 25, 2026 •

Copy link
Copy Markdown

⚠️ No Changeset found

Latest commit: 2e642cb

Merging this PR will not cause a version bump for any packages. If these changes should not result in a new version, you're good to go. If these changes should result in a version bump, you need to add a changeset.

Click here to learn what changesets are, and how to add one.

Click here if you're a maintainer who wants to add a changeset to this PR

@devin-ai-integration devin-ai-integration Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ Devin Review: No Issues Found

Devin Review analyzed this PR and found no bugs or issues to report.

Devin Review

@ScriptedAlchemy
ScriptedAlchemy force-pushed the fix/retire-unused-sync-controls branch from 35c547d to 33d47ea Compare October 25, 2026 10:35
@ScriptedAlchemy
ScriptedAlchemy force-pushed the fix/search-eval-canonical-corpus branch from d5b13e2 to 46532d2 Compare October 25, 2026 10:35
@ScriptedAlchemy
ScriptedAlchemy force-pushed the fix/retire-unused-sync-controls branch from 33d47ea to 0edfed2 Compare October 25, 2026 11:27
@ScriptedAlchemy
ScriptedAlchemy force-pushed the fix/search-eval-canonical-corpus branch from 46532d2 to deb6c24 Compare October 25, 2026 11:27
@ScriptedAlchemy
ScriptedAlchemy force-pushed the fix/retire-unused-sync-controls branch from 0edfed2 to b9864dc Compare October 25, 2026 19:42
@ScriptedAlchemy
ScriptedAlchemy force-pushed the fix/search-eval-canonical-corpus branch from deb6c24 to e6d05d4 Compare October 25, 2026 19:42
@ScriptedAlchemy
ScriptedAlchemy force-pushed the fix/retire-unused-sync-controls branch from b9864dc to 91804a5 Compare October 25, 2026 20:06
@ScriptedAlchemy
ScriptedAlchemy force-pushed the fix/search-eval-canonical-corpus branch from e6d05d4 to e5d734b Compare October 25, 2026 20:06
@ScriptedAlchemy
ScriptedAlchemy force-pushed the fix/retire-unused-sync-controls branch from d368062 to 9cf7a68 Compare October 26, 2026 03:42
@ScriptedAlchemy
ScriptedAlchemy force-pushed the fix/search-eval-canonical-corpus branch from 379a571 to e852784 Compare October 26, 2026 03:42
@ScriptedAlchemy
ScriptedAlchemy force-pushed the fix/retire-unused-sync-controls branch from 9cf7a68 to 4f5bbe9 Compare October 26, 2026 03:51
@ScriptedAlchemy
ScriptedAlchemy force-pushed the fix/search-eval-canonical-corpus branch from e852784 to 3e31ec1 Compare October 26, 2026 03:51
@ScriptedAlchemy
ScriptedAlchemy force-pushed the fix/retire-unused-sync-controls branch from 4f5bbe9 to fa929ee Compare October 26, 2026 03:59
@ScriptedAlchemy
ScriptedAlchemy force-pushed the fix/search-eval-canonical-corpus branch from 3e31ec1 to e4d884f Compare October 26, 2026 03:59
@ScriptedAlchemy
ScriptedAlchemy force-pushed the fix/retire-unused-sync-controls branch from fa929ee to fb0786d Compare October 26, 2026 05:15
@ScriptedAlchemy
ScriptedAlchemy force-pushed the fix/search-eval-canonical-corpus branch from e4d884f to 12bc45d Compare October 26, 2026 05:15
@ScriptedAlchemy
ScriptedAlchemy force-pushed the fix/retire-unused-sync-controls branch from fb0786d to 25c2f98 Compare October 26, 2026 05:34
@ScriptedAlchemy
ScriptedAlchemy force-pushed the fix/search-eval-canonical-corpus branch from 12bc45d to 0ffccb3 Compare October 26, 2026 05:35
@ScriptedAlchemy
ScriptedAlchemy force-pushed the fix/retire-unused-sync-controls branch from 25c2f98 to 603f723 Compare October 26, 2026 08:58
Restack unique #4530 onto the rewritten #4528 tip.
@ScriptedAlchemy
ScriptedAlchemy force-pushed the fix/search-eval-canonical-corpus branch from 0ffccb3 to 2e642cb Compare October 26, 2026 08:58
ScriptedAlchemy added a commit that referenced this pull request Oct 26, 2026

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant