Skip to content

Keep live checks aligned with corpus and output bounds - #12

Merged
exploitintel merged 1 commit into
mainfrom
fix/live-corpus-checkpoints
Sep 25, 2026
Merged

exploitintel merged 1 commit into
mainfrom
fix/live-corpus-checkpoints

Conversation

@exploitintel

Copy link
Copy Markdown
Owner

Live verification could report two false failures after normal corpus changes: an upstream Git file named by the wide-fence test was removed, and the shared MCP output budget omitted catalog_kind from a partial row. Check the complete API file inventory before declaring the optional Git case unavailable, and compare filtered row identities and available fields against the API's bounded page while honoring disclosed truncation. Incorrect values, changed ordering, incomplete inventories, and API errors still fail. The independent oversized ExploitDB read remains mandatory.

Validation: Ruff, formatting, mypy, and 1,970 hermetic tests passed at 97.33% coverage. Against the current production API with the deployed SDK version, all 227 live checks passed; two explicit corpus-conditional skips cover an unavailable undocumented verdict and the removed wide-fence file. Three independent review lenses found no significant issues; reviewers also exercised negative controls. No runtime behavior or deployment changes.

@claude

claude Bot commented Sep 25, 2026

Copy link
Copy Markdown

Reviewed this PR against the read-only MCP server's provenance/escaping/truncation-disclosure/no-scope-creep checklist.

Scope: test-only changes (tests/test_live.py, tests/test_live_parameter_effects.py) plus a CONTRIBUTING.md note. No production code is touched, so none of the runtime-facing concerns (corpus text escaping, derived judgment/ranking, stored-analysis fail-open, PoC token leakage, Host/transport-security defaults) are in play here.

Checked specifically:

  • The CAPPED_WIDE_FENCE skip in test_live.py only fires after confirming, via the complete file inventory (listing["items"] length matches detail["file_count"]), that the named file is truly absent — a failed request or partial listing still fails the test rather than skipping.
  • The test_catalog_kind_holds_on_every_row rewrite correctly ties the "field may be missing" allowance to result.structured.truncated, which matches bounded_result()'s actual behavior in structured.py (it sets truncated=True whenever a dict field is dropped for budget reasons). A present-but-wrong catalog_kind still fails regardless of truncation, so this doesn't mask real mismatches — it only tolerates the documented truncation case.
  • Both helpers used (tools._post_for_artifact, tools._api.get) are existing, already-allowlisted read paths (/api/v1/pocs, /api/v1/poc-files, /api/v1/pocs/{id}) also exercised elsewhere in the test suite and in production code (tools.py:671); no new endpoints or write paths introduced.

No findings. LGTM.

@exploitintel
exploitintel merged commit e937f60 into main Sep 25, 2026
4 checks passed
@exploitintel
exploitintel deleted the fix/live-corpus-checkpoints branch September 25, 2026 18:15
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant