Skip to content

Select text detectors before scanning - #47

Merged
sidmohan0 merged 2 commits into
mainfrom
feat/text-detector-selection
Oct 8, 2026
Merged

sidmohan0 merged 2 commits into
mainfrom
feat/text-detector-selection

Conversation

@sidmohan0

@sidmohan0 sidmohan0 commented Oct 8, 2026 •

Copy link
Copy Markdown
Contributor

Problem

Text scans run every enabled detector even when callers only want certain entity types. MCP filters findings afterward, so unwanted matches still consume time and memory. See DataFog/datafog-mcp#59.

Solution

  • Add optional entities selection to text scans and skip unselected detectors before matching.
  • Preserve existing results when selection is omitted, along with Unicode offsets, ENV/SQL boundaries, and locale/UUID activation rules.
  • Reject invalid selections through the existing configuration-error contract.
  • Expose the option in Python, Node, and WASM types and verify it across the installed packages. Structured scanning remains unchanged.

Validation: Rust format, workspace/all-features Clippy, and 122 tests pass (one existing benchmark ignored). Python installed-wheel/typing, Node package/TypeScript, and WASM browser/TypeScript checks pass. Node validation used host Node 25; CI should confirm the declared Node 24 environment.

Five fresh-process runs per case on Mac arm64/Python 3.11, using 10 MB synthetic log fixtures from #59:

Case Published Core: median scan / peak RSS Selected candidate: median scan / peak RSS
No selected entities 0.243 s / 178 MB 0.114 s / 61 MB
~1,000 selected entities per MB 0.246 s / 179 MB 0.125 s / 66 MB

The control's 271,710 unrequested findings disappear; the mixed case returns the 9,999 planted selected findings. Candidate scans with selection omitted retain the published counts and similar performance. These are Core-only measurements, not end-to-end MCP/model benchmarks.

Closes #46. Downstream MCP integration is tracked in DataFog/datafog-mcp#63. No package version or release is changed; MCP adoption requires a subsequent Core release.

Draft MCP implementation: DataFog/datafog-mcp#65.

@sidmohan0
sidmohan0 marked this pull request as ready for review October 8, 2026 15:47
@sidmohan0
sidmohan0 merged commit 74e315b into main Oct 8, 2026
20 of 21 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Allow text scans to select entity types before detection

1 participant