Add shape (character-class skeleton) index layer#5
Merged
Conversation
Accelerates literal-free pattern regexes (phones, SSNs, IPs, dates) that
have no fixed n-gram to prune on. Text is indexed under a second, coarser
tokenizer: letters collapse to L, digits to D, structural punctuation is
kept, everything else is a boundary. Only shapes containing a non-letter
are emitted, so a short shape stays selective (a phone DDD-DDDD is rare
even though every character class is common).
Queries project the regex through the same coarse alphabet and reuse the
existing extractRegexLiterals, so there is no bespoke analyzer and no
hand-coded assumptions about what a query looks like: /\d{3}-\d{2}-\d{4}/
becomes D{3}-D{2}-D{4} and the trusted extractor yields DDD-DD-DDDD.
The index stays a superset candidate filter and the per-row scan still
delivers exact grep, so shape tokens (mandatory, hence never a false
negative) only change which blocks are fetched, never the results. On
25k WildChat rows this takes SSN/date/time/phone regexes from full scans
to 5-26% of blocks for ~2% more index. The layer is always on and not
configurable; its settings live in kv metadata (format, not config) so
queries tokenize identically and shape-less indexes are detected.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds a second, coarser tokenization layer to the index so literal-free pattern regexes (phone numbers, SSNs, IPs, dates) can prune blocks by their digit/punctuation structure instead of falling back to a full scan.
How it works
L, every digit toD, and a small set of structural punctuation (@.-_/:) is kept verbatim; everything else is a run boundary. Shape n-grams (length 4) are taken over this tiny alphabet.ngramcolumn, prefixed with a control byte (\x01) so they can never collide with text n-grams.extractRegexLiteralsto find the mandatory shape runs, so there are no hand-coded assumptions about what a query "looks like".Correctness
Shape tokens are only required when mandatory across the whole pattern (single projected branch) and only spent on a branch with no literal n-gram to prune on, so they never cause a false negative. The layer is gated on new kv metadata (
hypgrep.shape_ngram_length,hypgrep.shape_chars), so an index built without it is never asked for shape tokens.i+ucase folding andv-mode class syntax bail out conservatively.Cost
About 2% index growth.
Tests
Adds
test/shape.test.jscovering shape extraction, projection through groups/classes/ranges, flag edge cases, and end-to-end brute-force equivalence (no false negatives).