feat(knowledge): add token, sentence, recursive, and regex chunkers

[waleedlatif1]

Summary

• Add 4 new chunking strategies: token, sentence, recursive, regex
• Extract shared chunking utilities (estimateTokens, cleanText, addOverlap, splitAtWordBoundaries) into utils.ts
• Wire strategy selection through full stack: UI → API → service → document processor
• Default to auto-detection (existing behavior), users can optionally select a strategy
• Add ReDoS protection for user-supplied regex patterns
• Add Zod validation for strategy options at API layer

Type of Change

• New feature

Testing

Tested manually. All 53 existing chunker tests pass.

Checklist

• Code follows project style guidelines
• Self-reviewed my changes
• Tests added/updated and passing
• No new warnings introduced
• I confirm that I have read and agree to the terms outlined in the Contributor License Agreement (CLA)

[vercel]

The latest updates on your projects. Learn more about Vercel for GitHub.

1 Skipped Deployment

Project	Deployment	Actions	Updated (UTC)
docs	Skipped		Apr 11, 2026 4:33am

[cursor]

PR Summary

Medium Risk
Touches the document ingestion/chunking pipeline end-to-end (UI → API → processing), which can change chunk boundaries, token counts, and embedding behavior across all new uploads. Regex-based splitting adds some input-safety checks but still carries performance/edge-case risk if patterns or separators behave unexpectedly on large inputs.

Overview
Adds user-selectable chunking strategies for knowledge bases (token, sentence, recursive, regex, plus auto) and wires them through creation UI, /api/knowledge validation, query types, stored chunkingConfig, and processDocument strategy selection.

Introduces new chunker implementations (TokenChunker, SentenceChunker, RecursiveChunker, RegexChunker) plus shared helpers in chunkers/utils.ts (token estimation, cleaning, overlap, word-boundary splitting, metadata building), and refactors existing chunkers (notably TextChunker, JsonYamlChunker, DocsChunker, StructuredDataChunker) to use the shared utilities and adjusted detection/token-estimation behavior.

Enhances safety/validation around strategy inputs (e.g., overlap < chunk size, regex pattern required and length-limited, basic catastrophic-backtracking checks) and updates/extends chunker test coverage accordingly.

^{Reviewed by Cursor Bugbot for commit 97a0bd4. Configure here.}

[greptile-apps]

Greptile Summary

This PR adds four new chunking strategies (token, sentence, recursive, regex) by extracting shared utilities into utils.ts and wiring strategy selection through the full stack — UI → API → service → document processor. Previously flagged regressions (wrong overlap in TokenChunker, misleading label for the text strategy) have been addressed in follow-up commits.

Confidence Score: 5/5

Safe to merge; all remaining findings are P2 edge-case suggestions with no data-loss or correctness risk on typical inputs.

The two previously blocking issues (token chunker overlap, misleading label) are resolved. The two remaining findings require uncommon user inputs (regex capture groups, trailing commas in separators) and neither corrupts data for the default or normal usage paths.

apps/sim/lib/chunkers/regex-chunker.ts (capture group handling), apps/sim/app/workspace/[workspaceId]/knowledge/components/create-base-modal/create-base-modal.tsx (separator filtering)

Important Files Changed

Filename	Overview
apps/sim/lib/chunkers/utils.ts	New shared utility module; buildChunks, addOverlap, splitAtWordBoundaries, and estimateTokens are all well-implemented with correct edge-case handling.
apps/sim/lib/chunkers/token-chunker.ts	New fixed-size token chunker; previously flagged buildChunks(chunks, 0) bug has been fixed to buildChunks(chunks, this.chunkOverlap).
apps/sim/lib/chunkers/regex-chunker.ts	New regex chunker with ReDoS guards; split() with capture groups will include matched delimiter text as spurious chunk content when user patterns contain capturing groups.
apps/sim/lib/chunkers/sentence-chunker.ts	New sentence-aware chunker; lookbehind regex correctly avoids abbreviations, overlap is applied at sentence granularity, and the minSentencesPerChunk guard is intentional.
apps/sim/lib/chunkers/recursive-chunker.ts	New recursive delimiter chunker; RECIPES are well-chosen, empty-string sentinel is handled correctly, overlap and buildChunks usage is consistent with other chunkers.
apps/sim/app/workspace/[workspaceId]/knowledge/components/create-base-modal/create-base-modal.tsx	Strategy selector wired correctly; customSeparators parsing omits a .filter() for empty strings, causing bypassed separators and early fallback to word-boundary splitting.
apps/sim/app/api/knowledge/route.ts	New Zod validation for strategy and strategyOptions including regex-requires-pattern refinement; the .default('auto').optional() combination is redundant but harmless.
apps/sim/lib/knowledge/documents/document-processor.ts	Strategy selection wired end-to-end through applyStrategy; strategy is correctly read from the knowledge base config in processDocumentAsync so async job processing is unaffected.
apps/sim/lib/chunkers/types.ts	Clean type additions: ChunkingStrategy, StrategyOptions, and chunker-specific option interfaces are all well-defined and consistently used.
apps/sim/lib/knowledge/types.ts	ChunkingConfig extended with optional strategy/strategyOptions; backwards-compatible with existing knowledge bases that have no strategy stored.
apps/sim/hooks/queries/kb/knowledge.ts	CreateKnowledgeBaseParams updated to include strategy/strategyOptions in chunkingConfig; mutation invalidates the correct query key prefix.

Flowchart

%%{init: {'theme': 'neutral'}}%%
flowchart TD
    UI["CreateBaseModal\n(strategy selector)"] -->|"POST /api/knowledge"| API["route.ts\nZod validation\n(strategy + strategyOptions)"]
    API --> KB_SVC["knowledge/service.ts\nstoreChunkingConfig in DB"]
    KB_SVC --> JOB["dispatchDocumentProcessingJob"]
    JOB --> PROC["processDocumentAsync\nreads strategy from DB"]
    PROC --> DP["document-processor.ts\napplyStrategy()"]
    DP -->|"auto"| AUTO{"auto-detect\ncontent type"}
    AUTO -->|json/yaml| JY["JsonYamlChunker"]
    AUTO -->|csv/xlsx| SD["StructuredDataChunker"]
    AUTO -->|"default"| TC["TextChunker"]
    DP -->|"token"| TK["TokenChunker"]
    DP -->|"sentence"| SC["SentenceChunker"]
    DP -->|"recursive"| RC["RecursiveChunker\n(plain/markdown/code recipe)"]
    DP -->|"regex"| RX["RegexChunker\n(user pattern + ReDoS guard)"]
    TK & SC & RC & RX & TC & JY & SD --> UTILS["utils.ts\nestimateTokens · cleanText\nsplitAtWordBoundaries · buildChunks · addOverlap"]

Loading

_{Reviews (5): Last reviewed commit: "fix(chunkers): restore separator-as-join..." | Re-trigger Greptile}

[waleedlatif1]

@greptile

[waleedlatif1]

@cursor review

[waleedlatif1]

@cursor review

[waleedlatif1]

@greptile

[waleedlatif1]

@cursor review

[waleedlatif1]

@greptile

[waleedlatif1]

@cursor review

[waleedlatif1]

@greptile

[waleedlatif1]

@cursor review

[waleedlatif1]

@greptile

[waleedlatif1]

@cursor review

[cursor]

Cursor Bugbot has reviewed your changes and found 2 potential issues.

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.}

^{Reviewed by Cursor Bugbot for commit 97a0bd4. Configure here.}