feat(app): merge and source-list primitives for cross-source search - #2883
feat(app): merge and source-list primitives for cross-source search#2883teeohhem wants to merge 1 commit into
Conversation
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
|
Greptile SummaryThis PR adds reusable primitives for cross-source search before their page-level integration.
Confidence Score: 4/5The PR is not yet safe to merge because malformed or numeric Unix-second row timestamps can still produce incorrectly filtered or ordered merged results. The merge implementation continues to pass unchecked Files Needing Attention: packages/app/src/utils/multiSourceMerge.ts
|
| Filename | Overview |
|---|---|
| packages/app/src/utils/multiSourceMerge.ts | Adds frontier-based stream merging, but the previously reported timestamp validation and numeric normalization defect remains. |
| packages/app/src/utils/sourceParams.ts | Adds list resolution by composing the existing single-source resolver, deduplicating resolved IDs and applying an optional cap. |
| packages/app/src/utils/tests/multiSourceMerge.test.ts | Covers frontier, lagging-stream, direction, and merge behavior but does not cover invalid or numeric Unix-second timestamps. |
| packages/app/src/utils/tests/sourceParams.test.ts | Covers pending, partial resolution, kind filtering, deduplication, and selection limits. |
Flowchart
%%{init: {'theme': 'neutral'}}%%
flowchart LR
A[Ordered source streams] --> B[Compute coverage per active stream]
B --> C[Choose shared frontier]
C --> D[Hold rows beyond frontier]
D --> E[Tag and timestamp-sort eligible rows]
C --> F[Identify streams tied at frontier]
G[Source parameter list] --> H[Resolve IDs and names]
H --> I[Deduplicate and cap]
I --> J[Resolved and unresolved selections]
Reviews (2): Last reviewed commit: "feat(app): merge and source-list primiti..." | Re-trigger Greptile
![greptile-apps[bot]](https://avatars.githubusercontent.com/in/867647?s=60&v=4){kind=small}
| const parseTs = (row: Record<string, any>): number => { | ||
| const raw = row[timestampKey]; | ||
| let ts = tsCache.get(raw); | ||
| if (ts === undefined) { | ||
| ts = new Date(raw).getTime(); | ||
| tsCache.set(raw, ts); |
There was a problem hiding this comment.
Missing or invalid values produce NaN, which bypasses frontier comparisons and undermines sorting; numeric Unix-second values are interpreted as milliseconds and placed near 1970. Validate each parsed timestamp and normalize supported numeric values before using them for coverage, filtering, or ordering.
Two pure pieces the search page needs before it can span sources. The merge interleaves several already-ordered row streams into one timestamp-ordered list, bounded by a "safe frontier": the timestamp every stream has covered. Rows older than that are held back, so the timeline never shows a gap a slower stream could still fill, and the caller is told which streams are holding the frontier so only those need another page. Streams that error or are excluded stop bounding the frontier rather than freezing the list. The param resolver extends the existing single-source one to a list, deduping, capping, and reporting entries it could not resolve so one bad name doesn't sink the whole selection. Both are unit-tested; nothing calls them yet.
99420f8 to
99dec83
Compare
E2E Test Results✅ All tests passed • 140 passed • 1 skipped • 632s
Tests ran across 4 shards in parallel. |
1 participant
Stacked on #2882.
Two pure, fully-tested pieces the search page needs before it can span sources. ~660 lines, more than half of it tests.
The frontier merge interleaves several already-ordered row streams into one timestamp-ordered list, bounded by the timestamp every stream has covered. Rows past that are held back, so the merged timeline never shows a gap a slower stream could still fill; the caller learns which streams hold the frontier so only those fetch another page. Streams that error or are excluded stop bounding the frontier instead of freezing the list. The subtle cases (a window drained vs. a page cut short at its LIMIT, ascending vs. descending, a stream that hasn't returned yet) are what the tests pin down.
The source-list param resolver extends the existing single-source resolver to a list — dedupes, caps, and reports entries it couldn't resolve so one bad name doesn't sink the selection.
No callers yet; the search page picks them up later in the stack.