diff --git a/DEVELOPER_GUIDE.md b/DEVELOPER_GUIDE.md index 13fb0d9b7..4c8b6d6aa 100644 --- a/DEVELOPER_GUIDE.md +++ b/DEVELOPER_GUIDE.md @@ -2523,7 +2523,7 @@ catalog and unknown-field errors are generated from that same schema. | Ad-hoc issue-draft selection | `search --format issue-drafts` reads the complete filtered ad-hoc population, then applies `--first-per-file`, deterministic `--sample`, and `min(--limit, --total-limit)` in that order. Guarded searches retain their finite candidate inspection contract: `source_total_count` is omitted, `source_minimum_count` reports the observed lower bound, `source_total_count_authoritative=false`, `source_fetch_limit` reports the bounded fetch, and `truncated=true` preserves incomplete-population state. Existing `result_count`, `result_limit`, `omitted_count`, and `truncated` fields describe the returned selection accurately; additive `source_total_count`, `returned_count`, `limit_per_query`, `total_limit`, `first_per_file`, and `sample` fields make the applied contract auditable. Replay commands are serialized from normalized parsed options, use POSIX-safe single-quote escaping, and retain raw/exact/prefix modes, path/language/facet/guard filters, selection controls, evidence formatting, duplicate preflight, and issue hints. | | Recipe SARIF | `search --recipe --format sarif` emits one result per bounded recipe result. Rule IDs use `recipe/query`; standard `fingerprints.cdidx/v1` values are derived from the normalized source location; result properties preserve recipe/query identity, severity, confidence, and per-query truncation; run properties preserve scope, applied result limits, aggregate counts, and conservative omitted-result metadata. `--max-json-bytes` preflights the complete schema-valid document and exact UTF-8 byte count, including escaping and the final newline, through a counting writer before materializing only the selected prefix. If the full document does not fit, serialization omits only whole trailing results and adds run/query source, emitted, omitted, byte-strategy, and replay metadata; emitted rules and locations remain intact. This truncation exits `11` unless `--allow-partial` is set. A cap below the zero-result document minimum emits no SARIF and reports the required bytes; explicit `--json` may place a versioned error object on stdout when that object fits the cap. Replay metadata removes the byte cap when a complete report exceeds the parser's maximum supported cap. Bound SARIF with `--limit` / `--total-limit`; row selectors such as `--sample`, `--first-per-file`, and `--per-file-limit` are rejected instead of being silently ignored. Recipe severity maps `critical` / `high` to `error`, `medium` to `warning`, and `low` / `info` to `note`. | | Recipe classifier output | Recipe run JSON may add `audit_classifications` to individual `CompactSearchResult` rows when a recipe classifier can classify the hit, and query/count payloads may add `classifier_counts` when classified rows are present. These fields are additive; use them to separate triage domains such as DTO/result-wrapper `.Result` properties versus Task/ValueTask blocking waits without changing the raw search query. JSON read/write recipes also classify source-proximate `cdidx-audit: json-trust` annotations by origin, direction, sensitivity, trust, and rationale. Classification reads bounded indexed source rather than the projected snippet, including for guard-projected rows, and lexically verifies a real C# line comment so regular/verbatim/raw string contents and conditional-compilation regions cannot provide trust evidence. It evaluates every retained match site after overlap deduplication, collapses declaration-type facets—including expression-bodied method/local-function return types and generic return types split before the audited type—when the same containing statement has a later constructor facet across the same or following lines, consumes each annotation at the first lexical audited match across all selected JSON child queries, and binds it only to the next operation by parsing C# tokens. Annotation lookup searches the bounded indexed prefix rather than a fixed line gap. Nullable declarations, direct casts, nested-generic first arguments, and declaration-resolved direct receivers remain valid while an earlier statement, evaluated operand, indexer target, unresolved bare receiver, one-hop/chained property-valued assignment or invocation receiver, preprocessor directive, completed expression, control-flow block, or comma-separated operation is rejected as `not_adjacent`. Rows with distinct evidence remain conservatively `mixed_boundaries`; missing, invalid, direction-mismatched, or `review_required` evidence remains `ambiguous_trust`. The classifier groups rows by file and reconstructs the maximum required bounded prefix once; its per-query lexical cache retains only that one file prefix, records the prefix actually reconstructed, and remembers source-budget exhaustion so high-line matches neither repeat reconstruction nor poison a lower-line result. For `json-parse-apis`, every retained structured or compact row receives exactly one `parser_guard_evidence` classification from lexically masked containing-symbol context. A byte/depth/item/file-size bound must relate to the consumed payload and takes precedence over streaming/cancellation for the same operation; otherwise a streaming/cancellation signal or the non-authoritative `unbounded_materialization` fallback is emitted. If one row represents multiple operations, any unbounded operation keeps the row unbounded. Compact rows preserve the same classification evidence used for `classifier_counts`. Classification never removes or reorders the raw result. Source-backed classification runs only for JSON/NDJSON/compact/count-JSON shapes that serialize row classifications or classifier counts; text, scalar count, compact summary, SARIF, issue-draft, and `--search-fields` projection paths skip it. | -| NDJSON terminal records | Default NDJSON for `search`, `symbols`, and `files` appends one final `terminal_record` after result rows; search also emits it for zero-result responses, while raw `symbols` and `files` keep zero-result NDJSON empty. Recipe/audit search row streams share the same writer. Terminals report returned and observed total counts, `total_count_authoritative` / `total_count_lower_bound`, selection or interruption reason, applied limits, omitted rows, and recovery guidance. `--max-json-bytes` covers the complete stdout stream, including newlines and this terminal record; when additive selector-accounting fields prevent the terminal from fitting, the writer omits those optional fields before declaring the terminal impossible. A cap that still cannot fit the terminal fails before stdout. Capped output rejects `--profile`, `--verbose`, and `--json-envelope`. Byte-cap partial output exits with `CommandExitCodes.PartialResult` (`11`) unless `--allow-partial` explicitly opts into exit `0`. `--results-only` is the explicit terminal-record opt-out for these NDJSON row streams and is rejected with array, compact, summary, or count output. | +| NDJSON terminal records | Default NDJSON for `search`, `symbols`, and `files` appends one final `terminal_record` after result rows; search also emits it for zero-result responses, while raw `symbols` and `files` keep zero-result NDJSON empty. Recipe/audit search row streams share the same writer. Terminals report returned and observed total counts, `total_count_authoritative` / `total_count_lower_bound`, selection or interruption reason, applied limits, omitted rows, and recovery guidance. A raw cursor-capable stream with `has_more: true` and at least one emitted result adds a shared generation- and query-bound `response:v2` `next_cursor` whose offset advances by the actual emitted count. Replaying it with unchanged filters and ordering selects the bounded envelope; its `metadata.stream_terminal` preserves the same continuation. Final and zero-result pages omit continuation fields. A partial terminal that cannot safely advance instead omits the cursor and adds machine-readable `next_cursor_unavailable_reason`: `no_result_row_emitted` covers a terminal-only byte-capped response; `stream_not_cursor_capable` covers recipe/named or row-selector streams; `pagination_window_exhausted` covers the 10,000-row response window; `index_generation_changed_during_query` fails closed when the generation captured before row materialization differs from the generation used to encode the cursor; and `index_generation_unavailable` covers failure to obtain either comparison snapshot. `--max-json-bytes` covers the complete stdout stream, including newlines, cursor, and terminal record; each byte-fit candidate regenerates the cursor from its emitted count, and optional selector accounting is dropped before declaring the terminal impossible. A cap that still cannot fit the terminal fails before stdout. Capped output rejects `--profile`, `--verbose`, and `--json-envelope`. Byte-cap partial output exits with `CommandExitCodes.PartialResult` (`11`) unless `--allow-partial` explicitly opts into exit `0`. `--results-only` is the explicit terminal-record opt-out for these NDJSON row streams and is rejected with array, compact, summary, or count output. | | C# outline callable display | `DbSymbolReader.Outline` derives `display_name` only at read time and never changes canonical `symbols.name`, qualified paths, folded identity, or exact-query aliases. A complete C# generic method signature normally uses arity placeholders (`` or ``); if one would collide with a concrete parameter type, it deterministically selects collision-free `TArg` placeholders. Replacement applies only to unqualified method-type-parameter references, preserving qualified concrete types and escaped-keyword distinctions. Literal-aware scanning keeps delimiters in attributes and default values from changing parameter boundaries. The display omits `where` constraints, drops non-identity `this` / `params` / `scoped`, and retains overload-significant `ref` / `out` / `in` (including `ref readonly`). Non-generic and non-C# formatting stays on the existing path. Missing, truncated, or syntactically incomplete persisted signatures retain the legacy `Name@line` fallback for old-index compatibility. | | `outline` / `unused` cursor binding | `outline --json` accepts `--kind `, `--limit` / `--top`, opaque `--cursor `, and `--outline-fields ` for bounded machine output. Controlled outline responses keep the normal envelope and add `total_symbol_count`, `returned_symbol_count`, `cursor_offset`, `next_cursor`, `has_more`, and `result_stable_at`, plus `kind_filter` and `selected_fields` when active. Projection parsing canonicalizes aliases and removes duplicates before validation; unknown field names are reported together as one terminal usage error with valid candidates, while the empty-selection error is reserved for deliberately empty CSV input. `outline` and `unused` cursors bind their offset to the normalized path/scope, filters, ordering, and index generation; reuse after changing those inputs or refreshing the index fails with explicit restart-required guidance. Legacy `outline:` / `unused:` inputs remain accepted for transition, but every newly emitted cursor is opaque and bound. | | `hotspots --json` grouping semantics | `hotspots` and MCP `symbol_hotspots` emit `grouped_by`, `grouping_unit`, `count_kind`, `limit_applies_to`, `score_fields`, `ranking_fields`, and matching `query_context` fields. `--limit` applies to returned symbols, files, name/kind groups, or SQL statements; `--count` ignores `--limit` and reports total groups. Explicit `statement` grouping is SQL-only (`--lang sql` / `lang: "sql"`). | @@ -6763,7 +6763,7 @@ total / cursor / truncation、partial-family metadata、definition body の pagi | ad-hoc issue-draft selection | `search --format issue-drafts` は filter 済みの ad-hoc 母集団全体を読み、`--first-per-file`、決定的な `--sample`、`min(--limit, --total-limit)` の順に適用します。guard 付き検索は有限の candidate inspection 契約を維持し、`source_total_count` を省略し、観測下限を `source_minimum_count`、非 authoritative 状態を `source_total_count_authoritative=false`、bounded fetch を `source_fetch_limit` で報告し、母集団が未完了であることを `truncated=true` で保持します。既存の `result_count`、`result_limit`、`omitted_count`、`truncated` field は返却 selection を正確に表し、additive な `source_total_count`、`returned_count`、`limit_per_query`、`total_limit`、`first_per_file`、`sample` field により適用済み契約を監査できます。replay command は正規化済み parse option から serialize し、POSIX-safe な単一引用符 escape を使い、raw / exact / prefix mode、path / language / facet / guard filter、selection control、evidence formatting、duplicate preflight、issue hint を維持します。 | | Recipe SARIF | `search --recipe --format sarif` は、上限付き recipe result ごとに result を1件出力します。rule ID は `recipe/query` を使い、標準の `fingerprints.cdidx/v1` は正規化済み source location から導出します。result properties は recipe/query identity、severity、confidence、query ごとの truncation を保持し、run properties は scope、適用済み result limit、集計 count、保守的な omitted-result metadata を保持します。`--max-json-bytes` は escape と末尾改行を含む schema-valid な完全 document と正確な UTF-8 byte 数を counting writer で検査し、選択した prefix だけを materialize します。完全な document が収まらない場合、末尾の result だけを1件単位で省略し、run / query に source、emitted、omitted、byte strategy、replay metadata を追加します。出力済み result の rule と location は維持し、この truncation は `--allow-partial` がなければ `11` を返します。result 0件の document の最小値より小さい cap では SARIF を出力せず必要 byte 数を報告し、明示的な `--json` では error object 自体が cap に収まる場合に version 付き error を stdout へ出力することがあります。完全な report が parser の対応可能な最大 cap を超える場合、replay metadata は byte cap を外します。SARIF の上限には `--limit` / `--total-limit` も使い、`--sample`、`--first-per-file`、`--per-file-limit` のような row selector は黙って無視せず拒否します。recipe severity は `critical` / `high` を `error`、`medium` を `warning`、`low` / `info` を `note` に対応付けます。 | | Recipe classifier output | recipe classifier が hit を分類できる場合、recipe run JSON は個別の `CompactSearchResult` row に `audit_classifications` を追加することがあり、分類済み row がある query / count payload は `classifier_counts` を追加することがあります。これらは additive field です。raw search query を変えずに、DTO / result-wrapper の `.Result` property と Task / ValueTask の blocking wait などの triage domain を分離するために使います。JSON の read / write recipe は source-proximate な `cdidx-audit: json-trust` 注釈も origin、direction、sensitivity、trust、rationale で分類します。分類は guard により1行へ投影された row を含め、投影済み snippet ではなく上限付きの indexed source を読み、実際の C# line comment であることを lexical に検証するため regular / verbatim / raw string の内容や条件コンパイル領域を trust evidence にしません。overlap dedup 後に残った各 match site を評価し、expression-bodied method / local function の戻り型、および audit 対象型より前で改行された generic 戻り型を含め、同じ containing statement の同一行または後続行に constructor facet がある宣言型 facet は畳み込み、選択されたすべての JSON child query を横断した最初の lexical な audit 対象 match で各注釈を消費します。注釈探索は固定の行差ではなく上限付き indexed prefix を検索します。C# token 解析では nullable 宣言、直接 cast、nested-generic の first argument、宣言から解決できる直接 receiver は有効なまま維持する一方、途中の実行コード、先行 statement、評価済み operand、indexer 代入先、解決不能な単純名 receiver、1段 / 連鎖した property-valued な代入 / 呼び出し receiver、preprocessor directive、完了済み expression、control-flow block、カンマ区切りの操作がある注釈は `not_adjacent` として次の操作へ流用しません。異なる evidence を持つ row は保守的に `mixed_boundaries` とし、注釈の欠落、不正、direction 不一致、`review_required` を `ambiguous_trust` とします。分類は row を file ごとにまとめて最大必要行までの上限付き prefix を1回復元します。query ごとの lexical cache はその file prefix 1件だけを保持し、実際に復元できた行数と source budget の枯渇を記録するため、高い行で再構築を繰り返さず、低い行の結果も汚染しません。`json-parse-apis` では、保持された structured / compact row ごとに、lexical mask 済みの containing-symbol context から `parser_guard_evidence` を必ず1つ付与します。byte / depth / item / file-size bound は消費される payload と関係する必要があり、同じ operation では streaming / cancellation より優先されます。それ以外では streaming / cancellation signal または非 authoritative な `unbounded_materialization` fallback を出力します。1つの row が複数 operation を表す場合は、未bounded operation が1つでもあれば row を unbounded のままにします。compact row は `classifier_counts` と同じ classification evidence を保持します。分類は raw result を削除も並べ替えもしません。source-backed 分類は row classification または classifier count を serialize する JSON / NDJSON / compact / count JSON shape だけで実行し、text、scalar count、compact summary、SARIF、issue-draft、`--search-fields` projection path では省略します。 | -| NDJSON terminal record | `search`、`symbols`、`files` の既定 NDJSON は result row の後に最後の `terminal_record` を 1 件追加します。`search` は 0 件応答にも終端を出力しますが、raw `symbols` / `files` の 0 件 NDJSON は空のままです。recipe / audit search の row stream も同じ writer を使います。終端は返却件数と観測済み総件数、`total_count_authoritative` / `total_count_lower_bound`、selection または中断理由、適用上限、省略行数、復旧案内を報告します。`--max-json-bytes` は改行と終端レコードを含む stdout stream 全体を対象にし、追加 selector-accounting field が原因で終端が収まらない場合は、終端自体を不可能と判定する前にそれらの任意 field を省略します。それでも終端が収まらない cap は stdout 出力前に失敗します。上限付き出力は `--profile`、`--verbose`、`--json-envelope` を拒否します。byte cap による部分出力は、`--allow-partial` で終了コード `0` を明示許可しない限り `CommandExitCodes.PartialResult`（`11`）を返します。`--results-only` はこれらの NDJSON row stream から終端レコードを明示的に除外するための option であり、array / compact / summary / count 出力との組み合わせは拒否されます。 | +| NDJSON terminal record | `search`、`symbols`、`files` の既定 NDJSON は result row の後に最後の `terminal_record` を 1 件追加します。`search` は 0 件応答にも終端を出力しますが、raw `symbols` / `files` の 0 件 NDJSON は空のままです。recipe / audit search の row stream も同じ writer を使います。終端は返却件数と観測済み総件数、`total_count_authoritative` / `total_count_lower_bound`、selection または中断理由、適用上限、省略行数、復旧案内を報告します。raw の cursor 対応 stream で `has_more: true` かつ result を 1 件以上出力した場合、実際の出力件数だけ offset を進めた、generation と query に束縛済みの共有 `response:v2` `next_cursor` を追加します。filter と ordering を変えず再利用すると bounded envelope を選択し、`metadata.stream_terminal` にも同じ continuation を保持します。最終 page と 0 件 page は continuation field を省略します。安全に進めない partial terminal は cursor を省略し、機械可読な `next_cursor_unavailable_reason` を追加します。terminal だけの byte-cap 応答は `no_result_row_emitted`、recipe / named / row-selector stream は `stream_not_cursor_capable`、10,000 row の response window 枯渇は `pagination_window_exhausted`、row materialize 前に取得した generation と cursor encode 時の generation が異なる場合は fail-closed な `index_generation_changed_during_query`、比較用snapshotの取得失敗は `index_generation_unavailable` を使います。`--max-json-bytes` は改行、cursor、終端レコードを含む stdout stream 全体を対象とし、各 byte-fit candidate の実出力件数から cursor を再生成します。追加 selector-accounting field が原因で終端が収まらない場合は、終端自体を不可能と判定する前にそれらの任意 field を省略します。それでも終端が収まらない cap は stdout 出力前に失敗します。上限付き出力は `--profile`、`--verbose`、`--json-envelope` を拒否します。byte cap による部分出力は、`--allow-partial` で終了コード `0` を明示許可しない限り `CommandExitCodes.PartialResult`（`11`）を返します。`--results-only` はこれらの NDJSON row stream から終端レコードを明示的に除外するための option であり、array / compact / summary / count 出力との組み合わせは拒否されます。 | | C# outline callable 表示 | `DbSymbolReader.Outline` は read 時にだけ `display_name` を導出し、canonical な `symbols.name`、qualified path、folded identity、完全一致 query alias は変更しません。完全な C# generic method signature は通常 arity placeholder（`` または ``）で表示し、具体的な parameter type と衝突する場合は決定的で衝突しない `TArg` placeholder を選びます。置換対象は修飾されていない method type parameter の参照だけであり、修飾された具体型と escaped keyword の区別は保持します。literal-aware な走査により、attribute と既定値内の区切り文字は parameter 境界を変えません。`where` constraint と identity に影響しない `this` / `params` / `scoped` は省略し、overload を区別する `ref` / `out` / `in`（`ref readonly` を含む）は保持します。非 generic および C# 以外の formatting は既存経路のままです。永続 signature が欠落、切り詰め、または構文的に不完全な場合は、旧 index 互換のため legacy `Name@line` fallback を維持します。 | | `outline` / `unused` cursor の束縛 | `outline --json` は bounded な機械向け出力として `--kind `、`--limit` / `--top`、opaque な `--cursor `、`--outline-fields ` を受け付けます。制御付き outline 応答は通常の envelope を維持し、`total_symbol_count`、`returned_symbol_count`、`cursor_offset`、`next_cursor`、`has_more`、`result_stable_at` を追加し、該当時は `kind_filter` と `selected_fields` も返します。projection parser は検証前に alias を canonicalize して重複を除きます。未知field名はvalid候補を伴う1つの終端usage errorにまとめ、empty-selection errorは意図的に空のCSV入力にだけ使います。`outline` と `unused` の cursor は offset を正規化済み path/scope、filter、ordering、index generation に束縛するため、条件変更後または index 更新後の再利用は restart-required の明示案内付きで失敗します。移行用に legacy の `outline:` / `unused:` 入力は受理しますが、新しく出力する cursor はすべて opaque かつ束縛済みです。 | | `hotspots --json` grouping semantics | `hotspots` と MCP `symbol_hotspots` は `grouped_by`、`grouping_unit`、`count_kind`、`limit_applies_to`、`score_fields`、`ranking_fields` と、対応する `query_context` field を返します。`--limit` は返却される symbol、file、name/kind group、SQL statement に適用されます。`--count` は `--limit` を無視し、total group 数を返します。明示的な `statement` grouping は SQL 専用です（`--lang sql` / `lang: "sql"`）。 | diff --git a/TESTING_GUIDE.md b/TESTING_GUIDE.md index f105da7ee..2853bbc56 100644 --- a/TESTING_GUIDE.md +++ b/TESTING_GUIDE.md @@ -672,6 +672,8 @@ Candidate-ordered parallel-index recovery tests must prove that the fatal result uses one two-file recipe fixture to prove a compact search can emit a negative-score `next_cursor`, replay that exact cursor as a separated `--cursor` value, and return the next distinct page without an option-parsing error. - `QueryCommandRunnerTests.RunSearch_CursorDoesNotConsumeRecognizedShortOption_Issue4664` keeps the negative-cursor exception narrow by proving a recognized short option after `--cursor` remains an option and still produces the missing-cursor-value diagnostic. +- `QueryCommandRunnerIssue5230Tests` + walks cursor-capable `search`, `symbols`, and `files` NDJSON across multiple pages without gaps or duplicates; verifies final and zero-result terminals omit continuations; rejects filter-mismatched and stale-generation cursors; keeps byte-capped output within budget while advancing both the envelope and nested terminal continuation by the actual emitted count (using `no_result_row_emitted` instead of an unsafe zero-progress cursor if no row fits); and fails closed for row selectors, a generation change during materialization, and page sizes whose next replay would cross the 10,000-row pagination boundary. - `QueryCommandRunnerTests.RunSearch_UnknownRecipeQuerySuggestsOnlyFromActiveRecipe_Issue4862`, `RunSearch_RecipeQuerySuggestionHandlesZeroOneAndManyQueries_Issue4862`, and `RunSearch_ExternalRecipeAliasesResolveAndSuggestCanonicalActiveQueries_Issue4862` keep human/JSON usage errors, canonical names, current/deprecated aliases, deterministic typo correction, active-recipe isolation, normalized filter preservation, and shell-safe replay quoting in one recipe-selection contract. Suggestions and replay selectors must remain canonical names from the active recipe even when an alias is the nearest match. Include/exclude typo recovery must preserve selector semantics, unresolved repeated selectors, and raw FTS mode. Missing close matches and recipe-name corrections with child selectors must not offer a replay that broadens the selection, while aliases that collide with canonical names or multiple query owners must be removed with bounded diagnostics. @@ -1829,6 +1831,8 @@ dotnet test --filter "FullyQualifiedName~GitHelperTests" は 2 file の recipe fixture を 1 つ使い、compact search が負の score で始まる `next_cursor` を生成し、その値を separated `--cursor` として変更せず再利用しても option parse error にならず、別の次ページを返すことを検証します。 - `QueryCommandRunnerTests.RunSearch_CursorDoesNotConsumeRecognizedShortOption_Issue4664` は負の cursor に対する例外を狭く保ち、`--cursor` の直後にある既知の short option は cursor 値として消費されず、従来どおり cursor 値欠如の診断を返すことを検証します。 +- `QueryCommandRunnerIssue5230Tests` + は cursor 対応の `search`、`symbols`、`files` NDJSON を複数 page にわたって gap / duplicate なしで走査し、最終 terminal と 0 件 terminal が continuation を省略すること、filter mismatch と stale generation の cursor を拒否すること、byte-cap 出力が budget 内に収まり envelope と内包 terminal の continuation がともに実際の出力件数だけ進むこと（row が 1 件も収まらない場合は unsafe な zero-progress cursor の代わりに `no_result_row_emitted` を返すこと）、row selector、materialize 中の generation 変更、次回の同一 page size で 10,000 row の pagination 境界を越える場合に fail closed することを検証します。 - `QueryCommandRunnerTests.RunSearch_UnknownRecipeQuerySuggestsOnlyFromActiveRecipe_Issue4862`、`RunSearch_RecipeQuerySuggestionHandlesZeroOneAndManyQueries_Issue4862`、`RunSearch_ExternalRecipeAliasesResolveAndSuggestCanonicalActiveQueries_Issue4862` は human / JSON usage error、canonical 名、現行 / deprecated alias、決定的な typo correction、active recipe への限定、正規化済み filter の保持、shell-safe な replay quoting を1つの recipe-selection 契約として維持します。alias が最も近い候補でも、suggestion と replay selector は active recipe の canonical 名でなければなりません。 include / exclude の typo recovery は selector の意味、未解決の繰り返し selector、raw FTS mode を保持します。近い候補がない場合、および child selector を伴う recipe 名訂正では selection を広げる replay を提示せず、canonical 名または複数の query owner と衝突する alias は上限付き diagnostic とともに除去しなければなりません。 diff --git a/USER_GUIDE.md b/USER_GUIDE.md index c00a6cdba..0c540fbbe 100644 --- a/USER_GUIDE.md +++ b/USER_GUIDE.md @@ -401,6 +401,20 @@ array. `languages --json` accepts `--limit` / `--top`, `--cursor`, and the ordinary unbounded JSON shape. Pass each `next_cursor` back to the same command and filters. A cursor is bound to that selection and index generation, so changed inputs or a refreshed index require restarting the pagination. +The default raw NDJSON streams for `search`, `symbols`, and `files` use the same +opaque `response:v2` continuation contract. When a terminal record has +`has_more: true` and at least one result row was emitted, its `next_cursor` +resumes after the last emitted row; replay it with the unchanged command, +query, filters, ordering, and page limit. The replay is returned in the shared +bounded envelope, whose `metadata.stream_terminal` mirrors the continuation. +Final and zero-result pages do not advertise a cursor. If a partial stream +cannot make safe progress—for example, a byte cap leaves room only for the +terminal record, the query uses row selectors or a recipe/named search, the +10,000-row pagination window is exhausted or the unchanged page limit would +cross it on replay, or the index generation changes while rows are being +read—the terminal omits the cursor and reports +`next_cursor_unavailable_reason`. Cursor bytes are included in the complete +`--max-json-bytes` measurement. When a bounded `find --all` scan exits partially, its terminal record includes `next_cursor`; replaying it resumes after the last scanned line. The bounded-response commands `search`, `definition`, `find`, `status`, @@ -4159,7 +4173,20 @@ row を省略した場合は、`omitted_match_count`、`truncated`、`has_more` 受け付け、これらを指定した場合だけ bounded envelope を選択するため、通常の上限なし JSON 形状は変わりません。`next_cursor` は同じ command と filter に渡してください。 cursor はその選択条件と index generation に束縛されるため、入力変更後または index -更新後は pagination を最初からやり直す必要があります。上限に達した +更新後は pagination を最初からやり直す必要があります。 +`search`、`symbols`、`files` の既定 raw NDJSON stream も同じ opaque な +`response:v2` continuation 契約を使います。terminal record が `has_more: true` で、 +result row を 1 件以上出力した場合、`next_cursor` は最後に出力した row の次から再開します。 +command、query、filter、ordering、page limit を変えずに再利用してください。再開応答は +共有 bounded envelope となり、`metadata.stream_terminal` にも同じ continuation が +反映されます。最終 page と 0 件 page は cursor を公開しません。byte cap により terminal +record しか出力できない場合、row selector または recipe / named search を使う場合、 +10,000 row の pagination window を使い切ったか、同じ page limit での再開時にその上限を +越える場合、row の読み取り中に index generation が変わった場合のように、安全に再開できない +partial stream では cursor を省略し、terminal の +`next_cursor_unavailable_reason` で理由を報告します。cursor の byte 数も +`--max-json-bytes` による stream 全体の計測に含まれます。 +上限に達した `find --all` scan が partial exit した場合、terminal record の `next_cursor` を 再利用すると最後に scan した line の次から継続します。 bounded-response command の `search`、`definition`、`find`、`status`、 diff --git a/changelog.d/unreleased/5230.fixed.md b/changelog.d/unreleased/5230.fixed.md new file mode 100644 index 000000000..b5fc41940 --- /dev/null +++ b/changelog.d/unreleased/5230.fixed.md @@ -0,0 +1,25 @@ +--- +category: fixed +issues: + - 5230 +affected: + - src/CodeIndex/Cli/QueryCommandRunner.cs + - src/CodeIndex/Cli/QueryCommandRunner.ArgumentParser.cs + - src/CodeIndex/Cli/QueryCommandRunner.Database.cs + - src/CodeIndex/Cli/JsonOutputContracts.cs + - src/CodeIndex/Cli/JsonEnvelopeWrapper.Bounded.cs + - src/CodeIndex/Cli/QueryCommandRunner.Ndjson.cs + - src/CodeIndex/Cli/QueryCommandRunner.SearchResults.cs + - tests/CodeIndex.Tests/QueryCommandRunnerIssue5230Tests.cs + - USER_GUIDE.md + - DEVELOPER_GUIDE.md + - TESTING_GUIDE.md +--- + +## English + +- **Truncated discovery NDJSON now exposes resumable cursors (#5230)** — Raw `search`, `symbols`, and `files` terminal records include a stable, generation-bound `next_cursor` after the last emitted row by reusing the response-v2 contract from #4730. Continuation rejects changed filters and stale indexes, advances both byte-trimmed envelope cursor copies by the actual emitted row count, omits cursors on final and zero-result pages, and reports a machine-readable reason when selectors, the next unchanged page would exceed the pagination window, or a concurrent index-generation change makes safe progress impossible. + +## 日本語 + +- **切り詰められた discovery NDJSON が再開用 cursor を公開するようになりました (#5230)** — raw `search`、`symbols`、`files` の terminal record は、#4730 の response-v2 契約を再利用し、最後に出力した row の後から再開できる安定した generation-bound `next_cursor` を含みます。continuation は filter 変更と stale index を拒否し、byte trimming 後の envelope 内にある両方の cursor を実際に出力した row 数だけ進めます。最終 page と 0 件 page では cursor を省略し、selector、次回の同一 page が pagination window を越える場合、同時発生した index-generation 変更により安全に進められない場合は機械可読な理由を報告します。 diff --git a/src/CodeIndex/Cli/JsonEnvelopeWrapper.Bounded.cs b/src/CodeIndex/Cli/JsonEnvelopeWrapper.Bounded.cs index 920342b31..5644a074e 100644 --- a/src/CodeIndex/Cli/JsonEnvelopeWrapper.Bounded.cs +++ b/src/CodeIndex/Cli/JsonEnvelopeWrapper.Bounded.cs @@ -668,7 +668,7 @@ JsonObject BuildCandidate( metadata["cursor_offset"] = controls.Offset; metadata["page_limit"] = controls.PageLimit; metadata["has_more"] = hasMore; - metadata["next_cursor"] = selectedScanCursor + var nextCursor = selectedScanCursor ?? (hasMore && count > 0 ? FormatResponseCursor( nextOffset, @@ -680,6 +680,15 @@ JsonObject BuildCandidate( controls.ResumeMatchOrdinal, controls.ResumeByteOffset) : null); + metadata["next_cursor"] = nextCursor; + if (scanCursor is not null + && metadata["stream_terminal"] is JsonObject adjustedTerminal) + { + if (nextCursor is null) + adjustedTerminal.Remove("next_cursor"); + else + adjustedTerminal["next_cursor"] = nextCursor; + } metadata["truncated"] = scanCursor is not null || totalCount > count || byteLimitOmittedPathCount > 0; metadata["pagination_window_limit"] = MaxPageWindow; metadata["pagination_window_exhausted"] = paginationWindowExhausted; @@ -2620,6 +2629,67 @@ private static ResponseSnapshot BuildResponseSnapshot(DbReader reader) reader.GetIndexedHeadForResponse()); } + internal sealed record NdjsonResponseCursorContext( + Func? CursorFactory, + string? UnavailableReason); + + internal static string CaptureResponseGenerationFingerprint(DbReader reader) + => BuildResponseSnapshot(reader).GenerationFingerprint; + + private static string? TryCaptureResponseGenerationFingerprint(string dbPath) + { + try + { + using var db = new DbContext(DbOpenIntent.QueryOnly, dbPath); + if (!db.TryValidateIsCodeIndexDb(out _)) + return null; + return CaptureResponseGenerationFingerprint(new DbReader(db)); + } + catch + { + return null; + } + } + + internal static NdjsonResponseCursorContext BuildNdjsonResponseCursorFactory( + string command, + string[] args, + string dbPath, + string? expectedGenerationFingerprint) + { + if (expectedGenerationFingerprint is null) + return new(null, "index_generation_unavailable"); + var currentGenerationFingerprint = TryCaptureResponseGenerationFingerprint(dbPath); + if (currentGenerationFingerprint is null) + return new(null, "index_generation_unavailable"); + if (!string.Equals( + expectedGenerationFingerprint, + currentGenerationFingerprint, + StringComparison.Ordinal)) + { + return new(null, "index_generation_changed_during_query"); + } + + var responseOffset = GetBoundedResponseOffset(command); + var queryFingerprint = BuildResponseFingerprint(command, args); + return new( + returnedCount => FormatResponseCursor( + checked(responseOffset + returnedCount), + queryFingerprint, + expectedGenerationFingerprint), + null); + } + + internal static bool IsNdjsonResponseCursorWithinWindow( + string command, + int returnedCount, + int replayPageLimit) + { + var nextOffset = checked(GetBoundedResponseOffset(command) + returnedCount); + return nextOffset < MaxPageWindow + && checked(nextOffset + replayPageLimit) <= MaxPageWindow; + } + private static ResponseSnapshot BuildFallbackResponseSnapshot(string appVersion) => new(BuildResponseValueFingerprint("catalog\0" + appVersion), null, null); diff --git a/src/CodeIndex/Cli/JsonOutputContracts.cs b/src/CodeIndex/Cli/JsonOutputContracts.cs index dde3bbcec..541337200 100644 --- a/src/CodeIndex/Cli/JsonOutputContracts.cs +++ b/src/CodeIndex/Cli/JsonOutputContracts.cs @@ -829,6 +829,10 @@ internal sealed record JsonStreamDoneResult( [property: JsonPropertyName("truncated")] bool Truncated, [property: JsonPropertyName("has_more")] bool HasMore, [property: JsonPropertyName("total_count_authoritative")] bool TotalCountAuthoritative, + [property: JsonPropertyName("next_cursor")] + [property: JsonIgnore(Condition = JsonIgnoreCondition.WhenWritingNull)] string? NextCursor = null, + [property: JsonPropertyName("next_cursor_unavailable_reason")] + [property: JsonIgnore(Condition = JsonIgnoreCondition.WhenWritingNull)] string? NextCursorUnavailableReason = null, [property: JsonPropertyName("total_count_lower_bound")] [property: JsonIgnore(Condition = JsonIgnoreCondition.WhenWritingNull)] int? TotalCountLowerBound = null, [property: JsonPropertyName("selection_reason")] diff --git a/src/CodeIndex/Cli/QueryCommandRunner.ArgumentParser.cs b/src/CodeIndex/Cli/QueryCommandRunner.ArgumentParser.cs index 14963911f..e71675512 100644 --- a/src/CodeIndex/Cli/QueryCommandRunner.ArgumentParser.cs +++ b/src/CodeIndex/Cli/QueryCommandRunner.ArgumentParser.cs @@ -250,7 +250,7 @@ internal QueryCommandOptions Parse(string[] args) resolvedDbPath = DbContext.ToReadOnlyUri(resolvedDbPath); } - return BuildOptions(dbResolution, resolvedDbPath); + return BuildOptions(dbResolution, resolvedDbPath, args); } private void NormalizeOutputMode() @@ -407,10 +407,14 @@ private void ValidateEnvironmentDefaults() AddParseError(defaultMaxLineWidthError); } - private QueryCommandOptions BuildOptions(DbPathResolution dbResolution, string resolvedDbPath) + private QueryCommandOptions BuildOptions( + DbPathResolution dbResolution, + string resolvedDbPath, + string[] invocationArgs) { return new QueryCommandOptions { + InvocationArgs = [.. invocationArgs], DbPath = resolvedDbPath, DbPathExplicit = dbPathExplicit, ReadOnly = readOnly, diff --git a/src/CodeIndex/Cli/QueryCommandRunner.Database.cs b/src/CodeIndex/Cli/QueryCommandRunner.Database.cs index 42fdd2397..ebc6b7427 100644 --- a/src/CodeIndex/Cli/QueryCommandRunner.Database.cs +++ b/src/CodeIndex/Cli/QueryCommandRunner.Database.cs @@ -90,6 +90,11 @@ private static int WithDb( } reader.IncludeGenerated = options.IncludeGenerated; + options.InvocationGenerationFingerprint = options.Json + && options.JsonOutputFormat == JsonOutputFormatNdjson + && !options.ResultsOnly + ? JsonEnvelopeWrapper.CaptureResponseGenerationFingerprint(reader) + : null; var previousProjectRoot = s_activeQueryProjectRoot; var projectRootResolution = ResolveProjectFilterRoot(dbPath, options.DbPathExplicit); s_activeQueryProjectRoot = projectRootResolution.Root; diff --git a/src/CodeIndex/Cli/QueryCommandRunner.Ndjson.cs b/src/CodeIndex/Cli/QueryCommandRunner.Ndjson.cs index ae36279e6..847882bf1 100644 --- a/src/CodeIndex/Cli/QueryCommandRunner.Ndjson.cs +++ b/src/CodeIndex/Cli/QueryCommandRunner.Ndjson.cs @@ -7,6 +7,7 @@ namespace CodeIndex.Cli; public static partial class QueryCommandRunner { private const int NdjsonResponseBudgetRetryHeadroomBytes = 1024; + internal static Action? NdjsonRowsMaterializedForTesting { get; set; } private sealed record NdjsonOutputRecord(string Line, bool CountsAsResult = true); @@ -57,6 +58,15 @@ private static NdjsonStreamWriteResult WriteNdjsonStream( var emittedRecords = records.Count; string? terminalLine = null; + NdjsonRowsMaterializedForTesting?.Invoke(); + var continuationCursorFactory = reader is not null && IsCursorCapableNdjson(commandName, options) + ? new Lazy(() => + JsonEnvelopeWrapper.BuildNdjsonResponseCursorFactory( + commandName, + options.InvocationArgs, + options.DbPath, + options.InvocationGenerationFingerprint)) + : null; string BuildTerminal( int returnedCount, @@ -67,7 +77,21 @@ string BuildTerminal( int omittedRecordCount, string? recoveryGuidance, bool includeSelectionAccounting) - => BuildJsonStreamDoneLine( + { + var hasMore = truncated || interrupted; + if (hasMore && totalCountAuthoritative) + { + var nextOffset = checked( + JsonEnvelopeWrapper.GetBoundedResponseOffset(commandName) + returnedCount); + hasMore = nextOffset < totalCount; + } + var (nextCursor, unavailableReason) = BuildNdjsonContinuation( + commandName, + returnedCount, + options.Limit, + hasMore, + continuationCursorFactory); + return BuildJsonStreamDoneLine( returnedCount, totalCount, jsonOptions, @@ -89,7 +113,11 @@ string BuildTerminal( selectedTotal: includeSelectionAccounting ? selectedTotal : null, selectorOmittedCount: includeSelectionAccounting ? selectorOmittedCount : null, limitOmittedCount: includeSelectionAccounting ? limitOmittedCount : null, - selectors: includeSelectionAccounting ? selectors : null); + selectors: includeSelectionAccounting ? selectors : null, + nextCursor: nextCursor, + nextCursorUnavailableReason: unavailableReason, + hasMore: hasMore); + } if (options.MaxJsonBytes.HasValue) { @@ -198,6 +226,40 @@ string BuildTerminal( exitCode); } + private static (string? Cursor, string? UnavailableReason) BuildNdjsonContinuation( + string commandName, + int returnedCount, + int replayPageLimit, + bool hasMore, + Lazy? cursorFactory) + { + if (!hasMore) + return (null, null); + if (returnedCount <= 0) + return (null, "no_result_row_emitted"); + if (cursorFactory is null) + return (null, "stream_not_cursor_capable"); + if (!JsonEnvelopeWrapper.IsNdjsonResponseCursorWithinWindow( + commandName, + returnedCount, + replayPageLimit)) + return (null, "pagination_window_exhausted"); + + var cursorContext = cursorFactory.Value; + if (cursorContext.CursorFactory is null) + return (null, cursorContext.UnavailableReason ?? "index_generation_unavailable"); + + return (cursorContext.CursorFactory(returnedCount), null); + } + + private static bool IsCursorCapableNdjson(string commandName, QueryCommandOptions options) + => commandName is "symbols" or "files" + || commandName == "search" + && options.RecipeName is null + && options.NamedSearchQueries.Count == 0 + && !options.ListRecipes + && !HasSearchRowSelectors(options); + private static NdjsonStreamWriteResult WriteResultOnlyNdjson( IReadOnlyList records, QueryCommandOptions options, diff --git a/src/CodeIndex/Cli/QueryCommandRunner.SearchResults.cs b/src/CodeIndex/Cli/QueryCommandRunner.SearchResults.cs index b06023891..9080a476d 100644 --- a/src/CodeIndex/Cli/QueryCommandRunner.SearchResults.cs +++ b/src/CodeIndex/Cli/QueryCommandRunner.SearchResults.cs @@ -1751,7 +1751,10 @@ private static string BuildJsonStreamDoneLine( int? selectedTotal = null, int? selectorOmittedCount = null, int? limitOmittedCount = null, - List? selectors = null) + List? selectors = null, + string? nextCursor = null, + string? nextCursorUnavailableReason = null, + bool? hasMore = null) { var includeDiagnostics = HasReadOnlyFallbackDiagnostics(reader); return JsonSerializer.Serialize( @@ -1762,7 +1765,9 @@ private static string BuildJsonStreamDoneLine( TotalCount: totalCount, Interrupted: interrupted, Truncated: truncated, - HasMore: truncated || interrupted, + HasMore: hasMore ?? (truncated || interrupted), + NextCursor: nextCursor, + NextCursorUnavailableReason: nextCursorUnavailableReason, TotalCountAuthoritative: totalCountAuthoritative, TotalCountLowerBound: totalCountAuthoritative ? null : totalCount, SelectionReason: selectionReason, diff --git a/src/CodeIndex/Cli/QueryCommandRunner.cs b/src/CodeIndex/Cli/QueryCommandRunner.cs index 1065556fb..561cb05e8 100644 --- a/src/CodeIndex/Cli/QueryCommandRunner.cs +++ b/src/CodeIndex/Cli/QueryCommandRunner.cs @@ -85,6 +85,8 @@ public sealed class QueryCommandOptions internal QueryCommandInvocationContext InvocationContext { get; set; } = QueryCommandInvocationContext.Search; internal JsonSerializerOptions? InvocationJsonOptions { get; set; } internal bool InvocationMachineErrorOutputRequested { get; set; } + internal string[] InvocationArgs { get; init; } = []; + internal string? InvocationGenerationFingerprint { get; set; } public string DbPath { get; init; } = Path.Combine(".cdidx", "codeindex.db"); public bool DbPathExplicit { get; init; } public bool ReadOnly { get; init; } diff --git a/tests/CodeIndex.Tests/QueryCommandRunnerIssue5230Tests.cs b/tests/CodeIndex.Tests/QueryCommandRunnerIssue5230Tests.cs new file mode 100644 index 000000000..dad6aef07 --- /dev/null +++ b/tests/CodeIndex.Tests/QueryCommandRunnerIssue5230Tests.cs @@ -0,0 +1,533 @@ +using System.Text; +using System.Text.Json; +using System.Text.Json.Nodes; +using CodeIndex.Cli; +using CodeIndex.Database; + +namespace CodeIndex.Tests; + +[Collection("Console sensitive")] +public sealed class QueryCommandRunnerIssue5230Tests +{ + private readonly JsonSerializerOptions _jsonOptions = ProgramRunner.CreateDefaultJsonOptions(); + + [Fact] + public void Symbols_NdjsonCursorWalksEveryPageAndFinalPageOmitsContinuation_Issue5230() + { + var projectRoot = TestProjectHelper.CreateTempProject("ndjson_symbols_cursor_5230"); + try + { + var dbPath = TestProjectHelper.CreateProjectDb(projectRoot); + for (var index = 1; index <= 5; index++) + { + TestProjectHelper.InsertIndexedFile( + dbPath, + $"src/Issue5230Type{index}.cs", + "csharp", + $"public sealed class Issue5230Type{index} {{ }}\n"); + } + + var args = new[] + { + "symbols", "Issue5230Type", "--db", dbPath, "--json", "--limit", "2", + }; + var (firstExitCode, firstStdout, firstStderr) = CaptureConsole(() => + ProgramRunner.Run(args, _jsonOptions, "1.0.0-test")); + + Assert.Equal(CommandExitCodes.Success, firstExitCode); + Assert.Equal(string.Empty, firstStderr); + var firstRecords = ParseNdjson(firstStdout); + Assert.Equal(3, firstRecords.Length); + var names = firstRecords[..^1] + .Select(row => row.GetProperty("name").GetString()) + .ToList(); + var terminal = firstRecords[^1]; + Assert.True(terminal.GetProperty("has_more").GetBoolean()); + var cursor = Assert.IsType(terminal.GetProperty("next_cursor").GetString()); + Assert.StartsWith("response:v2:", cursor, StringComparison.Ordinal); + + JsonElement finalMetadata = default; + while (cursor is not null) + { + var (exitCode, stdout, stderr) = CaptureConsole(() => ProgramRunner.Run( + args.Concat(["--cursor", cursor]).ToArray(), + _jsonOptions, + "1.0.0-test")); + + Assert.Equal(CommandExitCodes.Success, exitCode); + Assert.Equal(string.Empty, stderr); + using var document = JsonDocument.Parse(stdout); + names.AddRange(document.RootElement.GetProperty("results") + .EnumerateArray() + .Select(row => row.GetProperty("name").GetString())); + finalMetadata = document.RootElement.GetProperty("metadata").Clone(); + cursor = finalMetadata.GetProperty("next_cursor").GetString(); + } + + Assert.Equal(5, names.Count); + Assert.Equal(5, names.Distinct(StringComparer.Ordinal).Count()); + Assert.False(finalMetadata.GetProperty("has_more").GetBoolean()); + Assert.Equal(JsonValueKind.Null, finalMetadata.GetProperty("next_cursor").ValueKind); + var finalTerminal = finalMetadata.GetProperty("stream_terminal"); + Assert.False(finalTerminal.GetProperty("has_more").GetBoolean()); + Assert.False(finalTerminal.TryGetProperty("next_cursor", out _)); + Assert.False(finalTerminal.TryGetProperty("next_cursor_unavailable_reason", out _)); + } + finally + { + TestProjectHelper.DeleteDirectory(projectRoot); + } + } + + [Fact] + public void SearchAndFiles_NdjsonTerminalsShareTheResumableCursorContract_Issue5230() + { + var projectRoot = TestProjectHelper.CreateTempProject("ndjson_shared_cursor_5230"); + try + { + var dbPath = TestProjectHelper.CreateProjectDb(projectRoot); + for (var index = 1; index <= 3; index++) + { + TestProjectHelper.InsertIndexedFile( + dbPath, + $"src/Issue5230Needle{index}.txt", + "text", + $"Issue5230Needle marker {index}\n"); + } + + AssertRawCursorResumesWithoutRepeatingFirstPath( + ["search", "Issue5230Needle", "--db", dbPath, "--exact-substring", "--json", "--limit", "1"]); + AssertRawCursorResumesWithoutRepeatingFirstPath( + ["files", "--path", "src/*.txt", "--db", dbPath, "--json", "--limit", "1"]); + + var (selectorExitCode, selectorStdout, selectorStderr) = CaptureConsole(() => + ProgramRunner.Run( + [ + "search", "Issue5230Needle", "--db", dbPath, "--exact-substring", + "--json", "--limit", "1", "--first-per-file", + ], + _jsonOptions, + "1.0.0-test")); + Assert.Equal(CommandExitCodes.Success, selectorExitCode); + Assert.Equal(string.Empty, selectorStderr); + var selectorTerminal = ParseNdjson(selectorStdout)[^1]; + Assert.True(selectorTerminal.GetProperty("has_more").GetBoolean()); + Assert.False(selectorTerminal.TryGetProperty("next_cursor", out _)); + Assert.Equal( + "stream_not_cursor_capable", + selectorTerminal.GetProperty("next_cursor_unavailable_reason").GetString()); + } + finally + { + TestProjectHelper.DeleteDirectory(projectRoot); + } + } + + [Fact] + public void Symbols_NdjsonSuppressesCursorWhenGenerationChangesDuringQuery_Issue5230() + { + var projectRoot = TestProjectHelper.CreateTempProject("ndjson_cursor_generation_race_5230"); + try + { + var dbPath = TestProjectHelper.CreateProjectDb(projectRoot); + for (var index = 1; index <= 3; index++) + { + TestProjectHelper.InsertIndexedFile( + dbPath, + $"src/Issue5230Race{index}.cs", + "csharp", + $"public sealed class Issue5230Race{index} {{ }}\n"); + } + + var generationChanged = false; + QueryCommandRunner.NdjsonRowsMaterializedForTesting = () => + { + if (generationChanged) + return; + + generationChanged = true; + using var db = new DbContext(DbOpenIntent.WriteIndex, dbPath); + var writer = new DbWriter(db.Connection); + writer.SetMeta( + DbContext.IndexedHeadTimestampMetaKey, + "2026-09-01T23:59:59.0000000+00:00"); + }; + + var (exitCode, stdout, stderr) = CaptureConsole(() => ProgramRunner.Run( + ["symbols", "Issue5230Race", "--db", dbPath, "--json", "--limit", "1"], + _jsonOptions, + "1.0.0-test")); + + Assert.True(generationChanged); + Assert.Equal(CommandExitCodes.Success, exitCode); + Assert.Equal(string.Empty, stderr); + var terminal = ParseNdjson(stdout)[^1]; + Assert.True(terminal.GetProperty("has_more").GetBoolean()); + Assert.False(terminal.TryGetProperty("next_cursor", out _)); + Assert.Equal( + "index_generation_changed_during_query", + terminal.GetProperty("next_cursor_unavailable_reason").GetString()); + } + finally + { + QueryCommandRunner.NdjsonRowsMaterializedForTesting = null; + TestProjectHelper.DeleteDirectory(projectRoot); + } + } + + [Fact] + public void Symbols_NdjsonSuppressesUnusableCursorAtPaginationWindow_Issue5230() + { + var projectRoot = TestProjectHelper.CreateTempProject("ndjson_cursor_window_5230"); + try + { + var dbPath = TestProjectHelper.CreateProjectDb(projectRoot); + var source = string.Join( + '\n', + Enumerable.Range(0, 10_001) + .Select(index => $"public sealed class Issue5230Window{index:D5} {{ }}")); + TestProjectHelper.InsertIndexedFile( + dbPath, + "src/Issue5230Window.cs", + "csharp", + source); + + var args = new[] + { + "symbols", "Issue5230Window", "--db", dbPath, "--json", "--limit", "1", + }; + var (_, firstStdout, _) = CaptureConsole(() => + ProgramRunner.Run(args, _jsonOptions, "1.0.0-test")); + var firstCursor = Assert.IsType(ParseNdjson(firstStdout)[^1] + .GetProperty("next_cursor") + .GetString()); + var boundaryCursor = ReplaceResponseCursorOffset(firstCursor, 9_999); + + var (exitCode, stdout, stderr) = CaptureConsole(() => ProgramRunner.Run( + args.Concat(["--cursor", boundaryCursor]).ToArray(), + _jsonOptions, + "1.0.0-test")); + + Assert.Equal(CommandExitCodes.Success, exitCode); + Assert.Equal(string.Empty, stderr); + using var document = JsonDocument.Parse(stdout); + var metadata = document.RootElement.GetProperty("metadata"); + Assert.True(metadata.GetProperty("pagination_window_exhausted").GetBoolean()); + Assert.False(metadata.GetProperty("has_more").GetBoolean()); + Assert.Equal(JsonValueKind.Null, metadata.GetProperty("next_cursor").ValueKind); + var terminal = metadata.GetProperty("stream_terminal"); + Assert.True(terminal.GetProperty("has_more").GetBoolean()); + Assert.False(terminal.TryGetProperty("next_cursor", out _)); + Assert.Equal( + "pagination_window_exhausted", + terminal.GetProperty("next_cursor_unavailable_reason").GetString()); + + var (largePageExitCode, largePageStdout, largePageStderr) = CaptureConsole(() => + ProgramRunner.Run( + [ + "symbols", "Issue5230Window", "--db", dbPath, + "--json", "--limit", "6000", + ], + _jsonOptions, + "1.0.0-test")); + Assert.Equal(CommandExitCodes.Success, largePageExitCode); + Assert.Equal(string.Empty, largePageStderr); + var largePageTerminal = ParseNdjson(largePageStdout)[^1]; + Assert.True(largePageTerminal.GetProperty("has_more").GetBoolean()); + Assert.False(largePageTerminal.TryGetProperty("next_cursor", out _)); + Assert.Equal( + "pagination_window_exhausted", + largePageTerminal.GetProperty("next_cursor_unavailable_reason").GetString()); + } + finally + { + TestProjectHelper.DeleteDirectory(projectRoot); + } + } + + [Fact] + public void Symbols_NdjsonCursorRejectsFilterMismatchAndIndexGenerationChange_Issue5230() + { + var projectRoot = TestProjectHelper.CreateTempProject("ndjson_cursor_validation_5230"); + try + { + var dbPath = TestProjectHelper.CreateProjectDb(projectRoot); + for (var index = 1; index <= 3; index++) + { + TestProjectHelper.InsertIndexedFile( + dbPath, + $"src/Issue5230Validation{index}.cs", + "csharp", + $"public sealed class Issue5230Validation{index} {{ }}\n"); + } + + var args = new[] + { + "symbols", "Issue5230Validation", "--kind", "class", + "--db", dbPath, "--json", "--limit", "1", + }; + var (_, firstStdout, _) = CaptureConsole(() => + ProgramRunner.Run(args, _jsonOptions, "1.0.0-test")); + var cursor = Assert.IsType(ParseNdjson(firstStdout)[^1] + .GetProperty("next_cursor") + .GetString()); + + var (mismatchExitCode, mismatchStdout, mismatchStderr) = CaptureConsole(() => + ProgramRunner.Run( + [ + "symbols", "Issue5230Validation", "--kind", "interface", + "--db", dbPath, "--json", "--limit", "1", "--cursor", cursor, + ], + _jsonOptions, + "1.0.0-test")); + Assert.Equal(CommandExitCodes.UsageError, mismatchExitCode); + Assert.Equal(string.Empty, mismatchStdout); + Assert.Contains("does not match this command, query, or filter set", mismatchStderr, StringComparison.Ordinal); + + using (var db = new DbContext(DbOpenIntent.WriteIndex, dbPath)) + { + var writer = new DbWriter(db.Connection); + writer.SetMeta( + DbContext.IndexedHeadTimestampMetaKey, + "2026-08-31T23:59:59.0000000+00:00"); + } + + var (staleExitCode, staleStdout, staleStderr) = CaptureConsole(() => + ProgramRunner.Run( + args.Concat(["--cursor", cursor]).ToArray(), + _jsonOptions, + "1.0.0-test")); + Assert.Equal(CommandExitCodes.UsageError, staleExitCode); + Assert.Equal(string.Empty, staleStdout); + Assert.Contains("index generation changed", staleStderr, StringComparison.Ordinal); + } + finally + { + TestProjectHelper.DeleteDirectory(projectRoot); + } + } + + [Fact] + public void Search_ZeroAndByteBudgetedTerminalsKeepContinuationProgressTruthful_Issue5230() + { + var projectRoot = TestProjectHelper.CreateTempProject("ndjson_cursor_budget_5230"); + try + { + var dbPath = TestProjectHelper.CreateProjectDb(projectRoot); + for (var index = 1; index <= 10; index++) + { + TestProjectHelper.InsertIndexedFile( + dbPath, + $"src/Issue5230Large{index}.txt", + "text", + $"Issue5230Large {index} {new string('x', 4_000)}\n"); + } + + var (zeroExitCode, zeroStdout, zeroStderr) = CaptureConsole(() => + ProgramRunner.Run( + ["search", "Issue5230Missing", "--db", dbPath, "--json", "--limit", "1"], + _jsonOptions, + "1.0.0-test")); + Assert.Equal(CommandExitCodes.Success, zeroExitCode); + Assert.Equal(string.Empty, zeroStderr); + var zeroTerminal = ParseNdjson(zeroStdout)[^1]; + Assert.False(zeroTerminal.GetProperty("has_more").GetBoolean()); + Assert.False(zeroTerminal.TryGetProperty("next_cursor", out _)); + + var (_, firstPageStdout, _) = CaptureConsole(() => ProgramRunner.Run( + [ + "search", "Issue5230Large", "--db", dbPath, "--exact-substring", + "--json", "--limit", "1", "--max-line-width", "4096", + ], + _jsonOptions, + "1.0.0-test")); + var firstPageRecords = ParseNdjson(firstPageStdout); + var firstPagePath = firstPageRecords[0].GetProperty("path").GetString(); + var firstPageCursor = Assert.IsType(firstPageRecords[^1] + .GetProperty("next_cursor") + .GetString()); + JsonElement trimmedMetadata = default; + JsonElement[] trimmedResults = []; + var trimmedBudget = 0; + foreach (var budget in Enumerable.Range(6, 25).Select(index => index * 1_000)) + { + var (exitCode, stdout, _) = CaptureConsole(() => ProgramRunner.Run( + [ + "search", "Issue5230Large", "--db", dbPath, "--exact-substring", + "--json", "--limit", "8", "--max-line-width", "4096", + "--cursor", firstPageCursor, "--max-json-bytes", budget.ToString(), + ], + _jsonOptions, + "1.0.0-test")); + if (exitCode != CommandExitCodes.Success || string.IsNullOrWhiteSpace(stdout)) + continue; + + using var document = JsonDocument.Parse(stdout); + var results = document.RootElement.GetProperty("results"); + if (results.GetArrayLength() is <= 0 or >= 8) + continue; + + Assert.True(Encoding.UTF8.GetByteCount(stdout) <= budget); + trimmedMetadata = document.RootElement.GetProperty("metadata").Clone(); + trimmedResults = results.EnumerateArray().Select(result => result.Clone()).ToArray(); + trimmedBudget = budget; + break; + } + + Assert.NotEqual(0, trimmedBudget); + var trimmedCursor = Assert.IsType(trimmedMetadata.GetProperty("next_cursor").GetString()); + Assert.Equal( + trimmedCursor, + trimmedMetadata.GetProperty("stream_terminal").GetProperty("next_cursor").GetString()); + var trimmedPaths = trimmedResults + .Select(result => result.GetProperty("path").GetString()) + .ToHashSet(StringComparer.Ordinal); + var (trimmedReplayExitCode, trimmedReplayStdout, trimmedReplayStderr) = CaptureConsole(() => + ProgramRunner.Run( + [ + "search", "Issue5230Large", "--db", dbPath, "--exact-substring", + "--json", "--limit", "8", "--max-line-width", "4096", + "--cursor", trimmedCursor, + ], + _jsonOptions, + "1.0.0-test")); + Assert.Equal(CommandExitCodes.Success, trimmedReplayExitCode); + Assert.Equal(string.Empty, trimmedReplayStderr); + using (var replayDocument = JsonDocument.Parse(trimmedReplayStdout)) + { + var replayPaths = replayDocument.RootElement.GetProperty("results") + .EnumerateArray() + .Select(result => result.GetProperty("path").GetString()) + .ToArray(); + Assert.DoesNotContain(replayPaths, path => trimmedPaths.Contains(path)); + var allPaths = trimmedPaths + .Concat(replayPaths) + .Append(firstPagePath) + .ToHashSet(StringComparer.Ordinal); + Assert.Equal(10, allPaths.Count); + } + + JsonElement budgetTerminal = default; + JsonElement[] budgetRecords = []; + var selectedBudget = 0; + foreach (var budget in Enumerable.Range(0, 161).Select(index => 200 + (index * 25))) + { + var (exitCode, stdout, _) = CaptureConsole(() => ProgramRunner.Run( + [ + "search", "Issue5230Large", "--db", dbPath, "--exact-substring", + "--json", "--limit", "10", "--max-line-width", "4096", + "--max-json-bytes", budget.ToString(), + ], + _jsonOptions, + "1.0.0-test")); + if (exitCode != CommandExitCodes.PartialResult || string.IsNullOrWhiteSpace(stdout)) + continue; + + var records = ParseNdjson(stdout); + var candidate = records[^1]; + if (!candidate.TryGetProperty("terminal_record", out var isTerminal) + || !isTerminal.GetBoolean()) + { + continue; + } + + Assert.True(Encoding.UTF8.GetByteCount(stdout) <= budget); + budgetTerminal = candidate; + budgetRecords = records; + selectedBudget = budget; + break; + } + + Assert.NotEqual(0, selectedBudget); + Assert.True(budgetTerminal.GetProperty("has_more").GetBoolean()); + var returnedCount = budgetTerminal.GetProperty("count").GetInt32(); + Assert.Equal(returnedCount, budgetRecords.Length - 1); + if (returnedCount == 0) + { + Assert.False(budgetTerminal.TryGetProperty("next_cursor", out _)); + Assert.Equal( + "no_result_row_emitted", + budgetTerminal.GetProperty("next_cursor_unavailable_reason").GetString()); + } + else + { + var cursor = Assert.IsType(budgetTerminal.GetProperty("next_cursor").GetString()); + var emittedPaths = budgetRecords[..^1] + .Select(record => record.GetProperty("path").GetString()) + .ToHashSet(StringComparer.Ordinal); + var (resumeExitCode, resumeStdout, resumeStderr) = CaptureConsole(() => + ProgramRunner.Run( + [ + "search", "Issue5230Large", "--db", dbPath, "--exact-substring", + "--json", "--limit", "10", "--cursor", cursor, + ], + _jsonOptions, + "1.0.0-test")); + Assert.Equal(CommandExitCodes.Success, resumeExitCode); + Assert.Equal(string.Empty, resumeStderr); + using var resumeDocument = JsonDocument.Parse(resumeStdout); + Assert.DoesNotContain( + resumeDocument.RootElement.GetProperty("results").EnumerateArray(), + result => emittedPaths.Contains(result.GetProperty("path").GetString())); + } + } + finally + { + TestProjectHelper.DeleteDirectory(projectRoot); + } + } + + private void AssertRawCursorResumesWithoutRepeatingFirstPath(string[] args) + { + var (firstExitCode, firstStdout, firstStderr) = CaptureConsole(() => + ProgramRunner.Run(args, _jsonOptions, "1.0.0-test")); + Assert.Equal(CommandExitCodes.Success, firstExitCode); + Assert.Equal(string.Empty, firstStderr); + var records = ParseNdjson(firstStdout); + var firstPath = records[0].GetProperty("path").GetString(); + var terminal = records[^1]; + Assert.True(terminal.GetProperty("has_more").GetBoolean()); + var cursor = Assert.IsType(terminal.GetProperty("next_cursor").GetString()); + Assert.StartsWith("response:v2:", cursor, StringComparison.Ordinal); + + var (secondExitCode, secondStdout, secondStderr) = CaptureConsole(() => ProgramRunner.Run( + args.Concat(["--cursor", cursor]).ToArray(), + _jsonOptions, + "1.0.0-test")); + Assert.Equal(CommandExitCodes.Success, secondExitCode); + Assert.Equal(string.Empty, secondStderr); + using var secondDocument = JsonDocument.Parse(secondStdout); + var secondPath = secondDocument.RootElement.GetProperty("results")[0] + .GetProperty("path") + .GetString(); + Assert.NotEqual(firstPath, secondPath); + var metadata = secondDocument.RootElement.GetProperty("metadata"); + Assert.Equal( + metadata.GetProperty("next_cursor").GetString(), + metadata.GetProperty("stream_terminal").GetProperty("next_cursor").GetString()); + } + + private static JsonElement[] ParseNdjson(string stdout) + => stdout.Split(Environment.NewLine, StringSplitOptions.RemoveEmptyEntries) + .Select(line => JsonDocument.Parse(line).RootElement.Clone()) + .ToArray(); + + private static string ReplaceResponseCursorOffset(string cursor, int offset) + { + const string prefix = "response:v2:"; + Assert.StartsWith(prefix, cursor, StringComparison.Ordinal); + var encoded = cursor[prefix.Length..] + .Replace('-', '+') + .Replace('_', '/'); + encoded += new string('=', (4 - (encoded.Length % 4)) % 4); + var payload = JsonNode.Parse(Convert.FromBase64String(encoded))!.AsObject(); + payload["offset"] = offset; + return prefix + Convert.ToBase64String(Encoding.UTF8.GetBytes(payload.ToJsonString())) + .TrimEnd('=') + .Replace('+', '-') + .Replace('/', '_'); + } + + private static (int ExitCode, string Stdout, string Stderr) CaptureConsole(Func action) + => ConsoleCapture.Capture(action); +}