[Mike Krüger (mkrueger)]

Improves what users see during long-running ingestion and documents expectations and limitations for customers.

Progress reporting

import previously only printed a final summary, and bulk printed progress only when an operation completed, so it went silent during slow writes or SDK retry waits. Both now share an IngestionProgress heartbeat that reports about every two seconds:

• import: items processed, imported/failed, elapsed seconds, items/s, observed write RU charge. Dry runs report offline validation counts instead. Because input is streamed without a counting pass, there is no total, percentage, or ETA.
• bulk: processed/total operations, succeeded/failed, elapsed seconds, operations/s, observed RU charge. Progress starts after validation, confirmation, and journal verification.

Progress is human-readable output only. Quiet mode, structured output (json/csv), redirected stdout, and MCP calls emit nothing extra, and final result shapes are unchanged. Reporting failures never abandon in-flight writes.

Sizing guidance and measurements

New docs/ingestion.md covers pilot sizing, how to read progress, limits (2 MB per item, stateful bulk caps, temporary spool disk, no rollback, retry safety), and local emulator measurements.

The opt-in IngestionBenchmarkTests (skipped unless COSMOSDB_SHELL_INGESTION_BENCHMARK_CSV is set) ran the full matrix of import and bulk create with 1K/10K/100K 1 KiB items at concurrency 1/16/32. All 18 cases verified their final container counts (666,000 writes). Concurrency 16 was about 3x faster than 1; 32 showed no consistent gain:

Items	Import (c=16)	Bulk (c=16)
1,000	5.2 s	5.3 s
10,000	51.8 s	51.7 s
100,000	8.6 min	8.7 min

These are local emulator command-path numbers, not Azure throughput promises. Representative Azure measurements are still needed before publishing customer completion-time estimates.

Local emulator timeout

Commands connected to a localhost emulator were canceled after 10 seconds in total, even while making progress. That broke long imports, rm *, exports, queries, and piped scripts, and disconnected the shell. The limit is now an inactivity timeout:

• Every emulator response restarts it (via a custom RequestHandler). Completed import/bulk writes and locally validated bulk operations also restart it. Request timeouts and 503s do not count, because the SDK synthesizes them when the emulator doesn't answer.
• It is paused during confirmation prompts and the sproc/trigger/udf editors, which used to abort after 10 seconds of user input.
• A stalled or unreachable emulator still times out and disconnects.

rm progress and in-place progress line

Pattern-based and piped rm now report items scanned and deleted. In a terminal, progress for import, bulk, and rm is redrawn on a single line (truncated to the console width), cleared before mid-run warnings, and erased when the command finishes. Progress appears only when stdout is a terminal; pipes and OS or shell redirection suppress it.

Validation

• dotnet build for the shell and test projects: 0 warnings, 0 errors
• Offline suite (Category!=Emulator): 4,927 passed
• Emulator suite: all new tests pass; 2 existing failures (Info_Container_FormatTable_RedirectsGrid, Sproc_Exec_WithoutResponseBody_ReturnsNull) also fail on unmodified main
• End to end against the emulator with the default limit: 3,000-item import (23.9 s), 1,000-pair piped script (22.5 s), and rm * on 3,000 items (48.5 s) all complete; previously each stopped at about 10 s
• l10n/CosmosDBShell.json regenerated with the new and changed strings

Example:

[github-code-quality]

Code coverage overview

Languages: C#

C# / code-coverage/dotnet

main 437e8b4	dev/mkrueger/feature... 5740191	+/-
72%	73%	+1%

View coverage of the 10 most impacted files.

File	main 437e8b4	dev/mkrueger/feature... 5740191	+/-
D:\a\CosmosDBSh...BulkExecutor.cs	100%	98%	-2%
D:\a\CosmosDBSh...hell\Program.cs	82%	83%	+1%
D:\a\CosmosDBSh...\BulkCommand.cs	91%	93%	+2%
D:\a\CosmosDBSh...lInterpreter.cs	76%	80%	+4%
D:\a\CosmosDBSh...ThemeCommand.cs	60%	70%	+10%
D:\a\CosmosDBSh...ds\RmCommand.cs	16%	42%	+26%
D:\a\CosmosDBSh...\EditCommand.cs	5%	51%	+46%
D:\a\CosmosDBSh...WatchCommand.cs	23%	89%	+66%
D:\a\CosmosDBSh...tionProgress.cs	0%	95%	+95%
D:\a\CosmosDBSh...ivityTimeout.cs	0%	100%	+100%

ℹ️ Calculations represent line coverage. Results may change as calculations are updated.

---

_{Updated October 29, 2026 17:55 UTC}

[Mike Krüger (mkrueger)]

Addressed all 6 findings from the Copilot review overview in dc523ea (no suppressed findings were listed). Each inline thread has a reply and is resolved. That commit also fixes the build-test-package failure: OperationInactivityTimeoutTests.Progress_RestartsTimeout depended on wall-clock timing on the slow CI runner, so the timer tests now use a deterministic manual clock.

[Mike Krüger (mkrueger)]

Second Copilot review round addressed in 3e0daee: the 6 overview findings plus the code-quality suggestion. Each inline thread has a reply and is resolved. The 5 "previously missed" findings are also fixed:

• Disconnect on bulk timeout despite structured errors: same fix as the structured-error thread.
• Pause the invoking interpreter while editing: same fix as the confirmation thread.
• Ignore all non-fatal progress cleanup exceptions: clearing the line now ignores every non-fatal exception, like rendering does.
• Preserve key metrics in width-aware output: added compact *-compact progress messages, used when the full line doesn't fit the console width; truncation is only a last resort.
• Atomically clear and render warnings: RenderWarnings holds the progress-line lock from clearing through writing the warnings, so a heartbeat can't redraw in between.

[Mike Krüger (mkrueger)]

Third Copilot review round: all 3 overview findings (console-width-dependent tests, lost script location, and exit code for emulator timeouts) are addressed in b2f1799, and their threads are replied to and resolved. The overview listed no suppressed or previously missed findings this round.

[Mike Krüger (mkrueger)]

Fourth Copilot review: no open issues, all 3 previous threads are resolved, and CI is green.

On the overview note that progress stays visible for upstream stages of a pipeline: this is intentional, and the docs now say so (74f2a49). In-shell | stages pass data as CommandState results, not as console text. The progress line goes straight to the terminal and is erased before the stage returns, so it never reaches the piped data or a final > redirect, and a long first stage still shows it is working. Operating-system pipes and redirection, --quiet, structured output, and MCP still suppress progress entirely.

[Mike Krüger (mkrueger)]

Resolved the two latest findings in 1d96c97, with individual explanations before resolution. A successfully stopped watch in stdin now prevents separately supplied startup commands from running. Timed-out import JSON now matches the connection-error exit code and message while retaining settled counts/RUs; direct and block regressions also validate redirected JSON and MCP responses. Both builds pass with zero warnings/errors; focused tests pass (120), and offline tests pass (4,978 passed, 6 expected skips).

[Copilot]

🔵 Needs a closer look

Completed writes remain unrecorded during slow input reads, leaving heartbeat counts and RU charges stale.

No open issues 🎉

2 resolved since last review

• Stop follow-up commands after handled watch cancellation
• Synchronize structured errors when converting import timeouts

_{Review effort: Balanced}

[Mike Krüger (mkrueger)]

Addressed the latest review overview even though it reported no open threads: completed import writes could remain unrecorded during slow asynchronous input reads. Commit 6d92a29 drains completions during those reads and safely settles the active read before source disposal on failure/cancellation. Added live-count/RU and cleanup regressions; both builds pass without warnings/errors, and offline tests pass (4,981 passed, 6 expected skips). Added a substantive follow-up on the original resolved progress thread for the suppressed duplicate.

[Mike Krüger (mkrueger)]

Synchronized the branch with the newly updated main branch (#263) without including unrelated local benchmark/documentation edits. The merged app and tests build without warnings/errors; offline suite: 5,009 passed, 6 expected skips. This also refreshes GitHub mergeability after it reported a conflicting base.

[Copilot]

🔵 Needs a closer look

Safety-sensitive interactions across startup, cancellation, and command execution require human assessment beyond the published review.

No open issues 🎉

_{Review effort: Balanced}

[Mike Krüger (mkrueger)]

Fixed the newly enabled emulator CI failures in 3d438a3. Real requests are paced in the timing fixture so a fast emulator cannot invalidate the >timeout assertions; the shared error helper now accepts structured partial failures. Focused validation against the running emulator passed (49 tests), and the offline suite passed (5,009 tests, 6 expected skips). The preceding merged-head build/package/localization/CodeQL checks passed. The latest Copilot review reports no open issues; its generic human-assessment note identifies no additional concrete defect. Startup/cancellation safety paths have explicit regressions, and the overview-only slow-input progress concern was implemented rather than dismissed. Also added substantive confirmations to the pre-existing resolved threads whose previous replies were placeholders.

[Copilot]

🟡 Changes recommended

Unresolved cancellation and timeout gaps can let interrupted commands continue or disconnect active sessions, while progress reporting and machine-readable output can also be incorrect.

10 open issues

• Honor cancellation while loading journal records · New
• Refresh activity for successfully settled writes · New
• Settle completed writes while awaiting spool reads · New
• Report activity for locally rejected import records · New
• Publish scan progress and refresh activity during pattern scans · New
• Propagate handled cancellation through expression evaluation · New
• Cancel startup operations on Ctrl+C during prompts · New
• Pause external commands executed in expressions · New
• Keep interrupt markers out of machine-readable output · New
• Create startup cancellation before reading stdin · New

_{Review effort: Balanced}

[Copilot]

🔵 Needs a closer look

Unresolved issues can leave progress stale, trigger false emulator timeouts, and prevent cancellation from promptly stopping startup or enclosing command work.

10 open issues

• Create startup cancellation before reading stdin
• Keep interrupt markers out of machine-readable output
• Pause external commands executed in expressions
• Cancel startup operations on Ctrl+C during prompts
• Propagate handled cancellation through expression evaluation
• Publish scan progress and refresh activity during pattern scans
• Report activity for locally rejected import records
• Settle completed writes while awaiting spool reads
• Refresh activity for successfully settled writes
• Honor cancellation while loading journal records

_{Review effort: Balanced}

[Mike Krüger (mkrueger)]

Resolved all ten newly published findings in 1cd1568, each with a substantive reply before resolution. Covered bulk journal cancellation/settlement, bulk slow-source progress, local import validation, per-row removal scans, handled watch expressions, confirmation interrupts, external expressions, clean machine stdout, and cancellable stdin reads. Both builds pass cleanly. The configured offline suite passes (5,042 passed, 6 expected skips), and the four real-emulator inactivity regressions pass separately. Startup/prompt/output focused coverage passed 106 tests; focused bulk regressions passed 11 tests. Unrelated benchmark and ingestion-documentation changes remain uncommitted and untouched.

[Copilot]

🟡 Changes recommended

Interrupting a command with an explicit structured format can corrupt stdout with ^C because the check considers only the session default.

1 open issue

• Use the active command format for interrupt markers · New

10 resolved since last review

• Create startup cancellation before reading stdin
• Keep interrupt markers out of machine-readable output
• Pause external commands executed in expressions
• Cancel startup operations on Ctrl+C during prompts
• Propagate handled cancellation through expression evaluation
• Publish scan progress and refresh activity during pattern scans
• Report activity for locally rejected import records
• Settle completed writes while awaiting spool reads
• Refresh activity for successfully settled writes
• Honor cancellation while loading journal records

_{Review effort: Balanced}

[Mike Krüger (mkrueger)]

Completed the final format-specific interrupt finding and response-lifetime follow-ups in 5740191. Explicit command JSON/CSV formats now suppress interrupt markers even with a human-readable session default; nested structured contexts remain protected. Mock setup no longer consumes the local-validation timing budget. Dismissed the proposed SystemException-only catch with a technical explanation and non-SystemException regressions preserving first-failure semantics. Both builds pass cleanly; full offline suite: 5,058 passed, 6 expected skips. All published review threads currently have substantive dispositions and are resolved.

[Copilot]

🔵 Needs a closer look

Cancellation, redirection, and inactivity handling interact across multiple execution paths, warranting human assessment of their combined behavior.

No open issues 🎉

1 resolved since last review

• Use the active command format for interrupt markers

_{Review effort: Balanced}

[Mike Krüger (mkrueger)]

Final verification at 5740191: all 47 review threads are resolved, including overview/suppressed findings and explained inapplicable suggestions. Latest Copilot review reports no open issues; its remaining overview is a general combined-behavior review caution, with no additional concrete defect. All applicable checks are green: build/test/package, fuzzer, localization, CodeQL, offline integration, Linux emulator, Windows partition-key integration, and CLA. Local offline validation passed 5,058 tests with 6 expected skips; four real-emulator inactivity regressions also passed. Branch is mergeable and synchronized with main. Unrelated local benchmark and ingestion-documentation edits were preserved; temporary isolated build outputs were removed.