perf: remove autocomplete stalls and add a measured performance study

[Maxteabag]

Autocomplete with 5,000 stored routines repeatedly scanned the entire catalog for each candidate, producing a measured 1,841 ms completion/dropdown operation. Building one routine-identity index reduces that median to 35 ms while preserving observed suggestions and namespace disambiguation.

The source changes also stop empty idle-queue polling, honor a saved disabled-worker setting before CLI prewarm, repair absolute diagnostic-path arguments, and bound oversized literal-cell display without truncating backend values. The long-cell change's whole-operation speedup was inconclusive in the repeated trials.

Includes a standalone HTML performance paper, 11 SVG figures, an interactive statistics/trace viewer, raw observations and reproducible PTY/database labs. The 152 formal jobs produced 1,349 observations across startup, idle CPU, completion, rendering, PostgreSQL/MySQL/MariaDB under controlled latency, SQLite/DuckDB, process workers and IPC. Experimental alternatives remain in the lab.

Validation:

• Python 3.13: 1,801 passed, 14 skipped across unit, UI, CLI, SQLite and DuckDB suites; skip reasons are retained.
• Python 3.10: 408 targeted compatibility tests passed.
• Desktop/mobile report checks passed: figures, links, CSV export, trace loading, no page errors or external asset requests.
• Headless long-cell visual fixture verifies 48 complete original values.
• Measured source: baseline 9db49c2, candidate 0640c96; measurement-script hashes match lab snapshot 03e68b7.

Explorer-focus idle CPU fell 72% in the five-second workload. Launch-and-exit CPU fell 16% with the worker disabled; a default first-refresh speedup was not established. The paper documents confidence intervals, rendering regressions in rejected prototypes, and the boundary between measured opportunities and implemented changes.

[Maxteabag]

This PR is part of Peter's experiment with giving Codex a broad performance-improvement brief and evaluating the code, measured results, reproducible experiments and report it produces. The original study used gpt-6-astra with xhigh reasoning effort.

Original prompt — spelling, punctuation and grammar corrected; meaning preserved

I want you to deeply investigate various ways of reducing startup time, CPU usage, and any stutters in the app (FPS, performance, etc.). I need you to use the existing diagnostic tools (watchdog, debugger, tracer system, etc.), but also create all the diagnostic tools you want in order to debug this and create labs and experiments.

I want you to go in many different directions and be creative in how you make this system faster and more lightweight. I want you to actually do the experiments, not just code in theory. You need to create simulated environments, whether that be with Docker, integration tests, unit tests, or whatever is appropriate.

Also experiment with ways to speed up connections to various databases.

In the end, I expect a full HTML scientific paper with graphs, statistics, recommended actions, and measurements showing how much changed in percentages, milliseconds, bandwidth, CPU usage, etc.

You have complete freedom to make this system amazing and polished.

Time and token usage for the original completed study

Metric	Recorded amount
Elapsed wall time	2 hours, 1 minute, 30 seconds (rounded)
Recorded model calls	115
Input tokens, including cached input	28,116,203
Cached input tokens — included above	26,985,344
Uncached input tokens — calculated difference	1,130,859
Output tokens, including reasoning	191,959
Reasoning tokens — included in output	108,297
Total input + output tokens	28,308,162

The recorded duration is 7,289,746 ms, from approximately 02:46:19 to 04:47:49 UTC on 10 September 2026. It includes investigation, implementation, lab iterations, tests, report construction, visual checks and publication. These totals exclude this follow-up comment. Token accounting is cumulative across model calls: reused context is counted again, and 95.98% of input tokens were cache hits. The cached-input and reasoning rows are subsets, not additional tokens to add to the total.

What happened

• The work started in separate baseline/candidate worktrees, preserving the original dirty checkout. Existing startup/import profiling, debug events and the stall watchdog were combined with new PTY, event-loop, CPU, byte-counting and database labs.
• The formal study completed 152 jobs and 1,349 observations, including real PostgreSQL, MySQL, MariaDB, SQLite and DuckDB experiments, plus simulated network latency and multiple rendering/IPC strategies.
• The main finding was a quadratic routine-name scan in autocomplete. Fixing it reduced the measured 5,000-routine completion/dropdown operation from 1,841 ms to 35 ms. Empty-queue polling and premature worker creation also produced verified CPU savings. The changes remain workload-specific; a default startup wall-time improvement and an overall long-cell speedup were not established.
• Some attractive-looking approaches were rejected as defaults: bulk rendering and threaded preparation completed sooner but caused longer stalls. Other ideas, including the sampled row backend and database connection reuse, remain documented experiments or follow-up work.
• The deliverable is the HTML paper with 11 figures, confidence intervals, raw observations, reusable diagnostics and recommendations. Validation recorded 1,801 passed / 14 skipped, plus a separate 408-test Python 3.10 compatibility pass. The PR is a draft; it is not a release or deployment.

[Maxteabag]

Open the HTML performance report

The report is now hosted on GitHub Pages and opens directly in your browser. No download, local setup or GitHub sign-in is needed.

It includes all 11 figures, the interactive statistics table and trace viewer, plus downloadable raw data and SVG figures. The live page, controls and downloads have been checked at desktop and phone widths.