[Bug]: Startup hangs ~30s in large repos: three full-tree scans ignore .gitignore

[faceleg]

Bug Description

Upstream issue draft: autohand startup hangs ~30s in large repos

Title: Startup hangs ~30s in large repos: three full-tree scans ignore .gitignore

Version: 0.9.7-alpha.f4775fd
Environment: macOS arm64 (Apple M4, 10 cores, 16GB), Node v26.3.0, Bun 1.4.2

Summary

In a repo with 10 GB / 218K files (.derivedData 50K files, .claude/worktrees 121K files across 6 worktrees — all gitignored), autohand blocks ~30s before the first prompt. Three independent startup scans walk the tree, and none of them respect .gitignore:

1. collectWorkspaceFiles() / walkWorkspaceFiles — only skips the hardcoded FILE_WALK_IGNORED_DIRECTORIES = {".git",".hg",".svn","node_modules","dist","build","coverage"}. It does not use the GitIgnoreParser that already exists elsewhere in the codebase. Simulating the exact algorithm in plain Node: 231,337 files, ~4.3s — and this runs inside initializeAgentManagers → workspace_files at startup.

2. FFFSearchProvider.create(workspaceRoot) — calls ffiCreate(basePath, …, aiMode: true) with no disableWatch, no disableContentIndexing, no cache budgets, then blocks on waitForScan(1e4) (10s timeout). FFF respects gitignore, but on first run it still walks/stats/content-indexes the tree and installs a watcher. The TUI shows a " Scanning <source>..." phase.

3. Skills discovery walker — skip set is {".git",".hg","node_modules","vendor","dist","build","target","coverage",".venv","venv",".next",".turbo",".vitest",".worktrees","__pycache__"} — again no .derivedData/.claude. Its descend rule accepts any ^\.[\w-]+$ top-level dir, so it descends into .derivedData and .claude (20–30K entry budgets), plus walks ~/.autohand, ~/.claude, ~/.codex at depth 9.

4. MCP connectAll — 4 servers spawned in parallel at startup (uvx, npx, remote HTTP, xcrun); measured 0.15–4.2s each, plus zombie MCP processes from prior sessions.

All of this is awaited before the first instruction (performBackgroundInit → ensureInitComplete).

Repro

# repo with a gitignored dir containing 50K+ files (e.g. .derivedData) and
# .claude/worktrees with several full checkouts
time autohand            # ~30s before prompt
AUTOHAND_DEBUG=1 autohand 2>/tmp/ah-debug.log   # observe Scanning phase
mv .derivedData /tmp/ && mv .claude/worktrees /tmp/ && time autohand  # ~2-4s

Expected behavior

• Startup must not block on scanning gitignored directories; the first prompt should appear quickly.
• Scans should be lazy (deferred until fff_find/find_grep/skills are actually used) or respect .gitignore.

Suggested fixes

• Make FILE_WALK_IGNORED_DIRECTORIES gitignore-aware (reuse GitIgnoreParser) or configurable.
• Add common build/tool dirs to both walker skip sets: .derivedData, .claude, .venv*, .opencode, .cache, .ruff_cache, .superpowers, .xcodebuildmcp, .test-batch-state, .worktrees.
• Add config to disable FFF watch/content-indexing at startup, or make FFF init lazy.
• Consider a startup budget: don't gate the first instruction on background scans.

Local evidence (XXX repo)

• Repo size: 10 GB, 218,241 files (find . = 5.5s)
• .derivedData: 4.7 GB / 50,493 files
• .claude/worktrees: 2.4 GB / 121,588 files (6 worktrees)
• .git: 839 MB
• Simulated walkWorkspaceFiles (exact skip set): 231,337 files, 4.3s
• MCP cold-start measurements: npx xcodebuildmcp 1.4s, uvx memory 4.2s, xcrun mcpbridge 0.15s
• Zombie MCP processes observed: 3× xcodebuildmcp + 1× mcp-remote from prior sessions

Steps to Reproduce

.

Expected Behavior

.

Actual Behavior

.

Autohand Version

0.9.7

Operating System

macOS (Apple Silicon)

OS Version

No response

Installation Method

curl install script

LLM Model (if applicable)

No response

Error Logs

Configuration (optional)

Checklist

• I have searched for existing issues that describe this bug
• I have removed any sensitive information (API keys, personal data) from the logs/config

[L4XB]

Read the three scans at f4775fd while looking at this. Two of the four items hold as written; the first one names the wrong walker, which matters because it is the one with the measured 4.3s attached to it.

workspace_files at startup already respects .gitignore. initializeAgentManagers runs host.workspaceFileCollector.collectWorkspaceFiles() (src/core/agent/AgentLifecycleRunner.ts), which is WorkspaceFileCollector, not the FILE_WALK_IGNORED_DIRECTORIES walker:

• it runs git ls-files --cached --others --exclude-standard, so gitignored paths never reach it, and filters the result through ignoreFilter.isIgnored() on top;
• the whole scan is bounded by DEFAULT_WORKSPACE_SCAN_BUDGET_MS = 5_000, enforced by a timer that SIGKILLs git and keeps the complete lines it has;
• the filesystem walk is a fallback that runs only when git ls-files returns zero entries, and it checks this.ignoreFilter.isIgnored(rel) per entry before descending, plus shouldSkipPath.

So in a git repo where .derivedData and .claude/worktrees are gitignored, this step should neither walk them nor exceed 5s. If it does on your machine, the interesting question is which of those two mechanisms failed: git ls-files returning zero (making the walk the live path), or GitIgnoreParser not loading the patterns. That is worth a measurement before any skip list grows, because a new hardcoded directory would paper over whichever one broke.

The FILE_WALK_IGNORED_DIRECTORIES walker is a different surface. It lives in src/search/fffSearchProvider.ts and its only caller is fileSearchWithFilesystemWalk, reached from one place: the isMissingExecutableError(error) branch of the file-search path, i.e. when the search binary is not available. It is gitignore-blind, and worth fixing for what it is, but it is not on the startup path and does not explain the 30s.

Items 2, 3 and 4 look right to me. The FFF create with no disableWatch / disableContentIndexing and a waitForScan(1e4) is a 10s bound taken at startup by construction, and four MCP spawns in parallel at 0.15 to 4.2s each is the other end of the budget. For item 3 I could not find the skip set you quote in the TypeScript sources: __pycache__ appears in src/actions/metadata.ts (a tree lister that does use GitIgnoreParser), and the .turbo / .vitest / .worktrees set appears only inside the generated bundle string in src/discovery/workerSource.ts. If that walker is the discovery worker, its real source is wherever that bundle is built from, and it would help to name the file in the issue.

Happy to take the fffSearchProvider walker as its own change: reuse GitIgnoreParser there so the fallback stops enumerating gitignored trees. I would rather not grow either skip list until the startup measurement says which mechanism actually failed, since a hardcoded .derivedData would hide it.