Skip to content

Stop rebuilding the projection on every cone change - #33

Open
tyrielv wants to merge 1 commit into
feature/sparse-indexfrom
tyrielv/sparse-index/no-cone-rebuild
Open

tyrielv wants to merge 1 commit into
feature/sparse-indexfrom
tyrielv/sparse-index/no-cone-rebuild

Conversation

@tyrielv

@tyrielv tyrielv commented Sep 18, 2026

Copy link
Copy Markdown
Owner

What

Removes the projection rebuild that automatic sparse-index cone management triggered after every cone change. On a large enlistment that rebuild cost about 2.5 seconds per widen. It produces a byte-identical projection, so it was pure overhead.

Why the call was there, and why that reason was wrong

The call at AutoSparseIndexConeManager.ApplyCone was justified by this comment:

The collapse rewrote the on-disk index with hooks disabled, so the mount got no PostIndexChanged notification.

That premise is false. Of the three index-rewrite commands, only ForceCollapseSparseIndex omits core.hookspath=:

Method core.hookspath= Hooks
ForceExpandSparseIndex present disabled
CollapseSparseIndexAtClone present disabled
ForceCollapseSparseIndex absent intact

Its own remark says so — it must keep the hooks path intact, because git consults the virtualfilesystem hook while writing the index. Only usePreCommandHook: false is set, which exports COMMAND_HOOK_LOCK=true and affects the pre-command hook.

So git does run its post-index-change hook. update-index sets updated_skipworktree unconditionally, so the mount was already being notified with:

(updatedWorkingDirectory: false, updatedSkipWorktreeBits: true)

which is the exact inverse, on both flags, of what the manual call requested.

Why the projection cannot change

A cone change alters only how the index represents a directory — one sparse-directory entry versus individual file entries. SparseDirectoryExpander normalizes both back to the full HEAD tree, so the projected set is invariant.

Verified by dumping the full projected set (one line per entry: file path + blob SHA, folder path + inclusion flag) after each rebuild:

entries SHA-256
before widen 2,628,724 0536687E…96BFDB13
after widen 2,628,724 0536687E…96BFDB13

Byte-identical. The folder inclusion flags are the direct test of the skip-worktree concern: had a widen changed any entry's projected-ness, those lines would differ.

Negative control: moving HEAD did change the dump, so the comparison is not vacuous.

The operation's own invalidation is untouched

Widen and narrow run inside the pre-command hook of an operation that may itself change skip-worktree. Only one of the three mechanisms is removed:

Trigger Notification After
our cone rewrite git → modified-paths only unchanged
the user's operation git → its own flags unchanged
our manual call hand-rolled removed

A path checkout that clears skip-worktree still reports updated_skipworktree=1 and still invalidates.

Verification

  • Unit tests: 1261 passed, 0 failed, 0 skipped
  • Live mount, widen after the change: InvalidateProjection: 0, zero projection rebuilds. Git's notification drives a AddMissingModifiedFilesAndRemoveThemFromPlaceholderList pass in 4 ms instead.
  • Projection correctness after the change: 53/53 root directories present; a deep placeholder hydrates to a blob matching HEAD exactly; status clean.

RequestIndexProjectionUpdate existed only for this call site and is removed with it.

Notes

  • This supersedes cache replay as the primary fix for cone-widen cost. Caching would have optimised work that does not need to happen.
  • Only the widen direction was diffed. Cone narrowing is not reachable in the current implementation — the cone only grows — so it could not be exercised.
  • A separate, pre-existing defect was found while validating this and is not addressed here: after widen-then-reset --hard, a clean tracked file keeps an individual index entry with skip-worktree cleared but is no longer in ModifiedPaths, so it drops out of the projection. Latent rather than visible, because the file stays hydrated on disk.

Automatic sparse-index cone management triggered a full projection rebuild
after every widen and narrow, costing about 2.5 seconds on a large
enlistment. That rebuild produces a byte-identical projection, so it was
pure overhead.

The call was justified by a comment stating that the collapse runs with
hooks disabled and therefore produces no PostIndexChanged notification.
That premise is wrong. ForceCollapseSparseIndex is the one index-rewrite
command that deliberately leaves core.hookspath intact, so git runs its own
post-index-change hook for it. update-index sets updated_skipworktree
unconditionally, so the mount is already notified with
(updatedWorkingDirectory: false, updatedSkipWorktreeBits: true) - the exact
inverse of what the manual call requested on both flags.

A cone change alters only how the index represents a directory: one
sparse-directory entry versus individual file entries.
SparseDirectoryExpander normalizes both forms back to the full HEAD tree, so
the projected set cannot change. Verified by dumping the full projected set
before and after a widen on a large enlistment: 2,628,724 entries, identical
SHA-256, including every blob SHA and every folder inclusion flag. Moving
HEAD was used as a negative control and did change the dump, so the
comparison is not vacuous.

After this change a widen performs no projection rebuild. Git's own
notification drives a 4 ms modified-paths validation instead.

RequestIndexProjectionUpdate existed only for this call site and is removed
with it.

Assisted-by: Claude Opus 5
Signed-off-by: Tyrie Vella <tyrielv@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant