perf: Optimize the heck out of the storage of token trees

[ChayimFriedman2]

This is basically the most optimized (memory-wise) storage possible, found after multiple measurements. The price we pay for this ultra-extra optimization is a bunch of unsafe, encapsulated in tt/src/storage.rs. Since macros and therefore token trees are so common in Rust code, I think this is worth it.

Some stats:

• On rust-analyzer itself, memory usage is reduced by 30mb. rust-analyzer doesn't use macros a lot and the previous optimization already took the most, but when considering that all token trees in r-a now consumes only about 40mb, this is still surprising.
• On buck2, ~133mb ~143mb is saved.
• On omicron, ~352mb ~436mb is saved, and this is after the previous optimization already ripped 880mb! It is only using ~264mb ~180mb for token trees now, in total!

The basic idea is to use a variable-length encoding into a bytes array. Multiple measurements were done in order to determine the most common forms of token trees along with their frequencies, and to find the best encoding.

In addition, we also now sort the compressed spans by their frequencies (in a descending order), so that even if a TopSubtree has more than 2^4 unique compressed spans, we will still use the more efficient encoding for the biggest number of spans possible. This is made possible by the fact that unlike the previous encoding, now we don't force one span encoding for all tokens (or in fact even for the two spans in one subtree).

The basic form is not expected to change and this is pretty much ready for review, but I still want to add more documentation and tests.

[ChayimFriedman2]

Exceptionally, I requested a review from GitHub Copilot, since this has a bunch of subtle logic and magic constants that can be subtly mistaken and AI is good at catching such things.

[Copilot]

Pull request overview

Reworks token-tree storage to reduce memory through compact variable-length encoding.

Changes:

• Adds byte-level token-tree encoding with frequency-ranked spans.
• Updates iteration, cursors, builders, and symbol ownership handling.
• Adjusts dependent APIs, tests, and test features.

Reviewed changes

Copilot reviewed 9 out of 10 changed files in this pull request and generated 6 comments.

Show a summary per file

File	Description
crates/tt/src/storage.rs	Implements compressed storage and rebuilding.
crates/tt/src/lib.rs	Adapts token-tree views and enums.
crates/tt/src/iter.rs	Updates iteration for encoded storage.
crates/tt/src/buffer.rs	Reworks cursor state and decoding.
crates/mbe/src/lib.rs	Uses cursor-based iterator advancement.
crates/intern/src/symbol.rs	Adds raw symbol ownership APIs.
crates/hir-expand/src/fixup.rs	Adapts token-tree comparison.
crates/cfg/Cargo.toml	Updates test feature dependencies.

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

View changes since this review

[Copilot]

Pull request overview

Copilot reviewed 9 out of 10 changed files in this pull request and generated 2 comments.

Suppressed comments (1)

crates/tt/src/storage.rs:1042

• Removing the invisible subtree header does not remove its open/close span counts. encode_all serializes every key in this map and uses the counts to choose compact indices, so flattened repetitions retain unused span parts and can displace spans that are actually frequent. Remove both delimiter spans from the frequency map before removing the header.

            self.token_trees.remove(last_subtree_idx);

View changes since this review

[ChayimFriedman2]

I instructed Claude to review locally, it constructed a fuzzer (!) that checked 58,000 round-trip inputs with many variants (or at least that is what it said), and confirmed no mismatches were found. Well this is relaxing.

[ChayimFriedman2]

I optimized it even more a bit: by outlining the symbols as well into their own table, as was able to save more ~10mb on buck2 and ~84mb on omicron (rust-analyzer does not change). In addition, this makes the byte buffer a POD, which allows us to have faster comparison/hashing/cloning for it and also to avoid needing to use MaybeUninit to preserve provenance.