Skip to content

decompress: sink nextState/nbBits loads in ZSTD_decodeSequence - #4771

Open
uarif1 wants to merge 1 commit into
facebook:devfrom
uarif1:zstd_register_skip
Open

decompress: sink nextState/nbBits loads in ZSTD_decodeSequence#4771
uarif1 wants to merge 1 commit into
facebook:devfrom
uarif1:zstd_register_skip

Conversation

@uarif1

@uarif1 uarif1 commented Sep 8, 2026

Copy link
Copy Markdown

The generic path loads all twelve fields of the three ZSTD_seqSymbol entries up front, but the six nextState/nbBits values are not consumed until the final FSE state update. Keeping them live across the whole offset, matchLength and litLength decode costs six registers, and gcc 11.5.0 -O3 loses that fight: two of the three nextState values are spilled and reloaded per sequence in the hot loop of ZSTD_decompressSequences_bmi2().

mov    %bx,0x20(%rsp)
...
mov    %bx,0x28(%rsp)
...
movzwl 0x20(%rsp),%ecx
...
movzwl 0x28(%rsp),%ecx

Read them at the point of use instead, turning each spill/reload pair back into an L1 hit on a line already touched for nbAdditionalBits and baseValue.

This is an identity transform. llDInfo, mlDInfo and ofDInfo point into dctx->{LL,ML,OF}Tptr, which is not written during the decode loop, and each load is an argument expression sequenced before its own state store. Only the generic arm changes; the aarch64 arm reads DInfo through ZSTD_memcpy'd stack copies specifically to get one 64-bit load per entry, and that is left alone.

Measured on an AMD EPYC 9D85, gcc 11.5.0 -O3, decompressing in memory with a reused DCtx. The four spill/reload instructions are gone, and instructions retired over a 128 KiB-block text corpus drop 3.1% (5.123G -> 4.963G). Throughput, best of 5 interleaved runs:

corpus     payload    before     after
text        16 KiB  1524.1     1573.4 MB/s   +3.2%
text       128 KiB  2414.8     2471.2 MB/s   +2.3%
JSON logs  128 KiB  1794.0     1830.4 MB/s   +2.0%

4 KiB payloads are flat, as expected: there the per-block header decode dominates over the sequence loop. The library shrinks by 584 bytes of text.

The generic path loads all twelve fields of the three ZSTD_seqSymbol
entries up front, but the six nextState/nbBits values are not consumed
until the final FSE state update. Keeping them live across the whole
offset, matchLength and litLength decode costs six registers, and gcc
11.5.0 -O3 loses that fight: two of the three nextState values are
spilled and reloaded per sequence in the hot loop of
ZSTD_decompressSequences_bmi2().

    mov    %bx,0x20(%rsp)
    ...
    mov    %bx,0x28(%rsp)
    ...
    movzwl 0x20(%rsp),%ecx
    ...
    movzwl 0x28(%rsp),%ecx

Read them at the point of use instead, turning each spill/reload pair
back into an L1 hit on a line already touched for nbAdditionalBits and
baseValue.

This is an identity transform. llDInfo, mlDInfo and ofDInfo point into
dctx->{LL,ML,OF}Tptr, which is not written during the decode loop, and
each load is an argument expression sequenced before its own state
store. Only the generic arm changes; the __aarch64__ arm reads DInfo
through ZSTD_memcpy'd stack copies specifically to get one 64-bit load
per entry, and that is left alone.

Measured on an AMD EPYC 9D85, gcc 11.5.0 -O3, decompressing in memory
with a reused DCtx. The four spill/reload instructions are gone, and
instructions retired over a 128 KiB-block text corpus drop 3.1%
(5.123G -> 4.963G). Throughput, best of 5 interleaved runs:

    corpus     payload    before     after
    text        16 KiB  1524.1     1573.4 MB/s   +3.2%
    text       128 KiB  2414.8     2471.2 MB/s   +2.3%
    JSON logs  128 KiB  1794.0     1830.4 MB/s   +2.0%

4 KiB payloads are flat, as expected: there the per-block header decode
dominates over the sequence loop. The library shrinks by 584 bytes of
text.

Signed-off-by: Usama Arif <usama.arif@linux.dev>
@meta-cla meta-cla Bot added the CLA Signed label Sep 8, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant