decompress: sink nextState/nbBits loads in ZSTD_decodeSequence - #4771
Open
uarif1 wants to merge 1 commit into
Open
decompress: sink nextState/nbBits loads in ZSTD_decodeSequence#4771uarif1 wants to merge 1 commit into
uarif1 wants to merge 1 commit into
Conversation
The generic path loads all twelve fields of the three ZSTD_seqSymbol
entries up front, but the six nextState/nbBits values are not consumed
until the final FSE state update. Keeping them live across the whole
offset, matchLength and litLength decode costs six registers, and gcc
11.5.0 -O3 loses that fight: two of the three nextState values are
spilled and reloaded per sequence in the hot loop of
ZSTD_decompressSequences_bmi2().
mov %bx,0x20(%rsp)
...
mov %bx,0x28(%rsp)
...
movzwl 0x20(%rsp),%ecx
...
movzwl 0x28(%rsp),%ecx
Read them at the point of use instead, turning each spill/reload pair
back into an L1 hit on a line already touched for nbAdditionalBits and
baseValue.
This is an identity transform. llDInfo, mlDInfo and ofDInfo point into
dctx->{LL,ML,OF}Tptr, which is not written during the decode loop, and
each load is an argument expression sequenced before its own state
store. Only the generic arm changes; the __aarch64__ arm reads DInfo
through ZSTD_memcpy'd stack copies specifically to get one 64-bit load
per entry, and that is left alone.
Measured on an AMD EPYC 9D85, gcc 11.5.0 -O3, decompressing in memory
with a reused DCtx. The four spill/reload instructions are gone, and
instructions retired over a 128 KiB-block text corpus drop 3.1%
(5.123G -> 4.963G). Throughput, best of 5 interleaved runs:
corpus payload before after
text 16 KiB 1524.1 1573.4 MB/s +3.2%
text 128 KiB 2414.8 2471.2 MB/s +2.3%
JSON logs 128 KiB 1794.0 1830.4 MB/s +2.0%
4 KiB payloads are flat, as expected: there the per-block header decode
dominates over the sequence loop. The library shrinks by 584 bytes of
text.
Signed-off-by: Usama Arif <usama.arif@linux.dev>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The generic path loads all twelve fields of the three ZSTD_seqSymbol entries up front, but the six nextState/nbBits values are not consumed until the final FSE state update. Keeping them live across the whole offset, matchLength and litLength decode costs six registers, and gcc 11.5.0 -O3 loses that fight: two of the three nextState values are spilled and reloaded per sequence in the hot loop of ZSTD_decompressSequences_bmi2().
Read them at the point of use instead, turning each spill/reload pair back into an L1 hit on a line already touched for nbAdditionalBits and baseValue.
This is an identity transform. llDInfo, mlDInfo and ofDInfo point into dctx->{LL,ML,OF}Tptr, which is not written during the decode loop, and each load is an argument expression sequenced before its own state store. Only the generic arm changes; the aarch64 arm reads DInfo through ZSTD_memcpy'd stack copies specifically to get one 64-bit load per entry, and that is left alone.
Measured on an AMD EPYC 9D85, gcc 11.5.0 -O3, decompressing in memory with a reused DCtx. The four spill/reload instructions are gone, and instructions retired over a 128 KiB-block text corpus drop 3.1% (5.123G -> 4.963G). Throughput, best of 5 interleaved runs:
4 KiB payloads are flat, as expected: there the per-block header decode dominates over the sequence loop. The library shrinks by 584 bytes of text.