perf: bulk-copy bitpacked streams instead of element-wise decode

[radmirnovii]

perf: bulk-copy bitpacked streams instead of element-wise decode

Description

DecompressDeltaBitpack{Uint16,Uint32,Uint64} convert the compressed stream from bytes to words with a per-element binary.LittleEndian loop (copyAsUints32/64). The loop is instruction-bound at ~4.5 GiB/s regardless of input size. On little-endian hosts the in-memory representation of a []uint32/[]uint64 is exactly the on-disk little-endian stream, so the conversion can be a single bulk copy (runtime.memmove), which runs at memory/cache bandwidth (30–70 GiB/s).

The implementations are split by build tag: bulk copy on every little-endian GOARCH (copy_uints.go), the element-wise reference on big-endian (copy_uints_be.go) — behavior for BE builds is unchanged. The smallest inputs (≤3 words) keep the plain loop on the LE path too: the crossover was measured (BenchmarkSmallCopyCrossover — the loop wins at 1–2 words, memmove wins from 4 on, −48% at 8). The fail-fast contract on a corrupted stream is preserved: a ragged length (not a multiple of the word size) panics, exactly as the element-wise decode did.

No format change, no assembly, CGO_ENABLED/purego paths untouched.

Open question for maintainers: seq-db ships only little-endian release targets (.goreleaser.yml) and has no big-endian CI, so the BE fallback is dead code that nothing exercises. At the same time, as far as we can tell the codebase is currently endian-correct everywhere (all serialization goes through binary.LittleEndian; the few unsafe casts are in-memory scratch reuse) — dropping the fallback would introduce the first LE-only assumption. This PR therefore preserves the status quo (package builds everywhere, BE keeps the element-wise reference). Would you prefer to drop copy_uints_be.go and let big-endian builds fail at compile time instead? Happy to do either.

Where this path runs

DecompressDeltaBitpack* is the only road from disk bytes to index values:

Call site	Block type	Hot paths above it
seqids/blocks.go (BlockMIDs.Unpack)	MID blocks (u64)	search window borders (2 unpacks per query per fraction), LID→ID resolution of results, time-series aggregation, histogram, fetch (findLIDs per fetched ID)
lids/block.go	LID postings (u32, two streams)	every leaf token on LID-block cache miss; blocks are shared containers (Block{LIDs, Offsets}) — each unpack converts two streams, and the offsets stream is tiny whenever a block is dominated by one heavy token
token/block_loader.go (u16/u32)	token blocks	GetToken/Narrow/SelectEntries/FindContains on token-block cache miss

The same loaders serve local sealed fractions, remote (S3) fractions and compaction reads.

Since the nanosecond-MID migration the compressed MID stream is ~36% of the raw block (bitlen≈23), so the conversion was a third of the MID decompression cost (12–18% of IndexSearch in profiles).

Measurements

Conversion microbenchmarks (benchstat, n=10, Cascade Lake; arm64/M1 shows the same shape, copy −84%):

	loop	memcpy	Δ
u64, 11.4 KiB (real MID stream size)	2388 ns	160 ns	−93.3% (4.5 → 67.7 GiB/s)
u64, 32 KiB (raw block)	6764 ns	955 ns	−85.9%
u32, 4 KiB (LID scale)	850 ns	31 ns	−96.3%
u32, 16 KiB	6798 ns	329 ns	−95.2%

Why not just a better loop? A safe-Go rewrite (indexed writes instead of append, variant=indexed in the benchmark) gives only −18%: any scalar loop moves 8 bytes per iteration, while memmove moves cache lines with vector registers (×12 over the indexed loop) — that is what the unsafe buys.

Full Decompress for every production shape, main vs this PR (n=10, pre-sized buffers modeling the production pools):

Benchmark	main	this PR	Δ
DecompressMIDBlock — 4096 nanosecond MIDs, bitlen≈23	8.22 µs	5.93 µs	−27.9%
DecompressLIDBlock — 65536-entry postings block	79.8 µs	56.8 µs	−28.8%
DecompressSmallBlock/n=127 — raw-residual stream	254.8 ns	39.5 ns	−84.5%
DecompressSmallBlock/n=16	45.6 ns	24.5 ns	−46.4%
DecompressSmallBlock/n=1 — 2-word stream	16.0 ns	19.6 ns	+3.6 ns (+22%)

Small streams occur in production as the offsets arrays of blocks dominated by one heavy token, the tail block of every fraction, and freshly sealed small fractions (LID blocks are shared multi-token containers, so per-token postings length does not reach this function directly). The n=1 case regresses by 3.6 ns absolute: the copy helper grew past the inlining budget (the ragged-length check and the two-path body), so the tiny-input call pays a function-call overhead — a few nanoseconds once per block unpack; no search scenario regresses end-to-end (see below); called out here for completeness rather than hidden by the geomean (−44%).

Search-level, this branch vs main (in-process harness: sealed fraction, 2M synthetic structured logs, production sealing params; n=10):

Scenario	main	this PR	Δ
needle-in-haystack, miss (trace_id:<uuid>)	38.80 µs	37.78 µs	−2.6% (p=0.001)
needle-in-haystack, hit	51.27 µs	47.26 µs	−7.8% (p=0.000)
level:error AND service:X, limit 100	382.3 µs	328.8 µs	−14.0% (p=0.000)
3-token AND	537.2 µs	525.7 µs	−2.1% (p=0.000)
count group by (2M docs)	287.5 ms	289.3 ms	+0.6% (p=0.029, noise-level)
fetch, 100 scattered IDs (findLIDs probes a different MID block per ID)	1.355 ms	1.144 ms	−15.6% (p=0.000)
fetch, 100 adjacent IDs	65.7 µs	61.8 µs	−5.9% (p=0.000)
prefix wildcard (trace_id:ab*cd*)	1.085 ms	0.979 ms	−9.8% (p=0.000)
histogram over 2M docs (bulk GetMIDs)	37.9 ms	36.7 ms	−3.1% (p=0.001)

The scattered-fetch scenario is the strongest production case: findLIDs decompresses ~1.5 MID blocks per fetched ID (54.7% of Fetch CPU), and the conversion is a third of that.

Tests

packer/copy_uints_test.go:

• differential test against the reference decode loop: 13 sizes × 8 source offsets (block payload starts at +4 after the uint32 header, so every alignment occurs) × 4 dst reuse modes;
• FuzzCopyAsUints32/64 — differential fuzzing against the reference;
• TestCopyAsUintsRaggedPanics — pins the fail-fast contract on corrupted-stream lengths;
• TestDecompressBoundarySizes — round-trip at the raw-residual/bitpacked switchover (0, 1, 63, 127, 128, 129, 255, 256, 257, 4096 values, u32 and u64);
• benchmarks: variant=loop vs variant=memcpy for the conversion, BenchmarkSmallCopyCrossover justifying the small-input threshold, plus the end-to-end Decompress* regression anchors above.

Verified additionally: full go test ./..., -race on the affected packages, cross-compilation for the release matrix (linux/darwin × amd64/arm64), wasm and big-endian (GOARCH=s390x); byte-identical query results vs main on a 57-query equivalence dump (IDs/aggregations/histogram/doc hashes) over the 2M-doc dataset.

---

• I have read and followed all requirements in CONTRIBUTING.md;
• I used LLM/AI assistance to make this pull request;

Model: Claude (Opus/Fable, agent session)
Prompt: performance research session on cache-aware layouts for seq-db search
structures; this change was found by profiling IndexSearch (phase-0 harness),
implemented and measured by the agent, reviewed by the PR author.

[cheb0]

Overall looks good to me, but I still have weird feelings about build tags solely for this micro optimization. I'd honestly redo this (see comments).

Please also note reviewing a PR fully generated with AI is a frustrating experience. Please clean up suggested places (I guess it's our job now). I'd also clean up description - looks completely unreadable to me.

[cheb0]

I have odd feelings about introducing build tags for this small optimization. Initially I did this, it just didn't get into main branch: 22a3e36#diff-aab3afc5841441065dc926e0dccdfca3010a8eb081ccc5c04bb394f902bde1d3R17

I though it's easier to just check if host is LE and then use if branch.

The main problem I see with the proposed solution is I don't know much about these platforms and build tags look like overkill for this kind of micro-optimization. I also looked through arrow-go library, haven't found any build tags, but endianess field is there in the schema.

What are you thoughts?

[cheb0]

Please clean up the comment. That's a typical huge claude comment explaining trivial things. And there a lot of questionable numbers and statements as well.

[cheb0]

We can remove this smallCopyWords, I'd generally expect we pass large blocks (128 bytes or more) through this function.

[cheb0]

The file named copy_uints.go. Do you think it's a good name for this particular file? I'd personally put it somewhere to utils package and named copy_utils.go

[cheb0]

Same for comments, please simplify.

[cheb0]

len(src) / sizeOfUint64?

[cheb0]

Why do we need a comment here? Look weird :)

[cheb0]

I'd remove these benchmarks. Probably most benchmarks in this file. The problem is we actually run all microbenchmarks on each commit so that we can track numbers in time. However, these benches do not carry any useful info/insights.

Our project already has such non-informative benches, so yes, there is work to do.

Besides, everybody know memcpy should be faster than hand-rolled loop unless the theoretical compiler targets this particular optimization.