Conversation
Equality between a string and a byte-backed RedisValue fell through to (string?)x == (string?)y, which materialises the blob as a transient string before comparing. That allocation is proportional to the payload: at 64KB it is a 131KB string, over the large-object-heap threshold, on every comparison. And because the whole payload is decoded before anything is compared, a value that differs in its first byte costs exactly as much as one that matches. Answer the same question without building the string: decode a chunk at a time into a small stack buffer and compare as we go, stopping at the chunk that contains a mismatch. A bounds check first - UTF-8 yields at most one char per byte, and never more than three bytes per char - rejects mismatched lengths before touching any content. Sequence-backed values carry up to four bytes across a segment boundary, since a sequence split mid-character has to be resumed rather than treated as invalid data. The relation is deliberately unchanged: still "the blob, read as UTF-8 text, equals the string". Comparing the other way - encoding the string and matching raw bytes - looks equivalent but is not, because UTF-8 does not round-trip strings holding unpaired surrogates: "\uD800" encodes to the same bytes as "�", so the two would compare equal while GetHashCode, which hashes the decoded form, keeps them in different buckets. Equal values with different hash codes silently break Dictionary and HashSet. Measured against the local six-node figures in RedisValueEqualityBenchmarks (net10.0, string vs byte[]): 64KB, equal 43,121ns / 131,110B -> 5,397ns / 0B (8x) 64KB, differs first 57,286ns / 131,110B -> 52ns / 0B (1092x) 1KB, differs first 119ns / 2,072B -> 55ns / 0B (2.2x) 16B, equal 59ns / 56B -> 56ns / 0B Allocation is gone at every size. Small values are otherwise a wash, and two cases are marginally slower - 1KB differing at the last byte goes 138ns -> 153ns - where the per-chunk overhead lands on a path that has to decode everything anyway. Down-level targets keep the previous implementation: System.Text.Unicode.Utf8 does not exist there. The tests compare against the old behaviour directly, over text that does not survive a UTF-8 round trip (truncated, overlong and invalid sequences, surrogate pairs, lone surrogates) and, for sequences, at every possible split point.
This was referenced Sep 16, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Picks up the intent of #3116 — mixed string/blob equality allocates — but implements it so the relation, and
therefore the hash contract, is untouched. Supersedes that PR's
Equalshunk; its other two parts are alreadymoot (the author reverted the
GetHashCodechange, and theStartsWithnon-ASCII bug was fixed in0531cd54).The problem, measured
operator ==falls through to(string?)x == (string?)ywhen one side is a string and the other byte-backed,which materialises the blob as a transient string.
RedisValueEqualityBenchmarks(new here, net10.0):BlobVsBlobcontrolTwo things stand out. The allocation is proportional, so at 64KB it is a large-object-heap allocation on
every comparison — the Gen0/Gen1/Gen2 columns all read
41.6260. And a value differing in its first bytecosts the same as one that matches, because the whole payload is decoded before anything is compared.
The change
Decode a chunk at a time into a 128-char stack buffer and compare as we go, stopping at the chunk containing a
mismatch. An O(1) bounds check runs first: UTF-8 yields at most one char per byte and never more than three
bytes per char, so a mismatched length is rejected without touching content. Sequence-backed values carry up to
four bytes across a segment boundary, because a sequence split mid-character must be resumed rather than
treated as invalid data.
Allocation is gone at every size. Small values are otherwise a wash, and two cases are marginally slower —
1KB differing at the last byte goes 138 ns → 153 ns — where per-chunk overhead lands on a path that has to
decode everything regardless.
Decoding still costs ~6× a raw
memcmp(5,397 ns against the 899 nsBlobVsBlobfloor). That gap is the priceof not changing the relation, which is the next section.
Why not compare in the byte domain
Encoding the string and matching raw bytes would reach the
memcmpfloor, and it is what #3116 does. It is notthe same relation. UTF-8 does not round-trip strings holding unpaired surrogates:
GetHashCodehashes the decoded form, and says so in a comment. MakingEqualsbyte-based without changingit in the same breath puts equal values in different buckets, which silently breaks
DictionaryandHashSet.Keeping the decode-domain relation means that question never arises.
Down-level targets keep the previous implementation —
System.Text.Unicode.Utf8does not exist there.Testing
The new tests in
RedisValueEquivalencyUnitTestscompare against the old behaviour directly, asserting(asString == asBlob) == (s == (string?)asBlob)over text that does not survive a UTF-8 round trip: truncated2/3/4-byte sequences, overlong encodings, surrogates encoded as UTF-8, invalid bytes mid-payload, emoji, and
strings holding lone surrogates. For sequence-backed values this runs at every possible split point, which
is what covers the carry logic. A third test asserts the hash contract directly.
Cases are built in code rather than as
InlineDatabecause a lone surrogate survives neither a UTF-8 sourcefile nor xunit's argument serialization — and those are the interesting inputs.
Sabotaging the bounds guard fails the segmented test, so it is not passing vacuously. Full suite: 6365 passed,
0 failed. All six TFMs build with 0 warnings.
Follow-up, deliberately not here
Benchmarking hashing while I was in here turned up something bigger:
HashBlobandHashStringare within 4%of each other at every size (~31µs vs ~30µs at 64KB), so the decode is not the cost — Marvin is, at roughly
2 GB/s. Hashing a bounded prefix (and suffix, to keep common-prefix collisions expensive) would be worth
~1000× at that size, and would benefit long strings just as much as blobs. That is a separate change with a
hash-flooding tradeoff to weigh, so it is not in this PR.