Skip to content

Detect RFC 8785 duplicate keys that differ only in encoding - #1108

Closed
Shaisolaris wants to merge 1 commit into
ruby:masterfrom
Shaisolaris:rfc8785-duplicate-keys-across-encodings
Closed

Shaisolaris wants to merge 1 commit into
ruby:masterfrom
Shaisolaris:rfc8785-duplicate-keys-across-encodings

Conversation

@Shaisolaris

Copy link
Copy Markdown

With rfc8785: true, a Hash whose String keys have the same text in different encodings is written with that key twice:

latin1 = "\u00e9".encode(Encoding::ISO_8859_1)
JSON.generate({ latin1 => 1, "\u00e9" => 2 }, rfc8785: true)
# => {"é":1,"é":2}

The two keys are distinct Hash keys, but both are written as the UTF-8 string "é", so the canonical output has a duplicate property name. RFC 8785 output is I-JSON, which does not allow duplicate names.

The sort proc already converts keys to UTF-8 and checks for duplicates when an as_json hook is given. It now does the same when a String or Symbol key is not ASCII or valid UTF-8, even without as_json, so this case raises the usual "detected duplicate key" GeneratorError. Hashes whose keys are all ASCII or UTF-8 still skip the conversion.

Tests: added test_rfc8785_duplicate_keys_in_different_encodings to test/json/json_generator_test.rb. It fails on master and passes with the change. Locally (CRuby 3.3, C extension) the rfc8785 tests and the 11 test files under test/json that I ran pass.

@byroot

byroot commented Oct 12, 2026

Copy link
Copy Markdown
Member

Not very convinced by this. String hash keys not encoded in UTF-8 are supposed to be invalid. We should just raise.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants