Skip to content

Raise on non-UTF-8 string keys in RFC 8785 mode - #1109

Open
Shaisolaris wants to merge 1 commit into
ruby:masterfrom
Shaisolaris:rfc8785-raise-non-utf8-keys
Open

Shaisolaris wants to merge 1 commit into
ruby:masterfrom
Shaisolaris:rfc8785-raise-non-utf8-keys

Conversation

@Shaisolaris

Copy link
Copy Markdown

Follow-up to #1108. As suggested there, string keys that are not UTF-8 are treated as invalid in RFC 8785 mode, so the generator now raises instead of transcoding them.

On master:

latin1 = "\xE9".dup.force_encoding(Encoding::ISO_8859_1)
JSON.generate({ latin1 => 1, "é" => 2 }, rfc8785: true)
# => "{\"é\":1,\"é\":2}"   (duplicate key in canonical output)
JSON.generate({ "\xC3\xA9".b => 1 }, rfc8785: true)
# => Encoding::UndefinedConversionError (from the key sort, not JSON::GeneratorError)
JSON.generate({ "\xFF".dup.force_encoding("UTF-8") => 1 }, rfc8785: true)
# => Encoding::InvalidByteSequenceError

With this change, all three raise JSON::GeneratorError. The checks are in rfc8785_sort_keys_proc, on the path where no as_json conversion is used:

  • ASCII-only keys are still accepted in any ASCII-compatible encoding (for example "a".b, US-ASCII strings, and Symbols).
  • Keys in encodings that are not ASCII-compatible, such as UTF-16LE, are rejected even when their content is ASCII. Before, { "a".encode("UTF-16LE") => 1 } produced {"a":1}.
  • Keys that go through as_json (with strict: true) are converted as before.
  • Only rfc8785: true is affected. Plain JSON.generate keeps its current behavior.

Tests:

  • Added test_rfc8785_non_utf8_string_keys. It fails on master and passes with this change.
  • All test/json/*_test.rb files pass locally with the C extension on Ruby 3.3.8, with the same counts as master apart from the new test. I did not run the TruffleRuby or JRuby generators or the CI matrix.

String hash keys that are not UTF-8 are invalid, so with rfc8785: true
the generator now raises JSON::GeneratorError for them instead of
transcoding them.  A Latin-1 key used to be written the same as the
UTF-8 key with the same characters, so the output could contain a
duplicate key.  Binary and invalid UTF-8 keys raised Encoding errors
from the sort instead of GeneratorError.

ASCII-only keys are still accepted in any ASCII-compatible encoding.
Keys handed to as_json are converted as before.

See ruby#1108.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant