Add opt-in RFC 9839 Unicode repertoire validation

[ydah]

JSON strings can contain escaped NUL, legacy controls, and noncharacters even when the document is syntactically valid. Applications whose protocols restrict these characters currently need their own checks across nested keys and values. RFC 9839 §5 describes applying Unicode repertoire restrictions to JSON data, including characters represented by escapes.

Changes

This adds unicode_subset and on_invalid_char options to parsing and generation. Applications can enforce the same policy on incoming and outgoing data, report violations at the JSON boundary, or explicitly replace prohibited characters. The defaults, unicode_subset: nil and on_invalid_char: :raise, preserve existing behavior.

The repertoires follow RFC 9839 §4. All three allow tab, LF, CR, unassigned code points, and private-use code points.

Usage

source = '{"name":"alice\u0000"}'

# Existing behavior remains available.
JSON.parse(source)
# => {"name"=>"alice\u0000"}

# Enforce a repertoire in both directions.
coder = JSON::Coder.new(unicode_subset: :assignables)

coder.load(source)
# raises JSON::ParserError: U+0000 is not allowed ...
# line: 1, column: 15, json_path: "$.name"

coder.dump({ name: "alice\u0000" })
# raises JSON::GeneratorError: U+0000 is not allowed ...

# Alternatively, explicitly accept replacement.
JSON.parse(source, unicode_subset: :assignables, on_invalid_char: :replace)
# => {"name"=>"alice�"}

Validation and edge cases

Validation covers object keys and string values after escape decoding and before output escaping. With a subset selected, replacement handles malformed UTF-8 by maximal subpart and preserves input strings. Distinct keys can become equal after replacement; leave allow_duplicate_key: false (the default) to reject these collisions.

With no subset selected, :replace only relaxes parsing of unpaired surrogate escapes; it does not enable general character replacement. Generation rejects the combination of on_invalid_char: :replace and rfc8785: true. Raw JSON::Fragment contents and custom to_json output remain the caller's responsibility.

Performance

Benchmarks: Measured on CRuby 4.0.6, arm64 macOS, using benchmark/data/twitter.json (631,514 bytes). The baseline is cdcfdb8; the implementation is 66de252.

Each sample measures 1,000 operations after 300 warmup iterations. File loading, input preparation, and Coder construction are excluded. Execution order alternates between rounds, with GC.start before each sample.

The :assignables measurements use the default on_invalid_char: :raise on input that satisfies the repertoire. Replacement and error handling are not timed.

Default behavior

Default behavior:

Operation	Baseline (s / 1,000 calls)	This PR (s / 1,000 calls)	Change
JSON.parse	0.596201	0.596832	+0.11%
JSON.generate	0.228508	0.227185	−0.58%
JSON::Coder#dump	0.228294	0.228336	+0.02%

Unicode repertoire validation

Enabling unicode_subset: :assignables:

Operation	Default (s / 1,000 calls)	:assignables (s / 1,000 calls)	Overhead
JSON.parse	0.596832	0.672650	+12.70%
JSON.generate	0.227185	0.256503	+12.90%
JSON::Coder#dump	0.225009	0.253952	+12.86%

Measurement methodology

Times are medians of elapsed seconds per 1,000 operations, using every sample in each reported comparison. Percentages are calculated as (after_median / before_median - 1) * 100, using the unrounded medians; negative values mean faster execution. Parsing uses 72 paired rounds (eight runs of nine); generation uses nine. The enabled Coder#dump comparison was measured in a separate nine-round run, which accounts for its different default timing.

Benchmark reproduction

Benchmark code: baseline/default comparison and JSON.generate validation

Build both checkouts with the same Ruby and save these two files together.

worker.rb:

require 'json'

$stdout.sync = true
source = File.read(ARGV.fetch(0))
object = JSON.parse(source)

operations = {
  'parse' => -> { JSON.parse(source) },
  'generate' => -> { JSON.generate(object) },
}

if ARGV.fetch(2) == 'generate'
  coder = JSON::Coder.new
  operations['generate_coder'] = -> { coder.dump(object) }
end

if ARGV.fetch(1) == 'current'
  operations['parse_enabled'] =
    -> { JSON.parse(source, unicode_subset: :assignables) }
  operations['generate_enabled'] =
    -> { JSON.generate(object, unicode_subset: :assignables) }
end

operations.each_value { |operation| 300.times { operation.call } }
puts 'ready'

while line = $stdin.gets
  name, count = line.split
  operation = operations.fetch(name)
  GC.start
  start = Process.clock_gettime(Process::CLOCK_MONOTONIC)
  count.to_i.times { operation.call }
  puts Process.clock_gettime(Process::CLOCK_MONOTONIC) - start
end

compare.py:

import json, os, statistics, subprocess, sys
from pathlib import Path

baseline, current, dataset, mode, runs = sys.argv[1:]
ruby = os.environ.get("RUBY", "ruby")
worker_script = str(Path(__file__).with_name("worker.rb"))

if mode == "parse":
    cases = [
        ("baseline", "parse"),
        ("current", "parse"),
        ("current", "parse_enabled"),
        ("baseline", "generate"),
        ("current", "generate"),
        ("current", "generate_enabled"),
    ]
    comparisons = [
        ("default", "current_parse", "baseline_parse"),
        ("enabled", "current_parse_enabled", "current_parse"),
    ]
else:
    assert mode == "generate"
    cases = [
        ("baseline", "generate"),
        ("current", "generate"),
        ("baseline", "generate_coder"),
        ("current", "generate_coder"),
        ("current", "generate_enabled"),
    ]
    comparisons = [
        ("default", "current_generate", "baseline_generate"),
        ("coder_default", "current_generate_coder", "baseline_generate_coder"),
        ("enabled", "current_generate_enabled", "current_generate"),
    ]

samples = {name + "_" + op: [] for name, op in cases}

for _ in range(int(runs)):
    workers = {}
    try:
        for name, root in [("baseline", baseline), ("current", current)]:
            worker = subprocess.Popen(
                [ruby, "-I" + str(Path(root) / "lib"),
                 worker_script, dataset, name, mode],
                stdin=subprocess.PIPE,
                stdout=subprocess.PIPE,
                text=True,
            )
            workers[name] = worker
            assert worker.stdout.readline().strip() == "ready"

        for round in range(9):
            order = cases if round % 2 == 0 else reversed(cases)
            for name, op in order:
                worker = workers[name]
                worker.stdin.write(op + " 1000\n")
                worker.stdin.flush()
                samples[name + "_" + op].append(
                    float(worker.stdout.readline())
                )
    finally:
        for worker in workers.values():
            worker.stdin.close()
            worker.wait()

medians = {k: statistics.median(v) for k, v in samples.items()}
print(json.dumps({
    "samples": samples,
    "median_seconds_per_1000": medians,
    "change_percent": {
        label: (medians[num] / medians[den] - 1) * 100
        for label, num, den in comparisons
    },
}, indent=2))

Run eight sets of nine rounds for parsing and one set for generation:

export RUBY=/path/to/ruby

python3 compare.py \
  /path/to/baseline /path/to/branch /path/to/twitter.json parse 8

python3 compare.py \
  /path/to/baseline /path/to/branch /path/to/twitter.json generate 1

Benchmark code: additional Coder#dump validation comparison

coder.rb:

require 'json'

source = File.binread(ARGV.fetch(0))
object = JSON.parse(source)
coders = {
  default: JSON::Coder.new,
  assignables: JSON::Coder.new(unicode_subset: :assignables),
}

raise 'Output differs' unless
  coders[:default].dump(object) == coders[:assignables].dump(object)

begin
  coders[:assignables].dump("\u0089")
  raise 'Repertoire validation is not active'
rescue JSON::GeneratorError
end

coders.each_value { |coder| 300.times { coder.dump(object) } }
samples = coders.transform_values { [] }

9.times do |round|
  order = round.even? ? coders.keys : coders.keys.reverse
  order.each do |name|
    coder = coders.fetch(name)
    GC.start
    start = Process.clock_gettime(Process::CLOCK_MONOTONIC)
    1000.times { coder.dump(object) }
    samples.fetch(name) <<
      Process.clock_gettime(Process::CLOCK_MONOTONIC) - start
  end
end

median = ->(values) { values.sort.fetch(values.size / 2) }
medians = samples.transform_values { |values| median.call(values) }

puts JSON.pretty_generate({
  samples: samples,
  median_seconds_per_1000: medians,
  overhead_percent: (medians[:assignables] / medians[:default] - 1) * 100,
})

export RUBY=/path/to/ruby

"$RUBY" -I/path/to/branch/lib coder.rb /path/to/twitter.json

[byroot]

Applications whose protocols restrict these characters currently need their own checks across nested keys and values.

Do you need this in one of your application? Isn't it simpler and faster to apply this restriction on the whole JSON payload prior to parsing?