Skip to content

Add located root-array streaming with per-element source spans (1.8.0-dev) - #78

Merged
ThorFuchs merged 1 commit into
mainfrom
feature/located-top-level-array
Nov 9, 2026
Merged

ThorFuchs merged 1 commit into
mainfrom
feature/located-top-level-array

Conversation

@ThorFuchs

Copy link
Copy Markdown
Owner

Summary

Adds an additive public JSONStreaming API for the next minor line (1.8.0-dev):
streamed root-array elements together with their original source location.

public enum JSONLocatedTopLevelArrayEvent: Sendable {
    case element(index: UInt64, value: JSON, sourceSpan: JSONSourceSpan)
    case documentCompleted(JSONStreamSummary)
}
public struct JSONLocatedTopLevelArraySequence<Source: AsyncSequence>: AsyncSequence
    where Source.Element == JSONByteChunk
  • sourceSpan is a zero-based, half-open range of UTF-8 byte offsets in the complete stream.
    It covers the element token or subtree only; whitespace and separators are excluded. Offsets
    come from the shared token machine, so they don't depend on how the source splits chunks.
  • Spans are provisional until documentCompleted, like the elements themselves. The reader
    keeps no copy of the original bytes. A consumer that needs raw slices keeps its own immutable
    source and applies the span to it.
  • One grammar, one driver. JSONArrayStreamDriver goes from private to internal and gains
    elementStart tracking plus nextLocated(). The existing next() delegates to it, so both
    facades share strict grammar, per-element and lifetime quotas, error values, statistics,
    cancellation and the completion boundary. JSONTopLevelArraySequence keeps its public API and
    behavior.
  • No new product, dependency, platform or deployment change. Thirteen products are unchanged.

Why / consumer

Chat Library's ProviderExport01 (private app) vendors a fork of Swift-JSON
(ProviderExport01/Vendor/Swift-JSON) only because it needs a source span for each record in
large provider exports. A comparison against 1.7.1 found that this API is the fork's only private
delta. It depends on the internal stream driver, so it can't be built outside the package. With
this in a tagged 1.8.0, Chat Library can drop the vendored tree and depend on
.package(url: "https://github.com/ThorFuchs/Swift-JSON.git", exact: "1.8.0"). The code here
matches that private candidate's API (1e950354), with full DocC comments added.

Overlap with other 1.8 work

  • Prepare 1.9 bounded store workflows and Transform selection #74 (codex/1.8-selection-main-qa) and Add bounded structural batches and read-only stores for 1.8 #71 (codex/1.8-batches-and-store) don't touch
    Sources/JSONStreaming.
  • Expected textual conflicts are limited to shared inventories. Each needs a simple merge:
    • CHANGELOG.md: both add ## Unreleased — 1.8.0-dev. Merge the bullets under one heading.
    • Documentation/Qualification/TestDiscovery.json: counts are additive.
    • Scripts/testing_events.py: this PR adds one parameterized entry.
    • Sources/JSONCLIEngine/CLIBuildIdentity.swift: regenerate with
      python3 Scripts/generate_cli_identity.py after merging.
  • CLIReleaseVersion stays at 1.7.1. The version bump, tag and release are left to the
    maintainer.

Tests

New Tests/JSONStreamingTests/LocatedTopLevelArrayTests.swift. It imports only JSON and
JSONStreaming, so it tests as a public consumer. It adds 6 functions and 18 parameterized cases:

  • exact spans for scalars, nested objects and arrays, leading and trailing document whitespace,
    tabs and CRLF, and separator spacing. Each input runs with 1-byte chunks and with a two-chunk
    split at every byte position, and every slice re-parses to the yielded value
  • UTF-8 byte (not character or scalar) offsets: 2-, 3- and 4-byte scalars, surrogate escapes,
    Cyrillic keys and combining sequences
  • empty arrays ([], with whitespace) yield only completion with records == 0
  • parity with JSONTopLevelArraySequence on 18 valid and malformed inputs at every split: the
    same JSONFailure, indices, values, records and statistics, and no completion after a failure
  • the same limit failures (maxDepth, totalRecords, nodes, arrayItems, chunkBytes), plus
    per-element quota reset and lifetime counters over a 10,000-element generated array
  • the value-only reader's output is unchanged and its statistics match the located reader

Inventories updated: TestDiscovery.json (JSONStreamingTests 30→36 on native, portable and WASI
lists; parameterized invocations 390→408 and WASI 353→371) and Scripts/testing_events.py.

Local results

Source: de1fabb2c34ff7310dbf20165ecdbb474d4ef342, macOS 27 / SDK 27.2, Xcode toolchain Apple Swift 6.4.2
(swiftlang-6.4.2.1.9), Swift language mode 6.

Command Exit Result
python3 Scripts/ci.py --lane native 0 passed; 58/58 commands. Debug and release swift test both ran 14 bundles with counts [10, 16, 16, 23, 34, 36, 47, 57, 79, 88, 97, 98, 190, 190] and 408 parameterized cases. CLI, ownership, Transform, schema-examples, DSL, navigation and qualification sub-gates also passed. source_inputs_unchanged: true
python3 Scripts/ci.py --lane docs 0 passed; DocC built with warnings as errors, and 1509/1509 authored public symbols are documented

Host note: the swiftly 6.4.0 toolchain on this machine was removed during the session. The lanes
above ran with ~/.swiftly/bin taken off PATH (Xcode /usr/bin/swift) and with
SDKROOT=MacOSX27.2.sdk. Without that, the bare swiftc probes in qualify_cli_ownership.py and
qualify_navigation.py pick a mismatched compiler or the Command Line Tools SDK 27.0. This is a
host setup problem, not caused by this change.

Not run: safety, linux, linux-safety, wasm and archive lanes, and performance
measurement. The LimitsMatrix.json accounting-site line numbers for
JSONTopLevelArraySequence.swift were already out of date before this change; no script checks
them, and they're left as they were.

Docs

  • DocC: JSONStreaming.md (overview paragraph and topics) and CompletionAndBackpressure.md
    (reader table row and a new "Locate root-array elements" section with an example).
  • Documentation/Architecture/Streaming.md and API.md.
  • CHANGELOG.md: a new Unreleased — 1.8.0-dev entry.

🤖 Generated with Claude Code

Add JSONLocatedTopLevelArraySequence and JSONLocatedTopLevelArrayEvent to
JSONStreaming for 1.8.0-dev. Each root-array element also reports its
JSONSourceSpan: zero-based half-open UTF-8 byte offsets in the complete
stream, excluding whitespace and separators, independent of chunking.

Both root-array facades share the one internal JSONArrayStreamDriver, so
grammar, quotas, errors, statistics and completion are identical; the
value-only next() delegates to nextLocated(). No bytes are retained.

Add public-consumer tests for spans, UTF-8, chunk splits, empty arrays,
facade parity on errors/limits and unchanged value-only behavior. Update
DocC, architecture notes, CHANGELOG, test discovery counts, the
parameterized-case registry and the regenerated CLI source identity.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@ThorFuchs
ThorFuchs merged commit 2fa0ce3 into main Nov 9, 2026
13 of 14 checks passed
@ThorFuchs
ThorFuchs deleted the feature/located-top-level-array branch November 9, 2026 18:02
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant