Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
11 changes: 11 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
@@ -1,5 +1,16 @@
# Changelog

## Unreleased — 1.8.0-dev

- Add `JSONLocatedTopLevelArraySequence` and `JSONLocatedTopLevelArrayEvent` to
JSONStreaming. Each root-array element also reports its `JSONSourceSpan`:
zero-based half-open UTF-8 byte offsets in the complete stream, excluding
whitespace and separators, independent of chunk boundaries. Spans stay
provisional until `documentCompleted`; no original bytes are retained.
- Both root-array facades share one strict grammar and pull driver, so quotas,
errors, statistics and completion are identical. `JSONTopLevelArraySequence`
behavior and all existing products, dependencies and platforms are unchanged.

## 1.7.1 — prepared source; publication separate

- Fix packaged Draft 2020-12 resource discovery when SwiftPM's native test host
Expand Down
2 changes: 1 addition & 1 deletion Documentation/Architecture/API.md
Original file line number Diff line number Diff line change
Expand Up @@ -30,7 +30,7 @@ and executed by public consumers.
| JSONBridges | BinaryInteger.jsonNumber, FixedWidthInteger/Double/Decimal.init(jsonNumber:), Double.jsonNumberFromRoundTripText | Optional one-delegate numeric conversions; explicit fidelity, no new authority |
| JSONSchema | JSONSchemaDocument, JSONSchemaRegistry, JSONSchemaCompiler, CompiledJSONSchema, JSONSchemaValidator | Offline immutable plan, invocation-owned evaluation and private native regex |
| JSONSchemaExamples | JSONSchemaExampleGenerator, JSONSchemaExampleResult | Conservative deterministic witnesses certified by the existing native validator; separate finite preparation and cumulative generation |
| JSONStreaming | JSONEventSequence, JSONValueSequence, NDJSONValueSequence, JSONTopLevelArraySequence, JSONAsyncWriter | Exclusive pull state, shared codec, strict framing and awaited handoffs |
| JSONStreaming | JSONEventSequence, JSONValueSequence, NDJSONValueSequence, JSONTopLevelArraySequence, JSONLocatedTopLevelArraySequence, JSONAsyncWriter | Exclusive pull state, shared codec, strict framing and awaited handoffs |
| JSONPatch | JSONPatch, JSONPatchOperation, JSONPatchResult, JSONMergePatch | Transaction staging, exact comparison and iterative editing |
| swift-json | format, validate, pointer, patch, query, stream, schema, render | Bounded process/file adapters; the public Transform facade and CLI use the same private query compiler/VM |
| JSONHTML | JSONHTMLRenderer, JSONHTMLRenderingOptions, JSONHTMLLimits, JSONHTMLRenderReport, JSONHTMLAnnotation, JSONExportDocument, JSONExportOptions, JSONExportReport | Structured forms/cards and bounded native previews, exact HTML, native Apple transfer, immutable prepared JSON/text/Markdown/static HTML/opt-in styled RTF and cumulative budgets |
Expand Down
2 changes: 1 addition & 1 deletion Documentation/Architecture/Streaming.md
Original file line number Diff line number Diff line change
Expand Up @@ -28,7 +28,7 @@ Large-stream tests use a generator that allocates one admitted chunk at a time a

## Root-array streaming

JSONTopLevelArraySequence returns an explicit array stream event: element(index, value) and documentCompleted(summary). It consumes one root array through the shared strict token machine and materializes only the current element. It has no selector language.
JSONTopLevelArraySequence returns an explicit array stream event: element(index, value) and documentCompleted(summary). It consumes one root array through the shared strict token machine and materializes only the current element. It has no selector language. JSONLocatedTopLevelArraySequence uses the same driver and yields element(index, value, sourceSpan) with zero-based half-open UTF-8 byte offsets in the complete stream; it retains no original bytes, and spans are provisional until documentCompleted.

This serves large export files and datasets whose records are array elements rather than separate NDJSON lines. Each yielded value is a complete valid JSON subtree; the enclosing document is not certified until the closing bracket, trailing whitespace and EOF have all been checked. A malformed tail throws after any earlier yields. Early termination or cancellation never produces documentCompleted.

Expand Down
10 changes: 5 additions & 5 deletions Documentation/Qualification/TestDiscovery.json
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@
"JSONTests": 190,
"JSONSchemaTests": 79,
"JSONRegexRuntimeTests": 16,
"JSONStreamingTests": 30,
"JSONStreamingTests": 36,
"JSONPatchTests": 16,
"JSONIntegrationTests": 34,
"JSONHTMLTests": 97,
Expand All @@ -16,7 +16,7 @@
"JSONTransformTests": 23,
"JSONDSLTests": 47
},
"parameterized_invocations": 390,
"parameterized_invocations": 408,
"composition_external_functions": 5,
"qa_functions": {
"JSONTests": 82,
Expand Down Expand Up @@ -76,7 +76,7 @@
"JSONTests": 190,
"JSONSchemaTests": 79,
"JSONRegexRuntimeTests": 16,
"JSONStreamingTests": 30,
"JSONStreamingTests": 36,
"JSONPatchTests": 16,
"JSONIntegrationTests": 34,
"JSONHTMLTests": 81,
Expand All @@ -92,7 +92,7 @@
"JSONTests": 190,
"JSONSchemaTests": 79,
"JSONRegexRuntimeTests": 16,
"JSONStreamingTests": 30,
"JSONStreamingTests": 36,
"JSONPatchTests": 16,
"JSONIntegrationTests": 34,
"JSONHTMLTests": 81,
Expand All @@ -103,7 +103,7 @@
"JSONTransformTests": 23,
"JSONDSLTests": 47
},
"wasi_parameterized_invocations": 353,
"wasi_parameterized_invocations": 371,
"portable_shared_state_functions": {
"JSONTests": 4,
"JSONSchemaTests": 3,
Expand Down
2 changes: 1 addition & 1 deletion Scripts/testing_events.py
Original file line number Diff line number Diff line change
Expand Up @@ -19,7 +19,7 @@
'strictArrayIndexGrammar(text:)': 5, 'malformedUTF8(bytes:)': 4,
'invalidPointerEscapes(text:)': 4, 'invalidCancellationConfiguration(quantum:)': 3,
'strictMalformedInput(text:)': 10, 'negativeZeroHasOneIdentity(text:)': 3,
'everyByteSplitUsesIdenticalGrammar(text:)': 4, 'malformedRootArrayNeverCompletes(text:)': 6,
'everyByteSplitUsesIdenticalGrammar(text:)': 4, 'malformedRootArrayNeverCompletes(text:)': 6, 'locatedAndValueOnlyFacadesAgreeOnEveryOutcome(text:)': 18,
'streamEverySplitAgreesWithNativeParser(text:)': 3,
'successfulAlternativesCannotSwallowRegexExhaustion(keyword:)': 3,
'URIIdentityUsesSyntaxNormalizationWithoutUnicodeFolding(input:expected:)': 9,
Expand Down
2 changes: 1 addition & 1 deletion Sources/JSONCLIEngine/CLIBuildIdentity.swift
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,6 @@
// Generated by Scripts/generate_cli_identity.py; excludes this file.
// SHA-256 of the ordered Package.swift + Sources content manifest, not a Git commit.
package enum CLIBuildIdentity {
package static let revision = "sha256:cc33adc50dbf28731ccac0919bfaf10716e459d3e69ffdd05189301d1d13b2f9"
package static let revision = "sha256:9d42eb6dbc4c0e23e7329ae1b3c278421d996c7446638e46cc5c1a07ab1240ac"
package static let kind = "source-sha256"
}
Original file line number Diff line number Diff line change
Expand Up @@ -10,6 +10,7 @@ Choose framing, consume through its certified boundary and await output acceptan
| JSONValueSequence | Complete values | Configured single-document or whitespace boundary |
| NDJSONValueSequence | Complete physical records | LF/CRLF or valid final record at EOF |
| JSONTopLevelArraySequence | Provisional indexed elements, then completion | Outer closing array, trailing whitespace and EOF |
| JSONLocatedTopLevelArraySequence | Provisional indexed elements with source spans, then completion | Outer closing array, trailing whitespace and EOF |

Each reader uses owned ``JSONByteChunk`` values from a caller AsyncSequence.
An iterator owns its mutable state and must be advanced serially. No detached
Expand All @@ -32,6 +33,33 @@ results until completion. Consumers choosing incremental acceptance own their
persistence rules. A single-document JSONValueSequence instead waits until EOF
before yielding its whole materialized root.

## Locate root-array elements

``JSONLocatedTopLevelArraySequence`` shares the root-array grammar, pull driver,
quotas, errors and completion boundary with ``JSONTopLevelArraySequence``. Each
``JSONLocatedTopLevelArrayEvent/element(index:value:sourceSpan:)`` also carries
the element's zero-based half-open UTF-8 byte span in the complete stream,
including any leading document whitespace before the array. Whitespace and
separators between elements are excluded; a container's span covers its whole
subtree. Offsets count encoded bytes, not characters or decoded scalars, and do
not depend on how the source splits chunks.

The reader owns no copy of the original bytes. A consumer that needs a record's
raw slice retains an immutable source separately and applies the span to it.
Like the element itself, a span stays provisional until
``JSONLocatedTopLevelArrayEvent/documentCompleted(_:)``.

```swift
for try await event in JSONLocatedTopLevelArraySequence(chunks) {
switch event {
case .element(let index, let value, let span):
stage(index, value, bytes[Int(span.start)..<Int(span.end)])
case .documentCompleted:
commitStagedRecords()
}
}
```

Core quotas apply per root-array element; maxDepth includes the outer container.
Iterator-lifetime input, work, records and totalScannedNodes remain cumulative.
State is retired at record boundaries. The selected scalar reader retains no
Expand All @@ -54,4 +82,5 @@ checks on return from caller I/O do not preempt arbitrary work inside that callb
- ``JSONValueSequence``
- ``NDJSONValueSequence``
- ``JSONTopLevelArraySequence``
- ``JSONLocatedTopLevelArraySequence``
- ``JSONAsyncWriter``
8 changes: 8 additions & 0 deletions Sources/JSONStreaming/Documentation.docc/JSONStreaming.md
Original file line number Diff line number Diff line change
Expand Up @@ -27,6 +27,12 @@ iterator-lifetime input/work/record/node counters remain cumulative. Record
state is retired at boundaries; the scalar implementation retains no reusable
scratch pool and holds only the currently admitted chunk and active element.

``JSONLocatedTopLevelArraySequence`` drives the same reader and additionally
reports each element's `JSONSourceSpan`: zero-based half-open UTF-8 byte
offsets in the complete stream, excluding surrounding whitespace and separators.
It retains no original bytes; callers that need raw slices keep their own
immutable source and treat spans as provisional until document completion.

``JSONAsyncWriter`` awaits full acceptance of each bounded chunk before the
next handoff. Cancellation/error stops further pulls/writes; bytes already
accepted by a sink cannot be recalled. Application code that requires atomic
Expand All @@ -51,6 +57,8 @@ publication must stage bytes in its own sink.
- ``NDJSONOptions``
- ``JSONTopLevelArraySequence``
- ``JSONTopLevelArrayEvent``
- ``JSONLocatedTopLevelArraySequence``
- ``JSONLocatedTopLevelArrayEvent``

### Awaited output
- ``JSONAsyncWriter``
55 changes: 55 additions & 0 deletions Sources/JSONStreaming/JSONLocatedTopLevelArraySequence.swift
Original file line number Diff line number Diff line change
@@ -0,0 +1,55 @@
// SPDX-License-Identifier: Apache-2.0
import JSON
/// A provisional root-array element with its original source span, or certified completion.
///
/// Spans are zero-based half-open UTF-8 byte offsets in the complete stream.
/// Whitespace and separators outside the element are excluded. A span does not
/// certify the enclosing array, stable backing bytes or durable admission.
public enum JSONLocatedTopLevelArrayEvent: Sendable {
/// A zero-based indexed element and its byte span; the enclosing document remains provisional.
case element(index: UInt64, value: JSON, sourceSpan: JSONSourceSpan)
/// The outer array, trailing whitespace and EOF have been certified.
case documentCompleted(JSONStreamSummary)
}
/// A pull-driven root-array reader that also reports each element's original byte span.
///
/// Uses the same strict grammar, pull driver, finite quotas and completion boundary
/// as ``JSONTopLevelArraySequence``; only the yielded event carries a location.
/// No original-byte buffer, alternate parser, prefetch task or file is owned here.
/// Callers needing raw slices must retain an immutable source separately and keep
/// each element provisional until `documentCompleted`. The framing option is ignored.
public struct JSONLocatedTopLevelArraySequence<Source: AsyncSequence>: AsyncSequence where Source.Element == JSONByteChunk {
/// The located event yielded by this sequence, as declared below.
public typealias Element = JSONLocatedTopLevelArrayEvent
let source: Source
let options: JSONStreamingOptions
/// Wraps a caller-owned source for one root array; framing is always single-document.
///
/// - Parameters:
/// - source: The caller-owned asynchronous source of owned UTF-8 chunks.
/// - options: The value configuration used by this facade or operation.
public init(_ source: Source, options: JSONStreamingOptions = .init()) { self.source = source; self.options = options }
/// Creates an independent iterator with exclusive mutable state and no prefetch task.
public func makeAsyncIterator() -> Iterator { .init(source: source.makeAsyncIterator(), limits: options.limits) }
/// Exclusive pull iterator; never call next concurrently.
///
/// Each iterator owns its budget, parsing state and span origin. Conditional
/// Sendable of the sequence does not make this mutable iterator safe for concurrent access.
public struct Iterator: AsyncIteratorProtocol {
private var source: Source.AsyncIterator?
private let limits: JSONLimits
private var driver: JSONArrayStreamDriver<Source.AsyncIterator>?
fileprivate init(source: Source.AsyncIterator, limits: JSONLimits) { self.source = source; self.limits = limits }
/// Pulls at most one located result through exclusively owned iterator state.
///
/// Continue until the certified boundary or nil required by the sequence contract.
/// An error or cancellation makes an initialized iterator terminal and prevents later pulls.
/// - Returns: The next located event, or nil after termination.
/// - Throws: Source, syntax or resource errors, or cancellation.
public mutating func next() async throws -> Element? {
if driver == nil { driver = try JSONArrayStreamDriver(upstream: source!, limits: limits); source = nil }
return try await driver!.nextLocated()
}
}
}
extension JSONLocatedTopLevelArraySequence: Sendable where Source: Sendable {}
18 changes: 15 additions & 3 deletions Sources/JSONStreaming/JSONTopLevelArraySequence.swift
Original file line number Diff line number Diff line change
Expand Up @@ -40,13 +40,23 @@ public struct JSONTopLevelArraySequence<Source: AsyncSequence>: AsyncSequence wh
}
}
extension JSONTopLevelArraySequence: Sendable where Source: Sendable {}
private final class JSONArrayStreamDriver<Upstream: AsyncIteratorProtocol> where Upstream.Element == JSONByteChunk {
final class JSONArrayStreamDriver<Upstream: AsyncIteratorProtocol> where Upstream.Element == JSONByteChunk {
let input: JSONStreamInput<Upstream>
var machine = JSONTokenMachine(topLevelArrayStreaming: true)
var assembler = JSONValueAssembler()
var began = false, closed = false, terminal = false, resetOnNext = false
private var elementStart: UInt64?
init(upstream: Upstream, limits: JSONLimits) throws { input = try JSONStreamInput(upstream: upstream, limits: limits) }
func next() async throws -> JSONTopLevelArrayEvent? {
guard let event = try await nextLocated() else { return nil }
switch event {
case .element(let index, let value, _): return .element(index: index, value: value)
case .documentCompleted(let summary): return .documentCompleted(summary)
}
}
// Both facades share this one grammar and pull driver. Spans are absolute
// token offsets from the machine, never counts of delivered chunks or text.
func nextLocated() async throws -> JSONLocatedTopLevelArrayEvent? {
guard !terminal else { return nil }
do {
try input.meter.checkpoint(); try input.meter.require(.pendingValues, 1)
Expand All @@ -72,12 +82,14 @@ private final class JSONArrayStreamDriver<Upstream: AsyncIteratorProtocol> where
guard case .endArray = event.kind else { throw JSONFailure(.syntax, code: .invalidSyntax) }
closed = true; continue
}
if elementStart == nil { elementStart = event.start }
if let value = try assembler.consume(event, meter: &input.meter) {
let index = input.meter.used(.totalRecords)
try input.meter.charge(.totalRecords); assembler = JSONValueAssembler(); resetOnNext = true
return .element(index: index, value: value)
let span = JSONSourceSpan(start: elementStart!, end: event.end); elementStart = nil
return .element(index: index, value: value, sourceSpan: span)
}
}
} catch { terminal = true; input.failed = true; assembler = JSONValueAssembler(); throw error }
} catch { terminal = true; input.failed = true; assembler = JSONValueAssembler(); elementStart = nil; throw error }
}
}
Loading