Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 2 additions & 1 deletion AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -57,7 +57,8 @@ store/follow API on 2026-11-10; later API changes stay additive and human-gated.
one invocation-owned ledger per operation. Positions remain unforgeable and
source-instance-local. No persistence or mutable store is exposed. See
Documentation/Architecture/StructuralIntelligence/BoundedWorkflows1.8.md.
Wider retrieval/persistence and JSON text-sequence framing remain separate work.
Wider retrieval/persistence remains separate work. RFC 7464 JSON text sequences
are 1.10 development in JSONStreaming (`JSONTextSequence`, `JSONTextSequenceWriter`).

JSONTransform uses one private compiler/VM and the CLI's accounted adapter.
Preserve its accepted 42 authored declarations and exact semantics. The original
Expand Down
7 changes: 7 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -28,6 +28,13 @@
complete the page. Its CSP authorizes the two stylesheets by hash and sets
`script-src 'none'`. The comparison is a package walker in JSON shared with
`JSONPatch.difference`, so JSONHTML still depends on JSON only.
- Add RFC 7464 JSON text sequences (`application/json-seq`) to JSONStreaming.
`JSONTextSequence` reads texts framed by the record separator byte with the
same pull model and budgets as NDJSON. Unterminated top-level scalars are
treated as possibly truncated (RFC 7464 section 2.4). Malformed texts fail by
default; `JSONTextSequenceOptions.MalformedTexts.skip` skips them and counts
them on the iterator, but never skips resource failures. `JSONTextSequenceWriter`
writes RS, strict JSON and LF per value, as one sequence or in awaited chunks.

## Unreleased — 1.9.1 hygiene

Expand Down
6 changes: 3 additions & 3 deletions Documentation/Qualification/TestDiscovery.json
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@
"JSONTests": 198,
"JSONSchemaTests": 79,
"JSONRegexRuntimeTests": 16,
"JSONStreamingTests": 36,
"JSONStreamingTests": 42,
"JSONPatchTests": 22,
"JSONIntegrationTests": 45,
"JSONHTMLTests": 103,
Expand Down Expand Up @@ -76,7 +76,7 @@
"JSONTests": 198,
"JSONSchemaTests": 79,
"JSONRegexRuntimeTests": 16,
"JSONStreamingTests": 36,
"JSONStreamingTests": 42,
"JSONPatchTests": 22,
"JSONIntegrationTests": 45,
"JSONHTMLTests": 87,
Expand All @@ -92,7 +92,7 @@
"JSONTests": 198,
"JSONSchemaTests": 79,
"JSONRegexRuntimeTests": 16,
"JSONStreamingTests": 36,
"JSONStreamingTests": 42,
"JSONPatchTests": 22,
"JSONIntegrationTests": 45,
"JSONHTMLTests": 87,
Expand Down
4 changes: 3 additions & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -90,7 +90,7 @@ Choose the products your application needs:
| **JSON** | Exact values, strict parsing/writing, native Codable, Pointer, presence, checked construction, source documents and diagnostics |
| **JSONSchema** | Draft 2020-12 compilation/evaluation, offline registries, bounded regex, annotations and raw declaration inspection |
| **JSONSchemaExamples** | Opt-in deterministic exact candidates, conservative typed construction and native schema certification, introduced in 1.6.0 |
| **JSONStreaming** | Pull syntax events and values, strict NDJSON, root-array elements and awaited output chunks |
| **JSONStreaming** | Pull syntax events and values, strict NDJSON, RFC 7464 text sequences, root-array elements and awaited output chunks |
| **JSONHTML** | Exact HTML and immutable exports; structured forms/cards and bounded previews in 1.6.0 |
| **swift-json** | Bounded command-line JSON workflows (available since 1.5.0) |
| **JSONBridges** | Opt-in native integer, Double and Foundation Decimal conversions (available in the 1.4.0 tag and 1.5.0 release) |
Expand Down Expand Up @@ -471,6 +471,8 @@ producer queue. Root-array elements and syntax events remain provisional until
completion. A document completes only after its closing syntax, trailing whitespace
and EOF. `JSONValueSequence` waits for its framing boundary before yielding a value;
`NDJSONValueSequence` rejects blank/malformed physical records and never skips them.
`JSONTextSequence` reads RFC 7464 `application/json-seq` texts; it treats unterminated
top-level scalars as possibly truncated and skips malformed texts only when asked.
Since 1.8.0, `JSONLocatedTopLevelArraySequence` drives the same root-array reader and
also yields each element's `JSONSourceSpan`: zero-based half-open UTF-8 byte offsets
in the complete stream, excluding whitespace and separators. It retains no original
Expand Down
2 changes: 1 addition & 1 deletion Sources/JSONCLIEngine/CLIBuildIdentity.swift
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,6 @@
// Generated by Scripts/generate_cli_identity.py; excludes this file.
// SHA-256 of the ordered Package.swift + Sources content manifest (DocC catalogs excluded), not a Git commit.
package enum CLIBuildIdentity {
package static let revision = "sha256:dafab040289a71fed6924ffe8aa7a7273e6315c5701d1fd35fd5d1b812a52ccd"
package static let revision = "sha256:c0c4442af4a6e52b5e232fc41405acbf92616b366832ae0df0bfb843e18a3773"
package static let kind = "source-sha256"
}
12 changes: 11 additions & 1 deletion Sources/JSONStreaming/Documentation.docc/JSONStreaming.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
# ``JSONStreaming``

Pull-driven events and values, strict NDJSON, root-array elements and awaited sinks.
Pull-driven events and values, strict NDJSON, RFC 7464 text sequences, root-array elements and awaited sinks.

Sources are caller-owned AsyncSequences of owned ``JSONByteChunk`` values.
Each iterator has exclusive mutable state, performs one pull at a time and
Expand All @@ -19,6 +19,12 @@ blank/malformed records and accepts a valid final record without a final LF.
There is no skipped-record recovery. Error locations include absolute byte and
physical record positions.

``JSONTextSequence`` reads RFC 7464 JSON text sequences (`application/json-seq`):
each text starts with a record separator byte. Unterminated top-level scalars
are treated as possibly truncated. Malformed texts fail by default and can be
skipped explicitly. ``JSONTextSequenceWriter`` writes the same framing. See
<doc:TextSequences>.

``JSONTopLevelArraySequence`` emits indexed elements followed by one explicit
document-completed summary. Earlier elements remain provisional with respect
to the enclosing document. A malformed/truncated tail produces no completion.
Expand All @@ -43,6 +49,7 @@ publication must stage bytes in its own sink.
### Guides

- <doc:CompletionAndBackpressure>
- <doc:TextSequences>

### Pull sequences
- ``JSONByteChunk``
Expand All @@ -55,10 +62,13 @@ publication must stage bytes in its own sink.
### Record framing and completion
- ``NDJSONValueSequence``
- ``NDJSONOptions``
- ``JSONTextSequence``
- ``JSONTextSequenceOptions``
- ``JSONTopLevelArraySequence``
- ``JSONTopLevelArrayEvent``
- ``JSONLocatedTopLevelArraySequence``
- ``JSONLocatedTopLevelArrayEvent``

### Awaited output
- ``JSONAsyncWriter``
- ``JSONTextSequenceWriter``
54 changes: 54 additions & 0 deletions Sources/JSONStreaming/Documentation.docc/TextSequences.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,54 @@
# JSON text sequences

Read and write RFC 7464 `application/json-seq` streams.

## Overview

A JSON text sequence frames each JSON text with a leading record separator byte
(RS, `0x1E`) and, when written, a trailing LF. Because RS can never appear
unescaped inside JSON, a reader can find the next text even after a malformed
one. That makes the format useful for logs and long-running streams where one
damaged record should not hide the rest.

``JSONTextSequence`` reads texts from caller-owned chunks with the same pull
model, cumulative iterator budget and per-text value quotas as
``NDJSONValueSequence``:

```swift
for try await value in JSONTextSequence(chunks) {
handle(value)
}
```

Each text runs from an RS to the next RS or the end of input and is parsed by
the strict grammar, so surrounding whitespace is allowed and a text may span
several lines. Consecutive RS bytes are not empty texts.

### Truncated and malformed texts

A top-level number, `true`, `false` or `null` that does not end in whitespace
may have been cut short ("123" could be the start of "1234"), so RFC 7464
section 2.4 treats it as possibly truncated. The reader reports it as an
`unexpectedEOF` syntax failure. Bytes before the first RS are malformed as well.

By default a malformed text ends the sequence with its failure, located by
absolute byte offset, text number (`record`) and line and column within the
text. With ``JSONTextSequenceOptions/MalformedTexts/skip`` the reader skips
malformed texts and continues with the next RS, as RFC 7464 recommends;
``JSONTextSequence/Iterator/skippedTexts`` counts them. Resource failures,
cancellation and source errors are never skipped.

### Writing

``JSONTextSequenceWriter`` writes each value as RS, the strict writer's bytes
and LF. ``JSONTextSequenceWriter/encode(_:)`` produces a complete sequence under
one ledger. ``JSONTextSequenceWriter/write(_:to:)`` writes one text in awaited,
bounded chunks, like ``JSONAsyncWriter``; call it once per value. Bytes a sink
has accepted cannot be recalled, but a reader recognizes a partially written
text as malformed.

## Topics

- ``JSONTextSequence``
- ``JSONTextSequenceOptions``
- ``JSONTextSequenceWriter``
120 changes: 120 additions & 0 deletions Sources/JSONStreaming/TextSequence/JSONTextSequence.swift
Original file line number Diff line number Diff line change
@@ -0,0 +1,120 @@
// SPDX-License-Identifier: Apache-2.0
// Original RFC 7464 (application/json-seq) reader over the shared stream input.
import JSON
/// A pull reader for RFC 7464 JSON text sequences (`application/json-seq`).
///
/// Each text starts with the record separator byte (RS, `0x1E`) and runs to the
/// next RS or the end of input. Consecutive RS bytes are not empty texts. Each text
/// is parsed by the strict UTF-8 grammar, so surrounding whitespace, including the
/// LF that writers append, is allowed.
///
/// A text that is a top-level number, `true`, `false` or `null` must end in
/// whitespace; otherwise it may have been truncated and is malformed (RFC 7464
/// section 2.4). Bytes before the first RS are a malformed text too. Malformed
/// texts end the sequence by default; ``JSONTextSequenceOptions/MalformedTexts/skip``
/// continues with the next text instead.
///
/// Failures report the absolute byte offset, the one-based text number as `record`,
/// and the line and column within the text.
public struct JSONTextSequence<Source: AsyncSequence>: AsyncSequence where Source.Element == JSONByteChunk {
/// Each complete JSON text, in order.
public typealias Element = JSON
let source: Source
let options: JSONTextSequenceOptions
/// Wraps a caller-owned source of `application/json-seq` bytes.
///
/// - Parameters:
/// - source: The caller-owned asynchronous source of owned UTF-8 chunks.
/// - options: Limits and the malformed-text policy.
public init(_ source: Source, options: JSONTextSequenceOptions = .init()) { self.source = source; self.options = options }
/// Creates an independent iterator with exclusive mutable state and no prefetch task.
public func makeAsyncIterator() -> Iterator { .init(source: source.makeAsyncIterator(), options: options) }
/// Exclusive pull iterator; never call next concurrently.
///
/// Each iterator owns its budget and parsing state. Conditional Sendable of the
/// sequence does not make this mutable iterator safe for concurrent access.
public struct Iterator: AsyncIteratorProtocol {
private var source: Source.AsyncIterator?
private let options: JSONTextSequenceOptions
private var driver: JSONTextSequenceDriver<Source.AsyncIterator>?
fileprivate init(source: Source.AsyncIterator, options: JSONTextSequenceOptions) { self.source = source; self.options = options }
/// The number of malformed texts skipped so far; always zero with the `fail` policy.
public var skippedTexts: Int { driver?.skipped ?? 0 }
/// Pulls at most one value through exclusively owned iterator state.
///
/// An error or cancellation makes an initialized iterator terminal and prevents later pulls.
/// - Returns: The next JSON text's value, or nil at the end of input.
/// - Throws: Source, syntax or resource errors, or cancellation.
public mutating func next() async throws -> JSON? {
if driver == nil { driver = try JSONTextSequenceDriver(upstream: source!, options: options); source = nil }
return try await driver!.next()
}
}
}
extension JSONTextSequence: Sendable where Source: Sendable {}

private final class JSONTextSequenceDriver<Upstream: AsyncIteratorProtocol> where Upstream.Element == JSONByteChunk {
static var separator: UInt8 { 0x1E }
let input: JSONStreamInput<Upstream>
let skipMalformed: Bool
var terminal = false, resetOnNext = false, afterSeparator = false
var offset: UInt64 = 0, texts: UInt64 = 0
var skipped = 0
init(upstream: Upstream, options: JSONTextSequenceOptions) throws {
input = try JSONStreamInput(upstream: upstream, limits: options.limits); skipMalformed = options.malformedTexts == .skip
}
func next() async throws -> JSON? {
guard !terminal else { return nil }
do {
try input.meter.checkpoint(); try input.meter.require(.pendingValues, 1)
while true {
if resetOnNext { try input.resetRecord(); resetOnNext = false }
let introduced = afterSeparator, start = offset
var text: [UInt8] = [], ended = false
while true {
guard try await input.ensureBytes() else { ended = true; break }
let byte = input.bytes[input.index]; input.index += 1; offset += 1
try input.meter.work()
if byte == Self.separator { break }
try input.meter.require(.inputBytes, UInt64(text.count) + 1); try input.meter.retain(1); text.append(byte)
}
afterSeparator = !ended
// Empty segments: input that starts with RS, consecutive RS bytes, or a final RS.
if text.isEmpty {
if ended { terminal = true; return nil }
continue
}
texts += 1; resetOnNext = true
do {
let value = try parse(text, introduced: introduced)
try input.meter.charge(.totalRecords)
if ended { terminal = true }
return value
} catch let failure as JSONFailure {
let relative = failure.location?.byteOffset ?? 0
let located = JSONFailure(failure.category, code: failure.code,
location: .init(byteOffset: start + relative, line: failure.location?.line,
column: failure.location?.column, record: texts),
resource: failure.resource)
guard skipMalformed, failure.category == .syntax else { throw located }
try input.meter.charge(.totalRecords); skipped += 1
if ended { terminal = true; return nil }
}
}
} catch { terminal = true; input.failed = true; throw error }
}
/// Parses one text; a missing leading RS or a possibly truncated scalar is a syntax failure.
private func parse(_ text: [UInt8], introduced: Bool) throws -> JSON {
guard introduced else { throw JSONFailure(.syntax, code: .invalidSyntax, location: .init(byteOffset: 0, line: 1, column: 1)) }
let value = try JSONParser().parse(text, meter: &input.meter)
switch value {
case .object, .array, .string: return value
default:
// RFC 7464 section 2.4: a top-level scalar without trailing whitespace may be truncated.
guard let last = text.last, last == 0x20 || last == 0x09 || last == 0x0A || last == 0x0D else {
throw JSONFailure(.syntax, code: .unexpectedEOF, location: .init(byteOffset: UInt64(text.count)))
}
return value
}
}
}
Original file line number Diff line number Diff line change
@@ -0,0 +1,26 @@
// SPDX-License-Identifier: Apache-2.0
import JSON
/// Finite limits and the malformed-text policy for RFC 7464 JSON text sequences.
public struct JSONTextSequenceOptions: Sendable {
/// What a reader does with a text that is not one complete JSON value.
public enum MalformedTexts: Sendable, Equatable {
/// Throw the syntax failure and end the sequence (the default).
case fail
/// Skip the text and continue with the next record separator, as RFC 7464
/// recommends. Resource failures, cancellation and source errors still end
/// the sequence. The iterator counts skipped texts.
case skip
}
/// Finite capacities and work quotas applied to library-owned operations.
public var limits: JSONLimits
/// The policy for texts that fail to parse or are possibly truncated.
public var malformedTexts: MalformedTexts
/// Creates the configuration; finite limits are checked at operation entry.
///
/// - Parameters:
/// - limits: Finite inclusive capacities and work quotas, per text and for the iterator lifetime.
/// - malformedTexts: Whether malformed texts end the sequence or are skipped.
public init(limits: JSONLimits = .balanced, malformedTexts: MalformedTexts = .fail) {
self.limits = limits; self.malformedTexts = malformedTexts
}
}
Loading
Loading