Pure-julia OpenAPI internals rewrite

[quinnj]

Replace the legacy generated-client and server implementation with the normalized OpenAPI 3.0, 3.1, and 3.2 pipeline.\n\nKeep the provisional JSON Schema engine isolated inside OpenAPI. Generate clients against that engine until its API is ready to move upstream.\n\nKeep HTTP optional through an extension. Leave server framework integration to downstream packages such as Servo.\n\nAdd adversarial, conformance, external-corpus, runtime HTTP, and JuliaC trim-compilation coverage.\n\nReview hardening:\n- Quote specification-derived generated source and prevent generated identifier collisions.\n- Match parameterized media types and make streaming cancellation deterministic.\n- Preserve raw server request bodies and decode form and multipart values by schema.\n- Follow documented success responses, including empty responses and JSON null.\n- Restore license and TagBot requirements, declare the namespaced public API, and run the official schema suite in CI.\n\nValidation:\n- Julia 1.11 full package test suite.\n- Julia 1.12 full package test suite and JuliaC trim compilation.\n- Official JSON Schema suite: 9,884 of 9,884 cases.\n- Independent Fable 5 implementation review: CLEAN.\n\nBREAKING CHANGE: The legacy OpenAPI 0.2 API is replaced by the namespaced document and client-generation API.\n\nCo-authored by Codex

[tanmaykm]

Thanks @quinnj . I did some trials with the new client. Majority of the specs that I tried work fine. But here are a few things which I feel should be addressed:

• The response code declaration in spec seems too strict. Specs sometimes do not document response code when they do not return any data. With the new client, this throws an error.
• Missing Content-Type on a response would throw exception. The existing client deserializes by status alone. The new client throws UnexpectedContentType. Sloppy servers, which we do encounter in practice, would become client errors.
• Date time fields do not support time zone.
• Streaming support is not there

I will also try this out with some more complicated specs, maybe the k8s api spec.

[quinnj]

Thanks for the thorough trial run, @tanmaykm — all four points are addressed as of ac0689d:

Undocumented response codes no longer error. A 2XX status the spec doesn't describe now succeeds: an empty body returns nothing, a non-empty body returns the raw bytes. Undocumented error statuses still throw ApiError (with the raw body attached) so failures stay visible.

Missing/misreported Content-Type falls back to decoding by status. When a response has no Content-Type, or misreports it while only one media type is documented for that status, the client decodes with the documented media type — the legacy "deserialize by status alone" behavior. UnexpectedContentType is now reserved for the genuinely ambiguous case: several documented media types and a header that matches none of them.

Time zones. Offsets like 2020-01-02T03:04:05+05:30 parse in the default Dates.DateTime mapping by normalizing to UTC (and zone-less date-times are accepted as UTC). For preserved offsets, generate with OpenAPI.client(doc; datetime = :zoned): format: date-time fields then map to TimeZones.ZonedDateTime with RFC 3339 round-tripping, matching the legacy client's ZonedDateTime behavior. The default stays DateTime so plain clients keep no TimeZones dependency (and stay juliac --trim friendly); the option is a one-liner where zone fidelity matters.

Streaming responses. Every generated operation now accepts stream_to = Channel(n). The call returns at the response head and a background task decodes items onto the channel: application/json bodies split into consecutive JSON documents each decoded against the documented schema (the k8s watch convention), JSON Lines/NDJSON decode per line to the array's element type, JSON text sequences split on RFC 7464 records, text/* yields lines, and other media yield raw chunks. Decode/validation failures close the channel with the error, error statuses throw ApiError with the buffered body, and closing the channel from the consumer side aborts the transfer (connection torn down, not leaked).

events = Channel{Any}(16)
K8sClient.watch_core_v1_namespaced_pod(...; stream_to = events)
for event in events
    ...
end

Test coverage added for all of the above, including a raw chunked-transfer fixture that splits items across wire chunks.

I also pre-flighted the k8s trial you mentioned: the v3 documents pair application/json with application/json;stream=watch on every list operation, and the normalizer was rejecting those as case-insensitive duplicate media types. Parameterized keys are now kept distinct, and the core api/v1 document (112 paths, all the watch operations) generates and compiles cleanly. Would still very much appreciate your run against the rest of the k8s groups.

[update prompted and reviewed by quinnj, posted by claude]

[tanmaykm]

Thanks @quinnj — I re-ran my trials at bd96d53. All four earlier issues check out as fixed:

• Undocumented 2xx: my full petstore scenario suite (CRUD, multipart upload, path escaping, auth, response headers, 404 handling) now passes against a legacy 0.2.x julia-server — operations that document only error codes return nothing on a 200 instead of throwing.
• Missing Content-Type: a 200 with no Content-Type header now decodes by the documented media type.
• Time zones: datetime = :zoned round-trips 2024-03-04T05:06:07.890+05:30 with the offset preserved, both in encode/decode and over the wire.
• Streaming: tested against a chunked-transfer server where I control wire timing. First item is delivered immediately while the server stalls before the second (no small-chunk buffering), NDJSON decodes per line, close(channel) tears the connection down within ~0.4s, and with_http_info=true returns at the response head. This matches the behavior we fixed in fix(client): discard truncated JSON document on mid-stream EOF #97 through fix(client): :http streaming stalls small chunks until 8KB buffer fills #102. One difference from the legacy client: a truncated final JSON document closes the channel with a DecodeError instead of ending silently — I think that's the better behavior, just noting it.

The k8s core v1 document also now generates in strict mode (113 paths, ~6.6 MiB module, loads in ~6.5s) — the ac0689d media-key fix works.

I then ran the generated k8s client against a real cluster (k3s v1.35 via kubectl proxy), which turned up two new issues:

1. validate_responses=false doesn't reach model decoding. Listing pods fails with SchemaValidationError: the live API returns "lastProbeTime": null while the spec declares Time as a non-nullable date-time string. The generated model _decode methods call _validate_schema unconditionally, so the client-level flag never applies, and the typed field would reject the null anyway. As far as I can tell there is no configuration under which the generated client can decode a real pod list. Could the flag be threaded through model _decode (and explicit null on a non-nullable optional field tolerated when validation is off)? Without that, any server that drifts from its published contract is a hard error with no escape hatch.
2. Watch decoding fails on the k8s spec's declared schema. The stream itself works against the real cluster (connects, splits items, cancels cleanly), but k8s declares application/json;stream=watch responses with the List schema while the wire carries WatchEvent objects, so every item fails to decode. Buffered responses have codec! as an escape hatch, but the streaming path doesn't consult custom codecs. Either letting stream_to use a registered codec/decoder, or documenting spec-patching as the supported route for k8s-style watches, would close this.

[quinnj]

Thanks — both issues are fixed in fd4558c. validate_responses=false now reaches nested model decoding and tolerates explicit null on optional fields while preserving ABSENT for missing fields. Streaming now uses parameter-aware custom decoders per framed item, so application/json;stream=watch can override the incorrect List schema without affecting normal JSON. I added live regressions for both, and CI is green. Could you retry the pod-list and watch cases?

[work by codex; reviewed by quinnj]

[tanmaykm]

Thanks. Retried both cases at c2a5244 against the real cluster. validate_responses=false now works end to end: a tolerant client decodes live pod lists on the unmodified k8s spec (every lastProbeTime: null arrives as nothing), and a strict client still rejects them as before. The streaming codec hook also works, but I hit one nuance with the registration pattern the README suggests.

The parameterized codec registration never fires against a real k8s cluster. codec!(client, "application/json;stream=watch"; decode=...) is matched against the received Content-Type, and _media_selection_score requires the documented parameters to be a subset of the received ones. A real apiserver always replies Content-Type: application/json — with no parameters, even when the request Accepts application/json;stream=watch (verified with curl against k8s v1.35). So the score is zero, the codec is skipped, and each frame still fails to decode against the List schema. The added regression passes because its fixture server replies Content-Type: application/json; stream=watch, which real k8s never sends.

What does work: registering the codec for plain "application/json" on a client instance dedicated to watch calls —

watch_client = K8s.Client(server; validate_responses = false)
K8s.codec!(watch_client, "application/json"; decode = (bytes, _) -> JSON.parse(String(bytes)))

— verified live, events stream out fine. The dedicated instance is needed because a plain application/json codec also overrides normal buffered JSON responses on that client.

Two possible upstream resolutions, either would do:

1. When selecting a stream-item codec, also try the documented media type of the selected entry (or the media type passed as accept=), not just the received header — then the parameterized registration scopes the override the way the README describes, even for servers that reply with the bare type.
2. Or simply document the dedicated-client + plain-media pattern above as the intended escape hatch for this class of server.

Not a blocker for us either way — patching the k8s spec at generation time is something we do currently though we would like to avoid it when possible — but the README example as written won't work against the API it was presumably written for.

[quinnj]

Thanks @tanmaykm — good catch on the fixture not matching the real apiserver. This is fixed in 1ff9ba8, along the lines of your option 1: when no registered decoder matches the received Content-Type, streaming calls now fall back to the media type the call requested via accept. So the README pattern works against a real cluster:

K8s.codec!(client, "application/json;stream=watch"; decode = (bytes, _) -> JSON.parse(String(bytes)))
K8s.watch_core_v1_namespaced_pod(...; client, accept = "application/json;stream=watch", stream_to = events)

The override is scoped to exactly the calls that pass that accept, so plain buffered application/json responses on the same client are unaffected — no dedicated client instance needed. The regression test now mirrors the real apiserver (both media types documented, bare Content-Type: application/json in the reply), and the README section was updated to describe the accept requirement. CI is green.

Could you give the watch case one more try against your cluster?

[work by claude; reviewed by quinnj]

[tanmaykm]

Expanding on the "generated-module ↔ runtime contract is private" blocker from #104

I dug into what the contract actually consists of, using a small generated client/server pair as evidence. It's larger than the _decode/_encode framing in #104 suggests, and one part of it can fail silently. Details below, plus a cheap mechanical fix that's independent of whatever policy we choose.

The coupling has three layers

1. Imported names. A generated client imports 25 names from OpenAPI.Runtime (see GENERATED_CLIENT_IMPORTS in src/client.jl); a generated server imports a different overlapping set of ~25, mostly _-prefixed internals (_decode_sequential_json, _header_type_variant, _select_media, …). The union is ~40 names, plus ~10 referenced qualified (Runtime.Client, Runtime.Spec, Runtime.credential!, …). None are in the public list in src/OpenAPI.jl — Runtime itself isn't declared public. Renaming any one breaks previously generated modules at import time. Loud failure, acceptable.

2. Data shapes baked into generated source. Generated modules don't just call functions — they embed runtime-internal data structures as literals:

• Runtime.Spec(; security_schemes, resources, roots, dialects, directional_required, default_server) — a keyword-constructor contract.
• The _OP_* operation tables: NamedTuples with exact field names (id, method, path, parameters, request, responses, security, servers) and nested tuple shapes destructured by Runtime._request (client) and _operation_arguments/_server_response (server). Every parameter-descriptor field (arg, name, type, location, style, explode, allow_reserved, shape, schema, content, required) is contract.
• Schema refs as (resource = ..., pointer = ...) NamedTuples passed to _validate_schema with direction = :neutral.

Changing any shape produces MethodErrors or key errors deep inside the runtime when an old generated file runs against a new OpenAPI — mysterious, but at least an error.

3. The silent one. Generated files emit positional SchemaEngine.Dialect constructor calls:

SchemaEngine.Dialect(:draft202012, "https://json-schema.org/draft/2020-12/schema", "\$id",
                     true, true, true, true, false, true, true)

Dialect (src/schema_engine/dialects.jl) ends in seven positional Bool fields (ref_siblings, modern_items, unevaluated, dynamic_refs, recursive_refs, applicator, validation). Reordering two of them, or changing one's meaning, keeps old generated code constructing successfully — with silently wrong validation semantics. That's a wire-behavior change with no error anywhere.

Why this must be resolved before the tag

Once 1.0 tags, semver says internals can change in a minor release. But changing them breaks every generated artifact in every user's repo — files that look like user code and that Pkg can't fix. So post-tag we're in one of two bad states: ship a minor that breaks the ecosystem (sometimes silently, per layer 3), or treat Runtime as frozen-in-practice — an undocumented, untested ~40-name-plus-shapes API nobody dares touch. The tag converts "undecided" into the second state by default.

Proposal: stamp + guard now, policy independently

Two mechanical gaps exist today regardless of which policy wins (freeze-by-policy / make Runtime public / design a narrow seam):

1. No generator version stamp. The generated banner records the spec's title/version, not the OpenAPI.jl version that produced the file. Given a broken generated module in the wild, there's no way to tell which release generated it.
2. No load-time guard. Nothing asserts compatibility at module load.

Suggested fix (~15 lines): add a Runtime.CONTRACT_VERSION::Int constant; emit into every generated module a header line recording the generating OpenAPI.jl version and a load-time assertion like Runtime.require_contract(1) that throws a clear "generated by OpenAPI.jl x.y.z against runtime contract 1; this runtime provides contract 2 — regenerate" error. Bump the constant whenever any of the three layers change. This converts every layer-2/3 failure from mysterious-or-silent into actionable, and it stays useful under any eventual policy — a narrow seam still has a version, and a public API still benefits from recording provenance.

Independent of that, #104's ask stands: document that generated modules are version-coupled baked artifacts. That's true today whether or not we write it down.

[quinnj]

Thanks @tanmaykm — this is exactly the right framing, and your three-layer breakdown held up precisely against the code. All of it is now addressed, plus a few things the deep-dive surfaced along the way.

Stamp + guard (your proposal, adopted as-is). Runtime.CONTRACT_VERSION::Int now exists, and every generated module — client, server, and the framework-extension path through OpenAPI.server_module_source — emits Runtime.require_contract(N, "<generating version>"). The banner also stamps the producing release: # Generated by OpenAPI.jl v1.0.0 from "Title" version "1.2.3". Do not edit.. On mismatch you get: "this generated module was produced by OpenAPI.jl v0.0.0 against generated-code contract 1, but the loaded OpenAPI.jl v1.0.0 provides contract 3; regenerate the module with OpenAPI.client or OpenAPI.server." One refinement over the sketch: the guard is emitted before the private import OpenAPI.Runtime: list, so a layer-1 rename also surfaces the actionable error rather than a bare UndefVarError. (require_contract itself is now the one permanently frozen name, and is documented as such.)

Layer 3 is gone, not just guarded. Generated code no longer bakes positional Dialect literals at all: the five standard dialects are emitted as SchemaEngine.dialect(:draft202012) lookups (the runtime stays the single owner of flag semantics), and vocabulary-customized dialects go through a new keyword-only constructor that maps values through fieldnames(Dialect) — a field reorder can no longer silently reassign the Bools from either direction, and old positional literals now fail loudly at load.

Layer 2 hardened the same way. Runtime.Spec generated-data keywords are now required (an omitted keyword can't silently substitute an empty vector), media and security records moved from positional tuples to named fields with fail-closed checks (unknown API-key location, header shape/style drift, and descriptor-without-roots now error instead of taking fallback paths), and the runtime reconstructs dialect aliases through keywords. Contract version is at 3 to account for these shape changes; 1.0 will tag at whatever it is then.

Two adjacent finds worth flagging. While sweeping, we found and fixed a real semantic gap: planning decided $ref-sibling applicability from the OAS version alone, but OAS 3.1 documents can select draft-07 (or another dialect) where $ref siblings don't apply — model shape, requiredness, nullability, and descriptions now follow the compiled per-node dialect, with regressions covering OAS 3.1 + draft-07.

And the #104 docstring blocker. Spec descriptions now flow into generated model docstrings (schema description plus a bullet per documented field), and operation docstrings gained parameter/request-body bullets — for both client and server output, with hostile-content escaping tests (quotes, $, """, backslashes, unicode). The coupling policy is documented in the README ("Generated modules are baked artifacts") and a 0.2.x → 1.0 MIGRATION.md now covers the openapi-generator lane, API mapping, and dropped capabilities.

Validated on Julia 1.10 and 1.12 full suites, the external corpus (32/32 across Petstore/Discord/Stripe/GitHub), and the official JSON Schema suite (9,884/9,884). Byte-determinism of generated output is unchanged.

Remaining from #104 that this doesn't close: the 0.2 compat-pin decision (and whether a release-0.2 branch gets cut — that's yours to call on the upstream repo), and the live server smoke test against a production-shaped spec.

[tanmaykm]

A release-0.2 branch is now cut out from main where we'll maintain the old version. PR to openapi-generator up at OpenAPITools/openapi-generator#24789 to pin it to the 0.2 release.

I think we can merge this once that's in place and take up any new requirements on main.