Adopt the WIT mutation schema; retire the byte protocol - #18
Merged
Conversation
Adds a second mutation channel expressing the same op vocabulary as an explicit WIT schema (`interface mutations`, `stream<operation>`, export `run-typed`) alongside the existing hand-rolled byte format on `run`, so the two encodings can be measured against each other. The byte channel stays the default and is untouched — it is the baseline. Both channels compile into the same component; `mountApp`'s `channel` option picks one, and both feed the identical `DomApplier`, so the DOM work is common and the delta is purely encode/transport/decode. `host/tests/typed_test.ts` asserts the two produce identical DOM for the same interaction sequence; a one-off instrumented run confirmed they also emit identical operation counts. Findings are written up in bench/README.md's "Channel A/B" section. In short: the typed channel costs 4-5.5 us per operation, which reproduces every row of the two-column table (~5x on create-10k, 12x on clear, invisible on the small ops). ~85% of that is polyengine's uncached canonical-ABI lift, not the component model or the schema — memoizing the layout functions alone takes create-10k from 5.07x to 1.74x. The floor is not 1.0x though: a typed channel must allocate JS objects per operation where the byte decoder allocates nothing (~190 vs ~45 ns/op in a pure-JS bound). The schema also costs something before any measurement: WIT forbids recursive types, so register-template's tree becomes an index arena, which admits malformed graphs the byte grammar cannot express and needs validation the byte decoder never needed. Also fixes bench/ops.ts's stale mountApp call (the bench had rotted against the current host API) and adds bench/ to `deno task check`, the gate that would have caught it.
The section claimed the residual gap after fixing polyengine's uncached lift was a ~4x floor no runtime work could remove, because a typed channel must allocate JS objects per element. That held polyengine's value-mapping contract fixed, which is the thing worth changing, and it was wrong in the direction that flattered the conclusion. Measured against the same guest memory at the same rendezvous, interleaved so no strategy sees different JIT state, and asserted element-by-element against the interpreter and against the shipped applyTyped: interpreted load() + toHost (today) ~4080 ns/element compiled lift (identical JS values) ~232 ns/element 17x compiled visitor (operands as args) ~127 ns/element 32x the byte decoder, same sink ~14 ns/element Closure specialization needs no eval, no emitted modules and no contract change. The visitor shape is where the supposedly irreducible allocation goes. Extrapolated, create-10k lands at ~1.9x rather than 5.8x, and the residual becomes guest-side lowering rather than lift. Reported upstream as polymorph-components/polyengine#261.
The pinned runtime now carries the four optimization PRs that came out of polyengine#261 (#263 layout-node cache, #264 embedder adapter tables, #265 flatten-count memoization, #270 variant kind/value). The typed mutation channel's cost was almost entirely that lift path, so the A/B was only meaningful once they landed. Per-operation cost of the typed channel, medians of three full runs: op ops before after ratio then -> now create-10k 90 006 4.9 us 0.78 us 5.80x -> ~1.8x create-1k 9 006 5.5 us 0.70 us 4.99x -> ~1.6x append-1k 9 002 5.2 us 0.77 us 1.59x -> ~1.1x clear 10 002 4.1 us 0.15 us 12.02x -> ~1.4x 6-27x less per operation, from four upstream commits and no change to this repo's encoders on either side. The before block is kept as a dated table rather than overwritten; the before/after is what makes either column mean anything. Also retracts a claim from the previous revision: it said the residual after fixing the lift would be guest-side lowering rather than lift. That was a subtraction between numbers measured under different conditions. With the lift fixed, the end-to-end delta (~0.78 us/op) and the measured lift cost (~0.7-0.96 us/element) are the same size, so lift still accounts for essentially all of it — and a compiled lift plus a visitor-shaped read are still worth ~3x and ~1.8x on top upstream. The verdict changes with the numbers: at 1.1-1.8x the typed channel is a real option rather than a non-starter, and the case for or against it is now mostly about the schema. That case still has the WIT recursion ban in it, which no runtime work fixes. deno.lock moves 0.5.2 -> 0.6.3 as a consequence of the bump.
The spike measured the two channels, upstream fixed the lift that made the
typed one expensive (polyengine#261, pinned at 22b5d3d), and at 1.1-1.8x the
maintainability win is worth paying for. So the explicit WIT schema is now
the mutation protocol and the hand-rolled byte format is gone.
polymorph:dioxus@0.5.0. One export:
export run: async func() -> stream<operation>
What that deletes, which is the point of the change:
- the normative wire-format prose in wit/world.wit — the opcode table, the
framing grammar, the primitive operand encodings. `interface mutations` is
the schema now, and bindgen owns both sides of it.
- src/protocol.rs's Batch: the opcode constants, utf16_len, the dynstr
escape, the path/strref encoders, take_frame.
- host/src/decoder.ts: the cursor, decodeBatch's 18-arm switch, FrameDecoder's
partial-frame staging.
- tests/vectors.rs and vectors/*: golden byte vectors existed to keep two
hand-written implementations of one undocumented-by-construction format in
agreement. There is one implementation of each side now, generated.
- the surface-probe fixture's inlined 150-line encoder.
Net -3189 lines.
Two things the schema buys beyond deletion: `option<str-ref>` replaces the
0xffff sentinel, which frees the reserved id (the interner's guard is now the
plain u16 ceiling), and every operand's type is checked at the boundary
instead of by a decoder's cursor arithmetic.
One thing it costs, recorded in wit/world.wit and bench/README.md rather than
glossed: WIT forbids recursive types, so register-template carries an index
arena instead of the natural tree, and the arena admits malformed index
graphs a recursive grammar could not express. host/src/operations.ts
validates them.
Also corrects a citation that was wrong in both halves of the tree: the
guest scheduler's persistent park was justified by the host's parked
readDirect session, which the typed channel does not have. The park is legal
by a different clause of the same rule — polyengine#162's retention rule is
disjunctive, and a retained end alone suffices, which mountApp has for the
instance's lifetime. ("A15" is also no longer a real label; upstream pruned
amendment labels, so this now cites the contract section and HostActivity.)
host/tests/operations_test.ts loses its differential reference when the byte
channel goes, so it is now an absolute assertion: literal innerHTML at each
step of the counter interaction sequence, captured while the differential
test was still green and both channels were proven identical.
Gates: cargo check/clippy/test, deno task check, deno task test (90),
all five examples build and validate, and the Playwright lane (8 tests,
real Chromium) which had not been run in this branch before.
Two conflicts, both from main and this branch fixing the same bug independently (bench/ops.ts's stale pre-`source` mountApp call): - The `Translator` type: took main's. Deriving it from the embedder's own `UntranslatedArtifacts["translator"]` is better than my `Awaited<ReturnType<typeof defaultTranslator>>` — it stays pinned to what `mountApp`'s `source` actually accepts rather than to one way of producing one. - A `waitFor` message: took this branch's. Main interpolated the transport name into it; there is one channel now, so there is no transport to name. deno.json needed no resolution — both sides widened the `check` task to cover bench/ with the identical string. .github/workflows/ci.yml merged cleanly and was semantically stale, which git cannot see: its comments described a cross-language golden-vector gate (vectors/ pinning the Rust encoder, decoder_test.ts pinning the TypeScript decoder against the same files) that this branch deletes along with the byte protocol. Corrected to say what actually guards the boundary now — bindgen owns both sides of the WIT schema, and host/tests/operations_test.ts's literal-DOM assertions are the end-to-end net, which is what makes the counter build step load-bearing rather than a nicety. The CI steps themselves needed no change: just deps / check / fixtures / example counter / test are all still the right gates. Ran that exact sequence locally on the merged tree — check, fixtures, counter, test (90 passed).
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The mutation channel's wire format was a hand-rolled byte encoding whose only specification was prose in a WIT doc comment — an opcode table, a framing grammar and a set of operand encodings, implemented twice by hand (Rust encoder, TypeScript decoder) and kept in agreement by golden vectors and review. This replaces it with an explicit WIT schema and deletes the byte format.
polymorph:dioxus@0.5.0, one export:Why now
This branch started as a spike to answer whether the typed schema was affordable. It was not: the typed channel cost ~5 µs per operation, ~5x the byte channel on op-heavy renders and 12x on
clear.That turned out to be almost entirely one runtime's interpreted canonical-ABI lift, not the component model and not the schema. Reported as polyengine#261; upstream landed four fixes; this branch pins 22b5d3d.
At 1.1–1.8x, on a channel whose cost is a small fraction of the DOM work either way, the maintainability win is worth paying for. The full A/B — methodology, instrumented op counts, cost attribution, and the noise-floor caveats — is preserved in
bench/README.mdas a historical section.What this deletes
Deletion is the point of the change; net −3189 lines.
wit/world.wit— opcode table, framing grammar, primitive operand encodings.interface mutationsis the schema now, and bindgen owns both sides.src/protocol.rs'sBatch— opcode constants,utf16_len, the dynstr escape, path/strref encoders,take_frame.host/src/decoder.ts— the cursor,decodeBatch's 18-arm switch,FrameDecoder's partial-frame staging.tests/vectors.rsandvectors/*— golden vectors existed to keep two hand-written implementations of one format in agreement. There is one generated implementation of each side now.What the schema buys, and what it costs
Beyond the deletion:
option<str-ref>replaces the0xffffsentinel, which frees the reserved id (the interner's guard is now the plainu16ceiling), and every operand is type-checked at the boundary instead of by a decoder's cursor arithmetic.The cost is recorded rather than glossed. WIT forbids recursive types —
record template-element { children: list<template-node> }is rejected outright ("typetemplate-nodedepends on itself") — soregister-templatecarries an index arena instead of the natural tree. The arena admits out-of-range and cyclic index graphs a recursive grammar could not express, andhost/src/operations.tsvalidates them. For that one operation, the schema relocates the "two implementations must agree" problem rather than retiring it.Also: no
readDirect(it isstream<u8>-only), so the host pays a copy per batch, anduse mutations.{operation}adds an import to the world.A citation that was wrong in both halves
The guest scheduler's persistent park between renders was justified by the host's parked
readDirectsession, which the typed channel does not have. The park is legal by a different clause of the same rule: polyengine#162's retention rule is disjunctive — "a retained end, a parked host operation, or an unfinished producer pump" — andmountAppholds the lifted readable end for the instance's lifetime. ("A15" is also no longer a real label; upstream pruned amendment labels, so this cites the contract section andHostActivityinstead.)Testing
Deleting the byte channel deletes the reference implementation of the differential test that proved the typed channel correct — a differential test with nothing to differ against proves nothing.
host/tests/operations_test.tsis now an absolute assertion: literalinnerHTMLat each of eight steps of the counter interaction sequence, captured while the differential test was still green and both channels were proven byte-for-byte identical. The provenance is stated in the test header.The
applyOperationsunit tests are unaffected and are now a larger share of the net: arena rehydration including multi-root, rejection of out-of-range and cyclic indices, the non-textattr-valuecases,option<str-ref>present and absent,bubblesboth ways,create-placeholder.Gates:
cargo check/clippy -D warnings/test,deno task check,deno task test(90), all five examples build andwasm-tools validateclean, and the Playwright lane (8 tests, real Chromium) — which had not been run in this branch before this commit.