Skip to content

feat(core): encode/decode UnsignedDataSet proto for all duty types#508

Merged
emlautarom1 merged 6 commits into
mainfrom
emlautarom1/core-unsigned-proto-codec
Jul 1, 2026
Merged

feat(core): encode/decode UnsignedDataSet proto for all duty types#508
emlautarom1 merged 6 commits into
mainfrom
emlautarom1/core-unsigned-proto-codec

Conversation

@emlautarom1-agent

@emlautarom1-agent emlautarom1-agent Bot commented Jun 30, 2026

Copy link
Copy Markdown
Contributor

Summary

Implements the all-duty-type UnsignedDataSet protobuf codec in pluto-core, so unsigned duty data for every duty type can be carried over the consensus wire and decoded by peers.

What it implements

  • unsigned_data_set_to_proto(&UnsignedDataSet) -> Result<pbcore::UnsignedDataSet, ParSigExCodecError> — encodes attester, proposer, aggregator, and sync-contribution unsigned data.
  • unsigned_data_set_from_proto decoding for proposer, aggregator, and sync-contribution duty data (attester was already supported).
  • SSZ encode/decode helpers in ssz_codec for the unsigned VersionedProposal, VersionedAggregatedAttestation, and SyncContribution.

The encoding is SSZ-first with a JSON fallback on decode, matching Charon's UnsignedDataSet wire format so a Pluto node interoperates with Charon nodes in a mixed-client cluster.

The JSON-fallback path reuses the existing versioned deserializers in signeddata instead of duplicating the per-fork dispatch: VersionedAggregatedAttestation decodes through VersionedAttestation's Deserialize, and a new Deserialize for VersionedProposal is added beside its signed sibling (sharing the VersionedRawBlockJson wrapper). The looks_like_json prefix check is shared with parsigex_codec. As a result the diff also touches signeddata.rs and parsigex_codec.rs.

Ports charon/core/proto.go + core/unsigneddata.go. Part of #402.

Testing

Round-trip (encode → decode) unit tests for all four duty types, plus the non-versioned aggregator and JSON-fallback decode paths.

Add the all-duty-type proto codec covering attester, proposer,
aggregator, and sync-contribution unsigned data. The encoder/decoder
is wire-compatible with Charon's SSZ-first UnsignedDataSet format
(with legacy JSON fallback on decode), unblocking non-attester duties
through consensus.
@emlautarom1-agent emlautarom1-agent Bot force-pushed the emlautarom1/core-unsigned-proto-codec branch from f5013ec to 5e0331f Compare June 30, 2026 20:37
…allback

The aggregator JSON fallback re-implemented the per-fork dispatch and
validator-index parsing already provided by signeddata::VersionedAttestation's
Deserialize impl. Since VersionedAggregatedAttestation is a newtype over the
same inner type, deserialize directly into VersionedAttestation and rewrap,
dropping the redundant VersionedAggregatedAttestationJson wrapper. This also
gains the version/payload validation from VersionedAttestation::new.
… JSON

The proposer JSON fallback re-implemented the per-fork (version, blinded)
dispatch that VersionedSignedProposal's Deserialize already does. Add a
Deserialize impl for the unsigned VersionedProposal in signeddata next to its
signed sibling, sharing the VersionedRawBlockJson wrapper, and have the codec
just call serde_json::from_slice. Drops the one-off VersionedProposalJson
wrapper and its parse_json/block_field/field/parse_default helpers.
looks_like_json was defined three times (a nested fn in parsigex_codec and
two copies in unsigneddata, one of them inline). Promote the parsigex_codec
copy to a pub(crate) fn and reuse it everywhere.
charon writes a two-slot offset table for AttestationData even though both
fields are fixed-size; ssz_derive would omit it, so the manual layout is
required for wire compatibility. Document this to prevent a future regression.
@emlautarom1 emlautarom1 marked this pull request as ready for review June 30, 2026 21:53
@emlautarom1

Copy link
Copy Markdown
Collaborator

@claude

@claude

claude Bot commented Jun 30, 2026

Copy link
Copy Markdown

Claude finished @emlautarom1's task in 3m 44s —— View job


Code Review: UnsignedDataSet proto codec

  • Read the PR diff (unsigneddata.rs, ssz_codec.rs, signeddata.rs, parsigex_codec.rs)
  • Cross-check against Charon Go source (core/proto.go, core/unsigneddata.go, core/ssz.go)
  • Review correctness, style, and tests
  • Post review feedback

Overall this is a careful, well-documented port. The doc-comments that explain why the layout matches Charon (the hand-rolled attester offset table, the SSZ-first/JSON-fallback rationale) are excellent and make the parity easy to verify. Functional equivalence with Charon checks out across the board. I have one substantive concern about test coverage and a few minor notes.


🟠 Main concern — the Deneb/Electra/Fulu BlockContents SSZ layout is untested

The most novel, highest-risk code in this PR is the two hand-declared containers in ssz_codec.rs:

  • DenebBlockContents (crates/core/src/ssz_codec.rs:599)
  • ElectraBlockContents (crates/core/src/ssz_codec.rs:610)

These are brand-new structs whose byte layout must match Charon's eth2deneb.BlockContents / eth2electra.BlockContents exactly for cross-client interop — but no test exercises them:

  • The proposer round-trip (round_trip_proposer) and the proposer JSON-fallback test both use sample_versioned_proposal_phase0 only (unsigneddata.rs:525). Phase0 never goes through DenebBlockContents/ElectraBlockContents.
  • Unlike the signed codecs, the new unsigned types have no Go golden-vector fixtures under crates/core/testdata/ssz/ (the existing go_fixture_* tests at ssz_codec.rs:1137+ only cover signed/attestation types).

The consequence: a field-order or offset bug in the block-contents layout (the exact failure mode the doc-comment at ssz_codec.rs:587 warns about for the attester case) would pass every test in this PR while silently breaking interop with Charon nodes for Deneb+ proposals — which is essentially every proposal on mainnet today.

Two suggestions, ideally both:

  1. Add a Deneb (and Electra) round-trip case to round_trip_proposer so the new containers are at least exercised end-to-end within Pluto.
  2. Add Go golden-vector fixtures for an unsigned Deneb VersionedProposal, a VersionedAggregatedAttestation, and a SyncContribution, mirroring the existing go_fixture_versioned_proposal_phase0 pattern. This is what actually proves charon wire-compat for the new types.

Add Deneb/Electra round-trip + Go golden fixtures for unsigned types →


🟡 Minor

  • Unbounded blob/proof lists on decodeDenebBlockContents/ElectraBlockContents use plain Vec<deneb::KZGProof> / Vec<deneb::Blob> (ssz_codec.rs:601-602, 612-613). Charon's fastssz enforces the MAX_BLOBS_PER_BLOCK / commitment list bounds on decode; a derived Vec does not. Wire bytes round-trip fine, so this isn't an interop bug, but a peer could send an over-length list that decodes successfully here. Worth a comment noting bounds are enforced upstream (or adding a length check) if the p2p message-size cap isn't considered sufficient.

  • looks_like_json evaluated twicedecode_versioned_proposal (unsigneddata.rs:161 & 167) and decode_sync_contribution (unsigneddata.rs:226 & 232) call looks_like_json(data) twice. Minor; a single let is_json = looks_like_json(data); reads cleaner. (The attester path at unsigneddata.rs:242 already avoids this by trying SSZ unconditionally first, which is also the closest match to Charon's unmarshal.)

  • Documented ordering divergence (no action needed) — Charon's unmarshal (core/proto.go:287) always attempts SSZ first, then JSON only if the { prefix is present. The short-circuit form here (skip SSZ when looks_like_json) is functionally equivalent because a {-prefixed buffer can't be a valid versioned-SSZ header (first byte 0x7B ⇒ invalid DataVersion). Just flagging it's an intentional, safe reordering rather than a literal port.


✅ Verified against Charon

  • Header sizes match core/ssz.go constants: proposal version(8)+blinded(1)+offset(4)=13, aggregate version(8)+offset(4)=12 (versionedBlindedOffset/versionedOffset, ssz.go:636).
  • Aggregator decode reproduces Charon's two-stage fallback: versioned → non-versioned AggregatedAttestation (raw phase0), for both SSZ and JSON paths (unsigneddata.rs:184 vs unmarshalUnsignedData unsigneddata.go:674).
  • Deserialize for VersionedProposal faithfully mirrors VersionedProposal.UnmarshalJSON (unsigneddata.go:490), including the pre-merge-blinded rejection and Deneb+ BlockContents {block, kzg_proofs, blobs} shape with optional/null-tolerant fields.
  • Electra/Fulu aggregated attestations correctly route through electra::Attestation (committee bits), matching VersionedAggregatedAttestation.sszValFromVersion (ssz.go:619).
  • SyncContribution is a bare altair::SyncCommitteeContribution with no versioned header, matching core/unsigneddata.go:618.
  • Block values (consensus_block_value/execution_payload_value) correctly dropped from the wire and defaulted to zero on decode, as in Charon.
    · branch emlautarom1/core-unsigned-proto-codec

@emlautarom1 emlautarom1 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Manually checked, LGTM. The serialization/deserialization code is just messy by nature due to this JSON/SSZ split.

Comment thread crates/core/src/parsigex_codec.rs
@emlautarom1 emlautarom1 merged commit c241b50 into main Jul 1, 2026
15 of 16 checks passed
@emlautarom1 emlautarom1 deleted the emlautarom1/core-unsigned-proto-codec branch July 1, 2026 17:21
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants