feat(nat): Add NAT64 egress by generalizing the NAT66 tier into galactic-nat - #525
Merged
Merged
Conversation
ecv
reviewed
Sep 11, 2026
ecv
left a comment
ecv
left a comment
Contributor
There was a problem hiding this comment.
Diff exceeds auto-review threshold (7536 lines across 98 files) — flagging for human review.
1 task
privateip
force-pushed
the
feat/nat64-egress
branch
from
September 12, 2026 12:26
be8f597 to
af7b4a0
Compare
scotwells
previously approved these changes
Sep 12, 2026
privateip
force-pushed
the
feat/nat64-egress
branch
from
September 13, 2026 02:09
7bad333 to
56054ad
Compare
NAT64 and NAT66 are the same function -- stateful egress PAT with a VRF-scoped session table, port allocation, and decap/re-encap on the return path -- over different address families, so this generalizes the existing datapath rather than adding a second, near-duplicate one beside it. nat66prog becomes natprog and nat66.c becomes nat.c, now holding five programs instead of one: a dispatcher, nat_ingress, that classifies a packet and tail-calls one of four translation leaves (nat66_forward, nat66_return, nat64_forward, nat64_return). The split is not stylistic. Holding both families' full translation paths in one program -- each with its own header synthesis, checksum handling, and an unrolled port-allocation probe loop, all force-inlined -- is a verifier complexity risk with no upside, and it is the arrangement this design's own "keep the two families structurally separate" requirement makes worst. Tail calls give each leaf its own instruction budget and keep a defect in one family's translation from growing the other's program. Shared, and the whole reason to combine rather than duplicate: conn_key and conn_value, the port allocator, the tenant session accounting, and the drop counters. conn_key gains a family byte so a NAT64 row, whose IPv4 addresses are stored IPv4-mapped, cannot alias a genuine IPv6 flow in ::ffff:0:0/96 -- the separation is structural rather than merely unlikely. Not shared, because they are genuinely different work: NAT64 rewrites the network header across address families (RFC 7915), which means synthesizing an IPv4 header, computing an IPv4 header checksum that has no IPv6 counterpart, and taking a checksum delta across a pseudo-header whose *family* changed rather than only whose addresses did. Per-tenant session limits are enforced in the datapath at flow-creation time, against one budget shared across both families. A tenant at its ceiling has the triggering packet dropped, never an established session evicted. The limit check is read-only and the increment happens only once a masquerade port is actually claimed, so a failed allocation never consumes budget. tenant_state also separates the two admission-failure causes an operator would act on differently -- at limit, versus this shard not serving the family -- since aggregating them makes the common "why did this connection fail" question unanswerable from counters alone. Scope is TCP and UDP. ICMP translation and fragment handling are absent by decision, not oversight, and each is counted under its own drop reason rather than passed silently, so the gap reads off a counter. serves_v6/serves_v4 gate the whole thing: a shard with no IPv4 fields set never executes an instruction of NAT64 translation, and its dispatch is the NAT66-only program's. Drop reasons 0-8 keep their previous indices so an existing counter series stays the same series. Also fixes a pre-existing NAT66 forward-path defect, found by the regression test added here rather than sought out. strip_outer_header widened the head past the link header and then reclaimed 14 bytes, which exposes the old packet's bytes at that offset -- the tail of the outer destination address and the front of the inner IPv6 header -- rather than a link header, and never wrote one. Every decapsulated packet left the forward path with those bytes as its EtherType, and eth_type_trans dispatches an XDP_PASS frame on exactly that field, so a NAT66-translated packet reached no protocol handler at all. The header is now carried across the move. This is a behavior change to NAT66 rather than an additive one, and is called out separately for that reason; it is not reachable in any deployment today, galactic-nat66 having only ever run in containerlab. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…e, and CNI Renames the whole component from galactic-nat66 to galactic-nat, and extends every layer above the datapath to describe a shard that may serve NAT66, NAT64, or both. Config gains the IPv4 half as an all-or-nothing pair: a public IPv4 address with no NAT64 prefix has nothing to translate for, and a prefix with no address has nothing to translate into. Either alone yields a shard that drops every NAT64 packet sent to it, so both are rejected at startup rather than at the first packet. The IPv6 public address stops being unconditionally required, a shard now being allowed to serve NAT64 alone -- what stays required is that it serve something, since a shard serving neither family loads a datapath that claims nothing and presents as a blackhole rather than as the misconfiguration it is. The prefix must be a /96 with no bits below its length. That is the only RFC 6052 length placing the embedded IPv4 address in one aligned four-byte run, which is what lets the datapath extract it without a bit-shuffle on the hot path. NATConfig.ServesNAT66/ServesNAT64 give that decision one home, so the binary and the reconciler read the same answer instead of each re-deriving it from which fields happen to be non-empty. EgressShardReconciler publishes the shard's SID, both masquerade addresses, and the NAT64 prefix. It deliberately does not advertise the IPv4 address: a NAT64 reply arrives from the IPv4 internet rather than across this fabric, so a BGPAdvertisement into the EVPN mesh would not make it reachable by the party that needs to reach it. Publishing it in status is what makes that underlay prerequisite checkable rather than implicit. CNI now installs two egress routes per VRF: the existing ::/0 default and, where the fabric has NAT64, a more-specific route for the NAT64 prefix. Both point at the same shard SID -- a shard decides which translation a packet gets from its inner destination, not from which route carried it -- so the second route exists to make that prefix reachable where no default covers it. The two really are independent: a fabric may offer NAT64 without NAT66, and then no default route exists for that traffic to fall into. EgressDefaultRouteAdd becomes a thin wrapper over a prefix-general EgressPrefixRouteAdd, leaving its existing callers and tests untouched. The binary enables IPv4 forwarding on the uplink when serving NAT64. Without it the kernel drops every translated packet after the datapath has already counted it as successfully translated, which reads as a healthy shard with a silently broken path. A periodic resync recomputes per-tenant session counts from the connection table. The datapath cannot keep that count accurate alone: nat_conn_table is an LRU map, so the kernel evicts rows with nothing to decrement, and nothing ages an idle flow out. Left alone the count only climbs until every tenant reads as over limit -- a per-tenant ceiling degrading into a dead shard. Counting forward rows alone is what makes the total right, every flow owning one forward row and one reverse row with only the forward row carrying a VRFID. Metrics now break connection counts down by family and expose each tenant's sessions, limit, and the two admission-failure causes separately. Both families are reported even at zero, a family absent from the output reading as "not scraped" rather than "no flows". The session limit defaults to unlimited on purpose. A ceiling nobody chose becomes an outage nobody can explain, since this design collects the admission-failure counters but does not surface them anywhere a tenant or an on-call engineer reads. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…ic-nat Follows the binary and package renames through every asset that names them: config/galactic-nat/, containers/galactic-nat/, docs/nat/, the containerlab overlay, and the Taskfile. The manifests and the configuration reference now describe a shard as serving NAT66, NAT64, or both, rather than as a NAT66 shard with required fields. The DaemonSet base documents each new environment variable alongside the two things about them that are not inferable from the field names: the NAT64 pair is all-or-nothing, and nothing in this repo makes GALACTIC_NAT_SHARD_PUB_ADDR4 reachable, since a NAT64 reply arrives from the IPv4 internet rather than across the fabric. The containerlab overlay stays NAT66-only, and says why. This lab's transit mesh is IPv6-only with no IPv4 upstream, so enabling NAT64 there would produce shards that translate outbound traffic correctly and then blackhole it -- indistinguishable, from every counter this component exposes, from a NAT64 deployment that is simply broken. The README records what turning it on would require and which validation the lab therefore does not provide. Also adds galactic-nat to .github/workflows/publish.yaml, which never built it. config/galactic-nat/base/daemonset.yaml has always referenced ghcr.io/datum-cloud/galactic-nat while no job published that image and no bundle entry stamped its tag, so the component could not be applied from a published bundle at all. This is the same failure the repository already hit once with galactic-router, recorded in CLAUDE.md's own history note -- a manifest advertising an image nothing builds. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Follows the docs/nat66 -> docs/nat move through the index and the architecture docs, and drops "NAT66" from the phrases that now describe a tier serving both address families. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The IPv6 and IPv4 masquerade addresses are the same kind of thing -- one public address per family a shard serves -- so the pair reads as a pair: GALACTIC_NAT_SHARD_PUB_ADDR6 alongside _ADDR4, --nat-shard-pub-addr6 alongside --nat-shard-pub-addr4, shard_pub_addr6 alongside shard_pub_addr4. The unsuffixed name was carried over from when NAT66 was the only thing this component did, and leaving it made the asymmetry look deliberate: readers had to know that the bare name meant IPv6 specifically rather than "the shard's address" generally. EgressShard's own status fields were already symmetric (shardAddressIPv6/shardAddressIPv4), so the inconsistency was internal to this repo, not just cosmetic. Renamed through every layer that names it -- the eBPF struct field, the map layer, the config struct, the flag, the env var, the manifests, the lab overlay, and the docs -- rather than only the env var, since a half-applied convention is worse than either one applied consistently. No deployment is broken that this PR had not already broken: every GALACTIC_NAT66_* variable is renamed here regardless, so a shard's environment has to be rewritten either way. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Removes the session ceiling, the per-tenant accounting map behind it, the userspace resync that kept its counts honest, and the metrics that exposed them. The feature did not work per tenant. Egress route installation registers the operator-configured shard SID verbatim, so every tenant VRF encapsulates to the same destination and the datapath reads the same SRv6 Argument for all of them. Confirmed on the wire in the gvpc lab: three tenants in three VPCs all encapsulated to one address, and one counter absorbed all of them. What shipped was therefore a per-shard ceiling wearing per-tenant clothes -- set it to any finite value and the first tenant to reach it locks out every other tenant on that shard. Keeping it would have meant carrying the machinery for a guarantee it cannot make. Fixing it properly needs the Argument threaded through route installation, and that needs a design decision about which Argument a shard SID should carry, since shard-SID Argument space is per-shard-locator while attachment Arguments are allocated per BGPRouter. That is not MVP work. The isolation the limit depended on is unaffected and still correct where it is exercised: conn_key keeps its family byte and its tenant_arg, and the datapath honors whatever Argument arrives. Only the accounting built on top of it is gone. DROP_REASON_NAT64_SHARD_UNAVAILABLE survives as a shard-wide counter. It was the one admission-failure cause that never depended on per-tenant state, and it stays the signal for a half-configured NAT64 shard. The remaining drop reasons keep their indices; only the trailing tenant-limit reason is removed, so no existing counter series changes meaning. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…rgument alone Two tenants on different nodes could share one connection row, and the second tenant's replies were then re-encapsulated toward the first tenant's node. That is cross-tenant delivery, not a dropped packet. The Argument was carrying tenant identity alone, and cannot. Arguments are allocated per BGPRouter -- allocateArgument scans one router's own BGPVRFInstances -- so two tenants on different nodes holding the same value is ordinary rather than exceptional. Today it is worse than that: nothing writes a per-tenant Argument into a shard SID at all, so every tenant presents whichever constant the operator configured. Tenant identity is now composed from two fields that are each already in the packet: the SRv6 outer header's source, unique per worker node, and the Argument, unique per node. Neither identifies a tenant fabric-wide; together they do. No new identifier, no allocation change, and no control-plane round trip on the attach path. This does not need the Argument to be correct to help. It is the half of the fix that is safe to make now; threading a per-tenant Argument through route installation is the other half, and needs the shard to advertise a prefix covering its Argument space rather than the single /128 it advertises today. Until that lands, the outer source is doing all the work, which closes every cross-node collision and leaves only same-node ones. The probe loop now mutates the reverse key in place instead of copying it per iteration. A copy inside an unrolled loop is a second whole key on the stack, and with the key 16 bytes wider the program no longer fits in BPF's 512-byte stack. The caller has no use for the key afterward, so there was nothing to preserve. Decoding leaves an all-zero encapsulation source as the zero Addr rather than "::", so a reverse row -- which carries no such source, its uniqueness coming from the shard's own address and masquerade port -- stays distinguishable from a forward row and the wire round trip stays symmetric. The regression test fails without this, both tenants receiving the same masquerade port and the table holding two rows where four belong. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
… ULA Every existing lab tenant is given a site-unique ipv6_subnet, so no two of them ever present the same inner source address to a shard and the connection key's collision path is never reached. The datapath change that key exists for was covered only by a unit test. ns70 gives iad and sjc the identical subnet, so a pod on each holds fd20:70:ffff::100:0. Both sites resolve dfw's shard SID first -- a node cannot resolve its own self-originated SID -- so both flows arrive at dfw-worker's shard encapsulated from different nodes. Sending an identical 5-tuple from each makes every field of the connection key identical except the encapsulation source. That is not a contrived shape. ULAs are RFC 4193 locally assigned and not globally coordinated, so two tenants picking the same prefix is a real possibility, and it is exactly the case the SRv6 Argument was supposed to disambiguate and does not, because nothing yet writes a per-tenant Argument into a shard SID. verify:nat-collision drives it and asserts the two tenants receive distinct masquerade ports. Confirmed live: 37584 and 37585, four connection rows rather than two, no drops, and a repeat of both flows reusing their own rows rather than each other's. The ports come out adjacent because identical hash inputs give both flows the same probe starting point and the second claims the next candidate -- the allocator separating them only because the keys already differ. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…ds exist verify:nat-sharding reads EgressShards and BGPAdvertisements. That proves the control plane agrees with itself and nothing about whether a packet is translated -- a shard can report Ready with an attached datapath while every packet it handles is silently discarded downstream, which is close to what was actually happening before the forward path's link header was fixed. verify:nat-egress drives a real packet from a dfw tenant and reads what comes out on sjc's uplink. It asserts the three things that have each been broken at some point: the EtherType (a decapsulated frame needs its link header rebuilt, or the kernel dispatches it on stale packet bytes and it reaches no protocol handler at all), the source address, and the masquerade port's range. It also requires the connection table to have grown and the shard to have recorded no drops. The sender and the capture are deliberately on different sites: a node cannot resolve its own self-originated shard SID, so dfw's tenants fall through to sjc's shard. A single-site version of this test would pass against a shard that never receives anything. The shard address is read from EgressShard status rather than hardcoded, so the test exercises the reconciler's published value and cannot drift from whichever address the shard is actually configured with. Each assertion was checked to fire on the input it is meant to reject, not only to pass on good input. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
privateip
force-pushed
the
feat/nat64-egress
branch
from
September 13, 2026 02:27
19c0bb6 to
873cd9d
Compare
…h run The check sent from a fixed source port, so every run presented the same 5-tuple. Connection rows live in a pinned map and survive a DaemonSet restart, so a rerun found the existing rows and re-asserted the two ports an earlier run had allocated -- without touching the allocator at all. Observed directly: after rebuilding and rolling the DaemonSets, the check reported the same pair of ports as the previous run. That is the failure mode where a test keeps passing after the code it covers stops working. Varying the source port per run makes each run create new flows, so the separation is proven fresh rather than remembered. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
scotwells
previously approved these changes
Sep 13, 2026
…with Every containerlab operation on the gvpc topology after a successful deploy failed to resolve three of its own nodes: Failed to list containers based on topology: could not get container for node iad-worker2: Node: iad-worker2. containers not found. Kind creates iad's extra workers sequentially as iad-worker2/3/4, and deploy:rename-control renames them to iad-worker-rr/iad-gateway1/ iad-gateway2 to match the Kubernetes node names their kubeadmConfigPatches set. gvpc.clab.yaml named the pre-rename containers. Deploy itself worked, because links are wired once while the old names still apply -- which is what the file's own comment claimed made the rename "purely cosmetic". It is not cosmetic for inspect or destroy, which resolve the topology against whatever is running now. Fixed by naming the containers here as they end up, and moving deploy:rename-control ahead of deploy:topology so they already carry those names when containerlab attaches to them as ext-containers. The alternative -- dropping the rename -- would have left the container and Kubernetes node names disagreeing, which is what the rename exists to fix and what makes `docker exec <node name>` work across the lab's scripts. Verified against the running lab: all 13 nodes resolve, no errors, and the NAT verify tasks still pass. The reordering itself is not exercised until the next full deploy. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
galactic-cni's credential-refresh container crash-looped on startup, taking the DaemonSet rollout with it: R8 has pointer with unsupported alu operation, pointer arithmetic with it prohibited for !root load program: permission denied "!root" is the tell. The container does run as root; what it lacks is CAP_PERFMON, without which the verifier applies its unprivileged ruleset and refuses pointer+scalar arithmetic on packet data regardless of uid. config/galactic-gateway and config/galactic-nat both grant PERFMON alongside BPF and document why. This container never did, and did not need to: usid.c stayed inside what unprivileged BPF permits until the GRO-merged decap path began offsetting a packet pointer by a computed header length (bf50804, "Forward GRO-merged return traffic through SRv6 decap"). The program had been loading on the strength of not yet doing anything that required the capability, which is luck rather than a property to depend on. Confirmed by patching the running DaemonSet: the container reaches Ready with no restarts and logs "eBPF uSID datapath loaded, pinned, and attached". This is an independent regression rather than anything to do with the NAT work carried on this branch -- it is fixed here because it blocks deploying the branch at all. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Collaborator
Author
|
@scotwells this PR is now ready for final review |
ecv
approved these changes
Sep 13, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Implements enhancements#879. Depends on network#22 (merged; go.mod points at it).
IPv6-only instances get a path to the IPv4 internet.
galactic-nat66becomesgalactic-nat, serving NAT66, NAT64, or both from one session table, one port allocator, and one CRD.Where this follows the design, and where it doesn't
Shared, as designed: the session table, port allocator, tenant accounting, counters, CRD, and route installation.
conn_keygains a family byte so a NAT64 row's IPv4-mapped addresses cannot alias a real IPv6 flow in::ffff:0:0/96.Split, against the letter of the design: the two translation cores are separate eBPF programs behind a tail-call dispatcher, not branches in one program. NAT66 is pure PAT — the IPv6 header stays an IPv6 header. NAT64 synthesizes an IPv4 header, computes a header checksum IPv6 has no counterpart for, and takes a checksum delta across a pseudo-header whose family changed. Inlining both into one program is a verifier-complexity risk, and the design's own "keep the families structurally separate" requirement makes that arrangement worst. Tail calls give each leaf its own instruction budget.
Scope reduced: TCP and UDP only. ICMP translation and fragment handling are absent, so this is not yet a complete RFC 6146 implementation and PMTUD across the translator does not work. Each gap increments its own named drop counter rather than passing silently.
Test plan
task lint— cleantask test:unittask test:e2e— needs a Kind clusterdeploy/containerlab/resources/galactic-nat/README.mdrecords what turning it on would require.