Conversation
Contributor
Author
|
Example just showing the (fake) router fronting traffic to the containerlab VPC: |
Contributor
Author
|
Example attaching a real VM host interface to the fake VPC in containerlab: |
Contributor
Author
Validated cross-machine: Mac at home ↔ VPC on a DC VM, over the public internetClient on a bare-metal macOS laptop at home, the VPC (router + 3 regions) on a Linux VM in a data center, dialed by iroh On the DC VM (docker) — stand up the containerized VPC: On the Mac — join the VPC (dialed by id), then verify client→VPC: Back on the VM — router dials the Mac by endpoint id, verifies VPC→client: Full write-up in |
Bumps iroh/iroh-base/iroh-relay 0.95->1.2.0, iroh-tickets 0.2->1.0, and
iroh-proxy-utils to its current HEAD (the first rev with real iroh-1.0
support). Mechanical fallout from the major bump:
- discovery -> address_lookup: iroh 1.x replaced the pluggable Discovery
trait with AddressLookup/.address_lookup(...). Datum's custom-origin
DNS discovery has a direct successor in
address_lookup::DnsAddressLookup::builder(origin), so
DiscoveryMode::{Dns,Hybrid,Default} behavior is preserved.
- Endpoint::builder(..).bind() now hard-errors without an explicit
crypto_provider (no more implicit global-default fallback); connect-lib
now depends on rustls directly to supply one.
- iroh_tickets::Ticket renamed to_bytes/from_bytes to
encode_bytes/decode_bytes; SecretKey::generate() dropped its RNG arg.
- iroh-n0des doesn't support iroh 1.x (still pinned to iroh ^0.96) and
briefly regressed metrics collection; n0-computer's 1.x-compatible
successor is published as iroh-services (same Client/ApiSecret shapes),
so ListenNode/ConnectNode's n0des plumbing now points at that crate
instead. N0DES_API_SECRET/BUILD_N0DES_API_SECRET env vars are unchanged.
Verified with cargo build/test/clippy --workspace (91/91 tests passing).
Adds the client-side data plane for datum-cloud/enhancements#894: a `datumctl connect vpc join` verb that creates a local TUN interface and joins a galactic VPC as a remote member over an iroh connection, the sibling of `tunnel` but carrying raw IPv6 packets to a galactic-side router instead of HTTP requests to Envoy. connect-lib: - datum_apis/vpc_attachment.rs: the VPCAttachment CRD type (spec + status contract — assigned address, advertised prefixes, MTU, claiming router's iroh EndpointId), a Layer-3 analogue of ConnectorAdvertisement over the same reusable Connector primitive. - vpc/{mod,transport,routing}.rs: TUN creation, VpcListener/ VpcAcceptHandler (accept side — the client is dialed in, as tunnel's client is dialed by Envoy), VpcDialer (dial side), length-prefixed packet framing over one iroh bidi stream on a new IROH_VPC_ALPN, and wg-quick-style routing-mode installation (vpc-only vs default-route with the ::/1+8000::/1 split and the anti-loop gateway pin). - The WireGuard-linux interface/routing model is the reference; its crypto/handshake/keepalive/roaming are all subsumed by iroh. AllowedIPs collapses to a mode toggle (one peer) plus a single-EndpointId source check. - A unit test exercises the framing via tokio::io::duplex (no root). bin: `vpc join` subcommand; new mock-galactic-router crate stands in for the (imagined, not-yet-built) galactic-side router for testing. connect-plugin: `vpc` cobra tree mirroring tunnel's supervisor pattern. --router-id is optional (trust-on-first-connect) so the client can start unattended before any router has claimed its attachment — the real control-plane flow, where the client necessarily starts first. Once CRD-backed, "no router yet" must mean "don't accept," not "accept anyone" (flagged in code and design/vpc-attachment.md §6). Privileged ops (TUN creation, `ip` commands) fail with a CAP_NET_ADMIN hint instead of elevating — nothing invokes sudo. deploy/containerlab/vpc: a lab proving the data plane end to end (TUN creation, iroh accept/dial handshake, framing, routing) against the mock router. Verified via docker run: real TUN devices across two containers, 0% packet loss pinging both directions. Not yet wired: `vpc join` takes the attachment's effects as flags rather than watching a real VPCAttachment; the galactic-side router is imagined, not built (see design/vpc-attachment.md §5–6).
Adds a 3-region VPC lab (mirroring galactic's dfw/sjc/iad model): a router
sits on the VPC, the local device joins through it over iroh, and a
full-mesh ping matrix proves every device reaches every other.
- deploy/containerlab/vpc/test-3region.sh: self-contained on plain Docker
(no containerlab/root beyond Docker access), idempotent, non-zero exit on
any failure. The router is a plain IPv6 L3 hub — kernel forwarding moves
packets between the three regional /64 segments and the tunnel TUN, while
mock-galactic-router only pumps the tunnel. That needed no code changes:
it's the same split a real galactic router has. Verified: all 12 ordered
pairs reachable, repeatable across runs.
- deploy/containerlab/vpc/vpc-3region.clab.yaml: the same topology in
containerlab form (parses/checks clean; deploy needs root, supply it
yourself).
- routing.rs: correct install_routes' doc — routes are plain dev routes out
the tunnel (one peer, the router, so no next-hop gateway is needed), and
the advertised prefix must actually cover the target subnets. During
bring-up a /48 aggregate silently excluded fd00:cafe:{a,b,c}::/64 (whose
third 16-bit group differs), so the lab uses /32; documented in the script
and design doc §7 so it doesn't bite again.
No sudo anywhere; privileged ops still fail with a CAP_NET_ADMIN hint.
The router joins the full-mesh matrix at its tunnel-side address (fd00:cafe:100::1), so its own stack is verified reachable from every device and as a source to every device — 5 devices, 20 ordered pairs, all passing. Previously the router was only exercised implicitly as the forwarding hub on the client<->region paths.
test-host-client.sh runs the `vpc join` client on the host (real TUN in the host netns) against the 3-region VPC in containers. The router dials the host over iroh at the docker transport gateway (verified reachable); the host TUN step needs the caller's root, so it's a three-step up / (sudo join) / dial flow. Verifies host<->VPC reachability both directions.
The mock router previously required --peer-addr <ip:port>, which defeats the point of iroh — you dial by identity and discovery resolves reachability. mock-galactic-router now builds its endpoint with connect-lib's build_endpoint (same relays + discovery every client gets) and dials by EndpointId; the client is resolvable because it already publishes to iroh discovery. Both test scripts drop the port: they scrape only the endpoint id and the router dials by id. --peer-addr stays as an optional offline/same-host pin. Verified: 3-region full mesh (20 pairs) with the router connecting purely by endpoint id via discovery. Matches the production design (§5): the real router dials the client's EndpointId published under its Connector status.
…ab subnets
- Both labs now retry each reachability ping (5x/1s) instead of a single
shot. Dialing by endpoint id brings the connection up via a relay path
first, then upgrades to a direct path; a brief blip during that upgrade can
drop a packet or two, which shouldn't fail a reachability check. A genuinely
unreachable pair still fails after all attempts, so this doesn't mask breaks.
(Fixes an intermittent `host -> router` failure seen in the host-client lab;
the all-containers lab's local->router is 15/15 solid, confirming the mock
logic is stable once the path settles.)
- test-host-client.sh now uses fd00:cafe:1{a,b,c}::/64 for its regions,
distinct from test-3region.sh's fd00:cafe:{a,b,c}::/64, so the two labs'
docker networks can't overlap ("Pool overlaps" errors) across leftover runs.
The host is the docker host, so it had a direct bridge route to every region
network — host->region pings shortcut the tunnel entirely, and the convergence
gate (which pinged a region) passed instantly via that bridge, letting the
matrix run before the iroh path had settled. That surfaced as an intermittent
host->router failure (the one genuinely tunnel-only check); once the relay->
direct path upgrade completes it's a clean ~0.8ms direct path.
Fixes, matching the intended architecture (docker plays no part in host<->VPC):
- Split the docker router<->region fabric (fd00:d0c:1{a,b,c}::/64) from the VPC
prefix. Each region's VPC address (fd00:cafe:1{a,b,c}::1) is now a /128 atop
its docker link, routed only through the router. The host has no route into
fd00:cafe:: except the TUN, so every host->VPC packet goes through iroh.
- Gate on the real tunnel path (host -> router's tunnel address) with a 90s
settle window, instead of a region ping that never touched the tunnel.
Verified the VPC fabric (router->region, region->region over the /128 routes)
container-side; the host->tunnel path needs the caller's sudo to exercise.
The client and the VPC no longer have to be on the same machine — a laptop can join a VPC running on a remote VM. Split the docker-side and client-side steps so the client side needs no docker (it was failing with "docker: command not found" when run off-host): - dial (VM/docker) now drives the router and verifies only the VPC->client direction (region -> client TUN, via docker exec), gating on that instead of a "ping from this host" that assumed the client was local. - New client-check (client machine, no docker) pings the whole VPC from the client for the client->VPC direction. - up's output and the README explain which command runs where, how to place the binary + credentials helper on a remote client, and that only one client may hold the VPC address at a time. client-check verified to run with docker absent from PATH.
The iroh 1.x upgrade bumped iroh-proxy-utils to 0.3.0 at rev e87788bb, but the
flake's cargoLock outputHashes still keyed the old iroh-proxy-utils-0.1.0 hash,
so `nix build` failed ("A hash was specified for iroh-proxy-utils-0.1.0, but
there is no corresponding git dependency"). Rekey to 0.3.0 with the vendored
hash nix reports for the new rev.
The vpc client only worked on Linux: it forced a `datum-vpc0` TUN name (macOS requires utunN → "invalid device tun name") and shelled out to iproute2 (`ip`), which macOS doesn't have. Since issue #894 targets the Mac, add macOS support: - create_tun_device: on macOS, let the kernel assign the utunN name (a non-utun name is rejected) rather than forcing one. New device_name() reads the actual kernel-assigned name back; both binaries now configure routes/addresses using that name and report it. - routing.rs: OS-gate configure_interface/install_routes — iproute2 on Linux, ifconfig + route on macOS (address assignment, interface routes, the DefaultRoute host-route pin, and default-route lookup all have macOS forms). - No data-plane change: the tun crate hides the macOS utun 4-byte protocol header, so the packet pump is identical. Linux builds/tests/clippy pass; the macOS path is written per-target but not yet run on real hardware (no Mac in CI here) — see design/vpc-attachment.md §6.
client-check ran the Linux ping form (`ping -6 ... -W2`) on the client machine. On macOS `-W` is milliseconds (Linux: seconds), so `-W2` timed out at ~2ms and every probe failed — "tunnel did not settle" — even though the tunnel was up (the VM-side dial checks passed because they ping inside Linux containers). Add host_ping1(): on macOS use `ping6 -c1 -W2000` (2000ms), on Linux `ping -6 -c1 -W2`. hping and the client-check settle gate use it; container pings (cping, dial's gate) stay on Linux ping.
On macOS, assigning the utun address with a subnet prefix (/64) made the kernel install a connected route whose next hop is the utun's own link-local, and it then chose that link-local as the source for destinations in that subnet — notably the router's own tunnel address (fd00:cafe:1100::1) — which the router can't answer across the tunnel. Symptom: client-check's `client -> router` failed while `client -> region` (a different /prefix, routed via the explicit VPC route) and `ping6 -S <utun-addr> <router>` both worked. Assign the address as a host (/128) on macOS so no connected subnet route is created; every VPC address then follows the explicit VPC-prefix route with the correct global source. The tunnel is point-to-point (one peer), so an on-link subnet on the interface isn't needed. Linux keeps the caller's prefix length.
The /128 attempt regressed the working direction (region -> client broke), so the /64 assignment is restored — with /64 the data plane worked in both directions; the only failure was an auto-source `ping6` to the router's own tunnel address, which macOS sources from the utun link-local (a source the router can't answer) because that address shares the client's on-link subnet. Rather than re-architect the interface addressing, client-check now forces the source to the client's VPC address on macOS (`ping6 -S <vpc-addr>`), which the earlier manual test confirmed works. Reaching VPC members in other prefixes was never affected. Documented the on-link source-selection quirk and a cleaner follow-up (on-link route as an interface route) in design/vpc-attachment.md §6.
macOS ping6's -W is a valueless flag, not a milliseconds waittime like Linux
ping / macOS ping(8), so `-W2000` parsed as invalid ("invalid option -- 2")
and every client-check probe errored out. macOS ping6 has no per-packet reply
timeout flag; drop it and use `ping6 -c1 -S <vpc-addr> <dst>`, which returns
promptly when the tunnel is up (confirmed: `ping6 -c3 -S <addr> <router>` gets
replies). Linux keeps `ping -6 -c1 -W2`.
…d run Records the end-to-end validation with the client on a bare-metal macOS laptop and the VPC on a Linux VM, connected over the public internet dialed by iroh EndpointId — a topology diagram plus the up / vpc join + client-check / dial transcripts.
…UN <-> iroh forwarder Lock all box borders to consistent columns (left/right walls, nested router box, docker-bridge and region-box connectors) and relabel the client-side box from the internal '[packet pump]' to '[TUN <-> iroh forwarder]' so it reads without knowing the code.
Member
|
Thanks, going to give this a spin |
The router now allocates each client's IPv6 address from a pool and
sends it as an Assignment frame (JSON, length-prefixed) on the iroh
stream before packet pumping begins — the wire analogue of the future
VPCAttachmentStatus.assignedAddress CRD flow. Clients no longer need
--address, --prefix-len, or --vpc-prefix; the TUN is created
unaddressed and configured from the assignment.
Wire protocol: bumps ALPN to datum-connect/vpc/1. The first frame on
every stream is a JSON-encoded Assignment{address, prefix_len,
vpc_prefixes}, followed by raw length-prefixed packets as before.
Transport (connect-lib/lib): extracts write_frame/read_frame helpers
from pump, adds Assignment + send/recv_assignment, VpcAcceptHandler
gains on_assign callback, VpcDialer gains dial_and_send_assignment
returning raw stream halves for N:1 mux.
Mock router: rewrites from single-peer/single-pump to N clients with
--peer-id (repeatable), --pool (CIDR), --advertise (VPC prefixes).
Per-client task bridges frames between the client stream and a shared
TUN; TUN reader demuxes by IPv6 destination to the matching client.
Client (Rust + Go): drops --address/--prefix-len/--vpc-prefix from
both binaries. Emits vpc_listening (with endpoint_id) immediately,
then vpc_ready (with assigned address) after assignment.
Labs: test-3region.sh and test-host-client.sh updated; dial now
accepts multiple peer IDs for multi-client testing.
Tested: cargo test (94 pass), go vet clean, test-3region.sh full-mesh
20/20 PASS with router-allocated addressing.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Extends the
connectplugin with avpcverb (datumctl connect vpc join): creates a local TUN interface that joins a Datum Cloud galactic VPC as a remote member over iroh —tunnel's sibling, carrying raw IPv6 packets to a galactic-side router instead of HTTP to Envoy. Implements datum-cloud/enhancements#894.connect-lib:VPCAttachmentCRD type,vpcmodule (TUN, iroh accept/dial, packet framing, wg-quick-style routing modes). WireGuard's interface/routing model is the reference; iroh subsumes its crypto/handshake/keepalive.connect-plugin:vpc joincobra command.mock-galactic-router+ containerlab labs. Control-plane wiring and the real galactic router are still scaffolding — seedesign/vpc-attachment.md§5–6.Test plan
cargo test/clippy --workspace,go build/vet/test ./...deploy/containerlab/vpc/test-3region.sh: 3-region VPC, local joins through the router (dialed by endpoint id, no ip/port), full mesh (5 devices, 20 pairs) all reachableHow to test
Everything runs on plain Docker (no containerlab/root needed for the all-in-containers path); labs live in
deploy/containerlab/vpc/. The router dials the client purely by irohEndpointId— no ip/port — via discovery, as production does.All-in-containers full mesh (one shot):
Stands up a 3-region VPC + router + local client (all containers), then pings every ordered pair (5 devices, 20 pairs) and exits non-zero on any failure.
--keepleaves it up;--downtears down.Client on your host, VPC in containers (proves a real machine joining the containerized VPC over iroh). Three steps, because the host TUN needs your root and the client must start first (it's the iroh accept side that prints the endpoint id the router dials):
Both ends reach iroh discovery + relays over normal outbound internet.
The canonical containerlab topology (
vpc-3region.clab.yaml) expresses the same all-in-containers topology forcontainerlab deploy(needs root); see the labREADME.md.