Skip to content

feat: Assign an egress shard's SID in spec #25

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
privateip merged 1 commit into from
Sep 23, 2026
Merged

feat: Assign an egress shard's SID in spec #25

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
26 changes: 21 additions & 5 deletions api/v1alpha1/egressshard_types.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -205,6 +205,7 @@ type AddressClaimRef struct {
// is therefore accepted and is unattributable, which is a fact about the
// allocation rather than something a schema can repair.
//
// +kubebuilder:validation:XValidation:rule="!(has(oldSelf.shardSID) && size(oldSelf.shardSID) > 0) || (has(self.shardSID) && size(self.shardSID) > 0)",message="shardSID cannot be unassigned once assigned"
// +kubebuilder:validation:XValidation:rule="(has(self.shardAddressIPv4) && size(self.shardAddressIPv4) > 0) == (has(self.nat64Prefix) && size(self.nat64Prefix) > 0)",message="shardAddressIPv4 and nat64Prefix must be set together"
// +kubebuilder:validation:XValidation:rule="!(has(oldSelf.shardAddressIPv6) && size(oldSelf.shardAddressIPv6) > 0) || (has(self.shardAddressIPv6) && size(self.shardAddressIPv6) > 0)",message="shardAddressIPv6 cannot be unassigned once assigned"
// +kubebuilder:validation:XValidation:rule="!(has(oldSelf.shardAddressIPv4) && size(oldSelf.shardAddressIPv4) > 0) || (has(self.shardAddressIPv4) && size(self.shardAddressIPv4) > 0)",message="shardAddressIPv4 cannot be unassigned once assigned"
Expand All @@ -216,6 +217,24 @@ type EgressShardSpec struct {
// +kubebuilder:validation:Required
TargetRef TargetRef `json:"targetRef"`

// ShardSID is this shard's own SRv6 uSID, the outer destination a tenant
// VRF's egress route encapsulates toward. The shard claims a packet by its
// Block and Node-ID alone, the Argument carrying the tenant, so those 64
// bits must be reserved for this shard and disjoint from every BGPRouter
// locator: a value another node already uses silently diverts that node's
// traffic.
//
// Write-once for the same reason the addresses are: every established
// flow's reply is re-encapsulated from this SID, so changing it strands
// them. Operator-chosen until the addressing service hands out identifiers
// of this kind.
//
// Empty means no SID is assigned yet, and the shard claims no packet.
// +optional
// +kubebuilder:validation:XValidation:rule="self == '' || (isIP(self) && ip(self).family() == 6)",message="shardSID must be a valid IPv6 address"
// +kubebuilder:validation:XValidation:rule="self == oldSelf || oldSelf == ''",message="shardSID is immutable once assigned"
ShardSID string `json:"shardSID,omitempty"`

// ShardAddressIPv6 is the dedicated, publicly-routable IPv6 address this
// shard translates to — every NAT66 masquerade port it allocates lives
// within this address, so any node can route a reply to the owning shard
Expand Down Expand Up @@ -300,11 +319,8 @@ type EgressShardStatus struct {
// packet gets is decided from the inner destination, not from a second
// SID.
//
// Still chosen by an operator and reported here rather than assigned in
// spec, unlike the addresses: a value another node already uses silently
// diverts that node's traffic, so the assignment belongs to the
// addressing service, which does not hand out identifiers of this kind
// yet.
// Reports the SID this node's datapath is programmed with, from
// Spec.ShardSID.
// +optional
// +kubebuilder:validation:XValidation:rule="self == '' || (isIP(self) && ip(self).family() == 6)",message="shardSID must be a valid IPv6 address"
ShardSID string `json:"shardSID,omitempty"`
Expand Down
11 changes: 9 additions & 2 deletions api/v1alpha1/egressshard_types_test.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -24,6 +24,7 @@ func newTestEgressShard() *EgressShard {
},
Spec: EgressShardSpec{
TargetRef: TargetRef{Kind: "Node", Name: "node-a"},
ShardSID: "2001:db8:ff01:2001::",
ShardAddressIPv6: "2001:db8:f00d::100",
ShardAddressIPv6ClaimRef: &AddressClaimRef{
APIGroup: "ipam.miloapis.com",
Expand All @@ -42,13 +43,17 @@ func TestEgressShardDeepCopy(t *testing.T) {
orig := newTestEgressShard()
dup := orig.DeepCopy()

dup.Spec.ShardSID = "2001:db8:ff01:2002::"
dup.Spec.ShardAddressIPv6 = "2001:db8:f00d::200"
dup.Spec.ShardAddressIPv4 = "198.51.100.7"
dup.Spec.NAT64Prefix = "64:ff9b::/96"
dup.Labels[LabelEgressShardPool] = "dedicated-public"
dup.Spec.ShardAddressIPv6ClaimRef.Name = "some-other-claim"
dup.Status.Conditions = append(dup.Status.Conditions, metav1.Condition{Type: ConditionTypeProgrammed})

if orig.Spec.ShardSID != "2001:db8:ff01:2001::" {
t.Errorf("ShardSID mutated: got %q", orig.Spec.ShardSID)
}
if orig.Spec.ShardAddressIPv6 != "2001:db8:f00d::100" {
t.Errorf("ShardAddressIPv6 mutated: got %q", orig.Spec.ShardAddressIPv6)
}
Expand Down Expand Up @@ -107,7 +112,8 @@ func TestEgressShardJSONRoundTrip(t *testing.T) {
t.Fatalf("Unmarshal: %v", err)
}

if got.Spec.ShardAddressIPv6 != orig.Spec.ShardAddressIPv6 ||
if got.Spec.ShardSID != orig.Spec.ShardSID ||
got.Spec.ShardAddressIPv6 != orig.Spec.ShardAddressIPv6 ||
got.Spec.ShardAddressIPv4 != orig.Spec.ShardAddressIPv4 ||
got.Spec.NAT64Prefix != orig.Spec.NAT64Prefix ||
got.Spec.TargetRef != orig.Spec.TargetRef {
Expand Down Expand Up @@ -148,6 +154,7 @@ func TestEgressShardSpecFieldNames(t *testing.T) {
}

want := map[string]string{
"shardSID": "2001:db8:ff01:2001::",
"shardAddressIPv6": "2001:db8:f00d::100",
"shardAddressIPv4": "198.51.100.7",
"nat64Prefix": "64:ff9b::/96",
Expand Down Expand Up @@ -196,7 +203,7 @@ func TestEgressShardSpecOmitEmpty(t *testing.T) {
}

for _, key := range []string{
"shardAddressIPv6", "shardAddressIPv4", "nat64Prefix",
"shardSID", "shardAddressIPv6", "shardAddressIPv4", "nat64Prefix",
"shardAddressIPv6ClaimRef", "shardAddressIPv4ClaimRef",
} {
if _, ok := m[key]; ok {
Expand Down
31 changes: 26 additions & 5 deletions config/crd/network.datumapis.com_egressshards.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -264,6 +264,27 @@ spec:
x-kubernetes-validations:
- message: shardAddressIPv6ClaimRef is immutable once assigned
rule: self == oldSelf
shardSID:
description: |-
ShardSID is this shard's own SRv6 uSID, the outer destination a tenant
VRF's egress route encapsulates toward. The shard claims a packet by its
Block and Node-ID alone, the Argument carrying the tenant, so those 64
bits must be reserved for this shard and disjoint from every BGPRouter
locator: a value another node already uses silently diverts that node's
traffic.

Write-once for the same reason the addresses are: every established
flow's reply is re-encapsulated from this SID, so changing it strands
them. Operator-chosen until the addressing service hands out identifiers
of this kind.

Empty means no SID is assigned yet, and the shard claims no packet.
type: string
x-kubernetes-validations:
- message: shardSID must be a valid IPv6 address
rule: self == '' || (isIP(self) && ip(self).family() == 6)
- message: shardSID is immutable once assigned
rule: self == oldSelf || oldSelf == ''
targetRef:
description: TargetRef identifies the Node this shard executes on.
properties:
Expand All @@ -283,6 +304,9 @@ spec:
- targetRef
type: object
x-kubernetes-validations:
- message: shardSID cannot be unassigned once assigned
rule: '!(has(oldSelf.shardSID) && size(oldSelf.shardSID) > 0) || (has(self.shardSID)
&& size(self.shardSID) > 0)'
- message: shardAddressIPv4 and nat64Prefix must be set together
rule: (has(self.shardAddressIPv4) && size(self.shardAddressIPv4) > 0)
== (has(self.nat64Prefix) && size(self.nat64Prefix) > 0)
Expand Down Expand Up @@ -414,11 +438,8 @@ spec:
packet gets is decided from the inner destination, not from a second
SID.

Still chosen by an operator and reported here rather than assigned in
spec, unlike the addresses: a value another node already uses silently
diverts that node's traffic, so the assignment belongs to the
addressing service, which does not hand out identifiers of this kind
yet.
Reports the SID this node's datapath is programmed with, from
Spec.ShardSID.
type: string
x-kubernetes-validations:
- message: shardSID must be a valid IPv6 address
Expand Down
3 changes: 2 additions & 1 deletion docs/api/bgp.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1058,6 +1058,7 @@ _Appears in:_
| Field | Description | Default | Validation |
| --- | --- | --- | --- |
| `targetRef` _[TargetRef](#targetref)_ | TargetRef identifies the Node this shard executes on. | | Required: \{\} <br /> |
| `shardSID` _string_ | ShardSID is this shard's own SRv6 uSID, the outer destination a tenant<br />VRF's egress route encapsulates toward. The shard claims a packet by its<br />Block and Node-ID alone, the Argument carrying the tenant, so those 64<br />bits must be reserved for this shard and disjoint from every BGPRouter<br />locator: a value another node already uses silently diverts that node's<br />traffic.<br />Write-once for the same reason the addresses are: every established<br />flow's reply is re-encapsulated from this SID, so changing it strands<br />them. Operator-chosen until the addressing service hands out identifiers<br />of this kind.<br />Empty means no SID is assigned yet, and the shard claims no packet. | | |
| `shardAddressIPv6` _string_ | ShardAddressIPv6 is the dedicated, publicly-routable IPv6 address this<br />shard translates to — every NAT66 masquerade port it allocates lives<br />within this address, so any node can route a reply to the owning shard<br />using ordinary unicast routing on it alone, with no per-flow state<br />lookup anywhere but that shard.<br />Empty means no IPv6 address is assigned to this shard. | | |
| `shardAddressIPv6ClaimRef` _[AddressClaimRef](#addressclaimref)_ | ShardAddressIPv6ClaimRef records the addressing-service claim<br />ShardAddressIPv6 came from — see AddressClaimRef. Set it in the same<br />write as the address whenever that address came from a claim.<br />Nothing here resolves it and no component reads it to program anything;<br />it is the only trail from a translating address back to the allocation<br />accountable for it. | | |
| `shardAddressIPv4` _string_ | ShardAddressIPv4 is the dedicated, publicly-routable IPv4 address this<br />shard translates to, and the source an IPv4-only destination sees.<br />Unlike ShardAddressIPv6, reachability for it is not established by a<br />BGPAdvertisement into the EVPN fabric: an IPv4 reply arrives from the<br />internet, so the underlay or an upstream announcement must attract this<br />address to this node.<br />Empty means no IPv4 address is assigned to this shard. | | |
Expand All @@ -1084,7 +1085,7 @@ _Appears in:_
| Field | Description | Default | Validation |
| --- | --- | --- | --- |
| `observedGeneration` _integer_ | ObservedGeneration is the .metadata.generation this status was computed from. | | |
| `shardSID` _string_ | ShardSID is this shard's own uSID locator — a real SRv6 uSID (unlike<br />the ShardAddress fields, which are plain routable addresses),<br />advertised into BGP the same way any other node-reachability route is<br />(a /128 BGPAdvertisement, no VRFID/Function) so every other node learns<br />a kernel SEG6 route toward it before installing a tenant VRF's egress<br />route against it. One SID serves both families: which translation a<br />packet gets is decided from the inner destination, not from a second<br />SID.<br />Still chosen by an operator and reported here rather than assigned in<br />spec, unlike the addresses: a value another node already uses silently<br />diverts that node's traffic, so the assignment belongs to the<br />addressing service, which does not hand out identifiers of this kind<br />yet. | | |
| `shardSID` _string_ | ShardSID is this shard's own uSID locator — a real SRv6 uSID (unlike<br />the ShardAddress fields, which are plain routable addresses),<br />advertised into BGP the same way any other node-reachability route is<br />(a /128 BGPAdvertisement, no VRFID/Function) so every other node learns<br />a kernel SEG6 route toward it before installing a tenant VRF's egress<br />route against it. One SID serves both families: which translation a<br />packet gets is decided from the inner destination, not from a second<br />SID.<br />Reports the SID this node's datapath is programmed with, from<br />Spec.ShardSID. | | |
| `shardAddressIPv6` _string_ | ShardAddressIPv6 is the IPv6 masquerade source this shard's datapath is<br />programmed with. Empty means it translates no IPv6 flow. | | |
| `shardAddressIPv4` _string_ | ShardAddressIPv4 is the IPv4 masquerade source this shard's datapath is<br />programmed with. Empty means it performs no NAT64. Publishing it is also<br />what makes the underlay reachability prerequisite in<br />Spec.ShardAddressIPv4 checkable. | | |
| `nat64Prefix` _string_ | NAT64Prefix is the prefix this shard's datapath is programmed to<br />translate. Empty whenever ShardAddressIPv4 is empty. | | |
Expand Down
3 changes: 2 additions & 1 deletion docs/api/gateway.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -159,6 +159,7 @@ _Appears in:_
| Field | Description | Default | Validation |
| --- | --- | --- | --- |
| `targetRef` _[TargetRef](#targetref)_ | TargetRef identifies the Node this shard executes on. | | Required: \{\} <br /> |
| `shardSID` _string_ | ShardSID is this shard's own SRv6 uSID, the outer destination a tenant<br />VRF's egress route encapsulates toward. The shard claims a packet by its<br />Block and Node-ID alone, the Argument carrying the tenant, so those 64<br />bits must be reserved for this shard and disjoint from every BGPRouter<br />locator: a value another node already uses silently diverts that node's<br />traffic.<br />Write-once for the same reason the addresses are: every established<br />flow's reply is re-encapsulated from this SID, so changing it strands<br />them. Operator-chosen until the addressing service hands out identifiers<br />of this kind.<br />Empty means no SID is assigned yet, and the shard claims no packet. | | |
| `shardAddressIPv6` _string_ | ShardAddressIPv6 is the dedicated, publicly-routable IPv6 address this<br />shard translates to — every NAT66 masquerade port it allocates lives<br />within this address, so any node can route a reply to the owning shard<br />using ordinary unicast routing on it alone, with no per-flow state<br />lookup anywhere but that shard.<br />Empty means no IPv6 address is assigned to this shard. | | |
| `shardAddressIPv6ClaimRef` _[AddressClaimRef](#addressclaimref)_ | ShardAddressIPv6ClaimRef records the addressing-service claim<br />ShardAddressIPv6 came from — see AddressClaimRef. Set it in the same<br />write as the address whenever that address came from a claim.<br />Nothing here resolves it and no component reads it to program anything;<br />it is the only trail from a translating address back to the allocation<br />accountable for it. | | |
| `shardAddressIPv4` _string_ | ShardAddressIPv4 is the dedicated, publicly-routable IPv4 address this<br />shard translates to, and the source an IPv4-only destination sees.<br />Unlike ShardAddressIPv6, reachability for it is not established by a<br />BGPAdvertisement into the EVPN fabric: an IPv4 reply arrives from the<br />internet, so the underlay or an upstream announcement must attract this<br />address to this node.<br />Empty means no IPv4 address is assigned to this shard. | | |
Expand All @@ -185,7 +186,7 @@ _Appears in:_
| Field | Description | Default | Validation |
| --- | --- | --- | --- |
| `observedGeneration` _integer_ | ObservedGeneration is the .metadata.generation this status was computed from. | | |
| `shardSID` _string_ | ShardSID is this shard's own uSID locator — a real SRv6 uSID (unlike<br />the ShardAddress fields, which are plain routable addresses),<br />advertised into BGP the same way any other node-reachability route is<br />(a /128 BGPAdvertisement, no VRFID/Function) so every other node learns<br />a kernel SEG6 route toward it before installing a tenant VRF's egress<br />route against it. One SID serves both families: which translation a<br />packet gets is decided from the inner destination, not from a second<br />SID.<br />Still chosen by an operator and reported here rather than assigned in<br />spec, unlike the addresses: a value another node already uses silently<br />diverts that node's traffic, so the assignment belongs to the<br />addressing service, which does not hand out identifiers of this kind<br />yet. | | |
| `shardSID` _string_ | ShardSID is this shard's own uSID locator — a real SRv6 uSID (unlike<br />the ShardAddress fields, which are plain routable addresses),<br />advertised into BGP the same way any other node-reachability route is<br />(a /128 BGPAdvertisement, no VRFID/Function) so every other node learns<br />a kernel SEG6 route toward it before installing a tenant VRF's egress<br />route against it. One SID serves both families: which translation a<br />packet gets is decided from the inner destination, not from a second<br />SID.<br />Reports the SID this node's datapath is programmed with, from<br />Spec.ShardSID. | | |
| `shardAddressIPv6` _string_ | ShardAddressIPv6 is the IPv6 masquerade source this shard's datapath is<br />programmed with. Empty means it translates no IPv6 flow. | | |
| `shardAddressIPv4` _string_ | ShardAddressIPv4 is the IPv4 masquerade source this shard's datapath is<br />programmed with. Empty means it performs no NAT64. Publishing it is also<br />what makes the underlay reachability prerequisite in<br />Spec.ShardAddressIPv4 checkable. | | |
| `nat64Prefix` _string_ | NAT64Prefix is the prefix this shard's datapath is programmed to<br />translate. Empty whenever ShardAddressIPv4 is empty. | | |
Expand Down
Loading
Loading