Skip to content

docs(networking): explain FQDN policy limitations with CNI chaining #766

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
Fedor Batonogov (batonogov) wants to merge 1 commit into
base: main
Choose a base branch
from
Open

docs(networking): explain FQDN policy limitations with CNI chaining #766

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
39 changes: 39 additions & 0 deletions content/en/docs/next/networking/architecture.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -76,6 +76,45 @@ balancer.
The rest of this document describes the default `kubeovn-cilium` variant.
{{% /alert %}}

### FQDN and Layer 7 Policy Limitations

{{% alert color="warning" %}}
On the default Kube-OVN + Cilium stack, do not rely on `toFQDNs` rules
for FQDN-based egress enforcement. Cilium's `generic-veth` chaining mode
has [upstream Layer 7 policy limitations](https://docs.cilium.io/en/stable/installation/cni-chaining-generic-veth/).
FQDN rules depend on DNS traffic passing through Cilium's DNS proxy;
adding `rules.dns` to enable that proxy can also disrupt pod DNS in this
configuration. Acceptance of a `CiliumNetworkPolicy` by the API does not
confirm that its rules work on the dataplane.
{{% /alert %}}

For workloads in the management cluster, use L3/L4 rules with endpoint,
entity, or CIDR selectors as appropriate for the destination. These rules
restrict addresses and ports; they do not provide domain-name filtering.
See [cozystack/cozystack#3820](https://github.com/cozystack/cozystack/issues/3820)
for the reported limitation.

#### Keeping VM Networking While Using FQDN Policies

When workloads require FQDN or other L7 policies alongside the platform's
Kube-OVN-based VM networking, an option is to run those workloads in a
[managed Kubernetes cluster](/docs/next/kubernetes/) with Cilium as its
primary CNI. Keep the management cluster on Kube-OVN + Cilium; apply the
workload policies inside the managed cluster, where Cilium is not chained
to Kube-OVN.

This separates the two networking requirements without changing the
management cluster's CNI. It adds another cluster to operate, and its
policies apply to its own workloads, not to pods or VMs in the management
cluster. FQDN rules allow destination IPs learned from matching DNS
responses; they do not distinguish names sharing an IP address, inspect
encrypted application content, or replace a WAF.

Before relying on the policies, test pod DNS, an allowed destination, a
denied destination, and any required internal Service or LoadBalancer
paths in the deployed cluster. Check all policies selecting the workload:
another egress allow rule can broaden access beyond the FQDN allowlist.

### Pod CIDR Allocation (Kube-OVN)

Kube-OVN uses a **shared Pod CIDR** model:
Expand Down
39 changes: 39 additions & 0 deletions content/en/docs/v1.6/networking/architecture.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -76,6 +76,45 @@ balancer.
The rest of this document describes the default `kubeovn-cilium` variant.
{{% /alert %}}

### FQDN and Layer 7 Policy Limitations

{{% alert color="warning" %}}
On the default Kube-OVN + Cilium stack, do not rely on `toFQDNs` rules
for FQDN-based egress enforcement. Cilium's `generic-veth` chaining mode
has [upstream Layer 7 policy limitations](https://docs.cilium.io/en/stable/installation/cni-chaining-generic-veth/).
FQDN rules depend on DNS traffic passing through Cilium's DNS proxy;
adding `rules.dns` to enable that proxy can also disrupt pod DNS in this
configuration. Acceptance of a `CiliumNetworkPolicy` by the API does not
confirm that its rules work on the dataplane.
{{% /alert %}}

For workloads in the management cluster, use L3/L4 rules with endpoint,
entity, or CIDR selectors as appropriate for the destination. These rules
restrict addresses and ports; they do not provide domain-name filtering.
See [cozystack/cozystack#3820](https://github.com/cozystack/cozystack/issues/3820)
for the reported limitation.

#### Keeping VM Networking While Using FQDN Policies

When workloads require FQDN or other L7 policies alongside the platform's
Kube-OVN-based VM networking, an option is to run those workloads in a
[managed Kubernetes cluster](/docs/v1.6/kubernetes/) with Cilium as its
primary CNI. Keep the management cluster on Kube-OVN + Cilium; apply the
workload policies inside the managed cluster, where Cilium is not chained
to Kube-OVN.

This separates the two networking requirements without changing the
management cluster's CNI. It adds another cluster to operate, and its
policies apply to its own workloads, not to pods or VMs in the management
cluster. FQDN rules allow destination IPs learned from matching DNS
responses; they do not distinguish names sharing an IP address, inspect
encrypted application content, or replace a WAF.

Before relying on the policies, test pod DNS, an allowed destination, a
denied destination, and any required internal Service or LoadBalancer
paths in the deployed cluster. Check all policies selecting the workload:
another egress allow rule can broaden access beyond the FQDN allowlist.

### Pod CIDR Allocation (Kube-OVN)

Kube-OVN uses a **shared Pod CIDR** model:
Expand Down