Add UID-based nftables firewall for NATS and monit connections by rkoster · Pull Request #399 · cloudfoundry/bosh-agent

rkoster · 2026-02-06T14:30:33Z

Summary

This PR implements a UID-based nftables firewall to protect NATS (mbus) and monit connections, replacing the previous cgroup-based iptables approach.

Motivation

The existing cgroup-based firewall approach has limitations in nested container environments (like Garden containers running on BOSH VMs). In these environments:

The cgroup hierarchy may not be accessible or reliable
iptables cgroup matching doesn't work consistently
This leaves NATS and monit ports exposed to unprivileged processes

The UID-based approach solves this by using nftables with meta skuid matching, which works reliably regardless of container nesting.

Changes

New `platform/firewall` Package

firewall.go - Defines Manager and NatsFirewallHook interfaces
nftables_firewall.go - Linux implementation using github.com/google/nftables (netlink-based, no CLI required)
nftables_firewall_other.go - Stub for non-Linux platforms

Firewall Rules

Creates an nftables table bosh_agent with two output chains:

Chain	Purpose	Rule
`monit_access`	Protect monit (port 2822)	Allow only UID 0, drop others
`nats_access`	Protect NATS/director connection	Allow only UID 0, drop others

Platform Integration

Added GetNatsFirewallHook() to Platform interface
LinuxPlatform initializes firewall and implements the hook
Stub implementations for Windows and dummy platforms

NATS Handler Integration

nats_handler.go calls BeforeConnect hook before NATS connection and on each reconnection
Supports DNS re-resolution on reconnect for HA director failover scenarios

Testing

23 unit tests for firewall functionality (setup, cleanup, error handling)
4 new tests for NATS handler firewall hook integration
All tests use dependency injection for nftables connection and DNS resolver

Technical Details

Uses github.com/google/nftables library which communicates via netlink (no nft CLI needed)
Works on Ubuntu Jammy and Noble without additional package installation
IPv4 and IPv6 support
Firewall rules are idempotent (chains are flushed before adding rules)

Testing Performed

Unit tests: ginkgo -r platform/firewall mbus
Manual testing on Noble stemcell in nested Garden environment:
- Non-root user: curl 127.0.0.1:2822 hangs (blocked)
- Root user: curl 127.0.0.1:2822 returns 401 (allowed through firewall)

Implement a firewall mechanism that restricts NATS (mbus) connections to the bosh-agent process only, using UID-based filtering with nftables. Key changes: - Add platform/firewall package with Manager and NatsFirewallHook interfaces - Implement NftablesFirewall that creates UID-based egress rules - Add GetNatsFirewallHook() to Platform interface - Integrate BeforeConnect hook in nats_handler.go for connection/reconnection - Support DNS re-resolution on reconnect for HA failover scenarios - Add stub implementations for Windows and dummy platforms The firewall rules allow only the agent's UID to connect to NATS/director ports while blocking other processes, improving security posture.

Add comprehensive unit tests for the new firewall functionality: platform/firewall tests (23 tests): - SetupMonitFirewall: table/chain/rule creation, error handling - SetupNATSFirewall: IPv4/IPv6, DNS resolution, https/empty URL handling - BeforeConnect: delegation to SetupNATSFirewall - Cleanup: table deletion and error handling mbus/nats_handler tests (4 new tests): - Firewall hook is called on Start - BeforeConnect receives correct mbus URL - Handler still starts when hook returns nil - Warning logged but no failure when BeforeConnect errors Also: - Add DNSResolver interface for testable DNS resolution - Inject resolver dependency via NewNftablesFirewallWithDeps - Configure test logging to use GinkgoWriter for visibility

- Fix ST1023 linter error: omit type from var declaration - Add linux_header.txt for counterfeiter to add build tags to Linux-only fakes - Regenerate fake_nftables_conn.go and fake_dnsresolver.go with //go:build linux tag - This fixes macOS/Windows build failures due to google/nftables being Linux-only

mariash · 2026-02-06T19:15:49Z

mbus/nats_handler.go

 	var natsOptions = []nats.Option{
 		nats.RetryOnFailedConnect(true),
 		nats.DisconnectErrHandler(func(c *nats.Conn, err error) {
 			h.logger.Debug(natsHandlerLogTag, "Nats disconnected with Error: %v", err.Error())


I know this was not introduced in your PR, but I see you handle err nil on line 170 and this seems to have similar issue?

mariash · 2026-02-06T19:31:45Z

mbus/nats_handler.go

 	}
+
+	// Update firewall rules before initial connection
+	h.updateFirewallForNATS()


The SetupNetworking in ubuntu_net_manager is still calling SetupNatsFirewall. Will this be an issue to have both iptables with cgroup matching and nftables?

mariash · 2026-02-06T19:34:33Z

platform/firewall/nftables_firewall.go

+		&expr.Cmp{
+			Op:       expr.CmpOpEq,
+			Register: 1,
+			Data:     net.ParseIP("127.0.0.1").To4(),


Do we also want to add a rule for ipv6 for loopback?

mariash · 2026-02-06T19:35:32Z

platform/firewall/firewall.go

+	SetupNATSFirewall(mbusURL string) error
+
+	// Cleanup removes all agent-managed firewall rules
+	Cleanup() error


I don't see this is being called?

Good catch, have removed it, since even during a restart of the agent (which sometimes happens during an update_settings) it is better to keep the firewall rules in place.

mariash · 2026-02-06T19:37:49Z

platform/linux_platform.go

+		p.firewallManager = mgr
+
+		// Set up monit firewall rules immediately
+		if err := mgr.SetupMonitFirewall(); err != nil {


Wondering if this can be more explicitly called during setup and not in a getter?

mariash · 2026-02-06T19:39:00Z

platform/firewall/nftables_firewall.go

+	return f.conn.Flush()
+}
+
+func (f *NftablesFirewall) ensureTable() error {


Looks like these 2 methods always return nil?

Alphasite · 2026-02-06T19:46:13Z

Im a little worried by the general approach of teaching the agent about os-version specific things. Specifically I worry that it will (further?) violate layering by pushing version specific customisation from the stemcell into the agent.

Its probably ok here since this somewhat sits between agent setup and stemcell config, but i wanted to at least mention it even if nothing comes of it.

- Fix nil pointer dereference in DisconnectErrHandler when err is nil - Remove iptables-based SetupNatsFirewall code (replaced by nftables) - Remove unused Cleanup() method from firewall interface - Move firewall initialization from lazy getter to explicit SetupFirewall() - Add comment explaining IPv6 loopback is intentionally not protected (monit only binds to 127.0.0.1:2822)

rkoster · 2026-02-06T20:10:58Z

Im a little worried by the general approach of teaching the agent about os-version specific things. Specifically I worry that it will (further?) violate layering by pushing version specific customisation from the stemcell into the agent.

There currently is nothing OS specific about this feature, because it works on both noble an jammy. So this is an effort to simplify and centralise all the different nats and monit firewall codepaths into the agent, where it can more easily be tested (compared to the stemcell builder).

The nftables library batches operations until Flush() is called, so AddTable/AddChain/AddRule never return errors. Removing the misleading error return types from these internal helper methods.

aramprice · 2026-02-07T00:13:08Z

platform/firewall/linux_header.txt

Why this file?

This was confusing as I was expecting a C header based on naming.

Maybe clearer would be to put it in firewallfakes/ and name it linux_build_flag_header.txt or - if not in the directory - have the name indicate counterfeiter or fakes: {counterfeiter|fakes}_linux_build_flag_header.txt.

rkoster added 3 commits February 6, 2026 12:15

rkoster requested review from aramprice, beyhan and mariash February 6, 2026 14:57

cf-foundation-community-automation bot added this to Foundational Infrastructure Working Group Feb 6, 2026

cf-foundation-community-automation bot moved this to Inbox in Foundational Infrastructure Working Group Feb 6, 2026

rkoster moved this from Inbox to Pending Review | Discussion in Foundational Infrastructure Working Group Feb 6, 2026

mariash reviewed Feb 6, 2026

View reviewed changes

Remove unused error returns from internal firewall helper methods

ca5711e

The nftables library batches operations until Flush() is called, so AddTable/AddChain/AddRule never return errors. Removing the misleading error return types from these internal helper methods.

rkoster requested a review from mariash February 6, 2026 20:14

aramprice reviewed Feb 7, 2026

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add UID-based nftables firewall for NATS and monit connections#399

Add UID-based nftables firewall for NATS and monit connections#399
rkoster wants to merge 5 commits intocloudfoundry:mainfrom
rkoster:feature/uid-based-firewall

rkoster commented Feb 6, 2026 •

edited

Loading

Uh oh!

mariash Feb 6, 2026

Uh oh!

mariash Feb 6, 2026

Uh oh!

mariash Feb 6, 2026

Uh oh!

mariash Feb 6, 2026

Uh oh!

rkoster Feb 6, 2026

Uh oh!

mariash Feb 6, 2026

Uh oh!

mariash Feb 6, 2026

Uh oh!

Alphasite commented Feb 6, 2026

Uh oh!

rkoster commented Feb 6, 2026

Uh oh!

aramprice Feb 7, 2026

Uh oh!

aramprice Feb 7, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Conversation

rkoster commented Feb 6, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Motivation

Changes

New platform/firewall Package

Firewall Rules

Platform Integration

NATS Handler Integration

Testing

Technical Details

Testing Performed

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Alphasite commented Feb 6, 2026

Uh oh!

rkoster commented Feb 6, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

rkoster commented Feb 6, 2026 •

edited

Loading

New `platform/firewall` Package