Skip to content

fix(nebius): persist firewall rules across instance stop/start cycles#106

Open
mmoulikk wants to merge 6 commits intomainfrom
nebius-netowrk-state
Open

fix(nebius): persist firewall rules across instance stop/start cycles#106
mmoulikk wants to merge 6 commits intomainfrom
nebius-netowrk-state

Conversation

@mmoulikk
Copy link
Copy Markdown
Contributor

Summary

  • Adds a systemd drop-in (After=netfilter-persistent.service) to ufw.service via cloud-init runcmd, fixing a race condition where both services call iptables-restore concurrently on boot, causing UFW to fail with iptables-restore: line 4 failed
  • Installs iptables-persistent and runs netfilter-persistent save to snapshot the complete iptables state (UFW chains + DOCKER-USER rules) so it survives stop/start cycles
  • Provides defense-in-depth: netfilter-persistent restores the snapshot first, then UFW starts after it and re-applies cleanly

Changes

cloud/v1/providers/nebius/instance.gogenerateCloudInitUserData()

  1. Added iptables-persistent to cloud-init packages list (installs netfilter-persistent service)
  2. Added systemd drop-in creation via printf piped to tee — writes After=netfilter-persistent.service to /etc/systemd/system/ufw.service.d/after-netfilter.conf, followed by systemctl daemon-reload
  3. Added sudo netfilter-persistent save as the final runcmd command to snapshot the complete iptables state to /etc/iptables/rules.v4
  4. Fixed YAML rendering of runcmd entries: commands are now double-quoted with \ and " properly escaped, preventing YAML-special characters from being misinterpreted by the cloud-init parser

Test plan

  • Provisioned a new Nebius instance, confirmed cloud-init sets up the systemd drop-in, UFW rules, DOCKER-USER rules, and netfilter-persistent snapshot
  • Stopped and started the instance, confirmed host-ssh health check passes
  • Verified via journalctl that UFW starts after netfilter-persistent (no race)
  • Verified sudo iptables -L DOCKER-USER -n -v shows all rules after reboot
  • Verified sudo ufw status verbose shows correct rules after reboot

@mmoulikk mmoulikk requested a review from a team as a code owner March 30, 2026 14:26
@patelspratik
Copy link
Copy Markdown
Contributor

patelspratik commented Apr 1, 2026

is this solution an option?

  1. Use ufw alone with ufw enable persistence

UFW already persists its rules to /etc/ufw/*.rules and restores them on boot via its own service. The DOCKER-USER iptables rules are what don't survive. Instead of snapshotting the entire iptables state, you could:

  • Keep UFW for port rules (it already persists)
  • Move the DOCKER-USER rules into /etc/ufw/after.rules (UFW's native hook for custom iptables rules)

This eliminates the need for iptables-persistent entirely. UFW handles everything, no race condition because there's only one service.

@mmoulikk
Copy link
Copy Markdown
Contributor Author

mmoulikk commented Apr 2, 2026

is this solution an option?

  1. Use ufw alone with ufw enable persistence

UFW already persists its rules to /etc/ufw/*.rules and restores them on boot via its own service. The DOCKER-USER iptables rules are what don't survive. Instead of snapshotting the entire iptables state, you could:

  • Keep UFW for port rules (it already persists)
  • Move the DOCKER-USER rules into /etc/ufw/after.rules (UFW's native hook for custom iptables rules)

This eliminates the need for iptables-persistent entirely. UFW handles everything, no race condition because there's only one service.

Hi @patelspratik ,

Yeah, we did consider moving the DOCKER-USER rules into /etc/ufw/after.rules so everything could be managed under UFW. Though it can work, Docker’s documentation states that Docker and UFW are incompatible in how they use firewall rules, so adding those rules into UFW can make things a bit tricky to manage and easier to mess up.
Ref: Docker and UFW

For now, we’re keeping UFW for general port rules and handling DOCKER-USER rules separately. This reduces coupling — if something goes wrong with UFW because of Docker-related rules, it doesn’t automatically affect the rest of the setup including SSH. Using iptables-persistent just ensures the custom Docker rules survive reboots, since Docker doesn’t persist those rules itself and UFW doesn’t manage them.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants