Skip to content

sources: assign an IP that moved to another machine in the run that sees the move - #619

Open
tyderian1978 wants to merge 1 commit into
bb-Ricardo:developmentfrom
tyderian1978:fix/ip-moved-between-machines
Open

tyderian1978 wants to merge 1 commit into
bb-Ricardo:developmentfrom
tyderian1978:fix/ip-moved-between-machines

Conversation

@tyderian1978

Copy link
Copy Markdown

Disclaimer: AI generated, based off the work we're doing with Netbox.

What happens

When an address moves from one running VM to another, the next run doesn't move it. Typical cases are the cluster and listener addresses of a Windows Server failover cluster or SQL Server availability group, a keepalived/ucarp VIP, or an address re-used by a replacement VM. That run logs

WARNING: Current interface 'vNIC 1 (...) (node-b)' for IP '10.40.17.12/19' and this one 'vNIC 1 (...) (node-a)' are both enabled. IP assignment skipped because it is unclear which one is the correct one!

takes the address off node-b's interface and tags it NetBox-synced: Orphaned. Only the run after that assigns it to node-a. So after every failover the address is assigned to nothing and tagged orphaned for one sync interval, and the change log gets three entries for it instead of one.

We have about 50 such addresses on about 25 clusters. Patching moves most of them, and moves them back a night or two later. #250 reported the same warning for HA setups in 2022; excluding the addresses through permitted_subnets avoids the warning, but then they aren't synced at all.

Why

In the run after the move, node-b no longer reports the address, so add_update_interface() calls remove_interface_association() on it. That only queues the change: unset_attribute() adds it to unset_items, which the NetBox update sends in its first phase. The object's data still names node-b's interface.

When node-a is parsed, the matching loop finds the address on an enabled interface of another active VM and skips it. That holds whichever VM is parsed first, and in the reevaluation pass meant for this case too: by the time node-a is parsed again, the removal is still only queued.

Fix

  • In the matching loop, an address whose assigned_object_id is queued for unsetting counts as unassigned: the interface it is on stopped reporting it in this run. remove_interface_association() is only called from the removal loop in add_update_interface(), so a queued unset means exactly that.
  • NBIPAddress.update() drops a queued unset of assigned_object_id / assigned_object_type when the address is assigned again, so NetBox gets one PATCH to the new interface instead of an unassign followed by an assign.

An address both machines still report stays ambiguous and is skipped, as before.

Tests

tests/test_ip_moved_between_machines.py, in the style of test_ip_non_unique_roles.py:

  • the old machine is parsed first, then the new one;
  • the new machine is parsed first, then the old one, then the new one again (the reevaluation pass);
  • control: both machines still report the address, so it is skipped and stays where it is;
  • control: the old machine still loses it as its primary IP.

Three of them fail on development without the change (the "both report it" control passes); all four pass with it. Full suite: 611 passed, 8 skipped.

Also checked end to end against vcsim and NetBox 4.7.2: two running VMs from tests/fixtures/vcsim/vc001 handing an address back and forth.

  • Before: the warning, the address unassigned and tagged orphaned, and assigned to the new VM one run later.
  • After: one change in the run that sees the move, in both directions, and nothing in the run after.

…ees the move

When an address moves from one running machine to another (a Windows
failover cluster's or SQL listener's address, a keepalived/ucarp VIP, an
address handed to a replacement VM), the next run logged

  Current interface 'X' for IP 'a.b.c.d' and this one 'Y' are both
  enabled. IP assignment skipped because it is unclear which one is the
  correct one!

and left the address assigned to nothing, tagged orphaned, until the run
after that assigned it to the new machine.

The old machine no longer reports the address, so add_update_interface()
queues its removal (remove_interface_association() -> unset_attribute()).
That only records the removal for the NetBox update phase; the object's
data still names the old interface. The matching loop for the new machine,
and the reevaluation pass meant for exactly this case, therefore kept
seeing the address on an enabled interface of another active machine.

Treat an address whose assignment is queued for removal as free, and drop
the queued removal when it is assigned again, so NetBox gets a single PATCH
to the new interface. An address both machines still report stays
ambiguous and is skipped as before.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant