Skip to content

fix(neutron): run the neutron RPC server so the ironic agent registers - #2350

Merged
cardoe merged 5 commits into
mainfrom
neutron-enable-rpc-server
Sep 22, 2026
Merged

cardoe merged 5 commits into
mainfrom
neutron-enable-rpc-server

Conversation

@cardoe

@cardoe cardoe commented Sep 15, 2026

Copy link
Copy Markdown
Contributor

What does this change do?

neutron-server runs under uWSGI, which serves the API only, so nothing starts
neutron's RPC listeners and rpc_workers is inert. neutron-rpc-server is what
receives from agents — the comment claiming it only sends requests to them is what
kept it off. With no consumer on q-reports-plugin, neutron-ironic-agent's
report_state casts are dropped silently and it never registers: dev's 25
Baremetal Node agents Alive = False since 2025-11-19, rax-prod-dfw3 and
rax-prod-iad3-rxdb-infra the same. That leaves the baremetal mechanism driver
inert too, since it won't bind to a dead agent. The same process hosts the ML2
callbacks (q-plugin) and service-plugin RPC (trunk, L3), and
neutron-periodic-workers is where the router driver's periodics will run.

  • Enable deployment_rpc_server, with 2 replicas so state reports survive a rollout
    or node drain, and fix that comment.
  • Add the missing pod.etcSources for neutron_rpc_server and
    neutron_periodic_worker: secret_ks_etc is false, so the chart mounts
    neutron-ks-etc only where it is listed, and both pods otherwise get an empty
    /etc/neutron/neutron.conf.d instead of the service user credentials. The
    periodic worker itself stays disabled.
  • Pin networking-baremetal to stable/2026.1 in containers/neutron/Dockerfile.
    The base image installs it unpinned from PyPI and lands 8.1.0, the post-2026.1
    series, whose CI only ever ran against constraints/upper/master.
  • Bump neutron's chartVersion to 2026.1.45+5b923efb7, which threads ovn.ini
    into the ironic agent so it takes ovn_nb_connection from the chart instead of
    falling back to tcp:127.0.0.1:6641 with OVN event-driven reconciliation disabled.

This also wakes the baremetal driver as a candidate binder ahead of undersync;
try_to_bind_segment_for_agent is the guard that keeps the hierarchical VXLAN→VLAN
binding with undersync, so the pin and the backports below belong here.

Do not merge yet

networking-baremetal backports to stable/2026.1, and bumps
NETWORKING_BAREMETAL_GIT_REF:

  • 1005793 — Remove binding:host_id setting for L2VNI subports
  • 1005795 — Remove invalid requested-chassis opt for localnet ports

openstack-helm. Both are merged upstream; the first is in the chart bump above, the
second is in 65f89744e and needs a chart release past 5b923efb7:

  • 1005787 — thread ovn.ini into the ironic agent
  • 1005789 — add a readiness probe to the ironic agent

Upgrade impact

  • This change requires operator action to upgrade. If checked, add the
    upgrade-impact label and a release note: run scriv create from the
    repository root and describe the required action in the generated
    changelog.d/ file. See RELEASING.md.

Picked up by a normal resync, no operator action, matching how 9edf31f enabled the
OVN maintenance worker.

@mfencik

mfencik commented Sep 16, 2026

Copy link
Copy Markdown
Contributor

Would be good to explain what is the current problem, what is it causing?

cardoe commented Sep 18, 2026

Copy link
Copy Markdown
Contributor Author

neutron-server runs under uWSGI, which serves the API only, so nothing starts neutron's RPC listeners and rpc_workers is inert. neutron-rpc-server is what receives from agents, not what sends to them — the comment saying otherwise is what kept it off.

With no consumer on q-reports-plugin, neutron-ironic-agent's report_state casts are dropped silently and it never registers: dev's 25 Baremetal Node agents have been Alive = False since 2025-11-19, both prod clusters the same. That leaves the baremetal mechanism driver inert too, since it is agent-based and won't bind to a dead agent — four configured, three running. The same process hosts the ML2 callbacks (q-plugin) and service-plugin RPC (trunk, L3), and neutron-periodic-workers is where the router driver's periodics will run.

The etcSources half is a second bug: secret_ks_etc is false, so the chart mounts neutron-ks-etc only where it's listed and the rpc-server would have come up with no service credentials. The networking-baremetal pin carries the guard that stops the now-live baremetal driver taking the VLAN binding from undersync, which is why it's in the same PR.


Generated by Claude Code

@cardoe
cardoe force-pushed the neutron-enable-rpc-server branch 7 times, most recently from 4d60ae9 to d97b431 Compare September 22, 2026 13:38
The airship base image installs networking-baremetal from PyPI unpinned,
so a rebuild picks up whatever is current. Today that is 8.1.0, from the
series after 2026.1, whose CI only ever ran against master constraints.
The 2026.1 series is 7.x, and stable/2026.1 is the only branch gated
against the 2026.1 library set the image ships.

Install it from git alongside neutron and neutron-lib so the version is
deterministic and matches our release.
neutron-server runs under uWSGI, which serves the API only, so the
rpc_workers setting in neutron.conf is inert and nothing consumes the
agent state report topic. neutron-ironic-agent therefore never registers
as a baremetal agent: report_state is a cast, so the agent logs
successful reports while every one of them is discarded.

Enable the rpc-server deployment and drop the claim that it only sends
requests to agents, which is what kept it off.

secret_ks_etc is false here, so every component has to list its
etcSources explicitly. neutron_rpc_server and neutron_periodic_worker
were missing and would get an empty /etc/neutron/neutron.conf.d instead
of the keystone, ironic, nova and placement service user credentials.

Run two replicas so agent state reports survive a rollout or a drain.

Assisted-by: Claude Opus 5
Picks up the ironic agent's ovn-neutron-init container and its
--config-file /tmp/pod-shared/ovn.ini argument, so ironic-neutron-agent
takes ovn_nb_connection from the chart instead of falling back to
tcp:127.0.0.1:6641 with OVN event-driven reconciliation disabled.
The periodic worker can be used by flavor plugins like our router
plugins to run operations like validations and cleanups out of band from
API calls. Add the missing container reference and fix up the OVN
metadata agent container as well.
We are not using the networking-baremetal l2vni support yet so the trunk
reconciler runs and attempts to recreate the anchor network and fails
since we do not have geneve enabled by default. So disable it but also
when we do go to enable it in the future, pre-emptively change to vxlan.
@cardoe
cardoe force-pushed the neutron-enable-rpc-server branch from d97b431 to 8a08303 Compare September 22, 2026 13:44
@haseebsyed12

Copy link
Copy Markdown
Contributor

need to change the image pr-2350

@cardoe
cardoe force-pushed the neutron-enable-rpc-server branch from 8a08303 to be2a2cd Compare September 22, 2026 13:54
@cardoe
cardoe marked this pull request as ready for review September 22, 2026 13:55
@cardoe
cardoe added this pull request to the merge queue Sep 22, 2026
Merged via the queue into main with commit a58d380 Sep 22, 2026
42 checks passed
@cardoe
cardoe deleted the neutron-enable-rpc-server branch September 22, 2026 14:08
neutron_l2gw: "ghcr.io/rackerlabs/understack/neutron:2026.1"
neutron_linuxbridge_agent: "ghcr.io/rackerlabs/understack/neutron:2026.1"
neutron_metadata: "ghcr.io/rackerlabs/understack/neutron:2026.1"
neutron_ovn_metadata: "ghcr.io/rackerlabs/understack/neutron:2026.1"

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this change ?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants