Skip to content

Support for Jumbo Frames (RFD 689)#10472

Open
rcgoodfellow wants to merge 10 commits into
mainfrom
ry/opte-984
Open

Support for Jumbo Frames (RFD 689)#10472
rcgoodfellow wants to merge 10 commits into
mainfrom
ry/opte-984

Conversation

@rcgoodfellow
Copy link
Copy Markdown
Contributor

@rcgoodfellow rcgoodfellow commented May 20, 2026

Implements the Omicron side of RFD 689.

Depends on:

Some other fixes included here.

  • Updates lldpd to fix client/server version mismatch.
  • Don't squash lldpd 404 errors into 500 Oxide API errors.
  • Add support for static IPv4 routes over IPv6 nexthops (this was an oversight in initial IPv6 routing support and something I needed for my testing setup).

Testing

Lab Environment Performance

I've tested this on berlin. When the fleet level jumbo flag is set and instances have opted into jumbo, they come up with an 8500 byte MTU in linux. The numbers I see for TCP from berlin running through one of the external d4x4 routers is.

  • 1500 MTU
    • ~6 gbps single stream
    • ~10 gbps 16 streams (-P 16 on the iperf3 client)
  • 8500 MTU
    • ~12 gbps single stream
    • ~30 gbps 16 streams

For UDP

  • 1500 MTU
    • ~1 gbps single stream
    • ~3 gbps 16 streams
  • 8500 MTU
    • ~3 gbps single stream
    • ~11 gbps 16 streams

I also tested that a pair of clients with and without jumbo enabled can communicate over both TCP and UDP.

Control Bits

The fleet-level jumbo enable and instance-level jumbo enable have been tested in combination. For an instance to have jumbo frames enabled, both the fleet level setting and the instance level setting must be set to true. I've tested all combinations of these settings with an instance restart in between changes to pick up the setting, and things behave as expected.

Comment thread openapi/nexus/nexus-2026052000.0.0-7e307e.json Outdated
dhcp_config: DhcpCfg::default(),
// Services do not use attached subnets, only instances.
attached_subnets: vec![],
// TODO(RFD 689): plumb the fleet-wide jumbo-frames opt-in
Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Note to self: make follow up issue for this.

Comment thread tools/install_opte.sh
curl -fL -o "$P5P_PATH" "$P5P_URL"

RC=0
pfexec pkg install -g "$P5P_PATH" driver/network/opte || RC=$?
Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This pattern was not working for me in a4x2 with Helios 3. The new form does work.

@rcgoodfellow rcgoodfellow marked this pull request as ready for review May 24, 2026 05:39
@rcgoodfellow rcgoodfellow changed the title implement RFD 689 Support for Jumbo Frames (RFD 689) May 24, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants