master red since #4527 (Bazel/LTO build): CI-evidence bisect, aes-mbff PSM-0069, and dead UpdateRules automation

[oharboe]

master has been red since 2026-09-17. This is a CI-evidence bisect confirming where it broke, plus two findings that are not just threshold drift. Posting it so nobody has to re-derive it — I expect the cause is already suspected, but as far as I can tell it hasn't been pinned down in writing.

A local git bisect isn't practical here (each step means rebuilding OpenROAD and running 62 designs, and the builds aren't reproducible anyway), so this is bisected from Jenkins build artifacts.

Bisect

Jenkins pins the ORFS SHA per build, so the builds themselves are the bisect steps:

Build	ORFS commit	metadata-check failures	aes-mbff asap7
master #1600	fbc8fa48d (#4532)	0 — last green	pass
master #1601	bc5af8cbd (#4533)	1	pass
master #1602	3000fc797 (#4527)	13	PSM-0069
master #1603–#1605	through 3a964e13f	13 (same set)	PSM-0069

bc5af8cbd..3000fc797 contains nothing but #4527's own commits:

3000fc797 Merge pull request #4527 from vvbandeira/openroad-bazel
f0a810dee Merge branch 'master' into openroad-bazel
84d3ba1eb fix(build): keep backslashes in --openroad-args
e7d41e0d4 build: enable LTO for the Bazel OpenROAD build
af01027ac fix(docker): include OpenROAD version in dev tags
4e30d84a9 build: install Bazel in the dev image, not the builder stage
bba680a85 build: install OpenROAD with Bazel

The single failure at #1601 is asap7/mock-alu, cts__timing__setup__ws -289.509 >= -289 — 0.18% over, ordinary drift, unrelated.

This is an interaction, which is why neither PR's own CI caught it

#4527's own merge build, PR-4527-merge #10, tested the same #4527 content (head f0a810dee, LTO included) but merged onto pre-#4533 master (fbc8fa48d). It showed 8 failing designs — matching the QoR bot's "62 designs checked — 8 with regression(s)" on that PR — and aes-mbff asap7 passed all 27 rules there.

So:

	without #4527	with #4527
without #4533	0 failures (#1600)	8 failures, aes-mbff OK (PR-4527-merge #10)
with #4533	1 failure (#1601)	13 failures + aes-mbff PSM-0069 (#1602)

Neither change alone produces the current state. #4527 is the dominant factor, but the aes-mbff hard failure and 7 of the 13 designs only appear with both.

Caveat worth stating: #4533 also carried an OpenROAD submodule bump of 145 commits (90e29809c..60807181d, including the gpl convergence fixes), so "#4533" here means that whole bundle, not only the nonPlaceArea change.

aes-mbff asap7 — a hard error, not a threshold

This one cannot be fixed by regenerating rules. From logs/asap7/aes-mbff/base/6_report.log:

[WARNING PSM-0043] Shorted shape on net VDD to terminal _tray_size4_1108/CLK
    (net clknet_leaf_3_clk) at (29.583um, 35.893um) - (29.589um, 35.927um), layer: M3.
[ERROR PSM-0069] Check connectivity failed on VDD.
Error: final_report.tcl, 7 PSM-0069

A clock-leaf pin on a multi-bit flop tray shorting to VDD on M3. Clean at #1600 and #1601, and clean in PR-4527-merge #10.

The 13 failing designs (master #1605)

Design	Metric	Measured	Rule
asap7/cva6	finish__timing__setup__tns	-1047.73	>= -851
asap7/mock-alu	cts__timing__setup__ws	-290.017	>= -289
asap7/mock-alu	globalroute__timing__setup__ws	-303.113	>= -300
gf180/riscv32i	cts__timing__setup__tns	-3.95563	>= -3.85
gt2n/aes	globalroute__timing__setup__ws	-83.6302	>= -75.2
gt2n/aes	globalroute__timing__setup__tns	-817.748	>= -585
gt2n/aes	finish__timing__setup__ws	-83.6302	>= -75.2
gt2n/aes	finish__timing__setup__tns	-817.748	>= -585
ihp-sg13g2/aes	cts__timing__setup__tns	-62.5691	>= -62.5
ihp-sg13g2/aes	globalroute__timing__setup__tns	-98.8159	>= -93.2
ihp-sg13g2/aes	finish__timing__setup__tns	-73.6913	>= -71.7
ihp-sg13g2/riscv32i	globalroute__timing__setup__ws	-2.58651	>= -2.55
ihp-sg13g2/riscv32i	globalroute__timing__setup__tns	-2332.19	>= -2250
ihp-sg13g2/riscv32i	finish__timing__setup__ws	-2.46482	>= -2.34
nangate45/bp_fe_top	globalroute__timing__setup__tns	-4.30451	>= -3.06
nangate45/bp_fe_top	finish__timing__setup__tns	-5.41024	>= -3.9
nangate45/dynamic_node	cts__timing__setup__tns	-797.044	>= -795
nangate45/dynamic_node	globalroute__timing__setup__tns	-803.169	>= -797
nangate45/dynamic_node	finish__timing__setup__tns	-826.946	>= -824
sky130hd/chameleon	cts__timing__setup__tns	-19.596	>= -19.5
sky130hd/chameleon	globalroute__antenna_diodes_count	139	<= 100
sky130hd/chameleon	globalroute__timing__setup__tns	-12.3447	>= -10.4
sky130hd/chameleon	finish__timing__setup__tns	-8.9768	>= -5.01
sky130hd/jpeg	globalroute__timing__setup__tns	-150.077	>= -142
sky130hd/microwatt	globalroute__antenna_diodes_count	3379	<= 3103
sky130hd/microwatt	detailedroute__antenna__violating__nets	96	<= 81
sky130hd/microwatt	detailedroute__antenna_diodes_count	1608	<= 1463
sky130hs/aes	cts__timing__setup__ws	-0.147874	>= -0.14
sky130hs/ibex	cts__timing__setup__tns	-123.543	>= -98.8

Most are sub-1% and are plain rules drift. These four are large enough that I'd want them looked at rather than loosened:

• gt2n/aes — GR/finish TNS -585 → -818 (+40%), WS -75.2 → -83.6
• asap7/cva6 — finish TNS -851 → -1048 (+23%)
• sky130hd/chameleon — finish TNS -5.01 → -8.98, GR antenna diodes 100 → 139
• sky130hd/microwatt — DRT antenna violating nets 81 → 96, diodes 1463 → 1608

If enabling LTO on the Bazel build is what moved these, it's worth knowing whether that's acceptable before it's baked into new goldens.

Blast radius

Every open PR inherits this through pr-merge: #4546, #4547 and #4535 are all red for this reason alone and cannot be landed.

The UpdateRules automation is dead

This is probably why the tree is still red rather than fixed in an afternoon.

.github/workflows/github-actions-update-rules.yml (the set-new-golden repository_dispatch handler) and .github/workflows/github-actions-manual-update-rules.yml both run:

python flow/util/updateRules.py --keyFile ... --apiURL ... --commitSHA ...

flow/util/updateRules.py was deleted on 2025-02-25 in 63cfaa332 "util: remove updateRules, not used". Consequences:

• the dispatch-driven workflow has zero runs, ever;
• the manual one last ran 2026-04-05 and failed;
• nothing in the repo emits the set-new-golden dispatch any more;
• github-actions-cron-update-OR.yml still applies the UpdateRules label to the nightly submodule-bump PR, where it now does nothing.

So applying the UpdateRules label to a PR has no effect. Either the script should come back or those two workflows and the label should go.

Regenerating the rules without a local run

Since the flow isn't reproducible locally, the goldens can be taken straight from CI — genRuleFile.py only reads metadata.json and never runs the flow:

B=https://jenkins.openroad.tools/job/OpenROAD-flow-scripts-Public/job/master/1605/artifact/reports
# note the report dir is nangate45/bp_fe while the design dir is nangate45/bp_fe_top
curl -sSL -o flow/reports/<platform>/<design>/base/metadata.json \
     $B/<platform>/<design>/base/metadata.json
make DESIGN_CONFIG=./designs/<platform>/<design>/config.mk update_rules

update_rules is genRuleFile.py --failing --tighten, so it only moves rules that actually fail and tightens ones that have gone slack.

That fixes the 13 metadata-check failures. It does nothing for aes-mbff, which needs a real fix or a revert.

🤖 Generated with Claude Code