[jkaczman]

Adds an optional supplementary bench-pgo label to be used to customize the CI metal bench.

When a PR comes in with the PGO bench label, CI will now additionally run and compare PGO benchmarks (see #1872 for what PGO is). The benchmarks ran on the PGO binaries are the same benchmarks used for the regular binaries (except microbench). After running, it'll post 4 comments (2 regular, 2 PGO) on the PR.

This is mainly intended for PRs that improve performance. When we start to include PGO builds in our releases (#1872), it would be interesting to see if there's overlap with PGO / what the perf improvement looks like on a PGO release build.

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

[github-actions]

Metal bench (arm64) passed. All 148 measures within bounds.

suite	throughput	cpu/txn	user cycles/txn	user instr/txn	kernel cycles/txn	ctx/txn
pooler		21.78 (-0.2% ± 0.5%)	22,481 (-0.6% ± 1.1%)	35,907 (-0.0% ± 0.04%)	32,189 (+0.1% ± 1.1%)
lb		22.57 (+0.2% ± 0.6%)	23,215 (-0.6% ± 1.6%)	36,875 (-0.0% ± 0.04%)	33,261 (+0.6% ± 1.1%)
sharding-w2		40.32 (-0.7% ± 1.8%)	55,660 (-0.2% ± 0.8%)	102,572 (+0.0% ± 0.01%)	47,265 (-1.4% ± 3.8%)	1.545 (+0.1% ± 0.4%)
sharding-w8	219,214 (+0.1% ± 0.5%)	33.3 (-0.1% ± 0.4%)	51,865 (-0.2% ± 0.5%)	57,911 (+0.0% ± 0.03%)	40,241 (-0.0% ± 0.4%)	0.1463 (+0.3% ± 0.9%)

Counters per txn

suite	frontend stall	l1i refill	backend stall
pooler	11,185 (-1.7% ± 2.7%)	3,153 (-0.8% ± 1.3%)	2,545 (+1.9% ± 2.2%)
lb	11,663 (-1.5% ± 3.9%)	3,231 (-0.6% ± 1.9%)	2,606 (+2.1% ± 3.0%)
sharding-w2	27,058 (-0.4% ± 2.0%)	5,210 (-0.2% ± 1.4%)	7,431 (-0.2% ± 3.5%)
sharding-w8	26,582 (-0.6% ± 0.9%)	4,797 (-0.4% ± 0.3%)	9,721 (+0.6% ± 1.2%)

Microbench: all 135 measures within bounds.

lb flamegraph vs dbcf92a

Open interactive for hover, zoom and search.

Open interactive.

microbench flamegraph vs dbcf92a

Open interactive for hover, zoom and search.

Open interactive.

pooler flamegraph vs dbcf92a

Open interactive for hover, zoom and search.

Open interactive.

sharding-w2 flamegraph vs dbcf92a

Open interactive for hover, zoom and search.

Open interactive.

sharding-w8 flamegraph vs dbcf92a

Open interactive for hover, zoom and search.

Open interactive.

Logs

lb, microbench, pooler, sharding-w2, sharding-w8, verdict.json

Compared 7a9a21b with dbcf92a on m8g.metal-24xl (spot).

[github-actions]

Metal bench (x86) passed. All 148 measures within bounds.

suite	throughput	cpu/txn	user cycles/txn	user instr/txn	kernel cycles/txn	ctx/txn
pooler		26.13 (-0.0% ± 0.2%)	20,523 (-0.0% ± 0.1%)	36,693 (+0.0% ± 0.07%)	33,229 (-0.0% ± 0.2%)
lb		26.53 (-0.1% ± 0.2%)	21,130 (-0.0% ± 0.1%)	37,643 (-0.1% ± 0.1%)	33,289 (-0.2% ± 0.2%)
sharding-w2		42.61 (+0.1% ± 1.5%)	50,438 (+0.1% ± 0.9%)	106,040 (-0.0% ± 0.07%)	38,129 (+0.0% ± 3.6%)	1.523 (-0.0% ± 0.4%)
sharding-w8	202,588 (+0.1% ± 0.3%)	34.23 (-0.1% ± 0.3%)	49,479 (-0.1% ± 0.5%)	60,305 (-0.0% ± 0.05%)	31,182 (-0.0% ± 0.3%)	0.1635 (+0.1% ± 1.0%)

Counters per txn

suite	frontend stall	l1i refill	backend stall
pooler	3,623 (-0.1% ± 0.6%)	1,192 (-0.1% ± 0.5%)	41,541 (-0.1% ± 0.4%)
lb	3,757 (+0.0% ± 0.3%)	1,222 (-0.0% ± 0.4%)	42,807 (-0.0% ± 0.3%)
sharding-w2	6,419 (+0.9% ± 1.4%)	1,708 (+0.5% ± 0.9%)	122,155 (-0.2% ± 2.7%)
sharding-w8	3,371 (+0.0% ± 0.4%)	1,027 (-0.2% ± 0.3%)	187,439 (-0.2% ± 0.8%)

Microbench: all 135 measures within bounds.

lb flamegraph vs dbcf92a

Open interactive for hover, zoom and search.

Open interactive.

microbench flamegraph vs dbcf92a

Open interactive for hover, zoom and search.

Open interactive.

pooler flamegraph vs dbcf92a

Open interactive for hover, zoom and search.

Open interactive.

sharding-w2 flamegraph vs dbcf92a

Open interactive for hover, zoom and search.

Open interactive.

sharding-w8 flamegraph vs dbcf92a

Open interactive for hover, zoom and search.

Open interactive.

Logs

lb, microbench, pooler, sharding-w2, sharding-w8, verdict.json

Compared 7a9a21b with dbcf92a on c7i.metal-48xl (spot).

[github-actions]

Metal bench PGO (arm64).

12 rounds of 3f2b2ea vs d20ec1a.

suite	measure	base	head	launches per side	delta (95% CI)
pooler	throughput	16,994	16,994	36	+0.00% ± 0.00%
pooler	cpu-per-txn	18.45	18.45	36	+0.04% ± 0.19%
pooler	user-cycles-per-txn	17,405	17,603	36	+1.13% ± 0.09%
pooler	user-instructions-per-txn	31,081	31,658	36	+1.86% ± 0.03%
pooler	kernel-instructions-per-txn	52,240	52,261	36	+0.04% ± 0.15%
pooler	kernel-cycles-per-txn	29,410	29,246	36	-0.56% ± 0.33%
lb	throughput	15,998	15,998	36	+0.00% ± 0.00%
lb	cpu-per-txn	19.21	19.18	36	-0.18% ± 0.31%
lb	user-cycles-per-txn	18,409	18,473	36	+0.35% ± 0.16%
lb	user-instructions-per-txn	32,634	33,217	36	+1.79% ± 0.05%
lb	kernel-instructions-per-txn	53,251	53,276	36	+0.05% ± 0.17%
lb	kernel-cycles-per-txn	30,277	30,069	36	-0.68% ± 0.57%
sharding-w8	throughput	271,047	269,998	8	-0.39% ± 0.90%
sharding-w8	cpu-per-txn	25.67	25.83	8	+0.65% ± 0.96%
sharding-w8	user-cycles-per-txn	40,778	41,103	8	+0.80% ± 0.55%
sharding-w8	user-instructions-per-txn	49,444	50,405	8	+1.94% ± 0.01%
sharding-w8	kernel-instructions-per-txn	43,179	43,166	8	-0.03% ± 0.08%
sharding-w8	kernel-cycles-per-txn	30,018	30,161	8	+0.47% ± 1.73%
sharding-w8	ctx-switches-per-txn	0.2047	0.203	8	-0.85% ± 1.18%
sharding-w2	throughput	9,999	9,999	8	+0.00% ± 0.00%
sharding-w2	cpu-per-txn	27.58	27.94	8	+1.33% ± 1.00%
sharding-w2	user-cycles-per-txn	36,990	37,526	8	+1.45% ± 0.61%
sharding-w2	user-instructions-per-txn	87,166	88,143	8	+1.12% ± 0.03%
sharding-w2	kernel-instructions-per-txn	54,623	54,622	8	-0.00% ± 0.19%
sharding-w2	kernel-cycles-per-txn	33,500	34,046	8	+1.63% ± 2.18%
sharding-w2	ctx-switches-per-txn	1.563	1.563	8	-0.05% ± 0.15%

Compared d20ec1a with 3f2b2ea on m8gd.metal-24xl (spot).

[github-actions]

Metal bench PGO (x86).

12 rounds of 3f2b2ea vs d20ec1a.

suite	measure	base	head	launches per side	delta (95% CI)
pooler	throughput	16,994	16,994	36	+0.00% ± 0.00%
pooler	cpu-per-txn	32.77	32.8	36	+0.10% ± 0.28%
pooler	user-cycles-per-txn	20,179	19,836	36	-1.70% ± 0.59%
pooler	user-instructions-per-txn	27,975	28,033	36	+0.21% ± 0.01%
pooler	kernel-instructions-per-txn	38,845	38,841	36	-0.01% ± 0.08%
pooler	kernel-cycles-per-txn	55,011	55,440	36	+0.78% ± 0.40%
lb	throughput	15,997	15,996	36	-0.00% ± 0.01%
lb	cpu-per-txn	33.73	33.78	36	+0.13% ± 0.16%
lb	user-cycles-per-txn	21,400	20,920	36	-2.24% ± 0.69%
lb	user-instructions-per-txn	29,072	29,120	36	+0.17% ± 0.03%
lb	kernel-instructions-per-txn	39,489	39,486	36	-0.01% ± 0.10%
lb	kernel-cycles-per-txn	55,758	56,324	36	+1.02% ± 0.15%
sharding-w8	throughput	152,443	153,356	8	+0.60% ± 1.68%
sharding-w8	cpu-per-txn	46.43	46.29	8	-0.29% ± 0.74%
sharding-w8	user-cycles-per-txn	54,888	54,820	8	-0.12% ± 0.67%
sharding-w8	user-instructions-per-txn	46,521	46,519	8	-0.00% ± 0.11%
sharding-w8	kernel-instructions-per-txn	34,222	34,233	8	+0.03% ± 0.11%
sharding-w8	kernel-cycles-per-txn	62,742	62,435	8	-0.48% ± 1.10%
sharding-w8	ctx-switches-per-txn	0.1861	0.1896	8	+1.89% ± 1.02%
sharding-w2	throughput	9,999	9,999	8	-0.00% ± 0.00%
sharding-w2	cpu-per-txn	51.76	51.6	8	-0.29% ± 0.68%
sharding-w2	user-cycles-per-txn	43,994	43,883	8	-0.24% ± 1.27%
sharding-w2	user-instructions-per-txn	88,921	88,985	8	+0.07% ± 0.04%
sharding-w2	kernel-instructions-per-txn	43,260	43,276	8	+0.04% ± 0.07%
sharding-w2	kernel-cycles-per-txn	69,648	69,201	8	-0.63% ± 1.02%
sharding-w2	ctx-switches-per-txn	1.427	1.43	8	+0.18% ± 0.33%

Compared d20ec1a with 3f2b2ea on m7a.metal-48xl (spot).