Repository navigation
Keep PyLong loop carries as twodigits in shifts #156443
Description
Feature or enhancement
Proposal:
Several functions in longobject.c (v_lshift, v_rshift, and x_divrem) narrowed a loop carry value to digit or sdigit, then widened it again on the next iteration.
On AArch64, that 64-to-32-to-64 conversion inserts an extra mov on the loop-carried critical path. Keeping the carry at two-digit width until the function returns drops that mov and shortens the carry chain. Results are unchanged for valid limbs.
Use v_lshift on AArch64 as an example. The code inside the loop is:
ldr w0, [x4, x2, lsl #2] ; a[i]
mov w3, w3 ; carry chain
lsl x0, x0, x24 ; a[i] << d
orr x0, x0, x3 ; carry chain
and w1, w0, #0x3fffffff
ubfx x3, x0, #30, #32 ; carry chain
str w1, [x26, x2, lsl #2]
add x2, x2, #1
cmp x25, x2
b.ne
removing the narrowing, the code will be optimized to:
ldr w0, [x4, x2, lsl #2] ; a[i]
lsl x0, x0, x24 ; a[i] << d
orr x0, x0, x3 ; carry chain
and w1, w0, #0x3fffffff
str w1, [x26, x2, lsl #2]
add x2, x2, #1
lsr x3, x0, #30 ; carry chain
cmp x25, x2
b.ne
The instruction count on the loop carried chain is reduced from 3 to 2.
Has this already been discussed elsewhere?
No response given
Links to previous discussion of this feature:
No response
Activity
aisk commentedon Aug 27, 2026
It would be good to have some benchmark numbers on both aarch64 and x86-64 before anyone starts on this.
XiaohongGong commentedon Aug 28, 2026
Yeah, thanks for looking at this issue. I'm working on preparing the patch for improvement. And will publish the PR soon~
We can observe ~20% performance gain for
pidigitson an AArch64 machine, while about ~4% on a X86_64 machine. I will provide the performance details soon. Thanks!XiaohongGong commentedon Sep 1, 2026
Here is the performance-change graph for
pyperformanceon my x86 and AArch64 machines. It compares the normalized runtime before and after the change: values greater than 1 indicate a performance improvement, while values below 1 indicate a regression.AArch64 (NVIDIA Grace Neoverse-V2):
X86 (AMD EPYC 9124 16-Core):
From the above data, we can observe about 20% performance uplifts for
pidigitson AArch64 while not significant regression on other benchmarks and on X86 machines.[-]Keep PyLong loop carry at two-digit width in shift and division[/-][+]Keep PyLong loop carries as twodigits in shifts and division[/+]Merge branch 'main' into pythongh-156443-longobject
picnixz commentedon Sep 7, 2026
Please include the data as table with geometric mean as well (it's a bit hard to read the graph). In addition, x86 still see regressions of about 4% I guess which is non-negligible on macrobenchmarks.
Merge branch 'main' into pythongh-156443-longobject
Merge branch 'main' into pythongh-156443-longobject
XiaohongGong commentedon Sep 8, 2026
There is a substantial amount of performance data, so I previously presented it as a graph. The data in table is shown below. For the roughly 3%–5% regressions observed in several benchmarks, I reran the tests multiple times but could not reproduce them consistently. This suggests that the regressions may be due to measurement noise rather than the change itself.
Here is the pyperformance result on my AArch64 (NVIDIA Grace Neoverse-V2) machine:
And here is the pyperformance result on my X86_64 (AMD EPYC 9124 16-Core) machine:
XiaohongGong commentedon Sep 11, 2026
I reviewed the benchmarks showing regressions greater than 3%. Based on the stddev values below, the observed regressions appear more likely to be caused by system noise than by the PR change itself. Please let me know if you would recommend any additional testing. Thanks a lot!
AArch64 (NVIDIA Grace Neoverse-V2) machine:
X86_64 (AMD EPYC 9124 16-Core) machine:
Merge branch 'main' into pythongh-156443-longobject
Merge branch 'main' into pythongh-156443-longobject
Merge branch 'main' into pythongh-156443-longobject
[-]Keep PyLong loop carries as twodigits in shifts and division[/-][+]Keep PyLong loop carries as twodigits in shifts[/+]