Repository navigation
optparse: parsing many positional arguments takes quadratic time #161282
Description
Bug report
Bug description:
OptionParser.parse_args() takes quadratic time in the number of positional arguments. _process_args() moves each positional argument to largs and then removes it from the front of the remaining arguments with del rargs[0], which shifts the whole rest of the list every time:
Lines 1420 to 1422 in 7e3290d
Many positional arguments are common with shell globs or xargs, for example tool -v *.txt in a large directory.
import optparse, time
parser = optparse.OptionParser()
parser.add_option('-v', action='store_true')
for n in (25_000, 50_000, 100_000, 200_000):
args = ['-v'] + [f'file{i}' for i in range(n)]
start = time.perf_counter()
parser.parse_args(args)
print(f'{n:>7,}: {(time.perf_counter() - start) * 1000:.1f} ms')| positional args | main | with the fix below |
|---|---|---|
| 25,000 | 33.2 ms | 1.0 ms |
| 50,000 | 150.2 ms | 1.9 ms |
| 100,000 | 616.4 ms | 4.1 ms |
| 200,000 | 2467.2 ms | 8.2 ms |
Doubling the arguments makes it about 4 times slower. For comparison, argparse parses the same 200,000 arguments in about 30 ms. (macOS arm64, release build of main at 7e3290d, best of 3.)
Proposed fix
Append each run of consecutive positional arguments to largs, then remove the whole run from rargs with a single del rargs[:end]. A run ends at the next argument that _process_args() would handle as an option (arg[:1] == "-" and len(arg) > 1, which includes --), so options, -- and callbacks are processed exactly as before, and callbacks see the same parser.largs and parser.rargs.
Options are still removed from rargs one at a time, since callbacks are documented to inspect and modify parser.rargs. That part is only quadratic in the number of options, which is rarely large.
CPython versions tested on:
CPython main branch
Operating systems tested on:
macOS
Activity
bedevere-app commentedon Nov 4, 2026
Thank you for taking the time to open this issue. Python is a community-driven project, and we want it to be a welcoming place to learn and collaborate.
This issue appears to have been written with AI assistance. Please explain your issue in your own words and describe how you checked the information you provided. This helps us understand your issue and leaves more time to address other issues.
Please read our guidelines for using AI tools.
thejesh23 commentedon Nov 4, 2026
optparse removes positional arguments one at a time using
del rargs[0], causing quadratic performance due to repeated list shifting. With 200k arguments, parsing takes ~2.5s instead of ~8ms.I changed it to remove consecutive positional arguments in a single operation using
del rargs[:end], keeping option handling and callbacks unchanged.FYI I used caude code to investigate and prepare the fix, then verified it myself with benchmarks, the full test suite, and a fuzz test covering ~200k random argument lists. The output matched the old implementation.