Lexer: extract matches with PREG_PATTERN_ORDER (~40% faster lexing)

[JanTvrdik]

Lexer::tokenize() spends most of its time building match arrays that it throws
away again. Two changes, which only pay off together, take ~40% off lexing and
~19% off end-to-end PHPDoc parsing. No API change: tokenize() still returns
list<array{string, int, int}>.

1. The numeric patterns hold 12 capturing groups nobody reads

TOKEN_FLOAT and TOKEN_INTEGER use (_[0-9]++)* for the _ digit
separators — 12 capturing groups between them. Nothing reads them, but
preg_match_all() collects every group of the pattern for every token it finds,
so each token carries 12 unused entries. Made them non-capturing.

2. PREG_SET_ORDER allocates one array per token

With the groups gone, PREG_SET_ORDER still allocates a two-entry array per
match — the whole match and the MARK — which tokenize() reads out and drops
immediately. PREG_PATTERN_ORDER instead collects the whole PHPDoc into two
arrays: $matches[0] for the values, $matches['MARK'] for the token types.
Two array headers per PHPDoc rather than one per token.

The two are multiplicative rather than additive, which is why they are one PR:

	lexing
baseline	—
non-capturing groups only	−7%
PREG_PATTERN_ORDER only (groups left capturing)	−7%
both	−41%

PREG_PATTERN_ORDER with capturing groups present is no better than the status
quo, because it then has to build one array per group of one entry per token.
That makes a capturing group in a token pattern a silent 35-percentage-point
regression, so LexerTest::testTokenPatternsHaveNoCapturingGroups() pins it.

Numbers

Corpus: 37 150 unique docblocks, 8.38 MiB, 2 355 859 tokens, collected by
tokenizing 10 001 PHP files and keeping every unique T_DOC_COMMENT — 20 OSS
libraries, one full vendor tree (nikic/php-parser, nette, symfony, react,
phpunit, jetbrains/phpstorm-stubs, doctrine, twig), phpstan-src/src, and this
library. Pinned to one core, php -n, minimum of 15 rounds with the variants
interleaved.

	lexing	parse (lex + PhpDocParser)
PHP 8.1	−42%
PHP 8.4	−40%
PHP 8.5	−40%	−19%
PHP 8.5, lines+indexes+comments		−19%
PHP 8.5, opcache + tracing JIT	−45%	−23%

Lexing is ~41% of total parse time in the baseline, which is why a lexer-only
change moves the end-to-end number this much.

Behaviour

Tokens are byte-identical to before, and so are the ASTs printed from them, over
all 37 150 docblocks (xxh128 over both). make check passes on PHP 8.5:
1750 tests, phpcs, PHPStan.

The only behavioural edge case is a subject that produces zero matches — an
empty string — where $matches['MARK'] is absent; that returns just the
TOKEN_END token, as before.

Not in this PR

While benchmarking I also tried changing the token representation, in case it is
of interest for a future major:

	lexing	parse	bytes/token
Token objects instead of tuples	+21%	+8%	164 (−39%)
struct-of-arrays TokenList	−56%	−31%	124 (−54%)

Token objects trade 21% lexing time for 39% less memory (new Token(...) is a
userland call where the array literal is one opcode; the parser does get ~2%
faster from reading properties instead of indexing arrays, but not enough).
Three parallel packed arrays win on both axes, because $matches[0] becomes the
values array with no per-token allocation at all — but that one changes the
return type of tokenize(), so PHPStan would have to follow. Happy to open an
issue with the details if either is worth discussing.

[Copilot]

Pull request overview

Optimizes PHPStan\PhpDocParser\Lexer\Lexer::tokenize() by reducing preg_match_all() allocations while preserving the existing list<array{string, int, int}> return type and token stream semantics.

Changes:

• Switch preg_match_all() usage from PREG_SET_ORDER to default PREG_PATTERN_ORDER and iterate over $matches[0] + $matches['MARK'] to avoid per-token match array allocations.
• Update numeric token regexes (TOKEN_FLOAT, TOKEN_INTEGER) to remove capturing groups by converting them to non-capturing groups.
• Add a regression test that asserts the lexer regexp produces no capturing groups (so $matches contains only [0, 'MARK']).

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated no comments.

File	Description
src/Lexer/Lexer.php	Uses PREG_PATTERN_ORDER match layout and removes capturing groups from numeric token patterns to reduce allocation overhead.
tests/PHPStan/Lexer/LexerTest.php	Adds a test to prevent reintroducing capturing groups into token patterns, preserving the performance win.

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[ondrejmirtes]

Yes, pleasen open an issue about future gains coupled with BC breaks.

[ondrejmirtes]

Thank you!

[SerafimArts]

Token objects trade 21% lexing time for 39% less memory (new Token(...) is a
userland call where the array literal is one opcode; the parser does get ~2%
faster from reading properties instead of indexing arrays, but not enough).

If you want to create objects, I recommend using prototypes and thus bypassing the constructor call, like:

$prototype = new Token();

foreach ($values as $value) {
    $token = clone $prototype;
    $token->id = $xxx;
    $token->value = $yyy;
    $token->zzz = $zzz;
    
    $tokens[] = $token;
}

Like this: https://github.com/phplrt/phplrt/blob/4.x/libs/components/lexer/src/Internal/Tokenizer.php#L160-L172

This is a hack that can significantly improve the performance of creating objects on hot paths.