Skip to content

Commit 87c3936

Browse files
committed
Tokenizer/PHP: efficiency fix
Reminder: the PHP::processAdditional()` method walks _back_ from the end of the file to the beginning. The type handling retokenization layer is triggered for each `&`, `|` and `)` the tokenizer encounters. When something is recognized as a valid type declaration, the relevant tokens will all be retokenized in one go the first time the type handling layer is triggered, which means that - as the type tokens will have been retokenized already -, the type handling layer will not be triggered again for any of the other type related tokens in the type. However, if the type is *not* recognized as a valid type, the type handling layer will keep getting retriggered and will (correctly) keep concluding this is not a valid type. The change in this PR, prevents the type handling layer from doing any work when it is retriggered on a token which was previously already seen and concluded to be, either not part of a type or part of an invalid type. This should make the tokenizer marginally faster for complex types containing an error, like `(A&B|(C&D)|(E&F)`.
1 parent 07477a7 commit 87c3936

File tree

1 file changed

+12
-1
lines changed

1 file changed

+12
-1
lines changed

src/Tokenizers/PHP.php

Lines changed: 12 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -2602,7 +2602,9 @@ protected function processAdditional()
26022602

26032603
$this->createAttributesNestingMap();
26042604

2605-
$numTokens = count($this->tokens);
2605+
$numTokens = count($this->tokens);
2606+
$lastSeenTypeToken = $numTokens;
2607+
26062608
for ($i = ($numTokens - 1); $i >= 0; $i--) {
26072609
// Check for any unset scope conditions due to alternate IF/ENDIF syntax.
26082610
if (isset($this->tokens[$i]['scope_opener']) === true
@@ -3038,6 +3040,12 @@ protected function processAdditional()
30383040
|| $this->tokens[$i]['code'] === T_BITWISE_AND
30393041
|| $this->tokens[$i]['code'] === T_CLOSE_PARENTHESIS
30403042
) {
3043+
if ($lastSeenTypeToken < $i) {
3044+
// We've already examined this code to check if it is a type declaration and concluded it wasn't.
3045+
// No need to do it again.
3046+
continue;
3047+
}
3048+
30413049
/*
30423050
Convert "|" to T_TYPE_UNION or leave as T_BITWISE_OR.
30433051
Convert "&" to T_TYPE_INTERSECTION or leave as T_BITWISE_AND.
@@ -3250,6 +3258,9 @@ protected function processAdditional()
32503258
break;
32513259
}//end for
32523260

3261+
// Remember the last token we examined as part of the (non-)"type declaration".
3262+
$lastSeenTypeToken = $x;
3263+
32533264
if ($confirmed === false
32543265
&& $suspectedType === 'property or parameter'
32553266
&& isset($this->tokens[$i]['nested_parenthesis']) === true

0 commit comments

Comments
 (0)