Skip to content

Commit e851138

Browse files
committed
[mlir] Do not set lastToken in AsmParser's resetToken function
The member lastToken is the last token that was consumed by the parser. Resetting the lexer position to a different position does not cause any token to be consumed, so lastToken should not be updated. Setting it to curToken can cause the scopeLoc.end location of OperationDefinition to be off-by-one, pointing to the first token after the operation. An example for an operation for which the scopeLoc.end location was wrong before is: %0 = torch.vtensor.literal(dense_resource<__elided__> : tensor<768xbf16>) : !torch.vtensor<[768],bf16> . Here the scope end loc always pointed to the next token
1 parent 281d178 commit e851138

File tree

1 file changed

+6
-2
lines changed

1 file changed

+6
-2
lines changed

mlir/lib/AsmParser/Parser.h

Lines changed: 6 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -130,10 +130,14 @@ class Parser {
130130
consumeToken();
131131
}
132132

133-
/// Reset the parser to the given lexer position.
133+
/// Reset the parser to the given lexer position. Resetting the parser/lexer
134+
/// position does not update 'state.lastToken'. 'state.lastToken' is the
135+
/// last parsed token, and is used to provide the scope end location for
136+
/// OperationDefinitions. To ensure the correctness of the end location, the
137+
/// last consumed token of an OperationDefinition needs to be the last token
138+
/// belonging to it.
134139
void resetToken(const char *tokPos) {
135140
state.lex.resetPointer(tokPos);
136-
state.lastToken = state.curToken;
137141
state.curToken = state.lex.lexToken();
138142
}
139143

0 commit comments

Comments
 (0)