Parse 0.2 after … as float literal, not member access
#1277
Conversation
ahoppen
force-pushed
the
ahoppen/float-literal-lexing
branch
from
193e7aa to
9cf5678
Compare
|
@swift-ci Please test |
| @@ -335,6 +337,8 @@ extension Lexer.Cursor { | |||
| flags.insert(.isAtStartOfLine) | |||
| } | |||
|
|
|||
| self.previousTokenKind = result.tokenKind.base | |||
There was a problem hiding this comment.
There's Cursor.backUp where previousTokenKind wouldn't be correct any more. We also don't set the state in there, but I suppose the assumption there is that we're always in normal when splitting tokens.
(Just have to pass the token kind through to backUp)
There was a problem hiding this comment.
Cursor.backUp is still correct because the previous token still has the same kind if we resetForSplit, which is the only caller of this function (and should remain to be the only caller).
If we have just lexed <.. as a binary operator, previousToken is binaryOperator. If we now back up by two characters to consume only <, we are now placed at the first . but the previous token continues to be of kind binaryOperator. So no change of previousTokenKind is necessary here.
There was a problem hiding this comment.
Makes sense. It would be nice to have a comment mentioning this (just in case we ever end up calling backUp elsewhere). Thanks for the explanation!
Deciding whether `0.2` should be lexed as a float literal or a member access is a little more difficult than just looking at the previous character because `0.2` might be preceeded by an operator like `…` or `.^.`, in which case it should be lexed as a float literal and not a member access. We might be able to do some disambiguation magic on whether the character before the period is also an operator continuation point but that seems fairly brittle to me. The sanest way of doing this, is to store the previously lexed token’s kind in the cursor and checking that. I measured and did not see a performance regregssion when parsing MovieSwiftUI. rdar://103273988
ahoppen
force-pushed
the
ahoppen/float-literal-lexing
branch
from
9cf5678 to
da28695
Compare
|
@swift-ci Please test |
|
@swift-ci Please test Linux |
|
@swift-ci Please test Linux |
Deciding whether
0.2should be lexed as a float literal or a member access is a little more difficult than just looking at the previous character because0.2might be preceeded by an operator like…or.^., in which case it should be lexed as a float literal and not a member access.We might be able to do some disambiguation magic on whether the character before the period is also an operator continuation point but that seems fairly brittle to me. The sanest way of doing this, is to store the previously lexed token’s kind in the cursor and checking that.
I measured and did not see a performance regregssion when parsing MovieSwiftUI.
rdar://103273988