Return nil from parsing functions that didn’t consume any tokens
#656
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Recovery should be handled as unexpected nodes, which are usually produced by `expect`.
Previously, we returned a missing syntax in these cases, which made it harder for recovery to determine whether anything was consumed.
CodaFi
reviewed
Aug 30, 2022
| while currentToken.tokenKind != .eof { | ||
| extraneousTokens.append(RawSyntax(consumeAnyToken())) | ||
| } | ||
| let unexpectedBeforeEof = extraneousTokens.isEmpty ? nil : RawUnexpectedNodesSyntax(elements: extraneousTokens, arena: self.arena) |
There was a problem hiding this comment.
Does this mean that if we encounter an un-parsable decl in the middle of the file that we'll drain the rest of the tokens until the EOF into this token list?
There was a problem hiding this comment.
What do you mean by an un-parsable decl? If you mean
- A declaration with syntax errors: No, absolutely not. That declaration will get stuck into the the tree with the syntax errors.
- A declaration that’s so broken that the parser doesn’t make any progress trying to parse it: Yes. If we encounter such cases, we should improve recovery inside the parsing logic of that declaration so that the parser makes progress and puts the unexpected tokens into the declarations’s syntax node.
The idea is that unexpectedBeforeEof should contain those tokens that were previously errorToken of the last CodeBlockItem in the top level code block.
|
@swift-ci Please test |
|
After some offline discussion, we decided that returning a missing node is the correct design here. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Depends on #655
Previously, we returned a missing syntax in these cases, which made it harder for recovery to determine whether anything was consumed.