[Syntax] test diagnostics in Lexer with libSyntax #14954
Merged
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
To test diagnostics in Lexer with libSyntax,
add
Diags
arguments toswift::tokenize
andswift:tokenizeWithTrivia
."with libSyntax
means
TriviaRetentionMode::WithTrivia` in Lexer.I updated
swift-syntax-test.cpp
to passingDiags
fromgetTokensFromFile
intotokenizeWithTrivia
.It makes
doFullLexRoundTrip
anddoDumpRawTokenSyntax
are print diagnostics to stderr.To keep compatibility,
I changed some
Diags.hadAnyError() ? EXIT_FAILURE : EXIT_SUCCESS
lines to justEXIT_SUCCESS
.Because, in previous
Diags
exists but not used so never had error.And I add test case to check this.
To emit embedded nul character warnings,
I use little trick to make source code which is embedded nul.
This warning comes from standard Lexer logic and shared with
WithTrivia
andWithoutTrivia
.The purpose of this test case is checking testing of diagnostics mechanism through
tokenizeWithTrivia
.In
lexer_invalid_nul.swift
,tr '\132' '\0'
means to change characterZ
to nul character.To avoid replace character in argument of tr itself,
it uses octal form of
Z
, is\132
.I planned to implement some diagnostics in Lexer which already exists with
WithoutTrivia
but not existsWithTrivia
.This PR also prepares to this my plan.