Skip to content

[Lexer] Allow multiline delimiter to be escaped in raw Strings #23115

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
wants to merge 12 commits into from
Closed

[Lexer] Allow multiline delimiter to be escaped in raw Strings #23115

Show file tree
Hide file tree
Changes from 2 commits
Commits
Show all changes
12 commits
Select commit Hold shift + click to select a range
52068f0
[Lexer] SR-10011: Allow the multiline delimiter to be escaped in raw …
maustinstar Mar 5, 2019
0d7decf
Merge remote-tracking branch 'refs/remotes/origin/master'
maustinstar Mar 5, 2019
6ba19f5
Remove tests from incorrect location. Adapt tests for escaping multil…
maustinstar Mar 9, 2019
abfa95e
Iterate through buffer to find raw delimiters before newline
maustinstar Mar 9, 2019
e1b8ddf
Added missing break statement
maustinstar Mar 9, 2019
22d4de8
Apply markup edits from code review
xwu Mar 9, 2019
4a6155b
Use delimiterMatches after checking for quote.
maustinstar Mar 11, 2019
9e378c9
Edit comments
maustinstar Mar 11, 2019
c923786
Merge remote-tracking branch 'refs/remotes/origin/master'
maustinstar Mar 13, 2019
ff29bdb
Add test cases
maustinstar Mar 15, 2019
c2e9818
Check for CustomDelimiterLen > 0
maustinstar Mar 25, 2019
4540f70
Alter test; remove unterminated multiline string that eats entire file
maustinstar Apr 18, 2019
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
16 changes: 14 additions & 2 deletions lib/Parse/Lexer.cpp
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1843,9 +1843,21 @@ void Lexer::lexStringLiteral(unsigned CustomDelimiterLen) {
assert((QuoteChar == '"' || QuoteChar == '\'') && "Unexpected start");

bool IsMultilineString = advanceIfMultilineDelimiter(CurPtr, Diags);
if (IsMultilineString && *CurPtr != '\n' && *CurPtr != '\r')
diagnose(CurPtr, diag::lex_illegal_multiline_string_start)
// Test for single-line Strings that may resemble multiline delimiter
if (IsMultilineString && *CurPtr != '\n' && *CurPtr != '\r') {
const char *TmpPtr = CurPtr + 1;
TmpPtr = skipToEndOfInterpolatedExpression(TmpPtr, BufferEnd, false);
TmpPtr = TmpPtr - CustomDelimiterLen;
if (CustomDelimiterLen != 0
&& delimiterMatches(CustomDelimiterLen, TmpPtr, Diags)) {
// Undo effects from falsely detecting multiline delimiter
CurPtr = CurPtr - 2;
IsMultilineString = false;
} else {
diagnose(CurPtr, diag::lex_illegal_multiline_string_start)
.fixItInsert(Lexer::getSourceLoc(CurPtr), "\n");
}
}

bool wasErroneous = false;
while (true) {
Expand Down
26 changes: 26 additions & 0 deletions unittests/Parse/LexerTests.cpp
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -124,6 +124,32 @@ TEST_F(LexerTest, StringLiteralWithNUL1) {
EXPECT_EQ(Toks[1].getLength(), 0U);
}

TEST_F(LexerTest, StringLiteralFalseMultilineDelimiter) {
const char *Source =
"#\"\"\"meow\"#\n"
"#\"\"\"#"
;
std::vector<tok> ExpectedTokens{
tok::string_literal, tok::string_literal
};
std::vector<Token> Toks = checkLex(Source, ExpectedTokens);
EXPECT_EQ(Toks[0].getLength(), 10U);
EXPECT_EQ(Toks[1].getLength(), 5U);
}

TEST_F(LexerTest, StringLiteralInvalidMultilineDelimiter) {
const char *Source =
"#\"\"\"meow#\n"
"#\"\"\"meow"
;
std::vector<tok> ExpectedTokens{
tok::unknown, tok::unknown
};
std::vector<Token> Toks = checkLex(Source, ExpectedTokens);
EXPECT_EQ(Toks[0].getLength(), 9U);
EXPECT_EQ(Toks[1].getLength(), 8U);
}

TEST_F(LexerTest, ContentStartHashbangSkip) {
const char *Source = "#!/usr/bin/swift\naaa";

Expand Down