Skip to content

[DNM] Parse /.../ regex literals #41767

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
wants to merge 4 commits into from
Closed

[DNM] Parse /.../ regex literals #41767

wants to merge 4 commits into from

Conversation

hamishknight
Copy link
Contributor

@hamishknight hamishknight commented Mar 10, 2022 •
edited
Loading

Try to lex regex literals with /.../ delimiters when in an expression context in the parser, with a rule that it may not start with a space, tab or ) character.

@hamishknight hamishknight mentioned this pull request Mar 10, 2022
Closed
@hamishknight hamishknight requested a review from rintaro March 10, 2022 16:44
@hamishknight
Copy link
Contributor Author

@rintaro The lexing logic is still WIP as:

  • Sub-lexing still needs to taken care of
  • The heuristic needs formalizing
  • The lexing loop needs to handle additional cases e.g invisible ASCII
  • Prefix operators on regex literals needs to be implemented

But let me know if you have any initial thoughts

@hamishknight
Copy link
Contributor Author

@swift-ci please test

@hamishknight
Copy link
Contributor Author

@swift-ci please test source compatibility

@rintaro
Copy link
Member

rintaro commented Mar 10, 2022

  • Sub-lexing still needs to taken care of

What do you think are the issues sub-lexing might cause?

  • The lexing loop needs to handle additional cases e.g invisible ASCII

Could you elaborate? Any example?

@hamishknight
Copy link
Contributor Author

What do you think are the issues sub-lexing might cause?

I don't think it currently causes any issues, e.g for delayed body parsing we start on the opening {. But it seems like a sub-lexer should be able to handle the case where it starts on an opening /.

Could you elaborate? Any example?

String literals currently diagnose invisible ASCII characters they come across, we probably ought to do the same for regex literals.

https://github.com/apple/swift/blob/d3702bacbb418afd4d4915cbebd77265dad50a65/lib/Parse/Lexer.cpp#L1338-L1344

@hamishknight
Copy link
Contributor Author

@swift-ci please test macOS

@hamishknight
Copy link
Contributor Author

hamishknight commented Mar 11, 2022 •
edited
Loading

@swift-ci please build toolchain

macOS toolchain download: https://ci.swift.org/job/swift-PR-toolchain-macos/53/artifact/branch-main/swift-PR-41767-53-osx.tar.gz

@hamishknight hamishknight mentioned this pull request Mar 16, 2022
Closed
@hamishknight
Emit error on prefix operator containing /
e783f8f
@hamishknight
[WIP] Parse /.../ regex literals
34bec95 
Try to lex regex literals with `/.../` delimiters.
@hamishknight
[Lexer] Delay token diagnostics
b938fbe 
Queue up diagnostics when lexing, waiting until
`Lexer::lex` is called before emitting them. This
allows us to re-lex without having to deal with
previously invalid tokens.
@hamishknight
Error if /.../ is parsed by default
88a55e2 
Loudly error if `-enable-experimental-string-processing`
is not used. This allows us to easily check for
source compatibility conficts.
@hamishknight
Copy link
Contributor Author

Updated to re-lex regex literals from the parser

@hamishknight
Copy link
Contributor Author

@swift-ci please test

@hamishknight
Copy link
Contributor Author

@swift-ci please test source compatibility

@hamishknight hamishknight deleted the regular-grammar branch March 31, 2022 19:08
@hamishknight
Copy link
Contributor Author

Continuing work on #42119

@hamishknight hamishknight mentioned this pull request Mar 31, 2022
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@rintaro rintaro Awaiting requested review from rintaro rintaro is a code owner

@bnbarham bnbarham Awaiting requested review from bnbarham bnbarham will be requested when the pull request is marked ready for review bnbarham is a code owner

@CodaFi CodaFi Awaiting requested review from CodaFi CodaFi will be requested when the pull request is marked ready for review CodaFi is a code owner

@DougGregor DougGregor Awaiting requested review from DougGregor DougGregor will be requested when the pull request is marked ready for review DougGregor is a code owner

@hborla hborla Awaiting requested review from hborla hborla will be requested when the pull request is marked ready for review hborla is a code owner

@slavapestov slavapestov Awaiting requested review from slavapestov slavapestov will be requested when the pull request is marked ready for review slavapestov is a code owner

@xedin xedin Awaiting requested review from xedin xedin will be requested when the pull request is marked ready for review xedin is a code owner

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@hamishknight @rintaro