Skip to content

Add API to enable or disable bare slash regex parsing #448

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 1 commit into from
May 30, 2022
Merged

Add API to enable or disable bare slash regex parsing #448

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
50 changes: 42 additions & 8 deletions Sources/SwiftSyntaxParser/SyntaxParser.swift
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -58,6 +58,12 @@ public enum SyntaxParser {
/// - source: The source string to parse.
/// - parseTransition: Optional mechanism for incremental re-parsing.
/// - filenameForDiagnostics: Optional file name used for SourceLocation.
/// - languageVersion: Interpret input according to a specific Swift
/// language version number. If `nil`, this inherits the default from
/// the syntax parser library.
/// - enableBareSlashRegexLiteral: Enable or disable the use of forward
/// slash regular-expression literal syntax. If `nil`, this inherits the
/// default from the syntax parser library.
/// - diagnosticHandler: Optional callback that will be called for all
/// diagnostics the parser emits
/// - Returns: A top-level Syntax node representing the contents of the tree,
Expand All @@ -67,6 +73,8 @@ public enum SyntaxParser {
source: String,
parseTransition: IncrementalParseTransition? = nil,
filenameForDiagnostics: String = "",
languageVersion: String? = nil,
enableBareSlashRegexLiteral: Bool? = nil,
diagnosticHandler: ((Diagnostic) -> Void)? = nil
) throws -> SourceFileSyntax {
guard nodeHashVerifyResult && cnodeLayoutHashVerifyResult else {
Expand All @@ -78,8 +86,12 @@ public enum SyntaxParser {
var utf8Source = source
utf8Source.makeContiguousUTF8()

let rawSyntax = parseRaw(utf8Source, parseTransition, filenameForDiagnostics,
diagnosticHandler)
let rawSyntax = parseRaw(source: utf8Source,
parseTransition: parseTransition,
filenameForDiagnostics: filenameForDiagnostics,
languageVersion: languageVersion,
enableBareSlashRegexLiteral: enableBareSlashRegexLiteral,
diagnosticHandler: diagnosticHandler)

let base = _SyntaxParserInterop.nodeFromRetainedOpaqueRawSyntax(rawSyntax)
guard let file = base.as(SourceFileSyntax.self) else {
Expand All @@ -92,33 +104,55 @@ public enum SyntaxParser {
///
/// - Parameters:
/// - url: The file URL to parse.
/// - languageVersion: Interpret input according to a specific Swift
/// language version number. If `nil`, this inherits the default from
/// the syntax parser library.
/// - enableBareSlashRegexLiteral: Enable or disable the use of forward
/// slash regular-expression literal syntax. If `nil`, this inherits the
/// default from the syntax parser library.
/// - diagnosticHandler: Optional callback that will be called for all
/// diagnostics the parser emits
/// - Returns: A top-level Syntax node representing the contents of the tree,
/// if the parse was successful.
/// - Throws: `ParserError`
public static func parse(_ url: URL,
diagnosticHandler: ((Diagnostic) -> Void)? = nil) throws -> SourceFileSyntax {
public static func parse(
_ url: URL,
languageVersion: String? = nil,
enableBareSlashRegexLiteral: Bool? = nil,
Comment on lines +120 to +121
Copy link
Member

@rintaro rintaro May 24, 2022 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I feel like these should be a Options object. But I'm not sure how the API should look like, for example

// An array of enums
SyntaxParse.parse(
    url,
    options: [.languageVersion("5.6"), .bareSlashRegexLiteral(true)],
    diagnosticHandler: diagnosticHandler)

// Just a "options" object
SyntaxParse.parse(
    url,
    options: SyntaxParseOptions(languageVersion: "5.1", enableBareSlashRegexLiteral: true),
    diagnosticHandler: diagnosticHandler)

The advantage of options object is that you can easily pass around and store the object in the clients, when they want to create options at not exact the call site.

WDYT?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Though it's still only in the pitch phase, it seems like a long-term API for this should build on the ideas from @DougGregor 's future feature strings.

The value passed into SyntaxParser.parse could just be an array of strings denoting the feature names. While that's not as discoverable as strongly-typed arguments, if Swift Package Manager is already planning to use that approach (for the reasons described in that pitch), then the string array would at least be familiar and consistent for users. And it eliminates some potential instability from API users, since SwiftSyntax doesn't need to be updated every time new features are added.

This would also have the advantage of letting SwiftSyntax clients, like swift-format, provide the same -enable-future-feature flag and pass those strings verbatim to SwiftSyntax, instead of each client having to either provide their own flags for each of these features or do their own mapping.

With all that being said, I don't know what the best design for this API should look like now if you want to land it before the future-feature work while still keeping that as the eventual end state.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Alex and I chatted about this and we decided to accept this as is for now until the "language features" proposal is settled.

diagnosticHandler: ((Diagnostic) -> Void)? = nil
) throws -> SourceFileSyntax {
// Avoid using `String(contentsOf:)` because it creates a wrapped NSString.
let fileData = try Data(contentsOf: url)
let source = fileData.withUnsafeBytes { buf in
return String(decoding: buf.bindMemory(to: UInt8.self), as: UTF8.self)
}
return try parse(source: source, filenameForDiagnostics: url.path,
languageVersion: languageVersion,
enableBareSlashRegexLiteral: enableBareSlashRegexLiteral,
diagnosticHandler: diagnosticHandler)
}

private static func parseRaw(
_ source: String,
_ parseTransition: IncrementalParseTransition?,
_ filenameForDiagnostics: String,
_ diagnosticHandler: ((Diagnostic) -> Void)?
source: String,
parseTransition: IncrementalParseTransition?,
filenameForDiagnostics: String,
languageVersion: String?,
enableBareSlashRegexLiteral: Bool?,
diagnosticHandler: ((Diagnostic) -> Void)?
) -> CClientNode {
precondition(source.isContiguousUTF8)
let c_parser = swiftparse_parser_create()
defer {
swiftparse_parser_dispose(c_parser)
}
if let languageVersion = languageVersion {
languageVersion.withCString { languageVersionCString in
swiftparse_parser_set_language_version(c_parser, languageVersionCString)
}
}
if let enableBareSlashRegexLiteral = enableBareSlashRegexLiteral {
swiftparse_parser_set_enable_bare_slash_regex_literal(c_parser, enableBareSlashRegexLiteral)
}

let nodeHandler = { (cnode: CSyntaxNodePtr!) -> UnsafeMutableRawPointer in
return _SyntaxParserInterop.getRetainedOpaqueRawSyntax(cnode: cnode, source: source)
Expand Down
31 changes: 25 additions & 6 deletions Sources/lit-test-helper/main.swift
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -67,6 +67,12 @@ func printHelp() {
-out FILENAME
The file to which the source representation of the post-edit syntax
tree shall be written.
-swift-version
Interpret input according to a specific Swift language version
number
-enable-bare-slash-regex [0|1]
Enable or disable the use of forward slash regular-expression
literal syntax
""")
}

Expand Down Expand Up @@ -195,6 +201,13 @@ struct IncrementalEdit {
let replacement: String
}

func getSwiftLanguageVersionInfo(args: CommandLineArguments) -> (languageVersion: String?, enableBareSlashRegexLiteral: Bool?) {
return (
args["-swift-version"],
args["-enable-bare-slash-regex"].map({ $0 == "1" })
)
}

/// Rewrites a parsed tree with all constructed nodes.
class TreeReconstructor : SyntaxRewriter {
override func visit(_ token: TokenSyntax) -> Syntax {
Expand All @@ -207,8 +220,9 @@ class TreeReconstructor : SyntaxRewriter {

func performClassifySyntax(args: CommandLineArguments) throws {
let treeURL = URL(fileURLWithPath: try args.getRequired("-source-file"))
let versionInfo = getSwiftLanguageVersionInfo(args: args)

let tree = try SyntaxParser.parse(treeURL)
let tree = try SyntaxParser.parse(treeURL, languageVersion: versionInfo.languageVersion, enableBareSlashRegexLiteral: versionInfo.enableBareSlashRegexLiteral)
let result = ClassifiedSyntaxTreePrinter.print(Syntax(tree))
do {
// Sanity check that we get the same result if the tree has constructed nodes.
Expand Down Expand Up @@ -286,8 +300,9 @@ func performParseIncremental(args: CommandLineArguments) throws {
URL(fileURLWithPath: try args.getRequired("-old-source-file"))
let postEditURL = URL(fileURLWithPath: try args.getRequired("-source-file"))
let expectedReparseRegions = try args.getReparseRegions()
let versionInfo = getSwiftLanguageVersionInfo(args: args)

let preEditTree = try SyntaxParser.parse(preEditURL)
let preEditTree = try SyntaxParser.parse(preEditURL, languageVersion: versionInfo.languageVersion, enableBareSlashRegexLiteral: versionInfo.enableBareSlashRegexLiteral)
let edits = try parseIncrementalEditArguments(args: args)
let regionCollector = IncrementalParseReusedNodeCollector()
let editTransition = IncrementalParseTransition(
Expand Down Expand Up @@ -390,7 +405,8 @@ func verifyReusedRegions(expectedReparseRegions: [SourceRegion],

func performRoundtrip(args: CommandLineArguments) throws {
let sourceURL = URL(fileURLWithPath: try args.getRequired("-source-file"))
let tree = try SyntaxParser.parse(sourceURL)
let versionInfo = getSwiftLanguageVersionInfo(args: args)
let tree = try SyntaxParser.parse(sourceURL, languageVersion: versionInfo.languageVersion, enableBareSlashRegexLiteral: versionInfo.enableBareSlashRegexLiteral)
let treeText = tree.description

if let outURL = args["-out"].map(URL.init(fileURLWithPath:)) {
Expand Down Expand Up @@ -418,13 +434,15 @@ class NodePrinter: SyntaxAnyVisitor {

func printSyntaxTree(args: CommandLineArguments) throws {
let treeURL = URL(fileURLWithPath: try args.getRequired("-source-file"))
let tree = try SyntaxParser.parse(treeURL)
let versionInfo = getSwiftLanguageVersionInfo(args: args)
let tree = try SyntaxParser.parse(treeURL, languageVersion: versionInfo.languageVersion, enableBareSlashRegexLiteral: versionInfo.enableBareSlashRegexLiteral)
let printer = NodePrinter()
printer.walk(tree)
}

func printParserDiags(args: CommandLineArguments) throws {
let treeURL = URL(fileURLWithPath: try args.getRequired("-source-file"))
let versionInfo = getSwiftLanguageVersionInfo(args: args)

var diagCounter : (error: Int, warning: Int, note: Int) = (0, 0, 0)

Expand All @@ -442,7 +460,7 @@ func printParserDiags(args: CommandLineArguments) throws {
print(diagnostic.debugDescription)
}

_ = try SyntaxParser.parse(treeURL, diagnosticHandler: handleDiagnostic)
_ = try SyntaxParser.parse(treeURL, languageVersion: versionInfo.languageVersion, enableBareSlashRegexLiteral: versionInfo.enableBareSlashRegexLiteral, diagnosticHandler: handleDiagnostic)

print("\(diagCounter.error) error(s) \(diagCounter.warning) warnings(s) \(diagCounter.note) note(s)")
}
Expand Down Expand Up @@ -473,8 +491,9 @@ func diagnose(args: CommandLineArguments) throws {
}

let treeURL = URL(fileURLWithPath: try args.getRequired("-source-file"))
let versionInfo = getSwiftLanguageVersionInfo(args: args)

let tree = try SyntaxParser.parse(treeURL, diagnosticHandler: printDiagnostic)
let tree = try SyntaxParser.parse(treeURL, languageVersion: versionInfo.languageVersion, enableBareSlashRegexLiteral: versionInfo.enableBareSlashRegexLiteral, diagnosticHandler: printDiagnostic)

class DiagnoseUnknown: SyntaxAnyVisitor {
let diagnosticHandler: ((Diagnostic) -> Void)
Expand Down
12 changes: 8 additions & 4 deletions lit_tests/print_regex.swift
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,6 +1,10 @@
// RUN: %empty-directory(%t)
// RUN: %lit-test-helper -print-tree -source-file %s | %FileCheck %s --check-prefix REGEX
// RUN: %lit-test-helper -print-tree -source-file %s | %FileCheck %s --check-prefix REGEX_DISABLED
// RUN: %lit-test-helper -print-tree -source-file %s -enable-bare-slash-regex 1 -swift-version 5 | %FileCheck %s --check-prefix REGEX_ENABLED

_ = /abc/
// REGEX: _ </TokenSyntax></DiscardAssignmentExprSyntax><AssignmentExprSyntax><TokenSyntax>= </TokenSyntax></AssignmentExprSyntax><RegexLiteralExprSyntax><TokenSyntax>/abc/</TokenSyntax></RegexLiteralExprSyntax></ExprListSyntax></SequenceExprSyntax></CodeBlockItemSyntax></CodeBlockItemListSyntax><TokenSyntax>
// REGEX: </TokenSyntax></SourceFileSyntax>
_ = /abc/
// REGEX_DISABLED: _ </TokenSyntax></DiscardAssignmentExprSyntax><AssignmentExprSyntax><TokenSyntax>= </TokenSyntax></AssignmentExprSyntax><PrefixOperatorExprSyntax><TokenSyntax>/</TokenSyntax><PostfixUnaryExprSyntax><IdentifierExprSyntax><TokenSyntax>abc</TokenSyntax></IdentifierExprSyntax><TokenSyntax>/</TokenSyntax></PostfixUnaryExprSyntax></PrefixOperatorExprSyntax></ExprListSyntax></SequenceExprSyntax></CodeBlockItemSyntax></CodeBlockItemListSyntax><TokenSyntax>
// REGEX_DISABLED: </TokenSyntax></SourceFileSyntax>

// REGEX_ENABLED: _ </TokenSyntax></DiscardAssignmentExprSyntax><AssignmentExprSyntax><TokenSyntax>= </TokenSyntax></AssignmentExprSyntax><RegexLiteralExprSyntax><TokenSyntax>/abc/</TokenSyntax></RegexLiteralExprSyntax></ExprListSyntax></SequenceExprSyntax></CodeBlockItemSyntax></CodeBlockItemListSyntax><TokenSyntax>
// REGEX_ENABLED: </TokenSyntax></SourceFileSyntax>