Skip to content

Commit 316f847

Browse files
committed
Split SwiftSyntaxParser into a separate module
When SwiftSyntax is just being used to generate code and not to parse Swift source files, there is no reason that a compatible `_InternalSwiftSyntaxParser.dylib` needs to be present. We do, however, currently have the requirement of always needing a `_InternalSwiftSyntaxParser.dylib` because `SyntaxParser` lives inside the `SwiftSyntax` module and thus `SwiftSyntax` needs to link against `_InternalSwiftSyntaxParser.dylib`. To lift the requirement, split `SwiftSyntax` into two modules: `SwiftSyntax` has no requirement on `_InternalSwiftSyntaxParser.dylib`. It only offers functionality to programatically generate `SwiftSyntax` trees and walk them. The new `SwiftSyntaxParser` module links against `_InternalSwiftSyntaxParser.dylib` and provides the ability to generate `SwiftSyntax` trees from source files. To efficiently generate `SwiftSyntax` nodes from the parser, `RawSyntax` (and its child types) does depend on the layout of the C nodes (for example, `TokenData` stores `CTriviaPiece`s as trailing trivia and only constructs the trivia’s Swift representation when requested). I don’t think it’s possible to eliminate that behaviour without eager translation and thus decided to include a copy of the C nodes inside the `SwiftSyntax`. When `SwiftSyntax` is used on its own, they are never used and thus their value is irrelevant. When a source file is parsed through `SwiftSyntaxParser`, it first verifies that the C node layouts match those defined in `_InternalSwiftSyntaxParser.dylib`, similar to how we currently compute the node hash. If the layouts are not compatible, we throw an error. We also verify that the files definingt these types match when invoking `build-script` (e.g. in CI).
1 parent 8738d98 commit 316f847

38 files changed

+526
-146
lines changed

Changelog.md

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -5,6 +5,9 @@ Note: This is in reverse chronological order, so newer entries are added to the
55
## `main`
66

77
* To clarify that the edits passed to `IncrementalParseTransition` are applied concurrently, introduce a new `ConcurrentEdit` type that provides the guarantee and allows translation of sequentially applied edits to the expected concurrent form.
8+
* The `SwiftSyntaxParser` type and a few related types now live in their own module (also named `SwiftSyntaxParser`). This allows using `SwiftSyntax` for code generation purposes without having a compatible `_InternalSwiftSyntaxParser.dylib` around.
9+
10+
`import SwiftSyntaxParser` where necessary.
811

912
## Swift 5.3
1013

Package.swift

Lines changed: 6 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -7,11 +7,13 @@ let package = Package(
77
name: "SwiftSyntax",
88
targets: [
99
.target(name: "_CSwiftSyntax"),
10-
.testTarget(name: "SwiftSyntaxTest", dependencies: ["SwiftSyntax"], exclude: ["Inputs"]),
10+
.testTarget(name: "SwiftSyntaxTest", dependencies: ["SwiftSyntax"]),
1111
.target(name: "SwiftSyntaxBuilder", dependencies: ["SwiftSyntax"]),
1212
.testTarget(name: "SwiftSyntaxBuilderTest", dependencies: ["SwiftSyntaxBuilder"]),
13-
.target(name: "lit-test-helper", dependencies: ["SwiftSyntax"]),
14-
.testTarget(name: "PerformanceTest", dependencies: ["SwiftSyntax"])
13+
.target(name: "SwiftSyntaxParser", dependencies: ["SwiftSyntax"]),
14+
.testTarget(name: "SwiftSyntaxParserTest", dependencies: ["SwiftSyntaxParser"], exclude: ["Inputs"]),
15+
.target(name: "lit-test-helper", dependencies: ["SwiftSyntax", "SwiftSyntaxParser"]),
16+
.testTarget(name: "PerformanceTest", dependencies: ["SwiftSyntax", "SwiftSyntaxParser"])
1517
// Also see targets added below
1618
]
1719
)
@@ -52,4 +54,5 @@ if ProcessInfo.processInfo.environment["SWIFT_BUILD_SCRIPT_ENVIRONMENT"] != nil
5254
}
5355

5456
package.products.append(.library(name: "SwiftSyntax", type: libraryType, targets: ["SwiftSyntax"]))
57+
package.products.append(.library(name: "SwiftSyntaxParser", type: libraryType, targets: ["SwiftSyntaxParser"]))
5558
package.products.append(.library(name: "SwiftSyntaxBuilder", type: libraryType, targets: ["SwiftSyntaxBuilder"]))

Sources/SwiftSyntax/CNodes.swift

Lines changed: 72 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,72 @@
1+
//===---------------------------- CNodes.swift ----------------------------===//
2+
//
3+
// This source file is part of the Swift.org open source project
4+
//
5+
// Copyright (c) 2014 - 2021 Apple Inc. and the Swift project authors
6+
// Licensed under Apache License v2.0 with Runtime Library Exception
7+
//
8+
// See https://swift.org/LICENSE.txt for license information
9+
// See https://swift.org/CONTRIBUTORS.txt for the list of Swift project authors
10+
//
11+
//===----------------------------------------------------------------------===//
12+
13+
@_implementationOnly import _CSwiftSyntax
14+
15+
// These two types need to be exposed publicly, so they can't rely on the C
16+
// types defined in _CSwiftSyntax.
17+
public typealias CSyntaxKind = UInt16
18+
public typealias CClientNode = UnsafeMutableRawPointer
19+
20+
typealias CSyntaxNode = swiftparse_syntax_node_t
21+
typealias CSyntaxNodePtr = UnsafePointer<CSyntaxNode>
22+
typealias CTokenKind = swiftparse_token_kind_t
23+
typealias CTriviaKind = swiftparse_trivia_kind_t
24+
typealias CTokenData = swiftparse_token_data_t
25+
typealias CLayoutData = swiftparse_layout_data_t
26+
typealias CTriviaPiecePtr = UnsafePointer<CTriviaPiece>
27+
typealias CTriviaPiece = swiftparse_trivia_piece_t
28+
29+
/// Computes a hash value that describes the layout of all C nodes which are
30+
/// passed as opaque values between `SwiftSyntaxParser` and `SwiftSyntax`.
31+
/// This should match the value returned by the `cNodeLayoutHash` function in
32+
/// the `SwiftSyntaxParser` module.
33+
public func cNodeLayoutHash() -> Int {
34+
var hasher = Hasher()
35+
36+
// These two types are not defined in terms of the C types in SwiftSyntax.
37+
// Let's include them specifically in the hash as well.
38+
hasher.combine(MemoryLayout<CSyntaxKind>.size)
39+
hasher.combine(MemoryLayout<CClientNode>.size)
40+
41+
hasher.combine(MemoryLayout<swiftparse_range_t>.size)
42+
hasher.combine(MemoryLayout<swiftparse_range_t>.offset(of: \.offset))
43+
hasher.combine(MemoryLayout<swiftparse_range_t>.offset(of: \.length))
44+
45+
hasher.combine(MemoryLayout<swiftparse_trivia_kind_t>.size)
46+
hasher.combine(MemoryLayout<swiftparse_token_kind_t>.size)
47+
hasher.combine(MemoryLayout<swiftparse_syntax_kind_t>.size)
48+
49+
hasher.combine(MemoryLayout<swiftparse_client_node_t>.size)
50+
51+
hasher.combine(MemoryLayout<swiftparse_trivia_piece_t>.size)
52+
hasher.combine(MemoryLayout<swiftparse_trivia_piece_t>.offset(of: \.length))
53+
hasher.combine(MemoryLayout<swiftparse_trivia_piece_t>.offset(of: \.kind))
54+
55+
hasher.combine(MemoryLayout<swiftparse_token_data_t>.size)
56+
hasher.combine(MemoryLayout<swiftparse_token_data_t>.offset(of: \.leading_trivia))
57+
hasher.combine(MemoryLayout<swiftparse_token_data_t>.offset(of: \.trailing_trivia))
58+
hasher.combine(MemoryLayout<swiftparse_token_data_t>.offset(of: \.leading_trivia_count))
59+
hasher.combine(MemoryLayout<swiftparse_token_data_t>.offset(of: \.trailing_trivia_count))
60+
hasher.combine(MemoryLayout<swiftparse_token_data_t>.offset(of: \.kind))
61+
hasher.combine(MemoryLayout<swiftparse_token_data_t>.offset(of: \.range))
62+
63+
hasher.combine(MemoryLayout<swiftparse_layout_data_t>.size)
64+
hasher.combine(MemoryLayout<swiftparse_layout_data_t>.offset(of: \.nodes))
65+
hasher.combine(MemoryLayout<swiftparse_layout_data_t>.offset(of: \.nodes_count))
66+
67+
hasher.combine(MemoryLayout<swiftparse_syntax_node_t>.size)
68+
hasher.combine(MemoryLayout<swiftparse_syntax_node_t>.offset(of: \.kind))
69+
hasher.combine(MemoryLayout<swiftparse_syntax_node_t>.offset(of: \.present))
70+
71+
return hasher.finalize()
72+
}

Sources/SwiftSyntax/IncrementalParseTransition.swift

Lines changed: 5 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -199,11 +199,11 @@ public struct ConcurrentEdits {
199199

200200
/// Provides a mechanism for the parser to skip regions of an incrementally
201201
/// updated source that was already parsed during a previous parse invocation.
202-
internal struct IncrementalParseLookup {
202+
public struct IncrementalParseLookup {
203203
fileprivate let transition: IncrementalParseTransition
204204
fileprivate var cursor: SyntaxCursor
205205

206-
init(transition: IncrementalParseTransition) {
206+
public init(transition: IncrementalParseTransition) {
207207
self.transition = transition
208208
self.cursor = .init(root: transition.previousTree.data.absoluteRaw)
209209
}
@@ -224,16 +224,16 @@ internal struct IncrementalParseLookup {
224224
///
225225
/// - Parameters:
226226
/// - offset: The byte offset of the source string that is currently parsed.
227-
/// - kind: The `SyntaxKind` that the parser expects at this position.
227+
/// - kind: The `CSyntaxKind` that the parser expects at this position.
228228
/// - Returns: A `SyntaxNode` node from the previous parse invocation,
229229
/// representing the contents of this region, if it is still valid
230230
/// to re-use. `nil` otherwise.
231-
mutating func lookUp(_ newOffset: Int, kind: SyntaxKind) -> SyntaxNode? {
231+
public mutating func lookUp(_ newOffset: Int, kind: CSyntaxKind) -> SyntaxNode? {
232232
guard let prevOffset = translateToPreEditOffset(newOffset) else {
233233
return nil
234234
}
235235
let prevPosition = AbsolutePosition(utf8Offset: prevOffset)
236-
let node = cursorLookup(prevPosition: prevPosition, kind: kind)
236+
let node = cursorLookup(prevPosition: prevPosition, kind: .fromRawValue(kind))
237237
if let delegate = reusedDelegate, let node = node {
238238
delegate.parserReusedNode(
239239
range: ByteSourceRange(offset: newOffset, length: node.byteSize),

Sources/SwiftSyntax/Misc.swift.gyb

Lines changed: 0 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -19,8 +19,6 @@
1919
//
2020
//===----------------------------------------------------------------------===//
2121

22-
@_implementationOnly import _InternalSwiftSyntaxParser
23-
2422
extension SyntaxNode {
2523
public var isUnknown: Bool { return raw.kind.isUnknown }
2624
public var asUnknown: UnknownSyntax? {
@@ -63,10 +61,3 @@ extension Syntax {
6361
}
6462
}
6563
}
66-
67-
extension SyntaxParser {
68-
static func verifyNodeDeclarationHash() -> Bool {
69-
return String(cString: swiftparse_syntax_structure_versioning_identifier()!) ==
70-
"${calculate_node_hash()}"
71-
}
72-
}

Sources/SwiftSyntax/RawSyntax.swift

Lines changed: 5 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -1216,12 +1216,12 @@ final class RawSyntax: ManagedBuffer<RawSyntaxBase, RawSyntaxDataElement> {
12161216
}
12171217

12181218
extension RawSyntax {
1219+
static func moveFromOpaque(_ cn: CClientNode) -> RawSyntax {
1220+
return Unmanaged<RawSyntax>.fromOpaque(cn).takeRetainedValue()
1221+
}
1222+
12191223
static func moveFromOpaque(_ cn: CClientNode?) -> RawSyntax? {
1220-
if let subnode = cn {
1221-
return Unmanaged<RawSyntax>.fromOpaque(subnode).takeRetainedValue()
1222-
} else {
1223-
return nil
1224-
}
1224+
return cn.map(moveFromOpaque)
12251225
}
12261226

12271227
static func getFromOpaque(_ cn: CClientNode?) -> RawSyntax? {
Lines changed: 54 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,54 @@
1+
//===--------------------- _SyntaxParserInterop.swift ---------------------===//
2+
//
3+
// This source file is part of the Swift.org open source project
4+
//
5+
// Copyright (c) 2014 - 2021 Apple Inc. and the Swift project authors
6+
// Licensed under Apache License v2.0 with Runtime Library Exception
7+
//
8+
// See https://swift.org/LICENSE.txt for license information
9+
// See https://swift.org/CONTRIBUTORS.txt for the list of Swift project authors
10+
//
11+
//===----------------------------------------------------------------------===//
12+
13+
/// Namespace for methods needed by the `SwiftSyntaxParser` module to
14+
/// efficiently create `SwiftSyntax` nodes from the C nodes created by the
15+
/// parser.
16+
public enum _SyntaxParserInterop {
17+
private static func getRetainedOpaque(rawSyntax: RawSyntax) -> CClientNode {
18+
return Unmanaged.passRetained(rawSyntax).toOpaque()
19+
}
20+
21+
/// Create a `RawSyntax` node for the given `cnode` and return an opaque
22+
/// pointer to the `RawSyntax` node (a `CClientNode`).
23+
/// After this method finishes, the `RawSyntax` node has a retain count of 1
24+
/// and is owned by whoever manages the returned `CClientNode`. Passing the
25+
/// `CClientNode` to `nodeFromRetainedOpaqueRawSyntax` transfers ownership
26+
/// back to `SwiftSyntax`.
27+
public static func getRetainedOpaqueRawSyntax(
28+
cnode: UnsafeRawPointer, source: String
29+
) -> CClientNode {
30+
let cnode = cnode.assumingMemoryBound(to: CSyntaxNode.self)
31+
// Transfer ownership of the object to the C parser. We get ownership back
32+
// via `moveFromCRawNode()`.
33+
let node = RawSyntax.create(from: cnode, source: source)
34+
return getRetainedOpaque(rawSyntax: node)
35+
}
36+
37+
/// Return an opaque pointer to the given `node`.
38+
/// After this method finishes, the `RawSyntax` node has a retain count of 1
39+
/// and is owned by whoever manages the returned `CClientNode`. Passing the
40+
/// `CClientNode` to `nodeFromRetainedOpaqueRawSyntax` transfers ownership
41+
/// back to `SwiftSyntax`.
42+
public static func getRetainedOpaqueRawSyntax(node: SyntaxNode)
43+
-> CClientNode {
44+
return getRetainedOpaque(rawSyntax: node.raw)
45+
}
46+
47+
/// After an opaque pointer to a `RawSyntax` node has been created using one
48+
/// of the methods above, transfer its ownership back to a `Syntax` node,
49+
/// which is managed by `SwiftSyntax`.
50+
public static func nodeFromRetainedOpaqueRawSyntax(_ cRoot: CClientNode)
51+
-> Syntax {
52+
return Syntax(SyntaxData.forRoot(RawSyntax.moveFromOpaque(cRoot)))
53+
}
54+
}

Sources/SwiftSyntax/gyb_generated/Misc.swift

Lines changed: 0 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -12,8 +12,6 @@
1212
//
1313
//===----------------------------------------------------------------------===//
1414

15-
@_implementationOnly import _InternalSwiftSyntaxParser
16-
1715
extension SyntaxNode {
1816
public var isUnknown: Bool { return raw.kind.isUnknown }
1917
public var asUnknown: UnknownSyntax? {
@@ -1964,10 +1962,3 @@ extension Syntax {
19641962
}
19651963
}
19661964
}
1967-
1968-
extension SyntaxParser {
1969-
static func verifyNodeDeclarationHash() -> Bool {
1970-
return String(cString: swiftparse_syntax_structure_versioning_identifier()!) ==
1971-
"e9565bceebb81b9c3a69c442a8576b029d7eaf9c"
1972-
}
1973-
}
Lines changed: 73 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,73 @@
1+
//===---------------------------- CNodes.swift ----------------------------===//
2+
//
3+
// This source file is part of the Swift.org open source project
4+
//
5+
// Copyright (c) 2014 - 2021 Apple Inc. and the Swift project authors
6+
// Licensed under Apache License v2.0 with Runtime Library Exception
7+
//
8+
// See https://swift.org/LICENSE.txt for license information
9+
// See https://swift.org/CONTRIBUTORS.txt for the list of Swift project authors
10+
//
11+
//===----------------------------------------------------------------------===//
12+
13+
@_implementationOnly import _InternalSwiftSyntaxParser
14+
15+
typealias CSyntaxNode = swiftparse_syntax_node_t
16+
typealias CTriviaPiece = swiftparse_trivia_piece_t
17+
typealias CSyntaxNodePtr = UnsafePointer<CSyntaxNode>
18+
typealias CTriviaPiecePtr = UnsafePointer<CTriviaPiece>
19+
typealias CSyntaxKind = swiftparse_syntax_kind_t
20+
typealias CTokenKind = swiftparse_token_kind_t
21+
typealias CTriviaKind = swiftparse_trivia_kind_t
22+
typealias CTokenData = swiftparse_token_data_t
23+
typealias CLayoutData = swiftparse_layout_data_t
24+
typealias CParseLookupResult = swiftparse_lookup_result_t
25+
typealias CClientNode = swiftparse_client_node_t
26+
typealias CDiagnostic = swiftparser_diagnostic_t
27+
typealias CFixit = swiftparse_diagnostic_fixit_t
28+
typealias CRange = swiftparse_range_t
29+
30+
/// Computes a hash value that describes the layout of all C nodes which are
31+
/// passed as opaque values between `SwiftSyntaxParser` and `SwiftSyntax`.
32+
/// This should match the value returned by the `cNodeLayoutHash` function in
33+
/// the `SwiftSyntax` module.
34+
public func cNodeLayoutHash() -> Int {
35+
var hasher = Hasher()
36+
37+
// These two types are not defined in terms of the C types in SwiftSyntax.
38+
// Let's include them specifically in the hash as well.
39+
hasher.combine(MemoryLayout<CSyntaxKind>.size)
40+
hasher.combine(MemoryLayout<CClientNode>.size)
41+
42+
hasher.combine(MemoryLayout<swiftparse_range_t>.size)
43+
hasher.combine(MemoryLayout<swiftparse_range_t>.offset(of: \.offset))
44+
hasher.combine(MemoryLayout<swiftparse_range_t>.offset(of: \.length))
45+
46+
hasher.combine(MemoryLayout<swiftparse_trivia_kind_t>.size)
47+
hasher.combine(MemoryLayout<swiftparse_token_kind_t>.size)
48+
hasher.combine(MemoryLayout<swiftparse_syntax_kind_t>.size)
49+
50+
hasher.combine(MemoryLayout<swiftparse_client_node_t>.size)
51+
52+
hasher.combine(MemoryLayout<swiftparse_trivia_piece_t>.size)
53+
hasher.combine(MemoryLayout<swiftparse_trivia_piece_t>.offset(of: \.length))
54+
hasher.combine(MemoryLayout<swiftparse_trivia_piece_t>.offset(of: \.kind))
55+
56+
hasher.combine(MemoryLayout<swiftparse_token_data_t>.size)
57+
hasher.combine(MemoryLayout<swiftparse_token_data_t>.offset(of: \.leading_trivia))
58+
hasher.combine(MemoryLayout<swiftparse_token_data_t>.offset(of: \.trailing_trivia))
59+
hasher.combine(MemoryLayout<swiftparse_token_data_t>.offset(of: \.leading_trivia_count))
60+
hasher.combine(MemoryLayout<swiftparse_token_data_t>.offset(of: \.trailing_trivia_count))
61+
hasher.combine(MemoryLayout<swiftparse_token_data_t>.offset(of: \.kind))
62+
hasher.combine(MemoryLayout<swiftparse_token_data_t>.offset(of: \.range))
63+
64+
hasher.combine(MemoryLayout<swiftparse_layout_data_t>.size)
65+
hasher.combine(MemoryLayout<swiftparse_layout_data_t>.offset(of: \.nodes))
66+
hasher.combine(MemoryLayout<swiftparse_layout_data_t>.offset(of: \.nodes_count))
67+
68+
hasher.combine(MemoryLayout<swiftparse_syntax_node_t>.size)
69+
hasher.combine(MemoryLayout<swiftparse_syntax_node_t>.offset(of: \.kind))
70+
hasher.combine(MemoryLayout<swiftparse_syntax_node_t>.offset(of: \.present))
71+
72+
return hasher.finalize()
73+
}

Sources/SwiftSyntax/Diagnostic.swift renamed to Sources/SwiftSyntaxParser/Diagnostic.swift

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -12,6 +12,8 @@
1212
// This file provides the Diagnostic, Note, and FixIt types.
1313
//===----------------------------------------------------------------------===//
1414

15+
import SwiftSyntax
16+
1517
/// A FixIt represents a change to source code in order to "correct" a
1618
/// diagnostic.
1719
public enum FixIt: Codable, CustomDebugStringConvertible {

Sources/SwiftSyntax/DiagnosticEngine.swift renamed to Sources/SwiftSyntaxParser/DiagnosticEngine.swift

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -14,6 +14,7 @@
1414
//===----------------------------------------------------------------------===//
1515

1616
import Foundation
17+
import SwiftSyntax
1718

1819
/// The DiagnosticEngine allows Swift tools to emit diagnostics.
1920
public class DiagnosticEngine {

Sources/SwiftSyntax/JSONDiagnosticConsumer.swift renamed to Sources/SwiftSyntaxParser/JSONDiagnosticConsumer.swift

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -14,6 +14,7 @@
1414
//===----------------------------------------------------------------------===//
1515

1616
import Foundation
17+
import SwiftSyntax
1718

1819
public final class JSONDiagnosticConsumer: DiagnosticConsumer {
1920
/// Enumerates the possible places this consumer might output diagnostics.
Lines changed: 27 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,27 @@
1+
%{
2+
# -*- mode: Swift -*-
3+
from gyb_syntax_support import calculate_node_hash
4+
# Ignore the following admonition it applies to the resulting .swift file only
5+
}%
6+
//// Automatically Generated From NodeDeclarationHash.swift.gyb.
7+
//// Do Not Edit Directly!
8+
//===--------------------- NodeDeclarationHash.swift ----------------------===//
9+
//
10+
// This source file is part of the Swift.org open source project
11+
//
12+
// Copyright (c) 2014 - 2021 Apple Inc. and the Swift project authors
13+
// Licensed under Apache License v2.0 with Runtime Library Exception
14+
//
15+
// See https://swift.org/LICENSE.txt for license information
16+
// See https://swift.org/CONTRIBUTORS.txt for the list of Swift project authors
17+
//
18+
//===----------------------------------------------------------------------===//
19+
20+
@_implementationOnly import _InternalSwiftSyntaxParser
21+
22+
extension SyntaxParser {
23+
static func verifyNodeDeclarationHash() -> Bool {
24+
return String(cString: swiftparse_syntax_structure_versioning_identifier()!) ==
25+
"${calculate_node_hash()}"
26+
}
27+
}

0 commit comments

Comments
 (0)