Skip to content

[libSyntax] Make RawSyntax a struct #18276

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 2 commits into from
Jul 27, 2018
Merged

[libSyntax] Make RawSyntax a struct #18276

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
8c8f4eb
[libSyntax] Make RawSyntax a struct
ahoppen May 24, 2018
478518f
[swiftSyntax] Add type annotations to speed up compile time
ahoppen Jun 26, 2018
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
140 changes: 74 additions & 66 deletions tools/SwiftSyntax/RawSyntax.swift
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -29,6 +29,16 @@ extension CodingUserInfoKey {
struct SyntaxNodeId: Hashable, Codable {
private let rawValue: UInt

// Start creating fresh node IDs for user generated nodes on in the upper
// half of the UInt value range so that they don't easily collide with node
// ids generated by the C++ side of libSyntax
private static var highestUsedId: UInt = UInt.max / 2

/// Generates a syntax node ID that has not been used yet
fileprivate static func generateFreshId() -> SyntaxNodeId {
return SyntaxNodeId(rawValue: highestUsedId + 1)
}

fileprivate init(rawValue: UInt) {
self.rawValue = rawValue
}
Expand All @@ -43,53 +53,57 @@ struct SyntaxNodeId: Hashable, Codable {
}
}

/// The data that is specific to a tree or token node
fileprivate indirect enum RawSyntaxData {
Copy link
Member

@rintaro rintaro Jul 27, 2018 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why this is declared as indirect? I know the original RawSyntax was also indirect. @harlanhaskins, what was the reason? maybe MemoryLayout<RawSyntax>.size?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

At the time of writing, it ensured both cases were heap allocated and that a copy = a retain, so we could share raw data. Pretty sure that is still the case.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's because .node has RawSyntax as its member which again has RawSyntaxData as one of its members. That's a cycle and thus the object size wouldn't be determined at runtime if the enum is not indirect.

Or TL;DR: It doesn't compile if the enum isn't indirect.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It should still compile. It’s already got a layer of indirection via the array of RawSyntax?.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@ahoppen It does compile because [RawSyntaxData] provides indirect semantics.

Copy link
Member

@rintaro rintaro Jul 27, 2018 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

At the time of writing, it ensured both cases were heap allocated and that a copy = a retain, so we could share raw data.

Makes sense. Thanks :)

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It does compile because [RawSyntaxData] provides indirect semantics.

Hmm. I was pretty sure it didn't compile at some point without indirect. But maybe that was just my imagination. @harlanhaskins's explanation makes sense. I'll leave it as is.

/// A tree node with a kind and an array of children
case node(kind: SyntaxKind, layout: [RawSyntax?])
/// A token with a token kind, leading trivia, and trailing trivia
case token(kind: TokenKind, leadingTrivia: Trivia, trailingTrivia: Trivia)
}

/// Represents the raw tree structure underlying the syntax tree. These nodes
/// have no notion of identity and only provide structure to the tree. They
/// are immutable and can be freely shared between syntax nodes.
indirect enum RawSyntax: Codable {
/// A tree node with a kind, an array of children, and a source presence.
case node(SyntaxKind, [RawSyntax?], SourcePresence, SyntaxNodeId?)
struct RawSyntax: Codable {
fileprivate let data: RawSyntaxData
let presence: SourcePresence
let id: SyntaxNodeId

init(kind: SyntaxKind, layout: [RawSyntax?], presence: SourcePresence,
id: SyntaxNodeId? = nil) {
self.data = .node(kind: kind, layout: layout)
self.presence = presence
self.id = id ?? SyntaxNodeId.generateFreshId()
}

/// A token with a token kind, leading trivia, trailing trivia, and a source
/// presence.
case token(TokenKind, Trivia, Trivia, SourcePresence, SyntaxNodeId?)
init(kind: TokenKind, leadingTrivia: Trivia, trailingTrivia: Trivia,
presence: SourcePresence, id: SyntaxNodeId? = nil) {
self.data = .token(kind: kind, leadingTrivia: leadingTrivia,
trailingTrivia: trailingTrivia)
self.presence = presence
self.id = id ?? SyntaxNodeId.generateFreshId()
}

/// The syntax kind of this raw syntax.
var kind: SyntaxKind {
switch self {
case .node(let kind, _, _, _): return kind
case .token(_, _, _, _, _): return .token
switch data {
case .node(let kind, _): return kind
case .token(_, _, _): return .token
}
}

var tokenKind: TokenKind? {
switch self {
case .node(_, _, _, _): return nil
case .token(let kind, _, _, _, _): return kind
switch data {
case .node(_, _): return nil
case .token(let kind, _, _): return kind
}
}

/// The layout of the children of this Raw syntax node.
var layout: [RawSyntax?] {
switch self {
case .node(_, let layout, _, _): return layout
case .token(_, _, _, _, _): return []
}
}

/// The source presence of this node.
var presence: SourcePresence {
switch self {
case .node(_, _, let presence, _): return presence
case .token(_, _, _, let presence, _): return presence
}
}

/// The ID of this node
var id: SyntaxNodeId? {
switch self {
case .node(_, _, _, let id): return id
case .token(_, _, _, _, let id): return id
switch data {
case .node(_, let layout): return layout
case .token(_, _, _): return []
}
}

Expand Down Expand Up @@ -152,12 +166,13 @@ indirect enum RawSyntax: Codable {
let presence = try container.decode(SourcePresence.self, forKey: .presence)
if let kind = try container.decodeIfPresent(SyntaxKind.self, forKey: .kind) {
let layout = try container.decode([RawSyntax?].self, forKey: .layout)
self = .node(kind, layout, presence, id)
self.init(kind: kind, layout: layout, presence: presence, id: id)
} else {
let kind = try container.decode(TokenKind.self, forKey: .tokenKind)
let leadingTrivia = try container.decode(Trivia.self, forKey: .leadingTrivia)
let trailingTrivia = try container.decode(Trivia.self, forKey: .trailingTrivia)
self = .token(kind, leadingTrivia, trailingTrivia, presence, id)
self.init(kind: kind, leadingTrivia: leadingTrivia,
trailingTrivia: trailingTrivia, presence: presence, id: id)
}
if let callback = decoder.userInfo[.rawSyntaxDecodedCallback] as?
(RawSyntax) -> Void {
Expand All @@ -168,23 +183,18 @@ indirect enum RawSyntax: Codable {
/// Encodes the RawSyntax to the provided Foundation Encoder.
func encode(to encoder: Encoder) throws {
var container = encoder.container(keyedBy: CodingKeys.self)
switch self {
case let .node(kind, layout, presence, id):
if let id = id {
try container.encode(id, forKey: .id)
}
switch self.data {
case let .node(kind, layout):
try container.encode(id, forKey: .id)
try container.encode(kind, forKey: .kind)
try container.encode(layout, forKey: .layout)
try container.encode(presence, forKey: .presence)
case let .token(kind, leadingTrivia, trailingTrivia, presence, id):
if let id = id {
try container.encode(id, forKey: .id)
}
case let .token(kind, leadingTrivia, trailingTrivia):
try container.encode(id, forKey: .id)
try container.encode(kind, forKey: .tokenKind)
try container.encode(leadingTrivia, forKey: .leadingTrivia)
try container.encode(trailingTrivia, forKey: .trailingTrivia)
try container.encode(presence, forKey: .presence)
}
try container.encode(presence, forKey: .presence)
}

/// Creates a RawSyntax node that's marked missing in the source with the
Expand All @@ -195,7 +205,7 @@ indirect enum RawSyntax: Codable {
/// - Returns: A new RawSyntax `.node` with the provided kind and layout, with
/// `.missing` source presence.
static func missing(_ kind: SyntaxKind) -> RawSyntax {
return .node(kind, [], .missing, nil)
return RawSyntax(kind: kind, layout: [], presence: .missing)
}

/// Creates a RawSyntax token that's marked missing in the source with the
Expand All @@ -204,7 +214,7 @@ indirect enum RawSyntax: Codable {
/// - Returns: A new RawSyntax `.token` with the provided kind, no
/// leading/trailing trivia, and `.missing` source presence.
static func missingToken(_ kind: TokenKind) -> RawSyntax {
return .token(kind, [], [], .missing, nil)
return RawSyntax(kind: kind, leadingTrivia: [], trailingTrivia: [], presence: .missing)
}

/// Returns a new RawSyntax node with the provided layout instead of the
Expand All @@ -213,10 +223,10 @@ indirect enum RawSyntax: Codable {
/// is returned.
/// - Parameter newLayout: The children of the new node you're creating.
func replacingLayout(_ newLayout: [RawSyntax?]) -> RawSyntax {
switch self {
case let .node(kind, _, presence, _):
return .node(kind, newLayout, presence, nil)
case .token(_, _, _, _, _): return self
switch data {
case let .node(kind, _):
return RawSyntax(kind: kind, layout: newLayout, presence: presence)
case .token(_, _, _): return self
}
}

Expand Down Expand Up @@ -259,14 +269,14 @@ extension RawSyntax: TextOutputStreamable {
/// - Parameter stream: The stream on which to output this node.
func write<Target>(to target: inout Target)
where Target: TextOutputStream {
switch self {
case .node(_, let layout, _, _):
switch data {
case .node(_, let layout):
for child in layout {
guard let child = child else { continue }
child.write(to: &target)
}
case let .token(kind, leadingTrivia, trailingTrivia, presence, _):
guard case .present = presence else { return }
case let .token(kind, leadingTrivia, trailingTrivia):
guard isPresent else { return }
for piece in leadingTrivia {
piece.write(to: &target)
}
Expand All @@ -280,14 +290,14 @@ extension RawSyntax: TextOutputStreamable {

extension RawSyntax {
func accumulateAbsolutePosition(_ pos: AbsolutePosition) {
switch self {
case .node(_, let layout, _, _):
switch data {
case .node(_, let layout):
for child in layout {
guard let child = child else { continue }
child.accumulateAbsolutePosition(pos)
}
case let .token(kind, leadingTrivia, trailingTrivia, presence, _):
guard case .present = presence else { return }
case let .token(kind, leadingTrivia, trailingTrivia):
guard isPresent else { return }
for piece in leadingTrivia {
piece.accumulateAbsolutePosition(pos)
}
Expand All @@ -299,31 +309,29 @@ extension RawSyntax {
}

var leadingTrivia: Trivia? {
switch self {
case .node(_, let layout, _, _):
switch data {
case .node(_, let layout):
for child in layout {
guard let child = child else { continue }
guard let result = child.leadingTrivia else { continue }
return result
}
return nil
case let .token(_, leadingTrivia, _, presence, _):
guard case .present = presence else { return nil }
case let .token(_, leadingTrivia, _):
return leadingTrivia
}
}

var trailingTrivia: Trivia? {
switch self {
case .node(_, let layout, _, _):
switch data {
case .node(_, let layout):
for child in layout.reversed() {
guard let child = child else { continue }
guard let result = child.trailingTrivia else { continue }
return result
}
return nil
case let .token(_, _, trailingTrivia, presence, _):
guard case .present = presence else { return nil }
case let .token(_, _, trailingTrivia):
return trailingTrivia
}
}
Expand Down
5 changes: 1 addition & 4 deletions tools/SwiftSyntax/SwiftSyntax.swift
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -78,10 +78,7 @@ public final class SyntaxTreeDeserializer {
}

private func addToLookupTable(_ node: RawSyntax) {
guard let id = node.id else {
return
}
nodeLookupTable[id] = node
nodeLookupTable[node.id] = node
}
}

Expand Down
40 changes: 22 additions & 18 deletions tools/SwiftSyntax/Syntax.swift
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -283,36 +283,40 @@ public struct TokenSyntax: _SyntaxBase, Hashable {
/// Returns a new TokenSyntax with its kind replaced
/// by the provided token kind.
public func withKind(_ tokenKind: TokenKind) -> TokenSyntax {
guard case let .token(_, leadingTrivia, trailingTrivia, presence, _) = raw else {
guard raw.kind == .token else {
fatalError("TokenSyntax must have token as its raw")
}
let (root, newData) = data.replacingSelf(.token(tokenKind, leadingTrivia,
trailingTrivia, presence,
nil))
let newRaw = RawSyntax(kind: tokenKind, leadingTrivia: raw.leadingTrivia!,
trailingTrivia: raw.trailingTrivia!,
presence: raw.presence)
let (root, newData) = data.replacingSelf(newRaw)
return TokenSyntax(root: root, data: newData)
}

/// Returns a new TokenSyntax with its leading trivia replaced
/// by the provided trivia.
public func withLeadingTrivia(_ leadingTrivia: Trivia) -> TokenSyntax {
guard case let .token(kind, _, trailingTrivia, presence, _) = raw else {
guard raw.kind == .token else {
fatalError("TokenSyntax must have token as its raw")
}
let (root, newData) = data.replacingSelf(.token(kind, leadingTrivia,
trailingTrivia, presence,
nil))
let newRaw = RawSyntax(kind: raw.tokenKind!, leadingTrivia: leadingTrivia,
trailingTrivia: raw.trailingTrivia!,
presence: raw.presence)
let (root, newData) = data.replacingSelf(newRaw)
return TokenSyntax(root: root, data: newData)
}

/// Returns a new TokenSyntax with its trailing trivia replaced
/// by the provided trivia.
public func withTrailingTrivia(_ trailingTrivia: Trivia) -> TokenSyntax {
guard case let .token(kind, leadingTrivia, _, presence, _) = raw else {
guard raw.kind == .token else {
fatalError("TokenSyntax must have token as its raw")
}
let (root, newData) = data.replacingSelf(.token(kind, leadingTrivia,
trailingTrivia, presence,
nil))
let newRaw = RawSyntax(kind: raw.tokenKind!,
leadingTrivia: raw.leadingTrivia!,
trailingTrivia: trailingTrivia,
presence: raw.presence)
let (root, newData) = data.replacingSelf(newRaw)
return TokenSyntax(root: root, data: newData)
}

Expand All @@ -333,26 +337,26 @@ public struct TokenSyntax: _SyntaxBase, Hashable {

/// The leading trivia (spaces, newlines, etc.) associated with this token.
public var leadingTrivia: Trivia {
guard case .token(_, let leadingTrivia, _, _, _) = raw else {
guard raw.kind == .token else {
fatalError("TokenSyntax must have token as its raw")
}
return leadingTrivia
return raw.leadingTrivia!
}

/// The trailing trivia (spaces, newlines, etc.) associated with this token.
public var trailingTrivia: Trivia {
guard case .token(_, _, let trailingTrivia, _, _) = raw else {
guard raw.kind == .token else {
fatalError("TokenSyntax must have token as its raw")
}
return trailingTrivia
return raw.trailingTrivia!
}

/// The kind of token this node represents.
public var tokenKind: TokenKind {
guard case .token(let kind, _, _, _, _) = raw else {
guard raw.kind == .token else {
fatalError("TokenSyntax must have token as its raw")
}
return kind
return raw.tokenKind!
}

public static func ==(lhs: TokenSyntax, rhs: TokenSyntax) -> Bool {
Expand Down
9 changes: 5 additions & 4 deletions tools/SwiftSyntax/SyntaxBuilders.swift.gyb
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -46,8 +46,9 @@ public struct ${Builder} {
if let list = layout[idx] {
layout[idx] = list.appending(elt.raw)
} else {
layout[idx] = RawSyntax.node(
.${child.swift_syntax_kind}, [elt.raw], .present, nil)
layout[idx] = RawSyntax(kind: SyntaxKind.${child.swift_syntax_kind},
layout: [elt.raw],
presence: SourcePresence.present)
}
}
% else:
Expand All @@ -68,8 +69,8 @@ public struct ${Builder} {
% end
% end

return SyntaxData(raw: .node(.${node.swift_syntax_kind},
layout, .present, nil))
return SyntaxData(raw: RawSyntax(kind: .${node.swift_syntax_kind},
layout: layout, presence: .present))
}
}

Expand Down
Loading