[🦩] Add proof of concept for ASTGen.

[zoecarver]

Uses SyntaxTransformVisitor to start implementing ASTGen. Just a proof of concept for now.

Refs swiftlang/swift#60943

[zoecarver]

This should probably get re-worked.

[ahoppen]

I think we should encode information that elements in a source file can only be StmtSyntax, ExprSyntax or DeclSyntax in the syntax tree so we can do an exhaustive switch here.

The best idea that I’ve come up with is to code-gen enums like the following based on the node_choices of CodeBlockItem.Item in CommonNodes.py

public struct CodeBlockItemSyntax /* CodeBlockItemSyntax conformances */ {
  enum Item {
    case stmt(StmtSyntax)
    case expr(ExprSyntax)
    case decl(DeclSyntax)
    case tokenList(TokenListSyntax)
    case nonEmptyTokenList(NonEmptyTokenListSyntax)
  
    func as(_ t: StmtSyntax.Type) -> StmtSyntax? { … }
    func as<T: StmtSyntaxProtocol>(_ t: T.Type) -> T? { … }
    func as(_ t: ExprSyntax.Type) -> ExprSyntax? { … }
    func as<T: ExprSyntaxProtocol>(_ t: T.Type) -> T? { … }
    func as(_ t: DeclSyntax.Type) -> DeclSyntax? { … }
    func as<T: DeclSyntaxProtocol>(_ t: T.Type) -> T? { … }
    func as(_ t: TokenList.Type) -> TokenListSyntax? { … }
    func as(_ t: nonEmptyTokenList.Type) -> NonEmptyTokenListSyntax? { … }
  }
  // existing code in CodeBlockItemSyntax
}

and change the type of item in CodeBlockItemSyntax from Syntax to CodeBlockItemSyntax.Item.
I think (haven’t tested) that this should be mostly (i.e. covering most common API use cases) source-comaptible with the current usage of these APIs.

That way you could exhaustivly switch over it here as

switch element.item {
    case stmt(StmtSyntax):
      out.append(SwiftTopLevelCodeDecl_createStmt(ctx, declContext, loc, swiftASTNodes, loc))
    case expr(ExprSyntax):
      out.append(SwiftTopLevelCodeDecl_createExpr(ctx, declContext, loc, swiftASTNodes, loc))
    case decl(DeclSyntax):
      out.append(swiftASTNodes)
    case tokenList(TokenListSyntax):
      fatalError("Should not be generated by SwiftParser")
    case nonEmptyTokenList(NonEmptyTokenListSyntax):
      fatalError("Should not be generated by SwiftParser")
}

I would really like to see this refactoring done before we fully start ASTGen. I don’t think it needs to block this PR but it’s something we should do ASAP.

A quick improvement without this refactoring: You can use element.item.as(SyntaxEnum.self) here and then switch over the enum cases.

[DougGregor]

I think this would be absolutely fantastic! The hardest thing about working with the syntax tree is that you don't know what cases to cover at any given point without working back from the grammar.

[DougGregor]

All that said, I don't know that this pull request is the place for this discussion. It's a general improvement to the ergonomics of the SwiftSyntax APIs.

[ahoppen]

I’m not really a fan that all these functions return UnsafeMutableRawPointer. I imagine it will be to easy to e.g. pass a FunctionDecl to a parameter that really expects an Identifier Can we do something more typed here? Like structs that wrap a void * or something like that?

[bnbarham]

I was going to mention this on the C++ side, but since you have here - IMO it'd be good to define a bunch of opaque types (eg. typedef void *swift_source_loc_t) and use those instead of void * directly.

And another comment mentions labels - we could just add swift_name to the C functions, eg. __attribute__((swift_name("FuncDecl_create(ctx:staticLoc:....)")))

[ahoppen]

What do you think about renaming this to `sourceFileBufferStart?

[ahoppen]

Can’t you just delete this?

[zoecarver]

No, we have a default implementation in the protocol that will be ambiguous.

[zoecarver]

We need the _disfavoredOverload attribute, so we need to add this overload.

[ahoppen]

I assume this is a pointer to ASTContext? In the absence of proper typing, could you rename the variable to astContext to at least give some clue what this is?

[ahoppen]

What do you think about extracting this to a helper function?

func loc<SyntaxType: SyntaxProtocol>(of node: SyntaxType) -> UnsafePointer<CChar> {
  return self.base.advanced(by: node.position.utf8Offset).raw
}

[zoecarver]

Great idea.

[ahoppen]

I really wish we had argument labels here. Just an idea: You define FuncDecl_create on the C++ side and on the Swift side you define

struct FuncDecl {
  var pointer: UnsafeMutableRawPointer

  init(<all the parameters>) {
    self.pointer = FuncDecl_create(<forward all the parameters>)
  }
}

The downside is that if you modify the parameters on FuncDecl_create, you also need to modify them on FuncDecl.init. But we could maybe also use FuncDecl as a typed wrapper instead of UnsafeMutableRawPointer in ASTGen.

And another thought: It might be good to namespace these types in an enum AST so they don’t conflict with types from SwiftSyntax or SwiftSyntaxBuilder.

[DougGregor]

I'll look more into details, but from a layering perspective I don't think this ASTGen library belongs in the swift-syntax package. It's too specific to the compiler itself, since it is layered on top of the compiler's (C++) AST representation, so it's a layering violation to have it here.

[zoecarver]

Closing this patch as I don't think we need it anymore.