[Macros] Support user-defined macros as compiler plugins #61734

rxwei · 2022-10-26T02:22:12Z

Allow user-defined macros to be loaded from dynamic libraries and evaluated.

Introduce a _CompilerPluginSupport module installed into the toolchain. Its _CompilerPlugin protocol acts as a stable interface between the compiler and user-defined macros.
Introduce a -load-plugin-library <path> attribute which allows users to specify dynamic libraries to be loaded into the compiler.

A macro library must declare a public top-level computed property public var allMacros: [Any.Type] and be compiled to a dynamic library. The compiler will call the getter of this property to obtain and register all macros.

Known issues:

We current do not have a way to strip out unnecessary symbols from the plugin dylib, i.e. produce a plugin library that does not contain SwiftSyntax symbols that will collide with the compiler itself.
MacroExpansionExpr's type is hard-coded as (Int, String). It should instead be specified by the macro via protocol requirements such as signature and genericSignature. We need more protocol requirements in _CompilerPlugin to handle this.
dlopen is not secure and is only for prototyping use here.

Friend PR: swiftlang/swift-syntax#1022

rxwei · 2022-10-26T02:23:38Z

swiftlang/swift-syntax#1022
@swift-ci please test

rxwei · 2022-10-26T18:24:28Z

swiftlang/swift-syntax#1022
@swift-ci please test

rxwei · 2022-10-26T23:33:30Z

swiftlang/swift-syntax#1022
@swift-ci please test

rxwei · 2022-10-27T04:44:53Z

@swift-ci please test macOS

rxwei · 2022-10-27T04:45:03Z

swiftlang/swift-syntax#1022
@swift-ci please test macOS

rxwei · 2022-10-27T05:01:29Z

swiftlang/swift-syntax#1022
@swift-ci please test

rxwei · 2022-10-28T03:40:36Z

swiftlang/swift-syntax#1022
@swift-ci please test

rxwei · 2022-10-28T04:56:34Z

swiftlang/swift-syntax#1022
@swift-ci please test

include/swift/AST/SearchPathOptions.h

DougGregor · 2022-10-28T05:29:33Z

lib/AST/CompilerPlugin.cpp

+  // TODO: Consider using runtime lookup to get all types that conform to the
+  // macro protocol instead of finding a global `allMacros` property.
+  std::string name;
+  // Build a mangling tree for


This is... quite clever. I wonder if we could do something simpler by forcing some more structure into the plugins themselves. For example, if allCases had a @_silgen_name("swift_plugin_allMacros") on it, then the name would be the same for all plugins and we wouldn't have to do mangling tricks at all.

Wouldn't multiple plugins end up with colliding symbols if we used @_silgen_name("swift_plugin_allMacros")?

We could definitely enforce some naming pattern, e.g. @_silgen_name("swift_plugin_<module_name>_allMacros"). But I still believe it's more hassle-free to just let the compiler handle this, even though building a mangling tree isn't pretty.

I was thinking we'd dlopen with RTLD_LOCAL to keep all of the symbols separate, so different plugins could have the same symbol. But your solution might be more hassle-free. Either way, we can change it later; this shouldn't block merging.

Interesting. Separately I wonder if RTLD_LOCAL is a solution to the duplicate symbols problem.

lib/AST/CompilerPlugin.cpp

lib/ASTGen/Sources/ASTGen/Macros.swift

DougGregor · 2022-10-28T05:37:20Z

lib/Sema/TypeCheckMacros.cpp

+
+  // Built-in macros go through `MacroSystem` in Swift Syntax linked to this
+  // compiler.
+  if (isBuiltinMacro) {


This is fine for now, but we should seek to unify them by having a single Macro structure that abstracts over builtin vs. loaded from a plugin so the rest of the compiler doesn't care.

I think handling of both of these can be cleaned up a bit, so don't fret over it in this PR.

stdlib/toolchain/CompilerPluginSupport/CMakeLists.txt

DougGregor · 2022-10-28T05:40:35Z

test/Sema/macro_expansion.swift

+// FIXME: Swift parser is not enabled on Linux CI yet.
+// REQUIRES: OS=macosx
+
+let _ = #stringify(1.byteSwapped + 2.advanced(by: 10))


Amazing! ;)

rxwei · 2022-10-28T18:46:52Z

swiftlang/swift-syntax#1022
@swift-ci please test macOS

rxwei · 2022-10-28T19:45:03Z

swiftlang/swift-syntax#1022
@swift-ci please test macOS

rxwei · 2022-10-28T21:11:55Z

swiftlang/swift-syntax#1022
@swift-ci please test linux

rxwei · 2022-10-28T23:27:06Z

swiftlang/swift-syntax#1022
@swift-ci please test macOS

Allow user-defined macros to be loaded from dynamic libraries and evaluated. - Introduce a _CompilerPluginSupport module installed into the toolchain. Its `_CompilerPlugin` protocol acts as a stable interface between the compiler and user-defined macros. - Introduce a `-load-plugin-library <path>` attribute which allows users to specify dynamic libraries to be loaded into the compiler. A macro library must declare a public top-level computed property `public var allMacros: [Any.Type]` and be compiled to a dynamic library. The compiler will call the getter of this property to obtain and regsiter all macros. Known issues: - We current do not have a way to strip out unnecessary symbols from the plugin dylib, i.e. produce a plugin library that does not contain SwiftSyntax symbols that will collide with the compiler itself. - `MacroExpansionExpr`'s type is hard-coded as `(Int, String)`. It should instead be specified by the macro via protocol requirements such as `signature` and `genericSignature`. We need more protocol requirements in `_CompilerPlugin` to handle this. - `dlopen` is not secure and is only for prototyping use here.

rxwei · 2022-10-29T19:41:13Z

swiftlang/swift-syntax#1022
@swift-ci please test

DougGregor

A couple of comments that we can address with follow-ups, but this is looking good! Let's merge

DougGregor · 2022-10-31T19:48:07Z

lib/AST/CompilerPlugin.cpp

+  // TODO: Consider using runtime lookup to get all types that conform to the
+  // macro protocol instead of finding a global `allMacros` property.
+  std::string name;
+  // Build a mangling tree for


I was thinking we'd dlopen with RTLD_LOCAL to keep all of the symbols separate, so different plugins could have the same symbol. But your solution might be more hassle-free. Either way, we can change it later; this shouldn't block merging.

DougGregor · 2022-10-31T19:53:34Z

lib/ASTGen/Sources/ASTGen/Macros.swift

@@ -121,7 +121,8 @@ public func getMacroEvaluationContext(


 @_cdecl("swift_ASTGen_evaluateMacro")
-public func evaluateMacro(
+@usableFromInline
+func evaluateMacro(


Yeah, this is a good idea to do in general for our entry points from the C++ world.

DougGregor · 2022-10-31T19:54:38Z

lib/CompilerPluginSupport/CMakeLists.txt

+
+  target_link_libraries("${library_name}"
+    PRIVATE
+    swiftCore


I'm surprised we need this, but... okay.

It might not be needed. Another problem is that currently the _CompilerPluginSupport module files aren't installed into the toolchain. I'll merge this and investigate it next.

DougGregor · 2022-10-31T19:56:28Z

lib/Sema/TypeCheckMacros.cpp

+
+  // Built-in macros go through `MacroSystem` in Swift Syntax linked to this
+  // compiler.
+  if (swift_ASTGen_lookupMacro(macroName.str().c_str())) {


We should track where macros come from (builtin vs. loaded) in the macro itself, which would simplify this code, but that can come later.

DougGregor · 2022-10-31T19:59:11Z

include/swift/AST/CompilerPlugin.h

+    ConformanceDescriptor = 0,
+    // static func _name() -> (UnsafePointer<UInt8>, count: Int8)
+    Name = 1,
+    // static func _kind() -> _CompilerPluginKind


Since the _CompilerPlugin protocol is resilient, we don't have to hard-code these. Rather, we could look up the requirement symbols and do the address calculation to find the entries dynamically.

I was thinking about that, but most of the protocol layout utilities are down in IRGen. Here we could look up a requirement but it doesn't provide the offset info. We could go though all requirements up front and build up an offset table ahead of time.

The requirement is the offset :). See https://github.com/apple/swift/blob/main/stdlib/public/runtime/Metadata.cpp#L5156-L5157

Make `Macro` refine `_CompilerPlugin` protocol when `_CompilerPluginSupport` exists. (`_CompilerPluginSupport` may not exist when ASTGen or SwiftCompilerSources is built with host tools.) This enables macros to talk to the compiler without an ABI-stable SwiftSyntax. Friend PR: swiftlang/swift#61734

rxwei marked this pull request as ready for review October 26, 2022 02:22

rxwei mentioned this pull request Oct 26, 2022

[Macros] Support user-defined macros as compiler plugins swiftlang/swift-syntax#1022

Merged

rxwei force-pushed the macro-compiler-plugin branch from 1ae1e0e to f154896 Compare October 26, 2022 18:24

rxwei requested review from ahoppen, bnbarham, CodaFi, DougGregor and rintaro as code owners October 26, 2022 18:24

rxwei force-pushed the macro-compiler-plugin branch from f154896 to 8c9922f Compare October 26, 2022 23:28

rxwei force-pushed the macro-compiler-plugin branch from 8c9922f to e864a54 Compare October 27, 2022 05:01

rxwei force-pushed the macro-compiler-plugin branch 8 times, most recently from a745059 to 340b2f7 Compare October 28, 2022 03:38

rxwei force-pushed the macro-compiler-plugin branch 5 times, most recently from 85123ee to fdce2b7 Compare October 28, 2022 04:51

rxwei force-pushed the macro-compiler-plugin branch from fdce2b7 to 1ed0057 Compare October 28, 2022 04:55

DougGregor reviewed Oct 28, 2022

View reviewed changes

rxwei force-pushed the macro-compiler-plugin branch 2 times, most recently from 894e1e7 to afc2687 Compare October 28, 2022 18:35

rxwei force-pushed the macro-compiler-plugin branch from afc2687 to 845f224 Compare October 28, 2022 19:44

rxwei force-pushed the macro-compiler-plugin branch from 845f224 to 2aef407 Compare October 29, 2022 19:40

rxwei requested a review from DougGregor October 29, 2022 21:02

DougGregor approved these changes Oct 31, 2022

View reviewed changes

rxwei merged commit 4ce1ebb into swiftlang:main Oct 31, 2022

rxwei deleted the macro-compiler-plugin branch October 31, 2022 21:07

[Macros] Support user-defined macros as compiler plugins #61734

[Macros] Support user-defined macros as compiler plugins #61734

Uh oh!

Conversation

rxwei commented Oct 26, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

rxwei commented Oct 26, 2022

Uh oh!

rxwei commented Oct 26, 2022

Uh oh!

rxwei commented Oct 26, 2022

Uh oh!

rxwei commented Oct 27, 2022

Uh oh!

rxwei commented Oct 27, 2022

Uh oh!

rxwei commented Oct 27, 2022

Uh oh!

rxwei commented Oct 28, 2022

Uh oh!

rxwei commented Oct 28, 2022

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

rxwei Oct 28, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

rxwei Oct 28, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

rxwei commented Oct 28, 2022

Uh oh!

rxwei commented Oct 28, 2022

Uh oh!

rxwei commented Oct 28, 2022

Uh oh!

rxwei commented Oct 28, 2022

Uh oh!

rxwei commented Oct 29, 2022

Uh oh!

DougGregor left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

rxwei Oct 31, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

rxwei commented Oct 26, 2022 •

edited

Loading

rxwei Oct 28, 2022 •

edited

Loading

rxwei Oct 28, 2022 •

edited

Loading

rxwei Oct 31, 2022 •

edited

Loading