Implement SE-0325 Additional Package Plugin APIs (#3758)

* Extend the plugin API by passing down more information about the package graph

The original TargetBuildContext input structure was focused on build tools, which was the subject of SE-0303.  In order to prepare for new kinds of plugins such as custom commands, which have much more varied purposes and need a more diverse set of input information, we generalize this structure into a PluginContext that contains information about a whole package graph.

The graph — which is a directed acyclic graph and not trivially encodable into a transport format — is flattened and serialized into the plugin input JSON, and is reconstituted on the plugin side.  Path deduplication is used to reduce the amount of information sent over the wire.

A compatibility layer lets old plugins keep working with the old TargetBuildContext structure.  This is preliminary pending approval of an upcoming pitch and evolution proposal.

* Change Product and Target to protocols, and change the classes to structs.  Since structs cannot be nested in protocols, `Package.Dependency` moves out to a top-level `PackageDependency`, and similarly `Target.Dependency` moves out to `TargetDependency`.  Also define identifiers for Package, Product, and Target since they are now value types.  We cannot make them conform to `Identifiable` and still use arrays of `Product` and `Target` due to the Self requirement of `Identifiable`.  We can maybe resolve this in the future.  For now we use the ID that we already have from the model that we were handed from libSwiftPM as the identifier, but the exact value doesn't matter.

* Move the `builtProductsDirectory` in the package plugin context to be a private property.

The original intent was that it could be used in forming paths to built executables, but this is instead done through the `tools` to paths mapping.

* Bring draft implementation in line with proposal:

- add Origin to Package struct
- add SourceModuleTarget protocol, and separate SwiftSourceModuleTarget and ClangSourceModuleTarget
- fill in BinaryArtifactTarget and SystemLibraryTarget
- add ToolsVersion to Package struct

* Add BuildEnvironment to plugin evaluation.

* Adjust to changes to binary targets to have a `unknown` artifact type

* Add support for `pkgConfig` for system library targets.

Author: abertelrud

Fixtures/Miscellaneous/Plugins/ContrivedTestPlugin/Package.swift

@@ -32,5 +32,14 @@ let package = Package(
                 "MySourceGenBuildTool",
             ]
         ),
+        // Sample system library target for testing.
+        .systemLibrary(
+            name: "libpcre",
+            path: "Sources/libpcre",
+            pkgConfig: "libpcre",
+            providers: [
+                .apt(["libpcre-dev"])
+            ]
+        )
     ]
 )

Fixtures/Miscellaneous/Plugins/ContrivedTestPlugin/Sources/libpcre/module.modulemap

@@ -0,0 +1,5 @@
+module readline [system] {
+  header "sdk_libpcre.h"
+  link "libpcre"
+  export *
+}

Fixtures/Miscellaneous/Plugins/ContrivedTestPlugin/Sources/libpcre/sdk_libpcre.h

@@ -0,0 +1 @@
+#include <libpcre/libpcre.h>

Sources/Commands/SwiftTool.swift

@@ -766,6 +766,7 @@ public class SwiftTool {
             let result = try graph.invokePlugins(
                 outputDir: outputDir,
                 builtToolsDir: builtToolsDir,
+                buildEnvironment: buildEnvironment,
                 pluginScriptRunner: pluginScriptRunner,
                 observabilityScope: self.observabilityScope,
                 fileSystem: localFileSystem

Sources/PackagePlugin/CMakeLists.txt

@@ -7,13 +7,16 @@
 # See http://swift.org/CONTRIBUTORS.txt for Swift project authors
 
 add_library(PackagePlugin
-  PublicAPI/Command.swift
-  PublicAPI/Context.swift
-  PublicAPI/Diagnostics.swift
-  PublicAPI/FileList.swift
-  PublicAPI/Path.swift
-  PublicAPI/Protocols.swift
-  Implementation.swift)
+  Command.swift
+  Context.swift
+  Diagnostics.swift
+  Errors.swift
+  PackageModel.swift
+  Path.swift
+  Plugin.swift
+  PluginInput.swift
+  PluginOutput.swift
+  Protocols.swift)
 
 target_compile_options(PackagePlugin PUBLIC
   $<$<COMPILE_LANGUAGE:Swift>:-package-description-version$<SEMICOLON>999.0>)

Sources/PackagePlugin/PublicAPI/Command.swift renamed to Sources/PackagePlugin/Command.swift

@@ -6,14 +6,64 @@
 
  See http://swift.org/LICENSE.txt for license information
  See http://swift.org/CONTRIBUTORS.txt for Swift project authors
- */
+*/
+
+/// Provides information about the package for which the plugin is invoked,
+/// as well as contextual information based on the plugin's stated intent
+/// and requirements.
+public struct PluginContext {
+    /// Information about the package to which the plugin is being applied.
+    public let package: Package
+
+    /// The path of a writable directory into which the plugin or the build
+    /// commands it constructs can write anything it wants. This could include
+    /// any generated source files that should be processed further, and it
+    /// could include any caches used by the build tool or the plugin itself.
+    /// The plugin is in complete control of what is written under this di-
+    /// rectory, and the contents are preserved between builds.
+    ///
+    /// A plugin would usually create a separate subdirectory of this directory
+    /// for each command it creates, and the command would be configured to
+    /// write its outputs to that directory. The plugin may also create other
+    /// directories for cache files and other file system content that either
+    /// it or the command will need.
+    public let pluginWorkDirectory: Path
+
+    /// The path of the directory into which built products associated with
+    /// targets in the graph are written. This is a private implementation
+    /// detail.
+    let builtProductsDirectory: Path
+
+    /// Looks up and returns the path of a named command line executable tool.
+    /// The executable must be provided by an executable target or a binary
+    /// target on which the package plugin target depends. This function throws
+    /// an error if the tool cannot be found. The lookup is case sensitive.
+    public func tool(named name: String) throws -> Tool {
+        if let path = self.toolNamesToPaths[name] { return Tool(name: name, path: path) }
+        throw PluginContextError.toolNotFound(name: name)
+    }
+
+    /// A mapping from tool names to their definitions. Not directly available
+    /// to the plugin, but used by the `tool(named:)` API.
+    let toolNamesToPaths: [String: Path]
+    
+    /// Information about a particular tool that is available to a plugin.
+    public struct Tool {
+        /// Name of the tool (suitable for display purposes).
+        public let name: String
+
+        /// Full path of the built or provided tool in the file system.
+        public let path: Path
+    }
+}
+
+
 
 /// Provides information about the target being built, as well as contextual
 /// information such as the paths of the directories to which commands should
 /// be configured to write their outputs. This information should be used as
 /// part of generating the commands to be run during the build.
-public final class TargetBuildContext: Decodable {
-    
+public struct TargetBuildContext {
     /// The name of the target being built, as specified in the manifest.
     public let targetName: String
 
@@ -83,48 +133,21 @@ public final class TargetBuildContext: Decodable {
     /// The executable must be provided by an executable target or a binary
     /// target on which the package plugin target depends. This function throws
     /// an error if the tool cannot be found. The lookup is case sensitive.
-    public func tool(named name: String, line: UInt = #line) throws -> ToolInfo {
-        if let tool = self.tools[name] { return tool }
-        throw TargetBuildContextError.toolNotFound(name: name, line: line)
+    public func tool(named name: String) throws -> ToolInfo {
+        if let path = self.toolNamesToPaths[name] { return ToolInfo(name: name, path: path) }
+        throw PluginContextError.toolNotFound(name: name)
     }
 
     /// A mapping from tool names to their definitions. Not directly available
-    /// to the plugin but used by the `tool(named:)` API.
-    private let tools: [String: ToolInfo]
+    /// to the plugin, but used by the `tool(named:)` API.
+    let toolNamesToPaths: [String: Path]
 
     /// Information about a particular tool that is available to a plugin.
-    public struct ToolInfo: Codable {
-        
-        /// Name of the tool, suitable for display purposes.
+    public struct ToolInfo {
+        /// Name of the tool (suitable for display purposes).
         public let name: String
 
-        /// Path of the built or provided tool in the file system.
+        /// Full path of the built or provided tool in the file system.
         public let path: Path
     }
-
-    /// Internal
-    internal let pluginAction: PluginAction
-
-    internal enum PluginAction: String, Decodable {
-        case createBuildToolCommands
-    }
-}
-
-public enum TargetBuildContextError: Error {
-    
-    /// Could not find a tool with the given name. This could be either because
-    /// it doesn't exist, or because the plugin doesn't have a dependency on it.
-    case toolNotFound(name: String, line: UInt)
-}
-
-extension TargetBuildContextError: CustomStringConvertible {
-    
-    public var description: String {
-        switch self {
-        case .toolNotFound(let name, let line):
-            // FIXME: How to convey this line number to where it gets shown for errors at top level.
-            // Need to customize "Fatal error: Error raised at top level: plugin doesn’t have access to any tool named ‘MySourceGenBuildTools’ [5]: file Swift/ErrorType.swift, line 200"
-            return "plugin doesn’t have access to any tool named ‘\(name)’ [line \(line)]"
-        }
-    }
 }

Sources/PackagePlugin/PublicAPI/Context.swift renamed to Sources/PackagePlugin/Context.swift

@@ -12,32 +12,41 @@
 /// plugin. After emitting one or more errors, the plugin should return a
 /// non-zero exit code.
 public struct Diagnostics {
+    // Internal variable collecting the diagnostics that have been emitted.
+    static var emittedDiagnostics: [Diagnostic] = []
 
-    // This prevents a Diagnostics from being instantiated by the script.
+    // This prevents a Diagnostics struct from being instantiated by the script.
     internal init() {}
-    
+
     /// Severity of the diagnostic.
     public enum Severity: String, Encodable {
         case error, warning, remark
     }
-
+    
     /// Emits an error with a specified severity and message, and optional file path and line number.
     public static func emit(_ severity: Severity, _ message: String, file: Path? = #file, line: Int? = #line) {
-        output.diagnostics.append(Diagnostic(severity: severity, message: message, file: file, line: line))
+        self.emittedDiagnostics.append(Diagnostic(severity: severity, message: message, file: file, line: line))
     }
 
     /// Emits an error with the specified message, and optional file path and line number.
-    public static func error(_ message: String, file: Path? = nil, line: Int? = nil) {
-        output.diagnostics.append(Diagnostic(severity: .error, message: message, file: file, line: line))
+    public static func error(_ message: String, file: Path? = #file, line: Int? = #line) {
+        self.emit(.error, message, file: file, line: line)
     }
 
     /// Emits a warning with the specified message, and optional file path and line number.
-    public static func warning(_ message: String, file: Path? = nil, line: Int? = nil) {
-        output.diagnostics.append(Diagnostic(severity: .warning, message: message, file: file, line: line))
+    public static func warning(_ message: String, file: Path? = #file, line: Int? = #line) {
+        self.emit(.warning, message, file: file, line: line)
     }
 
     /// Emits a remark with the specified message, and optional file path and line number.
-    public static func remark(_ message: String, file: Path? = nil, line: Int? = nil) {
-        output.diagnostics.append(Diagnostic(severity: .remark, message: message, file: file, line: line))
+    public static func remark(_ message: String, file: Path? = #file, line: Int? = #line) {
+        self.emit(.remark, message, file: file, line: line)
     }
 }
+
+struct Diagnostic {
+    var severity: Diagnostics.Severity
+    var message: String
+    var file: Path?
+    var line: Int?
+}

Sources/PackagePlugin/PublicAPI/Diagnostics.swift renamed to Sources/PackagePlugin/Diagnostics.swift

@@ -0,0 +1,38 @@
+/*
+ This source file is part of the Swift.org open source project
+
+ Copyright (c) 2021 Apple Inc. and the Swift project authors
+ Licensed under Apache License v2.0 with Runtime Library Exception
+
+ See http://swift.org/LICENSE.txt for license information
+ See http://swift.org/CONTRIBUTORS.txt for Swift project authors
+ */
+
+public enum PluginContextError: Error {
+    /// Could not find a tool with the given name. This could be either because
+    /// it doesn't exist, or because the plugin doesn't have a dependency on it.
+    case toolNotFound(name: String)
+}
+
+extension PluginContextError: CustomStringConvertible {
+    public var description: String {
+        switch self {
+        case .toolNotFound(let name):
+            return "Plugin does not have access to a tool named ‘\(name)’"
+        }
+    }
+}
+
+public enum PluginDeserializationError: Error {
+    /// The input JSON is malformed in some way; the message provides more details.
+    case malformedInputJSON(_ message: String)
+}
+
+extension PluginDeserializationError: CustomStringConvertible {
+    public var description: String {
+        switch self {
+        case .malformedInputJSON(let message):
+            return "Malformed input JSON: \(message)"
+        }
+    }
+}