Merge commit 'a111b4748be9fbc893abb8a6c4871bb59dabbde0' into 6.0/merge-main-2024-06-22

Author: ahoppen

CONTRIBUTING.md

@@ -68,6 +68,9 @@ If you want test your changes to SourceKit-LSP inside your editor, you can point
 "swift.sourcekit-lsp.serverPath": "/path/to/sourcekit-lsp/.build/arm64-apple-macosx/debug/sourcekit-lsp",
 ```
 
+> [!NOTE]
+> VS Code will note that that the `swift.sourcekit-lsp.serverPath` setting is deprecated. That’s because mixing and matching versions of sourcekit-lsp and Swift toolchains is generally not supported, so the settings is reserved for developers of SourceKit-LSP, which includes you. You can ignore this warning, If you have the `swift.path` setting to a recent [Swift Development Snapshot](https://www.swift.org/install).
+
 > [!TIP]
 > The easiest way to debug SourceKit-LSP is usually to write a test case that reproduces the behavior and then debug that. If that’s not possible, you can attach LLDB to the sourcekit-lsp launched by your and set breakpoints to debug. To do so on the command line, run
 > ```bash

Documentation/Enable Experimental Background Indexing.md

@@ -2,35 +2,23 @@
 
 Background indexing in SourceKit-LSP is available as an experimental feature. This guide shows how to set up background indexing and which caveats to expect.
 
-> [!WARNING]
-> Background indexing is still an experimental feature and still has some non-trivial limitations and bugs (see the known issues).
-
 ## Set Up
 
-Background indexing is still under active, rapid development, so please build SourceKit-LSP from the `main` branch using the following commands. Things are changing rapidly, so rebuilding daily is not a bad practice.
-
-```bash
-$ git clone https://github.com/apple/sourcekit-lsp.git
-$ cd sourcekit-lsp
-$ swift package update
-$ swift build -c release
-```
-
-Next, point your editor to use the just-built copy of SourceKit-LSP and enable background indexing by passing `--experimental-feature background-indexing` to sourcekit-lsp. In VS Code, this can be done by adding the following to your settings.json
-```json
-"swift.sourcekit-lsp.serverPath": "/path/to/sourcekit-lsp/.build/release/sourcekit-lsp",
-"swift.sourcekit-lsp.serverArguments": [ "--experimental-feature", "background-indexing" ],
-```
-
-Background indexing requires a Swift 6 toolchain. You can download Swift 6 nightly toolchains from https://www.swift.org/download/#swift-60-development.
+1. Install a `main` or `release/6.0` Swift Development Snapshot from https://www.swift.org/install.
+2. Point your editor to the newly installed toolchain.
+   - In VS Code on macOS this can be done by adding the following to your `settings.json`: `"swift.path": "/Library/Developer/Toolchains/swift-latest.xctoolchain/usr/bin"`
+   - In VS Code on other platforms, you need to set the `swift.path` to the `usr/bin` directory of your toolchain’s install location.
+   - Other editors likely also have a way to pick the Swift toolchain, the exact steps vary by your setup.
+3. Enable the experimental `background-indexing` feature.
+   - In VS Code, add the following to your `settings.json`: `"swift.sourcekit-lsp.serverArguments": [ "--experimental-feature", "background-indexing" ]`
+   - In other editors, change the invocation that launches `sourcekit-lsp` to pass the `--experimental-feature background-indexing` command line arguments. The exact steps vary by your setup.
 
 ## Known issues
 
-- Not really a background indexing related issue but Swift nightly toolchain snapshots are crashing on macOS 14.4 and 14.5 (swift#73327)[https://github.com/apple/swift/issues/73327]
-  - Workaround: Run the toolchains on an older version of macOS, if possible
 - Background Indexing is only supported for SwiftPM projects [#1269](https://github.com/apple/sourcekit-lsp/issues/1269), [#1271](https://github.com/apple/sourcekit-lsp/issues/1271)
 - If a module or one of its dependencies has a compilation error, it cannot be properly prepared for indexing because we are running a regular `swift build` to generate its modules [#1254](https://github.com/apple/sourcekit-lsp/issues/1254) rdar://128683404
-  - Workaround: Ensure that your files dependencies are in a buildable state to get an up-to-date index and proper cross-module functionality
+  - Workaround 1: Ensure that your files dependencies are in a buildable state to get an up-to-date index and proper cross-module functionality
+  - Workaround 2: Enable the `swiftpm-prepare-for-indexing` experimental feature, which continues to build Swift module even in the presence of errors.
 - If you change a function in a way that changes its USR but keeps it API compatible (such as adding a defaulted parameter), references to it will be lost and not re-indexed automatically [#1264](https://github.com/apple/sourcekit-lsp/issues/1264)
   - Workaround: Make some edit to the files that had references to re-index them
 - The index build is currently completely separate from the command line build generated using `swift build`. Building *does not* update the index (break your habits of always building!) [#1270](https://github.com/apple/sourcekit-lsp/issues/1270)
@@ -40,5 +28,5 @@ Background indexing requires a Swift 6 toolchain. You can download Swift 6 night
 ## Filing issues
 
 If you hit any issues that are not mentioned above, please [file a GitHub issue](https://github.com/apple/sourcekit-lsp/issues/new/choose) and attach the following information, if possible:
-- A diagnostic bundle generated by running `path/to/sourcekit-lsp diagnose --toolchain path/to/the/toolchain/you/are/using`
-- Your project including the `.index-build` folder.
+- A diagnostic bundle generated by running `path/to/sourcekit-lsp diagnose`.
+- Your project including the `.index-build` folder, if possible.

Package.swift

@@ -1,8 +1,14 @@
-// swift-tools-version:5.8
+// swift-tools-version: 5.8
 
 import Foundation
 import PackageDescription
 
+let strictConcurrencySettings: [SwiftSetting] = [
+  .enableUpcomingFeature("StrictConcurrency"),
+  .enableUpcomingFeature("RegionBasedIsolation"),
+  .enableUpcomingFeature("InferSendableFromCaptures"),
+]
+
 let package = Package(
   name: "SourceKitLSP",
   platforms: [.macOS(.v13)],
@@ -34,6 +40,7 @@ let package = Package(
         .product(name: "SwiftToolsSupport-auto", package: "swift-tools-support-core"),
       ],
       exclude: ["CMakeLists.txt"],
+      swiftSettings: strictConcurrencySettings,
       linkerSettings: sourcekitLSPLinkSettings
     ),
 
@@ -46,7 +53,7 @@ let package = Package(
         "LanguageServerProtocol"
       ],
       exclude: ["CMakeLists.txt"],
-      swiftSettings: [.enableExperimentalFeature("StrictConcurrency")]
+      swiftSettings: strictConcurrencySettings
     ),
 
     // MARK: CAtomics
@@ -87,7 +94,8 @@ let package = Package(
         .product(name: "SwiftParser", package: "swift-syntax"),
         .product(name: "SwiftToolsSupport-auto", package: "swift-tools-support-core"),
       ],
-      exclude: ["CMakeLists.txt"]
+      exclude: ["CMakeLists.txt"],
+      swiftSettings: strictConcurrencySettings
     ),
 
     .testTarget(
@@ -100,7 +108,8 @@ let package = Package(
         "SKCore",
         "SKTestSupport",
         .product(name: "SwiftToolsSupport-auto", package: "swift-tools-support-core"),
-      ]
+      ],
+      swiftSettings: strictConcurrencySettings
     ),
 
     // MARK: InProcessClient
@@ -113,7 +122,8 @@ let package = Package(
         "SKCore",
         "SourceKitLSP",
       ],
-      exclude: ["CMakeLists.txt"]
+      exclude: ["CMakeLists.txt"],
+      swiftSettings: strictConcurrencySettings
     ),
 
     // MARK: LanguageServerProtocol
@@ -122,15 +132,16 @@ let package = Package(
       name: "LanguageServerProtocol",
       dependencies: [],
       exclude: ["CMakeLists.txt"],
-      swiftSettings: [.enableExperimentalFeature("StrictConcurrency")]
+      swiftSettings: strictConcurrencySettings
     ),
 
     .testTarget(
       name: "LanguageServerProtocolTests",
       dependencies: [
         "LanguageServerProtocol",
         "LSPTestSupport",
-      ]
+      ],
+      swiftSettings: strictConcurrencySettings
     ),
 
     // MARK: LanguageServerProtocolJSONRPC
@@ -143,15 +154,16 @@ let package = Package(
         "LSPLogging",
       ],
       exclude: ["CMakeLists.txt"],
-      swiftSettings: [.enableExperimentalFeature("StrictConcurrency")]
+      swiftSettings: strictConcurrencySettings
     ),
 
     .testTarget(
       name: "LanguageServerProtocolJSONRPCTests",
       dependencies: [
         "LanguageServerProtocolJSONRPC",
         "LSPTestSupport",
-      ]
+      ],
+      swiftSettings: strictConcurrencySettings
     ),
 
     // MARK: LSPLogging
@@ -164,15 +176,16 @@ let package = Package(
         .product(name: "Crypto", package: "swift-crypto"),
       ],
       exclude: ["CMakeLists.txt"],
-      swiftSettings: lspLoggingSwiftSettings + [.enableExperimentalFeature("StrictConcurrency")]
+      swiftSettings: lspLoggingSwiftSettings + strictConcurrencySettings
     ),
 
     .testTarget(
       name: "LSPLoggingTests",
       dependencies: [
         "LSPLogging",
         "SKTestSupport",
-      ]
+      ],
+      swiftSettings: strictConcurrencySettings
     ),
 
     // MARK: LSPTestSupport
@@ -185,7 +198,8 @@ let package = Package(
         "LanguageServerProtocolJSONRPC",
         "SKSupport",
         "SwiftExtensions",
-      ]
+      ],
+      swiftSettings: strictConcurrencySettings
     ),
 
     // MARK: SemanticIndex
@@ -199,14 +213,16 @@ let package = Package(
         "SwiftExtensions",
         .product(name: "IndexStoreDB", package: "indexstore-db"),
       ],
-      exclude: ["CMakeLists.txt"]
+      exclude: ["CMakeLists.txt"],
+      swiftSettings: strictConcurrencySettings
     ),
 
     .testTarget(
       name: "SemanticIndexTests",
       dependencies: [
         "SemanticIndex"
-      ]
+      ],
+      swiftSettings: strictConcurrencySettings
     ),
 
     // MARK: SKCore
@@ -227,15 +243,16 @@ let package = Package(
         .product(name: "SwiftToolsSupport-auto", package: "swift-tools-support-core"),
       ],
       exclude: ["CMakeLists.txt"],
-      swiftSettings: [.enableExperimentalFeature("StrictConcurrency")]
+      swiftSettings: strictConcurrencySettings
     ),
 
     .testTarget(
       name: "SKCoreTests",
       dependencies: [
         "SKCore",
         "SKTestSupport",
-      ]
+      ],
+      swiftSettings: strictConcurrencySettings
     ),
 
     // MARK: SKSupport
@@ -252,7 +269,7 @@ let package = Package(
         .product(name: "SwiftToolsSupport-auto", package: "swift-tools-support-core"),
       ],
       exclude: ["CMakeLists.txt"],
-      swiftSettings: [.enableExperimentalFeature("StrictConcurrency")]
+      swiftSettings: strictConcurrencySettings
     ),
 
     .testTarget(
@@ -262,7 +279,8 @@ let package = Package(
         "SKSupport",
         "SKTestSupport",
         "SwiftExtensions",
-      ]
+      ],
+      swiftSettings: strictConcurrencySettings
     ),
 
     // MARK: SKSwiftPMWorkspace
@@ -279,7 +297,7 @@ let package = Package(
         .product(name: "SwiftToolsSupport-auto", package: "swift-tools-support-core"),
       ],
       exclude: ["CMakeLists.txt"],
-      swiftSettings: [.enableExperimentalFeature("StrictConcurrency")]
+      swiftSettings: strictConcurrencySettings
     ),
 
     .testTarget(
@@ -293,7 +311,8 @@ let package = Package(
         "SourceKitLSP",
         .product(name: "SwiftPM-auto", package: "swift-package-manager"),
         .product(name: "SwiftToolsSupport-auto", package: "swift-tools-support-core"),
-      ]
+      ],
+      swiftSettings: strictConcurrencySettings
     ),
 
     // MARK: SKTestSupport
@@ -312,9 +331,8 @@ let package = Package(
         .product(name: "ISDBTestSupport", package: "indexstore-db"),
         .product(name: "SwiftToolsSupport-auto", package: "swift-tools-support-core"),
       ],
-      resources: [
-        .copy("INPUTS")
-      ]
+      resources: [.copy("INPUTS")],
+      swiftSettings: strictConcurrencySettings
     ),
 
     // MARK: SourceKitD
@@ -330,7 +348,7 @@ let package = Package(
         .product(name: "SwiftToolsSupport-auto", package: "swift-tools-support-core"),
       ],
       exclude: ["CMakeLists.txt", "sourcekitd_uids.swift.gyb"],
-      swiftSettings: [.enableExperimentalFeature("StrictConcurrency")]
+      swiftSettings: strictConcurrencySettings
     ),
 
     .testTarget(
@@ -340,7 +358,8 @@ let package = Package(
         "SKCore",
         "SKTestSupport",
         "SwiftExtensions",
-      ]
+      ],
+      swiftSettings: strictConcurrencySettings
     ),
 
     // MARK: SourceKitLSP
@@ -369,7 +388,8 @@ let package = Package(
         .product(name: "SwiftToolsSupport-auto", package: "swift-tools-support-core"),
         .product(name: "SwiftPM-auto", package: "swift-package-manager"),
       ],
-      exclude: ["CMakeLists.txt"]
+      exclude: ["CMakeLists.txt"],
+      swiftSettings: strictConcurrencySettings
     ),
 
     .testTarget(
@@ -393,14 +413,16 @@ let package = Package(
         // be used by test cases that test macros (see `SwiftPMTestProject.macroPackageManifest`).
         .product(name: "SwiftCompilerPlugin", package: "swift-syntax"),
         .product(name: "SwiftSyntaxMacros", package: "swift-syntax"),
-      ]
+      ],
+      swiftSettings: strictConcurrencySettings
     ),
 
     // MARK: SwiftExtensions
 
     .target(
       name: "SwiftExtensions",
-      exclude: ["CMakeLists.txt"]
+      exclude: ["CMakeLists.txt"],
+      swiftSettings: strictConcurrencySettings
     ),
   ]
 )

Sources/Diagnose/CommandLineArgumentsReducer.swift

@@ -62,6 +62,7 @@ fileprivate class CommandLineArgumentReducer {
   ///
   /// If `simultaneousRemove` is set, the reducer will try to remove that many arguments at once. This is useful to
   /// quickly remove multiple arguments from the request.
+  @MainActor
   private func reduce(initialRequestInfo: RequestInfo, simultaneousRemove: Int) async throws -> RequestInfo {
     guard initialRequestInfo.compilerArgs.count > simultaneousRemove else {
       // Trying to remove more command line arguments than we have. This isn't going to work.

Sources/Diagnose/DiagnoseCommand.swift

@@ -226,6 +226,7 @@ public struct DiagnoseCommand: AsyncParsableCommand {
         "--predicate", #"subsystem = "org.swift.sourcekit-lsp" AND process = "sourcekit-lsp""#,
         "--info",
         "--debug",
+        "--signpost",
       ],
       outputRedirection: .stream(
         stdout: { bytes in
@@ -368,7 +369,9 @@ public struct DiagnoseCommand: AsyncParsableCommand {
       if let bundleOutputPath = self.bundleOutputPath {
         URL(fileURLWithPath: bundleOutputPath)
       } else {
-        FileManager.default.temporaryDirectory.appendingPathComponent("sourcekit-lsp-diagnose-\(date)")
+        FileManager.default.temporaryDirectory
+          .appendingPathComponent("sourcekit-lsp-diagnose")
+          .appendingPathComponent("sourcekit-lsp-diagnose-\(date)")
       }
     try FileManager.default.createDirectory(at: bundlePath, withIntermediateDirectories: true)
 

Sources/Diagnose/IndexCommand.swift

@@ -21,7 +21,6 @@ import SwiftExtensions
 
 import struct TSCBasic.AbsolutePath
 import class TSCBasic.Process
-import var TSCBasic.stderrStream
 import class TSCUtility.PercentProgressAnimation
 
 private actor IndexLogMessageHandler: MessageHandler {

Sources/Diagnose/ReduceCommand.swift

@@ -16,7 +16,6 @@ import SKCore
 
 import struct TSCBasic.AbsolutePath
 import class TSCBasic.Process
-import var TSCBasic.stderrStream
 import class TSCUtility.PercentProgressAnimation
 
 public struct ReduceCommand: AsyncParsableCommand {
@@ -77,7 +76,7 @@ public struct ReduceCommand: AsyncParsableCommand {
       throw ReductionError("Unable to find sourcekitd.framework")
     }
 
-    let progressBar = PercentProgressAnimation(stream: stderrStream, header: "Reducing sourcekitd issue")
+    let progressBar = PercentProgressAnimation(stream: stderrStreamConcurrencySafe, header: "Reducing sourcekitd issue")
 
     let request = try String(contentsOfFile: sourcekitdRequestPath)
     let requestInfo = try RequestInfo(request: request)

Sources/Diagnose/ReduceFrontendCommand.swift

@@ -16,7 +16,6 @@ import SKCore
 
 import struct TSCBasic.AbsolutePath
 import class TSCBasic.Process
-import var TSCBasic.stderrStream
 import class TSCUtility.PercentProgressAnimation
 
 public struct ReduceFrontendCommand: AsyncParsableCommand {
@@ -86,7 +85,7 @@ public struct ReduceFrontendCommand: AsyncParsableCommand {
     }
 
     let progressBar = PercentProgressAnimation(
-      stream: stderrStream,
+      stream: stderrStreamConcurrencySafe,
       header: "Reducing swift-frontend crash"
     )