Add documentation about each module's purpose and move some files between modules

The purpose of the different modules wasn’t clearly defined, which lead to inconsistent responsibilities between the different modules. Define each module’s purpose and move a few files between modules to satisfy these definitions.

There are a few more larger changes that will need to be made for a fully consistent module structure. These are FIXMEs in the new Modules.md document and I’ll address them in follow-up PRs.

Author: ahoppen

Documentation/Modules.md

@@ -0,0 +1,121 @@
+# Modules
+
+The SourceKit-LSP package contains the following non-testing modules.
+
+### BuildServerProtocol
+
+Swift types to represent the [Build Server Protocol (BSP) specification, version TODO](TODO). These types should also be usable when implementing a BSP client and thus this module should not have any dependencies other than the LanguageServerProtocol module, with which it shares some types.
+
+FIXME: Add link for BSP and version
+
+### CAtomics
+
+Implementation of atomics for Swift using C. Once we can raise our deployment target to use the `Atomic` type from the Swift standard library, this module should be removed.
+
+### CSKTestSupport
+
+FIXME: Can we remove this?
+
+### Csourcekitd
+
+Header file defining the interface to sourcekitd. This should stay in sync with [sourcekitd.h](TODO) in the Swift repository.
+
+### Diagnose
+
+A collection of subcommands to the `sourcekit-lsp` executable that help SourceKit-LSP diagnose issues.
+
+### InProcessClient
+
+A simple type that allows launching a SourceKit-LSP server in-process, communicating in terms of structs from the `LanguageServerProtocol` module.
+
+This should be the dedicated entry point for clients that want to run SourceKit-LSP in-process instead of launching a SourceKit-LSP server out-of-process and communicating with it using JSON RPC.
+
+### LanguageServerProtocol
+
+Swift types to represent the [Language Server Protocol (LSP) specification, version 3.17](https://microsoft.github.io/language-server-protocol/specifications/lsp/3.17/specification/). These types should also be usable when implementing an LSP client and thus this module should not have any dependencies.
+
+### LanguageServerProtocolJSONRPC
+
+A connection to or from a SourceKit-LSP server. Since message parsing can fail, it needs to handle errors in some way and the design decision here is to use LSPLogging, which hardcodes `org.swift.sourcekit-lsp` as the default logging subsystem and thus makes the module unsuitable for generic clients.
+
+### LSPLogging
+
+Types that are API-compatible with OSLog to allow logging to OSLog when building for Darwin platforms and logging to stderr or files on non-Darwin platforms. This should not be dependent on any LSP specific types and be portable to other packages.
+
+FIXME: Rename the module to SKLogging
+
+### LSPTestSupport
+
+FIXME: Merge this module with SKTestSupport
+
+### SemanticIndex
+
+Contains the interface with which SourceKit-LSP queries the semantic index, adding up-to-date checks on top of the indexstore-db API. Also implements the types that manage background indexing.
+
+### SKCore
+
+FIXME: Currently serves two independent purposes and should be split up into two modules
+
+#### BuildSystem
+
+Defines the queries SourceKit-LSP can ask of a a build system, like getting compiler arguments for a file. Finding a target’s dependencies or preparing a target.
+
+This includes:
+- BuildConfiguration.swift
+- BuildServerBuildSystem.swift
+- BuildSetup.swift
+- BuildSystem.swift
+- BuildSystemDelegate.swift
+- BuildSystemManager.swift
+- CompilationDatabase.swift
+- CompilationDatabaseBuildSystem.swift
+- FallbackBuildSystem.swift
+- FileBuildSettings.swift
+- IndexTaskID.swift
+- MainFilesProvider.swift
+- PathPrefixMapping.swift
+- SplitShellCommand.swift
+- WorkspaceType.swift
+
+#### ToolchainRegistry
+
+Discovers Swift toolchains on the system.
+
+- Toolchain.swift
+- ToolchainRegistry.swift
+- XCToolchainPlist.swift
+
+
+### SKSupport
+
+Contains SourceKit-LSP-specific helper functions. These fall into three different categories:
+-  Extensions on top of `swift-tools-support-core`
+- Functionality that can only be implemented by combining two lower-level modules that don't have a shared dependency, like `LSPLogging` + `LanguageServerProtocol`
+- Types that should be sharable by the different modules that implement SourceKit-LSP but that are not generic enough to fit into `SwiftExtensions`, like `ExperimentalFeatures`.
+
+### SKSwiftPMWorkspace
+
+Implements the `BuildSystem` protocol for Swift packages.
+
+FIXME: Merge this into the BuildSystem module once SKCore is split.
+
+### SKTestSupport
+
+A collection of utilities useful for writing tests for SourceKit-LSP and which are not specific to a single test module.
+
+### sourcekit-lsp
+
+This executable target that produces the `sourcekit-lsp` binary.
+
+### SourceKitD
+
+A Swift interface to talk to sourcekitd.
+
+### SourceKitLSP
+
+This is the core module that implements the SourceKit-LSP server.
+
+### SwiftExtensions
+
+Extensions to the Swift standard library and Foundation. Should not have any other dependencies. Any types in here should theoretically make senses to put in the Swift standard library or Foundation and they shouldn't be specific to SourceKit-LSP
+

Package.swift

@@ -220,7 +220,9 @@ let package = Package(
     .testTarget(
       name: "SemanticIndexTests",
       dependencies: [
-        "SemanticIndex"
+        "LSPLogging",
+        "SemanticIndex",
+        "SKTestSupport",
       ],
       swiftSettings: strictConcurrencySettings
     ),
@@ -343,7 +345,6 @@ let package = Package(
       dependencies: [
         "Csourcekitd",
         "LSPLogging",
-        "SKSupport",
         "SwiftExtensions",
         .product(name: "SwiftToolsSupport-auto", package: "swift-tools-support-core"),
       ],
@@ -399,6 +400,7 @@ let package = Package(
         "LSPLogging",
         "LSPTestSupport",
         "LanguageServerProtocol",
+        "SemanticIndex",
         "SKCore",
         "SKSupport",
         "SKTestSupport",

Sources/LanguageServerProtocolJSONRPC/JSONRPCConnection.swift

@@ -19,9 +19,12 @@ import LanguageServerProtocol
 import struct CDispatch.dispatch_fd_t
 #endif
 
-/// A connection between a message handler (e.g. language server) in the same process as the connection object and a remote message handler (e.g. language client) that may run in another process using JSON RPC messages sent over a pair of in/out file descriptors.
+/// A connection between a message handler (e.g. language server) in the same process as the connection object and a
+/// remote message handler (e.g. language client) that may run in another process using JSON RPC messages sent over a
+// pair of in/out file descriptors.
 ///
-/// For example, inside a language server, the `JSONRPCConnection` takes the language service implementation as its `receiveHandler` and itself provides the client connection for sending notifications and callbacks.
+/// For example, inside a language server, the `JSONRPCConnection` takes the language service implementation as its
+// `receiveHandler` and itself provides the client connection for sending notifications and callbacks.
 public final class JSONRPCConnection: Connection {
 
   /// A name of the endpoint for this connection, used for logging, e.g. `clangd`.

Sources/SKSupport/BuildConfiguration.swift renamed to Sources/SKCore/BuildConfiguration.swift

@@ -1,23 +1,22 @@
 
 add_library(SKCore STATIC
+  BuildConfiguration.swift
   BuildServerBuildSystem.swift
   BuildSetup.swift
   BuildSystem.swift
   BuildSystemDelegate.swift
   BuildSystemManager.swift
   CompilationDatabase.swift
   CompilationDatabaseBuildSystem.swift
-  Debouncer.swift
-  ExperimentalFeatures.swift
   FallbackBuildSystem.swift
   FileBuildSettings.swift
   IndexTaskID.swift
   MainFilesProvider.swift
   PathPrefixMapping.swift
   SplitShellCommand.swift
-  TaskScheduler.swift
   Toolchain.swift
   ToolchainRegistry.swift
+  WorkspaceType.swift
   XCToolchainPlist.swift)
 set_target_properties(SKCore PROPERTIES
   INTERFACE_INCLUDE_DIRECTORIES ${CMAKE_Swift_MODULE_DIRECTORY})

Sources/SKCore/CMakeLists.txt

@@ -1,19 +1,15 @@
 
 add_library(SKSupport STATIC
   Atomics.swift
-  BuildConfiguration.swift
   ByteString.swift
   Connection+Send.swift
-  dlopen.swift
+  Debouncer.swift
   DocumentURI+CustomLogStringConvertible.swift
+  ExperimentalFeatures.swift
   FileSystem.swift
   LineTable.swift
-  PipeAsStringHandler.swift
   Process+Run.swift
-  Random.swift
-  Result.swift
   SwitchableProcessResultExitStatus.swift
-  WorkspaceType.swift
 )
 set_target_properties(SKSupport PROPERTIES
   INTERFACE_INCLUDE_DIRECTORIES ${CMAKE_Swift_MODULE_DIRECTORY})

Sources/SKSupport/WorkspaceType.swift renamed to Sources/SKCore/WorkspaceType.swift

@@ -12,10 +12,6 @@
 
 import LSPLogging
 
-#if canImport(os)
-import os
-#endif
-
 public struct LineTable: Hashable, Sendable {
   @usableFromInline
   var impl: [String.Index]

Sources/SKSupport/CMakeLists.txt

@@ -6,6 +6,7 @@ add_library(SemanticIndex STATIC
   IndexTestHooks.swift
   PreparationTaskDescription.swift
   SemanticIndexManager.swift
+  TaskScheduler.swift
   UpdateIndexStoreTaskDescription.swift
   UpToDateTracker.swift
 )

Sources/SKCore/Debouncer.swift renamed to Sources/SKSupport/Debouncer.swift

@@ -1,5 +1,6 @@
 
 add_library(SourceKitD STATIC
+  dlopen.swift
   SKDRequestArray.swift
   SKDRequestDictionary.swift
   SKDResponse.swift

Sources/SKCore/ExperimentalFeatures.swift renamed to Sources/SKSupport/ExperimentalFeatures.swift

@@ -12,16 +12,10 @@
 
 import Foundation
 import LSPLogging
-import SKSupport
 import SwiftExtensions
 
 import struct TSCBasic.AbsolutePath
 
-#if compiler(<5.11)
-extension DLHandle: @unchecked Sendable {}
-#else
-extension DLHandle: @unchecked @retroactive Sendable {}
-#endif
 extension sourcekitd_api_keys: @unchecked Sendable {}
 extension sourcekitd_api_requests: @unchecked Sendable {}
 extension sourcekitd_api_values: @unchecked Sendable {}

Sources/SKSupport/LineTable.swift

@@ -13,7 +13,6 @@
 @_exported import Csourcekitd
 import Dispatch
 import Foundation
-import SKSupport
 import SwiftExtensions
 
 #if compiler(>=6)

Sources/SKSupport/Random.swift

@@ -21,7 +21,7 @@ import Glibc
 import Musl
 #endif
 
-public final class DLHandle {
+public final class DLHandle: Sendable {
   #if os(Windows)
   typealias Handle = HMODULE
   #else

Sources/SemanticIndex/CMakeLists.txt

@@ -11,7 +11,6 @@
 //===----------------------------------------------------------------------===//
 
 import Csourcekitd
-import SKSupport
 
 extension sourcekitd_api_functions_t {
   public init(_ sourcekitd: DLHandle) throws {

Sources/SKCore/TaskScheduler.swift renamed to Sources/SemanticIndex/TaskScheduler.swift

@@ -833,7 +833,7 @@ extension SourceKitLSPServer: BuildSystemDelegate {
 
 extension LanguageServerProtocol.BuildConfiguration {
   /// Convert `LanguageServerProtocol.BuildConfiguration` to `SKSupport.BuildConfiguration`.
-  var configuration: SKSupport.BuildConfiguration {
+  var configuration: SKCore.BuildConfiguration {
     switch self {
     case .debug: return .debug
     case .release: return .release
@@ -843,7 +843,7 @@ extension LanguageServerProtocol.BuildConfiguration {
 
 private extension LanguageServerProtocol.WorkspaceType {
   /// Convert `LanguageServerProtocol.WorkspaceType` to `SkSupport.WorkspaceType`.
-  var workspaceType: SKSupport.WorkspaceType {
+  var workspaceType: SKCore.WorkspaceType {
     switch self {
     case .buildServer: return .buildServer
     case .compilationDatabase: return .compilationDatabase

Sources/SourceKitD/CMakeLists.txt

@@ -5,6 +5,8 @@ add_library(SwiftExtensions STATIC
   Collection+Only.swift
   Collection+PartitionIntoBatches.swift
   NSLock+WithLock.swift
+  PipeAsStringHandler.swift
+  ResultExtensions.swift
   Sequence+AsyncMap.swift
   Task+WithPriorityChangedHandler.swift
   ThreadSafeBox.swift

Sources/SourceKitD/DynamicallyLoadedSourceKitD.swift

@@ -88,15 +88,15 @@ extension PathPrefixMapping: @retroactive ExpressibleByArgument {}
 #endif
 
 #if compiler(<5.11)
-extension SKSupport.BuildConfiguration: ExpressibleByArgument {}
+extension SKCore.BuildConfiguration: ExpressibleByArgument {}
 #else
-extension SKSupport.BuildConfiguration: @retroactive ExpressibleByArgument {}
+extension SKCore.BuildConfiguration: @retroactive ExpressibleByArgument {}
 #endif
 
 #if compiler(<5.11)
-extension SKSupport.WorkspaceType: ExpressibleByArgument {}
+extension SKCore.WorkspaceType: ExpressibleByArgument {}
 #else
-extension SKSupport.WorkspaceType: @retroactive ExpressibleByArgument {}
+extension SKCore.WorkspaceType: @retroactive ExpressibleByArgument {}
 #endif
 
 @main
@@ -179,7 +179,7 @@ struct SourceKitLSP: AsyncParsableCommand {
   @Option(
     help: "Override default workspace type selection; one of 'swiftPM', 'compilationDatabase', or 'buildServer'"
   )
-  var defaultWorkspaceType: SKSupport.WorkspaceType?
+  var defaultWorkspaceType: SKCore.WorkspaceType?
 
   @Option(
     name: .customLong("compilation-db-search-path"),

Sources/SourceKitD/SourceKitD.swift

@@ -10,7 +10,7 @@
 //
 //===----------------------------------------------------------------------===//
 
-import SKCore
+import SKSupport
 import XCTest
 
 final class DebouncerTests: XCTestCase {

Sources/SKSupport/dlopen.swift renamed to Sources/SourceKitD/dlopen.swift

@@ -15,7 +15,22 @@ import SKSupport
 import SKTestSupport
 import XCTest
 
-final class SupportPerfTests: PerfTestCase {
+/// A linear congruential generator with user-specified seed value. Useful for generating a predictable "random" number sequence.
+fileprivate struct SimpleLCG: RandomNumberGenerator {
+
+  var state: UInt64
+
+  public init(seed: UInt64) {
+    state = seed
+  }
+
+  public mutating func next() -> UInt64 {
+    state = state &* 6364136223846793005 &+ 1442695040888963407
+    return state
+  }
+}
+
+final class LineTablePerfTests: PerfTestCase {
 
   func testLineTableAppendPerf() {