Add a documentation article for @_spi and when to use existentials

CONTRIBUTING.md

@@ -63,6 +63,22 @@ Tip: Running SwiftSyntax’s self-parse tests takes the majority of testing time
 2. Select the Arguments tab in the Run section
 3. Add a `SKIP_LONG_TESTS` environment variable with value `1`
 
+### Additional Verification
+
+swift-syntax has two additional verification methods (see the two sections below) that provide more extensive validation. They have significant runtime impact on swift-syntax and are thus not enabled by default when building swift-syntax, but are enabled in CI. If CI fails and you are unable to reproduce the failure locally, make sure that `SKIP_LONG_TESTS` is not set and try enabling these validations.
+
+#### RawSyntax Validation
+
+When the `SWIFTSYNTAX_ENABLE_RAWSYNTAX_VALIDATION` environment variable is set while building swift-syntax (or the check for that variable has been changed to always return `true` in Package.swift), SwiftSyntax will perform additional validation that the layout of the syntax tree is correct. It validates that every child of a syntax node has the correct kind (which should be guaranteed by the Swift type system in most cases) and, more importantly, validates that each token only has one of the token kinds that is specified in the syntax tree layout of the `CodeGeneration` package. 
+
+If this validation hits an assertion failure that a token is not accepted at a certain position in the syntax tree, double check if the token kind that is being stored in the syntax tree actually makes sense here. If it does not, check if there is a parser bug or whether you need to remap the token kind. If it does make sense, add the token kind to `.token(choices:)` of the syntax node in CodeGeneration, re-generate that source code and run tests again.
+
+#### Test Case Mutation
+
+When the `SWIFTPARSER_ENABLE_ALTERNATE_TOKEN_INTROSPECTION` environment variable is set while building swift-syntax (or the check for that variable has been changed to always return `true` in Package.swift), SwiftParser records alternative tokens that the parser was looking for at specific offsets in the source file (e.g. whether it also checked for a `struct` keyword when the source code contained a `class` keyword). It will then use that information to mutate the test case by e.g. substituting `class` by `struct`.
+
+When testing finds one of these failures, it will show you the syntax tree that produced the failure. Create a new test case with the source code the failure gives you and fix the failure.
+
 ### `lit`-based Tests
 
 A few tests are based LLVM’s `lit` and `FileCheck` tools.

CodeGeneration/Sources/generate-swiftsyntax/templates/swiftsyntax/SwiftSyntaxDoccIndex.swift

@@ -67,9 +67,40 @@ let nodesSections: String = {
   return result
 }()
 
+var contributingDocs: String = {
+  let contributingDocsFolder = URL(fileURLWithPath: #filePath)
+    .deletingLastPathComponent()
+    .deletingLastPathComponent()
+    .deletingLastPathComponent()
+    .deletingLastPathComponent()
+    .deletingLastPathComponent()
+    .deletingLastPathComponent()
+    .appendingPathComponent("Sources")
+    .appendingPathComponent("SwiftSyntax")
+    .appendingPathComponent("Documentation.docc")
+    .appendingPathComponent("Contributing")
+
+  let files = (try? FileManager.default.contentsOfDirectory(at: contributingDocsFolder, includingPropertiesForKeys: nil)) ?? []
+
+  return files.compactMap { file in
+    if file.pathExtension != "md" {
+      return nil
+    }
+    let doccName = file.lastPathComponent
+      .replacingOccurrences(of: ".md", with: "")
+      .replacingOccurrences(of: " ", with: "-")
+    return "- <doc:\(doccName)>"
+  }.sorted().joined(separator: "\n")
+}()
+
 let swiftSyntaxDoccIndex: String = {
-  let templateURL = URL(fileURLWithPath: #filePath).deletingLastPathComponent().appendingPathComponent("SwiftSyntaxDoccIndexTemplate.md")
+  let templateURL = URL(fileURLWithPath: #filePath)
+    .deletingLastPathComponent()
+    .appendingPathComponent("SwiftSyntaxDoccIndexTemplate.md")
   let template = try! String(contentsOf: templateURL)
 
-  return template.replacingOccurrences(of: "{{Nodes}}", with: nodesSections)
+  return
+    template
+    .replacingOccurrences(of: "{{Nodes}}", with: nodesSections)
+    .replacingOccurrences(of: "{{ContributingDocs}}", with: contributingDocs)
 }()

CodeGeneration/Sources/generate-swiftsyntax/templates/swiftsyntax/SwiftSyntaxDoccIndexTemplate.md

@@ -17,13 +17,17 @@ allows Swift tools to parse, inspect, generate, and transform Swift source code.
 ### Articles
 
 - <doc:Working-with-SwiftSyntax>
-- <doc:ChangingSwiftSyntax>
-- <doc:Create-SwiftSyntax-Release>
 
 ### Tutorials
 
 - <doc:Tutorial-Table-of-Contents>
 
+### Contributing
+
+These articles are intended for developers wishing to contribute to SwiftSyntax
+
+{{ContributingDocs}}
+
 ### Syntax
 
 - <doc:SwiftSyntax/Syntax>

Sources/SwiftSyntax/Documentation.docc/Contributing/Existentials.md

@@ -0,0 +1,10 @@
+# When to use protocols in SwiftSyntax
+
+Learn when to use protocols value types like ``ExprSyntax`` over protocols like ``ExprSyntaxProtocol``. 
+
+
+SwiftSyntax tries to minimize the use of existentials (aka. protocols spelled with `any` or protocols spelled without `some`) wherever possible. This is because when the stored value is more than 3 words (a word is the size of a pointer) large, these existentials store their data on the heap. The data stored inside `RawSyntax` is larger than 3 words and thus every time you pass a value around as a e.g. an `ExprSyntaxProtocol`, a new heap allocation will be made and that data needs to be reference-counted, which causes a very noticeable performance overhead. 
+
+swift-syntax offers two alternatives:
+- When passing a single node around, use `some ExprSyntaxProtocol`. This allows the concrete expression node (e.g. an ``IntegerLiteralExprSyntax``) to be passed directly without the need to wrap it in an existential and thus avoid the performance overhead.
+- When multiple expression nodes need to be represented that might be of different types, eg. in an array of expressions, use the ``ExprSyntax`` type. ``ExprSyntax`` is a struct and can thus be allocated on the stack. The downside is that specific expression nodes need to explicitly be upcast to `ExprSyntax` (`ExprSyntax(integerLiteral)`) and downcast to more specific types using a custom `as` method (`expr.as(IntegerLiteralExprSyntax.self)`).

Sources/SwiftSyntax/Documentation.docc/Contributing/SPI.md

@@ -0,0 +1,9 @@
+# @_spi attribute
+
+Learn when SwiftSyntax exposes declaration annotated as `@_spi`.
+
+Functions marked as `@_spi(RawSyntax)` (where ``RawSyntax`` can be any name) are considered *SPI* (System Programming Interface) and are only accessible if the module that declares them is imported as `@_spi(RawSyntax)`.
+
+Since functions marked as SPI are not part of the public API, swift-syntax makes no guarantee to their source stability. swift-syntax makes no effort to keep its SPI stable.
+
+Declarations are typically marked as SPI because they have some kind of caveat that makes them unsafe to use in general. For example, when accessing ``RawSyntax`` nodes, you need to manually guarantee that the ``SyntaxArena`` they’re allocated in will not be de-allocated. Other declarations have an `@_spi` to share them between different modules within the swift-syntax package. These would use the [`package` modifier](https://github.com/apple/swift-evolution/blob/main/proposals/0386-package-access-modifier.md) if not for the fact that swift-syntax needed to compile with the last two Swift releases (see <doc:Swift-Version>).

Sources/SwiftSyntax/Documentation.docc/Contributing/Swift Version.md

@@ -0,0 +1,7 @@
+# Language Features Usable in swift-syntax
+
+Defines which language features the swift-syntax package can use.
+
+We require that SwiftSyntax builds with the latest released compiler and the previous major version (e.g. with Swift 5.8 and Swift 5.7).
+
+*This documentation article needs to be expanded*

Sources/SwiftSyntax/Documentation.docc/generated/SwiftSyntax.md

@@ -17,13 +17,20 @@ allows Swift tools to parse, inspect, generate, and transform Swift source code.
 ### Articles
 
 - <doc:Working-with-SwiftSyntax>
-- <doc:ChangingSwiftSyntax>
-- <doc:Create-SwiftSyntax-Release>
 
 ### Tutorials
 
 - <doc:Tutorial-Table-of-Contents>
 
+### Contributing
+
+These articles are intended for developers wishing to contribute to SwiftSyntax
+
+- <doc:ChangingSwiftSyntax>
+- <doc:Existentials>
+- <doc:SPI>
+- <doc:Swift-Version>
+
 ### Syntax
 
 - <doc:SwiftSyntax/Syntax>