Merge branch 'master' of github.com:apple/swift into tensorflow-stage

* 'master' of github.com:apple/swift:
  [docs] Fix broken links in the documentation index (#33829)
  [Profiler] Increment closure body count prior to the prolog (#33946)
  ARCSequenceOpts: Add LoopSummary verifier (#33810)
  [build-script] Cleanup source directory layout help
  Address review comment.
  ABI checker: when invoking via build system, explicitly mention ABI breakge in diagnostics
  stdlib: Remove unused, unsafe helper function _withUninitializedString (#33704)
  test: Replace _silgen_name w/ _cdecl in CommandLineStressTest
  [AutoDiff] [Docs] Clarify 'Differentiable' derived conformances conditions.
  Revert "build-script: remove dead CMake options for Swift"

Author: Dave Abrahams

docs/DifferentiableProgramming.md

@@ -1219,53 +1219,73 @@ network layers and models are formed from smaller components stored as
 properties in structure types and class types. In order to use these types for
 differentiation, one must extend these types to conform to the `Differentiable`
 protocol. Luckily, this need not be done manually in most cases—the compiler
-automatically synthesizes conformances when a memberwise `Differentiable`
-conformance is declared.
+automatically synthesizes conformances when a `Differentiable` conformance is
+declared.
 
 ##### Synthesis conditions
 
 The compiler automatically synthesizes implementations of `Differentiable`
-protocol requirements for struct and class types. Here are the conditions for
-synthesis: The type must declare a conformance to `Differentiable` with a
-`@memberwise` attribute before the protocol name, either on the type declaration
-or on an extension in the same file. All stored properties of the conforming
-type must either be a `var` that conforms to `Differentiable` or be marked with
-the `@noDerivative` attribute. If a non-`Differentiable` or a `let` stored
-property is not marked with `@noDerivative`, then it is treated as if it has
-`@noDerivative` and the compiler emits a warning (with a fix-it in IDEs) asking
-the user to make the attribute explicit.
+protocol requirements for struct and class types. For a type, conditions for the
+synthesis are:
+
+1. There is a conformance to `Differentiable` declared for the type, either in
+   the original type declaration or in an extension.
+
+2. There is a `@memberwise` attribute in the conformance clause before the
+   protocol name.
+
+3. The conformance must be declared in the same file.
+
+Here is an example where the synthesis conditions are satisfied.
+
+```swift
+struct Model: @memberwise Differentiable {
+    var weight: SIMD4<Double>
+    var bias: Double
+    let metadata1: Float
+    let metadata2: Float
+    let usesBias: Bool
+}
+```
 
 ##### Default synthesis
 
-By default, the compiler synthesizes a nested `TangentVector` structure type
-that contains the `TangentVector`s of all stored properties that are not marked
-with `@noDerivative`. In other words, `@noDerivative` makes a stored property
-not be included in a type's tangent vectors.
+The compiler synthesizes a nested `TangentVector` structure type that contains
+the `TangentVector`s of all stored properties (terms and conditions apply) that
+conform to `Differentiable`, which we call **differentiable variables**.
+
+Mathematically, the synthesized implementation treats the data structure as a
+product manifold of the manifolds each differentiable variable's type
+represents. Differentiable variables' types are required to conform to
+`Differentiable` because the synthesized implementation needs to access each
+differentiable variable's type's `TangentVector` associated type and invoke each
+differentiable variable's implementation of `move(along:)` and
+`zeroTangentVectorInitializer`. Because the synthesized implementation needs to
+invoke `move(along:)` on each differentiable variable, the differentiable
+variables must have a `move(along:)` which satisfies the protocol requirement
+and can be invoked on the property. That is, the property must be either a
+variable (`var`) or a constant (`let`) with a non-`mutating` implementation of
+the `move(along:)` protocol requirement.
 
 The synthesized `TangentVector` has the same effective access level as the
 original type declaration. Properties in the synthesized `TangentVector` have
 the same effective access level as their corresponding original properties.
 
-A `move(along:)` method is synthesized with a body that calls `move(along:)` for
-each pair of the original property and its corresponding property in
-`TangentVector`.
+The synthesized `move(along:)` method calls `move(along:)` for each pair of a
+differentiable variable and its corresponding property in `TangentVector`.
 
-Similarly, when memberwise derivation is possible,
-`zeroTangentVectorInitializer` is synthesized to return a closure that captures
-and calls each stored property's `zeroTangentVectorInitializer` closure.
-When memberwise derivation is not possible (e.g. for custom user-defined
-`TangentVector` types), `zeroTangentVectorInitializer` is synthesized as a
-`{ TangentVector.zero }` closure.
-
-Here's an example:
+The synthesized `zeroTangentVectorInitializer` property returns a closure that
+captures and calls each stored property's `zeroTangentVectorInitializer`
+closure. When memberwise derivation is not possible (e.g. for custom
+user-defined `TangentVector` types), `zeroTangentVectorInitializer` is
+synthesized as a `{ TangentVector.zero }` closure.
 
 ```swift
 struct Foo<T: Differentiable, U: Differentiable>: @memberwise Differentiable {
-    // `x` and `y` are the "differentiation properties".
+    // `x` and `y` are the "differentiable variables".
     var x: T
     var y: U
-    @noDerivative var customFlag: Bool
-    @noDerivative let helperVariable: T
+    let customFlag: Bool
 
     // The compiler synthesizes:
     //
@@ -1279,7 +1299,6 @@ struct Foo<T: Differentiable, U: Differentiable>: @memberwise Differentiable {
     //         y.move(along: direction.y)
     //     }
     //
-    //     @noDerivative
     //     var zeroTangentVectorInitializer: () -> TangentVector {
     //         { [xTanInit = x.zeroTangentVectorInitializer,
     //            yTanInit = y.zeroTangentVectorInitializer] in
@@ -1289,6 +1308,74 @@ struct Foo<T: Differentiable, U: Differentiable>: @memberwise Differentiable {
 }
 ```
 
+###### Opt out of synthesis for a stored property
+
+The synthesized implementation of `Differentiable` protocol requirements already
+excludes stored properties that are not differentiable variables, such as stored
+properties that do not conform to `Differentiable` and `let`
+properties that do not have a non-mutating `move(along:)`. In addition to this
+behavior, we also introduce a `@noDerivative` declaration attribute, which can
+be attached to properties that the programmer does not wish to include in the
+synthesized `Differentiable` protocol requirement implementation.
+
+When a stored property is marked with `@noDerivative` in a type that declares a
+conformance to `Differentiable`, it will not be treated as a differentiable
+variable regardless of whether it conforms to `Differentiable`. That is, the
+synthesized implementation of protocol requirements will not include this
+property.
+
+```swift
+struct Foo<T: Differentiable, U: Differentiable>: @memberwise Differentiable {
+    // `x` and `y` are the "differentiable variables".
+    var x: T
+    var y: U
+    @noDerivative var customFlag: Bool
+    @noDerivative let helperVariable: T
+}
+```
+
+For clarity as to which stored properties are to be included for
+differentiation, the compiler will recommend that all stored properties that
+cannot be included as differentiable variables (due to either lacking a
+conformance to `Differentiable` or being a non-`class`-bound `let` property) be
+marked with `@noDerivative`. When a property is not included as a differentiable
+variable and is not marked with `@noDerivative`, the compiler produces a warning
+asking the user to make the exclusion explicit along with fix-it suggestions in
+IDEs.
+
+```swift
+struct Foo<T: Differentiable, U: Differentiable>: @memberwise Differentiable {
+    // `x` and `y` are the "differentiable variables".
+    var x: T
+    var y: U
+    var customFlag: Bool
+    let helperVariable: T
+}
+```
+
+```console
+test.swift:5:4: warning: stored property 'customFlag' has no derivative because 'Bool' does not conform to 'Differentiable'
+    var customFlag: Bool
+
+test.swift:5:4: note: add a '@noDerivative' attribute to make it explicit
+    var customFlag: Bool
+    ^
+    @noDerivative
+
+test.swift:6:4: warning: synthesis of the 'Differentiable.move(along:)' requirement for 'Foo' requires all stored properties not marked with `@noDerivative` to be mutable
+    let helperVariable: T
+
+test.swift:6:4: note: change 'let' to 'var' to make it mutable
+    let helperVariable: T
+    ^~~
+    var
+
+test.swift:6:4: note: add a '@noDerivative' attribute to make it explicit
+    let helperVariable: T
+    ^
+    @noDerivative
+```
+
 ##### Shortcut synthesis
 
 In certain cases, it is not ideal to keep `Self` and `TangentVector` as separate

docs/README.md

@@ -77,7 +77,7 @@ documentation, please create a thread on the Swift forums under the
     How to build Swift on Windows using Visual Studio.
   - [WindowsCrossCompile.md](/docs/WindowsCrossCompile.md):
     How to cross compile Swift for Windows on a non-Windows host OS.
-- [RunningIncludeWhatYouUse.md](/docs/RunningIncludeWhatYouUse.md):
+- [RunningIncludeWhatYouUse.md](/docs/HowToGuides/RunningIncludeWhatYouUse.md):
   Describes how to run [include-what-you-use](https://include-what-you-use.org)
   on the Swift project.
 
@@ -86,6 +86,8 @@ documentation, please create a thread on the Swift forums under the
 - [ByteTree.md](/docs/ByteTree.md):
   Describes the ByteTree binary format used for serializing syntax trees
   in `libSyntax`.
+- [WebAssembly.md](/docs/WebAssembly.md):
+  Explains some decisions that were made while implementing the WebAssembly target.
 
 ### Compiler and Runtime Subsystems
 
@@ -138,10 +140,11 @@ documentation, please create a thread on the Swift forums under the
 ### Language subsystems
 
 - Swift's Object Model
-  - [LogicalObjects.md]():
+  - [LogicalObjects.md](/docs/LogicalObjects.md):
     Describes the differences between logical and physical objects and
     introduces materialization and writeback.
-  - [MutationModel.rst]() <!--: NOTE: Outdated -->
+  - [MutationModel.rst](/docs/MutationModel.rst): Outdated.
+    <!-- NOTE: Outdated -->
 - [DocumentationComments.md](/docs/DocumentationComments.md):
   Describes the format of Swift's documentation markup, including
   specially-recognized sections.
@@ -174,6 +177,29 @@ documentation, please create a thread on the Swift forums under the
   <!-- NOTE: Outdated -->
 - [Lexicon.md](/docs/Lexicon.md):
   Canonical reference for terminology used throughout the project.
+  
+### ABI
+
+- [GenericSignature.md](/docs/ABI/GenericSignature.md):
+  Describes what generic signatures are and how they are used in the ABI,
+  including the algorithms for minimization and canonicalization.
+- [KeyPaths.md](/docs/ABI/KeyPaths.md):
+  Describes the layout of key path objects (instantiated by the runtime,
+  and therefore not strictly ABI). \
+  **TODO:** The layout of key path patterns (emitted by the compiler,
+  to represent key path literals) isn't documented yet.
+- [Mangling.rst](/docs/ABI/Mangling.rst):
+  Describes the stable mangling scheme, which produces unique symbols for
+  ABI-public declarations.
+- [RegisterUsage.md](/docs/ABI/RegisterUsage.md):
+  Summarizes the register allocation for ARM64 and x86_64 calling conventions,
+  including the context register (self) and error return register.
+- [TypeLayout.rst](/docs/ABI/TypeLayout.rst):
+  Describes the algorithms/strategies for fragile struct and tuple layout;
+  class layout; fragile enum layout; and existential container layout.
+- [TypeMetadata.rst](/docs/ABI/TypeMetadata.rst):
+  Describes the fields, values, and layout of metadata records, which can be
+  used (by reflection and debugger tools) to discover information about types.
 
 ## Recommended Practices
 
@@ -188,7 +214,7 @@ documentation, please create a thread on the Swift forums under the
 - [StdlibAPIGuidelines.rst](/docs/StdlibAPIGuidelines.rst):
   Provides guidelines for designing stdlib APIs.
   <!-- NOTE: Outdated -->
-- [StandardLibraryProgrammersManual](/docs/StandardLibraryProgrammersManual.md):
+- [StandardLibraryProgrammersManual.md](/docs/StandardLibraryProgrammersManual.md):
   Provides guidelines for working code in the stdlib.
 - [OptimizationTips.rst](/docs/OptimizationTips.rst):
   Provides guidelines for writing high-performance Swift code.
@@ -199,7 +225,7 @@ documentation, please create a thread on the Swift forums under the
 
 - [Branches.md](/docs/Branches.md):
   Describes how different branches are setup and what the automerger does.
-- [ContinuousIntegration.md](ContinuousIntegration.md):
+- [ContinuousIntegration.md](/docs/ContinuousIntegration.md):
   Describes the continuous integration setup, including the `@swift_ci` bot.
 
 ## Evolution Documents
@@ -211,9 +237,9 @@ documentation, please create a thread on the Swift forums under the
     Describes the goals and design for ABI stability.
   - [LibraryEvolutionManifesto.md](/docs/LibraryEvolutionManifesto.md):
     Describes the goals and design for Library Evolution.
-- [BuildManifesto.md](BuildManifesto.md):
+- [BuildManifesto.md](/docs/BuildManifesto.md):
   Provides an outline for modularizing the build system for the Swift toolchain.
-- [CppInteroperabilityManifesto.md](CppInteroperabilityManifesto.md):
+- [CppInteroperabilityManifesto.md](/docs/CppInteroperabilityManifesto.md):
   Describes the motivation and design for first-class Swift-C++ interoperability.
 - [DifferentiableProgramming.md](/docs/DifferentiableProgramming.md):
   Outlines a vision and design for first-class differentiable programming in Swift.
@@ -232,10 +258,13 @@ More recent proposals are located in the [apple/swift-evolution][] repository.
 You can see the status of different proposals at
 <https://apple.github.io/swift-evolution/>.
 
-[swift-evolution]: https://github.com/apple/swift-evolution
+[apple/swift-evolution]: https://github.com/apple/swift-evolution
 
 ### Surveys
 
+- [CallingConvention.rst](/docs/ABI/CallingConvention.rst):
+  This whitepaper discusses the Swift calling convention (high-level semantics;
+  ownership transfer; physical representation; function signature lowering).
 - [ErrorHandlingRationale.rst](/docs/ErrorHandlingRationale.rst):
   Surveys error-handling in a variety of languages, and describes the rationale
   behind the design of error handling in Swift.
@@ -247,10 +276,10 @@ You can see the status of different proposals at
 ### Archive
 
 These documents are known to be out-of-date and are superseded by other
-documentation, primarily [The Swift Programming Language (TSPL)][].
+documentation, primarily [The Swift Programming Language][] (TSPL).
 They are preserved mostly for historical interest.
 
-- [AccessControl.rst](/docs/AccessControl.swift)
+- [AccessControl.rst](/docs/AccessControl.rst)
 - [Arrays.rst](/docs/Arrays.rst)
   <!-- Has additional notes on bridging that may be of general interest? -->
 - [Generics.rst](/docs/Generics.rst)
@@ -262,7 +291,7 @@ They are preserved mostly for historical interest.
 
 ## External Resources
 
-External resources are listed in [docs/ExternalResources.md](ExternalResources.md).
+External resources are listed in [ExternalResources.md](/docs/ExternalResources.md).
 These cover a variety of topics,
 such as the design of different aspects of the Swift compiler and runtime
 and contributing to the project more effectively.
@@ -323,9 +352,9 @@ provide greater clarity to contributors wanting to add new documentation.
   `@_hasMissingDesignatedInitializers`. Some of this is covered in
   [TSPL's initialization section][] but that doesn't include newly added
   attributes.
-- [Modules.rst](/docs/Module.rst): for Swift pre-1.0.
+- [Modules.rst](/docs/Modules.rst): for Swift pre-1.0.
 - [Swift3Compatibility.md](/docs/Swift3Compatibility.md):
   Discusses the Swift 3 -> Swift 4 migration.
-- [StoredAndComputedVariables.rst](): for Swift pre-1.0.
+- [StoredAndComputedVariables.rst](/docs/StoredAndComputedVariables.rst): for Swift pre-1.0.
 
 [TSPL's initialization section]: https://docs.swift.org/swift-book/LanguageGuide/Initialization.html

lib/SILGen/SILGenFunction.cpp

@@ -524,10 +524,10 @@ void SILGenFunction::emitClosure(AbstractClosureExpr *ace) {
   auto resultIfaceTy = ace->getResultType()->mapTypeOutOfContext();
   auto captureInfo = SGM.M.Types.getLoweredLocalCaptures(
     SILDeclRef(ace));
+  emitProfilerIncrement(ace);
   emitProlog(captureInfo, ace->getParameters(), /*selfParam=*/nullptr,
              ace, resultIfaceTy, ace->isBodyThrowing(), ace->getLoc());
   prepareEpilog(true, ace->isBodyThrowing(), CleanupLocation(ace));
-  emitProfilerIncrement(ace);
   if (auto *ce = dyn_cast<ClosureExpr>(ace)) {
     emitStmt(ce->getBody());
   } else {

lib/SILOptimizer/ARC/ARCRegionState.cpp

@@ -18,10 +18,15 @@
 #include "swift/SILOptimizer/Analysis/LoopRegionAnalysis.h"
 #include "swift/SILOptimizer/Analysis/AliasAnalysis.h"
 #include "swift/SILOptimizer/Analysis/RCIdentityAnalysis.h"
+#include "llvm/Support/CommandLine.h"
 #include "llvm/Support/Debug.h"
 
 using namespace swift;
 
+llvm::cl::opt<bool> verifyARCLoopSummary(
+    "verify-arc-loop-summary", llvm::cl::init(false),
+    llvm::cl::desc("Verify if loop summary is correct in ARCLoopsOpts"));
+
 //===----------------------------------------------------------------------===//
 //                               ARCRegionState
 //===----------------------------------------------------------------------===//
@@ -282,6 +287,18 @@ bool ARCRegionState::processLoopBottomUp(
       OtherState->second.checkAndResetKnownSafety(
           I, OtherState->first, checkIfRefCountInstIsMatched, RCIA, AA);
     }
+#ifndef NDEBUG
+    // Verify updateForDifferentLoopInst is conservative enough that the flow
+    // sensitive native of the loop summarized instructions does not matter.
+    if (verifyARCLoopSummary) {
+      auto NewRefCountState = OtherState->second;
+      for (auto *I : State->getSummarizedInterestingInsts()) {
+        NewRefCountState.updateForDifferentLoopInst(I, AA);
+      }
+      assert(NewRefCountState.getLatticeState() ==
+             OtherState->second.getLatticeState());
+    }
+#endif
   }
 
   return false;
@@ -420,6 +437,18 @@ bool ARCRegionState::processLoopTopDown(
       OtherState->second.checkAndResetKnownSafety(
           I, OtherState->first, checkIfRefCountInstIsMatched, RCIA, AA);
     }
+#ifndef NDEBUG
+    // Verify updateForDifferentLoopInst is conservative enough that the flow
+    // sensitive native of the loop summarized instructions does not matter.
+    if (verifyARCLoopSummary) {
+      auto NewRefCountState = OtherState->second;
+      for (auto *I : State->getSummarizedInterestingInsts()) {
+        NewRefCountState.updateForDifferentLoopInst(I, AA);
+      }
+      assert(NewRefCountState.getLatticeState() ==
+             OtherState->second.getLatticeState());
+    }
+#endif
   }
 
   return false;