[AutoDiff] [stdlib] Add 'EuclideanDifferentiable' protocol. (#26827)

Some iterative mathematical optimization algorithms (e.g. gradient descent with weight decay), when applied on a model (of type `Model` that conforms to `Differentiable`), rely on computing tangent vectors (of type `Model.TangentVector`) based on a gradient and an existing parameter (a stored property of `Model`). However, models that are differentiable via Euclidean differentiation are not necessarily vector spaces such that the `Model` equals `Model.TangentVector`, because `Model` may contain properties that do not appear in `Model.TangentVector`, such as Boolean flags or other hyperparameters. This use case is extremely common in machine learning, where most differentiation happens in the Euclidean space.

To make this use case possible, we introduce `EuclideanDifferentiable`, which generalizes types that are a product manifold of a differentiable vector space and some arbitrary manifold where the product manifold's tangent space is equal to the differentiable vector space component. In other words, `T: EuclideanDifferentiable` means that `T == V × M` where `V: Differentiable & AdditiveArithmetic` and `T.TangentVector == V == V.TangentVector`. The protocol has a `vectorView` property, which returns the `V` part (the differentiable vector space part) of the value.

The `EuclideanDifferentiable` allows us to **generically** express the kind of special iterative optimization algorithms that use a models parameters when computing a tangent vector.

```swift
let 𝛁L: Model.TangentVector = ...
model.move(along: -η * 𝛁L - η * λ * model.vectorView))
```

TODO: We need to implement derived conformances for the `vectorView` property.

Thanks to @sgugger, @marcrasi, @mattjj, @dougalm, and @caseychu for their input on solving this use case.

Partially resolves [tensorflow/swift-apis#456](tensorflow/swift-apis#456).

Author: rxwei

stdlib/public/core/AutoDiff.swift

@@ -183,6 +183,41 @@ public extension Differentiable where TangentVector == Self {
   }
 }
 
+/// A type that consists of a differentiable vector space and some other
+/// non-differentiable component.
+///
+/// Mathematically, this represents a product manifold that consists of
+/// a differentiable vector space and some arbitrary manifold, where the tangent
+/// bundle of the entire product manifold is equal to the vector space
+/// component.
+///
+/// This abstraction is useful for representing common differentiable data
+/// structures that contain both differentiable vector properties and other
+/// stored properties that do not have a derivative, e.g.
+///
+/// ```swift
+/// struct Perceptron: @memberwise EuclideanDifferentiable {
+///     var weight: SIMD16<Float>
+///     var bias: Float
+///     @noDerivative var useBias: Bool
+/// }
+/// ```
+///
+/// - Note: Conform a type to `EuclideanDifferentiable` if it is differentiable
+///   only with respect to its vector space component and when its
+///   `TangentVector` is equal to its vector space component.
+public protocol EuclideanDifferentiable: Differentiable {
+  /// The differentiable vector component of `self`.
+  var vectorView: TangentVector { get set }
+}
+
+public extension EuclideanDifferentiable where TangentVector == Self {
+  var vectorView: TangentVector {
+    _read { yield self }
+    _modify { yield &self }
+  }
+}
+
 /// Returns `x` like an identity function. When used in a context where `x` is
 /// being differentiated with respect to, this function will not produce any 
 /// derivative at `x`.