[AutoDiff upstream] Upstream @derivative attribute type-checking.

[dan-zheng]

The @derivative attribute registers a function as a derivative of another
function-like declaration: a func, init, subscript, or var computed
property declaration.

The @derivative attribute also has an optional wrt: clause specifying the
parameters that are differentiated "with respect to", i.e. the differentiation
parameters. The differentiation parameters must conform to the Differentiable
protocol.

If the wrt: clause is unspecified, the differentiation parameters are inferred
to be all parameters that conform to Differentiable.

@derivative attribute type-checking verifies that the type of the derivative
function declaration is consistent with the type of the referenced original
declaration and the differentiation parameters.

Resolves TF-829.

---

Examples:

func sin(_ x: Float) -> Float { ... }

@derivative(of: sin)
func derivativeSin(_ x: Float) -> (value: Float, differential: (Float) -> Float) {
    (sin(x), { dx in dx * cos(x) })
}

public protocol ElementaryFunctions {
    static func log(_ x: Self) -> Self
    ...
}

public extension ElementaryFunctions
where Self: Differentiable & FloatingPoint, Self == Self.TangentVector {
    @inlinable
    @derivative(of: log)
    static func derivativeLog(_ x: Self) -> (value: Self, differential: (Self) -> Self) {
        (log(x), { dx in (1 / x) * dx })
    }
}

The differentiable programming manifesto has more information and examples.

---

Note: this code is upstreamed from tensorflow branch. Some functions are intentionally not static (fileprivate) because they have users in other files on tensorflow branch.

Known issues are referenced in comments:

• TF-874: Simplify type calculation helpers.
• TF-1042: Remove DerivativeAttrs cache from ASTContext.

We would like to upstream code and continue development on master, if that's acceptable!

cc @rxwei @marcrasi @bgogul

[dan-zheng]

@swift-ci Please test

[swift-ci]

Build failed
Swift Test Linux Platform
Git Sha - 9e09105

[dan-zheng]

@swift-ci Please test

[swift-ci]

Build failed
Swift Test OS X Platform
Git Sha - 9e09105

[swift-ci]

Build failed
Swift Test Linux Platform
Git Sha - 9e09105

[lattner]

Nice, I love the testcase, great job.

[dan-zheng]

@swift-ci Please test

[swift-ci]

Build failed
Swift Test Linux Platform
Git Sha - 2a63ab7

[swift-ci]

Build failed
Swift Test OS X Platform
Git Sha - 2a63ab7

[dan-zheng]

@swift-ci Please test macOS

[dan-zheng]

Merging to unblock progress on upstreaming AutoDiff. Happy to address additional feedback later!

[compnerd]

This broke the VS2019 build:
https://compnerd.visualstudio.com/windows-swift/_build/results?buildId=18893&view=logs&j=b9e62f99-1a98-5ed7-01d2-f4794231ed79&t=9d1c7fdf-e7ea-5e83-bbfd-5e0708569c31&l=3865