[API] [AD] Revamp @differentiable usages in stdlib.

[dan-zheng]

• Use FloatingPoint rather than BinaryFloatingPoint to constrain
  differentiability.
  • Follows from:
    • [stdlib] [AutoDiff] Constrain gradient operators on FloatingPoint, not BinaryFloatingPoint. #21673
    • Use FloatingPoint constraint insteaad of BinaryFloatingPoint. tensorflow/swift-bindings#11
• Use @differentiable where clauses to constrain differentiability
  of numeric operations.
  • The most common constraint is where Scalar : FloatingPoint because
    Tensor conditionally conforms to Differentiable where Scalar : FloatingPoint.
• Tensor now conditionally conforms to Differentiable where Scalar : Differentiable & FloatingPoint.
• Allow @differentiable where clause conformance requirements to protocol composition types.
• Make VJP applications use the correct substitution map.
  • If a custom @differentiable attribute defines a VJP and where clause
    requirements, VJP applications should use a substitution map involving
    those requirements.
  • Note: more related cases need to be handled, such as @differentiable
    attributes with where clause requirements but no VJP. These cases will
    be handled later.

Todos:

• Make more Tensor operations differentiable.
  • This includes reduction and broadcasting ops.
  • This is enabled by @differentiable where clause type-checking.
• Use VJP functions instead of adjoint functions.
  • I've started work on this.
  • I would prefer that this be done in a separate patch, after this patch
    adds the correct @differentiable where clauses.
• Add tests for newly @differentiable Tensor operations.

[dan-zheng]

WIP, triggering tests to see what breaks.
@swift-ci Please test tensorflow

[rxwei]

I think we should change Tensor's Differentiable conformance condition first. Currently it is

extension Tensor : Differentiable where Scalar : FloatingPoint

This kinda works, but what if, in a context where Scalar : FloatingPoint, you want to differentiate through some scalar operation on an element of Tensor<Scalar>? In that case, you'll want the scalar to be differentiable as well. Therefore, we should change the conformance condition to:

extension Tensor : Differentiable where Scalar : Differentiable & FloatingPoint

[dan-zheng]

Therefore, we should change the conformance condition to:

extension Tensor : Differentiable where Scalar : Differentiable & FloatingPoint

This makes sense. I'll adopt this change in this PR.

[dan-zheng]

@swift-ci Please test tensorflow

[dan-zheng]

Merging to unblock progress.

[rxwei]

Would you mind changing the PR description to mention the substitution map change?

[dan-zheng]

Would you mind changing the PR description to mention the substitution map change?

Done! Also added info about the last commit (Tensor : Differentiable conditional conformance change).

[rxwei]

Note: more related cases need to be handled, such as @differentiable attributes with where clause requirements but no VJP. These cases will be handled later.

Yeah, this will be handled (and required) in differentiation through generics.