[AutoDiff] Type-checking support for inout parameter differentiation.

[dan-zheng]

Semantically, an inout parameter is both a parameter and a result.

@differentiable and @derivative attributes now support original functions
with one "semantic result": either a formal result or an inout parameter.

The differential/pullback type of a function with inout differentiability
parameters also has inout parameters. This is ideal for performance.

Differential typing rules:

• Case 1: original function has no inout parameters.
  • Original: (T0, T1, ...) -> R
  • Differential: (T0.Tan, T1.Tan, ...) -> R.Tan
• Case 2: original function has a non-wrt inout parameter.
  • Original: (T0, inout T1, ...) -> Void
  • Differential: (T0.Tan, ...) -> T1.Tan
• Case 3: original function has a wrt inout parameter.
  • Original: (T0, inout T1, ...) -> Void
  • Differential: (T0.Tan, inout T1.Tan, ...) -> Void

Pullback typing rules:

• Case 1: original function has no inout parameters.
  • Original: (T0, T1, ...) -> R
  • Pullback: R.Tan -> (T0.Tan, T1.Tan, ...)
• Case 2: original function has a non-wrt inout parameter.
  • Original: (T0, inout T1, ...) -> Void
  • Pullback: (T1.Tan) -> (T0.Tan, ...)
• Case 3: original function has a wrt inout parameter.
  • Original: (T0, inout T1, ...) -> Void
  • Pullback: (inout T1.Tan) -> (T0.Tan, ...)

Resolves TF-1164.

---

Examples:

extension Float {
  @derivative(of: *=)
  static func _vjpMultiplyAssign(_ lhs: inout Float, _ rhs: Float) -> (
    value: Void, pullback: (inout Float) -> Float
  ) {
    defer { lhs *= rhs }
    return ((), { [lhs = lhs] v in
      let drhs = lhs * v
      v *= rhs
      return drhs
    })
  }
}

extension Array where Element: Differentiable {
  @derivative(of: append)
  mutating func _vjpAppend(_ element: Element) -> (
    value: Void, pullback: (inout TangentVector) -> Element.TangentVector
  ) {
    let appendedElementIndex = count
    defer { append(element) }
    return ((), { dself in dself.base[appendedElementIndex] })
  }
}

---

This patch includes all inout parameter differentiation changes currently
relevant to master branch, e.g. SIL derivative type calculation changes.

Remaining changes involves code that hasn't yet been upstreamed. Those changes
will be committed to tensorflow branch and eventually upstreamed.

End-to-end inout parameter differentiation is tested in #29956.

[dan-zheng]

Let's check that tests pass.
@swift-ci Please smoke test

[dan-zheng]

Note: I intentionally didn't add tests for @differentiable(jvp:vjp), since that's actively being removed (TF-1162).

[dan-zheng]

@swift-ci Please smoke test

[rxwei]

Case 3: original function has a wrt inout parameter.

• Original: (T0, inout T1, ...) -> R
• Differential: (T0.Tan, inout T1.Tan, ...) -> Void
  Case 3: original function has a wrt inout parameter.
• Original: (T0, inout T1, ...) -> R
• Pullback: (inout T1.Tan) -> (T0.Tan, ...)

I don't think these are correct. R should be a Void because those the wrt inout parameter already behaves as a result. There cannot be a second result.

[rxwei]

static func _vjpMultiplyAssign(_ lhs: inout Float, _ rhs: ${Self}) -> (

There are multiple occurrences of ${Self} in your code example in the PR description.

[dan-zheng]

@swift-ci Please smoke test