Update min(_:_:) and max(_:_:) gradients to match Python TensorFlow

[jon-tow]

Fixes #479

[jon-tow]

@dan-zheng @rxwei This should be ready.

[dan-zheng]

Thank you for the prompt fix! 🙂

[dan-zheng]

There appears to be one failure:

Test Case 'LossTests.testSigmoidCrossEntropyGrad' started at 2019-08-24 07:41:17.269
/swift-apis/Tests/TensorFlowTests/LossTests.swift:219: error: LossTests.testSigmoidCrossEntropyGrad : XCTAssertEqual failed: ("0.125") is not equal to ("0.0625") +/- ("1e-06") -

It's likely because sigmoidCrossEntropy internally calls max:

@differentiable(wrt: logits)
public func sigmoidCrossEntropy<Scalar: TensorFlowFloatingPoint>(
    logits: Tensor<Scalar>,
    labels: Tensor<Scalar>,
    reduction: @differentiable (Tensor<Scalar>) -> Tensor<Scalar> = _mean
) -> Tensor<Scalar> {
    let maxLogitsWithZero = max(logits, Tensor(0)) // `max` called here
    let result = log(1 + exp(-abs(logits)))
    return reduction(maxLogitsWithZero - logits * labels + result)
}

Could you please fix? Updating the expected value in the test should be good - validating the expected value against tf.losses.sigmoid_cross_entropy would be great (I'm not sure how expectedGradientsBeforeMean was computed in the test).

[jon-tow]

tf.losses.sigmoid_cross_entropy is computed via tf.python.ops.nn_impl.sigmoid_cross_entropy_with_logits. which is a bit different than the swift-api version in that it uses custom max and abs in order to compute gradients at 0. I'm still trying to find my way around this but will update as soon as possible.

[dan-zheng]

Thanks @jon-tow for the many adjustments!

[jon-tow]

No problem @dan-zheng. Thanks for the guidance!