Added support for 'Tensor.clipped(min:max:)' and its VJP.

[eaplatanios]

No description provided.

[Shashi456]

I guess after this PR, a lot of layers can be updated to avoid certain values beyond bounds. Like its done in keras.

[eaplatanios]

I guess after this PR, a lot of layers can be updated to avoid certain values beyond bounds. Like its done in keras.

Could you give examples of some of these layers in Keras? I'm curious to see where they clip implicitly.

[rxwei]

I feel the tensor-scalar and scalar-tensor overloads are not quite necessary and may not set a good example to follow for future APIs. Say, if some API has three 0-D tensor parameters (e.g. batch norm), would we want to write all 6 combinations of scalars parameters and tensor parameters?

[eaplatanios]

I agree. Would it be possible to make Scalar: TensorFlowScalar implicitly convertible to Tensor<Scalar>?

[rxwei]

No, implicit conversion is generally not possible in Swift except for literals. However, scalar literal conversion for Tensor has been very problematic.

[eaplatanios]

What is it that makes it problematic? Is it too inefficient for the type checker?

[rxwei]

Type checking ambiguity. s + t is possible where s is a scalar and t is a tensor, but if you make tensor also expressible by literal, the type checker won't know which overload you are calling for an expression like 3 + x (scalar-tensor or tensor-tensor).

[eaplatanios]

Can we not impose a hierarchy on the literal conversion rules?

[rxwei]

What do you mean exactly?

[rxwei]

Type checking precedence changes generally need to be an evolution proposal.

[eaplatanios]

Yes that would need an evolution proposal indeed. We could go for something even more powerful and support implicit type conversions using something like this:

@implicitConversion(precedence: ...)
func _<Scalar: TensorFlowScalar>(_ value: Scalar) -> Tensor<Scalar> {
  ...
}

Do you think something like this could potentially go through evolution? Also, how time consuming would it be support something like this in the compiler?

[eaplatanios]

Note that this would be separate from how literals are handled.

[rxwei]

This would be highly unlikely. I would suggest asking a question on the Swift evolution forum about it.

[eaplatanios]

Started a discussion thread here.