Lazy tensor: automatically promote constants to inputs based on history

[bgogul]

This PR implements the logic to promote constants to inputs (https://bugs.swift.org/browse/TF-704). Suppose that we have the following traces:

lazyTrace_sum2(%0: float) -> (float) {
   %1 = Const[dtype: float, value: 2.0]() 
   %2 = Add[T: float](%0, %1)
   return %2
}

lazyTrace_sum2(%0: float) -> (float) {
   %1 = Const[dtype: float, value: 3.0]() 
   %2 = Add[T: float](%0, %1)
   return %2
}

The above two functions are different and will need to be compiled individually. Suppose that we promote these constants to inputs as follows:

lazyTrace_sum3(%0: float, %1: float) -> (float) {
   %2 = Add[T: float](%0, %1)
   return %2
}

Only the inputs change and the function remains constant, which eliminates the additional compilation overhead.

[eaplatanios]

Thanks @bgogul! This is great. I'm really curious as to how big the performance gains / number of materializations reduced are after this change. I made some stylistic comments and I will try to go through the logic later today.

[bgogul]

Thanks @bgogul! This is great. I'm really curious as to how big the performance gains / number of materializations reduced are after this change. I made some stylistic comments and I will try to go through the logic later today.

Thanks for the review, @eaplatanios! I have addresed your style comments.

This is absolutely critical for performance. e.g., the step size constant changes often due to decay. If we don't promote the step size as an input, we will pay the cost of XLA compilation for almost every step. XLA compilation is not cheap; for MNIST, it takes about 6 s for compiling a single step of the training loop.

[bgogul]

@rxwei this is not super urgent, but can you take a look at this when you get a chance?

[rxwei]

@rxwei this is not super urgent, but can you take a look at this when you get a chance?

Sorry for the delay!

[bgogul]

@rxwei this is not super urgent, but can you take a look at this when you get a chance?

Sorry for the delay!

No worries. I addressed your comments and left some clarification questions.