cuda: fix layer split mode preventing cuda graph compilation

[koush]

When layer split is used, the output from one layer is the input to another layer.

I noticed that CUDA graphs would eventually give up due to too many changes in the graph. The input tensor would have a data address that would cycle between 4 (I think) allocated values. I dug into this a bit to see if I could fix that, so the cuda graph would not have a noisy input address, but my understanding is that this is done due to pipelining? Maybe that is the correct fix, and someone else could make that change. This change works under the assumption that the underlying platform is working as intended here.

In any case, I implemented CUDA graph compilation to account for input address changes by caching CUDA Graphs, and (quickly) finding the one that matches from a LRU set.

This showed appreciable gains in tokens per second:

bartowski/Qwen2.5-72B-Instruct-GGUF:IQ4_XS, 2 x RTX PRO 6000:

Before: 29 tokens/s
After: 35 tokens/s

[koush]

I just noticed this fix was done nearly the same time as mine and this change may be defunct:

#13814

For issue:
#13751

[koush]

Actually it turns out that I will likely need this change, or a derivative of it in the tensor parallel work I am doing, the graph is split for every RMS normalization where the gpus need to gather/reduce tensors. This results in graph churn.

[koush]

Will get another pull up once it is settled.