[ET-VK] Introduce copy constructor for vTensor to allow for zero-copy…

[kirklandsign]

… operators

Context

For buffer-backed tensors, orchestration operators such as slicing, transposition, views, etc. can be implemented by creating a new tensor that uses the same storage as another tensor, but with different metadata (i.e. sizes and strides).

This diff implements copy constructors for the Allocation, VulkanBuffer, and vTensor classes which enable the aforementioned behaviour. Class instances created from copy constructors do not own the underlying memory resource, hence the resource will not be freed upon destruction of the class instance. Note that this behaviour is similar to copying a pointer in C/C++, and is inherently unsafe because the original resource may be destroyed before the copy.

However, in practice this is not much of a concern, because tensors must be kept alive for the duration of inference, thus all tensors created during model inference will have the same lifetime. However, it does pose a problem for memory planned tensors, since from the memory planner's perspective the lifetime of the original tensor may be shorter than the aliased tensor, thus the shared memory may be overwritten by other tensors using the same allocation. Therefore this behaviour is not yet safe to use when memory planning is enabled; additional work will be needed on the export side to make sure aliased tensors have the same lifetime as the original tensor.

Why not use shared_ptr?

In the past, this behaviour was enabled by vTensor instances storing their vTensorStorage classes via a shared_ptr. This was a safer design, since shared_ptr would handle resource management of the underlying buffer or texture resource.

However, I decided not to go with shared_ptr design because of the overhead involved in making a heap allocation whenever a vTensor is constructed, and the subsequent pointer chasing required whenever data is accessed from a vTensor. It seemed too big a cost to pay, especially considering tensor aliasing only really makes sense for buffer-backed tensors (thus it is not expected to be a common occurrence).

Also, as mentioned above the lifetime of all created vTensor instances tend to have the same lifetime in practice, especially in the context of the ComputeGraph class. Also, the shared_ptr design would still encounter the problem with memory planning.

Differential Revision: D61417569

[ghstack-poisoned]

[pytorch-bot]

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/4791

• 📄 Preview Python docs built from this PR

Note: Links to docs will display an error until the docs builds have been completed.

❌ 1 New Failure

As of commit 8648f68 with merge base b75e7d7 ():

NEW FAILURE - The following job has failed:

• pull / test-llava-runner-linux / linux-job (gh)
  RuntimeError: Command docker exec -t 0f538b12607b6e64e5b75fe9277947145e6b5f35b5ee14521a53c1ee7f8da92f /exec failed with exit code 127

This comment was automatically generated by Dr. CI and updates every 15 minutes.