vulkan: Find optimal memory type but with fallback

[luciferous]

Some memory properties are nice to have, but not critical. eHostCached, for instance, isn't essential, and yet we fail on devices where this memory property isn't available.

ggml_vulkan: No suitable memory type found: ErrorOutOfDeviceMemory

This change differentiates between those properties that are critical and those that are just nice-to-have, and will fail only when critical properties aren't available.

Fixes #5319.

[luciferous]

@0cc4m Waiting for #5321 to merge before incorporating changes, but thought we can discuss the general idea.

[0cc4m]

For staging buffers we could probably just drop the cached requirement alltogether, since they only do memcpys to and from cpu. But overall this approach is probably a good idea and corresponds with something that VMA does. I'll give it a try soon and do a review. Thank you for your work!

[0cc4m]

I think it would be better to just add a fallback buffer type to the ggml_vk_create_buffers function, then it can also take over the fallback from DEVICE_LOCAL to HOST_VISIBLE/COHERENT for UMA devices (APUs).

[luciferous]

Thanks for the review @0cc4m.

For my understanding: are you suggesting we should always fallback to HOST_VISIBLE/COHERENT in ggml_vk_create_buffers when the wanted properties aren't available? I.e.,

memory_type_index = find_properties(&mem_props, &mem_req, req_flags);

if (memory_type_index == -1) {
    memory_type_index = find_properties(&mem_props, &mem_req, vk::MemoryPropertyFlagBits::eHostVisible | vk::MemoryPropertyFlagBits::eHostCoherent);
}

[0cc4m]

Thanks for the review @0cc4m.

For my understanding: are you suggesting we should always fallback to HOST_VISIBLE/COHERENT in ggml_vk_create_buffers when the wanted properties aren't available? I.e.,

memory_type_index = find_properties(&mem_props, &mem_req, req_flags);

if (memory_type_index == -1) {
    memory_type_index = find_properties(&mem_props, &mem_req, vk::MemoryPropertyFlagBits::eHostVisible | vk::MemoryPropertyFlagBits::eHostCoherent);
}

I mean instead of having required flags and desired flags we could just allow having two pairs of flags. If the first isn't available, fall back to the second. If that isn't available, throw an error. If the second set is empty, throw an error after the first fails.

static vk_buffer ggml_vk_create_buffer(ggml_backend_vk_context * ctx, size_t size, vk::MemoryPropertyFlags req_flags, vk::MemoryPropertyFlags fallback_flags = vk::MemoryPropertyFlags(0)) {
[...]
memory_type_index = find_properties(&mem_props, &mem_req, req_flags);

if (memory_type_index == -1 && fallback_flags != 0) {
    memory_type_index = find_properties(&mem_props, &mem_req, fallback_flags);
}

Something like this.

[0cc4m]

Thank you, looks good now.