[SYCL][E2E] Enable llvm-lit to accept device architecture

[DBDuncan]

Allow for selection of specific devices via level_zero:arch-intel_gpu_bmg_g21 which accepts device architectures listed under sycl-ls --verbose. Devices with multiple GPU's under the devices with multiple GPU's under the device selection style of backend:*device type* such as level_zero:gpu can cause incorrect aspects to be marked as unsupported in the tests ran through llvm-lit using // REQUIRES: aspect-. In addition, only the first device listed via sycl-ls under the label is ran.

This occurs because using the device selection style of level_zero:gpu there is currently no way to choose a specific device that falls under that category. Such as a battlemage GPU and an intel iGPU. sycl/test-e2e/lit.cfg.py takes all available device aspects and marks aspects as supported that are available for all the devices. This is problematic for aspects such as ext_oneapi_bindless_images which is supported on battlemage but not Intel iGPU's. Even if battlemage, which supports bindless images is used to run all the tests as it comes first in the device list shown under level_zero:gpu, tests requiring that aspect will not run.

[aelovikov-intel]

What problem are you trying to solve? Empty PR description and no comments in the change don't help to understand the idea behind this.

[DBDuncan]

Ah sorry. You are right. The issue is that on devices with say a battlemage GPU and an intel intergrated iGPU, when running llvm-lit with --param sycl_devices="level_zero:gpu" set, when sycl/test-e2e/lit.cfg.py is sorting out which aspects are enabled, it only signals aspects as supported to // REQUIRES: aspect- that are supported on all intel devices listed via level_zero:gpu from sycl-ls.

This is a problem with bindless images as the aspect exp_oneapi_bindless_images is not supported on intel iGPUs, but even if the tests are running on a supported device because that device comes first in the list it will not run them because the aspect in tests returns the feature as unsupported.

Currently, sycl/test-e2e/lit.cfg.py does not support the level_zero:0/1/2/etc device selection style which would fix this issue. So patch intends to fix this allowing for explicit selection for the device, not just platform and device type.

[aelovikov-intel]

First, that should go to the PR description, not the comment. Second, please add an example usage, e.g., sycl-ls shows this, we run llvm-lit like that, and it does the following (and print detected aspects from its output).

[DBDuncan]

I am putting this as a comment here as it feels like a lot to put in the commit message. Though I can put it there if you prefer or a cut down version. I might be going a bit too verbose.

On the system I am using, when running sycl-ls like this it shows these devices as available:

./bin/sycl-ls`

[level_zero:gpu][level_zero:0] Intel(R) oneAPI Unified Runtime over Level-Zero, Intel(R) Arc(TM) B580 Graphics 20.1.0 [1.6.32567+16]
[level_zero:gpu][level_zero:1] Intel(R) oneAPI Unified Runtime over Level-Zero, Intel(R) UHD Graphics 770 12.2.0 [1.6.32567+16]
[opencl:gpu][opencl:0] Intel(R) OpenCL Graphics, Intel(R) Arc(TM) B580 Graphics OpenCL 3.0 NEO  [25.05.32567]
[opencl:gpu][opencl:1] Intel(R) OpenCL Graphics, Intel(R) UHD Graphics 770 OpenCL 3.0 NEO  [25.05.32567]

I am abbreviating this, but adding --verbose shows the following aspects as available for the battlemage:

        Aspects           : gpu fp16 fp64 online_compiler online_linker queue_profiling usm_device_allocations usm_host_allocations usm_shared_allocations ext_intel_pci_address ext_intel_gpu_eu_count ext_intel_gpu_eu_simd_width ext_intel_gpu_slices ext_intel_gpu_subslices_per_slice ext_intel_gpu_eu_count_per_subslice atomic64 ext_intel_device_info_uuid ext_intel_gpu_hw_threads_per_eu ext_oneapi_cuda_async_barrier ext_intel_free_memory ext_intel_device_id ext_intel_memory_clock_rate ext_intel_memory_bus_width ext_intel_legacy_image ext_oneapi_bindless_images ext_oneapi_bindless_images_1d_usm ext_oneapi_bindless_images_2d_usm ext_oneapi_external_memory_import ext_intel_esimd ext_oneapi_ballot_group ext_oneapi_fixed_size_group ext_oneapi_opportunistic_group ext_oneapi_tangle_group ext_intel_matrix ext_oneapi_limited_graph ext_oneapi_private_alloca ext_oneapi_bindless_sampled_image_fetch_1d_usm ext_oneapi_bindless_sampled_image_fetch_2d_usm ext_oneapi_bindless_sampled_image_fetch_2d ext_oneapi_bindless_sampled_image_fetch_3d ext_oneapi_queue_profiling_tag ext_oneapi_virtual_mem ext_oneapi_image_array ext_oneapi_virtual_functions ext_intel_spill_memory_size ext_intel_current_clock_throttle_reasons ext_intel_power_limits ext_oneapi_async_memory_alloc

And the intel Intel(R) UHD Graphics 770:

        Aspects           : gpu fp16 online_compiler online_linker queue_profiling usm_device_allocations usm_host_allocations usm_shared_allocations ext_intel_pci_address ext_intel_gpu_eu_count ext_intel_gpu_eu_simd_width ext_intel_gpu_slices ext_intel_gpu_subslices_per_slice ext_intel_gpu_eu_count_per_subslice atomic64 ext_intel_device_info_uuid ext_intel_gpu_hw_threads_per_eu ext_oneapi_cuda_async_barrier ext_intel_device_id ext_intel_memory_clock_rate ext_intel_memory_bus_width ext_intel_legacy_image ext_oneapi_external_memory_import ext_intel_esimd ext_oneapi_ballot_group ext_oneapi_fixed_size_group ext_oneapi_opportunistic_group ext_oneapi_tangle_group ext_oneapi_limited_graph ext_oneapi_private_alloca ext_oneapi_bindless_sampled_image_fetch_1d_usm ext_oneapi_bindless_sampled_image_fetch_2d_usm ext_oneapi_bindless_sampled_image_fetch_2d ext_oneapi_bindless_sampled_image_fetch_3d ext_oneapi_queue_profiling_tag ext_oneapi_virtual_mem ext_oneapi_image_array ext_oneapi_virtual_functions ext_intel_spill_memory_size ext_intel_current_clock_throttle_reasons ext_intel_power_limits ext_oneapi_async_memory_alloc

When running llvm-lit via this command:
python3.12 bin/llvm-lit -sva --param sycl_devices="level_zero:gpu" ../sycl/test-e2e/bindless_images/

I get the following output before the tests start running:

llvm-lit: /home/duncanb/git/llvm/sycl/test-e2e/lit.cfg.py:613: note: Targeted devices: level_zero:gpu
llvm-lit: /home/duncanb/git/llvm/sycl/test-e2e/lit.cfg.py:815: note: Found pre-installed AOT device compiler ocloc
llvm-lit: /home/duncanb/git/llvm/sycl/test-e2e/lit.cfg.py:818: warning: Couldn't find pre-installed AOT device compiler opencl-aot
llvm-lit: /home/duncanb/git/llvm/sycl/test-e2e/lit.cfg.py:929: note: Aspects for level_zero:gpu: ext_intel_power_limits, ext_intel_current_clock_throttle_reasons, ext_oneapi_private_alloca, ext_intel_gpu_eu_count, ext_oneapi_virtual_mem, atomic64, queue_profiling, ext_oneapi_queue_profiling_tag, ext_oneapi_external_memory_import, ext_intel_device_info_uuid, ext_intel_pci_address, usm_host_allocations, ext_oneapi_virtual_functions, ext_oneapi_limited_graph, ext_oneapi_ballot_group, online_compiler, ext_intel_legacy_image, ext_oneapi_opportunistic_group, ext_oneapi_bindless_sampled_image_fetch_1d_usm, ext_intel_gpu_hw_threads_per_eu, fp16, ext_oneapi_bindless_sampled_image_fetch_2d_usm, usm_shared_allocations, ext_intel_memory_bus_width, ext_oneapi_fixed_size_group, ext_intel_gpu_eu_simd_width, ext_intel_gpu_subslices_per_slice, online_linker, ext_intel_spill_memory_size, ext_oneapi_async_memory_alloc, ext_oneapi_bindless_sampled_image_fetch_2d, ext_intel_esimd, ext_oneapi_cuda_async_barrier, gpu, usm_device_allocations, ext_intel_memory_clock_rate, ext_intel_device_id, ext_oneapi_image_array, ext_intel_gpu_eu_count_per_subslice, ext_oneapi_bindless_sampled_image_fetch_3d, ext_oneapi_tangle_group, ext_intel_gpu_slices
llvm-lit: /home/duncanb/git/llvm/sycl/test-e2e/lit.cfg.py:941: note: SG sizes for level_zero:gpu: 32, 16
llvm-lit: /home/duncanb/git/llvm/sycl/test-e2e/lit.cfg.py:950: note: Architectures for level_zero:gpu: intel_gpu_adl_s, intel_gpu_bmg_g21

Then, running llvm-lit using level_zero:0 like this:
python3.12 bin/llvm-lit -sva --param sycl_devices="level_zero:0" ../sycl/test-e2e/bindless_images/

Get the following output:

llvm-lit: /home/duncanb/git/llvm/sycl/test-e2e/lit.cfg.py:613: note: Targeted devices: level_zero:0
llvm-lit: /home/duncanb/git/llvm/sycl/test-e2e/lit.cfg.py:815: note: Found pre-installed AOT device compiler ocloc
llvm-lit: /home/duncanb/git/llvm/sycl/test-e2e/lit.cfg.py:818: warning: Couldn't find pre-installed AOT device compiler opencl-aot
llvm-lit: /home/duncanb/git/llvm/sycl/test-e2e/lit.cfg.py:929: note: Aspects for level_zero:0: ext_oneapi_tangle_group, ext_intel_legacy_image, ext_intel_gpu_slices, ext_oneapi_bindless_sampled_image_fetch_2d, ext_intel_power_limits, ext_oneapi_async_memory_alloc, ext_oneapi_private_alloca, ext_intel_matrix, usm_shared_allocations, online_linker, usm_host_allocations, ext_intel_memory_bus_width, ext_oneapi_external_memory_import, ext_oneapi_bindless_sampled_image_fetch_1d_usm, ext_oneapi_queue_profiling_tag, online_compiler, ext_oneapi_virtual_mem, usm_device_allocations, ext_oneapi_image_array, ext_intel_gpu_eu_count_per_subslice, ext_intel_memory_clock_rate, ext_oneapi_bindless_sampled_image_fetch_3d, ext_intel_spill_memory_size, ext_oneapi_ballot_group, ext_oneapi_cuda_async_barrier, ext_intel_gpu_eu_simd_width, ext_intel_gpu_subslices_per_slice, ext_intel_device_id, ext_oneapi_bindless_images, ext_oneapi_bindless_images_2d_usm, ext_intel_gpu_eu_count, queue_profiling, ext_oneapi_virtual_functions, fp64, ext_oneapi_fixed_size_group, ext_intel_device_info_uuid, ext_oneapi_bindless_sampled_image_fetch_2d_usm, ext_intel_pci_address, ext_oneapi_bindless_images_1d_usm, fp16, ext_intel_free_memory, ext_intel_current_clock_throttle_reasons, ext_intel_esimd, gpu, ext_oneapi_opportunistic_group, ext_oneapi_limited_graph, atomic64, ext_intel_gpu_hw_threads_per_eu
llvm-lit: /home/duncanb/git/llvm/sycl/test-e2e/lit.cfg.py:941: note: SG sizes for level_zero:0: 32, 16
llvm-lit: /home/duncanb/git/llvm/sycl/test-e2e/lit.cfg.py:950: note: Architectures for level_zero:0: intel_gpu_bmg_g21

As can be seen with the llvm-lit command using level_zero:gpu, the aspect ext_oneapi_bindless_images is not available but when using the more specific device selection style of `level_zero:0 it is.

As far as I am aware, it is not currently possible to make llvm-lit sycl run tests that require an aspect that is available on a device selectable via level_zero:gpu but not another that shows under the same label on the same system without making changes to the code.

Enabling the level_zero:0 device selection style via llvm-lit should make it easier to run tests on systems with multiple GPUs without needing to disable one GPU or force enable the aspect.

[DBDuncan]

@aelovikov-intel I apologize for taking so long. Other work got in the way.

I might have gone overboard with all the aspects I am showing but I felt I would be leaving out context without them. I appreciate your feedback if you want anything else.

[aelovikov-intel]

I don't like --param sycl_devices="level_zero:0" for CI purposes because it will be possible to silently loose coverage if devices order changes for some reason. IMO, a better approach is to somehow pass intel_gpu_bmg_g21 as the intended target. I'm not sure how to do that though.

I'd like to see the following logic: if we have multiple backend:device_type devices then extra parameter --param require-device-arch=<value> is required. Then, the lit.cfg.py should verify that only one of backend:device_type devices satisfy that condition and select that number internally.

However, that won't work with multi-tile GPUs that are seen as separate root devices. Maybe "only one" above is unnecessary and we should always select a numbered device out of several options matching a given backend:device_type. The filtering via --param require-device-arch=<value> above should still take place.

WDYT? + @sarnex , @uditagarwal97

[sarnex]

I agree, I don't like using the device ID number, totally agree we could silently loose coverage.

I like the require-device-arch idea, but ideally ONEAPI_DEVICE_SELECTOR would support level_zero:intel_gpu_bmg_g21 or something so the SYCL runtime would have to figure it out instead of the LIT code.

But since we don't have that, a LIT param is fine.

For the multi-tile case, I'm sure we can figure it out. The devices will probably have the same UUID or PCI bus address or something so we can detect it's multi-tile.

[DBDuncan]

This seems like a good solution. The only thing is because sycl_devices accepts multiple parameters, require-device-arch likely should as well.

I am thinking to not require to put as many require-device-arch as there are sycl_devices, and instead perhaps emit a warning if the number does not match up and then default to the current behaviour. Any thoughts?

[sarnex]

I think we can do a sensible simple default and then expand it in the future if we need it. Maybe we could just have a way to specify cpu and gpu devices so that we could run opencl:gpu and opencl:gpu testing at the same time but still pass require-device-arch.

Maybe something like

--param require-device-arch="gpu:intel_gpu_bmg_g21;cpu:sapphire_rapids"
(I have no idea what the arches are for CPU I just made something up

and we would apply the gpu specification to both level_zero:gpu and opencl:gpu but not opencl:cpu and vice versa.

[aelovikov-intel]

I think CPU arch isn't useful. Can you imagine a system with two different CPUs on it? We aren't MPI...

[sarnex]

Yeah I don't have a strong opinion, for now we could only support GPUs. Maybe name the option require-gpu-device-arch to make it clear? Important thing is to apply it for both level_zero:gpu and opencl:gpu and CUDA/HIP tho

[DBDuncan]

I have been investigating implementing this and I believe internally I will need to make it so that when --param sycl_devices="level_zero:gpu" --param require-gpu-device-arch="intel_gpu_bmg_g21 is set it will need to internally swap level_zero:gpu with the appropriate level_zero:N device selector as there seems to be no other way to specificity select a device.

[aelovikov-intel]

I have been investigating implementing this and I believe internally I will need to make it so that when --param sycl_devices="level_zero:gpu" --param require-gpu-device-arch="intel_gpu_bmg_g21 is set it will need to internally swap level_zero:gpu with the appropriate level_zero:N device selector as there seems to be no other way to specificity select a device.

That's what I was thinking. Maybe even go further and do a switch anytime there are multiple devices matching backend:device-type.

[DBDuncan]

@sarnex @aelovikov-intel Here is the new version using require-device-arch. Behaviour should be the same as before if require-device-arch is not used. Bit uncertain how to go about CI testing.

I designed it so you can only filter devices on a one to one basis. Meaning that if say you want to run both intel_gpu_adl_s and intel_gpu_bmg_g21 which both fall under the level_zero:gpu label, you would have to do this: --param sycl_devices="level_zero:gpu;level_zero:gpu` --param require_device_arch="intel_gpu_bmg_g21;intel_gpu_adl_s"

Feedback is appreciated! This design is perhaps slightly different then the design discussed in this thread before but should be a good starting point.

Apologies if the python code is not very pythonic. Have not written in it for a while but is nice to get some more experience.

[aelovikov-intel]

Brainstorming... What if we do it like this: --param sycl_devices=level_zero:arch-intel_gpu_bmg_g21 ?

[sarnex]

yeah i like andrei's idea, today with sycl_devices=level_zero:gpu we would only ever be testing one device so we ideally wouldnt change that with this new param, so if we can specify the device multiple times with a specific arch that would solve it

[DBDuncan]

Ooh, that actually could work. Internally would have to check for devices via level_zero:* and then filter based upon those results.

[DBDuncan]

I think I would have to make it so that if the second part of level_zero:arch-intel_gpu_bmg_g21 is not recognised based upon

available_devices = {
    "opencl": ("cpu", "gpu", "fpga"),
    "cuda": "gpu",
    "level_zero": "gpu",
    "hip": "gpu",
    "native_cpu": "cpu",
}

then assume it is a device architecture specific filter.

[aelovikov-intel]

My assumption was that arch- prefix can serve as the trigger for this logic.

[aelovikov-intel]

Good enough to me, none of us are python experts (to the best of my knowledge).

@sarnex ?

[DBDuncan]

I missed a third sycl-ls invocation. Slightly different this one as it does not call --verbose. Just modified it to add with test_env():.

[sarnex]

also not a python export but lgtm minus a minor question

[sarnex]

i really double this actually matters but is it possible to prevent a second call to sycl-ls here and capture what we need in the first call?

[aelovikov-intel]

Don't we use different device filter?

[sarnex]

yeah in this one we set a filter but it seems the other one has no device filter, so it's output should contain the info we need here, but if its not trivial to implement its not worth it

[DBDuncan]

The first sycl-ls, line 638 in the code is used to get all available devices if all is used so no filter is passed.

The second sycl-ls, line 882 is only called if the string arch- appears in the device filter. We could also grab the aspects at the same time but that feels messy.

Lastly, sycl-ls at line 938 is called again with all the filtered device filters to set available aspects and other properties.

[DBDuncan]

We prob could remove one of the calls to sycl-ls in the code but I think the separation of concerns with filtering the device filters and then using them to query sycl-ls again to set test parameters makes things less tangled.

[sarnex]

sure, thanks

[sarnex]

Once you fix the code formatting CI fail we can merge this IMO

[DBDuncan]

Once you fix the code formatting CI fail we can merge this IMO

I knew I forgot something. Had to install the black formatter extension. Should be good now.

[sarnex]

thanks, this is a really useful feature!

[DBDuncan]

Thanks! There is a very minor documentation update i missed at line 281 of sycl/test-e2e/README.md just to explain that device selection like level_zero:arch-intel_gpu_bmg_g21 is accepted. Like I pushed the fix just 2 mins after this was merged lol. Coming to the end of the day and is a very minor change. Will make a PR correcting this on Monday.

[sarnex]

Np, thanks, take it easy!

[hvdijk]

The refactor to call sp = subprocess.Run(...) in a function means that later on, in e.g.

        lit_config.error(
            "Cannot detect device aspect for {}\nstdout:\n{}\nstderr:\n{}".format(
                sycl_device, sp.stdout, sp.stderr
            )
        )

sp is no longer available and this crashes rather than reporting an appropriate error. Likewise for other lit_config.errors in the same file. Could you take a look? I can open a new issue for this if you prefer?

[DBDuncan]

The refactor to call sp = subprocess.Run(...) in a function means that later on, in e.g.

        lit_config.error(
            "Cannot detect device aspect for {}\nstdout:\n{}\nstderr:\n{}".format(
                sycl_device, sp.stdout, sp.stderr
            )
        )

sp is no longer available and this crashes rather than reporting an appropriate error. Likewise for other lit_config.errors in the same file. Could you take a look? I can open a new issue for this if you prefer?

Sorry Harald, just got back from holiday. Will fix this. Need to be more careful jumping from a compiled to interpreted language with things like this...

[hvdijk]

@RossBrunton also separately encountered this and already put up a fix as #18535 🙂

[DBDuncan]

Ah, ok lol