Update Documentation with OpenCL limiations

Describe user facing limitations of OpenCL graphs
backend and document failing tests this directly affects.

Author: EwanC

sycl/doc/design/CommandGraph.md

@@ -149,8 +149,8 @@ yet been implemented.
 Implementation of UR command-buffers
 for each of the supported SYCL 2020 backends.
 
-Backends which are implemented currently are: Level Zero, CUDA and OpenCL.
-More sub-sections will be added here as other backends are supported.
+Backends which are implemented currently are: [Level Zero](#level-zero),
+[CUDA](#cuda), and partial support for [OpenCL](#opencl).
 
 ### Level Zero
 
@@ -249,9 +249,41 @@ graph resubmission.
 
 ### OpenCL
 
-Command Buffers are defined in the OpenCL spec in the [cl_khr_command_buffer](https://registry.khronos.org/OpenCL/specs/3.0-unified/html/OpenCL_Ext.html#cl_khr_command_buffer) extension.
-
-There are some gaps in both the OpenCL and UR specifications for Command 
+SYCL-Graphs is only enabled for an OpenCL backend when the
+[cl_khr_command_buffer](https://registry.khronos.org/OpenCL/specs/3.0-unified/html/OpenCL_Ext.html#cl_khr_command_buffer)
+extension is available, however this information isn't available until runtime
+due to OpenCL implementations being loaded through an ICD.
+
+The `ur_exp_command_buffer` string is conditionally returned from the OpenCL
+command-buffer UR backend at runtime based on `cl_khr_command_buffer` support
+to indicate that the graphs extension should be enabled. This is information
+is propagated to the SYCL user via the
+`device.get_info<info::device::graph_support>()` query for graphs extension
+support.
+
+#### Limitations
+
+Due to the API mapping gaps documented in the following section, OpenCL as a
+SYCL backend cannot fully support the graphs API. Instead there are
+limitations in the types on nodes which a user can add to a graph, using
+an unsupported node type will cause an abort in graph finalization with the
+message
+`ur_die: Experimental Command-buffer entry point is not implemented for OpenCL adapter.`.
+
+The types of commands which are unsupported, and lead to this exception are:
+* `handler::copy(src, dest)` - Where `src` is an accessor and `dest` is a pointer.
+   This corresponds to a memory buffer read command.
+* `handler::copy(src, dest)` - Where `src` is an pointer and `dest` is an accessor.
+  This corresponds to a memory buffer write command.
+* `handler::copy(src, dest)` or `handler::memcpy(dest, src)` - Where both `src` and
+   `dest` are USM pointers. This corresponds to a USM copy command.
+
+Note that `handler::copy(src, dest)` where both `src` and `dest` are an accessor
+is supported, as a memory buffer copy command exists in the OpenCL extension.
+
+#### UR API Mapping
+
+There are some gaps in both the OpenCL and UR specifications for Command
 Buffers shown in the list below. There are implementations in the UR OpenCL
 adapter where there is matching support for each function in the list.
 
@@ -280,27 +312,29 @@ adapter where there is matching support for each function in the list.
 |  | clCommandSVMMemcpyKHR | No |
 |  | clCommandSVMMemFillKHR | No |
 
+#### UR Command-Buffer Implementation
+
 Many of the OpenCL functions take a `cl_command_queue` parameter which is not
 present in most of the UR functions. Instead, when a new command buffer is
-created in `urCommandBufferCreateExp` we also create and maintain a new 
-internal `ur_queue_handle_t` with a reference stored inside of the 
-`ur_exp_command_buffer_handle_t_` struct. This internal queue is then used with
-the various append functions. The internal queue is retained and released 
-whenever the owning command buffer is retained or released.
+created in `urCommandBufferCreateExp` we also create and maintain a new
+internal `ur_queue_handle_t` with a reference stored inside of the
+`ur_exp_command_buffer_handle_t_` struct. The internal queue is retained and
+released whenever the owning command buffer is retained or released.
 
 With command buffers being an OpenCL extension, each function is accessed by
 loading a function pointer to its implementation. These are defined in a common
 header file in the UR OpenCL adapter. The symbols for the functions are however
-defined in [OpenCL-Headers](https://github.com/KhronosGroup/OpenCL-Headers/blob/main/CL/cl_ext.h) but it is not known at this time what version of the headers will be used in the UR GitHub CI configuration, so loading the function
-pointers will be used until this can be verified. A future piece of work would 
-be replacing the custom defined symbols with the ones from OpenCL-Headers.
+defined in [OpenCL-Headers](https://github.com/KhronosGroup/OpenCL-Headers/blob/main/CL/cl_ext.h)
+but it is not known at this time what version of the headers will be used in
+the UR GitHub CI configuration, so loading the function pointers will be used
+until this can be verified. A future piece of work would be replacing the
+custom defined symbols with the ones from OpenCL-Headers.
+
+#### Available OpenCL Command-Buffer Implementations
 
-The `UR_DEVICE_INFO_EXTENSIONS` enum can be used with `urDeviceGetInfo` to
-query if a specified device supports OpenCL command buffers. This will append 
-`ur_exp_command_buffer` to a string pointer passed to the function if the 
-extension is supported.
+Publicly available implementations of `cl_khr_command_buffer` that can be used
+to enable the graphs extension in OpenCL:
 
-Known implementations of cl_khr_command_buffer:
 - [OneAPI-Construction-Kit](https://github.com/codeplaysoftware/oneapi-construction-kit) (must enable `OCL_EXTENSION_cl_khr_command_buffer` when building)
 - [PoCL](http://portablecl.org/)
 - [Command-Buffer Emulation Layer](https://github.com/bashbaug/SimpleOpenCLSamples/tree/main/layers/10_cmdbufemu)

sycl/doc/design/images/SYCL-Graph-Architecture.svg

@@ -7,6 +7,9 @@
 //
 // TODO enable cuda once buffer issue investigated and fixed
 // UNSUPPORTED: cuda
+//
+// Host to device copy command not supported for OpenCL
+// UNSUPPORTED: opencl
 
 #define GRAPH_E2E_EXPLICIT
 

sycl/test-e2e/Graph/Explicit/buffer_copy_host2target.cpp

@@ -7,6 +7,9 @@
 //
 // TODO enable cuda once buffer issue investigated and fixed
 // UNSUPPORTED: cuda
+//
+// Host to device copy command not supported for OpenCL
+// UNSUPPORTED: opencl
 
 #define GRAPH_E2E_EXPLICIT
 

sycl/test-e2e/Graph/Explicit/buffer_copy_host2target_2d.cpp

@@ -7,6 +7,9 @@
 //
 // TODO enable cuda once buffer issue investigated and fixed
 // UNSUPPORTED: cuda
+//
+// Host to device copy command not supported for OpenCL
+// UNSUPPORTED: opencl
 
 #define GRAPH_E2E_EXPLICIT
 

sycl/test-e2e/Graph/Explicit/buffer_copy_host2target_offset.cpp

@@ -7,6 +7,9 @@
 //
 // TODO enable cuda once buffer issue investigated and fixed
 // UNSUPPORTED: cuda
+//
+// Device to host copy command not supported for OpenCL
+// UNSUPPORTED: opencl
 
 #define GRAPH_E2E_EXPLICIT
 

sycl/test-e2e/Graph/Explicit/buffer_copy_target2host.cpp

@@ -7,6 +7,9 @@
 //
 // TODO enable cuda once buffer issue investigated and fixed
 // UNSUPPORTED: cuda
+//
+// Device to host copy command not supported for OpenCL
+// UNSUPPORTED: opencl
 
 #define GRAPH_E2E_EXPLICIT
 

sycl/test-e2e/Graph/Explicit/buffer_copy_target2host_2d.cpp

@@ -7,6 +7,9 @@
 //
 // TODO enable cuda once buffer issue investigated and fixed
 // UNSUPPORTED: cuda
+//
+// Device to host copy command not supported for OpenCL
+// UNSUPPORTED: opencl
 
 #define GRAPH_E2E_EXPLICIT
 

sycl/test-e2e/Graph/Explicit/buffer_copy_target2host_offset.cpp

@@ -4,6 +4,9 @@
 // RUN: %if ext_oneapi_level_zero %{env UR_L0_LEAKS_DEBUG=1 %{run} %t.out 2>&1 | FileCheck %s %}
 //
 // CHECK-NOT: LEAK
+//
+// USM copy command not supported for OpenCL
+// UNSUPPORTED: opencl
 
 #define GRAPH_E2E_EXPLICIT
 

sycl/test-e2e/Graph/Explicit/usm_copy.cpp

@@ -7,6 +7,9 @@
 //
 // TODO enable cuda once buffer issue investigated and fixed
 // UNSUPPORTED: cuda
+//
+// Host to device copy command not supported for OpenCL
+// UNSUPPORTED: opencl
 
 #define GRAPH_E2E_RECORD_REPLAY
 

sycl/test-e2e/Graph/RecordReplay/buffer_copy_host2target.cpp

@@ -7,6 +7,9 @@
 //
 // TODO enable cuda once buffer issue investigated and fixed
 // UNSUPPORTED: cuda
+//
+// Host to device copy command not supported for OpenCL
+// UNSUPPORTED: opencl
 
 #define GRAPH_E2E_RECORD_REPLAY
 

sycl/test-e2e/Graph/RecordReplay/buffer_copy_host2target_2d.cpp

@@ -7,6 +7,9 @@
 //
 // TODO enable cuda once buffer issue investigated and fixed
 // UNSUPPORTED: cuda
+//
+// Host to device copy command not supported for OpenCL
+// UNSUPPORTED: opencl
 
 #define GRAPH_E2E_RECORD_REPLAY
 

sycl/test-e2e/Graph/RecordReplay/buffer_copy_host2target_offset.cpp

@@ -7,6 +7,9 @@
 //
 // TODO enable cuda once buffer issue investigated and fixed
 // UNSUPPORTED: cuda
+//
+// Device to host copy command not supported for OpenCL
+// UNSUPPORTED: opencl
 
 #define GRAPH_E2E_RECORD_REPLAY
 

sycl/test-e2e/Graph/RecordReplay/buffer_copy_target2host.cpp

@@ -7,6 +7,9 @@
 //
 // TODO enable cuda once buffer issue investigated and fixed
 // UNSUPPORTED: cuda
+//
+// Device to host copy command not supported for OpenCL
+// UNSUPPORTED: opencl
 
 #define GRAPH_E2E_RECORD_REPLAY
 

sycl/test-e2e/Graph/RecordReplay/buffer_copy_target2host_2d.cpp

@@ -7,6 +7,9 @@
 //
 // TODO enable cuda once buffer issue investigated and fixed
 // UNSUPPORTED: cuda
+//
+// Device to host copy command not supported for OpenCL
+// UNSUPPORTED: opencl
 
 #define GRAPH_E2E_RECORD_REPLAY
 

sycl/test-e2e/Graph/RecordReplay/buffer_copy_target2host_offset.cpp

@@ -4,6 +4,9 @@
 // RUN: %if ext_oneapi_level_zero %{env UR_L0_LEAKS_DEBUG=1 %{run} %t.out 2>&1 | FileCheck %s %}
 //
 // CHECK-NOT: LEAK
+//
+// USM copy command not supported for OpenCL
+// UNSUPPORTED: opencl
 
 #define GRAPH_E2E_RECORD_REPLAY
 

sycl/test-e2e/Graph/RecordReplay/usm_copy.cpp

@@ -4,6 +4,9 @@
 // RUN: %if ext_oneapi_level_zero %{env UR_L0_LEAKS_DEBUG=1 %{run} %t.out 2>&1 | FileCheck %s %}
 //
 // CHECK-NOT: LEAK
+//
+// USM copy command not supported for OpenCL
+// UNSUPPORTED: opencl
 
 // Tests memcpy operation using device USM and an in-order queue.
 

sycl/test-e2e/Graph/RecordReplay/usm_copy_in_order.cpp

@@ -21,8 +21,13 @@ int main() {
   auto Backend = Device.get_backend();
 
   if ((Backend == backend::ext_oneapi_level_zero) ||
-      (Backend == backend::ext_oneapi_cuda) || (Backend == backend::opencl)) {
+      (Backend == backend::ext_oneapi_cuda)) {
     assert(SupportsGraphs == exp_ext::graph_support_level::native);
+  } else if (Backend == backend::opencl) {
+    // OpenCL backend support is conditional on the cl_khr_command_buffer
+    // extension being available
+    assert(SupportsGraphs == exp_ext::graph_support_level::native ||
+           SupportsGraphs == exp_ext::graph_support_level::unsupported);
   } else {
     assert(SupportsGraphs == exp_ext::graph_support_level::unsupported);
   }