Skip to content

Commit 41221e2

Browse files
[SYCL] Add Level-Zero interop with specification of ownership (#3231)
The Level-Zero backend specified assuming ownership of the native handles used in the interoperability. There was however request made that this is not sufficient in all cases, e.g. when some memory was created outside SYCL RT in some Level-Zero Context, then working with that memory should always be within the same context. Thus if anyone wanted to "return" something back from SYCL, then the context should still exist and be the same. This patch only does for context, but probably we'd need to deal with other objects too at some point. Signed-off-by: Sergey V Maslov [email protected]
1 parent ae0cb4f commit 41221e2

File tree

11 files changed

+99
-35
lines changed

11 files changed

+99
-35
lines changed

sycl/doc/extensions/LevelZeroBackend/LevelZeroBackend.md

Lines changed: 33 additions & 12 deletions
Original file line numberDiff line numberDiff line change
@@ -87,7 +87,7 @@ a SYCL object that encapsulates a corresponding Level-Zero object:
8787
|-------------|:------------|
8888
|``` make<platform>(ze_driver_handle_t);```|Constructs a SYCL platform instance from a Level-Zero ```ze_driver_handle_t```.|
8989
|``` make<device>(const platform &, ze_device_handle_t);```|Constructs a SYCL device instance from a Level-Zero ```ze_device_handle_t```. The platform argument gives a SYCL platform, encapsulating a Level-Zero driver supporting the passed Level-Zero device.|
90-
|``` make<context>(const vector_class<device> &, ze_context_handle_t);```| Constructs a SYCL context instance from a Level-Zero ```ze_context_handle_t```. The context is created against the devices passed in. There must be at least one device given and all the devices must be from the same SYCL platform and thus from the same Level-Zero driver.|
90+
|``` make<context>(const vector_class<device> &, ze_context_handle_t, ownership = transfer);```| Constructs a SYCL context instance from a Level-Zero ```ze_context_handle_t```. The context is created against the devices passed in. There must be at least one device given and all the devices must be from the same SYCL platform and thus from the same Level-Zero driver. The ```ownership``` argument specifies if the SYCL runtime should take ownership of the passed native handle. The default behavior is to transfer the ownership to the SYCL runtime. See section 4.4 for details.|
9191
|``` make<queue>(const context &, ze_command_queue_handle_t);```| Constructs a SYCL queue instance from a Level-Zero ```ze_command_queue_handle_t```. The context argument must be a valid SYCL context encapsulating a Level-Zero context. The queue is attached to the first device in the passed SYCL context.|
9292
|``` make<program>(const context &, ze_module_handle_t);```| Constructs a SYCL program instance from a Level-Zero ```ze_module_handle_t```. The context argument must be a valid SYCL context encapsulating a Level-Zero context. The Level-Zero module must be fully linked (i.e. not require further linking through [```zeModuleDynamicLink```](https://spec.oneapi.com/level-zero/latest/core/api.html?highlight=zemoduledynamiclink#_CPPv419zeModuleDynamicLink8uint32_tP18ze_module_handle_tP28ze_module_build_log_handle_t)), and thus the SYCL program is created in the "linked" state.|
9393
@@ -96,23 +96,43 @@ NOTE: We shall consider adding other interoperability as needed, if possible.
9696
### 4.4 Level-Zero handles' ownership and thread-safety
9797
9898
The Level-Zero runtime doesn't do reference-counting of its objects, so it is crucial to adhere to these
99-
practices of how Level-Zero handles are manged.
99+
practices of how Level-Zero handles are managed. By default, the ownership is transferred to the SYCL runtime, but
100+
some interoparability API supports overriding this behavior and keep the ownership in the application.
101+
Use this enumeration for explicit specification of the ownership:
102+
``` C++
103+
namespace sycl {
104+
namespace level_zero {
105+
106+
enum class ownership { transfer, keep };
107+
108+
} // namesace level_zero
109+
} // namespace sycl
110+
```
100111
101-
#### 4.4.1 SYCL runtime takes ownership
112+
#### 4.4.1 SYCL runtime takes ownership (default)
102113
103114
Whenever the application creates a SYCL object from the corresponding Level-Zero handle via one of the ```make<T>()``` functions,
104-
the SYCL runtime takes ownership of the Level-Zero handle. The application must not use the Level-Zero handle after
105-
the last host copy of the SYCL object is destroyed (as described in the core SYCL specification under
106-
"Common reference semantics"), and the application must not destroy the Level-Zero handle itself.
115+
the SYCL runtime takes ownership of the Level-Zero handle, if no explicit ```ownership::keep``` was specified.
116+
The application must not use the Level-Zero handle after the last host copy of the SYCL object is destroyed (
117+
as described in the core SYCL specification under "Common reference semantics"), and the application must not
118+
destroy the Level-Zero handle itself.
119+
120+
#### 4.4.2 Application keeps ownership (explicit)
121+
122+
If SYCL object is created with an interoperability API explicitly asking to keep the native handle ownership in the application with
123+
```ownership::keep``` then the SYCL runtime does not take the ownership and will not destroy the Level-Zero handle at the destruction of the SYCL object.
124+
The application is responsible for destroying the native handle when it no longer needs it, but it must not destroy the
125+
handle before the last host copy of the SYCL object is destroyed (as described in the core SYCL specification under
126+
"Common reference semantics").
107127

108-
#### 4.4.2 SYCL runtime assumes ownership
128+
#### 4.4.3 Obtaining native handle does not change ownership
109129

110-
The application may call the ```get_native<T>()``` member function of a SYCL object to retrieve the underlying Level-Zero handle,
111-
however, the SYCL runtime continues to retain ownership of this handle. The application must not use this handle after
112-
the last host copy of the SYCL object is destroyed (as described in the core SYCL specification under
113-
"Common reference semantics"), and the application must not destroy the Level-Zero handle.
130+
The application may call the ```get_native<T>()``` member function of a SYCL object to retrieve the underlying Level-Zero handle.
131+
Doing so does not change the ownership of the the Level-Zero handle. Therefore, the application may not use this
132+
handle after the last host copy of the SYCL object is destroyed (as described in the core SYCL specification under
133+
"Common reference semantics") unless the SYCL object was created by the application with ```ownership::keep```.
114134

115-
#### 4.4.3 Considerations for multi-threaded environment
135+
#### 4.4.4 Considerations for multi-threaded environment
116136

117137
The Level-Zero API is not thread-safe, refer to <https://spec.oneapi.com/level-zero/latest/core/INTRO.html#multithreading-and-concurrency>.
118138
Applications must make sure that the Level-Zero handles themselves aren't used simultaneously from different threads.
@@ -123,4 +143,5 @@ the application should not attempt further direct use of those handles.
123143
|Rev|Date|Author|Changes|
124144
|-------------|:------------|:------------|:------------|
125145
|1|2021-01-26|Sergey Maslov|Initial public working draft
146+
|2|2021-02-22|Sergey Maslov|Introduced explicit ownership for context
126147

sycl/include/CL/sycl/backend/level_zero.hpp

Lines changed: 18 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -51,12 +51,20 @@ struct interop<backend::level_zero, accessor<DataT, Dimensions, AccessMode,
5151

5252
namespace level_zero {
5353

54-
// Implementation of various "make" functions resides in libsycl.so
54+
// Since Level-Zero is not doing any reference counting itself, we have to
55+
// be explicit about the ownership of the native handles used in the
56+
// interop functions below.
57+
//
58+
enum class ownership { transfer, keep };
59+
60+
// Implementation of various "make" functions resides in libsycl.so and thus
61+
// their interface needs to be backend agnostic.
5562
__SYCL_EXPORT platform make_platform(pi_native_handle NativeHandle);
5663
__SYCL_EXPORT device make_device(const platform &Platform,
5764
pi_native_handle NativeHandle);
5865
__SYCL_EXPORT context make_context(const vector_class<device> &DeviceList,
59-
pi_native_handle NativeHandle);
66+
pi_native_handle NativeHandle,
67+
bool keep_ownership = false);
6068
__SYCL_EXPORT program make_program(const context &Context,
6169
pi_native_handle NativeHandle);
6270
__SYCL_EXPORT queue make_queue(const context &Context,
@@ -82,11 +90,17 @@ T make(const platform &Platform,
8290
/// created SYCL context. Provided devices and native context handle must
8391
/// be associated with the same platform.
8492
/// \param Interop is a Level Zero native context handle.
93+
/// \param Ownership (optional) specifies who will assume ownership of the
94+
/// native context handle. Default is that SYCL RT does, so it destroys
95+
/// the native handle when the created SYCL object goes out of life.
96+
///
8597
template <typename T, typename std::enable_if<
8698
std::is_same<T, context>::value>::type * = nullptr>
8799
T make(const vector_class<device> &DeviceList,
88-
typename interop<backend::level_zero, T>::type Interop) {
89-
return make_context(DeviceList, detail::pi::cast<pi_native_handle>(Interop));
100+
typename interop<backend::level_zero, T>::type Interop,
101+
ownership Ownership = ownership::transfer) {
102+
return make_context(DeviceList, detail::pi::cast<pi_native_handle>(Interop),
103+
Ownership == ownership::keep);
90104
}
91105

92106
// Construction of SYCL program.

sycl/include/CL/sycl/detail/pi.h

Lines changed: 6 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -34,9 +34,10 @@
3434
// pi_device_binary_property_set PropertySetsBegin;
3535
// pi_device_binary_property_set PropertySetsEnd;
3636
// 2. A number of types needed to define pi_device_binary_property_set added.
37+
// 3. Added new ownership argument to piextContextCreateWithNativeHandle.
3738
//
38-
#define _PI_H_VERSION_MAJOR 2
39-
#define _PI_H_VERSION_MINOR 3
39+
#define _PI_H_VERSION_MAJOR 3
40+
#define _PI_H_VERSION_MINOR 4
4041

4142
#define _PI_STRING_HELPER(a) #a
4243
#define _PI_CONCAT(a, b) _PI_STRING_HELPER(a.b)
@@ -983,6 +984,8 @@ piextContextGetNativeHandle(pi_context context, pi_native_handle *nativeHandle);
983984
/// \param devices is the list of devices in the context. Parameter is ignored
984985
/// if devices can be queried from the context native handle for a
985986
/// backend.
987+
/// \param ownNativeHandle tells if SYCL RT should assume the ownership of
988+
/// the native handle, if it can.
986989
/// \param context is the PI context created from the native handle.
987990
/// \return PI_SUCCESS if successfully created pi_context from the handle.
988991
/// PI_OUT_OF_HOST_MEMORY if can't allocate memory for the pi_context
@@ -991,7 +994,7 @@ piextContextGetNativeHandle(pi_context context, pi_native_handle *nativeHandle);
991994
/// native handle. PI_UNKNOWN_ERROR in case of another error.
992995
__SYCL_EXPORT pi_result piextContextCreateWithNativeHandle(
993996
pi_native_handle nativeHandle, pi_uint32 numDevices,
994-
const pi_device *devices, pi_context *context);
997+
const pi_device *devices, bool ownNativeHandle, pi_context *context);
995998

996999
//
9971000
// Queue

sycl/plugins/cuda/pi_cuda.cpp

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1698,6 +1698,7 @@ pi_result cuda_piextContextGetNativeHandle(pi_context context,
16981698
pi_result cuda_piextContextCreateWithNativeHandle(pi_native_handle nativeHandle,
16991699
pi_uint32 num_devices,
17001700
const pi_device *devices,
1701+
bool ownNativeHandle,
17011702
pi_context *context) {
17021703
cl::sycl::detail::pi::die(
17031704
"Creation of PI context from native handle not implemented");

sycl/plugins/level_zero/pi_level_zero.cpp

Lines changed: 7 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -1954,7 +1954,7 @@ pi_result piContextCreate(const pi_context_properties *Properties,
19541954
ZE_CALL(zeContextCreate((*Devices)->Platform->ZeDriver, &ContextDesc,
19551955
&ZeContext));
19561956
try {
1957-
*RetContext = new _pi_context(ZeContext, NumDevices, Devices);
1957+
*RetContext = new _pi_context(ZeContext, NumDevices, Devices, true);
19581958
(*RetContext)->initialize();
19591959
} catch (const std::bad_alloc &) {
19601960
return PI_OUT_OF_HOST_MEMORY;
@@ -2013,6 +2013,7 @@ pi_result piextContextGetNativeHandle(pi_context Context,
20132013
pi_result piextContextCreateWithNativeHandle(pi_native_handle NativeHandle,
20142014
pi_uint32 NumDevices,
20152015
const pi_device *Devices,
2016+
bool OwnNativeHandle,
20162017
pi_context *RetContext) {
20172018
PI_ASSERT(NativeHandle, PI_INVALID_VALUE);
20182019
PI_ASSERT(Devices, PI_INVALID_DEVICE);
@@ -2021,7 +2022,7 @@ pi_result piextContextCreateWithNativeHandle(pi_native_handle NativeHandle,
20212022

20222023
try {
20232024
*RetContext = new _pi_context(pi_cast<ze_context_handle_t>(NativeHandle),
2024-
NumDevices, Devices);
2025+
NumDevices, Devices, OwnNativeHandle);
20252026
(*RetContext)->initialize();
20262027
} catch (const std::bad_alloc &) {
20272028
return PI_OUT_OF_HOST_MEMORY;
@@ -2045,7 +2046,8 @@ pi_result piContextRelease(pi_context Context) {
20452046
PI_ASSERT(Context, PI_INVALID_CONTEXT);
20462047

20472048
if (--(Context->RefCount) == 0) {
2048-
auto ZeContext = Context->ZeContext;
2049+
ze_context_handle_t DestoryZeContext =
2050+
Context->OwnZeContext ? Context->ZeContext : nullptr;
20492051

20502052
// Clean up any live memory associated with Context
20512053
pi_result Result = Context->finalize();
@@ -2059,7 +2061,8 @@ pi_result piContextRelease(pi_context Context) {
20592061
// and therefore it must be valid at that point.
20602062
// Technically it should be placed to the destructor of pi_context
20612063
// but this makes API error handling more complex.
2062-
ZE_CALL(zeContextDestroy(ZeContext));
2064+
if (DestoryZeContext)
2065+
ZE_CALL(zeContextDestroy(DestoryZeContext));
20632066

20642067
return Result;
20652068
}

sycl/plugins/level_zero/pi_level_zero.hpp

Lines changed: 7 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -171,8 +171,9 @@ struct _pi_device : _pi_object {
171171

172172
struct _pi_context : _pi_object {
173173
_pi_context(ze_context_handle_t ZeContext, pi_uint32 NumDevices,
174-
const pi_device *Devs)
175-
: ZeContext{ZeContext}, Devices{Devs, Devs + NumDevices},
174+
const pi_device *Devs, bool OwnZeContext)
175+
: ZeContext{ZeContext},
176+
OwnZeContext{OwnZeContext}, Devices{Devs, Devs + NumDevices},
176177
ZeCommandListInit{nullptr}, ZeEventPool{nullptr},
177178
NumEventsAvailableInEventPool{}, NumEventsLiveInEventPool{} {
178179
// Create USM allocator context for each pair (device, context).
@@ -201,6 +202,10 @@ struct _pi_context : _pi_object {
201202
// resources that may be used by multiple devices.
202203
ze_context_handle_t ZeContext;
203204

205+
// Indicates if we own the ZeContext or it came from interop that
206+
// asked to not transfer the ownership to SYCL RT.
207+
bool OwnZeContext;
208+
204209
// Keep the PI devices this PI context was created for.
205210
std::vector<pi_device> Devices;
206211

sycl/plugins/opencl/pi_opencl.cpp

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -536,10 +536,12 @@ pi_result piContextCreate(const pi_context_properties *properties,
536536
pi_result piextContextCreateWithNativeHandle(pi_native_handle nativeHandle,
537537
pi_uint32 num_devices,
538538
const pi_device *devices,
539+
bool ownNativeHandle,
539540
pi_context *piContext) {
540541
(void)num_devices;
541542
(void)devices;
542543
assert(piContext != nullptr);
544+
assert(ownNativeHandle == false);
543545
*piContext = reinterpret_cast<pi_context>(nativeHandle);
544546
return PI_SUCCESS;
545547
}

sycl/source/backend/level_zero.cpp

Lines changed: 10 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -48,7 +48,8 @@ __SYCL_EXPORT device make_device(const platform &Platform,
4848
//----------------------------------------------------------------------------
4949
// Implementation of level_zero::make<context>
5050
__SYCL_EXPORT context make_context(const vector_class<device> &DeviceList,
51-
pi_native_handle NativeHandle) {
51+
pi_native_handle NativeHandle,
52+
bool KeepOwnership) {
5253
const auto &Plugin = pi::getPlugin<backend::level_zero>();
5354
// Create PI context first.
5455
pi_context PiContext;
@@ -57,12 +58,19 @@ __SYCL_EXPORT context make_context(const vector_class<device> &DeviceList,
5758
DeviceHandles.push_back(detail::getSyclObjImpl(Dev)->getHandleRef());
5859
}
5960
Plugin.call<PiApiKind::piextContextCreateWithNativeHandle>(
60-
NativeHandle, DeviceHandles.size(), DeviceHandles.data(), &PiContext);
61+
NativeHandle, DeviceHandles.size(), DeviceHandles.data(), !KeepOwnership,
62+
&PiContext);
6163
// Construct the SYCL context from PI context.
6264
return detail::createSyclObjFromImpl<context>(
6365
std::make_shared<context_impl>(PiContext, async_handler{}, Plugin));
6466
}
6567

68+
// TODO: remove this version (without ownership) when allowed to break ABI.
69+
__SYCL_EXPORT context make_context(const vector_class<device> &DeviceList,
70+
pi_native_handle NativeHandle) {
71+
return make_context(DeviceList, NativeHandle, false);
72+
}
73+
6674
//----------------------------------------------------------------------------
6775
// Implementation of level_zero::make<program>
6876
__SYCL_EXPORT program make_program(const context &Context,

sycl/source/backend/opencl.cpp

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -51,7 +51,7 @@ __SYCL_EXPORT context make_context(pi_native_handle NativeHandle) {
5151
// Create PI context first.
5252
pi::PiContext PiContext;
5353
Plugin.call<PiApiKind::piextContextCreateWithNativeHandle>(
54-
NativeHandle, 0, nullptr, &PiContext);
54+
NativeHandle, 0, nullptr, false, &PiContext);
5555
// Construct the SYCL context from PI context.
5656
return detail::createSyclObjFromImpl<context>(
5757
std::make_shared<context_impl>(PiContext, async_handler{}, Plugin));

sycl/source/detail/context_impl.cpp

Lines changed: 10 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -56,7 +56,8 @@ context_impl::context_impl(const vector_class<cl::sycl::device> Devices,
5656
getPlugin().call<PiApiKind::piContextCreate>(
5757
Props, DeviceIds.size(), DeviceIds.data(), nullptr, nullptr, &MContext);
5858
#else
59-
cl::sycl::detail::pi::die("CUDA support was not enabled at compilation time");
59+
cl::sycl::detail::pi::die(
60+
"CUDA support was not enabled at compilation time");
6061
#endif
6162
} else {
6263
getPlugin().call<PiApiKind::piContextCreate>(nullptr, DeviceIds.size(),
@@ -96,7 +97,12 @@ context_impl::context_impl(RT::PiContext PiContext, async_handler AsyncHandler,
9697
// TODO catch an exception and put it to list of asynchronous exceptions
9798
// getPlugin() will be the same as the Plugin passed. This should be taken
9899
// care of when creating device object.
99-
getPlugin().call<PiApiKind::piContextRetain>(MContext);
100+
//
101+
// TODO: Move this backend-specific retain of the context to SYCL-2020 style
102+
// make_context<backend::opencl> interop, when that is created.
103+
if (getPlugin().getBackend() == cl::sycl::backend::opencl) {
104+
getPlugin().call<PiApiKind::piContextRetain>(MContext);
105+
}
100106
MKernelProgramCache.setContextPtr(this);
101107
}
102108

@@ -153,8 +159,8 @@ KernelProgramCache &context_impl::getKernelProgramCache() const {
153159
return MKernelProgramCache;
154160
}
155161

156-
bool
157-
context_impl::hasDevice(shared_ptr_class<detail::device_impl> Device) const {
162+
bool context_impl::hasDevice(
163+
shared_ptr_class<detail::device_impl> Device) const {
158164
for (auto D : MDevices)
159165
if (getSyclObjImpl(D) == Device)
160166
return true;

0 commit comments

Comments
 (0)