[SYCL][Level Zero] Enable multi-CCS support. #4038

rbegam · 2021-07-01T23:48:43Z

Signed-off-by: rehana begam [email protected]

smaslov-intel · 2021-07-06T23:47:02Z

@rbegam , please fix clang-format.
Did you already created a test?

sycl/plugins/level_zero/pi_level_zero.cpp

rbegam · 2021-07-09T02:19:52Z

ping @smaslov-intel @jandres742

sycl/plugins/level_zero/pi_level_zero.cpp

sycl/plugins/level_zero/pi_level_zero.hpp

sycl/plugins/level_zero/pi_level_zero.cpp

smaslov-intel · 2021-07-16T00:27:57Z

sycl/plugins/level_zero/pi_level_zero.cpp

+      pi_result Res = Device->Platform->populateDeviceCacheIfNeeded();
+      if (Res != PI_SUCCESS) {
+        return Res;
+      }
+
+      return ReturnValue(pi_uint32{(unsigned int)(Device->SubDevices.size())});


I think we can run this exact code for root-devices as well

smaslov-intel · 2021-07-16T00:30:35Z

sycl/plugins/level_zero/pi_level_zero.cpp

-    ZE_CALL(zeDeviceGetSubDevices, (ZeDevice, &ZeSubDeviceCount, nullptr));
-    if (ZeSubDeviceCount < 2) {
-      return ReturnValue(pi_device_partition_property{0});
+    if (!Device->isSubDevice()) {


You are going to return non-zero vector for sub-devices, why is it OK? Also please change the code to get # of sub-devices from the cache (instead of repeating call to zeDeviceGetSubDevices)

Signed-off-by: rehana begam <[email protected]>

smaslov-intel · 2021-07-19T23:22:23Z

sycl/plugins/level_zero/pi_level_zero.cpp

+    const char *CopyEngine = std::getenv("SYCL_PI_LEVEL_ZERO_USE_COPY_ENGINE");
+    bool UseCopyEngine = (!CopyEngine || (std::stoi(CopyEngine) != 0));


Please rewrite it similar to zeSerialize so that it is only executed once

smaslov-intel · 2021-07-19T23:23:48Z

sycl/plugins/level_zero/pi_level_zero.cpp

+      return PI_ERROR_UNKNOWN;
+    }
+
+    ZeComputeEngineIndex = 0;


This deserves a comment why we are using "0" and how it scales to HW

smaslov-intel · 2021-07-19T23:27:33Z

sycl/plugins/level_zero/pi_level_zero.cpp

+        }
+
+        if (Ordinals.empty()) {
+          return PI_ERROR_UNKNOWN;


There are no sub-sub-devices, why error?

right, it should not be an error. this code is the reason for the pre-commit fail. I've fixed it, will upload soon.

Signed-off-by: rehana begam <[email protected]>

rbegam · 2021-07-20T00:55:41Z

ping @smaslov-intel @jandres742

Signed-off-by: rehana begam <[email protected]>

jandres742 · 2021-07-20T01:00:12Z

LGTM

…ackend_plugin * upstream/sycl: (26 commits) [SPIR-V][NFC] Move non-upstreamed FuncParam decorations into internal:: (intel#4138) [SYCL] Move free function queries to experimental namespace (intel#4090) [SYCL][XPTI] Enable PI calls notifications with arguments (intel#4148) [SYCL] Revert queue::wait() to its old behaviour with Level Zero (intel#4153) [SYCL] Add missing <cstring> header to spirv.hpp (intel#4157) [SYCL] Adds info query for atomic_memory_order_capabilities on device and context (intel#4105) [SYCL] Improve performance of generic shuffles (intel#3815) [SYCL] Fix the error with namespaces caused during rebase of intel#4014 (intel#4151) [ESIMD] Fix 'ambiguous operator' error with length 1 simd operands (intel#4149) [libdevice][NFC] Fix libdevice dependencies list (intel#4130) [SPIR-V] Reland Encode debug info producer in SPIR-V (intel#4082) [SYCL][ROCm] Add ROCm support to get_device_count_by_type (intel#4113) [SYCL] Fix sRGB device info (intel#4145) [SYCL][ROCm] Fix kernel launch with multiple dimensions (intel#4063) [SYCL][ROCm] Fix compilation for AMD GPU with -fsycl-dead-args-optimization (intel#4126) [SYCL][Level Zero] Enable multi-CCS support. (intel#4038) [SYCL] Pass bound arch to unbundler (intel#4112) [ESIMD][doc] Added documentation for some ESIMD math APIs (intel#3995) [ESIMD] rename gather4/scatter4 to gather_rgba/scatter_rgba (intel#4120) [SYCL][NFC] Remove unused variable. (intel#4131) ...

rbegam requested a review from smaslov-intel as a code owner July 1, 2021 23:48

smaslov-intel suggested changes Jul 6, 2021

View reviewed changes

rbegam mentioned this pull request Jul 8, 2021

[LIT][L0] test for multi-CCS support in level_zero. intel/llvm-test-suite#345

Merged

jandres742 reviewed Jul 8, 2021

View reviewed changes

sycl/plugins/level_zero/pi_level_zero.cpp Outdated Show resolved Hide resolved

jandres742 reviewed Jul 8, 2021

View reviewed changes

sycl/plugins/level_zero/pi_level_zero.cpp Outdated Show resolved Hide resolved

jandres742 reviewed Jul 9, 2021

View reviewed changes

sycl/plugins/level_zero/pi_level_zero.cpp Outdated Show resolved Hide resolved

jandres742 reviewed Jul 9, 2021

View reviewed changes

sycl/plugins/level_zero/pi_level_zero.cpp Show resolved Hide resolved

smaslov-intel changed the title ~~[DPC++] enable multi-context (c-slice) support.~~ [DPC++] enable multi-CCS support. Jul 9, 2021

smaslov-intel suggested changes Jul 9, 2021

View reviewed changes

sycl/plugins/level_zero/pi_level_zero.hpp Outdated Show resolved Hide resolved

sycl/plugins/level_zero/pi_level_zero.cpp Outdated Show resolved Hide resolved

sycl/plugins/level_zero/pi_level_zero.cpp Show resolved Hide resolved

sycl/plugins/level_zero/pi_level_zero.cpp Show resolved Hide resolved

rbegam requested a review from a team as a code owner July 15, 2021 22:31

smaslov-intel reviewed Jul 16, 2021

View reviewed changes

rbegam force-pushed the c-slice branch from b19efe5 to 5cfc674 Compare July 17, 2021 04:35

rbegam added 15 commits July 19, 2021 11:32

[DPC++] enable multi-context (c-slice) support.

f3c884d

Signed-off-by: rehana begam <[email protected]>

fix sub sub device count.

d5f4387

Signed-off-by: rehana begam <[email protected]>

remove SubSubDevicesCount.

607746e

Signed-off-by: rehana begam <[email protected]>

clang formatted.

beb8763

Signed-off-by: rehana begam <[email protected]>

clang format.

e5b85fb

Signed-off-by: rehana begam <[email protected]>

inlining.

00aef90

Signed-off-by: rehana begam <[email protected]>

clang formatted.

89fdf06

Signed-off-by: rehana begam <[email protected]>

update pi_level_zero.cpp.

d388a8b

Signed-off-by: rehana begam <[email protected]>

clang formatted.

485ecaf

Signed-off-by: rehana begam <[email protected]>

update max sub-device count.

fa22bcd

Signed-off-by: rehana begam <[email protected]>

add comments.

6ff74c2

Signed-off-by: rehana begam <[email protected]>

add index.

0d9dfa6

Signed-off-by: rehana begam <[email protected]>

update context initialization.

a52ca46

Signed-off-by: rehana begam <[email protected]>

clang formatted.

bf50ccd

Signed-off-by: rehana begam <[email protected]>

add comments.

33c4449

Signed-off-by: rehana begam <[email protected]>

bader requested a review from smaslov-intel July 19, 2021 18:33

smaslov-intel suggested changes Jul 19, 2021

View reviewed changes

fix pre-commit fail.

306cc9f

Signed-off-by: rehana begam <[email protected]>

rbegam force-pushed the c-slice branch from 5875765 to 306cc9f Compare July 20, 2021 00:34

update comments.

37b114e

Signed-off-by: rehana begam <[email protected]>

clang formatted.

3527982

Signed-off-by: rehana begam <[email protected]>

smaslov-intel approved these changes Jul 20, 2021

View reviewed changes

bader changed the title ~~[DPC++] enable multi-CCS support.~~ [SYCL][Level Zero] Enable multi-CCS support. Jul 20, 2021

bader merged commit 9bc3bc4 into intel:sycl Jul 20, 2021

		const char *CopyEngine = std::getenv("SYCL_PI_LEVEL_ZERO_USE_COPY_ENGINE");
		bool UseCopyEngine = (!CopyEngine \|\| (std::stoi(CopyEngine) != 0));

[SYCL][Level Zero] Enable multi-CCS support. #4038

[SYCL][Level Zero] Enable multi-CCS support. #4038

Uh oh!

Conversation

rbegam commented Jul 1, 2021

Uh oh!

smaslov-intel commented Jul 6, 2021

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

rbegam commented Jul 9, 2021

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

smaslov-intel Jul 16, 2021

Choose a reason for hiding this comment

Uh oh!

smaslov-intel Jul 16, 2021

Choose a reason for hiding this comment

Uh oh!

smaslov-intel Jul 19, 2021

Choose a reason for hiding this comment

Uh oh!

smaslov-intel Jul 19, 2021

Choose a reason for hiding this comment

Uh oh!

smaslov-intel Jul 19, 2021

Choose a reason for hiding this comment

Uh oh!

rbegam Jul 19, 2021

Choose a reason for hiding this comment

Uh oh!

rbegam commented Jul 20, 2021

Uh oh!

jandres742 commented Jul 20, 2021

Uh oh!

Uh oh!