Skip to content

[SYCL] Support USM buffer location property in malloc_host #6220

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 3 commits into from
Jun 8, 2022
Merged

[SYCL] Support USM buffer location property in malloc_host #6220

merged 3 commits into from
Jun 8, 2022

Conversation

pcolberg
Copy link
Contributor

@pcolberg pcolberg commented May 31, 2022

See malloc_shared implementation in #6218

See extension specification in #6219

@tiwaria1 @bsyrowik @GarveyJoe @aditikum @ajaykumarkannan

@pcolberg pcolberg requested a review from a team as a code owner May 31, 2022 21:27
@pcolberg pcolberg requested a review from smaslov-intel May 31, 2022 21:27
@pcolberg pcolberg marked this pull request as draft May 31, 2022 21:30
@pcolberg pcolberg added the enhancement New feature or request label May 31, 2022
@pcolberg pcolberg requested a review from cperkinsintel May 31, 2022 22:01
@pcolberg pcolberg marked this pull request as ready for review May 31, 2022 22:02
@pcolberg
Copy link
Contributor Author

pcolberg commented Jun 1, 2022

8b86af77b872c58b0c68e53b0495350c5bc4dd9c passes HIP but still fails L0:

2022-06-01T02:12:48.5256663Z FAIL: SYCL :: USM/usm_pooling.cpp (897 of 899)
2022-06-01T02:12:48.5257504Z ******************** TEST 'SYCL :: USM/usm_pooling.cpp' FAILED ********************
2022-06-01T02:12:48.5258681Z Script:
2022-06-01T02:12:48.5258949Z --
2022-06-01T02:12:48.5259736Z : 'RUN: at line 2';    /__w/llvm/llvm/toolchain/bin/clang++      -fsycl -fsycl-targets=spir64 /__w/llvm/llvm/llvm_test_suite/SYCL/USM/usm_pooling.cpp -o /__w/llvm/llvm/build/SYCL/USM/Output/usm_pooling.cpp.tmp.out
2022-06-01T02:12:48.5263032Z : 'RUN: at line 7';   env ZE_DEBUG=1  env SYCL_DEVICE_FILTER=ext_oneapi_level_zero:gpu,host  SYCL_PI_LEVEL_ZERO_DISABLE_USM_ALLOCATOR=1 /__w/llvm/llvm/build/SYCL/USM/Output/usm_pooling.cpp.tmp.out h 2> /__w/llvm/llvm/build/SYCL/USM/Output/usm_pooling.cpp.tmp1.out; cat /__w/llvm/llvm/build/SYCL/USM/Output/usm_pooling.cpp.tmp1.out | FileCheck /__w/llvm/llvm/llvm_test_suite/SYCL/USM/usm_pooling.cpp --check-prefix CHECK-NOPOOL
2022-06-01T02:12:48.5264380Z : 'RUN: at line 8';   env ZE_DEBUG=1  env SYCL_DEVICE_FILTER=ext_oneapi_level_zero:gpu,host  SYCL_PI_LEVEL_ZERO_DISABLE_USM_ALLOCATOR=1 /__w/llvm/llvm/build/SYCL/USM/Output/usm_pooling.cpp.tmp.out d 2> /__w/llvm/llvm/build/SYCL/USM/Output/usm_pooling.cpp.tmp1.out; cat /__w/llvm/llvm/build/SYCL/USM/Output/usm_pooling.cpp.tmp1.out | FileCheck /__w/llvm/llvm/llvm_test_suite/SYCL/USM/usm_pooling.cpp --check-prefix CHECK-NOPOOL
2022-06-01T02:12:48.5266056Z : 'RUN: at line 9';   env ZE_DEBUG=1  env SYCL_DEVICE_FILTER=ext_oneapi_level_zero:gpu,host  SYCL_PI_LEVEL_ZERO_DISABLE_USM_ALLOCATOR=1 /__w/llvm/llvm/build/SYCL/USM/Output/usm_pooling.cpp.tmp.out s 2> /__w/llvm/llvm/build/SYCL/USM/Output/usm_pooling.cpp.tmp1.out; cat /__w/llvm/llvm/build/SYCL/USM/Output/usm_pooling.cpp.tmp1.out | FileCheck /__w/llvm/llvm/llvm_test_suite/SYCL/USM/usm_pooling.cpp --check-prefix CHECK-NOPOOL
2022-06-01T02:12:48.5267311Z : 'RUN: at line 12';   env ZE_DEBUG=1  env SYCL_DEVICE_FILTER=ext_oneapi_level_zero:gpu,host  SYCL_PI_LEVEL_ZERO_USM_ALLOCATOR=";;1M,4,64K" /__w/llvm/llvm/build/SYCL/USM/Output/usm_pooling.cpp.tmp.out h 2> /__w/llvm/llvm/build/SYCL/USM/Output/usm_pooling.cpp.tmp1.out; cat /__w/llvm/llvm/build/SYCL/USM/Output/usm_pooling.cpp.tmp1.out | FileCheck /__w/llvm/llvm/llvm_test_suite/SYCL/USM/usm_pooling.cpp --check-prefix CHECK-12345
2022-06-01T02:12:48.5268802Z : 'RUN: at line 13';   env ZE_DEBUG=1  env SYCL_DEVICE_FILTER=ext_oneapi_level_zero:gpu,host  SYCL_PI_LEVEL_ZERO_USM_ALLOCATOR=";;1M,4,64K" /__w/llvm/llvm/build/SYCL/USM/Output/usm_pooling.cpp.tmp.out d 2> /__w/llvm/llvm/build/SYCL/USM/Output/usm_pooling.cpp.tmp1.out; cat /__w/llvm/llvm/build/SYCL/USM/Output/usm_pooling.cpp.tmp1.out | FileCheck /__w/llvm/llvm/llvm_test_suite/SYCL/USM/usm_pooling.cpp --check-prefix CHECK-12345
2022-06-01T02:12:48.5270410Z : 'RUN: at line 14';   env ZE_DEBUG=1  env SYCL_DEVICE_FILTER=ext_oneapi_level_zero:gpu,host  SYCL_PI_LEVEL_ZERO_USM_ALLOCATOR=";;1M,4,64K" /__w/llvm/llvm/build/SYCL/USM/Output/usm_pooling.cpp.tmp.out s 2> /__w/llvm/llvm/build/SYCL/USM/Output/usm_pooling.cpp.tmp1.out; cat /__w/llvm/llvm/build/SYCL/USM/Output/usm_pooling.cpp.tmp1.out | FileCheck /__w/llvm/llvm/llvm_test_suite/SYCL/USM/usm_pooling.cpp --check-prefix CHECK-12345
2022-06-01T02:12:48.5271935Z : 'RUN: at line 17';   env ZE_DEBUG=1  env SYCL_DEVICE_FILTER=ext_oneapi_level_zero:gpu,host  SYCL_PI_LEVEL_ZERO_USM_ALLOCATOR=";;2M,1,64K" /__w/llvm/llvm/build/SYCL/USM/Output/usm_pooling.cpp.tmp.out h 2> /__w/llvm/llvm/build/SYCL/USM/Output/usm_pooling.cpp.tmp1.out; cat /__w/llvm/llvm/build/SYCL/USM/Output/usm_pooling.cpp.tmp1.out | FileCheck /__w/llvm/llvm/llvm_test_suite/SYCL/USM/usm_pooling.cpp --check-prefix CHECK-1245
2022-06-01T02:12:48.5273868Z : 'RUN: at line 18';   env ZE_DEBUG=1  env SYCL_DEVICE_FILTER=ext_oneapi_level_zero:gpu,host  SYCL_PI_LEVEL_ZERO_USM_ALLOCATOR=";;2M,1,64K" /__w/llvm/llvm/build/SYCL/USM/Output/usm_pooling.cpp.tmp.out d 2> /__w/llvm/llvm/build/SYCL/USM/Output/usm_pooling.cpp.tmp1.out; cat /__w/llvm/llvm/build/SYCL/USM/Output/usm_pooling.cpp.tmp1.out | FileCheck /__w/llvm/llvm/llvm_test_suite/SYCL/USM/usm_pooling.cpp --check-prefix CHECK-1245
2022-06-01T02:12:48.5275138Z : 'RUN: at line 19';   env ZE_DEBUG=1  env SYCL_DEVICE_FILTER=ext_oneapi_level_zero:gpu,host  SYCL_PI_LEVEL_ZERO_USM_ALLOCATOR=";;2M,1,64K" /__w/llvm/llvm/build/SYCL/USM/Output/usm_pooling.cpp.tmp.out s 2> /__w/llvm/llvm/build/SYCL/USM/Output/usm_pooling.cpp.tmp1.out; cat /__w/llvm/llvm/build/SYCL/USM/Output/usm_pooling.cpp.tmp1.out | FileCheck /__w/llvm/llvm/llvm_test_suite/SYCL/USM/usm_pooling.cpp --check-prefix CHECK-1245
2022-06-01T02:12:48.5276432Z : 'RUN: at line 22';   env ZE_DEBUG=1  env SYCL_DEVICE_FILTER=ext_oneapi_level_zero:gpu,host  SYCL_PI_LEVEL_ZERO_USM_ALLOCATOR=";2M;2M,4,64K" /__w/llvm/llvm/build/SYCL/USM/Output/usm_pooling.cpp.tmp.out h 2> /__w/llvm/llvm/build/SYCL/USM/Output/usm_pooling.cpp.tmp1.out; cat /__w/llvm/llvm/build/SYCL/USM/Output/usm_pooling.cpp.tmp1.out | FileCheck /__w/llvm/llvm/llvm_test_suite/SYCL/USM/usm_pooling.cpp --check-prefix CHECK-1245
2022-06-01T02:12:48.5277665Z : 'RUN: at line 23';   env ZE_DEBUG=1  env SYCL_DEVICE_FILTER=ext_oneapi_level_zero:gpu,host  SYCL_PI_LEVEL_ZERO_USM_ALLOCATOR=";2M;2M,4,64K" /__w/llvm/llvm/build/SYCL/USM/Output/usm_pooling.cpp.tmp.out d 2> /__w/llvm/llvm/build/SYCL/USM/Output/usm_pooling.cpp.tmp1.out; cat /__w/llvm/llvm/build/SYCL/USM/Output/usm_pooling.cpp.tmp1.out | FileCheck /__w/llvm/llvm/llvm_test_suite/SYCL/USM/usm_pooling.cpp --check-prefix CHECK-1245
2022-06-01T02:12:48.5278945Z : 'RUN: at line 24';   env ZE_DEBUG=1  env SYCL_DEVICE_FILTER=ext_oneapi_level_zero:gpu,host  SYCL_PI_LEVEL_ZERO_USM_ALLOCATOR=";2M;2M,4,64K" /__w/llvm/llvm/build/SYCL/USM/Output/usm_pooling.cpp.tmp.out s 2> /__w/llvm/llvm/build/SYCL/USM/Output/usm_pooling.cpp.tmp1.out; cat /__w/llvm/llvm/build/SYCL/USM/Output/usm_pooling.cpp.tmp1.out | FileCheck /__w/llvm/llvm/llvm_test_suite/SYCL/USM/usm_pooling.cpp --check-prefix CHECK-1245
2022-06-01T02:12:48.5280163Z : 'RUN: at line 27';   env ZE_DEBUG=1  env SYCL_DEVICE_FILTER=ext_oneapi_level_zero:gpu,host  SYCL_PI_LEVEL_ZERO_USM_ALLOCATOR=";;2M,4,4M" /__w/llvm/llvm/build/SYCL/USM/Output/usm_pooling.cpp.tmp.out h 2> /__w/llvm/llvm/build/SYCL/USM/Output/usm_pooling.cpp.tmp1.out; cat /__w/llvm/llvm/build/SYCL/USM/Output/usm_pooling.cpp.tmp1.out | FileCheck /__w/llvm/llvm/llvm_test_suite/SYCL/USM/usm_pooling.cpp --check-prefix CHECK-15
2022-06-01T02:12:48.5281384Z : 'RUN: at line 28';   env ZE_DEBUG=1  env SYCL_DEVICE_FILTER=ext_oneapi_level_zero:gpu,host  SYCL_PI_LEVEL_ZERO_USM_ALLOCATOR=";;2M,4,4M" /__w/llvm/llvm/build/SYCL/USM/Output/usm_pooling.cpp.tmp.out d 2> /__w/llvm/llvm/build/SYCL/USM/Output/usm_pooling.cpp.tmp1.out; cat /__w/llvm/llvm/build/SYCL/USM/Output/usm_pooling.cpp.tmp1.out | FileCheck /__w/llvm/llvm/llvm_test_suite/SYCL/USM/usm_pooling.cpp --check-prefix CHECK-15
2022-06-01T02:12:48.5282580Z : 'RUN: at line 29';   env ZE_DEBUG=1  env SYCL_DEVICE_FILTER=ext_oneapi_level_zero:gpu,host  SYCL_PI_LEVEL_ZERO_USM_ALLOCATOR=";;2M,4,4M" /__w/llvm/llvm/build/SYCL/USM/Output/usm_pooling.cpp.tmp.out s 2> /__w/llvm/llvm/build/SYCL/USM/Output/usm_pooling.cpp.tmp1.out; cat /__w/llvm/llvm/build/SYCL/USM/Output/usm_pooling.cpp.tmp1.out | FileCheck /__w/llvm/llvm/llvm_test_suite/SYCL/USM/usm_pooling.cpp --check-prefix CHECK-15
2022-06-01T02:12:48.5283110Z --
2022-06-01T02:12:48.5283247Z Exit Code: 1
2022-06-01T02:12:48.5283390Z 
2022-06-01T02:12:48.5283462Z Command Output (stdout):
2022-06-01T02:12:48.5283638Z --
2022-06-01T02:12:48.5283778Z $ ":" "RUN: at line 2"
2022-06-01T02:12:48.5283997Z note: command had no output on stdout or stderr
2022-06-01T02:12:48.5284539Z $ "/__w/llvm/llvm/toolchain/bin/clang++" "-fsycl" "-fsycl-targets=spir64" "/__w/llvm/llvm/llvm_test_suite/SYCL/USM/usm_pooling.cpp" "-o" "/__w/llvm/llvm/build/SYCL/USM/Output/usm_pooling.cpp.tmp.out"
2022-06-01T02:12:48.5284901Z note: command had no output on stdout or stderr
2022-06-01T02:12:48.5285088Z $ ":" "RUN: at line 7"
2022-06-01T02:12:48.5285279Z note: command had no output on stdout or stderr
2022-06-01T02:12:48.5285645Z $ "env" "ZE_DEBUG=1" "env" "SYCL_DEVICE_FILTER=ext_oneapi_level_zero:gpu,host" "SYCL_PI_LEVEL_ZERO_DISABLE_USM_ALLOCATOR=1" "/__w/llvm/llvm/build/SYCL/USM/Output/usm_pooling.cpp.tmp.out" "h"
2022-06-01T02:12:48.5285970Z # command output:
2022-06-01T02:12:48.5286288Z Running on device GPU (Intel(R) UHD Graphics [0x9bca]) Intel(R) Level-Zero plugin
2022-06-01T02:12:48.5286460Z 
2022-06-01T02:12:48.5286596Z $ "cat" "/__w/llvm/llvm/build/SYCL/USM/Output/usm_pooling.cpp.tmp1.out"
2022-06-01T02:12:48.5286971Z note: command had no output on stdout or stderr
2022-06-01T02:12:48.5287370Z $ "FileCheck" "/__w/llvm/llvm/llvm_test_suite/SYCL/USM/usm_pooling.cpp" "--check-prefix" "CHECK-NOPOOL"
2022-06-01T02:12:48.5287615Z # command stderr:
2022-06-01T02:12:48.5288018Z /__w/llvm/llvm/llvm_test_suite/SYCL/USM/usm_pooling.cpp:105:23: error: CHECK-NOPOOL-NEXT: is not on the line after the previous match
2022-06-01T02:12:48.5288369Z // CHECK-NOPOOL-NEXT: ZE ---> [[API]](
2022-06-01T02:12:48.5288546Z                       ^
2022-06-01T02:12:48.5288773Z <stdin>:43:1: note: 'next' match was here
2022-06-01T02:12:48.5289115Z ZE ---> zeMemAllocHost(Context->ZeContext, &ZeHostDesc, Size, Alignment, ResultPtr)
2022-06-01T02:12:48.5289407Z ^
2022-06-01T02:12:48.5289578Z <stdin>:36:20: note: previous match ended here
2022-06-01T02:12:48.5289771Z Test zeMemAllocHost
2022-06-01T02:12:48.5289924Z                    ^
2022-06-01T02:12:48.5290211Z <stdin>:37:1: note: non-matching line after previous match is here
2022-06-01T02:12:48.5290416Z ==========================
2022-06-01T02:12:48.5290560Z ^
2022-06-01T02:12:48.5290636Z 
2022-06-01T02:12:48.5290700Z Input file: <stdin>
2022-06-01T02:12:48.5290916Z Check file: /__w/llvm/llvm/llvm_test_suite/SYCL/USM/usm_pooling.cpp
2022-06-01T02:12:48.5291067Z 
2022-06-01T02:12:48.5291236Z -dump-input=help explains the following input dump.
2022-06-01T02:12:48.5291383Z 
2022-06-01T02:12:48.5291436Z Input was:
2022-06-01T02:12:48.5291574Z <<<<<<
2022-06-01T02:12:48.5291707Z           .
2022-06-01T02:12:48.5291840Z           .
2022-06-01T02:12:48.5291974Z           .
2022-06-01T02:12:48.5292201Z          38: SYCL over Level-Zero 1.3.22905 
2022-06-01T02:12:48.5292380Z          39: ========================== 
2022-06-01T02:12:48.5292547Z          40: ========================== 
2022-06-01T02:12:48.5292790Z          41: SYCL over Level-Zero 1.3.22905 
2022-06-01T02:12:48.5292966Z          42: ========================== 
2022-06-01T02:12:48.5293310Z          43: ZE ---> zeMemAllocHost(Context->ZeContext, &ZeHostDesc, Size, Alignment, ResultPtr) 
2022-06-01T02:12:48.5293615Z next:105     !~~~~~~~~~~~~~~~~~~~~~~                                                              error: match on wrong line
2022-06-01T02:12:48.5293822Z          44: ========================== 
2022-06-01T02:12:48.5294058Z          45: SYCL over Level-Zero 1.3.22905 
2022-06-01T02:12:48.5294238Z          46: ========================== 
2022-06-01T02:12:48.5294399Z          47: ========================== 
2022-06-01T02:12:48.5294631Z          48: SYCL over Level-Zero 1.3.22905 
2022-06-01T02:12:48.5294797Z           .
2022-06-01T02:12:48.5294934Z           .
2022-06-01T02:12:48.5295062Z           .
2022-06-01T02:12:48.5295200Z >>>>>>
2022-06-01T02:12:48.5295283Z 
2022-06-01T02:12:48.5295374Z error: command failed with exit status: 1
2022-06-01T02:12:48.5295556Z 
2022-06-01T02:12:48.5295630Z --
2022-06-01T02:12:48.5295714Z 
2022-06-01T02:12:48.5295769Z ********************

Comment on lines 74 to 95
const platform &Platform = Ctxt.get_platform();

std::vector<pi_usm_mem_properties> Props;

if (Platform.has_extension("cl_intel_mem_alloc_buffer_location") &&
PropList.has_property<cl::sycl::ext::intel::experimental::property::
usm::buffer_location>()) {
Props.emplace_back(PI_MEM_USM_ALLOC_BUFFER_LOCATION);
Props.emplace_back(
PropList
.get_property<cl::sycl::ext::intel::experimental::property::
usm::buffer_location>()
.get_buffer_location());
}

if (Props.empty()) {
// Explicitly pass null pointer since the OpenCL driver may not support
// empty property lists, i.e., containing only a null terminator.
Error = Plugin.call_nocheck<PiApiKind::piextUSMHostAlloc>(
&RetVal, C, nullptr, Size, Alignment);
} else {
Props.emplace_back(0); // null-terminate property list
Error = Plugin.call_nocheck<PiApiKind::piextUSMHostAlloc>(
&RetVal, C, Props.data(), Size, Alignment);
}

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

the spec https://github.com/sherry-yuan/llvm/blob/9301c51e755d118cee94fb2df391ccb3549d6a23/sycl/doc/extensions/proposed/sycl_ext_intel_runtime_buffer_location.asciidoc says:

this property has no effect when passed to sycl::malloc_shared() or sycl::malloc_host()

so why is this code not ignoring it?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The spec is being updated to not say that: #6219

@tiwaria1
Copy link
Contributor

tiwaria1 commented Jun 1, 2022

8b86af7 passes HIP but still fails L0:

2022-06-01T02:12:48.5256663Z FAIL: SYCL :: USM/usm_pooling.cpp (897 of 899)
2022-06-01T02:12:48.5257504Z ******************** TEST 'SYCL :: USM/usm_pooling.cpp' FAILED ********************
2022-06-01T02:12:48.5258681Z Script:
2022-06-01T02:12:48.5258949Z --
2022-06-01T02:12:48.5259736Z : 'RUN: at line 2';    /__w/llvm/llvm/toolchain/bin/clang++      -fsycl -fsycl-targets=spir64 /__w/llvm/llvm/llvm_test_suite/SYCL/USM/usm_pooling.cpp -o /__w/llvm/llvm/build/SYCL/USM/Output/usm_pooling.cpp.tmp.out
2022-06-01T02:12:48.5263032Z : 'RUN: at line 7';   env ZE_DEBUG=1  env SYCL_DEVICE_FILTER=ext_oneapi_level_zero:gpu,host  SYCL_PI_LEVEL_ZERO_DISABLE_USM_ALLOCATOR=1 /__w/llvm/llvm/build/SYCL/USM/Output/usm_pooling.cpp.tmp.out h 2> /__w/llvm/llvm/build/SYCL/USM/Output/usm_pooling.cpp.tmp1.out; cat /__w/llvm/llvm/build/SYCL/USM/Output/usm_pooling.cpp.tmp1.out | FileCheck /__w/llvm/llvm/llvm_test_suite/SYCL/USM/usm_pooling.cpp --check-prefix CHECK-NOPOOL
2022-06-01T02:12:48.5264380Z : 'RUN: at line 8';   env ZE_DEBUG=1  env SYCL_DEVICE_FILTER=ext_oneapi_level_zero:gpu,host  SYCL_PI_LEVEL_ZERO_DISABLE_USM_ALLOCATOR=1 /__w/llvm/llvm/build/SYCL/USM/Output/usm_pooling.cpp.tmp.out d 2> /__w/llvm/llvm/build/SYCL/USM/Output/usm_pooling.cpp.tmp1.out; cat /__w/llvm/llvm/build/SYCL/USM/Output/usm_pooling.cpp.tmp1.out | FileCheck /__w/llvm/llvm/llvm_test_suite/SYCL/USM/usm_pooling.cpp --check-prefix CHECK-NOPOOL
2022-06-01T02:12:48.5266056Z : 'RUN: at line 9';   env ZE_DEBUG=1  env SYCL_DEVICE_FILTER=ext_oneapi_level_zero:gpu,host  SYCL_PI_LEVEL_ZERO_DISABLE_USM_ALLOCATOR=1 /__w/llvm/llvm/build/SYCL/USM/Output/usm_pooling.cpp.tmp.out s 2> /__w/llvm/llvm/build/SYCL/USM/Output/usm_pooling.cpp.tmp1.out; cat /__w/llvm/llvm/build/SYCL/USM/Output/usm_pooling.cpp.tmp1.out | FileCheck /__w/llvm/llvm/llvm_test_suite/SYCL/USM/usm_pooling.cpp --check-prefix CHECK-NOPOOL
2022-06-01T02:12:48.5267311Z : 'RUN: at line 12';   env ZE_DEBUG=1  env SYCL_DEVICE_FILTER=ext_oneapi_level_zero:gpu,host  SYCL_PI_LEVEL_ZERO_USM_ALLOCATOR=";;1M,4,64K" /__w/llvm/llvm/build/SYCL/USM/Output/usm_pooling.cpp.tmp.out h 2> /__w/llvm/llvm/build/SYCL/USM/Output/usm_pooling.cpp.tmp1.out; cat /__w/llvm/llvm/build/SYCL/USM/Output/usm_pooling.cpp.tmp1.out | FileCheck /__w/llvm/llvm/llvm_test_suite/SYCL/USM/usm_pooling.cpp --check-prefix CHECK-12345
2022-06-01T02:12:48.5268802Z : 'RUN: at line 13';   env ZE_DEBUG=1  env SYCL_DEVICE_FILTER=ext_oneapi_level_zero:gpu,host  SYCL_PI_LEVEL_ZERO_USM_ALLOCATOR=";;1M,4,64K" /__w/llvm/llvm/build/SYCL/USM/Output/usm_pooling.cpp.tmp.out d 2> /__w/llvm/llvm/build/SYCL/USM/Output/usm_pooling.cpp.tmp1.out; cat /__w/llvm/llvm/build/SYCL/USM/Output/usm_pooling.cpp.tmp1.out | FileCheck /__w/llvm/llvm/llvm_test_suite/SYCL/USM/usm_pooling.cpp --check-prefix CHECK-12345
2022-06-01T02:12:48.5270410Z : 'RUN: at line 14';   env ZE_DEBUG=1  env SYCL_DEVICE_FILTER=ext_oneapi_level_zero:gpu,host  SYCL_PI_LEVEL_ZERO_USM_ALLOCATOR=";;1M,4,64K" /__w/llvm/llvm/build/SYCL/USM/Output/usm_pooling.cpp.tmp.out s 2> /__w/llvm/llvm/build/SYCL/USM/Output/usm_pooling.cpp.tmp1.out; cat /__w/llvm/llvm/build/SYCL/USM/Output/usm_pooling.cpp.tmp1.out | FileCheck /__w/llvm/llvm/llvm_test_suite/SYCL/USM/usm_pooling.cpp --check-prefix CHECK-12345
2022-06-01T02:12:48.5271935Z : 'RUN: at line 17';   env ZE_DEBUG=1  env SYCL_DEVICE_FILTER=ext_oneapi_level_zero:gpu,host  SYCL_PI_LEVEL_ZERO_USM_ALLOCATOR=";;2M,1,64K" /__w/llvm/llvm/build/SYCL/USM/Output/usm_pooling.cpp.tmp.out h 2> /__w/llvm/llvm/build/SYCL/USM/Output/usm_pooling.cpp.tmp1.out; cat /__w/llvm/llvm/build/SYCL/USM/Output/usm_pooling.cpp.tmp1.out | FileCheck /__w/llvm/llvm/llvm_test_suite/SYCL/USM/usm_pooling.cpp --check-prefix CHECK-1245
2022-06-01T02:12:48.5273868Z : 'RUN: at line 18';   env ZE_DEBUG=1  env SYCL_DEVICE_FILTER=ext_oneapi_level_zero:gpu,host  SYCL_PI_LEVEL_ZERO_USM_ALLOCATOR=";;2M,1,64K" /__w/llvm/llvm/build/SYCL/USM/Output/usm_pooling.cpp.tmp.out d 2> /__w/llvm/llvm/build/SYCL/USM/Output/usm_pooling.cpp.tmp1.out; cat /__w/llvm/llvm/build/SYCL/USM/Output/usm_pooling.cpp.tmp1.out | FileCheck /__w/llvm/llvm/llvm_test_suite/SYCL/USM/usm_pooling.cpp --check-prefix CHECK-1245
2022-06-01T02:12:48.5275138Z : 'RUN: at line 19';   env ZE_DEBUG=1  env SYCL_DEVICE_FILTER=ext_oneapi_level_zero:gpu,host  SYCL_PI_LEVEL_ZERO_USM_ALLOCATOR=";;2M,1,64K" /__w/llvm/llvm/build/SYCL/USM/Output/usm_pooling.cpp.tmp.out s 2> /__w/llvm/llvm/build/SYCL/USM/Output/usm_pooling.cpp.tmp1.out; cat /__w/llvm/llvm/build/SYCL/USM/Output/usm_pooling.cpp.tmp1.out | FileCheck /__w/llvm/llvm/llvm_test_suite/SYCL/USM/usm_pooling.cpp --check-prefix CHECK-1245
2022-06-01T02:12:48.5276432Z : 'RUN: at line 22';   env ZE_DEBUG=1  env SYCL_DEVICE_FILTER=ext_oneapi_level_zero:gpu,host  SYCL_PI_LEVEL_ZERO_USM_ALLOCATOR=";2M;2M,4,64K" /__w/llvm/llvm/build/SYCL/USM/Output/usm_pooling.cpp.tmp.out h 2> /__w/llvm/llvm/build/SYCL/USM/Output/usm_pooling.cpp.tmp1.out; cat /__w/llvm/llvm/build/SYCL/USM/Output/usm_pooling.cpp.tmp1.out | FileCheck /__w/llvm/llvm/llvm_test_suite/SYCL/USM/usm_pooling.cpp --check-prefix CHECK-1245
2022-06-01T02:12:48.5277665Z : 'RUN: at line 23';   env ZE_DEBUG=1  env SYCL_DEVICE_FILTER=ext_oneapi_level_zero:gpu,host  SYCL_PI_LEVEL_ZERO_USM_ALLOCATOR=";2M;2M,4,64K" /__w/llvm/llvm/build/SYCL/USM/Output/usm_pooling.cpp.tmp.out d 2> /__w/llvm/llvm/build/SYCL/USM/Output/usm_pooling.cpp.tmp1.out; cat /__w/llvm/llvm/build/SYCL/USM/Output/usm_pooling.cpp.tmp1.out | FileCheck /__w/llvm/llvm/llvm_test_suite/SYCL/USM/usm_pooling.cpp --check-prefix CHECK-1245
2022-06-01T02:12:48.5278945Z : 'RUN: at line 24';   env ZE_DEBUG=1  env SYCL_DEVICE_FILTER=ext_oneapi_level_zero:gpu,host  SYCL_PI_LEVEL_ZERO_USM_ALLOCATOR=";2M;2M,4,64K" /__w/llvm/llvm/build/SYCL/USM/Output/usm_pooling.cpp.tmp.out s 2> /__w/llvm/llvm/build/SYCL/USM/Output/usm_pooling.cpp.tmp1.out; cat /__w/llvm/llvm/build/SYCL/USM/Output/usm_pooling.cpp.tmp1.out | FileCheck /__w/llvm/llvm/llvm_test_suite/SYCL/USM/usm_pooling.cpp --check-prefix CHECK-1245
2022-06-01T02:12:48.5280163Z : 'RUN: at line 27';   env ZE_DEBUG=1  env SYCL_DEVICE_FILTER=ext_oneapi_level_zero:gpu,host  SYCL_PI_LEVEL_ZERO_USM_ALLOCATOR=";;2M,4,4M" /__w/llvm/llvm/build/SYCL/USM/Output/usm_pooling.cpp.tmp.out h 2> /__w/llvm/llvm/build/SYCL/USM/Output/usm_pooling.cpp.tmp1.out; cat /__w/llvm/llvm/build/SYCL/USM/Output/usm_pooling.cpp.tmp1.out | FileCheck /__w/llvm/llvm/llvm_test_suite/SYCL/USM/usm_pooling.cpp --check-prefix CHECK-15
2022-06-01T02:12:48.5281384Z : 'RUN: at line 28';   env ZE_DEBUG=1  env SYCL_DEVICE_FILTER=ext_oneapi_level_zero:gpu,host  SYCL_PI_LEVEL_ZERO_USM_ALLOCATOR=";;2M,4,4M" /__w/llvm/llvm/build/SYCL/USM/Output/usm_pooling.cpp.tmp.out d 2> /__w/llvm/llvm/build/SYCL/USM/Output/usm_pooling.cpp.tmp1.out; cat /__w/llvm/llvm/build/SYCL/USM/Output/usm_pooling.cpp.tmp1.out | FileCheck /__w/llvm/llvm/llvm_test_suite/SYCL/USM/usm_pooling.cpp --check-prefix CHECK-15
2022-06-01T02:12:48.5282580Z : 'RUN: at line 29';   env ZE_DEBUG=1  env SYCL_DEVICE_FILTER=ext_oneapi_level_zero:gpu,host  SYCL_PI_LEVEL_ZERO_USM_ALLOCATOR=";;2M,4,4M" /__w/llvm/llvm/build/SYCL/USM/Output/usm_pooling.cpp.tmp.out s 2> /__w/llvm/llvm/build/SYCL/USM/Output/usm_pooling.cpp.tmp1.out; cat /__w/llvm/llvm/build/SYCL/USM/Output/usm_pooling.cpp.tmp1.out | FileCheck /__w/llvm/llvm/llvm_test_suite/SYCL/USM/usm_pooling.cpp --check-prefix CHECK-15
2022-06-01T02:12:48.5283110Z --
2022-06-01T02:12:48.5283247Z Exit Code: 1
2022-06-01T02:12:48.5283390Z 
2022-06-01T02:12:48.5283462Z Command Output (stdout):
2022-06-01T02:12:48.5283638Z --
2022-06-01T02:12:48.5283778Z $ ":" "RUN: at line 2"
2022-06-01T02:12:48.5283997Z note: command had no output on stdout or stderr
2022-06-01T02:12:48.5284539Z $ "/__w/llvm/llvm/toolchain/bin/clang++" "-fsycl" "-fsycl-targets=spir64" "/__w/llvm/llvm/llvm_test_suite/SYCL/USM/usm_pooling.cpp" "-o" "/__w/llvm/llvm/build/SYCL/USM/Output/usm_pooling.cpp.tmp.out"
2022-06-01T02:12:48.5284901Z note: command had no output on stdout or stderr
2022-06-01T02:12:48.5285088Z $ ":" "RUN: at line 7"
2022-06-01T02:12:48.5285279Z note: command had no output on stdout or stderr
2022-06-01T02:12:48.5285645Z $ "env" "ZE_DEBUG=1" "env" "SYCL_DEVICE_FILTER=ext_oneapi_level_zero:gpu,host" "SYCL_PI_LEVEL_ZERO_DISABLE_USM_ALLOCATOR=1" "/__w/llvm/llvm/build/SYCL/USM/Output/usm_pooling.cpp.tmp.out" "h"
2022-06-01T02:12:48.5285970Z # command output:
2022-06-01T02:12:48.5286288Z Running on device GPU (Intel(R) UHD Graphics [0x9bca]) Intel(R) Level-Zero plugin
2022-06-01T02:12:48.5286460Z 
2022-06-01T02:12:48.5286596Z $ "cat" "/__w/llvm/llvm/build/SYCL/USM/Output/usm_pooling.cpp.tmp1.out"
2022-06-01T02:12:48.5286971Z note: command had no output on stdout or stderr
2022-06-01T02:12:48.5287370Z $ "FileCheck" "/__w/llvm/llvm/llvm_test_suite/SYCL/USM/usm_pooling.cpp" "--check-prefix" "CHECK-NOPOOL"
2022-06-01T02:12:48.5287615Z # command stderr:
2022-06-01T02:12:48.5288018Z /__w/llvm/llvm/llvm_test_suite/SYCL/USM/usm_pooling.cpp:105:23: error: CHECK-NOPOOL-NEXT: is not on the line after the previous match
2022-06-01T02:12:48.5288369Z // CHECK-NOPOOL-NEXT: ZE ---> [[API]](
2022-06-01T02:12:48.5288546Z                       ^
2022-06-01T02:12:48.5288773Z <stdin>:43:1: note: 'next' match was here
2022-06-01T02:12:48.5289115Z ZE ---> zeMemAllocHost(Context->ZeContext, &ZeHostDesc, Size, Alignment, ResultPtr)
2022-06-01T02:12:48.5289407Z ^
2022-06-01T02:12:48.5289578Z <stdin>:36:20: note: previous match ended here
2022-06-01T02:12:48.5289771Z Test zeMemAllocHost
2022-06-01T02:12:48.5289924Z                    ^
2022-06-01T02:12:48.5290211Z <stdin>:37:1: note: non-matching line after previous match is here
2022-06-01T02:12:48.5290416Z ==========================
2022-06-01T02:12:48.5290560Z ^
2022-06-01T02:12:48.5290636Z 
2022-06-01T02:12:48.5290700Z Input file: <stdin>
2022-06-01T02:12:48.5290916Z Check file: /__w/llvm/llvm/llvm_test_suite/SYCL/USM/usm_pooling.cpp
2022-06-01T02:12:48.5291067Z 
2022-06-01T02:12:48.5291236Z -dump-input=help explains the following input dump.
2022-06-01T02:12:48.5291383Z 
2022-06-01T02:12:48.5291436Z Input was:
2022-06-01T02:12:48.5291574Z <<<<<<
2022-06-01T02:12:48.5291707Z           .
2022-06-01T02:12:48.5291840Z           .
2022-06-01T02:12:48.5291974Z           .
2022-06-01T02:12:48.5292201Z          38: SYCL over Level-Zero 1.3.22905 
2022-06-01T02:12:48.5292380Z          39: ========================== 
2022-06-01T02:12:48.5292547Z          40: ========================== 
2022-06-01T02:12:48.5292790Z          41: SYCL over Level-Zero 1.3.22905 
2022-06-01T02:12:48.5292966Z          42: ========================== 
2022-06-01T02:12:48.5293310Z          43: ZE ---> zeMemAllocHost(Context->ZeContext, &ZeHostDesc, Size, Alignment, ResultPtr) 
2022-06-01T02:12:48.5293615Z next:105     !~~~~~~~~~~~~~~~~~~~~~~                                                              error: match on wrong line
2022-06-01T02:12:48.5293822Z          44: ========================== 
2022-06-01T02:12:48.5294058Z          45: SYCL over Level-Zero 1.3.22905 
2022-06-01T02:12:48.5294238Z          46: ========================== 
2022-06-01T02:12:48.5294399Z          47: ========================== 
2022-06-01T02:12:48.5294631Z          48: SYCL over Level-Zero 1.3.22905 
2022-06-01T02:12:48.5294797Z           .
2022-06-01T02:12:48.5294934Z           .
2022-06-01T02:12:48.5295062Z           .
2022-06-01T02:12:48.5295200Z >>>>>>
2022-06-01T02:12:48.5295283Z 
2022-06-01T02:12:48.5295374Z error: command failed with exit status: 1
2022-06-01T02:12:48.5295556Z 
2022-06-01T02:12:48.5295630Z --
2022-06-01T02:12:48.5295714Z 
2022-06-01T02:12:48.5295769Z ********************

@smaslov-intel Could you help us resolve this failure?

@smaslov-intel
Copy link
Contributor

@smaslov-intel Could you help us resolve this failure?

@tiwaria1 : please show the test output before and after the change

@tiwaria1
Copy link
Contributor

tiwaria1 commented Jun 3, 2022

@smaslov-intel Could you help us resolve this failure?

@tiwaria1 : please show the test output before and after the change

Hi @smaslov-intel, how do I get this build, do I have to build this change locally or is there a way to download the build from github for this PR? And do you know where the llvm_test_suite is located on the repo? The failing test is usm_pooling.cpp.

@pcolberg
Copy link
Contributor Author

pcolberg commented Jun 3, 2022

@tiwaria1 I believe this is the failing test: llvm-test-suite/SYCL/USM/usm_pooling.cpp.

@pcolberg
Copy link
Contributor Author

pcolberg commented Jun 3, 2022

0408af8 passes L0.

See #6218 (comment) for unrelated ESIMD Emu runner breakage.

@pcolberg
Copy link
Contributor Author

pcolberg commented Jun 3, 2022

e6cd0ea fails L0.

@pcolberg
Copy link
Contributor Author

pcolberg commented Jun 3, 2022

d627549 fails L0.

@pcolberg
Copy link
Contributor Author

pcolberg commented Jun 3, 2022

ec6e188 fails L0, OCL GEN9, OCL x64, and HIP AMDGPU.

@pcolberg
Copy link
Contributor Author

pcolberg commented Jun 3, 2022

8c261b4 passes L0.

@pcolberg
Copy link
Contributor Author

pcolberg commented Jun 3, 2022

6deb2a6 fails L0.

@pcolberg
Copy link
Contributor Author

pcolberg commented Jun 3, 2022

019e0c9 fails L0.

@pcolberg
Copy link
Contributor Author

pcolberg commented Jun 3, 2022

64dabbc fails L0 and HIP.

@pcolberg
Copy link
Contributor Author

pcolberg commented Jun 4, 2022

After trying many slight variations of this PR, I am unsure why L0 GEN9 is failing in USM/usm_pooling.cpp. The first commit passing the property list through from malloc_host() to alignedAllocHost() passes L0 fine. But as soon as anything is added to alignedAllocHost(), as little as querying the platform of the context,
https://github.com/intel/llvm/blob/ec6e188a60baf759a62bacbd1e8b434857dfe32e/sycl/source/detail/usm/usm_impl.cpp#L73-L80
the L0 test starts failing. I am wondering if the fault is not in the implementation, but in the failing test? I don't quite understand yet what the test does, is it tracing L0 backend calls and matching for the expected calls? Anyway, this is probably much easier to debug using a local build on the L0 GEN9 platform rather than CI.

@tiwaria1
Copy link
Contributor

tiwaria1 commented Jun 6, 2022

@smaslov-intel, @cperkinsintel Do you have access to L0 GEN9 platform, could you help us get access to debug this further? Should I file an issue on intel/llvm-test-suite to see if the test owner can help us understand the test?

@smaslov-intel
Copy link
Contributor

I don't quite understand yet what the test does, is it tracing L0 backend calls and matching for the expected calls?

@tiwaria1 : yes, that's what it is testing, expects specific calls to native L0 RT based on the SYCL USM pooling settings. With your changes do you expect any difference in PI calls being made by SYCL RT for the failing test? (I don't think so, and then the failure does look unexpected).

this is probably much easier to debug using a local build on the L0 GEN9 platform rather than CI.

Absolutely. @pvchupin : do we have any wiki on how to locally reproduce L0 failures encountered in CI testing?
@tiwaria1 : point me to the exact changes (preferably rebased to the top of the trunk) that exhibit the failure, I'll try to reproduce it locally myself.

@pcolberg
Copy link
Contributor Author

pcolberg commented Jun 6, 2022

9f7c265 is failing check-clang with an unrelated error, see #6243 (comment):

llvm-lit: /__w/llvm/llvm/src/llvm/utils/lit/lit/llvm/config.py:449: note: using clang: /__w/llvm/llvm/build/bin/clang
-- Testing: 15943 tests, 16 workers --
Testing:  0.. 10.. 20.. 30.. 40.. 50.. 60.. 70.. 80.. 90
FAIL: Clang :: SemaSYCL/reqd-sub-group-size-ast.cpp (15036 of 15943)
******************** TEST 'Clang :: SemaSYCL/reqd-sub-group-size-ast.cpp' FAILED ********************
Script:
--
: 'RUN: at line 1';   /__w/llvm/llvm/build/bin/clang -cc1 -internal-isystem /__w/llvm/llvm/build/lib/clang/15.0.0/include -nostdsysteminc -fsycl-is-device -internal-isystem /__w/llvm/llvm/src/clang/test/SemaSYCL/Inputs -sycl-std=2017 -Wno-sycl-2017-compat -ast-dump /__w/llvm/llvm/src/clang/test/SemaSYCL/reqd-sub-group-size-ast.cpp | /__w/llvm/llvm/build/bin/FileCheck /__w/llvm/llvm/src/clang/test/SemaSYCL/reqd-sub-group-size-ast.cpp
--
Exit Code: 2

Command Output (stderr):
--
/__w/llvm/llvm/src/clang/test/SemaSYCL/reqd-sub-group-size-ast.cpp:35:4: error: misspelled directive 'CHECK_NEXT:'
// CHECK_NEXT: IntelReqdSubGroupSizeAttr {{.*}} reqd_sub_group_size
   ^

--

********************
Testing:  0.. 10.. 20.. 30.. 40.. 50.. 60.. 70.. 80.. 90.. 
********************
Failed Tests (1):
  Clang :: SemaSYCL/reqd-sub-group-size-ast.cpp


Testing Time: 99.33s
  Skipped          :     4
  Unsupported      :  1405
  Passed           : 29698
  Expectedly Failed:    29
  Failed           :     1

@bader
Copy link
Contributor

bader commented Jun 6, 2022

9f7c265 is failing check-clang with an unrelated error:

llvm-lit: /__w/llvm/llvm/src/llvm/utils/lit/lit/llvm/config.py:449: note: using clang: /__w/llvm/llvm/build/bin/clang
-- Testing: 15943 tests, 16 workers --
Testing:  0.. 10.. 20.. 30.. 40.. 50.. 60.. 70.. 80.. 90
FAIL: Clang :: SemaSYCL/reqd-sub-group-size-ast.cpp (15036 of 15943)
******************** TEST 'Clang :: SemaSYCL/reqd-sub-group-size-ast.cpp' FAILED ********************
Script:
--
: 'RUN: at line 1';   /__w/llvm/llvm/build/bin/clang -cc1 -internal-isystem /__w/llvm/llvm/build/lib/clang/15.0.0/include -nostdsysteminc -fsycl-is-device -internal-isystem /__w/llvm/llvm/src/clang/test/SemaSYCL/Inputs -sycl-std=2017 -Wno-sycl-2017-compat -ast-dump /__w/llvm/llvm/src/clang/test/SemaSYCL/reqd-sub-group-size-ast.cpp | /__w/llvm/llvm/build/bin/FileCheck /__w/llvm/llvm/src/clang/test/SemaSYCL/reqd-sub-group-size-ast.cpp
--
Exit Code: 2

Command Output (stderr):
--
/__w/llvm/llvm/src/clang/test/SemaSYCL/reqd-sub-group-size-ast.cpp:35:4: error: misspelled directive 'CHECK_NEXT:'
// CHECK_NEXT: IntelReqdSubGroupSizeAttr {{.*}} reqd_sub_group_size
   ^

--

********************
Testing:  0.. 10.. 20.. 30.. 40.. 50.. 60.. 70.. 80.. 90.. 
********************
Failed Tests (1):
  Clang :: SemaSYCL/reqd-sub-group-size-ast.cpp


Testing Time: 99.33s
  Skipped          :     4
  Unsupported      :  1405
  Passed           : 29698
  Expectedly Failed:    29
  Failed           :     1

This seems to be introduced by 4071659 a few hours ago.
@smanna12, could you fix this issue ASAP, please?
@pvchupin, FYI.

@pvchupin
Copy link
Contributor

pvchupin commented Jun 6, 2022

Absolutely. @pvchupin : do we have any wiki on how to locally reproduce L0 failures encountered in CI testing?

You need to follow https://github.com/intel/llvm/blob/sycl/sycl/doc/GetStartedGuide.md and may also need to pass --ci-defaults to the configure.py to match extended testing CI is doing.

@pcolberg
Copy link
Contributor Author

pcolberg commented Jun 6, 2022

@tiwaria1 : point me to the exact changes (preferably rebased to the top of the trunk) that exhibit the failure, I'll try to reproduce it locally myself.

Thanks @smaslov-intel! I have rebased onto sycl. The second commit, b4e2489 is failing L0 in USM/usm_pooling.cpp. The first commit, 181c015 passes L0.

pvchupin pushed a commit that referenced this pull request Jun 7, 2022
…locations (#6219)

Spec states that the runtime buffer_location property will not have an effect on shared/host allocations. This is incorrect as the property is meant to be used with host/shared allocations with the same behavior as device allocations.

#6218
#6220
@smaslov-intel
Copy link
Contributor

@tiwaria1 : point me to the exact changes (preferably rebased to the top of the trunk) that exhibit the failure, I'll try to reproduce it locally myself.

Thanks @smaslov-intel! I have rebased onto sycl. The second commit, b4e2489 is failing L0 in USM/usm_pooling.cpp. The first commit, 181c015 passes L0.

Please see https://github.com/intel/llvm/pull/6220/files#r891017817

@pcolberg pcolberg requested a review from a team as a code owner June 7, 2022 14:58
@smaslov-intel
Copy link
Contributor

The change looks good, and what about #6220 (comment)?

@pcolberg
Copy link
Contributor Author

pcolberg commented Jun 8, 2022

@smaslov-intel, this PR is ready for review. Thank you for investigating and providing the pointers that helped quickly resolve the test failure 🙂

@pcolberg
Copy link
Contributor Author

pcolberg commented Jun 8, 2022

@intel/llvm-gatekeepers This is ready for merge 🙂

steffenlarsen pushed a commit that referenced this pull request Jun 8, 2022
…h malloc_shared (#6269)

Use the same call to USMDeviceAlloc with an empty property list in all
cases, to allow for straight-forward extension with future properties.

Query buffer location extension only if buffer location property is passed.

This amends #5634

See also #6220
@steffenlarsen steffenlarsen merged commit 2c7caab into intel:sycl Jun 8, 2022
@pcolberg pcolberg deleted the malloc_host_buffer_location branch June 8, 2022 15:25
pvchupin pushed a commit that referenced this pull request Jun 8, 2022
… to malloc_shared (#6268)

This aligns the implementation with malloc_host and avoids an unneeded
call in the case where the FPGA backend is used without buffer location.

This amends #6218

See also #6220
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

Successfully merging this pull request may close these issues.

6 participants