Skip to content

[SYCL] correct sub-device count calculation for numa partitioning #6005

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 2 commits into from
Apr 14, 2022
Merged
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
53 changes: 46 additions & 7 deletions sycl/source/detail/device_impl.cpp
Original file line number Diff line number Diff line change
Expand Up @@ -133,9 +133,11 @@ device_impl::create_sub_devices(const cl_device_partition_property *Properties,
Plugin.call<sycl::errc::invalid, PiApiKind::piDevicePartition>(
MDevice, Properties, SubDevicesCount, SubDevices.data(),
&ReturnedSubDevices);
// TODO: check that returned number of sub-devices matches what was
// requested, otherwise this walk below is wrong.
//
if (ReturnedSubDevices != SubDevicesCount) {
throw sycl::exception(
errc::invalid,
"Could not partition to the specified number of sub-devices");
}
// TODO: Need to describe the subdevice model. Some sub_device management
// may be necessary. What happens if create_sub_devices is called multiple
// times with the same arguments?
Expand All @@ -161,8 +163,14 @@ std::vector<device> device_impl::create_sub_devices(size_t ComputeUnits) const {
if (!is_partition_supported(info::partition_property::partition_equally)) {
throw cl::sycl::feature_not_supported();
}
size_t SubDevicesCount =
get_info<info::device::max_compute_units>() / ComputeUnits;
// If count exceeds the total number of compute units in the device, an
// exception with the errc::invalid error code must be thrown.
auto MaxComputeUnits = get_info<info::device::max_compute_units>();
if (ComputeUnits > MaxComputeUnits)
throw sycl::exception(errc::invalid,
"Total counts exceed max compute units");

size_t SubDevicesCount = MaxComputeUnits / ComputeUnits;
const cl_device_partition_property Properties[3] = {
CL_DEVICE_PARTITION_EQUALLY, (cl_device_partition_property)ComputeUnits,
0};
Expand All @@ -184,7 +192,33 @@ device_impl::create_sub_devices(const std::vector<size_t> &Counts) const {
static const cl_device_partition_property P[] = {
CL_DEVICE_PARTITION_BY_COUNTS, CL_DEVICE_PARTITION_BY_COUNTS_LIST_END, 0};
std::vector<cl_device_partition_property> Properties(P, P + 3);
Properties.insert(Properties.begin() + 1, Counts.begin(), Counts.end());

// Fill the properties vector with counts and validate it
auto It = Properties.begin() + 1;
size_t TotalCounts = 0;
size_t NonZeroCounts = 0;
for (auto Count : Counts) {
TotalCounts += Count;
NonZeroCounts += (Count != 0) ? 1 : 0;
It = Properties.insert(It, Count);
}
Comment on lines +197 to +204
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please add a comment about what is happening here.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

OK


// If the number of non-zero values in counts exceeds the device’s maximum
// number of sub devices (as returned by info::device::
// partition_max_sub_devices) an exception with the errc::invalid
// error code must be thrown.
if (NonZeroCounts > get_info<info::device::partition_max_sub_devices>())
throw sycl::exception(errc::invalid,
"Total non-zero counts exceed max sub-devices");

// If the total of all the values in the counts vector exceeds the total
// number of compute units in the device (as returned by
// info::device::max_compute_units), an exception with the errc::invalid
// error code must be thrown.
if (TotalCounts > get_info<info::device::max_compute_units>())
throw sycl::exception(errc::invalid,
"Total counts exceed max compute units");

return create_sub_devices(Properties.data(), Counts.size());
}

Expand All @@ -205,7 +239,12 @@ std::vector<device> device_impl::create_sub_devices(
const pi_device_partition_property Properties[3] = {
PI_DEVICE_PARTITION_BY_AFFINITY_DOMAIN,
(pi_device_partition_property)AffinityDomain, 0};
size_t SubDevicesCount = get_info<info::device::partition_max_sub_devices>();

pi_uint32 SubDevicesCount = 0;
const detail::plugin &Plugin = getPlugin();
Plugin.call<sycl::errc::invalid, PiApiKind::piDevicePartition>(
MDevice, Properties, 0, nullptr, &SubDevicesCount);

return create_sub_devices(Properties, SubDevicesCount);
}

Expand Down