Skip to content

[SYCL][clang-offload-bundler] Add support for BC files in archives. #11034

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 24 commits into from
Oct 23, 2023
Merged

[SYCL][clang-offload-bundler] Add support for BC files in archives. #11034

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
24 commits
Select commit Hold shift + click to select a range
cdce858
Add support for BC files in archives.
Aug 31, 2023
b0dd2c0
Add testing of -list, -check-section and -unbundle for archives with …
Aug 31, 2023
0b4fa66
Apply clang-format
Aug 31, 2023
d3ee76c
Use 'rm -f' to ensure that command will succeed
Sep 1, 2023
dd02094
Test BC archive unbundling with multiple files for the same target
Sep 1, 2023
4f153c4
Only apply clang-format to this PRs changes
Sep 1, 2023
bda9f68
Test unbundling multiple targets
Sep 1, 2023
0383d7e
Refactor common code
Sep 5, 2023
f752c81
Use llvm-ar not ar
Sep 7, 2023
1efe986
Rename FHP to FH
Oct 16, 2023
ac55b07
Change memory buffer check to assert
Oct 18, 2023
9509557
Use simpler kernel code
Oct 18, 2023
9f699dc
Add toplevel comment describing test and simplify test function
Oct 18, 2023
686a48f
Fix clang-format issue
Oct 19, 2023
9477f20
Avoid piping in back-ticked command to avoid Window's problems
Oct 19, 2023
225b30f
clang-offload-bundler tests require linux since b/c the target string…
Oct 19, 2023
4eee644
Create Windows version of test by updating target string
Oct 19, 2023
3ddec6a
Disable commands that use backticks on Windows. Give Linux tests a l…
Oct 20, 2023
8f5b407
Correctly disable RUN commands with a backtick
Oct 20, 2023
8f28b72
Update disabling comment for clarity
Oct 20, 2023
336e0b3
Correctly disable RUN commands with a backtick
Oct 20, 2023
182b3f1
Save test output to a temp file so it is not created twice
Oct 23, 2023
439a409
Revise comment and put in correct location
Oct 23, 2023
85ad4e7
Use C++ style comments
Oct 23, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
111 changes: 67 additions & 44 deletions clang/lib/Driver/OffloadBundler.cpp
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1040,39 +1040,49 @@ class ArchiveFileHandler final : public FileHandler {
for (auto &C : Ar->children(Err)) {
++ChildIndex;
auto BinOrErr = C.getAsBinary();

std::unique_ptr<FileHandler> FH{nullptr};
std::unique_ptr<MemoryBuffer> Buf{nullptr};

if (!BinOrErr) {
if (auto Err = isNotObjectErrorInvalidFileType(BinOrErr.takeError()))
return Err;
continue;
}

auto &Bin = BinOrErr.get();
if (!Bin->isObject())
continue;
// Handle bundled BC Files
FH = std::make_unique<BinaryFileHandler>(BundlerConfig);
auto MR = C.getMemoryBufferRef();
assert(MR);
Buf = MemoryBuffer::getMemBuffer(*MR, false);
} else {
auto &Bin = BinOrErr.get();
if (!Bin->isObject())
continue;

auto CheckOrErr = CheckIfObjectFileContainsExcludedTargets(C);
if (!CheckOrErr)
return CheckOrErr.takeError();
auto CheckOrErr = CheckIfObjectFileContainsExcludedTargets(C);
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hi @maksimsab

We are trying to add support here for cases where the archive file will contain .bc file. Previously, we handle only those archive files which have .o files.
Can you please comment if we will need this check for archives containing .bc files?

Thanks

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't know at the moment whether we will need it in the future.
This particular piece of code was related to handling of "FPGA" part. Frankly speaking, we don't have a concrete design for them.

if (!CheckOrErr)
return CheckOrErr.takeError();

if (*CheckOrErr) {
LLVM_DEBUG(outs() << "Add child to ban list. Index: " << ChildIndex
<< "\n");
ExcludedChildIndexes.emplace(ChildIndex);
}
if (*CheckOrErr) {
LLVM_DEBUG(outs()
<< "Add child to ban list. Index: " << ChildIndex << "\n");
ExcludedChildIndexes.emplace(ChildIndex);
}

auto Obj = std::unique_ptr<ObjectFile>(cast<ObjectFile>(Bin.release()));
auto Buf = MemoryBuffer::getMemBuffer(Obj->getMemoryBufferRef(), false);
auto Obj = std::unique_ptr<ObjectFile>(cast<ObjectFile>(Bin.release()));
Buf = MemoryBuffer::getMemBuffer(Obj->getMemoryBufferRef(), false);

// Collect the list of bundles from the object.
ObjectFileHandler OFH(std::move(Obj), BundlerConfig);
if (Error Err = OFH.ReadHeader(*Buf))
FH = std::make_unique<ObjectFileHandler>(std::move(Obj), BundlerConfig);
}

// Collect the list of bundles from the object or bundled BC file.
if (Error Err = FH->ReadHeader(*Buf))
return Err;
Expected<std::optional<StringRef>> NameOrErr = OFH.ReadBundleStart(*Buf);
Expected<std::optional<StringRef>> NameOrErr = FH->ReadBundleStart(*Buf);
if (!NameOrErr)
return NameOrErr.takeError();
while (*NameOrErr) {
++Bundles[**NameOrErr];
NameOrErr = OFH.ReadBundleStart(*Buf);
NameOrErr = FH->ReadBundleStart(*Buf);
if (!NameOrErr)
return NameOrErr.takeError();
}
Expand Down Expand Up @@ -1125,28 +1135,40 @@ class ArchiveFileHandler final : public FileHandler {
continue;
}

std::unique_ptr<FileHandler> FH{nullptr};
std::unique_ptr<MemoryBuffer> Buf{nullptr};
StringRef Ext("o");
if (BundlerConfig.FilesType == "aocr" ||
BundlerConfig.FilesType == "aocx")
Ext = BundlerConfig.FilesType;

auto BinOrErr = C.getAsBinary();
if (!BinOrErr) {
// Not a recognized binary file. Specifically not an object file
if (auto Err = isNotObjectErrorInvalidFileType(BinOrErr.takeError()))
return Err;
continue;
}

auto &Bin = BinOrErr.get();
if (!Bin->isObject())
continue;

auto Obj = std::unique_ptr<ObjectFile>(cast<ObjectFile>(Bin.release()));
auto Buf = MemoryBuffer::getMemBuffer(Obj->getMemoryBufferRef(), false);

auto ChildNameOrErr = C.getName();
if (!ChildNameOrErr)
return ChildNameOrErr.takeError();
if (BundlerConfig.FilesType == "aoo") {
// Handle bundled BC Files
Ext = "bc";
FH = std::make_unique<BinaryFileHandler>(BundlerConfig);
auto MR = C.getMemoryBufferRef();
assert(MR);
Buf = MemoryBuffer::getMemBuffer(*MR, false);
} else
continue;
} else {
auto &Bin = BinOrErr.get();
if (!Bin->isObject())
continue;
auto Obj = std::unique_ptr<ObjectFile>(cast<ObjectFile>(Bin.release()));
Buf = MemoryBuffer::getMemBuffer(Obj->getMemoryBufferRef(), false);
FH = std::make_unique<ObjectFileHandler>(std::move(Obj), BundlerConfig);
}

ObjectFileHandler OFH(std::move(Obj), BundlerConfig);
if (Error Err = OFH.ReadHeader(*Buf))
if (Error Err = FH->ReadHeader(*Buf))
return Err;
Expected<std::optional<StringRef>> NameOrErr = OFH.ReadBundleStart(*Buf);
Expected<std::optional<StringRef>> NameOrErr = FH->ReadBundleStart(*Buf);
if (!NameOrErr)
return NameOrErr.takeError();
while (*NameOrErr) {
Expand All @@ -1156,10 +1178,7 @@ class ArchiveFileHandler final : public FileHandler {
if (Mode == OutputType::FileList) {
// Create temporary file where the device part will be extracted to.
SmallString<128u> ChildFileName;
StringRef Ext("o");
if (BundlerConfig.FilesType == "aocr" ||
BundlerConfig.FilesType == "aocx")
Ext = BundlerConfig.FilesType;

auto EC = sys::fs::createTemporaryFile(TempFileNameBase, Ext,
ChildFileName);
if (EC)
Expand All @@ -1169,7 +1188,7 @@ class ArchiveFileHandler final : public FileHandler {
if (EC)
return createFileError(ChildFileName, EC);

if (Error Err = OFH.ReadBundle(ChildOS, *Buf))
if (Error Err = FH->ReadBundle(ChildOS, *Buf))
return Err;

if (ChildOS.has_error())
Expand All @@ -1180,13 +1199,17 @@ class ArchiveFileHandler final : public FileHandler {
OS << ChildFileName << "\n";
} else if (Mode == OutputType::Object) {
// Extract the bundle to the output file in single file mode.
if (Error Err = OFH.ReadBundle(OS, *Buf))
if (Error Err = FH->ReadBundle(OS, *Buf))
return Err;
} else if (Mode == OutputType::Archive) {
auto ChildNameOrErr = C.getName();
if (!ChildNameOrErr)
return ChildNameOrErr.takeError();

// Extract the bundle to a buffer.
SmallVector<char> Data;
raw_svector_ostream ChildOS{Data};
if (Error Err = OFH.ReadBundle(ChildOS, *Buf))
if (Error Err = FH->ReadBundle(ChildOS, *Buf))
return Err;

// Add new archive member.
Expand All @@ -1195,10 +1218,10 @@ class ArchiveFileHandler final : public FileHandler {
Member.Buf = MemoryBuffer::getMemBufferCopy(ChildOS.str(), Name);
Member.MemberName = Member.Buf->getBufferIdentifier();
}
if (Error Err = OFH.ReadBundleEnd(*Buf))
if (Error Err = FH->ReadBundleEnd(*Buf))
return Err;
}
NameOrErr = OFH.ReadBundleStart(*Buf);
NameOrErr = FH->ReadBundleStart(*Buf);
if (!NameOrErr)
return NameOrErr.takeError();
}
Expand Down
123 changes: 123 additions & 0 deletions clang/test/Driver/clang-offload-bundler-bc-archive-support-linux.cpp
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,123 @@
// Target "host-x86_64-unknown-linux-gnu" only works on Linux
// REQUIRES: system-linux

// Ensure that bundled BC files in archives can work with:
// TEST1: clang-offload-bundler -list
// TEST2: clang-offload-bundler -check-section
// TEST3: clang-offload-bundler -unbundle with single target
// TEST4: clang-offload-bundler -unbundle with multiple targets
//
// In all these tests also ensure functionality with bundled object files still
// works correctly.

////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////
// Make bundled object with targets:
// sycl-spir64-unknown-unknown
// host-x86_64-unknown-linux-gnu
// RUN: %clangxx -fsycl -c %s -o %t_bundled.o

////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////
// Make three distinct BC files
// RUN: %clangxx -fsycl -fsycl-device-only -DTYPE1 %s -o %t1.bc
// RUN: %clangxx -fsycl -fsycl-device-only -DTYPE2 %s -o %t2.bc
// RUN: %clangxx -fsycl -fsycl-device-only -DTYPE3 %s -o %t3.bc

////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////
// Bundle BC files to different targets:
// host-spir64-unknown-unknown
// host-spir64_gen
// host-spir64_x86_64
// RUN: clang-offload-bundler -type=bc -targets=host-spir64-unknown-unknown -input=%t1.bc -output=%t1_bundled.bc
// RUN: clang-offload-bundler -type=bc -targets=host-spir64_gen -input=%t2.bc -output=%t2_bundled.bc
// RUN: clang-offload-bundler -type=bc -targets=host-spir64_x86_64 -input=%t3.bc -output=%t3_bundled.bc

////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////
// Make archive with bundled BC and o files
// RUN: rm -f %t_bundled.a
// RUN: llvm-ar cr %t_bundled.a %t1_bundled.bc %t2_bundled.bc %t3_bundled.bc %t_bundled.o

////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////
// TEST1
// Check that -list with various archive types can find all targets
// RUN: clang-offload-bundler -list -type=ao -input=%t_bundled.a > %t_list_ao.txt
// RUN: clang-offload-bundler -list -type=aoo -input=%t_bundled.a > %t_list_aoo.txt
// RUN: FileCheck --check-prefixes=CHECK-LIST < %t_list_ao.txt %s
// RUN: FileCheck --check-prefixes=CHECK-LIST < %t_list_aoo.txt %s

// CHECK-LIST-DAG: sycl-spir64-unknown-unknown
// CHECK-LIST-DAG: host-x86_64-unknown-linux-gnu
// CHECK-LIST-DAG: host-spir64-unknown-unknown
// CHECK-LIST-DAG: host-spir64_gen
// CHECK-LIST-DAG: host-spir64_x86_64

// RUN: wc -l %t_list_ao.txt | FileCheck --check-prefixes=CHECK-LIST-LENGTH %s
// RUN: wc -l %t_list_aoo.txt | FileCheck --check-prefixes=CHECK-LIST-LENGTH %s

// CHECK-LIST-LENGTH: 5

////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////
// TEST2
// Test -check-section
// RUN: clang-offload-bundler -check-section -type=ao -input=%t_bundled.a -targets=sycl-spir64-unknown-unknown
// RUN: clang-offload-bundler -check-section -type=ao -input=%t_bundled.a -targets=host-x86_64-unknown-linux-gnu
// RUN: clang-offload-bundler -check-section -type=ao -input=%t_bundled.a -targets=host-spir64-unknown-unknown
// RUN: clang-offload-bundler -check-section -type=ao -input=%t_bundled.a -targets=host-spir64_gen
// RUN: clang-offload-bundler -check-section -type=ao -input=%t_bundled.a -targets=host-spir64_x86_64
// RUN: not clang-offload-bundler -check-section -type=ao -input=%t_bundled.a -targets=sycl-spir64-unknown-unknown-a
// RUN: not clang-offload-bundler -check-section -type=ao -input=%t_bundled.a -targets=host-x86_64-unknown-linux-gnu-b
// RUN: not clang-offload-bundler -check-section -type=ao -input=%t_bundled.a -targets=host-spir64-unknown-unknown-c
// RUN: not clang-offload-bundler -check-section -type=ao -input=%t_bundled.a -targets=host-spir64_gen-d
// RUN: not clang-offload-bundler -check-section -type=ao -input=%t_bundled.a -targets=host-spir64_x86_64-e
// RUN: clang-offload-bundler -check-section -type=aoo -input=%t_bundled.a -targets=sycl-spir64-unknown-unknown
// RUN: clang-offload-bundler -check-section -type=aoo -input=%t_bundled.a -targets=host-x86_64-unknown-linux-gnu
// RUN: clang-offload-bundler -check-section -type=aoo -input=%t_bundled.a -targets=host-spir64-unknown-unknown
// RUN: clang-offload-bundler -check-section -type=aoo -input=%t_bundled.a -targets=host-spir64_gen
// RUN: clang-offload-bundler -check-section -type=aoo -input=%t_bundled.a -targets=host-spir64_x86_64
// RUN: not clang-offload-bundler -check-section -type=aoo -input=%t_bundled.a -targets=sycl-spir64-unknown-unknown-a
// RUN: not clang-offload-bundler -check-section -type=aoo -input=%t_bundled.a -targets=host-x86_64-unknown-linux-gnu-b
// RUN: not clang-offload-bundler -check-section -type=aoo -input=%t_bundled.a -targets=host-spir64-unknown-unknown-c
// RUN: not clang-offload-bundler -check-section -type=aoo -input=%t_bundled.a -targets=host-spir64_gen-d
// RUN: not clang-offload-bundler -check-section -type=aoo -input=%t_bundled.a -targets=host-spir64_x86_64-e

////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////
// Unbundle object file to use as a reference result
// RUN: clang-offload-bundler -unbundle -type=o -input=%t_bundled.o -targets=sycl-spir64-unknown-unknown -output=%t_unbundled_A.o
// RUN: clang-offload-bundler -unbundle -type=o -input=%t_bundled.o -targets=host-x86_64-unknown-linux-gnu -output=%t_unbundled_B.o

////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////
// TEST3
// Test archive unbundling
// RUN: clang-offload-bundler -unbundle -type=aoo -input=%t_bundled.a -targets=sycl-spir64-unknown-unknown -output=%t_list1.txt
// RUN: clang-offload-bundler -unbundle -type=aoo -input=%t_bundled.a -targets=host-x86_64-unknown-linux-gnu -output=%t_list2.txt
// RUN: clang-offload-bundler -unbundle -type=aoo -input=%t_bundled.a -targets=host-spir64-unknown-unknown -output=%t_list3.txt
// RUN: clang-offload-bundler -unbundle -type=aoo -input=%t_bundled.a -targets=host-spir64_gen -output=%t_list4.txt
// RUN: clang-offload-bundler -unbundle -type=aoo -input=%t_bundled.a -targets=host-spir64_x86_64 -output=%t_list5.txt

// RUN: cmp %t_unbundled_A.o `cat %t_list1.txt`
// RUN: cmp %t_unbundled_B.o `cat %t_list2.txt`
// RUN: cmp %t1.bc `cat %t_list3.txt`
// RUN: cmp %t2.bc `cat %t_list4.txt`
// RUN: cmp %t3.bc `cat %t_list5.txt`

////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////
// TEST4
// Test archive unbundling for multiple targets
// RUN: clang-offload-bundler -unbundle -type=aoo -input=%t_bundled.a -targets=sycl-spir64-unknown-unknown,host-spir64_gen -output=%t_listA.txt -output=%t_listB.txt

// RUN: cmp %t_unbundled_A.o `cat %t_listA.txt`
// RUN: cmp %t2.bc `cat %t_listB.txt`

#include <sycl/sycl.hpp>

SYCL_EXTERNAL int foo(int x) {

#ifdef TYPE1
return x+13;
#elif TYPE2
return x+17;
#elif TYPE3
return x+23;
#else
return x+29;
#endif
}
Loading