Skip to content

[OffloadBundler] Compress bundles over 4GB #122307

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 1 commit into from
Jan 21, 2025

Conversation

yxsamliu
Copy link
Collaborator

@yxsamliu yxsamliu commented Jan 9, 2025

Added initial support for version 3 of the compressed offload bundle format, which uses 64-bit fields for Total File Size and Uncompressed Binary Size. This enables support for files larger than 4GB. The support is currently experimental and can be enabled by setting the environment variable COMPRESSED_BUNDLE_FORMAT_VERSION=3.

@llvmbot llvmbot added clang Clang issues not falling into any other category clang:driver 'clang' and 'clang++' user-facing binaries. Not 'clang-cl' labels Jan 9, 2025
@llvmbot
Copy link
Member

llvmbot commented Jan 9, 2025

@llvm/pr-subscribers-clang-driver

@llvm/pr-subscribers-clang

Author: Yaxun (Sam) Liu (yxsamliu)

Changes

Added initial support for version 3 of the compressed offload bundle format, which uses 64-bit fields for Total File Size and Uncompressed Binary Size. This enables support for files larger than 4GB. The support is currently experimental and can be enabled by setting the environment variable COMPRESSED_BUNDLE_FORMAT_VERSION=3.


Full diff: https://github.com/llvm/llvm-project/pull/122307.diff

5 Files Affected:

  • (modified) clang/docs/ClangOffloadBundler.rst (+2)
  • (modified) clang/docs/ReleaseNotes.rst (+1)
  • (modified) clang/include/clang/Driver/OffloadBundler.h (+39-11)
  • (modified) clang/lib/Driver/OffloadBundler.cpp (+106-32)
  • (modified) clang/test/Driver/clang-offload-bundler-zlib.c (+24)
diff --git a/clang/docs/ClangOffloadBundler.rst b/clang/docs/ClangOffloadBundler.rst
index 3c241027d405ca..bceb4060992fc8 100644
--- a/clang/docs/ClangOffloadBundler.rst
+++ b/clang/docs/ClangOffloadBundler.rst
@@ -542,3 +542,5 @@ The compressed offload bundle begins with a header followed by the compressed bi
 
 - **Compressed Data**:
     The actual compressed binary data follows the header. Its size can be inferred from the total size of the file minus the header size.
+
+    > **Note**: Version 3 of the format is under development. It uses 64-bit fields for Total File Size and Uncompressed Binary Size to support files larger than 4GB. To experiment with version 3, set the environment variable `COMPRESSED_BUNDLE_FORMAT_VERSION=3`. This support is experimental and not recommended for production use.
\ No newline at end of file
diff --git a/clang/docs/ReleaseNotes.rst b/clang/docs/ReleaseNotes.rst
index 93915e5db7d131..ea3c27bde40fb8 100644
--- a/clang/docs/ReleaseNotes.rst
+++ b/clang/docs/ReleaseNotes.rst
@@ -1053,6 +1053,7 @@ RISC-V Support
 
 CUDA/HIP Language Changes
 ^^^^^^^^^^^^^^^^^^^^^^^^^
+* Added initial support for version 3 of the compressed offload bundle format, which uses 64-bit fields for Total File Size and Uncompressed Binary Size. This enables support for files larger than 4GB. The support is currently experimental and can be enabled by setting the environment variable `COMPRESSED_BUNDLE_FORMAT_VERSION=3`.
 
 CUDA Support
 ^^^^^^^^^^^^
diff --git a/clang/include/clang/Driver/OffloadBundler.h b/clang/include/clang/Driver/OffloadBundler.h
index 57ecbdcb7d040e..31c11e25ecd9f2 100644
--- a/clang/include/clang/Driver/OffloadBundler.h
+++ b/clang/include/clang/Driver/OffloadBundler.h
@@ -39,6 +39,7 @@ class OffloadBundlerConfig {
   bool Verbose = false;
   llvm::compression::Format CompressionFormat;
   int CompressionLevel;
+  uint16_t CompressedBundleVersion;
 
   unsigned BundleAlignment = 1;
   unsigned HostInputIndex = ~0u;
@@ -100,36 +101,63 @@ struct OffloadTargetInfo {
 // - Version (2 bytes)
 // - Compression Method (2 bytes) - Uses the values from
 // llvm::compression::Format.
-// - Total file size (4 bytes). Available in version 2 and above.
-// - Uncompressed Size (4 bytes).
+// - Total file size (4 bytes in V2, 8 bytes in V3).
+// - Uncompressed Size (4 bytes in V1/V2, 8 bytes in V3).
 // - Truncated MD5 Hash (8 bytes).
 // - Compressed Data (variable length).
-
 class CompressedOffloadBundle {
 private:
   static inline const size_t MagicSize = 4;
   static inline const size_t VersionFieldSize = sizeof(uint16_t);
   static inline const size_t MethodFieldSize = sizeof(uint16_t);
-  static inline const size_t FileSizeFieldSize = sizeof(uint32_t);
-  static inline const size_t UncompressedSizeFieldSize = sizeof(uint32_t);
+  // Legacy size fields for V1/V2
+  static inline const size_t FileSizeFieldSizeV2 = sizeof(uint32_t);
+  static inline const size_t UncompressedSizeFieldSizeV2 = sizeof(uint32_t);
+  // New size fields for V3
+  static inline const size_t FileSizeFieldSizeV3 = sizeof(uint64_t);
+  static inline const size_t UncompressedSizeFieldSizeV3 = sizeof(uint64_t);
   static inline const size_t HashFieldSize = sizeof(uint64_t);
+
+  // Keep V1 header size for backward compatibility
   static inline const size_t V1HeaderSize =
       MagicSize + VersionFieldSize + MethodFieldSize +
-      UncompressedSizeFieldSize + HashFieldSize;
+      UncompressedSizeFieldSizeV2 + HashFieldSize;
+
+  // Keep V2 header size for backward compatibility
   static inline const size_t V2HeaderSize =
-      MagicSize + VersionFieldSize + FileSizeFieldSize + MethodFieldSize +
-      UncompressedSizeFieldSize + HashFieldSize;
+      MagicSize + VersionFieldSize + FileSizeFieldSizeV2 + MethodFieldSize +
+      UncompressedSizeFieldSizeV2 + HashFieldSize;
+
+  // Add V3 header size with 64-bit fields
+  static inline const size_t V3HeaderSize =
+      MagicSize + VersionFieldSize + FileSizeFieldSizeV3 + MethodFieldSize +
+      UncompressedSizeFieldSizeV3 + HashFieldSize;
+
   static inline const llvm::StringRef MagicNumber = "CCOB";
-  static inline const uint16_t Version = 2;
 
 public:
+  static inline const uint16_t DefaultVersion = 2;
+
+  // Helper method to get header size based on version
+  static size_t getHeaderSize(uint16_t Version) {
+    switch (Version) {
+    case 1:
+      return V1HeaderSize;
+    case 2:
+      return V2HeaderSize;
+    case 3:
+      return V3HeaderSize;
+    default:
+      llvm_unreachable("Unsupported version");
+    }
+  }
+
   static llvm::Expected<std::unique_ptr<llvm::MemoryBuffer>>
   compress(llvm::compression::Params P, const llvm::MemoryBuffer &Input,
-           bool Verbose = false);
+           uint16_t Version, bool Verbose = false);
   static llvm::Expected<std::unique_ptr<llvm::MemoryBuffer>>
   decompress(const llvm::MemoryBuffer &Input, bool Verbose = false);
 };
-
 } // namespace clang
 
 #endif // LLVM_CLANG_DRIVER_OFFLOADBUNDLER_H
diff --git a/clang/lib/Driver/OffloadBundler.cpp b/clang/lib/Driver/OffloadBundler.cpp
index 87d7303d938c90..f67fe50bcc26f8 100644
--- a/clang/lib/Driver/OffloadBundler.cpp
+++ b/clang/lib/Driver/OffloadBundler.cpp
@@ -926,7 +926,8 @@ CreateFileHandler(MemoryBuffer &FirstInput,
                            "'" + FilesType + "': invalid file type specified");
 }
 
-OffloadBundlerConfig::OffloadBundlerConfig() {
+OffloadBundlerConfig::OffloadBundlerConfig()
+    : CompressedBundleVersion(CompressedOffloadBundle::DefaultVersion) {
   if (llvm::compression::zstd::isAvailable()) {
     CompressionFormat = llvm::compression::Format::Zstd;
     // Compression level 3 is usually sufficient for zstd since long distance
@@ -942,16 +943,13 @@ OffloadBundlerConfig::OffloadBundlerConfig() {
       llvm::sys::Process::GetEnv("OFFLOAD_BUNDLER_IGNORE_ENV_VAR");
   if (IgnoreEnvVarOpt.has_value() && IgnoreEnvVarOpt.value() == "1")
     return;
-
   auto VerboseEnvVarOpt = llvm::sys::Process::GetEnv("OFFLOAD_BUNDLER_VERBOSE");
   if (VerboseEnvVarOpt.has_value())
     Verbose = VerboseEnvVarOpt.value() == "1";
-
   auto CompressEnvVarOpt =
       llvm::sys::Process::GetEnv("OFFLOAD_BUNDLER_COMPRESS");
   if (CompressEnvVarOpt.has_value())
     Compress = CompressEnvVarOpt.value() == "1";
-
   auto CompressionLevelEnvVarOpt =
       llvm::sys::Process::GetEnv("OFFLOAD_BUNDLER_COMPRESSION_LEVEL");
   if (CompressionLevelEnvVarOpt.has_value()) {
@@ -964,6 +962,26 @@ OffloadBundlerConfig::OffloadBundlerConfig() {
           << "Warning: Invalid value for OFFLOAD_BUNDLER_COMPRESSION_LEVEL: "
           << CompressionLevelStr.str() << ". Ignoring it.\n";
   }
+  auto CompressedBundleFormatVersionOpt =
+      llvm::sys::Process::GetEnv("COMPRESSED_BUNDLE_FORMAT_VERSION");
+  if (CompressedBundleFormatVersionOpt.has_value()) {
+    llvm::StringRef VersionStr = CompressedBundleFormatVersionOpt.value();
+    uint16_t Version;
+    if (!VersionStr.getAsInteger(10, Version)) {
+      if (Version >= 2 && Version <= 3)
+        CompressedBundleVersion = Version;
+      else
+        llvm::errs()
+            << "Warning: Invalid value for COMPRESSED_BUNDLE_FORMAT_VERSION: "
+            << VersionStr.str()
+            << ". Valid values are 2 or 3. Using default version "
+            << CompressedBundleVersion << ".\n";
+    } else
+      llvm::errs()
+          << "Warning: Invalid value for COMPRESSED_BUNDLE_FORMAT_VERSION: "
+          << VersionStr.str() << ". Using default version "
+          << CompressedBundleVersion << ".\n";
+  }
 }
 
 // Utility function to format numbers with commas
@@ -980,12 +998,11 @@ static std::string formatWithCommas(unsigned long long Value) {
 llvm::Expected<std::unique_ptr<llvm::MemoryBuffer>>
 CompressedOffloadBundle::compress(llvm::compression::Params P,
                                   const llvm::MemoryBuffer &Input,
-                                  bool Verbose) {
+                                  uint16_t Version, bool Verbose) {
   if (!llvm::compression::zstd::isAvailable() &&
       !llvm::compression::zlib::isAvailable())
     return createStringError(llvm::inconvertibleErrorCode(),
                              "Compression not supported");
-
   llvm::Timer HashTimer("Hash Calculation Timer", "Hash calculation time",
                         ClangOffloadBundlerTimerGroup);
   if (Verbose)
@@ -1002,7 +1019,6 @@ CompressedOffloadBundle::compress(llvm::compression::Params P,
   auto BufferUint8 = llvm::ArrayRef<uint8_t>(
       reinterpret_cast<const uint8_t *>(Input.getBuffer().data()),
       Input.getBuffer().size());
-
   llvm::Timer CompressTimer("Compression Timer", "Compression time",
                             ClangOffloadBundlerTimerGroup);
   if (Verbose)
@@ -1012,11 +1028,31 @@ CompressedOffloadBundle::compress(llvm::compression::Params P,
     CompressTimer.stopTimer();
 
   uint16_t CompressionMethod = static_cast<uint16_t>(P.format);
-  uint32_t UncompressedSize = Input.getBuffer().size();
-  uint32_t TotalFileSize = MagicNumber.size() + sizeof(TotalFileSize) +
-                           sizeof(Version) + sizeof(CompressionMethod) +
-                           sizeof(UncompressedSize) + sizeof(TruncatedHash) +
-                           CompressedBuffer.size();
+
+  // Store sizes in 64-bit variables first
+  uint64_t UncompressedSize64 = Input.getBuffer().size();
+  uint64_t TotalFileSize64;
+
+  // Calculate total file size based on version
+  if (Version == 2) {
+    // For V2, ensure the sizes don't exceed 32-bit limit
+    if (UncompressedSize64 > std::numeric_limits<uint32_t>::max())
+      return createStringError(llvm::inconvertibleErrorCode(),
+                               "Uncompressed size exceeds version 2 limit");
+    if ((MagicNumber.size() + sizeof(uint32_t) + sizeof(Version) +
+         sizeof(CompressionMethod) + sizeof(uint32_t) + sizeof(TruncatedHash) +
+         CompressedBuffer.size()) > std::numeric_limits<uint32_t>::max())
+      return createStringError(llvm::inconvertibleErrorCode(),
+                               "Total file size exceeds version 2 limit");
+
+    TotalFileSize64 = MagicNumber.size() + sizeof(uint32_t) + sizeof(Version) +
+                      sizeof(CompressionMethod) + sizeof(uint32_t) +
+                      sizeof(TruncatedHash) + CompressedBuffer.size();
+  } else { // Version 3
+    TotalFileSize64 = MagicNumber.size() + sizeof(uint64_t) + sizeof(Version) +
+                      sizeof(CompressionMethod) + sizeof(uint64_t) +
+                      sizeof(TruncatedHash) + CompressedBuffer.size();
+  }
 
   SmallVector<char, 0> FinalBuffer;
   llvm::raw_svector_ostream OS(FinalBuffer);
@@ -1024,10 +1060,22 @@ CompressedOffloadBundle::compress(llvm::compression::Params P,
   OS.write(reinterpret_cast<const char *>(&Version), sizeof(Version));
   OS.write(reinterpret_cast<const char *>(&CompressionMethod),
            sizeof(CompressionMethod));
-  OS.write(reinterpret_cast<const char *>(&TotalFileSize),
-           sizeof(TotalFileSize));
-  OS.write(reinterpret_cast<const char *>(&UncompressedSize),
-           sizeof(UncompressedSize));
+
+  // Write size fields according to version
+  if (Version == 2) {
+    uint32_t TotalFileSize32 = static_cast<uint32_t>(TotalFileSize64);
+    uint32_t UncompressedSize32 = static_cast<uint32_t>(UncompressedSize64);
+    OS.write(reinterpret_cast<const char *>(&TotalFileSize32),
+             sizeof(TotalFileSize32));
+    OS.write(reinterpret_cast<const char *>(&UncompressedSize32),
+             sizeof(UncompressedSize32));
+  } else { // Version 3
+    OS.write(reinterpret_cast<const char *>(&TotalFileSize64),
+             sizeof(TotalFileSize64));
+    OS.write(reinterpret_cast<const char *>(&UncompressedSize64),
+             sizeof(UncompressedSize64));
+  }
+
   OS.write(reinterpret_cast<const char *>(&TruncatedHash),
            sizeof(TruncatedHash));
   OS.write(reinterpret_cast<const char *>(CompressedBuffer.data()),
@@ -1037,18 +1085,17 @@ CompressedOffloadBundle::compress(llvm::compression::Params P,
     auto MethodUsed =
         P.format == llvm::compression::Format::Zstd ? "zstd" : "zlib";
     double CompressionRate =
-        static_cast<double>(UncompressedSize) / CompressedBuffer.size();
+        static_cast<double>(UncompressedSize64) / CompressedBuffer.size();
     double CompressionTimeSeconds = CompressTimer.getTotalTime().getWallTime();
     double CompressionSpeedMBs =
-        (UncompressedSize / (1024.0 * 1024.0)) / CompressionTimeSeconds;
-
+        (UncompressedSize64 / (1024.0 * 1024.0)) / CompressionTimeSeconds;
     llvm::errs() << "Compressed bundle format version: " << Version << "\n"
                  << "Total file size (including headers): "
-                 << formatWithCommas(TotalFileSize) << " bytes\n"
+                 << formatWithCommas(TotalFileSize64) << " bytes\n"
                  << "Compression method used: " << MethodUsed << "\n"
                  << "Compression level: " << P.level << "\n"
                  << "Binary size before compression: "
-                 << formatWithCommas(UncompressedSize) << " bytes\n"
+                 << formatWithCommas(UncompressedSize64) << " bytes\n"
                  << "Binary size after compression: "
                  << formatWithCommas(CompressedBuffer.size()) << " bytes\n"
                  << "Compression rate: "
@@ -1060,6 +1107,7 @@ CompressedOffloadBundle::compress(llvm::compression::Params P,
                  << "Truncated MD5 hash: "
                  << llvm::format_hex(TruncatedHash, 16) << "\n";
   }
+
   return llvm::MemoryBuffer::getMemBufferCopy(
       llvm::StringRef(FinalBuffer.data(), FinalBuffer.size()));
 }
@@ -1067,9 +1115,9 @@ CompressedOffloadBundle::compress(llvm::compression::Params P,
 llvm::Expected<std::unique_ptr<llvm::MemoryBuffer>>
 CompressedOffloadBundle::decompress(const llvm::MemoryBuffer &Input,
                                     bool Verbose) {
-
   StringRef Blob = Input.getBuffer();
 
+  // Check minimum header size (using V1 as it's the smallest)
   if (Blob.size() < V1HeaderSize)
     return llvm::MemoryBuffer::getMemBufferCopy(Blob);
 
@@ -1082,31 +1130,56 @@ CompressedOffloadBundle::decompress(const llvm::MemoryBuffer &Input,
 
   size_t CurrentOffset = MagicSize;
 
+  // Read version
   uint16_t ThisVersion;
   memcpy(&ThisVersion, Blob.data() + CurrentOffset, sizeof(uint16_t));
   CurrentOffset += VersionFieldSize;
 
+  // Verify header size based on version
+  if (ThisVersion >= 2 && ThisVersion <= 3) {
+    size_t RequiredSize = (ThisVersion == 2) ? V2HeaderSize : V3HeaderSize;
+    if (Blob.size() < RequiredSize)
+      return createStringError(inconvertibleErrorCode(),
+                               "Compressed bundle header size too small");
+  }
+
+  // Read compression method
   uint16_t CompressionMethod;
   memcpy(&CompressionMethod, Blob.data() + CurrentOffset, sizeof(uint16_t));
   CurrentOffset += MethodFieldSize;
 
-  uint32_t TotalFileSize;
+  // Read total file size (version 2+)
+  uint64_t TotalFileSize = 0;
   if (ThisVersion >= 2) {
-    if (Blob.size() < V2HeaderSize)
-      return createStringError(inconvertibleErrorCode(),
-                               "Compressed bundle header size too small");
-    memcpy(&TotalFileSize, Blob.data() + CurrentOffset, sizeof(uint32_t));
-    CurrentOffset += FileSizeFieldSize;
+    if (ThisVersion == 2) {
+      uint32_t TotalFileSize32;
+      memcpy(&TotalFileSize32, Blob.data() + CurrentOffset, sizeof(uint32_t));
+      TotalFileSize = TotalFileSize32;
+      CurrentOffset += FileSizeFieldSizeV2;
+    } else { // Version 3
+      memcpy(&TotalFileSize, Blob.data() + CurrentOffset, sizeof(uint64_t));
+      CurrentOffset += FileSizeFieldSizeV3;
+    }
   }
 
-  uint32_t UncompressedSize;
-  memcpy(&UncompressedSize, Blob.data() + CurrentOffset, sizeof(uint32_t));
-  CurrentOffset += UncompressedSizeFieldSize;
+  // Read uncompressed size
+  uint64_t UncompressedSize = 0;
+  if (ThisVersion <= 2) {
+    uint32_t UncompressedSize32;
+    memcpy(&UncompressedSize32, Blob.data() + CurrentOffset, sizeof(uint32_t));
+    UncompressedSize = UncompressedSize32;
+    CurrentOffset += UncompressedSizeFieldSizeV2;
+  } else { // Version 3
+    memcpy(&UncompressedSize, Blob.data() + CurrentOffset, sizeof(uint64_t));
+    CurrentOffset += UncompressedSizeFieldSizeV3;
+  }
 
+  // Read hash
   uint64_t StoredHash;
   memcpy(&StoredHash, Blob.data() + CurrentOffset, sizeof(uint64_t));
   CurrentOffset += HashFieldSize;
 
+  // Determine compression format
   llvm::compression::Format CompressionFormat;
   if (CompressionMethod ==
       static_cast<uint16_t>(llvm::compression::Format::Zlib))
@@ -1372,7 +1445,8 @@ Error OffloadBundler::BundleFiles() {
     auto CompressionResult = CompressedOffloadBundle::compress(
         {BundlerConfig.CompressionFormat, BundlerConfig.CompressionLevel,
          /*zstdEnableLdm=*/true},
-        *BufferMemory, BundlerConfig.Verbose);
+        *BufferMemory, BundlerConfig.CompressedBundleVersion,
+        BundlerConfig.Verbose);
     if (auto Error = CompressionResult.takeError())
       return Error;
 
diff --git a/clang/test/Driver/clang-offload-bundler-zlib.c b/clang/test/Driver/clang-offload-bundler-zlib.c
index 7e5857296756cb..4f1e63faeee69c 100644
--- a/clang/test/Driver/clang-offload-bundler-zlib.c
+++ b/clang/test/Driver/clang-offload-bundler-zlib.c
@@ -42,6 +42,30 @@
 // NOHOST-DAG: hip-amdgcn-amd-amdhsa--gfx906
 //
 
+// Check compression/decompression of offload bundle using version 3 format.
+//
+// RUN: env OFFLOAD_BUNDLER_COMPRESS=1 OFFLOAD_BUNDLER_VERBOSE=1 COMPRESSED_BUNDLE_FORMAT_VERSION=3 \
+// RUN:   clang-offload-bundler -type=bc -targets=hip-amdgcn-amd-amdhsa--gfx900,hip-amdgcn-amd-amdhsa--gfx906 \
+// RUN:   -input=%t.tgt1 -input=%t.tgt2 -output=%t.hip.bundle.bc 2>&1 | \
+// RUN:   FileCheck -check-prefix=COMPRESS %s
+// RUN: clang-offload-bundler -type=bc -list -input=%t.hip.bundle.bc | FileCheck -check-prefix=NOHOST %s
+// RUN: env OFFLOAD_BUNDLER_VERBOSE=1 \
+// RUN:   clang-offload-bundler -type=bc -targets=hip-amdgcn-amd-amdhsa--gfx900,hip-amdgcn-amd-amdhsa--gfx906 \
+// RUN:   -output=%t.res.tgt1 -output=%t.res.tgt2 -input=%t.hip.bundle.bc -unbundle 2>&1 | \
+// RUN:   FileCheck -check-prefix=DECOMPRESS %s
+// RUN: diff %t.tgt1 %t.res.tgt1
+// RUN: diff %t.tgt2 %t.res.tgt2
+//
+// COMPRESS: Compressed bundle format version: 3
+// COMPRESS: Compression method used: zlib
+// COMPRESS: Compression level: 6
+// DECOMPRESS: Compressed bundle format version: 3
+// DECOMPRESS: Decompression method: zlib
+// DECOMPRESS: Hashes match: Yes
+// NOHOST-NOT: host-
+// NOHOST-DAG: hip-amdgcn-amd-amdhsa--gfx900
+// NOHOST-DAG: hip-amdgcn-amd-amdhsa--gfx906
+
 // Check -compression-level= option
 
 // RUN: clang-offload-bundler -type=bc -targets=hip-amdgcn-amd-amdhsa--gfx900,hip-amdgcn-amd-amdhsa--gfx906 \

Copy link
Contributor

@jhuber6 jhuber6 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I've never been a fan of this type of implicit behavior, since it will change spuriously depending on what the user wants.

I also wonder if we couldn't do something similar with ELF compression so we don't need to manually do all this stuff.

@yxsamliu
Copy link
Collaborator Author

I've never been a fan of this type of implicit behavior, since it will change spuriously depending on what the user wants.

I also wonder if we couldn't do something similar with ELF compression so we don't need to manually do all this stuff.

Linux only supports compressed debug info sections.

The env var is for debugging purpose only. After runtime support is mature, we will remove the env var and also eventually remove support of generation of old versions of compressed bundle format.

Added initial support for version 3 of the compressed offload bundle format,
which uses 64-bit fields for Total File Size and Uncompressed Binary Size.
This enables support for files larger than 4GB. The support is currently
experimental and can be enabled by setting the environment variable
`COMPRESSED_BUNDLE_FORMAT_VERSION=3`.
@yxsamliu
Copy link
Collaborator Author

ping

Copy link
Contributor

@jhuber6 jhuber6 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Seems fine. What's required to move HIP over to using the binary format natively by the way? Guessing we'd need to update a bunch of tools in the fork.

@yxsamliu
Copy link
Collaborator Author

Seems fine. What's required to move HIP over to using the binary format natively by the way? Guessing we'd need to update a bunch of tools in the fork.

need to teach comgr to load the new offload binary

@jhuber6
Copy link
Contributor

jhuber6 commented Jan 20, 2025

Seems fine. What's required to move HIP over to using the binary format natively by the way? Guessing we'd need to update a bunch of tools in the fork.

need to teach comgr to load the new offload binary

Should be easy enough, since there's a library function for that in LLVM. Is comgr the only location that would need to change? Nothing in ROCr or HIP runtime?

@yxsamliu
Copy link
Collaborator Author

Seems fine. What's required to move HIP over to using the binary format natively by the way? Guessing we'd need to update a bunch of tools in the fork.

need to teach comgr to load the new offload binary

Should be easy enough, since there's a library function for that in LLVM. Is comgr the only location that would need to change? Nothing in ROCr or HIP runtime?

currently HIP runtime loads fat binary by itself for non-compressed fat binary. It needs to switch to use comgr for loading fat binary. Some math libs compile assembly code to code objects then bundle them to fat binary. Such math libs need to switch to use tools for packing offload binary. Also, zstd compression of code objects for different GPU arch by taking advantage of redundancy in ISA is critical for math libraries. The offload binary needs to support zstd compression.

@jhuber6
Copy link
Contributor

jhuber6 commented Jan 20, 2025

currently HIP runtime loads fat binary by itself for non-compressed fat binary. It needs to switch to use comgr for loading fat binary. Some math libs compile assembly code to code objects then bundle them to fat binary. Such math libs need to switch to use tools for packing offload binary. Also, zstd compression of code objects for different GPU arch by taking advantage of redundancy in ISA is critical for math libraries. The offload binary needs to support zstd compression.

We could probably just keep backwards compatibility by checking file magic now that the bundles have them. Does it do anything special during optimization? The offload binary is just a big blob, basically looking like this. I figured any optimization algorithm would be able to figure out the repetition itself.

<header>
<big binary blob>
<header>
<big binary blob>
...
<header>
<big binary blob>

@yxsamliu yxsamliu merged commit 4e2efc3 into llvm:main Jan 21, 2025
7 of 9 checks passed
@llvm-ci
Copy link
Collaborator

llvm-ci commented Jan 21, 2025

LLVM Buildbot has detected a new failure on builder llvm-clang-x86_64-sie-ubuntu-fast running on sie-linux-worker while building clang at step 6 "test-build-unified-tree-check-all".

Full details are available at: https://lab.llvm.org/buildbot/#/builders/144/builds/16114

Here is the relevant piece of the build log for the reference
Step 6 (test-build-unified-tree-check-all) failure: test (failure)
******************** TEST 'Clang :: Driver/clang-offload-bundler-zlib.c' FAILED ********************
Exit Code: 1

Command Output (stderr):
--
RUN: at line 7: /home/buildbot/buildbot-root/llvm-clang-x86_64-sie-ubuntu-fast/build/bin/clang -O0 -target x86_64-scei-ps4 /home/buildbot/buildbot-root/llvm-clang-x86_64-sie-ubuntu-fast/llvm-project/clang/test/Driver/clang-offload-bundler-zlib.c -c -emit-llvm -o /home/buildbot/buildbot-root/llvm-clang-x86_64-sie-ubuntu-fast/build/tools/clang/test/Driver/Output/clang-offload-bundler-zlib.c.tmp.bc
+ /home/buildbot/buildbot-root/llvm-clang-x86_64-sie-ubuntu-fast/build/bin/clang -O0 -target x86_64-scei-ps4 /home/buildbot/buildbot-root/llvm-clang-x86_64-sie-ubuntu-fast/llvm-project/clang/test/Driver/clang-offload-bundler-zlib.c -c -emit-llvm -o /home/buildbot/buildbot-root/llvm-clang-x86_64-sie-ubuntu-fast/build/tools/clang/test/Driver/Output/clang-offload-bundler-zlib.c.tmp.bc
RUN: at line 12: touch /home/buildbot/buildbot-root/llvm-clang-x86_64-sie-ubuntu-fast/build/tools/clang/test/Driver/Output/clang-offload-bundler-zlib.c.tmp.empty
+ touch /home/buildbot/buildbot-root/llvm-clang-x86_64-sie-ubuntu-fast/build/tools/clang/test/Driver/Output/clang-offload-bundler-zlib.c.tmp.empty
RUN: at line 17: echo 'Content of device file 1' > /home/buildbot/buildbot-root/llvm-clang-x86_64-sie-ubuntu-fast/build/tools/clang/test/Driver/Output/clang-offload-bundler-zlib.c.tmp.tgt1
+ echo 'Content of device file 1'
RUN: at line 18: echo 'Content of device file 2' > /home/buildbot/buildbot-root/llvm-clang-x86_64-sie-ubuntu-fast/build/tools/clang/test/Driver/Output/clang-offload-bundler-zlib.c.tmp.tgt2
+ echo 'Content of device file 2'
RUN: at line 23: env OFFLOAD_BUNDLER_COMPRESS=1 OFFLOAD_BUNDLER_VERBOSE=1    clang-offload-bundler -type=bc -targets=hip-amdgcn-amd-amdhsa--gfx900,hip-amdgcn-amd-amdhsa--gfx906    -input=/home/buildbot/buildbot-root/llvm-clang-x86_64-sie-ubuntu-fast/build/tools/clang/test/Driver/Output/clang-offload-bundler-zlib.c.tmp.tgt1 -input=/home/buildbot/buildbot-root/llvm-clang-x86_64-sie-ubuntu-fast/build/tools/clang/test/Driver/Output/clang-offload-bundler-zlib.c.tmp.tgt2 -output=/home/buildbot/buildbot-root/llvm-clang-x86_64-sie-ubuntu-fast/build/tools/clang/test/Driver/Output/clang-offload-bundler-zlib.c.tmp.hip.bundle.bc 2>&1 |    /home/buildbot/buildbot-root/llvm-clang-x86_64-sie-ubuntu-fast/build/bin/FileCheck -check-prefix=COMPRESS /home/buildbot/buildbot-root/llvm-clang-x86_64-sie-ubuntu-fast/llvm-project/clang/test/Driver/clang-offload-bundler-zlib.c
+ env OFFLOAD_BUNDLER_COMPRESS=1 OFFLOAD_BUNDLER_VERBOSE=1 clang-offload-bundler -type=bc -targets=hip-amdgcn-amd-amdhsa--gfx900,hip-amdgcn-amd-amdhsa--gfx906 -input=/home/buildbot/buildbot-root/llvm-clang-x86_64-sie-ubuntu-fast/build/tools/clang/test/Driver/Output/clang-offload-bundler-zlib.c.tmp.tgt1 -input=/home/buildbot/buildbot-root/llvm-clang-x86_64-sie-ubuntu-fast/build/tools/clang/test/Driver/Output/clang-offload-bundler-zlib.c.tmp.tgt2 -output=/home/buildbot/buildbot-root/llvm-clang-x86_64-sie-ubuntu-fast/build/tools/clang/test/Driver/Output/clang-offload-bundler-zlib.c.tmp.hip.bundle.bc
+ /home/buildbot/buildbot-root/llvm-clang-x86_64-sie-ubuntu-fast/build/bin/FileCheck -check-prefix=COMPRESS /home/buildbot/buildbot-root/llvm-clang-x86_64-sie-ubuntu-fast/llvm-project/clang/test/Driver/clang-offload-bundler-zlib.c
�[1m/home/buildbot/buildbot-root/llvm-clang-x86_64-sie-ubuntu-fast/llvm-project/clang/test/Driver/clang-offload-bundler-zlib.c:59:14: �[0m�[0;1;31merror: �[0m�[1mCOMPRESS: expected string not found in input
�[0m// COMPRESS: Compressed bundle format version: 3
�[0;1;32m             ^
�[0m�[1m<stdin>:4:21: �[0m�[0;1;30mnote: �[0m�[1mscanning from here
�[0mCompression level: 6
�[0;1;32m                    ^
�[0m�[1m<stdin>:7:1: �[0m�[0;1;30mnote: �[0m�[1mpossible intended match here
�[0mCompression rate: 1.86
�[0;1;32m^
�[0m
Input file: <stdin>
Check file: /home/buildbot/buildbot-root/llvm-clang-x86_64-sie-ubuntu-fast/llvm-project/clang/test/Driver/clang-offload-bundler-zlib.c

-dump-input=help explains the following input dump.

Input was:
<<<<<<
�[1m�[0m�[0;1;30m            1: �[0m�[1m�[0;1;46mCompressed bundle format version: 2 �[0m
�[0;1;30m            2: �[0m�[1m�[0;1;46mTotal file size (including headers): 125 bytes �[0m
�[0;1;30m            3: �[0m�[1m�[0;1;46m�[0mCompression method used: zlib�[0;1;46m �[0m
�[0;1;32mcheck:36       ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~
�[0m�[0;1;30m            4: �[0m�[1m�[0;1;46m�[0mCompression level: 6�[0;1;46m �[0m
�[0;1;32mcheck:37       ^~~~~~~~~~~~~~~~~~~~
�[0m�[0;1;31mcheck:59'0                         X error: no match found
�[0m�[0;1;30m            5: �[0m�[1m�[0;1;46mBinary size before compression: 188 bytes �[0m
�[0;1;31mcheck:59'0     ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
�[0m�[0;1;30m            6: �[0m�[1m�[0;1;46mBinary size after compression: 101 bytes �[0m
�[0;1;31mcheck:59'0     ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
�[0m�[0;1;30m            7: �[0m�[1m�[0;1;46mCompression rate: 1.86 �[0m
�[0;1;31mcheck:59'0     ~~~~~~~~~~~~~~~~~~~~~~~
�[0m�[0;1;35mcheck:59'1     ?                       possible intended match
�[0m�[0;1;30m            8: �[0m�[1m�[0;1;46mCompression ratio: 53.72% �[0m
�[0;1;31mcheck:59'0     ~~~~~~~~~~~~~~~~~~~~~~~~~~
�[0m�[0;1;30m            9: �[0m�[1m�[0;1;46mCompression speed: 1.50 MB/s �[0m
...

@llvm-ci
Copy link
Collaborator

llvm-ci commented Jan 21, 2025

LLVM Buildbot has detected a new failure on builder arc-builder running on arc-worker while building clang at step 6 "test-build-unified-tree-check-all".

Full details are available at: https://lab.llvm.org/buildbot/#/builders/3/builds/10570

Here is the relevant piece of the build log for the reference
Step 6 (test-build-unified-tree-check-all) failure: test (failure)
******************** TEST 'Clang :: Driver/clang-offload-bundler-zlib.c' FAILED ********************
Exit Code: 1

Command Output (stderr):
--
RUN: at line 7: /buildbot/worker/arc-folder/build/bin/clang -O0 -target x86_64-unknown-linux-gnu /buildbot/worker/arc-folder/llvm-project/clang/test/Driver/clang-offload-bundler-zlib.c -c -emit-llvm -o /buildbot/worker/arc-folder/build/tools/clang/test/Driver/Output/clang-offload-bundler-zlib.c.tmp.bc
+ /buildbot/worker/arc-folder/build/bin/clang -O0 -target x86_64-unknown-linux-gnu /buildbot/worker/arc-folder/llvm-project/clang/test/Driver/clang-offload-bundler-zlib.c -c -emit-llvm -o /buildbot/worker/arc-folder/build/tools/clang/test/Driver/Output/clang-offload-bundler-zlib.c.tmp.bc
RUN: at line 12: touch /buildbot/worker/arc-folder/build/tools/clang/test/Driver/Output/clang-offload-bundler-zlib.c.tmp.empty
+ touch /buildbot/worker/arc-folder/build/tools/clang/test/Driver/Output/clang-offload-bundler-zlib.c.tmp.empty
RUN: at line 17: echo 'Content of device file 1' > /buildbot/worker/arc-folder/build/tools/clang/test/Driver/Output/clang-offload-bundler-zlib.c.tmp.tgt1
+ echo 'Content of device file 1'
RUN: at line 18: echo 'Content of device file 2' > /buildbot/worker/arc-folder/build/tools/clang/test/Driver/Output/clang-offload-bundler-zlib.c.tmp.tgt2
+ echo 'Content of device file 2'
RUN: at line 23: env OFFLOAD_BUNDLER_COMPRESS=1 OFFLOAD_BUNDLER_VERBOSE=1    clang-offload-bundler -type=bc -targets=hip-amdgcn-amd-amdhsa--gfx900,hip-amdgcn-amd-amdhsa--gfx906    -input=/buildbot/worker/arc-folder/build/tools/clang/test/Driver/Output/clang-offload-bundler-zlib.c.tmp.tgt1 -input=/buildbot/worker/arc-folder/build/tools/clang/test/Driver/Output/clang-offload-bundler-zlib.c.tmp.tgt2 -output=/buildbot/worker/arc-folder/build/tools/clang/test/Driver/Output/clang-offload-bundler-zlib.c.tmp.hip.bundle.bc 2>&1 |    /buildbot/worker/arc-folder/build/bin/FileCheck -check-prefix=COMPRESS /buildbot/worker/arc-folder/llvm-project/clang/test/Driver/clang-offload-bundler-zlib.c
+ env OFFLOAD_BUNDLER_COMPRESS=1 OFFLOAD_BUNDLER_VERBOSE=1 clang-offload-bundler -type=bc -targets=hip-amdgcn-amd-amdhsa--gfx900,hip-amdgcn-amd-amdhsa--gfx906 -input=/buildbot/worker/arc-folder/build/tools/clang/test/Driver/Output/clang-offload-bundler-zlib.c.tmp.tgt1 -input=/buildbot/worker/arc-folder/build/tools/clang/test/Driver/Output/clang-offload-bundler-zlib.c.tmp.tgt2 -output=/buildbot/worker/arc-folder/build/tools/clang/test/Driver/Output/clang-offload-bundler-zlib.c.tmp.hip.bundle.bc
+ /buildbot/worker/arc-folder/build/bin/FileCheck -check-prefix=COMPRESS /buildbot/worker/arc-folder/llvm-project/clang/test/Driver/clang-offload-bundler-zlib.c
/buildbot/worker/arc-folder/llvm-project/clang/test/Driver/clang-offload-bundler-zlib.c:59:14: error: COMPRESS: expected string not found in input
// COMPRESS: Compressed bundle format version: 3
             ^
<stdin>:4:21: note: scanning from here
Compression level: 6
                    ^
<stdin>:7:1: note: possible intended match here
Compression rate: 1.86
^

Input file: <stdin>
Check file: /buildbot/worker/arc-folder/llvm-project/clang/test/Driver/clang-offload-bundler-zlib.c

-dump-input=help explains the following input dump.

Input was:
<<<<<<
            1: Compressed bundle format version: 2 
            2: Total file size (including headers): 125 bytes 
            3: Compression method used: zlib 
            4: Compression level: 6 
check:59'0                         X error: no match found
            5: Binary size before compression: 188 bytes 
check:59'0     ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
            6: Binary size after compression: 101 bytes 
check:59'0     ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
            7: Compression rate: 1.86 
check:59'0     ~~~~~~~~~~~~~~~~~~~~~~~
check:59'1     ?                       possible intended match
            8: Compression ratio: 53.72% 
check:59'0     ~~~~~~~~~~~~~~~~~~~~~~~~~~
            9: Compression speed: 1.46 MB/s 
check:59'0     ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
           10: Truncated MD5 hash: 0x2bcafb899a29dc94 
...

@llvm-ci
Copy link
Collaborator

llvm-ci commented Jan 21, 2025

LLVM Buildbot has detected a new failure on builder clang-aarch64-quick running on linaro-clang-aarch64-quick while building clang at step 5 "ninja check 1".

Full details are available at: https://lab.llvm.org/buildbot/#/builders/65/builds/10933

Here is the relevant piece of the build log for the reference
Step 5 (ninja check 1) failure: stage 1 checked (failure)
******************** TEST 'Clang :: Driver/clang-offload-bundler-zlib.c' FAILED ********************
Exit Code: 1

Command Output (stderr):
--
RUN: at line 7: /home/tcwg-buildbot/worker/clang-aarch64-quick/stage1/bin/clang -O0 -target aarch64-unknown-linux-gnu /home/tcwg-buildbot/worker/clang-aarch64-quick/llvm/clang/test/Driver/clang-offload-bundler-zlib.c -c -emit-llvm -o /home/tcwg-buildbot/worker/clang-aarch64-quick/stage1/tools/clang/test/Driver/Output/clang-offload-bundler-zlib.c.tmp.bc
+ /home/tcwg-buildbot/worker/clang-aarch64-quick/stage1/bin/clang -O0 -target aarch64-unknown-linux-gnu /home/tcwg-buildbot/worker/clang-aarch64-quick/llvm/clang/test/Driver/clang-offload-bundler-zlib.c -c -emit-llvm -o /home/tcwg-buildbot/worker/clang-aarch64-quick/stage1/tools/clang/test/Driver/Output/clang-offload-bundler-zlib.c.tmp.bc
RUN: at line 12: touch /home/tcwg-buildbot/worker/clang-aarch64-quick/stage1/tools/clang/test/Driver/Output/clang-offload-bundler-zlib.c.tmp.empty
+ touch /home/tcwg-buildbot/worker/clang-aarch64-quick/stage1/tools/clang/test/Driver/Output/clang-offload-bundler-zlib.c.tmp.empty
RUN: at line 17: echo 'Content of device file 1' > /home/tcwg-buildbot/worker/clang-aarch64-quick/stage1/tools/clang/test/Driver/Output/clang-offload-bundler-zlib.c.tmp.tgt1
+ echo 'Content of device file 1'
RUN: at line 18: echo 'Content of device file 2' > /home/tcwg-buildbot/worker/clang-aarch64-quick/stage1/tools/clang/test/Driver/Output/clang-offload-bundler-zlib.c.tmp.tgt2
+ echo 'Content of device file 2'
RUN: at line 23: env OFFLOAD_BUNDLER_COMPRESS=1 OFFLOAD_BUNDLER_VERBOSE=1    clang-offload-bundler -type=bc -targets=hip-amdgcn-amd-amdhsa--gfx900,hip-amdgcn-amd-amdhsa--gfx906    -input=/home/tcwg-buildbot/worker/clang-aarch64-quick/stage1/tools/clang/test/Driver/Output/clang-offload-bundler-zlib.c.tmp.tgt1 -input=/home/tcwg-buildbot/worker/clang-aarch64-quick/stage1/tools/clang/test/Driver/Output/clang-offload-bundler-zlib.c.tmp.tgt2 -output=/home/tcwg-buildbot/worker/clang-aarch64-quick/stage1/tools/clang/test/Driver/Output/clang-offload-bundler-zlib.c.tmp.hip.bundle.bc 2>&1 |    /home/tcwg-buildbot/worker/clang-aarch64-quick/stage1/bin/FileCheck -check-prefix=COMPRESS /home/tcwg-buildbot/worker/clang-aarch64-quick/llvm/clang/test/Driver/clang-offload-bundler-zlib.c
+ /home/tcwg-buildbot/worker/clang-aarch64-quick/stage1/bin/FileCheck -check-prefix=COMPRESS /home/tcwg-buildbot/worker/clang-aarch64-quick/llvm/clang/test/Driver/clang-offload-bundler-zlib.c
+ env OFFLOAD_BUNDLER_COMPRESS=1 OFFLOAD_BUNDLER_VERBOSE=1 clang-offload-bundler -type=bc -targets=hip-amdgcn-amd-amdhsa--gfx900,hip-amdgcn-amd-amdhsa--gfx906 -input=/home/tcwg-buildbot/worker/clang-aarch64-quick/stage1/tools/clang/test/Driver/Output/clang-offload-bundler-zlib.c.tmp.tgt1 -input=/home/tcwg-buildbot/worker/clang-aarch64-quick/stage1/tools/clang/test/Driver/Output/clang-offload-bundler-zlib.c.tmp.tgt2 -output=/home/tcwg-buildbot/worker/clang-aarch64-quick/stage1/tools/clang/test/Driver/Output/clang-offload-bundler-zlib.c.tmp.hip.bundle.bc
/home/tcwg-buildbot/worker/clang-aarch64-quick/llvm/clang/test/Driver/clang-offload-bundler-zlib.c:59:14: error: COMPRESS: expected string not found in input
// COMPRESS: Compressed bundle format version: 3
             ^
<stdin>:4:21: note: scanning from here
Compression level: 6
                    ^
<stdin>:7:1: note: possible intended match here
Compression rate: 1.86
^

Input file: <stdin>
Check file: /home/tcwg-buildbot/worker/clang-aarch64-quick/llvm/clang/test/Driver/clang-offload-bundler-zlib.c

-dump-input=help explains the following input dump.

Input was:
<<<<<<
            1: Compressed bundle format version: 2 
            2: Total file size (including headers): 125 bytes 
            3: Compression method used: zlib 
            4: Compression level: 6 
check:59'0                         X error: no match found
            5: Binary size before compression: 188 bytes 
check:59'0     ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
            6: Binary size after compression: 101 bytes 
check:59'0     ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
            7: Compression rate: 1.86 
check:59'0     ~~~~~~~~~~~~~~~~~~~~~~~
check:59'1     ?                       possible intended match
            8: Compression ratio: 53.72% 
check:59'0     ~~~~~~~~~~~~~~~~~~~~~~~~~~
            9: Compression speed: 0.54 MB/s 
check:59'0     ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
           10: Truncated MD5 hash: 0x2bcafb899a29dc94 
...

@llvm-ci
Copy link
Collaborator

llvm-ci commented Jan 21, 2025

LLVM Buildbot has detected a new failure on builder clang-armv8-quick running on linaro-clang-armv8-quick while building clang at step 5 "ninja check 1".

Full details are available at: https://lab.llvm.org/buildbot/#/builders/154/builds/10570

Here is the relevant piece of the build log for the reference
Step 5 (ninja check 1) failure: stage 1 checked (failure)
******************** TEST 'Clang :: Driver/clang-offload-bundler-zlib.c' FAILED ********************
Exit Code: 1

Command Output (stderr):
--
RUN: at line 7: /home/tcwg-buildbot/worker/clang-armv8-quick/stage1/bin/clang -O0 -target armv8l-unknown-linux-gnueabihf /home/tcwg-buildbot/worker/clang-armv8-quick/llvm/clang/test/Driver/clang-offload-bundler-zlib.c -c -emit-llvm -o /home/tcwg-buildbot/worker/clang-armv8-quick/stage1/tools/clang/test/Driver/Output/clang-offload-bundler-zlib.c.tmp.bc
+ /home/tcwg-buildbot/worker/clang-armv8-quick/stage1/bin/clang -O0 -target armv8l-unknown-linux-gnueabihf /home/tcwg-buildbot/worker/clang-armv8-quick/llvm/clang/test/Driver/clang-offload-bundler-zlib.c -c -emit-llvm -o /home/tcwg-buildbot/worker/clang-armv8-quick/stage1/tools/clang/test/Driver/Output/clang-offload-bundler-zlib.c.tmp.bc
RUN: at line 12: touch /home/tcwg-buildbot/worker/clang-armv8-quick/stage1/tools/clang/test/Driver/Output/clang-offload-bundler-zlib.c.tmp.empty
+ touch /home/tcwg-buildbot/worker/clang-armv8-quick/stage1/tools/clang/test/Driver/Output/clang-offload-bundler-zlib.c.tmp.empty
RUN: at line 17: echo 'Content of device file 1' > /home/tcwg-buildbot/worker/clang-armv8-quick/stage1/tools/clang/test/Driver/Output/clang-offload-bundler-zlib.c.tmp.tgt1
+ echo 'Content of device file 1'
RUN: at line 18: echo 'Content of device file 2' > /home/tcwg-buildbot/worker/clang-armv8-quick/stage1/tools/clang/test/Driver/Output/clang-offload-bundler-zlib.c.tmp.tgt2
+ echo 'Content of device file 2'
RUN: at line 23: env OFFLOAD_BUNDLER_COMPRESS=1 OFFLOAD_BUNDLER_VERBOSE=1    clang-offload-bundler -type=bc -targets=hip-amdgcn-amd-amdhsa--gfx900,hip-amdgcn-amd-amdhsa--gfx906    -input=/home/tcwg-buildbot/worker/clang-armv8-quick/stage1/tools/clang/test/Driver/Output/clang-offload-bundler-zlib.c.tmp.tgt1 -input=/home/tcwg-buildbot/worker/clang-armv8-quick/stage1/tools/clang/test/Driver/Output/clang-offload-bundler-zlib.c.tmp.tgt2 -output=/home/tcwg-buildbot/worker/clang-armv8-quick/stage1/tools/clang/test/Driver/Output/clang-offload-bundler-zlib.c.tmp.hip.bundle.bc 2>&1 |    /home/tcwg-buildbot/worker/clang-armv8-quick/stage1/bin/FileCheck -check-prefix=COMPRESS /home/tcwg-buildbot/worker/clang-armv8-quick/llvm/clang/test/Driver/clang-offload-bundler-zlib.c
+ env OFFLOAD_BUNDLER_COMPRESS=1 OFFLOAD_BUNDLER_VERBOSE=1 clang-offload-bundler -type=bc -targets=hip-amdgcn-amd-amdhsa--gfx900,hip-amdgcn-amd-amdhsa--gfx906 -input=/home/tcwg-buildbot/worker/clang-armv8-quick/stage1/tools/clang/test/Driver/Output/clang-offload-bundler-zlib.c.tmp.tgt1 -input=/home/tcwg-buildbot/worker/clang-armv8-quick/stage1/tools/clang/test/Driver/Output/clang-offload-bundler-zlib.c.tmp.tgt2 -output=/home/tcwg-buildbot/worker/clang-armv8-quick/stage1/tools/clang/test/Driver/Output/clang-offload-bundler-zlib.c.tmp.hip.bundle.bc
+ /home/tcwg-buildbot/worker/clang-armv8-quick/stage1/bin/FileCheck -check-prefix=COMPRESS /home/tcwg-buildbot/worker/clang-armv8-quick/llvm/clang/test/Driver/clang-offload-bundler-zlib.c
/home/tcwg-buildbot/worker/clang-armv8-quick/llvm/clang/test/Driver/clang-offload-bundler-zlib.c:59:14: error: COMPRESS: expected string not found in input
// COMPRESS: Compressed bundle format version: 3
             ^
<stdin>:4:21: note: scanning from here
Compression level: 6
                    ^
<stdin>:7:1: note: possible intended match here
Compression rate: 1.86
^

Input file: <stdin>
Check file: /home/tcwg-buildbot/worker/clang-armv8-quick/llvm/clang/test/Driver/clang-offload-bundler-zlib.c

-dump-input=help explains the following input dump.

Input was:
<<<<<<
            1: Compressed bundle format version: 2 
            2: Total file size (including headers): 125 bytes 
            3: Compression method used: zlib 
            4: Compression level: 6 
check:59'0                         X error: no match found
            5: Binary size before compression: 188 bytes 
check:59'0     ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
            6: Binary size after compression: 101 bytes 
check:59'0     ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
            7: Compression rate: 1.86 
check:59'0     ~~~~~~~~~~~~~~~~~~~~~~~
check:59'1     ?                       possible intended match
            8: Compression ratio: 53.72% 
check:59'0     ~~~~~~~~~~~~~~~~~~~~~~~~~~
            9: Compression speed: 0.02 MB/s 
check:59'0     ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
           10: Truncated MD5 hash: 0x2bcafb899a29dc94 
...

@llvm-ci
Copy link
Collaborator

llvm-ci commented Jan 21, 2025

LLVM Buildbot has detected a new failure on builder llvm-clang-x86_64-gcc-ubuntu running on sie-linux-worker3 while building clang at step 6 "test-build-unified-tree-check-all".

Full details are available at: https://lab.llvm.org/buildbot/#/builders/174/builds/11719

Here is the relevant piece of the build log for the reference
Step 6 (test-build-unified-tree-check-all) failure: test (failure)
******************** TEST 'Clang :: Driver/clang-offload-bundler-zlib.c' FAILED ********************
Exit Code: 1

Command Output (stderr):
--
RUN: at line 7: /home/buildbot/buildbot-root/llvm-clang-x86_64-gcc-ubuntu/build/bin/clang -O0 -target x86_64-unknown-linux-gnu /home/buildbot/buildbot-root/llvm-clang-x86_64-gcc-ubuntu/llvm-project/clang/test/Driver/clang-offload-bundler-zlib.c -c -emit-llvm -o /home/buildbot/buildbot-root/llvm-clang-x86_64-gcc-ubuntu/build/tools/clang/test/Driver/Output/clang-offload-bundler-zlib.c.tmp.bc
+ /home/buildbot/buildbot-root/llvm-clang-x86_64-gcc-ubuntu/build/bin/clang -O0 -target x86_64-unknown-linux-gnu /home/buildbot/buildbot-root/llvm-clang-x86_64-gcc-ubuntu/llvm-project/clang/test/Driver/clang-offload-bundler-zlib.c -c -emit-llvm -o /home/buildbot/buildbot-root/llvm-clang-x86_64-gcc-ubuntu/build/tools/clang/test/Driver/Output/clang-offload-bundler-zlib.c.tmp.bc
RUN: at line 12: touch /home/buildbot/buildbot-root/llvm-clang-x86_64-gcc-ubuntu/build/tools/clang/test/Driver/Output/clang-offload-bundler-zlib.c.tmp.empty
+ touch /home/buildbot/buildbot-root/llvm-clang-x86_64-gcc-ubuntu/build/tools/clang/test/Driver/Output/clang-offload-bundler-zlib.c.tmp.empty
RUN: at line 17: echo 'Content of device file 1' > /home/buildbot/buildbot-root/llvm-clang-x86_64-gcc-ubuntu/build/tools/clang/test/Driver/Output/clang-offload-bundler-zlib.c.tmp.tgt1
+ echo 'Content of device file 1'
RUN: at line 18: echo 'Content of device file 2' > /home/buildbot/buildbot-root/llvm-clang-x86_64-gcc-ubuntu/build/tools/clang/test/Driver/Output/clang-offload-bundler-zlib.c.tmp.tgt2
+ echo 'Content of device file 2'
RUN: at line 23: env OFFLOAD_BUNDLER_COMPRESS=1 OFFLOAD_BUNDLER_VERBOSE=1    clang-offload-bundler -type=bc -targets=hip-amdgcn-amd-amdhsa--gfx900,hip-amdgcn-amd-amdhsa--gfx906    -input=/home/buildbot/buildbot-root/llvm-clang-x86_64-gcc-ubuntu/build/tools/clang/test/Driver/Output/clang-offload-bundler-zlib.c.tmp.tgt1 -input=/home/buildbot/buildbot-root/llvm-clang-x86_64-gcc-ubuntu/build/tools/clang/test/Driver/Output/clang-offload-bundler-zlib.c.tmp.tgt2 -output=/home/buildbot/buildbot-root/llvm-clang-x86_64-gcc-ubuntu/build/tools/clang/test/Driver/Output/clang-offload-bundler-zlib.c.tmp.hip.bundle.bc 2>&1 |    /home/buildbot/buildbot-root/llvm-clang-x86_64-gcc-ubuntu/build/bin/FileCheck -check-prefix=COMPRESS /home/buildbot/buildbot-root/llvm-clang-x86_64-gcc-ubuntu/llvm-project/clang/test/Driver/clang-offload-bundler-zlib.c
+ /home/buildbot/buildbot-root/llvm-clang-x86_64-gcc-ubuntu/build/bin/FileCheck -check-prefix=COMPRESS /home/buildbot/buildbot-root/llvm-clang-x86_64-gcc-ubuntu/llvm-project/clang/test/Driver/clang-offload-bundler-zlib.c
+ env OFFLOAD_BUNDLER_COMPRESS=1 OFFLOAD_BUNDLER_VERBOSE=1 clang-offload-bundler -type=bc -targets=hip-amdgcn-amd-amdhsa--gfx900,hip-amdgcn-amd-amdhsa--gfx906 -input=/home/buildbot/buildbot-root/llvm-clang-x86_64-gcc-ubuntu/build/tools/clang/test/Driver/Output/clang-offload-bundler-zlib.c.tmp.tgt1 -input=/home/buildbot/buildbot-root/llvm-clang-x86_64-gcc-ubuntu/build/tools/clang/test/Driver/Output/clang-offload-bundler-zlib.c.tmp.tgt2 -output=/home/buildbot/buildbot-root/llvm-clang-x86_64-gcc-ubuntu/build/tools/clang/test/Driver/Output/clang-offload-bundler-zlib.c.tmp.hip.bundle.bc
�[1m/home/buildbot/buildbot-root/llvm-clang-x86_64-gcc-ubuntu/llvm-project/clang/test/Driver/clang-offload-bundler-zlib.c:59:14: �[0m�[0;1;31merror: �[0m�[1mCOMPRESS: expected string not found in input
�[0m// COMPRESS: Compressed bundle format version: 3
�[0;1;32m             ^
�[0m�[1m<stdin>:4:21: �[0m�[0;1;30mnote: �[0m�[1mscanning from here
�[0mCompression level: 6
�[0;1;32m                    ^
�[0m�[1m<stdin>:7:1: �[0m�[0;1;30mnote: �[0m�[1mpossible intended match here
�[0mCompression rate: 1.86
�[0;1;32m^
�[0m
Input file: <stdin>
Check file: /home/buildbot/buildbot-root/llvm-clang-x86_64-gcc-ubuntu/llvm-project/clang/test/Driver/clang-offload-bundler-zlib.c

-dump-input=help explains the following input dump.

Input was:
<<<<<<
�[1m�[0m�[0;1;30m            1: �[0m�[1m�[0;1;46mCompressed bundle format version: 2 �[0m
�[0;1;30m            2: �[0m�[1m�[0;1;46mTotal file size (including headers): 125 bytes �[0m
�[0;1;30m            3: �[0m�[1m�[0;1;46m�[0mCompression method used: zlib�[0;1;46m �[0m
�[0;1;32mcheck:36       ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~
�[0m�[0;1;30m            4: �[0m�[1m�[0;1;46m�[0mCompression level: 6�[0;1;46m �[0m
�[0;1;32mcheck:37       ^~~~~~~~~~~~~~~~~~~~
�[0m�[0;1;31mcheck:59'0                         X error: no match found
�[0m�[0;1;30m            5: �[0m�[1m�[0;1;46mBinary size before compression: 188 bytes �[0m
�[0;1;31mcheck:59'0     ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
�[0m�[0;1;30m            6: �[0m�[1m�[0;1;46mBinary size after compression: 101 bytes �[0m
�[0;1;31mcheck:59'0     ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
�[0m�[0;1;30m            7: �[0m�[1m�[0;1;46mCompression rate: 1.86 �[0m
�[0;1;31mcheck:59'0     ~~~~~~~~~~~~~~~~~~~~~~~
�[0m�[0;1;35mcheck:59'1     ?                       possible intended match
�[0m�[0;1;30m            8: �[0m�[1m�[0;1;46mCompression ratio: 53.72% �[0m
�[0;1;31mcheck:59'0     ~~~~~~~~~~~~~~~~~~~~~~~~~~
�[0m�[0;1;30m            9: �[0m�[1m�[0;1;46mCompression speed: 1.49 MB/s �[0m
...

@llvm-ci
Copy link
Collaborator

llvm-ci commented Jan 21, 2025

LLVM Buildbot has detected a new failure on builder clang-debian-cpp20 running on clang-debian-cpp20 while building clang at step 6 "test-build-unified-tree-check-all".

Full details are available at: https://lab.llvm.org/buildbot/#/builders/108/builds/8358

Here is the relevant piece of the build log for the reference
Step 6 (test-build-unified-tree-check-all) failure: test (failure)
******************** TEST 'Clang :: Driver/clang-offload-bundler-zlib.c' FAILED ********************
Exit Code: 1

Command Output (stderr):
--
RUN: at line 7: /vol/worker/clang-debian-cpp20/clang-debian-cpp20/build/bin/clang -O0 -target x86_64-unknown-linux-gnu /vol/worker/clang-debian-cpp20/clang-debian-cpp20/llvm-project/clang/test/Driver/clang-offload-bundler-zlib.c -c -emit-llvm -o /vol/worker/clang-debian-cpp20/clang-debian-cpp20/build/tools/clang/test/Driver/Output/clang-offload-bundler-zlib.c.tmp.bc
+ /vol/worker/clang-debian-cpp20/clang-debian-cpp20/build/bin/clang -O0 -target x86_64-unknown-linux-gnu /vol/worker/clang-debian-cpp20/clang-debian-cpp20/llvm-project/clang/test/Driver/clang-offload-bundler-zlib.c -c -emit-llvm -o /vol/worker/clang-debian-cpp20/clang-debian-cpp20/build/tools/clang/test/Driver/Output/clang-offload-bundler-zlib.c.tmp.bc
RUN: at line 12: touch /vol/worker/clang-debian-cpp20/clang-debian-cpp20/build/tools/clang/test/Driver/Output/clang-offload-bundler-zlib.c.tmp.empty
+ touch /vol/worker/clang-debian-cpp20/clang-debian-cpp20/build/tools/clang/test/Driver/Output/clang-offload-bundler-zlib.c.tmp.empty
RUN: at line 17: echo 'Content of device file 1' > /vol/worker/clang-debian-cpp20/clang-debian-cpp20/build/tools/clang/test/Driver/Output/clang-offload-bundler-zlib.c.tmp.tgt1
+ echo 'Content of device file 1'
RUN: at line 18: echo 'Content of device file 2' > /vol/worker/clang-debian-cpp20/clang-debian-cpp20/build/tools/clang/test/Driver/Output/clang-offload-bundler-zlib.c.tmp.tgt2
+ echo 'Content of device file 2'
RUN: at line 23: env OFFLOAD_BUNDLER_COMPRESS=1 OFFLOAD_BUNDLER_VERBOSE=1    clang-offload-bundler -type=bc -targets=hip-amdgcn-amd-amdhsa--gfx900,hip-amdgcn-amd-amdhsa--gfx906    -input=/vol/worker/clang-debian-cpp20/clang-debian-cpp20/build/tools/clang/test/Driver/Output/clang-offload-bundler-zlib.c.tmp.tgt1 -input=/vol/worker/clang-debian-cpp20/clang-debian-cpp20/build/tools/clang/test/Driver/Output/clang-offload-bundler-zlib.c.tmp.tgt2 -output=/vol/worker/clang-debian-cpp20/clang-debian-cpp20/build/tools/clang/test/Driver/Output/clang-offload-bundler-zlib.c.tmp.hip.bundle.bc 2>&1 |    /vol/worker/clang-debian-cpp20/clang-debian-cpp20/build/bin/FileCheck -check-prefix=COMPRESS /vol/worker/clang-debian-cpp20/clang-debian-cpp20/llvm-project/clang/test/Driver/clang-offload-bundler-zlib.c
+ /vol/worker/clang-debian-cpp20/clang-debian-cpp20/build/bin/FileCheck -check-prefix=COMPRESS /vol/worker/clang-debian-cpp20/clang-debian-cpp20/llvm-project/clang/test/Driver/clang-offload-bundler-zlib.c
+ env OFFLOAD_BUNDLER_COMPRESS=1 OFFLOAD_BUNDLER_VERBOSE=1 clang-offload-bundler -type=bc -targets=hip-amdgcn-amd-amdhsa--gfx900,hip-amdgcn-amd-amdhsa--gfx906 -input=/vol/worker/clang-debian-cpp20/clang-debian-cpp20/build/tools/clang/test/Driver/Output/clang-offload-bundler-zlib.c.tmp.tgt1 -input=/vol/worker/clang-debian-cpp20/clang-debian-cpp20/build/tools/clang/test/Driver/Output/clang-offload-bundler-zlib.c.tmp.tgt2 -output=/vol/worker/clang-debian-cpp20/clang-debian-cpp20/build/tools/clang/test/Driver/Output/clang-offload-bundler-zlib.c.tmp.hip.bundle.bc
/vol/worker/clang-debian-cpp20/clang-debian-cpp20/llvm-project/clang/test/Driver/clang-offload-bundler-zlib.c:59:14: error: COMPRESS: expected string not found in input
// COMPRESS: Compressed bundle format version: 3
             ^
<stdin>:4:21: note: scanning from here
Compression level: 6
                    ^
<stdin>:7:1: note: possible intended match here
Compression rate: 1.86
^

Input file: <stdin>
Check file: /vol/worker/clang-debian-cpp20/clang-debian-cpp20/llvm-project/clang/test/Driver/clang-offload-bundler-zlib.c

-dump-input=help explains the following input dump.

Input was:
<<<<<<
            1: Compressed bundle format version: 2 
            2: Total file size (including headers): 125 bytes 
            3: Compression method used: zlib 
            4: Compression level: 6 
check:59'0                         X error: no match found
            5: Binary size before compression: 188 bytes 
check:59'0     ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
            6: Binary size after compression: 101 bytes 
check:59'0     ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
            7: Compression rate: 1.86 
check:59'0     ~~~~~~~~~~~~~~~~~~~~~~~
check:59'1     ?                       possible intended match
            8: Compression ratio: 53.72% 
check:59'0     ~~~~~~~~~~~~~~~~~~~~~~~~~~
            9: Compression speed: 1.14 MB/s 
check:59'0     ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
           10: Truncated MD5 hash: 0x2bcafb899a29dc94 
...

@llvm-ci
Copy link
Collaborator

llvm-ci commented Jan 21, 2025

LLVM Buildbot has detected a new failure on builder premerge-monolithic-linux running on premerge-linux-1 while building clang at step 7 "test-build-unified-tree-check-all".

Full details are available at: https://lab.llvm.org/buildbot/#/builders/153/builds/20312

Here is the relevant piece of the build log for the reference
Step 7 (test-build-unified-tree-check-all) failure: test (failure)
******************** TEST 'Clang :: Driver/clang-offload-bundler-zlib.c' FAILED ********************
Exit Code: 1

Command Output (stderr):
--
RUN: at line 7: /build/buildbot/premerge-monolithic-linux/build/bin/clang -O0 -target x86_64-unknown-linux-gnu /build/buildbot/premerge-monolithic-linux/llvm-project/clang/test/Driver/clang-offload-bundler-zlib.c -c -emit-llvm -o /build/buildbot/premerge-monolithic-linux/build/tools/clang/test/Driver/Output/clang-offload-bundler-zlib.c.tmp.bc
+ /build/buildbot/premerge-monolithic-linux/build/bin/clang -O0 -target x86_64-unknown-linux-gnu /build/buildbot/premerge-monolithic-linux/llvm-project/clang/test/Driver/clang-offload-bundler-zlib.c -c -emit-llvm -o /build/buildbot/premerge-monolithic-linux/build/tools/clang/test/Driver/Output/clang-offload-bundler-zlib.c.tmp.bc
RUN: at line 12: touch /build/buildbot/premerge-monolithic-linux/build/tools/clang/test/Driver/Output/clang-offload-bundler-zlib.c.tmp.empty
+ touch /build/buildbot/premerge-monolithic-linux/build/tools/clang/test/Driver/Output/clang-offload-bundler-zlib.c.tmp.empty
RUN: at line 17: echo 'Content of device file 1' > /build/buildbot/premerge-monolithic-linux/build/tools/clang/test/Driver/Output/clang-offload-bundler-zlib.c.tmp.tgt1
+ echo 'Content of device file 1'
RUN: at line 18: echo 'Content of device file 2' > /build/buildbot/premerge-monolithic-linux/build/tools/clang/test/Driver/Output/clang-offload-bundler-zlib.c.tmp.tgt2
+ echo 'Content of device file 2'
RUN: at line 23: env OFFLOAD_BUNDLER_COMPRESS=1 OFFLOAD_BUNDLER_VERBOSE=1    clang-offload-bundler -type=bc -targets=hip-amdgcn-amd-amdhsa--gfx900,hip-amdgcn-amd-amdhsa--gfx906    -input=/build/buildbot/premerge-monolithic-linux/build/tools/clang/test/Driver/Output/clang-offload-bundler-zlib.c.tmp.tgt1 -input=/build/buildbot/premerge-monolithic-linux/build/tools/clang/test/Driver/Output/clang-offload-bundler-zlib.c.tmp.tgt2 -output=/build/buildbot/premerge-monolithic-linux/build/tools/clang/test/Driver/Output/clang-offload-bundler-zlib.c.tmp.hip.bundle.bc 2>&1 |    /build/buildbot/premerge-monolithic-linux/build/bin/FileCheck -check-prefix=COMPRESS /build/buildbot/premerge-monolithic-linux/llvm-project/clang/test/Driver/clang-offload-bundler-zlib.c
+ env OFFLOAD_BUNDLER_COMPRESS=1 OFFLOAD_BUNDLER_VERBOSE=1 clang-offload-bundler -type=bc -targets=hip-amdgcn-amd-amdhsa--gfx900,hip-amdgcn-amd-amdhsa--gfx906 -input=/build/buildbot/premerge-monolithic-linux/build/tools/clang/test/Driver/Output/clang-offload-bundler-zlib.c.tmp.tgt1 -input=/build/buildbot/premerge-monolithic-linux/build/tools/clang/test/Driver/Output/clang-offload-bundler-zlib.c.tmp.tgt2 -output=/build/buildbot/premerge-monolithic-linux/build/tools/clang/test/Driver/Output/clang-offload-bundler-zlib.c.tmp.hip.bundle.bc
+ /build/buildbot/premerge-monolithic-linux/build/bin/FileCheck -check-prefix=COMPRESS /build/buildbot/premerge-monolithic-linux/llvm-project/clang/test/Driver/clang-offload-bundler-zlib.c
/build/buildbot/premerge-monolithic-linux/llvm-project/clang/test/Driver/clang-offload-bundler-zlib.c:59:14: error: COMPRESS: expected string not found in input
// COMPRESS: Compressed bundle format version: 3
             ^
<stdin>:4:21: note: scanning from here
Compression level: 6
                    ^
<stdin>:7:1: note: possible intended match here
Compression rate: 1.86
^

Input file: <stdin>
Check file: /build/buildbot/premerge-monolithic-linux/llvm-project/clang/test/Driver/clang-offload-bundler-zlib.c

-dump-input=help explains the following input dump.

Input was:
<<<<<<
            1: Compressed bundle format version: 2 
            2: Total file size (including headers): 125 bytes 
            3: Compression method used: zlib 
            4: Compression level: 6 
check:59'0                         X error: no match found
            5: Binary size before compression: 188 bytes 
check:59'0     ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
            6: Binary size after compression: 101 bytes 
check:59'0     ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
            7: Compression rate: 1.86 
check:59'0     ~~~~~~~~~~~~~~~~~~~~~~~
check:59'1     ?                       possible intended match
            8: Compression ratio: 53.72% 
check:59'0     ~~~~~~~~~~~~~~~~~~~~~~~~~~
            9: Compression speed: 1.02 MB/s 
check:59'0     ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
           10: Truncated MD5 hash: 0x2bcafb899a29dc94 
...

yxsamliu added a commit that referenced this pull request Jan 21, 2025
yxsamliu added a commit that referenced this pull request Jan 21, 2025
Reland the patch after fixing the lit test.
@llvm-ci
Copy link
Collaborator

llvm-ci commented Jan 21, 2025

LLVM Buildbot has detected a new failure on builder llvm-clang-win-x-aarch64 running on as-builder-2 while building clang at step 10 "test-check-clang".

Full details are available at: https://lab.llvm.org/buildbot/#/builders/193/builds/5054

Here is the relevant piece of the build log for the reference
Step 10 (test-check-clang) failure: Test just built components: check-clang completed (failure)
******************** TEST 'Clang :: Driver/clang-offload-bundler-zlib.c' FAILED ********************
Exit Code: 1

Command Output (stdout):
--
# RUN: at line 7
c:\buildbot\as-builder-2\x-aarch64\build\bin\clang.exe -O0 -target aarch64-unknown-linux-gnu C:\buildbot\as-builder-2\x-aarch64\llvm-project\clang\test\Driver\clang-offload-bundler-zlib.c -c -emit-llvm -o C:\buildbot\as-builder-2\x-aarch64\build\tools\clang\test\Driver\Output\clang-offload-bundler-zlib.c.tmp.bc
# executed command: 'c:\buildbot\as-builder-2\x-aarch64\build\bin\clang.exe' -O0 -target aarch64-unknown-linux-gnu 'C:\buildbot\as-builder-2\x-aarch64\llvm-project\clang\test\Driver\clang-offload-bundler-zlib.c' -c -emit-llvm -o 'C:\buildbot\as-builder-2\x-aarch64\build\tools\clang\test\Driver\Output\clang-offload-bundler-zlib.c.tmp.bc'
# RUN: at line 12
touch C:\buildbot\as-builder-2\x-aarch64\build\tools\clang\test\Driver\Output\clang-offload-bundler-zlib.c.tmp.empty
# executed command: touch 'C:\buildbot\as-builder-2\x-aarch64\build\tools\clang\test\Driver\Output\clang-offload-bundler-zlib.c.tmp.empty'
# RUN: at line 17
echo 'Content of device file 1' > C:\buildbot\as-builder-2\x-aarch64\build\tools\clang\test\Driver\Output\clang-offload-bundler-zlib.c.tmp.tgt1
# executed command: echo 'Content of device file 1'
# RUN: at line 18
echo 'Content of device file 2' > C:\buildbot\as-builder-2\x-aarch64\build\tools\clang\test\Driver\Output\clang-offload-bundler-zlib.c.tmp.tgt2
# executed command: echo 'Content of device file 2'
# RUN: at line 23
env OFFLOAD_BUNDLER_COMPRESS=1 OFFLOAD_BUNDLER_VERBOSE=1    clang-offload-bundler -type=bc -targets=hip-amdgcn-amd-amdhsa--gfx900,hip-amdgcn-amd-amdhsa--gfx906    -input=C:\buildbot\as-builder-2\x-aarch64\build\tools\clang\test\Driver\Output\clang-offload-bundler-zlib.c.tmp.tgt1 -input=C:\buildbot\as-builder-2\x-aarch64\build\tools\clang\test\Driver\Output\clang-offload-bundler-zlib.c.tmp.tgt2 -output=C:\buildbot\as-builder-2\x-aarch64\build\tools\clang\test\Driver\Output\clang-offload-bundler-zlib.c.tmp.hip.bundle.bc 2>&1 |    c:\buildbot\as-builder-2\x-aarch64\build\bin\filecheck.exe -check-prefix=COMPRESS C:\buildbot\as-builder-2\x-aarch64\llvm-project\clang\test\Driver\clang-offload-bundler-zlib.c
# executed command: env OFFLOAD_BUNDLER_COMPRESS=1 OFFLOAD_BUNDLER_VERBOSE=1 clang-offload-bundler -type=bc -targets=hip-amdgcn-amd-amdhsa--gfx900,hip-amdgcn-amd-amdhsa--gfx906 '-input=C:\buildbot\as-builder-2\x-aarch64\build\tools\clang\test\Driver\Output\clang-offload-bundler-zlib.c.tmp.tgt1' '-input=C:\buildbot\as-builder-2\x-aarch64\build\tools\clang\test\Driver\Output\clang-offload-bundler-zlib.c.tmp.tgt2' '-output=C:\buildbot\as-builder-2\x-aarch64\build\tools\clang\test\Driver\Output\clang-offload-bundler-zlib.c.tmp.hip.bundle.bc'
# executed command: 'c:\buildbot\as-builder-2\x-aarch64\build\bin\filecheck.exe' -check-prefix=COMPRESS 'C:\buildbot\as-builder-2\x-aarch64\llvm-project\clang\test\Driver\clang-offload-bundler-zlib.c'
# .---command stderr------------
# | C:\buildbot\as-builder-2\x-aarch64\llvm-project\clang\test\Driver\clang-offload-bundler-zlib.c:59:14: error: COMPRESS: expected string not found in input
# | // COMPRESS: Compressed bundle format version: 3
# |              ^
# | <stdin>:4:21: note: scanning from here
# | Compression level: 6
# |                     ^
# | <stdin>:7:1: note: possible intended match here
# | Compression rate: 1.86
# | ^
# | 
# | Input file: <stdin>
# | Check file: C:\buildbot\as-builder-2\x-aarch64\llvm-project\clang\test\Driver\clang-offload-bundler-zlib.c
# | 
# | -dump-input=help explains the following input dump.
# | 
# | Input was:
# | <<<<<<
# |             1: Compressed bundle format version: 2 
# |             2: Total file size (including headers): 125 bytes 
# |             3: Compression method used: zlib 
# |             4: Compression level: 6 
# | check:59'0                         X error: no match found
# |             5: Binary size before compression: 188 bytes 
# | check:59'0     ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
# |             6: Binary size after compression: 101 bytes 
# | check:59'0     ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
# |             7: Compression rate: 1.86 
# | check:59'0     ~~~~~~~~~~~~~~~~~~~~~~~
...

@llvm-ci
Copy link
Collaborator

llvm-ci commented Jan 21, 2025

LLVM Buildbot has detected a new failure on builder clang-aarch64-sve2-vla-2stage running on linaro-g4-01 while building clang at step 12 "ninja check 2".

Full details are available at: https://lab.llvm.org/buildbot/#/builders/199/builds/1090

Here is the relevant piece of the build log for the reference
Step 12 (ninja check 2) failure: stage 2 checked (failure)
******************** TEST 'Clang :: Driver/clang-offload-bundler-zlib.c' FAILED ********************
Exit Code: 1

Command Output (stderr):
--
RUN: at line 7: /home/tcwg-buildbot/worker/clang-aarch64-sve2-vla-2stage/stage2/bin/clang -O0 -target aarch64-unknown-linux-gnu /home/tcwg-buildbot/worker/clang-aarch64-sve2-vla-2stage/llvm/clang/test/Driver/clang-offload-bundler-zlib.c -c -emit-llvm -o /home/tcwg-buildbot/worker/clang-aarch64-sve2-vla-2stage/stage2/tools/clang/test/Driver/Output/clang-offload-bundler-zlib.c.tmp.bc
+ /home/tcwg-buildbot/worker/clang-aarch64-sve2-vla-2stage/stage2/bin/clang -O0 -target aarch64-unknown-linux-gnu /home/tcwg-buildbot/worker/clang-aarch64-sve2-vla-2stage/llvm/clang/test/Driver/clang-offload-bundler-zlib.c -c -emit-llvm -o /home/tcwg-buildbot/worker/clang-aarch64-sve2-vla-2stage/stage2/tools/clang/test/Driver/Output/clang-offload-bundler-zlib.c.tmp.bc
RUN: at line 12: touch /home/tcwg-buildbot/worker/clang-aarch64-sve2-vla-2stage/stage2/tools/clang/test/Driver/Output/clang-offload-bundler-zlib.c.tmp.empty
+ touch /home/tcwg-buildbot/worker/clang-aarch64-sve2-vla-2stage/stage2/tools/clang/test/Driver/Output/clang-offload-bundler-zlib.c.tmp.empty
RUN: at line 17: echo 'Content of device file 1' > /home/tcwg-buildbot/worker/clang-aarch64-sve2-vla-2stage/stage2/tools/clang/test/Driver/Output/clang-offload-bundler-zlib.c.tmp.tgt1
+ echo 'Content of device file 1'
RUN: at line 18: echo 'Content of device file 2' > /home/tcwg-buildbot/worker/clang-aarch64-sve2-vla-2stage/stage2/tools/clang/test/Driver/Output/clang-offload-bundler-zlib.c.tmp.tgt2
+ echo 'Content of device file 2'
RUN: at line 23: env OFFLOAD_BUNDLER_COMPRESS=1 OFFLOAD_BUNDLER_VERBOSE=1    clang-offload-bundler -type=bc -targets=hip-amdgcn-amd-amdhsa--gfx900,hip-amdgcn-amd-amdhsa--gfx906    -input=/home/tcwg-buildbot/worker/clang-aarch64-sve2-vla-2stage/stage2/tools/clang/test/Driver/Output/clang-offload-bundler-zlib.c.tmp.tgt1 -input=/home/tcwg-buildbot/worker/clang-aarch64-sve2-vla-2stage/stage2/tools/clang/test/Driver/Output/clang-offload-bundler-zlib.c.tmp.tgt2 -output=/home/tcwg-buildbot/worker/clang-aarch64-sve2-vla-2stage/stage2/tools/clang/test/Driver/Output/clang-offload-bundler-zlib.c.tmp.hip.bundle.bc 2>&1 |    /home/tcwg-buildbot/worker/clang-aarch64-sve2-vla-2stage/stage2/bin/FileCheck -check-prefix=COMPRESS /home/tcwg-buildbot/worker/clang-aarch64-sve2-vla-2stage/llvm/clang/test/Driver/clang-offload-bundler-zlib.c
+ env OFFLOAD_BUNDLER_COMPRESS=1 OFFLOAD_BUNDLER_VERBOSE=1 clang-offload-bundler -type=bc -targets=hip-amdgcn-amd-amdhsa--gfx900,hip-amdgcn-amd-amdhsa--gfx906 -input=/home/tcwg-buildbot/worker/clang-aarch64-sve2-vla-2stage/stage2/tools/clang/test/Driver/Output/clang-offload-bundler-zlib.c.tmp.tgt1 -input=/home/tcwg-buildbot/worker/clang-aarch64-sve2-vla-2stage/stage2/tools/clang/test/Driver/Output/clang-offload-bundler-zlib.c.tmp.tgt2 -output=/home/tcwg-buildbot/worker/clang-aarch64-sve2-vla-2stage/stage2/tools/clang/test/Driver/Output/clang-offload-bundler-zlib.c.tmp.hip.bundle.bc
+ /home/tcwg-buildbot/worker/clang-aarch64-sve2-vla-2stage/stage2/bin/FileCheck -check-prefix=COMPRESS /home/tcwg-buildbot/worker/clang-aarch64-sve2-vla-2stage/llvm/clang/test/Driver/clang-offload-bundler-zlib.c
/home/tcwg-buildbot/worker/clang-aarch64-sve2-vla-2stage/llvm/clang/test/Driver/clang-offload-bundler-zlib.c:59:14: error: COMPRESS: expected string not found in input
// COMPRESS: Compressed bundle format version: 3
             ^
<stdin>:4:21: note: scanning from here
Compression level: 6
                    ^
<stdin>:7:1: note: possible intended match here
Compression rate: 1.86
^

Input file: <stdin>
Check file: /home/tcwg-buildbot/worker/clang-aarch64-sve2-vla-2stage/llvm/clang/test/Driver/clang-offload-bundler-zlib.c

-dump-input=help explains the following input dump.

Input was:
<<<<<<
            1: Compressed bundle format version: 2 
            2: Total file size (including headers): 125 bytes 
            3: Compression method used: zlib 
            4: Compression level: 6 
check:59'0                         X error: no match found
            5: Binary size before compression: 188 bytes 
check:59'0     ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
            6: Binary size after compression: 101 bytes 
check:59'0     ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
            7: Compression rate: 1.86 
check:59'0     ~~~~~~~~~~~~~~~~~~~~~~~
check:59'1     ?                       possible intended match
            8: Compression ratio: 53.72% 
check:59'0     ~~~~~~~~~~~~~~~~~~~~~~~~~~
            9: Compression speed: 1.08 MB/s 
check:59'0     ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
           10: Truncated MD5 hash: 0x2bcafb899a29dc94 
...

@llvm-ci
Copy link
Collaborator

llvm-ci commented Jan 21, 2025

LLVM Buildbot has detected a new failure on builder clang-aarch64-global-isel running on linaro-clang-aarch64-global-isel while building clang at step 7 "ninja check 1".

Full details are available at: https://lab.llvm.org/buildbot/#/builders/125/builds/5078

Here is the relevant piece of the build log for the reference
Step 7 (ninja check 1) failure: stage 1 checked (failure)
******************** TEST 'Clang :: Driver/clang-offload-bundler-zlib.c' FAILED ********************
Exit Code: 1

Command Output (stderr):
--
RUN: at line 7: /home/tcwg-buildbot/worker/clang-aarch64-global-isel/stage1/bin/clang -O0 -target aarch64-unknown-linux-gnu /home/tcwg-buildbot/worker/clang-aarch64-global-isel/llvm/clang/test/Driver/clang-offload-bundler-zlib.c -c -emit-llvm -o /home/tcwg-buildbot/worker/clang-aarch64-global-isel/stage1/tools/clang/test/Driver/Output/clang-offload-bundler-zlib.c.tmp.bc
+ /home/tcwg-buildbot/worker/clang-aarch64-global-isel/stage1/bin/clang -O0 -target aarch64-unknown-linux-gnu /home/tcwg-buildbot/worker/clang-aarch64-global-isel/llvm/clang/test/Driver/clang-offload-bundler-zlib.c -c -emit-llvm -o /home/tcwg-buildbot/worker/clang-aarch64-global-isel/stage1/tools/clang/test/Driver/Output/clang-offload-bundler-zlib.c.tmp.bc
RUN: at line 12: touch /home/tcwg-buildbot/worker/clang-aarch64-global-isel/stage1/tools/clang/test/Driver/Output/clang-offload-bundler-zlib.c.tmp.empty
+ touch /home/tcwg-buildbot/worker/clang-aarch64-global-isel/stage1/tools/clang/test/Driver/Output/clang-offload-bundler-zlib.c.tmp.empty
RUN: at line 17: echo 'Content of device file 1' > /home/tcwg-buildbot/worker/clang-aarch64-global-isel/stage1/tools/clang/test/Driver/Output/clang-offload-bundler-zlib.c.tmp.tgt1
+ echo 'Content of device file 1'
RUN: at line 18: echo 'Content of device file 2' > /home/tcwg-buildbot/worker/clang-aarch64-global-isel/stage1/tools/clang/test/Driver/Output/clang-offload-bundler-zlib.c.tmp.tgt2
+ echo 'Content of device file 2'
RUN: at line 23: env OFFLOAD_BUNDLER_COMPRESS=1 OFFLOAD_BUNDLER_VERBOSE=1    clang-offload-bundler -type=bc -targets=hip-amdgcn-amd-amdhsa--gfx900,hip-amdgcn-amd-amdhsa--gfx906    -input=/home/tcwg-buildbot/worker/clang-aarch64-global-isel/stage1/tools/clang/test/Driver/Output/clang-offload-bundler-zlib.c.tmp.tgt1 -input=/home/tcwg-buildbot/worker/clang-aarch64-global-isel/stage1/tools/clang/test/Driver/Output/clang-offload-bundler-zlib.c.tmp.tgt2 -output=/home/tcwg-buildbot/worker/clang-aarch64-global-isel/stage1/tools/clang/test/Driver/Output/clang-offload-bundler-zlib.c.tmp.hip.bundle.bc 2>&1 |    /home/tcwg-buildbot/worker/clang-aarch64-global-isel/stage1/bin/FileCheck -check-prefix=COMPRESS /home/tcwg-buildbot/worker/clang-aarch64-global-isel/llvm/clang/test/Driver/clang-offload-bundler-zlib.c
+ env OFFLOAD_BUNDLER_COMPRESS=1 OFFLOAD_BUNDLER_VERBOSE=1 clang-offload-bundler -type=bc -targets=hip-amdgcn-amd-amdhsa--gfx900,hip-amdgcn-amd-amdhsa--gfx906 -input=/home/tcwg-buildbot/worker/clang-aarch64-global-isel/stage1/tools/clang/test/Driver/Output/clang-offload-bundler-zlib.c.tmp.tgt1 -input=/home/tcwg-buildbot/worker/clang-aarch64-global-isel/stage1/tools/clang/test/Driver/Output/clang-offload-bundler-zlib.c.tmp.tgt2 -output=/home/tcwg-buildbot/worker/clang-aarch64-global-isel/stage1/tools/clang/test/Driver/Output/clang-offload-bundler-zlib.c.tmp.hip.bundle.bc
+ /home/tcwg-buildbot/worker/clang-aarch64-global-isel/stage1/bin/FileCheck -check-prefix=COMPRESS /home/tcwg-buildbot/worker/clang-aarch64-global-isel/llvm/clang/test/Driver/clang-offload-bundler-zlib.c
/home/tcwg-buildbot/worker/clang-aarch64-global-isel/llvm/clang/test/Driver/clang-offload-bundler-zlib.c:59:14: error: COMPRESS: expected string not found in input
// COMPRESS: Compressed bundle format version: 3
             ^
<stdin>:4:21: note: scanning from here
Compression level: 6
                    ^
<stdin>:7:1: note: possible intended match here
Compression rate: 1.86
^

Input file: <stdin>
Check file: /home/tcwg-buildbot/worker/clang-aarch64-global-isel/llvm/clang/test/Driver/clang-offload-bundler-zlib.c

-dump-input=help explains the following input dump.

Input was:
<<<<<<
            1: Compressed bundle format version: 2 
            2: Total file size (including headers): 125 bytes 
            3: Compression method used: zlib 
            4: Compression level: 6 
check:59'0                         X error: no match found
            5: Binary size before compression: 188 bytes 
check:59'0     ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
            6: Binary size after compression: 101 bytes 
check:59'0     ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
            7: Compression rate: 1.86 
check:59'0     ~~~~~~~~~~~~~~~~~~~~~~~
check:59'1     ?                       possible intended match
            8: Compression ratio: 53.72% 
check:59'0     ~~~~~~~~~~~~~~~~~~~~~~~~~~
            9: Compression speed: 0.14 MB/s 
check:59'0     ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
           10: Truncated MD5 hash: 0x2bcafb899a29dc94 
...

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
clang:driver 'clang' and 'clang++' user-facing binaries. Not 'clang-cl' clang Clang issues not falling into any other category
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants