[BOLT][DWARF] Add parallelization for processing of DWO debug information #100282

sayhaan · 2024-07-23T23:46:33Z

Enables parallelization for the processing of DWO CUs.

github-actions · 2024-07-23T23:52:44Z

✅ With the latest revision this PR passed the Python code formatter.

bolt/include/bolt/Core/ParallelUtilities.h

llvmbot · 2024-07-24T00:46:16Z

@llvm/pr-subscribers-bolt

Author: Sayhaan Siddiqui (sayhaan)

Changes

Enables parallelization for the processing of DWO CUs.

Patch is 47.54 KiB, truncated to 20.00 KiB below, full version: https://github.com/llvm/llvm-project/pull/100282.diff

49 Files Affected:

(modified) bolt/include/bolt/Core/ParallelUtilities.h (+1-1)
(modified) bolt/lib/Core/ParallelUtilities.cpp (+7-4)
(modified) bolt/lib/Rewrite/DWARFRewriter.cpp (+13-2)
(modified) bolt/test/X86/dwarf4-cross-cu-backward-different-abbrev.test (+1-1)
(modified) bolt/test/X86/dwarf4-cross-cu-forward-different-abbrev.test (+1-1)
(modified) bolt/test/X86/dwarf4-cross-cu-loclist-dwarf4-loclist--dwarf5-loclist.test (+1-1)
(modified) bolt/test/X86/dwarf4-df-dualcu-loclist.test (+1-1)
(modified) bolt/test/X86/dwarf4-df-dualcu.test (+1-1)
(modified) bolt/test/X86/dwarf4-df-input-lowpc-ranges-cus.test (+1-1)
(modified) bolt/test/X86/dwarf4-split-dwarf-no-address.test (+1-1)
(modified) bolt/test/X86/dwarf4-split-gdb-index-types-gdb-generated.test (+1-1)
(modified) bolt/test/X86/dwarf4-subprogram-multiple-ranges-cus.test (+1-1)
(modified) bolt/test/X86/dwarf4-types-dwarf5-types.test (+1-1)
(modified) bolt/test/X86/dwarf4-types-dwarf5.test (+1-1)
(modified) bolt/test/X86/dwarf5-addr-section-reuse.s (+1-1)
(modified) bolt/test/X86/dwarf5-call-pc-function-null-check.test (+1-1)
(modified) bolt/test/X86/dwarf5-call-pc.test (+1-1)
(modified) bolt/test/X86/dwarf5-cu-no-debug-addr.test (+1-1)
(modified) bolt/test/X86/dwarf5-df-input-lowpc-ranges-cus.test (+1-1)
(modified) bolt/test/X86/dwarf5-df-mono-dualcu.test (+1-1)
(modified) bolt/test/X86/dwarf5-df-output-dir-same-name.test (+1-1)
(modified) bolt/test/X86/dwarf5-df-types-debug-names.test (+1-1)
(modified) bolt/test/X86/dwarf5-df-types-dup-dwp-input.test (+1-1)
(modified) bolt/test/X86/dwarf5-df-types-modify-dwo-name-mixed.test (+1-1)
(modified) bolt/test/X86/dwarf5-df-types-modify-dwo-name.test (+1-1)
(modified) bolt/test/X86/dwarf5-dwarf4-gdb-index-types-gdb-generated-gdb11.test (+1-1)
(modified) bolt/test/X86/dwarf5-dwarf4-gdb-index-types-gdb-generated-gdb9.test (+1-1)
(modified) bolt/test/X86/dwarf5-dwarf4-gdb-index-types-lld-generated.test (+1-1)
(modified) bolt/test/X86/dwarf5-dwarf4-monolithic.test (+1-1)
(modified) bolt/test/X86/dwarf5-dwarf4-types-backward-forward-cross-reference.test (+1-1)
(modified) bolt/test/X86/dwarf5-empty-arange.test (+1-1)
(modified) bolt/test/X86/dwarf5-ftypes-dwo-mono-input-dwp-output.test (+1-1)
(modified) bolt/test/X86/dwarf5-gdb-index-types-gdb-generated-gdb11.test (+1-1)
(modified) bolt/test/X86/dwarf5-gdb-index-types-gdb-generated-gdb9.test (+1-1)
(modified) bolt/test/X86/dwarf5-gdb-index-types-lld-generated.test (+1-1)
(modified) bolt/test/X86/dwarf5-locexpr-referrence.test (+1-1)
(modified) bolt/test/X86/dwarf5-loclist-offset-form.test (+1-1)
(modified) bolt/test/X86/dwarf5-one-loclists-two-bases.test (+1-1)
(modified) bolt/test/X86/dwarf5-return-pc-form-addr.test (+1-1)
(modified) bolt/test/X86/dwarf5-return-pc.test (+1-1)
(modified) bolt/test/X86/dwarf5-split-dwarf4-monolithic.test (+1-1)
(modified) bolt/test/X86/dwarf5-split-gdb-index-types-gdb-generated.test (+1-1)
(modified) bolt/test/X86/dwarf5-subprogram-multiple-ranges-cus.test (+1-1)
(modified) bolt/test/X86/dwarf5-two-cu-str-offset-table.test (+1-1)
(modified) bolt/test/X86/dwarf5-two-loclists.test (+1-1)
(modified) bolt/test/X86/dwarf5-two-rnglists.test (+1-1)
(modified) bolt/test/X86/dwarf5-type-unit-no-cu-str-offset-table.test (+1-1)
(modified) bolt/test/X86/dwarf5-types-backward-cross-reference.s (+1-1)
(modified) bolt/test/X86/dwarf5-types-debug-names.test (+1-1)

diff --git a/bolt/include/bolt/Core/ParallelUtilities.h b/bolt/include/bolt/Core/ParallelUtilities.h
index e7b35a79acc78..854438e4bba88 100644
--- a/bolt/include/bolt/Core/ParallelUtilities.h
+++ b/bolt/include/bolt/Core/ParallelUtilities.h
@@ -50,7 +50,7 @@ enum SchedulingPolicy {
 };
 
 /// Return the managed thread pool and initialize it if not initialized.
-ThreadPoolInterface &getThreadPool();
+ThreadPoolInterface &getThreadPool(const int ThreadsCount = opts::ThreadCount);
 
 /// Perform the work on each BinaryFunction except those that are accepted
 /// by SkipPredicate, scheduling heuristic is based on SchedPolicy.
diff --git a/bolt/lib/Core/ParallelUtilities.cpp b/bolt/lib/Core/ParallelUtilities.cpp
index a24c37c06f1ac..63ecec83ca816 100644
--- a/bolt/lib/Core/ParallelUtilities.cpp
+++ b/bolt/lib/Core/ParallelUtilities.cpp
@@ -49,7 +49,7 @@ namespace ParallelUtilities {
 
 namespace {
 /// A single thread pool that is used to run parallel tasks
-std::unique_ptr<DefaultThreadPool> ThreadPoolPtr;
+std::unique_ptr<ThreadPoolInterface> ThreadPoolPtr;
 
 unsigned computeCostFor(const BinaryFunction &BF,
                         const PredicateTy &SkipPredicate,
@@ -102,12 +102,15 @@ inline unsigned estimateTotalCost(const BinaryContext &BC,
 
 } // namespace
 
-ThreadPoolInterface &getThreadPool() {
+ThreadPoolInterface &getThreadPool(const int ThreadsCount) {
   if (ThreadPoolPtr.get())
     return *ThreadPoolPtr;
 
-  ThreadPoolPtr = std::make_unique<DefaultThreadPool>(
-      llvm::hardware_concurrency(opts::ThreadCount));
+  if (ThreadsCount > 1)
+    ThreadPoolPtr = std::make_unique<DefaultThreadPool>(
+        llvm::hardware_concurrency(ThreadsCount));
+  else
+    ThreadPoolPtr = std::make_unique<SingleThreadExecutor>();
   return *ThreadPoolPtr;
 }
 
diff --git a/bolt/lib/Rewrite/DWARFRewriter.cpp b/bolt/lib/Rewrite/DWARFRewriter.cpp
index 674b5f17adb3f..963a02e05000c 100644
--- a/bolt/lib/Rewrite/DWARFRewriter.cpp
+++ b/bolt/lib/Rewrite/DWARFRewriter.cpp
@@ -326,6 +326,12 @@ static cl::opt<bool> KeepARanges(
         "keep or generate .debug_aranges section if .gdb_index is written"),
     cl::Hidden, cl::cat(BoltCategory));
 
+static cl::opt<int>
+    DebugThreadCount("debug-thread-count",
+                     cl::desc("specifies thread count for the multithreading "
+                              "for updating DWO debug info"),
+                     cl::init(1), cl::cat(BoltCategory));
+
 static cl::opt<std::string> DwarfOutputPath(
     "dwarf-output-path",
     cl::desc("Path to where .dwo files or dwp file will be written out to."),
@@ -738,6 +744,8 @@ void DWARFRewriter::updateDebugInfo() {
   CUPartitionVector PartVec = partitionCUs(*BC.DwCtx);
   for (std::vector<DWARFUnit *> &Vec : PartVec) {
     DIEBlder.buildCompileUnits(Vec);
+    ThreadPoolInterface &ThreadPool =
+        ParallelUtilities::getThreadPool(opts::DebugThreadCount);
     for (DWARFUnit *CU : DIEBlder.getProcessedCUs()) {
       createRangeLocListAddressWriters(*CU);
       std::optional<DWARFUnit *> SplitCU;
@@ -757,9 +765,12 @@ void DWARFRewriter::updateDebugInfo() {
               : std::optional<std::string>(opts::DwarfOutputPath.c_str());
       std::string DWOName = DIEBlder.updateDWONameCompDir(
           *StrOffstsWriter, *StrWriter, *CU, DwarfOutputPath, std::nullopt);
-      processSplitCU(*CU, **SplitCU, DIEBlder, *TempRangesSectionWriter,
-                     AddressWriter, DWOName, DwarfOutputPath);
+      ThreadPool.async([&, DwarfOutputPath, DWOName] {
+        processSplitCU(*CU, **SplitCU, DIEBlder, *TempRangesSectionWriter,
+                       AddressWriter, DWOName, DwarfOutputPath);
+      });
     }
+    ThreadPool.wait();
     for (DWARFUnit *CU : DIEBlder.getProcessedCUs())
       processMainBinaryCU(*CU, DIEBlder);
     finalizeCompileUnits(DIEBlder, *Streamer, OffsetMap,
diff --git a/bolt/test/X86/dwarf4-cross-cu-backward-different-abbrev.test b/bolt/test/X86/dwarf4-cross-cu-backward-different-abbrev.test
index 555887a067589..580a61e0aa4be 100644
--- a/bolt/test/X86/dwarf4-cross-cu-backward-different-abbrev.test
+++ b/bolt/test/X86/dwarf4-cross-cu-backward-different-abbrev.test
@@ -3,7 +3,7 @@
 # RUN: llvm-mc -dwarf-version=4 -filetype=obj -triple x86_64-unknown-linux %p/Inputs/dwarf4-cross-reference-different-abbrev-dst.s -o %t.o
 # RUN: llvm-mc -dwarf-version=4 -filetype=obj -triple x86_64-unknown-linux %p/Inputs/dwarf4-cross-reference-different-abbrev-src.s -o %t1.o
 # RUN: %clang %cflags -gdwarf-4 %t.o %t1.o -o %t.exe
-# RUN: llvm-bolt %t.exe -o %t.bolt --update-debug-sections
+# RUN: llvm-bolt %t.exe -o %t.bolt --update-debug-sections --debug-thread-count=72
 # RUN: llvm-dwarfdump --show-form --verbose --debug-info %t.exe | FileCheck --check-prefix=PRECHECK %s
 # RUN: llvm-dwarfdump --show-form --verbose --debug-info %t.bolt | FileCheck --check-prefix=POSTCHECK %s
 
diff --git a/bolt/test/X86/dwarf4-cross-cu-forward-different-abbrev.test b/bolt/test/X86/dwarf4-cross-cu-forward-different-abbrev.test
index 74c9491d95d36..5b42034eabf7f 100644
--- a/bolt/test/X86/dwarf4-cross-cu-forward-different-abbrev.test
+++ b/bolt/test/X86/dwarf4-cross-cu-forward-different-abbrev.test
@@ -3,7 +3,7 @@
 # RUN: llvm-mc -dwarf-version=4 -filetype=obj -triple x86_64-unknown-linux %p/Inputs/dwarf4-cross-reference-different-abbrev-dst.s -o %t.o
 # RUN: llvm-mc -dwarf-version=4 -filetype=obj -triple x86_64-unknown-linux %p/Inputs/dwarf4-cross-reference-different-abbrev-src.s -o %t1.o
 # RUN: %clang %cflags -gdwarf-4 %t1.o %t.o -o %t.exe
-# RUN: llvm-bolt %t.exe -o %t.bolt --update-debug-sections
+# RUN: llvm-bolt %t.exe -o %t.bolt --update-debug-sections --debug-thread-count=72
 # RUN: llvm-dwarfdump --show-form --verbose --debug-info %t.exe | FileCheck --check-prefix=PRECHECK %s
 # RUN: llvm-dwarfdump --show-form --verbose --debug-info %t.bolt | FileCheck --check-prefix=POSTCHECK %s
 
diff --git a/bolt/test/X86/dwarf4-cross-cu-loclist-dwarf4-loclist--dwarf5-loclist.test b/bolt/test/X86/dwarf4-cross-cu-loclist-dwarf4-loclist--dwarf5-loclist.test
index 6bcf8892ed0a8..4dfb0df51902d 100644
--- a/bolt/test/X86/dwarf4-cross-cu-loclist-dwarf4-loclist--dwarf5-loclist.test
+++ b/bolt/test/X86/dwarf4-cross-cu-loclist-dwarf4-loclist--dwarf5-loclist.test
@@ -4,7 +4,7 @@
 # RUN: llvm-mc -dwarf-version=5 -filetype=obj -triple x86_64-unknown-linux %p/Inputs/dwarf5-loclist.s -o %t1.o
 # RUN: llvm-mc -dwarf-version=4 -filetype=obj -triple x86_64-unknown-linux %p/Inputs/dwarf4-two-entries-loclist.s -o %t2.o
 # RUN: %clang %cflags %t1.o %t2.o %t.o -o %t.exe
-# RUN: llvm-bolt %t.exe -o %t.bolt --update-debug-sections
+# RUN: llvm-bolt %t.exe -o %t.bolt --update-debug-sections --debug-thread-count=72
 # RUN: llvm-dwarfdump --show-form --verbose --debug-info %t.exe | FileCheck --check-prefix=PRECHECK %s
 # RUN: llvm-dwarfdump --show-form --verbose --debug-info %t.bolt | FileCheck --check-prefix=POSTCHECK %s
 
diff --git a/bolt/test/X86/dwarf4-df-dualcu-loclist.test b/bolt/test/X86/dwarf4-df-dualcu-loclist.test
index 57c75e282421a..03aa77e819596 100644
--- a/bolt/test/X86/dwarf4-df-dualcu-loclist.test
+++ b/bolt/test/X86/dwarf4-df-dualcu-loclist.test
@@ -6,7 +6,7 @@
 ; RUN: llvm-mc -dwarf-version=4 -filetype=obj -triple x86_64-unknown-linux %p/Inputs/dwarf4-df-dualcu-loclist-helper.s \
 ; RUN: -split-dwarf-file=helper.dwo -o helper.o
 ; RUN: %clang %cflags -gdwarf-5 -O2 -gsplit-dwarf=split main.o helper.o -o main.exe
-; RUN: llvm-bolt main.exe -o main.exe.bolt --update-debug-sections
+; RUN: llvm-bolt main.exe -o main.exe.bolt --update-debug-sections --debug-thread-count=72
 ; RUN: llvm-dwarfdump --show-form --verbose --debug-info main.dwo | FileCheck -check-prefix=PRE-BOLT-DWO-MAIN %s
 ; RUN: llvm-dwarfdump --show-form --verbose --debug-info main.dwo.dwo | FileCheck -check-prefix=BOLT-DWO-MAIN %s
 ; RUN: llvm-dwarfdump --show-form --verbose --debug-info helper.dwo | FileCheck -check-prefix=PRE-BOLT-DWO-HELPER %s
diff --git a/bolt/test/X86/dwarf4-df-dualcu.test b/bolt/test/X86/dwarf4-df-dualcu.test
index 468c924de3a90..7cc6dd0bb1614 100644
--- a/bolt/test/X86/dwarf4-df-dualcu.test
+++ b/bolt/test/X86/dwarf4-df-dualcu.test
@@ -6,7 +6,7 @@
 ; RUN: llvm-mc -dwarf-version=4 -filetype=obj -triple x86_64-unknown-linux %p/Inputs/dwarf4-df-dualcu-helper.s \
 ; RUN: -split-dwarf-file=helper.dwo -o helper.o
 ; RUN: %clang %cflags -gdwarf-5 -gsplit-dwarf=split main.o helper.o -o main.exe -fno-pic -no-pie
-; RUN: llvm-bolt main.exe -o main.exe.bolt --update-debug-sections --always-convert-to-ranges
+; RUN: llvm-bolt main.exe -o main.exe.bolt --update-debug-sections --always-convert-to-ranges --debug-thread-count=72
 ; RUN: llvm-dwarfdump --show-form --verbose --debug-info main.exe | FileCheck -check-prefix=PRE-BOLT %s
 ; RUN: llvm-dwarfdump --show-form --verbose --debug-ranges main.exe.bolt &> %t/foo.txt
 ; RUN: llvm-dwarfdump --show-form --verbose --debug-info main.exe.bolt >> %t/foo.txt
diff --git a/bolt/test/X86/dwarf4-df-input-lowpc-ranges-cus.test b/bolt/test/X86/dwarf4-df-input-lowpc-ranges-cus.test
index a97377400eeb7..021cf43c03981 100644
--- a/bolt/test/X86/dwarf4-df-input-lowpc-ranges-cus.test
+++ b/bolt/test/X86/dwarf4-df-input-lowpc-ranges-cus.test
@@ -6,7 +6,7 @@
 ; RUN: llvm-mc -dwarf-version=4 -filetype=obj -triple x86_64-unknown-linux %p/Inputs/dwarf4-df-input-lowpc-ranges-other.s \
 ; RUN: -split-dwarf-file=mainOther.dwo -o other.o
 ; RUN: %clang %cflags -gdwarf-4 -gsplit-dwarf=split main.o other.o -o main.exe
-; RUN: llvm-bolt main.exe -o main.exe.bolt --update-debug-sections
+; RUN: llvm-bolt main.exe -o main.exe.bolt --update-debug-sections --debug-thread-count=72
 ; RUN: llvm-dwarfdump --show-form --verbose --debug-ranges main.exe.bolt &> %t/foo.txt
 ; RUN: llvm-dwarfdump --show-form --verbose --debug-info main.exe.bolt >> %t/foo.txt
 ; RUN: cat %t/foo.txt | FileCheck -check-prefix=BOLT %s
diff --git a/bolt/test/X86/dwarf4-split-dwarf-no-address.test b/bolt/test/X86/dwarf4-split-dwarf-no-address.test
index fc6d8d324b959..223b9248089ae 100644
--- a/bolt/test/X86/dwarf4-split-dwarf-no-address.test
+++ b/bolt/test/X86/dwarf4-split-dwarf-no-address.test
@@ -6,7 +6,7 @@
 ; RUN: llvm-mc --split-dwarf-file=helper.dwo --triple=x86_64-unknown-linux-gnu \
 ; RUN: --filetype=obj %p/Inputs/dwarf4-split-dwarf-no-address-helper.s -o=helper.o
 ; RUN: %clang %cflags -gdwarf-4 -gsplit-dwarf=split main.o helper.o -o main.exe -fno-pic -no-pie
-; RUN: llvm-bolt main.exe -o main.exe.bolt --update-debug-sections
+; RUN: llvm-bolt main.exe -o main.exe.bolt --update-debug-sections --debug-thread-count=72
 ; RUN: llvm-dwarfdump --show-form --verbose --debug-info main.exe.bolt | FileCheck -check-prefix=BOLT %s
 
 ;; Testing that there are no asserts/crashes when one of the DWARF4 CUs does not modify .debug_addr
diff --git a/bolt/test/X86/dwarf4-split-gdb-index-types-gdb-generated.test b/bolt/test/X86/dwarf4-split-gdb-index-types-gdb-generated.test
index c9b12574caa3a..20b6288c5532c 100644
--- a/bolt/test/X86/dwarf4-split-gdb-index-types-gdb-generated.test
+++ b/bolt/test/X86/dwarf4-split-gdb-index-types-gdb-generated.test
@@ -7,7 +7,7 @@
 # RUN: llvm-mc --split-dwarf-file=helper.dwo -dwarf-version=4 -filetype=obj -triple x86_64-unknown-linux %p/Inputs/dwarf4-split-gdb-index-types-helper.s -o helpergdb.o
 # RUN: %clang %cflags maingdb.o helpergdb.o -o maingdb.exe -Wl,-q
 # RUN: llvm-objcopy maingdb.exe --add-section=.gdb_index=%p/Inputs/dwarf4-split-gdb-index-types-v8.gdb-index
-# RUN: llvm-bolt maingdb.exe -o maingdb.exe.bolt --update-debug-sections
+# RUN: llvm-bolt maingdb.exe -o maingdb.exe.bolt --update-debug-sections --debug-thread-count=72
 # RUN: llvm-dwarfdump --gdb-index maingdb.exe.bolt | FileCheck --check-prefix=POSTCHECK %s
 
 ## Tests that BOLT correctly handles gdb-index generated by GDB with split-dwarf DWARF4.
diff --git a/bolt/test/X86/dwarf4-subprogram-multiple-ranges-cus.test b/bolt/test/X86/dwarf4-subprogram-multiple-ranges-cus.test
index c9ade995b7087..9eda54cd76a50 100644
--- a/bolt/test/X86/dwarf4-subprogram-multiple-ranges-cus.test
+++ b/bolt/test/X86/dwarf4-subprogram-multiple-ranges-cus.test
@@ -3,7 +3,7 @@
 # RUN: llvm-mc -dwarf-version=4 -filetype=obj -triple x86_64-unknown-linux %p/Inputs/dwarf4-subprogram-multiple-ranges-main.s -o %t1.o
 # RUN: llvm-mc -dwarf-version=4 -filetype=obj -triple x86_64-unknown-linux %p/Inputs/dwarf4-subprogram-multiple-ranges-other.s -o %t2.o
 # RUN: %clang %cflags %t1.o %t2.o -o %t.exe -Wl,-q
-# RUN: llvm-bolt %t.exe -o %t.bolt --update-debug-sections
+# RUN: llvm-bolt %t.exe -o %t.bolt --update-debug-sections --debug-thread-count=72
 # RUN: llvm-objdump %t.bolt --disassemble > %t1.txt
 # RUN: llvm-dwarfdump --show-form --verbose --debug-info %t.bolt >> %t1.txt
 # RUN: cat %t1.txt | FileCheck --check-prefix=POSTCHECK %s
diff --git a/bolt/test/X86/dwarf4-types-dwarf5-types.test b/bolt/test/X86/dwarf4-types-dwarf5-types.test
index a253f22836090..6e58ea9c21fea 100644
--- a/bolt/test/X86/dwarf4-types-dwarf5-types.test
+++ b/bolt/test/X86/dwarf4-types-dwarf5-types.test
@@ -3,7 +3,7 @@
 # RUN: llvm-mc -dwarf-version=4 -filetype=obj -triple x86_64-unknown-linux %p/Inputs/dwarf4-types-dwarf5-types-main.s -o %tmain.o
 # RUN: llvm-mc -dwarf-version=5 -filetype=obj -triple x86_64-unknown-linux %p/Inputs/dwarf4-types-dwarf5-types-helper.s -o %thelper.o
 # RUN: %clang %cflags %tmain.o %thelper.o -o %t.exe -Wl,-q
-# RUN: llvm-bolt %t.exe -o %t.bolt --update-debug-sections
+# RUN: llvm-bolt %t.exe -o %t.bolt --update-debug-sections --debug-thread-count=72
 # RUN: llvm-dwarfdump --show-form --verbose --debug-info %t.bolt | FileCheck --check-prefix=POSTCHECK %s
 # RUN: llvm-dwarfdump --show-form --verbose --debug-types %t.bolt | FileCheck --check-prefix=POSTCHECKTU %s
 
diff --git a/bolt/test/X86/dwarf4-types-dwarf5.test b/bolt/test/X86/dwarf4-types-dwarf5.test
index 1eb42683e40ee..e1c5d1199fbad 100644
--- a/bolt/test/X86/dwarf4-types-dwarf5.test
+++ b/bolt/test/X86/dwarf4-types-dwarf5.test
@@ -3,7 +3,7 @@
 # RUN: llvm-mc -dwarf-version=4 -filetype=obj -triple x86_64-unknown-linux %p/Inputs/dwarf4-types-dwarf5-main.s -o %tmain.o
 # RUN: llvm-mc -dwarf-version=5 -filetype=obj -triple x86_64-unknown-linux %p/Inputs/dwarf4-types-dwarf5-helper.s -o %thelper.o
 # RUN: %clang %cflags %tmain.o %thelper.o -o %t.exe -Wl,-q
-# RUN: llvm-bolt %t.exe -o %t.bolt --update-debug-sections
+# RUN: llvm-bolt %t.exe -o %t.bolt --update-debug-sections --debug-thread-count=72
 # RUN: llvm-dwarfdump --show-form --verbose --debug-info %t.bolt | FileCheck --check-prefix=POSTCHECK %s
 # RUN: llvm-dwarfdump --show-form --verbose --debug-types %t.bolt | FileCheck --check-prefix=POSTCHECKTU %s
 
diff --git a/bolt/test/X86/dwarf5-addr-section-reuse.s b/bolt/test/X86/dwarf5-addr-section-reuse.s
index cf511d6d111e0..cf7fddd2240ac 100644
--- a/bolt/test/X86/dwarf5-addr-section-reuse.s
+++ b/bolt/test/X86/dwarf5-addr-section-reuse.s
@@ -3,7 +3,7 @@
 # RUN: llvm-mc -dwarf-version=5 -filetype=obj -triple x86_64-unknown-linux %p/Inputs/dwarf5-helper2-addr-section-reuse.s -o %thelper2.o
 # RUN: %clang %cflags -dwarf-5 %thelper1.o %tmain.o %thelper2.o -o %t.exe -Wl,-q
 # RUN: llvm-dwarfdump --debug-info %t.exe | FileCheck --check-prefix=PRECHECK %s
-# RUN: llvm-bolt %t.exe -o %t.exe.bolt --update-debug-sections
+# RUN: llvm-bolt %t.exe -o %t.exe.bolt --update-debug-sections --debug-thread-count=72
 # RUN: llvm-dwarfdump --debug-info %t.exe.bolt | FileCheck --check-prefix=POSTCHECK %s
 
 ## This test checks that when a binary is bolted if CU is not modified and has DW_AT_addr_base that is shared
diff --git a/bolt/test/X86/dwarf5-call-pc-function-null-check.test b/bolt/test/X86/dwarf5-call-pc-function-null-check.test
index 761a4da696217..3f1ec9da158e2 100644
--- a/bolt/test/X86/dwarf5-call-pc-function-null-check.test
+++ b/bolt/test/X86/dwarf5-call-pc-function-null-check.test
@@ -3,7 +3,7 @@
 # RUN: llvm-mc -dwarf-version=5 -filetype=obj -triple x86_64-unknown-linux %p/Inputs/dwarf5-call-pc-function-null-check-main.s -o %tmain.o
 # RUN: llvm-mc -dwarf-version=5 -filetype=obj -triple x86_64-unknown-linux %p/Inputs/dwarf5-call-pc-function-null-check-helper.s -o %thelper.o
 # RUN: %clang %cflags -dwarf-5 %tmain.o %thelper.o -o %t.exe -Wl,-q
-# RUN: llvm-bolt %t.exe -o %t.exe.bolt --update-debug-sections
+# RUN: llvm-bolt %t.exe -o %t.exe.bolt --update-debug-sections --debug-thread-count=72
 # RUN: llvm-dwarfdump --show-form --verbose --debug-info %t.exe > %t.txt
 # RUN: llvm-dwarfdump --show-form --verbose --debug-info %t.exe.bolt >> %t.txt
 # RUN: cat %t.txt | FileCheck --check-prefix=CHECK %s
diff --git a/bolt/test/X86/dwarf5-call-pc.test b/bolt/test/X86/dwarf5-call-pc.test
index dc7773dc053d9..1d7617faafa11 100644
--- a/bolt/test/X86/dwarf5-call-pc.test
+++ b/bolt/test/X86/dwarf5-call-pc.test
@@ -3,7 +3,7 @@
 # RUN: llvm-mc -dwarf-version=5 -filetype=obj -triple x86_64-unknown-linux %p/Inputs/dwarf5-call-pc-main.s -o %tmain.o
 # RUN: llvm-mc -dwarf-version=5 -filetype=obj -triple x86_64-unknown-linux %p/Inputs/dwarf5-call-pc-helper.s -o %thelper.o
 # RUN: %clang %cflags -dwarf-5 %tmain.o %thelper.o -o %t.exe -Wl,-q
-# RUN: llvm-bolt %t.exe -o %t.exe.bolt --update-debug-sections -reorder-blocks=reverse
+# RUN: llvm-bolt %t.exe -o %t.exe.bolt --update-debug-sections -reorder-blocks=reverse --debug-thread-count=72
 # RUN: llvm-dwarfdump --show-form --verbose --debug-info %t.exe > %tmain.txt
 # RUN: llvm-objdump %t.exe --disassemble >> %tmain.txt
 # RUN: llvm-dwarfdump --show-form --verbose --debug-info %t.exe.bolt > %tmainbolt.txt
diff --git a/bolt/test/X86/dwarf5-cu-no-debug-addr.test b/bolt/test/X86/dwarf5-cu-no-debug-addr.test
index e78b68680d6cc..f8b475ddb2b45 100644
--- a/bolt/test/X86/dwarf5-cu-no-debug-addr.test
+++ b/bolt/test/X86/dwarf5-cu-no-debug-addr.test
@@ -3,7 +3,7 @@
 # RUN: llvm-mc -dwarf-version=5 -filetype=obj -triple x86_64-unknown-linux %p/Inputs/dwarf5-cu-no-debug-addr-main.s -o %t1main.o
 # RUN: llvm-mc -dwarf-version=5 -filetype=obj -triple x86_64-unknown-linux %p/Inputs/dwarf5-cu-no-debug-addr-helper.s -o %t1helper.o
 # RUN: %clang %cflags -dwarf-5 %t1main.o %t1helper.o -o %t.exe -Wl,-q
-# RUN: llvm-bolt %t.exe -o %t.bolt --update-debug-sections
+# RUN: llvm-bolt %t.exe -o %t.bolt --update-debug-sections --debug-thread-count=72
 # RUN: llvm-dwarfdump --show-form --verbose --debug-info %t.exe  | FileCheck --check-prefix=PRECHECK %s
 # RUN: llvm-dwarfdump --show-form --verbose --debug-info %t.bolt | FileCheck --check-prefix=POSTCHECK %s
 
diff --git a/bolt/test/X86/dwarf5-df-input-lowpc-ranges-cus.test b/bolt/test/X86/dwarf5-df-input-lowpc-ranges-cus.test
index a325395fd5320..18f9105eb04bb 100644
--- a/bolt/test/X86/dwarf5-df-input-lowpc-ranges-cus.test
+++ b/bolt/test/X86/dwarf5-df-input-lowpc-ranges-cus.test
@@ -6,7 +6,7 @@
 ; RUN: llvm-mc -dwarf-version=5 -filetype=obj -triple x86_64-unknown-linux %p/Inputs/dwarf5-df-input-lowpc-ranges-other.s \
 ; RUN: -split-dwarf-file=mainOther.dwo -o other.o
 ; RUN: %clang %cflags main.o other.o -o main.exe
-; RUN: llvm-bolt main.exe -o main.exe.bolt --update-debug-sections
+; RUN: llvm-bolt main.exe -o main.exe.bolt --update-debug-sections --debug-thread-count=72
 ; RUN: llvm-dwarfdump --show-form --verbose --debug-rnglists main.exe.bolt &> %t/foo.txt
 ; RUN: llvm-dwarfdump --show-form --verbose --debug-addr main.exe.bolt >> %t/foo.txt
 ; RUN: llvm-dwarfdump --show-form --verbose --debug-info main.exe.bolt >> %t/foo.txt
diff --git a/bolt/test/X86/dwarf5-df-mono-dualcu.test b/bolt/test/X86/dwarf5-df-mono-dualcu.test
index 13272cc1c3c4d..69cd054a4c20a 100644
--- a/bolt/test/X86/dwarf5-df-mono-dualcu.test
+++ b/bolt/test/X86/dwarf5-df-mono-dualcu.test
@@ -5,7 +5,7 @@
 ; RUN: -split-dwarf-file=main.dwo -o main.o
 ; RUN: llvm-mc -filetype=obj -triple x86_64-unknown-linux-gnu %p/Inputs/dwarf5-df-mono-helper.s -o=helper.o
 ; RUN: %clang %cflags -gdwarf-5 main.o helper.o -o main.exe -fno-pic -no-pie
-; RUN: llvm-bolt main.exe -o main.exe.bolt --update-debug-sections --always-convert-to-ranges
+; RUN: llvm-bolt main.exe -o main.exe.bolt --update-debug-sections --always-convert-to-ranges --debug-thread-count=72
 ; RUN: llvm-dwarfdump --show-form --verbose --debug-info main.exe | FileCheck -check-prefix=PRE-BOLT %s
 ; RUN: llvm-dwarfdump --show-form --verbose --debug-addr main.exe.bolt &> %t/foo.txt
 ; RUN: llvm-dwarfdump --show-form --verbose --debug-info main.exe.bolt >> %t/foo.txt
diff --git a/bolt/test/X86/dwarf5-df-output-dir-same-name.test b/bolt/test/X86/dwarf5-...
[truncated]

ayermolo · 2024-07-24T01:05:36Z

I wonder if we can modify bolt/test/lit.local.cfg to add common llvm-bolt flags. So we don't have to copy past the same option everythere.
@aaupov WDYT?

bolt/lib/Rewrite/DWARFRewriter.cpp

bolt/lib/Core/ParallelUtilities.cpp

github-actions · 2024-07-24T23:01:27Z

✅ With the latest revision this PR passed the C/C++ code formatter.

aaupov · 2024-07-25T03:43:36Z

I wonder if we can modify bolt/test/lit.local.cfg to add common llvm-bolt flags. So we don't have to copy past the same option everythere. @aaupov WDYT?

If you add a top-level test folder (e.g. bolt/test/debug) and move relevant tests there, I'm fine with defining common BOLT flags used in those tests in lit.local.cfg.

bolt/lib/Core/ParallelUtilities.cpp

ayermolo · 2024-07-25T18:13:32Z

I wonder if we can modify bolt/test/lit.local.cfg to add common llvm-bolt flags. So we don't have to copy past the same option everythere. @aaupov WDYT?

If you add a top-level test folder (e.g. bolt/test/debug) and move relevant tests there, I'm fine with defining common BOLT flags used in those tests in lit.local.cfg.

OK, let's leave this for later.

ayermolo · 2024-07-25T18:56:39Z

Also update bolt/docs/CommandLineArgumentReference.md
And cu-processing-batch-size needs to be set.

bolt/include/bolt/Core/DebugData.h

bolt/lib/Core/BinaryContext.cpp

bolt/lib/Rewrite/DWARFRewriter.cpp

ayermolo · 2024-08-08T18:40:33Z

lGTM

aaupov

LG overall, but tests make me uneasy: that we're opting in all of them and setting thread-count to a rather high value of 72 (more than CUs per test). I would suggest reducing that to 4 so it's less suspicious.

Also, processing in parallel is not NFC to me, even if the output stays exactly the same (use significantly more resources). So I'd suggest dropping [NFC] from the title.

…formation Summary: Test Plan: Reviewers: Subscribers: Tasks: Tags: Differential Revision: https://phabricator.intern.facebook.com/D60151517

sayhaan force-pushed the T187681160-multithreading branch from 10a1442 to cbcf4b3 Compare July 23, 2024 23:48

sayhaan force-pushed the T187681160-multithreading branch from cbcf4b3 to 091ea2e Compare July 24, 2024 00:43

sayhaan commented Jul 24, 2024

View reviewed changes

bolt/include/bolt/Core/ParallelUtilities.h Outdated Show resolved Hide resolved

sayhaan marked this pull request as ready for review July 24, 2024 00:45

sayhaan requested review from aaupov, maksfb, rafaelauler, ayermolo and dcci as code owners July 24, 2024 00:45

llvmbot added the BOLT label Jul 24, 2024

sayhaan force-pushed the T187681160-multithreading branch from 091ea2e to 4fe6a42 Compare July 24, 2024 19:28

ayermolo reviewed Jul 24, 2024

View reviewed changes

bolt/lib/Rewrite/DWARFRewriter.cpp Outdated Show resolved Hide resolved

bolt/lib/Rewrite/DWARFRewriter.cpp Outdated Show resolved Hide resolved

bolt/lib/Core/ParallelUtilities.cpp Outdated Show resolved Hide resolved

ayermolo approved these changes Jul 25, 2024

View reviewed changes

ayermolo reviewed Jul 25, 2024

View reviewed changes

bolt/lib/Core/ParallelUtilities.cpp Outdated Show resolved Hide resolved

sayhaan force-pushed the T187681160-multithreading branch from 6645810 to 843e9e1 Compare August 2, 2024 16:37

ayermolo reviewed Aug 2, 2024

View reviewed changes

bolt/include/bolt/Core/DebugData.h Outdated Show resolved Hide resolved

bolt/lib/Core/BinaryContext.cpp Outdated Show resolved Hide resolved

sayhaan force-pushed the T187681160-multithreading branch from 47f366d to 95c1374 Compare August 7, 2024 17:17

ayermolo reviewed Aug 7, 2024

View reviewed changes

bolt/lib/Rewrite/DWARFRewriter.cpp Outdated Show resolved Hide resolved

bolt/lib/Rewrite/DWARFRewriter.cpp Show resolved Hide resolved

ayermolo reviewed Aug 8, 2024

View reviewed changes

bolt/lib/Rewrite/DWARFRewriter.cpp Outdated Show resolved Hide resolved

aaupov approved these changes Aug 8, 2024

View reviewed changes

sayhaan added 2 commits August 8, 2024 12:57

[BOLT][DWARF][NFC] Add parallelization for processing of DWO debug in…

a383d2b

…formation Summary: Test Plan: Reviewers: Subscribers: Tasks: Tags: Differential Revision: https://phabricator.intern.facebook.com/D60151517

Updates

e27c829

sayhaan force-pushed the T187681160-multithreading branch from 27c7cf3 to e27c829 Compare August 8, 2024 19:57

Updates

9f6dc99

sayhaan changed the title ~~[BOLT][DWARF][NFC] Add parallelization for processing of DWO debug information~~ [BOLT][DWARF] Add parallelization for processing of DWO debug information Aug 8, 2024

sayhaan merged commit 6aad62c into llvm:main Aug 8, 2024
5 of 6 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

[BOLT][DWARF] Add parallelization for processing of DWO debug information #100282

[BOLT][DWARF] Add parallelization for processing of DWO debug information #100282

Uh oh!

sayhaan commented Jul 23, 2024

Uh oh!

github-actions bot commented Jul 23, 2024 •

edited

Loading

Uh oh!

Uh oh!

llvmbot commented Jul 24, 2024

Uh oh!

ayermolo commented Jul 24, 2024

Uh oh!

Uh oh!

Uh oh!

Uh oh!

github-actions bot commented Jul 24, 2024 •

edited

Loading

Uh oh!

aaupov commented Jul 25, 2024

Uh oh!

Uh oh!

ayermolo commented Jul 25, 2024

Uh oh!

ayermolo commented Jul 25, 2024

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

ayermolo commented Aug 8, 2024

Uh oh!

aaupov left a comment •

edited

Loading

Uh oh!

Uh oh!

Uh oh!

[BOLT][DWARF] Add parallelization for processing of DWO debug information #100282

[BOLT][DWARF] Add parallelization for processing of DWO debug information #100282

Uh oh!

Conversation

sayhaan commented Jul 23, 2024

Uh oh!

github-actions bot commented Jul 23, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

llvmbot commented Jul 24, 2024

Uh oh!

ayermolo commented Jul 24, 2024

Uh oh!

Uh oh!

Uh oh!

Uh oh!

github-actions bot commented Jul 24, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

aaupov commented Jul 25, 2024

Uh oh!

Uh oh!

ayermolo commented Jul 25, 2024

Uh oh!

ayermolo commented Jul 25, 2024

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

ayermolo commented Aug 8, 2024

Uh oh!

aaupov left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

github-actions bot commented Jul 23, 2024 •

edited

Loading

github-actions bot commented Jul 24, 2024 •

edited

Loading

aaupov left a comment •

edited

Loading