Skip to content

[Reland] Report only loaded debug info in statistics dump (#81706) #82207

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 1 commit into from
Feb 19, 2024

Conversation

kusmour
Copy link
Contributor

@kusmour kusmour commented Feb 19, 2024

Updates:

  • The previous patch changed the default behavior to not load dwos in DWARFUnit
    SymbolFileDWARFDwo *GetDwoSymbolFile(bool load_all_debug_info = false);
    SymbolFileDWARFDwo *GetDwoSymbolFile(bool load_all_debug_info = true);
  • TestDebugInfoSize.py
    • with symbol on-demand, by default statistics dump only reports skeleton debug info size
    • statistics dump -f will load all dwos. debug info = skeleton debug info + all dwo debug info

Currently running statistics dump will trigger lldb to load debug info that's not yet loaded (eg. dwo files). Resulted in a delay in the command return, which, can be interrupting.

This patch also added a new option --load-all-debug-info asking statistics to dump all possible debug info, which will force loading all debug info available if not yet loaded.

@llvmbot
Copy link
Member

llvmbot commented Feb 19, 2024

@llvm/pr-subscribers-lldb

Author: Wanyi (kusmour)

Changes

Updates:

  • The previous patch changed the default behavior to not load dwos in DWARFUnit
    SymbolFileDWARFDwo *GetDwoSymbolFile(bool load_all_debug_info = false);
    SymbolFileDWARFDwo *GetDwoSymbolFile(bool load_all_debug_info = true);
  • TestDebugInfoSize.py
    • with symbol on-demand, by default statistics dump only reports skeleton debug info size
    • statistics dump -f will load all dwos. debug info = skeleton debug info + all dwo debug info

Currently running statistics dump will trigger lldb to load debug info that's not yet loaded (eg. dwo files). Resulted in a delay in the command return, which, can be interrupting.

This patch also added a new option --load-all-debug-info asking statistics to dump all possible debug info, which will force loading all debug info available if not yet loaded.


Patch is 41.07 KiB, truncated to 20.00 KiB below, full version: https://github.com/llvm/llvm-project/pull/82207.diff

25 Files Affected:

  • (modified) lldb/bindings/interface/SBStatisticsOptionsDocstrings.i (+6)
  • (modified) lldb/include/lldb/API/SBStatisticsOptions.h (+8)
  • (modified) lldb/include/lldb/Symbol/SymbolFile.h (+11-3)
  • (modified) lldb/include/lldb/Symbol/SymbolFileOnDemand.h (+1-1)
  • (modified) lldb/include/lldb/Target/Statistics.h (+1)
  • (modified) lldb/source/API/SBStatisticsOptions.cpp (+8)
  • (modified) lldb/source/Commands/CommandObjectStats.cpp (+3)
  • (modified) lldb/source/Commands/Options.td (+5-1)
  • (modified) lldb/source/Plugins/SymbolFile/Breakpad/SymbolFileBreakpad.cpp (+1-1)
  • (modified) lldb/source/Plugins/SymbolFile/Breakpad/SymbolFileBreakpad.h (+1-1)
  • (modified) lldb/source/Plugins/SymbolFile/DWARF/DWARFUnit.cpp (+3-2)
  • (modified) lldb/source/Plugins/SymbolFile/DWARF/DWARFUnit.h (+1-1)
  • (modified) lldb/source/Plugins/SymbolFile/DWARF/SymbolFileDWARF.cpp (+2-2)
  • (modified) lldb/source/Plugins/SymbolFile/DWARF/SymbolFileDWARF.h (+1-1)
  • (modified) lldb/source/Plugins/SymbolFile/DWARF/SymbolFileDWARFDwo.cpp (+1-1)
  • (modified) lldb/source/Plugins/SymbolFile/DWARF/SymbolFileDWARFDwo.h (+1-1)
  • (modified) lldb/source/Plugins/SymbolFile/NativePDB/SymbolFileNativePDB.cpp (+1-1)
  • (modified) lldb/source/Plugins/SymbolFile/NativePDB/SymbolFileNativePDB.h (+1-1)
  • (modified) lldb/source/Symbol/SymbolFile.cpp (+1-1)
  • (modified) lldb/source/Symbol/SymbolFileOnDemand.cpp (+2-2)
  • (modified) lldb/source/Target/Statistics.cpp (+3-1)
  • (modified) lldb/test/API/commands/target/debuginfo/TestDebugInfoSize.py (+11)
  • (modified) lldb/test/API/functionalities/stats_api/TestStatisticsAPI.py (+39)
  • (added) lldb/test/API/functionalities/stats_api/main-main.dwo.yaml (+37)
  • (added) lldb/test/API/functionalities/stats_api/main.yaml (+543)
diff --git a/lldb/bindings/interface/SBStatisticsOptionsDocstrings.i b/lldb/bindings/interface/SBStatisticsOptionsDocstrings.i
index f72cf84319e19b..087f6ab8786630 100644
--- a/lldb/bindings/interface/SBStatisticsOptionsDocstrings.i
+++ b/lldb/bindings/interface/SBStatisticsOptionsDocstrings.i
@@ -6,3 +6,9 @@
 ) lldb::SBStatisticsOptions::SetSummaryOnly;
 %feature("docstring", "Gets whether the statistics only dump a summary."
 ) lldb::SBStatisticsOptions::GetSummaryOnly;
+%feature("docstring", "
+    Sets whether the statistics will force loading all possible debug info."
+) lldb::SBStatisticsOptions::SetReportAllAvailableDebugInfo;
+%feature("docstring", "
+    Gets whether the statistics will force loading all possible debug info."
+) lldb::SBStatisticsOptions::GetReportAllAvailableDebugInfo;
diff --git a/lldb/include/lldb/API/SBStatisticsOptions.h b/lldb/include/lldb/API/SBStatisticsOptions.h
index 8019ed4315ca21..a0055135e36c2a 100644
--- a/lldb/include/lldb/API/SBStatisticsOptions.h
+++ b/lldb/include/lldb/API/SBStatisticsOptions.h
@@ -25,6 +25,14 @@ class LLDB_API SBStatisticsOptions {
   void SetSummaryOnly(bool b);
   bool GetSummaryOnly();
 
+  /// If set to true, the debugger will load all debug info that is available
+  /// and report statistics on the total amount. If this is set to false, then
+  /// only report statistics on the currently loaded debug information.
+  /// This can avoid loading debug info from separate files just so it can
+  /// report the total size which can slow down statistics reporting.
+  void SetReportAllAvailableDebugInfo(bool b);
+  bool GetReportAllAvailableDebugInfo();
+
 protected:
   friend class SBTarget;
   const lldb_private::StatisticsOptions &ref() const;
diff --git a/lldb/include/lldb/Symbol/SymbolFile.h b/lldb/include/lldb/Symbol/SymbolFile.h
index f356f7b789fa38..d20766788192f7 100644
--- a/lldb/include/lldb/Symbol/SymbolFile.h
+++ b/lldb/include/lldb/Symbol/SymbolFile.h
@@ -381,7 +381,8 @@ class SymbolFile : public PluginInterface {
 
   /// Metrics gathering functions
 
-  /// Return the size in bytes of all debug information in the symbol file.
+  /// Return the size in bytes of all loaded debug information or total possible
+  /// debug info in the symbol file.
   ///
   /// If the debug information is contained in sections of an ObjectFile, then
   /// this call should add the size of all sections that contain debug
@@ -391,7 +392,14 @@ class SymbolFile : public PluginInterface {
   /// entire file should be returned. The default implementation of this
   /// function will iterate over all sections in a module and add up their
   /// debug info only section byte sizes.
-  virtual uint64_t GetDebugInfoSize() = 0;
+  ///
+  /// \param load_all_debug_info
+  ///   If true, force loading any symbol files if they are not yet loaded and
+  ///   add to the total size. Default to false.
+  ///
+  /// \returns
+  ///   Total currently loaded debug info size in bytes
+  virtual uint64_t GetDebugInfoSize(bool load_all_debug_info = false) = 0;
 
   /// Return the time taken to parse the debug information.
   ///
@@ -534,7 +542,7 @@ class SymbolFileCommon : public SymbolFile {
 
   void Dump(Stream &s) override;
 
-  uint64_t GetDebugInfoSize() override;
+  uint64_t GetDebugInfoSize(bool load_all_debug_info = false) override;
 
   bool GetDebugInfoIndexWasLoadedFromCache() const override {
     return m_index_was_loaded_from_cache;
diff --git a/lldb/include/lldb/Symbol/SymbolFileOnDemand.h b/lldb/include/lldb/Symbol/SymbolFileOnDemand.h
index 4e3009941aa7d6..8073d1816860e3 100644
--- a/lldb/include/lldb/Symbol/SymbolFileOnDemand.h
+++ b/lldb/include/lldb/Symbol/SymbolFileOnDemand.h
@@ -178,7 +178,7 @@ class SymbolFileOnDemand : public lldb_private::SymbolFile {
 
   void PreloadSymbols() override;
 
-  uint64_t GetDebugInfoSize() override;
+  uint64_t GetDebugInfoSize(bool load_all_debug_info = false) override;
   lldb_private::StatsDuration::Duration GetDebugInfoParseTime() override;
   lldb_private::StatsDuration::Duration GetDebugInfoIndexTime() override;
 
diff --git a/lldb/include/lldb/Target/Statistics.h b/lldb/include/lldb/Target/Statistics.h
index f838fa17f80c24..c4f17b503a1f99 100644
--- a/lldb/include/lldb/Target/Statistics.h
+++ b/lldb/include/lldb/Target/Statistics.h
@@ -132,6 +132,7 @@ struct ConstStringStats {
 
 struct StatisticsOptions {
   bool summary_only = false;
+  bool load_all_debug_info = false;
 };
 
 /// A class that represents statistics for a since lldb_private::Target.
diff --git a/lldb/source/API/SBStatisticsOptions.cpp b/lldb/source/API/SBStatisticsOptions.cpp
index 77a7e26a6bd4b5..7e826c4c93ebcd 100644
--- a/lldb/source/API/SBStatisticsOptions.cpp
+++ b/lldb/source/API/SBStatisticsOptions.cpp
@@ -44,6 +44,14 @@ void SBStatisticsOptions::SetSummaryOnly(bool b) {
 
 bool SBStatisticsOptions::GetSummaryOnly() { return m_opaque_up->summary_only; }
 
+void SBStatisticsOptions::SetReportAllAvailableDebugInfo(bool b) {
+  m_opaque_up->load_all_debug_info = b;
+}
+
+bool SBStatisticsOptions::GetReportAllAvailableDebugInfo() {
+  return m_opaque_up->load_all_debug_info;
+}
+
 const lldb_private::StatisticsOptions &SBStatisticsOptions::ref() const {
   return *m_opaque_up;
 }
diff --git a/lldb/source/Commands/CommandObjectStats.cpp b/lldb/source/Commands/CommandObjectStats.cpp
index b23b7024c82176..a92bb5d1165ee6 100644
--- a/lldb/source/Commands/CommandObjectStats.cpp
+++ b/lldb/source/Commands/CommandObjectStats.cpp
@@ -78,6 +78,9 @@ class CommandObjectStatsDump : public CommandObjectParsed {
       case 's':
         m_stats_options.summary_only = true;
         break;
+      case 'f':
+        m_stats_options.load_all_debug_info = true;
+        break;
       default:
         llvm_unreachable("Unimplemented option");
       }
diff --git a/lldb/source/Commands/Options.td b/lldb/source/Commands/Options.td
index dd732e35220287..ad4321d9a386cc 100644
--- a/lldb/source/Commands/Options.td
+++ b/lldb/source/Commands/Options.td
@@ -1419,6 +1419,10 @@ let Command = "statistics dump" in {
   def statistics_dump_all: Option<"all-targets", "a">, Group<1>,
     Desc<"Include statistics for all targets.">;
   def statistics_dump_summary: Option<"summary", "s">, Group<1>,
-    Desc<"Dump only high-level summary statistics."
+    Desc<"Dump only high-level summary statistics. "
          "Exclude targets, modules, breakpoints etc... details.">;
+  def statistics_dump_force: Option<"load-all-debug-info", "f">, Group<1>,
+    Desc<"Dump the total possible debug info statistics. "
+    "Force loading all the debug information if not yet loaded, and collect "
+    "statistics with those.">;
 }
diff --git a/lldb/source/Plugins/SymbolFile/Breakpad/SymbolFileBreakpad.cpp b/lldb/source/Plugins/SymbolFile/Breakpad/SymbolFileBreakpad.cpp
index b1f7397d6b0f00..3977dc3a6d67c5 100644
--- a/lldb/source/Plugins/SymbolFile/Breakpad/SymbolFileBreakpad.cpp
+++ b/lldb/source/Plugins/SymbolFile/Breakpad/SymbolFileBreakpad.cpp
@@ -918,7 +918,7 @@ void SymbolFileBreakpad::ParseUnwindData() {
   m_unwind_data->win.Sort();
 }
 
-uint64_t SymbolFileBreakpad::GetDebugInfoSize() {
+uint64_t SymbolFileBreakpad::GetDebugInfoSize(bool load_all_debug_info) {
   // Breakpad files are all debug info.
   return m_objfile_sp->GetByteSize();
 }
diff --git a/lldb/source/Plugins/SymbolFile/Breakpad/SymbolFileBreakpad.h b/lldb/source/Plugins/SymbolFile/Breakpad/SymbolFileBreakpad.h
index 41e4e3b258014c..83215bf3c87e4a 100644
--- a/lldb/source/Plugins/SymbolFile/Breakpad/SymbolFileBreakpad.h
+++ b/lldb/source/Plugins/SymbolFile/Breakpad/SymbolFileBreakpad.h
@@ -141,7 +141,7 @@ class SymbolFileBreakpad : public SymbolFileCommon {
 
   llvm::StringRef GetPluginName() override { return GetPluginNameStatic(); }
 
-  uint64_t GetDebugInfoSize() override;
+  uint64_t GetDebugInfoSize(bool load_all_debug_info = false) override;
 
 private:
   // A class representing a position in the breakpad file. Useful for
diff --git a/lldb/source/Plugins/SymbolFile/DWARF/DWARFUnit.cpp b/lldb/source/Plugins/SymbolFile/DWARF/DWARFUnit.cpp
index 23e0b8a7f2c06b..e28036d34b34a6 100644
--- a/lldb/source/Plugins/SymbolFile/DWARF/DWARFUnit.cpp
+++ b/lldb/source/Plugins/SymbolFile/DWARF/DWARFUnit.cpp
@@ -896,8 +896,9 @@ void DWARFUnit::ComputeAbsolutePath() {
     m_file_spec->MakeAbsolute(GetCompilationDirectory());
 }
 
-SymbolFileDWARFDwo *DWARFUnit::GetDwoSymbolFile() {
-  ExtractUnitDIEIfNeeded();
+SymbolFileDWARFDwo *DWARFUnit::GetDwoSymbolFile(bool load_all_debug_info) {
+  if (load_all_debug_info)
+    ExtractUnitDIEIfNeeded();
   if (m_dwo)
     return &llvm::cast<SymbolFileDWARFDwo>(m_dwo->GetSymbolFileDWARF());
   return nullptr;
diff --git a/lldb/source/Plugins/SymbolFile/DWARF/DWARFUnit.h b/lldb/source/Plugins/SymbolFile/DWARF/DWARFUnit.h
index 9f6d127056fa56..28981b51bfcb33 100644
--- a/lldb/source/Plugins/SymbolFile/DWARF/DWARFUnit.h
+++ b/lldb/source/Plugins/SymbolFile/DWARF/DWARFUnit.h
@@ -241,7 +241,7 @@ class DWARFUnit : public UserID {
   FileSpec GetFile(size_t file_idx);
   FileSpec::Style GetPathStyle();
 
-  SymbolFileDWARFDwo *GetDwoSymbolFile();
+  SymbolFileDWARFDwo *GetDwoSymbolFile(bool load_all_debug_info = true);
 
   die_iterator_range dies() {
     ExtractDIEsIfNeeded();
diff --git a/lldb/source/Plugins/SymbolFile/DWARF/SymbolFileDWARF.cpp b/lldb/source/Plugins/SymbolFile/DWARF/SymbolFileDWARF.cpp
index 781f5c5a436778..42211b9a21b0e3 100644
--- a/lldb/source/Plugins/SymbolFile/DWARF/SymbolFileDWARF.cpp
+++ b/lldb/source/Plugins/SymbolFile/DWARF/SymbolFileDWARF.cpp
@@ -2672,7 +2672,7 @@ static bool UpdateCompilerContextForSimpleTemplateNames(TypeQuery &match) {
   return any_context_updated;
 }
 
-uint64_t SymbolFileDWARF::GetDebugInfoSize() {
+uint64_t SymbolFileDWARF::GetDebugInfoSize(bool load_all_debug_info) {
   DWARFDebugInfo &info = DebugInfo();
   uint32_t num_comp_units = info.GetNumUnits();
 
@@ -2687,7 +2687,7 @@ uint64_t SymbolFileDWARF::GetDebugInfoSize() {
     if (cu == nullptr)
       continue;
 
-    SymbolFileDWARFDwo *dwo = cu->GetDwoSymbolFile();
+    SymbolFileDWARFDwo *dwo = cu->GetDwoSymbolFile(load_all_debug_info);
     if (dwo)
       debug_info_size += dwo->GetDebugInfoSize();
   }
diff --git a/lldb/source/Plugins/SymbolFile/DWARF/SymbolFileDWARF.h b/lldb/source/Plugins/SymbolFile/DWARF/SymbolFileDWARF.h
index 01518b26ca669e..2f8f80f8765cb8 100644
--- a/lldb/source/Plugins/SymbolFile/DWARF/SymbolFileDWARF.h
+++ b/lldb/source/Plugins/SymbolFile/DWARF/SymbolFileDWARF.h
@@ -186,7 +186,7 @@ class SymbolFileDWARF : public SymbolFileCommon {
   GetMangledNamesForFunction(const std::string &scope_qualified_name,
                              std::vector<ConstString> &mangled_names) override;
 
-  uint64_t GetDebugInfoSize() override;
+  uint64_t GetDebugInfoSize(bool load_all_debug_info = false) override;
 
   void FindTypes(const lldb_private::TypeQuery &match,
                  lldb_private::TypeResults &results) override;
diff --git a/lldb/source/Plugins/SymbolFile/DWARF/SymbolFileDWARFDwo.cpp b/lldb/source/Plugins/SymbolFile/DWARF/SymbolFileDWARFDwo.cpp
index ea23b75c3d708d..a0a7012dfaf0d7 100644
--- a/lldb/source/Plugins/SymbolFile/DWARF/SymbolFileDWARFDwo.cpp
+++ b/lldb/source/Plugins/SymbolFile/DWARF/SymbolFileDWARFDwo.cpp
@@ -85,7 +85,7 @@ lldb::offset_t SymbolFileDWARFDwo::GetVendorDWARFOpcodeSize(
   return GetBaseSymbolFile().GetVendorDWARFOpcodeSize(data, data_offset, op);
 }
 
-uint64_t SymbolFileDWARFDwo::GetDebugInfoSize() {
+uint64_t SymbolFileDWARFDwo::GetDebugInfoSize(bool load_all_debug_info) {
   // Directly get debug info from current dwo object file's section list
   // instead of asking SymbolFileCommon::GetDebugInfo() which parses from
   // owning module which is wrong.
diff --git a/lldb/source/Plugins/SymbolFile/DWARF/SymbolFileDWARFDwo.h b/lldb/source/Plugins/SymbolFile/DWARF/SymbolFileDWARFDwo.h
index d5f48f2a8ed4e2..c2c420f711d345 100644
--- a/lldb/source/Plugins/SymbolFile/DWARF/SymbolFileDWARFDwo.h
+++ b/lldb/source/Plugins/SymbolFile/DWARF/SymbolFileDWARFDwo.h
@@ -47,7 +47,7 @@ class SymbolFileDWARFDwo : public SymbolFileDWARF {
                                           const lldb::offset_t data_offset,
                                           const uint8_t op) const override;
 
-  uint64_t GetDebugInfoSize() override;
+  uint64_t GetDebugInfoSize(bool load_all_debug_info = false) override;
 
   bool ParseVendorDWARFOpcode(uint8_t op, const DataExtractor &opcodes,
                               lldb::offset_t &offset,
diff --git a/lldb/source/Plugins/SymbolFile/NativePDB/SymbolFileNativePDB.cpp b/lldb/source/Plugins/SymbolFile/NativePDB/SymbolFileNativePDB.cpp
index 745685a1b31d05..7fded6a31a3af5 100644
--- a/lldb/source/Plugins/SymbolFile/NativePDB/SymbolFileNativePDB.cpp
+++ b/lldb/source/Plugins/SymbolFile/NativePDB/SymbolFileNativePDB.cpp
@@ -2156,7 +2156,7 @@ SymbolFileNativePDB::GetTypeSystemForLanguage(lldb::LanguageType language) {
   return type_system_or_err;
 }
 
-uint64_t SymbolFileNativePDB::GetDebugInfoSize() {
+uint64_t SymbolFileNativePDB::GetDebugInfoSize(bool load_all_debug_info) {
   // PDB files are a separate file that contains all debug info.
   return m_index->pdb().getFileSize();
 }
diff --git a/lldb/source/Plugins/SymbolFile/NativePDB/SymbolFileNativePDB.h b/lldb/source/Plugins/SymbolFile/NativePDB/SymbolFileNativePDB.h
index 82577771f355c8..669c44aa131edc 100644
--- a/lldb/source/Plugins/SymbolFile/NativePDB/SymbolFileNativePDB.h
+++ b/lldb/source/Plugins/SymbolFile/NativePDB/SymbolFileNativePDB.h
@@ -77,7 +77,7 @@ class SymbolFileNativePDB : public SymbolFileCommon {
 
   void InitializeObject() override;
 
-  uint64_t GetDebugInfoSize() override;
+  uint64_t GetDebugInfoSize(bool load_all_debug_info = false) override;
 
   // Compile Unit function calls
 
diff --git a/lldb/source/Symbol/SymbolFile.cpp b/lldb/source/Symbol/SymbolFile.cpp
index e318e2beb6547b..16ed98d7840f78 100644
--- a/lldb/source/Symbol/SymbolFile.cpp
+++ b/lldb/source/Symbol/SymbolFile.cpp
@@ -227,7 +227,7 @@ SymbolFileCommon::GetTypeSystemForLanguage(lldb::LanguageType language) {
   return type_system_or_err;
 }
 
-uint64_t SymbolFileCommon::GetDebugInfoSize() {
+uint64_t SymbolFileCommon::GetDebugInfoSize(bool load_all_debug_info) {
   if (!m_objfile_sp)
     return 0;
   ModuleSP module_sp(m_objfile_sp->GetModule());
diff --git a/lldb/source/Symbol/SymbolFileOnDemand.cpp b/lldb/source/Symbol/SymbolFileOnDemand.cpp
index bdb1951d51259d..c6d9f0071c392b 100644
--- a/lldb/source/Symbol/SymbolFileOnDemand.cpp
+++ b/lldb/source/Symbol/SymbolFileOnDemand.cpp
@@ -535,11 +535,11 @@ void SymbolFileOnDemand::PreloadSymbols() {
   return m_sym_file_impl->PreloadSymbols();
 }
 
-uint64_t SymbolFileOnDemand::GetDebugInfoSize() {
+uint64_t SymbolFileOnDemand::GetDebugInfoSize(bool load_all_debug_info) {
   // Always return the real debug info size.
   LLDB_LOG(GetLog(), "[{0}] {1} is not skipped", GetSymbolFileName(),
            __FUNCTION__);
-  return m_sym_file_impl->GetDebugInfoSize();
+  return m_sym_file_impl->GetDebugInfoSize(load_all_debug_info);
 }
 
 StatsDuration::Duration SymbolFileOnDemand::GetDebugInfoParseTime() {
diff --git a/lldb/source/Target/Statistics.cpp b/lldb/source/Target/Statistics.cpp
index ec0a4c84692dea..7f866ae0ef3242 100644
--- a/lldb/source/Target/Statistics.cpp
+++ b/lldb/source/Target/Statistics.cpp
@@ -224,6 +224,7 @@ llvm::json::Value DebuggerStats::ReportStatistics(
     const lldb_private::StatisticsOptions &options) {
 
   const bool summary_only = options.summary_only;
+  const bool load_all_debug_info = options.load_all_debug_info;
 
   json::Array json_targets;
   json::Array json_modules;
@@ -280,7 +281,8 @@ llvm::json::Value DebuggerStats::ReportStatistics(
         ++debug_index_saved;
       module_stat.debug_index_time = sym_file->GetDebugInfoIndexTime().count();
       module_stat.debug_parse_time = sym_file->GetDebugInfoParseTime().count();
-      module_stat.debug_info_size = sym_file->GetDebugInfoSize();
+      module_stat.debug_info_size =
+          sym_file->GetDebugInfoSize(load_all_debug_info);
       module_stat.symtab_stripped = module->GetObjectFile()->IsStripped();
       if (module_stat.symtab_stripped)
         ++num_stripped_modules;
diff --git a/lldb/test/API/commands/target/debuginfo/TestDebugInfoSize.py b/lldb/test/API/commands/target/debuginfo/TestDebugInfoSize.py
index a70212fb426268..c5f345d45af89c 100644
--- a/lldb/test/API/commands/target/debuginfo/TestDebugInfoSize.py
+++ b/lldb/test/API/commands/target/debuginfo/TestDebugInfoSize.py
@@ -120,6 +120,7 @@ def test_dwos_loaded_symbols_on_demand(self):
         target = self.dbg.CreateTarget(exe)
         self.assertTrue(target, lldbtest.VALID_TARGET)
 
+        # By default dwo files will not be loaded
         stats = target.GetStatistics()
         stream = lldb.SBStream()
         res = stats.GetAsJSON(stream)
@@ -129,6 +130,16 @@ def test_dwos_loaded_symbols_on_demand(self):
             debug_stats,
             'Make sure the "totalDebugInfoByteSize" key is in target.GetStatistics()',
         )
+        self.assertEqual(
+            debug_stats["totalDebugInfoByteSize"], SKELETON_DEBUGINFO_SIZE)
+        
+        # Force loading all the dwo files
+        stats_options = lldb.SBStatisticsOptions()
+        stats_options.SetReportAllAvailableDebugInfo(True)
+        stats = target.GetStatistics(stats_options)
+        stream = lldb.SBStream()
+        stats.GetAsJSON(stream)
+        debug_stats = json.loads(stream.GetData())
         self.assertEqual(
             debug_stats["totalDebugInfoByteSize"],
             SKELETON_DEBUGINFO_SIZE + MAIN_DWO_DEBUGINFO_SIZE + FOO_DWO_DEBUGINFO_SIZE,
diff --git a/lldb/test/API/functionalities/stats_api/TestStatisticsAPI.py b/lldb/test/API/functionalities/stats_api/TestStatisticsAPI.py
index 457a2022fe4b8f..eee91bfadead97 100644
--- a/lldb/test/API/functionalities/stats_api/TestStatisticsAPI.py
+++ b/lldb/test/API/functionalities/stats_api/TestStatisticsAPI.py
@@ -117,3 +117,42 @@ def test_command_stats_api(self):
         self.assertNotIn("bt", command_stats)
         # Verify bt's regex command is not duplicatedly captured.
         self.assertNotIn("_regexp-bt", command_stats)
+
+    @add_test_categories(["dwo"])
+    def test_command_stats_force(self):
+        """
+        Test reporting all pssible debug info stats by force loading all debug
+        info. For example, dwo files
+        """
+        src_dir = self.getSourceDir()
+        dwo_yaml_path = os.path.join(src_dir, "main-main.dwo.yaml")
+        exe_yaml_path = os.path.join(src_dir, "main.yaml")
+        dwo_path = self.getBuildArtifact("main-main.dwo")
+        exe_path = self.getBuildArtifact("main")
+        self.yaml2obj(dwo_yaml_path, dwo_path)
+        self.yaml2obj(exe_yaml_path, exe_path)
+
+        # Turn on symbols on-demand loading
+        self.runCmd("settings set symbols.load-on-demand true")
+
+        # We need the current working directory to be set to the build directory
+        os.chdir(self.getBuildDir())
+        # Create a target with the object file we just created from YAML
+        target = self.dbg.CreateTarget(exe_path)
+        self.assertTrue(target, VALID_TARGET)
+
+        # Get statistics
+        stats_options = lldb.SBStatisticsOptions()
+        stats = target.GetStatistics(stats_options)
+        stream = lldb.SBStream()
+        stats.GetAsJSON(stream)
+        debug_stats = json.loads(stream.GetData())
+        self.assertEqual(debug_stats["totalDebugInfoByteSize"], 193)
+
+        # Get statistics with force loading
+        stats_options.SetReportAllAvailableDebugInfo(True)
+        stats_force = target.GetStatistics(stats_options)
+        stream_force = lldb.SBStream()
+        stats_force.GetAsJSON(stream_force)
+        debug_stats_force = json.loads(stream_force.GetData())
+        self.assertEqual(debug_stats_force["totalDebugInfoByteSize"], 445)
diff --git a/lldb/test/API/functionalities/stats_api/main-main.dwo.yaml b/lldb/test/API/functionalities/stats_api/main-main.dwo.yaml
new file mode 100644
index 00000000000000..d24dde3ce96d97
--- /dev/null
+++ b/lldb/test/API/functionalities/stats_api/main-main.dwo.yaml
@@ -0,0 +1,37 @@
+--- !ELF
+FileHeader:
+  Class:           ELFCLASS64
+  Data:            ELFDATA2LSB
+  Type:            ET_REL
+  Machine:         EM_X86_64
+  SectionHeaderStringTable: .strtab
+Sections:
+  - Name:            .debug_str...
[truncated]

Copy link

github-actions bot commented Feb 19, 2024

✅ With the latest revision this PR passed the Python code formatter.

@kusmour
Copy link
Contributor Author

kusmour commented Feb 19, 2024

For the 2 links of test failures in the original PR:
I've run ninja check-lldb on x86 linux to confirm the lldb-shell test failures in https://green.lab.llvm.org/green/view/LLDB/job/as-lldb-cmake/16273/ are fixed.
Unfortunately, I wasn't able to repro most test failures in https://lab.llvm.org/buildbot/#/builders/68/builds/69018 with or without my patch. I am assuming there're other factors causing the failures.

@kusmour kusmour force-pushed the lldb-debuginfo-size branch from bd9ea7d to 8c4f6de Compare February 19, 2024 05:18
Currently running `statistics dump` will trigger lldb to load debug info
that's not yet loaded (eg. dwo files). Resulted in a delay in the
command return, which, can be interrupting.

This patch also added a new option `--load-all-debug-info` asking
statistics to dump all possible debug info, which will force loading all
debug info available if not yet loaded.
@kusmour kusmour force-pushed the lldb-debuginfo-size branch from 8c4f6de to f71ff3b Compare February 19, 2024 05:28
Copy link
Member

@JDevlieghere JDevlieghere left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks! I applied your patch locally and the test suite passes cleanly on macOS as well. LGTM!

@kusmour kusmour merged commit dd7386d into llvm:main Feb 19, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants