[lldb] Decouple importing the std C++ module from the way the program is compiled

Summary:
At the moment, when trying to import the `std` module in LLDB, we look at the imported modules used in the compiled program
and try to infer the Clang configuration we need from the DWARF module-import. That was the initial idea but turned out to
cause a few problems or inconveniences:

* It requires that users compile their programs with C++ modules. Given how experimental C++ modules are makes this feature inaccessible
for many users. Also it means that people can't just get the benefits of this feature for free when we activate it by default
(and we can't just close all the associated bug reports).
* Relying on DWARF's imported module tags (that are only emitted by default on macOS) means this can only be used when using DWARF (and with -glldb on Linux).
* We essentially hardcoded the C standard library paths on some platforms (Linux) or just couldn't support this feature on other platforms (macOS).

This patch drops the whole idea of looking at the imported module DWARF tags and instead just uses the support files of the compilation unit.
If we look at the support files and see file paths that indicate where the C standard library and libc++ are, we can just create the module
configuration this information. This fixes all the problems above which means we can enable all the tests now on Linux, macOS and with other debug information
than what we currently had. The only debug information specific code is now the iteration over external type module when -gmodules is used (as `std` and also the
`Darwin` module are their own external type module with their own files).

The meat of this patch is the CppModuleConfiguration which looks at the file paths from the compilation unit and then figures out the include paths
based on those paths. It's quite conservative in that it only enables modules if we find a single C library and single libc++ library. It's still missing some
test mode where we try to compile an expression before we actually activate the config for the user (which probably also needs some caching mechanism),
but for now it works and makes the feature usable.

Reviewers: aprantl, shafik, jdoerfert

Reviewed By: aprantl

Subscribers: mgorny, abidh, JDevlieghere, lldb-commits

Tags: #c_modules_in_lldb, #lldb

Differential Revision: https://reviews.llvm.org/D67760

llvm-svn: 372716

Author: Teemperor

lldb/include/lldb/Symbol/CompileUnit.h

@@ -225,6 +225,14 @@ class CompileUnit : public std::enable_shared_from_this<CompileUnit>,
 
   DebugMacros *GetDebugMacros();
 
+  /// Apply a lambda to each external lldb::Module referenced by this
+  /// compilation unit. Recursively also descends into the referenced external
+  /// modules of any encountered compilation unit.
+  ///
+  /// \param[in] lambda
+  ///     The lambda that should be applied to every module.
+  void ForEachExternalModule(llvm::function_ref<void(lldb::ModuleSP)> f);
+
   /// Get the compile unit's support file list.
   ///
   /// The support file list is used by the line table, and any objects that

lldb/include/lldb/Symbol/SymbolFile.h

@@ -122,6 +122,9 @@ class SymbolFile : public PluginInterface {
   virtual size_t ParseFunctions(CompileUnit &comp_unit) = 0;
   virtual bool ParseLineTable(CompileUnit &comp_unit) = 0;
   virtual bool ParseDebugMacros(CompileUnit &comp_unit) = 0;
+  virtual void
+  ForEachExternalModule(CompileUnit &comp_unit,
+                        llvm::function_ref<void(lldb::ModuleSP)> f) {}
   virtual bool ParseSupportFiles(CompileUnit &comp_unit,
                                  FileSpecList &support_files) = 0;
   virtual size_t ParseTypes(CompileUnit &comp_unit) = 0;

lldb/include/lldb/Target/Platform.h

@@ -257,19 +257,6 @@ class Platform : public PluginInterface {
 
   virtual bool SetRemoteWorkingDirectory(const FileSpec &working_dir);
 
-  /// Retrieve the system include directories on this platform for the
-  /// given language.
-  ///
-  /// \param[in] lang
-  ///     The language for which the include directories should be queried.
-  ///
-  /// \param[out] directories
-  ///     The include directories for this system.
-  virtual std::vector<std::string>
-  GetSystemIncludeDirectories(lldb::LanguageType lang) {
-    return {};
-  }
-
   virtual UserIDResolver &GetUserIDResolver() = 0;
 
   /// Locate a file for a platform.

lldb/packages/Python/lldbsuite/test/commands/expression/import-std-module/basic/Makefile

@@ -1,4 +1,3 @@
 USE_LIBCPP := 1
-CXXFLAGS += $(MANDATORY_CXXMODULE_BUILD_CFLAGS)
 CXX_SOURCES := main.cpp
 include Makefile.rules

lldb/packages/Python/lldbsuite/test/commands/expression/import-std-module/basic/TestImportStdModule.py

@@ -11,12 +11,8 @@ class ImportStdModule(TestBase):
 
     mydir = TestBase.compute_mydir(__file__)
 
-    # FIXME: This should work on more setups, so remove these
-    # skipIf's in the future.
     @add_test_categories(["libc++"])
     @skipIf(compiler=no_match("clang"))
-    @skipIf(oslist=no_match(["linux"]))
-    @skipIf(debug_info=no_match(["dwarf"]))
     def test(self):
         self.build()
 
@@ -34,12 +30,8 @@ def test(self):
         self.expect("expr char char_a = 'b'; char char_b = 'a'; std::swap(char_a, char_b); char_a",
                     substrs=["(char) $3 = 'a'"])
 
-    # FIXME: This should work on more setups, so remove these
-    # skipIf's in the future.
     @add_test_categories(["libc++"])
     @skipIf(compiler=no_match("clang"))
-    @skipIf(oslist=no_match(["linux"]))
-    @skipIf(debug_info=no_match(["dwarf"]))
     def test_non_cpp_language(self):
         self.build()
 

lldb/packages/Python/lldbsuite/test/commands/expression/import-std-module/conflicts/Makefile

@@ -1,4 +1,3 @@
 USE_LIBCPP := 1
-CXXFLAGS += $(MANDATORY_CXXMODULE_BUILD_CFLAGS)
 CXX_SOURCES := main.cpp
 include Makefile.rules

lldb/packages/Python/lldbsuite/test/commands/expression/import-std-module/conflicts/TestStdModuleWithConflicts.py

@@ -16,12 +16,8 @@ class TestImportStdModuleConflicts(TestBase):
 
     mydir = TestBase.compute_mydir(__file__)
 
-    # FIXME: This should work on more setups, so remove these
-    # skipIf's in the future.
     @add_test_categories(["libc++"])
     @skipIf(compiler=no_match("clang"))
-    @skipIf(oslist=no_match(["linux"]))
-    @skipIf(debug_info=no_match(["dwarf"]))
     def test(self):
         self.build()
 

lldb/packages/Python/lldbsuite/test/commands/expression/import-std-module/deque-basic/Makefile

@@ -1,4 +1,3 @@
 USE_LIBCPP := 1
-CXXFLAGS += $(MANDATORY_CXXMODULE_BUILD_CFLAGS)
 CXX_SOURCES := main.cpp
 include Makefile.rules

lldb/packages/Python/lldbsuite/test/commands/expression/import-std-module/deque-basic/TestBasicDeque.py

@@ -10,12 +10,8 @@ class TestBasicDeque(TestBase):
 
     mydir = TestBase.compute_mydir(__file__)
 
-    # FIXME: This should work on more setups, so remove these
-    # skipIf's in the future.
     @add_test_categories(["libc++"])
     @skipIf(compiler=no_match("clang"))
-    @skipIf(oslist=no_match(["linux"]))
-    @skipIf(debug_info=no_match(["dwarf"]))
     def test(self):
         self.build()
 

lldb/packages/Python/lldbsuite/test/commands/expression/import-std-module/deque-dbg-info-content/Makefile

@@ -1,4 +1,3 @@
 USE_LIBCPP := 1
-CXXFLAGS += $(MANDATORY_CXXMODULE_BUILD_CFLAGS)
 CXX_SOURCES := main.cpp
 include Makefile.rules

lldb/packages/Python/lldbsuite/test/commands/expression/import-std-module/deque-dbg-info-content/TestDbgInfoContentDeque.py

@@ -10,12 +10,8 @@ class TestDbgInfoContentDeque(TestBase):
 
     mydir = TestBase.compute_mydir(__file__)
 
-    # FIXME: This should work on more setups, so remove these
-    # skipIf's in the future.
     @add_test_categories(["libc++"])
     @skipIf(compiler=no_match("clang"))
-    @skipIf(oslist=no_match(["linux"]))
-    @skipIf(debug_info=no_match(["dwarf"]))
     def test(self):
         self.build()
 

lldb/packages/Python/lldbsuite/test/commands/expression/import-std-module/forward_list-basic/Makefile

@@ -1,4 +1,3 @@
 USE_LIBCPP := 1
-CXXFLAGS += $(MANDATORY_CXXMODULE_BUILD_CFLAGS)
 CXX_SOURCES := main.cpp
 include Makefile.rules

lldb/packages/Python/lldbsuite/test/commands/expression/import-std-module/forward_list-basic/TestBasicForwardList.py

@@ -10,12 +10,8 @@ class TestBasicForwardList(TestBase):
 
     mydir = TestBase.compute_mydir(__file__)
 
-    # FIXME: This should work on more setups, so remove these
-    # skipIf's in the future.
     @add_test_categories(["libc++"])
     @skipIf(compiler=no_match("clang"))
-    @skipIf(oslist=no_match(["linux"]))
-    @skipIf(debug_info=no_match(["dwarf"]))
     def test(self):
         self.build()
 

lldb/packages/Python/lldbsuite/test/commands/expression/import-std-module/forward_list-dbg-info-content/Makefile

@@ -1,4 +1,3 @@
 USE_LIBCPP := 1
-CXXFLAGS += $(MANDATORY_CXXMODULE_BUILD_CFLAGS)
 CXX_SOURCES := main.cpp
 include Makefile.rules

lldb/packages/Python/lldbsuite/test/commands/expression/import-std-module/forward_list-dbg-info-content/TestDbgInfoContentForwardList.py

@@ -10,12 +10,8 @@ class TestDbgInfoContentForwardList(TestBase):
 
     mydir = TestBase.compute_mydir(__file__)
 
-    # FIXME: This should work on more setups, so remove these
-    # skipIf's in the future.
     @add_test_categories(["libc++"])
     @skipIf(compiler=no_match("clang"))
-    @skipIf(oslist=no_match(["linux"]))
-    @skipIf(debug_info=no_match(["dwarf"]))
     def test(self):
         self.build()
 

lldb/packages/Python/lldbsuite/test/commands/expression/import-std-module/list-basic/Makefile

@@ -1,4 +1,3 @@
 USE_LIBCPP := 1
-CXXFLAGS += $(MANDATORY_CXXMODULE_BUILD_CFLAGS)
 CXX_SOURCES := main.cpp
 include Makefile.rules

lldb/packages/Python/lldbsuite/test/commands/expression/import-std-module/list-basic/TestBasicList.py

@@ -10,12 +10,8 @@ class TestBasicList(TestBase):
 
     mydir = TestBase.compute_mydir(__file__)
 
-    # FIXME: This should work on more setups, so remove these
-    # skipIf's in the future.
     @add_test_categories(["libc++"])
     @skipIf(compiler=no_match("clang"))
-    @skipIf(oslist=no_match(["linux"]))
-    @skipIf(debug_info=no_match(["dwarf"]))
     def test(self):
         self.build()
 

lldb/packages/Python/lldbsuite/test/commands/expression/import-std-module/list-dbg-info-content/Makefile

@@ -1,4 +1,3 @@
 USE_LIBCPP := 1
-CXXFLAGS += $(MANDATORY_CXXMODULE_BUILD_CFLAGS)
 CXX_SOURCES := main.cpp
 include Makefile.rules

lldb/packages/Python/lldbsuite/test/commands/expression/import-std-module/list-dbg-info-content/TestDbgInfoContentList.py

@@ -11,12 +11,8 @@ class TestDbgInfoContentList(TestBase):
 
     mydir = TestBase.compute_mydir(__file__)
 
-    # FIXME: This should work on more setups, so remove these
-    # skipIf's in the future.
     @add_test_categories(["libc++"])
     @skipIf(compiler=no_match("clang"))
-    @skipIf(oslist=no_match(["linux"]))
-    @skipIf(debug_info=no_match(["dwarf"]))
     def test(self):
         self.build()
 

lldb/packages/Python/lldbsuite/test/commands/expression/import-std-module/no-std-module/Makefile

@@ -1,4 +1,3 @@
 USE_LIBCPP := 1
-CXXFLAGS += $(MANDATORY_CXXMODULE_BUILD_CFLAGS)
 CXX_SOURCES := main.cpp
 include Makefile.rules

lldb/packages/Python/lldbsuite/test/commands/expression/import-std-module/no-std-module/TestMissingStdModule.py

@@ -17,12 +17,8 @@ class STLTestCase(TestBase):
 
     mydir = TestBase.compute_mydir(__file__)
 
-    # FIXME: This should work on more setups, so remove these
-    # skipIf's in the future.
     @add_test_categories(["libc++"])
     @skipIf(compiler=no_match("clang"))
-    @skipIf(oslist=no_match(["linux"]))
-    @skipIf(debug_info=no_match(["dwarf"]))
     def test(self):
         self.build()
 

lldb/packages/Python/lldbsuite/test/commands/expression/import-std-module/queue/Makefile

@@ -1,4 +1,3 @@
 USE_LIBCPP := 1
-CXXFLAGS += $(MANDATORY_CXXMODULE_BUILD_CFLAGS)
 CXX_SOURCES := main.cpp
 include Makefile.rules

lldb/packages/Python/lldbsuite/test/commands/expression/import-std-module/queue/TestQueue.py

@@ -10,12 +10,8 @@ class TestQueue(TestBase):
 
     mydir = TestBase.compute_mydir(__file__)
 
-    # FIXME: This should work on more setups, so remove these
-    # skipIf's in the future.
     @add_test_categories(["libc++"])
     @skipIf(compiler=no_match("clang"))
-    @skipIf(oslist=no_match(["linux"]))
-    @skipIf(debug_info=no_match(["dwarf"]))
     def test(self):
         self.build()
 

lldb/packages/Python/lldbsuite/test/commands/expression/import-std-module/shared_ptr-dbg-info-content/Makefile

@@ -1,4 +1,3 @@
 USE_LIBCPP := 1
-CXXFLAGS += $(MANDATORY_CXXMODULE_BUILD_CFLAGS)
 CXX_SOURCES := main.cpp
 include Makefile.rules

lldb/packages/Python/lldbsuite/test/commands/expression/import-std-module/shared_ptr-dbg-info-content/TestSharedPtrDbgInfoContent.py

@@ -10,12 +10,8 @@ class TestSharedPtrDbgInfoContent(TestBase):
 
     mydir = TestBase.compute_mydir(__file__)
 
-    # FIXME: This should work on more setups, so remove these
-    # skipIf's in the future.
     @add_test_categories(["libc++"])
     @skipIf(compiler=no_match("clang"))
-    @skipIf(oslist=no_match(["linux"]))
-    @skipIf(debug_info=no_match(["dwarf"]))
     def test(self):
         self.build()
 

lldb/packages/Python/lldbsuite/test/commands/expression/import-std-module/shared_ptr/Makefile

@@ -1,4 +1,3 @@
 USE_LIBCPP := 1
-CXXFLAGS += $(MANDATORY_CXXMODULE_BUILD_CFLAGS)
 CXX_SOURCES := main.cpp
 include Makefile.rules

lldb/packages/Python/lldbsuite/test/commands/expression/import-std-module/shared_ptr/TestSharedPtr.py

@@ -10,12 +10,8 @@ class TestSharedPtr(TestBase):
 
     mydir = TestBase.compute_mydir(__file__)
 
-    # FIXME: This should work on more setups, so remove these
-    # skipIf's in the future.
     @add_test_categories(["libc++"])
     @skipIf(compiler=no_match("clang"))
-    @skipIf(oslist=no_match(["linux"]))
-    @skipIf(debug_info=no_match(["dwarf"]))
     def test(self):
         self.build()
 

lldb/packages/Python/lldbsuite/test/commands/expression/import-std-module/stack/Makefile

@@ -1,4 +1,3 @@
 USE_LIBCPP := 1
-CXXFLAGS += $(MANDATORY_CXXMODULE_BUILD_CFLAGS)
 CXX_SOURCES := main.cpp
 include Makefile.rules

lldb/packages/Python/lldbsuite/test/commands/expression/import-std-module/stack/TestStack.py

@@ -10,12 +10,8 @@ class TestStack(TestBase):
 
     mydir = TestBase.compute_mydir(__file__)
 
-    # FIXME: This should work on more setups, so remove these
-    # skipIf's in the future.
     @add_test_categories(["libc++"])
     @skipIf(compiler=no_match("clang"))
-    @skipIf(oslist=no_match(["linux"]))
-    @skipIf(debug_info=no_match(["dwarf"]))
     def test(self):
         self.build()
 

lldb/packages/Python/lldbsuite/test/commands/expression/import-std-module/sysroot/Makefile

@@ -3,7 +3,6 @@
 # system headers.
 NO_TEST_COMMON_H := 1
 
-CXXFLAGS += $(MANDATORY_CXXMODULE_BUILD_CFLAGS)
 CXXFLAGS += -I $(SRCDIR)/root/usr/include/c++/include/ -I $(SRCDIR)/root/usr/include/ -nostdinc -nostdinc++ -nostdlib++
 CXX_SOURCES := main.cpp
 include Makefile.rules

lldb/packages/Python/lldbsuite/test/commands/expression/import-std-module/sysroot/TestStdModuleSysroot.py

@@ -11,11 +11,7 @@ class ImportStdModule(TestBase):
 
     mydir = TestBase.compute_mydir(__file__)
 
-    # FIXME: This should work on more setups, so remove these
-    # skipIf's in the future.
     @skipIf(compiler=no_match("clang"))
-    @skipIf(oslist=no_match(["linux"]))
-    @skipIf(debug_info=no_match(["dwarf"]))
     def test(self):
         self.build()
 

lldb/packages/Python/lldbsuite/test/commands/expression/import-std-module/unique_ptr-dbg-info-content/Makefile

@@ -1,4 +1,3 @@
 USE_LIBCPP := 1
-CXXFLAGS += $(MANDATORY_CXXMODULE_BUILD_CFLAGS)
 CXX_SOURCES := main.cpp
 include Makefile.rules

lldb/packages/Python/lldbsuite/test/commands/expression/import-std-module/unique_ptr-dbg-info-content/TestUniquePtrDbgInfoContent.py

@@ -10,12 +10,8 @@ class TestUniquePtrDbgInfoContent(TestBase):
 
     mydir = TestBase.compute_mydir(__file__)
 
-    # FIXME: This should work on more setups, so remove these
-    # skipIf's in the future.
     @add_test_categories(["libc++"])
     @skipIf(compiler=no_match("clang"))
-    @skipIf(oslist=no_match(["linux"]))
-    @skipIf(debug_info=no_match(["dwarf"]))
     def test(self):
         self.build()
 

lldb/packages/Python/lldbsuite/test/commands/expression/import-std-module/unique_ptr/Makefile

@@ -1,4 +1,3 @@
 USE_LIBCPP := 1
-CXXFLAGS += $(MANDATORY_CXXMODULE_BUILD_CFLAGS)
 CXX_SOURCES := main.cpp
 include Makefile.rules

lldb/packages/Python/lldbsuite/test/commands/expression/import-std-module/unique_ptr/TestUniquePtr.py

@@ -10,12 +10,8 @@ class TestUniquePtr(TestBase):
 
     mydir = TestBase.compute_mydir(__file__)
 
-    # FIXME: This should work on more setups, so remove these
-    # skipIf's in the future.
     @add_test_categories(["libc++"])
     @skipIf(compiler=no_match("clang"))
-    @skipIf(oslist=no_match(["linux"]))
-    @skipIf(debug_info=no_match(["dwarf"]))
     def test(self):
         self.build()
 

lldb/packages/Python/lldbsuite/test/commands/expression/import-std-module/vector-basic/Makefile

@@ -1,4 +1,3 @@
 USE_LIBCPP := 1
-CXXFLAGS += $(MANDATORY_CXXMODULE_BUILD_CFLAGS)
 CXX_SOURCES := main.cpp
 include Makefile.rules

lldb/packages/Python/lldbsuite/test/commands/expression/import-std-module/vector-basic/TestBasicVector.py

@@ -10,12 +10,8 @@ class TestBasicVector(TestBase):
 
     mydir = TestBase.compute_mydir(__file__)
 
-    # FIXME: This should work on more setups, so remove these
-    # skipIf's in the future.
     @add_test_categories(["libc++"])
     @skipIf(compiler=no_match("clang"))
-    @skipIf(oslist=no_match(["linux"]))
-    @skipIf(debug_info=no_match(["dwarf"]))
     def test(self):
         self.build()
 

lldb/packages/Python/lldbsuite/test/commands/expression/import-std-module/vector-bool/Makefile

@@ -1,4 +1,3 @@
 USE_LIBCPP := 1
-CXXFLAGS += $(MANDATORY_CXXMODULE_BUILD_CFLAGS)
 CXX_SOURCES := main.cpp
 include Makefile.rules

lldb/packages/Python/lldbsuite/test/commands/expression/import-std-module/vector-bool/TestBoolVector.py

@@ -10,12 +10,8 @@ class TestBoolVector(TestBase):
 
     mydir = TestBase.compute_mydir(__file__)
 
-    # FIXME: This should work on more setups, so remove these
-    # skipIf's in the future.
     @add_test_categories(["libc++"])
     @skipIf(compiler=no_match("clang"))
-    @skipIf(oslist=no_match(["linux"]))
-    @skipIf(debug_info=no_match(["dwarf"]))
     def test(self):
         self.build()
 

lldb/packages/Python/lldbsuite/test/commands/expression/import-std-module/vector-dbg-info-content/Makefile

@@ -1,4 +1,3 @@
 USE_LIBCPP := 1
-CXXFLAGS += $(MANDATORY_CXXMODULE_BUILD_CFLAGS)
 CXX_SOURCES := main.cpp
 include Makefile.rules