Reland "[llvm-cov] Support multi-source object files for convert-for-testing"

`llvm-cov convert-for-testing` only functions properly when the input binary contains a single source file. When the binary has multiple source files, a `Malformed coverage data` error will occur when the generated .covmapping is read back. This is because the testing format lacks support for indicating the size of its file records, and current implementation just assumes there's only one record in it. This patch fixes this problem by introducing a new testing format version.

Changes to the code:

- Add a new format version. The version number is stored in the the last 8 bytes of the orignial magic number field to be backward-compatible.
- Output a LEB128 number before the file records section to indicate its size in the new version.
- Change the format parsing code correspondingly.
- Update the document to formalize the testing format.
- Additionally, fix the bug when converting COFF binaries.

Reviewed By: phosek, gulfem

Differential Revision: https://reviews.llvm.org/D156611

Author: yhgu2000

llvm/docs/CommandGuide/llvm-cov.rst

@@ -543,3 +543,25 @@ OPTIONS
 
  Fail if an object file cannot be found for a binary ID present in the profile,
  neither on the command line nor via binary ID lookup.
+
+CONVERT-FOR-TESTING COMMAND
+---------------------------
+
+.. warning::
+  This command is for the LLVM developers who are working on ``llvm-cov`` only.
+
+SYNOPSIS
+^^^^^^^^
+
+:program:`llvm-cov convert-for-testing` *BIN* -o *OUT*
+
+DESCRIPTION
+^^^^^^^^^^^
+
+The :program:`llvm-cov convert-for-testing` command serves the purpose of
+testing `llvm-cov` itself. It can extract all code coverage data from the
+binary *BIN* to the file *OUT*, thereby reducing the size of test files. The
+output file typically bears the :program:`.covmapping` extension.
+
+The :program:`.covmapping` files can be read back by ``llvm-cov`` just as
+ordinary binary files.

llvm/docs/CoverageMappingFormat.rst

@@ -318,15 +318,15 @@ In version 2, the function record for *foo* was defined as follows:
 Coverage Mapping Header:
 ------------------------
 
-The coverage mapping header has the following fields:
+As shown above, the coverage mapping header has the following fields:
 
 * The number of function records affixed to the coverage header. Always 0, but present for backwards compatibility.
 
 * The length of the string in the third field of *__llvm_coverage_mapping* that contains the encoded translation unit filenames.
 
 * The length of the string in the third field of *__llvm_coverage_mapping* that contains any encoded coverage mapping data affixed to the coverage header. Always 0, but present for backwards compatibility.
 
-* The format version. The current version is 4 (encoded as a 3).
+* The format version. The current version is 6 (encoded as a 5).
 
 .. _function records:
 
@@ -610,3 +610,70 @@ The source range record contains the following fields:
 * *columnEnd*: The ending column of the mapping region. If the high bit is set,
   the current mapping region is a gap area. A count for a gap area is only used
   as the line execution count if there are no other regions on a line.
+
+Testing Format
+==============
+
+.. warning::
+  This section is for the LLVM developers who are working on ``llvm-cov`` only.
+
+``llvm-cov`` uses a special file format (called ``.covmapping`` below) for
+testing purposes. This format is private and should have no use for general
+users. As a developer, you can get such files by the ``convert-for-testing``
+subcommand of ``llvm-cov``.
+
+The structure of the ``.covmapping`` files follows:
+
+``[magicNumber : u64, version : u64, profileNames, coverageMapping, coverageRecords]``
+
+Magic Number and Version
+------------------------
+
+The magic is ``0x6d766f636d766c6c``, which is the ASCII string
+``llvmcovm`` in little-endian.
+
+There are two versions for now:
+
+- Version1, encoded as ``0x6174616474736574`` (ASCII string ``testdata``).
+- Version2, encoded as 1.
+
+The only difference between Version1 and Version2 is in the encoding of the
+``coverageMapping`` fields, which is explained later.
+
+Profile Names
+------------
+
+``profileNames``, ``coverageMapping`` and ``coverageRecords`` are 3 sections
+extracted from the original binary file.
+
+``profileNames`` encodes the size, address and the raw data of the section:
+
+``[profileNamesSize : LEB128, profileNamesAddr : LEB128, profileNamesData : bytes]``
+
+Coverage Mapping
+---------------
+
+This field is padded with zero bytes to make it 8-byte aligned.
+
+``coverageMapping`` contains the records of the source files. In version 1,
+only one record is stored:
+
+``[padding : bytes, coverageMappingData : bytes]``
+
+Version 2 relaxes this restriction by encoding the size of
+``coverageMappingData`` as a LEB128 number before the data:
+
+``[coverageMappingSize : LEB128, padding : bytes, coverageMappingData : bytes]``
+
+The current version is 2.
+
+Coverage Records
+---------------
+
+This field is padded with zero bytes to make it 8-byte aligned.
+
+``coverageRecords`` is encoded as:
+
+``[padding : bytes, coverageRecordsData : bytes]``
+
+The rest data in the file is considered as the ``coverageRecordsData``.

llvm/include/llvm/ProfileData/Coverage/CoverageMapping.h

@@ -1027,6 +1027,20 @@ enum CovMapVersion {
   CurrentVersion = INSTR_PROF_COVMAP_VERSION
 };
 
+// Correspond to "llvmcovm", in little-endian.
+constexpr uint64_t TestingFormatMagic = 0x6d766f636d766c6c;
+
+enum class TestingFormatVersion : uint64_t {
+  // The first version's number corresponds to the string "testdata" in
+  // little-endian. This is for a historical reason.
+  Version1 = 0x6174616474736574,
+  // Version1 has a defect that it can't store multiple file records. Version2
+  // fix this problem by adding a new field before the file records section.
+  Version2 = 1,
+  // The current testing format version is Version2.
+  CurrentVersion = Version2
+};
+
 template <int CovMapVersion, class IntPtrT> struct CovMapTraits {
   using CovMapFuncRecordType = CovMapFunctionRecordV3;
   using NameRefType = uint64_t;

llvm/include/llvm/ProfileData/Coverage/CoverageMappingWriter.h

@@ -54,6 +54,27 @@ class CoverageMappingWriter {
   void write(raw_ostream &OS);
 };
 
+/// Writer for the coverage mapping testing format.
+class TestingFormatWriter {
+  uint64_t ProfileNamesAddr;
+  StringRef ProfileNamesData;
+  StringRef CoverageMappingData;
+  StringRef CoverageRecordsData;
+
+public:
+  TestingFormatWriter(uint64_t ProfileNamesAddr, StringRef ProfileNamesData,
+                      StringRef CoverageMappingData,
+                      StringRef CoverageRecordsData)
+      : ProfileNamesAddr(ProfileNamesAddr), ProfileNamesData(ProfileNamesData),
+        CoverageMappingData(CoverageMappingData),
+        CoverageRecordsData(CoverageRecordsData) {}
+
+  /// Encode to the given output stream.
+  void
+  write(raw_ostream &OS,
+        TestingFormatVersion Version = TestingFormatVersion::CurrentVersion);
+};
+
 } // end namespace coverage
 
 } // end namespace llvm

llvm/lib/ProfileData/Coverage/CoverageMappingReader.cpp

@@ -822,8 +822,6 @@ static Error readCoverageMappingData(
   return Error::success();
 }
 
-static const char *TestingFormatMagic = "llvmcovmtestdata";
-
 Expected<std::unique_ptr<BinaryCoverageReader>>
 BinaryCoverageReader::createCoverageReaderFromBuffer(
     StringRef Coverage, FuncRecordsStorage &&FuncRecords,
@@ -865,7 +863,16 @@ loadTestingFormat(StringRef Data, StringRef CompilationDir) {
   uint8_t BytesInAddress = 8;
   support::endianness Endian = support::endianness::little;
 
-  Data = Data.substr(StringRef(TestingFormatMagic).size());
+  // Read the magic and version.
+  Data = Data.substr(sizeof(TestingFormatMagic));
+  if (Data.size() < sizeof(uint64_t))
+    return make_error<CoverageMapError>(coveragemap_error::malformed);
+  auto TestingVersion =
+      support::endian::byte_swap<uint64_t, support::endianness::little>(
+          *reinterpret_cast<const uint64_t *>(Data.data()));
+  Data = Data.substr(sizeof(uint64_t));
+
+  // Read the ProfileNames data.
   if (Data.empty())
     return make_error<CoverageMapError>(coveragemap_error::truncated);
   unsigned N = 0;
@@ -886,41 +893,62 @@ loadTestingFormat(StringRef Data, StringRef CompilationDir) {
   if (Error E = ProfileNames.create(Data.substr(0, ProfileNamesSize), Address))
     return std::move(E);
   Data = Data.substr(ProfileNamesSize);
+
+  // In Version2, the size of CoverageMapping is stored directly.
+  uint64_t CoverageMappingSize;
+  if (TestingVersion == uint64_t(TestingFormatVersion::Version2)) {
+    N = 0;
+    CoverageMappingSize = decodeULEB128(Data.bytes_begin(), &N);
+    if (N > Data.size())
+      return make_error<CoverageMapError>(coveragemap_error::malformed);
+    Data = Data.substr(N);
+    if (CoverageMappingSize < sizeof(CovMapHeader))
+      return make_error<CoverageMapError>(coveragemap_error::malformed);
+  } else if (TestingVersion != uint64_t(TestingFormatVersion::Version1)) {
+    return make_error<CoverageMapError>(coveragemap_error::unsupported_version);
+  }
+
   // Skip the padding bytes because coverage map data has an alignment of 8.
-  size_t Pad = offsetToAlignedAddr(Data.data(), Align(8));
+  auto Pad = offsetToAlignedAddr(Data.data(), Align(8));
   if (Data.size() < Pad)
     return make_error<CoverageMapError>(coveragemap_error::malformed);
   Data = Data.substr(Pad);
   if (Data.size() < sizeof(CovMapHeader))
     return make_error<CoverageMapError>(coveragemap_error::malformed);
   auto const *CovHeader = reinterpret_cast<const CovMapHeader *>(
       Data.substr(0, sizeof(CovMapHeader)).data());
-  CovMapVersion Version =
-      (CovMapVersion)CovHeader->getVersion<support::endianness::little>();
-  StringRef CoverageMapping;
-  BinaryCoverageReader::FuncRecordsStorage CoverageRecords;
+  auto Version =
+      CovMapVersion(CovHeader->getVersion<support::endianness::little>());
+
+  // In Version1, the size of CoverageMapping is calculated.
+  if (TestingVersion == uint64_t(TestingFormatVersion::Version1)) {
+    if (Version < CovMapVersion::Version4) {
+      CoverageMappingSize = Data.size();
+    } else {
+      auto FilenamesSize =
+          CovHeader->getFilenamesSize<support::endianness::little>();
+      CoverageMappingSize = sizeof(CovMapHeader) + FilenamesSize;
+    }
+  }
+
+  auto CoverageMapping = Data.substr(0, CoverageMappingSize);
+  Data = Data.substr(CoverageMappingSize);
+
+  // Read the CoverageRecords data.
   if (Version < CovMapVersion::Version4) {
-    CoverageMapping = Data;
-    if (CoverageMapping.empty())
-      return make_error<CoverageMapError>(coveragemap_error::truncated);
-    CoverageRecords = MemoryBuffer::getMemBuffer("");
+    if (!Data.empty())
+      return make_error<CoverageMapError>(coveragemap_error::malformed);
   } else {
-    uint32_t FilenamesSize =
-        CovHeader->getFilenamesSize<support::endianness::little>();
-    uint32_t CoverageMappingSize = sizeof(CovMapHeader) + FilenamesSize;
-    CoverageMapping = Data.substr(0, CoverageMappingSize);
-    if (CoverageMapping.empty())
-      return make_error<CoverageMapError>(coveragemap_error::truncated);
-    Data = Data.substr(CoverageMappingSize);
     // Skip the padding bytes because coverage records data has an alignment
     // of 8.
     Pad = offsetToAlignedAddr(Data.data(), Align(8));
     if (Data.size() < Pad)
       return make_error<CoverageMapError>(coveragemap_error::malformed);
-    CoverageRecords = MemoryBuffer::getMemBuffer(Data.substr(Pad));
-    if (CoverageRecords->getBufferSize() == 0)
-      return make_error<CoverageMapError>(coveragemap_error::truncated);
+    Data = Data.substr(Pad);
   }
+  BinaryCoverageReader::FuncRecordsStorage CoverageRecords =
+      MemoryBuffer::getMemBuffer(Data);
+
   return BinaryCoverageReader::createCoverageReaderFromBuffer(
       CoverageMapping, std::move(CoverageRecords), std::move(ProfileNames),
       BytesInAddress, Endian, CompilationDir);
@@ -1081,14 +1109,19 @@ BinaryCoverageReader::create(
     StringRef CompilationDir, SmallVectorImpl<object::BuildIDRef> *BinaryIDs) {
   std::vector<std::unique_ptr<BinaryCoverageReader>> Readers;
 
-  if (ObjectBuffer.getBuffer().startswith(TestingFormatMagic)) {
-    // This is a special format used for testing.
-    auto ReaderOrErr =
-        loadTestingFormat(ObjectBuffer.getBuffer(), CompilationDir);
-    if (!ReaderOrErr)
-      return ReaderOrErr.takeError();
-    Readers.push_back(std::move(ReaderOrErr.get()));
-    return std::move(Readers);
+  if (ObjectBuffer.getBuffer().size() > sizeof(TestingFormatMagic)) {
+    uint64_t Magic =
+        support::endian::byte_swap<uint64_t, support::endianness::little>(
+            *reinterpret_cast<const uint64_t *>(ObjectBuffer.getBufferStart()));
+    if (Magic == TestingFormatMagic) {
+      // This is a special format used for testing.
+      auto ReaderOrErr =
+          loadTestingFormat(ObjectBuffer.getBuffer(), CompilationDir);
+      if (!ReaderOrErr)
+        return ReaderOrErr.takeError();
+      Readers.push_back(std::move(ReaderOrErr.get()));
+      return std::move(Readers);
+    }
   }
 
   auto BinOrErr = createBinary(ObjectBuffer);

llvm/lib/ProfileData/Coverage/CoverageMappingWriter.cpp

@@ -249,3 +249,37 @@ void CoverageMappingWriter::write(raw_ostream &OS) {
   // Ensure that all file ids have at least one mapping region.
   assert(CurrentFileID == (VirtualFileMapping.size() - 1));
 }
+
+void TestingFormatWriter::write(raw_ostream &OS, TestingFormatVersion Version) {
+  auto ByteSwap = [](uint64_t N) {
+    return support::endian::byte_swap<uint64_t, support::endianness::little>(N);
+  };
+
+  // Output a 64bit magic number.
+  auto Magic = ByteSwap(TestingFormatMagic);
+  OS.write(reinterpret_cast<char *>(&Magic), sizeof(Magic));
+
+  // Output a 64bit version field.
+  auto VersionLittle = ByteSwap(uint64_t(Version));
+  OS.write(reinterpret_cast<char *>(&VersionLittle), sizeof(VersionLittle));
+
+  // Output the ProfileNames data.
+  encodeULEB128(ProfileNamesData.size(), OS);
+  encodeULEB128(ProfileNamesAddr, OS);
+  OS << ProfileNamesData;
+
+  // Version2 adds an extra field to indicate the size of the
+  // CoverageMappingData.
+  if (Version == TestingFormatVersion::Version2)
+    encodeULEB128(CoverageMappingData.size(), OS);
+
+  // Coverage mapping data is expected to have an alignment of 8.
+  for (unsigned Pad = offsetToAlignment(OS.tell(), Align(8)); Pad; --Pad)
+    OS.write(uint8_t(0));
+  OS << CoverageMappingData;
+
+  // Coverage records data is expected to have an alignment of 8.
+  for (unsigned Pad = offsetToAlignment(OS.tell(), Align(8)); Pad; --Pad)
+    OS.write(uint8_t(0));
+  OS << CoverageRecordsData;
+}