[pdb] Provide a better error message when overflowing the public/global symbol record stream #140884

zmodem · 2025-05-21T12:03:02Z

Before:

lld-link: error: Stream Error: The stream is too short to perform the requested operation.
lld-link: error: failed to write PDB file ./unit_tests.exe.pdb

After:

lld-link: error: the public (2127832912 bytes) and global (2200532960 bytes) symbols
are too large to fit in a PDB file; the maximum total is 4294967295 bytes.
lld-link: error: failed to write PDB file ./unit_tests.exe.pdb

…al symbol record stream Before: lld-link: error: Stream Error: The stream is too short to perform the requested operation. lld-link: error: failed to write PDB file ./unit_tests.exe.pdb After: lld-link: error: the public (2127832912 bytes) and global (2200532960 bytes) symbols are too large to fit in a PDB file. lld-link: error: failed to write PDB file ./unit_tests.exe.pdb

llvmbot · 2025-05-21T12:03:40Z

@llvm/pr-subscribers-platform-windows

Author: Hans Wennborg (zmodem)

Changes

Before:

lld-link: error: Stream Error: The stream is too short to perform the requested operation.
lld-link: error: failed to write PDB file ./unit_tests.exe.pdb

After:

lld-link: error: the public (2127832912 bytes) and global (2200532960 bytes) symbols are too large to fit in a PDB file.
lld-link: error: failed to write PDB file ./unit_tests.exe.pdb

Full diff: https://github.com/llvm/llvm-project/pull/140884.diff

1 Files Affected:

(modified) llvm/lib/DebugInfo/PDB/Native/GSIStreamBuilder.cpp (+7-2)

diff --git a/llvm/lib/DebugInfo/PDB/Native/GSIStreamBuilder.cpp b/llvm/lib/DebugInfo/PDB/Native/GSIStreamBuilder.cpp
index c195754c0c679..0289a7543205a 100644
--- a/llvm/lib/DebugInfo/PDB/Native/GSIStreamBuilder.cpp
+++ b/llvm/lib/DebugInfo/PDB/Native/GSIStreamBuilder.cpp
@@ -25,6 +25,7 @@
 #include "llvm/DebugInfo/PDB/Native/RawTypes.h"
 #include "llvm/Support/BinaryItemStream.h"
 #include "llvm/Support/BinaryStreamWriter.h"
+#include "llvm/Support/FormatVariadic.h"
 #include "llvm/Support/Parallel.h"
 #include "llvm/Support/TimeProfiler.h"
 #include "llvm/Support/xxhash.h"
@@ -39,7 +40,7 @@ using namespace llvm::codeview;
 // Helper class for building the public and global PDB hash table buckets.
 struct llvm::pdb::GSIHashStreamBuilder {
   // Sum of the size of all public or global records.
-  uint32_t RecordByteSize = 0;
+  uint64_t RecordByteSize = 0;
 
   std::vector<PSHashRecord> HashRecords;
 
@@ -319,7 +320,11 @@ Error GSIStreamBuilder::finalizeMsfLayout() {
     return Idx.takeError();
   PublicsStreamIndex = *Idx;
 
-  uint32_t RecordBytes = PSH->RecordByteSize + GSH->RecordByteSize;
+  uint64_t RecordBytes = PSH->RecordByteSize + GSH->RecordByteSize;
+  if (RecordBytes > UINT32_MAX)
+    return make_error<StringError>(formatv("the public ({0} bytes) and global "
+          "({1} bytes) symbols are too large to fit in a PDB file.",
+          PSH->RecordByteSize, GSH->RecordByteSize), inconvertibleErrorCode());
 
   Idx = Msf.addStream(RecordBytes);
   if (!Idx)

llvmbot · 2025-05-21T12:03:40Z

@llvm/pr-subscribers-debuginfo

Author: Hans Wennborg (zmodem)

Changes

Before:

lld-link: error: Stream Error: The stream is too short to perform the requested operation.
lld-link: error: failed to write PDB file ./unit_tests.exe.pdb

After:

lld-link: error: the public (2127832912 bytes) and global (2200532960 bytes) symbols are too large to fit in a PDB file.
lld-link: error: failed to write PDB file ./unit_tests.exe.pdb

Full diff: https://github.com/llvm/llvm-project/pull/140884.diff

1 Files Affected:

(modified) llvm/lib/DebugInfo/PDB/Native/GSIStreamBuilder.cpp (+7-2)

diff --git a/llvm/lib/DebugInfo/PDB/Native/GSIStreamBuilder.cpp b/llvm/lib/DebugInfo/PDB/Native/GSIStreamBuilder.cpp
index c195754c0c679..0289a7543205a 100644
--- a/llvm/lib/DebugInfo/PDB/Native/GSIStreamBuilder.cpp
+++ b/llvm/lib/DebugInfo/PDB/Native/GSIStreamBuilder.cpp
@@ -25,6 +25,7 @@
 #include "llvm/DebugInfo/PDB/Native/RawTypes.h"
 #include "llvm/Support/BinaryItemStream.h"
 #include "llvm/Support/BinaryStreamWriter.h"
+#include "llvm/Support/FormatVariadic.h"
 #include "llvm/Support/Parallel.h"
 #include "llvm/Support/TimeProfiler.h"
 #include "llvm/Support/xxhash.h"
@@ -39,7 +40,7 @@ using namespace llvm::codeview;
 // Helper class for building the public and global PDB hash table buckets.
 struct llvm::pdb::GSIHashStreamBuilder {
   // Sum of the size of all public or global records.
-  uint32_t RecordByteSize = 0;
+  uint64_t RecordByteSize = 0;
 
   std::vector<PSHashRecord> HashRecords;
 
@@ -319,7 +320,11 @@ Error GSIStreamBuilder::finalizeMsfLayout() {
     return Idx.takeError();
   PublicsStreamIndex = *Idx;
 
-  uint32_t RecordBytes = PSH->RecordByteSize + GSH->RecordByteSize;
+  uint64_t RecordBytes = PSH->RecordByteSize + GSH->RecordByteSize;
+  if (RecordBytes > UINT32_MAX)
+    return make_error<StringError>(formatv("the public ({0} bytes) and global "
+          "({1} bytes) symbols are too large to fit in a PDB file.",
+          PSH->RecordByteSize, GSH->RecordByteSize), inconvertibleErrorCode());
 
   Idx = Msf.addStream(RecordBytes);
   if (!Idx)

github-actions · 2025-05-21T12:06:23Z

✅ With the latest revision this PR passed the C/C++ code formatter.

tru · 2025-05-21T12:46:18Z

Looks good, but I wonder if we should hint at any way to resolve the problem? I am not sure I would know what to do when this happen and end up on this Pull Request after some research :)

aganea · 2025-05-21T13:43:20Z

llvm/lib/DebugInfo/PDB/Native/GSIStreamBuilder.cpp

+  uint64_t RecordBytes = PSH->RecordByteSize + GSH->RecordByteSize;
+  if (RecordBytes > UINT32_MAX)
+    return make_error<StringError>(
+        formatv("the public ({0} bytes) and global ({1} bytes) "


Can we also display what the maximum size is? It’s obvious for us, but not for regular users.

Github shows some kind of syntax error, maybe you have local fixup changes that didn't get pushed.

Oops. Done.

zmodem · 2025-05-21T15:56:05Z

I wonder if we should hint at any way to resolve the problem? I am not sure I would know what to do when this happen and end up on this Pull Request after some research :)

I don't think there's room for much info into the error message, but we can certainly put some notes here.

This kind of overflow is likely due to a huge amount of function template instantiations which would go away if inlined. So if compiling with -O1 or higher is an option, that would be the best solution. If that's not an option, but the user controls the source code, applying force_inline attributes to common templates may be an alternative.
There are flags to reduce the amount of debug info, such as -gline-tables-only, but that will give a much more limited debugging experience.
If the above doesn't help, it may be necessary to split the binary into multiple .exe/.dll files.

rnk

Looks good, thanks!

rnk · 2025-05-21T18:26:01Z

llvm/lib/DebugInfo/PDB/Native/GSIStreamBuilder.cpp

+  uint64_t RecordBytes = PSH->RecordByteSize + GSH->RecordByteSize;
+  if (RecordBytes > UINT32_MAX)
+    return make_error<StringError>(
+        formatv("the public ({0} bytes) and global ({1} bytes) "


Github shows some kind of syntax error, maybe you have local fixup changes that didn't get pushed.

rnk · 2025-05-21T18:27:15Z

llvm/lib/DebugInfo/PDB/Native/GSIStreamBuilder.cpp

+    return make_error<StringError>(
+        formatv("the public ({0} bytes) and global ({1} bytes) "
+                "symbols are too large to fit in a PDB file; ",
+                "the maximum total is 4294967295 bytes.


nit: maybe pass UINT32_MAX as a parameter to the formatter so we don't have to hardcode the decimal value.

nico · 2025-05-22T13:40:54Z

llvm/lib/DebugInfo/PDB/Native/GSIStreamBuilder.cpp

+    return make_error<StringError>(
+        formatv("the public ({0} bytes) and global ({1} bytes) "
+                "symbols are too large to fit in a PDB file; "
+                "the maximum total is {2} bytes.",


nit: "The public and global symbols are too large" can be read as if this is about two symbols, the public symbol and the global symbol. Maybe just "The public symbols and global symbols"? "The public and global symbol tables?" (I think that suggestion, while not being ambiguous, is incorrect, but I'm not sure. If it is, we obviously shouldn't use it :P)

Sorry for the bikeshed; feel free to ignore if you're happy with this as-is.

Thanks, going with "the public symbols and global symbols" in 6bb05ea

…stream Follow-up to #140884

…als record stream Follow-up to llvm/llvm-project#140884

…al symbol record stream (llvm#140884) Before: lld-link: error: Stream Error: The stream is too short to perform the requested operation. lld-link: error: failed to write PDB file ./unit_tests.exe.pdb After: lld-link: error: the public (2127832912 bytes) and global (2200532960 bytes) symbols are too large to fit in a PDB file; the maximum total is 4294967295 bytes. lld-link: error: failed to write PDB file ./unit_tests.exe.pdb

…stream Follow-up to llvm#140884

…al symbol record stream (llvm#140884) Before: lld-link: error: Stream Error: The stream is too short to perform the requested operation. lld-link: error: failed to write PDB file ./unit_tests.exe.pdb After: lld-link: error: the public (2127832912 bytes) and global (2200532960 bytes) symbols are too large to fit in a PDB file; the maximum total is 4294967295 bytes. lld-link: error: failed to write PDB file ./unit_tests.exe.pdb

zmodem requested review from rnk and ZequanWu May 21, 2025 12:03

llvmbot added debuginfo platform:windows labels May 21, 2025

format

a16712c

aganea reviewed May 21, 2025

View reviewed changes

mention the max total

e639bc4

rnk approved these changes May 21, 2025

View reviewed changes

use UINT32_MAX, fix syntax

66aa97b

ZequanWu approved these changes May 21, 2025

View reviewed changes

zmodem merged commit 213d0d2 into llvm:main May 22, 2025
9 of 11 checks passed

nico reviewed May 22, 2025

View reviewed changes

zmodem added a commit that referenced this pull request May 23, 2025

[pdb] Tweak the message about overflowing the publics/globals record …

6bb05ea

…stream Follow-up to #140884

llvm-sync bot pushed a commit to arm/arm-toolchain that referenced this pull request May 23, 2025

Automerge: [pdb] Tweak the message about overflowing the publics/glob…

b839e16

…als record stream Follow-up to llvm/llvm-project#140884

sivan-shani pushed a commit to sivan-shani/llvm-project that referenced this pull request Jun 3, 2025

[pdb] Tweak the message about overflowing the publics/globals record …

4b18bad

…stream Follow-up to llvm#140884

[pdb] Provide a better error message when overflowing the public/global symbol record stream #140884

[pdb] Provide a better error message when overflowing the public/global symbol record stream #140884

Uh oh!

Conversation

zmodem commented May 21, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

llvmbot commented May 21, 2025

Uh oh!

llvmbot commented May 21, 2025

Uh oh!

github-actions bot commented May 21, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

tru commented May 21, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

zmodem commented May 21, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

rnk left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

zmodem commented May 21, 2025 •

edited

Loading

github-actions bot commented May 21, 2025 •

edited

Loading

zmodem commented May 21, 2025 •

edited

Loading