Skip to content

Commit 5c0df5f

Browse files
kazutakahiratakazutakahirata
authored
[memprof] Add CallStackRadixTreeBuilder (#93784)
Call stacks are a huge portion of the MemProf profile, taking up 70+% of the profile file size. This patch implements a radix tree to compress call stacks, which are known to have long common prefixes. Specifically, CallStackRadixTreeBuilder, introduced in this patch, takes call stacks in the MemProf profile, sorts them in the dictionary order to maximize the common prefix between adjacent call stacks, and then encodes a radix tree into a single array that is ready for serialization. The resulting radix array is essentially a concatenation of call stack arrays, each encoded with its length followed by the payload, except that these arrays contain "instructions" like "skip 7 elements forward" to borrow common prefixes from other call stacks. This patch does not integrate with the MemProf serialization/deserialization infrastructure yet. Once integrated, the radix tree is expected to roughly halve the file size of the MemProf profile.
1 parent 210692b commit 5c0df5f

File tree

3 files changed

+373
-0
lines changed

3 files changed

+373
-0
lines changed

‎llvm/include/llvm/ProfileData/MemProf.h

Lines changed: 107 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -923,6 +923,113 @@ struct IndexedMemProfData {
923923
llvm::MapVector<CallStackId, llvm::SmallVector<FrameId>> CallStackData;
924924
};
925925

926+
// Construct a radix tree of call stacks.
927+
//
928+
// A set of call stacks might look like:
929+
//
930+
// CallStackId 1: f1 -> f2 -> f3
931+
// CallStackId 2: f1 -> f2 -> f4 -> f5
932+
// CallStackId 3: f1 -> f2 -> f4 -> f6
933+
// CallStackId 4: f7 -> f8 -> f9
934+
//
935+
// where each fn refers to a stack frame.
936+
//
937+
// Since we expect a lot of common prefixes, we can compress the call stacks
938+
// into a radix tree like:
939+
//
940+
// CallStackId 1: f1 -> f2 -> f3
941+
// |
942+
// CallStackId 2: +---> f4 -> f5
943+
// |
944+
// CallStackId 3: +---> f6
945+
//
946+
// CallStackId 4: f7 -> f8 -> f9
947+
//
948+
// Now, we are interested in retrieving call stacks for a given CallStackId, so
949+
// we just need a pointer from a given call stack to its parent. For example,
950+
// CallStackId 2 would point to CallStackId 1 as a parent.
951+
//
952+
// We serialize the radix tree above into a single array along with the length
953+
// of each call stack and pointers to the parent call stacks.
954+
//
955+
// Index: 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14
956+
// Array: L3 f9 f8 f7 L4 f6 J3 L4 f5 f4 J3 L3 f3 f2 f1
957+
// ^ ^ ^ ^
958+
// | | | |
959+
// CallStackId 4: 0 --+ | | |
960+
// CallStackId 3: 4 --------------+ | |
961+
// CallStackId 2: 7 -----------------------+ |
962+
// CallStackId 1: 11 -----------------------------------+
963+
//
964+
// - LN indicates the length of a call stack, encoded as ordinary integer N.
965+
//
966+
// - JN indicates a pointer to the parent, encoded as -N.
967+
//
968+
// The radix tree allows us to reconstruct call stacks in the leaf-to-root
969+
// order as we scan the array from left ro right while following pointers to
970+
// parents along the way
971+
//
972+
// For example, if we are decoding CallStackId 2, we start a forward traversal
973+
// at Index 7, noting the call stack length of 4 and obtaining f5 and f4. When
974+
// we see J3 at Index 10, we resume a forward traversal at Index 13 = 10 + 3,
975+
// picking up f2 and f1. We are done after collecting 4 frames as indicated at
976+
// the beginning of the traversal.
977+
//
978+
// On-disk IndexedMemProfRecord will refer to call stacks by their indexes into
979+
// the radix tree array, so we do not explicitly encode mappings like:
980+
// "CallStackId 1 -> 11".
981+
class CallStackRadixTreeBuilder {
982+
// The radix tree array.
983+
std::vector<LinearFrameId> RadixArray;
984+
985+
// Mapping from CallStackIds to indexes into RadixArray.
986+
llvm::DenseMap<CallStackId, LinearCallStackId> CallStackPos;
987+
988+
// In build, we partition a given call stack into two parts -- the prefix
989+
// that's common with the previously encoded call stack and the frames beyond
990+
// the common prefix -- the unique portion. Then we want to find out where
991+
// the common prefix is stored in RadixArray so that we can link the unique
992+
// portion to the common prefix. Indexes, declared below, helps with our
993+
// needs. Intuitively, Indexes tells us where each of the previously encoded
994+
// call stack is stored in RadixArray. More formally, Indexes satisfies:
995+
//
996+
// RadixArray[Indexes[I]] == Prev[I]
997+
//
998+
// for every I, where Prev is the the call stack in the root-to-leaf order
999+
// previously encoded by build. (Note that Prev, as passed to
1000+
// encodeCallStack, is in the leaf-to-root order.)
1001+
//
1002+
// For example, if the call stack being encoded shares 5 frames at the root of
1003+
// the call stack with the previously encoded call stack,
1004+
// RadixArray[Indexes[0]] is the root frame of the common prefix.
1005+
// RadixArray[Indexes[5 - 1]] is the last frame of the common prefix.
1006+
std::vector<LinearCallStackId> Indexes;
1007+
1008+
using CSIdPair = std::pair<CallStackId, llvm::SmallVector<FrameId>>;
1009+
1010+
// Encode a call stack into RadixArray. Return the starting index within
1011+
// RadixArray.
1012+
LinearCallStackId encodeCallStack(
1013+
const llvm::SmallVector<FrameId> *CallStack,
1014+
const llvm::SmallVector<FrameId> *Prev,
1015+
const llvm::DenseMap<FrameId, LinearFrameId> &MemProfFrameIndexes);
1016+
1017+
public:
1018+
CallStackRadixTreeBuilder() = default;
1019+
1020+
// Build a radix tree array.
1021+
void build(llvm::MapVector<CallStackId, llvm::SmallVector<FrameId>>
1022+
&&MemProfCallStackData,
1023+
const llvm::DenseMap<FrameId, LinearFrameId> &MemProfFrameIndexes);
1024+
1025+
const std::vector<LinearFrameId> &getRadixArray() const { return RadixArray; }
1026+
1027+
const llvm::DenseMap<CallStackId, LinearCallStackId> &
1028+
getCallStackPos() const {
1029+
return CallStackPos;
1030+
}
1031+
};
1032+
9261033
// Verify that each CallStackId is computed with hashCallStack. This function
9271034
// is intended to help transition from CallStack to CSId in
9281035
// IndexedAllocationInfo.

‎llvm/lib/ProfileData/MemProf.cpp

Lines changed: 159 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -409,6 +409,165 @@ CallStackId hashCallStack(ArrayRef<FrameId> CS) {
409409
return CSId;
410410
}
411411

412+
// Encode a call stack into RadixArray. Return the starting index within
413+
// RadixArray. For each call stack we encode, we emit two or three components
414+
// into RadixArray. If a given call stack doesn't have a common prefix relative
415+
// to the previous one, we emit:
416+
//
417+
// - the frames in the given call stack in the root-to-leaf order
418+
//
419+
// - the length of the given call stack
420+
//
421+
// If a given call stack has a non-empty common prefix relative to the previous
422+
// one, we emit:
423+
//
424+
// - the relative location of the common prefix, encoded as a negative number.
425+
//
426+
// - a portion of the given call stack that's beyond the common prefix
427+
//
428+
// - the length of the given call stack, including the length of the common
429+
// prefix.
430+
//
431+
// The resulting RadixArray requires a somewhat unintuitive backward traversal
432+
// to reconstruct a call stack -- read the call stack length and scan backward
433+
// while collecting frames in the leaf to root order. build, the caller of this
434+
// function, reverses RadixArray in place so that we can reconstruct a call
435+
// stack as if we were deserializing an array in a typical way -- the call stack
436+
// length followed by the frames in the leaf-to-root order except that we need
437+
// to handle pointers to parents along the way.
438+
//
439+
// To quickly determine the location of the common prefix within RadixArray,
440+
// Indexes caches the indexes of the previous call stack's frames within
441+
// RadixArray.
442+
LinearCallStackId CallStackRadixTreeBuilder::encodeCallStack(
443+
const llvm::SmallVector<FrameId> *CallStack,
444+
const llvm::SmallVector<FrameId> *Prev,
445+
const llvm::DenseMap<FrameId, LinearFrameId> &MemProfFrameIndexes) {
446+
// Compute the length of the common root prefix between Prev and CallStack.
447+
uint32_t CommonLen = 0;
448+
if (Prev) {
449+
auto Pos = std::mismatch(Prev->rbegin(), Prev->rend(), CallStack->rbegin(),
450+
CallStack->rend());
451+
CommonLen = std::distance(CallStack->rbegin(), Pos.second);
452+
}
453+
454+
// Drop the portion beyond CommonLen.
455+
assert(CommonLen <= Indexes.size());
456+
Indexes.resize(CommonLen);
457+
458+
// Append a pointer to the parent.
459+
if (CommonLen) {
460+
uint32_t CurrentIndex = RadixArray.size();
461+
uint32_t ParentIndex = Indexes.back();
462+
// The offset to the parent must be negative because we are pointing to an
463+
// element we've already added to RadixArray.
464+
assert(ParentIndex < CurrentIndex);
465+
RadixArray.push_back(ParentIndex - CurrentIndex);
466+
}
467+
468+
// Copy the part of the call stack beyond the common prefix to RadixArray.
469+
assert(CommonLen <= CallStack->size());
470+
for (FrameId F : llvm::drop_begin(llvm::reverse(*CallStack), CommonLen)) {
471+
// Remember the index of F in RadixArray.
472+
Indexes.push_back(RadixArray.size());
473+
RadixArray.push_back(MemProfFrameIndexes.find(F)->second);
474+
}
475+
assert(CallStack->size() == Indexes.size());
476+
477+
// End with the call stack length.
478+
RadixArray.push_back(CallStack->size());
479+
480+
// Return the index within RadixArray where we can start reconstructing a
481+
// given call stack from.
482+
return RadixArray.size() - 1;
483+
}
484+
485+
void CallStackRadixTreeBuilder::build(
486+
llvm::MapVector<CallStackId, llvm::SmallVector<FrameId>>
487+
&&MemProfCallStackData,
488+
const llvm::DenseMap<FrameId, LinearFrameId> &MemProfFrameIndexes) {
489+
// Take the vector portion of MemProfCallStackData. The vector is exactly
490+
// what we need to sort. Also, we no longer need its lookup capability.
491+
llvm::SmallVector<CSIdPair, 0> CallStacks = MemProfCallStackData.takeVector();
492+
493+
// Return early if we have no work to do.
494+
if (CallStacks.empty()) {
495+
RadixArray.clear();
496+
CallStackPos.clear();
497+
return;
498+
}
499+
500+
// Sort the list of call stacks in the dictionary order to maximize the length
501+
// of the common prefix between two adjacent call stacks.
502+
llvm::sort(CallStacks, [&](const CSIdPair &L, const CSIdPair &R) {
503+
// Call stacks are stored from leaf to root. Perform comparisons from the
504+
// root.
505+
return std::lexicographical_compare(
506+
L.second.rbegin(), L.second.rend(), R.second.rbegin(), R.second.rend(),
507+
[&](FrameId F1, FrameId F2) { return F1 < F2; });
508+
});
509+
510+
// Reserve some reasonable amount of storage.
511+
RadixArray.clear();
512+
RadixArray.reserve(CallStacks.size() * 8);
513+
514+
// Indexes will grow as long as the longest call stack.
515+
Indexes.clear();
516+
Indexes.reserve(512);
517+
518+
// CallStackPos will grow to exactly CallStacks.size() entries.
519+
CallStackPos.clear();
520+
CallStackPos.reserve(CallStacks.size());
521+
522+
// Compute the radix array. We encode one call stack at a time, computing the
523+
// longest prefix that's shared with the previous call stack we encode. For
524+
// each call stack we encode, we remember a mapping from CallStackId to its
525+
// position within RadixArray.
526+
//
527+
// As an optimization, we encode from the last call stack in CallStacks to
528+
// reduce the number of times we follow pointers to the parents. Consider the
529+
// list of call stacks that has been sorted in the dictionary order:
530+
//
531+
// Call Stack 1: F1
532+
// Call Stack 2: F1 -> F2
533+
// Call Stack 3: F1 -> F2 -> F3
534+
//
535+
// If we traversed CallStacks in the forward order, we would end up with a
536+
// radix tree like:
537+
//
538+
// Call Stack 1: F1
539+
// |
540+
// Call Stack 2: +---> F2
541+
// |
542+
// Call Stack 3: +---> F3
543+
//
544+
// Notice that each call stack jumps to the previous one. However, if we
545+
// traverse CallStacks in the reverse order, then Call Stack 3 has the
546+
// complete call stack encoded without any pointers. Call Stack 1 and 2 point
547+
// to appropriate prefixes of Call Stack 3.
548+
const llvm::SmallVector<FrameId> *Prev = nullptr;
549+
for (const auto &[CSId, CallStack] : llvm::reverse(CallStacks)) {
550+
LinearCallStackId Pos =
551+
encodeCallStack(&CallStack, Prev, MemProfFrameIndexes);
552+
CallStackPos.insert({CSId, Pos});
553+
Prev = &CallStack;
554+
}
555+
556+
// "RadixArray.size() - 1" below is problematic if RadixArray is empty.
557+
assert(!RadixArray.empty());
558+
559+
// Reverse the radix array in place. We do so mostly for intuitive
560+
// deserialization where we would read the length field and then the call
561+
// stack frames proper just like any other array deserialization, except
562+
// that we have occasional jumps to take advantage of prefixes.
563+
for (size_t I = 0, J = RadixArray.size() - 1; I < J; ++I, --J)
564+
std::swap(RadixArray[I], RadixArray[J]);
565+
566+
// "Reverse" the indexes stored in CallStackPos.
567+
for (auto &[K, V] : CallStackPos)
568+
V = RadixArray.size() - 1 - V;
569+
}
570+
412571
void verifyIndexedMemProfRecord(const IndexedMemProfRecord &Record) {
413572
for (const auto &AS : Record.AllocSites) {
414573
assert(AS.CSId == hashCallStack(AS.CallStack));

‎llvm/unittests/ProfileData/MemProfTest.cpp

Lines changed: 107 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -662,4 +662,111 @@ TEST(MemProf, MissingFrameId) {
662662
ASSERT_TRUE(FrameIdConv.LastUnmappedId.has_value());
663663
EXPECT_EQ(*FrameIdConv.LastUnmappedId, 3U);
664664
}
665+
666+
// Verify CallStackRadixTreeBuilder can handle empty inputs.
667+
TEST(MemProf, RadixTreeBuilderEmpty) {
668+
llvm::DenseMap<FrameId, llvm::memprof::LinearFrameId> MemProfFrameIndexes;
669+
llvm::MapVector<CallStackId, llvm::SmallVector<FrameId>> MemProfCallStackData;
670+
llvm::memprof::CallStackRadixTreeBuilder Builder;
671+
Builder.build(std::move(MemProfCallStackData), MemProfFrameIndexes);
672+
ASSERT_THAT(Builder.getRadixArray(), testing::IsEmpty());
673+
const auto &Mappings = Builder.getCallStackPos();
674+
ASSERT_THAT(Mappings, testing::IsEmpty());
675+
}
676+
677+
// Verify CallStackRadixTreeBuilder can handle one trivial call stack.
678+
TEST(MemProf, RadixTreeBuilderOne) {
679+
llvm::DenseMap<FrameId, llvm::memprof::LinearFrameId> MemProfFrameIndexes = {
680+
{11, 1}, {12, 2}, {13, 3}};
681+
llvm::SmallVector<llvm::memprof::FrameId> CS1 = {13, 12, 11};
682+
llvm::MapVector<CallStackId, llvm::SmallVector<FrameId>> MemProfCallStackData;
683+
MemProfCallStackData.insert({llvm::memprof::hashCallStack(CS1), CS1});
684+
llvm::memprof::CallStackRadixTreeBuilder Builder;
685+
Builder.build(std::move(MemProfCallStackData), MemProfFrameIndexes);
686+
EXPECT_THAT(Builder.getRadixArray(), testing::ElementsAreArray({
687+
3U, // Size of CS1,
688+
3U, // MemProfFrameIndexes[13]
689+
2U, // MemProfFrameIndexes[12]
690+
1U // MemProfFrameIndexes[11]
691+
}));
692+
const auto &Mappings = Builder.getCallStackPos();
693+
ASSERT_THAT(Mappings, SizeIs(1));
694+
EXPECT_THAT(Mappings, testing::Contains(testing::Pair(
695+
llvm::memprof::hashCallStack(CS1), 0U)));
696+
}
697+
698+
// Verify CallStackRadixTreeBuilder can form a link between two call stacks.
699+
TEST(MemProf, RadixTreeBuilderTwo) {
700+
llvm::DenseMap<FrameId, llvm::memprof::LinearFrameId> MemProfFrameIndexes = {
701+
{11, 1}, {12, 2}, {13, 3}};
702+
llvm::SmallVector<llvm::memprof::FrameId> CS1 = {12, 11};
703+
llvm::SmallVector<llvm::memprof::FrameId> CS2 = {13, 12, 11};
704+
llvm::MapVector<CallStackId, llvm::SmallVector<FrameId>> MemProfCallStackData;
705+
MemProfCallStackData.insert({llvm::memprof::hashCallStack(CS1), CS1});
706+
MemProfCallStackData.insert({llvm::memprof::hashCallStack(CS2), CS2});
707+
llvm::memprof::CallStackRadixTreeBuilder Builder;
708+
Builder.build(std::move(MemProfCallStackData), MemProfFrameIndexes);
709+
EXPECT_THAT(Builder.getRadixArray(),
710+
testing::ElementsAreArray({
711+
2U, // Size of CS1
712+
static_cast<uint32_t>(-3), // Jump 3 steps
713+
3U, // Size of CS2
714+
3U, // MemProfFrameIndexes[13]
715+
2U, // MemProfFrameIndexes[12]
716+
1U // MemProfFrameIndexes[11]
717+
}));
718+
const auto &Mappings = Builder.getCallStackPos();
719+
ASSERT_THAT(Mappings, SizeIs(2));
720+
EXPECT_THAT(Mappings, testing::Contains(testing::Pair(
721+
llvm::memprof::hashCallStack(CS1), 0U)));
722+
EXPECT_THAT(Mappings, testing::Contains(testing::Pair(
723+
llvm::memprof::hashCallStack(CS2), 2U)));
724+
}
725+
726+
// Verify CallStackRadixTreeBuilder can form a jump to a prefix that itself has
727+
// another jump to another prefix.
728+
TEST(MemProf, RadixTreeBuilderSuccessiveJumps) {
729+
llvm::DenseMap<FrameId, llvm::memprof::LinearFrameId> MemProfFrameIndexes = {
730+
{11, 1}, {12, 2}, {13, 3}, {14, 4}, {15, 5}, {16, 6}, {17, 7}, {18, 8},
731+
};
732+
llvm::SmallVector<llvm::memprof::FrameId> CS1 = {14, 13, 12, 11};
733+
llvm::SmallVector<llvm::memprof::FrameId> CS2 = {15, 13, 12, 11};
734+
llvm::SmallVector<llvm::memprof::FrameId> CS3 = {17, 16, 12, 11};
735+
llvm::SmallVector<llvm::memprof::FrameId> CS4 = {18, 16, 12, 11};
736+
llvm::MapVector<CallStackId, llvm::SmallVector<FrameId>> MemProfCallStackData;
737+
MemProfCallStackData.insert({llvm::memprof::hashCallStack(CS1), CS1});
738+
MemProfCallStackData.insert({llvm::memprof::hashCallStack(CS2), CS2});
739+
MemProfCallStackData.insert({llvm::memprof::hashCallStack(CS3), CS3});
740+
MemProfCallStackData.insert({llvm::memprof::hashCallStack(CS4), CS4});
741+
llvm::memprof::CallStackRadixTreeBuilder Builder;
742+
Builder.build(std::move(MemProfCallStackData), MemProfFrameIndexes);
743+
EXPECT_THAT(Builder.getRadixArray(),
744+
testing::ElementsAreArray({
745+
4U, // Size of CS1
746+
4U, // MemProfFrameIndexes[14]
747+
static_cast<uint32_t>(-3), // Jump 3 steps
748+
4U, // Size of CS2
749+
5U, // MemProfFrameIndexes[15]
750+
3U, // MemProfFrameIndexes[13]
751+
static_cast<uint32_t>(-7), // Jump 7 steps
752+
4U, // Size of CS3
753+
7U, // MemProfFrameIndexes[17]
754+
static_cast<uint32_t>(-3), // Jump 3 steps
755+
4U, // Size of CS4
756+
8U, // MemProfFrameIndexes[18]
757+
6U, // MemProfFrameIndexes[16]
758+
2U, // MemProfFrameIndexes[12]
759+
1U // MemProfFrameIndexes[11]
760+
}));
761+
const auto &Mappings = Builder.getCallStackPos();
762+
ASSERT_THAT(Mappings, SizeIs(4));
763+
EXPECT_THAT(Mappings, testing::Contains(testing::Pair(
764+
llvm::memprof::hashCallStack(CS1), 0U)));
765+
EXPECT_THAT(Mappings, testing::Contains(testing::Pair(
766+
llvm::memprof::hashCallStack(CS2), 3U)));
767+
EXPECT_THAT(Mappings, testing::Contains(testing::Pair(
768+
llvm::memprof::hashCallStack(CS3), 7U)));
769+
EXPECT_THAT(Mappings, testing::Contains(testing::Pair(
770+
llvm::memprof::hashCallStack(CS4), 10U)));
771+
}
665772
} // namespace

0 commit comments

Comments
 (0)