[llvm-exegesis] Add middle half repetition mode

[boomanaiden154]

This patch adds two new repetition modes to llvm-exegesis, particularly loop and duplicate repetition modes of what I am terming the middle half repetition mode. The middle half repetition mode essentially runs each measurement twice, one with twice the number of iterations of the other. These two measurements are then agregated by taking their difference. This subtracts away any setup/overhead that is unrelated to the code in the snippet, providing more accurate results.

Using this mode on a couple toy examples, I am able to get exact (integer) throughput values on all of them in contrast to the default duplicate/loop repetition modes which show a little bit of noise on the snippet value.

[llvmbot]

@llvm/pr-subscribers-llvm-binary-utilities

@llvm/pr-subscribers-tools-llvm-exegesis

Author: Aiden Grossman (boomanaiden154)

Changes

This patch adds two new repetition modes to llvm-exegesis, particularly loop and duplicate repetition modes of what I am terming the middle half repetition mode. The middle half repetition mode essentially runs each measurement twice, one with twice the number of iterations of the other. These two measurements are then agregated by taking their difference. This subtracts away any setup/overhead that is unrelated to the code in the snippet, providing more accurate results.

Using this mode on a couple toy examples, I am able to get exact (integer) throughput values on all of them in contrast to the default duplicate/loop repetition modes which show a little bit of noise on the snippet value.

---

Full diff: https://github.com/llvm/llvm-project/pull/77020.diff

3 Files Affected:

• (modified) llvm/tools/llvm-exegesis/lib/BenchmarkResult.h (+10-2)
• (modified) llvm/tools/llvm-exegesis/lib/SnippetRepetitor.cpp (+2)
• (modified) llvm/tools/llvm-exegesis/llvm-exegesis.cpp (+49-18)

diff --git a/llvm/tools/llvm-exegesis/lib/BenchmarkResult.h b/llvm/tools/llvm-exegesis/lib/BenchmarkResult.h
index 0d08febae20cb3..3be58fa7f6853c 100644
--- a/llvm/tools/llvm-exegesis/lib/BenchmarkResult.h
+++ b/llvm/tools/llvm-exegesis/lib/BenchmarkResult.h
@@ -78,7 +78,7 @@ struct BenchmarkKey {
 struct BenchmarkMeasure {
   // A helper to create an unscaled BenchmarkMeasure.
   static BenchmarkMeasure Create(std::string Key, double Value) {
-    return {Key, Value, Value};
+    return {Key, Value, Value, Value};
   }
   std::string Key;
   // This is the per-instruction value, i.e. measured quantity scaled per
@@ -87,6 +87,8 @@ struct BenchmarkMeasure {
   // This is the per-snippet value, i.e. measured quantity for one repetition of
   // the whole snippet.
   double PerSnippetValue;
+  // This is the raw value collected from the full execution.
+  double RawValue;
 };
 
 // The result of an instruction benchmark.
@@ -101,7 +103,13 @@ struct Benchmark {
   // The number of instructions inside the repeated snippet. For example, if a
   // snippet of 3 instructions is repeated 4 times, this is 12.
   unsigned NumRepetitions = 0;
-  enum RepetitionModeE { Duplicate, Loop, AggregateMin };
+  enum RepetitionModeE {
+    Duplicate,
+    Loop,
+    AggregateMin,
+    MiddleHalfDuplicate,
+    MiddleHalfLoop
+  };
   // Note that measurements are per instruction.
   std::vector<BenchmarkMeasure> Measurements;
   std::string Error;
diff --git a/llvm/tools/llvm-exegesis/lib/SnippetRepetitor.cpp b/llvm/tools/llvm-exegesis/lib/SnippetRepetitor.cpp
index cc5a045a8be5dd..7e19e8a9bb5211 100644
--- a/llvm/tools/llvm-exegesis/lib/SnippetRepetitor.cpp
+++ b/llvm/tools/llvm-exegesis/lib/SnippetRepetitor.cpp
@@ -135,8 +135,10 @@ SnippetRepetitor::Create(Benchmark::RepetitionModeE Mode,
                          const LLVMState &State) {
   switch (Mode) {
   case Benchmark::Duplicate:
+  case Benchmark::MiddleHalfDuplicate:
     return std::make_unique<DuplicateSnippetRepetitor>(State);
   case Benchmark::Loop:
+  case Benchmark::MiddleHalfLoop:
     return std::make_unique<LoopSnippetRepetitor>(State);
   case Benchmark::AggregateMin:
     break;
diff --git a/llvm/tools/llvm-exegesis/llvm-exegesis.cpp b/llvm/tools/llvm-exegesis/llvm-exegesis.cpp
index ffbf94ce0fcb26..b963ca0252a72e 100644
--- a/llvm/tools/llvm-exegesis/llvm-exegesis.cpp
+++ b/llvm/tools/llvm-exegesis/llvm-exegesis.cpp
@@ -106,10 +106,13 @@ static cl::opt<exegesis::Benchmark::RepetitionModeE> RepetitionMode(
     cl::values(
         clEnumValN(exegesis::Benchmark::Duplicate, "duplicate",
                    "Duplicate the snippet"),
-        clEnumValN(exegesis::Benchmark::Loop, "loop",
-                   "Loop over the snippet"),
+        clEnumValN(exegesis::Benchmark::Loop, "loop", "Loop over the snippet"),
         clEnumValN(exegesis::Benchmark::AggregateMin, "min",
-                   "All of the above and take the minimum of measurements")),
+                   "All of the above and take the minimum of measurements"),
+        clEnumValN(exegesis::Benchmark::MiddleHalfDuplicate,
+                   "middle-half-duplicate", "Middle half duplicate mode"),
+        clEnumValN(exegesis::Benchmark::MiddleHalfLoop, "middle-half-loop",
+                   "Middle half loop mode")),
     cl::init(exegesis::Benchmark::Duplicate));
 
 static cl::opt<bool> BenchmarkMeasurementsPrintProgress(
@@ -399,29 +402,37 @@ static void runBenchmarkConfigurations(
   std::optional<ProgressMeter<>> Meter;
   if (BenchmarkMeasurementsPrintProgress)
     Meter.emplace(Configurations.size());
+
+  SmallVector<unsigned, 2> MinInstructions = {NumRepetitions};
+  if (RepetitionMode == Benchmark::MiddleHalfDuplicate ||
+      RepetitionMode == Benchmark::MiddleHalfLoop)
+    MinInstructions.push_back(NumRepetitions * 2);
+
   for (const BenchmarkCode &Conf : Configurations) {
     ProgressMeter<>::ProgressMeterStep MeterStep(Meter ? &*Meter : nullptr);
     SmallVector<Benchmark, 2> AllResults;
 
     for (const std::unique_ptr<const SnippetRepetitor> &Repetitor :
          Repetitors) {
-      auto RC = ExitOnErr(Runner.getRunnableConfiguration(
-          Conf, NumRepetitions, LoopBodySize, *Repetitor));
-      std::optional<StringRef> DumpFile;
-      if (DumpObjectToDisk.getNumOccurrences())
-        DumpFile = DumpObjectToDisk;
-      auto [Err, BenchmarkResult] =
-          Runner.runConfiguration(std::move(RC), DumpFile);
-      if (Err) {
-        // Errors from executing the snippets are fine.
-        // All other errors are a framework issue and should fail.
-        if (!Err.isA<SnippetExecutionFailure>()) {
-          llvm::errs() << "llvm-exegesis error: " << toString(std::move(Err));
-          exit(1);
+      for (unsigned IterationRepetitions : MinInstructions) {
+        auto RC = ExitOnErr(Runner.getRunnableConfiguration(
+            Conf, IterationRepetitions, LoopBodySize, *Repetitor));
+        std::optional<StringRef> DumpFile;
+        if (DumpObjectToDisk.getNumOccurrences())
+          DumpFile = DumpObjectToDisk;
+        auto [Err, BenchmarkResult] =
+            Runner.runConfiguration(std::move(RC), DumpFile);
+        if (Err) {
+          // Errors from executing the snippets are fine.
+          // All other errors are a framework issue and should fail.
+          if (!Err.isA<SnippetExecutionFailure>()) {
+            llvm::errs() << "llvm-exegesis error: " << toString(std::move(Err));
+            exit(1);
+          }
+          BenchmarkResult.Error = toString(std::move(Err));
         }
-        BenchmarkResult.Error = toString(std::move(Err));
+        AllResults.push_back(std::move(BenchmarkResult));
       }
-      AllResults.push_back(std::move(BenchmarkResult));
     }
     Benchmark &Result = AllResults.front();
 
@@ -455,6 +466,26 @@ static void runBenchmarkConfigurations(
               Measurement.PerSnippetValue, NewMeasurement.PerSnippetValue);
         }
       }
+    } else if (RepetitionMode ==
+                   Benchmark::RepetitionModeE::MiddleHalfDuplicate ||
+               RepetitionMode == Benchmark::RepetitionModeE::MiddleHalfLoop) {
+      for (const Benchmark &OtherResult :
+           ArrayRef<Benchmark>(AllResults).drop_front()) {
+        if (OtherResult.Measurements.empty())
+          continue;
+        assert(OtherResult.Measurements.size() == Result.Measurements.size());
+        for (auto I : zip(Result.Measurements, OtherResult.Measurements)) {
+          BenchmarkMeasure &Measurement = std::get<0>(I);
+          const BenchmarkMeasure &NewMeasurement = std::get<1>(I);
+          Measurement.RawValue = NewMeasurement.RawValue - Measurement.RawValue;
+          Measurement.PerInstructionValue = Measurement.RawValue;
+          Measurement.PerInstructionValue /= Result.NumRepetitions;
+          Measurement.PerSnippetValue = Measurement.RawValue;
+          Measurement.PerSnippetValue *=
+              static_cast<double>(Result.Key.Instructions.size()) /
+              Result.NumRepetitions;
+        }
+      }
     }
 
     // With dummy counters, measurements are rather meaningless,

[boomanaiden154]

@legrosbuffle Can you review this patch when you get a chance? Thanks!

[legrosbuffle]

Can you please update the documentation too ?

[legrosbuffle]

The approach LGTM, only comments on the code.

[boomanaiden154]

@legrosbuffle This should be ready to again when you have a chance to take a look. Thanks!