EarlyCSE: fix CmpPredicate duplicate-hashing

[artagnon]

Strip hash_value() for CmpPredicate, as different callers have different hashing use-cases. In this case, there is just one caller, namely EarlyCSE, which calls hash_combine() on a CmpPredicate, which used to call hash_combine() on a CmpInst::Predicate prior to 4a0d53a (PatternMatch: migrate to CmpPredicate). This has uncovered a bug where two icmp instructions differing in just the fact that one of them has the samesign flag on it are hashed differently, leading to divergent hashing, and a crash. Fix this crash by dropping samesign information on icmp instructions before hashing them, preserving the former behavior.

Fixes #119893.

[llvmbot]

@llvm/pr-subscribers-llvm-transforms

@llvm/pr-subscribers-llvm-ir

Author: Ramkumar Ramachandra (artagnon)

Changes

Fixes #119893.

---

Full diff: https://github.com/llvm/llvm-project/pull/119902.diff

3 Files Affected:

• (modified) llvm/include/llvm/IR/CmpPredicate.h (-6)
• (modified) llvm/lib/IR/Instructions.cpp (-4)
• (modified) llvm/lib/Transforms/Scalar/EarlyCSE.cpp (+2-1)

diff --git a/llvm/include/llvm/IR/CmpPredicate.h b/llvm/include/llvm/IR/CmpPredicate.h
index ce78e4311f9f82..9aa1449465f5ff 100644
--- a/llvm/include/llvm/IR/CmpPredicate.h
+++ b/llvm/include/llvm/IR/CmpPredicate.h
@@ -71,13 +71,7 @@ class CmpPredicate {
 
   /// Get the swapped predicate of a CmpInst.
   static CmpPredicate getSwapped(const CmpInst *Cmp);
-
-  /// Provided to facilitate storing a CmpPredicate in data structures that
-  /// require hashing.
-  friend hash_code hash_value(const CmpPredicate &Arg); // NOLINT
 };
-
-[[nodiscard]] hash_code hash_value(const CmpPredicate &Arg);
 } // namespace llvm
 
 #endif
diff --git a/llvm/lib/IR/Instructions.cpp b/llvm/lib/IR/Instructions.cpp
index d1da02c744f18c..2d6fe40f4c1de0 100644
--- a/llvm/lib/IR/Instructions.cpp
+++ b/llvm/lib/IR/Instructions.cpp
@@ -3946,10 +3946,6 @@ CmpPredicate CmpPredicate::getSwapped(const CmpInst *Cmp) {
   return getSwapped(get(Cmp));
 }
 
-hash_code llvm::hash_value(const CmpPredicate &Arg) { // NOLINT
-  return hash_combine(Arg.Pred, Arg.HasSameSign);
-}
-
 //===----------------------------------------------------------------------===//
 //                        SwitchInst Implementation
 //===----------------------------------------------------------------------===//
diff --git a/llvm/lib/Transforms/Scalar/EarlyCSE.cpp b/llvm/lib/Transforms/Scalar/EarlyCSE.cpp
index 682c5c3d8c6340..3a0ae6b01a1144 100644
--- a/llvm/lib/Transforms/Scalar/EarlyCSE.cpp
+++ b/llvm/lib/Transforms/Scalar/EarlyCSE.cpp
@@ -290,7 +290,8 @@ static unsigned getHashValueImpl(SimpleValue Val) {
       Pred = CmpInst::getInversePredicate(Pred);
       std::swap(A, B);
     }
-    return hash_combine(Inst->getOpcode(), Pred, X, Y, A, B);
+    return hash_combine(Inst->getOpcode(),
+                        static_cast<CmpInst::Predicate>(Pred), X, Y, A, B);
   }
 
   if (CastInst *CI = dyn_cast<CastInst>(Inst))

[nikic]

This looks good to me, but should add a test? You can probably use -earlycse-debug-hash to produce a minimal test case.

[artagnon]

Sounds good, will work on a test if this is not super-urgent.

[nikic]

LGTM