[DAG] Reducing instructions by better legalization handling of AVGFLOORU for illegal data types #101223
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
@llvm/pr-subscribers-backend-x86 @llvm/pr-subscribers-llvm-selectiondag Author: Julius Alexandre (medievalghoul) ChangesPrevious reverted merge: #99913 Previous error generated by LLVM Bot: </details> It doesn't seem as if there's a fix for the error message. I wasn't able to reproduce this error. All my attempts to do To reiterate:https://rust.godbolt.org/z/T7eKP3Tvo x86: https://alive2.llvm.org/ce/z/ze88Hw cc: @RKSimon @topperc Full diff: https://github.com/llvm/llvm-project/pull/101223.diff 4 Files Affected:
diff --git a/llvm/lib/CodeGen/SelectionDAG/TargetLowering.cpp b/llvm/lib/CodeGen/SelectionDAG/TargetLowering.cpp
index 6fd23b5ab9f5f..7fa83a5999dfe 100644
--- a/llvm/lib/CodeGen/SelectionDAG/TargetLowering.cpp
+++ b/llvm/lib/CodeGen/SelectionDAG/TargetLowering.cpp
@@ -9379,6 +9379,26 @@ SDValue TargetLowering::expandAVG(SDNode *N, SelectionDAG &DAG) const {
}
}
+ // avgflooru(lhs, rhs) -> or(lshr(add(lhs, rhs),1),shl(overflow, typesize-1))
+ if (Opc == ISD::AVGFLOORU && VT.isScalarInteger() && !isTypeLegal(VT)) {
+ SDValue UAddWithOverflow =
+ DAG.getNode(ISD::UADDO, dl, DAG.getVTList(VT, MVT::i1), {RHS, LHS});
+
+ SDValue Sum = UAddWithOverflow.getValue(0);
+ SDValue Overflow = UAddWithOverflow.getValue(1);
+
+ // Right shift the sum by 1
+ SDValue One = DAG.getShiftAmountConstant(1, VT, dl);
+ SDValue LShrVal = DAG.getNode(ISD::SRL, dl, VT, Sum, One);
+
+ SDValue ZeroExtOverflow = DAG.getNode(ISD::ANY_EXTEND, dl, VT, Overflow);
+ SDValue OverflowShl =
+ DAG.getNode(ISD::SHL, dl, VT, ZeroExtOverflow,
+ DAG.getConstant(VT.getScalarSizeInBits() - 1, dl, VT));
+
+ return DAG.getNode(ISD::OR, dl, VT, LShrVal, OverflowShl);
+ }
+
// avgceils(lhs, rhs) -> sub(or(lhs,rhs),ashr(xor(lhs,rhs),1))
// avgceilu(lhs, rhs) -> sub(or(lhs,rhs),lshr(xor(lhs,rhs),1))
// avgfloors(lhs, rhs) -> add(and(lhs,rhs),ashr(xor(lhs,rhs),1))
diff --git a/llvm/test/CodeGen/RISCV/avgflooru.ll b/llvm/test/CodeGen/RISCV/avgflooru.ll
index b58aaab6aaf4a..fa88c3760e455 100644
--- a/llvm/test/CodeGen/RISCV/avgflooru.ll
+++ b/llvm/test/CodeGen/RISCV/avgflooru.ll
@@ -164,18 +164,20 @@ define i32 @test_ext_i32(i32 %a0, i32 %a1) nounwind {
define i64 @test_fixed_i64(i64 %a0, i64 %a1) nounwind {
; RV32I-LABEL: test_fixed_i64:
; RV32I: # %bb.0:
-; RV32I-NEXT: and a4, a1, a3
-; RV32I-NEXT: xor a1, a1, a3
-; RV32I-NEXT: srli a3, a1, 1
-; RV32I-NEXT: add a3, a4, a3
-; RV32I-NEXT: slli a1, a1, 31
-; RV32I-NEXT: xor a4, a0, a2
-; RV32I-NEXT: srli a4, a4, 1
-; RV32I-NEXT: or a1, a4, a1
-; RV32I-NEXT: and a2, a0, a2
-; RV32I-NEXT: add a0, a2, a1
+; RV32I-NEXT: add a4, a3, a1
+; RV32I-NEXT: add a0, a2, a0
; RV32I-NEXT: sltu a1, a0, a2
-; RV32I-NEXT: add a1, a3, a1
+; RV32I-NEXT: add a2, a4, a1
+; RV32I-NEXT: beq a2, a3, .LBB6_2
+; RV32I-NEXT: # %bb.1:
+; RV32I-NEXT: sltu a1, a2, a3
+; RV32I-NEXT: .LBB6_2:
+; RV32I-NEXT: slli a1, a1, 31
+; RV32I-NEXT: srli a3, a2, 1
+; RV32I-NEXT: or a1, a3, a1
+; RV32I-NEXT: slli a2, a2, 31
+; RV32I-NEXT: srli a0, a0, 1
+; RV32I-NEXT: or a0, a0, a2
; RV32I-NEXT: ret
;
; RV64I-LABEL: test_fixed_i64:
@@ -195,18 +197,20 @@ define i64 @test_fixed_i64(i64 %a0, i64 %a1) nounwind {
define i64 @test_ext_i64(i64 %a0, i64 %a1) nounwind {
; RV32I-LABEL: test_ext_i64:
; RV32I: # %bb.0:
-; RV32I-NEXT: and a4, a1, a3
-; RV32I-NEXT: xor a1, a1, a3
-; RV32I-NEXT: srli a3, a1, 1
-; RV32I-NEXT: add a3, a4, a3
-; RV32I-NEXT: slli a1, a1, 31
-; RV32I-NEXT: xor a4, a0, a2
-; RV32I-NEXT: srli a4, a4, 1
-; RV32I-NEXT: or a1, a4, a1
-; RV32I-NEXT: and a2, a0, a2
-; RV32I-NEXT: add a0, a2, a1
+; RV32I-NEXT: add a4, a3, a1
+; RV32I-NEXT: add a0, a2, a0
; RV32I-NEXT: sltu a1, a0, a2
-; RV32I-NEXT: add a1, a3, a1
+; RV32I-NEXT: add a2, a4, a1
+; RV32I-NEXT: beq a2, a3, .LBB7_2
+; RV32I-NEXT: # %bb.1:
+; RV32I-NEXT: sltu a1, a2, a3
+; RV32I-NEXT: .LBB7_2:
+; RV32I-NEXT: slli a1, a1, 31
+; RV32I-NEXT: srli a3, a2, 1
+; RV32I-NEXT: or a1, a3, a1
+; RV32I-NEXT: slli a2, a2, 31
+; RV32I-NEXT: srli a0, a0, 1
+; RV32I-NEXT: or a0, a0, a2
; RV32I-NEXT: ret
;
; RV64I-LABEL: test_ext_i64:
diff --git a/llvm/test/CodeGen/X86/avgflooru-i128.ll b/llvm/test/CodeGen/X86/avgflooru-i128.ll
new file mode 100644
index 0000000000000..da16a7da48ca6
--- /dev/null
+++ b/llvm/test/CodeGen/X86/avgflooru-i128.ll
@@ -0,0 +1,145 @@
+; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py UTC_ARGS: --version 5
+; RUN: llc -mtriple=x86_64 < %s | FileCheck %s
+
+define i128 @avgflooru_i128(i128 %x, i128 %y) {
+; CHECK-LABEL: avgflooru_i128:
+; CHECK: # %bb.0: # %start
+; CHECK-NEXT: movq %rdi, %rax
+; CHECK-NEXT: addq %rdx, %rax
+; CHECK-NEXT: adcq %rcx, %rsi
+; CHECK-NEXT: setb %cl
+; CHECK-NEXT: shrdq $1, %rsi, %rax
+; CHECK-NEXT: movzbl %cl, %edx
+; CHECK-NEXT: shldq $63, %rsi, %rdx
+; CHECK-NEXT: retq
+start:
+ %xor = xor i128 %y, %x
+ %lshr = lshr i128 %xor, 1
+ %and = and i128 %y, %x
+ %add = add i128 %lshr, %and
+ ret i128 %add
+}
+
+declare void @use(i8)
+
+define i128 @avgflooru_i128_multi_use(i128 %x, i128 %y) nounwind {
+; CHECK-LABEL: avgflooru_i128_multi_use:
+; CHECK: # %bb.0: # %start
+; CHECK-NEXT: pushq %rbp
+; CHECK-NEXT: pushq %r15
+; CHECK-NEXT: pushq %r14
+; CHECK-NEXT: pushq %r13
+; CHECK-NEXT: pushq %r12
+; CHECK-NEXT: pushq %rbx
+; CHECK-NEXT: pushq %rax
+; CHECK-NEXT: movq %rcx, %rbx
+; CHECK-NEXT: movq %rdx, %r14
+; CHECK-NEXT: movq %rsi, %r15
+; CHECK-NEXT: movq %rdi, %r12
+; CHECK-NEXT: movq %rdx, %r13
+; CHECK-NEXT: xorq %rdi, %r13
+; CHECK-NEXT: movq %rcx, %rbp
+; CHECK-NEXT: xorq %rsi, %rbp
+; CHECK-NEXT: movq %r13, %rdi
+; CHECK-NEXT: movq %rbp, %rsi
+; CHECK-NEXT: callq use@PLT
+; CHECK-NEXT: shrdq $1, %rbp, %r13
+; CHECK-NEXT: shrq %rbp
+; CHECK-NEXT: movq %r13, %rdi
+; CHECK-NEXT: movq %rbp, %rsi
+; CHECK-NEXT: callq use@PLT
+; CHECK-NEXT: addq %r14, %r12
+; CHECK-NEXT: adcq %rbx, %r15
+; CHECK-NEXT: setb %al
+; CHECK-NEXT: shrdq $1, %r15, %r12
+; CHECK-NEXT: movzbl %al, %edx
+; CHECK-NEXT: shldq $63, %r15, %rdx
+; CHECK-NEXT: movq %r12, %rax
+; CHECK-NEXT: addq $8, %rsp
+; CHECK-NEXT: popq %rbx
+; CHECK-NEXT: popq %r12
+; CHECK-NEXT: popq %r13
+; CHECK-NEXT: popq %r14
+; CHECK-NEXT: popq %r15
+; CHECK-NEXT: popq %rbp
+; CHECK-NEXT: retq
+start:
+ %xor = xor i128 %y, %x
+ call void @use(i128 %xor)
+ %lshr = lshr i128 %xor, 1
+ call void @use(i128 %lshr)
+ %and = and i128 %y, %x
+ %add = add i128 %lshr, %and
+ ret i128 %add
+}
+
+; This test case shouldn't combine because it's not
+; an avgflooru operation
+
+define i128 @avgflooru_i128_negative(i128 %x, i128 %y) {
+; CHECK-LABEL: avgflooru_i128_negative:
+; CHECK: # %bb.0: # %start
+; CHECK-NEXT: movq %rdi, %rax
+; CHECK-NEXT: andq %rsi, %rcx
+; CHECK-NEXT: notq %rsi
+; CHECK-NEXT: andq %rdi, %rdx
+; CHECK-NEXT: notq %rax
+; CHECK-NEXT: addq %rdx, %rax
+; CHECK-NEXT: adcq %rcx, %rsi
+; CHECK-NEXT: movq %rsi, %rdx
+; CHECK-NEXT: retq
+start:
+ %xor = xor i128 %x, -1
+ %and = and i128 %y, %x
+ %add = add i128 %xor, %and
+ ret i128 %add
+}
+
+; This negative test case shouldn't combine, i32 is already properly
+; handled in terms of legalization, compared to the i128
+
+define i32 @avgflooru_i128_negative2(i32 %x, i32 %y) {
+; CHECK-LABEL: avgflooru_i128_negative2:
+; CHECK: # %bb.0: # %start
+; CHECK-NEXT: movl %edi, %ecx
+; CHECK-NEXT: movl %esi, %eax
+; CHECK-NEXT: addq %rcx, %rax
+; CHECK-NEXT: shrq %rax
+; CHECK-NEXT: # kill: def $eax killed $eax killed $rax
+; CHECK-NEXT: retq
+start:
+ %xor = xor i32 %y, %x
+ %lshr = lshr i32 %xor, 1
+ %and = and i32 %y, %x
+ %add = add i32 %lshr, %and
+ ret i32 %add
+}
+
+define <2 x i128> @avgflooru_i128_vec(<2 x i128> %x, <2 x i128> %y) {
+; CHECK-LABEL: avgflooru_i128_vec:
+; CHECK: # %bb.0: # %start
+; CHECK-NEXT: movq %rdi, %rax
+; CHECK-NEXT: addq {{[0-9]+}}(%rsp), %rsi
+; CHECK-NEXT: adcq {{[0-9]+}}(%rsp), %rdx
+; CHECK-NEXT: setb %dil
+; CHECK-NEXT: movzbl %dil, %edi
+; CHECK-NEXT: shldq $63, %rdx, %rdi
+; CHECK-NEXT: addq {{[0-9]+}}(%rsp), %rcx
+; CHECK-NEXT: adcq {{[0-9]+}}(%rsp), %r8
+; CHECK-NEXT: setb %r9b
+; CHECK-NEXT: movzbl %r9b, %r9d
+; CHECK-NEXT: shldq $63, %r8, %r9
+; CHECK-NEXT: shldq $63, %rsi, %rdx
+; CHECK-NEXT: shldq $63, %rcx, %r8
+; CHECK-NEXT: movq %r8, 16(%rax)
+; CHECK-NEXT: movq %rdx, (%rax)
+; CHECK-NEXT: movq %r9, 24(%rax)
+; CHECK-NEXT: movq %rdi, 8(%rax)
+; CHECK-NEXT: retq
+start:
+ %xor = xor <2 x i128> %y, %x
+ %lshr = lshr <2 x i128> %xor, <i128 1, i128 1>
+ %and = and <2 x i128> %y, %x
+ %add = add <2 x i128> %lshr, %and
+ ret <2 x i128> %add
+}
diff --git a/llvm/test/CodeGen/X86/avgflooru-scalar.ll b/llvm/test/CodeGen/X86/avgflooru-scalar.ll
index d21c9d65ea9c8..0c91a9da5720a 100644
--- a/llvm/test/CodeGen/X86/avgflooru-scalar.ll
+++ b/llvm/test/CodeGen/X86/avgflooru-scalar.ll
@@ -168,26 +168,14 @@ define i32 @test_ext_i32(i32 %a0, i32 %a1) nounwind {
define i64 @test_fixed_i64(i64 %a0, i64 %a1) nounwind {
; X86-LABEL: test_fixed_i64:
; X86: # %bb.0:
-; X86-NEXT: pushl %ebx
-; X86-NEXT: pushl %edi
-; X86-NEXT: pushl %esi
-; X86-NEXT: movl {{[0-9]+}}(%esp), %esi
-; X86-NEXT: movl {{[0-9]+}}(%esp), %edi
-; X86-NEXT: movl {{[0-9]+}}(%esp), %eax
; X86-NEXT: movl {{[0-9]+}}(%esp), %ecx
-; X86-NEXT: movl %eax, %ebx
-; X86-NEXT: xorl %esi, %ebx
-; X86-NEXT: movl %ecx, %edx
-; X86-NEXT: xorl %edi, %edx
-; X86-NEXT: shrdl $1, %edx, %ebx
-; X86-NEXT: andl %edi, %ecx
-; X86-NEXT: shrl %edx
-; X86-NEXT: andl %esi, %eax
-; X86-NEXT: addl %ebx, %eax
-; X86-NEXT: adcl %ecx, %edx
-; X86-NEXT: popl %esi
-; X86-NEXT: popl %edi
-; X86-NEXT: popl %ebx
+; X86-NEXT: movl {{[0-9]+}}(%esp), %eax
+; X86-NEXT: addl {{[0-9]+}}(%esp), %ecx
+; X86-NEXT: adcl {{[0-9]+}}(%esp), %eax
+; X86-NEXT: setb %dl
+; X86-NEXT: movzbl %dl, %edx
+; X86-NEXT: shldl $31, %eax, %edx
+; X86-NEXT: shldl $31, %ecx, %eax
; X86-NEXT: retl
;
; X64-LABEL: test_fixed_i64:
@@ -208,26 +196,14 @@ define i64 @test_fixed_i64(i64 %a0, i64 %a1) nounwind {
define i64 @test_ext_i64(i64 %a0, i64 %a1) nounwind {
; X86-LABEL: test_ext_i64:
; X86: # %bb.0:
-; X86-NEXT: pushl %ebx
-; X86-NEXT: pushl %edi
-; X86-NEXT: pushl %esi
-; X86-NEXT: movl {{[0-9]+}}(%esp), %esi
-; X86-NEXT: movl {{[0-9]+}}(%esp), %edi
-; X86-NEXT: movl {{[0-9]+}}(%esp), %eax
; X86-NEXT: movl {{[0-9]+}}(%esp), %ecx
-; X86-NEXT: movl %eax, %ebx
-; X86-NEXT: xorl %esi, %ebx
-; X86-NEXT: movl %ecx, %edx
-; X86-NEXT: xorl %edi, %edx
-; X86-NEXT: shrdl $1, %edx, %ebx
-; X86-NEXT: andl %edi, %ecx
-; X86-NEXT: shrl %edx
-; X86-NEXT: andl %esi, %eax
-; X86-NEXT: addl %ebx, %eax
-; X86-NEXT: adcl %ecx, %edx
-; X86-NEXT: popl %esi
-; X86-NEXT: popl %edi
-; X86-NEXT: popl %ebx
+; X86-NEXT: movl {{[0-9]+}}(%esp), %eax
+; X86-NEXT: addl {{[0-9]+}}(%esp), %ecx
+; X86-NEXT: adcl {{[0-9]+}}(%esp), %eax
+; X86-NEXT: setb %dl
+; X86-NEXT: movzbl %dl, %edx
+; X86-NEXT: shldl $31, %eax, %edx
+; X86-NEXT: shldl $31, %ecx, %eax
; X86-NEXT: retl
;
; X64-LABEL: test_ext_i64:
|
|
|
topperc
approved these changes
Jul 30, 2024
|
@topperc Just in case you do see the LLVM Bot emails, they're all false. It has nothing to do with this merge, I'm not sure why it does that. But the error is related to |
lukel97
added a commit
to lukel97/llvm-project
that referenced
this pull request
Mar 13, 2025
…ructions by better legalization handling of AVGFLOORU for illegal data types" (llvm#101223)
lukel97
added a commit
to lukel97/llvm-project
that referenced
this pull request
Mar 13, 2025
…ctions by better legalization handling of AVGFLOORU for illegal data types" (llvm#101223)
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Previous reverted merge: #99913
Previous error generated by LLVM Bot:
Error generated by avgflooru-i128.ll
It doesn't seem as if there's a fix for the error message. I wasn't able to reproduce this error. All my attempts to do
ninja checkwas a successful. This may be an issue with LLVM Bot? I removed theAArch64/avgflooru-i128.lltest in order to prevent this issue from happening.
To reiterate:
https://rust.godbolt.org/z/T7eKP3Tvo
x86: https://alive2.llvm.org/ce/z/ze88Hw
cc: @RKSimon @topperc