Skip to content

[SelectionDAG] Fix bug related to demanded bits/elts for BITCAST #139085

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
wants to merge 3 commits into from
+9,274 −8,374
Closed

[SelectionDAG] Fix bug related to demanded bits/elts for BITCAST #139085

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
c3081f4
[SelectionDAG] Fix bug related to demanded bits/elts for BITCAST
bjope May 8, 2025
4e7858e
Adding DoNotPoisonEltMask to SimplifyDemandedVectorElts
bjope May 14, 2025
99dde85
Add DoNotPoisonEltMask to SimplifyDemandedVectorEltsForTargetNode
bjope May 22, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
12 changes: 11 additions & 1 deletion llvm/include/llvm/CodeGen/TargetLowering.h
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4192,6 +4192,15 @@ class TargetLowering : public TargetLoweringBase {
/// results of this function, because simply replacing TLO.Old
/// with TLO.New will be incorrect when this parameter is true and TLO.Old
/// has multiple uses.
/// Vector elements that aren't demanded can be turned into poison unless the
/// corresponding bi in the \p DoNotPoisonEltMask is set.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
/// corresponding bi in the \p DoNotPoisonEltMask is set.
/// corresponding bit in the \p DoNotPoisonEltMask is set.

bool SimplifyDemandedVectorElts(SDValue Op, const APInt &DemandedEltMask,
const APInt &DoNotPoisonEltMask,
APInt &KnownUndef, APInt &KnownZero,
TargetLoweringOpt &TLO, unsigned Depth = 0,
bool AssumeSingleUse = false) const;
/// Version of SimplifyDemandedVectorElts without the DoNotPoisonEltMask
/// argument. All undemanded elements can be turned into poison.
bool SimplifyDemandedVectorElts(SDValue Op, const APInt &DemandedEltMask,
APInt &KnownUndef, APInt &KnownZero,
TargetLoweringOpt &TLO, unsigned Depth = 0,
Expand Down Expand Up @@ -4269,7 +4278,8 @@ class TargetLowering : public TargetLoweringBase {
/// (used to simplify the caller). The KnownUndef/Zero elements may only be
/// accurate for those bits in the DemandedMask.
virtual bool SimplifyDemandedVectorEltsForTargetNode(
SDValue Op, const APInt &DemandedElts, APInt &KnownUndef,
SDValue Op, const APInt &DemandedElts, const APInt &DoNotPoisonEltMask,
APInt &KnownUndef,
APInt &KnownZero, TargetLoweringOpt &TLO, unsigned Depth = 0) const;

/// Attempt to simplify any target nodes based on the demanded bits/elts,
Expand Down
4 changes: 3 additions & 1 deletion llvm/lib/CodeGen/SelectionDAG/DAGCombiner.cpp
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1414,7 +1414,9 @@ bool DAGCombiner::SimplifyDemandedVectorElts(SDValue Op,
bool AssumeSingleUse) {
TargetLowering::TargetLoweringOpt TLO(DAG, LegalTypes, LegalOperations);
APInt KnownUndef, KnownZero;
if (!TLI.SimplifyDemandedVectorElts(Op, DemandedElts, KnownUndef, KnownZero,
APInt DoNotPoisonElts = APInt::getZero(DemandedElts.getBitWidth());
if (!TLI.SimplifyDemandedVectorElts(Op, DemandedElts, DoNotPoisonElts,
KnownUndef, KnownZero,
TLO, 0, AssumeSingleUse))
return false;

Expand Down
190 changes: 128 additions & 62 deletions llvm/lib/CodeGen/SelectionDAG/TargetLowering.cpp
View file
Open in desktop

Large diffs are not rendered by default.

116 changes: 77 additions & 39 deletions llvm/lib/Target/X86/X86ISelLowering.cpp
View file
Open in desktop

Large diffs are not rendered by default.

1 change: 1 addition & 0 deletions llvm/lib/Target/X86/X86ISelLowering.h
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1302,6 +1302,7 @@ namespace llvm {

bool SimplifyDemandedVectorEltsForTargetNode(SDValue Op,
const APInt &DemandedElts,
const APInt &DoNotPoisonElts,
APInt &KnownUndef,
APInt &KnownZero,
TargetLoweringOpt &TLO,
Expand Down
11 changes: 5 additions & 6 deletions llvm/test/CodeGen/AArch64/reduce-or.ll
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -218,13 +218,12 @@ define i8 @test_redor_v3i8(<3 x i8> %a) {
; CHECK-NEXT: movi v0.2d, #0000000000000000
; CHECK-NEXT: mov v0.h[0], w0
; CHECK-NEXT: mov v0.h[1], w1
; CHECK-NEXT: fmov x8, d0
; CHECK-NEXT: mov v0.h[2], w2
; CHECK-NEXT: fmov x9, d0
; CHECK-NEXT: lsr x10, x9, #32
; CHECK-NEXT: lsr x9, x9, #16
; CHECK-NEXT: orr w8, w8, w10
; CHECK-NEXT: orr w0, w8, w9
; CHECK-NEXT: fmov x8, d0
; CHECK-NEXT: lsr x9, x8, #32
; CHECK-NEXT: lsr x10, x8, #16
; CHECK-NEXT: orr w8, w8, w9
; CHECK-NEXT: orr w0, w8, w10
; CHECK-NEXT: ret
;
; GISEL-LABEL: test_redor_v3i8:
Expand Down
11 changes: 5 additions & 6 deletions llvm/test/CodeGen/AArch64/reduce-xor.ll
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -207,13 +207,12 @@ define i8 @test_redxor_v3i8(<3 x i8> %a) {
; CHECK-NEXT: movi v0.2d, #0000000000000000
; CHECK-NEXT: mov v0.h[0], w0
; CHECK-NEXT: mov v0.h[1], w1
; CHECK-NEXT: fmov x8, d0
; CHECK-NEXT: mov v0.h[2], w2
; CHECK-NEXT: fmov x9, d0
; CHECK-NEXT: lsr x10, x9, #32
; CHECK-NEXT: lsr x9, x9, #16
; CHECK-NEXT: eor w8, w8, w10
; CHECK-NEXT: eor w0, w8, w9
; CHECK-NEXT: fmov x8, d0
; CHECK-NEXT: lsr x9, x8, #32
; CHECK-NEXT: lsr x10, x8, #16
; CHECK-NEXT: eor w8, w8, w9
; CHECK-NEXT: eor w0, w8, w10
; CHECK-NEXT: ret
;
; GISEL-LABEL: test_redxor_v3i8:
Expand Down
12 changes: 7 additions & 5 deletions llvm/test/CodeGen/AArch64/vecreduce-and-legalization.ll
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -125,12 +125,14 @@ define i8 @test_v9i8(<9 x i8> %a) nounwind {
define i32 @test_v3i32(<3 x i32> %a) nounwind {
; CHECK-LABEL: test_v3i32:
; CHECK: // %bb.0:
; CHECK-NEXT: ext v1.16b, v0.16b, v0.16b, #8
; CHECK-NEXT: mov v1.16b, v0.16b
; CHECK-NEXT: mov w8, #-1 // =0xffffffff
; CHECK-NEXT: mov v1.s[3], w8
; CHECK-NEXT: ext v1.16b, v1.16b, v1.16b, #8
; CHECK-NEXT: and v0.8b, v0.8b, v1.8b
; CHECK-NEXT: fmov x8, d0
; CHECK-NEXT: lsr x8, x8, #32
; CHECK-NEXT: and v1.8b, v0.8b, v1.8b
; CHECK-NEXT: fmov x9, d1
; CHECK-NEXT: and w0, w9, w8
; CHECK-NEXT: lsr x9, x8, #32
; CHECK-NEXT: and w0, w8, w9
; CHECK-NEXT: ret
%b = call i32 @llvm.vector.reduce.and.v3i32(<3 x i32> %a)
ret i32 %b
Expand Down
98 changes: 51 additions & 47 deletions llvm/test/CodeGen/AMDGPU/cvt_f32_ubyte.ll
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1904,69 +1904,74 @@ define amdgpu_kernel void @load_v7i8_to_v7f32(ptr addrspace(1) noalias %out, ptr
; VI-NEXT: v_addc_u32_e32 v1, vcc, 0, v1, vcc
; VI-NEXT: v_add_u32_e32 v2, vcc, 5, v0
; VI-NEXT: v_addc_u32_e32 v3, vcc, 0, v1, vcc
; VI-NEXT: flat_load_ubyte v10, v[2:3]
; VI-NEXT: v_add_u32_e32 v2, vcc, 6, v0
; VI-NEXT: v_addc_u32_e32 v3, vcc, 0, v1, vcc
; VI-NEXT: v_add_u32_e32 v4, vcc, 1, v0
; VI-NEXT: v_add_u32_e32 v4, vcc, 4, v0
; VI-NEXT: v_addc_u32_e32 v5, vcc, 0, v1, vcc
; VI-NEXT: v_add_u32_e32 v6, vcc, 2, v0
; VI-NEXT: v_add_u32_e32 v6, vcc, 1, v0
; VI-NEXT: v_addc_u32_e32 v7, vcc, 0, v1, vcc
; VI-NEXT: v_add_u32_e32 v8, vcc, 3, v0
; VI-NEXT: v_add_u32_e32 v8, vcc, 2, v0
; VI-NEXT: v_addc_u32_e32 v9, vcc, 0, v1, vcc
; VI-NEXT: flat_load_ubyte v6, v[6:7]
; VI-NEXT: flat_load_ubyte v7, v[8:9]
; VI-NEXT: flat_load_ubyte v8, v[2:3]
; VI-NEXT: flat_load_ubyte v2, v[0:1]
; VI-NEXT: v_add_u32_e32 v10, vcc, 3, v0
; VI-NEXT: v_addc_u32_e32 v11, vcc, 0, v1, vcc
; VI-NEXT: v_add_u32_e32 v12, vcc, 6, v0
; VI-NEXT: v_addc_u32_e32 v13, vcc, 0, v1, vcc
; VI-NEXT: flat_load_ubyte v2, v[2:3]
; VI-NEXT: flat_load_ubyte v4, v[4:5]
; VI-NEXT: v_add_u32_e32 v0, vcc, 4, v0
; VI-NEXT: v_addc_u32_e32 v1, vcc, 0, v1, vcc
; VI-NEXT: flat_load_ubyte v9, v[0:1]
; VI-NEXT: flat_load_ubyte v5, v[6:7]
; VI-NEXT: flat_load_ubyte v7, v[8:9]
; VI-NEXT: flat_load_ubyte v3, v[10:11]
; VI-NEXT: flat_load_ubyte v6, v[12:13]
; VI-NEXT: flat_load_ubyte v0, v[0:1]
; VI-NEXT: v_mov_b32_e32 v8, 0x3020504
; VI-NEXT: s_mov_b32 s3, 0xf000
; VI-NEXT: s_mov_b32 s2, -1
; VI-NEXT: s_waitcnt vmcnt(6)
; VI-NEXT: v_cvt_f32_ubyte0_e32 v5, v10
; VI-NEXT: v_lshlrev_b32_e32 v9, 8, v2
; VI-NEXT: s_waitcnt vmcnt(5)
; VI-NEXT: v_or_b32_e32 v4, v9, v4
; VI-NEXT: s_waitcnt vmcnt(4)
; VI-NEXT: v_cvt_f32_ubyte0_e32 v3, v7
; VI-NEXT: v_cvt_f32_ubyte0_e32 v1, v5
; VI-NEXT: s_waitcnt vmcnt(3)
; VI-NEXT: v_cvt_f32_ubyte0_e32 v2, v7
; VI-NEXT: s_waitcnt vmcnt(2)
; VI-NEXT: v_cvt_f32_ubyte0_e32 v0, v2
; VI-NEXT: v_cvt_f32_ubyte0_e32 v2, v6
; VI-NEXT: s_waitcnt vmcnt(1)
; VI-NEXT: v_cvt_f32_ubyte0_e32 v1, v4
; VI-NEXT: v_cvt_f32_ubyte0_e32 v6, v8
; VI-NEXT: v_cvt_f32_ubyte0_e32 v3, v3
; VI-NEXT: v_perm_b32 v4, v4, s0, v8
; VI-NEXT: s_waitcnt vmcnt(0)
; VI-NEXT: v_cvt_f32_ubyte0_e32 v4, v9
; VI-NEXT: buffer_store_dwordx3 v[4:6], off, s[0:3], 0 offset:16
; VI-NEXT: v_cvt_f32_ubyte0_e32 v0, v0
; VI-NEXT: v_cvt_f32_ubyte0_e32 v6, v6
; VI-NEXT: v_cvt_f32_ubyte1_e32 v5, v4
; VI-NEXT: v_cvt_f32_ubyte0_e32 v4, v4
; VI-NEXT: buffer_store_dwordx4 v[0:3], off, s[0:3], 0
; VI-NEXT: buffer_store_dwordx3 v[4:6], off, s[0:3], 0 offset:16
; VI-NEXT: s_endpgm
;
; GFX10-LABEL: load_v7i8_to_v7f32:
; GFX10: ; %bb.0:
; GFX10-NEXT: s_load_dwordx4 s[0:3], s[4:5], 0x24
; GFX10-NEXT: v_lshlrev_b32_e32 v0, 3, v0
; GFX10-NEXT: v_mov_b32_e32 v8, 0
; GFX10-NEXT: v_mov_b32_e32 v4, 0
; GFX10-NEXT: v_mov_b32_e32 v7, 0
; GFX10-NEXT: s_waitcnt lgkmcnt(0)
; GFX10-NEXT: s_clause 0x5
; GFX10-NEXT: global_load_ubyte v4, v0, s[2:3] offset:6
; GFX10-NEXT: global_load_ubyte v5, v0, s[2:3] offset:6
; GFX10-NEXT: global_load_ubyte v1, v0, s[2:3] offset:3
; GFX10-NEXT: global_load_ubyte v2, v0, s[2:3] offset:2
; GFX10-NEXT: global_load_ubyte v5, v0, s[2:3] offset:1
; GFX10-NEXT: global_load_short_d16 v7, v0, s[2:3] offset:4
; GFX10-NEXT: global_load_ubyte v6, v0, s[2:3] offset:1
; GFX10-NEXT: global_load_short_d16 v4, v0, s[2:3] offset:4
; GFX10-NEXT: global_load_ubyte v0, v0, s[2:3]
; GFX10-NEXT: s_waitcnt vmcnt(5)
; GFX10-NEXT: v_cvt_f32_ubyte0_e32 v6, v4
; GFX10-NEXT: s_waitcnt vmcnt(4)
; GFX10-NEXT: v_cvt_f32_ubyte0_e32 v3, v1
; GFX10-NEXT: s_waitcnt vmcnt(3)
; GFX10-NEXT: v_cvt_f32_ubyte0_e32 v2, v2
; GFX10-NEXT: s_waitcnt vmcnt(2)
; GFX10-NEXT: v_cvt_f32_ubyte0_e32 v1, v5
; GFX10-NEXT: v_cvt_f32_ubyte0_e32 v1, v6
; GFX10-NEXT: v_cvt_f32_ubyte0_e32 v6, v5
; GFX10-NEXT: s_waitcnt vmcnt(1)
; GFX10-NEXT: v_cvt_f32_ubyte1_e32 v5, v7
; GFX10-NEXT: v_cvt_f32_ubyte0_e32 v4, v7
; GFX10-NEXT: v_cvt_f32_ubyte1_e32 v5, v4
; GFX10-NEXT: v_cvt_f32_ubyte0_e32 v4, v4
; GFX10-NEXT: s_waitcnt vmcnt(0)
; GFX10-NEXT: v_cvt_f32_ubyte0_e32 v0, v0
; GFX10-NEXT: global_store_dwordx3 v8, v[4:6], s[0:1] offset:16
; GFX10-NEXT: global_store_dwordx4 v8, v[0:3], s[0:1]
; GFX10-NEXT: global_store_dwordx3 v7, v[4:6], s[0:1] offset:16
; GFX10-NEXT: global_store_dwordx4 v7, v[0:3], s[0:1]
; GFX10-NEXT: s_endpgm
;
; GFX9-LABEL: load_v7i8_to_v7f32:
Expand All @@ -1984,8 +1989,8 @@ define amdgpu_kernel void @load_v7i8_to_v7f32(ptr addrspace(1) noalias %out, ptr
; GFX9-NEXT: s_waitcnt vmcnt(5)
; GFX9-NEXT: v_cvt_f32_ubyte0_e32 v6, v1
; GFX9-NEXT: s_waitcnt vmcnt(4)
; GFX9-NEXT: v_cvt_f32_ubyte1_e32 v5, v2
; GFX9-NEXT: v_cvt_f32_ubyte0_e32 v4, v2
; GFX9-NEXT: v_cvt_f32_ubyte1_sdwa v5, v2 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:WORD_0
; GFX9-NEXT: v_cvt_f32_ubyte0_sdwa v4, v2 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:WORD_0
; GFX9-NEXT: s_waitcnt vmcnt(3)
; GFX9-NEXT: v_cvt_f32_ubyte0_e32 v3, v3
; GFX9-NEXT: s_waitcnt vmcnt(2)
Expand All @@ -2001,34 +2006,33 @@ define amdgpu_kernel void @load_v7i8_to_v7f32(ptr addrspace(1) noalias %out, ptr
; GFX11-LABEL: load_v7i8_to_v7f32:
; GFX11: ; %bb.0:
; GFX11-NEXT: s_load_b128 s[0:3], s[4:5], 0x24
; GFX11-NEXT: v_and_b32_e32 v0, 0x3ff, v0
; GFX11-NEXT: v_mov_b32_e32 v8, 0
; GFX11-NEXT: v_dual_mov_b32 v7, 0 :: v_dual_and_b32 v0, 0x3ff, v0
; GFX11-NEXT: v_mov_b32_e32 v4, 0
; GFX11-NEXT: s_delay_alu instid0(VALU_DEP_2)
; GFX11-NEXT: v_lshlrev_b32_e32 v0, 3, v0
; GFX11-NEXT: s_waitcnt lgkmcnt(0)
; GFX11-NEXT: s_clause 0x5
; GFX11-NEXT: global_load_u8 v4, v0, s[2:3] offset:6
; GFX11-NEXT: global_load_u8 v5, v0, s[2:3] offset:6
; GFX11-NEXT: global_load_u8 v1, v0, s[2:3] offset:3
; GFX11-NEXT: global_load_u8 v2, v0, s[2:3] offset:2
; GFX11-NEXT: global_load_u8 v5, v0, s[2:3] offset:1
; GFX11-NEXT: global_load_d16_b16 v7, v0, s[2:3] offset:4
; GFX11-NEXT: global_load_u8 v6, v0, s[2:3] offset:1
; GFX11-NEXT: global_load_d16_b16 v4, v0, s[2:3] offset:4
; GFX11-NEXT: global_load_u8 v0, v0, s[2:3]
; GFX11-NEXT: s_waitcnt vmcnt(5)
; GFX11-NEXT: v_cvt_f32_ubyte0_e32 v6, v4
; GFX11-NEXT: s_waitcnt vmcnt(4)
; GFX11-NEXT: v_cvt_f32_ubyte0_e32 v3, v1
; GFX11-NEXT: s_waitcnt vmcnt(3)
; GFX11-NEXT: v_cvt_f32_ubyte0_e32 v2, v2
; GFX11-NEXT: s_waitcnt vmcnt(2)
; GFX11-NEXT: v_cvt_f32_ubyte0_e32 v1, v5
; GFX11-NEXT: v_cvt_f32_ubyte0_e32 v1, v6
; GFX11-NEXT: v_cvt_f32_ubyte0_e32 v6, v5
; GFX11-NEXT: s_waitcnt vmcnt(1)
; GFX11-NEXT: v_cvt_f32_ubyte1_e32 v5, v7
; GFX11-NEXT: v_cvt_f32_ubyte0_e32 v4, v7
; GFX11-NEXT: v_cvt_f32_ubyte1_e32 v5, v4
; GFX11-NEXT: v_cvt_f32_ubyte0_e32 v4, v4
; GFX11-NEXT: s_waitcnt vmcnt(0)
; GFX11-NEXT: v_cvt_f32_ubyte0_e32 v0, v0
; GFX11-NEXT: s_clause 0x1
; GFX11-NEXT: global_store_b96 v8, v[4:6], s[0:1] offset:16
; GFX11-NEXT: global_store_b128 v8, v[0:3], s[0:1]
; GFX11-NEXT: global_store_b96 v7, v[4:6], s[0:1] offset:16
; GFX11-NEXT: global_store_b128 v7, v[0:3], s[0:1]
; GFX11-NEXT: s_endpgm
%tid = call i32 @llvm.amdgcn.workitem.id.x()
%gep = getelementptr <7 x i8>, ptr addrspace(1) %in, i32 %tid
Expand Down
Loading