Skip to content

AMDGPU: Do not generate non-temporal hint when Load_Tr intrinsic did not specify it #79104

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 17 commits into from
Jan 23, 2024
Merged

AMDGPU: Do not generate non-temporal hint when Load_Tr intrinsic did not specify it #79104

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
17 commits
Select commit Hold shift + click to select a range
f2dbe39
[AMDGPU] Make v32f16/v32i16 legal
changpeng Oct 27, 2023
940cb66
Merge branch 'llvm:main' into main
changpeng Oct 27, 2023
57b3c57
[AMDGPU] Make v32f16/v32i16 legal
changpeng Oct 27, 2023
b52722d
Merge branch 'llvm:main' into main
changpeng Oct 27, 2023
acb3527
Merge branch 'llvm:main' into main
changpeng Nov 10, 2023
23d3456
[GISel] Guide return in llvm::getIConstantSplatVal
changpeng Nov 10, 2023
4cd8d09
[GISel] Guide return in llvm::getIConstantSplatVal
changpeng Nov 10, 2023
2155fe1
Merge branch 'main' of https://github.com/changpeng/llvm-project into…
changpeng Nov 14, 2023
dd2349b
Merge branch 'llvm:main' into main
changpeng Dec 5, 2023
7c8f914
Merge branch 'llvm:main' into main
changpeng Dec 7, 2023
7d3f144
Merge branch 'llvm:main' into main
changpeng Jan 15, 2024
05a76b9
Merge branch 'llvm:main' into main
changpeng Jan 18, 2024
d6d9463
Merge branch 'llvm:main' into main
changpeng Jan 23, 2024
cd972f4
Merge branch 'llvm:main' into main
changpeng Jan 23, 2024
4000266
AMDGPU: Do not generate non-temporal hint when Load_Tr intrinsic did …
changpeng Jan 23, 2024
ead5cf5
Merge branch 'llvm:main' into main
changpeng Jan 23, 2024
7fd1c2d
AMDGPU: Do not generate non-temporal hint when Load_Tr intrinsic did …
changpeng Jan 23, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
9 changes: 9 additions & 0 deletions llvm/lib/Target/AMDGPU/SIISelLowering.cpp
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1348,6 +1348,14 @@ bool SITargetLowering::getTgtMemIntrinsic(IntrinsicInfo &Info,
MachineMemOperand::MOVolatile;
return true;
}
case Intrinsic::amdgcn_global_load_tr: {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This case should also be handled in getAdrModeArguments below.

Info.opc = ISD::INTRINSIC_W_CHAIN;
Info.memVT = MVT::getVT(CI.getType());
Info.ptrVal = CI.getOperand(0);
Info.align.reset();
Info.flags |= MachineMemOperand::MOLoad;
return true;
}
case Intrinsic::amdgcn_ds_gws_init:
case Intrinsic::amdgcn_ds_gws_barrier:
case Intrinsic::amdgcn_ds_gws_sema_v:
Expand Down Expand Up @@ -1407,6 +1415,7 @@ bool SITargetLowering::getAddrModeArguments(IntrinsicInst *II,
SmallVectorImpl<Value*> &Ops,
Type *&AccessTy) const {
switch (II->getIntrinsicID()) {
case Intrinsic::amdgcn_global_load_tr:
case Intrinsic::amdgcn_ds_ordered_add:
case Intrinsic::amdgcn_ds_ordered_swap:
case Intrinsic::amdgcn_ds_append:
Expand Down
24 changes: 8 additions & 16 deletions llvm/test/CodeGen/AMDGPU/llvm.amdgcn.global.load.tr-w32.ll
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -13,9 +13,8 @@ define amdgpu_kernel void @global_load_tr_b64(ptr addrspace(1) %addr, ptr addrsp
; GFX12-SDAG-W32-NEXT: s_load_b128 s[0:3], s[0:1], 0x24
; GFX12-SDAG-W32-NEXT: v_mov_b32_e32 v2, 0
; GFX12-SDAG-W32-NEXT: s_wait_kmcnt 0x0
; GFX12-SDAG-W32-NEXT: global_load_tr_b64 v[0:1], v2, s[0:1] offset:32 th:TH_LOAD_NT
; GFX12-SDAG-W32-NEXT: global_load_tr_b64 v[0:1], v2, s[0:1] offset:32
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks!

; GFX12-SDAG-W32-NEXT: s_wait_loadcnt 0x0
; GFX12-SDAG-W32-NEXT: global_inv scope:SCOPE_SYS
; GFX12-SDAG-W32-NEXT: global_store_b64 v2, v[0:1], s[2:3]
; GFX12-SDAG-W32-NEXT: s_nop 0
; GFX12-SDAG-W32-NEXT: s_sendmsg sendmsg(MSG_DEALLOC_VGPRS)
Expand All @@ -26,9 +25,8 @@ define amdgpu_kernel void @global_load_tr_b64(ptr addrspace(1) %addr, ptr addrsp
; GFX12-GISEL-W32-NEXT: s_load_b128 s[0:3], s[0:1], 0x24
; GFX12-GISEL-W32-NEXT: v_mov_b32_e32 v2, 0
; GFX12-GISEL-W32-NEXT: s_wait_kmcnt 0x0
; GFX12-GISEL-W32-NEXT: global_load_tr_b64 v[0:1], v2, s[0:1] offset:32 th:TH_LOAD_NT
; GFX12-GISEL-W32-NEXT: global_load_tr_b64 v[0:1], v2, s[0:1] offset:32
; GFX12-GISEL-W32-NEXT: s_wait_loadcnt 0x0
; GFX12-GISEL-W32-NEXT: global_inv scope:SCOPE_SYS
; GFX12-GISEL-W32-NEXT: global_store_b64 v2, v[0:1], s[2:3]
; GFX12-GISEL-W32-NEXT: s_nop 0
; GFX12-GISEL-W32-NEXT: s_sendmsg sendmsg(MSG_DEALLOC_VGPRS)
Expand All @@ -46,9 +44,8 @@ define amdgpu_kernel void @global_load_tr_b128_i16(ptr addrspace(1) %addr, ptr a
; GFX12-SDAG-W32-NEXT: s_load_b128 s[0:3], s[0:1], 0x24
; GFX12-SDAG-W32-NEXT: v_mov_b32_e32 v4, 0
; GFX12-SDAG-W32-NEXT: s_wait_kmcnt 0x0
; GFX12-SDAG-W32-NEXT: global_load_tr_b128 v[0:3], v4, s[0:1] offset:32 th:TH_LOAD_NT
; GFX12-SDAG-W32-NEXT: global_load_tr_b128 v[0:3], v4, s[0:1] offset:32
; GFX12-SDAG-W32-NEXT: s_wait_loadcnt 0x0
; GFX12-SDAG-W32-NEXT: global_inv scope:SCOPE_SYS
; GFX12-SDAG-W32-NEXT: global_store_b128 v4, v[0:3], s[2:3]
; GFX12-SDAG-W32-NEXT: s_nop 0
; GFX12-SDAG-W32-NEXT: s_sendmsg sendmsg(MSG_DEALLOC_VGPRS)
Expand All @@ -59,9 +56,8 @@ define amdgpu_kernel void @global_load_tr_b128_i16(ptr addrspace(1) %addr, ptr a
; GFX12-GISEL-W32-NEXT: s_load_b128 s[0:3], s[0:1], 0x24
; GFX12-GISEL-W32-NEXT: v_mov_b32_e32 v4, 0
; GFX12-GISEL-W32-NEXT: s_wait_kmcnt 0x0
; GFX12-GISEL-W32-NEXT: global_load_tr_b128 v[0:3], v4, s[0:1] offset:32 th:TH_LOAD_NT
; GFX12-GISEL-W32-NEXT: global_load_tr_b128 v[0:3], v4, s[0:1] offset:32
; GFX12-GISEL-W32-NEXT: s_wait_loadcnt 0x0
; GFX12-GISEL-W32-NEXT: global_inv scope:SCOPE_SYS
; GFX12-GISEL-W32-NEXT: global_store_b128 v4, v[0:3], s[2:3]
; GFX12-GISEL-W32-NEXT: s_nop 0
; GFX12-GISEL-W32-NEXT: s_sendmsg sendmsg(MSG_DEALLOC_VGPRS)
Expand All @@ -79,9 +75,8 @@ define amdgpu_kernel void @global_load_tr_b128_half(ptr addrspace(1) %addr, ptr
; GFX12-SDAG-W32-NEXT: s_load_b128 s[0:3], s[0:1], 0x24
; GFX12-SDAG-W32-NEXT: v_mov_b32_e32 v4, 0
; GFX12-SDAG-W32-NEXT: s_wait_kmcnt 0x0
; GFX12-SDAG-W32-NEXT: global_load_tr_b128 v[0:3], v4, s[0:1] offset:32 th:TH_LOAD_NT
; GFX12-SDAG-W32-NEXT: global_load_tr_b128 v[0:3], v4, s[0:1] offset:32
; GFX12-SDAG-W32-NEXT: s_wait_loadcnt 0x0
; GFX12-SDAG-W32-NEXT: global_inv scope:SCOPE_SYS
; GFX12-SDAG-W32-NEXT: global_store_b128 v4, v[0:3], s[2:3]
; GFX12-SDAG-W32-NEXT: s_nop 0
; GFX12-SDAG-W32-NEXT: s_sendmsg sendmsg(MSG_DEALLOC_VGPRS)
Expand All @@ -92,9 +87,8 @@ define amdgpu_kernel void @global_load_tr_b128_half(ptr addrspace(1) %addr, ptr
; GFX12-GISEL-W32-NEXT: s_load_b128 s[0:3], s[0:1], 0x24
; GFX12-GISEL-W32-NEXT: v_mov_b32_e32 v4, 0
; GFX12-GISEL-W32-NEXT: s_wait_kmcnt 0x0
; GFX12-GISEL-W32-NEXT: global_load_tr_b128 v[0:3], v4, s[0:1] offset:32 th:TH_LOAD_NT
; GFX12-GISEL-W32-NEXT: global_load_tr_b128 v[0:3], v4, s[0:1] offset:32
; GFX12-GISEL-W32-NEXT: s_wait_loadcnt 0x0
; GFX12-GISEL-W32-NEXT: global_inv scope:SCOPE_SYS
; GFX12-GISEL-W32-NEXT: global_store_b128 v4, v[0:3], s[2:3]
; GFX12-GISEL-W32-NEXT: s_nop 0
; GFX12-GISEL-W32-NEXT: s_sendmsg sendmsg(MSG_DEALLOC_VGPRS)
Expand All @@ -112,9 +106,8 @@ define amdgpu_kernel void @global_load_tr_b128_bfloat(ptr addrspace(1) %addr, pt
; GFX12-SDAG-W32-NEXT: s_load_b128 s[0:3], s[0:1], 0x24
; GFX12-SDAG-W32-NEXT: v_mov_b32_e32 v4, 0
; GFX12-SDAG-W32-NEXT: s_wait_kmcnt 0x0
; GFX12-SDAG-W32-NEXT: global_load_tr_b128 v[0:3], v4, s[0:1] offset:32 th:TH_LOAD_NT
; GFX12-SDAG-W32-NEXT: global_load_tr_b128 v[0:3], v4, s[0:1] offset:32
; GFX12-SDAG-W32-NEXT: s_wait_loadcnt 0x0
; GFX12-SDAG-W32-NEXT: global_inv scope:SCOPE_SYS
; GFX12-SDAG-W32-NEXT: global_store_b128 v4, v[0:3], s[2:3]
; GFX12-SDAG-W32-NEXT: s_nop 0
; GFX12-SDAG-W32-NEXT: s_sendmsg sendmsg(MSG_DEALLOC_VGPRS)
Expand All @@ -125,9 +118,8 @@ define amdgpu_kernel void @global_load_tr_b128_bfloat(ptr addrspace(1) %addr, pt
; GFX12-GISEL-W32-NEXT: s_load_b128 s[0:3], s[0:1], 0x24
; GFX12-GISEL-W32-NEXT: v_mov_b32_e32 v4, 0
; GFX12-GISEL-W32-NEXT: s_wait_kmcnt 0x0
; GFX12-GISEL-W32-NEXT: global_load_tr_b128 v[0:3], v4, s[0:1] offset:32 th:TH_LOAD_NT
; GFX12-GISEL-W32-NEXT: global_load_tr_b128 v[0:3], v4, s[0:1] offset:32
; GFX12-GISEL-W32-NEXT: s_wait_loadcnt 0x0
; GFX12-GISEL-W32-NEXT: global_inv scope:SCOPE_SYS
; GFX12-GISEL-W32-NEXT: global_store_b128 v4, v[0:3], s[2:3]
; GFX12-GISEL-W32-NEXT: s_nop 0
; GFX12-GISEL-W32-NEXT: s_sendmsg sendmsg(MSG_DEALLOC_VGPRS)
Expand Down
24 changes: 8 additions & 16 deletions llvm/test/CodeGen/AMDGPU/llvm.amdgcn.global.load.tr-w64.ll
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -13,9 +13,8 @@ define amdgpu_kernel void @global_load_tr_b64(ptr addrspace(1) %addr, ptr addrsp
; GFX12-SDAG-W64-NEXT: s_load_b128 s[0:3], s[0:1], 0x24
; GFX12-SDAG-W64-NEXT: v_mov_b32_e32 v0, 0
; GFX12-SDAG-W64-NEXT: s_wait_kmcnt 0x0
; GFX12-SDAG-W64-NEXT: global_load_tr_b64 v1, v0, s[0:1] offset:32 th:TH_LOAD_NT
; GFX12-SDAG-W64-NEXT: global_load_tr_b64 v1, v0, s[0:1] offset:32
; GFX12-SDAG-W64-NEXT: s_wait_loadcnt 0x0
; GFX12-SDAG-W64-NEXT: global_inv scope:SCOPE_SYS
; GFX12-SDAG-W64-NEXT: global_store_b32 v0, v1, s[2:3]
; GFX12-SDAG-W64-NEXT: s_nop 0
; GFX12-SDAG-W64-NEXT: s_sendmsg sendmsg(MSG_DEALLOC_VGPRS)
Expand All @@ -26,9 +25,8 @@ define amdgpu_kernel void @global_load_tr_b64(ptr addrspace(1) %addr, ptr addrsp
; GFX12-GISEL-W64-NEXT: s_load_b128 s[0:3], s[0:1], 0x24
; GFX12-GISEL-W64-NEXT: v_mov_b32_e32 v0, 0
; GFX12-GISEL-W64-NEXT: s_wait_kmcnt 0x0
; GFX12-GISEL-W64-NEXT: global_load_tr_b64 v1, v0, s[0:1] offset:32 th:TH_LOAD_NT
; GFX12-GISEL-W64-NEXT: global_load_tr_b64 v1, v0, s[0:1] offset:32
; GFX12-GISEL-W64-NEXT: s_wait_loadcnt 0x0
; GFX12-GISEL-W64-NEXT: global_inv scope:SCOPE_SYS
; GFX12-GISEL-W64-NEXT: global_store_b32 v0, v1, s[2:3]
; GFX12-GISEL-W64-NEXT: s_nop 0
; GFX12-GISEL-W64-NEXT: s_sendmsg sendmsg(MSG_DEALLOC_VGPRS)
Expand All @@ -46,9 +44,8 @@ define amdgpu_kernel void @global_load_tr_b128_i16(ptr addrspace(1) %addr, ptr a
; GFX12-SDAG-W64-NEXT: s_load_b128 s[0:3], s[0:1], 0x24
; GFX12-SDAG-W64-NEXT: v_mov_b32_e32 v2, 0
; GFX12-SDAG-W64-NEXT: s_wait_kmcnt 0x0
; GFX12-SDAG-W64-NEXT: global_load_tr_b128 v[0:1], v2, s[0:1] offset:32 th:TH_LOAD_NT
; GFX12-SDAG-W64-NEXT: global_load_tr_b128 v[0:1], v2, s[0:1] offset:32
; GFX12-SDAG-W64-NEXT: s_wait_loadcnt 0x0
; GFX12-SDAG-W64-NEXT: global_inv scope:SCOPE_SYS
; GFX12-SDAG-W64-NEXT: global_store_b64 v2, v[0:1], s[2:3]
; GFX12-SDAG-W64-NEXT: s_nop 0
; GFX12-SDAG-W64-NEXT: s_sendmsg sendmsg(MSG_DEALLOC_VGPRS)
Expand All @@ -59,9 +56,8 @@ define amdgpu_kernel void @global_load_tr_b128_i16(ptr addrspace(1) %addr, ptr a
; GFX12-GISEL-W64-NEXT: s_load_b128 s[0:3], s[0:1], 0x24
; GFX12-GISEL-W64-NEXT: v_mov_b32_e32 v2, 0
; GFX12-GISEL-W64-NEXT: s_wait_kmcnt 0x0
; GFX12-GISEL-W64-NEXT: global_load_tr_b128 v[0:1], v2, s[0:1] offset:32 th:TH_LOAD_NT
; GFX12-GISEL-W64-NEXT: global_load_tr_b128 v[0:1], v2, s[0:1] offset:32
; GFX12-GISEL-W64-NEXT: s_wait_loadcnt 0x0
; GFX12-GISEL-W64-NEXT: global_inv scope:SCOPE_SYS
; GFX12-GISEL-W64-NEXT: global_store_b64 v2, v[0:1], s[2:3]
; GFX12-GISEL-W64-NEXT: s_nop 0
; GFX12-GISEL-W64-NEXT: s_sendmsg sendmsg(MSG_DEALLOC_VGPRS)
Expand All @@ -79,9 +75,8 @@ define amdgpu_kernel void @global_load_tr_b128_half(ptr addrspace(1) %addr, ptr
; GFX12-SDAG-W64-NEXT: s_load_b128 s[0:3], s[0:1], 0x24
; GFX12-SDAG-W64-NEXT: v_mov_b32_e32 v2, 0
; GFX12-SDAG-W64-NEXT: s_wait_kmcnt 0x0
; GFX12-SDAG-W64-NEXT: global_load_tr_b128 v[0:1], v2, s[0:1] offset:32 th:TH_LOAD_NT
; GFX12-SDAG-W64-NEXT: global_load_tr_b128 v[0:1], v2, s[0:1] offset:32
; GFX12-SDAG-W64-NEXT: s_wait_loadcnt 0x0
; GFX12-SDAG-W64-NEXT: global_inv scope:SCOPE_SYS
; GFX12-SDAG-W64-NEXT: global_store_b64 v2, v[0:1], s[2:3]
; GFX12-SDAG-W64-NEXT: s_nop 0
; GFX12-SDAG-W64-NEXT: s_sendmsg sendmsg(MSG_DEALLOC_VGPRS)
Expand All @@ -92,9 +87,8 @@ define amdgpu_kernel void @global_load_tr_b128_half(ptr addrspace(1) %addr, ptr
; GFX12-GISEL-W64-NEXT: s_load_b128 s[0:3], s[0:1], 0x24
; GFX12-GISEL-W64-NEXT: v_mov_b32_e32 v2, 0
; GFX12-GISEL-W64-NEXT: s_wait_kmcnt 0x0
; GFX12-GISEL-W64-NEXT: global_load_tr_b128 v[0:1], v2, s[0:1] offset:32 th:TH_LOAD_NT
; GFX12-GISEL-W64-NEXT: global_load_tr_b128 v[0:1], v2, s[0:1] offset:32
; GFX12-GISEL-W64-NEXT: s_wait_loadcnt 0x0
; GFX12-GISEL-W64-NEXT: global_inv scope:SCOPE_SYS
; GFX12-GISEL-W64-NEXT: global_store_b64 v2, v[0:1], s[2:3]
; GFX12-GISEL-W64-NEXT: s_nop 0
; GFX12-GISEL-W64-NEXT: s_sendmsg sendmsg(MSG_DEALLOC_VGPRS)
Expand All @@ -112,9 +106,8 @@ define amdgpu_kernel void @global_load_tr_b128_bfloat(ptr addrspace(1) %addr, pt
; GFX12-SDAG-W64-NEXT: s_load_b128 s[0:3], s[0:1], 0x24
; GFX12-SDAG-W64-NEXT: v_mov_b32_e32 v2, 0
; GFX12-SDAG-W64-NEXT: s_wait_kmcnt 0x0
; GFX12-SDAG-W64-NEXT: global_load_tr_b128 v[0:1], v2, s[0:1] offset:32 th:TH_LOAD_NT
; GFX12-SDAG-W64-NEXT: global_load_tr_b128 v[0:1], v2, s[0:1] offset:32
; GFX12-SDAG-W64-NEXT: s_wait_loadcnt 0x0
; GFX12-SDAG-W64-NEXT: global_inv scope:SCOPE_SYS
; GFX12-SDAG-W64-NEXT: global_store_b64 v2, v[0:1], s[2:3]
; GFX12-SDAG-W64-NEXT: s_nop 0
; GFX12-SDAG-W64-NEXT: s_sendmsg sendmsg(MSG_DEALLOC_VGPRS)
Expand All @@ -125,9 +118,8 @@ define amdgpu_kernel void @global_load_tr_b128_bfloat(ptr addrspace(1) %addr, pt
; GFX12-GISEL-W64-NEXT: s_load_b128 s[0:3], s[0:1], 0x24
; GFX12-GISEL-W64-NEXT: v_mov_b32_e32 v2, 0
; GFX12-GISEL-W64-NEXT: s_wait_kmcnt 0x0
; GFX12-GISEL-W64-NEXT: global_load_tr_b128 v[0:1], v2, s[0:1] offset:32 th:TH_LOAD_NT
; GFX12-GISEL-W64-NEXT: global_load_tr_b128 v[0:1], v2, s[0:1] offset:32
; GFX12-GISEL-W64-NEXT: s_wait_loadcnt 0x0
; GFX12-GISEL-W64-NEXT: global_inv scope:SCOPE_SYS
; GFX12-GISEL-W64-NEXT: global_store_b64 v2, v[0:1], s[2:3]
; GFX12-GISEL-W64-NEXT: s_nop 0
; GFX12-GISEL-W64-NEXT: s_sendmsg sendmsg(MSG_DEALLOC_VGPRS)
Expand Down