Skip to content

Commit ebda2d5

Browse files
Pierre-vhamd-aakash
Pierre-vh
authored and
amd-aakash
committed
[AMDGPU][SIMemoryLegalizer] Fix order of GL0/1_INV on GFX10/11 (llvm#81450)
Fixes SWDEV-443292 Change-Id: I2eeb68b9d82a560683a96efb0207e82a93de901a
1 parent de05ddb commit ebda2d5

16 files changed

+1503
-1501
lines changed

‎llvm/docs/AMDGPUUsage.rst

Lines changed: 16 additions & 16 deletions
Original file line numberDiff line numberDiff line change
@@ -11845,8 +11845,8 @@ table :ref:`amdgpu-amdhsa-memory-model-code-sequences-gfx10-gfx11-table`.
1184511845
before invalidating
1184611846
the caches.
1184711847

11848-
3. buffer_gl0_inv;
11849-
buffer_gl1_inv
11848+
3. buffer_gl1_inv;
11849+
buffer_gl0_inv
1185011850

1185111851
- Must happen before
1185211852
any following
@@ -11875,8 +11875,8 @@ table :ref:`amdgpu-amdhsa-memory-model-code-sequences-gfx10-gfx11-table`.
1187511875
before invalidating
1187611876
the caches.
1187711877

11878-
3. buffer_gl0_inv;
11879-
buffer_gl1_inv
11878+
3. buffer_gl1_inv;
11879+
buffer_gl0_inv
1188011880

1188111881
- Must happen before
1188211882
any following
@@ -11982,8 +11982,8 @@ table :ref:`amdgpu-amdhsa-memory-model-code-sequences-gfx10-gfx11-table`.
1198211982
invalidating the
1198311983
caches.
1198411984

11985-
3. buffer_gl0_inv;
11986-
buffer_gl1_inv
11985+
3. buffer_gl1_inv;
11986+
buffer_gl0_inv
1198711987

1198811988
- Must happen before
1198911989
any following
@@ -12013,8 +12013,8 @@ table :ref:`amdgpu-amdhsa-memory-model-code-sequences-gfx10-gfx11-table`.
1201312013
invalidating the
1201412014
caches.
1201512015

12016-
3. buffer_gl0_inv;
12017-
buffer_gl1_inv
12016+
3. buffer_gl1_inv;
12017+
buffer_gl0_inv
1201812018

1201912019
- Must happen before
1202012020
any following
@@ -12209,8 +12209,8 @@ table :ref:`amdgpu-amdhsa-memory-model-code-sequences-gfx10-gfx11-table`.
1220912209
the
1221012210
fence-paired-atomic.
1221112211

12212-
2. buffer_gl0_inv;
12213-
buffer_gl1_inv
12212+
2. buffer_gl1_inv;
12213+
buffer_gl0_inv
1221412214

1221512215
- Must happen before any
1221612216
following global/generic
@@ -12923,8 +12923,8 @@ table :ref:`amdgpu-amdhsa-memory-model-code-sequences-gfx10-gfx11-table`.
1292312923
invalidating the
1292412924
caches.
1292512925

12926-
4. buffer_gl0_inv;
12927-
buffer_gl1_inv
12926+
4. buffer_gl1_inv;
12927+
buffer_gl0_inv
1292812928

1292912929
- Must happen before
1293012930
any following
@@ -12998,8 +12998,8 @@ table :ref:`amdgpu-amdhsa-memory-model-code-sequences-gfx10-gfx11-table`.
1299812998
invalidating the
1299912999
caches.
1300013000

13001-
4. buffer_gl0_inv;
13002-
buffer_gl1_inv
13001+
4. buffer_gl1_inv;
13002+
buffer_gl0_inv
1300313003

1300413004
- Must happen before
1300513005
any following
@@ -13226,8 +13226,8 @@ table :ref:`amdgpu-amdhsa-memory-model-code-sequences-gfx10-gfx11-table`.
1322613226
requirements of
1322713227
release.
1322813228

13229-
2. buffer_gl0_inv;
13230-
buffer_gl1_inv
13229+
2. buffer_gl1_inv;
13230+
buffer_gl0_inv
1323113231

1323213232
- Must happen before
1323313233
any following

‎llvm/lib/Target/AMDGPU/SIMemoryLegalizer.cpp

Lines changed: 4 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1999,8 +1999,11 @@ bool SIGfx10CacheControl::insertAcquire(MachineBasicBlock::iterator &MI,
19991999
switch (Scope) {
20002000
case SIAtomicScope::SYSTEM:
20012001
case SIAtomicScope::AGENT:
2002-
BuildMI(MBB, MI, DL, TII->get(AMDGPU::BUFFER_GL0_INV));
2002+
// The order of invalidates matter here. We must invalidate "outer in"
2003+
// so L1 -> L0 to avoid L0 pulling in stale data from L1 when it is
2004+
// invalidated.
20032005
BuildMI(MBB, MI, DL, TII->get(AMDGPU::BUFFER_GL1_INV));
2006+
BuildMI(MBB, MI, DL, TII->get(AMDGPU::BUFFER_GL0_INV));
20042007
Changed = true;
20052008
break;
20062009
case SIAtomicScope::WORKGROUP:

0 commit comments

Comments
 (0)