release/19.x: [AArch64] Don't replace dst of SWP instructions with (X|W)ZR (#102139) #102316
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
@statham-arm What do you think about merging this PR to the release branch? |
|
@llvm/pr-subscribers-backend-aarch64 Author: None (llvmbot) ChangesBackport beb37e2 Requested by: @pratlucas Full diff: https://github.com/llvm/llvm-project/pull/102316.diff 2 Files Affected:
diff --git a/llvm/lib/Target/AArch64/AArch64DeadRegisterDefinitionsPass.cpp b/llvm/lib/Target/AArch64/AArch64DeadRegisterDefinitionsPass.cpp
index 2bc14f9821e63..161cf24dd4037 100644
--- a/llvm/lib/Target/AArch64/AArch64DeadRegisterDefinitionsPass.cpp
+++ b/llvm/lib/Target/AArch64/AArch64DeadRegisterDefinitionsPass.cpp
@@ -108,6 +108,10 @@ static bool atomicReadDroppedOnZero(unsigned Opcode) {
case AArch64::LDUMINW: case AArch64::LDUMINX:
case AArch64::LDUMINLB: case AArch64::LDUMINLH:
case AArch64::LDUMINLW: case AArch64::LDUMINLX:
+ case AArch64::SWPB: case AArch64::SWPH:
+ case AArch64::SWPW: case AArch64::SWPX:
+ case AArch64::SWPLB: case AArch64::SWPLH:
+ case AArch64::SWPLW: case AArch64::SWPLX:
return true;
}
return false;
diff --git a/llvm/test/CodeGen/AArch64/Atomics/aarch64-atomic-exchange-fence.ll b/llvm/test/CodeGen/AArch64/Atomics/aarch64-atomic-exchange-fence.ll
new file mode 100644
index 0000000000000..2adbc709d238d
--- /dev/null
+++ b/llvm/test/CodeGen/AArch64/Atomics/aarch64-atomic-exchange-fence.ll
@@ -0,0 +1,64 @@
+; RUN: llc %s -o - -verify-machineinstrs -mtriple=aarch64 -mattr=+lse -O0 | FileCheck %s
+; RUN: llc %s -o - -verify-machineinstrs -mtriple=aarch64 -mattr=+lse -O1 | FileCheck %s
+
+; When their destination register is WZR/ZZR, SWP operations are not regarded as
+; a read for the purpose of a DMB.LD in the AArch64 memory model.
+; This test ensures that the AArch64DeadRegisterDefinitions pass does not
+; replace the desitnation register of SWP instructions with the zero register
+; when the read value is unused.
+
+define dso_local i32 @atomic_exchange_monotonic(ptr %ptr, ptr %ptr2, i32 %value) {
+; CHECK-LABEL: atomic_exchange_monotonic:
+; CHECK: // %bb.0:
+; CHECK-NEXT: swp
+; CHECK-NOT: wzr
+; CHECK-NEXT: dmb ishld
+; CHECK-NEXT: ldr w0, [x1]
+; CHECK-NEXT: ret
+ %r0 = atomicrmw xchg ptr %ptr, i32 %value monotonic
+ fence acquire
+ %r1 = load atomic i32, ptr %ptr2 monotonic, align 4
+ ret i32 %r1
+}
+
+define dso_local i32 @atomic_exchange_acquire(ptr %ptr, ptr %ptr2, i32 %value) {
+; CHECK-LABEL: atomic_exchange_acquire:
+; CHECK: // %bb.0:
+; CHECK-NEXT: swpa
+; CHECK-NOT: wzr
+; CHECK-NEXT: dmb ishld
+; CHECK-NEXT: ldr w0, [x1]
+; CHECK-NEXT: ret
+ %r0 = atomicrmw xchg ptr %ptr, i32 %value acquire
+ fence acquire
+ %r1 = load atomic i32, ptr %ptr2 monotonic, align 4
+ ret i32 %r1
+}
+
+define dso_local i32 @atomic_exchange_release(ptr %ptr, ptr %ptr2, i32 %value) {
+; CHECK-LABEL: atomic_exchange_release:
+; CHECK: // %bb.0:
+; CHECK-NEXT: swpl
+; CHECK-NOT: wzr
+; CHECK-NEXT: dmb ishld
+; CHECK-NEXT: ldr w0, [x1]
+; CHECK-NEXT: ret
+ %r0 = atomicrmw xchg ptr %ptr, i32 %value release
+ fence acquire
+ %r1 = load atomic i32, ptr %ptr2 monotonic, align 4
+ ret i32 %r1
+}
+
+define dso_local i32 @atomic_exchange_acquire_release(ptr %ptr, ptr %ptr2, i32 %value) {
+; CHECK-LABEL: atomic_exchange_acquire_release:
+; CHECK: // %bb.0:
+; CHECK-NEXT: swpal
+; CHECK-NOT: wzr
+; CHECK-NEXT: dmb ishld
+; CHECK-NEXT: ldr w0, [x1]
+; CHECK-NEXT: ret
+ %r0 = atomicrmw xchg ptr %ptr, i32 %value acq_rel
+ fence acquire
+ %r1 = load atomic i32, ptr %ptr2 monotonic, align 4
+ ret i32 %r1
+}
|
statham-arm
approved these changes
Aug 7, 2024
There was a problem hiding this comment.
Seems sensible to me. It's fixing a genuine codegen fault (a subtle one, but of course that makes it worse – harder to spot when it occurs!). And it's a small safe change that disables one very small case of a conceptually simple optimisation, unlikely to introduce other bugs.
|
I second Simon, looks good to me |
…2139) This change updates the AArch64DeadRegisterDefinition pass to ensure it does not replace the destination register of a SWP instruction with the zero register when its value is unused. This is necessary to ensure that the ordering of such instructions in relation to DMB.LD barries adheres to the definitions of the AArch64 Memory Model. The memory model states the following (ARMARM version DDI 0487K.a §B2.3.7): ``` Barrier-ordered-before An effect E1 is Barrier-ordered-before an effect E2 if one of the following applies: [...] * All of the following apply: - E1 is a Memory Read effect. - E1 is generated by an instruction whose destination register is not WZR or XZR. - E1 appears in program order before E3. - E3 is either a DMB LD effect or a DSB LD effect. - E3 appears in program order before E2. ``` Prior to this change, by replacing the destination register of such SWP instruction with WZR/XZR, the ordering relation described above was incorrectly removed from the generated code. The new behaviour is ensured in this patch by adding the relevant `SWP[L](B|H|W|X)` instructions to list in the `atomicReadDroppedOnZero` predicate, which already covered the `LD<Op>` instructions that are subject to the same effect. Fixes llvm#68428. (cherry picked from commit beb37e2)
|
@pratlucas (or anyone else). If you would like to add a note about this fix in the release notes (completely optional). Please reply to this comment with a one or two sentence description of the fix. When you are done, please add the release:note label to this PR. |
|
Prevents a concurrency-related compiler bug (a reordering bug introduced by LLVM) that arises when optimisations rewrite the destination register of SWP instructions to be the zero register when compiling an atomic exchange operation. For more information on this bug and how it was found, please see: https://lukegeeson.com/publications/2024-03-05-CGO/ |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Backport beb37e2
Requested by: @pratlucas