[libc++][Android] Enable Android testing in BuildKite CI

[rprichard]

Enable testing for two NDK configurations:

• android-ndk-21-def-x86
• android-ndk-33-goog-x86_64

Fixes: #69270

[llvmbot]

@llvm/pr-subscribers-libcxx

Author: Ryan Prichard (rprichard)

Changes

Enable testing for two NDK configurations:

• android-ndk-21-def-x86
• android-ndk-33-goog-x86_64

Fixes: #69270

---

Full diff: https://github.com/llvm/llvm-project/pull/69275.diff

1 Files Affected:

• (modified) libcxx/utils/ci/buildkite-pipeline.yml (+30)

diff --git a/libcxx/utils/ci/buildkite-pipeline.yml b/libcxx/utils/ci/buildkite-pipeline.yml
index 7a125d16af59493..50b778739902fb4 100644
--- a/libcxx/utils/ci/buildkite-pipeline.yml
+++ b/libcxx/utils/ci/buildkite-pipeline.yml
@@ -1135,3 +1135,33 @@ steps:
           - exit_status: -1  # Agent was lost
             limit: 2
       timeout_in_minutes: 120
+
+  - group: ":android: Android"
+    steps:
+    - label: "Android 5.0, x86 NDK"
+      command: "libcxx/utils/ci/run-buildbot android-ndk-21-def-x86"
+      artifact_paths:
+        - "**/test-results.xml"
+        - "**/*.abilist"
+      agents:
+        queue: "libcxx-builders"
+        os: "android"
+      retry:
+        automatic:
+          - exit_status: -1  # Agent was lost
+            limit: 2
+      timeout_in_minutes: 120
+
+    - label: "Android 13, x86_64 NDK"
+      command: "libcxx/utils/ci/run-buildbot android-ndk-33-goog-x86_64"
+      artifact_paths:
+        - "**/test-results.xml"
+        - "**/*.abilist"
+      agents:
+        queue: "libcxx-builders"
+        os: "android"
+      retry:
+        automatic:
+          - exit_status: -1  # Agent was lost
+            limit: 2
+      timeout_in_minutes: 120

[EricWF]

woo-hoo! I think this is the only change we need to land!

I will ask one thing though. I'm a bit concerned that we don't yet have the capacity to handle this change as a pre-commit check. Could we initially turn it on at some other schedule, something like post-commit, hourly, or rolling head, so we can gain some confidence. Then we'll upgrade it after a week or so?

[ldionne]

We should add the support mention in the documentation in this patch.

[EricWF]

We accidentally already did that in another change.

We can go ahead and revert that documentation change and move it here, if you would like? I suggested @rprichard didn't do that yesterday because I believed this follow up would land shortly.

[rprichard]

Oh, that documentation change was already reverted in #69660.

[ldionne]

woo-hoo! I think this is the only change we need to land!

I will ask one thing though. I'm a bit concerned that we don't yet have the capacity to handle this change as a pre-commit check. Could we initially turn it on at some other schedule, something like post-commit, hourly, or rolling head, so we can gain some confidence. Then we'll upgrade it after a week or so?

IMO we should do like we do for all the other jobs and run it as pre-commit. We should tackle that issue separately like we've been discussing, but we shouldn't try to mash up improving our CI latency with this new platform support effort. Concretely, the libc++ policy is to require pre-commit CI for all the supported platforms, so we should just stick to it until we have the technical means to do several layers of testing and we have updated our policy to account for that.

[EricWF]

woo-hoo! I think this is the only change we need to land!
I will ask one thing though. I'm a bit concerned that we don't yet have the capacity to handle this change as a pre-commit check. Could we initially turn it on at some other schedule, something like post-commit, hourly, or rolling head, so we can gain some confidence. Then we'll upgrade it after a week or so?

IMO we should do like we do for all the other jobs and run it as pre-commit. We should tackle that issue separately like we've been discussing, but we shouldn't try to mash up improving our CI latency with this new platform support effort. Concretely, the libc++ policy is to require pre-commit CI for all the supported platforms, so we should just stick to it until we have the technical means to do several layers of testing and we have updated our policy to account for that.

I'm suggesting we "soft launch" or canary the new android builders in a way that won't affect the current pre-commit CI; that way, if a problem were to immediately arise, it wouldn't affect all of the ongoing libc++ work.

The "canary" step would last at most a week. But before promoting it, I want to see the bot successfully complete a few builds.

If there's a way to add this check to the pre-commit in a way that, were it to break tonight, would not block other developers, then let's do that. Otherwise, let's find another way to canary.

@ldionne WDYT?

[ldionne]

Oh, that's what you mean. Yeah, I think that makes sense. It is possible to mark Buildkite jobs as "soft fail" -- they will be reported as failures but they shouldn't mark the whole build as failed: https://buildkite.com/changelog/56-command-steps-can-now-be-made-to-soft-fail

[EricWF]

Perfect. Let's do that. The jobs appear to be passing, but giving them a few days of "soft-failing" sounds like the best approach.

[EricWF]

@rprichard ping me when you've updated this change. Then it LGTM.

[EricWF]

LGTM. I don't think we need to wait for the bots to come back (they already have on the same change once before).

Let's merge it!