[ci] Write test results to unique file names #113160
Conversation
|
Edit: the plugin requires docker which we cannot use in CI. Instead I've written a new script to make a summary - #113447. Having the results as artifacts means anyone can use something like https://github.com/lukejpreston/xunit-viewer on their own machine too. |
|
@llvm/pr-subscribers-clang Author: David Spickett (DavidSpickett) ChangesIn this patch I'm using a new lit option so that the pipeline writes many results files, one for each time lit is run: (I added this to lit recently) Now if I run the Linux build: I get multiple test result files. In my case some tests fail so runtimes aren't checked, but all projects are so there is 1 file for lldb and one for lld: This change just collects the XML files as artifacts. Once I know that's working, I can setup a test reporting plugin to build a summary from them. Full diff: https://github.com/llvm/llvm-project/pull/113160.diff 5 Files Affected:
diff --git a/.ci/generate-buildkite-pipeline-premerge b/.ci/generate-buildkite-pipeline-premerge
index 7676ff716c4185..e52133751f09b1 100755
--- a/.ci/generate-buildkite-pipeline-premerge
+++ b/.ci/generate-buildkite-pipeline-premerge
@@ -272,7 +272,7 @@ if [[ "${linux_projects}" != "" ]]; then
artifact_paths:
- 'artifacts/**/*'
- '*_result.json'
- - 'build/test-results.xml'
+ - 'build/test-results*.xml'
agents: ${LINUX_AGENTS}
retry:
automatic:
@@ -295,7 +295,7 @@ if [[ "${windows_projects}" != "" ]]; then
artifact_paths:
- 'artifacts/**/*'
- '*_result.json'
- - 'build/test-results.xml'
+ - 'build/test-results*.xml'
agents: ${WINDOWS_AGENTS}
retry:
automatic:
diff --git a/.ci/monolithic-linux.sh b/.ci/monolithic-linux.sh
index b78dc59432b65c..17ea51c08fafd3 100755
--- a/.ci/monolithic-linux.sh
+++ b/.ci/monolithic-linux.sh
@@ -37,6 +37,8 @@ trap show-stats EXIT
projects="${1}"
targets="${2}"
+lit_args="-v --xunit-xml-output ${BUILD_DIR}/test-results.xml --use-unique-output-file-name --timeout=1200 --time-tests"
+
echo "--- cmake"
pip install -q -r "${MONOREPO_ROOT}"/mlir/python/requirements.txt
pip install -q -r "${MONOREPO_ROOT}"/lldb/test/requirements.txt
@@ -47,7 +49,7 @@ cmake -S "${MONOREPO_ROOT}"/llvm -B "${BUILD_DIR}" \
-D LLVM_ENABLE_ASSERTIONS=ON \
-D LLVM_BUILD_EXAMPLES=ON \
-D COMPILER_RT_BUILD_LIBFUZZER=OFF \
- -D LLVM_LIT_ARGS="-v --xunit-xml-output ${BUILD_DIR}/test-results.xml --timeout=1200 --time-tests" \
+ -D LLVM_LIT_ARGS="${lit_args}" \
-D LLVM_ENABLE_LLD=ON \
-D CMAKE_CXX_FLAGS=-gmlt \
-D LLVM_CCACHE_BUILD=ON \
@@ -87,7 +89,8 @@ if [[ "${runtimes}" != "" ]]; then
-D CMAKE_BUILD_TYPE=RelWithDebInfo \
-D CMAKE_INSTALL_PREFIX="${INSTALL_DIR}" \
-D LIBCXX_TEST_PARAMS="std=c++03" \
- -D LIBCXXABI_TEST_PARAMS="std=c++03"
+ -D LIBCXXABI_TEST_PARAMS="std=c++03" \
+ -D LLVM_LIT_ARGS="${lit_args}"
echo "--- ninja runtimes C++03"
@@ -104,7 +107,8 @@ if [[ "${runtimes}" != "" ]]; then
-D CMAKE_BUILD_TYPE=RelWithDebInfo \
-D CMAKE_INSTALL_PREFIX="${INSTALL_DIR}" \
-D LIBCXX_TEST_PARAMS="std=c++26" \
- -D LIBCXXABI_TEST_PARAMS="std=c++26"
+ -D LIBCXXABI_TEST_PARAMS="std=c++26" \
+ -D LLVM_LIT_ARGS="${lit_args}"
echo "--- ninja runtimes C++26"
@@ -121,7 +125,8 @@ if [[ "${runtimes}" != "" ]]; then
-D CMAKE_BUILD_TYPE=RelWithDebInfo \
-D CMAKE_INSTALL_PREFIX="${INSTALL_DIR}" \
-D LIBCXX_TEST_PARAMS="enable_modules=clang" \
- -D LIBCXXABI_TEST_PARAMS="enable_modules=clang"
+ -D LIBCXXABI_TEST_PARAMS="enable_modules=clang" \
+ -D LLVM_LIT_ARGS="${lit_args}"
echo "--- ninja runtimes clang modules"
diff --git a/.ci/monolithic-windows.sh b/.ci/monolithic-windows.sh
index 91e719c52d4363..9ec44c22442d06 100755
--- a/.ci/monolithic-windows.sh
+++ b/.ci/monolithic-windows.sh
@@ -53,7 +53,7 @@ cmake -S "${MONOREPO_ROOT}"/llvm -B "${BUILD_DIR}" \
-D LLVM_ENABLE_ASSERTIONS=ON \
-D LLVM_BUILD_EXAMPLES=ON \
-D COMPILER_RT_BUILD_LIBFUZZER=OFF \
- -D LLVM_LIT_ARGS="-v --xunit-xml-output ${BUILD_DIR}/test-results.xml --timeout=1200 --time-tests" \
+ -D LLVM_LIT_ARGS="-v --xunit-xml-output ${BUILD_DIR}/test-results.xml --use-unique-output-file-name --timeout=1200 --time-tests" \
-D COMPILER_RT_BUILD_ORC=OFF \
-D CMAKE_C_COMPILER_LAUNCHER=sccache \
-D CMAKE_CXX_COMPILER_LAUNCHER=sccache \
diff --git a/clang/README.md b/clang/README.md
index b98182d8a3f684..7cc2136be13488 100644
--- a/clang/README.md
+++ b/clang/README.md
@@ -23,3 +23,5 @@ If you're interested in more (including how to build Clang) it is best to read t
* If you find a bug in Clang, please file it in the LLVM bug tracker:
https://github.com/llvm/llvm-project/issues
+
+Test line.
\ No newline at end of file
diff --git a/llvm/README.txt b/llvm/README.txt
index b9b71a3b6daff1..d5e0796876bd13 100644
--- a/llvm/README.txt
+++ b/llvm/README.txt
@@ -15,3 +15,5 @@ documentation setup.
If you are writing a package for LLVM, see docs/Packaging.rst for our
suggestions.
+
+Test line.
\ No newline at end of file
|
|
You can see the artifacts for Linux and Windows here - https://buildkite.com/llvm-project/github-pull-requests/builds/111819#0192afb2-3217-4a3d-9366-a28c33440288. I couldn't think of a logical way to make lit label them by the paths or testsuites they run (because they can be many different combinations), and the only way to truly know what you'd run is to go back to running the check- targets in a bash for loop. So the names are randomly changed using this new option. I'm surprised buildkite doesn't have a download all artifacts option, but we can add a zipped results file if it's needed. For reporting test status in the job, separate files is fine. |
In this patch I'm using a new lit option so that the pipeline writes many results files, one for each time lit is run: ``` --use-unique-output-file-name When enabled, lit will add a unique element to the output file name, before the extension. For example "results.xml" will become "results.<something>.xml". The "<something>" is not ordered in any way and is chosen so that existing files are not overwritten. [Default: Off] ``` (I added this to lit recently) Now if I run the Linux build: $ bash ./.ci/monolithic-linux.sh "clang;lldb;lld" "check-lldb-shell check-lld" "libcxx;libcxxabi" "check-libcxx check-libcxxabi" I get multiple test result files. In my case some tests fail so runtimes aren't checked, but all projects are so there is 1 file for lldb and one for lld: $ ls build/*.xml build/test-results.klc82utf.xml build/test-results.majylh73.xml This change just collects the XML files as artifacts. Once I know that's working, I can setup a test reporting plugin to build a summary from them.
DavidSpickett
force-pushed
the
ci-xml
branch
from
4aaa3a1 to
cee1011
Compare
In this patch I'm using a new lit option so that the pipeline writes many results files, one for each time lit is run:
(I added this to lit recently)
Alternatives were considered:
And after feedback I decided that using an option to lit itself is the cleanest way to go. It can be removed when we no longer need it.
If I run the Linux build after this change:
I get multiple test result files. In my case some tests fail so runtimes aren't checked, but all projects are so there is 1 file for lldb and one for lld:
This change just collects the XML files as artifacts. Once I know that's working, I can set up test reporting to make a summary of them.