Skip to content

[build-script] Tests should fail if one of test commands failed. #2217

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 1 commit into from
Apr 28, 2016

Conversation

rintaro
Copy link
Member

@rintaro rintaro commented Apr 16, 2016

What's in this pull request?

Test script:

false
true

should be considered as failed, and true should never be executed.

Use -e option for test script execution: sh -e -x -c "script.."


Before merging this pull request to apple/swift repository:

  • Test pull request on Swift continuous integration.

@gribozavr
Copy link
Contributor

@swift-ci Please test and merge

@rintaro
Copy link
Member Author

rintaro commented Apr 16, 2016

Good! It seems, this change unveiled a hidden problem 😁
https://ci.swift.org/job/swift-PR-osx/908/console

+ /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/clang++ -DGTEST_HAS_RTTI=0 -D_DEBUG -D__STDC_CONSTANT_MACROS -D__STDC_FORMAT_MACROS -D__STDC_LIMIT_MACROS -Itools/swift-reflection-test -I/Users/buildnode/jenkins/workspace/swift-PR-osx/swift/tools/swift-reflection-test -I/Users/buildnode/jenkins/workspace/swift-PR-osx/swift/include -Iinclude -I/Users/buildnode/jenkins/workspace/swift-PR-osx/buildbot_incremental/llvm-macosx-x86_64/include -I/Users/buildnode/jenkins/workspace/swift-PR-osx/llvm/include -I/Users/buildnode/jenkins/workspace/swift-PR-osx/buildbot_incremental/llvm-macosx-x86_64/tools/clang/include -I/Users/buildnode/jenkins/workspace/swift-PR-osx/llvm/tools/clang/include -I/Users/buildnode/jenkins/workspace/swift-PR-osx/cmark/src -I/Users/buildnode/jenkins/workspace/swift-PR-osx/buildbot_incremental/cmark-macosx-x86_64/src -fno-stack-protector -stdlib=libc++ -fPIC -fvisibility-inlines-hidden -Wall -W -Wno-unused-parameter -Wwrite-strings -Wcast-qual -Wmissing-field-initializers -Wcovered-switch-default -Wnon-virtual-dtor -Wdelete-non-virtual-dtor -Werror=date-time -std=c++11 -fcolor-diagnostics -Wdocumentation -Wimplicit-fallthrough -Wunreachable-code -Woverloaded-virtual -O3 -isysroot /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.11.sdk -UNDEBUG -fno-exceptions -fno-rtti -target i386-apple-ios7.0 -isysroot /Applications/Xcode.app/Contents/Developer/Platforms/iPhoneSimulator.platform/Developer/SDKs/iPhoneSimulator9.3.sdk -arch i386 -F /Applications/Xcode.app/Contents/Developer/Platforms/iPhoneSimulator.platform/Developer/SDKs/iPhoneSimulator9.3.sdk/../../../Developer/Library/Frameworks -mios-simulator-version-min=7.0 -O2 -momit-leaf-frame-pointer -g0 -UNDEBUG -MMD -MT tools/swift-reflection-test/CMakeFiles/swift-reflection-test-iphonesimulator-i386.dir/swift-reflection-test.cpp.o -MF tools/swift-reflection-test/CMakeFiles/swift-reflection-test-iphonesimulator-i386.dir/swift-reflection-test.cpp.o.d -o tools/swift-reflection-test/CMakeFiles/swift-reflection-test-iphonesimulator-i386.dir/swift-reflection-test.cpp.o -c /Users/buildnode/jenkins/workspace/swift-PR-osx/swift/tools/swift-reflection-test/swift-reflection-test.cpp
�[0;1;31merror: �[0munable to open output file 'tools/swift-reflection-test/CMakeFiles/swift-reflection-test-iphonesimulator-i386.dir/swift-reflection-test.cpp.o': 'No such file or directory'�[0m
1 error generated.
*** Failed while running tests for swift (check-swift-all-iphonesimulator-i386)
/Users/buildnode/jenkins/workspace/swift-PR-osx/swift/utils/build-script: command terminated with a non-zero exit status 1, aborting
/Users/buildnode/jenkins/workspace/swift-PR-osx/swift/utils/build-script: using preset 'buildbot_all_platforms,tools=RA,stdlib=RA', which expands to /Users/buildnode/jenkins/workspace/swift-PR-osx/swift/utils/build-script --test --validation-test --ios --tvos --watchos --build-subdir=buildbot_incremental --release --assertions --llbuild --swiftpm -- --reconfigure --verbose-build --build-ninja --build-swift-stdlib-unittest-extra --compiler-vendor=apple --skip-test-ios-host --skip-test-tvos-host --skip-test-watchos-host
/Users/buildnode/jenkins/workspace/swift-PR-osx/swift/utils/build-script: command terminated with a non-zero exit status 1, aborting
Build step 'Execute shell' marked build as failure

Failed on compiling swift-reflection-test for iphonesimulator-i386
@bitjammer Could you confirm?

@rintaro
Copy link
Member Author

rintaro commented Apr 28, 2016

I believe the problem is resolved in #2327.
Could someone please re-run test and merge this?

@gribozavr
Copy link
Contributor

@swift-ci Please test and merge

@swift-ci swift-ci merged commit d86ca51 into swiftlang:master Apr 28, 2016
@gribozavr
Copy link
Contributor

@rintaro Could you take a look at the buildbot failure?

https://ci.swift.org/job/oss-swift-incremental-RA-linux-ubuntu-14_04-long-test/60/console

clang: �[0;1;31merror: �[0mno such file or directory: 'lib/swift/linux/x86_64/swift_begin.o'�[0m
clang: �[0;1;31merror: �[0mno such file or directory: 'stdlib/private/StdlibCollectionUnittest/linux/x86_64/StdlibCollectionUnittest.o'�[0m
clang: �[0;1;31merror: �[0mno such file or directory: 'lib/swift/linux/x86_64/swift_end.o'�[0m
clang: �[0;1;31merror: �[0mno such file or directory: 'lib/swift/linux/x86_64/libswiftStdlibUnittest.so'�[0m
clang: �[0;1;31merror: �[0mno such file or directory: 'lib/swift/linux/x86_64/libswiftCore.so'�[0m

@rintaro
Copy link
Member Author

rintaro commented Apr 29, 2016

I'm looking into it.

@rintaro
Copy link
Member Author

rintaro commented Apr 29, 2016

First of all, some build commands are assuming ${PWD} is the build directory.
I think,

pushd ${build_dir}
sh -e -x -c "$("${build_cmd[@]}" -n -v ${target} | sed -e 's/[^]]*] //')"
popd

should resolve the current problem...

But, I think, the root cause is that the test dependencies are not fully built before tests execution.

@gribozavr
Copy link
Contributor

I agree, that command shouldn't be building anything, looks like we are missing some dependencies.

@rintaro
Copy link
Member Author

rintaro commented Apr 29, 2016

/usr/bin/cmake --build /home/buildnode/jenkins/workspace/oss-swift-incremental-RA-linux-ubuntu-14_04-long-test/Ninja-ReleaseAssert/swift-linux-x86_64 \
 -- -j48 all swift-test-stdlib-linux-x86_64

Long test implies validation-test. the stdlib target should be swift-stdlib-linux-x86_64?

@gribozavr
Copy link
Contributor

Yes, I think it should be the full library.

@rintaro
Copy link
Member Author

rintaro commented Apr 29, 2016

Testing:

--- a/utils/build-script-impl
+++ b/utils/build-script-impl
@@ -1083,7 +1083,7 @@ for deployment_target in "${STDLIB_DEPLOYMENT_TARGETS[@]}"; do
         if [[ "${BUILD_SWIFT_STDLIB_UNITTEST_EXTRA}" == "1" ]] ; then
             SWIFT_STDLIB_TARGETS+=("swift-stdlib-${deployment_target}")
         else
-            if [[ "${VALIDATION_TEST}" == "1" ]] ; then
+            if [[ "${VALIDATION_TEST}" == "1" || "${LONG_TEST}" == "1" ]] ; then
                 SWIFT_STDLIB_TARGETS+=("swift-stdlib-${deployment_target}")
             else
                 SWIFT_STDLIB_TARGETS+=("swift-test-stdlib-${deployment_target}")

@rintaro
Copy link
Member Author

rintaro commented Apr 29, 2016

Passed locally. Could you try apply that and wait the bot?

$ utils/build-script --preset="buildbot_incremental_linux,long_test"

...

--- Running tests for swift ---
--- check-swift-only_long-linux-x86_64 ---
+ cd /home/rintaro/Documents/swift-oss/build/Ninja-ReleaseAssert/swift-linux-x86_64/stdlib/public/SwiftShims
+ /usr/bin/cmake -E make_directory /home/rintaro/Documents/swift-oss/build/Ninja-ReleaseAssert/swift-linux-x86_64/./lib/swift
+ /usr/bin/cmake -E create_symlink /home/rintaro/Documents/swift-oss/build/Ninja-ReleaseAssert/llvm-linux-x86_64/lib/clang/3.9.0 /home/rintaro/Documents/swift-oss/build/Ninja-ReleaseAssert/swift-linux-x86_64/./lib/swift/clang
+ /usr/bin/cmake -E make_directory /home/rintaro/Documents/swift-oss/build/Ninja-ReleaseAssert/swift-linux-x86_64/./lib/swift/install-tmp/install-tmp
+ /usr/bin/cmake -E remove /home/rintaro/Documents/swift-oss/build/Ninja-ReleaseAssert/swift-linux-x86_64/./lib/swift/install-tmp/install-tmp/clang
+ /usr/bin/cmake -E create_symlink ../clang/3.9.0 /home/rintaro/Documents/swift-oss/build/Ninja-ReleaseAssert/swift-linux-x86_64/./lib/swift/install-tmp/install-tmp/clang
+ cd /home/rintaro/Documents/swift-oss/build/Ninja-ReleaseAssert/swift-linux-x86_64/test
+ /usr/bin/cmake -E remove_directory /home/rintaro/Documents/swift-oss/build/Ninja-ReleaseAssert/swift-linux-x86_64/./swift-test-results/x86_64-unknown-linux-gnu
+ /usr/bin/cmake -E make_directory /home/rintaro/Documents/swift-oss/build/Ninja-ReleaseAssert/swift-linux-x86_64/./swift-test-results/x86_64-unknown-linux-gnu
+ /usr/bin/python /home/rintaro/Documents/swift-oss/llvm/utils/lit/lit.py -sv --xunit-xml-output=/home/rintaro/Documents/swift-oss/build/Ninja-ReleaseAssert/swift-linux-x86_64/./swift-test-results/x86_64-unknown-linux-gnu/lit-tests.xml --param run_only_tests=all --param run_only_tests=long_tests /home/rintaro/Documents/swift-oss/build/Ninja-ReleaseAssert/swift-linux-x86_64/test-linux-x86_64 /home/rintaro/Documents/swift-oss/build/Ninja-ReleaseAssert/swift-linux-x86_64/test/../validation-test-linux-x86_64
lit.py: lit.cfg:259: note: using swift: /home/rintaro/Documents/swift-oss/build/Ninja-ReleaseAssert/swift-linux-x86_64/bin/swift
lit.py: lit.cfg:259: note: using swiftc: /home/rintaro/Documents/swift-oss/build/Ninja-ReleaseAssert/swift-linux-x86_64/bin/swiftc
...

@gribozavr
Copy link
Contributor

@rintaro Mind submitting a PR? I'll accept immediately.

@rintaro
Copy link
Member Author

rintaro commented Apr 29, 2016

Sure.

@rintaro rintaro deleted the build-script-tests-execution branch May 3, 2016 04:20
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants