Skip to content

[llvm][docs] Expand HowToAddABuilder with guidance on testing locally #115024

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 7 commits into from
Nov 12, 2024

Conversation

asb
Copy link
Contributor

@asb asb commented Nov 5, 2024

Once llvm/llvm-zorg#289 and llvm/llvm-zorg#293 land, it's quite reasonable to ask people to test their builder configurations locally. This patch adds documentation on how to do so.

I think review at this stage is useful, but of course if there's more review feedback on llvm/llvm-zorg#289 it's possible some details may change. This won't be committed until those llvm-zorg PRs land of course.

Once <llvm/llvm-zorg#289> and
<llvm/llvm-zorg#293> land, it's quite reasonable
to ask people to test their builder configurations locally. This patch
adds documentation on how to do so.

I think review at this stage is useful, but of course if there's more
review feedback on <llvm/llvm-zorg#289> it's
possible some details may change. This won't be committed until those
llvm-zorg PRs land of course.
@DavidSpickett
Copy link
Collaborator

This is beside the point of this change but I wonder if this document could do with a note at the start to the effect that the "master" terminology is what the Buildbot project uses and is therefore used here in that context (see buildbot/buildbot#5382).

Since we have precedent for not using the term ourselves, when we moved the master branch to main.

asb added a commit to llvm/llvm-zorg that referenced this pull request Nov 8, 2024
This makes it (relatively) easy to test a builder setup locally. The
buildmaster and the web interface should be bound only to local
interfaces for security reasons (you can use ssh port forwarding if
wanting to run on a server).

See llvm/llvm-project#115024 for instructions on how to use, but note #293 is needed to avoid disabling certain builders or needing to hack your local buildbot install.
@asb
Copy link
Contributor Author

asb commented Nov 10, 2024

Thank you for the detailed and thoughtful suggestions - I believe I've addressed them all now.

@asb
Copy link
Contributor Author

asb commented Nov 10, 2024

This is beside the point of this change but I wonder if this document could do with a note at the start to the effect that the "master" terminology is what the Buildbot project uses and is therefore used here in that context (see buildbot/buildbot#5382).

Since we have precedent for not using the term ourselves, when we moved the master branch to main.

Yes, I've simply followed what we already do in LLVM by using "buildmaster" as upstream buildbot does (though they did of course rename "buildslave" to "worker"). I'm not sure what the right thing is here.

@asb asb force-pushed the 2024q4-document-buildmaster-test-mode branch from d8a5e75 to 4950b7c Compare November 10, 2024 08:49
@asb
Copy link
Contributor Author

asb commented Nov 10, 2024

This is beside the point of this change but I wonder if this document could do with a note at the start to the effect that the "master" terminology is what the Buildbot project uses and is therefore used here in that context (see buildbot/buildbot#5382).
Since we have precedent for not using the term ourselves, when we moved the master branch to main.

Yes, I've simply followed what we already do in LLVM by using "buildmaster" as upstream buildbot does (though they did of course rename "buildslave" to "worker"). I'm not sure what the right thing is here.

I've submitted a PR to weak the naming of the environment variable to just BUILDBOT_TEST as I think that's sufficiently descriptive and wouldn't require renaming either downstream in LLVM or in buildbot upstream. llvm/llvm-zorg#301

@DavidSpickett DavidSpickett changed the title [docs] Expand HowToAddABuilder with guidance on testing locally [llvm][docs] Expand HowToAddABuilder with guidance on testing locally Nov 12, 2024
@DavidSpickett
Copy link
Collaborator

I'm not sure what the right thing is here.

Adding a note at the start would be my choice, I'll make a PR for it.

@DavidSpickett
Copy link
Collaborator

#115856

Copy link
Collaborator

@DavidSpickett DavidSpickett left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM


* Either wait until the poller sets off a build, or alternatively force a
build to start in the web UI (which is also the best place to review the
build results).
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewing the results seems like a step in itself to me so break out the parenthetical into its own bullet point.


.. code-block:: bash

ssh -N -L 8011:localhost:8011 -L 9990:localhost:9990 username@server_address
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

buildmaster_server_address to be extra clear here.

@asb
Copy link
Contributor Author

asb commented Nov 12, 2024

Thank you, made those final tweaks and reflected the change to BUILDBOT_TEST now the relevant PR in llvm-zorg landed.

I'll hold off merging until llvm/llvm-zorg#293 as the instructions don't work without that fix sadly.

@asb asb merged commit 8da61a3 into llvm:main Nov 12, 2024
9 checks passed
@llvm-ci
Copy link
Collaborator

llvm-ci commented Nov 12, 2024

LLVM Buildbot has detected a new failure on builder libc-riscv32-qemu-yocto-fullbuild-dbg running on rv32gc-qemu-system while building llvm at step 4 "annotate".

Full details are available at: https://lab.llvm.org/buildbot/#/builders/196/builds/931

Here is the relevant piece of the build log for the reference
Step 4 (annotate) failure: 'python ../llvm-zorg/zorg/buildbot/builders/annotated/libc-linux.py ...' (failure)
...
Ran 2 tests.  PASS: 2  FAIL: 0
[811/928] Running unit test libc.test.src.sys.statvfs.linux.statvfs_test
sh: line 1: /timer.3119: Permission denied
[==========] Running 2 tests from 1 test suite.
[ RUN      ] LlvmLibcSysStatvfsTest.StatvfsBasic
[       OK ] LlvmLibcSysStatvfsTest.StatvfsBasic (1 ms)
[ RUN      ] LlvmLibcSysStatvfsTest.StatvfsInvalidPath
[       OK ] LlvmLibcSysStatvfsTest.StatvfsInvalidPath (2 ms)
Ran 2 tests.  PASS: 2  FAIL: 0
[812/928] Running unit test libc.test.src.sys.statvfs.linux.fstatvfs_test
FAILED: projects/libc/test/src/sys/statvfs/linux/CMakeFiles/libc.test.src.sys.statvfs.linux.fstatvfs_test /home/libcrv32buildbot/bbroot/libc-riscv32-qemu-yocto-fullbuild-dbg/build/projects/libc/test/src/sys/statvfs/linux/CMakeFiles/libc.test.src.sys.statvfs.linux.fstatvfs_test 
cd /home/libcrv32buildbot/bbroot/libc-riscv32-qemu-yocto-fullbuild-dbg/build/projects/libc/test/src/sys/statvfs/linux && /home/libcrv32buildbot/cross.sh /home/libcrv32buildbot/bbroot/libc-riscv32-qemu-yocto-fullbuild-dbg/build/projects/libc/test/src/sys/statvfs/linux/libc.test.src.sys.statvfs.linux.fstatvfs_test.__build__
sh: line 1: /timer.3114: Permission denied
[==========] Running 2 tests from 1 test suite.
[ RUN      ] LlvmLibcSysFStatvfsTest.FStatvfsBasic
[       OK ] LlvmLibcSysFStatvfsTest.FStatvfsBasic (1 ms)
[ RUN      ] LlvmLibcSysFStatvfsTest.FStatvfsInvalidPath
/home/libcrv32buildbot/bbroot/libc-riscv32-qemu-yocto-fullbuild-dbg/llvm-project/libc/test/src/sys/statvfs/linux/fstatvfs_test.cpp:40: FAILURE
Failed to match LIBC_NAMESPACE::mkdirat(AT_FDCWD, TEST_DIR, S_IRWXU) against Succeeds(0).
Expected return value to be equal to 0 but got -1.
Expected errno to be equal to "Success" but got "File exists".
[  FAILED  ] LlvmLibcSysFStatvfsTest.FStatvfsInvalidPath
Ran 2 tests.  PASS: 1  FAIL: 1
[813/928] Running unit test libc.test.src.sys.stat.mkdirat_test
sh: line 1: /timer.3126: Permission denied
[==========] Running 2 tests from 1 test suite.
[ RUN      ] LlvmLibcMkdiratTest.CreateAndRemove
[       OK ] LlvmLibcMkdiratTest.CreateAndRemove (3 ms)
[ RUN      ] LlvmLibcMkdiratTest.BadPath
[       OK ] LlvmLibcMkdiratTest.BadPath (218 us)
Ran 2 tests.  PASS: 2  FAIL: 0
[814/928] Running unit test libc.test.src.sys.auxv.linux.getauxval_test
sh: line 1: /timer.3134: Permission denied
[==========] Running 1 test from 1 test suite.
[ RUN      ] LlvmLibcGetauxvalTest.Basic
[       OK ] LlvmLibcGetauxvalTest.Basic (2 ms)
Ran 1 tests.  PASS: 1  FAIL: 0
[815/928] Running unit test libc.test.src.sys.wait.wait4_test
sh: line 1: /timer.3135: Permission denied
[==========] Running 1 test from 1 test suite.
[ RUN      ] LlvmLibcwait4Test.NoHangTest
[       OK ] LlvmLibcwait4Test.NoHangTest (629 us)
Ran 1 tests.  PASS: 1  FAIL: 0
[816/928] Running unit test libc.test.src.sys.epoll.linux.epoll_ctl_test
sh: line 1: /timer.3254: Permission denied
[==========] Running 1 test from 1 test suite.
[ RUN      ] LlvmLibcEpollCtlTest.Basic
[       OK ] LlvmLibcEpollCtlTest.Basic (2 ms)
Ran 1 tests.  PASS: 1  FAIL: 0
Step 8 (libc-unit-tests) failure: libc-unit-tests (failure)
...
Ran 2 tests.  PASS: 2  FAIL: 0
[811/928] Running unit test libc.test.src.sys.statvfs.linux.statvfs_test
sh: line 1: /timer.3119: Permission denied
[==========] Running 2 tests from 1 test suite.
[ RUN      ] LlvmLibcSysStatvfsTest.StatvfsBasic
[       OK ] LlvmLibcSysStatvfsTest.StatvfsBasic (1 ms)
[ RUN      ] LlvmLibcSysStatvfsTest.StatvfsInvalidPath
[       OK ] LlvmLibcSysStatvfsTest.StatvfsInvalidPath (2 ms)
Ran 2 tests.  PASS: 2  FAIL: 0
[812/928] Running unit test libc.test.src.sys.statvfs.linux.fstatvfs_test
FAILED: projects/libc/test/src/sys/statvfs/linux/CMakeFiles/libc.test.src.sys.statvfs.linux.fstatvfs_test /home/libcrv32buildbot/bbroot/libc-riscv32-qemu-yocto-fullbuild-dbg/build/projects/libc/test/src/sys/statvfs/linux/CMakeFiles/libc.test.src.sys.statvfs.linux.fstatvfs_test 
cd /home/libcrv32buildbot/bbroot/libc-riscv32-qemu-yocto-fullbuild-dbg/build/projects/libc/test/src/sys/statvfs/linux && /home/libcrv32buildbot/cross.sh /home/libcrv32buildbot/bbroot/libc-riscv32-qemu-yocto-fullbuild-dbg/build/projects/libc/test/src/sys/statvfs/linux/libc.test.src.sys.statvfs.linux.fstatvfs_test.__build__
sh: line 1: /timer.3114: Permission denied
[==========] Running 2 tests from 1 test suite.
[ RUN      ] LlvmLibcSysFStatvfsTest.FStatvfsBasic
[       OK ] LlvmLibcSysFStatvfsTest.FStatvfsBasic (1 ms)
[ RUN      ] LlvmLibcSysFStatvfsTest.FStatvfsInvalidPath
/home/libcrv32buildbot/bbroot/libc-riscv32-qemu-yocto-fullbuild-dbg/llvm-project/libc/test/src/sys/statvfs/linux/fstatvfs_test.cpp:40: FAILURE
Failed to match LIBC_NAMESPACE::mkdirat(AT_FDCWD, TEST_DIR, S_IRWXU) against Succeeds(0).
Expected return value to be equal to 0 but got -1.
Expected errno to be equal to "Success" but got "File exists".
[  FAILED  ] LlvmLibcSysFStatvfsTest.FStatvfsInvalidPath
Ran 2 tests.  PASS: 1  FAIL: 1
[813/928] Running unit test libc.test.src.sys.stat.mkdirat_test
sh: line 1: /timer.3126: Permission denied
[==========] Running 2 tests from 1 test suite.
[ RUN      ] LlvmLibcMkdiratTest.CreateAndRemove
[       OK ] LlvmLibcMkdiratTest.CreateAndRemove (3 ms)
[ RUN      ] LlvmLibcMkdiratTest.BadPath
[       OK ] LlvmLibcMkdiratTest.BadPath (218 us)
Ran 2 tests.  PASS: 2  FAIL: 0
[814/928] Running unit test libc.test.src.sys.auxv.linux.getauxval_test
sh: line 1: /timer.3134: Permission denied
[==========] Running 1 test from 1 test suite.
[ RUN      ] LlvmLibcGetauxvalTest.Basic
[       OK ] LlvmLibcGetauxvalTest.Basic (2 ms)
Ran 1 tests.  PASS: 1  FAIL: 0
[815/928] Running unit test libc.test.src.sys.wait.wait4_test
sh: line 1: /timer.3135: Permission denied
[==========] Running 1 test from 1 test suite.
[ RUN      ] LlvmLibcwait4Test.NoHangTest
[       OK ] LlvmLibcwait4Test.NoHangTest (629 us)
Ran 1 tests.  PASS: 1  FAIL: 0
[816/928] Running unit test libc.test.src.sys.epoll.linux.epoll_ctl_test
sh: line 1: /timer.3254: Permission denied
[==========] Running 1 test from 1 test suite.
[ RUN      ] LlvmLibcEpollCtlTest.Basic
[       OK ] LlvmLibcEpollCtlTest.Basic (2 ms)
Ran 1 tests.  PASS: 1  FAIL: 0

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants