Skip to content

Commit c6e3f2a

Browse files
Alex-WelshAlex-Welsh
committed
Add docs page for running Tempest
1 parent b189927 commit c6e3f2a

File tree

2 files changed

+304
-0
lines changed

2 files changed

+304
-0
lines changed

‎doc/source/operations/index.rst

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -13,3 +13,4 @@ This guide is for operators of the StackHPC Kayobe configuration project.
1313
rocky-linux-9
1414
ubuntu-jammy
1515
secret-rotation
16+
tempest

‎doc/source/operations/tempest.rst

Lines changed: 303 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,303 @@
1+
======================================
2+
Running Tempest with Kayobe Automation
3+
======================================
4+
5+
Overview
6+
========
7+
8+
This document describes how to configure and run `Tempest
9+
<https://docs.openstack.org/tempest/latest/>`_ using `kayobe-automation
10+
<https://github.com/stackhpc/kayobe-automation>`_ from the ``.automation``
11+
submodule included with ``stackhpc-kayobe-config``.
12+
13+
The following guide will assume all commands are run from your
14+
``kayobe-config`` root and the environment has been configured to run Kayobe
15+
commands unless stated otherwise.
16+
17+
Building a Kayobe container
18+
===========================
19+
``kayobe-automation`` runs in a container on the Ansible control host. This
20+
means that Docker must be installed on the Ansible control host if it is not
21+
already.
22+
23+
.. warning::
24+
25+
Docker can cause networking issues when it is installed. By default it will
26+
create a bridge and change ``iptables`` rules. These can be
27+
disabled by setting the following in ``/etc/docker/daemon.json``:
28+
29+
.. code-block:: json
30+
31+
{
32+
"bridge": "none",
33+
"iptables": false
34+
}
35+
36+
To install Docker on Ubuntu:
37+
38+
.. code-block:: bash
39+
40+
# Add Docker's official GPG key:
41+
sudo apt-get update
42+
sudo apt-get install ca-certificates curl
43+
sudo install -m 0755 -d /etc/apt/keyrings
44+
sudo curl -fsSL https://download.docker.com/linux/ubuntu/gpg -o /etc/apt/keyrings/docker.asc
45+
sudo chmod a+r /etc/apt/keyrings/docker.asc
46+
47+
# Add the repository to Apt sources:
48+
echo \
49+
"deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.asc] https://download.docker.com/linux/ubuntu \
50+
$(. /etc/os-release && echo "$VERSION_CODENAME") stable" | \
51+
sudo tee /etc/apt/sources.list.d/docker.list > /dev/null
52+
sudo apt-get update
53+
sudo apt-get install docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin
54+
55+
Installing Docker on CentOS/Rocky:
56+
57+
.. code-block:: bash
58+
59+
sudo yum install -y yum-utils
60+
sudo yum-config-manager --add-repo https://download.docker.com/linux/centos/docker-ce.repo
61+
62+
Ensure Docker is running:
63+
64+
.. code-block:: bash
65+
66+
sudo systemctl start docker
67+
68+
The Docker ``buildx`` plugin must be installed. If you are using an existing
69+
installation of docker, you may need to install it with:
70+
71+
.. code-block:: bash
72+
73+
sudo dnf/apt install docker-buildx-plugin
74+
sudo docker buildx install
75+
# or if that fails:
76+
sudo docker plugin install buildx
77+
78+
Build a Kayobe automation image:
79+
80+
.. code-block:: bash
81+
82+
git submodule init
83+
git submodule update
84+
# If running on Ubuntu, the fact cache can confuse Kayobe in the CentOS-based container
85+
mv etc/kayobe/facts{,-old}
86+
sudo DOCKER_BUILDKIT=1 docker build --file .automation/docker/kayobe/Dockerfile --tag kayobe:latest .
87+
88+
Configuration
89+
=============
90+
91+
Kayobe automation configuration files are stored in the ``.automation.conf/``
92+
directory. It contains:
93+
94+
- A script used to export environment variables for meta configuration of
95+
Tempest - ``.automation.conf/config.sh``.
96+
- Tempest configuration override files, stored in ``.automation.conf/tempest/``
97+
and conventionally named ``tempest.overrides.conf`` or
98+
``tempest-<environment>.overrides.conf``.
99+
- Tempest load lists, stored in ``.automation.conf/tempest/load-lists``.
100+
- Tempest skip lists, stored in ``.automation.conf/tempest/skip-lists``.
101+
102+
config.sh
103+
---------
104+
105+
``config.sh`` is a mandatory shell script, primarily used to export environment
106+
variables for the meta configuration of Tempest.
107+
108+
See:
109+
https://github.com/stackhpc/docker-rally/blob/master/bin/rally-verify-wrapper.sh
110+
for a full list of tempest parameters that can be overridden.
111+
112+
The most common variables to override are:
113+
114+
- ``TEMPEST_CONCURRENCY`` - The maximum number of tests to run in parallel at
115+
one time. Higher values are faster but increase the risk of timeouts. 1-2 is
116+
safest in CI/Tenks/Multinode/AIO etc. 8-32 is typical in production. Default
117+
value is 2.
118+
- ``KAYOBE_AUTOMATION_TEMPEST_LOADLIST``: the filename of a load list in the
119+
``load-lists`` directory. Default value is ``default`` (symlink to refstack).
120+
- ``KAYOBE_AUTOMATION_TEMPEST_SKIPLIST``: the filename of a load list in the
121+
``skip-lists`` directory. Default value is unset.
122+
- ``TEMPEST_OPENRC``: The **contents** of an ``openrc.sh`` file, to be used by
123+
Tempest to create resources on the cloud. Default is to read in the contents
124+
of ``etc/kolla/public-openrc.sh``.
125+
126+
tempest.overrides.conf
127+
----------------------
128+
129+
Tempest uses a configuration file to define which tests are run and how to run
130+
them. A full sample configuration file can be found `here
131+
<https://docs.openstack.org/tempest/latest/sampleconf.html>`_. Sensible
132+
defaults exist for all values and in most situations, a blank
133+
``*overrides.conf`` file will successfully run many tests. It will however also
134+
skip many tests which may otherwise be appropriate to run.
135+
136+
Below is an example file including many of the most common overrides. It makes
137+
many assumptions about the environment, so make sure you understand all the
138+
options before applying them.
139+
140+
.. NOTE(upgrade): Microversions change for each release
141+
.. code-block:: ini
142+
143+
[openstack]
144+
# Use a StackHPC-built image without a default password.
145+
img_url=https://github.com/stackhpc/cirros/releases/download/20231206/cirros-d231206-x86_64-disk.img
146+
147+
[auth]
148+
# Expect unlimited quotas for CPU cores and RAM
149+
compute_quotas = cores:-1,ram:-1
150+
151+
[compute]
152+
# Required for migration testing
153+
min_compute_nodes = 2
154+
# Required to test some API features
155+
min_microversion = 2.1
156+
max_microversion = 2.90
157+
# Flavors for creating test servers and server resize. The ``alt`` flavor should be larger.
158+
flavor_ref = <flavor UUID>
159+
flavor_ref_alt = <different flavor UUID>
160+
volume_multiattach = true
161+
162+
[compute-feature-enabled]
163+
# Required for migration testing
164+
resize = true
165+
live_migration = true
166+
block_migration_for_live_migration = false
167+
volume_backed_live_migration = true
168+
169+
[placement]
170+
min_microversion = "1.0"
171+
max_microversion = "1.39"
172+
173+
[volume]
174+
storage_protocol = ceph
175+
# Required to test some API features
176+
min_microversion = 3.0
177+
max_microversion = 3.68
178+
179+
Tempest configuration override files are stored in
180+
``.automation.conf/tempest/``. The default file used is
181+
``tempest.overrides.conf`` or ``tempest-<environment>.overrides.conf``
182+
depending on whether a Kayobe environment is enabled. This can be changed by
183+
setting ``KAYOBE_AUTOMATION_TEMPEST_CONF_OVERRIDES`` to a different file path.
184+
An ``overrides.conf`` file must be supplied, even if it is blank.
185+
186+
Load Lists
187+
----------
188+
189+
Load lists are a newline-separated list of tests to run. They are stored in
190+
``.automation.conf/tempest/load-lists/``. The directory contains three objects
191+
by default:
192+
193+
- ``tempest-full`` - A complete list of all possible tests.
194+
- ``platform.2022.11-test-list.txt`` - A reduced list of tests to match the
195+
`Refstack <https://docs.opendev.org/openinfra/refstack/latest/>`_ standard.
196+
- ``default`` - A symlink to ``platform.2022.11-test-list.txt``.
197+
198+
Test lists can be selected by changing ``KAYOBE_AUTOMATION_TEMPEST_SKIPLIST``
199+
in ``config.sh``. The default value is ``default``, which symlinks to
200+
``platform.2022.11-test-list.txt``.
201+
202+
A common use case is to use the ``failed-tests`` list output from a previous
203+
tempest run as a load list, to retry the failed tests after making changes.
204+
205+
Skip Lists
206+
----------
207+
208+
Skip lists are a newline-separated list of tests to Skip. They are stored in
209+
``.automation.conf/tempest/skip-lists/``. Each line consists of a pattern to
210+
match against test names, and a string explaining why the test is being
211+
skipped e.g.
212+
213+
.. code-block::
214+
215+
tempest.scenario.test_network_basic_ops.TestNetworkBasicOps.test_subnet_details.*: "Cirros image doesn't have /var/run/udhcpc.eth0.pid"
216+
217+
There is no requirement for a skip list, and none is selected by default. A
218+
skip list can be selected by setting ``KAYOBE_AUTOMATION_TEMPEST_SKIPLIST`` in
219+
``config.sh``.
220+
221+
Tempest runner
222+
--------------
223+
224+
While the Kayobe automation container is always deployed to the ansible control
225+
host, the Tempest container is deployed to the host in the ``tempest_runner``
226+
group, which can be any host in the Kayobe inventory. The group should only
227+
ever contain one host. The seed is usually used as the tempest runner however
228+
it is also common to use the Ansible control host or an infrastructure VM. The
229+
main requirement of the host is that it can reach the OpenStack API.
230+
231+
Running Tempest
232+
===============
233+
234+
Kayobe automation will need to SSH to the tempest runner (even if they are on
235+
the same host), so requires an SSH key exported as
236+
``KAYOBE_AUTOMATION_SSH_PRIVATE_KEY`` e.g.
237+
238+
.. code-block:: bash
239+
240+
export KAYOBE_AUTOMATION_SSH_PRIVATE_KEY=$(cat ~/.ssh/id_rsa)
241+
242+
Tempest outputs will be sent to the ``tempest-artifacts/`` directory. Create
243+
one if it does not exist.
244+
245+
.. code-block:: bash
246+
247+
mkdir tempest-artifacts
248+
249+
The contents of ``tempest-artifacts`` will be overwritten. Ensure any previous
250+
test results have been copied away.
251+
252+
The Tempest playbook is invoked through the Kayobe container using this
253+
command from the base of the ``kayobe-config`` directory:
254+
255+
.. code-block:: bash
256+
257+
sudo -E docker run --detach -it --rm --network host -v $(pwd):/stack/kayobe-automation-env/src/kayobe-config -v $(pwd)/tempest-artifacts:/stack/tempest-artifacts -e KAYOBE_ENVIRONMENT -e KAYOBE_VAULT_PASSWORD -e KAYOBE_AUTOMATION_SSH_PRIVATE_KEY kayobe:latest /stack/kayobe-automation-env/src/kayobe-config/.automation/pipeline/tempest.sh -e ansible_user=stack
258+
259+
By default, ``no_log`` is set to stop credentials from leaking. This can be
260+
disabled by adding ``-e rally_no_sensitive_log=false`` to the end.
261+
262+
To follow the progress of the Kayobe automation container, either remove
263+
``--detach`` from the above command, or follow the docker logs of the
264+
``kayobe`` container.
265+
266+
To follow the progress of the tempest tests themselves, follow the logs of the
267+
``tempest`` container on the ``tempest_runner`` host.
268+
269+
.. code-block:: bash
270+
271+
ssh <tempest-runner>
272+
sudo docker logs -f tempest
273+
274+
Tempest will keep running until completion if the ``kayobe`` container is
275+
stopped. The ``tempest`` container must be stopped manually. Doing so will
276+
however stop test resources (such as networks, images, and VMs) from being
277+
automatically cleaned up. They must instead be manually removed. They should be
278+
clearly labeled with either rally or tempest in the name, often alongside some
279+
randomly generated string.
280+
281+
Outputs
282+
-------
283+
284+
Tempest outputs will be sent to the ``tempest-artifacts/`` directory. It
285+
contain the following artifacts:
286+
287+
- ``docker.log`` - The logs from the ``tempest`` docker container
288+
- ``failed-tests`` - A simple list of tests that failed
289+
- ``rally-junit.xml`` - An XML file listing all tests in the test list and
290+
their status (skipped/succeeded/failed). Usually not useful.
291+
- ``rally-verify-report.html`` - An HTML page with all test results including
292+
an error trace for failed tests. It is often best to ``scp`` this file back
293+
to your local machine to view it. This is the most user-friendly way to view
294+
the test results, however can be awkward to host.
295+
- ``rally-verify-report.json`` - A JSON blob with all test results including an
296+
error trace for failed tests. It contains all the same data as the HTML
297+
report but without formatting.
298+
- ``stderr.log`` - The stderr log. Usually not useful.
299+
- ``stdout.log`` - The stdout log. Usually not useful.
300+
- ``tempest-load-list`` - The load list that Tempest was invoked with.
301+
- ``tempest.log`` - Detailed logs from Tempest. Contains more data than the
302+
``verify`` reports, but can be difficult to parse. Useful for tracing specific
303+
errors.

0 commit comments

Comments
 (0)