Skip to content

Commit b20d96f

Browse files
Alex-WelshAlex-Welsh
committed
Add docs page for running Tempest
1 parent b189927 commit b20d96f

File tree

2 files changed

+323
-0
lines changed

2 files changed

+323
-0
lines changed

‎doc/source/operations/index.rst

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -13,3 +13,4 @@ This guide is for operators of the StackHPC Kayobe configuration project.
1313
rocky-linux-9
1414
ubuntu-jammy
1515
secret-rotation
16+
tempest

‎doc/source/operations/tempest.rst

Lines changed: 322 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,322 @@
1+
======================================
2+
Running Tempest with Kayobe Automation
3+
======================================
4+
5+
Overview
6+
========
7+
8+
This document describes how to configure and run `Tempest
9+
<https://docs.openstack.org/tempest/latest/>`_ using `kayobe-automation
10+
<https://github.com/stackhpc/kayobe-automation>`_ from the ``.automation``
11+
submodule included with ``stackhpc-kayobe-config``.
12+
13+
The best way of running Tempest is to use CI/CD workflows. Before proceeding,
14+
consider whether it would be possible to use/set up a CI/CD workflow instead.
15+
16+
The following guide will assume all commands are run from your
17+
``kayobe-config`` root and the environment has been configured to run Kayobe
18+
commands unless stated otherwise.
19+
20+
Prerequisites
21+
=============
22+
23+
Installing Docker
24+
-----------------
25+
26+
``kayobe-automation`` runs in a container on the Ansible control host. This
27+
means that Docker must be installed on the Ansible control host if it is not
28+
already.
29+
30+
.. warning::
31+
32+
Docker can cause networking issues when it is installed. By default it will
33+
create a bridge and change ``iptables`` rules. These can be
34+
disabled by setting the following in ``/etc/docker/daemon.json``:
35+
36+
.. code-block:: json
37+
38+
{
39+
"bridge": "none",
40+
"iptables": false
41+
}
42+
43+
To install Docker on Ubuntu:
44+
45+
.. code-block:: bash
46+
47+
# Add Docker's official GPG key:
48+
sudo apt-get update
49+
sudo apt-get install ca-certificates curl
50+
sudo install -m 0755 -d /etc/apt/keyrings
51+
sudo curl -fsSL https://download.docker.com/linux/ubuntu/gpg -o /etc/apt/keyrings/docker.asc
52+
sudo chmod a+r /etc/apt/keyrings/docker.asc
53+
54+
# Add the repository to Apt sources:
55+
echo \
56+
"deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.asc] https://download.docker.com/linux/ubuntu \
57+
$(. /etc/os-release && echo "$VERSION_CODENAME") stable" | \
58+
sudo tee /etc/apt/sources.list.d/docker.list > /dev/null
59+
sudo apt-get update
60+
sudo apt-get install docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin
61+
62+
Installing Docker on CentOS/Rocky:
63+
64+
.. code-block:: bash
65+
66+
sudo dnf install -y dnf-utils
67+
sudo dnf-config-manager --add-repo https://download.docker.com/linux/centos/docker-ce.repo
68+
sudo dnf install docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin
69+
70+
Ensure Docker is running & enabled:
71+
72+
.. code-block:: bash
73+
74+
sudo systemctl start docker
75+
sudo systemctl enable docker
76+
77+
The Docker ``buildx`` plugin must be installed. If you are using an existing
78+
installation of docker, you may need to install it with:
79+
80+
.. code-block:: bash
81+
82+
sudo dnf/apt install docker-buildx-plugin
83+
sudo docker buildx install
84+
# or if that fails:
85+
sudo docker plugin install buildx
86+
87+
88+
89+
Building a Kayobe container
90+
---------------------------
91+
92+
Build a Kayobe automation image:
93+
94+
.. code-block:: bash
95+
96+
git submodule init
97+
git submodule update
98+
# If running on Ubuntu, the fact cache can confuse Kayobe in the CentOS-based container
99+
mv etc/kayobe/facts{,-old}
100+
sudo DOCKER_BUILDKIT=1 docker build --file .automation/docker/kayobe/Dockerfile --tag kayobe:latest .
101+
102+
Configuration
103+
=============
104+
105+
Kayobe automation configuration files are stored in the ``.automation.conf/``
106+
directory. It contains:
107+
108+
- A script used to export environment variables for meta configuration of
109+
Tempest - ``.automation.conf/config.sh``.
110+
- Tempest configuration override files, stored in ``.automation.conf/tempest/``
111+
and conventionally named ``tempest.overrides.conf`` or
112+
``tempest-<environment>.overrides.conf``.
113+
- Tempest load lists, stored in ``.automation.conf/tempest/load-lists``.
114+
- Tempest skip lists, stored in ``.automation.conf/tempest/skip-lists``.
115+
116+
config.sh
117+
---------
118+
119+
``config.sh`` is a mandatory shell script, primarily used to export environment
120+
variables for the meta configuration of Tempest.
121+
122+
See:
123+
https://github.com/stackhpc/docker-rally/blob/master/bin/rally-verify-wrapper.sh
124+
for a full list of Tempest parameters that can be overridden.
125+
126+
The most common variables to override are:
127+
128+
- ``TEMPEST_CONCURRENCY`` - The maximum number of tests to run in parallel at
129+
one time. Higher values are faster but increase the risk of timeouts. 1-2 is
130+
safest in CI/Tenks/Multinode/AIO etc. 8-32 is typical in production. Default
131+
value is 2.
132+
- ``KAYOBE_AUTOMATION_TEMPEST_LOADLIST``: the filename of a load list in the
133+
``load-lists`` directory. Default value is ``default`` (symlink to refstack).
134+
- ``KAYOBE_AUTOMATION_TEMPEST_SKIPLIST``: the filename of a load list in the
135+
``skip-lists`` directory. Default value is unset.
136+
- ``TEMPEST_OPENRC``: The **contents** of an ``openrc.sh`` file, to be used by
137+
Tempest to create resources on the cloud. Default is to read in the contents
138+
of ``etc/kolla/public-openrc.sh``.
139+
140+
tempest.overrides.conf
141+
----------------------
142+
143+
Tempest uses a configuration file to define which tests are run and how to run
144+
them. A full sample configuration file can be found `here
145+
<https://docs.openstack.org/tempest/latest/sampleconf.html>`_. Sensible
146+
defaults exist for all values and in most situations, a blank
147+
``*overrides.conf`` file will successfully run many tests. It will however also
148+
skip many tests which may otherwise be appropriate to run.
149+
150+
`Shakespeare <https://github.com/stackhpc/shakespeare>`_ is a tool for
151+
generating Tempest configuration files. It contains elements for different
152+
cloud features, which can be combined to template out a detailed configuration
153+
file. This is the best-practice approach.
154+
155+
Below is an example of a manually generated file including many of the most
156+
common overrides. It makes many assumptions about the environment, so make sure
157+
you understand all the options before applying them.
158+
159+
.. NOTE(upgrade): Microversions change for each release
160+
.. code-block:: ini
161+
162+
[openstack]
163+
# Use a StackHPC-built image without a default password.
164+
img_url=https://github.com/stackhpc/cirros/releases/download/20231206/cirros-d231206-x86_64-disk.img
165+
166+
[auth]
167+
# Expect unlimited quotas for CPU cores and RAM
168+
compute_quotas = cores:-1,ram:-1
169+
170+
[compute]
171+
# Required for migration testing
172+
min_compute_nodes = 2
173+
# Required to test some API features
174+
min_microversion = 2.1
175+
max_microversion = 2.90
176+
# Flavors for creating test servers and server resize. The ``alt`` flavor should be larger.
177+
flavor_ref = <flavor UUID>
178+
flavor_ref_alt = <different flavor UUID>
179+
volume_multiattach = true
180+
181+
[compute-feature-enabled]
182+
# Required for migration testing
183+
resize = true
184+
live_migration = true
185+
block_migration_for_live_migration = false
186+
volume_backed_live_migration = true
187+
188+
[placement]
189+
min_microversion = "1.0"
190+
max_microversion = "1.39"
191+
192+
[volume]
193+
storage_protocol = ceph
194+
# Required to test some API features
195+
min_microversion = 3.0
196+
max_microversion = 3.68
197+
198+
Tempest configuration override files are stored in
199+
``.automation.conf/tempest/``. The default file used is
200+
``tempest.overrides.conf`` or ``tempest-<environment>.overrides.conf``
201+
depending on whether a Kayobe environment is enabled. This can be changed by
202+
setting ``KAYOBE_AUTOMATION_TEMPEST_CONF_OVERRIDES`` to a different file path.
203+
An ``overrides.conf`` file must be supplied, even if it is blank.
204+
205+
Load Lists
206+
----------
207+
208+
Load lists are a newline-separated list of tests to run. They are stored in
209+
``.automation.conf/tempest/load-lists/``. The directory contains three objects
210+
by default:
211+
212+
- ``tempest-full`` - A complete list of all possible tests.
213+
- ``platform.2022.11-test-list.txt`` - A reduced list of tests to match the
214+
`Refstack <https://docs.opendev.org/openinfra/refstack/latest/>`_ standard.
215+
- ``default`` - A symlink to ``platform.2022.11-test-list.txt``.
216+
217+
Test lists can be selected by changing ``KAYOBE_AUTOMATION_TEMPEST_LOADLIST``
218+
in ``config.sh``. The default value is ``default``, which symlinks to
219+
``platform.2022.11-test-list.txt``.
220+
221+
A common use case is to use the ``failed-tests`` list output from a previous
222+
Tempest run as a load list, to retry the failed tests after making changes.
223+
224+
Skip Lists
225+
----------
226+
227+
Skip lists are a newline-separated list of tests to Skip. They are stored in
228+
``.automation.conf/tempest/skip-lists/``. Each line consists of a pattern to
229+
match against test names, and a string explaining why the test is being
230+
skipped e.g.
231+
232+
.. code-block::
233+
234+
tempest.scenario.test_network_basic_ops.TestNetworkBasicOps.test_subnet_details.*: "Cirros image doesn't have /var/run/udhcpc.eth0.pid"
235+
236+
There is no requirement for a skip list, and none is selected by default. A
237+
skip list can be selected by setting ``KAYOBE_AUTOMATION_TEMPEST_SKIPLIST`` in
238+
``config.sh``.
239+
240+
Tempest runner
241+
--------------
242+
243+
While the Kayobe automation container is always deployed to the ansible control
244+
host, the Tempest container is deployed to the host in the ``tempest_runner``
245+
group, which can be any host in the Kayobe inventory. The group should only
246+
ever contain one host. The seed is usually used as the tempest runner however
247+
it is also common to use the Ansible control host or an infrastructure VM. The
248+
main requirement of the host is that it can reach the OpenStack API.
249+
250+
Running Tempest
251+
===============
252+
253+
Kayobe automation will need to SSH to the Tempest runner (even if they are on
254+
the same host), so requires an SSH key exported as
255+
``KAYOBE_AUTOMATION_SSH_PRIVATE_KEY`` e.g.
256+
257+
.. code-block:: bash
258+
259+
export KAYOBE_AUTOMATION_SSH_PRIVATE_KEY=$(cat ~/.ssh/id_rsa)
260+
261+
Tempest outputs will be sent to the ``tempest-artifacts/`` directory. Create
262+
one if it does not exist.
263+
264+
.. code-block:: bash
265+
266+
mkdir tempest-artifacts
267+
268+
The contents of ``tempest-artifacts`` will be overwritten. Ensure any previous
269+
test results have been copied away.
270+
271+
The Tempest playbook is invoked through the Kayobe container using this
272+
command from the base of the ``kayobe-config`` directory:
273+
274+
.. code-block:: bash
275+
276+
sudo -E docker run --detach -it --rm --network host -v $(pwd):/stack/kayobe-automation-env/src/kayobe-config -v $(pwd)/tempest-artifacts:/stack/tempest-artifacts -e KAYOBE_ENVIRONMENT -e KAYOBE_VAULT_PASSWORD -e KAYOBE_AUTOMATION_SSH_PRIVATE_KEY kayobe:latest /stack/kayobe-automation-env/src/kayobe-config/.automation/pipeline/tempest.sh -e ansible_user=stack
277+
278+
By default, ``no_log`` is set to stop credentials from leaking. This can be
279+
disabled by adding ``-e rally_no_sensitive_log=false`` to the end.
280+
281+
To follow the progress of the Kayobe automation container, either remove
282+
``--detach`` from the above command, or follow the docker logs of the
283+
``kayobe`` container.
284+
285+
To follow the progress of the Tempest tests themselves, follow the logs of the
286+
``tempest`` container on the ``tempest_runner`` host.
287+
288+
.. code-block:: bash
289+
290+
ssh <tempest-runner>
291+
sudo docker logs -f tempest
292+
293+
Tempest will keep running until completion if the ``kayobe`` container is
294+
stopped. The ``tempest`` container must be stopped manually. Doing so will
295+
however stop test resources (such as networks, images, and VMs) from being
296+
automatically cleaned up. They must instead be manually removed. They should be
297+
clearly labeled with either rally or tempest in the name, often alongside some
298+
randomly generated string.
299+
300+
Outputs
301+
-------
302+
303+
Tempest outputs will be sent to the ``tempest-artifacts/`` directory. It
304+
contain the following artifacts:
305+
306+
- ``docker.log`` - The logs from the ``tempest`` docker container
307+
- ``failed-tests`` - A simple list of tests that failed
308+
- ``rally-junit.xml`` - An XML file listing all tests in the test list and
309+
their status (skipped/succeeded/failed). Usually not useful.
310+
- ``rally-verify-report.html`` - An HTML page with all test results including
311+
an error trace for failed tests. It is often best to ``scp`` this file back
312+
to your local machine to view it. This is the most user-friendly way to view
313+
the test results, however can be awkward to host.
314+
- ``rally-verify-report.json`` - A JSON blob with all test results including an
315+
error trace for failed tests. It contains all the same data as the HTML
316+
report but without formatting.
317+
- ``stderr.log`` - The stderr log. Usually not useful.
318+
- ``stdout.log`` - The stdout log. Usually not useful.
319+
- ``tempest-load-list`` - The load list that Tempest was invoked with.
320+
- ``tempest.log`` - Detailed logs from Tempest. Contains more data than the
321+
``verify`` reports, but can be difficult to parse. Useful for tracing specific
322+
errors.

0 commit comments

Comments
 (0)