Skip to content

Commit 474d7ed

Browse files
Alex-WelshAlex-Welsh
committed
Add docs page for running Tempest
1 parent b189927 commit 474d7ed

File tree

2 files changed

+321
-0
lines changed

2 files changed

+321
-0
lines changed

‎doc/source/operations/index.rst

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -13,3 +13,4 @@ This guide is for operators of the StackHPC Kayobe configuration project.
1313
rocky-linux-9
1414
ubuntu-jammy
1515
secret-rotation
16+
tempest

‎doc/source/operations/tempest.rst

Lines changed: 320 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,320 @@
1+
======================================
2+
Running Tempest with Kayobe Automation
3+
======================================
4+
5+
Overview
6+
========
7+
8+
This document describes how to configure and run `Tempest
9+
<https://docs.openstack.org/tempest/latest/>`_ using `kayobe-automation
10+
<https://github.com/stackhpc/kayobe-automation>`_ from the ``.automation``
11+
submodule included with ``stackhpc-kayobe-config``.
12+
13+
The best way of running Tempest is to use CI/CD workflows. Before proceeding,
14+
consider whether it would be possible to use/set up a CI/CD workflow instead.
15+
16+
The following guide will assume all commands are run from your
17+
``kayobe-config`` root and the environment has been configured to run Kayobe
18+
commands unless stated otherwise.
19+
20+
Prerequisites
21+
=============
22+
23+
Installing Docker
24+
-----------------
25+
26+
``kayobe-automation`` runs in a container on the Ansible control host. This
27+
means that Docker must be installed on the Ansible control host if it is not
28+
already.
29+
30+
.. warning::
31+
32+
Docker can cause networking issues when it is installed. By default it will
33+
create a bridge and change ``iptables`` rules. These can be
34+
disabled by setting the following in ``/etc/docker/daemon.json``:
35+
36+
.. code-block:: json
37+
38+
{
39+
"bridge": "none",
40+
"iptables": false
41+
}
42+
43+
To install Docker on Ubuntu:
44+
45+
.. code-block:: bash
46+
47+
# Add Docker's official GPG key:
48+
sudo apt-get update
49+
sudo apt-get install ca-certificates curl
50+
sudo install -m 0755 -d /etc/apt/keyrings
51+
sudo curl -fsSL https://download.docker.com/linux/ubuntu/gpg -o /etc/apt/keyrings/docker.asc
52+
sudo chmod a+r /etc/apt/keyrings/docker.asc
53+
54+
# Add the repository to Apt sources:
55+
echo \
56+
"deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.asc] https://download.docker.com/linux/ubuntu \
57+
$(. /etc/os-release && echo "$VERSION_CODENAME") stable" | \
58+
sudo tee /etc/apt/sources.list.d/docker.list > /dev/null
59+
sudo apt-get update
60+
sudo apt-get install docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin
61+
62+
Installing Docker on CentOS/Rocky:
63+
64+
.. code-block:: bash
65+
66+
sudo dnf install -y dnf-utils
67+
sudo dnf-config-manager --add-repo https://download.docker.com/linux/centos/docker-ce.repo
68+
sudo dnf install docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin
69+
70+
Ensure Docker is running & enabled:
71+
72+
.. code-block:: bash
73+
74+
sudo systemctl start docker
75+
sudo systemctl enable docker
76+
77+
The Docker ``buildx`` plugin must be installed. If you are using an existing
78+
installation of docker, you may need to install it with:
79+
80+
.. code-block:: bash
81+
82+
sudo dnf/apt install docker-buildx-plugin
83+
sudo docker buildx install
84+
# or if that fails:
85+
sudo docker plugin install buildx
86+
87+
Building a Kayobe container
88+
---------------------------
89+
90+
Build a Kayobe automation image:
91+
92+
.. code-block:: bash
93+
94+
git submodule init
95+
git submodule update
96+
# If running on Ubuntu, the fact cache can confuse Kayobe in the CentOS-based container
97+
mv etc/kayobe/facts{,-old}
98+
sudo DOCKER_BUILDKIT=1 docker build --file .automation/docker/kayobe/Dockerfile --tag kayobe:latest .
99+
100+
Configuration
101+
=============
102+
103+
Kayobe automation configuration files are stored in the ``.automation.conf/``
104+
directory. It contains:
105+
106+
- A script used to export environment variables for meta configuration of
107+
Tempest - ``.automation.conf/config.sh``.
108+
- Tempest configuration override files, stored in ``.automation.conf/tempest/``
109+
and conventionally named ``tempest.overrides.conf`` or
110+
``tempest-<environment>.overrides.conf``.
111+
- Tempest load lists, stored in ``.automation.conf/tempest/load-lists``.
112+
- Tempest skip lists, stored in ``.automation.conf/tempest/skip-lists``.
113+
114+
config.sh
115+
---------
116+
117+
``config.sh`` is a mandatory shell script, primarily used to export environment
118+
variables for the meta configuration of Tempest.
119+
120+
See:
121+
https://github.com/stackhpc/docker-rally/blob/master/bin/rally-verify-wrapper.sh
122+
for a full list of Tempest parameters that can be overridden.
123+
124+
The most common variables to override are:
125+
126+
- ``TEMPEST_CONCURRENCY`` - The maximum number of tests to run in parallel at
127+
one time. Higher values are faster but increase the risk of timeouts. 1-2 is
128+
safest in CI/Tenks/Multinode/AIO etc. 8-32 is typical in production. Default
129+
value is 2.
130+
- ``KAYOBE_AUTOMATION_TEMPEST_LOADLIST``: the filename of a load list in the
131+
``load-lists`` directory. Default value is ``default`` (symlink to refstack).
132+
- ``KAYOBE_AUTOMATION_TEMPEST_SKIPLIST``: the filename of a load list in the
133+
``skip-lists`` directory. Default value is unset.
134+
- ``TEMPEST_OPENRC``: The **contents** of an ``openrc.sh`` file, to be used by
135+
Tempest to create resources on the cloud. Default is to read in the contents
136+
of ``etc/kolla/public-openrc.sh``.
137+
138+
tempest.overrides.conf
139+
----------------------
140+
141+
Tempest uses a configuration file to define which tests are run and how to run
142+
them. A full sample configuration file can be found `here
143+
<https://docs.openstack.org/tempest/latest/sampleconf.html>`_. Sensible
144+
defaults exist for all values and in most situations, a blank
145+
``*overrides.conf`` file will successfully run many tests. It will however also
146+
skip many tests which may otherwise be appropriate to run.
147+
148+
`Shakespeare <https://github.com/stackhpc/shakespeare>`_ is a tool for
149+
generating Tempest configuration files. It contains elements for different
150+
cloud features, which can be combined to template out a detailed configuration
151+
file. This is the best-practice approach.
152+
153+
Below is an example of a manually generated file including many of the most
154+
common overrides. It makes many assumptions about the environment, so make sure
155+
you understand all the options before applying them.
156+
157+
.. NOTE(upgrade): Microversions change for each release
158+
.. code-block:: ini
159+
160+
[openstack]
161+
# Use a StackHPC-built image without a default password.
162+
img_url=https://github.com/stackhpc/cirros/releases/download/20231206/cirros-d231206-x86_64-disk.img
163+
164+
[auth]
165+
# Expect unlimited quotas for CPU cores and RAM
166+
compute_quotas = cores:-1,ram:-1
167+
168+
[compute]
169+
# Required for migration testing
170+
min_compute_nodes = 2
171+
# Required to test some API features
172+
min_microversion = 2.1
173+
max_microversion = 2.90
174+
# Flavors for creating test servers and server resize. The ``alt`` flavor should be larger.
175+
flavor_ref = <flavor UUID>
176+
flavor_ref_alt = <different flavor UUID>
177+
volume_multiattach = true
178+
179+
[compute-feature-enabled]
180+
# Required for migration testing
181+
resize = true
182+
live_migration = true
183+
block_migration_for_live_migration = false
184+
volume_backed_live_migration = true
185+
186+
[placement]
187+
min_microversion = "1.0"
188+
max_microversion = "1.39"
189+
190+
[volume]
191+
storage_protocol = ceph
192+
# Required to test some API features
193+
min_microversion = 3.0
194+
max_microversion = 3.68
195+
196+
Tempest configuration override files are stored in
197+
``.automation.conf/tempest/``. The default file used is
198+
``tempest.overrides.conf`` or ``tempest-<environment>.overrides.conf``
199+
depending on whether a Kayobe environment is enabled. This can be changed by
200+
setting ``KAYOBE_AUTOMATION_TEMPEST_CONF_OVERRIDES`` to a different file path.
201+
An ``overrides.conf`` file must be supplied, even if it is blank.
202+
203+
Load Lists
204+
----------
205+
206+
Load lists are a newline-separated list of tests to run. They are stored in
207+
``.automation.conf/tempest/load-lists/``. The directory contains three objects
208+
by default:
209+
210+
- ``tempest-full`` - A complete list of all possible tests.
211+
- ``platform.2022.11-test-list.txt`` - A reduced list of tests to match the
212+
`Refstack <https://docs.opendev.org/openinfra/refstack/latest/>`_ standard.
213+
- ``default`` - A symlink to ``platform.2022.11-test-list.txt``.
214+
215+
Test lists can be selected by changing ``KAYOBE_AUTOMATION_TEMPEST_LOADLIST``
216+
in ``config.sh``. The default value is ``default``, which symlinks to
217+
``platform.2022.11-test-list.txt``.
218+
219+
A common use case is to use the ``failed-tests`` list output from a previous
220+
Tempest run as a load list, to retry the failed tests after making changes.
221+
222+
Skip Lists
223+
----------
224+
225+
Skip lists are a newline-separated list of tests to Skip. They are stored in
226+
``.automation.conf/tempest/skip-lists/``. Each line consists of a pattern to
227+
match against test names, and a string explaining why the test is being
228+
skipped e.g.
229+
230+
.. code-block::
231+
232+
tempest.scenario.test_network_basic_ops.TestNetworkBasicOps.test_subnet_details.*: "Cirros image doesn't have /var/run/udhcpc.eth0.pid"
233+
234+
There is no requirement for a skip list, and none is selected by default. A
235+
skip list can be selected by setting ``KAYOBE_AUTOMATION_TEMPEST_SKIPLIST`` in
236+
``config.sh``.
237+
238+
Tempest runner
239+
--------------
240+
241+
While the Kayobe automation container is always deployed to the ansible control
242+
host, the Tempest container is deployed to the host in the ``tempest_runner``
243+
group, which can be any host in the Kayobe inventory. The group should only
244+
ever contain one host. The seed is usually used as the tempest runner however
245+
it is also common to use the Ansible control host or an infrastructure VM. The
246+
main requirement of the host is that it can reach the OpenStack API.
247+
248+
Running Tempest
249+
===============
250+
251+
Kayobe automation will need to SSH to the Tempest runner (even if they are on
252+
the same host), so requires an SSH key exported as
253+
``KAYOBE_AUTOMATION_SSH_PRIVATE_KEY`` e.g.
254+
255+
.. code-block:: bash
256+
257+
export KAYOBE_AUTOMATION_SSH_PRIVATE_KEY=$(cat ~/.ssh/id_rsa)
258+
259+
Tempest outputs will be sent to the ``tempest-artifacts/`` directory. Create
260+
one if it does not exist.
261+
262+
.. code-block:: bash
263+
264+
mkdir tempest-artifacts
265+
266+
The contents of ``tempest-artifacts`` will be overwritten. Ensure any previous
267+
test results have been copied away.
268+
269+
The Tempest playbook is invoked through the Kayobe container using this
270+
command from the base of the ``kayobe-config`` directory:
271+
272+
.. code-block:: bash
273+
274+
sudo -E docker run --detach -it --rm --network host -v $(pwd):/stack/kayobe-automation-env/src/kayobe-config -v $(pwd)/tempest-artifacts:/stack/tempest-artifacts -e KAYOBE_ENVIRONMENT -e KAYOBE_VAULT_PASSWORD -e KAYOBE_AUTOMATION_SSH_PRIVATE_KEY kayobe:latest /stack/kayobe-automation-env/src/kayobe-config/.automation/pipeline/tempest.sh -e ansible_user=stack
275+
276+
By default, ``no_log`` is set to stop credentials from leaking. This can be
277+
disabled by adding ``-e rally_no_sensitive_log=false`` to the end.
278+
279+
To follow the progress of the Kayobe automation container, either remove
280+
``--detach`` from the above command, or follow the docker logs of the
281+
``kayobe`` container.
282+
283+
To follow the progress of the Tempest tests themselves, follow the logs of the
284+
``tempest`` container on the ``tempest_runner`` host.
285+
286+
.. code-block:: bash
287+
288+
ssh <tempest-runner>
289+
sudo docker logs -f tempest
290+
291+
Tempest will keep running until completion if the ``kayobe`` container is
292+
stopped. The ``tempest`` container must be stopped manually. Doing so will
293+
however stop test resources (such as networks, images, and VMs) from being
294+
automatically cleaned up. They must instead be manually removed. They should be
295+
clearly labeled with either rally or tempest in the name, often alongside some
296+
randomly generated string.
297+
298+
Outputs
299+
-------
300+
301+
Tempest outputs will be sent to the ``tempest-artifacts/`` directory. It
302+
contain the following artifacts:
303+
304+
- ``docker.log`` - The logs from the ``tempest`` docker container
305+
- ``failed-tests`` - A simple list of tests that failed
306+
- ``rally-junit.xml`` - An XML file listing all tests in the test list and
307+
their status (skipped/succeeded/failed). Usually not useful.
308+
- ``rally-verify-report.html`` - An HTML page with all test results including
309+
an error trace for failed tests. It is often best to ``scp`` this file back
310+
to your local machine to view it. This is the most user-friendly way to view
311+
the test results, however can be awkward to host.
312+
- ``rally-verify-report.json`` - A JSON blob with all test results including an
313+
error trace for failed tests. It contains all the same data as the HTML
314+
report but without formatting.
315+
- ``stderr.log`` - The stderr log. Usually not useful.
316+
- ``stdout.log`` - The stdout log. Usually not useful.
317+
- ``tempest-load-list`` - The load list that Tempest was invoked with.
318+
- ``tempest.log`` - Detailed logs from Tempest. Contains more data than the
319+
``verify`` reports, but can be difficult to parse. Useful for tracing specific
320+
errors.

0 commit comments

Comments
 (0)