Dev guide darktable testing tools

This page provides guidance on how to use:

• an integration test suite focused on regression testing for changes to IOP modules
• how to use the tool darktable-cli to help with further ad-hoc testing
• the CMocka unit testing framework

Integration tests

The base suite of integration tests can be found in src/tests/integration. There is a detailed README.txt included.

The tests use an industry measure of colour distance, created by the CIE, called Delta E (dE.) The threshold in use by Darktable developers is currently dE < 2. A limit of zero is not practical as the complex colour space computations used by darktable WILL vary slightly depending on libraries, versions, and underlying platform and OS.

The integration test scripts require some additional tools in order to be able to carry out this measurement. See below for a list of these pre-requisites; once they are installed you simply change to your src/tests/integration directory and run (for example)

./run 0001-exposure

To use these tests you require an environment which supports the following:

• ImageMagick's "compare" tool - note this is NOT the "compare" script shipped elsewhere in the current source tree
• a python3 environment, which must include:
  • opencv
  • numpy
  • colour (the package is called "colour-science" in most distros; note English spelling)

The intention is to include a test for every IOP. If you are testing changes to a module and there is no integration test please create one by following the instructions in README.txt or raise an issue.

• The PNG format is used when comparing images in these tests because it is lossless and should be pixel-accurate to the intended output of darktable's pipeline.
• The "expected" PNG result is managed within the source tree and was originally generated in @turbogit's development environment.
• The tests force LCMS2 off (otherwise they would test the accuracy of the upstream LCMS2, not darktable!)
• The test runs once with OpenCL disabled and once with it enabled, to allow comparison of results from these two codepaths in addition to the comparison with the known-good baseline image provided by the maintainers.
• The output png shows red pixels where significant differences between the images are found.

History

• These tests were added in PR #4818
• Issue 5203 added further explicit options to ensure test stability: memory limit, thread limit, disable LCMS2.

darktable-cli

Darktable includes a tool for running a set of operations repeatably from the command line. This is found in the bin subdirectory of the installed location. It is useful for carrying out repeatable, automated operations during testing. It is the basis for the included suite of pipeline tests discussed in the following section.

Usage: darktable-cli <input file> [<xmp file>] <output file> [options] [--core <darktable options>]

You can use this tool to:

• re-run existing XMP files on the same input, repeatably, in order to test how code or configuration changes affect the output;
• apply styles repeatably
• create consistent batch operations for use in testing performance

The --core flag will pass the remainder of the command line through to darktable itself, which is useful for setting debugging options.

Debug options useful for testing include:

• -d perf - adds profiling (timing) output to the console as darktable runs through the steps in a pipeline.
• -t <threads> sets the maximum number of threads used by OpenMP. This is useful for reducing to single-threaded if you suspect multithreading is causing problems, and also useful for determining whether more threads is actually helping with performance.

Further reading

• darktable-cli section of the user manual

Unit testing

In PR 4154 a framework for unit testing using the CMocka toolset was created. Pre-reqs:

• darktable must be built with -DBUILD_TESTING=On
• apt install libcmocka-dev or pacman -S mingw-w64-x86_64-cmocka