User Tools

Site Tools


dev:regtesting

This is an old revision of the document!


CP2K Regression Testing

CP2K comes with over 2500 test input files (located in cp2k/tests) which serve as both examples on how to use the many features in CP2K and also as a method for developers to test modifications and extensions to CP2K. In order to reduce the chance of bugs being introduced into the code, and ensure that all parts of the code are working, we recommend that all developers complete a successful run of the full regression test suite before committing changes to SVN.

Dashboard

A number of regtests are run automatically by various members of our community. The results of these tests are collected centrally at the Dashboard . If errors are detected, the developer responsible for the change should fix it immediately.

Code Coverage

We aim that the regression test suite covers all the functionality of CP2K. For this purpose we regularly create Coverage Reports of the testsuit. If you see parts of the code which are not well tested, please contribute to improving coverage by writing new tests!

How does it work?

The regression test suite is run using the do_regtest script. It performs the following tasks:

  • update to the current SVN version (including new tests)
  • performs a realclean build of the source
  • executes a list of tests
  • compares the results (outputs) with those of the last known result (reference)
  • produces a summary

Running the regtests

Step 1: Preparation

  • You must be able to build and run cp2k on the given machine, the rest should be 'easy'
  • Decide on a directory for doing the regtest, there will be plenty of files in this dir (after a while) so make it something like $HOME/rt
  • Checkout a version of cp2k into $HOME/rt. If you set up your svn account so that you don't need to provide a password for doing svn update, things will be more convenient later on.
  • Set up the arch files so that you can cleanly build cp2k (test this)

Step 2: Configuration

The do_regtest script is controlled via a config file. It should contain the following variables:

Config Variable Default Meaning
dir_base the base directory for testing (e.g. $HOME/rt)
FORT_C_NAME compiler selection (e.g. intel)
cp2k_version sopt,sdbg,popt….
dir_triplet the result of tools/get_arch_name, where the executable can be found
cp2k_dir cp2k normally cp2k
maxtasks Maximum number of CP2K threads to execute concurrently (== total number of CPU cores to be used for regtesting)
emptycheck NO useful for automatic testers, no testing if nothing changed in SVN (YES/NO)
leakcheck NO if using gfortran with the LeakSanitizer, set this variable to “YES” in order to get memory leak checking
cp2k_prefix
cp2k_postfix
make
awk
datum_full
datum_short
default_err_tolerance

Furthermore:

# Script configuration. The value of the follow variables can be redefined, see below
#   dir_base, FORT_C_NAME, cp2k_version, dir_triplet, cp2k_dir, cp2k_prefix, cp2k_postfix,
#   make, awk, datum_full, datum_short, default_err_tolerance
#   nosvn, quick, svndate, noreset, ndirtoskip, skip_dirs, restrict_dir, ndirtorestrict, doretest

Step 3: Execution

The do_regtest script has the following command line switches. Most of them can also be included into the config file.

Command Line Switch Meaning
-nosvn do not access the SVN for any updating, makes regtesting fully local
-quick rebuild the code if needed, but do not perform a realclean before (noquick is not needed anymore)
-noreset do not reset the reference outputs automatically
-svndate <str> specify any string to svn update (most likely used as “2005-02-17”)
-skipdir <dir> this switch can repeat, exclude certain dirs from regtesting, useful to speed-up regtesting after very localised changes (e.g. -skipdir QS/regtest)
-restrictdir <dir> this switch can repeat, restrict regtesting to certain dirs, useful to speed-up regtesting after very localised changes (e.g. -restrictdir QS/regtest)
-config <path> loads a site/compiler/environment specific configuration
-retest regtesting will be restricted only to those directories that contained failed tests in the previous run
-noemptycheck use this to force regression testing even if no changes exist in SVN
-nobuild do not build cp2k, but use one built by the user (turns on quick option automatically)

The do_regtest script has the following possible exists statuses:

Exit Code Meaning
0 clean exit with testing
1 problem with svn update
3 problem with realclean
4 build errors
5 problem with retest option - no TEST directory with latest test results found
6 problem with retest option - no error summary exists in the last TEST directory
7 reference directory is locked
8 ctrl-C (SIGINT) and various other signals trapped
100 no svn changes since last run - clean exit without testing

Step 4: Interpretation

A test results can be any of the following:

Test Result Meaning
OK if the results match those of a previous run precisely. The execution time is also given.
NEW if they have not been executed previously. The reference result is generated automatically in this run. Tests can also be NEW if they have been reset, i.e. been newly added to the TEST_FILES_RESET files.
RUNTIME FAILURE if they stopped unexpectedly (e.g. core dump, or stop)
WRONG RESULT if they produce a result that deviates (even a tiny bit) from an old reference

The last two outcomes generally mean that a bug has been introduced, which requires investigation. Since regtesting only yields information relative to a previously known result, it is most useful to do a regtest before and after you make changes. To allow per-test numerical difference higher than that set as a default, add third column in appropriate TEST_FILES file with a relative value of the difference.

Adding and Resetting Tests

The test-suite is fully controlled by the following files in the cp2k/tests directories

File Name Content
TEST_DIRS is just a list of directories that contain tests. You can add your directory here.
TEST_FILES the list of input files that need to be executed. You can add your file name here. Adding a comment about what it tests might help later debugging problems if a regtest fails
TEST_FILES_RESET you can add files for which the reference output became invalid (e.g. bug fix) to this list fo files. However be absolutely sure that the change is due to a bug fix, do not reset these that fail because of unclear reasons. Try to add a comment to the svn message and/or the file itself
TEST_TYPES this file allows you to create a new test type. I.e. to specify for which words should be grepped and what field should be used in the numerical comparison.
dev/regtesting.1430227782.txt.gz · Last modified: 2020/08/21 10:14 (external edit)