CP2K Open Source Molecular Dynamics

User Tools

Site Tools

Trace:
dev:regtesting

Differences

This shows you the differences between two versions of the page.

Link to this comparison view

Both sides previous revisionPrevious revision
Next revision		Previous revision
Next revisionBoth sides next revision
dev:regtesting [2015/04/28 14:21] – oschuettdev:regtesting [2018/10/08 19:59] – oschuett
Line 1: Line 1:
 ====== CP2K Regression Testing ====== ====== CP2K Regression Testing ======
  
-CP2K comes with over 2500 test input files (located in [[src>cp2k/tests]]) which serve as both examples on how to use the many features in CP2K and also as a method for developers to test modifications and extensions to CP2K. In order to reduce the chance of bugs being introduced into the code, and ensure that all parts of the code are working, we recommend that all developers complete a successful run of the full regression test suite before committing changes to SVN.+CP2K comes with over 2500 test input files (located in [[src>tests]]) which serve as both examples on how to use the many features in CP2K and also as a method for developers to test modifications and extensions to CP2K. In order to reduce the chance of bugs being introduced into the code, and ensure that all parts of the code are working, we recommend that : 
 +  * all developers complete a regression test before committing any changes to SVN. 
 +  * all users complete a test before using a self-compiled binary for their projects
  
 ==== Dashboard ==== ==== Dashboard ====
-A number of regtests are run automatically by various members of our community. The results of these tests are collected centrally at the [[http://dashboard.cp2k.org | Dashboard ]]. If errors are detected, the developer responsible for the change should fix it immediately.+A number of regtests are run automatically by various members of our community. The results of these tests are collected centrally at the [[http://dashboard.cp2k.org | Dashboard ]]. If errors are detected, the developer responsible for the change should fix it immediately. The output logs provide the arch file used for these tests, which might suggest useful settings for that particular architecture.
  
 ==== Code Coverage ==== ==== Code Coverage ====
-We aim that the regression test suite covers all the functionality of CP2K. For this purpose we regularly create [[http://www.cp2k.org/static/coverage/|Coverage Reports]] of the testsuit. If you see parts of the code which are not well tested, please contribute to improving coverage by writing new tests!+We aim that the regression test suite covers all the functionality of CP2K. For this purpose we regularly create [[http://www.cp2k.org/static/coverage/|Coverage Reports]] of the test-suite. If you see parts of the code which are not well tested, please contribute to improving coverage by writing new tests!
  
 ===== How does it work? ===== ===== How does it work? =====
  
-The regression test suite is run using the [[src>cp2k/tools/regtesting/do_regtest | do_regtest]] script. +The regression test suite is run using the [[src>tools/regtesting/do_regtest | do_regtest]] script. 
 It performs the following tasks: It performs the following tasks:
    * update to the current SVN version (including new tests)    * update to the current SVN version (including new tests)
Line 21: Line 23:
  
 ===== Running the regtests ===== ===== Running the regtests =====
 +
 +==== Step 0: make based testing ====
 +
 +  * If you are able to build and run cp2k on the local machine the easiest way is to start the regtesting by running ''make ARCH=... VERSION=... test''
 +  * Be careful about the value of ''-j'' parameter you will use; running too many tests in parallel can cause tests to fail due to lack of system and/or GPU memory.
 +  * If this fails (e.g. on batch systems), continue with Step 1 else go to interpretation.
  
 ==== Step 1: Preparation ==== ==== Step 1: Preparation ====
-  * You must be able to build and run cp2k on the given machine, the rest should be 'easy' 
   * Decide on a directory for doing the regtest, there will be plenty of files in this dir (after a while) so make it something like ''$HOME/rt''   * Decide on a directory for doing the regtest, there will be plenty of files in this dir (after a while) so make it something like ''$HOME/rt''
   * Checkout a version of cp2k into ''$HOME/rt''. If you set up your svn account so that you don't need to provide a password for doing svn update, things will be more convenient later on.   * Checkout a version of cp2k into ''$HOME/rt''. If you set up your svn account so that you don't need to provide a password for doing svn update, things will be more convenient later on.
   * Set up the arch files so that you can cleanly build cp2k (test this)   * Set up the arch files so that you can cleanly build cp2k (test this)
  
-==== Step 2: Configuration ====+==== Step 2: Running ==== 
 +<code> 
 +$ tools/regtesting/do_regtest --help 
 +Usage: do_regtest [OPTION] 
 +Run the CP2K regression test suite 
 +Example: do_regtest -c my_regtest.conf
  
-The //do_regtest// script is controlled via a config file. It should contain the following variables:+General: 
 +  -h, -help, --help         print this help screen. 
 +  -c, -config FILE          read any of the following configuration switches from FILE. 
 +  -cp2kdir PATH             location of cp2k source tree relative to current working directory.
  
-^ Config Variable ^ Default ^ Meaning  ^ +Svn: 
-| ''dir_base''    |         | the base directory for testing (e.g. $HOME/rt)| +  -svndate DATE             checkout svn revision of given DATE. Default: use trunk. 
-| ''FORT_C_NAME'' |         | compiler selection (e.g. intel)| +  -nosvn                    do not run svn update. Default: off. 
-| ''cp2k_version''|         | sopt,sdbg,popt....| +  -noemptycheck             run tests even when no changes in svn were made. Default: on.
-| ''dir_triplet'' |         | the result of tools/get_arch_name, where the executable can be found | +
-| ''cp2k_dir''    |  cp2k   | normally cp2k| +
-| ''maxtasks''    |         | Maximum number of CP2K threads to execute concurrently (== total number of CPU cores to be used for regtesting) | +
-| ''emptycheck''  |  NO     | useful for automatic testers, no testing if nothing changed in SVN (YES/NO)| +
-| ''leakcheck''   |  NO     | if using gfortran with the LeakSanitizer, set this variable to "YES" in order to get memory leak checking| +
-| ''cp2k_prefix'' |         | | +
-| ''cp2k_postfix''|         | | +
-| ''make''|         | | +
-| ''awk''|         | | +
-| ''datum_full''|         | | +
-| ''datum_short''|         | | +
-| ''default_err_tolerance''| 1.0E-14 | |+
  
 +Build:
 +  -version VERSION          VERSION passed to make. Default: sopt.
 +  -arch ARCH                ARCH passed to make. Default: Linux-x86-64-gfortran.
 +  -quick                    rebuild if needed, but without realclean. Default: off.
 +  -nobuild                  do not build cp2k, rely on user's build. Default: off.
  
-==== Step 3: Execution ==== +Runtime: 
-Normally, you just have to call the //do_regtest// script with your config-file like this:+  -mpiranks NRANKS          number of mpi-ranks. Default: 2 for parallel versions, 1 for serial. 
 +  -ompthreads NTHREADS      number of OpenMP threads. Default: 2 for smp versions otherwise 1. 
 +  -maxtasks NPROCS          total number of processor to use. Default: `nproc --all`. 
 +  -jobmaxtime SECONDS       maximum execution time of a single test. Default 600.
  
-<code> +Testing: 
-$ ./cp2k/tools/regtesting/do_regtest -config <path_to_your_config> +  -noreset                  do not reset the reference outputs. Default: off. 
-</code>+  -skiptest                 do not run test, only svn update and build. Default: off. 
 +  -skipdir TESTDIR          do not run tests in TESTDIR. This switch can repeated. 
 +  -restrictdir TESTDIR      run only tests in TESTDIR. This switch can repeated. 
 +  -retest                   run only tests in directories, which had failing tests in previous run.
  
-However, the //do_regtest// script supports a number of addional command line switches. Most of them can also be included into the config file. +Exit codes: 
-^ Command Line Switch    ^ Meaning   ^ +    0   clean exit with testing 
-| ''-nosvn''             | do not access the SVN for any updating, makes regtesting fully local | +    1   problem with svn update 
-| ''-quick''             | rebuild the code if needed, but do not perform a realclean before (noquick is not needed anymore) | +    3   problem with realclean 
-| ''-noreset''           | do not reset the reference outputs automatically | +    4   build errors 
-| ''-svndate <str>''     | specify any string to svn update (most likely used as "2005-02-17") | +    5   problem with retest option - no TEST directory with latest test results found 
-| ''-skipdir <dir>''     | this switch can repeat, exclude certain dirs from regtesting, useful to speed-up regtesting after very localised changes (e.g. -skipdir QS/regtest) | +    6   problem with retest option - no error summary exists in the last TEST directory 
-| ''-restrictdir <dir>'' | this switch can repeat, restrict regtesting to certain dirs, useful to speed-up regtesting after very localised changes (e.g. -restrictdir QS/regtest) | +    7   reference directory is locked 
-| ''-config <path>''     | loads a site/compiler/environment specific configuration | +    8   ctrl-C (SIGINT) and various other signals trapped 
-| ''-retest''            | regtesting will be restricted only to those directories that contained failed tests in the previous run | +  100   no svn changes since last run - clean exit without testing
-| ''-noemptycheck''      | use this to force regression testing even if no changes exist in SVN | +
-| ''-nobuild''           | do not build cp2k, but use one built by the user (turns on quick option automatically) | +
- +
-The //do_regtest// script has the following possible exists statuses: +
- +
-^ Exit Code   ^Meaning                                                                          ^ +
-|    0        | clean exit with testing                                                         | +
-|    1        | problem with svn update                                                         | +
-|    3        | problem with realclean                                                          | +
-|    4        | build errors                                                                    | +
-|    5        | problem with retest option - no TEST directory with latest test results found   | +
-|    6        | problem with retest option - no error summary exists in the last TEST directory | +
-|    7        | reference directory is locked                                                   | +
-|    8        | ctrl-C (SIGINT) and various other signals trapped                               | +
-|    100      | no svn changes since last run - clean exit without testing                      |+
  
 +For more information visit: <http://cp2k.org/dev:regtesting>
 +</code>
  
-==== Step 4: Interpretation ====+==== Step 3: Interpretation ====
 A test results can be any of the following: A test results can be any of the following:
 ^  Test Result         ^  Meaning | ^  Test Result         ^  Meaning |
Line 97: Line 98:
  
 ===== Adding and Resetting Tests===== ===== Adding and Resetting Tests=====
-The test-suite is fully controlled by the following files in the [[src>cp2k/tests]] directories+The test-suite is fully controlled by the following files in the [[src>tests]] directories
  
 ^ File Name            ^ Content   | ^ File Name            ^ Content   |
 | ''TEST_DIRS''        | is just a list of directories that contain tests. You can add your directory here. | | ''TEST_DIRS''        | is just a list of directories that contain tests. You can add your directory here. |
 | ''TEST_FILES''       | the list of input files that need to be executed. You can add your file name here. Adding a comment about what it tests might help later debugging problems if a regtest fails | | ''TEST_FILES''       | the list of input files that need to be executed. You can add your file name here. Adding a comment about what it tests might help later debugging problems if a regtest fails |
-| ''TEST_FILES_RESET'' | you can add files for which the reference output became invalid (e.g. bug fix) to this list fo files. However be absolutely sure that the change is due to a bug fix, do not reset these that fail because of unclear reasons. Try to add a comment to the svn message and/or the file itself |+| ''TEST_FILES_RESET'' | you can add files for which the reference output became invalid (e.g. bug fix) to this list of files. However be absolutely sure that the change is due to a bug fix, do not reset these that fail because of unclear reasons. Try to add a comment to the svn message and/or the file itself |
 | ''TEST_TYPES''       | this file allows you to create a new test type. I.e. to specify for which words should be grepped and what field should be used in the numerical comparison. | | ''TEST_TYPES''       | this file allows you to create a new test type. I.e. to specify for which words should be grepped and what field should be used in the numerical comparison. |
  
  
dev/regtesting.txt · Last modified: 2023/10/19 14:03 by krack