User Tools

Site Tools


dev:regtesting

Differences

This shows you the differences between two versions of the page.

Link to this comparison view

Both sides previous revisionPrevious revision
Next revision
Previous revision
Next revisionBoth sides next revision
dev:regtesting [2018/10/08 19:58] oschuettdev:regtesting [2019/01/13 11:30] oschuett
Line 1: Line 1:
 ====== CP2K Regression Testing ====== ====== CP2K Regression Testing ======
  
-CP2K comes with over 2500 test input files (located in [[src>cp2k/tests]]) which serve as both examples on how to use the many features in CP2K and also as a method for developers to test modifications and extensions to CP2K. In order to reduce the chance of bugs being introduced into the code, and ensure that all parts of the code are working, we recommend that +CP2K comes with over 3000 test input files (located in [[src>tests]]) which serve as both examples on how to use the many features in CP2K and also as a method for developers to test modifications and extensions to CP2K. In order to reduce the chance of bugs being introduced into the code, and ensure that all parts of the code are working. We also recommend that all users complete a test before using a self-compiled binary for their projects.
-  * all developers complete a regression test before committing any changes to SVN. +
-  * all users complete a test before using a self-compiled binary for their projects+
  
 ==== Dashboard ==== ==== Dashboard ====
Line 15: Line 13:
 The regression test suite is run using the [[src>tools/regtesting/do_regtest | do_regtest]] script.  The regression test suite is run using the [[src>tools/regtesting/do_regtest | do_regtest]] script. 
 It performs the following tasks: It performs the following tasks:
-   * update to the current SVN version (including new tests) 
    * performs a realclean build of the source    * performs a realclean build of the source
    * executes a list of tests    * executes a list of tests
Line 32: Line 29:
 ==== Step 1: Preparation ==== ==== Step 1: Preparation ====
   * Decide on a directory for doing the regtest, there will be plenty of files in this dir (after a while) so make it something like ''$HOME/rt''   * Decide on a directory for doing the regtest, there will be plenty of files in this dir (after a while) so make it something like ''$HOME/rt''
-  * Checkout a version of cp2k into ''$HOME/rt''. If you set up your svn account so that you don't need to provide a password for doing svn update, things will be more convenient later on.+  * Clone a version of cp2k into ''$HOME/rt''.
   * Set up the arch files so that you can cleanly build cp2k (test this)   * Set up the arch files so that you can cleanly build cp2k (test this)
  
Line 46: Line 43:
   -c, -config FILE          read any of the following configuration switches from FILE.   -c, -config FILE          read any of the following configuration switches from FILE.
   -cp2kdir PATH             location of cp2k source tree relative to current working directory.   -cp2kdir PATH             location of cp2k source tree relative to current working directory.
- +  -dirout PATH              root path of output folder (defaultcurrent working directory)
-Svn: +  -mpiexec EXE              name of the executable to run mpi-ranks. Default: mpiexec.
-  -svndate DATE             checkout svn revision of given DATE. Defaultuse trunk+
-  -nosvn                    do not run svn update. Default: off. +
-  -noemptycheck             run tests even when no changes in svn were made. Default: on.+
  
 Build: Build:
Line 61: Line 55:
   -mpiranks NRANKS          number of mpi-ranks. Default: 2 for parallel versions, 1 for serial.   -mpiranks NRANKS          number of mpi-ranks. Default: 2 for parallel versions, 1 for serial.
   -ompthreads NTHREADS      number of OpenMP threads. Default: 2 for smp versions otherwise 1.   -ompthreads NTHREADS      number of OpenMP threads. Default: 2 for smp versions otherwise 1.
-  -maxtasks NPROCS          total number of processor to use. Default: `nproc --all`.+  -maxtasks NPROCS          total number of processor to use. Default: `nproc`. 
 +  -maxbuildtasks NPROCS     total number of processor to use for compilation. Default taken from -maxtasks.
   -jobmaxtime SECONDS       maximum execution time of a single test. Default 600.   -jobmaxtime SECONDS       maximum execution time of a single test. Default 600.
 +  -farming                  test via a single farming run (only for parallel cp2k).
  
 Testing: Testing:
   -noreset                  do not reset the reference outputs. Default: off.   -noreset                  do not reset the reference outputs. Default: off.
-  -skiptest                 do not run test, only svn update and build. Default: off.+  -shard ISHARD NSHARDS     do sharding: Divide the test dirs into NSHARD subsets and only run the ISHARD-th set. 
 +  -skiptest                 do not run test, only build. Default: off. 
 +  -skipunittest             do not run unit tests. Default: off.
   -skipdir TESTDIR          do not run tests in TESTDIR. This switch can repeated.   -skipdir TESTDIR          do not run tests in TESTDIR. This switch can repeated.
   -restrictdir TESTDIR      run only tests in TESTDIR. This switch can repeated.   -restrictdir TESTDIR      run only tests in TESTDIR. This switch can repeated.
Line 73: Line 71:
 Exit codes: Exit codes:
     0   clean exit with testing     0   clean exit with testing
-    1   problem with svn update 
     3   problem with realclean     3   problem with realclean
     4   build errors     4   build errors
Line 80: Line 77:
     7   reference directory is locked     7   reference directory is locked
     8   ctrl-C (SIGINT) and various other signals trapped     8   ctrl-C (SIGINT) and various other signals trapped
-  100   no svn changes since last run - clean exit without testing 
  
 For more information visit: <http://cp2k.org/dev:regtesting> For more information visit: <http://cp2k.org/dev:regtesting>
Line 98: Line 94:
  
 ===== Adding and Resetting Tests===== ===== Adding and Resetting Tests=====
-The test-suite is fully controlled by the following files in the [[src>cp2k/tests]] directories+The test-suite is fully controlled by the following files in the [[src>tests]] directories
  
 ^ File Name            ^ Content   | ^ File Name            ^ Content   |
 | ''TEST_DIRS''        | is just a list of directories that contain tests. You can add your directory here. | | ''TEST_DIRS''        | is just a list of directories that contain tests. You can add your directory here. |
 | ''TEST_FILES''       | the list of input files that need to be executed. You can add your file name here. Adding a comment about what it tests might help later debugging problems if a regtest fails | | ''TEST_FILES''       | the list of input files that need to be executed. You can add your file name here. Adding a comment about what it tests might help later debugging problems if a regtest fails |
-| ''TEST_FILES_RESET'' | you can add files for which the reference output became invalid (e.g. bug fix) to this list of files. However be absolutely sure that the change is due to a bug fix, do not reset these that fail because of unclear reasons. Try to add a comment to the svn message and/or the file itself |+| ''TEST_FILES_RESET'' | you can add files for which the reference output became invalid (e.g. bug fix) to this list of files. However be absolutely sure that the change is due to a bug fix, do not reset these that fail because of unclear reasons. Try to add a comment to the git message and/or the file itself |
 | ''TEST_TYPES''       | this file allows you to create a new test type. I.e. to specify for which words should be grepped and what field should be used in the numerical comparison. | | ''TEST_TYPES''       | this file allows you to create a new test type. I.e. to specify for which words should be grepped and what field should be used in the numerical comparison. |
  
  
dev/regtesting.txt · Last modified: 2023/10/19 14:03 by krack