dev:regtesting

# Differences

This shows you the differences between two versions of the page.

 dev:regtesting [2014/10/12 06:36]krack Adjust number dev:regtesting [2019/01/13 11:30] (current)oschuett Both sides previous revision Previous revision 2019/01/13 11:30 oschuett 2018/10/08 19:59 oschuett 2018/10/08 19:58 oschuett 2015/11/18 09:37 rivanvx Better wording2015/11/18 09:37 rivanvx Add a note about -j in make test2015/09/01 08:04 vondele [Step 1: Preparation] 2015/09/01 07:59 vondele [Dashboard] 2015/09/01 07:58 vondele [CP2K Regression Testing] 2015/09/01 07:57 vondele [CP2K Regression Testing] 2015/07/09 14:50 oschuett 2015/05/31 10:33 [Code Coverage] 2015/05/29 10:29 [Adding and Resetting Tests] 2015/04/28 14:23 oschuett 2015/04/28 14:21 oschuett 2015/04/28 13:34 oschuett 2015/04/28 13:30 oschuett 2015/04/28 13:29 oschuett 2015/02/27 10:08 2014/10/23 07:51 krack Update link to new location2014/10/23 06:47 krack Update link to new location2014/10/12 06:36 krack Adjust number2014/06/26 11:40 krack Add ssmp regtest2014/06/26 10:28 krack Update link2014/05/30 10:22 krack Add link to regtest file2014/05/11 19:31 krack Version number2014/05/11 19:31 krack G95 end of life2014/05/09 09:21 vondele 2014/03/06 09:45 krack Update link for regression plot2014/02/26 15:02 krack Change line to heading2014/02/08 21:51 oschuett regtesting renamed to dev:regtesting2014/01/31 15:00 vondele add svn commit feed2014/01/16 09:23 oschuett 2014/01/16 09:21 oschuett 2014/01/13 09:16 external edit2013/12/11 08:49 krack 2013/12/11 08:37 krack Add test result2013/09/12 08:38 ibethune created Next revision Previous revision 2019/01/13 11:30 oschuett 2018/10/08 19:59 oschuett 2018/10/08 19:58 oschuett 2015/11/18 09:37 rivanvx Better wording2015/11/18 09:37 rivanvx Add a note about -j in make test2015/09/01 08:04 vondele [Step 1: Preparation] 2015/09/01 07:59 vondele [Dashboard] 2015/09/01 07:58 vondele [CP2K Regression Testing] 2015/09/01 07:57 vondele [CP2K Regression Testing] 2015/07/09 14:50 oschuett 2015/05/31 10:33 [Code Coverage] 2015/05/29 10:29 [Adding and Resetting Tests] 2015/04/28 14:23 oschuett 2015/04/28 14:21 oschuett 2015/04/28 13:34 oschuett 2015/04/28 13:30 oschuett 2015/04/28 13:29 oschuett 2015/02/27 10:08 2014/10/23 07:51 krack Update link to new location2014/10/23 06:47 krack Update link to new location2014/10/12 06:36 krack Adjust number2014/06/26 11:40 krack Add ssmp regtest2014/06/26 10:28 krack Update link2014/05/30 10:22 krack Add link to regtest file2014/05/11 19:31 krack Version number2014/05/11 19:31 krack G95 end of life2014/05/09 09:21 vondele 2014/03/06 09:45 krack Update link for regression plot2014/02/26 15:02 krack Change line to heading2014/02/08 21:51 oschuett regtesting renamed to dev:regtesting2014/01/31 15:00 vondele add svn commit feed2014/01/16 09:23 oschuett 2014/01/16 09:21 oschuett 2014/01/13 09:16 external edit2013/12/11 08:49 krack 2013/12/11 08:37 krack Add test result2013/09/12 08:38 ibethune created Line 1: Line 1: - ==== CP2K Regression Testing ===== + ====== CP2K Regression Testing ​====== - === Regression Testing Script === + CP2K comes with over 3000 test input files (located in [[src>​tests]]) which serve as both examples on how to use the many features in CP2K and also as a method for developers to test modifications and extensions to CP2K. In order to reduce the chance of bugs being introduced into the code, and ensure that all parts of the code are working. We also recommend that all users complete a test before using a self-compiled binary for their projects. - CP2K comes with over 2500 test input files (located in ''​cp2k/​tests''​) which serve as both examples on how to use the many features in CP2K and also as a method for developers to test modifications and extensions to CP2K. In order to reduce ​the chance of bugs being introduced into the code, and ensure that all parts of the code are working, we recommend ​that all developers complete a successful run of the full regression test suite before committing changes to SVN. + ==== Dashboard ==== + A number of regtests are run automatically by various members of our community. The results of these tests are collected centrally at the [[http://​dashboard.cp2k.org | Dashboard ]]. If errors are detected, ​the developer responsible for the change should fix it immediately. The output logs provide ​the arch file used for these tests, which might suggest useful settings for that particular architecture. - The regression test suite is run using the ''​do_regtest''​ script in ''​cp2k/tools/regtesting''​.  ​Instructions are included at the top of the script. + ==== Code Coverage ==== + We aim that the regression test suite covers all the functionality of CP2K. For this purpose we regularly create [[http://www.cp2k.org/​static/​coverage/​|Coverage Reports]] ​of the test-suite. If you see parts of the code which are not well tested, please contribute to improving coverage by writing new tests! - In addition, when new features are added to CP2K, please add new tests which cover those features and commit them to SVN along with the code. + ===== How does it work? ===== - === Automated Regression Tester === + The regression test suite is run using the [[src>​tools/​regtesting/​do_regtest | do_regtest]] script. + It performs the following tasks: + * performs a realclean build of the source + * executes a list of tests + * compares the results (outputs) with those of the last known result (reference) + * produces a summary - An automated regression test service is available at http://​cp2k-www.epcc.ed.ac.uk .  This runs the test suite on a range of architectures and versions of CP2K every time a change is made to SVN.  If errors are found, the developer responsible for the change will receive and email notifying them of the problem(s) and these should be fixed immediately. - We plan to extend ​the range of platforms, compilers etc. that are covered by the automated regression tester in future. ​ If you would like to set up automated regression testing on a local machine, or have other ideas about how the test service can be improved, please email [[mailto:​ibethuneNOSPAM@epcc.ed.ac.uk| Iain Bethune ]](remove NOSPAM) from address before sending + ===== Running ​the regtests ===== - === CP2K Code Coverage ​=== + ==== Step 0: make based testing ==== - We aim that the regression test suite covers all the functionality of CP2K.  ​Code coverage reports from recent runs of the tests are available at  http://www.cp2k.org/​static/​coverage/ ​If you see parts of the code which are not well tested, please contribute ​to improving coverage by writing new tests! + * If you are able to build and run cp2k on the local machine ​the easiest way is to start the regtesting by running ''​make ARCH=... VERSION=... test''​ + * Be careful about the value of ''​-j''​ parameter you will use; running too many tests in parallel can cause tests to fail due to lack of system and/or GPU memory. + * If this fails (e.g. on batch systems), continue with Step 1 else go to interpretation. - === Results of the Automatic Regression Tester ​=== + ==== Step 1: Preparation ==== - * [[http://sourceforge.net/p/cp2k/​code/​HEAD/​tree/​trunk/​cp2k/​arch/​Linux-x86-64-gfortran-regtest.pdbg | GFortran compiler ​(v4.9.1): ARCH=Linux-x86-64-gfortran-regtest VERSION=pdbg]] + * Decide on a directory for doing the regtest, there will be plenty of files in this dir (after a while) so make it something like ''​$HOME/rt''​ + * Clone a version of cp2k into ''​$HOME/rt''​. + * Set up the arch files so that you can cleanly build cp2k (test this) - [[http://cp2k.web.psi.ch/​regtest/​trunk/​Linux-x86-64-gfortran-regtest/​pdbg|{{http://​cp2k.web.psi.ch/​regtest/​trunk/​Linux-x86-64-gfortran-regtest/​pdbg/​regplot.png}}]] + ==== Step 2: Running ==== + <​code>​ + \$ tools/regtesting/do_regtest ​--help + Usage: do_regtest [OPTION] + Run the CP2K regression test suite + Example: do_regtest ​-c my_regtest.conf - * [[http://​sourceforge.net/​p/​cp2k/​code/​HEAD/​tree/​trunk/​cp2k/​arch/​Linux-x86-64-gfortran-regtest.ssmp | GFortran compiler (v4.9.1): ARCH=Linux-x86-64-gfortran-regtest VERSION=ssmp]] + General: + ​-h, -help, --help         print this help screen. + -c, -config FILE          read any of the following configuration switches from FILE. + -cp2kdir PATH             ​location of cp2k source tree relative to current working directory. + -dirout PATH              root path of output folder (default: current working directory). + ​-mpiexec EXE              name of the executable to run mpi-ranks. Default: mpiexec. - [[http://cp2k.web.psi.ch/​regtest/​trunk/​Linux-x86-64-gfortran-regtest/​ssmp|{{http://cp2k.web.psi.ch/​regtest/​trunk/​Linux-x86-64-gfortran-regtest/ssmp/regplot.png}}]] + Build: + -version VERSION ​         VERSION passed to make. Default: sopt. + -arch ARCH                ARCH passed to make. Default: ​Linux-x86-64-gfortran. + ​-quick                    rebuild if needed, but without realclean. Default: off. + -nobuild ​                 do not build cp2k, rely on user's build. Default: off. + + Runtime: + -mpiranks NRANKS ​         number of mpi-ranks. Default: 2 for parallel versions, 1 for serial. + ​-ompthreads NTHREADS ​     number of OpenMP threads. Default: 2 for smp versions otherwise 1. + ​-maxtasks NPROCS ​         total number of processor to use. Default: nproc. + ​-maxbuildtasks NPROCS ​    total number of processor to use for compilation. Default taken from -maxtasks. + -jobmaxtime SECONDS ​      ​maximum execution time of a single test. Default 600. + -farming ​                 test via a single farming run (only for parallel cp2k). + + Testing: + -noreset ​                 do not reset the reference outputs. Default: off. + -shard ISHARD NSHARDS ​    do sharding: Divide the test dirs into NSHARD subsets and only run the ISHARD-th set. + -skiptest ​                do not run test, only build. Default: off. + -skipunittest ​            do not run unit tests. Default: off. + -skipdir TESTDIR ​         do not run tests in TESTDIR. This switch can repeated. + -restrictdir TESTDIR ​     run only tests in TESTDIR. This switch can repeated. + -retest ​                  run only tests in directories,​ which had failing tests in previous run. + + Exit codes: + 0   clean exit with testing + 3   ​problem with realclean + 4   build errors + 5   ​problem with retest option - no TEST directory with latest test results found + 6   ​problem with retest option - no error summary exists in the last TEST directory + 7   ​reference directory is locked + 8   ​ctrl-C (SIGINT) and various other signals trapped + + For more information visit: ​ + ​ + + ==== Step 3: Interpretation ==== + A test results can be any of the following:​ + ^  Test Result ​        ​^ ​ Meaning | + | ''​OK'' ​              | if the results match those of a previous run precisely. The execution time is also given. | + | ''​NEW'' ​             | if they have not been executed previously. The reference result is generated automatically in this run. Tests can also be NEW if they have been reset, i.e. been newly added to the TEST_FILES_RESET files. | + | ''​RUNTIME FAILURE'' ​ | if they stopped unexpectedly (e.g. core dump, or stop) | + | ''​WRONG RESULT'' ​    | if they produce a result that deviates (even a tiny bit) from an old reference | + + The last two outcomes generally mean that a bug has been introduced, which requires investigation. + Since regtesting only yields information relative to a previously known result, it is most useful to do a regtest before and after you make changes. To allow per-test numerical difference higher than that set as a default, add third column in appropriate TEST_FILES file with a relative value of the difference. + + + ===== Adding and Resetting Tests===== + The test-suite is fully controlled by the following files in the [[src>​tests]] directories + + ^ File Name            ^ Content ​  | + | ''​TEST_DIRS'' ​       | is just a list of directories that contain tests. You can add your directory here. | + | ''​TEST_FILES'' ​      | the list of input files that need to be executed. You can add your file name here. Adding a comment about what it tests might help later debugging problems if a regtest fails | + | ''​TEST_FILES_RESET''​ | you can add files for which the reference output became invalid (e.g. bug fix) to this list of files. However be absolutely sure that the change is due to a bug fix, do not reset these that fail because of unclear reasons. Try to add a comment to the git message and/or the file itself | + | ''​TEST_TYPES'' ​      | this file allows you to create a new test type. I.e. to specify for which words should be grepped and what field should be used in the numerical comparison. | - === Latest commits === - {{rss>​http://​sourceforge.net/​p/​cp2k/​code/​feed 20 author date 10m}}