CP2K Open Source Molecular Dynamics

User Tools

Site Tools

Trace: • docs • c2h2_bond_energy_2018 • sublime • gfn1xtb • dye_tio • mp2 • regtesting • xas_tdp • quickstep • pwtools
dev:regtesting

Differences

This shows you the differences between two versions of the page.

Link to this comparison view

Both sides previous revisionPrevious revision
Next revision		Previous revision
dev:regtesting [2018/10/08 19:59] – oschuettdev:regtesting [2023/10/19 14:03] (current) – Update flags krack
Line 1: Line 1:
 ====== CP2K Regression Testing ====== ====== CP2K Regression Testing ======
  
-CP2K comes with over 2500 test input files (located in [[src>tests]]) which serve as both examples on how to use the many features in CP2K and also as a method for developers to test modifications and extensions to CP2K. In order to reduce the chance of bugs being introduced into the code, and ensure that all parts of the code are working, we recommend that : +CP2K comes with over 3000 test input files (located in [[src>tests]]) which serve as both examples on how to use the many features in CP2K and also as a method for developers to test modifications and extensions to CP2K. In order to reduce the chance of bugs being introduced into the code, and ensure that all parts of the code are working. We also recommend that all users complete a test before using a self-compiled binary for their projects.
-  * all developers complete a regression test before committing any changes to SVN. +
-  * all users complete a test before using a self-compiled binary for their projects+
  
 ==== Dashboard ==== ==== Dashboard ====
Line 13: Line 11:
 ===== How does it work? ===== ===== How does it work? =====
  
-The regression test suite is run using the [[src>tools/regtesting/do_regtest | do_regtest]] script. +The regression test suite is run using the [[src>tests/do_regtest.py | do_regtest]] script. 
 It performs the following tasks: It performs the following tasks:
-   * update to the current SVN version (including new tests) 
    * performs a realclean build of the source    * performs a realclean build of the source
    * executes a list of tests    * executes a list of tests
Line 26: Line 23:
 ==== Step 0: make based testing ==== ==== Step 0: make based testing ====
  
-  * If you are able to build and run cp2k on the local machine the easiest way is to start the regtesting by running ''make ARCH=... VERSION=... test''+  * If you are able to build and run cp2k on the local machine the easiest way is to start the regtesting by running ''make ARCH=... VERSION=... test''. The options listed in Step 2 can also be specified with ''make'' via ''TESTOPTS=<options>'', e.g. ''TESTOPTS+="--mpiranks 4 --ompthreads 4"'' requires 4 MPI ranks and 4 OpenMP threads for each MPI rank.
   * Be careful about the value of ''-j'' parameter you will use; running too many tests in parallel can cause tests to fail due to lack of system and/or GPU memory.   * Be careful about the value of ''-j'' parameter you will use; running too many tests in parallel can cause tests to fail due to lack of system and/or GPU memory.
   * If this fails (e.g. on batch systems), continue with Step 1 else go to interpretation.   * If this fails (e.g. on batch systems), continue with Step 1 else go to interpretation.
Line 32: Line 29:
 ==== Step 1: Preparation ==== ==== Step 1: Preparation ====
   * Decide on a directory for doing the regtest, there will be plenty of files in this dir (after a while) so make it something like ''$HOME/rt''   * Decide on a directory for doing the regtest, there will be plenty of files in this dir (after a while) so make it something like ''$HOME/rt''
-  * Checkout a version of cp2k into ''$HOME/rt''. If you set up your svn account so that you don't need to provide a password for doing svn update, things will be more convenient later on.+  * Clone a version of cp2k into ''$HOME/rt''.
   * Set up the arch files so that you can cleanly build cp2k (test this)   * Set up the arch files so that you can cleanly build cp2k (test this)
  
 ==== Step 2: Running ==== ==== Step 2: Running ====
 <code> <code>
-$ tools/regtesting/do_regtest --help +$ tests/do_regtest.py -h 
-Usage: do_regtest [OPTION] +usage: do_regtest.py [-h] [--mpiranks MPIRANKS] [--ompthreads OMPTHREADS] 
-Run the CP2K regression test suite +                     [--maxtasks MAXTASKS] [--num_gpus NUM_GPUS] 
-Example: do_regtest -c my_regtest.conf+                     [--timeout TIMEOUT] [--maxerrors MAXERRORS] 
 +                     [--mpiexec MPIEXEC] [--smoketest] [--valgrind] 
 +                     [--keepalive] [--flagslow] [--debug] 
 +                     [--restrictdir RESTRICTDIR] [--skipdir SKIPDIR] 
 +                     [--workbasedir WORKBASEDIR] arch version
  
-General: +Runs CP2K regression test suite.
-  -h, -help, --help         print this help screen. +
-  -c, -config FILE          read any of the following configuration switches from FILE. +
-  -cp2kdir PATH             location of cp2k source tree relative to current working directory.+
  
-Svn: +positional arguments: 
-  -svndate DATE             checkout svn revision of given DATE. Default: use trunk. +  arch 
-  -nosvn                    do not run svn update. Default: off. +  version
-  -noemptycheck             run tests even when no changes in svn were made. Default: on.+
  
-Build: +options: 
-  -version VERSION          VERSION passed to make. Default: sopt. +  -h, --help            show this help message and exit 
-  -arch ARCH                ARCH passed to make. Default: Linux-x86-64-gfortran. +  --mpiranks MPIRANKS 
-  -quick                    rebuild if needed, but without realclean. Default: off. +  --ompthreads OMPTHREADS 
-  -nobuild                  do not build cp2k, rely on user's build. Default: off. +  --maxtasks MAXTASKS 
- +  --num_gpus NUM_GPUS 
-Runtime: +  --timeout TIMEOUT 
-  -mpiranks NRANKS          number of mpi-ranks. Default: 2 for parallel versions, 1 for serial. +  --maxerrors MAXERRORS 
-  -ompthreads NTHREADS      number of OpenMP threads. Default: 2 for smp versions otherwise 1. +  --mpiexec MPIEXEC 
-  -maxtasks NPROCS          total number of processor to use. Default: `nproc --all`. +  --smoketest           Runs only the first test of each directory. 
-  -jobmaxtime SECONDS       maximum execution time of a single test. Default 600. +  --valgrind            Runs tests under Valgrind memcheck. Best used together with --keepalive. 
- +  --keepalive           Use a persistent cp2k-shell process to reduce startup time. 
-Testing: +  --flagslow            Flag slow tests in the final summary and status report. 
-  -noreset                  do not reset the reference outputs. Default: off. +  --debug 
-  -skiptest                 do not run test, only svn update and build. Default: off. +  --restrictdir RESTRICTDIR 
-  -skipdir TESTDIR          do not run tests in TESTDIR. This switch can repeated. +  --skipdir SKIPDIR 
-  -restrictdir TESTDIR      run only tests in TESTDIR. This switch can repeated. +  --workbasedir WORKBASEDIR
-  -retest                   run only tests in directories, which had failing tests in previous run. +
- +
-Exit codes: +
-    0   clean exit with testing +
-    1   problem with svn update +
-    3   problem with realclean +
-    4   build errors +
-    5   problem with retest option - no TEST directory with latest test results found +
-    6   problem with retest option - no error summary exists in the last TEST directory +
-    7   reference directory is locked +
-    8   ctrl-C (SIGINT) and various other signals trapped +
-  100   no svn changes since last run - clean exit without testing +
- +
-For more information visit: <http://cp2k.org/dev:regtesting>+
 </code> </code>
  
Line 103: Line 86:
 | ''TEST_DIRS''        | is just a list of directories that contain tests. You can add your directory here. | | ''TEST_DIRS''        | is just a list of directories that contain tests. You can add your directory here. |
 | ''TEST_FILES''       | the list of input files that need to be executed. You can add your file name here. Adding a comment about what it tests might help later debugging problems if a regtest fails | | ''TEST_FILES''       | the list of input files that need to be executed. You can add your file name here. Adding a comment about what it tests might help later debugging problems if a regtest fails |
-| ''TEST_FILES_RESET'' | you can add files for which the reference output became invalid (e.g. bug fix) to this list of files. However be absolutely sure that the change is due to a bug fix, do not reset these that fail because of unclear reasons. Try to add a comment to the svn message and/or the file itself |+| ''TEST_FILES_RESET'' | you can add files for which the reference output became invalid (e.g. bug fix) to this list of files. However be absolutely sure that the change is due to a bug fix, do not reset these that fail because of unclear reasons. Try to add a comment to the git message and/or the file itself |
 | ''TEST_TYPES''       | this file allows you to create a new test type. I.e. to specify for which words should be grepped and what field should be used in the numerical comparison. | | ''TEST_TYPES''       | this file allows you to create a new test type. I.e. to specify for which words should be grepped and what field should be used in the numerical comparison. |
  
 +===== Run with sbatch =====
 +
 +The ''%%do_regtest%%'' script in ''%%tools/regtesting%%'' is often run from ''%%make VERSION=... ARCH=... test%%'' where it sets up everything just nicely. On a cluster you usually do want to separate the steps and invoke ''%%do_regtest%%'' manually.
 +
 +What you need:
 +
 +  * ''%%sbatch%%'' template script
 +  * a CP2K source tree with a built CP2K
 +
 +==== Instructions ====
 +
 +The way the regtest script works is that it goes through all the directories (for example ''%%tests/QS/regtest-admm-1/%%'') and launches all tests in that directory. After each directories tests are started it checks whether the number of maximum tasks is reached, if not it also spawns the tests from the next directory. If the maximum number of tasks to run has been reached it waits until enough of them have finished to spawn tests from the next directory. Since the tests are usually rather short this procedure seldomly causes oversubscription.
 +
 +Also, ''%%srun%%'' should simply wait until nodes are free should there be no more free nodes available within the given allocation. Hence, the more nodes (or total number of tasks) you allocate for the ''%%sbatch%%'' the more tests can run in parallel. But we have to make sure ''%%do_regtest%%'' knows about that number by setting ''%%-maxtasks ${SLURM_NTASKS}%%'', ''%%SLURM_NTASKS%%'' is automatically set by ''%%sbatch%%'' to the number of tasks you specified either when running ''%%sbatch%%'' or in the preamble of the ''%%sbatch%%'' script.
 +
 +Append the following to your ''%%sbatch%%'' template and at least adapt the value for ''%%CP2K_BASE_DIR%%'' and possibly also the ''%%CP2K_TEST_DIR%%'':
 +
 +<code bash>
 +CP2K_BASE_DIR="/PATH/TO/YOUR/CP2K/SOURCE/TREE"
 +CP2K_TEST_DIR="${SCRATCH}/cp2k_regtesting"
 +# CP2K_REGTEST_SCRIPT_DIR=""  # only set if needed (see below)
 +
 +CP2K_ARCH="local"
 +CP2K_VERSION="psmp"
 +
 +# the following is the default, adjust if you want to run single tests with more than 2 ranks/tasks
 +NTASKS_SINGLE_TEST=2
 +NNODES_SINGLE_TEST=1  # otherwise srun will distribute the 2 tasks over 2 nodes
 +SRUN_CMD="srun --cpu-bind=verbose,cores"
 +
 +# the following should be sufficiently generic:
 +
 +mkdir -p "${CP2K_TEST_DIR}"
 +cd "${CP2K_TEST_DIR}"
 +
 +cp2k_rel_dir=$(realpath --relative-to="${CP2K_TEST_DIR}" "${CP2K_BASE_DIR}")
 +# srun does not like `-np`, override the complete command instead:
 +export cp2k_run_prefix="${SRUN_CMD} -N ${NNODES_SINGLE_TEST} -n ${NTASKS_SINGLE_TEST}"
 +
 +"${CP2K_REGEST_SCRIPT_DIR:-${CP2K_BASE_DIR}/tools/regtesting}/do_regtest" \
 +  -arch "${CP2K_ARCH}" \
 +  -version "${CP2K_VERSION}" \
 +  -nobuild \
 +  -mpiranks ${NTASKS_SINGLE_TEST} \
 +  -ompthreads ${OMP_NUM_THREADS} \
 +  -maxtasks ${SLURM_NTASKS} \
 +  -cp2kdir "${cp2k_rel_dir}" \
 +  |& tee "${CP2K_TEST_DIR}/${CP2K_ARCH}.${CP2K_VERSION}.log"
 +
 +# the above will output both to the slurm-*.out as well as a log file,
 +# if you want only the log file replace the `|& tee` with a `>&`.
 +
 +# More options:
 +# -farming   ... enable farming mode, see below
 +# -retest    ... only do tests which failed in a previous run
 +</code>
 +
 +A complete ''%%sbatch%%'' script to run the regtests on CSCS’ Alps (Eiger) could look as follows:
 +
 +<code bash>
 +#!/bin/bash -l
 +#SBATCH --time=01:00:00
 +#SBATCH --nodes=4
 +#SBATCH --ntasks-per-node=128
 +#SBATCH --cpus-per-task=2
 +#SBATCH --ntasks-per-core=1
 +
 +# More SBATCH options:
 +# If you need 512GB memory nodes (otherwise only 256GB guaranteed):
 +#    #SBATCH --mem=497G
 +# To run on the debug queue (max 10 nodes, 30 min):
 +#    #SBATCH--partition=debug
 +
 +set -o errexit
 +set -o nounset
 +set -o pipefail
 +
 +export MPICH_OFI_STARTUP_CONNECT=1
 +export OMP_NUM_THREADS=${SLURM_CPUS_PER_TASK}
 +export OMP_PROC_BIND=close
 +export OMP_PLACES=cores
 +
 +source "${MODULESHOME}/init/bash"
 +
 +module load cpeGNU
 +module load \
 +    cray-fftw \
 +    ELPA/2020.11.001 \
 +    libxsmm/1.16.1 \
 +    libxc/5.1.3 \
 +    Libint-CP2K/2.6.0 \
 +    gcc/10.2.0
 +
 +# Let the user see the currently loaded modules in the slurm log for completeness:
 +module list
 +
 +CP2K_BASE_DIR="/users/timuel/work/cp2k"
 +CP2K_TEST_DIR="${SCRATCH}/cp2k_regtesting"
 +
 +CP2K_ARCH="Eiger-gfortran"
 +CP2K_VERSION="psmp"
 +
 +NTASKS_SINGLE_TEST=2
 +NNODES_SINGLE_TEST=1
 +SRUN_CMD="srun --cpu-bind=verbose,cores"
 +
 +# to run tests across nodes (to check for communication effects), use:
 +# NNODES_SINGLE_TEST=4
 +# SRUN_CMD="srun --cpu-bind=verbose,cores --ntasks-per-node 2"
 +
 +# the following should be sufficiently generic:
 +
 +mkdir -p "${CP2K_TEST_DIR}"
 +cd "${CP2K_TEST_DIR}"
 +
 +cp2k_rel_dir=$(realpath --relative-to="${CP2K_TEST_DIR}" "${CP2K_BASE_DIR}")
 +# srun does not like `-np`, override the complete command instead:
 +export cp2k_run_prefix="${SRUN_CMD} -N ${NNODES_SINGLE_TEST} -n ${NTASKS_SINGLE_TEST}"
 +
 +"${CP2K_REGEST_SCRIPT_DIR:-${CP2K_BASE_DIR}/tools/regtesting}/do_regtest" \
 +  -arch "${CP2K_ARCH}" \
 +  -version "${CP2K_VERSION}" \
 +  -nobuild \
 +  -mpiranks ${NTASKS_SINGLE_TEST} \
 +  -ompthreads ${OMP_NUM_THREADS} \
 +  -maxtasks ${SLURM_NTASKS} \
 +  -cp2kdir "${cp2k_rel_dir}" \
 + |& tee "${CP2K_TEST_DIR}/${CP2K_ARCH}.${CP2K_VERSION}.log"
 +</code>
 +
 +==== Test parallelization via Farming ====
 +
 +Since the startup time of ''%%srun%%'' can be significant (the ''%%--bcast%%'' argument for ''%%srun%%'' might help if the binary is statically linked as does putting the CP2K directory on to a scratch volume) there is the option to use CP2Ks //Farming// mode to run all tests within one ''%%srun%%'' and one CP2K instance. This mode is directly supported by ''%%do_regtest%%'' which then dynamically creates the required CP2K input file. The drawback is that if one regtest triggers a ''%%CPABORT%%'' (or another signal) the rest of the tests will never run.
 +
 +Simply add the option ''%%-farming%%'' to the list of options for the ''%%do_regtest%%'' call to enable this.
 +
 +==== Minimal directory setup ====
 +
 +If you want to test a precompiled executable there is a minimal directory layout you have to reproduce to run the regtest:
 +
 +  * ''%%cp2k-prebuilt/exe/prebuilt/*.psmp%%'' … directory with all the executables
 +  * ''%%cp2k-prebuilt/tests%%'' … directory containing the tests (can NOT be a symlink)
 +  * ''%%cp2k-prebuilt/data%%'' … containing CP2Ks data
 +
 +An example if your HPC center uses EasyBuild to provide the CP2K package:
 +
 +<code>
 +cp2k-prebuilt
 +├── data -> /apps/eiger/UES/jenkins/1.4.0/software/CP2K/8.1-cpeGNU-21.04/data
 +├── exe
 +│   └── prebuilt -> /apps/eiger/UES/jenkins/1.4.0/software/CP2K/8.1-cpeGNU-21.04/bin
 +└── tests
 +</code>
 +
 +and then update the variables as follows:
 +
 +<code bash>
 +CP2K_BASE_DIR="/PATH/TO/THE/MINIMAL/DIR/cp2k-prebuilt"
 +CP2K_TEST_DIR="${SCRATCH}/cp2k_regtesting"
 +CP2K_REGTEST_SCRIPT_DIR="/PATH/TO/A/FULL/CP2K/DIR/tools/regtesting"
 +
 +CP2K_ARCH="prebuilt"
 +CP2K_VERSION="psmp"
 +</code>
  
 +**Note**: if the ''%%tools/regtesting%%'' is not in that minimal directory tree as shown above you may get an error about the ''%%timings.py%%'' not found and there will be no timings. If you need those you should link/copy the regtesting scripts into ''%%tools/regtest%%'' of that minimal directory tree, at which you point you can leave the ''%%CP2K_REGTEST_SCRIPT_DIR%%'' variable undefined again.
dev/regtesting.1539028744.txt.gz · Last modified: (external edit)