Testing

Why set-up a testing policy ?

Code_TYMPAN is quite a big application and one can not expect that some fix or new feature will just work : a fix somewhere can - and will - trigger a bug or a mere unexpected behaviour later and elsewhere… In order to circumvent this major source of complexity it is crucial to cover new bug and features with automated tests.

Automated tests are testing procedures :

• which can be run automatically, without requiring for a user to perform a series of actions in the GUI typically

• which are implemented a piece of code running them

• which will either succeed in case what they are intended to check is asserted or fail which does not mean crashing on an error but instead reporting this was expected but it got that instead.

Moreover, it is much easier to debug a small automated test than a complete application like Code_TYMPAN. So coding a test should be done at the same time - or even before - coding the actual feature so that the developer can draw the most benefit from it while devising the feature.

What is a testing framework ?

Now, writing a stand alone program to test each new feature or bug newly reported can be a tedious task. That’s why automated testing framework exists : to make this most important task lighter. They allow to concentrate on asserting the feature or bug, not the plumbing of the reporting (see tests/test_examples.cpp).

In Code_TYMPAN we use the testing framework GTest aka Google Test : it was chosen for it simplicity and the quality of its documentation. You are strongly encouraged to read the primer now.

GTest (like most unit testing frameworks) builds upon the notion of test case : a test case represent one simple setting in which you want to assert some behaviour of your feature, or lack of some bug. You create a test case (which actually is a class under the hood) with the TEST macro:

TEST(TestSuite, testname)
{
  // Set up the objects and their state
  ...
  EXPECT_EQUAL(expected, value_tested);
  ...
}

When you run the resulting executable, the body of each test case in your .cpp file will be run separately, just like the main function of as many small programs, and reported as pass, fail or error. In case of failure each failing EXPECT or ASSERT will be detailed in the report.

It is a good practice to organise automated tests in a directory tree by the components or interactions they test. Ideally each class and major interaction should be tested in its own test file. For a complex application like Code_TYMPAN there will thus be a whole tree a tests to run, that’s why a test driver is most useful.

How to launch automated tests ?

Launching by hand all the automated tests of the tests tree and aggregating their reports would be a daunting task. Here the CTest test driver comes into play. CTest is a companion to CMake which uses information already available in CMake to automate the execution and reporting of a whole set of tests.

In order to be able to run tests with CTest, one must set the environement before. Section Running Tests gives the way to do so using command lines.

Another way to use CTest is to launch the tests with Visual Studio. In this case, environement must also be set. The script 0_Launch_Visual_Studio_with_env.bat allows to do so.

CTest accepts numerous options.

-C indicates the configuration which can be Debug or Release in our case. With --output-on-failure option, the test will be quiet if successfull and verbose if it fails. The single -V option make CTest a bit more verbose and a double -VV makes it a lot more verbose. You can also filter (out) by test executable name (with reg-exp if needed) and many advanced features not covered here.

Test coverage

In order to be representative, the tests must execute a maximum of code when running.

At this time, the study and the results of test coverage are documented in this issue : https://gitlab.com/tympan/code_tympan/-/issues/324

Writing automated tests

Todo

Best practices. How can you do.

Data for Testing

The directory tests/data stores various pieces of data used by some of the automated tests.

For example the tests/data/projects-panel provides a set of XML files dedicated to test some features. These files deal with very simple Code_TYMPAN project: single site, one building, one pair of source/receptor, etc. which are very useful for testing. Moreover, for each of them, a typical screenshot is provided in doc/_static/images/tests/

• Geometric Data

By hand validation

Because the automated tests coverage is yet unsufficient, and because some high level, global validation procedure is always useful a by hand validation procedure is proposed.

For now it this procedure is documented in this external PDF document and the resources it refers to are stored under tests/data/manual_validation/.

Tips and Tricks

Launch Unit tests

There are several ways to run unit tests.

The simplest is to use the scripts provided which allow you to launch all the tests (see above).

Another way to run the tests is to use the command line, here is the procedure:

cd c:/projects/code_tympan_build_d
SetEnvTympanTests.bat
## It is necessary to set the TYMPAN_PYTHON_INTERP variable for the test_m_b_altimetrie.cpp test because it calls the altimetry module
set TYMPAN_PYTHON_INTERP=C:\dists\python\venv312tympan\Scripts\python.exe
set TYMPAN_SOLVERDIR=C:\projects\code_tympan_install\pluginsd
ctest -C Debug

To include visual tests, it is necessary to uncomment “set RUN_VISUAL_TESTS=ON”.

Please note that this involves the use of the ‘descartes’ package.

To run a specific test:

ctest -C Debug -R test_altimetry_builder -V

If a test blocks at number %N%, it can be necessary to kill Python in the task manager and restart with:

ctest -C Debug -I %N%

Launch Python tests

cd C:\projects\code_tympan_build_d\python\tests
python -m unittest discover

Debug PYTHON test :

C:\dists\python\venv312tympan\Scripts\python.exe -m pdb "C:/projects/code_tympan_build_d/python/tests/test_altimetry_roads.py" "-v"

or add in the python file :

breakpoint()

Debug

To keep temporary files set the environment variable:

set TYMPAN_DEBUG=keep_tmp_files;monothread

To debug by running a Python script please set:

set TYMPAN_DEBUG=keep_tmp_files;interactive;monothread

To Activate the virtual environment:

C:\dists\python\venv312tympan\Scripts\activate.bat

To Launch Visual Studio 2019 with environment variables correctly set (after Set commands):

Please use “0_Launch_Visual_Studio_with_env.bat” available here

Little explanation of “solve_tympan_project.py” parameters:

- input_proj: input project file

- output_proj: output project file

- output_mesh: output project mesh

- solverdir: solver

- size_criterion: triangles size used in the mesh, not limited if 0

- refine_mesh: specifies whether the mesh should be refined or not

- use_vol_landtakes : specifies whether landtakes should be used or not

To debug a calculation launched by Python with UseVolumesLandtake set to False:

python C:/projects/code_tympan_install_d/bin/solve_tympan_project.py
C:\projects\tympan\path_to_file\input_uvl_false.xml
C:\projects\tympan\path_to_file\result_uvl_false.xml
C:\projects\tympan\path_to_file\mesh_uvl_false.ply
C:/projects/code_tympan_install_d/pluginsd 0.0 True False

With UseVolumesLandatke to True :

python C:/projects/code_tympan_install_d/bin/solve_tympan_project.py
C:\projects\tympan\path_to_file\input_uvl_true.xml
C:\projects\tympan\path_to_file\result_uvl_true.xml
C:\projects\tympan\path_to_file\mesh_uvl_true.ply
C:/projects/code_tympan_install_d/pluginsd 0.0 True True

Physics tests

Code_TYMPAN’s physics is tested by a series of 10 automatic tests :

• Test n°1 : Comparing a surfacic source and a point source

• Test n°2 : Source on the surface of a cube (cannot run currently)

• Test n°3 : Accoustic power of a building

• Test n°4 : Atmospheric absorption calculation

• Test n°5 : Ground effect with an homogeneous ground - smoothing formula

• Test n°6 : Ground effect with an homogeneous ground - s parameter influence

• Test n°7 : Ground effect with an impedence discontinuity

• Test n°8 : Diffraction by a thin screen

• Test n°9 : Diffraction by a cubic screen

• Test n°10 : Diffraction by a hill

How to launch the tests

The tests are located in the tympan_tools git repository, in the test_physique folder. In this folder, you will find a launch.bat file, which will launch the tests. You can select the version of Code_TYMPAN you want to test by changing the TYMPAN_INSTALL_DIR variable in launch.bat.

To run the tests, simply double click on launch.bat. A cmd window will pop up and for each test that is run, will tell you wether it is “OK” or “Non OK”. The tests 1, 5, 6, 7, 8, 9, and 10 will also produce plots in the output folder.

How the tests works

launch.bat will setup the environment necessary to run Code_TYMPAN and call Unittest_physique.py.

Unittest_physique.py will run each test by calling a python script located in the corresponding folder.

These tests will run a calculation using Code_TYMPAN, and compare the results with references. Those references are obtained either by analytical computations, or by results from a previous Code_TYMPAN version. All the data necessary for a test, such as Code_TYMPAN models or results of previous versions are located in the folder for said test.

The functions used by several tests are located in util_mat.py.