From c47789baf0071943ad8d14dcf34aa4247d4f0f11 Mon Sep 17 00:00:00 2001 From: Archit Tamarapu Date: Tue, 15 Nov 2022 10:29:42 +0100 Subject: [PATCH] add a README for the renderer test scripts --- tests/renderer/README.md | 45 ++++++++++++++++++++++++++++++++++++++++ 1 file changed, 45 insertions(+) create mode 100644 tests/renderer/README.md diff --git a/tests/renderer/README.md b/tests/renderer/README.md new file mode 100644 index 0000000000..b819e43c5c --- /dev/null +++ b/tests/renderer/README.md @@ -0,0 +1,45 @@ +# External Renderer Tests + +See also the [contribution page](https://forge.3gpp.org/rep/ivas-codec-pc/ivas-codec/-/wikis/Contributions/2-external-renderer) for related presentations. + +## Running Tests + +### Two testfiles are located in `tests/renderer/`: + +* `test_renderer.py` +* `test_renderer_vs_decoder.py` \ + (requires `DEC_TO_REND_FLOAT_DUMP` defined in `options.h`, cannot be run in parallel i.e. `-n 1` argument must be used for the `pytest-xdist` plugin) + +### Run tests with: + +`python3 -m pytest -q -n auto tests/renderer/test_renderer.py` + +OR + +:warning: `DEC_TO_REND_FLOAT_DUMP` must be defined before compiling! :warning: + +`python3 -m pytest -q -n 1 tests/renderer/test_renderer_vs_decoder.py` + +### Important flags (see [pytest docs](https://docs.pytest.org/en/7.2.x/) for more information): + +* `-k` flag can filter test cases, e.g. `-k "test_ism_binaural_static"` +* `-rA` reports ALL (pass, xpass, xfail, fail) instead of the default behaviour of reporting only failed tests\ +this option will also report captured logs, **required for obtaining the commandline of testcases that pass or xfail** +* `--last-failed` re-runs only the cases that failed in the last test run +* `--collect-only` is useful when adding new testcases to check if argument parametrization is working correctly + +### Directory tree + +``` +. +├── compare_audio.py -> Python implementation of CompAudio, used for comparisons in tests +├── constants.py -> Important paths, formats, metadata files and commandline templates +├── cut -> Default location for output files for test conditions +├── data -> Input test vectors +├── ref -> Default location for output files for reference conditions +├── run_test_renderer_vs_decoder.sh -> Compiles and runs test_renderer_vs_decoder.py with CMake and DEC_TO_REND_FLOAT_DUMP defined +├── test_renderer_be_comparison.py -> Tests for CI Merge Request pipeline to compare renderer bit-exactness +├── test_renderer.py -> Comparison of renderer against standalone executables / python scripts (to be deprecated soon) +├── test_renderer_vs_decoder.py -> Comparison of renderer against decoder (to be deprecated soon) +└── utils.py -> Wrapper functions for executables for use in testcases +``` -- GitLab