Commit c47789ba authored by Archit Tamarapu's avatar Archit Tamarapu
Browse files

add a README for the renderer test scripts

parent 394140be
Loading
Loading
Loading
Loading
Loading
+45 −0
Original line number Diff line number Diff line
# External Renderer Tests

See also the [contribution page](https://forge.3gpp.org/rep/ivas-codec-pc/ivas-codec/-/wikis/Contributions/2-external-renderer) for related presentations.

## Running Tests

### Two testfiles are located in `tests/renderer/`:

* `test_renderer.py`
* `test_renderer_vs_decoder.py` \
  (requires `DEC_TO_REND_FLOAT_DUMP` defined in `options.h`, cannot be run in parallel i.e. `-n 1`  argument must be used for the `pytest-xdist` plugin)

### Run tests with:

`python3 -m pytest -q -n auto tests/renderer/test_renderer.py`

OR

:warning: `DEC_TO_REND_FLOAT_DUMP` must be defined before compiling! :warning:

`python3 -m pytest -q -n 1 tests/renderer/test_renderer_vs_decoder.py`

### Important flags (see [pytest docs](https://docs.pytest.org/en/7.2.x/) for more information):

* `-k` flag can filter test cases, e.g. `-k "test_ism_binaural_static"`
* `-rA` reports ALL (pass, xpass, xfail, fail) instead of the default behaviour of reporting only failed tests\
this option will also report captured logs, **required for obtaining the commandline of testcases that pass or xfail**
* `--last-failed` re-runs only the cases that failed in the last test run
* `--collect-only` is useful when adding new testcases to check if argument parametrization is working correctly

### Directory tree

```
.
├── compare_audio.py                    ->  Python implementation of CompAudio, used for comparisons in tests
├── constants.py                        ->  Important paths, formats, metadata files and commandline templates
├── cut                                 ->  Default location for output files for test conditions
├── data                                ->  Input test vectors
├── ref                                 ->  Default location for output files for reference conditions
├── run_test_renderer_vs_decoder.sh     ->  Compiles and runs test_renderer_vs_decoder.py with CMake and DEC_TO_REND_FLOAT_DUMP defined
├── test_renderer_be_comparison.py      ->  Tests for CI Merge Request pipeline to compare renderer bit-exactness
├── test_renderer.py                    ->  Comparison of renderer against standalone executables / python scripts (to be deprecated soon)
├── test_renderer_vs_decoder.py         ->  Comparison of renderer against decoder (to be deprecated soon)
└── utils.py                            ->  Wrapper functions for executables for use in testcases
```