The IVAS tests are using the [pytest](https://docs.pytest.org/) framework.
## Installing test dependencies
To use the `pytest` framework, you will need a few Python packages (in addition to Python itself).
As with other Python packages, there are different possibilities to install those packages.
Please chose the option that works best for you.
`Note`:
The installation of Python is not described, here.
In the following, it is assumed that `Python >= 3.7` is already installed / present.
### Global install
```bash
pip install-r tests/requirements.txt
```
### User install
```bash
pip install--user-r tests/requirements.txt
```
### Virtual environment install
```bash
# set up virtual environment
python3 -m venv VENV_NAME
# change to virtual environment
source VENV_NAME/bin/activate
# install required packages
pip install-r tests/requirements.txt
```
## Preparing the tests
`Note:`
Currently, shortened test vectors are used to speed up the testing.
Those shortened test vectors, some with gain adjustment, need to be created, once.
```bash
# create shortened test vectors
python3 tests/create_short_testvectors.py
```
The tests rely on references which need to be generated upfront using reference binaries.
When the reference binaries are named `IVAS_cod_ref(.exe)` and `IVAS_dec_ref.(exe)`, pytest will find and use them.
When the reference binaries are named differently, you need to specify them via the `--ref_encoder_path` and `--ref_decoder_path` options.
The tests will used the binaries `IVAS_cod(.exe)` and `IVAS_dec.(exe)` for testing. Please make sure that the binaries have been built before running the tests.
```bash
# create references
# the following binaies need to be present:
# - IVAS_cod(.exe)
# - IVAS_dec.(exe)
# - IVAS_cod_ref(.exe)
# - IVAS_dec_ref.(exe)
# pytest command lines to be executed from project root folder:
pytest tests -n auto --update_ref 1 -m create_ref
pytest tests -n auto --update_ref 1 -m create_ref_part2
```
## Running the tests
To run all tests from the tests folder:
```bash
# pytest command line to be executed from project root folder:
pytest tests -n auto
```
## Re-running some tests
When there are test failures, you may want to run, after having fixed the code, only those test cases which had failures. This can be achieved using the `--last-failed` option.
```bash
# rerun only the tests that failed at the last run
pytest tests -n auto --last-failed
```
To run a specific test case, you can e.g. pick a test case from the `short test summary info` and use that test case as an argument to `pytest`. E.g.
When there a many test failures, you can use the `-x` (or `--exitfirst`) option to stop testing on the first failure.
Commonly used options like `-n auto` can be added to addopts within the [pytest] section in `pytest.ini`. This saves some typing when calling `pytest`.
The `-v` (or `--verbose`) option is usually helpful to see what is going on. Therefore, `-v` is currently part of addopts in `pytest.ini`. If you don't like this verbosity, you can specify the `-q` (`--quiet`) option when running `pytest`.
