Merge branch 'main' of https://forge.3gpp.org/rep/ivas-codec-pc/ivas-codec (36faca00) · Commits · IVAS Codec Public Collaboration / IVAS Codec

tests/README.md

+4 −0

Original line number	Diff line number	Diff line
		@@ -108,6 +108,10 @@ Commonly used options like `-n auto` are added to addopts within the [pytest] se

		The `-v` (or `--verbose`) option is helpful to see what is going on.

		If an error occurs that lacks details or if you see a "keyboard interrupt" reported by xdist, try running pytest with `-n 0`. This may provide more accurate details for the source of the error.

		Avoid using pytest.exit in the pytest test code, and instead raise an exception. This is because xdist does not capture pytest.exit properly, hiding the error message and reporting a false keyboard interrupt.

		## Custom options

		`Note:`

tests/conftest.py

+5 −5

Original line number	Diff line number	Diff line
		@@ -174,7 +174,7 @@ def dut_encoder_path(request) -> str:
		path = str(path.resolve())

		if not os.path.isfile(path):
		pytest.exit(f"\nDUT encoder binary {path} not found!\n!")
		raise FileNotFoundError(f"DUT encoder binary {path} not found!\n!")

		return path

		@@ -296,7 +296,7 @@ def ref_encoder_path(request) -> str:
		path = str(path.resolve())

		if not os.path.isfile(path):
		pytest.exit(f"\nREF encoder binary {path} not found!\n!")
		raise FileNotFoundError(f"REF encoder binary {path} not found!\n!")

		return path

		@@ -322,7 +322,7 @@ def dut_decoder_path(request) -> str:
		path = str(path.resolve())

		if not os.path.isfile(path):
		pytest.exit(f"\nDUT decoder binary {path} not found!\n!")
		raise FileNotFoundError(f"DUT decoder binary {path} not found!\n!")

		return path

		@@ -430,7 +430,7 @@ def ref_decoder_path(request) -> str:
		path = str(path.resolve())

		if not os.path.isfile(path):
		pytest.exit(f"\nREF decoder binary {path} not found!\n!")
		raise FileNotFoundError(f"REF decoder binary {path} not found!\n!")

		return path

		@@ -468,7 +468,7 @@ def reference_path(request) -> str:

		if request.config.option.update_ref == "0":
		if not os.path.isdir(path):
		pytest.exit(f"\nREF path {path} not found!\nPlease generate the references, first!\n!")
		raise FileNotFoundError(f"REF path {path} not found!\nPlease generate the references, first!\n!")

		return path

tests/renderer/README.md

0 → 100644

+45 −0

Original line number	Diff line number	Diff line
		# External Renderer Tests

		See also the [contribution page](https://forge.3gpp.org/rep/ivas-codec-pc/ivas-codec/-/wikis/Contributions/2-external-renderer) for related presentations.

		## Running Tests

		### Two testfiles are located in `tests/renderer/`:

		* `test_renderer.py`
		* `test_renderer_vs_decoder.py` \
		(requires `DEC_TO_REND_FLOAT_DUMP` defined in `options.h`, cannot be run in parallel i.e. `-n 1` argument must be used for the `pytest-xdist` plugin)

		### Run tests with:

		`python3 -m pytest -q -n auto tests/renderer/test_renderer.py`

		OR

		:warning: `DEC_TO_REND_FLOAT_DUMP` must be defined before compiling! :warning:

		`python3 -m pytest -q -n 1 tests/renderer/test_renderer_vs_decoder.py`

		### Important flags (see [pytest docs](https://docs.pytest.org/en/7.2.x/) for more information):

		* `-k` flag can filter test cases, e.g. `-k "test_ism_binaural_static"`
		* `-rA` reports ALL (pass, xpass, xfail, fail) instead of the default behaviour of reporting only failed tests\
		this option will also report captured logs, required for obtaining the commandline of testcases that pass or xfail
		* `--last-failed` re-runs only the cases that failed in the last test run
		* `--collect-only` is useful when adding new testcases to check if argument parametrization is working correctly

		### Directory tree

		```
		.
		├── compare_audio.py -> Python implementation of CompAudio, used for comparisons in tests
		├── constants.py -> Important paths, formats, metadata files and commandline templates
		├── cut -> Default location for output files for test conditions
		├── data -> Input test vectors
		├── ref -> Default location for output files for reference conditions
		├── run_test_renderer_vs_decoder.sh -> Compiles and runs test_renderer_vs_decoder.py with CMake and DEC_TO_REND_FLOAT_DUMP defined
		├── test_renderer_be_comparison.py -> Tests for CI Merge Request pipeline to compare renderer bit-exactness
		├── test_renderer.py -> Comparison of renderer against standalone executables / python scripts (to be deprecated soon)
		├── test_renderer_vs_decoder.py -> Comparison of renderer against decoder (to be deprecated soon)
		└── utils.py -> Wrapper functions for executables for use in testcases
		```