clarifications

# IVAS Conformance Scripts

# IVAS Conformance Scripts, ver 3.1

This folder contains scripts for running IVAS conformance tests.

### LEVEL1, LEVEL2 and LEVEL3 behavior

For all levels, the default test-group baseline is `ENC` + `DEC`.

`REND`, `ISAR`, and `ISAR_ENC` are optional and are only included if explicitly selected as plain test-group tokens or added via `+REND`, `+ISAR`, `+ISAR_ENC`, or `+ISAR*`.

With no `--filter`, all test groups are selected: `ENC`, `DEC`, `REND`, `ISAR_ENC`, and `ISAR`.

When `--filter LEVEL1` is used, the default test-group baseline is `ENC` + `DEC`.

`REND`, `ISAR`, and `ISAR_ENC` are optional and are only included if explicitly selected as plain test-group tokens or added via `+REND`, `+ISAR` and `+ISAR_ENC`.

When `--filter LEVEL1` is specified, the following default tests are run:

  - Example: `--filter LEVEL1 DEC MONO` keeps only `MONO` tests from the LEVEL1-eligible DEC set.

  - Example: `--filter LEVEL1 JBM` keeps all LEVEL1-eligible ENC tests but only JBM tests from the LEVEL1-eligible DEC tests.

- `+TOKEN` adds tests to the final LEVEL1 selection, even if they would otherwise be restricted.

  - Example: `--filter LEVEL1 DEC JBM +BINAURAL` runs only JBM-matching LEVEL1 DEC tests and additionally includes DEC tests containing `BINAURAL` keyword, i.e. `BINAURAL`, `BINAURAL_IR`, `BINAURAL_ROOM_IR`, `BINAURAL_REVERB`.

  - Example: `--filter LEVEL1 DEC JBM +BINAURAL*` runs JBM-matching LEVEL1 DEC tests and additionally includes DEC tests with `BINAURAL` output formats, i.e. `BINAURAL`, `BINAURAL_IR`, `BINAURAL_ROOM_IR`, `BINAURAL_REVERB`.

- `-TOKEN` removes matching tests from the final LEVEL1 selection (including tests added via `+TOKEN`).

  - Example: `--filter LEVEL1 DEC +JBM -VOIP` adds JBM-matching DEC tests and then excludes any DEC tests containing the keyword `VOIP`.

  - Example: `--filter LEVEL1 DEC +JBM -VOIP` adds JBM-matching tests to LEVEL1 DEC tests and excludes DEC `VOIP` tests.

- Renderer and ISAR tests are not run by default in `LEVEL1`.

  - Add `+REND`, `+ISAR`, and/or `+ISAR_ENC` in `--filter` to include them.

  - Use `+ISAR*` if you want wildcard expansion across all `ISAR*`-prefixed test-group tokens.

When `--filter LEVEL2` is specified, all selection rules above remain the same,

except the bitrate cap is set to 192 kbps:

When `--filter LEVEL2` is specified, there are no default tag restrictions,

and the bitrate cap is set to 192 kbps for level-limited checks:

- Encoder (`ENC`) tests: only tests with bitrate up to 192 kbps (inclusive).

- Decoder (`DEC`) tests:

  - `MONO` output format: all bitrates.

  - `STEREO` output format: all bitrates.

When `--filter LEVEL3` is specified, there are no restrictions on the bitrate or output formats.

When `--filter LEVEL3` is specified, there are no restrictions on the bitrate or output formats,

and no default tag restrictions.

Examples (non-BE):

- Default behavior (same as LEVEL3 baseline): run only ENC and DEC test groups

- Default behavior (no filter): run all test groups

  ```shell

  PYTHONPATH=scripts python scripts/ivas_conformance/runConformance.py --testvecDir $PWD/testvec --ref_build_path=testvec/bin --analyse

  ```

- LEVEL3 plus renderer and ISAR test groups

- LEVEL3 explicit (equivalent tag scope to default; no bitrate/output restrictions)

  ```shell

  PYTHONPATH=scripts python scripts/ivas_conformance/runConformance.py --testvecDir $PWD/testvec --ref_build_path=testvec/bin --analyse --filter LEVEL3 +REND +ISAR +ISAR_ENC

  PYTHONPATH=scripts python scripts/ivas_conformance/runConformance.py --testvecDir $PWD/testvec --ref_build_path=testvec/bin --analyse --filter LEVEL3

  ```

- LEVEL1 baseline (ENC+DEC with LEVEL1 restrictions)