porting fixes from MR !2572

# IVAS Conformance Scripts

# IVAS Conformance Scripts, ver 3.1

This folder contains scripts for running IVAS conformance tests.

<details>

<summary>Example Output of CUT execution</summary>

<pre><code>

```console

Accumulating commands from Readme_IVAS_dec.txt

Accumulating commands from Readme_IVAS_rend.txt

Accumulating commands from Readme_IVAS_enc.txt

Accumulating commands from Readme_IVAS_ISAR_post_rend.txt

Accumulating commands from Readme_IVAS_ISAR_dec.txt

Accumulating commands from Readme_IVAS_JBM_dec.txt

No of tests :

    ENC : 381

    DEC : 637

    REND : 666

    ISAR_ENC : 1032

    ISAR : 1032

Executing tests for ENC   (381 tests)

Executing tests for DEC   (637 tests)

Executing tests for REND   (666 tests)

Executing tests for ISAR_ENC   (1032 tests)

Executing tests for ISAR   (1032 tests)

</code></pre>

Executing tests for ENC  (381 tests):

---------------------------

[ENC] OK

...

Summary of results:

---------------------

[ENC] OK

[DEC] OK

[REND] OK

[ISAR_ENC] OK

[ISAR] OK

```

</details>

This should generate outputs in scripts/CUT_OUTPUTS folder which looks like below:-

### Perform the BE comparison on the CUT outputs on reference platform

The BE comparison is performed to the CUT outputs using the command below. Encoded outputs will be decoded using the reference decoder executables as part of the process. The BE comparison is then performed between the CUT and reference decoded outputs. This includes comparison of ".wav"-files, and ".csv" and ".met" metadata files. If any non-BE results are observed, this is reported on the command-line and link to an analysis ".csv" file is given. The analysis file shows which exact files were non-BE. An example passing output is shown below. If all test sets print `PASSED BE TEST`, then CUT outputs are BE-conformant.

The BE comparison is performed to the CUT outputs using the command below. Encoded outputs are decoded using the reference decoder executables as part of the process. The BE comparison is then performed between the CUT and reference decoded outputs. This includes comparison of `.wav` files and `.csv`/`.met` metadata files. If non-BE results are observed, this is reported on the command line and in the generated analysis CSV output.

```shell

PYTHONPATH=scripts python scripts/ivas_conformance/runConformance.py --testvecDir $PWD/testvec --ref_build_path=testvec/bin --analyse --be-test

<details>

<summary>Example Output of BE comparison</summary>

<pre><code>

```console

Accumulating commands from Readme_IVAS_dec.txt

Accumulating commands from Readme_IVAS_enc.txt

Accumulating commands from Readme_IVAS_rend.txt

Accumulating commands from Readme_IVAS_JBM_dec.txt

Accumulating commands from Readme_IVAS_ISAR_dec.txt

Accumulating commands from Readme_IVAS_ISAR_post_rend.txt

No of tests :

    ENC : 374

    DEC : 638

    REND : 911

    ISAR_ENC : 1032

    ISAR : 1252

Analysing tests for ENC   (374 tests)

<ENC> PASSED BE TEST

Analysing tests for DEC   (638 tests)

<DEC> PASSED BE TEST

Analysing tests for REND   (911 tests)

<REND> PASSED BE TEST

Analysing tests for ISAR_ENC   (1032 tests)

<ISAR_ENC> PASSED BE TEST

Analysing tests for ISAR   (1252 tests)

<ISAR> PASSED BE TEST

</code></pre>

Analysing tests for ENC  (374 tests):

---------------------------

[ENC] OK (ERRORS=0, BE=374, NON-BE=0, MLD CORRIDOR FAILURES=0)

[DEC] OK (ERRORS=0, BE=638, NON-BE=0, MLD CORRIDOR FAILURES=0)

...

Summary of results:

---------------------

[ENC] OK

[DEC] OK

[REND] OK

[ISAR_ENC] OK

[ISAR] OK

```

</details>

### Perform the MLD based non-BE analysis on the CUT outputs on reference platform (Ubuntu 24.04)

<details>

<summary>Example Output of non-BE analysis</summary>

<pre><code>

```console

Accumulating commands from Readme_IVAS_dec.txt

Accumulating commands from Readme_IVAS_enc.txt

Accumulating commands from Readme_IVAS_rend.txt

Accumulating commands from Readme_IVAS_JBM_dec.txt

Accumulating commands from Readme_IVAS_ISAR_dec.txt

Accumulating commands from Readme_IVAS_ISAR_post_rend.txt

No of tests :

    ENC : 374

    DEC : 638

    REND : 911

    ISAR_ENC : 1032

    ISAR : 1252

Analysing tests for ENC   (374 tests)

##########################################################

<ENC> Total Frames: 3074220

<ENC> MAX MLD across all frames : 0.0

<ENC> Frames with MLD == 0 : 3074220 frames (100.0%)

&lt;ENC&gt; Frames with MLD <= 0.5 : 3074220 frames (100.0%)

&lt;ENC&gt; Frames with MLD <= 1 : 3074220 frames (100.0%)

&lt;ENC&gt; Frames with MLD <= 2 : 3074220 frames (100.0%)

&lt;ENC&gt; Frames with MLD <= 5 : 3074220 frames (100.0%)

<ENC> BE samples percentage = 100.0

<ENC> max absolute diff = 0.0, sample range (-32768, 32767)

##########################################################

MLD Corridor passed for ENC with max MLD diff of 0.0

Analysing tests for DEC   (638 tests)

##########################################################

<DEC> Total Frames: 5079252

<DEC> MAX MLD across all frames : 0.0

<DEC> Frames with MLD == 0 : 5079252 frames (100.0%)

&lt;DEC&gt; Frames with MLD <= 0.5 : 5079252 frames (100.0%)

&lt;DEC&gt; Frames with MLD <= 1 : 5079252 frames (100.0%)

&lt;DEC&gt; Frames with MLD <= 2 : 5079252 frames (100.0%)

&lt;DEC&gt; Frames with MLD <= 5 : 5079252 frames (100.0%)

<DEC> BE samples percentage = 100.0

<DEC> max absolute diff = 0.0, sample range (-32768, 32767)

##########################################################

MLD Corridor passed for DEC with max MLD diff of 0.0

Analysing tests for REND   (911 tests)

Analysing tests for ENC  (374 tests):

---------------------------

##########################################################

<REND> Total Frames: 5576907

<REND> MAX MLD across all frames : 0.0

<REND> Frames with MLD == 0 : 5576907 frames (100.0%)

&lt;REND&gt; Frames with MLD <= 0.5 : 5576907 frames (100.0%)

&lt;REND&gt; Frames with MLD <= 1 : 5576907 frames (100.0%)

&lt;REND&gt; Frames with MLD <= 2 : 5576907 frames (100.0%)

&lt;REND&gt; Frames with MLD <= 5 : 5576907 frames (100.0%)

<REND> BE samples percentage = 100.0

<REND> max absolute diff = 0.0, sample range (-32768, 32767)

<ENC> Total Frames: 3074220

<ENC> MAX MLD across all frames : 0.0

<ENC> Frames with MLD == 0 : 3074220 frames (100.0%)

<ENC> Frames with MLD <= 0.5 : 3074220 frames (100.0%)

<ENC> Frames with MLD <= 1 : 3074220 frames (100.0%)

<ENC> Frames with MLD <= 2 : 3074220 frames (100.0%)

<ENC> Frames with MLD <= 5 : 3074220 frames (100.0%)

<ENC> BE samples percentage = 100.0

<ENC> max absolute diff = 0.0, sample range (-32768, 32767)

##########################################################

MLD Corridor passed for REND with max MLD diff of 0.0

Analysing tests for ISAR_ENC   (1032 tests)

[ENC] OK (ERRORS=0, BE=374, NON-BE=0, MLD CORRIDOR FAILURES=0)

##########################################################

<ISAR_ENC> Total Frames: 2125956

<ISAR_ENC> MAX MLD across all frames : 0.0

<ISAR_ENC> Frames with MLD == 0 : 2125956 frames (100.0%)

&lt;ISAR_ENC&gt; Frames with MLD <= 0.5 : 2125956 frames (100.0%)

&lt;ISAR_ENC&gt; Frames with MLD <= 1 : 2125956 frames (100.0%)

&lt;ISAR_ENC&gt; Frames with MLD <= 2 : 2125956 frames (100.0%)

&lt;ISAR_ENC&gt; Frames with MLD <= 5 : 2125956 frames (100.0%)

<ISAR_ENC> BE samples percentage = 100.0

<ISAR_ENC> max absolute diff = 0.0, sample range (-32768, 32767)

##########################################################

MLD Corridor passed for ISAR_ENC with max MLD diff of 0.0

Analysing tests for ISAR   (1252 tests)

...

##########################################################

<ISAR> Total Frames: 2590956

<ISAR> MAX MLD across all frames : 0.0

<ISAR> Frames with MLD == 0 : 2590956 frames (100.0%)

&lt;ISAR&gt; Frames with MLD <= 0.5 : 2590956 frames (100.0%)

&lt;ISAR&gt; Frames with MLD <= 1 : 2590956 frames (100.0%)

&lt;ISAR&gt; Frames with MLD <= 2 : 2590956 frames (100.0%)

&lt;ISAR&gt; Frames with MLD <= 5 : 2590956 frames (100.0%)

<ISAR> BE samples percentage = 100.0

<ISAR> max absolute diff = 0.0, sample range (-32768, 32767)

##########################################################

Summary of results:

---------------------

[ENC] OK

[DEC] OK

[REND] OK

[ISAR_ENC] OK

[ISAR] OK

```

MLD Corridor passed for ISAR with max MLD diff of 0.0

</code></pre>

</details>

## Executing specific tests only

All CUT tests can be run specifically for IVAS Encoder,IVAS Decoder,IVAS Renderer, ISAR Encoder and ISAR Decoder only. The commandline allows for ```-test-mode=<PARAM>``` for this functionality, examples:

- Run CUT IVAS Encoder Tests Only (on Target Platform)

  ```shell

  PYTHONPATH=scripts python scripts/ivas_conformance/runConformance.py --testvecDir $PWD/testvec --cut_build_path=CUT_BIN_DIR --test-mode=ENC

  ```

- Analyse BE conformance for CUT IVAS Encoder Outputs Only (on Reference Platform)

  ```shell

  PYTHONPATH=scripts python scripts/ivas_conformance/runConformance.py --testvecDir $PWD/testvec --ref_build_path=testvec/bin --test-mode=ENC --analyse --be-test

  ```

- Analyse NON-BE conformance for CUT IVAS Encoder Outputs Only (on Reference Platform)

  ```shell

  PYTHONPATH=scripts python scripts/ivas_conformance/runConformance.py --testvecDir $PWD/testvec --ref_build_path=testvec/bin --test-mode=ENC --analyse

  ```

- Run CUT IVAS Decoder Tests Only (on Target Platform)

  ```shell

  PYTHONPATH=scripts python scripts/ivas_conformance/runConformance.py --testvecDir $PWD/testvec --cut_build_path=CUT_BIN_DIR --test-mode=DEC

  ```

- Analyse BE conformance for CUT IVAS Decoder Outputs Only

  ```shell

  PYTHONPATH=scripts python scripts/ivas_conformance/runConformance.py --testvecDir $PWD/testvec --test-mode=DEC --analyse --be-test

  ```

- Analyse NON-BE conformance CUT IVAS Decoder Outputs Only (on Reference Platform)

Use `--filter TOKEN` to control conformance level, test groups, output formats, and substring matching.

- Token types:

  - `LEVEL1`, `LEVEL2`, `LEVEL3`: conformance levels. `LEVEL3` is the default.

  - `ENC`, `DEC`, `REND`, `ISAR`, `ISAR_ENC`: test groups.

  - `MONO`, `STEREO`, `EXT`, `HOA`, `SBA`, `MC`: output-format tokens.

    - `HOA` expands to `HOA2`, `HOA3`.

    - `SBA` expands to `FOA`, `HOA2`, `HOA3`.

    - `MC` expands to `5_1`, `7_1`, `5_1_4`, `5_1_2`, `7_1_4`.

  - Any other token is treated as a case-insensitive substring match.

- Token modifiers:

  - `TOKEN`: restrictive token. Multiple restrictive tokens combine with logical AND.

  - `+TOKEN`: additive token. Adds matching tests even if they would otherwise be excluded.

  - `-TOKEN`: subtractive token. Removes matching tests from the final selection.

  - `TOKEN*`: wildcard token. Matches all known tokens starting with the given prefix.

  - `+TOKEN*`, `-TOKEN*`: wildcard token with add/remove behavior.

### LEVEL1, LEVEL2 and LEVEL3 behavior

For all levels, the default test-group baseline is `ENC` + `DEC`.

`REND`, `ISAR`, and `ISAR_ENC` are optional and are only included if explicitly selected as plain test-group tokens or added via `+REND`, `+ISAR` and `+ISAR_ENC`.

When `--filter LEVEL1` is specified, the following default tests are run:

- Encoder (`ENC`) tests: only tests with bitrate up to 80 kbps (inclusive).

- Decoder (`DEC`) tests:

  - `EXT` output format: only bitrate up to 80 kbps (inclusive).

  - `MONO` output format: all bitrates.

  - `STEREO` output format: all bitrates.

- The default `LEVEL1` tests may be restricted by adding more tokens (acting as logical AND).

  - Example: `--filter LEVEL1 DEC MONO` keeps only `MONO` tests from the LEVEL1-eligible DEC set.

  - Example: `--filter LEVEL1 JBM` keeps all LEVEL1-eligible ENC tests but only JBM tests from the LEVEL1-eligible DEC tests.

- `+TOKEN` adds tests to the final LEVEL1 selection, even if they would otherwise be restricted.

  - Example: `--filter LEVEL1 DEC JBM +BINAURAL*` runs JBM-matching LEVEL1 DEC tests and additionally includes DEC tests with `BINAURAL` output formats, i.e. `BINAURAL`, `BINAURAL_IR`, `BINAURAL_ROOM_IR`, `BINAURAL_REVERB`.

- `-TOKEN` removes matching tests from the final LEVEL1 selection (including tests added via `+TOKEN`).

  - Example: `--filter LEVEL1 DEC +JBM -VOIP` adds JBM-matching tests to LEVEL1 DEC tests and excludes DEC `VOIP` tests.

- Renderer and ISAR tests are not run by default in `LEVEL1`.

  - Add `+REND`, `+ISAR`, and/or `+ISAR_ENC` in `--filter` to include them.

  - Use `+ISAR*` if you want wildcard expansion across all `ISAR*`-prefixed test-group tokens.

When `--filter LEVEL2` is specified, all selection rules above remain the same,

except the bitrate cap is set to 192 kbps:

- Encoder (`ENC`) tests: only tests with bitrate up to 192 kbps (inclusive).

- Decoder (`DEC`) tests:

  - `EXT` output format: only bitrate up to 192 kbps (inclusive).

  - `MONO` output format: all bitrates.

  - `STEREO` output format: all bitrates.

When `--filter LEVEL3` is specified, there are no restrictions on the bitrate or output formats.

Examples (non-BE):

- Default behavior (same as LEVEL3 baseline): run only ENC and DEC test groups

  ```shell

  PYTHONPATH=scripts python scripts/ivas_conformance/runConformance.py --testvecDir $PWD/testvec  --test-mode=DEC --analyse

  ```

- Run CUT IVAS Renderer Tests Only (on Target Platform)

  ```shell

  PYTHONPATH=scripts python scripts/ivas_conformance/runConformance.py --testvecDir $PWD/testvec --cut_build_path=CUT_BIN_DIR --test-mode=REND

  PYTHONPATH=scripts python scripts/ivas_conformance/runConformance.py --testvecDir $PWD/testvec --ref_build_path=testvec/bin --analyse

  ```

- Analyse BE conformance for CUT Renderer Outputs Only

- LEVEL3 plus renderer and ISAR test groups

  ```shell

  PYTHONPATH=scripts python scripts/ivas_conformance/runConformance.py --testvecDir $PWD/testvec --test-mode=REND --analyse --be-test

  PYTHONPATH=scripts python scripts/ivas_conformance/runConformance.py --testvecDir $PWD/testvec --ref_build_path=testvec/bin --analyse --filter LEVEL3 +REND +ISAR +ISAR_ENC

  ```

- Analyse NON-BE conformance CUT Renderer Outputs Only

- LEVEL1 baseline (ENC+DEC with LEVEL1 restrictions)

  ```shell

  PYTHONPATH=scripts python scripts/ivas_conformance/runConformance.py --testvecDir $PWD/testvec --test-mode=REND --analyse

  PYTHONPATH=scripts python scripts/ivas_conformance/runConformance.py --testvecDir $PWD/testvec --ref_build_path=testvec/bin --analyse --filter LEVEL1

  ```

- Run CUT ISAR Encoder Tests Only (on Target Platform)

- LEVEL1 plus renderer and ISAR test groups

  ```shell

  PYTHONPATH=scripts python scripts/ivas_conformance/runConformance.py --testvecDir $PWD/testvec --cut_build_path=CUT_BIN_DIR --test-mode=ISAR_ENC

  PYTHONPATH=scripts python scripts/ivas_conformance/runConformance.py --testvecDir $PWD/testvec --ref_build_path=testvec/bin --analyse --filter LEVEL1 +REND +ISAR +ISAR_ENC

  ```

- Analyse BE conformance for CUT ISAR Encoder Outputs Only (on Reference Platform)

- LEVEL1 with additional case-insensitive command substring filtering

  ```shell

  PYTHONPATH=scripts python scripts/ivas_conformance/runConformance.py --testvecDir $PWD/testvec --ref_build_path=testvec/bin --test-mode=ISAR_ENC --analyse --be-test

  PYTHONPATH=scripts python scripts/ivas_conformance/runConformance.py --testvecDir $PWD/testvec --ref_build_path=testvec/bin --analyse --filter LEVEL1 DEC voip

  ```

- Analyse NON-BE conformance for CUT ISAR Encoder Outputs Only (on Reference Platform)

- LEVEL1 with additive BINAURAL decoder matching

  ```shell

  PYTHONPATH=scripts python scripts/ivas_conformance/runConformance.py --testvecDir $PWD/testvec --ref_build_path=testvec/bin --test-mode=ISAR_ENC --analyse

  PYTHONPATH=scripts python scripts/ivas_conformance/runConformance.py --testvecDir $PWD/testvec --ref_build_path=testvec/bin --analyse --filter LEVEL1 +BINAURAL

  ```

- Run CUT ISAR Decoder Tests Only (on Target Platform)

- LEVEL1 with restrictive and additive terms together

  ```shell

  PYTHONPATH=scripts python scripts/ivas_conformance/runConformance.py --testvecDir $PWD/testvec --cut_build_path=CUT_BIN_DIR --test-mode=ISAR

  PYTHONPATH=scripts python scripts/ivas_conformance/runConformance.py --testvecDir $PWD/testvec --ref_build_path=testvec/bin --analyse --filter LEVEL1 DEC JBM +BINAURAL

  ```

- Analyse BE conformance for CUT ISAR Decoder Outputs Only

- LEVEL2 baseline (ENC+DEC with LEVEL2 restrictions)

  ```shell

  PYTHONPATH=scripts python scripts/ivas_conformance/runConformance.py --testvecDir $PWD/testvec  --test-mode=ISAR --analyse --be-test

  PYTHONPATH=scripts python scripts/ivas_conformance/runConformance.py --testvecDir $PWD/testvec --ref_build_path=testvec/bin --analyse --filter LEVEL2

  ```

- Analyse NON-BE conformance CUT ISAR Decoder Outputs Only

- LEVEL2 plus renderer and ISAR test groups

  ```shell

  PYTHONPATH=scripts python scripts/ivas_conformance/runConformance.py --testvecDir $PWD/testvec  --test-mode=ISAR --analyse

  PYTHONPATH=scripts python scripts/ivas_conformance/runConformance.py --testvecDir $PWD/testvec --ref_build_path=testvec/bin --analyse --filter LEVEL2 +REND +ISAR +ISAR_ENC

  ```