Merge branch '1378_conformance_script_isar' into 'main'

# IVAS conformance scripts

# IVAS Conformance Scripts

This folder contains scripts for running IVAS conformance tests. This is a placeholder file for instructions.

This folder contains scripts for running IVAS conformance tests.

no-BE conformance USAGE

## Setup for Reference Platform

Following CMDs needs to be executed from ivas-codec root folder:

  Reference platform is Ubuntu 24.04

################generate testvec package and encoder refs (temporary step to obtain testvec package, this step will be removed in final delivery)  ############

sh ivas_be_conf_test_gen.sh 

PYTHONPATH=scripts python scripts/ivas_conformance/runConformance.py --testvecDir $PWD/testvec --ref_build_path=testvec/ref_bin --cut_build_path=testvec/cut_bin --test-mode=ENC --regenerate-enc-refs

- Verify the Ubuntu Linux release is 24.04

  ```shell

  lsb_release -d | grep Ubuntu

  ```

  <span style="color: green;"> # It might be similar to Ubuntu 24.04.3 LTS </span>

Encoder conformance:

	- PYTHONPATH=scripts python scripts/ivas_conformance/runConformance.py --testvecDir $PWD/testvec --ref_build_path=testvec/ref_bin --cut_build_path=testvec/cut_bin --test-mode=ENC

Decoder conformance:

	- PYTHONPATH=scripts python scripts/ivas_conformance/runConformance.py --testvecDir $PWD/testvec --ref_build_path=testvec/ref_bin --cut_build_path=testvec/cut_bin --test-mode=DEC

Renderer conformance:

	- PYTHONPATH=scripts python scripts/ivas_conformance/runConformance.py --testvecDir $PWD/testvec --ref_build_path=testvec/ref_bin --cut_build_path=testvec/cut_bin --test-mode=REND

Split renderer confomance:

	- <TBD>

- Install Clang 18 compiler

  ```shell

  sudo apt install clang-18

  clang-18 --version

  ```

  Example version observed on Ubuntu 24.04.3 LTS

  ```text

  Ubuntu clang version 18.1.3 (1ubuntu1)

  Target: x86_64-pc-linux-gnu

  Thread model: posix

  InstalledDir: /usr/bin

  ```

  It might be required to set Clang-18 as the default clang on the machine

  ```shell

  sudo update-alternatives --install /usr/bin/clang clang /usr/bin/clang-18 100

  ```

- Install Python3.13 on Ubuntu 24.04 LTS

  ```shell

  sudo apt install python3.13 python3.13-venv

  ```

- Create virtual environment for Python 3.13 and install requirements

  ```shell

  python3.13 -m venv pyConformance

  source pyConformance/bin/activate

  cd ivas-codec

  python -m pip install -r tests/requirements.txt

  ```

## Reference Conformance Package Generation

<details>

<summary> <i> Expand for detailed procedure </i> </summary>

To generate reference conformance package for distribution

### Generate Reference Outputs and Readme.txt files

  ```shell

  sh scripts/ivas_conformance/ivas_be_conf_test_gen.sh

  ```

<details>

<summary> Example output of reference test generation </summary>

<pre><code>

::::::::::::::::::::::::

------------------------------------------

Generated html report: file:///home/dolby/git/ivas-codec/report_cmd.htm

------------------------------------------

=================================================

2571 passed, 538 skipped, 230 xfailed in 377.10s (0:06:17)

=================================================

Identified 5430 files from scripts

Removed 1515 files

Kept 5422 files

</code></pre>

</details>

### Generate Reference Decoded Outputs for the Reference Encoded files

  ```shell

  PYTHONPATH=scripts python scripts/ivas_conformance/runConformance.py --testvecDir $PWD/testvec --ref_build_path=testvec/bin  --regenerate-enc-refs

  ```

### Generate a conformance package zip

  ```shell

  zip -r conformance.zip testvec scripts/ivas_conformance scripts/tools

  ```

</details>

## Run CUT tests on Target platform

  To run CUT binaries on the targeted platform, it is necessary to replicate the initial setup for python and dependency packages. The CUT build of the IVAS binaries should be made available in a selected folder and needed for the next step

  To run IVAS DUT commands on the TARGET platform (may be different from ubuntu/linux)

  ```shell

  PYTHONPATH=scripts python scripts/ivas_conformance/runConformance.py --testvecDir $PWD/testvec --cut_build_path=CUT_BIN_DIR

  ```

<details>

<summary>Example Output of CUT execution</summary>

<pre><code>

Accumulating commands from Readme_IVAS_dec.txt

Accumulating commands from Readme_IVAS_rend.txt

Accumulating commands from Readme_IVAS_enc.txt

Accumulating commands from Readme_IVAS_ISAR_post_rend.txt

Accumulating commands from Readme_IVAS_ISAR_dec.txt

Accumulating commands from Readme_IVAS_JBM_dec.txt

No of tests :

    ENC : 381

    DEC : 637

    REND : 666

    ISAR_ENC : 1032

    ISAR : 1032

Executing tests for ENC   (381 tests)

Executing tests for DEC   (637 tests)

Executing tests for REND   (666 tests)

Executing tests for ISAR_ENC   (1032 tests)

Executing tests for ISAR   (1032 tests)

</code></pre>

</details>

  This should generate outputs in scripts/CUT_OUTPUTS folder which looks like below:-

  ```shell

  CUT_OUTPUTS

    +- runlog.txt       : Dump of all the commands run and the outputs   (mostly jumbled up due to multiprocessing)

    +- failedCmds.txt   : Log of all the shell commands that failed execution

    +- dec/             : Folder containing all decoder tests CUT outputs

    +- enc/             : Folder containing all encoder tests CUT outputs

    +- renderer_short/  : Folder containing all renderer tests CUT outputs

    +- split_rendering/ : Folder containing all split rendering enc/dec tests

  ```

## Perform the MLD based analysis on the CUT outputs on reference platform (Ubuntu 24.04)

   If CUT test execution is done on a different platform, the scripts/CUT_OUTPUTS must be copied and provided in the reference platform's scripts/CUT_OUTPUTS. Then the following command is used to perform MLD based analysis on the same, encoded outputs will be implicitly decoded using reference decoder executables and MLD analysis performed on the reference decoded outputs.

  ```shell

   PYTHONPATH=scripts python scripts/ivas_conformance/runConformance.py --testvecDir $PWD/testvec --ref_build_path=testvec/bin --analyse

  ```

<details>

<summary>Example Output of CUT Analysis</summary>

<pre><code>

Accumulating commands from Readme_IVAS_dec.txt

Accumulating commands from Readme_IVAS_rend.txt

Accumulating commands from Readme_IVAS_enc.txt

Accumulating commands from Readme_IVAS_ISAR_post_rend.txt

Accumulating commands from Readme_IVAS_ISAR_dec.txt

Accumulating commands from Readme_IVAS_JBM_dec.txt

No of tests :

    ENC : 381

    DEC : 637

    REND : 666

    ISAR_ENC : 1032

    ISAR : 1032

Analysing tests for ENC   (381 tests)

##########################################################

<ENC> Total Frames: 2635800

<ENC> MAX MLD across all frames : 0.0

<ENC> Frames with MLD == 0 : 2635800 frames (100.0%)

&lt;ENC&gt; Frames with MLD <= 1 : 2635800 frames (100.0%)

&lt;ENC&gt; Frames with MLD <= 2 : 2635800 frames (100.0%)

&lt;ENC&gt; Frames with MLD <= 5 : 2635800 frames (100.0%)

<ENC> BE frames percentage = 100.0

<ENC> max absolute diff = 0.0, sample range (-32768, 32767)

##########################################################

Analysing tests for DEC   (637 tests)

##########################################################

<DEC> Total Frames: 4342140

<DEC> MAX MLD across all frames : 0.0

<DEC> Frames with MLD == 0 : 4342140 frames (100.0%)

&lt;DEC&gt; Frames with MLD <= 1 : 4342140 frames (100.0%)

&lt;DEC&gt; Frames with MLD <= 2 : 4342140 frames (100.0%)

&lt;DEC&gt; Frames with MLD <= 5 : 4342140 frames (100.0%)

<DEC> BE frames percentage = 100.0

<DEC> max absolute diff = 0.0, sample range (-32768, 32767)

##########################################################

Analysing tests for REND   (666 tests)

##########################################################

<REND> Total Frames: 4799952

<REND> MAX MLD across all frames : 0.0

<REND> Frames with MLD == 0 : 4799952 frames (100.0%)

&lt;REND&gt; Frames with MLD <= 1 : 4799952 frames (100.0%)

&lt;REND&gt; Frames with MLD <= 2 : 4799952 frames (100.0%)

&lt;REND&gt; Frames with MLD <= 5 : 4799952 frames (100.0%)

<REND> BE frames percentage = 100.0

<REND> max absolute diff = 0.0, sample range (-32768, 32767)

##########################################################

Analysing tests for ISAR_ENC   (1032 tests)

##########################################################

<ISAR_ENC> Total Frames: 2125956

<ISAR_ENC> MAX MLD across all frames : 0.0

<ISAR_ENC> Frames with MLD == 0 : 2125956 frames (100.0%)

&lt;ISAR_ENC&gt; Frames with MLD <= 1 : 2125956 frames (100.0%)

&lt;ISAR_ENC&gt; Frames with MLD <= 2 : 2125956 frames (100.0%)

&lt;ISAR_ENC&gt; Frames with MLD <= 5 : 2125956 frames (100.0%)

<ISAR_ENC> BE frames percentage = 100.0

<ISAR_ENC> max absolute diff = 0.0, sample range (-32768,32767)

##########################################################

Analysing tests for ISAR   (1032 tests)

##########################################################

<ISAR> Total Frames: 2125956

<ISAR> MAX MLD across all frames : 0.0

<ISAR> Frames with MLD == 0 : 2125956 frames (100.0%)

&lt;ISAR&gt; Frames with MLD <= 1 : 2125956 frames (100.0%)

&lt;ISAR&gt; Frames with MLD <= 2 : 2125956 frames (100.0%)

&lt;ISAR&gt; Frames with MLD <= 5 : 2125956 frames (100.0%)

<ISAR> BE frames percentage = 100.0

<ISAR> max absolute diff = 0.0, sample range (-32768, 32767)

##########################################################

</code></pre>

</details>

## Executing specific tests only

All CUT tests can be run specifically for IVAS Encoder,IVAS Decoder,IVAS Renderer, ISAR Encoder and ISAR Decoder only. The commandline allows for ```-test-mode=<PARAM>``` for this functionality, examples : -

- Run DUT IVAS Encoder Tests Only (on Target Platform)

  ```shell

  PYTHONPATH=scripts python scripts/ivas_conformance/runConformance.py --testvecDir $PWD/testvec --cut_build_path=CUT_BIN_DIR --test-mode=ENC

  ```

- Analyse BE conformance for DUT IVAS Encoder Outputs Only (on Reference Platform)

  ```shell

  PYTHONPATH=scripts python scripts/ivas_conformance/runConformance.py --testvecDir $PWD/testvec --ref_build_path=testvec/bin --test-mode=ENC --analyse --be-test

  ```

- Analyse NON-BE conformance for DUT IVAS Encoder Outputs Only (on Reference Platform)

  ```shell

  PYTHONPATH=scripts python scripts/ivas_conformance/runConformance.py --testvecDir $PWD/testvec --ref_build_path=testvec/bin --test-mode=ENC --analyse

  ```

- Run DUT IVAS Decoder Tests Only (on Target Platform)

  ```shell

  PYTHONPATH=scripts python scripts/ivas_conformance/runConformance.py --testvecDir $PWD/testvec --cut_build_path=CUT_BIN_DIR --test-mode=DEC

  ```

- Analyse BE conformance for DUT IVAS Decoder Outputs Only

  ```shell

  PYTHONPATH=scripts python scripts/ivas_conformance/runConformance.py --testvecDir $PWD/testvec --test-mode=DEC --analyse --be-test

  ```

- Analyse NON-BE conformance DUT IVAS Decoder Outputs Only (on Reference Platform)

  ```shell

  PYTHONPATH=scripts python scripts/ivas_conformance/runConformance.py --testvecDir $PWD/testvec  --test-mode=DEC --analyse

  ```

- Run DUT IVAS Renderer Tests Only (on Target Platform)

  ```shell

  PYTHONPATH=scripts python scripts/ivas_conformance/runConformance.py --testvecDir $PWD/testvec --cut_build_path=CUT_BIN_DIR --test-mode=REND

  ```

- Analyse BE conformance for DUT Renderer Outputs Only 

  ```shell

  PYTHONPATH=scripts python scripts/ivas_conformance/runConformance.py --testvecDir $PWD/testvec --test-mode=REND --analyse --be-test

  ```

- Analyse NON-BE conformance DUT Renderer Outputs Only 

  ```shell

  PYTHONPATH=scripts python scripts/ivas_conformance/runConformance.py --testvecDir $PWD/testvec --test-mode=REND --analyse

  ```

- Run DUT ISAR Encoder Tests Only (on Target Platform)

  ```shell

  PYTHONPATH=scripts python scripts/ivas_conformance/runConformance.py --testvecDir $PWD/testvec --cut_build_path=CUT_BIN_DIR --test-mode=ISAR_ENC

  ```

- Analyse BE conformance for DUT ISAR Encoder Outputs Only (on Reference Platform)

  ```shell

  PYTHONPATH=scripts python scripts/ivas_conformance/runConformance.py --testvecDir $PWD/testvec --ref_build_path=testvec/bin --test-mode=ISAR_ENC --analyse --be-test

  ```

- Analyse NON-BE conformance for DUT ISAR Encoder Outputs Only (on Reference Platform)

  ```shell

  PYTHONPATH=scripts python scripts/ivas_conformance/runConformance.py --testvecDir $PWD/testvec --ref_build_path=testvec/bin --test-mode=ISAR_ENC --analyse

  ```

- Run DUT ISAR Decoder Tests Only (on Target Platform)

  ```shell

  PYTHONPATH=scripts python scripts/ivas_conformance/runConformance.py --testvecDir $PWD/testvec --cut_build_path=CUT_BIN_DIR --test-mode=ISAR

  ```

- Analyse BE conformance for DUT ISAR Decoder Outputs Only

  ```shell

  PYTHONPATH=scripts python scripts/ivas_conformance/runConformance.py --testvecDir $PWD/testvec  --test-mode=ISAR --analyse --be-test

  ```

- Analyse NON-BE conformance DUT ISAR Decoder Outputs Only 

  ```shell

  PYTHONPATH=scripts python scripts/ivas_conformance/runConformance.py --testvecDir $PWD/testvec  --test-mode=ISAR --analyse

  ```

cp IVAS_dec IVAS_dec_ref

cp IVAS_rend IVAS_rend_ref

cp ISAR_post_rend ISAR_post_rend_ref

python3 scripts/prepare_combined_format_inputs.py

python3 -m pytest -q tests/codec_be_on_mr_nonselection tests/renderer_short/test_renderer.py tests/split_rendering/test_split_rendering.py -v -n auto --update_ref 1 --create_ref --keep_files --html=report_cmd.html --self-contained-html

python3 scripts/parse_commands.py report_cmd.html Readme_IVAS.txt  

rm -rf testvec

scipy>=1.5.2

numpy>=1.19.2