Commit 961ad4a7 authored by Ripinder Singh's avatar Ripinder Singh
Browse files

Split DUT test execution and analysis to two different steps 



* DUT tests can be executed in pass 1
* Analysis of DUT outputs in pass 2
* Updates to Readme.md

Signed-off-by: default avatarRipinder Singh <ripinder.singh@dolby.com>
parent 1a67be0a

scripts/ivas_conformance/README.md

+271 −15
Original line number Diff line number Diff line 
# IVAS conformance scripts

# IVAS Conformance Scripts



This folder contains scripts for running IVAS conformance tests. This is a placeholder file for instructions.

This folder contains scripts for running IVAS conformance tests.



no-BE conformance USAGE

## Setup for Reference Platform

  Reference platform is Ubuntu 24.04



Following CMDs needs to be executed from ivas-codec root folder:

- Verify the Ubuntu Linux release is 24.04



################generate testvec package and encoder refs (temporary step to obtain testvec package, this step will be removed in final delivery)  ############

  ```shell

  lsb_release -d | grep Ubuntu

  ```



  <span style="color: green;"># It might be similar to Ubuntu 24.04.3 LTS</span>



- Install Clang 18 compiler



  ```shell

  sudo apt install clang-18

  clang-18 --version

  ```



  Example version observed on Ubuntu 24.04.3 LTS



  ```text

  Ubuntu clang version 18.1.3 (1ubuntu1)

  Target: x86_64-pc-linux-gnu

  Thread model: posix

  InstalledDir: /usr/bin

  ```



  It might be required to set Clang-18 as the defauly clang on the machine



  ```shell

  sudo update-alternatives --install /usr/bin/clang clang /usr/bin/clang-18 100

  ```



- Install Python3.13 on Ubuntu 24.04 LTS



  ```shell

  sudo apt install python3.13 python3.13-venv

  ```



- Create virtual environment for Python 3.13 and install requirements



  ```shell

  python3.13 -m venv pyConformance

  source pyConformance/bin/activate

  cd ivas-codec

  python -m pip install -r tests/requirements.txt

  ```



## Reference Conformance Package Generation

  To generate reference conformance package for distribution

  #### Generate Reference Outputs and Readme.txt files



  ```shell

  sh ivas_be_conf_test_gen.sh

PYTHONPATH=scripts python scripts/ivas_conformance/runConformance.py --testvecDir $PWD/testvec --ref_build_path=testvec/ref_bin --cut_build_path=testvec/cut_bin --test-mode=ENC --regenerate-enc-refs

  ```



   <details>

    <summary>Example Output</summary>

    <pre><code>

    ::::::::::::::::::::::::

    ------------------------------------------

    Generated html report: file:///home/dolby/git/ivas-codec/report_cmd.htm

    ------------------------------------------

    =================================================

    2571 passed, 538 skipped, 230 xfailed in 377.10s (0:06:17)

    =================================================

    Identified 5430 files from scripts

    Removed 1515 files

    Kept 5422 files

    </code></pre>

  </details>



#### Generate Reference Decoded Outputs for the Reference Encoded files



  ```shell

  PYTHONPATH=scripts python scripts/ivas_conformance/runConformance.py --testvecDir $PWD/testvec --ref_build_path=testvec/bin  --regenerate-enc-refs

  ```



#### Generate a conformance package zip



  ```shell

  zip -r conformance.zip testvec scripts/ivas_conformance scripts/tools

  ```



## Run CUT tests on Target platform



  To run CUT binaries on the targeted platform, it is necessary to replicate the initial setup for python and dependency packages. The CUT build of the IVAS binaries should be made available in a selected folder and needed for the next step



  To run IVAS DUT commands on the TARGET platform (may be different from ubuntu/linux)



  ```shell

  PYTHONPATH=scripts python scripts/ivas_conformance/runConformance.py --testvecDir $PWD/testvec --cut_build_path=CUT_BIN_DIR

  ```



  <details>

    <summary>Example Output of CUT execution</summary>

    <pre><code>

    Accumulating commands from Readme_IVAS_dec.txt

    Accumulating commands from Readme_IVAS_rend.txt

    Accumulating commands from Readme_IVAS_enc.txt

    Accumulating commands from Readme_IVAS_ISAR_post_rend.txt

    Accumulating commands from Readme_IVAS_ISAR_dec.txt

    Accumulating commands from Readme_IVAS_JBM_dec.txt

    No of tests :

        ENC : 381

        DEC : 637

        REND : 666

        ISAR_ENC : 1032

        ISAR : 1032

    Executing tests for ENC   (381 tests)

    Executing tests for DEC   (637 tests)

    Executing tests for REND   (666 tests)

    Executing tests for ISAR_ENC   (1032 tests)

    Executing tests for ISAR   (1032 tests)

    </code></pre>

  </details>



  This should generate outputs in scripts/CUT_OUTPUTS folder which looks like below:-



  ```shell

    CUT_OUTPUTS

      +- runlog.txt       : Dump of all the commands run and the outputs   (mostlyjumbled up due to multiprocessing)

      +- failedCmds.txt   : Log of all the shell commands that failed execution

      +- dec/             : Folder containing all decoder tests CUT outputs

      +- enc/             : Folder containing all encoder tests CUT outputs

      +- renderer_short/  : Folder containing all renderer tests CUT outputs

      +- split_rendering/ : Folder containing all split rendering enc/dec testsCUT outputs

  ```



## Perform the MLD based analysis on the CUT outputs on refernce platform (Ubuntu 24.04)



   If CUT test execution is done on a different platform, the scripts/CUT_OUTPUTS must be copied and provided in the reference platform's scripts/CUT_OUTPUTS. Then the following command is used to perform MLD based analysis on the same, encoded outputs will be implicitly decoded using reference decoder executables and MLD analysis performed on the reference decoded outputs.





  ```shell

   PYTHONPATH=scripts python scripts/ivas_conformance/runConformance.py --testvecDir $PWD/testvec --ref_build_path=testvec/bin --analyse

  ```



<details>

<summary>Example Output of CUT Analysis</summary>

<pre><code>

Accumulating commands from Readme_IVAS_dec.txt

Accumulating commands from Readme_IVAS_rend.txt

Accumulating commands from Readme_IVAS_enc.txt

Accumulating commands from Readme_IVAS_ISAR_post_rend.txt

Accumulating commands from Readme_IVAS_ISAR_dec.txt

Accumulating commands from Readme_IVAS_JBM_dec.txt

No of tests :

    ENC : 381

    DEC : 637

    REND : 666

    ISAR_ENC : 1032

    ISAR : 1032

Analysing tests for ENC   (381 tests)

##########################################################

&lt;ENC&gt; Total Frames: 2635800

&lt;ENC&gt; MAX MLD across all frames : 0.0

&lt;ENC&gt; Frames with MLD == 0 : 2635800 frames (100.0%)

&lt;ENC&gt; Frames with MLD <= 1 : 2635800 frames (100.0%)

&lt;ENC&gt; Frames with MLD <= 2 : 2635800 frames (100.0%)

&lt;ENC&gt; Frames with MLD <= 5 : 2635800 frames (100.0%)

&lt;ENC&gt; BE frames percentage = 100.0

&lt;ENC&gt; max absolute diff = 0.0, sample range (-32768, 32767)

##########################################################

Analysing tests for DEC   (637 tests)

##########################################################

&lt;DEC&gt; Total Frames: 4342140

&lt;DEC&gt; MAX MLD across all frames : 0.0

&lt;DEC&gt; Frames with MLD == 0 : 4342140 frames (100.0%)

&lt;DEC&gt; Frames with MLD <= 1 : 4342140 frames (100.0%)

&lt;DEC&gt; Frames with MLD <= 2 : 4342140 frames (100.0%)

&lt;DEC&gt; Frames with MLD <= 5 : 4342140 frames (100.0%)

&lt;DEC&gt; BE frames percentage = 100.0

&lt;DEC&gt; max absolute diff = 0.0, sample range (-32768, 32767)

##########################################################

Analysing tests for REND   (666 tests)

##########################################################

&lt;REND&gt; Total Frames: 4799952

&lt;REND&gt; MAX MLD across all frames : 0.0

&lt;REND&gt; Frames with MLD == 0 : 4799952 frames (100.0%)

&lt;REND&gt; Frames with MLD <= 1 : 4799952 frames (100.0%)

&lt;REND&gt; Frames with MLD <= 2 : 4799952 frames (100.0%)

&lt;REND&gt; Frames with MLD <= 5 : 4799952 frames (100.0%)

&lt;REND&gt; BE frames percentage = 100.0

&lt;REND&gt; max absolute diff = 0.0, sample range (-32768, 32767)

##########################################################

Analysing tests for ISAR_ENC   (1032 tests)

##########################################################

&lt;ISAR_ENC&gt; Total Frames: 2125956

&lt;ISAR_ENC&gt; MAX MLD across all frames : 0.0

&lt;ISAR_ENC&gt; Frames with MLD == 0 : 2125956 frames (100.0%)

&lt;ISAR_ENC&gt; Frames with MLD <= 1 : 2125956 frames (100.0%)

&lt;ISAR_ENC&gt; Frames with MLD <= 2 : 2125956 frames (100.0%)

&lt;ISAR_ENC&gt; Frames with MLD <= 5 : 2125956 frames (100.0%)

&lt;ISAR_ENC&gt; BE frames percentage = 100.0

&lt;ISAR_ENC&gt; max absolute diff = 0.0, sample range (-32768,32767)

##########################################################

Analysing tests for ISAR   (1032 tests)

##########################################################

&lt;ISAR&gt; Total Frames: 2125956

&lt;ISAR&gt; MAX MLD across all frames : 0.0

&lt;ISAR&gt; Frames with MLD == 0 : 2125956 frames (100.0%)

&lt;ISAR&gt; Frames with MLD <= 1 : 2125956 frames (100.0%)

&lt;ISAR&gt; Frames with MLD <= 2 : 2125956 frames (100.0%)

&lt;ISAR&gt; Frames with MLD <= 5 : 2125956 frames (100.0%)

&lt;ISAR&gt; BE frames percentage = 100.0

&lt;ISAR&gt; max absolute diff = 0.0, sample range (-32768, 32767)

##########################################################

</code></pre>

</details>



## Executing specific tests only



All CUT tests can be run specifically for IVAS Encoder,IVAS Decoder,IVAS Renderer, ISAR Encoder and ISAR Decoder only. The commandline allows for ```-test-mode=<PARAM>``` for this functionality, examples : -



- Run DUT IVAS Encoder Tests Only (on Target Platform)



  ```shell

  PYTHONPATH=scripts python scripts/ivas_conformance/runConformance.py --testvecDir $PWD/testvec --cut_build_path=CUT_BIN_DIR --test-mode=ENC

  ```



- Analyse DUT IVAS Encoder Outputs Only (on Reference Platform)



  ```shell

  PYTHONPATH=scripts python scripts/ivas_conformance/runConformance.py --testvecDir $PWD/testvec --ref_build_path=testvec/bin --test-mode=ENC --analyse

  ```



- Run DUT IVAS Decoder Tests Only (on Target Platform)



  ```shell

  PYTHONPATH=scripts python scripts/ivas_conformance/runConformance.py --testvecDir $PWD/testvec --cut_build_path=CUT_BIN_DIR --test-mode=DEC

  ```



- Analyse DUT IVAS Decoder Outputs Only (on Reference Platform)



  ```shell

  PYTHONPATH=scripts python scripts/ivas_conformance/runConformance.py --testvecDir $PWD/testvec --ref_build_path=testvec/bin --test-mode=DEC --analyse

  ```



- Run DUT IVAS Renderer Tests Only (on Target Platform)



  ```shell

  PYTHONPATH=scripts python scripts/ivas_conformance/runConformance.py --testvecDir $PWD/testvec --cut_build_path=CUT_BIN_DIR --test-mode=REND

  ```



- Analyse DUT Renderer Outputs Only (on Reference Platform)



  ```shell

  PYTHONPATH=scripts python scripts/ivas_conformance/runConformance.py --testvecDir $PWD/testvec --ref_build_path=testvec/bin --test-mode=REND --analyse

  ```



- Run DUT ISAR Encoder Tests Only (on Target Platform)



  ```shell

  PYTHONPATH=scripts python scripts/ivas_conformance/runConformance.py --testvecDir $PWD/testvec --cut_build_path=CUT_BIN_DIR --test-mode=ISAR_ENC

  ```



- Analyse DUT ISAR Encoder Outputs Only (on Reference Platform)



  ```shell

  PYTHONPATH=scripts python scripts/ivas_conformance/runConformance.py --testvecDir $PWD/testvec --ref_build_path=testvec/bin --test-mode=ISAR_ENC --analyse

  ```



- Run DUT ISAR Decoder Tests Only (on Target Platform)



  ```shell

  PYTHONPATH=scripts python scripts/ivas_conformance/runConformance.py --testvecDir $PWD/testvec --cut_build_path=CUT_BIN_DIR --test-mode=ISAR

  ```



- Analyse DUT ISAR Decoder Outputs Only (on Reference Platform)



Encoder conformance:

	- PYTHONPATH=scripts python scripts/ivas_conformance/runConformance.py --testvecDir $PWD/testvec --ref_build_path=testvec/ref_bin --cut_build_path=testvec/cut_bin --test-mode=ENC

Decoder conformance:

	- PYTHONPATH=scripts python scripts/ivas_conformance/runConformance.py --testvecDir $PWD/testvec --ref_build_path=testvec/ref_bin --cut_build_path=testvec/cut_bin --test-mode=DEC

Renderer conformance:

	- PYTHONPATH=scripts python scripts/ivas_conformance/runConformance.py --testvecDir $PWD/testvec --ref_build_path=testvec/ref_bin --cut_build_path=testvec/cut_bin --test-mode=REND

Split renderer confomance:

	- <TBD>
 
  ```shell

  PYTHONPATH=scripts python scripts/ivas_conformance/runConformance.py --testvecDir $PWD/testvec --ref_build_path=testvec/bin --test-mode=ISAR --analyse

  ```

scripts/ivas_conformance/runConformance.py

+385 −308

File changed.

Preview size limit exceeded, changes collapsed.