Commit 961ad4a7 authored by Ripinder Singh's avatar Ripinder Singh
Browse files

Split DUT test execution and analysis to two different steps



* DUT tests can be executed in pass 1
* Analysis of DUT outputs in pass 2
* Updates to Readme.md

Signed-off-by: default avatarRipinder Singh <ripinder.singh@dolby.com>
parent 1a67be0a
Loading
Loading
Loading
Loading
Loading
+271 −15
Original line number Diff line number Diff line
# IVAS conformance scripts
# IVAS Conformance Scripts

This folder contains scripts for running IVAS conformance tests. This is a placeholder file for instructions.
This folder contains scripts for running IVAS conformance tests.

no-BE conformance USAGE
## Setup for Reference Platform
  Reference platform is Ubuntu 24.04

Following CMDs needs to be executed from ivas-codec root folder:
- Verify the Ubuntu Linux release is 24.04

################generate testvec package and encoder refs (temporary step to obtain testvec package, this step will be removed in final delivery)  ############
  ```shell
  lsb_release -d | grep Ubuntu
  ```

  <span style="color: green;"># It might be similar to Ubuntu 24.04.3 LTS</span>

- Install Clang 18 compiler

  ```shell
  sudo apt install clang-18
  clang-18 --version
  ```

  Example version observed on Ubuntu 24.04.3 LTS

  ```text
  Ubuntu clang version 18.1.3 (1ubuntu1)
  Target: x86_64-pc-linux-gnu
  Thread model: posix
  InstalledDir: /usr/bin
  ```

  It might be required to set Clang-18 as the defauly clang on the machine

  ```shell
  sudo update-alternatives --install /usr/bin/clang clang /usr/bin/clang-18 100
  ```

- Install Python3.13 on Ubuntu 24.04 LTS

  ```shell
  sudo apt install python3.13 python3.13-venv
  ```

- Create virtual environment for Python 3.13 and install requirements

  ```shell
  python3.13 -m venv pyConformance
  source pyConformance/bin/activate
  cd ivas-codec
  python -m pip install -r tests/requirements.txt
  ```

## Reference Conformance Package Generation
  To generate reference conformance package for distribution
  #### Generate Reference Outputs and Readme.txt files

  ```shell
  sh ivas_be_conf_test_gen.sh
PYTHONPATH=scripts python scripts/ivas_conformance/runConformance.py --testvecDir $PWD/testvec --ref_build_path=testvec/ref_bin --cut_build_path=testvec/cut_bin --test-mode=ENC --regenerate-enc-refs
  ```

   <details>
    <summary>Example Output</summary>
    <pre><code>
    ::::::::::::::::::::::::
    ------------------------------------------
    Generated html report: file:///home/dolby/git/ivas-codec/report_cmd.htm
    ------------------------------------------
    =================================================
    2571 passed, 538 skipped, 230 xfailed in 377.10s (0:06:17)
    =================================================
    Identified 5430 files from scripts
    Removed 1515 files
    Kept 5422 files
    </code></pre>
  </details>

#### Generate Reference Decoded Outputs for the Reference Encoded files

  ```shell
  PYTHONPATH=scripts python scripts/ivas_conformance/runConformance.py --testvecDir $PWD/testvec --ref_build_path=testvec/bin  --regenerate-enc-refs
  ```

#### Generate a conformance package zip

  ```shell
  zip -r conformance.zip testvec scripts/ivas_conformance scripts/tools
  ```

## Run CUT tests on Target platform

  To run CUT binaries on the targeted platform, it is necessary to replicate the initial setup for python and dependency packages. The CUT build of the IVAS binaries should be made available in a selected folder and needed for the next step

  To run IVAS DUT commands on the TARGET platform (may be different from ubuntu/linux)

  ```shell
  PYTHONPATH=scripts python scripts/ivas_conformance/runConformance.py --testvecDir $PWD/testvec --cut_build_path=CUT_BIN_DIR
  ```

  <details>
    <summary>Example Output of CUT execution</summary>
    <pre><code>
    Accumulating commands from Readme_IVAS_dec.txt
    Accumulating commands from Readme_IVAS_rend.txt
    Accumulating commands from Readme_IVAS_enc.txt
    Accumulating commands from Readme_IVAS_ISAR_post_rend.txt
    Accumulating commands from Readme_IVAS_ISAR_dec.txt
    Accumulating commands from Readme_IVAS_JBM_dec.txt
    No of tests :
        ENC : 381
        DEC : 637
        REND : 666
        ISAR_ENC : 1032
        ISAR : 1032
    Executing tests for ENC   (381 tests)
    Executing tests for DEC   (637 tests)
    Executing tests for REND   (666 tests)
    Executing tests for ISAR_ENC   (1032 tests)
    Executing tests for ISAR   (1032 tests)
    </code></pre>
  </details>

  This should generate outputs in scripts/CUT_OUTPUTS folder which looks like below:-

  ```shell
    CUT_OUTPUTS
      +- runlog.txt       : Dump of all the commands run and the outputs   (mostlyjumbled up due to multiprocessing)
      +- failedCmds.txt   : Log of all the shell commands that failed execution
      +- dec/             : Folder containing all decoder tests CUT outputs
      +- enc/             : Folder containing all encoder tests CUT outputs
      +- renderer_short/  : Folder containing all renderer tests CUT outputs
      +- split_rendering/ : Folder containing all split rendering enc/dec testsCUT outputs
  ```

## Perform the MLD based analysis on the CUT outputs on refernce platform (Ubuntu 24.04)

   If CUT test execution is done on a different platform, the scripts/CUT_OUTPUTS must be copied and provided in the reference platform's scripts/CUT_OUTPUTS. Then the following command is used to perform MLD based analysis on the same, encoded outputs will be implicitly decoded using reference decoder executables and MLD analysis performed on the reference decoded outputs.


  ```shell
   PYTHONPATH=scripts python scripts/ivas_conformance/runConformance.py --testvecDir $PWD/testvec --ref_build_path=testvec/bin --analyse
  ```

<details>
<summary>Example Output of CUT Analysis</summary>
<pre><code>
Accumulating commands from Readme_IVAS_dec.txt
Accumulating commands from Readme_IVAS_rend.txt
Accumulating commands from Readme_IVAS_enc.txt
Accumulating commands from Readme_IVAS_ISAR_post_rend.txt
Accumulating commands from Readme_IVAS_ISAR_dec.txt
Accumulating commands from Readme_IVAS_JBM_dec.txt
No of tests :
    ENC : 381
    DEC : 637
    REND : 666
    ISAR_ENC : 1032
    ISAR : 1032
Analysing tests for ENC   (381 tests)
##########################################################
&lt;ENC&gt; Total Frames: 2635800
&lt;ENC&gt; MAX MLD across all frames : 0.0
&lt;ENC&gt; Frames with MLD == 0 : 2635800 frames (100.0%)
&lt;ENC&gt; Frames with MLD <= 1 : 2635800 frames (100.0%)
&lt;ENC&gt; Frames with MLD <= 2 : 2635800 frames (100.0%)
&lt;ENC&gt; Frames with MLD <= 5 : 2635800 frames (100.0%)
&lt;ENC&gt; BE frames percentage = 100.0
&lt;ENC&gt; max absolute diff = 0.0, sample range (-32768, 32767)
##########################################################
Analysing tests for DEC   (637 tests)
##########################################################
&lt;DEC&gt; Total Frames: 4342140
&lt;DEC&gt; MAX MLD across all frames : 0.0
&lt;DEC&gt; Frames with MLD == 0 : 4342140 frames (100.0%)
&lt;DEC&gt; Frames with MLD <= 1 : 4342140 frames (100.0%)
&lt;DEC&gt; Frames with MLD <= 2 : 4342140 frames (100.0%)
&lt;DEC&gt; Frames with MLD <= 5 : 4342140 frames (100.0%)
&lt;DEC&gt; BE frames percentage = 100.0
&lt;DEC&gt; max absolute diff = 0.0, sample range (-32768, 32767)
##########################################################
Analysing tests for REND   (666 tests)
##########################################################
&lt;REND&gt; Total Frames: 4799952
&lt;REND&gt; MAX MLD across all frames : 0.0
&lt;REND&gt; Frames with MLD == 0 : 4799952 frames (100.0%)
&lt;REND&gt; Frames with MLD <= 1 : 4799952 frames (100.0%)
&lt;REND&gt; Frames with MLD <= 2 : 4799952 frames (100.0%)
&lt;REND&gt; Frames with MLD <= 5 : 4799952 frames (100.0%)
&lt;REND&gt; BE frames percentage = 100.0
&lt;REND&gt; max absolute diff = 0.0, sample range (-32768, 32767)
##########################################################
Analysing tests for ISAR_ENC   (1032 tests)
##########################################################
&lt;ISAR_ENC&gt; Total Frames: 2125956
&lt;ISAR_ENC&gt; MAX MLD across all frames : 0.0
&lt;ISAR_ENC&gt; Frames with MLD == 0 : 2125956 frames (100.0%)
&lt;ISAR_ENC&gt; Frames with MLD <= 1 : 2125956 frames (100.0%)
&lt;ISAR_ENC&gt; Frames with MLD <= 2 : 2125956 frames (100.0%)
&lt;ISAR_ENC&gt; Frames with MLD <= 5 : 2125956 frames (100.0%)
&lt;ISAR_ENC&gt; BE frames percentage = 100.0
&lt;ISAR_ENC&gt; max absolute diff = 0.0, sample range (-32768,32767)
##########################################################
Analysing tests for ISAR   (1032 tests)
##########################################################
&lt;ISAR&gt; Total Frames: 2125956
&lt;ISAR&gt; MAX MLD across all frames : 0.0
&lt;ISAR&gt; Frames with MLD == 0 : 2125956 frames (100.0%)
&lt;ISAR&gt; Frames with MLD <= 1 : 2125956 frames (100.0%)
&lt;ISAR&gt; Frames with MLD <= 2 : 2125956 frames (100.0%)
&lt;ISAR&gt; Frames with MLD <= 5 : 2125956 frames (100.0%)
&lt;ISAR&gt; BE frames percentage = 100.0
&lt;ISAR&gt; max absolute diff = 0.0, sample range (-32768, 32767)
##########################################################
</code></pre>
</details>

## Executing specific tests only

All CUT tests can be run specifically for IVAS Encoder,IVAS Decoder,IVAS Renderer, ISAR Encoder and ISAR Decoder only. The commandline allows for ```-test-mode=<PARAM>``` for this functionality, examples : -

- Run DUT IVAS Encoder Tests Only (on Target Platform)

  ```shell
  PYTHONPATH=scripts python scripts/ivas_conformance/runConformance.py --testvecDir $PWD/testvec --cut_build_path=CUT_BIN_DIR --test-mode=ENC
  ```

- Analyse DUT IVAS Encoder Outputs Only (on Reference Platform)

  ```shell
  PYTHONPATH=scripts python scripts/ivas_conformance/runConformance.py --testvecDir $PWD/testvec --ref_build_path=testvec/bin --test-mode=ENC --analyse
  ```

- Run DUT IVAS Decoder Tests Only (on Target Platform)

  ```shell
  PYTHONPATH=scripts python scripts/ivas_conformance/runConformance.py --testvecDir $PWD/testvec --cut_build_path=CUT_BIN_DIR --test-mode=DEC
  ```

- Analyse DUT IVAS Decoder Outputs Only (on Reference Platform)

  ```shell
  PYTHONPATH=scripts python scripts/ivas_conformance/runConformance.py --testvecDir $PWD/testvec --ref_build_path=testvec/bin --test-mode=DEC --analyse
  ```

- Run DUT IVAS Renderer Tests Only (on Target Platform)

  ```shell
  PYTHONPATH=scripts python scripts/ivas_conformance/runConformance.py --testvecDir $PWD/testvec --cut_build_path=CUT_BIN_DIR --test-mode=REND
  ```

- Analyse DUT Renderer Outputs Only (on Reference Platform)

  ```shell
  PYTHONPATH=scripts python scripts/ivas_conformance/runConformance.py --testvecDir $PWD/testvec --ref_build_path=testvec/bin --test-mode=REND --analyse
  ```

- Run DUT ISAR Encoder Tests Only (on Target Platform)

  ```shell
  PYTHONPATH=scripts python scripts/ivas_conformance/runConformance.py --testvecDir $PWD/testvec --cut_build_path=CUT_BIN_DIR --test-mode=ISAR_ENC
  ```

- Analyse DUT ISAR Encoder Outputs Only (on Reference Platform)

  ```shell
  PYTHONPATH=scripts python scripts/ivas_conformance/runConformance.py --testvecDir $PWD/testvec --ref_build_path=testvec/bin --test-mode=ISAR_ENC --analyse
  ```

- Run DUT ISAR Decoder Tests Only (on Target Platform)

  ```shell
  PYTHONPATH=scripts python scripts/ivas_conformance/runConformance.py --testvecDir $PWD/testvec --cut_build_path=CUT_BIN_DIR --test-mode=ISAR
  ```

- Analyse DUT ISAR Decoder Outputs Only (on Reference Platform)

Encoder conformance:
	- PYTHONPATH=scripts python scripts/ivas_conformance/runConformance.py --testvecDir $PWD/testvec --ref_build_path=testvec/ref_bin --cut_build_path=testvec/cut_bin --test-mode=ENC
Decoder conformance:
	- PYTHONPATH=scripts python scripts/ivas_conformance/runConformance.py --testvecDir $PWD/testvec --ref_build_path=testvec/ref_bin --cut_build_path=testvec/cut_bin --test-mode=DEC
Renderer conformance:
	- PYTHONPATH=scripts python scripts/ivas_conformance/runConformance.py --testvecDir $PWD/testvec --ref_build_path=testvec/ref_bin --cut_build_path=testvec/cut_bin --test-mode=REND
Split renderer confomance:
	- <TBD>
  ```shell
  PYTHONPATH=scripts python scripts/ivas_conformance/runConformance.py --testvecDir $PWD/testvec --ref_build_path=testvec/bin --test-mode=ISAR --analyse
  ```
+385 −308

File changed.

Preview size limit exceeded, changes collapsed.