From 1a38f45ac412d17bae3f0aa60e253ed672da7011 Mon Sep 17 00:00:00 2001 From: Tapani Pihlajakuja Date: Sat, 15 Nov 2025 13:45:22 +0200 Subject: [PATCH 1/3] Updates to conformance README.md --- scripts/ivas_conformance/README.md | 163 +++++++++++++++++------------ 1 file changed, 98 insertions(+), 65 deletions(-) diff --git a/scripts/ivas_conformance/README.md b/scripts/ivas_conformance/README.md index f8e2428506..53bee7f5f7 100644 --- a/scripts/ivas_conformance/README.md +++ b/scripts/ivas_conformance/README.md @@ -4,7 +4,7 @@ This folder contains scripts for running IVAS conformance tests. ## Setup Used for Conformance package generation Platform - Reference platform is Ubuntu 24.04 +Reference platform is Ubuntu 24.04 - Verify the Ubuntu Linux release is 24.04 @@ -12,7 +12,7 @@ This folder contains scripts for running IVAS conformance tests. lsb_release -d | grep Ubuntu ``` - # It might be similar to Ubuntu 24.04.3 LTS + # It might be similar to Ubuntu 24.04.3 LTS - Install Clang 18 compiler @@ -21,7 +21,7 @@ This folder contains scripts for running IVAS conformance tests. clang-18 --version ``` - Example version observed on Ubuntu 24.04.3 LTS +Example version observed on Ubuntu 24.04.3 LTS ```text Ubuntu clang version 18.1.3 (1ubuntu1) @@ -30,7 +30,7 @@ This folder contains scripts for running IVAS conformance tests. InstalledDir: /usr/bin ``` - It might be required to set Clang-18 as the default clang on the machine +It might be required to set Clang-18 as the default clang on the machine ```shell sudo update-alternatives --install /usr/bin/clang clang /usr/bin/clang-18 100 @@ -53,19 +53,19 @@ This folder contains scripts for running IVAS conformance tests. ## Run CUT tests on Target platform - To run CUT binaries on the targeted platform, it is necessary to replicate the initial setup for python and dependency packages. The CUT build of the IVAS binaries should be made available in a selected folder and needed for the next step +To run CUT binaries on the targeted platform, it is necessary to replicate the initial setup for python and dependency packages. The CUT build of the IVAS binaries should be made available in a selected folder and needed for the next step - To run IVAS DUT commands on the TARGET platform (may be different from ubuntu/linux) +To run IVAS DUT commands on the TARGET platform (may be different from ubuntu/linux) - ```shell - PYTHONPATH=scripts python scripts/ivas_conformance/runConformance.py --testvecDir $PWD/testvec --cut_build_path=CUT_BIN_DIR - ``` +```shell +PYTHONPATH=scripts python scripts/ivas_conformance/runConformance.py --testvecDir $PWD/testvec --cut_build_path=CUT_BIN_DIR +```
Example Output of CUT execution 

 Accumulating commands from Readme_IVAS_dec.txt
-Accumulating commands from Readme_IVAS_rend.txt
+Accumulating commands from Readme_IVAS_rend.txt   
 Accumulating commands from Readme_IVAS_enc.txt
 Accumulating commands from Readme_IVAS_ISAR_post_rend.txt
 Accumulating commands from Readme_IVAS_ISAR_dec.txt
@@ -84,103 +84,136 @@ Executing tests for ISAR   (1032 tests)
- This should generate outputs in scripts/CUT_OUTPUTS folder which looks like below:- +This should generate outputs in scripts/CUT_OUTPUTS folder which looks like below:- - ```shell - CUT_OUTPUTS - +- runlog.txt : Dump of all the commands run and the outputs (mostly jumbled up due to multiprocessing) - +- failedCmds.txt : Log of all the shell commands that failed execution - +- dec/ : Folder containing all decoder tests CUT outputs - +- enc/ : Folder containing all encoder tests CUT outputs - +- renderer_short/ : Folder containing all renderer tests CUT outputs - +- split_rendering/ : Folder containing all split rendering enc/dec tests - ``` +```shell +CUT_OUTPUTS + +- runlog.txt : Dump of all the commands run and the outputs (mostly jumbled up due to multiprocessing) + +- failedCmds.txt : Log of all the shell commands that failed execution + +- dec/ : Folder containing all decoder tests CUT outputs + +- enc/ : Folder containing all encoder tests CUT outputs + +- renderer_short/ : Folder containing all renderer tests CUT outputs + +- split_rendering/ : Folder containing all split rendering enc/dec tests +``` -## Perform the MLD based analysis on the CUT outputs on reference platform (Ubuntu 24.04) +## Perform analysis on the reference platform (Ubuntu 24.04) - If CUT test execution is done on a different platform, the scripts/CUT_OUTPUTS must be copied and provided in the reference platform's scripts/CUT_OUTPUTS. Then the following command is used to perform MLD based analysis on the same, encoded outputs will be implicitly decoded using reference decoder executables and MLD analysis performed on the reference decoded outputs. +If CUT test execution is done on a different platform, the scripts/CUT_OUTPUTS must be copied and provided in the reference platform's scripts/CUT_OUTPUTS. Then the BE analysis or non-BE analysis procedure below should be followed. - ```shell - PYTHONPATH=scripts python scripts/ivas_conformance/runConformance.py --testvecDir $PWD/testvec --ref_build_path=testvec/bin --analyse - ``` +### Perform the BE comparison on the CUT outputs on reference platform + +The BE comparison is performed to the CUT outputs using the command below. Encoded outputs will be decoded using the reference decoder executables as part of the process. The BE comparison is then performed between the CUT and reference decoded outputs. This includes comparison of ".wav"-files, and ".csv" and ".met" metadata files. If any non-BE results are observed, this is reported on the command-line and link to an analysis ".csv" file is given. The analysis file shows which exact files were non-BE. + +```shell +PYTHONPATH=scripts python scripts/ivas_conformance/runConformance.py --testvecDir $PWD/testvec --ref_build_path=testvec/bin --analyse --be-test +``` + + +### Perform the MLD based non-BE analysis on the CUT outputs on reference platform (Ubuntu 24.04) + +The MLD-based non-BE analysis is performed to the CUT outputs with the command below. Encoded outputs will be decoded using the reference decoder executables as part of the process. The MLD analysis is then performed between the CUT and reference decoded outputs. Comparison to MLD corridor is also done as part of this process. The example output is also shown below. + +```shell +PYTHONPATH=scripts python scripts/ivas_conformance/runConformance.py --testvecDir $PWD/testvec --ref_build_path=testvec/bin --analyse +```
-Example Output of CUT Analysis +Example Output of non-BE analysis 

 Accumulating commands from Readme_IVAS_dec.txt
-Accumulating commands from Readme_IVAS_rend.txt
 Accumulating commands from Readme_IVAS_enc.txt
-Accumulating commands from Readme_IVAS_ISAR_post_rend.txt
-Accumulating commands from Readme_IVAS_ISAR_dec.txt
+Accumulating commands from Readme_IVAS_rend.txt
 Accumulating commands from Readme_IVAS_JBM_dec.txt
+Accumulating commands from Readme_IVAS_ISAR_dec.txt
+Accumulating commands from Readme_IVAS_ISAR_post_rend.txt
 No of tests :
-    ENC : 381
-    DEC : 637
-    REND : 666
+    ENC : 374
+    DEC : 638
+    REND : 911
     ISAR_ENC : 1032
-    ISAR : 1032
-Analysing tests for ENC   (381 tests)
+    ISAR : 1252
+Analysing tests for ENC   (374 tests)
+
 ##########################################################
-<ENC> Total Frames: 2635800
+<ENC> Total Frames: 3074220
 <ENC> MAX MLD across all frames : 0.0
-<ENC> Frames with MLD == 0 : 2635800 frames (100.0%)
-<ENC> Frames with MLD <= 1 : 2635800 frames (100.0%)
-<ENC> Frames with MLD <= 2 : 2635800 frames (100.0%)
-<ENC> Frames with MLD <= 5 : 2635800 frames (100.0%)
-<ENC> BE frames percentage = 100.0
+<ENC> Frames with MLD == 0 : 3074220 frames (100.0%)
+<ENC> Frames with MLD <= 0.5 : 3074220 frames (100.0%)
+<ENC> Frames with MLD <= 1 : 3074220 frames (100.0%)
+<ENC> Frames with MLD <= 2 : 3074220 frames (100.0%)
+<ENC> Frames with MLD <= 5 : 3074220 frames (100.0%)
+<ENC> BE samples percentage = 100.0
 <ENC> max absolute diff = 0.0, sample range (-32768, 32767)
 ##########################################################
-Analysing tests for DEC   (637 tests)
+
+MLD Corridor passed for ENC with max MLD diff of 0.0
+Analysing tests for DEC   (638 tests)
+
 ##########################################################
-<DEC> Total Frames: 4342140
+<DEC> Total Frames: 5079252
 <DEC> MAX MLD across all frames : 0.0
-<DEC> Frames with MLD == 0 : 4342140 frames (100.0%)
-<DEC> Frames with MLD <= 1 : 4342140 frames (100.0%)
-<DEC> Frames with MLD <= 2 : 4342140 frames (100.0%)
-<DEC> Frames with MLD <= 5 : 4342140 frames (100.0%)
-<DEC> BE frames percentage = 100.0
+<DEC> Frames with MLD == 0 : 5079252 frames (100.0%)
+<DEC> Frames with MLD <= 0.5 : 5079252 frames (100.0%)
+<DEC> Frames with MLD <= 1 : 5079252 frames (100.0%)
+<DEC> Frames with MLD <= 2 : 5079252 frames (100.0%)
+<DEC> Frames with MLD <= 5 : 5079252 frames (100.0%)
+<DEC> BE samples percentage = 100.0
 <DEC> max absolute diff = 0.0, sample range (-32768, 32767)
 ##########################################################
-Analysing tests for REND   (666 tests)
+
+MLD Corridor passed for DEC with max MLD diff of 0.0
+Analysing tests for REND   (911 tests)
+
 ##########################################################
-<REND> Total Frames: 4799952
+<REND> Total Frames: 5576907
 <REND> MAX MLD across all frames : 0.0
-<REND> Frames with MLD == 0 : 4799952 frames (100.0%)
-<REND> Frames with MLD <= 1 : 4799952 frames (100.0%)
-<REND> Frames with MLD <= 2 : 4799952 frames (100.0%)
-<REND> Frames with MLD <= 5 : 4799952 frames (100.0%)
-<REND> BE frames percentage = 100.0
+<REND> Frames with MLD == 0 : 5576907 frames (100.0%)
+<REND> Frames with MLD <= 0.5 : 5576907 frames (100.0%)
+<REND> Frames with MLD <= 1 : 5576907 frames (100.0%)
+<REND> Frames with MLD <= 2 : 5576907 frames (100.0%)
+<REND> Frames with MLD <= 5 : 5576907 frames (100.0%)
+<REND> BE samples percentage = 100.0
 <REND> max absolute diff = 0.0, sample range (-32768, 32767)
 ##########################################################
+
+MLD Corridor passed for REND with max MLD diff of 0.0
 Analysing tests for ISAR_ENC   (1032 tests)
+
 ##########################################################
 <ISAR_ENC> Total Frames: 2125956
 <ISAR_ENC> MAX MLD across all frames : 0.0
 <ISAR_ENC> Frames with MLD == 0 : 2125956 frames (100.0%)
+<ISAR_ENC> Frames with MLD <= 0.5 : 2125956 frames (100.0%)
 <ISAR_ENC> Frames with MLD <= 1 : 2125956 frames (100.0%)
 <ISAR_ENC> Frames with MLD <= 2 : 2125956 frames (100.0%)
 <ISAR_ENC> Frames with MLD <= 5 : 2125956 frames (100.0%)
-<ISAR_ENC> BE frames percentage = 100.0
-<ISAR_ENC> max absolute diff = 0.0, sample range (-32768,32767)
+<ISAR_ENC> BE samples percentage = 100.0
+<ISAR_ENC> max absolute diff = 0.0, sample range (-32768, 32767)
 ##########################################################
-Analysing tests for ISAR   (1032 tests)
+
+MLD Corridor passed for ISAR_ENC with max MLD diff of 0.0
+Analysing tests for ISAR   (1252 tests)
+
 ##########################################################
-<ISAR> Total Frames: 2125956
+<ISAR> Total Frames: 2590956
 <ISAR> MAX MLD across all frames : 0.0
-<ISAR> Frames with MLD == 0 : 2125956 frames (100.0%)
-<ISAR> Frames with MLD <= 1 : 2125956 frames (100.0%)
-<ISAR> Frames with MLD <= 2 : 2125956 frames (100.0%)
-<ISAR> Frames with MLD <= 5 : 2125956 frames (100.0%)
-<ISAR> BE frames percentage = 100.0
+<ISAR> Frames with MLD == 0 : 2590956 frames (100.0%)
+<ISAR> Frames with MLD <= 0.5 : 2590956 frames (100.0%)
+<ISAR> Frames with MLD <= 1 : 2590956 frames (100.0%)
+<ISAR> Frames with MLD <= 2 : 2590956 frames (100.0%)
+<ISAR> Frames with MLD <= 5 : 2590956 frames (100.0%)
+<ISAR> BE samples percentage = 100.0
 <ISAR> max absolute diff = 0.0, sample range (-32768, 32767)
 ##########################################################
-
+ +MLD Corridor passed for ISAR with max MLD diff of 0.0
+ ## Executing specific tests only -All CUT tests can be run specifically for IVAS Encoder,IVAS Decoder,IVAS Renderer, ISAR Encoder and ISAR Decoder only. The commandline allows for ```-test-mode=``` for this functionality, examples : - +All CUT tests can be run specifically for IVAS Encoder,IVAS Decoder,IVAS Renderer, ISAR Encoder and ISAR Decoder only. The commandline allows for ```-test-mode=``` for this functionality, examples: - Run DUT IVAS Encoder Tests Only (on Target Platform) -- GitLab From ca659e0cec1943c0b2f574731b8444ad27b189f4 Mon Sep 17 00:00:00 2001 From: Tapani Pihlajakuja Date: Sat, 15 Nov 2025 13:51:59 +0200 Subject: [PATCH 2/3] Add mention that that non-BE is comparison is performed only for wav-files. --- scripts/ivas_conformance/README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/scripts/ivas_conformance/README.md b/scripts/ivas_conformance/README.md index 53bee7f5f7..42dd191f8e 100644 --- a/scripts/ivas_conformance/README.md +++ b/scripts/ivas_conformance/README.md @@ -112,7 +112,7 @@ PYTHONPATH=scripts python scripts/ivas_conformance/runConformance.py --testvecDi ### Perform the MLD based non-BE analysis on the CUT outputs on reference platform (Ubuntu 24.04) -The MLD-based non-BE analysis is performed to the CUT outputs with the command below. Encoded outputs will be decoded using the reference decoder executables as part of the process. The MLD analysis is then performed between the CUT and reference decoded outputs. Comparison to MLD corridor is also done as part of this process. The example output is also shown below. +The MLD-based non-BE analysis is performed to the CUT outputs with the command below. Encoded outputs will be decoded using the reference decoder executables as part of the process. The MLD analysis is then performed between the CUT and reference decoded outputs (only ".wav" files are compared). Comparison to MLD corridor is also done as part of this process. The example output is also shown below. ```shell PYTHONPATH=scripts python scripts/ivas_conformance/runConformance.py --testvecDir $PWD/testvec --ref_build_path=testvec/bin --analyse -- GitLab From e938dde2660986c8f4d373669bfe31aeeec90538 Mon Sep 17 00:00:00 2001 From: Tapani Pihlajakuja Date: Sat, 15 Nov 2025 14:14:39 +0200 Subject: [PATCH 3/3] Add more instructions and example for BE output. --- scripts/ivas_conformance/README.md | 39 ++++++++++++++++++++++++++---- 1 file changed, 34 insertions(+), 5 deletions(-) diff --git a/scripts/ivas_conformance/README.md b/scripts/ivas_conformance/README.md index 42dd191f8e..2caca4838b 100644 --- a/scripts/ivas_conformance/README.md +++ b/scripts/ivas_conformance/README.md @@ -12,7 +12,7 @@ Reference platform is Ubuntu 24.04 lsb_release -d | grep Ubuntu ``` - # It might be similar to Ubuntu 24.04.3 LTS +It might be similar to Ubuntu 24.04.3 LTS - Install Clang 18 compiler @@ -98,21 +98,49 @@ CUT_OUTPUTS ## Perform analysis on the reference platform (Ubuntu 24.04) -If CUT test execution is done on a different platform, the scripts/CUT_OUTPUTS must be copied and provided in the reference platform's scripts/CUT_OUTPUTS. Then the BE analysis or non-BE analysis procedure below should be followed. +If CUT test execution is done on a different platform, the scripts/CUT_OUTPUTS must be copied and provided in the reference platform's scripts/CUT_OUTPUTS. Then the BE analysis or non-BE analysis procedure below should be followed. It is recommended to perform first analysis with BE comparison and then analysis with non-BE comparison if non-BE outputs were found. Note, non-BE conformance applies currently only if metadata file output is BE and there are non-BE results only in wave-file output. ### Perform the BE comparison on the CUT outputs on reference platform -The BE comparison is performed to the CUT outputs using the command below. Encoded outputs will be decoded using the reference decoder executables as part of the process. The BE comparison is then performed between the CUT and reference decoded outputs. This includes comparison of ".wav"-files, and ".csv" and ".met" metadata files. If any non-BE results are observed, this is reported on the command-line and link to an analysis ".csv" file is given. The analysis file shows which exact files were non-BE. +The BE comparison is performed to the CUT outputs using the command below. Encoded outputs will be decoded using the reference decoder executables as part of the process. The BE comparison is then performed between the CUT and reference decoded outputs. This includes comparison of ".wav"-files, and ".csv" and ".met" metadata files. If any non-BE results are observed, this is reported on the command-line and link to an analysis ".csv" file is given. The analysis file shows which exact files were non-BE. An example passing output is shown below. If all test sets print `PASSED BE TEST`, then CUT outputs are BE-conformant. ```shell PYTHONPATH=scripts python scripts/ivas_conformance/runConformance.py --testvecDir $PWD/testvec --ref_build_path=testvec/bin --analyse --be-test ``` +
+Example Output of BE comparison +

+Accumulating commands from Readme_IVAS_dec.txt
+Accumulating commands from Readme_IVAS_enc.txt
+Accumulating commands from Readme_IVAS_rend.txt
+Accumulating commands from Readme_IVAS_JBM_dec.txt
+Accumulating commands from Readme_IVAS_ISAR_dec.txt
+Accumulating commands from Readme_IVAS_ISAR_post_rend.txt
+No of tests :
+    ENC : 374
+    DEC : 638
+    REND : 911
+    ISAR_ENC : 1032
+    ISAR : 1252
+Analysing tests for ENC   (374 tests)
+<ENC> PASSED BE TEST
+Analysing tests for DEC   (638 tests)
+<DEC> PASSED BE TEST
+Analysing tests for REND   (911 tests)
+<REND> PASSED BE TEST
+Analysing tests for ISAR_ENC   (1032 tests)
+<ISAR_ENC> PASSED BE TEST
+Analysing tests for ISAR   (1252 tests)
+<ISAR> PASSED BE TEST
+
+
+ ### Perform the MLD based non-BE analysis on the CUT outputs on reference platform (Ubuntu 24.04) -The MLD-based non-BE analysis is performed to the CUT outputs with the command below. Encoded outputs will be decoded using the reference decoder executables as part of the process. The MLD analysis is then performed between the CUT and reference decoded outputs (only ".wav" files are compared). Comparison to MLD corridor is also done as part of this process. The example output is also shown below. +The MLD-based non-BE analysis is performed to the CUT outputs with the command below. Encoded outputs will be decoded using the reference decoder executables as part of the process. The MLD analysis is then performed between the CUT and reference decoded outputs (only ".wav" files are compared). Comparison to MLD corridor is also done as part of this process. An example passing output is shown below. If all test sets print `MLD Corridor passed for...` and there were no non-BE metadata comparisons in BE-test, then CUT outputs are Non-BE conformant. ```shell PYTHONPATH=scripts python scripts/ivas_conformance/runConformance.py --testvecDir $PWD/testvec --ref_build_path=testvec/bin --analyse @@ -207,7 +235,8 @@ Analysing tests for ISAR (1252 tests) <ISAR> max absolute diff = 0.0, sample range (-32768, 32767) ########################################################## -MLD Corridor passed for ISAR with max MLD diff of 0.0 +MLD Corridor passed for ISAR with max MLD diff of 0.0 + -- GitLab