Commit 4c0f489c authored by Vladimir Malenovsky's avatar Vladimir Malenovsky
Browse files

docs: update CUT/BE/non-BE output examples to current script format

parent 6e860438

scripts/ivas_conformance/README.md

+38 −84
Original line number Diff line number Diff line
@@ -75,17 +75,27 @@ Accumulating commands from Readme_IVAS_enc.txt 
Accumulating commands from Readme_IVAS_ISAR_post_rend.txt

Accumulating commands from Readme_IVAS_ISAR_dec.txt

Accumulating commands from Readme_IVAS_JBM_dec.txt



No of tests :

    ENC : 381

    DEC : 637

    REND : 666

    ISAR_ENC : 1032

    ISAR : 1032

Executing tests for ENC   (381 tests)

Executing tests for DEC   (637 tests)

Executing tests for REND   (666 tests)

Executing tests for ISAR_ENC   (1032 tests)

Executing tests for ISAR   (1032 tests)



Executing tests for ENC  (381 tests):

---------------------------

[ENC] OK



...



Summary of results:

---------------------

[ENC] OK

[DEC] OK

[REND] OK

[ISAR_ENC] OK

[ISAR] OK

</code></pre>

</details>
@@ -108,7 +118,7 @@ If CUT test execution is done on a different platform, the scripts/CUT_OUTPUTS m 


### Perform the BE comparison on the CUT outputs on reference platform 



The BE comparison is performed to the CUT outputs using the command below. Encoded outputs will be decoded using the reference decoder executables as part of the process. The BE comparison is then performed between the CUT and reference decoded outputs. This includes comparison of ".wav"-files, and ".csv" and ".met" metadata files. If any non-BE results are observed, this is reported on the command-line and link to an analysis ".csv" file is given. The analysis file shows which exact files were non-BE. An example passing output is shown below. If all test sets print `PASSED BE TEST`, then CUT outputs are BE-conformant.

The BE comparison is performed to the CUT outputs using the command below. Encoded outputs are decoded using the reference decoder executables as part of the process. The BE comparison is then performed between the CUT and reference decoded outputs. This includes comparison of `.wav` files and `.csv`/`.met` metadata files. If non-BE results are observed, this is reported on the command line and in the generated analysis CSV output.

  

```shell

PYTHONPATH=scripts python scripts/ivas_conformance/runConformance.py --testvecDir $PWD/testvec --ref_build_path=testvec/bin --analyse --be-test
@@ -123,29 +133,37 @@ Accumulating commands from Readme_IVAS_rend.txt 
Accumulating commands from Readme_IVAS_JBM_dec.txt

Accumulating commands from Readme_IVAS_ISAR_dec.txt

Accumulating commands from Readme_IVAS_ISAR_post_rend.txt



No of tests :

    ENC : 374

    DEC : 638

    REND : 911

    ISAR_ENC : 1032

    ISAR : 1252

Analysing tests for ENC   (374 tests)

&lt;ENC&gt; PASSED BE TEST

Analysing tests for DEC   (638 tests)

&lt;DEC&gt; PASSED BE TEST

Analysing tests for REND   (911 tests)

&lt;REND&gt; PASSED BE TEST

Analysing tests for ISAR_ENC   (1032 tests)

&lt;ISAR_ENC&gt; PASSED BE TEST

Analysing tests for ISAR   (1252 tests)

&lt;ISAR&gt; PASSED BE TEST



Analysing tests for ENC  (374 tests):

---------------------------



[ENC] OK (ERRORS=0, BE=374, NON-BE=0, MLD CORRIDOR FAILURES=0)

[DEC] OK (ERRORS=0, BE=638, NON-BE=0, MLD CORRIDOR FAILURES=0)



...



Summary of results:

---------------------

[ENC] OK

[DEC] OK

[REND] OK

[ISAR_ENC] OK

[ISAR] OK

</code></pre>

</details>





### Perform the MLD based non-BE analysis on the CUT outputs on reference platform (Ubuntu 24.04)



The MLD-based non-BE analysis is performed to the CUT outputs with the command below. Encoded outputs will be decoded using the reference decoder executables as part of the process. The MLD analysis is then performed between the CUT and reference decoded outputs (only ".wav" files are compared). Comparison to MLD corridor is also done as part of this process. An example passing output is shown below. If all test sets print `MLD Corridor passed for...` and there were no non-BE metadata comparisons in BE-test, then CUT outputs are Non-BE conformant.

The non-BE analysis below compares CUT and reference outputs by running MLD on audio (`.wav`) and, when MASA metadata are generated, for the matching reference/CUT `.met` files. For encoder tests, encoded CUT bitstreams are first decoded with the reference decoder before analysis. Per-frame MLD and MASA metadata values are written to `scripts/CUT_OUTPUTS` and checked against corridor references in `testvec/testv/mld_ref` (`mld_ref_<TAG>.csv` and `masa_ref_<TAG>.csv`).





```shell

PYTHONPATH=scripts python scripts/ivas_conformance/runConformance.py --testvecDir $PWD/testvec --ref_build_path=testvec/bin --analyse
@@ -160,12 +178,14 @@ Accumulating commands from Readme_IVAS_rend.txt 
Accumulating commands from Readme_IVAS_JBM_dec.txt

Accumulating commands from Readme_IVAS_ISAR_dec.txt

Accumulating commands from Readme_IVAS_ISAR_post_rend.txt



No of tests :

    ENC : 374

    DEC : 638

    REND : 911

    ISAR_ENC : 1032

    ISAR : 1252



Analysing tests for ENC  (374 tests):

---------------------------
@@ -183,73 +203,7 @@ Analysing tests for ENC (374 tests): 


[ENC] OK (ERRORS=0, BE=374, NON-BE=0, MLD CORRIDOR FAILURES=0)



Analysing tests for DEC  (638 tests):

---------------------------



##########################################################

&lt;DEC&gt; Total Frames: 5079252

&lt;DEC&gt; MAX MLD across all frames : 0.0

&lt;DEC&gt; Frames with MLD == 0 : 5079252 frames (100.0%)

&lt;DEC&gt; Frames with MLD <= 0.5 : 5079252 frames (100.0%)

&lt;DEC&gt; Frames with MLD <= 1 : 5079252 frames (100.0%)

&lt;DEC&gt; Frames with MLD <= 2 : 5079252 frames (100.0%)

&lt;DEC&gt; Frames with MLD <= 5 : 5079252 frames (100.0%)

&lt;DEC&gt; BE samples percentage = 100.0

&lt;DEC&gt; max absolute diff = 0.0, sample range (-32768, 32767)

##########################################################



[DEC] OK (ERRORS=0, BE=638, NON-BE=0, MLD CORRIDOR FAILURES=0)



Analysing tests for REND  (911 tests):

---------------------------



##########################################################

&lt;REND&gt; Total Frames: 5576907

&lt;REND&gt; MAX MLD across all frames : 0.0

&lt;REND&gt; Frames with MLD == 0 : 5576907 frames (100.0%)

&lt;REND&gt; Frames with MLD <= 0.5 : 5576907 frames (100.0%)

&lt;REND&gt; Frames with MLD <= 1 : 5576907 frames (100.0%)

&lt;REND&gt; Frames with MLD <= 2 : 5576907 frames (100.0%)

&lt;REND&gt; Frames with MLD <= 5 : 5576907 frames (100.0%)

&lt;REND&gt; BE samples percentage = 100.0

&lt;REND&gt; max absolute diff = 0.0, sample range (-32768, 32767)

##########################################################



[REND] OK (ERRORS=0, BE=911, NON-BE=0, MLD CORRIDOR FAILURES=0)



Analysing tests for ISAR_ENC  (1032 tests):

---------------------------



##########################################################

&lt;ISAR_ENC&gt; Total Frames: 2125956

&lt;ISAR_ENC&gt; MAX MLD across all frames : 0.0

&lt;ISAR_ENC&gt; Frames with MLD == 0 : 2125956 frames (100.0%)

&lt;ISAR_ENC&gt; Frames with MLD <= 0.5 : 2125956 frames (100.0%)

&lt;ISAR_ENC&gt; Frames with MLD <= 1 : 2125956 frames (100.0%)

&lt;ISAR_ENC&gt; Frames with MLD <= 2 : 2125956 frames (100.0%)

&lt;ISAR_ENC&gt; Frames with MLD <= 5 : 2125956 frames (100.0%)

&lt;ISAR_ENC&gt; BE samples percentage = 100.0

&lt;ISAR_ENC&gt; max absolute diff = 0.0, sample range (-32768, 32767)

##########################################################



[ISAR_ENC] OK (ERRORS=0, BE=1032, NON-BE=0, MLD CORRIDOR FAILURES=0)



Analysing tests for ISAR  (1252 tests):

---------------------------



##########################################################

&lt;ISAR&gt; Total Frames: 2590956

&lt;ISAR&gt; MAX MLD across all frames : 0.0

&lt;ISAR&gt; Frames with MLD == 0 : 2590956 frames (100.0%)

&lt;ISAR&gt; Frames with MLD <= 0.5 : 2590956 frames (100.0%)

&lt;ISAR&gt; Frames with MLD <= 1 : 2590956 frames (100.0%)

&lt;ISAR&gt; Frames with MLD <= 2 : 2590956 frames (100.0%)

&lt;ISAR&gt; Frames with MLD <= 5 : 2590956 frames (100.0%)

&lt;ISAR&gt; BE samples percentage = 100.0

&lt;ISAR&gt; max absolute diff = 0.0, sample range (-32768, 32767)

##########################################################



[ISAR] OK (ERRORS=0, BE=1252, NON-BE=0, MLD CORRIDOR FAILURES=0)

...



Summary of results:

---------------------