Commit e938dde2 authored by Tapani Pihlajakuja's avatar Tapani Pihlajakuja
Browse files

Add more instructions and example for BE output.

parent ca659e0c
Loading
Loading
Loading
Loading
Loading
+34 −5
Original line number Diff line number Diff line
@@ -12,7 +12,7 @@ Reference platform is Ubuntu 24.04
  lsb_release -d | grep Ubuntu
  ```

<span style="color: green;"> # It might be similar to Ubuntu 24.04.3 LTS </span>
It might be similar to Ubuntu 24.04.3 LTS

- Install Clang 18 compiler

@@ -98,21 +98,49 @@ CUT_OUTPUTS

## Perform analysis on the reference platform (Ubuntu 24.04)

If CUT test execution is done on a different platform, the scripts/CUT_OUTPUTS must be copied and provided in the reference platform's scripts/CUT_OUTPUTS. Then the BE analysis or non-BE analysis procedure below should be followed.
If CUT test execution is done on a different platform, the scripts/CUT_OUTPUTS must be copied and provided in the reference platform's scripts/CUT_OUTPUTS. Then the BE analysis or non-BE analysis procedure below should be followed. It is recommended to perform first analysis with BE comparison and then analysis with non-BE comparison if non-BE outputs were found. Note, non-BE conformance applies currently only if metadata file output is BE and there are non-BE results only in wave-file output.


### Perform the BE comparison on the CUT outputs on reference platform 

The BE comparison is performed to the CUT outputs using the command below. Encoded outputs will be decoded using the reference decoder executables as part of the process. The BE comparison is then performed between the CUT and reference decoded outputs. This includes comparison of ".wav"-files, and ".csv" and ".met" metadata files. If any non-BE results are observed, this is reported on the command-line and link to an analysis ".csv" file is given. The analysis file shows which exact files were non-BE.
The BE comparison is performed to the CUT outputs using the command below. Encoded outputs will be decoded using the reference decoder executables as part of the process. The BE comparison is then performed between the CUT and reference decoded outputs. This includes comparison of ".wav"-files, and ".csv" and ".met" metadata files. If any non-BE results are observed, this is reported on the command-line and link to an analysis ".csv" file is given. The analysis file shows which exact files were non-BE. An example passing output is shown below. If all test sets print `PASSED BE TEST`, then CUT outputs are BE-conformant.
  
```shell
PYTHONPATH=scripts python scripts/ivas_conformance/runConformance.py --testvecDir $PWD/testvec --ref_build_path=testvec/bin --analyse --be-test
```

<details>
<summary>Example Output of BE comparison</summary>
<pre><code>
Accumulating commands from Readme_IVAS_dec.txt
Accumulating commands from Readme_IVAS_enc.txt
Accumulating commands from Readme_IVAS_rend.txt
Accumulating commands from Readme_IVAS_JBM_dec.txt
Accumulating commands from Readme_IVAS_ISAR_dec.txt
Accumulating commands from Readme_IVAS_ISAR_post_rend.txt
No of tests :
    ENC : 374
    DEC : 638
    REND : 911
    ISAR_ENC : 1032
    ISAR : 1252
Analysing tests for ENC   (374 tests)
&lt;ENC&gt; PASSED BE TEST
Analysing tests for DEC   (638 tests)
&lt;DEC&gt; PASSED BE TEST
Analysing tests for REND   (911 tests)
&lt;REND&gt; PASSED BE TEST
Analysing tests for ISAR_ENC   (1032 tests)
&lt;ISAR_ENC&gt; PASSED BE TEST
Analysing tests for ISAR   (1252 tests)
&lt;ISAR&gt; PASSED BE TEST
</code></pre>
</details>


### Perform the MLD based non-BE analysis on the CUT outputs on reference platform (Ubuntu 24.04)

The MLD-based non-BE analysis is performed to the CUT outputs with the command below. Encoded outputs will be decoded using the reference decoder executables as part of the process. The MLD analysis is then performed between the CUT and reference decoded outputs (only ".wav" files are compared). Comparison to MLD corridor is also done as part of this process. The example output is also shown below.
The MLD-based non-BE analysis is performed to the CUT outputs with the command below. Encoded outputs will be decoded using the reference decoder executables as part of the process. The MLD analysis is then performed between the CUT and reference decoded outputs (only ".wav" files are compared). Comparison to MLD corridor is also done as part of this process. An example passing output is shown below. If all test sets print `MLD Corridor passed for...` and there were no non-BE metadata comparisons in BE-test, then CUT outputs are Non-BE conformant.

```shell
PYTHONPATH=scripts python scripts/ivas_conformance/runConformance.py --testvecDir $PWD/testvec --ref_build_path=testvec/bin --analyse
@@ -207,7 +235,8 @@ Analysing tests for ISAR (1252 tests)
&lt;ISAR&gt; max absolute diff = 0.0, sample range (-32768, 32767)
##########################################################

MLD Corridor passed for ISAR with max MLD diff of 0.0</code></pre>
MLD Corridor passed for ISAR with max MLD diff of 0.0
</code></pre>
</details>