Commit d201088a authored by Marek Szczerba's avatar Marek Szczerba
Browse files

Merge branch 'main' into 89-add-reverb-parameters-generation-scripts

parents 1278a0b6 679f4bcf

.gitignore

+18 −16
Original line number Diff line number Diff line
@@ -12,22 +12,24 @@ venv/ 
*.bs

*.192

mc.double

experiments/selection/*/proc_input*/cat*/*.wav

experiments/selection/*/proc_input*/cat*/*.csv

experiments/selection/*/proc_input*/cat*/*.pcm

experiments/selection/*/proc_input*/FOA*/*.wav

experiments/selection/*/proc_input*/HOA2*/*.wav

experiments/selection/*/background_noise/*.wav

experiments/selection/*/background_noise/*.txt

experiments/selection/*/proc_input*/*.wav

experiments/selection/*/proc_input*/*.pcm

experiments/selection/*/proc_input*/*.csv

experiments/selection/*/proc_input*/*.log

experiments/selection/*/proc_input*/*.yml

experiments/selection/*/proc_output*/

experiments/selection/*/config/*-lab_*.yml

experiments/selection/*/gen_input/IRs/*.wav

experiments/selection/*/gen_input/items_mono/*.wav

experiments/*/*/proc_input*/cat*/*.wav

experiments/*/*/proc_input*/cat*/*.csv

experiments/*/*/proc_input*/cat*/*.met

experiments/*/*/proc_input*/cat*/*.pcm

experiments/*/*/proc_input*/FOA*/*.wav

experiments/*/*/proc_input*/HOA2*/*.wav

experiments/*/*/background_noise/*.wav

experiments/*/*/background_noise/*.csv

experiments/*/*/background_noise/*.txt

experiments/*/*/proc_input*/*.wav

experiments/*/*/proc_input*/*.pcm

experiments/*/*/proc_input*/*.csv

experiments/*/*/proc_input*/*.log

experiments/*/*/proc_input*/*.yml

experiments/*/*/proc_output*/

experiments/*/*/config/*-lab_*.yml

experiments/*/*/gen_input/IRs/*.wav

experiments/*/*/gen_input/items_mono/*.wav

*~

tests/temp_output_*

tests/cut

README.md

+379 −130
Original line number Diff line number Diff line 
****<!---

\*\*\*\*<!---



(C) 2022-2025 IVAS codec Public Collaboration with portions copyright Dolby International AB, Ericsson AB,

Fraunhofer-Gesellschaft zur Foerderung der angewandten Forschung e.V., Huawei Technologies Co. LTD.,
@@ -36,7 +36,7 @@ 


## Requirements



- Python >= 3.9

- Python 3.13.7 (may work with lower python versions, not guaranteed)

- numpy, scipy and pyyaml ([see requirements.txt](./requirements.txt))



---
@@ -47,10 +47,261 @@ 


---



# Running characterization experiments



For an overview of the experiments, see [characterization_tests.md](characterization_tests.md)



## Prerequisites



- A working python environment (minimum version >= 3.9) with the required packages installed.

- The required additional binaries from ITU STL or the respective 3GPP contributions:

  - The binaries must be placed in ivas_processing_scripts/bin OR

  - a path specified in [binary_paths.yml](ivas_processing_scripts/binary_paths.yml) configuration file (please read the instructions carefully if using).

  - Please note the required patch for networkSimulator_g192 that must be manually applied.

- EVS fixed point binaries

  - These should be placed in the bin folder and named EVS_cod_fx and EVS_dec_fx

- Compiled versions of the codec from the BASOP repository:

  - Binaries compiled with ivas_float_update should be placed in the bin folder and named IVAS_cod_flt_ref and IVAS_dec_flt_ref

  - Binaries compiled with main (BASOP) should be placed in the bin folder and named IVAS_cod_fx and IVAS_dec_fx

- IVAS_rend from the floating-point repository:

  - This should be placed in the bin folder and named named IVAS_rend

- A local clone of this repository



## Setup



- Ensure you are on the latest version of the main branch of the processing scripts repo (`git pull`)

- Navigate to the box.com folder “Inbox_selected” for the experiment you wish to run.

- Download the experiment folder, e.g., “Inbox_selected/P.800/P.800-1”

- Copy/move the downloaded items into the corresponding folder in experiments/characterization in your copy of the processing scripts repo.

- e.g., `cp -r ~/Downloads/P.800-1/\* experiments/characterization/P800-1/`



## Processing



Run the processing from the root directory of the repository by invoking the python module with the experiment configuration YAML file:



```sh

python3 -m ivas_processing_scripts experiments/characterization/P800-1/config/P800-1_cat1_STEREO-16LKFS.yml

```



Repeat the above step for all configuration files to process all categories and levels.



Notes:



- The python module does NOT accept a list of configs; they must be run individually.

- This can be somewhat worked around with a shell loop, but please note this can make finding an error more difficult:



```sh

# for POSIX shells

for config in $(ls experiments/characterization/P800-1/config/*.yml); do python3 -m ivas_processing_scripts $config; done

```



```batch

:: for cmd

for %f in (experiments\characterization\P800-1\config\*.yml) do python -m ivas_processing_scripts "%f"

```



## Follow up tasks



After the processing is finished, the outputs will be present in the respective `proc_output` directories.



### Item collection



- For P.800 tests, these outputs need to be collected according to the test plan to assemble the final outputs required for P.800 test software.



  - This can be done using the scripts in [collection-scripts](collection-scripts/characterization/README.md) (click for README) by running:



  ```batch

  python collection-scripts\characterization\p800_categorized_samples.py

  python collection-scripts\characterization\p800_preliminaries.py

  python collection-scripts\characterization\bs1534.py

  ```



  - These scripts collect items from each experiments `proc_output*` folder(s) and puts the needed files for the listening test into a `proc_final` folder. This folder needs to be uploaded for the dry run and the final delivery of the listening items to the labs.



### Hash generation and checking for duplicates



The hashes for the `proc_final` can be generated using the [get_md5.py](other/get_md5.py) script.

This script also checks for identical hashes and thus identifies duplicates in the output files which are reported in a printout.

When generating hashes one should check if duplicates are reported and if yes, what files are identical - note that there might be duplicates between the actual test and the preliminaries/training which is ok.

If there is a case with three or more items being the same or two items being the same inside the test or the preliminaries, the input files should be checked for duplicates.



Script usage:



```shell

> python other/get_md5.py --help

usage: get_md5.py [-h] output_dir out_file



Cross-platform script for generating MD5 hashes of output files for the characterization testing experiments.



positional arguments:

  output_dir  Path to the output directory of the respective experiment that you want to generate Hashes for. Can be proc_output or proc_final.

  out_file    Output text file with filenames and Hashes



options:

  -h, --help  show this help message and exit

```



For example:



```shell

python other/get_md5.py experiments/characterization/P800-1/proc_final p800-1_hashes.txt

```



## Directory structure



### P.800 experiments



#### Input structure for P.800



```bash

.

└── P800-1

    ├── background_noise

    │   └── <background noise files.wav>

    ├── config

    │   └── <configurations from git.yaml>

    └── proc_input

        ├── cat1

        │   ├── [l]p[ee]a[y]s[zz].c[nn].wav

        │   ├── [l]p[ee]a[y]s[zz].c[nn].wav.{0,1,2,3}.csv # for ISM metadata

        │   └── [l]p[ee]a[y]s[zz].c[nn].wav.met           # for MASA metadata

        ├── cat2...

        └── ...cat6

```



#### Scripts outputs for P.800



```bash

.

└── P800-1

    │   # the below directories are created by the processing and collection scripts

    ├── proc_final

    │   ├── c01

    │   │    └── [l]p[ee]a[y]s[zz].c[nn].wav

    │   ├── c02...

    │   ├── ...c36

    │   └── preliminaries

    │       └── [l]p[ee]a[y]s[zz].c[nn].wav

    └── proc_output

        ├── cat1

        │   ├── out_-16LKFS

        │   │   ├── c01

        │   │   │   └── [l]p[ee]a[y]s[zz].c[nn].wav

        │   │   ├── c02...

        │   │   └── ...cXX

        │   ├── out_-26LKFS

        │   └── out_-36LKFS

        ├── cat2...

        └── ...cat6

```



#### Output structure for P.800



The folders for upload to box.com and for delivery to the listeing labs should look like the structure below.

This is essentially the contents of the `proc_final` directory produced from the collection scripts.



```bash

.

└── P800-1

    ├── c01

    │    └── [l]p[ee]a[y]s[zz].c[nn].wav

    ├── c02...

    ├── ...c36

    ├── preliminaries

    │   └── [l]p[ee]a[y]s[zz].c[nn].wav

    └── P800-1_hashes.txt

```



### BS.1534 experiments



#### Input structure for BS.1534



```bash

.

└── BS1534-1

    ├── config

    │   └── <configurations from git.yaml>

    ├── proc_input[_FMT] # for multiple input formats, suffix is present

    │   ├── [l]m[ee]a[y]s[zz].c[nn].wav

    │   ├── [l]m[ee]a[y]s[zz].c[nn].wav.{0,1,2,3}.csv # for ISM metadata

    │   └── [l]m[ee]a[y]s[zz].c[nn].wav.met           # for MASA metadata

```



#### Scripts outputs for BS.1534



```bash

.

└── BS1534-1

    │   # the below directories are created by the processing and collection scripts

    ├── proc_final

    │   ├── c01

    │   │   └── [l]m[ee]a[y]s[zz].c[nn].wav

    │   ├── c02...

    │   ├── ...c08

    │   └── preliminaries

    │       └── [l]m[ee]a[y]s[zz].c[nn].wav

    └── proc_output[_FMT] # for multiple input formats, suffix is present

        ├── c01

        │   └── [l]m[ee]a[y]s[zz].c[nn].wav

        ├── c02...

        └── ...cXX

```



#### Output structure for BS.1534



The folders for upload to box.com and for delivery to the listeing labs should look like the structure below.

This is essentially the contents of the `proc_final` directory produced from the collection scripts.



```bash

.

└── BS1534-1

    ├── c01

    │   └── [l]m[ee]a[y]s[zz].c[nn].wav

    ├── c02...

    ├── ...c08

    ├── preliminaries

    │   └── [l]m[ee]a[y]s[zz].c[nn].wav

    └── BS1534-1_hashes.txt

```



## Reporting issues



If you encounter an issue while running the characterization tests, please follow the steps below depending on the type of issue you encounter.



### Crash in fixed point codec



Example output for a crash:



```shell

SystemError: Command returned non-zero exit status (-6): ivas_processing_scripts/bin/IVAS_cod_fx -max_band SWB -q -stereo 24400 32 experiments/characterization/P800_ACR-1/proc_output/cat1/out_-26LKFS/tmp_c27/20ms_aligned_files_concatenated.cod_fmt.wav experiments/characterization/P800_ACR-1/proc_output/cat1/out_-26LKFS/tmp_c27/20ms_aligned_files_concatenated.ivas.192

```



1. Copy the **entire traceback** to a text file for upload

2. Inspect the output for more information, and possibly the command that caused the crash, example:

3. Try to determine the output directory used, usually with the prefix `tmp_` in `proc_output`, in this case `tmp_c27`

4. Inspect the output files in the given directory, checking `.log` files for further info about commands that were run

5. Try to reproduce the crash outside of the scripts

6. Upload the required files to a new folder under [_EXTERNAL_IVAS_CHAR_MC/Issues](https://dolby.box.com/s/oc7k3ck8n04wbcknxu71t3wyvkq2cev4)

7. File an issue on the [BASOP repository](https://forge.3gpp.org/rep/sa4/audio/ivas-basop/-/issues)

    - Report git SHA for float reference codec, fixed point codec and processing scripts!



### Crash / unexpected behaviour in the scripts



1. Copy the **entire traceback** to a text file for upload

2. Inspect the output for more information, and where the exception occurred

3. If it is related to a certain condition in the script identify the related `tmp_` folder in `proc_output`

4. Collect the following files for upload

    - The `.yml` configuration file in **`proc_output`** - this is produced by the scripts and is NOT the one used to run the experiment

    - The main log file for the experiment, named the same as the above file with suffix `.log`

    - Any files in the related `tmp_` condition directory, if applicable. Use your discretion to choose pertinent files.

5. Upload the required files to a new folder under [_EXTERNAL_IVAS_CHAR_MC/Issues](https://dolby.box.com/s/oc7k3ck8n04wbcknxu71t3wyvkq2cev4)

6. File an issue on the [Processing scripts repository](https://forge.3gpp.org/rep/ivas-codec-pc/ivas-processing-scripts/-/issues)

    - Report git SHA for float reference codec, fixed point codec and processing scripts!



# Quick guide: How to set up a listening test



This section describes step by step how to set up a listening test according to the Processing Plan (IVAS-7) and Test Plan (IVAS-8).

In the following sections the only purpose of the curly brackets is to mark the variables that thave to be replaced with the actual values.

In the following sections the only purpose of the curly brackets is to mark the variables that have to be replaced with the actual values.



## P800
@@ -76,15 +327,17 @@ Each entry under `scenes:` describes one test item, specifying: 
- `input`: list of mono `.wav` files

- `azimuth` / `elevation`: spatial placement (°)

- `level`: loudness in dB

- `shift`: timing offsets in seconds



Dynamic positioning (e.g., `"-20:1.0:360"`) means the source will move over time, stepping every 20 ms.

- `shift`: signal offset/overlap in seconds

- `background`: background noise file (applicable to STEREO and SBA only)

- `background_level`: level of the background noise (applicable to STEREO and SBA only)



The total duration of the output signal can be controlled using the `duration` field. The output signal may optionally be rendered to the BINAURAL format by specifying the `binaural_output` field.

Dynamic positioning (e.g., `"-20:1.0:360"`) means the source will move over time, stepping every 20 ms. The maximum total duration of the output signal can be controlled using the `duration` field. The output signal may optionally be rendered to the BINAURAL format by specifying the `binaural_output` field. The `shift` parameter ensures time adjustment (offset) of the input signal (positive value delays the signal). Aternatively, the notation `X(i_ref)` generates overlap by `X` seconds from the reference signal `i_ref` (0-based index) (positive value creates gap).



Start by running a single scene to verify settings. Output includes both audio and optional metadata files. You can enable multiprocessing by setting `multiprocessing: true`.



### Item processing

The addition of custom background noise at specific level is supported for the STEREO and SBA formats only. For ISMs it's not applicable. For OMASA and OSBA formats, it is expected that the backround noise is provided in the FOA/HOA2/HOA3 format as the first item in the `input` list.



### Item processing (selection)



The input has to be in the folder `experiments/selection/P800-{X}/proc_input_{l}`. If item generation is performed previous to this step, the corresponding files are already in the right folder.

If this step is performed independently of the previous one the input files have to be copied to the respective folder sorted by category.
@@ -102,7 +355,7 @@ For more information about this processing step see 


The set up for the MUSHRA test only consists of the item processing.



### Item processing

### Item processing (Selection)



To process a BS1534-{X}{x} (X = 1, 2, ...7, x = a, b) listening test, the input files have to be placed in the folder `experiments/selection/BS1534-{X}{x}/proc_input_{l}` and the command

`python generate_test.py BS1534-{X}{x},{l}` has to be run from the root of the repository. 'l' stands for the listening lab designator: a (Force Technology), b (HEAD acoustics), c (MQ University), d (Mesaqin.com).
@@ -418,6 +671,7 @@ conditions_to_generate: 
      ### Encoder input sampling rate in Hz (resampling performed in case of mismatch); default = null (no resampling)

      # fs: 32000

      ### Additional commandline options; default = null

      ### NOTE: if using -stereo_dmx_evs ensure you specify dec:fmt as MONO to avoid errors

      # opts: ["-q", "-dtx", 4]

      ### Extended metadata flag for ISM > 64kbps, ignored otherwise; default = false

      # extended_metadata: true
@@ -431,6 +685,13 @@ conditions_to_generate: 
      # fs: 48000

      ### Additional commandline options; default = null

      # opts: ["-q", "-no_delay_cmp"]

      ### Per-item renderer configuration. Set to true to search for a file with suffix .cfg; default = false

      # render_config: true

      ### Head-tracking trajectory file for binaural output OR 'true' which will search for a file with the suffix .ht.csv next to the input; default = null

      ###   NOTE: this automatically configures the '-T' argument to the decoder, so may conflict if also specified in `opts`

      # trajectory: "path/to/file"

      ### Limit the trajectory to 3DoF via truncation; default = false

      # only_3dof: false



  ### IVAS condition ###############################

  c07:
@@ -516,8 +777,10 @@ postprocessing: 
  # bin_lfe_gain: 1

  ### Flag whether output should be limited to avoid clipping (can alter target loudness); default = true

  # limit: false

  ### Head-tracking trajectory file for binaural output; default = null

  ### Head-tracking trajectory file for binaural output OR 'true' which will search for a file with the suffix .ht.csv in the input dir; default = null

  # trajectory: "path/to/file"

  ### Limit the trajectory to 3DoF via truncation; default = false

  # only_3dof: false

```



</details>
@@ -615,21 +878,7 @@ The processing chain is as follows: 


## Additional Executables



The following additional executables are needed for the different processing steps:



| Processing step                                                                             | Executable                 | Where to find                                                                                                   |

| ------------------------------------------------------------------------------------------- | -------------------------- | --------------------------------------------------------------------------------------------------------------- |

| Loudness measurement and adjustment                                                         | bs1770demo                 | <https://github.com/ErikNorvell-Ericsson/STL> (Note branch)                                                     |

| MNRU                                                                                        | p50fbmnru                  | <https://github.com/openitu/STL>                                                                                |

| ESDRU                                                                                       | esdru                      | <https://github.com/openitu/STL>                                                                                |

| Frame error pattern application                                                             | eid-xor                    | <https://github.com/openitu/STL>                                                                                |

| Error pattern generation                                                                    | gen-patt                   | <https://www.itu.int/rec/T-REC-G.191-201003-S/en> (Note: Version in <https://github.com/openitu/STL> is buggy!) |

| Reverberation module                                                                        | reverb                     | <https://github.com/openitu/STL>                                                                                |

| Filtering, Resampling                                                                       | filter                     | <https://www.3gpp.org/ftp/tsg_sa/WG4_CODEC/TSGS4_76/docs/S4-131277.zip>                                         |

| Random offset/seed generation (necessary for background noise and FER bitstream processing) | random                     | <https://www.3gpp.org/ftp/tsg_sa/WG4_CODEC/TSGS4_76/docs/S4-131277.zip>                                         |

| JBM network simulator                                                                       | networkSimulator_g192      | <https://www.3gpp.org/ftp/tsg_sa/WG4_CODEC/TSGS4_76/docs/S4-131277.zip>                                         |

| MASA rendering (also used in loudness measurement of MASA items)                            | masaRenderer, masaAnalyzer | <https://www.3gpp.org/ftp/TSG_SA/WG4_CODEC/TSGS4_122_Athens/Docs/S4-230221.zip>                                 |

| EVS reference conditions                                                                    | EVS_cod, EVS_dec           | <https://www.3gpp.org/ftp/Specs/archive/26_series/26.443/26443-h00.zip>                                         |

Additional executables are needed for the different processing steps, please see the dedicated [README](ivas_processing_scripts/bin/README.md).



The necessary binaries have to be either placed in the [ivas_processing_scripts/bin](./ivas_processing_scripts/bin) folder or the path has to be specified in

[ivas_processing_scripts/binary_paths.yml](./ivas_processing_scripts/binary_paths.yml).

characterization_tests.md

0 → 100644
+56 −0
Original line number Diff line number Diff line 
# Characterization experiments



Each link in the table leads to the config directory of the respective experiment.



## P.800



| Experiment                                              | Format               |

| ------------------------------------------------------- | -------------------- |

| [P800-1](experiments/characterization/P800-1/config/)   | Stereo               |

| [P800-2](experiments/characterization/P800-2/config/)   | Stereo               |

| [P800-3](experiments/characterization/P800-3/config/)   | FOA                  |

| [P800-4](experiments/characterization/P800-4/config/)   | HOA2                 |

| [P800-5](experiments/characterization/P800-5/config/)   | HOA3                 |

| [P800-6](experiments/characterization/P800-6/config/)   | MC 5.1, 7.1          |

| [P800-7](experiments/characterization/P800-7/config/)   | MC 5.1+4, 7.1+4      |

| [P800-8](experiments/characterization/P800-8/config/)   | MC - Mixed CICP      |

| [P800-9](experiments/characterization/P800-9/config/)   | 1-2 Objects          |

| [P800-10](experiments/characterization/P800-10/config/) | 3-4 Objects          |

| [P800-11](experiments/characterization/P800-11/config/) | 1-4 Objects          |

| [P800-12](experiments/characterization/P800-12/config/) | MASA 1 TC            |

| [P800-13](experiments/characterization/P800-13/config/) | MASA 2 TCs           |

| [P800-14](experiments/characterization/P800-14/config/) | MASA 1-2 TC          |

| [P800-15](experiments/characterization/P800-15/config/) | OSBA (1-2 objects)   |

| [P800-16](experiments/characterization/P800-16/config/) | OSBA (3-4 objects)   |

| [P800-17](experiments/characterization/P800-17/config/) | OSBA (1-4 objects)   |

| [P800-18](experiments/characterization/P800-18/config/) | OMASA (1-2 objects)  |

| [P800-19](experiments/characterization/P800-19/config/) | OMASA (3-4 objects)  |

| [P800-20](experiments/characterization/P800-20/config/) | OMASA (1-4 objects)  |

| [P800-21](experiments/characterization/P800-21/config/) | JBM with Stereo      |

| [P800-22](experiments/characterization/P800-22/config/) | JBM with 1-2 objects |

| [P800-23](experiments/characterization/P800-23/config/) | JBM with FOA         |



## BS.1534



| Experiment                                                  | Format                     |

| ----------------------------------------------------------- | -------------------------- |

| [BS1534-1](experiments/characterization/BS1534-1/config/)   | Stereo                     |

| [BS1534-2](experiments/characterization/BS1534-2/config/)   | Stereo                     |

| [BS1534-3](experiments/characterization/BS1534-3/config/)   | FOA                        |

| [BS1534-4](experiments/characterization/BS1534-4/config/)   | FOA                        |

| [BS1534-5](experiments/characterization/BS1534-5/config/)   | HOA3                       |

| [BS1534-6](experiments/characterization/BS1534-6/config/)   | Multichannel 5.1           |

| [BS1534-7](experiments/characterization/BS1534-7/config/)   | Multi-channel 5.1, 7.1     |

| [BS1534-8](experiments/characterization/BS1534-8/config/)   | Multi-channel 5.1+2, 5.1+4 |

| [BS1534-9](experiments/characterization/BS1534-9/config/)   | Multi-channel 7.1+4        |

| [BS1534-10](experiments/characterization/BS1534-10/config/) | ISM 1-2                    |

| [BS1534-11](experiments/characterization/BS1534-11/config/) | ISM 3-4                    |

| [BS1534-12](experiments/characterization/BS1534-12/config/) | ISM 3-4                    |

| [BS1534-13](experiments/characterization/BS1534-13/config/) | MASA (1TC)                 |

| [BS1534-14](experiments/characterization/BS1534-14/config/) | MASA (1TC)                 |

| [BS1534-15](experiments/characterization/BS1534-15/config/) | MASA (2TC)                 |

| [BS1534-16](experiments/characterization/BS1534-16/config/) | OSBA (1-4 obj.)            |

| [BS1534-17](experiments/characterization/BS1534-17/config/) | OSBA (1-4 obj.)            |

| [BS1534-18](experiments/characterization/BS1534-18/config/) | OMASA (1-4 obj.)           |

| [BS1534-19](experiments/characterization/BS1534-19/config/) | Stereo downmix for EVS     |

| [BS1534-20](experiments/characterization/BS1534-20/config/) | ISM 6 DoF (4 objects)      |

collection-scripts/characterization/README.md

0 → 100644
+36 −0

File added.

Preview size limit exceeded, changes collapsed.

collection-scripts/characterization/bs1534.py

0 → 100644
+67 −0

File added.

Preview size limit exceeded, changes collapsed.