Merge branch 'main' of...

*.bs

*.192

mc.double

experiments/selection/*/proc_input*/cat*/*.wav

experiments/selection/*/proc_input*/cat*/*.csv

experiments/selection/*/proc_input*/cat*/*.pcm

experiments/selection/*/proc_input*/FOA*/*.wav

experiments/selection/*/proc_input*/HOA2*/*.wav

experiments/selection/*/background_noise/*.wav

experiments/selection/*/background_noise/*.txt

experiments/selection/*/proc_input*/*.wav

experiments/selection/*/proc_input*/*.pcm

experiments/selection/*/proc_input*/*.csv

experiments/selection/*/proc_input*/*.log

experiments/selection/*/proc_input*/*.yml

experiments/selection/*/proc_output*/

experiments/selection/*/config/*-lab_*.yml

experiments/selection/*/gen_input/IRs/*.wav

experiments/selection/*/gen_input/items_mono/*.wav

experiments/*/*/proc_input*/cat*/*.wav

experiments/*/*/proc_input*/cat*/*.csv

experiments/*/*/proc_input*/cat*/*.met

experiments/*/*/proc_input*/cat*/*.pcm

experiments/*/*/proc_input*/FOA*/*.wav

experiments/*/*/proc_input*/HOA2*/*.wav

experiments/*/*/background_noise/*.wav

experiments/*/*/background_noise/*.csv

experiments/*/*/background_noise/*.txt

experiments/*/*/proc_input*/*.wav

experiments/*/*/proc_input*/*.pcm

experiments/*/*/proc_input*/*.csv

experiments/*/*/proc_input*/*.log

experiments/*/*/proc_input*/*.yml

experiments/*/*/proc_output*/

experiments/*/*/config/*-lab_*.yml

experiments/*/*/gen_input/IRs/*.wav

experiments/*/*/gen_input/items_mono/*.wav

*~

tests/temp_output_*

tests/cut

****<!---

\*\*\*\*<!---

(C) 2022-2025 IVAS codec Public Collaboration with portions copyright Dolby International AB, Ericsson AB,

Fraunhofer-Gesellschaft zur Foerderung der angewandten Forschung e.V., Huawei Technologies Co. LTD.,

---

# Running characterization experiments

For an overview of the experiments, see [characterization_tests.md](characterization_tests.md)

## Prerequisites

- A working python environment (minimum version >= 3.9) with the required packages installed.

- The required additional binaries from ITU STL or the respective 3GPP contributions:

  - The binaries must be placed in ivas_processing_scripts/bin OR

  - a path specified in [binary_paths.yml](ivas_processing_scripts/binary_paths.yml) configuration file (please read the instructions carefully if using).

  - Please note the required patch for networkSimulator_g192 that must be manually applied.

- EVS fixed point binaries

  - These should be placed in the bin folder and named EVS_cod_fx and EVS_dec_fx

- Compiled versions of the codec from the BASOP repository:

  - Binaries compiled with ivas_float_update should be placed in the bin folder and named IVAS_cod_flt_ref and IVAS_dec_flt_ref

  - Binaries compiled with main (BASOP) should be placed in the bin folder and named IVAS_cod_fx and IVAS_dec_fx

- IVAS_rend from the floating-point repository:

  - This should be placed in the bin folder and named named IVAS_rend

- A local clone of this repository

## Setup

- Ensure you are on the latest version of the main branch of the processing scripts repo (`git pull`)

- Navigate to the box.com folder “Inbox_selected” for the experiment you wish to run.

- Download the experiment folder, e.g., “Inbox_selected/P.800/P.800-1”

- Copy/move the downloaded items into the corresponding folder in experiments/characterization in your copy of the processing scripts repo.

- e.g., `cp -r ~/Downloads/P.800-1/\* experiments/characterization/P800-1/`

## Processing

Run the processing from the root directory of the repository by invoking the python module with the experiment configuration YAML file:

```sh

python3 -m ivas_processing_scripts experiments/characterization/P800-1/config/P800-1_cat1_STEREO-16LKFS.yml

```

Repeat the above step for all configuration files to process all categories and levels.

Notes:

- The python module does NOT accept a list of configs; they must be run individually.

- This can be somewhat worked around with a shell loop, but please note this can make finding an error more difficult:

```sh

for config in $(ls experiments/characterization/P800-1/config/*.yml); do python3 -m ivas_processing_scripts $config; done

```

## Follow up tasks

After the processing is finished, the outputs will be present in the respective `proc_output` directories.

- For P.800 tests, these outputs need to be collected according to the test plan to assemble the final outputs required for P.800 test software.

- For MUSHRA tests, the outputs may be used directly, or in case of multi-input format experiments again need to be manually assembled.

At this point of time there is no script to automatically perform this.

## Directory structure

### P.800 experiments

```bash

.

└── P800-1

    ├── background_noise

    │   └── <background noise files.wav>

    ├── config

    │   └── <configurations from git.yaml>

    ├── proc_input

    │   ├── cat1

    │   │   ├── <item_name.wav>

    │   │   ├── <item_name.wav.{0,1,2,3}.csv> for ISM metadata

    │   │   └── <item_name.wav.met> for MASA metadata

    │   ├── cat2...

    │   └── ...cat6

    └── proc_output

        ├── cat1

        │   ├── out_-16LKFS

        │   │   ├── c01

        │   │   │   └── <item_name.c01.wav>

        │   │   ├── c02...

        │   │   └── ...cXX

        │   ├── out_-26LKFS

        │   └── out_-36LKFS

        ├── cat2...

        └── ...cat6

```

### BS.1534 experiments

```bash

.

└── BS1534-1

    ├── config

    │   └── <configurations from git.yaml>

    ├── proc_input[_FMT] # for multiple input formats, suffix is present

    │   ├── <item_name.wav>

    │   ├── <item_name.wav.{0,1,2,3}.csv> for ISM metadata

    │   └── <item_name.wav.met> for MASA metadata

    └── proc_output[_FMT] # for multiple input formats, suffix is present

        ├── c01

        │   └── <item_name.c01.wav>

        ├── c02...

        └── ...cXX

```

# Quick guide: How to set up a listening test

This section describes step by step how to set up a listening test according to the Processing Plan (IVAS-7) and Test Plan (IVAS-8).

The addition of custom background noise at specific level is supported for the STEREO and SBA formats only. For ISMs it's not applicable. For OMASA and OSBA formats, it is expected that the backround noise is provided in the FOA/HOA2/HOA3 format as the first item in the `input` list.

### Item processing

### Item processing (selection)

The input has to be in the folder `experiments/selection/P800-{X}/proc_input_{l}`. If item generation is performed previous to this step, the corresponding files are already in the right folder.

If this step is performed independently of the previous one the input files have to be copied to the respective folder sorted by category.

The set up for the MUSHRA test only consists of the item processing.

### Item processing

### Item processing (Selection)

To process a BS1534-{X}{x} (X = 1, 2, ...7, x = a, b) listening test, the input files have to be placed in the folder `experiments/selection/BS1534-{X}{x}/proc_input_{l}` and the command

`python generate_test.py BS1534-{X}{x},{l}` has to be run from the root of the repository. 'l' stands for the listening lab designator: a (Force Technology), b (HEAD acoustics), c (MQ University), d (Mesaqin.com).

## Additional Executables

The following additional executables are needed for the different processing steps:

| Processing step                                                                             | Executable                 | Where to find                                                                                                   |

| ------------------------------------------------------------------------------------------- | -------------------------- | --------------------------------------------------------------------------------------------------------------- |

| Loudness measurement and adjustment                                                         | bs1770demo                 | <https://github.com/ErikNorvell-Ericsson/STL> (Note branch)                                                     |

| MNRU                                                                                        | p50fbmnru                  | <https://github.com/openitu/STL>                                                                                |

| ESDRU                                                                                       | esdru                      | <https://github.com/openitu/STL>                                                                                |

| Frame error pattern application                                                             | eid-xor                    | <https://github.com/openitu/STL>                                                                                |

| Error pattern generation                                                                    | gen-patt                   | <https://www.itu.int/rec/T-REC-G.191-201003-S/en> (Note: Version in <https://github.com/openitu/STL> is buggy!) |

| Reverberation module                                                                        | reverb                     | <https://github.com/openitu/STL>                                                                                |

| Filtering, Resampling                                                                       | filter                     | <https://www.3gpp.org/ftp/tsg_sa/WG4_CODEC/TSGS4_76/docs/S4-131277.zip>                                         |

| Random offset/seed generation (necessary for background noise and FER bitstream processing) | random                     | <https://www.3gpp.org/ftp/tsg_sa/WG4_CODEC/TSGS4_76/docs/S4-131277.zip>                                         |

| JBM network simulator                                                                       | networkSimulator_g192      | <https://www.3gpp.org/ftp/tsg_sa/WG4_CODEC/TSGS4_76/docs/S4-131277.zip>                                         |

| MASA rendering (also used in loudness measurement of MASA items)                            | masaRenderer, masaAnalyzer | <https://www.3gpp.org/ftp/TSG_SA/WG4_CODEC/TSGS4_122_Athens/Docs/S4-230221.zip>                                 |

| EVS reference conditions                                                                    | EVS_cod, EVS_dec           | <https://www.3gpp.org/ftp/Specs/archive/26_series/26.443/26443-h00.zip>                                         |

Additional executables are needed for the different processing steps, please see the dedicated [README](ivas_processing_scripts/bin/README.md).

The necessary binaries have to be either placed in the [ivas_processing_scripts/bin](./ivas_processing_scripts/bin) folder or the path has to be specified in

[ivas_processing_scripts/binary_paths.yml](./ivas_processing_scripts/binary_paths.yml).

# Characterization experiments

Each link in the table leads to the config directory of the respective experiment.

## P.800

| Experiment                                              | Format               |

| ------------------------------------------------------- | -------------------- |

| [P800-1](experiments/characterization/P800-1/config/)   | Stereo               |

| [P800-2](experiments/characterization/P800-2/config/)   | Stereo               |

| [P800-3](experiments/characterization/P800-3/config/)   | FOA                  |

| [P800-4](experiments/characterization/P800-4/config/)   | HOA2                 |

| [P800-5](experiments/characterization/P800-5/config/)   | HOA3                 |

| [P800-6](experiments/characterization/P800-6/config/)   | MC 5.1, 7.1          |

| [P800-7](experiments/characterization/P800-7/config/)   | MC 5.1+4, 7.1+4      |

| [P800-8](experiments/characterization/P800-8/config/)   | MC - Mixed CICP      |

| [P800-9](experiments/characterization/P800-9/config/)   | 1-2 Objects          |

| [P800-10](experiments/characterization/P800-10/config/) | 3-4 Objects          |

| [P800-11](experiments/characterization/P800-11/config/) | 1-4 Objects          |

| [P800-12](experiments/characterization/P800-12/config/) | MASA 1 TC            |

| [P800-13](experiments/characterization/P800-13/config/) | MASA 2 TCs           |

| [P800-14](experiments/characterization/P800-14/config/) | MASA 1-2 TC          |

| [P800-15](experiments/characterization/P800-15/config/) | OSBA (1-2 objects)   |

| [P800-16](experiments/characterization/P800-16/config/) | OSBA (3-4 objects)   |

| [P800-17](experiments/characterization/P800-17/config/) | OSBA (1-4 objects)   |

| [P800-18](experiments/characterization/P800-18/config/) | OMASA (1-2 objects)  |

| [P800-19](experiments/characterization/P800-19/config/) | OMASA (3-4 objects)  |

| [P800-20](experiments/characterization/P800-20/config/) | OMASA (1-4 objects)  |

| [P800-21](experiments/characterization/P800-21/config/) | JBM with Stereo      |

| [P800-22](experiments/characterization/P800-22/config/) | JBM with 1-2 objects |

| [P800-23](experiments/characterization/P800-23/config/) | JBM with FOA         |

## BS.1534

| Experiment                                                  | Format                     |

| ----------------------------------------------------------- | -------------------------- |

| [BS1534-1](experiments/characterization/BS1534-1/config/)   | Stereo                     |

| [BS1534-2](experiments/characterization/BS1534-2/config/)   | Stereo                     |

| [BS1534-3](experiments/characterization/BS1534-3/config/)   | FOA                        |

| [BS1534-4](experiments/characterization/BS1534-4/config/)   | FOA                        |

| [BS1534-5](experiments/characterization/BS1534-5/config/)   | HOA3                       |

| [BS1534-6](experiments/characterization/BS1534-6/config/)   | Multichannel 5.1           |

| [BS1534-7](experiments/characterization/BS1534-7/config/)   | Multi-channel 5.1, 7.1     |

| [BS1534-8](experiments/characterization/BS1534-8/config/)   | Multi-channel 5.1+2, 5.1+4 |

| [BS1534-9](experiments/characterization/BS1534-9/config/)   | Multi-channel 7.1+4        |

| [BS1534-10](experiments/characterization/BS1534-10/config/) | ISM 1-2                    |

| [BS1534-11](experiments/characterization/BS1534-11/config/) | ISM 3-4                    |

| [BS1534-12](experiments/characterization/BS1534-12/config/) | ISM 3-4                    |

| [BS1534-13](experiments/characterization/BS1534-13/config/) | MASA (1TC)                 |

| [BS1534-14](experiments/characterization/BS1534-14/config/) | MASA (1TC)                 |

| [BS1534-15](experiments/characterization/BS1534-15/config/) | MASA (2TC)                 |

| [BS1534-16](experiments/characterization/BS1534-16/config/) | OSBA (1-4 obj.)            |

| [BS1534-17](experiments/characterization/BS1534-17/config/) | OSBA (1-4 obj.)            |

| [BS1534-18](experiments/characterization/BS1534-18/config/) | OMASA (1-4 obj.)           |

| [BS1534-19](experiments/characterization/BS1534-19/config/) | Stereo downmix for EVS     |

| [BS1534-20](experiments/characterization/BS1534-20/config/) | ISM 6 DoF (4 objects)      |

    ### EVS conditions ###

    c03:

        type: evs

        rend:

        ivas_rend:

            fmt: STEREO

        bitrates:

            - 16400

            bin: ivas_processing_scripts/bin/EVS_cod_fx

            fmt: MONO

            ivas_prerend: true

        dec:

            bin: ivas_processing_scripts/bin/EVS_dec_fx

    c04:

        type: evs

        rend:

        ivas_rend:

            fmt: STEREO

        bitrates:

            - 32000

            bin: ivas_processing_scripts/bin/EVS_cod_fx

            fmt: MONO

            ivas_prerend: true

        dec:

            bin: ivas_processing_scripts/bin/EVS_dec_fx

    ### IVAS FX conditions ###

    c05:

        type: ivas

    ### EVS conditions ###

    c03:

        type: evs

        rend:

        ivas_rend:

            fmt: STEREO

        bitrates:

            - 64000

            bin: ivas_processing_scripts/bin/EVS_cod_fx

            fmt: MONO

            ivas_prerend: true

        dec:

            bin: ivas_processing_scripts/bin/EVS_dec_fx

    c04:

        type: evs

        rend:

        ivas_rend:

            fmt: STEREO

        bitrates:

            - 128000

            bin: ivas_processing_scripts/bin/EVS_cod_fx

            fmt: MONO

            ivas_prerend: true

        dec:

            bin: ivas_processing_scripts/bin/EVS_dec_fx

    ### IVAS FX conditions ###

    c05:

        type: ivas