Merge branch 'main' of forge.3gpp.org:ivas-codec-pc/ivas-codec into...

---

# Python scripts for Testing the IVAS code and Generating test items

## Contents

0. [Requirements](#0-requirements)

1. [Scripts and classes for testing IVAS code](#1-scripts-and-classes-for-testing-ivas-code)

   - [Classes](#11-classes)

   - [Output directory structure](#12-output-directory-structure)

   - [Scripts](#13-scripts)

- [Python scripts for Testing the IVAS code and Generating test items](#python-scripts-for-testing-the-ivas-code-and-generating-test-items)

  - [Contents](#contents)

  - [0. Requirements](#0-requirements)

  - [- numpy and scipy for `generate_test_items.py`, `testBitexact.py` and `self_test.py`](#--numpy-and-scipy-for-generate_test_itemspy-testbitexactpy-and-self_testpy)

  - [1.  Scripts and classes for testing IVAS code](#1--scripts-and-classes-for-testing-ivas-code)

    - [1.1 Classes](#11-classes)

    - [1.2 Output directory structure](#12-output-directory-structure)

    - [1.3 Scripts](#13-scripts)

      - [Common commandline options for the scripts](#common-commandline-options-for-the-scripts)

     - [runIvasCodec.py](#runivascodecpy)

     - [IvasBuildAndRun.py](#ivasbuildandrunpy)

     - [IvasBuildAndRunChecks.py](#ivasbuildandruncheckspy)

     - [testBitexact.py](#testbitexactpy)

     - [self_test.py](#self_testpy)

2. [Script for generating listening test items](#2-script-for-generating-listening-test-items)

    - [generate_test_items.py](#21-generate_test_itemspy)

    - [Test configuration file](#22-test-configuration-file)

    - [Supported test conditions](#23-supported-test-conditions)

    - [Supported input/output/rendered audio formats](#24-supported-inputoutputrendered-audio-formats)

    - [Pre-renderer Metadata definiton](#25-pre-renderer-metadata-definition)

3. [Script for converting formats and binauralizing](#3-script-for-converting-formats-and-binauralizing)

    - [Binauralizing with head rotation](#31-binauralizing-with-head-rotation)

    - [Generating binaural reference signals](#32-generating-binaural-reference-signals)

      - [`runIvasCodec.py`](#runivascodecpy)

      - [`IvasBuildAndRun.py`](#ivasbuildandrunpy)

      - [`IvasBuildAndRunChecks.py`](#ivasbuildandruncheckspy)

      - [`testBitexact.py`](#testbitexactpy)

      - [`self_test.py`](#self_testpy)

  - [2. Script for generating listening test items](#2-script-for-generating-listening-test-items)

    - [2.1. `generate_test_items.py`](#21-generate_test_itemspy)

    - [2.2. Test configuration file](#22-test-configuration-file)

    - [2.3. Supported test conditions](#23-supported-test-conditions)

    - [2.4. Supported input/output/rendered audio formats](#24-supported-inputoutputrendered-audio-formats)

    - [2.5. Processing](#25-processing)

    - [2.6. Renderer Metadata definition](#26-renderer-metadata-definition)

  - [3. Script for converting formats and binauralizing](#3-script-for-converting-formats-and-binauralizing)

    - [3.1. Binauralizing with head rotation](#31-binauralizing-with-head-rotation)

    - [3.2. Generating binaural reference signals](#32-generating-binaural-reference-signals)

---

|--------------------------------------------------|----------------------|------------------------------------------------|

| MONO                                             |    yes/yes/yes       | mono signals                                   |

| STEREO                                           |    yes/yes/yes       | stereo signals                                 |

| ISM or ISMx                                      |    yes/no/no         | Objects with metadata, description using pre-renderer metadata |

| ISM or ISMx                                      |    yes/no/no         | Objects with metadata, description using renderer metadata |

| MASA or MASAx                                    |    yes/no/no         | mono or stereo signals with spatial metadata !!!metadata must share same basename as waveform file but with .met extension!!! |

| FOA/HOA2/HOA3 or PLANAR(FOA/HOAx)                |    yes/yes/yes       | Ambisonic signals or planar ambisonic signals  |

| BINAURAL/BINAURAL_ROOM                           |    no/yes/yes        | Binaural signals                               |

| 5_1/5_1_2/5_1_4/7_1/7_1_4 or CICP[6/12/14/16/19] |    yes/yes/yes       | Multi-channel signals for predefined loudspeaker layout |

| META                                             |    yes/yes/no        | Audio scene described by a pre-renderer config |

| META                                             |    yes/yes/no        | Audio scene described by a renderer config     |

---

---

### 2.6. Pre-renderer Metadata definition

### 2.6. Renderer Metadata definition

To run, the pre-renderer requires a config file describing the input scene.The expected format of the config file is as follows:

To run, the renderer requires a config file describing the input scene.The expected format of the config file is as follows:

---

- Line 1: Path to a "multitrack" audio file. This should be a single multichannel wav/pcm file that contains all input audio. For example channels 1-4 can be an FOA scene,channel 5 - an object and channels 6-11 - a 5.1 channel bed. If the path is not absolute, it is considered relative to the pre-renderer executable, not the config file. This path has lower priority than the one given on the command line: *The path in the config file is ignored if the --inputAudio argument to the  pre-renderer executable is specified.* When running together with the encoder using EVS_cod_prerenderer.sh, the path in  the config file is ignored and the one provided as script argument is used  instead.

- Line 1: Path to a "multitrack" audio file. This should be a single multichannel wav/pcm file that contains all input audio. For example channels 1-4 can be an FOA scene,channel 5 - an object and channels 6-11 - a 5.1 channel bed. If the path is not absolute, it is considered relative to the renderer executable, not the config file. This path has lower priority than the one given on the command line: *The path in the config file is ignored if the --inputAudio argument to the renderer executable is specified.*

---

- Line 2:  Contains number of inputs. An input can either be an Ambisonics scene, anobject or a channel bed.This is NOT the total number of channels in the input audio file.The pre-renderer currently supports simultaneously:  *Up to 2 SBA inputs,  Up to 2 MC inputs* Up to 16 ISM inputsThese limits can be freely changed with pre-processor macros, if needed.

- Line 2:  Contains number of inputs. An input can either be an Ambisonics scene, anobject or a channel bed.This is NOT the total number of channels in the input audio file.The renderer currently supports simultaneously:  *Up to 2 SBA inputs,  Up to 2 MC inputs* Up to 16 ISM inputsThese limits can be freely changed with pre-processor macros, if needed.

---

- Following lines: 

## 3. Script for converting formats and binauralizing

The script audio3dtool.py can convert between different input and output formats and binauralize the signals. 

The script audio3dtools.py can convert between different input and output formats and binauralize signals. 

Execute `python -m pyaudio3dtools.audio3dtools --help` for usage.

### 3.1. Binauralizing with head rotation

```

python -m pyaudio3dtools.audio3dtools -i cicp6_input.wav  -o . -F BINAURAL_REF -T .\trajectories\full_circle_in_15s

```

### 3.3. Rendering ISM to Custom loudspeakers with auxiliary binaural output

ISM metadata can either be specified via an input text file in the Renderer Metadata definition format, or via the commandline using the same style as IVAS:

```

python -m pyaudio3dtools.audio3dtools -i ism2.wav -f ISM2 -m ism1.csv NULL -F 7_1_4 -o . -b -T .\trajectories\full_circle_in_15s 

```

        "markers", "create_ref: mark test capable of producing references"

    )

    config.addinivalue_line(

        "markers", "create_ref_serial: reference creation test that should not run in parallel"

        "markers", "create_ref_part2: reference creation test that depends on create_ref references"

    )

import platform

from pathlib import Path

sys.path.append('scripts/ivas_pytests/tests/')

from cut_pcm import cut_samples

BIN_EXT = ".exe" if platform.system() == "Windows" else ""

HERE = Path(__file__).parent.resolve()

DEFAULT_ENCODER_DUT = str(HERE.joinpath(f"../../IVAS_cod{BIN_EXT}").resolve())

    build_crend_unittest(dut_src_dir)

def create_short_testvectors():

    """

    Create short (5sec) testvectors.

    """

    print("Creating short (5sec) testvectors")

    num_channels = "4" # currently only FOA

    cut_from = "0.0"

    cut_len = "5.0"

    for fs in ['48', '32', '16']:

        in_file = f"{TEST_VECTOR_DIR}/stvFOA{fs}c.pcm"

        cut_gain = "1.0"

        cut_file = f"{TEST_VECTOR_DIR}/stvFOA{fs}c_cut.pcm"

        cut_samples(in_file, cut_file, num_channels, fs + "000", cut_from, cut_len, cut_gain)

        cut_gain = "16.0"

        cut_file = f"{TEST_VECTOR_DIR}/stvFOA{fs}c_cut_{cut_gain}.pcm"

        cut_samples(in_file, cut_file, num_channels, fs + "000", cut_from, cut_len, cut_gain)

def main(argv):

    # check for python >= 3.7

    if sys.version_info[0] < 3 or sys.version_info[1] < 7:

    else:

        # create references

        print(f"Creating references within the references directory {REFERENCE_DIR}")

        create_short_testvectors()

        if platform.system() == "Windows":

            base_cmd = ["pytest"]

        else:

            base_cmd = ["python3", "-m", "pytest"]

        base_cmd += [

            "scripts/ivas_pytests/tests",

            "-n",

            args.numprocesses,

            "--update_ref",

            "1",

            "-v",

        my_env["TESTVECTOR_PATH_REL_TRUNK"] = "/scripts/ivas_pytests/testv/"  # leading "/" is important

        my_env["CREND_UNIT_TEST_BIN"] = CREND_UNITTEST_REF

        print("pytest command line to be executed from project root folder:")

        print(" ".join(base_cmd + ["-m", "create_ref", "-n", args.numprocesses]))

        subprocess.run(base_cmd + ["-m", "create_ref", "-n", args.numprocesses], check=False, env=my_env)

        print(" ".join(base_cmd + ["-m", "create_ref"]))

        subprocess.run(base_cmd + ["-m", "create_ref"], check=False, env=my_env)

        print("pytest command line to be executed from project root folder:")

        print(" ".join(base_cmd + ["-m", "create_ref_serial"]))

        subprocess.run(base_cmd + ["-m", "create_ref_serial"], check=False, env=my_env)

        print(" ".join(base_cmd + ["-m", "create_ref_part2"]))

        subprocess.run(base_cmd + ["-m", "create_ref_part2"], check=False, env=my_env)

    if args.create_only:

        return

dtx_set = ['0', '1']

ivas_br_list = ['32000', '64000', '96000', '256000']

sampling_rate_list = ['48', '32', '16']

# we need signals amplified by 24dB - until those are available, use existing 'stvFOA'

agc_tag_list = ['stvFOA']

# TODO: create and use new 24dB signals

# agc_tag_list = ['stvFOA_24dB_']

agc_list = [0, 1]

ch_count_foa = 4

AbsTol = '3'

                raise  # raises the error again

@pytest.mark.create_ref_serial

# assumption:

# - the needed reference bitstreams are created by test_spar_foa_enc_system

# -> reference bitstreams are not any longer created as part of this test

# -> the parameters of this test (except additional parameter plc_pattern) need to be a subset of the parameters in test_spar_foa_enc_system

# -> the reference generation for this test (reference decoder output) needs to be done after completion of test_spar_foa_enc_system

#    -> therefore the marker create_ref_part2

@pytest.mark.create_ref_part2

@pytest.mark.parametrize("ivas_br", ivas_br_list)

@pytest.mark.parametrize("dtx", dtx_set)

@pytest.mark.parametrize("tag", tag_list)

@pytest.mark.parametrize("plc_pattern", plc_patterns)

@pytest.mark.parametrize("fs", sampling_rate_list)

@pytest.mark.parametrize("agc", agc_list)

def test_spar_foa_plc_system(

    dut_decoder_frontend: DecoderFrontend,

    data_system_tests_path,

    reference_path,

    dut_base_path,

    ref_encoder_path,

    ref_decoder_path,

    update_ref,

    ivas_br,

    dtx,

    tag,

    plc_pattern,

    fs

):

    tag = tag + fs + 'c'

    agc = 0

    if update_ref == 1 and plc_pattern == plc_patterns[0] and ref_encoder_path:

        # enc

        spar_foa_ref_enc(ref_encoder_path, data_system_tests_path, reference_path, tag, fs, ivas_br, dtx, agc)

    #dec

    spar_foa_dec_plc(dut_decoder_frontend, data_system_tests_path, reference_path, dut_base_path, ref_decoder_path, tag, ch_count_foa, fs, ivas_br, dtx, plc_pattern, update_ref, agc)

@pytest.mark.create_ref_serial

@pytest.mark.parametrize("ivas_br", ivas_br_list)

@pytest.mark.parametrize("dtx", dtx_set)

@pytest.mark.parametrize("tag", agc_tag_list)

@pytest.mark.parametrize("plc_pattern", plc_patterns)

@pytest.mark.parametrize("fs", sampling_rate_list)

def test_spar_foa_agc_plc_system(

    dut_decoder_frontend: DecoderFrontend,

    data_system_tests_path,

    reference_path,

    dut_base_path,

    ref_encoder_path,

    ref_decoder_path,

    update_ref,

    ivas_br,

    dtx,

    tag,

    plc_pattern,

    fs

    fs,

    agc

):

    tag = tag + fs + 'c'

    agc = 1

    if update_ref == 1 and plc_pattern == plc_patterns[0] and ref_encoder_path:

        # enc

        spar_foa_ref_enc(ref_encoder_path, data_system_tests_path, reference_path, tag, fs, ivas_br, dtx, agc)

    #dec

    spar_foa_dec_plc(dut_decoder_frontend, data_system_tests_path, reference_path, dut_base_path, ref_decoder_path, tag, ch_count_foa, fs, ivas_br, dtx, plc_pattern, update_ref, agc)

#########################################################

############ test function ##############################

def spar_foa_ref_enc(

    ref_encoder_path,

    test_vector_path,

    reference_path,

    tag,

    sampling_rate,

    ivas_br,

    dtx,

    agc

):

    #########  run cmd   #####################################

    ref_out_dir = f"{reference_path}/spar_foa_bs/pkt"

    check_and_makedir(ref_out_dir)

    tag_out = f"{tag}_ivasbr{ivas_br[:-3]}k_DTX{dtx}"

    if agc == 1:

        tag_out += '_AGC1'

    input_path = f"{test_vector_path}/{tag}.pcm"

    cut_file = f"{test_vector_path}/{tag}_cut.pcm"

    # we expect that the reference generation for create_ref did already run and the cut PCM files are available

    assert os.path.exists(cut_file)

    input_path = cut_file

    ref_pkt_file = f"{ref_out_dir}/{tag_out}.pkt"

    dtx_mode = dtx == '1'

    ref_encoder = EncoderFrontend(ref_encoder_path, "REF")

    # call REF encoder

    ref_encoder.run(

        ivas_br,

        sampling_rate,

        input_path,

        ref_pkt_file,

        sba_order="+1",

        max_band="FB",

        agc_op=agc,

        dtx_mode=dtx_mode,

    )

def spar_foa_dec_plc(

    decoder_frontend,

    test_vector_path,

        ##report failure

        assert not test_fail

    if ref_decoder_path:

        # Unclear whether this clean-up is still needed

        # TODO: check temp file generation

        ##File removal##

        for count in range(ch_count, 25):

            ch_id = str(count + 1)

            temp_raw_file = f"{ref_out_dir}/out{ch_id}ch.raw"

            if os.path.isfile(temp_raw_file):

                os.remove(temp_raw_file)

        temp_md_file = f"{ref_out_dir}/MD_OUT_DUMMY.BIN"

        if os.path.isfile(temp_md_file):

            os.remove(temp_md_file)

sample_rate_list = ['48', '32', '16']

bypass_list = [1, 2]

# we need signals amplified by 24dB - until those are available, use existing 'stvFOA'

agc_tag_list = ['stvFOA']

# TODO: create and use new 24dB signals

# agc_tag_list = ['stvFOA_24dB_']

agc_list = [0, 1]

sample_rate_bw_idx_list = [('48', 'SWB'), ('48', 'WB'), ('32', 'WB')]

    dtx = '0'

    max_bw = "FB"

    agc = 0

    use_agc_ref = 0

    sba_order = "+1"

    output_config = "FOA"

        max_bw,

        bypass,

        agc,

        use_agc_ref,

        sba_order,

        update_ref

    )

        max_bw,

        bypass,

        agc,

        use_agc_ref,

        output_config,

        update_ref

    )

@pytest.mark.parametrize("dtx", dtx_set)

@pytest.mark.parametrize("tag", tag_list)

@pytest.mark.parametrize("fs", sample_rate_list)

#@pytest.mark.parametrize("agc", agc_list)

@pytest.mark.parametrize("agc", agc_list)

def test_spar_foa_enc_system(

    dut_encoder_frontend: EncoderFrontend,

    dut_decoder_frontend: DecoderFrontend,

    dtx,

    tag,

    fs,

#    agc

    agc

):

    tag = tag + fs + 'c'

    max_bw = "FB"

    bypass = -1

    agc = 0

    use_agc_ref = 0

    sba_order = "+1"

    output_config = "FOA"

    if agc == 1:

        cut_gain = "16.0"

    else:

        cut_gain = "1.0"

    # enc

    spar_foa_enc(

        max_bw,

        bypass,

        agc,

        use_agc_ref,

        sba_order,

        update_ref

        update_ref,

        cut_gain=cut_gain

    )

    # dec

        max_bw,

        bypass,

        agc,

        use_agc_ref,

        output_config,

        update_ref

    )

#    tag = tag + '_' + fs + 'k'

    max_bw = "FB"

    bypass = -1

    use_agc_ref = 0

    sba_order = "+2"

    output_config = "HOA2"

        max_bw,

        bypass,

        agc,

        use_agc_ref,

        sba_order,

        update_ref,

        '.wav'

        max_bw,

        bypass,

        agc,

        use_agc_ref,

        output_config,

        update_ref

    )

#    tag = tag + '_' + fs + 'k'

    max_bw = "FB"

    bypass = -1

    use_agc_ref = 0

    sba_order = "+3"

    output_config = "HOA3"

        max_bw,

        bypass,

        agc,

        use_agc_ref,

        sba_order,

        update_ref,

        '.wav'

        max_bw,

        bypass,

        agc,

        use_agc_ref,

        output_config,

        update_ref

    )

    tag = tag + fs + 'c'

    bypass = -1

    agc = 0

    use_agc_ref = 0

    sba_order = "+1"

    output_config = "FOA"

        bw,

        bypass,

        agc,

        use_agc_ref,

        sba_order,

        update_ref

    )

        bw,

        bypass,

        agc,

        use_agc_ref,

        output_config,

        update_ref

    )

@pytest.mark.create_ref

@pytest.mark.parametrize("ivas_br", ivas_br_FOA)

@pytest.mark.parametrize("dtx", dtx_set)

@pytest.mark.parametrize("tag", agc_tag_list)

@pytest.mark.parametrize("fs", sample_rate_list)

def test_spar_foa_enc_agc_system(

    dut_encoder_frontend: EncoderFrontend,

    dut_decoder_frontend: DecoderFrontend,

    data_system_tests_path,

    reference_path,

    dut_base_path,

    ref_encoder_path,

    ref_decoder_path,

    update_ref,

    ivas_br,

    dtx,

    tag,

    fs

):

    tag = tag + fs + 'c'

    max_bw = "FB"

    bypass = -1

    agc = 1

    use_agc_ref = 1

    sba_order = "+1"

    output_config = "FOA"

    # enc

    spar_foa_enc(

        dut_encoder_frontend,

        data_system_tests_path,

        ref_encoder_path,

        reference_path,

        dut_base_path,

        tag,

        fs,

        ivas_br,

        dtx,

        max_bw,

        bypass,

        agc,

        use_agc_ref,

        sba_order,

        update_ref,

        cut_gain="16.0"

    )

    # dec

    spar_foa_dec(

        dut_decoder_frontend,

        ref_decoder_path,

        reference_path,

        dut_base_path,

        tag,

        ch_count_foa,

        fs,

        ivas_br,

        dtx,

        max_bw,

        bypass,

        agc,

        use_agc_ref,

        output_config,

        update_ref

    )

    ivas_max_bw,

    bypass,

    agc,

    use_agc_ref,

    sba_order,

    update_ref,

    in_extension = '.pcm',

    tag_out = f"{tag}_ivasbr{ivas_br[:-3]}k_DTX{dtx}"

    short_tag_ext = ""

    if agc == 1 and use_agc_ref == 1:

    if agc == 1:

        short_tag_ext += '_AGC1'

    # we update only bypass = 0/2 (bypass 1 is the same as the baseline)

    if bypass in [0, 2]:

    dtx_mode = dtx == '1'

    if in_extension == '.pcm':

        # cut input PCM file - currently with fixed (i.e. not test dependant) values

        # use shortened and potentially gain adjusted input PCM file - create if not present

        # cut input PCM file: currently with mostly fixed (i.e. not test dependant) values

        num_channels = "4" # currently only FOA inputs end up, here

        cut_from = "0.0"

        cut_len = "5.0"

    ivas_max_bw,

    bypass,

    agc,

    use_agc_ref,

    output_config,

    update_ref,

    keep_files=False

    tag_out = f"{tag}_ivasbr{ivas_br[:-3]}k_DTX{dtx}"

    short_tag_ext = ""

    if agc == 1 and use_agc_ref == 1:

    if agc == 1:

        short_tag_ext += '_AGC1'

    # we update only bypass = 0/2 (bypass 1 is the same as the baseline)

    if bypass in [0, 2]:

        for count in range(ch_count):

            ch_id = str(count + 1)

            # TEST

            fsize1 = os.path.getsize(f"{dut_out_dir}/out{ch_id}ch.raw")

            fsize2 = os.path.getsize(f"{ref_out_dir}/out{ch_id}ch.raw")

            print(f"Want to compare {dut_out_dir}/out{ch_id}ch.raw ({fsize1} bytes) with {ref_out_dir}/out{ch_id}ch.raw ({fsize2} bytes)")

            if cmp_custom(

                f"{dut_out_dir}/out{ch_id}ch.raw",

                f"{ref_out_dir}/out{ch_id}ch.raw",

        ##report failure

        assert not test_fail

    if ref_decoder_path:

        # Unclear whether this clean-up is still needed

        # TODO: check temp file generation

        ##File removal##

        for count in range(ch_count, 25):

            ch_id = str(count + 1)

            temp_raw_file = f"{ref_out_dir}/out{ch_id}ch.raw"

            if os.path.isfile(temp_raw_file):

                os.remove(temp_raw_file)

        temp_md_file = f"{ref_out_dir}/MD_OUT_DUMMY.BIN"

        if os.path.isfile(temp_md_file):

            os.remove(temp_md_file)