Apple Clang 16 Release vs Debug differences in MASA

Commit SHA: 32c65a1d (probably)

This issue shows differences in Release vs. Debug build run of short self-test on Mac M4, OS version 15.6.1, Apple Clang 16. This was discussed in IVAS conformance call on Monday September 22nd.

This is the issue for MASA format. Mostly small differences but one case is 1012 so it is worth to take look at that.

Test showed these results

FAILED tests/codec_be_on_mr_nonselection/test_param_file.py::test_param_file_tests[stv-MASA 1dir 1TC at 13.2 kbps, 48kHz in, 48kHz out, DTX on, 7_1_4 out] - Failed: Difference between ref and dut in output only (Non-BE - MAX_ABS_DIFF: 1)
FAILED tests/codec_be_on_mr_nonselection/test_param_file.py::test_param_file_tests[stv-MASA 1dir 1TC at 32 kbps, 48kHz in, 48kHz out, 7_1_4, random FER at 5%] - Failed: Difference between ref and dut in output only (Non-BE - MAX_ABS_DIFF: 1)
FAILED tests/codec_be_on_mr_nonselection/test_param_file.py::test_param_file_tests[stv-MASA 1dir 1TC bitrate switching from 13.2 kbps to 128 kbps, 48kHz in, 32kHz out, 5_1 out, JBM Prof 5] - Failed: Difference between ref and dut in output only (Non-BE - MAX_ABS_DIFF: 1)
FAILED tests/codec_be_on_mr_nonselection/test_param_file.py::test_param_file_tests[stv-MASA 1dir 1TC bitrate switching from 13.2 kbps to 128 kbps, 48kHz in, 48kHz out, 5_1 out] - Failed: Difference between ref and dut in output only (Non-BE - MAX_ABS_DIFF: 1)
FAILED tests/codec_be_on_mr_nonselection/test_param_file.py::test_param_file_tests[stv-MASA 1dir 1TC bitrate switching from 13.2 kbps to 128 kbps, 48kHz in, 48kHz out, BINAURAL out] - Failed: Difference between ref and dut in output only (Non-BE - MAX_ABS_DIFF: 1)
FAILED tests/codec_be_on_mr_nonselection/test_param_file.py::test_param_file_tests[stv-MASA 1dir 1TC bitrate switching from 13.2 kbps to 128 kbps, 48kHz in, 48kHz out, FOA out] - Failed: Difference between ref and dut in output only (Non-BE - MAX_ABS_DIFF: 1)
FAILED tests/codec_be_on_mr_nonselection/test_param_file.py::test_param_file_tests[stv-MASA 1dir 1TC bitrate switching from 13.2 kbps to 128 kbps, 48kHz in, 48kHz out, STEREO out] - Failed: Difference between ref and dut in output only (Non-BE - MAX_ABS_DIFF: 1)
FAILED tests/codec_be_on_mr_nonselection/test_param_file.py::test_param_file_tests[stv-MASA 1dir 2TC at 128 kbps, 48kHz in, 48kHz out, DTX on, EXT out] - Failed: Difference between ref and dut in output (Non-BE - MAX_ABS_DIFF: 1012) and metadata
FAILED tests/codec_be_on_mr_nonselection/test_param_file.py::test_param_file_tests[stv-MASA 1dir 2TC at 13.2 kbps, 48kHz in, 48kHz out, 5_1 out, random FER at 5%, bandwidth switching] - Failed: Difference between ref and dut in output only (Non-BE - MAX_ABS_DIFF: 1)
FAILED tests/codec_be_on_mr_nonselection/test_param_file.py::test_param_file_tests[stv-MASA 1dir 2TC at 16.4 kbps, 48kHz in, 48kHz out, DTX on, FOA out] - Failed: Difference between ref and dut in output only (Non-BE - MAX_ABS_DIFF: 1)
FAILED tests/codec_be_on_mr_nonselection/test_param_file.py::test_param_file_tests[stv-MASA 1dir 2TC at 160 kbps, 48kHz in, 48kHz out, HOA3 out, random FER at 5%] - Failed: Difference between ref and dut in output only (Non-BE - MAX_ABS_DIFF: 1)
FAILED tests/codec_be_on_mr_nonselection/test_param_file.py::test_param_file_tests[stv-MASA 1dir 2TC at 256 kbps, 48kHz in, 48kHz out, 5_1 out] - Failed: Difference between ref and dut in output only (Non-BE - MAX_ABS_DIFF: 1)
FAILED tests/codec_be_on_mr_nonselection/test_param_file.py::test_param_file_tests[stv-MASA 1dir 2TC at 32 kbps, 48kHz in, 48kHz out, BINAURAL ROOM IR out, HR, OT] - Failed: Difference between ref and dut in output only (Non-BE - MAX_ABS_DIFF: 1)
FAILED tests/codec_be_on_mr_nonselection/test_param_file.py::test_param_file_tests[stv-MASA 1dir 2TC at 32 kbps, 48kHz in, 48kHz out, BINAURAL ROOM IR out, HR] - Failed: Difference between ref and dut in output only (Non-BE - MAX_ABS_DIFF: 1)
FAILED tests/codec_be_on_mr_nonselection/test_param_file.py::test_param_file_tests[stv-MASA 1dir 2TC at 32.0 kbps, 48kHz in, 48kHz out, DTX on, 5_1 out] - Failed: Difference between ref and dut in output only (Non-BE - MAX_ABS_DIFF: 2)
FAILED tests/codec_be_on_mr_nonselection/test_param_file.py::test_param_file_tests[stv-MASA 1dir 2TC at 48 kbps, 48kHz in, 48kHz out, 7_1_4 out, random FER at 5%] - Failed: Difference between ref and dut in output only (Non-BE - MAX_ABS_DIFF: 3)
FAILED tests/codec_be_on_mr_nonselection/test_param_file.py::test_param_file_tests[stv-MASA 1dir 2TC at 64 kbps, 48kHz in, 48kHz out, EXT out] - Failed: Difference between ref and dut in output only (Non-BE - MAX_ABS_DIFF: 5)
FAILED tests/codec_be_on_mr_nonselection/test_param_file.py::test_param_file_tests[stv-MASA 1dir 2TC bitrate switching from 13.2 kbps to 512 kbps, 48kHz in, 16kHz out, BINAURAL out, JBM Prof 5] - Failed: Difference between ref and dut in output only (Non-BE - MAX_ABS_DIFF: 3)
FAILED tests/codec_be_on_mr_nonselection/test_param_file.py::test_param_file_tests[stv-MASA 1dir 2TC bitrate switching from 13.2 kbps to 512 kbps, 48kHz in, 32kHz out, MONO out, JBM Prof 5] - Failed: Difference between ref and dut in output only (Non-BE - MAX_ABS_DIFF: 4)
FAILED tests/codec_be_on_mr_nonselection/test_param_file.py::test_param_file_tests[stv-MASA 1dir 2TC bitrate switching from 13.2 kbps to 512 kbps, 48kHz in, 48kHz out, 7_1 out] - Failed: Difference between ref and dut in output only (Non-BE - MAX_ABS_DIFF: 2)
FAILED tests/codec_be_on_mr_nonselection/test_param_file.py::test_param_file_tests[stv-MASA 1dir 2TC bitrate switching from 13.2 kbps to 512 kbps, 48kHz in, 48kHz out, BINAURAL out (Model from file)] - Failed: Difference between ref and dut in output only (Non-BE - MAX_ABS_DIFF: 5)
FAILED tests/codec_be_on_mr_nonselection/test_param_file.py::test_param_file_tests[stv-MASA 1dir 2TC bitrate switching from 13.2 kbps to 512 kbps, 48kHz in, 48kHz out, DTX on, BINAURAL out, random FER at 5%] - Failed: Difference between ref and dut in output only (Non-BE - MAX_ABS_DIFF: 19)
FAILED tests/codec_be_on_mr_nonselection/test_param_file.py::test_param_file_tests[stv-MASA 1dir 2TC bitrate switching from 13.2 kbps to 512 kbps, 48kHz in, 48kHz out, MONO out] - Failed: Difference between ref and dut in output only (Non-BE - MAX_ABS_DIFF: 2)
FAILED tests/codec_be_on_mr_nonselection/test_param_file.py::test_param_file_tests[stv-MASA 2dir 1TC at 128 kbps, 48kHz in, 48kHz out, BINAURAL out, HR, exo] - Failed: Difference between ref and dut in output only (Non-BE - MAX_ABS_DIFF: 1)
FAILED tests/codec_be_on_mr_nonselection/test_param_file.py::test_param_file_tests[stv-MASA 2dir 1TC at 128 kbps, 48kHz in, 48kHz out, BINAURAL out, random FER at 5%] - Failed: Difference between ref and dut in output only (Non-BE - MAX_ABS_DIFF: 1)
FAILED tests/codec_be_on_mr_nonselection/test_param_file.py::test_param_file_tests[stv-MASA 2dir 1TC at 128 kbps, 48kHz in, 48kHz out, BINAURAL out] - Failed: Difference between ref and dut in output only (Non-BE - MAX_ABS_DIFF: 1)
FAILED tests/codec_be_on_mr_nonselection/test_param_file.py::test_param_file_tests[stv-MASA 2dir 2TC at 128 kbps, 48kHz in, 48kHz out, FOA out] - Failed: Difference between ref and dut in output only (Non-BE - MAX_ABS_DIFF: 1)
FAILED tests/codec_be_on_mr_nonselection/test_param_file.py::test_param_file_tests[stv-MASA 2dir 2TC at 192 kbps, 48kHz in, 48kHz out, 5_1_4 out, random FER at 5%] - Failed: Difference between ref and dut in output only (Non-BE - MAX_ABS_DIFF: 1)
FAILED tests/codec_be_on_mr_nonselection/test_param_file.py::test_param_file_tests[stv-MASA 2dir 2TC at 32 kbps, 48kHz in, 48kHz out, EXT out] - Failed: Difference between ref and dut in output only (Non-BE - MAX_ABS_DIFF: 5)
FAILED tests/codec_be_on_mr_nonselection/test_param_file.py::test_param_file_tests[stv-MASA 2dir 2TC at 48 kbps, 48kHz in, 48kHz out, 5_1 out, bandwidth switching] - Failed: Difference between ref and dut in output only (Non-BE - MAX_ABS_DIFF: 8)
FAILED tests/codec_be_on_mr_nonselection/test_param_file.py::test_param_file_tests[stv-MASA 2dir 2TC at 512 kbps, 48kHz in, 48kHz out, 5_1 out] - Failed: Difference between ref and dut in output only (Non-BE - MAX_ABS_DIFF: 1)
FAILED tests/codec_be_on_mr_nonselection/test_param_file.py::test_param_file_tests[stv-MASA 2dir 2TC at 64 kbps, 48kHz in, 48kHz out, BINAURAL out, HR, exo] - Failed: Difference between ref and dut in output only (Non-BE - MAX_ABS_DIFF: 3)
FAILED tests/codec_be_on_mr_nonselection/test_param_file.py::test_param_file_tests[stv-MASA 2dir 2TC at 64 kbps, 48kHz in, 48kHz out, BINAURAL out, HR] - Failed: Difference between ref and dut in output only (Non-BE - MAX_ABS_DIFF: 3)
FAILED tests/codec_be_on_mr_nonselection/test_param_file.py::test_param_file_tests[stv-MASA 2dir 2TC at 64 kbps, 48kHz in, 48kHz out, EXTERNAL out, random FER at 5%] - Failed: Difference between ref and dut in output only (Non-BE - MAX_ABS_DIFF: 4)
FAILED tests/codec_be_on_mr_nonselection/test_param_file.py::test_param_file_tests[stv-MASA 2dir 2TC bitrate switching from 13.2 kbps to 512 kbps, 48kHz in, 48kHz out, 7_1 out] - Failed: Difference between ref and dut in output only (Non-BE - MAX_ABS_DIFF: 1)

Build and tests were done with

cmake . -Bbuild-debug
cmake --build build-debug -- -j
cmake . -Bbuild-release -DCMAKE_BUILD_TYPE=Release
cmake --build build-release -- -j
pytest tests/codec_be_on_mr_nonselection -v --update_ref 1 --ref_encoder_path=./build-debug/IVAS_cod --ref_decoder_path=./build-debug/IVAS_dec
pytest tests/codec_be_on_mr_nonselection -v --update_ref 0 --dut_encoder_path=./build-release/IVAS_cod --dut_decoder_path=./build-release/IVAS_dec

Admin message

Apple Clang 16 Release vs Debug differences in MASA