Apple Clang 16 Release vs Debug differences in Stereo

Commit SHA: 32c65a1d (probably)

This issue shows differences in Release vs. Debug build run of short self-test on Mac M4, OS version 15.6.1, Apple Clang 16. This was discussed in IVAS conformance call on Monday September 22nd.

This is the issue for Stereo format. Some larger differences with maximum being 2425 which does require taking a look.

Test showed these results

FAILED tests/codec_be_on_mr_nonselection/test_param_file.py::test_param_file_tests[stv-stereo at 13.2 kbps, 16kHz in, 16kHz out] - Failed: Difference between ref and dut in output only (Non-BE - MAX_ABS_DIFF: 2)
FAILED tests/codec_be_on_mr_nonselection/test_param_file.py::test_param_file_tests[stv-stereo at 13.2 kbps, 32kHz in, 16kHz out, DTX on, random FER at 5%] - Failed: Difference between ref and dut in output only (Non-BE - MAX_ABS_DIFF: 1)
FAILED tests/codec_be_on_mr_nonselection/test_param_file.py::test_param_file_tests[stv-stereo at 13.2 kbps, 32kHz in, 32kHz out, DTX on, random FER at 5%, bandwidth switching] - Failed: Difference between ref and dut in output only (Non-BE - MAX_ABS_DIFF: 5)
FAILED tests/codec_be_on_mr_nonselection/test_param_file.py::test_param_file_tests[stv-stereo at 13.2 kbps, 32kHz in, 32kHz out, DTX on] - Failed: Difference between ref and dut in output only (Non-BE - MAX_ABS_DIFF: 4)
FAILED tests/codec_be_on_mr_nonselection/test_param_file.py::test_param_file_tests[stv-stereo at 13.2 kbps, 32kHz in, 32kHz out] - Failed: Difference between ref and dut in output only (Non-BE - MAX_ABS_DIFF: 5)
FAILED tests/codec_be_on_mr_nonselection/test_param_file.py::test_param_file_tests[stv-stereo at 13.2 kbps, 48kHz in, 48kHz out ] - Failed: Difference between ref and dut in output only (Non-BE - MAX_ABS_DIFF: 11)
FAILED tests/codec_be_on_mr_nonselection/test_param_file.py::test_param_file_tests[stv-stereo at 16.4 kbps, 16kHz in, 16kHz out] - Failed: Difference between ref and dut in output only (Non-BE - MAX_ABS_DIFF: 3)
FAILED tests/codec_be_on_mr_nonselection/test_param_file.py::test_param_file_tests[stv-stereo at 16.4 kbps, 32kHz in, 16kHz out, DTX on] - Failed: Difference between ref and dut in output only (Non-BE - MAX_ABS_DIFF: 2)
FAILED tests/codec_be_on_mr_nonselection/test_param_file.py::test_param_file_tests[stv-stereo at 16.4 kbps, 32kHz in, 16kHz out, random FER at 5%, DTX on] - Failed: Difference between ref and dut in output only (Non-BE - MAX_ABS_DIFF: 2)
FAILED tests/codec_be_on_mr_nonselection/test_param_file.py::test_param_file_tests[stv-stereo at 16.4 kbps, 32kHz in, 16kHz out, random FER at 5%] - Failed: Difference between ref and dut in output only (Non-BE - MAX_ABS_DIFF: 3)
FAILED tests/codec_be_on_mr_nonselection/test_param_file.py::test_param_file_tests[stv-stereo at 16.4 kbps, 32kHz in, 32kHz out, random FER at 5%] - Failed: Difference between ref and dut in output only (Non-BE - MAX_ABS_DIFF: 5)
FAILED tests/codec_be_on_mr_nonselection/test_param_file.py::test_param_file_tests[stv-stereo at 16.4 kbps, 32kHz in, 32kHz out] - Failed: Difference between ref and dut in output only (Non-BE - MAX_ABS_DIFF: 4)
FAILED tests/codec_be_on_mr_nonselection/test_param_file.py::test_param_file_tests[stv-stereo at 24.4 kbps, 16kHz in, 16kHz out, DTX on, MONO out] - Failed: Difference between ref and dut in output only (Non-BE - MAX_ABS_DIFF: 1763)
FAILED tests/codec_be_on_mr_nonselection/test_param_file.py::test_param_file_tests[stv-stereo at 24.4 kbps, 16kHz in, 16kHz out] - Failed: Difference between ref and dut in output only (Non-BE - MAX_ABS_DIFF: 2)
FAILED tests/codec_be_on_mr_nonselection/test_param_file.py::test_param_file_tests[stv-stereo at 24.4 kbps, 16kHz in, 32kHz out, DTX on] - Failed: Difference between ref and dut in output only (Non-BE - MAX_ABS_DIFF: 2425)
FAILED tests/codec_be_on_mr_nonselection/test_param_file.py::test_param_file_tests[stv-stereo at 24.4 kbps, 32kHz in, 16kHz out, DTX on, MONO out, random FER at 5%] - Failed: Difference between ref and dut in output only (Non-BE - MAX_ABS_DIFF: 613)
FAILED tests/codec_be_on_mr_nonselection/test_param_file.py::test_param_file_tests[stv-stereo at 24.4 kbps, 32kHz in, 32kHz out, DTX on, MONO out, random FER at 5%] - Failed: Difference between ref and dut in output only (Non-BE - MAX_ABS_DIFF: 633)
FAILED tests/codec_be_on_mr_nonselection/test_param_file.py::test_param_file_tests[stv-stereo at 24.4 kbps, 32kHz in, 32kHz out, DTX on, MONO out] - Failed: Difference between ref and dut in output only (Non-BE - MAX_ABS_DIFF: 635)
FAILED tests/codec_be_on_mr_nonselection/test_param_file.py::test_param_file_tests[stv-stereo at 24.4 kbps, 32kHz in, 32kHz out, DTX on] - Failed: Difference between ref and dut in output only (Non-BE - MAX_ABS_DIFF: 908)
FAILED tests/codec_be_on_mr_nonselection/test_param_file.py::test_param_file_tests[stv-stereo at 24.4 kbps, 32kHz in, 32kHz out] - Failed: Difference between ref and dut in output only (Non-BE - MAX_ABS_DIFF: 6)
FAILED tests/codec_be_on_mr_nonselection/test_param_file.py::test_param_file_tests[stv-stereo at 24.4 kbps, 32kHz in, 48kHz out, DTX on, random FER at 5%] - Failed: Difference between ref and dut in output only (Non-BE - MAX_ABS_DIFF: 909)
FAILED tests/codec_be_on_mr_nonselection/test_param_file.py::test_param_file_tests[stv-stereo at 24.4 kbps, 48kHz in, 48kHz out, DTX on] - Failed: Difference between ref and dut in output only (Non-BE - MAX_ABS_DIFF: 9)
FAILED tests/codec_be_on_mr_nonselection/test_param_file.py::test_param_file_tests[stv-stereo at 32 kbps, 16kHz in, 16kHz out, random FER at 5%, DTX on] - Failed: Difference between ref and dut in output only (Non-BE - MAX_ABS_DIFF: 2)
FAILED tests/codec_be_on_mr_nonselection/test_param_file.py::test_param_file_tests[stv-stereo at 32 kbps, 16kHz in, 16kHz out] - Failed: Difference between ref and dut in output only (Non-BE - MAX_ABS_DIFF: 1)
FAILED tests/codec_be_on_mr_nonselection/test_param_file.py::test_param_file_tests[stv-stereo at 32 kbps, 16kHz in, 48kHz out, MONO out, random FER at 5%] - Failed: Difference between ref and dut in output only (Non-BE - MAX_ABS_DIFF: 7)
FAILED tests/codec_be_on_mr_nonselection/test_param_file.py::test_param_file_tests[stv-stereo at 32 kbps, 32kHz in, 16kHz out, DTX on, MONO out, random FER at 5%] - Failed: Difference between ref and dut in output only (Non-BE - MAX_ABS_DIFF: 1)
FAILED tests/codec_be_on_mr_nonselection/test_param_file.py::test_param_file_tests[stv-stereo at 32 kbps, 32kHz in, 32kHz out, DTX on, MONO out, random FER at 5%] - Failed: Difference between ref and dut in output only (Non-BE - MAX_ABS_DIFF: 2)
FAILED tests/codec_be_on_mr_nonselection/test_param_file.py::test_param_file_tests[stv-stereo at 32 kbps, 32kHz in, 32kHz out, MONO out, random FER at 5%] - Failed: Difference between ref and dut in output only (Non-BE - MAX_ABS_DIFF: 4)
FAILED tests/codec_be_on_mr_nonselection/test_param_file.py::test_param_file_tests[stv-stereo at 32 kbps, 32kHz in, 32kHz out, NOOP] - Failed: Difference between ref and dut in output only (Non-BE - MAX_ABS_DIFF: 449)
FAILED tests/codec_be_on_mr_nonselection/test_param_file.py::test_param_file_tests[stv-stereo at 32 kbps, 32kHz in, 32kHz out, random FER at 5%] - Failed: Difference between ref and dut in output only (Non-BE - MAX_ABS_DIFF: 6)
FAILED tests/codec_be_on_mr_nonselection/test_param_file.py::test_param_file_tests[stv-stereo at 32 kbps, 32kHz in, 32kHz out] - Failed: Difference between ref and dut in output only (Non-BE - MAX_ABS_DIFF: 5)
FAILED tests/codec_be_on_mr_nonselection/test_param_file.py::test_param_file_tests[stv-stereo at 32 kbps, 32kHz in, 48kHz out, MONO out, random FER at 5%] - Failed: Difference between ref and dut in output only (Non-BE - MAX_ABS_DIFF: 21)
FAILED tests/codec_be_on_mr_nonselection/test_param_file.py::test_param_file_tests[stv-stereo at 32 kbps, 32kHz in, 48kHz out, random FER at 5%] - Failed: Difference between ref and dut in output only (Non-BE - MAX_ABS_DIFF: 9)
FAILED tests/codec_be_on_mr_nonselection/test_param_file.py::test_param_file_tests[stv-stereo at 32 kbps, 48 kHz in, 32 kHz out, DTX on, JBM Prof 0] - Failed: Difference between ref and dut in output only (Non-BE - MAX_ABS_DIFF: 48)
FAILED tests/codec_be_on_mr_nonselection/test_param_file.py::test_param_file_tests[stv-stereo at 32 kbps, 48kHz in, 16kHz out, DTX on, MONO out, random FER at 5%] - Failed: Difference between ref and dut in output only (Non-BE - MAX_ABS_DIFF: 33)
FAILED tests/codec_be_on_mr_nonselection/test_param_file.py::test_param_file_tests[stv-stereo at 32 kbps, 48kHz in, 32kHz out, random FER at 5%] - Failed: Difference between ref and dut in output only (Non-BE - MAX_ABS_DIFF: 5)
FAILED tests/codec_be_on_mr_nonselection/test_param_file.py::test_param_file_tests[stv-stereo at 32 kbps, 48kHz in, 48kHz out, DTX on, MONO out, random FER at 5%] - Failed: Difference between ref and dut in output only (Non-BE - MAX_ABS_DIFF: 35)
FAILED tests/codec_be_on_mr_nonselection/test_param_file.py::test_param_file_tests[stv-stereo at 32 kbps, 48kHz in, 48kHz out, DTX on, random FER at 5%, bandwidth switching] - Failed: Difference between ref and dut in output only (Non-BE - MAX_ABS_DIFF: 8)
FAILED tests/codec_be_on_mr_nonselection/test_param_file.py::test_param_file_tests[stv-stereo at 32 kbps, 48kHz in, 48kHz out, bandwidth switching] - Failed: Difference between ref and dut in output only (Non-BE - MAX_ABS_DIFF: 9)
FAILED tests/codec_be_on_mr_nonselection/test_param_file.py::test_param_file_tests[stv-stereo at 32 kbps, 48kHz in, 48kHz out, random FER at 18% with long bursts] - Failed: Difference between ref and dut in output only (Non-BE - MAX_ABS_DIFF: 10)
FAILED tests/codec_be_on_mr_nonselection/test_param_file.py::test_param_file_tests[stv-stereo at 48 kbps, 16 kHz in, 16 kHz out, DTX on, JBM Prof 5] - Failed: Difference between ref and dut in output only (Non-BE - MAX_ABS_DIFF: 2)
FAILED tests/codec_be_on_mr_nonselection/test_param_file.py::test_param_file_tests[stv-stereo at 48 kbps, 16 kHz in, 16 kHz out, DTX on] - Failed: Difference between ref and dut in output only (Non-BE - MAX_ABS_DIFF: 2)
FAILED tests/codec_be_on_mr_nonselection/test_param_file.py::test_param_file_tests[stv-stereo at 48 kbps, 16 kHz in, 16 kHz, DTX on, MONO out, random FER at 5%] - Failed: Difference between ref and dut in output only (Non-BE - MAX_ABS_DIFF: 1)
FAILED tests/codec_be_on_mr_nonselection/test_param_file.py::test_param_file_tests[stv-stereo at 48 kbps, 16kHz in, 16kHz out] - Failed: Difference between ref and dut in output only (Non-BE - MAX_ABS_DIFF: 2)
FAILED tests/codec_be_on_mr_nonselection/test_param_file.py::test_param_file_tests[stv-stereo at 48 kbps, 32 kHz in, 32 kHz out, DTX on, MONO out] - Failed: Difference between ref and dut in output only (Non-BE - MAX_ABS_DIFF: 3)
FAILED tests/codec_be_on_mr_nonselection/test_param_file.py::test_param_file_tests[stv-stereo at 48 kbps, 32 kHz in, 32 kHz out, DTX on] - Failed: Difference between ref and dut in output only (Non-BE - MAX_ABS_DIFF: 5)
FAILED tests/codec_be_on_mr_nonselection/test_param_file.py::test_param_file_tests[stv-stereo at 48 kbps, 32kHz in, 32kHz out, MONO out] - Failed: Difference between ref and dut in output only (Non-BE - MAX_ABS_DIFF: 6)
FAILED tests/codec_be_on_mr_nonselection/test_param_file.py::test_param_file_tests[stv-stereo at 48 kbps, 32kHz in, 32kHz out, random FER at 5%] - Failed: Difference between ref and dut in output only (Non-BE - MAX_ABS_DIFF: 8)
FAILED tests/codec_be_on_mr_nonselection/test_param_file.py::test_param_file_tests[stv-stereo at 48 kbps, 48 kHz in, 48 kHz out, DTX on, FER with burst error before SID] - Failed: Difference between ref and dut in output only (Non-BE - MAX_ABS_DIFF: 8)
FAILED tests/codec_be_on_mr_nonselection/test_param_file.py::test_param_file_tests[stv-stereo at 48 kbps, 48 kHz in, 48 kHz out, DTX on, MONO out, random FER at 5%] - Failed: Difference between ref and dut in output only (Non-BE - MAX_ABS_DIFF: 5)
FAILED tests/codec_be_on_mr_nonselection/test_param_file.py::test_param_file_tests[stv-stereo at 48 kbps, 48 kHz in, 48 kHz out, DTX on, bandwidth switching] - Failed: Difference between ref and dut in output only (Non-BE - MAX_ABS_DIFF: 6)
FAILED tests/codec_be_on_mr_nonselection/test_param_file.py::test_param_file_tests[stv-stereo at 64 kbps, 16kHz in, 16kHz out, random FER at 10%] - Failed: Difference between ref and dut in output only (Non-BE - MAX_ABS_DIFF: 3)
FAILED tests/codec_be_on_mr_nonselection/test_param_file.py::test_param_file_tests[stv-stereo at 64 kbps, 32kHz in, 32kHz out, bandwidth switching] - Failed: Difference between ref and dut in output only (Non-BE - MAX_ABS_DIFF: 6)
FAILED tests/codec_be_on_mr_nonselection/test_param_file.py::test_param_file_tests[stv-stereo bitrate switching from 13.2 kbps to 128 kbps, 32kHz in, 32kHz out] - Failed: Difference between ref and dut in output only (Non-BE - MAX_ABS_DIFF: 4)
FAILED tests/codec_be_on_mr_nonselection/test_param_file.py::test_param_file_tests[stv-stereo bitrate switching from 13.2 kbps to 128 kbps, 48kHz in, 48kHz out, DTX on, EXT out] - Failed: Difference between ref and dut in output only (Non-BE - MAX_ABS_DIFF: 7)

Build and tests were done with

cmake . -Bbuild-debug
cmake --build build-debug -- -j
cmake . -Bbuild-release -DCMAKE_BUILD_TYPE=Release
cmake --build build-release -- -j
pytest tests/codec_be_on_mr_nonselection -v --update_ref 1 --ref_encoder_path=./build-debug/IVAS_cod --ref_decoder_path=./build-debug/IVAS_dec
pytest tests/codec_be_on_mr_nonselection -v --update_ref 0 --dut_encoder_path=./build-release/IVAS_cod --dut_decoder_path=./build-release/IVAS_dec

Admin message

Apple Clang 16 Release vs Debug differences in Stereo