Skip to content

Large differences in OMASA output between Debug and Release builds

Commit:

Noticed during IVAS conformance analysis, there are surprisingly large differences in output of OMASA when comparing codec built with Debug and Release options. See test results for OMASA below

test_param_file_tests[stv-OMASA 1Dir1TC 2ISM at 16.4 kbps, 16kHz in, 48kHz out, 5.1 out] - Failed: Difference between ref and dut in output only (Non-BE - MAX_ABS_DIFF: 4024)
test_param_file_tests[stv-OMASA 1Dir1TC 4ISM 48 kbps 48kHz in, 48kHz out, BINAURAL_ROOM_REVERB out custom configuration] - Failed: Difference between ref and dut in output only (Non-BE - MAX_ABS_DIFF: 4628)
test_param_file_tests[stv-OMASA 1Dir1TC 4ISM at 48 kbps, 48kHz in, 48kHz out, BINAURAL_ROOM_REVERB out] - Failed: Difference between ref and dut in output only (Non-BE - MAX_ABS_DIFF: 4109)
test_param_file_tests[stv-OMASA 1Dir1TC 4ISM at br sw techs 13.2 to 512 kbps start 32 kbps, 48kHz in, 48kHz out, BINAURAL out, JBM Prof 5] - Failed: Difference between ref and dut in output only (Non-BE - MAX_ABS_DIFF: 730)
test_param_file_tests[stv-OMASA 1Dir2TC 3ISM at 32 kbps, 48kHz in, 16kHz out, STEREO out] - Failed: Difference between ref and dut in output only (Non-BE - MAX_ABS_DIFF: 6276)
test_param_file_tests[stv-OMASA 1Dir2TC 3ISM at br sw techs 13.2 to 512 kbps start 24.4 kbps, 32kHz in, 48kHz out, 5.1.2 out, JBM Prof 5] - Failed: Difference between ref and dut in output only (Non-BE - MAX_ABS_DIFF: 4428)
test_param_file_tests[stv-OMASA 1Dir2TC 3ISM at br sw techs 13.2 to 512 kbps start 24.4 kbps, 32kHz in, 48kHz out, 5.1.4 out] - Failed: Difference between ref and dut in output only (Non-BE - MAX_ABS_DIFF: 3661)
test_param_file_tests[stv-OMASA 1Dir2TC 4ISM at 32 kbps, 48kHz in, 48kHz out, FOA out, FER at 5%] - Failed: Difference between ref and dut in output only (Non-BE - MAX_ABS_DIFF: 4671)
test_param_file_tests[stv-OMASA 1Dir2TC 4ISM at br sw techs 13.2 to 512 kbps start 80 kbps, 48kHz in, 48kHz out, FOA out, JBM Prof 5] - Failed: Difference between ref and dut in output only (Non-BE - MAX_ABS_DIFF: 1864)
test_param_file_tests[stv-OMASA 1Dir2TC 4ISM at br sw techs 13.2 to 512 kbps start 80 kbps, 48kHz in, 48kHz out, HOA3 out] - Failed: Difference between ref and dut in output only (Non-BE - MAX_ABS_DIFF: 1715)
test_param_file_tests[stv-OMASA 2Dir1TC 3ISM at 24.4 kbps, 48kHz in, 16kHz out, FOA out, FER at 10%] - Failed: Difference between ref and dut in output only (Non-BE - MAX_ABS_DIFF: 214)
test_param_file_tests[stv-OMASA 2Dir1TC 3ISM at br sw techs 13.2 to 512 kbps start 48 kbps, 48kHz in, 32kHz out, STEREO out, FER at 10%] - Failed: Difference between ref and dut in output only (Non-BE - MAX_ABS_DIFF: 1271)
test_param_file_tests[stv-OMASA 2Dir1TC 3ISM at br sw techs 13.2 to 512 kbps start 48 kbps, 48kHz in, 32kHz out, STEREO out, JBM Prof 5] - Failed: Difference between ref and dut in output only (Non-BE - MAX_ABS_DIFF: 1167)
test_param_file_tests[stv-OMASA 2Dir1TC 4ISM at 24.4 kbps, 48kHz in, 48kHz out, STEREO out] - Failed: Difference between ref and dut in output only (Non-BE - MAX_ABS_DIFF: 3417)
test_param_file_tests[stv-OMASA 2Dir2TC 1ISM at br sw techs 13.2 to 512 kbps start 24.4 kbps, 48kHz in, 48kHz out, BINAURAL_ROOM_IR out, JBM Prof 5] - Failed: Difference between ref and dut in output only (Non-BE - MAX_ABS_DIFF: 3284)
test_param_file_tests[stv-OMASA 2Dir2TC 2ISM at 32 kbps, 48kHz in, 48kHz out, STEREO out, FER at 5%] - Failed: Difference between ref and dut in output only (Non-BE - MAX_ABS_DIFF: 3)
test_param_file_tests[stv-OMASA 2Dir2TC 2ISM at br sw techs 13.2 to 512 kbps start 48 kbps, 48kHz in, 48kHz out, 7.1 out, JBM Prof 5] - Failed: Difference between ref and dut in output only (Non-BE - MAX_ABS_DIFF: 3359)
test_param_file_tests[stv-OMASA 2Dir2TC 2ISM br sw techs 13.2 to 512 kbps start 48 kbps, 48kHz in, 48kHz out, BINAURAL out, object editing] - Failed: Difference between ref and dut in output only (Non-BE - MAX_ABS_DIFF: 4396)
test_param_file_tests[stv-OMASA 2Dir2TC 3ISM at 48 kbps, 48kHz in, 48kHz out, MONO out] - Failed: Difference between ref and dut in output only (Non-BE - MAX_ABS_DIFF: 2112)
test_param_file_tests[stv-OMASA 2Dir2TC 3ISM at br sw techs 13.2 to 512 kbps start 160 kbps, 48kHz in, 48kHz out, MONO out, JBM Prof 5] - Failed: Difference between ref and dut in output only (Non-BE - MAX_ABS_DIFF: 2108)
test_param_file_tests[stv-OMASA 2Dir2TC 3ISM at br sw techs 13.2 to 512 kbps start 160 kbps, 48kHz in, 48kHz out, MONO out] - Failed: Difference between ref and dut in output only (Non-BE - MAX_ABS_DIFF: 2106)
test_param_file_tests[stv-OMASA 2Dir2TC 4ISM at 13.2 kbps, 48kHz in, 48kHz out, MONO out] - Failed: Difference between ref and dut in output only (Non-BE - MAX_ABS_DIFF: 6228)
test_param_file_tests[stv-OMASA 2Dir2TC 4ISM at 192 kbps, 48kHz in, 48kHz out, STEREO out] - Failed: Difference between ref and dut in output only (Non-BE - MAX_ABS_DIFF: 1)
test_param_file_tests[stv-OMASA 2Dir2TC 4ISM at 80 kbps, 48kHz in, 48kHz out, BINAURAL out, default object editing, 1SEP-PARAM] - Failed: Difference between ref and dut in output only (Non-BE - MAX_ABS_DIFF: 2)
test_param_file_tests[stv-OMASA 2Dir2TC 4ISM at br sw techs 13.2 to 512 kbps start 80 kbps, 48kHz in, 48kHz out, EXT out] - Failed: Difference between ref and dut in output (Non-BE - MAX_ABS_DIFF: 1806) and metadata

Otherwise, the max abs diff for other formats in mostly under 6. The cause of difference should be found and ideally removed if possible. This might have beneficial effects to BASOP as well.

The results above were on Linux and to replicate, you can build binaries, e.g., like this

cmake . -Bbuild-debug
cmake --build build-debug -- -j

cmake . -Bbuild-release -DCMAKE_BUILD_TYPE=Release
cmake --build build-release -- -j

And then you have the binaries in the folders. Running the pytest with this setup should work with

pytest tests/codec_be_on_mr_nonselection -v --update_ref 1 --ref_encoder_path=./build-debug/IVAS_cod --ref_decoder_path=./build-debug/IVAS_dec
pytest tests/codec_be_on_mr_nonselection -v --update_ref 0 --dut_encoder_path=./build-release/IVAS_cod --dut_decoder_path=./build-release/IVAS_dec