Resolve "Assertion errors triggered when float code is compiled with "-funsafe-math-optimizations" option"

multrusrequested to merge
float-1544-assertion-errors-triggered-when-float-code-is-compiled-with-funsafe-math-optimizations-part01 into main

Reason why this change is needed

  • With optimization and -funsafe-math-optimizations, the encoder runs into an assert:

    lib_enc/ivas_stereo_dft_td_itd.c:295: void stereo_td_itd(ITD_DATA *, float (*)[420], const int16_t, const int16_t, Encoder_State **, const int16_t, float **): Assertion `( fabsf( hITD->itd[k_offset] ) <= itd_max ) && "ITD value is too high!"' failed.

  • Reason for this is an imprecision in stereo_dft_quantize_itd()

  • But: In the code we are truncating to integer, but the asserts actually checks the floating-point value --> change the assert

Description of the change

  • Adapt the assert to only check on the integer value.

Affected operating points

  • Everything is BE

Related to #1544 (closed)

Edited by multrus

Merge request reports