Draft: Resolve "Assertion errors triggered when float code is compiled with "-funsafe-math-optimizations" option"

multrusrequested to merge
float-1544-assertion-errors-triggered-when-float-code-is-compiled-with-funsafe-math-optimizations-part01 into main

Reason why this change is needed

  • With optimization and -funsafe-math-optimizations, the encoder runs into an assert:

    lib_enc/ivas_stereo_dft_td_itd.c:295: void stereo_td_itd(ITD_DATA *, float (*)[420], const int16_t, const int16_t, Encoder_State **, const int16_t, float **): Assertion `( fabsf( hITD->itd[k_offset] ) <= itd_max ) && "ITD value is too high!"' failed.

  • Reason for this is an imprecision in stereo_dft_quantize_itd()

Description of the change

  • Replace single precision arithmetic by double precision arithmetic in stereo_dft_quantize_itd() when converting the ITDs back to encoder input samplig rate

Affected operating points

  • For "normal" optimization, the encoder is expected to produce bit-exact output.

Related to #1544

Edited by multrus

Merge request reports