basop-2471-correction-of-some-overflow-operations [non-BE][split-non-BE]

vaillancourrequested to merge
basop-2471-correction-of-some-overflow-operations into main

Reason why this change is needed

  • Two distinct undesired overflows were identified, both impacting bit‑exactness.

    1. Normalized correlation initialization
      The first overflow occurs during the initialization of the normalized correlation. Correcting it may slightly affect ACELP operating points. Only minimal differences were observed in the short version of the self_test.

    2. TCX operating point
      The second overflow affects the TCX operating point. A few differences were observed, with the corrected spectrum being closer to the floating‑point reference. No subjective differences have been found so far.

      @multrus . if someone at FhG wants to have a look given TCX could be affected

Description of the change

  • preventing the overflow by using an appropriate data size or by saturating the value

Affected operating points

  • Not bit-exact but bitstream compatible and should nor have a bit impact on quality

Closes #2471 (closed)

Edited by vaillancour

Merge request reports