[Non BE] Resolve issue 1246: high level stereo DTX can cause inf in coherence
requested to merge 1246-encoder-can-create-a-wrong-bitstream-for-all-stereo-bitrate-48-kbps-with-dtx-on into main
- Related issues: #1246 (closed)
- Requested reviewers: @vaillancour
Reason why this change is needed
- A high level input signal triggering DTX can lead to numerical overflow in the coherence calculation, where a weighed mean is derived using the DFT power spectrum. If the level is high, the spectral values are large which results in values exceeding FLT_MAX.
Description of the change
- A parenthesis is inserted to compute the division before the multiplication with the weighting factor.
- Mathematically equivalent, but will likely introduce differences due to numerical precision.
Affected operating points
- Non-BE may occur for all operating points using CPE below 48 kbps, triggering DTX. Impact is expected to be negligible for nominal level input.
Related to #1246 (closed)