Draft: Resolve "Encoder MLD sensitivity due to precision differences in sinf/cosf math kernels"

Vladimir Malenovskyrequested to merge
float-1603-encoder-mld-sensitivity-due-to-precision-differences-in-sinf-cosf-math-kernels into main

Reason why this change is needed

Two functions in the IVAS encoder rely on sinf()/cosf() for computations that feed directly into discrete or threshold-sensitive decisions. On platforms where sinf/cosf are evaluated with <=2 ULP error, the resulting bitstream diverges from the reference, causing high MLD scores on the SBA 24.4 kbps / 32 kHz input test case.

IVAS_cod -q -sba 3 24400 32 testvec/testv/stv3OA32c_cut_16.0.wav stv3OA32c_cut_16.0.wav_SBA_at_24_4_kbps_32kHz_in_32kHz_out_BINAURAL_out_HR_OT.192
IVAS_dec -q -fr 20 -t testvec/testv/headrot.csv -otr avg BINAURAL 32 stv3OA32c_cut_16.0.wav_SBA_at_24_4_kbps_32kHz_in_32kHz_out_BINAURAL_out_HR_OT.192 stv3OA32c_cut_16.0.wav_SBA_at_24_4_kbps_32kHz_in_32kHz_out_BINAURAL_out_HR_OT.dec.wav

scripts/tools/Linux/wav-diff stv3OA32c_cut_16.0.wav_SBA_at_24_4_kbps_32kHz_in_32kHz_out_BINAURAL_out_HR_OT.dec.wav.48k.ref.wav  stv3OA32c_cut_16.0.wav_SBA_at_24_4_kbps_32kHz_in_32kHz_out_BINAURAL_out_HR_OT.dec.wav.48k.wav

Affected Files/Functions :

File Function Call type
lib_com/ivas_fb_mixer.c ivas_get_hanning_win() Init-time
lib_com/ivas_fb_mixer.c ivas_get_ld_fb_resp() Init-time
lib_com/ivas_agc_com.c ivas_agc_initWindowFunc() Init-time

ISSUE 1 — lib_com/ivas_fb_mixer.c :

The 12-band filterbank uses hardcoded ROM tables (ivas_fb_fr_12band_1ms_re[], ivas_fb_fr_12band_1ms_im[]) for 48 kHz input — always bit-exact. For 32 kHz, the coefficients are calculated on the fly in ivas_get_ld_fb_resp(), which calls ivas_get_hanning_win(). This results in platform-dependent behavior.

ISSUE 2 — lib_com/ivas_agc_com.c :

The AGC is active for nchan_transport == 1 (SBA FOA paths). At encoder open, ivas_agc_initWindowFunc() fills winFunc[] — a gain attenuation base function. Per frame, the encoder computes a discrete gain exponent using ceilf().

a = 0.5f * (1.f - powf(10.f, -6.f/20.f)); // a ≈ 0.2481  (no sinf/cosf here)
for (i = 0; i < length; i++) 
{
  winFunc[i] = 1.f + a *   (cosf(EVS_PI*i/N) - 1.f);  // range: 1.0 → ~0.504
}  
/* per-frame — called EVERY frame */

gainExpVal = ceilf(  -logf(maxAbs * NORM) /  logf(winFunc[idx]));  // ← integer!
gain = powf(winFunc[idx], gainExpVal);
pcm_out[idx] *= gain;  // applied to PCM

Observed Impact on MLD: 10.1 (before fix) -> 0.0 (after fix)

Description of the change

Affected operating points

Closes #1603

Merge request reports