Stereo DMX: Usage of BASOP_Util_Divide3232_Scale_newton() to calculate hPHA->fad_g_prc_fx

Basic info

• Float reference:
  • Encoder (float): b63c033f
  • Decoder (float): b63c033f
• Fixed point:
  • Encoder (fixed): 8d9f712d
  • Decoder (fixed): n/a

Bug description

This is a follow-up on !1376 (merged): When trying to replace BASOP_Util_Divide3232_Scale_cadence() by BASOP_Util_Divide3232_Scale_newton() in ivas_stereo_dmx_evs_fx.c, line 2540, we observe relatively big outliers in the STV stvST32c.wav at 13.2 kbit/s:

BASOP, using BASOP_Util_Divide3232_Scale_newton():

BASOP, using BASOP_Util_Divide3232_Scale_cadence():

Float:

The following change was applied to the code:

diff --git a/lib_enc/ivas_stereo_dmx_evs_fx.c b/lib_enc/ivas_stereo_dmx_evs_fx.c
index 625ff8ee..2e76c973 100644
--- a/lib_enc/ivas_stereo_dmx_evs_fx.c
+++ b/lib_enc/ivas_stereo_dmx_evs_fx.c
@@ -2537,7 +2537,13 @@ ivas_error stereo_dmx_evs_init_encoder_fx(
     move16();
     fad_g = hStereoDmxEVS->hPHA->fad_g_prc_fx;
     // fad_r = 1.0f / (float) ( fad_len + 1 );
+#if 1
+    fad_r = BASOP_Util_Divide3232_Scale_newton( 1, add( fad_len, 1 ), &tmp_e );
+    fprintf(stderr, "\nfad_r: %d\n", fad_r);
+#else
     fad_r = BASOP_Util_Divide3232_Scale_cadence( 1, add( fad_len, 1 ), &tmp_e );
+    fprintf(stderr, "\nfad_r: %d\n", fad_r);
+#endif
     fad_r = L_shl_r( fad_r, tmp_e );
     fad_len2 = shr( fad_len, 1 );
     FOR( ( n = 0, m = ( fad_len - 1 ) ); n < fad_len2; ( n++, m-- ) )

The output of the division itself only differs by 1 bit:

BASOP_Util_Divide3232_Scale_newton(): fad_r = 857653376

BASOP_Util_Divide3232_Scale_cadence(): fad_r: 857653375

With the following code, an potential error is amplified:

fad_r = L_shl_r( fad_r, tmp_e );
fad_len2 = shr( fad_len, 1 );
FOR( ( n = 0, m = ( fad_len - 1 ) ); n < fad_len2; ( n++, m-- ) )
{
    fad_g[n] = imult3216( fad_r, add( n, 1 ) );
    move32();
    fad_g[m] = L_sub( MAX_32, fad_g[n] );
    move32();
}

I'm not exactly sure about the actual reason for this big outlier, but it's a bit concerning that this one bit of difference in precision causes such a big outlier.

Ways to reproduce

IVAS_cod -stereo_dmx_evs 13200 32 ivas-codec/scripts/testv/stvST32c.wav out.192
IVAS_dec 32 out.192 out.wav