Skip to content

ASAN: write to nullptr in stereo encoder at 64kbps possible

Using signal 35.wav from the SQAM set of testsignals, running

make clean
make -j CLANG=2
./IVAS_cod -stereo 64000 48 /home/ame-tmp2/knj/SQAM_stereo/35.wav bit

at commit 3350c00b results in

AddressSanitizer:DEADLYSIGNAL
=================================================================
==913880==ERROR: AddressSanitizer: SEGV on unknown address 0x000000000000 (pc 0x000000d7de2e bp 0x7ffda7e5ab20 sp 0x7ffda7e5aaf0 T0)
==913880==The signal is caused by a WRITE memory access.
==913880==Hint: address points to the zero page.
    #0 0xd7de2e in shl_o /local/knj/ivas-basop/lib_com/basop32.c:787:23
    #1 0xd7e16c in shr_o /local/knj/ivas-basop/lib_com/basop32.c:912:19
    #2 0xd7de99 in shr /local/knj/ivas-basop/lib_com/basop32.c:952:12
    #3 0xabf449 in IGF_CalculateStereoEnvelope_fx /local/knj/ivas-basop/lib_enc/igf_enc.c:1262:104
    #4 0xab90c6 in IGFEncApplyStereo_fx /local/knj/ivas-basop/lib_enc/igf_enc.c:3216:9
    #5 0x8998f5 in ProcessStereoIGF_fx /local/knj/ivas-basop/lib_enc/tcx_utils_enc.c:1692:5
    #6 0xbf40f9 in stereo_mdct_core_enc /local/knj/ivas-basop/lib_enc/ivas_stereo_mdct_core_enc.c:448:21
    #7 0xaf9385 in ivas_core_enc /local/knj/ivas-basop/lib_enc/ivas_core_enc.c:302:17
    #8 0xb2acde in ivas_cpe_enc /local/knj/ivas-basop/lib_enc/ivas_cpe_enc.c:815:20
    #9 0x58fed8 in ivas_enc /local/knj/ivas-basop/lib_enc/ivas_enc.c:772:24
    #10 0x4ec12f in IVAS_ENC_EncodeFrameToSerial /local/knj/ivas-basop/lib_enc/lib_enc.c:1737:24
    #11 0x4d3805 in main /local/knj/ivas-basop/apps/encoder.c:690:24
    #12 0x7ffaefdca249 in __libc_start_call_main csu/../sysdeps/nptl/libc_start_call_main.h:58:16
    #13 0x7ffaefdca304 in __libc_start_main csu/../csu/libc-start.c:360:3
    #14 0x41f580 in _start (/local/knj/ivas-basop/IVAS_cod+0x41f580)

AddressSanitizer can not provide additional info.
SUMMARY: AddressSanitizer: SEGV /local/knj/ivas-basop/lib_com/basop32.c:787:23 in shl_o
==913880==ABORTING

This happens in frame 29 (second frame with non-zero signal). Line from igf_enc.c:

                        IF( last_core_acelp || hPrivateData->wasTransient || EQ_32( hPrivateData->prevDampingFactor_IIR_fx[sfb], L_shl( -1, sub( 15, hPrivateData->prevDampingFactor_IIR_e[sfb] ) ) ) )
                        {
                            hPrivateData->prevDampingFactor_IIR_fx[sfb] = s_max( currDampingFactor_fx, shr( 3277 /*0.1f * 32767*/, currDampingFactor_e ) ); /*resultant exponent stored in hPrivateData->prevDampingFactor_IIR_e[sfb]*/ //<-----

currDampingFactor_fx has value 16348, currDampingFactor_e has value -5. The null pointer access happens apparently because shr does not allow overflows (snippet from basop32.c:949):

#ifdef BASOP_NOGLOB
Word16 shr( Word16 var1, Word16 var2 )
{
    return shr_o( var1, var2, NULL );
}
Word16 shr_sat( Word16 var1, Word16 var2 )
{
    Flag Overflow;
    return shr_o( var1, var2, &Overflow );
}
#endif /* BASOP_NOGLOB */

I can not judge if if the saturating variant of the BASOP should be used instead or if the scaling before needs to be changed - pinging @vaillancour and @multrus.

Edited by Jan Kiene