Skip to content

UBSAN: left shift of negative values in 1st stage of MSVQ

For two places in the 1st stage of the MSVQ encoder, the undef behaviour sanitizer reports left shift of negative values:

lib_enc/lsf_msvq_ma_enc.c:151:73: runtime error: left shift of negative value -126
SUMMARY: UndefinedBehaviorSanitizer: undefined-behavior lib_enc/lsf_msvq_ma_enc.c:151:73 in 
lib_com/lsf_tools.c:2075:58: runtime error: left shift of negative value -1
SUMMARY: UndefinedBehaviorSanitizer: undefined-behavior lib_com/lsf_tools.c:2075:58 in 

The respective locations in the code look like this:

                tmp = dct_target[c2] - (float) ( ( (Word16) cbpW8[c2] ) << dct_col_shift_tab[c2] ); /* Word8 storage MSE inner loop */
        dct_vec[col] = (float) ( ( (Word16) cbpW8[col] ) << dct_col_shift_tab[col] );

In both cases, the negative value comes from a table (cbpW8), so this is likely not just a symptom of something going wrong somewhere else.

Can this be replaced by multiplication with 2^dct_col_shift_tab[col] instead? That should be the same operation, but in an always well-defined way. There are no overflows reported, but that may be shadowed by this errors, so afterwards needs to be rechecked. No idea if that will affect BE, though (probably depends on the platform)...

Was found when running self_test.py with the UBSAN. There, it triggers for a lot of testcases. Here is an explicit command line from the first one:

make clean
make -j CLANG=3
./IVAS_cod -STEREO -dtx 13200 16 scripts/testv/stvST16n.wav bit