Undefined behaviour in BASOP implementation causes assert in Stereo DMX EVS encoder with optimized build
# Basic info
<!--- Add commit SHA used to reproduce-->
- Fixed point: 73dd51941
# Bug description
When building the basop codec with an optimization level \>0, one LTV testcase for the EVS compatible stereo downmix crashes. It seems that the input value to the square root gets negative in frame 9956:
{width="900" height="352"}
This was found on Linux (conformance reference platform Ubuntu 24.04 + clang 18) as well as on Mac (clang 18).
## Analysis
TL;DR: the `W_round48_L_o` BASOP implementation relies on c undefined behaviour for checking overflow and applying saturation. The compiler optimizes parts of that away in -O2. `-fwrapv` build flag can be used to prevent that, but it would be better to not rely on C undef behaviour in the BASOPs.
### Longer version:
Mpy_32_32_r(INT32_MIN, INT32_MIN) should saturate to INT32_MAX (= 0x7FFFFFFF). When the codec is built with -O0 this works correctly. But when built with -O2 the same input produces INT32_MIN (= 0x80000000), a sign-flipped, wrapped-around result. The wrong value propagates through the IVAS stereo downmix path: Nr\*Nr becomes negative, so L_tmp1 = Nr² + Ni² becomes negative, the divide that follows produces a negative quotient, and Sqrt32 then triggers the assert.
The actual bug is inside `W_round48_L_o` (called from Mpy_32_32_r via W_shr → W_round48_L). It detects 64-bit signed overflow like this:
```c
L64_var_out = L64_var1 + L64_var2;
if (((L64_var1 ^ L64_var2) & L64_MIN) == 0) { // same sign?
if ((L64_var_out ^ L64_var1) & L64_MIN) { // result flipped sign?
L64_var_out = (L64_var1 < 0) ? L64_MIN : L64_MAX; // saturate
set_overflow(Overflow);
}
}
```
Under the C standard, signed integer overflow is undefined behavior. At -O2 the compiler is allowed to assume this never occurs in L64_var1 + L64_var2 and thus assumes (L64_var_out ^ L64_var1) & L64_MIN is always zero. The compiler deletes the saturation check, the addition wraps at runtime, and W_extract_h returns the wrong upper half (0x80000000).
### Why -fwrapv fixes it
`-fwrapv` tells the compiler that signed integer overflow has well-defined two's-complement wrap-around semantics. With that guarantee, the compiler can no longer assume the overflow check is always false, so the saturation branch survives optimization and W_round48_L_o correctly clamps the result to L64_MAX. Mpy_32_32_r then returns 0x7FFFFFFF and the downstream math stays in valid range.
A cleaner long-term fix would be to rewrite the overflow detection without relying on undef behaviour so the code is correct at any optimization level without needing -fwrapv.
# Ways to reproduce
Here is a test program for the root cause:
[mpy_repro.c](/uploads/19b485e677505576c9382d3d08ae55f1/mpy_repro.c)
Build like this:
```bash
cc -O0 mpy_repro.c -o mpy_O0
cc -O2 mpy_repro.c -o mpy_O2
cc -O2 -fwrapv mpy_repro.c -o mpy_O2_wrap
```
Command line for encoder:
<!--Commandline or script-->
```bash
make clean
OPTIM=2 make -j
./IVAS_cod -stereo_dmx_evs 24400 48 ltv48_STEREO.wav bit
```
<!--- Below are labels that will be added but are not shown in description. This is a template to help fill them.
Add further information to the first row and remove and add labels as necessary.-->
issue