Precision issues for Orientation tracking REF_VEC mode in VectorRotationToQuaternion_fx

Basic info

• Float reference: f9ca922123
• Fixed point: cf7f073c (main) ensure FIX_BASOP_2361_OTR is active

Bug description

The headrotation trajectory full-circle-with-up-and-down-4s.csv is expected to be cancelled out by the static head trajectory const000.csv + the reference vector movement full-circle-with-up-and-down-4s-ccw-Vector3.csv. However this is not the case and a single digit MAX_ABS_DIFF is observed.

Further details in comment below. The root cause is related to precision of the vector math functions.

Ways to reproduce

Run the tests as described in the original issue #2361 with the branch/fix mentioned above.

make -j
python3 -m pytest -n auto -rA tests/renderer_short/test_renderer.py

Or run e.g. a failing testcase directly (with STV):

# Reference
./IVAS_rend -q -no_delay_cmp -fs 48 -fr 5 -i scripts/testv/stv714MC48c.wav -if 7_1_4 -of BINAURAL -o ref.wav -T scripts/trajectories/full-circle-with-up-and-down-4s.csv
# OTR REF VEC
./IVAS_rend -q -no_delay_cmp -fs 48 -fr 5 -i scripts/testv/stv714MC48c.wav -if 7_1_4 -of BINAURAL -o otr_ref_vec.wav -T scripts/trajectories/const000.csv -rvf scripts/trajectories/full-circle-with-up-and-down-4s-ccw-Vector3.csv -otr ref_vec

# diff the files ref.wav and otr_ref_vec.wav