Skip to content

[CI] Sanitizer test inconsistencies

I had a look at the latest sanitizer pipeline results.

It is a bit confusing that the name of the schedule is “Short testvector sanitizer tests”, while the LTV is used. I guess it can simply be renamed.

The number of test cases differ. We saw it in the total that errors get a count both as a failure and an error, which results in a higher total. But here the number of cases differs between ivas-pytest-enc-usan (249) and ivas-pytest-enc-asan (241) and there are no errors recorded.

https://forge.3gpp.org/rep/sa4/audio/ivas-basop/-/pipelines/46964

  • ivas-pytest-enc-msan: 70 failures, seem to be all testcase timeouts. Even with the faster runners, we seem to exceed the limit for some cases.
  • ivas-pytest-enc-usan: 249 failures (all test cases). I looked at a few and they fail with messages like “Failed: Undefined Behaviour runtime error encountered”, but it is listed as a failure and not an error.
  • ivas-pytest-enc-asan: All 241 test cases passed.
  • ivas-pytest-dec-usan: 580 failures, with runtime errors reported (all non-skipped test cases). Same as for ivas-pytest-enc-usan.
  • ivas-pytest-dec-asan: All 580 test cases passed.
  • ivas-pytest-dec-msan: 1 error reported, 579 passes. Here the message is  “Failed: REF decoder terminated with a non-0 return code: 1"

It seems the return value of the encoder and decoder are not handled in the same way.