[CI] Renderer testcases in BASOP long MLD tests do not report runtime errors correctly
E.g. in this job https://forge.3gpp.org/rep/sa4/audio/ivas-basop/-/jobs/734550, the check for runtime errors reports errors in the console log, but the html report lists the respective testcases as "FAILED" instead of "ERROR" as it should be. The XML report has an "<error>" tag for the testcases and also reports the number of errors correctly.
Artifacts for later reference: [reports.zip](/uploads/f5b759305d36751c4a890e0be49c151f/reports.zip)
Pytest reports testcases as FAILED if an exception is raised during test execution. If the exception is raised during setup or teardown of a fixture, pytest reports ERROR instead. In our test code, this is handled in the test_info fixture by checking if an error message is present:
```python
# fixture returns test information, enabling per-testcase SNR
@pytest.fixture(scope="function")
def test_info(request):
yield request
# Check for errors during teardown to report error instead of failure
if hasattr(request, "error"):
pytest.fail(request.error)
```
The code in `tests/renderer[_short]/utils.py` which handles running the renderer command should correctly set test_info.error:
```python
def _run_cmd(cmd, test_info=None, env=None):
"""
Helper function for running some command.
Raises a SystemError if either the return code is non-zero or a USAN printout is detected
"""
proc = sp.run(cmd, capture_output=True, text=True, env=env)
stdout = proc.stdout + proc.stderr
# check for USAN error first
if "UndefinedBehaviorSanitizer" in stdout:
error = f"USAN error detected in stdout of command: {' '.join(cmd)}\n{stdout}"
if test_info is not None:
test_info.error = error
raise SystemError(error)
# then handle possible crash
try:
proc.check_returncode()
except sp.CalledProcessError as e:
error = f"Command returned non-zero exit status ({e.returncode}): {' '.join(e.cmd)}\n{e.stderr}\n{e.stdout}"
if test_info is not None:
test_info.error = error
raise SystemError(error)
```
So, if the renderer binary returned non-zero, the error should be recorded in the `test_info` object and cause `pytest.fail` to be run in the teardown and issue ERROR + raise SystemError which would mark the testcase as FAILED as well -> that is why crashing testcases are reported as both FAILED and ERROR in basically all other tests.
The problem is that this does not seem work in CI. I couldn't reproduce this neither locally (there, I got both FAILED and ERROR as expected) nor in the same docker image as the CI uses. The behaviour also seems to be runner-independent.
Bottomline for now: when checking the LTV MLD jobs, a failed gitlab job (red label) means that there was a crash somewhere, even if the html report does not show it. You need to Ctrl-f for "error message" in the XML report to find the crashed testcase.
issue