Smoke test in merge request pipeline does not detect failures

The smoke test running in the MR pipelines currently runs (almost) all modes to check for broken modes, but does not fail when something is actually broken.