Skip to content

[CI] implement check for regressions in merge request pipeline

In the daily results we observed frequently that some of the measures got worse, e.g. as a side effect of fixing something else. Ideally, this should be caught already in the MR pipelines and changes that cause regressions elsewhere should not be merged in the first place.

Currently, the MR pipeline runs the testcases for the reference (i.e. build from ivas-float-update branch) and for the build from the respective branch, then compares those wrt. MLD, max abs diff, SSNR, ODG. This gives a picture of how the branch compares to the flaot reference, but does not show how the branch compares to main.

Proposal for new setup:

  • Run the same thing, but use the main branch for building the reference (make sure that branch is up-to-date) => this way, only differences to the main branch are reflected in the pipeline results. The differences to the float reference branch are reported daily for the ltv set of testvectors anyway.
  • Do not allow merging if a regression (over a threshold) is detected -> Needs to be discussed if it fits the workflow. We can use a tag in the MR title to explicitly allow bypassing this if needed. Imo, no significant regressions should be allowed to be merged back, otherwise the possibility of circling around back and forth with fixing one thing and breaking another is too high. This may be applied to the decoder tests only for now. The encoder is maybe still too unstable and too many testcases may fail so development might get stuck.
  • One could extend this to not allow crashing decoders to be merged
  • If one still wants to have the comparison to the float reference, on could pre-generate the reference outputs and put them on the runners to save time, then add a mechanism that re-runs reference creation in case there are updates on the reference branch and for checking that all outputs are there by comparing to the main branch outputs. But imo this is not needed anymore at this stage.

Pinging @norvell @kinuthia @vaillancour @malenovsky @pihlajakuja @venkateshsa for comments.