Commit 35ef2902 authored by Jan Kiene's avatar Jan Kiene
Browse files

Merge branch 'main' into unify-complexity-measurements

parents f0e20c4f 97c123d5

README.md

+84 −2
Original line number Diff line number Diff line
@@ -6,9 +6,91 @@ Home of CI configs and scripts for: 
- [IVAS BASOP](https://forge.3gpp.org/rep/sa4/audio/ivas-basop)



Refer to the  [IVAS Wiki](https://forge.3gpp.org/rep/ivas-codec-pc/ivas-codec/-/wikis/home) for more info on the CI setup.

This repository is mainly meant to share scripts and configs between the two repositories and (for BASOP) between differently configured branches.

This repository is mainly meant to share scripts and configs between the two repositories and between differently configured branches.



## Notes about usage of the snippets

## Structure



``` bash

.

├── includes  # folder for (possibly shared) yaml files which are included in the main ones

│   ├── default-variables-basop.yml

│   ├── default-variables.yml

│   ├── job-templates.yml

│   ├── rules-basop.yml

│   └── rules.yml

├── main-basop.yml  # CI config for ivas-basop repo

├── main.yml  # shared CI config

├── ...

└── snippets  # Folder for scripts used in CI only. Shared ones go in the folder directly. Can be small ones or bigger scripts which implement a whole job

    ├── basop

    │   ├── ...

    ├── float

    │   └── ...

    └── ...

```



## Usage



For both repositories (ivas-codec and ivas-basop), there should be one main yml file in here which includes stuff from `includes`.

Ideally, there is no other CI code in the respective repositories except a minimal `.gitlab-ci.yml` file which includes from here (currently only achieved for ivas-basop), e.g. (from ivas-basop):



``` yaml

variables:

  # note: GitLab cannot reference variables defined by users in the include ref:, we need to use a YAML anchor for this

  # see https://docs.gitlab.com/ci/yaml/includes/#use-variables-with-include for more information

  IVAS_CODEC_CI_REF: &IVAS_CODEC_CI_REF main



# all CI code and config is included from https://forge.3gpp.org/rep/ivas-codec-pc/ivas-codec-ci

include:

  - project: ivas-codec-pc/ivas-codec-ci

    ref: *IVAS_CODEC_CI_REF

    file: main-basop.yml

```



This way, any CI change can be developed centrally in this repository and does not need to be merged/ported to different repositories/branches (e.g. keeping ivas-basop main and ivas-float-update in synch).



### Adding new CI code



1. Implement your new CI code on a branch in this repo, e.g. `add-awesome-test` and ideally add a merge request for easier review.

2. Test it by opening a branch in the respective ivas-* repository and changing IVAS_CODEC_CI_REF to your WIP branch, so the yaml there should read:



``` yaml

  IVAS_CODEC_CI_REF: &IVAS_CODEC_CI_REF add-awesome-test

```



3. If your change affects multiple repos/branches, you might need to use multiple test branches

4. Once you are done and review is complete, merge `add-awesome-test` to `main` here. The test branches in the other repos are not needed and SHOULD NOT BE MERGED.



### Overwriting CI variables in the ivas-* repos' CI configs



Usually, everything should be done in the files here, but if there is the need to still overwrite some configuration/control CI variables - e.g. to switch off something temporarily or in a branch-specific manner - this should be done by in the variables: section at the top of the `.gitlab-ci.yml` file.



Example: The variable `DISABLE_HRTF` is used in ivas-basop to disable all testcase that use external HRTF files.

The relevant part in the `.gitlab-ci.yml` file in ivas-basop looks like given below.

In practice, one would uncomment the last three lines to switch off the HRTF testcase in the respective branch only.



``` yaml

variables:

  # note: GitLab cannot reference variables defined by users in the include ref:, we need to use a YAML anchor for this

  # see https://docs.gitlab.com/ci/yaml/includes/#use-variables-with-include for more information

  IVAS_CODEC_CI_REF: &IVAS_CODEC_CI_REF main

  # If you need to set some config variable only in a local branch, then add an overwrite here

  # One example is DISABLE_HRTF - this will be set on a branch which is about to be merged and will be removed in a subsequent second MR

  # this is more easily done directly here in the child repo



  # these lines are suposed to stay commented out to serve as an example

  # # set this to true to skip the external HRTF testcases in pytest calls

  # DISABLE_HRTF: "true"

```



Useful configuration variables:



- `DISABLE_HRTF` - adds `-k not model` to `PYTEST_ADDOPTS` which skips the testcases with -hrtf in the selftest. Probably more for historic reasons.

- `BASOP_CI_SCRIPTS_BRANCH` - used to change the branch from which testscripts are pulled, see [here](https://forge.3gpp.org/rep/ivas-codec-pc/ivas-codec/-/wikis/Software-development/Test-Setup-For-BASOP-Code#how-do-i-check-that-my-new-testcase-does-work-for-basop) for more info.



### Some guidelines/hints



- use only one include section at the top of the main yml file to keep things clear, remember that later stuff overwrites earlier stuff

- in general prefer adding scripts here over using yaml anchors in a CI config file

- if your script shall modify some environment variable, you need to run it with `source` instead of calling it.

- the merge CI config in the ivas-* repos can be viewed in the pipeline editor under "full configuration" - this is especially useful when making changes which should not functionally affect the CI, then it is helpful to copy the full merged config with and without the changes and compare the files directly or as parsed yaml dict locally.

includes/basop-default-variables.yml

deleted100644 → 0
+0 −5
Original line number Diff line number Diff line 
variables:

  FLOAT_REF_COMMIT_FILE: "float-ref-git-sha.txt"

  MERGE_SOURCE_FLOAT_REF_COMMIT_FILE: "merge-source-float-ref-git-sha.txt"

  MERGE_TARGET_COMMIT_FILE: "merge-target-git-sha.txt"

  FLOAT_REF_COMMIT: "ivas-float-update"

includes/default-variables-basop.yml

0 → 100644
+100 −0
Original line number Diff line number Diff line 
variables:

  EVS_BE_TEST_DIR_BASOP: "/usr/local/be_2_evs_basop"

  # TODO: cleanup and use global var defined in ivas-codec-ci

  FLOAT_REF_BRANCH: "ivas-float-update"

  FLOAT_REF_BRANCH_MERGE_SOURCE: ""

  SCRIPTS_DIR: "/usr/local/scripts"

  LONG_TEST_SUITE: "tests/codec_be_on_mr_nonselection tests/renderer --param_file scripts/config/self_test_ltv.prm --use_ltv"

  LONG_TEST_SUITE_NO_RENDERER: "tests/codec_be_on_mr_nonselection --param_file scripts/config/self_test_ltv.prm --use_ltv"

  SHORT_TEST_SUITE: "tests/codec_be_on_mr_nonselection"

  SHORT_TEST_SUITE_ENCODER: "tests/codec_be_on_mr_nonselection/test_param_file.py --param_file scripts/config/self_test_basop_encoder.prm"

  LONG_TEST_SUITE_ENCODER: "tests/codec_be_on_mr_nonselection/test_param_file.py --param_file scripts/config/self_test_ltv_basop_encoder.prm"

  TEST_SUITE: ""

  # These path variables are used by the pytest calls.

  # They can be overwritten in the job templates to e.g. only test encoder or decoder in the chain

  DUT_ENCODER_PATH: "./IVAS_cod_dut"

  DUT_DECODER_PATH: "./IVAS_dec_dut"

  DUT_RENDERER_PATH: "./IVAS_rend_dut"

  DUT_POST_RENDERER_PATH: "./ISAR_post_rend_dut"

  REF_ENCODER_PATH: "./IVAS_cod_ref"

  REF_DECODER_PATH: "./IVAS_dec_ref"

  REF_RENDERER_PATH: "./IVAS_rend_ref"

  REF_POST_RENDERER_PATH: "./ISAR_post_rend_ref"

  MERGE_TARGET_ENCODER_PATH: "./IVAS_cod_merge_target"

  MERGE_TARGET_DECODER_PATH: "./IVAS_dec_merge_target"

  MERGE_TARGET_RENDERER_PATH: "./IVAS_rend_merge_target"

  MERGE_TARGET_POST_RENDERER_PATH: "./ISAR_post_rend_merge_target"

  MERGE_SOURCE_FLOAT_REF_ENCODER_PATH: "./IVAS_cod_merge_source_float_ref"

  MERGE_SOURCE_FLOAT_REF_DECODER_PATH: "./IVAS_dec_merge_source_float_ref"

  MERGE_SOURCE_FLOAT_REF_RENDERER_PATH: "./IVAS_rend_merge_source_float_ref"

  MERGE_SOURCE_FLOAT_REF_POST_RENDERER_PATH: "./ISAR_post_rend_merge_source_float_ref"

  # These path variables are used for building the binaries

  # They should never be overwritten!

  DUT_ENCODER_PATH_FOR_BUILD_DO_NOT_MODIFY: "./IVAS_cod_dut"

  DUT_DECODER_PATH_FOR_BUILD_DO_NOT_MODIFY: "./IVAS_dec_dut"

  DUT_RENDERER_PATH_FOR_BUILD_DO_NOT_MODIFY: "./IVAS_rend_dut"

  DUT_POST_RENDERER_PATH_FOR_BUILD_DO_NOT_MODIFY: "./ISAR_post_rend_dut"

  REF_ENCODER_PATH_FOR_BUILD_DO_NOT_MODIFY: "./IVAS_cod_ref"

  REF_DECODER_PATH_FOR_BUILD_DO_NOT_MODIFY: "./IVAS_dec_ref"

  REF_RENDERER_PATH_FOR_BUILD_DO_NOT_MODIFY: "./IVAS_rend_ref"

  REF_POST_RENDERER_PATH_FOR_BUILD_DO_NOT_MODIFY: "./ISAR_post_rend_ref"

  MERGE_TARGET_ENCODER_PATH_FOR_BUILD_DO_NOT_MODIFY: "./IVAS_cod_merge_target"

  MERGE_TARGET_DECODER_PATH_FOR_BUILD_DO_NOT_MODIFY: "./IVAS_dec_merge_target"

  MERGE_TARGET_RENDERER_PATH_FOR_BUILD_DO_NOT_MODIFY: "./IVAS_rend_merge_target"

  MERGE_TARGET_POST_RENDERER_PATH_FOR_BUILD_DO_NOT_MODIFY: "./ISAR_post_rend_merge_target"

  MERGE_SOURCE_FLOAT_REF_ENCODER_PATH_FOR_BUILD_DO_NOT_MODIFY: "./IVAS_cod_merge_source_float_ref"

  MERGE_SOURCE_FLOAT_REF_DECODER_PATH_FOR_BUILD_DO_NOT_MODIFY: "./IVAS_dec_merge_source_float_ref"

  MERGE_SOURCE_FLOAT_REF_RENDERER_PATH_FOR_BUILD_DO_NOT_MODIFY: "./IVAS_rend_merge_source_float_ref"

  MERGE_SOURCE_FLOAT_REF_POST_RENDERER_PATH_FOR_BUILD_DO_NOT_MODIFY: "./ISAR_post_rend_merge_source_float_ref"

  LEVEL_SCALING: "1.0"

  PRM_FILES: "scripts/config/self_test.prm scripts/config/self_test_ltv.prm"

  TESTCASE_TIMEOUT_STV: 900

  TESTCASE_TIMEOUT_LTV: 2400

  TESTCASE_TIMEOUT_LTV_SANITIZERS: 10800

  CI_REGRESSION_THRESH_MLD: "0.1"

  CI_REGRESSION_THRESH_MAX_ABS_DIFF: "50"

  CI_REGRESSION_THRESH_SSNR: "-1"

  CI_REGRESSION_THRESH_ODG: "-0.05"

  INSTR_DIR: "scripts/c-code_instrument"

  BUILD_WITH_DEBUG_MODE_INFO: ""

  ENCODER_TEST: ""

  DELTA_ODG: ""

  COMPARE_DMX: ""

  SPLIT_COMPARISON: ""

  SKIP_REGRESSION_CHECK: ""

  FAILED_TESTCASES_LIST: "failed-testcases.txt"

  ERRORS_TESTCASES_LIST: "errors-testcases.txt"

  PYTEST_CACHE_ARTIFACT: "pytest_cache.zip"

  MEASURES_FOR_REPORT: "MLD MAX_ABS_DIFF MIN_SSNR MIN_ODG"

  FLOAT_REF_COMMIT_FILE: "float-ref-git-sha.txt"

  CUT_COMMIT_FILE: "CuT-git-sha.txt"

  MERGE_TARGET_COMMIT_FILE: "merge-target-git-sha.txt"

  MERGE_SOURCE_FLOAT_REF_COMMIT_FILE: "merge-source-float-ref-git-sha.txt"

  RUNNER_TAG: "ivas-basop-linux"

  LOGS_BACKUP_SOURCE_DIR: ""

  LOGS_BACKUP_TARGET_DIR: ""

  # set this to true to skip the external HRTF testcases in pytest calls

  DISABLE_HRTF: "false"

  MANUAL_PIPELINE_TYPE:

    description: "Type for the manual pipeline run. Use 'pytest-compare' to run comparison test against reference float codec."

    value: 'default'

    options:

      - 'default'

      - 'pytest-compare'

      - 'pytest-compare-enc-dmx'

      - 'pytest-compare-long'

      - 'pytest-compare-long-fx-fx'

      - 'pytest-compare-to-input'

      - 'pytest-saturation-smoke-test'

      - 'evs-26444'

      - 'sanitizer'

      - 'pytest-renderer'

      - 'complexity'

      - 'coverage'

      - 'voip-be-test'

      - 'renderer-framesize-be'

      - 'peaq-enc-passthrough'

      - 'long-term-logs'

      - 'backup-long-term-logs'

      - 'test-long-self-test'

      - 'ivas-conformance-linux'

includes/default-variables-float.yml

0 → 100644
+33 −0
Original line number Diff line number Diff line 
variables:

  EVS_BE_TEST_DIR: "/usr/local/be_2_evs_test"

  EVS_BE_WIN_TEST_DIR: "C:/Users/gitlab-runner/testvec"

  PROCESSING_SCRIPTS_BIN_DIR: "/test-bin"

  SANITIZER_TESTS: "CLANG1 CLANG2 CLANG3"

  OUT_FORMATS_CHANNEL_BASED: "stereo mono 5_1 5_1_2 5_1_4 7_1 7_1_4"

  OUT_FORMATS_SCENE_BASED: "FOA HOA2 HOA3"

  OUT_FORMATS_BINAURAL: "BINAURAL BINAURAL_ROOM_IR BINAURAL_ROOM_REVERB"

  OUT_FORMATS_ALL: "$OUT_FORMATS_CHANNEL_BASED $OUT_FORMATS_SCENE_BASED $OUT_FORMATS_BINAURAL EXT"

  MANUAL_PIPELINE_TYPE:

    description: "Type for the manual pipeline run. Use 'test-be-release' to run BE test against release codec."

    value: 'default'

    options:

      - 'default'

      - 'test-be-release'

      - 'test-long-self-test'

      - 'ivas-conformance'

      - 'ivas-conformance-linux'

      - 'check-clipping'

      - 'test-branch-vs-input-passthrough'

      - 'coverage'



  PYTEST_ARGS: ""

  LONG_TEST_SUITE: "tests/codec_be_on_mr_nonselection tests/renderer --param_file scripts/config/self_test_ltv.prm --use_ltv"

  LONG_TEST_SUITE_NO_RENDERER: "tests/codec_be_on_mr_nonselection --param_file scripts/config/self_test_ltv.prm --use_ltv"

  SHORT_TEST_SUITE: "tests/codec_be_on_mr_nonselection"

  SHORT_TEST_SUITE_ENCODER: "tests/codec_be_on_mr_nonselection/test_param_file.py --param_file scripts/config/self_test_basop_encoder.prm"

  LONG_TEST_SUITE_ENCODER: "tests/codec_be_on_mr_nonselection/test_param_file.py --param_file scripts/config/self_test_ltv_basop_encoder.prm"

  TEST_SUITE: ""

  COVERAGE_OUTPUT_FILE_STV: "coverage-stv.info"

  COVERAGE_OUTPUT_FILE_LTV: "coverage-ltv.info"

  COVERAGE_OUTPUT_FILE_CONFORMANCE: "coverage-conformance.info"

  COVERAGE_OUTPUT_FILE_MERGED: "coverage-merged.info"

includes/default-variables.yml

+3 −3
Original line number Diff line number Diff line
@@ -9,11 +9,11 @@ variables: 
  GIT_CLEAN_FLAGS: -ffdxq

  TESTCASE_TIMEOUT_STV_SANITIZERS: 240

  TESTCASE_TIMEOUT_LTV_SANITIZERS: 2400

  BASOP_REFERENCE_BRANCH: "ivas-float-update"

  SCALE_FACTOR: "3.162"

  BASOP_CI_BRANCH_PC_REPO: "basop-ci-branch"

  PYTEST_ADDOPTS: ""

  # overwrite this via .gitlab/variables.yml in the respective other repo to disable "(model from file)" testcases

  # this contains the name of the branch in ivas-codec repo that is used to copy the ci scripts from

  BASOP_CI_SCRIPTS_BRANCH: "main"

  # overwrite this via .gitlab-ci.yml in the respective other repo to disable "(model from file)" testcases

  DISABLE_HRTF: "false"

  PROJECT_ID_FLOAT: "49"

  PROJECT_ID_BASOP: "77"