Commit 6d00302a authored by Jan Kiene's avatar Jan Kiene
Browse files

Merge branch 'main' into renderer-short-for-ltv-tests

parents 1fecc9d1 3263ad0d

README.md

+46 −17
Original line number Diff line number Diff line
@@ -12,14 +12,9 @@ This repository is mainly meant to share scripts and configs between the two rep 


``` bash

.

├── includes  # folder for (possibly shared) yaml files which are included in the main ones

│   ├── default-variables-basop.yml

│   ├── default-variables.yml

│   ├── job-templates.yml

│   ├── rules-basop.yml

│   └── rules.yml

├── includes  # folder for shared yaml files which are included in the main ones

├── main-basop.yml  # CI config for ivas-basop repo

├── main.yml  # shared CI config

├── main-float.yml  # CI config for ivas-basop repo

├── ...

└── snippets  # Folder for scripts used in CI only. Shared ones go in the folder directly. Can be small ones or bigger scripts which implement a whole job

    ├── basop
@@ -32,7 +27,7 @@ This repository is mainly meant to share scripts and configs between the two rep 
## Usage



For both repositories (ivas-codec and ivas-basop), there should be one main yml file in here which includes stuff from `includes`.

Ideally, there is no other CI code in the respective repositories except a minimal `.gitlab-ci.yml` file which includes from here (currently only achieved for ivas-basop), e.g. (from ivas-basop):

There is no other CI code in the respective repositories except a minimal `.gitlab-ci.yml` file which includes from here, e.g. (from ivas-basop):



``` yaml

variables:
@@ -47,11 +42,37 @@ include: 
    file: main-basop.yml

```



This way, any CI change can be developed centrally in this repository and does not need to be merged/ported to different repositories/branches (e.g. keeping ivas-basop main and ivas-float-update in synch).

This way, any CI change can be developed centrally in this repository and does not need to be merged/ported to different repositories/branches.



### Adding new CI code

### Adding a new job



In general, job definitions should go into `main-float.yml` or `main-basop-yml` if they are specific to one of the repos.

Shared jobs should go into `includes/`.

It might also be helpful for bundling stuff together to put repo-specific code into `includes/` as well.



The workflow and rules section for each CI config are located at the top of the respective `main-*.yml` file.

After implementing your job, adapt or add there so that your job runs when desired.



Prefer adding scripts in `snippets/` over yaml anchors.

Also check the [guidelines](#some-guidelines-hints) below.



For shared jobs, it might be needed to check in which project the CI is running.

You can do that by checking the gitlab-predefined `CI_PROJECT_ID` variable against `PROJECT_ID_FLOAT` and/or `PROJECT_ID_BASOP`, e.g.:



``` yaml

script:

    - |

      if [ "$CI_PROJECT_ID" == "$PROJECT_ID_BASOP" ]; then

        bash "${CI_PROJECT_DIR}"/ivas-codec-ci/snippets/basop/update-scripts-repo.sh

      fi

```



### Testing new CI code



1. Implement your new CI code on a branch in this repo, e.g. `add-awesome-test` and ideally add a merge request for easier review.



- Note: your job needs to inherit the `before_script:` section from `.test-job-linux` or `.test-job-windows`, respectively. Without that, the job will not clone this repo (i.e. ivas-codec-ci) and the snippets will be missing. If your job does not make use of any snippet, in theory you can do without this, but as a general rule, you should inherit from one of these. If your job uses a before_script: section as well, see [guidelines section below](#some-guidelines-hints).



2. Test it by opening a branch in the respective ivas-* repository and changing IVAS_CODEC_CI_REF to your WIP branch, so the yaml there should read:



``` yaml
@@ -61,6 +82,21 @@ This way, any CI change can be developed centrally in this repository and does n 
3. If your change affects multiple repos/branches, you might need to use multiple test branches

4. Once you are done and review is complete, merge `add-awesome-test` to `main` here. The test branches in the other repos are not needed and SHOULD NOT BE MERGED.



### Some guidelines/hints



- if your new job needs to overwrite the before_script: section, one can use this construct to still get the ci repo cloning:



```yaml

before_script:

  - !reference [.test-job-linux, before_script]

  - ...

```



- use only one include section at the top of the main yml file to keep things clear, remember that later stuff overwrites earlier stuff

- in general prefer adding scripts here over using yaml anchors in a CI config file

- if your script shall modify some environment variable, you need to run it with `source` instead of calling it.

- the merge CI config in the ivas-* repos can be viewed in the pipeline editor under "full configuration" - this is especially useful when making changes which should not functionally affect the CI, then it is helpful to copy the full merged config with and without the changes and compare the files directly or as parsed yaml dict locally.



### Overwriting CI variables in the ivas-* repos' CI configs



Usually, everything should be done in the files here, but if there is the need to still overwrite some configuration/control CI variables - e.g. to switch off something temporarily or in a branch-specific manner - this should be done by in the variables: section at the top of the `.gitlab-ci.yml` file.
@@ -87,10 +123,3 @@ Useful configuration variables: 


- `DISABLE_HRTF` - adds `-k not model` to `PYTEST_ADDOPTS` which skips the testcases with -hrtf in the selftest. Probably more for historic reasons.

- `BASOP_CI_SCRIPTS_BRANCH` - used to change the branch from which testscripts are pulled, see [here](https://forge.3gpp.org/rep/ivas-codec-pc/ivas-codec/-/wikis/Software-development/Test-Setup-For-BASOP-Code#how-do-i-check-that-my-new-testcase-does-work-for-basop) for more info.
 


### Some guidelines/hints



- use only one include section at the top of the main yml file to keep things clear, remember that later stuff overwrites earlier stuff

- in general prefer adding scripts here over using yaml anchors in a CI config file

- if your script shall modify some environment variable, you need to run it with `source` instead of calling it.

- the merge CI config in the ivas-* repos can be viewed in the pipeline editor under "full configuration" - this is especially useful when making changes which should not functionally affect the CI, then it is helpful to copy the full merged config with and without the changes and compare the files directly or as parsed yaml dict locally.

includes/build-jobs.yml

0 → 100644
+137 −0
Original line number Diff line number Diff line 
# ---------------------------------------------------------------

# Build jobs

# ---------------------------------------------------------------



# ensure that codec builds on linux

build-codec-linux-make:

  rules:

    - if: $MANUAL_PIPELINE_TYPE == 'long-term-logs' || $MANUAL_PIPELINE_TYPE == 'ivas-conformance-linux' || $MANUAL_PIPELINE_TYPE == 'backup-long-term-logs' || $UPDATE_PAGES || $MANUAL_PIPELINE_TYPE == 'complexity' || $MEASURE_COMPLEXITY_LINUX

      when: never

    - if: $CI_PIPELINE_SOURCE == 'web'

    - if: $CI_PIPELINE_SOURCE == 'push' && $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH

    - if: $CI_PIPELINE_SOURCE == 'merge_request_event'

    - if: $CI_PIPELINE_SOURCE == 'schedule'

    - if: $CI_PIPELINE_SOURCE == 'push'

      when: never

  extends:

    - .build-job-linux

  timeout: "10 minutes"

  script:

    - bash "${CI_PROJECT_DIR}"/ivas-codec-ci/snippets/print-common-info.sh

    - bash "${CI_PROJECT_DIR}"/ivas-codec-ci/snippets/activate-Werror-linux.sh

    - make -j



# ensure that codec builds on linux with cmake

build-codec-linux-cmake:

  rules:

    - if: $CI_MERGE_REQUEST_TITLE =~ /^(\[Draft\]|\(Draft\)|Draft:)/

      when: never

    - if: $MANUAL_PIPELINE_TYPE == 'long-term-logs' || $MANUAL_PIPELINE_TYPE == 'backup-long-term-logs' || $MANUAL_PIPELINE_TYPE == 'ivas-conformance-linux' || $UPDATE_PAGES

      when: never

    - if: $CI_PIPELINE_SOURCE == 'push' && $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH

    - if: $CI_PIPELINE_SOURCE == 'merge_request_event'

    - if: $CI_PIPELINE_SOURCE == 'push'

      when: never

  extends:

    - .build-job-linux

  timeout: "10 minutes"

  script:

    - bash "${CI_PROJECT_DIR}"/ivas-codec-ci/snippets/print-common-info.sh

    - |

      if [ "$CI_PROJECT_ID" == "$PROJECT_ID_BASOP" ]; then

        bash "${CI_PROJECT_DIR}"/ivas-codec-ci/snippets/basop/update-scripts-repo.sh

      fi

    - bash "${CI_PROJECT_DIR}"/ivas-codec-ci/snippets/activate-Werror-linux.sh

    - cmake -B cmake-build -G "Unix Makefiles"

    - cmake --build cmake-build -- -j



# ensure that codec builds on linux with instrumentation active

build-codec-linux-instrumented-make:

  rules:

    - if: $CI_MERGE_REQUEST_TITLE =~ /^(\[Draft\]|\(Draft\)|Draft:)/

      when: never

    - if: $MANUAL_PIPELINE_TYPE == 'long-term-logs' || $MANUAL_PIPELINE_TYPE == 'backup-long-term-logs' || $UPDATE_PAGES

      when: never

    - if: $MEASURE_COMPLEXITY_LINUX || $MANUAL_PIPELINE_TYPE == 'complexity'

    - if: $CI_PIPELINE_SOURCE == 'push' && $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH

    - if: $CI_PIPELINE_SOURCE == 'merge_request_event'

    - if: $CI_PIPELINE_SOURCE == 'push'

      when: never

  extends:

    - .build-job-linux

  timeout: "12 minutes"

  variables:

    INSTRUMENTATION_ARGS: ""

  script:

    - bash "${CI_PROJECT_DIR}"/ivas-codec-ci/snippets/print-common-info.sh

    - |

      if [ "$CI_PROJECT_ID" == "$PROJECT_ID_BASOP" ]; then

        bash "${CI_PROJECT_DIR}"/ivas-codec-ci/snippets/basop/update-scripts-repo.sh

        INSTRUMENTATION_ARGS="-m MEM_ONLY"

      fi

    - bash scripts/prepare_instrumentation.sh $INSTRUMENTATION_ARGS

    - make -j -C $INSTR_DIR



# make sure that the codec builds with msan, asan and usan

build-codec-sanitizers-linux:

  rules:

    - if: $CI_MERGE_REQUEST_TITLE =~ /^(\[Draft\]|\(Draft\)|Draft:)/

      when: never

    - if: $MANUAL_PIPELINE_TYPE == 'long-term-logs' || $MANUAL_PIPELINE_TYPE == 'backup-long-term-logs' || $UPDATE_PAGES

      when: never

    - if: $MANUAL_PIPELINE_TYPE == 'sanitizer'

    - if: $CI_PIPELINE_SOURCE == 'push' && $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH

    - if: $CI_PIPELINE_SOURCE == 'merge_request_event'

    - if: $CI_PIPELINE_SOURCE == 'push'

      when: never

  extends:

    - .build-job-linux

  timeout: "10 minutes"

  script:

    - bash "${CI_PROJECT_DIR}"/ivas-codec-ci/snippets/print-common-info.sh

    - |

      if [ "$CI_PROJECT_ID" == "$PROJECT_ID_BASOP" ]; then

        bash "${CI_PROJECT_DIR}"/ivas-codec-ci/snippets/basop/update-scripts-repo.sh

      fi

    - bash "${CI_PROJECT_DIR}"/ivas-codec-ci/snippets/activate-Werror-linux.sh

    - bash ci/build_codec_sanitizers_linux.sh



# ensure the codec builds with DEBUGGING activated

# warnings are not treated as errors here

build-codec-linux-debugging-make:

  extends:

    - .build-job-linux

  rules:

    - if: $CI_MERGE_REQUEST_TITLE =~ /^(\[Draft\]|\(Draft\)|Draft:)/

      when: never

    - if: $MANUAL_PIPELINE_TYPE == 'long-term-logs' || $MANUAL_PIPELINE_TYPE == 'backup-long-term-logs' || $UPDATE_PAGES

      when: never

    - if: $MANUAL_PIPELINE_TYPE == 'pytest-compare-enc-dmx'

    - if: $CI_PIPELINE_SOURCE == 'merge_request_event'

    - if: $CI_PIPELINE_SOURCE == 'push'

      when: never

  timeout: "10 minutes"

  script:

    - bash "${CI_PROJECT_DIR}"/ivas-codec-ci/snippets/print-common-info.sh

    - bash "${CI_PROJECT_DIR}"/ivas-codec-ci/snippets/enable-debugging-macro.sh

    - make -j



build-codec-windows-msbuild:

  rules:

    - if: $MANUAL_PIPELINE_TYPE == 'long-term-logs' || $MANUAL_PIPELINE_TYPE == 'backup-long-term-logs' || $MANUAL_PIPELINE_TYPE == 'ivas-conformance-linux' || $UPDATE_PAGES || $MANUAL_PIPELINE_TYPE == 'complexity' || $MEASURE_COMPLEXITY_LINUX

      when: never

    - if: $CI_PIPELINE_SOURCE == 'web'

    - if: $CI_PIPELINE_SOURCE == 'push' && $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH

    - if: $CI_PIPELINE_SOURCE == 'merge_request_event'

    - if: $CI_PIPELINE_SOURCE == 'schedule'

    - if: $CI_PIPELINE_SOURCE == 'push'

      when: never

  extends:

    - .build-job-windows

  timeout: "10 minutes"

  tags:

    - ivas-windows

  script:

    - powershell $env:CI_PROJECT_DIR\ivas-codec-ci\snippets\print-common-info-windows.ps1

    - powershell $env:CI_PROJECT_DIR\ivas-codec-ci\snippets\activate-WX-windows.ps1

    - MSBuild.exe -maxcpucount .\Workspace_msvc\Workspace_msvc.sln /property:Configuration=Debug

includes/complexity-jobs.yml

+2 −0
Original line number Diff line number Diff line
@@ -193,6 +193,8 @@ complexity-sba-hoa3-in-ext-out: 
    - if: $MANUAL_PIPELINE_TYPE == 'complexity' || $MEASURE_COMPLEXITY_LINUX

      when: delayed

      start_in: 4 hours 30 minutes

  variables:

    JOB_ID_INJECT: ""

  script:

    - in_format=HOA3

    - out_format=EXT

includes/default-variables-basop.yml

deleted100644 → 0
+0 −93
Original line number Diff line number Diff line 
variables:

  EVS_BE_TEST_DIR_BASOP: "/usr/local/be_2_evs_basop"

  # TODO: cleanup and use global var defined in ivas-codec-ci

  FLOAT_REF_BRANCH_MERGE_SOURCE: ""

  SCRIPTS_DIR: "/usr/local/scripts"

  # These path variables are used by the pytest calls.

  # They can be overwritten in the job templates to e.g. only test encoder or decoder in the chain

  DUT_ENCODER_PATH: "./IVAS_cod_dut"

  DUT_DECODER_PATH: "./IVAS_dec_dut"

  DUT_RENDERER_PATH: "./IVAS_rend_dut"

  DUT_POST_RENDERER_PATH: "./ISAR_post_rend_dut"

  REF_ENCODER_PATH: "./IVAS_cod_ref"

  REF_DECODER_PATH: "./IVAS_dec_ref"

  REF_RENDERER_PATH: "./IVAS_rend_ref"

  REF_POST_RENDERER_PATH: "./ISAR_post_rend_ref"

  MERGE_TARGET_ENCODER_PATH: "./IVAS_cod_merge_target"

  MERGE_TARGET_DECODER_PATH: "./IVAS_dec_merge_target"

  MERGE_TARGET_RENDERER_PATH: "./IVAS_rend_merge_target"

  MERGE_TARGET_POST_RENDERER_PATH: "./ISAR_post_rend_merge_target"

  MERGE_SOURCE_FLOAT_REF_ENCODER_PATH: "./IVAS_cod_merge_source_float_ref"

  MERGE_SOURCE_FLOAT_REF_DECODER_PATH: "./IVAS_dec_merge_source_float_ref"

  MERGE_SOURCE_FLOAT_REF_RENDERER_PATH: "./IVAS_rend_merge_source_float_ref"

  MERGE_SOURCE_FLOAT_REF_POST_RENDERER_PATH: "./ISAR_post_rend_merge_source_float_ref"

  # These path variables are used for building the binaries

  # They should never be overwritten!

  DUT_ENCODER_PATH_FOR_BUILD_DO_NOT_MODIFY: "./IVAS_cod_dut"

  DUT_DECODER_PATH_FOR_BUILD_DO_NOT_MODIFY: "./IVAS_dec_dut"

  DUT_RENDERER_PATH_FOR_BUILD_DO_NOT_MODIFY: "./IVAS_rend_dut"

  DUT_POST_RENDERER_PATH_FOR_BUILD_DO_NOT_MODIFY: "./ISAR_post_rend_dut"

  REF_ENCODER_PATH_FOR_BUILD_DO_NOT_MODIFY: "./IVAS_cod_ref"

  REF_DECODER_PATH_FOR_BUILD_DO_NOT_MODIFY: "./IVAS_dec_ref"

  REF_RENDERER_PATH_FOR_BUILD_DO_NOT_MODIFY: "./IVAS_rend_ref"

  REF_POST_RENDERER_PATH_FOR_BUILD_DO_NOT_MODIFY: "./ISAR_post_rend_ref"

  MERGE_TARGET_ENCODER_PATH_FOR_BUILD_DO_NOT_MODIFY: "./IVAS_cod_merge_target"

  MERGE_TARGET_DECODER_PATH_FOR_BUILD_DO_NOT_MODIFY: "./IVAS_dec_merge_target"

  MERGE_TARGET_RENDERER_PATH_FOR_BUILD_DO_NOT_MODIFY: "./IVAS_rend_merge_target"

  MERGE_TARGET_POST_RENDERER_PATH_FOR_BUILD_DO_NOT_MODIFY: "./ISAR_post_rend_merge_target"

  MERGE_SOURCE_FLOAT_REF_ENCODER_PATH_FOR_BUILD_DO_NOT_MODIFY: "./IVAS_cod_merge_source_float_ref"

  MERGE_SOURCE_FLOAT_REF_DECODER_PATH_FOR_BUILD_DO_NOT_MODIFY: "./IVAS_dec_merge_source_float_ref"

  MERGE_SOURCE_FLOAT_REF_RENDERER_PATH_FOR_BUILD_DO_NOT_MODIFY: "./IVAS_rend_merge_source_float_ref"

  MERGE_SOURCE_FLOAT_REF_POST_RENDERER_PATH_FOR_BUILD_DO_NOT_MODIFY: "./ISAR_post_rend_merge_source_float_ref"

  LEVEL_SCALING: "1.0"

  PRM_FILES: "scripts/config/self_test.prm scripts/config/self_test_ltv.prm"

  TESTCASE_TIMEOUT_STV: 900

  TESTCASE_TIMEOUT_LTV: 2400

  TESTCASE_TIMEOUT_LTV_SANITIZERS: 10800

  CI_REGRESSION_THRESH_MLD: "0.1"

  CI_REGRESSION_THRESH_MAX_ABS_DIFF: "50"

  CI_REGRESSION_THRESH_SSNR: "-1"

  CI_REGRESSION_THRESH_ODG: "-0.05"

  INSTR_DIR: "scripts/c-code_instrument"

  BUILD_WITH_DEBUG_MODE_INFO: ""

  ENCODER_TEST: ""

  DELTA_ODG: ""

  COMPARE_DMX: ""

  SPLIT_COMPARISON: ""

  SKIP_REGRESSION_CHECK: ""

  FAILED_TESTCASES_LIST: "failed-testcases.txt"

  ERRORS_TESTCASES_LIST: "errors-testcases.txt"

  PYTEST_CACHE_ARTIFACT: "pytest_cache.zip"

  MEASURES_FOR_REPORT: "MLD MAX_ABS_DIFF MIN_SSNR MIN_ODG"

  FLOAT_REF_COMMIT_FILE: "float-ref-git-sha.txt"

  CUT_COMMIT_FILE: "CuT-git-sha.txt"

  MERGE_TARGET_COMMIT_FILE: "merge-target-git-sha.txt"

  MERGE_SOURCE_FLOAT_REF_COMMIT_FILE: "merge-source-float-ref-git-sha.txt"

  RUNNER_TAG: "ivas-linux"

  LOGS_BACKUP_SOURCE_DIR: ""

  LOGS_BACKUP_TARGET_DIR: ""

  # set this to true to skip the external HRTF testcases in pytest calls

  DISABLE_HRTF: "false"

  MANUAL_PIPELINE_TYPE:

    description: "Type for the manual pipeline run. Use 'pytest-compare' to run comparison test against reference float codec."

    value: 'default'

    options:

      - 'default'

      - 'pytest-compare'

      - 'pytest-compare-enc-dmx'

      - 'pytest-compare-long'

      - 'pytest-compare-long-fx-fx'

      - 'pytest-compare-to-input'

      - 'pytest-saturation-smoke-test'

      - 'evs-26444'

      - 'sanitizer'

      - 'pytest-renderer'

      - 'complexity'

      - 'coverage'

      - 'voip-be-test'

      - 'renderer-framesize-be'

      - 'peaq-enc-passthrough'

      - 'long-term-logs'

      - 'backup-long-term-logs'

      - 'test-long-self-test'

      - 'ivas-conformance-linux'

includes/default-variables-float.yml

deleted100644 → 0
+0 −28
Original line number Diff line number Diff line 
variables:

  EVS_BE_TEST_DIR: "/usr/local/be_2_evs_test"

  EVS_BE_WIN_TEST_DIR: "C:/Users/gitlab-runner/testvec"

  PROCESSING_SCRIPTS_BIN_DIR: "/test-bin"

  SANITIZER_TESTS: "CLANG1 CLANG2 CLANG3"

  OUT_FORMATS_CHANNEL_BASED: "stereo mono 5_1 5_1_2 5_1_4 7_1 7_1_4"

  OUT_FORMATS_SCENE_BASED: "FOA HOA2 HOA3"

  OUT_FORMATS_BINAURAL: "BINAURAL BINAURAL_ROOM_IR BINAURAL_ROOM_REVERB"

  OUT_FORMATS_ALL: "$OUT_FORMATS_CHANNEL_BASED $OUT_FORMATS_SCENE_BASED $OUT_FORMATS_BINAURAL EXT"

  MANUAL_PIPELINE_TYPE:

    description: "Type for the manual pipeline run. Use 'test-be-release' to run BE test against release codec."

    value: 'default'

    options:

      - 'default'

      - 'test-be-release'

      - 'test-long-self-test'

      - 'ivas-conformance'

      - 'ivas-conformance-linux'

      - 'check-clipping'

      - 'test-branch-vs-input-passthrough'

      - 'coverage'

      - 'complexity'



  PYTEST_ARGS: ""

  COVERAGE_OUTPUT_FILE_STV: "coverage-stv.info"

  COVERAGE_OUTPUT_FILE_LTV: "coverage-ltv.info"

  COVERAGE_OUTPUT_FILE_CONFORMANCE: "coverage-conformance.info"

  COVERAGE_OUTPUT_FILE_MERGED: "coverage-merged.info"