diff --git a/README.md b/README.md index 60405aaf47b52942b65c2ae112f623f9f784621c..d17d5eb438477b60f94515cec75119ec7b4fe0e2 100644 --- a/README.md +++ b/README.md @@ -12,14 +12,9 @@ This repository is mainly meant to share scripts and configs between the two rep ``` bash . -├── includes # folder for (possibly shared) yaml files which are included in the main ones -│   ├── default-variables-basop.yml -│   ├── default-variables.yml -│   ├── job-templates.yml -│   ├── rules-basop.yml -│   └── rules.yml +├── includes # folder for shared yaml files which are included in the main ones ├── main-basop.yml # CI config for ivas-basop repo -├── main.yml # shared CI config +├── main-float.yml # CI config for ivas-basop repo ├── ... └── snippets # Folder for scripts used in CI only. Shared ones go in the folder directly. Can be small ones or bigger scripts which implement a whole job ├── basop @@ -32,7 +27,7 @@ This repository is mainly meant to share scripts and configs between the two rep ## Usage For both repositories (ivas-codec and ivas-basop), there should be one main yml file in here which includes stuff from `includes`. -Ideally, there is no other CI code in the respective repositories except a minimal `.gitlab-ci.yml` file which includes from here (currently only achieved for ivas-basop), e.g. (from ivas-basop): +There is no other CI code in the respective repositories except a minimal `.gitlab-ci.yml` file which includes from here, e.g. (from ivas-basop): ``` yaml variables: @@ -47,11 +42,37 @@ include: file: main-basop.yml ``` -This way, any CI change can be developed centrally in this repository and does not need to be merged/ported to different repositories/branches (e.g. keeping ivas-basop main and ivas-float-update in synch). +This way, any CI change can be developed centrally in this repository and does not need to be merged/ported to different repositories/branches. -### Adding new CI code +### Adding a new job + +In general, job definitions should go into `main-float.yml` or `main-basop-yml` if they are specific to one of the repos. +Shared jobs should go into `includes/`. +It might also be helpful for bundling stuff together to put repo-specific code into `includes/` as well. + +The workflow and rules section for each CI config are located at the top of the respective `main-*.yml` file. +After implementing your job, adapt or add there so that your job runs when desired. + +Prefer adding scripts in `snippets/` over yaml anchors. +Also check the [guidelines](#some-guidelines-hints) below. + +For shared jobs, it might be needed to check in which project the CI is running. +You can do that by checking the gitlab-predefined `CI_PROJECT_ID` variable against `PROJECT_ID_FLOAT` and/or `PROJECT_ID_BASOP`, e.g.: + +``` yaml +script: + - | + if [ "$CI_PROJECT_ID" == "$PROJECT_ID_BASOP" ]; then + bash "${CI_PROJECT_DIR}"/ivas-codec-ci/snippets/basop/update-scripts-repo.sh + fi +``` + +### Testing new CI code 1. Implement your new CI code on a branch in this repo, e.g. `add-awesome-test` and ideally add a merge request for easier review. + +- Note: your job needs to inherit the `before_script:` section from `.test-job-linux` or `.test-job-windows`, respectively. Without that, the job will not clone this repo (i.e. ivas-codec-ci) and the snippets will be missing. If your job does not make use of any snippet, in theory you can do without this, but as a general rule, you should inherit from one of these. If your job uses a before_script: section as well, see [guidelines section below](#some-guidelines-hints). + 2. Test it by opening a branch in the respective ivas-* repository and changing IVAS_CODEC_CI_REF to your WIP branch, so the yaml there should read: ``` yaml @@ -61,6 +82,20 @@ This way, any CI change can be developed centrally in this repository and does n 3. If your change affects multiple repos/branches, you might need to use multiple test branches 4. Once you are done and review is complete, merge `add-awesome-test` to `main` here. The test branches in the other repos are not needed and SHOULD NOT BE MERGED. +### Some guidelines/hints + +- if your new job needs to overwrite the before_script: section, one can use this construct to still get the ci repo cloning: + +```yaml +before_script: + - !reference [.test-job-linux, before_script] + - ... +``` + +- use only one include section at the top of the main yml file to keep things clear, remember that later stuff overwrites earlier stuff +- if your script shall modify some environment variable, you need to run it with `source` instead of calling it. +- the merge CI config in the ivas-* repos can be viewed in the pipeline editor under "full configuration" - this is especially useful when making changes which should not functionally affect the CI, then it is helpful to copy the full merged config with and without the changes and compare the files directly or as parsed yaml dict locally. + ### Overwriting CI variables in the ivas-* repos' CI configs Usually, everything should be done in the files here, but if there is the need to still overwrite some configuration/control CI variables - e.g. to switch off something temporarily or in a branch-specific manner - this should be done by in the variables: section at the top of the `.gitlab-ci.yml` file. @@ -87,10 +122,3 @@ Useful configuration variables: - `DISABLE_HRTF` - adds `-k not model` to `PYTEST_ADDOPTS` which skips the testcases with -hrtf in the selftest. Probably more for historic reasons. - `BASOP_CI_SCRIPTS_BRANCH` - used to change the branch from which testscripts are pulled, see [here](https://forge.3gpp.org/rep/ivas-codec-pc/ivas-codec/-/wikis/Software-development/Test-Setup-For-BASOP-Code#how-do-i-check-that-my-new-testcase-does-work-for-basop) for more info. - -### Some guidelines/hints - -- use only one include section at the top of the main yml file to keep things clear, remember that later stuff overwrites earlier stuff -- in general prefer adding scripts here over using yaml anchors in a CI config file -- if your script shall modify some environment variable, you need to run it with `source` instead of calling it. -- the merge CI config in the ivas-* repos can be viewed in the pipeline editor under "full configuration" - this is especially useful when making changes which should not functionally affect the CI, then it is helpful to copy the full merged config with and without the changes and compare the files directly or as parsed yaml dict locally. diff --git a/includes/build-jobs.yml b/includes/build-jobs.yml index 948b69c61cfa54e6f6a80421f60ef8c06be265f1..0fc5b68c613647b2ce6a05bb4114e3956e4ef2fe 100644 --- a/includes/build-jobs.yml +++ b/includes/build-jobs.yml @@ -1,3 +1,101 @@ +# --------------------------------------------------------------- +# Build jobs +# --------------------------------------------------------------- + +# ensure that codec builds on linux +build-codec-linux-make: + rules: + - if: $MANUAL_PIPELINE_TYPE == 'long-term-logs' || $MANUAL_PIPELINE_TYPE == 'ivas-conformance-linux' || $MANUAL_PIPELINE_TYPE == 'backup-long-term-logs' || $UPDATE_PAGES || $MANUAL_PIPELINE_TYPE == 'complexity' || $MEASURE_COMPLEXITY_LINUX + when: never + - if: $CI_PIPELINE_SOURCE == 'web' + - if: $CI_PIPELINE_SOURCE == 'push' && $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH + - if: $CI_PIPELINE_SOURCE == 'merge_request_event' + - if: $CI_PIPELINE_SOURCE == 'schedule' + - if: $CI_PIPELINE_SOURCE == 'push' + when: never + extends: + - .build-job-linux + timeout: "10 minutes" + script: + - bash "${CI_PROJECT_DIR}"/ivas-codec-ci/snippets/print-common-info.sh + - bash "${CI_PROJECT_DIR}"/ivas-codec-ci/snippets/activate-Werror-linux.sh + - make -j + +# ensure that codec builds on linux with cmake +build-codec-linux-cmake: + rules: + - if: $CI_MERGE_REQUEST_TITLE =~ /^(\[Draft\]|\(Draft\)|Draft:)/ + when: never + - if: $MANUAL_PIPELINE_TYPE == 'long-term-logs' || $MANUAL_PIPELINE_TYPE == 'backup-long-term-logs' || $MANUAL_PIPELINE_TYPE == 'ivas-conformance-linux' || $UPDATE_PAGES + when: never + - if: $CI_PIPELINE_SOURCE == 'push' && $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH + - if: $CI_PIPELINE_SOURCE == 'merge_request_event' + - if: $CI_PIPELINE_SOURCE == 'push' + when: never + extends: + - .build-job-linux + timeout: "10 minutes" + script: + - bash "${CI_PROJECT_DIR}"/ivas-codec-ci/snippets/print-common-info.sh + - | + if [ "$CI_PROJECT_ID" == "$PROJECT_ID_BASOP" ]; then + bash "${CI_PROJECT_DIR}"/ivas-codec-ci/snippets/basop/update-scripts-repo.sh + fi + - bash "${CI_PROJECT_DIR}"/ivas-codec-ci/snippets/activate-Werror-linux.sh + - cmake -B cmake-build -G "Unix Makefiles" + - cmake --build cmake-build -- -j + +# ensure that codec builds on linux with instrumentation active +build-codec-linux-instrumented-make: + rules: + - if: $CI_MERGE_REQUEST_TITLE =~ /^(\[Draft\]|\(Draft\)|Draft:)/ + when: never + - if: $MANUAL_PIPELINE_TYPE == 'long-term-logs' || $MANUAL_PIPELINE_TYPE == 'backup-long-term-logs' || $UPDATE_PAGES + when: never + - if: $MEASURE_COMPLEXITY_LINUX || $MANUAL_PIPELINE_TYPE == 'complexity' + - if: $CI_PIPELINE_SOURCE == 'push' && $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH + - if: $CI_PIPELINE_SOURCE == 'merge_request_event' + - if: $CI_PIPELINE_SOURCE == 'push' + when: never + extends: + - .build-job-linux + timeout: "12 minutes" + variables: + INSTRUMENTATION_ARGS: "" + script: + - bash "${CI_PROJECT_DIR}"/ivas-codec-ci/snippets/print-common-info.sh + - | + if [ "$CI_PROJECT_ID" == "$PROJECT_ID_BASOP" ]; then + bash "${CI_PROJECT_DIR}"/ivas-codec-ci/snippets/basop/update-scripts-repo.sh + INSTRUMENTATION_ARGS="-m MEM_ONLY" + fi + - bash scripts/prepare_instrumentation.sh $INSTRUMENTATION_ARGS + - make -j -C $INSTR_DIR + +# make sure that the codec builds with msan, asan and usan +build-codec-sanitizers-linux: + rules: + - if: $CI_MERGE_REQUEST_TITLE =~ /^(\[Draft\]|\(Draft\)|Draft:)/ + when: never + - if: $MANUAL_PIPELINE_TYPE == 'long-term-logs' || $MANUAL_PIPELINE_TYPE == 'backup-long-term-logs' || $UPDATE_PAGES + when: never + - if: $MANUAL_PIPELINE_TYPE == 'sanitizer' + - if: $CI_PIPELINE_SOURCE == 'push' && $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH + - if: $CI_PIPELINE_SOURCE == 'merge_request_event' + - if: $CI_PIPELINE_SOURCE == 'push' + when: never + extends: + - .build-job-linux + timeout: "10 minutes" + script: + - bash "${CI_PROJECT_DIR}"/ivas-codec-ci/snippets/print-common-info.sh + - | + if [ "$CI_PROJECT_ID" == "$PROJECT_ID_BASOP" ]; then + bash "${CI_PROJECT_DIR}"/ivas-codec-ci/snippets/basop/update-scripts-repo.sh + fi + - bash "${CI_PROJECT_DIR}"/ivas-codec-ci/snippets/activate-Werror-linux.sh + - bash ci/build_codec_sanitizers_linux.sh + # ensure the codec builds with DEBUGGING activated # warnings are not treated as errors here build-codec-linux-debugging-make: @@ -17,3 +115,23 @@ build-codec-linux-debugging-make: - bash "${CI_PROJECT_DIR}"/ivas-codec-ci/snippets/print-common-info.sh - bash "${CI_PROJECT_DIR}"/ivas-codec-ci/snippets/enable-debugging-macro.sh - make -j + +build-codec-windows-msbuild: + rules: + - if: $MANUAL_PIPELINE_TYPE == 'long-term-logs' || $MANUAL_PIPELINE_TYPE == 'backup-long-term-logs' || $MANUAL_PIPELINE_TYPE == 'ivas-conformance-linux' || $UPDATE_PAGES || $MANUAL_PIPELINE_TYPE == 'complexity' || $MEASURE_COMPLEXITY_LINUX + when: never + - if: $CI_PIPELINE_SOURCE == 'web' + - if: $CI_PIPELINE_SOURCE == 'push' && $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH + - if: $CI_PIPELINE_SOURCE == 'merge_request_event' + - if: $CI_PIPELINE_SOURCE == 'schedule' + - if: $CI_PIPELINE_SOURCE == 'push' + when: never + extends: + - .build-job-windows + timeout: "10 minutes" + tags: + - ivas-windows + script: + - powershell $env:CI_PROJECT_DIR\ivas-codec-ci\snippets\print-common-info-windows.ps1 + - powershell $env:CI_PROJECT_DIR\ivas-codec-ci\snippets\activate-WX-windows.ps1 + - MSBuild.exe -maxcpucount .\Workspace_msvc\Workspace_msvc.sln /property:Configuration=Debug diff --git a/includes/default-variables-basop.yml b/includes/default-variables-basop.yml deleted file mode 100644 index 76ac9679c3152622f305b8f7bd8bd96b3d902f80..0000000000000000000000000000000000000000 --- a/includes/default-variables-basop.yml +++ /dev/null @@ -1,99 +0,0 @@ -variables: - EVS_BE_TEST_DIR_BASOP: "/usr/local/be_2_evs_basop" - # TODO: cleanup and use global var defined in ivas-codec-ci - FLOAT_REF_BRANCH_MERGE_SOURCE: "" - SCRIPTS_DIR: "/usr/local/scripts" - LONG_TEST_SUITE: "tests/codec_be_on_mr_nonselection tests/renderer --param_file scripts/config/self_test_ltv.prm --use_ltv" - LONG_TEST_SUITE_NO_RENDERER: "tests/codec_be_on_mr_nonselection --param_file scripts/config/self_test_ltv.prm --use_ltv" - SHORT_TEST_SUITE: "tests/codec_be_on_mr_nonselection" - SHORT_TEST_SUITE_ENCODER: "tests/codec_be_on_mr_nonselection/test_param_file.py --param_file scripts/config/self_test_basop_encoder.prm" - LONG_TEST_SUITE_ENCODER: "tests/codec_be_on_mr_nonselection/test_param_file.py --param_file scripts/config/self_test_ltv_basop_encoder.prm" - TEST_SUITE: "" - # These path variables are used by the pytest calls. - # They can be overwritten in the job templates to e.g. only test encoder or decoder in the chain - DUT_ENCODER_PATH: "./IVAS_cod_dut" - DUT_DECODER_PATH: "./IVAS_dec_dut" - DUT_RENDERER_PATH: "./IVAS_rend_dut" - DUT_POST_RENDERER_PATH: "./ISAR_post_rend_dut" - REF_ENCODER_PATH: "./IVAS_cod_ref" - REF_DECODER_PATH: "./IVAS_dec_ref" - REF_RENDERER_PATH: "./IVAS_rend_ref" - REF_POST_RENDERER_PATH: "./ISAR_post_rend_ref" - MERGE_TARGET_ENCODER_PATH: "./IVAS_cod_merge_target" - MERGE_TARGET_DECODER_PATH: "./IVAS_dec_merge_target" - MERGE_TARGET_RENDERER_PATH: "./IVAS_rend_merge_target" - MERGE_TARGET_POST_RENDERER_PATH: "./ISAR_post_rend_merge_target" - MERGE_SOURCE_FLOAT_REF_ENCODER_PATH: "./IVAS_cod_merge_source_float_ref" - MERGE_SOURCE_FLOAT_REF_DECODER_PATH: "./IVAS_dec_merge_source_float_ref" - MERGE_SOURCE_FLOAT_REF_RENDERER_PATH: "./IVAS_rend_merge_source_float_ref" - MERGE_SOURCE_FLOAT_REF_POST_RENDERER_PATH: "./ISAR_post_rend_merge_source_float_ref" - # These path variables are used for building the binaries - # They should never be overwritten! - DUT_ENCODER_PATH_FOR_BUILD_DO_NOT_MODIFY: "./IVAS_cod_dut" - DUT_DECODER_PATH_FOR_BUILD_DO_NOT_MODIFY: "./IVAS_dec_dut" - DUT_RENDERER_PATH_FOR_BUILD_DO_NOT_MODIFY: "./IVAS_rend_dut" - DUT_POST_RENDERER_PATH_FOR_BUILD_DO_NOT_MODIFY: "./ISAR_post_rend_dut" - REF_ENCODER_PATH_FOR_BUILD_DO_NOT_MODIFY: "./IVAS_cod_ref" - REF_DECODER_PATH_FOR_BUILD_DO_NOT_MODIFY: "./IVAS_dec_ref" - REF_RENDERER_PATH_FOR_BUILD_DO_NOT_MODIFY: "./IVAS_rend_ref" - REF_POST_RENDERER_PATH_FOR_BUILD_DO_NOT_MODIFY: "./ISAR_post_rend_ref" - MERGE_TARGET_ENCODER_PATH_FOR_BUILD_DO_NOT_MODIFY: "./IVAS_cod_merge_target" - MERGE_TARGET_DECODER_PATH_FOR_BUILD_DO_NOT_MODIFY: "./IVAS_dec_merge_target" - MERGE_TARGET_RENDERER_PATH_FOR_BUILD_DO_NOT_MODIFY: "./IVAS_rend_merge_target" - MERGE_TARGET_POST_RENDERER_PATH_FOR_BUILD_DO_NOT_MODIFY: "./ISAR_post_rend_merge_target" - MERGE_SOURCE_FLOAT_REF_ENCODER_PATH_FOR_BUILD_DO_NOT_MODIFY: "./IVAS_cod_merge_source_float_ref" - MERGE_SOURCE_FLOAT_REF_DECODER_PATH_FOR_BUILD_DO_NOT_MODIFY: "./IVAS_dec_merge_source_float_ref" - MERGE_SOURCE_FLOAT_REF_RENDERER_PATH_FOR_BUILD_DO_NOT_MODIFY: "./IVAS_rend_merge_source_float_ref" - MERGE_SOURCE_FLOAT_REF_POST_RENDERER_PATH_FOR_BUILD_DO_NOT_MODIFY: "./ISAR_post_rend_merge_source_float_ref" - LEVEL_SCALING: "1.0" - PRM_FILES: "scripts/config/self_test.prm scripts/config/self_test_ltv.prm" - TESTCASE_TIMEOUT_STV: 900 - TESTCASE_TIMEOUT_LTV: 2400 - TESTCASE_TIMEOUT_LTV_SANITIZERS: 10800 - CI_REGRESSION_THRESH_MLD: "0.1" - CI_REGRESSION_THRESH_MAX_ABS_DIFF: "50" - CI_REGRESSION_THRESH_SSNR: "-1" - CI_REGRESSION_THRESH_ODG: "-0.05" - INSTR_DIR: "scripts/c-code_instrument" - BUILD_WITH_DEBUG_MODE_INFO: "" - ENCODER_TEST: "" - DELTA_ODG: "" - COMPARE_DMX: "" - SPLIT_COMPARISON: "" - SKIP_REGRESSION_CHECK: "" - FAILED_TESTCASES_LIST: "failed-testcases.txt" - ERRORS_TESTCASES_LIST: "errors-testcases.txt" - PYTEST_CACHE_ARTIFACT: "pytest_cache.zip" - MEASURES_FOR_REPORT: "MLD MAX_ABS_DIFF MIN_SSNR MIN_ODG" - FLOAT_REF_COMMIT_FILE: "float-ref-git-sha.txt" - CUT_COMMIT_FILE: "CuT-git-sha.txt" - MERGE_TARGET_COMMIT_FILE: "merge-target-git-sha.txt" - MERGE_SOURCE_FLOAT_REF_COMMIT_FILE: "merge-source-float-ref-git-sha.txt" - RUNNER_TAG: "ivas-linux" - LOGS_BACKUP_SOURCE_DIR: "" - LOGS_BACKUP_TARGET_DIR: "" - # set this to true to skip the external HRTF testcases in pytest calls - DISABLE_HRTF: "false" - MANUAL_PIPELINE_TYPE: - description: "Type for the manual pipeline run. Use 'pytest-compare' to run comparison test against reference float codec." - value: 'default' - options: - - 'default' - - 'pytest-compare' - - 'pytest-compare-enc-dmx' - - 'pytest-compare-long' - - 'pytest-compare-long-fx-fx' - - 'pytest-compare-to-input' - - 'pytest-saturation-smoke-test' - - 'evs-26444' - - 'sanitizer' - - 'pytest-renderer' - - 'complexity' - - 'coverage' - - 'voip-be-test' - - 'renderer-framesize-be' - - 'peaq-enc-passthrough' - - 'long-term-logs' - - 'backup-long-term-logs' - - 'test-long-self-test' - - 'ivas-conformance-linux' diff --git a/includes/default-variables-float.yml b/includes/default-variables-float.yml deleted file mode 100644 index 9baec3b3458dec5a232da1d01a0282392fecbf36..0000000000000000000000000000000000000000 --- a/includes/default-variables-float.yml +++ /dev/null @@ -1,34 +0,0 @@ -variables: - EVS_BE_TEST_DIR: "/usr/local/be_2_evs_test" - EVS_BE_WIN_TEST_DIR: "C:/Users/gitlab-runner/testvec" - PROCESSING_SCRIPTS_BIN_DIR: "/test-bin" - SANITIZER_TESTS: "CLANG1 CLANG2 CLANG3" - OUT_FORMATS_CHANNEL_BASED: "stereo mono 5_1 5_1_2 5_1_4 7_1 7_1_4" - OUT_FORMATS_SCENE_BASED: "FOA HOA2 HOA3" - OUT_FORMATS_BINAURAL: "BINAURAL BINAURAL_ROOM_IR BINAURAL_ROOM_REVERB" - OUT_FORMATS_ALL: "$OUT_FORMATS_CHANNEL_BASED $OUT_FORMATS_SCENE_BASED $OUT_FORMATS_BINAURAL EXT" - MANUAL_PIPELINE_TYPE: - description: "Type for the manual pipeline run. Use 'test-be-release' to run BE test against release codec." - value: 'default' - options: - - 'default' - - 'test-be-release' - - 'test-long-self-test' - - 'ivas-conformance' - - 'ivas-conformance-linux' - - 'check-clipping' - - 'test-branch-vs-input-passthrough' - - 'coverage' - - 'complexity' - - PYTEST_ARGS: "" - LONG_TEST_SUITE: "tests/codec_be_on_mr_nonselection tests/renderer --param_file scripts/config/self_test_ltv.prm --use_ltv" - LONG_TEST_SUITE_NO_RENDERER: "tests/codec_be_on_mr_nonselection --param_file scripts/config/self_test_ltv.prm --use_ltv" - SHORT_TEST_SUITE: "tests/codec_be_on_mr_nonselection" - SHORT_TEST_SUITE_ENCODER: "tests/codec_be_on_mr_nonselection/test_param_file.py --param_file scripts/config/self_test_basop_encoder.prm" - LONG_TEST_SUITE_ENCODER: "tests/codec_be_on_mr_nonselection/test_param_file.py --param_file scripts/config/self_test_ltv_basop_encoder.prm" - TEST_SUITE: "" - COVERAGE_OUTPUT_FILE_STV: "coverage-stv.info" - COVERAGE_OUTPUT_FILE_LTV: "coverage-ltv.info" - COVERAGE_OUTPUT_FILE_CONFORMANCE: "coverage-conformance.info" - COVERAGE_OUTPUT_FILE_MERGED: "coverage-merged.info" diff --git a/includes/default-variables.yml b/includes/default-variables.yml index d72bce41107a4e6ce82a6a59ad58c59b73804ede..6a2a5ad680be309c3cecc819f48a0b8e6eaf1e72 100644 --- a/includes/default-variables.yml +++ b/includes/default-variables.yml @@ -17,3 +17,4 @@ variables: DISABLE_HRTF: "false" PROJECT_ID_FLOAT: "49" PROJECT_ID_BASOP: "77" + INSTR_DIR: "scripts/c-code_instrument" diff --git a/includes/job-templates.yml b/includes/job-templates.yml index 031f613ca77a44f8d98771bdab606448883e54fd..8ec7b0331d652f95ef451b2c7fff30e5303b383a 100644 --- a/includes/job-templates.yml +++ b/includes/job-templates.yml @@ -19,10 +19,10 @@ - ivas-windows before_script: - | - if (-not $env:IVAS_CODEC_CI_REF) { echo "Error: IVAS_CODEC_CI_REF not set" ; exit 1 } + if (-not $env:IVAS_CODEC_CI_REF) { echo "Error: IVAS_CODEC_CI_REF not set" ; exit 1 } else { echo "IVAS_CODEC_CI_REF set to: $env:IVAS_CODEC_CI_REF" } # note: git clone -b does not work with SHA refs, hence using the safe way of a full clone and git switch # also, we cannot use && with powershell < 7 unfortunately, hence we cannot reliably detect if the snippet fails - - git clone $env:CI_SERVER_URL/ivas-codec-pc/ivas-codec-ci.git ivas-codec-ci; pushd ivas-codec-ci ; git switch --detach $env:IVAS_CODEC_CI_REF ; popd + - git clone $env:CI_SERVER_URL/ivas-codec-pc/ivas-codec-ci.git ivas-codec-ci; pushd ivas-codec-ci ; git checkout $env:IVAS_CODEC_CI_REF ; popd .build-job-linux: stage: build diff --git a/includes/pre.yml b/includes/pre.yml new file mode 100644 index 0000000000000000000000000000000000000000..640d659b10b550b2d6ad6c7fd079f6cba09bf14b --- /dev/null +++ b/includes/pre.yml @@ -0,0 +1,21 @@ +# --------------------------------------------------------------- +# .pre jobs for setting up things +# --------------------------------------------------------------- + +# See: https://gitlab.com/gitlab-org/gitlab/-/issues/194023 +# Solution to make main branch pipelines uninterruptible while all other +# pipelines can be interrupted by default. This works because all jobs +# after uninterruptible jobs will be uninterruptible. Resource group +# setting avoids rare case where two fast merges could still interrupt +# pipeline. This should be revisited if there are updates to Gitlab. +uninterruptible: + stage: .pre + interruptible: false + resource_group: uninterruptible + script: + - echo "$CI_COMMIT_BRANCH is uninterruptible" + rules: + - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH + when: always + tags: + - ivas-linux diff --git a/includes/rules-basop.yml b/includes/rules-basop.yml deleted file mode 100644 index b17e44ad32f37b8a6ecbfed6962b03bbe2bf36c9..0000000000000000000000000000000000000000 --- a/includes/rules-basop.yml +++ /dev/null @@ -1,56 +0,0 @@ -# overwrites the default rules -# should be refactored and unified -.rules-basis: - rules: - # see https://docs.gitlab.com/ee/ci/yaml/workflow.html#switch-between-branch-pipelines-and-merge-request-pipelines - - if: $CI_COMMIT_BRANCH && $CI_OPEN_MERGE_REQUESTS && $CI_PIPELINE_SOURCE == "push" - when: never - - if: $CI_PIPELINE_SOURCE == 'merge_request_event' - variables: - IVAS_PIPELINE_NAME: 'MR pipeline: $CI_MERGE_REQUEST_SOURCE_BRANCH_NAME' - ### disabled for now because pipeline cd is redundant with MR pipeline with current workflow - # - if: $CI_PIPELINE_SOURCE == 'push' && $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH # Pushes to main - # variables: - # IVAS_PIPELINE_NAME: 'Push pipeline: $CI_COMMIT_BRANCH' - - if: $CI_PIPELINE_SOURCE == 'web' && $MANUAL_PIPELINE_TYPE == 'default' # for testing - variables: - IVAS_PIPELINE_NAME: 'Web run pipeline: $CI_COMMIT_BRANCH' - - if: $CI_PIPELINE_SOURCE == 'web' && $MANUAL_PIPELINE_TYPE == 'pytest-compare' - variables: - IVAS_PIPELINE_NAME: 'Run comparison tools against float ref: $CI_COMMIT_BRANCH' - - if: $CI_PIPELINE_SOURCE == 'web' && $MANUAL_PIPELINE_TYPE == 'pytest-compare-enc-dmx' - variables: - IVAS_PIPELINE_NAME: 'Run encoder dmx comparison against float ref: $CI_COMMIT_BRANCH' - - if: $CI_PIPELINE_SOURCE == 'web' && $MANUAL_PIPELINE_TYPE == 'pytest-compare-long' - variables: - IVAS_PIPELINE_NAME: 'Run comparison tools against float ref (long test vectors): $CI_COMMIT_BRANCH' - - if: $CI_PIPELINE_SOURCE == 'web' && $MANUAL_PIPELINE_TYPE == 'pytest-compare-to-input' - variables: - IVAS_PIPELINE_NAME: 'Run comparison tools against input (pass-through only): $CI_COMMIT_BRANCH' - - if: $CI_PIPELINE_SOURCE == 'web' && $MANUAL_PIPELINE_TYPE == 'pytest-saturation-smoke-test' - variables: - IVAS_PIPELINE_NAME: 'Run saturation smoke-test: $CI_COMMIT_BRANCH' - - if: $CI_PIPELINE_SOURCE == 'web' && $MANUAL_PIPELINE_TYPE == 'evs-26444' - variables: - IVAS_PIPELINE_NAME: 'EVS 26.444 test: $CI_COMMIT_BRANCH' - - if: $CI_PIPELINE_SOURCE == 'web' && $MANUAL_PIPELINE_TYPE == 'sanitizer' - variables: - IVAS_PIPELINE_NAME: 'Short testvectors sanitizers' - - if: $CI_PIPELINE_SOURCE == 'web' && $MANUAL_PIPELINE_TYPE == 'pytest-renderer' - variables: - IVAS_PIPELINE_NAME: 'Renderer test: $CI_COMMIT_BRANCH' - - if: $CI_PIPELINE_SOURCE == 'web' && $MANUAL_PIPELINE_TYPE == 'complexity' - variables: - IVAS_PIPELINE_NAME: 'Complexity Measurement on $CI_COMMIT_BRANCH' - - if: $CI_PIPELINE_SOURCE == 'web' && $MANUAL_PIPELINE_TYPE == 'voip-be-test' - variables: - IVAS_PIPELINE_NAME: 'Voip BE test on $CI_COMMIT_BRANCH' - - if: $CI_PIPELINE_SOURCE == 'web' && $MANUAL_PIPELINE_TYPE == 'peaq-enc-passthrough' - variables: - IVAS_PIPELINE_NAME: 'PEAQ encoder pass-through test: $CI_COMMIT_BRANCH' - - if: $CI_PIPELINE_SOURCE == 'web' && $MANUAL_PIPELINE_TYPE == 'ivas-conformance' - variables: - IVAS_PIPELINE_NAME: 'IVAS conformance: $CI_COMMIT_BRANCH' - - if: $CI_PIPELINE_SOURCE == 'schedule' # Scheduled in any branch - variables: - IVAS_PIPELINE_NAME: 'Scheduled pipeline: $CI_COMMIT_BRANCH' diff --git a/includes/rules.yml b/includes/rules.yml deleted file mode 100644 index 6e652788ad7521fc17519cf548d9742164e60260..0000000000000000000000000000000000000000 --- a/includes/rules.yml +++ /dev/null @@ -1,42 +0,0 @@ -# templates for rules -.rules-basis: - rules: - - if: $MIRROR_ACCESS_TOKEN # Don't run in the mirror update pipeline (only then MIRROR_ACCESS_TOKEN is defined) - when: never - - if: $CI_PIPELINE_SOURCE == 'schedule' # Don't run in any scheduled pipelines by default (use schedule templates below to enable again for certain conditions) - when: never - - if: $CI_PIPELINE_SOURCE == 'trigger' # Don't run triggered pipeline by default - when: never - - if: $MANUAL_PIPELINE_TYPE == 'test-be-release' # Skip all the normal jobs when testing manually against release codec - when: never - - if: $MANUAL_PIPELINE_TYPE == 'test-long-self-test' # Skip all the normal jobs when testing manually against release codec - when: never - - if: $MANUAL_PIPELINE_TYPE == 'ivas-conformance' - when: never - - if: $MANUAL_PIPELINE_TYPE == 'ivas-conformance-linux' - when: never - - if: $MANUAL_PIPELINE_TYPE == 'check-clipping' - when: never - - if: $MANUAL_PIPELINE_TYPE == 'test-branch-vs-input-passthrough' - when: never - - if: $CI_PIPELINE_SOURCE == 'web' && $MANUAL_PIPELINE_TYPE == 'complexity' - variables: - IVAS_PIPELINE_NAME: 'Complexity Measurement on $CI_COMMIT_BRANCH' - - when: on_success - -.rules-merge-request-to-main: - extends: .rules-basis - rules: - - if: $CI_PIPELINE_SOURCE == 'merge_request_event' && $CI_MERGE_REQUEST_TARGET_BRANCH_NAME == "main" - - if: $CI_PIPELINE_SOURCE == 'push' - when: never - -.rules-main-push: - extends: .rules-basis - rules: - - if: $CI_PIPELINE_SOURCE == 'push' && $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH - -.rules-main-scheduled: - extends: .rules-basis - rules: - - if: $CI_PIPELINE_SOURCE == 'schedule' && $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH diff --git a/includes/smoke-test.yml b/includes/smoke-test.yml new file mode 100644 index 0000000000000000000000000000000000000000..897838ce119d9cf0186b93dd3c762c835a3c0884 --- /dev/null +++ b/includes/smoke-test.yml @@ -0,0 +1,28 @@ +# test that runs all modes with 1s input signals +.smoke-test-template: + extends: + - .test-job-linux-needs-testv-dir + tags: + - ivas-linux-fast + before_script: + - !reference [.test-job-linux-needs-testv-dir, before_script] + - bash "${CI_PROJECT_DIR}"/ivas-codec-ci/snippets/print-common-info.sh + # LTV update needed as ltv ISM metadata files are used + - bash "${CI_PROJECT_DIR}"/ivas-codec-ci/snippets/update-ltv-repo.sh + after_script: + ### analyze for failures + - if ! [ -s smoke_test_output.txt ] || ! [ -s smoke_test_output_jbm.txt ] || ! [ -s smoke_test_output_hrtf.txt ]; then echo "Error in smoke test"; exit 1; fi + - ret_val=0 + - if cat smoke_test_output.txt | grep -c "failed"; then echo "Smoke test without JBM failed"; ret_val=1; fi + - if cat smoke_test_output_jbm.txt | grep -c "failed"; then echo "Smoke test JBM part failed"; ret_val=1; fi + - if cat smoke_test_output_hrtf.txt | grep -c "failed"; then echo "Smoke test with external hrtf files failed"; ret_val=1; fi + - exit $ret_val + artifacts: + name: "mr-$CI_MERGE_REQUEST_IID--sha-$CI_COMMIT_SHORT_SHA--job-$CI_JOB_NAME--results" + expire_in: 1 week + when: always + paths: + - smoke_test_output.txt + - smoke_test_output_jbm.txt + - smoke_test_output_hrtf.txt + expose_as: "Smoke test results" diff --git a/main-basop.yml b/main-basop.yml index 3c168bc2a75249b5ca15e547399a2032a6485c4d..cf54cf2308f08665211e7eeeb4f8f43ed50027ac 100644 --- a/main-basop.yml +++ b/main-basop.yml @@ -1,24 +1,130 @@ include: - - local: main.yml - - local: includes/default-variables-basop.yml - - local: includes/rules-basop.yml + - local: includes/pre.yml + - local: includes/default-variables.yml + - local: includes/job-templates.yml - local: includes/complexity-jobs.yml + - local: includes/smoke-test.yml - local: includes/build-jobs.yml +default: + # by default, we want all jobs to be interruptible, and we need to configure this explicitly + interruptible: true + stages: - .pre - maintenance - prevalidate - build - # only in basop - compare - # only in basop - check-be - test - postvalidate - deploy - +variables: + EVS_BE_TEST_DIR_BASOP: "/usr/local/be_2_evs_basop" + FLOAT_REF_BRANCH_MERGE_SOURCE: "" + SCRIPTS_DIR: "/usr/local/scripts" + LONG_TEST_SUITE: "tests/codec_be_on_mr_nonselection tests/renderer --param_file scripts/config/self_test_ltv.prm --use_ltv" + LONG_TEST_SUITE_NO_RENDERER: "tests/codec_be_on_mr_nonselection --param_file scripts/config/self_test_ltv.prm --use_ltv" + SHORT_TEST_SUITE: "tests/codec_be_on_mr_nonselection" + SHORT_TEST_SUITE_ENCODER: "tests/codec_be_on_mr_nonselection/test_param_file.py --param_file scripts/config/self_test_basop_encoder.prm" + LONG_TEST_SUITE_ENCODER: "tests/codec_be_on_mr_nonselection/test_param_file.py --param_file scripts/config/self_test_ltv_basop_encoder.prm" + TEST_SUITE: "" + ### These path variables are used by the pytest calls. + ### They can be overwritten in the job definitions in the variables: section to e.g. only test encoder or decoder in the chain + ### E.g. a job that tests the BASOP encoder against the float ref via MLD etc. would overwrite DUT_DECODER_PATH with REF_DECODER_PATH in its variables: section + # "DUT_" - used in merge request pipelines for the branch build (Device Under Test) + DUT_ENCODER_PATH: "./IVAS_cod_dut" + DUT_DECODER_PATH: "./IVAS_dec_dut" + DUT_RENDERER_PATH: "./IVAS_rend_dut" + DUT_POST_RENDERER_PATH: "./ISAR_post_rend_dut" + # "REF_" - used in merge request regression jobs for the floating-point ref (ivas-codec main) + REF_ENCODER_PATH: "./IVAS_cod_ref" + REF_DECODER_PATH: "./IVAS_dec_ref" + REF_RENDERER_PATH: "./IVAS_rend_ref" + REF_POST_RENDERER_PATH: "./ISAR_post_rend_ref" + # "MERGE_TARGET_" - used in merge request jobs to denote the build from the target branch (mostly main) + MERGE_TARGET_ENCODER_PATH: "./IVAS_cod_merge_target" + MERGE_TARGET_DECODER_PATH: "./IVAS_dec_merge_target" + MERGE_TARGET_RENDERER_PATH: "./IVAS_rend_merge_target" + MERGE_TARGET_POST_RENDERER_PATH: "./ISAR_post_rend_merge_target" + # "MERGE_SOURCE_FLOAT_REF_" - used for the respective float reference branch of an MR. Can be main or the respective companion branch in ivas-codec repo + MERGE_SOURCE_FLOAT_REF_ENCODER_PATH: "./IVAS_cod_merge_source_float_ref" + MERGE_SOURCE_FLOAT_REF_DECODER_PATH: "./IVAS_dec_merge_source_float_ref" + MERGE_SOURCE_FLOAT_REF_RENDERER_PATH: "./IVAS_rend_merge_source_float_ref" + MERGE_SOURCE_FLOAT_REF_POST_RENDERER_PATH: "./ISAR_post_rend_merge_source_float_ref" + ### These path variables are used for building the binaries + ### They should never be overwritten! + DUT_ENCODER_PATH_FOR_BUILD_DO_NOT_MODIFY: "./IVAS_cod_dut" + DUT_DECODER_PATH_FOR_BUILD_DO_NOT_MODIFY: "./IVAS_dec_dut" + DUT_RENDERER_PATH_FOR_BUILD_DO_NOT_MODIFY: "./IVAS_rend_dut" + DUT_POST_RENDERER_PATH_FOR_BUILD_DO_NOT_MODIFY: "./ISAR_post_rend_dut" + REF_ENCODER_PATH_FOR_BUILD_DO_NOT_MODIFY: "./IVAS_cod_ref" + REF_DECODER_PATH_FOR_BUILD_DO_NOT_MODIFY: "./IVAS_dec_ref" + REF_RENDERER_PATH_FOR_BUILD_DO_NOT_MODIFY: "./IVAS_rend_ref" + REF_POST_RENDERER_PATH_FOR_BUILD_DO_NOT_MODIFY: "./ISAR_post_rend_ref" + MERGE_TARGET_ENCODER_PATH_FOR_BUILD_DO_NOT_MODIFY: "./IVAS_cod_merge_target" + MERGE_TARGET_DECODER_PATH_FOR_BUILD_DO_NOT_MODIFY: "./IVAS_dec_merge_target" + MERGE_TARGET_RENDERER_PATH_FOR_BUILD_DO_NOT_MODIFY: "./IVAS_rend_merge_target" + MERGE_TARGET_POST_RENDERER_PATH_FOR_BUILD_DO_NOT_MODIFY: "./ISAR_post_rend_merge_target" + MERGE_SOURCE_FLOAT_REF_ENCODER_PATH_FOR_BUILD_DO_NOT_MODIFY: "./IVAS_cod_merge_source_float_ref" + MERGE_SOURCE_FLOAT_REF_DECODER_PATH_FOR_BUILD_DO_NOT_MODIFY: "./IVAS_dec_merge_source_float_ref" + MERGE_SOURCE_FLOAT_REF_RENDERER_PATH_FOR_BUILD_DO_NOT_MODIFY: "./IVAS_rend_merge_source_float_ref" + MERGE_SOURCE_FLOAT_REF_POST_RENDERER_PATH_FOR_BUILD_DO_NOT_MODIFY: "./ISAR_post_rend_merge_source_float_ref" + LEVEL_SCALING: "1.0" + PRM_FILES: "scripts/config/self_test.prm scripts/config/self_test_ltv.prm" + TESTCASE_TIMEOUT_STV: 900 + TESTCASE_TIMEOUT_LTV: 2400 + TESTCASE_TIMEOUT_LTV_SANITIZERS: 10800 + CI_REGRESSION_THRESH_MLD: "0.1" + CI_REGRESSION_THRESH_MAX_ABS_DIFF: "50" + CI_REGRESSION_THRESH_SSNR: "-1" + CI_REGRESSION_THRESH_ODG: "-0.05" + BUILD_WITH_DEBUG_MODE_INFO: "" + ENCODER_TEST: "" + DELTA_ODG: "" + COMPARE_DMX: "" + SPLIT_COMPARISON: "" + SKIP_REGRESSION_CHECK: "" + FAILED_TESTCASES_LIST: "failed-testcases.txt" + ERRORS_TESTCASES_LIST: "errors-testcases.txt" + PYTEST_CACHE_ARTIFACT: "pytest_cache.zip" + MEASURES_FOR_REPORT: "MLD MAX_ABS_DIFF MIN_SSNR MIN_ODG" + FLOAT_REF_COMMIT_FILE: "float-ref-git-sha.txt" + CUT_COMMIT_FILE: "CuT-git-sha.txt" + MERGE_TARGET_COMMIT_FILE: "merge-target-git-sha.txt" + MERGE_SOURCE_FLOAT_REF_COMMIT_FILE: "merge-source-float-ref-git-sha.txt" + RUNNER_TAG: "ivas-linux" + LOGS_BACKUP_SOURCE_DIR: "" + LOGS_BACKUP_TARGET_DIR: "" + # set this to true to skip the external HRTF testcases in pytest calls + DISABLE_HRTF: "false" + MANUAL_PIPELINE_TYPE: + description: "Type for the manual pipeline run. Use 'pytest-compare' to run comparison test against reference float codec." + value: 'default' + options: + - 'default' + - 'pytest-compare' + - 'pytest-compare-enc-dmx' + - 'pytest-compare-long' + - 'pytest-compare-long-fx-fx' + - 'pytest-compare-to-input' + - 'pytest-saturation-smoke-test' + - 'evs-26444' + - 'sanitizer' + - 'pytest-renderer' + - 'complexity' + - 'coverage' + - 'voip-be-test' + - 'renderer-framesize-be' + - 'peaq-enc-passthrough' + - 'long-term-logs' + - 'backup-long-term-logs' + - 'test-long-self-test' + - 'ivas-conformance-linux' + +# This sets when pipelines are created. Jobs have more specific rules to restrict them. workflow: name: '$IVAS_PIPELINE_NAME' rules: @@ -85,28 +191,69 @@ workflow: IVAS_PIPELINE_NAME: 'Test long self-test against main pipeline: $CI_COMMIT_BRANCH' - if: $CI_PIPELINE_SOURCE == 'web' && $MANUAL_PIPELINE_TYPE == 'ivas-conformance-linux' variables: - IVAS_PIPELINE_NAME: 'IVAS Conformance Linux: $CI_COMMIT_BRANCH' + IVAS_PIPELINE_NAME: 'IVAS Conformance Linux: $CI_COMMIT_BRANCH' - if: $CI_PIPELINE_SOURCE == 'schedule' # Scheduled in any branch variables: - IVAS_PIPELINE_NAME: 'Scheduled pipeline: $CI_COMMIT_BRANCH' + IVAS_PIPELINE_NAME: 'Scheduled pipeline: $CI_COMMIT_BRANCH' # --------------------------------------------------------------- -# Generic script anchors +# Rules templates # --------------------------------------------------------------- -# These can be used later on to do common tasks - -# Prints useful information for every job and should be used at the beginning of each job -.print-common-info-windows: &print-common-info-windows - - | - echo "Printing common information for build job." - echo "Current job is run on commit $CI_COMMIT_SHA" - echo "Commit time was $CI_COMMIT_TIMESTAMP" - ("echo 'System time is'", "Get-Date -Format 'dddd dd/MM/yyyy HH:mm K'") | Invoke-Expression - -.activate-WX-windows: &activate-WX-windows - - (Get-Content -Path "CMakeLists.txt") -replace '# \(add_compile_options\("\/WX"\)\)', '$1' | Set-Content -Path "CMakeLists.txt" - - Get-ChildItem -Path "Workspace_msvc" -Filter "*.vcxproj" | ForEach-Object { (Get-Content -Path $_.FullName) -replace 'false', 'true' | Set-Content -Path $_.FullName } +.rules-basis: + rules: + # see https://docs.gitlab.com/ee/ci/yaml/workflow.html#switch-between-branch-pipelines-and-merge-request-pipelines + - if: $CI_COMMIT_BRANCH && $CI_OPEN_MERGE_REQUESTS && $CI_PIPELINE_SOURCE == "push" + when: never + - if: $CI_PIPELINE_SOURCE == 'merge_request_event' + variables: + IVAS_PIPELINE_NAME: 'MR pipeline: $CI_MERGE_REQUEST_SOURCE_BRANCH_NAME' + ### disabled for now because pipeline cd is redundant with MR pipeline with current workflow + # - if: $CI_PIPELINE_SOURCE == 'push' && $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH # Pushes to main + # variables: + # IVAS_PIPELINE_NAME: 'Push pipeline: $CI_COMMIT_BRANCH' + - if: $CI_PIPELINE_SOURCE == 'web' && $MANUAL_PIPELINE_TYPE == 'default' # for testing + variables: + IVAS_PIPELINE_NAME: 'Web run pipeline: $CI_COMMIT_BRANCH' + - if: $CI_PIPELINE_SOURCE == 'web' && $MANUAL_PIPELINE_TYPE == 'pytest-compare' + variables: + IVAS_PIPELINE_NAME: 'Run comparison tools against float ref: $CI_COMMIT_BRANCH' + - if: $CI_PIPELINE_SOURCE == 'web' && $MANUAL_PIPELINE_TYPE == 'pytest-compare-enc-dmx' + variables: + IVAS_PIPELINE_NAME: 'Run encoder dmx comparison against float ref: $CI_COMMIT_BRANCH' + - if: $CI_PIPELINE_SOURCE == 'web' && $MANUAL_PIPELINE_TYPE == 'pytest-compare-long' + variables: + IVAS_PIPELINE_NAME: 'Run comparison tools against float ref (long test vectors): $CI_COMMIT_BRANCH' + - if: $CI_PIPELINE_SOURCE == 'web' && $MANUAL_PIPELINE_TYPE == 'pytest-compare-to-input' + variables: + IVAS_PIPELINE_NAME: 'Run comparison tools against input (pass-through only): $CI_COMMIT_BRANCH' + - if: $CI_PIPELINE_SOURCE == 'web' && $MANUAL_PIPELINE_TYPE == 'pytest-saturation-smoke-test' + variables: + IVAS_PIPELINE_NAME: 'Run saturation smoke-test: $CI_COMMIT_BRANCH' + - if: $CI_PIPELINE_SOURCE == 'web' && $MANUAL_PIPELINE_TYPE == 'evs-26444' + variables: + IVAS_PIPELINE_NAME: 'EVS 26.444 test: $CI_COMMIT_BRANCH' + - if: $CI_PIPELINE_SOURCE == 'web' && $MANUAL_PIPELINE_TYPE == 'sanitizer' + variables: + IVAS_PIPELINE_NAME: 'Short testvectors sanitizers' + - if: $CI_PIPELINE_SOURCE == 'web' && $MANUAL_PIPELINE_TYPE == 'pytest-renderer' + variables: + IVAS_PIPELINE_NAME: 'Renderer test: $CI_COMMIT_BRANCH' + - if: $CI_PIPELINE_SOURCE == 'web' && $MANUAL_PIPELINE_TYPE == 'complexity' + variables: + IVAS_PIPELINE_NAME: 'Complexity Measurement on $CI_COMMIT_BRANCH' + - if: $CI_PIPELINE_SOURCE == 'web' && $MANUAL_PIPELINE_TYPE == 'voip-be-test' + variables: + IVAS_PIPELINE_NAME: 'Voip BE test on $CI_COMMIT_BRANCH' + - if: $CI_PIPELINE_SOURCE == 'web' && $MANUAL_PIPELINE_TYPE == 'peaq-enc-passthrough' + variables: + IVAS_PIPELINE_NAME: 'PEAQ encoder pass-through test: $CI_COMMIT_BRANCH' + - if: $CI_PIPELINE_SOURCE == 'web' && $MANUAL_PIPELINE_TYPE == 'ivas-conformance' + variables: + IVAS_PIPELINE_NAME: 'IVAS conformance: $CI_COMMIT_BRANCH' + - if: $CI_PIPELINE_SOURCE == 'schedule' # Scheduled in any branch + variables: + IVAS_PIPELINE_NAME: 'Scheduled pipeline: $CI_COMMIT_BRANCH' .rules-merge-request: rules: @@ -131,12 +278,6 @@ workflow: - if: $CI_PIPELINE_SOURCE == 'push' when: never -.rules-merge-request-to-float-ref: - rules: - - if: $CI_PIPELINE_SOURCE == 'merge_request_event' && $CI_MERGE_REQUEST_TARGET_BRANCH_NAME == "ivas-float-update" - - if: $CI_PIPELINE_SOURCE == 'push' - when: never - .rules-merge-request-to-main-no-draft: rules: - if: $CI_MERGE_REQUEST_TITLE =~ /^(\[Draft\]|\(Draft\)|Draft:)/ @@ -145,14 +286,6 @@ workflow: - if: $CI_PIPELINE_SOURCE == 'push' when: never -.rules-merge-request-to-float-ref-no-draft: - rules: - - if: $CI_MERGE_REQUEST_TITLE =~ /^(\[Draft\]|\(Draft\)|Draft:)/ - when: never - - if: $CI_PIPELINE_SOURCE == 'merge_request_event' && $CI_MERGE_REQUEST_TARGET_BRANCH_NAME == "ivas-float-update" - - if: $CI_PIPELINE_SOURCE == 'push' - when: never - .rules-pytest-to-ref-pre: rules: - if: $CI_PIPELINE_SOURCE == 'web' && $MANUAL_PIPELINE_TYPE == "ivas-conformance-linux" @@ -228,11 +361,7 @@ workflow: # Job templates # --------------------------------------------------------------- -# override for centrally defined job to make use of the basop runners -.job-linux: - tags: - - ivas-linux - +# override for centrally defined job to add the before_script section # custom variant of this template, we need to update the scripts repo before every build .test-job-linux: extends: @@ -252,8 +381,7 @@ workflow: .test-job-linux-needs-testv-dir: extends: .test-job-linux before_script: - - !reference [ .job-linux, before_script ] - - bash "${CI_PROJECT_DIR}"/ivas-codec-ci/snippets/basop/update-scripts-repo.sh + - !reference [ .test-job-linux, before_script ] - if [ ! -d "$TESTV_DIR" ]; then mkdir -p $TESTV_DIR; fi - cp -r scripts/testv/* $TESTV_DIR/ @@ -273,11 +401,10 @@ workflow: IMAGES_ARTIFACT_NAME: "images_$CI_JOB_NAME" IMAGES_ARTIFACT_SPLIT: "images_split_$CI_JOB_NAME" script: - - !reference [ .job-linux, before_script ] + - !reference [ .test-job-linux, before_script ] - set -euxo pipefail - bash "${CI_PROJECT_DIR}"/ivas-codec-ci/snippets/print-common-info.sh - - bash "${CI_PROJECT_DIR}"/ivas-codec-ci/snippets/basop/update-scripts-repo.sh - if [ $USE_LTV -eq 1 ]; then - bash "${CI_PROJECT_DIR}"/ivas-codec-ci/snippets/update-ltv-repo.sh - bash "${CI_PROJECT_DIR}"/ivas-codec-ci/snippets/copy-ltv-files-to-testv-dir.sh @@ -419,7 +546,6 @@ workflow: - set -euxo pipefail - - bash "${CI_PROJECT_DIR}"/ivas-codec-ci/snippets/basop/update-scripts-repo.sh - python3 tests/create_short_testvectors.py - if [ $USE_LTV -eq 1 ]; then @@ -545,8 +671,6 @@ workflow: - exit 0 - fi - - bash "${CI_PROJECT_DIR}"/ivas-codec-ci/snippets/basop/update-scripts-repo.sh - - if [ $USE_LTV -eq 1 ]; then - bash "${CI_PROJECT_DIR}"/ivas-codec-ci/snippets/update-ltv-repo.sh - bash "${CI_PROJECT_DIR}"/ivas-codec-ci/snippets/copy-ltv-files-to-testv-dir.sh @@ -680,10 +804,9 @@ workflow: stage: test needs: ["build-codec-linux-make"] script: - - !reference [ .job-linux, before_script ] + - !reference [ .test-job-linux, before_script ] - bash "${CI_PROJECT_DIR}"/ivas-codec-ci/snippets/print-common-info.sh - - bash "${CI_PROJECT_DIR}"/ivas-codec-ci/snippets/basop/update-scripts-repo.sh - bash "${CI_PROJECT_DIR}"/ivas-codec-ci/snippets/copy-ltv-files-to-testv-dir.sh - set -euxo pipefail @@ -716,16 +839,15 @@ workflow: .ivas-pytest-compare-to-input-anchor: &ivas-pytest-compare-to-input-anchor stage: test - needs: ["build-codec-linux-make"] + needs: ["build-codec-linux-make", "pytest-to-ref-pre"] timeout: "360 minutes" variables: SUMMARY_HTML_ARTIFACT_NAME: "summary_$CI_JOB_NAME.html" IMAGES_ARTIFACT_NAME: "images_$CI_JOB_NAME" script: - - !reference [ .job-linux, before_script ] + - !reference [ .test-job-linux, before_script ] - bash "${CI_PROJECT_DIR}"/ivas-codec-ci/snippets/print-common-info.sh - - bash "${CI_PROJECT_DIR}"/ivas-codec-ci/snippets/basop/update-scripts-repo.sh - if [ $USE_LTV -eq 1 ]; then - bash "${CI_PROJECT_DIR}"/ivas-codec-ci/snippets/update-ltv-repo.sh - bash "${CI_PROJECT_DIR}"/ivas-codec-ci/snippets/copy-ltv-files-to-testv-dir.sh @@ -783,28 +905,6 @@ workflow: - report-junit-ref.xml - report-junit-dut.xml -# --------------------------------------------------------------- -# .pre jobs for setting up things -# --------------------------------------------------------------- - -# See: https://gitlab.com/gitlab-org/gitlab/-/issues/194023 -# Solution to make main branch pipelines uninterruptible while all other -# pipelines can be interrupted by default. This works because all jobs -# after uninterruptible jobs will be uninterruptible. Resource group -# setting avoids rare case where two fast merges could still interrupt -# pipeline. This should be revisited if there are updates to Gitlab. -uninterruptible: - stage: .pre - interruptible: false - resource_group: uninterruptible - script: - - echo "$CI_COMMIT_BRANCH is uninterruptible" - rules: - - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH - when: always - tags: - - ivas-linux - # --------------------------------------------------------------- # verification jobs # --------------------------------------------------------------- @@ -889,7 +989,6 @@ clang-format-check: needs: [] timeout: "10 minutes" script: - - bash "${CI_PROJECT_DIR}"/ivas-codec-ci/snippets/basop/update-scripts-repo.sh # Set up variables. This can't be done in the "variables" section because variables are not expanded properly there - PATCH_FILE_NAME="$ARTIFACT_BASE_NAME".patch - > @@ -947,115 +1046,6 @@ pytest-to-ref-pre: - $FLOAT_REF_COMMIT_FILE reports: dotenv: commits.env -# --------------------------------------------------------------- -# Build jobs -# --------------------------------------------------------------- - -# TODO: template rules for the build jobs -# ensure that codec builds on linux -build-codec-linux-make: - rules: - - if: $MANUAL_PIPELINE_TYPE == 'long-term-logs' || $MANUAL_PIPELINE_TYPE == 'ivas-conformance-linux' || $MANUAL_PIPELINE_TYPE == 'backup-long-term-logs' || $UPDATE_PAGES || $MANUAL_PIPELINE_TYPE == 'complexity' || $MEASURE_COMPLEXITY_LINUX - when: never - - if: $CI_PIPELINE_SOURCE == 'web' - - if: $CI_PIPELINE_SOURCE == 'push' && $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH - - if: $CI_PIPELINE_SOURCE == 'merge_request_event' - - if: $CI_PIPELINE_SOURCE == 'schedule' - - if: $CI_PIPELINE_SOURCE == 'push' - when: never - extends: - - .build-job-linux - timeout: "10 minutes" - script: - - bash "${CI_PROJECT_DIR}"/ivas-codec-ci/snippets/print-common-info.sh - - bash "${CI_PROJECT_DIR}"/ivas-codec-ci/snippets/activate-Werror-linux.sh - - make -j - -build-codec-linux-cmake: - rules: - - if: $CI_MERGE_REQUEST_TITLE =~ /^(\[Draft\]|\(Draft\)|Draft:)/ - when: never - - if: $MANUAL_PIPELINE_TYPE == 'long-term-logs' || $MANUAL_PIPELINE_TYPE == 'backup-long-term-logs' || $MANUAL_PIPELINE_TYPE == 'ivas-conformance-linux' || $UPDATE_PAGES - when: never - - if: $CI_PIPELINE_SOURCE == 'push' && $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH - - if: $CI_PIPELINE_SOURCE == 'merge_request_event' - - if: $CI_PIPELINE_SOURCE == 'push' - when: never - extends: - - .build-job-linux - timeout: "10 minutes" - script: - - bash "${CI_PROJECT_DIR}"/ivas-codec-ci/snippets/print-common-info.sh - - bash "${CI_PROJECT_DIR}"/ivas-codec-ci/snippets/basop/update-scripts-repo.sh - - bash "${CI_PROJECT_DIR}"/ivas-codec-ci/snippets/activate-Werror-linux.sh - - cmake -B cmake-build -G "Unix Makefiles" - - cmake --build cmake-build -- -j - -# ensure that codec builds on linux with instrumentation active -build-codec-linux-instrumented-make: - rules: - - if: $CI_MERGE_REQUEST_TITLE =~ /^(\[Draft\]|\(Draft\)|Draft:)/ - when: never - - if: $MANUAL_PIPELINE_TYPE == 'long-term-logs' || $MANUAL_PIPELINE_TYPE == 'backup-long-term-logs' || $UPDATE_PAGES - when: never - - if: $MEASURE_COMPLEXITY_LINUX || $MANUAL_PIPELINE_TYPE == 'complexity' - - if: $CI_PIPELINE_SOURCE == 'push' && $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH - - if: $CI_PIPELINE_SOURCE == 'merge_request_event' - - if: $CI_PIPELINE_SOURCE == 'push' - when: never - extends: - - .build-job-linux - timeout: "12 minutes" - script: - - bash "${CI_PROJECT_DIR}"/ivas-codec-ci/snippets/print-common-info.sh - - bash "${CI_PROJECT_DIR}"/ivas-codec-ci/snippets/basop/update-scripts-repo.sh - - if [ "$CI_MERGE_REQUEST_TARGET_BRANCH_NAME" == "ivas-float-update" ] || [ "$CI_COMMIT_REF_NAME" == "ivas-float-update" ]; then - - bash scripts/prepare_instrumentation.sh - - else - - bash scripts/prepare_instrumentation.sh -m MEM_ONLY - - fi - - make -j -C $INSTR_DIR - -# make sure that the codec builds with msan, asan and usan -build-codec-sanitizers-linux: - rules: - - if: $CI_MERGE_REQUEST_TITLE =~ /^(\[Draft\]|\(Draft\)|Draft:)/ - when: never - - if: $MANUAL_PIPELINE_TYPE == 'long-term-logs' || $MANUAL_PIPELINE_TYPE == 'backup-long-term-logs' || $UPDATE_PAGES - when: never - - if: $MANUAL_PIPELINE_TYPE == 'sanitizer' - - if: $CI_PIPELINE_SOURCE == 'push' && $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH - - if: $CI_PIPELINE_SOURCE == 'merge_request_event' - - if: $CI_PIPELINE_SOURCE == 'push' - when: never - extends: - - .build-job-linux - timeout: "10 minutes" - script: - - bash "${CI_PROJECT_DIR}"/ivas-codec-ci/snippets/print-common-info.sh - - bash "${CI_PROJECT_DIR}"/ivas-codec-ci/snippets/basop/update-scripts-repo.sh - - bash "${CI_PROJECT_DIR}"/ivas-codec-ci/snippets/activate-Werror-linux.sh - - bash ci/build_codec_sanitizers_linux.sh - -build-codec-windows-msbuild: - rules: - - if: $MANUAL_PIPELINE_TYPE == 'long-term-logs' || $MANUAL_PIPELINE_TYPE == 'backup-long-term-logs' || $MANUAL_PIPELINE_TYPE == 'ivas-conformance-linux' || $UPDATE_PAGES || $MANUAL_PIPELINE_TYPE == 'complexity' || $MEASURE_COMPLEXITY_LINUX - when: never - - if: $CI_PIPELINE_SOURCE == 'web' - - if: $CI_PIPELINE_SOURCE == 'push' && $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH - - if: $CI_PIPELINE_SOURCE == 'merge_request_event' - - if: $CI_PIPELINE_SOURCE == 'schedule' - - if: $CI_PIPELINE_SOURCE == 'push' - when: never - extends: - - .build-job-windows - timeout: "10 minutes" - tags: - - ivas-windows - script: - - *print-common-info-windows - - *activate-WX-windows - - MSBuild.exe -maxcpucount .\Workspace_msvc\Workspace_msvc.sln /property:Configuration=Debug # --------------------------------------------------------------- # Test jobs for merge requests @@ -1247,7 +1237,6 @@ renderer-pytest-on-merge-request: REPORT_CSV: "report--$CI_JOB_NAME-$CI_JOB_ID--sha-$CI_COMMIT_SHORT_SHA.csv" script: - bash "${CI_PROJECT_DIR}"/ivas-codec-ci/snippets/print-common-info.sh - - bash "${CI_PROJECT_DIR}"/ivas-codec-ci/snippets/basop/update-scripts-repo.sh - source "${CI_PROJECT_DIR}"/ivas-codec-ci/snippets/basop/build-merge-target-binaries.sh - source "${CI_PROJECT_DIR}"/ivas-codec-ci/snippets/merge-request-comparison-setup-codec.sh - bash "${CI_PROJECT_DIR}"/ivas-codec-ci/snippets/basop/merge-request-print-pinned-commits.sh @@ -1320,7 +1309,6 @@ ivas-pytest-on-merge-request: - ivas-linux script: - bash "${CI_PROJECT_DIR}"/ivas-codec-ci/snippets/print-common-info.sh - - bash "${CI_PROJECT_DIR}"/ivas-codec-ci/snippets/basop/update-scripts-repo.sh - source "${CI_PROJECT_DIR}"/ivas-codec-ci/snippets/basop/build-merge-target-binaries.sh - source "${CI_PROJECT_DIR}"/ivas-codec-ci/snippets/merge-request-comparison-setup-codec.sh - bash "${CI_PROJECT_DIR}"/ivas-codec-ci/snippets/basop/merge-request-print-pinned-commits.sh @@ -1391,7 +1379,6 @@ split-rendering-pytest-on-merge-request: REPORT_CSV: "report--$CI_JOB_NAME-$CI_JOB_ID--sha-$CI_COMMIT_SHORT_SHA.csv" script: - bash "${CI_PROJECT_DIR}"/ivas-codec-ci/snippets/print-common-info.sh - - bash "${CI_PROJECT_DIR}"/ivas-codec-ci/snippets/basop/update-scripts-repo.sh - source "${CI_PROJECT_DIR}"/ivas-codec-ci/snippets/basop/build-merge-target-binaries.sh - source "${CI_PROJECT_DIR}"/ivas-codec-ci/snippets/merge-request-comparison-setup-codec.sh - bash "${CI_PROJECT_DIR}"/ivas-codec-ci/snippets/basop/merge-request-print-pinned-commits.sh @@ -1451,52 +1438,28 @@ split-rendering-pytest-on-merge-request: - $REPORT_XML # test that runs all modes with 1s input signals -codec-smoke-test: +# TODO: activate once everything is fixed in BASOP +.codec-smoke-test-basop: extends: - - .test-job-linux-needs-testv-dir - - .rules-merge-request-to-float-ref-no-draft - timeout: "20 minutes" + - .smoke-test-template + - .rules-merge-request-to-main + timeout: "30 minutes" stage: test - needs: ["build-codec-linux-make"] + needs: + [ + "build-codec-linux-cmake", + "build-codec-linux-make", + ] script: - - bash "${CI_PROJECT_DIR}"/ivas-codec-ci/snippets/print-common-info.sh - # LTV update needed as ltv ISM metadata files are used - - bash "${CI_PROJECT_DIR}"/ivas-codec-ci/snippets/update-ltv-repo.sh - bash "${CI_PROJECT_DIR}"/ivas-codec-ci/snippets/basop/update-scripts-repo.sh - - bash ci/smoke_test.sh || true - - tar cJf logs.tar.xz out/logs/ - - ls -al logs.tar.* - ### analyze for failures - - if ! [ -s smoke_test_output.txt ] || ! [ -s smoke_test_output_jbm.txt ] || ! [ -s smoke_test_output_hrtf.txt ]; then echo "Error in smoke test"; exit 1; fi - - ret_val=0 - - if cat smoke_test_output.txt | grep -c "failed"; then echo "Smoke test without JBM failed"; ret_val=1; fi - - if cat smoke_test_output_jbm.txt | grep -c "failed"; then echo "Smoke test JBM part failed"; ret_val=1; fi - - if cat smoke_test_output_hrtf.txt | grep -c "failed"; then - - if [ "$DISABLE_HRTF" = "false" ]; then - - echo "Smoke test with external hrtf files failed" - - ret_val=1 - - else - - echo "Smoke test with external hrtf files failed, but DISABLE_HRTF is set. Ignoring failure." - - fi - - fi - - exit $ret_val - artifacts: - name: "mr-$CI_MERGE_REQUEST_IID--sha-$CI_COMMIT_SHORT_SHA--job-$CI_JOB_NAME--results" - expire_in: 1 week - when: always - paths: - - smoke_test_output.txt - - smoke_test_output_jbm.txt - - smoke_test_output_hrtf.txt - - logs.tar.xz - expose_as: "Smoke test results" + - bash ci/smoke_test.sh # from float # check for crashes if first received frame on decoder side is an SID -check-first-frame-is-sid: +.check-first-frame-is-sid: extends: - .test-job-linux-needs-testv-dir - - .rules-merge-request-to-float-ref-no-draft + - .rules-merge-request-to-main-no-draft stage: test needs: ["build-codec-linux-make"] script: @@ -1538,10 +1501,10 @@ check-first-frame-is-sid: ### From Float # Check interop IVAS_cod_test -> IVAS_dec_ref -ivas-interop-on-merge-request: +.ivas-interop-on-merge-request: extends: - .test-job-linux-needs-testv-dir - - .rules-merge-request-to-float-ref-no-draft + - .rules-merge-request-to-main-no-draft - .test-job-linux-compares-to-fixed-target stage: test timeout: "20 minutes" @@ -1596,7 +1559,6 @@ split-rendering-smoke-test: stage: test script: - bash "${CI_PROJECT_DIR}"/ivas-codec-ci/snippets/print-common-info.sh - - bash "${CI_PROJECT_DIR}"/ivas-codec-ci/snippets/basop/update-scripts-repo.sh - make -j - testcase_timeout=10 @@ -1620,7 +1582,6 @@ lc3-wrapper-unit-test: stage: test script: - bash "${CI_PROJECT_DIR}"/ivas-codec-ci/snippets/print-common-info.sh - - bash "${CI_PROJECT_DIR}"/ivas-codec-ci/snippets/basop/update-scripts-repo.sh - cmake -B cmake-build -G "Unix Makefiles" -DCOPY_EXECUTABLES_FROM_BUILD_DIR=true - cmake --build cmake-build -- -j @@ -1631,10 +1592,11 @@ lc3-wrapper-unit-test: # from float # test renderer executable -renderer-smoke-test: +# TODO: adapt for BASOP +.renderer-smoke-test: extends: - .test-job-linux - - .rules-merge-request-to-float-ref-no-draft + - .rules-merge-request-to-main-no-draft needs: ["build-codec-linux-make"] stage: test script: @@ -1656,10 +1618,10 @@ renderer-smoke-test: # from float # test renderer executable with cmake + asan -renderer-asan: +# TODO: turn into scheduled job +.renderer-asan: extends: - .test-job-linux - - .rules-merge-request-to-float-ref-no-draft needs: ["build-codec-linux-cmake"] stage: test script: @@ -1683,10 +1645,10 @@ renderer-asan: # from float # test renderer executable with cmake + msan -renderer-msan: +# TODO: turn into scheduled job +.renderer-msan: extends: - .test-job-linux - - .rules-merge-request-to-float-ref-no-draft needs: ["build-codec-linux-cmake"] stage: test tags: @@ -1712,10 +1674,10 @@ renderer-msan: # from float # test renderer executable with cmake + usan -renderer-usan: +# TODO: turn into scheduled job +.renderer-usan: extends: - .test-job-linux - - .rules-merge-request-to-float-ref-no-draft needs: ["build-codec-linux-cmake"] stage: test tags: @@ -1772,7 +1734,6 @@ renderer-usan: # set CLANG_NUM, SELFTEST_SANITY_TIMEOUT and SELF_TEST_PRM_FILE in before_script section .sanitizer-selftest-anchor: &sanitizer-selftest-anchor - bash "${CI_PROJECT_DIR}"/ivas-codec-ci/snippets/print-common-info.sh - - bash "${CI_PROJECT_DIR}"/ivas-codec-ci/snippets/basop/update-scripts-repo.sh - bash "${CI_PROJECT_DIR}"/ivas-codec-ci/snippets/copy-ltv-files-to-testv-dir.sh - testcase_timeout=$SELFTEST_SANITY_TIMEOUT - export UBSAN_OPTIONS=suppressions=scripts/ubsan.supp,report_error_type=1,print_stacktrace=1 @@ -2437,7 +2398,6 @@ voip-be-on-merge-request: - .test-job-linux-needs-testv-dir rules: - if: $CI_PIPELINE_SOURCE == 'web' && $MANUAL_PIPELINE_TYPE == "voip-be-test" - - if: $CI_PIPELINE_SOURCE == 'merge_request_event' && $CI_MERGE_REQUEST_TARGET_BRANCH_NAME == "ivas-float-update" # - if: $CI_PIPELINE_SOURCE == 'merge_request_event' && $CI_MERGE_REQUEST_TARGET_BRANCH_NAME == "main" # only have MR pipelines for MRs to main stage: test needs: ["build-codec-linux-make"] diff --git a/main-float.yml b/main-float.yml index b21622bd39d57f38fbf0606f2f94b7d945f92b5d..5195bbb0c37ff289d2679d9dc152fc7f9a2d3935 100644 --- a/main-float.yml +++ b/main-float.yml @@ -1,9 +1,15 @@ include: - - local: main.yml - - local: includes/default-variables-float.yml + - local: includes/pre.yml + - local: includes/default-variables.yml + - local: includes/job-templates.yml - local: includes/complexity-jobs.yml + - local: includes/smoke-test.yml - local: includes/build-jobs.yml +default: + # by default, we want all jobs to be interruptible, and we need to configure this explicitly + interruptible: true + stages: - .pre - maintenance @@ -15,6 +21,41 @@ stages: - postvalidate - deploy +variables: + EVS_BE_TEST_DIR: "/usr/local/be_2_evs_test" + EVS_BE_WIN_TEST_DIR: "C:/Users/gitlab-runner/testvec" + PROCESSING_SCRIPTS_BIN_DIR: "/test-bin" + SANITIZER_TESTS: "CLANG1 CLANG2 CLANG3" + OUT_FORMATS_CHANNEL_BASED: "stereo mono 5_1 5_1_2 5_1_4 7_1 7_1_4" + OUT_FORMATS_SCENE_BASED: "FOA HOA2 HOA3" + OUT_FORMATS_BINAURAL: "BINAURAL BINAURAL_ROOM_IR BINAURAL_ROOM_REVERB" + OUT_FORMATS_ALL: "$OUT_FORMATS_CHANNEL_BASED $OUT_FORMATS_SCENE_BASED $OUT_FORMATS_BINAURAL EXT" + MANUAL_PIPELINE_TYPE: + description: "Type for the manual pipeline run. Use 'test-be-release' to run BE test against release codec." + value: 'default' + options: + - 'default' + - 'test-be-release' + - 'test-long-self-test' + - 'ivas-conformance' + - 'ivas-conformance-linux' + - 'check-clipping' + - 'test-branch-vs-input-passthrough' + - 'coverage' + - 'complexity' + + PYTEST_ARGS: "" + LONG_TEST_SUITE: "tests/codec_be_on_mr_nonselection tests/renderer --param_file scripts/config/self_test_ltv.prm --use_ltv" + LONG_TEST_SUITE_NO_RENDERER: "tests/codec_be_on_mr_nonselection --param_file scripts/config/self_test_ltv.prm --use_ltv" + SHORT_TEST_SUITE: "tests/codec_be_on_mr_nonselection" + SHORT_TEST_SUITE_ENCODER: "tests/codec_be_on_mr_nonselection/test_param_file.py --param_file scripts/config/self_test_basop_encoder.prm" + LONG_TEST_SUITE_ENCODER: "tests/codec_be_on_mr_nonselection/test_param_file.py --param_file scripts/config/self_test_ltv_basop_encoder.prm" + TEST_SUITE: "" + COVERAGE_OUTPUT_FILE_STV: "coverage-stv.info" + COVERAGE_OUTPUT_FILE_LTV: "coverage-ltv.info" + COVERAGE_OUTPUT_FILE_CONFORMANCE: "coverage-conformance.info" + COVERAGE_OUTPUT_FILE_MERGED: "coverage-merged.info" + # This sets when pipelines are created. Jobs have more specific rules to restrict them. workflow: name: "$IVAS_PIPELINE_NAME" @@ -61,19 +102,58 @@ workflow: variables: IVAS_PIPELINE_NAME: 'Complexity Measurement on $CI_COMMIT_BRANCH' +# --------------------------------------------------------------- +# Rules templates +# --------------------------------------------------------------- + +.rules-basis: + rules: + - if: $MIRROR_ACCESS_TOKEN # Don't run in the mirror update pipeline (only then MIRROR_ACCESS_TOKEN is defined) + when: never + - if: $CI_PIPELINE_SOURCE == 'schedule' # Don't run in any scheduled pipelines by default (use schedule templates below to enable again for certain conditions) + when: never + - if: $CI_PIPELINE_SOURCE == 'trigger' # Don't run triggered pipeline by default + when: never + - if: $MANUAL_PIPELINE_TYPE == 'test-be-release' # Skip all the normal jobs when testing manually against release codec + when: never + - if: $MANUAL_PIPELINE_TYPE == 'test-long-self-test' # Skip all the normal jobs when testing manually against release codec + when: never + - if: $MANUAL_PIPELINE_TYPE == 'ivas-conformance' + when: never + - if: $MANUAL_PIPELINE_TYPE == 'ivas-conformance-linux' + when: never + - if: $MANUAL_PIPELINE_TYPE == 'check-clipping' + when: never + - if: $MANUAL_PIPELINE_TYPE == 'test-branch-vs-input-passthrough' + when: never + - if: $CI_PIPELINE_SOURCE == 'web' && $MANUAL_PIPELINE_TYPE == 'complexity' + variables: + IVAS_PIPELINE_NAME: 'Complexity Measurement on $CI_COMMIT_BRANCH' + - when: on_success + +.rules-merge-request-to-main: + extends: .rules-basis + rules: + - if: $CI_PIPELINE_SOURCE == 'merge_request_event' && $CI_MERGE_REQUEST_TARGET_BRANCH_NAME == "main" + - if: $CI_PIPELINE_SOURCE == 'push' + when: never + +.rules-main-push: + extends: .rules-basis + rules: + - if: $CI_PIPELINE_SOURCE == 'push' && $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH + +.rules-main-scheduled: + extends: .rules-basis + rules: + - if: $CI_PIPELINE_SOURCE == 'schedule' && $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH + # --------------------------------------------------------------- # Generic script anchors # --------------------------------------------------------------- # These can be used later on to do common tasks -.print-common-info-windows: &print-common-info-windows - - | - echo "Printing common information for build job." - echo "Current job is run on commit $CI_COMMIT_SHA" - echo "Commit time was $CI_COMMIT_TIMESTAMP" - ("echo 'System time is'", "Get-Date -Format 'dddd dd/MM/yyyy HH:mm K'") | Invoke-Expression - .merge-request-comparison-setup-codec: &merge-request-comparison-setup-codec ### build test binaries, initial clean for paranoia reasons - bash "${CI_PROJECT_DIR}"/ivas-codec-ci/snippets/disable-debugging-macro.sh @@ -132,10 +212,6 @@ workflow: - cp $LTV_DIR_WIN\*.met scripts\testv - cp $LTV_DIR_WIN\*.csv scripts\testv -.activate-WX-windows: &activate-WX-windows - - (Get-Content -Path "CMakeLists.txt") -replace '# \(add_compile_options\("\/WX"\)\)', '$1' | Set-Content -Path "CMakeLists.txt" - - Get-ChildItem -Path "Workspace_msvc" -Filter "*.vcxproj" | ForEach-Object { (Get-Content -Path $_.FullName) -replace 'false', 'true' | Set-Content -Path $_.FullName } - # to be reused in MR and LTV-scheduled sanitizer test jobs # set CLANG_NUM, SELFTEST_SANITY_TIMEOUT and SELF_TEST_PRM_FILE in before_script section .sanitizer-selftest-anchor: @@ -163,28 +239,6 @@ workflow: - if [ $exit_code20 -ne 0 ] || [ $exit_code10 -ne 0 ] || [ $exit_code5 -ne 0 ]; then exit 1; fi -# --------------------------------------------------------------- -# .pre jobs for setting up things -# --------------------------------------------------------------- - -# See: https://gitlab.com/gitlab-org/gitlab/-/issues/194023 -# Solution to make main branch pipelines uninterruptible while all other -# pipelines can be interrupted by default. This works because all jobs -# after uninterruptible jobs will be uninterruptible. Resource group -# setting avoids rare case where two fast merges could still interrupt -# pipeline. This should be revisited if there are updates to Gitlab. -uninterruptible: - stage: .pre - interruptible: false - resource_group: uninterruptible - script: - - echo "$CI_COMMIT_BRANCH is uninterruptible" - rules: - - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH - when: always - tags: - - ivas-linux - # --------------------------------------------------------------- # Validation jobs # --------------------------------------------------------------- @@ -252,6 +306,7 @@ branch-is-up-to-date-with-main-post: when: never tags: - ivas-linux-fast + needs: [] before_script: - !reference [.job-linux, before_script] - python3 scripts/prepare_combined_format_inputs.py @@ -286,7 +341,6 @@ branch-is-up-to-date-with-main-post: junit: - report-junit.xml -### The stv jobs are run for both main and ivas-float-update basop-compat-stv: extends: - .basop-ci-branch-compat-template @@ -330,120 +384,46 @@ basop-compat-encoder-ltv: script: - exit 0 - -# --------------------------------------------------------------- -# Build jobs -# --------------------------------------------------------------- - -build-codec-linux-make: - extends: - - .build-job-linux - - .rules-basis - script: - - bash "${CI_PROJECT_DIR}"/ivas-codec-ci/snippets/print-common-info.sh - - bash "${CI_PROJECT_DIR}"/ivas-codec-ci/snippets/activate-Werror-linux.sh - - make -j - -build-codec-linux-cmake: - extends: - - .build-job-linux - - .rules-basis - script: - - bash "${CI_PROJECT_DIR}"/ivas-codec-ci/snippets/print-common-info.sh - - bash "${CI_PROJECT_DIR}"/ivas-codec-ci/snippets/activate-Werror-linux.sh - - mkdir build - - cd build - - cmake .. - - cd .. - - make -C build -j - -build-codec-instrumented-linux: - extends: - - .build-job-linux - - .rules-basis - timeout: "10 minutes" - script: - - bash "${CI_PROJECT_DIR}"/ivas-codec-ci/snippets/print-common-info.sh - - ./scripts/prepare_instrumentation.sh - - make -j -C scripts/c-code_instrument - artifacts: - when: always - access: all - expire_in: "30 days" - paths: - - wmc_tool_output.txt - -# make sure that the codec builds with msan, asan and usan -build-codec-sanitizers-linux: - extends: - - .build-job-linux - - .rules-basis - script: - - bash "${CI_PROJECT_DIR}"/ivas-codec-ci/snippets/print-common-info.sh - - bash "${CI_PROJECT_DIR}"/ivas-codec-ci/snippets/activate-Werror-linux.sh - - bash ci/build_codec_sanitizers_linux.sh - -build-codec-windows-cmake: - extends: - - .build-job-windows - - .rules-basis - script: - - *print-common-info-windows - - *activate-WX-windows - - cmake -G "Visual Studio 15 2017" . -Bbuild - - cmake --build build -j - -build-codec-windows-msbuild: - extends: - - .build-job-windows - - .rules-basis - script: - - *print-common-info-windows - - *activate-WX-windows - - MSBuild.exe -maxcpucount .\Workspace_msvc\Workspace_msvc.sln /property:Configuration=Debug - # --------------------------------------------------------------- # Test jobs for merge requests # --------------------------------------------------------------- # test that runs all modes with 1s input signals -codec-smoke-test: +codec-smoke-test-float: extends: - - .test-job-linux-needs-testv-dir + - .smoke-test-template - .rules-merge-request-to-main timeout: "20 minutes" - tags: - - ivas-linux-fast stage: test needs: [ "build-codec-linux-cmake", "build-codec-linux-make", - "build-codec-instrumented-linux", + "build-codec-linux-instrumented-make", "build-codec-sanitizers-linux", ] script: - - bash "${CI_PROJECT_DIR}"/ivas-codec-ci/snippets/print-common-info.sh - # LTV update needed as ltv ISM metadata files are used - - bash "${CI_PROJECT_DIR}"/ivas-codec-ci/snippets/update-ltv-repo.sh + - bash ci/smoke_test.sh - # for MRs to basop-ci-branch-pc, we want to test with the BASOP float reference build - # per default, the smoke test builds the current repo again with WMOPS activated to catch unbalanced instrumentation macros - # TODO: make manual for basop-CI merges - - if [ "$CI_MERGE_REQUEST_TARGET_BRANCH_NAME" = "basop-ci-branch" ]; then - - bash "${CI_PROJECT_DIR}"/ivas-codec-ci/snippets/float/build-binaries-from-basop-repo.sh ivas-float-update - - bash ci/smoke_test.sh coverage - - else - - bash ci/smoke_test.sh - - fi - - ### analyze for failures - - if ! [ -s smoke_test_output.txt ] || ! [ -s smoke_test_output_jbm.txt ] || ! [ -s smoke_test_output_hrtf.txt ]; then echo "Error in smoke test"; exit 1; fi - - ret_val=0 - - if cat smoke_test_output.txt | grep -c "failed"; then echo "Smoke test without JBM failed"; ret_val=1; fi - - if cat smoke_test_output_jbm.txt | grep -c "failed"; then echo "Smoke test JBM part failed"; ret_val=1; fi - - if cat smoke_test_output_hrtf.txt | grep -c "failed"; then echo "Smoke test with external hrtf files failed"; ret_val=1; fi - - exit $ret_val +basop-compat-smoke-test: + extends: + - .smoke-test-template + rules: + - if: $CI_PIPELINE_SOURCE == 'merge_request_event' && $CI_MERGE_REQUEST_TARGET_BRANCH_NAME == "main" + when: manual + - if: $CI_PIPELINE_SOURCE == 'push' + when: never + timeout: "20 minutes" + stage: basop-compat + needs: [] + allow_failure: true + script: + # get basop code and build binaries + - git clone -b main https://forge.3gpp.org/rep/sa4/audio/ivas-basop.git --single-branch --depth 1 + - cp -r scripts tests ci pytest.ini ivas-basop/ + - pushd ivas-basop + - bash ci/smoke_test.sh + - popd artifacts: name: "mr-$CI_MERGE_REQUEST_IID--sha-$CI_COMMIT_SHORT_SHA--job-$CI_JOB_NAME--results" expire_in: 1 week @@ -452,6 +432,7 @@ codec-smoke-test: - smoke_test_output.txt - smoke_test_output_jbm.txt - smoke_test_output_hrtf.txt + - ivas-basop-commit-sha.txt expose_as: "Smoke test results" # code selftest testvectors with memory-sanitizer binaries @@ -504,7 +485,7 @@ pytest-compare-20ms-and-5ms-rendering: [ "build-codec-linux-cmake", "build-codec-linux-make", - "build-codec-instrumented-linux", + "build-codec-linux-instrumented-make", "build-codec-sanitizers-linux", ] script: @@ -831,7 +812,7 @@ ivas-pytest-on-merge-request: - .test-job-linux-needs-testv-dir - .rules-merge-request-to-main stage: compare - needs: ["build-codec-linux-cmake", "codec-smoke-test"] + needs: ["build-codec-linux-cmake", "codec-smoke-test-float"] timeout: "14 minutes" parallel: matrix: @@ -939,7 +920,7 @@ evs-pytest-on-merge-request: - .test-job-linux - .rules-merge-request-to-main stage: compare - needs: ["build-codec-linux-cmake", "codec-smoke-test"] + needs: ["build-codec-linux-cmake", "codec-smoke-test-float"] timeout: "10 minutes" script: - bash "${CI_PROJECT_DIR}"/ivas-codec-ci/snippets/print-common-info.sh @@ -1138,13 +1119,14 @@ check-bitexactness-ext-and-transport-format: be-2-evs-windows: extends: - .rules-merge-request-to-main + - .job-windows tags: - ivas-windows stage: test needs: ["build-codec-windows-msbuild"] timeout: "20 minutes" # To be revisited script: - - *print-common-info-windows + - powershell $CI_PROJECT_DIR\ivas-codec-ci\snippets\print-common-info-windows.ps1 - $winoutdata = $null - MSBuild.exe -maxcpucount .\Workspace_msvc\Workspace_msvc.sln /property:Configuration=Release | tee -variable winoutdata @@ -1298,6 +1280,8 @@ test-be-to-release: expose_as: "test-be-to-release results" ivas-conformance: + extends: + - .job-windows tags: - ivas-windows stage: test @@ -1309,7 +1293,7 @@ ivas-conformance: exit_codes: - 123 script: - - *print-common-info-windows + - powershell $CI_PROJECT_DIR\ivas-codec-ci\snippets\print-common-info-windows.ps1 - MSBuild.exe -maxcpucount .\Workspace_msvc\Workspace_msvc.sln /property:Configuration=Debug - cp -force IVAS_cod.exe IVAS_cod_ref.exe - cp -force IVAS_dec.exe IVAS_dec_ref.exe diff --git a/main.yml b/main.yml deleted file mode 100644 index 9062a0c8b0338dcfd470a71a9d88b70f95b7aa8f..0000000000000000000000000000000000000000 --- a/main.yml +++ /dev/null @@ -1,9 +0,0 @@ -include: - # variables can be overwritten easily if needed - - local: includes/default-variables.yml - - local: includes/rules.yml - - local: includes/job-templates.yml - -default: - # by default, we want all jobs to be interruptible, and we need to configure this explicitly - interruptible: true diff --git a/snippets/activate-WX-windows.ps1 b/snippets/activate-WX-windows.ps1 new file mode 100644 index 0000000000000000000000000000000000000000..a0286487c3ca9aefea336c309f6421b7f5c62ced --- /dev/null +++ b/snippets/activate-WX-windows.ps1 @@ -0,0 +1,2 @@ +(Get-Content -Path "CMakeLists.txt") -replace '# \(add_compile_options\("\/WX"\)\)', '$1' | Set-Content -Path "CMakeLists.txt" +Get-ChildItem -Path "Workspace_msvc" -Filter "*.vcxproj" | ForEach-Object { (Get-Content -Path $_.FullName) -replace 'false', 'true' | Set-Content -Path $_.FullName } diff --git a/snippets/float/build-binaries-from-basop-repo.sh b/snippets/float/build-binaries-from-basop-repo.sh index 9cc0fd03efc42d7e92cad7e38cf8e0bdf82fdb73..f5eb0fe477b77cf34d1b8d1980682b64bdbb3c67 100644 --- a/snippets/float/build-binaries-from-basop-repo.sh +++ b/snippets/float/build-binaries-from-basop-repo.sh @@ -9,14 +9,20 @@ fi branchname=$1 clone_dir="ivas-basop" +commit_file="ivas-basop-commit-sha.txt" cd "${CI_PROJECT_DIR}" git clone -b $branchname https://forge.3gpp.org/rep/sa4/audio/ivas-basop.git --single-branch --depth 1 $clone_dir -make -C $clone_dir -j +pushd $clone_dir +git rev-parse HEAD >"$commit_file" +echo "Building BASOP codec @ $(git rev-parse HEAD)" +make -j +popd cp $clone_dir/IVAS_cod ./ cp $clone_dir/IVAS_dec ./ cp $clone_dir/IVAS_rend ./ cp $clone_dir/ISAR_post_rend ./ +cp $clone_dir/$commit_file ./ diff --git a/snippets/print-common-info-windows.ps1 b/snippets/print-common-info-windows.ps1 new file mode 100644 index 0000000000000000000000000000000000000000..c55144e8db1ab960392fdd964eac18d6f3d36e6f --- /dev/null +++ b/snippets/print-common-info-windows.ps1 @@ -0,0 +1,4 @@ +echo "Printing common information for build job." +echo "Current job is run on commit $CI_COMMIT_SHA" +echo "Commit time was $CI_COMMIT_TIMESTAMP" +("echo 'System time is'", "Get-Date -Format 'dddd dd/MM/yyyy HH:mm K'") | Invoke-Expression