Commit 15461140 authored by Vladimir Malenovsky's avatar Vladimir Malenovsky
Browse files

align filter terminology and ISAR selection docs

parent e9df032c
Loading
Loading
Loading
Loading
Loading
+28 −34
Original line number Diff line number Diff line
@@ -224,26 +224,27 @@ Summary of results:

## Executing specific tests only

Use `--filter` to control test levels, tags or output format filtering.

- Levels:
  - `LEVEL3` (default): no bitrate restrictions.
  - `LEVEL1`: applies the restrictions below.
  - `LEVEL2`: same rules as `LEVEL1`, but with bitrate cap increased from 80 kbps to 192 kbps.
- Tag selection (case-insensitive): `ENC`, `DEC`, `REND`, `ISAR`, `ISAR_ENC`.
- Command substring filtering is case-insensitive.
- Plain substring terms are restrictive and combined using logical AND.
  - Example: `--filter DEC JBM` runs only decoder tests that contain `JBM`.
- `+TERM` adds matching tests to the selection.
  - Example: `--filter LEVEL1 DEC +JBM` adds all `JBM` decoder tests to the default `LEVEL1 DEC` selection.
- `-TERM` removes matching tests from the selection.
  - Example: `--filter DEC -JBM` runs decoder tests except those containing `JBM`.
Use `--filter TOKEN` to control conformance level, test groups, output formats, and substring matching.

- Token types:
  - `LEVEL1`, `LEVEL2`, `LEVEL3`: conformance levels. `LEVEL3` is the default.
  - `ENC`, `DEC`, `REND`, `ISAR`, `ISAR_ENC`: test groups.
  - `MONO`, `STEREO`, `EXT`, `HOA`, `SBA`, `MC`: output-format tokens.
    - `HOA` expands to `HOA2`, `HOA3`.
    - `SBA` expands to `FOA`, `HOA2`, `HOA3`.
    - `MC` expands to `5_1`, `7_1`, `5_1_4`, `5_1_2`, `7_1_4`.
  - Any other token is treated as a case-insensitive substring match.
- Token modifiers:
  - `TOKEN`: restrictive token. Multiple restrictive tokens combine with logical AND.
  - `+TOKEN`: additive token. Adds matching tests even if they would otherwise be excluded.
  - `-TOKEN`: subtractive token. Removes matching tests from the final selection.
  - `TOKEN*`: wildcard token. Matches all known tokens starting with the given prefix.
  - `+TOKEN*`, `-TOKEN*`: wildcard token with add/remove behavior.

### LEVEL1, LEVEL2 and LEVEL3 behavior

For all levels, the default tag baseline is `ENC` + `DEC`.
`REND`, `ISAR`, and `ISAR_ENC` are optional and are only included if explicitly selected
as plain tags or added via `+REND` and/or `+ISAR`.
For all levels, the default test-group baseline is `ENC` + `DEC`.
`REND`, `ISAR`, and `ISAR_ENC` are optional and are only included if explicitly selected as plain test-group tokens or added via `+REND`, `+ISAR`, `+ISAR_ENC`, or `+ISAR*`.

When `--filter LEVEL1` is specified, the following default tests are run:

@@ -253,19 +254,19 @@ When `--filter LEVEL1` is specified, the following default tests are run:
  - `MONO` output format: all bitrates.
  - `STEREO` output format: all bitrates.

- The default `LEVEL1` tests may be restricted by adding more terms (acting as logical AND).
- The default `LEVEL1` tests may be restricted by adding more tokens (acting as logical AND).
  - Example: `--filter LEVEL1 DEC MONO` keeps only `MONO` tests from the LEVEL1-eligible DEC set.
  - Example: `--filter LEVEL1 JBM` keeps all LEVEL1-eligible ENC tests but only JBM tests from the LEVEL1-eligible DEC tests.
- `+TERM` adds tests to the final LEVEL1 selection, even if they would otherwise be restricted.
- `+TOKEN` adds tests to the final LEVEL1 selection, even if they would otherwise be restricted.
  - Example: `--filter LEVEL1 DEC JBM +BINAURAL` runs only JBM-matching LEVEL1 DEC tests and additionally includes DEC tests containing `BINAURAL` keyword, i.e. `BINAURAL`, `BINAURAL_IR`, `BINAURAL_ROOM_IR`, `BINAURAL_REVERB`.
- `-TERM` removes matching tests from the final LEVEL1 selection (including tests added via `+TERM`).
- `-TOKEN` removes matching tests from the final LEVEL1 selection (including tests added via `+TOKEN`).
  - Example: `--filter LEVEL1 DEC +JBM -VOIP` adds JBM-matching DEC tests and then excludes any DEC tests containing the keyword `VOIP`.
- Renderer and ISAR tests are not run by default in `LEVEL1`.
  - Add `+REND` and/or `+ISAR` in `--filter` to include them.
  - If `+ISAR` is provided, both `ISAR` and `ISAR_ENC` test groups are run.
  - Add `+REND`, `+ISAR`, and/or `+ISAR_ENC` in `--filter` to include them.
  - Use `+ISAR*` if you want wildcard expansion across all `ISAR*`-prefixed test-group tokens.

When `--filter LEVEL2` is specified, all selection rules above remain the same,
except bitrate caps are relaxed to 192 kbps:
except the bitrate cap is set to 192 kbps:

- Encoder (`ENC`) tests: only tests with bitrate up to 192 kbps (inclusive).
- Decoder (`DEC`) tests:
@@ -273,14 +274,7 @@ except bitrate caps are relaxed to 192 kbps:
  - `MONO` output format: all bitrates.
  - `STEREO` output format: all bitrates.

When `--filter LEVEL3` is specified, all selection rules above remain the same,
with no bitrate caps:

- Encoder (`ENC`) tests: all bitrates.
- Decoder (`DEC`) tests:
  - `EXT` output format: all bitrates.
  - `MONO` output format: all bitrates.
  - `STEREO` output format: all bitrates.
When `--filter LEVEL3` is specified, there are no restrictions on the bitrate or output formats.

Examples (non-BE):

@@ -293,7 +287,7 @@ Examples (non-BE):
- LEVEL3 plus renderer and ISAR test groups

  ```shell
  PYTHONPATH=scripts python scripts/ivas_conformance/runConformance.py --testvecDir $PWD/testvec --ref_build_path=testvec/bin --analyse --filter LEVEL3 +REND +ISAR
  PYTHONPATH=scripts python scripts/ivas_conformance/runConformance.py --testvecDir $PWD/testvec --ref_build_path=testvec/bin --analyse --filter LEVEL3 +REND +ISAR +ISAR_ENC
  ```

- LEVEL1 baseline (ENC+DEC with LEVEL1 restrictions)
@@ -305,7 +299,7 @@ Examples (non-BE):
- LEVEL1 plus renderer and ISAR test groups

  ```shell
  PYTHONPATH=scripts python scripts/ivas_conformance/runConformance.py --testvecDir $PWD/testvec --ref_build_path=testvec/bin --analyse --filter LEVEL1 +REND +ISAR
  PYTHONPATH=scripts python scripts/ivas_conformance/runConformance.py --testvecDir $PWD/testvec --ref_build_path=testvec/bin --analyse --filter LEVEL1 +REND +ISAR +ISAR_ENC
  ```

- LEVEL1 with additional case-insensitive command substring filtering
@@ -335,5 +329,5 @@ Examples (non-BE):
- LEVEL2 plus renderer and ISAR test groups

  ```shell
  PYTHONPATH=scripts python scripts/ivas_conformance/runConformance.py --testvecDir $PWD/testvec --ref_build_path=testvec/bin --analyse --filter LEVEL2 +REND +ISAR
  PYTHONPATH=scripts python scripts/ivas_conformance/runConformance.py --testvecDir $PWD/testvec --ref_build_path=testvec/bin --analyse --filter LEVEL2 +REND +ISAR +ISAR_ENC
  ```
+6 −14
Original line number Diff line number Diff line
@@ -1123,7 +1123,7 @@ class MLDConformance:
            passes_level = self._matchesLevel2(tag, rawCmdline)

        passes_requested_formats = True
        if tag in {"ENC", "DEC"} and requested_formats:
        if requested_formats:
            cmd_formats = self._outputFormatsInCommand(rawCmdline)
            passes_requested_formats = bool(cmd_formats.intersection(requested_formats))

@@ -1876,19 +1876,19 @@ if __name__ == "__main__":
        help=(
            "Select which tests to run. Default baseline: ENC+DEC tests (REND/ISAR optional).\n"
            "\n"
            "TOKEN TYPES:\n"
            "Token types:\n"
            "  LEVEL1, LEVEL2, LEVEL3              — Conformance level. LEVEL1: ≤80 kbps; LEVEL2: ≤192 kbps; LEVEL3: unlimited (default).\n"
            "  ENC, DEC, REND, ISAR, ISAR_ENC      — Test groups (ENC+DEC are baseline; REND/ISAR optional).\n"
            "  MONO, STEREO, EXT, HOA, SBA, MC     — Output formats (ENC/DEC only). Aliases: HOA→{HOA2,HOA3}, SBA→{FOA,HOA2,HOA3}, MC→{5_1,7_1,5_1_4,5_1_2,7_1_4}.\n"
            "  ENC, DEC, REND, ISAR, ISAR_ENC      — Test groups.\n"
            "  MONO, STEREO, EXT, HOA, SBA, MC     — Output formats. Aliases: HOA→{HOA2,HOA3}, SBA→{FOA,HOA2,HOA3}, MC→{5_1,7_1,5_1_4,5_1_2,7_1_4}.\n"
            "  (any other)                         — Substring match (case-insensitive). Multiple terms combine with AND.\n"
            "\n"
            "TOKEN MODIFIERS:\n"
            "Token modifiers:\n"
            "  +TOKEN                              — Add matching tests to selection (even if they would be excluded).\n"
            "  -TOKEN                              — Remove matching tests from selection.\n"
            "  TOKEN*                              — Wildcard: match all tokens starting with TOKEN (e.g., ISAR* → {ISAR, ISAR_ENC}; BINAURAL* → {BINAURAL, BINAURAL_IR, BINAURAL_ROOM_IR, ...}).\n"
            "  +TOKEN*, -TOKEN*                    — Wildcard with add/remove modifiers (e.g., +ISAR*, -BINAURAL*).\n"
            "\n"
            "EXAMPLES:\n"
            "Examples:\n"
            "  --filter LEVEL1                     Run LEVEL1 ENC+DEC (≤80 kbps).\n"
            "  --filter LEVEL2                     Run LEVEL2 ENC+DEC (≤192 kbps).\n"
            "  --filter LEVEL1 DEC MONO            Restrict to DEC MONO tests at LEVEL1.\n"
@@ -2067,14 +2067,6 @@ if __name__ == "__main__":
    tag_remove_tokens = list(dict.fromkeys(tag_remove_tokens))
    decoder_format_tokens = list(dict.fromkeys(decoder_format_tokens))

    # If ISAR is requested/added/removed, apply same action to ISAR_ENC.
    if "ISAR" in tag_tokens and "ISAR_ENC" not in tag_tokens:
        tag_tokens.append("ISAR_ENC")
    if "ISAR" in tag_add_tokens and "ISAR_ENC" not in tag_add_tokens:
        tag_add_tokens.append("ISAR_ENC")
    if "ISAR" in tag_remove_tokens and "ISAR_ENC" not in tag_remove_tokens:
        tag_remove_tokens.append("ISAR_ENC")

    # All levels share the same default tag baseline: ENC + DEC.
    # REND/ISAR/ISAR_ENC are optional and must be explicitly selected or added.
    # Plain tag_tokens restrict the baseline; +tag_tokens add beyond it.