diff --git a/scripts/ivas_conformance/README.md b/scripts/ivas_conformance/README.md index d56974ef0a113d6a44b660f5d66f6403dc0e0db5..6f57b4d113dbc6d1b098ac37104baacbe5b7b7a4 100644 --- a/scripts/ivas_conformance/README.md +++ b/scripts/ivas_conformance/README.md @@ -68,25 +68,37 @@ PYTHONPATH=scripts python scripts/ivas_conformance/runConformance.py --testvecDi
Example Output of CUT execution -

+
+```console
 Accumulating commands from Readme_IVAS_dec.txt
 Accumulating commands from Readme_IVAS_rend.txt   
 Accumulating commands from Readme_IVAS_enc.txt
 Accumulating commands from Readme_IVAS_ISAR_post_rend.txt
 Accumulating commands from Readme_IVAS_ISAR_dec.txt
 Accumulating commands from Readme_IVAS_JBM_dec.txt
+
 No of tests :
     ENC : 381
     DEC : 637
     REND : 666
     ISAR_ENC : 1032
     ISAR : 1032
-Executing tests for ENC   (381 tests)
-Executing tests for DEC   (637 tests)
-Executing tests for REND   (666 tests)
-Executing tests for ISAR_ENC   (1032 tests)
-Executing tests for ISAR   (1032 tests)
-
+ +Executing tests for ENC (381 tests): +--------------------------- +[ENC] OK + +... + +Summary of results: +--------------------- +[ENC] OK +[DEC] OK +[REND] OK +[ISAR_ENC] OK +[ISAR] OK +``` +
This should generate outputs in scripts/CUT_OUTPUTS folder which looks like below:- @@ -108,7 +120,7 @@ If CUT test execution is done on a different platform, the scripts/CUT_OUTPUTS m ### Perform the BE comparison on the CUT outputs on reference platform -The BE comparison is performed to the CUT outputs using the command below. Encoded outputs will be decoded using the reference decoder executables as part of the process. The BE comparison is then performed between the CUT and reference decoded outputs. This includes comparison of ".wav"-files, and ".csv" and ".met" metadata files. If any non-BE results are observed, this is reported on the command-line and link to an analysis ".csv" file is given. The analysis file shows which exact files were non-BE. An example passing output is shown below. If all test sets print `PASSED BE TEST`, then CUT outputs are BE-conformant. +The BE comparison is performed to the CUT outputs using the command below. Encoded outputs are decoded using the reference decoder executables as part of the process. The BE comparison is then performed between the CUT and reference decoded outputs. This includes comparison of `.wav` files and `.csv`/`.met` metadata files. If non-BE results are observed, this is reported on the command line and in the generated analysis CSV output. ```shell PYTHONPATH=scripts python scripts/ivas_conformance/runConformance.py --testvecDir $PWD/testvec --ref_build_path=testvec/bin --analyse --be-test @@ -116,36 +128,46 @@ PYTHONPATH=scripts python scripts/ivas_conformance/runConformance.py --testvecDi
Example Output of BE comparison -

+
+```console
 Accumulating commands from Readme_IVAS_dec.txt
 Accumulating commands from Readme_IVAS_enc.txt
 Accumulating commands from Readme_IVAS_rend.txt
 Accumulating commands from Readme_IVAS_JBM_dec.txt
 Accumulating commands from Readme_IVAS_ISAR_dec.txt
 Accumulating commands from Readme_IVAS_ISAR_post_rend.txt
+
 No of tests :
     ENC : 374
     DEC : 638
     REND : 911
     ISAR_ENC : 1032
     ISAR : 1252
-Analysing tests for ENC   (374 tests)
-<ENC> PASSED BE TEST
-Analysing tests for DEC   (638 tests)
-<DEC> PASSED BE TEST
-Analysing tests for REND   (911 tests)
-<REND> PASSED BE TEST
-Analysing tests for ISAR_ENC   (1032 tests)
-<ISAR_ENC> PASSED BE TEST
-Analysing tests for ISAR   (1252 tests)
-<ISAR> PASSED BE TEST
-
+ +Analysing tests for ENC (374 tests): +--------------------------- + +[ENC] OK (ERRORS=0, BE=374, NON-BE=0, MLD CORRIDOR FAILURES=0) +[DEC] OK (ERRORS=0, BE=638, NON-BE=0, MLD CORRIDOR FAILURES=0) + +... + +Summary of results: +--------------------- +[ENC] OK +[DEC] OK +[REND] OK +[ISAR_ENC] OK +[ISAR] OK +``` +
### Perform the MLD based non-BE analysis on the CUT outputs on reference platform (Ubuntu 24.04) -The MLD-based non-BE analysis is performed to the CUT outputs with the command below. Encoded outputs will be decoded using the reference decoder executables as part of the process. The MLD analysis is then performed between the CUT and reference decoded outputs (only ".wav" files are compared). Comparison to MLD corridor is also done as part of this process. An example passing output is shown below. If all test sets print `MLD Corridor passed for...` and there were no non-BE metadata comparisons in BE-test, then CUT outputs are Non-BE conformant. +The non-BE analysis below compares CUT and reference outputs by running MLD on audio (`.wav`) and, when MASA metadata are generated, for the matching reference/CUT `.met` files. For encoder tests, encoded CUT bitstreams are first decoded with the reference decoder before analysis. Per-frame MLD and MASA metadata values are written to `scripts/CUT_OUTPUTS` and checked against corridor references in `testvec/testv/mld_ref` (`mld_ref_.csv` and `masa_ref_.csv`). + ```shell PYTHONPATH=scripts python scripts/ivas_conformance/runConformance.py --testvecDir $PWD/testvec --ref_build_path=testvec/bin --analyse @@ -153,188 +175,159 @@ PYTHONPATH=scripts python scripts/ivas_conformance/runConformance.py --testvecDi
Example Output of non-BE analysis -

+
+```console
 Accumulating commands from Readme_IVAS_dec.txt
 Accumulating commands from Readme_IVAS_enc.txt
 Accumulating commands from Readme_IVAS_rend.txt
 Accumulating commands from Readme_IVAS_JBM_dec.txt
 Accumulating commands from Readme_IVAS_ISAR_dec.txt
 Accumulating commands from Readme_IVAS_ISAR_post_rend.txt
+
 No of tests :
     ENC : 374
     DEC : 638
     REND : 911
     ISAR_ENC : 1032
     ISAR : 1252
-Analysing tests for ENC   (374 tests)
-
-##########################################################
-<ENC> Total Frames: 3074220
-<ENC> MAX MLD across all frames : 0.0
-<ENC> Frames with MLD == 0 : 3074220 frames (100.0%)
-<ENC> Frames with MLD <= 0.5 : 3074220 frames (100.0%)
-<ENC> Frames with MLD <= 1 : 3074220 frames (100.0%)
-<ENC> Frames with MLD <= 2 : 3074220 frames (100.0%)
-<ENC> Frames with MLD <= 5 : 3074220 frames (100.0%)
-<ENC> BE samples percentage = 100.0
-<ENC> max absolute diff = 0.0, sample range (-32768, 32767)
-##########################################################
 
-MLD Corridor passed for ENC with max MLD diff of 0.0
-Analysing tests for DEC   (638 tests)
+Analysing tests for ENC  (374 tests):
+---------------------------
 
 ##########################################################
-<DEC> Total Frames: 5079252
-<DEC> MAX MLD across all frames : 0.0
-<DEC> Frames with MLD == 0 : 5079252 frames (100.0%)
-<DEC> Frames with MLD <= 0.5 : 5079252 frames (100.0%)
-<DEC> Frames with MLD <= 1 : 5079252 frames (100.0%)
-<DEC> Frames with MLD <= 2 : 5079252 frames (100.0%)
-<DEC> Frames with MLD <= 5 : 5079252 frames (100.0%)
-<DEC> BE samples percentage = 100.0
-<DEC> max absolute diff = 0.0, sample range (-32768, 32767)
+ Total Frames: 3074220
+ MAX MLD across all frames : 0.0
+ Frames with MLD == 0 : 3074220 frames (100.0%)
+ Frames with MLD <= 0.5 : 3074220 frames (100.0%)
+ Frames with MLD <= 1 : 3074220 frames (100.0%)
+ Frames with MLD <= 2 : 3074220 frames (100.0%)
+ Frames with MLD <= 5 : 3074220 frames (100.0%)
+ BE samples percentage = 100.0
+ max absolute diff = 0.0, sample range (-32768, 32767)
 ##########################################################
 
-MLD Corridor passed for DEC with max MLD diff of 0.0
-Analysing tests for REND   (911 tests)
-
-##########################################################
-<REND> Total Frames: 5576907
-<REND> MAX MLD across all frames : 0.0
-<REND> Frames with MLD == 0 : 5576907 frames (100.0%)
-<REND> Frames with MLD <= 0.5 : 5576907 frames (100.0%)
-<REND> Frames with MLD <= 1 : 5576907 frames (100.0%)
-<REND> Frames with MLD <= 2 : 5576907 frames (100.0%)
-<REND> Frames with MLD <= 5 : 5576907 frames (100.0%)
-<REND> BE samples percentage = 100.0
-<REND> max absolute diff = 0.0, sample range (-32768, 32767)
-##########################################################
-
-MLD Corridor passed for REND with max MLD diff of 0.0
-Analysing tests for ISAR_ENC   (1032 tests)
-
-##########################################################
-<ISAR_ENC> Total Frames: 2125956
-<ISAR_ENC> MAX MLD across all frames : 0.0
-<ISAR_ENC> Frames with MLD == 0 : 2125956 frames (100.0%)
-<ISAR_ENC> Frames with MLD <= 0.5 : 2125956 frames (100.0%)
-<ISAR_ENC> Frames with MLD <= 1 : 2125956 frames (100.0%)
-<ISAR_ENC> Frames with MLD <= 2 : 2125956 frames (100.0%)
-<ISAR_ENC> Frames with MLD <= 5 : 2125956 frames (100.0%)
-<ISAR_ENC> BE samples percentage = 100.0
-<ISAR_ENC> max absolute diff = 0.0, sample range (-32768, 32767)
-##########################################################
+[ENC] OK (ERRORS=0, BE=374, NON-BE=0, MLD CORRIDOR FAILURES=0)
 
-MLD Corridor passed for ISAR_ENC with max MLD diff of 0.0
-Analysing tests for ISAR   (1252 tests)
+...
 
-##########################################################
-<ISAR> Total Frames: 2590956
-<ISAR> MAX MLD across all frames : 0.0
-<ISAR> Frames with MLD == 0 : 2590956 frames (100.0%)
-<ISAR> Frames with MLD <= 0.5 : 2590956 frames (100.0%)
-<ISAR> Frames with MLD <= 1 : 2590956 frames (100.0%)
-<ISAR> Frames with MLD <= 2 : 2590956 frames (100.0%)
-<ISAR> Frames with MLD <= 5 : 2590956 frames (100.0%)
-<ISAR> BE samples percentage = 100.0
-<ISAR> max absolute diff = 0.0, sample range (-32768, 32767)
-##########################################################
+Summary of results:
+---------------------
+[ENC] OK
+[DEC] OK
+[REND] OK
+[ISAR_ENC] OK
+[ISAR] OK
+```
 
-MLD Corridor passed for ISAR with max MLD diff of 0.0
-
## Executing specific tests only -All CUT tests can be run specifically for IVAS Encoder,IVAS Decoder,IVAS Renderer, ISAR Encoder and ISAR Decoder only. The commandline allows for ```-test-mode=``` for this functionality, examples: - -- Run CUT IVAS Encoder Tests Only (on Target Platform) - - ```shell - PYTHONPATH=scripts python scripts/ivas_conformance/runConformance.py --testvecDir $PWD/testvec --cut_build_path=CUT_BIN_DIR --test-mode=ENC - ``` - -- Analyse BE conformance for CUT IVAS Encoder Outputs Only (on Reference Platform) - - ```shell - PYTHONPATH=scripts python scripts/ivas_conformance/runConformance.py --testvecDir $PWD/testvec --ref_build_path=testvec/bin --test-mode=ENC --analyse --be-test - ``` - -- Analyse NON-BE conformance for CUT IVAS Encoder Outputs Only (on Reference Platform) - - ```shell - PYTHONPATH=scripts python scripts/ivas_conformance/runConformance.py --testvecDir $PWD/testvec --ref_build_path=testvec/bin --test-mode=ENC --analyse - ``` - -- Run CUT IVAS Decoder Tests Only (on Target Platform) - - ```shell - PYTHONPATH=scripts python scripts/ivas_conformance/runConformance.py --testvecDir $PWD/testvec --cut_build_path=CUT_BIN_DIR --test-mode=DEC - ``` - -- Analyse BE conformance for CUT IVAS Decoder Outputs Only - - ```shell - PYTHONPATH=scripts python scripts/ivas_conformance/runConformance.py --testvecDir $PWD/testvec --test-mode=DEC --analyse --be-test - ``` - -- Analyse NON-BE conformance CUT IVAS Decoder Outputs Only (on Reference Platform) - - ```shell - PYTHONPATH=scripts python scripts/ivas_conformance/runConformance.py --testvecDir $PWD/testvec --test-mode=DEC --analyse - ``` - -- Run CUT IVAS Renderer Tests Only (on Target Platform) +Use `--filter TOKEN` to control conformance level, test groups, output formats, and substring matching. + +- Token types: + - `LEVEL1`, `LEVEL2`, `LEVEL3`: conformance levels. `LEVEL3` is the default. + - `ENC`, `DEC`, `REND`, `ISAR`, `ISAR_ENC`: test groups. + - `MONO`, `STEREO`, `EXT`, `HOA`, `SBA`, `MC`: output-format tokens. + - `HOA` expands to `HOA2`, `HOA3`. + - `SBA` expands to `FOA`, `HOA2`, `HOA3`. + - `MC` expands to `5_1`, `7_1`, `5_1_4`, `5_1_2`, `7_1_4`. + - Any other token is treated as a case-insensitive substring match. +- Token modifiers: + - `TOKEN`: restrictive token. Multiple restrictive tokens combine with logical AND. + - `+TOKEN`: additive token. Adds matching tests even if they would otherwise be excluded. + - `-TOKEN`: subtractive token. Removes matching tests from the final selection. + - `TOKEN*`: wildcard token. Matches all known tokens starting with the given prefix. + - `+TOKEN*`, `-TOKEN*`: wildcard token with add/remove behavior. + +### LEVEL1, LEVEL2 and LEVEL3 behavior + +For all levels, the default test-group baseline is `ENC` + `DEC`. +`REND`, `ISAR`, and `ISAR_ENC` are optional and are only included if explicitly selected as plain test-group tokens or added via `+REND`, `+ISAR`, `+ISAR_ENC`, or `+ISAR*`. + +When `--filter LEVEL1` is specified, the following default tests are run: + +- Encoder (`ENC`) tests: only tests with bitrate up to 80 kbps (inclusive). +- Decoder (`DEC`) tests: + - `EXT` output format: only bitrate up to 80 kbps (inclusive). + - `MONO` output format: all bitrates. + - `STEREO` output format: all bitrates. + +- The default `LEVEL1` tests may be restricted by adding more tokens (acting as logical AND). + - Example: `--filter LEVEL1 DEC MONO` keeps only `MONO` tests from the LEVEL1-eligible DEC set. + - Example: `--filter LEVEL1 JBM` keeps all LEVEL1-eligible ENC tests but only JBM tests from the LEVEL1-eligible DEC tests. +- `+TOKEN` adds tests to the final LEVEL1 selection, even if they would otherwise be restricted. + - Example: `--filter LEVEL1 DEC JBM +BINAURAL` runs only JBM-matching LEVEL1 DEC tests and additionally includes DEC tests containing `BINAURAL` keyword, i.e. `BINAURAL`, `BINAURAL_IR`, `BINAURAL_ROOM_IR`, `BINAURAL_REVERB`. +- `-TOKEN` removes matching tests from the final LEVEL1 selection (including tests added via `+TOKEN`). + - Example: `--filter LEVEL1 DEC +JBM -VOIP` adds JBM-matching DEC tests and then excludes any DEC tests containing the keyword `VOIP`. +- Renderer and ISAR tests are not run by default in `LEVEL1`. + - Add `+REND`, `+ISAR`, and/or `+ISAR_ENC` in `--filter` to include them. + - Use `+ISAR*` if you want wildcard expansion across all `ISAR*`-prefixed test-group tokens. + +When `--filter LEVEL2` is specified, all selection rules above remain the same, +except the bitrate cap is set to 192 kbps: + +- Encoder (`ENC`) tests: only tests with bitrate up to 192 kbps (inclusive). +- Decoder (`DEC`) tests: + - `EXT` output format: only bitrate up to 192 kbps (inclusive). + - `MONO` output format: all bitrates. + - `STEREO` output format: all bitrates. + +When `--filter LEVEL3` is specified, there are no restrictions on the bitrate or output formats. + +Examples (non-BE): + +- Default behavior (same as LEVEL3 baseline): run only ENC and DEC test groups ```shell - PYTHONPATH=scripts python scripts/ivas_conformance/runConformance.py --testvecDir $PWD/testvec --cut_build_path=CUT_BIN_DIR --test-mode=REND + PYTHONPATH=scripts python scripts/ivas_conformance/runConformance.py --testvecDir $PWD/testvec --ref_build_path=testvec/bin --analyse ``` -- Analyse BE conformance for CUT Renderer Outputs Only +- LEVEL3 plus renderer and ISAR test groups ```shell - PYTHONPATH=scripts python scripts/ivas_conformance/runConformance.py --testvecDir $PWD/testvec --test-mode=REND --analyse --be-test + PYTHONPATH=scripts python scripts/ivas_conformance/runConformance.py --testvecDir $PWD/testvec --ref_build_path=testvec/bin --analyse --filter LEVEL3 +REND +ISAR +ISAR_ENC ``` -- Analyse NON-BE conformance CUT Renderer Outputs Only +- LEVEL1 baseline (ENC+DEC with LEVEL1 restrictions) ```shell - PYTHONPATH=scripts python scripts/ivas_conformance/runConformance.py --testvecDir $PWD/testvec --test-mode=REND --analyse + PYTHONPATH=scripts python scripts/ivas_conformance/runConformance.py --testvecDir $PWD/testvec --ref_build_path=testvec/bin --analyse --filter LEVEL1 ``` -- Run CUT ISAR Encoder Tests Only (on Target Platform) +- LEVEL1 plus renderer and ISAR test groups ```shell - PYTHONPATH=scripts python scripts/ivas_conformance/runConformance.py --testvecDir $PWD/testvec --cut_build_path=CUT_BIN_DIR --test-mode=ISAR_ENC + PYTHONPATH=scripts python scripts/ivas_conformance/runConformance.py --testvecDir $PWD/testvec --ref_build_path=testvec/bin --analyse --filter LEVEL1 +REND +ISAR +ISAR_ENC ``` -- Analyse BE conformance for CUT ISAR Encoder Outputs Only (on Reference Platform) +- LEVEL1 with additional case-insensitive command substring filtering ```shell - PYTHONPATH=scripts python scripts/ivas_conformance/runConformance.py --testvecDir $PWD/testvec --ref_build_path=testvec/bin --test-mode=ISAR_ENC --analyse --be-test + PYTHONPATH=scripts python scripts/ivas_conformance/runConformance.py --testvecDir $PWD/testvec --ref_build_path=testvec/bin --analyse --filter LEVEL1 DEC voip ``` -- Analyse NON-BE conformance for CUT ISAR Encoder Outputs Only (on Reference Platform) +- LEVEL1 with additive BINAURAL decoder matching ```shell - PYTHONPATH=scripts python scripts/ivas_conformance/runConformance.py --testvecDir $PWD/testvec --ref_build_path=testvec/bin --test-mode=ISAR_ENC --analyse + PYTHONPATH=scripts python scripts/ivas_conformance/runConformance.py --testvecDir $PWD/testvec --ref_build_path=testvec/bin --analyse --filter LEVEL1 +BINAURAL ``` -- Run CUT ISAR Decoder Tests Only (on Target Platform) +- LEVEL1 with restrictive and additive terms together ```shell - PYTHONPATH=scripts python scripts/ivas_conformance/runConformance.py --testvecDir $PWD/testvec --cut_build_path=CUT_BIN_DIR --test-mode=ISAR + PYTHONPATH=scripts python scripts/ivas_conformance/runConformance.py --testvecDir $PWD/testvec --ref_build_path=testvec/bin --analyse --filter LEVEL1 DEC JBM +BINAURAL ``` -- Analyse BE conformance for CUT ISAR Decoder Outputs Only +- LEVEL2 baseline (ENC+DEC with LEVEL2 restrictions) ```shell - PYTHONPATH=scripts python scripts/ivas_conformance/runConformance.py --testvecDir $PWD/testvec --test-mode=ISAR --analyse --be-test + PYTHONPATH=scripts python scripts/ivas_conformance/runConformance.py --testvecDir $PWD/testvec --ref_build_path=testvec/bin --analyse --filter LEVEL2 ``` -- Analyse NON-BE conformance CUT ISAR Decoder Outputs Only +- LEVEL2 plus renderer and ISAR test groups ```shell - PYTHONPATH=scripts python scripts/ivas_conformance/runConformance.py --testvecDir $PWD/testvec --test-mode=ISAR --analyse + PYTHONPATH=scripts python scripts/ivas_conformance/runConformance.py --testvecDir $PWD/testvec --ref_build_path=testvec/bin --analyse --filter LEVEL2 +REND +ISAR +ISAR_ENC ``` diff --git a/scripts/ivas_conformance/runConformance.py b/scripts/ivas_conformance/runConformance.py index 50f851db9eb6f0d1908c0602bd02a9e0c8c02861..d066a3092e82f66f886ac16768f7586927ee4543 100644 --- a/scripts/ivas_conformance/runConformance.py +++ b/scripts/ivas_conformance/runConformance.py @@ -53,6 +53,64 @@ import time sys.path.append(os.path.join(os.path.dirname(os.path.abspath(__file__)), "..")) + +def _preprocess_filter_args(): + """Preprocess sys.argv to handle --filter with minus-prefixed tokens. + + Argparse stops consuming args when it encounters e.g. '-TERM', treating it as a flag. + This function escapes filter tokens by wrapping minus-prefixed tokens with a marker, + allowing argparse to treat them as regular arguments. + + Stores the escape mapping in _FILTER_ESCAPES for later unescaping. + + Returns: modified sys.argv with filter tokens escaped + """ + global _FILTER_ESCAPES + _FILTER_ESCAPES = {} + marker_prefix = "@FILT_" + result_argv = [] + i = 0 + + while i < len(sys.argv): + arg = sys.argv[i] + + # When we see --filter, process all following tokens until next option (--) or end + if arg == "--filter": + result_argv.append(arg) + i += 1 + + # Collect all filter tokens, escaping those that look like flags + while i < len(sys.argv): + tok = sys.argv[i] + + # Stop if we hit the next option (--something or -X where X is not escaped) + if tok.startswith("--") or (tok.startswith("-") and len(tok) <= 2 and tok != "-"): + # Exception: stop only if it's a known argparse option + known_short_options = {"-h", "--help"} + if tok in known_short_options or tok.startswith("--"): + break + break + + # Escape tokens starting with - to protect them from argparse + if tok.startswith("-"): + escaped = f"{marker_prefix}{len(_FILTER_ESCAPES)}" + _FILTER_ESCAPES[escaped] = tok + result_argv.append(escaped) + else: + result_argv.append(tok) + + i += 1 + else: + result_argv.append(arg) + i += 1 + + + return result_argv + + +# Module-level dict to track filter token escapes +_FILTER_ESCAPES = {} + def readfile( filename: str, nchannels: int = 1, fs: int = 48000, outdtype="float" ) -> Tuple[np.ndarray, int]: @@ -197,6 +255,29 @@ IVAS_Bins = { "ISAR": "ISAR_post_rend", } +DECODER_OUTPUT_FORMATS = { + "MONO", + "STEREO", + "BINAURAL", + "BINAURAL_ROOM_IR", + "BINAURAL_ROOM_REVERB", + "5_1", + "7_1", + "5_1_4", + "5_1_2", + "7_1_4", + "FOA", + "HOA2", + "HOA3", + "EXT", +} + +DECODER_OUTPUT_FORMAT_ALIASES = { + "HOA": {"HOA2", "HOA3"}, + "SBA": {"FOA", "HOA2", "HOA3"}, + "MC": {"5_1", "7_1", "5_1_4", "5_1_2", "7_1_4"}, +} + def validate_build_binaries(parser, build_path: str, build_label: str) -> None: """Validate that a build path exists and contains all IVAS binaries.""" @@ -302,6 +383,14 @@ class MLDConformance: os.makedirs(os.path.join(self.testvDir, odir), exist_ok=True) os.makedirs(os.path.join(self.outputDir, odir), exist_ok=True) + nested_subdirs = [ + os.path.join("renderer_short", "ref"), + os.path.join("split_rendering", "cut"), + os.path.join("split_rendering", "ref"), + ] + for odir in nested_subdirs: + os.makedirs(os.path.join(self.outputDir, odir), exist_ok=True) + self.logFile = os.path.join(self.outputDir, "runlog.txt") self.failedCmdsFile = os.path.join(self.outputDir, "failedCmds.txt") self.errorBlocksDir = os.path.join(self.outputDir, "error_blocks") @@ -352,7 +441,7 @@ class MLDConformance: def setupDUT(self): self.cut_build_path = self.args.cut_build_path - self.filter = self.args.filter + self.filter = getattr(self.args, "filter_display", self.args.filter) exe_platform = platform.system() if exe_platform == "Windows": exe_platform = "Win64" @@ -581,10 +670,6 @@ class MLDConformance: ) else: print(f"{pyTestTag} not found in ISAR decoder") - print("No of tests :") - for tag in testDesciptor.keys(): - print(f" {tag} : {len(testDesciptor[tag])}") - return testDesciptor def genEncoderReferences(self, tag: str, encPytestTag: str): @@ -923,21 +1008,155 @@ class MLDConformance: def analyseOneCommandFromTuple(self, args): return self.analyseOneCommand(*args) + def _extractKbpsValues(self, rawCmdline: str) -> list[float]: + """Extract all bitrate values from command line (e.g., from 'at_32_kbps' or 'from_32_kbps_to_96_kbps').""" + values = [] + for match in re.findall(r"(\d+(?:_\d+)?)_kbps", rawCmdline.lower()): + values.append(float(match.replace("_", "."))) + return values + + def _isBitrateAtMost(self, rawCmdline: str, max_kbps: float) -> bool: + """Check if all bitrates in command line are <= max_kbps. + + For bitrate switching tests (e.g., 'from_32_kbps_to_96_kbps'), this checks + that the upper (target) bitrate does not exceed max_kbps.""" + values = self._extractKbpsValues(rawCmdline) + return bool(values) and max(values) <= float(max_kbps) + + def _isBitrateAtMost80(self, rawCmdline: str) -> bool: + return self._isBitrateAtMost(rawCmdline, 80.0) + + def _isBitrateAtMost192(self, rawCmdline: str) -> bool: + return self._isBitrateAtMost(rawCmdline, 192.0) + + def _outputFormatsInCommand(self, rawCmdline: str) -> set[str]: + text = rawCmdline.upper() + formats = set() + + # Match format token between '_out_' and '_out' to avoid matching input-format words. + # Example: '..._48kHz_out_MONO_out_...' -> captures MONO only. + for match in re.finditer(r"_OUT_([A-Z0-9_]+?)_OUT(?:\b|_)", text): + fmt = match.group(1) + if fmt == "EXTERNAL": + fmt = "EXT" + if fmt in DECODER_OUTPUT_FORMATS: + formats.add(fmt) + + # Avoid matching words like EXTENDED; key on EXT_OUT style output naming. + if ( + "_EXT_OUT" in text + or "_EXTERNAL_OUT" in text + or " EXT_OUT" in text + or " EXTERNAL_OUT" in text + ): + formats.add("EXT") + + return formats + + def _matchesAllTerms(self, rawCmdline: str, terms: list[str]) -> bool: + text = rawCmdline.lower() + return all(term.lower() in text for term in terms) + + def _matchesAnyTerm(self, rawCmdline: str, terms: list[str]) -> bool: + text = rawCmdline.lower() + return any(term.lower() in text for term in terms) + + def _matchesLevel1(self, tag: str, rawCmdline: str) -> bool: + if tag == "ENC": + return self._isBitrateAtMost80(rawCmdline) + + if tag == "DEC": + formats = self._outputFormatsInCommand(rawCmdline) + requested_formats = set(getattr(self.args, "filter_decoder_formats", [])) + + ext_ok = "EXT" in formats and self._isBitrateAtMost80(rawCmdline) + mono_ok = "MONO" in formats + stereo_ok = "STEREO" in formats + default_level1_dec_ok = ext_ok or mono_ok or stereo_ok + + if requested_formats: + # Plain decoder format tokens are restrictive under LEVEL1. + return default_level1_dec_ok and bool(formats.intersection(requested_formats)) + + return default_level1_dec_ok + + # For REND/ISAR/ISAR_ENC under LEVEL1, tag-level inclusion is decided at testTags parsing. + return True + + def _matchesLevel2(self, tag: str, rawCmdline: str) -> bool: + if tag == "ENC": + return self._isBitrateAtMost192(rawCmdline) + + if tag == "DEC": + formats = self._outputFormatsInCommand(rawCmdline) + requested_formats = set(getattr(self.args, "filter_decoder_formats", [])) + + ext_ok = "EXT" in formats and self._isBitrateAtMost192(rawCmdline) + mono_ok = "MONO" in formats + stereo_ok = "STEREO" in formats + default_level2_dec_ok = ext_ok or mono_ok or stereo_ok + + if requested_formats: + # Plain decoder format tokens are restrictive under LEVEL2. + return default_level2_dec_ok and bool(formats.intersection(requested_formats)) + + return default_level2_dec_ok + + # For REND/ISAR/ISAR_ENC under LEVEL2, tag-level inclusion is decided at testTags parsing. + return True + + def _testPassesFilter(self, tag: str, rawCmdline: str) -> bool: + level = getattr(self.args, "filter_level", "LEVEL3") + restrictive_terms = getattr(self.args, "filter_restrictive_terms", []) + additive_terms = getattr(self.args, "filter_add_terms", []) + subtractive_terms = getattr(self.args, "filter_remove_terms", []) + requested_formats = set(getattr(self.args, "filter_decoder_formats", [])) + + # '-' terms always remove tests from the final selection. + if subtractive_terms and self._matchesAnyTerm(rawCmdline, subtractive_terms): + return False + + passes_level = True + if level == "LEVEL1": + passes_level = self._matchesLevel1(tag, rawCmdline) + elif level == "LEVEL2": + passes_level = self._matchesLevel2(tag, rawCmdline) + + passes_requested_formats = True + if requested_formats: + cmd_formats = self._outputFormatsInCommand(rawCmdline) + passes_requested_formats = bool(cmd_formats.intersection(requested_formats)) + + passes_restrictive_terms = self._matchesAllTerms(rawCmdline, restrictive_terms) + base_selected = passes_level and passes_restrictive_terms and passes_requested_formats + + if base_selected: + return True + + # '+' terms add tests even if they fail restrictive filters. + if additive_terms and self._matchesAnyTerm(rawCmdline, additive_terms): + return True + + return False + + def getSelectedTestsForTag(self, tag: str) -> list[str]: + selected = [] + for pyTestsTag in self.TestDesc[tag].keys(): + rawCmdline = self.TestDesc[tag][pyTestsTag].rawCmdline + if self._testPassesFilter(tag, rawCmdline): + selected.append(pyTestsTag) + return selected + def runTag(self, tag: str) -> bool: failed_before = self.getFailedCommandCount() - selectedTests = list() - if self.filter: - for pyTestsTag in self.TestDesc[tag].keys(): - if self.filter in self.TestDesc[tag][pyTestsTag].rawCmdline: - selectedTests.append(pyTestsTag) - else: - selectedTests = list(self.TestDesc[tag].keys()) + selectedTests = self.getSelectedTestsForTag(tag) self.totalTests = len(selectedTests) print( - f"Executing tests for {tag} {'Filter=' + self.filter if self.filter else ''} ({self.totalTests} tests)", + f"Executing tests for {tag} ({self.totalTests} tests):", flush=True, ) + print("---------------------------") if not self.args.no_multi_processing: with Pool() as pool: args = [ @@ -1013,19 +1232,14 @@ class MLDConformance: with open(self.sampleStats[tag], "w") as f: f.write(f"PYTESTTAG, MAXDIFF, RMSdB, BEFRAMES_PERCENT, MAX_MLD\n") - selectedTests = [] - if self.filter: - for pyTestsTag in self.TestDesc[tag].keys(): - if self.filter in self.TestDesc[tag][pyTestsTag].rawCmdline: - selectedTests.append(pyTestsTag) - else: - selectedTests = list(self.TestDesc[tag].keys()) + selectedTests = self.getSelectedTestsForTag(tag) self.totalTests = len(selectedTests) print( - f"Analysing tests for {tag} {'Filter=' + self.filter if self.filter else ''} ({self.totalTests} tests)", + f"Analysing tests for {tag} ({self.totalTests} tests):", flush=True, ) + print("---------------------------") def handle_test_result( testPrefix, @@ -1118,6 +1332,7 @@ class MLDConformance: if self.args.regenerate_mld_ref: return command_fail_count == 0 and analysis_ok + print() if command_fail_count == 0 and failure_count == 0 and analysis_ok: print( f"[{tag}] OK (ERRORS={command_fail_count}, BE={be_count}, NON-BE={non_be_count}, MLD CORRIDOR FAILURES={failure_count})\n" @@ -1147,7 +1362,7 @@ class MLDConformance: contextPrefix: str = "", emitConsole: bool = True, returnOutput: bool = False, - ) -> int: + ) -> Union[int, Tuple[int, str]]: contextPrefix = contextPrefix or (f"[{tag}]" if tag else "") return self._process( command, @@ -1166,12 +1381,14 @@ class MLDConformance: contextPrefix: str = "", emitConsole: bool = True, returnOutput: bool = False, - ) -> int: + ) -> Union[int, Tuple[int, str]]: prefix = (contextPrefix + " ") if contextPrefix else "" if self.args.verbose and emitConsole: print(f"{prefix}Command: {command}", flush=True) if self.args.dryrun: self.appendRunlog(command=command) + if returnOutput: + return 0, "" return 0 c = subprocess.run( @@ -1435,6 +1652,12 @@ class MLDConformance: keys = IVAS_Bins.keys() if selectTag == "all" else [selectTag] for tag in keys: if os.path.exists(self.BEcsv[tag]): + # For filtered runs it is valid to have zero selected tests for a tag. + # In that case the BE csv contains only the header; skip loadtxt to avoid warnings. + with open(self.BEcsv[tag], "r") as f: + non_empty_lines = [line for line in f if line.strip()] + if len(non_empty_lines) <= 1: + continue BEresult = np.loadtxt( self.BEcsv[tag], delimiter=",", @@ -1442,10 +1665,10 @@ class MLDConformance: skiprows=1, usecols=1, ) - if np.sum(BEresult) > 0: - print(f"<{tag}> FAILED BE TEST, check {self.BEcsv[tag]}") - else: - print(f"<{tag}> PASSED BE TEST") + # if np.sum(BEresult) > 0: + # print(f"<{tag}> FAILED BE TEST, check {self.BEcsv[tag]}") + # else: + # print(f"<{tag}> PASSED BE TEST") def computeCorridor(self, mldRefWithTags, mldCutWithTags, tag, threshold=0.1): indRef = np.argsort(mldRefWithTags["pyTestTag"]) @@ -1471,16 +1694,25 @@ class MLDConformance: else: ref_count = refMLD.shape[0] dut_count = dutMLD.shape[0] - ref_preview = ", ".join(refTags[:3]) if ref_count else "" - dut_preview = ", ".join(dutTags[:3]) if dut_count else "" - warn_msg = ( - f"Warning: {tag} corridor comparison skipped because reference and DUT frame tags do not match " - f"(ref_count={ref_count}, dut_count={dut_count}, ref_first=[{ref_preview}], dut_first=[{dut_preview}])." - ) - print(f"\033[93m{warn_msg}\033[00m") - self.appendRunlog(context=warn_msg) - self.appendFailed(context=warn_msg) - corridor_failed = True + # If filters are active, frame count mismatch is expected (DUT has fewer tests than reference). + # Skip the warning and don't treat it as a failure in this case. + if getattr(self.args, "filter_display", None): + skip_msg = ( + f"[{tag}] Corridor comparison skipped (filtered test set: ref_count={ref_count}, dut_count={dut_count})" + ) + self.appendRunlog(context=skip_msg) + corridor_failed = False + else: + ref_preview = ", ".join(refTags[:3]) if ref_count else "" + dut_preview = ", ".join(dutTags[:3]) if dut_count else "" + warn_msg = ( + f"Warning: {tag} corridor comparison skipped because reference and DUT frame tags do not match " + f"(ref_count={ref_count}, dut_count={dut_count}, ref_first=[{ref_preview}], dut_first=[{dut_preview}])." + ) + print(f"\033[93m{warn_msg}\033[00m") + self.appendRunlog(context=warn_msg) + self.appendFailed(context=warn_msg) + corridor_failed = True return not corridor_failed @@ -1589,7 +1821,8 @@ class MLDConformance: if __name__ == "__main__": parser = argparse.ArgumentParser( - description="Compare .wav files in two folders using mld per frame" + description="Compare .wav files in two folders using mld per frame", + formatter_class=argparse.RawTextHelpFormatter, ) parser.add_argument( @@ -1637,16 +1870,35 @@ if __name__ == "__main__": parser.add_argument( "--filter", - type=str, - default=None, - help="Filter test based on text provided", - ) - parser.add_argument( - "--test-mode", - type=str, + nargs="+", default=None, - choices=["ENC", "DEC", "REND", "ISAR", "ISAR_ENC"], - help='Choose one test group to run ["ENC", "DEC", "REND", "ISAR", "ISAR_ENC"]. If omitted, all are run.', + metavar="TOKEN", + help=( + "Select which tests to run. Default baseline: ENC+DEC tests (REND/ISAR optional).\n" + "\n" + "Token types:\n" + " LEVEL1, LEVEL2, LEVEL3 — Conformance level. LEVEL1: ≤80 kbps; LEVEL2: ≤192 kbps; LEVEL3: unlimited (default).\n" + " ENC, DEC, REND, ISAR, ISAR_ENC — Test groups.\n" + " MONO, STEREO, EXT, HOA, SBA, MC — Output formats. Aliases: HOA→{HOA2,HOA3}, SBA→{FOA,HOA2,HOA3}, MC→{5_1,7_1,5_1_4,5_1_2,7_1_4}.\n" + " (any other) — Substring match (case-insensitive). Multiple terms combine with AND.\n" + "\n" + "Token modifiers:\n" + " +TOKEN — Add matching tests to selection (even if they would be excluded).\n" + " -TOKEN — Remove matching tests from selection.\n" + " TOKEN* — Wildcard: match all tokens starting with TOKEN (e.g., ISAR* → {ISAR, ISAR_ENC}; BINAURAL* → {BINAURAL, BINAURAL_IR, BINAURAL_ROOM_IR, ...}).\n" + " +TOKEN*, -TOKEN* — Wildcard with add/remove modifiers (e.g., +ISAR*, -BINAURAL*).\n" + "\n" + "Examples:\n" + " --filter LEVEL1 Run LEVEL1 ENC+DEC (≤80 kbps).\n" + " --filter LEVEL2 Run LEVEL2 ENC+DEC (≤192 kbps).\n" + " --filter LEVEL1 DEC MONO Restrict to DEC MONO tests at LEVEL1.\n" + " --filter +REND +ISAR Add REND and ISAR to baseline ENC+DEC.\n" + " --filter DEC HOA Run DEC tests with HOA2/HOA3 outputs.\n" + " --filter DEC -voip Run DEC tests except those matching 'voip'.\n" + " --filter ISAR* Add ISAR and ISAR_ENC (wildcard expansion).\n" + " --filter DEC +BINAURAL* Run LEVEL3 DEC + all BINAURAL variant outputs.\n" + " --filter +ISAR* -voip Add ISAR/ISAR_ENC groups then remove 'voip' tests.\n" + ), ) parser.add_argument( "--be-test", @@ -1673,11 +1925,10 @@ if __name__ == "__main__": help="Do not run DUT, use existing mld and bitdiff stats files to generate analysis only", ) parser.add_argument( - "-c", "--clean-output-dir", default=False, action="store_true", - help="Do not run DUT, use existing mld and bitdiff stats files to generate analysis only", + help="Delete and recreate the CUT_OUTPUTS directory before running, discarding any previous DUT outputs", ) parser.add_argument( "--regenerate-mld-ref", @@ -1685,7 +1936,25 @@ if __name__ == "__main__": action="store_true", help="Run analysis and unconditionally regenerate mld_ref2 files for all tags", ) - args = parser.parse_args() + # Preprocess sys.argv to handle --filter with minus-prefixed tokens like -JBM + modified_argv = _preprocess_filter_args() + args = parser.parse_args(modified_argv[1:]) # Skip program name; parse_args expects args without it + + # Unescape filter tokens that were escaped during sys.argv preprocessing + if args.filter: + marker_prefix = "@FILT_" + unescaped_filter = [] + for tok in args.filter: + # Unescape any tokens that were wrapped by preprocessing + if tok in _FILTER_ESCAPES: + unescaped_filter.append(_FILTER_ESCAPES[tok]) + else: + unescaped_filter.append(tok) + args.filter = unescaped_filter + + # Explicit LEVEL3 means default level behavior (same as no explicit level token). + if args.filter and len(args.filter) == 1 and args.filter[0].upper() == "LEVEL3": + args.filter = None if not os.path.isdir(args.testvecDir): parser.error( @@ -1697,9 +1966,139 @@ if __name__ == "__main__": if args.cut_build_path: validate_build_binaries(parser, args.cut_build_path, "CUT") + # Parse --filter into level + optional tag selection + optional format/substring filters. + raw_filter = " ".join(args.filter) if args.filter else "" + filter_tokens = [tok for tok in re.split(r"[\s,]+", raw_filter.strip()) if tok] + + valid_tags = set(IVAS_Bins.keys()) + valid_levels = {"LEVEL1", "LEVEL2", "LEVEL3"} + valid_decoder_formats = set(DECODER_OUTPUT_FORMATS).union( + DECODER_OUTPUT_FORMAT_ALIASES.keys() + ) + + level_tokens = [] + tag_tokens = [] + tag_add_tokens = [] + tag_remove_tokens = [] + decoder_format_tokens = [] + restrictive_terms = [] + additive_terms = [] + subtractive_terms = [] + + for tok in filter_tokens: + sign = "" + base_tok = tok + if len(tok) > 1 and tok[0] in {"+", "-"}: + sign = tok[0] + base_tok = tok[1:] + + upper_tok = base_tok.upper() + + # Prefix wildcard selection for tags and decoder output formats, e.g. ISAR*, BINAURAL*. + if upper_tok.endswith("*") and len(upper_tok) > 1: + prefix = upper_tok[:-1] + matched_tags = sorted(t for t in valid_tags if t.startswith(prefix)) + matched_format_tokens = sorted( + f for f in valid_decoder_formats if f.startswith(prefix) + ) + + if not matched_tags and not matched_format_tokens: + parser.error( + f"Wildcard token '{tok}' did not match any known tag or decoder output format." + ) + + if sign == "+": + tag_add_tokens.extend(matched_tags) + elif sign == "-": + tag_remove_tokens.extend(matched_tags) + else: + tag_tokens.extend(matched_tags) + + expanded_wildcard_formats = [] + for fmt in matched_format_tokens: + expanded_wildcard_formats.extend( + sorted(DECODER_OUTPUT_FORMAT_ALIASES.get(fmt, {fmt})) + ) + + if sign == "+": + additive_terms.extend(expanded_wildcard_formats) + elif sign == "-": + subtractive_terms.extend(expanded_wildcard_formats) + else: + decoder_format_tokens.extend(expanded_wildcard_formats) + + continue + + if upper_tok in valid_levels: + if sign: + parser.error(f"Level token '{tok}' cannot be prefixed with '+' or '-'.") + level_tokens.append(upper_tok) + elif upper_tok in valid_tags: + if sign == "+": + tag_add_tokens.append(upper_tok) + elif sign == "-": + tag_remove_tokens.append(upper_tok) + else: + tag_tokens.append(upper_tok) + elif upper_tok in valid_decoder_formats: + expanded_formats = DECODER_OUTPUT_FORMAT_ALIASES.get(upper_tok, {upper_tok}) + if sign == "+": + additive_terms.append(base_tok) + elif sign == "-": + subtractive_terms.append(base_tok) + else: + decoder_format_tokens.extend(sorted(expanded_formats)) + else: + if sign == "+": + additive_terms.append(base_tok) + elif sign == "-": + subtractive_terms.append(base_tok) + else: + restrictive_terms.append(base_tok) + + if len(set(level_tokens)) > 1: + parser.error("Multiple filter levels specified. Use only one of LEVEL1, LEVEL2, LEVEL3.") + + filter_level = level_tokens[0] if level_tokens else "LEVEL3" + + # Preserve order while removing duplicates. + tag_tokens = list(dict.fromkeys(tag_tokens)) + tag_add_tokens = list(dict.fromkeys(tag_add_tokens)) + tag_remove_tokens = list(dict.fromkeys(tag_remove_tokens)) + decoder_format_tokens = list(dict.fromkeys(decoder_format_tokens)) + + # All levels share the same default tag baseline: ENC + DEC. + # REND/ISAR/ISAR_ENC are optional and must be explicitly selected or added. + # Plain tag_tokens restrict the baseline; +tag_tokens add beyond it. + if tag_tokens: + selected_tag_set = set(tag_tokens) + else: + selected_tag_set = {"ENC", "DEC"} + for tag in tag_add_tokens: + selected_tag_set.add(tag) + for tag in tag_remove_tokens: + selected_tag_set.discard(tag) + testTags = [tag for tag in IVAS_Bins.keys() if tag in selected_tag_set] + + args.filter_display = raw_filter if raw_filter else None + args.filter_restrictive_terms = restrictive_terms + args.filter_add_terms = additive_terms + args.filter_remove_terms = subtractive_terms + args.filter_decoder_formats = decoder_format_tokens + args.filter_level = filter_level + conformance = MLDConformance(args) conformance.accumulateCommands() + if args.filter_display: + print(f"Applying filter: {args.filter_display}") + print() + print("No of tests:") + for tag in IVAS_Bins.keys(): + n = len(conformance.getSelectedTestsForTag(tag)) if tag in testTags else 0 + print(f" {tag} : {n}") + print() + if args.regenerate_enc_refs: conformance.runReferenceGeneration(encTag="ISAR_ENC") conformance.runReferenceGeneration(encTag="ENC") @@ -1709,7 +2108,6 @@ if __name__ == "__main__": if args.regenerate_mld_ref: args.analyse = True - testTags = IVAS_Bins.keys() if args.test_mode is None else [args.test_mode] tag_results = {} for tag in testTags: if args.report_only: @@ -1727,4 +2125,5 @@ if __name__ == "__main__": for tag in testTags: tag_status = "OK" if tag_results.get(tag, False) else "FAILED" print(f"[{tag}] {tag_status}") + print()