Commit 60781006 authored by Vladimir Malenovsky's avatar Vladimir Malenovsky
Browse files

unify --test-mode and --filter into --filter

parent 74ca16fc
Loading
Loading
Loading
Loading
+13 −73
Original line number Diff line number Diff line
@@ -247,94 +247,34 @@ MLD Corridor passed for ISAR with max MLD diff of 0.0

## Executing specific tests only

All CUT tests can be run specifically for IVAS Encoder,IVAS Decoder,IVAS Renderer, ISAR Encoder and ISAR Decoder only. The commandline allows for ```-test-mode=<PARAM>``` for this functionality, examples:
Use `--filter` to run or analyze a subset of tests.

- Run CUT IVAS Encoder Tests Only (on Target Platform)
- Tag selection is supported with comma-separated tags: `ENC`, `DEC`, `REND`, `ISAR`, `ISAR_ENC`.
- Command substring filtering is also supported.
- Matching is case-insensitive.

  ```shell
  PYTHONPATH=scripts python scripts/ivas_conformance/runConformance.py --testvecDir $PWD/testvec --cut_build_path=CUT_BIN_DIR --test-mode=ENC
  ```

- Analyse BE conformance for CUT IVAS Encoder Outputs Only (on Reference Platform)

  ```shell
  PYTHONPATH=scripts python scripts/ivas_conformance/runConformance.py --testvecDir $PWD/testvec --ref_build_path=testvec/bin --test-mode=ENC --analyse --be-test
  ```

- Analyse NON-BE conformance for CUT IVAS Encoder Outputs Only (on Reference Platform)

  ```shell
  PYTHONPATH=scripts python scripts/ivas_conformance/runConformance.py --testvecDir $PWD/testvec --ref_build_path=testvec/bin --test-mode=ENC --analyse
  ```

- Run CUT IVAS Decoder Tests Only (on Target Platform)

  ```shell
  PYTHONPATH=scripts python scripts/ivas_conformance/runConformance.py --testvecDir $PWD/testvec --cut_build_path=CUT_BIN_DIR --test-mode=DEC
  ```

- Analyse BE conformance for CUT IVAS Decoder Outputs Only

  ```shell
  PYTHONPATH=scripts python scripts/ivas_conformance/runConformance.py --testvecDir $PWD/testvec --test-mode=DEC --analyse --be-test
  ```

- Analyse NON-BE conformance CUT IVAS Decoder Outputs Only (on Reference Platform)

  ```shell
  PYTHONPATH=scripts python scripts/ivas_conformance/runConformance.py --testvecDir $PWD/testvec  --test-mode=DEC --analyse
  ```

- Run CUT IVAS Renderer Tests Only (on Target Platform)

  ```shell
  PYTHONPATH=scripts python scripts/ivas_conformance/runConformance.py --testvecDir $PWD/testvec --cut_build_path=CUT_BIN_DIR --test-mode=REND
  ```

- Analyse BE conformance for CUT Renderer Outputs Only 

  ```shell
  PYTHONPATH=scripts python scripts/ivas_conformance/runConformance.py --testvecDir $PWD/testvec --test-mode=REND --analyse --be-test
  ```

- Analyse NON-BE conformance CUT Renderer Outputs Only 

  ```shell
  PYTHONPATH=scripts python scripts/ivas_conformance/runConformance.py --testvecDir $PWD/testvec --test-mode=REND --analyse
  ```

- Run CUT ISAR Encoder Tests Only (on Target Platform)

  ```shell
  PYTHONPATH=scripts python scripts/ivas_conformance/runConformance.py --testvecDir $PWD/testvec --cut_build_path=CUT_BIN_DIR --test-mode=ISAR_ENC
  ```

- Analyse BE conformance for CUT ISAR Encoder Outputs Only (on Reference Platform)

  ```shell
  PYTHONPATH=scripts python scripts/ivas_conformance/runConformance.py --testvecDir $PWD/testvec --ref_build_path=testvec/bin --test-mode=ISAR_ENC --analyse --be-test
  ```
Examples:

- Analyse NON-BE conformance for CUT ISAR Encoder Outputs Only (on Reference Platform)
- Run all renderer tests only:

  ```shell
  PYTHONPATH=scripts python scripts/ivas_conformance/runConformance.py --testvecDir $PWD/testvec --ref_build_path=testvec/bin --test-mode=ISAR_ENC --analyse
  PYTHONPATH=scripts python scripts/ivas_conformance/runConformance.py --testvecDir $PWD/testvec --cut_build_path=CUT_BIN_DIR --filter rend
  ```

- Run CUT ISAR Decoder Tests Only (on Target Platform)
- Analyze BE conformance for encoder tests only:

  ```shell
  PYTHONPATH=scripts python scripts/ivas_conformance/runConformance.py --testvecDir $PWD/testvec --cut_build_path=CUT_BIN_DIR --test-mode=ISAR
  PYTHONPATH=scripts python scripts/ivas_conformance/runConformance.py --testvecDir $PWD/testvec --ref_build_path=testvec/bin --analyse --be-test --filter enc
  ```

- Analyse BE conformance for CUT ISAR Decoder Outputs Only
- Analyze non-BE conformance for DEC and REND tests that include `voip` in the command:

  ```shell
  PYTHONPATH=scripts python scripts/ivas_conformance/runConformance.py --testvecDir $PWD/testvec  --test-mode=ISAR --analyse --be-test
  PYTHONPATH=scripts python scripts/ivas_conformance/runConformance.py --testvecDir $PWD/testvec --ref_build_path=testvec/bin --analyse --filter dec,rend voip
  ```

- Analyse NON-BE conformance CUT ISAR Decoder Outputs Only 
- Run only ISAR encoder tests:

  ```shell
  PYTHONPATH=scripts python scripts/ivas_conformance/runConformance.py --testvecDir $PWD/testvec  --test-mode=ISAR --analyse
  PYTHONPATH=scripts python scripts/ivas_conformance/runConformance.py --testvecDir $PWD/testvec --cut_build_path=CUT_BIN_DIR --filter isar_enc
  ```
+27 −11
Original line number Diff line number Diff line
@@ -935,8 +935,9 @@ class MLDConformance:
        failed_before = self.getFailedCommandCount()
        selectedTests = list()
        if self.filter:
            filter_text = self.filter.lower()
            for pyTestsTag in self.TestDesc[tag].keys():
                if self.filter in self.TestDesc[tag][pyTestsTag].rawCmdline:
                if filter_text in self.TestDesc[tag][pyTestsTag].rawCmdline.lower():
                    selectedTests.append(pyTestsTag)
        else:
            selectedTests = list(self.TestDesc[tag].keys())
@@ -1023,8 +1024,9 @@ class MLDConformance:

        selectedTests = []
        if self.filter:
            filter_text = self.filter.lower()
            for pyTestsTag in self.TestDesc[tag].keys():
                if self.filter in self.TestDesc[tag][pyTestsTag].rawCmdline:
                if filter_text in self.TestDesc[tag][pyTestsTag].rawCmdline.lower():
                    selectedTests.append(pyTestsTag)
        else:
            selectedTests = list(self.TestDesc[tag].keys())
@@ -1634,14 +1636,10 @@ if __name__ == "__main__":
        "--filter",
        type=str,
        default=None,
        help="Filter test based on text provided",
    )
    parser.add_argument(
        "--test-mode",
        type=str,
        default=None,
        choices=["ENC", "DEC", "REND", "ISAR", "ISAR_ENC"],
        help='Choose one test group to run ["ENC", "DEC", "REND", "ISAR", "ISAR_ENC"]. If omitted, all are run.',
        help=(
            "Filter tests. Supports tags ENC DEC REND ISAR ISAR_ENC (comma/space-separated) "
            "and case-insensitive command substring filtering. Examples: '--filter rend' or '--filter REND binaural'."
        ),
    )
    parser.add_argument(
        "--be-test",
@@ -1704,7 +1702,25 @@ if __name__ == "__main__":
    if args.regenerate_mld_ref:
        args.analyse = True

    testTags = IVAS_Bins.keys() if args.test_mode is None else [args.test_mode]
    # Parse --filter into optional tag selection + optional command substring filter.
    raw_filter = args.filter or ""
    filter_tokens = [tok for tok in re.split(r"[\s,]+", raw_filter.strip()) if tok]
    valid_tags = set(IVAS_Bins.keys())
    tag_tokens = []
    non_tag_tokens = []
    for tok in filter_tokens:
        upper_tok = tok.upper()
        if upper_tok in valid_tags:
            tag_tokens.append(upper_tok)
        else:
            non_tag_tokens.append(tok)

    # Preserve order while removing duplicates.
    tag_tokens = list(dict.fromkeys(tag_tokens))

    testTags = tag_tokens if tag_tokens else list(IVAS_Bins.keys())
    args.filter = " ".join(non_tag_tokens) if non_tag_tokens else None

    tag_results = {}
    for tag in testTags:
        if args.report_only: