Skip to content

Coverage analysis of test vectors

A coverage analysis of the updated test vector set has been done following these steps:

  1. Compile code with make GCOV=1
  2. Put test vectors to be tested in scripts/testv
  3. Run self_test.py --create -t 1 (optionally pointing out config/self_test_evs.prm for EVS coverage test)
  4. Collect output with lcov -c -d ../obj -o coverage.info
  5. Generate HTML output with genhtml coverage.info -o coverage_html
  6. Potentially reset counters with lcov -z -d ../obj in case a new measurement should be done.

An old test vector set was used as reference when doing the comparison. In total, five coverage analyses were done:

  1. coverage_ivas_old.zip: self_test.prm, old testv (IVAS operating points)
  2. coverage_ivas_new.zip: self_test.prm, new testv (IVAS operating points)
  3. coverage_evs_old.zip: self_test_evs.prm, old testv (EVS operating points)
  4. coverage_evs_new.zip: self_test_evs.prm, new testv (EVS operating points)
  5. coverage_evs_26444.zip: self_test_evs.prm, testv based on 26.444 (EVS operating points)

Coverage summary IVAS:

testv Lines Functions
old 78.0% (92577 of 118748 lines) 84.4% (3091 of 3662 functions)
new 78.4% (93048 of 118748 lines) 84.5% (3094 of 3662 functions)

Coverage summary EVS:

testv Lines Functions
old 49.5% (58762 of 118753 lines) 56.7% (2077 of 3662 functions)
new 48.0% (57050 of 118753 lines) 55.7% (2040 of 3662 functions)
26.444-based 49.9% (59281 of 118753 lines) 56.8% (2081 of 3662 functions)

Since the test vectors used for EVS had a drop in overall coverage, a variant based on the 26.444 EVS test vectors was created. These are simply the input files with .INP renamed to .pcm, and the stvn{1,2}.pcm were concatenated into one noise vector stvn.pcm. If this variant can be used remains to be checked.

Details on lost and added coverage of lines and functions can be seen in the following attached text files:

  1. coverage_diff_ivas.txt
  2. coverage_diff_evs.txt
  3. coverage_diff_evs_26444.txt

Please note there is a section on lost coverage followed by a section on added coverage. Please feel free to review the added and lost coverage.

Edited by norvell