Improve speed of comparison in PyTest tests
While working with some tests in development, I noticed that the cmp_custom
-function used in our PyTest-based tests is a bit slow. It does a very thorough work but this ends up making passing BE tests significantly slower than they need to. To give some data, we can compare the length of a self test run using functions cmp_custom
and filecmp.cmp
(probably fastest binary diff we can get). With no references present, the run times on my 16-core laptop are 4 min 39 s and 3 min 12 s respectively. If references are present, then this becomes 3 min 7 s and 1 min 36 s respectively. These values are for a fully passing test. For reference, I used this kind of test invocation with prepared binaries:
pytest tests/test_param_file.py --update_ref=2 --dut_encoder_path=build/IVAS_cod --dut_decoder_path=build/IVAS_dec --ref_encoder_path=build/IVAS_cod --ref_decoder_path=build/IVAS_dec
As in usual use case most of the tests should pass, an easy optimization would be to check first the fast diff using filecmp.cmp
and if that fails, then calculate a more detailed diff. This would of course cause some overhead for failing cases but might be a reasonable sacrifice. A comprehensive optimization would be to optimise or rewrite the cmp_custom
-function. E.g., files could be read in one go instead of one sample at a time.