Evaluating Non-Adequate Test-Case Reduction

چکیده

Given two test cases, one longer and one shorter, the shorter test case is preferred for many purposes. A shorter test case usually runs faster, is easier to understand, and is more convenient for debugging. However, shorter test cases also tend to cover less code and detect fewer faults than longer test cases. While traditional work focused on reducing test suites while preserving code coverage, one line of recent work has introduced the idea of reducing individual test cases, rather than test suites, while still preserving code coverage; and another line of recent work has proposed non-adequately reducing test suites. This paper empirically evaluates a new combination of these ideas: non-adequate reduction of test cases, which allows for a wide range of trade-offs between test case size and fault detection. Our study introduces and evaluates C%-coverage reduction (where a test case is reduced to retain at least C% of its original coverage) and N-mutant reduction (where a test case is reduced to kill at least N of the mutants it originally killed). We evaluate the reduction trade-offs with varying values of C and N for four real-world C projects, using both manually written and automatically generated test cases: Mozilla’s SpiderMonkey JavaScript engine, the YAFFS2 flash file system, Grep, and Gzip. The results show that it is possible to substantially reduce the size of many test cases while still preserving much of their fault-detection capability. [Global TODOs: 1. mention C%-coverage beforeN -mutant 2. order subjects in a consistently : SpiderMonkey, YAFFS2, Grep, Gzip3. command-line arguments versus parameters (use macro [parameters]) 4. Use top captions for tables and bottom captions for figures consistently 5. Stick with either decimal or percentage for figures. ]