Effect of Testing Coverage on Software Reliability - an Experimental Investigation

Logical expressions are often used to formalize software specifications of safety-critical systems. These logical expressions can be tested using software testing methods (criteria) that include Decision Coverage (DC), Condition Coverage (CC), Decision/Condition (D/CC), and Modified Condition/Decision Coverage (MC/DC). Selection of the appropriate testing method is an important practical task. A significant characteristic for this selection process is understanding the effect of testing methods on software reliability, specifically their ability to reveal faults. This paper provides experimental results for determining the probabilistic characteristics of effectiveness of testing criteria. A logical expression, which is typical for nuclear reactor protection system logic, is used as a case study for this research. Probabilities for a test set to reveal a fault in the logical expression are evaluated for DC, CC, D/CC, and MC/DC. Our experimental results show that, when compared with random testing, using DC, CC, or D/CC criteria do not provide significant benefits. At the same time, the results confirm that MC/DC is a reasonable and effective technique to test logical expressions in software.