Sources of Failure in the Public Switched Telephone Network

To operate successfully, most large distributed systems depend on software, hardware, and human operators and maintainers to function correctly. Failure of any one of these elements can dis­ rupt or bring down an entire system. One such distributed system, the US Public Switched Telephone Network (PSTN), is the US portion of pos­ sibly the largest distributed system in existence.1 Like all telephone switching networks, the PSTN performs a fairly simple task: It connects point A with point B. Paradoxically, this seemingly trivial task requires some of the most complex and sophisticated computing sys­ tems in existence. Software for a switch with even a relatively small set of features may comprise several million lines of code. The PSTN contains thousands of switches. Switches include redundant hardware and extensive self-check­ ing and recovery software. For several decades, AT&T has expected its switches to experience not more than two hours of failure in 40 years,2 a failure rate of 5.7 × 10− 6. Since 1992, telephone companies have been required to notify the US Federal Communications Commission (FCC) of outages affecting more than 30,000 customers. I used these outage records to deter­ mine the principal causes of PSTN failures. To account for the possible effects of seasonal fluctuations in callprocessing volume, I analyzed failures over two years, from April 1992 to March 1994, beginning with the earliest FCC reports. I made quantitative measures of how each failure source affects system dependability, in an effort to shed some light on the dependability of different components (including software). The PSTN’s dependability stems from a design that successfully exploits the loose coupling of system com­ ponents. Because the PSTN has many similarities with other types of distributed systems, the analysis may suggest factors to consider in the design of distributed systems in general. Major sources of failure were human error (on the part of both telephone company personnel and oth­ ers), acts of nature, and overloads. Overloads caused nearly half of all downtime (44 percent) in terms of outage minutes. An unexpected finding, given the complexity of the PSTN and its heavy reliance on software, was that software errors caused less system downtime (2 per­ cent) than any other source of failure except vandal­ ism. Hardware and software failures were similar in terms of average number of customers affected (96,000 and 118,000) and duration of outage (160 and 119 minutes). Errors on the part of telephone company personnel and acts of nature caused similar amounts of down­ time (14 and 18 percent).