Application-transparent fault tolerance in distributed systems

We present a new software architecture in which all concepts necessary to achieve fault tolerance can be added to an application automatically without any source code changes. As a case study, we consider the problem of providing a reliable service despite node failures by executing a group of replicat-ed servers. Replica creation and management as well as failure detection and recovery are performed automatically by a separate fault tolerance layer (ft-layer) which is inserted between the server application and the operating system kernel. The layer is invisible for the application since it provides the same functional interface as the operating system kernel, thus making the fault tolerance property of the service completely transparent for the application. A major advantage of our architecture is that the layer encapsulates both fault tolerance mechanisms and policies. This allows for maximum flexibility in the choice of appropriate methods for fault tolerance without any changes in the application code. Due to the steadily increasing complexity of application programs more and more sophisticated concepts are required to make these programs resistent against system component failures. During the past years several methods for achieving this fault tolerance have been developed and meanwhile are well understood. Basic concepts which are necessary for reliable applications have been identified and thoroughly studied (process surveillance [Bec91], membership information distribution [Cri91], checkpoint-ing and recovery [KoT87], reliable and order-preserving multicast protocols [BeG93], various replication mechanisms [Pow91], etc.). However, up to now little has been done to support a non-expert application programmer in building reliable applications. Although their development is facilitated by toolkits (e.g. the ISIS toolkit [BiJ87]) which provide the basic building blocks of fault tolerance, the programmer still requires precise knowledge of the concepts implemented by these toolkits in order to use them successfully. Even with the help of toolkits, fault tolerance policies and mechanisms still have to be programmed explicitly as part of the application code. This close interrelation of the fault tolerance mechanisms and the application code has two severe drawbacks: • Distributed fault tolerant applications become much more difficult to test because the implemented fault tolerance concepts have to be taken into account when selecting proper test scenarios. Often the failure handling parts of such an application are particularly difficult to test as subtle combinations of failure situations have to be simulated. • Once the fault tolerance policy is chosen, the application is restricted only to this choice. Switching …