Database Availability for Transaction Processing

Modern businesses store A transaction processing critical data in database system relies on its management systems. Much database management of the daily activity system to supply high of business includes availability. Digital manipulation of data offers a network-based in the database. As product, the VAX DBMS businesses extend their system, and a relational operations worldwide, data-based product, the their databases are shared VAX Rdb/VMS database among office locations system, for its transaction in different parts of processing systems. These the world. Consequently, database systems have these businesses require several strategies to transaction processing survive failures, disk systems to be available head crashes, revectored for use at all times. This bad blocks, database requirement translates corruptions, memory directly to a goal of corruptions, and memory perfect availability overwrites by faulty for database management application programs. systems. They use base hardware VAX DBMS and VAX Rdb/VMS technologies and also database systems are based employ novel software on network and relational techniques, such as data models, respectively. parallel transaction Both systems use a kernel recovery, recovery on of code that is largely surviving nodes of a responsible for providing VAXcluster system, high availability. This restore and roll-forward layer of code is maintained operations on areas of the by the KODA group. KODA database, on-line backup, is the physical subsystem verification and repair for VAX DBMS and VAX Rdb utilities, and executive /VMS database systems. mode protection of trusted It is responsible for all database management system I/O, buffer management, code. concurrency control, Introduction transaction consistency, locking, journaling, and access methods. Digital Technical Journal Vol. 3 No. 1 Winter 1991 1 Database Availability for Transaction Processing In this paper, we define of downtime are useful. database availability, Unexpected downtime is and describe downtime caused by factors that situations and how such are beyond the control of situations can be resolved. the transaction processing We then discuss the system. For example, a disk mechanisms that have failure is quite possible been implemented to at any time during normal provide minimal loss of processing of transactions. availability. However, scheduled downtime is entirely within the Database Availability control of the database administrator. High The unit of work in availability demands that transaction processing we eliminate scheduled systems is a transaction. downtime and ensure fast We therefore define system recovery from database availability as unexpected failures. the ability to execute The layers of the software transactions. One way the and hardware services which database management system compose a transaction provides high availability processing system are is by guaranteeing the dependent on one another properties of transactions: for high availability. atomicity, serializability, The dependency among these and durability.[1] For services is illustrated example, if a transaction in Figure 1. Each service that has made updates to depends on the availability the database is aborted, of the service in the other transactions must lower layers. Errors and not be allowed to see these failures can occur in updates; the updates made any layer, but may not by the aborted transaction be detected immediately. must be removed from the For example, in the case database before other of a database management transactions may use that system, the effects of a data. Yet, data that has database corruption may not been accessed by the not be apparent until aborted transaction must long after the corruption continue to be available to (error) has occurred. other transactions. Hence it is difficult to Downtime is the deal with such errors. On term used to refer the other hand, failures to periods when the are noticed immediately. database is unavailable. Failures usually make the Downtime is caused by system unavailable and are either an unexpected the cause of unexpected failure (unexpected downtime. downtime) or scheduled maintenance on the database (scheduled downtime). Such classifications 2 Digital Technical Journal Vol. 3 No. 1 Winter 1991 Database Availability for Transaction Processing Each layer can provide Application Program only as much availability Exceptions as the immediate lower Although transaction layer. Hence we can also processing systems are express the perfectbased on the client availability goal of a /server architecture, database management system Digital's database systems as the goal of matching are process based. The the availability of the privileged database immediately lower layer, management system code which in our case is the is packaged in a shareable operating system. library and linked with At the outset, it is clear the application programs. that a database management Therefore, bugs in the system layered on top of applications have a an operating system and good chance of affecting hence only as available as the consistency of the the underlying operating database. Such bugs in system. However, a database applications are one type management system is in of failure that can make general not as available the database unavailable. as the underlying layer The VAX DBMS and VAX Rdb because of the need to /VMS systems guard against guarantee the properties of this class of failure by transactions. executing the database management system code Unexpected Downtime in VAX'S executive mode. Since application programs In this section we discuss execute in user mode, they the causes of unexpected do not have access to data downtime and the techniques structures used by the that minimize downtime. database management system. A database monitor must be When a faulty application started on a node before program attempts such an a user's process running access, the VMS operating on that node can access system detects it and a database. The monitor generates an exception. oversees all database This exception then forces activity on the node. an image rundown of the It allows processes to application program. attach to and detach from In general, when an image databases and detects rundown is initiated, failures. On detecting Digital's database a failure, the monitor management products use starts a process to recover the condition-handling the transactions that did facility of VMS to abort not complete because of the transaction. Condition the failure. Note that handling of image rundown this database monitor is performed at two levels. is different from the TP Two condition handlers monitor. [2] are established, one in Digital Technical Journal Vol. 3 No. 1 Winter 1991 3 Database Availability for Transaction Processing user mode and the other in is examined at different kernel mode. The user mode points in the code. If any exit handler is usually inconsistency is found, a invoked, which rolls back bug-check utility is called the current transaction that dumps the internal and unbinds it from the database format to a file. database. In this case, the The utility then raises an rest of the users on the exception that is handled system are not affected at by the monitor, and the all. The database remains DBR process is started as available. The execution of described above. the user mode exit handler To deal with corruptions is, however, not guaranteed to the database that are by the VMS operating undetected with this system. Under some abnormal mechanism, an explicit circumstances, the user utility is provided that mode exit handlers may verifies the structural not be executed at all. consistency of the In such circumstances, the database. This verify kernel mode exit handler is utility may be executed invoked by the VMS system. on-line, while users This handler resides in are still accessing the the database monitor. The database. Such verification monitor starts a database may also be executed by recovery (DBR) process. a database administrator It is the responsibility (DBA) in response to a bugof the DBR process to roll check dump. Once such a back the effects of the corruption is detected, an aborted transaction. To do on-line utility provides this, the DBR process first the ability to repair the establishes a database database. freeze. This freeze prevents other processes In general, corruption in from acquiring locks that databases causes unexpected were held by the aborted downtime. Digital provides transaction and hence see the means of detecting and update uncommitted such corruption on-line data. (The VMS lock manager and repairing them on-line releases all locks held by through recovery utilities. a process when that process Process Failure dies.) The DBR process then proceeds to roll back the In the VMS system, a aborted transaction. process failure is always Code Corruptions preceded by image rundown of the current image It is important to prevent running as part of the coding mistakes within the process. Therefore, a DBMS from irretrievably process failure is detected corrupting the database. by the database monitor, To protect the database which then starts a DBR management system from process to handle recovery. coding mistakes, internal Node Failure data structure consistency 4 Digital Technical Journal Vol. 3 No. 1 Winter 1991 Database Availability for Transaction Processing Among the many mechanisms the recovery. The only Digital provides for differences in the case of availability is node process, node, or cluster failover within a cluster. failure is the mechanism When a node fails, another by which the monitor is node on the cluster detects informed of the failure. the failure and rolls back Disk Head Crash the lost transactions from the failed node. Thus the Some failures can result failure of one node does in the loss or corruption not cause transactions of the data on the stable on other active nodes storage device (disk). of the cluster to come Digital has a mechanism for to a halt (except for bringing the database back the time the DBR process to a consistent state in enforces a freeze). It is such cases. the database monitor that A disk head crash is a detects node failure and failure of hardware that starts a recovery process is usually characterized by for every lost transaction the inability to read from on the failed node. The or write to the disk. Hence database becomes available database storage areas as soon as recovery is residing on that disk are complete for all the users unavailable and possibly on the failed node. irretrievable. A disk Power Failure head crash automatically Power failure is a hardware aborts transactions that failure. As soon as power need to read from or is restored, the VMS system write to that disk. In boots. When a process addition, recovery of these attaches to the database, aborted transactions is a number of messages are not possible since the passed between the process recovery processes need that is attaching and the access to the same disk. monitor. If the database is In this case, the database corrupt (because of power is shut down and access is failure), the monitor is so denied until the storage informed by the attaching areas on the failed disk process, and again the are brought on-line. Areas monitor starts recovery are restored from backups processes to return the and then rolled forward database to a consistent until consistent with the state. The database becomes rest of the database. The available as soon as after image journal (AIJ) recovery is complete for files are used to roll all such failed users. the areas forward. As soon as all the areas on As described above, the failed disk have been recovery is always restored onto a good disk accomplished by the and rolled forward, the monitor process starting database becomes available. DBR processes to do Bad Disk Blocks Digital Technical Journal Vol. 3 No. 1 Winter 1991 5 Database Availability for Transaction Processing Bad blocks are hardware off against the downtime errors that often are needed to restore and roll not detected when they forward. happen. The bad blocks are Site Failure revectored, and the next time the disk block is A site failure occurs when read, an error is reported. neither the computers nor Bad blocks simply mean the disks are available. A that the contents of a disk site failure is usually block are lost forever. caused by a natural The database administrator disaster such as an detects the problem only earthquake. The best when a database application recourse for recovery is fails to fetch data on the archival storage. Digital revectored block. Such an provides mechanisms to error may cause a certain back up the database and transaction or a set of AIJ files to tape. These transactions to fail, no tapes must then be stored matter how many attempts at a site away from the are made to execute site at which the database the transactions. This resides. Should a disaster failure constitutes reduced happen, these backup tapes availability; parts of the can be used to restore database are unavailable the database. However, to transactions. Exactly the recovery may not be how much of the database complete. It cannot restore remains available depends the effects of those on which blocks were committed transactions that revectored. were not backed up to tape. The mechanism provided After a disaster, the to reduce the possible database can be restored downtime is early and rolled forward to the detection. Digital's state of the completion database systems provide of the last AIJ that a verification utility that was backed up to tape. can be executed while users Any transactions that are running transactions. committed after the last The verification utility AIJ was backed up cannot be checks the structural recovered at the alternate consistency of the site. Such transaction database. Once a bad block losses can be minimized by is detected by such a frequently backing up the utility, that area of the AIJ files. database may be restored Memory Errors and rolled forward. These two operations make the Memory errors are quite whole database temporarily infrequent, and when they unavailable; however, the happen, they usually are bad block is corrected, and not detected. If the error future downtime is avoided. happens to a data record, The downtime caused by the it may never be detected bad block may be traded by any utility, but may 6 Digital Technical Journal Vol. 3 No. 1 Winter 1991 Database Availability for Transaction Processing be seen as incorrect Digital's database data by the user. If the systems allow two types verification utility is of transactions: update run on-line, it may also and "snapshot." The ability detect the errors. Again, to back up data on-line the database may only be depends on the snapshot partially available, as transaction capability of in the case of bad blocks. the database. However, it is possible to Database backup is a repair the database while standard way of recovering users are still accessing from media failures. the database. Digital's Digital's database systems database management provide the ability to products provide explicit do transaction consistent repair facilities for backups of data on-line this purpose. The loss of while users continue to availability during repair change the database. is not worse than the loss due to the memory error The general scheme for itself. snapshot transactions is As explained previously, as follows. The update the database monitor transactions of the plays an important database preserve the part in ensuring previous versions of the database consistency database records in the and availability. Most snapshot file. All versions unexpected failure of a database record are scenarios are detected chained. Only the current by the monitor, which then version of the record is starts recovery processes. in the database area. The In addition, some failures older versions are kept might require the use of in the snapshot area. The backup files to restore the versions of the records are database. tagged with the transaction numbers (TSNs). When a snapshot transaction (for Scheduled Downtime example, a database backup) Most database systems needs to read a database have scheduled maintenance record, it traverses the operations that require a chain for that database database shutdown. Database record and then uses the backup for media recovery appropriate version of the and verification to check record. structural consistency There are two modes are examples of operations of database operation that may require scheduled with respect to snapshot downtime. In this section activity. In one mode, we describe ways to perform all update transactions many of these operations write snapshot copies of while the database is any records they update. executing transactions. In the deferred snapshot Backup mode, the updates cause Digital Technical Journal Vol. 3 No. 1 Winter 1991 7 Database Availability for Transaction Processing snapshot copies to be database to check the written only if a snapshot structural consistency transaction is active and of the database. These requires old versions of utilities may also be a record. In this mode, a executed on-line through snapshot transaction cannot the use of snapshot start until all currently transactions. active update transactions AIJ Backup (which are not writing snapshot records) have The backup and the AIJ completed; that is, the log are the two mechanisms snapshot transaction must that provide media recovery wait for a quiet point in for Digital's database time. If there are either management products. The active or pending snapshot AIJ file is continuously transactions when an update written to by all user transaction starts, the processes updating the update transaction must database. We need to write snapshot copies. provide some ability Here we see a trade-off to back up the AIJ file between update transactions since it monotonically and snapshot transactions. increases in size and The database is completely eventually fills up the available to snapshot disk. Digital's database transactions if all update systems offer the ability transactions always write to back up the AIJ file snapshot copies. On the to tape (or another other hand, if the deferred device) on-line. The only snapshot mode is enabled, restriction is that a quiet update transactions point must be established need not write snapshot for a short period during copies if a snapshot which the backup operation transaction in not active. takes place. A quiet point This approach obviously is defined as a point when results in some loss of the database is quiescent, availability to snapshot i.e., there are no active transactions. transactions. Verification On-line Schema Changes Database corruption Digital's database can also result in management systems allow downtime. Although database users to change metadata corruption is not probable, on-line, while users it is possible. Any are still accessing the database system that database. Although this may supports critical data be standard for relational must provide facilities database management to ensure the consistency systems, it is not standard of the database. Digital's for network databases. The database management systems VAX DBMS system provides a provide verification utility called the database utilities that scan the restructuring utility (DRU) 8 Digital Technical Journal Vol. 3 No. 1 Winter 1991 Database Availability for TransactionProcessing to allow for on-line schemaReferencesmodifications.1. P. Bernstein, W.AcknowledgmentsEmberton, and V.Trehan, "DECdta -Many engineers haveDigital's Distributedcontributed to theTransaction Processingdevelopment of theArchitecture," Digitalalgorithms described inTechnical Journal, vol.this paper. We have chosen3, no. 1 (Winter 1991,not to enumerate all suchthis issue): 10-17.contributions. However,2. T. Speer and M. Storm,we would like to recognize"Digital's Transactionthe contributions of PeterProcessing Monitors,"Spiro, Ashok Joshi, JeffDigital TechnicalArnold, and Rick AndersonJournal, vol. 3, no.who, together with the1 (Winter 1991, thisauthors, are members of theissue): 18-32.KODA team. Digital Technical Journal Vol. 3 No. 1 Winter 19919=============================================================================Copyright 1991 Digital Equipment Corporation. Forwarding and copying of thisarticle is permitted for personal and educational purposes without feeprovided that Digital Equipment Corporation's copyright is retained with thearticle and that the content is not modified. This article is not to bedistributed for commercial advantage. Abstracting with credit of DigitalEquipment Corporation's authorship is permitted. All rights reserved.=============================================================================