Production Experience with Multi-terabyte Object Databases in High Energy Physics
چکیده
CERN, the European Organisation for Nuclear Research, is currently building a new large-scale particle accelerator, the Large Hadron Collider (LHC). Four major physics experiments are scheduled to start collecting data at the LHC in 2005. They will collect raw data with rates from 100MB/s and up to 1.5GB/s, resulting in approximately 5PB (5∗10 bytes) of raw and processed data per year. The total data sample collected over the running period of the LHC (over 15 years) is expected to be in the range of 100PB. To provide a solution for management issues of such large amount of persistent data the RD45 project has been started at CERN. RD45 proposed a storage system consisting of Object Oriented Database Management System (ODBMS) combined with a Mass Storage System (MSS). Since then a growing number of physics experiments at CERN and outside have adopted this strategy to successfully deploy data in production environments. We describe here some of the largest production deployments of object databases in High Energy Physics and present the experience we have gathered in recent months running multiple Objectivity/DB-based data stores of up to tens of TB. Introduction CERN, the European Organisation for Nuclear Research, is presently preparing for a new largescale particle accelerator, the Large Hadron Collider (LHC). LHC will replace its predecessor, the Large Electron-Positron collider (LEP), in the 27km long ring tunnel running under the SwissFrench border near Geneva. Four physics experiments formed by collaborations of scientists from many countries that participate in the CERN program are scheduled to start collecting data at the LHC in 2005. Two of these, ATLAS and CMS, will record about 1PB of data per year with the rate of around 100MB/second. The third experiment, ALICE, will have an exceptionally high data rate – up to 1.5GB/second, but its shorter activity time during a year will lead to a similar data volume. Taking into account the fourth experiment, LHCb, the global amount of raw, derived and simulated data is expected to reach approximately 5PB per year. As the LHC is expected to run for more then 15 years, the total volume of data that must be stored is in the order of 100PB (10 bytes). Data Processing in High Energy Physics The data volume is only part of the LHC data management challenge. Data complexity and highly distributed environment of the international collaborations contribute to it significantly. The data models in High Energy Physics (HEP) are quite complicated, reflecting both the internal structure of the large and complex particle detectors and the multitude of processing steps that are performed to extract scientific results from the initial input. The number of different data types (classes) used to model the detector measurements in an experiment can range from several hundreds to a few thousand. The data structure describing a single particle collision (called an event) contains thousands of individual objects with a large number of relations between them. Fortunately, except for the state of the detector itself, there is no correlation between the individual events, which makes it possible to process the data with a high degree of parallelism. Obtaining and storing the raw measurements is the first step in the data processing chain. Before the data can be analysed, it needs to be processed by detector specific reconstruction algorithms that produce higher level abstractions (e.g. particle tracks) from the basic electronic signals. The reconstructed objects are then stored in addition to the original data together with associated references. Since this processing step requires a significant computing power, it is typically performed on large processor farms, reconstructing many events in parallel. This however introduces the problem of concurrent updates of shared data. Once the reconstruction phase is finished, the analysis of the physics content can begin. It will be performed by a large community of physicists from hundreds of institutes worldwide. The RD45 Project During the early stages of the LHC software projects it became clear that the data management techniques used by the previous experiments would not be suitable for LHC. The existing storage packages that had been developed earlier inside the HEP community were not able to scale to the expected data volumes. Being FORTRAN based they were also poorly adapted to the new software environment that was being established. All LHC experiments have now committed to the object oriented technology and are implementing their core systems mainly in C++. Java is becoming increasingly popular, particularly in the area of data visualisation and user interfaces. The preferred data storage solution would therefore have to support the concepts of OO (abstraction, inheritance, and polymorphism) and allow storing and retrieving data from different OO languages. Taking into account the long lifetime of the LHC project, it would also need to be able to adapt to major changes in software and hardware technology. To look for a solution for the data persistence needs of the LHC experiments, the RD45 project has been initiated at CERN. RD45 performed a market survey, evaluating various technologies including persistent object managers, language extensions, object databases and object brokers. Resulting from this evaluation was the conclusion that only a full Object Database Management System (ODBMS) conforming to the ODBMS manifesto [ODBMS manifesto, Cattell] could fulfil the requirements for data intensive tasks in a distributed and heterogeneous environment. The RD45 focus moved therefore quite early to solutions compliant with ODMG [ODMG] ODBMS systems. During the evaluation of the commercially available ODBMS systems, the scalable architecture of Objectivity/DB made it an attractive basis for a multi-PB object store. Among the other surveyed products, including O2, ObjectStore and POET, only Versant showed similar on-paper scalability. Therefore, the bulk of the work in the HEP community focused on Objectivity/DB, although we continue to watch and evaluate alternatives, including ORDBMS. Objectivity/DB Architecture The storage hierarchy of Objectivity/DB consists of several layers: pages, containers, databases and a federation. The hierarchy is reflected in the 64-bit object identifiers (OIDs) that permit direct navigation to any persistent object. The highest-level storage entity, the federation, corresponds to a system of distributed databases with a central object schema and a database catalogue. A single federation may contain up to 2 databases. Each database maps to a single file that can be located on any host on the network. Data in a remote database can be accessed via the Objectivity/DB data server (AMS). Databases are divided into containers, which contain pages with objects. Containers are the smallest locking entity in Objectivity/DB. In principle, databases of any size can be created. In practice, however, when considering copying the file (i.e. onto tape), the manageable database file size is from 1GB to 10GB. With 10GB files, the maximum size of a federation would be 0.5PB. For the LHC start time (2005) we could assume using 100GB files. This would allow us to create federations of size 5PB (assuming filling every database to the limit, which is not possible in a real system), which is still below the required capacity for the LHC data store. An extension of the available address space has therefore been requested from Objectivity and it is hoped that it will be available in one of the next releases. Interface to a Mass Storage System Although disk capacity is growing fast, it is still not feasible to keep petabytes of data permanently on disk. Therefore, a large fraction of the data (up to 90%) needs to be stored on tape or similar media in a Mass Storage System. An integration layer between Objectivity/DB and MSS has been designed and implemented as a collaborative effort between Objectivity, SLAC and CERN and is today incorporated in the standard release of the database product. The design introduces a file system abstraction layer (OOFS) in the database data server (AMS), based on the standard POSIX file system interface. This layer performs all file I/O operations requested by AMS, communicating with a MSS if necessary. The interaction with the MSS is transparent to the user, except for a longer waiting time. An implementation of the interface for a given MSS can be provided in a form of shared library. The versions of OOFS created at SLAC and CERN are generic and can work with any MSS that allows storing and retrieval of files using UNIX type filenames. Currently HPSS is the system of choice, although CERN also deploys an in-house developed MSS – CASTOR. In the early prototypes of the interface, each I/O operation performed by the database was directly forwarded to the HPSS (POSIX compliant) I/O interface. This design resulted in a very limited performance, since the preferred I/O granularity of both systems is different. Objectivity/DB operates on data blocks (database pages) of size in the order of 10KB and requires of the order of 1000s of I/O operations per second to deliver satisfactory performance. Mass storage systems are usually efficient in sequential transfers of large data volumes of 10MB or more. In the simple test environment at CERN, HPSS could perform about 10 I/O operations per second, which was much too low for use with Objectivity/DB. As a result, a revised design of the interface has been developed. This design assumes a local disk cache with database files that can be accessed directly by the AMS. When required, entire database files are transferred to or from MSS. The database file size ranges from 1GB to 10GB, which limits the time that an application would need to wait for file retrieval to an acceptable value. Maintaining the local file cache adds complexity to the ODBMS/MSS interface, but gives much better performance than the initial approach. In addition to the MSS integration, the OOFS layer supports the implementation of the following features in the AMS server: • Defer Protocol – used to notify the database client application how long an operation, e.g. retrieving a file from MSS, is going to take. • Redirect Protocol – allows database requests to be redirected to other data servers for load balancing purposes • Authentication Protocol – allows for the implementation of security mechanisms based on public or private keys. Data Backup and Distribution To ensure a reliable use of an object database in production, a backup strategy had to be established. Since objects located in different database files can be associated with each other, the backup and recovery of a consistent state of the whole federation is more complicated than in the case of individual uncorrelated files. Objectivity provides tools that allow complete or incremental backups of the federation. However, in order to perform this task, they need to access all database files. In case where the federation contains petabytes of data stored in a MSS, this solution is not applicable. The current solution assumes that only the most vital databases need to be protected. In the case of HEP, the bulk of the data has a statistical value and a loss of a small part of it is not critical. Similarly, any data that can be easily recomputed from the raw events is only stored once. On the other hand, the federation catalogue and class schema are of central importance to the system and can not be lost. An extension of the current backup tools that would allow backups to be performed on a subset of the federation has been requested from Objectivity. In the meantime, SLAC and CERN have implemented partial backups based on file copies. To guarantee consistency, the backup tool acquires an update lock for all relevant databases. These files are then copied, the lock released and the copies backed up to tape. Preferably, the operation is performed during the database maintenance downtime. A similar procedure is used to distribute data to remote institutes. The BaBar collaboration have regional data centres for North America (SLAC), France (IN2P3), Italy (INFN) and the UK (RAL) each of them providing data subsets for local analysis. The BaBar Production System The first large-scale physics experiment to enter a full production mode with a system based on Objectivity/DB was BaBar experiment at SLAC, in May 1999. BaBar is a collaboration of ~650 scientists from 85 institutes in 10 countries. Its goal is the study of the B-meson decay from ee collisions at the PEP-II accelerator. The BaBar detector is delivering events of 32KB of raw data with 100Hz frequency. The data is distributed among nodes of the computing farm of 150 Sun-T1 (CPU servers) for the on-line data processing (online prompt reconstruction OPR). The results are written to a single Objectivity federation with 30MB/s data rate. The system configuration used for the OPR database consists of 3 data servers (Sun E4500, 4 CPU, 2*800GB Veritas filesystems, GigaBit Ethernet) and separate hosts for lock server, journal file server and database catalogue server. Data servers are using a modified AMS (Objectivity data server) with an interface to HPSS. The OPR system and federation is used solely for automated data processing and recording and is not accessible for any other users. A second farm with 150 nodes is used for offline data reprocessing and 80 hosts are dedicated for running users’ physics analysis applications. Data written into the OPR federation is duplicated into separate, dedicated Objectivity federations before it is made available to the users. The synchronisation is done periodically during scheduled maintenance time. A smaller amount of data (detector calibration) needs also to be transferred back to the OPR federation. This setup is more complicated and difficult to use than a single federation, but it eliminates the risk of off-line jobs locking the data and interfering with the OPR. The data analysis takes place not only at SLAC, but also at several external institutes distributed around the world. There are currently about 5 such sites, each of which maintains its own federation and exchanges data with SLAC. To be able to exchange data, the federations have to share the same schema: BaBar uses more than 500 different persistent classes. As a first project to use Objectivity/DB on a large scale, BaBar has encountered a number of scalability problems. Some of them have been successfully resolved whilst others require changes in the database architecture that are currently being discussed with Objectivity. The solved problems include: • Increasing the operating system limit of file descriptors available to the data server process • Careful application redesign to minimise lock contention and network traffic to the lockserver • Job control system and transaction time tuning to balance resource usage • Developing techniques to transfer data and databases between federations • Database schema distribution • Creating an interface to MSS The remaining problems include the maximum number of databases in a single federation. With 130TB of data, BaBar has already filled up ~30000 databases from the 65000 available. Also, large numbers of database clients (>150) can cause a saturation of the lock server. To resolve this problem Objectivity has been asked to introduce a read-only database type that would not require locks. This feature is expected to be delivered in Objectivity/DB v6. Later in this year BaBar is planning to quadruple its data rates and increase the size of the computing farm accordingly. The CMS Event Trigger Study CMS is one of the four LHC experiments at CERN that will begin collecting data in 2005. Although there is still some time to prepare, the experiment-wide physics application framework is already being developed and the software is complete enough to allow data processing. From March to May 2000 CMS has conducted an Event Trigger (a filter discarding uninteresting data) study on simulated data generated with a Monte Carlo (MC) program, to check the correctness of Figure 1 CMS processing farm configuration and data flow diagram output DB files jetmet federation + journals jetmet lockserver muon federation + journals