Configuration Databases in the ATLAS DAQ Prototype R.Jones 1, I.Soloviev 1,2 1 CERN, Geneva, Switzerland 2 on leave from PNPI, St.-Petersburg, Russia Abstract A prototype of the ATLAS Prototype-1 DAQ Configuration Databases, based on persistent object managers, has been developed. We give here some details of the architecture, implementation and test results. Conclusions and plans for the further work are presented. 1 Introduction 1.1 ATLAS Prototype DAQ Project The goal of the Prototype -1 ATLAS DAQ project [1] is to produce a prototype system representing a full slice of a DAQ suitable for evaluating candidate technologies and architectures for the final ATLAS DAQ system. The work started on defining the requirements in 1996 and it is expected that the prototype will be running at the end of 1998. 1.2 Configuration Databases Requirements The ATLAS DAQ system needs several configuration databases to store a large number of parameters describing DAQ system architecture, and components, running modes and the system running status. In accordance with the User Requirements Document (URD) [2] prepared for the backend subsystem [3], these databases must store multiple versions of data, support schema evolution facilities, an Application Programming Interface (API) and a Graphical User Interface (GUI) to store, retrieve and navigate databases contents. During DAQ runs the configuration databases are used in read-only mode and they must not degrade performance of DAQ applications and the API must be available for all DAQ platforms including real-time operating systems running on embedded processors. The off-line database usage supports concurrent updates, authorization checks, contents consistency checks and basic system management functions. 2 Architecture The ATLAS DAQ group has evaluated various commercial and shareware data persistence systems (relational databases, object databases and object managers) but no single system satisfied all the documented user requirements [4], [5]. As a consequence, it was decided to adopt a two-tier architecture, using a light-weight in-memory persistent object manager to support the real-time requirements and a full Object Database Management System (ODBMS) as a back-up and for long-term data management, as it is shown on Figure 1. The data consistency between these systems will be provided by translation applications run off-line time and the CASE tools will synchronize development. For the run-time object manager, a package called OKS (Object Kernel Support) [6], [7] has been developed on top of Rogue Wave s Tools.h++ C++ class library [8]. The OKS system is based on an object model that supports objects, classes, associations, methods, data abstraction, inheritance, polymorphism, object identifiers, com-
posite objects, integrity constraints, schema evolution, data migration, active notification and queries. The OKS system stores database schema and data in portable ASCII files and allows different schema and data files to be merged into a single database. It includes Motif based GUIs to design database schema and to manipulate OKS objects. For the back-up and for long-term data management, a commercial object oriented database management system Objectivity/DB [9] has been chosen. It was introduced to CERN by the RD45 project [10]. We evaluated the basic DBMS features (schema evolution, access control, versioning, back-up/restore facilities etc.) and the C++ programming interface. Figure 1. Two-tier architecture of configuration databases DAQ Application DAQ Application DAQ Application Run Time DAQ Application DAQ Application Run-time persistent object manager satisfies real-time requirements supports ATLAS DAQ platforms GUI data editor Data Translator Data Translator Full object database (backup & long-term data management) supports data and schema versioning supports concurrent updates has authorisation control GUI schema and data editors 2.1 Database Files Architecture Both OKS and Objectivity/DB can group several database files into a single database, called federated database. It seems reasonable to organize configuration databases into tree-like structure that represents the actual hierarchy of DAQ system as shown on Figure 2. The advantages of organizing the databases in such manner is discussed below: It improves performance of applications which use configuration databases and it reduces their memory consumption (e.g. to receive a value of some detector s parameter it is enough to load only the description of the actual detector [one data file] instead of the whole database). It allows file system based authorization control on the level of subsystems or any part of a subsystem (e.g., only one group of users is responsible for changing some piece of database information, any other users can receive read-only access to it etc.) It simplifies the implementation ofdaq system initialization and partitioning. We understand partitioning to be the ability to run two or more DAQ systems in parallel concurrently (e.g. Detector X and Detector Y are taking data together while Detector Z is performing calibration). Of course, we would need some means of controlling which run concurrently so that they do not use the same equipment (e.g. 2 using detector Y) [11]. Here we understand DAQ ini-
tialization to be the ability to configure a partition independent part of the system (e.g. event builder switch that is used by several or piece of and which are used by all like partition and resource managers). To initialize the DAQ system a setup data file from Root/shared/setup directory has to be used, that has references to data files stored on Root/shared directory (i.e. dataflow, event filter, backend and detector interface directories). To define a partition for Detector X, a setup data file from Root/detector X/ directory has to be used, that has references to data files stored on Root/ shared and Root/detector X directories. If a partition uses more than one detector, it is placed on Root/ directory and it has links with data files from related detectors. Figure 2. Configuration databases file structure Conf. database root directory shared run parameters detector X detector Y... detector Z schemes setup dataflow event filter backend detector interface schema parameters schema parameters schema parameters 2.2 Data Access Libraries Data Access Libraries (DAL) are used to hide the details of the low-level persistent object system from a DAQ programmer (so their code will not be changed if the data manager is replaced) and details of database schema (i.e. map schema to application view ). Two DALs have been implemented for the DAQ configuration database: data flow application view DAL (it maps database object schema to data flow application view) and automatically generated C++ DAL (it maps database schema to C++ classes and database objects to C++ objects with get...()/set...() methods). 2.3 CASE Tools CASE tools have been used to synchronize development between the persistent object storage systems (to generate schema and data mover applications) and to generate DALs. The StP (Software Thru Pictures) [12] CASE tool with OMT (Object Modeling Technique) object model is used for automatic generation of some configuration databases code.
A translator has been developed between the OMT object model and OKS object model and a prototype translator has been developed between the OMT object model and Objectivity object model. The C++ DAL (see above) is generated with StP as well. 2.4 Remote Database Access To provide world-wide access to configuration databases, a CORBA interface has been implemented. It allows Java based DAQ Status Display to access the database from any computer connected via Internet. The remote read-only configuration database access is implemented on top of ILU (freeware CORBA implementation) [13]. It consists of a C++ server containing OKS and an IDL interface that allows clients to be written in different programming languages (C/C++, Java), to access database information world-wide. The prototype client has been written in Java as a part of the future Status Display. 3 Implementation Currently two configuration database schemes have been developed and related databases populated. The DAQ configuration database includes descriptions of DAQ partitioning, and components, run control hierarchy, run parameters, detector hierarchy and DAQ setup/shutdown sequences representing a total by 50 interconnected classes. It is used by DAQ backend components (Run Control, Process Manager and DAQ Supervisor) and by DAQ dataflow [14]. The Message Reporting System (MRS) configuration database includes description of predefined messages (two classes in total). Both databases are implemented on top of OKS. 3.1 Tests and Benchmarks This section describes benchmarks and tests which have been performed on the configuration databases to evaluate the performance for different configurations (including expected DAQ configurations), to test compartibility of OKS with commercial persistent object managers and to prove it s reliability. The details of test s requirements [15] and results [16] are published as separate documents and the rest of the section gives a short overview of results. 3.1.1 OO7 Benchmark The OO7 Benchmark [17] has been designed as a first step toward providing a comprehensive ODBMS performance profile. Among the performance characteristics tested by OO7 are: the speed of various traversals, the efficiency of various updates and the performance of the query processor on various queries. The tests have been made for different configurations, where the number of objects varies from 5,000 to 1,000,000. The benchmark produces a set of numbers rather than a single number. Both OKS and Objectivity/DB have been tested. The benchmark shows that OKS requires less system resources when working with relatively small configurations. For the biggest configuration the sizes of database files are comparable (99.2 MB for Objectivity/DB and 115.7 for OKS), but OKS requires more time and memory for initialization (remember that OKS is on-memory database). The OKS shows better performance of read-only operations for configurations of any size: the traversals are faster by about 10 times, the queries are faster up to 100 times (1000 times for path lookup). The update operations are faster for OKS if the configuration is not large. The results of the benchmark show that OKS is preferred when working with small configurations (which are close to the expected DAQ configurations) and from
this point of view it is better for run-time usage. The Objectivity/DB works better with large databases (which correspond to multiple DAQ configurations including versioning) in an off-line environment. In addition, the OO7 benchmark determines that the OKS object model is comparable with the object models supported by commercial object databases. Successful completion of the benchmark also indicates that OKS is a reliable implementation of a light-weight object manager. 3.1.2 Performance Benchmark of DAQ Configuration Databases Using OKS The intention of this benchmark is to evaluate the performance for expected DAQ configurations (single read-out crate [~400 database objects], prototype-1 DAQ [10 crates and ~5,000 database objects] and final ATLAS DAQ [200 crates and ~100,000 database objects]) on different computers running various operating systems (Sun Solaris, HP-UX, LynxOS and Windows NT) and to show what are the requirements to achieve acceptable response times. The benchmark has been performed with OKS only (and not Objectivity/DB) because it is the run-time object manager and hence directly affects the DAQ performance during data taking activities. The tests identified that performance depends not only CPU frequency but also architecture. By the benchmark s results the fastest computers require about one second for initialization of a medium-sized configuration (i.e. prototype-1 DAQ) and about half of second for it s complete traverse and shutdown. The average front-end computer (100 MHz PowerPC with 32 MB running real-time LynxOS) requires about 3,000 ms to initialise the database and load a medium-sized configuration, 500 ms to close the configuration and shutdown database, less than 0.01 ms to access an object by name, 10 ms to execute query with 10% match lookup and less than 0.0025 ms to access the value of an object s attribute or relationship by name. The benchmark process with loaded medium-sized configuration allocates about 5 MBytes of memory (including C++ run-time and common libraries) and requires about 500 KBytes of disk space. 3.1.3 Code Analysis Tests Because the configuration databases are used by practically all control during their initialization phase, the question of reliability and correct deallocation of system resources is important. Run-time memory related problems have been detected with the aid of the Purify [18] CASE tool and general code coverage tests are planned with the Logiscope [19] CASE tool. 4 Conclusions The two-tier architecture helps cover all requirements for the configuration databases (high performance, availability for real-time operating system running embedded processors, GUI, data and schema versioning) and eases data exchange with offline applications. The CASE tools simplify implementation, increase re-use of databases in case of schema modification and improve robustness (i.e. error free code). The federated database (i.e. merging multiple database files in run-time) improves performance, adds authorization control and simplifies DAQ partitioning and initialization. The performed tests and benchmarks show robustness and compatibility of OKS and Objectivity/DB. By using an object model we were able to profit from inheritance to simplify the database schema in comparison with Entity-Relationship approach.
5 Acknowledgements We gratefully acknowledge the contribution of all our colleagues in the prototype DAQ project to the organization and contents of the database schema and access libraries. 6 References [1] The ATLAS DAQ and Event Filter Prototype -1 Project G.Ambrosin, et al., Computer Physics Communications 110, 1998, pp.95-102 [2] ATLAS DAQ Back-end User Requirements Document ATLAS DAQ Note 87, 1996, is available through Internet at URL http://atddoc.cern.ch/ Atlas/DaqSoft/document/draft_1.html [3] The Back-end subsystem of the ATLAS prototype DAQ CHEP 98 Proceedings [4] Object Oriented database system evaluation for the DAQ system Maria Skiadelli, Diploma thesis at the University of Athens, 1995, see Internet at URL http://rd13doc.cern.ch/public/doc/note108/thesis.book.html [5] Experience using a distributed Object Oriented Database for a DAQ system R. Jones, et al., World Scientific Publishing Co., 1996, ISBN 981-02-2783-3 [6] The OKS Persistent In-memory Object Manager R.Jones, L. Mapelli, Yu.Ryabov, I.Soloviev, IEEE Transactions on Nuclear Science, vol 45, No 4, August, 1998 [7] OKS Documentation (User s Guide, Tools Manual, Reference Manual) I.Soloviev, 1998, ATLAS DAQ/EF P-1 Technical Note 33, is available through Internet at URL http://atddoc.cern.ch/atlas/notes/033/welcome.html [8] Tools.h++ Foundation Class Library for C++ programming Rogue Wave Software, Inc., March 1996, for more information see Internet at URL http:// www.roguewave.com/products/tools/tools.html [9] Using Objectivity/C++ version 4 Objectivity, Inc., July 1996, see Internet at URL http://www.objectivity.com/ [10] Object Databases and Their Impact On Storage Related Aspects of HEP Computing CERN/LHCC 97-7, LCB/RD45, February 7, 1997, is available through Internet at URL http://wwwinfo.cern.ch/pl/cernlib/rd45/reports.htm [11] High-level design of the resource manager I.Alexandrov et al., 1998, ATLAS DAQ/EF P-1 Technical Note 52, is available through Internet at URL http://atddoc.cern.ch/atlas/notes/052/note052-1.html [12] User Manual, Creating OMT Models Interactive Development Environment (IDE) [13] Use of Corba in the Atlas prototype DAQ A.Amorim et al., IEEE Transactions on Nuclear Science, vol 45, No 4, August, 1998 [14] The Dataflow for ATLAS DAQ/EF Prototype -1 G.Ambrosini et al., ATLAS Internal Note, DAQ-NO-091, see http://atddoc.cern.ch/ Atlas/Notes/069/Note069-1.html [15] Test Plan of the Configuration Databases for the Atlas DAQ Prototype -1 I.Soloviev, 1998, ATLAS DAQ/EF P-1 Technical Note 99, is available through Internet at URL http://atddoc.cern.ch/atlas/notes/099/note099-1.html [16] Test Report of the Configuration Databases for the Atlas DAQ Prototype -1 to be published about the end of September as ATLAS DAQ/EF P-1 Technical Note [17] A Status Report on the OO7 OODBMS Benchmark Effort Michael J. Carey David J. DeWitt Jeffrey F. Naughton, Computer Sciences Department University of Wisconsin-Madison, proceedings of OOPSLA 94 [18] Purify, Rational Software Corp. see for more information Internet at URL http://www.pure.com/products/purify/ [19] Logiscope, Verilog see for more information Internet at URL http://www.verilogusa.com/log/logiscop.htm