Software Design Document

Transcription

1 Software Design Document TESS SPOC Array Database EXP-TESS-ARC-SW-0020 January 29, 2015

2 Sean McCauliff, Sr. Software Engineer (Author) Date Masoud Mansouri-Samani, TESS-SPOC Systems Engineer, ARC Date David Lung, TESS-SPOC System Safety and Mission Assurance Lead, ARC Date Jon Jenkins, TESS-SPOC Analysis Lead, ARC Date Dwight Sanderfer, TESS-SPOC Manager, ARC Date page 1 of 15

3 Contents 1 Introduction Purpose and Scope Intended Audience Related Documents Compliance Documents Reference Documents Architectural Design Context Functional Descriptions Heritage Architecture Application Programming Interface Storage Allocation Deployment Appendices 13 A Requirements 13 B Test Cases 14 page 2 of 15

4 List of Figures 1 Array Database Context ADB Client API CRUD and Transaction Control Methods ADB Client API Data Objects page 3 of 15

5 List of Tables page 4 of 15

6 Todo list page 5 of 15

7 Change Date Notes 4 Jun 2014 You-are-here diagram. Heritage section. Add SDD to the build system. Initial revision. 6 Jun 2014 The need for a new storage allocator. 11 Jun 2014 Grammar. Architecture section. Data objects API. ADB client API. 12 Jun 2014 ADB Document ID. 14 Jun 2014 Modified the publish task to put the docid into the published pdf filenames. Modified the build task to create build/doc-id.sty files specific to each tex document and in turn modified those documents to use it. Added ext.docid to all the docs/sources build.gradle files. 23 Oct 2014 Modify documents to use new automatic editing history. Use length of history entry to estimate the number of lines needed for the history entry. 29 Oct 2014 Update tests. Use ArrayId instead of ObjectId and other minor renaming issues. 28 Jan 2015 Add draft marker. Update signatures page. page 6 of 15

8 1 Introduction This document describes the software design of the TESS Science Processing and Operations Center (SPOC) Array Database (ADB) CSCI. 1.1 Purpose and Scope The purpose of this document is to provide a description of the Array Database CSCI including the (1) architectural design, (2) applicable requirements, (3) functional decomposition, (4) test cases, (5) traceability of requirements to functions that realize them and test cases that verify them, (6) data flow, (7) interface and input/output classes, (8) execution sequence, and (9) planned utilization of NASA Advanced Supercomputing (NAS) Pleiades cluster resources. The TESS SPOC pipeline derives heavily from the Kepler SOC pipeline which has been instrumental in the discovery of thousands of extrasolar planets and candidates, and is largely a proven commodity. Differences in the ADB CSCI between Kepler and TESS are emphasized in this document; these are driven by differences in the science data and data acquisition rate between the two transit photometry missions. 1.2 Intended Audience The intended audience of the document is composed of systems engineers, software engineers, scientists and operations staff of TESS. 1.3 Related Documents Compliance Documents This Software Design Description will be compliant with the following documents. Functional Specification (EXP-TESS-ARC-SW-0018) Test Cases (EXP-TESS-ARC-SW-0019) Software Requirements Document (SRD) (EXP-TESS-ARC-RQMT-0010) System Architecture Document (EXP-TESS-ARC-SW-0003) Reference Documents Pipeline Infrastructure Software Design Document (SDD) (EXP-TESS-ARC-SW-0004) page 7 of 15

9 2 Architectural Design 2.1 Context A block diagram of the SPOC pipeline is shown in figure 1. The ADB CSCI is highlighted along with Oracle (RDBMS). Together ADB and Oracle compose the Datastore component. This is the long term storage for all SPOC data transmitted between pipeline components. Pipeline GUI Original Pixel Data POC Target Lists Compression Tables Synthetic Data TESS Input Catalog Target Lists Data Receipt (DR) Raw Pixels Calibration (CAL) Compression Tables Compression (COMP) Compression Tables Calibrated Pixels CAL Metrics Synthetic Data Compute Optimal Aperture (COA) Optimal Apertures Photometric Analysis (PA) PA Metrics SPOC Science Pipeline & Other Infrastructure Raw Flux Centroids Photometer Management End-to-End Model (Lilith) Presearch Data Conditioning (PDC) Models Pipeline Infrastructure (PI) Corrected Flux Models Models (MOD) Transiting Planet Search (TPS) Models Datastore (ADB/Oracle) Threshold-crossing Events (TCEs) CDPP Photometer Performance Assessment (PPA) Data Validation (DV) Archive (AR) Photometer Performance Metrics Planet Candidates and Reports Exports MAST TSO Figure 1: Array Database Context 2.2 Functional Descriptions The Array Database CSCI is decomposed into a set of functions that enable it to realize the requirements allocated to it. The ADB requirements are enumerated in appendix A. Descriptions of the ADB functions follow with references to the requirements that they realize.. L5 F.ADB Array Data Base (ADB) Description The ADB server is a specialized database management system used to provide an abstraction and simple query interface over the accumulated TESS mission data. The ADB is one component of the datastore which also includes a relational datbase component. The ADB and the relational database combined store the minimal necessary data for running the TESS pipeline. We use the ADB instead of a relational database for performance reasons. For the data types that the ADB handles we have speedups of 8 to 80 times that of a relational database. Realizes R.SPOC.10: The SPOC shall provide automation, persistence, and configuration management services to pipeline algorithms. L6 F.ADB.1 Provide network-based client interface page 8 of 15

10 F.ADB : Array Data Base (ADB) Description This means that the ADB should allow access over a network to the data stored within. Specifically this interface should provide a mechanism for storing arrays, sparse arrays and arbitrary binary data (BLOBs) or some more mission specific version of those data types.. L6 F.ADB.2 Provide persistence to disk F.ADB : Array Data Base (ADB) Description When a transaction is complete the data should be stored on disk as opposed to stored in memory or some volatile storage medium.. L6 F.ADB.3 Provide management interface F.ADB : Array Data Base (ADB) Description This allows an administrator or an engineer to assess the current state of the ADB server and provides for limited control over the current transactions. More specifically the management interface lists the uptime of the server; the number of connected clients; the total number of threads allowed to perform I/O and a means to increase that number; the active transactions and a means to force the rollback of stuck transactions. Some management functionality is available via the ADB command line interface full functionality is available via a Java management extension (JMX) interface. Should the ADB server be implemented in some other language than the management interface can be exported via HTTP or some other remote management service.. L6 F.ADB.4 Provide transaction support F.ADB : Array Data Base (ADB) Description A transaction is the unit of work for the ADB. A transaction must maintain the following properties often referred to as ACID: Atomic, Consistent, Isolated and Durable. The atomicity property is that either all the I/O operations in a transaction will succeed or fail; there can be no partial success. Consistency ensures that integrity constraints are always obeyed. Isolation provides a view of the database to different concurrent processes which appears that no other concurrent processes are running. Durability means that a transaction is not complete until all its data has been written to a persistent storage medium. The ADB server has specific interpretations of some of the ACID properties. Atomicity can be enforced on a single write issued by a client or multiple writes. The only constraint enforced by the ADB server is to ensure that data provenance is kept consistent with the data it is associated with. Transactions have approximately "READ_COMMITTED" level of transaction isolation. In this isolation level transactions can read data that has been completely committed as part of another transaction before the other transaction is complete. It does not lock arrays that have been read as part of a transaction. Durability is handled under a separate function. 2.3 Heritage Kepler used ADB (then poorly named as the filestore or FS in Kepler nomenclature) to store one dimensional arrays, sparse arrays and regular files (also known as Binary Large Objects or BLOBs). Compared with a traditional RDBMs ADB has two advantages. Basic read and write operations have a speedup of about 8 to 80 compared with writing the same data into an RDBMS with an appropriate schema. The natural representation of the data is a much closer match to an array database than to a relational database. Compared with flat files the ADB has the advantage of maintaining transactional properties ACID (Atomic, Consistent, Isolated and Durable) over the data. ADB read and write operations are atomic since all operations that are part of the same transaction either succeed or fail. They are consistent in the sense that originator identifiers (see the PI SDD) always identify the source of every array element or BLOB. This is part of the data accountability requirement. Isolation is the view that each client of the database looks as if they get their own copy of the database. ADB implements a looser version of READ_COMMITTED transaction isolation level. That is transaction A may read transition B s changes after B has committed even though transaction A has not yet committed. Durability is ability to have permanent changes remain page 9 of 15

11 permanent. ADB is resilient to software errors that would cause the ADB process to crash; the ADB sever can be killed at any time without threat of data corruption. ADB can not guarantee durability in the face of power failures or OS kernel panics. [F.ADB.4] TESS ADB will maintain all these features. There are a few additional features that are needed for more efficient TESS pipeline operations. Often we want to represent data that is attached to some cadence or time interval. For example, we want to associate a motion polynomial BLOB with a start cadence of 1000 and an end cadence of Overtime we build up these motion polynomial BLOBs and so ADB clients want to query over a large interval of time and get back every BLOB in that interval. Kepler handles this problem in a very awkward manner where the relational database is asked to process the query and create the list of things that need to be fetched. For TESS we would bring this into ADB itself. Unlike Kepler, TESS ADB will need to be able to delete a large amount of data in order to keep the total storage utilization within the bounds set by the mission. This is about three months of data storage. Since some stars will be observed for longer periods only stars not going to be re-observed after 3 months will need to be dropped. TESS, will have much more pixel data. TESS pixel masks are larger (generally 10 x 10) than Kepler and TESS will be storing full frame images as arrays. Efficiently storing this data means exploiting the data locality implicit in the arrangement of pixels. These two differences (highly selective delete and a need for greater data locality) entail a new storage allocation algorithm for ADB. The new storage allocator will store a large number of arrays in the same file. Deleting an array of part of a array will involve revoking all or some of the file system blocks that are allocated to it. This is similar to how file systems allocate disk block to files. As disk capacity becomes larger we need fewer spinning hard drives to store the same amount of data. This is generally a good thing. But fewer spinning drives means random access performance of a storage system will decrease. This decreases linearly with the number of hard disks in the storage system. As a result TESS will need to exploit more data locality by having reads and writes be more contiguous. The Kepler ADB only had support for one dimensional arrays. By implementing multidimensional arrays for TESS ADB we can insure that I/O operations are less frequent and larger. [F.ADB.2]ADB provides for remote monitoring over Java Management Extensions (JMX) and a custom command line interface. 3 Architecture 3.1 Application Programming Interface [F.ADB.1] The ADB application programming interface provides three kinds of interfaces. Create, Read, Update, Delete (CRUD) methods(1) to manipulate the data objects (2) and transactions over the CRUD methods(3). To begin a transaction a the client can just call begintransaction() on the ADB client instance. Committing or rolling back a transaction is accomplished by calling committransaction() or rollbacktransaction() on the ADB client instance. Transactions will automatically rollback after a configurable transaction timeout. Figure 2 is a UML class diagram that shows the transaction API and the CRUD API. Blue elements indicate new functionality for TESS. Figure 3 shows data objects managed by ADB. Arrays also have subclasses for double precision numbers in additional to those shown in figure 3. Additional array types supporting any Java primitive type are trivial to add. Blue elements indicate new functionality for TESS. The MultidimensionalArrayClient allows for the manipulation of MultidimensionalArrays. We show only a single precision float implementation, but we also plan for a 32-bit signed integer multidimensional array implementation. The CRUD methods associated with MultidimensionalArrays allow for the specification of the indices into the array dimensions should be returned. A query spec for a MultidimensionalArray can be constructed from strings written in MATLAB style. For example, accessing all columns and rows one through 10 from a two dimensional array can be written as "1:10, :". IntervalBlobs are arbitrary binary data, usually these are MAT files, that are too complicated to store as simple arrays and storing them in the database would require much in the way of database schema engineering. An IntervalBlob also has a cadence interval associated with it that determines the time interval over which it is valid. The CRUD API allows for the specification of a object identifier and an interval over which blobs should be fetched. It s possible that multiple BLOBs could be valid over the specified query interval. The value returned, BlobFileSeries, associates an originator id and a cadence interval for each BLOB returned. 3.2 Storage Allocation [F.ADB.2] The Kepler ADB storage allocator attempts to pack multiple 1D arrays into a single file. It does this in a way that assumes that it knows the order in which bytes will be written into in the future. This turns out to work well most of the time, but as the ADB file system instance for Kepler has aged more files have become fragmented and sparse. This leads to more random access reads than are really necessary. The TESS ADB storage allocator will use our existing B+-tree index implementation to store an ArrayId and the extent information for each array. An extent is simply a tuple of the logical array address, the start address into the file of the extent, page 10 of 15

12 Figure 2: ADB Client API CRUD and Transaction Control Methods and a length. This length is the number of blocks allocated to the array. When the space for an extent is exhausted a new extent is allocated from an index of free space in the file. When an array (or part of array) needs to be deleted it s extents are deleted from the index. Efficient storage of originator identifiers can be realized by using an R*-tree. This is similar to a B+-tree, but it stores with it informtion on the hyper-rectanges which bound an area within the multidimensional array. This would be used to associate originator ids with arbitary areas. 3.3 Deployment ADB is never deployed within the NAS supercomputing environment. It always exists on a server within the SPOC network. page 11 of 15

13 Figure 3: ADB Client API Data Objects page 12 of 15

14 Appendices A Requirements SPOC requirements which have been allocated to ADB are enumerated below with references to the functions which realize them. The ADB functions are enumerated in section 2.2. The test cases which verify the requirements are listed in appendix B.. L6 R.ADB.1 ADB shall provide long-term persistent storage for pipeline data R.SPOC.10 : The SPOC shall provide automation, persistence, and configuration management services to pipeline algorithms Rationale Provides a common source for pipeline modules and other code to access pipeline data with distributed access.. L7 R.ADB.1.1 ADB shall provide CRUD for arrays, sparse arrays and blobs. R.ADB.1 : ADB shall provide long-term persistent storage for pipeline data Rationale This is the bulk of the TESS data. Verified By T.1: Automated End-to-end Pipeline Test. L7 R.ADB.1.2 ADB shall maintain ACID properties for managed data types R.ADB.1 : ADB shall provide long-term persistent storage for pipeline data Rationale Allow safe concurrent access to data Verified By T.18: ADB Test 1. T.19: ADB Test 2.. L7 R.ADB.1.3 ADB shall provide query interface for managed data types. R.ADB.1 : ADB shall provide long-term persistent storage for pipeline data Rationale Provide flexible access to pipeline data Verified By T.1: Automated End-to-end Pipeline Test. L7 R.ADB.1.4 ADB shall provide a management interface (JMX, command line) R.ADB.1 : ADB shall provide long-term persistent storage for pipeline data Rationale Support troubleshooting. L7 page 13 of 15

15 R.ADB.1.5 ADB shall allow concurrent access to data. R.ADB.1 : ADB shall provide long-term persistent storage for pipeline data Rationale Support concurrent, distributed pipeline processing Verified By T.1: Automated End-to-end Pipeline Test T.19: ADB Test 2.. L7 R.ADB.1.6 ADB shall be sized to hold at least 3 months worth of mission data at a time R.ADB.1 : ADB shall provide long-term persistent storage for pipeline data Rationale Allow enough time for data to be processed, reviewed, reprocessed if necessary, and exported before taking the data offline. Verified By T.21: ADB Test 4. L4 SPOC_22 The SPOC hardware shall be sized to store at least 3 sectors worth of data. Rationale Ensures that we have enough storage capacity to accommodate the data we receive and process. B Test Cases The test cases which are used to verify ADB requirements are listed below. For further information on the requirements themselves, see appendix A.. L5 T.18 ADB Test 1. Description Write multiple data types in a single transaction to the ADB server. Verify that all the data are written correctly by reading back the values that were written. Stopping the server and restarting it. Then reading back the values that were written. Starting a transaction and overwriting existing data, but then not committing the transaction, but rolling back instead. Data should still have original values once all transactions have been terminated. Verifies R.ADB.1.2: ADB shall maintain ACID properties for managed data types. L5 T.19 ADB Test 2. Description Create two transactions in different threads (A and B). Thread A will write data to ADB and then wait until Thread B releases thread A. Thread B will write different data to the same ArrayIds as thread A. Thread B will commit and verify that it sees B s data but not A s data. Thread B releases thread A. Thread A will verify that it still sees thread A s data and not thread B s data. Verifies R.ADB.1.2: ADB shall maintain ACID properties for managed data types R.ADB.1.5: ADB shall allow concurrent access to data. page 14 of 15

16 . L5 T.20 ADB Test 3 Description Browse the status of a running ADB server with the management interfaces. Verifies R.COMP.2.1: COMP shall compute a mean black table. L5 T.21 ADB Test 4 Description Maintain a spreadsheet of the estimated size of the file system space consumed by ADB. Verifies R.ADB.1.6: ADB shall be sized to hold at least 3 months worth of mission data at a time page 15 of 15