Objectivity Technical Overview. Release 9

Transcription

1 Objectivity Technical Overview Release 9

2 Objectivity Technical Overview Part Number: 9-OTO-1 Release 9 The information in this document is subject to change without notice. Objectivity, Inc. assumes no responsibility for any errors that may appear in this document. Copyright by Objectivity, Inc. All rights reserved. This document may not be copied, photocopied, reproduced, translated, or converted to any electronic or machine-readable form in whole or in part without prior written approval of Objectivity, Inc. Objectivity and Objectivity/DB are registered trademarks of Objectivity, Inc. Active Schema, Objectivity/DB Active Schema, Assist, Objectivity/Assist, ooassistant, Objectivity/DB ooassistant, Objectivity/DB Fault Tolerant Option, Objectivity/FTO, Objectivity/DB Data Replication Option, Objectivity/DRO, Objectivity/DB High Availability, Objectivity/HA, Objectivity/DB Hot Failover, Objectivity/DB In-Process Lock Server, Objectivity/IPLS, Objectivity/DB Open File System, Objectivity/OFS, Objectivity/DB Parallel Query Engine, Objectivity/PQE, Objectivity/DB Persistence Designer, Objectivity/DB Secure Framework, Objectivity/Secure, Objectivity/C++, Objectivity/C++ Data Definition Language, Objectivity/DDL, Objectivity/C++ Active Schema, Objectivity/C++ Standard Template Library, Objectivity/C++ STL, Objectivity/C++ Spatial Index Framework, Objectivity/Spatial, Objectivity for Java, Objectivity/.NET, Objectivity/. NET for C#, Objectivity/Python, Objectivity/Smalltalk, Objectivity/SQL++, Objectivity/SQL++ ODBC Driver, Objectivity/ODBC, Objectivity Event Notification Services, and Persistence Designer are trademarks of Objectivity, Inc. Other trademarks and products are the property of their respective owners. ODMG information in this document is based in whole or in part on material from The Object Database Standard: ODMG 2.0, edited by R.G.G. Cattell, and is reprinted with permission of Morgan Kaufmann Publishers. Copyright 1997 by Morgan Kaufmann Publishers. The software and information contained herein are proprietary to, and comprise valuable trade secrets of, Objectivity, Inc., which intends to preserve as trade secrets such software and information. This software is furnished pursuant to a written license agreement and may be used, copied, transmitted, and stored only in accordance with the terms of such license and with the inclusion of the above copyright notice. This software and information or any other copies thereof may not be provided or otherwise made available to any other person. RESTRICTED RIGHTS NOTICE: Use, duplication or disclosure by the U.S. Government is subject to restrictions set forth in the Objectivity, Inc. license agreement and as provided in DFARS (a) and (a) (1998), and FAR , as applicable. Objectivity, Inc., 640 West California Avenue, Suite 210, Sunnyvale, CA

3 Contents Chapter 1 Introduction 7 Standards 10 Typical Applications 11 About Objectivity, Inc. 12 Chapter 2 Product Family Overview 15 Objectivity/DB Product Group 15 Powerful Object-Oriented Development 23 Objectivity/C++ Product Group 26 Objectivity for Java Product Group 28 Objectivity/Smalltalk Product Group 29 Objectivity/SQL++ Product Group 30 Chapter 3 Data Modeling 31 Complex Data Models 31 Object Relationships 33 Evolution of Data Models and Object Conversion 35 Chapter 4 Distributed Database Architectures 37 Evolution of Client/Server 37 Distributed Database Architecture 39 Chapter 5 Objectivity/DB 43 Safe Object Reference Architecture 43 Storage for Exabytes of Interrelated Objects 43 Platform Interoperability 44 3

4 Location Independence of Objects 44 Accessing Objects 44 Locking 46 Transactions 46 Object Clustering 47 Object-Level Versioning 47 Chapter 6 Objectivity/DB High Availability 49 Partitioning a Federated Database 49 Replicating Databases 51 Objectivity/DB High Availability Scenarios 54 Chapter 7 Developing with Objectivity/C++ 63 Development Process 63 Data Modeling in C++ 65 Developing Applications 67 Using C++ Class Libraries 73 Chapter 8 Developing with Objectivity for Java 77 Development Process 77 Data Modeling in Java 78 Developing Applications 81 Database Garbage Collection 87 Object and Class Migration 87 Chapter 9 Developing with Objectivity/Smalltalk 89 Development Process 89 Data Modeling in Smalltalk 90 Developing Applications 92 Using Smalltalk Class Libraries 95 Automatic Online Database Garbage Collection 95 Automatic Storage Management 96 Automatic Object and Class Migration 96 4 Objectivity Technical Overview

5 Chapter 10 Objectivity/SQL++ 97 Using Objectivity/SQL++ 97 Objectivity/SQL++ Examples 100 Objectivity/SQL++ Extensions 101 Interactive SQL Objectivity/SQL++ ODBC Driver 106 Chapter 11 Objectivity Solutions 107 Objectivity/DB Active Schema 107 Objectivity/DB Open File System 109 Objectivity/DB Parallel Query Engine 110 Objectivity/DB Secure Framework 113 Objectivity Event Notification Services 115 Chapter 12 Development and Administration Tools 119 Data Modeling and Application Development Tools 119 Administering Objectivity/DB 125 Appendix A Page Server Performance 129 Assumptions 130 Page Servers Are More Efficient Than Object Servers 131 Page Server Has Less Network Delay 132 Page Server Uses Less Network Bandwidth 132 Appendix B Standards Support 135 Objectivity Involvement in Standards Organizations 135 Technology Standards 136 Appendix C Migrating a C++ Application 137 The Original C++ Example 137 The Objectivity/C++ Application 144 Objectivity Technical Overview 5

6 Appendix D Migrating a Java Application 153 The Original Java Example 153 The Objectivity for Java Application 158 Appendix E Migrating a Smalltalk Application 165 The Original Smalltalk Example 165 The Objectivity/Smalltalk Application 170 Appendix F Resources Appendix 175 Publication Resources 175 Web Resources 176 Index Objectivity Technical Overview

7 1 Introduction Why Use an ODBMS? Database management for today s high-end applications. Objectivity/DB is a distributed object database management system (ODBMS). As an ODBMS, Objectivity/DB is ideal for applications that require complex data models. As a distributed ODBMS, Objectivity/DB supports large numbers of users and provides high-performance access to large volumes of physically distributed data. Objectivity/DB manages data transparently to high-end applications. Its distributed database architecture provides scalability as well as performance. This architecture supports a wide range of application architectures, including client/server, mixed-tier, and service-oriented. Objectivity/DB integrates easily with application software. It allows you to directly store and manage objects through standard language interfaces such as, C#/.NET, C++, Java, Python, Smalltalk, SQL and XML using traditional programming techniques and tools. Integrates into your existing environment. Objectivity/DB also integrates into existing computing environments. It supports complete interoperability across major hardware platforms and operating systems. Through its ANSI-standard SQL interface, you can leverage your SQL-programming knowledge to use popular client/server tools compliant with Microsoft s Open Database Connectivity (ODBC) interface. Relational database management systems (RDBMSs) have been in use for many years. These systems are well-suited for applications that use tabular data models. Typical RDBMS applications include billing and inventory tracking, where simple, fixed-length data is represented in table form. RDBMSs perform basic operations such as joins, projections, and selections in short transactions. These systems have served well for traditional online transaction processing (OLTP), accounting, and online entry applications requiring database control of textual and numerical data. 7

8 Introduction Why Use an ODBMS? Some vendors in the RDBMS market added objects to their RDBMS engines and called them Object Relational DBMS (ORDBMS). These extensions to their engines include Extended Data Types (EDTs) and complex objects. However, the underlying mismatch between the object-oriented software and the database still exist, resulting in substantial performance penalties. If you can solve the problem with an RDBMS without a lot of effort and also achieve the performance, throughput and scalability that you need then there's no compelling reason to use an ODBMS. However, if any of the following conditions are met then you should consider using Objectivity/DB: Complex Relationships If there are a lot of many-to-many relationships, tree structures or network (graph) structures. Complex Data If the data has multiple varying length components, particularly multi-dimensional arrays, arrays of structs or binary streams. Distributed Databases If the databases need to be geographically distributed, used in a detached mode or are accessed from a processor grid (or farm ). Multiple Languages or Platforms If the system will use multiple languages and/or platforms. Repetitive Use of Large Working Sets of Objects If your application performs many successive transactions on the same working set of objects. Massive Scalability If you have very large amounts (Exabytes) of data, databases at large numbers of sites or a very large numbers of clients. Expensive Mapping Layer If you are expending effort on decomposing objects to fit into rows and columns you will be able save time by eliminating the mapping layer. You will no longer need to do ad hoc queries to locate individual objects by storing binary data as BLOBs (Binary Large Objects) or construct artificial JOIN tables to express many-to-many, tree and network relationships. 8 Objectivity Technical Overview

9 Introduction Why Use an ODBMS? Many applications require storage of complex data. Many of today s applications are more sophisticated than traditional OLTP. These new applications cover a broad domain, including authoring systems, bioinformatics, collaborative engineering, finance, groupware, intelligence, logistics, manufacturing, multimedia, process control, and telecommunications. These applications require additional data management capabilities not available from RDBMSs and ORDBMSs, such as complex data models, high-performance object management, relationship navigation, and transaction models for both long and short transactions. Combines the power of object technology with traditional DBMS features. Objectivity/DB meets the feature and performance requirements for mission-critical applications by combining the flexibility of object technology with the data management capability of RDBMSs and ORDBMSs. It provides high-performance data management without sacrificing integrity or multiuser concurrency. Like RDBMSs and ORDBMSs, Objectivity/DB supports the ACID properties of DBMSs. These properties are Atomicity, Consistency, Isolation, and Durability. Atomicity means that each transaction is executed completely or not at all. Consistency means that transactions are serializable, allowing the database to remain in a consistent state. Isolation indicates that the operations within one transaction are not affected by other transactions. This is accomplished through locking mechanisms. Durability means that committed data persists despite machine failures. Objectivity Technical Overview 9

10 Introduction Standards ODBMSs differ from RDBMSs and ORDBMSs in several ways. Some of the characteristics of DBMSs are listed in Table 1-1. If your database application requires one or more of the ODBMS characteristics, Objectivity/DB may be the right DBMS choice. Even if your application does not have these requirements, but you want the benefits of distributed data, Objectivity/DB may provide the best solution. Table 1-1: ODBMS, RDBMS, and ORDBMS Comparison Database Requirements ODBMSs RDBMSs ORDBMSs ACID properties Yes Yes Yes Data model complexity High Low a Variable User-defined data types Yes Not yet Yes Versioning Yes No No Application development Object-oriented 4GL 4GL, 3GL Polymorphism Yes No Maybe Inheritance Yes No Maybe a. Varies with different vendors Standards Over the past several years, computer users have demanded open systems across all aspects of software technology, from operating systems to languages to databases. Standards organizations such as the Object Data Management Group (ODMG) were established to ensure that present and future ODBMS systems will support the open environments users need. As a founding member of ODMG, Objectivity played a leadership role in helping develop these standards. Objectivity played a key role in standards organizations. Objectivity has played a key role in the Object Management Group (OMG), actively participating in the CORBA Persistent Object Service standard, and supporting ANSI standards for C++ and SQL. Objectivity is also active in the formal and informal processes for extending the Java language and defining database access standards in a grid environment. 10 Objectivity Technical Overview

11 Introduction Typical Applications Typical Applications Objectivity/DB is effectively employed as a database storage solution in a variety of enterprises. This section describes some typical areas of deployment. Bioinformatics Applications. Objectivity/DB is appropriate for a variety of bioinformatics applications. Objectivity/DB s ability to store and process complex structures (collections, trees and networks), images and data streams makes it an ideal database for applications such as: Comparing and visualizing nucleotide or protein sequences Analyzing and visualizing phylogenetic trees and 3D protein structures Protein structure prediction Primer design and analysis Strategic recombinant design Computer-Aided Molecular Design Data-Intensive Sciences Applications. Objectivity/DB is used in a wide range of data-intensive science applications. Objectivity/DB is typically implemented in the following areas: Modeling and simulation Experiment design and simulation Process simulations Macro-nano Earth Sciences and bioinformatics models Analyzing and archiving Equipment calibration records High speed data ingest to a persistent store Experiment results management Relationship hunting Data correlation for experiments in the astrophysical, biological, chemical, fusion, HEP and other domains Government Applications. Objectivity/DB technology is used in many different government applications. Government customers typically implement Objectivity/DB in the following areas: C4ISR and data fusion High Performance computing Simulation & virtual reality assisted training Correlation and analysis Video/Audio/ELINT/Telemedicine streaming and processing Logistics and operational modeling Objectivity Technical Overview 11

12 Introduction About Objectivity, Inc. Manufacturing and Process Control Applications. Objectivity/DB is typically used by manufacturing and process control administrators to manage and store complex data. Some typical manufacturing and process control applications of Objectivity/DB are in the areas of: Designing, modeling and simulating Process and plant design Process control simulations Virtual reality assisted training Monitoring, analyzing and responding Instrumentation systems Statistical process control Quality assurance data collection and archiving Plant management Manufacturing planning and logistics Relationship hunting Long term fault/anomaly correlation Downtime analysis Telecom Applications. Objectivity has extensive experience in the deployment of distributed ODBMS technology in advanced telecommunications systems. Objectivity/DB is in use within wireline, wireless and cable systems. Objectivity/DB is employed in many telecommunications applications. These systems have been implemented either by equipment manufacturers and software tool vendors or by systems integrators in partnership with service providers. Typical deployments include: Intelligent networking Network management and element management systems Telecommunications software development tools Multimedia repositories About Objectivity, Inc. Objectivity is an industry leader. Objectivity, Inc. is the leading provider of distributed, scalable object databases with unrivaled support for mixed-language development and mixed hardware. World-wide sales support and consulting. Objectivity provides high-quality technical support and consulting to guarantee customer success with the 12 Objectivity Technical Overview

13 Introduction About Objectivity, Inc. development of data models and system architecture for optimal performance and reliability. Expert engineers with experience in DBMSs and application development comprise Objectivity s customer support and critical response teams. In international markets, Objectivity works with experienced distribution and consulting firms to provide local support to customers. For the location of a sales office in your area, visit the Objectivity Web site (www. objectivity.com/). Objectivity Technical Overview 13

14 Introduction About Objectivity, Inc. 14 Objectivity Technical Overview

15 2 Product Family Overview Objectivity/DB is a distributed ODBMS. It is designed for mission-critical and production environments and offers high performance, virtually unlimited scalability, and interoperability across all major platforms and operating systems. The Objectivity software product family is shown in Figure 2-1. Objectivity Professional Services HPSS and Security Frameworks Parallel Query Engine (PQE) Open-File System (OFS) Active Schema High Availability Application Programming Interfaces Assist (Eclipse IDE) IPLS Objectivity/DB and Developer Tools Figure 2-1 Objectivity Product Family Objectivity/DB Product Group The core database product is Objectivity/DB. It consists of shared libraries, several servers, and a comprehensive set of administrative tools. The shared libraries and servers provide storage and retrieval of local and distributed data, and ensure data integrity through transaction management. The administrative tools make administration of Objectivity/DB databases easy and flexible. 15

16 Product Family Overview Distributed Database Architecture Other members of the Objectivity/DB product group include Objectivity/DB High Availability, Objectivity/DB In-Process Lock Server, Objectivity/DB Parallel Query Engine and several framework products. Distributed Database Architecture At the heart of Objectivity/DB is a unique, distributed database architecture. Because of this architecture, Objectivity/DB can deliver consistent high performance even as the amount of stored data and number of users grows. Objectivity/DB databases can support Exabytes of data spread physically across a corporate enterprise. Objectivity/DB offers the most advanced distributed database architecture available today. This architecture distributes processing and data storage tasks transparently across machines, thereby allowing applications to benefit from high-performance local processing through modern networks. This architecture also provides scalability and high availability to distributed data. Objectivity/DB is partially grid-enabled. Future releases of Objectivity/DB will virtualize communications with processes and the placement of files. Shared Libraries and Remote Servers Figure 2-2 shows Objectivity/DB s distributed architecture: distributed data and multiple servers. Instead of a traditional, centralized database server, Objectivity/DB provides a client-side shared library and several types of specialized server processes lock servers, data servers, and query servers. Lock servers grant permissions (locks) to an application seeking to access data. Data servers deliver data from storage on disk to application memory. The shared library, which is linked to the application, acts as a data server by obtaining data directly from the local file system. The shared library can also contact separate data-server processes on local or remote machines. A remote data server can be either Objectivity/DB s advanced multithreaded server (AMS) or an industry standard file server. See Chapter 4, Distributed Database Architectures, for more information. Query servers improve the performance of certain queries. Query servers are provided with Objectivity/DB Parallel Query Engine. See Chapter 11, Objectivity Solutions, for more information. 16 Objectivity Technical Overview

17 Product Family Overview Distributed Database Architecture Host 1 Host 2 Host 3 Application Process Objectivity Kernel Local Filesystem Application Process Objectivity Kernel Data Server AMS Data Server AMS Lock Server Query Server Figure 2-2 Distributed Data and Multiple Servers High-Performance Page-Server Architecture Objectivity/DB remote data servers use a page-server architecture to provide optimal performance over a wide range of work loads. Page servers typically outperform object-server architectures by providing significantly higher transfer rates. Objectivity/DB adds cross-transaction caching for an added performance boost for applications. Objectivity/DB data servers offer the highest possible performance for a wide range of multiuser applications. Flexible locking mechanisms, including multiple readers, one writer (MROW) for non-blocking writes, allow Objectivity/DB to achieve high concurrency. Two implementations of remote data servers are available. The first is Objectivity/DB s advanced multithreaded server (AMS). AMS provides multithreaded fast access to data and significantly improves performance when updating data stored in remote databases. Remote data servers can also be provided by industry standard file servers offered by companies such as EMC, Hewlett Packard, IBM, Silicon Graphics and Sun Microsystems. Objectivity Technical Overview 17

18 Product Family Overview Development and Administration Tools Multiple, Distributed I/O Servers As was shown in Figure 2-2, Objectivity/DB enables you to distribute data transparently across multiple machines in a network. As a result, multiple data servers can share the processing load for server operations. This eliminates the processing bottleneck that may occur when data is stored on a single machine. Because the physical location of objects is transparent to Objectivity/DB applications, you can locate and relocate objects on any machine without changing your application code. Parallel I/O and Locking for Multiple Client Processes To further improve performance, Objectivity/DB data server processes execute I/O requests from multiple application processes in parallel, even if the requested data resides on the same machine. Each application process can perform its I/O and lock operations without waiting for a server process to complete an I/O or lock request from another application. Development and Administration Tools Objectivity/DB provides a graphical tool, Objectivity/Assist, to help you develop databases and database applications, and browse and query the data. Objectivity/Assist uses the popular open-source Eclipse Platform. In addition to development tools, Objectivity/DB provides a set of administration tools and APIs to help you successfully administer and deploy your databases and applications to end users. These tools include database configuration, online incremental backup and restore, online object relocation, recovery tools, and database compaction, among others. For additional information, see Chapter 12, Development and Administration Tools. Safe Object Reference Architecture Consistent with ODMG standards, Objectivity/DB accesses objects through indirect handles or object references instead of direct pointers. Through handles and object references, Objectivity/DB guarantees object integrity, even if the object is moved to another location in memory after the transaction is completed. 18 Objectivity Technical Overview

19 Product Family Overview Interoperability Interoperability Objectivity/DB provides complete and transparent interoperability (platform heterogeneity) among leading hardware architectures and operating systems, allowing you to distribute or move databases among different machines. This support lets you use your existing hardware investment, and gives you flexibility in choosing future hardware. Objectivity/DB also provides interoperability between supported languages, such as C++ and Java, within the constraints of those languages. Application-Side Caching With Objectivity/DB, an object is brought intact directly from disk the first time it is needed. The object is then stored for easy access in the cache where the application process is running. Since navigation between cached objects occurs at in-memory speeds, this application-side caching of objects further improves performance. As a result, object navigation operations are significantly faster with Objectivity/DB than with RDBMSs and ODBMSs that only perform caching in centralized servers. High-Performance Concurrency Support Data Modeling Features Many complex applications today are designed for workgroup environments. These applications unite computing and communications, allowing people to collaborate on complex problems through workgroups. Applications of this kind often use object-oriented technology, a variety of server configurations (client/server, mixed-tier, and so on), and elegant graphical user interfaces. In addition to text, users share and operate on complex object types such as images, three-dimensional graphics, and sound. Objectivity/DB provides concurrency support through locking, transaction semantics, and the ability to support multiple readers, one writer (MROW) on the same data at the same time. Objectivity/DB supports several data modeling mechanisms consistent with the various supported programming languages: For C++ applications, you design your data model using the Data Definition Language (DDL). The DDL is standard C++ with a few extensions supporting Objectivity/DB features such as object relationships. For Java or Python applications, you define classes that take advantage of the capabilities of Objectivity/DB. Objectivity Technical Overview 19

20 Product Family Overview Data Modeling Features For Smalltalk applications, you use standard VisualWorks classes and Objectivity/DB classes to develop your data model. For SQL applications, Objectivity/DB automatically generates an equivalent SQL data definition for all object classes, making it easier to integrate legacy applications and ODBC tools. Layered Hierarchy for Object Storage As shown in Figure 2-3, Objectivity/DB provides logical storage through persistent objects, databases, and federated databases. These objects continue to exist after your application terminates, and can be shared among applications, with locking managed by Objectivity/DB. The highest level of storage is the federated database. Each federated database logically contains one or more databases. It also contains the data model (or schema) and a catalog of all databases that comprise the federation. Each database contains persistent objects. Objectivity/DB also supports the clustering of objects within containers to enhance performance. See Object Clustering on page 47. An object can consist of object-oriented language-specific constructs, with the addition of variable-size arrays, relationships to other objects, references to other objects, and type constraints. Your application can create and delete persistent objects dynamically (except for the federated database itself). Multi-File Databases A database is physically represented by a database file, which contains: The database s default container Zero or more containers created by applications A catalog of the containers belonging to the database Database files are typically used to distribute related containers and their basic objects to different physical locations in your network. For a very large database (VLDB), the database s file size can become too large to manage easily. To solve this problem, an application can choose to store selected external containers in their own separate container files. External containers enable the creation of multi-file databases. Together, database files and container files are the federated database s data files. 20 Objectivity Technical Overview

21 Product Family Overview Data Modeling Features Federated Database Database Container Database Container Container Figure 2-3 Layered Storage Within a Federated Database Complex Interrelationships In your data model, you can design objects with complex interrelationships that Objectivity/DB maintains automatically. These relationships can be unidirectional or bidirectional. For C++ applications, these relationships are called associations. Composite Objects Relationships also allow you to create composite objects consisting of groups of interrelated objects. These objects can behave as one object for locking and delete operations. Advanced Schema Evolution Capabilities Objectivity/DB allows you to easily change your data model (schema) as the needs of your applications and users change. This feature, known as schema evolution, allows you to change classes, their contents, and their inheritance. Supported operations include: renaming and deleting a class; adding, deleting, and changing data members, associations, and object references; and adding, deleting, and changing base classes, among many others. For many operations, Objectivity/DB automates the conversion process for existing objects changed by schema evolution. For more complex changes, Objectivity Technical Overview 21

22 Product Family Overview Objectivity/DB High Availability Objectivity/DB provides programming interfaces that support the data conversion process. Along with Objectivity s patented schema evolution, Objectivity/DB Active Schema allows an application to completely reconfigure itself to accommodate changing conditions, without taking the application or database offline. Objectivity/DB Active Schema also guarantees the integrity of the resulting database schema through the use of a unique Schema Engine. See Chapter 11, Objectivity Solutions, for more information. Active Schema is available in the C++ and Java bindings. Objectivity/DB High Availability Objectivity/DB High Availability is an option that supports fault tolerance and improved performance for large database deployments. This product improves data availability after network failures by distributing lock management and replicating the database schema and catalog to multiple locations in a network. Using Objectivity/DB High Availability, database administrators can divide a federated database into autonomous partitions. Partitioning replicates metadata in each partition so that each partition is self-sufficient and insulated from network and other system failures. Objectivity/DB High Availability is transparent to deployed applications. Objectivity/DB High Availability improves data availability and read performance by replicating a subset of data in multiple locations in a network. Replicating data can also eliminate the danger of a single point of failure to the database. Objectivity/DB Parallel Query Engine Objectivity/DB Parallel Query Engine enables you to perform fast queries over very large numbers of objects in a very large database (VLDB). This product also supports more complex search or filtering algorithms, exploits the use of distributed resources (for example, many cheaper server hosts instead of a very large central server), and will eventually support searches of non-objectivity DBMSs. You typically use Objectivity/PQE when you need to scan large amounts of data in order to find comparatively few objects satisfying a predicate condition. Objectivity/PQE speeds up such a search by employing concurrent multithreaded query servers to scan different portions of the data in parallel. When each query-server thread searches a set of databases, it reads their data into its own cache, instead of into the cache of the querying application. This can significantly reduce network traffic when the databases are on remote hosts. 22 Objectivity Technical Overview

23 Product Family Overview Objectivity/DB In-Process Lock Server Objectivity/PQE has modifiable components for determining the target containers and databases, controlling the local thread pool, and filtering objects before they are returned to the client. For example, to optimize the query according to how you ve organized your data, you can customize the search so that every query-server thread scans selected containers of selected databases. Objectivity/DB In-Process Lock Server In addition to a standalone lock server process, Objectivity offers an in-process lock server (IPLS), which can be linked to an Objectivity application. From the perspective of other distributed clients, the Objectivity/IPLS performs the same way as a regular lock server. However, when most of the persistent operations are coming from a single multithreaded process then the inter-process communication is greatly reduced, and greater performance can result. Objectivity/DB Open File System Objectivity/DB Open File System (Objectivity/OFS) provides a customizable interface between Objectivity s distributed, scalable database and high-end storage solutions. This capability is used to provide an interface to High Performance Storage Systems (HPSS) and can also be used to create an interface to other Hierarchical Storage Manager (HSM) systems. Objectivity/DB has been run successfully with the SGI InfiniteStorage Shared Filesystem CXFS for Storage Area Networks (SANs). Objectivity/DB data files can be protected using standard operating system security mechanisms. Because these mechanisms may be inadequate for some applications, it is possible to call a Generalized Security Architecture hook from within the Objectivity/OFS layer. The functionality invoked by the hooks can interact with the appropriate security mechanisms in use at a particular site. This makes it feasible to have a federation with partitions spread across multiple sites, each imposing their own security policies. Kerberos authentication is used by default, but the client to AMS protocol can be modified to use other mechanisms. SQL++ users can also use the standard SQL GRANT and REVOKE mechanisms. Powerful Object-Oriented Development The Objectivity/DB product family is designed for mission-critical application development and deployment. It supports object-oriented design and implementation through industry-standard languages. With Objectivity/DB, you can develop object-oriented applications using standard languages, compilers and development tools you already know. There is no need to use non-standard compilers, debuggers, or tools. Objectivity Technical Overview 23

24 Product Family Overview C++ Support C++ Support C# and.net Support Java Support Objectivity/DB is fully compatible with standard integrated design environments. Because of this compatibility, you can choose among many popular development environments when developing Objectivity/DB applications. Objectivity/C++ provides a programming interface that is transparent to native C++ compilers. With Objectivity/C++, you can add object database features to your applications using standard C++ functionality, including single and multiple inheritance and templates. Objectivity enables complete support for the.net platform, thereby enabling the development of massively parallel database applications. Objectivity has made its high performance persistence engine a managed runtime that is callable from any language that compiles to IL (Intermediate Language). Therefore, persistent objects can now exist in all of the Microsoft. NET supported languages such as C#, VB.NET C++ and J#, as well as non-microsoft languages that support the.net Common Language Runtime, including Ruby, LISP, and FORTAN. Objectivity provides full support for Java with a language binding compliant with the ODMG standard. Objectivity s unique single process model architecture for Java-based applications provides the highest performing solution in the industry, because the application program and the Java Virtual Machine run in the same process, ensuring that valuable processing cycles are not wasted on constant process switching. Along with full support for Java multithreading, Objectivity for Java represents a truly industrial-strength solution for complex computing needs. A Java 2 Platform Enterprise Edition (J2EE) Connection Architecture resource adapter allows Objectivity for Java to interface with any application server supporting JCA. Rather than requiring manual vendor-specific connection management coding, the JCA feature uses the Common Client Interface (CCI) to manage connections to an Enterprise Information System. The resource adapter has been tested with the BEA WebLogic and IBM WebSphere application servers, making it easier to incorporate Objectivity for Java into systems that use a Service-Oriented Architecture. 24 Objectivity Technical Overview

25 Product Family Overview Python Support Python Support Smalltalk Support SQL++ Support XML Support Namespace Support The Objectivity/Python binding provides an easy to use, dynamic, scriptable interface to Objectivity/DB. Objectivity/Python can be used to develop new applications or to link together existing applications. Python is a high-level interpreted language that provides an excellent interface to the operating system and network. Easy to script, Python is ideal for database administration, QA testing, and the development of analysis programs. With Objectivity/Smalltalk, you use VisualWorks from Cincom to develop applications. You can also use the ENVY/Developer from Object Technology International for multiuser development environments. As is standard in Smalltalk, with Objectivity/Smalltalk you benefit from automatic garbage collection and object evolution. The Objectivity/SQL++ product is an ANSI SQL-92 standard SQL engine you can use to access and manipulate objects stored in Objectivity/DB databases. Like SQL, Objectivity/SQL++ allows you to create tables, delete tables, and update objects. It provides an object relational interface to an Objectivity/DB federation. XML is an increasingly important standard for facilitating data transfer between business systems. The Objectivity XML Interface tools support: Exporting objects from the latest version of Objectivity/DB in XML format Importing XML formatted data into Objectivity/DB Objectivity/DB provides full support for namespaces (Java packages) for persistence-capable classes. For example, C++ persistence-capable classes may be defined in a namespace, and namespace qualifiers may be used to refer to them. Similarly, Java persistence-capable classes may be defined in a package, and a package may be used to refer to them. Objectivity/DB Active Schema provides a namespace API for iteration and scope lookup. Objectivity Technical Overview 25

26 Product Family Overview Language Interoperability Language Interoperability Scalable Collections Persistent objects can be created by applications written in any of the Objectivity/DB application program interfaces. Once created, persistent objects can be accessed by applications using the same API or by applications using any other Objectivity/DB API, provided the corresponding class definitions in all applications are compatible with schemas of the shared federated database. This allows for more flexibility to select the language best suited to the application development environment. Objectivity/DB supports language-independent persistent collections that are scalable to hundreds of millions of objects. For example, a collection created in C++ would be accessible in Java or Smalltalk, and vice versa. The scalable collections are patterned on JDK 1.5 specifications, and the Objectivity for Java implementation uses the Java 2 interfaces. Objectivity/C++ Product Group Objectivity/DB supports industry-standard object-oriented programming languages for developing database schemas and applications. You use these languages to define the objects to be stored in your database and to create applications that access and manipulate the stored objects. The Objectivity/C++ group consists of Objectivity/C++ and Objectivity/C++ Data Definition Language. Objectivity/C++ provides many features for implementing C++ applications. Table 2-1 briefly summarizes some of these features. Table 2-1: Objectivity/C++ Development Features Feature Creation and deletion functions Association functions Create and delete objects. Description Set, delete, and navigate associations (relationships) between objects. 26 Objectivity Technical Overview

27 Product Family Overview Objectivity/C++ Product Group Table 2-1: Objectivity/C++ Development Features (Continued) Feature Access functions Version functions Copy and move functions Object and class migration function Class libraries for C++ applications Description Name and look up objects. Use map dictionaries to look up objects based on keyed values. Define index values for objects and use indexing functions to access objects across containers, databases, and federated databases. Traverse objects using iterators that select objects based on object type, name, an indexed value, containment, association, or user-defined selection criteria. Query and access objects using the Objectivity/SQL++ interface. Create versions of objects. Track version genealogy. Locate any version at any time. Specify any object as the default version. Copy or move objects from one container to another. Evolve existing classes and create new ones. Variable-size array classes for creating and manipulating persistent and transient arrays that can vary in size dynamically. String classes for creating and manipulating variable and fixed-size strings. Map dictionary classes for creating dynamic structures that map user-defined keys to objects. Scalable collection classes for bags, sets, lists, trees and other common groupings of objects. Objectivity Technical Overview 27

28 Product Family Overview Objectivity for Java Product Group Objectivity for Java Product Group Objectivity provides full support for Java with a language binding compliant with the latest ODMG standards, thereby enabling the rapid development and deployment of complex, mission-critical applications. Objectivity for Java provides many features for implementing Java applications. Table 2-2: Objectivity for Java Development Features Feature Description Full support for Java Compliant with Java 2. Single process model architecture Flexible persistence model Java multithreading support ADE support Platform portability The application program and the Java Virtual Machine run in the same process, ensuring that processing cycles are not wasted on constant process switching. Classes can be made persistence-capable through inheritance or by an interface; full control of when and how an object is stored in the database is provided. Transactions can share threads or have a thread dedicated to a particular transaction, providing an industrial-strength solution for complex computing needs. Complete support and compatibility with leading application development environments (ADEs) including debuggers. Investment in application development is preserved because Objectivity for Java delivers true portability across a wide variety of hardware platforms. 28 Objectivity Technical Overview

29 Product Family Overview Objectivity/Smalltalk Product Group Objectivity/Smalltalk Product Group The Objectivity/Smalltalk product group extends standard VisualWorks Smalltalk from Cincom for developing Objectivity/DB applications. Objectivity/Smalltalk provides many classes and methods you can use when developing Smalltalk database applications. Table 2-3 briefly summarizes some features these classes and methods provide. Table 2-3: Objectivity/Smalltalk Development Features Feature Creation and deletion functions Association (relation) functions Access functions Version functions Class libraries for Smalltalk applications Automatic online database garbage collection Automatic storage management features Automatic object and class migration Implicit feature of Smalltalk. Description Set, delete, and navigate relations between objects. Use standard Smalltalk dictionaries to lookup objects based on keyed values. Traverse objects using iterators that select objects based on relations. Create versions of objects. Track version genealogy. Locate any version at any time. Specify any object as the default version. Supports Smalltalk class libraries from Cincom. Supports standard automatic garbage collection of transient and persistent objects. Provides automatic mechanisms for locking, tracking, and clustering objects. Supports automatic migration of objects and classes as in standard Smalltalk. Objectivity Technical Overview 29

30 Product Family Overview Objectivity/SQL++ Product Group Objectivity/SQL++ Product Group Object Extensions With Objectivity/SQL++, applications and SQL-compliant tools can use ANSI-standard SQL to create, access, manipulate, and delete objects stored in an Objectivity/DB database. Objectivity/SQL++ provides full support of ANSI SQL In addition to supporting standard ANSI SQL, Objectivity/SQL++ supports object extensions to SQL. These extensions support inheritance and allow you to access data members, associations, object references, and non-sql types such as arrays. One use for this feature is to use associations and object references as implicitly-defined indexes for SQL join operations. Stored Procedures and Triggers User-Defined Security ODBC Tool Support Objectivity/SQL++ supports stored procedures. This feature lets you call C++ and C procedures directly from SQL. You can use built-in stored procedures, or build your own using a standard set of data types, data structures, macros, and member functions. Objectivity/SQL++ also supports triggers. Triggers are C++ procedures that are called whenever SQL performs a database modification. You can implement and register your own triggers for SQL INSERT, UPDATE (pre-update and post-update), and DELETE operations. Through Objectivity/SQL++, you can control access to your database by implementing security based on a user name, a password, or both. Objectivity/SQL++ provides an Objectivity/SQL++ ODBC Driver (level 3), which is an SQL programming interface for end user applications. It allows client/server tools that are based on the Microsoft Open Database Connectivity (ODBC) interface to access and manipulate Objectivity/DB objects. These off-the-shelf tools are available from numerous vendors and include graphical user interface builders, form entry systems, report generators, and ad hoc query tools. Any tool that is ODBC-compliant can access data in an Objectivity/DB database. 30 Objectivity Technical Overview

31 3 Data Modeling This chapter describes Objectivity/DB data modeling common to all Objectivity/DB applications. Data modeling is the key factor in choosing any DBMS. Data modeling with Objectivity/DB is identical to modeling data in non-database applications; that is, you decide which objects you want to use in your application and how those objects behave and interact. With Objectivity/DB, you can also specify objects you want to store in the database and relationships between objects. The data model you define for your database and its applications is called the schema. Complex Data Models Today s complex business applications require complex data models. While RDBMSs continue to offer good solutions for data models of low complexity, ODBMSs are well suited for implementing highly complex data models. Some of the most complex models include acyclic graph and network data models, such as those used for bill-of-material modeling and network topologies. For complex data models, RDBMSs fall short in terms of performance. Representing complex data in an RDBMS requires numerous tables that must be joined together to represent a desired data set. However, these join operations are inefficient, particularly when joining three or more tables. To work around this inefficiency, RDBMS application developers instead implement simpler database data models and increase the complexity of the data model used by applications, as illustrated in Figure 3-1. Where possible, developers reduce the complexity of the operations used against the relational database, and assemble the desired complex data within the application using mapping code, which can amount to between 20 and 45 percent of typical applications, and which can be a considerable update and maintenance burden. 31

32 Data Modeling Efficient Navigation Between Objects For example, rather than using a series of join operations for a query, developers write code that performs a sequence of queries on individual tables, using the result of each query as a filter for the next until the desired result is obtained. Efficient Navigation Between Objects Because in ODBMSs the data model is the same in the application and the database, the application can make use of navigation between objects. This is more efficient and provides better performance than when RDBMSs search through indexes and table joins. Data Model Maps to Application and Database ODBMSs are ideal for data models of high complexity. With Objectivity/DB, the data model you design is tightly integrated with the applications you develop. As shown in Figure 3-1, there is a natural mapping of the data model to both the application and the database. There is no need to translate between application objects and those stored in the database because the application and database share the same data model. This inherent mapping of data on disk to that used in the application significantly improves performance because the data does not have to be constructed from simpler parts. Mapping of Data Model By RDBMS By ODBMS Application Application Mapping Code Database Database Figure 3-1 Data Model Mapping of Application and Database 32 Objectivity Technical Overview

33 Data Modeling Object Relationships Object Relationships Objectivity/DB also provides a mechanism for defining relationships between objects. This mechanism provides higher-level capabilities than simple pointers for modeling and managing relationships between objects. Relationships are maintained automatically by Objectivity/DB. Objectivity/DB supports unidirectional and bidirectional relationships, relationship cardinality, and the use of relationships to create composite objects. Relationships Are Unidirectional or Bidirectional Relationship Cardinality A unidirectional relationship is one in which only one class defines a relationship link to another class. A relationship between two objects in which each object s class has a relationship link to the other class is called a bidirectional relationship. Bidirectional relationships allow you to locate an associated object from either of the two associated objects. An important advantage of using bidirectional relationships in your data model is that Objectivity/DB maintains the referential integrity of your object relationships, and therefore contributes to the robustness of your application. A relationship has two ends. A relationship s cardinality is the number of objects on one side of a relationship that may be related with objects on the other side. For example, in Figure 3-2, there is a one-to-many relationship between a library object and book objects. Objectivity/DB relationships support four categories of cardinality: 1:1 One-to-one 1:m One-to-many n:1 Many-to-one n:m Many-to-many Objectivity Technical Overview 33

34 Data Modeling Relationships in Composite Objects fromlibrary Book inloan allbooks title author subject isbn hasbook Library name address phone hasloans fromlibrary Loan duedate Figure 3-2 Data Modeling With Relationships Relationships in Composite Objects You can use relationships to create composite objects, which group interrelated objects so that they behave as one object for propagating operations such as deletion and locking. Any type of object can be used in a composite object, and an object can be part of any number of composite objects. When you define relationships between your classes, you can specify which operations should propagate, and the direction of propagation. As shown in Figure 3-3, a book can be a composite object combining chapters and pages. When you delete the book, you also delete its chapters and pages. P P Book Chapters Pages Figure 3-3 Composite Object Propagation along a relationship is optional. That is, an operation works on a single object, unless the operation is explicitly issued with a propagate attribute. If the attribute is present, the operation applied to the target object will propagate to associated objects. 34 Objectivity Technical Overview

35 Data Modeling Evolution of Data Models and Object Conversion Evolution of Data Models and Object Conversion Over time, you will make changes to your applications to incorporate new features. Accordingly, you will need to make comparable changes to the data model your applications use. This process is known as schema evolution. Data Models and Objects Can Evolve With Your Needs As shown in Figure 3-4, Objectivity/DB allows you to evolve a data model as the needs of your applications and users change. Through schema evolution, you can go beyond simply adding new classes to your schema. You can also make changes to existing classes, their contents, and their inheritance. For example, you can use schema evolution to: Rename and delete classes. Add, delete, and change data members, associations, and object references. Add, delete, and change base classes. If this is combined with the performance improvements in new hardware happening every 6 to 12 months, then Objectivity/DB can leverage processor and I/O subsystem improvements more easily. Existing Schema Evolved Schema Schema Evolution Process Before Schema Evolution After Schema Evolution Figure 3-4 Schema Evolution Objectivity Technical Overview 35

36 Data Modeling Automatic and User-Defined Object Conversion Options Automatic and User-Defined Object Conversion Options Objectivity/DB automates the conversion process for existing objects that have been changed by schema evolution. This object conversion capability is especially important in production environments where an upgrade to a new software version usually requires that existing objects be converted to the new data model. Control Timing and Granularity of Conversion The object conversion process is shown in Figure 3-5. You can choose the timing of object conversion as either deferred, on-demand, or immediate. A deferred conversion mode converts objects the first time they are accessed. An on-demand mode invokes object conversion from within an application to convert objects at a specified granularity. The immediate mode implies that a single-use upgrade application will be used to help convert all objects in a federated database within a single transaction. This upgrade application only is necessary for replacing a base class or deleting a class that has inherited references, and is supported by special upgrade functions. These three conversion modes give organizations the freedom to choose the type of schema evolution timing that is best for them, depending on their application and operational requirements. For any of these conversion modes, you can also provide conversion functions to calculate values of new data based on old values during the conversion process. Evolved Schema Evolved Schema Automatic Conversion Upgrade Application Before Object Conversion After Object Conversion Figure 3-5 Object Conversion 36 Objectivity Technical Overview

37 4 Distributed Database Architectures Objectivity/DB delivers consistent high performance for database applications as the number of users, data servers, and data volume increases. It provides much of this performance because of its distributed architecture, which helps eliminate processing bottlenecks that affect databases with centralized architectures, including RDBMSs and most ODBMSs. Evolution of Client/Server As shown in Figure 4-1, database architectures have evolved from an asymmetric, server-dominant design to a design that puts increasing processing on client machines. Initially, one server handled most of the processing tasks for one database application. 37

38 Distributed Database Architectures Server-Dominant Models Created Bottlenecks CLIENT Presentation Presentation Presentation Presentation Application Function Application Function Application Function Application Function Application Function Data Management Data Management Data Management Data Management Data Management SERVER Based on a source from the Gartner Group Figure 4-1 Client/Server Evolution Server-Dominant Models Created Bottlenecks Early database management systems were developed during the mainframe era, when a single mainframe served hundreds of terminals. Mainframe servers managed applications, I/O processing, database operations, and terminal display services. This architecture was adopted by RDBMSs and proliferated with the introduction of minicomputers. When PCs became available, they replaced the terminals used in database configurations. Each PC had only 1/100th of the computational and I/O power of a mainframe, but this was enough to locally run graphical user interface (GUI) presentation services. This distribution of GUI services enabled mainframes to focus on number-crunching and I/O operations, including centralized database services. As the power of PCs increased, more and more application functionality was moved to the client. This transition was limited by the traditional data server, which remained centralized and monolithic. To take further advantage of the inexpensive power of PC microprocessors, hardware manufacturers moved from 38 Objectivity Technical Overview

39 Distributed Database Architectures Distributed Database Architecture a single microprocessor design to multiple parallel processors. RDBMSs followed with products that tried to capitalize on the speed of parallel processing, but were still limited by the centralized data server design of the past. Centralized RDBMSs also placed I/O, concurrency control, and most database system services on the server. In an attempt to reduce the load on the server process, some RDBMSs placed services that control page contents on the client processes. Despite this change, the majority of system services and all I/O services remained on the server, forcing every single page used by the client to be read through the server process. The centralized server model used during the mainframe era began to break down in light of these new configurations. For example, consider a network with ten workstations and one server where the ten workstations could each process ten million instructions per second (MIPS) and the server could process 25 MIPS. Using the centralized mainframe model wastes most of the 100 MIPS of compute power of the ten client workstations. Adding more workstations just makes the situation worse, since additional workstations add to the load on the server, without adding usable resources. This server-dominant architecture quickly becomes a performance bottleneck. Distributed Database Architecture The distributed database architecture used by Objectivity/DB solves the performance bottleneck problems seen in other DBMSs by permitting distribution of database services among machines, including object caching, updates, traversals, and conversion operations for interoperability. I/O services are moved to separate server processes on the machines where the data resides. Concurrency control is centralized and even that overall task can be subdivided between distributed servers. For a visual representation of distributed database architecture, see Figure 2-2 on page 17. Objectivity Technical Overview 39

40 Distributed Database Architectures Increased Scalability Increased Scalability This balanced database architecture gives Objectivity/DB databases increased scalability. Making good use of the CPU resources that centralized implementation wastes, most of the computation can be done on the same machine as the application. The result is that Objectivity/DB can handle many more client processes than centralized systems can, and the processing power and performance of the ODBMS increases as each new machine is added to the system. By using a distributed architecture, Objectivity/DB databases can grow to virtually unlimited size and complexity while maintaining a linear performance profile. As a result, Objectivity/DB databases can support Exabytes of data spread physically across many machines. Applications can access this enterprise-wide data transparently, without knowledge of where the data actually resides. Petabytes of Data Objectivity/DB was used in a project which built the world's largest database. The SLAC BaBar project used Objectivity/DB to store the results of their High Energy Physics experiments. Data arrived at a rate of around one Terabyte per day. They used 2400 processors to analyze and work with the data. They eventually stored over one Petabyte of data, compressed down to about 895 Terabytes. This overall system architecture with a bank of machines receiving and storing streams of data, more powerful processors analyzing, correlating and refining the data and a group of specialists using the results (perhaps replicated to multiple sites) is common among Objectivity/DB deployments. High Ingest Rate Objectivity/DB's distributed, parallel architecture makes it highly suited to building true or near real-time applications. A very high ingest rate benchmark was run on two SGI Origin 32-processor machines. One machine was used for ingesting streams of data, correlating objects with other objects and storing both the objects and the discovered relationships. The other was used for supporting a heavy load of mixed queries. Objectivity/DB sustained an ingest rate of over one Terabyte per hour while concurrently supporting the query load. The 100 Terabytes of raw data was stored on 120 Terabytes of disk. The competing relational implementations required 300 Terabytes of disk to store the same raw data. Objectivity/DB associations were used to store the relationships between the incoming objects. This mechanism was not only faster than creating JOIN tables, but it also helped remove the need to index every object, as many of the objects were only used after finding related objects with a query. 40 Objectivity Technical Overview

41 Distributed Database Architectures Service-Oriented Architectures Service-Oriented Architectures A Service-Oriented Architecture is a group of communicating services, usually connected via Web services. Figure 4-2 shows a basic Service-Oriented Architecture. In the figure, a service consumer and a service provider are having a conversation made possible because of the way their connections are defined. Objectivity/DB has J2EE Connector Architecture (JCA) interfaces that have been tested with application servers such as BEA WebLogic and IBM WebSphere. JCA is a protocol used between the service provider and the service consumer. Service Provider service request Service Consumer service response Figure 4-2 Basic Service-Oriented Architecture High-Performance Page Servers Parallel I/O and Locking Objectivity/DB server processes perform page read and write operations, with one or more pages containing a cluster of objects. This approach differs from the approach taken by central server architectures, where rows or objects are the unit of transfer between clients and servers. However, most RDBMSs transfer internal data to and from disk as fixed-size pages. The Objectivity/DB approach of reading objects as clusters and caching them in the client can greatly improve performance, since it reduces the number of times objects need to be read from disk. In many cases, objects needed by an application will already reside in its shared library s cache. For a more detailed quantitative analysis of page server and object database architectures, see Appendix A, Page Server Performance. Objectivity/DB servers execute multiple disk I/O requests in parallel, even if the requested objects reside on the same machine. Because Objectivity/DB uses a separate server to manage lock requests, I/O requests can be processed Objectivity Technical Overview 41

42 Distributed Database Architectures Support for Dedicated I/O servers independently of lock requests. Even in single machine environments, where the machine host manages both locking and application I/O requests, the machine can process lock requests concurrently with I/O operations. Support for Dedicated I/O servers Many server machines offered today are specialized for I/O operations. They are not designed for CPU-intensive applications, and in fact often do not allow user applications to run on them. Because Objectivity/DB separates I/O services from the CPU-intensive system services, it can take advantage of these servers. Objectivity/DB application processes can directly access the high-speed I/O servers for their objects. Centralized systems instead have to place their server on a different machine, and their applications are forced to access data indirectly through that third machine, losing much of the advantage of the I/O server. 42 Objectivity Technical Overview

43 5 Objectivity/DB This chapter describes the basic database concepts that are common to all Objectivity/DB database applications. Safe Object Reference Architecture Objectivity/DB transparently and safely locates the objects you need in the database. Each language binding employs a different technique: Objectivity/C++ guarantees integrity of objects by using smart-pointers instead of direct pointers to access objects. This important architectural feature follows the ODMG guidelines, and protects Objectivity/DB from serious object corruption that can be caused by an errant application. For Objectivity for Java and Objectivity/Smalltalk applications, you access objects through standard references. Storage for Exabytes of Interrelated Objects Any Objectivity/DB transaction can transparently and simultaneously access all data in a database through 64-bit object references. For 64-bit file systems, this allows you to access up to approximately 2 64 (or 2 x ) bytes (10,000,000 Terabytes) of data and up to 2 58 (or ) objects. For 32-bit file systems, due to operating system file size constraints, you can access up to 2 47 (or ) bytes (100 Terabytes) of data and up to 2 42 (or ) objects. Many Objectivity customers have successfully developed and deployed federations with tens of thousands of databases, millions of containers, and Petabytes of storage. For up-to-date information about these customers and their experiences, contact your Objectivity sales representative. 43

44 Objectivity/DB Platform Interoperability Platform Interoperability Objectivity/DB provides complete interoperability among different architectures, which are supported combinations of hardware, operating system, and compiler. Objectivity/DB supports a variety of 32-bit and 64-bit architectures on two main types of platforms (Windows and UNIX). Objectivity/DB applications, servers, and tools can operate in virtually any combination across these architectures. Objects can be distributed and freely moved across different database architectures and shared among different kinds of machines. Objectivity/DB automatically and transparently handles inherent differences between these architectures, such as byte order, floating point format, and packing and padding alignment. Objectivity/DB translates on the object level and performs translation only when necessary. Location Independence of Objects The physical locations of objects are entirely transparent to Objectivity/DB applications. This allows you to locate and relocate objects on any machine without changing applications that access them. For example, for performance reasons you may want to move certain objects to the machine that accesses them most often. If access patterns to these objects change, you can move the objects again dynamically. Accessing Objects 64-bit OIDs At the lowest level, Objectivity/DB accesses persistent objects through 64-bit object identifiers (OIDs). Each object in an Objectivity/DB federated database has a unique OID. OIDs allow Objectivity/DB to locate and manage objects with flexibility and safety. OIDs contain information about the database, container, page, and slot where an object is located. The OID is split into 4 logical components as follows: First 16 bits Logical database identifier Second 16 bits Logical container identifier Third 16 bits Logical page within container Last 16 bits Logical slot within page 44 Objectivity Technical Overview

45 Objectivity/DB Consistency for Concurrent Database Access In general, the OID does not change for the lifetime of the object, unless you explicitly move an object to another container. If you move an object that is linked to other objects through bidirectional relationships, those links to the moved object are automatically adjusted. Consistency for Concurrent Database Access Objectivity/DB allows concurrent access to a database by multiple users and processes on a network. Objectivity/DB ensures that databases in a federation remain consistent when multiple applications access a database concurrently. As shown in Table 5-1, Objectivity/DB provides several concurrency features that facilitate today s high-performance applications. Table 5-1: Special Objectivity/DB Concurrency Features Feature Flexible transaction model Read-only database Multiple readers, one writer (MROW) Object-level versioning Hierarchical locking Lock waiting Active deadlock detection across databases Downgrade locks on checkpoint Description Supports transactions of any duration. A database can be marked as read only to avoid overheads. Improves concurrency by allowing multiple readers to read the most recent state of an object that is locked for update by another transaction. Allows a C++ or Smalltalk application to selectively operate on specific versions of objects. Two-phase hierarchical locking controls read and update access to objects. Allows a transaction to wait for a user-defined period for access to an object that is already locked by another transaction. Warns against potential deadlock situations in which one or more transactions are queued and each is waiting for a lock that will never become available. Allows an application to downgrade (from update to read) all locks obtained during a transaction when the transaction has been checkpointed. Objectivity Technical Overview 45

46 Objectivity/DB Locking Locking Objectivity/DB is a multiprocessing system which allows simultaneous access to multiple objects. To ensure data consistency, database access is managed through the use of locks. Locks Coordinate Database Access A process can obtain either a read lock, which allows other processes to read the objects, or an update lock, which prevents all other processes from reading or modifying the objects. A process can also obtain shared and exclusive locks that support propagation across composite objects. Objectivity/DB supports locking at three levels of granularity federated database, database, and container. Lock Server Manages Locks Locks are granted automatically and transparently to an application. You can also explicitly set locks. For each federated database, locks are managed by a lock server. For non-partitioned federated databases, the lock server is a centralized resource; there is one lock server running per federated database. All applications accessing the federated database request a lock for objects from this lock server. For federated databases that are partitioned using Objectivity/DB High Availability, there is one lock server per partition. Transactions Checkpointing Database applications perform work on objects within a transaction. Transactions allow a collection of operations to appear as a single atomic operation to the database that is, either all of the operations are performed or none are. Once a transaction is applied to the database, the changes made to objects during the transaction are stored on disk. However, an application may choose to abort a transaction at any time up to the commit point, leaving the database in its original state. You can checkpoint the changes made during a transaction. This stores your changes to the database, allowing other applications to access the changes. However, the transaction remains active and by default all previously set locks are still held by the application. Optionally, you can downgrade (from update to read) all locks obtained during a transaction when you checkpoint the transaction. 46 Objectivity Technical Overview

47 Objectivity/DB Object Clustering Object Clustering As shown in Figure 5-1, you can group objects within a database. This feature is known as object clustering, and is implemented using containment objects called containers. Using object clustering, you can improve cache performance by grouping objects based on their access patterns. Objects placed in the same container are physically clustered together in memory pages and on disk, so access to all objects in a single container is very efficient. Database Federated Database Container Database Container Container Figure 5-1 Object Clustering Object-Level Versioning Create Object Versions With Objectivity/DB, your C++ or Smalltalk application can create separate versions of an object and track its version genealogy over time. This feature is essential for applications that perform design and configuration management of complex systems. Your application can perform linear and branch versioning of objects, and can specify a particular version as the default version. For example, a linear version genealogy for a mechanical component object is shown in Figure 5-2. Each time the component is updated, another version is created. In this case, the most recent version is likely to be selected as the default. Objectivity Technical Overview 47

48 Objectivity/DB Create Object Versions Version 2.1 Version 2.2 Latest Version 1.1 Version 1.2 Version 1.3 Generic Version Figure 5-2 Object Versioning In Figure 5-2, note that the version names (or scope names) are set by the user, not by Objectivity/DB. The generic object provides a static OID for relationships with other objects. Associations to versioned objects can be given properties that define their behavior when an object instance is opened for update and versioning is turned on. Within your application, you can set relationships to any particular version of an object, or to the genealogy as a whole through a generic object. From the generic object, you can directly access any object version in the genealogy. Objectivity/DB versioning offers flexibility in design. The versioning functions do not enforce a particular configuration management model. Instead, they enable you to create systems that vary widely in scope and depth of function, as well as semantics. 48 Objectivity Technical Overview

49 6 Objectivity/DB High Availability Objectivity/DB High Availability (Objectivity/HA) supports two integral features: autonomous partitions and replication. Autonomous partitions allow you to improve fault tolerance by partitioning federated databases into independent units. Replication improves the availability of particular databases by allowing you to replicate them that is, to create and distribute managed copies (called images) among multiple autonomous partitions. Partitioning a Federated Database With Objectivity/HA, you create autonomous workgroups by separating your federated databases into independent pieces called autonomous partitions. You can distribute these partitions across multiple workstations and multiple sites. Replicated Services Reduce Chances of Single Points of Failure The key benefit of Objectivity/HA is that it insulates Objectivity/DB applications from network or system failures. It distributes and relocates Objectivity/DB services so that each partition is self-sufficient in case a failure occurs in other partitions. Applications that are well-suited for Objectivity/HA have the following requirements: Require distributed data for fault tolerance from network failures. Require up-to-date data (cannot use stale reads). Frequently access data that is local to the partition. Occasionally access data that is in another remote partition. As shown in Figure 6-1, the replication and distribution services include locking for concurrency control, the data model (schema) for the federated database, and federated database catalog information. Concurrency control is performed by multiple lock servers, one for each partition. Schema information and catalog 49

50 Objectivity/DB High Availability Private and Public Partitions information are stored in special containers within a system database in each partition. Autonomous Partition A Autonomous Partition B Host 1 Host 2 Host 3 Host 4 Application Process Objectivity Kernel Data Server (AMS) Application Process Objectivity Kernel Data Server (AMS) Data Server (AMS) Application Process Objectivity Kernel Data Server (AMS) Lock Server Lock Server Network Figure 6-1 Partitioned Federated Database During normal operation, users within a partition can access objects transparently anywhere in the network or federation of databases. When a network failure occurs, a partition continues to operate autonomously with unimpeded access to information within the partition or to other partitions unaffected by the failure. The impact of the failure is limited to the partition in which it occurred. Private and Public Partitions You can create private and public autonomous partitions with Objectivity/HA. A private partition is one which you prevent remote access to even if it is physically accessible. This could be done for various reasons, including: The databases in that private autonomous partition are for archival purposes. 50 Objectivity Technical Overview

51 Objectivity/DB High Availability Runtime Setup and Configuration The private autonomous partition is for use by a particular workgroup or individual only. To make an autonomous partition private, you can configure it as though it is offline. Only applications that were started in such an offline partition have access to it. When you configure it online, it becomes accessible to all applications that have access to the federated database. In the case of an autonomous partition used for archiving databases, you would configure it online to move data or databases into that partition, and configure it offline when the moving was completed. You can configure a partition to be offline or online either by using an Objectivity/DB command-line tool, or by making an API call from within an application. By utilizing this feature of private and public partitions, you can have a single federated database within an organization with publicly accessible partitions as well as multiple private partitions for departments, projects or teams as required. Runtime Setup and Configuration Since Objectivity/HA operates transparently to Objectivity/DB applications, you can use Objectivity/HA administrative tools to create and change your partitioned environment at any time without modifying your application code. Objectivity/HA provides a rich set of tools and APIs for creating and configuring a partitioned federated database. These tools allow you to perform tasks such as creating, opening, modifying, and deleting partitions, as well as moving databases between partitions. You can also use Objectivity/DB s administration tools to administer your partioned federated database for example, to perform online incremental backup and restore. Replicating Databases With Objectivity/DB High Availability, you can replicate databases on any host anywhere in your network. Objectivity/HA improves the availability of critical data and enhances read performance when accessing this data. Using this product, you can: Create multiple copies of a database on a single machine or across a heterogeneous network of machines. Setting up full peer-to-peer copies of databases eliminates single point-of-failure scenarios. Other configurations can be designed to withstand multiple failure. Provide full read/write access to replicated databases. When one database image (copy) is updated, other images are automatically and transparently updated as well. Objectivity Technical Overview 51

52 Objectivity/DB High Availability Images Dynamically determine write privileges to replicated databases in failure situations based on a calculation of available databases (a quorum). Set up a tie-breaker for configurations where an even number of database images might be separated into two equal parts by a network failure. Provide your own function for determining whether a quorum exists if only one machine is available in a two-machine hot failover configuration. Applications that are well-suited for Objectivity/HA have the requirements and characteristics listed in Table 6-1: Table 6-1: Objectivity/HA Application Requirements Application Requirements High availability Application Characteristics Low to moderate update frequency. Stale reads are acceptable. Generally distributed. If not necessary, Objectivity/HA or redundant server solution may be better. High read performance Low update frequency. High read frequency. May or may not be distributed. When a client application reads data from a database that is replicated, Objectivity/DB reads the requested information from the image of the database in the boot partition of the application. Otherwise, the system reads a single image in a different partition. If one image of a particular database becomes unavailable due to a network or machine failure, a process can read data from a different image. The presence of multiple copies of a database permits multiple network or machine failures before databases become unavailable. Images When you create a database in an autonomous partition, you create a copy of a database file, or image, controlled by that autonomous partition. With Objectivity/HA, you can create additional images of that database in other partitions. Each image of a database belongs to a single partition, and each partition can have at most one image of any single database. Each image of a particular database contains all the data in the database. Data replication does not change the containment hierarchy, and it does not affect the logical model of your applications. Object identifiers (OIDs) for objects are also the same across images. 52 Objectivity Technical Overview

53 Objectivity/DB High Availability Quorum Not all images need to be available for applications to access a database. If a machine crashes or a network failure occurs, the database system will transparently substitute another image. To guarantee database consistency, however, a certain number of images (based on a quorum calculation) must be available in order for database access operations to succeed. Quorum Objectivity/HA performs a dynamic quorum calculation when an application accesses a database image. A quorum calculation is the count of database images reachable by the application. If the number of reachable images is greater than half of the total number of images, a quorum of the images exists. If an application can access a quorum of images, the application can continue reading and updating the database. Reading and updating an image is not permitted if an application cannot access a quorum of images. This guarantees that another application on the network cannot access another image at the same time. In some configurations, the number of images accessible to an application may not be sufficient to constitute a quorum. For these cases, you can assign weights to the images to ensure a quorum can be reached. See Weighted Scenario on page 56 for more information. By default, Objectivity/HA does not allow clients without a quorum of database images to read from or write to the non-quorum images. However, it does provide functions you can use to create applications that can read from the non-quorum potentially out-of-date images. Such reads are called stale reads. Automatic Resynchronization When one or more database image files become unavailable to the quorum, Objectivity/DB keeps track of the pages written to the quorum databases. Objectivity/DB later uses this information to update out-of-date non-quorum databases so that all images are resynchronized. Figure 6-2 is an example of a distributed, mixed-tier architecture that uses data replication. Objectivity Technical Overview 53

54 Objectivity/DB High Availability Objectivity/DB High Availability Scenarios Autonomous Partition A Autonomous Partition B Host 1 Host 2 Host 3 Host 4 Application Process Objectivity Kernel Data Server (AMS) Application Process Objectivity Kernel Data Server (AMS) Data Server (AMS) Application Process Objectivity Kernel Data Server (AMS) A B C C D B Lock Server Lock Server Network Figure 6-2 Replicated Databases B and C Objectivity/DB High Availability Scenarios Objectivity/HA can be used in many situations to improve availability and read performance. When a machine becomes unreachable to an application, it is generally not possible to determine exactly what caused the problem. It could be that the machine itself has crashed, or perhaps the machine is fine, but the Objectivity/DB database image or the partition s lock server is inaccessible or perhaps the network has gone down, either partially or completely. Further, just because a machine is inaccessible from one partition does not necessarily mean it is unavailable from another. Objectivity/HA allows you to configure your system to eliminate or reduce the downtime from these kinds of failure scenarios independent from the cause. 54 Objectivity Technical Overview