Distributed Architecture of Oracle Database In-memory Niloy Mukherjee, Shasank Chavan, Maria Colgan, Dinesh Das, Mike Gleeson, Sanket Hase, Allison Holloway, Hui Jin, Jesse Kamp, Kartik Kulkarni, Tirthankar Lahiri, Juan Loaiza, Neil Macnaughton, Vineet Marwah, Atrayee Mullick, Andy Witkowski, Jiaqi Yan, Mohamed Zait Håkon Åmdal Suresh Kumar Mukhiya 1/25
Motivation Ad-hoc real-time analysis (OLAP) High performance Large amount of data while still keeping the system suitable for transactional workloads (OLTP) All without explicit optimizer plan changes or query rewrites 2/25
Oracle Database In-Memory Dual Format Row storage for online transactional processing (OLTP) Column storage for online analytical processing (OLAP) Database objects optimized for memory, while still being persisted on disk. Memory is cheap From cache on disk-access to primary storage. 3/25
Need for a distributed architecture Scaling out Scaling up Scaling out allows for elastic expansion Avoid single point of failure Hard to program? Majority of analytics do not process huge datasets at the same time Cheap hardware on a single server can process 90% of Facebook s jobs. Main memory bus might become bottleneck Simpler implementation? Single point of failure Long recovery period We are motivated by these observations to design an extremely scalable, high-available fault-tolerant distributed architecture within the Oracle Database In-Memory Option 4/25
Distributed Oracle DBIM Oracle Real Application Cluster (RAC) allows for scaling ut. across multiple machines. Shared nothing -architecture Persisted in row-based blocks 5/25
In-Memory compression units (IMCUs) Compressed with user-defined compression levels Optimized for OLTP Optimized for OLAP Optimized for Storage High performance Single Input Multiple Data (SIMD) instructions In-Memory Storage indexes Bloom filter based joins 6/25
Shared Database Buffer Cache Global Cache Service (GCS) tracks and maintains locations and access modes of all data blocks in the global cache. Handle all OLTP operations Guarantees strict ACID and robustness properties Cache Fusion Protocol 7/25
In-memory column store Shared-nothing container of in-memory segments on each instance Distributed together with the underlying row blocks Falls back to disk storage if not present in-memory 8/25
In-memory Transaction manager Each server is responsible for transactional consistency for incoming DML statements When looking up the transactional log causes too much overhead, the IMCUs are rebuilt 9/25
Distribution manager a. extremely scalable application-transparent distribution of IMCUs across a RAC cluster allowing for efficient utilization of collective memory across inmemory column stores b. high availability of IMCUs across the cluster guaranteeing in-memory faulttolerance for queries c. application-transparent distribution of IMCUs across NUMA nodes within a single server to improve vertical scale-up performance d. efficient recovery against instance failures by guaranteeing minimal rebalancing of IMCUs on cluster topology changes e. seamless interaction with Oracle s SQL execution engine ensuring affinitized high performance parallel scan execution at local memory bandwidths, without explicit optimizer plan changes. 10/25
Distribution scheme Partition Subpartition Block range Auto 11/25
Distribution mechanism Global phase for distribution consensus Decentralized population phase Each instance comes up with the same object location using a hashing algorithm Spread equally across a NUMA node 12/25
Redistribution If a server goes down, a new distribution is calculated Only the objects that has been redistributed is moved, the other ones stay the same. 13/25
Availability None 1-safe (N-1)-safe 14/25
Distributed SQL Execution 15/25
Uniqueness of Architecture Paper Compares uniqueness with: SAP HANA and IBM DB2 with BLU with respect to Distribution Scalability Availability Recovery The architecture provides complete scale out solution with collective memory utilization, redundancy, availability and efficient failure handling by redistribution. 16/25
Evaluation Hardware Setup Distribution Experiments Distributed Query Execution In-Memory Distribution Awareness In-Memory fault Tolerance NUMA Aware Query Execution Evaluation to Validate Quality Attributes like: Performance Scalability Availability 17/25
Evaluation- Hardware Setup Conducted on Oracle Exadata Database Machine, a state of the art database SMP Server and storage cluster system NUMA Experiment is conducted on an X4-8 single node machine with 8 15-core Intel Xeon processor and 2TB DRAM Rest of the experiments are conducted on X4-2 RAC Configuration comprising up to 8 database server nodes, each with 2 12-core Intel Xeon processor and 256GB DRAM and 14 shared storage servers amounting to 200TB total storage capacity 18/25
Evaluation- Distribution Experiments To verify whether IMCU throughput scales out with the number of database server instances in the RAC Cluster. Two experiments were performed: Non-partitioned Table Distribution 13-column and 1 billion row nonpartitioned atomic table with size of 64 GB Composite-partitioned Table Distribution TPC-H lineitem schema is chosen for this experiment. 19/25
Evaluation- Distributed Query Execution - set of 3 experiments performed in 13- column 64 gb atomics table Table is auto-distributed based on block ranges without redundancy Four sets of query sets are selected for each of these experiments 20/25
Evaluation- Distributed Query Execution Q1, Q2, Q3 nonlinear scale out where queries are CPU-Bound Queries in set 4 exercise in-memory storages. Throughput of such queries is not expected to scale with number of instances. 21/25
Evaluation - In Memory Distribution Awareness To observe and validate impact of in-memory distribution awareness in execution of cluster-wide analytic query performance. Performance gains in orders of 20x to 40x over executions with distribution awareness disabled in parallel query granule generation phase. 22/25
Evaluation - In Memory Fault Tolerance To validate in-memory fault tolerance of distributed query execution under 1-safe redundancy. Single instance failure has no visible effect on the elapsed times of the queries. 23/25
Evaluation - NUMA Aware query Execution To observe weather IMCU NUMA-affined query execution yields better throughput compared to the same inmemory execution but without NUMA awareness. 150-250% improvements in query elapsed times when execution in IMCU NUMAaware. 24/25
Conclusion This paper summarizes: Motivation for development of in-memory database optimized for OLTAP environment Fault-tolerant distributed architecture of Oracle database in-memory option Uniqueness of the architecture among its peers Evaluation with sets of experiments showing how performance can be enhanced with in-memory option 25/25