Application-Tier In-Memory Analytics Best Practices and Use Cases

Transcription

1

2 Application-Tier In-Memory Analytics Best Practices and Use Cases Susan Cheung Vice President Product Management Oracle, Server Technologies Oct 01, 2014 Guest Speaker: Kiran Tailor Senior Oracle DBA and Architect Chartered Institute of Management Accountants

3 Safe Harbor Statement The following is intended to outline our general product direction. It is intended for information purposes only, and may not be incorporated into any contract. It is not a commitment to deliver any material, code, or functionality, and should not be relied upon in making purchasing decisions. The development, release, and timing of any features or functionality described for Oracle s products remains at the sole discretion of Oracle.

4 Program Agenda TimesTen In-Memory Database Overview TimesTen for In-Memory Analytics Common Use Cases and Best Practices Real-life Customer Use Case at CIMA Product roadmap Q&A

5 Oracle TimesTen In-Memory Database Microseconds Response Time in Application Tier Relational Database Pure in-memory ACID compliant Standard SQL Extremely Fast Microseconds response time Very high throughput Persistent and Recoverable Database and Transaction logs persisted on disk and flash storage Compatible with Oracle Database Data types, PL/SQL, OCI, ODP.NET, PHP, R Integrated with RAC, Data Guard, Enterprise Manager, SQL Developer, etc. 5

6 TimesTen In-Memory Database Persistence and Recovery TimesTen database persistence can be configured on Flash, SSD, hard disk storage (HDD) TimesTen Checkpoint Files TimesTen Transaction Log Files All transactions are logged and persisted Redo, undo, and recovery Dual database Checkpoint files Database restart Database recovery after failures Separate storage for Transaction Logs and Checkpoint files if using HDD 6

7 Application Development with TimesTen SQL PL/SQL Oracle R ODP.NET ODBC OCI, Pro*C ttclasses Languages C/C++/C# Java/JDBC J2EE App Servers OR Mapping

8 TimesTen In-Memory Database Low Latency - Microseconds Response Time TPTBM Read and Update E GHz 1 socket, 15 cores/socket, 2 threads/core TimesTen (100M rows, 17GB) Microseconds Response Time Millionths of a Second 3.58 SELECT Query Millionths of a Second 9.07 UPDATE Transaction

9 TimesTen In-Memory Database Read Scalability 25 Million Queries per Second 30,000, Million Queries per Second TPTBM 100% Read E GHz 4 sockets, 15 cores/socket, 2 threads/core TimesTen (100M rows, 17GB) Queries Per Second 25,000,000 20,000,000 15,000,000 10,000,000 5,000, ,192, , Concurrent Processes 9

10 TimesTen In-Memory Database 1.3 Million Update Transactions Per Second Per Processor 1,400,000 Update Throughput Per Processor 1.3 Million Update TPS TPTBM 100% Update E GHz 1 socket, 15 cores/socket, 2 threads/core TimesTen (100M rows, 17GB) Update Transactions Per Second 1,200,000 1,000, , , , , Concurrent Update Processes 10

11 TimesTen In-Memory Database 3.4 Million Transactions Per Second Per Processor 3,500,000 Mixed Workload Throughput Per Processor 3.4 Million Transactions Per Second TPTBM 100% Mixed Workload ( ) E GHz 1 socket, 15 cores/socket, 2 threads/core TimesTen (100M rows, 17GB) Transactions Per Second 3,000,000 2,500,000 2,000,000 1,500,000 1,000, , Concurrent Processes Workload = 80% select, 10% updates, 5% inserts, 5% deletes 11

12 Oracle TimesTen Pure In-Memory Relational Database Very Fast Response Time for Very High Throughput in Application Tier Pre-Oracle acquisition 1998 First commercial In-Memory RDBMS HA Replication Online Upgrades Application-tier Cache for Oracle Database TimesTen 6 TimesTen 11g TimesTen 11g x TimesTen Enhancements Oracle RAC integration National Language Support Oracle Data Types support SQL Developer Integration Enterprise Manager integration PL/SQL and OCI Support Oracle Clusterware Integration Cache Grid for Scale Out ODP.NET Support BLOB, CLOB, NCLOB data types Parallel Replication In-Memory Analytics Columnar Compression Index Advisor Oracle R Support In-Memory Star Join Oracle Golden Gate Integration Parallel data import from Oracle Database Parallel database restart Highly concurrent range indexes

13 Analytics Use Cases for Oracle TimesTen

14 In-Memory Analytics - Summary Aggregates Scenarios Aggregations are suitable for analysis of higher-level grains of FACT data Knowledge of query patterns required Aggregated tables and indexes typically much smaller than detail-tables Reports using summary aggregates typically provide exceptional response time (sub-second) Operational Data Store 1 2 Reports Using Aggregate Set 1 OLTP Data Warehouse Summary Aggregate Tables and Indexes stored in TimesTen Reports Using Aggregate Set 2

15 In-Memory Analytics - Data Mart Use Cases Subset of Data from a Data Warehouse Data mart and operational data stores For analysis where summary aggregates are not sufficient Require access to detail source tables (fact and dimension tables) Consider hot set of data* Data volume constrained by availability of RAM in the system May use compression to include more data Common use cases have both aggregations and detail tables If the entire data warehouse is too big to fit into memory, only load portion of the data to TimesTen

16 In-Memory Analytics ODS and Data Mart Detail Tables Scenarios Operational Reports Operational Data Store OLTP Data Warehouse Aggregates + Detail tables + indexes Reports with Aggregates AND Detail Tables

17 Best Practices Tips and Tricks

18 TimesTen In-Memory Database Storage Speed Matters to Database Restart Time Storage speed matters Faster read operations using Flash and Solid State Disks Higher throughput with parallel threads Significant time reduction for database load to memory TimesTen Checkpoint Files 18

19 Dual Parallelism Oracle TimesTen and Intel NVMe P3700 Speed up Database Load Time to Memory 100GB Database Load to Memory Time in Seconds Database Load Time HDD 15K RPM Sata SSD Intel NVMe P3700 TimesTen on Oracle Linux, Intel Xeon E Ghz, 2 processors 20 core server 19

20 Dual Parallelism Oracle TimesTen and Intel NVMe P3700 Multiple Flash Cards Lower Load Time and Higher Throughput GB Database Load to Memory Time in Seconds 42 Single NVMe P Striped Two P GB Database Load to Memory Read Rate Single NVMe P3700 Striped Two P3700 Cards 2.2 GB/s 3.3 GB/s TimesTen on Oracle Linux, Intel Xeon E v2 2.8 Ghz, Two 15-core Processors 20

21 TimesTen Cost-Based SQL Optimizer Proper Indexes and Up-to-date Statistics Optimized Query Execution Plans Hash indexes Best performance for equality matches Must be properly sized for good performance Undersized hash indexes can result in severe performance penalties Range Indexes Range indexes best for range search Can ALTER Unique PK range indexes Default index type for primary key indexes to hash indexes Up-to-date table and column statistics Important for a cost-based SQL optimizer

22 Summary TimesTen is an excellent choice for in-memory analytics in the application tier Correct setup and usage is vital to achieve good performance and easy management OS, configuration, operations, persistence storage, tuning, etc. Use the tools available to make your life easier and get the best results Many more best practices available Visit TimesTen product center on Oracle Technology Network Refer to TimesTen documentation for more information (available online)

23 Most Widely Used Relational In-Memory Database Deployed by Thousands of Companies

24 Charter Institute of Management Accountants (CIMA) Real-life Use Case of In-Memory Analytics Kiran Tailor Senior Oracle DBA and Architect Chartered Institute of Management Accountants

25 CIMA, Exalytics and TimesTen Kiran Tailor Senior Oracle DBA and Architect The Chartered Institute of Management Accountants Date: 01/10/2014

26 About Me Kiran Tailor, Senior Oracle DBA/Architect Sixteen years working with Oracle Started working with Oracle Forms, moved to databases Oracle Databases for 15 years OBIEE for more than five years CIMA CIMA,Exalytics,TimesTen 01/10/2014 2

27 CIMA Our mission is to help people and businesses to succeed in the public and private sectors We have more than 218,000 members and students in 177 countries They work at the heart of business in industry, commerce and not for profit organisations We have 28 offices around the world, employing 450 people We constantly update our qualification. This ensures that our members and students are first choice for employers who are recruiting financially trained business leaders In 2012, together with the AICPA, we launched the CGMA designation CIMA CIMA,Exalytics,TimesTen 01/10/2014 3

28 Our Challenges Meaningful information from the systems, that we could action upon instantly CIMA CIMA,Exalytics,TimesTen 01/10/2014 4

29 Our Challenges As systems were slow. Staff/Business managers constantly phoning for information CIMA CIMA,Exalytics,TimesTen 01/10/2014 5

30 Our Challenges Marketing Campaign Reports 8 minutes Exam Reports 2 minutes individually, thousands to do Anything with Activities 15 minutes+ CIMA CIMA,Exalytics,TimesTen 01/10/2014 6

31 CIMA Before Exalytics DW OLTP Siebel CRM CIMA CIMA,Exalytics,TimesTen 01/10/2014 7

32 Exalytics/TimesTen common Setup DW OLTP Siebel CRM CIMA CIMA,Exalytics,TimesTen 01/10/2014 8

33 TimesTen CIMA Set Up Siebel DW CIMA CIMA,Exalytics,TimesTen 01/10/ Siebel CRM

34 TimesTen DAC CIMA Set Up How DAC identifies records in TimesTen CIMA CIMA,Exalytics,TimesTen 10

35 TimesTen DAC CIMA Set Up How DAC identifies records that need to be copied to TimesTen CIMA CIMA,Exalytics,TimesTen 01/10/2014

36 TimesTen DAC CIMA Set Up How DAC identifies records in TimesTen at task level CIMA CIMA,Exalytics,TimesTen 01/10/

37 TimesTen DAC CIMA Set Up How DAC identifies which records have changed in the warehouse that need to be copied to TimesTen CIMA CIMA,Exalytics,TimesTen Siebel CRM 01/10/

38 Outcome Full warehouse approx 300gb loaded into memory 100x + performance improvement, in some cases much higher Full load took approx 3 hours Incremental nightly copy approximately 6 minutes Achieved huge performance uplift with zero application code changes Enabled previously unachievable analysis, leading to greater knowledge of customers and challenges across the company 100% In-memory Daily load time 6 mins ipad apps roll-out ongoing Zero Application code changes Performance Improvement INTELLIGENCE AND BUSINESS RESULTS CIMA CIMA,Exalytics,TimesTen

39 Feedback It s all a lot better. It s so quick now it s difficult to measure without an atomic stopwatch. UK User I didn t used to use the old dashboards to the extent that I do now- maybe that s partly because I used to get so fed up with the speed that I gave up whereas now it s there at the press of a button UK User Report UK - WAN connection on standard Analytics seconds Outside of UK - Wifi connection on Analytics not possible due to error Outside of UK - Wifi connection on Times Ten In Memory DW 1.1 second via webex CIMA CIMA,Exalytics,TimesTen

40 Deployment Tips In RPD make sure you use correct TimesTen ODBC settings We used DAC to create the table and index scripts INITIAL load: run the table scripts on the TimesTen Server, load the data using the DAC, run the create index scripts on the TimesTen server, update the stats on TimesTen OBIEE Caching: Linux Ram Disk cache into memory! If using Exalytics and LDAP for OBIEE, follow the weblogic admin guide for configuration Keep checking support for Patch updates If any columns are being used to do updates make sure they are indexed. Update the stats constantly especially if you are updating and inserting rows CIMA CIMA,Exalytics,TimesTen 01/10/

41 Dziękujemy! Thank you! Спасибо! 謝謝! ﺷﻛرا ﻟك Kiran Tailor

42 Product Roadmap

43 TimesTen 12c Roadmap In-Memory Database Technology in Application Tier In-memory columnar storage Row-based and Columnar co-exist in the same TimesTen database Entire database is in memory (just like today) TimesTen Grid scale-out architecture Single image Distributed database, transactional and persistent Transparent data distribution and parallel query processing Elastic scalability and availability

44 TimesTen In-Memory Columnar Everything In Memory Row-based Storage plus Columnar Storage plus Process Billion rows per second Columnar and row-based tables co-exist User choose preferred format at table level Store more data in memory Row-based tables No Indexes required Lower administration cost Columnar Co-exist in same TimesTen in-memory database Offload analytics processing Data and Operational isolations from production data warehouse Complementary to Oracle Database In-Memory option

45 TimesTen In-Memory Grid Database Architecture Distributed Scale Out Always Available Fault Tolerant Distributed In-Memory RDBMS Single database image across all hosts Application transparent data distribution Connect to any instance and access to ALL data on the entire Grid Parallel query processing High availability (k-safety, k >= 2) Fault-tolerant Scale-out to hundreds of machines App App App App Single Image IMDB Instance 1 Instance 2 TimesTen Grid Instance N Instance 3 Instance 5 Instance 4

46 Q&A

47

48