5 Database technology Trends. Guy Harrison, Executive Director, Information Management R&D

Transcription

1 5 Database technology Trends Guy Harrison, Executive Director, Information Management R&D

2 Introductions Web: guyharrison.net

3

4

5

6 But Seriously

7 5 Database Technology Trends 1. The end of one size fits all 2. Big Data and Hadoop 3. NoSQL 4. Columnar architectures 5. In-memory databases

8 Trend #1: The end of one size fits all 8

9 History of databases Pre-computer technologies: Printing press Dewey decimal system Punched cards Magnetic tape flat (sequential) files Magnetic Disk IDMS ADABAS System R Oracle V2 Access Postgres MySQL HBase Dynamo MongoDB Redis VoltDB Neo4J Relational Model defined IMS Network Model Hierarchical model Indexed-Sequential Access Mechanism (ISAM) SQL Server Sybase Informix Ingres DB2 dbase Aerospike Hana Riak Cassandra Vertica Hadoop

10 Why? 3 rd Platform drives new demands on the database: Global High Availability Data volumes Unstructured data Transaction rates Latency A single architecture cannot meet all those demands

11 It takes all sorts In-memory processing (Spark) Analytic/BI software (SAS, Tableau) Web Server Data Warehouse RDBMS (Oracle, Terradata ) In-memory Analytics (HANA, Exalytics ) Hadoop Web DBMS (MySQL, Mongo, Cassandra) Operational RDBMS (Oracle, SQL Server, ) ERP & inhouse CRM

12 Oracle engineered systems

13 Trend #2: Big Data and Hadoop 14

14 The 3-4 V s Value Volume Terabytes Petabytes Exabytes Zetabytes Variety Structured Unstructured Human Generated Machine Generated Velocity Transaction rates User populations Machines

15 The Industrial revolution of data

16 2005

17 2009

18 The instrumented human Compass Camera Mike/earphones Heads up display Emotion/Attention monitor Bluetooth Personal Area Network 3G/WiFi Wide Area Network GPS Storage Pulse, temp monitor Silent alarms Pedometer, sleep monitoring

19

20 The instrumented world

21 Big Data is the culmination of cloud, social and mobile

22 More Data Storing all data including machine generated and sol, Social, community, demographic data in original format for ever To More Effect Smarter use of data (data science) to achieve competitive or human benefit

23 More Data Storing all data including machine generated and sol, Social, community, demographic data in original format for ever To More Effect Smarter use of data (data science) to achieve competitive or human benefit

24 Pioneers of big data

25

26

27

28

29

30 Google Software Architecture (circa 2005) Google Applications Map Reduce BigTable Google File System (GFS)

31 Map Reduce Start Map Map Map Map Map Map Map Map Map Map Map Map Map Map Map Map Map Map Map Map Map Map Map Map Map Map Map Map Map Map Map Map Map Map Map Reduce

32 Hadoop: 1.0: Open Source Map-Reduce Stack

33 Hadoop at Yahoo 2010(biggest cluster): 4000 nodes 16PB disk 64 TB of RAM 32,000 Cores 2014: 16 Clusters 32,500 nodes

34

35 Hadoop family Oozie (Workflow manager) Hive (Query) Pig (Scripting) SQOOP (RDBMS loader) Flume (Log Loader) Map Reduce / YARN Hbase (database) Zookeeper (locking) Hadoop File System (HDFS)

36 Economies Exadata vs Hadoop $$/TB (Hardware only) Hadoop $750 Exadata $4,911 $0 $1,000 $2,000 $3,000 $4,000 $5,000 $6,000

37 Hadoop is the most concrete Big Data technology Toad: your companion in the Big Data revolution

38 More Data Storing all data including machine generated and sol, Social, community, demographic data in original format for ever To More Effect Smarter use of data (data science) to achieve competitive or human benefit

39 More Data Storing all data including machine generated and sol, Social, community, demographic data in original format for ever To More Effect Smarter use of data (data science) to achieve competitive or human benefit

40 Big Data Analytics AKA Data Science Machine Learning Programs that evolve with experience Collective Intelligence Programs that use inputs from crowds to simulate intelligence Predictive Analytics Programs that extrapolate from past to future

41

42 Collective Intelligence Siri call me an ambulance From now on, I ll call you An Ambulance. OK?

43 Data science 250 Predictive Analytics Classification Clustering Model training and deployment y = x

44 Trend #3: NoSQL

45 Web Servers Memcached Servers Database Servers Read Only Slaves Shard (A-F) Shard (G-O) Shard (P-Z)

46 CAP Theorem says something has to give CAP (Brewer s) Theorem says you can only have two out of three of Consistency, Partition Tolerance, Availability Partition Tolerance System stays up when network between nodes fail Consistency Everyone always sees the same data NO GO Availability System stays up when nodes fail Oracle RAC lives here Most NoSQL lives here

47 Major influences on non-relational Amazon Dynamo Eventually consistent transaction model Consistent hashing Google BigTable Column Family model for sparse distributed columnar data OODBMS and XML DBs Paved the way for the document database

48 Amazon Dynamo Model

49 BigTable Data Model NameId Name 1 Dick 2 Jane SiteId SiteName 1 Ebay 2 Google 3 Facebook 4 ILoveLarry.com 5 MadBillFans.com Name Site Counter Dick Ebay 507,018 Dick Google 690,414 Jane Google 716,426 Dick Facebook 723,649 Jane Facebook 643,261 Jane ILoveLarry.com 856,767 Dick MadBillFans.com 675,230 NameId SiteId Counter , , , , , , ,230 Id Name Ebay Google Facebook (other columns) MadBillFans.com 1 Dick 507, , , ,230 Id Name Google Facebook (other columns) ILoveLarry.com 2 Jane 716, , ,767

50 OODBMS -1990s The OODBMS Manifesto (Atkinson/Bancilhon/DeWitt/Dittrich/Maier/Zdo nik, '90) "A relational database is like a garage that forces you to take your car apart and store the pieces in little drawers Also SQL is ugly A Object database is like a closet which requires that you hang up your suit with tie, underwear, belt socks and shoes all attached (Dave Ensor) IPgd1Tg8ByE/UkOzHg1FmI/AAAAAAAACB0/QYg8kE Vp5_0/s1600/db4o_vs_orm.png

51 Revenge of the Object Nerds Document databases Structured documents XML and JSON (JavaScript Object Notation) become more prevalent within applications Web programmers start storing these in BLOBS in MySQL Emergence of XML and JSON databases

52 Memchache DB MongoDB Key Value Oracle NoSQL Voldemort JSON based CouchDB Dynamo DynamoDB Document RethinkDB Riak XML based MarkLogic BerkeleyDB XML Cassandra Hbase Neo4J Table Based BigTable HyperTable Graph Database Infinite Graph Accumulo FlockDB

53 It s not a database, it s a key value store

54 No Means Yes!

55 Trend #4: Column-oriented DB Dell - Restricted - Confidential

56 Row orientation vs column orientation Row oriented database ID Name DOB Salary Sales Expenses 1001 Dick 21/12/60 67, Jane 12/12/55 55, Robert 17/02/80 22, Dan 15/03/75 65, Steven 11/11/81 76, Block ID Name DOB Salary Sales Expenses Dick 21/12/60 67, Jane 12/12/55 55, Robert 17/02/80 22, Dan 15/03/75 65, Steven 11/11/81 76, Block 1 Dick Jane Robert Dan Steven 2 21/12/60 12/12/55 17/02/80 15/03/75 11/11/ ,000 55,000 22,000 65,200 76, Column oriented database

57 Analytical Queries Row oriented database SELECT SUM(salary) FROM saleperson Block ID Name DOB Salary Sales Expenses Dick 21/12/60 67, Jane 12/12/55 55, Robert 17/02/80 22, Dan 15/03/75 65, Steven 11/11/81 76, Block 1 Dick Jane Robert Dan Steven 2 21/12/60 12/12/55 17/02/80 15/03/75 11/11/ ,000 55,000 22,000 65,200 76, Column oriented database

58 Compression Row oriented database Poor compression ratio (low repetition) Block ID Name DOB Salary Sales Expenses Dick 21/12/60 67, Jane 12/12/55 55, Robert 17/02/80 22, Dan 15/03/75 65, Steven 11/11/81 76, Good compression ratio (high repetition) Block 1 Dick Jane Robert Dan Steven 2 21/12/60 12/12/55 17/02/80 15/03/75 11/11/ ,000 55,000 22,000 65,200 76, Column oriented database

59 Inserts Row oriented database INSERT INTO salesperson Block ID Name DOB Salary Sales Expenses Dick 21/12/60 67, Jane 12/12/55 55, Robert 17/02/80 22, Dan 15/03/75 65, Steven 11/11/81 76, Block 1 Dick Jane Robert Dan Steven 2 21/12/60 12/12/55 17/02/80 15/03/75 11/11/ ,000 55,000 22,000 65,200 76, Column oriented database

60 C-Store (Vertica) Solution for inserts Bulk sequential loads Merged Query Read Optimized Store Columnar Disk-based Highly Compressed Bulk loadable Asynchronous Tuple Mover Continual Parallel inserts Write Optimized Store Row oriented Uncompressed Single row inserts

61 Exadata Hybrid Columnar Compression (EHCC) Compression Unit (~<1M) Block (8K) Block Block Block Column 1 Column 2 Column 3 Column 4 Row Row Row

62 Exadata Hybrid Columnar Compression SELECT SUM(Column4) FROM table Provides high compression ratio Manageable impact on row read/write operations Some optimization of analytic queries

63 Trend #5: The End of Disk? 68

64 5MB HDD circa 1956

65 The more that things change...

66 Faster or slower? IO/CPU -390 CPU 1,013 IO/Capacity -630 Disk Capacity 1,635 IO Rate 260-1, ,000 1,500 2,000 %age change

67 Solid state disk to the rescue DDR RAM Drive SATA flash drive PCI flash drive SSD storage Server

68 Cheaper by the IO SSD DDR-RAM SSD PCI flash SSD SATA Flash Magnetic Disk 4, ,000 2,000 3,000 4,000 5,000 Seek time (us)

69 $$/GB $$/GB But not by the GB HDD MLC SDD SLC SSD

70 $/GB Tiered storage management Main Memory DDR SSD Flash SSD Fast Disk (SAS, RAID 0+1) $/IOP Slow Disk (SATA, RAID 5) Tape, Flat Files, Hadoop

71 Cost (US$/GB) Size (GB) In-Memory databases $100, Cost of RAM falling 50% each 18 months. $10, Some databases can fit entirely within the RAM of a single server or cluster of servers $1, $ US$/GB Size (GB) $ $ Year

72 Oracle Times Ten Clients In-memory transactional database Disk-based Checkpoints and disk-based logging By default, COMMITs are not durable (writes to the transaction log are asynchronous). Can configure synchronous replication or synchronous log writes to avoid data loss Columnar compression and analytic functions in the Exalytics version Memory Point in time snapshot Commits Checkpoints Transaction Logs

73 SAP Hana Memory Column store Persistence Layer Txn logs Row Store Savepoints Data files Delta store Note: Table must be either row or column not both

74 Exalytics Instantaneous!

75 You keep using that word. I do not think it means what you think it means

76 Exalytics Hardware: 2 TB RAM 4 10GBe, 2 InfiniBand ports 6x1.2TB SAS (7.2 TB) 3x800GB (2.4TB) SSD Software: Oracle BI ESSBase Oracle R Times-Ten 12c In-memory

77 VoltDB Clients Clients Clients Single threaded access to memory: no latch/mutex waits Transactions in selfcontained stored procedures: minimal locking K-Safety for COMMIT: No sync waits CPU CPU CPU CPU CPU CPU In-memory Partition In-memory Partition In-memory Partition In-memory Partition In-memory Partition In-memory Partition

78 Spark (sort of) in-memory Hadoop In Memory compute Spark Streaming Mlib Machine Learning SparkSQL HDFS compatible Libraries for data processing, machine learning, streaming, SQL, etc Spark: in-memory distributed compute Python and Scala interfaces Part of the Berkeley Data Analytic Stack HDFS Tachyon in memory File system Mesos Cluster manager

79 Oracle 12c in-memory database Column store Memory (SGA) Row store Column Store (IMCU) OLTP Analytics (SMU) Redo Logs Data files

80 What does all this mean for me?

81 Trend #6: shameless product plugs will increase over the next 120 seconds 89

82 Toad: your companion in the Big Data revolution

83 Toad for Hadoop

84 SharePlex for Hadoop JMS Queue Hadoop Poster HBase Real Time replication Change Data Capture Redo-logs Batched HDFS File Copy Audit / Change Data

85 Toad BI Suite join and analyse data from any source

86 Dell Statistica

87 Dell In-Memory Appliances for Cloudera Enterprise Starter Configuration 8 Node Cluster R720-4 Infrastructure Nodes R720XD- 4 Data Nodes Force10- S55 ~176TB (disk raw space) ~1.5TB (raw memory) Mid-Size Configuration 16 Node Cluster R720-4 Infrastructure Nodes R720XD- 12 Data Nodes Force10- S4810P Force10- S55 ~528TB (disk raw space) ~4.5 TB (raw memory) Small Enterprise Configuration 24 Node Cluster R720-4 Infrastructure Nodes R720XD- 20 Data Nodes ~880TB (disk raw space) ~7.5 TB (raw memory) Expansion Unit- R720XD-4 Data, Cloudera Enterprise Data Hub, Scale in Blocks

88 Dell appliances for any database Dell provides appliances and reference architectures specifically designed for: Oracle SQL Server HANA SSD database acceleration Large memory footprints

89 Big Data for the rest of us Success in Big Data requires capabilities at multiple technology levels: hardware, software infrastructure, business intelligence and analytics Only Dell can deliver capabilities at every technology layer Only Dell s solutions are designed and priced to suit mid-market initial deployments and to scale to the largest enterprise Advanced Analytics Business Intelligence Data Integration Systems Management Hadoop and database software Server and Storage Toad Data point Boomi Statistica Boomi, Toad Intelligence Central Dell Foglight and TOAD Dell appliances for Hadoop, Oracle, etc Dell servers and storage arrays

90 Thank you.