Using Hadoop, Cloud and Tiered Storage For Peak Performance

Transcription

1 Using Hadoop, Cloud and Tiered Storage For Peak Performance Presented by: David Gorbet, Vice President, Engineering, MarkLogic Corporation

2 AGILITY SLIDE: 2

3 Local Disk SAN NAS SLIDE: 3

4 TIERED STORAGE ELASTICITY SLIDE: 4

5 TIERED STORAGE SLIDE: 5

6 With Tiered Storage You Can Provide multiple Service Level Agreements (SLAs) in a single system Decrease time and costs of ETL to bring offline content back online Empower your operations team without imposing burdens on your developers SLIDE: 6

7 Example: Investment Bank SLIDE: 7

8 Total Size (TB) 96 Operational 504 Compliance 1,044 Analytic SLIDE: 8

9 Total Size (TB) ,044 Total Cost ($000) $592 $2,066 $2,080 Effective Unit Cost ($/GB) $25 $4 $1.50 Operational Compliance Analytic SLIDE: 9

10 Tiered Storage Here s how you enable tiered storage Define data tiers based on a range index Have content balanced into forests by tier Move an entire tier to different storage Query one tier or the other tier or both at once! All with no downtime, and 100% consistency! SLIDE: 10

11 OPTIMIZE STORAGE $ SAVE MONEY Put data in the economically right place and decrease ETL costs ENFORCE DATA GOVERNANCE Maintain security, compliance over lifetime of data ALWAYS-ON DATA Keep data available and easily accessible for applications or reporting SLIDE: 11

12 ELASTICITY SLIDE: 12

13 With Elasticity You Can Know when to scale How much to scale Programmatically expand and contract On premise or in the cloud SLIDE: 13

14 Example: MarkMail.org SLIDE: 14

15 Add 18 more nodes (:15) SLIDE: 15

16 Add 18 more nodes (:15) Migrate content across the 24 nodes (:45) SLIDE: 16

17 Add 18 more nodes (:15) Migrate content across the 24 nodes (:45) Enrich the data (5:00) SLIDE: 17

18 Total time (9:00) Add 18 more nodes (:15) Migrate content across the 24 nodes (:45) Enrich the data (5:00) Collapse back to 6 nodes (3:00) SLIDE: 18

19 Elasticity Scale up and down with Tools to understand in detail how your cluster is performing, and to find bottlenecks Fine-grained tuning parameters for optimization of indexes, cache sizes, etc. Cloud orchestration APIs to expand and contract clusters programmatically on-prem or in the cloud Continuous, online rebalancing of content across nodes in a cluster to keep performance optimal for your cluster size SLIDE: 19

20 ELASTICITY HELPS YOU KNOW YOUR SYSTEM Track usage, identify hot spots, support chargebacks HARNESS THE CLOUD Public or private, shared or dedicated resources run the same application anywhere BE MORE PROACTIVE Manage to SLAs, in a programmatic way SLIDE: 20

21 Demo Expand cluster to six nodes Rebalance content across the cluster See the effects in Monitoring History Split data into tiers and work with a tier independently Consolidate back to a single node, and archive to S3 SLIDE: 21

22 DEMO SLIDE: 22

23 TIERED STORAGE ELASTICITY SLIDE: 23

24 HADOOP SLIDE: 24

25 Oozie Scheduler Pig Scripting Mahout Machine Learning R Connectors Statistics Hive SQL Query ZooKeeper Coordination MapReduce Distributed Processing Framework HDFS Hadoop Distributed File System SLIDE: 25

26 Oozie Scheduler Pig Scripting Mahout Machine Learning R Connectors Statistics Hive SQL Query MarkLogic Real-time Database ZooKeeper Coordination MapReduce Distributed Processing Framework HDFS HDFS Hadoop Hadoop Distributed Distributed File System File System SLIDE: 26

27 Local Disk HDFS SLIDE: 27

28 DO MORE WITH HADOOP SECURE Minimize duplication, costly ETL, reduce risk REAL-TIME Enterprise-class database for real-time search, delivery & analytics RUN APPLICATIONS Run mission critical applications directly on HDFS SLIDE: 28

29 TIERED STORAGE ELASTICITY HADOOP SLIDE: 29

30 Enterprise NoSQL: Powerful, Agile & Trusted Powerful Deliver more value, build more powerful applications Agile Prepare for and respond to change quickly Trusted Enterprise-ready and secure for mission-critical apps Semantics: RDF & SPARQL Flexible Indexes Schema- Agnostic Elastic ACID Transactions XA Distributed Transactions Triple Index JSON Storage Tiered Storage Cloud Ready Common Criteria Security Certification Role-based Security & LDAP Support Full Text Search Alerting & Event Processing mlcp Content Pump Programmatic Controls & Metering Configuration Management Monitoring & Management Scalable Geospatial Query HDFS & Amazon S3 Storage Application Builder Automated Failover Backup/ Restore REST & Java APIs In-database MapReduce Hadoop Connector SQL Support Point-intime Recovery Replication Analytic Functions Visualization Widgets Information Studio BI Integration Journal Archiving Database Rollback SLIDE: 30