<Insert Picture Here> Big Data for the Enterprise

Transcription

1 <Insert Picture Here> Big Data for the Enterprise David Rajan, Director Technology

2 It s the Market Conversation Big Data features on Radio 4 Today Programme 2

3 Big Data Appliance was GA in January IDC "Oracle's comprehensiveness in approaching both the big data landscape overall and how the products fit together represents a formidable challenge to any vendor hoping to offer end to end business oriented Big Data solutions" Ovum Positive note that applauds Oracle Big Data Appliance on: (1) using proven, open CDH (2) price (3) appliance approach vs build your own WinterCorp Oracle Big Data Appliance in connection with Orac le Exadata Database Machine and Oracle Exalytics In-Memory Machine offers a new route for Oracle customers to leverage the business opportunities associated with Big Data WinterCorp recommends that Oracle customers with an interest in big data take a look at Oracle Big Data Appliance. Source: IDC Oracle's All-Out Assault on the Big Data Market: Offering Hadoop, R, Cubes, and Scalable IMDB in Familiar Packages, February 2012 Source: Ovum Oracle Mainstreams its Hadoop Platform with Cloudera OEM Deal, January 2012 Source: WinterCorp Big Data: Business Opportunities, Requirements and Oracle s Approach 3

4 What is Big Data? <Insert Picture Here>

5 Buzz Word Bingo Hadoop MapReduce Cloudera Unstructured Cassandra Mahout MapR NoSQL Semi-Structured HBase R Pig 10gen HortonWorks Sqoop Hive 5

6 Internet of Things Creates a Digital Exhaust CERN 40TB/s IDC 1,200 EB generated in 2008 All catalogued books in the US Library of Congress total 15 TB Total amount of information in existence = 1.2 ZB Enterprises responsible 85% of this 95% is unstructured weblogs, social data feeds, machine generated or sensor data (text, numbers, thermostat, RFID, sound, vibration, image detection) audio and voice (VoIP), video, TV, news feeds, radio, scanners, , chat, GPS and telematics, satellite, badge swipes, weather, flood data, stock tickers, SMS messages, network data or other in-flight events Source: The Economist The data deluge

7 101 Introduction and Jargon Buster What is unstructured data? Weblogs, social data feeds, machine generated or sensor data (text, numbers, thermostat, RFID, sound, vibration, image detection), audio and voice (VoIP), video, images, TV, news feeds, radio, scanners, , chat, GPS and telematics, satellite, badge swipes, weather, flood data, stock tickers, SMS messages, network data or other high volume in-flight events Where did Big Data come from? Google wrote three papers on large scale data analytics between 2003 and They espoused the approach Google was taking with its own architecture. This stimulated an Open Source movement to develop many NoSQL variant databases and also a Yahoo employee, called Doug Cutting, who developed the first Hadoop Library for search indexing. Big social websites facing a Petabyte storage challenge, especially Facebook, embraced and pioneered these technologies. It s all about unstructured data... ish Bunch small NoSQL firms and some Hadoop distributions Primarily Cloudera It s came out of the Google, Yahoo, FB... big web properties who had a Petabyte issue Whose behind it? NoSQL Databases are open source and managed by many small independent companies such as 10gen who have MongoDB and DataStax or Acunu who distribute Cassandra (Developed at Facebook). Hadoop is also Open Source and is managed as a top level Apache project. Three companies provide distributions adding key functionality such as manageability Cloudera, HortonWorks (Yahoo spin-out) and MapR. What is NoSQL and Oracle s relationship with it? NoSQL databases are simple data stores and come in four main variants which are used for different purposes Key Value, Columnar, Graph and Document. They are extremely developer centric with limited functionality leaving the developer to understand, manage and build such things as relationships and consistency. They are mostly eventually consistent meaning you may not always read the latest update to a piece of data with the write eventually propagating through the system. Oracle has built its own Oracle NoSQL Database based on the Oracle Berkeley DB which is also the foundation for other NoSQL products such as Voldermort and GenieDB. The Oracle approach to NoSQL is to bring key enterprise features and capabilities required by our enterprise customers. Key Technology #1 NoSQL Databases which are very developer centric and simple What is Hadoop and Oracle s relationship with it? Hadoop is a Java framework that contains many components. This framework allows many low cost Key Technology #2 Hadoop Java Framework for Massively Parallel Processing i.e. 1000s low cost blades processing data Key Technology #3 R commodity rack mounted blade servers using local (not expensive SAN or NFS storage) internal disk to be networked together into extremely large clusters of 100s or thousands of machines (Yahoo has over 4500 in a single cluster). Bound together in this manner has the potential to deliver pools of Open Source Analytics compute with thousands of cores and petabytes of storage. Data is stored locally on individual nodes with processing passed locally to the data. That processing is managed by another component of Hadoop called Map Reduce. Other technologies inside the Hadoop framework include Hive a data warehouse and Hbase a NoSQL Key Value database. Plus others. Oracle has selected to partner and distribute the Cloudera distribution of Hadoop. 7

8 Definition the Vs of Big Data? Volume It s about 100s TB and Petabytes... BUT could be smaller volumes size is in the eye of beholder Velocity Relates to streams of high frequency data arriving e.g. From a sensor, network or high volume event Variety Highlights the new types of data from the internet of things that can be stored and analysed Value At volume low value data can highlight high value patterns, trends and insights of significant commercial value 8

9 <Insert Picture Here> The Technology Basics 9

10 Hadoop Hive Pig HDFS Data Analysis SQL Map Reduce Sqoop Avro Serialisation Database imp/exp Map Reduce HBase Zoo Keeper Coordination Columnar Database A distributed file system Schedules and parallelises Jobs HDFS Distributed File System 10

11 NoSQL Database Broad class of non-relational databases Avoid joins; don t require structured schema Relaxed consistency; Typically not ACID Focus on throughput (Read and Write) Short discrete operations Typically no ad hoc query capability Fault tolerent and designed to scale across hundreds of servers Variable feature set with specialisations e.g. keyvalue, document, graph and columnar 11

12 R and the Rise of the Data Scientist The future belongs to the companies and people that turn data into products O Reilly Radar Report What is Data Science? Coder Maths R is a language and environment for statistical computing and graphics Statistician Creative Big Picture

13 BI And Data Discovery Are Peers Drives need for Business Intelligence Proven Answers To Known Questions Data Discovery Fast Answers To New Questions Creates new KPIs for 13

14 <Insert Picture Here> Why Does Big Data Matter? 14

15 Market Opportunity paradigm shift around unstructured data is expected to grow from $3.2 b in 2010 to $16.9 b in A compound annual growth rate (CAGR) of 40% or about 7 times that of the overall ICT market. Source: IDC Worldwide Big Data Technology and Services Market Forecast 7/3/2012 Source: * McKinsey Global Institute: Big Data The next frontier for innovation, competition and productivity (May 2011) 15

16 Why does this matter to you? Creating competitive advantage from the unstructured data asset Opportunity grow the business New data = Strategic asset Do we have an unstructured data asset? Does this have value for us? New insight = Competitive advantage Identify top line competitive advantage New products and services Fear factor, our competitors are so we should too New technologies = Key enabler Need understand new technologies Need to experiment 16

17 Is advantage about our mindset to data? ,000 ounces $ ,000 ounces $59 20% CAGR since 2004 data will be the new oil a valuable resource of the 21 st Century Source: World Economic Forum Personal Data: The Emergence of a New Asset Class January

18 What s the need? Business Use Cases Financial Services Compliance and regulatory reporting Risk analysis and management Fraud detection and security analytics CRM and customer loyalty programs Credit risk, scoring and analysis High speed Arbitrage trading Trade surveillance Abnormal trading pattern analysis Rogue Trader Detection Customer Rating in Insurance NoSQL Government Fraud detection Threat detection Cyber security Relational Retail CRM and Customer Loyalty Customer Interaction and Experience Marketing campaign and sales program optimisation Web clickstream./ Weblog Analysis Sentiment Analysis Brand management Hadoop Telco / Communication Network performance analysis Revenue assurance and price optimization Customer churn prevention Campaign management and customer loyalty Call Detail Record (CDR) analysis Network performance and optimization Mobile User Location analysis Healthcare Health insurance fraud detection Campaign and sales program Patient care quality and program analysis Supply-chain management Drug discovery and development analysis Other Early Warning System from sensor data capture Image and face analysis search and detection. Batch ETL of unstructured data into a Warehouse 18

19 Real World Illustration Website behaviour / Web Logs Which customers might buy Product X? Big Data Hadoop / NoSQL System Processes weblogs 1TB / Day from existing OLTP Transaction Systems 2/3 Content 1/3 Content Sales Inventory Payments Campaigns 150 TB Warehouse

20 Real World Illustrations* Struggle translate store success to web 1/5 th Amazon. Social network analysis - what customers saying about products/wants Search engine keyword analysis Inter-store swing 300m Kosmix - WalmartLabs Correlate information from different businesses -Theme park - Resort hotels - Cable channel - Website - Music Faster results = more Questions = low cost Boost customer engagement thro. Behaviour pattern anal. & recommendation Mine text for Sentiment analysis - Facebook and - Twitter - news reports - other information on the Internet Understand, with 80 percent accuracy, how consumers feel about GE and its various divisions * Public examples taken from the from the web 20

21 Real World Illustrations* Improve customer experience through understanding phone usage patterns and layering maps of network traffic PB stored in Hadoop 100s TB in DWH Avoided silo d apps and created an ecosystem for data sharing and analytics Analyze billions of records to gain a better understanding of the impact of new and existing financial products. Can now examine things like credit and operational risk of products across different lines of business including home loans, insurance and online banking, Mine welogs and feed output into Datawarehouse Output from hadoop weblog analysis forms at least 40% of the data loaded each day and is about 1/3 of the data warehouse content. 1 TB+ of transactional data / day loaded total size 150 TB * Public examples taken from the from the web 21

22 Real World Illustrations* Building web analytics on hadoop for top 20 global web properties Increase clicks by improving stickiness Process 500m events daily 1PB on 80 nodes Optimise content placement next to current article user is reading; great ad revenues Solving fraud via a data hub capturing global daily and intraday transactions Hadoop used for fraud detection and risk management Model on entire universe not samples Culls information it collects on the site to find people you may know. It scores as many as 120 billion relationships per day using a statistical model to determine the probability that you may know another member. * Public examples taken from the from the web 22

23 PWC Technology Forecast Disney Sums Up Well it s exploratory... Provides indications... Not absolute conclusions... Requires a different mindset, exploration... Hypothesis, with pre-conceptions... Know results won t be as specific as a profitmargin/calculation... Still expect demonstrable value and expect to get it without a lot of extra expense

24 <Insert Picture Here> Understanding the Oracle Product Offering 24

25 Oracle Big Data Appliance Hardware 648TB per rack 216 cores 864GB Memory Infiniband Software Cloudera Distribution of Hadoop Oracle No*SQL DB (Community Edition) R Oracle Connectors Value Pre-engineered / Pre-configured Enterprise features & support R scale out Data integration Re-use of skils 25

26 We ship Cloudera Hadoop 26

27 Oracle NoSQL Database NoSQL to meet Enterprise requirements Simple Data Model Key-value pair with major+sub-key paradigm Read/insert/update/delete operations ACID Transactions Scalability Dynamic data partitioning and distribution Optimized data access via intelligent driver High availability One or more replicas Disaster recovery through location of replicas No single point of failure Transparent load balancing Reads from master or replicas Driver is network topology & latency aware Elastic (Planned for Release 2) Online addition/removal of Storage Nodes Automatic data redistribution Storage Nodes Data Center A Storage Nodes Data Center B 27

28 We provide Four Big Data Connectors Direct Access Hadoop HDFS 11g HDFS Files Load into Oracle Orchestrate Read from Oracle 28

29 11g Advanced Analytics Option 29

30 Oracle Exalytics Oracle TimesTen In-Memory Database Tightly integrated with Oracle Exadata Enables speed-of-thought visualization Oracle Business Intelligence Foundation Suite, Endeca and RTD

31 Big Data in Life-Cycle DECIDE ACQUIRE Make Better Decisions Using Big Data ANALYZE ORGANIZE 31

32 A Complete Data Architecture Big Data Appliance Exadata Exalytics * CEP (Complex Event Processing) - Part of Fusion Middleware is also key 32