Oracle Big Data for Dummies

Transcription

1 Oracle Big Data for Dummies Sai Janakiram Penumuru WW Product Expert Cloud Platforms Hewlett-Packard, India

2 The Father of Microbiology first microbiologist Antonie Philips van Leeuwenhoek 2

3 Sai Janakiram Penumuru o Twelve years in Oracle DBA / Oracle Apps DBA / Cloud Architect o Current Position: WW Product Expert, Cloud Platform - Oracle in hp o Co-Fonder & Director of Finance - All India Oracle Users Group (AIOUG) o Oracle Database 12c Beta Tester o Oracle VM SIG Leader 3 o Blog:

4 Agenda A New Style of IT Defining Big Data: Market Drivers and Trends Why Big Data Now? Overview of Oracle Big Data Appliance Oracle Big Data Implementation Steps Oracle Integrated Software Solution Oracle Big Data Best Practices Demo The Call to Action (Next Steps)

5 A new era of accelerated innovation Forever changing how consumers and businesses interact, enabling new opportunities 2013 Every 60 seconds 98,000+ tweets 695,000 status updates 11million instant messages 698,445 Google searches 168 million+ s sent 1,820TB of data created Growing Internet of Things (IoT) By 2020 Devices DATA Mobile Apps 30 Billion (1) 40 Trillion GB (2) 10 Million (3) 217 new mobile web users Pervasive Connectivity Smart Device Expansion Explosion of Information for 8 Billion (4) A new style of IT required for IoT solutions 5 (1) IDC Directions 2013: Why the Datacenter of the Future Will Leverage a Converged Infrastructure, March 2013, Matt Eastwood ; (2) & (3) IDC Predictions 2012: Competing for 2020, Document , December 2011, Frank Gens; (4)

6 Defining Big Data: Market Drivers and Trends

7 Big Data defined Big data is high-volume, -velocity and variety information assets that demand cost-effective, innovative forms of information processing for enhanced insight and decision making Velocity Variety Big Data Volume Value Information Sources $ CRM, SCM, ERP Video IT Ops Transactional Data Mobile Audio Texts Social Media Search Images Big Data is no longer just a buzzword ¹Source: Gartner, The Importance of 'Big Data': A Definition, June

8 Information from the Internet of Things: We have gone beyond the decimal system Today data scientist uses Yottabytes to describe how much government data the NSA or FBI have on people altogether. In the near future, Brontobyte will be the measurement to describe the type of sensor data that will be generated from the IoT 8 Brontobyte This will be our digital universe tomorrow Yottabyte This is our digital universe today = 250 trillion of DVDs Exabyte 1 EB of data is created on the internet each day = 250 million DVDs worth of information. The proposed Square Kilometer Array telescope will generated an EB of data per day Terabyte 500TB of new data per day are ingested in Facebook databases Megabyte Geopbyte This will take us beyond our decimal system Zettabyte 1.3 ZB of network traffic by Petabyte The CERN Large Hadron Collider generates 1PB per second 10 9 Gigabyte

9 Second half of the chessboard The wheat and chessboard problem On the entire chessboard there would be = 18,446,744,073,709,551,615 grains of rice Weighing 461,168,602,000 metric tons Heap of rice larger than Mount Everest Around 1,000 times the global production of rice in 2010 (464,000,000 metric tons). 9

10 A real world example: Sensor data collected from US commercial jet engines during 1 year 20 TB , terabytes of information per engine every hour twin-engine Boeing 737 Average duration for US flights in hours # of commercial flights in the sky in the United States on any given day days in a year 1,041,600,500 TB 10

11 Big data is derived from a variety of sources Variety Big Data Velocity Volume 11

12 A day in the life of Big Data An intelligent end-to-end approach delivers the right information to the right person at the right time Executive Dashboards Enterprise Search Customer Interaction Predictive Analytics Web Engagement Variety Velocity Volume Social Media Video Audio MGD Texts Transactional Data CRM (Sales) Transactional Operational Strategic Web ERP (Procurement) Supply Chain (Ops) Word, Excel Logs Clickstream Data HR Images Machine Generated Data 12

13 Why Big Data Now?

14 Command of information drives increased business performance Right information. Right person. Right time. Insight What is happening now? Better, Right Time Decisions Decisions Active information In Flight information Inactive information Foresight What will happen? 14 Hindsight what happened?

15 Big Data opportunities across industries and use cases Innovative analytic use cases are cutting across structured, unstructured and semi structured data Finance Government Telecom Manufacturing Energy Healthcare Fraud detection Anti-money laundering Risk management Law enforcement Counter terrorism Traffic flow optimization Broadcast monitoring Churn prevention Advertising optimization Supply chain optimization Defect tracking RFID Correlation Warranty management Weather forecasting Natural resource exploration Drug development Scientific research Evidence based medicine Healthcare outcomes analysis Sentiment analysis Social CRM / network analysis Churn mitigation Brand monitoring Cross and Up sell Loyalty & promotion analysis Web application optimization Horizontal use cases Marketing campaign optimization Brand management Social media analytics Pricing optimization Internal risk assessment Customer behavior analysis Revenue assurance Logistics optimization Clickstream analysis Influencer analysis IT infrastructure analysis Legal discovery Equipment monitoring Enterprise search 15 Sources: IDC: 2012 Worldwide Big Data Technology and Services Forecast: , Gartner: 2012 Big Data Drives Rapid Changes in Infrastructure and $232 Billion in IT Spending Through 2016

16 Which Big Data outcomes are your peers focusing on? Solutions which improve operational efficiency and/or enhance business effectiveness Improve service levels Track new and/or changing trends Maximize margins Identify cross sell and upsell opportunities Support corporate strategy development Analyze/form credit risk assessment Perform product line management 16

17 Overview of Oracle Big Data Appliance

18 Oracle Big Data Appliance 18

19 Cloudera s Distribution for Hadoop File System Mount UI Framework SDK FUSE-DFS HUE HUE SDK Workflow Scheduling Metadata APACHE OOZIE APACHE OOZIE APACHE HIVE Data Integration Languages / Compilers APACHE PIG, APACHE HIVE, APACHE MAHOUT Fast Read/Write Access APACHE FLUME, SQOOP HDFS, MAPREDUCE APACHE HBASE Coordination APACHE ZOOKEEPER 19

20 BDA Full Rack: Hadoop + Oracle NoSQL Database Node1 Node2 Node3 Node4 Node5 Node6 Node7 Node8 Node9 Node10 Node11 Node12 Node13 Node14 Node15 Node16 Node17 Node18 First NameNode, DataNode, ZooKeeper,failover controller, Balancer, Puppet master & Agent, NoSQL Second NameNode,DataNode,ZooKeeper,failover controller, MySQL backup Server, Puppet Agent, NoSQL Admin Job Tracker, DataNode, ZooKeeper, CMserver, ODI agent, MySQL primary server, Hue, Hive, Beeswax,Puppet Agent, NoSQL DataNode, TaskTracker, Cloudera Mgr Agents,Puppet Agents, NoSQL DataNode, TaskTracker, Cloudera Mgr Agents,Puppet Agents, NoSQL DataNode, TaskTracker, Cloudera Mgr Agents,Puppet Agents, NoSQL DataNode, TaskTracker, Cloudera Mgr Agents,Puppet Agents, NoSQL DataNode, TaskTracker, Cloudera Mgr Agents,Puppet Agents, NoSQL DataNode, TaskTracker, Cloudera Mgr Agents,Puppet Agents, NoSQL DataNode, TaskTracker, Cloudera Mgr Agents,Puppet Agents, NoSQL DataNode, TaskTracker, Cloudera Mgr Agents,Puppet Agents, NoSQL DataNode, TaskTracker, Cloudera Mgr Agents,Puppet Agents, NoSQL DataNode, TaskTracker, Cloudera Mgr Agents,Puppet Agents, NoSQL DataNode, TaskTracker, Cloudera Mgr Agents,Puppet Agents, NoSQL DataNode, TaskTracker, Cloudera Mgr Agents,Puppet Agents, NoSQL DataNode, TaskTracker, Cloudera Mgr Agents,Puppet Agents, NoSQL DataNode, TaskTracker, Cloudera Mgr Agents,Puppet Agents, NoSQL DataNode, TaskTracker, Cloudera Mgr Agents,Puppet Agents, NoSQL 20

21 Oracle Big Data Implementation Steps

22 Implementation Step -1 Acquire Organize Analyze Visualize Collect tweets into HDFS using Twitter API Specify search terms Terms may be Categorized Bulk collect or stream 22

23 Implementation Step - 2 Acquire Organize Analyze Visualize MapReduce jobs compute sentiment Summarize by airline, category, time and user Load into Oracle Database using Oracle SQL Connector for HDFS Map Twitter ID to customer 23

24 Implementation Step - 3 Acquire Organize Analyze Visualize Combine other customer data sets to provide a deeper level of analysis, such as social and economic importance. 24

25 Implementation Step - 4 Acquire Organize Analyze Visualize Visualize social ad economic importance by using Oracle Database analytics and OBIEE dashboards to drive decision making 25

26 Oracle Integrated Software Solution

27 Oracle Big Data Appliance Software Overview Shows the relationships among the tools and identifies the tasks that they perform 27

28 Oracle Engineered Systems Data Variety Unstructured Schema-less Big Data Appliance Oracle BI Schema Exadata 28 Information Density Acquire Organize Analyze

29 Oracle Big Data Appliance Usage Model Oracle Big Data Appliance Oracle Exadata Oracle Exalytics InfiniBand InfiniBand Stream Acquire Organize Analyze & Visualize 29

30 Oracle Big Data Best Practices

31 Big Data Best Practices Few general guidelines to build a successful big data architecture foundation Align Big Data with Specific Business Goals Ease Skills Shortage with Standards and Governance Optimize Knowledge Transfer with a Center of Excellence Top Payoff is Aligning Unstructured with Structured Data Plan Your Sandbox For Performance Align with the Cloud Operating Model 31

32 Demo - How to get Oracle Big Data Lite VM - Start/Stop Services - Software Walk through - Getting data from Twitter

33 Oracle Big Data Lite Virtual Machine Prepare your host system Version 4.0 Technical Requirements: Dedicate 2 cores 5 GB memory 30GB disk space to the virtual machine Install will require ~45GB disk space including temporary files To get started: Download and install Oracle VM VirtualBox and 7-zip Download each of the 7-zip files Run the 7-zip extractor on the BigDataLite file only. This will create the BigDataLite-4.0.ova VirtualBox appliance file In VirtualBox, import BigDataLite-4.0.ova Start BigDataLite-4.0 Log in as oracle/welcome1 33

34 Download Oracle Big Data Lite Virtual Machine Version 4.0 File Deployment Guide BigDataLite ( bytes) BigDataLite ( bytes) BigDataLite ( bytes) BigDataLite ( bytes) BigDataLite ( bytes) BigDataLite ( bytes) BigDataLite ( bytes) BigDataLite ( bytes) BigDataLite ( bytes) BigDataLite ( bytes) BigDataLite ( bytes) BigDataLite ( bytes) BigDataLite ( bytes) md5sum.txt (346 bytes) Cloudera JDBC Drivers Description Start Here! Deployment Guide provides step-by-step instructions for download and deployment. To get started: Download and install Oracle VM VirtualBox and 7- zip Download each of the 7-zip files Run the 7-zip extractor on the BigDataLite file only. This will create the BigDataLite-4.0.ova VirtualBox appliance file In VirtualBox, import BigDataLite-4.0.ova Start BigDataLite-4.0 Log in as oracle/welcome1 See the Deployment Guide for details. Download and install the Cloudera JDBC drivers to enable Oracle SQL Developer and Data Modeler to connect to Hive. Oracle Enterprise Linux 6.4 Oracle Database 12c Release 1 Enterprise Edition ( ) - including Oracle Big Data SQL-enabled external tables Cloudera Distribution including Apache Hadoop (CDH5.1.2) Cloudera Manager (5.1.2) Oracle Big Data Connectors 4.0 Oracle SQL Connector for HDFS Oracle Loader for Hadoop Oracle Data Integrator 12c Oracle R Advanced Analytics for Hadoop Oracle XQuery for Hadoop Oracle NoSQL Database Enterprise Edition 12cR1 (3.0.14) Oracle JDeveloper 12c (12.1.3) Oracle SQL Developer and Data Modeler Oracle Data Integrator 12cR1 (12.1.3) Oracle GoldenGate 12c Oracle R Distribution

35 Installation Steps Prepare your host system Version 4.0 Run the 7-zip extractor on the BigDataLite file only. This will create the BigDataLite-4.0.ova VirtualBox appliance file Start Oracle Virtual Box Manager and Import the Appliance 35

36 What s Next

37 Harness the power of Big Data the time is now Requires a cultural shift, enterprise-wide Build an information driven corporate culture By using information effectively to understand and align with your business needs and preferences of your customers is the key to creating a competitive advantage in today's customer-empowered environment. 37

38 Sangam14 The Largest Independent Oracle Conference In India Tom Kyte Maria Colgan 38 Sangam14-6th Annual Oracle Users Conference - INDIA. Friday, 7 th, Saturday 8 th & Sunday 9 th November Bangalore.

39 Thank you oracle.com/bigdata