A Perfect Storm. Oracle Big Data Science for Enterprise R and SAS Users. Marcos Arancibia, Consulting Product Manager marcos.arancibia@oracle.

Transcription

1 A Perfect Storm Oracle Big Data Science for Enterprise R and SAS Users Mark Hornick, Director, Advanced Analytics Marcos Arancibia, Consulting Product Manager marcos.arancibia@oracle.com

2 Safe Harbor Statement The following is intended to outline our general product direction. It is intended for information purposes only, and may not be incorporated into any contract. It is not a commitment to deliver any material, code, or functionality, and should not be relied upon in making purchasing decisions. The development, release, and timing of any features or functionality described for Oracle s products remains at the sole discretion of Oracle. 2

3 Perfect Storm: something that describes an actual phenomenon that happens to occur in such a confluence, resulting in an event of unusual magnitude - Wikipedia 3

4 Big Data Cloud Masters in Data Science 4

5 Events of unusual magnitude? Massive migrations A changing of the guard 5

6 Agenda What is R? Who is using R and why? Overview of Oracle R Technologies Global customer tour Demonstration 6

7 What is R? R is an Open Source scripting language and environment for statistical computing and graphics Started in 1994 as an Alternative to SAS, SPSS and other proprietary Statistical Environments The R environment R is an integrated suite of software facilities for data manipulation, calculation and graphical display Millions of R users worldwide Widely taught in Universities Many Corporate Analysts and Data Scientists know and use R Thousands of open sources packages to enhance productivity such as: Bioinformatics Spatial Statistics Financial Market Analysis Social Network Analysis

8 Why statisticians, data analysts, data scientists use R R environment is.. Powerful Extensible Graphical Extensive statistics OOTB functionality with many knobs but smart defaults Ease of installation and use Free R is a statistics language similar to Base SAS or SPSS statistics

9 R s Popularity Number of Google Scholar hits SPSS SAS R SPSS has a clear lead, but you can see that its dominance peaked in 2007 and its use is now in sharp decline. SAS never came close to SPSS level of dominance, and it peaked in Robert A. Muenchen

10 R s Popularity Number of Google Scholar Documents SAS and SPSS removed R the use of R is experiencing very rapid growth and is pulling away from the pack, solidifying its position in third place. Robert A. Muenchen

11 R s Popularity Job Trends R vs. SPSS R vs. SAS

12 KDNuggets Poll: Languages used for Analytics / Data Mining 12

13 Customer Pain Points with Advanced Analytics for example It takes too long to get my data or to get the right data I can t analyze or mine all of my data it has to be sampled Putting models and results into production is ad hoc and complex Recoding models into SQL, C, or Java takes time and is error prone Our company is concerned about data security, backup and recovery We need to build 10s of thousands of models fast to meet business objectives 13

14 Oracle R Technologies 14

15 Oracle R Distribution Ability to dynamically load Intel Math Kernel Library AMD Core Math Library Solaris Sun Performance Library Oracle Support An Oracle-Supported Redistribution of Open Source R Enhanced linear algebra performance via dynamically loaded libraries Improve scalability at client and database for embedded R execution Enterprise support for customers of Oracle Advanced Analytics option, Big Data Appliance, and Oracle Linux Free download Oracle contributes bug fixes and enhancements to open source R

16 ROracle Oracle Database ROracle R package enabling scalable and performant connectivity to Oracle Database Open source, publicly available on CRAN Oracle is maintainer Oracle Database Interface (DBI) for R Re-implemented and optimized driver based on OCI Execute SQL statements from R interface Enables transactional behavior for insert, update, and delete

17 Oracle R Enterprise Oracle Advanced Analytics Option to Oracle Database Eliminate memory constraint of client R engine Minimize or eliminate data movement latency Leverage Oracle Database as HPC environment Execute R scripts through database server machine for scalability and performance Leverage parallel, distributed in-database data mining algorithms Execute and manage R scripts via SQL Operationalize R scripts in production applications eliminate porting R code Avoid reinventing code to integrate R results into existing applications Client R Engine ORE packages Oracle Database User tables In-db stats SQL Interfaces SQL*Plus, SQLDeveloper, Database Server Machine 17

18 Database-centric architecture Smart meter scenario Oracle Database Data c1 c2 ci cn R Datastore R Script Repository f(dat,args, ) f(dat,args, ) f(dat,args, ) f(dat,args, ) 200K Households f(dat,args, ) { R Script build model Model c1 Model c2 Model ci Model cn }

19 Database-centric architecture Smart meter scenario Oracle Database Data c1 c2 ci cn R Datastore Model R Script Repository f(dat,args, ) f(dat,args, ) f(dat,args, ) f(dat,args, ) 200K Households f(dat,args, ) { } R Script score data scores c1 scores c2 scores ci scores cn

20 Build models and store in database, partition on CUST_ID ore.groupapply (CUST_USAGE_DATA, 14 lines CUST_USAGE_DATA$CUST_ID, function(dat, ds.name) { cust_id <- dat$cust_id[1] mod <- lm(consumption ~. -CUST_ID, dat) mod$effects <- mod$residuals <- mod$fitted.values <- NULL name <- paste("mod", cust_id,sep="") assign(name, mod) ds.name1 <- paste(ds.name,".",cust_id,sep="") ore.save(list=paste("mod",cust_id,sep=""), name=ds.name1, overwrite=true) TRUE }, ds.name="mydatastore", ore.connect=true, parallel=true ) 20

21 Score customers in database, partition on CUST_ID ore.groupapply(cust_usage_data_new, CUST_USAGE_DATA_NEW$CUST_ID, 16 lines function(dat, ds.name) { cust_id <- dat$cust_id[1] ds.name1 <- paste(ds.name,".",cust_id,sep="") ore.load(ds.name1) name <- paste("mod", cust_id,sep="") mod <- get(name) prd <- predict(mod, newdata=dat) prd[as.integer(rownames(prd))] <- prd res <- cbind(cust_id=cust_id, PRED = prd) data.frame(res) }, ds.name="mydatastore", ore.connect=true, parallel=true, FUN.VALUE=data.frame(CUST_ID=numeric(0), PRED=numeric(0)) ) 21

22 Execution (sec) Performance with DOP= Models Data: 26,280,000 rows Total build time: 65.2 seconds Total scoring time: 25.7 seconds (all data) 50,000 Models Data: 1,314,000,000 rows Total build time: minutes Total scoring time: 18 minutes (all data) 10,000 Models Data: 262,800,000 rows Total build time: 516 seconds Total scoring time: 217 seconds (all data) 1 Model/Customer # rows (millions) Build Time Score Time 22

23 Seconds (log scale) Oracle Advanced Analytics on Exadata X3-2 ¼ Rack Scalability of the new distributed ore.lm() Linear Regression Model Building using 30 numeric variables: Leading Vendor on a machine connected directly to the same Exadata box took 2+ hours for ETL+Exec on 34mi records 7, minutes Leading Vendor-34mi OAA-34mi OAA-180mi OAA-299mi OAA-2.99Bi Engine/Database size (records)

24 Oracle Advanced Analytics Option Fastest Way to Deliver Scalable Enterprise-wide Predictive Analytics Better Decisions with Deeper Insights & Predictive Analytics Understand and predict customer behavior for churn, fraud, cross-sell, and many other business problems Easy to Use Data analysts: Mining work flow GUI (part of SQL Developer) Data scientists: R and SQL languages supported DBA: SQL integration Comprehensive Analytics on a Simple Architecture Performance and scalability of the Oracle Database Lowest Total Cost of Ownership No need for separate analytical servers Components Oracle R Enterprise Oracle Data Mining

25 HCache Hadoop Abstraction Layer Oracle R Advanced Analytics for Hadoop Oracle Big Data Connectors option to Big Data Appliance ORD R Client R script {CRAN packages} Hadoop Job Mapper Reducer R HDFS R MapReduce R Hive R sqoop/olh Hadoop Cluster MapReduce Nodes {CRAN packages} HDFS Nodes ORD Transparent access to Hadoop Cluster from R Manipulate data in HDFS, Hive, database, and file system Write and execute MapReduce jobs with R Leverage CRAN R packages to work on HDFS-resident data Prepackaged parallel, distributed algorithms Oracle Database

26 Oracle Big Data Platform Oracle Big Data Appliance Optimized for Hadoop, R, and NoSQL Processing Oracle Big Data Connectors Oracle Exadata System of Record Optimized for DW/OLTP Oracle Exalytics Optimized for Analytics & In-Memory Workloads Hadoop Oracle R Distribution Oracle NoSQL Database Applications Oracle R Oracle Big Data Advanced Analytics Connectors for Hadoop + Oracle Data Integrator Oracle R Oracle Oracle Enterprise Advanced Advanced Analytics Analytics Oracle Data Mining Data Warehouse Oracle Database Oracle R Distribution Oracle Enterprise Performance Management Oracle Business Intelligence Applications Oracle Business Intelligence Tools Oracle Endeca Information Discovery Stream Acquire Organize Discover & Analyze

27 A Global Customer Tour 27

28 Quick Houston Facts: Most populous city in Texas Metropolitan area is the fifth-most populated in the U.S., with over 6 million people Leading in energy, manufacturing, aeronautics, transportation, health care sectors and building oilfield equipment Only New York City is home to more Fortune 500 Headquarters. Panoramic Houston skyline Oracle R Enterprise at Apache Oil: Segmentation of drilling problems to understand potential problems ahead of time Predictive maintenance of assets to prevent waiting a day for replacement of drill bits or other components helps optimize Revenues The space shuttle Challenger atop its Boeing 747 SCA, flying over Johnson Space Center,

29 Quick Guatemala Facts: 15.8 mi inhabitants Guatemala City is the Capital Spanish spoken by 93% of Population 21 Mayan and 2 Amerindian languages also spoken Service sector is largest component of GDP at 63%, followed by industry sector at 23.8% and agriculture sector at 13.2% (2010 est.) Mayan City of Tikal Oracle R Enterprise at TIGO: Customer Behavior of 5.5M customers with 1.8B transactions Generate 5 models per customer to understand mobility using Lat/Long of the Cell Tower of each transaction Evaluate 27.5M segmentation models in 25 minutes, or over 1M models/minute Guatemala City Today 29

30 Quick Cincinnati Facts: First major American city founded after American Revolution First major inland purely American city in country Cincinnati Reds have a storied history as being first professional club, hosting first night game, and dominating 1970s as the "Big Red Machine" American Sign Museum Oracle Advanced Analytics at dunnhumby: Very long ETL time eliminated with in-database Advanced Analytics Modeling behavior of millions of shoppers Coupon optimization for Retailers on Billions of transactions Insights and exploration on all data vs. samples Average Project time from 19 days to 3 days Music Hall Cincinnati Reds: Great American Ballpark

31 Quick Orlando Facts: Nicknamed "The City Beautiful" Symbol is the fountain at Lake Eola Theme Park Capital of the World 51+ million tourists a year, 3.6 million international Walt Disney World Resort: Magic Kingdom, Hollywood Studios, Epcot, Animal Kingdom Universal Studios Orlando SeaWorld Oracle R Enterprise at Olive Garden: Olive Garden, traditionally managing its 830 restaurants nationally, transitioned to a localized approach with the help of predictive analytics Evaluated 115 million transactions in just 5 percent the time required by previous BI tool Supporting Olive Garden s latest remodel campaign, continuing to uncover millions in profits by optimizing pricing and menu assortment TODAY, 2:00PM! Moscone South 308 CON2898

32 Quick Lima, Peru Facts: Capital and the largest city of Peru with 9M citizens Most populous metropolitan area of Peru Fifth largest city in the Americas (as defined by "city proper") Home to one of the oldest higher learning institutions in the New World National University of San Marcos, founded on May 12, 1551 Oracle R Enterprise at Financiera Uno: Reduce time to build credit scoring models to ensure their market relevancy Scale to handle big data volumes Rapidly deploy credit scoring models into production applications

33 Quick London Facts: One of world's leading financial centers Has fifth-or sixth-largest metropolitan area GDP in the world depending on measurement World cultural capital World's most-visited city as measured by international arrivals World's largest city airport system measured by passenger traffic Oracle R Enterprise at Major Financial Company: Earnings calculations reduced from 7 hours to 4 minutes Scoring on written premium reduced from 100 minutes to 7 minutes Scoring on earned premium reduced from 25 minutes to 8 minutes with added functionality Load time improved by factor of 15X 33

34 Quick Geneva Facts: Most populous city of Romandy, the Frenchspeaking part of Switzerland A financial center Worldwide center for diplomacy Headquarters of many of the agencies of the United Nations and the Red Cross Hosts highest number of international organizations in the world Oracle R Enterprise at CERN: Real time monitoring and anomaly detection of tens of thousands of events per second CERN Central Logging Service: complex in-database time series analysis and forecasting Electrical Consumption forecast for LHC and experiments 34

35 Quick Croatia Facts: Member of European Union (EU) and United Nations (UN) Tourism is a significant source of revenue during the summer Ranked 18th most popular tourist destination in the world Oracle R Enterprise at ZABA Bank: Historical Customer Behavior Analysis shortened from several months to 2 weeks Specialized Variable Clustering algorithm running in parallel to replace leading vendor solution Faster model development resulted in better model quality and increasing bottom line

36 Quick Korea Facts: Roughly half of the country's 50 million people reside in the metropolitan area surrounding its capital, Seoul Seoul Capital Area is the second largest in the world with over 25 million residents Eighth largest country in international trade A regional power with world's 10th largest defense budget Oracle R Enterprise at BISTEL: Oracle ORE enables BISTEL to perform analytics with much more data faster and enables them to gain more insight (root cause and prediction) With Oracle Exadata BISTEL can do enterprise advanced process control in Mega/Giga fabs in high-tech manufacturing

37 37

38 Video 39

39 Demonstration 40

40 compare 41

41 See these Advanced Analytics Talks at OOW 14 CON Developing Relevant Dining Visits with Oracle Advanced Analytics at Olive Garden CON Extending the Power of In-Database Analytics with Oracle Big Data Appliance CON Predictive Analytics with Oracle Data Mining CON Market Basket Analysis at Dunkin Brands CON Big Data and Predictive Analytics: Fiserv Data Mining Case Study 42

42 Learn More about Oracle s R Technologies Join us at our booth Moscone South SLD-115 Oracle Demogrounds 43

43 Oracle Confidential Internal/Restricted/Highly Restricted 44

44