R / TERR. Ana Costa e SIlva, PhD Senior Data Scientist TIBCO. Copyright TIBCO Software Inc.

Size: px

Start display at page:

Winfred Hood
8 years ago
Views:

1 R / TERR Ana Costa e SIlva, PhD Senior Data Scientist TIBCO Copyright TIBCO Software Inc.

2 Tower of Big and Fast Data Visual Data Discovery Hundreds of Records Millions of Records Key peformance indicators Data Mining Real Time Analytics Billions of Records (Big Data) Trillions of Records (Fast Data) Copyright TIBCO Software Inc. 2

Mining Real Time Analytics Billions of Records (Big Data)

3 Tower of Big and Fast Data Spotfire Analyst Spotfire Business Author Spotfire Consumer Visual Data Discovery Spotfire Event Analytics Real Time Analytics Hundreds of Records Millions of Records Billions of Records (Big Data) Key peformance indicators Spotfire Mobile Metrics Data Mining TIBCO Enterprise Runtime for R Trillions of Records (Fast Data) Copyright TIBCO Software Inc. 3

Records Billions of Records (Big Data) Key peformance indicators Spotfire Mobile Metrics Data

4 TERR TIBCO Enterprise Runtime for R (TERR) Latest in family of statistics scripting engines: S, S-PLUS, R, TERR Commercial Releases: v1.0 Nov 2012, v2.0 Nov 2013, v2.1 Feb 2014, Developer Edition: Engine internals rebuilt from scratch Redesigned data object representation Redesigned memory management facilities Addresses long-standing problems with S language Fast and scalable engine!! 4

com/community/products/analytics/terr Engine internals rebuilt from scratch Redesigned data object

5 TERR Performance Model Fitting: 5 Million Rows Model Scoring: 20 Million Rows TERR 7X faster 84X 5

6 TERR: The Fastest Road to Big Data TERR: TIBCO Enterprise Runtime for R Most stable and performant access to analytics Zero learning curve for R programmers Supports in-database, in-hadoop functionality Teradata, Oracle, ; Apache, Horton, Cloudera, MapR, Deployment TERR Server execution: TIBCO Spotfire Statistics Services CEP Integration: TIBCO Business Events, Streambase Grid Integration: TIBCO GridServer Infrastructure Integration: TIBCO Business Works, 6

Apache, Horton, Cloudera, MapR, Deployment TERR Server execution: TIBCO Spotfire Statistics Services CEP

7 RStudio integration TERR now compatible with the most popular IDE in the R Community Professional-quality development environment to use with TERR Features Syntax highlighting, code completion, and smart indentation Execute R code directly from the source editor Manage multiple working directories using projects Quickly navigate code TERR integration with RStudio IDE

code completion, and smart indentation Execute R code directly from the source editor Manage

8 Demo 1 8

9 Hadoop / TERR: Write Your Mapper Use Standard R Syntax; Run using TERR If you can understand this, you can write mapreduce: cat input mapper sort reducer mapper <- function(d) { words <- strsplit(paste(d, collapse = ' '), '[[:punct:][:space:]]+')[[1]] # split on punctuation and spaces words <-words[!(words == '')] # get rid of empty words caused by whitespace at beginning of lines df <- data.frame(word = words) df$cnt<-1 hswritetable(df, sep = "\t") } 9

'[[:punct:][:space:]]+')[[1]] # split on punctuation and spaces words <-words[!

10 Write Your Reducer Use Standard R Syntax; Run using TERR If you can understand this, you can write mapreduce: cat input mapper sort reducer reducer <- function(d) { # d$wordis all one value per mapreduce cat(paste(d$word[1], sum(d$cnt), collapse="\t"), "\n") } 10

$per mapreduce cat(paste(d$word[1], sum(d$cnt), collapse="\t"),$

11 TERR Map Reduce From the command line: $ hadoop-streaming map mapper.r reduce reducer.r input inputfile output outputfile From TERR: optionally call remotely via TIBCO Spotfire Statistics Services Return.code <- system( hadoop-streaming map mapper.r reduce reducer.r input inputfile output outputfile ) 11

r input inputfile output outputfile From TERR: optionally call remotely via

12 Hadoop Big Data Tools Complex Technical Confusing TIBCO Approach Authors and Consumers Hide Complexity, Empower Users Visual Query data on demand Fit interface to User skills 12

13 TERR Map Reduce Spotfire via Statistics Services Mapper.R TERRscript Reducer.R via TERRscript Hadoop Streaming $ hadoop-streaming map mapper.r reduce reducer.r -input inputfile output outputfile HDFS Each Node Processes its own data using TERR Data Node Data Node Data Node Data Node 13

R via TERRscript Hadoop Streaming $ hadoop-streaming map mapper.

14 Demo 2 14

15 TERR MapReduce from Spotfire Parameterize MapReduce, Generate and Edit MapReduce code, Test Locally, I/O from Spotfire Deploy through Hadoop Streaming MapReduce Interface from/to Spotfire Receive analysis results directly back into Spotfire for visualisation and further analysis Copyright TIBCO Software Inc.

MapReduce Interface from/to Spotfire Receive analysis results directly back

16 Contact Thank you! Ana Costa e Silva, PhD Senior Data Scientist ansilva@tibco.com TERR Developer Edition: Copyright TIBCO Software Inc. Copyright TIBCO Software Inc. 16

com TERR Developer Edition: www.tibcommunity.

6.0, 6.5 and Beyond. The Future of Spotfire. Tobias Lehtipalo Sr. Director of Product Management

6.0, 6.5 and Beyond. The Future of Spotfire. Tobias Lehtipalo Sr. Director of Product Management 6.0, 6.5 and Beyond The Future of Spotfire Tobias Lehtipalo Sr. Director of Product Management Key peformance indicators Hundreds of Records Visual Data Discovery Millions of Records Data Mining or Data