The Open Analytics Platform

Size: px

Start display at page:

Download "The Open Analytics Platform"

Shanon Washington
7 years ago
Views:

1 The Open Analytics Platform Bernd Wiswedel KNIME.com AG

2 Agenda KNIME.com AG The KNIME Platform Recognition Small Sales Pitch KNIME and R the best of two worlds KNIME (Node) Development 2

3 A Brief History of KNIME 2004: KNIME development commences 2006: KNIME v1 released 2006: Spin-off in Konstanz, Germany 2008: KNIME moves to Zurich 2010: Enterprise products released 2011: KNIME.com AG founded 2013: KNIME opens San Francisco office 2014: KNIME opens Berlin office KNIME saved my life in a world of scripts that I do not want to learn! 3

4 The KNIME Platform 4

5 The KNIME Platform 5

6 KNIME loads and integrates data from diverse data sources: Different data bases Various file formats (CSV, XML, SDF, etc.) Data Loading 6

7 KNIME provides huge repository of modules for easy-to-use, modular Data preprocessing Data fusion Data transformation Data Loading ETL 7

8 In addition to standard data mining techniques, KNIME adds cutting edge data analysis algorithms. ( thanks to its academic roots) Data Loading ETL Data Mining 8

9 Interactive views provide data overviews and insights into the learned models. Interactive linking&brushing techniques allow for powerful exploration of models and data. Data Loading ETL Data Mining Visualization 9

10 Due to its open API and node-in-a-sandbox -approach additional (also external) tools are easily integrated, e.g. Access to the R Project (statistical analysis/visualizations) Complete integration of the machine learning library WEKA Application area specific integration, e.g. CDK (Chemical Development Kit), RDKit, ImageJ, KNIME is Eclipse-based: Integrating other Eclipse projects such as BIRT, DTP, etc. provides even more functionality Data Loading ETL Data Mining Visualization External Tools 10

11 Over 1000 native and embedded nodes included: MySQL, Oracle, etc. SAS, SPSS, etc. Excel, Flat, etc. Hive etc. XML, PMML Text, Doc, Image Web Crawlers Industry Specific Community / 3rd ETL Row, Column Matrix Text, Image Time Series Java Python Community / 3rd Statistics Data Mining Machine Learning Web Analytics Text Mining Network Analysis Social Media Analysis WEKA R Community / 3rd R JFreeChart Community / 3rd via BIRT PMML XML Databases Excel, Flat, etc. Hive etc. Text, Doc, Image Industry Specific Community / 3rd 11

12 Commercial Partners integrate their proprietary tools KNIME serves as an integration platform for tools of various vendors (or your inhouse/legacy applications) 3rd Party Tools Visualization External Tools 12

13 Small KNIME Demo 13

14 Who s Using KNIME? >25,000 Individuals using KNIME >3,000 Organizations using KNIME >300 Customers paying for KNIME as of January k Annual Unique Downloads 40k 20k Open Source Users

15 Broad Range of KNIME Application Areas Pharma Manufacturing Health Care Advanced Analytics Customer Intelligence Finance Retail 15

16 Top in User Satisfaction 2012 & 2013 Rexer Analytics Survey 16

17 Sales Pitch: The KNIME Server at Work 17

18 KNIME in Action: Big Data As long as your machine can handle it, KNIME will play along. 18

19 KNIME and Big Data 19

20 KNIME and Big Data 20

21 KNIME and R The best of two worlds 21

22 Why use KNIME andr? R Powerful statistics (b)leading edge algorithms Powerful/flexible graphics Widely accepted language Open source analytics Cross platform Vibrant communities KNIME Powerful GUI Good Extract/Transform/Load Integrates diverse tools Enterprise grade solutions

23 Two Integrations Community (RServe Integration) R Interactive (Today's topic)

24 Overview of new nodes Different input and output options Grey ports enable workspace branching

25 The Interactive Editor Columns Variables Code Editor Workspace Overview Console

26 Templates List Summary Preview

27 R Source nodes Get data from an R data frame Assign output to a data frame named knime.out Use with foreign, RCurl, or...

28 R Snippet nodes Generic data manipulation Edit tables or workspaces Derive knime.out from knime.in Use for cumulative stats, plyr, or...

29 R Data Mining nodes Use R models in KNIME Learner (knime.model) & Predictor motif R to PMML support for model portability

30 R View nodes Generic R plots Plot(knime.in) Use with many packages including ggplot2

31 R in Action: Choropleth Generation

32 R in Action: Dose Response modeling

33 A Peak under the Hood: KNIME (Node) Development 33

34 KNIME Analytics Platform: Technology Overview KNIME Workflow Manager & User Interface Node Interface Node Interface Node Interface Node Interface Node Interface KNIME I/O KNIME Native Algorithms Open Source Integrations (R, BIRT, ) Community Extensions Partner Extensions Data Mgmt & Execution Ctrl Data Mgmt & Execution Ctrl Data Mgmt & Execution Ctrl Data Mgmt & Execution Ctrl Data Mgmt & Execution Ctrl Meta Data Handling KNIME Data Management and Execution Layer Data Management Execution Control Data Type Extensions In Memory Data Handling Automatic Data Caching Distributed Data Storage Multi Core Execution Cluster Execution Distributed Execution

35 Node Architecture class Node (final) class class Node- Node- Dialog- Model Pane (abstract) (abstract) class Node- View (abstract) KNIME interacts only with a Node Nodetakes care of embedding the node in the infrastructure New nodes implement Model/View/Dialog class NodeFactory (abstract) 35

36 Node Extension Wizard Included in the KNIME Developer Version Allows creation of plugin projects including functioning KNIME nodes (with sample code) Helpful to easily create all node classes Generates all Java classes Node is registered with the plugin project Launch KNIME and enjoy the new node working! 36

37 Node Extension Wizard 37

38 Node Extension Wizard Specify all settings to create a new KNIME node In a completely new plugin project, or Into an existing project Node type: Sink, Source, Learner, Predictor, Manipulator, Visualizer, Meta, or Other Include sample code or not 38

39 Node Extension Wizard Contains all Java classes (including sample code) Node is registered in the plugin.xml NodeDialogand NodeViewclass are also created and registered to the NodeFactory 39

40 Resources KNIME pages( APPLICATIONS for example workflows LEARNING HUB underresources KNIME Tech pages (tech.knime.org) FORUM for questions and answers DOCUMENTATION for documentation, FAQ, changelogs,... LABS where to find new experimental nodes COMMUNITY CONTRIBUTIONS for development instructions and third party nodes KNIME TV channel on 40

41 Thank you

KNIME TUTORIAL. Anna Monreale KDD-Lab, University of Pisa Email: annam@di.unipi.it

KNIME TUTORIAL. Anna Monreale KDD-Lab, University of Pisa Email: annam@di.unipi.it KNIME TUTORIAL Anna Monreale KDD-Lab, University of Pisa Email: annam@di.unipi.it Outline Introduction on KNIME KNIME components Exercise: Market Basket Analysis Exercise: Customer Segmentation Exercise: