Business Intelligence in Microservice Architecture. Debarshi bol.com

Save this PDF as:
 WORD  PNG  TXT  JPG

Size: px
Start display at page:

Download "Business Intelligence in Microservice Architecture. Debarshi Basak @ bol.com"

Transcription

1 Business Intelligence in Microservice Architecture Debarshi bol.com

2 What can you expect? - Introduction Monolithic days Mapreduce Era Flink Era Operational Aspect

3 Who am I? Debarshi Basak Software engineer at bol.com Part of Bigdata platform team and online marketing

4 About bol.com - Leader in Dutch ecommerce Scrum employees 40+ scrum teams Young and relaxed You build it. You run it. You love it.

5 How big is big data at bol.com? - > products for sale Catalog > newsletter responses new clicks every day - 26 node cluster More than 300 jobs a month in production

6 What is Microservice?

7 What is Microservice? App Database

8 What is Microservice? Services Services App Services Services Services Services Database Services

9 Business Intelligence 101

10 Business Intelligence 101

11 Business Intelligence Analyzing data and presenting actionable items

12 Business Intelligence Analyzing data and presenting actionable items - Automated and Continuous Integration with internal and external data sources

13 Business Intelligence Analyzing data and presenting actionable items - Automated and Continuous Integration with internal and external data sources - Flexible Analytics

14 Business Intelligence ETL - Extract from source Transform the data Load into target data models

15 Business Intelligence ETL - - Extract from source Transform the data Load into target data models Business data modelling - Kimball s dimensional modeling technique OLAP cubes

16

17 Monolithic days

18 Evolution of BI Systems (at bol.com) Online systems

19 Evolution of BI Systems (at bol.com) Online systems Data hub

20 Evolution of BI Systems (at bol.com) Online systems Replication Data hub

21 Evolution of BI Systems (at bol.com) Online systems Data hub Replication Data warehouse

22 Evolution of BI Systems (at bol.com) Online systems Data hub Data warehouse ETL

23 Evolution of BI Systems (at bol.com) - Easy to implement Complexities are abstracted Data Overheads Latency

24 Evolution of BI Systems (at bol.com) Online systems

25 Evolution of BI Systems (at bol.com) Online systems Publish Message Broker

26 Evolution of BI Systems (at bol.com) Online systems Message Broker Listener Data warehouse

27 Evolution of BI Systems (at bol.com) Online systems Message Broker Listener Data warehouse

28 Evolution of BI Systems (at bol.com) - Loss of Messages and Consistency guarantees Database are kind of not made for this Complex implementation Nightmare for operations

29 Challenges in Microservice Architecture

30 Challenges in Microservice Architecture - Too many sources Can affect scalability and stability of reports BI cannot scale Extraction logic, transformation operations for each Service Joins.

31 Hadoop era

32 History of Hadoop at bol.com Operational experience in hbase, hadoop based tooling - Supplier connector - Recommendation system

33 Service Definition RPC over HTTP Message Queues Service Bulk Interfaces

34 Bulk Interfaces t1-productid1 f:price::15,50 t2-productid2 f:price::15,35 t3-productid1 f:price::15,25 t4-productid1 f:price::15,75 t5-productid3 f:price::15,50

35 Bulk Interfaces t1-productid1 f:price::15,50 t2-productid2 f:price::15,35 t3-productid1 f:price::15,25 t4-productid1 f:price::15,75 t5-productid3 f:price::15,50 We can replay event to get the latest state of the event. This is also known as Event Sourcing Pattern. Similar key design can be found in OpenTSDB

36 Re-imagine traditional BI on hadoop Source systems Staging-Transformation Loading in reporting tooling

37 Re-imagine traditional BI on hadoop Source systems Staging-Transformation Loading in reporting tooling

38 Supplier Service Offers Services Pricing Services Data warehouse

39 From Queues Supplier Service Offers Services Pricing Services Data warehouse

40 Supplier Service Offers Services Pricing Services Data warehouse

41 Supplier Service Offers Services Pricing Services Data warehouse

42 Supplier Service Offers Services Pricing Services Data warehouse

43 Supplier Service Offers Services Pricing Services Data warehouse

44 Automation Supplier Service Integration Unit Offers Services Pricing Services Data warehouse

45 What kind of jobs we build - Aggregation jobs - - Single service, aggregate on a function key Interface concatenation - Multiple services combined on one/many functional keys.

46 Automation { "bulk_interface" : "transport_acc_public_v1_transporter_versions", "table_name" : "transporter", "primary_key" : "f:transporter.transporterid", "ctl_name" : "transporter", "service_version" : "v2", "col_map":[ { "hbase_col_name": "f:transporter.transporterid", "ora_col_name":"id", "function_name":"transporter id", "data_type" : "NUMBER" }, { "hbase_col_name": "f:transporter.transportercode", "ora_col_name":"code", "function_name":"transporter code", "data_type" : "VARCHAR2(40)" }, { "hbase_col_name": "f:transporter.transportername", "ora_col_name":"transportername", "function_name":"transporter Name", "data_type" : "VARCHAR2(40)" } ] }

47 Problem Source systems Staging-Transformation Loading in reporting tooling

48 Problem Source systems Staging-Transformation Loading in reporting tooling

49 Problem Source systems Staging-Transformation Loading in reporting tooling

50 But everything is stream. Nature of data in most of use cases is asynchronous. Clicks are asynchronous Orders are asynchronous Updates are asynchronous In fact, Batch is a bounded stream.

51 Streaming era

52 Enter Flink Low entry barrier Java/Scala functional apis. Operational expertise.

53 Emulating Stream You don t always need queues for stream Streaming HBase tables.

54 Give a starting point in stream t1-productid1 f:price::15,50 t2-productid2 f:price::15,35 t3-productid1 f:price::15,25 t4-productid1 f:price::15,75 t5-productid3 f:price::15,50

55 Give next x records t1-productid1 f:price::15,50 t2-productid2 f:price::15,35 t3-productid1 f:price::15,25 t4-productid1 f:price::15,75 t5-productid3 f:price::15,50

56 Give next x records t1-productid1 f:price::15,50 t2-productid2 f:price::15,35 t3-productid1 f:price::15,25 t4-productid1 f:price::15,75 t5-productid3 f:price::15,50

57 while(true){ Give next x records } t1-productid1 f:price::15,50 t2-productid2 f:price::15,35 t3-productid1 f:price::15,25 t4-productid1 f:price::15,75 t5-productid3 f:price::15,50

58 Offers Product Catalog

59 ProductId Offers ProductId Product Catalog Product-Offer-Join

60 Offers Product Catalog Product-Offer-Join

61 Offers Product Catalog Product-Offer-Join

62 Offers Product Catalog Product-Offer-Join

63 Offers Product Catalog Product-Offer-Join

64 Offers Product Catalog Product-Offer-Join

65 Offers Eventual Consistency ProductId Product Catalog Product-Offer-Join

66 Offers Product Catalog Sources Product-Offer-Other Sources-Join

67 Offers Product Catalog Sources Cube or star

68 Offers Product Catalog Sources Star

69 Can we automate this?

70 Can we automate this? Yes, We can.

71 Can we automate this? cube_builder.from( table("productoffer_tst_public_v1.0_sellingoffers_versions") ).on( key("f:globalid", key -> new StringBuilder(key).reverse().toString()) ).lookup( key("f:globalid"), table("financecategory_tst_public_v1_productfinancecategorycurrents") ).to( table("final_join_version"), table("reverse_index_lookup", key("f:globalid"), columns("f:offerid")), table("final_join_version1", columns("f:sellingofferdata.listprice")) ).build().execute();

72 Operational Aspect Build

73 Operational Aspect Build

74 Operational Aspect Build Docker Registry

75 Operational Aspect Build Docker Registry Deploy

76 Operational Aspect Build Docker Registry Deploy

77 Operational Aspect Build Docker Registry Deploy

78 Lessons learned - Dedicated team for hadoop Think not tools but how to solve problems Flink can be flinky Frameworks are out there Kylin Think infrastructure too

79

Beyond Lambda - how to get from logical to physical. Artur Borycki, Director International Technology & Innovations

Beyond Lambda - how to get from logical to physical. Artur Borycki, Director International Technology & Innovations Beyond Lambda - how to get from logical to physical Artur Borycki, Director International Technology & Innovations Simplification & Efficiency Teradata believe in the principles of self-service, automation

More information

A very short talk about Apache Kylin Business Intelligence meets Big Data. Fabian Wilckens EMEA Solutions Architect

A very short talk about Apache Kylin Business Intelligence meets Big Data. Fabian Wilckens EMEA Solutions Architect A very short talk about Apache Kylin Business Intelligence meets Big Data Fabian Wilckens EMEA Solutions Architect 1 The challenge today 2 Very quickly: OLAP Online Analytical Processing How many beers

More information

MS 20467: Designing Business Intelligence Solutions with Microsoft SQL Server 2012

MS 20467: Designing Business Intelligence Solutions with Microsoft SQL Server 2012 MS 20467: Designing Business Intelligence Solutions with Microsoft SQL Server 2012 Description: This five-day instructor-led course teaches students how to design and implement a BI infrastructure. The

More information

LEARNING SOLUTIONS website milner.com/learning email training@milner.com phone 800 875 5042

LEARNING SOLUTIONS website milner.com/learning email training@milner.com phone 800 875 5042 Course 20467A: Designing Business Intelligence Solutions with Microsoft SQL Server 2012 Length: 5 Days Published: December 21, 2012 Language(s): English Audience(s): IT Professionals Overview Level: 300

More information

Apache Kylin Introduction Dec 8, 2014 @ApacheKylin

Apache Kylin Introduction Dec 8, 2014 @ApacheKylin Apache Kylin Introduction Dec 8, 2014 @ApacheKylin Luke Han Sr. Product Manager lukhan@ebay.com @lukehq Yang Li Architect & Tech Leader yangli9@ebay.com Agenda What s Apache Kylin? Tech Highlights Performance

More information

Establish and maintain Center of Excellence (CoE) around Data Architecture

Establish and maintain Center of Excellence (CoE) around Data Architecture Senior BI Data Architect - Bensenville, IL The Company s Information Management Team is comprised of highly technical resources with diverse backgrounds in data warehouse development & support, business

More information

Pulsar Realtime Analytics At Scale. Tony Ng April 14, 2015

Pulsar Realtime Analytics At Scale. Tony Ng April 14, 2015 Pulsar Realtime Analytics At Scale Tony Ng April 14, 2015 Big Data Trends Bigger data volumes More data sources DBs, logs, behavioral & business event streams, sensors Faster analysis Next day to hours

More information

Big Data Pipeline and Analytics Platform

Big Data Pipeline and Analytics Platform Big Data Pipeline and Analytics Platform Using NetflixOSS and Other Open Source Software Sudhir Tonse (@stonse) Danny Yuan (@g9yuayon) Netflix is a log generating company that also happens to stream movies

More information

SAS Business Intelligence Online Training

SAS Business Intelligence Online Training SAS Business Intelligence Online Training IQ Training facility offers best online SAS Business Intelligence training. Our SAS Business Intelligence online training is regarded as the best training in Hyderabad

More information

Designing Business Intelligence Solutions with Microsoft SQL Server 2012 Course 20467A; 5 Days

Designing Business Intelligence Solutions with Microsoft SQL Server 2012 Course 20467A; 5 Days Lincoln Land Community College Capital City Training Center 130 West Mason Springfield, IL 62702 217-782-7436 www.llcc.edu/cctc Designing Business Intelligence Solutions with Microsoft SQL Server 2012

More information

Designing Business Intelligence Solutions with Microsoft SQL Server B; 5 days, Instructor-led

Designing Business Intelligence Solutions with Microsoft SQL Server B; 5 days, Instructor-led Designing Business Intelligence Solutions with Microsoft SQL Server 2012 20467B; 5 days, Instructor-led Course Description This five-day instructor-led course teaches students how to design and implement

More information

East Asia Network Sdn Bhd

East Asia Network Sdn Bhd Course: Analyzing, Designing, and Implementing a Data Warehouse with Microsoft SQL Server 2014 Elements of this syllabus may be change to cater to the participants background & knowledge. This course describes

More information

Getting Real Real Time Data Integration Patterns and Architectures

Getting Real Real Time Data Integration Patterns and Architectures Getting Real Real Time Data Integration Patterns and Architectures Nelson Petracek Senior Director, Enterprise Technology Architecture Informatica Digital Government Institute s Enterprise Architecture

More information

Luncheon Webinar Series May 13, 2013

Luncheon Webinar Series May 13, 2013 Luncheon Webinar Series May 13, 2013 InfoSphere DataStage is Big Data Integration Sponsored By: Presented by : Tony Curcio, InfoSphere Product Management 0 InfoSphere DataStage is Big Data Integration

More information

How to Enhance Traditional BI Architecture to Leverage Big Data

How to Enhance Traditional BI Architecture to Leverage Big Data B I G D ATA How to Enhance Traditional BI Architecture to Leverage Big Data Contents Executive Summary... 1 Traditional BI - DataStack 2.0 Architecture... 2 Benefits of Traditional BI - DataStack 2.0...

More information

Data Integration Checklist

Data Integration Checklist The need for data integration tools exists in every company, small to large. Whether it is extracting data that exists in spreadsheets, packaged applications, databases, sensor networks or social media

More information

Data Warehousing Fundamentals: A Comprehensive Guide for IT Professionals

Data Warehousing Fundamentals: A Comprehensive Guide for IT Professionals Data Warehousing Fundamentals: A Comprehensive Guide for IT Professionals Ponniah, Paulraj ISBN-13: 9780470462072 Table of Contents Preface. Part 1 OVERVIEW AND CONCEPTS. 1 The Compelling Need for Data

More information

More Data in Less Time

More Data in Less Time More Data in Less Time Leveraging Cloudera CDH as an Operational Data Store Daniel Tydecks, Systems Engineering DACH & CE Goals of an Operational Data Store Load Data Sources Traditional Architecture Operational

More information

COURSE 20463C: IMPLEMENTING A DATA WAREHOUSE WITH MICROSOFT SQL SERVER

COURSE 20463C: IMPLEMENTING A DATA WAREHOUSE WITH MICROSOFT SQL SERVER Page 1 of 8 ABOUT THIS COURSE This 5 day course describes how to implement a data warehouse platform to support a BI solution. Students will learn how to create a data warehouse with Microsoft SQL Server

More information

Decoding the Big Data Deluge a Virtual Approach. Dan Luongo, Global Lead, Field Solution Engineering Data Virtualization Business Unit, Cisco

Decoding the Big Data Deluge a Virtual Approach. Dan Luongo, Global Lead, Field Solution Engineering Data Virtualization Business Unit, Cisco Decoding the Big Data Deluge a Virtual Approach Dan Luongo, Global Lead, Field Solution Engineering Data Virtualization Business Unit, Cisco High-volume, velocity and variety information assets that demand

More information

European Archival Records and Knowledge Preservation Database Archiving in the E-ARK Project

European Archival Records and Knowledge Preservation Database Archiving in the E-ARK Project European Archival Records and Knowledge Preservation Database Archiving in the E-ARK Project Janet Delve, University of Portsmouth Kuldar Aas, National Archives of Estonia Rainer Schmidt, Austrian Institute

More information

Business Intelligence for Big Data

Business Intelligence for Big Data Business Intelligence for Big Data Will Gorman, Vice President, Engineering May, 2011 2010, Pentaho. All Rights Reserved. www.pentaho.com. What is BI? Business Intelligence = reports, dashboards, analysis,

More information

Managing Data in Motion

Managing Data in Motion Managing Data in Motion Data Integration Best Practice Techniques and Technologies April Reeve ELSEVIER AMSTERDAM BOSTON HEIDELBERG LONDON NEW YORK OXFORD PARIS SAN DIEGO SAN FRANCISCO SINGAPORE SYDNEY

More information

Implementing a Data Warehouse with Microsoft SQL Server MOC 20463

Implementing a Data Warehouse with Microsoft SQL Server MOC 20463 Implementing a Data Warehouse with Microsoft SQL Server MOC 20463 Course Outline Module 1: Introduction to Data Warehousing This module provides an introduction to the key components of a data warehousing

More information

COURSE OUTLINE MOC 20463: IMPLEMENTING A DATA WAREHOUSE WITH MICROSOFT SQL SERVER

COURSE OUTLINE MOC 20463: IMPLEMENTING A DATA WAREHOUSE WITH MICROSOFT SQL SERVER COURSE OUTLINE MOC 20463: IMPLEMENTING A DATA WAREHOUSE WITH MICROSOFT SQL SERVER MODULE 1: INTRODUCTION TO DATA WAREHOUSING This module provides an introduction to the key components of a data warehousing

More information

Integrating Hadoop. Into Business Intelligence & Data Warehousing. Philip Russom TDWI Research Director for Data Management, April 9 2013

Integrating Hadoop. Into Business Intelligence & Data Warehousing. Philip Russom TDWI Research Director for Data Management, April 9 2013 Integrating Hadoop Into Business Intelligence & Data Warehousing Philip Russom TDWI Research Director for Data Management, April 9 2013 TDWI would like to thank the following companies for sponsoring the

More information

Implementing a Data Warehouse with Microsoft SQL Server 2012

Implementing a Data Warehouse with Microsoft SQL Server 2012 Course Code: M10777 Vendor: Microsoft Course Overview Duration: 5 RRP: 1,935 Implementing a Data Warehouse with Microsoft SQL Server 2012 Overview This 5-day instructor-led course describes how to implement

More information

Implement a Data Warehouse with Microsoft SQL Server 20463C; 5 days

Implement a Data Warehouse with Microsoft SQL Server 20463C; 5 days Lincoln Land Community College Capital City Training Center 130 West Mason Springfield, IL 62702 217-782-7436 www.llcc.edu/cctc Implement a Data Warehouse with Microsoft SQL Server 20463C; 5 days Course

More information

Cisco IT Hadoop Journey

Cisco IT Hadoop Journey Cisco IT Hadoop Journey Srini Desikan, Program Manager IT 2015 MapR Technologies 1 Agenda Hadoop Platform Timeline Key Decisions / Lessons Learnt Data Lake Hadoop s place in IT Data Platforms Use Cases

More information

Big Data for Investment Research Management

Big Data for Investment Research Management IDT Partners www.idtpartners.com Big Data for Investment Research Management Discover how IDT Partners helps Financial Services, Market Research, and Investment Management firms turn big data into actionable

More information

Managing Big Data with Hadoop & Vertica. A look at integration between the Cloudera distribution for Hadoop and the Vertica Analytic Database

Managing Big Data with Hadoop & Vertica. A look at integration between the Cloudera distribution for Hadoop and the Vertica Analytic Database Managing Big Data with Hadoop & Vertica A look at integration between the Cloudera distribution for Hadoop and the Vertica Analytic Database Copyright Vertica Systems, Inc. October 2009 Cloudera and Vertica

More information

Lambda Architecture. Near Real-Time Big Data Analytics Using Hadoop. January 2015. Email: bdg@qburst.com Website: www.qburst.com

Lambda Architecture. Near Real-Time Big Data Analytics Using Hadoop. January 2015. Email: bdg@qburst.com Website: www.qburst.com Lambda Architecture Near Real-Time Big Data Analytics Using Hadoop January 2015 Contents Overview... 3 Lambda Architecture: A Quick Introduction... 4 Batch Layer... 4 Serving Layer... 4 Speed Layer...

More information

BIG DATA ANALYTICS REFERENCE ARCHITECTURES AND CASE STUDIES

BIG DATA ANALYTICS REFERENCE ARCHITECTURES AND CASE STUDIES BIG DATA ANALYTICS REFERENCE ARCHITECTURES AND CASE STUDIES Relational vs. Non-Relational Architecture Relational Non-Relational Rational Predictable Traditional Agile Flexible Modern 2 Agenda Big Data

More information

Implementing a Data Warehouse with Microsoft SQL Server

Implementing a Data Warehouse with Microsoft SQL Server This course describes how to implement a data warehouse platform to support a BI solution. Students will learn how to create a data warehouse 2014, implement ETL with SQL Server Integration Services, and

More information

Real Time Big Data Processing

Real Time Big Data Processing Real Time Big Data Processing Cloud Expo 2014 Ian Meyers Amazon Web Services Global Infrastructure Deployment & Administration App Services Analytics Compute Storage Database Networking AWS Global Infrastructure

More information

Big Data Analytics Platform @ Nokia

Big Data Analytics Platform @ Nokia Big Data Analytics Platform @ Nokia 1 Selecting the Right Tool for the Right Workload Yekesa Kosuru Nokia Location & Commerce Strata + Hadoop World NY - Oct 25, 2012 Agenda Big Data Analytics Platform

More information

Enterprise Data Integration for Microsoft Dynamics CRM

Enterprise Data Integration for Microsoft Dynamics CRM Enterprise Data Integration for Microsoft Dynamics CRM Daniel Cai http://danielcai.blogspot.com About me Daniel Cai Developer @KingswaySoft a software company offering integration software and solutions

More information

Oracle9i Data Warehouse Review. Robert F. Edwards Dulcian, Inc.

Oracle9i Data Warehouse Review. Robert F. Edwards Dulcian, Inc. Oracle9i Data Warehouse Review Robert F. Edwards Dulcian, Inc. Agenda Oracle9i Server OLAP Server Analytical SQL Data Mining ETL Warehouse Builder 3i Oracle 9i Server Overview 9i Server = Data Warehouse

More information

SQL Server 2005 Features Comparison

SQL Server 2005 Features Comparison Page 1 of 10 Quick Links Home Worldwide Search Microsoft.com for: Go : Home Product Information How to Buy Editions Learning Downloads Support Partners Technologies Solutions Community Previous Versions

More information

Big Data on Tap Jonathan Gray

Big Data on Tap Jonathan Gray Unified Integration for Data-Driven Applications Big Data on Tap Jonathan Gray Founder & CEO November 7, 2016 Hadoop Enables New Applications and Architectures ENTERPRISE DATA LAKES BIG DATA ANALYTICS

More information

MDM and Data Warehousing Complement Each Other

MDM and Data Warehousing Complement Each Other Master Management MDM and Warehousing Complement Each Other Greater business value from both 2011 IBM Corporation Executive Summary Master Management (MDM) and Warehousing (DW) complement each other There

More information

Republic Polytechnic School of Information and Communications Technology C355 Business Intelligence. Module Curriculum

Republic Polytechnic School of Information and Communications Technology C355 Business Intelligence. Module Curriculum Republic Polytechnic School of Information and Communications Technology C355 Business Intelligence Module Curriculum This document addresses the content related abilities, with reference to the module.

More information

Course Outline. Module 1: Introduction to Data Warehousing

Course Outline. Module 1: Introduction to Data Warehousing Course Outline Module 1: Introduction to Data Warehousing This module provides an introduction to the key components of a data warehousing solution and the highlevel considerations you must take into account

More information

Evolution of Web Application Architecture International PHP Conference. Kore Nordmann / @koredn / <kore@qafoo.com> June 9th, 2015

Evolution of Web Application Architecture International PHP Conference. Kore Nordmann / @koredn / <kore@qafoo.com> June 9th, 2015 Evolution of Web Application Architecture International PHP Conference Kore Nordmann / @koredn / June 9th, 2015 Evolution Problem Too many visitors Evolution Evolution Lessons Learned:

More information

SAS BI Course Content; Introduction to DWH / BI Concepts

SAS BI Course Content; Introduction to DWH / BI Concepts SAS BI Course Content; Introduction to DWH / BI Concepts SAS Web Report Studio 4.2 SAS EG 4.2 SAS Information Delivery Portal 4.2 SAS Data Integration Studio 4.2 SAS BI Dashboard 4.2 SAS Management Console

More information

Architectural patterns for building real time applications with Apache HBase. Andrew Purtell Committer and PMC, Apache HBase

Architectural patterns for building real time applications with Apache HBase. Andrew Purtell Committer and PMC, Apache HBase Architectural patterns for building real time applications with Apache HBase Andrew Purtell Committer and PMC, Apache HBase Who am I? Distributed systems engineer Principal Architect in the Big Data Platform

More information

Designing Business Intelligence Solutions with Microsoft SQL Server 2012

Designing Business Intelligence Solutions with Microsoft SQL Server 2012 CÔNG TY CỔ PHẦN TRƯỜNG CNTT TÂN ĐỨC TAN DUC INFORMATION TECHNOLOGY SCHOOL JSC LEARN MORE WITH LESS! Course 20467B: Designing Business Intelligence Solutions with Microsoft SQL Server 2012 Length: 5 Days

More information

Linux A first-class citizen in Windows Azure. Bruno Terkaly bterkaly@microsoft.com Principal Software Engineer Mobile/Cloud/Startup/Enterprise

Linux A first-class citizen in Windows Azure. Bruno Terkaly bterkaly@microsoft.com Principal Software Engineer Mobile/Cloud/Startup/Enterprise Linux A first-class citizen in Windows Azure Bruno Terkaly bterkaly@microsoft.com Principal Software Engineer Mobile/Cloud/Startup/Enterprise 1 First, I am software developer (C/C++, ASM, C#, Java, Node.js,

More information

Extending the Enterprise Data Warehouse with Hadoop Robert Lancaster. Nov 7, 2012

Extending the Enterprise Data Warehouse with Hadoop Robert Lancaster. Nov 7, 2012 Extending the Enterprise Data Warehouse with Hadoop Robert Lancaster Nov 7, 2012 Who I Am Robert Lancaster Solutions Architect, Hotel Supply Team rlancaster@orbitz.com @rob1lancaster Organizer of Chicago

More information

Monitor Your Key Performance Indicators using WSO2 Business Activity Monitor

Monitor Your Key Performance Indicators using WSO2 Business Activity Monitor Published on WSO2 Inc (http://wso2.com) Home > Stories > Monitor Your Key Performance Indicators using WSO2 Business Activity Monitor Monitor Your Key Performance Indicators using WSO2 Business Activity

More information

Implementing a Data Warehouse with Microsoft SQL Server

Implementing a Data Warehouse with Microsoft SQL Server Page 1 of 7 Overview This course describes how to implement a data warehouse platform to support a BI solution. Students will learn how to create a data warehouse with Microsoft SQL 2014, implement ETL

More information

Orchestrating Distributed Deployments with Docker and Containers 1 / 30

Orchestrating Distributed Deployments with Docker and Containers 1 / 30 Orchestrating Distributed Deployments with Docker and Containers 1 / 30 Who am I? Jérôme Petazzoni (@jpetazzo) French software engineer living in California Joined Docker (dotcloud) more than 4 years ago

More information

I/O Considerations in Big Data Analytics

I/O Considerations in Big Data Analytics Library of Congress I/O Considerations in Big Data Analytics 26 September 2011 Marshall Presser Federal Field CTO EMC, Data Computing Division 1 Paradigms in Big Data Structured (relational) data Very

More information

Course: SAS BI(business intelligence) and DI(Data integration)training - Training Duration: 30 + Days. Take Away:

Course: SAS BI(business intelligence) and DI(Data integration)training - Training Duration: 30 + Days. Take Away: Course: SAS BI(business intelligence) and DI(Data integration)training - Training Duration: 30 + Days Take Away: Class notes and Books, Data warehousing concept Assignments for practice Interview questions,

More information

Course Outline: Course: Implementing a Data Warehouse with Microsoft SQL Server 2012 Learning Method: Instructor-led Classroom Learning

Course Outline: Course: Implementing a Data Warehouse with Microsoft SQL Server 2012 Learning Method: Instructor-led Classroom Learning Course Outline: Course: Implementing a Data with Microsoft SQL Server 2012 Learning Method: Instructor-led Classroom Learning Duration: 5.00 Day(s)/ 40 hrs Overview: This 5-day instructor-led course describes

More information

Implementing a Data Warehouse with Microsoft SQL Server 2012

Implementing a Data Warehouse with Microsoft SQL Server 2012 Course 10777A: Implementing a Data Warehouse with Microsoft SQL Server 2012 Course Details Course Outline Module 1: Introduction to Data Warehousing This module provides an introduction to the key components

More information

Implementing a Data Warehouse with Microsoft SQL Server 2012 MOC 10777

Implementing a Data Warehouse with Microsoft SQL Server 2012 MOC 10777 Implementing a Data Warehouse with Microsoft SQL Server 2012 MOC 10777 Course Outline Module 1: Introduction to Data Warehousing This module provides an introduction to the key components of a data warehousing

More information

Ganzheitliches Datenmanagement

Ganzheitliches Datenmanagement Ganzheitliches Datenmanagement für Hadoop Michael Kohs, Senior Sales Consultant @mikchaos The Problem with Big Data Projects in 2016 Relational, Mainframe Documents and Emails Data Modeler Data Scientist

More information

The Inside Scoop on Hadoop

The Inside Scoop on Hadoop The Inside Scoop on Hadoop Orion Gebremedhin National Solutions Director BI & Big Data, Neudesic LLC. VTSP Microsoft Corp. Orion.Gebremedhin@Neudesic.COM B-orgebr@Microsoft.com @OrionGM The Inside Scoop

More information

Microsoft. Course 20463C: Implementing a Data Warehouse with Microsoft SQL Server

Microsoft. Course 20463C: Implementing a Data Warehouse with Microsoft SQL Server Course 20463C: Implementing a Data Warehouse with Microsoft SQL Server Length : 5 Days Audience(s) : IT Professionals Level : 300 Technology : Microsoft SQL Server 2014 Delivery Method : Instructor-led

More information

PLATFORA INTERACTIVE, IN-MEMORY BUSINESS INTELLIGENCE FOR HADOOP

PLATFORA INTERACTIVE, IN-MEMORY BUSINESS INTELLIGENCE FOR HADOOP PLATFORA INTERACTIVE, IN-MEMORY BUSINESS INTELLIGENCE FOR HADOOP Your business is swimming in data, and your business analysts want to use it to answer the questions of today and tomorrow. YOU LOOK TO

More information

20463C: Implementing a Data Warehouse with Microsoft SQL Server

20463C: Implementing a Data Warehouse with Microsoft SQL Server 20463C: Implementing a Data Warehouse with Microsoft SQL Server Course Details Course Code: Duration: Notes: 20463C 5 days This course syllabus should be used to determine whether the course is appropriate

More information

Time-Series Databases and Machine Learning

Time-Series Databases and Machine Learning Time-Series Databases and Machine Learning Jimmy Bates November 2017 1 Top-Ranked Hadoop 1 3 5 7 Read Write File System World Record Performance High Availability Enterprise-grade Security Distribution

More information

Ten Things You Need to Know About Data Virtualization

Ten Things You Need to Know About Data Virtualization White Paper Ten Things You Need to Know About Data Virtualization What is Data Virtualization? Data virtualization is an agile data integration method that simplifies information access. Data virtualization

More information

Designing Self-Service Business Intelligence and Big Data Solutions

Designing Self-Service Business Intelligence and Big Data Solutions CÔNG TY CỔ PHẦN TRƯỜNG CNTT TÂN ĐỨC TAN DUC INFORMATION TECHNOLOGY SCHOOL JSC LEARN MORE WITH LESS! Course 20467C: Designing Self-Service Business Intelligence and Big Data Solutions Length: 5 Days Audience:

More information

IST722 Data Warehousing

IST722 Data Warehousing IST722 Data Warehousing Components of the Data Warehouse Michael A. Fudge, Jr. Recall: Inmon s CIF The CIF is a reference architecture Understanding the Diagram The CIF is a reference architecture CIF

More information

Hadoop Data Hubs and BI. Supporting the migration from siloed reporting and BI to centralized services with Hadoop

Hadoop Data Hubs and BI. Supporting the migration from siloed reporting and BI to centralized services with Hadoop Hadoop Data Hubs and BI Supporting the migration from siloed reporting and BI to centralized services with Hadoop John Allen October 2014 Introduction John Allen; computer scientist Background in data

More information

Developing Business Intelligence and Data Visualization Applications with Web Maps

Developing Business Intelligence and Data Visualization Applications with Web Maps Developing Business Intelligence and Data Visualization Applications with Web Maps Introduction Business Intelligence (BI) means different things to different organizations and users. BI often refers to

More information

Capitalize on Big Data for Competitive Advantage with Bedrock TM, an integrated Management Platform for Hadoop Data Lakes

Capitalize on Big Data for Competitive Advantage with Bedrock TM, an integrated Management Platform for Hadoop Data Lakes Capitalize on Big Data for Competitive Advantage with Bedrock TM, an integrated Management Platform for Hadoop Data Lakes Highly competitive enterprises are increasingly finding ways to maximize and accelerate

More information

BIG DATA TRENDS AND TECHNOLOGIES

BIG DATA TRENDS AND TECHNOLOGIES BIG DATA TRENDS AND TECHNOLOGIES THE WORLD OF DATA IS CHANGING Cloud WHAT IS BIG DATA? Big data are datasets that grow so large that they become awkward to work with using onhand database management tools.

More information

EII - ETL - EAI What, Why, and How!

EII - ETL - EAI What, Why, and How! IBM Software Group EII - ETL - EAI What, Why, and How! Tom Wu 巫 介 唐, wuct@tw.ibm.com Information Integrator Advocate Software Group IBM Taiwan 2005 IBM Corporation Agenda Data Integration Challenges and

More information

Information Architecture

Information Architecture The Bloor Group Actian and The Big Data Information Architecture WHITE PAPER The Actian Big Data Information Architecture Actian and The Big Data Information Architecture Originally founded in 2005 to

More information

From Spark to Ignition:

From Spark to Ignition: From Spark to Ignition: Fueling Your Business on Real-Time Analytics Eric Frenkiel, MemSQL CEO June 29, 2015 San Francisco, CA What s in Store For This Presentation? 1. MemSQL: A real-time database for

More information

Implementing a Data Warehouse with Microsoft SQL Server

Implementing a Data Warehouse with Microsoft SQL Server Course Code: M20463 Vendor: Microsoft Course Overview Duration: 5 RRP: 2,025 Implementing a Data Warehouse with Microsoft SQL Server Overview This course describes how to implement a data warehouse platform

More information

Course 20463:Implementing a Data Warehouse with Microsoft SQL Server

Course 20463:Implementing a Data Warehouse with Microsoft SQL Server Course 20463:Implementing a Data Warehouse with Microsoft SQL Server Type:Course Audience(s):IT Professionals Technology:Microsoft SQL Server Level:300 This Revision:C Delivery method: Instructor-led (classroom)

More information

DATA INTEGRATION. in the world of microservices

DATA INTEGRATION. in the world of microservices DATA INTEGRATION in the world of microservices About me Valentine Gogichashvili Head of Data Engineering @ZalandoTech twitter: @valgog google+: +valgog email: valentine.gogichashvili@zalando.de One of

More information

Native Connectivity to Big Data Sources in MSTR 10

Native Connectivity to Big Data Sources in MSTR 10 Native Connectivity to Big Data Sources in MSTR 10 Bring All Relevant Data to Decision Makers Support for More Big Data Sources Optimized Access to Your Entire Big Data Ecosystem as If It Were a Single

More information

GigaSpaces Real-Time Analytics for Big Data

GigaSpaces Real-Time Analytics for Big Data GigaSpaces Real-Time Analytics for Big Data GigaSpaces makes it easy to build and deploy large-scale real-time analytics systems Rapidly increasing use of large-scale and location-aware social media and

More information

Three Open Blueprints For Big Data Success

Three Open Blueprints For Big Data Success White Paper: Three Open Blueprints For Big Data Success Featuring Pentaho s Open Data Integration Platform Inside: Leverage open framework and open source Kickstart your efforts with repeatable blueprints

More information

LEVERAGING ORACLE DRM TO INTEGRATE ORACLE EBS CUSTOMER DATA WITH SALESFORCE CERVELLO WHITEPAPER

LEVERAGING ORACLE DRM TO INTEGRATE ORACLE EBS CUSTOMER DATA WITH SALESFORCE CERVELLO WHITEPAPER LEVERAGING ORACLE DRM TO INTEGRATE ORACLE EBS CUSTOMER DATA WITH SALESFORCE CERVELLO WHITEPAPER INTRODUCTION Hierarchy management for both reporting and operational environments can create technical, business

More information

Addressing Risk Data Aggregation and Risk Reporting Ben Sharma, CEO. Big Data Everywhere Conference, NYC November 2015

Addressing Risk Data Aggregation and Risk Reporting Ben Sharma, CEO. Big Data Everywhere Conference, NYC November 2015 Addressing Risk Data Aggregation and Risk Reporting Ben Sharma, CEO Big Data Everywhere Conference, NYC November 2015 Agenda 1. Challenges with Risk Data Aggregation and Risk Reporting (RDARR) 2. How a

More information

CRM Magic with Data Migration & Integration

CRM Magic with Data Migration & Integration CRM Magic with Data Migration & Integration Daniel Cai http://www.kingswaysoft.com http://danielcai.blogspot.com About me Daniel Cai Principal Developer @KingswaySoft An independent software company offering

More information

SQL Server 2012 Business Intelligence Boot Camp

SQL Server 2012 Business Intelligence Boot Camp SQL Server 2012 Business Intelligence Boot Camp Length: 5 Days Technology: Microsoft SQL Server 2012 Delivery Method: Instructor-led (classroom) About this Course Data warehousing is a solution organizations

More information

ARBES BI MODERN SOLUTION for the support of strategic, tactical and operative management

ARBES BI MODERN SOLUTION for the support of strategic, tactical and operative management ARBES BI MODERN SOLUTION for the support of strategic, tactical and operative management www.arbes.com ARBES BI BUSINESS INTELLIGENCE The majority of companies today already know how important it is to

More information

Building Scalable Big Data Infrastructure Using Open Source Software. Sam William sampd@stumbleupon.

Building Scalable Big Data Infrastructure Using Open Source Software. Sam William sampd@stumbleupon. Building Scalable Big Data Infrastructure Using Open Source Software Sam William sampd@stumbleupon. What is StumbleUpon? Help users find content they did not expect to find The best way to discover new

More information

CORPORATE OVERVIEW. Big Data. Shared. Simply. Securely.

CORPORATE OVERVIEW. Big Data. Shared. Simply. Securely. CORPORATE OVERVIEW Big Data. Shared. Simply. Securely. INTRODUCING PHEMI SYSTEMS PHEMI unlocks the power of your data with out-of-the-box privacy, sharing, and governance PHEMI Systems brings advanced

More information

EMC Federation Big Data Solutions. Copyright 2015 EMC Corporation. All rights reserved.

EMC Federation Big Data Solutions. Copyright 2015 EMC Corporation. All rights reserved. EMC Federation Big Data Solutions 1 Introduction to data analytics Federation offering 2 Traditional Analytics! Traditional type of data analysis, sometimes called Business Intelligence! Type of analytics

More information

Big Data Analytics - Accelerated. stream-horizon.com

Big Data Analytics - Accelerated. stream-horizon.com Big Data Analytics - Accelerated stream-horizon.com StreamHorizon & Big Data Integrates into your Data Processing Pipeline Seamlessly integrates at any point of your your data processing pipeline Implements

More information

Multi-Datacenter Replication

Multi-Datacenter Replication www.basho.com Multi-Datacenter Replication A Technical Overview & Use Cases Table of Contents Table of Contents... 1 Introduction... 1 How It Works... 1 Default Mode...1 Advanced Mode...2 Architectural

More information

Implementing a Data Warehouse with Microsoft SQL Server

Implementing a Data Warehouse with Microsoft SQL Server CÔNG TY CỔ PHẦN TRƯỜNG CNTT TÂN ĐỨC TAN DUC INFORMATION TECHNOLOGY SCHOOL JSC LEARN MORE WITH LESS! Course 20463 Implementing a Data Warehouse with Microsoft SQL Server Length: 5 Days Audience: IT Professionals

More information

Lambda Architecture for Batch and Real- Time Processing on AWS with Spark Streaming and Spark SQL. May 2015

Lambda Architecture for Batch and Real- Time Processing on AWS with Spark Streaming and Spark SQL. May 2015 Lambda Architecture for Batch and Real- Time Processing on AWS with Spark Streaming and Spark SQL May 2015 2015, Amazon Web Services, Inc. or its affiliates. All rights reserved. Notices This document

More information

Virtualizing Apache Hadoop. June, 2012

Virtualizing Apache Hadoop. June, 2012 June, 2012 Table of Contents EXECUTIVE SUMMARY... 3 INTRODUCTION... 3 VIRTUALIZING APACHE HADOOP... 4 INTRODUCTION TO VSPHERE TM... 4 USE CASES AND ADVANTAGES OF VIRTUALIZING HADOOP... 4 MYTHS ABOUT RUNNING

More information

Big Data for Investment Research Management

Big Data for Investment Research Management IDT Partners www.idtpartners.com Big Data for Investment Research Management Discover how IDT Partners helps Financial Services, Market Research, and Investment firms turn big data into actionable research

More information

Big Data & Analytics Reference Architecture

Big Data & Analytics Reference Architecture An Oracle White Paper September 2013 Oracle Enterprise Transformation Solutions Series Big Data & Analytics Reference Architecture Executive Overview... 3 Introduction... 5 Reference Architecture Conceptual

More information

Whitepaper. The Emerging Big Data System - Testing Perspective. : Digital Assurance Practice : Nagarajan K R :

Whitepaper. The Emerging Big Data System - Testing Perspective. : Digital Assurance Practice : Nagarajan K R : Whitepaper The Emerging Big Data System Presented by Author Email Id : Digital Assurance Practice : Nagarajan K R : nagarajankr@hexaware.com Hexaware Technologies. All rights reserved. Table of Contents

More information

Oracle BI 10g: Analytics Overview

Oracle BI 10g: Analytics Overview Oracle BI 10g: Analytics Overview Student Guide D50207GC10 Edition 1.0 July 2007 D51731 Copyright 2007, Oracle. All rights reserved. Disclaimer This document contains proprietary information and is protected

More information

Putting Apache Kafka to Use!

Putting Apache Kafka to Use! Putting Apache Kafka to Use! Building a Real-time Data Platform for Event Streams! JAY KREPS, CONFLUENT! A Couple of Themes! Theme 1: Rise of Events! Theme 2: Immutability Everywhere! Level! Example! Immutable

More information

Unlock your data for fast insights: dimensionless modeling with in-memory column store. By Vadim Orlov

Unlock your data for fast insights: dimensionless modeling with in-memory column store. By Vadim Orlov Unlock your data for fast insights: dimensionless modeling with in-memory column store By Vadim Orlov I. DIMENSIONAL MODEL Dimensional modeling (also known as star or snowflake schema) was pioneered by

More information

Apache Hadoop: Past, Present, and Future

Apache Hadoop: Past, Present, and Future The 4 th China Cloud Computing Conference May 25 th, 2012. Apache Hadoop: Past, Present, and Future Dr. Amr Awadallah Founder, Chief Technical Officer aaa@cloudera.com, twitter: @awadallah Hadoop Past

More information

Developing Scalable Smart Grid Infrastructure to Enable Secure Transmission System Control

Developing Scalable Smart Grid Infrastructure to Enable Secure Transmission System Control Developing Scalable Smart Grid Infrastructure to Enable Secure Transmission System Control EP/K006487/1 UK PI: Prof Gareth Taylor (BU) China PI: Prof Yong-Hua Song (THU) Consortium UK Members: Brunel University

More information