An approach to monitoring, data analytics, and decision support for levee supervision

Size: px
Start display at page:

Download "An approach to monitoring, data analytics, and decision support for levee supervision"

Transcription

1 An approach to monitoring, data analytics, and decision support for levee supervision M. Bubak, B. Baliś, D. Harężlak, M. Kasztelnik, P. Nowakowski, T. Bartyński, T. Gubala, M. Malawski, M. Pawlik, B. Wilk AGH University of Science and Technology Department of Computer Science Krakow, Poland ISMOP: a computerized levee monitoring system Comprehensive research project on monitoring and assessment of levees which comprises: Construction of an artificial levee Design and construction of sensors for levee instrumentation Design and development of a sensor communication infrastructure Optimal collection and transmission of sensor data Levee modeling and simulation Comparison of simulated and real levee behavior Central System: software platforms for execution management, data management, visualization and decision support

2 ISMOP Consortium Department of Computer Science AGH Department of Hydrogeology and Engineering Geology AGH Department of Geoinformatics and Applied Computer Science AGH NeoSentio, Kraków Sweco Hydroprojekt Kraków in collaboration with the Czernichów Community Project leader: Prof. Krzysztof Zieliński ISMOP: target users Main goal: support the decision making for Flood protection Levee maintenance Target users: National and regional flood protection agencies Local authorities (levee maintenance)

3 ISMOP central system Visualization & decision support Execution management Data management Monitoring & decision support high level workflow Stand by mode Monitoring data collection (low frequency) Initialon line analysis (trends, deviations in sensor readings) Presentation of external info: weather prediction, flood wave prediction, etc. Threat assessment mode Increased frequency of sensor data collection Resourceintensive threat level evaluation Alert mode Prediction of levee behavior Notification of authorities

4 Visualization & decision support: selected challenges Interoperability with external systems (e.g. ISOK, regional flood protection agencies) Visualization of relevant information to effectively support the decision making process Adaptability to other domains (e.g. monitoring of communication infrastructure) Leveragingopen standards (OGC, INSPIRE) for data & metadata models Solution: research in progress Open domain agnostic design (metadata and public APIs design are crucial) Visualization & decision support system

5 Execution management: selected challenges Scale up to 100s 1000s kilometers of levees Monitored area divided into sections Managed by multiple instances of a Monitoring Application, dynamically deployed on demand Highly variable resource demands: from very low in standby mode to high in threat assessment mode Dynamic provisioning of resources from private or public clouds Autoscaling algorithms and policies Execution and Provisioning Platform (EXP) Composite App Execution Platform App model Provisioning platform Optimizer & Scheduler Scaling rules Data access (DAP) Events Enact HyperFlow Enactment Engine Execute App state Autoscaler Initial deployment Reconfigure app Provisioner Start/Stop/Reconfigure measurements Cloud Input data Executor Trigger app execution Monitoring

6 Data management: selected challenges Diverse data sets (spatial, time series, binary, metadata) and data usage patterns Multiple data stores and models to address diverse needs Data intensive processing Threat level evaluation scenario: up to 130 GB of data to search per 1km of a levee Big data infrastructure Map Reduce data search Example of data intensive analysis: threat level evaluation Real time sensor data Decision support Pre simulated scenarios (up to 130 GB of data / 1km of levee) Data intensive search Ranking of matching scenarios Global levee state & behavior prediction

7 Data Access & Analytics Platform (DAP) From sensor network communication infrastructure Sensor data collection Apache Flume propagation propagation DB interfaces Time Series DB GIS database HDFS Namenode Storage node 2nd NN Storage node Offline data analytics Storage node synchronization Online data access API Hive Pig Data access To application services Previous experience: UrbanFlood Online Flood Early Warning System Authorities Science SS S SS S Control Centre Control Centre Public

8 Conclusion ISMOP: comprehensive solution for levee monitoring & decision support Currently at the initial stage of research ISMOP Central System: visualization & decision support, execution management, data management

An approach to monitoring, data analytics, and decision support for levee supervision

An approach to monitoring, data analytics, and decision support for levee supervision An approach to monitoring, data analytics, and decision support for levee supervision M. Bubak, B. Baliś, D. Harężlak, M. Kasztelnik, P. Nowakowski, T. Bartyński, T. Gubala, M. Malawski, M. Pawlik, B.

More information

GridSpace2 Towards Science-as-a-Service Model

GridSpace2 Towards Science-as-a-Service Model Polish Roadmap towards Domain-Specific Infrastructure for Supporting Computational Science in European Research Area GridSpace2 Towards Science-as-a-Service Model Eryk Ciepiela, Bartosz Wilk, Daniel Harężlak,

More information

Big Data, Cloud Computing, Spatial Databases Steven Hagan Vice President Server Technologies

Big Data, Cloud Computing, Spatial Databases Steven Hagan Vice President Server Technologies Big Data, Cloud Computing, Spatial Databases Steven Hagan Vice President Server Technologies Big Data: Global Digital Data Growth Growing leaps and bounds by 40+% Year over Year! 2009 =.8 Zetabytes =.08

More information

DataNet Flexible Metadata Overlay over File Resources

DataNet Flexible Metadata Overlay over File Resources 1 DataNet Flexible Metadata Overlay over File Resources Daniel Harężlak 1, Marek Kasztelnik 1, Maciej Pawlik 1, Bartosz Wilk 1, Marian Bubak 1,2 1 ACC Cyfronet AGH, 2 AGH University of Science and Technology,

More information

Securing the Big Data Ecosystem

Securing the Big Data Ecosystem Securing the Big Data Ecosystem SESSION ID: STU-T07A Davi Ottenheimer Senior Director of Trust, EMC @daviottenheimer COWS NOT PETS ( ) (xx) /-------\/ / * ---- ^^ ^^ Systematic Treatment of Illness Easily

More information

A Platform for Collaborative e-science Applications. Marian Bubak ICS / Cyfronet AGH Krakow, PL bubak@agh.edu.pl

A Platform for Collaborative e-science Applications. Marian Bubak ICS / Cyfronet AGH Krakow, PL bubak@agh.edu.pl A Platform for Collaborative e-science Applications Marian Bubak ICS / Cyfronet AGH Krakow, PL bubak@agh.edu.pl Outline Motivation Idea of an experiment Virtual laboratory Examples of experiments Summary

More information

Experiments with Complex Scientific Applications on Hybrid Cloud Infrastructures

Experiments with Complex Scientific Applications on Hybrid Cloud Infrastructures Experiments with Complex Scientific Applications on Hybrid Cloud Infrastructures Maciej'Malawski 1,2,'Piotr'Nowakowski 1,'Tomasz'Gubała 1,'Marek'Kasztelnik 1,' Marian'Bubak 1,2,'Rafael'Ferreira'da'Silva

More information

AUTOMATIC PROXY GENERATION AND LOAD-BALANCING-BASED DYNAMIC CHOICE OF SERVICES

AUTOMATIC PROXY GENERATION AND LOAD-BALANCING-BASED DYNAMIC CHOICE OF SERVICES Computer Science 13 (3) 2012 http://dx.doi.org/10.7494/csci.2012.13.3.45 Jarosław Dąbrowski Sebastian Feduniak Bartosz Baliś Tomasz Bartyński Włodzimierz Funika AUTOMATIC PROXY GENERATION AND LOAD-BALANCING-BASED

More information

Mr. Apichon Witayangkurn apichon@iis.u-tokyo.ac.jp Department of Civil Engineering The University of Tokyo

Mr. Apichon Witayangkurn apichon@iis.u-tokyo.ac.jp Department of Civil Engineering The University of Tokyo Sensor Network Messaging Service Hive/Hadoop Mr. Apichon Witayangkurn apichon@iis.u-tokyo.ac.jp Department of Civil Engineering The University of Tokyo Contents 1 Introduction 2 What & Why Sensor Network

More information

Deploying Hadoop with Manager

Deploying Hadoop with Manager Deploying Hadoop with Manager SUSE Big Data Made Easier Peter Linnell / Sales Engineer plinnell@suse.com Alejandro Bonilla / Sales Engineer abonilla@suse.com 2 Hadoop Core Components 3 Typical Hadoop Distribution

More information

Distributed Cloud Environment for PL-Grid Applications

Distributed Cloud Environment for PL-Grid Applications Distributed Environment for PL-Grid Applications Piotr Nowakowski, Tomasz Bartyński, Tomasz Gubała, Daniel Harężlak, Marek Kasztelnik, J. Meizner, P. Suder, M. Bubak ACC CYFRONET AGH KUKDM 2015 Zakopane,

More information

JFlooder - Application performance testing with QoS assurance

JFlooder - Application performance testing with QoS assurance JFlooder - Application performance testing with QoS assurance Tomasz Duszka 1, Andrzej Gorecki 1, Jakub Janczak 1, Adam Nowaczyk 1 and Dominik Radziszowski 1 Institute of Computer Science, AGH UST, al.

More information

Savanna Hadoop on. OpenStack. Savanna Technical Lead

Savanna Hadoop on. OpenStack. Savanna Technical Lead Savanna Hadoop on OpenStack Sergey Lukjanov Savanna Technical Lead Mirantis, 2013 Agenda Savanna Overview Savanna Use Cases Roadmap & Current Status Architecture & Features Overview Hadoop vs. Virtualization

More information

Certified Big Data and Apache Hadoop Developer VS-1221

Certified Big Data and Apache Hadoop Developer VS-1221 Certified Big Data and Apache Hadoop Developer VS-1221 Certified Big Data and Apache Hadoop Developer Certification Code VS-1221 Vskills certification for Big Data and Apache Hadoop Developer Certification

More information

Lecture 32 Big Data. 1. Big Data problem 2. Why the excitement about big data 3. What is MapReduce 4. What is Hadoop 5. Get started with Hadoop

Lecture 32 Big Data. 1. Big Data problem 2. Why the excitement about big data 3. What is MapReduce 4. What is Hadoop 5. Get started with Hadoop Lecture 32 Big Data 1. Big Data problem 2. Why the excitement about big data 3. What is MapReduce 4. What is Hadoop 5. Get started with Hadoop 1 2 Big Data Problems Data explosion Data from users on social

More information

A standards-based open source processing chain for ocean modeling in the GEOSS Architecture Implementation Pilot Phase 8 (AIP-8)

A standards-based open source processing chain for ocean modeling in the GEOSS Architecture Implementation Pilot Phase 8 (AIP-8) NATO Science & Technology Organization Centre for Maritime Research and Experimentation (STO-CMRE) Viale San Bartolomeo, 400 19126 La Spezia, Italy A standards-based open source processing chain for ocean

More information

Control-M As an Application Management Platform

Control-M As an Application Management Platform Control-M As an Application Management Platform Zagreb Lipanj 2015 Vedran Vesel, Imaves Control-M stručnjak Legal Notice The information contained in this presentation is the confidential information of

More information

Ankush Cluster Manager - Hadoop2 Technology User Guide

Ankush Cluster Manager - Hadoop2 Technology User Guide Ankush Cluster Manager - Hadoop2 Technology User Guide Ankush User Manual 1.5 Ankush User s Guide for Hadoop2, Version 1.5 This manual, and the accompanying software and other documentation, is protected

More information

An Industrial Perspective on the Hadoop Ecosystem. Eldar Khalilov Pavel Valov

An Industrial Perspective on the Hadoop Ecosystem. Eldar Khalilov Pavel Valov An Industrial Perspective on the Hadoop Ecosystem Eldar Khalilov Pavel Valov agenda 03.12.2015 2 agenda Introduction 03.12.2015 2 agenda Introduction Research goals 03.12.2015 2 agenda Introduction Research

More information

Capitalize on Big Data for Competitive Advantage with Bedrock TM, an integrated Management Platform for Hadoop Data Lakes

Capitalize on Big Data for Competitive Advantage with Bedrock TM, an integrated Management Platform for Hadoop Data Lakes Capitalize on Big Data for Competitive Advantage with Bedrock TM, an integrated Management Platform for Hadoop Data Lakes Highly competitive enterprises are increasingly finding ways to maximize and accelerate

More information

Modern Data Architecture for Predictive Analytics

Modern Data Architecture for Predictive Analytics Modern Data Architecture for Predictive Analytics David Smith VP Marketing and Community - Revolution Analytics John Kreisa VP Strategic Marketing- Hortonworks Hortonworks Inc. 2013 Page 1 Your Presenters

More information

Data Lake In Action: Real-time, Closed Looped Analytics On Hadoop

Data Lake In Action: Real-time, Closed Looped Analytics On Hadoop 1 Data Lake In Action: Real-time, Closed Looped Analytics On Hadoop 2 Pivotal s Full Approach It s More Than Just Hadoop Pivotal Data Labs 3 Why Pivotal Exists First Movers Solve the Big Data Utility Gap

More information

HPC technology and future architecture

HPC technology and future architecture HPC technology and future architecture Visual Analysis for Extremely Large-Scale Scientific Computing KGT2 Internal Meeting INRIA France Benoit Lange benoit.lange@inria.fr Toàn Nguyên toan.nguyen@inria.fr

More information

Horizontal IoT Application Development using Semantic Web Technologies

Horizontal IoT Application Development using Semantic Web Technologies Horizontal IoT Application Development using Semantic Web Technologies Soumya Kanti Datta Research Engineer Communication Systems Department Email: Soumya-Kanti.Datta@eurecom.fr Roadmap Introduction Challenges

More information

Collaborative Big Data Analytics. Copyright 2012 EMC Corporation. All rights reserved.

Collaborative Big Data Analytics. Copyright 2012 EMC Corporation. All rights reserved. Collaborative Big Data Analytics 1 Big Data Is Less About Size, And More About Freedom TechCrunch!!!!!!!!! Total data: bigger than big data 451 Group Findings: Big Data Is More Extreme Than Volume Gartner!!!!!!!!!!!!!!!

More information

AN INTEGRATED SOLUTION FOR MANAGING EXPLORATION DATA

AN INTEGRATED SOLUTION FOR MANAGING EXPLORATION DATA www.wipro.com AN INTEGRATED SOLUTION FOR MANAGING EXPLORATION DATA Sandipan Chakraborti, Senior Architect ENU Table of Contents 03... Introduction 04... Lack of Holistic View 06... Figure 1: Conceptual

More information

NASA s Big Data Challenges in Climate Science

NASA s Big Data Challenges in Climate Science NASA s Big Data Challenges in Climate Science Tsengdar Lee, Ph.D. High-end Computing Program Manager NASA Headquarters Presented at IEEE Big Data 2014 Workshop October 29, 2014 1 2 7-km GEOS-5 Nature Run

More information

ITG Software Engineering

ITG Software Engineering Introduction to Cloudera Course ID: Page 1 Last Updated 12/15/2014 Introduction to Cloudera Course : This 5 day course introduces the student to the Hadoop architecture, file system, and the Hadoop Ecosystem.

More information

Pivotal HD Enterprise

Pivotal HD Enterprise PRODUCT DOCUMENTATION Pivotal HD Enterprise Version 1.1.1 Release Notes Rev: A02 2014 GoPivotal, Inc. Table of Contents 1 Welcome to Pivotal HD Enterprise 4 2 PHD Components 5 2.1 Core Apache Stack 5 2.2

More information

Data-intensive HPC: opportunities and challenges. Patrick Valduriez

Data-intensive HPC: opportunities and challenges. Patrick Valduriez Data-intensive HPC: opportunities and challenges Patrick Valduriez Big Data Landscape Multi-$billion market! Big data = Hadoop = MapReduce? No one-size-fits-all solution: SQL, NoSQL, MapReduce, No standard,

More information

Cloud services in PL-Grid and EGI Infrastructures

Cloud services in PL-Grid and EGI Infrastructures 1 Cloud services in PL-Grid and EGI Infrastructures J. Meizner, M. Radecki, M. Pawlik, T. Szepieniec ACK Cyfronet AGH Cracow Grid Workshop 2012, Kraków, 22.10.2012 Overview 2 Different types of Compute

More information

Investigating Hadoop for Large Spatiotemporal Processing Tasks

Investigating Hadoop for Large Spatiotemporal Processing Tasks Investigating Hadoop for Large Spatiotemporal Processing Tasks David Strohschein dstrohschein@cga.harvard.edu Stephen Mcdonald stephenmcdonald@cga.harvard.edu Benjamin Lewis blewis@cga.harvard.edu Weihe

More information

UrbanFlood Monitoring, rich browsers and cloud service technologies for an online EWS hosting platform Work Package 6 D6.4 version 1.0, 29 Nov.

UrbanFlood Monitoring, rich browsers and cloud service technologies for an online EWS hosting platform Work Package 6 D6.4 version 1.0, 29 Nov. UrbanFlood Monitoring, rich browsers and cloud service technologies for an online EWS hosting platform Work Package 6 D6.4 version 1.0, 29 Nov. 2010 November 2010 URBAN FLOOD A project funded under the

More information

CAPTURING & PROCESSING REAL-TIME DATA ON AWS

CAPTURING & PROCESSING REAL-TIME DATA ON AWS CAPTURING & PROCESSING REAL-TIME DATA ON AWS @ 2015 Amazon.com, Inc. and Its affiliates. All rights reserved. May not be copied, modified, or distributed in whole or in part without the express consent

More information

Data Governance in the Hadoop Data Lake. Kiran Kamreddy May 2015

Data Governance in the Hadoop Data Lake. Kiran Kamreddy May 2015 Data Governance in the Hadoop Data Lake Kiran Kamreddy May 2015 One Data Lake: Many Definitions A centralized repository of raw data into which many data-producing streams flow and from which downstream

More information

Future @ Cloud: Cloud Computing meets Smart Ecosystems

Future @ Cloud: Cloud Computing meets Smart Ecosystems Future @ Cloud: Cloud Computing meets Smart Ecosystems Joerg Doerr, Fraunhofer IESE, Kaiserslautern, Germany Joerg.Doerr@iese.fraunhofer.de Fraunhofer-Institute for Experimental Software Engineering (IESE)

More information

Simplifying Big Data Analytics: Unifying Batch and Stream Processing. John Fanelli,! VP Product! In-Memory Compute Summit! June 30, 2015!!

Simplifying Big Data Analytics: Unifying Batch and Stream Processing. John Fanelli,! VP Product! In-Memory Compute Summit! June 30, 2015!! Simplifying Big Data Analytics: Unifying Batch and Stream Processing John Fanelli,! VP Product! In-Memory Compute Summit! June 30, 2015!! Streaming Analy.cs S S S Scale- up Database Data And Compute Grid

More information

Leveraging Cloud-Based Mapping Solutions

Leveraging Cloud-Based Mapping Solutions Leveraging Cloud-Based Mapping Solutions GeoAlberta October 28, 2014 Laura Kerssens Safe Software Agenda To the Cloud Using Basic Services Cloud Applications Web Services Cloud-Hosted Databases Real-time

More information

Hadoop 101. Lars George. NoSQL- Ma4ers, Cologne April 26, 2013

Hadoop 101. Lars George. NoSQL- Ma4ers, Cologne April 26, 2013 Hadoop 101 Lars George NoSQL- Ma4ers, Cologne April 26, 2013 1 What s Ahead? Overview of Apache Hadoop (and related tools) What it is Why it s relevant How it works No prior experience needed Feel free

More information

Bringing Big Data to People

Bringing Big Data to People Bringing Big Data to People Microsoft s modern data platform SQL Server 2014 Analytics Platform System Microsoft Azure HDInsight Data Platform Everyone should have access to the data they need. Process

More information

Craig McWilliams Craig Burrell. Bringing Smarter, Safer Transport to NZ

Craig McWilliams Craig Burrell. Bringing Smarter, Safer Transport to NZ Craig McWilliams Craig Burrell Bringing Smarter, Safer Transport to NZ World Class Transport. Smarter, Stronger, Safer. Bringing Smarter Safer Transport to NZ Craig Burrell Infrastructure Advisory Director

More information

How To Scale A Server Farm

How To Scale A Server Farm Basics of Cloud Computing Lecture 3 Scaling Applications on the Cloud Satish Srirama Outline Scaling Information Systems Scaling Enterprise Applications in the Cloud Auto Scaling 25/02/2014 Satish Srirama

More information

Cloudera Administrator Training for Apache Hadoop

Cloudera Administrator Training for Apache Hadoop Cloudera Administrator Training for Apache Hadoop Duration: 4 Days Course Code: GK3901 Overview: In this hands-on course, you will be introduced to the basics of Hadoop, Hadoop Distributed File System

More information

E6893 Big Data Analytics Lecture 2: Big Data Analytics Platforms

E6893 Big Data Analytics Lecture 2: Big Data Analytics Platforms E6893 Big Data Analytics Lecture 2: Big Data Analytics Platforms Ching-Yung Lin, Ph.D. Adjunct Professor, Dept. of Electrical Engineering and Computer Science Mgr., Dept. of Network Science and Big Data

More information

Big Data Analytics Platform @ Nokia

Big Data Analytics Platform @ Nokia Big Data Analytics Platform @ Nokia 1 Selecting the Right Tool for the Right Workload Yekesa Kosuru Nokia Location & Commerce Strata + Hadoop World NY - Oct 25, 2012 Agenda Big Data Analytics Platform

More information

Development and Execution of Collaborative Application on the ViroLab Virtual Laboratory

Development and Execution of Collaborative Application on the ViroLab Virtual Laboratory Development and Execution of Collaborative Application on the ViroLab Virtual Laboratory Marek Kasztelnik 3, Tomasz Guba la 2,3, Maciej Malawski 1, and Marian Bubak 1,3 1 Institute of Computer Science

More information

SAP MRS Multiresource Scheduling Info session - 2013. Atul Wakankar May 2013

SAP MRS Multiresource Scheduling Info session - 2013. Atul Wakankar May 2013 SAP MRS Multiresource Scheduling Info session - 2013 Atul Wakankar May 2013 MRS: Foundation for the End-to-end Scheduling Process Resource Management for various Industries and different Scenarios Oil

More information

BIG DATA TRENDS AND TECHNOLOGIES

BIG DATA TRENDS AND TECHNOLOGIES BIG DATA TRENDS AND TECHNOLOGIES THE WORLD OF DATA IS CHANGING Cloud WHAT IS BIG DATA? Big data are datasets that grow so large that they become awkward to work with using onhand database management tools.

More information

Cloud Cruiser and Azure Public Rate Card API Integration

Cloud Cruiser and Azure Public Rate Card API Integration Cloud Cruiser and Azure Public Rate Card API Integration In this article: Introduction Azure Rate Card API Cloud Cruiser s Interface to Azure Rate Card API Import Data from the Azure Rate Card API Defining

More information

Agenda. Big Data & Hadoop ViPR HDFS Pivotal Big Data Suite & ViPR HDFS ViON Customer Feedback #EMCVIPR

Agenda. Big Data & Hadoop ViPR HDFS Pivotal Big Data Suite & ViPR HDFS ViON Customer Feedback #EMCVIPR 1 Agenda Big Data & Hadoop ViPR HDFS Pivotal Big Data Suite & ViPR HDFS ViON Customer Feedback 2 A World of Connected Devices Need a new data management architecture for Internet of Things 21% the % of

More information

PRIVACY AWARE ACCESS CONTROL FOR CLOUD-BASED DATA PLATFORMS

PRIVACY AWARE ACCESS CONTROL FOR CLOUD-BASED DATA PLATFORMS www.openi-ict.eu Open-Source, Web-Based, Framework for Integrating Applications with Social Media Services and Personal Cloudlets PRIVACY AWARE ACCESS CONTROL FOR CLOUD-BASED DATA PLATFORMS Open-Source,

More information

ArcGIS Enterprise Systems: Designing, Testing and Monitoring

ArcGIS Enterprise Systems: Designing, Testing and Monitoring Federal GIS Conference February 9 10, 2015 Washington, DC ArcGIS Enterprise Systems: Designing, Testing and Monitoring Jim VanOstenbridge, jvanostenbridge@esri.com> Martin Hamann, MHamann@esri.com Andrew

More information

Open Cloud System. (Integration of Eucalyptus, Hadoop and AppScale into deployment of University Private Cloud)

Open Cloud System. (Integration of Eucalyptus, Hadoop and AppScale into deployment of University Private Cloud) Open Cloud System (Integration of Eucalyptus, Hadoop and into deployment of University Private Cloud) Thinn Thu Naing University of Computer Studies, Yangon 25 th October 2011 Open Cloud System University

More information

Comprehensive Analytics on the Hortonworks Data Platform

Comprehensive Analytics on the Hortonworks Data Platform Comprehensive Analytics on the Hortonworks Data Platform We do Hadoop. Page 1 Page 2 Back to 2005 Page 3 Vertical Scaling Page 4 Vertical Scaling Page 5 Vertical Scaling Page 6 Horizontal Scaling Page

More information

From Big Data to Smart Data Thomas Hahn

From Big Data to Smart Data Thomas Hahn Siemens Future Forum @ HANNOVER MESSE 2014 From Big to Smart Hannover Messe 2014 The Evolution of Big Digital data ~ 1960 warehousing ~1986 ~1993 Big data analytics Mining ~2015 Stream processing Digital

More information

CLOUD MANAGED SERVICES FRAMEWORK E-BOOK

CLOUD MANAGED SERVICES FRAMEWORK E-BOOK CLOUD MANAGED SERVICES FRAMEWORK E-BOOK TABLE OF CONTENTS 1 Introduction 2 2 Operational Insight 3 3 Cloud Management Process Control 4 4 Infrastructure, Application & Data Security 5 5 Continuous Improvement

More information

Pro Apache Hadoop. Second Edition. Sameer Wadkar. Madhu Siddalingaiah

Pro Apache Hadoop. Second Edition. Sameer Wadkar. Madhu Siddalingaiah Pro Apache Hadoop Second Edition Sameer Wadkar Madhu Siddalingaiah Contents J About the Authors About the Technical Reviewer Acknowledgments Introduction xix xxi xxiii xxv Chapter 1: Motivation for Big

More information

PROPOSAL To Develop an Enterprise Scale Disease Modeling Web Portal For Ascel Bio Updated March 2015

PROPOSAL To Develop an Enterprise Scale Disease Modeling Web Portal For Ascel Bio Updated March 2015 Enterprise Scale Disease Modeling Web Portal PROPOSAL To Develop an Enterprise Scale Disease Modeling Web Portal For Ascel Bio Updated March 2015 i Last Updated: 5/8/2015 4:13 PM3/5/2015 10:00 AM Enterprise

More information

A Brief Introduction to Apache Tez

A Brief Introduction to Apache Tez A Brief Introduction to Apache Tez Introduction It is a fact that data is basically the new currency of the modern business world. Companies that effectively maximize the value of their data (extract value

More information

The Way to SOA Concept, Architectural Components and Organization

The Way to SOA Concept, Architectural Components and Organization The Way to SOA Concept, Architectural Components and Organization Eric Scholz Director Product Management Software AG Seite 1 Goals of business and IT Business Goals Increase business agility Support new

More information

Big Data Big Data/Data Analytics & Software Development

Big Data Big Data/Data Analytics & Software Development Big Data Big Data/Data Analytics & Software Development Danairat T. danairat@gmail.com, 081-559-1446 1 Agenda Big Data Overview Business Cases and Benefits Hadoop Technology Architecture Big Data Development

More information

Qsoft Inc www.qsoft-inc.com

Qsoft Inc www.qsoft-inc.com Big Data & Hadoop Qsoft Inc www.qsoft-inc.com Course Topics 1 2 3 4 5 6 Week 1: Introduction to Big Data, Hadoop Architecture and HDFS Week 2: Setting up Hadoop Cluster Week 3: MapReduce Part 1 Week 4:

More information

BIG DATA AND ANALYTICS

BIG DATA AND ANALYTICS BIG DATA AND ANALYTICS Björn Bjurling, bgb@sics.se Daniel Gillblad, dgi@sics.se Anders Holst, aho@sics.se Swedish Institute of Computer Science AGENDA What is big data and analytics? and why one must bother

More information

Hadoop Development & BI- 0 to 100

Hadoop Development & BI- 0 to 100 Development Master the Data Analysis tools like Pig and hive Data Science Hadoop Development & BI- 0 to 100 Build a recommendation engine Hadoop Development - 0 to 100 HADOOP SCHOOL OF TRAINING Basics

More information

ESS event: Big Data in Official Statistics. Antonino Virgillito, Istat

ESS event: Big Data in Official Statistics. Antonino Virgillito, Istat ESS event: Big Data in Official Statistics Antonino Virgillito, Istat v erbi v is 1 About me Head of Unit Web and BI Technologies, IT Directorate of Istat Project manager and technical coordinator of Web

More information

INTERNATIONAL JOURNAL OF PURE AND APPLIED RESEARCH IN ENGINEERING AND TECHNOLOGY

INTERNATIONAL JOURNAL OF PURE AND APPLIED RESEARCH IN ENGINEERING AND TECHNOLOGY INTERNATIONAL JOURNAL OF PURE AND APPLIED RESEARCH IN ENGINEERING AND TECHNOLOGY A PATH FOR HORIZING YOUR INNOVATIVE WORK BIG DATA HOLDS BIG PROMISE FOR SECURITY NEHA S. PAWAR, PROF. S. P. AKARTE Computer

More information

Factories of the Future Horizon 2020: LEIT ICT WP2016-17. FoF-11-2016: Digital Automation

Factories of the Future Horizon 2020: LEIT ICT WP2016-17. FoF-11-2016: Digital Automation Factories of the Future Horizon 2020: LEIT ICT WP2016-17 FoF-11-2016: Digital Automation Directorate General for Communication Networks, Content and Technology FoF PPP in Work Programme 2016-2017 FoF-11-2016:

More information

ENABLING GLOBAL HADOOP WITH EMC ELASTIC CLOUD STORAGE

ENABLING GLOBAL HADOOP WITH EMC ELASTIC CLOUD STORAGE ENABLING GLOBAL HADOOP WITH EMC ELASTIC CLOUD STORAGE Hadoop Storage-as-a-Service ABSTRACT This White Paper illustrates how EMC Elastic Cloud Storage (ECS ) can be used to streamline the Hadoop data analytics

More information

Cloud-based Infrastructures. Serving INSPIRE needs

Cloud-based Infrastructures. Serving INSPIRE needs Cloud-based Infrastructures Serving INSPIRE needs INSPIRE Conference 2014 Workshop Sessions Benoit BAURENS, AKKA Technologies (F) Claudio LUCCHESE, CNR (I) June 16th, 2014 This content by the InGeoCloudS

More information

Angelos Tzotsos IMIS Athena, Scientific & Technical Manager OSGeo Charter Member Copernicus Big Data Workshop, March 2014

Angelos Tzotsos IMIS Athena, Scientific & Technical Manager OSGeo Charter Member Copernicus Big Data Workshop, March 2014 Angelos Tzotsos IMIS Athena, Scientific & Technical Manager OSGeo Charter Member Copernicus Big Data Workshop, March 2014 Genesis of the project Consortium established on active research, commercial, and

More information

Azure Data Lake Analytics

Azure Data Lake Analytics Azure Data Lake Analytics Compose and orchestrate data services at scale Fully managed service to support orchestration of data movement and processing Connect to relational or non-relational data

More information

Open source Google-style large scale data analysis with Hadoop

Open source Google-style large scale data analysis with Hadoop Open source Google-style large scale data analysis with Hadoop Ioannis Konstantinou Email: ikons@cslab.ece.ntua.gr Web: http://www.cslab.ntua.gr/~ikons Computing Systems Laboratory School of Electrical

More information

Case Study : 3 different hadoop cluster deployments

Case Study : 3 different hadoop cluster deployments Case Study : 3 different hadoop cluster deployments Lee moon soo moon@nflabs.com HDFS as a Storage Last 4 years, our HDFS clusters, stored Customer 1500 TB+ data safely served 375,000 TB+ data to customer

More information

Hortonworks CISC Innovation day

Hortonworks CISC Innovation day Hortonworks CISC Innovation day Simon gregory sgregory@hortonworks.com Here was the ask Hortonworks' data reposition - how this works and the types of data you work with. 1: Data Types & Value. What have

More information

Chase Wu New Jersey Ins0tute of Technology

Chase Wu New Jersey Ins0tute of Technology CS 698: Special Topics in Big Data Chapter 4. Big Data Analytics Platforms Chase Wu New Jersey Ins0tute of Technology Some of the slides have been provided through the courtesy of Dr. Ching-Yung Lin at

More information

ANALYTICS IN BIG DATA ERA

ANALYTICS IN BIG DATA ERA ANALYTICS IN BIG DATA ERA ANALYTICS TECHNOLOGY AND ARCHITECTURE TO MANAGE VELOCITY AND VARIETY, DISCOVER RELATIONSHIPS AND CLASSIFY HUGE AMOUNT OF DATA MAURIZIO SALUSTI SAS Copyr i g ht 2012, SAS Ins titut

More information

Enabling Access to Environmental. on the Web

Enabling Access to Environmental. on the Web ENVISION Overview Enabling Access to Environmental Models, Data, and Services on the Web Dumitru Roman dumitru.roman@sintef.no Bucharest, November 13, 2012 http://www.envision-project.eu/ ENVISION Motivation

More information

Deploying Predictive Analytics Solutions in the Rail Industry and Seeing a Return on Investment

Deploying Predictive Analytics Solutions in the Rail Industry and Seeing a Return on Investment Robert Morris, Ph.D. Co- Founder & Chief Science Officer Predikto, Inc. Deploying Predictive Analytics Solutions in the Rail Industry and Seeing a Return on Investment SHOWING BUSINESS VALUE FROM DEPLOYED

More information

COMP9321 Web Application Engineering

COMP9321 Web Application Engineering COMP9321 Web Application Engineering Semester 2, 2015 Dr. Amin Beheshti Service Oriented Computing Group, CSE, UNSW Australia Week 11 (Part II) http://webapps.cse.unsw.edu.au/webcms2/course/index.php?cid=2411

More information

Real-time Monitoring Platform to Improve the Drinking Water Network Efficiency

Real-time Monitoring Platform to Improve the Drinking Water Network Efficiency Real-time Monitoring Platform to Improve the Drinking Water Network Efficiency SUMMARY Recent advances in real-time water sensing technology have enabled new opportunities for improved assessment and management

More information

Big Data: Making Sense of it all!

Big Data: Making Sense of it all! Big Data: Making Sense of it all! Jamie Engesser E-mail : jamie@hortonworks.com Page 1 Data Driven Business? Facts not Intuition! Data driven decisions are better decisions its as simple as that. Using

More information

Application of Petri Nets to Evaluation of Grid Applications Efficiency

Application of Petri Nets to Evaluation of Grid Applications Efficiency Application of Petri Nets to Evaluation of Grid Applications Efficiency Wojciech Rząsa 1, Marian Bubak 2,3 (1) Rzeszow University of Technology, Poland (2) Science AGH, Krakow, Poland (3) ACC Cyfronet,

More information

Hadoop Introduction. Olivier Renault Solution Engineer - Hortonworks

Hadoop Introduction. Olivier Renault Solution Engineer - Hortonworks Hadoop Introduction Olivier Renault Solution Engineer - Hortonworks Hortonworks A Brief History of Apache Hadoop Apache Project Established Yahoo! begins to Operate at scale Hortonworks Data Platform 2013

More information

HDP Hadoop From concept to deployment.

HDP Hadoop From concept to deployment. HDP Hadoop From concept to deployment. Ankur Gupta Senior Solutions Engineer Rackspace: Page 41 27 th Jan 2015 Where are you in your Hadoop Journey? A. Researching our options B. Currently evaluating some

More information

Data processing goes big

Data processing goes big Test report: Integration Big Data Edition Data processing goes big Dr. Götz Güttich Integration is a powerful set of tools to access, transform, move and synchronize data. With more than 450 connectors,

More information

Big Data Applications in Disaster Management

Big Data Applications in Disaster Management Taiwan 2012 Bridging Big Data Infrastructure Workshop Case Studies and Needs: Disaster Mitigation Whey-Fone Tsai National Center for High-performance Computing National Applied Research Laboratories Hsinchu,

More information

XpoLog Competitive Comparison Sheet

XpoLog Competitive Comparison Sheet XpoLog Competitive Comparison Sheet New frontier in big log data analysis and application intelligence Technical white paper May 2015 XpoLog, a data analysis and management platform for applications' IT

More information

Apache Hadoop: The Big Data Refinery

Apache Hadoop: The Big Data Refinery Architecting the Future of Big Data Whitepaper Apache Hadoop: The Big Data Refinery Introduction Big data has become an extremely popular term, due to the well-documented explosion in the amount of data

More information

Information Builders Mission & Value Proposition

Information Builders Mission & Value Proposition Value 10/06/2015 2015 MapR Technologies 2015 MapR Technologies 1 Information Builders Mission & Value Proposition Economies of Scale & Increasing Returns (Note: Not to be confused with diminishing returns

More information

A common interface for multi-rule-engine distributed systems

A common interface for multi-rule-engine distributed systems A common interface for multi-rule-engine distributed systems Pierre de Leusse, Bartosz Kwolek and Krzysztof Zieliński Distributed System Research Group, AGH University of Science and Technology Krakow,

More information

Oracle s Big Data solutions. Roger Wullschleger. <Insert Picture Here>

Oracle s Big Data solutions. Roger Wullschleger. <Insert Picture Here> s Big Data solutions Roger Wullschleger DBTA Workshop on Big Data, Cloud Data Management and NoSQL 10. October 2012, Stade de Suisse, Berne 1 The following is intended to outline

More information

Big Data and Analytics: A Conceptual Overview. Mike Park Erik Hoel

Big Data and Analytics: A Conceptual Overview. Mike Park Erik Hoel Big Data and Analytics: A Conceptual Overview Mike Park Erik Hoel In this technical workshop This presentation is for anyone that uses ArcGIS and is interested in analyzing large amounts of data We will

More information

Agile Infrastructure Update Monitoring

Agile Infrastructure Update Monitoring Agile Infrastructure Update Monitoring Pedro Andrade IT/GT 6 th July 2012 IT Technical Forum CERN IT Department CH-1211 Genève 23 Switzerland www.cern.ch/it Overview Introduction Motivation, Challenge,

More information

Big Data Spatial Analytics An Introduction

Big Data Spatial Analytics An Introduction 2013 Esri International User Conference July 8 12, 2013 San Diego, California Technical Workshop Big Data Spatial Analytics An Introduction Marwa Mabrouk Mansour Raad Esri iu UC2013. Technical Workshop

More information

Data Analyst Program- 0 to 100

Data Analyst Program- 0 to 100 Development Data Analyst Program- 0 to 100 Master the Data Analysis tools like Pig and hive Data Science Build a recommendation engine 1 Data Analyst Program- 0 to 100 HADOOP SCHOOL OF TRAINING Basics

More information

Providing drivers with actionable intelligence can minimize accidents, reduce driver claims and increase your bottom line. Equip motorists with the

Providing drivers with actionable intelligence can minimize accidents, reduce driver claims and increase your bottom line. Equip motorists with the Providing drivers with actionable intelligence can minimize accidents, reduce driver claims and increase your bottom line. Equip motorists with the ability to make informed decisions based on reliable,

More information

Big Data Course Highlights

Big Data Course Highlights Big Data Course Highlights The Big Data course will start with the basics of Linux which are required to get started with Big Data and then slowly progress from some of the basics of Hadoop/Big Data (like

More information

Adobe s Story of Integrating Hadoop and SAP HANA with SAP Data Services

Adobe s Story of Integrating Hadoop and SAP HANA with SAP Data Services Orange County Convention Center Orlando, Florida June 3-5, 2014 Adobe s Story of Integrating Hadoop and SAP HANA with SAP Data Services Kevin Davis, Senior Data Warehouse Engineer, Adobe Hemant Puranik,

More information

International Journal of Advancements in Research & Technology, Volume 3, Issue 2, February-2014 10 ISSN 2278-7763

International Journal of Advancements in Research & Technology, Volume 3, Issue 2, February-2014 10 ISSN 2278-7763 International Journal of Advancements in Research & Technology, Volume 3, Issue 2, February-2014 10 A Discussion on Testing Hadoop Applications Sevuga Perumal Chidambaram ABSTRACT The purpose of analysing

More information

TECHNOLOGY WHITE PAPER Jun 2012

TECHNOLOGY WHITE PAPER Jun 2012 TECHNOLOGY WHITE PAPER Jun 2012 Technology Stack C# Windows Server 2008 PHP Amazon Web Services (AWS) Route 53 Elastic Load Balancing (ELB) Elastic Compute Cloud (EC2) Amazon RDS Amazon S3 Elasticache

More information