DICE: Quality- Driven Development of Data- Intensive Cloud ApplicaPons

Size: px
Start display at page:

Download "DICE: Quality- Driven Development of Data- Intensive Cloud ApplicaPons"

Transcription

1 DICE: Quality- Driven Development of Data- Intensive Cloud ApplicaPons G. Casale, D. Ardagna, M. Artac, F. Barbier, E. Di Ni6o, A. Henry, G. Iuhasz, C. Joubert, J. Merseguer, V. I. Munteanu, J. F. Pérez, D. Petcu, M. Rossi, C. Sheridan, I. Spais, D. Vladušic DICE Horizon 2020 Project Grant Agreement no h>p://www.dice- h2020.eu Funded by the Horizon 2020 Framework Programme of the European Union

2 Overview and goals o MDE oyen features quality assurance (QA) techniques for developers o How should quality- aware MDE support data- intensive soyware systems? o ExisPng models and QA techniques largely ignore properpes of data o Characterize the behavior of new technologies o DICE: a quality- aware MDE methodology inspired by DevOps for data- intensive cloud applicapons 2

3 MoPvaPon o SoYware market rapidly shiying to Big Data 32% compound annual growth rate in EU through % Big data projects are successful [CapGemini 2015] o European call for soyware quality assurance (QA) ISTAG: call to define environments for understanding the consequences of different implementanon alternanves (e.g. quality, robustness, performance, maintenance, evolvability,...) o QA evolving too slowly compared to the trends in soyware development (Big data, Cloud, DevOps...) 3

4 DataInc example o DataInc is a small soyware vendor selling cloud- based environmental soyware o DICEnv, a warning system for floods in rural regions o monitoring local environmental condipons o fetching precipitapons data from satellite image stream o DICEnv exploits Big Data technologies and cloud capacity for online water simulapons and MapReduce for batch processing of historical data o DICEnv is a cripcal system: o is expected to remain up 24/7 o should quickly ramp up data intake rates, as well as memory and compute capacipes, to update more frequently the hazard management control room 4

5 DataInc example o The contract requires delivering an inipal version of DICEnv within 3 months serving a small area, increasing coverage on a monthly basis o Challenges: o How to implement a complex cloud applicapon in such a short Pme? o How to sapsfy all the quality requirements? o What architecture should be adopted? 5

6 Quality- Aware MDE Today Plalorm- Indep. Domain s AnalyPcal s MARTE Architecture QA s Plalorm DescripPon Plalorm- Specific Cost- Quality s Code generapon C++ Java C# 6

7 Quality- Aware MDE Today MARTE Plalorm- Indep. Architecture Domain s Issues PIM layer: stanc characterisncs of data dynamic characterisncs of data data dependencies DICEnv modeling issues: Plalorm DescripPon Code generapon C++ Plalorm- Specific Java C# individual dependencies between components and data streams relaponships between compute and memory requirements lack of an explicit annotapon for data characterispcs 7

8 Quality- Aware MDE Today MARTE Plalorm- Indep. Architecture Domain s Issues at PSM layer: heterogeneity of Big Data technologies automanc translanon of PSM models into deployment plans Plalorm DescripPon Code generapon C++ Plalorm- Specific Java C# QA tools limitapons: contenpon at processing resources with limited features for memory consumppon fork and joining are complex to be described analypcally preserving tractability 8

9 An HolisPc Approach: DICE DICE DICE IDE Plalorm DescripPon MARTE Deployment & ConPnuous IntegraPon Plalorm- Indep. Architecture Plalorm- Specific Domain s Data Awareness ConPnuous ValidaPon QA s Big Data ConPnuous Monitoring 9

10 Embracing DevOps o SoYware development process is evolving Developer: I want to change my code Operator: I want systems to be stable o...but code changes are the cause of most instabilipes! o DevOps closes the gap between Dev and Ops Lean release cycles with automated tests and tools Deep modelling of systems is the key to automapon Agile Development DevOps Business Dev Ops 10

11 Embracing DevOps o QA must become lean as well ConPnuous quality checks and model versioning o ling of the operapons Dev needs awareness of infrastructure and costs o ConPnuous feedback Forward and backward model synchronisapon Tracking of self- adaptapon events (e.g. auto- scaling) o Big data coming from conpnuous monitoring QA has its own Big data, use machine learning? 11

12 Benefits o Tackling skill shortage and steep learning curves Data- aware methods, models, and OSS tools o Shorter Pme to market for Big Data applicapons Cost reducpon, without sacrificing product quality o Decrease development and tespng costs Select oppmal architectures that can meet SLAs o Reduce number and severity of quality incidents IteraPve refinement of applicapon design 12

13 DICE Plalorm Independent (DPIM) 13

14 DICE Plalorm and Technology Specific (DTSM) 14

15 DICE Plalorm, Technology and Deployment Specific (DDSM) 15

16 DICE Profile: PIM Level o FuncPonal approach to data to be expanded o Data dependencies graph relaponships between data, archives and streams o QA focuses on quanptapve aspects of data o StaPc characterispcs of data volumes, value, storage locapon, replicapon pa>ern, consistency policies, data access costs, known schedules of data transfers, data access control / privacy,... o Dynamic characterispcs of data cache hit/miss probabilipes, read/write/update rates, burspness,... 16

17 DICE Profile: PSM Level o Need for technology- specific abstracpons Hadoop: Number of mappers and reducers,... In- memory DBs: Peak memory and variable threading Streaming: merge/split/operators, networking,... Storage: Supported operapons, cost/byte,... NoSQL: Consistency policies,... o GeneraPon of deployment plan Proposed Chef + TOSCA extension o Interest is both on private and public clouds 17

18 DICE QA: Quality Dimensions o Reliability o Efficiency Availability Fault- tolerance Performance Time behaviour Costs o Safety Risk of harm Privacy & data protecpon 18

19 DICE QA: Tools o Discrete- event simulanon: assess reliability and efficiency in Big Data applicapons, accounpng for stochaspc evolupon of the environment o stochaspc Petri nets or queueing networks, rely on simulapon o Formal verificanon tools: assess safety risks in Big Data applicapons, e.g. find design flaws causing order and Pming violapons in message and state sequences o temporal logic formulae and bounded model checking, sapsfiability modulo theories solvers o quanpfier- eliminapon techniques to extend temporal logic- based verificapon 19

20 DICE QA: Tools o Architecture opnmizanon tool: find architectural improvements to oppmise costs and quality o decomposipon- based analysis approach o resort to fluid approximapon of stochaspc models o Feedback analysis: automated extracpon from the monitored data of key parameters required to define simulapon and verificapon models o extract model parameters through log mining and stapspcal espmapon methods o breakdown resource consumppon into its atomic components on the end- to- end path of requests 20

21 DICE Project h>p://www.dice- h2020.eu o Horizon 2020 Research & InnovaPon AcPon Quality- Aware Development for Big Data applicapons Feb Jan 2019, 4M Euros budget 9 partners (Academia & SMEs), 7 EU countries 21

DICE: Developing Data-Intensive Cloud Applications with Iterative Quality Enhancements

DICE: Developing Data-Intensive Cloud Applications with Iterative Quality Enhancements DICE: Developing Data-Intensive Cloud Applications with Iterative Quality Enhancements Giuliano Casale Imperial College London Project Coordinator DICE Horizon 2020 Project Grant Agreement no. 644869 http://www.dice-h2020.eu

More information

BIG DATA IN THE CLOUD : CHALLENGES AND OPPORTUNITIES MARY- JANE SULE & PROF. MAOZHEN LI BRUNEL UNIVERSITY, LONDON

BIG DATA IN THE CLOUD : CHALLENGES AND OPPORTUNITIES MARY- JANE SULE & PROF. MAOZHEN LI BRUNEL UNIVERSITY, LONDON BIG DATA IN THE CLOUD : CHALLENGES AND OPPORTUNITIES MARY- JANE SULE & PROF. MAOZHEN LI BRUNEL UNIVERSITY, LONDON Overview * Introduction * Multiple faces of Big Data * Challenges of Big Data * Cloud Computing

More information

Augmented Search for Web Applications. New frontier in big log data analysis and application intelligence

Augmented Search for Web Applications. New frontier in big log data analysis and application intelligence Augmented Search for Web Applications New frontier in big log data analysis and application intelligence Business white paper May 2015 Web applications are the most common business applications today.

More information

Augmented Search for IT Data Analytics. New frontier in big log data analysis and application intelligence

Augmented Search for IT Data Analytics. New frontier in big log data analysis and application intelligence Augmented Search for IT Data Analytics New frontier in big log data analysis and application intelligence Business white paper May 2015 IT data is a general name to log data, IT metrics, application data,

More information

Testing Big data is one of the biggest

Testing Big data is one of the biggest Infosys Labs Briefings VOL 11 NO 1 2013 Big Data: Testing Approach to Overcome Quality Challenges By Mahesh Gudipati, Shanthi Rao, Naju D. Mohan and Naveen Kumar Gajja Validate data quality by employing

More information

Getting Started & Successful with Big Data

Getting Started & Successful with Big Data Getting Started & Successful with Big Data @Pentaho #BigDataWebSeries 2013, Pentaho. All Rights Reserved. pentaho.com. Worldwide +1 (866) 660-7555 Your Hosts Today Davy Nys VP EMEA & APAC Pentaho Paul

More information

HPC ABDS: The Case for an Integrating Apache Big Data Stack

HPC ABDS: The Case for an Integrating Apache Big Data Stack HPC ABDS: The Case for an Integrating Apache Big Data Stack with HPC 1st JTC 1 SGBD Meeting SDSC San Diego March 19 2014 Judy Qiu Shantenu Jha (Rutgers) Geoffrey Fox gcf@indiana.edu http://www.infomall.org

More information

Why continuous delivery needs devops, and why devops needs infrastructure-as-code. Sriram Narayan @sriramnarayan 25-Oct-2012

Why continuous delivery needs devops, and why devops needs infrastructure-as-code. Sriram Narayan @sriramnarayan 25-Oct-2012 Why continuous delivery needs devops, and why devops needs infrastructure-as-code Sriram Narayan @sriramnarayan 25-Oct-2012 about me Part of ThoughtWorks Studios Go team Have consulted as Tech Principal,

More information

Reference Architecture, Requirements, Gaps, Roles

Reference Architecture, Requirements, Gaps, Roles Reference Architecture, Requirements, Gaps, Roles The contents of this document are an excerpt from the brainstorming document M0014. The purpose is to show how a detailed Big Data Reference Architecture

More information

INTRODUCING SOFTWARE PERFORMANCE ANTIPATTERNS IN CLOUD COMPUTING ENVIRONMENTS: DOES IT HELP OR HURT? [VISION PAPER]

INTRODUCING SOFTWARE PERFORMANCE ANTIPATTERNS IN CLOUD COMPUTING ENVIRONMENTS: DOES IT HELP OR HURT? [VISION PAPER] INTRODUCING SOFTWARE PERFORMANCE ANTIPATTERNS IN CLOUD COMPUTING ENVIRONMENTS: DOES IT HELP OR HURT? [VISION PAPER] Catia Trubiani Gran Sasso Science Institute (L Aquila, Italy) http://cs.gssi.infn.it/catia.trubiani

More information

DevOps for the Mainframe

DevOps for the Mainframe DevOps for the Mainframe Rosalind Radcliffe IBM Distinguished Engineer, Enterprise Modernization Solution Architect rradclif@us.ibm.com 1 Please note IBM s statements regarding its plans, directions, and

More information

Augmented Search for Software Testing

Augmented Search for Software Testing Augmented Search for Software Testing For Testers, Developers, and QA Managers New frontier in big log data analysis and application intelligence Business white paper May 2015 During software testing cycles,

More information

Cisco Network Services Orchestrator enabled by Tail-f Multi-Vendor Service Automation & Network Programmability Stefan Vallin, Ph D

Cisco Network Services Orchestrator enabled by Tail-f Multi-Vendor Service Automation & Network Programmability Stefan Vallin, Ph D Cisco Network Services Orchestrator enabled by Tail-f Multi-Vendor Service Automation & Network Programmability Stefan Vallin, Ph D Product Manager NSO 10 June 2015 Key Market Trends and Challenges Changing

More information

NoSQL for SQL Professionals William McKnight

NoSQL for SQL Professionals William McKnight NoSQL for SQL Professionals William McKnight Session Code BD03 About your Speaker, William McKnight President, McKnight Consulting Group Frequent keynote speaker and trainer internationally Consulted to

More information

Hadoop and Map-Reduce. Swati Gore

Hadoop and Map-Reduce. Swati Gore Hadoop and Map-Reduce Swati Gore Contents Why Hadoop? Hadoop Overview Hadoop Architecture Working Description Fault Tolerance Limitations Why Map-Reduce not MPI Distributed sort Why Hadoop? Existing Data

More information

Hadoop Architecture. Part 1

Hadoop Architecture. Part 1 Hadoop Architecture Part 1 Node, Rack and Cluster: A node is simply a computer, typically non-enterprise, commodity hardware for nodes that contain data. Consider we have Node 1.Then we can add more nodes,

More information

Improving Agility of Cloud Ecosystems with MODAClouds Introduction and objectives for the second year

Improving Agility of Cloud Ecosystems with MODAClouds Introduction and objectives for the second year Improving Agility of Cloud Ecosystems with MODAClouds Introduction and objectives for the second year Elisabetta Di Nitto Politecnico di Milano elisabetta.dinitto@polimi.it MODAClouds () 2 MODAClouds objectives

More information

Securing the Cloud with IBM Security Systems. IBM Security Systems. 2012 IBM Corporation. 2012 2012 IBM IBM Corporation Corporation

Securing the Cloud with IBM Security Systems. IBM Security Systems. 2012 IBM Corporation. 2012 2012 IBM IBM Corporation Corporation Securing the Cloud with IBM Security Systems 1 2012 2012 IBM IBM Corporation Corporation IBM Point of View: Cloud can be made secure for business As with most new technology paradigms, security concerns

More information

Ganzheitliches Datenmanagement

Ganzheitliches Datenmanagement Ganzheitliches Datenmanagement für Hadoop Michael Kohs, Senior Sales Consultant @mikchaos The Problem with Big Data Projects in 2016 Relational, Mainframe Documents and Emails Data Modeler Data Scientist

More information

Towards a Thriving Data Economy: Open Data, Big Data, and Data Ecosystems

Towards a Thriving Data Economy: Open Data, Big Data, and Data Ecosystems Towards a Thriving Data Economy: Open Data, Big Data, and Data Ecosystems Volker Markl volker.markl@tu-berlin.de dima.tu-berlin.de dfki.de/web/research/iam/ bbdc.berlin Based on my 2014 Vision Paper On

More information

Collaborative DevOps Learn the magic of Continuous Delivery. Saurabh Agarwal Product Engineering, DevOps Solutions agarwasa@us.ibm.

Collaborative DevOps Learn the magic of Continuous Delivery. Saurabh Agarwal Product Engineering, DevOps Solutions agarwasa@us.ibm. Collaborative DevOps Learn the magic of Continuous Delivery Saurabh Agarwal Product Engineering, DevOps Solutions agarwasa@us.ibm.com Please note IBM s statements regarding its plans, directions, and intent

More information

SigMo Platform based approach for automation of workflows in large scale IT-Landscape. Tarmo Ploom 2/21/2014

SigMo Platform based approach for automation of workflows in large scale IT-Landscape. Tarmo Ploom 2/21/2014 SigMo Platform based approach for automation of workflows in large scale IT-Landscape 2/21/2014 Agenda Starting situation Problem Solution variants Friction of project based approach Platform approach

More information

Automated Model-Based Testing of Embedded Real-Time Systems

Automated Model-Based Testing of Embedded Real-Time Systems Automated Model-Based Testing of Embedded Real-Time Systems Jan Peleska jp@tzi.de University of Bremen Bieleschweig Workshop 7 2006-05-05 Outline Technologie-Zentrum Informatik Objectives Basic concepts

More information

SESSION 703 Wednesday, November 4, 9:00am - 10:00am Track: Advancing ITSM

SESSION 703 Wednesday, November 4, 9:00am - 10:00am Track: Advancing ITSM SESSION 703 Wednesday, November 4, 9:00am - 10:00am Track: Advancing ITSM Optimizing ITSM for Cloud Computing Reginald Lo Director, Accelerate Management, VMware rlo@vmware.com Session Description Organizations

More information

Data Management in the Cloud

Data Management in the Cloud Data Management in the Cloud Ryan Stern stern@cs.colostate.edu : Advanced Topics in Distributed Systems Department of Computer Science Colorado State University Outline Today Microsoft Cloud SQL Server

More information

Deploying Hadoop with Manager

Deploying Hadoop with Manager Deploying Hadoop with Manager SUSE Big Data Made Easier Peter Linnell / Sales Engineer plinnell@suse.com Alejandro Bonilla / Sales Engineer abonilla@suse.com 2 Hadoop Core Components 3 Typical Hadoop Distribution

More information

Business Intelligence meets Big Data: An Overview on Security and Privacy

Business Intelligence meets Big Data: An Overview on Security and Privacy Business Intelligence meets Big Data: An Overview on Security and Privacy Claudio A. Ardagna Ernesto Damiani Dipartimento di Informatica - Università degli Studi di Milano NSF Workshop on Big Data Security

More information

Standards for Big Data in the Cloud

Standards for Big Data in the Cloud Standards for Big Data in the Cloud James Kobielus Chair, CSCC Big Data Working Group Big Data Evangelist, Senior Program Director, Product Marketing, Big Data Analytics, IBM jgkobiel@us.ibm.com 15 October

More information

Big Data Are You Ready? Thomas Kyte http://asktom.oracle.com

Big Data Are You Ready? Thomas Kyte http://asktom.oracle.com Big Data Are You Ready? Thomas Kyte http://asktom.oracle.com The following is intended to outline our general product direction. It is intended for information purposes only, and may not be incorporated

More information

Master of Science Service Oriented Architecture for Enterprise. Courses description

Master of Science Service Oriented Architecture for Enterprise. Courses description Master of Science Service Oriented Architecture for Enterprise Courses description SCADA and PLC networks The course aims to consolidate and transfer of extensive knowledge regarding the architecture,

More information

Workprogramme 2013 objective 1.2. Sandro D Elia. Software & Service Architectures and Infrastructures

Workprogramme 2013 objective 1.2. Sandro D Elia. Software & Service Architectures and Infrastructures Workprogramme 2013 objective 1.2 Sandro D Elia Software & Service Architectures and Infrastructures Target Outcomes Delivering services in an effective, efficient and reliable manner across the future

More information

ATA DRIVEN GLOBAL VISION CLOUD PLATFORM STRATEG N POWERFUL RELEVANT PERFORMANCE SOLUTION CLO IRTUAL BIG DATA SOLUTION ROI FLEXIBLE DATA DRIVEN V

ATA DRIVEN GLOBAL VISION CLOUD PLATFORM STRATEG N POWERFUL RELEVANT PERFORMANCE SOLUTION CLO IRTUAL BIG DATA SOLUTION ROI FLEXIBLE DATA DRIVEN V ATA DRIVEN GLOBAL VISION CLOUD PLATFORM STRATEG N POWERFUL RELEVANT PERFORMANCE SOLUTION CLO IRTUAL BIG DATA SOLUTION ROI FLEXIBLE DATA DRIVEN V WHITE PAPER Create the Data Center of the Future Accelerate

More information

NE-10750A Monitoring and Operating a Private Cloud with System Center 2012

NE-10750A Monitoring and Operating a Private Cloud with System Center 2012 NE-10750A and Operating a with System Center 2012 Summary Duration Vendor Audience 5 Days Microsoft IT Professionals Published Level Technology 16 June 2012 200 Microsoft System Center 2012 Delivery Method

More information

101-301 Guide to Mobile Testing

101-301 Guide to Mobile Testing 101-301 Guide to Mobile Testing Perfecto Mobile & Toronto Association of System and Software Eran Kinsbruner & Joe Larizza 2014 What To Do? Great News Your first Mobile Project has arrived! You have been

More information

Considerations for Adopting PaaS (Platform as a Service)

Considerations for Adopting PaaS (Platform as a Service) Considerations for Adopting PaaS (Platform as a Service) Michael Dolan (mdolan@pivotal.io) Senior Field Engineer April 2015 1 Becoming The Agile Enterprise To effectively achieve its missions, the Department

More information

Making Sense of Big Data in Insurance

Making Sense of Big Data in Insurance Making Sense of Big Data in Insurance Amir Halfon, CTO, Financial Services, MarkLogic Corporation BIG DATA?.. SLIDE: 2 The Evolution of Data Management For your application data! Application- and hardware-specific

More information

IAN MASSINGHAM. Technical Evangelist Amazon Web Services

IAN MASSINGHAM. Technical Evangelist Amazon Web Services IAN MASSINGHAM Technical Evangelist Amazon Web Services From 2014: Cloud computing has become the new normal Deploying new applications to the cloud by default Migrating existing applications as quickly

More information

Trends and Research Opportunities in Spatial Big Data Analytics and Cloud Computing NCSU GeoSpatial Forum

Trends and Research Opportunities in Spatial Big Data Analytics and Cloud Computing NCSU GeoSpatial Forum Trends and Research Opportunities in Spatial Big Data Analytics and Cloud Computing NCSU GeoSpatial Forum Siva Ravada Senior Director of Development Oracle Spatial and MapViewer 2 Evolving Technology Platforms

More information

Evaluating NoSQL for Enterprise Applications. Dirk Bartels VP Strategy & Marketing

Evaluating NoSQL for Enterprise Applications. Dirk Bartels VP Strategy & Marketing Evaluating NoSQL for Enterprise Applications Dirk Bartels VP Strategy & Marketing Agenda The Real Time Enterprise The Data Gold Rush Managing The Data Tsunami Analytics and Data Case Studies Where to go

More information

Well packaged sets of preinstalled, integrated, and optimized software on select hardware in the form of engineered systems and appliances

Well packaged sets of preinstalled, integrated, and optimized software on select hardware in the form of engineered systems and appliances INSIGHT Oracle's All- Out Assault on the Big Data Market: Offering Hadoop, R, Cubes, and Scalable IMDB in Familiar Packages Carl W. Olofson IDC OPINION Global Headquarters: 5 Speen Street Framingham, MA

More information

BIG DATA AND THE ENTERPRISE DATA WAREHOUSE WORKSHOP

BIG DATA AND THE ENTERPRISE DATA WAREHOUSE WORKSHOP BIG DATA AND THE ENTERPRISE DATA WAREHOUSE WORKSHOP Business Analytics for All Amsterdam - 2015 Value of Big Data is Being Recognized Executives beginning to see the path from data insights to revenue

More information

Big Data Analytics. Chances and Challenges. Volker Markl

Big Data Analytics. Chances and Challenges. Volker Markl Volker Markl Professor and Chair Database Systems and Information Management (DIMA), Technische Universität Berlin www.dima.tu-berlin.de Big Data Analytics Chances and Challenges Volker Markl DIMA BDOD

More information

Keywords Big Data, NoSQL, Relational Databases, Decision Making using Big Data, Hadoop

Keywords Big Data, NoSQL, Relational Databases, Decision Making using Big Data, Hadoop Volume 4, Issue 1, January 2014 ISSN: 2277 128X International Journal of Advanced Research in Computer Science and Software Engineering Research Paper Available online at: www.ijarcsse.com Transitioning

More information

Kinjal Mody 24 th November 2015. Service Design in Practice

Kinjal Mody 24 th November 2015. Service Design in Practice Kinjal Mody 24 th November 2015 Service Design in Practice What is Service Design? ITIL Definition: Service Design is the design of appropriate and innovative services, including their architectures, processes,

More information

Enabling Continuous Delivery by Leveraging the Deployment Pipeline

Enabling Continuous Delivery by Leveraging the Deployment Pipeline Enabling Continuous Delivery by Leveraging the Deployment Pipeline Jason Carter Principal (972) 689-6402 Jason.carter@parivedasolutions.com Pariveda Solutions, Inc. Dallas,TX Table of Contents Matching

More information

Big data blue print for cloud architecture

Big data blue print for cloud architecture Big data blue print for cloud architecture -COGNIZANT Image Area Prabhu Inbarajan Srinivasan Thiruvengadathan Muralicharan Gurumoorthy Praveen Codur 2012, Cognizant Next 30 minutes Big Data / Cloud challenges

More information

Technology Enablement

Technology Enablement SOLUTION OVERVIEW 1 ABOUT TECHMILEAGE Founded in 2008 / Tempe, Arizona Over 100 engagements Full range of business & technology services Software Development, Big Data, Cloud/AWS, BI, Advanced Analytics

More information

Mark Bennett. Search and the Virtual Machine

Mark Bennett. Search and the Virtual Machine Mark Bennett Search and the Virtual Machine Agenda Intro / Business Drivers What to do with Search + Virtual What Makes Search Fast (or Slow!) Virtual Platforms Test Results Trends / Wrap Up / Q & A Business

More information

Big Data Storage Architecture Design in Cloud Computing

Big Data Storage Architecture Design in Cloud Computing Big Data Storage Architecture Design in Cloud Computing Xuebin Chen 1, Shi Wang 1( ), Yanyan Dong 1, and Xu Wang 2 1 College of Science, North China University of Science and Technology, Tangshan, Hebei,

More information

Automated Machine Learning For Autonomic Computing

Automated Machine Learning For Autonomic Computing Automated Machine Learning For Autonomic Computing ICAC 2012 Numenta Subutai Ahmad Autonomic Machine Learning ICAC 2012 Numenta Subutai Ahmad 35% 30% 25% 20% 15% 10% 5% 0% Percentage of Machine Learning

More information

BIG DATA & DATA SCIENCE

BIG DATA & DATA SCIENCE BIG DATA & DATA SCIENCE ACADEMY PROGRAMS IN-COMPANY TRAINING PORTFOLIO 2 TRAINING PORTFOLIO 2016 Synergic Academy Solutions BIG DATA FOR LEADING BUSINESS Big data promises a significant shift in the way

More information

DevOps Best Practices for Mobile Apps. Sanjeev Sharma IBM Software Group

DevOps Best Practices for Mobile Apps. Sanjeev Sharma IBM Software Group DevOps Best Practices for Mobile Apps Sanjeev Sharma IBM Software Group Me 18 year in the software industry 15+ years he has been a solution architect with IBM Areas of work: o DevOps o Enterprise Architecture

More information

Turning Big Data into Big Insights

Turning Big Data into Big Insights mwd a d v i s o r s Turning Big Data into Big Insights Helena Schwenk A special report prepared for Actuate May 2013 This report is the fourth in a series and focuses principally on explaining what s needed

More information

Anatomy of an Enterprise Software Delivery Project

Anatomy of an Enterprise Software Delivery Project Chapter 2 Anatomy of an Enterprise Software Delivery Project Chapter Summary I present an example of a typical enterprise software delivery project. I examine its key characteristics and analyze specific

More information

The Virtualization Practice

The Virtualization Practice The Virtualization Practice White Paper: Managing Applications in Docker Containers Bernd Harzog Analyst Virtualization and Cloud Performance Management October 2014 Abstract Docker has captured the attention

More information

Creating and Managing Frequent and High-Quality SaaS Product Releases

Creating and Managing Frequent and High-Quality SaaS Product Releases Creating and Managing Frequent and High-Quality SaaS Product Releases Originally published in Sandhill.com by Paul Ressler, Principal Frequent product releases are important to the SaaS business model

More information

PART IV Performance oriented design, Performance testing, Performance tuning & Performance solutions. Outline. Performance oriented design

PART IV Performance oriented design, Performance testing, Performance tuning & Performance solutions. Outline. Performance oriented design PART IV Performance oriented design, Performance testing, Performance tuning & Performance solutions Slide 1 Outline Principles for performance oriented design Performance testing Performance tuning General

More information

DevOps. Happiest People Happiest Customers

DevOps. Happiest People Happiest Customers DevOps Happiest People Happiest Customers Contents Introduction...3 What Is DevOps?...3 Do We Really Need DevOps?...4 Survey of DevOps Quantifiable Benefits...5 How Does DevOps Work Anyways?...5 Challenges

More information

Cloud Computing. Theory and Practice. Dan C. Marinescu. Morgan Kaufmann is an imprint of Elsevier HEIDELBERG LONDON AMSTERDAM BOSTON

Cloud Computing. Theory and Practice. Dan C. Marinescu. Morgan Kaufmann is an imprint of Elsevier HEIDELBERG LONDON AMSTERDAM BOSTON Cloud Computing Theory and Practice Dan C. Marinescu AMSTERDAM BOSTON HEIDELBERG LONDON NEW YORK OXFORD PARIS SAN DIEGO SAN FRANCISCO SINGAPORE SYDNEY TOKYO M< Morgan Kaufmann is an imprint of Elsevier

More information

DevOps Course Content

DevOps Course Content DevOps Course Content INTRODUCTION TO DEVOPS What is DevOps? History of DevOps Dev and Ops DevOps definitions DevOps and Software Development Life Cycle DevOps main objectives Infrastructure As A Code

More information

Databases & Business Intelligence Part 1

Databases & Business Intelligence Part 1 Welcome back! We will have more fun. Databases & Business Intelligence Part 1 BUSA345 Lecture #8-1 Claire Hitosugi, PhD, MBA In the previous lecture We learned Define Open Source Software (OSS) and provide

More information

Managing Cloud Server with Big Data for Small, Medium Enterprises: Issues and Challenges

Managing Cloud Server with Big Data for Small, Medium Enterprises: Issues and Challenges Managing Cloud Server with Big Data for Small, Medium Enterprises: Issues and Challenges Prerita Gupta Research Scholar, DAV College, Chandigarh Dr. Harmunish Taneja Department of Computer Science and

More information

Challenges and Pains in Mobile Apps Testing

Challenges and Pains in Mobile Apps Testing Challenges and Pains in Mobile Apps Testing Sales office Table of Contents Abstract... 3 Mobile Test Automation... 3 Challenges & Pains... 4 EZ TestApp Concept and Elements... 5 About TenKod Ltd.... 8

More information

Decoding the Big Data Deluge a Virtual Approach. Dan Luongo, Global Lead, Field Solution Engineering Data Virtualization Business Unit, Cisco

Decoding the Big Data Deluge a Virtual Approach. Dan Luongo, Global Lead, Field Solution Engineering Data Virtualization Business Unit, Cisco Decoding the Big Data Deluge a Virtual Approach Dan Luongo, Global Lead, Field Solution Engineering Data Virtualization Business Unit, Cisco High-volume, velocity and variety information assets that demand

More information

Detecting Anomalous Behavior with the Business Data Lake. Reference Architecture and Enterprise Approaches.

Detecting Anomalous Behavior with the Business Data Lake. Reference Architecture and Enterprise Approaches. Detecting Anomalous Behavior with the Business Data Lake Reference Architecture and Enterprise Approaches. 2 Detecting Anomalous Behavior with the Business Data Lake Pivotal the way we see it Reference

More information

Project Overview. Collabora'on Mee'ng with Op'mis, 20-21 Sept. 2011, Rome

Project Overview. Collabora'on Mee'ng with Op'mis, 20-21 Sept. 2011, Rome Project Overview Collabora'on Mee'ng with Op'mis, 20-21 Sept. 2011, Rome Cloud-TM at a glance "#$%&'$()!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!"#$%&!"'!()*+!!!!!!!!!!!!!!!!!!!,-./01234156!("*+!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!&7"7#7"7!("*+!!!!!!!!!!!!!!!!!!!89:!;62!("$+!

More information

UNIFY YOUR (BIG) DATA

UNIFY YOUR (BIG) DATA UNIFY YOUR (BIG) DATA ANALYTIC STRATEGY GIVE ANY USER ANY ANALYTIC ON ANY DATA Scott Gnau President, Teradata Labs scott.gnau@teradata.com t Unify Your (Big) Data Analytic Strategy Technology excitement:

More information

MapReduce, Hadoop and Amazon AWS

MapReduce, Hadoop and Amazon AWS MapReduce, Hadoop and Amazon AWS Yasser Ganjisaffar http://www.ics.uci.edu/~yganjisa February 2011 What is Hadoop? A software framework that supports data-intensive distributed applications. It enables

More information

HYPER-CONVERGED INFRASTRUCTURE STRATEGIES

HYPER-CONVERGED INFRASTRUCTURE STRATEGIES 1 HYPER-CONVERGED INFRASTRUCTURE STRATEGIES MYTH BUSTING & THE FUTURE OF WEB SCALE IT 2 ROADMAP INFORMATION DISCLAIMER EMC makes no representation and undertakes no obligations with regard to product planning

More information

Big Data Use Cases. To Start Today. Paul Scholey Sales Director, EMEA. 2013, Pentaho. All Rights Reserved. pentaho.com. Worldwide +1 (866) 660-7555

Big Data Use Cases. To Start Today. Paul Scholey Sales Director, EMEA. 2013, Pentaho. All Rights Reserved. pentaho.com. Worldwide +1 (866) 660-7555 Big Use Cases To Start Today Paul Scholey Sales Director, EMEA 1 Exabytes of We all know the amount of data in the world is growing exponentially 40000 30000 YOU ARE HERE 20000 FROM 2010 TO 2015 77% of

More information

Cloud Training Portal. Trainings. Process Guide. October 2012 ECPG-3. Version 1.2

Cloud Training Portal. Trainings. Process Guide. October 2012 ECPG-3. Version 1.2 Cloud Training Portal Trainings Process Guide October 2012 ECPG-3 Version 1.2 Content Content... 2 1. PREFACE... 3 1.1. About this Guide... 3 1.2. Trainers and Contacts... 3 1.3. Audience... 3 1.4. Typographic

More information

THE STATEFUL CONDITION: OR HOW I LEARNED TO STOP WORRYING AND EMBRACE THE CLOUD

THE STATEFUL CONDITION: OR HOW I LEARNED TO STOP WORRYING AND EMBRACE THE CLOUD THE STATEFUL CONDITION: OR HOW I LEARNED TO STOP WORRYING AND EMBRACE THE CLOUD Eric Jeanes NET+ Program Management, Internet2 2015 Internet2 Thanks to Brad Greer, U of Washington [ 2 ] 2015 Internet2

More information

MEng, BSc Applied Computer Science

MEng, BSc Applied Computer Science School of Computing FACULTY OF ENGINEERING MEng, BSc Applied Computer Science Year 1 COMP1212 Computer Processor Effective programming depends on understanding not only how to give a machine instructions

More information

Integrating a Big Data Platform into Government:

Integrating a Big Data Platform into Government: Integrating a Big Data Platform into Government: Drive Better Decisions for Policy and Program Outcomes John Haddad, Senior Director Product Marketing, Informatica Digital Government Institute s Government

More information

Enabling Continuous Delivery for Java Projects with Oracle Cloud Services (Oracle PaaS) Siva Rama Krishna Oracle India

Enabling Continuous Delivery for Java Projects with Oracle Cloud Services (Oracle PaaS) Siva Rama Krishna Oracle India Enabling Continuous Delivery for Java Projects with Oracle Services (Oracle PaaS) Siva Rama Krishna Oracle India Agenda What is Continuous Delivery? What is Oracle PaaS? Enabling Continuous Delivery with

More information

Cloud for Large Enterprise Where to Start. Terry Wise Director, Business Development Amazon Web Services

Cloud for Large Enterprise Where to Start. Terry Wise Director, Business Development Amazon Web Services Cloud for Large Enterprise Where to Start Terry Wise Director, Business Development Amazon Web Services Amazon Retail Business Tens of millions of active customer accounts Seven countries: US, UK, Germany,

More information

The 5G Infrastructure Public-Private Partnership

The 5G Infrastructure Public-Private Partnership The 5G Infrastructure Public-Private Partnership NetFutures 2015 5G PPP Vision 25/03/2015 19/06/2015 1 5G new service capabilities User experience continuity in challenging situations such as high mobility

More information

Consulting and Systems Integration (1) Networks & Cloud Integration Engineer

Consulting and Systems Integration (1) Networks & Cloud Integration Engineer Ericsson is a world-leading provider of telecommunications equipment & services to mobile & fixed network operators. Over 1,000 networks in more than 180 countries use Ericsson equipment, & more than 40

More information

A Guide Through the BPM Maze

A Guide Through the BPM Maze A Guide Through the BPM Maze WHAT TO LOOK FOR IN A COMPLETE BPM SOLUTION With multiple vendors, evolving standards, and ever-changing requirements, it becomes difficult to recognize what meets your BPM

More information

Driving force. What future software needs. Potential research topics

Driving force. What future software needs. Potential research topics Improving Software Robustness and Efficiency Driving force Processor core clock speed reach practical limit ~4GHz (power issue) Percentage of sustainable # of active transistors decrease; Increase in #

More information

Formal Verification Problems in a Bigdata World: Towards a Mighty Synergy

Formal Verification Problems in a Bigdata World: Towards a Mighty Synergy Dept. of Computer Science Formal Verification Problems in a Bigdata World: Towards a Mighty Synergy Matteo Camilli matteo.camilli@unimi.it http://camilli.di.unimi.it ICSE 2014 Hyderabad, India June 3,

More information

BIG DATA ANALYTICS REFERENCE ARCHITECTURES AND CASE STUDIES

BIG DATA ANALYTICS REFERENCE ARCHITECTURES AND CASE STUDIES BIG DATA ANALYTICS REFERENCE ARCHITECTURES AND CASE STUDIES Relational vs. Non-Relational Architecture Relational Non-Relational Rational Predictable Traditional Agile Flexible Modern 2 Agenda Big Data

More information

EMA Radar for Application Discovery and Dependency Mapping (ADDM): Q4 2013. AppEnsure Profile

EMA Radar for Application Discovery and Dependency Mapping (ADDM): Q4 2013. AppEnsure Profile EMA Radar for Application Discovery and Dependency Mapping (ADDM): Q4 2013 By Dennis Drogseth, VP of Research ENTERPRISE MANAGEMENT ASSOCIATES (EMA ) Radar Report December 2013 AppEnsure Introduction Santa

More information

DELIVERING AGILE QUALITY ASSURANCE THROUGH EXTREME AUTOMATION

DELIVERING AGILE QUALITY ASSURANCE THROUGH EXTREME AUTOMATION DELIVERING AGILE QUALITY ASSURANCE THROUGH EXTREME AUTOMATION Enterprises that keep pace with rapid technology advancements are witnessing dynamic changes in their business environments. Enterprise applications

More information

Big Data Driven Knowledge Discovery for Autonomic Future Internet

Big Data Driven Knowledge Discovery for Autonomic Future Internet Big Data Driven Knowledge Discovery for Autonomic Future Internet Professor Geyong Min Chair in High Performance Computing and Networking Department of Mathematics and Computer Science College of Engineering,

More information

Simplifying Big Data Analytics: Unifying Batch and Stream Processing. John Fanelli,! VP Product! In-Memory Compute Summit! June 30, 2015!!

Simplifying Big Data Analytics: Unifying Batch and Stream Processing. John Fanelli,! VP Product! In-Memory Compute Summit! June 30, 2015!! Simplifying Big Data Analytics: Unifying Batch and Stream Processing John Fanelli,! VP Product! In-Memory Compute Summit! June 30, 2015!! Streaming Analy.cs S S S Scale- up Database Data And Compute Grid

More information

3 Reasons Enterprises Struggle with Storm & Spark Streaming and Adopt DataTorrent RTS

3 Reasons Enterprises Struggle with Storm & Spark Streaming and Adopt DataTorrent RTS . 3 Reasons Enterprises Struggle with Storm & Spark Streaming and Adopt DataTorrent RTS Deliver fast actionable business insights for data scientists, rapid application creation for developers and enterprise-grade

More information

ON-PREMISE OR IN THE CLOUD, A SINGLE JAVA EE APPLICATION PLATFORM

ON-PREMISE OR IN THE CLOUD, A SINGLE JAVA EE APPLICATION PLATFORM ON-PREMISE OR IN THE CLOUD, A SINGLE JAVA EE APPLICATION PLATFORM TECHNOLOGY OVERVIEW FEATURES Fully certified Java EE 6 container Full web services stack Modular architecture optimized for cloud and virtual

More information

Sofware Engineering, Services and Cloud Computing

Sofware Engineering, Services and Cloud Computing Work Programme 2013 Objective ICT-2013.1.2: Sofware Engineering, Services and Cloud Computing DG CONNECT Unit E2: Software and Service, Cloud Target Outcomes Delivering services in an effective, efficient

More information

Data Modeling for Big Data

Data Modeling for Big Data Data Modeling for Big Data by Jinbao Zhu, Principal Software Engineer, and Allen Wang, Manager, Software Engineering, CA Technologies In the Internet era, the volume of data we deal with has grown to terabytes

More information

FUJITSU Transformational Application Managed Services

FUJITSU Transformational Application Managed Services FUJITSU Application Managed Services Going digital What does it mean for Applications Management? Most public and private sector enterprises recognize that going digital will drive business agility and

More information

Big Data and Industrial Internet

Big Data and Industrial Internet Big Data and Industrial Internet Keijo Heljanko Department of Computer Science and Helsinki Institute for Information Technology HIIT School of Science, Aalto University keijo.heljanko@aalto.fi 16.6-2015

More information

1 st Symposium on Colossal Data and Networking (CDAN-2016) March 18-19, 2016 Medicaps Group of Institutions, Indore, India

1 st Symposium on Colossal Data and Networking (CDAN-2016) March 18-19, 2016 Medicaps Group of Institutions, Indore, India 1 st Symposium on Colossal Data and Networking (CDAN-2016) March 18-19, 2016 Medicaps Group of Institutions, Indore, India Call for Papers Colossal Data Analysis and Networking has emerged as a de facto

More information

Contents. Preface Acknowledgements. Chapter 1 Introduction 1.1

Contents. Preface Acknowledgements. Chapter 1 Introduction 1.1 Preface xi Acknowledgements xv Chapter 1 Introduction 1.1 1.1 Cloud Computing at a Glance 1.1 1.1.1 The Vision of Cloud Computing 1.2 1.1.2 Defining a Cloud 1.4 1.1.3 A Closer Look 1.6 1.1.4 Cloud Computing

More information

Seamless adaptive multi- cloud management of service- based applications. European Open Cloud Collaboration Workshop, May 15, 2014, Brussels

Seamless adaptive multi- cloud management of service- based applications. European Open Cloud Collaboration Workshop, May 15, 2014, Brussels Seamless adaptive multi- cloud management of service- based applications European Open Cloud Collaboration Workshop, May 15, 2014, Brussels Interoperability and portability are a few of the main challenges

More information

Luncheon Webinar Series May 13, 2013

Luncheon Webinar Series May 13, 2013 Luncheon Webinar Series May 13, 2013 InfoSphere DataStage is Big Data Integration Sponsored By: Presented by : Tony Curcio, InfoSphere Product Management 0 InfoSphere DataStage is Big Data Integration

More information

A Service for Data-Intensive Computations on Virtual Clusters

A Service for Data-Intensive Computations on Virtual Clusters A Service for Data-Intensive Computations on Virtual Clusters Executing Preservation Strategies at Scale Rainer Schmidt, Christian Sadilek, and Ross King rainer.schmidt@arcs.ac.at Planets Project Permanent

More information

Service Virtualization: Managing Change in a Service-Oriented Architecture

Service Virtualization: Managing Change in a Service-Oriented Architecture Service Virtualization: Managing Change in a Service-Oriented Architecture Abstract Load balancers, name servers (for example, Domain Name System [DNS]), and stock brokerage services are examples of virtual

More information