Big Data Infrastructures for Processing Sentinel Data

Save this PDF as:
 WORD  PNG  TXT  JPG

Size: px
Start display at page:

Download "Big Data Infrastructures for Processing Sentinel Data"

Transcription

1 Big Data Infrastructures for Processing Sentinel Data Wolfgang Wagner Department for Geodesy and Geoinformation Technische Universität Wien Earth Observation Data Centre for Water Resources Monitoring What is Big Data? Big Data, Big Hype? Steve Dodson (2014) in An intrusion of privacy A successful business model of few big primarily American enterprises Sven Schade (2015) describes the Big Data era as a situation in where the volume, variety, velocity and veracity (3+1 Vs) in which data sets and streams become available challenges current management and processing capabilities Schade, S. (2015) Big Data breaking barriers - first steps on a long trail, ISPRS Archives, XL-7/W3,

2 Infrastructures for Processing Big Data Google Council Bluffs data center ( Sentinel Programme A fleet of European earth observation satellites for environmental monitoring

3 Sentinel-1 A Game Changer C-band SAR satellite in continuation of ERS-1/2 and ENVISAT High spatio-temporal coverage Spatial resolution m Temporal resolution < 3 days over Europe and Canada with 2 satellites Excellent data quality Highly dynamic land surface processes can be captured Impact on water management, health and other applications could be high if the challenges in the ground segment can be overcome Sentinel-1 Image of Upper Austria taken on 13/04/2015 Solar panel and SAR antenna of Sentinel-1 launched 3 April Image was acquired by the satellite's onboard camera. ESA

4 Sentinel-1 Data Volume From Byte to PetaByte 1 Byte 1 GigaByte 1 KiloByte 1 TeraByte 1 MegaByte 1 PetaByte

5 Speed of Data Transmission Download of 500 Gigabyte ( daily Sentinel-1 data volume over land) Wireless with 7 Mbit/s Landline with 1 Gbit/s Download of 1 Petabyte ( 7 years of Sentinel-1 data over land) Landline with 1 Gbit/s Speed of Data Processing Assumed processing speed of Sentinel-1 data with one computer/node ~ 4 Mbit/s Processing of 500 Gigabyte ( daily Sentinel-1 data over land) 1 computer Processing of 1 Petabyte ( 7 years of Sentinel-1 data over land) 1 computer 100 nodes 1000 nodes One needs supercomputers for processing Sentinel data!

6 Approaching Technological Frontiers? Information and communications technology (ICT) has improved dramatically over the past decades Moore s law, which states that the number of transistors in a dense integrated circuit doubles approximately every two years, still holds But there are physical limits to every technology! e.g. for any thermodynamic cycle operating between temperatures and none can exceed the efficiency of a Carnot cycle: = 1 Increasingly we face challenges related to Data volume Bandwidth and I/O Algorithmic complexity Earth Observation Ground Segment Past

7 Earth Observation Ground Segment Present Earth Observation Ground Segment Future

8 A New Paradigm for Earth Observation Reasons Fast growing volume and increasing variety of EO data Increasing complexity of algorithms with increasing resolution Higher scientific standards Algorithms must be validated with big data sets and competing algorithms Algorithms ensembles needed Solution Consequence Bring users and their software to the data Need for cooperation & specialisation An Opportunity for New Business Models Business Model of Munich-based company CloudEO

9 Big Data Infrastructures for the Sentinels Private Sector Google Earth Engine Amazon Web Services Offers Landsat data (complete from 2015 onwards) for its cloud user Helix Nebula Science Cloud etc. Consortium of European ICT providers teaming up with ESA, CERN, etc. Public Sector Initiatives trigged mainly by national space programmes THEIA Land Data Centre (France) Climate, Environment and Monitoring from Space (CEMS) (UK) OPUS/Copernicus Centre (Germany) European Space Agency etc. Thematic Exploitation Platforms Mission Exploitation Platforms Google Earth Engine Premier platform for the scientific analysis of high-resolution imagery Combines the strength of an ICT giant with expertise in earth observation (team of > 100 programmers) Rolled out on three Google data centres (US, Europe, Asia) Access through Java Script or Python API Programming in Googlish, i.e. code can only run on Google Earth Engine Image-oriented data structure, including image pyramids for interactive analysis Commercial applications are not free Data download possible (original and processed data) Landsat: complete archive MODIS: many geophysical variables Sentinel-1: already about scenes Sentinel-2: will likely follow soon

10 Snapshot of Google Earth Engine Interface showing Sentinel-1 data holding as of 4/9/2016 ( Earth Observation Data Centre (EODC) Founded in May 2014 as a Public-Private Partnership Mission EODC works together with its partners from science, the public- and the private sectors in order to foster the use of EO data for monitoring of water and land EODC acts as a community facilitator Joint developments Cloud infrastructure Operational data services Software Open Source EODC works towards a federation of data centres

11 EODC Cooperation Network Work is done within the Communities Infrastructure Sentinel-1 Sentinel-2 Already 13 Cooperation Partners from 6 countries Austria, Australia, Czech Republic, Italy, France, The Netherlands EODC Infrastructure in Vienna Virtual Machines (VMs) Supercomputer VSC-3 Rank 85 of the World s most powerful computers (11/2014) 24/7 Operations & Rolling Archive Petabyte-Scale Disk Storage Tape Storage

12 EODC Status Operations started in June 2015 after a one year development phase Operational data reception and processing by ZAMG Computer cluster to operated by EODC Virtual Machines via OpenStack Cloud Services Supercomputer VSC-3 operated by TU Wien Data and Platform Services Community Building PaaS User VMs Repositories Community File Repository VSC-3 Login Node NORA Router Job Scheduler High Availabilty Continuous Integration Various Inspection Tools Web Conferencing Development Collaboration Sentinel-1 Data EODC Sentinel-1 data are currently available ~2,5 hours after its processing time and 6,25 hours after acquisition time (median value for August 2015) acquisitions with TB (>1,5 times our 10-year ENVISAT ASAR archive) Ramp-up of Sentinel-1 acquisition scenario to full operational status

13 Supercomputing Experiment Vienna Scientific Cluster 3 High-performance computing (HPC) system with 2020 nodes Each node has 2 processors Intel Xeon E5-2650v2, 2.6 GHz, and 64 Gbytes of RAM Simple Linux Utility for Resource Management (SLURM) Experiment Geocoding of 624 Sentinel-1 images from Austria, Sudan and Zambia with Sentinel-1 toolbox Each image is about 1 Gbyte in size Serial processing with one processor would take about two weeks Approach Parallel processing on 312 nodes whereas 2 images were simultaneously launched on a single computing node Results Processing was completed within 45 min (without queuing)

14 Conclusions Big Data is a broad term for data sets so large or complex that traditional data processing applications are inadequate. Earth Observation is entering the Big Data era Big Data infrastructures for processing of Sentinel data are being developed along two main lines Deploy EO specific services on general-purpose cloud computing environments Building of new, or expansion of existing dedicated EO data centres Acknowledgements My colleagues at TU Wien and EODC: Christian Briese, Vahid Naeimi, Bernhard Bauer- Marschallinger, Christoph Paulik, Alena Dostalova, Stefano Elefante, Thomas Mistelbauer, Hans Thüminger, and Andreas Roncat Austrian Space Application Programme: Projects Prepare4EODC and WetMon European Space Agency: Contract No /12/I-BG EODC Water Study

Cloud Computing and Content Delivery Network use within Earth Observation Ground Segments: experiences and lessons learnt

Cloud Computing and Content Delivery Network use within Earth Observation Ground Segments: experiences and lessons learnt Cloud Computing and Content Delivery Network use within Earth Observation Ground Segments: experiences and lessons learnt J.Farres EOP-GS ESRIN 6/6/2012 Page 1 Agenda 1. Introduction 2. ESA Experiences

More information

Big Data and Cloud Computing for GHRSST

Big Data and Cloud Computing for GHRSST Big Data and Cloud Computing for GHRSST Jean-Francois Piollé (jfpiolle@ifremer.fr) Frédéric Paul, Olivier Archer CERSAT / Institut Français de Recherche pour l Exploitation de la Mer Facing data deluge

More information

Emerging remote environmental monitoring techniques. Remote Sensing

Emerging remote environmental monitoring techniques. Remote Sensing Emerging remote environmental monitoring techniques Remote Sensing Satellite and airborne Remote Sensing techniques Emerging trends in remote sensing are occurring largely in four broad areas: 1. advances

More information

A Future Scenario of interconnected EO Platforms How will EO data be used in 2025?

A Future Scenario of interconnected EO Platforms How will EO data be used in 2025? A Future Scenario of interconnected EO Platforms How will EO data be used in 2025? ESA UNCLASSIFIED For Official Use European EO data asset Heritage missions Heritage Core GS (data preservation, curation

More information

Big Data from Space: the Copernicus contribution

Big Data from Space: the Copernicus contribution Big Data from Space: the Copernicus contribution P. Bargellini, H. Laur EO Ground Segment and Mission Operations Department, European Space Agency Sentinel-1A: Radar Vision Acquired on 28 February 2016

More information

A Cloud Computing Approach for Big DInSAR Data Processing

A Cloud Computing Approach for Big DInSAR Data Processing A Cloud Computing Approach for Big DInSAR Data Processing through the P-SBAS Algorithm Zinno I. 1, Elefante S. 1, Mossucca L. 2, De Luca C. 1,3, Manunta M. 1, Terzo O. 2, Lanari R. 1, Casu F. 1 (1) IREA

More information

German Copernicus Data Access and Exploitation Platform BiDS 16, Teneriffa, Spain,

German Copernicus Data Access and Exploitation Platform BiDS 16, Teneriffa, Spain, DLR.de Chart 1 German Copernicus Data Access and Exploitation Platform BiDS 16, Teneriffa, Spain, 2016-03-16 Christoph Reck Gina Campuzano Klaus Dengler Torsten Heinen Mario Winkler DLR Oberpfaffenhofen

More information

EGI services for distribution and federation of data and computing

EGI services for distribution and federation of data and computing EGI services for distribution and federation of data and computing Tiziana Ferrari Technical Director, EGI.eu tiziana.ferrari@egi.eu March 2014 EGI-InSPIRE RI-261323 1 Accelerating Excellent Science MISSION.

More information

The European Space Agency s Synthetic Aperture Radar Programme From Experiment to Service Provision

The European Space Agency s Synthetic Aperture Radar Programme From Experiment to Service Provision The European Space Agency s Synthetic Aperture Radar Programme From Experiment to Service Provision Evert Attema ESA, Directorate of Earth Observation Programme! The idea of an independent European space

More information

Mission Operations and Ground Segment

Mission Operations and Ground Segment ESA Earth Observation Info Days Mission Operations and Ground Segment ESA EO Ground Segment and Mission Operations department (EOP-G) May 2013 EOEP 2013 Page 1 ESA Unclassified For Official Use MISSION

More information

ESA Earth Observation Big Data R&D Past, Present, & Future Activities

ESA Earth Observation Big Data R&D Past, Present, & Future Activities ESA Earth Observation Big Data R&D Past, Present, & Future Activities [Sveinung.Loekken, Jordi.Farres]@esa.int Ground Segment and Mission Operations Department, Earth Observation Programmes Directorate,

More information

Copernicus Space Component ESA Data Access Overview J. Martin (ESA), R. Knowelden (Airbus D&S)

Copernicus Space Component ESA Data Access Overview J. Martin (ESA), R. Knowelden (Airbus D&S) Copernicus Space Component ESA Data Access Overview J. Martin (ESA), R. Knowelden (Airbus D&S) Introduction and Context Operational Scenarios Translation into interfaces Translation into services Current

More information

IMPLEMENTING GREEN IT

IMPLEMENTING GREEN IT Saint Petersburg State University of Information Technologies, Mechanics and Optics Department of Telecommunication Systems IMPLEMENTING GREEN IT APPROACH FOR TRANSFERRING BIG DATA OVER PARALLEL DATA LINK

More information

Cloud Computing Where ISR Data Will Go for Exploitation

Cloud Computing Where ISR Data Will Go for Exploitation Cloud Computing Where ISR Data Will Go for Exploitation 22 September 2009 Albert Reuther, Jeremy Kepner, Peter Michaleas, William Smith This work is sponsored by the Department of the Air Force under Air

More information

On Demand Satellite Image Processing

On Demand Satellite Image Processing On Demand Satellite Image Processing Next generation technology for processing Terabytes of imagery on the Cloud WHITEPAPER MARCH 2015 Introduction Profound changes are happening with computing hardware

More information

Space Work Programme 2015

Space Work Programme 2015 Space Work Programme 79 billion from 2014 to 2020 2 There is a place for SPACE everywhere 3 Space in Horizon 2020 Four objectives (specific programme) 1. Enhance competitiveness, non-dependence, and innovation

More information

Big Data and the Earth Observation and Climate Modelling Communities: JASMIN and CEMS

Big Data and the Earth Observation and Climate Modelling Communities: JASMIN and CEMS Big Data and the Earth Observation and Climate Modelling Communities: JASMIN and CEMS Workshop on the Future of Big Data Management 27-28 June 2013 Philip Kershaw Centre for Environmental Data Archival

More information

CEDA Storage. Dr Matt Pritchard. Centre for Environmental Data Archival (CEDA) www.ceda.ac.uk

CEDA Storage. Dr Matt Pritchard. Centre for Environmental Data Archival (CEDA) www.ceda.ac.uk CEDA Storage Dr Matt Pritchard Centre for Environmental Data Archival (CEDA) www.ceda.ac.uk How we store our data NAS Technology Backup JASMIN/CEMS CEDA Storage Data stored as files on disk. Data is migrated

More information

Stanford SDN-Based Private Cloud. Johan van Reijendam (jvanreij@stanford.edu) Stanford University

Stanford SDN-Based Private Cloud. Johan van Reijendam (jvanreij@stanford.edu) Stanford University Stanford SDN-Based Private Cloud (jvanreij@stanford.edu) Stanford University Executive Summary The Web and its infrastructure continue to make phenomenal progress, allowing the creation and scaling of

More information

RevoScaleR Speed and Scalability

RevoScaleR Speed and Scalability EXECUTIVE WHITE PAPER RevoScaleR Speed and Scalability By Lee Edlefsen Ph.D., Chief Scientist, Revolution Analytics Abstract RevoScaleR, the Big Data predictive analytics library included with Revolution

More information

ESA Earth Observation and the need for high speed networking

ESA Earth Observation and the need for high speed networking ESA Earth Observation and the need for high speed networking Pisa, 11 th May 25 11 th May 25 GARR Conference 5 1 ESA Earth Observation 11 th May 25 GARR Conference 5 2 The European Space Agency The European

More information

Silviu Panica, Marian Neagul, Daniela Zaharie and Dana Petcu (Romania)

Silviu Panica, Marian Neagul, Daniela Zaharie and Dana Petcu (Romania) Silviu Panica, Marian Neagul, Daniela Zaharie and Dana Petcu (Romania) Outline Introduction EO challenges; EO and classical/cloud computing; EO Services The computing platform Cluster -> Grid -> Cloud

More information

CERN s Scientific Programme and the need for computing resources

CERN s Scientific Programme and the need for computing resources This document produced by Members of the Helix Nebula consortium is licensed under a Creative Commons Attribution 3.0 Unported License. Permissions beyond the scope of this license may be available at

More information

Agenda. Company Platform Customers Partners Competitive Analysis

Agenda. Company Platform Customers Partners Competitive Analysis KidoZen Overview Agenda Company Platform Customers Partners Competitive Analysis Our Vision Power the backend of the post- web enterprise Key Challenges of the Mobile Enterprise Enterprise systems integration

More information

Function and Performance Test of Open Source CloudStack Platform for

Function and Performance Test of Open Source CloudStack Platform for Function and Performance Test of pen Source CloudStack Platform for HPC Service Function and Performance Test of pen Source CloudStack Platform for HPC Service 1 Jaegyoon Hahm, 2 Sang Boem Lim, 3 Guohua

More information

Cloud Computing and Amazon Web Services

Cloud Computing and Amazon Web Services Cloud Computing and Amazon Web Services Gary A. McGilvary edinburgh data.intensive research 1 OUTLINE 1. An Overview of Cloud Computing 2. Amazon Web Services 3. Amazon EC2 Tutorial 4. Conclusions 2 CLOUD

More information

Brief overview of ESA EO Missions & Programmes

Brief overview of ESA EO Missions & Programmes Available GOCE Products Brief overview of ESA EO Missions & Programmes Pierre-Philippe Mathieu, ESA-EOP SAGOMA KO Meeting, 24 Nov, Liege, Belgium ESA EO : Overall Framework CCI, STSE GMES EOMD Available

More information

Cloud Platforms in the Enterprise

Cloud Platforms in the Enterprise Cloud Platforms in the Enterprise A Guide for IT Leaders @DChappellAssoc Copyright 2014 Chappell & Associates The Three Most Important IT Events In the last decade Initial pubic offering of Salesforce.com,

More information

SEAIP 2009 Presentation

SEAIP 2009 Presentation SEAIP 2009 Presentation By David Tan Chair of Yahoo! Hadoop SIG, 2008-2009,Singapore EXCO Member of SGF SIG Imperial College (UK), Institute of Fluid Science (Japan) & Chicago BOOTH GSB (USA) Alumni Email:

More information

ADDRESSING GRAND CHALLENGES IN EARTH OBSERVATION SCIENCE: THE EARTH OBSERVATION DATA CENTRE FOR WATER RESOURCES MONITORING

ADDRESSING GRAND CHALLENGES IN EARTH OBSERVATION SCIENCE: THE EARTH OBSERVATION DATA CENTRE FOR WATER RESOURCES MONITORING ADDRESSING GRAND CHALLENGES IN EARTH OBSERVATION SCIENCE: THE EARTH OBSERVATION DATA CENTRE FOR WATER RESOURCES MONITORING W. Wagner a, h, *, J. Fröhlich a, G. Wotawa b, h, R. Stowasser b, M. Staudinger

More information

Astrium GEO UK Multi-Mission PDGS Facilities and Services

Astrium GEO UK Multi-Mission PDGS Facilities and Services Astrium GEO UK Multi-Mission PDGS Facilities and Services Introduction UK-MM-PAC ERS-1/2 UK-MM-PAC ENVISAT Data Hosting Activities Future Activities Infoterra Archives LTDP Preparation Collaborative Activities

More information

Remote sensing information cloud service: research and practice

Remote sensing information cloud service: research and practice Remote sensing information cloud service: research and practice Yang Banghui Dr., Ren Fuhu Prof. and Wang jinnian Prof. yangbh@radi.ac.cn +8613810963452 Content 1 Background 2 Studying and Designing 3

More information

Is Big Data a Big Deal? What Big Data Does to Science

Is Big Data a Big Deal? What Big Data Does to Science Is Big Data a Big Deal? What Big Data Does to Science Netherlands escience Center Wilco Hazeleger Wilco Hazeleger Student @ Wageningen University and Reading University Meteorology PhD @ Utrecht University,

More information

Copernicus and Big Data: Challenges and Opportunities

Copernicus and Big Data: Challenges and Opportunities Copernicus and Big Data: Challenges and Opportunities Alessandro Annoni European Commission Joint Research Centre www.jrc.ec.europa.eu Serving society Stimulating innovation Supporting legislation Big

More information

High Performance Computing at CINECA

High Performance Computing at CINECA High Performance Computing at CINECA Marzia Rivi m.rivi@cineca.it Supercomputing Applications & Innovation Department - CINECA CINECA Non profit Consortium, made up of 51 Italian universities, The National

More information

Long Term Preservation of Earth Observation Data

Long Term Preservation of Earth Observation Data Long Term Preservation of Earth Observation Data QA4EO Workshop RAL, October 18-20 th 2011 Mirko Albani and Bojan Bojkov* (ESA/ESRIN) Page 1 Outline Earth Observation data preservation: the need and the

More information

Cloud Computing. Alex Crawford Ben Johnstone

Cloud Computing. Alex Crawford Ben Johnstone Cloud Computing Alex Crawford Ben Johnstone Overview What is cloud computing? Amazon EC2 Performance Conclusions What is the Cloud? A large cluster of machines o Economies of scale [1] Customers use a

More information

The 5G Infrastructure Public-Private Partnership

The 5G Infrastructure Public-Private Partnership The 5G Infrastructure Public-Private Partnership NetFutures 2015 5G PPP Vision 25/03/2015 19/06/2015 1 5G new service capabilities User experience continuity in challenging situations such as high mobility

More information

Big Data in the context of Preservation and Value Adding

Big Data in the context of Preservation and Value Adding Big Data in the context of Preservation and Value Adding R. Leone, R. Cosac, I. Maggio, D. Iozzino ESRIN 06/11/2013 ESA UNCLASSIFIED Big Data Background ESA/ESRIN organized a 'Big Data from Space' event

More information

Enabling multi-cloud resources at CERN within the Helix Nebula project. D. Giordano (CERN IT-SDC) HEPiX Spring 2014 Workshop 23 May 2014

Enabling multi-cloud resources at CERN within the Helix Nebula project. D. Giordano (CERN IT-SDC) HEPiX Spring 2014 Workshop 23 May 2014 Enabling multi-cloud resources at CERN within the Helix Nebula project D. Giordano (CERN IT-) HEPiX Spring 2014 Workshop This document produced by Members of the Helix Nebula consortium is licensed under

More information

HPC Cloud. Focus on your research. Floris Sluiter Project leader SARA

HPC Cloud. Focus on your research. Floris Sluiter Project leader SARA HPC Cloud Focus on your research Floris Sluiter Project leader SARA Why an HPC Cloud? Christophe Blanchet, IDB - Infrastructure Distributing Biology: Big task to port them all to your favorite architecture

More information

Volunteer Computing, Grid Computing and Cloud Computing: Opportunities for Synergy. Derrick Kondo INRIA, France

Volunteer Computing, Grid Computing and Cloud Computing: Opportunities for Synergy. Derrick Kondo INRIA, France Volunteer Computing, Grid Computing and Cloud Computing: Opportunities for Synergy Derrick Kondo INRIA, France Outline Cloud Grid Volunteer Computing Cloud Background Vision Hide complexity of hardware

More information

Solution for private cloud computing

Solution for private cloud computing The CC1 system Solution for private cloud computing 1 Outline What is CC1? Features Technical details Use cases By scientist By HEP experiment System requirements and installation How to get it? 2 What

More information

The Hartree Centre helps businesses unlock the potential of HPC

The Hartree Centre helps businesses unlock the potential of HPC The Hartree Centre helps businesses unlock the potential of HPC Fostering collaboration and innovation across UK industry with help from IBM Overview The need The Hartree Centre needs leading-edge computing

More information

emergency.lu aims to improve the worldwide preparedness and rapid response capacity by filling the communication gap in the first hours and days

emergency.lu aims to improve the worldwide preparedness and rapid response capacity by filling the communication gap in the first hours and days emergency.lu emergency.lu aims to improve the worldwide preparedness and rapid response capacity by filling the communication gap in the first hours and days after a disaster emergency.lu is a multi-layer

More information

JASMIN Cloud ESGF and UV- CDAT Conference 09-11 December 2014 STFC / Stephen Kill

JASMIN Cloud ESGF and UV- CDAT Conference 09-11 December 2014 STFC / Stephen Kill JASMIN Cloud ESGF and UV- CDAT Conference 09-11 December 2014 STFC / Stephen Kill Philip Kershaw (1, 2), Jonathan Churchill (5), Bryan Lawrence (1, 3, 4), Stephen Pascoe (1, 4) and MaE Pritchard (1) Centre

More information

BIG DATA FUNDAMENTALS

BIG DATA FUNDAMENTALS BIG DATA FUNDAMENTALS Timeframe Minimum of 30 hours Use the concepts of volume, velocity, variety, veracity and value to define big data Learning outcomes Critically evaluate the need for big data management

More information

High Performance Computing (HPC)

High Performance Computing (HPC) High Performance Computing (HPC) High Performance Computing (HPC) White Paper Attn: Name, Title Phone: xxx.xxx.xxxx Fax: xxx.xxx.xxxx 1.0 OVERVIEW When heterogeneous enterprise environments are involved,

More information

Research IT Plan. UCD IT Services. Seirbhísí TF UCD

Research IT Plan. UCD IT Services. Seirbhísí TF UCD Research IT Plan UCD IT Services Research IT Plan The main goal of this plan is to provide a sustainable and evolving campus Cyberinfrastructure for the UCD research community. We will continue the development

More information

Integrated Risk Management System Components in the GEO Architecture Implementation Pilot Phase 2 (AIP-2)

Integrated Risk Management System Components in the GEO Architecture Implementation Pilot Phase 2 (AIP-2) Meraka Institute ICT for Earth Observation PO Box 395 Pretoria 0001, Gauteng, South Africa Telephone: +27 12 841 3028 Facsimile: +27 12 841 4720 University of KwaZulu- Natal School of Computer Science

More information

COMP/CS 605: Intro to Parallel Computing Lecture 01: Parallel Computing Overview (Part 1)

COMP/CS 605: Intro to Parallel Computing Lecture 01: Parallel Computing Overview (Part 1) COMP/CS 605: Intro to Parallel Computing Lecture 01: Parallel Computing Overview (Part 1) Mary Thomas Department of Computer Science Computational Science Research Center (CSRC) San Diego State University

More information

A Study of Data Management Technology for Handling Big Data

A Study of Data Management Technology for Handling Big Data Available Online at www.ijcsmc.com International Journal of Computer Science and Mobile Computing A Monthly Journal of Computer Science and Information Technology IJCSMC, Vol. 3, Issue. 9, September 2014,

More information

wu.cloud: Insights Gained from Operating a Private Cloud System

wu.cloud: Insights Gained from Operating a Private Cloud System wu.cloud: Insights Gained from Operating a Private Cloud System Stefan Theußl, Institute for Statistics and Mathematics WU Wirtschaftsuniversität Wien March 23, 2011 1 / 14 Introduction In statistics we

More information

Doing Multidisciplinary Research in Data Science

Doing Multidisciplinary Research in Data Science Doing Multidisciplinary Research in Data Science Assoc.Prof. Abzetdin ADAMOV CeDAWI - Center for Data Analytics and Web Insights Qafqaz University aadamov@qu.edu.az http://ce.qu.edu.az/~aadamov 16 May

More information

U"lizing the SDSC Cloud Storage Service

Ulizing the SDSC Cloud Storage Service U"lizing the SDSC Cloud Storage Service PASIG Conference January 13, 2012 Richard L. Moore rlm@sdsc.edu San Diego Supercomputer Center University of California San Diego SAN DIEGO SUPERCOMPUTER CENTER

More information

ESA UNCLASSIFIED For Official Use. Request for Information. Implementation of Thematic Exploitation Platforms

ESA UNCLASSIFIED For Official Use. Request for Information. Implementation of Thematic Exploitation Platforms ESA UNCLASSIFIED For Official Use Request for Information Implementation of Thematic Exploitation Platforms Table of contents: 1! INTRODUCTION*...*3! 1.1! SCOPE AND PURPOSE!...!3! 1.2! CONTACTS AT ESA!...!3!

More information

Big Data Analytics. Prof. Dr. Lars Schmidt-Thieme

Big Data Analytics. Prof. Dr. Lars Schmidt-Thieme Big Data Analytics Prof. Dr. Lars Schmidt-Thieme Information Systems and Machine Learning Lab (ISMLL) Institute of Computer Science University of Hildesheim, Germany 33. Sitzung des Arbeitskreises Informationstechnologie,

More information

White Paper. Version 1.2 May 2015 RAID Incorporated

White Paper. Version 1.2 May 2015 RAID Incorporated White Paper Version 1.2 May 2015 RAID Incorporated Introduction The abundance of Big Data, structured, partially-structured and unstructured massive datasets, which are too large to be processed effectively

More information

Big Data and Analytics: Getting Started with ArcGIS. Mike Park Erik Hoel

Big Data and Analytics: Getting Started with ArcGIS. Mike Park Erik Hoel Big Data and Analytics: Getting Started with ArcGIS Mike Park Erik Hoel Agenda Overview of big data Distributed computation User experience Data management Big data What is it? Big Data is a loosely defined

More information

Evoluzione dell Infrastruttura di Calcolo e Data Analytics per la ricerca

Evoluzione dell Infrastruttura di Calcolo e Data Analytics per la ricerca Evoluzione dell Infrastruttura di Calcolo e Data Analytics per la ricerca Carlo Cavazzoni CINECA Supercomputing Application & Innovation www.cineca.it 21 Aprile 2015 FERMI Name: Fermi Architecture: BlueGene/Q

More information

Utilizing the SDSC Cloud Storage Service

Utilizing the SDSC Cloud Storage Service Utilizing the SDSC Cloud Storage Service PASIG Conference January 13, 2012 Richard L. Moore rlm@sdsc.edu San Diego Supercomputer Center University of California San Diego Traditional supercomputer center

More information

Data Centric Computing Revisited

Data Centric Computing Revisited Piyush Chaudhary Technical Computing Solutions Data Centric Computing Revisited SPXXL/SCICOMP Summer 2013 Bottom line: It is a time of Powerful Information Data volume is on the rise Dimensions of data

More information

WHAT WOULD IT MEAN IF YOU COULD VISUALISE ALL YOUR DATA IN LESS THAN A SECOND FROM WHEREVER YOU ARE?

WHAT WOULD IT MEAN IF YOU COULD VISUALISE ALL YOUR DATA IN LESS THAN A SECOND FROM WHEREVER YOU ARE? WHAT WOULD IT MEAN IF YOU COULD VISUALISE ALL YOUR DATA IN LESS THAN A SECOND FROM WHEREVER YOU ARE? ACCURATE, REAL-TIME 3D DATA VISUALISATION. Geoverse delivers instant, interactive access to your entire

More information

Cluster Scalability of ANSYS FLUENT 12 for a Large Aerodynamics Case on the Darwin Supercomputer

Cluster Scalability of ANSYS FLUENT 12 for a Large Aerodynamics Case on the Darwin Supercomputer Cluster Scalability of ANSYS FLUENT 12 for a Large Aerodynamics Case on the Darwin Supercomputer Stan Posey, MSc and Bill Loewe, PhD Panasas Inc., Fremont, CA, USA Paul Calleja, PhD University of Cambridge,

More information

CLOUD COMPUTING. When It's smarter to rent than to buy

CLOUD COMPUTING. When It's smarter to rent than to buy CLOUD COMPUTING When It's smarter to rent than to buy Is it new concept? Nothing new In 1990 s, WWW itself Grid Technologies- Scientific applications Online banking websites More convenience Not to visit

More information

Data-intensive Computing on the Cloud: Concepts, Technologies and Applications B. Ramamurthy bina@buffalo.edu This talks is partially supported by

Data-intensive Computing on the Cloud: Concepts, Technologies and Applications B. Ramamurthy bina@buffalo.edu This talks is partially supported by Data-intensive Computing on the Cloud: Concepts, Technologies and Applications B. Ramamurthy bina@buffalo.edu This talks is partially supported by National Science Foundation grants DUE: #0920335, OCI:

More information

Data Centric Systems (DCS)

Data Centric Systems (DCS) Data Centric Systems (DCS) Architecture and Solutions for High Performance Computing, Big Data and High Performance Analytics High Performance Computing with Data Centric Systems 1 Data Centric Systems

More information

Trends and Research Opportunities in Spatial Big Data Analytics and Cloud Computing NCSU GeoSpatial Forum

Trends and Research Opportunities in Spatial Big Data Analytics and Cloud Computing NCSU GeoSpatial Forum Trends and Research Opportunities in Spatial Big Data Analytics and Cloud Computing NCSU GeoSpatial Forum Siva Ravada Senior Director of Development Oracle Spatial and MapViewer 2 Evolving Technology Platforms

More information

Cloud Computing for Control Systems CERN Openlab Summer Student Program 9/9/2011 ARSALAAN AHMED SHAIKH

Cloud Computing for Control Systems CERN Openlab Summer Student Program 9/9/2011 ARSALAAN AHMED SHAIKH Cloud Computing for Control Systems CERN Openlab Summer Student Program 9/9/2011 ARSALAAN AHMED SHAIKH CONTENTS Introduction... 4 System Components... 4 OpenNebula Cloud Management Toolkit... 4 VMware

More information

Experiences and challenges in the development of the JASMIN cloud service for the environmental science community

Experiences and challenges in the development of the JASMIN cloud service for the environmental science community JASMIN (STFC/Stephen Kill) Experiences and challenges in the development of the JASMIN cloud service for the environmental science community ECMWF Visualisa-on in Meteorology Week, 28 September 2015 Philip

More information

SAP Mobile Platform. SAP Mobile Platform. Cloud Performance and Scalability SAP AG or an SAP affiliate company. All rights reserved.

SAP Mobile Platform. SAP Mobile Platform. Cloud Performance and Scalability SAP AG or an SAP affiliate company. All rights reserved. SAP Mobile Platform SAP Mobile Platform Cloud Performance and Scalability Table of Contents 4 Performance Test Configurations The Test Plans 7 Performance Test Results Single-User Test Results Multiuser

More information

PARALLEL & CLUSTER COMPUTING CS 6260 PROFESSOR: ELISE DE DONCKER BY: LINA HUSSEIN

PARALLEL & CLUSTER COMPUTING CS 6260 PROFESSOR: ELISE DE DONCKER BY: LINA HUSSEIN 1 PARALLEL & CLUSTER COMPUTING CS 6260 PROFESSOR: ELISE DE DONCKER BY: LINA HUSSEIN Introduction What is cluster computing? Classification of Cluster Computing Technologies: Beowulf cluster Construction

More information

Chapter 19 Cloud Computing for Multimedia Services

Chapter 19 Cloud Computing for Multimedia Services Chapter 19 Cloud Computing for Multimedia Services 19.1 Cloud Computing Overview 19.2 Multimedia Cloud Computing 19.3 Cloud-Assisted Media Sharing 19.4 Computation Offloading for Multimedia Services 19.5

More information

Systems, Storage and Software in the National Supercomputing Service. CSCS User Assembly, Luzern, 26 th March 2010 Neil Stringfellow

Systems, Storage and Software in the National Supercomputing Service. CSCS User Assembly, Luzern, 26 th March 2010 Neil Stringfellow Systems, Storage and Software in the National Supercomputing Service CSCS User Assembly, Luzern, 26 th March 2010 Neil Stringfellow Cray XT5 Monte Rosa 22,168 processors 1844 twelve-way nodes 2 AMD 2.4

More information

Computing in clouds: Where we come from, Where we are, What we can, Where we go

Computing in clouds: Where we come from, Where we are, What we can, Where we go Computing in clouds: Where we come from, Where we are, What we can, Where we go Luc Bougé ENS Cachan/Rennes, IRISA, INRIA Biogenouest With help from many colleagues: Gabriel Antoniu, Guillaume Pierre,

More information

Cloud computing. Intelligent Services for Energy-Efficient Design and Life Cycle Simulation. as used by the ISES project

Cloud computing. Intelligent Services for Energy-Efficient Design and Life Cycle Simulation. as used by the ISES project Intelligent Services for Energy-Efficient Design and Life Cycle Simulation Project number: 288819 Call identifier: FP7-ICT-2011-7 Project coordinator: Technische Universität Dresden, Germany Website: ises.eu-project.info

More information

Cloud Storage. Parallels. Performance Benchmark Results. White Paper. www.parallels.com

Cloud Storage. Parallels. Performance Benchmark Results. White Paper. www.parallels.com Parallels Cloud Storage White Paper Performance Benchmark Results www.parallels.com Table of Contents Executive Summary... 3 Architecture Overview... 3 Key Features... 4 No Special Hardware Requirements...

More information

Milestone Solution Partner IT Infrastructure MTP Certification Report Scality RING Software-Defined Storage 11-16-2015

Milestone Solution Partner IT Infrastructure MTP Certification Report Scality RING Software-Defined Storage 11-16-2015 Milestone Solution Partner IT Infrastructure MTP Certification Report Scality RING Software-Defined Storage 11-16-2015 Table of Contents Introduction... 4 Certified Products... 4 Key Findings... 5 Solution

More information

ARM-UAV Mission Gateway System

ARM-UAV Mission Gateway System ARM-UAV Mission Gateway System S. T. Moore and S. Bottone Mission Research Corporation Santa Barbara, California Introduction The Atmospheric Radiation Measurement-unmanned aerospace vehicle (ARM-UAV)

More information

SixSq Cloud Capabilities

SixSq Cloud Capabilities SixSq Cloud Capabilities SlipStream: Mutli-cloud Management Platform Marc-Elian Bégin, CEO, Co-founder, SixSq HEPIA Cloud Masters, Lausanne, 2015 Locations Global Headquarters Geneva, Switzerland North

More information

Business-centric Storage FUJITSU Hyperscale Storage System ETERNUS CD10000

Business-centric Storage FUJITSU Hyperscale Storage System ETERNUS CD10000 Business-centric Storage FUJITSU Hyperscale Storage System ETERNUS CD10000 Clear the way for new business opportunities. Unlock the power of data. Overcoming storage limitations Unpredictable data growth

More information

Forestry Thematic Exploitation Platform Earth Observation Open Science 2.0

Forestry Thematic Exploitation Platform Earth Observation Open Science 2.0 Forestry Thematic Exploitation Platform Earth Observation Open Science 2.0 Tuomas Häme VTT Technical Research of Finland Ltd and the Forestry TEP Team Objective One-stop shop for forestry remote sensing

More information

IES+Perto Project Cloud Computing

IES+Perto Project Cloud Computing IES+Perto Project Cloud Computing Instituições de Ensino Superior Mais Perto Higher Education Institutions Closer Universities: Aveiro, Coimbra, Porto José António Sousa (UP), Fernando Correia (UP), Mário

More information

Challenges in Delivering Large-scale Services over Cloud Environments

Challenges in Delivering Large-scale Services over Cloud Environments Computation World 2014 Panel CLOUD/SERVICES Challenges in Delivering Large-scale Services over Cloud Environments Moderator Christoph Reich, Furtwangen University of Applied Science, Germany Panelists

More information

Big Data Analytics. An Introduction. Oliver Fuchsberger University of Paderborn 2014

Big Data Analytics. An Introduction. Oliver Fuchsberger University of Paderborn 2014 Big Data Analytics An Introduction Oliver Fuchsberger University of Paderborn 2014 Table of Contents I. Introduction & Motivation What is Big Data Analytics? Why is it so important? II. Techniques & Solutions

More information

AN OPENGIS WEB MAP SERVER FOR THE ESA MULTI-MISSION CATALOGUE

AN OPENGIS WEB MAP SERVER FOR THE ESA MULTI-MISSION CATALOGUE AN OPENGIS WEB MAP SERVER FOR THE ESA MULTI-MISSION CATALOGUE T. Westin a, *, C. Caspar b, L. Edgardh a, L. Schylberg c a Spacemetric AB, Tingsv 19, 19161 Sollentuna, Sweden - tw@spacemetric.se b ESA Esrin,

More information

Elastic Cloud Computing in the Open Cirrus Testbed implemented via Eucalyptus

Elastic Cloud Computing in the Open Cirrus Testbed implemented via Eucalyptus Elastic Cloud Computing in the Open Cirrus Testbed implemented via Eucalyptus International Symposium on Grid Computing 2009 (Taipei) Christian Baun The cooperation of and Universität Karlsruhe (TH) Agenda

More information

High Performance Applications over the Cloud: Gains and Losses

High Performance Applications over the Cloud: Gains and Losses High Performance Applications over the Cloud: Gains and Losses Dr. Leila Ismail Faculty of Information Technology United Arab Emirates University leila@uaeu.ac.ae http://citweb.uaeu.ac.ae/citweb/profile/leila

More information

ACCESS TO ERS AND ENVISAT DATA. CGMS is informed about the ESA Earth Observation data policy and data access, in particular in Near Real Time.

ACCESS TO ERS AND ENVISAT DATA. CGMS is informed about the ESA Earth Observation data policy and data access, in particular in Near Real Time. Prepared by ESA Agenda Item: III.3 Discussed in WG3 ACCESS TO ERS AND ENVISAT DATA CGMS is informed about the ESA Earth Observation data policy and data access, in particular in Near Real Time. ACCESS

More information

Virtualization with Windows

Virtualization with Windows Virtualization with Windows at CERN Juraj Sucik, Emmanuel Ormancey Internet Services Group Agenda Current status of IT-IS group virtualization service Server Self Service New virtualization features in

More information

Surfing the Data Tsunami: A New Paradigm for Big Data Processing and Analytics

Surfing the Data Tsunami: A New Paradigm for Big Data Processing and Analytics Surfing the Data Tsunami: A New Paradigm for Big Data Processing and Analytics Dr. Liangxiu Han Future Networks and Distributed Systems Group (FUNDS) School of Computing, Mathematics and Digital Technology,

More information

HA Linux Cloud Server

HA Linux Cloud Server HA Linux Cloud Server October 100% Uptime Features Quicklist Easy setup: Sign up online CPU: Intel Xeon 2.27GHz - Upgrades to up to 8 Cores available Dedicated Memory: 1GB to 16GB SAN Storage: 50GB to

More information

Parallel Processing using the LOTUS cluster

Parallel Processing using the LOTUS cluster Parallel Processing using the LOTUS cluster Alison Pamment / Cristina del Cano Novales JASMIN/CEMS Workshop February 2015 Overview Parallelising data analysis LOTUS HPC Cluster Job submission on LOTUS

More information

EO data hosting and processing core capabilities and emerging solutions

EO data hosting and processing core capabilities and emerging solutions EO data hosting and processing core capabilities and emerging solutions Andrew Groom 4 th March 2015 Contents An introduction to Airbus Defence and Space, Geo-Intelligence Elements of the C3S vision EO

More information

The Collaboratorium & Remote Visualization at SARA. Tijs de Kler SARA Visualization Group (tijs.dekler@sara.nl)

The Collaboratorium & Remote Visualization at SARA. Tijs de Kler SARA Visualization Group (tijs.dekler@sara.nl) The Collaboratorium & Remote Visualization at SARA Tijs de Kler SARA Visualization Group (tijs.dekler@sara.nl) The Collaboratorium! Goals Support collaboration, presentations and visualization for the

More information

IBM Platform Computing Cloud Service Ready to use Platform LSF & Symphony clusters in the SoftLayer cloud

IBM Platform Computing Cloud Service Ready to use Platform LSF & Symphony clusters in the SoftLayer cloud IBM Platform Computing Cloud Service Ready to use Platform LSF & Symphony clusters in the SoftLayer cloud February 25, 2014 1 Agenda v Mapping clients needs to cloud technologies v Addressing your pain

More information

Grid vs. Cloud Computing

Grid vs. Cloud Computing Grid vs. Cloud Computing The similarities and differences between Cloud Computing and Extreme-Scale Computation on Demand 2008 Parabon Inc. All rights reserved. 2009 Parabon 1 Computation, Inc. All rights

More information

SR-IOV: Performance Benefits for Virtualized Interconnects!

SR-IOV: Performance Benefits for Virtualized Interconnects! SR-IOV: Performance Benefits for Virtualized Interconnects! Glenn K. Lockwood! Mahidhar Tatineni! Rick Wagner!! July 15, XSEDE14, Atlanta! Background! High Performance Computing (HPC) reaching beyond traditional

More information

Data Centers and Cloud Computing

Data Centers and Cloud Computing Data Centers and Cloud Computing CS377 Guest Lecture Tian Guo 1 Data Centers and Cloud Computing Intro. to Data centers Virtualization Basics Intro. to Cloud Computing Case Study: Amazon EC2 2 Data Centers

More information

The air conditioning is organized on the principle of hot and cold aisles. Controlled power outlets from APC. Automatic server security control

The air conditioning is organized on the principle of hot and cold aisles. Controlled power outlets from APC. Automatic server security control COMPANY «UNIT-IS» is a team of professionals with over a decade of experience in the telecommunications and IT-development projects! The team consists of qualified system and network engineers, developers,

More information