Computing at the HL-LHC

Save this PDF as:
 WORD  PNG  TXT  JPG

Size: px
Start display at page:

Download "Computing at the HL-LHC"

Transcription

1 Computing at the HL-LHC Predrag Buncic on behalf of the Trigger/DAQ/Offline/Computing Preparatory Group ALICE: Pierre Vande Vyvre, Thorsten Kollegger, Predrag Buncic; ATLAS: David Rousseau, Benedetto Gorini, Nikos Konstantinidis; CMS: Wesley Smith, Christoph Schwick, Ian Fisk, Peter Elmer ; LHCb: Renaud Legac, Niko Neufeld LHCb Predrag Buncic, October 3, 2013 ECFA Workshop Aix-Les-Bains - 1

2 HL-HLC CPU needs (per event) will grow with track multiplicity (pileup) Storage needs are proportional to accumulated luminosity Run1 Run2 Run3 ALICE + LHCb Run4 ATLAS + CMS 1. Resource estimates 2. The probable evolution of the distributed computed technologies Predrag Buncic, October 3, 2013 ECFA Workshop Aix-Les-Bains - 2

3 LHCb & Run 3 40 MHz 40 MHz 50 khz 5-40 MHz Reconstruction + Compression 20 khz (0.1 MB/event) Storage 50 khz (1.5 MB/event) 2 GB/s PEAK OUTPUT 75 GB/s Predrag Buncic, October 3, 2013 ECFA Workshop Aix-Les-Bains - 3

4 ATLAS & Run 4 Level 1 Level 1 HLT Storage 5-10 khz (2MB/event) HLT Storage 10 khz (4MB/event) GB/s PEAK OUTPUT 40 GB/s Predrag Buncic, October 3, 2013 ECFA Workshop Aix-Les-Bains - 4

5 access rate Big Data standard data processing applicable BIG DATA size Data that exceeds the boundaries and sizes of normal processing capabilities, forcing you to take a non-traditional approach Predrag Buncic, October 3, 2013 ECFA Workshop Aix-Les-Bains - 5

6 How do we score today? Tweeter Stock database Library of Congres Digital collection RAW + Derived Run1 Climatic Data Center database LHC raw data per year YouTube videos per year Digital Health records Google index Facebook new content per year PB Predrag Buncic, October 3, 2013 ECFA Workshop Aix-Les-Bains - 6

7 PB Data: Outlook for HL-LHC CMS ATLAS ALICE LHCb Run 1 Run 2 Run 3 Run 4 Very rough estimate of a new RAW data per year of running using a simple extrapolation of current data volume scaled by the output rates. To be added: derived data (ESD, AOD), simulation, user data Predrag Buncic, October 3, 2013 ECFA Workshop Aix-Les-Bains - 7

8 Digital Universe Expansion Predrag Buncic, October 3, 2013 ECFA Workshop Aix-Les-Bains - 8

9 Data storage issues Our data problems may still look small on the scale of storage needs of internet giants Business , video, music, smartphones, digital cameras generate more and more need for storage The cost of storage will probably continue to go down but Commodity high capacity disks may start to look more like tapes, optimized for multimedia storage, sequential access Need to be combined with flash memory disks for fast random access The residual cost of disk servers will remain While we might be able to write all this data, how long it will take to read it back? Need for sophisticated parallel I/O and processing. + We have to store this amount of data every year and for many years to come (Long Term Data Preservation ) Predrag Buncic, October 3, 2013 ECFA Workshop Aix-Les-Bains - 9

10 Exabyte Scale We are heading towards Exabyte scale Predrag Buncic, October 3, 2013 ECFA Workshop Aix-Les-Bains - 10

11 CPU: Online requirements ALICE & Run 3,4 HLT farms for online reconstruction/compression or event rejections with aim to reduce data volume to manageable levels Event rate x 100, data volume x 10 ALICE estimate: 200k today s core equivalent, 2M HEP-SPEC-06 LHCb estimate: 880k today s core equivalent, 8M HEP-SPEC-06 Looking into alternative platforms to optimize performance and minimize cost GPUs ( 1 GPU = 3 CPUs for ALICE online tracking use case) ARM, Atom ATLAS & Run 4 Trigger rate x 10, CMS 4MB/event, ATLAS 2MB/event CMS: Assuming linear scaling with pileup, a total factor of 50 increase (10M HEP-SPEC-06) of HLT power would be needed wrt. today s farm Predrag Buncic, October 3, 2013 ECFA Workshop Aix-Les-Bains - 11

12 MHS CPU: Online + Offline Moore s law limit GRID ATLAS CMS LHCb ALICE Room for improvement GRID Historical growth of 25%/year 20 ONLINE 0 Run 1 Run 2 Run 3 Run 4 Very rough estimate of new CPU requirements for online and offline processing per year of data taking using a simple extrapolation of current requirements scaled by the number of events. Little headroom left, we must work on improving the performance. Predrag Buncic, October 3, 2013 ECFA Workshop Aix-Les-Bains - 12

13 How to improve the performance? Clock frequency Vectors Instruction Pipelining Instruction Level Parallelism (ILP) Hardware threading Multi-core Multi-socket Multi-node } Very little gain to be expected and no action to be taken Potential gain in throughput and in time-to-finish Gain in memory footprint and time-to-finish but not in throughput NEXT TALK Improving the algorithms is the only way to reclaim factors in performance! Running independent jobs per core (as we do now) is optimal solution for High Throughput Computing applications THIS TALK Predrag Buncic, October 3, 2013 ECFA Workshop Aix-Les-Bains - 13

14 WLCG Collaboration Distributed infrastructure of 150 computing centers in 40 countries 300+ k CPU cores (~ 2M HEP-SPEC-06) The biggest site with ~50k CPU cores, 12 T2 with 2-30k CPU cores Distributed data, services and operation infrastructure Predrag Buncic, October 3, 2013 ECFA Workshop Aix-Les-Bains - 14

15 Distributed computing today LHCb Running millions of jobs and processing hundreds of PB of data Efficiency (CPU/Wall time) from 70% (analysis) to 85% (organized activities, reconstruction, simulation) 60-70% of all CPU time on the grid is spent in running simulation Predrag Buncic, October 3, 2013 ECFA Workshop Aix-Les-Bains - 15

16 Can it scale? GlideinWMS LHCb Each experiment today runs their own Grid overlay on top of shared WLCG infrastructure In all cases the underlying distributed computing architecture is similar based on pilot job model Can we scale these systems expected levels? Lots of commonality and potential for consolidation Predrag Buncic, October 3, 2013 ECFA Workshop Aix-Les-Bains - 16

17 Grid Tier model Network capabilities and data access technologies significantly improved our ability to use resources independent of location Predrag Buncic, October 3, 2013 ECFA Workshop Aix-Les-Bains - 17

18 FAX Deployment Global Data Federation B. Bockelman, CHEP 2012 FAX Federated ATLAS Xrootd, AAA Any Data, Any Time, AX Anywhere is a 15PB (CMS), federation, AliEn (ALICE) including ATLAS T3s and multip layers of hierarchy. Predrag Buncic, October 3, 2013 ECFA Workshop Aix-Les-Bains - 18

19 Virtualization Virtualization provides better system utilization by putting all those many cores to work, resulting in power & cost savings. Predrag Buncic, October 3, 2013 ECFA Workshop Aix-Les-Bains - 19

20 Software integration problem Application Libraries Traditional model Horizontal layers Independently developed Maintained by the different groups Different lifecycle Application is deployed on top of the stack Breaks if any layer changes Needs to be certified every time when something changes Results in deployment and support nightmare Difficult to do upgrades Even worse to switch to new OS versions Predrag Buncic, October 3, 2013 ECFA Workshop Aix-Les-Bains - 20

21 Decoupling Apps and Ops Application Libraries Tools Databases OS Application driven approach 1. Start by analysing the application requirements and dependencies 2. Add required tools and libraries Use virtualization to 1. Build minimal OS 2. Bundle all this into Virtual Machine image Separates lifecycles of the application and underlying computing infrastructure Predrag Buncic, October 3, 2013 ECFA Workshop Aix-Les-Bains - 21

22 Cloud: Putting it all together Server consolidation using virtualization improves resource usage API IaaS = Infrastructure as a Service Public, on demand, pay as you go infrastructure with goals and capabilities similar to those of academic grids Predrag Buncic, October 3, 2013 ECFA Workshop Aix-Les-Bains - 22

23 Public Cloud At present 9 large sites/zones up to ~2M CPU cores/site, ~4M total 10 x more cores on 1/10 of the sites compared to our Grid 100 x more users (500,000) Predrag Buncic, October 3, 2013 ECFA Workshop Aix-Les-Bains - 23

24 Could we make our own Cloud? Open Source Cloud components Open Source Cloud middleware VM building tools and infrastructure CernVM+CernVM/FS, boxgrinder.. Common API From the Grid heritage Common Authentication/Authorization High performance global data federation/storage Scheduled file transfer Experiment distributed computing frameworks already adapted to Cloud To do Unified access and cross Cloud scheduling Centralized key services at a few large centers Reduce complexity in terms of number of sites Predrag Buncic, October 3, 2013 ECFA Workshop Aix-Les-Bains - 24

25 Grid on top of Clouds Private CERN Cloud Analysis Reconstruction Custodial Data Store AliEn CAF On Demand Reconstruction Calibration Data Store T1/T2/T3 AliEn O 2 Online-Offline Facility HLT DAQ AliEn CAF On Demand Simulation Analysis Simulation Data Cache Vision Public Cloud(s) Predrag Buncic, October 3, 2013 ECFA Workshop Aix-Les-Bains - 25 AliEn

26 And if this is not enough 18,688 nodes with total of 299,008 processor cores and one GPU per node. And, there are spare CPU cycles available Ideal for event generators/simulation type workloads (60-70% of all CPU cycles used by the experiments). Simulation frameworks must be adapted to efficiently use such resources where available Predrag Buncic, October 3, 2013 ECFA Workshop Aix-Les-Bains - 26

27 Conclusions Resources needed for Computing at HL-LHC are going to be large but not unprecedented Data volume is going to grow dramatically Projected CPU needs are reaching the Moore s law ceiling Grid growth of 25% per year is by far not sufficient Speeding up the simulation is very important Parallelization at all levels is needed for improving performance (application, I/O) Requires reengineering of experiment software New technologies such as clouds and virtualization may help to reduce the complexity and help to fully utilize available resources Predrag Buncic, October 3, 2013 ECFA Workshop Aix-Les-Bains - 27

Next Generation Operating Systems

Next Generation Operating Systems Next Generation Operating Systems Zeljko Susnjar, Cisco CTG June 2015 The end of CPU scaling Future computing challenges Power efficiency Performance == parallelism Cisco Confidential 2 Paradox of the

More information

Write a technical report Present your results Write a workshop/conference paper (optional) Could be a real system, simulation and/or theoretical

Write a technical report Present your results Write a workshop/conference paper (optional) Could be a real system, simulation and/or theoretical Identify a problem Review approaches to the problem Propose a novel approach to the problem Define, design, prototype an implementation to evaluate your approach Could be a real system, simulation and/or

More information

The CMS Tier0 goes Cloud and Grid for LHC Run 2. Dirk Hufnagel (FNAL) for CMS Computing

The CMS Tier0 goes Cloud and Grid for LHC Run 2. Dirk Hufnagel (FNAL) for CMS Computing The CMS Tier0 goes Cloud and Grid for LHC Run 2 Dirk Hufnagel (FNAL) for CMS Computing CHEP, 13.04.2015 Overview Changes for the Tier0 between Run 1 and Run 2 CERN Agile Infrastructure (in GlideInWMS)

More information

(Possible) HEP Use Case for NDN. Phil DeMar; Wenji Wu NDNComm (UCLA) Sept. 28, 2015

(Possible) HEP Use Case for NDN. Phil DeMar; Wenji Wu NDNComm (UCLA) Sept. 28, 2015 (Possible) HEP Use Case for NDN Phil DeMar; Wenji Wu NDNComm (UCLA) Sept. 28, 2015 Outline LHC Experiments LHC Computing Models CMS Data Federation & AAA Evolving Computing Models & NDN Summary Phil DeMar:

More information

Tier0 plans and security and backup policy proposals

Tier0 plans and security and backup policy proposals Tier0 plans and security and backup policy proposals, CERN IT-PSS CERN - IT Outline Service operational aspects Hardware set-up in 2007 Replication set-up Test plan Backup and security policies CERN Oracle

More information

Flash Memory Arrays Enabling the Virtualized Data Center. July 2010

Flash Memory Arrays Enabling the Virtualized Data Center. July 2010 Flash Memory Arrays Enabling the Virtualized Data Center July 2010 2 Flash Memory Arrays Enabling the Virtualized Data Center This White Paper describes a new product category, the flash Memory Array,

More information

Status and Evolution of ATLAS Workload Management System PanDA

Status and Evolution of ATLAS Workload Management System PanDA Status and Evolution of ATLAS Workload Management System PanDA Univ. of Texas at Arlington GRID 2012, Dubna Outline Overview PanDA design PanDA performance Recent Improvements Future Plans Why PanDA The

More information

The CMS openstack, opportunistic, overlay, online-cluster Cloud (CMSooooCloud)

The CMS openstack, opportunistic, overlay, online-cluster Cloud (CMSooooCloud) 15-19 April 2013, Bologna Italy HEPiX Spring 2013 Workshop Wojciech Ozga Faculty of Computer Science, Electronics and Telecommunication AGH University of Science and Technology in Krakow, Poland CERN,

More information

Building a Volunteer Cloud

Building a Volunteer Cloud Building a Volunteer Cloud Ben Segal, Predrag Buncic, David Garcia Quintas / CERN Daniel Lombrana Gonzalez / University of Extremadura Artem Harutyunyan / Yerevan Physics Institute Jarno Rantala / Tampere

More information

Outdated Architectures Are Holding Back the Cloud

Outdated Architectures Are Holding Back the Cloud Outdated Architectures Are Holding Back the Cloud Flash Memory Summit Open Tutorial on Flash and Cloud Computing August 11,2011 Dr John R Busch Founder and CTO Schooner Information Technology JohnBusch@SchoonerInfoTechcom

More information

NT1: An example for future EISCAT_3D data centre and archiving?

NT1: An example for future EISCAT_3D data centre and archiving? March 10, 2015 1 NT1: An example for future EISCAT_3D data centre and archiving? John White NeIC xx March 10, 2015 2 Introduction High Energy Physics and Computing Worldwide LHC Computing Grid Nordic Tier

More information

DSS. High performance storage pools for LHC. Data & Storage Services. Łukasz Janyst. on behalf of the CERN IT-DSS group

DSS. High performance storage pools for LHC. Data & Storage Services. Łukasz Janyst. on behalf of the CERN IT-DSS group DSS High performance storage pools for LHC Łukasz Janyst on behalf of the CERN IT-DSS group CERN IT Department CH-1211 Genève 23 Switzerland www.cern.ch/it Introduction The goal of EOS is to provide a

More information

Performance monitoring at CERN openlab. July 20 th 2012 Andrzej Nowak, CERN openlab

Performance monitoring at CERN openlab. July 20 th 2012 Andrzej Nowak, CERN openlab Performance monitoring at CERN openlab July 20 th 2012 Andrzej Nowak, CERN openlab Data flow Reconstruction Selection and reconstruction Online triggering and filtering in detectors Raw Data (100%) Event

More information

Novel, Highly-Parallel Software for the Online Storage System of the ATLAS Experiment at CERN: Design and Performances

Novel, Highly-Parallel Software for the Online Storage System of the ATLAS Experiment at CERN: Design and Performances Novel, Highly-Parallel Software for the Online Storage System of the ATLAS Experiment at CERN: Design and Performances Tommaso Colombo a,b Wainer Vandelli b a Università degli Studi di Pavia b CERN IEEE

More information

Capacity Estimation for Linux Workloads

Capacity Estimation for Linux Workloads Capacity Estimation for Linux Workloads Session L985 David Boyes Sine Nomine Associates 1 Agenda General Capacity Planning Issues Virtual Machine History and Value Unique Capacity Issues in Virtual Machines

More information

RO-11-NIPNE, evolution, user support, site and software development. IFIN-HH, DFCTI, LHCb Romanian Team

RO-11-NIPNE, evolution, user support, site and software development. IFIN-HH, DFCTI, LHCb Romanian Team IFIN-HH, DFCTI, LHCb Romanian Team Short overview: The old RO-11-NIPNE site New requirements from the LHCb team User support ( solution offered). Data reprocessing 2012 facts Future plans The old RO-11-NIPNE

More information

Datacenter Operating Systems

Datacenter Operating Systems Datacenter Operating Systems CSE451 Simon Peter With thanks to Timothy Roscoe (ETH Zurich) Autumn 2015 This Lecture What s a datacenter Why datacenters Types of datacenters Hyperscale datacenters Major

More information

Virtualization, Grid, Cloud: Integration Paths for Scientific Computing

Virtualization, Grid, Cloud: Integration Paths for Scientific Computing Virtualization, Grid, Cloud: Integration Paths for Scientific Computing Or, where and how will my efficient large-scale computing applications be executed? D. Salomoni, INFN Tier-1 Computing Manager Davide.Salomoni@cnaf.infn.it

More information

IT of SPIM Data Storage and Compression. EMBO Course - August 27th! Jeff Oegema, Peter Steinbach, Oscar Gonzalez

IT of SPIM Data Storage and Compression. EMBO Course - August 27th! Jeff Oegema, Peter Steinbach, Oscar Gonzalez IT of SPIM Data Storage and Compression EMBO Course - August 27th Jeff Oegema, Peter Steinbach, Oscar Gonzalez 1 Talk Outline Introduction and the IT Team SPIM Data Flow Capture, Compression, and the Data

More information

Today: Data Centers & Cloud Computing" Data Centers"

Today: Data Centers & Cloud Computing Data Centers Today: Data Centers & Cloud Computing" Data Centers Cloud Computing Lecture 25, page 1 Data Centers" Large server and storage farms Used by enterprises to run server applications Used by Internet companies

More information

US NSF s Scientific Software Innovation Institutes

US NSF s Scientific Software Innovation Institutes US NSF s Scientific Software Innovation Institutes S 2 I 2 awards invest in long-term projects which will realize sustained software infrastructure that is integral to doing transformative science. (Can

More information

Data Centers and Cloud Computing

Data Centers and Cloud Computing Data Centers and Cloud Computing CS377 Guest Lecture Tian Guo 1 Data Centers and Cloud Computing Intro. to Data centers Virtualization Basics Intro. to Cloud Computing Case Study: Amazon EC2 2 Data Centers

More information

ATLAS Software and Computing Week April 4-8, 2011 General News

ATLAS Software and Computing Week April 4-8, 2011 General News ATLAS Software and Computing Week April 4-8, 2011 General News Refactor requests for resources (originally requested in 2010) by expected running conditions (running in 2012 with shutdown in 2013) 20%

More information

How Solace Message Routers Reduce the Cost of IT Infrastructure

How Solace Message Routers Reduce the Cost of IT Infrastructure How Message Routers Reduce the Cost of IT Infrastructure This paper explains how s innovative solution can significantly reduce the total cost of ownership of your messaging middleware platform and IT

More information

Betriebssystem-Virtualisierung auf einem Rechencluster am SCC mit heterogenem Anwendungsprofil

Betriebssystem-Virtualisierung auf einem Rechencluster am SCC mit heterogenem Anwendungsprofil Betriebssystem-Virtualisierung auf einem Rechencluster am SCC mit heterogenem Anwendungsprofil Volker Büge 1, Marcel Kunze 2, OIiver Oberst 1,2, Günter Quast 1, Armin Scheurer 1 1) Institut für Experimentelle

More information

Big Data and Storage Management at the Large Hadron Collider

Big Data and Storage Management at the Large Hadron Collider Big Data and Storage Management at the Large Hadron Collider Dirk Duellmann CERN IT, Data & Storage Services Accelerating Science and Innovation CERN was founded 1954: 12 European States Science for Peace!

More information

HYPER-CONVERGED INFRASTRUCTURE STRATEGIES

HYPER-CONVERGED INFRASTRUCTURE STRATEGIES 1 HYPER-CONVERGED INFRASTRUCTURE STRATEGIES MYTH BUSTING & THE FUTURE OF WEB SCALE IT 2 ROADMAP INFORMATION DISCLAIMER EMC makes no representation and undertakes no obligations with regard to product planning

More information

Achieving Nanosecond Latency Between Applications with IPC Shared Memory Messaging

Achieving Nanosecond Latency Between Applications with IPC Shared Memory Messaging Achieving Nanosecond Latency Between Applications with IPC Shared Memory Messaging In some markets and scenarios where competitive advantage is all about speed, speed is measured in micro- and even nano-seconds.

More information

ioscale: The Holy Grail for Hyperscale

ioscale: The Holy Grail for Hyperscale ioscale: The Holy Grail for Hyperscale The New World of Hyperscale Hyperscale describes new cloud computing deployments where hundreds or thousands of distributed servers support millions of remote, often

More information

Delivering Quality in Software Performance and Scalability Testing

Delivering Quality in Software Performance and Scalability Testing Delivering Quality in Software Performance and Scalability Testing Abstract Khun Ban, Robert Scott, Kingsum Chow, and Huijun Yan Software and Services Group, Intel Corporation {khun.ban, robert.l.scott,

More information

Parallelism and Cloud Computing

Parallelism and Cloud Computing Parallelism and Cloud Computing Kai Shen Parallel Computing Parallel computing: Process sub tasks simultaneously so that work can be completed faster. For instances: divide the work of matrix multiplication

More information

Cloud Based Application Architectures using Smart Computing

Cloud Based Application Architectures using Smart Computing Cloud Based Application Architectures using Smart Computing How to Use this Guide Joyent Smart Technology represents a sophisticated evolution in cloud computing infrastructure. Most cloud computing products

More information

Software-defined Storage Architecture for Analytics Computing

Software-defined Storage Architecture for Analytics Computing Software-defined Storage Architecture for Analytics Computing Arati Joshi Performance Engineering Colin Eldridge File System Engineering Carlos Carrero Product Management June 2015 Reference Architecture

More information

Unisys ClearPath Forward Fabric Based Platform to Power the Weather Enterprise

Unisys ClearPath Forward Fabric Based Platform to Power the Weather Enterprise Unisys ClearPath Forward Fabric Based Platform to Power the Weather Enterprise Introducing Unisys All in One software based weather platform designed to reduce server space, streamline operations, consolidate

More information

Data storage services at CC-IN2P3

Data storage services at CC-IN2P3 Centre de Calcul de l Institut National de Physique Nucléaire et de Physique des Particules Data storage services at CC-IN2P3 Jean-Yves Nief Agenda Hardware: Storage on disk. Storage on tape. Software:

More information

Cloud Computing and Amazon Web Services

Cloud Computing and Amazon Web Services Cloud Computing and Amazon Web Services Gary A. McGilvary edinburgh data.intensive research 1 OUTLINE 1. An Overview of Cloud Computing 2. Amazon Web Services 3. Amazon EC2 Tutorial 4. Conclusions 2 CLOUD

More information

Introduction to the NI Real-Time Hypervisor

Introduction to the NI Real-Time Hypervisor Introduction to the NI Real-Time Hypervisor 1 Agenda 1) NI Real-Time Hypervisor overview 2) Basics of virtualization technology 3) Configuring and using Real-Time Hypervisor systems 4) Performance and

More information

Status of Grid Activities in Pakistan. FAWAD SAEED National Centre For Physics, Pakistan

Status of Grid Activities in Pakistan. FAWAD SAEED National Centre For Physics, Pakistan Status of Grid Activities in Pakistan FAWAD SAEED National Centre For Physics, Pakistan 1 Introduction of NCP-LCG2 q NCP-LCG2 is the only Tier-2 centre in Pakistan for Worldwide LHC computing Grid (WLCG).

More information

Enabling Technologies for Distributed Computing

Enabling Technologies for Distributed Computing Enabling Technologies for Distributed Computing Dr. Sanjay P. Ahuja, Ph.D. Fidelity National Financial Distinguished Professor of CIS School of Computing, UNF Multi-core CPUs and Multithreading Technologies

More information

Copyright 2013, Oracle and/or its affiliates. All rights reserved.

Copyright 2013, Oracle and/or its affiliates. All rights reserved. 1 Oracle SPARC Server for Enterprise Computing Dr. Heiner Bauch Senior Account Architect 19. April 2013 2 The following is intended to outline our general product direction. It is intended for information

More information

Solving the Mysteries of the Universe with Big Data

Solving the Mysteries of the Universe with Big Data Solving the Mysteries of the Universe with Big Data Sverre Jarp Former CERN openlab CTO Big Data Innovation Summit, Stockholm, 8 th May 2014 Accelerating Science and Innovation 1 What is CERN? The European

More information

Improving the performance of data servers on multicore architectures. Fabien Gaud

Improving the performance of data servers on multicore architectures. Fabien Gaud Improving the performance of data servers on multicore architectures Fabien Gaud Grenoble University Advisors: Jean-Bernard Stefani, Renaud Lachaize and Vivien Quéma Sardes (INRIA/LIG) December 2, 2010

More information

Why Choose VMware vsphere for Desktop Virtualization? WHITE PAPER

Why Choose VMware vsphere for Desktop Virtualization? WHITE PAPER Why Choose VMware vsphere for Desktop Virtualization? WHITE PAPER Table of Contents Thin, Legacy-Free, Purpose-Built Hypervisor.... 3 More Secure with Smaller Footprint.... 4 Less Downtime Caused by Patches...

More information

A Close Look at PCI Express SSDs. Shirish Jamthe Director of System Engineering Virident Systems, Inc. August 2011

A Close Look at PCI Express SSDs. Shirish Jamthe Director of System Engineering Virident Systems, Inc. August 2011 A Close Look at PCI Express SSDs Shirish Jamthe Director of System Engineering Virident Systems, Inc. August 2011 Macro Datacenter Trends Key driver: Information Processing Data Footprint (PB) CAGR: 100%

More information

Introduction to Cloud Computing

Introduction to Cloud Computing Introduction to Cloud Computing Cloud Computing I (intro) 15 319, spring 2010 2 nd Lecture, Jan 14 th Majd F. Sakr Lecture Motivation General overview on cloud computing What is cloud computing Services

More information

The CMS openstack, opportunistic, overlay, online-cluster Cloud (CMSooooCloud)"

The CMS openstack, opportunistic, overlay, online-cluster Cloud (CMSooooCloud) The CMS openstack, opportunistic, overlay, online-cluster Cloud (CMSooooCloud)" J.A. Coarasa " CERN, Geneva, Switzerland" for the CMS TriDAS group." " CHEP2013, 14-18 October 2013, Amsterdam, The Netherlands

More information

Beyond High Performance Computing: What Matters to CERN

Beyond High Performance Computing: What Matters to CERN Beyond High Performance Computing: What Matters to CERN Pierre VANDE VYVRE for the ALICE Collaboration ALICE Data Acquisition Project Leader CERN, Geneva, Switzerland 2 CERN CERN is the world's largest

More information

Data Centers and Cloud Computing. Data Centers. MGHPCC Data Center. Inside a Data Center

Data Centers and Cloud Computing. Data Centers. MGHPCC Data Center. Inside a Data Center Data Centers and Cloud Computing Intro. to Data centers Virtualization Basics Intro. to Cloud Computing Data Centers Large server and storage farms 1000s of servers Many TBs or PBs of data Used by Enterprises

More information

Batch and Cloud overview. Andrew McNab University of Manchester GridPP and LHCb

Batch and Cloud overview. Andrew McNab University of Manchester GridPP and LHCb Batch and Cloud overview Andrew McNab University of Manchester GridPP and LHCb Overview Assumptions Batch systems The Grid Pilot Frameworks DIRAC Virtual Machines Vac Vcycle Tier-2 Evolution Containers

More information

Power Management in Cloud Computing using Green Algorithm. -Kushal Mehta COP 6087 University of Central Florida

Power Management in Cloud Computing using Green Algorithm. -Kushal Mehta COP 6087 University of Central Florida Power Management in Cloud Computing using Green Algorithm -Kushal Mehta COP 6087 University of Central Florida Motivation Global warming is the greatest environmental challenge today which is caused by

More information

Introduction to grid technologies, parallel and cloud computing. Alaa Osama Allam Saida Saad Mohamed Mohamed Ibrahim Gaber

Introduction to grid technologies, parallel and cloud computing. Alaa Osama Allam Saida Saad Mohamed Mohamed Ibrahim Gaber Introduction to grid technologies, parallel and cloud computing Alaa Osama Allam Saida Saad Mohamed Mohamed Ibrahim Gaber OUTLINES Grid Computing Parallel programming technologies (MPI- Open MP-Cuda )

More information

CMS Experience Provisioning Cloud Resources with GlideinWMS. Anthony Tiradani HTCondor Week 2015 20 May 2015

CMS Experience Provisioning Cloud Resources with GlideinWMS. Anthony Tiradani HTCondor Week 2015 20 May 2015 CMS Experience Provisioning Cloud Resources with GlideinWMS Anthony Tiradani Week 2015 20 May 2015 glideinwms Quick Facts glideinwms is an open- source Fermilab CompuJng Sector product driven by CMS Heavy

More information

Introducing EEMBC Cloud and Big Data Server Benchmarks

Introducing EEMBC Cloud and Big Data Server Benchmarks Introducing EEMBC Cloud and Big Data Server Benchmarks Quick Background: Industry-Standard Benchmarks for the Embedded Industry EEMBC formed in 1997 as non-profit consortium Defining and developing application-specific

More information

BENCHMARKING CLOUD DATABASES CASE STUDY on HBASE, HADOOP and CASSANDRA USING YCSB

BENCHMARKING CLOUD DATABASES CASE STUDY on HBASE, HADOOP and CASSANDRA USING YCSB BENCHMARKING CLOUD DATABASES CASE STUDY on HBASE, HADOOP and CASSANDRA USING YCSB Planet Size Data!? Gartner s 10 key IT trends for 2012 unstructured data will grow some 80% over the course of the next

More information

TOP FIVE REASONS WHY CUSTOMERS USE EMC AND VMWARE TO VIRTUALIZE ORACLE ENVIRONMENTS

TOP FIVE REASONS WHY CUSTOMERS USE EMC AND VMWARE TO VIRTUALIZE ORACLE ENVIRONMENTS TOP FIVE REASONS WHY CUSTOMERS USE EMC AND VMWARE TO VIRTUALIZE ORACLE ENVIRONMENTS Leverage EMC and VMware To Improve The Return On Your Oracle Investment ESSENTIALS Better Performance At Lower Cost Run

More information

Parallel Computing: Strategies and Implications. Dori Exterman CTO IncrediBuild.

Parallel Computing: Strategies and Implications. Dori Exterman CTO IncrediBuild. Parallel Computing: Strategies and Implications Dori Exterman CTO IncrediBuild. In this session we will discuss Multi-threaded vs. Multi-Process Choosing between Multi-Core or Multi- Threaded development

More information

Windows Server Virtualization An Overview

Windows Server Virtualization An Overview Microsoft Corporation Published: May 2006 Abstract Today s business climate is more challenging than ever and businesses are under constant pressure to lower costs while improving overall operational efficiency.

More information

Cluster, Grid, Cloud Concepts

Cluster, Grid, Cloud Concepts Cluster, Grid, Cloud Concepts Kalaiselvan.K Contents Section 1: Cluster Section 2: Grid Section 3: Cloud Cluster An Overview Need for a Cluster Cluster categorizations A computer cluster is a group of

More information

SOFTWARE-DEFINED: MAKING CLOUDS MORE EFFICIENT. Julian Chesterfield, Director of Emerging Technologies

SOFTWARE-DEFINED: MAKING CLOUDS MORE EFFICIENT. Julian Chesterfield, Director of Emerging Technologies SOFTWARE-DEFINED: MAKING CLOUDS MORE EFFICIENT Julian Chesterfield, Director of Emerging Technologies DEFINING SOFTWARE DEFINED! FLEXIBILITY IN SOFTWARE Leveraging commodity CPU cycles to provide traditional

More information

Arif Goelmhd Goelammohamed Solutions Architect. @agoelammohamed. Hyperconverged Infrastructure: The How-To and Why Now?

Arif Goelmhd Goelammohamed Solutions Architect. @agoelammohamed. Hyperconverged Infrastructure: The How-To and Why Now? Arif Goelmhd Goelammohamed Solutions Architect @agoelammohamed Hyperconverged Infrastructure: The How-To and Why Now? Agenda: 1. SimpliVity Overview 2. The Problem 3. The Solution 4. Demo Simplify IT with

More information

WM Technical Evolution Group Report

WM Technical Evolution Group Report WM Technical Evolution Group Report Davide Salomoni, INFN for the WM TEG February 7, 2012 The WM TEG, organization Mailing list, wlcg-teg-workload-mgmt@cern.ch 48 people currently subscribed; representation

More information

Preview of a Novel Architecture for Large Scale Storage

Preview of a Novel Architecture for Large Scale Storage Preview of a Novel Architecture for Large Scale Storage Andreas Petzold, Christoph-Erdmann Pfeiler, Jos van Wezel Steinbuch Centre for Computing STEINBUCH CENTRE FOR COMPUTING - SCC KIT University of the

More information

Stream Processing on GPUs Using Distributed Multimedia Middleware

Stream Processing on GPUs Using Distributed Multimedia Middleware Stream Processing on GPUs Using Distributed Multimedia Middleware Michael Repplinger 1,2, and Philipp Slusallek 1,2 1 Computer Graphics Lab, Saarland University, Saarbrücken, Germany 2 German Research

More information

Long term analysis in HEP: Use of virtualization and emulation techniques

Long term analysis in HEP: Use of virtualization and emulation techniques Long term analysis in HEP: Use of virtualization and emulation techniques Yves Kemp DESY IT First Workshop on Data Preservation and Long Term Analysis in HEP, DESY 26.1.2009 Outline Why virtualization

More information

Programming models for heterogeneous computing. Manuel Ujaldón Nvidia CUDA Fellow and A/Prof. Computer Architecture Department University of Malaga

Programming models for heterogeneous computing. Manuel Ujaldón Nvidia CUDA Fellow and A/Prof. Computer Architecture Department University of Malaga Programming models for heterogeneous computing Manuel Ujaldón Nvidia CUDA Fellow and A/Prof. Computer Architecture Department University of Malaga Talk outline [30 slides] 1. Introduction [5 slides] 2.

More information

Datacenters and Cloud Computing. Jia Rao Assistant Professor in CS http://cs.uccs.edu/~jrao/cs5540/spring2014/index.html

Datacenters and Cloud Computing. Jia Rao Assistant Professor in CS http://cs.uccs.edu/~jrao/cs5540/spring2014/index.html Datacenters and Cloud Computing Jia Rao Assistant Professor in CS http://cs.uccs.edu/~jrao/cs5540/spring2014/index.html What is Cloud Computing? A model for enabling ubiquitous, convenient, ondemand network

More information

Microsoft Private Cloud Fast Track

Microsoft Private Cloud Fast Track Microsoft Private Cloud Fast Track Microsoft Private Cloud Fast Track is a reference architecture designed to help build private clouds by combining Microsoft software with Nutanix technology to decrease

More information

Testing & Assuring Mobile End User Experience Before Production. Neotys

Testing & Assuring Mobile End User Experience Before Production. Neotys Testing & Assuring Mobile End User Experience Before Production Neotys Agenda Introduction The challenges Best practices NeoLoad mobile capabilities Mobile devices are used more and more At Home In 2014,

More information

Virtualization. (and cloud computing at CERN)

Virtualization. (and cloud computing at CERN) Virtualization (and cloud computing at CERN) Ulrich Schwickerath Special thanks: Sebastien Goasguen Belmiro Moreira, Ewan Roche, Romain Wartel See also related presentations: CloudViews2010 conference,

More information

Xeon+FPGA Platform for the Data Center

Xeon+FPGA Platform for the Data Center Xeon+FPGA Platform for the Data Center ISCA/CARL 2015 PK Gupta, Director of Cloud Platform Technology, DCG/CPG Overview Data Center and Workloads Xeon+FPGA Accelerator Platform Applications and Eco-system

More information

Data Centers and Cloud Computing. Data Centers

Data Centers and Cloud Computing. Data Centers Data Centers and Cloud Computing Intro. to Data centers Virtualization Basics Intro. to Cloud Computing 1 Data Centers Large server and storage farms 1000s of servers Many TBs or PBs of data Used by Enterprises

More information

Virtuoso and Database Scalability

Virtuoso and Database Scalability Virtuoso and Database Scalability By Orri Erling Table of Contents Abstract Metrics Results Transaction Throughput Initializing 40 warehouses Serial Read Test Conditions Analysis Working Set Effect of

More information

Lecture 2 Cloud Computing & Virtualization. Cloud Application Development (SE808, School of Software, Sun Yat-Sen University) Yabo (Arber) Xu

Lecture 2 Cloud Computing & Virtualization. Cloud Application Development (SE808, School of Software, Sun Yat-Sen University) Yabo (Arber) Xu Lecture 2 Cloud Computing & Virtualization Cloud Application Development (SE808, School of Software, Sun Yat-Sen University) Yabo (Arber) Xu Outline Introduction to Virtualization The Major Approaches

More information

Solving the Mysteries of the Universe with Big Data

Solving the Mysteries of the Universe with Big Data Solving the Mysteries of the Universe with Big Data Sverre Jarp CERN openlab CTO Big Data Innovation Summit, Boston, 12 th September 2013 Accelerating Science and Innovation 1 What is CERN? The European

More information

Enabling Technologies for Distributed and Cloud Computing

Enabling Technologies for Distributed and Cloud Computing Enabling Technologies for Distributed and Cloud Computing Dr. Sanjay P. Ahuja, Ph.D. 2010-14 FIS Distinguished Professor of Computer Science School of Computing, UNF Multi-core CPUs and Multithreading

More information

The Mainframe Virtualization Advantage: How to Save Over Million Dollars Using an IBM System z as a Linux Cloud Server

The Mainframe Virtualization Advantage: How to Save Over Million Dollars Using an IBM System z as a Linux Cloud Server Research Report The Mainframe Virtualization Advantage: How to Save Over Million Dollars Using an IBM System z as a Linux Cloud Server Executive Summary Information technology (IT) executives should be

More information

Protect Data... in the Cloud

Protect Data... in the Cloud QUASICOM Private Cloud Backups with ExaGrid Deduplication Disk Arrays Martin Lui Senior Solution Consultant Quasicom Systems Limited Protect Data...... in the Cloud 1 Mobile Computing Users work with their

More information

Optimizing Storage for Better TCO in Oracle Environments. Part 1: Management INFOSTOR. Executive Brief

Optimizing Storage for Better TCO in Oracle Environments. Part 1: Management INFOSTOR. Executive Brief Optimizing Storage for Better TCO in Oracle Environments INFOSTOR Executive Brief a QuinStreet Excutive Brief. 2012 To the casual observer, and even to business decision makers who don t work in information

More information

Virtualization. Dr. Yingwu Zhu

Virtualization. Dr. Yingwu Zhu Virtualization Dr. Yingwu Zhu What is virtualization? Virtualization allows one computer to do the job of multiple computers. Virtual environments let one computer host multiple operating systems at the

More information

A Novel Cloud Based Elastic Framework for Big Data Preprocessing

A Novel Cloud Based Elastic Framework for Big Data Preprocessing School of Systems Engineering A Novel Cloud Based Elastic Framework for Big Data Preprocessing Omer Dawelbeit and Rachel McCrindle October 21, 2014 University of Reading 2008 www.reading.ac.uk Overview

More information

Cloud Storage. Parallels. Performance Benchmark Results. White Paper. www.parallels.com

Cloud Storage. Parallels. Performance Benchmark Results. White Paper. www.parallels.com Parallels Cloud Storage White Paper Performance Benchmark Results www.parallels.com Table of Contents Executive Summary... 3 Architecture Overview... 3 Key Features... 4 No Special Hardware Requirements...

More information

Long Term Data Preserva0on LTDP = Data Sharing In Time and Space

Long Term Data Preserva0on LTDP = Data Sharing In Time and Space Long Term Data Preserva0on LTDP = Data Sharing In Time and Space Jamie.Shiers@cern.ch Big Data, Open Data Workshop, May 2014 International Collaboration for Data Preservation and Long Term Analysis in

More information

Emerging Technology for the Next Decade

Emerging Technology for the Next Decade Emerging Technology for the Next Decade Cloud Computing Keynote Presented by Charles Liang, President & CEO Super Micro Computer, Inc. What is Cloud Computing? Cloud computing is Internet-based computing,

More information

Hyper-converged IT drives: - TCO cost savings - data protection - amazing operational excellence

Hyper-converged IT drives: - TCO cost savings - data protection - amazing operational excellence Hyper-converged IT drives: - TCO cost savings - data protection - amazing operational excellence Sebastian Nowicki SimpliVity is one of the biggest innovations in enterprise computing since ware. ~John

More information

Cloud: App-Centric Scalability, Availability, Reliability and Security. Prakash Sinha, Director, Product Management October 27, 2009

Cloud: App-Centric Scalability, Availability, Reliability and Security. Prakash Sinha, Director, Product Management October 27, 2009 Cloud: App-Centric Scalability, Availability, Reliability and Security Prakash Sinha, Director, Product Management October 27, 2009 Agenda Cloud Computing Evolution Creating an Enterprise Cloud Use Cases,

More information

An Energy-Aware Design and Reporting Tool for On-Demand Service Infrastructures

An Energy-Aware Design and Reporting Tool for On-Demand Service Infrastructures An Energy-Aware Design and Reporting Tool for On-Demand Service Infrastructures Miguel Gómez, David Perales and Emilio J. Torres E2GC2 Workshop. IEEE/ACM GRID 2009. Oct 13 th, 2009 Outline 01 Introduction

More information

MaxDeploy Ready. Hyper- Converged Virtualization Solution. With SanDisk Fusion iomemory products

MaxDeploy Ready. Hyper- Converged Virtualization Solution. With SanDisk Fusion iomemory products MaxDeploy Ready Hyper- Converged Virtualization Solution With SanDisk Fusion iomemory products MaxDeploy Ready products are configured and tested for support with Maxta software- defined storage and with

More information

Impact of Big Data growth On Transparent Computing

Impact of Big Data growth On Transparent Computing Impact of Big Data growth On Transparent Computing Michael A. Greene Intel Vice President, Software and Services Group, General Manager, System Technologies and Optimization 1 Transparent Computing (TC)

More information

Building a Scalable Big Data Infrastructure for Dynamic Workflows

Building a Scalable Big Data Infrastructure for Dynamic Workflows Building a Scalable Big Data Infrastructure for Dynamic Workflows INTRODUCTION Organizations of all types and sizes are looking to big data to help them make faster, more intelligent decisions. Many efforts

More information

Online Performance Monitoring of the Third ALICE Data Challenge (ADC III)

Online Performance Monitoring of the Third ALICE Data Challenge (ADC III) EUROPEAN ORGANIZATION FOR NUCLEAR RESEARCH European Laboratory for Particle Physics Publication ALICE reference number ALICE-PUB-1- version 1. Institute reference number Date of last change 1-1-17 Online

More information

Last time. Data Center as a Computer. Today. Data Center Construction (and management)

Last time. Data Center as a Computer. Today. Data Center Construction (and management) Last time Data Center Construction (and management) Johan Tordsson Department of Computing Science 1. Common (Web) application architectures N-tier applications Load Balancers Application Servers Databases

More information

DIABLO TECHNOLOGIES MEMORY CHANNEL STORAGE AND VMWARE VIRTUAL SAN : VDI ACCELERATION

DIABLO TECHNOLOGIES MEMORY CHANNEL STORAGE AND VMWARE VIRTUAL SAN : VDI ACCELERATION DIABLO TECHNOLOGIES MEMORY CHANNEL STORAGE AND VMWARE VIRTUAL SAN : VDI ACCELERATION A DIABLO WHITE PAPER AUGUST 2014 Ricky Trigalo Director of Business Development Virtualization, Diablo Technologies

More information

Computing in High- Energy-Physics: How Virtualization meets the Grid

Computing in High- Energy-Physics: How Virtualization meets the Grid Computing in High- Energy-Physics: How Virtualization meets the Grid Yves Kemp Institut für Experimentelle Kernphysik Universität Karlsruhe Yves Kemp Barcelona, 10/23/2006 Outline: Problems encountered

More information

A REVIEW PAPER ON THE HADOOP DISTRIBUTED FILE SYSTEM

A REVIEW PAPER ON THE HADOOP DISTRIBUTED FILE SYSTEM A REVIEW PAPER ON THE HADOOP DISTRIBUTED FILE SYSTEM Sneha D.Borkar 1, Prof.Chaitali S.Surtakar 2 Student of B.E., Information Technology, J.D.I.E.T, sborkar95@gmail.com Assistant Professor, Information

More information

HTCondor at the RAL Tier-1

HTCondor at the RAL Tier-1 HTCondor at the RAL Tier-1 Andrew Lahiff, Alastair Dewhurst, John Kelly, Ian Collier, James Adams STFC Rutherford Appleton Laboratory HTCondor Week 2014 Outline Overview of HTCondor at RAL Monitoring Multi-core

More information

Context-aware cloud computing for HEP

Context-aware cloud computing for HEP Department of Physics and Astronomy, University of Victoria, Victoria, British Columbia, Canada V8W 2Y2 E-mail: rsobie@uvic.ca The use of cloud computing is increasing in the field of high-energy physics

More information

Data Centers and Cloud Computing. Data Centers

Data Centers and Cloud Computing. Data Centers Data Centers and Cloud Computing Slides courtesy of Tim Wood 1 Data Centers Large server and storage farms 1000s of servers Many TBs or PBs of data Used by Enterprises for server applications Internet

More information

RAMCloud and the Low- Latency Datacenter. John Ousterhout Stanford University

RAMCloud and the Low- Latency Datacenter. John Ousterhout Stanford University RAMCloud and the Low- Latency Datacenter John Ousterhout Stanford University Most important driver for innovation in computer systems: Rise of the datacenter Phase 1: large scale Phase 2: low latency Introduction

More information

Cloud/SaaS enablement of existing applications

Cloud/SaaS enablement of existing applications Cloud/SaaS enablement of existing applications GigaSpaces: Nati Shalom, CTO & Founder About GigaSpaces Technologies Enabling applications to run a distributed cluster as if it was a single machine 75+

More information

An Oracle White Paper August 2011. Oracle VM 3: Server Pool Deployment Planning Considerations for Scalability and Availability

An Oracle White Paper August 2011. Oracle VM 3: Server Pool Deployment Planning Considerations for Scalability and Availability An Oracle White Paper August 2011 Oracle VM 3: Server Pool Deployment Planning Considerations for Scalability and Availability Note This whitepaper discusses a number of considerations to be made when

More information