I/O Scheduling Service for Multi-Application Clusters

Size: px
Start display at page:

Download "I/O Scheduling Service for Multi-Application Clusters"

Transcription

1 I/O Scheduling Service for Multi-Application Clusters, Guillaume Huard, Yves Denneulin Laboratoire ID-IMAG (UMR 5132), Grenoble, France. BULL - HPC, Échirolles, France. Przemyslaw Sowa sowa@icis.pcz.pl Institute of Computer and Information Sciences Czestochowa University of Technology, Poland.

2 Plan Part 1 - Parallel Input/Output and Clusters Part 2 - Controlling and Scheduling Multi-application I/O Part 3 - aioli, an Input/Output Scheduler for HPC Part 4 - Conclusion 2/23

3 Plan Part 1 - Parallel Input/Output and Clusters 1 Introduction Context Parallel I/O 2 Parallel I/O and Clusters Available Solutions 3 Objectives Part 2 - Controlling and Scheduling Multi-application I/O Part 3 - aioli, an Input/Output Scheduler for HPC Part 4 - Conclusion 3/23

4 Context Environment Clusters of SMPs Linux High Performance Computing Intensive I/O applications CPU bounded application I/O bounded application Remote hard drive I/O Parallel I/O Handling concurrent accesses to a same resource (a file) Accesses: different in size, in offset Example: a parallel matrix product 4/23

5 Parallel I/O - Example Parallel matrix product Specific parts to fetch according to the data distribution (columns/rows, BLOCK/BLOCK...) Matrix A Matrix B Matrix C a b c 1 2 3??? d e f 4 5 6??? g h i 7 8 9??? Matrices are stored "row by row" in files Matrix A a b c d e f g h i Matrix B /23

6 Parallel I/O - Example Parallel matrix product Specific parts to fetch according to the data distribution (columns/rows, BLOCK/BLOCK...) Matrix A Matrix B Matrix C a b c 1 2 3??? d e f 4 5 6??? g h i 7 8 9??? P0 1 read (n) n read (1) Matrices are stored "row by row" in files Matrix A a b c d e f g h i Matrix B /23

7 Parallel I/O - Example Parallel matrix product Specific parts to fetch according to the data distribution (columns/rows, BLOCK/BLOCK...) Matrix A Matrix B Matrix C a b c 1 2 3??? d e f 4 5 6??? g h i 7 8 9??? P0 1 read (n) n read (1) 1 read (n) n read (1) Matrices are stored "row by row" in files Matrix A a b c d e f g h i Matrix B /23

8 Parallel I/O - Example Parallel matrix product Specific parts to fetch according to the data distribution (columns/rows, BLOCK/BLOCK...) File decomposition: Lot of disjoint/contiguous requests at the same time lethal behaviour for I/O subsystem Matrix A Matrix B Matrix C a b c 1 2 3??? d e f 4 5 6??? g h i 7 8 9??? P0 Pi 1 read (n) n read (1) 1 read (n) n read (1) 1 read (n) n read (1) Matrix A Matrices are stored "row by row" in files a b c d e f g h i Pp 1 read (n) n read (1) Matrix B No defined order between requests many disk head movements n2 * read (n) n2 * n read (1) 5/23

9 Parallel I/O and Clusters Parallel I/O bottlenecks SMP 1 SMP 2 Compute nodes SMP 3 Network interconnection I/O server (NFS) SMP 4 SMP n Application 1 6/23

10 Parallel I/O and Clusters Parallel I/O bottlenecks SMP 1 SMP 2 Compute nodes SMP 3 Network interconnection I/O server (NFS) SMP 4 SMP n Application 1 Hypothesis: network has fewer impact than the I/O subsystems. 6/23

11 Parallel I/O Solutions (1/2) Performance constraints Reduce the number of requests: decrease overhead implied by the different syscalls Requests Scheduling: avoid expensive seeks and maximize large accessses Exploit cache mechanisms (read-ahead strategies,...) Available solutions Parallel File Systems : balance requests between several data servers Designed for Parallel I/O (PIOUS, VESTA,...): gave up in the time More generic (PVFS, GPFS, Lustre, Parallel NFS) +/- complete / +/- efficient Libraries (MPI I/O): Focus on performance and portability aspects Sophisticated API Development overhead / Language bindings No global coordination 7/23

12 Parallel I/O Solutions (2/2) Libraries in multi-application environment Application 1 Compute nodes SMP 1 SMP 2 SMP 3 ROMIO ROMIO Network Interconnection I/O server 1 I/O server 2 Storage nodes SMP 4 I/O server p SMP n Application 2 8/23

13 Parallel I/O Solutions (2/2) Libraries in multi-application environment I/O server Synchronous behaviour (2 concurrent applications execute a cat like operation)? Appli 1 Appli 2 No informations about applications, only about files File 1 File 2 SEEK? 8/23

14 Parallel I/O Solutions (2/2) Libraries in multi-application environment I/O server Synchronous behaviour (2 concurrent applications execute a cat like operation)? Appli 1 Appli 2 No informations about applications, only about files File 1 File 2 8/23

15 Parallel I/O Solutions (2/2) Libraries in multi-application environment I/O server Synchronous behaviour (2 concurrent applications execute a cat like operation)?! Appli 1 Appli 2 No informations about applications, only about files File 1 File 2 SEEK 8/23

16 Parallel I/O Solutions (2/2) Libraries in multi-application environment I/O server Synchronous behaviour (2 concurrent applications execute a cat like operation)?! Appli 1 Appli 2 No informations about applications, only about files File 1 File 2 8/23

17 Parallel I/O Solutions (2/2) Libraries in multi-application environment I/O server Synchronous behaviour (2 concurrent applications execute a cat like operation)?!! Appli 1 Appli 2 No informations about applications, only about files File 1 File 2 SEEK 8/23

18 Parallel I/O Solutions (2/2) Libraries in multi-application environment I/O server Synchronous behaviour (2 concurrent applications execute a cat like operation)?!! Appli 1 Appli 2 No informations about applications, only about files File 1 File 2 8/23

19 Parallel I/O Solutions (2/2) Libraries in multi-application environment I/O server Synchronous behaviour (2 concurrent applications execute a cat like operation)?!!! Appli 1 Appli 2 No informations about applications, only about files File 1 File 2 SEEK 8/23

20 Parallel I/O Solutions (2/2) Libraries in multi-application environment I/O server Synchronous behaviour (2 concurrent applications execute a cat like operation)?!!!! Appli 1 Appli 2 To switch from one file to the other implies a disk head movement File 1 File 2 Libraries are not suited global synchronization is required 8/23

21 A New Approach Objectives Supply Parallel I/O algorithms scheduling / aggregating / overlapping access mono-application efficiency Only through the use of the ubiquitous POSIX calls: open/read/write/lseek/close portability / simplicity Address requests in a global manner multi-application efficiency Naive approach: Processing all the requests from one application before serving another one Not suited for a cluster Tradeoff between fairness and performance 9/23

22 Plan Part 1 - Parallel Input/Output and Clusters Part 2 - Controlling and Scheduling Multi-application I/O 4 Scheduling in Multi-application Environment General Algorithm 5 Synchronous Behaviour Part 3 - aioli, an Input/Output Scheduler for HPC Part 4 - Conclusion 10/23

23 Multi-application Scheduling Appli 1 Appli2 SMP 1 Appli 3 SMP 2 I/O server 1 Compute nodes Appli 5 SMP 3 Appli 4 Network interconnection I/O server 2 Storage nodes SMP 4 Appli 7 I/O server p Appli n SMP n 11/23

24 Multi-application Scheduling Appli 1 Appli2 SMP 1 Appli 3 SMP 2 I/O server 1 Compute nodes Appli 5 SMP 3 Appli 4 Network interconnection I/O server 2 Storage nodes SMP 4 Appli 7 I/O server p Appli n SMP n 11/23

25 Multi-application Scheduling Appli 1 Appli2 SMP 1 Appli 3 SMP 2 Compute nodes Appli 5 SMP 3 Appli 4 Network interconnection I/O server (NFS) SMP 4 Appli 7 Appli n SMP n 11/23

26 Multi-application Scheduling Online problem: several requests from distinct applications are delivered to the file systems. Wished criteria : Efficiency with fairness constraints Maximize the minimum of instantaneous throughput for each application Algorithm : Multi-Level Feedback variant (quantum approach) Step 1 Processing Waiting A1 A2 A1 A2 A1 A2 A1 A3 Pre processing (offset dependance) A1 A1 A1 A1 A2 A2 q = 10, T = 20 Scheduling A2 A3 q = 10, T = 15 q = 10, T = 5 A3 A1 A1 A1 A1 A2 A2 A2 1 element requires 5 time units to be processed 11/23

27 Multi-application Scheduling Online problem: several requests from distinct applications are delivered to the file systems. Wished criteria : Efficiency with fairness constraints Maximize the minimum of instantaneous throughput for each application Algorithm : Multi-Level Feedback variant (quantum approach) Step 2 Processing Waiting A3 A1 A1 A1 A1 A2 A2 A2 A2 A2 A3 Pre processing (offset dependance) A1 A1 A1 A1 A2 A2 A2 A2 A2 A3 q = 20, T = 20 q = 20, T = 25 q = 10, T = 5 Scheduling A1 A1 A1 A1 A2 A2 A2 A2 A2 A3 1 element requires 5 time units to be processed 11/23

28 Multi-application Scheduling Online problem: several requests from distinct applications are delivered to the file systems. Wished criteria : Efficiency with fairness constraints Maximize the minimum of instantaneous throughput for each application Algorithm : Multi-Level Feedback variant (quantum approach) Step 3 Processing Waiting A1 A1 A1 A1 A2 A2 A2 A2 A2 A3 A3 A2 A3 A3 Pre processing (offset dependance) A2 A2 A2 A2 A2 A2 A3 A3 A3 A3 q = 40, T = 30 q = 20, T = 20 Scheduling A2 A2 A2 A2 A2 A2 A3 A3 A3 A3 1 element requires 5 time units to be processed 11/23

29 Multi-application Scheduling Online problem: several requests from distinct applications are delivered to the file systems. Wished criteria : Efficiency with fairness constraints Maximize the minimum of instantaneous throughput for each application Algorithm : Multi-Level Feedback variant (quantum approach) The grow of a quantum could be set for each application (QoS) Step 3 Processing Waiting A1 A1 A1 A1 A2 A2 A2 A2 A2 A3 A3 A2 A3 A3 Pre processing (offset dependance) A2 A2 A2 A2 A2 A2 A3 A3 A3 A3 q = 40, T = 30 q = 20, T = 20 Scheduling A2 A2 A2 A2 A2 A2 A3 A3 A3 A3 1 element requires 5 time units to be processed 11/23

30 Manage Synchronous Behaviour in an Efficient Way I/O server Synchronous behaviour (2 concurrent applications execute a cat like operation)?!!!! Appli 1 Appli 2 To switch from one file to the other implies a disk head movement Serialize and define dedicated windows File 1 File 2 12/23

31 Manage Synchronous Behaviour in an Efficient Way I/O server Synchronous behaviour (2 concurrent applications execute a cat like operation)? Coordination server Appli 1 Appli 2 Serialize and define dedicated windows File 1 File 2 SEEK? 12/23

32 Manage Synchronous Behaviour in an Efficient Way I/O server Synchronous behaviour (2 concurrent applications execute a cat like operation)? Coordination server Appli 1 Appli 2 Serialize and define dedicated windows File 1 File 2 12/23

33 Manage Synchronous Behaviour in an Efficient Way I/O server Synchronous behaviour (2 concurrent applications execute a cat like operation)? Coordination server Appli 1 Appli 2 Serialize and define dedicated windows File 1 File 2 12/23

34 Manage Synchronous Behaviour in an Efficient Way I/O server Synchronous behaviour (2 concurrent applications execute a cat like operation)? Coordination server Appli 1 Appli 2 Serialize and define dedicated windows File 1 File 2 12/23

35 Manage Synchronous Behaviour in an Efficient Way I/O server Synchronous behaviour (2 concurrent applications execute a cat like operation)?! Coordination server Appli 1 Appli 2 Serialize and define dedicated windows File 1 File 2 SEEK 12/23

36 Manage Synchronous Behaviour in an Efficient Way I/O server Synchronous behaviour (2 concurrent applications execute a cat like operation)?! Coordination server Appli 1 Appli 2 Serialize and define dedicated windows File 1 File 2 12/23

37 Manage Synchronous Behaviour in an Efficient Way I/O server Synchronous behaviour (2 concurrent applications execute a cat like operation)?! Coordination server Appli 1 Appli 2 Serialize and define dedicated windows File 1 File 2 Decrease the number of seeks and exploit read-ahead mechanism 12/23

38 Plan Part 1 - Parallel Input/Output and Clusters Part 2 - Controlling and Scheduling Multi-application I/O Part 3 - aioli, an Input/Output Scheduler for HPC 6 aioli - Generic Framework 7 aioli - Evaluations Evaluations - Multi-nodes Eval - 2 applications Evaluations - 10 applications Part 4 - Conclusion 13/23

39 aioli - Generic Framework SMP 1 SMP 2 I/O server 1 Compute Network Storage nodes nodes interconnection SMP 3 I/O server 2 SMP 4 I/O server p Objectives SMP n A central point is required to apply our global strategy : model Client/Server like PANDA (explicit centralization) is not scalable Exploit available central point Major change : I/O systems become clients of our framework! 14/23

40 aioli - Framework Implementation Virtual File System (server side) syscalls layer userspace kernelspace NFS server XDR RPC Network layer aioli VFS Client Virtual File System ExtX ReiserFS aioli Optimize the VFS on server side 15/23

41 aioli - Framework Implementation Virtual File System (server side) NFS server (Version 3) NFS server syscalls layer aioli NFS client aioli userspace kernelspace XDR RPC Network layer Virtual File System ExtX ReiserFS Optimize the NFS server 15/23

42 aioli - Framework Implementation Virtual File System (server side) NFS server (Version 3) Technical aspects Linux kernel module - 3 functions to plug an I/O system Schedulers Pool Sched. Instance Sched. Instance Sched. Instance Request Queue Management Statistics Control I/O Controller aioli System I/O Controller I/O Controller aioli Client 1 (Virtual File System)... I/O Systems aioli Client n (Network File System) Block Device Block Device Network 15/23

43 aioli - Evaluations Platform Grid5000 : Sophia-Antipolis cluster (AMD 64, IDE HDD, Giga Ethernet) 1 to 96 nodes 1 dedicated NFS server IOR benchmark (LLNL) Experiments 1 parallel application : Valid parallel I/O detection and transparent optimisations 2 applications : Analyse mutual impact and interest of a global strategy 10 applications : Evaluate a real case 16/23

44 Evaluations : One Parallel Application 4 GB File decomposition including 32 MPI instances kernel , sophia cluster, NFS version 3, mpich 1.2.5, ROMIO POSIX 1 POSIX 32 Observations Time! Completion time (sec) File access granularity (KBytes) 32 processes deployed on 32 nodes Mono-application OK! next step: global coordination 17/23

45 Evaluations : One Parallel Application 4 GB File decomposition including 32 MPI instances kernel , sophia cluster, NFS version 3, mpich 1.2.5, ROMIO POSIX 1 POSIX 32 MPI IO 32 Observations Time! Completion time (sec) File access granularity (KBytes) 32 processes deployed on 32 nodes Mono-application OK! next step: global coordination 17/23

46 Evaluations : One Parallel Application 4 GB File decomposition including 32 MPI instances kernel , sophia cluster, NFS version 3, mpich 1.2.5, ROMIO Completion time (sec) POSIX 1 POSIX 32 MPI IO 32 POSIX + aioli 32 Observations Time! aioli provides significant improvements 11 < T P osix T aioli < < T MP IIO T aioli < File access granularity (KBytes) 32 processes deployed on 32 nodes Mono-application OK! next step: global coordination 17/23

47 Evaluations : One Parallel Application 4 GB File decomposition including 32 MPI instances kernel , sophia cluster, NFS version 3, mpich 1.2.5, ROMIO POSIX + aioli 32 POSIX 1 POSIX 32 MPI IO 32 Observations Time! Bandwidth (MB/s) aioli provides significant improvements 11 < T P osix T aioli < < T MP IIO T aioli < File access granularity (KBytes) 32 processes deployed on 32 nodes Mono-application OK! next step: global coordination 17/23

48 Bandwidth (MB/s) Evaluations : One Parallel Application 4 GB File decomposition including 32 MPI instances kernel , sophia cluster, NFS version 3, mpich 1.2.5, ROMIO POSIX 1 POSIX + aioli File access granularity (KBytes) 1 process ( prefetch ) Mono-application OK! next step: global coordination Observations Time! aioli provides significant improvements 11 < T P osix T aioli < 50 T MP IIO 3.5 < < 6.5 T aioli Synchronous behaviours benefit from aioli 17/23

49 Evaluations : One Parallel Application 4 GB File decomposition including 32 MPI instances kernel , sophia cluster, NFS version 3, mpich 1.2.5, ROMIO Bandwidth (MB/s) POSIX + aioli 1 POSIX + aioli 32 POSIX 1 POSIX 32 MPI IO File access granularity (KBytes) Observations Time! aioli provides significant improvements 11 < T P osix T aioli < 50 T MP IIO 3.5 < < 6.5 T aioli Synchronous behaviours benefit from aioli Mono-application OK! next step: global coordination 17/23

50 Evaluations : Multi-application Mode - Case 1 Impact of a 4 GB decomposition (32 processes - 32 nodes) over a cat of 16 MB kernel , sophia cluster, NFS version 3, mpich 1.2.5, ROMIO IOR NFS cat NFS IOR NFS cat NFS Completion time (sec) IOR file access granularity (KBytes) cat granularity 32KBytes IOR file access granularity (KBytes) "cat" granularity 32KBytes read - POSIX read - MPI I/O Observations Impact : 16MB in 100 sec (Y axis : log scale), The use of MPI I/O reduces the load on the server /23

51 Evaluations : Multi-application Mode - Case 1 Impact of a 4 GB decomposition (32 processes - 32 nodes) over a cat of 16 MB kernel , sophia cluster, NFS version 3, mpich 1.2.5, ROMIO IOR NFS cat NFS IOR NFS cat NFS Completion time (sec) IOR aioli cat aioli IOR file access granularity (KBytes) cat granularity 32KBytes IOR file access granularity (KBytes) "cat" granularity 32KBytes read - POSIX read - MPI I/O Observations Impact : 16MB in 100 sec (Y axis : log scale), The use of MPI I/O reduces the load on the server aioli improves the performances for both applications IOR aioli cat aioli /23

52 Evaluations : Multi-application Mode - Case 2 10 concurrent applications, 96 nodes, 6 GB kernel , sophia cluster, NFS version 3, mpich 1.2.5, ROMIO Completion time Application details POSIX MPI IO 4 decompos. 6 sequential. 6 GB Time are given in seconds. 19/23

53 Evaluations : Multi-application Mode - Case 2 10 concurrent applications, 96 nodes, 6 GB kernel , sophia cluster, NFS version 3, mpich 1.2.5, ROMIO Application details Completion time NFS POSIX MPI IO read decompos. - 2GB (32 nodes, granularity=128kb) write decompos. - 2GB (32 nodes, granularity=128kb) read decompos MB (16 nodes, granularity=8kb) write decompos MB (8 nodes, granularity=64kb) read sequential. - 1GB (1 node, granularity=2mb) write sequential MB (1 node, granularity=2mb) read sequential. - 32MB (1 node, granularity=4kb) write sequential. - 32MB (1 node, granularity=4kb) read sequential. - 4MB (1 node, granularity=32kb) write sequential. - 4MB (1 node, granularity=32kb) 39 2 Time are given in seconds. 19/23

54 Evaluations : Multi-application Mode - Case 2 10 concurrent applications, 96 nodes, 6 GB kernel , sophia cluster, NFS version 3, mpich 1.2.5, ROMIO Application details Completion time NFS POSIX MPI IO read decompos. - 2GB (32 nodes, granularity=128kb) write decompos. - 2GB (32 nodes, granularity=128kb) read decompos MB (16 nodes, granularity=8kb) write decompos MB (8 nodes, granularity=64kb) read sequential. - 1GB (1 node, granularity=2mb) write sequential MB (1 node, granularity=2mb) read sequential. - 32MB (1 node, granularity=4kb) write sequential. - 32MB (1 node, granularity=4kb) read sequential. - 4MB (1 node, granularity=32kb) write sequential. - 4MB (1 node, granularity=32kb) 39 2 Time are given in seconds. 19/23

55 Evaluations : Multi-application Mode - Case 2 10 concurrent applications, 96 nodes, 6 GB kernel , sophia cluster, NFS version 3, mpich 1.2.5, ROMIO Completion time Application details NFS NFS+aIOLi POSIX MPI IO POSIX MPI IO 4 decompos. 6 sequential. 6 GB Time are given in seconds. 19/23

56 Evaluations : Multi-application Mode - Case 2 10 concurrent applications, 96 nodes, 6 GB kernel , sophia cluster, NFS version 3, mpich 1.2.5, ROMIO Completion time Applications NFS NFS+aIOLi POSIX MPI IO POSIX MPI IO read decompos. - 2GB (32 nodes, granularity=128kb) write decompos. - 2GB (32 nodes, granularity=128kb) read decompos MB (16 nodes, granularity=8kb) write decompos MB (8 nodes, granularity=64kb) read sequential - 1GB (1 node, granularity=2mb) write sequential MB (1 node, granularity=2mb) read sequential. - 32MB (1 node, granularity=4kb) write sequential. - 32MB (1 node, granularity=4kb) read sequential. - 4MB (1 node, granularity=32kb) write sequential. - 4MB (1 node, granularity=32kb) Time are given in seconds. 19/23

57 Evaluations : Multi-application Mode - Case 2 10 concurrent applications, 96 nodes, 6 GB kernel , sophia cluster, NFS version 3, mpich 1.2.5, ROMIO Completion time Applications NFS NFS+aIOLi POSIX MPI IO POSIX MPI IO read decompos. - 2GB (32 nodes, granularity=128kb) write decompos. - 2GB (32 nodes, granularity=128kb) read decompos MB (16 nodes, granularity=8kb) write decompos MB (8 nodes, granularity=64kb) read sequential - 1GB (1 node, granularity=2mb) write sequential MB (1 node, granularity=2mb) read sequential. - 32MB (1 node, granularity=4kb) write sequential. - 32MB (1 node, granularity=4kb) read sequential. - 4MB (1 node, granularity=32kb) write sequential. - 4MB (1 node, granularity=32kb) Time are given in seconds. 19/23

58 Plan Part 1 - Parallel Input/Output and Clusters Part 2 - Controlling and Scheduling Multi-application I/O Part 3 - aioli, an Input/Output Scheduler for HPC Part 4 - Conclusion 8 Conclusion 9 Current and Future Works 20/23

59 Conclusion Performances and I/O in multi-application environment Control and schedule I/O requests in a global way : multi-criteria problem : efficiency and fairness MLF variant proposal aioli, an I/O scheduler for HPC Generic framework to evaluate new strategies for I/O scheduling Implementation in kernel space : intrusive from system point of view but efficient Code available under GPL Joint project since the end of 2005 with ICIS University (Poland) 21/23

60 Current and Future Works aioli, works in progress Future Take into account data stripping (on RAID devices and Parallel File Systems) Collaboration around a parallel version of NFS to evaluate the interest of an higher I/ scheduler (Brasil/France/Poland) Interconnexion with Lustre File system Control I/O requests at different points : multi-level scheduler on client side (compute node) / on server and hard drive side cascade scheduling Exploiting the meta-node concept used by modern FS to provide consistency as a central point to plug aioli 22/23

61 Questions? LIPS Project BULL - INRIA - ID-IMAG Laboratory - ICIS Institute Thanks 23/23

Overlapping Data Transfer With Application Execution on Clusters

Overlapping Data Transfer With Application Execution on Clusters Overlapping Data Transfer With Application Execution on Clusters Karen L. Reid and Michael Stumm reid@cs.toronto.edu stumm@eecg.toronto.edu Department of Computer Science Department of Electrical and Computer

More information

Operating System for the K computer

Operating System for the K computer Operating System for the K computer Jun Moroo Masahiko Yamada Takeharu Kato For the K computer to achieve the world s highest performance, Fujitsu has worked on the following three performance improvements

More information

An Alternative Storage Solution for MapReduce. Eric Lomascolo Director, Solutions Marketing

An Alternative Storage Solution for MapReduce. Eric Lomascolo Director, Solutions Marketing An Alternative Storage Solution for MapReduce Eric Lomascolo Director, Solutions Marketing MapReduce Breaks the Problem Down Data Analysis Distributes processing work (Map) across compute nodes and accumulates

More information

BlobSeer: Towards efficient data storage management on large-scale, distributed systems

BlobSeer: Towards efficient data storage management on large-scale, distributed systems : Towards efficient data storage management on large-scale, distributed systems Bogdan Nicolae University of Rennes 1, France KerData Team, INRIA Rennes Bretagne-Atlantique PhD Advisors: Gabriel Antoniu

More information

Sun Storage Perspective & Lustre Architecture. Dr. Peter Braam VP Sun Microsystems

Sun Storage Perspective & Lustre Architecture. Dr. Peter Braam VP Sun Microsystems Sun Storage Perspective & Lustre Architecture Dr. Peter Braam VP Sun Microsystems Agenda Future of Storage Sun s vision Lustre - vendor neutral architecture roadmap Sun s view on storage introduction The

More information

Write a technical report Present your results Write a workshop/conference paper (optional) Could be a real system, simulation and/or theoretical

Write a technical report Present your results Write a workshop/conference paper (optional) Could be a real system, simulation and/or theoretical Identify a problem Review approaches to the problem Propose a novel approach to the problem Define, design, prototype an implementation to evaluate your approach Could be a real system, simulation and/or

More information

MOSIX: High performance Linux farm

MOSIX: High performance Linux farm MOSIX: High performance Linux farm Paolo Mastroserio [mastroserio@na.infn.it] Francesco Maria Taurino [taurino@na.infn.it] Gennaro Tortone [tortone@na.infn.it] Napoli Index overview on Linux farm farm

More information

Petascale Software Challenges. Piyush Chaudhary piyushc@us.ibm.com High Performance Computing

Petascale Software Challenges. Piyush Chaudhary piyushc@us.ibm.com High Performance Computing Petascale Software Challenges Piyush Chaudhary piyushc@us.ibm.com High Performance Computing Fundamental Observations Applications are struggling to realize growth in sustained performance at scale Reasons

More information

THE EXPAND PARALLEL FILE SYSTEM A FILE SYSTEM FOR CLUSTER AND GRID COMPUTING. José Daniel García Sánchez ARCOS Group University Carlos III of Madrid

THE EXPAND PARALLEL FILE SYSTEM A FILE SYSTEM FOR CLUSTER AND GRID COMPUTING. José Daniel García Sánchez ARCOS Group University Carlos III of Madrid THE EXPAND PARALLEL FILE SYSTEM A FILE SYSTEM FOR CLUSTER AND GRID COMPUTING José Daniel García Sánchez ARCOS Group University Carlos III of Madrid Contents 2 The ARCOS Group. Expand motivation. Expand

More information

Using Linux Clusters as VoD Servers

Using Linux Clusters as VoD Servers HAC LUCE Using Linux Clusters as VoD Servers Víctor M. Guĺıas Fernández gulias@lfcia.org Computer Science Department University of A Corunha funded by: Outline Background: The Borg Cluster Video on Demand.

More information

Globus Striped GridFTP Framework and Server. Raj Kettimuthu, ANL and U. Chicago

Globus Striped GridFTP Framework and Server. Raj Kettimuthu, ANL and U. Chicago Globus Striped GridFTP Framework and Server Raj Kettimuthu, ANL and U. Chicago Outline Introduction Features Motivation Architecture Globus XIO Experimental Results 3 August 2005 The Ohio State University

More information

- An Essential Building Block for Stable and Reliable Compute Clusters

- An Essential Building Block for Stable and Reliable Compute Clusters Ferdinand Geier ParTec Cluster Competence Center GmbH, V. 1.4, March 2005 Cluster Middleware - An Essential Building Block for Stable and Reliable Compute Clusters Contents: Compute Clusters a Real Alternative

More information

DSS. Diskpool and cloud storage benchmarks used in IT-DSS. Data & Storage Services. Geoffray ADDE

DSS. Diskpool and cloud storage benchmarks used in IT-DSS. Data & Storage Services. Geoffray ADDE DSS Data & Diskpool and cloud storage benchmarks used in IT-DSS CERN IT Department CH-1211 Geneva 23 Switzerland www.cern.ch/it Geoffray ADDE DSS Outline I- A rational approach to storage systems evaluation

More information

Distributed RAID Architectures for Cluster I/O Computing. Kai Hwang

Distributed RAID Architectures for Cluster I/O Computing. Kai Hwang Distributed RAID Architectures for Cluster I/O Computing Kai Hwang Internet and Cluster Computing Lab. University of Southern California 1 Presentation Outline : Scalable Cluster I/O The RAID-x Architecture

More information

COSC 6374 Parallel Computation. Parallel I/O (I) I/O basics. Concept of a clusters

COSC 6374 Parallel Computation. Parallel I/O (I) I/O basics. Concept of a clusters COSC 6374 Parallel I/O (I) I/O basics Fall 2012 Concept of a clusters Processor 1 local disks Compute node message passing network administrative network Memory Processor 2 Network card 1 Network card

More information

Cray DVS: Data Virtualization Service

Cray DVS: Data Virtualization Service Cray : Data Virtualization Service Stephen Sugiyama and David Wallace, Cray Inc. ABSTRACT: Cray, the Cray Data Virtualization Service, is a new capability being added to the XT software environment with

More information

COSC 6374 Parallel Computation. Parallel I/O (I) I/O basics. Concept of a clusters

COSC 6374 Parallel Computation. Parallel I/O (I) I/O basics. Concept of a clusters COSC 6374 Parallel Computation Parallel I/O (I) I/O basics Spring 2008 Concept of a clusters Processor 1 local disks Compute node message passing network administrative network Memory Processor 2 Network

More information

CSAR: Cluster Storage with Adaptive Redundancy

CSAR: Cluster Storage with Adaptive Redundancy CSAR: Cluster Storage with Adaptive Redundancy Manoj Pillai, Mario Lauria Department of Computer and Information Science The Ohio State University Columbus, OH, 4321 Email: pillai,lauria@cis.ohio-state.edu

More information

POWER ALL GLOBAL FILE SYSTEM (PGFS)

POWER ALL GLOBAL FILE SYSTEM (PGFS) POWER ALL GLOBAL FILE SYSTEM (PGFS) Defining next generation of global storage grid Power All Networks Ltd. Technical Whitepaper April 2008, version 1.01 Table of Content 1. Introduction.. 3 2. Paradigm

More information

Running a Workflow on a PowerCenter Grid

Running a Workflow on a PowerCenter Grid Running a Workflow on a PowerCenter Grid 2010-2014 Informatica Corporation. No part of this document may be reproduced or transmitted in any form, by any means (electronic, photocopying, recording or otherwise)

More information

New Storage System Solutions

New Storage System Solutions New Storage System Solutions Craig Prescott Research Computing May 2, 2013 Outline } Existing storage systems } Requirements and Solutions } Lustre } /scratch/lfs } Questions? Existing Storage Systems

More information

Can High-Performance Interconnects Benefit Memcached and Hadoop?

Can High-Performance Interconnects Benefit Memcached and Hadoop? Can High-Performance Interconnects Benefit Memcached and Hadoop? D. K. Panda and Sayantan Sur Network-Based Computing Laboratory Department of Computer Science and Engineering The Ohio State University,

More information

Bottleneck Detection in Parallel File Systems with Trace-Based Performance Monitoring

Bottleneck Detection in Parallel File Systems with Trace-Based Performance Monitoring Julian M. Kunkel - Euro-Par 2008 1/33 Bottleneck Detection in Parallel File Systems with Trace-Based Performance Monitoring Julian M. Kunkel Thomas Ludwig Institute for Computer Science Parallel and Distributed

More information

CERN Cloud Storage Evaluation Geoffray Adde, Dirk Duellmann, Maitane Zotes CERN IT

CERN Cloud Storage Evaluation Geoffray Adde, Dirk Duellmann, Maitane Zotes CERN IT SS Data & Storage CERN Cloud Storage Evaluation Geoffray Adde, Dirk Duellmann, Maitane Zotes CERN IT HEPiX Fall 2012 Workshop October 15-19, 2012 Institute of High Energy Physics, Beijing, China SS Outline

More information

Why Compromise? A discussion on RDMA versus Send/Receive and the difference between interconnect and application semantics

Why Compromise? A discussion on RDMA versus Send/Receive and the difference between interconnect and application semantics Why Compromise? A discussion on RDMA versus Send/Receive and the difference between interconnect and application semantics Mellanox Technologies Inc. 2900 Stender Way, Santa Clara, CA 95054 Tel: 408-970-3400

More information

Oracle Database Scalability in VMware ESX VMware ESX 3.5

Oracle Database Scalability in VMware ESX VMware ESX 3.5 Performance Study Oracle Database Scalability in VMware ESX VMware ESX 3.5 Database applications running on individual physical servers represent a large consolidation opportunity. However enterprises

More information

Performance Monitoring of Parallel Scientific Applications

Performance Monitoring of Parallel Scientific Applications Performance Monitoring of Parallel Scientific Applications Abstract. David Skinner National Energy Research Scientific Computing Center Lawrence Berkeley National Laboratory This paper introduces an infrastructure

More information

Violin: A Framework for Extensible Block-level Storage

Violin: A Framework for Extensible Block-level Storage Violin: A Framework for Extensible Block-level Storage Michail Flouris Dept. of Computer Science, University of Toronto, Canada flouris@cs.toronto.edu Angelos Bilas ICS-FORTH & University of Crete, Greece

More information

Advanced Techniques with Newton. Gerald Ragghianti Advanced Newton workshop Sept. 22, 2011

Advanced Techniques with Newton. Gerald Ragghianti Advanced Newton workshop Sept. 22, 2011 Advanced Techniques with Newton Gerald Ragghianti Advanced Newton workshop Sept. 22, 2011 Workshop Goals Gain independence Executing your work Finding Information Fixing Problems Optimizing Effectiveness

More information

E) Modeling Insights: Patterns and Anti-patterns

E) Modeling Insights: Patterns and Anti-patterns Murray Woodside, July 2002 Techniques for Deriving Performance Models from Software Designs Murray Woodside Second Part Outline ) Conceptual framework and scenarios ) Layered systems and models C) uilding

More information

Avid ISIS 7000. www.avid.com

Avid ISIS 7000. www.avid.com Avid ISIS 7000 www.avid.com Table of Contents Overview... 3 Avid ISIS Technology Overview... 6 ISIS Storage Blade... 6 ISIS Switch Blade... 7 ISIS System Director... 7 ISIS Client Software... 8 ISIS Redundant

More information

Cloud Computing through Virtualization and HPC technologies

Cloud Computing through Virtualization and HPC technologies Cloud Computing through Virtualization and HPC technologies William Lu, Ph.D. 1 Agenda Cloud Computing & HPC A Case of HPC Implementation Application Performance in VM Summary 2 Cloud Computing & HPC HPC

More information

Distributed File Systems

Distributed File Systems Distributed File Systems Paul Krzyzanowski Rutgers University October 28, 2012 1 Introduction The classic network file systems we examined, NFS, CIFS, AFS, Coda, were designed as client-server applications.

More information

Parallel Programming Survey

Parallel Programming Survey Christian Terboven 02.09.2014 / Aachen, Germany Stand: 26.08.2014 Version 2.3 IT Center der RWTH Aachen University Agenda Overview: Processor Microarchitecture Shared-Memory

More information

David Rioja Redondo Telecommunication Engineer Englobe Technologies and Systems

David Rioja Redondo Telecommunication Engineer Englobe Technologies and Systems David Rioja Redondo Telecommunication Engineer Englobe Technologies and Systems About me David Rioja Redondo Telecommunication Engineer - Universidad de Alcalá >2 years building and managing clusters UPM

More information

System Software for High Performance Computing. Joe Izraelevitz

System Software for High Performance Computing. Joe Izraelevitz System Software for High Performance Computing Joe Izraelevitz Agenda Overview of Supercomputers Blue Gene/Q System LoadLeveler Job Scheduler General Parallel File System HPC at UR What is a Supercomputer?

More information

Vers des mécanismes génériques de communication et une meilleure maîtrise des affinités dans les grappes de calculateurs hiérarchiques.

Vers des mécanismes génériques de communication et une meilleure maîtrise des affinités dans les grappes de calculateurs hiérarchiques. Vers des mécanismes génériques de communication et une meilleure maîtrise des affinités dans les grappes de calculateurs hiérarchiques Brice Goglin 15 avril 2014 Towards generic Communication Mechanisms

More information

Agenda. HPC Software Stack. HPC Post-Processing Visualization. Case Study National Scientific Center. European HPC Benchmark Center Montpellier PSSC

Agenda. HPC Software Stack. HPC Post-Processing Visualization. Case Study National Scientific Center. European HPC Benchmark Center Montpellier PSSC HPC Architecture End to End Alexandre Chauvin Agenda HPC Software Stack Visualization National Scientific Center 2 Agenda HPC Software Stack Alexandre Chauvin Typical HPC Software Stack Externes LAN Typical

More information

The High Performance Internet of Things: using GVirtuS for gluing cloud computing and ubiquitous connected devices

The High Performance Internet of Things: using GVirtuS for gluing cloud computing and ubiquitous connected devices WS on Models, Algorithms and Methodologies for Hierarchical Parallelism in new HPC Systems The High Performance Internet of Things: using GVirtuS for gluing cloud computing and ubiquitous connected devices

More information

Commoditisation of the High-End Research Storage Market with the Dell MD3460 & Intel Enterprise Edition Lustre

Commoditisation of the High-End Research Storage Market with the Dell MD3460 & Intel Enterprise Edition Lustre Commoditisation of the High-End Research Storage Market with the Dell MD3460 & Intel Enterprise Edition Lustre University of Cambridge, UIS, HPC Service Authors: Wojciech Turek, Paul Calleja, John Taylor

More information

Using Linux Clusters as VoD Servers

Using Linux Clusters as VoD Servers HAC LUCE Using Linux Clusters as VoD Servers Víctor M. Guĺıas Fernández gulias@lfcia.org Computer Science Department University of A Corunha funded by: Outline Background: The Borg Cluster Video on Demand.

More information

Beyond Embarrassingly Parallel Big Data. William Gropp www.cs.illinois.edu/~wgropp

Beyond Embarrassingly Parallel Big Data. William Gropp www.cs.illinois.edu/~wgropp Beyond Embarrassingly Parallel Big Data William Gropp www.cs.illinois.edu/~wgropp Messages Big is big Data driven is an important area, but not all data driven problems are big data (despite current hype).

More information

A Comparison on Current Distributed File Systems for Beowulf Clusters

A Comparison on Current Distributed File Systems for Beowulf Clusters A Comparison on Current Distributed File Systems for Beowulf Clusters Rafael Bohrer Ávila 1 Philippe Olivier Alexandre Navaux 2 Yves Denneulin 3 Abstract This paper presents a comparison on current file

More information

How To Install Linux Titan

How To Install Linux Titan Linux Titan Distribution Presented By: Adham Helal Amgad Madkour Ayman El Sayed Emad Zakaria What Is a Linux Distribution? What is a Linux Distribution? The distribution contains groups of packages and

More information

Technological Overview of High-Performance Computing. Gerolf Ziegenhain - TU Kaiserslautern, Germany

Technological Overview of High-Performance Computing. Gerolf Ziegenhain - TU Kaiserslautern, Germany Technological Overview of High-Performance Computing Gerolf Ziegenhain - TU Kaiserslautern, Germany Outline of This Talk Give a glance at the important technologies The most important stuff is mentioned

More information

Network Attached Storage. Jinfeng Yang Oct/19/2015

Network Attached Storage. Jinfeng Yang Oct/19/2015 Network Attached Storage Jinfeng Yang Oct/19/2015 Outline Part A 1. What is the Network Attached Storage (NAS)? 2. What are the applications of NAS? 3. The benefits of NAS. 4. NAS s performance (Reliability

More information

Lustre * Filesystem for Cloud and Hadoop *

Lustre * Filesystem for Cloud and Hadoop * OpenFabrics Software User Group Workshop Lustre * Filesystem for Cloud and Hadoop * Robert Read, Intel Lustre * for Cloud and Hadoop * Brief Lustre History and Overview Using Lustre with Hadoop Intel Cloud

More information

Portable, Scalable, and High-Performance I/O Forwarding on Massively Parallel Systems. Jason Cope copej@mcs.anl.gov

Portable, Scalable, and High-Performance I/O Forwarding on Massively Parallel Systems. Jason Cope copej@mcs.anl.gov Portable, Scalable, and High-Performance I/O Forwarding on Massively Parallel Systems Jason Cope copej@mcs.anl.gov Computation and I/O Performance Imbalance Leadership class computa:onal scale: >100,000

More information

How To Monitor Infiniband Network Data From A Network On A Leaf Switch (Wired) On A Microsoft Powerbook (Wired Or Microsoft) On An Ipa (Wired/Wired) Or Ipa V2 (Wired V2)

How To Monitor Infiniband Network Data From A Network On A Leaf Switch (Wired) On A Microsoft Powerbook (Wired Or Microsoft) On An Ipa (Wired/Wired) Or Ipa V2 (Wired V2) INFINIBAND NETWORK ANALYSIS AND MONITORING USING OPENSM N. Dandapanthula 1, H. Subramoni 1, J. Vienne 1, K. Kandalla 1, S. Sur 1, D. K. Panda 1, and R. Brightwell 2 Presented By Xavier Besseron 1 Date:

More information

Enabling High performance Big Data platform with RDMA

Enabling High performance Big Data platform with RDMA Enabling High performance Big Data platform with RDMA Tong Liu HPC Advisory Council Oct 7 th, 2014 Shortcomings of Hadoop Administration tooling Performance Reliability SQL support Backup and recovery

More information

LS-DYNA Best-Practices: Networking, MPI and Parallel File System Effect on LS-DYNA Performance

LS-DYNA Best-Practices: Networking, MPI and Parallel File System Effect on LS-DYNA Performance 11 th International LS-DYNA Users Conference Session # LS-DYNA Best-Practices: Networking, MPI and Parallel File System Effect on LS-DYNA Performance Gilad Shainer 1, Tong Liu 2, Jeff Layton 3, Onur Celebioglu

More information

Cluster Implementation and Management; Scheduling

Cluster Implementation and Management; Scheduling Cluster Implementation and Management; Scheduling CPS343 Parallel and High Performance Computing Spring 2013 CPS343 (Parallel and HPC) Cluster Implementation and Management; Scheduling Spring 2013 1 /

More information

Implementing MPI-IO Shared File Pointers without File System Support

Implementing MPI-IO Shared File Pointers without File System Support Implementing MPI-IO Shared File Pointers without File System Support Robert Latham, Robert Ross, Rajeev Thakur, Brian Toonen Mathematics and Computer Science Division Argonne National Laboratory Argonne,

More information

How To Visualize Performance Data In A Computer Program

How To Visualize Performance Data In A Computer Program Performance Visualization Tools 1 Performance Visualization Tools Lecture Outline : Following Topics will be discussed Characteristics of Performance Visualization technique Commercial and Public Domain

More information

Rodrigo Fernandes de Mello, Evgueni Dodonov, José Augusto Andrade Filho

Rodrigo Fernandes de Mello, Evgueni Dodonov, José Augusto Andrade Filho Middleware for High Performance Computing Rodrigo Fernandes de Mello, Evgueni Dodonov, José Augusto Andrade Filho University of São Paulo São Carlos, Brazil {mello, eugeni, augustoa}@icmc.usp.br Outline

More information

Redpaper. Performance Metrics in TotalStorage Productivity Center Performance Reports. Introduction. Mary Lovelace

Redpaper. Performance Metrics in TotalStorage Productivity Center Performance Reports. Introduction. Mary Lovelace Redpaper Mary Lovelace Performance Metrics in TotalStorage Productivity Center Performance Reports Introduction This Redpaper contains the TotalStorage Productivity Center performance metrics that are

More information

Intel DPDK Boosts Server Appliance Performance White Paper

Intel DPDK Boosts Server Appliance Performance White Paper Intel DPDK Boosts Server Appliance Performance Intel DPDK Boosts Server Appliance Performance Introduction As network speeds increase to 40G and above, both in the enterprise and data center, the bottlenecks

More information

Chronicle: Capture and Analysis of NFS Workloads at Line Rate

Chronicle: Capture and Analysis of NFS Workloads at Line Rate Chronicle: Capture and Analysis of NFS Workloads at Line Rate Ardalan Kangarlou, Sandip Shete, and John Strunk Advanced Technology Group 1 Motivation Goal: To gather insights from customer workloads via

More information

Sockets vs. RDMA Interface over 10-Gigabit Networks: An In-depth Analysis of the Memory Traffic Bottleneck

Sockets vs. RDMA Interface over 10-Gigabit Networks: An In-depth Analysis of the Memory Traffic Bottleneck Sockets vs. RDMA Interface over 1-Gigabit Networks: An In-depth Analysis of the Memory Traffic Bottleneck Pavan Balaji Hemal V. Shah D. K. Panda Network Based Computing Lab Computer Science and Engineering

More information

Use of Hadoop File System for Nuclear Physics Analyses in STAR

Use of Hadoop File System for Nuclear Physics Analyses in STAR 1 Use of Hadoop File System for Nuclear Physics Analyses in STAR EVAN SANGALINE UC DAVIS Motivations 2 Data storage a key component of analysis requirements Transmission and storage across diverse resources

More information

GPFS Storage Server. Concepts and Setup in Lemanicus BG/Q system" Christian Clémençon (EPFL-DIT)" " 4 April 2013"

GPFS Storage Server. Concepts and Setup in Lemanicus BG/Q system Christian Clémençon (EPFL-DIT)  4 April 2013 GPFS Storage Server Concepts and Setup in Lemanicus BG/Q system" Christian Clémençon (EPFL-DIT)" " Agenda" GPFS Overview" Classical versus GSS I/O Solution" GPFS Storage Server (GSS)" GPFS Native RAID

More information

GraySort on Apache Spark by Databricks

GraySort on Apache Spark by Databricks GraySort on Apache Spark by Databricks Reynold Xin, Parviz Deyhim, Ali Ghodsi, Xiangrui Meng, Matei Zaharia Databricks Inc. Apache Spark Sorting in Spark Overview Sorting Within a Partition Range Partitioner

More information

Ceph. A file system a little bit different. Udo Seidel

Ceph. A file system a little bit different. Udo Seidel Ceph A file system a little bit different Udo Seidel Ceph what? So-called parallel distributed cluster file system Started as part of PhD studies at UCSC Public announcement in 2006 at 7 th OSDI File system

More information

Infrastructure for Load Balancing on Mosix Cluster

Infrastructure for Load Balancing on Mosix Cluster Infrastructure for Load Balancing on Mosix Cluster MadhuSudhan Reddy Tera and Sadanand Kota Computing and Information Science, Kansas State University Under the Guidance of Dr. Daniel Andresen. Abstract

More information

Distributed Data Management

Distributed Data Management Introduction Distributed Data Management Involves the distribution of data and work among more than one machine in the network. Distributed computing is more broad than canonical client/server, in that

More information

Last Class: OS and Computer Architecture. Last Class: OS and Computer Architecture

Last Class: OS and Computer Architecture. Last Class: OS and Computer Architecture Last Class: OS and Computer Architecture System bus Network card CPU, memory, I/O devices, network card, system bus Lecture 3, page 1 Last Class: OS and Computer Architecture OS Service Protection Interrupts

More information

Configuration Maximums VMware Infrastructure 3

Configuration Maximums VMware Infrastructure 3 Technical Note Configuration s VMware Infrastructure 3 When you are selecting and configuring your virtual and physical equipment, you must stay at or below the maximums supported by VMware Infrastructure

More information

PCI Express High Speed Networks. Complete Solution for High Speed Networking

PCI Express High Speed Networks. Complete Solution for High Speed Networking PCI Express High Speed Networks Complete Solution for High Speed Networking Ultra Low Latency Ultra High Throughput Maximizing application performance is a combination of processing, communication, and

More information

Unstructured Data Accelerator (UDA) Author: Motti Beck, Mellanox Technologies Date: March 27, 2012

Unstructured Data Accelerator (UDA) Author: Motti Beck, Mellanox Technologies Date: March 27, 2012 Unstructured Data Accelerator (UDA) Author: Motti Beck, Mellanox Technologies Date: March 27, 2012 1 Market Trends Big Data Growing technology deployments are creating an exponential increase in the volume

More information

High Performance Computing. Course Notes 2007-2008. HPC Fundamentals

High Performance Computing. Course Notes 2007-2008. HPC Fundamentals High Performance Computing Course Notes 2007-2008 2008 HPC Fundamentals Introduction What is High Performance Computing (HPC)? Difficult to define - it s a moving target. Later 1980s, a supercomputer performs

More information

System Models for Distributed and Cloud Computing

System Models for Distributed and Cloud Computing System Models for Distributed and Cloud Computing Dr. Sanjay P. Ahuja, Ph.D. 2010-14 FIS Distinguished Professor of Computer Science School of Computing, UNF Classification of Distributed Computing Systems

More information

Lustre Networking BY PETER J. BRAAM

Lustre Networking BY PETER J. BRAAM Lustre Networking BY PETER J. BRAAM A WHITE PAPER FROM CLUSTER FILE SYSTEMS, INC. APRIL 2007 Audience Architects of HPC clusters Abstract This paper provides architects of HPC clusters with information

More information

Network Infrastructure Services CS848 Project

Network Infrastructure Services CS848 Project Quality of Service Guarantees for Cloud Services CS848 Project presentation by Alexey Karyakin David R. Cheriton School of Computer Science University of Waterloo March 2010 Outline 1. Performance of cloud

More information

Quantum StorNext. Product Brief: Distributed LAN Client

Quantum StorNext. Product Brief: Distributed LAN Client Quantum StorNext Product Brief: Distributed LAN Client NOTICE This product brief may contain proprietary information protected by copyright. Information in this product brief is subject to change without

More information

The Assessment of Benchmarks Executed on Bare-Metal and Using Para-Virtualisation

The Assessment of Benchmarks Executed on Bare-Metal and Using Para-Virtualisation The Assessment of Benchmarks Executed on Bare-Metal and Using Para-Virtualisation Mark Baker, Garry Smith and Ahmad Hasaan SSE, University of Reading Paravirtualization A full assessment of paravirtualization

More information

Making Multicore Work and Measuring its Benefits. Markus Levy, president EEMBC and Multicore Association

Making Multicore Work and Measuring its Benefits. Markus Levy, president EEMBC and Multicore Association Making Multicore Work and Measuring its Benefits Markus Levy, president EEMBC and Multicore Association Agenda Why Multicore? Standards and issues in the multicore community What is Multicore Association?

More information

InfoScale Storage & Media Server Workloads

InfoScale Storage & Media Server Workloads InfoScale Storage & Media Server Workloads Maximise Performance when Storing and Retrieving Large Amounts of Unstructured Data Carlos Carrero Colin Eldridge Shrinivas Chandukar 1 Table of Contents 01 Introduction

More information

CHAPTER 2 MODELLING FOR DISTRIBUTED NETWORK SYSTEMS: THE CLIENT- SERVER MODEL

CHAPTER 2 MODELLING FOR DISTRIBUTED NETWORK SYSTEMS: THE CLIENT- SERVER MODEL CHAPTER 2 MODELLING FOR DISTRIBUTED NETWORK SYSTEMS: THE CLIENT- SERVER MODEL This chapter is to introduce the client-server model and its role in the development of distributed network systems. The chapter

More information

Windows 8 SMB 2.2 File Sharing Performance

Windows 8 SMB 2.2 File Sharing Performance Windows 8 SMB 2.2 File Sharing Performance Abstract This paper provides a preliminary analysis of the performance capabilities of the Server Message Block (SMB) 2.2 file sharing protocol with 10 gigabit

More information

Department of Computer Sciences University of Salzburg. HPC In The Cloud? Seminar aus Informatik SS 2011/2012. July 16, 2012

Department of Computer Sciences University of Salzburg. HPC In The Cloud? Seminar aus Informatik SS 2011/2012. July 16, 2012 Department of Computer Sciences University of Salzburg HPC In The Cloud? Seminar aus Informatik SS 2011/2012 July 16, 2012 Michael Kleber, mkleber@cosy.sbg.ac.at Contents 1 Introduction...................................

More information

EOFS Workshop Paris Sept, 2011. Lustre at exascale. Eric Barton. CTO Whamcloud, Inc. eeb@whamcloud.com. 2011 Whamcloud, Inc.

EOFS Workshop Paris Sept, 2011. Lustre at exascale. Eric Barton. CTO Whamcloud, Inc. eeb@whamcloud.com. 2011 Whamcloud, Inc. EOFS Workshop Paris Sept, 2011 Lustre at exascale Eric Barton CTO Whamcloud, Inc. eeb@whamcloud.com Agenda Forces at work in exascale I/O Technology drivers I/O requirements Software engineering issues

More information

SRNWP Workshop. HP Solutions and Activities in Climate & Weather Research. Michael Riedmann European Performance Center

SRNWP Workshop. HP Solutions and Activities in Climate & Weather Research. Michael Riedmann European Performance Center SRNWP Workshop HP Solutions and Activities in Climate & Weather Research Michael Riedmann European Performance Center Agenda A bit of marketing: HP Solutions for HPC A few words about recent Met deals

More information

File System & Device Drive. Overview of Mass Storage Structure. Moving head Disk Mechanism. HDD Pictures 11/13/2014. CS341: Operating System

File System & Device Drive. Overview of Mass Storage Structure. Moving head Disk Mechanism. HDD Pictures 11/13/2014. CS341: Operating System CS341: Operating System Lect 36: 1 st Nov 2014 Dr. A. Sahu Dept of Comp. Sc. & Engg. Indian Institute of Technology Guwahati File System & Device Drive Mass Storage Disk Structure Disk Arm Scheduling RAID

More information

A REVIEW PAPER ON THE HADOOP DISTRIBUTED FILE SYSTEM

A REVIEW PAPER ON THE HADOOP DISTRIBUTED FILE SYSTEM A REVIEW PAPER ON THE HADOOP DISTRIBUTED FILE SYSTEM Sneha D.Borkar 1, Prof.Chaitali S.Surtakar 2 Student of B.E., Information Technology, J.D.I.E.T, sborkar95@gmail.com Assistant Professor, Information

More information

Mixing Hadoop and HPC Workloads on Parallel Filesystems

Mixing Hadoop and HPC Workloads on Parallel Filesystems Mixing Hadoop and HPC Workloads on Parallel Filesystems Esteban Molina-Estolano *, Maya Gokhale, Carlos Maltzahn *, John May, John Bent, Scott Brandt * * UC Santa Cruz, ISSDM, PDSI Lawrence Livermore National

More information

Performance Analysis of RAIDs in Storage Area Network

Performance Analysis of RAIDs in Storage Area Network Performance Analysis of RAIDs in Storage Area Network Sneha M. Assistant Professor, Department of Computer Science and Engineering, R V College of Engineering Bengaluru-560059 ABSTRACT Direct Attached

More information

AIX NFS Client Performance Improvements for Databases on NAS

AIX NFS Client Performance Improvements for Databases on NAS AIX NFS Client Performance Improvements for Databases on NAS October 20, 2005 Sanjay Gulabani Sr. Performance Engineer Network Appliance, Inc. gulabani@netapp.com Diane Flemming Advisory Software Engineer

More information

Multicore Parallel Computing with OpenMP

Multicore Parallel Computing with OpenMP Multicore Parallel Computing with OpenMP Tan Chee Chiang (SVU/Academic Computing, Computer Centre) 1. OpenMP Programming The death of OpenMP was anticipated when cluster systems rapidly replaced large

More information

International Journal of Computer & Organization Trends Volume20 Number1 May 2015

International Journal of Computer & Organization Trends Volume20 Number1 May 2015 Performance Analysis of Various Guest Operating Systems on Ubuntu 14.04 Prof. (Dr.) Viabhakar Pathak 1, Pramod Kumar Ram 2 1 Computer Science and Engineering, Arya College of Engineering, Jaipur, India.

More information

PARALLEL & CLUSTER COMPUTING CS 6260 PROFESSOR: ELISE DE DONCKER BY: LINA HUSSEIN

PARALLEL & CLUSTER COMPUTING CS 6260 PROFESSOR: ELISE DE DONCKER BY: LINA HUSSEIN 1 PARALLEL & CLUSTER COMPUTING CS 6260 PROFESSOR: ELISE DE DONCKER BY: LINA HUSSEIN Introduction What is cluster computing? Classification of Cluster Computing Technologies: Beowulf cluster Construction

More information

Maximizing Hadoop Performance and Storage Capacity with AltraHD TM

Maximizing Hadoop Performance and Storage Capacity with AltraHD TM Maximizing Hadoop Performance and Storage Capacity with AltraHD TM Executive Summary The explosion of internet data, driven in large part by the growth of more and more powerful mobile devices, has created

More information

Configuration Maximums VMware vsphere 4.0

Configuration Maximums VMware vsphere 4.0 Topic Configuration s VMware vsphere 4.0 When you select and configure your virtual and physical equipment, you must stay at or below the maximums supported by vsphere 4.0. The limits presented in the

More information

Agenda. Enterprise Application Performance Factors. Current form of Enterprise Applications. Factors to Application Performance.

Agenda. Enterprise Application Performance Factors. Current form of Enterprise Applications. Factors to Application Performance. Agenda Enterprise Performance Factors Overall Enterprise Performance Factors Best Practice for generic Enterprise Best Practice for 3-tiers Enterprise Hardware Load Balancer Basic Unix Tuning Performance

More information

Symmetric Multiprocessing

Symmetric Multiprocessing Multicore Computing A multi-core processor is a processing system composed of two or more independent cores. One can describe it as an integrated circuit to which two or more individual processors (called

More information

The Hadoop Distributed Filesystem: Balancing Portability and Performance

The Hadoop Distributed Filesystem: Balancing Portability and Performance The Hadoop Distributed Filesystem: Balancing Portability and Performance Jeffrey Shafer, Scott Rixner, and Alan L. Cox Rice University Houston, TX Email: {shafer, rixner, alc}@rice.edu Abstract Hadoop

More information

Optimizing Shared Resource Contention in HPC Clusters

Optimizing Shared Resource Contention in HPC Clusters Optimizing Shared Resource Contention in HPC Clusters Sergey Blagodurov Simon Fraser University Alexandra Fedorova Simon Fraser University Abstract Contention for shared resources in HPC clusters occurs

More information

IT of SPIM Data Storage and Compression. EMBO Course - August 27th! Jeff Oegema, Peter Steinbach, Oscar Gonzalez

IT of SPIM Data Storage and Compression. EMBO Course - August 27th! Jeff Oegema, Peter Steinbach, Oscar Gonzalez IT of SPIM Data Storage and Compression EMBO Course - August 27th Jeff Oegema, Peter Steinbach, Oscar Gonzalez 1 Talk Outline Introduction and the IT Team SPIM Data Flow Capture, Compression, and the Data

More information

Cluster Grid Interconects. Tony Kay Chief Architect Enterprise Grid and Networking

Cluster Grid Interconects. Tony Kay Chief Architect Enterprise Grid and Networking Cluster Grid Interconects Tony Kay Chief Architect Enterprise Grid and Networking Agenda Cluster Grid Interconnects The Upstart - Infiniband The Empire Strikes Back - Myricom Return of the King 10G Gigabit

More information

HPC and Big Data. EPCC The University of Edinburgh. Adrian Jackson Technical Architect a.jackson@epcc.ed.ac.uk

HPC and Big Data. EPCC The University of Edinburgh. Adrian Jackson Technical Architect a.jackson@epcc.ed.ac.uk HPC and Big Data EPCC The University of Edinburgh Adrian Jackson Technical Architect a.jackson@epcc.ed.ac.uk EPCC Facilities Technology Transfer European Projects HPC Research Visitor Programmes Training

More information

Design Issues in a Bare PC Web Server

Design Issues in a Bare PC Web Server Design Issues in a Bare PC Web Server Long He, Ramesh K. Karne, Alexander L. Wijesinha, Sandeep Girumala, and Gholam H. Khaksari Department of Computer & Information Sciences, Towson University, 78 York

More information