Towards Autonomic Grid Data Management with Virtualized Distributed File Systems

Size: px
Start display at page:

Download "Towards Autonomic Grid Data Management with Virtualized Distributed File Systems"

Transcription

1 Towards Autonomic Grid Data Management with Virtualized Distributed File Systems Ming Zhao, Jing Xu, Renato Figueiredo Advanced Computing and Information Systems Electrical and Computer Engineering University of Florida {ming, jxu, Advanced Computing and Information Systems laboratory

2 Motivation Grid data provisioning Wide-area latency/bandwidth, dynamism of resources How to provide data with application-tailored optimizations? job job job Compute server job job job Compute server Aggressive prefetching Genome sequence GRID Highly reliable access Simulation data Data server data job job job Compute server Sparse file access Virtual machine Data server Advanced Computing and Information Systems laboratory 2

3 Motivation Grid data management Dynamic data access, resource sharing, changing resource availability How to manage data provisioning in dynamically changing environments? job job job Compute server job job job Compute server Aggressive prefetching Genome sequence GRID Highly reliable access Simulation data Data server data job job job Compute server Sparse file access Virtual machine Data server Advanced Computing and Information Systems laboratory 3

4 Overview Goal: Efficient data management in heterogeneous, dynamic and large-scale Grid environments Challenges: Application-tailored data provisioning Data management in dynamically changing environment Contribution: Autonomic Grid data management Virtualized Grid-wide file systems for user-transparent Grid data access with application-tailored enhancements Autonomic data management services for self-managing, goal-driven control over Grid file system sessions Advanced Computing and Information Systems laboratory 4

5 Outline Background Grid virtual file system and data management Architecture Autonomic data management services Evaluation Experiments and analysis Summary Related work, conclusion and future work Advanced Computing and Information Systems laboratory 5

6 Grid Virtual File Systems Grid Virtual File System (GVFS) Distributed file system virtualization through user-level NFS proxies Enhancements for Grid environment (disk caching, secure tunneling) Dynamic, independent, application-tailored GVFS sessions data job $ C1 job $ GVFS Session I GRID Secure Tunneling S1 data C2 job $ job $ GVFS Session II S2 [Cluster Computing, Vol 9/1, 2006] Advanced Computing and Information Systems laboratory 6

7 Data Management Services WSRF-based management services for GVFS sessions Data Scheduler Service (DSS) Scheduling and customization of GVFS sessions Data Replication Service (DRS) Management of application data sets and their replicas M DRS DSS Server-side File System Service () Management of serverside GVFS proxies C1 GVFS Session I GRID S1 Client-side File System Service () Management of clientside mount points, GVFS proxies and disk caches C2 GVFS Session II S2 [HPDC 2005] Advanced Computing and Information Systems laboratory 7

8 Autonomic Services Autonomic Data Scheduler Service Analyze Plan Performance Monitor Storage Configuration Execute Reconfigure M DSS DRS Autonomic Data Server-side Replication File System Service Service Analyze Plan Reliability Performance Replication Load balance Monitor Execute Storage Response Replicate Reassign Analyze Response C1 Autonomic Client-side File System Service Performance Monitor Prim-server Redirect Plan Execute GVFS Session I C2 GRID GVFS Session II S1 S2 Advanced Computing and Information Systems laboratory 8 [Russell, IBM Systems Journal, 2003]

9 Autonomic Data Scheduler Service Manages concurrent GVFS sessions on shared resources Considers both application performance and resource utilization policy Session i s utility: U i = Performance i * Priority i Configures sessions to maximize U i i job job Analyze Performance Monitor C1 $ $ Storage Plan Configuration Execute Reconfigure S2 S1 E.g. Disk cache management Hit rate is important in wide-area environment Allocates client-side storage among sessions Monitors storage usage via client-side Configures the sessions caches to maximize global utility Applies configurations via client-side Advanced Computing and Information Systems laboratory 9

10 Autonomic Data Replication Service Replication degree and placement Increases reliability against server-side failures (crash, network partitioning) The reliability of a session s data set d : Reliability d > R min Reduces replication overhead The cost of creating replicas for data set d Cost d < C max Replaces replicas based on utility U d = Value d * Reliability d, maximizes U d i Analyze Storage Replica regeneration Monitors server status via server-side Reconfigures sessions replicas after a failure Generates replicas via server-side s C1 Reliability Monitor Plan Replication Execute Replicate replica S1 S2 Advanced Computing and Information Systems laboratory 10

11 Autonomic File System Service Fail-over against server failures Minimizes impact on the application Non-interrupt session redirection Monitors server response time Detects failure when request times out Redirects session to backup server Regenerate replicas via DSS/DRS Primary server selection Maximizes performance against dynamic environment Monitors connections with effective, low-overhead mechanism: Small random writes to a hidden file on the mounted partition Predicts performance with simple effective forecasting algorithm Chooses the best connection and switches transparently C1 Analyze Performance Configuration Monitor Response job $ Plan Execute Redirect S2 S1 Advanced Computing and Information Systems laboratory 11

12 Experimental Setup File system clients and servers VMware-based virtual machines Wide-area networks NIST Net emulated links I/O part of Grid applications IOzone file system benchmark Grid data management User-level NFS v3 based GVFS proxies WSRF-Lite based management services Advanced Computing and Information Systems laboratory 12

13 Response Time (sec) Autonomic Session Redirection Scenario (I) Data transfer under network fluctuation E.g. caused by parallel TCP transfers Setup Client connected to two servers with independently emulated WAN links Randomly applied latencies with values from a real wide-area measurement C1 50 S1 Advanced Computing and Information Systems laboratory 13 S Number of parallel TCP connections Server 1 Server 2 Autonomic session redirection Time (second) Autonomic session redirection achieves the best performance by adapting to the changing network condition Network RTT (ms)

14 Autonomic Session Redirection Scenario (II) Data transfer under server load variation Setup Randomly generated background load applied on the data servers C1 S1 S2 Response Time (sec) Server 1 Server 2 Autonomic session redirection Time (second) Autonomic session redirection achieves the best performance by adapting to the changing server load Advanced Computing and Information Systems laboratory 14

15 Autonomic Data Replication Scenario Data transfer and replication under dynamic server failures Setup Replica degree: 2 (1 primary + 1 backup) Failures randomly injected on the servers Consecutive executions of the benchmark Case I: Independent sessions (different inputs) C1 S1 S2 primary primary server server failed failed new new replica replica created created backup backup server server failed failed new new replica replica created created primary primary server server failed failed new new replica replica created created backup backup server server failed failed new new replica replica created created time (s) st 1st run run done done 2nd 2nd run run done done 3rd 3rd run run done done 4th 4th run run done done redirected redirected to to backup backup redirected redirected to to backup backup Autonomic data replication provides transparent error detection/recovery Advanced Computing and Information Systems laboratory 15

16 Autonomic Data Replication Scenario Data transfer and replication under dynamic server failures Setup Replica degree: 2 (1 primary + 1 backup) Failures randomly injected on the servers Consecutive executions of the benchmark C1 Case II: Dependent sessions: same input, cache reuse S1 S2 primary server failed backup server failed new replica created new replica created 0 time (s): redirected to backup 1st run done 2nd run done 3rd run done 4th run done 5th run done 6th run done 7th run done 8th run done 9th run done 10th run done Client-side disk cache further helps to minimize the impact of failures Advanced Computing and Information Systems laboratory 16

17 Related Work Data management in Grid environment GridFTP, GASS, Condor, Legion Middleware control over Grid data transfers Batch-Aware Distributed File System [NSDI 04] Self-optimizing, fault-tolerant bulk data transfer framework based on Condor and Stork [ISPDC 03] Replication and storage management Wide-area data replication based on Globus RFT IBM autonomic storage manager Advanced Computing and Information Systems laboratory 17

18 Summary Problem: Efficient data management in heterogeneous, dynamic and large-scale Grid environments Solution: Autonomic data management system based on GVFS and self-managing, goal-driven services Future work: Autonomic session checkpointing and migration Decentralized coordinating per-domain DSS/DRS Advanced Computing and Information Systems laboratory 18

19 References [1] M. Zhao, J. Zhang and R. Figueiredo, Distributed File System Virtualization Techniques Supporting On-Demand Virtual Machine Environments for Grid Computing, Cluster Computing, [2] M. Zhao, V. Chadha and R. Figueiredo, Supporting Application-Tailored Grid File System Sessions with WSRF-Based Services, HPDC [3] J. Xu, S. Adabala and J. A.B. Fortes, Towards Autonomic Virtual Applications in the In-VIGO System, ICAC [4] L. W. Russell et al, Clockwork: A New Movement in Autonomic Systems, IBM Systems Journal, 42(1), [5] J. Bent et al, Explicit Control in a Batch-Aware Distributed File System, NSDI [6] T. Kosar et al, A Framework for Self-optimizing, Fault-tolerant, High Performance Bulk Data Transfers in a Heterogeneous Grid Environment, ISPDC [7] A. Chervenak et al, Wide Area Data Replication for Scientific Collaborations, Grid [8] Y. Zhong, S. G. Dropsho, C. Ding, Miss Rate Prediction across All Program Inputs, PACT WSRF::Lite An Implementation of Web Services Resource Framework GVFS Virtualized distributed file system for Grid environment In-VIGO Virtualization middleware for computational Grids Grid virtualization middleware Advanced Computing and Information Systems laboratory 19

20 Acknowledgments In-VIGO team NSF Middleware Initiative NSF Research Resources IBM Shared University Research Intel ISTG R&D Council Questions? Advanced Computing and Information Systems laboratory 20

Proactive, Resource-Aware, Tunable Real-time Fault-tolerant Middleware

Proactive, Resource-Aware, Tunable Real-time Fault-tolerant Middleware Proactive, Resource-Aware, Tunable Real-time Fault-tolerant Middleware Priya Narasimhan T. Dumitraş, A. Paulos, S. Pertet, C. Reverte, J. Slember, D. Srivastava Carnegie Mellon University Problem Description

More information

Efficient Data Management Support for Virtualized Service Providers

Efficient Data Management Support for Virtualized Service Providers Efficient Data Management Support for Virtualized Service Providers Íñigo Goiri, Ferran Julià and Jordi Guitart Barcelona Supercomputing Center - Technical University of Catalonia Jordi Girona 31, 834

More information

An Efficient Use of Virtualization in Grid/Cloud Environments. Supervised by: Elisa Heymann Miquel A. Senar

An Efficient Use of Virtualization in Grid/Cloud Environments. Supervised by: Elisa Heymann Miquel A. Senar An Efficient Use of Virtualization in Grid/Cloud Environments. Arindam Choudhury Supervised by: Elisa Heymann Miquel A. Senar Index Introduction Motivation Objective State of Art Proposed Solution Experimentations

More information

Using Resource Virtualization Techniques to Grid-enable Coupled Coastal Ocean Models

Using Resource Virtualization Techniques to Grid-enable Coupled Coastal Ocean Models Using Resource Virtualization Techniques to Grid-enable Coupled Coastal Ocean Models Renato Figueiredo Arijit Ganguly Advanced Computing and Information Systems Lab Peter Sheng, Justin Davis, Vladimir

More information

Direct NFS - Design considerations for next-gen NAS appliances optimized for database workloads Akshay Shah Gurmeet Goindi Oracle

Direct NFS - Design considerations for next-gen NAS appliances optimized for database workloads Akshay Shah Gurmeet Goindi Oracle Direct NFS - Design considerations for next-gen NAS appliances optimized for database workloads Akshay Shah Gurmeet Goindi Oracle Agenda Introduction Database Architecture Direct NFS Client NFS Server

More information

nanohub.org An Overview of Virtualization Techniques

nanohub.org An Overview of Virtualization Techniques An Overview of Virtualization Techniques Renato Figueiredo Advanced Computing and Information Systems (ACIS) Electrical and Computer Engineering University of Florida NCN/NMI Team 2/3/2006 1 Outline Resource

More information

Server Virtualization with Windows Server Hyper-V and System Center

Server Virtualization with Windows Server Hyper-V and System Center Course 20409B: Server Virtualization with Windows Server Hyper-V and System Center Course Details Course Outline Module 1: Evaluating the Environment for Virtualization This module provides an overview

More information

Berkeley Ninja Architecture

Berkeley Ninja Architecture Berkeley Ninja Architecture ACID vs BASE 1.Strong Consistency 2. Availability not considered 3. Conservative 1. Weak consistency 2. Availability is a primary design element 3. Aggressive --> Traditional

More information

Distributed File System Support for Virtual Machines in Grid Computing

Distributed File System Support for Virtual Machines in Grid Computing Distributed File System Support for Virtual Machines in Grid Computing Ming Zhao Jian Zhang Renato Figueiredo Advanced Computing and Information Systems Laboratory (ACIS) Electrical and Computer Engineering

More information

On the Use of Fuzzy Modeling in Virtualized Data Center Management Jing Xu, Ming Zhao, José A. B. Fortes, Robert Carpenter*, Mazin Yousif*

On the Use of Fuzzy Modeling in Virtualized Data Center Management Jing Xu, Ming Zhao, José A. B. Fortes, Robert Carpenter*, Mazin Yousif* On the Use of Fuzzy Modeling in Virtualized Data Center Management Jing Xu, Ming Zhao, José A. B. Fortes, Robert Carpenter*, Mazin Yousif* Electrical and Computer Engineering, University of Florida *Intel

More information

LARGE-SCALE DATA STORAGE APPLICATIONS

LARGE-SCALE DATA STORAGE APPLICATIONS BENCHMARKING AVAILABILITY AND FAILOVER PERFORMANCE OF LARGE-SCALE DATA STORAGE APPLICATIONS Wei Sun and Alexander Pokluda December 2, 2013 Outline Goal and Motivation Overview of Cassandra and Voldemort

More information

Group Based Load Balancing Algorithm in Cloud Computing Virtualization

Group Based Load Balancing Algorithm in Cloud Computing Virtualization Group Based Load Balancing Algorithm in Cloud Computing Virtualization Rishi Bhardwaj, 2 Sangeeta Mittal, Student, 2 Assistant Professor, Department of Computer Science, Jaypee Institute of Information

More information

IMCM: A Flexible Fine-Grained Adaptive Framework for Parallel Mobile Hybrid Cloud Applications

IMCM: A Flexible Fine-Grained Adaptive Framework for Parallel Mobile Hybrid Cloud Applications Open System Laboratory of University of Illinois at Urbana Champaign presents: Outline: IMCM: A Flexible Fine-Grained Adaptive Framework for Parallel Mobile Hybrid Cloud Applications A Fine-Grained Adaptive

More information

Fault-Tolerant Framework for Load Balancing System

Fault-Tolerant Framework for Load Balancing System Fault-Tolerant Framework for Load Balancing System Y. K. LIU, L.M. CHENG, L.L.CHENG Department of Electronic Engineering City University of Hong Kong Tat Chee Avenue, Kowloon, Hong Kong SAR HONG KONG Abstract:

More information

Dynamic Load Balancing of Virtual Machines using QEMU-KVM

Dynamic Load Balancing of Virtual Machines using QEMU-KVM Dynamic Load Balancing of Virtual Machines using QEMU-KVM Akshay Chandak Krishnakant Jaju Technology, College of Engineering, Pune. Maharashtra, India. Akshay Kanfade Pushkar Lohiya Technology, College

More information

CS 6343: CLOUD COMPUTING Term Project

CS 6343: CLOUD COMPUTING Term Project CS 6343: CLOUD COMPUTING Term Project Group A1 Project: IaaS cloud middleware Create a cloud environment with a number of servers, allowing users to submit their jobs, scale their jobs Make simple resource

More information

Plug-and-play Virtual Appliance Clusters Running Hadoop. Dr. Renato Figueiredo ACIS Lab - University of Florida

Plug-and-play Virtual Appliance Clusters Running Hadoop. Dr. Renato Figueiredo ACIS Lab - University of Florida Plug-and-play Virtual Appliance Clusters Running Hadoop Dr. Renato Figueiredo ACIS Lab - University of Florida Advanced Computing and Information Systems laboratory Introduction You have so far learned

More information

Course Outline. Create and configure virtual hard disks. Create and configure virtual machines. Install and import virtual machines.

Course Outline. Create and configure virtual hard disks. Create and configure virtual machines. Install and import virtual machines. Server Virtualization with Windows Server Hyper-V and System Centre Duration 5 days Course Code SSM20409 Format Instructor Led Overview This five day course will provide you with the knowledge and skills

More information

Server-Virtualisierung mit Windows Server Hyper-V und System Center MOC 20409

Server-Virtualisierung mit Windows Server Hyper-V und System Center MOC 20409 Server-Virtualisierung mit Windows Server Hyper-V und System Center MOC 20409 Course Outline Module 1: Evaluating the Environment for Virtualization This module provides an overview of Microsoft virtualization

More information

Server Virtualization with Windows Server Hyper-V and System Center

Server Virtualization with Windows Server Hyper-V and System Center Server Virtualization with Windows Server Hyper-V and System Center About this Course This five day course will provide you with the knowledge and skills required to design and implement Microsoft Server

More information

Google File System. Web and scalability

Google File System. Web and scalability Google File System Web and scalability The web: - How big is the Web right now? No one knows. - Number of pages that are crawled: o 100,000 pages in 1994 o 8 million pages in 2005 - Crawlable pages might

More information

Hadoop Architecture. Part 1

Hadoop Architecture. Part 1 Hadoop Architecture Part 1 Node, Rack and Cluster: A node is simply a computer, typically non-enterprise, commodity hardware for nodes that contain data. Consider we have Node 1.Then we can add more nodes,

More information

MOSIX: High performance Linux farm

MOSIX: High performance Linux farm MOSIX: High performance Linux farm Paolo Mastroserio [mastroserio@na.infn.it] Francesco Maria Taurino [taurino@na.infn.it] Gennaro Tortone [tortone@na.infn.it] Napoli Index overview on Linux farm farm

More information

THE EXPAND PARALLEL FILE SYSTEM A FILE SYSTEM FOR CLUSTER AND GRID COMPUTING. José Daniel García Sánchez ARCOS Group University Carlos III of Madrid

THE EXPAND PARALLEL FILE SYSTEM A FILE SYSTEM FOR CLUSTER AND GRID COMPUTING. José Daniel García Sánchez ARCOS Group University Carlos III of Madrid THE EXPAND PARALLEL FILE SYSTEM A FILE SYSTEM FOR CLUSTER AND GRID COMPUTING José Daniel García Sánchez ARCOS Group University Carlos III of Madrid Contents 2 The ARCOS Group. Expand motivation. Expand

More information

Quantum StorNext. Product Brief: Distributed LAN Client

Quantum StorNext. Product Brief: Distributed LAN Client Quantum StorNext Product Brief: Distributed LAN Client NOTICE This product brief may contain proprietary information protected by copyright. Information in this product brief is subject to change without

More information

The Data Grid: Towards an Architecture for Distributed Management and Analysis of Large Scientific Datasets

The Data Grid: Towards an Architecture for Distributed Management and Analysis of Large Scientific Datasets The Data Grid: Towards an Architecture for Distributed Management and Analysis of Large Scientific Datasets!! Large data collections appear in many scientific domains like climate studies.!! Users and

More information

Storage I/O Control: Proportional Allocation of Shared Storage Resources

Storage I/O Control: Proportional Allocation of Shared Storage Resources Storage I/O Control: Proportional Allocation of Shared Storage Resources Chethan Kumar Sr. Member of Technical Staff, R&D VMware, Inc. Outline The Problem Storage IO Control (SIOC) overview Technical Details

More information

Eloquence Training What s new in Eloquence B.08.00

Eloquence Training What s new in Eloquence B.08.00 Eloquence Training What s new in Eloquence B.08.00 2010 Marxmeier Software AG Rev:100727 Overview Released December 2008 Supported until November 2013 Supports 32-bit and 64-bit platforms HP-UX Itanium

More information

Outline. MCSA: Server Virtualization

Outline. MCSA: Server Virtualization MCSA: Server Virtualization Description Get hands-on instruction and practice implementing Microsoft Server Virtualization with Windows Server 2012 R2 Hyper-V and System Center 2012 R2 Virtual Machine

More information

Data Management in an International Data Grid Project. Timur Chabuk 04/09/2007

Data Management in an International Data Grid Project. Timur Chabuk 04/09/2007 Data Management in an International Data Grid Project Timur Chabuk 04/09/2007 Intro LHC opened in 2005 several Petabytes of data per year data created at CERN distributed to Regional Centers all over the

More information

Capacity planning for IBM Power Systems using LPAR2RRD. www.lpar2rrd.com www.stor2rrd.com

Capacity planning for IBM Power Systems using LPAR2RRD. www.lpar2rrd.com www.stor2rrd.com Capacity planning for IBM Power Systems using LPAR2RRD Agenda LPAR2RRD and STOR2RRD basic introduction Capacity Planning practical view CPU Capacity Planning LPAR2RRD Premium features Future STOR2RRD quick

More information

Petascale Software Challenges. Piyush Chaudhary piyushc@us.ibm.com High Performance Computing

Petascale Software Challenges. Piyush Chaudhary piyushc@us.ibm.com High Performance Computing Petascale Software Challenges Piyush Chaudhary piyushc@us.ibm.com High Performance Computing Fundamental Observations Applications are struggling to realize growth in sustained performance at scale Reasons

More information

Server Virtualization with Windows Server Hyper-V and System Center

Server Virtualization with Windows Server Hyper-V and System Center Server Virtualization with Windows Server Hyper-V and System Center MS20409 DESCRIZIONE: This five day course will provide you with the knowledge and skills required to design and implement Microsoft Server

More information

Server Virtualization with Windows Server Hyper-V and System Center

Server Virtualization with Windows Server Hyper-V and System Center Course 20409 Server Virtualization with Windows Server Hyper-V and System Center Length: Language(s): Audience(s): 5 Days English IT Professionals Level: 300 Technology: Windows Server 2012 Type: Delivery

More information

Figure 1. The cloud scales: Amazon EC2 growth [2].

Figure 1. The cloud scales: Amazon EC2 growth [2]. - Chung-Cheng Li and Kuochen Wang Department of Computer Science National Chiao Tung University Hsinchu, Taiwan 300 shinji10343@hotmail.com, kwang@cs.nctu.edu.tw Abstract One of the most important issues

More information

RAMCloud and the Low- Latency Datacenter. John Ousterhout Stanford University

RAMCloud and the Low- Latency Datacenter. John Ousterhout Stanford University RAMCloud and the Low- Latency Datacenter John Ousterhout Stanford University Most important driver for innovation in computer systems: Rise of the datacenter Phase 1: large scale Phase 2: low latency Introduction

More information

Data Grids. Lidan Wang April 5, 2007

Data Grids. Lidan Wang April 5, 2007 Data Grids Lidan Wang April 5, 2007 Outline Data-intensive applications Challenges in data access, integration and management in Grid setting Grid services for these data-intensive application Architectural

More information

Server Virtualization with Windows Server Hyper-V and System Center

Server Virtualization with Windows Server Hyper-V and System Center Course 20409 : Server Virtualization with Windows Server Hyper-V and System Center Page 1 of 8 Server Virtualization with Windows Server Hyper-V and System Center Course 20409: 4 days; Instructor-Led Introduction

More information

INUVIKA TECHNICAL GUIDE

INUVIKA TECHNICAL GUIDE --------------------------------------------------------------------------------------------------- INUVIKA TECHNICAL GUIDE FILE SERVER HIGH AVAILABILITY OVD Enterprise External Document Version 1.0 Published

More information

Distribution transparency. Degree of transparency. Openness of distributed systems

Distribution transparency. Degree of transparency. Openness of distributed systems Distributed Systems Principles and Paradigms Maarten van Steen VU Amsterdam, Dept. Computer Science steen@cs.vu.nl Chapter 01: Version: August 27, 2012 1 / 28 Distributed System: Definition A distributed

More information

Lab Testing Summary Report

Lab Testing Summary Report Lab Testing Summary Report July 2006 Report 060725 Product Category: WAN Optimization Appliances Vendors Tested: Citrix Systems TM Riverbed Technology TM Key findings and conclusions: The WANScaler 8800

More information

The Google File System

The Google File System The Google File System Motivations of NFS NFS (Network File System) Allow to access files in other systems as local files Actually a network protocol (initially only one server) Simple and fast server

More information

A High-Performance Virtual Storage System for Taiwan UniGrid

A High-Performance Virtual Storage System for Taiwan UniGrid Journal of Information Technology and Applications Vol. 1 No. 4 March, 2007, pp. 231-238 A High-Performance Virtual Storage System for Taiwan UniGrid Chien-Min Wang; Chun-Chen Hsu and Jan-Jan Wu Institute

More information

Software-defined Storage Architecture for Analytics Computing

Software-defined Storage Architecture for Analytics Computing Software-defined Storage Architecture for Analytics Computing Arati Joshi Performance Engineering Colin Eldridge File System Engineering Carlos Carrero Product Management June 2015 Reference Architecture

More information

High Performance Data-Transfers in Grid Environment using GridFTP over InfiniBand

High Performance Data-Transfers in Grid Environment using GridFTP over InfiniBand High Performance Data-Transfers in Grid Environment using GridFTP over InfiniBand Hari Subramoni *, Ping Lai *, Raj Kettimuthu **, Dhabaleswar. K. (DK) Panda * * Computer Science and Engineering Department

More information

Understanding Data Locality in VMware Virtual SAN

Understanding Data Locality in VMware Virtual SAN Understanding Data Locality in VMware Virtual SAN July 2014 Edition T E C H N I C A L M A R K E T I N G D O C U M E N T A T I O N Table of Contents Introduction... 2 Virtual SAN Design Goals... 3 Data

More information

Real-Time Analysis of CDN in an Academic Institute: A Simulation Study

Real-Time Analysis of CDN in an Academic Institute: A Simulation Study Journal of Algorithms & Computational Technology Vol. 6 No. 3 483 Real-Time Analysis of CDN in an Academic Institute: A Simulation Study N. Ramachandran * and P. Sivaprakasam + *Indian Institute of Management

More information

Appendix A Core Concepts in SQL Server High Availability and Replication

Appendix A Core Concepts in SQL Server High Availability and Replication Appendix A Core Concepts in SQL Server High Availability and Replication Appendix Overview Core Concepts in High Availability Core Concepts in Replication 1 Lesson 1: Core Concepts in High Availability

More information

Gfarm: Present Status and Future Evolution

Gfarm: Present Status and Future Evolution OpenSFS APAC Lustre User Group 2013 Tokyo October 17, 2013 Gfarm: Present Status and Future Evolution Osamu Tatebe University of Tsukuba Gfarm file system Award-winning file system since 2000 Distributed

More information

CASE STUDY: Oracle TimesTen In-Memory Database and Shared Disk HA Implementation at Instance level. -ORACLE TIMESTEN 11gR1

CASE STUDY: Oracle TimesTen In-Memory Database and Shared Disk HA Implementation at Instance level. -ORACLE TIMESTEN 11gR1 CASE STUDY: Oracle TimesTen In-Memory Database and Shared Disk HA Implementation at Instance level -ORACLE TIMESTEN 11gR1 CASE STUDY Oracle TimesTen In-Memory Database and Shared Disk HA Implementation

More information

Project Convergence: Integrating Data Grids and Compute Grids. Eugene Steinberg, CTO Grid Dynamics May, 2008

Project Convergence: Integrating Data Grids and Compute Grids. Eugene Steinberg, CTO Grid Dynamics May, 2008 Project Convergence: Integrating Data Grids and Compute Grids Eugene Steinberg, CTO May, 2008 Data-Driven Scalability Challenges in HPC Data is far away Latency of remote connection Latency of data movement

More information

Recommendations for Performance Benchmarking

Recommendations for Performance Benchmarking Recommendations for Performance Benchmarking Shikhar Puri Abstract Performance benchmarking of applications is increasingly becoming essential before deployment. This paper covers recommendations and best

More information

CS550. Distributed Operating Systems (Advanced Operating Systems) Instructor: Xian-He Sun

CS550. Distributed Operating Systems (Advanced Operating Systems) Instructor: Xian-He Sun CS550 Distributed Operating Systems (Advanced Operating Systems) Instructor: Xian-He Sun Email: sun@iit.edu, Phone: (312) 567-5260 Office hours: 2:10pm-3:10pm Tuesday, 3:30pm-4:30pm Thursday at SB229C,

More information

Autonomic Power & Performance Management of Large-scale Data Centers

Autonomic Power & Performance Management of Large-scale Data Centers Autonomic Power & Performance Management of Large-scale Data Centers Bithika Khargharia 1, Salim Hariri 1, Ferenc Szidarovszky 1, Manal Houri 2, Hesham El-Rewini 2, Samee Khan 3, Ishfaq Ahmad 3 and Mazin

More information

Study of Load Balancing of Resource Namespace Service

Study of Load Balancing of Resource Namespace Service Study of Load Balancing of Resource Namespace Service Masahiro Nakamura, Osamu Tatebe University of Tsukuba Background Resource Namespace Service (RNS) is published as GDF.101 by OGF RNS is intended to

More information

Server Virtualization with Windows Server Hyper-V and System Center

Server Virtualization with Windows Server Hyper-V and System Center Course Code: M20409 Vendor: Microsoft Course Overview Duration: 5 RRP: 2,025 Server Virtualization with Windows Server Hyper-V and System Center Overview This five day course will provide you with the

More information

Datasheet iscsi Protocol

Datasheet iscsi Protocol Protocol with DCB PROTOCOL PACKAGE Industry s premiere validation system for SAN technologies Overview Load DynamiX offers SCSI over TCP/IP transport () support to its existing powerful suite of file,

More information

Load Balancing in Distributed Web Server Systems With Partial Document Replication

Load Balancing in Distributed Web Server Systems With Partial Document Replication Load Balancing in Distributed Web Server Systems With Partial Document Replication Ling Zhuo, Cho-Li Wang and Francis C. M. Lau Department of Computer Science and Information Systems The University of

More information

Virtualization Technologies and Blackboard: The Future of Blackboard Software on Multi-Core Technologies

Virtualization Technologies and Blackboard: The Future of Blackboard Software on Multi-Core Technologies Virtualization Technologies and Blackboard: The Future of Blackboard Software on Multi-Core Technologies Kurt Klemperer, Principal System Performance Engineer kklemperer@blackboard.com Agenda Session Length:

More information

QNAP in vsphere Environment

QNAP in vsphere Environment QNAP in vsphere Environment HOW TO USE QNAP NAS AS A VMWARE DATASTORE VIA NFS Copyright 2009. QNAP Systems, Inc. All Rights Reserved. V1.8 How to use QNAP NAS as a VMware Datastore via NFS QNAP provides

More information

16th International Conference on Control Systems and Computer Science (CSCS16 07)

16th International Conference on Control Systems and Computer Science (CSCS16 07) 16th International Conference on Control Systems and Computer Science (CSCS16 07) TOWARDS AN IO INTENSIVE GRID APPLICATION INSTRUMENTATION IN MEDIOGRID Dacian Tudor 1, Florin Pop 2, Valentin Cristea 2,

More information

Technological Trend. A Framework for Highly-Available Cascaded Real-Time Internet Services. Service Composition. Service Composition

Technological Trend. A Framework for Highly-Available Cascaded Real-Time Internet Services. Service Composition. Service Composition A Framework for Highly-Available Cascaded Real-Time Internet Services Bhaskaran Raman Qualifying Examination Proposal Feb 12, 2001 Examination Committee: Prof. Anthony D. Joseph (Chair) Prof. Randy H.

More information

Overlapping Data Transfer With Application Execution on Clusters

Overlapping Data Transfer With Application Execution on Clusters Overlapping Data Transfer With Application Execution on Clusters Karen L. Reid and Michael Stumm reid@cs.toronto.edu stumm@eecg.toronto.edu Department of Computer Science Department of Electrical and Computer

More information

A Batch Computing Service for the Spot Market. Supreeth Subramanya, Tian Guo, Prateek Sharma, David Irwin, Prashant Shenoy

A Batch Computing Service for the Spot Market. Supreeth Subramanya, Tian Guo, Prateek Sharma, David Irwin, Prashant Shenoy SpotOn: A Batch Computing Service for the Spot Market Supreeth Subramanya, Tian Guo, Prateek Sharma, David Irwin, Prashant Shenoy University of Massachusetts Amherst infrastructure cloud Cost vs. Availability

More information

Transparency in Distributed Systems

Transparency in Distributed Systems Transparency in Distributed Systems By Sudheer R Mantena Abstract The present day network architectures are becoming more and more complicated due to heterogeneity of the network components and mainly

More information

The Lattice Project: A Multi-Model Grid Computing System. Center for Bioinformatics and Computational Biology University of Maryland

The Lattice Project: A Multi-Model Grid Computing System. Center for Bioinformatics and Computational Biology University of Maryland The Lattice Project: A Multi-Model Grid Computing System Center for Bioinformatics and Computational Biology University of Maryland Parallel Computing PARALLEL COMPUTING a form of computation in which

More information

The glite File Transfer Service

The glite File Transfer Service The glite File Transfer Service Peter Kunszt Paolo Badino Ricardo Brito da Rocha James Casey Ákos Frohner Gavin McCance CERN, IT Department 1211 Geneva 23, Switzerland Abstract Transferring data reliably

More information

Globus Striped GridFTP Framework and Server. Raj Kettimuthu, ANL and U. Chicago

Globus Striped GridFTP Framework and Server. Raj Kettimuthu, ANL and U. Chicago Globus Striped GridFTP Framework and Server Raj Kettimuthu, ANL and U. Chicago Outline Introduction Features Motivation Architecture Globus XIO Experimental Results 3 August 2005 The Ohio State University

More information

Intro to Virtualization

Intro to Virtualization Cloud@Ceid Seminars Intro to Virtualization Christos Alexakos Computer Engineer, MSc, PhD C. Sysadmin at Pattern Recognition Lab 1 st Seminar 19/3/2014 Contents What is virtualization How it works Hypervisor

More information

Tushar Joshi Turtle Networks Ltd

Tushar Joshi Turtle Networks Ltd MySQL Database for High Availability Web Applications Tushar Joshi Turtle Networks Ltd www.turtle.net Overview What is High Availability? Web/Network Architecture Applications MySQL Replication MySQL Clustering

More information

Heterogeneous Workload Consolidation for Efficient Management of Data Centers in Cloud Computing

Heterogeneous Workload Consolidation for Efficient Management of Data Centers in Cloud Computing Heterogeneous Workload Consolidation for Efficient Management of Data Centers in Cloud Computing Deep Mann ME (Software Engineering) Computer Science and Engineering Department Thapar University Patiala-147004

More information

Efficient Data Replication Scheme based on Hadoop Distributed File System

Efficient Data Replication Scheme based on Hadoop Distributed File System , pp. 177-186 http://dx.doi.org/10.14257/ijseia.2015.9.12.16 Efficient Data Replication Scheme based on Hadoop Distributed File System Jungha Lee 1, Jaehwa Chung 2 and Daewon Lee 3* 1 Division of Supercomputing,

More information

Delivering SDS simplicity and extreme performance

Delivering SDS simplicity and extreme performance Delivering SDS simplicity and extreme performance Real-World SDS implementation of getting most out of limited hardware Murat Karslioglu Director Storage Systems Nexenta Systems October 2013 1 Agenda Key

More information

Distributed File Systems

Distributed File Systems Distributed File Systems Mauro Fruet University of Trento - Italy 2011/12/19 Mauro Fruet (UniTN) Distributed File Systems 2011/12/19 1 / 39 Outline 1 Distributed File Systems 2 The Google File System (GFS)

More information

Chapter 18: Database System Architectures. Centralized Systems

Chapter 18: Database System Architectures. Centralized Systems Chapter 18: Database System Architectures! Centralized Systems! Client--Server Systems! Parallel Systems! Distributed Systems! Network Types 18.1 Centralized Systems! Run on a single computer system and

More information

ESPRESSO: An Encryption as a Service for Cloud Storage Systems

ESPRESSO: An Encryption as a Service for Cloud Storage Systems 8th International Conference on Autonomous Infrastructure, Management and Security ESPRESSO: An Encryption as a Service for Cloud Storage Systems Kang Seungmin 30 th Jun., 2014 Outline Introduction and

More information

Introduction 1 Performance on Hosted Server 1. Benchmarks 2. System Requirements 7 Load Balancing 7

Introduction 1 Performance on Hosted Server 1. Benchmarks 2. System Requirements 7 Load Balancing 7 Introduction 1 Performance on Hosted Server 1 Figure 1: Real World Performance 1 Benchmarks 2 System configuration used for benchmarks 2 Figure 2a: New tickets per minute on E5440 processors 3 Figure 2b:

More information

Distributed Systems and Recent Innovations: Challenges and Benefits

Distributed Systems and Recent Innovations: Challenges and Benefits Distributed Systems and Recent Innovations: Challenges and Benefits 1. Introduction Krishna Nadiminti, Marcos Dias de Assunção, and Rajkumar Buyya Grid Computing and Distributed Systems Laboratory Department

More information

Write a technical report Present your results Write a workshop/conference paper (optional) Could be a real system, simulation and/or theoretical

Write a technical report Present your results Write a workshop/conference paper (optional) Could be a real system, simulation and/or theoretical Identify a problem Review approaches to the problem Propose a novel approach to the problem Define, design, prototype an implementation to evaluate your approach Could be a real system, simulation and/or

More information

Cyberinfrastructure Education and Hands-on Training Using the CH3D-GTM Virtual Appliance on SURAGrid

Cyberinfrastructure Education and Hands-on Training Using the CH3D-GTM Virtual Appliance on SURAGrid Cyberinfrastructure Education and Hands-on Training Using the CH3D-GTM Virtual Appliance on SURAGrid Renato Figueiredo http://grid-appliance.org J. Davis, J. Fortes, P. Sheng, V. Paramygin, B. Tutak, D.

More information

Rodrigo Fernandes de Mello, Evgueni Dodonov, José Augusto Andrade Filho

Rodrigo Fernandes de Mello, Evgueni Dodonov, José Augusto Andrade Filho Middleware for High Performance Computing Rodrigo Fernandes de Mello, Evgueni Dodonov, José Augusto Andrade Filho University of São Paulo São Carlos, Brazil {mello, eugeni, augustoa}@icmc.usp.br Outline

More information

Cloud Computing with Azure PaaS for Educational Institutions

Cloud Computing with Azure PaaS for Educational Institutions International Journal of Information and Computation Technology. ISSN 0974-2239 Volume 4, Number 2 (2014), pp. 139-144 International Research Publications House http://www. irphouse.com /ijict.htm Cloud

More information

Key Components of WAN Optimization Controller Functionality

Key Components of WAN Optimization Controller Functionality Key Components of WAN Optimization Controller Functionality Introduction and Goals One of the key challenges facing IT organizations relative to application and service delivery is ensuring that the applications

More information

Nirav H. Kapadia. kapadia@purdue.edu fortes@purdue.edu. Abstract

Nirav H. Kapadia. kapadia@purdue.edu fortes@purdue.edu. Abstract The PUNCH Virtual File System: Seamless Access to Decentralized Storage Services in a Computational Grid Renato J. Figueiredo gueire@purdue.edu Nirav H. Kapadia kapadia@purdue.edu Jose A.B.Fortes fortes@purdue.edu

More information

GPFS-OpenStack Integration. Dinesh Subhraveti IBM Research

GPFS-OpenStack Integration. Dinesh Subhraveti IBM Research GPFS-OpenStack Integration Dinesh Subhraveti IBM Research GPFS File Placement Optimization Tradi5onal shared architecture Shared nothing architecture SAN I/O bo*leneck Scale out performance GPFS cluster

More information

SQL Clusters in Virtualized Environments April 10 th, 2013

SQL Clusters in Virtualized Environments April 10 th, 2013 SQL Clusters in Virtualized Environments April 10 th, 2013 Session Overview This session s primary objective is to help you gain insight into where clustering fits in the overall picture of HA/DR as it

More information

20409B: Server Virtualization with Windows Server Hyper-V and System Center

20409B: Server Virtualization with Windows Server Hyper-V and System Center 20409B: Server with Windows Server Hyper-V and System Center Course Details Course Code: Duration: Notes: 20409B 5 days Elements of this syllabus are subject to change. About this course This five day

More information

Validating Long-distance VMware vmotion

Validating Long-distance VMware vmotion Technical Brief Validating Long-distance VMware vmotion with NetApp FlexCache and F5 BIG-IP F5 BIG-IP enables long distance VMware vmotion live migration and optimizes NetApp FlexCache replication. Key

More information

2. Research and Development on the Autonomic Operation. Control Infrastructure Technologies in the Cloud Computing Environment

2. Research and Development on the Autonomic Operation. Control Infrastructure Technologies in the Cloud Computing Environment R&D supporting future cloud computing infrastructure technologies Research and Development on Autonomic Operation Control Infrastructure Technologies in the Cloud Computing Environment DEMPO Hiroshi, KAMI

More information

Distributed File Systems

Distributed File Systems Distributed File Systems Paul Krzyzanowski Rutgers University October 28, 2012 1 Introduction The classic network file systems we examined, NFS, CIFS, AFS, Coda, were designed as client-server applications.

More information

Resume. Wenjing. Date of birth: June 11th, 1982 Nationality: Chinese Phone number: 8610-88236012-608 Cell phone: 13366466802 wuwj@ihep.ac.

Resume. Wenjing. Date of birth: June 11th, 1982 Nationality: Chinese Phone number: 8610-88236012-608 Cell phone: 13366466802 wuwj@ihep.ac. Resume Personal information First name: Wenjing surname: Wu Gender: Female Date of birth: June 11th, 1982 Nationality: Chinese Phone number: 8610-88236012-608 Cell phone: 13366466802 Email: wuwj@ihep.ac.cn

More information

Ceph. A file system a little bit different. Udo Seidel

Ceph. A file system a little bit different. Udo Seidel Ceph A file system a little bit different Udo Seidel Ceph what? So-called parallel distributed cluster file system Started as part of PhD studies at UCSC Public announcement in 2006 at 7 th OSDI File system

More information

Distributed System Principles

Distributed System Principles Distributed System Principles 1 What is a Distributed System? Definition: A distributed system consists of a collection of autonomous computers, connected through a network and distribution middleware,

More information

Welcome to the unit of Hadoop Fundamentals on Hadoop architecture. I will begin with a terminology review and then cover the major components

Welcome to the unit of Hadoop Fundamentals on Hadoop architecture. I will begin with a terminology review and then cover the major components Welcome to the unit of Hadoop Fundamentals on Hadoop architecture. I will begin with a terminology review and then cover the major components of Hadoop. We will see what types of nodes can exist in a Hadoop

More information

Big data management with IBM General Parallel File System

Big data management with IBM General Parallel File System Big data management with IBM General Parallel File System Optimize storage management and boost your return on investment Highlights Handles the explosive growth of structured and unstructured data Offers

More information

Optimization of Cluster Web Server Scheduling from Site Access Statistics

Optimization of Cluster Web Server Scheduling from Site Access Statistics Optimization of Cluster Web Server Scheduling from Site Access Statistics Nartpong Ampornaramveth, Surasak Sanguanpong Faculty of Computer Engineering, Kasetsart University, Bangkhen Bangkok, Thailand

More information

VIDEO SURVEILLANCE WITH SURVEILLUS VMS AND EMC ISILON STORAGE ARRAYS

VIDEO SURVEILLANCE WITH SURVEILLUS VMS AND EMC ISILON STORAGE ARRAYS VIDEO SURVEILLANCE WITH SURVEILLUS VMS AND EMC ISILON STORAGE ARRAYS Successfully configure all solution components Use VMS at the required bandwidth for NAS storage Meet the bandwidth demands of a 2,200

More information

Centralized Systems. A Centralized Computer System. Chapter 18: Database System Architectures

Centralized Systems. A Centralized Computer System. Chapter 18: Database System Architectures Chapter 18: Database System Architectures Centralized Systems! Centralized Systems! Client--Server Systems! Parallel Systems! Distributed Systems! Network Types! Run on a single computer system and do

More information

Efficient Load Balancing using VM Migration by QEMU-KVM

Efficient Load Balancing using VM Migration by QEMU-KVM International Journal of Computer Science and Telecommunications [Volume 5, Issue 8, August 2014] 49 ISSN 2047-3338 Efficient Load Balancing using VM Migration by QEMU-KVM Sharang Telkikar 1, Shreyas Talele

More information