Servicing Seismic and Oil Reservoir Simulation Data through Grid Data Services

Save this PDF as:
 WORD  PNG  TXT  JPG

Size: px
Start display at page:

Download "Servicing Seismic and Oil Reservoir Simulation Data through Grid Data Services"

Transcription

1 Servicing Seismic and Oil Reservoir Simulation Data through Grid Data Services Sivaramakrishnan Narayanan, Tahsin Kurc, Umit Catalyurek and Joel Saltz Multiscale Computing Lab Biomedical Informatics Department The Ohio State University

2 Multiscale Computing Lab Joel Saltz Gagan Agrawal Umit Catalyurek Shannon Hastings Vijay S Kumar Tahsin Kurc Steve Langella Scott Oster Tony Pan Benjamin Rutt Narayanan Sivaramakrishnan, Li Weng Michael Zhang

3 mplementing effective oil and gas production Simulate multiple realizations of multiple geostatistical models and production strategies Evaluate geologic uncertainty and production strategies simultaneously Enable on-demand exploration and comparison of multiple scenarios Integration of a robust, Grid-based computational and data handling infrastructure Distributed databases of reservoir and geophysical data Storage and computing resources at multiple institutions Summary data from datasets Data Seismic, well pressures, reservoir simulations Analysis Production rates, bypass oil, net present value Spatio-temporal queries Store and index simulation results Obtain initial, boundary conditions, input parameters for simulations Generate requests new simulations, ne seismic studies Workflow Run new reservoir simulations

4 Characteristics and Issues Spatio-temporal datasets Simulations carried out/data captured on 3D meshes over many time steps Multiple data attributes per data point (gas pressure, oil saturation, seismic traces, etc). Very large datasets Tens of gigabytes to 100+ TB data Lots of simulation runs Up to thousands of runs for a study are possible Data can be stored in distributed collection of files Distributed datasets Data may be captured at multiple locations by multiple groups Simulations are carried out at multiple sites Common operations: subsetting, filtering, interpolations, projections, comparisons, frequency counts

5 ata Management, Access and Integration Tracking of metadata associated with data Metadata defining simulation parameters, mesh description, files associated with simulations, etc. Metadata defining seismic measurements (location, year, files storing data, etc.) Support for data subsetting and filtering on filebased, distributed datasets Support for on-demand data product generation Track metadata associated with data analysis workflows Grid data services and distributed querying Make data and data products available through Grid service interfaces

6 ata Virtualization pplications developers generally prefer storing data in files upport high level queries on multi-dimensional distributed datasets any possible data abstractions, query interfaces Grid virtualized object-relational database or XML database Grid virtualized objects with user defined methods invoked to access and process data Our Approach Support a basic SQL Select query with a virtual relational table view or a virtual XML database view A lightweight layer on top of datasets Runtime middleware carries out query execution, query planning

7 iddleware Support Data Virtualization: STORM Large data querying capabilities, layered on DataCutter Distributed data virtualization Indexing, Subsetting, Data Cluster/Decluster, Parallel Data Transfer Data Analysis/Processing Workflows: DataCutter Component Framework for Combined Task/Data Parallelism Filtering/Program coupling Service: Distributed C++ component framework On demand data product generation Distributed Metadata and Data Management: Mobius Create, manage, version data definitions Management of metadata and data instances Data integration Grid Data Services (OGSA-DAI) Defines services and interfaces that can be used by clients to specify operations on data resources and data

8 ata Management, Access, Integration Schema Management Mobius OGSA-DAI OGSA-DAI SQL Virtualization of Files STORM Grid Protocols XML Virtualization Metadata Management Mobius OGSA-DAI OGSA-DAI Data Product Generation DataCutter Grid-level data services via OGSA-DAI Management of data definitions and metadata, XML virtualization via Mobius Object-relational virtualization and subsetting of file based datasets via STORM On-demand data product generation via DataCutter STORM, Mobius, DataCutter support data operations on heterogeneous collections of storage and compute clusters

9 ata Management, Access, and Integration SQL Virtualization of Files STORM Data Product Generation DataCutter Schema Management Mobius rid-data Service GSA-DAI) Grid Service Protocols Simulation Data XML Virtualization Metadata Management Mobius Grid-data Service (OGSA-DAI) Grid-data Service (OGSA-DAI) SQL Virtualization of Files STORM Seismic Data Data Product Generation DataCutter XML Virtualization Metadata Management Mobius Grid-data Service (OGSA-DAI) SQL Virtualization of Files STORM Data Product Generation DataCutter XML Virtualization Metadata Management Mobius Seismic/Simulation Data

10 ata Querying and Processing Reservoir Simulations Sp (or CDP) # & position Seismic Data Geostatistics Model 1 Model 2 m realizations Array # Array # Receiver group # Receiver & position group # Receiver & position group # & position Component # Component # Component # Model n Production Strategies Well Pattern 1 Well Pattern 2 Array # Receiver group # Receiver & position group # Receiver & position group # & position Component # Component # Component # Well Pattern p Receiver group # Receiver & position group # Receiver & position group # & position Component # Component # Component #

11 TORM Support efficient selection of the data of interest from distributed scientific datasets and transfer of data from storage clusters to compute clusters Data Subsetting Model Virtual Tables Select Queries Distributed Arrays SELECT <DataElements> FROM Dataset-1, Dataset-2,, Dataset-n WHERE <Expression> AND <Filter(<DataElement>)> GROUP-BY-PROCESSOR ComputeAttribute(<DataElement>)

12 TORM Services Query Meta-data Indexing Data Filtering Partition Source Generation Data Mover

13 rid Data Resource Grid has emerged as an integrated infrastructure for distributed computation OGSA-DAI initiative is to deliver high level data management functionality for the Grid. Defines services and interfaces that can be used by clients to specify operations on data resources and data OGSA-DAI services can be configured to expose a specific database management system. To be a GDS, a service must accept perform documents and return results Interpretation of perform documents is open to interpretation Traditionally wrap SQL queries

14 TORM Data Resource GDS JDBC Driver Data Resource Storm Daemon Data Mover Filter STORM instance Extractor

15 xperimental Setup mob 8 nodes Dual 1.4 GHz AMD Optron 8 GB memory 1.5 TB local disk Xio 16 2 Xeon 2.4 GHz 4 GB memory 7.3 TB FAStT600 disk array Dataset Attributes Record Size Records (millions) Dataset (GB) Cluster, Num nodes Oil Reservoir bytes 3, Mob,03 Seismic bytes 247 1,056 Xio,16 TXm 6 24 bytes X 24 * X / 1M Mob,01 All nodes running linux Gigabit switch

16 TORM Results Seismic Datasets 10-25GB per file. About 30-35TB of Data. STORM I/O Performance Bandwidth (MB/s) Threads 4 Threads Max # XIO nodes

17 omparison with MySQL - 1 Varying table size. Per tuple cost is lesser Execution Time (secs) Table Size (million rows) MySQL-cold MySQL-hot STORM-cold STORM-hot

18 omparison with MySQL - 2 Varying query size Also compare them as data resources Execution Time (secs) MySQL STORM MySQL-DAI STORM-DAI Query Size (num of records)

19 il Reservoir Data Results Improvements due to: treating records as array of bytes, combining results at client Execution Time (secs) STORM STORM-DAI-o STORM-DAI-1 STORM-DAI-50 3 DAIs Query Size (number of records)

20 eismic Data Results 96 x 11GB files on 16 nodes Execution Time (secs) Query Size (number of records) STORM STORM-DAI-o STORM-DAI-1 2-DAIs

21 onclusions Overview of work related to Large Scale Scientific Data Management at Multi-Scale Computing Lab Exposed STORM as a Grid Data Service Results on use case: Oil reservoir management For more info / to download STORM, DataCutter, Mobius or

Application of Grid-Enabled Technologies for Solving Optimization Problems in Data Driven Reservoir Systems

Application of Grid-Enabled Technologies for Solving Optimization Problems in Data Driven Reservoir Systems Application of Grid-Enabled Technologies for Solving Optimization Problems in Data Driven Reservoir Systems M. Parashar, H. Klie, U. Catalyurek, T. Kurc, V. Matossian, J. Saltz, M.F. Wheeler ITR Collaborators

More information

Database Support for Data-driven Scientific Applications in the Grid

Database Support for Data-driven Scientific Applications in the Grid Database Support for Data-driven Scientific Applications in the Grid Sivaramakrishnan Narayanan, Tahsin Kurc, Umit Catalyurek, Joel Saltz Dept. of Biomedical Informatics The Ohio State University Columbus,

More information

XML Database Support for Distributed Execution of Data-intensive Scientific Workflows

XML Database Support for Distributed Execution of Data-intensive Scientific Workflows XML Database Support for Distributed Execution of Data-intensive Scientific Workflows Shannon Hastings, Matheus Ribeiro, Stephen Langella, Scott Oster, Umit Catalyurek, Tony Pan, Kun Huang, Renato Ferreira,

More information

Data Grids. Lidan Wang April 5, 2007

Data Grids. Lidan Wang April 5, 2007 Data Grids Lidan Wang April 5, 2007 Outline Data-intensive applications Challenges in data access, integration and management in Grid setting Grid services for these data-intensive application Architectural

More information

A Generic Model for Querying Multiple Databases in a Distributed Environment Using JDBC and an Uniform Interface

A Generic Model for Querying Multiple Databases in a Distributed Environment Using JDBC and an Uniform Interface A Generic Model for Querying Multiple Databases in a Distributed Environment Using JDBC and an Uniform Interface Barkha Bhagwant Keni, M.Madiajagan, B.Vijayakumar Abstract - This paper discusses about

More information

Towards Dynamic Data-Driven Management of the Ruby Gulch Waste Repository

Towards Dynamic Data-Driven Management of the Ruby Gulch Waste Repository Towards Dynamic Data-Driven Management of the Ruby Gulch Waste Repository Manish Parashar 1, Vincent Matossian 1, Hector Klie 2, Sunil G. Thomas 2, Mary F. Wheeler 2,TahsinKurc 3,JoelSaltz 3, and Roelof

More information

Scaling Objectivity Database Performance with Panasas Scale-Out NAS Storage

Scaling Objectivity Database Performance with Panasas Scale-Out NAS Storage White Paper Scaling Objectivity Database Performance with Panasas Scale-Out NAS Storage A Benchmark Report August 211 Background Objectivity/DB uses a powerful distributed processing architecture to manage

More information

The Data Grid: Towards an Architecture for Distributed Management and Analysis of Large Scientific Datasets

The Data Grid: Towards an Architecture for Distributed Management and Analysis of Large Scientific Datasets The Data Grid: Towards an Architecture for Distributed Management and Analysis of Large Scientific Datasets!! Large data collections appear in many scientific domains like climate studies.!! Users and

More information

In-Situ Bitmaps Generation and Efficient Data Analysis based on Bitmaps. Yu Su, Yi Wang, Gagan Agrawal The Ohio State University

In-Situ Bitmaps Generation and Efficient Data Analysis based on Bitmaps. Yu Su, Yi Wang, Gagan Agrawal The Ohio State University In-Situ Bitmaps Generation and Efficient Data Analysis based on Bitmaps Yu Su, Yi Wang, Gagan Agrawal The Ohio State University Motivation HPC Trends Huge performance gap CPU: extremely fast for generating

More information

A simulation and data analysis system for large-scale, data-driven oil reservoir simulation studies

A simulation and data analysis system for large-scale, data-driven oil reservoir simulation studies CONCURRENCY AND COMPUTATION: PRACTICE AND EXPERIENCE Concurrency Computat.: Pract. Exper. 2005; 17:1441 1467 Published online in Wiley InterScience (www.interscience.wiley.com). DOI: 10.1002/cpe.898 A

More information

NoSQL Performance Test In-Memory Performance Comparison of SequoiaDB, Cassandra, and MongoDB

NoSQL Performance Test In-Memory Performance Comparison of SequoiaDB, Cassandra, and MongoDB bankmark UG (haftungsbeschränkt) Bahnhofstraße 1 9432 Passau Germany www.bankmark.de info@bankmark.de T +49 851 25 49 49 F +49 851 25 49 499 NoSQL Performance Test In-Memory Performance Comparison of SequoiaDB,

More information

Analisi di un servizio SRM: StoRM

Analisi di un servizio SRM: StoRM 27 November 2007 General Parallel File System (GPFS) The StoRM service Deployment configuration Authorization and ACLs Conclusions. Definition of terms Definition of terms 1/2 Distributed File System The

More information

THE EXPAND PARALLEL FILE SYSTEM A FILE SYSTEM FOR CLUSTER AND GRID COMPUTING. José Daniel García Sánchez ARCOS Group University Carlos III of Madrid

THE EXPAND PARALLEL FILE SYSTEM A FILE SYSTEM FOR CLUSTER AND GRID COMPUTING. José Daniel García Sánchez ARCOS Group University Carlos III of Madrid THE EXPAND PARALLEL FILE SYSTEM A FILE SYSTEM FOR CLUSTER AND GRID COMPUTING José Daniel García Sánchez ARCOS Group University Carlos III of Madrid Contents 2 The ARCOS Group. Expand motivation. Expand

More information

Abstract. 1. Introduction. Ohio State University Columbus, OH 43210 {langella,oster,hastings,kurc,saltz}@bmi.osu.edu

Abstract. 1. Introduction. Ohio State University Columbus, OH 43210 {langella,oster,hastings,kurc,saltz}@bmi.osu.edu Dorian: Grid Service Infrastructure for Identity Management and Federation Stephen Langella 1, Scott Oster 1, Shannon Hastings 1, Frank Siebenlist 2, Tahsin Kurc 1, Joel Saltz 1 1 Department of Biomedical

More information

Globus Striped GridFTP Framework and Server. Raj Kettimuthu, ANL and U. Chicago

Globus Striped GridFTP Framework and Server. Raj Kettimuthu, ANL and U. Chicago Globus Striped GridFTP Framework and Server Raj Kettimuthu, ANL and U. Chicago Outline Introduction Features Motivation Architecture Globus XIO Experimental Results 3 August 2005 The Ohio State University

More information

Achieving Nanosecond Latency Between Applications with IPC Shared Memory Messaging

Achieving Nanosecond Latency Between Applications with IPC Shared Memory Messaging Achieving Nanosecond Latency Between Applications with IPC Shared Memory Messaging In some markets and scenarios where competitive advantage is all about speed, speed is measured in micro- and even nano-seconds.

More information

Client-aware Cloud Storage

Client-aware Cloud Storage Client-aware Cloud Storage Feng Chen Computer Science & Engineering Louisiana State University Michael Mesnier Circuits & Systems Research Intel Labs Scott Hahn Circuits & Systems Research Intel Labs Cloud

More information

A Grid Based Image Archival and Analysis System

A Grid Based Image Archival and Analysis System A Grid Based Image Archival and Analysis System Shannon Hastings, MS, Scott Oster, MS, Steve Langella, MS, Tahsin M. Kurc, PhD, Tony Pan, MS, Umit V. Catalyurek, PhD, Joel H. Saltz, MD, PhD Department

More information

Comparing the performance of the Landmark Nexus reservoir simulator on HP servers

Comparing the performance of the Landmark Nexus reservoir simulator on HP servers WHITE PAPER Comparing the performance of the Landmark Nexus reservoir simulator on HP servers Landmark Software & Services SOFTWARE AND ASSET SOLUTIONS Comparing the performance of the Landmark Nexus

More information

Managing Large Imagery Databases via the Web

Managing Large Imagery Databases via the Web 'Photogrammetric Week 01' D. Fritsch & R. Spiller, Eds. Wichmann Verlag, Heidelberg 2001. Meyer 309 Managing Large Imagery Databases via the Web UWE MEYER, Dortmund ABSTRACT The terramapserver system is

More information

Agility Database Scalability Testing

Agility Database Scalability Testing Agility Database Scalability Testing V1.6 November 11, 2012 Prepared by on behalf of Table of Contents 1 Introduction... 4 1.1 Brief... 4 2 Scope... 5 3 Test Approach... 6 4 Test environment setup... 7

More information

Enabling Technologies for Distributed Computing

Enabling Technologies for Distributed Computing Enabling Technologies for Distributed Computing Dr. Sanjay P. Ahuja, Ph.D. Fidelity National Financial Distinguished Professor of CIS School of Computing, UNF Multi-core CPUs and Multithreading Technologies

More information

Virtualization of a Cluster Batch System

Virtualization of a Cluster Batch System Virtualization of a Cluster Batch System Christian Baun, Volker Büge, Benjamin Klein, Jens Mielke, Oliver Oberst and Armin Scheurer Die Kooperation von Cluster Batch System Batch system accepts computational

More information

High Performance Data-Transfers in Grid Environment using GridFTP over InfiniBand

High Performance Data-Transfers in Grid Environment using GridFTP over InfiniBand High Performance Data-Transfers in Grid Environment using GridFTP over InfiniBand Hari Subramoni *, Ping Lai *, Raj Kettimuthu **, Dhabaleswar. K. (DK) Panda * * Computer Science and Engineering Department

More information

Introduction. Need for ever-increasing storage scalability. Arista and Panasas provide a unique Cloud Storage solution

Introduction. Need for ever-increasing storage scalability. Arista and Panasas provide a unique Cloud Storage solution Arista 10 Gigabit Ethernet Switch Lab-Tested with Panasas ActiveStor Parallel Storage System Delivers Best Results for High-Performance and Low Latency for Scale-Out Cloud Storage Applications Introduction

More information

Data-Intensive Science and Scientific Data Infrastructure

Data-Intensive Science and Scientific Data Infrastructure Data-Intensive Science and Scientific Data Infrastructure Russ Rew, UCAR Unidata ICTP Advanced School on High Performance and Grid Computing 13 April 2011 Overview Data-intensive science Publishing scientific

More information

Using the Grid for the interactive workflow management in biomedicine. Andrea Schenone BIOLAB DIST University of Genova

Using the Grid for the interactive workflow management in biomedicine. Andrea Schenone BIOLAB DIST University of Genova Using the Grid for the interactive workflow management in biomedicine Andrea Schenone BIOLAB DIST University of Genova overview background requirements solution case study results background A multilevel

More information

Run your own Oracle Database Benchmarks with Hammerora

Run your own Oracle Database Benchmarks with Hammerora Run your own Oracle Database Benchmarks with Hammerora Steve Shaw Intel Corporation UK Keywords: Database, Benchmark Performance, TPC-C, TPC-H, Hammerora Introduction The pace of change in database infrastructure

More information

Removing Performance Bottlenecks in Databases with Red Hat Enterprise Linux and Violin Memory Flash Storage Arrays. Red Hat Performance Engineering

Removing Performance Bottlenecks in Databases with Red Hat Enterprise Linux and Violin Memory Flash Storage Arrays. Red Hat Performance Engineering Removing Performance Bottlenecks in Databases with Red Hat Enterprise Linux and Violin Memory Flash Storage Arrays Red Hat Performance Engineering Version 1.0 August 2013 1801 Varsity Drive Raleigh NC

More information

Data Services @neurist and beyond

Data Services @neurist and beyond s @neurist and beyond Siegfried Benkner Department of Scientific Computing Faculty of Computer Science University of Vienna http://www.par.univie.ac.at Department of Scientific Computing Parallel Computing

More information

CMS Query Suite. CS4440 Project Proposal. Chris Baker Michael Cook Soumo Gorai

CMS Query Suite. CS4440 Project Proposal. Chris Baker Michael Cook Soumo Gorai CMS Query Suite CS4440 Project Proposal Chris Baker Michael Cook Soumo Gorai I) Motivation Relational databases are great places to efficiently store large amounts of information. However, information

More information

Dell Reference Configuration for Hortonworks Data Platform

Dell Reference Configuration for Hortonworks Data Platform Dell Reference Configuration for Hortonworks Data Platform A Quick Reference Configuration Guide Armando Acosta Hadoop Product Manager Dell Revolutionary Cloud and Big Data Group Kris Applegate Solution

More information

Deliverable 2.1.4. 150 Billion Triple dataset hosted on the LOD2 Knowledge Store Cluster. LOD2 Creating Knowledge out of Interlinked Data

Deliverable 2.1.4. 150 Billion Triple dataset hosted on the LOD2 Knowledge Store Cluster. LOD2 Creating Knowledge out of Interlinked Data Collaborative Project LOD2 Creating Knowledge out of Interlinked Data Project Number: 257943 Start Date of Project: 01/09/2010 Duration: 48 months Deliverable 2.1.4 150 Billion Triple dataset hosted on

More information

Parallel Processing of Large-Scale XML-Based Application Documents on Multi-core Architectures with PiXiMaL

Parallel Processing of Large-Scale XML-Based Application Documents on Multi-core Architectures with PiXiMaL 1 / 35 Parallel Processing of Large-Scale XML-Based Application Documents on Multi-core Architectures with PiXiMaL Michael R. Head Madhusudhan Govindaraju Department of Computer Science Grid Computing

More information

Efficient Data Access and Data Integration Using Information Objects Mica J. Block

Efficient Data Access and Data Integration Using Information Objects Mica J. Block Efficient Data Access and Data Integration Using Information Objects Mica J. Block Director, ACES Actuate Corporation mblock@actuate.com Agenda Information Objects Overview Best practices Modeling Security

More information

An Oracle White Paper June 2012. High Performance Connectors for Load and Access of Data from Hadoop to Oracle Database

An Oracle White Paper June 2012. High Performance Connectors for Load and Access of Data from Hadoop to Oracle Database An Oracle White Paper June 2012 High Performance Connectors for Load and Access of Data from Hadoop to Oracle Database Executive Overview... 1 Introduction... 1 Oracle Loader for Hadoop... 2 Oracle Direct

More information

Oracle TimesTen In-Memory Database on Oracle Exalogic Elastic Cloud

Oracle TimesTen In-Memory Database on Oracle Exalogic Elastic Cloud An Oracle White Paper July 2011 Oracle TimesTen In-Memory Database on Oracle Exalogic Elastic Cloud Executive Summary... 3 Introduction... 4 Hardware and Software Overview... 5 Compute Node... 5 Storage

More information

POSIX and Object Distributed Storage Systems

POSIX and Object Distributed Storage Systems 1 POSIX and Object Distributed Storage Systems Performance Comparison Studies With Real-Life Scenarios in an Experimental Data Taking Context Leveraging OpenStack Swift & Ceph by Michael Poat, Dr. Jerome

More information

Benchmarking Cassandra on Violin

Benchmarking Cassandra on Violin Technical White Paper Report Technical Report Benchmarking Cassandra on Violin Accelerating Cassandra Performance and Reducing Read Latency With Violin Memory Flash-based Storage Arrays Version 1.0 Abstract

More information

Enabling Technologies for Distributed and Cloud Computing

Enabling Technologies for Distributed and Cloud Computing Enabling Technologies for Distributed and Cloud Computing Dr. Sanjay P. Ahuja, Ph.D. 2010-14 FIS Distinguished Professor of Computer Science School of Computing, UNF Multi-core CPUs and Multithreading

More information

THE FLORIDA STATE UNIVERSITY COLLEGE OF ARTS AND SCIENCE COMPARING HADOOPDB: A HYBRID OF DBMS AND MAPREDUCE TECHNOLOGIES WITH THE DBMS POSTGRESQL

THE FLORIDA STATE UNIVERSITY COLLEGE OF ARTS AND SCIENCE COMPARING HADOOPDB: A HYBRID OF DBMS AND MAPREDUCE TECHNOLOGIES WITH THE DBMS POSTGRESQL THE FLORIDA STATE UNIVERSITY COLLEGE OF ARTS AND SCIENCE COMPARING HADOOPDB: A HYBRID OF DBMS AND MAPREDUCE TECHNOLOGIES WITH THE DBMS POSTGRESQL By VANESSA CEDENO A Dissertation submitted to the Department

More information

GraySort on Apache Spark by Databricks

GraySort on Apache Spark by Databricks GraySort on Apache Spark by Databricks Reynold Xin, Parviz Deyhim, Ali Ghodsi, Xiangrui Meng, Matei Zaharia Databricks Inc. Apache Spark Sorting in Spark Overview Sorting Within a Partition Range Partitioner

More information

Adam Rauch Partner, LabKey Software adam@labkey.com. Extending LabKey Server Part 1: Retrieving and Presenting Data

Adam Rauch Partner, LabKey Software adam@labkey.com. Extending LabKey Server Part 1: Retrieving and Presenting Data Adam Rauch Partner, LabKey Software adam@labkey.com Extending LabKey Server Part 1: Retrieving and Presenting Data Extending LabKey Server LabKey Server is a large system that combines an extensive set

More information

Exploiting Remote Memory Operations to Design Efficient Reconfiguration for Shared Data-Centers over InfiniBand

Exploiting Remote Memory Operations to Design Efficient Reconfiguration for Shared Data-Centers over InfiniBand Exploiting Remote Memory Operations to Design Efficient Reconfiguration for Shared Data-Centers over InfiniBand P. Balaji, K. Vaidyanathan, S. Narravula, K. Savitha, H. W. Jin D. K. Panda Network Based

More information

Performance Characteristics of VMFS and RDM VMware ESX Server 3.0.1

Performance Characteristics of VMFS and RDM VMware ESX Server 3.0.1 Performance Study Performance Characteristics of and RDM VMware ESX Server 3.0.1 VMware ESX Server offers three choices for managing disk access in a virtual machine VMware Virtual Machine File System

More information

Chapter 7. Using Hadoop Cluster and MapReduce

Chapter 7. Using Hadoop Cluster and MapReduce Chapter 7 Using Hadoop Cluster and MapReduce Modeling and Prototyping of RMS for QoS Oriented Grid Page 152 7. Using Hadoop Cluster and MapReduce for Big Data Problems The size of the databases used in

More information

JBoss Seam Performance and Scalability on Dell PowerEdge 1855 Blade Servers

JBoss Seam Performance and Scalability on Dell PowerEdge 1855 Blade Servers JBoss Seam Performance and Scalability on Dell PowerEdge 1855 Blade Servers Dave Jaffe, PhD, Dell Inc. Michael Yuan, PhD, JBoss / RedHat June 14th, 2006 JBoss Inc. 2006 About us Dave Jaffe Works for Dell

More information

Consumer vs Professional How to Select the Best Graphics Card For Your Workflow

Consumer vs Professional How to Select the Best Graphics Card For Your Workflow Consumer vs Professional How to Select the Best Graphics Card For Your Workflow Allen Bourgoyne Director, ISV Alliances, AMD Professional Graphics Learning Objectives At the end of this class, you will

More information

www.thinkparq.com www.beegfs.com

www.thinkparq.com www.beegfs.com www.thinkparq.com www.beegfs.com KEY ASPECTS Maximum Flexibility Maximum Scalability BeeGFS supports a wide range of Linux distributions such as RHEL/Fedora, SLES/OpenSuse or Debian/Ubuntu as well as a

More information

Agenda. HPC Software Stack. HPC Post-Processing Visualization. Case Study National Scientific Center. European HPC Benchmark Center Montpellier PSSC

Agenda. HPC Software Stack. HPC Post-Processing Visualization. Case Study National Scientific Center. European HPC Benchmark Center Montpellier PSSC HPC Architecture End to End Alexandre Chauvin Agenda HPC Software Stack Visualization National Scientific Center 2 Agenda HPC Software Stack Alexandre Chauvin Typical HPC Software Stack Externes LAN Typical

More information

Dragon Medical Enterprise Network Edition Technical Note: Requirements for DMENE Networks with virtual servers

Dragon Medical Enterprise Network Edition Technical Note: Requirements for DMENE Networks with virtual servers Dragon Medical Enterprise Network Edition Technical Note: Requirements for DMENE Networks with virtual servers This section includes system requirements for DMENE Network configurations that utilize virtual

More information

HP reference configuration for entry-level SAS Grid Manager solutions

HP reference configuration for entry-level SAS Grid Manager solutions HP reference configuration for entry-level SAS Grid Manager solutions Up to 864 simultaneous SAS jobs and more than 3 GB/s I/O throughput Technical white paper Table of contents Executive summary... 2

More information

Principles and Experiences: Building a Hybrid Metadata Service for Service Oriented Architecture based Grid Applications

Principles and Experiences: Building a Hybrid Metadata Service for Service Oriented Architecture based Grid Applications Principles and Experiences: Building a Hybrid Metadata Service for Service Oriented Architecture based Grid Applications Mehmet S. Aktas, Geoffrey C. Fox, and Marlon Pierce Abstract To link multiple varying

More information

Cluster Scalability of ANSYS FLUENT 12 for a Large Aerodynamics Case on the Darwin Supercomputer

Cluster Scalability of ANSYS FLUENT 12 for a Large Aerodynamics Case on the Darwin Supercomputer Cluster Scalability of ANSYS FLUENT 12 for a Large Aerodynamics Case on the Darwin Supercomputer Stan Posey, MSc and Bill Loewe, PhD Panasas Inc., Fremont, CA, USA Paul Calleja, PhD University of Cambridge,

More information

Converged storage architecture for Oracle RAC based on NVMe SSDs and standard x86 servers

Converged storage architecture for Oracle RAC based on NVMe SSDs and standard x86 servers Converged storage architecture for Oracle RAC based on NVMe SSDs and standard x86 servers White Paper rev. 2015-11-27 2015 FlashGrid Inc. 1 www.flashgrid.io Abstract Oracle Real Application Clusters (RAC)

More information

SQL Server Business Intelligence on HP ProLiant DL785 Server

SQL Server Business Intelligence on HP ProLiant DL785 Server SQL Server Business Intelligence on HP ProLiant DL785 Server By Ajay Goyal www.scalabilityexperts.com Mike Fitzner Hewlett Packard www.hp.com Recommendations presented in this document should be thoroughly

More information

Microsoft SQL Server OLTP Best Practice

Microsoft SQL Server OLTP Best Practice Microsoft SQL Server OLTP Best Practice The document Introduction to Transactional (OLTP) Load Testing for all Databases provides a general overview on the HammerDB OLTP workload and the document Microsoft

More information

The Construction of Seismic and Geological Studies' Cloud Platform Using Desktop Cloud Visualization Technology

The Construction of Seismic and Geological Studies' Cloud Platform Using Desktop Cloud Visualization Technology Send Orders for Reprints to reprints@benthamscience.ae 1582 The Open Cybernetics & Systemics Journal, 2015, 9, 1582-1586 Open Access The Construction of Seismic and Geological Studies' Cloud Platform Using

More information

Best Practices for Optimizing Your Linux VPS and Cloud Server Infrastructure

Best Practices for Optimizing Your Linux VPS and Cloud Server Infrastructure Best Practices for Optimizing Your Linux VPS and Cloud Server Infrastructure Q1 2012 Maximizing Revenue per Server with Parallels Containers for Linux www.parallels.com Table of Contents Overview... 3

More information

Solution for private cloud computing

Solution for private cloud computing The CC1 system Solution for private cloud computing 1 Outline What is CC1? Features Technical details System requirements and installation How to get it? 2 What is CC1? The CC1 system is a complete solution

More information

Sawmill Log Analyzer Best Practices!! Page 1 of 6. Sawmill Log Analyzer Best Practices

Sawmill Log Analyzer Best Practices!! Page 1 of 6. Sawmill Log Analyzer Best Practices Sawmill Log Analyzer Best Practices!! Page 1 of 6 Sawmill Log Analyzer Best Practices! Sawmill Log Analyzer Best Practices!! Page 2 of 6 This document describes best practices for the Sawmill universal

More information

Clusters: Mainstream Technology for CAE

Clusters: Mainstream Technology for CAE Clusters: Mainstream Technology for CAE Alanna Dwyer HPC Division, HP Linux and Clusters Sparked a Revolution in High Performance Computing! Supercomputing performance now affordable and accessible Linux

More information

TECHNICAL GUIDELINES FOR APPLICANTS TO PRACE 7 th CALL (Tier-0)

TECHNICAL GUIDELINES FOR APPLICANTS TO PRACE 7 th CALL (Tier-0) TECHNICAL GUIDELINES FOR APPLICANTS TO PRACE 7 th CALL (Tier-0) Contributing sites and the corresponding computer systems for this call are: GCS@Jülich, Germany IBM Blue Gene/Q GENCI@CEA, France Bull Bullx

More information

Ultimate Guide to Oracle Storage

Ultimate Guide to Oracle Storage Ultimate Guide to Oracle Storage Presented by George Trujillo George.Trujillo@trubix.com George Trujillo Twenty two years IT experience with 19 years Oracle experience. Advanced database solutions such

More information

HP SN1000E 16 Gb Fibre Channel HBA Evaluation

HP SN1000E 16 Gb Fibre Channel HBA Evaluation HP SN1000E 16 Gb Fibre Channel HBA Evaluation Evaluation report prepared under contract with Emulex Executive Summary The computing industry is experiencing an increasing demand for storage performance

More information

Cray DVS: Data Virtualization Service

Cray DVS: Data Virtualization Service Cray : Data Virtualization Service Stephen Sugiyama and David Wallace, Cray Inc. ABSTRACT: Cray, the Cray Data Virtualization Service, is a new capability being added to the XT software environment with

More information

A Service for Data-Intensive Computations on Virtual Clusters

A Service for Data-Intensive Computations on Virtual Clusters A Service for Data-Intensive Computations on Virtual Clusters Executing Preservation Strategies at Scale Rainer Schmidt, Christian Sadilek, and Ross King rainer.schmidt@arcs.ac.at Planets Project Permanent

More information

Database Server Configuration Best Practices for Aras Innovator 10

Database Server Configuration Best Practices for Aras Innovator 10 Database Server Configuration Best Practices for Aras Innovator 10 Aras Innovator 10 Running on SQL Server 2012 Enterprise Edition Contents Executive Summary... 1 Introduction... 2 Overview... 2 Aras Innovator

More information

Solution for private cloud computing

Solution for private cloud computing The CC1 system Solution for private cloud computing 1 Outline What is CC1? Features Technical details Use cases By scientist By HEP experiment System requirements and installation How to get it? 2 What

More information

Sockets vs. RDMA Interface over 10-Gigabit Networks: An In-depth Analysis of the Memory Traffic Bottleneck

Sockets vs. RDMA Interface over 10-Gigabit Networks: An In-depth Analysis of the Memory Traffic Bottleneck Sockets vs. RDMA Interface over 1-Gigabit Networks: An In-depth Analysis of the Memory Traffic Bottleneck Pavan Balaji Hemal V. Shah D. K. Panda Network Based Computing Lab Computer Science and Engineering

More information

Business white paper. HP Process Automation. Version 7.0. Server performance

Business white paper. HP Process Automation. Version 7.0. Server performance Business white paper HP Process Automation Version 7.0 Server performance Table of contents 3 Summary of results 4 Benchmark profile 5 Benchmark environmant 6 Performance metrics 6 Process throughput 6

More information

Moving Virtual Storage to the Cloud

Moving Virtual Storage to the Cloud Moving Virtual Storage to the Cloud White Paper Guidelines for Hosters Who Want to Enhance Their Cloud Offerings with Cloud Storage www.parallels.com Table of Contents Overview... 3 Understanding the Storage

More information

Why Not Oracle Standard Edition? A Dbvisit White Paper By Anton Els

Why Not Oracle Standard Edition? A Dbvisit White Paper By Anton Els Why Not Oracle Standard Edition? A Dbvisit White Paper By Anton Els Copyright 2011-2013 Dbvisit Software Limited. All Rights Reserved Nov 2013 Executive Summary... 3 Target Audience... 3 Introduction...

More information

THE ATLAS DISTRIBUTED DATA MANAGEMENT SYSTEM & DATABASES

THE ATLAS DISTRIBUTED DATA MANAGEMENT SYSTEM & DATABASES THE ATLAS DISTRIBUTED DATA MANAGEMENT SYSTEM & DATABASES Vincent Garonne, Mario Lassnig, Martin Barisits, Thomas Beermann, Ralph Vigne, Cedric Serfon Vincent.Garonne@cern.ch ph-adp-ddm-lab@cern.ch XLDB

More information

EMC Unified Storage for Microsoft SQL Server 2008

EMC Unified Storage for Microsoft SQL Server 2008 EMC Unified Storage for Microsoft SQL Server 2008 Enabled by EMC CLARiiON and EMC FAST Cache Reference Copyright 2010 EMC Corporation. All rights reserved. Published October, 2010 EMC believes the information

More information

Managing a Fibre Channel Storage Area Network

Managing a Fibre Channel Storage Area Network Managing a Fibre Channel Storage Area Network Storage Network Management Working Group for Fibre Channel (SNMWG-FC) November 20, 1998 Editor: Steven Wilson Abstract This white paper describes the typical

More information

Data Sharing Options for Scientific Workflows on Amazon EC2

Data Sharing Options for Scientific Workflows on Amazon EC2 Data Sharing Options for Scientific Workflows on Amazon EC2 Gideon Juve, Ewa Deelman, Karan Vahi, Gaurang Mehta, Benjamin P. Berman, Bruce Berriman, Phil Maechling Francesco Allertsen Vrije Universiteit

More information

Expansion of a Framework for a Data- Intensive Wide-Area Application to the Java Language

Expansion of a Framework for a Data- Intensive Wide-Area Application to the Java Language Expansion of a Framework for a Data- Intensive Wide-Area Application to the Java Sergey Koren Table of Contents 1 INTRODUCTION... 3 2 PREVIOUS RESEARCH... 3 3 ALGORITHM...4 3.1 APPROACH... 4 3.2 C++ INFRASTRUCTURE

More information

CMS Tier-3 cluster at NISER. Dr. Tania Moulik

CMS Tier-3 cluster at NISER. Dr. Tania Moulik CMS Tier-3 cluster at NISER Dr. Tania Moulik What and why? Grid computing is a term referring to the combination of computer resources from multiple administrative domains to reach common goal. Grids tend

More information

Implementation & Capacity Planning Specification

Implementation & Capacity Planning Specification White Paper Implementation & Capacity Planning Specification Release 7.1 October 2014 Yellowfin, and the Yellowfin logo are trademarks or registered trademarks of Yellowfin International Pty Ltd. All other

More information

Distributed Data Storage Based on Web Access and IBP Infrastructure. Faculty of Informatics Masaryk University Brno, The Czech Republic

Distributed Data Storage Based on Web Access and IBP Infrastructure. Faculty of Informatics Masaryk University Brno, The Czech Republic Distributed Data Storage Based on Web Access and IBP Infrastructure Lukáš Hejtmánek Faculty of Informatics Masaryk University Brno, The Czech Republic Summary New web based distributed data storage infrastructure

More information

Oracle Database Scalability in VMware ESX VMware ESX 3.5

Oracle Database Scalability in VMware ESX VMware ESX 3.5 Performance Study Oracle Database Scalability in VMware ESX VMware ESX 3.5 Database applications running on individual physical servers represent a large consolidation opportunity. However enterprises

More information

Performance Evaluation of the XDEM framework on the OpenStack Cloud Computing Middleware

Performance Evaluation of the XDEM framework on the OpenStack Cloud Computing Middleware Performance Evaluation of the XDEM framework on the OpenStack Cloud Computing Middleware 1 / 17 Performance Evaluation of the XDEM framework on the OpenStack Cloud Computing Middleware X. Besseron 1 V.

More information

The Data Access Handbook

The Data Access Handbook The Data Access Handbook Achieving Optimal Database Application Performance and Scalability John Goodson and Robert A. Steward PRENTICE HALL Upper Saddle River, NJ Boston Indianapolis San Francisco New

More information

A Theory of the Spatial Computational Domain

A Theory of the Spatial Computational Domain A Theory of the Spatial Computational Domain Shaowen Wang 1 and Marc P. Armstrong 2 1 Academic Technologies Research Services and Department of Geography, The University of Iowa Iowa City, IA 52242 Tel:

More information

PUBLIC Performance Optimization Guide

PUBLIC Performance Optimization Guide SAP Data Services Document Version: 4.2 Support Package 6 (14.2.6.0) 2015-11-20 PUBLIC Content 1 Welcome to SAP Data Services....6 1.1 Welcome.... 6 1.2 Documentation set for SAP Data Services....6 1.3

More information

The ORIENTGATE data platform

The ORIENTGATE data platform Seminar on Proposed and Revised set of indicators June 4-5, 2014 - Belgrade (Serbia) The ORIENTGATE data platform WP2, Action 2.4 Alessandra Nuzzo, Sandro Fiore, Giovanni Aloisio Scientific Computing and

More information

Data Integrator Performance Optimization Guide

Data Integrator Performance Optimization Guide Data Integrator Performance Optimization Guide Data Integrator 11.7.2 for Windows and UNIX Patents Trademarks Copyright Third-party contributors Business Objects owns the following

More information

HPC and Big Data. EPCC The University of Edinburgh. Adrian Jackson Technical Architect a.jackson@epcc.ed.ac.uk

HPC and Big Data. EPCC The University of Edinburgh. Adrian Jackson Technical Architect a.jackson@epcc.ed.ac.uk HPC and Big Data EPCC The University of Edinburgh Adrian Jackson Technical Architect a.jackson@epcc.ed.ac.uk EPCC Facilities Technology Transfer European Projects HPC Research Visitor Programmes Training

More information

A Comparison of Oracle Performance on Physical and VMware Servers

A Comparison of Oracle Performance on Physical and VMware Servers A Comparison of Oracle Performance on Physical and VMware Servers By Confio Software Confio Software 4772 Walnut Street, Suite 100 Boulder, CO 80301 303-938-8282 www.confio.com Comparison of Physical and

More information

Scala Storage Scale-Out Clustered Storage White Paper

Scala Storage Scale-Out Clustered Storage White Paper White Paper Scala Storage Scale-Out Clustered Storage White Paper Chapter 1 Introduction... 3 Capacity - Explosive Growth of Unstructured Data... 3 Performance - Cluster Computing... 3 Chapter 2 Current

More information

Moving Virtual Storage to the Cloud. Guidelines for Hosters Who Want to Enhance Their Cloud Offerings with Cloud Storage

Moving Virtual Storage to the Cloud. Guidelines for Hosters Who Want to Enhance Their Cloud Offerings with Cloud Storage Moving Virtual Storage to the Cloud Guidelines for Hosters Who Want to Enhance Their Cloud Offerings with Cloud Storage Table of Contents Overview... 1 Understanding the Storage Problem... 1 What Makes

More information

EFETnet Software System Requirements

EFETnet Software System Requirements EFETnet Software System Requirements Version 6 Dated 2015-01-01 Ponton GmbH Page 1 of 7 Change Log Revision Date Mark Changes 1 2004-12-15 TZ MH Original Schedule to EFETnet Maintenance & Support Contract

More information

CONFIGURATION GUIDELINES: EMC STORAGE FOR PHYSICAL SECURITY

CONFIGURATION GUIDELINES: EMC STORAGE FOR PHYSICAL SECURITY White Paper CONFIGURATION GUIDELINES: EMC STORAGE FOR PHYSICAL SECURITY DVTel Latitude NVMS performance using EMC Isilon storage arrays Correct sizing for storage in a DVTel Latitude physical security

More information

Can High-Performance Interconnects Benefit Memcached and Hadoop?

Can High-Performance Interconnects Benefit Memcached and Hadoop? Can High-Performance Interconnects Benefit Memcached and Hadoop? D. K. Panda and Sayantan Sur Network-Based Computing Laboratory Department of Computer Science and Engineering The Ohio State University,

More information

Study of Load Balancing of Resource Namespace Service

Study of Load Balancing of Resource Namespace Service Study of Load Balancing of Resource Namespace Service Masahiro Nakamura, Osamu Tatebe University of Tsukuba Background Resource Namespace Service (RNS) is published as GDF.101 by OGF RNS is intended to

More information

Performance and scalability of a large OLTP workload

Performance and scalability of a large OLTP workload Performance and scalability of a large OLTP workload ii Performance and scalability of a large OLTP workload Contents Performance and scalability of a large OLTP workload with DB2 9 for System z on Linux..............

More information

QoS & Traffic Management

QoS & Traffic Management QoS & Traffic Management Advanced Features for Managing Application Performance and Achieving End-to-End Quality of Service in Data Center and Cloud Computing Environments using Chelsio T4 Adapters Chelsio

More information

Objectives. Distributed Databases and Client/Server Architecture. Distributed Database. Data Fragmentation

Objectives. Distributed Databases and Client/Server Architecture. Distributed Database. Data Fragmentation Objectives Distributed Databases and Client/Server Architecture IT354 @ Peter Lo 2005 1 Understand the advantages and disadvantages of distributed databases Know the design issues involved in distributed

More information