PEXTA: A PGAS I/O for Hyperspectral Data Storage
|
|
|
- Claribel McKinney
- 9 years ago
- Views:
Transcription
1 PEXTA: A PGAS I/O for Hyperspectral Data Storage G. Nimako, E.J Otoo and Michael Sears School of Computer Science The University of the Witwatersrand Johannesburg, South Africa CHPC Annual Meeting, December 2013.
2 Introduction The conceptual model of a dataset as a multidimensional array has been around since the early days of relational database model. Records of multi-attribute files, perceived as points in multi-dimensional space, form the bases of many of the efficient data structures and algorithms. Examples: KD-Tree, Quad-Tree, Grid-File, multi-dimensional OLAP and data warehousing. Scientific data-sets are maintained predominantly as multi-dimensional arrays - HDF5, NetCDF, New SciDB, etc. Otoo,Nimako&Sears () Using Chunked Ext... 2 / 30
3 Introduction The conceptual model of a dataset as a multidimensional array has been around since the early days of relational database model. Records of multi-attribute files, perceived as points in multi-dimensional space, form the bases of many of the efficient data structures and algorithms. Examples: KD-Tree, Quad-Tree, Grid-File, multi-dimensional OLAP and data warehousing. Scientific data-sets are maintained predominantly as multi-dimensional arrays - HDF5, NetCDF, New SciDB, etc. Otoo,Nimako&Sears () Using Chunked Ext... 2 / 30
4 Motivation Processing massively large scientific data (Big Data), in the realm of Peta-scale and Exa-scale computing using Partitioned Global Address Space (PGAS) computational model. This also subsumes the hadoop-like computing model. The PGAS computational model are exemplified by: Global Array Toolkit, Unified Parallel C (UPC), OpenShmem Application to hyper-spectral image data storage and analysis The database consistent with this model of computation is SciDB It is an Array Data Model on shared parallel architectures with unbounded non-uniform dimensions with holes, arbitrary nesting, sophisticated storage representations, including overlapping chunks, etc. Otoo,Nimako&Sears () Using Chunked Ext... 3 / 30
5 Motivation: The Global Array Toolkit Distributed dense arrays that can be accessed through a shared memory-like style Single, shared data structure/global indexing Otoo,Nimako&Sears () Using Chunked Ext... 4 / 30
6 Motivation: The Global Array Toolkit Distributed dense arrays that can be accessed through a shared memory-like style Single, shared data structure/global indexing Otoo,Nimako&Sears () Using Chunked Ext... 4 / 30
7 Multidimensional Array Files Mapping of conventional array elements onto linear storage, using row-major or column-major order allows for extendability of the array in only one dimension. Otoo,Nimako&Sears () Using Chunked Ext... 5 / 30
8 Multidimensional Array Files Mapping of conventional array elements onto linear storage, using row-major or column-major order allows for extendability of the array in only one dimension. The question then arises: How can we store arrays elements so that the index range of any dimension can be extended? Otoo,Nimako&Sears () Using Chunked Ext... 5 / 30
9 Multidimensional Array Files Mapping of conventional array elements onto linear storage, using row-major or column-major order allows for extendability of the array in only one dimension. The question then arises: How can we store arrays elements so that the index range of any dimension can be extended? The problem has been addressed in some earlier papers. Notably by : G. Nimako, E. Otoo and D. Kwofie, PARCO Journal [2013] G. Nimako, E. Otoo and D. Kwofie, IASDS of IEEE Cluster [2013] G. Nimako, E. Otoo and D. Kwofie, P2S2 [2012] Tsuji, M. Kuroda and K. Higuchi [2008] A. Rosenberg and L. Stockmeyer [1977] Otoo,Nimako&Sears () Using Chunked Ext... 5 / 30
10 Main Contributions We present a computable access function F ( i 0, i 1,..., i k 1 ) that maps the k-dimensional index i 0, i 1,..., i k 1 to a value that gives the linear address location l j in time O(k + log N) using O(k 2 N 1/k ) addition space. This allows us to store elements of extendible array directly as array-files. Otoo,Nimako&Sears () Using Chunked Ext... 6 / 30
11 Main Contributions We present a computable access function F ( i 0, i 1,..., i k 1 ) that maps the k-dimensional index i 0, i 1,..., i k 1 to a value that gives the linear address location l j in time O(k + log N) using O(k 2 N 1/k ) addition space. This allows us to store elements of extendible array directly as array-files. We introduce Parallel Chunked Extendible Dense Array I/O (PEXTA) for the GA Otoo,Nimako&Sears () Using Chunked Ext... 6 / 30
12 Main Contributions We present a computable access function F ( i 0, i 1,..., i k 1 ) that maps the k-dimensional index i 0, i 1,..., i k 1 to a value that gives the linear address location l j in time O(k + log N) using O(k 2 N 1/k ) addition space. This allows us to store elements of extendible array directly as array-files. We introduce Parallel Chunked Extendible Dense Array I/O (PEXTA) for the GA PEXTA can be used as a drop-in replacement for the Disk Resident Array of the GA-Toolkit Otoo,Nimako&Sears () Using Chunked Ext... 6 / 30
13 Main Contributions We present a computable access function F ( i 0, i 1,..., i k 1 ) that maps the k-dimensional index i 0, i 1,..., i k 1 to a value that gives the linear address location l j in time O(k + log N) using O(k 2 N 1/k ) addition space. This allows us to store elements of extendible array directly as array-files. We introduce Parallel Chunked Extendible Dense Array I/O (PEXTA) for the GA PEXTA can be used as a drop-in replacement for the Disk Resident Array of the GA-Toolkit PEXTA provides an efficient I/O for Parallel Implementation of Endmember Extraction Algorithms From Hyperspectral Data. Otoo,Nimako&Sears () Using Chunked Ext... 6 / 30
14 Rest of the Talk Extendible Arrays of Data Chunked Extendible Arrays PEXTA I/O Model PEXTA Implementation for GA Using PEXTA for Parallel Implementation of Endmember Extraction Algorithms From Hyperspectral Data Summary and Future Directions Otoo,Nimako&Sears () Using Chunked Ext... 7 / 30
15 Conventional Array Using Row-Major Ordering Consider an 3-D array A[U 0 ][U 1 ][U 2 ] whose elements are allocated in row-major order. The location of an element A i 0, i 1, i 2, is given by loc(a 0, 0, 0 + q where q is computed as: q = i 0 C 0 + i 1 C 1 + i 2 C 2 and C 0 = U 1 U 2 C 1 = U 2 C 2 = 1 The values C 0, C 1, C 2 are held in the dope-vector. This reveals the strategy of our allocation scheme. Otoo,Nimako&Sears () Using Chunked Ext... 8 / 30
16 Conventional Array Using Row-Major Ordering Consider an 3-D array A[3][3][2] whose elements are allocated in row-major order as below This reveals the strategy of our allocation scheme. Otoo,Nimako&Sears () Using Chunked Ext... 9 / 30
17 Conventional Array Using Row-Major Ordering Consider an 3-D array A[3][3][2] whose elements are allocated in row-major order as below This reveals the strategy of our allocation scheme. Otoo,Nimako&Sears () Using Chunked Ext... 9 / 30
18 Conventional Array Using Row-Major Ordering Consider an 3-D array A[3][3][2] whose elements are allocated in row-major order as below This reveals the strategy of our allocation scheme. Otoo,Nimako&Sears () Using Chunked Ext... 9 / 30
19 Conventional Array Using Row-Major Ordering Consider an 3-D array A[3][3][2] whose elements are allocated in row-major order as below This reveals the strategy of our allocation scheme. Otoo,Nimako&Sears () Using Chunked Ext... 9 / 30
20 Linear Mapping for a Dense Extendible Array The mapping function for extendible array uses axial-vectors to store information that includes the dope-vector at the time of extending the dimension A list of axial-vectors is maintained for each dimension. Let A[U 0 ][U 1 ][U 2 ] be an arbitrary 3-dimensional array, where U j implies that the bound can be extended. Similarly we employ the notation: F() when referring to conventional array mapping function. F () when referring to a mapping function for an extendible array. Otoo,Nimako&Sears () Using Chunked Ext / 30
21 Linear Mapping for a Dense Extendible Array The mapping function for extendible array uses axial-vectors to store information that includes the dope-vector at the time of extending the dimension A list of axial-vectors is maintained for each dimension. Let A[U 0 ][U 1 ][U 2 ] be an arbitrary 3-dimensional array, where U j implies that the bound can be extended. Similarly we employ the notation: F() when referring to conventional array mapping function. F () when referring to a mapping function for an extendible array. Otoo,Nimako&Sears () Using Chunked Ext / 30
22 Linear Mapping for a Dense Extendible Array The mapping function for extendible array uses axial-vectors to store information that includes the dope-vector at the time of extending the dimension A list of axial-vectors is maintained for each dimension. Let A[U 0 ][U 1 ][U 2 ] be an arbitrary 3-dimensional array, where U j implies that the bound can be extended. Similarly we employ the notation: F() when referring to conventional array mapping function. F () when referring to a mapping function for an extendible array. Otoo,Nimako&Sears () Using Chunked Ext / 30
23 Illustration of the Linear Mapping for a Dense Extendible Array Otoo,Nimako&Sears () Using Chunked Ext / 30
24 Illustration of the Linear Mapping for a Dense Extendible Array Otoo,Nimako&Sears () Using Chunked Ext / 30
25 Illustration of the Linear Mapping for a Dense Extendible Array Otoo,Nimako&Sears () Using Chunked Ext / 30
26 Illustration of the Linear Mapping for a Dense Extendible Array Otoo,Nimako&Sears () Using Chunked Ext / 30
27 Illustration of the Linear Mapping for a Dense Extendible Array Otoo,Nimako&Sears () Using Chunked Ext / 30
28 Illustration of the Linear Mapping for a Dense Extendible Array Otoo,Nimako&Sears () Using Chunked Ext / 30
29 Illustration of the Linear Mapping for a Dense Extendible Array Otoo,Nimako&Sears () Using Chunked Ext / 30
30 Illustration of the Linear Mapping for a Dense Extendible Array Otoo,Nimako&Sears () Using Chunked Ext / 30
31 Illustration of the Linear Mapping for a Dense Extendible Array Otoo,Nimako&Sears () Using Chunked Ext / 30
32 Illustration of the Linear Mapping for a Dense Extendible Array Otoo,Nimako&Sears () Using Chunked Ext / 30
33 Illustration of the Linear Mapping for a Dense Extendible Array Otoo,Nimako&Sears () Using Chunked Ext / 30
34 Illustration of the Linear Mapping for a Dense Extendible Array Otoo,Nimako&Sears () Using Chunked Ext / 30
35 Illustration of the Linear Mapping for a Dense Extendible Array Otoo,Nimako&Sears () Using Chunked Ext / 30
36 Linear Mapping for a Dense Extendible Array Suppose that in a k-dimensional extendible array A[U 0][U 1][U 2]...[U k 1 ], dimension l is extended by λ l, then the index range increases from U l to U l + λ l. Let the location A 0, 0,..., U l,..., 0 (i.e. the starting location of an allocated hyperslab ) be denoted as l Z 0 where Z 0 U U = k 1 l r=0 U r. l The Mapping Function k 1 q = F ( i 0, i 1, i 2,..., i k 1 )) = Z 0 U + (i l U l )C l + i j Cj l j=0 j l k 1 Cl = U j j=0 j l C j = k 1 r=j+1 r l U r Otoo,Nimako&Sears () Using Chunked Ext / 30
37 Linear Mapping for a Dense Extendible Array Suppose that in a k-dimensional extendible array A[U 0][U 1][U 2]...[U k 1 ], dimension l is extended by λ l, then the index range increases from U l to U l + λ l. Let the location A 0, 0,..., U l,..., 0 (i.e. the starting location of an allocated hyperslab ) be denoted as l Z 0 where Z 0 U U = k 1 l r=0 U r. l The Mapping Function k 1 q = F ( i 0, i 1, i 2,..., i k 1 )) = Z 0 U + (i l U l )C l + i j Cj l j=0 j l k 1 Cl = U j j=0 j l C j = k 1 r=j+1 r l U r Otoo,Nimako&Sears () Using Chunked Ext / 30
38 Chunking Extendible Dense Arrays The use of the list for axial-vectors can be expensive and depends in particular on the interruptible expansions. (worst case is when we have cubical extensions). Such interruptible expansions cause the addition of a new entry in the list of axial vectors. Chunking the array gives some additional advantages: It gives contiguous storage allocations for the elements of the chunks. When arrays are allocated onto secondary storage, I/O can be made in multiples of the chunk size. The allocation is done in chunks as opposed to the single elements. Otoo,Nimako&Sears () Using Chunked Ext / 30
39 Chunking Extendible Dense Arrays The use of the list for axial-vectors can be expensive and depends in particular on the interruptible expansions. (worst case is when we have cubical extensions). Such interruptible expansions cause the addition of a new entry in the list of axial vectors. Chunking the array gives some additional advantages: It gives contiguous storage allocations for the elements of the chunks. When arrays are allocated onto secondary storage, I/O can be made in multiples of the chunk size. The allocation is done in chunks as opposed to the single elements. Otoo,Nimako&Sears () Using Chunked Ext / 30
40 Illustration of Chunked Extendible Dense Arrays Otoo,Nimako&Sears () Using Chunked Ext / 30
41 PEXTA Overview and Implementation We model the PEXTA after MPI-I/O Definitions The filetype describes the data layout in file The buftype describes how data is laid out in a process s buffer The elementary datatype, etype, ensures consistency between the type signature of both filetype and buftype. The principal-array data and the axial-vectors, that correspond to the leaf-nodes of the O 2 -Tree, are stored on disk as the principal-array file, F p and metadata file, F m, respectively. Although F p will be stored on disk, it is accessed through the buffers of each process in a communicator group. Otoo,Nimako&Sears () Using Chunked Ext / 30
42 PEXTA Overview and Implementation We model the PEXTA after MPI-I/O Definitions The filetype describes the data layout in file The buftype describes how data is laid out in a process s buffer The elementary datatype, etype, ensures consistency between the type signature of both filetype and buftype. The principal-array data and the axial-vectors, that correspond to the leaf-nodes of the O 2 -Tree, are stored on disk as the principal-array file, F p and metadata file, F m, respectively. Although F p will be stored on disk, it is accessed through the buffers of each process in a communicator group. Otoo,Nimako&Sears () Using Chunked Ext / 30
43 PEXTA Overview and Implementation PEXTA Model supports both regular and irregular distributions of global array of the GA-Toolkit. We have completed implementation for regular distribution. Processes can perform independent or collective reads and writes of their respective sub-arrays. An external/independent process can request for an exclusive write lock token on the metadata file, F m and append chunks of data to F p Otoo,Nimako&Sears () Using Chunked Ext / 30
44 PEXTA Overview and Implementation PEXTA Model supports both regular and irregular distributions of global array of the GA-Toolkit. We have completed implementation for regular distribution. Processes can perform independent or collective reads and writes of their respective sub-arrays. An external/independent process can request for an exclusive write lock token on the metadata file, F m and append chunks of data to F p Otoo,Nimako&Sears () Using Chunked Ext / 30
45 PEXTA Overview and Implementation When a new session starts, the processes within a communicator group reads the newly updated metadata file into their individual O 2 -Trees. Currently, we do not support in-memory extensions. Otoo,Nimako&Sears () Using Chunked Ext / 30
46 PEXTA Overview and Implementation When a new session starts, the processes within a communicator group reads the newly updated metadata file into their individual O 2 -Trees. Currently, we do not support in-memory extensions. Otoo,Nimako&Sears () Using Chunked Ext / 30
47 PEXTA Overview and Implementation The PEXTA API provides routines quite similar to DRA for creating disk resident array We however provide additional functions for extendibility. Functions int PEXTA Init(int maxarrays, double maxarraysize, double totaldiskspace, double maxmemory); int PEXTA Create(int type, int ndim, size t dims[],char *name, char* filename, int mode, int mode, size t reqdims[], int g a, int *p a); int PEXTA IRead(int g a, int p a, int *request); int PEXTA Extend(int p a, size t edims[],size t odims[], int *request); etc. Otoo,Nimako&Sears () Using Chunked Ext / 30
48 Hyperspectral Data Otoo,Nimako&Sears () Using Chunked Ext / 30
49 HPC in Remote Sensing Applications Recently, high-performance computing systems have become more widespread in remote sensing applications. Parallel techniques for endmember extraction may allow rapid inspection of massive data archives It will also enable generation of comprehensive spectral libraries and other useful information An opportunity to explore methodologies in areas such as multisource data registration and mining that previously looked too computationally intensive for practical applications due to the immense files involved Otoo,Nimako&Sears () Using Chunked Ext / 30
50 Hyperspectral Data Analysis Mixed pixels (due to insufficient spatial resolution and mixing effects in surfaces) Sub-pixel targets (very important and crucial in many hyperspectral applications) Otoo,Nimako&Sears () Using Chunked Ext / 30
51 Spectral Unmixing of Hyperspectral Data The goal is to find extreme pixel vectors (endmembers) that can be used to unmix other mixed pixels in the data using a linear mixture model. Each mixed pixel can be obtained as a combination of endmember fractional abundances. A crucial issue is how to find spectral endmembers. Endmember Extraction Algorithms: Boardman s Pixel Purity Index (PPI) Algorithm Harsanyi and Chang s Orthogonal Subspace Projection (OSP) Winter s N-FINDR Algorithm Otoo,Nimako&Sears () Using Chunked Ext / 30
52 Parallel Endmember Extraction Using PEXTA and GA The master processor scatters the data among the distributed workers We provide collective operations in PEXTA for data partitioning. Alternatively, the GA collective operation for data partitioning can be used. Otoo,Nimako&Sears () Using Chunked Ext / 30
53 Parallel Endmember Extraction Using PEXTA and GA Spectral-Domain Partitioning A single pixel vector (spectral signature) may be stored in different processing units It requires communications for individual pixel-based calculations such as spectral angle computations. Otoo,Nimako&Sears () Using Chunked Ext / 30
54 Parallel Endmember Extraction Using PEXTA and GA Spatial-Domain Partitioning Every pixel vector (spectral signature) is stored in the same processing unit. This is beneficial for a proposed sliding-window algorithm in terms of low-level image processing computations. Otoo,Nimako&Sears () Using Chunked Ext / 30
55 Summary and Future Work Poor parallel I/O performance has been a bottleneck in many data-intensive applications The organisation of extendible arrays using a mapping function is highly appropriate for most scientific datasets where the model of the data is perceived as a large multi-dimensional array. Main application areas is for hyper-spectral image processing and a host other scientific applications that are based on multidimensional data models. Remote sensing missions require efficient I/O able to cope with the extremely high dimensionality of the collected data without compromising mission payload. Future work on the use of in memory resident caches of both the chunks and the array are anticipated. Otoo,Nimako&Sears () Using Chunked Ext / 30
56 Thanks! Thank You Questions? Otoo,Nimako&Sears () Using Chunked Ext / 30
Data Centric Systems (DCS)
Data Centric Systems (DCS) Architecture and Solutions for High Performance Computing, Big Data and High Performance Analytics High Performance Computing with Data Centric Systems 1 Data Centric Systems
Data Deduplication in a Hybrid Architecture for Improving Write Performance
Data Deduplication in a Hybrid Architecture for Improving Write Performance Data-intensive Salable Computing Laboratory Department of Computer Science Texas Tech University Lubbock, Texas June 10th, 2013
Buffer Management 5. Buffer Management
5 Buffer Management Copyright 2004, Binnur Kurt A journey of a byte Buffer Management Content 156 A journey of a byte Suppose in our program we wrote: outfile
A Quantitative and Comparative Analysis of Endmember Extraction Algorithms From Hyperspectral Data
650 IEEE TRANSACTIONS ON GEOSCIENCE AND REMOTE SENSING, VOL. 42, NO. 3, MARCH 2004 A Quantitative and Comparative Analysis of Endmember Extraction Algorithms From Hyperspectral Data Antonio Plaza, Pablo
Introduction to Parallel Programming and MapReduce
Introduction to Parallel Programming and MapReduce Audience and Pre-Requisites This tutorial covers the basics of parallel programming and the MapReduce programming model. The pre-requisites are significant
Two Parts. Filesystem Interface. Filesystem design. Interface the user sees. Implementing the interface
File Management Two Parts Filesystem Interface Interface the user sees Organization of the files as seen by the user Operations defined on files Properties that can be read/modified Filesystem design Implementing
Binary search tree with SIMD bandwidth optimization using SSE
Binary search tree with SIMD bandwidth optimization using SSE Bowen Zhang, Xinwei Li 1.ABSTRACT In-memory tree structured index search is a fundamental database operation. Modern processors provide tremendous
Big Data and Cloud Computing for GHRSST
Big Data and Cloud Computing for GHRSST Jean-Francois Piollé ([email protected]) Frédéric Paul, Olivier Archer CERSAT / Institut Français de Recherche pour l Exploitation de la Mer Facing data deluge
RevoScaleR Speed and Scalability
EXECUTIVE WHITE PAPER RevoScaleR Speed and Scalability By Lee Edlefsen Ph.D., Chief Scientist, Revolution Analytics Abstract RevoScaleR, the Big Data predictive analytics library included with Revolution
ORACLE OLAP. Oracle OLAP is embedded in the Oracle Database kernel and runs in the same database process
ORACLE OLAP KEY FEATURES AND BENEFITS FAST ANSWERS TO TOUGH QUESTIONS EASILY KEY FEATURES & BENEFITS World class analytic engine Superior query performance Simple SQL access to advanced analytics Enhanced
HPC enabling of OpenFOAM R for CFD applications
HPC enabling of OpenFOAM R for CFD applications Towards the exascale: OpenFOAM perspective Ivan Spisso 25-27 March 2015, Casalecchio di Reno, BOLOGNA. SuperComputing Applications and Innovation Department,
In-Memory Columnar Databases HyPer. Arto Kärki University of Helsinki 30.11.2012
In-Memory Columnar Databases HyPer Arto Kärki University of Helsinki 30.11.2012 1 Introduction Columnar Databases Design Choices Data Clustering and Compression Conclusion 2 Introduction The relational
Introduction to Data Mining
Introduction to Data Mining 1 Why Data Mining? Explosive Growth of Data Data collection and data availability Automated data collection tools, Internet, smartphones, Major sources of abundant data Business:
Data Warehousing und Data Mining
Data Warehousing und Data Mining Multidimensionale Indexstrukturen Ulf Leser Wissensmanagement in der Bioinformatik Content of this Lecture Multidimensional Indexing Grid-Files Kd-trees Ulf Leser: Data
Data warehousing with PostgreSQL
Data warehousing with PostgreSQL Gabriele Bartolini http://www.2ndquadrant.it/ European PostgreSQL Day 2009 6 November, ParisTech Telecom, Paris, France Audience
PARALLELS CLOUD STORAGE
PARALLELS CLOUD STORAGE Performance Benchmark Results 1 Table of Contents Executive Summary... Error! Bookmark not defined. Architecture Overview... 3 Key Features... 5 No Special Hardware Requirements...
Unsupervised Data Mining (Clustering)
Unsupervised Data Mining (Clustering) Javier Béjar KEMLG December 01 Javier Béjar (KEMLG) Unsupervised Data Mining (Clustering) December 01 1 / 51 Introduction Clustering in KDD One of the main tasks in
Indexing Techniques in Data Warehousing Environment The UB-Tree Algorithm
Indexing Techniques in Data Warehousing Environment The UB-Tree Algorithm Prepared by: Yacine ghanjaoui Supervised by: Dr. Hachim Haddouti March 24, 2003 Abstract The indexing techniques in multidimensional
Feedback guided load balancing in a distributed memory environment
Feedback guided load balancing in a distributed memory environment Constantinos Christofi August 18, 2011 MSc in High Performance Computing The University of Edinburgh Year of Presentation: 2011 Abstract
Hypertable Architecture Overview
WHITE PAPER - MARCH 2012 Hypertable Architecture Overview Hypertable is an open source, scalable NoSQL database modeled after Bigtable, Google s proprietary scalable database. It is written in C++ for
EXPLOITING SHARED MEMORY TO IMPROVE PARALLEL I/O PERFORMANCE
EXPLOITING SHARED MEMORY TO IMPROVE PARALLEL I/O PERFORMANCE Andrew B. Hastings Sun Microsystems, Inc. Alok Choudhary Northwestern University September 19, 2006 This material is based on work supported
COSC 6374 Parallel Computation. Parallel I/O (I) I/O basics. Concept of a clusters
COSC 6374 Parallel I/O (I) I/O basics Fall 2012 Concept of a clusters Processor 1 local disks Compute node message passing network administrative network Memory Processor 2 Network card 1 Network card
Chapter 11 I/O Management and Disk Scheduling
Operating Systems: Internals and Design Principles, 6/E William Stallings Chapter 11 I/O Management and Disk Scheduling Dave Bremer Otago Polytechnic, NZ 2008, Prentice Hall I/O Devices Roadmap Organization
CHAPTER - 5 CONCLUSIONS / IMP. FINDINGS
CHAPTER - 5 CONCLUSIONS / IMP. FINDINGS In today's scenario data warehouse plays a crucial role in order to perform important operations. Different indexing techniques has been used and analyzed using
Scala Storage Scale-Out Clustered Storage White Paper
White Paper Scala Storage Scale-Out Clustered Storage White Paper Chapter 1 Introduction... 3 Capacity - Explosive Growth of Unstructured Data... 3 Performance - Cluster Computing... 3 Chapter 2 Current
SAP HANA - Main Memory Technology: A Challenge for Development of Business Applications. Jürgen Primsch, SAP AG July 2011
SAP HANA - Main Memory Technology: A Challenge for Development of Business Applications Jürgen Primsch, SAP AG July 2011 Why In-Memory? Information at the Speed of Thought Imagine access to business data,
DAME Astrophysical DAta Mining Mining & & Exploration Exploration GRID
DAME Astrophysical DAta Mining & Exploration on GRID M. Brescia S. G. Djorgovski G. Longo & DAME Working Group Istituto Nazionale di Astrofisica Astronomical Observatory of Capodimonte, Napoli Department
How To Virtualize A Storage Area Network (San) With Virtualization
A New Method of SAN Storage Virtualization Table of Contents 1 - ABSTRACT 2 - THE NEED FOR STORAGE VIRTUALIZATION 3 - EXISTING STORAGE VIRTUALIZATION METHODS 4 - A NEW METHOD OF VIRTUALIZATION: Storage
CS 377: Operating Systems. Outline. A review of what you ve learned, and how it applies to a real operating system. Lecture 25 - Linux Case Study
CS 377: Operating Systems Lecture 25 - Linux Case Study Guest Lecturer: Tim Wood Outline Linux History Design Principles System Overview Process Scheduling Memory Management File Systems A review of what
Orthogonal Projections
Orthogonal Projections and Reflections (with exercises) by D. Klain Version.. Corrections and comments are welcome! Orthogonal Projections Let X,..., X k be a family of linearly independent (column) vectors
Massive Labeled Solar Image Data Benchmarks for Automated Feature Recognition
Massive Labeled Solar Image Data Benchmarks for Automated Feature Recognition Michael A. Schuh1, Rafal A. Angryk2 1 Montana State University, Bozeman, MT 2 Georgia State University, Atlanta, GA Introduction
M2074 - Designing and Implementing OLAP Solutions Using Microsoft SQL Server 2000 5 Day Course
Module 1: Introduction to Data Warehousing and OLAP Introducing Data Warehousing Defining OLAP Solutions Understanding Data Warehouse Design Understanding OLAP Models Applying OLAP Cubes At the end of
CHAPTER 17: File Management
CHAPTER 17: File Management The Architecture of Computer Hardware, Systems Software & Networking: An Information Technology Approach 4th Edition, Irv Englander John Wiley and Sons 2010 PowerPoint slides
Andreas Rauber and Philipp Tomsich Institute of Software Technology Vienna University of Technology, Austria {andi,phil}@ifs.tuwien.ac.
An Architecture for Modular On-Line Analytical Processing Systems: Supporting Distributed and Parallel Query Processing Using Co-operating CORBA Objects Andreas Rauber and Philipp Tomsich Institute of
Cloud Computing at Google. Architecture
Cloud Computing at Google Google File System Web Systems and Algorithms Google Chris Brooks Department of Computer Science University of San Francisco Google has developed a layered system to handle webscale
Similarity-Based Unsupervised Band Selection for Hyperspectral Image Analysis
564 IEEE GEOSCIENCE AND REMOTE SENSING LETTERS, VOL. 5, NO. 4, OCTOBER 2008 Similarity-Based Unsupervised Band Selection for Hyperspectral Image Analysis Qian Du, Senior Member, IEEE, and He Yang, Student
Review. Data Warehousing. Today. Star schema. Star join indexes. Dimension hierarchies
Review Data Warehousing CPS 216 Advanced Database Systems Data warehousing: integrating data for OLAP OLAP versus OLTP Warehousing versus mediation Warehouse maintenance Warehouse data as materialized
CUBE INDEXING IMPLEMENTATION USING INTEGRATION OF SIDERA AND BERKELEY DB
CUBE INDEXING IMPLEMENTATION USING INTEGRATION OF SIDERA AND BERKELEY DB Badal K. Kothari 1, Prof. Ashok R. Patel 2 1 Research Scholar, Mewar University, Chittorgadh, Rajasthan, India 2 Department of Computer
Topics in basic DBMS course
Topics in basic DBMS course Database design Transaction processing Relational query languages (SQL), calculus, and algebra DBMS APIs Database tuning (physical database design) Basic query processing (ch
Big Graph Processing: Some Background
Big Graph Processing: Some Background Bo Wu Colorado School of Mines Part of slides from: Paul Burkhardt (National Security Agency) and Carlos Guestrin (Washington University) Mines CSCI-580, Bo Wu Graphs
Chapter 11: File System Implementation. Operating System Concepts with Java 8 th Edition
Chapter 11: File System Implementation 11.1 Silberschatz, Galvin and Gagne 2009 Chapter 11: File System Implementation File-System Structure File-System Implementation Directory Implementation Allocation
DSS based on Data Warehouse
DSS based on Data Warehouse C_13 / 6.01.2015 Decision support system is a complex system engineering. At the same time, research DW composition, DW structure and DSS Architecture based on DW, puts forward
High Performance Computing. Course Notes 2007-2008. HPC Fundamentals
High Performance Computing Course Notes 2007-2008 2008 HPC Fundamentals Introduction What is High Performance Computing (HPC)? Difficult to define - it s a moving target. Later 1980s, a supercomputer performs
Comp 5311 Database Management Systems. 16. Review 2 (Physical Level)
Comp 5311 Database Management Systems 16. Review 2 (Physical Level) 1 Main Topics Indexing Join Algorithms Query Processing and Optimization Transactions and Concurrency Control 2 Indexing Used for faster
In-situ Visualization
In-situ Visualization Dr. Jean M. Favre Scientific Computing Research Group 13-01-2011 Outline Motivations How is parallel visualization done today Visualization pipelines Execution paradigms Many grids
Facts about Visualization Pipelines, applicable to VisIt and ParaView
Facts about Visualization Pipelines, applicable to VisIt and ParaView March 2013 Jean M. Favre, CSCS Agenda Visualization pipelines Motivation by examples VTK Data Streaming Visualization Pipelines: Introduction
LLNL s Parallel I/O Testing Tools and Techniques for ASC Parallel File Systems
UCRL-CONF-203489 LAWRENCE LIVERMORE NATIONAL LABORATORY LLNL s Parallel I/O Testing Tools and Techniques for ASC Parallel File Systems W. E. Loewe, R. M. Hedges, T. T. McLarty, and C. J. Morrone April
Big Table A Distributed Storage System For Data
Big Table A Distributed Storage System For Data OSDI 2006 Fay Chang, Jeffrey Dean, Sanjay Ghemawat et.al. Presented by Rahul Malviya Why BigTable? Lots of (semi-)structured data at Google - - URLs: Contents,
Graph Mining on Big Data System. Presented by Hefu Chai, Rui Zhang, Jian Fang
Graph Mining on Big Data System Presented by Hefu Chai, Rui Zhang, Jian Fang Outline * Overview * Approaches & Environment * Results * Observations * Notes * Conclusion Overview * What we have done? *
The Scientific Data Mining Process
Chapter 4 The Scientific Data Mining Process When I use a word, Humpty Dumpty said, in rather a scornful tone, it means just what I choose it to mean neither more nor less. Lewis Carroll [87, p. 214] In
A Novel Cloud Based Elastic Framework for Big Data Preprocessing
School of Systems Engineering A Novel Cloud Based Elastic Framework for Big Data Preprocessing Omer Dawelbeit and Rachel McCrindle October 21, 2014 University of Reading 2008 www.reading.ac.uk Overview
CSE 544 Principles of Database Management Systems. Magdalena Balazinska Fall 2007 Lecture 5 - DBMS Architecture
CSE 544 Principles of Database Management Systems Magdalena Balazinska Fall 2007 Lecture 5 - DBMS Architecture References Anatomy of a database system. J. Hellerstein and M. Stonebraker. In Red Book (4th
BIG DATA CAN DRIVE THE BUSINESS AND IT TO EVOLVE AND ADAPT RALPH KIMBALL BUSSUM 2014
BIG DATA CAN DRIVE THE BUSINESS AND IT TO EVOLVE AND ADAPT RALPH KIMBALL BUSSUM 2014 Ralph Kimball Associates 2014 The Data Warehouse Mission Identify all possible enterprise data assets Select those assets
REGULATIONS FOR THE DEGREE OF MASTER OF SCIENCE IN COMPUTER SCIENCE (MSc[CompSc])
![REGULATIONS FOR THE DEGREE OF MASTER OF SCIENCE IN COMPUTER SCIENCE (MSc[CompSc])](/thumbs/24/2595933.jpg "REGULATIONS FOR THE DEGREE OF MASTER OF SCIENCE IN COMPUTER SCIENCE (MSc[CompSc])")
244 REGULATIONS FOR THE DEGREE OF MASTER OF SCIENCE IN COMPUTER SCIENCE (MSc[CompSc]) (See also General Regulations) Any publication based on work approved for a higher degree should contain a reference
Big Data and Big Analytics
Big Data and Big Analytics Introducing SciDB Open source, massively parallel DBMS and analytic platform Array data model (rather than SQL, Unstructured, XML, or triple-store) Extensible micro-kernel architecture
Topological Properties
Advanced Computer Architecture Topological Properties Routing Distance: Number of links on route Node degree: Number of channels per node Network diameter: Longest minimum routing distance between any
Chapter 7 Memory Management
Operating Systems: Internals and Design Principles Chapter 7 Memory Management Eighth Edition William Stallings Frame Page Segment A fixed-length block of main memory. A fixed-length block of data that
Interconnection Networks. Interconnection Networks. Interconnection networks are used everywhere!
Interconnection Networks Interconnection Networks Interconnection networks are used everywhere! Supercomputers connecting the processors Routers connecting the ports can consider a router as a parallel
RCFile: A Fast and Space-efficient Data Placement Structure in MapReduce-based Warehouse Systems CLOUD COMPUTING GROUP - LITAO DENG
1 RCFile: A Fast and Space-efficient Data Placement Structure in MapReduce-based Warehouse Systems CLOUD COMPUTING GROUP - LITAO DENG Background 2 Hive is a data warehouse system for Hadoop that facilitates
α = u v. In other words, Orthogonal Projection
Orthogonal Projection Given any nonzero vector v, it is possible to decompose an arbitrary vector u into a component that points in the direction of v and one that points in a direction orthogonal to v
How To Use Data Mining For Knowledge Management In Technology Enhanced Learning
Proceedings of the 6th WSEAS International Conference on Applications of Electrical Engineering, Istanbul, Turkey, May 27-29, 2007 115 Data Mining for Knowledge Management in Technology Enhanced Learning
Testing the In-Memory Column Store for in-database physics analysis. Dr. Maaike Limper
Testing the In-Memory Column Store for in-database physics analysis Dr. Maaike Limper About CERN CERN - European Laboratory for Particle Physics Support the research activities of 10 000 scientists from
Operating Systems CSE 410, Spring 2004. File Management. Stephen Wagner Michigan State University
Operating Systems CSE 410, Spring 2004 File Management Stephen Wagner Michigan State University File Management File management system has traditionally been considered part of the operating system. Applications
FROM RELATIONAL TO OBJECT DATABASE MANAGEMENT SYSTEMS
FROM RELATIONAL TO OBJECT DATABASE MANAGEMENT SYSTEMS V. CHRISTOPHIDES Department of Computer Science & Engineering University of California, San Diego ICS - FORTH, Heraklion, Crete 1 I) INTRODUCTION 2
SCALABLE DATA SERVICES
1 SCALABLE DATA SERVICES 2110414 Large Scale Computing Systems Natawut Nupairoj, Ph.D. Outline 2 Overview MySQL Database Clustering GlusterFS Memcached 3 Overview Problems of Data Services 4 Data retrieval
L20: GPU Architecture and Models
L20: GPU Architecture and Models scribe(s): Abdul Khalifa 20.1 Overview GPUs (Graphics Processing Units) are large parallel structure of processing cores capable of rendering graphics efficiently on displays.
The Cubetree Storage Organization
The Cubetree Storage Organization Nick Roussopoulos & Yannis Kotidis Advanced Communication Technology, Inc. Silver Spring, MD 20905 Tel: 301-384-3759 Fax: 301-384-3679 {nick,kotidis}@act-us.com 1. Introduction
Oracle EXAM - 1Z0-117. Oracle Database 11g Release 2: SQL Tuning. Buy Full Product. http://www.examskey.com/1z0-117.html
Oracle EXAM - 1Z0-117 Oracle Database 11g Release 2: SQL Tuning Buy Full Product http://www.examskey.com/1z0-117.html Examskey Oracle 1Z0-117 exam demo product is here for you to test the quality of the
IDL. Get the answers you need from your data. IDL
Get the answers you need from your data. IDL is the preferred computing environment for understanding complex data through interactive visualization and analysis. IDL Powerful visualization. Interactive
Parallel I/O on JUQUEEN
Parallel I/O on JUQUEEN 3. February 2015 3rd JUQUEEN Porting and Tuning Workshop Sebastian Lührs, Kay Thust [email protected], [email protected] Jülich Supercomputing Centre Overview Blue Gene/Q
Chapter 7 Memory Management
Operating Systems: Internals and Design Principles, 6/E William Stallings Chapter 7 Memory Management Patricia Roy Manatee Community College, Venice, FL 2008, Prentice Hall Memory Management Subdividing
Business Intelligence. Data Mining and Optimization for Decision Making
Brochure More information from http://www.researchandmarkets.com/reports/2325743/ Business Intelligence. Data Mining and Optimization for Decision Making Description: Business intelligence is a broad category
Section 1.1. Introduction to R n
The Calculus of Functions of Several Variables Section. Introduction to R n Calculus is the study of functional relationships and how related quantities change with each other. In your first exposure to
Chapter 11: File System Implementation. Chapter 11: File System Implementation. Objectives. File-System Structure
Chapter 11: File System Implementation Chapter 11: File System Implementation File-System Structure File-System Implementation Directory Implementation Allocation Methods Free-Space Management Efficiency
1Z0-117 Oracle Database 11g Release 2: SQL Tuning. Oracle
1Z0-117 Oracle Database 11g Release 2: SQL Tuning Oracle To purchase Full version of Practice exam click below; http://www.certshome.com/1z0-117-practice-test.html FOR Oracle 1Z0-117 Exam Candidates We
Big Data Paradigms in Python
Big Data Paradigms in Python San Diego Data Science and R Users Group January 2014 Kevin Davenport! http://kldavenport.com [email protected] @KevinLDavenport Thank you to our sponsors: Setting up
Alejandro Vaisman Esteban Zimanyi. Data. Warehouse. Systems. Design and Implementation. ^ Springer
Alejandro Vaisman Esteban Zimanyi Data Warehouse Systems Design and Implementation ^ Springer Contents Part I Fundamental Concepts 1 Introduction 3 1.1 A Historical Overview of Data Warehousing 4 1.2 Spatial
Visualization of large data sets using MDS combined with LVQ.
Visualization of large data sets using MDS combined with LVQ. Antoine Naud and Włodzisław Duch Department of Informatics, Nicholas Copernicus University, Grudziądzka 5, 87-100 Toruń, Poland. www.phys.uni.torun.pl/kmk
Multimedia Databases. Wolf-Tilo Balke Philipp Wille Institut für Informationssysteme Technische Universität Braunschweig http://www.ifis.cs.tu-bs.
Multimedia Databases Wolf-Tilo Balke Philipp Wille Institut für Informationssysteme Technische Universität Braunschweig http://www.ifis.cs.tu-bs.de 14 Previous Lecture 13 Indexes for Multimedia Data 13.1
MultiMedia and Imaging Databases
MultiMedia and Imaging Databases Setrag Khoshafian A. Brad Baker Technische H FACHBEREIGM W-C^KA VK B_l_3JLJ0 T H E K Inventar-N*.: Sachgebiete: Standort: Morgan Kaufmann Publishers, Inc. San Francisco,
y = Xβ + ε B. Sub-pixel Classification
Sub-pixel Mapping of Sahelian Wetlands using Multi-temporal SPOT VEGETATION Images Jan Verhoeye and Robert De Wulf Laboratory of Forest Management and Spatial Information Techniques Faculty of Agricultural
Overlapping Data Transfer With Application Execution on Clusters
Overlapping Data Transfer With Application Execution on Clusters Karen L. Reid and Michael Stumm [email protected] [email protected] Department of Computer Science Department of Electrical and Computer
IMPROVING DATA INTEGRATION FOR DATA WAREHOUSE: A DATA MINING APPROACH
IMPROVING DATA INTEGRATION FOR DATA WAREHOUSE: A DATA MINING APPROACH Kalinka Mihaylova Kaloyanova St. Kliment Ohridski University of Sofia, Faculty of Mathematics and Informatics Sofia 1164, Bulgaria
low-level storage structures e.g. partitions underpinning the warehouse logical table structures
DATA WAREHOUSE PHYSICAL DESIGN The physical design of a data warehouse specifies the: low-level storage structures e.g. partitions underpinning the warehouse logical table structures low-level structures
Data Mining. Cluster Analysis: Advanced Concepts and Algorithms
Data Mining Cluster Analysis: Advanced Concepts and Algorithms Tan,Steinbach, Kumar Introduction to Data Mining 4/18/2004 1 More Clustering Methods Prototype-based clustering Density-based clustering Graph-based
ANALYSIS OF AVIRIS DATA: A COMPARISON OF THE PERFORMANCE OF COMMERCIAL SOFTWARE WITH PUBLISHED ALGORITHMS. William H. Farrand 1
ANALYSIS OF AVIRIS DATA: A COMPARISON OF THE PERFORMANCE OF COMMERCIAL SOFTWARE WITH PUBLISHED ALGORITHMS William H. Farrand 1 1. Introduction An early handicap to the effective use of AVIRIS data was
WHITE PAPER. Permabit Albireo Data Optimization Software. Benefits of Albireo for Virtual Servers. January 2012. Permabit Technology Corporation
WHITE PAPER Permabit Albireo Data Optimization Software Benefits of Albireo for Virtual Servers January 2012 Permabit Technology Corporation Ten Canal Park Cambridge, MA 02141 USA Phone: 617.252.9600 FAX:
Globus Striped GridFTP Framework and Server. Raj Kettimuthu, ANL and U. Chicago
Globus Striped GridFTP Framework and Server Raj Kettimuthu, ANL and U. Chicago Outline Introduction Features Motivation Architecture Globus XIO Experimental Results 3 August 2005 The Ohio State University