PEXTA: A PGAS I/O for Hyperspectral Data Storage

PEXTA: A PGAS I/O for Hyperspectral Data Storage G. Nimako, E.J Otoo and Michael Sears School of Computer Science The University of the Witwatersrand Johannesburg, South Africa CHPC Annual Meeting, December 2013.

Introduction The conceptual model of a dataset as a multidimensional array has been around since the early days of relational database model. Records of multi-attribute files, perceived as points in multi-dimensional space, form the bases of many of the efficient data structures and algorithms. Examples: KD-Tree, Quad-Tree, Grid-File, multi-dimensional OLAP and data warehousing. Scientific data-sets are maintained predominantly as multi-dimensional arrays - HDF5, NetCDF, New SciDB, etc. Otoo,Nimako&Sears () Using Chunked Ext... 2 / 30

Motivation Processing massively large scientific data (Big Data), in the realm of Peta-scale and Exa-scale computing using Partitioned Global Address Space (PGAS) computational model. This also subsumes the hadoop-like computing model. The PGAS computational model are exemplified by: Global Array Toolkit, Unified Parallel C (UPC), OpenShmem Application to hyper-spectral image data storage and analysis The database consistent with this model of computation is SciDB It is an Array Data Model on shared parallel architectures with unbounded non-uniform dimensions with holes, arbitrary nesting, sophisticated storage representations, including overlapping chunks, etc. Otoo,Nimako&Sears () Using Chunked Ext... 3 / 30

Motivation: The Global Array Toolkit Distributed dense arrays that can be accessed through a shared memory-like style Single, shared data structure/global indexing Otoo,Nimako&Sears () Using Chunked Ext... 4 / 30

Multidimensional Array Files Mapping of conventional array elements onto linear storage, using row-major or column-major order allows for extendability of the array in only one dimension. Otoo,Nimako&Sears () Using Chunked Ext... 5 / 30

Multidimensional Array Files Mapping of conventional array elements onto linear storage, using row-major or column-major order allows for extendability of the array in only one dimension. The question then arises: How can we store arrays elements so that the index range of any dimension can be extended? Otoo,Nimako&Sears () Using Chunked Ext... 5 / 30

Multidimensional Array Files Mapping of conventional array elements onto linear storage, using row-major or column-major order allows for extendability of the array in only one dimension. The question then arises: How can we store arrays elements so that the index range of any dimension can be extended? The problem has been addressed in some earlier papers. Notably by : G. Nimako, E. Otoo and D. Kwofie, PARCO Journal [2013] G. Nimako, E. Otoo and D. Kwofie, IASDS of IEEE Cluster [2013] G. Nimako, E. Otoo and D. Kwofie, P2S2 [2012] Tsuji, M. Kuroda and K. Higuchi [2008] A. Rosenberg and L. Stockmeyer [1977] Otoo,Nimako&Sears () Using Chunked Ext... 5 / 30

Main Contributions We present a computable access function F ( i 0, i 1,..., i k 1 ) that maps the k-dimensional index i 0, i 1,..., i k 1 to a value that gives the linear address location l j in time O(k + log N) using O(k 2 N 1/k ) addition space. This allows us to store elements of extendible array directly as array-files. Otoo,Nimako&Sears () Using Chunked Ext... 6 / 30

Main Contributions We present a computable access function F ( i 0, i 1,..., i k 1 ) that maps the k-dimensional index i 0, i 1,..., i k 1 to a value that gives the linear address location l j in time O(k + log N) using O(k 2 N 1/k ) addition space. This allows us to store elements of extendible array directly as array-files. We introduce Parallel Chunked Extendible Dense Array I/O (PEXTA) for the GA Otoo,Nimako&Sears () Using Chunked Ext... 6 / 30

Main Contributions We present a computable access function F ( i 0, i 1,..., i k 1 ) that maps the k-dimensional index i 0, i 1,..., i k 1 to a value that gives the linear address location l j in time O(k + log N) using O(k 2 N 1/k ) addition space. This allows us to store elements of extendible array directly as array-files. We introduce Parallel Chunked Extendible Dense Array I/O (PEXTA) for the GA PEXTA can be used as a drop-in replacement for the Disk Resident Array of the GA-Toolkit Otoo,Nimako&Sears () Using Chunked Ext... 6 / 30

Main Contributions We present a computable access function F ( i 0, i 1,..., i k 1 ) that maps the k-dimensional index i 0, i 1,..., i k 1 to a value that gives the linear address location l j in time O(k + log N) using O(k 2 N 1/k ) addition space. This allows us to store elements of extendible array directly as array-files. We introduce Parallel Chunked Extendible Dense Array I/O (PEXTA) for the GA PEXTA can be used as a drop-in replacement for the Disk Resident Array of the GA-Toolkit PEXTA provides an efficient I/O for Parallel Implementation of Endmember Extraction Algorithms From Hyperspectral Data. Otoo,Nimako&Sears () Using Chunked Ext... 6 / 30

Rest of the Talk Extendible Arrays of Data Chunked Extendible Arrays PEXTA I/O Model PEXTA Implementation for GA Using PEXTA for Parallel Implementation of Endmember Extraction Algorithms From Hyperspectral Data Summary and Future Directions Otoo,Nimako&Sears () Using Chunked Ext... 7 / 30

Conventional Array Using Row-Major Ordering Consider an 3-D array A[U 0 ][U 1 ][U 2 ] whose elements are allocated in row-major order. The location of an element A i 0, i 1, i 2, is given by loc(a 0, 0, 0 + q where q is computed as: q = i 0 C 0 + i 1 C 1 + i 2 C 2 and C 0 = U 1 U 2 C 1 = U 2 C 2 = 1 The values C 0, C 1, C 2 are held in the dope-vector. This reveals the strategy of our allocation scheme. Otoo,Nimako&Sears () Using Chunked Ext... 8 / 30

Conventional Array Using Row-Major Ordering Consider an 3-D array A[3][3][2] whose elements are allocated in row-major order as below This reveals the strategy of our allocation scheme. Otoo,Nimako&Sears () Using Chunked Ext... 9 / 30

Linear Mapping for a Dense Extendible Array The mapping function for extendible array uses axial-vectors to store information that includes the dope-vector at the time of extending the dimension A list of axial-vectors is maintained for each dimension. Let A[U 0 ][U 1 ][U 2 ] be an arbitrary 3-dimensional array, where U j implies that the bound can be extended. Similarly we employ the notation: F() when referring to conventional array mapping function. F () when referring to a mapping function for an extendible array. Otoo,Nimako&Sears () Using Chunked Ext... 10 / 30

Illustration of the Linear Mapping for a Dense Extendible Array Otoo,Nimako&Sears () Using Chunked Ext... 11 / 30

Linear Mapping for a Dense Extendible Array Suppose that in a k-dimensional extendible array A[U 0][U 1][U 2]...[U k 1 ], dimension l is extended by λ l, then the index range increases from U l to U l + λ l. Let the location A 0, 0,..., U l,..., 0 (i.e. the starting location of an allocated hyperslab ) be denoted as l Z 0 where Z 0 U U = k 1 l r=0 U r. l The Mapping Function k 1 q = F ( i 0, i 1, i 2,..., i k 1 )) = Z 0 U + (i l U l )C l + i j Cj l j=0 j l k 1 Cl = U j j=0 j l C j = k 1 r=j+1 r l U r Otoo,Nimako&Sears () Using Chunked Ext... 15 / 30

Chunking Extendible Dense Arrays The use of the list for axial-vectors can be expensive and depends in particular on the interruptible expansions. (worst case is when we have cubical extensions). Such interruptible expansions cause the addition of a new entry in the list of axial vectors. Chunking the array gives some additional advantages: It gives contiguous storage allocations for the elements of the chunks. When arrays are allocated onto secondary storage, I/O can be made in multiples of the chunk size. The allocation is done in chunks as opposed to the single elements. Otoo,Nimako&Sears () Using Chunked Ext... 16 / 30

Illustration of Chunked Extendible Dense Arrays Otoo,Nimako&Sears () Using Chunked Ext... 17 / 30

PEXTA Overview and Implementation We model the PEXTA after MPI-I/O Definitions The filetype describes the data layout in file The buftype describes how data is laid out in a process s buffer The elementary datatype, etype, ensures consistency between the type signature of both filetype and buftype. The principal-array data and the axial-vectors, that correspond to the leaf-nodes of the O 2 -Tree, are stored on disk as the principal-array file, F p and metadata file, F m, respectively. Although F p will be stored on disk, it is accessed through the buffers of each process in a communicator group. Otoo,Nimako&Sears () Using Chunked Ext... 18 / 30

PEXTA Overview and Implementation PEXTA Model supports both regular and irregular distributions of global array of the GA-Toolkit. We have completed implementation for regular distribution. Processes can perform independent or collective reads and writes of their respective sub-arrays. An external/independent process can request for an exclusive write lock token on the metadata file, F m and append chunks of data to F p Otoo,Nimako&Sears () Using Chunked Ext... 19 / 30

PEXTA Overview and Implementation When a new session starts, the processes within a communicator group reads the newly updated metadata file into their individual O 2 -Trees. Currently, we do not support in-memory extensions. Otoo,Nimako&Sears () Using Chunked Ext... 20 / 30

PEXTA Overview and Implementation The PEXTA API provides routines quite similar to DRA for creating disk resident array We however provide additional functions for extendibility. Functions int PEXTA Init(int maxarrays, double maxarraysize, double totaldiskspace, double maxmemory); int PEXTA Create(int type, int ndim, size t dims[],char *name, char* filename, int mode, int mode, size t reqdims[], int g a, int *p a); int PEXTA IRead(int g a, int p a, int *request); int PEXTA Extend(int p a, size t edims[],size t odims[], int *request); etc. Otoo,Nimako&Sears () Using Chunked Ext... 21 / 30

Hyperspectral Data Otoo,Nimako&Sears () Using Chunked Ext... 22 / 30

HPC in Remote Sensing Applications Recently, high-performance computing systems have become more widespread in remote sensing applications. Parallel techniques for endmember extraction may allow rapid inspection of massive data archives It will also enable generation of comprehensive spectral libraries and other useful information An opportunity to explore methodologies in areas such as multisource data registration and mining that previously looked too computationally intensive for practical applications due to the immense files involved Otoo,Nimako&Sears () Using Chunked Ext... 23 / 30

Hyperspectral Data Analysis Mixed pixels (due to insufficient spatial resolution and mixing effects in surfaces) Sub-pixel targets (very important and crucial in many hyperspectral applications) Otoo,Nimako&Sears () Using Chunked Ext... 24 / 30

Spectral Unmixing of Hyperspectral Data The goal is to find extreme pixel vectors (endmembers) that can be used to unmix other mixed pixels in the data using a linear mixture model. Each mixed pixel can be obtained as a combination of endmember fractional abundances. A crucial issue is how to find spectral endmembers. Endmember Extraction Algorithms: Boardman s Pixel Purity Index (PPI) Algorithm Harsanyi and Chang s Orthogonal Subspace Projection (OSP) Winter s N-FINDR Algorithm Otoo,Nimako&Sears () Using Chunked Ext... 25 / 30

Parallel Endmember Extraction Using PEXTA and GA The master processor scatters the data among the distributed workers We provide collective operations in PEXTA for data partitioning. Alternatively, the GA collective operation for data partitioning can be used. Otoo,Nimako&Sears () Using Chunked Ext... 26 / 30

Parallel Endmember Extraction Using PEXTA and GA Spectral-Domain Partitioning A single pixel vector (spectral signature) may be stored in different processing units It requires communications for individual pixel-based calculations such as spectral angle computations. Otoo,Nimako&Sears () Using Chunked Ext... 27 / 30

Parallel Endmember Extraction Using PEXTA and GA Spatial-Domain Partitioning Every pixel vector (spectral signature) is stored in the same processing unit. This is beneficial for a proposed sliding-window algorithm in terms of low-level image processing computations. Otoo,Nimako&Sears () Using Chunked Ext... 28 / 30

Summary and Future Work Poor parallel I/O performance has been a bottleneck in many data-intensive applications The organisation of extendible arrays using a mapping function is highly appropriate for most scientific datasets where the model of the data is perceived as a large multi-dimensional array. Main application areas is for hyper-spectral image processing and a host other scientific applications that are based on multidimensional data models. Remote sensing missions require efficient I/O able to cope with the extremely high dimensionality of the collected data without compromising mission payload. Future work on the use of in memory resident caches of both the chunks and the array are anticipated. Otoo,Nimako&Sears () Using Chunked Ext... 29 / 30

Thanks! Thank You Questions? Otoo,Nimako&Sears () Using Chunked Ext... 30 / 30