Integrating Parallel File I/O and Database Support for High-Performance Scientific Data Management

Transcription

1 In Pro of SC: High Performane Networking and Computing, Novemer Integrating Parallel File I/O and Dataase Support for High-Performane Sientifi Data Management Jaehun No Rajeev Thakur Alok Choudhary Math and Computer Siene Division Dept of Eletrial and Computer Eng Argonne National Laoratory Northwestern University Argonne, IL 39, USA Evanston, IL 8, USA jano, Astrat Many sientifi appliations have large I/O requirements, in terms of oth the size of data and the numer of files or data sets Management, storage, effiient aess, and analysis of this data present an extremely hallenging task Traditionally, two different solutions are used for this prolem: file I/O or dataases File I/O an provide high performane ut is tedious to use with large numers of files and large and omplex data sets Dataases an e onvenient, flexile, and powerful ut do not perform and sale well for parallel superomputing appliations We have developed a software system, alled Sientifi Data Manager (SDM), that aims to omine the good features of oth file I/O and dataases SDM provides a high-level API to the user and, internally, uses a parallel file system to store real data and a dataase to store appliation-related metadata SDM takes advantage of various I/O optimizations availale in MPI-IO, suh as olletive I/O and nonontiguous requests, in a manner that is transparent to the user As a result, users an write and retrieve data with the performane of parallel file I/O, without having to other with the details of atually performing file I/O In this paper, we desrie the design and implementation of SDM With the help of two parallel appliation templates, ASTRO3D and an Euler solver, we illustrate how some of the design riteria affet performane //$ IEEE

2 Introdution Many large-sale sientifi experiments and simulations generate very large amounts of data [8, ] (on the order of several hundred gigaytes to teraytes), spanning thousands of files or data sets Management, storage, effiient aess, and analysis of this data present an extremely hallenging task Currently availale tehniques for this purpose are either raw file-i/o interfaes, suh as MPI-IO [, 9], or full-fledged dataases File-I/O interfaes provide high performane ut are too umersome to use with large, omplex data sets and large numers of files For example, the user must rememer file names and the organization of data in a file and must speify the exat loation in the file from whih data must e aessed Dataases, on the other hand, provide a onvenient, high-level interfae and powerful data-retrieval apaility, ut they do not measure up to the performane requirements of large-sale sientifi appliations running on superomputers We have developed a software system, alled Sientifi Data Manager (SDM), that aims to omine the good features of oth file I/O and dataases SDM provides a high-level, user-friendly interfae Internally, SDM interats with a dataase to store appliation-related metadata and uses MPI-IO to store the real data on a high-performane parallel file system SDM takes advantage of various I/O optimizations availale in MPI-IO, suh as olletive I/O and nonontiguous requests, in a manner that is transparent to the user As a result, users an aess data with the performane of parallel file I/O, without having to other with the details of file I/O Figure illustrates the asi idea In this paper, we desrie the design and implementation of SDM SDM is urrently implemented to use either MySQL [] or PostgreSQL [] as the dataase for metadata and MPI-IO for file I/O In designing suh a system, we have a wide hoie of how to organize the data in files We have implemented three different approahes At one extreme, level, we store all data sets in separate files as they are generated At the other extreme, level 3, we store data sets in a very small numer of files and, using a dataase tale, keep trak of where in the files eah data set is stored We also have an intermediate approah, alled level We examine the performane impliations of using eah of these approahes y studying the performane results otained for two appliation templates, ASTRO3D and an Euler solver, on an IBM SP and SGI Origin The rest of this paper is organized as follows In Setion we disuss our goals in developing SDM In Setion 3 we desrie how SDM is implemented Performane results are presented in Setion We disuss related work in Setion 5 We onlude and outline our plans for future work in Setion Design Ojetives We had three major goals in developing SDM: provide high-performane parallel I/O, provide a high-level appliation programming interfae (API) that eliminates the need for the user to other with the details of low-level file I/O or dataases, and store enough metadata in a dataase so that the user an easily retrieve previously stored data High-Performane I/O To ahieve high-performane I/O, we deided to use a parallel file-i/o system to store real data and use MPI-IO to aess this data MPI-IO, the I/O interfae defined as part of the MPI- standard [, 9], is rapidly emerging as the standard, portale API for I/O in parallel appliations High-performane implementations of MPI-IO, oth vendor and puli-domain implementations, are availale for most platforms [9, 5, 5,, 3] MPI-IO is speifially designed to enale the optimizations that are ritial for high-performane parallel I/O Examples of these optimizations inlude olletive I/O, the aility to aess nonontiguous data sets with a single funtion,

3 Appliation Data SDM Metadata MPI-IO Dataase Parallel file system HPSS Hierarhial storage management system Figure : SDM arhiteture and the aility to pass hints to the implementation aout aess patterns, file-striping parameters, and so forth High-Level API Our goal was to provide an API that did not require the user to know either MPI-IO or dataases The user an speify the data with a high-level desription, together with annotations, and use a similar API for data retrieval SDM internally translates the user s request into appropriate MPI-IO alls, inluding reating MPI derived datatypes for nonontiguous data [33] SDM also interats with the dataase when neessary, y using emedded SQL funtions Convenient Data-Retrieval Capaility SDM allows the user to speify names and other attriutes to e assoiated with a data set SDM internally selets a file name into whih the data will e stored; the mapping etween data sets and file names is stored in the dataase The user an retrieve a data set y speifying a unique set of attriutes for the desired data 3 Implementation To desrie the metadata storage in the dataase, the SDM API, and the organization of data in files, we use an example, ASTRO3D, an astrophysis appliation developed at the University of Chiago For simpliity of explanation, we onsider the two-dimensional version of this three-dimensional appliation (The performane results presented in this paper are for the full three-dimensional version) In this appliation, data is stored in several arrays that are lok-distriuted in eah dimension At various time steps, several of these arrays are written to files for data analysis, restart, and visualization Six floating-point arrays are written for data analysis and another six for restart; seven harater arrays are written for visualization The frequenies of the writes an e varied We use the term data set to refer to eah array eing written and data group to refer to all the arrays written at a time step for a partiular purpose suh as data analysis For simpliity of explanation, let us

4 Appliation Client API MPI I/O Data Base run_tale runid dimension 3 prolem_size numoftimesteps year month day hour min aess_pattern_tale File System Data Data Data runid dataset asi_pattern data_type storage_order aess_pattern gloal_size runid a a a exeution_tale dataset timesteps a a a file_offset file_name Figure : Dataase tales used in SDM assume that three arrays are written for data analysis, another three for restart, and four for visualization (Note that all arrays six, six, and seven were used in the performane experiments reported in this paper) Let us further assume that the data-analysis and restart dumps are performed every six time steps and the visualization dumps are performed every four time steps Let e the three data sets for data analysis and e the data group for data analysis Similarly, we have! " for visualization and #$% % %& for restart 3 Dataase Tales to Store Metadata SDM uses three dataase tales for storing metadata: run tale, aess pattern tale, and exeution tale (see Figure ) These tales are made for eah appliation Eah time an appliation writes data sets, SDM enters the prolem size, dimension, urrent date, and a unique identifiation numer (runid) to the run tale The aess pattern tale inludes the properties of eah data set, suh as data type, storage order, data aess pattern, and gloal size SDM uses this information to make appropriate MPI-IO alls to aess the real data The exeution tale stores a gloally determined file offset denoting the starting offset in the file of eah data set 3

5 SDM_initialize(App); A = SDM_make_datalist(3, { a, a, a }); write all read all A[]data_type = FLOAT; A[]aess_pattern[] = BLOCK; A[]aess_pattern[] = BLOCK; SDM_assoiate_attriutes(3, &A[]); handlea = SDM_set_attriutes(3, A); headera = SDM_make_header(3, A, FLOAT, ); SDM_suarray(handleA, 3,, StartingPoints, SuArraySizes, NULL); for (i=; i<lasttimestep; i++) { } Computation SDM_writeH(headerA, a, i, headerbuf); SDM_writeH(headerA, a, i, headerbuf); SDM_writeH(headerA, a, i, headerbuf); SDM_write(handleA, a, i, uf); SDM_write(handleA, a, i, uf); SDM_write(handleA, a, i, uf); initialize(&date); dateyear = 999; datemonth = ; dateday = ; handlea = SDM_selet_attriutes(3, A); headera = SDM_selet_attriutesH(3, A); SDM_suarray(handleA, 3,, StartingPoints, SuArraySizes, NULL); for (i=; i<lasttimestep; i++) { } SDM_readH(headerA, SDM_readH(headerA, SDM_readH(headerA, SDM_read(handleA, SDM_read(handleA, SDM_read(handleA, Computation a a a, i, headerbuf);, i, headerbuf);, i, headerbuf); a, i, uf); a, i, uf); a, i, uf); SDM_finalize(3, handlea); SDM_finalizeH(3, headera); Figure 3: Example of using the SDM API to perform I/O in ASTRO3D 3 Appliation Programming Interfae Figure 3 shows how the SDM API is used to perform I/O for data analysis (data group A) in a twodimensional version of ASTRO3D To use SDM, the user must first all the funtion SDM initialize This funtion initializes the SDM environment and estalishes a onnetion to the dataase Next, to speify groups of data sets, the user must all the funtion SDM make datalist This funtion assigns properties to the first data set in a group The same properties an e assigned to other data sets in the same group y alling SDM assoiate attriutes The main reason for making groups of data sets is that SDM an then use different ways of organizing data in files, with different performane impliations For example, eah data set an e written in a separate file, or the data sets of a group an e written to a single file In the ase of write operations, the user must all SDM set attriutes to set the attriutes assoiated with a group and to return a set of handles to e used for further I/O operations If an appliation writes header information along with the data, SDM make header must e used to return an array of handles for writing the header information In ASTRO3D, eah data set in group A has a header onsisting of six

6 floating-point variales The funtion SDM writeh is used to write the header In the ase of read operations, data from a speifi run an e retrieved y speifying attriutes of the data, suh as the date of the run If the date is not speified, data from the last run will e read Also, the properties of the data sets need not e speified eause SDM retrieves this information from the dataase Data an e seleted using the SDM selet attriutes funtion To retrieve header information, SDM selet attriutesh must e alled The main SDM funtions for writing and reading data are SDM write and SDM read Before alling these funtions, the user must provide the information neessary for SDM to perform I/O, suh as the starting points and sizes of the suarray in eah dimension in the ase of lok distriution, or the size of proess grids and distriution arguments in eah dimension in the ase of yli distriution To perform I/O, the handle of a group, position of a data set within the handle (group), urrent time step, and pointer to the appliation uffer are passed to the SDM I/O funtion Note that the user does not have to provide file names SDM generates the file name and reords the name in the dataase Finally, the user must all SDM finalize and SDM finalizeh to lose all files, lose the onnetion to the dataase server, and free all memory alloated y SDM 33 File Organization SDM supports three different ways of organizing data in files In level, eah data set generated at eah time step is written to a separate file, as shown in Figure This file organization is simple, ut it inurs the ost of a file open and lose at eah time step, whih on some file systems an e quite high, as we shall see in the performane results For a large numer of data sets and time steps, this method an e expensive eause of the large numer of file opens In level, eah data set (within a group) is written to a separate file, ut different iterations of the same data set are appended to the same file, as illustrated in Figure 5 This method results in a smaller numer of files and smaller file-open osts The offset in the file where data is appended is stored in the exeution tale In level 3, all iterations of all data sets elonging to a group are stored in a single file, as shown in Figure As in level, the file offset for eah data set is stored in the exeution tale y proess in the SDM write funtion If a file system has high open and lose osts, SDM an generate a very small numer of files y hoosing the level-3 file organization On the other hand, if an appliation produes a large numer of data sets of large size, level 3 would result in very large files, whih may affet performane We study the performane impliations of the three file-organization levels in the next setion Performane Results We otained all performane results on the IBM SP and SGI Origin at Argonne National Laoratory The IBM SP has 8 ompute nodes and I/O nodes Eah I/O node ontrols four SSA disks, eah of 9 Gyte apaity The parallel file system on the mahine is IBM s PIOFS [] The SGI Origin has 8 proessors and Fire Channel ontrollers onneted to a total of disks, eah of 9 Gyte apaity The file system on the Origin is SGI s XFS [, 3] XFS supports an optimization alled diret I/O, whih we used in our experiments When ertain alignment restritions are met, the user an hoose the diret-i/o option, in whih the file system moves data diretly etween the user s uffer and the storage devie, ypassing the file-system ahe Diret I/O thus eliminates an extra memory opy into the ahe and an perform well if the I/O size is large and the mahine has a high-andwidth I/O system Diret I/O 5

7 (th time step) A group ASTRO3D a RId a ASTRO3D a RId a ASTRO3D a RId a ASTRO3D a RId a ASTRO3D a RId a ASTRO3D a RId a B group ASTRO3D RId ASTRO3D RId ASTRO3D RId ASTRO3D RId ASTRO3D RId ASTRO3D RId ASTRO3D 3 RId 3 ASTRO3D 3 RId 3 C group ASTRO3D RId ASTRO3D RId ASTRO3D RId Figure : Level- file organization at th time step in ASTRO3D The supersript on a data set denotes the time step at whih the data set has een written to a file, RId denotes the urrent identifiation numer (runid), and eah shadowed ox (along with the name eside it) shows the SDM-generated file for storing the orresponding data set an e used from an MPI-IO program the ROMIO implementation of MPI-IO that we used supports diret I/O [35] We present performane results with oth diret I/O and regular (uffered) I/O We used two appliation templates, ASTRO3D and a three-dimensional Euler solver, in our performane experiments For ASTRO3D, we used a prolem size of '( )+*,'( )+*-' ( ) We ran the program for one time step and performed the data analysis, restart, and visualization dumps at that time step This resulted in a total of around 88 Mytes of data The seond appliation is a three-dimensional Euler solver for the prolem of three-dimensional transoni flow aout an M wing [] This appliation is a mesh-strutured ode that writes the physial values and residual of eah node at ertain iterations The struture of these values is a distriuted gloal vetor, and eah value has five omponents (density, energy, and three oordinates of momentum) In addition, the appliation writes the physial oordinates and pressure at eah mesh point In our experiments, we ran the ode for 5 iterations and wrote data at every 5 iterations The prolem size was /*3* Costs of Dataase Aess SDM uses TCP/IP to onnet to the dataase servers We performed our experiments with two different dataases, MySQL [] and PostgreSQL [] Figure 7 shows the dataase-aess ost in the SDM write operation on the Origin As mentioned in Setion 3, the onnetion to and disonnetion from the dataase server our one in SDM initialize and SDM finalize, respetively In SDM set attriutes, proess aesses the run tale and aess pattern tale to store attriutes, and in the write operation,

8 (th time step) A group ASTRO3Da RId a a ASTRO3Da RId a a ASTRO3Da RId a a B group ASTRO3D RId ASTRO3D RId ASTRO3D RId ASTRO3D 3 RId 3 3 C group ASTRO3D RId ASTRO3D RId ASTRO3D RId Figure 5: Level- file organization at th time step in ASTRO3D The supersript on a data set denotes the time step at whih the data set has een written to a file, RId denotes the urrent identifiation numer (runid), and eah shadowed ox (along with the name eside it) shows the SDM-generated file for storing the orresponding data set (th time step) A group ASTRO3D a RId a a a a a a B group ASTRO3D RId 3 3 C group ASTRO3D RId Figure : Level-3 file organization at th time step in ASTRO3D The supersript on a data set denotes the time step at whih the data set has een written to a file, RId denotes the urrent identifiation numer (runid), and eah shadowed ox (along with the name eside it) shows the SDM-generated file for storing the orresponding data set 7

9 9 8 = > > < Exeution Time (Se) 8 onnet/disonnet selet/update/insert I/O Bandwidth (MB/se) 8 Postgres 5 Mysql 7 8 Postgres 5 Mysql 7 8 : Astro3D Euler Solver level ; / level < 3 level ; / level 3 (Write Operation) (Read Operation) Figure 7: Cost of aessing the dataase for the two appliations Figure 8: I/O andwidth for ASTRO3D on the IBM SP it stores the file offset into the exeution tale In ASTRO3D the aess to the exeution tale ourred 9 times, and in the Euler solver the aess to the exeution tale ourred times As an e seen in Figure 7, the dataase-aess ost using oth the dataase servers was less than se This ost, however, will hange aording to the numer of I/O operations ourring in the appliations We oserved that MySQL performs etter than PostgreSQL Therefore, we used only MySQL for the rest of the performane experiments Results for ASTRO3D Figure 8 shows the write and read andwidths for ASTRO3D on the IBM SP using 3 proessors for the three levels of file organization Sine we ran only one iteration of the program, levels and resulted in the same file organization Level 3 ahieved muh higher andwidth eause only three different files were reated, and, therefore, only three file opens ourred The high ost of file opens on the PIOFS file system [3] resulted in lower performane for levels and, where 9 separate files were reated The impat of file-open time an indeed e quite large Figures 9 and show the write and read andwidths for ASTRO3D on the SGI using proessors We measured performane for oth diret I/O and uffered I/O For writing data, diret I/O performed etter than uffered I/O There are two reasons for this First, with uffered I/O, XFS serializes onurrent writes to the same file, whereas with diret I/O, onurrent writes are allowed to proeed in parallel Seond, diret I/O eliminates a opy into the file-system ahe For reading data, uffered I/O performed etter Again, there are two reasons for this One reason is that XFS does not serialize uffered reads; therefore, diret reads do not have any extra advantage in the area of parallelism The seond reason is that XFS performs a read-ahead (prefeth) in the ase of uffered reads, ut not in ase of diret reads The read-ahead poliy works well for this appliation, and uffered reads therefore perform etter Sine the ost of file opens is small on XFS, the three levels of file organization performed nearly the same 8

10 > < > < Write Bandwidth (MB/se) 5 5 Read Bandwidth (MB/se) 5 5 level ; / level < 3 level ; / level 3 > (Diret I/O) (Buffered I/O) level ; / level < 3 level ; / level 3 > (Diret I/O) (Buffered I/O) Figure 9: Write andwidth for ASTRO3D on the SGI Origin Figure : Read andwidth for ASTRO3D on the SGI Origin 3 Results for the Euler Solver Figures and show the write and read andwidths for the Euler solver on the IBM SP using 3 proessors The total data written was around Mytes In level 3, only two files were generated, one for writing the oordinates and pressure at eah mesh node and the other for writing the physial values and residual at eah node In level, six vetors (that is, the three oordinates, pressure, physial values of eah node, and nodal residual) were written separately, resulting in a total of six files In level, the six vetors generated every five iterations were written separately, resulting in a total of files As Figures and show, level 3 performed the est eause of the high open ost on PIOFS In level, the file-open ost took around 8% of the total exeution time; in level, it took around 3%; and in level 3, it took around % of the total exeution time Figures 3 and show the write and read andwidths for the Euler solver using proessors on the SGI For this appliation, we used only uffered I/O We ould not use diret I/O eause the memory alloation for distriuted vetors was done inside the numerial lirary (PETS [3]) that the appliation uses, and thus we ould not align the uffers to the ahe line as required for diret I/O For the write operation, levels and 3 performed slightly etter than level For the read operation, however, level performed the est The reason is that the read-ahead poliy of XFS for uffered reads operates on a per-file asis and therefore works to the appliation s advantage when it has a greater numer of files 5 Related Work The main differene etween this work and other efforts is that this work aims to omine the good features of parallel file I/O and dataases, whereas other efforts fous on either parallel I/O or data management, not oth We riefly mention related efforts elow SRB (Storage Resoure Broker) [] provides a uniform interfae to aess various storage systems, suh 9

11 Write Bandwidth (MB/se) 3 Read Bandwidth (MB/se) 3 level? level? level? 3 level? level? level? 3 Figure : Write andwidth for the Euler solver on the IBM SP Figure : Read andwidth for the Euler solver on the IBM SP as file systems, Unitree, HPSS, and dataase ojets However, it does not support optimizations suh as olletive I/O that MPI-IO provides Shoshani et al [8, 9] desrie an arhiteture for optimizing aess to large volumes of sientifi data stored on tapes Chervenak et al [5] desrie a general arhiteture for managing distriuted sientifi data sets in a grid environment An arhiteture for data-intensive distriuted omputing using DPSS is desried in [37, 38] The Ative Data Repository [7] optimizes storage, retrieval and proessing of very large multi-dimensional datasets An initial disussion of a framework for sientifi data management similar to the one desried in this paper is given in [] Several efforts have involved optimizing I/O in parallel file systems and runtime liraries [3,, 7, 3,, 8,, 7, 3] However, file systems and liraries have a lower-level interfae than SDM, requiring more work from the user Conlusions and Future Work We have presented the design and implementation of an environment for high-performane sientifi data management, alled Sientifi Data Manager (SDM), that is uilt on top of MPI-IO and also interats with a dataase for storing metadata SDM provides a simple, high-level interfae and performs all neessary I/O optimizations transparently to the user We also experimented with different ways of organizing data in files, alled level level 3 In general, when file-open ost on a partiular file system is high, level 3 performs well eause it minimizes the numer of files reated If the file-open ost is small, the performane of the three levels depends on how the numer and size of files affets performane on the partiular file system An appropriate file-organization poliy an therey e hosen for a partiular file system On the XFS file system, we found that the file-open ost was so small that it did not signifiantly affet I/O performane Instead, our experiment foused on the use of diret I/O and uffered I/O in the ASTRO3D template For writing data, we found that diret I/O performed muh etter than uffered I/O y avoiding the overhead of opying the data into the XFS uffer ahe and also eause XFS allows diret writes to

12 8 Write Bandwidth (MB/se) Read Bandwidth (MB/se) level? level? level? 3 level? level? level? 3 Figure 3: Write andwidth for the Euler solver on the SGI Origin Figure : Read andwidth for the Euler solver on the SGI Origin proeed onurrently For reading data, however, uffered I/O performed etter eause of its read-ahead poliy We are developing SDM further to support other types of appliations suh as unstrutured-grid appliations and to support visualization We also plan to investigate whether an SDM-like system that stores data in files using MPI-IO and metadata in a real dataase an e effetively used as a strategy for implementing liraries suh as HDF [3] and netcdf [] Aknowledgments This work was supported y the Mathematial, Information, and Computational Sienes Division suprogram of the Offie of Advaned Sientifi Computing Researh, US Department of Energy, under Contrat W-3-9-Eng-38 Referenes [] Appliations Working Group of the Salale I/O Initiative Preliminary Survey of I/O Intensive Appliations appsps, 99 [] Chaitanya Baru, Reagan Moore, Arot Rajasekar, and Mihael Wan The SDSC Storage Resoure Broker In Proeedings of CASCON 98, Deemer 998 [3] Roert Bennett, Kelvin Bryant, Alan Sussman, Raja Das, and Joel Saltz Jovian: A Framework for Optimizing Parallel I/O In Proeedings of the Salale Parallel Liraries Conferene, pages IEEE Computer Soiety Press, Otoer 99 [] Rajesh Bordawekar, Juan Miguel del Rosario, and Alok Choudhary Design and Evaluation of Primitives for Parallel I/O In Proeedings of Superomputing 93, pages 5, Novemer 993

13 [5] Ann Chervenak, Ian Foster, Carl Kesselman, Charles Salisury, and Steven Tueke The Data Grid: Towards an Arhiteture for the Distriuted Management and Analysis of Large Sientifi Datasets In Proeedings of the Network Storage Symposium (NetStore 99), Otoer 999 [] A Choudhary, M Kandemir, H Nagesh, J No, X Shen, V Taylor, S More, and R Thakur Data Management for Large-Sale Sientifi Computations in High Performane Distriuted Systems In Pro of the Eighth IEEE Int l Symposium on High Performane Distriuted Computing, pages 3 7, August 999 [7] Peter F Corett and Dror G Feitelson The Vesta parallel file system ACM Transations on Computer Systems, (3):5, August 99 [8] Juan Miguel del Rosario and Alok Choudhary High Performane I/O for Parallel Computers: Prolems and Prospets Computer, 7(3):59 8, Marh 99 [9] Samuel A Fineerg, Parkson Wong, Bill Nitzerg, and Chris Kuszmaul PMPIO A Portale Implementation of MPI-IO In Proeedings of the Sixth Symposium on the Frontiers of Massively Parallel Computation, pages IEEE Computer Soiety Press, Otoer 99 [] W D Gropp, D E Keyes, L C MInnes, and M D Tidriri Gloalized Newton-Krylov-Shwarz Algorithms and Software for Parallel Impliit CFD Tehnial Report ICASE TR 98- (to appear in Int J Superomputer Appliations), 998 [] William Gropp, Ewing Lusk, and Rajeev Thakur Using MPI-: Advaned Features of the Message-Passing Interfae MIT Press, Camridge, MA, 999 [] Mike Holton and Raj Das XFS: A Next Generation Journalled -Bit Filesystem With Guaranteed Rate I/O Tehnial report, SGI, In, 99 [3] Jay Huer, Christopher L Elford, Daniel A Reed, Andrew A Chien, and David S Blumenthal PPFS: A High Performane Portale Parallel File System In Proeedings of the 9th ACM International Conferene on Superomputing, pages ACM Press, July 995 [] IBM Corp IBM AIX Parallel I/O File System: Installation, Administration, and Use Doument Numer SH3-5-, August 995 [5] Terry Jones, Rihard Mark, Jeanne Martin, John May, Elsie Piere, and Linda Stanerry An MPI-IO Interfae to HPSS In Proeedings of the Fifth NASA Goddard Conferene on Mass Storage Systems, pages I:37 5, Septemer 99 [] David Kotz Disk-direted I/O for MIMD Multiproessors ACM Transations on Computer Systems, 5(): 7, Feruary 997 [7] Tahsin Kur, Chialin Chang, Renato Ferreira, Alan Sussman, and Joel Saltz Querying Very Large Multidimensional Datasets in ADR In Proeedings of SC99: High Performane Networking and Computing, Novemer 999 [8] Tara M Madhyastha and Daniel A Reed Intelligent, Adaptive File System Poliy Seletion In Proeedings of the Sixth Symposium on the Frontiers of Massively Parallel Computation, pages 7 79 IEEE Computer Soiety Press, Otoer 99 [9] Message Passing Interfae Forum MPI-: Extensions to the Message-Passing Interfae, July [] MySQL Referene Manual Version 33-alpha [] NetCDF [] Nils Nieuwejaar and David Kotz The Galley Parallel File System Parallel Computing, 3():7 7, June 997

14 [3] PETS for MPI [] Postgres Gloal Development Group PostgreSQL User s Guide, 99 [5] Jean-Pierre Prost MPI-IO/PIOFS World-Wide We page at 99 [] D Sanders, Y Park, and M Brodowiz Implementation and Performane of MPI-IO File Aess Using MPI Datatypes Tehnial Report UH-CS-9-, University of Houston, Novemer 99 [7] K E Seamons, Y Chen, P Jones, J Jozwiak, and M Winslett Server-Direted Colletive I/O in Panda In Proeedings of Superomputing 95 ACM Press, Deemer 995 [8] A Shoshani, L M Bernardo, H Norderg, D Rotem, and A Sim Storage Management for High Energy Physis Appliations In Proeedings of Computing in High Energy Physis (CHEP 98), 998 [9] A Shoshani, L M Bernardo, H Norderg, D Rotem, and A Sim Multidimensional Indexing and Query Coordination for Tertiary Storage Management In Pro of SSDBM 99, pages 5, July 999 [3] Adam Sweeney, Doug Douette, Wei Hu, Curtis Anderson, Mike Nishimoto, and Geoff Pek Salaility in the XFS File System In Pro of USENIX 99 Annual Tehnial Conferene, San Diego, CA, January 99 [3] Rajeev Thakur and Alok Choudhary An Extended Two-Phase Method for Aessing Setions of Out-of-Core Arrays Sientifi Programming, 5():3 37, Winter 99 [3] Rajeev Thakur, William Gropp, and Ewing Lusk An Experimental Evaluation of the Parallel I/O Systems of the IBM SP and Intel Paragon Using a Prodution Appliation In Proeedings of the 3rd International Conferene of the Austrian Center for Parallel Computation (ACPC) with Speial Emphasis on Parallel Dataases and Parallel I/O, pages 35 Leture Notes in Computer Siene 7 Springer-Verlag, Septemer 99 [33] Rajeev Thakur, William Gropp, and Ewing Lusk A Case for Using MPI s Derived Datatypes to Improve I/O Performane In Proeedings of SC98: High Performane Networking and Computing, Novemer 998 [3] Rajeev Thakur, William Gropp, and Ewing Lusk On Implementing MPI-IO Portaly and with High Performane In Proeedings of the th Workshop on I/O in Parallel and Distriuted Systems, pages 3 3 ACM Press, May 999 [35] Rajeev Thakur, Ewing Lusk, and William Gropp Users Guide for ROMIO: A High-Performane, Portale MPI-IO Implementation Tehnial Report ANL/MCS-TM-3, Mathematis and Computer Siene Division, Argonne National Laoratory, Revised Deemer 999 [3] The NCSA HDF home page [37] Brian Tierney, William Johnston, Jason Lee, and Mary Thompson A Data Intensive Distriuted Computing Arhiteture for Grid Appliations Future Generation Computer Systems, To appear [38] Brian Tierney, Jason Lee, Brian Crowley, and Mason Holding A Network-Aware Distriuted Storage Cahe for Data Intensive Environments In Proeedings of the Eighth IEEE International Symposium on High Performane Distriuted Computing, August 999 3