THE EXPAND PARALLEL FILE SYSTEM A FILE SYSTEM FOR CLUSTER AND GRID COMPUTING. José Daniel García Sánchez ARCOS Group University Carlos III of Madrid

Transcription

1 THE EXPAND PARALLEL FILE SYSTEM A FILE SYSTEM FOR CLUSTER AND GRID COMPUTING José Daniel García Sánchez ARCOS Group University Carlos III of Madrid

2 Contents 2 The ARCOS Group. Expand motivation. Expand design. Expand evaluation. Conclusions. Ongoing Work.

3 University Carlos III of Madrid 3 Founded in 1989 Three faculties: Faculty of Social Sciences and Law. Faculty of Humanities, Documentation and Communication. Higher Technical School.

4 The ARCOS Group 4 The Computer Architecture, Communications and Systems Group is part of the Department of Computer Science. 20 full time members 9 PhD s (2 full professors + 4 associate professors + 3 visiting professors). 11 PhD students

5 Research lines 5 Data management on Grid environments. Parallel file systems. Optimization of irregular applications. OS for Wireless Sensor Networks. Real-time systems.

6 Some products 6 Expand: A parallel file system for cluster and grid environment. WinPFS: Windows Parallel File System. MiMPI: MPI implementation for heterogeneous cluster environments

7 Contents 7 The ARCOS Group. Expand motivation. Expand design. Expand evaluation. Conclusions. Ongoing Work.

8 June-93 April-94 February-95 December-95 October-96 August-97 June-98 April-99 February-00 December-00 October-01 August-02 June-03 April-04 February-05 December-05 October-06 8 Trends in the supercomputing environment Clusters in top500.org 75 % of supercomputers in top500 are clusters.

9 Trends in the supercomputing 9 environment Number of transistors per chip still doubling every 1.5 years. Does not mean doubling frequency, performance, More space more cores per chip.

10 Trends in the supercomputing 10 environment Grid Computing: Interconnecting supercomputers to aggregate geographically distributed resources. Applications are deployed somewhere in the grid. Applications read input data and produce output data.

11 Trends in the supercomputing 11 environment Clusters becoming the preferred option for supercomputing. Processors with increasing capacity. Grid computing using clusters as a building block. I/O will remain as a major bottleneck.

12 Storage system typical architecture 12 Clients Communication network Storage network I/O server

13 Aggregated bandwidth (MB/s) Problems with storage architectures Clients NAS

14 Solution: Parallelism 14 Parallel applications Parallel computers Exploit parallelism at multiple layers Parallel file systems Parallel devices

15 Parallel File System Architecture 15 Computing node Computing node Computing node Apps Client Communication Network I/O Server I/O Server I/O Server File 1 File 2

16 Parallel File System Architecture 16 Computing node Computing node Computing node Apps Client Communication Network GPFS I/O Server I/O Server I/O Server Storage Network File 1 File 2

17 Parallel File System Architecture 17 GPFS Computing node Computing node Computing node Client Apps Storage Network File 1 File 2

18 18 Expand Parallel File System: Motivation Provide a high performance storage system by using standard protocols and servers. Easy integration of heterogeneous systems. Reuse and aggregation of existing resources. Parallel data access.

19 Why Expand? 19 A standard data server already includes almost all the needed functionality. Reuse. Standard protocols and servers make resources universally available. Easy to deploy. Independence of the underlying storage infrastructure. Portability.

20 Objective 20 Offer a new approach to build PFS for cluster and grid environments by using standard data servers. Advantages: No server change needed. Operations at client side. Independence of client and server OS s. Operations through standard protocols. Simplified PFS construction. Take advantage of already implemented server high performance mechanisms. Allows mixing servers with different platforms and OS s. Easy installation and configuration.

21 Contents 21 The ARCOS Group. Expand motivation. Expand design. Expand evaluation. Conclusions. Ongoing Work.

22 Computing node Architecture 22 POSIX MPI-IO Expand NFS GridFTP RNS-WS Local Parallel access Server protocol Distributed partition File 1 File 2

23 File structure 23 Expand file: Metadata sub-file. Several data sub-files. Data distributed across several servers. File-to-server flexible mapping policy. Sub-files Expand file Server 1 Server 2 Server

24 Directory structure 24 Logical View Mapping Physical View Dir1 /Expand Dir2 Dir3 Server 1 Server 2 Server 3 /export1 /export2 /export3 Dir1 Dir2 Dir3 Dir1 Dir2 Dir3 Dir1 Dir2 Dir3 Dir4 Dir4 Dir4 Dir4 FileA FileA FileA FileA

25 Metadata management 25 Metadata distributed management. Two levels. Without locking. No metadata manager. Metadata distributed across servers. Master node. Hashing on name. Load balancing. Expand file block Server 1 Server 2 Server metadata 2 5 8

26 Metadata management 26 Metadata distributed management. Two levels. Without locking. No metadata manager. Metadata distributed across servers. Master node. Hashing on name. Load balancing. Expand file Server 1 Server 2 Server 3 Metadata FileA Metadata FileC block Metadata FileD Metadata FileF Metadata FileB Metadata FileE

27 Parallel access 27 read(fd, buffer,count) buffer Data blocks Expand Threads Server 1 Server 2 Server

28 MPI-IO interface using ROMIO 28 MPI-IO ADIO Unix NFS PFVS Expand IBM PIOFS SGI XFS Distributed partition

29 Dynamic partition reconfiguration 29 Server 1 Server 2 Server 3 Server Instantaneous. Deferred. hash(file) = server 3

30 Dynamic partition reconfiguration 30 Server 1 Server 2 Server 3 Server

31 Expand cluster versions 31 Linux/NFS Java/NFS NFS Server NFS Server NFS Server NFS Server Distributed partition Distributed partition

32 Contents 32 The ARCOS Group. Expand motivation. Expand design. Expand adaptation for Grid Computing. Expand evaluation. Conclusions. Ongoing Work.

33 Requirements for a Grid File System 33 Hierarchical logical space name. Resource Namespace Service (RNS). Standard access interface. POSIX and MPI-IO. Data access. GridFTP. Security. Grid Security Infrastructure (GSI). Performance optimization and improvement. Paralle I/O.

34 Computing node Adapting Expand to Grid environments 34 POSIX MPI-IO Computing node Expand Computing node NFS GridFTP RNS-WS Local RNS NFS NFS NFS Internet + GSI GridFTP GridFTP GridFTP GridFTP Distributed partition Site 1 Site 2 Site 3 Site 4 Distributed partition

35 Contents 35 The ARCOS Group. Expand motivation. Expand design. Expand evaluation. Conclusions. Ongoing Work.

36 Evaluation 36 How does Expand behaves compared to other existing solutions? Cluster PFVS. GPFS. Grid Globus Grid services.

37 Cluster environment 37 8 biprocessors (Pentium VI, 3.2 GHz). 2 GB RAM per node. Network: Gigabit ethernet. Expand. PVFS. GPFS.

38 Cluster benchmarnking 38 High performance. Parallel access to a file: IOR benchmark. FLASH I/O benchmark. Metadata operations. High throughput. Image processing. Dynamic partition reconfiguration.

39 High performance: 39 Parallel access to a file Parallel program (IOR) making interleaved writes and reads to a single file with different access sizes. MPI-IO interface Process 1 Process 2 File

40 Bandwidth (MB/s) 40 High performance: Parallel access to a file for writing processes (writing) XPN PVFS GPFS 0 access size

41 Bandwidth (MB/s) 41 High performance: Parallel access to a file for reading processes (reading) XPN PVFS GPFS 0 access size

42 Bandwidth (MB/s) 42 High performance: Parallel access to a file for writing 140 Parallel access writing (8 KB) XPN PVFS GPFS Number of processes

43 Bandwidth (MB/s) 43 High performance. Parallel access to a file for writing XPN PVFS GPFS Parallel access writing (256 KB) Number of processes

44 High Performance: FLASH-IO 44 FLASH is a parallel application simulating thermonuclear flashes. FLASH-IO simulates I/O operations performed by FLASH. Data size is proportional to number of running processes. 1 process MB 16 processes 1.16 GB

45 Bandwidth (MB/s) High Performance: FLASH-IO XPN PVFS GPFS Benchmark FLASH-IO Number of processes

46 Files/second Metadata: Creating empty files File creation (empty files)) process 4 processes XPN PVFS GPFS Filesystem

47 Files/second Metadata: Creating small files File creation (small files) process 4 processes XPN PVFS GPFS Filesystem

48 High throughput 48 Parallel application processing a set of 256 images. Each process works on a subset of images independently. No concurrent access to a file. Sizes: Image file 5 MB. Full dataset 2.5 GB. The process applies to each image file a fixed bitmask to generate a new image file.

49 Time (s) 49 High throughput: Image processing in C Image processing (C application) XPN PVFS GPFS Number of processes

50 Time (s) 50 High throughput: Image processing in Java 450 Image processing (Java application) XPN PVFS GPFS Number of processes

51 Bandwidth (MB/s) Reconstruction time (min) 51 Dynamic partition reconfiguration: Adding new nodes Bandwidth (MB/s) Reconstruction time (min) Reconstruction Model 0

52 Grid evaluation environment 52 Evaluation for high throughput. Perform 500 jobs. Each job selects randomly a file (among 200) to access. File size is 200 MB.

53 Testbed environment 53 Intel Pentium GHz GridFTP Intel Xeon Doble Procesador 2.4GHz Intel(R) Pentium(R) 4 CPU 2.40GHz GridFTP GridFTP GridFTP Intel Pentium 4 CPU 2.80GHz AMD Athlon

54 Evaluated scenarios 54 Typical Grid Completely transfer file to local node. Processing starts after transfer finishes. Globus services for transfer. globus-url-copy Expand Direct remote access to file. No previous transfer to node needed! José Daniel García Sánchez ARCOS Group University Carlos III of Madrid

55 Model 1 / Scenario server Complete transfer to local node. Application access local copy. GridFTP Files José Daniel García Sánchez ARCOS Group University Carlos III of Madrid

56 Model 2 / Scenario servers. Distributed files. Each server stores 50 files. Complete transfer to local node. Application accesses local copy. GridFTP GridFTP GridFTP GridFTP Files José Daniel García Sánchez ARCOS Group University Carlos III of Madrid

57 Computing node Scenario 2 (Expand) 57 Expand with 1, 2 and 4 servers. POSIX MPI-IO Expand NFS GridFTP RNS-WS Local Local node accesses remotely needed data. No previous transfer needed. RNS Internet + GSI GridFTP GridFTP GridFTP GridFTP Site 1 Site 2 Site 3 Site 4 José Daniel García Sánchez ARCOS Group University Carlos III of Madrid Distributed partition

58 Time (min) Grid Evaluation site 4 sites (distributed files) Expand (1 server) Expand (2 servers) Expand (4 servers) Model

59 Contents 59 The ARCOS Group. Expand motivation. Expand design. Expand evaluation. Conclusions. Ongoing Work.

60 Conclusions 60 It is feasible to build parallel file system by using standard protocols and servers. Our solution is easily adaptable to different environments/situations (cluster and grid are examples). Performance results are comparable to other solutions (even comercial).

61 Contents 61 The ARCOS Group. Expand motivation. Expand design. Expand evaluation. Conclusions. Ongoing Work.

62 Ongoing work 62 Add new protocols (e.g. Web Services) Evaluation in large clusters and grid environments. Use Expand to improve performance when accessing replicated data.

63 Ongoing work 63 Use Expand as intermediate file system in large clusters. Apps Expand Cluster File System (PFVS, GPFS, etc.) Compute nodes Parallel access Network I/O nodes

64 THE EXPAND PARALLEL FILE SYSTEM A FILE SYSTEM FOR CLUSTER AND GRID COMPUTING José Daniel García Sánchez ARCOS Group University Carlos III of Madrid