In search of the right way for extreme-scale HPC file system metadata

Transcription

1 ++ In search of the right way for extreme-scale HPC file system metadata Qing Zheng 1, Kai Ren 1, Garth Gibson 1, Bradley W. Settlemyer 2 1 Carnegie MellonUniversity 2 Los AlamosNationalLaboratory [LA-UR ]

2 HPC defined by parallel scientific applications o running on compute nodes with pooled storage for highly parallel I/O App 3 App 2 App 1 App 4 App 5 compute nodes w/ fast interconnect (10,000+) pooled storage nodes (100+) Parallel Data Lab - LANL/HPC_showcase 2

3 App 3 App 2 HPC Checkpointing copying memory from compute nodes to a backend file system memory (1PB) LustreFS storage (80PB 1.45TB/s) App 1 App 4 App 5 compute nodes w/ fast interconnect (10,000+) pooled storage nodes (100+) Parallel Data Lab - LANL/HPC_showcase 3

4 App 3 App 2 HPC Checkpointing copying memory from compute nodes to a parallel file system memory (1PB) LustreFS storage (80PB 1.45TB/s) App 1 App 4 App 5 compute nodes w/ fast interconnect (10,000+) pooled storage nodes (100+) Parallel Data Lab - LANL/HPC_showcase 4

5 HPC Checkpointing two-tier storage for higher bandwidth App 3 App 2 memory (1PB) LustreFS storage (80PB 1.45TB/s) flash burst buffer (4PB 3.3TB/s) App 1 App 4 App 5 compute nodes w/ fast interconnect (10,000+) pooled storage nodes (100+) Parallel Data Lab - LANL/HPC_showcase 5

6 HPC Checkpointing metadata can also become a bottleneck App 3 App 2 memory (1PB) head node LustreFS metadata (5,000 creates/s) App 1 App 4 App 5 compute nodes w/ fast interconnect 320,000 CPU cores 640,000 CPU threads Parallel Data Lab - pooled storage nodes (100+) LANL/HPC_showcase 6

7 HPC Checkpointing file creations are non-trivial in extrema-scale App 3 App 2 memory (1PB) head node 128 s to create all files LustreFS metadata (5,000 creates/s) App 1 App 4 App 5 compute nodes w/ fast interconnect 320,000 CPU cores 640,000 CPU threads Parallel Data Lab - pooled storage nodes (100+) LANL/HPC_showcase 7

8 s s FILE CREATES DATA Parallel Data Lab - LANL/HPC_showcase 8

9 2X CPU cores and 2X storage bandwidth s s FILE CREATES DATA Parallel Data Lab - LANL/HPC_showcase 9

10 Bias for data Lustre File System metadata service data service [a single (or a few) machines] [a large collection of machines] metadata eventually a problem!! Parallel Data Lab - LANL/HPC_showcase 10

11 SCALING-UP 30,000 creates/s [6X faster] Harden Lustre File System Parallel Data Lab - LANL/HPC_showcase 11

12 SCALING-UP 30,000 creates/s [6X faster] expensive w/ limited improvements Parallel Data Lab - LANL/HPC_showcase 12

13 SCALING-OUT Use multiple dedicated machines to serve file system metadata DISTRIBUTED METADATA Parallel Data Lab - LANL/HPC_showcase 13

14 SCALING-OUT / home ROOT src zhengq zhengq garth bws f 2 f 1 src f 1 f 2 f 3 hierarchical directory structure dynamic directory allocation & splitting home garth bws f 3 Parallel Data Lab - LANL/HPC_showcase 14

15 SCALING-OUT / home ROOT src zhengq zhengq garth bws dynamic directory allocation & splitting home requires a large # of dedicated bws garth machines f directory structure 3 src f 1 f 2 f 3 f 2 f 1 Parallel Data Lab - LANL/HPC_showcase 15

16 GOAL Cost-effective highly-parallel metadata Orders of magnitude faster then LustreFS Parallel Data Lab - LANL/HPC_showcase 16

17 ++ Parallel Data Lab - LANL/HPC_showcase 17

18 Traditional Metadata Model / App 1 App 3 App 2 global coordination zhengq garth home bws compute nodes src f 1 f 2 f 3 shared file system view dedicated mediator Parallel Data Lab - LANL/HPC_showcase 18

19 Traditional Metadata Model / App 1 App 3 App 2 global coordination zhengq garth home bws src f 1 f 2 f 3 shared file view compute nodes A single authoritative view of the file system Allows different apps to shared data and dedicated achieve synchronization mediator Parallel Data Lab - LANL/HPC_showcase 19

20 HPC I/O is much simpler INPUT HPC Application OUTPUT CHECKPOINT DATA Parallel Data Lab - LANL/HPC_showcase 20

21 HPC I/O is much simpler INPUT HPC Application OUTPUT CHECKPOINT DATA HPC applications don t need the FS to achieve synchronization Different apps operate on different sets of files Parallel Data Lab - LANL/HPC_showcase 21

22 HPC I/O is much simpler INPUT HPC Application OUTPUT A single unified file system view overkill for HPC applications Global coordination via a dedicated service bad for performance CHECKPOINT DATA HPC applications don t need the FS to achieve synchronization Different apps operate on different sets of files Parallel Data Lab - LANL/HPC_showcase 22

23 Middleware Design Parallel Scientific Applications ++ metadata ops data storage Shared Underlying Storage [parallel data access oblivious of file system semantics] metadata storage Parallel Data Lab - LANL/HPC_showcase 23

24 Serverless Metadata App 1 / data proj s3d metadata App 2 / input proj ocean metadata App 3 / proj mpas repo metadata compute nodes Parallel Data Lab - LANL/HPC_showcase 24

25 Serverless Metadata App 1 / data proj s3d metadata App 2 / input proj ocean metadata App 3 / proj mpas repo metadata No such global namespace managed by a centralized mediator Each app only communicates compute with nodesits private metadata servers Parallel Data Lab - LANL/HPC_showcase 25

26 Serverless Metadata / proj mpas proj / repo s3d proj App 3 data App ocean 1 Fundamental Assumption metadata input metadata App HPC applications don t communicate 2 with each other metadata No such global namespace managed by a centralized mediator Each app only communicates compute with nodesits private metadata servers Parallel Data Lab - LANL/HPC_showcase 26 /

27 App Execution Model [snapshot = static view of a file system] snapshot HPC Application new snapshot SNAPSHOT CATALOG Implemented as database, k/v-store, zookeeper, Parallel Data Lab - LANL/HPC_showcase 27

28 App Execution Model [snapshot = static view of a file system] snapshot HPC Application Keeps new snapshot each app isolated from other running apps SNAPSHOT CATALOG Implemented as database, k/v-store, zookeeper, Truly parallel metadata processing among different apps Parallel Data Lab - LANL/HPC_showcase 28

29 Sharing Data Between Apps [snapshot = static view of a file system] snapshot_0 snapshot_1 SNAPSHOT CATALOG APP 1 HPC workflow Parallel Data Lab - LANL/HPC_showcase 29

30 Sharing Data Between Apps [snapshot = static view of a file system] snapshot_0 snapshot_1 snapshot_2 SNAPSHOT CATALOG APP 1 App 2 HPC workflow Parallel Data Lab - LANL/HPC_showcase 30

31 Sharing Data Between Apps [snapshot = static view of a file system] snapshot_0 snapshot_1 snapshot_2 snapshot_3 SNAPSHOT CATALOG APP 1 App 2 APP 3 HPC workflow Parallel Data Lab - LANL/HPC_showcase 31

32 Sharing Data Between Apps [snapshot = static view of a file system] snapshot_0 snapshot_1 snapshot_2 snapshot_3 SNAPSHOT CATALOG APP 1 App 2 APP 3 Converts online to batched offline sharing HPC workflow Allows apps to decide file visibility and timings for communication Parallel Data Lab - LANL/HPC_showcase 32

33 Sharing Data Between Apps snapshot_0 APP 1 [snapshot = static view of a file system] snapshot_1 snapshot_2 App 2 APP 3 snapshot_3 SNAPSHOT Avoids unnecessary coordination CATALOG synchronously enforced by a centralized metadata service Converts online to batched offline sharing HPC workflow Allows apps to decide file visibility and timings for communication Parallel Data Lab - LANL/HPC_showcase 33

34 Snapshot Merging snapshot_145 snapshot_8 snapshot_54 file system namespace Parallel Data Lab - LANL/HPC_showcase 34

35 Snapshot Merging snapshot_145 snapshot_8 snapshot_54 file system view Conflicts resolved at read path to avoid unnecessary computation Resolution performed by apps instead of a super authority Parallel Data Lab - LANL/HPC_showcase 35

36 App 3 App 2 HPC in 10 years two-tier storage for cost-effective fast data client-funded namespace for highly-parallel metadata capacity tier performance tier App 1 App 4 App 5 compute nodes w/ fast interconnect (10,000+) pooled storage nodes (100+) Parallel Data Lab - LANL/HPC_showcase 36

37 Reference Scaling the File System Control Plane with Client-Funded Metadata Servers (PDSW14) Scaling File System Metadata Performance with Stateless Caching and Bulk Insertion (SC14) Parallel Data Lab - LANL/HPC_showcase 37

38 QUESTIONS