Deploying Server-side File System Monitoring at NERSC

Size: px

Start display at page:

Download "Deploying Server-side File System Monitoring at NERSC"

Corey Hubbard
8 years ago
Views:

1 System Monitoring at Cray Users Group Proceedings May 7, 2009 National Energy Research Scientific Computing Center Lawrence Berkeley National Lab 1.1

2 Contents

3 Monitoring the I/O Subsystem CN OSS fc switch RAID CN OSS MDS Net fc switch RAID 10.0 network /LMT network Liberty Almanack 1.3

4 OSS /usr/lib/cerebro/* cerebro_metric_lmt_oss.so cerebro_metric_lmt_ost.so cerebro_metric_lmt_mds.so cerebro_monitor_lmt.so 1.4

5 LMT OSS /proc/stat /proc/meminfo /proc/fs/lustre/obdfilter/*/ filesfree filestotal kbytesfree kbytestotal numrefs stats uuid 1.5

6 An OSS Tuple Protocol Version Host Name CPU Utilization Memory Utilization 1.0;nid04187; ;

7 Data Values Protocol Version Host Name UUID Bytes Read Bytes Written Kbytes Free Kbytes Used Inodes Free Inodes Used 1.7

8 MDS Operations mysql> select * from OPERATION_INFO; OPERATION_NAME UNITS OPERATION_NAME UNITS req_waittime usec mds_getattr_lock usec req_qdepth reqs mds_close usec req_active reqs mds_reint usec reqbuf_avail bufs mds_readpage usec ost_reply usec mds_connect usec ost_getattr usec mds_disconnect usec ost_setattr usec mds_getstatus usec ost_read bytes mds_statfs usec ost_write bytes mds_pin usec ost_create usec mds_unpin usec ost_destroy usec mds_sync usec ost_get_info usec mds_done_writing usec ost_connect usec mds_set_info usec ost_disconnect usec mds_quotacheck usec ost_punch usec mds_quotactl usec ost_open usec mds_getxattr usec ost_close usec mds_setxattr usec ost_statfs usec ldlm_enqueue usec

9 1.9

10 Four IOR Tests Aggregate rates from :45:00 read rate write rate Data Rate (MB/s) :45 22:48 22:51 22:54 22:57 23:00 23:03 23:06 23:09 23:12 23:15 23:18 Time (PDT) 1.10

11 24 Hours of LMT Data read rate write rate Data Rate (MB/s) :00 10:00 12:00 14:00 16:00 18:00 20:00 22:00 00:00 02:00 04:00 06:00 Time (PDT) 1.11

12 Daily Averages Data Rate (GB/s) Average daily rates Read Write /01 08/01 09/01 10/01 11/01 12/01 01/01 02/01 03/01 Time (PDT) 1.12

13 f λ (k) = λk e λ k! 1.13

14 C(m) = N f λ (int(m/m)) 1.13

15 Poisson : λ = 2 Poisson distribution 100 M lambda = 2, M = 125MB, N = 250M 10 M 1 M count 100 K 10 K 1 K GB 0.4 GB 0.6 GB 0.8 GB 1 GB 1.2 GB 1.4 GB 1.6 GB 1.8 GB 2.0 GB m - The amount of data transferred during 5 second interval. 1.14

16 Poisson : λ = 20 Poisson distribution 100 M lambda = 20, M = 40MB, N = 250M 10 M 1 M count 100 K 10 K 1 K GB 0.4 GB 0.6 GB 0.8 GB 1 GB 1.2 GB 1.4 GB 1.6 GB 1.8 GB 2.0 GB m - The amount of data transferred during 5 second interval. 1.15

17 250 M LMT Observations of LMT observed rates read write 10 M 1 M Count 100 K 10 K 1 K MB 1.16

18 Two weeks of recent observations of LMT observed rates 1 M read write 100 K Count 10 K 1 K MB 1.17

19 I would like to acknowledge and thank: Al Chu The author of. Herb Wartens The author of the Lustre Monitorining Tool plug-ins. Both work at Lawrence Livermore National Lab, which supported the development of these tools. Both were very generous with their time as I deployed the software on Franklin. 1.18

20 The software is available from: Both applications are open source and available from Sourceforge. http: //sourceforge.net/projects/cerebro LMT If you would like hints and encouragement with getting this software deployed, contact me: If you get results from your deployment that you would like to share, please do so. 1.19

Deploying Server-side File System Monitoring at NERSC

Deploying Server-side File System Monitoring at NERSC Andrew Uselton NERSC, Lawrence Berkeley National Laboratory Berkeley, CA 94720 May 7, 2009 Abstract The Franklin Cray XT4 at the NERSC center was equipped