Network monitoring with perfsonar. Duncan Rand Imperial College London

Transcription

1 Network monitoring with perfsonar Duncan Rand Imperial College London

2 A little history: Gridmon Cast your minds back to GridPP16 at QMUL about 6 years ago; we saw a not too dissimilar network monitoring status report by Robin GridPP Gridmon network monitoring system developed at Daresbury GridMonV1 ran in full mesh between 12 original GridPP sites Functionality: iperf: throughput ping: packet-loss, RTT udpmon: UDP throughput and packet loss 2

3 Gridmon Version 2 of Gridmon software proposed to run based on Monarch model of envisaged LHC data flow from Tier-1 at RAL to Tier-2s star from RAL to all ~20 Tier-2s because of concerns about contention sites had less bandwidth in those days a full mesh only within distributed Tier-2s (i.e. Scotgrid, NorthGrid etc) Status of site roll-out similar to now, i.e. at the tail-end URL still works: 3

4 4

5 LHC data: CMS Switching to the present, where is LHC VO data that s being copied to Tier-2s actually coming from? CMS Tier-2s have always got data from sites apart from their local Tier-1 For example: snapshot of data flow into Imperial College taken over last week 46 TB dataset transfer mostly from Fermilab 5

6 CMS Over last 6 months UK CMS Tier-2s have got only 14% of their data from UK and about 28% comes from one site: Fermilab worth monitoring link 6

7 ATLAS Similar 48 hour picture for typical ATLAS site: QMUL 35% (11TB) from BNL 7

8 ATLAS ATLAS UK Tier-2s over last year get 54% of data from outside the UK RAL-LCG2 still site with the largest overall proportion (34%) So we clearly need an international network performance monitoring and diagnostic system.. 8

9 perfsonar PerfSONAR stands for PERFormance Service Oriented Network monitoring Architecture perfsonar is an infrastructure for network performance monitoring, making it easier to solve end-to-end performance problems on paths crossing several networks A collaboration between ESnet (the US DoE Energy Sciences network) GÉANT2 (Europe s high-bandwidth academic internet) Internet2 (advanced networking consortium) RNP (Rede Nacional de Ensino e Pesquisa: Brazil s academic internet backbone) Also a protocol and several interoperable software packages Two flavours: perfsonar-mdm (multi-domain monitoring, Geant flavour) perfsonar-ps (US version) 9

10 UK deployment UK s DRI money fortuitously timed as it enabled GridPP sites to go out and buy suitable perfsonar monitoring hosts en masse Some sites bought two hosts, one for bandwidth monitoring and one for latency/packet loss monitoring, others only bought one RAL Tier-1 pioneered installation followed by Oxford and QMUL Wiki set up to help new sites with installation Currently 16 UK sites including Tier-1 running perfsonar boxes Last remaining sites are those with very little (if any) GridPP man-power 10

11 Sites join GridPP community and thereby appear for other sites to schedule tests to Full mesh of bandwidth and latency tests between all UK sites Typically measure bandwidth using 30s iperf test every 4 hours and one-way latency using NTP synchronised clock and sending a set of packets to each destination site 11

12 Bandwidth throughput results 12

13 13

14 Packet loss/ latency results 14

15 15

16 BNL dashboard When the US deployed perfsonar they found it difficult to get an overview; they were continuously having to monitor each site s perfsonar rate graphs So they developed BNL dashboard Matrices for bandwidth and packet loss between sites all on one page USCMS, USATLAS, Italy, Canada, France, LHCOPN, LHCONE (selected sites) clouds UK sites have been added to UK cloud as and when they came online now form the largest country cloud (16 sites) Inter-cloud groups to monitor international performance 16

17 BNL dashboard 17

18 How can perfsonar help us: ATLAS T2D ATLAS now have standard Tier-2 and higher performance Tier-2 known as T2D T2Ds have demonstrated good network connectivity and ATLAS ships out more data directly to them; uses them to export data locally within cloud to other T2s Requirements are to be able to copy to and from 9/12 Tier1s at >5MB/s reliably over a sustained period (about a month) A number of UK ATLAS Tier-2 have been T2Ds for a while (Glasgow, Manchester QMUL, Lancaster, Oxford, ECDF) Others more recently promoted: RHUL and Cambridge Up until recently once promoted apparently unlikely to get demoted Now ADC are hinting at demotions; ECDF recently put on negative watch A real networking problem to solve Have set up an ATLAS-UK inter-cloud group to help with such current issues 18

19 Other ATLAS sites to UK inter-cloud 19

20 Current issues I: UK Tier-2s to FZK/KIT Many UK Tier-2 sites have been getting poor file transfer rates into KIT for a while perfsonar results also poor; diagnosed to KIT firewall problem Manchester and Oxford were moved outside of firewall dramatic change KIT see LHCONE as the solution to this problem GGUS-Ticket-ID: #

21 Current issues II: Taiwan to QMUL On 4 th September QMUL upgraded their WAN to 10 Gbps Saw an increased in bandwidth to other UK sites as expected But after a temporary increase there was a dramatic decrease in ATLAS sonar file transfer rates from ASGC traceroute and perfsonar reverse traceroute commands suggest possible differing routes to and from Taiwan GGUS-Ticket-ID: #

22 Current issues II: Taiwan to QMUL On 4 th September QMUL upgraded their WAN to 10 Gbps Saw an increased in bandwidth to other UK sites as expected But after a temporary increase there was a dramatic decrease in ATLAS sonar file transfer rates from ASGC traceroute and perfsonar reverse traceroute commands suggest possible differing routes to and from Taiwan GGUS-Ticket-ID: #

23 Current issues II: Taiwan to QMUL On 4 th September QMUL upgraded their WAN to 10 Gbps Saw an increased in bandwidth to other UK sites as expected But after a temporary increase there was a dramatic decrease in ATLAS sonar file transfer rates from ASGC traceroute and perfsonar reverse traceroute commands suggest possible differing routes to and from Taiwan Solved! GGUS-Ticket-ID: #

24 Current issues III: Oxford to BNL Since April Oxford gets very poor results copying ATLAS sonar files to BNL but throughput to BNL is OK (0.8 Gbps) as is Oxford to AGLT2 (since June) traceroute indicates the same routing for other UK sites to BNL GGUS-Ticket-ID: #

25 AGLT2 MWT2 BNL 25

26 Current issues IV: Birmingham to BNL Similar problem copying from BHAM to BNL Poor FTS rates but OK to MWT2 and good perfsonar throughput rates What about reverse traceroute? GGUS-Ticket-ID: #

27 X X X X 27

28 CMS use of perfsonar CMS is developing the way in which it intends to use perfsonar USCMS + Brazil have a cloud on the BNL dashboard UK CMS sites in UK cloud form our national CMS cloud RALPP is part of prototype inter-cloud group with Nebraska, Florida, DESY, Beijing, SPRACE, Bari, TIFR, SINP, & Taiwan Purpose: testing network connections between widely separated CMS computing sites (T2s), a few per geographic region 28

29 Conclusion Clearly need international standard network monitoring to be able to monitor links and actively diagnose them when problems occur perfsonar, together with timely DRI hardware, provides this capability UK is now in a strong position: most (16) of the UK Tier-2s have now deployed their perfsonar hosts and have been added to the BNL dashboard perfsonar is already being used to assist in diagnosing network issues Plenty to do in terms of understanding how best to use the diagnostic tool to optimise and debug links between sites UK needs to keep up to date with developments associated with LHCONE Thanks to Tom Wlodek (BNL), Shawn McKee and Jason Zurawski (Internet2) for their help 29