perfsonar Multi-Domain Monitoring Service Deployment and Support: The LHC-OPN Use Case

Transcription

1 perfsonar Multi-Domain Monitoring Service Deployment and Support: The LHC-OPN Use Case Fausto Vetter, Domenico Vicinanza DANTE TNC 2010, Vilnius, 2 June 2010

2 Agenda Large Hadron Collider Optical Private Network (LHC-OPN) Multi-Domain monitoring challenge: perfsonar GÉANT Multi Domain Monitoring Service GÉANT Service Desk The LHCOPN case: Deployment Support Monitoring

3 LHC-OPN Large Hadron Collider Optical Private Network (LHC-OPN): Dedicated network to support LHC experiment Large amount of data in a grid environment Network architecture is organized in Tiers 1 Tier0, 11 Tier1, 140+ Tier2 Primary users are researchers around different institutes Requirement: Large amount of data being exchanged Strategy: Keep traffic segregated from Internet Solution: Optical Private Network (LHC-OPN) among Tier 0/1s Challenge: monitoring effectively in a multi-domain environment

4 LHC-OPN Topology Dual-star topology 10 Gb/s links Cross-border fibers resiliency Multi-domain LHC-OPN Topology

5 Monitoring the LHC-OPN: The requirements Focus of monitoring: Network Layer (IP) Physical Layer (Links) Regular Active Point-to-Point Measurements One-Way Delay, One-Way Delay Variation, Achievable Bandwidth, Historical Traceroute Changes Regular Passive Point-to-Point Measurements Utilization, Input Errors, Packet Discards End-to-End link monitoring Managed service Unified view of the network status and information across all sites Homogeneous installations and centralized operations

6 Monitoring the LHC-OPN: The solution - perfsonar The Tool: perfsonar GÉANT multi-domain monitoring (MDM) tool Based on Open Grid Forum Standard Monitoring Protocol Customized, fully managed and supported for LHCOPN Objective: Identify network problems across multiple domains Correctly, efficiently and quickly Allowing proactive actions Strategy: perform network monitoring actions in different network domains make the information available thanks to a common protocol cross-domain monitoring capability access network performance metrics from across multiple domains

7 perfsonar as unifying layer across domains perfsonar Services Domain 1 Domain 2 perfsonar Each domain has its own local monitoring Domain 3 Domain 4 perfsonar UI (visualization) Scripts/API

8 Monitoring the LHC-OPN: The benefits Effective monitoring across the several LHC-OPN domains perfsonar enables multi-domain monitoring Problems can be tracked through the participating domains from a single interface proactively solving problems across domains Effective, distributed monitoring can identify problems even before users suffer them through a customized web portal Monitoring portal designed according to LHCOPN needs Easy to integrate into involved NOCs workflows Less disruptions and faster recovery Easy to take and foster collaborative efforts Fully managed solution: Low overhead for the Tier0/1 network operators involved Configuration, Operation and Support carried out by GÉANT SD

9 perfsonar at LHC-OPN 12 sites (1 Tier0, CERN, and 11 Tier1) involved Several Countries around Europe, Asia and America Access to network measurements data from multiple network domains Customized version of perfsonar MDM service for Tier0/1 sites (main contributor to LHCOPN operations) Customized visualization tool accessed: Dedicated web portal Specific weather maps and further diagnosis tools to visualize measurements results Monitoring tools, hardware and operating system packed in monitoring boxes, To be easily deployed at any location Remotely accessible by the service desk for operations and support

10 GÉANT MDM Service Design for LHCOPN Two servers installed in each site (Tier0 and Tier1) : Server 1 (HADES): one way delay, one way delay variation, achievable bandwidth, historical traceroute changes Server 2 (MDM): regular passive measurements carried out for collecting interface utilisation, input error and packet discards statistics from the sites network elements Each site provided: Gigabit port on the border router Switch Time Sources DNS Servers

11 perfsonar MDM in LHC-OPN

12 The result as displayed by the LHC-OPN Portal

13 Weather-map E2Emon Link Status

14 Weather-map E2Emon Link Status

15 GÉANT Application Service Desk Deployment carried out by the GÉANT Application Service Desk Dedicated Staff Manage the Users Relationship Responsible for Incident Management Interact with Problem Management/Product Management to Improve Products Acts as a Single Point of Contact: Usage of Products Deployment of Products Debugging Issues on Products Focus on transition and operations of the services delivered

16 GÉANT MDM Service Transition Service deployment: two workflows Server 1: OS and Software installed and configured by a GÉANT partner Server 2: OS and Software entirely installed and configured remotely Phase details: Pre-Shipment: gather information about deployment details Pre-Shipment Form Shipment: servers shipment to GÉANT partner and customer Receive Boxes: customer and configuration partner receives boxes Preparation: Pre-Deployment Form Third party supplier prepares servers Physical Installation Deployment: software installation Configuration: service configuration Validation

17 MDM Service Deployment Agenda

18 perfsonar services monitoring Service Monitoring Infrastructure (based on Nagios+Cacti): Customised set of testing scripts and health checks 35 Checks per server, covering hardware, software and services Automatic notification, detailed history Three layer monitoring: Hardware layer: CPU, MEM, disk space, network interfaces, TCP/UDP traffic, temperature Resource layer: login attempts, Tomcat RRT, exist RTT, MySQL, NTP Service layer: perfsonar services availability and performance Additional tools: Syslog server (with MySQL support) security log auditing (with automatic report tools)

19 GÉANT MDM Service Operations: the monitoring interfaces

20 GÉANT MDM Service Operations: incident management procedures Well defined procedures for Incident Management: Incident Management Third party supplier involved

21 Conclusions GÉANT Application Service Desk: Effective single point of contact in complex deployments LHC-OPN use case: great opportunity for service & support infrastructure Reasons for a successful deployment: Preparation phase is crucial Adequate tools for event and incident management Customer collaboration was the main player on the deployment. Continuous service improvement Periodic meetings with involved parties Quality audits about the deployment

22 Final Remarks Thanks to: perfsonar community GÉANT partners DANTE perfsonar development team CERN and its partners Thanks for your attention Any questions and/or comments?

23 E2E Monitoring

24 HADES