Proposal for a perfsonar Multi Domain Monitoring Service for LHCOPN

Transcription

1 Proposal for a perfsonar Multi Domain Monitoring Service for LHCOPN Service Specification Version 10

2 Proposal for a perfsonar Multi Domain Monitoring Service for LHCOPN Document History Product/Version Number: Proposal for a perfsonar Multi Domain Monitoring Service for LHCOPN 10 Date of Release: 29/11/07 Contract Number: Instrument type: Integrated Infrastructure Initiative (I3) Activity: SA3 Work Item: WI-15 Dissemination Level RE (Restricted) Lead Partner Document Code GN v10 Authors: Loukik Kudarimoti

3 Table of Contents 1 Summary 2 2 Service Overview 3 3 LHCOPN Monitoring Requirements 7 4 Managed Service specification Appliance specification Deployment Location Site Responsibilities Security perfsonar MDM Service in beta 25 5 Support 26 6 Extending the MDM Service Measurement footprint Additional Network Metrics and Software 27 7 Conclusions 28 8 References 29 Appendix A Hardware Specification 30 A.1 Hardware configuration 30 Appendix B Software Description 34 B.1 Measurement Tools 34 B.2 perfsonar Software 37 Appendix C Hardware - Software - Metrics mapping 42 Proposal for a perfsonar Multi Domain Monitoring Service for LHCOPN 10 i

4 1 Summary The LHC project is designing and implementing an Optical Private Network (OPN) dedicated to support the LHC experiments. As with the operation of any network, monitoring plays a critical role. This document draws up a specification for a monitoring service that can fulfil the monitoring requirements of LHCOPN. The monitoring service, named as the perfsonar Multi Domain Monitoring Service, is delivered by using products designed and developed by an international consortium of backbone network operators. This consortium, called perfsonar, has developed products that are highly advanced in their capabilities to measure and display network metrics. Users from various organisations across the world can gain seamless access to measurement data being gathered by any network with the help of these products. The primary users of the perfsonar Multi Domain monitoring service are the Tier-0 and Tier- 1 sites. The primary users of the monitoring data and measurement capabilities made accessible by the supported perfsonar products are the Network Operations Centre responsible for the IP operations of the LHCOPN, the End-to-End Co-ordination Unit (E2ECU) and the Tier-0 and Tier-1 personnel. Other users can also make use of the monitoring data and the measurement capabilities. The perfsonar Multi Domain Monitoring service has been successfully rolled out to six backbone networks across Europe, delivered with the help of a dedicated Service Desk setup at DANTE. This service provides a complete solution that includes support for the monitoring software, the operating system and the required hardware that are needed to provide reliable and precise monitoring of the network. It can be readily deployed at the Tier-0 and Tier-1 sites across the world. This document specifies the service by defining its scope and describing the products that will be provided as part of the service. It lists the requirements expected from a site where it will be deployed and very importantly, it discusses the ideal locations for these products to be deployed within the various sites. Deployment of hardware and operating system at the Tier-0 and Tier-1 sites raises some security concerns that are discussed in this document along with some information on the measures taken by the service to address such concerns. To summarize, the objectives of the LHCOPN, its monitoring requirements and the geographically distributed users provide a challenging multi domain environment in which the network has to be operated and monitored. The combination of perfsonar products and the perfsonar Multi Domain Monitoring service along with many years of backbone monitoring experience demonstrate how these requirements and challenges can be met. Page 2

5 2 Service Overview The perfsonar Multi Domain Monitoring (MDM) service provides users with access to network measurement data from multiple network domains. Users can collect monitoring data from all domains where the service is deployed in order to visualise network characteristics and troubleshoot related issues. For the specific monitoring of the LHC Optical Private Network (LHCOPN), DANTE, CERN and the 11 Tier-1 sites are collaborating on a customised version of the perfsonar MDM service. This customised service will be used to monitor the IP and circuit operations of the LHCOPN. Probes will be placed at the 12 nodes (the 11 Tier-1 sites and CERN) in order to monitor the interconnections between these nodes. The perfsonar MDM service desk at DANTE will operate this service. Users of this service and of the collected measurement data will be the LHCOPN IP Coordination Unit (LIPCU), the End-to-End Coordination Unit (E2ECU) and the Tier-1 sites who are the contributors to the LHCOPN operations. As part of a public trial phase, a similar service has already been rolled out to 6 network domains in Europe with 5 further networks to be introduced in the first quarter of The GÉANT2 project will extend this trial to LHCOPN monitoring. Similarly, ESnet and Internet2 are distributing compatible perfsonar measurement systems to their backbones and to various customer locations. Features The MDM service s measurement tools and the hardware and operating system they require are packaged in an appliance that can be deployed at any location. For LHCOPN it will be deployed close to the border router(s) of the Tier-0/1 sites. Active and passive measurement techniques will be used to monitor the network connectivity between Tier-0 and Tier-1 sites. Regular active measurements will probe the network connections in order to determine: Page 3

6 one-way delay and delay variation TCP achievable bandwidth ICMP-based round-trip time traceroute information to monitor routing path changes Regular passive measurements will be carried out to collect utilisation, input error and packet discards statistics from the sites network elements. LHCOPN-specific network weathermaps and diagnosis tools will be provided to visualise the measurement results. The service desk will use these data to generate monthly service reports. The monitoring infrastructure will be operated in a secure mode with well-defined responsibilities. The service specifications provide an overview of risks assessment. Network configurations and router firewalls play an important role in the mitigation of these risks. Therefore, all Tier-0/1 sites are required to meet specific responsibilities in order to maximise network security and reliability. Benefits of a managed service The service will provide up-to-date information about network health and assist operational teams in diagnosing network-related issues. Deploying the monitoring infrastructure at the administrative boundaries (the Tier-0/1 border routers) establishes demarcation points that help distinguish network issues in the LHCOPN from those at the sites. The advantage of a centrally managed service is that it provides a common view of the network status with low overheads for the Tier-0/1 network operators involved. The service provides reports based on the monitored data that can be used to analyse network statistics (for example, usage or errors) and performance bottlenecks. Reports can also be used to monitor adherence of network providers to Service Level Agreements and to detect usage trends. Deployment of identical tools and the use of a common service in all the Tier-0/1 sites will provide a unified network information view across all sites. The tools can be used on a day-today basis as part of normal site operations as well as for network engineering and capacity planning. The Tier-0/1 sites can also use the tools to demonstrate the quality of the wide-area networking service they deliver to the LCG project using the LHCOPN. Service desk responsibilities The perfsonar MDM service desk at DANTE will: Manage the measurement tools, data archives and visualisation tools. Manage the hardware and operating systems in order to ensure the reliability of the measurement tools and the data. Page 4

7 Ensure the integrity of the monitoring infrastructure, availability of the measured data, data quality and its archival. Site responsibilities All sites are required to contribute the following components to the MDM service: a terminal server dedicated IP interface on the border router(s) a PSTN/ISDN line for out-of-band access a Gigabit Ethernet switch a GPS antenna a protected power source rack space These contributions are further detailed in the service specifications set by the service desk. In addition, sites also need to assign local administrators whose responsibilities include: Securing the deployed measurement infrastructure by controlling access for third party support personnel (when required by the service desk). Carrying out the initial hardware installation and assisting the remote software installation. Updating the service desk with site network topology and configuration changes. Providing End-to-End monitoring (E2Emon) status information about circuits ending at the site. Securing network access to the infrastructure. The appliances must be connected to a dedicated IP interface on the border router(s). At each site, the appliances, the terminal server and the dedicated IP interface require a dedicated subnet with 11 public IP addresses. These IP addresses must be within the LHCOPN address range, so that measurements are routed via the LHCOPN circuits. They also must be reachable by the DANTE service desk and the operation coordinators (LIPCU and E2ECU). Traffic flows on the dedicated router interfaces must be restricted to: measurement activity between the appliances at the 12 sites, management access for the perfsonar MDM service desk which is outside the LHCOPN, and monitoring data access for users of measurement data. These users are operation coordinators (LIPCU and E2ECU) and the operators at the Tier-0/1 sites. In order to collect statistics from the border router interfaces which connect the site to the LHCOPN, the appliances must be allowed to poll the IP interfaces using read-only SNMP access. Page 5

8 This document provides a detailed specification for the perfsonar Multi-Domain Monitoring. A summary of the LHCOPN monitoring requirements that were discussed recently is in chapter 3. Detailed specification of the service begins with chapter 4. Section 4.1 provides an overview of the appliance. Section 4.2 suggests ideal locations within the Tier-0 and Tier-1 sites where these appliances can be deployed. Site responsibilities mentioned above are detailed in section 4.3. Section 4.4 is discusses how these appliances and the monitoring service can work with the security policies of the sites. It also summarizes the risks that have been identified, the mitigation of these risks as well as the security controls and contingency plans that have been defined. An overview of the support provided by the service desk is in section 5. The service specification ends with chapter 6 describing the potential for future service extension and the appendices providing specifications for the hardware that the sites have to deploy to contribute to the monitoring service. Technical details for the hardware and software used for the appliance and the software that will be installed on these appliances are also included in the appendices. Page 6

9 3 LHCOPN Monitoring Requirements The table below provides a summary of the network metrics and features that are required to be monitored for the LHCOPN. The details of these requirements, which have been proposed and discussed by the LHCOPN group, are available in [1]. Network Metrics Description Features 1 Delay measurements between Tier-0 and Tier-1 sites 2 Achievable Bandwidth measurements between Tier-0 and Tier-1 sites Regularly measure one way delay and jitter experienced during high volume data transfers Regularly measure bandwidth achievable on the network using the TCP protocol Periodic and ondemand scheduling of measurements Archive measurement results for trend analysis Visualise measurement results Raise alarms when defined thresholds are crossed Periodic and ondemand scheduling of measurements Archive measurement results for trend analysis Visualise measurement results Raise alarms when defined thresholds are crossed Page 7

10 3 Status of circuits (paths) connecting Tier- 1 sites to Tier-0 4 IP interface statistics for IP connectivity between Tier-0 and Tier-1 sites 5 Traceroute information Real time status of circuits indicating at least whether the circuits are up or down Regularly measure IP link statistics such as link capacity, link utilisation, input errors, and output drops Check the routing path using traceroute Measure status Archive status for trend analysis Visualise status of all circuits Raise alarms when status changes Measure metrics various Archive measurement results for trend analysis Visualise measurement results Raise alarms when defined thresholds are crossed Archive results for analysis Visualise measurement results Raise alarms when path is outside the LHCOPN or when path is assymetric 6 Packet Loss and Reordering Actively measure packet loss and reordering (in addition to passive interface statistics collected for IP interface drops and discards) Archive results for analysis Visualise measurement results Raise alarms when defined thresholds are crossed Page 8

11 4 Managed Service specification 4.1 Appliance specification Hardware In order to deploy perfsonar software, hardware with recommended configuration is necessary. For this purpose, the following hardware will be deployed at Tier-0 and the 11 Tier-1 sites Two servers provided by Sun Microsystems Another server provided by Bee Systems [10] The complete hardware specifications for the above items are available in 8Appendix A Operating System The Operating system installed on the servers is Red Hat Enterprise Linux 5. The Operating System will be updated with the latest patches available at the time of deployment. This Linux Operating System installed on the servers is of enterprise quality supplied and supported by the Red Hat Corporation. If any security vulnerabilities are found, the Red Hat Corporation, as per the RHEL 5 licensing agreements, promises to fix such issues and prepare necessary patches and software upgrades. Whenever any security patches and software updates are available, Red Hat notifies its users (i.e. the service desk). The service desk will upgrade the Operating System by installing the latest patches and software updates whenever Red Hat makes them available. Software upgrades and all other necessary maintenance of Operating System will be done remotely with the help of the servers remote management capabilities. Page 9

12 4.1.3 Software suite The software suite that meets the monitoring requirements of LHCOPN consists of various software products. These products can be grouped into three main categories. Measurement tools and scripts perfsonar software (i.e., web service software) for providing multi domain access to data gathered by measurement tools and scripts perfsonar visualisation tools for viewing data accessed via the perfsonar software The third category of products in the list above includes visualisation tools that are web based and do not need to be deployed at the Tier-0 or any of the Tier-1 sites. All software products belonging to the first two categories (i.e., measurement tools and perfsonar software) will need to be deployed at the Tier-0 and all the Tier-1 sites. The list below introduces all the products belonging to these two categories. The service desk will install all necessary software including measurement tools, perfsonar suite and the required dependency software. The service desk will maintain the software that they have installed on the appliances. This maintenance includes applying patches and upgrading software to newer versions whenever updates are available. Software maintenance will be done in maintenance windows unless urgent maintenance is required Measurement Tools The following is a list of the Measurement Tools that will be deployed at the Tier-0 and Tier-1 sites. Further information for these tools is available in the annex. BWCTL and Iperf Tool TCP Achievable Bandwidth measurements CACTI Open source tool to gather and store IP interface statistics via SNMP HADES System One Way Delay Measurement system OWAMP One Way Delay Active Measurement tool perfsonar software with multi domain capabilities The following is a list of perfsonar software that will be deployed at the Tier-0 and Tier-1 sites. Further information for these tools is available in the annex. Authentication Service Stores user identities and attributes, provides Authentication BWCTL Measurement Point Capability to schedule TCP throughput measurements HADES Measurement Archive Access to one way delay measurement results Lookup Service XMLDB Discovery of capabilities of appliances Page 10

13 perfsonobuoy Active measurement scheduler and measurement archive Pinger Measurement Point Capability to make measurement a ping measurement Pinger Measurement Archive Access to Ping Measurement results RRD Measurement Archive Access to IP Interface statistics SQL Measurement Archive Access to circuit status information 4.2 Deployment Location The primary objective of the Multi Domain Monitoring service is to monitor the LHCOPN network connectivity. Therefore, the monitoring hardware specified in the previous section will be deployed at the Tier-0 and all Tier-1 sites. The monitoring hardware, at all sites, has to be connected to the links that are part of the LHCOPN. This is required to ensure that any active measurements made by the monitoring hardware will use the links that are used for data transfer between Tier-0 and Tier-1 and also between any two Tier-1 sites. The figure below provides an abstract view of the connectivity between Tier-0 and Tier-1 sites and illustrates some of the active measurements that will be made by the appliances deployed at these sites. Fig. 1: Illustration of active measurements over network connections between Tier-0 and Tier- 1 sites Page 11

14 Connecting the appliances via a dedicated LAN to a separate interface on the router(s) helps in monitoring the characteristics of the network connectivity between the Tier-0 and Tier-1 sites. As the appliances are connected to a separate LAN, measurements made by these appliances are not affected by any of the problems in the LANs on which the server farms (or cluster elements) are connected. Such a setup also allows the server farms to be firewalled from these appliances thus making it easier to work with existing security policies of the site. The appliances should be connected as close as possible to the border router(s) of the Tier-0 and Tier-1 sites. A dedicated switch should be used to connect these appliances to the network. This dedicated switch should be connected to a dedicated interface on the border router(s). The sections below list out common scenarios that describe border router network topologies used to connect Tier-0 and Tier-1 sites to the LHC Optical Private Network. Each scenario also identifies the location where MDM appliances can be deployed and lists out sites that have been preliminarily identified as candidates for that scenario. Scenario 1: Sites with one border router with primary and backup links This scenario describes a straightforward scenario where a Tier-1 site makes use of a single border router to connect to Tier-0 and Tier-1 sites using the LHCOPN. The border router typically contains outward interfaces connecting to a primary and zero or more backup links. The inward interfaces are connected to various LANs used by server farms. Figure 2: Illustration of Scenario 1 - appliances are connected to dedicated interfaces on the border router Page 12

15 Figure 2 is an illustration of this scenario where MDM appliances are connected via a dedicated switch to a dedicated interface on the border router of the Tier-1 sites. The Tier-1 site in the figure consists of one border router with a few interfaces. One of the interfaces (green in colour) is dedicated to the monitoring appliances connected via a dedicated switch. The LANs on which the server farms (or cluster elements) are present, can be seen connected to the router via separate interfaces. Sites suitable for Scenario 1: BNL CNAF FNAL IN2P3 NDGF PIC RAL TRIUMF Scenario 2a: Sites with two border routers with primary and backup links In this scenario, Tier-0 and Tier-1 sites could make use of two border routers to connect to other sites via the LHCOPN. The primary link used for this connection is connected to one of the routers while the backup link is connected to the other. The site s LAN connections connect to the LHCOPN via one of the routers. In case the primary link fails, the backup links, usually on a different router, are used by the LANs to communicate with Tier-0/ other Tier-1 sites. In case of failure of one of the routers, Figure 3: Illustration of scenario 2a - appliances are deployed in sites with two border routers Page 13

16 Figure 3 illustrates scenario 2a where site s LANs are distributed across the two border routers in order to connect to the LHCOPN. One of the routers contains the primary link which is used for data transfer unless there are link/router failures. The MDM appliances can be seen connected to a dedicated interface on one of the routers via a dedicated switch. The primary link to the LHCOPN can be seen connected to this chosen router as well. In case the primary link fails, the routers send all traffic via the backup links and this would include active measurement traffic as well. In case the primary router fails, the MDM appliances would loose connectivity to LHCOPN. Candidate sites suitable for Scenario 2a: GRIDKA Scenario 2b: Sites with two border routers using Hot Standby Routing Protocol or other similar protocols (Virtual Router technique) In this scenario, some of the Tier-0 and Tier-1 sites could also make use of two border routers to connect to the LHCOPN network but with one of these routers acting as a Hot Standby. This is achieved with the help of popular techniques such as Cisco s Hot Standby Routing Protocol or the Virtual Routing technique. These techniques make use of a third virtual router that acts as the recipient of all the traffic being sent by the server farms in the site s LANs. One of the border routers, assigned as primary, will play the role of the virtual router as well. The other router constantly monitors the status of the primary/virtual router. In case of router failures, the standby router assumes the role of the virtual router and traffic gets routed via the standby router instead. Figure 4: Illustration of scenario 2b - Appliances being connected to interfaces on two dedicated routers Page 14

17 In figure 4, the appliances can be seen connected to the virtual router via a switch. Effectively, the appliances are connected to both the border routers via dedicated interfaces. Ideally the switch used is dedicated to the MDM appliances but it is important to note that the appliances have been deployed and connected to the LHCOPN network in a manner that ensures that the active measurement traffic get treated the same way as the actual data traffic between the Tier-0 and the Tier-1 site. Sites suitable for Scenario 2b: SARA Scenario 2c: Sites with two border routers arranged in a Triangle along with an internal router In this scenario, two border routers of the site are connected using primary and backup links to the LHCOPN. A third internal router is then used to route the site s LHCOPN destined traffic towards a particular border router. In some cases, the decision on which border to use could be made based on the destination of the site s traffic. In case of failure of one of the routers, traffic would automatically get diverted to the other router. Figure 5: Illustration of scenario 2c - Appliances deployed at sites with two border routers connected to form a Triangle with an internal router In this scenario, the best deployment location for MDM appliances would be next to the internal router that decides the flow of traffic from the site towards the border routers. This is to ensure that the active measurement traffic from MDM appliances are treated the same way as the site s server farm traffic (bound for LHCOPN). Figure 5 illustrates such a deployment. Like in previous scenarios, the appliances can be seen connected to a dedicated switch which is then connected to a dedicated interface on the internal router. Page 15

18 Sites suitable for scenario 2c: ASGC CERN 4.3 Site Responsibilities In order to deploy the appliances, each Tier-1 site will need to provide the following: Hardware A dedicated Gigabit Ethernet switch and necessary cat 6 cabling o This switch allows a dedicated LAN to be set up for monitoring equipment within each site Terminal server according to the make and model specified in the appendix o The appliances have been tested to work with the specified terminal server model A PSTN or ISDN line for out-of-band access via the terminal server Electricity 9 power sockets, each providing v of power supply o Includes power socket for terminal server Total power consumption of 2 Kilo Watts o Includes power consumption by terminal server Connection via UPS (Uninterrupted Power Supply) to all the servers and the terminal server o o UPS Option 1: Provide UPS backup for at least 60 minutes and site administrators will notify MDM service providers of power failures well within this time period so that remedial measures can be undertaken. UPS Option 2: Make use of advanced UPS systems that are capable of notifying power failures to the servers. Note: There are additional power requirements for a terminal server if there isn t one already present at the site. The sub-section on Terminal Server below explains these requirements. Rack space Rack space of 8 units for servers in a standard19 rack o Includes ventilation space between servers 1 Additional unit of rack space for the terminal server Page 16

19 Network Connections 11 ports on the dedicated switch for data connectivity o Includes port for terminal server and net management cards on servers 3 ports on the specified terminal server for remote management o Terminal Server has to be connected to a switch for in-band access and to an ISDN line and/or PSTN line for out-of-band access 11 Public IP addresses and DNS entries per site (DNS entry managed by the Tier-0 and Tier-1 sites) o IP addresses assigned to the appliances must be within the public LHCOPN Address range o Furthermore, at each site, a dedicated a subnet within the LHCOPN address range must be assigned for the monitoring infrastructure. This subnet must be part of a /27 address range within the LHCOPN address range and must be dedicated to the servers, the terminal server and the dedicated IP interface. Network Firewall and traffic flow restrictions In order to secure the network access of the appliances as well as the site equipment, the following restrictions are important Monitoring and management related traffic from the appliances do not enter the local or remote sites LANs. This is to protect the sites sensitive equipment such as data stores and server farms. The only exception is traffic related to measurement data access from within the local or remote site, no other traffic from the site or any of the remote sites enters the LAN dedicated to the perfsonar MDM appliances. The list below details characterises the traffic flows to and from the appliances. It identifies the minimum set of firewall and access configurations required to deliver the MDM Service. The list has been drawn up based on the assumptions that the appliances make measurements with other appliances located in Tier-0 or Tier-1 sites, the data is accessed by Service desk at DANTE, the Operations groups LIPCU and E2ECU and the Tier-0 and Tier-1 personnel, and the service desk at DANTE along with DFN and Internet2 will carry out remote management of the boxes. The list is: SSH access to and from the servers and the terminal server o Access must be limited to set of address ranges that includes at least DANTE/GEANT2 and DFN and Internet2. o This is used for the management of the workstations. Telnet is not permitted. Page 17

20 HTTP traffic to and from the servers to all other appliances, to Tier-0/Tier-1 sites wishing to access data, the service desk at DANTE and other co-ordinating groups such as End to End Co-ordination Unit (E2ECU) and LHC IP operations Co-ordination Unit (LIPCU). o o Includes traffic originating from the site as well Ports in the range 8000:8200 will be receiving and sending http traffic SNMP read-only access to (at least) the border router(s) of the site connected to the LHCOPN network o o Access can be limited to the address range assigned to the servers deployed on the site Access can be limited to MIBs storing Interface statistics such as counts, drops, errors and others. ICMP packets received and sent by the servers to Tier-0/Tier-1 sites wishing to access data, the service desk at DANTE and other co-ordinating groups such as E2ECU and LIPCU TCP and UDP traffic to be allowed on the following ports to and from all other Tier-0 and Tier-1 sites, the service desk, the E2ECU and LIPCU o BWCTL requires TCP - ports 4823, 9910:9950, 5001 o BWCTL requires UDP port 5001 o OWAMP requires TCP 861 o OWAMP requires UDP 9800:10000 o perfsonobuoy requires TCP o HADES requires UDP ports 60000:61000 No restrictions on traffic exchange between perfsonar MDM servers located at the same site Local Administrators Each site is required to appoint local administrator(s) who can assist the service desk in carrying out installation, configuration and maintenance tasks. The administrators must follow the instructions provided by the service desk. Some such tasks are: Taking delivery of the servers, installing them and connecting them with other equipment such as terminal server, GPS antenna, Gigabit switch and IP interface Page 18

21 Providing network information such as topology and network configurations o Updating the service desk whenever there is any change. o Failure to do so could result in the appliances not being able to make any measurements o For example, if the capacity of an IP interface on the border router is changed or the interface itself is changed but the service desk is not updated about such changes, measured data could be inaccurate or measurement may not be possible at all. In such cases, it remains the responsibility of the local system administrators to ensure that such information is accurate and is provided to the service desk at the earliest. Assisting the service desk in maintenance procedures by doing tasks such as removing, shipping and replacing faulty hardware Notifying the service desk of scheduled maintenance and unscheduled events. In case of power failure, the service desk must be notified as soon as possible but definitely within 60 minutes (time before the UPS shuts down). Circuit Status Information for SQL Measurement Archive: The LHCOPN network connectivity between Tier-0 and Tier-1 sites is seen as a set of end-to-end circuits (also known as lighpaths). Each end-to-end path can be seen as a concatenation of multiple circuits. Each circuit is provided by one of the many network providers or administrative domains that are part of the LHCOPN (for example: Tier-0 and Tier-1 sites, GEANT2, ESnet, NRENs, USLHCnet, etc). A change in the status of one of the circuits impacts the status of the end-toend circuit that it is a constituent of. From the operational point of view, reliable status information about the end-to-end circuits as well as the status of the constituent circuits is important. For the circuits managed by the Tier-0 and Tier-1 sites, the manner in which this circuit status information can be gathered depends on the network equipment in use by the Tier-0/Tier-1 site. This is because each equipment manufacturer makes use of proprietary non-standardized interfaces to gather such information from the equipment and these interfaces are usually accessible only by the site s network operators. The MDM appliances provide the capability to access detailed information as well as the current and historical status (up, down, etc) of the circuits that are managed by the Tier-0 & Tier-1 sites. This capability is provided with the help of the SQL Measurement Archive software. In order to do this, the SQL Measurement Archive requires certain information about the circuits (local name, global name, etc) to be registered with it. Whenever there is any change to the status of any of the configured circuits or any change to the circuit information itself, the SQL Measurement archive needs to be updated at the earliest. The reliability of the information provided by the SQL Measurement Archive depends on the information that has been configured into it and also the promptness with which it is updated of any change in such information. The Tier-0/Tier-1 site administrators are expected to provide information about the circuits as well as their status to the SQL Measurement Archive. They are also expected to update the information whenever there is a change in status of the circuit or any other information related Page 19

22 to the circuit itself. Standardized perfsonar interfaces are available on the SQL Measurement Archive that can be used to provide such updates. Specialized software that can be used the Tier-0/Tier-1 site administrators in order to easily update such information can also be provided. While the SQL Measurement Archive software and the data stored in the archive is maintained by the service desk, the Tier-0/Tier-1 site administrators will be responsible for the accuracy of the circuit information provided. The administrators can approach the service desk for assistance in using the interfaces on the SQL Measurement Archive or in using the specialized software. Accurate Time Signal One of the MDM servers requires an accurate time source to be available at the deployment location in order to carry out one way delay measurements of high precision. The lack of an accurate time source at the site will result in drifting clocks that reduce the reliability of the measurements. A GPS antenna that receives time signals from the satellites has to be installed on site. There are a couple of options available for configuration of a GPS antenna and the receiver card required for interpreting the signals from the satellite. Option 1 (Recommended) - GPS Antenna: A GPS antenna with the make and model similar to the one listed in the appendix A has to be available at the Tier-0 and Tier-1 sites for this purpose. The time signal has to be delivered to one of the MDM servers via a co-axial cable (RG58). In this case, the server connected to the time signal acts as an NTP stratum-1 server. The figure 6 illustrates the connection between the one of the MDM servers (green box) and a GPS antenna. Other key points to note in this figure are the use of a splitter, a lightning protector and the GPS receiver card on the server. If a GPS antenna is not already present at the site, the site will need to install a GPS Antenna of the same make and model and also the necessary co-axial cable (RG58) connections. Ideally a splitter should be used to allow re-use of the signal by multiple servers/cards. If a similar GPS antenna is already present at the site, a co-axial cable (RG58) connection between a splitter connected to the antenna and the GPS card on one of the deployed servers is required. Page 20

23 Figure 6: Illustration of connections between GPS time source and a monitoring appliance Option 2 - PPS or NTP signal: If a GPS antenna is already available and is connected to an existing GPS signal receiver card, it is possible to re-use the clock on this card via a PPS signal (if the existing GPS card supports PPS) or by configuring the server with the GPS receiver card as an NTP stratum-1 time source. The PPS signal has to be delivered to the MDM appliance via a serial cable. However, device driver limitations and difficulties in modifying the Operating System Kernel of Red Hat Enterprise 5 create some constraints to this solution. As a result it is not the preferred solution and its use will have to be decided on a case-by-case basis. If the existing GPS signal receiving server can be configured as an NTP time source (stratum- 1), it can be used as a time source for MDM appliances provided: The Stratum-1 server (i.e. server receiving the GPS signal) is located adjacent to the MDM appliances and is connected to the same LAN switch as the MDM appliances The network path connecting the stratum-1 server and the appliances are less than 50% loaded at any time in order to avoid network congestions to reduce the accuracy of the time signal The delay and delay variation on the network path between the stratum-1 server and the MDM appliances are extremely low (<2 micro seconds of delay variation) in order to ensure accuracy of the time signal Since the network related constraints are difficult to predict or control, this option is not highly recommended and has to be decided on a case-by-case basis. Installations of GPS antenna and all necessary cabling work will have to be carried out by a local company/contractor and has to be chosen by the site management. Installation instructions and information on how to test installations will be available. Page 21

24 Site Security and physical access control All sites are required to secure the measurement infrastructure against unauthorized access. Personnel with authorization from the perfsonar MDM service desk at DANTE should be allowed to access the infrastructure. 3 rd party hardware support personnel will be required to visit the site on behalf of DANTE to carry out hardware repair or upgrade tasks. Such visits will be planned with a short notice as hardware repairs and replacements have to be carried out as soon as possible. The personnel will be given details about the site location and site security procedures that are required to be followed. In case of a security breach, the sites must follow their contingency plans and they must immediately inform the service desk. 4.4 Security Risk Management overview This sub-section provides an overview of some of the risks associated with the service. These risks threaten the normal operation of the monitoring service and in some cases, it could affect other LHCOPN operations. Mitigation of such risks is explained in each category. Network related vulnerabilities The service desk at DANTE along with DFN and Internet2 will require remote access to the appliances at all times for administration tasks. Remote access will also be required for the visualisation tools used by Operational groups and for measurement data and information exchange between appliances. The need for remote access to the appliances requires that the appliances be secured against network related vulnerabilities. The lack of network security can expose the machine to several threats. An immediate threat is attack from unauthorised IP addresses. Without firewalls protecting the servers anybody can probe the servers for vulnerabilities or information leading to a compromise of the servers. This risk can be mitigated by the correct use of firewall filters on the site s border routers and the servers. These filters limit access to a specific set of ports on the servers and also the IP addresses that are permitted to contact these ports. For the perfsonar MDM service, the list of ports and protocol types required by the monitoring applications and the user groups permitted to access these ports have already been specified in this document. The service desk will apply this list to the firewalls on the servers. The local administrators of the site will have to apply this list on the security firewalls in the site s routers (border routers) to protect the servers and also the Tier-0/1 site equipment. Page 22

25 A denial of service attack whether distributed or not would cause significant harm to the services the servers provide. Simply overloading the HTTP server with requests would greatly impact the users accessing the HTTP service. This risk is also mitigated by the previously described measure of restricting access to the HTTP ports on the servers to a limited group of users. Legitimate users of the service will be advised about fair use policies so that they don t overload the service on purpose. To mitigate the threat of a smurf attack [11] or ping flood the routers should be configured to enforce strict rate limiting policies. A router has far more capability than a server to filter this traffic without impact and by dropping the unwanted traffic closer to the source there is less of a congestion risk. Software vulnerabilities Any vulnerability in the software such as operating system or the applications, especially the ones listening on open ports such as SSH daemon, could end up with the system being compromised. This will pose a considerable risk to the normal operation of the appliances, to the stored data and possibly to other local and remote devices to which the compromised appliance has network access. As the appliances have SNMP read access to border routers of the site, there is a risk that large amounts of SNMP read requests could be made to the border routers. As part of risk mitigation, the operating system used on the servers is Red Hat Linux Enterprise Server 5.0. This operating system comes with a licensing agreement with Red Hat Corporation. As per the agreement, the Red Hat Corporation provides patches and updates for any discovered vulnerabilities. Alerts are sent on subscription channels whenever vulnerabilities are discovered and patches are available. The service desk at DANTE will promptly update the Operating System whenever such patches are made available. If a known vulnerability is expected to affect the appliances, appropriate measures will be taken, which could include shutdown of some of the monitoring tasks. Furthermore, the appliances are exposed to limited user groups with the help of network firewalls as detailed previously. The operating system will also be bolted down to run with only the necessary set of services. As a result, the risk of exploiting operating system vulnerabilities is reduced considerably. Restricted user access to the applications on the appliances with the help of network firewalls and in-built authentication mechanisms reduces the level of risk associated with application vulnerabilities. All applications required for monitoring purposes will be run in a low-privilege user account on the appliances. As a result, exploits would have very limited access to system commands. Security vulnerabilities in the applications will be handled with the help of agreements (contractual or MoU) between the service desk and many of the developers of such applications. Developers would be asked to prioritise working on patches for known vulnerability. Physical access vulnerabilities The appliances deployed within each site are susceptible to loss of connectivity and power supply as well as damage due to negligence or intentional acts from authorized or unauthorized personnel with physical access to the machines. Page 23

26 As mentioned in the site requirements previously, each site is expected to secure the monitoring infrastructure deployed within their site and control access to the infrastructure as per the requirements of the Service Desk at DANTE Security Controls and contingency planning Apart from the risk mitigation procedures mentioned previously, security control procedures such as periodic audits of appliances and internal procedures will be used to enhance security related aspects of the service and to ensure that the appliances are working as expected. Whenever a change is planned to be introduced into the service, change management procedures include identification of any new security related risks that could arise as a result of this change. In such cases, risk mitigation procedures would be carried out before the change is introduced into the service. Although the servers are operated and controlled legitimately, emergency procedures which include adding special rules in the firewalls of border router(s) and workstations will be prepared. These procedures are designed to help the service desk in situations where one or more of the servers are compromised or if a legitimate user causes network and monitoring service degradation. If a compromised server is detected by the service desk, either via security audits or via other means such as loss of connectivity, data loss and any other abnormal behaviour, emergency procedures will be followed. Local administrators might be requested to apply special rules in the site s border routers to which the servers are connected. Depending on the severity, the service desk may request the affected sites to remove power supply until the incident is resolved and integrity is restored. Removal of power or network access will result in some level of service degradation depending on the appliance(s) involved. Detailed contingency plans define the procedures to be followed by the service desk in order to restore normal or partial service operation. They also define the assistance expected from the local site administrators. Other contingency plans used by the service desk for the perfsonar MDM service for LHCOPN include procedures for situations such as restoring from backed up data in case of problems with the primary data source and rebuilding components or the complete appliance in case of hardware failures Measurement data access restrictions Many administrative domains (example: the CERN domain, the GEANT2 domain) make use of technologies such as digital certificates, etc. in order provide credentials to its users. Users make use of these credentials to access resources such as measurement data within their domain. The edugain framework makes it possible for users of domains with compatible authentication technologies (for example: Shibboleth [3]) to use their existing credentials in order to access resources made available by other domains. Such domains will however need to be part of the edugain federation. For domains that do not use edugain compatible authentication technologies, edugain and GIdP [4] can provide credentials to the users these Page 24

27 domains. edugain is also investigating the compatibility of VOMS (Virtual Organisation Membership Service) [5] with its framework. The perfsonar software provides the capability to restrict access to measurement data and the usage of measurement tools to authenticated and authorized users only. These capabilities are available via the edugain framework [2] with which the perfsonar framework is compatible. If access to the measured data and the measurement capabilities of the deployed perfsonar software has to be restricted to a selected group of users, then it is necessary for all these users to possess credentials compatible with the edugain framework. 4.5 perfsonar MDM Service in beta The perfsonar Multi Domain Monitoring service began as a beta service (i.e., public trial) during the 2 nd quarter of A dedicated service desk to support its users has been available since then. During this beta phase, the roll out of the service is limited to a specific number of users in order to carry out a controlled evaluation of the service so that suggestions and improvements can be gathered and quality and scope of the service can be improved before the service is made available to a wide community of users. During the first stage of the trial phase (called the MDM pilot) the service has been successfully rolled out to six backbone network providers. During the second stage (the MDM prototype), which begins during the first quarter of 2008, the number of users will almost be doubled to eleven. To meet the requirements of LHCOPN monitoring, the perfsonar MDM service has been customised and the trial phase is being extended to include the Tier-0 and the Tier-1 sites as well so that the feedback from the LHC community can be used to further enhance the MDM Service. Page 25

28 5 Support The Multi Domain Monitoring Service offered by DANTE is supported by a dedicated service desk located at DANTE. The service desk acts as the Single Point Of Contact (SPOC) that can be reached via telephone and . It provides support to the users with the help of underpinning contracts and support agreements with providers of hardware, Operating System and various development teams responsible for monitoring tools and perfsonar products. The service desk makes use of a Trouble Ticket System in order to acknowledge reported incidents (usually by users). In order to help the service desk in detecting incidents even before they are reported, special tools are in use that monitors the health of deployments and raise alerts if it detects problems with any of the deployments. Such proactive incident detection helps the service desk in speedy acknowledgement and resolution of certain incidents. The service desk also maintains an information repository designed to meet the needs of correlating and storing information related to the Multi Domain Monitoring Service. This repository, called as the Configuration Management Database, provides quick and detailed access to information related to any case being handled by the service desk. The service support structure and procedures that have been defined for the MDM service make use of the Information Technology Infrastructure Library (ITIL) standards. The IT Infrastructure Library (ITIL) is a framework definition that specifies a set of processes and functions to be used for the management of any IT Service. These definitions are based on best practices gathered from public and private sector industries world wide. Page 26