Sample Report by: Bernie Ledwick Network Integrity Service Ltd 17 March 2005 bl@e-nis.com Phone: 0161 860 7678
1 Tables. 1.1 Contents. 1 Tables... 2 1.1 Contents... 2 1.2 Figures... 2 2 Introduction.... 4 2.1 Overview.... 4 2.2 Constraints... 4 3 Network overview.... 5 3.1 Network Diagram... 5 3.2 Network description... 5 4 Findings... 6 4.1 Site-1.... 6 4.1.1 Firewall... 6 4.1.2 Switch2.... 7 4.1.3 Switch 3....11 4.1.4 Wireless access points....13 4.2 Site-2....14 4.2.1 Site-2 router....14 4.2.2 Site-3 Wireless Access Point....15 4.3 Site-3....16 4.3.1 Site-3 Router....16 4.3.2 Site-3 Switch 2...17 4.3.3 Site-3 Wireless access point....18 4.4 Site-4....19 4.4.2 Site-4 Switch 1 Port 21...22 5 Latency...23 6 Environmental...25 7 Conclusions....26 Appendix A. Network Diagram...27 1.2 Figures. Figure 1 Site-1 Firewall 09-Mar to 17Mar... 6 Figure 2 Site-1 Firewall - Mar 2005... 7 Figure 3 Switch 2 port 8 16-Mar to 17 Mar... 7 Figure 4 Site-1 Switch 2 port 8 9-Mar to 17-Mar... 8 Figure 5 Switch 2 port 8 - Mar 2005... 8 Figure 6 Switch 2 port 11-16-Mar to 17-Mar... 9 Figure 7 Switch 2 Port 11 09-Mar to 17-Mar... 9 Figure 8 Site-1 switch 2 port 18 traffic - Feb-Mar 2005...10 Figure 9 Site-1 switch port 18 errors - Feb-Mar 2005...10 Figure 10 Switch 2 Gigabit port 1...11 Figure 11 Site-4 switch 3 port 4 errors...11 Bernie Ledwick Page 2 19 April 2005
Figure 12 Site-4 switch 3 port 9 errors...12 Figure 13 Site-4 switch 3 port 15 errors...12 Figure 14 Conset router ADSL interface Mar 2005...14 Figure 15 Site-2 ISDN - Jan-Feb 2005...14 Figure 16 Site-2 ISDN - 9-Mar-17 Mar...15 Figure 17 Site-3 router ADSL interface 9-Mar to 17-Mar...16 Figure 18 Site-3 router ISDN interface - Feb-Mar 2005...16 Figure 19 Site-3 switch 2 port 3-16-Mar to 17-Mar...17 Figure 20 Site-3 switch 2 gigabit port 2 16-Mar to 17-Mar...17 Figure 21 Site-3 wireless access point Feb-Mar 2005...18 Figure 22 Site-4 switch 3 port 16-9-Mar to 17-Mar...19 Figure 23 Site-4 switch 3 port 24-9-Mar to 17-Mar...19 Figure 24 Site-4 switch 2 port 24-9-Mar to 17-Mar...20 Figure 25 Site-4 Switch 2 port 23-9-Mar to 17-Mar...20 Figure 26 Site-4 switch 1 port 21-9-Mar to 17-Mar...21 Figure 27 Site-4 switch 1 port 24-9-Mar to 17-Mar...21 Figure 28 Site-4 switch 1 port 21 traffic...22 Figure 29 Site-4 switch 1 port 21 errors...22 Figure 30 Site-2 router latency...24 Figure 31 Site 2 router errors on WAN port...24 Figure 32 Site 2 router traffic on WAN port...24 Figure 33 Computer room temperature...25 Figure 34 Network connectivity at Site 1...27 Bernie Ledwick Page 3 19 April 2005
2 Introduction. 2.1 Overview. The background to the report is as follows.. Network Integrity Services Ltd (NIS) were asked by XXX Company to assist them in monitoring various aspects of their network, as issues were being encountered on a regular basis and the IT department resources were fully stretched and finding it difficult to keep on top of the issues as they arose. It was agreed that the evaluation should cover the Site-1, Site-2, Site-3 and Site-4 sites, concentrating on bandwidth usage, ISDN activation and network latency. 2.2 Constraints. All routers and switches at the four sites are being monitored. However, it has not been possible to monitor the Site-1-Site-4 routers during the evaluation as these are maintained by xxx-communications-services-company, rather than yyycommunications-services-company. Two switches in Site-1 (switch6 and switch7) do not appear to be configured to allow monitoring. Bernie Ledwick Page 4 19 April 2005
3 Network overview. 3.1 Network Diagram. Please refer to Appendix 1. 3.2 Network description. The XXX Company branch network is based on a zzz-isp (supported by yyycommunications-services-company) VPN solution, over ADSL with ISDN backup. The main site is Site-1 where all the live servers reside. There are backup servers located in Site-4, with real-time data synchronisation via a high-capacity link between the Site-1 and Site-4 sites. This link is maintained by xxx-communications-servicescompany. Each of the remote sites has a wireless access point and the Site-1 site has two wireless access points. The NIM server was installed at Site-1 and has links back to NIS s servers in Manchester via the internet connection. NIS were provided with a dial-in VPN connection for configuration and monitoring purposes. Bernie Ledwick Page 5 19 April 2005
4 Findings. The reports below show anomalous results, which may actually show expected traffic, but may also show underlying issues which would require further analysis to determine the causes. 4.1 Site-1. 4.1.1 Firewall. Figure 1 Site-1 Firewall 09-Mar to 17Mar As can be seen on the above graph, the Site-1 firewall network interface is completely utilised for about 8 hours each night from 21:30-07:30. There are times when the firewall is completely utilised during the day, e.g. Monday evening and Tuesday midday. During these periods Internet users will experience reduced performance and the e-commerce applications will also be slow to use for customers. This needs monitoring for a longer period to see if the experience of Monday and Tuesday is repeated. Bernie Ledwick Page 6 19 April 2005
Figure 2 Site-1 Firewall - Mar 2005 The above graphs show a significant increase in the throughput of the firewall during the day, with traffic increasing to 400kb/s in the last week. It may be useful to turn on the NIM Internet Usage reporting to identify if the increase is linked to non work related usage of the Internet. 4.1.2 Switch2. 4.1.2.1 Port 8. Figure 3 Switch 2 port 8 16-Mar to 17 Mar Bernie Ledwick Page 7 19 April 2005
Figure 4 Site-1 Switch 2 port 8 9-Mar to 17-Mar Figure 5 Switch 2 port 8 - Mar 2005 There is a significant amount of traffic on a regular basis on this port, mainly overnight. However, traffic was not apparent in week 10. It is suspected this is related to a backup process, if it is expected to run every day then NIM can be configured to detect the failure of the process as seen during week 10. Bernie Ledwick Page 8 19 April 2005
4.1.2.2 Switch 2 Port 11. Figure 6 Switch 2 port 11-16-Mar to 17-Mar Figure 7 Switch 2 Port 11 09-Mar to 17-Mar Significant traffic can be seen on this port over the last 2 days, with no previous traffic. NIM can identify unexpected usage this may simply be a new employee or similar change of use. Bernie Ledwick Page 9 19 April 2005
4.1.2.3 Site-1 switch 2 port 18 Figure 8 Site-1 switch 2 port 18 traffic - Feb-Mar 2005 Figure 9 Site-1 switch port 18 errors - Feb-Mar 2005 There are errors showing on the port in weeks 9-11, which may indicate an issue with cabling or an interface. The user on this port will probably experience poor performance so correcting the issue will improve user satisfaction. Bernie Ledwick Page 10 19 April 2005
4.1.2.4 Switch 2 Gigabit port 1. Figure 10 Switch 2 Gigabit port 1 Significant, abnormal traffic can be seen over the weekend period. This needs to be explained as it may indicate data being transferred out of the system, or a virus. 4.1.3 Switch 3. Figure 11 Site-4 switch 3 port 4 errors Bernie Ledwick Page 11 19 April 2005
Figure 12 Site-4 switch 3 port 9 errors Figure 13 Site-4 switch 3 port 15 errors The above ports are showing errors in weeks 9-11, which may indicate an issue with cabling or an interface. See note on 4.1.2.3 Bernie Ledwick Page 12 19 April 2005
4.1.4 Wireless access points. Neither of the wireless access points seems to be utilised, with traffic only being apparent on ap2, yesterday. Bernie Ledwick Page 13 19 April 2005
4.2 Site-2. 4.2.1 Site-2 router. 4.2.1.1 ADSL interface. Figure 14 Conset router ADSL interface Mar 2005 Traffic can be seen to be reasonably within the limits of the ADSL connection, but there is evidence of some constant 10-15kb/s traffic in weeks 7 and 10. The volume recorded is within the limits for normal activity but may indicate an underlying issue. 4.2.1.2 ISDN interface. Figure 15 Site-2 ISDN - Jan-Feb 2005 Bernie Ledwick Page 14 19 April 2005
Figure 16 Site-2 ISDN - 9-Mar-17 Mar There is evidence that the ISDN line is being raised regularly in January and February, less so in March. NIM reports on these instances to NIS and to XXX Co IT staff. Reducing the time that the ISDN is in use for clearly saves money. 4.2.2 Site-3 Wireless Access Point. This does not seem to be utilised. Bernie Ledwick Page 15 19 April 2005
4.3 Site-3. 4.3.1 Site-3 Router. 4.3.1.1 Site-3 Router ASDL Interface. Figure 17 Site-3 router ADSL interface 9-Mar to 17-Mar The traffic on the ADSL interfaces seems to be reasonably in the bounds of the ADSL capability, but there has been a marked increase to 15-20kb/s in the underlying traffic this week. 4.3.1.2 Site-3 Router ISDN Interface. Figure 18 Site-3 router ISDN interface - Feb-Mar 2005 Bernie Ledwick Page 16 19 April 2005
The ISDN line is being activated on a regular basis for short periods. 4.3.2 Site-3 Switch 2. 4.3.2.1 Site-3 Switch 2 Port 3 and Gigabit Port 2. Figure 19 Site-3 switch 2 port 3-16-Mar to 17-Mar Figure 20 Site-3 switch 2 gigabit port 2 16-Mar to 17-Mar Abnormal traffic yesterday is showing between these two ports, up to 15:30. N.B. Gigabit port 2 is labelled as being connected to switch5, but we have no record of a switch5 in Site-3. Bernie Ledwick Page 17 19 April 2005
4.3.3 Site-3 Wireless access point. Figure 21 Site-3 wireless access point Feb-Mar 2005 This seems to have been utilised only in weeks 9 and 10. Bernie Ledwick Page 18 19 April 2005
4.4 Site-4. 4.4.1.1 Switches. Figure 22 Site-4 switch 3 port 16-9-Mar to 17-Mar Figure 23 Site-4 switch 3 port 24-9-Mar to 17-Mar Bernie Ledwick Page 19 19 April 2005
Figure 24 Site-4 switch 2 port 24-9-Mar to 17-Mar Figure 25 Site-4 Switch 2 port 23-9-Mar to 17-Mar Bernie Ledwick Page 20 19 April 2005
Figure 26 Site-4 switch 1 port 21-9-Mar to 17-Mar Figure 27 Site-4 switch 1 port 24-9-Mar to 17-Mar It would seem that the switches in Site-4 are daisy chained, e.g. 1->2->3. It may be beneficial to re-arrange the switch connections, so that, as in the case shown above, significant amounts of traffic would not need to visit 3 switches, thus removing a potential performance bottle neck. Bernie Ledwick Page 21 19 April 2005
4.4.2 Site-4 Switch 1 Port 21. Figure 28 Site-4 switch 1 port 21 traffic Figure 29 Site-4 switch 1 port 21 errors This port is showing errors in weeks 9-11, which may indicate an issue with cabling or an interface. Bernie Ledwick Page 22 19 April 2005
5 Latency. In the last few days of the evaluation period, we implemented a latency test of the network to the three remote sites. The test as implemented is a network ping test to the router and a switch at each of the remote locations. Whilst this gives an indication of performance, including packet loss, it can be, in some cases, misleading as routers and switches do not always respond readily to ping requests if they are busy. More sophisticated tests can be implemented, e.g. HTTP round trip timings, depending on what network equipment is available at the remote location. Some thought would be required as to the relevance of this type of information, but one important use could be as a direct comparison between different service providers. A sample output for the performed test is shown below and it can be seen that there is a strong correlation between traffic, CRC errors and slower response times. A small volume of packet loss was observed. All remote sites, except Site-4 connected by Megastream, exhibited similar characteristics, so it would be worth investigating the cause of the CRC errors as this could have performance implications for users at remote sites. Bernie Ledwick Page 23 19 April 2005
Figure 30 Site-2 router latency Figure 31 Site 2 router errors on WAN port Figure 32 Site 2 router traffic on WAN port Bernie Ledwick Page 24 19 April 2005
6 Environmental As well as monitoring the data traffic NIM has been recording the temperature in the computer room. Figure 33 Computer room temperature Problems with the air conditioning during the period are clear to see. NIM alerted staff at XXX Co to the problem allowing them to prevent a serious issue. Bernie Ledwick Page 25 19 April 2005
7 Conclusions. The above findings need to be discussed and investigated, as required, so that XXX Company IT personnel have an understanding of the types of issues that NIM can identify and report. In order for XXX Company to benefit fully from a wider implementation of NIM, some work will be required to more fully document the network (e.g. what is connected to which switch port) and an on-going programme of assessing what issues need to be reported automatically needs to be progressed. During the period covered by the report, NIM identified several issues where money was saved with the ISDN links and a number of issues that were unlikely to be found without NIM and which would have an impact on the user. A number of system processes were identified that need further investigation, and close monitoring if deemed critical. Bernie Ledwick Page 26 19 April 2005
Appendix A. Network Diagram Sample only Figure 34 Network connectivity at Site 1. Bernie Ledwick Page 27 19 April 2005