Rural Telephone Service Co., Inc. Troubleshooting and Remote Monitoring for IPTV Finding the needle in the haystack

Transcription

1 IneoQuest Case Study Case Study: Video Quality Monitoring Live IPTV Rural Telephone Service Co., Inc. Troubleshooting and Remote Monitoring for IPTV Finding the needle in the haystack Marc Todd IneoQuest Technologies, Inc. Shane Broyles Rural Telephone Service Co., Inc. Introduction Rural Telephone Service Co., Inc. (Rural Telephone) and its IPTV service area are located in Kansas. Rural Telephone started using the IneoQuest product in the fall of 2004 to help debug and troubleshoot Quality of Experience issues within the Rural Telephone IPTV deployment across their remote network. Today, Rural Telephone has 1 headend and 35 central offices spread across 5,100 square miles in western Kansas. Rural Telephone has worked with IneoQuest tools to improve the quality of the IneoQuest product and increase its usefulness in the emerging IPTV market. Over time, Rural Telephone saw the advantage of placing IneoQuest equipment at critical routers in remote locations for proactive monitoring. This proactive monitoring would increase their productivity and decrease travel time by quickly and easily identifying network degradation. Rural Telephone has 4 x OC-48 rings serving 31 central offices where every office has access to an IPTV service via Ethernet. This topology allows Rural Telephone to easily access content from any node in the network and also inject content such as local off air channels. Rural Telephone and IneoQuest Case Study for IPTV Troubleshooting and Monitoring

2 Rural Telephone uses IneoQuest products to continuously monitor the integrity of their network. Since the geographical area is very large, the IneoQuest monitoring and analysis tools provide Rural s technical team performance visibility from any location in the network. This case study describes how Rural Telephone, component vendors, and IneoQuest worked together to troubleshoot Rural Telephone s Video over IP network. Although Rural Telephone had not logged any direct customer complaints, there was a concern about Quality of Experience (QoE) occurrences due to the identification of significant packet loss as measured by the IneoQuest tools. The manifestation of media loss on about 10 percent of the IPTV streams was identified using the IneoQuest Singulus TM G1-T Video over IP Monitoring and Analysis Tool and the Media Delivery Index (MDI) measurement. The challenge was to find out where the loss was occurring, it s cause and how it could be fixed. Was this loss on the IP network or in the video network? This was the needle in the haystack? Every component in the system was suspected including the encoder, network cabling and routers. The real challenge is caused by the nature of IPTV itself. There are currently few standards and everyone including operators and equipment manufacturers are moving rapidly to deploy. The effect of this quick deployment may cause interoperability problems enhancing the need for proactive field monitoring strategies. This case study walks through the setup and test methodology to discover the problems found on the Rural Telephone network. Even though the problem was in Kansas, it was possible to monitor, identify, isolate and analyze the problem remotely from the IneoQuest engineering lab located in Boston. The monitoring and analysis of counters, log events, video and protocol were captured and analyzed from Boston, which was key to the development of this case study. The ability to debug a site from a remote location illustrated that it was possible to save unnecessary expenditures such as time, travel, accommodations and overtime. The test results indicated a problem of extra information being inserted into the video payload (PIDs in the Transport Stream) manifesting itself every 15 to 30 minutes on average per effected stream. This error only showed up from one encoder and affected 10 to 15 IPTV streams. When the event occurred, it affected the audio part of the signal only, producing a distinctive audio glitch. In addition to the network protocol capture, the MPEG2 TS Video and Audio were recorded by the G1-T in order to replay the identified fault for debug purposes or playback for further analysis by equipment manufacturers. The conclusion to this case study reinforces the need for monitoring and analysis of all streams in the IPTV network in order to assure Quality of Experience for everyone. The fact that customers had not yet complained makes this problem even more difficult to isolate. The question is whether or not this problem would have become worse over time. The audio glitch was not enough to have customers complain, but goes to the underlining perception of quality of the IPTV service. Rural Telephone s proactive approach caught this problem in a timely manner. If Rural Telephone had not been monitoring this problem, it may never have been detected. Monitoring the entire network gives everyone involved more confidence in interoperability and that the system is functioning as expected. Issue: Significant Media Loss Measured - Finding the needle in the haystack One major concern faced by any service provider is whether or not their customer is receiving good quality video and audio. As previously stated, Rural Telephone had not been receiving any customer complaints, but still needed to explain why IP packet loss was occurring. Over a period of a few hours, Rural Telephone had measured several hundred media loss events (MDI-MLR), which was a cause for great concern. In its current configuration, Rural Telephone has a few hundred variable bit rate (VBR) IPTV streams that are MPEG2 Transport Stream (TS) encoded. These video transport streams are a multiplex of the video, audio and other information used by the set top box (STB). The structure of the transport stream separates the video, audio and other information into 188 byte packets with a header ID information that identifies the packet as video, audio or other. In addition to the ID, the header contains a 4-bit sequence counter (Continuity Counter) used to monitor sequence of the particular TS packets. This counter is unique for each ID and increments from 0 to 15. Thus, a 188 byte video packet and the audio packets, using separate counters, will also increment from 0 to 15. In this application, the 188-byte packets are put into a UDP/IP packet by the encoder. There are (up to) 7 x 188 byte TS packets within one UDP/IP frame for a total of 1362 byte Ethernet frame. The structure described is the transport being used at Rural Telephone. Rural Telephone and IneoQuest Case Study for IPTV Troubleshooting and Monitoring - 6/4/

3 In this particular configuration there is limited information within the IP packet structure and payload, which would have allowed for simple debug. Layer-4 UDP encapsulation does not contain information such as sequence numbers making it impossible to determine if IP packet loss from a UDP stream had occurred. For this stream configuration, the Singulus G1-T needs to go the extra step and measure the continuity counters for each TS PID within the TS payload. The measurement, which identifies media loss rate within the MPEG TS, is called MDI-MLR or Media Delivery Index-Media Loss Rate. The need to monitor all Video over IP streams simultaneously is required to identify which stream is impaired due to packet loss. Of course, these measurements need to be completed in a non-intrusive manner. If an IP packet is dropped, the media loss counter will increment by a count of 0 to 7, as there are 7 TS packets in a typical IP frame. Figure 1 Rural Telephone Live Census and Identification of Media Loss on Streams in IPTV Network - Encoder IQ-A One issue to consider is that it is not possible to know if the IP packet was dropped or if the video TS payload was incorrect even before its encapsulation on the IP network. If any media loss is detected, alerts to the service provider need to identify that media loss events are occurring. It is possible to quantify the magnitude of media loss and also determine when the loss events have occurred due to event logging and time stamping. This event log can be very helpful in determining the occurrence and nature of IP packet loss. In their configuration, Rural Telephone was observing media loss counters increase by a count of 13, which added up to a few hundred-packet losses within a few hours. Figure 1 illustrates Media Loss Total (MLT) counters to be in the hundreds. Cumulative loss is identified in the total MDI-MLT column. A RED background identifies streams with excessive loss. Still, the open question remains; is the fault caused by the IP network or is the encoder at fault and why are the MLT counters increasing on a regular basis? Rural Telephone and IneoQuest Case Study for IPTV Troubleshooting and Monitoring - 6/4/

4 Setup: IPTV Network and Test Setup Monitoring of the network at critical nodes was essential to isolate the problem. The test involved two of Rural Telephone s Singulus G1-Ts; one at the encoder and the other at an aggregation switch. In order to help Rural Telephone isolate the problem, IneoQuest Engineering team monitored the control ports of the Singulus G1-T and recorded measurements as illustrated in Figure 2. Figure 2 Rural Telephone Live IPTV Network and Singulus Deployment Setup Rural Telephone currently has several hundred IPTV channels and uses IneoQuest equipment to monitor and probe at several points in the network. Figure 3 illustrates user-defined aliases set up on the IneoQuest monitoring equipment by Rural Telephone. The role of the Singulus G1-T is to dynamically and automatically detect Video over IP streams and associate user-defined aliases allowing the user the ability to easily identify stream as specific TV programs. Aliases are also important in the setting of stream alarm thresholds and characteristics such as VBR, CBR and encoding rates. Alarm thresholds can be defined on a per stream basis or group of streams. Alarms can be generated by multiple types of events such as; bandwidth utilization, MDI-DF, MDI-Loss, loss distance, loss period, PCR Bit rate variations, IP bit rate. All alarms are logged and can be ejected to either the SNMP manager or Syslog Daemon server. Figure 3 Rural Telephone IPTV Alias & Alarm Threshold Table Encoder (IQ-A) Rural Telephone and IneoQuest Case Study for IPTV Troubleshooting and Monitoring - 6/4/

5 Baseline: Initial Inspection and Test Approach In order to debug a Video over IP network, it is vital to look at the cause of the problem and not the symptoms. There are several types of information that can be very useful at the onset of debug. It is imperative to consider all available information before implementing a debug approach. First, isolate the issue giving the behavior and make sure there are no fundamental problems. Inspecting the RMON counters (Figure 4) will provide a general idea of the traffic flows, CRC errors and link utilization. Since video streams are formed from 1362 byte packets, we would expect most of the activity to occur within the bytes range. However, notice that we are seeing differently sized packets with some frequency at Singulus IQ-B as per Figure 2 Figure 4 RMON, QoS Stats and Line Capture Decode Using the IQMediaAnalyzer software, (which is one of three software packages that can be used with the Singulus G1-T), it is possible to capture traffic on the open line to inspect the network activity. The packets turned out to be some other control traffic, not video. The IP addresses are outside the scope of this paper; however, they are probably OK and most likely not causing any problems, as the utilization is less than 50 percent. Prior to testing and network characterization, it is recommended to clear all alarms and logs. Using both IQMediaAnalyzer and IQMediaMonitor (SNMP Tool Application) software we clear and inspect current alarms and higher level network stats to start our troubleshooting. Figure 5 IQMediaMonitor Windows - The Clearing of current Alarms for Units IQ-A and IQ-B Rural Telephone and IneoQuest Case Study for IPTV Troubleshooting and Monitoring - 6/4/

6 Troubleshoot: Setup Test for Overnight Monitoring The two units were cleared for a baseline test and ran overnight for 11 hours. The results depicted in figure 6a, identified by the red background color, showed media loss in several streams. Figure 6a is the aggregate of all streams from IQ-A and IQ- B taken by the IQMediaMonitor Application. Figure 6a IQMediaMonitor Inspection of Multiple Singulus Monitoring Units - for Units IQ-A and IQ-B Figure 6b and 6c show the IQMediaAnalyzer Applications networked to each Singulus G1-T unit. Figure 6b and c illustrates that most of the loss (identified by red background) occurred when PID 0x86 is present. This information is found in the pink PID column of figure 6b and c and provides critical clues to the origin of the problem. Figure 6b IQMediaAnalyzer Interface of Singulus G1-T Unit at IQ-A - Live After Running for 11 Hours on the Live Network Rural Telephone and IneoQuest Case Study for IPTV Troubleshooting and Monitoring - 6/4/

7 Figure 6c IQMediaAnalyzer Interface of Singulus G1-T Unit at IQ-B Live After Running for 11 Hours on the Live Network As per Figure 6b and 6c, the results show that the MDI Media Loss Total (MDI-MTL) numbers of streams seen at 2 points (IQ-A and IQ-B) in the network vary. This could be due to several issues: 1) The payload out of port 2 of the encoder may not be the same as port 1. It is possible that the switch may be dropping packets. Packet loss due to network transport is probable and was still under investigation during the development of this case study. Figure 6d IQMediaMonitor monitoring several sites 2) The network between the encoder and the IQ-B unit could have dropped the IP packet. These are all valid concerns and should be explored; however, for this test the magnitude and the pattern of packet loss was very consistent. Interestingly, each stream that has massive MTL contains the PID (188-byte TS packet) with the ID 0x86. Also, notice the maximum MLR (Media Loss Rate) in Figure 6b, 6c and 6e is 13, meaning in a 1 second sample, the maximum continuity counter loss was 13. Figure 7 also shows that the rate of loss has some periodicity. Rural Telephone and IneoQuest Case Study for IPTV Troubleshooting and Monitoring - 6/4/

8 Figure 6e IQMediaAnalyzer Scoped in Detail on IQ-A Simultaneous measurement of all 90 Video over IP Streams Figure 7 Charting Media Loss Log Events versus Time of Day The charting information has been extracted from the IQMediaMonitor Logging event, which provides time and correlated event information. Figure 8 is a sample of the log file which identifies: monitoring location, severity of alarm, source and destination of IP and port address, event description, current alarm and defined errors threshold and time of day. If needed, the log information can be extracted to 3 rd party application for further analysis. Rural Telephone and IneoQuest Case Study for IPTV Troubleshooting and Monitoring - 6/4/

9 Figure 8 IQMediaMonitor Log with Time Stamp Events Numbers Extracted to chart Trends As highlighted in Figure 8, the supermajority of the media loss events has a magnitude of 13. Using log information, it was possible to chart results as per figure 9a, which shows the error arrival time histogram and the comparison of 2 IPTV streams from IQ-A and IQ-B. Note the exact same events for 2 IPTV programs separated by a second or two over a period of 11 hours (7 charted), which is expected if the problem is at the encoder. The linear slope as shown in figure 9a indicates the periodicity of the problem. The periodicity suggest a deterministic problem which can be easily correlated due to it repeatable nature. Figure 9a Error Arrival Histogram and 2 IPTV programs at 2 Locations (IQ-A and IQ-B) - Chart over 7 Hour Period Rural Telephone and IneoQuest Case Study for IPTV Troubleshooting and Monitoring - 6/4/

10 Figure 9b Histogram of 6 IPTV Streams at 2 Locations (IQ-A and IQ-B) - Charted over a 4 Hour Period Further Analysis: Using the Overnight Results to Focus the Analysis Looking at the trends and periodic tendency of the media loss events shows that the IP network does not look like the cause of these large MLT counters. The fact that the same loss events are recorded at 2 locations over 11 hours suggests a potential issue at the encoder. There may still be IP packet loss from the encoder or the video TS is corrupted in some way. The next step is to isolate the fault using IQMediaAnalyzer, which looks deeper into the payload of the video. The Singulus G1-T can record several megabytes of video for inspection. As shown in Figure 10, the IQMediaAnalyzer in analysis mode records in loop mode waiting for a loss event, which should occur in about 30 minutes from our prior analysis. When the loss event occurs, the recording is stopped and stored which allows further inspection of the IP payload and MPEG TS video clip. Figure 10 IQMediaAnalyzer Recording a Suspected Bad IPTV Stream Loop recoding in hardware Rural Telephone and IneoQuest Case Study for IPTV Troubleshooting and Monitoring - 6/4/

11 At this point a large amount of diagnostic information has been collected. Confidence in the network has been established as Singulus G1-T IQ-A & IQ-B recorded very similar events overnight. Using the MDI counters the problem was isolated to one encoder IP address in particular. A handful of streams originating from an encoded port had very high MDI numbers for IP cumulative jitter and media loss. At this point the encoder has been identified as the cause of MDI loss. Upon further investigation, Figure 11 shows that PID 0x86 is present in a recorded segment that has exhibited media loss. In this segment, about 75 MB worth, there are only 3 PIDs that have the ID 0x86, which is the cause of the MDI-MRL magnitude 13 continuity counter errors. MDI-MLR Loss of 13 explanation: If a PID (for example ID 0x86) was periodically present in the video TS payload which had 3 x 188 byte TS packets, where continuity counters started counting from 0,1,2 and then stopped for a period, then another PID ID 0x86 continuity counters 0,1, 2 showed up soon after, what would the media loss event be? The continuity counter will count from 0 to 15 then roll over, therefore MDI-MRL would equal 16-3 = 13. Lastly, a playback of the recoded MPEG2 TS video stream (Audio and Video) demonstrated that an audio glitch could be heard in the location of the presence of the 0x86 PID. Figure 11 IQTSx MPEG2 Analyzer Inspecting Captured Video from Singulus G1-T Unit at IQ-A PID List Overview Rural Telephone and IneoQuest Case Study for IPTV Troubleshooting and Monitoring - 6/4/

12 Conclusions Finding the needle in the haystack is simple when looking in the right place While working with Rural Telephone and their equipment vendors, it was possible to quickly solve a network interoperability & configuration problem. This case study illustrates the requirement for Video over IP monitoring and analysis tools. Traditional layer 2 and 3 test equipment could not have detected these particular problems due to the nature of streaming media over IP. Monitoring transport stream (TS) loss, PCR bit rate and IP cumulative jitter on a per stream basis was the only way to isolate impairments caused by the encoder. The ability to simultaneously monitor and analyze all streams allowed for quick fault identification, which had the benefit of saving time and reducing travel costs. The MDI, helped track, identify, monitor and debug the problem. Without MDI, this issue could have gone on undetected for a long time causing customer frustration and potentially a reduction in take rates. This case study demonstrates the value of remote fault analysis and monitoring. Equipment vendors could perform similar analysis from remote sites to improve overall effectiveness and customer satisfaction. Video over IP is an emerging technology and IneoQuest is committed to being flexible and working with the service providers and equipment vendors to improve the quality of experience for the customer. Monitoring and measurements will be critical for IPTV deployments and QoE for next generation converged network services. Working with telecoms like Rural Telephone enabled the evolution of IPTV, providing a higher quality service and better understanding of interoperability problems found in Video over IP networks. IneoQuest would like to thank Rural Telephone for their support in developing this case study. IneoQuest Technologies, Inc Forbes Boulevard - Mansfield, MA Toll free: Fax: IneoQuest Europe - 12 avenue des Pres Montigny Le - Bretonneux - France Phone: Fax: Rural Telephone and IneoQuest Case Study for IPTV Troubleshooting and Monitoring - 6/4/