White Paper Troubleshooting CDMA Packet Data Networks to Ensure a Positive Customer Experience By: Maurice Lampell, Solution Architect Introduction The rising popularity of smartphones has led to massive growth in the volume and complexity of data traffic carried on wireless networks. This explosive growth will continue and will continue to contribute to customer ARPU only as long as customers are happy with the quality of their service. In this volatile environment, subscribers and operators share a crucial point of potential dissatisfaction: support calls. These are becoming more complex, are taking longer to resolve, and can cost millions in customer care. A variety of issues can lead to dramatic increases in the number and complexity of support calls: The mere introduction of data-capable smartphones and the accompanying services Network element configuration errors that impact data services The guarantees and expectations that accompany corporate or VIP service agreements Diminished service quality caused by rogue users consuming massive amounts of bandwidth Complaints from customers who are shocked by unexpected charges for data usage they didn t initiate such as automatic updates The ability to quickly identify the underlying problems and successfully resolve support calls depends on a highly scalable assurance solution that provides full-time monitoring and includes an integrated suite of monitoring and troubleshooting applications. A scalable solution that provides speed, accuracy and insight will put operators on the fast track to improved revenues, happy subscribers and a growing customer base. Speed: When answers are no more than three clicks away, a prompt, successful resolution is more likely. Accuracy: With accurate data from the network, operators and their support personnel can be more confident in their decisions. Insight: An end-to-end solution with comprehensive real-time and historical data provides a broader view for fast, accurate troubleshooting. The remainder of this paper provides a closer look at the issues faced by network operators; describes monitoring methods and optimum monitoring points; outlines the key elements of an advanced solution; and presents an example based on the JDSU CDMA Assurance Solution. Issues faced by network operators Operators with CDMA packet data networks face a number of multidimensional challenges, spanning handset proliferation, rapid service rollout, enterprise customers, roaming partners and network quality. A closer look at each of these will help highlight the potential issues. Handset proliferation Market forces have driven the rate of new handset introductions to unexpected levels. This proliferation is accompanied by a number of assurance challenges: Device configuration and authentication Device performance in general, and when using specific services WEBSITE: www.jdsu.com/test
White Paper: Troubleshooting CDMA Packet Data Networks to Ensure a Positive Customer Experience 2 Rapid service rollout Along with new handsets, operators are competing for consumer attention with new services that are vibrant and current. However, this introduces other challenges: Achieving a profitable service uptake rate. Understanding how uptake varies by variables such as handset type or customer segment. Measuring, monitoring and assuring adequate performance of those services. Ensuring that the services are completing successfully. Enterprise customers and roaming partners To increase coverage, it s critical to establish internetworking agreements with roaming partners, and to ensure that those interconnections work properly. Establishing a broad coverage area is also the key to attracting large enterprise customers. This introduces more potential issues: Ensuring that all network elements are provisioned properly to allow roaming. Measuring the quality of service being offered to inbound and outbound roamers. Ensuring that VIP customers are able to access their custom intranet services. Network quality Ultimately, data packet services can be successful only if they are deployed on top of a reliable network. Ongoing network assurance must be able to answer key questions like: Is traffic balanced across network elements? Is there any latency or data loss in the network? Are there pockets of signaling or user-plane issues? Is every element properly configured? How can the network quality be ensured as the network evolves and grows? Operators that meet these challenges can deliver consistent, high-quality service for their customers. The first step is choosing monitoring methods and monitoring points that provide a clear view of what is really happening within the network. Monitoring points The widely distributed nature of the CDMA packet data network infrastructure makes it difficult to extract meaningful metrics for network and service utilization. It is even more difficult to obtain a cohesive view of how signaling and user-plane information is flowing. Fortunately, viable monitoring solutions and optimal monitoring points are readily available. Monitoring methods Monitoring of CDMA packet data can be accomplished in two ways. One is to use network element-based information, such as billing records, peg counts and log files. The other is to use an external monitoring system that collects signaling and user-plane information to determine what is occurring within the data network. In both cases, it s necessary to write applications that extract the information, correlate the information across the various network elements or data collection points, and present the results. These applications have traditionally used network element records because it is seemingly cost-effective to incorporate the assurance function into existing equipment. Unfortunately, this method has some noteworthy drawbacks: The accuracy of network element information is highly dependent on the traffic load being experienced by the network element (and it may also depend on the capabilities of any given software release). Element vendors make predetermined choices about what information to export via APIs or log files, and how to handle processor/memory overloads. Because essential network-element functionality must take priority over assurance monitoring, the assurance feed will become less reliable when the system is congested. Network elements typically do not create sufficiently detailed records for failed data session analysis, nor do they provide timely enough reporting to enable real-time alarming. In many cases, it is important to know whether a data session failed because the user was busy or the network was busy. User busy is an opportunity to sell more service but network busy may be an indication of capacity problems. Multiple network elements must interact to create successful data sessions and enable user data to flow. Often four or five elements (e.g., FA, HA, AAA and PCF) are involved in the set-up process and a complex series of exchanges takes place between them. To fully understand the flow of an endto-end data session, it is necessary to correlate the individual signaling protocols into a single, unified view. External monitoring systems can operate independently of the traffic load experienced by network elements, and don t depend on which software release an element is running. What s more, an external system is able to create an impartial view of the network traffic, reflecting the reality of what is being transmitted.
White Paper: Troubleshooting CDMA Packet Data Networks to Ensure a Positive Customer Experience 3 Optimal monitoring points Figure 1 illustrates the various network elements that are involved in the delivery of CDMA packet data services. There are four principal element types: PCF/RNC, PDSN/FA, home agent (HA) and AAA server. For the monitoring of this network, a key question is, Where should it be tapped for maximum efficiency and effectiveness? Signaling information for tunnel set-up is available from the RP interface and must be collected to fully understand potential failures. The Mobile IP component of the tunnel setup is present in the P-H interface. Authentication information can be monitored at various points in the network: at the PCF/RNC, PDSN and HA; or at the AAA sites. User-plane information can be extracted in three different places: the A10 tunnel, the P-H tunnel, or the Pi interface to the Internet. The degree of difficulty in extracting the user-plane data varies by interface and may require unraveling several levels of tunneling. The most cost-effective location to monitor is around the PDSN sites. These offer access to the A11 signaling, the A10 user plane, the P-H interface and the Pi interface. Monitoring at the PCF/ RNC sites is not practical due to the large number of sites. Instead, it is simpler to extract user-plane information from the A10 tunnel and to discard the P-H user-plane data. By monitoring at the AAA sites, authentication information can be obtained from the three other types of network elements in a single location, offering cost-effective monitoring. Finally, HA monitoring is optional, given that the HA ingress interface has already been monitored at the PDSN sites and the HA egress interface can be seen at the AAA sites. The only loss of visibility is the Pi traffic emerging from the HA to the Internet but this traffic is already available at the A10 tunnel. Figure 1. Key network elements in the delivery of CDMA packet data services
White Paper: Troubleshooting CDMA Packet Data Networks to Ensure a Positive Customer Experience 4 Solutions For CDMA packet data network operators, a comprehensive network and service solution should include four key elements: session tracing; ad hoc real-time reporting; QoS and KPI analysis; and a data-mining toolkit. The JDSU CDMA Assurance Solution suite provides examples of these capabilities (Figure 2). Session Trace permits users to retrieve specific customer sessions by NAI, IMSI or ESN and time-period, independent of where the customer s network interaction took place. Responses appear within 30 to 90 seconds of their occurrence, enabling near-realtime troubleshooting. The QoS Analyzer invisibly monitors the aggregated performance data arriving from each of the monitored points every five minutes. Performance data is summarized into several hierarchies, permitting analysis by network element, service, enterprise customer or realm. For each measurement, the system automatically creates a historical baseline of its values to enable comparisons with future measurements. Various threshold levels can be programmed to support different levels of alarm urgency. Ad hoc real-time reporting (AHRTR) is designed to focus attention on a specific slice of the network, helping users verify changes. This is a peg-counting application and feedback from AHRTR occurs in near-real-time (one minute), making it a suitable tool for network change control. For example, the cutover of a new network element can be closely monitored as it occurs. Users can be very specific about the conditions they want to match for any given measurement. The enterprise data warehouse stores information collected from the JDSU CDMA Assurance Solution. This information is accessible through business-intelligence applications. Session Trace Real-time and historical troubleshooting Protocol-level output Documentable Latency: none History: 7 days Ad-hoc Real-Time Reporting Verification of network changes Concentrated analysis of a small segment of the network Latency: 1 minute History: 30 days QoS Analyzer/KPI Analyzer Alarming on Nodes & Services Performance metrics Top/Down, Bottom/ Up views Drill-down to Session Trace Latency: 5 minutes History: 6 months Data Mining Toolkit Long term trends Detailed analysis of traffic, broken down by multiple dimensions Full detail for 1 day Latency: 15 minutes History: 1 year Figure 2. Four major elements of a comprehensive network and service solution
White Paper: Troubleshooting CDMA Packet Data Networks to Ensure a Positive Customer Experience 5 Identifying and resolving service performance issues The following example highlights useful monitoring and troubleshooting that can help increase the likelihood of first-call resolution, reduce the duration of support calls and enhance customer satisfaction. The screen shots are from the JDSU CDMA Assurance Solution. In just three clicks, a user can navigate from an error bar to a service-usage screen that provides information essential to identifying the root cause of a problem. Within the QoS Analyzer, a service model identifies and alarms on threshold violations of the success rate for Web services that occurred on a specific date and time (Figure 3). As can be seen in the figure, the normal error rate is approximately 2 percent but spiked up significantly during the time period of interest. Figure 3. Chart of error rate for Web services First click: By clicking on one of the bars that exceeds the pre-defined threshold, the user is presented with the error breakdown shown in Figure 4. Cause-code counts are presented for each error type: timeout, service error and unidirectional packet flow. Figure 4. Breakdown of errors from a single error bar
White Paper: Troubleshooting CDMA Packet Data Networks to Ensure a Positive Customer Experience 6 Selecting a TopN analysis by IMSI for the HTTP Activation Status Error Timeout cause code (Figure 5) identifies the TopN IMSIs that experienced the problem at that time. As shown in the figure, most of the errors are associated with a single IMSI. Second click: Selecting a specific IMSI launches the Session Trace application, prepopulating the required parameters and opening a Session Trace window that displays the offending sessions for the selected IMSI (Figure 6). Figure 5. In the KPI Analyzer, each list of error types includes cause-code counts Figure 6. Session Trace displays provide insight into error-related sessions
White Paper: Troubleshooting CDMA Packet Data Networks to Ensure a Positive Customer Experience 7 Third click: Finally, double-clicking on any of the displayed messages provides access to service usage summaries that provide additional information about service data, transport data and more (Figure 7). These screens provide information that is essential to finding the root cause of a problem: HTTP URI of the accessed Web site IP addresses of network elements, the user device and the ASP Byte counts, throughput measurements and latency measurements In this case, the screens identify a rogue wireless device that was making repeated HTTP queries to a specific Web site. With that information in hand, the operator was able to take action and remedy the situation. Figure 7. Summaries of service data (top) and transport data (bottom)
White Paper: Troubleshooting CDMA Packet Data Networks to Ensure a Positive Customer Experience 8 Conclusion Industry forecasts suggest continued growth in the volume and complexity of data traffic carried on wireless networks. In this dynamic environment, customer satisfaction often depends on the ability to quickly identify underlying problems and successfully resolve support calls. One essential tool is a highly scalable assurance solution that provides full-time monitoring and includes an integrated suite of monitoring and troubleshooting applications. As shown here, the monitoring of external network interfaces can provide an adequate and impartial view of the network by collecting signaling and user-plane information and using it to see what s really happening within the data network. The JDSU CDMA Assurance Solution is a complete solution suite that addresses surveillance, troubleshooting and long-term planning through speed, accuracy and insight. It includes four key elements: session tracing; ad hoc real-time reporting; QoS and KPI analysis; and a data-mining toolkit. With these tools, network staff is just three clicks away from actionable information that will enable them to identify underlying problems, make informed decisions and, ultimately, improve customer satisfaction. Glossary AAA: ASP: CDMA: ESN: HA: HTTP: IMSI: IP: KPI: NAI: PCF: PDSN-FA: PDSN-HA: QoS: RNC: URI: VIP: Authentication, authorization, and accounting Application service provider Code Division Multiple Access Electronic serial number Home agent Hypertext transfer protocol International mobile subscriber identity Internet protocol Key performance indicator Network address indicator Packet control function Packet data serving node, foreign agent Packet data serving node, home agent Quality of service Radio network controller Uniform resource indicator Very important person Test & Measurement Regional Sales NORTH AMERICA TEL: 1 866 228 3762 FAX: +1 301 353 9216 LATIN AMERICA TEL: +1 954 688 5660 FAX: +1 954 345 4668 ASIA PACIFIC TEL: +852 2892 0990 FAX: +852 2892 0770 EMEA TEL: +49 7121 86 2222 FAX: +49 7121 86 1222 WEBSITE: www.jdsu.com/test Product specifications and descriptions in this document subject to change without notice. 2010 JDS Uniphase Corporation 30168044 500 0410 CDMA.WP.NSD.TM.AE April 2010