NTS Testing Labs Voice over IP (VoIP) Product Test Lab Compatibility & Functionality Test Outline Revision 1.0 NATIONAL TECHNICAL SYSTEMS The NTS Mission: Assisting our Clients in Navigating a Short Course to Global Markets Written by: Dmitri Grigorev Approved by: Osman Sakr Quality Assurance Approval Derek Fakehany Date 2/22/02 Date 2/22/02 Tracking No. WP201002 Rev. 1.0 FileName NTS White Paper - VoIP - WP201002.doc Copyright 2002, National Technical Systems, Inc. This document is provided to clients and potential clients of National Technical Systems (NTS) for informational purposes only. It may not be reproduced or stored, in whole or in part by any means, with out the express written permission of NTS. Document Tracking # WP201002 NTS White Paper - VoIP - WP201002.doc 1.0 Dmitri Grigorev 02/02
Table of Contents 1. Introduction...1 2. Scope of Work...1 2.1 Overview...1 2.2 Sample Test Procedures...2 2.2.1 Customer Premises Equipment (CPE)...2 2.2.2 Internet Protocol (IP)...3 2.2.3 Public Switched Telephone Network (PSTN)...4 2.2.4 Signaling System 7 (SS7)...4 3. Project Deliverables...5 3.1 NTS Test Reports...5 3.2 Test Matrix...5 3.3 Project Documentation Report...5 3.4 Report Transmission...6 3.5 Project Leadership...6 4. Conclusion...6 i
1. Introduction National Technical Systems (NTS) Testing Laboratories currently conducts Voice over IP (VoIP) Product Compatibility / Functionality Testing for a wide array of product manufacturers. Various types of Voice over IP (VoIP) Products exist in the market place. This document is not to be interpreted as a test plan or a test outline due to the complexity of technology involved, but rather as a general approach to test methodology development. 2. Scope of Work 2.1 Overview NTS typically conducts Teleconferencing Compatibility / Functionality Testing to verify that the product under test functions correctly without anomalies, difficulties or discrepancies. Below is a drawing that demonstrates the network under test, which will be discussed further in this document: For control protocol and gateway-gateway communication H.323 is used in this example. This protocol provides voice and video connections over IP network with no guaranteed QoS (Quality of Service). The scope of this sample test outline is: Verify Voice/Video Quality H.323 Standard Compliance SS7 Protocol Integrity Details of each of these items will be discussed in appropriate sections. - 1 -
2.2 Sample Test Procedures Before NTS can begin testing, equipment must be setup and configured to perform testing with the product-undertest. Once the test environment is configured, NTS Testing Engineers will confirm that the test environment appears to be functioning as expected. The quality assurance testing typically consists of the test areas outlined below. During testing, NTS Test Engineers typically exercise the gateway and gatekeeper functionality of the Voice over IP (VoIP) product. The gateway in focus will have four (4) interface sides: CPE Customer Premises Equipment IP Internet Protocol PSTN Public Switched Telephone Network SS7 Signaling System 7 Below is a table with corresponding test for each interface: Test Suite Performed Interface Voice/Video Quality CPE, PSTN H.323 Compliance IP SS7 Integrity SS7 The gatekeeper will provide connection control and other functions as specified by H.323 standard. IP interface tests will include gatekeeper tests as well. 2.2.1 Customer Premises Equipment (CPE) There are several voice and video tests that are performed on the CPE side. Voice tests are: PSQM PSQM (Perceptual Speech Quality Measurement) is a means for objectively assessing the quality of speech that has been degraded by a telephony network. It has a high correlation to subjective quality across a range of types of distortion, and is appropriate for testing networks that are subject to different coding types and transmission errors. This test is performed by an Abacus Test System (The Abacus system is a modular and scalable test platform that generates telephone traffic [bulk call simulator] and switches telephone traffic [central office emulator] simultaneously within the same system over a variety of telecom interfaces.). It will originate the call and also terminate it. Once the path is established and confirmed, it will transmit a voice sample over the path. This voice sample can be artificial speech or a recording of real speech. Once the sample is transmitted, the Abacus will perform PSQM algorithm processing and return a value in range 0 6.5; 0 being a very good channel and 6.5 being a very bad channel. PSQM+ PSQM+ is essentially the same test as PSQM except for some post processing. PSQM+ maps out better to the MOS table result, but is not yet ITU recommended. MOS MOS (Mean Opinion Score) is a subjective test performed by the user. In this test actual H.323 connection has to be established by two (2) users. Both evaluate the quality of sound on a scale 1 5; 5 being very good channel and 1 being unacceptable quality. Path Verification Path verification is another test performed by the Abacus Test System. The call generator will originate and terminate the calls. Once the call is terminated the path is verified by transmitting tones over the connection and measuring attenuation on the receive side. There are two types of verification available: 3-tone and resilient. 3-tone verification involves transmission of 3 tones at marginal frequencies and resilient is a single tone test. - 2 -
Jitter Jitter measurement is an evaluation of variation in transmission delay. This is a very important parameter in telephony network as it has the most impact on quality of connection. The test is performed by measuring delays over a period of time and then charting the dependency. Echo Cancellation Components in voice networks, usually, are quipped with Echo Cancellation feature. This function nullifies effects of reflection signals in the path. Testing this feature is done by two users sharing a talk path. The connection is tested for presence of echo as conversation goes on. Stress/Stability Stress/Stability test utilizes an Abacus to create maximum utilization of the gateway by generating/terminating calls on all available CPE ports. The test is run over a period of time of at least 24 hours. The results of the testing are analyzed by considering amount of dropped calls and calls with unconfirmed paths. The video tests executed include: Packet Loss, Latency, and Throughput Packet Loss, Latency, and Throughput tests are executed using Chariot software package, which generates data traffic and analyzes various aspects of network performance. Test criteria for these measurements is determined by manufacturer s specifications. Frame Rate Frame rate tests require special features of conferencing software. It is a task of the video application to measure the frame rate. Most of the conferencing software packages today provide frame rate measurements. 2.2.2 Internet Protocol (IP) There are two (2) areas of testing on the IP interface for which both interoperability and functionality tests are typically executed: Gateway-Gateway Gateway-Gateway communications include all transmissions of connection data as well as control components. This connectivity is achieved by translating protocols for call setup and release, converting media formats between different networks, and transferring information between the networks connected by the gateway. All these features are tested as summarized below: Call Control Call Control tests verify correct communication with the gatekeeper and another gateway as it is originated by the gateway under test. Proper handshaking and session control are verified by monitoring the line with protocol analyzer. Media Formatting Media Formatting tests evaluate the ability of the gateway to correctly and timely encode incoming media into transmission protocol. These tests are unit specific and are defined in accordance with manufacturer s specifications. TCP/IP Functionality TCP/IP functionality suite is designed to validate compliance of the unit under test with IP standards as they are defined in corresponding RFCs. These tests cover functionality and interoperability issues up to TCP/UDP layer. Gateway-Gatekeeper Gateway-gatekeeper transmissions include connection control signaling and handshaking. A gatekeeper can be considered the brain of the H.323 network. It is the focal point for all calls within the H.323 network. Following is the list of services for terminals and gateways that are tested under gateway-gatekeeper communications: Addressing Authorization Authentication Bandwidth management Billing - 3 -
2.2.3 Public Switched Telephone Network (PSTN) PSTN interface evaluation is typically accomplished during CPE testing. Since voice quality and other parameters are measured then, it does reflect the degree of functionality of PSTN interface. However, there are interoperability issues and physical layer tests that are also involved. In most cases, the gateway will use a T1 trunk to connect to the PSTN cloud. This link is tested per ANSI standards using the following test outline: General Electrical Pulse imbalance Power Level Power Density Receiver Impedance and RL Receiver Signal Parameters Signal Characteristics at NI Jitter/Wander/Phase Transients Network Signal at NI CI signal at NI Power Patterns Framing Formats Maintenance Verification Loopbacks Functional Performance Details of these tests are beyond the scope of this document. More information can be found in ANSI T1.403 standard documentation. 2.2.4 Signaling System 7 (SS7) Signaling System 7 (SS7) is an architecture for performing out-of-band signaling in support of the callestablishment, billing, routing, and information-exchange functions of the PSTN. It identifies functions to be performed by a signaling-system network and a protocol to enable their performance. Following is a general outline of the tests performed: Physical Layer Physical layer tests evaluate the physical and electrical characteristics of the signaling links of the SS7 network. Signaling links utilize DS 0 channels and carry raw signaling data at a rate of 56kbps or 64kbps (56 kbps is the more common implementation). Message Transfer Part Level 2 MTP Level 2 testing provides verification of link-layer functionality. It ensures that the two end points of a signaling link can reliably exchange signaling messages. It incorporates such capabilities as error checking, flow control, and sequence checking. Message Transfer Part Level 3 MTP Level 3 testing extends the verification of functionality provided by MTP level 2 to provide network layer functionality. It ensures that messages can be delivered between signaling points across the SS7 network regardless of whether they are directly connected. It includes such capabilities as node addressing, routing, alternate routing, and congestion control. - 4 -
Signaling Connection Control Part SCCP test provides verification of two major functions that are lacking in the MTP. The first of these is the capability to address applications within a signaling point. The MTP can only receive and deliver messages from a node as a whole; it does not deal with software applications within a node. While MTP network-management messages and basic call-setup messages are addressed to a node as a whole, other messages are used by separate applications (referred to as subsystems) within a node. Examples of subsystems are 800 call processing, calling-card processing, advanced intelligent network (AIN), and custom local-area signaling services (CLASS) services (e.g., repeat dialing and call return). The SCCP allows these subsystems to be addressed explicitly. Global Title Translation The second function provided by the SCCP is the ability to perform incremental routing using a capability called global title translation (GTT). The test of this feature verifies the ability of GTT to free originating signaling points from the burden of having to know every potential destination to which they might have to route a message. A switch can originate a query, for example, and address it to an STP along with a request for GTT. The receiving STP can then examine a portion of the message, make a determination as to where the message should be routed, and then route it. Transaction Capabilities Application Part (TCAP) TCAP testing is designed to monitor and analyze how TCAP defines the messages and protocol used to communicate between applications (deployed as subsystems) in nodes. It is used for database services such as calling card, 800, and AIN as well as switchto-switch services including repeat dialing and call return. Because TCAP messages must be delivered to individual applications within the nodes they address, they use the SCCP for transport. Operations, Maintenance, and Administration Part (OMAP) Testing of OMAP verifies that it properly defines messages and protocols designed to assist administrators of the SS7 network. To date, the most fully developed and deployed of these capabilities are procedures for validating network routing tables and for diagnosing link troubles. OMAP includes messages that use both the MTP and SCCP for routing. 3. Project Deliverables NTS clients receive the following standard NTS project deliverables during or at the conclusion of tests performed by NTS. (Project specific deliverables may also apply depending upon the specific project type). 1. NTS Test Reports (daily / as generated) 2. Test Matrix and/or Test Status Report 3. Project Documentation Report The Test Matrix and NTS Test / Problem Reports are transmitted to client daily or as developed. The Project Documentation Report is delivered following the conclusion of testing. 3.1 NTS Test Reports During testing, NTS generates reporting to document all testing performed. NTS test reporting is designed to capture configuration information, general procedures, test measurements and other test results related to all tests performed. 3.2 Test Matrix The NTS Test Matrix provides a quick overview of the status of entire project. The Test Matrix is a master grid outlining individual tests or test milestone coverage on the project. The matrix provides a quick overview of tests performed to date, with columns specifying the product(s) tested, a pass/fail/warn status, the date performed, test engineer identification, and any related Test Report numbers. 3.3 Project Documentation Report NTS binds the final copy of the project test matrix along with all NTS test reports and delivers this to clients following the completion of testing. - 5 -
3.4 Report Transmission NTS uses Internet e-mail as much as possible to transmit status reports, test reports, test matrices, documentation reports, etc. In some cases, materials may also be faxed or send via FedEx. If additional security is required, NTS will utilize encryption to prepare and send files in batches, using an agreed upon encryption mechanism (to be decided between NTS and a particular client prior to test execution). Catastrophic and critical problems are prioritized and reported upon as soon as possible. 3.5 Project Leadership A project leader is assigned to projects NTS performs for clients. The NTS project leader communicates with clients regarding test progress, status, and findings. The project leader s primary responsibility is to insure that testing is being executed as agreed upon, to provide management and technical expertise, and to work with the Client to make any required changes. The project leaders are supervised by NTS Operations Manager. If major revisions to a test plan is required, or if a client is not satisfied with the testing process, the client is encouraged to contact the Lab Quality Manager immediately. It is NTS goal to provide testing that meets Client s needs as accurately as possible. 4. Conclusion NTS Testing Laboratories has decades of experience and unsurpassed expertise in the area of compatibility and interoperability testing. Fewer product recalls, faster time to market, reduced technical support requirements and higher customer satisfaction are but a few of the many potential benefits of testing a product with NTS. A Voice over IP (VoIP) Product can provide a wide range of features. Some of which are more basic, or far more complex than those described above. Variations in a test plan vary depending on the features of the product and the test needs of the client. Voice over IP (VoIP) Product compatibility and functionality testing fill the need for real-world environment testing. The benefactors of this type of compatibility and functionality testing are lower product recall rates, lower volume of customer support calls and higher consumer satisfaction. - 6 -
NATIONAL TECHNICAL SYSTEMS National Technical Systems (NTS) is one of the largest consulting and certification laboratories in the United States. NTS operates extensive test facilities, throughout the United States and Japan, covering operational environment, stress environment, compatibility, interoperability, broadband network testing, EMI/EMC, FCC, NEBS, USB, WHQL, Product Safety testing and ISO Audit/Certification (we certify the National Space Administration for ISO compliance). NTS is a Nationally Recognized Testing Laboratory (NRTL) - as certified by the United States Government. We are accredited to ISO 17025 by the American Association for Laboratory Accreditation. NTS holds many certifications from foreign governments as a Conformity Assessment Body (CAB). We are the only one stop certification and compliance facility offering a broad range of testing capabilities. These capabilities include significant financial resources, numerous certification programs and a high level of lab security. We support extensive test procedures, operating systems and hardware. We are fully equipped for broadband testing and offer such services as DOCSIS third-party pre-certification and DSLAM compatibility and functionality testing. A variety of PSTN and Internet access methods are available ranging in speed from dial-up to 1Gb/sec. Several independent T1 Internet access points and frame relay circuits are available. An OC-12 link permits access ranging from standard T1 to OC-12 (644 Mb/sec) in a variety of formats: ATM, Frame Relay, SuperTrunk, HiCap and various other architectures. Our particular testing laboratory is located in Culver City, California. This facility is centrally located and is just 15 minutes from Los Angeles International Airport. IT IS EASY TO START! Send us your wish list covering the areas that you would like addressed in your test, and we will design a test proposal specific to your needs that covers only what you want tested. To turn your request into a formal (no cost to you) proposal, we need the following: 1. Product description and or specification. 2. Target Market What market your product will sell into. 3. Start and end date When you need to start and end this test. 4. Company information Name, contact, address, phone, fax and e-mail. 5. Any concerns about your product that you have or you know about. Let us know if you want a test proposal for your particular project. Typically, our Proposal Department can prepare one for you within five (5) working days, although more complex products may require additional time. The proposal will contain our test methodology, project schedule, defect classifications, regression information, and the project cost. This proposal should be considered a working document that you can change or modify to fit your exact test requirement and budget Our goal is to work together so that you can achieve your objectives. CONTACT US National Technical Systems Computer Testing Division 5730 Buckingham Parkway Culver City, CA 90230 Telephone: (310) 641-7700 Fax: (310) 641-5827 Web: http://www.ntscorp.com/ E-mail: labsales@ntscorp.com