The correct testing method is vital for the evaluation of VoIP services

Transcription

1 The correct testing method is vital for the evaluation of VoIP

2 No part of this brochure may be copied or published by means of printing, photocopying, microfilm or any other process whatsoever without prior permission in writing by Nextragen GmbH; nor may it be stored, processed, reproduced or distributed using electronic systems. Kindly note that the used designations and brand names of the respective companies are subject to trade or brand mark protection or are protected by patents. Copyright: 2012 Nextragen GmbH Stand: 02/2012 Publisher: Nextragen GmbH Lise-Meitner-Str Flensburg Germany 2

3 Management Summary TraceSim evaluates the speech quality of VoIP calls based on several testing methods. These methods are very different from each other and are suitable for differing purposes. In addition, not every method responds to the individual requirements of the respective testing situation. This paper will discuss the basics and fundamental differences of two of the most common measurement methods: E-model and PRESQ algorithm. Generally, all testing methods evaluate the speech quality of VoIP connections on a scale of one (poor quality) to five (excellent quality). This system is called Mean Opinion Score (MOS). MOS is defined in the ITU-T standard P.800. However, an MOS value stated in a testing protocol or in a Service Level Agreement (SLA) can only deliver an interpretable result of the actual network quality if the underlying testing method is known. Depending on the method various MOS values can be measured. For instance, MOS values resulting from measurements according to the E-model and the PESQ algorithm cannot be compared with each other. As a result, it should be made clear which testing method the MOS value is based on, which can be expressed as follows: MOS E-model = MOS value according to the E-model MOS PESQ = MOS value according to PESQ The ITU standard goes even further to define in more detail the characteristics of the MOS values. In practice and for testing purposes, however, information about the respective testing method used suffices. First, let us turn out attention to the E-model. The E-Model Considers Packet Parameters Only The E-model is defined in the ITU standard Rec. G.107. During E-model tests, the data streams of the RTP packets (RTP streams) are captured and evaluated according to the behaviour of the packet parameters. Such parameters include: Packet loss: packets which get lost during transportation; Jitter: average deviation from the network mean latency; Delay: delay between the sender and the receiver; Codec: VoIP codec used to decode the language. The four parameters are subsequently provided to the E-model algorithm which calculates the MOS value of the RTP stream. The E-model checks whether all packets of the respective RTP stream have arrived. In simple network setups (with one media gateway), this allows for an interpretable network quality evaluation. For instance, if the testing results of a simple LAN deliver negative MOS values, the RTP streams are obviously corrupted and the desired speech quality cannot be produced. Also, a series of subsequent tests can show whether the quality has improved or not. Yet, this testing method quickly comes up against limiting factors. The E-model can only measure the quality in LANs, but not the content of the RTP packets which carry important information, such as language. Neither can the E-model deliver proper MOS values when testing across various network segments. Figure 1 illustrates the methodological limitations of the E-model. The example shows a typical company network with two VoIP network locations covering three network segments (IP - ISDN/WAN - IP) which are connected via gateways. Such gateways convert VoIP packets into ISDN packets and vice versa. 3

4 Figure 1: The E-model can only measure erroneous RTP packets within the respective network segment. Errors which have been inherited from one segment to another are ignored since the correction packets do not carry information about the language (here: packet loss). MOS values of the VoIP quality based on the E-model method do not provide any information about the end-to-end (covering all three segments) network quality. Network A and network B must be measured separately and will deliver different MOS values. Codec errors in the gateways and transmission errors in the ISDN network cannot be identified. Reasons include during packet conversion from one format into the other the time stamps are newly generated, packet errors (jitter or placket loss) are reset. This measurement method considers packet layers, yet no packet content (the actual language). E-model tests are only carried out from one measurement point to another measurement point. Consequently, the E-model fails to calculate the end-to-end quality values and cannot deliver a complete evaluation of the entire network setup. If instead of the ISDN connection, a WAN is installed between network A and network B, network providers will often introduce a codec conversion, e. g. from G.711 to G.729, to save bandwidth. This conversion divides the network again into three segments. If such network setup is tested according to the E-model, no values about the end-to-end network quality can be delivered. Instead, the measured MOS values can only serve to determine the quality of the individual segments. The problem remains: inherited errors are ignored in the evaluation of the subsequently measured network segments. PESQ Takes The Transmitted Language Into Account The PESQ algorithm is based on the specification defined in the ITU-T standard Re. P.862. The PESQ algorithm uses the language signals in order to calculate the MOS values. A reference signal is compared with the received signal sent across the entire network. This method takes into account the language quality between the sender and the receiver on an end-to-end basis. 4

5 Figure 2: MOS values based on PESQ. Changes and quality impairments at the receiver s end are identified by comparing the reference signal with the received signal. The MOS value delivers an end-to-end evaluation of the speech quality of the entire network. The PESQ algorithm measures the speech signal which means that any signal changes or errors in the transmission path are registered. Compared to the E-model (which only provides information about packet transportation errors (e.g. packet loss, jitter, etc.) within a network segment), the PESQ algorithm evaluates the entire network from end to end. Also, quality impairments (errors occurring during codec conversion, e.g. from IP to ISDN), resulting from the conversion of codecs at network gateways are measured. Conclusion The difference between the two measuring methods lies in the parameters which are reflected in the MOS values. The E-model only refers to the characteristics of RTP packets. PESQ, on the other hand, calculates MOS values based on real speech signals. In particular, when a communication path is made up of several segments including media gateways, the E-model delivers ambiguous statements which can lead to a misinterpretation of the actual speech quality. Yet, the E-model is used on most of the measurement solutions which is due to two reasons: in simple setup the E-model delivers MOS values which sufficiently reflect the actual speech quality, the algorithm can be implemented into measurement devices whilst saving on licenses (other methods require licenses). However, as learned earlier on, stating of MOS values without indication on which testing method (E-model or PESQ) the values are based, does not provide any information about the actual network quality. Today, PESQ is the market standard by providers of professional measurement solutions. Due to recent advances and developments in the IP sector, the PESQ algorithm no longer covers all relevant network parameters. In 2011 the ITU introduced a new recommendation: ITU-T Rec. P.863 named Perceptual Objective Listening Quality Analysis (POLQA). POLQA upgrades PESQ and adds significant new capabilities for wideband and super-wideband ( Hz) voice signals along with support for most recent voice coding and VoIP transmission technologies. POLQA is suited to evaluate, optimize and monitor the voice quality of next-generation networks. In the course of the expansion of Next Generation Networks over the next few years, POLQA will become more and more relevant for providers of fixed IP networks and of wireless networks. 5

6 About Nextragen Nextragen GmbH is specialised in the development of Monitoring, Analysing and Testing software for VoIP/ video solutions to ensure the quality of End2End (QoS, QoE) for Next Generation Networks and triple-play. The company was founded in 2009 and is based in the Flensburg in Germany. Nextragen s customers, including carriers, enterprise customers and telecommunication businesses, make use of our solutions to monitor, analyse and test the quality, reliability and availability of VoIP and video applications. Nextragen s products, solutions and are 100% made in Germany and are distributed globally through certified partners. For more information, visit the company website at 6

7 Nextragen GmbH Lise-Meitner-Str Flensburg Telefon: Fax: Errors and omissions expected. 7