Test Report: Performance assessment of vehicle-mounted mobile phones

Transcription

1 Test Report: Performance assessment of vehicle-mounted mobile phes in cjuncti with hands-free terminals based Recommendatis ITU-T P.11 and ITU-T P.111 ITU 21

2 Test Report: Performance assessment of vehicle-mounted mobile phes Background ITU organized a test event Performance assessment of vehicle-mounted mobile phes in cjuncti with hands-free terminals based Recommendatis ITU-T P.11 and ITU-T P.111 The test event, held at ITU Headquarters, May 21, analysed the behaviour of a representative sample of mobile phes available today and capable of cnecting to handsfree systems. The tests were performed by HEAD acoustics GmbH, based the Chapter 12 tests of Recommendatis ITU-T P.11 and ITU-T P.111, standards for narrow-band and wideband communicatis involving motor vehicles. The tests requirements were adapted and applied to real-world scenarios. The methodology and results of the tests event will feed into an going process to refine the standards. This report was written by HEAD acoustics which is respsible for the test cducti and the analysis of the test results. Test results have been anymized in this report. Ctact perss: Mr Denis Andreev, denis.andreev@itu.int, Tel Mr Marc Lepage, marc.lepage@head-acoustics.de, Tel Mr Frank Kettler, frank.kettler@head-acoustics.de, Tel ii -

3 Test Report: Performance assessment of vehicle-mounted mobile phes Table of Ctents 1. Motivati Summary Cascaded Algorithms.... Test Descripti Test Setup Tests Adaptatis for the Test Event Selecti and Designati of Mobile Phes Result Representati and User Experience Analyses Results Overview - Narrowband Overview - Wideband Individual Summary Narrowband Individual Summary Wideband Cross Cnecti Tests References iii -

4 1. Motivati From a speech quality aspect, we need to regard the whole system, including the mobile phes, and not ly the hands-free unit. This cclusi from intensive discussis amg car manufacturers, hands-free suppliers and test laboratory during the ITU-T Test Event best reflects the motivati for this event: ITU-T initiated and hosted the Short Range Wireless Test Event in May 21 in Geneva [1] in order to verify the influence of mobile phes speech communicati quality of vehicle mounted hands-free systems. In the most comm use case for hands-free communicati, a driver s mobile phe is linked via wireless communicati to the vehicle s hands-free system which provides the mobile network access. Here the mobile phe acts as an audio gateway and should provide fully transparent voice transmissi in uplink and downlink. The relevant signal processing is performed solely by the vehicle s hands-free system; therefore mandating that the signal-processing functiality of a mobile phe be disabled while the phe is mounted a hands-free system. Experience has shown, however, that the internal signal processing in mobile phes is often not disabled despite the hands-free unit sending the appropriate ctrol command (AT command) to the mobile phe. In this situati, the mobile phe may significantly degrade the quality of the whole system, and car manufacturers have received complaints from customers having experienced speech quality shortfalls. Correspding tests to verify the performance of mobile phes in cjuncti with vehiclemounted hands-free terminals are described in Chapter 12 ( Verificati of the transmissi performance of short-range wireless (SRW) transmissi enabled phes ) of Recommendatis ITU-T P.11 [2] and ITU-T P.111 []. These tests have been used for testing during the event. The Test Event took place between May 12 th and 16 th at ITU-T headquarters in Geneva. tests were carried out a total number of state-of-the-art mobile phes of 12 different phe vendors in narrowband and wideband mode. The devices were selected and provided by 5 participating companies. For all tested devices the SRW transmissi was realized using the Bluetooth TM wireless technology standard. The most important findings are summarized in Chapter 2 of this report. Chapter gives a brief insight into the influence of cascaded algorithms. The setup together with a short introducti of the tests itself including adaptatis for the Test Event, an overview about the selected mobile phes and the result representati is given in Chapter. Chapter 5 analyses the results. References can be found in Chapter

5 Test Report: Performance assessment of vehicle-mounted mobile phes 2. Summary The most important findings can be summarized as follows: different mobiles from 12 different vendors were selected from the participating companies. A total number of tests were cducted in narrowband and wideband mode. The tests are based Recommendatis ITU-T P.11 and P.111 Chapter 12 together with adaptatis for the Test Event. The results and the most important findings will be fed back into the standardizati process during the next ITU-T SG 12 meeting. All tested mobile phes in narrowband and wideband mode respd with ok to the AT+= command, indicating that the internal signal processing (noise reducti, echo cancellati) is disabled. Approximately % of the tested phes violate the adapted round trip delay requirement of 21 ms. The maximum round delay was determined to nearly 5 ms for e phe in narrowband mode. Approximately % of the devices tested in narrowband mode do not disable noise reducti and echo cancellati, although they respded with ok the AT command. Customer complaints can be expected for these devices. Also other influences the transmitted signals, such as active volume ctrol the Bluetooth TM link, signal amplificati of up to 12 db in sending directi (highest measured amplificati) or active equalizers were detected during the tests. Only approximately % of the tested mobile phes can be regarded as fully transparent, as required. All -except e- devices tested in wideband mode disable noise reducti and echo cancellati. However, other signal influences like signal amplificati of up to 17 db in sending directi (highest measured amplificati in wideband mode, which will definitely lead to signal saturati and distorted voice) or active equalizers were detected. Cross cnecti tests in narrowband Bluetooth TM cnecti in combinati with wideband network access (or vice versa) indicate unexpected inband level ctrasts in both transmissi directis in case of wideband capable devices. Since dynamic codec switching between wideband and narrowband mode can occur in the mobile networks these level ctrasts are directly perceivable by the user and might lead to customer complaints. Additial tests without sending the AT+= command to the mobile phes (relevant for Bluetooth TM headsets communicating with mobile phes also via the hands-free profile), pointed out, that approximately 25% of the mobile phes disable internal signal processing, although it is not (!) requested by the accessories. This is an important issue for Bluetooth TM headsets without own implemented signal processing. In this case, the mobile phe should keep the algorithms active

6 Test Report: Performance assessment of vehicle-mounted mobile phes. Cascaded Algorithms According to the Hands-free Profile V1.6 [], a vehicle mounted hands-free system may request the mobile phe to disable the internal signal processing, such as echo cancellati and noise reducti. This is necessary in order to avoid cascaded algorithms which may significantly impair cversatial quality either in receiving directi in the vehicle or in sending directi, i.e. at the far end side. The block diagram in figure.1 indicates the two transmissi directis and shows the different compents, i.e. mobile network, the mobile phe, which acts as audio gateway between network and hands-free system, the hands-free telephe system, the audio playback system and the hands-free microphe system in the vehicle. Note that the cnecti between hands-free telephe system and mobile phe is today typically realized via Bluetooth TM. Receiving directi Sending directi Fig..1: Principle block diagram and definiti of transmissi directis Fig..2: Cascades signal processing Figure.2 shows the typical signal processing compents in this set-up. The hands-free algorithms provide the echo cancellati functiality ( ), the additial processing to suppress residual echo compents (echo suppressi ES ), noise reducti algorithm ( ) and gain adjustment including possible automatic gain ctrol ( ). Today s mobile phes provide the same kind of algorithms (see fig..2) which may lead to cascaded signal processing, if they are not bypassed or disabled. This should, under all circumstances, be avoided. The compents in the hands-free telephe system are already optimized the acoustic envirment in each vehicle. - -

7 Test Report: Performance assessment of vehicle-mounted mobile phes Experience shows that cascaded algorithms lead to degradatis in cversatial quality: Additial echo cancellati and echo suppressi significantly hamper the very important double talk performance. The echo suppressi unit ( ES in figure.2) typically introduces attenuati in the microphe path under single talk cditis, if ly the far end subscriber is talking and his voice is played back via the loudspeakers in the car. In this situati the driver hears the far end talkers voice in the car. The implemented signal processing (, ES) suppress the echo, which would be audible and annoying for the pers speaking at the far end side. If the driver interacts (both perss talk at the same time, designated as double talk ) any attenuati in the microphe path, introduced by the echo suppressi unit, needs to be quickly removed. Cascaded echo suppressi units (in case they are not disabled in the mobile phe) hamper the cversati. Driver s voice may be significantly attenuated or even partly suppressed (speech gaps in driver s voice, chopped speech), which is very annoying for the far end subscriber. On the other hand, cascaded noise reducti algorithms ( in Figure.2) degrade speech transmissi quality, especially if the driver is talking from the driving car. In this case his voice is transmitted together with background noise from the vehicle. The noise reducti algorithm shall lower the transmitted background noise without impairing driver s voice. However, due to technical limitatis quality degradati occurs. Driver s voice sounds artificial, unnatural and metallic and perhaps disturbed by other artifacts known as musical tes. This is audible and annoying for the far end cversatial partner. The degradati is even worse, if two cascaded noise reducti algorithms are active in such a cnecti. Other cascaded signal processing like additial gain or automatic gain ctrol introduced by the mobile phe may lead to signal saturati either in receiving directi (in this case the voice sounds distorted in the car) or in the microphe path (distorted driver s voice audible at the far end side). - -

8 Test Report: Performance assessment of vehicle-mounted mobile phes. Test Descripti The short range wireless cnecti was realized as a Bluetooth TM cnecti during the test event, representing the most comm use case cnecting a mobile phe to a vehicle hands-free system today..1 Test Setup The principal test setup is described in figure.1. The tests are carried out a mobile phe between two electrical interfaces, i.e. a mobile network simulator the network side and a Bluetooth TM reference interface the near end side. The Bluetooth TM reference interface MFE XI (measurement frtend provided by HEAD acoustics) communicates via the hands-free profile to the mobile phe under test. A narrowband and wideband Bluetooth TM cnecti can be setup depending the capability of the mobile phe, in particular if the mobile phe supports the wideband Bluetooth TM cnecti using the msbc codec. Tests in a narrowband Bluetooth TM cnecti use the CVSD speech codec. On the network side a network simulator (CMU 2, Rohde & Schwarz) was used providing the capability of establishing a narrowband or wideband cnecti to the mobile phe. In narrowband mode, the AMR codec operated at 12.2 kbit/s was used, the AMR- WB codec at kbit/s was used in WB mode. The Bluetooth TM reference interface MFE XI is directly cnected to the test system ACQUA. The network simulator is cnected via 6 Ohm analog input and output cnectors to the MFE VI.1 to the ACQUA measurement system (see test setup in figure.1). The clocks between both frtends, MFE XI and MFE VI.1 are synchrized via a digital AES/EBU cnecti. Fig..1: Test setup, mobile phe cnected to Bluetooth TM reference frtend MFE XI and network simulator Definiti of transmissi directis: Sending directi (see also fig..1): The sending directi (uplink) represents the transmissi from the Bluetooth TM interface (representing the car hands-free unit) via the mobile phe to the network simulator. Receiving directi (see also fig..1): The receiving directi (downlink) is defined as the transmissi from the network simulator through the mobile phe to the Bluetooth TM interface representing the hands-free unit in a vehicle. In order to verify the echo performance of the mobile phes in such a simulated Bluetooth TM cnecti, an echo path can be simulated in the MFE XI. This is indicated in figure.1. In this case, the downlink signal received via Bluetooth TM at the MFE XI is coupled back in sending directi of the Bluetooth TM cnecti with defined echo attenuati. These settings - 5 -

9 Test Report: Performance assessment of vehicle-mounted mobile phes are stored in the appropriate frtends. The test setup guarantees an automatic test run for all devices under all test cditis. The tests can be separated to verify the performance of the different communicati aspects: Tests in sending directi covering Juncti Loudness Rating tests, frequency respse tests, Automatic Gain Ctrol () tests, The correspding tests in receiving directi also cover these sensitivity tests, i.e. Juncti Loudness Rating, frequency respse, Automatic Gain Ctrol tests, Echo performance tests are realized by simulating an echo path the Bluetooth TM side. These tests are implemented in order to verify if the implemented echo cancellati signal processing in the mobile phe is disabled as required. Background noise transmissi tests are analyzed in sending directi. The main purpose is the verificati, whether noise reducti is disabled, as required. Double talk performance tests are carried out applying test signals in both directis (receiving and sending directi) simultaneously. The most critical use case, signal attenuati in sending directi caused by the implemented (and possibly active) echo suppressi units in the mobile phes, is analyzed..2 Tests Adaptatis for the Test Event The following test updates and extensi of tests were proposed and agreed to be performed during the Test Event. The main motivati for these test adaptatis is to provide realistically achievable quality of service requirements. Furthermore some test signals are selected and adapted to guarantee time-efficient test cducti during the Test Event. Both, the test updates and the extended test cases are described in detail and a motivati for the proposed modificatis/extensis is given in the following sectis. Updates of Existing Tests in Recommendatis ITU-T P.11 / P.111 Secti /2: SRW delay in SND/RCV The current delay measurement as described in Recommendati ITU-T P.11/111 is inapplicable since the described interfaces are not accessible. The given limit of 1 ms for the delay tests in sending (SND) and receiving (RCV) directi can therefore not be verified. Proposal: The terminal e-way delays in both transmissi directis T AG, UL (audio gateway uplink) and T AG,DL (audio gateway downlink) are informatively determined using the measurement interfaces as described in GPP TS [5] and the round-trip delay of the audio gateway T AG,RT is calculated as T AG,RT = T AG,UL + T AG,DL. A requirement of T AG,RT 21 ms is applied and verified during the test event. Additial remarks: The delay measurement is based the methodologies described in GPP TS [5] and investigated in more detail in GPP TSG-SA#68 document S-12 [6]. The delay definitis for a system under test (T S ) and the test system used for determining these delays (T TES ) are given as follows for mobile phes operated in handset mode: - 6 -

10 Test Report: Performance assessment of vehicle-mounted mobile phes The csideratis in S-12 lead to a requirement of T S 185 ms for this operating mode. For SRW/BT tests the setup needs to be modified as follows and the requirement has to be adapted: On e side the latencies for A/D and D/A cversi in handset mode can be neglected and delays of the implementati s specific speech processing algorithms need to be csidered ly in a reduced form since and are expected to be deactivated. It is proposed to use a reduced value of 17 ms to cover these aspects. On the other side additial processing delay for the BT Transmissi and coder block needs to be csidered. These latencies are assumed to be in a range from 2 to ms for each transmissi directi. In sum this leads to the proposed requirement of T S = T AG,RT 21 ms which was verified during the ITU-T Test Event to be achievable by modern state-of-the-art mobile phes. Secti : SRW loudness ratings (Requirements) The given requirement of ±.5 db in P.11 and P.111 is too narrow and too severe. The limit may already be violated by level loss caused by transcoding between AMR or EFR and CVSD (and vice versa) in todays narrowband Bluetooth TM cnecti or AMR-WB and msbc (and vice versa) in todays wideband Bluetooth TM cnectis in the audio gateway. Proposal: The requirement of ± db may be better suited and reflect todays signal processing more accurately. Secti /: SRW loudness ratings SND/RCV (Test) The current loudness rating tests use a periodical repetiti of CSS bursts for analysis. It should be csidered that the speech like characteristics of the CSS are limited, it was originally designed to reproduce the characteristics of real speech applied to echo cancellers

11 Test Report: Performance assessment of vehicle-mounted mobile phes Ccerns are given that the signal characteristics are not sufficient especially csidering more complex aspects of speech signals, as it may be necessary in order to evaluate e.g. transcoding effects. Proposal: A sequence of real speech signals taken from Recommendati ITU-T P.51 [7] is used as test signal. Two sentences (1 male, 1 female voice) from the Recommendati ITU- T P.51 single talk sequences are ccatenated in order to cover the low pitch frequency of male voices and the typically higher energy in the high frequency range for female voices: The last switch cannot be turned (sentence 1). The hogs were fed chopped corn and garbage (sentence 6). Secti 12.2./5: SRW linearity in SND/RCV The current linearity test uses an artificial voice sequence in Recommendati ITU-T P.11 and CSS in Recommendati ITU-T P.111. Although these test signals reproduce certain spectral and temporal characteristics of real speech, they are ly artificial speech-like test signals. Ccerns are given that the signal characteristics are not sufficient especially csidering more complex aspects of speech signals, as it may be necessary in order to evaluate e.g. transcoding effects. Furthermore the requirement for the deviati from the loudness ratings determined under nominal cditis of not more than ±.5 db is too narrow and too severe. Proposal: A sequence of real speech signals taken from Recommendati ITU-T P.51 is used as test signal. Two sentences (1 male, 1 female voice) from the Recommendati ITU- T P.51 single talk sequences are ccatenated in order to cover the low pitch frequency of male voices and the typically higher energy in the high frequency range for female voices: The last switch cannot be turned (sentence 1). The hogs were fed chopped corn and garbage (sentence 6). Furthermore, in order to limit the test durati, ly three different level adjustments relative to the nominal test signal level are csidered: -2 db, -1 db and +5 db. The deviati from the nominal test case should not exceed ± 2 db. It should be noted that higher signal levels might lead to codec saturati. It was observed that signal saturati may already occur for the +5 db test case. This test cditi was therefore not csidered in the further analyses and mighty be replaced by a - db test case. Secti 12..1/2: SRW sensitivity frequency respse SND/RCV The current frequency respse test uses an artificial voice sequence. Although this test signal reproduces certain spectral and temporal characteristics of real speech it is ly an artificial speech-like test signal. Ccerns are given that the signal characteristics are not sufficient especially csidering more complex aspects of speech signals, as it may be necessary in order to evaluate e.g. transcoding effects. Furthermore the currently used tolerance schemes are too narrow and too severe. The limit may already be violated by level loss caused by transcoding between AMR or EFR and CVSD (and vice versa) in todays narrowband Bluetooth TM cnecti or AMR-WB and msbc (and vice versa) in todays wideband Bluetooth TM cnectis in the audio gateway. Proposal: A sequence of real speech signals taken from Recommendati ITU-T P.51 is used as test signal. Two sentences (1 male, 1 female voice) from the Recommendati ITU

12 Test Report: Performance assessment of vehicle-mounted mobile phes T P.51 single talk sequences are ccatenated in order to cover the low pitch frequency of male voices and the typically higher energy in the high frequency range for female voices: The last switch cannot be turned (sentence 1) The hogs were fed chopped corn and garbage (sentence 6) The tests are performed with an AMR bitrate of 12.2 kbit/s respectively an AMR-WB bitrate of kbit/s. Furthermore, modified tolerance schemes for narrowband and wideband transmissis are proposed as follows (applicable in both directis, sending and receiving): Narrowband Wideband Frequency [Hz] Upper Limit Lower Limit Frequency [Hz] Upper Limit Lower Limit Not def Not def. rd octave FFT Size:96 Overlap:67,% Hanning Ref.: L/dB rd octave FFT Size:96 Overlap:75,% Hanning Ref.: L/dB f/hz f/hz 2 5 The proposed tolerance masks were verified to be realistically achievable during the Test Event. -15 Secti 12..2: SRW noise cancellati test in SND The original test signal csists of three pink noise bursts with a durati of 5 s each followed by a pause of s each. It should be csidered that noise reducti algorithms may need lger adaptati times than 5 s and thus might not be detected by this test ly due to the temporal structure of the test signal

13 Test Report: Performance assessment of vehicle-mounted mobile phes Original P.11/111 test signal Ctinuous car noise signal Pink random p/pa Fullsize_Car1_1Kmh p/pa t/s t/s Proposal: A ctinuous band limited inside car driving noise is applied with a level of -2 db m averaged over the frequency range from 2 Hz to.9 khz (NB) respectively 5 Hz to 8 khz (WB) and a minimum durati of 15 s. Secti /: SRW speech quality during single talk SND/RCV The determinati of the perceptual analytical listening speech quality is currently performed using the P.862 PESQ [8], [9], [1] method. Proposal: The more recent POLQA algorithm as described in Recommendati ITU-T P.86 [11] is used instead of PESQ. The British English speech sequence from Recommendati ITU-T P.51 is used and the POLQA scores are calculated based sentence pairs. The current requirement of MOS-LQOn,w. is kept and was verified during the Test Event. However, especially the wideband analyses in chapter 5.2 indicate, that the applicati of the POLQA analysis method together with this suggested limit, needs to be discussed. Ne of the tested devices meets this limit. Substituti for Secti /: Speech quality stability SND/RCV The test procedures for the speech quality stability described in Recommendati ITU-T P.11 and P.111 are not realistic to be performed in a laboratory envirment since different test positis in a vehicle and different combinatis of SRW links cannot accurately and reproducibly be simulated. Proposal for test substituti in laboratory envirments: Since the POLQA test result provides quality scores for each pair of sentences it is proposed to replace the original P.11/111 test by repetitis ( runs in total incl. the run covered by secti /) of the previously described test. Secti : Verificati of disabled echo ctrol The current test signal csists of an artificial voice adaptati sequence with a level of -16 db m followed by a pn sequence of 25 ms and a high level of - db m. Only the pn part of the signal is used for the analysis. This test signal is especially designed for the test of acoustic echo cancellers e.g. for mobile phes operated in handset mode. For the verificati of disabled echo ctrol using a simulated echo path a test signal with a lger - 1 -

14 Test Report: Performance assessment of vehicle-mounted mobile phes durati is preferable since it will lead to more reproducible results. Also a high signal level of - db m is not necessary since it can be expected that the echo ctrol is disabled. For wideband cnectis it is furthermore recommended to use a test signal which provides more energy in the higher frequency range above khz than the artificial voice sequence. Proposal: A real speech signal from Recommendati ITU-T P.51 is used. Two sentences (1 male, 1 female) from the Recommendati ITU-T P.51 single talk speech sequence are ccatenated in order to cover the full frequency spectrum: The last switch cannot be turned (sentence 1) The hogs were fed chopped corn and garbage (sentence 6) The test signal csists of two repetitis of this sequence. The first e is to ensure that the echo canceller, if it remains active, is fully adapted. The analyses are then performed during the secd repetiti. The test signal is applied with an active speech level of -16 db m in receiving directi. Only the 2 db artificial echo path is csidered the SRW reference interface side since db reduced signals are more likely to be affected by an active in the audio gateway. The analyzed echo loss (TCL w ) is corrected by the previously determined JLR values in SND and RCV. Since the influence of an modifying the measured signal levels cannot be 1% excluded the echo path spectrum should additially be analyzed in order to judge if the echo ctrol is active or not. For wideband cnectis the echo attenuati for the frequency range from 1 Hz to 7 khz (source signal level - level of the measured echo) corrected by the JLR in SND and RCV is calculated instead of the TCL w since the TCL w is defined ly from Hz to 6.7 khz. Additial Tests for Recommendati ITU-T P.11 and P.111 Proposal: and switched level tests The linearity tests described in Recommendati ITU-T P.11 are extended by an test and three switched level tests in order to further analyze whether an is active in the audio gateway. test signal Switched level test signal test signal p/pa.5 Switched level test signal p/pa t/s t/s

15 Test Report: Performance assessment of vehicle-mounted mobile phes Proposal: Verificati of volume ctrol activity The volume ctrol of the audio gateway should not affect the sensitivity in receiving directi. Therefore the loudness rating test in RCV is repeated with maximum and minimum volume setting adjusted in the audio gateway (if adjustable). The loudness rating difference for those two settings (and in principal for any other setting) shall be.2 db. This test procedure should be repeated in two independent service level cnectis. One with the remote volume ctrol feature enabled at the reference interface and e without this feature. Analysis results have shown that the volume ctrol activity often depends the availability of this feature. It was found out during the event, that the proposed requirement of.2 db is too severe for wideband cnectis. A wider range of.5 db is suggested for further tests. Proposal: Double talk test in SND In additi to the echo ctrol test an automatic double talk type test according to ITU-T P.52 Amendment 1 Appendix is informatively performed in SND in order to double check the disabled state of the echo ctrol signal processing. Proposal: Repetiti of tests with new service level cnecti and without sending the AT+= command. In order to verify that the deactivati of the and signal processing is really triggered by the dedicated AT command AT+= a subset of tests is repeated after setting up a new service level cnecti and WITHOUT sending the AT+= command. The tests to be repeated are: Delay SND/RCV JLR SND/RCV Sensitivity frequency respse SND/RCV Noise reducti test in SND Verificati of disabled echo ctrol Automatic DTT test in SND These tests are of special importance for accessories like headsets using the hands-free profile to cnect to mobile phes but relying the phes signal processing. Additial Tests for Recommendati ITU-T P.111 ly Codec negotiati mechanisms between the SRW (Bluetooth TM ) side and the mobile network side of mobile phes are not clearly defined. Thus NB/WB or WB/NB cross-cnectis are possible which may lead to unwanted level ctrasts. Proposal: Additial tests are performed under the following cditis: SRW cnecti in wideband and mobile network cnecti in narrowband mode. SRW cnecti in narrowband and mobile network cnecti in wideband mode. Following the discussis during the Test Event it was ccluded to analyze and compare the sensitivities based rd octave frequency respses, which is also used for loudness rating calculatis. The comparis of frequency respse curves provides more informati

16 Test Report: Performance assessment of vehicle-mounted mobile phes than loudness rating values. These tests are performed in send and receive directis. The results are compared for both cross-cnecti types as well as for the previously determined results according to chapter 1 of this document (no cross-cnecti).. Selecti and Designati of Mobile Phes The mobile phes were selected by the participating companies in the Test Event. The participati fee for each company covered eight tests in narrowband or wideband mode. Thus, a company could select eight different mobile phes to be tested in wideband mode or split the test modes between the mobile phes depending the transmissi capability, e.g. six tests selected mobile phes in narrowband mode and two tests in wideband mode. The motivati for the choice of mobile phes for each participant depended individual needs. The verificati of state-of-the-art phes typically used by customers today was mentied, other motivatis were the verificati of reported customer complaints, the suitability of a mobile phe for tuning hands-free performance and even the intenti to make known issues of phes public through the Test Event. In total, devices of twelve different mobile phe vendors have been tested during the event, thus representing a major part of vendors worldwide. All devices are commercially available. 27 mobile phes were tested in narrowband mode, 1 mobile phes were wideband capable and tested accordingly. Designati used for the tested devices (example): 1-WB Mobile phe No.1 tested in wideband mode. Result Representati and User Experience A two page test report was generated for each test run including a Quality Pie chart representati according to Recommendati ITU-T P.55 [12] of the most important results, the designati of the mobile phe including manufacturer, brand name, IMEI and software versi, a summary of the most important findings, additial graphs for the sending and receiving frequency respses, the noise reducti performance and the spectral echo attenuati tested with a 2 db simulated echo path. One example for the Quality Pie chart according to Recommendati ITU-T P.55 together with a short descripti of the selected parameters and requirements is shown in figure.2. The Quality Pie slices represent the following parameters and requirements (clock wise): The AT+= pie slice indicates the reply of the mobile phe this command from the reference Bluetooth TM interface. The pie slice is scaled between and Error. The pie slice is also digitally scaled between and Active. It is recommended that the volume ctrol the mobile phe should not influence the sensitivity of the Bluetooth TM cnecti, independent if the remote volume ctrol - 1 -

17 Test Report: Performance assessment of vehicle-mounted mobile phes feature is supported by the hands-free unit or not. In case the hands-free unit supports this feature, it is expected, that the volume ctrol adjustment the mobile phe may ctrol the hands-free unit playback volume. However, it is not recommended that the mobile phes volume ctrol influences the sensitivity of the Bluetooth TM link itself. The pie slice indicates the gain adjustment (amplificati or attenuati) introduced in sending or receiving directi by the mobile phe. This pie slice is scaled in db, the inner red circle represents the minimum requirement of ± db. The very important verificati, if -1 / 1 the implemented noise reducti Not transp. in the mobile phe is in the Bluetooth TM cnecti after sending the AT+= command, is indicated by the pie slice. The slice Fig..2: Quality Pie Chart according to represents two possible states, Recommendati ITU-T P.55 the noise reducti being disabled as expected ( in the Quality Pie) or being enabled ( ). The verificati of the active echo cancellati algorithm in the mobile phe after receiving the AT+= command from the Bluetooth TM reference interface is scaled in a similar way. Again two different stages, (as required) or are represented in the correspding pie slice ( ). The influence of implemented signal processing double talk performance is represented by the DT pie slice. Again two different stages are possible, the pie slice is scaled between transparent (as recommended) or not transp.. The sending sensitivity expressed by the measured Juncti Loudness Rating ( JLR (SND) ) is scaled in db. The limit, given by the inner red circle, represents an attenuati or amplificati of db (JLR range ± db). The Quality Pie slice is also scaled in db. It represents the violati of the suggested tolerance by the measured sending frequency respse () in db. Listening speech quality measured in sending directi using the POLQA algorithm described in Recommendati ITU-T P.86 is used for scaling of the MOS-LQO SND pie slice. The scale directly represents the measured MOS-LQO n or MOS- LQO w scores. The limit represented by the inner red circle is given by the numerical value. The receiving sensitivity (Juncti Loudness Rating ( ) is scaled in db. The inner red circle represents an attenuati or amplificati of db (JLR range ± db) MOS-LQO RCV -1 / /1 AT+ = Error Active

18 Test Report: Performance assessment of vehicle-mounted mobile phes The Quality Pie slice (receiving frequency respse) is scaled in db and indicates the violati of the suggested tolerance by the measured curve. Listening speech quality in receiving directi is given by the MOS-LQO RCV pie slice. The limit represented by the inner red circle is given by the numerical value as described above (see Chapter.2). In additi to these results that were analyzed after sending the AT+= command some tests were repeated without sending the command. In this case it is expected that the internal signal processing in the mobile phe is enabled as this ctrol mechanism may be used by the headset industry relying the signal processing in the mobile phes. The most important results are summarized in a short table in the test report. Test Report example: Summary in ITU-T P.55 Quality Pie Chart DUT informati, and identificati (incl. IMEI, SW) Analysis curves and numbers:,, noise reducti, echo attenuati Descripti of analysis parameter displayed in the Quality Pie Chart Additial test results without sending the AT+= command

19 Test Report: Performance assessment of vehicle-mounted mobile phes 5. Analyses Results 5.1 Overview - Narrowband The measured round-trip delay values in sending directi (uplink, UL), receiving directi (downlink, DL) and the round-trip delay (RT) are analyzed in figure 5.1a to 5.1c. The delay introduced by the test setup (frtend delay, network simulator) is csidered and subtracted already. Both e-way delays are informative ly, a limit of 21 ms is verified in figure 5.1c for the round trip delay as proposed in chapter.2 (red dotted line). Fig. 5.1a: Sending (uplink) delay T AG,UL, narrowband Fig. 5.1b: Receiving (downlink) delay T AG,DL, narrowband Fig. 5.1c: Round trip delay T AG,RT, narrowband, 27 tested mobile phes 8 out of 27 devices violate the round trip delay requirement of 21 ms in narrowband mode. devices (DUT 8, 27 and 28) show a slight violati, implementatis (DUT 2, 7 and 18) show delay values exceeding 26 ms. The maximum delay for the two devices 19 and 2 was determined between ms and 5 ms, which is unexpectedly high. It needs to be

20 Test Report: Performance assessment of vehicle-mounted mobile phes noted, that this round trip delay does not csider the network delay. Such high delays will significantly hamper the cversati between both subscribers in a telephe cversati. Figure 5.2 analyzes the 12 parameters used in the Quality Pie charts in order to provide a first overview about all 27 tested devices in narrowband mode. The green segments of each bar represent the number of mobile phes meeting the limits for each specific parameter. Vice versa the red segments indicate the number of limit violatis. The requirements itself are either taken from Recommendati ITU-T P.11, Chapter 12 or from the proposed adaptatis in Chapter.2 of this report. Fig. 5.2: Narrowband test result overview, 27 tested mobile phes The last two bars the right-hand side represent the additial tests carried out without sending the AT+= command. The bars represent the verificati of enabled noise reducti or echo cancellati (, ). It should be noted that, without sending the AT+= command, both algorithms are supposed to be enabled. The following cclusis can be drawn: All 27 phes tested in narrowband mode respd with Ok to the AT+= command. out of 27 devices keep the volume ctrol active the mobile phe ( ). This means that the actual adjusted playback level the mobile phe directly influences the sensitivity the Bluetooth TM link. This is, in general, not recommended. In case the hands-free telephe system supports the Remote Volume Ctrol feature the head unit allows the mobile phe to send appropriate commands to the head unit in order to remotely adjust the playback volume the head unit. However, it is not recommended to change the sensitivity the Bluetooth TM link within the mobile phe. This could worsen the signal to noise ratio in the digital transmissi

21 Test Report: Performance assessment of vehicle-mounted mobile phes Furthermore the playback volume in the vehicle during a phe call depends the volume setting of the phe, which is hard to ctrol without drivers distracti. Automatic gain ctrol ( ) either in receiving or sending directi is active in 5 of the 27 tested devices. % of the implementatis (9 devices) keep the noise reducti ( ) enabled, although they respded with Ok to the AT command to disable the signal processing. out of the 27 devices keep the implemented echo cancellati ( ) active. The double talk performance is hampered by 8 of the 27 devices. Surprisingly, these are twice as much compared to the devices keeping the echo cancellati algorithm active. This also indicates that the double talk performance is not ly influenced and degraded by the echo suppressi feature but also by other gain ctrol in the microphe path. The range of ± db for the sending sensitivity ( JLR SND ) is violated by 8 devices and met by 19 implementatis. It is worth to menti, that the strgest amplificati in the sending signal path was more than 12 db, which most likely leads to distorted drivers voice. 11 implementatis do not meet the adapted tolerance for the sending frequency respds ( ). The listening speech quality limit expressed by the MOS-LQO N ( ) is violated ly by e device. The sensitivity in receiving directi ( JLR RCV ) with a recommended range of ± db is violated by 5 of the 27 tested implementatis. The receiving frequency respse is violated by 16 of the 27 test devices. Listening speech quality in receiving directi with the preliminary limit of. MOS- LQO N is violated by ly 2 devices. The two additial test results in figure 5.2 measured without sending the AT command can be summarized as follows: out of 27 devices disable the implemented noise reducti, although the AT command is not sent. 7 devices disable the implemented echo canceller in the Bluetooth TM cnecti without having received the AT command. These devices obviously disable the internal signal processing in a Bluetooth TM link independent if the AT command is received or not. This is definitely another issue that should be verified for each implementati. Some Bluetooth TM headsets without own implemented signal processing may rely the algorithms in the paired mobile phe. If the mobile phe automatically disables these algorithms, echo disturbances and a low communicati quality in noisy envirment may occur

22 Test Report: Performance assessment of vehicle-mounted mobile phes 5.2 Overview - Wideband The measured delay values in wideband mode are analyzed in figure 5.a to 5.c. Again both e-way delays are informative ly and the suggested limit of 21 ms is indicated in figure 5.c by the red dotted line. Again the delay introduced by the test setup is csidered in these analyses. Fig. 5.a: Sending (uplink) delay T AG,UL, wideband Fig. 5.b: Receiving (downlink) delay T AG,DL, wideband Fig. 5.c: Round trip delay T AG,RT, wideband, 1 tested mobile phes The round trip delay limit of 21 ms is violated for 8 of the 1 tested wideband cnectis. The highest delay was determined to ms for device 2. The devices 8, 12, 27 and 28 show round trip delays of approximately 25 ms

23 Test Report: Performance assessment of vehicle-mounted mobile phes Fig. 5.: Wideband test result overview, 1 tested mobile phes The wideband results shown in figure 5. can be summarized as follows: All 1 wideband devices respd with Ok to the AT+= command. Only 1 device keeps the volume ctrol active ( ), the other 12 tested devices do not influence the sensitivity the Bluetooth TM link by the volume ctrol. Ne of the tested devices showed an automatic gain ctrol ( ), neither in receiving nor sending directi. The noise reducti was enabled in e of the 1 tested implementatis ( ). The implemented echo cancellati ( ) was disabled in all wideband tests. Ne of the tested mobile phes influenced double talk performance. The sending path can be regarded as transparent in all tests. The range of ± db for the sending sensitivity ( JLR SND ) is violated by 7 devices, 6 implementatis showed the expected sensitivity. The maximum amplificati was determined up to 17 db, which definitely distorts drivers voice. 6 implementatis violate the adapted tolerance for the sending frequency respse ( ). Ne of the mobile phes meets the listening speech quality limit of. MOS-LQOw using the P.86 recommended POLQA analysis method, neither in sending ( ) nor in receiving directi ( MOS-LQO RCV ). The verificati of the transmitted and recorded speech samples used for the POLQA analysis does not indicate strg disturbances. It is therefore necessary to verify the applicati of the P.86 analysis method together with suggested limits in the wideband test case

24 Test Report: Performance assessment of vehicle-mounted mobile phes The Juncti Loudness Rating range of ± db in receiving directi ( JLR RCV ) is violated by devices. These devices amplify the signal the Bluetooth TM link by approximately 5 db. The receiving frequency respse is violated by 5 of the 1 tested devices. The cross-cnecti tests can be summarized as follows: From the 1 tested wideband capable devices show unexpected sensitivity differences up to 1 db in cross-cnecti scenarios (BT-NB - WB / BT-WB - NB) compared to the correspding n-cross-cnecti case (BT-NB - NB / BT-WB - WB). This may lead to audible level ctrasts e.g. if the mobile network codec dynamically switches from wideband to narrowband mode during an going audio cnecti. The test results without sending the AT command can be summarized as follows: Approximately 1/ of the devices disable either noise reducti ( devices) or echo cancellati (5 devices), although AT+= has not been sent to the mobile phes. Again this may be critical in cjuncti with Bluetooth TM enabled headsets without own implemented signal processing. Disabled algorithms in this setup may lead to echo disturbances and a low communicati quality, e.g. if the headset is used in noisy envirment

25 Test Report: Performance assessment of vehicle-mounted mobile phes 5. Individual Summary Narrowband MOS-LQO RCV AT+ = Error 2.5 Active -1 / /1-1 / 1 Not transp. Device: 1-NB Volume ctrol disabled, no influence and disabled, transparent double talk performance All sensitivities within the recommended range device, no speech quality degradati for customers. w/o sending AT+= Noise reducti: enabled MOS-LQO RCV AT+ = Error 2.5 Active -1 / /1-1 / 1 Not transp. Device: 2-NB Volume ctrol disabled, no influence and disabled, transparent double talk performance Uplink sensitivity significantly out of range (7 db) No influence cversati, but risk of distorted drivers voice. w/o sending AT+= Noise reducti: enabled MOS-LQO RCV AT+ = Error 2.5 Active -1 / /1-1 / 1 Not transp. Device: -NB Volume ctrol disabled, slight influence enabled, disabled, transparent double talk performance Sensitivities in sending far out of range (12 db) Strgly distorted drivers voice, low quality of drivers voice in uplink, audible at the far end side. w/o sending AT+= Noise reducti: enabled Echo canceller: disabled MOS-LQO RCV AT+ = Error 2.5 Active -1 / /1-1 / 1 Not transp. Device: -NB Volume ctrol enabled, phes volume setting affects in vehicle playback level (up to 1 db), no influence and disabled, transparent double talk performance All sensitivities within the recommended range device, but playback level in vehicle depends current phe volume ctrol w/o sending AT+= Noise reducti: enabled

26 Test Report: Performance assessment of vehicle-mounted mobile phes MOS-LQO RCV AT+ = Error 2.5 Active -1 / /1-1 / 1 Not transp. Device: 5-NB Volume ctrol disabled, no influence and disabled, transparent double talk performance Receiving sensitivity out of range ( db), limited transmissi range device, speech sound in vehicle slightly thin w/o sending AT+= Noise reducti: enabled Echo canceller: disabled MOS-LQO RCV AT+ = Error 2.5 Active -1 / /1-1 / 1 Not transp. Device: 6-NB Volume ctrol disabled, no influence enabled, disabled, transparent double talk performance Uplink sensitivity far out of range (7 db) risk of distorted drivers voice, low quality of drivers voice in uplink, audible at the far end side. w/o sending AT+= Noise reducti: enabled MOS-LQO RCV AT+ = Error 2.5 Active -1 / /1-1 / 1 Not transp. Device: 7-NB Volume ctrol disabled, slight influence and disabled, transparent double talk performance Sending sensitivity slightly out of range ( db) Even though not a completely transparent device, no significant speech quality degradati for customers. w/o sending AT+= Noise reducti: enabled MOS-LQO RCV AT+ = Error 2.5 Active -1 / /1-1 / 1 Not transp. Device: 8-NB Volume ctrol disabled, negligible influence and disabled, transparent double talk performance Sending and receiving sensitivities out of range ( db) Even though not completely transparent, no significant speech quality degradati for customers. w/o sending AT+= Noise reducti: enabled - 2 -

27 Test Report: Performance assessment of vehicle-mounted mobile phes MOS-LQO RCV AT+ = Error 2.5 Active -1 / /1-1 / 1 Not transp. Device: 9-NB Volume ctrol disabled, no influence and enabled, double talk performance is not transparent if subscribers both talk at the same time Sensitivities ok, but limited transmissi range Not transparent device, customer complaints can be expected, speech sound in vehicle slightly muffled w/o sending AT+= Noise reducti: enabled MOS-LQO RCV AT+ = Error 2.5 Active -1 / /1-1 / 1 Not transp. Device: 1-NB Volume ctrol disabled, slight influence and disabled, transparent double talk performance Sending sensitivity just within recommended range Even though not a completely transparent device, no significant speech quality degradati for customers w/o sending AT+= Noise reducti: enabled MOS-LQO RCV AT+ = Error 2.5 Active -1 / /1-1 / 1 Not transp. Device: 11-NB Volume ctrol disabled, no influence and disabled, transparent double talk performance Sensitivities within recommended range Even though not a completely transparent device, speech sound in vehicle slightly muffled w/o sending AT+= Noise reducti: enabled MOS-LQO RCV AT+ = Error 2.5 Active -1 / /1-1 / 1 Not transp. Device: 1-NB Volume ctrol disabled, no influence and enabled, double talk performance is not transparent if subscribers both talk at the same time Sensitivities ok, but limited transmissi range in receiving Not transparent device, customer complaints can be expected, speech sound in vehicle slightly muffled w/o sending AT+= Noise reducti: enabled - 2 -

28 Test Report: Performance assessment of vehicle-mounted mobile phes MOS-LQO RCV AT+ = Error 2.5 Active -1 / /1-1 / 1 Not transp. Device: 15-NB Volume ctrol disabled, no influence and disabled, transparent double talk performance All sensitivities within the recommended range device, no speech quality degradati for customers. w/o sending AT+= Noise reducti: enabled MOS-LQO RCV AT+ = Error 2.5 Active -1 / /1-1 / 1 Not transp. Device: 16-NB Volume ctrol enabled, slight influence and disabled, double talk performance is not transparent if subscribers both talk at the same time Sending sensitivity out of range ( db) Not completely transparent device, playback level in vehicle depends current phe volume setting w/o sending AT+= Noise reducti: disabled Echo canceller: disabled MOS-LQO RCV AT+ = Error 2.5 Active -1 / /1-1 / 1 Not transp. Device: 18-NB Volume ctrol disabled, strg influence enabled, disabled, double talk performance is not transparent if subscribers both talk at the same time Sensitivities out of range, limited transmissi range in both directis Not completely transparent device, customer complaints can be expected, limited sound quality w/o sending AT+= Noise reducti: enabled Echo canceller: disabled MOS-LQO RCV AT+ = Error 2.5 Active -1 / /1-1 / 1 Not transp. Device: 19-NB Volume ctrol disabled, no influence enabled, disabled, transparent double talk performance Sensitivities ok, but limited transmissi range in sending low quality of drivers voice in uplink, audible at the far end side. w/o sending AT+= Noise reducti: enabled Echo canceller: disabled