Best Practices for Handset Testing with Spirent Communications Nomad

Transcription

1 Best Practices for Handset Testing with Spirent Communications Nomad Spirent Communications 7340 Executive Way Suite A Frederick, MD Document Version: 2.2

2 Contents 1 Introduction Time Synchronization Optimal Downlink Volume (Input Level) Optimal Uplink Volume (Output Level) High Speed Internet Connectivity Handset Placement and Settings Phone Number Entry Caller ID Block Redial After a Drop Best Practices for Mobile Terminated Call Performance Testing Appendix A: Nomad Troubleshooting Techniques Downlink Data Missing Uplink Data Missing Unsuccessful Online Merge Bluetooth The Bluetooth word mark and logos are owned by the Bluetooth SIG, Inc. and any use of such marks by Spirent Communications, Inc. is under license. Other trademarks and trade names are those of their respective owners. Copyright Spirent Communications, Inc

3 1 Introduction Best Practices for Handset Testing with Spirent Communications Nomad In order to obtain the most accurate results from the Nomad system, Spirent Communications has compiled a set of Best Practices to follow during testing. The Best Practice categories are listed below along with the applicable test types: Best Practice Voice Quality Call Performance Time Synchronization Optimal Downlink Volume (Input Level) Optimal Uplink Volume (Output Level) High Speed Internet Connectivity Handset Placement and Settings Phone Number Entry Caller ID Block Redial After a Drop Best Practices for Mobile Terminated Call Performance Testing Each of the Best Practices described are followed by Spirent Communications engineers during field testing, and are recommended for all users of the system. For additional guidance in resolving issues with the Nomad system, please see Appendix A of this document which addresses troubleshooting techniques. 2 Time Synchronization Nomad calculates MOS in Voice Quality tests by comparing audio samples degraded by the communication channel to the original source sample. Therefore, the audio waveform must be centered within the playback window in order for Nomad to accurately calculate MOS. The two waveforms below are both clipped. The first waveform is clipped on the left. The second waveform is clipped on the right. Nomad will not be able to score either sample accurately and will display a NULL value. Figure Nomad Waveform Clipped on the Left Copyright Spirent Communications, Inc

4 Figure Nomad Waveform Clipped on the Right The illumination of the MOS statistics area in red also indicates this problem. This waveform is centered within the window and MOS will be calculated correctly: Figure Centered Waveform Synchronize the test laptop s clock with the Audio Server to ensure consistent timing between the two: During testing, select the Time Sync button. Set the Time Synchronization Source as either Internet Time Server or GPS. o Note: Spirent does not recommend using Internet Time Server synchronization for Voice Quality testing performed while drive testing. Substantial latency may be introduced when the internet time protocol operates in a mobile environment. If possible, use GPS time synchronization in this case. Click the Sync Now button. Enable automatic time synchronization to prevent time drift in the synchronization between the test laptop and the Audio Server: In the Time Sync dialog, check Enable automatic time synchronization. Set the period between time synchronization updates in minutes. Spirent recommends setting this to 5 minutes for frequent updates. Copyright Spirent Communications, Inc

5 3 Optimal Downlink Volume (Input Level) Nomad calculates MOS in Voice Quality tests by comparing audio samples degraded by the communication channel to the original source sample. Therefore, the downlink waveform must be completely visible within the playback window and free of noise distortion for Nomad to accurately calculate downlink MOS. When using analog device connections, Spirent recommends running the Auto-Level Assist Wizard prior to testing a new device to determine the optimal volume settings for that device. Minor adjustments to the Wizard recommendations can be made during testing as described below. The Downlink Volume (Input Level) for this audio sample is too high. The top and bottom of the waveform are clipped and distortion is present. Nomad will not be able to score this sample accurately: Figure High Downlink Volume An additional indicator of clipping, the Overdrive light for the saturated channel blinks red on the Spirent ME hardware unit and on the Levels tab of the Settings dialog: Figure Overdrive Indicator Illuminated This sample represents a channel set at the appropriate Downlink Volume. The waveform is intact and downlink MOS will be calculated correctly: Copyright Spirent Communications, Inc

6 Figure Appropriate Downlink Volume To find the optimal Downlink Volume for any channel: Set the handset volume one level below the maximum. Further volume adjustments will be performed using the Nomad software. While a test session is in progress, click the Settings button and navigate to the Levels tab. For the channel of interest, move the Input slider to the far right until the Overdrive light illuminates in the GUI and on the hardware unit. Move the Input slider to the left one notch at a time and observe the Overdrive light during the Recording state. The first position at which the Overdrive light stays off for the current and next recording cycle should be used as the Input setting. The optimal Input slider position should fall within the Normal Operating Range of shown on the slider. The downlink insertion gain (PESQ) or attenuation (POLQA) shown in the test window will likely read around 0. Note that high downlink volume will not be an issue when using a Bluetooth connection. In cases where the downlink insertion gain (PESQ) or attenuation (POLQA) for a Bluetooth channel is far from 0 (roughly less than -8 db for insertion gain or greater than 8 db for attenuation), the Input Level may be adjusted by moving the slider one or two notches to the right. The slider should be kept within the Normal Operating Range. If MOS deteriorates after the adjustment, return the Input Level to the original value. 4 Optimal Uplink Volume (Output Level) Nomad calculates MOS in Voice Quality tests by comparing audio samples degraded by the communication channel to the original source sample. Therefore, the uplink waveform must be free of noise distortion in order for Nomad to accurately calculate uplink MOS. When using analog device connections, Spirent recommends running the Auto-Level Assist Wizard prior to testing a new device to determine the optimal volume settings for that device. Minor adjustments to the Wizard recommendations can be made during testing as described below. Copyright Spirent Communications, Inc

7 The uplink speech waveform is not displayed during testing. In most cases, uplink volume adjustment is only necessary if uplink MOS values are unexpectedly poor. The MOS Trend Window in this sample presents symptoms of an uplink volume setting that is too high: Figure High Uplink Volume This uplink MOS trend in this sample represents a channel set at an appropriate uplink volume: Figure Appropriate Uplink Volume To find the optimal Uplink Volume for any channel in both the Bluetooth and analog connection cases: While a test session is in progress, click the Settings button and navigate to the Levels tab. Move the Output slider for the channel of interest within the Normal Operating Range of If uplink MOS deteriorates after the adjustment, return the Output Level to the original value. Copyright Spirent Communications, Inc

8 5 High Speed Internet Connectivity Best Practices for Handset Testing with Spirent Communications Nomad Nomad requires a sustained internet connection of at least 50 kbps in the following situations: Determining the optimal volume levels for a Voice Quality test using Automatic Level Assist. Viewing live uplink data during a Voice Quality test. Starting a Mobile Terminated call campaign. Obtaining the report for a Mobile Terminated or Voice Quality call campaign. Automatic internet-based time sync during testing (GPS time sync is recommended). Follow these guidelines for the best results when using internet-dependent features: A sustained connection of 50 kbps is required to run these features. If the internet connection is lost during testing, these features will become unavailable. Functionality will return when the connection comes back up. The internet connection may need to be restarted manually. If possible, Spirent recommends using a different provider for the high speed internet connection than for the voice network. Spirent recommends using the Auto Level Assist Wizard to optimize volume prior to testing in a location with good RF coverage and a strong internet connection. 6 Handset Placement and Settings Noise coupling can induce buzzing or an equivalent noise on the audio path and reduce MOS scores. Follow these guidelines regarding handset placement and settings to minimize noise coupling: During testing, position the handsets as far away from the Spirent ME unit as possible. If an AC inverter is used, keep it away from both the hardware unit and the handsets. Ensure that the test PC is not running on batteries. Do not charge the handsets during testing. During testing, the handsets should be set to standard audio not speakerphone. If a 2.5 mm adaptor is required, ensure that the correct adaptor has been selected and that all adaptor connections are secure. Prior to testing, disable all Bluetooth connections with the exception of those used to couple the handset to the hardware unit. Copyright Spirent Communications, Inc

9 7 Phone Number Entry Best Practices for Handset Testing with Spirent Communications Nomad When performing Voice Quality testing using the Audio Server, it is imperative that the Number to call and Number calling from phone numbers match the actual numbers used. This ensures that uplink data may be obtained from the server and merged with the corresponding downlink data. Nomad provides the Connection Assistant utility to assist in entering the phone numbers with the highest accuracy. While Spirent strongly recommends the use of this utility for all customers, international users outside of North America are urged to take advantage of the Connection Assistant to ensure that phone numbers are entered in the correct format. To use the Connection Assistant: On the Test Setup tab, select the Session button for any configured Mobile task. Confirm that the Uplink Device is set to Audio Server. Click the Connection Assistant button to start the utility. Click the Edit Audio Server Settings button. In the Edit Audio Server Settings dialog, select the Audio Server Name and enter the Username and Password. Click OK to accept the changes. Figure Edit Audio Server Settings Dialog Dial the Audio Server using the handset to be connected to this channel. The handset does not need to be connected at this time. All active calls to the Audio Server will be represented in the Connection Assistant dropdown window using the {4 Digit DNIS}-{10 Digit ANI} convention: o {4 Digit DNIS} The last four digits of the Audio Server phone number called by the mobile. o {10 Digit ANI} The phone number of the mobile being tested. If the test handset does not appear in the list, use the Refresh button to update. It may take up to one minute for the handset to appear. Use the drop-down to select the correct number combination from the list. Click OK. Copyright Spirent Communications, Inc

10 Figure Connection Assistant The Number to call and Number calling from fields will be populated based on your selection, removing the risk of manually entering this information incorrectly. Figure Phone Number Settings Populated by the Connection Assistant 8 Caller ID Block Nomad relies on the phone numbers of both the calling and called devices to identify and merge session data. Nomad will not be able to correlate session uplink data if Caller ID Block is turned on. To this end, it is critical that Caller ID Block is turned OFF for all test devices. Copyright Spirent Communications, Inc

11 9 Redial After a Drop Best Practices for Handset Testing with Spirent Communications Nomad Unlike most RF collection tools, Nomad channels must be configured by the user in order to automatically redial after a dropped call during Voice Quality testing. Auto redial is available for all devices using a Bluetooth connection and for some devices using an analog connection. Bluetooth auto redial may be enabled via the Channel Settings button for any configured Voice Quality task. If auto redial is not enabled, the handset must be manually redialed before proceeding with testing after a drop. This is important to keep in mind during long call testing when calls are allowed to stay connected until they drop. To enable auto redial: In the case of a Bluetooth connection: o Auto redial is available for all devices. o In the Channel Settings dialog, set the Audio Interface to Bluetooth. o Check the Enable Bluetooth auto redial box and specify the number of seconds to pause before redialing. o With this configuration, a call that is dropped during testing will be automatically redialed. In the case of an analog connection: o Auto redial is available for phones which support the following options: Ability to maintain an analog and Bluetooth connection simultaneously. Ability to transfer from a Bluetooth connection to an analog connection. Ability for the Bluetooth connection to be re-established while the phone is on a call. o To utilize auto redial with a qualified device: Connect the test handset to the Spirent ME hardware using Bluetooth. Next, attach the test handset to the same channel using an analog connection. Verify that the Bluetooth connection is still active. Set the Audio interface for this channel as Analog. Check the Enable Bluetooth auto redial option. Specify the number of seconds to Pause before redialing. With this configuration, a call that is dropped during testing will be redialed using the Bluetooth connection. After three cycles of silence, Nomad will then issue a command to transfer the audio from the Bluetooth to the analog connection. Figure Auto Redial Option in Channel Settings Dialog Copyright Spirent Communications, Inc

12 10 Best Practices for Mobile Terminated Call Performance Testing In a Mobile Terminated Call Performance Test, the Call Server makes calls to the mobile test device for call control testing. Statistics gathered during this type of test are based on the point of view of the test handset. Spirent recommends following these best practices for the best results with Mobile Terminated testing: Confirm that the test handset is properly configured with voic . This will ensure that voic is properly detected during testing. The Access Timeout parameter for a Mobile Terminated call campaign must be set to a larger duration than it takes the phone to go to voic . If Access Timeout is too short, Nomad will fail to recognize a call sent to voic . Instead, such a call will be classified as an Access Timeout before voic picks up. A 45-second Access Timeout is generally sufficient for North American carriers. Nomad can take up to seven seconds to detect voic . In a Synchronous campaign it is best practice to set Duration at least seven seconds longer than Access Timeout to ensure that no Voic events are missed. Copyright Spirent Communications, Inc

13 Appendix A: Nomad Troubleshooting Techniques This Appendix provides techniques for troubleshooting and resolving some of the most common issues encountered when testing with the Nomad System: Downlink data missing (uplink data may be available) Uplink data missing (downlink data may be available) Unsuccessful online merge (data appears on Incomplete tab) If the troubleshooting tips presented here are insufficient, or if you encounter other issues, please contact Spirent Product Support for assistance. Downlink Data Missing In the event that downlink data is missing during Voice Quality testing (even though uplink data may be available), check these possible causes: Phone Volume Is the phone volume level acceptable? Confirm that the handset volume is set one level below the maximum. Time Synchronization Is the test laptop s clock synchronized with the Audio Server? If performing the test while drive testing, is the GPS time synchronization method used? See Chapter 2 Time Synchronization for instructions on checking and setting synchronization. Input Level Is the Input Level properly set? Input Level refers to the volume of the sourced downlink audio sample. See Chapter 3 Optimal Downlink Volume to make the proper adjustments. Adapter Issues Is there a problem with the 2.5 mm adapter? Confirm that you are using an appropriate adapter for the device. Ensure that all adapter connections are secure. If necessary, try a different adapter to help isolate the source of the problem. Task Type / Corresponding Device Has the appropriate Task Type (i.e. Mobile, Base, Source Only, Record Only) been applied to this channel? In the Session dialog, is the Uplink, Downlink, Sourcing or Recording Device set as appropriate? Copyright Spirent Communications, Inc

14 Uplink Data Missing Best Practices for Handset Testing with Spirent Communications Nomad In the event that uplink data is missing during Voice Quality testing (even though downlink data may be available), check these possible causes: Internet Access Does the test laptop have a sustained high-speed internet connection of at least 50 kbps? If the internet connection was lost during testing, live uplink data will become unavailable. Functionality will return when the connection comes back up. In some cases, the internet connection may need to be restarted manually. See Chapter 5 High Speed Internet Connectivity for more information on this topic. Phone Numbers On the Test Setup tab, open the Session dialog for the affected channel and confirm that the Phone Number Settings are correct. Spirent recommends using the Connection Assistant to confirm phone numbers as described in Chapter 7 Phone Number Entry. Phone Volume Is the phone volume level acceptable? Confirm that the handset volume is set one level below the maximum. Time Synchronization Is the test laptop s clock synchronized with the Audio Server? If performing the test while drive testing, is the GPS time synchronization method used? See Chapter 2 Time Synchronization for instructions on checking and setting synchronization. Output Level Is the Output Level properly set? Output Level refers to the volume of the sourced uplink audio sample. See Chapter 4 Optimal Uplink Volume to make the proper adjustments. Adapter Issues Is there a problem with the 2.5 mm adapter? Confirm that you are using an appropriate adapter for the device. Ensure that all adapter connections are secure. If necessary, try a different adapter to help isolate the source of the problem. Task Type / Corresponding Device Has the appropriate Task Type (i.e. Mobile, Base, Source Only, Record Only) been applied to this channel? In the Session dialog, is the Uplink, Downlink, Sourcing or Recording Device set as appropriate? Caller ID Block Confirm that Caller ID Block is turned OFF for the test device. Copyright Spirent Communications, Inc

15 Unsuccessful Online Merge Best Practices for Handset Testing with Spirent Communications Nomad On occasion, Nomad may be unsuccessful in merging the uplink and downlink data from a test session into a unified log file for report generation. Such files will appear on the Data Incomplete tab at the conclusion of a test (successfully merged files appear on the Data Complete tab). Files on the Incomplete tab contain an icon signifying each problematic channel: The half-circle icon indicates that Nomad does not have access to all of the data required to merge this file. This might appear if no calls were started for a channel during a test. This might also appear in the case of a Remote Unit test where Nomad can access the locally collected data but not the remotely collected data. The yellow triangle icon indicates that an error has occurred during data collection preventing Nomad from merging the uplink and downlink data associated with this test case. One likely error cause is the incorrect entry of the Phone Number Settings. Another common cause of this error is lack of internet connectivity at merge time. Click on the or indicator icon to invoke the Modify Incomplete Logging Session dialog to troubleshoot: Retry Auto-Complete If lack of internet connectivity is believed to be the reason for the error, re-establish the internet connection and then use the Retry Auto-Complete button to merge the data and move to the Complete tab. Phone Number Settings If the icon indicates Unable to download uplink scores, the Phone Number Settings for the session are likely incorrect. Select the Edit button to Edit Channel Configuration. Enter the correct values for Number to call and Number calling from and then Close the dialog. Click Retry Auto-Complete to merge the data. Uplink Scoring Utility Some files that are unable to be merged automatically can still be merged using the Uplink Scoring Utility. Contact Spirent if you require access to your uplink data for merging. Mark As Complete If you believe that a channel has been flagged as incomplete in error, use the Mark As Complete option to disregard the incomplete data warning. Any report generated with this data may contain missing or incomplete information. Copyright Spirent Communications, Inc