TESTING REPORT: Genesys IP Media exchange

Transcription

1 TESTING REPORT: Genesys IP Media exchange Executive Summary: The Converged Communication Industry s Premier ing Service Date of s: November, 2004 Vendor: Genesys Telecommunications Laboratories Daly City, CA GENESYS Product ed: IP Media exchange Version: (Pre-release) CT Labs was commissioned by Genesys to evaluate the call-handling performance of the IP Media exchange (IPMX). The IPMX components were deployed on one, two, and three Windows-based servers, respectively, for each of the three tests described in this report. All VoIP calls were generated using an automation script running on the Empirix Hammer FX-IP platform, using the SIP protocol. In the first test, the IPMX was deployed on a single server. The results of this test reveal that a single high-performance server with 150 agents configured can effectively process in excess of 250 simultaneous callers at a rate of 8 calls/second, with half listening to music on hold, and the other half connected to agents. In the second test, the IPMX was deployed on two servers. In this test, the IPMX with 288 agents configured was confirmed to handle over 400 simultaneous callers at a rate of 11 calls/second, again with half listening to music on hold, and half connected to agents. In the final test, the three-server IPMX configuration with 1000 agents configured resulted in successful processing of over 900 simultaneous callers at a rate of 10 calls/second. This was accomplished with 20% listening to music on hold, and the rest connected to agents. Overall, CT Labs found the Genesys IPMX to provide excellent callhandling performance in each of the three test configurations, supporting up to 1000 agents in our tests. Product Description: Genesys IPMX is an IP media switch designed specifically for the contact center. Genesys IPMX handles call control between IP-based devices such as media gateways, IP phones and conferencing units and gets its intelligence from Genesys applications. Genesys IPMX supports both Windows and UNIX Sun Solaris operating systems. All related configuration information is set up within Genesys Customer Interaction Management (CIM) Platform, allowing for rapid deployment to new sites or as a tie-in to legacy infrastructures. The Genesys IPMX system consists of three components: Distributed Media exchange (DMX) is an IP media switch, supporting SIP and H.323 protocols that are fully exposed / controllable by Genesys CIM Platform. The primary function of the DMX is call control providing a switching matrix, bridging IP interactions (voice and video) between endpoints and media gateways. In this manner, the DMX provides for the switching / call management functionally of IP calls, under the control of Genesys applications. Stream Manager works under the control of the DMX to provide tight control of real-time IP media (voice and video) packets and also provides an IP call treatment platform that includes call queuing, music, announcements and DTMF digit collection. SMCP T-Server is the messaging engine between the media-switch call control environment and the Genesys applications. It ensures all interactions, both voice and data, are synchronized and delivered to the appropriate agent. Page 1 of 8

2 CT Labs ing Report: Genesys IP Media exchange 1. Environment Diagrams: The testing environments for each of the three tests are shown in Figures 1, 2, and 3 below. 2 Empirix Hammer FX-IP's executing caller script HP Proliant DL 380 Netgear 1G Ethernet switch DMX + SMCP TS + SM in MoH Figure 1 One-server Setup 1 Empirix Hammer FX-IP executing agent script HP Proliant ML 370 SM in MoH HP Proliant DL 380 Netgear 1G Ethernet switch 2 Empirix Hammer FX-IP's executing caller script DMX + SMCP TS Figure 2 Two-server Setup 1 Empirix Hammer FX-IP executing agent script 2 HP Proliant ML 370s SM HP Proliant DL 380 Netgear 1G Ethernet switch 4 Empirix Hammer FX-IPs executing caller script DMX + SMCP TS Figure 3 Three-server Setup 4 Empirix Hammer FX-IPs executing agent script (See the next page for a description of the testing environments) Page 2 of 8

3 CT Labs ing Report: Genesys IP Media exchange 2. Environments Description: The Genesys SMCP T-Server (SMCP TS), Distributed Media exchange (DMX), and Stream Manager (SM) IPMX components were installed on dual 2.8gHz HP Proliant servers at CT Labs. The one-server configuration housed all the IPMX components together on the same server. The two-server configuration housed the DMX and SMCP TS on one server, and the SM on a server. The three-server configuration was performed with DMX and SMCP TS installed on one server, with the SM installed on two servers. CT Labs utilized the Empirix Hammer FX-IP testing platform to generate and answer native SIP calls with media for this test. Each Hammer FX-IP has the capacity to simulate up to 288 SIP endpoints with generated media. For the larger tests, up to eight Hammer FX-IP units were used. All the components in each test scenario were interconnected using a Netgear Gigabit Ethernet switch. 3. Call Flow: For each test performed, one or more Empirix Hammer FX-IP units initiated calls into the IPMX system, with one or more FX-IP units answering calls by acting as IPMX agents. Incoming SIP calls with media were answered by the IPMX and immediately connected to music on hold (MoH) for a predetermined amount of time (as specified by the Genesys IPMX configuration). Calls on hold were then transferred to an agent and answered by an agent-side Hammer FX-IP. Once connected, the Hammer test script verified the audio path and kept the call connected for a predetermined amount of time (as specified in the Genesys IPMX configuration). Samples of MoH audio were recorded and saved for later PAMS voice quality analysis. 4. Configuration Parameters: This section presents the parameters set for each of the three test runs, as shown in Table 1 below. Table 1: Configuration Parameters Configuration Parameter One-server Two-server Three-server Destination agents configured 1 : ,000 Incoming calls generated 2 (concurrent): Calls on hold (target 3 ): 50% 50% 20% Calls connected to an agent (target): 50% 50% 80% Call rate (Calls per second, average): MoH connect time (average, in seconds): Agent talk time (average, in seconds): The destination agents and MoH connect time were parameters set within the IPMX system. 2 The incoming calls generated, call rate, and agent talk time were parameters set within the Hammer FX-IP scripts. 3 The target percentages for calls on hold and calls connected to an agent are not direct settings, but are derived from the combined settings of MoH connect time and agent talk time used for the test. Page 3 of 8

4 CT Labs ing Report: Genesys IP Media exchange 5. ing Results Call Handling Performance This section presents call handling results for calls processed under each of the three test configurations. The call completion results are shown in Table 2 below. Table 2: Call Completion Results Call Completion Parameter One-server Two-server Three-server run time (minutes): Total calls, completed: 29,305 58,819 34,535 Total calls, not completed: Percent of completed calls 100% 100% 100% As the above data indicates, the Genesys IPMX system performed perfectly with a 100% call completion rate for each of the three test configurations outlined in this report. CT Labs also accumulated call answer performance data from the Empirix Hammer FX-IP call generation platforms, which is presented in Table 3 below. Table 3: Call Answer Delay Performance Results Call Answer Delay One-server Two-server Three-server Minimum (msec) Maximum (msec) Average (msec) With an average of no worse than 86 milliseconds, and a worse case call answer delay of no more than 350 milliseconds, the IPMX was found to have excellent call-answering performance. Page 4 of 8

5 CT Labs ing Report: Genesys IP Media exchange 6. Results CPU and Memory Utilization: In order to measure CPU and memory utilization for the Genesys IPMX servers, CT Labs used the integrated Windows Performance Monitor (PerfMon), an administrative tool for monitoring server performance. PerfMon was configured to track CPU (Processor) and memory (Private Bytes) resources on each IPMX server. For memory consumption, specific Genesys processes running on each server (autoserv6_1, dmx, sm, and smcp_server) were tracked, as were Network Bytes/Second on the SM servers. The average CPU performance results for each test configuration are shown in Figures 4 through 6 below % 1.04% 6.83% 8.18% 1.76% autoserv6_1 dmx sm smcp_server system/idle Figure 4: Average CPU Usage by Process One-server 87.16% 0.86% 4.90% 5.93% 1.15% autoserv6_1 dmx sm smcp_server system/idle Figure 5: Average CPU Usage by Process Two-server Page 5 of 8

6 CT Labs ing Report: Genesys IP Media exchange 88.66% 0.46% 6.77% 3.37% 0.74% autoserv6_1 dmx sm smcp_server system/idle Figure 6: Average CPU Usage by Process Three-server As the above figures indicate, the CPU processor usage remained low for all IPMX processes, and no significant increases or decreases in processor or memory use were noted over the duration of each test. 7. Music on Hold Quality Evaluation Technique Used: CT Labs performed automated end-to-end speech quality analysis using a batch-processing version of the Perceptual Analysis / Measurement System (PAMS) method. PAMS is an automated analysis technique that predicts the speech quality and listening effort for collected speech samples. This repeatable and objective test derives a set of scores by comparing one or more high-quality reference speech samples to the processed (degraded) resulting audio. This is an automated technique that can be performed quickly, returning quality measurement results that are often within 0.5 of a live listener MOS test rating. The PAMS listening quality and listening effort score is based upon a five point category judgment scale as is shown in Table 4 below: Table 4: PAMS Scoring Definitions Score Listening Quality Listening Effort 5 Excellent Complete relaxation possible; no effort required 4 Good Attention necessary; no appreciable effort required 3 Fair Moderate effort required 2 Poor Considerable effort required 1 Bad No meaning understood with any feasible effort Page 6 of 8

7 CT Labs ing Report: Genesys IP Media exchange 8. Results Music on Hold Voice Quality: This section presents the results of the CT Labs evaluation of the IPMX music on hold generation capability. The purpose of this test was to verify that the quality of the MoH streams were unaffected by high call volumes. The following table presents information relating to the calls in which MoH audio was sampled. Table 5: Music on Hold Quality, Calls Sampled Music on Hold Samples Parameter One-server Two-server Three-server run time (minutes): Total calls completed: 29,305 58,819 34,535 Calls sampled for MoH quality analysis: ,652 The MoH voice quality results are shown in Figure 7 below. PAMS Scoring Results 5 - Excellent PAMS score 4 - Good 3 - Fair 2 - Poor 1 - Bad Listening Effort Listening Quality One-server Two-server Three-server Configuration Figure 7: Music on Hold Quality Performance Results These results indicate the level of quality degradation of the sampled MoH audio throughout all test calls was essentially undetectable. No variation in scoring in either Listening Effort or Listening Quality was noted throughout the full range of samples evaluated across all three test configurations. Although a perfect PAMS score is 5.00, to the human ear anything above a 4.5 is essentially indistinguishable from the reference prompt. In this particular setup, the reference audio file used (PCM, 8 Khz sampling rate, 16 bit samples) had to be converted to a different format to be used as a Genesys MoH file. When played out by the Genesys IPMX during a call, it is converted to G.711, 8 Khz, 8 bit. On the receiving end of the call, the Hammer FX-IP saves it in PCM, 8 Khz, 16 bit format. This conversion process degrades the quality slightly, accounting for the slight but uniform expected reduction from a 5.0 score. Page 7 of 8

8 CT Labs ing Report: Genesys IP Media exchange 9. CT Labs Summation of Results: As CT Labs engineers ran a variety of tests with this pre-release version (version 7.0.2) of the IPMX software, CT Labs encountered a few issues with the IPMX product. While they did not adversely affect the test results presented in this report, they had the potential to degrade the performance of the IPMX product in the field. Genesys has indicated to CT Labs that these discovered issues will in fact be addressed in the next general availability release of the IPMX product. Overall, CT Labs found the Genesys IPMX to exhibit excellent call answer performance, efficient processor and memory usage, and excellent music on hold voice quality. About CT Labs CT Labs, an independent operating unit of Empirix, provides a full-service testing and product analysis lab exclusively serving the Converged Communication and IP Telephony industries. CT Labs testing automation specialists not only perform a wide variety of standard test suites that validate and exercise products, but can also create custom-developed tests for special testing needs. CT Labs prides itself on working with cutting edge technology and products. The CT Labs ed program was designed to provide independent testing and validation of products, including the execution of tests that verify real-world performance. Products that display the CT Labs ed label have been tested by CT Labs and have exhibited a high level of quality in their performance and use. For more information about this or other testing services, contact: CT Labs An independent operating unit of Empirix, Inc Warren Drive Rocklin, CA USA Phone: (916) Fax: (916) Copyright 2005 Empirix This report is intended to be reproduced in its entirety. No portion of this report may be reprinted without prior written permission from CT Labs. Page 8 of 8