3CX Phone System Enterprise 512SC Edition Performance Test 3CX Phone System is one of the most popular IP PBX systems that works flawlessly in a Microsoft Windows environment. It s renowned for its simplicity, scalability and also how easy it is to configure. 3CX licensing is primarily based on limiting the number of simultaneous calls that can be made. However, we have found that many customers look on 3CX with some disbelief at the sensibility of the 512 SC (simultaneous calls) license as 3CX is perceived more as an SMB phone system solution. Therefore, what happens when we test the system to its licensing maximum? We would like to thank 3CX for making the 512 SC Enterprise Edition available for our tests, especially since we made it explicit that we were not going to be lenient about any unfavorable findings. Our aim was to rigorously test 3CX to the limit; therefore we were not going to do a simple test using a basic SIP generator with 512 simultaneous calls running. We were going to go one step further... 3CX Phone System Thesis We wanted to check how 3CX Phone System will perform in a simulated office environment of a large enterprise, so we went to work and: introduced 3,000 internal extensions introduced a complex PBX configuration created a SIP generator and terminator that emulated different types of traffic and media emulated a SIP trunk to PSTN created a configuration in which 3CX routed the traffic using DDIs, ACD and SIP Location Service created a recording configuration and made sure that incoming calls are using IVRs We were also going to receive a recording of the system load, however we needed a reference load in the context of the operating system overhead and so we went in the direction of virtualization. Apart from some testing aspects, we decided that virtualization is one of the growing types of IP PBX implementations and generally used in an enterprise IT environment. We wanted to see whether using a hypervisor would cause issues in any way with real time communications when at a high load level.
1.0 Test Procedure 1.1 3CX Environment 1.1.1 Server A Dell PowerEdge was to host the hypervisor, VMware, and 3CX Phone System software: Model CPU RAM HDD RAID NIC Dell PowerEdge 1950 MKII 2x Intel Xeon E5320 @ 1,86 GHz 16 GB DDR2-667 PC2-5300 2x300 GB SAS 15k PERC 5/I running RAID1 2x Broadcom 1000 Base-T 1.1.2 Hypervisor Model VMware ESXi 4.1.0 1.1.3 Installed VM We decided that in order to simulate a working environment, we needed to create two more virtual machines apart from the one that would be tested. The VMs were in idle state and, apart from OS systems, no services were active. VM 3CX 512 Test VM1 VM2 Description VM for 3CX Idle VM1 Idle VM2 CPU 4 vcpu (4 Cores from Xeon CPU) 2 vcpu (2 Cores from Xeon CPU) 2 vcpu (2 Cores from Xeon CPU) RAM 8GB 2GB 2GB HDD 83GB SCSI (0:0) datastore1 22GB SCSI (0:0) datastore1 22GB SCSI (0:0) datastore1 RAID LSI Logic SAS LSI Logic Parallel LSI Logis SAS NIC 1x E1000 2x E1000 (Different vswitch) OS Windows Server 2008 R2 Standard 2xE1000 (Different vswitch) Debian Squeeze 6.0.2 Centos 6.0
1.1.3.1 3CX Software on VM The following software was installed on the VM with 3CX Phone System on board. In order to catch reference performance with Microsoft software we also used the option of Report Performance from the VMware vsphere Client 4.1.0. Operating System IP PBX OS Performance Monitor Microsoft Windows Server 2008 R2 Standard 3CX Phone System 10, Enterprise Edition 512SC, Service Pack 2 Microsoft Performance Monitor VM Performance Monitor VMware vsphere Client 4.1.0 1.2 Testing / Application Environment 1.2.1 Servers For the SIP generators and terminators of the various types of traffic, there were two machines with approximately the following parameters: CPU RAM HDD NIC 1x Intel QC @ 2GHz 4 GB DDR3 1x 1TB SATAII 1x Intel Pro E1000 1000 Base-T 1.2.2 Software The SIP generators and terminators used the following software configuration: OS Centos 6.0 Emulator / Generator SIP SIPp v3.2-tls-pcap RTP Generator Library libpcap 1.0.0 SSL Generator openssl 1.0.0 HTTP Generator Curl-loader 0.53 1.3 Testing Scenarios 1.3.1 Scenarios We differentiated the following actors in the test procedures: Extension internal extension that generates audio Extension internal extension registered to the PBX without generating audio
SIP Trunk simulating access to PSTN We simulated the following traffic types: IVR - incoming traffic from SIP trunk to DDI routed via IVR (DTMF) and directed to an ACD queue from which it directed to Extension IN incoming traffic directed from SIP Trunk to DDI and directly to Extension OUT outgoing voice traffic generated from Extension and directed to the SIP Trunk. LOCAL voice traffic generated by Extension and directed to other Extension In the table below we have specified all types of traffic together with the maximum number of simultaneous calls of each traffic type. For some types of traffic we have defined specific scenarios such as: call recording, Automatic Call Distribution based on a Round-Robin algorithm as well as using the PBX as an RTP / media proxy. Traffic Type Initiator Terminator Recording ACD Media Proxy IVR-NOREC SIP Trunk Extension IVR-REC SIP Trunk Extension IN-NOREC SIP Trunk Extension IN-REC SIP Trunk Extension OUT LOCAL Extension Extension Calls NO YES YES 82 YES YES YES 20 NO NO YES 189 NO NO YES 21 SIP Trunk N/A N/A NO 90 Extension N/A N/A NO 110 1.3.2 SIP Registration Generator Every Extension had to be registered as a SIP UA to the PBX so it could initiate or terminate a voice connection. The other 2378 extensions also had to register as SIP UA irrespective of the fact that they did not generate voice traffic. Traffic Type Initiator Terminator Method Calls REG-VOICE Extension 3CX Location Service Digest Authentication 622 IVR-REC Extension 3CX Location Service Digest Authentication 2378
1.3.3 HTTP Queries Generator The Admin and user GUI is based on Abyss WWW server from Aprelium. Our last test scenario component was generating HTTP GET queries to the WWW server as an approximation of how many users would want to check missed calls, for example, or reconfigure their follow me rules. Traffic Type Initiator Terminator Objective Queries HTTP GET curl-loader Abyss Web Server 3CX MyPhone login page 25/Sec 1.4 Test Method We specified 3 test methods, from which we would analyze results. 1.4.1 Idle Status In this method we wanted to find out the reference load on the server when it does not process any SIP connections. 1.4.2 8 Calls per 10 Seconds In this method we decided to generate 8 calls every 10 seconds, where every call had a length of 800 seconds. We wanted to see how the system performed when the load was rising at a linear pace and test the system resilience to a high load over a long period of time. Simultaneously, we were generating 25 queries per second to the WWW server, which is part of the 3CX Phone System application. 1.4.3 8 Calls per Second In this method we decided to generate 8 calls every second, where every call was 80 seconds long. This scenario is unlikely in the real world but we wanted to see how the system performed when the load was rising fast and test the system resilience to a high resource allocation in a short period of time. Simultaneously we were generating 25 queries per second to the WWW server, which is part of the 3CX Phone System application. 1.5 Collecting Results We used Microsoft Performance Monitor as well as VMware vsphere Client to CSV files. We were interested in: CPU Load, Memory usage, I/O Average, Network Bandwidth Average, CPU Time for 3CX Processes and Datastore Latency.
2.0 Preparation While preparing the test environment, we encountered a few elements that need highlighting to better understand the overall test and results. 2.1 SIP Generator In order to obtain an appropriate packet sequence that simulated SIP registration, call processing, authorization and call disconnect, we needed a very scalable SIP generator. We chose Sipp, an open source tool that has been used for years in various SIP stress tests. The main reason for this choice is that it has low hardware requirements and is simple to use. The only drawback, which we encountered while using Sipp, was that when under a high load, 3CX Phone System would occasionally send SIP messages in the wrong sequence. This generated errors in the next Sipp call attempts. The structure of each test assumes a rigorous SIP message sequence. Our workaround was to treat some SIP messages as optional. 2.2 RTP Generator The biggest challenge of testing VoIP solutions is the necessity to generate a large number of RTP streams in an appropriate codec. Since coding such a high number of RTP streams would require a processor or a transcoding card, we decided to use a Sipp module called PCAP Play. PCAP Play allowed us to play an 8 second sound file transmitted using G711A and loop it the required number of times in order to achieve desired audio sample length. 2.3 SIP/RTP Terminator In the case of the SIP/RTP Terminator we used the same tricks as in the Generator. The RTP stream was echoed back to Generator using the Sipp RTP Echo application. 2.4 3CX Phone System Configuration 3CX Phone System was configured with 3,000 user accounts and appropriate routing rules. Thankfully, 3CX Phone System allows a simple import of a CSV files. This cut the required time from years to 15 minutes, and all necessary info was quickly imported. Creating DDIs proved to be a problem since 3CX Phone System does not offer an automatic import of DDIs. Inserting them manually was a pain.
3.0 Results 3.1 Idle The graphs below present the resources usage for OS and 3CX Phone System Enterprise 512SC Edition while no calls were being processed and no SIP or HTTP queries were executed. 3.1.1 Microsoft Performance Monitor The results below were collected using Microsoft Performance Monitor and present the total usage of I/O hard drive (read/write) in bytes per second, CPU usage time (4 cores) in a percentage scale, available (not allocated) RAM memory in Megabytes, as well as the total bandwidth transferred through a network interface (Rx / Tx) in Bytes per second. All other results will be presented in the same fashion.
On the graphs below we see the total CPU usage for individual processes that are part of 3CX Phone System, while no connections (SIP or HTTP) were made. 3.1.2 VMware vsphere Below, we see the results collected using VMware vsphere Client Performance Report which presents the total I/O usage of physical hard drive (read/write) in kilobytes per second, total usage of CPU (all 4 cores) in a percentage scale, and used RAM in kilobytes per second. Other tests from vsphere will be presented in the same fashion for other test scenarios.
3.2 8 Calls per 10s The graphs below present resource usage while conducting the 8 calls per 10 seconds test scenario. In these graphs we have included the number of active calls. The test lasted for 25 minutes. The final peak in I/O results is a consequence of an RDP access login. The presented results using vsphere and PerfMon show very similar characteristics of resource usage. This simply means that resources are correctly allocated and freed. While using this test, we had 3% lost calls in the ACD mechanism or due to a wrong SIP message sequence sent to SIP generator. This could also be a timing issue between virtual machines. On the terminator side we did not notice any problems. It is noticeable that bigger resources are needed for call setups then for the maintenance of the call. This is not applicable to I/O HDD which, after call setup and IVR playback, had to record some of the calls. 3.2.1 Microsoft Performance Monitor Below are the results of this test using Microsoft Performance Monitor.
On the graph below one can observe the total CPU usage for individual processes of 3CX Phone System at a call rate of 8 calls per second. 3.2.2 VMware vsphere Below are the results of this test, collected using VMware vsphere Client Performance Report.
3.3 8 Call per Second Stress Test The graphs below present the resource usage while conducting the 8 calls per second test scenario. In these graphs we have included the number of active calls. The presented results, using vsphere and PerfMon, show very similar characteristics of resource usage. This simply means that resources are correctly allocated and freed. In this test, failed calls amounted to 8%. This is because of the large number of calls being setup simultaneously. Such scenarios are extremely unrealistic in real world production environments. Again, the errors were caused by calls being lost in the ACD engine and the wrong sequencing of SIP messages sent to SIP generator. We are aware that these 37 calls were dropped because of a lack of available CPU resources in the time of 13:20 peak. Again it is noticeable that system resources needed are higher for call setups rather then call maintenance. This does not apply to I/O HDD which behaved differently to the first test and recorded highest load at the end of the test. This is due to recording of calls. 3.3.1 Microsoft Performance Monitor Below are the results of this test collected using VMware vsphere Client Performance Report.
On the graph below one can observe that the total CPU usage for individual processes of 3CX application while the 8 calls per second test was conducted. 3.3.2 VMware vsphere Below are the results of this test collected using VMware vsphere Client Performance Report.
4.0 Summary A system that can handle advanced call scenarios for 512 simultaneous calls can easily handle a company with up to 3,000 extensions. Such a number is akin to a small ITSP. This means that 3CX has designed 3CX Phone System very ambitiously indeed. Our tests were conducted with a 50/50 feeling, especially given the demanding scenarios designed (virtualization, recording, abnormally high number of call setups per second). However our tests have shown that the 512 SC license is not just a marketing gimmick and 3CX Phone System is not only an SMB product but, with appropriate architecture design, has potential for even large enterprises. While examining these results we also bear in mind that the license cost and maintenance cost of 3CX Phone System is probably the lowest available on the market. Please note: Adding more resources to the machine, for example, more RAM & more CPU power, would resolve the failed calls issue we saw when pushing to the limit. About the Authors The tests on 3CX Phone System were carried out by Michał Podoski and Andrzej Pierścionek. Michal Podoski has been active in the IT and Telecommunication market for over 7 years. During that time, he held senior engineering and managerial positions. He is a carrier grade class specialist in networking and VoIP services. For many years, Michał has been using and abusing the potential of open source voice platforms. His main interests are securing in SIP, VAS, VoIP and cloud computing networks. He is currently a senior engineer at HaloKwadrat. Andrzej Pierscionek has gained experience in implementing software PBX infrastructure and creating call centre solutions. His main interests are networks of all sizes and VoIP. Andrzej is currently an engineer at HaloKwadrat. About HaloKwadrat HaloKwadrat is a 3CX Distributor in Warsaw, Poland and prides itself on being a leading distributor in the field of VoIP telephony. HaloKwadrat distinguishes itself by combining excellent pricing along with its great support of 3CX Phone System, Patton, Sangoma, and Polycom products. HaloKwadrat organizes basic and advanced 3CX training events each year and works with over 200 partners throughout Europe.