Oracle s Acme Packet Net-Net 6300 Performance Testing Report July 2013 Executive Summary To address the growth and complexity of telecommunications services, today s leading communications service providers (CSPs) now need more powerful, more capable Session Border Controllers (SBCs) not only to support large numbers of users, but also to manage the increasing array of applications across multiple end devices. In October 2012, Acme Packet, now Oracle, announced a new SBC platform, Oracle s Acme Packet Net-Net 6300, designed to deliver increased throughput, performance, and capacity for signaling, media, transcoding, and encryption in a 3U form factor. Oracle engaged Current Analysis and the Illinois Institute of Technology (IIT) Real-Time Communications (RTC) Lab to test and validate the performance of Oracle s Acme Packet Net-Net 6300 Session Border Controller (SBC). Performance tests were drawn from the IETF Methodology for Benchmarking SIP Network Devices specification. The use of a vendor-independent methodology enhances the objectivity of performance comparisons and gives network operators a higher degree of confidence in product performance. The key performance parameters to be validated for the Net-Net 6300 were SIP session establishment rate, capacity and registration against which four test suites were designed. The test suites exercised the Net- Net 6300 using a Tier 1 CSP traffic mix in both access and peering modes, with and without media and using a variety of codecs. The product met all its performance targets without errors, summarized as follows: A session establishment rate of 1,000 sessions per second established with media and 2,400 sessions per second without media A capacity of 80,000 concurrent sessions with media and 192,000 concurrent sessions without media A registration rate of 7,800 registrations per second leading to a potential registration capacity of 1,900,000 endpoints (limited only by test equipment) Product Profile Vendor: Oracle Product: Oracle s Acme Packet Net-Net 6300 Session Border Controller (SBC) Target Markets: SIP applications in CSP networks for peering and access network environments Testing Period: April 9-18, 2013 Report Released: July 1, 2013 Software Tested: Net-Net 6300 SCZ 7.0.2 f2 Tests Conducted: Session establishment rate Session capacity Registration capacity System Configuration: Common SBC features: UDP transport Topology hiding Media anchoring DDoS protection (trusted endpoints for peering and untrusted end-points for access) High availability (full redundancy for failover) Registration caching for access Europe: +33 (0) 1 41 14 83 15 www.currentanalysis.com 1
The above results are based on the maximum number of sessions which can be licensed on the system which is 80,000. Even so, it should also be recognized that the test parameters were still far in excess of the recommended performance for production networks e.g. the Net-Net 6300 is advertised as supporting 1 million subscribers, although the tests showed the Net-Net 6300 capable of supporting nearly double that population. Additionally, monitoring of processor utilization revealed very little variation across codec types, giving added assurance that media traffic mixes do not affect the performance of the product. Market Overview and Analysis SIP Applications in Communications Service Provider Networks CSPs moving to IP networks are being challenged on two levels going forward. First, these CSPs need to scale their signaling and routing networks, as the number of IP endpoints continues to grow rapidly. This is because customers are flocking away from basic telephone service, to new types of services and devices; and because voice providers are hollowing out their own expensive, legacy circuit-switched voice networks in favor of cost-saving IP infrastructure. Second, the complexity of IP-based communications is increasing. Operators are using SIP not just for voice, but to validate and help manage videoconferencing/telepresence; users conduct audio conferences that can be tied to collaboration sessions; and a mix of roaming subscribers and mobile devices can make SIP registration and initiation requests 'appear' anywhere across the network, at any time. The SIP command set itself is straightforward the trick is in the Session Border Controllers (SBCs) that need to authenticate, process/normalize, connect, monitor/manage, track, log and account for these calls. It's this comprehensive service wrap by the SBCs that makes IPT a quality experience for the subscriber, and a billable service for an operator. Modern SBC platforms combine purpose-built processing with tight integration of signaling and media controls in a single network element. To address the growth and growing complexity of services, providers will need more powerful, more capable SBCs. These SBCs will not just need to support large numbers of users, but also to support an array of applications across end devices. Objectives Oracle engaged a team of researchers from the Illinois Institute of Technology RTC Lab to create and execute a testing program designed to provide a benchmark for the Net-Net 6300 platform. The IIT team makes regular contributions to the Internet Engineering Task Force (IETF) standards developments. Oracle also engaged Current Analysis, as an independent third party to oversee the testing program and validate the platform s performance characteristics in terms of session establishment rate, session capacity and registration capacity. In order to characterize the performance of the platform, the testing objectives were divided into four separate test suites, each with its own objective: Test Suite 1 Baseline Session Establishment Rate Performance The objective of this test suite is to baseline the session establishment rate of the test bed configuration i.e. without the Net-Net 6300 (see Fig 1.). In this way, the maximum session establishment rate of the test bed is verified as sufficient for the later test suites when the Net-Net 6300 is inserted as the device under test (DUT). Europe: +33 (0) 1 41 14 83 15 www.currentanalysis.com 2
Test Suite 2 Net-Net 6300 Session Establishment Rate Performance After inserting the Net-Net 6300 into the test configuration (see Fig 2.), the session establishment rate performance of the Net-Net 6300 (DUT) is benchmarked both with and without media. Test Suite 3 Net-Net 6300 Session Capacity Performance The session capacity performance of the Net-Net 6300 (DUT) is benchmarked by establishing sessions with and without media. Test Suite 4 Net-Net 6300 Registration Performance The registration capacity performance of the Net-Net 6300 (DUT) is benchmarked by initiating registration requests. Test Methodology The testing methodology was authored by a lab testing team at RTC lab Illinois Institute of Technology in collaboration with Oracle. This was then validated, based on the stated goals of Oracle, by Current Analysis. All but one of the performance tests proposed were from the eighth revision of the draft IETF Methodology for Benchmarking SIP Network Devices referenced below and aligned with the test suites as shown in the following table. 1 Methodology for Benchmarking SIP Networking Devices draft-ietf-bmwg-sip-bench-meth-08 dated January 8, 2013 authored by C. Davids (Illinois Institute of Technology), V. Gurbani (Bell Laboratories, Alcatel-Lucent) and S. Poretsky (Allot Communications) Europe: +33 (0) 1 41 14 83 15 www.currentanalysis.com 3
Table 1: Oracle s Acme Packet Net-Net 6300 Performance Test Suites Relationship with IETF SIP Network Device Benchmarking Methodology Test Suite Test Objective IETF Methodology for Benchmarking SIP Network Devices Reference Scenario 1 Baseline session establishment rate performance 2 Net-Net 6300 session establishment rate performance 3 Net-Net 6300 session capacity performance 4 Net-Net 6300 registration performance 6.1 Baseline session establishment rate of the test bed 6.2 - Session establishment rate without media 6.4 - Session establishment rate with media on DUT/SUT 6.14 - Session capacity without media 6.15 - Session capacity with media 6.11 - Maximum registration rate registration capacity* Benchmarking SBC peering SBC peering SBC access * Currently undefined in the IETF methodology, but derived as an extension of 6.11. Sessions were constructed from 7-message call flows (INVITE/100/180/200/ACK/BYE/OK). With varying session hold times for each test, session establishment was ramped up over time on the basis of zero errors being detected. The call hold time for session establishment rate was set to 180 seconds and for session capacity the call hold time was set to 320 seconds without media and 133 seconds with media. Test Configuration The following diagram shows the high-level physical test configuration used for Oracle s Acme Packet Net-Net 6300 performance testing. Figure 1: Physical Test Bed Configuration Europe: +33 (0) 1 41 14 83 15 www.currentanalysis.com 4
Traffic Generation High performance EXFO QualityAssurer QA604 & QA805 traffic generators capable of IMS & VoLTE emulation were used for SIP protocol simulation, media generation/verification and voice quality measurements, including Mean Opinion (MOS) scores. Net-Net 6300 The DUT was configured without encryption and transcoding resources and as a high availability (full redundancy for failover) system. This would be a common access side SBC configuration for most fixed line consumer VoIP use cases and for SBC interconnect between CSP networks, wireless or wireline. Registrar The registrar was simulated using multiple instances of the OpenSIPS registrar running under a VMware ESXi bare metal hypervisor on a Dell PowerEdge R710 Rack Server. Traffic Mix The traffic mix reflected a Tier 1 service provider in a peering environment with voice (UDP) traffic. Separate traffic mixes for peering and access environments were used for the session establishment rates and registrations respectively. For peering configuration, destination prefix-based routing (route on area code) was utilized. For access configuration, domain-based routing (route to registrar domain) was utilized. Media codecs specifically tested are shown in Table 2 below. Table 2: Oracle s Acme Packet Net-Net 6300 Performance Test Suites Media Traffic Types Tested Test Suite Test Objective 2 Net-Net 6300 session establishment rate performance 3 Net-Net 6300 session capacity performance IETF Methodology for Benchmarking SIP Network Devices Reference 6.4 - Session establishment rate with media on DUT/SUT Media Traffic Type G.711 6.15 - Session capacity with media G.729 G.721 G.723 Performance Measurement Test results were derived from a comparison of measurements from the test equipment and device. Table 3: Performance Measurements Test Suites Test Objective EXFO Measurements Net-Net 6300 Measurements 1 Baseline session performance Outgoing / incoming calls N/A Successfully completed calls 2-3 Session performance show sip Unsuccessfully completed calls show sip sessions Retransmissions show sip invite show proc top 4 Registration performance Registration requested Registration successful/active Registration unsuccessful Retransmissions show sip show sip register show registration show proc top Europe: +33 (0) 1 41 14 83 15 www.currentanalysis.com 5
Test Description and Results Test Suite 1 - Baseline Session Establishment Rate Performance Configuration Figure 2: Baseline Session Establishment Rate Test Configuration Results In a transparent configuration emulating 180,000 endpoints, the following session establishment rates were verified. Table 4: Performance Results - Test Suite 1 - Baseline Session Establishment Rate Without Media With Media Baseline session establishment rate (sessions per second) 3750 2100 Europe: +33 (0) 1 41 14 83 15 www.currentanalysis.com 6
Test Suite 2 Oracle s Acme Packet Net-Net 6300 Session Establishment Rate Performance Configuration Figure 3: Oracle s Acme Packet Net-Net 6300 Session Establishment Rate Test Configuration Results With the Net-Net 6300 inserted into the test configuration, the following session establishment rates were verified. Table 5: Performance Results - Test Suite 2 Oracle s Acme Packet Net-Net 6300 Session Establishment Rate Performance Without Media With Media Net-Net 6300 session establishment rate (sessions per second) 2400 1000 Test Suite 3 Oracle s Acme Packet Net-Net 6300 Session Capacity Performance Configuration As for Test Suite 2. Europe: +33 (0) 1 41 14 83 15 www.currentanalysis.com 7
Results Table 6: Performance Results - Test Suite 3 Oracle s Acme Packet Net-Net 6300 Session Capacity Performance Without Media With Media Net-Net 6300 session capacity (sessions) 192,000* 80,000* * Capacity results were only limited by the test equipment used. Test Suite 4 Registration Performance Configuration Figure 4: Performance Results - Test Suite 4 Oracle s Acme Packet Net-Net 6300 Registration Performance Results Table 7: Performance Results - Test Suite 4 Oracle s Acme Packet Net-Net 6300 Registration Performance Registration Rate Registration Capacity 7800 registrations/second 1,900,000* registrations *Note capacity was restricted only by the limitations of the registrar capacity. Europe: +33 (0) 1 41 14 83 15 www.currentanalysis.com 8
Conclusions These test results confirm the performance of the Net-Net 6300 SBC in meeting the challenges of Tier 1 CSP traffic mixes in today s increasingly demanding SBC access and peering environments. All the tests met their expected performance criteria without errors under SIP benchmarking conditions for networking devices described in the latest IETF test specifications and referenced earlier in this report. The use of vendor-independent benchmarking enhances the objectivity of performance comparisons between vendors and gives network operators a higher degree of confidence that the equipment will perform to claimed parameters. By subjecting the Net-Net 6300 to this testing program, the full life cycle of the Net-Net 6300 s SIP session signaling performance has been verified for sessions with and without media. In summary, this testing program has validated the following: A session establishment rate of 1,000 sessions per second established with media and 2,400 sessions per second without media A capacity of 80,000 concurrent sessions with media and 192,000 concurrent sessions without media (limited only by the test environment) A registration rate of 7,800 registrations per second leading to a registration capacity of 1,900,000 endpoints (limited only by the test environment.) Note that the performance numbers are based on a maximum number of licensable sessions on the system which is 80,000. This license restriction was imposed by Oracle to ensure headroom in production networks for processes on the system such as message processing, registrations, presence, security, and call admission control, to cite a few examples. This performance headroom is used to accommodate the unpredictability of high traffic conditions occurring such as residential power outages or TV program voting responses and additional load due to instant messaging, file transfer and photo sharing invoked during voice calls e.g. rich communication services (RCS). Finally, although this report only documents the key results parameters required by the IETF benchmarking specifications, additional parameters (as seen in Table 3) such as processor occupancy were also monitored during the course of testing). CPU utilization with and without media was consistent, irrespective of the codec used (see Table 2). This small variation in occupancy indicates that operators deploying Oracle s Acme Packet Net-Net 6300 need not be concerned about re-dimensioning their SBC configuration for different traffic codec mixes. Acknowledgements Current Analysis would like to thank IIT Real-Time Communications (RTC) Lab Director Dr. Carol Davids and graduate students Michal Siuty and Tianjiao Sun for support throughout the course of this testing program. Special thanks are due to James Tessier of Oracle who headed up the program on behalf of the company and supplied the test bed environment including test equipment and Oracle s Acme Packet Net-Net 6300. He also arranged for Current Analysis to verify the test suite at every stage of its development prior, during and post-execution. Europe: +33 (0) 1 41 14 83 15 www.currentanalysis.com 9
About the Real Time Communications (RTC) Lab at the Illinois Institute of Technology (IIT) The Real-Time Communications (RTC) Lab at the Illinois Institute of Technology (IIT) is a unique venue in which industry and academia connect and collaborate. The lab is an educational facility dedicated to teaching, research and development activities that further the advancement of networked communications. The outcomes of RTC Lab projects include white papers, research papers, test reports, and software designs and implementations. These may be shared or confidential depending upon mutual understandings and agreements of the participants. Graduate research assistants carry out project tasks with the leadership and support of IIT faculty and staff and mentors from the industry and research communities. About Current Analysis Current Analysis provides Competitive Intelligence Solutions that enable companies to effectively anticipate and counter competitive threats, and win more business. This is achieved through the rapid delivery of tactical, actionable intelligence to help fast-paced companies compete in the highly volatile telecommunications, networking, and business software markets. Current Analysis 21335 Signal Hill Plaza, Suite 200 Sterling, VA 20164, United States Phone +1 703 404 9200 www.currentanalysis.com Europe: +33 (0) 1 41 14 83 15 www.currentanalysis.com 10