Telecom White Paper Automatig Performace Testig of Network Virtualizatio Platforms
About the Author Sahil Shah Sahil Shah is the Techical Lead of NextGe R&D Telecom iitiatives withi Tata Cosultacy Services' (TCS') Techology busiess uit. He is a Iteratioal Software Testig Qualificatios Board (ISTQB) certified techical test aalyst ad has over seve years of experiece i the IT idustry, spaig developmet ad testig i the telecom domai. Sahil specializes i iche techologies such as Software Defied Networkig (SDN) ad focuses o performace bechmarkig, security, cotroller developmet i OpeDayLight, virtualizatio/opestack ad specialized implemetatio of proof-ofcocepts leveragig the OpeStack framework.
Abstract I recet times, there has bee a expoetial growth i the demad for etwork virtualizatio. Network virtualizatio platform (NVP) service providers have lauched products that eable cloud data ceters to cater to a large ad diversified set of cliet eeds ad requiremets. However, i the absece of prove evaluatio frameworks, cloud service providers are ucertai about the performace of these solutios. This is drivig the etwork virtualizatio service providers to bechmark the performace of their products ad gai competitive advatage through higher reliability ad availability. I order to do this, they eed effective tools for performace evaluatio ad bechmarkig based o predetermied metrics. This paper highlights the eed for automatig the performace measuremet of etwork virtualizatio systems ad the challeges ivolved i the process. It discusses the framework ad approach for a automated performace testig tool for NVPs that leverage OpeStack etworkig techology. The paper also details the architecture, essetial compoets ad capabilities of the tool.
Cotets Buildig Scalable ad Efficiet Virtual Networks: The Need for Performace Evaluatio 6 Automatig Performace Testig of Network Virtualizatio Platforms 7 Achievig Tagible Beefits with Automated Testig 11 Preparig to Meet Future Virtualizatio Requiremets 11
List of Abbreviatios CRUD FWaaS IaaS LBaaS NFV NVP SDN VM VPNaaS Create, update, ad delete Firewall as a Service Ifrastructure as a service Load Balacer as a Service Network Fuctios Virtualizatio Network Virtualizatio Platform Software Defied Network Virtual Machie Virtual Private Network as a Service
Buildig Scalable ad Efficiet Virtual Networks: The Need for Performace Evaluatio Buildig a efficiet data ceter etwork remais a high priority for data ceter providers i order to streamlie costs, ad improve scalability ad agility. However, data ceter providers typically face certai key challeges i achievig this objective. These iclude: Moitorig, maagig, ad evaluatig hybrid etworks: With the emergece of cloud ad IaaS, the challege of processig demads for dyamic workloads has bee addressed by provisioig virtual etworks withi software defied data ceters. Today's data ceters are therefore a mix of physical ad virtual etworks. The challege lies i effectively evaluatig the performace of these hybrid etworks. Achievig scalable deploymet: I order to exted the virtual etworks across data ceters or to make them accessible to the outside world, the deploymet eeds to be scalable ad have the ability to support itegratio with other cloud deploymets. A robust etwork virtualizatio platform (NVP) is the aswer to addressig these challeges. Data ceter providers icreasigly use etwork orchestrators such as OpeStack i cloud provisioig. They are also lookig to esure that NVPs such as OpeCotrail or NSX provide comparable improvemets i the followig aspects: Cost: Elimiate maual cofiguratio i the etwork provisioig process ad simplify etwork hardware requiremets to achieve capital ad operatioal cost beefits Agility ad efficiecy: Reduce the time take to provisio custom, multi-tier etwork topologies ad provide cosistet results Security: Create closed user groups to provide secure access to legitimate users such as guests, parters, ad vedors Flexibility: Itegrate with ay cloud maagemet platform, ad operate o ay hypervisor ad etwork hardware Scalability: Scale to accommodate future requiremets with miimal chages Cosequetly, data ceter operators ad NVP vedors are costatly lookig for the ability to evaluate the performace of the etwork virtualizatio platforms. However, their itet behid performace measuremet is differet. Data ceter operators are focused o evaluatig various platforms to select the solutio best suited to their requiremets, or to predict bottleecks i upgradig to ewer versios of the same solutio. They are also iterested i uderstadig how adoptig differet cloud ifrastructure desig patters, icludig coverged ad o-coverged data ceter cofiguratios, affects performace. Platform vedors evaluate the performace of their products to bechmark them agaist other platforms. They ru regular performace tests itegrated with their developmet eviromet to keep a check o ay performace degradatio across multiple versios of their product. 6
Automatig Performace Testig of Network Virtualizatio Platforms The absece of a ed-to-ed idustry wide performace measuremet mechaism for virtualizatio platforms is a key challege. For istace, OpeStack ca itegrate with ay software defied etwork (SDN) cotroller that supports its etworkig API (Neutro) for creatig ad maagig virtual etworks. The SDN cotrollers ca further itegrate with multiple platforms through southboud plugis, makig the ed-to-ed deploymet a multi-tier architecture. I this sceario, a automated tool is essetial to effectively measure the performace of the ed-toed architecture as maual processes icrease costs, ad reduce the efficiecy ad accuracy of results. The ext challege is desigig a automated tool that eables users to ru performace test cases o the ed-toed deploymet. A ideal solutio must help capture results from various etwork tiers, collate the results, ad prepare the report i a ituitive format. The tool must also help defie performace parameters such as creatio ad deletio of virtual machies (VMs) - for example, VMs coected per hour ad VMs deleted per hour,- ad capture results based o them. As a huge ecosystem of products is ivolved i creatig the virtual etworks, the testig tool must effectively evaluate if the ed-to-ed deploymet coforms to the defied performace requiremets. The proposed performace testig tool addresses these requiremets ad automates the performace test process for virtualizatio platforms based o OpeStack techology. It is a ed-to-ed performace testig solutio supportig multiple deploymet scearios. Some of the supported scearios iclude: Performace Tool -> OpeStack -> ODL -> OpeCotrail/ovsdb Performace Tool -> OpeStack -> OpeCotrail/NSX Performace Tool -> OpeStack cofigured with vedor-specific Neutro plugi -> Vedor specific mechaism. Vedor specific Neutro plugis iclude plugis from Big Switch, Brocade, Cisco, Cloudbase, Mellaox, Midoet, NEC, NICIRA, PLUMgrid ad Ryu. Figure 1 depicts the compoets of the proposed solutio. Graphical User Iterface or Remote API Iterface Requests Test Sceario Opestack Neutro Request Status Jobs Scheduler Test Result Testig Framework Test Suites Test Scripts Other Ope Stack Requests Rally Iterface NFV Specific Request Results Reportig Captured Data Results Report NFV Iterface Database Performace Data Capture Utility Stores Data Capture Real Time Performace Data Opestack Rally Neutro Iterface Opestack NVS [OpeCotrail] Figure 1: Compoets of the performace testig tool 7
Graphical User Iterface The automated performace measuremet tool icludes a user-friedly graphical user iterface that eables users to: Defie the parameters to evaluate the performace of a particular deploymet. Users ca cofigure these parameters icludig load (umber of simultaeous users) ad scearios to ru (create, update, ad delete operatios o etworks, ports, subets, routers, floatig IPs, security groups, bootig VMs). These also iclude parameters specific to test scearios o the uderlyig OpeStack orchestratio system. Create performace test batches from the available test scearios, cofigure how the batches will ru (sequetial, parallel or with a delay i betwee), ad schedule them for performace rus. Moitor the performace rus from the jobs scheduler scree. Validate performace test rus agaist defied performace parameters ad provide the fial verdict of 'Pass' or 'Fail' for a particular test sceario. Save the performace ru results ad access them usig the iterface to query the database. Geerate reports i the specified format such as pdf/csv agaist the user selected metrics icludig creatio ad deletio of VMs, VMs coected per hour, ad VMs deleted per hour. Publish the saved results o a cetralized web page that ca be viewed across locatios usig a URL. Remote Applicatio Programmig Iterface The tool provides users a applicatio programmig iterface that eables them to write their ow customized scripts. NVP providers ca use these performace scripts as part of performace testig i a cotiuously itegrated developmet eviromet. Jobs Scheduler The task of this module is to schedule jobs based o user-defied ruig cofiguratio (parallel, sequetial ad periodic). For ruig the jobs i parallel, it creates a umber of threads that are equal to the umber of jobs ad ru the test scripts/suites available i the performace testig framework. I sequetial mode, it creates a sigle thread to ru the jobs sequetially without delay. I the periodic mode, a sigle thread rus the jobs sequetially with a user-defied iterval betwee two cosecutive jobs. The tool also displays the progress of the jobs i real time o the graphical user iterface. Performace Testig Framework (Testig the etworkig capabilities of OpeStack) The performace testig framework cosists of test suites ad idividual test scripts for ruig the etworkig test scearios. The suite accesses the Neutro iterface to execute the followig scearios: Networks: Create, update, list, ad delete etwork. Subets: Create, update, list, show, ad delete subet. Ports: Create, update, list, show, ad delete port. Layer three etworkig: Create, update list, ad delete router ad floatig IP; add ad remove iterface from router. 8
Meterig labels ad rules: Create, list, ad delete meterig label ad meterig label rules. Security groups ad rules: Create list ad delete security group ad security group rules. Virtual Private Network as a Service (VPNaaS): Create, update, list ad delete virtual private etwork (VPN) services, Iteret key exchage (IKE) policy, IP security policy, ad IP security site coectio. Load Balacer as a Service (LBaaS): Create, update, list ad delete load balacer, listeers, pools, members, ad health moitors. Firewall as a Service (FWaaS): Create, update, list ad delete firewall, firewall policy, ad firewall rule. The test suites eable the logical groupig of the test scripts ito tailor made scearios for ruig specific test cases. The test scearios described above are icluded as pre-defied test suites i the tool. Further test suites ca be added accordig to the deploymet. This eables users to add, modify or exted test suites based o their specific eeds. Here is a example of a deploymet sceario for which a test suite ca be added: 1. Create a virtual etwork 'TestNetwork' ad associate a subet with this etwork. 2. Boot a VM specifyig a sigle Network Iterface Card (NIC) coected to 'TestNetwork'. 3. Create port triggered by OpeStack Nova to create a port 'TestPort' attached with 'TestNetwork'. The IP will be allocated to the port from the Neutro API pool. 4. Destroy VM, which i tur, deletes the port. The allocated IP will be retured to the pool of available IP addresses. 5. Delete the subet ad etwork. The testig tool ca also be used to test the deployed etwork for real time varyig loads where a certai umber of cocurret users are simulated to sed multiple requests at a give time. The followig is a sample test cofiguratio for real time varyig loads: 5% users sedig create/update etworks requests 10% users sedig create/update subets requests 25% users sedig create/update ports requests 15% users sedig create/update router requests 10% users sedig create/update floatig IP requests 10% users sedig delete etwork requests 10% users sedig delete subet requests 15% users sedig delete ports ad router requests Database Cetralized data storage provides correlatio amog collected metrics. The database stores all performace test ru results ad test cofiguratios, ad the tool provides a iterface to query the database ad access the same. 9
Performace Data Capture Utility The testig framework triggers this utility whe a script is ru. It eables the tool to capture data from idividual odes. Apart from capturig the ed-to-ed respose time for a particular ru, this compoet features a smart logs parser. The parser parses the logs of idividual odes to calculate the time spet by a idividual module i processig the request. This capability for i-depth aalysis ca be leveraged to idetify performace bottleecks ad areas that require fie tuig. I additio, this module captures the logs for all the compoets, ad compresses ad saves them to a prespecified locatio (URL). This simplifies error diagosis as all the data is made available for a detailed aalysis. The cetralized URL also specifies the path of the logs agaist the test result, makig the logs accessible from ay locatio. The captured performace data is the stored o the tools database. For test scearios for OpeStack modules other tha etworkig, this utility captures performace data by queryig the OpeStack Rally database. Results Reportig This module is resposible for producig comprehesive reports that eable users to compare test rus ad report treds. Users are able to geerate detailed reports after test executio. They have the optio to view test results as tables ad graphs ad geerate reports based o the metrics selected from the iterface. The tool supports the followig metrics: Number of cocurret users Number of trasactios i give time (throughput); for example, maximum umber of trasactios per secod Respose time (latecy); for example, maximum respose time, average respose time ad miimum respose time Rally Iterface (Iterface to OpeStack modules except Neutro) The OpeStack platform is still evolvig, but the tool provides a iterface to all existig OpeStack test utilities across versios. Usig the Rally iterface, all the basic OpeStack modules ca be performace tested, icludig Keystoe, Nova, Heat, Glace, Cider, ad Celiometer. Apart from ruig performace tests, this iterface ca be utilized to ru fuctioality test cases o the OpeStack automated testig framework Tempest. This eables the tool to ru automated ed-to-ed regressio testig o the deploymet. Network Fuctios Virtualizatio (NFV) Iterface The customized iterface to the NFV solutio eables testig of NFV solutios. This is a vedor specific iterface ad ca be provided specifically if required by the NVP vedor. For example, if we have to test a OpeCotrail NVP directly o the performace metrics, this iterface ca be developed to itegrate with the OpeCotrail specific APIs, eablig the vedor to test their NFV solutio leveragig the beefits provided by the tool framework. 10
Achievig Tagible Beefits with Automated Testig The automated performace testig tool for virtualized etworks is a ed-to-ed solutio that supports multiple deploymet scearios. It offers NVP providers the opportuity to bechmark ad improve their products to gai competitive advatage. It also allows cloud service providers to evaluate ad select the product that meets their reliability, availability ad scalability requiremets. It drives the followig tagible busiess beefits: Lower capital costs: The tool ca test etwork performace across deploymets, helpig data ceter providers reduce capital expediture o multiple tools. The tool ca also be exteded to perform fuctioal testig, deliverig further savigs. Superior results with lower effort: Oe-click automatio reduces the time to test multi-tier topologies, providig a efficiet framework to achieve cosistet results. Icreased security: The solutio provides secure access to differet user groups, thus supportig ehaced group level security. Greater scalability: The testig tool ca scale up to test multiple etwork virtualizatios, hybrid etworks ad deploymets across cloud etworks. Preparig to Meet Future Virtualizatio Requiremets Network virtualizatio systems play a crucial role i meetig the growig busiess requiremets for ehaced etworkig resources ad ifrastructure, ad this demad is oly goig to grow. There is scope for a large umber of service providers to coexist i the ecosystem ad fulfill the virtualizatio requiremets of data ceter providers, with cost effectiveess, flexibility, scalability, ad efficiecy beig the decisive competitive factors. Performace evaluatio ad bechmarkig of etwork virtualizatio solutios ca help providers drive improvemets across all offerigs ad thus deserves extesive ivestigatio. The automated performace evaluatio tool proposed i this paper addresses curret challeges i etwork virtualizatio performace evaluatio, deliverig a comprehesive, automated solutio that is easy to implemet ad use. As virtualizatio techologies evolve ad offerigs mature, this tool ca be exteded to deliver testig across techologies ad o a broader set of metrics to meet future requiremets. 11
About TCS' Telecom Busiess Uit For over three decades, TCS has worked with compaies i the Commuicatios idustry, helpig them develop world class solutios ad systems, simplify operatios, augmet etwork services, diversify ito ew markets, ad stay ahead of the competitio. Our solutios address the etire ladscape of telecom busiesses, ragig from greefield start-ups to large cable ad satellite compaies ad equipmet vedors. Our partership model leverages ready-to-deploy platform solutios, proprietary frameworks, ad solutio accelerators to eable our cliets to lauch ew products ad services quickly. TCS' ivestmets i ext geeratio platforms, mobility solutios, capabilities, ad lab ifrastructure are aliged to address our cliets' key busiess imperatives of operatioal excellece, customer experiece, ad reveue ehacemet. We have helped eterprises ulock busiess value i the areas of Digital Chael ad Commerce, Customer Isights, Mobility Solutios, Hosted OSS/BSS (HOBS), IOT & M2M Platforms, Coected Device Maagemet, B2B Platform Ecosystem ad Services ad Network Abstractio layers. Cotact For more iformatio about TCS Telecom Busiess Uit, visit: http://www.tcs.com/idustries/telecom/pages/default.aspx Email: global.telecom@tcs.com Subscribe to TCS White Papers TCS.com RSS: http://www.tcs.com/rss_feeds/pages/feed.aspx?f=w Feedburer: http://feeds2.feedburer.com/tcswhitepapers About Tata Cosultacy Services (TCS) Tata Cosultacy Services is a IT services, cosultig ad busiess solutios orgaizatio that delivers real results to global busiess, esurig a level of certaity o other firm ca match. TCS offers a cosultig-led, itegrated portfolio of IT ad IT-eabled ifrastructure, egieerig ad TM assurace services. This is delivered through its uique Global Network Delivery Model, recogized as the bechmark of excellece i software developmet. A part of the Tata Group, Idia s largest idustrial coglomerate, TCS has a global footprit ad is listed o the Natioal Stock Exchage ad Bombay Stock Exchage i Idia. For more iformatio, visit us at www.tcs.com IT Services Busiess Solutios Cosultig All cotet / iformatio preset here is the exclusive property of Tata Cosultacy Services Limited (TCS). The cotet / iformatio cotaied here is correct at the time of publishig. No material from here may be copied, modified, reproduced, republished, uploaded, trasmitted, posted or distributed i ay form without prior writte permissio from TCS. Uauthorized use of the cotet / iformatio appearig here may violate copyright, trademark ad other applicable laws, ad could result i crimial or civil pealties. Copyright 2015 Tata Cosultacy Services Limited TCS Desig Services I M I 09 I 15