Network Virtualisation in Testing Richard Bishop 27 th January 2016 1
Introduction Trust IV Trust IV is an independent specialist provider of software QA solutions and professional IT services across the software development lifecycle. Established 2005 HPE Gold Partner Based in Manchester, UK Richard Bishop Head of Service Delivery at Trust IV Co-leader of Vivit in the UK 17 years experience in IT consultancy, primarily in software risk management Key experience in Finance, Telco and Retail sectors 2
2020 vision and challenges Accurately predicting the impact of what if scenarios Unknown or uncontrollable influences Variety of client access methods and channels Source: Cap Gemini / Sogeti WQR 2015 3
Live vs Test environments Can Development you accurately / Unit-Test mimic production Environment usage in your test environment? Tester s PC Application Server Database / Back-end Server 4
Live vs Test environments Integration Testing environment Tester s PC Other apps Application Servers Multiple Database / Back-end Servers Remote / third-party sites 5
Live vs Test environments Acceptance Testing environment Tester s PC / devices Shared Networks and Cloud Applications Other apps Multiple Database / Back-end Servers Remote / third-party sites 6
Live vs Test environments Possible problems Tester s PC / devices Shared Networks and Cloud Applications Other apps Multiple Database / Back-end Servers Remote / third-party sites 7
Live vs Test environments Problems resolved Shared Networks and Cloud Applications NV Other apps SV SV NV SV SV Multiple Database / Back-end Servers NV SV Remote / third-party sites 8
Real World example Trust IV Blog A day in the life of a forgetful performance engineer 9
Real World example Trust IV Blog A day in the life of a forgetful performance engineer 10
Third-Party links FaceBook 28 th September 2015 Source: venturebeat.com & catchpoint.com 11
Network Virtualisation Concept 12
NV use cases Uncontrolled / unreliable connectivity 13
3G Mobile Quality at Old Trafford 14
3G Mobile Quality at Old Trafford Thursday afternoon During the derby game Before game During game During game Average Latency: 42ms Min Latency: 37ms Max Latency: 62ms Packet loss 0.00% Download 3.4Mbps Source coordinates: (53.46,-2.29) Destination: m.bbc.co.uk Duration: 2min Date and time: 10/22/2015 16:44 Connection: Cellular data network Carrier: vodafone UK OS: ios 7.1.2 Device model: iphone 4 Average Latency: 45ms Min Latency: 39ms Max Latency: 124ms Packet loss 0.00% Download 4.0Mbps Source coordinates: (53.46,-2.29) Destination: m.bbc.co.uk Duration: 2min Date and time: 10/25/2015 14:37 Connection: Cellular data network Carrier: vodafone UK OS: ios 7.1.2 Device model: iphone 4 Average Latency: 43ms Min Latency: 39ms Max Latency: 153ms Packet loss 0.00% Download 5.4Mbps Source coordinates: (53.46,-2.29) Destination: m.bbc.co.uk Duration: 2min Date and time: 10/25/2015 14:41 Connection: Cellular data network Carrier: vodafone UK OS: ios 7.1.2 Device model: iphone 4 15
3G Mobile Quality at Old Trafford Test Case Conditions 3G phone with HSPA support iphone 4 specs Expected Results High latency ( >200 ms ) High packet loss ( > 1-2 % ) Limited bandwidth ( 1 Mbps ) Poor mobile data performance Actual Results Variable latency ( 39 153ms ) No packet loss ( 0% ) 3-5Mbps throughput ( 4.7Mbps avg ) Good but variable network performance 16
3G Mobile Quality at Old Trafford Analysis / Observations 5 Masts with 11 transmitters 4.7 Mbps throughput No packet loss Variable latency Unpredictable network 17
Mobile website performance Monitored key betting sites during Manchester derby at Old Trafford on 25 th October 2015 18
Observations BBC website for comparison Occasional spikes vs BetFred and Bet365 similar tech BetFred slow before 10am Early kick-off football games F1 Grand Prix Austin, Texas RWC Semi-final Twickenham PaddyPower and SkyBet similar PaddyPower fastest overall Showing all charts confusing Midday to 4pm shows busy periods Trendline of all betting sites shows two pre-game peaks 19
Is 4G always better? 20
Is 4G always better? Average Latency: 70ms Min Latency: 31ms Max Latency: 294ms Packet loss: 0.79% Download: 3.32Mbps Average Latency: 67ms Min Latency: 46ms Max Latency: 747ms Packet loss: 0.43% Download: 13.64Mbps 21
HPE Network Virtualization* * Network Virtualisation 22
Impact of mobile users Mobile Client Network & Security Resources Web Server DB Server Request Connection Pool Resources Response Bandwidth Packet Loss Jitter Latency Session Resources Database Resources 23
NV in performance testing WITHOUT accurate network conditions VS WITH accurate network conditions Capacity > 500+ Users Capacity < 200 Users 24
HPE Network Virtualization Products NV Analytics Analytics recommends performance improvements Grades applications / websites for mobile / desktop performance Simple results analysis, results shared easily within project teams. Now built into LoadRunner NV Test Manager Test Manager manages network profiles during Manual functional tests Automated functional tests e.g. LeanFT, UFT HPE Mobile Center tests NV Server NV Server can (optionally) be used to store test results User management and pooling of NV reports. Options to use servers to gather network statistics. Network Catcher Express ios and Android Client software Real world performance measurements. Contributes to world-wide database as well as providing personalised network performance statistics for testing. 25
NV in standalone mode Client and NV server AUT Servers 26
NV in proxy mode NV server configured as MITM proxy server. Default port 8888. Multiple MITM instances supported. Clients configured to use NV server as MITM proxy server. Individual flows can be configured based on IP address of client device AUT Servers 27
NV as a router NV Test Manager 2 NICs configured as router 28
NV in performance tests 3G 4G Wifi Controller Load Generator (s) Application Under Test Results analysis Filtered by location (network profile) Performance improvement recommendations Mobile or Desktop recommendations Drill down to resource usage etc. 29
Video Demo Mobile Center & NV integration (functional tests) NV & LoadRunner integration (performance tests) 30
Closing statement NV within the application development lifecycle Test composite and cloud-based apps more effectively Improve performance for all users Optimise the mobile user experience 31
Closing statement Retail sites Increasingly catering for mobile users Vulnerable to slow users consuming resources StormRunner Quality Center Performance Center Betting sites Customers impatient / less loyal - impact on customer retention NV SV Vulnerable to extreme variations in network Performance, require detailed testing Network Virtualization Service Virtualization Mobile Center ALM Sprinter BPT LoadRunner UFT What if testing only possible with SV / NV together 32
Questions Richard Bishop Trust IV Ltd richard.bishop@trustiv.co.uk @trustiv 33