Monitoring of Technical Testing. How to setup monitoring for analyzing the results of your technical tests and especially for load testing

Transcription

1 Monitoring of Technical Testing How to setup monitoring for analyzing the results of your technical tests and especially for load testing

2 Technical Testing overview None Functional Requirements (NFR) Validation of Non-functional Requirements Performance Stability Operability Security These requirements are usually applied to complex distributed server applications Supports many interactive users Processing high volume of data coming from several sources Test Automation Component and UI Testing Multi-browser testing API Testing Verification of your test environments Automated provisioning of test environments with underlying software, applications and test data 2 2

3 Best practices of validating NFR with monitoring Measure Response time and expected result from your automated test scenarios Define transactions in test scripts to measure specific steps You may need test scripts for cross web browser testing and different Mobile applications Measure Network latency from different location and ISP Measure performance, scalability & functionality of the SUT in your test environment with monitoring Application level Software level (Application Server, Database) Operating System Level Virtualization & Cloud Correlate the transactions in automated test scenarios with monitoring of the SUT 3 3

4 Example Load testing correlation with monitoring Web Page Transaction Response Time(s) Active users (Virtual User) Web Transaction rate throughput Active users (Virtual User) CPU Usage % - All servers in SUT 4 4

5 Apica experience from real customer engagements Challenges with monitoring of the SUT Sometimes we are not allowed setup monitoring agents on the hosts in the SUT Cloud /SaaS based applications Customer security policies An Application Performance Management (APM) is very useful for performance testing but we can seldom use it No APM platform deployed in the SUT We have no access to the deployed APM platform Challenges with automated generated test environment of SUT Monitoring software need to provisioned automatically End points to application services are often dynamic Constraints and limitations in load test scenarios End points to application services often hard coded in scripts Limited understanding of the need to generate load from different network locations and network capabilities. 5 5 high-speed, mobile 3G and so on

6 Measure & Optimize Web Browser Performance 6 6 6

7 Website Performance Issues It is very important to pinpoint and optimize the performance of your site from front-end perspective Generic guide lines for better performance of your web application Source: Minimize HTTP Requests Use a Content Delivery Network Add an Expires or a Cache-Control Header Gzip Components Put Scripts at the Bottom Make JavaScript and CSS External Reduce DNS Lookups Minify JavaScript and CSS Avoid Redirects Split Components Across Domains 7 7

8 Tools for testing Web page performance Popular free tools on the Web WebPageTest Google PageSpeed Insights YSLow The limitation is that you only can test a single URL in a real browser. No support for testing of complete scenario scripted with automated testing tool like Selenium IDE or Selenium Web Driver 8 8

9 Google Pagespeed Insight - Report 9 9

10 Using Web Performance monitoring platforms There are two flavors of web performance monitoring Active monitoring (Synthetics) Running scripts for complete scenarios (Selenium or similar) Running simple health checks URL, Ping, Port, Traceroute, DNS and so on. Passive monitoring (RUM) Listen to network traffic JavaScript injection Monitoring capabilities Monitor from multiple locations Compare different devices (Browser) Trend reporting Compare results against defined thresholds Custom Dashboards 10 10

11 Using Performance Analytic & Visualization platforms

12 Most popular Performance Analytics tool Splunk Splunk Enterprise makes it simple to collect, analyze, and act upon the value of the big data generated by your technology infrastructure, security systems and business applications-- giving you the insights to drive operational performance and business results. Elastic Stack The Elastic stack is a collection of open source software tools that make gathering insights from data easier. Formerly referred to as the ELK stack (Elasticsearch, Logstash, and Kibana), the growing list of tools that integrate with the platform (such as Beats) have outgrown the acronym. Grafana & InfluxDB Grafana is most commonly used for visualizing time series data for Internet infrastructure and application analytics InfluxDB fast and scalable Time-Series database

13 Example of Visualization &Dashboards 13 13

14 ZIG Apica Zebra Tester results In Grafana ZIG - Apica ZebraTester to work with InfluxDB and Grafana to show live and historical test results. ZIG is developed by Apica Professional Services. We use ZIG when we do performance testing in none Internet access environments and our SaaS based Load Testing Portal can not be used Similar solutions can be developed for other load testing and test automation tools

15 Collecting and analyzing application logs from a complete test environment with Elastic Stack We are collecting applications logs and web access logs in our cloud based demo environment at Apica. All logs are collected and correlated to a common searchable ElasticSearch database. Kibana is used for visualization Ad hoc Discovery of log events with queries Showing defined Dashboards 15 15

16 Architecture of log collection in the Apica demo environment We use Filebeat to collect the log files from our application servers and web cache. Our application servers (Jboss Wildfly) are deployed in Docker containers We use Logstash to parse and enrich the collected logs. We use Elastic Search as consolidated database for the logs

17 Using Application Performance Management tools

18 The APM conceptual framework and usage Gartner has defined a Conceptual Framework for APM The APM Conceptual Framework was designed to help prioritize an approach on what to focus on first for a quick implementation and overall understanding of the five-dimensional APM model APM is very useful in combination with technical testing and especially with web performance testing 18 18

19 The Top APM tools AppDynamics Get real-time, end-to-end management made for the most complex and distributed applications. AppDynamics APM helps you focus on what matters with features application mapping, dynamic baselining, and code-level diagnostics. New Relic New Relic APM gives you detailed performance metrics for every aspect of your environment, in real-time. View app performance down to specific lines of code, see how each component is connected, and resolve problems fast. Dynatrace The Dynatrace APM solution available for today s modern application architectures. Powered by PurePath Technology, Dynatrace captures timing and code level context for all transactions, end-to-end

20 APM Examples Transaction Trace 20 20

21 APM Create your own Dashboards for Load Testing 21 21

22 APM Advanced correlation with Load Testing 1. The load testing script create specific HTTP headers for all HTTP(s) requests. 2. The APM tool collects the HTTP headers and include them in the business transaction trace. 3. The benefit is that you can correlate the web page transactions in load testing scripts with transaction traces in APM on instance level

23 APM Next generation of tools The next generation of APM tools Advanced automatic Service Discovery with one unified agent. Broader coverage of platforms Machine learning Automated incident and change detection Real-time collection of metrics on a second level granularity Based on Big-Data analytic platforms Example is Instana Technical partnership with Apica Integration with Apica Load Testing planed 23 23

24 24 24