IERG 4080 Building Scalable Internet-based Services

Transcription

1 Department of Information Engineering, CUHK Term 1, 2015/16 IERG 4080 Building Scalable Internet-based Services Lecture 10 Load Testing Lecturer: Albert C. M. Au Yeung 18 th November, 2015

2 Software Performance Testing 2

3 Performance Testing How do you know that your system can perform well enough? Can it serve the users quickly? Can it serve the anticipated number of users? Can it cope with the anticipated amount of workload? Application Server Clients Load Balancer Application Server Database Server Application Server 3

4 Performance Testing Questions you should ask even before your application goes online How many users will the application need to serve at release, after 3 months, 6 months and 1 year? Where will the users come from (geographically, device used, etc.)? How many users will use the application simultaneously? What are typical scenarios of using the application? 4

5 Performance Testing What is it? To test a system in order to understand its responsiveness, throughput, reliability and scalability under certain conditions Why? Assess how ready is the system for production Evaluate the system against some criteria Compare different systems Locate problems and identify components to be tuned Understand the throughput of the system Taken from: 5

6 Performance Testing Different categories (or types) of performance testing Load test Stress test Capacity test Regression test Volume test 6

7 Stress Testing Subject the system to extremely high work load and see what problems will happen under those conditions Identify problems that might occur (e.g. synchronization issues, race conditions, memory leaks, ) Spike test is a subset study the system s behaviour and problems under sharp increase in work load in a short period of time Questions to answer: What is the max load a system can support before it breaks down? How does the system break down? Is the system able to recover after it has crashed? 7

8 Baselines and Benchmarking Baseline A reference point against which any performance-improving changes to the application can be evaluated A baseline can be created for different layers in the application (e.g. load balancer, application server, database server, etc.) Benchmarking The process of comparing your application s performance against a baseline, or against a standard widely accepted in the community 8

9 Load Testing 9

10 Load Testing The process of putting demand on a system and measure its response Understand the behaviour of the system under: 1. Normal load conditions 2. Anticipated peak load conditions Identify bottlenecks In Internet-based services, the load is defined in terms of the number of concurrent users or HTTP connections 10

11 Load Testing SLA Service Level Agreement An agreement between your application and the users on the quality of the service provided Common metrics Uptime (e.g %) ASA (Average speed to answer) TSF (Time service factor, e.g. 95% of requests within 1s) TAT (Turn-around time) MTTR (Mean time to recover) 11

12 12

13 Load Testing Some examples of load testing: Editing a very large document in a word processor Sending a huge file to a printer for printing Creating many concurrent users to browse a e-commerce Website Creating many concurrent users to view and edit a public wiki 13

14 Load Testing What should we study in a test? The time required for a request to be processed (i.e. response times) Errors Are requests finished? 14

15 Load Testing Things to consider in a load test What are the common scenarios when users are using your application Read-to-write ratio What will be the habits of the users Expected load in different situations (e.g. weekdays vs. weekends) 15

16 Load Testing What should you measure in a load test? Response times (how long does the user have to wait?) Error rates (how many requests are dropped?) How many requests can be answered in 1s? How many database queries can be performed in 1s? How many files can be written/read in 1s? 16

17 Load Testing Load Profiles The variation of work load across time and situations (ref. load profile in electrical engineering) Factors to be considered The possible activities of the users Common behaviours of the users (and their ratios) Time taken for a user to respond to changes in the application (think time) Abandonment rate (how often does a user chooses to leave?) 17

18 Load Profile Consider a e-commerce Website What are the possible actions? Browsing Searching Adding items to shopping cart Checking out and settling payments Changing their profiles Writing comments on items 18

19 Modelling Application Usage In order to really understand whether your system can serve your users efficiently and effectively, your tests should be realistic In order words, you need to have a good model of the application s usage Identifying application usage profiles is known as workload modelling 19

20 Modelling Application Usage What are the methods for coming up with a good model of workload? What are the objectives? What are the key scenarios? What are the common user behaviours? What are the navigation paths in key scenarios? How much data would each user generate? Which scenarios are more likely? What are your target load levels? 20

21 Modelling Application Usage 21

22 Tools for Load Testing 22

23 Load Testing Available Tools Ab (Apache HTTP server benchmarking tool) ( Apache JMeter ( httperf ( Tsung ( Funkload ( Locust ( Loads ( 23

24 Apache Bench (ab) A command line tool for benchmarking HTTP servers Can be used to easily study how many requests per second your Web server can handle Install by $ sudo apt-get install apache2-utils 24

25 Apache Bench (ab) Examples Send 200 requests with maximum 50 concurrent users to google.com $ ab -n 200 -c 50 Send 500 requests with maximum 10 concurrent users to yahoo.com, output data into a.csv file $ ab -n 500 -c 10 e output.csv 25

26 Apache Bench (ab) Benchmarking (be patient)...done Server Software: ATS Server Hostname: Server Port: 80 Document Path: / Document Length: 374 bytes Concurrency Level: 10 Time taken for tests: seconds Complete requests: 100 Failed requests: 0 Non-2xx responses: 100 Total transferred: bytes HTML transferred: bytes Requests per second: [#/sec] (mean) Time per request: [ms] (mean) Time per request: [ms] (mean, across all concurrent requests) Transfer rate: [Kbytes/sec] received Connection Times (ms) min mean[+/-sd] median max Connect: Processing: Waiting: Total: Percentage of the requests served within a certain time (ms) 50% 66 66% 68 75% 69 80% 70 90% 73 95% 76 98% 79 99% % 80 (longest request) 26

27 Apache Bench (ab) Plotting distribution of response times of requests 27

28 Locust.io Main Site: Doc: Open source: A user load testing tool written in Python You can define the user behaviour in Python code and load test your application A Web UI for monitoring the process in real-time Distributed allow running load tests over multiple machines 28

29 Locust.io Defining user behaviour from locust import HttpLocust, TaskSet, task class UserBehavior(TaskSet): def on_start(self): self.login() def login(self): self.client.post("/login", {"username": user", "password": def index(self): def profile(self): self.client.get("/profile") 29

30 Locust.io Defining user behaviour from locust import HttpLocust, TaskSet, task class UserBehavior(TaskSet): def on_start(self): self.login() def login(self): self.client.post("/login", {"username": user", "password": def index(self): def profile(self): self.client.get("/profile") Define one behaviour of a user (the actions that the user will take when using your application) on_start will be invoked whenever a new user is created (in this case the user signs into the service) 30

31 Locust.io Defining user behaviour from locust import HttpLocust, TaskSet, task class UserBehavior(TaskSet): def on_start(self): self.login() Define a function in which the user will carry out one action def login(self): self.client.post("/login", {"username": user", "password": def index(self): def profile(self): self.client.get("/profile") self.client is an HTTP client by which you can make requests to your application, it supports GET, POST, PUT, DELETE methods You can also pass parameters using a Python dictionary 31

32 Locust.io Defining user behaviour from locust import HttpLocust, TaskSet, task class UserBehavior(TaskSet): def on_start(self): self.login() def login(self): self.client.post("/login", {"username": user", "password": def index(self): def profile(self): defines an action that the user will carry out while he is online x represents the ratio of how often the user will perform this action. 32

33 Locust.io Define a user class WebsiteUser(HttpLocust): task_set = UserBehavior min_wait = 5000 max_wait = 9000 Defines a user task_set: the set of tasks the user will perform min_wait: minimum time in ms that the user will wait between tasks max_wait: maximum time in ms that the user will wait between tasks Run $ locust -f locust_file.py --host= 33

34 Locust.io If you want to simulate a lot of work load, you will need to use the distributed mode (using multiple machines) First, put the test script in both the master and the slave(s) machine. Then, start Locust in the master by $ locust -f locust_file.py --host= --master Start Locust in the slave(s) by $ locust -f locust_file.py --host= --slave --master-host= For details, see 34

35 Locust.io Web UI for running and monitoring tests (default running at port 8089) 35

36 Locust.io Running load test Total number of requests sent Number of failed requests Statistics for response time Size of the content returned Number of requests per second 36

37 Funkload A functional and load testing module written in Python Some features include: Can be used to emulate a Web browser Support cookies, https, HTTP authentication, file upload, etc. Can be used for functional testing, load testing or stress testing Generate reports in HTML format for review Installation: $ sudo apt-get install funkload 37

38 Funkload Creating a simple test import unittest from random import random from funkload.funkloadtestcase import FunkLoadTestCase class Simple(FunkLoadTestCase): def setup(self): self.server_url = self.conf_get('main', 'url') def test_simple(self): server_url = self.server_url res = self.get(server_url, description='get url') self.assertequal(res.code, 200) self.assertequal(res.body, "Hello World") if name in ('main', ' main '): unittest.main() 38

39 Funkload Configuration file (a.conf file with the same name as the class in your test script) [main] title = Demo description = Simple demo url = [test_simple] description = Access our Demo app [ftest] log_to = console file log_path = simple-test.log result_path = simple-test.xml sleep_time_min = 0 sleep_time_max = 0 [bench] cycles = 5:10:20 duration = 10 startup_delay = 0.01 sleep_time = 0.01 cycle_time = 1 log_to = log_path = simple-bench.log result_path = simple-bench.xml sleep_time_min = 0 sleep_time_max =

40 Funkload Running a benchmark $ fl-run-bench -c 1:10:20 test_simple.py Simple.test_simple Data will be stored in an XML file. HTML-formatted report can be generated by $ fl-build-report --html simple-bench.xml Example: 40

41 Profiling Python Programs 41

42 Profiling You have finished writing your code. How do you know that it is efficient? Is your implementation good enough? Can the application perform 20% faster or use 20% less memory? And how can you understand that if you want to? Profilers Ref: 42

43 Profiling A profile is a set of statistics that describes how often and for how long various parts of the program executed. Profiling helps us to understand: How fast is the program running? What are the bottlenecks? How much memory is the program using? 43

44 Timing Program Execution The simplest way: use the Unix utility time to measure the execution time of a program $ time python test_profiling.py real 0m2.458s user 0m2.108s sys 0m0.344s real the actual elapsed time from start to end user the amount of CPU time spent outside of kernel sys the amount of CPU time spent inside kernel specific functions 44

45 Profilers cprofile A C extension which adds relatively small overhead profile A pure Python module adds significant overhead to the program being profiled. Can be extended to build a customised profiler Other tools Line profiler ( 45

46 The profile Module import profile import random def swap(x, i, j): x[i], x[j] = x[j], x[i] return x def bubblesort(x): for i in xrange(len(x)): for j in xrange(i): if (x[i] < x[j]): x = swap(x, i, j) return x x = [] for i in xrange(1000): x.append(random.randint(1,50000)) profile.runctx('bubblesort(x)', globals(), {'x':x}) 46

47 The profile Module function calls in seconds Ordered by: standard name ncalls tottime percall cumtime percall filename:lineno(function) :0(len) :0(setprofile) <string>:1(<module>) profile:0(bubblesort(x)) profile:0(profiler) test_profiling.py:4(swap) test_profiling.py:8(bubblesort) 47

48 The cprofile Module Call cprofile on the command line to profile a program $ python m cprofile testing.py Save output to a text file $ python m cprofile o output.txt testing.py Read and print out file content $ python >>> import pstats >>> p = pstats.stats( output.txt ) >>> p.strip_dirs().sort_stats(-1).print_stats() 48

49 Load Test in Your Project 49

50 Load Test Your Application In your project, after finished the application, you are required to load test your application and study its performance Come up with a simple but reasonable workload model (e.g. a common scenario in which the user signs in and carry out several actions) Investigate the distribution of response times of major endpoints (e.g. how long does the user have to wait when submitting a new article or a new photo?) 50

51 End of Lecture 10 51