RALLY PERFORMANCE BENCHMARKING OF OPENSTACK. Tips and tools for creating and presenting wide format slides

RALLY PERFORMANCE BENCHMARKING OF OPENSTACK Tips and tools for creating and presenting wide format slides

Openstack cloud platform

Current Openstack - Ecosysten

OpenStack Basics 4 Everything is written in python End users can interact through a common web interface (Horizon) or directly to each service through their API All services authenticate through a common source Individual services interact with each other through their public APIs * Most daemons implemented WSGI middleware (Paste) Used extensively in OpenStack Configured through *-paste.ini files

Identity ( Keystone ) Keystone provides a single point of integration for OpenStack policy, catalog, token and authentication. keystone handles API requests as well as providing configurable catalog, policy, token and identity services. Standard backends include LDAP or SQL, as well as Key Value Stores (KVS). Most people will use this as a point of customization for their current authentication services. OpenStack Identity Service token backend keystone (ser vice & admin APIs) cat alog backend policy backend ident it y backend 5

Dashboard ( Horizon ) Django application that users can access in their web browser Communicates with each OpenStack service through their API (and sometimes their admin API) HTTP(S) Hor izon OpenSt ack Dashboard 6

Object Storage ( Swift ) account account DB swif t-proxy cont ainer cont ainer DB memcached object object st ore OpenSt ack Object St ore Stores and serves objects (files) Employs object level replication to safeguard data Accepts client requests via Objectstore API or HTTP from clients through swift-proxy Maintains distributed account and container databases Stores objects according the ring layout on filesystem with extended attributes (XFS, EXT4, etc.) 7

Image Service ( Glance ) glance-api accepts Image API calls for image discovery, image retrieval and image storage. glance-registry stores, processes and retrieves metadata about images (size, type, etc.). Database to store the image metadata. A storage repository for the actual image files. In many deployments, this is OpenStack Swift glance-api glance-regist r y glance dat abase OpenSt ack Image Ser vice 8

Compute ( Nova ) nova-compute libvirt, XenAPI, etc. hyper visor nova-api (OS, EC2, Admin) nova-consoleauth nova dat abase Queue nova-scheduler nova-console nova-cert/ objectstore nova- conduct or OpenSt ack Compute nova-api accepts and responds to end user compute API calls. Supports OpenStack Compute API, Amazon's EC2 API and a special Admin API (for privileged users to perform administrative actions). Initiates most of the orchestration activities (such as running an instance) Enforces some policy (mostly quota checks) Authentication is handled through middleware before getting to this daemon 9

Block Storage ( Cinder ) cinder-api accepts API requests and routes them to cinder-volume for action. cinder-volume acts upon the requests by reading or writing to the Cinder database to maintain state, interacting with other processes (like cinder-scheduler) through a message queue and directly upon block storage providing hardware or software. It can interact with a variety of storage providers through a driver architecture. Currently, there are drivers for IBM, SolidFire, NetApp, Nexenta, Zadara, linux iscsi and other storage providers. Much like nova-scheduler, the cinder-scheduler daemon picks the optimal block storage provider node volume provider OpenSt ack Block St orage 10 cinder-api cinder-volume cinder dat abase cinder-scheduler

Networking ( Quantum ) quant um-ser ver quantum-server accepts API requests and then routes them to the appropriate quantum plugin for action. Quantum ships with plugins and agents for: quant um agent (s) Queue quant um plugin(s) Cisco virtual and physical switches Nicira NVP product NEC OpenFlow products Open vswitch net work provider quant um dat abase Linux bridging Ryu Network Operating System Midokua OpenSt ack Net work Ser vice The common agents are L3 (layer 3), DHCP (dynamic host IP addressing) and the specific plug-in agent. 11

Rally - Why? "How does OpenStack work at scale?

Rally Components Server Providers - provide servers (virtual servers), with ssh access, in one L3 network. Deploy Engines - deploy OpenStack cloud on servers that are presented by Server Providers Verification - component that runs tempest (or another specific set of tests) against a deployed cloud, collects results & presents them in human readable form. Benchmark engine - allows to write parameterized benchmark scenarios & run them against the cloud.

Rally -Functionality

Rally Use Cases

Installing Rally on Ubuntu. Note : We would recommend using a separate machine for Rally. If the machine that you use to run rally has some Openstack components running, we suggest creating a virtual environment for running Rally because it may have conflicts with version of the client python libraries. Prerequisite sudo apt-get update sudo apt-get install libpq-dev git-core python-dev libevent-dev libssl-dev python-pip libffi-dev sudo pip install pbr

Installing Rally on Ubuntu. Installing Rally Clone git clone https://github.com/stackforge/rally.git && cd rally python setup.py install Configure sudo mkdir /etc/rally sudo cp etc/rally/rally.conf.sample /etc/rally/rally.conf sudo vim /etc/rally/rally.conf # Change the "connection" parameter, # e.g. to connection="sqlite://///home/<your_username>/.venv/rally-db/$sqlite_db" (or any other place) Create Database rally-manage db recreate

Rally : Deploy Engines. The task of a deploy engine is to control the process of deploying some OpenStack distribution like DevStack or FUEL before any benchmarking procedures take place. Every deploy engine should implement the following fairly simple interface: constuctor, which takes a deployment entity as its only parameter; deploy(), which should deploy the appropriate OpenStack distribution given the cloud config from the deployment object the engine was initialized with (possibly using one of available server providers) cleanup(), which should clean up the OpenStack deployment (again, possibly using one of available server providers).

Rally. Deploy Engine. Example If you already have a existing Openstack Deployment that you want to benchmark Use a json file that looks like one below. (with value specific to your system): { } "name": "DummyEngine", "endpoint": { "auth_url": "http://192.168.122.22:5000/v2.0/", "username": "admin", "password": "password", "tenant_name": "admin" } For Devstack based deployment { "name": "DevstackEngine", "localrc": { "ADMIN_PASSWORD": "secret", "NOVA_REPO": "git://example.com/nova/",...

constructor, which should take the deployment entity the provider should bind to and a config dictionary as its parameters; create_servers(image_uuid, type_id, amount), which should create the requested number of virtual machines of the given type using a specific image. The method should also return the list of created servers wrapped in special Server entities. destroy_servers(), which should destroy all virtual machines previously created by the same Rally Server Providers Server providers in Rally are typically used by deploy engines to manage virtual machines necessary for OpenStack deployment and its following benchmarking. The key feature of server providers is that they provide a unified interface for interacting with different virtualization technologies (LXS, Virsh etc.) and cloud suppliers (like Amazon). Every server provider should implement the following basic interface:

Rally Server Providers Ex DummyProvider If you already have an Openstack Deployment. This provider does nothing, but returns endpoints from configuration. This may be useful if you have specific software/hardware configuration ready to deploy OpenStack. { } "name": "ExampleEngine", "provider": { "name": "DummyProvider", "credentials": [{"user": "root", "host": "host1.net"}, {"user": "root", "host": "host2.net"}] }

Rally How_to_run Simple 1. Initialize your Deployment 2. Create a json for Supported Benchmarking scenario 3. Run Benchmarking against a deployment above

Rally Initialize Deployment 1. Create a Deployment configuration (json) file. If you are running Rally against a existing Openstack Deployment your should look like { "name": "DummyEngine", "endpoint": { "auth_url": <KEYSTONE_AUTH_URL>, "username": <ADMIN_USER_NAME>, "password": <ADMIN_PASSWORD>, "tenant_name": <ADMIN_TENANT> } } 2. Create a deployment using deployment create command $ rally deployment create --filename=dummy_deployment.json --name=dummy 3. If you want to list deployments $ rally deployment list 4. Switch to a different Deployment $ rally use deployment --deploy-id=<another deployment UUID>

Rally Set Benchmark scenario. Some sample configurations are provided at rally/doc/samples/tasks/. Lets pick up a scenario boot-and-delete.json from Nova. It looks like { } "NovaServers.boot_and_delete_server": [ { "args": {"flavor_id": 1, "image_id": "73257560-c59b-4275-a1ec-ab140e5b9979"}, "execution": "continuous", "config": {"times": 10, "active_users": 2, "tenants": 3, "users_per_tenant": 2} } ] Modify this to design your test-case. Similarly for all other cases other available json can be modified or you can even write a new one for a custom case. Rally by default uses the last created deployment. Use the switch deployment commands to run the tests with a different deployment.

Lets dig deeper Test name : NovaServers.boot_and_delete_server Execution : either continuous / periodic Times : Number of times the test needs to be run Active_users : Number of parallel threads (concurrent users) Tenants : Total number of tenants to be created Users_per_tenant : Number of users per single tenant Other parameters to be used only with supported tests Network : Name of network to be used Script : If a script is passed as input to the test Actions : soft_reboot / stop_start

Rally Run Benchmark. Run your benchmark scenario by pointing at the json you created in the previous step $ rally --verbose task start --task=my-task.json You can check the state of the task $ rally task list To check a complete task analysis $ rally task detailed <Task UUID>

test scenario NovaServers.boot_and_delete_server args position 0 args values: {u'args': {u'flavor_id': <Flavor UUID>, u'image_id': u'<image UUID>'}, u'config': {u'active_users': 1, u'times': 2}} +--------------------+---------------+---------------+---------------+ action max (sec) avg (sec) min (sec) +--------------------+---------------+---------------+---------------+ nova.boot_server 9.22798299789 8.90022659302 8.57247018814 nova.delete_server 4.24928498268 3.26377093792 2.27825689316 +--------------------+---------------+---------------+---------------+ Rally Result Example. $ rally task detailed <Task UUID> =========================================================================== ===== Task <Task UUID> is finished. Failed: False --------------------------------------------------------------------------------

Detailed performance benchmarking

Docker in OpenStack Havana Nova virt driver which integrates with docker REST API on backend Glance translator to integrate docker images with Glance Icehouse Heat plugin for docker Both options are still under development nova-docker virt driver docker heat plugin 9/12/2014 29

About This Benchmark Use case perspective As an OpenStack Cloud user I want a Ubuntu based VM with MySQL Why would I choose docker LXC vs a traditional hypervisor? OpenStack Cloudy perspective LXC vs. traditional VM from a Cloudy (OpenStack) perspective VM operational times (boot, start, stop, snapshot) Compute node resource usage (per VM penalty); density factor Guest runtime perspective CPU, memory, file I/O, MySQL OLTP Why KVM? Exceptional performance 9/12/2014 30

Benchmark Environment Topology @ SoftLayer rally dstat + Awesome! glance api / reg nova api / cond / etc keystone nova api / cond / etc cinder api / sch / vol KVM controller compute node rally dstat + Awesome! glance api / reg nova api / cond / etc keystone nova api / cond / etc cinder api / sch / vol docker lxc controller compute node 9/12/2014 31

Benchmark Specs Spec Controller Node (4CPU x 8G RAM) Compute Node (16CPU x 96G RAM) Environment Bare Metal @ SoftLayer Bare Metal @ SoftLayer Mother Board SuperMicro X8SIE-F Intel Xeon QuadCore SingleProc SATA [1Proc] SuperMicro X8DTU-F_R2 Intel Xeon HexCore DualProc [2Proc] CPU Intel Xeon-Lynnfield 3470-Quadcore [2.93GHz] (Intel Xeon-Westmere 5620-Quadcore [2.4GHz]) x 2 Memory (Kingston 4GB DDR3 2Rx8 4GB DDR3 2Rx8 [4GB]) x2 (Kingston 16GB DDR3 2Rx4 16GB DDR3 2Rx4 [16GB]) x 6 HDD (LOCAL) Digital WD Caviar RE3 WD5002ABYS [500GB]; SATAII Western Digital WD Caviar RE4 WD5003ABYX [500GB]; SATAII NIC eth0/eth1 @ 100 Mbps eth0/eth1 @100 Mbps Operating System Ubuntu 12.04 LTS 64bit Ubuntu 12.04 LTS 64bit Kernel 3.5.0-48-generic 3.8.0-38-generic IO Scheduler deadline deadline Hypervisor tested NA - KVM 1.0 + virtio + KSM (memory deduplication) - docker 0.10.0 + go1.2.1 + commit dc9c28f + AUFS OpenStack Trunk master via devstack Trunk master via devstack. Libvirt KVM nova driver / nova-docker virt driver OpenStack Benchmark Client OpenStack project rally NA Metrics Collection NA dstat Guest Benchmark Driver NA - Sysbench 0.4.12 - mbw 1.1.1.-2 - iibench (py) - netperf 2.5.0-1 VM Image NA - Scenario 1 (KVM): official ubuntu 12.04 image + mysql snapshotted and exported to qcow2 1080 MB - Scenario 2 (docker): guillermo/mysql -- 334.8 MB 9/12/2014 32

Test Descriptions: Cloudy Benchmarks Benchmark Benchmark Driver Description Serial VM boot (15 VMs) VM reboot (5 VMs rebooted 5 times each) VM snapshot (1 VM, 1 snapshot) OpenStack Cloudy Benchmarks OpenStack Rally - Boot VM from image - Wait for ACTIVE state - Repeat the above a total of 15 times - Delete VMs OpenStack Rally - Boot VM from image - Wait for ACTIVE state - Soft reboot VM 5 times - Delete VM - Repeat the above a total of 5 times OpenStack Rally - Boot VM from image - Wait for ACTIVE state - Snapshot VM to glance image - Delete VM 9/12/2014 33

Test Descriptions: Guest Benchmarks Benchmark Benchmark Driver Description Guest Runtime Benchmarks CPU performance Sysbench from within the guest - Run sysbench cpu test - Repeat a total of 10 times - Average results over the 10 times OLTP (MySQL) performance Sysbench from within the guest - Run sysbench OLTP test - Repeat a total of 10 times - Average results over the 10 times MySQL Indexed insertion benchmark - Run iibench for a total of 1M inserts printing stats at 100K intervals - Collect data over 5 runs & average File I/O performance Sysbench from within the guest - Synchronous IO - No-direct - Run sysbench OLTP test - Repeat a total of 10 times - Average results over the 10 times Memory performance Mbw from within the guest - Run mbw with array size of 1000 MiB and each test 10 times - Collect average over 10 runs per test Network performance Netperf - Run netperf server on controller - From guest run netperf client in IPv4 mode - Repeat text 5x - Average results 9/12/2014 34

OpenStack Cloudy Benchmark SERIALLY BOOT 15 VMS 9/12/2014 35

Active VMs Cloudy Performance: Serial VM Boot Benchmark scenario overview Boot VM via OpenStack nova Wait for VM to become active Repeat the above steps for a total of 15 VMs Delete all VMs 20 15 10 5 0 Benchmark Visualization 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 Time VMs 9/12/2014 36

Time in Seconds Cloudy Performance: Serial VM Boot 7 Docker 1.5x faster Average Server Boot Time 6 5.884197426 5 4 3 3.900927941 docker KVM 2 1 0 9/12/2014 37 docker KVM

CPU Usage In Percent CPU Usage In Percent 100 80 60 40 20 Cloudy Performance: Serial VM Boot 0 Docker: Compute Node CPU 1 3 5 7 9 11 13 15 17 19 21 23 25 27 29 31 33 35 37 39 41 43 45 47 49 51 53 55 57 59 61 63 65 67 69 71 73 75 77 79 81 83 85 87 89 Time Averages 1.14 0.44 usr sys 100 80 60 40 20 0 KVM: Compute Node CPU 1 5 9 13 17 21 25 29 33 37 41 45 49 53 57 61 65 69 73 77 81 85 89 93 97 101105109113117121125129133137 Time Averages 12.6 2.08 usr sys 9/12/2014 38

Memory Used Memory Used Cloudy Performance: Serial VM Boot 1 5 9 13 17 21 25 29 33 37 41 45 49 53 57 61 65 69 73 77 81 85 89 93 97 101 105 109 113 117 121 125 129 133 137 5.00E+09 4.00E+09 3.00E+09 2.00E+09 1.00E+09 0.00E+00 Docker: Compute Node Used Memory 1 3 5 7 9 11 13 15 17 19 21 23 25 27 29 31 33 35 37 39 41 43 45 47 49 51 53 55 57 59 61 63 65 67 69 71 73 75 77 79 81 83 85 87 89 Time Delta 687 MB Per VM 45.8 MB Memory 5.00E+09 4.00E+09 3.00E+09 2.00E+09 1.00E+09 0.00E+00 KVM: Compute Node Used Memory Delta 2775 MB Per VM 185 MB Memory Time 9/12/2014 39

1 Minute Load Average 1 Minute Load Average Cloudy Performance: Serial VM Boot Docker: Compute Node 1m Load Average Average 25 20 0.09 % 15 10 5 0 1 3 5 7 9 11 13 15 17 19 21 23 25 27 29 31 33 35 37 39 41 43 45 47 49 51 53 55 57 59 61 63 65 67 69 71 73 75 77 79 81 83 85 87 89 Time 1m KVM: Compute Node 1m Load Average 25 9.94 % 20 15 10 5 1m 0 1 5 9 13 17 21 25 29 33 37 41 45 49 53 57 61 65 69 73 77 81 85 89 93 97 101105109113117121125129133137 Time Average 9/12/2014 40

OpenStack Cloudy Benchmark SERIAL VM SOFT REBOOT 9/12/2014 41

Active VMs Cloudy Performance: Serial VM Reboot Benchmark scenario overview Boot a VM Wait for it to become active Soft reboot the VM Wait for it to become active Repeat soft reboot a total of 5 times Delete VM Benchmark Visualization Repeat 6 the above for a total of 5 VMs 5 4 3 2 1 0 1 3 5 7 9 11 13 15 17 19 21 23 25 27 29 31 33 35 37 39 41 43 45 47 49 51 53 55 Active VMs 9/12/2014 Time 42

Cloudy Performance: Serial VM Reboot Time In Seconds 140 120 Average Server Reboot Time Docker 18.9x faster 124.4525079 100 80 60 docker KVM 40 20 6.591313448 0 docker KVM 9/12/2014 43

Time In Seconds Cloudy Performance: Serial VM Reboot 9 7.857602167 8 Average Server Delete Time KVM 1.75x faster 7 6 5 4 4.451871514 docker KVM 3 2 1 0 docker KVM 9/12/2014 44

Cloudy Performance: Serial VM Reboot After investigating docker delete times Docker sends SIGTERM to container process to stop Bash is immune to SIGTERM Docker waits for X seconds before stop Default container image (init) command was using bash See: https://github.com/dotcloud/docker/issues/3766 Rebuild the docker mysql image Don t use bash for container image command No change to docker (hypervisor); image config only 9/12/2014 45 change

Cloudy Performance: Serial VM Reboot Time In Seconds 140 Average Server Reboot Time (Round 2) Docker 48.99x faster 124.4525079 120 100 80 60 docker KVM 40 20 2.541945305 0 docker 9/12/2014 46 KVM

Time In Seconds Cloudy Performance: Serial VM Reboot 5 4.5 4 3.5 3 4.093254519 Average Server Delete Time (Round 2) Docker 1.09x faster 4.451871514 2.5 2 docker KVM 1.5 1 0.5 0 docker KVM 9/12/2014 47

CPU Usage In Percent 1 74 147 220 293 366 439 512 585 658 731 804 877 950 1023 1096 1169 1242 1315 1388 1461 1534 1607 1680 1753 1826 1899 1972 2045 2118 2191 2264 2337 2410 2483 2556 2629 2702 2775 2848 2921 2994 3067 3140 CPU Usage In Percent 1 7 13 19 25 31 37 43 49 55 61 67 73 79 85 91 97 103 109 115 121 127 133 139 145 151 157 163 169 175 181 187 193 199 205 211 217 223 229 235 241 Cloudy Performance: Serial VM Reboot 10 8 6 4 2 0 Docker: Compute Node CPU Averages 0.34 0.11 usr sys Time 10 8 6 4 2 0 KVM: Compute Node CPU Averages 0.82 0.19 usr sys Time 9/12/2014 48

1 79 157 235 313 391 469 547 625 703 781 859 937 1015 1093 1171 1249 1327 1405 1483 1561 1639 1717 1795 1873 1951 2029 2107 2185 2263 2341 2419 2497 2575 2653 2731 2809 2887 2965 3043 3121 Memory Used 1 7 13 19 25 31 37 43 49 55 61 67 73 79 85 91 97 103 109 115 121 127 133 139 145 151 157 163 169 175 181 187 193 199 205 211 217 223 229 235 241 Memory Used Cloudy Performance: Serial VM Reboot 2.50E+09 2.00E+09 1.50E+09 1.00E+09 5.00E+08 0.00E+00 Docker: Compute Node Used Memory Delta 57 MB Memory Time 2.50E+09 2.00E+09 1.50E+09 1.00E+09 5.00E+08 0.00E+00 KVM: Compute Node Used Memory Delta 467 MB Memory Time 9/12/2014 49

1 Minute Load Average 1 71 141 211 281 351 421 491 561 631 701 771 841 911 981 1051 1121 1191 1261 1331 1401 1471 1541 1611 1681 1751 1821 1891 1961 2031 2101 2171 2241 2311 2381 2451 2521 2591 2661 2731 2801 2871 2941 3011 3081 3151 1 Minute Load Average 1 7 13 19 25 31 37 43 49 55 61 67 73 79 85 91 97 103 109 115 121 127 133 139 145 151 157 163 169 175 181 187 193 199 205 211 217 223 229 235 241 Cloudy Performance: Serial VM Reboot Docker: Compute Node 1m Load Average Average 3.5 0.05 % 2.5 1.5 0.5 1m -0.5 Time 4 3 2 1 0 KVM: Compute Node 1m Load Average Average 0.35 % 1m Time 9/12/2014 50

OpenStack Cloudy Benchmark SNAPSHOT VM TO IMAGE 9/12/2014 51

Cloudy Performance: Snapshot VM To Image Benchmark scenario overview Boot a VM Wait for it to become active Snapshot the VM Wait for image to become active Delete VM 9/12/2014 52

Time (Seconds) 50 Cloudy Performance: Snapshot VM To Image Docker 1.62x faster Average Snapshot Server Time 45 42.92771101 40 35 30 25 20 15 26.39477992 docker KVM 10 5 0 docker KVM 9/12/2014 53

CPU Usage In Percent 1 4 7 10 13 16 19 22 25 28 31 34 37 40 43 46 49 52 55 58 61 64 67 70 73 76 79 82 85 88 91 94 97 100 103 106 109 112 CPU Usage In Percent Cloudy Performance: Snapshot VM To 7 5 3 1 Docker: Compute Node CPU Image Averages 0.4 0.11 usr sys -1 1 3 5 7 9 11 13 15 17 19 21 23 25 27 29 31 33 35 37 39 41 43 45 47 49 51 53 55 57 59 61 63 65 67 Time 8 6 4 2 0 KVM: Compute Node CPU Averages 1.58 1.07 usr sys Time 9/12/2014 54

1 4 7 10 13 16 19 22 25 28 31 34 37 40 43 46 49 52 55 58 61 64 67 70 73 76 79 82 85 88 91 94 97 100 103 106 109 112 Memory Used Memory Used Cloudy Performance: Snapshot VM To 1.94E+09 1.92E+09 1.9E+09 1.88E+09 1.86E+09 1.84E+09 1.82E+09 Docker: Compute Node Used Memory Image 1 3 5 7 9 11 13 15 17 19 21 23 25 27 29 31 33 35 37 39 41 43 45 47 49 51 53 55 57 59 61 63 65 67 Time Delta 48 MB Memory 1.85E+09 1.8E+09 1.75E+09 1.7E+09 1.65E+09 KVM: Compute Node Used Memory Delta 114 MB Memory Time 9/12/2014 55

1 Minute Load Average 1 4 7 10 13 16 19 22 25 28 31 34 37 40 43 46 49 52 55 58 61 64 67 70 73 76 79 82 85 88 91 94 97 100 103 106 109 112 1 Minute Load Average 0.6 0.4 Cloudy Performance: Snapshot VM To Image Docker: Compute Node 1m Load Average Average 0.1 % 0.2 1m 0 1 3 5 7 9 11 13 15 17 19 21 23 25 27 29 31 33 35 37 39 41 43 45 47 49 51 53 55 57 59 61 63 65 67 Time 0.6 KVM: Compute Node 1m Load Average Average 0.37 % 0.4 0.2 0 1m Time 9/12/2014 56

Guest VM Benchmark GUEST PERFORMANCE BENCHMARKS 9/12/2014 57

Guest Performance: CPU Linux sysbench 0.4.12 cpu test Calculate prime numbers up to 20000 2 threads Instance size 4G RAM 2 CPU cores 9/12/2014 20G disk 58

Guest Performance: CPU 16 Calculate Primes 15.11 15.08 15.03 14 12 Seconds 10 8 6 Docker KVM Bare Metal 4 2 0 Docker KVM Bare Metal 9/12/2014 59

Guest Performance: Memory Linux mbw 1.1.1-2 Instance size 2 CPU 4G memory Execution options 10 runs; average 1000 MiB 9/12/2014 60

MiB/s 14000 Guest Performance: Memory Memory Benchmark Performance 12000 10000 8000 6000 4000 BareMetal docker KVM 2000 0 MEMCPY DUMB MCBLOCK Memory Tests 9/12/2014 61

Guest Performance: Network Netperf 2.5.0-1 Netserver running on controller Netperf on guest Run netperf 5 times & average results Instance size 2 CPU 4G memory 9/12/2014 Execution options 62

Throughput In 10^6 bits/second Guest Performance: Network 1000 Network Throughput 900 800 700 600 500 400 docker KVM 300 200 100 0 docker KVM 9/12/2014 63

Guest Performance: File I/O Linux sysbench 0.4.12 fileio test Synchronous IO Random read / write Total file size of 150G 16K block size Thread variations: 1, 8, 16, 32 Instance size 4G RAM 2 CPU cores 200G disk KVM specs Disk cache set to none Virtio Deadline scheduler (host & guest) Docker specs 9/12/2014 AUFS storage driver 64

Guest Performance: File I/O File I/O: Read 300 Mb 200 100 0 1 8 16 32 Docker KVM Threads File I/O: Write 200 150 Mb 100 50 0 1 8 16 32 Docker KVM Threads 9/12/2014 65

Kb/sec 1600 Guest Performance: File I/O Read / Write File I/O: Transfer Rate 1400 1200 1000 800 600 Docker KVM 400 200 0 1 8 16 32 Threads 9/12/2014 66

Guest Performance: MySQL OLTP Linux sysbench 0.4.12 oltp test Table size of 2,000,000 MySQL 5.5 (installed on Ubuntu 12.04 LTS with aptget) Variations Number of threads Read only & read / write Instance size 4G RAM 9/12/2014 67 2 CPU cores

Requests Total Transactions 60000 40000 Guest Performance: MySQL OLTP MySQL OLTP Read Transactions (Read) 20000 0 2 8 16 Threads Docker KVM 800000 600000 400000 200000 0 MySQL OLTP Read Requests 2 8 16 Threads Docker KVM 9/12/2014 68

Requests Total Transactions Guest Performance: MySQL OLTP (Read / Write) MySQL OLTP Read/Write Transactions 25000 20000 15000 10000 docker 5000 KVM 0 2 16 32 64 Threads 400000 300000 200000 100000 0 MySQL OLTP Read/Write Requests 2 16 32 64 Threads Docker KVM 9/12/2014 69

Guest Performance: MySQL Indexed Insertion Indexed insertion benchmark (iibench python script) A total of 1,000,000 insertions Print stats at 100K intervals Collect stats over 5 runs Average Instance size 4G RAM 9/12/2014 70 2 CPU cores

Seconds Per 100K Insertion Batch 140 120 Guest Performance: MySQL Indexed MySQL Indexed Insertion @ 100K Intervals Insertion 100 80 60 docker kvm 40 20 0 1 2 3 4 5 6 7 8 9 10 Table Size In 100K Increments 9/12/2014 71

BENCHMARK OBSERVATIONS 9/12/2014 72

CPU Usage In Percent Docker LXC CPU Growth 26x Lower Than VM 1 4 7 10 13 16 19 22 25 28 31 34 37 40 43 46 49 52 55 58 61 64 67 70 73 76 79 82 85 88 91 94 97 100 103 35 30 User CPU Growth Trend Slope 0.0091 0.237 25 y = 0.237x + 2.2993 20 15 docker KVM Linear (docker) Linear (KVM) 10 5 0 y = 0.0091x + 0.7349 9/12/2014 73

Memory Used Docker LXC Memory Growth 3x Lower 5.00E+09 4.50E+09 4.00E+09 3.50E+09 Memory Usage Growth Trend Than VM y = 3E+07x + 1E+09 Slope 1E+07 3E+07 3.00E+09 2.50E+09 2.00E+09 1.50E+09 1.00E+09 y = 1E+07x + 1E+09 docker KVM Linear (docker) Linear (KVM) 5.00E+08 0.00E+00 1 4 7 10 13 16 19 22 25 28 31 34 37 40 43 46 49 52 55 58 61 64 67 70 73 76 79 82 85 88 91 94 9/12/2014 74

Observations Cloudy operations with Docker LXC outperform VM 48x server reboot, 1.5x server boot, 1.62x server snapshot, etc. Docker LXC density potential compared to VMs 3x memory savings 26x CPU savings 3.22x smaller images in this test (note image sizes can vary based on required packages) Docker LXC containers run on bare metal Greater / equivalent in the VM performance at the micro benchmark level Micro benchmark results do not always reflect macro performance Always benchmark your real workload Docker image (init) command can impact performance Bash ignores SIGTERM Nova-docker virt driver and docker-registry components still under dev Nice work, but room for improvement (python anyone?) 9/12/2014 Real performance of Docker LXC ops capped my Cloud manager 75 Can start the SQL image from docker CLI in 0.191s

REFERENCES 9/12/2014 76

References & Related Links http://www.slideshare.net/bodenrussell/realizing-linux-containerslxc https://www.docker.io/ http://sysbench.sourceforge.net/ http://dag.wiee.rs/home-made/dstat/ http://www.openstack.org/ https://wiki.openstack.org/wiki/rally https://wiki.openstack.org/wiki/docker http://devstack.org/ http://www.linux-kvm.org/page/main_page https://github.com/stackforge/nova-docker https://github.com/dotcloud/docker-registry http://www.netperf.org/netperf/ 9/12/2014 77 http://www.tokutek.com/products/iibench/

REFERENCE 9/12/2014 78

Rally Boot 15 VM Configuration Rally config: { "NovaServers.boot_server": [ { "args": { "flavor_id": 2, "image_id": IMAGE_ID" }, "runner": { "type": "constant", "times": 15, "active_users": 1 }, "context": { "users": { "tenants": 1, "users_per_tenant": 1 } } } ] } Flavor +-----+-----------+-----------+------+-----------+------+-------+-------------+-----------+ ID Name Memory_MB Disk Ephemeral Swap VCPUs RXTX_Factor Is_Public +-----+-----------+-----------+------+-----------+------+-------+-------------+-----------+ 1 m1.tiny 512 1 0 1 1.0 True 2 m1.small 2048 20 0 1 1.0 True 3 m1.medium 4096 40 0 2 1.0 True 4 m1.large 8192 80 0 4 1.0 True 42 m1.nano 64 0 0 1 1.0 True 451 m1.heat 1024 0 0 2 1.0 True 5 m1.xlarge 16384 160 0 8 1.0 True 9/12/2014 6 m1.perf 4096 200 0 2 1.0 True 79 7 m1.sql 512 5 0 1 1.0 True

Cloudy Benchmark: Serially Boot 15 Docker VMs +------------------+-------+--------------+-------------+---------------+---------------+---------------+ action count max (sec) avg (sec) min (sec) 90 percentile 95 percentile +------------------+-------+--------------+-------------+---------------+---------------+---------------+ nova.boot_server 15 4.8055100441 3.900927941 3.64957404137 4.56917948723 4.80114896297 +------------------+-------+--------------+-------------+---------------+---------------+---------------+ +---------------+---------------+--------------+---------------+---------------+---------------+-------------+ max (sec) avg (sec) min (sec) 90 pecentile 95 percentile success/total total times +---------------+---------------+--------------+---------------+---------------+---------------+-------------+ 5.02777004242 4.11018741926 3.8767888546 4.77419886589 5.02389762402 1.0 15 +---------------+---------------+--------------+---------------+---------------+---------------+-------------+ KVM +------------------+-------+---------------+---------------+---------------+---------------+---------------+ action count max (sec) avg (sec) min (sec) 90 percentile 95 percentile +------------------+-------+---------------+---------------+---------------+---------------+---------------+ nova.boot_server 15 7.00468301773 5.88419742584 4.84723997116 6.13021831512 6.48467411995 +------------------+-------+---------------+---------------+---------------+---------------+---------------+ +---------------+--------------+---------------+--------------+---------------+---------------+-------------+ max (sec) avg (sec) min (sec) 90 pecentile 95 percentile success/total total times 9/12/2014 80 +---------------+--------------+---------------+--------------+---------------+---------------+-------------+

Rally Reboot 5x5 Configuration Rally config: { "NovaServers.boot_and_bounce_server": [ { "args": { "flavor_id": 2, "image_id": ID", "actions": [ {"soft_reboot": 5} ] }, "runner": { "type": "constant", "times": 5, "active_users": 1 }, "context": { "users": { "tenants": 1, "users_per_tenant": 1 } } 9/12/2014 81

Cloudy Performance: Serial VM Reboot Docker +--------------------+-------+---------------+---------------+---------------+---------------+---------------+ action count max (sec) avg (sec) min (sec) 90 percentile 95 percentile +--------------------+-------+---------------+---------------+---------------+---------------+---------------+ nova.reboot_server 25 6.85025811195 6.59131344795 6.50809788704 6.65774655342 6.72213015556 nova.boot_server 5 4.96266412735 4.13350987434 3.82891011238 4.61819009781 4.79042711258 nova.delete_server 5 8.74263191223 7.85760216713 6.61050701141 8.72944793701 8.73603992462 +--------------------+-------+---------------+---------------+---------------+---------------+---------------+ +---------------+---------------+---------------+---------------+---------------+---------------+-------------+ max (sec) avg (sec) min (sec) 90 pecentile 95 percentile success/total total times +---------------+---------------+---------------+---------------+---------------+---------------+-------------+ 45.8827331066 44.9664116383 43.5658369064 45.7298994541 45.8063162804 1.0 5 +---------------+---------------+---------------+---------------+---------------+---------------+-------------+ KVM +--------------------+-------+---------------+---------------+---------------+---------------+---------------+ action count max (sec) avg (sec) min (sec) 90 percentile 95 percentile +--------------------+-------+---------------+---------------+---------------+---------------+---------------+ nova.reboot_server 10 124.951740026 124.452507925 123.876129866 124.950992012 124.951366019 nova.boot_server 2 6.1228749752 5.58395600319 5.04503703117 6.0150911808 6.068983078 nova.delete_server 2 4.45799517632 4.45187151432 4.44574785233 4.45677044392 4.45738281012 +--------------------+-------+---------------+---------------+---------------+---------------+---------------+ +---------------+---------------+---------------+--------------+---------------+---------------+-------------+ max (sec) avg (sec) min (sec) 90 pecentile 95 percentile success/total total times +---------------+---------------+---------------+--------------+---------------+---------------+-------------+ 632.718183994 632.315432072 631.912680149 632.63763361 632.677908802 0.4 5 9/12/2014 82

Rally Snapshot Configuration Rally config: { "NovaServers.boot_and_bounce_server": [ { "args": { "flavor_id": 1, "image_id": IMAGE", "actions": [ {"soft_reboot": 5} ] }, "runner": { "type": "constant", "times": 5, "active_users": 1 }, "context": { "users": { "tenants": 1, "users_per_tenant": 1 } } } ] 9/12/2014 83 }

Cloudy Performance: Snapshot VM To Image Docker (note -- defect deleting image) +--------------------+-------+---------------+---------------+---------------+---------------+---------------+ action count max (sec) avg (sec) min (sec) 90 percentile 95 percentile +--------------------+-------+---------------+---------------+---------------+---------------+---------------+ nova.create_image 1 26.3947799206 26.3947799206 26.3947799206 26.3947799206 26.3947799206 nova.boot_server 2 4.120429039 3.85578501225 3.59114098549 4.06750023365 4.09396463633 nova.delete_server 2 8.61395692825 7.64416801929 6.67437911034 8.41999914646 8.51697803736 +--------------------+-------+---------------+---------------+---------------+---------------+---------------+ +-----------+-----------+-----------+--------------+---------------+---------------+-------------+ max (sec) avg (sec) min (sec) 90 pecentile 95 percentile success/total total times +-----------+-----------+-----------+--------------+---------------+---------------+-------------+ n/a n/a n/a n/a n/a 0 1 +-----------+-----------+-----------+--------------+---------------+---------------+-------------+ KVM +--------------------+-------+----------------+----------------+----------------+----------------+----------------+ action count max (sec) avg (sec) min (sec) 90 percentile 95 percentile +--------------------+-------+----------------+----------------+----------------+----------------+----------------+ nova.delete_image 1 0.700344085693 0.700344085693 0.700344085693 0.700344085693 0.700344085693 nova.create_image 1 42.92771101 42.92771101 42.92771101 42.92771101 42.92771101 nova.boot_server 2 40.9650099277 22.9950100183 5.02501010895 37.3710099459 39.1680099368 nova.delete_server 2 4.47270512581 4.46178817749 4.45087122917 4.47052173615 4.47161343098 +--------------------+-------+----------------+----------------+----------------+----------------+----------------+ +--------------+--------------+--------------+--------------+---------------+---------------+-------------+ max (sec) avg (sec) min (sec) 90 pecentile 95 percentile success/total total times +--------------+--------------+--------------+--------------+---------------+---------------+-------------+ 98.541975975 98.541975975 98.541975975 98.541975975 98.541975975 1.0 1 +--------------+--------------+--------------+--------------+---------------+---------------+-------------+ 9/12/2014 84

Configuring Docker LXC Container For 2CPU x 4G RAM Pin container to 2 CPUs / Mems Create cpuset cgroup Pin group to cpuset.mems to 0,1 Pin group to cpuset.cpus to 0,1 Add container root proc to tasks Limit container memory to 4G Create memory cgroup Set memory.limit_in_bytes to 4G Add container root proc to tasks 9/12/2014 85

Docker Command Line Start docker image from docker CLI root@devstack-compute:~/devstack$ time docker run -d guillermo/mysql 859f2a88adeb7190eb855eb060172fc2f137bff609f2f3be8ad8ee069d7e88f4 real user sys 0m0.191s 0m0.004s 0m0.004s 9/12/2014 86