Accelerating the adoption of Cloud Computing Are We Done Yet? Testing Your Deployment Summit, Paris, France November 4, 2014
Who Am I? Ken Pepple is the Chief Technology Officer of Solinea Prior to founding Solinea, he led the introduction of Internap's -based public cloud services while serving as their Director of Cloud Development Author of O'Reilly "Deploying book code contributor since Bexar release 2
Who we are Solinea is the leading software and services company that delivers integrated solutions, enabling enterprise adoption of Cloud Computing! Purpose-built for cloud Cloud is the only domain we focus on, with vertical industry and horizontal solutions specialization Experience Built the first production clouds and contributors to the platform since its inception Proven Delivery Success Track record of success architecting, building and operating production clouds private and public world-wide Differentiating IP Pre-configured management and monitoring platform for and other Open Infrastructure components Enterprise IT Experience We understand cloud adoption challenges of global enterprise companies 3
Agenda! Problem statement! Types of testing! Testing tools! Future improvements 4
Problem! Many components, even more connections! No two clouds are identical Hypervisor choices, network choices, storage choices! Many rarely used code paths (backups?)! Operational and integration concerns! Tools for operators! Requires skills across all the services 5
In a nutshell distributed systems Command Line Tools (nova-client, swift-client, etc.) Cloud Management Tools (CloudForms, Enstratius, etc.) GUI tools (Web Browsers, Cyberduck, iphone client, etc.) Internet / Enterprise Network HTTP(S) Object Object Identity (5000) Image (9292) Image Identity HTTP(S) Horizon Dashboard Compute / Admin Compute (8774, 8775, 8776) Amazon Web Services EC2 Block Storage VNC/Spice/RDP Network Block Storage Network Block Storage Compute Block Storage Network (9696) Compute Network Bock Storage Orchestration Data Processing Orchestration Database (8779) Data Processing (8386) account swift-proxy Object memcached container object glance-api Image glance database Image nova-compute libvirt, Xen, etc. nova-api (OS, EC2, Metadata, Admin) nova-console nova-*proxy nova-cert/ objectstore cinder-api cinder-volume cinder-backup neutron-server Queue heat-engine heat-api Queue trove-api Queue trove database sahara-api Queue sahara-conductor account DB container object DB store Object Store http://www.solinea.com glance-registry Identity Image Service Identity hypervisor nova-consoleauth nova database Queue nova-scheduler nova-conductor Compute volume provider Block Storage Queue cinder-scheduler cinder database neutron agent(s) network provider neutron database neutron plugin(s) Network Service heat database Orchestration cloudwatch-api trove-taskmgr trove-conductor trove-guestagent Database (mysql, redis, mongo, cassandra, etc.) Database sahara-engine Data Processing sahara database Identity Identity keystone (service & admin s) ceilometer-api ceilometer-collector Identity Identity Service token backend catalog backend policy backend identity backend Metering Service ceilometer database Identity Identity Identity Identity Object 6
What Should We Be Testing?! Functionality User scenarios Not code quality unless running trunk! Non-functional requirements Scalability Performance Security! Operations High availability and failover 7
How Should We Test?! Automated Repeatability Multiple checks! Continual! Ops not specialists Should need coders or QA people Testing should be an easy command! Integrated into operational procedures Let me send a canary in 8
Types of Testing! Unit testing! Integration! User Acceptance Tests! Operational 9
Functional Areas! Networking! Storage! Compute! User functional process! OUTSIDE applications Billing/metering, user signup, audit/compliance 10
Testing Options! Tempest! Rally! Manual 11
Tempest! testing project! Runs hundreds of tests Unit Scenario Stress CLI! Complex to configure! Not portable to every cloud deployment configuration! Requires significant investment in time 12
Rally An project to create a framework for validating, performance testing and benchmarking at scale with Tempest https://wiki.openstack.org/ wiki/rally 13
Rally Scenario $ rally task start summit/create-and-list-users.json ================================================================================ Task 57bc6d22-76c2-4450-b834-daa48c41d6e0 is started -------------------------------------------------------------------------------- ================================================================================ Task 57bc6d22-76c2-4450-b834-daa48c41d6e0 is finished. -------------------------------------------------------------------------------- test scenario KeystoneBasic.create_and_list_users args position 0 args values: {u'args': {u'name_length': 10}, u'runner': {u'concurrency': 10, u'times': 100, u'type': u'constant'}} +----------------------+-----------+-----------+-----------+---------------+---------------+---------+-------+ action min (sec) avg (sec) max (sec) 90 percentile 95 percentile success count +----------------------+-----------+-----------+-----------+---------------+---------------+---------+-------+ keystone.create_user 1.767 3.478 5.944 4.565 4.694 100.0% 100 keystone.list_users 0.077 0.464 1.516 1.01 1.143 100.0% 100 total 1.982 3.942 6.218 4.956 5.214 100.0% 100 +----------------------+-----------+-----------+-----------+---------------+---------------+---------+-------+ Whole scenario time without context preparation: 40.6584420204 14
Rally Database! Rally stores all scenario/task runs! Can be used to trend and benchmark results $ rally task list +--------------------------------------+----------------------------+----------+--------+-----+ uuid created_at status failed tag +--------------------------------------+----------------------------+----------+--------+-----+ 57bc6d22-76c2-4450-b834-daa48c41d6e0 2014-11-05 12:10:38.235211 finished False 12d34393-0520-4830-9516-e7404935c0e9 2014-11-05 12:32:39.632810 finished False 3b160652-ed3b-4e2b-b09c-14b6d900e18e 2014-11-05 12:40:21.544483 running False +--------------------------------------+----------------------------+----------+--------+-----+ 15
Rally Scenarios { } "KeystoneBasic.create_delete_user": [ { "args": { "name_length": 10 }, "runner": { "type": "constant", "times": 100, "concurrency": 10 } } ] Method Customization Concurrency 16
Rally Scenarios! Authenticate! Ceilometer! Cinder! Designate! Glance! Heat! Keystone! Neutron! Nova! quotas! Requests! Sahara! Tempest! VM! Zaqar 17
Rally Visualization 18
RALLY DEMO 19
Operational HA Tests! Kill process / hung process! Lost interface! Lost machine! Lost rack 20
Continual Testing! Testing should be part of monitoring and management! Order Canary Chao monkey may not be appropriate but happy path test should be Define rally scenarios or custom scripts to feed monitoring tools! Subset of tests that affect business critical functions 21
Putting it all together Rally [os-admin] OS commands Cloud Monitoring [goldstone] 22
Best Practices! ABT (Always be testing)! Manual testing " operational procedure Failover regularly as part of normal operations HA is an event, it is SOP! Testing is part of the deployment, monitoring and upgrade process 23
Goldstone Log Monitoring
Future! /info s for all services Describe deployment beyond services! Simplified DSL to configure tempest! Monitoring product incorporate testing 25
Thank You Ken Pepple, CTO ken@solinea.com http://www.solinea.com