Are We Done Yet? Testing Your OpenStack Deployment

Transcription

1 Accelerating the adoption of Cloud Computing Are We Done Yet? Testing Your Deployment Summit, Paris, France November 4, 2014

2 Who Am I? Ken Pepple is the Chief Technology Officer of Solinea Prior to founding Solinea, he led the introduction of Internap's -based public cloud services while serving as their Director of Cloud Development Author of O'Reilly "Deploying book code contributor since Bexar release 2

3 Who we are Solinea is the leading software and services company that delivers integrated solutions, enabling enterprise adoption of Cloud Computing! Purpose-built for cloud Cloud is the only domain we focus on, with vertical industry and horizontal solutions specialization Experience Built the first production clouds and contributors to the platform since its inception Proven Delivery Success Track record of success architecting, building and operating production clouds private and public world-wide Differentiating IP Pre-configured management and monitoring platform for and other Open Infrastructure components Enterprise IT Experience We understand cloud adoption challenges of global enterprise companies 3

4 Agenda! Problem statement! Types of testing! Testing tools! Future improvements 4

5 Problem! Many components, even more connections! No two clouds are identical Hypervisor choices, network choices, storage choices! Many rarely used code paths (backups?)! Operational and integration concerns! Tools for operators! Requires skills across all the services 5

6 In a nutshell distributed systems Command Line Tools (nova-client, swift-client, etc.) Cloud Management Tools (CloudForms, Enstratius, etc.) GUI tools (Web Browsers, Cyberduck, iphone client, etc.) Internet / Enterprise Network HTTP(S) Object Object Identity (5000) Image (9292) Image Identity HTTP(S) Horizon Dashboard Compute / Admin Compute (8774, 8775, 8776) Amazon Web Services EC2 Block Storage VNC/Spice/RDP Network Block Storage Network Block Storage Compute Block Storage Network (9696) Compute Network Bock Storage Orchestration Data Processing Orchestration Database (8779) Data Processing (8386) account swift-proxy Object memcached container object glance-api Image glance database Image nova-compute libvirt, Xen, etc. nova-api (OS, EC2, Metadata, Admin) nova-console nova-*proxy nova-cert/ objectstore cinder-api cinder-volume cinder-backup neutron-server Queue heat-engine heat-api Queue trove-api Queue trove database sahara-api Queue sahara-conductor account DB container object DB store Object Store glance-registry Identity Image Service Identity hypervisor nova-consoleauth nova database Queue nova-scheduler nova-conductor Compute volume provider Block Storage Queue cinder-scheduler cinder database neutron agent(s) network provider neutron database neutron plugin(s) Network Service heat database Orchestration cloudwatch-api trove-taskmgr trove-conductor trove-guestagent Database (mysql, redis, mongo, cassandra, etc.) Database sahara-engine Data Processing sahara database Identity Identity keystone (service & admin s) ceilometer-api ceilometer-collector Identity Identity Service token backend catalog backend policy backend identity backend Metering Service ceilometer database Identity Identity Identity Identity Object 6

7 What Should We Be Testing?! Functionality User scenarios Not code quality unless running trunk! Non-functional requirements Scalability Performance Security! Operations High availability and failover 7

8 How Should We Test?! Automated Repeatability Multiple checks! Continual! Ops not specialists Should need coders or QA people Testing should be an easy command! Integrated into operational procedures Let me send a canary in 8

9 Types of Testing! Unit testing! Integration! User Acceptance Tests! Operational 9

10 Functional Areas! Networking! Storage! Compute! User functional process! OUTSIDE applications Billing/metering, user signup, audit/compliance 10

11 Testing Options! Tempest! Rally! Manual 11

12 Tempest! testing project! Runs hundreds of tests Unit Scenario Stress CLI! Complex to configure! Not portable to every cloud deployment configuration! Requires significant investment in time 12

13 Rally An project to create a framework for validating, performance testing and benchmarking at scale with Tempest wiki/rally 13

14 Rally Scenario $ rally task start summit/create-and-list-users.json ================================================================================ Task 57bc6d22-76c b834-daa48c41d6e0 is started ================================================================================ Task 57bc6d22-76c b834-daa48c41d6e0 is finished test scenario KeystoneBasic.create_and_list_users args position 0 args values: {u'args': {u'name_length': 10}, u'runner': {u'concurrency': 10, u'times': 100, u'type': u'constant'}} action min (sec) avg (sec) max (sec) 90 percentile 95 percentile success count keystone.create_user % 100 keystone.list_users % 100 total % Whole scenario time without context preparation:

15 Rally Database! Rally stores all scenario/task runs! Can be used to trend and benchmark results $ rally task list uuid created_at status failed tag bc6d22-76c b834-daa48c41d6e :10: finished False 12d e c0e :32: finished False 3b ed3b-4e2b-b09c-14b6d900e18e :40: running False

16 Rally Scenarios { } "KeystoneBasic.create_delete_user": [ { "args": { "name_length": 10 }, "runner": { "type": "constant", "times": 100, "concurrency": 10 } } ] Method Customization Concurrency 16

17 Rally Scenarios! Authenticate! Ceilometer! Cinder! Designate! Glance! Heat! Keystone! Neutron! Nova! quotas! Requests! Sahara! Tempest! VM! Zaqar 17

18 Rally Visualization 18

19 RALLY DEMO 19

20 Operational HA Tests! Kill process / hung process! Lost interface! Lost machine! Lost rack 20

21 Continual Testing! Testing should be part of monitoring and management! Order Canary Chao monkey may not be appropriate but happy path test should be Define rally scenarios or custom scripts to feed monitoring tools! Subset of tests that affect business critical functions 21

22 Putting it all together Rally [os-admin] OS commands Cloud Monitoring [goldstone] 22

23 Best Practices! ABT (Always be testing)! Manual testing " operational procedure Failover regularly as part of normal operations HA is an event, it is SOP! Testing is part of the deployment, monitoring and upgrade process 23

24 Goldstone Log Monitoring

25 Future! /info s for all services Describe deployment beyond services! Simplified DSL to configure tempest! Monitoring product incorporate testing 25

26 Thank You Ken Pepple, CTO