Monitor Open stack environments from the bottom up and front to back Roger Ruttimann VP Engineering, GroundWork OpenSource November 17, 2015
What s ahead of us Overview: The Impact of Virtualization on IT operations How Open Stack fits into the Virtualization landscape Monitoring the changed landscape of IT infrastructure New Monitoring concepts Selection of the right tools to fit the right process Conclusion 2
Start with the facts: Open Stack is real! OpenStack deployments are not just happening in a far away land... Not secret potion or evil conspiracy Not just a geek movement Serious deployments for IT Operations not just in the Silicon Valley It s all OpenSource Has 5000 active members, growing daily Conferences draw close to 10,000 participants Many corporate sponsors It s disruptive and requires serious retooling for IT operations Let s review the challanges and what tools best fit the new realities Box Spy OSCM 2015 Monitor Open Stack environments from bottom up and front to back by Roger Ruttimann 3
Once upon a time SysAdmin task: Add To IT infrastructure { Deploy servers into datacenter Provisioning applications Define monitoring for each element Monitoring was done with SSH checks and port gets } repeat Port get ssh snmp A p p s A p p s Switch A p p s A p p s Switch A p p s Server Server Server Checking for up down and send email snmp Firewall Monitor snmp Router 4
Then came along compute virtualization Datacenter compute optimization Server resources were virtualized to improve efficiency which was < 30% Products like ESX allowed resource optimization not disturbing the provisioning process Rollout VM machines were provisioned Applications installed Monitoring defined as before New was an to the virtualization manager (example VSphere ) to poll data Monitor Port get ssh REST snmp snmp snmp A A A A A p p p p p p p p p p s s s s s Server Server virtualization Server Switch Firewall Router Switch 5
But it didn t stop there Software Defined Everything Compute (Hypervisors) Storage (SDS) Network (SDN) VCenter KVM RHEV-M Hybrid Cloud Public/Private Amazon WebServices Rackspace Azure Change created blind spots in coverage Suddenly ssh/snmp methods don t cover all Monitor Port get ssh REST snmp REST snmp REST snmp A A A A A p p p p p p p p p p s s s s s Server Server virtualization Server Switch Switch Virtualized Network Firewall Virtualized Storage Router Switches Switches Switches 6
Infrastructure landscape completely changed Hypervisor manager view Silo view Silo view Silo view Silo view Silo view Silo view Apps VMs Devices Files Shared I/O Fog Network & Storage 7
And the moment you think you ve seen it all DevOps is pushing the envelope even more Linux Containers Are the new kids in town Application Isolation Rapid deployment Elastic Scalability Port get ssh REST REST A A A A A p p p p p p p p p p s Docker s s Docker s Docker s Host Host Host Server Server Server virtualization Switch Switch Virtualized Network for deployment: Monitoring is somebody else s job REST Monitor Virtualized Storage Switches Switches Switches 8
Don t panic What happened over the last 5 years: Virtualize everything Private and Public clouds centric world not just for Applications REST is a standard CORBA, SOAP & proprietary s are now classified as dinosaurs Diverse Virtualization stacks ESX NSX vsan vsphere Nova Neutron Cinder Glance OpenStack Browser based UI / JavaScript is king Learn to speak REST & JSON EC2 VPC AWS Linux KVM Network ovirt S3 NFS 9
How does Open Stack fit in? Service oriented architecture Pluggable hypervisors, network, storage to support wide range of technologies Elastic Compute Units for better virtualization efficiency Dashboard--Horizon Standardized s to all services Façade Service Sounds But what s really good the reality doesn t? it? Object Store--Swift Image-- Glance Compute/ Hypervisor--Nova Storage--Cinder Network Quantum-Neutron Identity-- Keystone 10
Open Stack Profile Editor 11
Open Stack Profile Editor 12
is good but comprehensive coverage is better Performance data Unified View Virtual infrastructure Open Stack Container metrics Applications, Infrastructure checks Standardized Data Collection ssh/snmp/port get Provisioning/Configuration Manager Stack ManageIQ/ Cloud Forms Dashboard Ansible HP Helion VMWare VRealize Object Store Image Computer Storage Network Storage racks Mirantis Fuel Identity Servers Servers Servers Legacy network 13
Unified View Silo tools Impossible to correlate Naming mismatch Over-monitoring Encourages: It s not my problem check your system No Big Picture Dashboards Need for Stack monitoring bottom to top Aliasing of names Combine best of breed collectors Correlated metrics across all infrastructure Dashboard for each customer/client 14
Stack monitoring for Open Stack Function Source Host name Alias Network, Storage Netflow SNMP Hardware IPMI SNMP Drac-server1 os-eng-h1 Hypervisor Identity server Operating System check-mysql check-port check-proc check-mem check-load server1:port server1 os-eng-h1 os-eng-h1 OpenStack- Hypervisor CPU/Mem metrics, Network storage 10.10.0.1 os-eng-h1 OpenStack VM CPU/mem, Network, storage 10.10.10.123 eng-slicer-1 VM Operating system check-proc check-mem check-load eng-cent6-actg eng-slicer-1 DockerHost Memory, CPU eng-cent6-actg eng-slicer-1 APP Container Memory, CPU, Procs FAC3443DA77 Load-bal-01 Application Check_https 172.28.102.51 Load-bal-01 15
Automation / Continuous configuration discovery Adding Virtual machines, containers is automatic So is the monitoring V C C C V V V V V C Hypervisor Management Configuration collector Plugins Data collector D a t a Dashboards Monitor System Monitoring is like a flight recorder -- collecting all active data Continuous operations by.. synchronization of configuration re-configuration of plugins Collection of data (availability & performance) 16
OpenStack Monitoring: Tools selection Unified Monitoring View Data Management Collector Collector Collector Icinga2 Murano Application Software Catalog Fuel OPEN TSDB VM 1 OpenStack Linux OS Platform Hypervisor Hardware Cinder Box Spy Docker Node 2 Node 3 Box Spy Docker VM 1 VM 2 VM 3 VM 4 Agent IPMI Port Icinga2 OpenStack Storage Open Stack Icinga2 setup 1) Fuel, Open Stack deployment 2) Murano, Application deployment 3) Fuel Plugin, Configuration generate 4) Fuel Plugin, Configure monitoring Keystone OpenStack Controller 17
Performance Data Monitoring systems Grafana Dashboards VCenter Centralized Data Collector Monitoring send Perf Data to single Expandable storage cluster Expand on demand Dashboard to visualize and drill down Historic, raw performance data Group alike metrics to find outliers 18
Reference Architecture monitoring Aliasing and data normalization in backend Collect Integrate Visualize VCenter A P I Collector Collector Collector Data Collector Hub A P I Data Integration Normalization Aliasing A P I Growing number of s require integration Hub
Benefits Automation Reduced cost of maintenance Dynamic configuration Streamlined operations Personnel System Administrator & Dev Ops functions merge Capacity & Resource planning Complete Bottom (network) to Top (application) data collection Allocate resources at the right level Hardware No vendor lock-in Transparency through 20
Lessons learned Virtualization and containerization require new monitoring techniques Legacy hardware/software will be around for a while so don t throw tools away Everything -- needs integration Aliasing is hard but maybe easier than cross silo cooperation Use the best tool for the task Don t over-monitor to cover a gap 21
Conclusion Open Source tools are the driving force for innovation DevOps just selects the best tool to do the job A single monitoring tool to rule them all doesn t exist Integration is a complex task Don t expect teams will agree on naming, process, and workflow Automation and pragmatism will prevail as DevOps has demonstrated Virtualize everything will continue Automation and continuous discovery is necessary for rapid scale out Open Stack is the Open Source Virtualization platform but Monitoring coverage needs major improvements 22
Thank you Roger Ruttimann rruttimann@gwos.com 23