Just Enough Monitoring for the Busy IT Pro

Transcription

1 Just Enough Monitoring for the Busy IT Pro

2 Ezra Pagel Technical Product Manager Professionally focused on IT tools Virtualization, networking, financial modelling Chef: Spiceworks: Remote Agent Inventory Remote Sites Networking Monitoring core Ezra (Spiceworks) in the community

3 About You Not currently monitoring Homegrown or minimal monitoring Frustrated getting started Been burned Proactive!

4 Agenda Fundamentals of monitoring Architecture Spiceworks Network Monitor Tools landscape Q&A #SPICEWORLD2016

5 Disclaimer Opinions are my own Spiceworks is vendor-neutral IT pros and the Spiceworks Community members are not J Ignoring a lot of cloud monitoring tools, for now

6 Framework In which we attempt to reason about system monitoring

7 Fundamentals Reactive Outage minimization Watch for alerts, deal w/ or escalate Proactive Outage prevention Resource optimization Preemptively looking for problem trends

8 Maslow s Hierarchy of Needs Actualization: Creativity, fulfillment Esteem: Achievement, recognition Emotional: Family, friends, society Safety: Security, stability, plentiful resources Biological: Air, food, water, shelter

9 Monitoring Hierarchy of Needs Cause analysis, prediction Application perfomance Excellence Success Adaptive routing, balancing Network performance Application availability Purpose Route availability Disk space, CPU/Ram utilization, I/O load Health Port state, I/O load, dropped packets Power, OS state, connectivity Survival Power, OS state, management interface connectivity

10 Prioritize by working up the pyramid Technology continues to be more usable and more stable LoB exceptions Custom software Legacy apps Still not immune to sizing problems Never immune to HW outages Cable shorts, power failures, drive outages So we ll start w/ the hardware, power

11 Level 1 - What *things* should we monitor? Power, UPS, WAN, hosting provider Big 4 CPU Memory Disk Network Chassis Fan, temperature? External fundamentals Domain names SSL Certs

12 What *properties* of them should we monitor? Availability Up/down Responsive Utilization How much of the capacity is being used Saturation How overloaded is the system

13 Utilization vs Saturation Utilization Current CPU load % Memory in use Network bandwidth Disk space Saturation CPU queue length Network dropped packets Memory paging Disk queue length

14 Utilization vs Saturation

15 Utilization vs Saturation

16 Utilization/Saturation Try to find separate the utilization and saturation metrics Response time

17 Level 2 - What *things* should we monitor? Mail server, web server, databases, exchange, LoB apps, quickbooks, build servers, VPN appliances, Point-of-sale, etc Processes Services APIs Legacy Logfiles Output files Event viewer

18 APM- Properties to monitor? Availability Process/services running and reachable Responsive no timeouts HTTP request POST via powershell Throughput Requests/queries per second Service time!= response time Fall behind quickly, can grow linearly Task manager view Any instrumentation

19 EUX Synthetic monitoring Simulate user interactions Logging in, opening a ticket, purchasing, etc Useful for diffing changes Real-user monitoring Instrumentation in client or tracing of actual user sessions Looks at real user traffic Just-enough Check logs for session times Scheduled powershell job for common interaction Traceroute from machines SxS users

20 PITFALLS Sampling bias Averaging averages/percentiles Masking the most interesting data - What s the average of [0, 1, 2, 4, 2, 98, 23,11, 2, 3, 4,1, 2, 4,1]? Burst can cause saturation Still useful

21 Recap Be realistic. Select actionable, not aspirational coverage Start with availability monitoring, server and app Selectively enable performance/utilization notifications Disk space Selectively enable utilization Ignore spikes, you want sustained or quota-based checkpoints

22 Architecture Hypothetical/Typical

23 UI/dashboard Web-hosted or desktop client Configurability varies Suffer from overwhelming Demos great at trade shows though! #SPICEWORLD2016

24 U WOT M8

25 Data Storage Configuration data Device info Account credentials Alerts Schedules Historical data Time-series database RRD rules open source NoSQL Baseline performance Reporting SLA measurement

26 Real-time vs Historical The two primary ways to monitor systems are to (1) gather historical data related to availability and usage and (2) perform real-time monitoring to ensure that SAs are notified of failures. (Real-time monitoring) results in shorter outages and problems being fixed before customers notice, along with building the team s reputation for maintaining high-quality service. - T. Limoncelli, The Practice of System and Network Administration

27 Metric Collection Push, agent/service Pros: Access to deeper utilities beyond standard mgmt. APIs Can fire events closer to real-time Hook into OS eventing Cons Heavier on OS resources Maintenance PITA (updates, security) Not all systems supported

28 Metric Collection Pull - collector Pros: Centralized polling service Relies on standardized APIs Can poll systems that don t support agent (SNMP) Cons Suffers from sampling interval accuracy Might miss outliers (not sustained) Credential management can be more difficult Can stress already taxed system

29 Event processing Summarization Perform running calculations Maintain averages Compute deviations

30 Notification Inform users of events via channels Visual Tickets SMS Slack Twitter?

31 A word about the cloud Pros Deployment time No procurement/vm/laptop eval Still need on-prem agents/collectors Scalability Easier to scale up Initial cost

32 A word about the cloud Cons Connectivity SPF External data Security Performance pipe push Long term cost

33 Spiceworks Network Monitor A Whirlwind Tour

34 Monitoring Hierarchy of Needs Cause analysis, prediction Application perfomance Excellence Success Adaptive routing, balancing Network performance Application availability Purpose Route availability Disk space, CPU/Ram utilization, I/O load Health Port state, I/O load, dropped packets Power, OS state, connectivity Survival Power, OS state, management interface connectivity

35 Current Focus Cause analysis, prediction Application perfomance Excellence Success Adaptive routing, balancing Network performance Application availability Purpose Route availability Disk space, CPU/Ram utilization, I/O load Health Port state, I/O load, dropped packets Power, OS state, connectivity Survival Power, OS state, management interface connectivity

36 Summary 2016

37 Just Enough Choose actionable over aspirational Prioritize availability monitoring Customize alerting so they re relevant The best monitoring system is whichever one that you actually use.

38

39