SESSION 605 Thursday, March 26, 2:45 PM - 3:45 PM Track: Metrics and Measurements

Transcription

1 SESSION 605 Thursday, March 26, 2:45 PM - 3:45 PM Track: Metrics and Measurements Managing Cloud Service KPIs Hank Marquis Practice Director, Cloud, Global Knowledge hank.marquis@globalknowledge.com Session Description Measuring cloud service quality requires a very different approach from legacy IT measurements. Cloud quality is truly in the eye of the customer, but capturing their perception of that quality can be challenging. In this session, attendees will learn how to measure cloud service quality from the customer's point of view and then use those measurements to improve service proactively. Attendees will take home reusable templates for measuring services and establishing appropriate cloud-based KPIs, and then negotiating with or choosing providers based on those KPIs. (Advanced) Speaker Background Hank Marquis was working with cloud services before they were called the cloud. For nearly twenty years, Hank has worked to improve service delivery for a wide variety of service providers and their customers. He specializes in the changes required of IT to seamlessly move to the cloud.

2 Session 605: Managing Cloud Service Hank Marquis, Practice Leader, Cloud Computing, Global Knowledge ITIL Service Strategy Reviewer, ITIL Service Catalog author and examiner APMG Examiner, EXIN Council Member, itsmfusa committee member 29 Years IT Experience, Technician to CIO PhD Organizational Leadership-ICT Charted Fellow British Computer Society (FBCS, CITP) ITIL, ISO-20000, COBIT, Six Sigma, Project Management Certifications

3 Metrics Can Help You Make A Decision About Which Cloud Provider to Choose. But That s Another Presentation Cloud Consumer A? B? Metrics Provider A Provider B This Presentation is About Metrics To Manage Cloud Consumer-Provider Relationships, From the Point of View of the Consumer. Cloud Consumer Metrics Provider B

4 We re Going to Cover the Four Strategies to Managing Cloud KPIs Know Why to Measure Understand How to Measure Identify What to Measure Learn to Create Metrics Know what you want from your metrics WHY MEASURE

5 You Don t Want to Fly A Plane, You Want to Arrive Safely via Plane. Goal Arrive safely at destination by plane CSF Stay above the mountains KPI Altitude Measurement Elevation in feet Instrument Altimeter You Don t Want to Run Into A Mountain, You Want to Stay Above The Mountain. Goal Arrive safely at destination by plane CSF Stay above the mountains KPI Altitude Measurement Elevation in feet Instrument Altimeter

6 You Must Maintain Enough Altitude to Avoid The Mountains. Goal Arrive safely at destination by plane CSF Stay above the mountains KPI Altitude Measurement Elevation in feet Instrument Altimeter Be Specific About Exactly What You re Measuring and Why. Feet not Centimeters. Goal Arrive safely at destination by plane CSF Stay above the mountains KPI Altitude Measurement Elevation in feet Instrument Altimeter

7 You Will Rely Upon Your Altimeter to Gauge Altitude to Avoid the Mountains and Arrive Safely. Goal Arrive safely at destination by plane CSF Stay above the mountains KPI Altitude Measurement Elevation in feet Instrument Altimeter Choosing the right metrics for the right reasons HOW TO MEASURE

8 Do You Want To Validate A Previous Decision? Validate Direct Justify Intervene Direct Activities In Order To Meet Targets? Validate a Previous Decision Direct Justify Intervene

9 Intervene To Identify Changes And Corrective Actions? Validate previous decisions? Direct activities in order to meet targets? Justify Intervene Justify a Required Course Of Action? Validate previous decisions? Direct activities in order to meet targets? Justify Intervene to identify changes and corrective actions?

10 Choose One. Validate previous decisions? Direct activities in order to meet targets? Justify a Required Course Of Action? Intervene to identify changes and corrective actions? Metrics Are Only Useful in Pursuit of a Goal. If a Metric Doesn t Tell You About A Goal, Get Rid of IT. Goal The end toward which effort is directed. Critical Success Factor (CSF) Something that must happen to meet the goal. Key Performance Indicator (KPI) A metric that predicts if the CSF will occur.

11 You Need Multiple CSF for Each Goal. One CSF Usually Means You re Missing Something Important. Goal The end toward which effort is directed. Critical Success Factor (CSF) Something that must happen to meet the goal. Key Performance Indicator (KPI) A metric that predicts if the CSF will occur. You Normally Need 2-3 KPI for Each CSF in order To Really Predicts CSF Success or Failure. Goal The end toward which effort is directed. Critical Success Factor (CSF) Something that must happen to meet the goal. Key Performance Indicator (KPI) A metric that predicts if the CSF will occur.

12 Qualitative Measurements Are Subjective/Feelings Qualitative CSF: Maintain Customer Satisfaction with Salesforce.com KPI: Customer satisfaction MEASUREMENT: Variance no more than +/-10% Quantitative CSF: Control Amazon Web Services costs KPI: Cost per hour MEASUREMENT: consumption x cost Quantitative Measurements Are Objective/Numbers Qualitative CSF: Maintain Customer Satisfaction with Salesforce.com KPI: Customer satisfaction MEASUREMENT: Variance no more than +/-10% Quantitative CSF: Control Amazon Web Services costs KPI: Cost per hour MEASUREMENT: consumption x cost

13 Well Formed Metrics Make a Logical Top-Down Structure Goal Subjective Objective CSF C1 C2 C3 C4 Metric M1 M2 M3 M4 M5 M6 Effective Metrics Predict Goals Success Bottoms-Up Goal Subjective Objective CSF C1 C2 C3 C4 Metric M1 M2 M3 M4 M5 M6

14 Understanding the Cloud Actors and How They Determine What to Measure WHAT TO MEASURE All Cloud Metrics Will Come from One or More of These 5 Actors Cloud Consumer Cloud Provider Cloud Broker Cloud Auditor Cloud Carrier

15 Actor That Maintains Relationship With And Uses Service Cloud Consumer 5 E s Cloud Provider Cloud Broker Cloud Auditor Cloud Carrier Actor That Conducts Independent Assessments Of Cloud Services Cloud Consumer 5 E s Cloud Provider Cloud Broker Cloud Auditor Security Privacy Performance Cloud Carrier

16 Actor Responsible For Delivering Services To Cloud Consumers Cloud Consumer 5 E s Cloud Auditor Security Privacy Performance Cloud Provider SaaS, PaaS, IaaS Cloud Service Management Business Support Provisioning/Configuration Portability/Interoperability Security Privacy Cloud Broker Cloud Carrier Actor That Manages Use And Negotiates Relationships Between Cloud Providers And Cloud Consumers. Cloud Consumer 5 E s Cloud Auditor Security Privacy Performance Cloud Provider SaaS, PaaS, IaaS Cloud Service Management Business Support Provisioning/Configuration Portability/Interoperability Security Privacy Cloud Broker Service Intermediation Service Aggregation Service Arbitrage Cloud Carrier

17 Intermediate Actor Responsible For Connectivity And Transport Of Cloud Services. Cloud Consumer 5 E s Cloud Auditor Security Privacy Performance Cloud Provider SaaS, PaaS, IaaS Cloud Service Management Business Support Provisioning/Configuration Portability/Interoperability Security Privacy Cloud Broker Service Intermediation Service Aggregation Service Arbitrage Cloud Carrier Connectivity, Transport Your Cloud Metrics Will Reside and Reflect in One or More of these 3 Documents Master Service Agreement Service Agreement Service Level Agreement

18 MSA is a Top Level Legal Agreement Between Provider And Customer Covering General Aspects Master Service Agreement Service Agreement Service Level Agreement Legal Terms and Conditions Covered by MSA You Might Want To Measure Source: NIST Draft SLA Cloud Taxonomy

19 Lower Level Agreement Covering One Specific Service Master Service Agreement Top level legal agreement between provider and customer covering general aspects Service Agreement Service Level Agreement SLA is a Lower Level Agreement Covering The Performance Aspects Of A Service Master Service Agreement Top level legal agreement between provider and customer covering general aspects Lower level agreement covering one specific service Service Agreement Service Level Agreement

20 Service Agreement Business Metrics Source: NIST Draft SLA Cloud Taxonomy Technical Metrics for Your SLAs Source: NIST Draft SLA Cloud Taxonomy

21 Service Provider Performance Obviously Belongs in An SLA Obvious Availability Capacity Quality of support services Security performance Customer Satisfaction Maybe Not So Obvious Costs Consumption De-allocation of resources Portability Exit Strategy Cloud Service-Specific Items for Your SLA Aren t Always Obvious Obvious Availability Capacity Quality of support services Security performance Customer Satisfaction Maybe Not So Obvious Costs Consumption De-allocation of resources Portability Exit Strategy

22 Cloud Computing Key Performance Indicators You Should Measure Availability (e.g % during work days, 99.9% for nights/weekends) Performance (e.g. maximum response times) Security / privacy of the data (e.g. encrypting stored/transmitted data) Disaster Recovery expectations (e.g. recovery commitment) Location of the data (e.g. consistent with local legislation) Access to the data (e.g. data retrievable from provider in readable format) Portability of the data (e.g. ability to move data to a different provider) Process to identify problems and resolution expectations (e.g. call center) Change Management process (e.g. changes updates or new services) Dispute mediation process (e.g. escalation process, consequences) Consumption/costs (e.g., consumption x costs or cost escalations) Exit Strategy with expectations on the provider to ensure smooth transition Choosing what to measure. HOW TO CREATE METRICS

23 Definition Define Your Metrics Using a Top- Down Approach Goal Question Q1 Q2 Q3 Q4 Metric M1 M2 M3 M4 M5 M6 Goal Determines Question, Metric and Data to Collect Goal Attainment Question Answer Definition Metric Measurement Interpretation Data to Collect Collected Data Data Collection

24 Data Measurements Answer Question and Predict Goal Attainment Goal Attainment Question Answer Definition Metric Measurement Interpretation Data to Collect Collected Data Data Collection Start With a Business Goal Business Goal Ensure AWS CPU costs are identified and communicated to the application management team with enough time to prevent overruns of more than 25% per month. Cloud Management Goal?

25 Goal Begins with the OBJECT of the Measurement (Product, Process, Resource, etc.) Analyze Some OBJECT... CPU CONSUMPTION For The PURPOSE Of: Regarding the ISSUE of: From The POINT OF VIEW Of The... In The Following ENVIRONMENT: The Goals PURPOSE is a Verb Indicating an Outcome to Avoid or Achieve Analyze Some OBJECT... CPU CONSUMPTION For The PURPOSE Of: Identifying Regarding the ISSUE of: From The POINT OF VIEW Of The... In The Following ENVIRONMENT:

26 The ISSUE is What You Want to Monitor Analyze Some OBJECT... CPU CONSUMPTION For The PURPOSE Of: Identifying Regarding the ISSUE of: Costs of service From The POINT OF VIEW Of The... In The Following ENVIRONMENT: The VIEWPOINT is Who Will Be Taking Action on the Metric Analyze Some OBJECT... CPU CONSUMPTION For The PURPOSE Of: Identifying Regarding the ISSUE of: Costs of service From The POINT OF VIEW Of The... Application Management Team In The Following ENVIRONMENT:

27 ENVIRONMENT is The Context for the Measurement Analyze Some OBJECT... CPU CONSUMPTION For The PURPOSE Of: Identifying Regarding the ISSUE of: Costs of service From The POINT OF VIEW Of The... In The Following ENVIRONMENT: Application Management Team Amazon EC2 Turning Business Goals Into Measurable Goals Business Goal Ensure AWS CPU costs are identified and communicated to the application management team with enough time to prevent overruns of more than 25% per month. Cloud Management Goal Analyze CPU consumption to identify the costs of service of AWS EC2 from the viewpoint of the application management team.

28 Questions Come from Goals Goal Object, Purpose, Quality Focus, Environment, Viewpoint Question(s) That characterize performance Analyze CPU consumption to identify the costs of service of AWS EC2 from the viewpoint of the application management team. Where can we see CPU consumption costs today? Metric(s) That Answer Questions 53 Metrics Answer Questions Goal Object, Purpose, Quality Focus, Environment, Viewpoint Question(s) That characterize performance Metric(s) Data That Answers Questions Analyze CPU consumption to identify the costs of service of AWS EC2 from the viewpoint of the application management team. Where can we see CPU utilization today? AWS CPU utilization on AWS dashboard 54

29 Critical Success Factors From Questions Goal Object, Purpose, Quality Focus, Environment, Viewpoint Question(s) That characterize performance Metric(s) That Answer Questions Analyze CPU consumption to identify the costs of service of AWS EC2 from the viewpoint of the application management team. Where can we see CPU consumption costs today? AWS CPU utilization on AWS dashboard CSF: Notification within 30 minutes of increase in CPU costs of more than 20%. 55 Collected Data Feeds KPI s Goal Object, Purpose, Quality Focus, Environment, Viewpoint Question(s) That characterize performance Metric(s) That Answer Questions Analyze CPU consumption to identify the costs of service of AWS EC2 from the viewpoint of the application management team. Where can we see CPU consumption costs today? AWS CPU utilization on AWS dashboard CSF: Notification within 30 minutes of increase in CPU costs of more than 20%. KPI: Monitor AWS CPU utilization dashboard KPI: Verify alerts of 20% or more 56

30 Create Metrics Using the Goal Question Metric (GQM) Approach Develop a Detailed Goal with an Underlying Rationale Derive CSF/KPI/data Top-down, Interpret Bottom-up Include Actors Whose Viewpoint the Metric Measures Recommendations on using your new skills. SUMMARY

31 Use a Defined Structure to Create and Manage Cloud KPIs Follow Goal - CSF - KPI Cascade (up and down) Work with Business and IT to choose metrics wisely Use GQM and Include Qualitative and Quantitative CSFs and Metrics in Each Goal Lessons Learned With Respect To Cloud SLAs Read your cloud provider s SLA s very carefully An almost four-day Amazon outage did not breach Amazon s EC2 SLA, guarantees 99.95% availability of the service within a Region over a trailing 365 day period. Since it was a service (EBS/RDS) EC2 used (but not EC2 itself) that failed, the SLA was not violated legally. Get technical/dr staff involved to validate SLAs against common outage scenarios A simple power outage in Dublin affected Microsoft service to European customers. Have contingency plans in place

32 Thank you for attending this session. Don t forget to complete an evaluation