Rating Scales A Fundamental Analytic Method

Transcription

1 Rakesh Kumar, Manager, Competitive Intelligence Kevin Mann, Manager, Competitive Intelligence February 2012 A Fundamental Analytic Method

2 This presentation provides an introduction to using for market insights analysts with limited experience using this approach Topics Why rating scales work When to use a rating scales How to create a rating scale Which type of scale to use 2 Feb 2012

3 Have you ever been in a meeting when people had different points of view and had a hard time reaching agreement? No no no, you don t understand, our top priority should be city zoos! This guy belongs in a zoo! 3 Feb 2012

4 No need to suffer endless debates... rating scales simplify complex topics, bring objectivity to potentially emotional topics, are efficient to use, and lead to action Advantages of Defined Simplify complex topics Reduce multiple complex considerations into summary measures. Objective, less contentious Removes ambiguity making the rating factors explicit and easier to reach agreement on. Efficient actual rating steps take less time and validation is easier with subject matter experts and with stakeholders. Actionable action recommendations are designed in from the start. 4 Feb 2012

5 Many of us had our first experience of rating scales when we began to receive grades in school 5 Feb 2012

6 School grades combine scores from several different rating factors into a single score that can drive specific actions Grade Components Class participation 20% Homework 30% Tests and Quizzes 50% Expert Scoring of Class Achievement Action from Grades Advanced class placement Tutoring Admission to college 6 Feb 2012

7 Scoring models can also be created by crowd-sourcing, which has become typical for restaurants, and other entertainment Crowd-sourced Restaurant Scoring Rating Components Food Service Ambiance Price Action from Score Diner: Select restaurants to try out Owner: Improve items with low scores 7 Feb 2012

8 For movies, we have ratings for the appropriate age range based on several components and used for filtering Expert Movie Rating for Age/Maturity Rating Components Sexuality Language Violence Drug use Action from Rating Control who is permitted to attend a movie Select a movie I might prefer 8 Feb 2012

9 Bestseller and Top 40 lists are a special type of rating based primarily on categorization and counting to produced ranked lists NYT Bestsellers 2 of 23 categories Rating Elements Categorize each book Count sales for the week HARDCOVER FICTION 1. TAKEN, by Robert Crais 2. PRIVATE: #1 SUSPECT, by James Patterson and Maxine Paetro 3. DEATH COMES TO PEMBERLEY, by P. D. James 4. THE GIRL WHO KICKED THE HORNET S NEST, by Stieg Larsson 5. 11/22/63, by Stephen King HARDCOVER NONFICTION 1. AMERITOPIA, by Mark R. Levin 2. AMERICAN SNIPER, by Chris Kyle with Scott McEwen and Jim DeFelice 3. STEVE JOBS, by Walter Isaacson 4. QUIET, by Susan Cain 5. KILLING LINCOLN, by Bill O'Reilly and Martin Dugard Action from Rating Reader: What books should I evaluate for reading? Bookseller: What books should I carry in my store? 9 Feb 2012 Source: Feb

10 Detailed, granular ratings can help identify specific issues that need attention Gartner Assessment of Outsourcing Locations 2008 Action from Rating Select potential outsourcing locations Develop amelioration plans 10 Feb 2012 Source:

11 And, combining two ratings together can provide a strategic view as in the Gartner magic quadrant Rating Elements Completeness of vision Market understanding Marketing strategy Sales strategy Offering strategy Business model Industry strategy Innovation Geographic strategy Ability to execute Product/service Overall viability Sales Execution/Pricing Market responsiveness and track record Marketing execution Customer experience Operations 2010 Gartner Magic Quadrant for Application Performance Monitoring Action from Rating Customer: Select potential vendors Vendor: Focus on key competitors 11 Feb 2012

12 Or, as in this IDC example, two categorizations can be combined to provide a strategic view Manufacturing Value Chain Segmentation Focus High Engineering Technology Oriented Oriented Supply Complexity Asset Brand Low Oriented Oriented Source: IDC Manufacturing Insights, 2011 Low Demand Complexity High 12 Feb 2012

13 While there are many different types of rating scales, most projects will use one of these three core methods Three Types of Scoring (grades) Which is BEST? Use when best can be defined Extremely versatile Single or multi-dimensional Decimal accuracy to Harvey balls Counting (bestsellers) Which is MOST? Use when most is a good proxy for importance; e.g., what is the most frequent enhancement request? Grouping (movie rating) What GROUP is it in? Use when complex logic is needed Often used for filtering Often used for segmentation and sales targeting 13 Feb 2012

14 Let s look at one example of using a scoring system Three Types of Scoring (grades) Which is BEST? Use when best can be defined Extremely versatile Single or multi-dimensional Decimal accuracy to Harvey balls Counting (bestsellers) Which is MOST? Use when most is a good proxy for importance What s frequently requested? How many units shipped? Grouping (movie rating) What GROUP is it in? Use when complex logic is needed Often used for filtering Often used for segmentation and sales targeting 14 Feb 2012

15 To develop a scoring rating scale, define rating categories, create a scale for each category, rate using the scale, validate, and present Steps for Scoring-based Ratings 1. Define rating categories: Make categories actionable. 2. Create a defined scale for each category: Divide into sub-elements that can be rated. Define scales to create objectivity. 3. Rate using the scale and validate with SME s: Fact-based sources and SME validation create confidence. 4. Present and recommend action: Ratings should map to actions. 15 Feb 2012

16 For example, for a project designed to recommend improvement areas for CIO s, 10 capability areas were defined that CIO s could act on Rating Areas Defined in Actionable Focus Areas Capability Area (action areas) a) Business and IT Relationship b) Client Devices and User Support c) IT Appl. And Service Management d) IT Project Mgmt. & Execution e) IT Fin. Mgmt. & Procurement f) IT Organization and Skills g) Server and Storage Management h) Network i) Service Delivery and Support j) Site and Facility 16 Feb 2012

17 For each of the capability areas, we divide them into sub-elements that can be clearly defined and rated we leveraged ITIL models to create these Sub-elements Defined for the Business and IT Relationship Now +2yr Bus IT Capability Area (action areas) a) Business and IT Relationship b) Client Devices and User Support c) IT Appl. And Service Management d) IT Project Mgmt. & Execution e) IT Fin. Mgmt. & Procurement f) IT Organization and Skills g) Server and Storage Management h) Network i) Service Delivery and Support j) Site and Facility a) Business and IT Relationship IT Mission - as understood by the business IT Communication with Business Areas IT Investment Prioritization Metrics and Measurements Feb 2012

18 The defined rating scale provides clear definitions with an objective meaning, as well as implied improvement guidance Defined for the Sub-elements of Business and IT Relationship Now +2yr Bus IT a) Business and IT Relationship IT Mission - as understood by the business Manage technology -- technical focus. IT Communication with Business Areas Ad-hoc, based mainly on requests for support or complaints Provide applications to Help business areas -- support business areas -- leveraging technology for individual application process excellence. focus. Assigned support for Regular communications business areas with an focused on improving assessment of needs by impact of IT on business dept. results. IT Investment Prioritization Priority of projects based on opinions more than analysis, typically with an annual list of projects only for budgeting. Consistent framework for identifying, documenting, and prioritizing projects with sign-off by a business steering committee. Significant IT projects tied to business initiatives with business owners; or have measurable infrastructure goals. Metrics and Measurements Metrics limited to those created by individual products and applications, with limited or no reporting to the business. Metrics are summarized Monthly or quarterly into a monthly or review of IT-related quarterly reports, metrics specific to reveiwed by support of major management, that reflect business processes and technology-centric SLA's. departments; example: help desk calls by department with resolution. 18 Feb 2012

19 Business and IT pain point scales were also defined, but as a single scale used across all capabilities Business Pain Points Minimal Issues Minor Concerns Significant Issue Critical Issue Problems in this area are within the range of normal business operations. No significant concerns. Defined for Pain Points There are some problems; however, IT management has plans to address the issues, which have been communicated. Problems in this area have some business impacts and require the attention of non-it management, and improvement appears slow. Problems in this area are significantly affecting business operations, and significant changes are needed within the next 18 months. IT Pain Points Minimal Issues Minor Concerns Significant Issue Critical Issue Problems in this area are within the range of normal business operations. No significant concerns. There are some problems; however impacts are limited to IT internal operations and IT management has a plan to address the issues. Inefficiencies or problems in this area require an excessive amount of IT management or IT staff time, which affects IT productivity and effectiveness. Problems in this area are causing or may soon cause dissatisfaction with IT, or are absorbing far too much management attention. Issues need to be addressed within the next 18 months. 19 Feb 2012

20 Ratings were done collaboratively with customer teams in face-toface workshops, so the results were easily accepted Customer Workshop to Agree on Ratings 20 Feb 2012

21 Customers and IBM ers left the meetings with a shared agenda for areas needing improvement Current Capabilty Level and Gap to +2yr Target a) Business & IT Relationship b) Client devices and User Support c) IT Applications and Services Management d) IT Project Management and Execution e) IT Financial Management and Procurement f) IT Organization and Skills Cap. Level Now Cap. Level Gap 21 Feb 2012 g) Server and Storage Management h) Network i) Service Delivery and Support j) Site and Facility Source: IBM Market Insights, onsite workshop, September 2nd, 2009 Average of Current Capability Levels Note: pain scores were also incorporated into improvement prioritization

22 While there are many different types of rating scales, most projects will use one of these three core methods Three Types of Scoring (grades) Which is BEST? Use when best can be defined Extremely versatile Single or multi-dimensional Decimal accuracy to Harvey balls Counting (bestsellers) Which is MOST? Use when most is a good proxy for importance What s frequently requested? How many units shipped? Grouping (movie rating) What GROUP is it in? Use when complex logic is needed Often used for filtering Often used for segmentation and sales targeting 22 Feb 2012

23 For a counting-based rating scale, identify major categories, define sources, categorize findings, refine categories, present and recommend actions Steps for Counting-based Ratings 1. Identify major categories Make categories actionable. 2. Define sources and collection protocols The data pool should be meaningful. 3. Categorize, subcategorize, and recategorize until the categories seem meaningful and clear This is a creative process. 4. Present and recommend action: Ratings should map to actions. 23 Feb 2012

24 For example, for a project designed to identify areas to improve in IBM s cloud messaging, we identified seven message categories and four competitors Percent of Messages by Message Category by Competitor Message Categories Average Service Delivery Attribute Business benefits Audience Partnership association Workloads Industry Negative/Critical Messaging Totals 100% 100% 100% 100% 100% 100% Significant high level of messaging Significant low level of messaging 24 Feb 2012

25 To capture messages that a typical cloud buyer would see, we used a standard protocol for gathering information from six sources Collection Protocol by Messaging Source Messaging Sources Analyst Reports Whitepapers/Brochures Industry Websites Factiva Search (News) Company Websites Google Search Standard Collection Protocol Reports identified using simple search using competitor and cloud reference. Top 4 rated reports were read for each competitor and IBM. Total of 3 Whitepapers and Brochures per competitor and IBM. Generally found on websites or by conducting a web search. Three industry websites were targeted: CIO.com, eweek.com and Techtarget.com. 3 articles per competitor and IBM were read and messages parsed. Conducted a simple search using competitor name and cloud computing reference. 3 most recent news reports/press releases were evaluated. Collected messaging from home page and pages directly linked from the home page (1 click page depth). Simple search conducted using competitor name and cloud computing. Followed any organic links from first two pages that were not covered by an additional source already mentioned above. In total we captured messages from a 105 individual sources for the four competitors and IBM. Data was collected during two weeks: November 9 to 22, Feb 2012 Dec 2009

26 Analyzing the actual messaging around service delivery attributes, we developed sub-categories and counted message volume # of messages Service Delivery Attributes: No. of Messages by Top Five Sub-Categories Amazon Google IBM Microsoft Salesforce.com Service Delivery Attribute sub-category definitions Ease of use Offering that is easy to use and builds on existing skills, frameworks and languages Scalability Ability to scale applications up and down to manage demand shifts Security Mention of a secure environment or specific security plans/policies Speed of implementation Speed of provisioning and deployment Service Quality Service availability, reliability and dependability with features such as fault tolerance Source: Cloud Messaging Project Data, November Feb 2012 Dec 2009

27 In this case we also evaluated the quality of messaging to highlight areas for greater clarity Ease of Use Example Messages Competitors Amazon simple web service interface allows you to obtain and configure capacity with minimal friction Google End users can use the familiar Microsoft Outlook interface for , contacts and calendar as they transition to Gmail and Google Calendar Microsoft use their existing skills to develop cloud applications, familiar development experience IBM IBM could simplify the often arduous process CloudBurst, Ovum Report simple-to-use, self-service test platform Smart Business Development and Test, IBM Website create a dynamic IT infrastructure that is easier and less expensive to manage, upgrade and run Blue Cloud, IBM Website simple to invoke CloudBurst, IBM Whitepaper Salesforce.com Force.com automatically generates a rich user interface that you can customize with our drag-anddrop page layout editor Microsoft and Google provide more specific information about how ease of use is achieved than IBM. This could be an area for greater clarity in our messaging. Source: Cloud Messaging Project Data, November Feb 2012 Dec 2009

28 We leveraged the quantitative ratings and quality assessments to provide specific recommendations to improve IBM s messaging Five Focus Areas to Improve the Credibility of IBM s Cloud Positioning Security certifications - Competitors are providing details regarding existing certifications as well as efforts to obtain new certifications Customer references/case studies - Customer video references and specific case studies regarding service usage Third party sponsored whitepapers - ROI sponsored whitepapers lend analyst credibility to competitor marketing messages Partnership associations - New emerging competitors such as Salesforce.com are establishing partnerships with established industry brand names and giving credibility to their solutions Quantification of benefits - Detailed and specific information provided for messaging sub-categories Source: MI Analysis 28 Feb 2012 Dec 2009

29 While there are many different types of rating scales, most projects will use one of these three core methods Three Types of Scoring (grades) Which is BEST? Use when best can be defined Extremely versatile Single or multi-dimensional Decimal accuracy to Harvey balls Counting (bestsellers) Which is MOST? Use when most is a good proxy for importance What s frequently requested? How many units shipped? Grouping (movie rating) What GROUP is it in? Use when complex logic is needed Often used for filtering Often used for segmentation and sales targeting 29 Feb 2012

30 To develop a grouping or segmentation rating, define the groups and the logic of group membership, categorize based on data, and recommend actions based on group membership Steps for Grouping / Segmentation 1. Define groups Make groups actionable 2. Define logic / rules for membership in each group Can be complex and even statistically based, but must be explainable 3. Categorize into groups using the logic / rules and validate with SME s Ensure that SME s think your categorization makes sense 4. Present and recommend action: Groups should map to actions 30 Feb 2012

31 The paramount step was to prioritize the accounts in XXX and XXX to identify the customers that would be best candidates for ibm.com Develop Inputs Find Patterns and Relationships Identify Key Drivers XXX and XXX space (Both Destacked and Non-destacked accounts) Relative importance - an example Attributes Category Examples Revenue System x revenue in 08, 09, 10. Industry Employees Won revenue Opp. Owning Channels MAP Industry Group Computer services; Insurances; etc. Number of employees STG total won revenue (ssc=07, 08) in Business Partner; Faceto-Face; ibm.com Financial services sector; Distribution sector; etc. *Source: MAP and EIW 31 Feb 2012 Key Predictive Attributes of those that Look-Like best customers MAP Ind. Goup 95% Accounts receive a score which indicates the extent to which they are suitable to be destacked to ibm.com. IBM.CO M Won Rev. Share 5%

32 In 2011, the destacking practice in XXX and XXX should focus first on the accounts whose STAR ratings are 5 or 4. Propensity STAR Rating Accounts High Medium 5=Most appropriate for destacking Low 1=Least appropriate for destacking 9999 Total 9999 Accounts with High Propensity (STAR 5,4) are the most appropriate to be destacked to ibm.com than other accounts. Accounts with Low Propensity (STAR 1) are the least appropriate to be destacked to ibm.com. 32 Feb 2012

33 In summary, rating scales can cut through complexity and quickly lead to agreement and action Based on the ratings, it looks like we should focus on the finance industry Thank goodness for rating scales! 33 Feb 2012