MODELLING AND VALIDATION OF KPIs. Ella Roubtsova, Open University of the Netherlands Vaughan Michell, University of Reading, UK

Size: px

Start display at page:

Download "MODELLING AND VALIDATION OF KPIs. Ella Roubtsova, Open University of the Netherlands Vaughan Michell, University of Reading, UK"

Patrick Reeves
8 years ago
Views:

1 MODELLING AND VALIDATION OF KPIs { Ella Roubtsova, Open University of the Netherlands Vaughan Michell, University of Reading, UK

2 68% visible improvement in the skin condition in 4 weeks FUN is the most effective 'key performance indicator Roland Hanekroot What do you think about these sentences?

3 Why do we need KPIs?

4 fragmental knowledge about business processes Why are there misleading KPIs?

5 KPI Goals Requirements KPI Concepts validation Goal Business process Validation of properties of KPIs at the implementation phase

6 Name: PI29 Definition: efficiency of allocation of security officers type: discrete time frame: month scale: very_low-low-medium-high-very_high source: mission statement, job descriptions owner: forward and daily planning departments threshold: 2 units hardness: soft V. Popova A. Sharpanskykh De Montfort University, UK Existing Approaches to KPI design

7 MetricM with language MetricML Stefan Strecker Ulrich Frank David Heise Heiko Kattenstroth University of Duisburg-Essen, Germany Existing Approaches to KPI design

8 1. quantifiable 2. sensitive to changes of the business process state 3. linear 4. semantically reliable 5. oriented to improvement, not to conformance to plans Kechenham and Winchel Desired Properties of KPIs

9 A Key Performance Indicator indicates how the system performs To perform means - to carry through (as a process) - operate; - run; - execute RESEARCH PROBLEM : Find a method for validation of the properties of KPIs on the basis of fragmental knowledge about business s processes

10 KPI Goals Requirements Validation of KPI properties KPI Concepts Goal Abstract Business process Business process Validation of KPI applicability We want to validate the properties of KPIs at the design phase

11 Goal: Produce Goal: Serve Goal: Secure Goal: Green Requirements Requirements Requirements Requirements Concept Concept Concept Protocol Machine B Protocol Machine A D Protocol Machine B A X Y Protocol B D Machine Protocol Machine B A A A X D X X X Y B D Y D Y Y Add Protocol Modeling to existing methods for Validation of KPI properties

12 1. Fragments of processes as LTSs with data 2. Synchronous composition of fragments using CSP parallel operator extended for LTS with data 3. States are quiescent visible states used in goal definitions Why protocol modelling?

13 OBJECT ReferredPerson NAME PersonName INCLUDES Treatment, Assessment ATTRIBUTES PersonName: String, DateOfReferring:Date, STATES referred, left referred*leave=left, left*return=referred, referred*entertreatment=referred 1. Fragments as LTSs with data

14 EVENT Refer ATTRIBUTES ReferredPerson: ReferredPerson, PersonName:String, DateOfReferring:Date Events drive the transitions

15 Accept Refuse Accept Refuse Protocol Machine (1) Ignore Ignore Accept Refuse Protocol Machine (2) Ignore 2. Synchronous composition of fragments using CSP parallel operator extended for LTS with data

16 3. States are quiescent visible states used in goal definitions

17 PM = (E, s0, S,A, T), PM= PMi, i=1,,n KPI = f(t, I, VA, mpmi, ) where APMi 0 n mpmi PMi: Ψ(t,I,VA, )=true APMi Constructive Definition of a KPI

18 1. Quantifiable form 2. Sensitivity to change is dependency from arguments mpmi, APMi 3. Linearity is the linear dependency of KPI from the arguments (numbers of objects or values of their attributes) 4. Semantic reliability is measured by the number of additional assumptions that need to be made in order to derive the KPI value. If the set of additional assumptions is empty, the KPI is semantically reliable. 5. Improvement orientation is indicated as an empty set of scenarios enabling manipulation the KPI numbers to meet the planned values.

19 Case: Improving Access to Psychological Therapies (IAPT)

20 KPI1: Level of Need. Survey IAPT Indicators: Access

21 KPI3a: The number of people who have been referred KPI3b: The number of active referrals who have waited more than 28 days KPI4: The number of people who have entered psychological treatment IAPT Indicators: Access

22 HI1: Access Rate = KPI4/KPI1= (Number Of Entered Treatment / Level Of Need) IAPT Indicators: Access

at initial assessment achieve ``Caseness" and at the final session - did not.

23 KPI5: The number of people completed treatment. KPI6: The number of people moving to recovery. at initial assessment achieve ``Caseness" and at the final session - did not. KPI6b: The number of people who have completed treatment but were not at ``Caseness" at initial assessment. IAPT Indicators: Recovery

24 HI2: Recovery Rate = KPI6/ (KPI5-KPI6b)= Number Of People Moving To Recovery/ ( Number Of People Completed Treatment - Number Of Caseness People Before Treatment). IAPT Indicators: Recovery

25 Psychological Therapy Program of Improving Access to Psychological Therapies Survey of the Needs of Population AND Goal: A Referred Person has access to psychological therapies AND Goal:Measure access to psychological therapies Goal: Estimate the size of the population of people needed psychological therapy Goal: A Referred Person after treatment has improved conditions Goal: Measure effectiveness of treatment Goals of IAPT and underlying processes

26 Psychological Therapy Program of Improving Access to Psychological Therapies Survey of the Needs of Population AND Goal: A Referred Person has access to psychological therapies AND Goal:Measure access to psychological therapies Goal: Estimate the size of the population of people needed psychological therapy Goal: A Referred Person after treatment has improved conditions Goal: Measure effectiveness of treatment Concept:Referred Person Attributes: Name: String Date of Referring: Date Caseness Before: Boolean Caseness After: Boolean State: {Referred, Waited 28 days, Entered treatment, Completed treatment} Concept: Recovery Dashboard Attributes: KPI5: Number of people completed treatment KPI6: Number of people moving to recovery KPI6b: Number of people not at caseness before treatment HI2:RecoveryRate Concept: Access Dashboard Attributes KPI1:Number of people with disorder identified by the annual survey KPI4:Number of people with disorder who in the entered treatment HI1: Access Rate Concept:Survey Attributes: Date of Survey: Date Population of people with disorder: Integer Conceptual model

27 Protocol Model gives precise meaning to properties

28 Quantification means deriving a number or a numerical conclusion from a set of instances of selected concepts in the models. quantifiable selectinstate ("BehaviourName","State") selectbyref ("BehaviourName","AttributeName")

29 // KPI 5 Number of People Completed Treatment in the reported quarter public int getnumberofcompletedtreatment() { int NumberOfCompletedTreatment=0; Date StartRQ = getstartquarter(); Date EndRQ=getEndQuarter(); Instance[] completedtreatment = selectinstate("referredperson", "referred"); for (int i = 0; i < completedtreatment.length; i++) { Date completiondate=completedtreatment[i].getdate("dateofcompletion") String treatmentstate = completedtreatment[i].getstate("treatment") ; } if (completiondate.compareto(startrq)>=0 && completiondate.compareto(endrq)<=0 && treatmentstate.equals("completed")) NumberOfCompletedTreatment+=1; } return NumberOfCompletedTreatment;

30 calculated without assumptions semantically reliable KPI6, KPI6b and HI2 are not reliable -execution helps to communicate with users and find the information instead of assumptions ATTRIBUTES CasenessBefore:Boolean, CasenessAfter:Boolean

31 Manipulation of numbers is impossible oriented to improvement, not to conformance to plans HI2: Recovery Rate = Number Of People Moving To Recovery / (Number Of People Completed Treatment Number Of Caseness People Before Treatment)

32 Rules for the Caseness assessment Assignment of organizations responsible for the treatment and assessment. Possible solutions for improvement of KPIs for IAPT

33 A method for validation of properties of KPIs at the design phase Results

34 KPI Goals Requirements Validation of KPI properties KPI Concepts Goal Abstract Business process Business process Validation of KPI applicability Future work

EXTREME: EXecuTable Requirements Engineering, Management and Evolution. Dr. Ella Roubtsova Open University of the Netherlands

EXTREME: EXecuTable Requirements Engineering, Management and Evolution Dr. Ella Roubtsova Open University of the Netherlands Two phases of the Cognitive process Cognitive laws state that men: discover