Performance Measurement Maturity in New Zealand Universities. Zhivan Alach Unitec Institute of Technology AAIR Forum Nov 2015

Transcription

1 Performance Measurement Maturity in New Zealand Universities Zhivan Alach Unitec Institute of Technology AAIR Forum Nov 2015

2 STRUCTURE 1. What is performance measurement? 2. Performance measurement in higher education 3. Synthetic performance measurement maturity model 4. New Zealand context 5. Methodology: 1. Documentary analysis 2. Survey 6. Findings (by synthetic model elements) 7. Discussion 8. Further research

3 WHAT IS PERFORMANCE MEASUREMENT? For this study: the use of qualitative and quantitative indicators to measure organisational activities, efforts, and achievements - drawing heavily from Wang (2002)

4 PM IN HIGHER EDUCATION Followed (re-)emergence of PM in public sector 1980s/1990s United Kingdom leading way Global trend Similar drivers: Relaxation of central control over finance Move from elite to mass education Move from industrial to knowledge economies Increased funding pressures

5 SYNTHETIC PM MATURITY MODEL Literature full of recommendations Synthesis of these can provide a synthetic maturity model Asks how well is organisation X doing performance measurement? Not how useful is the performance measurement?

6 SYNTHETIC PM MATURITY MODEL Model consists of three groups and seven elements: Group One: Use and Results Element One: Usage of Performance Information Element Two: Strategic Alignment and Prioritisation Group Two: Key Design Elements Element Three: Outcomes Framework Element Four: Variety of Measures Element Five: Depth and Insight of Commentary Group Three: Key Shapers Element Six: Internal Ownership Element Seven: Accurate and Timely Underlying Data

7 NEW ZEALAND CONTEXT Eight universities Crown Entities public sector Ministry of Education Tertiary Education Commission Office of the Auditor General NZ public sector early adopter of PM NZ universities lagged behind

8 METHODOLOGY Two approaches used: Documentary analysis to show evolution of practice over time Survey of university staff for greater insight of current approaches

9 DOCUMENTARY ANALYSIS Annual Reports 6 years * 8 universities = 48 documents 1.64m words Focus on Statement of Service Performance Both quantitative and qualitative analyses Anonymised findings

10 SURVEY TOOL s scraped from university websites Aimed at tier 2/3 (dean, HoD) level 67-question survey Likert scales 57/471 response rate (12%) Identifiers: University Academic or administrative

11 ELEMENT ONE: USAGE OF PERFORMANCE INFO Documentary: Possible growth through positioning of SSP No information about trade-offs Survey Usage Index 3.47/5 (higher than expected) Difference between academic / admin not significant Seen as useful for quantity but not for quality

12 ELEMENT TWO: STRAT ALIGNMENT & PRIORITISATION Documentary: Increase in full alignment cf. partial Sometimes linkages not explicit Absence of prioritisation Survey Strategy Index 3.95/5 (higher than expected) Difference between academic / admin not significant Senior leaders seen as strongly supportive But little sign that performance used for prioritisation

13 ELEMENT THREE: USE OF OUTCOMES FRAMEWORK Documentary: Clear increase but still only half using Use of language in commentary Apparent confusion about use Survey Respondents familiar (3.76/5) with admin more familiar than academics Highest use of output measures (research?) Greater support for more outcome and enabler measures vs. outputs

14 ELEMENT FOUR: VARIETY & COMPREHENSIVENESS OF MEASURES Documentary: Decline in mean and median of reported measures Measure-years (measure*years reported) declined Little change in breadth (still very narrow) Dominated by inputs and processes Survey Variety Index 3.96/5 Financials used most frequently overall Academics used research more than financials Service/community measures rarely used

15 ELEMENT FIVE: DEPTH & INSIGHT OF COMMENTARY Documentary: Page-length mean/median constant (SSP) Word-length mean/median down/up (SSP) Very mixed approach to commentary Used to make up for measure shortfalls Survey Cannot be evaluated

16 ELEMENT SIX: INTERNAL OWNERSHIP Documentary: Cannot be evaluated Survey Ownership Index 3.56/5 Academics less supportive Problems seen in other fields often discounted Qualitative seen as of more value than quantitative

17 ELEMENT SEVEN: ACCURATE DATA Documentary: Cannot be evaluated Survey Hardest to evaluate without internal access Data Index: 3.32/5 Collecting data seen as greatest difficulty Validity and reliability largely unquestioned

18 DISCUSSION POINTS Little clear evidence of performance targets (apart from financials) guiding decision-making No output costing No clear trade-offs Most common usage research quantity But given this wouldn t we expect greater investment in low cost-per-output departments e.g. English, History, Political Studies? Not used for third role why?

19 DISCUSSION POINTS Strategic alignment may be window-dressing Are performance targets derived from strategy or simply reverse-aligned? Is use of performance targets in strategic documents purely symbolic? Compare to (for example) NZ Police and crime rate reduction targets

20 DISCUSSION POINTS Use of outcomes framework limited Given its frequency in NZ public sector this is interesting Output measures (research) self-reported, but documents show more inputs and processes Where does the university finish? Is the outcomes framework applicable to higher education?

21 DISCUSSION POINTS Limited breadth of measures in publicly reported documents No sign of increase over time Measures cited are prone to gaming

22 DISCUSSION POINTS Unsurprisingly, administrators more supportive than academics Support for performance measurement does not correlate with belief that it is of benefit External actors appear to have had a limited effect on practice Much performance measurement focuses on internal capability cf. other public sector

23 FURTHER STEPS Organisational culture likely central to findings Two tentative hypotheses: H1: Public sector universities will show lesser evolution in performance measurement practice than other public sector organisations, even when subjected to equal or similar external accountability regimes. H2: Public sector universities degree of maturity in the use of performance measurement will reflect the extent to which resources are allocated according to resultsfocused (outcome and output) performance targets.

24 ALSO TO DO Compare to polytechnics in New Zealand Compare to universities in Australia Case studies within universities based on findings to date Compare university practice to other parts of public sector (core, Crown Entity etc.)

25 THANK YOU