Social Impact Measurement System New South Wales Government National Disability Services

Transcription

1 Social Impact Measurement System New South Wales Government National Disability Services CADR Conference May 2014

2 Introduction Les Hems Director of Research & Development Net Balance Research Institute Arjun Ravi Senior Associate Net Balance

3 Social Impact Measurement System project objectives Project aim: To promote and encourage a more independent and thriving NGO sector whilst maximising choices for people with a disability, their families and carers Project outcomes: Providing NGOs with the resources to accurately measure and improve their social impact Assisting NGOs to provide people with a disability, their families and carers more choice and opportunities by being able to demonstrate their impact Helping NGOs to attract and form partnerships with potential investors including government by providing them with a methodology to demonstrate their social impact 2

4 Deliverables 1. A general outcome framework that captures the key areas of change being created for clients through supports and services offered by the Disability sector 2. A tool(s) that can be used by the NGO sector to measure their social outcomes and present their findings in a clear and transparent way. 3. Accompanying documentation/resources that will enable NGOs to utilise the tool(s) with minimal hands on external training or support following the pilot. 3

5 Introduction about the framework The NDS Social Impact Measurement (SIM) Project outcomes framework includes 21 life outcomes for people with a disability that are categorised into 3 high level domains: Being This relates to feeling safe, healthy and well. It includes the 3 subdomains of physical wellbeing, personal wellbeing and independence. Belonging This relates to full participation in the community. It includes the 2 subdomains of social wellbeing and living environment. Becoming This relates to realising potential. It includes the 2 subdomains of learning and growth and meaningful activity. 4

6 Physical wellbeing 1. I feel safe and secure 2. I am as healthy and well as I can be Personal wellbeing 3. I have a positive outlook on life 4. I feel good about myself 5. I am able to cope with changes and difficulties in my life 6. I can let people know what I need Independence 7. I have the support I need to undertake daily living activities 8. I choose how to live my life 9. I can move around freely 10. I can pay for the things I need 11. I feel comfortable about the balance between what I do myself and what I rely on others for Learning and growth 18. I have opportunities to learn 19. I feel satisfied with my accomplishments Making a contribution 20. I have a job I like Social wellbeing 12. I have meaningful and mutually supportive personal relationships with others 13. I feel connected to my community 14. I have the opportunity to participate in various activities in my community that I enjoy 15. I have the same legal and human rights as everyone else Living environment 16. I feel at home where I live 17. I find my community physically accessible 21. I undertake roles within my community or social group that are valuable and worthwhile to me

7 Methodology The development of the outcomes framework was an iterative process that involved the follow key steps: 1 3 Undertaking a literature review. This involved reviewing a broad range of literature from the disability sector, with a focus on identifying and analysing existing credible outcomes frameworks and social return on investment (SROI) analyses. Undertaking a gap analysis of outcomes identified. This involved extensive stakeholder consultation and review of further literature (as suggested by key stakeholders) to ensure that important life outcomes were not missing from the framework. 2 4 Synthesising outcomes. Simplifying outcomes and grouping them under appropriate outcome domains. 6

8 Literature review The literature review involved the identification and analysis of: Outcomes frameworks and social return on investment (SROI) analyses (from Australia and overseas) utilised for measuring outcomes in disability, health, aged care, and wellbeing. The review included 25 outcomes frameworks and 10 SROI analyses. Key policy documents related to the disability sector, at a: State level (e.g. Standards in action: Practice requirements and guidelines for services funded under the Disability Services Act) National level (e.g. National Standard for Disability Services) International level (e.g. United Nations Convention on the Rights of Persons with Disabilities) 7

9 Literature review The review synthesised outcome domains, individual outcomes and outcome indicators within existing frameworks or evaluations/reports. The literature review also analysed trends related to how existing frameworks are structured and communicated. For example, the New Zealand Disability Strategy used simple, first-person language to communicate outcomes. 8

10 Emotional wellbeing Autonomy; choice and decisionmaking; empowerment Physical wellbeing; health; vitality Independence Social inclusion Economic participation 59 different but closely interrelated outcomes were identified Increased mobility; moving and getting around; balance and functioning; increased confidence to be mobile Material wellbeing; living standards - assets, housing, income Economic security Safety in home and community I have my own home; stability of housing situation Protection from avoidable harm, disease and injuries Experiencing continuity and security Freedom from physical and emotional abuse, neglect, harassment and self-harm Mental health; reduced levels of poor mental health and depression; improved coping skills; resilience Confidence Self-esteem Increased hope; focus on positives rather than negatives; optimism Positive feelings; non-disturbing emotions; peace of mind; happy Life satisfaction Absence of negative feelings; disturbing emotions Social skills; gained social awareness; getting along with others Cognition; understanding & communicating Spirituality Meaningful personal relationships; supportive relationships; intimate relationships Social belonging; value: I feel valued Family resilience Increased sense of acceptance by others Reduced social isolation; social contact Being part of a community; community connectedness; welcoming communities Inclusive and accessible communities; integrated environments; environmental citizenship - accessibility to spaces and transport Physical belonging Rights protection, justice and legislation People are treated fairly Civil citizenship; political participation Perform different social roles Advocacy Leadership Cultural participation; citizenship; expressing culture Participation in hobbies and community groups; recreation; leisure; how you spend time Meaningful employment and activities People choose where they work Wellbeing at work Alternatives to employment Can pay for things; increased income Self-determination Self-care hygiene, dressing, eating & staying alone Increased life skills Carrying out everyday living tasks, and decision-making Being responsible Delaying / reducing the need for care and support Freedom to manage risks in your own way; choosing supports Personal development; competence; capacity building People choose personal goals Develop a sense of achievement; realise personal goals Learning and applying knowledge and skills Engagement Having fun; feeling good How to live; lifestyle People choose where and with whom they live Meaning and purpose Artistic expression Sense of identity 9

11 Gap Analysis A broad range of stakeholders including experts in the disability sector, disability service provides and experts in outcomes measurement were engaged to: Ensure key outcomes were not missing from the framework Provide input into how the framework should be structured and communicated Stakeholders include: Natalie Nicholles, new economics foundation (nef), Outcomes Measurement Expert Valmae Rose, future by design, Disability Sector Expert Various representatives of NSW disability providers at workshops and interviews 10

12 Synthesis of outcomes and domains Synthesising outcomes identified within existing frameworks or evaluations involved an iterative process of simplifying the outcomes to ensure that each one was unique (i.e. outcomes didn t overlap), whilst still ensuring the framework was a comprehensive representation of life outcomes for people with a disability. SCOPE framework University of Toronto Quality of Life Framework The Early Years Learning Framework Victoria Quality Framework nef dynamic model of wellbeing 11

13 Physical wellbeing 1. I feel safe and secure 2. I am as healthy and well as I can be Personal wellbeing 3. I have a positive outlook on life 4. I feel good about myself 5. I am able to cope with changes and difficulties in my life 6. I can let people know what I need Independence 7. I have the support I need to undertake daily living activities 8. I choose how to live my life 9. I can move around freely 10. I can pay for the things I need 11. I feel comfortable about the balance between what I do myself and what I rely on others for Learning and growth 18. I have opportunities to learn 19. I feel satisfied with my accomplishments Making a contribution 20. I have a job I like Social wellbeing 12. I have meaningful and mutually supportive personal relationships with others 13. I feel connected to my community 14. I have the opportunity to participate in various activities in my community that I enjoy 15. I have the same legal and human rights as everyone else Living environment 16. I feel at home where I live 17. I find my community physically accessible 21. I undertake roles within my community or social group that are valuable and worthwhile to me

14 Becoming Learning and growth Outcome 18 - I have opportunities to learn This outcomes relates to: Having the ability and opportunity to build knowledge, skills and experience lifelong learning. This relates to building your capabilities through formal and informal training or education that is responsive to your needs. Key references: International Classification of Functioning, Disability and Health UN Convention on Rights of Persons with Disabilities Shut out: The Experience of People with Disabilities and their Families in Australia Victoria Quality Framework WA Quality Framework SCOPE Outcomes Framework new economics foundation, National Accounts of Wellbeing Mental Well-being Impact Assessment Toolkit, UK New Zealand Disability Strategy

15 Worked example: Outcome 18 - I have opportunities to learn Evidence identified through literature review (key words are highlighted): Reference International Classification of Functioning, Disability and Health UN Convention on Rights of Persons with Disabilities Evidence for outcome Within Activities and Participation component there are nine chapters, one of which is Learning and applying knowledge. This includes the subdomains of purposeful sensory experience, basic learning and applying knowledge. These are assessed on the basis of performance and capacity. Article 24 Education: - Right to education without discrimination and the basis of equal opportunity - full development of human potential including personality, talents and creativity, as well as their mental and physical abilities - effective individualised support measures are provided 14

16 Worked example: Outcome 18 - I have opportunities to learn Reference UN Convention on Rights of Persons with Disabilities cont d Evidence for outcome - Employ teachers who are qualified in sign language and/or Braille - Use of appropriate augmentative and alternative modes, means and formats of communication, educational techniques and materials - Persons with disabilities are able to access general tertiary education, vocational training, adult education and lifelong learning without discrimination and on an equal basis with others Victoria Quality Framework SCOPE Outcomes Framework WA Quality Framework Within life area 1 Always learning there is the outcome People with a disability experience lifelong learning and education. Within outcome domain Economic citizenship there is a subdomain Education and training. Definition of this outcome domain is The quality of life in the economic domain including a person s agency (to make meaning, act, influence and contribute) in the areas of employment, education/training, a range of economic roles. Within outcome domain Alternatives to employment there is a subdomain Life long learning. 15

17 Worked example: Outcome 18 - I have opportunities to learn Reference Shut out report Evidence for outcome The education experience of people with a disability is one of the seven areas explored In the report. - Education represents an important opportunity to imagine and create an alternative future for individuals - True inclusion is about more than location it is about achieving the same quality of education - The education system continues to fail to respond to the needs of students with disabilities and, as a result, these students continue to lag behind on a range of attainment indicators these results are not a reflection of a lack of ability of students but of the failure of the system to meet their individual needs MWIA Toolkit, UK new economics foundation, National Accounts of Wellbeing Within outcome domain Education there is a subdomain Life long learning. Indicator under Positive functioning component and engagement subcomponent is To what extent do you get to learn new things? 16

18 Worked example: Outcome 18 - I have opportunities to learn Reference New Zealand Disability Strategy Evidence for outcome Objective 3 states Provide the best education for disabled people. This includes the following actions: - Ensure that no child is denied access to their local, regular school because of their impairment. - Support the development of effective communication by providing access to education in New Zealand Sign Language, communication technologies and human aids. - Ensure that teachers and other educators understand the learning needs of disabled people. - Ensure that disabled students, families, teachers and other educators have equitable access to the resources available to meet their needs. - Facilitate opportunities for disabled students to make contact with their disabled peers in other schools. - Improve schools responsiveness to and accountability for the needs of disabled students. - Promote appropriate and effective inclusive educational settings that will meet individual educational needs. - Improve post-compulsory education options for disabled people, including: promoting best practice, providing career guidance, increasing lifelong opportunities for learning and better aligning financial support with educational opportunities. 17

19 Worked example: Outcome 18 - I have opportunities to learn Synthesis of key concepts from literature review: Lifelong learning Ability and opportunity to learn new things Building skills and capability Training and education that is responsive to individuals needs 18

20 The SIMS tool The SIMS tool is a pair of interrelated excel worksheets that function on a client and organisational level respectively The tool has been through a series of iterations and has been workshopped with the SIMS project steering committee and other members of the disability sector The contents of the tool to be used during the pilot phase has been developed based on feedback from stakeholders pertaining to relevance, appropriateness, ease of use and data requirements 19

21 Summary of feedback and updates during the development process The aim of the tool is to track a client s quality of progress towards achieving outcomes NOT if they have achieved them or not. It was acknowledged that it is not realistic for a client to be able to achieve substantial outcomes within one plan period Goals in client plans will not be retrofitted to the Outcome framework, as this would be service-centric rather than client-centric Clients will thus choose Primary outcomes that would be directly affected by their set goals but provide baseline responses to all the outcomes in the client based tool This allows the organisation to identify any unintended outcomes resulting at the end of the plan period The theory of change presented will be outcome-centric rather than based on programs or disabilities Organisations will have the ability to provide their own qualitative theory of change for applicable outcomes and the option of linking it to NDIS supports if applicable 20

22 Summary of feedback and updates The contribution of multiple service providers to one client s outcomes is an important aspect to consider This shall be captured in the tool qualitatively buy capturing a list of all providers and supports utilised in a client s. It was agreed that quantifying the contribution would be difficult and potentially misleading Progress towards outcomes in the client focussed tool needs to be based on a simple and easily comprehensible scale Simple satisfaction terminology and a selection of images will be used It is acknowledged that the attainment on outcome is not based on clinical measures rather the self-reporting of clients based on their perception of personal change over the period of their plan 21

23 SIMS tool components Client outcomes tracking tool Organisational impact tracking tool Individual client specific Used to capture client level data on outcomes, service satisfaction and support service usage Provides outcome metrics that can be fed into the organisational impact measurement tool Unique to every client at the organisation Applicable to whole organisation Includes service provider data on ability to deliver outcomes (theory of change) Requires client level data from the COT tool to be input as available in order to aggregate results for entire organisation Provides a snapshot in time of clients quality of progress towards their outcomes 22

24 Client Outcomes tracking tool Data and Outputs Data input Tool outputs Monitoring of progress towards social outcomes facilitated by client s goals (Primary and Secondary) - Determined by a pre (at the start of the plan) and post (at the end of plan) assessment of clients subjective responses to a 8 point scale - Progress will also be recorded after 12 months of being on a plan if it is over a year long Number of supports used - Based on the support services specified by NDIS Number of service providers involved in clients plan - Listed in free text box Customer satisfaction survey questions - Questions to be crafted based on experience of service and level of choice and control A client progress dashboard with data in a format that can directly be entered into the Organisational Impact tracking tool - Quality of progress made towards clients outcomes through the period of their plan - Total cost of supports utilised by the client based on NDIS provided costs - Awareness of support contribution provided by other service agencies for the client - Feedback on clients experience of service at the service provider 23

25 24 Client Outcomes Tracking tool

26 25

27 Organisational Impact tracking tool Data and Outputs Data input Tool outputs Basic service provider information - Operational site address - Other service sites under the same umbrella - Contact - Phone (with options) - - Website Clients served information - Gender - Age range Option to specify specialisation in treating particular client groups (based on NMDS categorization) - Provider does not need to specify and can select ALL Establishing a theory of change per outcome - Free text on how the organisation facilitates the achievement of outcomes listed in the NB NDIS outcome framework - Ability to select relevant supports relevant to each outcome if appropriate Individual client level data from COT tool - outcome progress at end of their plans - service satisfaction progress at end of their plans - Individual client level supports costs Outcome-centric communications regarding organisation s ability to produce quality social outcomes for client (assisted by other aspects of toolkit) Aggregated client information on; - Distance travelled measures on quality of outcomes created, - Average start and finish points on outcome scales, - Most common primary outcomes selected, - Potential relationships between primary and secondary (unintended) outcomes, - Total and Average support costs per client/s, - Average number of additional support services used by clients, - List of other service providers used by clients, - Satisfaction levels (Service, control, choice) through a summary organisation dashboard 26

28 27 Organisational Impact tracking tool

29 Next Steps The tool is to be tested at out pilot organisations in June 2014 Net Balance to provide a days training and organisations to collect data over a 3 week period Feedback used to improve usability and add functionality if required SIMS toolkit to be finalised by August 28

30 Feedback NDS Outcomes Framework Do you feel the framework sufficiently addresses the impacts created by all areas of the disability sector Is the language used appropriate Are there any other frameworks we should consider to add to the robustness of the framework? Any other comments? 29

31 Feedback Social Impact Measurement Tool Do you feel the tool would be useful to service providers? Is there replication of present reporting requirements? Can we collect any additional data? Are the wording and graphics used on the measurement scales appropriate for a wide range of clients? Are the visuals useful? Would you like to see them presented in other formats? Any other comments 30