ontario association of children s aid societies A Framework for Quality Assurance in Ontario Child Welfare Agencies

Transcription

1 ontario association of A Framework for Quality Assurance in Ontario Child Welfare Agencies January 2004

2 acknowledgements The following people and affiliations committed to the development of this framework as sub-committee members of the OACAS Quality Assurance Committee. Ann Lumsden, Project Lead Susan Smythe, York Region Children s Aid Society, Author Terry LeBlanc, Simcoe Children s Aid Society, Author Deborah Goodman, Toronto Children s Aid Society, Author Bruce Leslie, Toronto Catholic Children s Aid Society, Author Several agencies affiliated with the OACAS Quality Assurance Committee contributed excellent quality assurance information during the research phase of the project. We are grateful for the support of the Committee and Gail Vandermeulen s commitment to this project. Special thanks to Shelly Thibert, Chatham-Kent Integrated Children s Service and Sandra May, Hamilton Children s Aid Society for their early work on the project. As well, thanks to Susan Manitowabi, Assistant Professor, Laurentian University, for her helpful feedback regarding First Nations communities. The following people also assisted by reading drafts and providing thoughtful comments on the publication: Claude Gingras, Frontenac Children s Aid Society Myra Hurst, Ontario Association of Children s Aid Societies Mary McConville, Toronto Catholic Children s Aid Society Marty McNamara, York Region Children s Aid Society Sandy Moshenko, Waterloo Family and Children s Services Marion Roberts, Ottawa-Carleton Children s Aid Society Marty Rutledge, Family Youth & Children s Services of Muskoka Jacquie Sharpe, American Society for Quality Toronto Section Maureen Wallis, Simcoe Children s Aid Society Finally, the OACAS edited and produced the final document. Special thanks to Brenda Nichol and Carrie Reid. ontario association of 1

3 table of contents Preamble: Quality Assurance in Child Welfare 4 A Definition for Quality 4 Aspects of Service Quality 5 A Definition for QA 5 The QA Cycle 5 Organizational Culture and QA 6 Why QA Now? 7 The Framework 8 Step 1: Build the Agency Strategy for Achieving Quality 9 Strategic Plan - A Clear Sense of Purpose 9 Strategic Goals 10 Quality Planning 11 Step 2: Create A Quality Assurance Culture 13 Commitment of the Board 13 Commitment and Involvement of Management 13 Buy-In and Participation of Staff, Foster Parents, Volunteers 13 Communication of Quality Objectives to Suppliers 14 Establishment of a Structure 14 Positioning QA Staff 15 QA Committees 15 Other Methods to Convey Quality Expectations 15 Communication Strategy 16 Step 3: Establish Service Quality Requirements 18 Standards 18 Outcomes 19 Outcome Indicators 19 Threshold for Action 20 A Word About Support Services 20 Step 4: Collect Data 21 Case Review 21 Program Evaluation 22 Research 23 Stakeholder Feedback 24 Other Considerations 24 Step 5: Analyze Data for Impact 26 Quality Data 26 How to Analyze Data 23 QA Reports 27 Communication of Findings 28 Step 6: Improve Quality and Maintain Excellence 29 Maintain Excellence 29 Identification of Opportunities 29 Quality Improvement 29 Best Practice Development 32 ontario association of 2

4 table of contents Endnotes 33 References 34 References for Consultations in Aboriginal Communities 35 Quality Assurance Links 36 Appendices A Mission to Outcomes 37 B Logic Model 38 C Canadian Child Welfare Outcomes Indicator Matrix 39 D US Department of Health and Human Services Child Welfare Outcomes 40 E United Kingdom, Looking After Children (LAC) 41 F Board Responsibilities Related to QA 42 G Possible Functions of a QA Department 43 H QA Staff Position 44 I QA Committee: Terms of Reference 47 J QA Planning Group Terms of Reference 50 K Self Assessment Tool for Agencies Planning and Implementing 52 Outcome Measurement L QA Newsletter 59 ontario association of 3

5 preamble Child welfare service in Ontario has historically promoted quality through a variety of activities. During the early 90s momentum was created to better coordinate these activities and support new approaches to assure quality service. A Provincial Committee on Quality Assurance was formed in A handbook of preferred quality assurance practices and tools was developed by this group in an effort to assist the field. In 2002, the Committee identified the need for a guide to further assist agencies with the implementation of a Quality Assurance program. As a result, the following Quality Assurance framework has been developed to provide the field with the how to steps for the establishment of a Quality Assurance (QA) program. This Framework is designed to: Incorporate work already done by the Ontario Association of Children s Aid Societies (OACAS) in the QA and Continuous Quality Improvement - A Handbook of Preferred Practices and Tools - draft Provide the field basic information about QA Establish a basis for standardization of QA practices across the field Be easily adapted by all agencies First Nations, integrated, large or small, urban or rural Build on resources and practices currently in place Provide practical examples from agencies in the field A Definition of Quality Industry Canada defines quality as, The totality of features and characteristics of a product or service that bear on its ability to satisfy stated or implied needs. Definitions that are more easily understood in relation to service delivery address the notion that quality is defined by the customer and is related to customer satisfaction. 1 Child welfare quality is defined by many customers, among them: Children and youth who use the service Parents and other caregivers who use the service Referral sources Community partners Other child welfare agencies The Ministry of Community, Family, and Children s Services (MCFCS) The community at large These customers may have potentially conflicting definitions for quality child welfare service; however, there is consensus related to outcomes of child safety, permanence, and well-being. The challenge for Ontario child welfare is to provide quality service that satisfies all constituents. ontario association of 4

6 preamble Aspects of Service Quality Service quality is composed of several interrelated aspects 2 : 1. Inputs refer to those aspects of quality that form the foundation of service delivery e.g., policies and procedures, well-trained staff with the proper credentials and appropriate facilities. Inputs are a necessary but not sufficient condition for services to be of acceptable quality. 2. Access refers to whether or not those eligible for service are actually receiving the service are the right people receiving the service? 3. Process relates to the way that service is provided. Examining this aspect of service quality answers the question; did the provider carry out the proper actions in the appropriate way? 4. Outcome refers to the results of the service and whether or not the service had the desired effect. 5. Output has to do with the volume of service provided and whether or not service is being delivered in sufficient quantity. A Definition for QA QA is defined as, A formal set of activities that reviews and affects the quality of service provided. These activities provide both internal and external parties the confidence that the organization will consistently meet the requirements for quality. 3 QA provides activities that permit the agency to determine if service is being delivered to the specifications of the customer by identifying gaps and providing a means for improvement and maintaining excellence. The QA Cycle As described in Table 1, QA can best be described as a cyclical process. ontario association of 5

7 preamble Table 1 QUALITY ASSURANCE Develop Agency Strategy QUALITY MONITORING Create a QA Culture Establish Service Quality Requirements QUALITY IMPROVEMENT Maintain Excellence Collect Data Corrective Action Analyze Data for Impact Analysis Delineate Problem Evaluation Satisfactory Evaluation Unsatisfactory Organizational Culture and QA In order to carry out the QA function, an organization must embrace certain beliefs among them: Everyone in the organization is responsible for quality Employees want to improve quality Customers and suppliers are partners in service provision Problems in quality come from processes and systems, not individuals Quality is best evaluated with hard data Concentrating on quality focuses resources 4 ontario association of 6

8 preamble Why QA Now? The child welfare field in Ontario has begun to translate an interest in the provision of quality service into the development of more formalized QA activities. This development has been driven both internally by Boards of Directors, staff, foster parents, and volunteers, and externally by clients, funders, referral sources and communities at large. Traditionally, QA activities have been closely associated with audits, usually externally based, and related to compliance monitoring. The following are five other benefits that can be derived from QA. Service Effectiveness Perhaps the most important benefit of a QA program is the provision of tools that describe, develop and maintain quality service through: Organizational motivation Performance measurement Program evaluation Solicitation of stakeholder feedback Ongoing development and implementation of the agency vision/mission With better information about how service is being provided - what works, whether standards are met, where the gaps are - the agency is able to carefully tailor service delivery to meet agency goals for excellence. Increased Accountability Much has been written about the responsibility of CASs to increase accountability relative to resources expended and the outcomes achieved. 5 Increasingly, funders and the public want to know the answers to key questions about the nature and achievements of child welfare practice. For example: Do children and young people under the care of the child welfare system make progress? Are the financial resources allocated to Children s Aid Societies being used appropriately to improve the welfare of children in their community? Do child welfare agencies provide information related to the adequacy/limitations of funding? Are child welfare personnel staff, foster parents, managers, volunteers and board members suitably prepared to perform their roles as investigators, care providers, leaders and governors? 6 Everyone Wants to Provide Quality Service A focus on QA enables child welfare agencies to promote and sustain their internal values when providing excellence in service delivery. 7 When it is possible to measure effectiveness and recognize improvement by celebrating successes, other internal benefits for the organization are derived. Efficiency Better information as a result of QA activities allows agencies to become more efficient in service delivery a boon when resources are limited. Although there are costs ontario association of 7

9 preamble involved in the implementation of a QA system, savings can be realized when the agency is able to focus on processes that yield the greatest return for clients. Risk Management A Board of Directors, as a governing body, has a responsibility for QA especially in regard to risk management. Through the use of QA activities the agency s exposure to risk can be reduced. For example, by ensuring that staff holding the proper credentials have been hired and properly trained, risks associated with poor service delivery can be reduced. As well, the agency s ability to consistently comply with standards set by the MCFCS will determine the degree of exposure. A QA program can assist the agency through close monitoring and improving those activities that are key to the agency s success/survival. The Framework The following Framework is a proposed set of steps necessary for the creation and implementation of a QA program: 1. Build the Agency Strategy for Achieving Quality 2. Create a Quality Assurance Culture 3. Establish Service Quality Requirements 4. Collect Data 5. Analyze Data for Impact 6. Improve Quality and Maintain Excellence Because every agency is different in terms of size, location, and stage of organizational development, each will operationalize the various steps in a unique fashion. Additionally, depending on each agency s stage of development, the time taken to fully implement the framework may vary and could take several years. ontario association of 8

10 quality assurance framework Step 1: Build the Agency Strategy for Achieving Quality The first step in implementing a QA program is to develop a clear sense of purpose. In order to provide quality service, the agency must first understand its ultimate goal. With a clear understanding of what needs to be accomplished, the agency can then map out a QA plan to realize these objectives. Agencies can use a strategic planning process that includes the development of a mission and vision to generate its sense of purpose. Everything the agency is doing to improve must be tied to its mission and vision. In a sense, the Strategic Plan is the broader QA Plan for the agency. Strategic Plan A Clear Sense of Purpose Peter Pecora et al describe the essential components of the strategic planning process visioning, developing a mission, strategic objectives, organizational values and principles for practice - as techniques for focusing the organization. 8 Through the Strategic Plan, the Board of Directors and the Senior Management of the agency can unmistakably demonstrate a focus on quality. Unless the agency s Strategic Plan, replete with the beacons that guide the organization, 9 indicates a priority for quality, members of the agency will not understand what is required. The mission, vision, philosophy and core values of an organization guide its work. These underlying values are developed with input from stakeholders and revisited from time to time. 10 Most child welfare agencies have developed mission and vision statements that derive from both the mandate of the Child and Family Services Act and the needs of their local community. The agency s QA strategy must flow from the mission and vision. Vision The Vision Statement identifies where the agency would like to be in the future. It is inspirational, inclusive, leading edge, and provides focus and clear decision-making criteria. 11 CAS Example Vision Statement A Child Loves Everybody First The Creator has entrusted us with the sacred responsibility for protecting our children, developing and sustaining strong families, and building healthy communities. The future of our communities is our children who need to be nurtured within their families and communities. Tikinagan Child and Family Services was created by the Chiefs of Nishnawbe-Aski Nation to strengthen our children, our families and our communities. Community responsibility for child protection is an essential aspect of Native self-government. ontario association of 9

11 quality assurance framework Mission The Mission Statement describes the overall purpose of the organization: Who are you? Who are your major customers? What do you do for them? What do you want to achieve? What sets you apart from everyone else? How do you do it? CAS Example - Mission Statement The Mission of the Children s Aid Society is to protect children and to strengthen and support the well-being of children and their families. (Peel CAS) Values The values statement reflects the philosophical intent, shared beliefs and key considerations that guide the agency s members in performing their work. CAS Example - Values Statement We Put the child s best interests first Champion enduring relationships for children and youth Advance equality, diversity and human rights Promote partnerships and teamwork Demonstrate professional excellence, leadership, innovation and advocacy (Children s Aid Society of Toronto) Strategic Goals The next step in the strategic planning process is the development of goals that will assist the agency in realizing its mission and vision. A goal is defined as the expression of direction or priority and is characterized by the indication of a direction and/or vision. Example Strategic Goals 12 Keep children safe ontario association of 10

12 quality assurance framework Quality Planning Once goals have been established through the strategic planning process, the agency needs to develop an action plan for quality assurance. Some agencies find it helpful to develop this plan in the form of a QA statement. A QA Statement A QA Statement clearly articulates the agency s intentions regarding quality by answering the following questions: Where does the agency want to go vis a vis QA? What needs to change in order to move toward the mission? How does the agency wish to provide services and at what levels? Who should benefit? What will those benefits be? The QA statement must be clearly connected to the organization s vision and mission. When this statement is developed, consideration should be given to all areas and functions within the organization. The exercise of developing this type of statement is worthwhile. A clear QA statement can assist with the identification of service needs. Agency initiatives or programs may be referenced within this statement. Staff responsible for developing and leading QA can use it as a reference regarding the areas that must be examined through QA initiatives. 13 The following are elements of a Quality Statement: Definition of quality Quality linked to agency goals Scope of quality effort Outcomes and goals short and long range Focus on customers internal and external Involvement of all employees Implementation by all staff, foster parents and volunteers Leadership by upper management 14 ontario association of 11

13 quality assurance framework CAS Example QA Statement The Catholic Children s Aid Society is committed to deliver high quality child protection services to families and children of the Catholic community of Toronto that address The safety and well-being of children and youth The need of children and youth to have a sense of permanence and continuity whether in their own home or in residential care Building partnerships with families and community resources that strengthen families and communities and increase the safety net for children. To this end, the Society is committed to Continuous review of services, service delivery systems and business processes Evaluation of the outcomes of service Receipt and review of feedback from clients, staff, caregivers, volunteers, partners and funders Professional excellence through ongoing training and learning. So that the Society can continually provide the range of supports required by our staff, adoptive and foster caregivers and volunteers and respond in a timely and appropriate manner to meet the needs of families and children by Creating opportunities for learning and improvement, Recognizing and rewarding individuals in the organization for their contributions, Developing a strong network of community services, Demonstrating accountability and making the most effective use of resources Advocating for needed services. And our clients will have the opportunity to Receive high quality services based on standards, best practices and research Be treated with dignity and respect of their cultural, racial and individual differences Communicate about the services they receive and what services they require from us ontario association of 12

14 quality assurance framework Step 2: Create a Quality Assurance Culture Having established the agency s vision, mission, and strategy for achieving quality, QA must become a part of the culture of the organization in order to move the agency from compliance monitoring to continuous quality improvement. This culture requires not only a clear and consistent direction but also a set of expectations regarding quality as established in Step 1. In addition to the beliefs that need to be embraced by the agency, there are other conditions that need to exist including practical steps that can be taken in order to instill a quality mission within the agency. The incorporation of QA throughout the agency is an iterative process that is addressed at key points during the life of the agency. Commitment of the Board In addition to sending a clear message via the Strategic Plan, the Board can also show a commitment to QA by ensuring that adequate resources (adequate staffing, appropriate information systems, training) are available for the QA program. Another opportunity for visible leadership occurs if the Board establishes a Board committee with responsibility for quality. Participation on this or an in-house QA committee indicates to the rest of the agency that QA is important. Regular monitoring of quality indicators is a further indication of Board concern. See Appendix F for Board responsibilities directly related to QA Commitment and Involvement of Management Where there is a quality problem, there is an 85% to 94% probability that management caused it is not simply responsible for it, but actively caused it. 15 One of the central premises related to QA is the belief that problems in quality come from systems, not individuals. Since management is responsible for systems, their involvement is key to the promotion and advancement of a QA Framework. The Executive Director, in particular, must provide leadership; without the commitment of the agency s top executive, all quality initiatives are doomed. The Director of Service also has a role to play in conveying the all-important message that service quality is the agency s paramount consideration. Senior and middle management must provide leadership to the agency through: Demonstration of activities and behaviours that indicate that quality is a top priority Support of an environment where opportunities for continuous learning are welcomed 16 Allocation of resources to QA Buy-In and Participation of Staff, Foster Parents, Volunteers Management s commitment to quality would be futile without the support and contribution of the entire agency s key stakeholder groups. Frontline staff, foster families and volunteers are in the best position to carry out the agency s quality objectives because of their direct client contact. They can also identify gaps in service and receive ontario association of 13

15 quality assurance framework feedback from the agency s customers 17. As a result, they can provide the agency with opportunities for quality improvement. Communication of Quality Objectives to Suppliers Suppliers are another important stakeholder group who must embrace the agency s quality objectives. In order to ensure that services purchased on behalf of agency clients are in keeping with the agency s quality expectations, suppliers (consulting psychiatrists or psychologists and outside paid institutions) need to be oriented to those expectations, and held accountable for the provision of quality service. Expectations should be spelled out in the service contract. Establishment of a Structure In order to operationalize the agency s commitment to QA, a structure designed to integrate QA into the way of doing things is necessary. The form that this takes will be unique to each agency. Decisions made regarding the way that the QA program is carried out will, again, signal the agency s intentions regarding QA. The following could be given consideration in establishment of the structure: Board, Board/Staff or Staff committees empowered to address issues of quality (see Appendices I and J for sample Terms of Reference for these types of committees) A QA department (see Appendix G for possible functions of a QA department) A full/part time QA staff position (see Appendix H for a job description) CAS Example - QA Staffing York CAS 1 FTE QA Specialist, reports to the Director of Corporate Services who in turn reports to the Executive Director Board Committee comprised of Executive Director, Director of Corporate Services and QA Specialist Staff Committee comprised of Executive Director, Director of Corporate Services, Director of Service, Managers, QA Specialist, Staff, Foster Parent and an Administrative Assistant Child and Family Services of Timmins and District Manager of Quality Assurance, Communications, and Training, reports to Executive Director Simcoe CAS 0.5 FTE Manager, reports to the Executive Director Staff Committee comprised of Executive Director, Director of Service, middle Managers, Frontline Staff and Statistician Family, Youth, and Children s Services of Muskoka Manager of Children in Care, reports to the Executive Director. Standing agenda item for middle management meetings ontario association of 14

16 quality assurance framework Positioning QA Staff QA staff may be privy to performance information related to other staff members. For this reason, it is important to ensure that QA positions are excluded from the bargaining unit. QA Committees QA Committees may vary widely in composition. The broader the representation, the stronger the message that is conveyed regarding the importance of quality to the agency. For example, a QA committee comprised of both frontline and management staff will indicate that all employees are responsible for quality. A broad-based committee provides opportunity for input from all stakeholders. CAS Example - QA Committees Waterloo Family and Children s Services: Quality Assurance Planning Group The purpose of the Quality Assurance Planning Group will be to develop and implement a Quality CAS Assurance Examples: Program QA for Committees Family and Children s Services in order to ensure that children and families are receiving the best service possible and that standards are being met. For the complete Terms of Reference refer to Appendix J Other Methods to Convey Quality Expectations In addition to resources dedicated to the implementation and oversight of QA processes, the agency can express quality expectations through: Communication to staff about standards and desired outcomes beginning with new staff during orientation Evaluation of staff performance on goals connected to quality (and related directly to the Mission of the agency) e.g., a Director of Service with responsibility for Intake could have the following goal: to have less than 20% of cases open for more than 60 days Training regarding QA tools available for use by staff and foster parents Documentation of expectations in policies and procedures Making QA a standing agenda item for senior and middle management and individual team meetings Inclusion of quality expectations and standards in provider contracts 18 Communication to customers, community partners and external stakeholders regarding standards and quality expectations ontario association of 15

17 quality assurance framework Communication Strategy 19 A key element of the entire QA initiative is the development of a communication strategy. Effective communication will: Keep the channels of communication open to nurture input of critical information Keep the agency up-to-date on the various QA initiatives Inform the agency of the successes and failures of its drive to improve services Become a resource and a source of motivation for staff in the pursuit of QA Initially, communication will deal with the reasoning and advantages behind the QA initiative. Once the QA program is under way, the communication strategy must shift to ensure that the initiatives and results of quality improvement activities are being publicized. This information will be used to ensure that: individuals and committees stay on track staff development and outcomes in key areas are highlighted feedback is solicited and information is distributed to those who require it in order to evaluate their progress on individual, department or corporate QA goals Information can be communicated by a variety of means, including regular status update meetings, team meetings, employee newsletters, electronic mail, bulletins and through supervision sessions. Regardless of the medium selected, it is important for communication to be timely and tailored to the needs of the audience. The preferred strategy would communicate to all stakeholders. Refer to Appendix L for an Example of a QA Newsletter. CAS Example - Communication Strategy Increasing Awareness of Quality Assurance at Kenora-Patricia Child and Family Services 1. PowerPoint presentation by the manager with QA function to the Board of Directors and agency staff unit by unit to increase awareness of QA and its role in the agency by: o Reviewing the outcomes of this year s Crown Ward Review and comparing them with the past two years outcomes o Examining how the agency functions as an organization and the role of Quality Assurance in the organization o Discussing the agency s Quality Assurance Program Proposal o Soliciting CAS broad Examples: representation Communication of staff, board and (and Training later other Strategies stakeholders) on the QA Committee. 2. Following the presentation, the Executive Director holds goal-setting meetings related to Quality Assurance with each unit to reinforce awareness of the commitment of the agency to QA. ontario association of 16

18 quality assurance framework CAS Example - Communication Strategy BEYOND COMPLIANCE at Simcoe CAS Training Components This training was done with all staff in the agency. It was completed at the team level, generally in their branch. A modified version was also presented to the Foster Parent Association. It included the following components: 1. Quality Assurance committee composition, definition of QA, principles of QA, committee outcomes 2. Watched ½ hour video on Quality Assurance entitled The Cornerstones of Quality (Toastmasters) 3. The group identified the QA Committee principles that were seen in the video 4. The group engaged in a discussion about activities that are currently happening that they would label QA. The group was given an exercise to complete to emphasize the need to examine the cause of the quality problem before jumping to solutions. Quality Tree Contest The purpose of the contest was as follows: 1. To engage the staff in the QA process 2. To define outcomes at the team level 3. To celebrate successes The contest included: 1. Each team received a kit to create a tree, apples and leaves. 2. The teams were to write outcomes that they had achieved on individual cases on apples (e.g., overload successfully handled 22 intakes in 6 weeks) and quality work they had completed on the leaves (e.g., worker formulated solid plan involving a newborn using community resources). 3. The leaves and apples were added to the tree. The team with the most apples and leaves after a 2-week period won the contest. ontario association of 17

19 quality assurance framework Step 3: Establish Service Quality Requirements In order to operationalize the quality objectives determined through the strategic planning process and/or articulated in the agency s QA Statement, standards for service delivery and desired outcomes need to be defined. When staff, foster parents and volunteers are clear about where the agency is heading and how that mission is translated into expectations about how service is to be provided, they are better able to carry out their task. With clearly stated standards and outcomes, it is possible to evaluate, through the QA process, whether the agency has achieved its intended vision/mission. Standards Standards for service delivery are determined in a variety of ways. One way is through Ministry standards relating to foster care, crown wards, children in care, adoption, and protection. Additionally, some agencies have adopted best practice standards for certain areas of service delivery (i.e., management of high risk infants), which have been gleaned from the literature or determined by the field to be effective for service delivery. Standards often address service quality aspects related to inputs, process, access and output. Some applicable examples are as follows: Inputs OACAS Human Resources Accreditation Standard 7.1 Qualifications Process Foster Care Licensing Term & Condition 29 Request to Change Placement Meeting to Determine Access OACAS Family Service Accreditation Standard 1.3 Service Eligibility, Output Benchmarks established in the Funding Formula Framework have become standards for the volume of service provided CAS Example Practice Standards Ministry Standard Ontario Reg. 213/00, Sec 4(1) Every society shall ensure that each child in the care of the society is given a medical and dental examination as soon as is practical after the admission of the child to care. ADMISSION MEDICALS OACAS Accreditation Standard Children in Care 1.2: When admitting a child to care: admission medical examination is arranged/carried out on admission or within 72 hours in emergency or exceptional circumstances. Agency Standard PROCEDURES: Admission Medical 2. The admission medical should take place upon admission to care or within 72 hours if the placement was an emergency or due to unusual circumstances. (Toronto Catholic CAS) ontario association of 18

20 quality assurance framework Outcomes Outcomes are the benefits or changes that participants/clients/patients/communities accrue during or after their involvement with the program/service. Changes can occur in the following areas: Behaviour Attitude Knowledge Values Status Condition It is assumed that most child welfare agencies mandates fall under three broad, wellestablished outcome areas - child safety, permanency and well-being. Arriving at a consensus on which specific child welfare outcomes to define, adopt, track and analyze over time through QA initiatives, is a key decision. Appendices C, D, and E provide examples of widely accepted outcomes and outcome indicators for child welfare practice. It is important to note that that effective outcome development includes feedback and involvement from staff, customers and the community at every step of the process. Outcome Indicators Outcome indicators are not the same as outcomes. An indicator helps assess achievement or progress of the outcome. For example, if the outcome is clients will be satisfied with the service, one indicator could be that 85% of clients state they are satisfied with the service. In other words, each outcome needs to be translated into one or more indicators. Indicators describe (often as a number and/or percentage) observable and measurable characteristics or changes that represent the degree of achievement of an outcome. The criteria applied will determine the selection of the outcome indicator. The following criteria should be considered when selecting indicators (Hatry, 1999) Relevance to the mission/objectives of the program and to the outcome the indicator is intended to measure Importance to the outcome it is intended to measure Understandability to the users of what is being measured and reported Span of Program Influence or Control with the intent not to avoid measuring important outcomes Feasibility of collecting reasonably valid data on the indicator Cost of collecting the indicator data Uniqueness of the indicator in relation to other indicators Inability to Manipulate data by program personnel Comprehensiveness of the set of indicators in the identification of all possible effects ontario association of 19