BOOK REVIEWS / COMPTES RENDUS DE LIVRES

LA REVUE The Canadian CANADIENNE Journal of D'ÉVALUATION Program EvaluationDE Vol. PROGRAMME 19 No. 1 Pages 179 181 ISSN 0834-1516 Copyright 2004 Canadian Evaluation Society 179 BOOK REVIEWS / COMPTES RENDUS DE LIVRES Thomas R. Guskey. (2000). Evaluating Professional Development. Thousand Oaks, CA: Corwin Press. 328 pages. Reviewed by Jill Chouinard The need for educational organizations to evaluate professional development activities is mounting. Cuts in education go ever deeper and threaten already meagre professional development budgets, and the imperative for reform and improved student learning outcomes echoes through the halls of all public schools. Guskey s book, Evaluating Professional Development, provides a timely and much needed contribution to the field of professional development evaluation. However, while it does provide a comprehensive and detailed account of professional development and evaluation practices within the broader perspective of school reform efforts, Guskey s adaptation of Kirkpatrick s four-level training evaluation model does not successfully capture the ethos of organizational change, the conditions necessary to enhance organizational learning. Despite this shortcoming, there is much in Guskey s book that is noteworthy. Evaluating Professional Development is well organized, with early chapters providing comprehensive descriptions of professional development, evaluation, and the link between professional development activities and student outcomes. Chapters 4 through 8 provide detailed accounts of Guskey s five-level model of evaluation (co-developed with Dennis Sparks). The remaining chapter describes who should conduct evaluations and provides recommendations for presenting evaluation results. Each chapter ends with relevant Questions for Reflection as well as sample evaluation forms and checklists. What sets Guskey s book apart from the plethora of available literature on professional development evaluation is the inclusion of context as a key variable in determining the effectiveness and success of professional development activities. Professional development is not depicted as a discrete event, but as a systematic, intentional, and ongoing process designed to bring about organizational change. Corresponding author: Jill Chouinard, 72 Harmer Avenue, Ottawa, ON K1Y 0T8; < jillanne@look.ca>

180 THE CANADIAN JOURNAL OF PROGRAM EVALUATION While the literature on school reform makes a similar claim about the relationship between professional development and school improvement, Guskey describes a complex relationship between organizational variables and professional development outcomes. This model thus becomes the critical foundation upon which professional evaluation is built. Guskey s five-level model of evaluation, based on Kirkpatrick s fourlevel model of training evaluation (reaction, learning, behaviour, and results), compensates for a significant shortcoming in Kirkpatrick s framework by including an additional level designed to measure organizational support and change. This level highlights the importance Guskey places on organizational context as a contributing factor influencing individual learning and learning outcomes. According to Guskey, organizational variables are key to the success of any professional development activity. They also can hinder or prevent success, even when the individual aspects of professional development are done right (p. 83). Thus, evaluating organizational context has the potential to greatly enhance the validity and usefulness of professional development evaluations and, most importantly, improve the implementation of professional development activities. Kirkpatrick s training evaluation model was designed to evaluate the effectiveness of a discrete training program unrelated to broader organizational change efforts. Kirkpatrick did not include climate as a contributing variable to be considered in measuring training effectiveness, though he did refer to climate conduciveness as a precondition of change (Kirkpatrick, 1996). Guskey provides a more systemic model of professional development and evaluation that includes organizational context in the evaluation process. But he fails to articulate a more inclusive or participatory evaluation process that might have helped integrate evaluation practices with ongoing learning in organizations. What is missing from this book is a more vigorous description of who could be involved in carrying out the evaluation. Guskey s approach to stakeholder participation is limited to professional development activities (design and implementation), and not the evaluation itself. By limiting stakeholder involvement, Guskey forgoes many organizational and individual benefits that enhanced participation in evaluation could produce. Involving teachers and other key decision makers in the evaluation of professional development activities has the potential to enhance individual and collective learning, both of which are essential features of successful implementation efforts.

LA REVUE CANADIENNE D'ÉVALUATION DE PROGRAMME 181 Despite the passage of time, Kirkpatrick s legacy continues to influence the professional development evaluation literature. While there have been a number of variations on Kirkpatrick s four levels, most fail to elaborate a role for key stakeholders in the evaluation process. Why stakeholder-based evaluation continues to remain outside of training evaluation practice, despite the extensive documentation of the role of collaboration in educational reform, is worthy of further inquiry. The current climate of educational change, however, may well force evaluation and professional development practitioners to revisit how they evaluate professional development activities and may in fact force them to look outside of the training evaluation literature for solutions. Despite these shortcomings, Evaluating Professional Development should be required reading for anyone who works in the field of professional development, and for anyone interested in implementing and evaluating professional development activities. Guskey s inclusion of context as a key variable in evaluating professional development is essential for the successful implementation and evaluation of professional development within organizations. REFERENCES Kirkpatrick, D.L. (1996). Evaluating training programs: The four levels. San Francisco: Berrett-Koehler.

182 The Canadian THE CANADIAN Journal of Program JOURNAL Evaluation OF PROGRAM Vol. 19 No. 1EVALUATION Pages 182 183 ISSN 0834-1516 Copyright 2004 Canadian Evaluation Society The Joint Committee on Standards for Educational Evaluation; Arlen R. Gullickson, Chair. (2003). The Student Evaluation Standards: How to Improve Evaluations of Students. Thousand Oaks, CA: Corwin Press. 264 pages. Reviewed by Suzanne Moreau The Student Evaluation Standards, established by 16 professional education associations, identifies evaluation principles that should result in improved student evaluations. The publication claims that sound student evaluation practices should possess four attributes or standards : propriety (considers legal and ethical issues, and the well-being of the students being evaluated); utility (considers usefulness, specifically that evaluations are informative, timely, and influential); feasibility (considers that evaluations can be implemented and are practical, diplomatic, and adequately supported); and accuracy (considers that evaluations will produce sound student performance information that leads to valid interpretations, justifiable conclusions, and appropriate follow-up). In British Columbia, a new school planning process was introduced in 2002/2003 to replace the former school accreditation program, with the primary goal of improving student achievement through more direct and meaningful parental involvement. As a pioneering parent member in a School Planning Council (SPC), and a program evaluation advisor by profession, I hope to connect the improvement of student evaluation practices with improvement in student achievement. Several issues affect student achievement at our school. Primarily, communication is the key to successful assessment for all involved, improving teachers ability to make defensible evaluations, parents capacity to make informed decisions, and, ultimately, students ability to achieve to their fullest potential. More specifically, the issues are: timely and comprehensible communication of a student s achievement with follow-up by teachers, the student, and parents; and consistent student evaluation practice among teachers in relation to provincial standards and broad expectations, so that students progress can be measured and acted upon with confidence. If our Corresponding author: Suzanne Moreau, Senior Evaluation Advisor, Ministry of Attorney General and Ministry of Public Safety and Solicitor General, Policy, Planning and Legislation Branch, Corporate Planning Division, Box 9263, Stn. Prov. Govt. 10th Floor, 1001 Douglas Street, Victoria, BC V8W 9J7; <Suzanne.Moreau@gems4.gov.bc.ca>

LA REVUE CANADIENNE D'ÉVALUATION DE PROGRAMME 183 SPC plan addresses student assessment practices, use of the Standards could have a positive and direct impact on student achievement at our school. There was a positive response when the school principal excerpted sections for our Council s planning process an indication of his acceptance of the Standards in our endeavour. While all four standards are important to consider in the planning process, of particular import for the SPC are the utility standards, which set out explicit values that are directly related to the school planning process and address problems inherent in ineffective communication and inconsistent practice. The school planning process sets goals that are linked to district goals and an accountability contract with the Ministry of Education. The practice of setting annual goals linked to a mission statement typically involves setting performance targets in an inclusive and collaborative manner, specifically involving administrators, teachers, parents, students, and members of the community. The following three utility standards are especially relevant to the planning process. Under U5 Explicit Values it is recommended that values are clarified and justified from all who have a role in shaping student evaluation practice, including parents and students. In addition, the frame of reference for interpretation of student evaluation data must be described and justified, specifically performance in relation to preset standards, peers, aptitude, and amount of improvement or amount learned. Second, under U6 Effective Reporting it is recommended that student evaluation reports should be clear, timely, accurate, and relevant so that they are useful to students and their parents. Specifically, prepare a school policy to guide the reporting system that ensures adherence to a consistent assessment and reporting scheme and avoids jargon, that is, it explains what letter grades mean and how they should be interpreted and used. Third, under U7 Follow up it is recommended that student evaluations include a follow-up procedure to improve understanding. Specifically, inform students about what will be assessed, and how the evaluation information will be used to identify strengths and weaknesses and used to design appropriate follow-up actions for students, parents and teachers. In summary, The Student Evaluation Standards has the potential to add value to the new school planning process in British Columbia. I hope we realize this potential at our school and that we may ultimately form a promising practice with broader application of these Standards at the district and provincial levels.

184 The Canadian THE CANADIAN Journal of Program JOURNAL Evaluation OF PROGRAM Vol. 19 No. 1EVALUATION Pages 184 185 ISSN 0834-1516 Copyright 2004 Canadian Evaluation Society David Gremblowski. (2001). The Practice of Health Program Evaluation. Thousand Oaks, CA: Sage Publications. 323 pages. Reviewed by Anna Byszewski As health care expenditures rise worldwide, health program evaluation is becoming more of a necessity. Gremblowski writes in an American context, in the dual environment of both private and public health care where the need to demonstrate cost effectiveness is central. Nevertheless, he makes a vital addition to the literature on the topic, demystifying a complex area while providing detailed descriptions of methodologies and instructions for avoiding potential threats to validity. His first intended audience is health program evaluation students; the second is program administrators and decision-makers. The text is framed as an easy to follow three-act play. Each chapter provides practical examples, with a summary section of key points, study questions, and detailed references. Gremblowski stresses that his is not a cookbook approach, and the road may be circular. Act I lays out the process of asking questions in the political realm. Scene 1 involves examining overt reasons, such as whether to continue or discontinue the program, and how to improve the program. The covert reasons include delaying decisions around the program or legitimizing it. This helps to develop a general question, which is often Did the program achieve its objectives? followed by Why is this the case? In Scene 2, the evaluator helps to translate a question into specific evaluation questions, with a good section on how to decide on questions and design. There is a discussion about the choices of prospective (ideal) versus retrospective designs. Act II focuses on answering the question via specific methods. This section introduces two major types of evaluations done in health programs: impact evaluations in which outcome/causality is central, and process evaluations (program implementation). Chapter 4 focuses on the former, and provides a menu of designs: experimental (RCT), quasi-experimental, and pre-experimental. Threats to the internal and external validity are described for each design, as well Corresponding author: Dr. Anna Byszewski, Associate Professor of Medicine, Faculty of Medicine, University of Ottawa; Medical Director, The Geriatric Day Hospital, Civic Campus, 1053 Carling Avenue, Ottawa ON K1Y4E9; <abyszewski@ottawahospital.on.ca>

LA REVUE CANADIENNE D'ÉVALUATION DE PROGRAMME 185 as measures to avoid these pitfalls. The author goes to great lengths to facilitate and deepen the reader s understanding of this area. Chapter 5 deals with cost-effectiveness analysis, an area that has probably received more attention in the American health program evaluation system than in Canada. The author insists that cost evaluation analysis is not a decision-making technique, though I would argue that such results might inevitably lead to judgement of program worth. Gremblowski suggests that program implementation evaluation can be more challenging than impact evaluation. Different implementation designs may be required, customized or created from scratch, using both quantitative and qualitative methodology. A useful and innovative framework is provided, a 2 X 2 table using the concepts of purpose (descriptive or comparative) and timing (cross-sectional or longitudinal). Excellent tables provide numerous examples of the types of questions that can be utilized. Chapters 7 through 9 deal with population sampling, measurement, and data collection and data analysis. These are excellent and exhaustive, similar to other evaluation texts, but presented from a health care perspective. Gremblowski concludes with Act III, in which the evaluator s major role is to return to the political realm and engage in results dissemination and use of answers in decision-making. It deals with practical points of a dissemination plan and developing recommendations in the form of a comprehensive report, executive summary, or short report. There are helpful hints in the form of actual examples of slides and overheads for presentation reports. In Gremblowski s view, the evaluator is the key expert who orchestrates the evaluation plan. Although negotiation with stakeholders is alluded to, this aspect might have been explored further. In today s health care scene, interdisciplinary teams make up most programs, and their involvement is crucial to the production of meaningful and useful findings. Future editions of this book could be expanded to include sections on this topic and to guide the reader in selecting who should be involved. Overall this text provides a comprehensive and practical text for health care evaluators, and is a significant addition to the library of evaluation. It is clearly written and well researched, and should be strongly recommended for curriculum in the field of health program evaluation.