Facilitating Praxis: Exposure to Authentic Evaluation for Students as Higher Education Consumers

Transcription

1 Facilitating Praxis: Exposure to Authentic Evaluation for Students as Higher Education Consumers Alison Moore 1 1 Empatheia Consulting Abstract This paper explores the benefits derived and challenges encountered for students and university staff alike when real life, problem based learning is part of a postgraduate evaluation subject. This examination acknowledges concern in recent times by purchasers of evaluation services; practitioners; and educators that core competencies to deliver consistently good evaluation practice are vital. Students enrolled in Australian university postgraduate courses in evaluation may be exposed to reallife scenarios. They can be expected, as part of the successful completion of the course, to deal with stakeholders and take an active role in problem solving. This requires them to deal with field based personnel and activities each with inherent tensions and contradictions. Such complexity may be far beyond their initial expectations of the course. Likewise lecturers face challenges in delivering applied courses. Not the least of these challenges is that the course must satisfy academic standards of inquiry and innovation, while matching industry expectations. Industry collaborators, although keen to offer students exposure to real life evaluation, may themselves lack sufficient resources to adequately respond to students within the constraints of a 13 week semester. Moreover, the agreement to partner with a university evaluation course coordinator, very often forged much earlier in the year, can be difficult to honour months later, as organisational and client priorities shift. Evaluation course design and delivery at a leading Australian university is examined, together with the responses from both lecturers and students enrolled from more than 20 different nations. Findings are outlined, together with implications suggested for evaluation scholarship and practice at a time of transition, when courses such as the one under examination are being re designed, replacing face to face with on line delivery. Introduction Today, program evaluation can be found in most government and non government sectors around the world, while businesses equally want to know if their investments are being maximised. Wherever funds are spent it is anticipated that positive results will be achieved, with measurable benefits for those communities and individuals involved. This change, the stakeholders affected and by what degree, are the core concerns for evaluation. 1

2 The evaluation literature and practice has expanded, with a variety of frameworks and models and a large body of research work. Currently there are more than a dozen journals dedicated to the topic. Evaluation societies and associations continue to be formed. They have existed in Australia, the United States, Canada, the UK and Europe for decades but others have formed more recently including in Latin America, Asia Pacific including Malaysia, and in Africa. By last count there were 75 professional evaluation societies and/or associations. Program evaluation as a profession is consolidating a positive reputation, attracting skilled personnel working in sectors as diverse as agriculture, health, resources and community development. Evaluation scholarship rooted in praxis Evaluation is by nature a vocational discipline. The proliferation of professional societies reflects an increasing accent on improving the practice of evaluation across the globe. My own evaluation practice includes a critical social science perspective a fascination in how relationships, collaborations and knowledge sharing occur. Investigating the taken for granted ways of thinking, understanding how a situation comes to be and how it might be improved. This gives rise to praxis, or as Carr and Kemmis describe, practical, thoughtful doing (1986, p.132) or as Schwandt (2005) describes as concerned with the competence, sensibility and sensitivity demanded in knowing what is right to do and good to be in a particular situation, given the situation s unique circumstances, peculiarities, contingencies, and demands (2005, p. 326). As a lecturer and practitioner in evaluation, I am keen to engage in praxis, exploring the benefits and challenges encountered for students and staff alike when real life evaluation scenarios are commissioned and completed as part of a postgraduate course at an Australian university. I am also interested in the degree to which this authentic learning, in which students engage, delivers them relevant skills and competencies when they either begin or return to professional life. Evaluation course delivery at higher education institutions A brief review of Australian university course profiles for reveals at least five universities are currently offering courses which teach the principles of evaluation theory and practice, at postgraduate level. It is not known if undergraduate programs offer any course in this subject area. The postgraduate courses on offer include two unit courses which contribute to a Masters degree in disciplines including international or participatory development; development practice; and human services or in one case only, a complete Masters program in evaluation. Less clear is the extent to which these courses make authentic evaluation practice part of the course design, with the exception of the course on which this paper concentrates. The intention of this paper is to investigate the extent to which the course as delivered with a real life evaluation component, delivers competencies for evaluator practitioners in the field and to reveal the importance ascribed to the evaluation practice component of the course, now that graduates have begun or returned to professional life graduates. 2

3 Course design This course delivery reflects a long history of association with industry to provide students real life experiences in evaluation. This paper concentrates attention on the course as it was delivered during , a period in which it required students to complete an actual evaluation of a program or project, negotiated in advance between the course lecturer and an industry party. The fundamentals of evaluation are at the core of the course, with a particular focus on social programs and students engage with industry personnel to complete an actual evaluation of an existing program. The concepts and skills sufficient to plan, conduct and communicate evaluations are covered. The use of program logic is tested while various methods for data collection and analysis are practiced. Contemporary issues of ethics, rigour, the political nature of evaluation, stakeholder negotiation, communicating evaluation findings and the construction of useful and relevant recommendations for change are discussed. Industry negotiation occurs before semester to secure appropriate programs for which students will apply the skills learnt during the course. Students work in groups to complete the program logic; an evaluation plan; the evaluation; briefings and reporting to industry partner. Just as Yeoman (2012) argues effective postgraduate course design places students at the centre of the learning process, the strategy for this course uses an action learning approach modelled on Kolb s (1986) learning cycle, which describes the experiential and reflective nature of adult learning as described in Figure 1 below. Figure 1: Action learning cycle (Kolb 1986, McGill and Beaty) This reflective process is one in which small groups negotiate a problem in a real or simulated environment (McGill and Beaty, 1995; 2001). Action learning is an appropriate approach when as they suggest, problems are, ambiguous or need negotiation and when there is no right answer and multiple solutions are available (1995, p 297). The 13 week semester university teaching cycle does constrain students, especially in terms of the amount of time available to negotiate within their own learning groups, build rapport with the industry client and for data collection. The industry partner may not see the immediacy of the task and both data and stakeholders may be difficult to access. Students are challenged too with the necessity of completing set assessment tasks while dealing with uncertainty. They may struggle to deal with the politics inherent in any evaluation, while student group dynamics may not always result in an equal division of tasks and individuals may not be comfortable with the model of working and learning within a group. 3

4 Collaborative learning is critical to completion of course elements and consistent with Yeoman (2012), the lecturer s role is a facilitator of learning, rather than a single point of knowledge. Students are expected to draw effectively upon their own and fellow students work and life experience, to link evaluation concepts and approaches covered in class to the assessment tasks. As he describes, collaborative learning entails the shifting of the focus of authority from the traditional teacher to the learning community and from a firm commanding classroom structure to a negotiated and fluid relationship between the members of the learning community. Hence, the role of the teacher is more of a facilitator rather than a teacher. The students in essence become responsible for their own learning and each other s (2012, p.307). Enrolled students fit a number of different profiles people who have some experience in evaluation, were intending to undertake evaluation work, or have a responsibility for evaluation studies and for those with experience in related areas such as policy development and analysis. There are also students who have not have any exposure whatsoever to evaluation theory or practice. Method An interpretive approach was chosen, acknowledging the subjective views on knowledge, practice and competency that respondents are likely to express. Also, my own subjective views and preconceptions during survey design, data analysis and interpretation should be acknowledged (Le Gallais 2008). Data is drawn from the cohort of students completing the course between 2009 and A web based survey was designed using Qualtrics software, with 10 questions including three which used either a 3 or 5 point Likert scale. The survey was conducted in mid 2013 with a purposive sample of 55. Key findings Twenty seven responses or a 50 per cent response rate was achieved. The survey participants identified themselves as having a broad range of professional interests and responsibilities. They included specialist evaluation practitioners (freelance, the non government and regional or national government sectors) in fields of health, governance, poverty reduction, church based charitable funds, education and sustainable agriculture; research analysts (university and non government); program managers; community development specialists; community engagement personnel in the resources sector; senior executive and public servants. They work in and are in some cases nationals from, every continent around the globe and many do not have English as a first language, with half of respondents currently working in evaluation. Reflecting the diversity of employment experience, 30 per cent of respondents reported more than three years professional experience in evaluation, with two of those having worked in the discipline for more than five years, compared to 62 per cent who were beginning their evaluation practice at more than one year but less than three years. Participants represented a variety of organisations including: World Bank 4

5 United Nations Development Program Wesley Mission CARE International University of Queensland Smith Family Newcrest Mining The data revealed four key themes which are now discussed in detail: Achieving competency with real life evaluation Critical learning components build capacity Industry collaboration is messy Time constraints challenge quality and delivery 1. Achieving competency with authentic evaluation Survey respondents were asked to rate the course in terms of their current work competency using a scale of: strongly agree; neither agree nor disagree; and strongly disagree. Figure 2 presents the findings for this question. In particular 82 per cent reported that completing the course gave them sufficient evaluation skills for the workplace, with the remaining having a neutral position, while 70 per cent found the real life component of the course to be useful for their current work and 30 per cent reporting a neutral view. Figure 2: Course contribution to building evaluation competencies Although not a specific objective of the course, the capacity of professionals to mentor others in evaluation practice is a laudable outcome of advanced skills building. This course promotes social interaction and participation and students are encouraged to work together for problem solving. However, when asked if the skills they have developed meant that they can confidently help others in evaluation, respondents were not universally confident in this role with 67 percent feeling confident. Statements about an evaluation exercise as a component of the course and working with industry were then described in negative terms, to test the consistency of results from the survey instrument. Overall, the results demonstrated that 85 per cent of 23 respondents either agreed or strongly agreed that a course component of a real life evaluation delivered appropriate workplace skills while 78 per cent either agreed or strongly agreed that working with an industry partner helped them 5

6 develop key negotiation skills, a competency vital for good evaluation practice. Figure 3 describes the responses. # Question Strongly agree Agree Neither agree nor disagree Disagree Strongly disagree Total Responses Mean 1 It is unnecessary to complete an evaluation as part of a postgraduate evaluation course to gain sufficient work skills in this field 2 Working with an organisation to conduct an evaluation as part of a postgraduate course did not help me to develop negotiation skills Figure 3: The importance to students of exposure to authentic evaluation for professional skill-building One participant added: Real life is what you get. The work that I did as part of [this course] was no different than previous and current experiences in the field of evaluating social programs: defining the type of evaluation based on clients needs/interest, chase the client to get information about the program, chase the stakeholders/beneficiaries to get their perspectives, the joys of timeconstraints and the challenges of team building. Other responses included: It was really exciting being part of a real life project that (we hoped) actually contributed to the organisation's reflection. And: I can't express enough how beneficial it was to learn about evaluation through a real life program. It's easy to forget theory, but first hand practical experience stays with you forever. 2. Critical learning components build capacity Overall, respondents highly rated key elements offered as part of the course. These components as shown in Figure 4 are namely: Working as a group on a real life project 93% Working with an organisation in an evaluation 85% The theoretical basis of evaluation 85% The practical frameworks and tools for evaluation practice 86% Practicing evaluation skills 96% As one respondent commented: When I took the course, I had some knowledge and experience in conducting evaluation, so the course helped me to strengthen my skills and reinforce and complement my previous knowledge. I liked the course just because of its practical component....it allows using the tools and seeing what works and what doesn't, facing the difficulties and challenges this type of process usually involves. I'm convinced the best way for us to learn is by putting knowledge into practice and the experience in the course confirmed that. 6

7 # Question Not at all Important 1 Working as a group on a real life project Very Unimportant Neither Important nor Unimportant Very Important Extremely Important Total Responses 0.00% 0.00% 7.41% 48.15% 44.44% Mean 2 Working with an organisation in an evaluation 3 The theoretical basis of evaluation 4 The practical frameworks and tools for evaluation practice 5 Practicing evaluation skills 3.70% 0.00% 11.11% 44.44% 40.74% % 0.00% 11.11% 62.96% 22.22% % 0.00% 0.00% 37.04% 59.26% % 0.00% 0.00% 33.33% 62.96% Figure 4: Relative importance of key elements in the course The practical frameworks and tools for evaluation practice and practicing evaluation skills were important to respondents knowledge and understanding of evaluation principles, and their ability to carry out an evaluation. This affirmed the approach taken in delivering the course, which included an orientation towards realistic and practical evaluation planning and execution which are of relevance to current challenges found in the workplace As one participant observed: It gives me the ability to compromise and improvise between evaluation theory and real life situation. Evaluation, [as part of the course] therefore, needs to be customised in each case to be more effective without neglecting the foundation of evaluation. By conducting real life evaluation, not only our evaluation skill has improved, but also our instinct and knowledge on what we can and should do during [any] particular situation. 3. Industry collaboration is messy Industry collaborators, although keen to offer students exposure to real life evaluation, may themselves lack sufficient resources to adequately respond to students during the semester. Moreover, the agreement to partner with a university evaluation course co ordinator, very often forged much earlier in the year, can be difficult to honour months later, as organisational and client priorities shift. Dealing with and understanding the particular characteristics of Australian industry, services sector and governance added extra complexity for many students, especially those from other countries as this respondent recalled: Being a foreign student, it took me some time to understand the context of the subject of the real life evaluation (e.g. the organisational context, the institutional landscape, the role of government, and the power and financial sources). it would be useful to understand these very early in the course. My Australian mates were able to easily get into the discussion, but it 7

8 took me awhile to catch up. Addressing this will be very helpful for the group discussion as well as when conducting the meetings and data collection with the subject organisation. 4. Time constraints challenge quality and delivery The limited time within the semester long course was an issue which for some students meant the quality of the evaluation outcome might be compromised, as these respondents described: [The] time constraint is the main problem in conducting a real life evaluation activity in this course. In my experience, some targeted respondents could not be interviewed to deepen our analysis. This will definitely impact on the quality of the evaluation report, both findings and recommendations [and] might not satisfy the clients. And In fact during the course we did not have much time conducting in depth interviews with the client then it is very difficult to say that we had enough evidence for our evaluation comments. Recommendations for improvements include these suggestions from respondents: I thought it was a tremendous experience and I learnt a lot from my group mates. In other courses timeframes to effectively work in the field are very short, which affects the quality of the results. The only recommendation I would make is have smaller groups as it is very hard to work with so many team members from so many different countries. And, [I] would like to receive a feed back from the organisation. This will give us an understanding on how useful our exercise was for the organisation in terms of adjusting and improving its performance based on our recommendations for the process and for outcomes. Conclusions Overall, the findings were clear in terms of building evaluation skills. Respondents recorded their knowledge, understanding and abilities reached a higher level of competency as a result of completing this course, one of several which are offered at Australian universities today. As one respondent recalled: I completed the [course in 2011]. [It] has been the most useful course so far and I'm not ashamed of sharing that I have had to go back to my notes of the course, whenever I [have] had deliver an evaluation proposal for a client. The evidence from this survey suggests that students were involved and interested in the topic, engaged fully in the evaluation task and undertook to collaborate with their peers in learning and practice. One respondent observed: I consider that the real life evaluation component is a singular opportunity I had to put in practice the theoretical basis and frameworks learned in class. Exercising team building and delegation of tasks; leadership and coordination skills complemented this exercise. 8

9 As courses at Australian universities are increasingly offered online, in some cases for evaluation studies, the opportunities for authentic practice such as the completion of a complete evaluation have shifted. Courses delivered online are being re designed, with consideration especially given to the reality that students in an online environment work in a different manner, compared to when enrolled in face face modes. Different assessment tasks are set to better match online teaching and learning modes. Many courses draw upon simulated work places and often incorporate extensive resources such as graphics, video, and sound files to assist in creation of authentic characteristics (Herrington and Reeves 2004). However on line delivery does not suit all students (Burke and Clark 2011) and these students, especially those with cultures of strong oral traditions can find participation on line quite difficult (2011, p.7). As one respondent argued: I believe this (the real life evaluation) is a key component of this course. I felt that the faceto face component of this course was really valuable, giving us opportunities for peer learning. I feel that I would have struggled if I had been undertaking this course as an online course and probably would not have chosen this elective if this had been the mode of delivery. The value of authentic, practice based evaluation inquiry, working with an industry partner has been generally found to be of value to graduates who are now professionals in the government, nongovernment and industry sectors, even if they are not currently actively engaged in evaluation. A challenge for the future will be to continue to provide authentic learning and practice, as universities embrace the digital world in both teaching and scholarship. 9

10 References Albrecht, J. N. (2012). Authentic Learning and Communities of Practice in Tourism Higher Education. Journal of Teaching in Travel & Tourism, 12(3), Carr, W., & Kemmis, S. (2003). Becoming critical: education knowledge and action research: Routledge. English, B. (2002). Competencies for evaluation practitioners: where to from here. Eval J Australasia, 2, Flyvbjerg, B. (2002). Making social science matter: Why social inquiry fails and how it can succeed again: Taylor & Francis. Herrington, J., Reeves, T. C., Oliver, R., & Woo, Y. (2004). Designing authentic activities in web based courses. Journal of Computing in Higher Education, 16(1), Holstein, J. A., & Gubrium, J. F. (2005). Interpretive practice and social action. The Sage handbook of qualitative research, 3, Holton, D., & Clarke, D. (2006). Scaffolding and metacognition. International journal of mathematical education in science and technology, 37(2), Kessels, J. W., & Poell, R. F. (2004). Andragogy and social capital theory: The implications for human resource development. Advances in Developing Human Resources, 6(2), Kincheloe, J. L., & McLaren, P. (2002). Rethinking critical theory and qualitative research. Ethnography and schools: Qualitative approaches to the study of education, Kolb, D. A. (1984). Experiential learning: Experience as the source of learning and development (Vol. 1): Prentice Hall Englewood Cliffs, NJ. Le Gallais, T. (2008). Wherever I go there I am: Reflections on reflexivity and the research stance. Reflective Practice, 9(2), MacGill, I., & Beaty, L. (1995). Action learning: a guide for professional, management and educational development. McGill, I., & Beaty, L. (2001). Action learning: A guide for professional, management and educational development: Psychology Press. Patton, M. Q. (2005). Qualitative research: Wiley Online Library. Schwandt, T. A. (2005). The centrality of practice to evaluation. American Journal of Evaluation, 26(1), Yeoman, I. (2012). Authentic Learning: My Reflective Journey With Postgraduates. Journal of Teaching in Travel & Tourism, 12(3),