Process Use of Evaluations Types of Use that Precede Lessons Learned and Feedback

Size: px
Start display at page:

Download "Process Use of Evaluations Types of Use that Precede Lessons Learned and Feedback"

Transcription

1 Evaluation Copyright 2002 SAGE Publications (London, Thousand Oaks and New Delhi) [ (200201)8:1; 29 45; ] Vol 8(1): Process Use of Evaluations Types of Use that Precede Lessons Learned and Feedback KIM FORSS Andante Tools for Thinking, Sweden CLAUS C. REBIEN COWI Ltd, Denmark JERKER CARLSSON * Among practitioners, it is often said that there is much that is useful that takes place during the evaluation process. The literature on the utilization of evaluation, however, mostly refers to lessons learned and recommendations implemented after the evaluation. This article seeks to contribute to the debate by exploring further the process use of evaluations the evaluation use that takes place before lessons learned are generated and feedback processes are initiated. Five different types of process use are identified: learning to learn; developing networks; creating shared understanding; strengthening the project; and boosting morale. Most resources in an evaluation process are spent in preparation and implementation. Very little is spent on feedback. A conventional way of improving evaluation use would be to shift resources from the preparation and implementation stages to the feedback stage. The article argues that in general, shifting resources from preparation and implementation to disseminating results would jeopardize the quality of the entire evaluation because it would ceteris paribus reduce the design, data collection and analytical quality. Instead it is argued that evaluation commissioners and evaluators should work explicitly to increase process use as the most costeffective way of strengthening the overall utility of an evaluation. KEYWORDS: empowerment evaluation; evaluation utility; learning; organizational change; participatory evaluation * Professor Jerker Carlsson was a good friend, dedicated colleague and inspiring coauthor of the present article. It is with great sadness that we the two co-authors honour his memory after he passed away at a much too early age in the autumn of

2 Evaluation 8(1) Introduction The literature on the utility of evaluation mostly refers to the end result of evaluation: the conclusions and recommendations in the report. In this article the intention is to explore the process uses of evaluation, defined as the utility to stakeholders of being involved in the planning and implementation of an evaluation. Indeed, members of organizations testify that the actual process of preparing for and being involved in an evaluation can be very helpful (Patton, 1997, 1998). The end report may not be utilized, but the evaluation may still have been effective, e.g. it may change management thinking about future options; mobilize staff around a course of action; or reinforce the impact of a programme of social change. An evaluation may become part of the treatment, rather than just being an independent assessment of effects. Purpose To explore the subject further it is necessary to develop a typology of different types of process use. These would be different from the usages often mentioned in literature (decision making, learning, control and conceptual use, etc.). Furthermore, we will explore how common different types of process use are, and which activities and institutional contexts they appear in. The purpose can thus be summarized: 1. to develop a typology that describes and analyses the process use of evaluation; and 2. to generate hypotheses regarding the context and activities relating to the process use of evaluations. Method The development of the typology in this article is based on the three following methods of research. Firstly, it is based upon a comprehensive review of the literature on evaluation use and on studies and experience with evaluation processes within specific organizations. Secondly, it is based upon two specific case studies concerning the Nordic Development Fund (NDF) and Swedish Museums. Staff members were interviewed about the benefits and problems concerned with their involvement in the evaluation to find out whether this was a worthwhile activity, or whether the evaluation would be equally effective if out-sourced. A careful analysis was carried out to ascertain what the staff members actually learned from the evaluation, and what benefits resulted from their involvement. There was a deliberate attempt to verify whether the patterns of process use suggested by Patton (1997 and 1998) were present or not. 1 Thirdly, in-depth analyses were carried out based on case study methodology of three evaluation processes with which the authors have been involved. 30

3 The Emerging Interest in Process Use Forss et al.: Process Use of Evaluations Utility is a prime virtue of evaluations, indeed it is the first criterion of evaluation quality mentioned by the Joint Committee on Standards: The utility standards are intended to ensure that an evaluation will serve the information needs of intended users (Sanders, 1994). Several manuals and handbooks urge evaluators to be practical and utilization-focused (Patton, 1978, 1982, 1997; Pawson and Tilley, 1997; Torres et al., 1996). Among the design features that will enhance the utility of an evaluation are the involvement of stakeholders in many aspects, such as identifying the problem, gathering data, and the design of a strategy to communicate the results. Nevertheless evidence from the field suggests that many evaluations are not utilized (Carlsson et al., 1999; Rebien, 1996; Thoenig, 2000). The institutional links between evaluations and decision making are weak (Cracknell, 1990) and it is difficult to prove that evaluations improve organizational learning (Samset et al., 1992). It is commonly believed that most evaluations end up gathering dust on shelves, and a number of scholars and critiques are raising the question: do we need all these evaluations? A more sophisticated analysis of evaluation utility may argue that evaluations can be used in many different ways. Even if the recommendations are not necessarily followed immediately, there might be other, more long-range effects. The knowledge generated in any programme context may be applied in a totally different context many years later. Hence, those who trace learning and utility with respect to evaluations confront a difficult task. People often forget where the knowledge and experience that they apply in decision making actually come from and if a past evaluation played a role, that role is likely to have been forgotten. The discussion of evaluation utility belongs to a wider area of how information is disseminated and utilized in a social system (McMaster, 1996; Eden and Spender, 1998). There is a similar discussion with respect to the links between basic research, applied and development research, and the commercialization of innovation. Institutions involved in agricultural research are seeking ways and means to measure the impact of research findings on the practices at farms, through the process of reaching out to a dissipated system of individual and independent farmers, which is challenging (Forss, 1999). So, an evaluation can be used in many different ways. Vedung (1997) suggests that an evaluation can be used in seven different ways: instrumentally, conceptually, legitimizing, interactively, tactically, ritually, and as a process. The legitimizing and ritual uses of evaluation can from a rational decision-making point of view be called illegitimate; they have little to do with the search for knowledge, and recommendations for action based on new findings. Instrumental, tactical and interactive uses imply strategies to share results (or withhold some results) and engage in discussion on future courses of action with respect to the evaluation object. The conceptual use relates strongly to learning, i.e. changing a mental framework, which may or may not influence future thinking and decision making. With respect to each of these different uses there is a field of reference: an established framework of books and articles. 31

4 Evaluation 8(1) Vedung argues that utilization must not necessarily take its starting point from the existence of a report. An evaluation report as such is not strictly necessary. Though many may see the word evaluation as synonymous with a mode of documentation a report this is a convention which is not supported by theory. The common definition of evaluation is a systematic inquiry into the worth or merit of an object (Sanders, 1994). The reader will note that this definition describes a process, an activity. Whether the activity will be documented and communicated via a written report is not a necessary requirement. Marra (2000) presents a good summary of how networking and the use of multiple channels of communication strengthen the use of evaluation findings. During their own work, the authors have found that people have often said that the findings were not particularly surprising; the effects or impact of the evaluated project were known about or at least the results the evaluators found had been entertained as the possible effects. Recommendations presented by evaluators are very often within the scope of what people in the evaluated organizations would naturally consider as possible courses of future action. So given this predictability, what do they mean when they say it was useful to go through the process? Patton (1997) defines process use as the kind of usage that: is indicated by individual changes in thinking and behaviour, and program or organizational changes in procedures and culture, that occur among those involved in evaluation as a result of the learning that occurs during the evaluation process. (1997: 90) Patton s definition is meant to capture the distinction between learning that occurs as a result of the findings that are reported in the evaluation, and the learning that occurs during the process, irrespective of what is said in a report. But, as a definition, it is not without problems. It would actually be quite hard to verify process use, if the concept is not further explained, or pinned down with more specific examples. Patton (1997: 91) briefly defines four primary kinds of process use: 1. enhancing shared understandings, especially about results; 2. supporting and reinforcing the programme through intervention-oriented evaluation; 3. increasing participants engagement, sense of ownership, and self-determination; and 4. programme or organizational development. The categories throw light on many of the issue of process use, but it is not always clear what distinguishes them from the use of findings. In the categories mentioned, it may seem that process use is closely intertwined with sharing the findings of an evaluation. In a follow-up to the above distinctions, Patton (1998) suggested additional types of process use, referred to as learning to learn, qualitative insights, and possibly goal displacement. Patton (1998) does not actually speak of categories of process use in the same rigorous sense as in the previously referred book (Patton, 1997). Patton s work touches on essential questions, and there is a need to further develop the typology, and explore varieties of process use. Yet another issue for 32

5 further exploration relates to the universality of process use. Can process use, however we define it and whichever subcategories we distinguish, be found on all and every type of evaluation? Is it even desirable to pursue process use in all evaluation settings? Assuming that some evaluations are initiated primarily for reasons of control and audit, would that diminish the potential for process use? Is process use mainly something that can be pursued with respect to participatory evaluation, or so called empowerment evaluations? Is it important to discuss the purpose of the evaluation before we start discussing how process use can be strengthened? In the analysis of the empirical examples below these issues are discussed. The Various Ways to Use an Evaluation Process Forss et al.: Process Use of Evaluations In the study concerning the NDF, we identified the following five types of process use. Learning to Learn Patton (1998) wrote that the evaluation field has its own particular culture, building on norms and values that evaluators take for granted, but which may be quite alien to people embedded in the culture of another profession. Patton (1998: 226) suggests that these values include clarity, specificity and focusing, being systematic and making assumptions explicit, operationalising programme concepts, ideas and goals, separating statement of fact from interpretations and judgments. It could of course be argued that in modern organizational settings, these values are quite universal; however, it is one thing to subscribe to a value in theory, quite another to act according to that particular logic. Even if values such as those mentioned by Patton are subscribed to by many, most would find that working life offers limited opportunities to act according to these values in practice. The values of evaluation are values conducive to enquiry. These values support a structured way of thinking about reality and generating knowledge. There are many ways of generating knowledge; such as with the help of guesswork, supernatural insight, divine inspiration, and transcendence. Historically, there can be no doubt that the application of a scientific method is the cheapest and richest way of generating knowledge, and the systematic conduct of an evaluation is a superior way of learning. To engage in evaluation is thus also a way of learning how to learn. Within the vast amount of literature on evaluation theory there are several articles on how to learn from evaluation. Patton (1998) can be read to suggest that this is a misguided perspective evaluation is actually synonymous with learning; it is the process which generates insight and knowledge. Failure to learn from evaluation actually only refers to failure to disseminate the findings, to organize the consumption. Those who learn from evaluation are those who do the evaluation, very few others learn to any significant extent. But those who do learn, may learn quite a lot, and they do so in the process of inquiry. The staff members of NDF unanimously testified to the value of learning to learn, to organize a systematic inquiry, to patiently search for opinions, facts, 33

6 Evaluation 8(1) evidence of impact etc. They learnt to assess responses, to search for explanations regarding why people profess certain opinions about a programme, and how that relates to self-interest. They learnt to seek information from supplementary sources, to crosscheck for validity and reliability. To engage in evaluation can thus be a way of strengthening organizational learning something which is very much in fashion. Developing Professional Networks NDF staff members often mentioned that being part of the evaluation allowed them to meet others whom they otherwise had little interaction with. They referred to clients, competing organizations, stakeholder groups, as well as members of their own organization. The importance of this is partly due to the peculiar nature of NDF financing. The NDF provides finances for part of a loan agreement, where other financiers play a dominant role. When a financial package is put together, it is important to have personal contacts. These lower transaction costs, by establishing trust and facilitating communication. In practice, members of staff of the different banks have a heavy workload and they seldom meet unless they have to, and then under severe time constraints. The evaluations conducted by NDF personnel often meant that they had to visit the other banks, as well as ministries of finance in borrowing countries, peers in other organizations, researchers, etc. Many of the personnel met during the conduct of the evaluation turned out to be useful contacts for other occasions. The next time a loan is negotiated, there is a face to connect to the mail address. Also, in the course of ordinary duties, people need information, and the new contacts added in the process of evaluation are possible sources of knowledge. In the present information society, building and maintaining a relationship are of vital importance for professional success and similarly an organization would need members with extensive networks (Peters, 1994). Evaluation appears to be a way of rapidly interacting with many people, and hence an opportunity to build networks. Creating Shared Understandings In the process of analysing the impact of a number of museums, the authors distinguished between the following four different roles of a museum. Acquiring and keeping things of the past; Maintaining these in proper conditions for future generations; Conducting research on the use and meaning of items in the museums collections; and Serving the general public with exhibitions, tours, displays, lectures and the like. Different personnel categories were found to have quite different priorities. Whereas curators saw the museum s main function to be in the building and maintenance of collections, the political leadership on museum boards often wanted to give priority to using the collections for public purposes. When considering how to spend an additional sum of money or choosing where to make budget cuts, 34

7 Forss et al.: Process Use of Evaluations they had quite different priorities. The evaluation made these differences explicit. Naturally, this did not take away dissent, but it may have led to a more successful arbitration of interests. It helped people understand each other s motives, and to some extent also to respect the differences. This communication and the mutual understanding which developed were quite unrelated to the thrust of the evaluation report. However, there are many different aspects of how communication influences evaluation work, internally on evaluation teams as well as between the evaluator and various stakeholders. It would seem that the usefulness of evaluation hinges directly upon the quality of communication in evaluation exercises, and that such communication requires skills and evaluator experience. It is therefore worthwhile discussing the precise content of such skills and how they can be developed. A comprehensive discussion is beyond the scope of this article but it is suggested that the skills include intangible qualities such as cultural sensitivity and applying what in German is termed Fingerspitzengefühl. This term describes being sensitive to any given situation in which the evaluator may find him or herself, and to how others may react to what one says or implies. Regardless of how intangible these qualities may be, the consequence of not having them will rapidly become manifest. The best way of acquiring such skills or personal qualities is through experience. Evaluation is defined by its practice. And evaluators are best trained through apprenticeship, i.e. through being involved as team members in evaluations with team leaders who can act as role models and mentors, and who can gradually expose the team member to ever increasing challenges. In addition to the apprenticeship model, the management training industry and business schools run many courses, seminars and entire degrees in which one can acquaint oneself with the theory of intercultural skills and communication. The purpose of this should be clear: to aid good communication and avoid misunderstandings, and to promote interaction and exchange of knowledge and experience. In addition to the individual being better equipped in terms of communicative skills, it is important, however, that structural barriers to communication are kept at a minimum. The usefulness of the evaluation tool in the future also hinges on its ability to provide timely and easily accessible answers to questions that must be answered in order for a project, a programme or an organization to progress. The tendency towards increasingly complex evaluation designs, with multiple teams and lasting for long periods of time, does not exactly facilitate communication, nor does it promote the usefulness of the evaluation tool in terms of being able to provide input to change processes. Strengthening the Project Evaluations are often undertaken to assess past performance, to analyse whether goals have been reached or not. In reality, the projects that are financed as specific organizational frameworks for action are embedded in organizations where many other activities occur at the same time. When the evaluator appears on the scene, he or she may often find that people are reminded of the project being evaluated. 35

8 Evaluation 8(1) During the process of an interview, what the project was (and is) about will be re-examined by the interviewees. The result is that they come out of the meeting with the evaluator with a clearer understanding of the project, and possibly with a new resolve to achieve the project s aims. Irrespective of what the evaluators write in their report, the purpose the project was meant to serve may thus have been strengthened. This effect is encountered particularly often in evaluations of technical assistance and of education. In a typical example, managers who have been through a short training programme come back to work and are immediately overwhelmed by routines. The ideas and the resolve to use them learnt during the training are soon forgotten. However, a meeting with an evaluator, who follows up the impact of training, can serve to remind them of their intentions to put the training to use, and if undertaken by a skilful evaluator, these intentions may actually be strengthened. Management training programmes often include an exercise whereby the participants have to think of themselves as change agents, and set targets for things they want to accomplish when they are back at work. They are often visited by some of the faculty six months later, or they reconvene with the whole course, to assess what they have achieved. Through such processes two birds are killed with one stone: evaluators get prime data on the changes brought about by the programme, and the risks that the programme is forgotten once the participants are back at work are reduced. Defining, gathering and analysing evaluation data become synonymous with strengthening the impact of the training programme. Patton (1998) calls this evaluation as an intervention ; the evaluation becomes an intentional intervention supporting programme outcomes. It is apparently more common in many social and health-related programmes. Here a follow-up meeting with a patient, a beneficiary or a person who has been through some kind of social programme, is taken as both an opportunity to collect information, a reminder of the results that are expected, and possibly a point at which to make new decisions. Collection of data is built into the programme process in ways that also strengthen the achievement of objectives. There is yet another kind of process use that strengthens the project, but which occurs through a different mechanism. Much of professional work today occurs under considerable time constraint. The productivity increases that have been witnessed over the past few decades have come at the cost of redundant time. As redundant by definition is redundant, the cost may appear low. However, general systems theory advises that redundancy is a way of adding variety in a system, and of increasing flexible adaptation (Morgan, 1986). One of the things that happens during an evaluation is that the work flow or whatever process is being evaluated is interrupted. Imagine an evaluation concerned with the impact of a programme to develop municipal democracy where the evaluators are a team of political scientists. During the course of the evaluation, the evaluation team will interact closely with politicians and bureaucrats. The way the team formulates questions, discusses activities and listens to experiences, may influence activities at the project level. 36

9 Forss et al.: Process Use of Evaluations Evaluators do not necessarily share their wisdom and experience; the process is rather more complex. It involves concentrated interaction with someone who is interested, who thinks sharply about the issues, and who may be ready to discuss difficult problems of resource allocation, strategic choice and responses. This is worthwhile in its own right and can lead to changes quite separate from the topics that arise out of the evaluation s terms of reference. The nature and extent of such changes may simply not be visible, or even relevant, for the evaluation report. Boosting Moral Brunsson (1989) stated that evaluation is a depressive activity. Evaluation leads us to question, to be sceptical, to reassess thus perhaps also to be distanced. Evaluators are not well known for good humour and enthusiasm, which are both essential ingredients in successful action. As Brunsson says, action thrives on commitment and enthusiasm. Evaluation, on the other hand, is an intellectual activity concerned with problems. In that sense evaluation is depressive. Practitioners who are confronted with an evaluation team may recognize these sentiments. It is therefore surprising that the staff members of NDF often mentioned that their own engagement in evaluations increased their commitment to the organization and to its activities. Taking part in evaluations generated enthusiasm and commitment, rather than the opposite. It served to boost morale. The paradox can be explained by looking at the role of the organization in relation to the activities that are evaluated. The staff of NDF were not evaluating their own work processes. They evaluated their clients, the borrowing organizations in developing countries. NDF is, in essence, an intellectual and reflective organization (in Brunsson s terminology). Furthermore, the day-to-day activities of the staff members are quite abstract and far removed from the down-to-earth implementation of projects. The evaluation process allowed them to see reality at close hand. It reminds them of the purposes they work for, and allows them to explore the relationship between their own organization and the development impact that is expected. Even though that confrontation with reality will sometimes lead to insights about the shortcomings of projects, mistakes and abuses of results, the net effect appears to boost morale. The question is whether that happens as a side-effect of other characteristics (travelling, network building, changing pace, learning to learn) or whether it is an independent aspect of process use. In practice, the effect of boosting morale is likely to be both an independent category of process use, and something which is influenced by other aspects of process use. The first reason is that organization theory, particularly the thoughts reflected in Brunsson s work and others following him, emphasize the negative, the depressive and paralysing consequences of evaluation (and related) activities. It is actually one of the major categories in thinking about the nature of organizations. Hence, to confront and question that view of evaluation, we put forward the opposite argument that evaluation is a way of boosting morale, strengthening action and generating enthusiasm. Resulting issues that need to be addressed 37

10 Evaluation 8(1) relate to whether this happens often, whether it can be designed to occur more frequently, and whether it is a desirable aspect of process use. The second reason to define this as an independent category is that even if boosting morale is a possible side effect of building networks or strengthening a programme, etc., it should also be noted that it is not the same as these processes. Morale, a supportive organizational culture, belongs to the intangible assets of an organization. It is part of the values that all agree are fundamental, but which elude description and measurement. Nevertheless, those who have entered a confused and dispirited organization will certainly recognize how tangibly different that is from a focused, inspired and committed organization. To avoid this potential aspect of process use because it is vague, difficult to understand and measure, would result in the fallacy of doing only those things that can be measured. In this context it may also be relevant to recall the Hawthorne effect. The process of evaluation may produce a similar effect. Evaluations are carried out by well-managed and professional organizations. Being exposed to an evaluation is a sign that one is part of such an organization. The fact that attention is shown, the project is investigated, viewpoints are listened to and data are collected could presumably give rise to similar positive effects as was seen in the Hawthorne experiment. Forming a Typology from the Five Types We have identified five different types of process use: learning to learn, developing networks, creating shared understanding, strengthening the project and boosting morale. 2 The first issue to consider is whether the typology is exhaustive or whether we have missed important aspects of process use. Patton mentions two other kinds of process use: qualitative insights and what gets measured gets done. Gaining qualitative insights is similar to some of our categories, namely creating shared understanding and strengthening the project. It seems to the authors that these two categories are quite different, though both build on some kind of qualitative insight. As a general phenomenon, a broad category called qualitative insight would not be distinct from learning to learn, but the latter is distinct from creating shared understanding and strengthening the project. Patton s category qualitative insights can thus be seen as a combination of two, if not three, of our categories. The category what gets measured gets done is a well-known side effect of evaluation and performance management. If it is known beforehand that an evaluation team is coming to check fire safety routines, foremen may carry out some extra fire drills, to make sure that standards are high. Therefore the evaluation will show that standards are high, and they are, but this may be due to the knowledge of the forthcoming evaluation. The fire drills replace other activities, and it is not known whether those activities would actually have been more important. It could be argued that this effect is the negative aspect of using the evaluation to strengthen the project. If the aim was to strengthen preparations for fire, then spreading the knowledge that this would be evaluated could be a way of accomplishing the effect. The fact is that an evaluation process is not neutral, it will 38

11 Forss et al.: Process Use of Evaluations reinforce and strengthen some aspects of the organization, presumably at an opportunity cost of time and money. Whether the effect is for better or for worse should be an aspect of the category, rather than a category in its own right. The next question is whether some types in the typology are redundant. Do they overlap each to such an extent that the list should be reduced to three or four kinds of process use? The types were defined as quite distinct from each other. They involve different kinds of learning processes for different purposes. Nevertheless, the categories are not independent of each other. In particular, the category of boosting morale could be seen as a secondary effect of learning to learn or developing networks. We have opted for treating boosting morale as a separate category, as it can be observed without being an effect of the others. It can occur together with them, or without them, and their presence does not necessarily boost morale either. However, it seems as if the different categories of process use will affect or are likely to affect stakeholders in an evaluation process in different ways. Table 1 below summarizes which actors are the most likely to benefit from each kind of process use. The table suggests that, primarily, those who do the evaluation will benefit; this refers to those who set the targets of the research, select the appropriate methods and collect and analyse data. The evaluators do not necessarily need to be an external team. As the evidence from the NDF case study indicates, this is a strategic choice. If the extent of process use is to be maximized in an organization, staff members should take part in evaluation work as much as possible. A high degree of process use is something that can be planned, designed and promoted. The choice of method will also have an impact on process use. It would seem that several aspects of process use depend upon personal communication; various kinds of interviewing methods would therefore seem more likely to generate it. Documentation analysis and large-scale surveys would probably be the methods that generate the least opportunities for process use. To further complicate the subject, it is also possible that the combination of personnel external evaluators, stakeholders and others whom the evaluators interact with will affect process use. An evaluation based on interviews with management in an organization may generate higher opportunities for micro-level change if the evaluators are a team of experts recruited in a peer group. If the evaluation effort is directed at technical levels of an organization, professionals of a more specific nature will probably increase the chances of process use. Table 1. Actors who will Benefit from Process Use Category of process use Actors who will benefit Learning to learn Developing networks Extending communication Strengthening the project Boosting morale To some extent those who commission the evaluation, and who furnish information, but primarily the evaluators Primarily the evaluators Primarily those who furnish information Primarily project management Primarily the evaluators 39

12 Evaluation 8(1) In sum, process use can be described as an outcome which may vary quite significantly. It is an outcome that can be seen as positive or negative, and comes at an opportunity cost in the use of time and money. It will affect the stakeholders in an evaluation process differently. High process use by some may imply a low use by others, or even a use that may hold a negative value. Can Process Use be Facilitated and Strengthened? It is well known that the utility of an evaluation process can be increased by extending communication when the results are presented. Working on the report, finding ways of supplementing a report with other presentations and connecting to decision makers are all worthwhile efforts. However, one practical implication of such suggestions is that budget resources are shifted from earlier stages in the evaluation process to latter stages. In a typical evaluation process, most of the money is spent on data collection and analysis, a minor share is spent on preparation for the evaluation, and even less is spent on disseminating the results. A rough approximation may be that 20 percent of the resources are spent in anticipating and formulating the evaluation; 75 percent of resources in conducting the evaluation, and 5 percent in disseminating the results. A high quality evaluation process is defined in terms of utility, feasibility, propriety and accuracy (Sanders, 1994). Whereas there can hardly be any doubt that a shift of resources from earlier to the last phase in the evaluation would enhance utility, a likely outcome would be that another quality aspect would suffer. Furthermore, there is a hypothesis that more learning takes place before and during the evaluation, than after. This does not mean that upon completion the evaluation is of no use; it does not necessarily collect dust on a shelf! But the report itself will normally provide little new insight. Most discoveries and new knowledge have been consumed and used during the evaluation process. The report merely marks the end of the evaluation process. It is perhaps not surprising that most of the learning occurs during the evaluation process at the time when most of the money is spent, rather than afterwards. But is it possible to increase the overall evaluation use without shifting resources from one evaluation stage to another? Could a marginal increase in utility be achieved at a lower cost, if the money is spent in preparation and implementation, rather than in disseminating the report? Indeed, can process use be traded off against use of findings and recommendations? Are they worth an equal amount? Can they even be compared? Two cases of process use are discussed below in relation to how process use was, or could have been, strengthened. Case 1 Enterprise Development A concrete example of process use is an evaluation of a small enterprise development programme in a county in Denmark. The programme was designed to assist small companies with less than 50 employees to access training and skillsupgrading courses for their staff. The programme hired six consultants with the 40

13 Forss et al.: Process Use of Evaluations task of identifying companies, working with them to identify training needs and identify the appropriate courses and vocational training centres where staff might acquire the necessary training. We the evaluators followed the programme for two years. The programme funders the EU Commission, the Danish county and Government and a number of vocational training centres wanted the evaluation to document accountability: whether taxpayers money had been well spent with good results. The final evaluation report after the programme was over shows the consequences of the programme to the companies: how many staff members were trained; what types of skills did they acquire; what did it mean to the company in terms of product development, changes in working routines, turnover, etc. But the real value of the evaluation was in its process use. The six consultants hired to the programme were convinced from the beginning that some of the barriers that exist in small companies for sending their staff for retraining and skills-development courses could be overcome. Small companies do not have staff training plans or personnel departments, and are even less likely to have human resource development functions or training budgets. Often staff members functions are so specialized that if even a single staff member is not present because he/she is sent for training, the production process will more or less be stopped. But these barriers did not discourage the consultants; other barriers did. The vocational training centres were co-sponsors of the programme. In addition to overall programme goals, their own interest was to sell courses to the companies. However, the programme consultants soon discovered that courses available at the training centres did not match all the priorities and training needs of the companies. Early in the life of the programme, the evaluation was able to document that this was the case. And although the vocational training centres were very dissatisfied, the consultants and the programme steering group using the evaluation in the process to provide legitimacy were able to identify additional providers of training courses which could fill the training needs gaps. In a similar way, the evaluation process was instrumental in changing the scope of the programme. It had been designed as a vocational training programme, but one of the consultants especially found this to be a very narrow approach to developing human resources in small companies. Developing skills of an individual staff member would, in his mind, have little effect on the company, unless linked with devices such as technological developments and changed working procedures, company work plans and strategies. Through dialogue with the evaluation team, and his five fellow consultants, a changed approach was developed during the first half of the programme period. The scope was changed in order that the identification of training needs, development of training plans and identification of training providers were supplemented with assistance and advice in areas completely new to most small companies. Training and training plans were linked to organizational and management development, identification of necessary changes in working procedures and the development of overall, strategic company plans. The process use of the evaluation led to programme 41

14 Evaluation 8(1) changes. And it was the new features of the revised programme that companies gave as the reason for assessing the programme to have been a success. Case 2 The Role of Experts in Technical Co-operation A number of overseas development agencies together commissioned an evaluation to analyse the effectiveness of expatriate experts on aid programmes. The evaluation was concerned with how expatriates are trained, receive job descriptions and the like. There was a general concern about details such as: an appropriate duration of an assignment abroad; type and extent of cross-cultural management skills; the difficulties of returning to a career at home; and the problems of accompanying spouses. The evaluation was primarily commissioned to learn about managing large technical assistance programmes. The recommendations were expected to feed into the decision-making mechanisms of the agencies that had commissioned the study. It was directed at senior management, in particular directors of personnel and recruitment. When the evaluation was completed, it was widely published, and the information requested did reach the target groups. The method chosen by the evaluators was interviewing expatriate advisers and their counterpart staff on a number of projects. In one such interview, the evaluators spoke to counterpart staff in an organization. They discussed personnel policies, and asked how the foreign experts were viewed. It was no surprise to find out that they were quite resented; arrogant behaviour, closed systems of decision making, reluctance to share knowledge and train local staff, high turnover, and extravagant living conditions were the main reasons. During the interview it turned out that several more expatriate experts were on their way; among them a financial adviser. The evaluators could see that the financial system was in shape, and that there was no need for assistance in that field. In fact, the local manager, who had been in charge for 15 years, was well educated and respected by others, did well, but he planned to leave as soon as the experts arrived. So, the assistance would destroy existing competence rather than help. The evaluators highlighted this problem to the aid agencies. Luckily, in this case there was a swift response, and recruitment for a financial expert was halted. The evaluation as a whole was of course not concerned with this particular project, and the main findings, conclusions and recommendations did not reflect this particular decision. But the process of evaluation generated change, it fed advice directly to decision makers, which led them to change a course of action. Conclusion In addition to the generally accepted notion that evaluations, through feedback processes, add new knowledge to organizational/programme development after the evaluation, there is in fact much that is useful that takes place during the evaluation process. In the article we have defined this as process use. A general example of process use is the well-known reaction of stakeholders when the final report is being presented: Your suggestions are fine. However, we 42

15 Forss et al.: Process Use of Evaluations have already adopted procedures along the lines that you suggest, so your recommendations do not in fact take account of the latest developments. Hence, the evaluation has sparked changes to the programme/organization during the evaluation process, making the report itself obsolete. In the article, we identify five different types of process use. These extend the categories of process use introduced by Michael Patton (1997 and 1998) and distinguish the actual experiences more sharply from the use of findings and recommendations. Taken together, they occur irrespective of the findings and recommendations that emerge through the collection, analysis and presentation of data. They reside in the process of evaluation, and they occur as a side effect of this unfolding process. Whereas some kinds of process use benefit those who carry out the evaluation, other kinds benefit those who are target groups for an intervention, and yet other kinds of process use may benefit those who commission the evaluation, or indeed the project management. At first sight, it seems as if most of the positive effects of process use will be experienced by the evaluators. But it is important to note that who the evaluators are does not affect this; in fact, internal evaluators may have a high process use, as may external evaluators. This could be an argument in favour of participatory approaches to evaluation to spread the utility of evaluation more evenly and widely. We do not doubt that process use exists, but we do not know how common it is. Additionally, it is difficult to ascertain whether it is a more common use of evaluation than the traditional use of lessons learned. But, since so many suggest that the use of lessons learned is severely limited, it may not be an exaggeration to suggest that in total, the different kinds of process use suggested here make up an equally important aspect of evaluation utility as the use of findings and recommendations. Hence there is a need to know more about process use, and to verify whether it occurs as often as some like to believe it does. The review of empirical examples suggests that process use occurs due to luck: it is a question of chance. While not underestimating good fortune, we suggest that process use could be planned and designed. Utility appears to be a weak spot in evaluation quality. The quest for utility is often focused on increasing the use of lessons learned, which implies shifting resources from implementation to the presentation and dissemination of results. If no resources are added, this will lower the quality of the other evaluation stages. Instead, experience with process use may indicate that further strengthening of data collection and analyses is needed in order to plan deliberately for use during the evaluation process. Notes 1. It is inevitable that there is a positive bias in this approach. We have looked for instances of use; we have not been engaged in a critical examination of whether evaluations are used or not, or even whether process use occurs as often as it could. The analysis suggests possible types of use it is not a critical examination of whether these actually occur. 2. This is a preliminary list of different process uses and is heavily indebted to Patton s first identification of kinds of process use (1998). The types are only possibilities; they 43

16 Evaluation 8(1) do not necessarily or automatically occur. Maybe they do not occur at all; maybe some are more common than others. Further empirical research is needed in this area. References Brunsson, N. (1989) The Organization of Hypocrisy. New York: John Wiley & Sons. Carlsson, J., M. Eriksson-Baaz, A. M. Fallenius and E. Lövgren (1999) Are Evaluations Useful? Cases from Swedish Development Cooperation. Stockholm: Sida. Cracknell, B. (1990) EEC Commission and Member States Review of Effectiveness of Feedback Mechanisms. Brussels: European Commission. Eden, C. and J. C. Spender (1998) Managerial and Organizational Cognition. Theory, Methods and Research. London: Sage. Forss, K. (1999) Questions and Methods in Impact Assessment; Outline of an Analytical Framework to Study Impact from Information and Communication. Wageningen: CTA. McMaster, M. (1996) The Intelligence Advantage: Organizing for Complexity. Boston, MA: Heinemann. Marra, M. (2000) How Much Does Evaluation Matter? Some Examples of the Utilization of the Evaluation of the World Bank s Anti-Corruption Activities, Evaluation 6(1): Morgan, P. (1986) Images of Organization. London: Sage. Patton, M. Q. (1978) Utilization-focused Evaluation. London: Sage. Patton, M. Q. (1982) Practical Evaluation. London: Sage. Patton, M. Q. (1997) Utilization-focused Evaluation. The New Century Text, 3rd edn. Thousand Oaks, CA: Sage. Patton, M. Q. (1998) Discovering Process Use, Evaluation 4(2): Pawson, R. and N. Tilley (1997) Realistic Evaluation. London: Sage. Peters, T. (1994) Crazy Times Call for Crazy Organizations. London: Pan Books. Rebien, C. (1996) Evaluating Development Assistance in Theory and Practice. Aldershot: Avebury. Samset, K., K. Forss and O. Hauglin (1992) Learning from Experience a Study of the Feedback from the Evaluation System in the Norwegian Aid Administration. Oslo: Ministry for Foreign Affairs. Sanders, J. R. (1994) The Program Evaluation Standards. Joint Committee on Standards for Educational Evaluation. London: Sage. Thoenig, J.-C. (2000) Evaluation as Usable Knowledge for Public Management Reforms, Evaluation 6(2): Torres, R. L., H. S. Preskill and M. E. Piontek (1996) Evaluation Strategies for Communication and Reporting. London: Sage. Vedung, E. (1997) Utvärdering och de sex användningarna, in B. Rombach and K. och Sahlin Andersson (eds) Från sanningssökande till styrmedel. Moderna utvärderingar i offentlig sektor. Uppsala: Nerenius & Santérus Förlag. KIM FORSS is the founder of Andante Tools for Thinking, an independent centre for research and competence development in the fields of evaluation and organization research. He has published work in the following areas: technical cooperation, organizational learning, evaluation research and UN reform. Please address correspondence to: Andante AB, Bisp Thomas v. 8, Strängnäs, Sweden. [ 44

17 Forss et al.: Process Use of Evaluations CLAUS C. REBIEN is Chief Projects Manager at COWI Ltd, the largest Danish consultancy. He has led numerous evaluations in Eastern Europe, Asia and Africa. In recent years his international experience has been applied to evaluations in the health and social sectors in his home country. Please address correspondence to: COWI Ltd, 2 Parallelvej, 2800 Kgs. Lyngby, Denmark. [ JERKER CARLSSON was a professor at the Institute of Peace and Development Research at the University of Gothenburg, and he was a co-founder of the Andante group of companies. He published work on development economics, economic history and the organization of partnerships and roles in development co-operation. Professor Carlsson passed away in September

The Work on Gender Mainstreaming in the Ministry of Employment by Agnete Andersen, legal adviser

The Work on Gender Mainstreaming in the Ministry of Employment by Agnete Andersen, legal adviser The Work on Gender Mainstreaming in the Ministry of Employment by Agnete Andersen, legal adviser Agnete Andersen The outset for the work on gender mainstreaming Gender equality is a difficult bird because

More information

Monitoring and Evaluation of. Interventions

Monitoring and Evaluation of. Interventions Monitoring and Evaluation of Sports in Development (SiD) Interventions presented by: Pamela K. Mbabazi, Ph.D. Faculty of Development Studies, Department of Development Studies Mbarara University of Science

More information

How to write your research proposal

How to write your research proposal How to write your research proposal by Maria Joyce, Lecturer, Faculty of Health and Social Care, University of Hull June 2004 The writing of a research proposal is generally understood to be a part of

More information

Project Management Simple Answers to Simple Questions

Project Management Simple Answers to Simple Questions Project Management Simple Answers to Simple Questions Originally I wrote this for one of my clients in 1991. The idea was to develop a brochure to promote project management in one of the client's departments.

More information

Research groups on masters level a pedagogical experiment

Research groups on masters level a pedagogical experiment Kerstin Rydbeck Research groups on masters level a pedagogical experiment Paper for the IFLA satellite meeting, 8 9 August 2010, in Borås, Sweden Introduction The two years master programme in ALM (archive,

More information

Challenges of Intercultural Management: Change implementation in the context of national culture

Challenges of Intercultural Management: Change implementation in the context of national culture 12-ICIT 9-11/4/07 in RoC Going for Gold ~ Best Practices in Ed. & Public Paper #: 07-08 Page- 1 /7 Challenges of Intercultural Management: Change implementation in the context of national culture Prof.

More information

Human Resource Management and Development

Human Resource Management and Development Human Resource Management and Development product: 4388 course code: c302 Human Resource Management and Development Centre for Financial and Management Studies, SOAS University of London First Edition

More information

Attribute 1: COMMUNICATION

Attribute 1: COMMUNICATION The positive are intended for use as a guide only and are not exhaustive. Not ALL will be applicable to ALL roles within a grade and in some cases may be appropriate to a Attribute 1: COMMUNICATION Level

More information

THE ROLE OF MARKETING IN MULTINATIONAL SUBSIDIARIES: STANDARDIZATION VERSUS LOCALIZATION

THE ROLE OF MARKETING IN MULTINATIONAL SUBSIDIARIES: STANDARDIZATION VERSUS LOCALIZATION THE ROLE OF MARKETING IN MULTINATIONAL SUBSIDIARIES: STANDARDIZATION VERSUS LOCALIZATION Miroslav Karlíãek, Zuzana Chytková, Nikola Hofiej, Hana Mohelská, Jakub Fischer Introduction In the international

More information

Board of Commissioners

Board of Commissioners Board of Commissioners SELF-STUDY HANDBOOK CHAPTER TWO Guidelines for Conducting an Institutional Self-Study TABLE OF CONTENTS Introduction 1 Purpose of the Self-Study 1 Institutional Evaluation 1 Institutional

More information

Report of a Peer Learning Activity held in Copenhagen / Malmö, 7 11 October 2007 Relationships between Teacher Education Institutions and schools

Report of a Peer Learning Activity held in Copenhagen / Malmö, 7 11 October 2007 Relationships between Teacher Education Institutions and schools EUROPEAN COMMISSION Directorate-General for Education and Culture Life Long Learning: policy and programmes School Education; Comenius Education and Training 2010 programme Cluster 'Teachers and Trainers'

More information

Guidance Note on Developing Terms of Reference (ToR) for Evaluations

Guidance Note on Developing Terms of Reference (ToR) for Evaluations Evaluation Guidance Note Series UNIFEM Evaluation Unit October 2009 Guidance Note on Developing Terms of Reference (ToR) for Evaluations Terms of Reference (ToR) What? Why? And How? These guidelines aim

More information

University of Bath. Welsh Baccalaureate Qualification Internal Evaluation. Themed Report: MARKETING AND PROMOTION

University of Bath. Welsh Baccalaureate Qualification Internal Evaluation. Themed Report: MARKETING AND PROMOTION University of Bath Welsh Baccalaureate Qualification Internal Evaluation Themed Report: MARKETING AND PROMOTION [This is one of eight themed reports which draw on issues relating to particular themes that

More information

On the attributes of a critical literature review. Saunders, Mark N. K. 1 & Rojon, Céline 2. United Kingdom.

On the attributes of a critical literature review. Saunders, Mark N. K. 1 & Rojon, Céline 2. United Kingdom. On the attributes of a critical literature review Saunders, Mark N. K. 1 & Rojon, Céline 2 1 School of Management, University of Surrey, Guildford, GU2 7XH, United Kingdom; 2 Department of Psychology &

More information

1. FORMULATING A RESEARCH QUESTION

1. FORMULATING A RESEARCH QUESTION W R I T I N G A R E S E A R C H G R A N T P R O P O S A L 1. FORMULATING A RESEARCH QUESTION 1.1. Identify a broad area of interest through literature searches, discussions with colleagues, policy makers

More information

BC Public Service Competencies

BC Public Service Competencies BC Public Service Competencies Competencies that support LEADING PEOPLE For Executive and Directors: Motivating for Peak Performance Motivating for peak performance involves knowledge and skills in using

More information

ASSESSMENT CENTER FOR IDENTIFYING POTENTIAL PROJECT MANAGERS: A CHANCE FOR SYSTEMATIC HUMAN RESOURCE DEVELOPMENT

ASSESSMENT CENTER FOR IDENTIFYING POTENTIAL PROJECT MANAGERS: A CHANCE FOR SYSTEMATIC HUMAN RESOURCE DEVELOPMENT ASSESSMENT CENTER FOR IDENTIFYING POTENTIAL PROJECT MANAGERS: A CHANCE FOR SYSTEMATIC HUMAN RESOURCE DEVELOPMENT Dipl. Psych. Ingo Heyn, ALLIANZ LEBENSVERSICHERUNGS-AG, Germany, 1999 Paper for the 6th

More information

Middlesbrough Manager Competency Framework. Behaviours Business Skills Middlesbrough Manager

Middlesbrough Manager Competency Framework. Behaviours Business Skills Middlesbrough Manager Middlesbrough Manager Competency Framework + = Behaviours Business Skills Middlesbrough Manager Middlesbrough Manager Competency Framework Background Middlesbrough Council is going through significant

More information

Communicating and influencing

Communicating and influencing HR SLA Page 1 of 9 Communicating and influencing I communicate confidently, professionally, authoritatively and with clarity both verbally and in writing. I use a range of effective communication skills

More information

2 nd EUA Funding Forum: Strategies for efficient funding of universities

2 nd EUA Funding Forum: Strategies for efficient funding of universities 2 nd EUA Funding Forum: Strategies for efficient funding of universities Bergamo, 9-10 October 2014 Forum report Liviu Matei, General rapporteur Table of contents I. Executive Summary 3 II. What is the

More information

Managing in the Round. 360 Degree Feedback and Middle Manager Development Programme

Managing in the Round. 360 Degree Feedback and Middle Manager Development Programme Managing in the Round 360 Degree Feedback and Middle Manager Development Programme 2 Managing in the Round 360 Degree Feedback and Middle Manager Development Programme What does Managing in the Round involve?

More information

TRAINING AND EDUCATION IN THE SERVICE OF MILITARY TRANSFORMATION

TRAINING AND EDUCATION IN THE SERVICE OF MILITARY TRANSFORMATION TRAINING AND EDUCATION IN THE SERVICE OF MILITARY TRANSFORMATION Ecaterina Livia TATAR Lecturer, Regional Department of Defense Resources Management Studies, Brasov, Romania The magnitude and challenges

More information

DESCRIBING OUR COMPETENCIES. new thinking at work

DESCRIBING OUR COMPETENCIES. new thinking at work DESCRIBING OUR COMPETENCIES new thinking at work OUR COMPETENCIES - AT A GLANCE 2 PERSONAL EFFECTIVENESS Influencing Communicating Self-development Decision-making PROVIDING EXCELLENT CUSTOMER SERVICE

More information

Behaviourally Based Questions

Behaviourally Based Questions Behaviourally Based Questions Index 1 HOW TO WRITE BEHAVIOURALLY BASED QUESTIONS Page 2 2 SAMPLE BEHAVIOURAL QUESTIONS Page 3 3 SAMPLE BEHAVIOURALLY BASED QUESTIONS FROM RIGHT JOB, RIGHT PERSON! CAPABILITY

More information

Copyright 2004.Pamela Cole. All rights reserved.

Copyright 2004.Pamela Cole. All rights reserved. Key concepts for working with the Role Behavior Analysis The Role Behavior Analysis (RBA), the companion instrument to the Personal Profile System (PPS), uses specific DiSC behavioral statements for defining,

More information

CHAPTER 1: A FRAMEWORK FOR INTERPERSONAL SKILL DEVELOPMENT

CHAPTER 1: A FRAMEWORK FOR INTERPERSONAL SKILL DEVELOPMENT CHAPTER 1: A FRAMEWORK FOR INTERPERSONAL SKILL DEVELOPMENT The first chapter sets the stage for improving interpersonal relations in organizations by presenting a framework for interpersonal skill development

More information

question is how does the implementation of change management strategies by tertiary institutions that are undergoing change lead to quality assurance?

question is how does the implementation of change management strategies by tertiary institutions that are undergoing change lead to quality assurance? 1 Sub theme: Governance and leadership in Tertiary Education Institutions Title of paper: Change management strategies -- mechanisms for quality assurance Author: Samantha Radway Institution: Church Teachers

More information

Master of Science in Management

Master of Science in Management Programme Syllabus for Master of Science in Management 120 higher education credits Second Cycle Established by the Faculty Board of the School of Business, Economics and Law, University of Gothenburg,

More information

Writing and Conducting Successful Performance Appraisals. Guidelines for Managers and Supervisors

Writing and Conducting Successful Performance Appraisals. Guidelines for Managers and Supervisors Writing and Conducting Successful Performance Appraisals Guidelines for Managers and Supervisors Performance reviews are like compasses They give your exact position and provide direction Office of Human

More information

PEOPLE INVOLVEMENT AND THEIR COMPETENCE IN QUALITY MANAGEMENT SYSTEMS * Jarmila ŠALGOVIČOVÁ, Matej BÍLÝ

PEOPLE INVOLVEMENT AND THEIR COMPETENCE IN QUALITY MANAGEMENT SYSTEMS * Jarmila ŠALGOVIČOVÁ, Matej BÍLÝ PEOPLE INVOLVEMENT AND THEIR COMPETENCE IN QUALITY MANAGEMENT SYSTEMS * Jarmila ŠALGOVIČOVÁ, Matej BÍLÝ Authors: Workplace: Assoc. Prof. Jarmila Šalgovičová, PhD., Prof. Matej Bílý, DrSC.* Institute of

More information

DESCRIPTOR OF THE STUDY FIELD OF PUBLIC ADMINISTRATION CHAPTER I GENERAL PROVISIONS

DESCRIPTOR OF THE STUDY FIELD OF PUBLIC ADMINISTRATION CHAPTER I GENERAL PROVISIONS DESCRIPTOR OF THE STUDY FIELD OF PUBLIC ADMINISTRATION CHAPTER I GENERAL PROVISIONS 1. The Descriptor of the Study Field of Public Administration (hereinafter referred to as the Descriptor ) shall govern

More information

WARSAW SCHOOL OF ECONOMICS

WARSAW SCHOOL OF ECONOMICS WARSAW SCHOOL OF ECONOMICS mgr Ewelina Florczak The summary of doctoral dissertation THE TITLE SOCIAL ENTERPRISE IN LOCAL ENVIRONMENT 1 Rationale topic A social enterprise as a business entity is subject

More information

Department of Political Science. College of Social Science. Undergraduate Bachelor s Degree in Political Science

Department of Political Science. College of Social Science. Undergraduate Bachelor s Degree in Political Science Student Outcomes Assessment Plan (SOAP) I. Mission Statement Department of Political Science College of Social Science Undergraduate Bachelor s Degree in Political Science The Department of Political Science

More information

Table of Contents. Excutive Summary

Table of Contents. Excutive Summary Presented by: 1 Table of Contents Excutive Summary I. Introduction II. Methodology III. Results of the Graduate Impact Survey IV. Implications and Outlook V. Literature 2 Executive Summary The Graduate

More information

Cultural Intelligence - A concept for bridging and benefiting from cultural differences

Cultural Intelligence - A concept for bridging and benefiting from cultural differences July 2007 Cultural Intelligence - A concept for bridging and benefiting from cultural differences By Elisabeth Plum (Translated and commented by Dympna Cunnane) What have international relations, mergers

More information

UNIDO. Competencies. Strengthening Organizational Core Values and Managerial Capabilities

UNIDO. Competencies. Strengthening Organizational Core Values and Managerial Capabilities Competencies Part One Strengthening Organizational Core Values and Managerial Capabilities Enquiries regarding the UNIDO Competency model can be addressed to S.Gardelliano@unido.org Contents Message from

More information

Brown, H. N., & Sorrell, J. M. (1993). Use of Clinical Journals to Enhance Thinking. Nurse Educator,18(5), 16-19.

Brown, H. N., & Sorrell, J. M. (1993). Use of Clinical Journals to Enhance Thinking. Nurse Educator,18(5), 16-19. Use of Clinical Journals to Enhance Critical Thinking By: Hazel N. Brown, EdD, RNC, and Jeanne M. Sorrell, DAEd, RN Brown, H. N., & Sorrell, J. M. (1993). Use of Clinical Journals to Enhance Thinking.

More information

The Relationship between the Fundamental Attribution Bias, Relationship Quality, and Performance Appraisal

The Relationship between the Fundamental Attribution Bias, Relationship Quality, and Performance Appraisal The Relationship between the Fundamental Attribution Bias, Relationship Quality, and Performance Appraisal Executive Summary Abstract The ability to make quality decisions that influence people to exemplary

More information

Corporate Governance Guidelines

Corporate Governance Guidelines Corporate Governance Guidelines Fuji Heavy Industries Ltd. Chapter 1. General Provisions Article 1. Purpose These guidelines set out the basic policy, framework and operating policy of the corporate governance

More information

G.F. Huon School of Psychology, University of New South Wales, Sydney, Australia

G.F. Huon School of Psychology, University of New South Wales, Sydney, Australia INTERVIEWING AND OBSERVATION G.F. Huon School of Psychology, University of New South Wales, Sydney, Australia Keywords: Unobtrusive observation, systematic observation, ethical considerations, reactivity,

More information

Sample Behavioural Questions by Competency

Sample Behavioural Questions by Competency Competencies that support LEADING PEOPLE Change Leadership Please tell us about a time when you led a significant change in your organization and how you helped others to deal with the change. Tell me

More information

Responsibility I Assessing Individual and Community Needs for Health Education

Responsibility I Assessing Individual and Community Needs for Health Education CHE Competencies Starting in the early 1990s, three national professional organizations the Society for Public Health Education, the American Association for Health Education, and the American Alliance

More information

Governance as Stewardship: Decentralization and Sustainable Human Development

Governance as Stewardship: Decentralization and Sustainable Human Development Governance as Stewardship: Decentralization and Sustainable Human Development by Jerry VanSant Research Triangle Institute USA EXECUTIVE SUMMARY Introduction The United Nations Development Programme (UNDP)

More information

COMMUNICATION STRATEGY FOR THE UNIVERSITY OF GOTHENBURG

COMMUNICATION STRATEGY FOR THE UNIVERSITY OF GOTHENBURG STYRDOKUMENT Dnr F V 2014/71 COMMUNICATION STRATEGY FOR THE UNIVERSITY OF GOTHENBURG Published Decision-maker Responsible body medarbetarportalen.gu.se/styrdokument Vice-Chancellor Kommunikationsenheten

More information

IMPLEMENTING THE TRIPLE AIM: A SENIOR LEADERSHIP PERSPECTIVE 1

IMPLEMENTING THE TRIPLE AIM: A SENIOR LEADERSHIP PERSPECTIVE 1 IMPLEMENTING THE TRIPLE AIM: A SENIOR LEADERSHIP PERSPECTIVE 1 Rafael BENGOA Patricia ARRATIBEL I. BACKGROUND There are numerous health care systems in the world undertaking the most important reforms

More information

TOOL KIT for RESIDENT EDUCATOR and MENT OR MOVES

TOOL KIT for RESIDENT EDUCATOR and MENT OR MOVES Get to Know My RE Observe Collect Evidence Mentor Moments Reflect Review Respond Tailor Support Provide Provide specific feedback specific Feedback What does my RE need? Practice Habits Of Mind Share Data

More information

On 4 December 1995, the National Faculty Meeting for Legal Studies agreed on the following statement:

On 4 December 1995, the National Faculty Meeting for Legal Studies agreed on the following statement: THE UNIVERSITY OF BERGEN The Faculty of Law THE LEVEL OF DOCTORAL DEGREES IN LAW Guidelines for the Faculties of Law at the University of Bergen and the University of Oslo, adopted by the board of the

More information

A Guide to Learning Outcomes, Degree Level Expectations and the Quality Assurance Process in Ontario

A Guide to Learning Outcomes, Degree Level Expectations and the Quality Assurance Process in Ontario A Guide to Learning Outcomes, Degree Level Expectations and the Quality Assurance Process in Ontario A Guide to Learning Outcomes, Degree Level Expectations and the Quality Assurance Process in Ontario

More information

Message from the Chief Executive of the RCM

Message from the Chief Executive of the RCM Message from the Chief Executive of the RCM The Midwifery Leadership Competency Framework has been derived from both the NHS Leadership Qualities Framework and the Clinical Leadership Competency Framework.

More information

Monitoring and Evaluation Plan Primer for DRL Grantees

Monitoring and Evaluation Plan Primer for DRL Grantees Monitoring and Evaluation Plan Primer for DRL Grantees I. What is a monitoring and evaluation plan? A monitoring and evaluation plan (M&E plan), sometimes also referred to as a performance monitoring or

More information

THE EFFECTIVENESS OF LOGISTICS ALLIANCES EUROPEAN RESEARCH ON THE PERFORMANCE MEASUREMENT AND CONTRACTUAL SUCCESS FACTORS IN LOGISTICS PARTNERSHIPS

THE EFFECTIVENESS OF LOGISTICS ALLIANCES EUROPEAN RESEARCH ON THE PERFORMANCE MEASUREMENT AND CONTRACTUAL SUCCESS FACTORS IN LOGISTICS PARTNERSHIPS CIIL An IESE-Mecalux Initiative STUDY-62 February, 2008 THE EFFECTIVENESS OF LOGISTICS ALLIANCES EUROPEAN RESEARCH ON THE MEASUREMENT AND CONTRACTUAL SUCCESS FACTORS IN LOGISTICS PARTNERSHIPS Joan Jané

More information

A PhD in Public Affairs?

A PhD in Public Affairs? A PhD in Public Affairs? The Basics A Doctor of Philosophy degree, abbreviated Ph.D. for the Latin Philosophiae Doctor, is an advanced academic degree earned in many fields, signifying major interests

More information

Establishing and Operating a Quality Management System Experiences of the EUROSAI Training Committee Seminar in Budapest

Establishing and Operating a Quality Management System Experiences of the EUROSAI Training Committee Seminar in Budapest Workshop Management of an SAI Berlin (Germany), 9-11 April 2008 Establishing and Operating a Quality Management System Experiences of the EUROSAI Training Committee Seminar in Budapest (Dr. Árpád Kovács,

More information

International Business Programme, Bachelor Course Descriptions 2015-2016

International Business Programme, Bachelor Course Descriptions 2015-2016 International Business Programme, Bachelor Course Descriptions 2015-2016 The following course descriptions briefly describe the course contents, how many credits the course is worth, if the course is given

More information

THE STANDARD FOR DOCTORAL DEGREES IN LAW AT THE FACULTY OF LAW, UNIVERSITY OF TROMSØ

THE STANDARD FOR DOCTORAL DEGREES IN LAW AT THE FACULTY OF LAW, UNIVERSITY OF TROMSØ THE FACULTY OF LAW THE STANDARD FOR DOCTORAL DEGREES IN LAW AT THE FACULTY OF LAW, UNIVERSITY OF TROMSØ Guidelines for the Faculty of Law in Tromsø, adopted by the Faculty Board on 31 May 2010. 1 Background

More information

Capability Procedure: Managing Poor Performance

Capability Procedure: Managing Poor Performance Capability Procedure: Managing Poor Performance This policy/apd supersedes: New Policy Name and title of author Christine White Human Resources Advisor Date written/revised September 2007 Approved by (Committee/Group)

More information

Interview studies. 1 Introduction... 1. 2 Applications of interview study designs... 2. 3 Outline of the design... 3

Interview studies. 1 Introduction... 1. 2 Applications of interview study designs... 2. 3 Outline of the design... 3 Interview studies Contents 1 Introduction... 1 2 Applications of interview study designs... 2 3 Outline of the design... 3 4 Strengths and weaknesses of interview study designs... 6 5 References... 7 1

More information

Institutional Entrepreneurs 1

Institutional Entrepreneurs 1 Preliminary draft Comments are welcome Institutional Entrepreneurs 1 David Daokui Li Tsinghua University Hong Kong University of Science and Technology Junxin Feng Hongping Jiang Tsinghua University December

More information

Finding the Right People for Your Program Evaluation Team: Evaluator and Planning Team Job Descriptions

Finding the Right People for Your Program Evaluation Team: Evaluator and Planning Team Job Descriptions : Evaluator and Planning Team Job Descriptions I. Overview II. Sample Evaluator Job Description III. Evaluator Competencies IV. Recruiting members of your strategic evaluation planning team V. Recruiting

More information

Master of Science in Management

Master of Science in Management Programme Syllabus for Master of Science in Management 120 higher education credits Second Cycle Established by the Faculty Board of the School of Business, Economics and Law, University of Gothenburg,

More information

Thinking in Big Data. Tony Shan. Nov 5, 2014

Thinking in Big Data. Tony Shan. Nov 5, 2014 Thinking in Big Data Tony Shan Nov 5, 2014 Table of Contents Introduction Big Data Skillset Challenges Concept of Big Data Competency and Model Big Data Thinking Conclusion 2 Introduction Big Data Competency

More information

Compass Interdisciplinary Virtual Conference 19-30 Oct 2009

Compass Interdisciplinary Virtual Conference 19-30 Oct 2009 Compass Interdisciplinary Virtual Conference 19-30 Oct 2009 10 Things New Scholars should do to get published Duane Wegener Professor of Social Psychology, Purdue University Hello, I hope you re having

More information

EXECUTIVE BEHAVIORAL INTERVIEW GUIDE

EXECUTIVE BEHAVIORAL INTERVIEW GUIDE EXECUTIVE BEHAVIORAL INTERVIEW GUIDE INTERVIEW GUIDE INSTRUCTIONS: This Interview Guide is intended to help hiring executives conduct behavioral interviews for executive classifications covered by the

More information

FOUNDATION YEAR Student Evaluation

FOUNDATION YEAR Student Evaluation FOUNDATION YEAR Student Evaluation I. Cover Sheet Student Name Home Phone Work Phone Cell Phone Email (Please Check) Full-time Part-time Field Instructor Work Phone Email Agency Cell Phone Site Address

More information

Contract Management Part One Making the Business Case for Investment

Contract Management Part One Making the Business Case for Investment Contract Management Part One Making the Business Case for Investment Executive Summary This paper is the first in a short series of three which will look at the business case for organisations to invest

More information

Information Technology Research in Developing Nations: Major Research Methods and Publication Outlets

Information Technology Research in Developing Nations: Major Research Methods and Publication Outlets Information Technology Research in Developing Nations: Major Research Methods and Publication Outlets Franklin Wabwoba, Anselimo Peters Ikoha Masinde Muliro University of Science and Technology, Computer

More information

MANAGEMENT COURSES Student Learning Outcomes 1

MANAGEMENT COURSES Student Learning Outcomes 1 MANAGEMENT COURSES Student Learning Outcomes 1 MGT 202: Business Professions 1. Describe and use the elements of effective decision making research, assessment and consequence. 2. Apply elements of effective

More information

The Role of Reputation in Professional Service Firms. Novak druce centre insights No. 6

The Role of Reputation in Professional Service Firms. Novak druce centre insights No. 6 The Role of Reputation in Professional Service Firms Novak druce centre insights No. 6 contents 01 Introduction 02 the Relationship between Reputation & Quality in Professional Service Firms (PSFs) 04

More information

Pharmaceutical Sales Certificate

Pharmaceutical Sales Certificate Pharmaceutical Sales Certificate Target Audience Medical representatives Objective The objective of this program is to provide the necessary skills and knowledge needed to succeed as medical representatives.

More information

Helen Featherstone, Visitor Studies Group and University of the West of England, Bristol

Helen Featherstone, Visitor Studies Group and University of the West of England, Bristol AUDIENCE RESEARCH AS A STRATEGIC MANAGEMENT TOOL Sofie Davis, Natural History Museum Helen Featherstone, Visitor Studies Group and University of the West of England, Bristol Acknowledgements: The authors

More information

MANAGEMENT AND MARKETING

MANAGEMENT AND MARKETING MANAGEMENT AND MARKETING The Department of Management and Marketing comprises some 18 full-time members of staff, academic and administrative, and over 2,000 students are currently registered for modules

More information

Research concerning the Social Services Act and complementary regulations

Research concerning the Social Services Act and complementary regulations Summary of Research Report 2009/10:RFR 3 Committee on Health and Welfare Research concerning the Social Services Act and complementary regulations SVERIGES RIKSDAG S-100 12 Stockholm Phone: +46 8 786 40

More information

Council on Social Work Education Educational Policies and Standards (EPAS) Competencies, Practice Behaviors and Advanced Practice Behaviors

Council on Social Work Education Educational Policies and Standards (EPAS) Competencies, Practice Behaviors and Advanced Practice Behaviors Council on Social Work Education Educational Policies and Standards (EPAS) Competencies, Practice Behaviors and Advanced Practice Behaviors 2.1.1 Core Competency: Identify with social work profession a.

More information

Developing Great Frontline Sales Managers: Four Key Sales Management Abilities

Developing Great Frontline Sales Managers: Four Key Sales Management Abilities W H I T E P A P E R Developing Great Frontline Sales Managers: Four Key Sales Management Abilities Why Sales Managers Need Management Training How does a sales manager learn how to manage a sales team?

More information

CHAPTER III RESEARCH METHODS

CHAPTER III RESEARCH METHODS CHAPTER III RESEARCH METHODS 3.1. Introduction Advertising management has become a prominent organizational activity in public and private corporate houses in India and abroad. Advertising is practiced

More information

Towards a new paradigm of science

Towards a new paradigm of science Essay Towards a new paradigm of science in scientific policy advising 2007 Eva Kunseler Kansanterveyslaitos Environment and Health Department Methods of Scientific Thought Prof. Lindqvist Introduction:

More information

Title: MPs Use of Social Networking Sites. A cross-national research

Title: MPs Use of Social Networking Sites. A cross-national research Title: MPs Use of Social Networking Sites. A cross-national research Bio: Norbert Merkovity lecturer at University of Szeged and researcher at National University of Public Service, Budapest, Hungary (merkovity@juris.u-szeged.hu).

More information

LONDON SCHOOL OF COMMERCE. Programme Specifications for the. Cardiff Metropolitan University. MSc in International Hospitality Management

LONDON SCHOOL OF COMMERCE. Programme Specifications for the. Cardiff Metropolitan University. MSc in International Hospitality Management LONDON SCHOOL OF COMMERCE Programme Specifications for the Cardiff Metropolitan University MSc in International Hospitality Management 1 Contents Programme Aims and Objectives 3 Programme Learning Outcomes

More information

Quality Assurance in Higher Education: A Literature Review

Quality Assurance in Higher Education: A Literature Review Quality Assurance in Higher Education: A Literature Review ANDREA WILGER National Center for Postsecondary Improvement 508 CERAS School of Education Stanford University, Stanford, CA 94305-3084 1997 National

More information

9 TH INTERNATIONAL ASECU CONFERENCE ON SYSTEMIC ECONOMIC CRISIS: CURRENT ISSUES AND PERSPECTIVES

9 TH INTERNATIONAL ASECU CONFERENCE ON SYSTEMIC ECONOMIC CRISIS: CURRENT ISSUES AND PERSPECTIVES Matilda Alexandrova Liliana Ivanova University of National and World Economy,Sofia, Bulgaria CRITICAL SUCCESS FACTORS OF PROJECT MANAGEMENT: EMPIRICAL EVIDENCE FROM PROJECTS SUPPORTED BY EU PROGRAMMES

More information

PERCEIVED QUALITY IN THE DELIVERY OF BUSINESS SUPPORT SERVICES: A CONCEPTUAL FRAMEWORK (WITH PRACTICAL IMPLICATIONS)

PERCEIVED QUALITY IN THE DELIVERY OF BUSINESS SUPPORT SERVICES: A CONCEPTUAL FRAMEWORK (WITH PRACTICAL IMPLICATIONS) PERCEIVED QUALITY IN THE DELIVERY OF BUSINESS SUPPORT SERVICES: A CONCEPTUAL FRAMEWORK (WITH PRACTICAL IMPLICATIONS) Nicola Bellini LINK Research Center Scuola Superiore Sant'Anna Pisa - Italy nbellini@sssup.it

More information

pm4dev, 2007 management for development series Introduction to Project Management PROJECT MANAGEMENT FOR DEVELOPMENT ORGANIZATIONS

pm4dev, 2007 management for development series Introduction to Project Management PROJECT MANAGEMENT FOR DEVELOPMENT ORGANIZATIONS pm4dev, 2007 management for development series Introduction to Project Management PROJECT MANAGEMENT FOR DEVELOPMENT ORGANIZATIONS PROJECT MANAGEMENT FOR DEVELOPMENT ORGANIZATIONS A methodology to manage

More information

By Jack Phillips and Patti Phillips How to measure the return on your HR investment

By Jack Phillips and Patti Phillips How to measure the return on your HR investment By Jack Phillips and Patti Phillips How to measure the return on your HR investment Using ROI to demonstrate your business impact The demand for HR s accountability through measurement continues to increase.

More information

Ph. D. Program in Education Specialization: Educational Leadership School of Education College of Human Sciences Iowa State University

Ph. D. Program in Education Specialization: Educational Leadership School of Education College of Human Sciences Iowa State University Ph. D. Program in Education Specialization: Educational Leadership School of Education College of Human Sciences Iowa State University The purpose of the doctoral program in Educational Leadership is to

More information

Performance planning:

Performance planning: Performance planning: A guide for staff development and feedback Version: 2.1 Last Updated: April, 2015 Ryerson University Human Resources 0 Table of contents Performance planning: Staff development and

More information

Origin and structure of the volume

Origin and structure of the volume Introduction This book presents a selection of articles dedicated to the analysis of the situation and the evolution of public relations and organisational strategic communication, with a main focus on

More information

SigmaRADIUS Leadership Effectiveness Report

SigmaRADIUS Leadership Effectiveness Report SigmaRADIUS Leadership Effectiveness Report Sample Report NOTE This is a sample report, containing illustrative results for only two dimensions on which 360 performance ratings were obtained. The full

More information

Total Quality Management in Engineering Education

Total Quality Management in Engineering Education Total Quality Management in Engineering Education EFTHIMIA STAIOU Department of Business Administration University of Macedonia Egnatia 156, P.O. Box 1591, 54006 Thessaloniki GREECE Abstract: - Higher

More information

Model of Learning Organizational Development of Primary School Network under the Office of Basic Education Commission

Model of Learning Organizational Development of Primary School Network under the Office of Basic Education Commission International Education Studies; Vol. 8, No. 7; 2015 ISSN 1913-9020 E-ISSN 1913-9039 Published by Canadian Center of Science and Education Model of Learning Organizational Development of Primary School

More information

KM Tools. Introduction. Communities of practice

KM Tools. Introduction. Communities of practice KM Tools Introduction The purpose of this note is to provide an overview of various tools that can be used for improving the sharing of knowledge and lessons. There are several inter-related tools, each

More information

UNIVERSITY OF ROCHESTER William E. Simon Graduate School of Business Administration. Proposal for a Clinical Faculty Track

UNIVERSITY OF ROCHESTER William E. Simon Graduate School of Business Administration. Proposal for a Clinical Faculty Track Contents: UNIVERSITY OF ROCHESTER William E. Simon Graduate School of Business Administration Proposal for a Clinical Faculty Track 1. Rationale and Objectives of the Proposal 2. The Simon School Mission

More information

Economic Commentaries

Economic Commentaries n Economic Commentaries Sweden has had a substantial surplus on its current account, and thereby also a corresponding financial surplus, for a long time. Nevertheless, Sweden's international wealth has

More information

HND degree top-up students perceptions of their experience at the University of Worcester: how can future students experiences be improved?

HND degree top-up students perceptions of their experience at the University of Worcester: how can future students experiences be improved? HND degree top-up students perceptions of their experience at the University of Worcester: how can future students experiences be improved? Lerverne Barber and Dr Nick Breeze University of Worcester (l.barber@worc.ac.uk

More information

Internal Audit. Audit of HRIS: A Human Resources Management Enabler

Internal Audit. Audit of HRIS: A Human Resources Management Enabler Internal Audit Audit of HRIS: A Human Resources Management Enabler November 2010 Table of Contents EXECUTIVE SUMMARY... 5 1. INTRODUCTION... 8 1.1 BACKGROUND... 8 1.2 OBJECTIVES... 9 1.3 SCOPE... 9 1.4

More information

UNIVERSITY OF READING

UNIVERSITY OF READING UNIVERSITY OF READING MARKING CRITERIA CLASSIFICATION FRAMEWORK FOR TAUGHT POSTGRADUATE PROGRAMMES (for non-greenlands cohorts entering Autumn Term 2008 and thereafter) (for Greenlands cohorts entering

More information

TENTATIVE SUGGESTIONS FOR THE CERTIFICATION OF PRACTICING PSYCHOLOGISTS

TENTATIVE SUGGESTIONS FOR THE CERTIFICATION OF PRACTICING PSYCHOLOGISTS FURTHER COMMUNICATIONS REGARDING "A PLAN FOR THE TECHNICAL TRAINING OF CONSULTING PSYCHOLOGISTS " Dr. Leta S. Hollmgworth, Teachers College, Columbia University, sends the following: TENTATIVE SUGGESTIONS

More information

A new European approach toward Quality Assurance in Vocational Education

A new European approach toward Quality Assurance in Vocational Education A new European approach toward Quality Assurance in Vocational Education I have been a teacher myself for over 15 years now. To keep developing myself as a teacher in my profession I signed in for a Masters

More information

Lessons Learned from MDG Monitoring From A Statistical Perspective

Lessons Learned from MDG Monitoring From A Statistical Perspective Lessons Learned from MDG Monitoring From A Statistical Perspective Report of the Task Team on Lessons Learned from MDG Monitoring of the IAEG-MDG United Nations March 2013 The views expressed in this paper

More information

and responding to the concerns of their constituents and of the general public in order to take advantage of new opportunities and to anticipate and

and responding to the concerns of their constituents and of the general public in order to take advantage of new opportunities and to anticipate and introduction For more than a century the business corporation has been a successful and widely adopted institutional arrangement for creating and distributing wealth. But the power and purpose of corporations

More information

THE NEED FOR PROFESSIONAL ACCOUNTING EDUCATION. John C. Burton Chief Accountant Securities and Exchange Commission

THE NEED FOR PROFESSIONAL ACCOUNTING EDUCATION. John C. Burton Chief Accountant Securities and Exchange Commission THE NEED FOR PROFESSIONAL ACCOUNTING EDUCATION by John C. Burton Chief Accountant Securities and Exchange Commission The University of Texas at Austin College of Business Administration February 28, 1974

More information