EVALUATION: WHY DO WE NEED IT?

Transcription

1 EVALUATION: WHY DO WE NEED IT? There are many times when we need to evaluate the services we provide; whether we are working in public sector providing front-line services to the community, or artists delivering projects or freelancing developing websites for business and personal clients. Evaluation helps us provide more effective and efficient services and projects. Evaluation enhances the quality of our creative practice, or increases our sustainability and/or profits! As the aim of evaluation is to become more effective and efficient, when we evaluate a service we need to think about two distinct areas; the processes involved in providing the service or arts project, and the effect or impact of delivering the service and/or project. When we speak of evaluating processes what do we mean? Basically we are talking about a thorough examination of the ways in which a service and/or project is delivered. How is the service/project managed? How well do different people involved in delivering the service communicate with each other? How is service delivery project managed? Where are the roadblocks to effective and efficient delivery of the service/project? What do we do that we don t need to do? What do we not do that we ought to do? The answers to these questions vary enormously. The management of a public sector service is very different to the management of a small design agency, or an artist delivering a workshop or involved in a Public Art project and for a freelance web developer it is different again. The basic questions, however, remain the same and can be easily summarized. How can we become more effective and more efficient? Before we can answer any of these questions we need to know what we actually do. It is amazing how many organisations from the very largest to the one-man operation find it very difficult to say in detail what they do. To say I provide web development services, or we deliver one to one arts projects for the elderly is fine, but it doesn t tell us anything of the day-to-day tasks we undertake to achieve this. To do this we need to monitor our activities. Monitoring is essentially very straightforward, and is an essential part of any business or public service. It can take many forms yet all serve the same function. It provides a record of what we actually do. So we have evaluated the processes of delivering our service or Evaluation: Why do we need it? Arts Unit KD

2 project, what about evaluating its impact? By impact we mean the impact on our audiences, participants, customers and on ourselves. The impact on our audiences (for example) is basically customer satisfaction, although depending what type of service we provide that can take many forms. The impact on ourself in turn impacts on our job satisfaction. Even if you work voluntarily within the delivery of an arts project, your own satisfaction impacts on how successful the project will be, and so as a manager this need to be considered. How do we find out about the impact of our service on our users and ourselves? The answer is obvious, we ask. We need to obtain regular feedback from our service users (customers, clients, audiences, participants) asking them how satisfied they are, what do we do well, and what could we do better? This is not to forget our own self reflection and evaluation, asking ourselves a host of questions. What do we do well? What could we do better? What skills do we have as individuals and as an organisation? What skills do we need to develop our service? How satisfied are we with our jobs? How could our job satisfaction be improved? Evaluation is not complicated, but it is essential. There is no benefit to anyone in providing a costly service that is relatively ineffective. Monitoring and consultation provide practical evidence on which to make rational judgments on the effectiveness and efficiency of the service we deliver. In other words, monitoring and consultation provide us with the tools for evaluation. This in turn helps us logically become more effective and efficient in delivering our service. When do you start to evaluate? Evaluation needs to be at the heart of the activity, not as an add on. Evaluation processes, systems, and methods should be included as part of any project planning or management task, and should be regularly referred to updated and results reviewed to aid the on going development of any project or service. Who, What and How? Evaluation processes can be organised and led in different ways according to the nature of the work: Use an internal evaluator (someone who is already involved in your work like a committee member or employee) Use an external evaluator (someone who can come in to review processes, projects and if being included into any project would need to be budgeted for) Co-opt participant evaluators (ask people who are involved as participants to give you their opinion, so you get the inside view of how things are working) Audiences (who are your audiences and how can you capture information form them?) Evaluation: Why do we need it? Arts Unit KD

3 In order for you to decide which format is best for you, you need to think about: Who is the audience for the evaluation and what kind of information do they require? Do they require facts and figures, or more of a feel of the organisation s work? What kinds of evaluation methods are appropriate for you and or your organisation? There is no point choosing a lengthy questionnaire if no one takes time to fill it out! What kind of sample is needed to gather a reliable level of information? Do you need to survey everyone or would a chat with a few be enough? Have you or your organisation considered all the implications of evaluation? What methods? And the costs? Answering these questions will help in deciding which methods are best for you to take forward. TOOLS FOR THE JOB Once an understanding of what and why you need evaluation and what is needed to concentrate on, the next step is to identify the best tools. Often a mixture of techniques is required and the inherent biases from one will rectify those of another. Following are examples (but is no way exhaustive) of ways you can evaluate: 1. Measuring Attendance Paper label stickers PROCESS: Cheap, low tech and capable of all sorts of variation. Use them to count numbers of people attending an event just subtract the empty spaces on the chart from the total. PROS: Different sizes and colours can differniate between children and adults, men form women, etc. CONS: Reliance on members of public to stick their sticker on the right chart, may need stewarding. Number Clicker PROCESS: For counting entries through a gate or turnstile PROS: Even simpler than stickers just read numbers, can be reset when desired. CONS: Not as adaptable or mobile, cannot distinguish between people (for example, who leave then return). Camera and Video PROCESS: For estimates of attendance, especially in free events when other techniques are unavailable PROS: Quick and easy. Evaluation: Why do we need it? Arts Unit KD

4 CONS: it is much harder to get an accurate picture, plus rely on the skills of the operator to capture the overview. Box Office and Gate returns PROCESS: For counting entries at the gate and postal information gives the reach of your activity PROS: The more advanced your technology the greater the range of information you can draw down. CONS: Can demand fully trained staff to operate, and may be costly depending on which package/system you adopt. Stewards and Police PROCESS: For estimates of attendance, espcieally at free events when other techniques are not available PROS: Professionals who deal with people counting on a regular basis often reach accurate estimates. CONS: If you rely on Police for example, you need to take into account that they are also there for other duties. It is an estimate and you must present it as that. 2. Measuring Responses at the time Questionnaire (closed questions) PROCESS: Tick boxes and yes-no answers, basic questions or fact, or asking for a score out of 5, 10 or a %. Many questionnaires lead on from open to clsed questions. Static techniques are used to get immediate responses. PROS: Provides a range of basic information and can be answered quickly. Greater success is achieved if linked to a prize draw etc. CONS: Often resented and messy, takes time and space, return rates are often poor 25% is high (source VAN). Questionnaire (Open Questions) PROCESS: Questions encourage a fuller response, and more considered answers. PROS: Gives more scope for people to express their thoughts and opinions. This system can lead to more long term connections to audiences or groups etc. CONS: Slower to complete and therefore fewer are returned. These questions much harder to design and to evaluate. Interview PROCESS: Individual or group recorded on paper, video or audio. Can either offer open or closed questions. PROS: Human and face to face, more spontaneous than many other techniques. Interviews can draw responses which other methods may miss. It can also be mobile, adaptable and friendly. CONS: Disruptive in the wrong scenario. It is very dependent on the quality of the interviewer. Questions and answers can be Evaluation: Why do we need it? Arts Unit KD

5 misunderstood. Taped interviews can be time consuming to process. Avoid leading questions. Camera and Video PROCESS: Photographs or video footage by designated person or a number of participants/documenters. PROS: Vivid and colourful, providing insights into atmosphere and feelings generated. Avoids exclusion of individuals who may not want to engage in other feed back/evlauation methods. A professional provides quality insight, with amateur work collated into a open social documentary and collage of the event/project. Great for future marketing and future funding bids. CONS: The least objective as the camera only captures the view of the photographer. Does not involve others in the evaluation process, Video footage harder to evaluate, distribute and if you do not have the time/budget to edit into something concise and useable. Graffiti wall, comments and Post-It notes PROCESS:These ra static techniques that offer people the chance to make immediate responses and on their own terms. PROS: Simple, open, informal and relatively private. Often elicits honest and unexpected responses. Very good for witty, catchy comments and for increasing a sense of participation. Demands little staffing. CONS: No control over the kinds of comments/feed back you will receive, or statements made. Not a representative model. Archery Target and Stickers PROCESS: Targets can be divided into sections to reflect the different elements. A simple gut-feeling response method. PROS: Simple and low tech. Easy to process afterwards CONS: No room for a more qualified response. Not very private anwers may be affected by others reponses or peer pressure. The pin-board questionnaire is a more formal method of this. 3. Measuring responses afterwards Group Feedback PROCESS: A means of consulting all participants soon after the event or activity ends. PROS: Deliveres considered responses form people on the inside and whose future commitment and involvement may be necessary for on going development of the project. The group discussion should be broad but it may be necessary to consult different groups separately, or focus on one area for a select period of time, in a what went well and what could we have done better exercise. CONS: It is vital the sessions are prepared and facilitated and do not become an arena for competing views. All points raised should be recorded and time allowed for all contributors to have their say. The recorders opions must not intrude. Evaluation: Why do we need it? Arts Unit KD

6 Published and virtual consultations PROCESS: Magazines, newsletters, letters, m discussion groups and web based chat/social networks. PROS: Provides feedback over time, the longer the discussion board thread is online the more in depth responses you are likely to gain. Gives freedom for participants time to consider response and submit them in their own time and in privacy. Useful in formulating new policies, manageing debates and drawing in new members. CONS: Lacks immediacy, and is hard to integrate into any timescale of evaluation. Tends to be heavily mediated. The following are a series of example formats which you can use, change, and fit to your requirements, and have been sourced from Voluntary Arts Network. Please remember they are only to give you an idea of how to go about in evaluating your project, and that evaluation should not be seen as an add on but integral to any projects and should be included in your project management plan at the very earliest stage. Should you require further information or want to discuss your project and your ideas around evaluation, please contact the Arts Unit direct on or arts@plymouth.gov.uk. Please also refer to other useful resources such as: Arts Council England: Voluntary Arts Network Kath Davies Principal Arts Officer Evaluation: Why do we need it? Arts Unit KD

7 1. ABOUT YOURSELF/ORGANISATION Thinking about yourself/your organisation, what word, or words, would you choose? 1. What kind of person/organisation am I? 2. My aims and objectives are (Sum up you aims in a Maximum of 50 words) 3. Being involved gives me (fulfilment, satisfaction, career step, creative opportunity) 4. Who am I aiming to engage with (community groups, peers, members of the public, children and young people) Evaluation: Why do we need it? Arts Unit KD

8 2. ABOUT THE GROUP AND ITS ACTIVITIES What three things have you got out of your work with the group? / or What three tings have your got out of attending Which of these words describe what you think/felt about the groups activities? (mark those that you feel apply to you) HARD WORK EASY FUN BORING USELESS VALUABLE NO NEW IDEAS GAVE ME NEW IDEAS NEW DONE IT ALL BEFORE TOO MUCH THEORY TOO PRACTICAL STATIC CREATIVE WORKED ALONE WORKED AS A GROUP Evaluation: Why do we need it? Arts Unit KD

9 3. YOU AND YOUR TIME IN THE GROUP In an average week, how much time do you spend on the groups activities? In the first set of circles, show the time spent in hours, e.g. 2 hours, with the whole group. Then in the second set, show the time spent on each activity outside the group. TIME SPENT ON ACTIVITIES WITH THE GROUP TIME SPENT ON ACTIVITIES OUTSIDE THE GROUP Key: 1 RESEARCHING 2 THINKING 3 TALKING & PLANNING 4 REHEARSING 5 PRACTISING 6 OTHER STUFF HAVE YOUR SAY: 1: Do you perform or exhibit with the group? YES NO If not, would you like to? YES NO 2. Do you go on outings with the group? YES NO If not, would you like to? YES NO 3. Do you work with other art groups? YES NO If not, would you like to? YES NO Evaluation: Why do we need it? Arts Unit KD

10 4. THE GROUP AND YOURSELF Look at the statements below and say whether you agree or disagree with them. You could also number them at the side form 1 10 (where 1 is important and 10 least important and you could include a column to order the importance of the statements, e.g. what is most important to you? Being a member of this group has Agree Disagree 1. Changed my hopes and plans for the future 2. Changed my social life for the better 3. Made me feel better about myself 4. Helped me develop career plans 5. Made me likely to take part in other activities 6. Made me aware of other opportunities 7. Helped me to do new activities/find new interests 8. Taught me skills/techniques in something new 9. Given me a lot of fun 10. Your skills Would you say you learn new skills in this group? If so why? Evaluation: Why do we need it? Arts Unit KD

11 4. A SIMPLE ONE WORD QUESTIONNAIRE Here s an example of an extremely simple and un-intrusive questionnaire, suitable for catching people s gut feeling about an event. Can I have a word? One word to describe the atmosphere of this event One word to describe the activities One word to describe the music/art One word to describe the facilities And finally, one word of constructive criticism By the way How far did you come to attend this event? How many were you in your party? Evaluation: Why do we need it? Arts Unit KD

12 5. THE ARCHERY TARGET The archery target is another form of recording responses at a particular event (the one shown below is designed for a music event). Targets can be anything from 12 cm square 3 metres or more! And responses can be recorded using stickers, post-it notes, marker pens etc. Using the target. The board is divided into eight sections (there could be more or less to suit your needs) and each section is further divided into four poor, fair, good and best in which people can register their responses to individual aspects, e.g. if you thought that the atmosphere was poor, put a sticker on the outside ring (marked poor ) of the atmosphere section. POOR FAIR The audience GOOD The music BEST The facilities The access Public information Evaluation: Why do we need it? Arts Unit KD