Debt Gateway/Money Active at Plymouth CAB

Transcription

1 Breaking the Cycle Embedding financial capability within debt advice at Plymouth Citizens Advice Bureau

2 Acknowledgements This research was funded by the Royal Bank of Scotland Group and written by Helen Aynsley, Evaluation and Policy Manager on RBS Innovate at Toynbee Hall. The author would like to thank Jane Guy, Training Manager at Plymouth Citizens Advice Bureau, for coordinating and providing the data for this report and for her on-going support and collaboration. Thanks also go to Sharon and Bill at the bureau for providing additional information and allowing the evaluator to sit in on their sessions. 2

3 Contents Executive Summary Introduction to RBS Innovate The Debt Gateway and Money Active at Plymouth CAB.. 10 Plymouth Citizens Advice Bureau.. 10 The Debt Gateway and Money Active Financial capability and debt advice. 12 Aims 14 3 The Model. 14 Assessment Money Active workshops.. 15 Follow up 17 Diagram: The Debt Gateway and Money Active Evaluation. 19 The trouble with evaluation. 19 Evaluation Stage 1: Set up of evaluation 21 Evaluation Stage 2: Methodology. 23 Debt Gateway and Money Active evaluation framework 28 5 Findings 29 Evaluation of the project Evaluation of the model. 38 Evaluation of the evaluation Conclusions and recommendations Appendix A References

4 Executive Summary 1. Introduction to RBS Innovate 1.1 From 2005 to 2009 RBS provided funding for small projects to test out new approaches to financial inclusion. This programme, called RBS Innovate and run through Toynbee Hall, helped 13 organisations get new financial inclusion initiatives off the ground. After three successful years the decision was taken that, in order for projects to provide maximum learning, it was important to focus on lessons learned and the long-term impact of interventions. The focus therefore shifted to the evaluation of projects with emphasis on three aims: 1. Help projects to set up their own internal evaluation structures; 2. Share the resulting evaluation frameworks with the rest of the financial inclusion sector and apply to other organisations where appropriate; 3. Showcase the work of the projects and offer models of replication. 1.2 RBS Innovate s role was therefore restructured to work with four organisations, each based in different areas of the UK. The decision of which four to choose was based not only on the projects themselves but also on the combination of the four projects it was important to ensure a good mix of size, remit, population served and geographical location. The four selected projects are: 1. Money Wise at Centrepoint (London) 2. Debt Gateway and Money Active at Plymouth CAB (Plymouth) 3. Save and Insure at Prince Bishops Credit Union (Consett) 4. Your Money Garden (Belfast) 2. Plymouth Citizens Advice Bureau 2.1 As well as identifying what specialist advice debt clients need, the Debt Gateway at Plymouth CAB identifies which clients could benefit from additional financial capability training. Those that could benefit are referred to a Money Active session either a workshop or a one to one. The aim of Money Active is to increase service users knowledge of budgeting, borrowing, savings, debt and where to go for additional support and help. There are three key aims to the Debt Gateway/Money Active model: 2.2 Increase financial capability: Plymouth CAB believes it is important to increase the knowledge, skills and confidence of service users to manage their money more effectively and take responsibility for personal financial decisions. 2.3 Decrease repeat appointments within the bureau: Debt service users often come back to the bureau multiple times, sometimes even for the same issue. It is hoped that by referring a number of service users to financial capability training, it may be possible to decrease the amount of clients who come back in for more advice, especially for the same issue. 4

5 2.4 Explore implementation: Consider different procedures to identify the best way for Plymouth CAB to embed financial capability within debt advice without taking away from debt service delivery. 3. The model 3.1 There are three steps to the Debt Gateway/Money Active model: 3.2 Assessment and referral: All debt clients that come to Plymouth CAB are assessed by the Debt Gateway. The advisor at the Debt Gateway assists clients with queries that can be handled there and then and refers clients that need specialist advice to the appropriate service. Clients that they feel could benefit from money management training are referred to a Money Active workshop or one to one. 3.3 Service delivery: Workshops are delivered in a group setting and cover four key subject areas: budgeting, borrowing and saving, dealing with debts and where to go for help. A4e sessions are delivered to clients facing barriers getting into or returning to work and the bureau also holds training sessions for advice staff who work in Plymouth but not in a CAB. 3.4 Follow up: Plymouth CAB do not usually follow up with clients who attend a Money Active session but they did so as part of this evaluation. The results were encouraging and the follow up not only provided clients with the opportunity to reflect on their progress since attending a Money Active session but provided useful feedback for the staff and volunteers who run the sessions. 4. The evaluation 4.1 Key questions to ask when evaluating an internal project: What are the key objectives of the project? Establish the purpose of the project and its aims and objectives right at the start. In the case of RBS Innovate, an initial meeting was held with Gill Tishler and Jane Guy at Plymouth CAB in order to gain an understanding of the Debt Gateway and Money Active and what they were seeking to achieve for participants through Money Active. It is important to have this discussion at the start of the evaluation as it lays the groundwork for subsequent evaluation stages How to strike the right balance between perfect methodology and actual capacity to deliver? In order to successfully embed evaluation activities within an organisation, it is necessary to be realistic about what is achievable as regards funding, staff, and available time. As Jane was the prime contact for the duration of the evaluation we designed an evaluation framework that would realistically fit in with her time and other commitments. This is an important process for any organisation setting up an internal evaluation; you could design the most robust evaluation framework that would guarantee a wealth of rich information, but if it is not going to be realistic to gather that information it will not be achievable. 5

6 4.1.3 What information is already being captured? Plymouth CAB already capture a lot of data for their own use and to report back to funders. Rather than starting from scratch it was useful to establish what information had to be captured for other purposes (i.e. funder reports) and what information was already available so that work was not replicated What kind of evaluation do we want to undertake? Do we want the evaluation to be formative or summative? A formative evaluation asks how can the programme be improved? and is undertaken in order to provide feedback with the goal of progress. Summative evaluations on the other hand ask what is the overall merit or worth of the programme? Should it be modified? Should it be continued? Gill and Jane decided that the evaluation at Plymouth CAB should be both formative and summative What methods are we going to use? Sometimes quantitative data such as numbers and figures will provide the most important information while other times more descriptive, qualitative information will be most useful; sometimes it will be both. Whilst an external evaluator can provide information about the different kinds of data and their respective benefits, it is up to the organisation to decide which method will give the richest data to suit their needs and help them to inform their decisions going forward. 5. Evaluation of Plymouth Citizens Advice Bureau 5.1 The most common reason a client was referred to a Money Active session (36%) was so that they could learn to prevent and deal with their debts followed closely by learn to budget their money and complete a financial statement (30%). 5.2 When asked what they would change about the session 99% of respondents said nothing % of participants said that they felt their attitude towards money had changed since attending the workshop, with 83% saying their attitude towards budgeting had changed % of respondents were able to give an example of an affordable source of credit and 79% of respondents were able to give an example of a priority bill weeks after attending a workshop. 6. Evaluation of the model 6.1 Plymouth CAB found that if a client takes part in a Money Active session any subsequent debt case work takes a third of the time it takes for clients who have not been through a Money Active session. 6.2 Take up of workshops was low throughout the duration of the evaluation but staff at Plymouth CAB were able to identify factors to mitigate this and have adjusted the service accordingly. 6

7 7. Evaluation of the evaluation 7.1 This model of evaluation provides a good balance between internal and external; the evaluation framework was developed in partnership with the external evaluator and Plymouth CAB and all major decisions were made jointly. Balanced alongside this is the fact that the evaluator offers an outside view and a different perspective. 7.2 Even though staff at Plymouth CAB followed up with Money Advice clients after the session for evaluation purposes only, this proved to be such a useful exercise for both service users and staff that it is recommended this be continued going forward. 7.3 Literacy, mental health and confidence issues means that Plymouth CAB will potentially want to explore other ways of evaluating the course in future that are not reliant on paper-based tools. As some participants need help filling in forms it means that the evaluation can take up a disproportionate amount of time and this takes away from the time needed to actually deliver the service. 7

8 Part 1 Introduction to RBS Innovate From 2005 to 2009 RBS provided funding for small projects to test out new approaches to financial inclusion. RBS called this programme RBS Innovate and through it helped 13 organisations get new financial inclusion initiatives off the ground. The overall aim of the RBS Innovate programme is to increase financial inclusion through the improvement of local financial inclusion services and the emphasis throughout the programme has been on innovative interventions, meaningful work with long term impact, improvement and development and dissemination of learning and best practice. The 13 projects that RBS Innovate funded were very diverse some helped individuals gain access to financial products, some produced guides and resources for trainers, others supported housing association tenants with specific needs but all focused on helping people experiencing financial exclusion. As the programme progressed RBS decided that in order for financial inclusion interventions to provide maximum learning for the sector, it was important to focus on lessons learned and long-term impact. The focus of RBS Innovate therefore shifted in the period from funding interventions to the evaluation of projects with the emphasis on three aims: 1. Help projects set up their own internal evaluation structures; 2. Share the resulting evaluation frameworks with the rest of the financial inclusion sector and apply to other organisations where appropriate; 3. Showcase the work of the projects and offer models of replication. Good projects are often under-evaluated so exploring how they can be scaled up and how this can work across different geographies and with different clients groups is key to promoting success in financial inclusion. RBS and Toynbee Hall therefore restructured the role of RBS Innovate to work with four organisations based in different areas of the UK over the course of one year. The four projects were selected using an open tender process. In December 2009 RBS Innovate announced that evaluation support was available for any organisation working in financial inclusion or financial capability. This announcement was disseminated via Transact, the National Forum for Financial Inclusion, the DWP Champions Initiative and the Citizens Advice National Financial Capability Forums. The application form was posted on the Transact web-site and the deadline was set for January In all, 33 applications were received and 4 projects were selected. Using an open tender process guaranteed that every organisation that applied was interested in setting up their own internal evaluation frameworks, having their organisation evaluated and sharing learning with the sector. Choosing only four organisations to work with was difficult as there were many interesting, innovative organisations interested in RBS Innovate and in the end the decision was based not only on the projects themselves but on the combination of the four projects; it was important to ensure a good mix of size, remit, population served and geographical location. The four organisations selected are outlined below. Moneywise at Centrepoint (London) Centrepoint s vision is to end youth homelessness and Moneywise fits into this vision by providing training on financial capability to educate Centrepoint young people 8

9 about personal financial management. While advice services regarding financial capability are available, young people find them difficult to access due to travel costs, intimidation and their adult focus. Embedding financial capability into wider work with homeless young people and young people at risk of homelessness helps to create socially and economically viable citizens. The objectives of Moneywise are achieved through AQA accredited workshops, one to ones with young people, capacity building through staff training and signposting to specific agencies where necessary. This holistic programme ensures that Centrepoint residents have access to high quality advice and leave Centrepoint services fully informed of the importance of financial capability. Money Active at Plymouth Citizens Advice Bureau (CAB) (Plymouth) At Plymouth CAB at least 70% of the issues staff deal with are related to financial inclusion and their Debt Rights and Remedies Unit provides specialist money and benefits advice to help tackle these issues. Through providing these services the bureau has become aware that, unless they find a way of equipping their service users with the skills and confidence to manage their financial affairs more effectively, they are likely to return possibly within a fairly short timescale with the same issues. To eradicate this, Plymouth CAB decided to refer a number of debt service users to management sessions delivered within the Money Active programme. Some service users are referred to a one to one session and some to a group session, both of which help service users deal with issues such as budgeting, borrowing and saving, debt and where to get help. Save and Insure at Prince Bishops Community Bank (Consett) Many people fall into financial difficulties when an unforeseen event such as a burst pipe or burglary occurs and they have no insurance or savings to replace lost items. As a result they often resort to doorstep lenders. Problems of loan sharks were recently highlighted when a resident paid 88,000 interest on a 500 loan. Through the Save and Insure scheme, tenants of Derwentside Homes in County Durham have been encouraged to open a credit union account. If they save at least 10 a month for six months they will receive a 20 credit to their account from the scheme. In addition new members receive a voucher worth 40 towards Derwentside Homes/RSA low cost contents insurance which covers a six month premium, based on a 9000 sum insured. After saving for twelve weeks individuals also have access to low cost loans from the credit union. Your Money Garden (Belfast) Your Money Garden is a project that provides financial capability education training primarily to women across Belfast, Newtownabbey and surrounding areas. The training is delivered across a series of workshops and aims to address gaps in knowledge around finances. The training helps participants to develop their money management, budgeting and practical skills so that they may increase their level of financial knowledge and economic activity. 9

10 Financial learning sessions are delivered to existing groups (community women s centres, etc.) as an effective way of reaching people in need. In order to ensure that participants have all the support they need, Your Money Garden works with partner organisations such as local CAB and credit unions. For the past year RBS Innovate has been working with the four projects to evaluate the impact of their work, provide evaluation frameworks for the sector and use lessons learned to make recommendations on how to replicate these models in different areas of the UK. Part 2 The Debt Gateway and Money Active at Plymouth CAB As well as identifying what specialist advice debt clients need, the Debt Gateway at Plymouth CAB identifies which clients could benefit from additional financial capability training. Those that could benefit are referred to a Money Active session either a workshop or a one to one. The aim of Money Active is to increase service users knowledge of budgeting, borrowing, savings, debt and where to go for additional help and support. It is hoped that providing this additional support will increase the financial capability of service users and therefore reduce future debt problems. 2.1 Plymouth Citizens Advice Bureau Plymouth Citizens Advice Bureau (CAB) is an independent registered charity that offers free, impartial, independent and confidential advice to their service users. They are part of the national network of Citizens Advice Bureaux that can be found across the UK see for your local bureau. Plymouth CAB operates across four key principles: 1. Free services: All of the services of Plymouth CAB are free because they believe it is everyone s right to have information about laws and services that affect them. They also believe they everyone should be able to make use of these laws where necessary and have access to the services that they are entitled to. 2. Confidentiality: All information given to Plymouth CAB is kept strictly confidential. 3. Independence: The service provided by Plymouth CAB is independent. Because of this, they are able to offer impartial advice to all enquirers and to take up any issue with the appropriate authority on behalf of individuals or groups. 4. Impartiality: All services offered through Plymouth CAB are impartial and are available to everybody. They believe it is important to value diversity, promote equality and challenge discrimination. Plymouth CAB gives information and advice to service users on a number of different subjects. Advisors are trained to deal with queries relating to housing, consumer issues, debt, welfare benefits, employment, family and personal issues, nationality and immigration. They help people in the community in a number of different ways; specific issues can be dealt with through face-to-face advice, on the telephone and by and additional information is also available on-line at Covering such a wide range of issues means that 10

11 Plymouth CAB is able to take a rounded view of the problems people face. This is important as they find that one significant change in circumstance can trigger a whole raft of problems: for example, losing a job can lead to the loss of a home, breakdown of a relationship and problems with debt. Falling ill can result in complicated benefit applications and anxiety about employment and bills. As a result, Plymouth CAB feel it is important to provide a holistic service to service users so that all of their needs are met wherever possible. Helping service users with all of their issues can also help tackle the root cause, thus preventing problems occurring in the future. As well as providing services under their roof, Plymouth CAB also provide the secretariat for the Advice for All group (A4A) which was launched in September A4A coordinates advice services across Plymouth and focuses on improving the quality and consistency of advice and also increasing the capacity to give advice. As part of this they work on referral mechanisms and signposting and how to work well in partnership. They also produce local fact sheets covering a wide range of topics including debt, mental health, benefits, job seeking, employment, drug and alcohol misuse, housing and energy efficiency. The full set of fact sheets can be found at and for further information on Advice 4 All visit Plymouth CAB has a good working relationship with the funder of its core service, Plymouth City Council, who has a city-wide financial inclusion strategy which supports the work of the CAB. The CAB has also developed an exceptional relationship with the local water company, South West Water, with an independent water gateway for their customers in debt across Devon & Cornwall which helps and advises customers where savings can be made andcan offer grants to those in hardship to clear their water debts. 2.2 The Debt Gateway and Money Active Of the 20,000 issues dealt with by Plymouth CAB in 2008/2009, at least 80% were related to financial inclusion i and they handled over 50 million worth of debt on behalf of their service users. Over the course of the past 5 years they have built up their Debt Rights & Remedies Unit which currently employs 12 staff. In order to assess which service users are in greatest need and ensure that they are seen as quickly as possible, service users who attend the drop in are given a short diagnostic interview. This is referred to as their Debt Gateway and it enables the bureau to gather brief details of the client s enquiry, assess the complexity and urgency of the issue and determine the best way to deal with it. Sometimes all a service user needs is some further information on their query, such as a leaflet or signposting to a web-site, whereas at other times they may need a referral to a specific service within the bureau or even out to an external organisation. A key role of the Debt Gateway is also to identify who could benefit from further financial education and capability training; those who could benefit are referred to a Money Active session. Money Active is a national scheme run through Citizens Advice across the UK and is funded by Nationwide Building Society. The aim of Money Active is to train and support staff and volunteers in Citizens Advice Bureaux across the UK to carry out financial education work in their area. It is a three year project, running from , that supports the recruitment and training of 1,300 11

12 volunteers to deliver financial education sessions. At Plymouth CAB these sessions take the form of either a two hour workshop or a one to one session, both of which cover basic skills around budgeting, banking, credit and saving. Clients are given the option of attending a workshop or a one to one session in recognition of the fact that they may not be comfortable sharing in a group session. In December 2010 Citizens Advice released a statement with interim results of Money Active. Two thirds of people who attended a Money Active session report feeling more confident about handling their finances as a result. Over a third said they would now open a savings account, and a third planned to set up a budget to help them manage their money. A further 19% intended to get a better deal from their energy supplier. It is estimated that Money Active has so far reached around 80,000 people ii. 2.3 Financial Capability and Debt Advice As mentioned above, debt advice is a core part of the service that Plymouth CAB provides and they have a well-established team who provide case work to over 1,000 service users per year. However, they know that, unless they find a way of equipping debt service users with the skills and confidence to manage their financial affairs more effectively in the future, they are likely to return to the bureau with the same issues. It is for this reason that Plymouth CAB place a particular emphasis on delivering financial capability sessions to their own debt service users. But what is financial capability? And how does it relate to debt advice? The true origin of the term financial capability is not entirely known but one of the first places it is mentioned is in The Financial Services and Markets Act of 2000 where it was stated that one of the four statutory objectives of the Financial Services Authority was to promote public understanding of the financial system iii. In 2003 the FSA set the foundations for the National Strategy for Financial Capability in place and established the Financial Capability Steering Group to oversee the work. The aim of the new strategy was to provide customers with the education, information and generic advice needed to make their financial decisions with confidence iv. Financial capability also has its roots in financial literacy. Around the same time as the National Strategy was set in place, the FSA joined forces with the Basic Skills Agency, as well as a range of expert partners across the financial capability sector, to develop a model for drawing the different elements of financial capability under one banner. They did this through the development of the Adult Financial Capability Framework: Following the recommendations of the Adult Financial Literacy Advisory Group report and the introduction of the new adult literacy and numeracy core curricula we were keen to develop a framework that outlined the skills and competencies that were deemed necessary for financial capability. The new framework broke financial capability down into three elements: 1. Financial knowledge and understanding 2. Financial skills and competence 3. Financial responsibility In 2005 the Financial Services Authority conducted their baseline survey which measured the financial capability of 5,300 adults in the UK. By this point, 12

13 understanding of financial capability had evolved and so the new survey measured financial capability along five new components: 1. Making ends meet 2. Keeping track of your finances 3. Planning ahead 4. Choosing financial products 5. Staying informed about financial matters These five elements were deemed to cover the core skills necessary for an individual to be considered financially capable and continue to frame understanding of financial capability today. There are a number of guides for providing financial capability support, both one to one and in a workshop environment. But how do you provide this work alongside the case work that goes with advising a service user with debt? Or with a group of people who are struggling to make ends meet? There are a few examples of financial capability delivery, including within the Citizens Advice Bureau: CAB Financial Capability Pilots: From August 2008 until July 2009 Citizens Advice was funded by the Department for Business, Enterprise and Regulatory Reform (BERR) to run a pilot project in 10 UK locations to people who were excluded from mainstream financial services. The project also provided training to frontline workers who delivered services directly to people experiencing financial exclusion. The FinCap project aimed to improve clients financial capability and encourage them to pass their learning on to friends and colleagues. In total, the FinCap project delivered 458 training events and trained 1,083 service users and 1,433 staff. The number of staff who said they were fairly or very confident in supporting service users on financial issues increased from 48% to 94% following the training. Service users who described themselves as fairly or very confident in managing their finances increased from 60% to 89% following the training v. CAB Financial Skills for Life: Launched in 2002, the Financial Skills for Life project was a joint venture between Citizen s Advice and Prudential that piloted new ways of delivering preventative financial education to a range of adults at risk of financial exclusion. The pilot ran for three years and in total there were nine pilots based in local bureaux that focused on developing good practice in delivering financial education to adults. The evaluation of the project was conducted by ECOTEC and identified some significant outcomes for service users. In the short term individuals involved in the project displayed greater confidence around personal finance and longer term benefits showed that these outcomes lasted; individuals surveyed months later showed that they had changed their patterns of behaviour, improved their economic wellbeing and increased their savings vi. Capitalise Financial Capability Pilot: Capitalise is London s debt advice partnership comprised of 17 organisations and managed by Toynbee Hall. 56 advisors from these organisations deliver face to face debt advice to people experiencing social and financial exclusion across the capital. The financial capability pilot was based on the idea that increasing financial capability is more effective as a crisis intervention than just dealing with specific debts and the aim was to improve the financial capability of Capitalise service users, including their skills, knowledge, awareness, confidence, 13

14 attitude and motivation in relation to their finances. For 12 months Capitalise ran the pilot, delivering financial capability to a sample of debt service users so that they had a test group and a control group. The results were significant: 78% of the pilot group felt more able to manage their money compared to 51% of the control group; 78% of the test group felt more able to plan ahead compared with 34% of the control group and 47% of the test group felt more able to choose financial products compared with 31% of the control group vii. These three examples illustrate the effect that financial capability can make on an individual s experience of debt as well as long term financial well being. Findings from all three reports suggest that embedding this work within wider debt advice structures is a key challenge; Capitalise, for example, found that the target pressures and inflexible structures of debt advice meant that there was little time for any other work. It is hoped that through this evaluation of the Debt Gateway and Money Active at Plymouth CAB, some of these challenges can be addressed. 2.4 There are three key aims of the Debt Gateway and Money Active working together: 1. Increase financial capability: While there is no one established definition of financial capability, Plymouth CAB focuses on increasing the knowledge, skills and confidence of service users to manage their money more effectively and take responsibility for personal financial decisions. Effective financial management is important for a number of areas in life, so the first key aim of Money Active is to increase the financial capability of service users at Plymouth CAB. 2. Decrease repeat appointments within the bureau: Debt service users often come back multiple times, sometimes even for the same issue. It is hoped that, by adding in financial capability training, it may be possible to decrease the amount of clients who come back in for more advice, especially for the same issue. 3. Explore implementation: Compare experiences of one to ones and group sessions to identify which are most successful in tackling debt service users money problems. Explore different methods of working to uncover the best way for Plymouth CAB to embed financial capability within debt advice without taking away from debt service delivery. Part 4 The model There are three key steps to the Gateway Service/Money Active model: 1. Assessment and referral 2. Service delivery 3. Follow up The below model represents the current structure of the Debt Gateway/Money Active model at Plymouth CAB as well as their plans going forward that have resulted from this evaluation. For further information on the findings from this evaluation, please see the section on findings. 14

15 4.1 Assessment and referral Plymouth CAB is open from Monday to Friday, 9am to 5pm for visitors and interview sessions. The Debt Gateway is open for those without an appointment from 10am to 1pm Monday to Friday and from 2pm to 5pm on Mondays. As a service run by a fulltime worker, the Gateway service has been running since November On average they will see around 10 service users per morning and issues can be very far ranging. Sometimes the issue will not actually be about debt eviction proceedings for example are often dealt with by those specialising in housing advice in which case the service user gets referred to another department. The Gateway advisor records the reason why the service user is seeking advice and will do what they can in the session but for any other specialist advice or for an extended advice session they will refer the service user to a case worker. Sometime a service user just needs signposting to a particular service or assistance in filling in forms in which case the Gateway advisor helps them with that at the time. This ultimately saves the bureau time in the long run as otherwise that service user would require a one to one appointment with a case worker. At the time of the evaluation Bill, the Gateway advisor, estimated that he referred approximately 10% of all service users that came to the Debt Gateway to a Money Active session. Referrals are made at the discretion of the Gateway advisor and based on their assessment of the service users money management skills. If a service user s debts appear to be as a result of their poor money management, for example, they will be referred to a Money Active session. The advisor records the reason for this referral as well as what type of session they referred the service user to (group or one to one). In most cases clients are referred to a workshop as this saves time and results show that participants benefit from the group environment. If a client is insistent that they do not want to attend a group session, or if their problem is very specific, they are referred to a one to one. Because referrals are up to the discretion of the Debt Gateway advisor, extensive training is very important. Money Active workshops are also advertised through flyers in the main bureau reception and in the interview rooms for anyone else who is interested in attending. Clients may also be referred to a Money Active session by a frontline worker. 4.2 Service delivery Plymouth CAB has been running Money Active sessions for approximately 4 years but in 2009 they decided to look at further ways of developing these sessions to meet the needs of their service users more effectively. They began this process by setting up regular team meetings with staff that were directly involved in the project. Through the course of these meetings they discussed how many of their service users were in financial difficulties simply because they had never had an opportunity to learn money management skills. A number of these service users were not even in debt but could benefit from additional training to prevent money problems developing in the future. As a result of these conversations a new training workshop was developed using power point, and additional materials were developed and collated for service users to take away with them e.g. budget sheets, money saving tips, Money Made Clear leaflets, etc. The presentations and course materials are updated on a regular basis to make sure that all information is current. At all of the workshops at least two members of staff or two volunteers are present to facilitate 15

16 the session and support those attending. Service users will often need support in filling in forms or completing course activities so it is useful to have more than one facilitator present. As the workshops are delivered in a group environment, the first part of the session involves setting ground rules for conduct. For example it s fine to disagree with and challenge each other but it s important to challenge views and not individuals. As a lot of the topics covered in the sessions touch on sensitive issues, such ground rules are important for engendering a safe and supportive environment. The workshop is then delivered under the heading of Money Management Skills and the key aim of the session is stated as to provide you with the skills to manage your money better. The facilitator then delivers the workshop along five main themes: Budgeting Borrowing and saving money Dealing with debts Signposting Questions and answers During the session on budgeting participants are asked to complete a financial statement. A financial statement is a form containing boxes for income and expenditure that is filled out by an individual based on the details of their personal financial circumstances. It is often used in debt advice as a tool for negotiating with creditors; once income and expenditure are recorded, whatever money is left over is used towards payment of debts. It can be a useful budgeting tool on its own however and people are often surprised when they are faced with the full picture of their income and expenditure. As one participant said in a Money Active workshop it s a real eye opener! Participants can be hesitant about staying at the start of the session when faced with the group environment. The session s coordinator will invite them to stay for a few minutes and see how they feel, emphasising that they are welcome to leave at any time. In actual fact, the group environment is an enormous strength of the service delivery and participants will usually stay throughout the duration of the workshop and contribute to the session. The workshops are very interactive and service users are encouraged to join in and participate which facilitates their learning. For example, during the session on borrowing and saving money service users are asked for examples of sources of credit that they use as well as sources of credit that they will not use. During one session they added loan sharks to the list as well as family and friends while acknowledging that the latter can be emotional. They talked about general distrust of banks banks are just organised mafia and interestingly, only one person out of the whole group had heard of a credit union. When they cited the Provident as a source of affordable credit the facilitator asked them to guess how much the Provident charges in interest and they guessed 29%; in actual fact the average Provident loan is charged at 272.2% APR viii. Delivering sessions in this group environment means that participants can learn as they go and ask the facilitator questions. It also means that they can learn from each other. One participant shared that her grandson uses her car but does not contribute any money towards maintenance or petrol and that the costs really add up. Another participant said that she had experienced the exact same problem with a family member and, in her case, she had asked for a financial contribution and that this 16

17 tactic had worked for her. This peer learning would not have arisen in a one to one it can only come from shared experiences. Plymouth CAB also provides training through A4e for customers wishing to get into work or to return to work. The workshop outline is almost exactly the same except that it focuses on employment with the first session covering things that stop someone returning to work. Aside from that, the workshops at A4e are delivered using a power point presentation and participants are given resources to take home with them, just like the other Money Active workshops at Plymouth CAB. Plymouth CAB also provides training to the staff and volunteers that support clients dealing with money problems. This training is delivered through interactive sessions where staff discuss their various client groups and how they might deliver training to them. For example, some clients have learning disabilities so a particular focus needs to be on independent living and where they can go to for further support. Clients with mental health issues need to know about other services in their area that can help them, such as Mind or Plymouth Options. During the training staff are shown the power point presentations that are used in the workshops with clients and are given guidance on how to deliver each session. For the session on budgeting, for example, it is important for staff to know how to challenge high expenditure on non-essential items that clients write into their financial statements. Staff are also given information on local doorstep lenders, local credit unions and how to make referrals. Follow up Currently Plymouth CAB do not follow up with service users who are referred to a Money Active session. As part of this evaluation, however, they did a follow up phone call to a sample of participants and the results show that participants found the sessions very useful, retained knowledge and started putting their financial plans into action. For further information on the findings of this follow up, see the section on findings. Going forward, it would be useful to continue to follow up with a sample of clients who are referred to a Money Active session. This would have two benefits: it is useful for evaluation purposes as it can demonstrate the impact that the sessions make on participants, but it is also an opportunity for participants to reflect on what they have learned. 17

18 The Debt Gateway and Money Active at Plymouth Citizens Advice Bureau Debt Gateway Money Active Workshop Money Active One to one Staff training Further specialist advice External agency Follow up (sample) Assessment and referral All debt clients come through the Debt Gateway Some clients simply need help filling in a form, information on their query or signposting to a different agency while others are referred for specialist advice Approximately 10% of clients are assessed as needing money management training and are referred to a Money Active session Delivery The majority of debt clients who are referred to a Money Active session are referred to a workshop Clients who are not comfortable with the workshop setting or have specific needs are referred to a Money Active one to one Staff are trained to deliver Money Active sessions through group training sessions Follow up Going forward Plymouth CAB are planning to continue following up with a sample of clients after they have attended the workshop to discuss their progress and what knowledge they have retained 18

19 Part 4 Evaluation Key questions to ask when evaluating an internal project: What are the key objectives of the project? How to strike the right balance between perfect methodology and actual capacity to deliver? What information is already being captured (for funders, stakeholder, etc.)? What kind of evaluation do we want to undertake? What methods are we going to use? 4.1 The trouble with evaluation A lot of organisations are required to evaluate their projects, often for the purpose of demonstrating to funders that agreed objectives have been met. These objectives are often demonstrated via outputs the number of clients seen, the number of workshops held, the number of hits to a web-site, the number of calls received, etc. This is useful information to have, but in order to demonstrate the true impact that an organisation is making on their client group, community or audience, an evaluation needs to go beyond outputs. In his original work in 1959, Donald Kirkpatrick made recommendations for evaluation that have laid the basis for a lot of thinking in the subject ever since. He argues that programme evaluation should concentrate on four levels: The four levels of programme evaluation: Level 1: Reaction evaluating the reaction of participants to the intervention Level 2: Learning measuring the knowledge, skills and attitudes gained from the intervention Level 3: Behaviour measuring the changes in behaviour that resulted from the intervention Level 4: Results relating the results of the intervention to wider organisational objectives Alongside outputs, many evaluations conducted in the financial inclusion and capability sector only capture the first level of evaluation. This is with good reason: reaction is the easiest to capture; it helps organisations to improve their programmes; and it can be easily analysed. Capturing reaction means asking questions such as what did you like about the training, what did you not like about the training and what suggestions do you have for the training going forward and is often recorded in an evaluation form after a course or an intervention has taken place. Just like outputs, reaction is useful information to capture, but it does not provide the whole picture. In order to capture real changes and attribute them to an intervention, it is necessary to go beyond this first level, to look at changes in behaviour and learning and, where possible, map these changes back to the activities of the organisation. 19

20 Confusion around how to capture outcomes is compounded by the fact that there is no single accepted definition of evaluation, let alone one method. Here are a few examples of definitions of evaluation that are widely used: Evaluation is about using monitoring and other information you collect to make judgments about your project. It is also about using the information to make changes and improvements (Charities Evaluation Service) Evaluation involves the systematic collection of data about the characteristics of a programme, product, policy or service. As part of this process, evaluation will often explore what needs to be changed, the procedures that are most likely to bring about this change, and whether there is evidence that change has occurred (Warr et al., 1970) Evaluation is the systematic assessment of the worth or merit of an object (Joint Committee on Standards for Educational Evaluation, 1994). This lack of one accepted definition is in part due to the fact that evaluation is a relatively new field; real interest in the area can only be dated as far back as the 1970s. The emphasis in early evaluations was on accuracy and specific measuring techniques but in recent years this has shifted ix. With the rise of action learning and self-development programmes, learning is now seen as arising within and through a situation rather than just through formal evaluations and issues of subjectivity and ethics are of prime concern in today s evaluations. Subjectivity is often at the heart of why many organisations struggle to evaluate their services beyond the reaction level. What actually is the relationship between inputs and outcomes? Can a change in learning or behaviour really be attributed to an intervention? What does success look like? These are some of the questions that organisations struggle with when designing their own evaluation strategies because they often involve making subjective judgments. This brings us to a key question: who is ultimately responsible for making these judgments? Who has a right to? While evaluation is a new field, financial capability is an even newer one, so there are not yet many established examples of evaluation best practice in the sector. Added to this is the fact that many outcomes in financial capability are soft outcomes which can be difficult to capture and, again, sometimes require subjective judgements to be made. Examples of soft outcomes include changes in knowledge, confidence, motivation and behaviour; all of which are extremely important and indicate that a real change has taken place but are not very straightforward to capture. So these are the key challenges that the sector faces when it comes to evaluating work in financial capability: There is no single accepted definition or method of evaluation; Financial capability is a young sector so does not have one accepted method of evaluation; Success in financial capability often means positive results in soft outcomes which can be difficult to measure; 20

21 Measuring outcomes often means making subjective judgements which some organisations may find difficult. Understanding that evaluating work in the sector is not always a straightforward process, a key aim of RSB Innovate is to provide the sector with evaluation frameworks that can be applied to different settings. Different organisations were deliberately chosen so that different approaches could be tested and best practice established. There was no guarantee of how this would work in practice, but the findings show that there are certain key identifiable themes across all four organisations. While the four organisations evaluated through RBS Innovate all deliver services directly to clients, the stages outlined below can also apply to other types of services working in financial inclusion. 4.2 Evaluation Stage 1: Set up of evaluation As described above, one of the three key aims of RBS Innovate was to design evaluation frameworks that are applicable to the whole sector. Whilst this is not necessarily going to be achieved through working with just four organisations, we deliberately chose organisations that are different so as to test out different approaches and establish common best practices. It is important to note that we did not start the process wedded to one evaluation framework or methodology. Instead, I (as the Evaluation and Policy Manager) started the evaluation process by visiting each individual project and approaching each evaluation with an open mind. In this way we were able to design an evaluation framework tailored to each organisation while at the same time staying mindful of key themes that emerged from each one. To start the evaluation process, an initial meeting was held in February 2010 with Gill Tishler, then Director of Plymouth CAB, and Jane Guy, A4A Training Manager. The purpose of this meeting was to learn more about Plymouth CAB and to gain an understanding of what they were looking to achieve through Money Active. As a well established organisation, they had already spent a lot of time thinking about what outcomes they wanted to achieve for service users and how their services could best enable those outcomes. As discussed previously, they had also been thinking a lot about their financial capability work and what more they could do for their service users. Their initial thinking had been to capture the cumulative impact of all of their work but, given the time constraints on the evaluation and current interest in embedding financial capability within debt advice, we decided to focus on this one area. The meeting presented an opportunity to talk about the service in depth and really hone down on the aims and objectives of the Debt Gateway and Money Active. This represents the first stage in developing an evaluation framework: decide what your key aims of your intervention are. As discussed above, a key question in evaluation is who has the right to make decisions on evaluation? With few exceptions it is the organisations themselves that know their client group or target audience best, as well as their organisational objectives. They also know the resource that has been invested in an intervention and the resulting benefit to their target group so it is often the organisations themselves that are best placed to decide the terms of an evaluation. 21