Monitoring and Evaluation of Employee Financial Wellness Programmes by Ulrich Walbruehl, Astrid Kuhlemann & Werner Just, Germany

Transcription

1 Kuhlemann, Walbrühl & Partner Business Development Consultants Bunsenstr. 5 D Gummersbach Tel Fax Monitoring and Evaluation of Employee Financial Wellness Programmes by Ulrich Walbruehl, Astrid Kuhlemann & Werner Just, Germany Introduction Monitoring and evaluation is a significant part of the quality management of any system implemented in a company. If efforts are not monitored, how would you know whether you are doing well, and whether you are still on the right track? There are many reasons for carrying out monitoring & evaluation activities. Debt counsellors and other stakeholders (including donors) need to know the extent to which the progam is meeting their objectives and leading to their desired outcomes. Monitoring & Evaluation builds greater transparency and accountability in respect of use of resources. Information generated through M&E provide project staff with a clearer basis for decision-making. Future planning and development is improved when guided by lessons learned from project experience. Monitoring and evaluation of financial wellness programmes When a financial wellness program is established, it is mandatory to decide how monitoring and evaluation will be conducted. It is an integral part of the program and should not be adopted later, but considered from the beginning. You need to think about different aspects: 1. Which goals do we want to achieve with the program? 2. Which are the critical indicators of success that tell whether I have reached the goal? 3. How can these aspects be measured? Which instruments can we use to measure these aspects? Who will apply these instruments? How can the data collected be analyzed and interpreted? Critical indicators of success Clients approach the financial wellness program with different expectations. Some may want to get rid of their debt, others would like to save some money for their retirement or some purchases they need to make. All of them will be unhappy or dissatisfied, and their quality of life will not be as good as it could. Financial wellness counseling has two basic goals that belong together: 1. the improvement of the client`s financial situation, 1

2 2. the improvement of the client`s his psychological and social situation, his wellbeing or quality of life. These two aspects are two sides of one coin, as people who join the program will often be unhappy with their financial situation. In a financial wellness program, both aspects are considered, and thus, results on both levels can be expected. That means, that we have to look at different sets of success indicators. Indicators of financial success First, there are the figures that indicate the client`s financial situation: Amount of Debt Which amount of debt does the client have when he becomes a program member? It is necessary to look at the total amount of debt. However, the total amount of debt may not even be known to the client, as he may have no overview of his financial status. In these cases, the total amount of debt is only known after several meetings with the counselor, when all relevant information has been gathered. Number of Creditors The more creditors there are, the more difficult it is to get an overview of the financial situation. In the beginning of the counseling process, it is important to find out how many people have lent money to the client. Number of judgements / garnishee orders Judgements and garnishee orders are a written proof of a bad financial situation. It is important that these are diminished in order to regain credibility and lead a life of financial well-being. Income/Expenditure-Ratio In order to get this figure, it is necessary to draw up a list of any income and regular obligation the client has. However, it is very likely that the counselor does this anyway in the counseling process, because he has to get a picture of the client`s financial situation and needs to convey this picture to the client as well. If expenditure exceed income regularly, the client is in deep trouble. In this case, the income-expenditure-ratio is below 1. In the counseling process, counselor and client will try to raise the ratio above 1, because otherwise no dept can be paid off and no savings can be built up. There might be other financial aspects that can be considered as a success factor, such as the amount of savings, or paying into a pension fund. Depending on the program`s goals, you should decide which factors need to be measured. Furthermore, there are psycho-social success factors to be considered. Indicators of psycho-social success Quality of life Quality of life cannot be measured as easily as merely financial aspects. However, there are ways to measures soft data with figures. To do so, you need a questionnaire with a rating scale. I recommend to use no more than eight or ten questions that are to be answered on a 5-point-scale. 2

3 Example: Never Rarely Sometimes Frequently Always Life has meaning and purpose With statements like I would describe myself as being happy you get a picture of the clients well-being and you can measure its improvements. Although, a question about the clients health should be included ( I feel well physically ) because recent studies have shown that people who suffer from a bad financial situation are likely to become ill, and financial wellness might lead to a better health, which is closely related to the quality of life. It is important to measure financial aspects and psycho-social aspects as well, although they appear in combination very often. However, it is possible improvements happen only in on area at first. That would mean that the counselor has to shift his attention. Further success indicators Changing financial behavior is another one of the program`s goals, as it is the prerequisite of a sustainable outcome of the counseling process. Only if the client changes his attitudes towards money and his behavior, he can successfully overcome the barriers that have kept him so far from being a wealthy and financially successful person. I recommend to ask the client five or six questions to be answered on a five-pointscale. Example: Never Rarely Sometimes Frequently Always I draw up a monthly budget Furthermore, the counsellor can be a valuable source of assessment, because he experiences the client for several sessions. Example: Never Rarely Sometimes Frequently Always The client is very active in fighting his/her debt situation Instruments for measuring financial wellness The main instrument you need for measuring financial wellness will be a questionnaire that touches on all the relevant aspects. This questionnaire has to be designed by the program directors. In the BMW FW-Program, a questionnaire was designed by the project team, with the help of experienced consultants. However, it was important that the debt counselors also were asked what they regarded as important, because they were the ones to apply the instruments. Who applies the instruments? I recommend that the section about financial aspects is filled in by the counselor, as he will have relevant data at hand in his file. Furthermore, the counselor can assess the clients behaviour in the counseling process. 3

4 The questions about quality of life and financial behaviour should be filled in by the client, if possible. Only if he is illiterate or has serious trouble understanding the questions, the counselor should conduct an interview and ask him these questions. It is important that the client answers the questions by himself, not the counselor. Please keep in mind that people who are in financially challenging situations often do not have a diploma or even a high school degree. In some cases, they might not have seen a questionnaire before. Formulate simple sentences and use a language easy to understand. How much data do I need before I analyse the data? I recommend that you do not take a random sample but collect data from every client who participates in the program. The reasons for this recommendation are: 1. Most of the data needed for monitoring and evaluation are needed to provide good advice in the counseling process anyway, so the effort for data gathering is small, 2. It is not easy to collect data from a real random sample. If you take only limited data for the M&E process, it is more likely that the counselors ask the people they get along with very easily, or the most successful clients, so that the data will be deviated. Furthermore, it is important that every set of data is really complete. Incomplete data sets lead to an imperfect view of reality and thus to wrong conclusions. Be careful to retrieve all the important questions of any client. Make sure that questionnaires are completed. When should data be collected? Data collection should generally happen in the beginning of the counseling process and after one year of counseling. When the counseling process finishes earlier, data collection can happen at the end of the process. If the process takes two years, you might take another set of data. Data analysis Once the data has been gathered, it must be filled in an excel sheet and analyzed. The BMW & GTZ financial wellness program has established a software program in cooperation with the University of Pretoria that facilitates this process. For every client and every question you need at least two data to make an assessment: One pre- and one post-test-figure. If you only have pre-test-figures of a client, these cannot be included in the analysis. This may be the case if the client has dropped out of the program by not showing up anymore, thus giving you no opportunity to make post-tests. Be sure to include only data from clients that have taken the pretest as well as the posttest. Display of results Excel or PowerPoint provide useful ways to display the program`s results by creating diagrams Average amount of debts in the beginning Average amount of debts after 1 year of counselling 4

5 This diagram shows that after one year of counselling, the average amount of debts is decreased by more than 15 %. Data Interpretation How can the results of the data analysis be interpreted? The first question you have to ask yourself is: did the clients` situation improve? The situation has improved if the amount of debts or the numbers of creditors have decreased. Even if, for some clients, these numbers have increased, the average should indicate an improvement. If this is not the case, the financial wellness program is not working sufficiently and may need to be redesigned. Figures that indicate success if they increase include the quality of life and the Income/Expenditure-ratio. But even if the figures are improving, how do we know that we are doing just well or even exceptionally well? In the first year of data collection, you will have no reference values to compare your results to. From the second year on, you can use the data of the first years` clients as reference figures. Furthermore, you can compare follow-up samples to a sample that includes all previous program participants. If you make an agreement with other financial wellness programs to use the same monitoring & evaluation tools, you can even use each other`s results as a benchmark to compare your results with. You can even use benchmark figures from international studies. In a study in Cologne, Germany, with 239 participants, a decrease of the amount of debts of 15,55 per cent was achieved within one year of counseling. The decrease of number of creditors was even higher: it amounted to 21,16 %. This is because counselors always try to get rid of creditors with small amounts of debts first. Because every single creditor is a burden to the debtor, this leads to a higher-than-average relief for the debtor. Further Indicators of Success Depending of the individual goal of the client, there could be more success indicators in any individual case. You can measure the amount of savings the client has gotten so far or whether he has a savings plan or not. You can also measure the amount of money he will get every month when retired, given that this amount is calculated regularly. However, these figures may be something that the counselor and his client monitor regularly within their meetings. In order to use them for the overall assessment of the program, figures must meet some requirements: 1. They have to mirror the general goals of the program 2. They must be important for all clients 3. They have to be collectable. Some figures might be something that the client would prefer not to talk about, e.g. his real amount of savings, or that are not available, e.g. like the amount of money he will get as a pension. For the overall assessment of the program, make sure that you use Success Indicators that can be measured. For instance, if you want to measure clients satisfaction with the counseling process, you have to reduce their statement to a figure. 5

6 Example: Not at all A little Somehow Very Completely I am satisfied with the financial wellness programme By collecting all statements from every client and calculating the average, you have a number you can refer to, e.g. the average satisfaction of our first-year`s clients was 3.8 out of five. However, if you ask clients: How did you like the financial wellness program?, you might get a wide range of different answers like It was useful, I wish I had come earlier, gorgeous, I am glad my financial situation has improved and Not bad, but I would have loved to learn more about dealing with creditors. You cannot reduce these answers to a single figure. They are valuable, though, because you get good hints at areas of possible improvement, but these statements only supplement your evaluation. Conclusion When setting up a monitoring & evaluation system, please keep the following in mind: 1. Establish the M & E system from the beginning Monitoring & evaluation is a vital part of the program, no appendix. 2. Measure hard factors and soft factors as well. In financial wellness, both kinds of factors are important. Do not reduce your measuring to the easily measurable, thus omitting the soft factors. 3. Give the counselors a thorough training of how to use the questionnaires At the beginning, counselors might get the feeling that they have to do a lot of additional paperwork. They will soon learn that monitoring the success of their efforts helps them in their daily work. Furthermore, it will contribute to the reputation of the program. 4. Make sure that the relevant data is complete for every client. Only complete sets of data lead to meaningful conclusions. 5. Leave the data analysis to a trained statistics specialist Although it is not extremely complicated, non-specialists might make mistakes with data entry and analysis. Ask a specialist to provide you with a PowerPoint-Version of the results or use the software program that has been developed by the GTZ-/BMW financial wellness program. 6. Do not hesitate to use your results in order to justify your program Financial wellness is important to everyone, but these issues are generally viewed as private. If there are other stakeholders and donors, they need to know how well the program is doing, because they pay for it and want to see results. Give them an overview of the program`s results regularly and make sure that the program is long lasting and sustainable. 6