RUNNING HEAD: EVALUATION PROPOSAL 1 Program Evaluation: Onboarding Program Kaitlin Gibbons The University of Tampa EME 640
EVALUATION PROPOSAL 2 Request For Proposal A mid-sized southern university is looking to develop and implement an onboarding program for their new staff members. The program would be implemented on a per person basis and would last for one year from the day they were hired. The purpose of the program is to help new employees get started and settled into their new roll on the university s campus. The program will include an online resource page for new hires on the campus learning management site Blackboard, as well as pamphlets, flyers, and mentoring program with a member in their office. The overall goal of the program is to welcome new hires onto the campus and acclimate them to the new environment and policies/procedures on campus. The program also moves the responsibility of acclimating the individuals away from the human resource department and towards the individual departments on campus. The desired outcome is for the new hires to feel comfortable in their position and become a productive member in their office, and ultimately lead to higher retention rates on the campus. Overall evaluation plan The human resources department on the campus and a graduate student assistant will conduct the overall evaluation plan for the onboarding program at the University. Three types of evaluation will take place during the development and implementation of the program: formative evaluation, summative evaluation, and confirmative evaluation. The total evaluation will take approximately 18 months to complete. This will allow for the development and testing of materials and program (formative evaluation phase), evaluation of the complete program (summative evaluation phase), and a follow up confirmation that the program was effective months after completion (confirmative evaluation phase). The evaluation will begin in the spring and last one full year, followed by a confirmative evaluation four months after.
EVALUATION PROPOSAL 3 The evaluation of the program is to ensure that the new onboarding program is successful and providing the right type of environment and information to new hires of staff at the university. The outcomes of the evaluation will determine whether or not to expand the program to new faculty members, not just staff, and if the program should continue. The evaluation will be composed of different data collection tools and assessment, which are: Subject Matter Expert (SME)- Formative evaluation Interview with individual One-to-One Evaluation-Formative evaluation Attitude acceptance survey (20 questions) Interview for additional information Attitude acceptance survey Summative evaluation 20 question survey about program Posttest survey-summative evaluation 20 question posttest about policies, procedures and campus items. Attitude acceptance survey-confirmative 20 question assessment of their position post program
EVALUATION PROPOSAL 4 Formative Evaluation April 1-15 April 1-15 April 1-10 April 1-15 April 15 April 15 May 1-8 May 1-15 May 8-22 May 15-30 May 21- June 4 Timeline for Formative Evaluation Recruit SME for evaluation Develop tools for SME evaluation Recruit four to six individuals for one-to-one evaluation Develop tools for One-to-one Evaluation Schedule SME Meeting for first week of May Schedule One-to-one Volunteers for the first two weeks of May Conduct SME Interviews on the Evaluation Conduct One-to-one evaluations Analyze data collected from SME Analyze data from One-to-one Make and finalize changes to the program Expert Review There will be two expert reviews performed on the material for the onboarding process. The first expert will be an individual in the human resources department from another university of the same or similar size and hierarchy that have implemented an onboarding process for its new staff members. This individual will be able to provide insight into what segments of the process were successful for their institution as well as locate any that seem vague or not needed. An email correspondence will be sent to a number of universities with an onboarding program and solicit a volunteer to review the program. The email will include a background about the university as well as the goals and purpose of the onboarding program. Unfortunately due to budget needs, the individual will not be compensated for their time. The program plan and materials developed will be sent via email to the individual for a total of two weeks. At the end of the two weeks, an interview meeting will be scheduled to discuss their thoughts on the materials. The meeting will be conducted over the phone or via Skype or Google Hangout. The interview will include closed and open-ended questions about the materials. The individual will also be asked to supply any additional information that they feel
EVALUATION PROPOSAL 5 would be appropriate for the program. A sample of the interview protocol can be seen in table 1 below. For the second expert review, a department head with in the university will be asked to review the materials. The reason for choosing a department head is that an individual in their position will be responsible for the new hires and implementing the process with in their department. This individual has also been faced with acclimating a new individual to their department and their department policies and will be able to provide insight into how a department is organized and how effective a program like this will be. An email will be sent out to all department heads on campus asking for a volunteer to evaluate the program and materials. The email will contain information about the program and what will be asked of them. Similar to the first expert review, this individual will be sent the program plan and materials via email. After two weeks, an interview meeting will be set up. For this meeting face to face is preferred since they will be on the campus. However because scheduling conflicts can occur, phone or video calls through Google Hangouts or Skype are an option. The same interview protocol will be used in the interview with this individual. Table 1 Sample of Interview Protocol 1. How comprehensive do you feel the materials Answer: were? 2. What items on the supervisor checklist would Answer: you change? 3. Were there any items listed in the procedure that Answer: you did not feel were necessary? 4. Were their items not listed in the procedure that Answer: you feel would be beneficial? 5. Would you add any information to Blackboard Answer: site? One-to-One Evaluation The second type of formative evaluation to be used will be a series of one-to-one evaluations. The individuals used in the one-to-one evaluation will be employees who were hired
EVALUATION PROPOSAL 6 at the university in the last six months to a year. These individuals would not have gone through an onboarding process and will have insight into what items in the plan would have been useful for them becoming familiar with the university and their various departments. The human resources department will be utilized to provide the campus emails of those individuals hired in the last 6 months to a year, and an email asking for volunteers will be developed and sent out. A total of four to six individuals are needed for this portion of the evaluation. The individuals will need to be able to dedicate either one three-hour session or three one-hour sessions to view the materials (the former is preferred). Upon the day of the evaluation, the individual will be brought into the conference room located in the human resource department. The room will be equipped with a laptop so the volunteer can evaluate the Blackboard site for the onboarding program. The volunteers will be asked to take notes on the materials and on things that they believe will be beneficial to add. The designer will be available in the room to answer any questions or concerns that the volunteer may have about the material. After all the materials have been reviewed, a short attitude acceptance survey will be given to the individuals to complete. The survey will include both Likert style and open ended questions and contain 20 questions. A sample of the survey can be seen in Table 2. They will also be interviewed by the designer and asked to share any additional thoughts about the program. Table 2 Survey Sample Questions 1. Would you have benefited from a program like Strongly Agree, Agree,, Strongly this when you were hired? 2. I would have felt more comfortable in my Strongly Agree, Agree,, Strongly position with a program like this 3. I have participated in an onboarding program in Strongly Agree, Agree,, Strongly the past 4. I found the Blackboard Site useful and Strongly Agree, Agree,, Strongly informative 5. What would you like to see added or changed to the materials for the program?
EVALUATION PROPOSAL 7 The revision suggestions and additional information given by both the SME and the oneto-one evaluations along with the surveys from the one-to-one will be taken into account and added to the final program materials and procedures. The goal is to refine the materials and program to ensure that it is the best possible product for its audience. Summative Evaluation May 1-15 May 1-31 June 1-30 June 1-30 June 1-30 July 1-31 July 1-31 July 1-31 July 1-31 August 1-4 August 1-15 Timeline for Summative Evaluation-Year After Implementation Develop attitude acceptance survey and posttest Recruit new staff member for evaluation Send out surveys and posttest to individuals who volunteered and were hired in June of the previous year Collect surveys and posttest Recruit new staff member for evaluation Analyze data from survey and posttest from June participants Send out surveys and posttest to individuals who volunteered and were hired in June of the previous year Collect surveys and Posttest Analyze data from survey and posttest from July participants Results from both months synthetized Adjustments to the program are made from survey results Upon completing the onboarding program for the year, all new staff members who were hired in the months of June and July will be asked to participate in the evaluation of the program. Ideally there will be ten total participants for this group. One year after they were brought on as staff, the individuals will be sent an attitude acceptance survey about their time in the onboarding program, and a brief posttest about various policies, procedures, and location of things on campus. This will be sent via their university email. The overall goal of the onboarding program
EVALUATION PROPOSAL 8 is, as stated earlier, to make the individuals feel welcomed and be knowledge about their new job and work environment. The attitude and acceptance survey will show if the individuals feel a part of the university community, their feelings toward the program, and if they felt it made a difference in their attitude toward their job. The survey will have similar questions as the ones that were asked in the one-to-one evaluation. Positive results are ideal for continuing this program, however if a trend develops in one area of negative result it could mean training is needed with supervisors in the onboarding program. Sample of the survey questions can be seen in Table 3. Table 3 Sample of Survey Questions 1. The onboarding program made me feel Strongly Agree, Agree,, Strongly more comfortable in my job 2. The information on the Blackboard site was Strongly Agree, Agree,, Strongly easy to find 3. I can locate all buildings on campus Strongly Agree, Agree,, Strongly 4. I feel that I understand my position better Strongly Agree, Agree,, Strongly because of the program 5. What is one thing you would change about the program? The posttest that will be distributed will be to see if they know the employee rules and regulations for the university and where things are located that they would need to know. This posttest will have no penalty for them for their participation; it will be used to evaluate the knowledge being shared with them. Low results from this posttest can show that aspects of the onboarding not dealing with acceptance are not being covered with the individual when they start their position. A sample of the posttest can be seen in Table 4.
EVALUATION PROPOSAL 9 Table 4 Sample of Posttest Questions 1. What are the two locations to find food on Answer: campus? 2. Where can I find my Payroll information Answer: 3. How much notice do I need to give for Answer: Vacation? 4. What is our schedule in the summer? Answer: 5. What are the hours of operation during the Answer: academic year? Confirmative Evaluation The confirmative evaluation will take place four months after the onboarding program is completed. The evaluation will be conducted to ensure that the information provided during onboarding and process that the individual went through has helped them on their job now that they are no longer undergoing the process. The same people who participated in the summative evaluation will be contacted to participate in confirmative evaluation. They will be asked to fill out another attitude and acceptance survey with Likert style and open-ended question to see how their experience has been since completing the program, and if their attitudes have changed since the first survey. It will include questions similar to the formative and summative attitude and acceptance surveys but will also include questions on how effective they feel they are at their position. A sample of this survey can be viewed in Table 5. Results from this survey will be analyzed against the previous summative survey. It is from here that a decision will be made on whether to keep the program or discontinue it.
EVALUATION PROPOSAL 10 Table 5 Sample of Confirmative Survey Questions 1. I am more likely to continue in this job Strongly Agree, Agree,, Strongly because of the Onboarding Program? 2. I still feel confident on how to do my job Strongly Agree, Agree,, Strongly correctly 3. I feel a part of my office community Strongly Agree, Agree,, Strongly 4. I feel comfortable to still approach my Strongly Agree, Agree,, Strongly supervisor with questions 5. What would you change about the program after completing it six months ago?