JOHN M. KNOX & ASSOCIATES, INC. DEPARTMENT OF EDUCATION PROGRAM AND FISCAL EVALUATION

Transcription

1 JOHN M. KNOX & ASSOCIATES, INC. A Report to the Hawaii State Board of Education DEPARTMENT OF EDUCATION PROGRAM AND FISCAL EVALUATION DOD INFORMATION TECHNOLOGY HUMAN RESOURCES RECRUITING SYSTEM PROGRAM ID NO October Bishop St., ASB Tower 1542 Honolulu, HI USA Phone (808) Fax (808) jmk@johnmknox.com Web

2 EXECUTIVE SUMMARY The DOD Information Technology Human Resources Recruiting System (IT HR Recruiting System) program was established in 2003 to begin work on developing online capabilities to support employee recruitment. The program s mission was to begin the development of online recruiting capability and determine its effectiveness in assisting the recruiting staff and reducing the cycle time between initial applicant contact and evaluation/selection for a position. The goals and objectives of this program were to: a. Identify bottlenecks in the recruiting process where increased automation would be likely to improve the processes. b. Implement interim online capabilities for process improvement. c. Evaluate the effectiveness of these capabilities. Staff from both the Office of Human Resources (OHR) and the Office of Information Technology Services (OITS) were involved in determining goals and objectives, and guiding the project. An initial set of online capabilities were designed and developed that included: a. An online employee application. b. A process for scanning and electronically storing the supporting documentation (transcripts, etc.) of each applicant. c. A database for tracking each applicant. Performance indicators were utilized to monitor progress towards the desired reduction in response and process times, from initial contact with an applicant to the availability of a complete set of applicant information ready for evaluation and selection by a hiring authority. Evaluation of prototype software developed under this program identified systemic limitations. Consequently, a broader, pre-packaged recruiting service (i.e., NeoGov) that met most of OHR s needs was then tailored for DOE use and implemented. NeoGov was able to handle the recruitment and placement of several thousand new employees, and only required a minimal increase in additional OHR staffing. The contractor s work progress was monitored by weekly and monthly meetings. When initial capabilities were put into use by OHR recruiting staff, regular evaluation meetings were held to communicate usage problems, program bugs, needed system changes, etc. John M. Knox & Associates, Inc. Page i

3 Despite changes in the functional scope of capabilities, falling behind schedule, and loss of project continuity, a prototype of an in-house recruiting software subsystem was produced. This prototype, though not developed to the point of production use, provided a baseline for future custom development. This program was funded by the U.S. Department of Defense (DOD) Supplement to Impact Aid, a federally funded program that provides funding to school districts (as a supplement to the U.S. Department of Education Federal Impact Aid Program) to support school districts that have a large proportion of students who are dependents of federally-connected, or military families. The DOD IT HR Recruiting System Program ended this fiscal year (FY) The findings of our evaluation are as follows: The DOD Information Technology Human Resources Recruiting System program attempted to carry out its mission of developing online capabilities to support employee recruiting efforts. However, the prototype software could not be developed to the point of production use within the required timeframe. Consequently, a prepackaged service was selected that did meet OHR s requirements and timeline. Regarding the goals and objectives of this program, no actual documentation was provided that identified the exact bottlenecks, further explained the interim online capabilities, or described the results of the evaluation of these capabilities. Without this information, we are unable to determine how and to what extent program objectives were met. As reported by the Program Manager, the program was established in No timeline was given to show how much time was spent on developing the prototype software or when the decision was made to use the pre-packaged service. Also, insufficient details were provided as to why the prototype was not able to be developed in time to meet OHR s needs. Since no supporting documents were furnished for review, we cannot speak to the fidelity or actual achievements of the program. Ultimately, the program s end products were a prototype software and identification of an existing software program tailored to meet OHR s needs. Performance indicators were utilized to monitor progress towards the desired reduction in response and process times, from initial contact with an applicant to the availability of a complete set of applicant information ready for evaluation and selection by a hiring authority. John M. Knox & Associates, Inc. Page ii

4 Also, weekly and monthly meetings monitored the progress of contractor work. The program did not provide us with a description of what measurements were used to assess progress in reducing response and process times. Since we did not receive any data regarding the time reduction (e.g., in days or weeks, etc.) involved in the hiring process, we cannot verify success of the program. In addition, discussions during weekly and monthly meetings do not constitute program monitoring and evaluation. Without capturing the real time benefit of the software, we cannot ascertain the true nature and value of the program s efforts. We find that the program apparently did not establish performance measures for process evaluation. Like our other findings, without data to show whether indicators were met or not, we find it impossible to ascertain the true nature and value of the program s efforts. Actual recommendations for this program are not required since this program ended with the current fiscal year. A related program will be implemented in FY , but no details have been made available for this evaluation. We do suggest, however, that the BOE consider withholding funding for this related program until the Program Manager is able to provide supporting documentation to justify it. Also, any continuation or variation of this program must include stronger accountability measures. John M. Knox & Associates, Inc. Page iii

5 TABLE OF CONTENTS EXECUTIVE SUMMARY... i TABLE OF CONTENTS... iv I. INTRODUCTION... 1 II. METHOD... 2 III. PROGRAM DESCRIPTION A. Program Issues... 3 B. Fiscal Issues... 5 IV. SUMMARY OF FINDINGS A. Execution of Mission and Purpose... 7 B. Meeting Objectives... 7 C. Program Implementation... 7 D. Performance Measures... 7 V. RECOMMENDATIONS... 8 VI. CONCLUSIONS... 8 VII. PROGRAM RESPONSE... 8 REFERENCES John M. Knox & Associates, Inc. Page iv

6 INTRODUCTION On August 17, 2006, the Hawaii State Board of Education approved Policy , Department of Education Program Evaluations as follows: The Board of Education (Board) is responsible for the review and approval of the Department of Education s (Department) budget. The Board is also responsible for monitoring the use of funds appropriated to the Department. As such the Board shall review the Department s programs and its use of funds for reasonableness and effectiveness regardless of the program s funding source, and reallocate moneys, if it is deemed necessary and/or prudent. The Department shall conduct a program and fiscal evaluation of its programs at least once every five years. The Department shall, by July 1 of each year, provide the Board with a five-year schedule for performing program evaluations. Evaluations shall focus on successful outcomes that enable the Department of Education to meet, or exceed, Board of Education standards and goals and perform in a cost effective manner. The Board shall decide whether to continue or discontinue a program after having reviewed the results of the program s evaluation. The Board shall determine whether to reallocate any funds from programs, which are to be discontinued. The policy requires the Department of Education (DOE) to conduct fiscal and program evaluations of every program in the Department on a five-year cycle. The DOE s Systems Accountability Office has contracted John M. Knox & Associates to conduct program and fiscal evaluations of several of its programs. These evaluations will enable the Department of Education and the Board of Education to: 1. Identify the extent to which programs are addressing their mission and purpose; 2. Identify issues related to program implementation; 3. Determine the extent to which the programs are meeting their objectives; and 4. Determine how program funds are being expended. John M. Knox & Associates, Inc. Page 1

7 METHOD Each Program Manager was initially asked to complete a checklist of required documents for the evaluation. Documents were to include the current fiscal year, as well as the previous three fiscal years. Program Managers were also able to indicate whether or not documents were available or applicable for the purposes of this evaluation. The following tasks were to be completed by the Program Manager for the evaluation: Program Evaluation: a. Identify the basis upon which the program was established. b. Identify the program s target population(s). c. Review the program mission, goals, and objectives. d. Review and assess the linkage between program goals and objectives and the program mission. e. Review and asses major program activities to determine the linkage between those activities, the program goals and objectives, and the program mission. f. Identify the number and type of staff responsible for program implementation. g. Identify program performance indicators and performance measures. h. Review and assess current and historical or trend data on program indicators and performance measures. i. Identify current and past program reports that describe the major program activities, outcomes, and impact. j. Interview program staff about issues related to program implementation and program outcomes. Fiscal Evaluation: a. Identify the funding source(s) for the program b. Collect, review, and assess trend data related to program allocations and expenditures. c. Review available program audits Documents submitted by the Program Manager included Fiscal Summary Historical Expenditure for Program Pivot by Organization ID Historical Expenditure for Program Pivot by Object Code Untitled Expenditure Report Our work was conducted from June 2007 to July John M. Knox & Associates, Inc. Page 2

8 PROGRAM DESCRIPTION Program Issues How, when and why the program was established The DOD Information Technology Human Resources Recruiting System (IT HR Recruiting System) program was established in 2003 to begin work on developing online capabilities to support employee recruiting. Target Population The beneficiaries of increased automation in the employee recruiting process are schools and students. The goal of this effort was to be able to process applicants faster and more accurately, so that school positions can be filled quicker and with better qualified employees. Stakeholders Served by the Program The ultimate stakeholders are the 180,000 DOE students. Secondary stakeholders are school administrators who will have improved ability to staff their schools and fill vacancies in a timely manner. The OHR recruiting staff is an additional stakeholder, since they will be able to service their users (schools and offices) with shorter response times. Program Mission The program s mission was to begin the development of online recruiting capability. The mission also involved determining effectiveness in (a) assisting the recruiting staff for a position and (b) reducing the cycle time between initial applicant contact and evaluation/selection for a position. Program Goals and Objectives The goals and objectives of this program were to: a. Identify bottlenecks in the recruiting process where increased automation would be likely to improve the processes. b. Implement interim online capabilities for process improvement. c. Evaluate the effectiveness of these capabilities. Staff from both the Office of Human Resources (OHR) and the Office of Information Technology Services (OITS) were involved in determining goals and objectives, and guiding the project. Major Program Activities An initial set of online capabilities were designed and developed including: a. An online employee application. b. A process for scanning and electronically storing applicant supporting documentation (transcripts, etc.) John M. Knox & Associates, Inc. Page 3

9 c. An applicant tracking database. The developed capabilities were used to provide the OHR staff with tools to assist them in performing recruitment activities for Board of Education (BOE) Support Services Personnel (BSSP) and Classified employees, per legislative mandate (This work had formerly been handled by the State Department of Human Resources Development (DHRD)). The developed capabilities were used heavily for Educational Assistant recruiting. When the evaluation of the capabilities being used for Educational Assistants identified systemic limitations, a broader turnkey recruiting capability, called NeoGov, was then implemented and put into use for Civil Service Classified employees. Staff The primary effort was accomplished via contracted services from Information Technology contractors. An advisory team comprised of key staff from both OHR and OITS guided this effort. In addition, DOE Subject Matter Experts (SMEs) were used at various stages to determine functional requirements. The participation of these staff members was on a part-time basis, as needed, and was not funded by the program. Program Performance Indicators and Measures Performance indicators were utilized to monitor progress towards the desired reduction in response and process times, from initial contact with an applicant to the availability of a complete set of applicant information ready for evaluation and selection by a hiring authority. Prototype software developed under this program was used initially for Educational Assistant recruiting. Evaluation of this use identified systemic limitations. It was determined that the needed changes, added functions, and associated testing and validation could not be accomplished in time to meet the forthcoming OHR school year recruiting cycle. Consequently, a broader prepackaged recruiting service (NeoGov), that met most of OHR s needs, was then tailored for DOE use and implemented for recruiting BSSP and Classified employees. NeoGov was able to handle the recruitment and placement of several thousand new employees, including a significant peak load at the start of the school year, and only required a minimal increase in additional OHR staffing. Program Monitoring and Evaluation The contractor s work progress was monitored by weekly and monthly meetings. When initial capabilities were put into use by OHR recruiting staff, regular evaluation meetings were held to communicate usage problems, program bugs, needed system changes, etc. John M. Knox & Associates, Inc. Page 4

10 Summary of Program Implementation and Outcomes Requirements from OHR users changed frequently, and additional requirements were often added. One result of this was that the functional scope of the capabilities being developed was steadily increasing, causing the contractor to fall behind schedule. The original contractor changed its technical approach, its project manager, and assigned personnel several times during the project which resulted in the loss of project continuity. Despite these issues, a prototype of an in-house recruiting software subsystem was produced and even though it was not developed to the point of production use, provided a baseline for future custom development. When it became clear that the prototype software could not be refined into a production quality system in time to meet OHR s needs, the program then turned to NeoGov, a pre-packaged service that did meet OHR s timeline and capabilities requirements. NeoGov provided a critical online recruiting and applicant tracking service for recruiting BSSP and Classified employees, which facilitated the DOE s delinking from DHRD. Fiscal Issues Funding Source This program is funded through U.S. DOD Supplement to Impact Aid. The Hawaii Department of Education (HIDOE) receives an annual supplemental appropriation from the U.S. Department of Defense (DOD) which the HIDOE, with approval from the BOE, uses at its discretion to support the state s educational system. The DOD appropriation supplements the Impact Aid funding that the state receives from the U.S. Department of Education (USDOE) to help defray the cost of educating the children of Hawaii s military and other federallyconnected families. DOD appropriations in recent years have ranged from $2.2 million to $2.5 million annually. Each year, the BOE reviews and approves the use of DOD funds and OFS tracks the use of these allocations with the DOD designation in the budget program title. DOD funds are used at the Department s discretion and do not lapse. These funds can be carried over from year to year until depleted. Program Allocations and Expenditures Fiscal Year (FY) Allocations Expenditures Balance $300, $3, $296, $296, $220, $75, $75, $75, $0.00 John M. Knox & Associates, Inc. Page 5

11 Funds were expended on: a. Travel to investigate vendor turnkey software and/or solutions that potentially could provide part or all of the needed capabilities. b. Contracted development of a process for scanning and electronically storing all the supporting documentation (transcripts, etc.) associated with applicants. This effort also included the development of a prototype applicant tracking software subsystem. c. When the initial prototype software was determined to be insufficient for production use, program funds were used to contract a service (NeoGov) which specializes in recruiting, applicant processing and applicant tracking for government employment. These capabilities were adapted for DOE use, the OHR staff was trained, and the system was put into production use for recruiting BSSP and Classified employees. Funds were expended in accordance with the guidelines found in the FMS User Policy and Process Flow Guide, and procurement law. Carryover Fiscal Year Balance $75, Lapsed Funds None. Program Audits or Fiscal Reviews None. Note: A recent amendment by the BOE now requires the DOE to provide the Board with monthly reports on federal funds received. The amendment was approved on October 5, BOE Policy , "Federal Funds Policy," also requires the DOE, at least six months prior to the lapsing of any federal funds, to provide the Board with an action plan for ensuring the federal funds will not lapse. Formal reviews of the use of federal funds received by the DOE will provide the BOE with a clearer understanding of the funds available to the DOE and the ability to evaluate the DOE's use of federal funds. The DOE already has regulations and procedures for routine reporting of the planned use of federal funds and approval of the proposed use of Impact Aid/U.S. DOD funds. John M. Knox & Associates, Inc. Page 6

12 SUMMARY OF FINDINGS Execution of Mission and Purpose The DOD Information Technology Human Resources Recruiting System Program attempted to develop online capabilities to support employee recruiting, but could not refine its prototype within the required timeframe. Instead, It selected an alternative in the form of a pre-packaged service that did meet OHR s requirements and timeline. Meeting Objectives The goals and objectives of this program were to: a. Identify bottlenecks in the recruiting process. b. Implement interim online capabilities for process improvement. c. Evaluate the effectiveness of these capabilities. No actual documentation was provided that identified the exact bottlenecks, further explained the interim online capabilities, or described the results of the evaluation of these capabilities. Without this information, we are unable to determine how and to what extent program objectives were met. Program Implementation As reported by the Program Manager, the program was established in No timeline was given that revealed how long the program spent on developing the prototype software or when the decision was made to use the pre-packaged service. Also, the program did not describe the details surrounding its inability to develop the prototype in time to meet OHR s needs. Since no supporting documents were furnished for review, we cannot speak to the fidelity or actual achievements of the program. Ultimately, the program s end products were a prototype software and identification of an existing software program tailored to meet OHR s needs. Performance Measures Performance indicators were utilized to monitor progress towards the desired reduction in response and process times, from initial contact with an applicant to a detailed compilation of applicant information that would assist HR in the selection process. Also, weekly and monthly meetings monitored the progress of contractor work. The program did not provide us with a description of what measurements were used to assess progress in reducing response and process times. We did not receive any data regarding the time reduction (e.g., in days or weeks, etc.) involved in the hiring process, so we cannot verify success of the program. In John M. Knox & Associates, Inc. Page 7

13 addition, discussions during weekly and monthly meetings do not constitute program monitoring and evaluation. Without capturing the real time benefit of the software, we cannot determine the true nature and value of the program s efforts and ascertain, then, that the program did not establish performance measures for process evaluation. RECOMMENDATIONS Actual recommendations for this program are not required since this program ended with the current fiscal year. A related program will be implemented in FY , but no details have been made available for this evaluation. We do suggest, however, that the BOE consider withholding funding for this related program until the Program Manager is able to provide supporting documentation to justify it. Also, any continuation or variation of this program must include stronger accountability measures. CONCLUSION Technology can be invaluable in gathering data and the selection process, but effective human relationships remain the key to recruiting the right applicant. Although technology allows handling larger workloads quickly and more efficiently, human interaction remains an essential part of the hiring process. Nevertheless, refinement of the Department s online capabilities should be an ongoing endeavor that incorporates continual advancements and improvements in technology. PROGRAM RESPONSE Thank you for your Program and Fiscal Review of the DOD Information Technology Human Resources Recruiting System Program and for the opportunity to submit a response. Here are my responses to follow-up questions that will provide additional information about the program that were not addressed in the report. 1. Was the focus only on recruiting and hiring school personnel or nonschool personnel too? The primary focus was the hiring of school personnel but the scope also included the hiring of non-school personnel. John M. Knox & Associates, Inc. Page 8

14 2. What was the total cost of contracted services from IT contractors? Total cost of contracted services and associated software licenses was $291,641. a. What portion of these services could have been done inhouse? There was insufficient in-house capacity to undertake the scope of services that was contracted. 3. What were the specific performance indicators utilized to monitor progress towards the desired reduction in response and process times? The goal was for the initial DOE response to an applicant to be no more than 48 hours. There was also an overriding objective of DOE being able to handle recruiting on its own by July 2006, including evaluating applicants and providing candidate lists to Schools and Offices, totally uncoupled from DHRD. This was successfully accomplished. 4. Is there a timeline of major project milestones? For example: a. Date project commenced August b. Activities leading to development of prototype software A series of requirements meetings were held between the OHR recruiting staff, OITS personnel and the contractor from September 2004 through February c. Prototype developed June 2005 d. Prototype found incapable of meeting project needs January 2006 e. Decision to go with NeoGov April 2006 f. Outcomes i. Comparison of expected vs. actual outcomes The prototype was put into preliminary use by the OHR Support Services Personnel (SSP) recruiting staff with emphasis on handling the recruiting of Educational Assistants. The level of bugs in the system and residual deficiencies in desired functionality led to a loss of confidence in the system by the HR recruiting staff. This in turn led to the investigation of alternatives that could be implemented rapidly. Based on favorable experience with NeoGov by Honolulu City & County the transition was made to NeoGov. 5. The contractor fell behind schedule because the functional scope of the capabilities being developed was steadily increasing. a. Was this due to lack of proper planning? It was primarily due to the lack of firm guidelines during the requirements sessions that would constrain the development effort to the original set of highest priority features. During the requirements sessions and during the prototype evaluation sessions, unforeseen additional requirements were continuously added by the DOE users. Some of these were indeed necessary but some were refinements that could easily John M. Knox & Associates, Inc. Page 9

15 have been deferred to a later enhancement : phase. However, the contractor kept accepting these additional requirements even though many of these implied additional development effort, additional schedule time, and increased complexity. As a result, the development effort experienced delays and the resulting prototype software had an unacceptable level of quality problems. This situation was exacerbated by inadequate Project Management on the contractor s part and abrupt changes of Project Management personnel during critical phases of the project. b. Did this result in higher costs? We were billed for costs beyond the original plan, but we ultimately disallowed most of the excess billings because the contractor didn t follow the original plan and because of the unacceptable number of errors (bugs) in the delivered prototype. c. If so, how much higher? We ultimately did not pay any higher cost than planned. Since the contractor had also been engaged for document scanning services, we had the contractor provide an equivalent dollar amount of extra document scanning services for those portions of their software development effort where we did not receive adequate value from their delivered software product. 6. How effective was NeoGov and how was this measured? NeoGov very adequately met the HR staff s needs and they were able to meet the goal of 48 hours or less response time to applicants. Of equal importance, the staff was able to assume the recruiting function from DHRD as part of the Act 253 delinking plan. 7. Was the total cost of this project limited to the initial $300,000 allocation for FY ? Yes a. If not, what were the additional expenses and what were they for? 8. What did we get for the money that was expended? (1) Functional requirements (business process definition) for recruiting of SSP employees; (2) Server hardware and licenses for commercial software products (FileNet and Liquid Office) that are the foundation of a DOE electronic document repository which now contains over 7 million documents and has freed up a large amount of office space formerly occupied by Lektriever cabinets; (3) Document scanning of specific groups of HR employee document records; and (4) NeoGov licenses including customization for DOE needs and initial setup and training. 9. To what extent were the expected outcomes achieved? NeoGov has been largely successful for handling SSP recruiting and has subsequently been adapted for recruiting of Classified employees in addition to the original use for SSP. It is serving OHR s needs very adequately as an John M. Knox & Associates, Inc. Page 10

16 interim system. It is expected to be gradually superseded by a single, comprehensive recruiting system designed to handle all DOE employee categories (SSP, Classified and Certificated) that is being developed as part of the Collaborative Human Resources Automation Project (CHAP) currently under way. The CHAP effort will provide a much tighter level of integration between the Recruiting functions, the School List resource allocation process, and the Position and Employee transaction Systems than is possible with NeoGov. 10. Will there be a related program implemented in FY ? If so, could you provide some details about the following: The next phase of related efforts addressing the Department s recruiting needs have been merged into the CHAP project. The initial Recruiting capabilities within the CHAP project are scheduled to be available in July a. Total budget amount b. Program purpose c. Desired outcomes d. Timelines e. Performance indicators John M. Knox & Associates, Inc. Page 11

17 REFERENCES Department of Defense. (n.d.) Military Students on the Move: A Toolkit for Installation Commanders. Retrieved June 26, 2007 from Department of Education (2006). Legislative Report: Federal and Trust Funds. Retrieved June 26, 2007 from olelo2.k12.hi.us/reports/tolegislature/b06- FederalAndTrustFunds.pdf Department of Education. (2002.) Legislative Report: Cost of Educating Federally Connected Students. Attachment 1: History of Impact Aid in Hawaii. Retrieved June 26, 2007 from doe.k12.hi.us/reports/tolegislature_2002/federalconnection/federalconnection.pdf John M. Knox & Associates, Inc. Page 12