TIERS/IEES REVIEW TEXAS HEALTH AND HUMAN SERVICES COMMISSION OFFICE OF INSPECTOR GENERAL. April 18, 2007 ALBERT HAWKINS EXECUTIVE COMMISSIONER

Transcription

1 TIERS/IEES REVIEW April 18, 2007 TEXAS HEALTH AND HUMAN SERVICES COMMISSION ALBERT HAWKINS EXECUTIVE COMMISSIONER OFFICE OF INSPECTOR GENERAL BRIAN FLOOD INSPECTOR GENERAL

2

3 Why OIG Did This Report On March 13, 2007, thirty Texas State Senators requested that the Texas Health and Human Services Commission (HHSC) Office of Inspector General (OIG) conduct a review or investigation and issue a report concerning the procurements, management, development, effectiveness, efficiency, usability, and cost effectiveness of both the Texas Integrated Eligibility Redesign System (TIERS) and the HHSC contract for Integrated Eligibility and Enrollment Services (IEES). The letter further requested that the report include findings concerning the public monies spent on TIERS and IEES, findings of whether public monies have been expended in a cost effective manner, and recommendations for the most cost effective future expenditures of public monies relating to TIERS or IEES. (See Appendix 1 Request Letter from thirty State Senators). Note Concerning References This report contains references to appendices, and references to attachments. Each reference to a document or to information contained in an appendix is cited in the body of the report. Each reference to an attachment is cited in an endnote. Concerning the attachments that are referenced in the endnotes, these are not included in the printed version of the report. However, each printed report includes a compact disc that contains copies of the referenced attachments in electronic form. April 2007 What OIG Found TIERS is the Texas Integrated Eligibility Redesign System, a software application being built under contract that is fully deployed in three of the 254 Texas counties, and that is intended to replace the system in use in the remaining 251 counties. IEES is the Integrated Eligibility and Enrollment System, a new business process for providing entitlement benefits to needy Texans under a call center model. Using TIERS, IEES was launched as a pilot in January 2006 under a privatization contract that is now being cancelled, which has been characterized as the largest contract of its kind in the state s history. 1 To date, more than half a billion taxpayer dollars have been spent on TIERS/IEES. As explained in the Results in Detail section of this report, we concur with the 2006 federal Single State Audit finding that the eligibility process supported by TIERS is not appropriately designed and/or operating effectively to enforce the respective eligibility decisions necessary to ensure clients are eligible and receive proper benefit amounts. To place this in perspective, based on an extrapolation of the findings of their audit, if TIERS were deployed statewide today, it could increase the state s questioned costs, as found in the recent audit, covering 1 million to 1.85 million more needy Texans, and ranging between $2.6 and $4.9 billion more taxpayer dollars. (See Appendix 2). However, we acknowledge that state authorities have recently recognized these risks and have suspended the IEES pilot and the IEES contract is in the process of being cancelled. Further, the Commission has announced that it will not proceed with any further rollout of TIERS until issues raised by the federal Food and Nutrition Service, which is the lead federal funding partner for IEES, have been resolved. (See Appendix 14). The following seven critical events have been identified as contributing to the suspension of the IEES pilot: 1. The removal of the original data warehousing requirement from the initial TIERS contract and subsequent failure to yet replace it with an adequate alternative solution. (See page 12) 2. The commitment of state authorities to the TIERS application without consideration of the possibility of other alternative technical solutions. (See page 16) 3. The decision to privatize did not use all the relevant data that was available. (See page 19) 4. Although state authorities initially used the statistically valid SAO scoring methodology, a different methodology was then utilized in the process of awarding the IEES contract. (See page 23) 5. State authorities released Deloitte from correction of numerous system deficiencies, resulting in a failure to enforce original, fundamental requirements of the initial TIERS contract. (See page 25) 6. State authorities released Accenture from a fundamental contract requirement, immediately before going live with IEES, by agreeing to the outsourcing of a core competency requiring program policy knowledge. (See page 27) 7. The absence of effective management of project and contract oversight. (See page 32)

4 What OIG Recommends Based on the results of the review, without investigation or audit, OIG s preliminary recommendations are as follows: 1. Place a single project manager over TIERS and Integrated Eligibility, with ultimate authority and accountability for the entire project. The project manager should have demonstrated success and experience commensurate with projects of this size and complexity. 2. Timely retain an independent expert consultant to determine the most cost effective of the alternatives for going forward, under a defined time frame. In regard to the software application to be deployed for eligibility determinations, it is specifically recommended that the consultant examine TIERS, SAVERR, and any other available system that is potentially capable of supporting the business process. 3. Pending the results of the independent study that is recommended under item 2, limit the expansion of clients in the TIERS platform and continue the decision, as reflected in the executive commissioner s letter, dated April 17, 2007 (see Appendix 14), to not proceed with any further rollout of TIERS until the identified issues are resolved. 4. An audit by the State Auditor s Office (SAO) of the economy, efficiency and effectiveness of the TIERS and SAVERR applications and the IEES program including the necessary ancillary and technological systems that HHS currently employs to conduct the eligibility program for Texas. 5. Engage the QAT, Quality Assurance Team, as provided by sections through of the Government Code, and the Quality Assurance Review of Major Resource Projects functions therein to monitor or assist the TIERS/IEES system implementation and operation issues. 6. Stabilize the current workforce environment.

5 Contents Review Appendices Tables Figures Note Concerning Scope of Review 1 Results in Brief 2 Background 6 Results in Detail 10 Removal of Data Warehouse 12 Commitment of State Authorities to TIERS Application 16 Decision to Privatize Did Not Use All Relevant Data 19 Departure from SAO Scoring Methodology 23 Departure from Requirements of Initial TIERS Contract 25 State went Live with Business Process Different from Winning Bid 27 Absence of Effective Management of Project and Contract Oversight 32 Table of Appendices 38 Table 1: TIERS/IEES Expenditures 10 Table 2: Test Methodology 20 Table 3: Unweighted (Raw) Results 24 Table 4: Weighted Results 24 Table 5: Timeliness 30 Figure 1: TIERS/IEES Events 7 i

6 Abbreviations FNS The Food and Nutrition Service of the United States Department of Agriculture HHSC Texas Health and Human Services Commission IEES Integrated Eligibility and Enrollment System OES Office of Eligibility Services, Texas Health and Human Services Commission OIG Office of Inspector General, Texas Health and Human Services Commission SAO State Auditor s Office SAVERR System of Application, Verification, Eligibility, Referral and Reporting (currently the system in use in 251 Texas counties to process cases seeking entitlement benefits) TAA Texas Access Alliance (the deliverer of services under the IEES contract with Accenture) TIES Texas Integrated Eligibility System (the predecessor to TIERS) TIERS Texas Integrated Eligibility Redesign System (currently the system in use in 3 Texas counties, and intended to replace SAVERR) ii

7 Note Concerning Scope of Review Due to time constraints, and consistent with the authority and requirements of section (f)(1) and (h) of the Government Code, the OIG performed a review of limited scope. OIG did not conduct a full investigation or audit in accordance with government auditing standards. The documents reviewed as part of this scope included, but were not limited to, those described below. The OIG review involved the following: Comparison testing of SAVERR versus TIERS for efficiency and effectiveness via processing of identical test cases in each respective system by highly experienced HHSC eligibility staff, who are knowledgeable in both systems and were selected by HHSC executive management; Performing a confidential survey of HHSC Assistance Response Team (ART) workers, who are among the most experienced and knowledgeable of HHSC eligibility staff; Witness interviews of state personnel, including the TIERS and IEES Executive Sponsors; A sample review of TIERS case actions, and other information requested from Accenture and various HHSC personnel; A review of documents provided in response to requests for HHSC Executive Correspondence relating to TIERS, IEES, call centers, and related topics; A review of correspondence between HHSC and Federal Funding Partners, including Advance Planning Documents and Updates; A review of HHSC records relating to budgets and expenditures for TIERS, IEES, and TIES (the predecessor to TIERS); A review of reports from previous internal and external audits and reviews by firms such as: o KPMG (vendor for the federal Single State Audit); o Clifton Gunderson (vendor for state audits); and o SAIC (vendor for TIERS/IEES Independent Verification and Validation). A review of RFP documents, proposals, and related information for TIERS and TIERS/IEES; A review of the TIERS vendor contract with Deloitte and the IEES vendor contract with Accenture, including all executed amendments; and A review of materials relating to the award of the IEES contract that were obtained by OIG during a prior review. Page 1

8 Results in Brief TIERS is the Texas Integrated Eligibility Redesign System, a software application being built under contract that is fully deployed in three of the 254 Texas counties, and that is intended to replace the system in use in the remaining 251 counties. IEES is the Integrated Eligibility and Enrollment System, a new business process for providing entitlement benefits to needy Texans under a call center model. Using TIERS, IEES was launched as a pilot in January 2006 under a privatization contract that is now being cancelled, which has been characterized as the largest contract of its kind in the state s history. 2 To date, more than half a billion taxpayer dollars have been spent on TIERS/IEES. As explained in the Results in Detail section of this report, we concur with the 2006 federal Single State Audit finding that the eligibility process supported by TIERS is not appropriately designed and/or operating effectively to enforce the respective eligibility decisions necessary to ensure clients are eligible and receive proper benefit amounts. To place this in perspective, based on an extrapolation of the findings of their audit, if TIERS were deployed statewide today, it could increase the state s questioned costs, as found in the recent audit, covering 1 million to 1.85 million more needy Texans, and ranging between $2.6 and $4.9 billion more taxpayer dollars. (See Appendix 2). However, we acknowledge that state authorities have recently recognized these risks and have suspended the IEES pilot and the IEES contract is in the process of being cancelled. Further, the Commission has announced that it will not proceed with any further rollout of TIERS until issues raised by the federal Food and Nutrition Service, which is the lead federal funding partner for IEES, have been resolved. (See Appendix 14). The following seven critical events have been identified as contributing to the suspension of the IEES pilot: 1. The removal of the original data warehousing requirement from the initial TIERS contract and subsequent failure to yet replace it with an adequate alternative solution. (See page 12). This has significantly contributed to the current lack of accurate, reliable, readily retrievable, and commercially useable historical case information from TIERS. Page 2

9 2. The commitment of state authorities to the TIERS application without consideration of the possibility of other alternative technical solutions. (See page 16) The selected funding strategy for IEES set the course that ultimately foreclosed the state from exploring other possible technical solutions. 3. The decision to privatize did not use all the relevant data that was available. (See page 19) The analysis by state authorities that determined privatization would be cost effective, relied only on pre TIERS historical data metrics. However, TIERS had been in operation for seven months before the release of the analysis. The relevance of the failure to consider TIERS data metrics is indicated by a comparison study of TIERS versus SAVERR, and statistically valid expert opinions on the respective systems. In one comparison study that was performed April 2007(see Appendix 3) that tested cases in each system, OIG found: o On average, the testers required over 45 minutes more time per case using TIERS than using SAVERR; o The testers also were required to go through anywhere between 94 and 155 more screens in TIERS than compared to SAVERR; o Four of the six test cases resulted in incorrect Food Stamp benefits being calculated using TIERS; and o One of the six test cases resulted in incorrect Food Stamp benefits being calculated using SAVERR. A survey administered to the Assistance Response Team (ART) workers of the Office of Eligibility Services (OES) resulted in the following statistically valid findings: o In the survey, SAVERR significantly statistically outperformed TIERS in every category except necessity of policy knowledge and system availability. The policy knowledge required and system availability for both systems is statistically the same. o In addition, SAVERR: complies with policy and Federal regulations more often than TIERS; determines eligibility correctly more often than TIERS; is easier for the worker to verify that eligibility was determined correctly, than in TIERS; calculates benefits correctly more often than TIERS; completes cases quicker than TIERS; experiences less down time than TIERS; and Page 3

10 is less prone to issuing overpayments than TIERS. 4. Although state authorities initially used the statistically valid SAO scoring methodology, a different methodology was then utilized in the process of awarding the IEES contract. (See page 23) OIG s review found little indication that the state authorities derived maximum benefit from the evaluation stat, methods, and tools known to be available at the time of determining which bidder offered Best Value. 5. State authorities released Deloitte from correction of numerous system deficiencies resulting in a failure to enforce original, fundamental requirements of the initial TIERS contract. (See page 25) In June 2005, State authorities agreed to pay Accenture more than the original contract price to fix defects identified prior to delivery of the TIERS application to Accenture. On October 26, 2005, Deloitte was given final system acceptance for the TIERS application despite numerous outstanding claimed defects. In December 2005, Accenture hired Deloitte as its subcontractor. Deloitte was rehired by the Commission in April 2007 to maintain the TIERS application. 6. State authorities released Accenture from a fundamental contract requirement immediately before going live with IEES, by agreeing to the outsourcing of a core competency requiring program policy knowledge. (See page 27) State authorities made the decision to go forward in knowing that the call center application, MAXIE, would not interface with the eligibility determination application, TIERS. State authorities knew or should have known that program policy knowledge was required to effectively enter data into the TIERS application. Accenture hired additional staff the same month of going live without the state core competencies of sufficient program policy knowledge necessary to correctly enter data into the TIERS application. 7. The absence of effective management of project and contract oversight. (See page 32) State management had no single person dedicated to running the entire IEES effort. Interviews of state authorities indicate they interpreted the per Page 4

11 formance based nature of the IEES contract to mean they had little or no vendor oversight responsibilities. Documents available and provided indicate insufficient Readiness Assessment Testing prior to rollout in January No documentation was provided showing scenario based User Acceptance Testing was performed prior to the IEES rollout. According to interviews of state authorities, adequate User Acceptance Testing had not been performed, although this was HHSC s responsibility under the IEES contract. Page 5

12 Background The events are illustrated in Figure 1 on the following page. The Texas Integrated Eligibility Redesign System, known as TIERS, was conceived in 1999, and operationally assigned to the former Texas Department of Human Services (DHS). 3 The TIERS mission statement, as set forth in the implementation plan, 4 states: To improve Texan s access to health and human services by replacing the current automated eligibility determination system, improving business efficiencies and effectiveness, and establishing the foundation for a comprehensive integrated eligibility determination process. To accomplish the above, specific objectives for TIERS were also included in the 1999 plan and are excerpted below: Reduce the number of manual processes and work around activities; Improve the efficiency and effectiveness of implementing policy changes; Reduce operating and maintenance costs and eliminate processing delays; Improve accuracy of eligibility and benefit decisions [to] increase enhanced federal funding; Improve timeliness of case transactions; and Design automated system and business processes to accommodate conversion of change centers to call centers. 5 As a result of a competitive procurement, in 2001 Deloitte Consulting, LP (Deloitte) was awarded the TIERS contract. Development proceeded, and TIERS was launched as a pilot in Travis and Hays counties in June TIERS was expanded to Williamson county in late Currently, the previous eligibility system remains in use in the state s remaining 251 counties. The 78 th Legislature passed HB 2292, which was effective September 1, This legislation combined twelve former health and human service agencies into five agencies that now comprise the state s current health and human services enterprise system. As a consequence of HB 2292, HHSC assumed operational responsibility for eligibility determinations, and became direct manager over the TIERS contract, and responsible for development of the TIERS application. 6 Page 6

13 Figure 1: TIERS/IEES Events Contracts With (2001) HHSC Contracts With (2007) TIERS Development ( ) (2007) Accenture Contract Terminated HB 2292 (2003) TIERS transfers (2003) IEES Pilot Suspended (2006) Calls for HHSC IEES Launch (2006) Business Case Analysis for Call Centers (2004) Accenture Awarded Contract (2005) RFP for IEES (Integrated Eligibility, CHIP, TIERS) (2004) Page 7

14 HB 2292 also added the current section of the Government Code, requiring that HHSC establish at least one but not more than four call centers for the purposes of determining and certifying or recertifying a person s eligibility..., if cost effective and also requiring HHSC to contract with at least one but not more than four private entities for the operation of call centers... unless [HHSC] determines that contracting would not be cost effective[.] To implement the above, HHSC formed an integrated eligibility project team whose primary purpose was to evaluate current eligibility processes and develop a business case or best value justification to determine if the use of a call center process for eligibility determination is cost effective and responsive to clients. 7 The resulting March 2004 analysis concluded: it is financially [emphasis added] feasible for the State to operate a converged call center to consolidate the eligibility determination function as mandated in HB 2292[; and] The due diligence conducted to date also supports the view that eligibility determination within a converged call center is operationally [emphasis added] feasible, although detailed implementation planning will not be completed until the next phase of the project. 8 The Business Case Analysis listed as a Next Step the preparation and release of a Request for Proposal (RFP) to determine whether outsourcing components of the call center model would be costeffective. 9 The resulting RFP was released in July 2004, seeking vendor bids for what was coined Integrated Eligibility and Enrollment Services (IEES), which involved three major functions: TIERS operation and maintenance; Operation of one or more call centers to support the eligibility process for entitlement programs; and Eligibility determination and enrollment for the Children s Health Insurance Program (CHIP). 10 The RFP for IEES thus sought a possible replacement for Deloitte (the TIERS vendor at the time), a possible replacement for ACS (the CHIP eligibility and enrollment vendor at the time), and a new vendor to Page 8

15 operate one or more call centers. The RFP called for a performance contract under which the state would not prescribe specifics but instead bidders were to offer proposed solutions, although the RFP also required that any solution offered must use TIERS. 11 The RFP also anticipated a division of responsibilities under which the vendor would perform data collection from Texas applicants seeking entitlement benefits, and state staff would be responsible for the eligibility decision. 12 After a bidding process under which IBM and Accenture were the finalists, Accenture was tentatively awarded the IEES contract in February of After negotiations, the contract with Accenture was executed on June 29, In accordance with the language of Accenture s proposal, the contracted deliverer of services under IEES was called the Texas Access Alliance (TAA). On September 30, 2005, the executive commissioner announced intended layoffs to state eligibility staff, explaining that layoffs were anticipated to take effect on or after May 1, 2006, depending on the success of the planned statewide rollout of IEES. 13 This sparked staff resignations over the ensuing months. The IEES project went live as a pilot in Travis and Hays counties on January 20, 2006, with planned further rollout across the state in stages throughout the calendar year. Citing the need for technical and operational improvements, 14 HHSC made the decision to suspend the IEES pilot in May 2006, with HHSC cancelling planned staff layoffs and assuming more operational responsibility for and greater oversight of the application and eligibility processes. By March 2007, HHSC opted to proceed with terminating the IEES contract by mutual agreement. (See Appendix 14). At that time, HHSC also commenced negotiations for 60 day transition contracts with Deloitte for further TIERS development and maintenance, and with Maximus a subcontractor of Accenture for certain other IEES related functions. The Commission has announced that it will not proceed with any further rollout of TIERS until issues raised by the federal Food and Nutrition Service, which is the lead federal funding partner for IEES, have been resolved. (See Appendix 14). Page 9

16 Results in Detail This section expands on OIG s findings with further detail, derived from the limited scope of our review, as explained in the Notes Concerning Scope of Review section. In this section, we first address TIERS/IEES expenditures. Next, we discuss each of seven critical events, more or less in chronological order. TIERS/IEES Expenditures TIERS/IEES expenses are summarized in Table 1. OIG personnel with state budgetary subject matter expertise reconciled the information that was obtainable against expense supplied details provided by the HHSC Financial Services Division. This examination returned a total of $522,696, spent on TIERS/IEES during the period from 1999 to the present. (Appendix 15). In addition, Accenture is claiming over $57 million in unpaid balances, in a letter dated March 27, (See Appendix 4). Table 1: TIERS/IEES Expenditures TIERS Expenditures Phase 1 TIERS 1 $ 289,074, Phase II TIERS $ 136,165, APDU TOTAL $ 425,239, Ancillary Expenses (other outsourcing) $ 363, TIERS TOTAL $ 425,603, IEES Total Accenture Payments (non TIERS) $ 76,993, Other Costs 2 $ 11,061, Worker Retention Bonuses 3 $ 8,480, IEES TOTAL (excluding TIERS) $ 96,535, TIERS/IEES TIERS Total $ 425,603, IEES Total $ 96,535, TIERS/IEES TOTAL $ 522,138, Of this amount, $150, relates to TIES, the TIERS predecessor. 2 Costs including postage/printing, contract management, IV&V contract. 3 To retain qualified staff, after layoffs were announced, HHSC management approved an Eligibility Retention Bonus consisting of two payments of $930 to certain individuals, paid 6 months apart, between June 2006 and February These are the expenses OIG was able to identify within the time allotted and limited scope of the review (See Notes Concerning Scope of Review). Additional expenses, that would not have been incurred Page 10

17 but for TIERS/IEES, appear to be indicated but could not be confirmed by OIG. Such additional expenses may be identified utilizing a comprehensive audit, and may include costs for HHSC staff that were assigned to TIERS/IEES under circumstances that precluded working their normal job duties, costs for overtime pay, and costs for temporary staff. OIG was not able to confirm costs for unemployment compensation benefits due to TIERS/IEES. The amount of unemployment compensation benefits paid by HHSC for fiscal years 2005, 2006, and to date in 2007 indicates an increase in FY2006, 15 but as next explained, OIG could not attribute these costs to TIERS/IEES. Rider 51 to HHSC appropriations for the biennium beginning September 1, 2005 contemplated a reduction of 3,472 full time positions in fiscal year 2006, and a reduction of 4,487 full time positions in fiscal year But this did not result in actual layoffs. Instead, staff resignations occurred in response to the notice of expected layoff that was sent out on September 30, 2005, and that explained layoffs would not occur until on or after May 1, To receive benefits, unemployment compensation law requires good cause when the separation was the worker s decision, and review of applicable law indicates leaving in response to possible layoff does not generally constitute good cause that would qualify a person for unemployment benefits. The TIERS expenses in Table 1 were calculated by reviewing multiple Advance Planning Documents for direct and indirect TIERS costs, such as for the purchase of specific hardware, for technical training, for the contract with Deloitte, and for contracts with other vendors. The IEES expenses in Table 1 were calculated by reviewing expenditures associated with the Accenture contract (including expenses noted through Amendment Y of the contract), adding the amount of Eligibility Retention Bonuses reported by HHSC as paid, and subtracting TIERS maintenance costs (that had already been counted in the review of TIERS costs). Page 11

18 Removal of Data Warehouse This has significantly contributed to the current lack of accurate, reliable, readily retrievable, and commercially useable historical case information from TIERS. The initial TIERS contract with Deloitte, executed in 2001, required that this system must allow the inquirer to access the full range of current and historical information maintained within the TIERS database. This includes the ability for the system to display any historical case action with the information current at the time of that case action. 18 This requirement is a fundamental requirement of federal law for every type of assistance for which TIERS determines eligibility. The requirement as implemented in TIERS was expected to be analogous to a capability in the SAVERR system known as the Generic Work Sheet, or GWS, which is a detailed snapshot in time that shows the ultimate eligibility decision, and also details the results of each eligibility criterion. The GWS includes details on the reported income and resources of each person or persons who applied for benefits, the calculations made based on eligibility policy, and the amount of benefits determined. State authorities decided early in TIERS development in June 2001 to remove the data warehousing requirement of the TIERS contract for this biennium. (See Appendix 5). A contract amendment was later executed, effective October 18, 2002, containing language to Remove TIERS Data Warehousing and Reporting Repository, while providing required Reports. (See Appendix 5). Although the terms data warehouse and reporting repository were not clearly defined in the original TIERS contract or in the October 2002 contract amendment, a later communication from the executive commissioner, dated July 1, 2005, states: My perception (based on something I was told recently) was that as TIERS was being designed initially, someone thought that the historical picture would be accessed through a separate system or by different means. This assumption has proved to be bad, consequently we now have the need to provide that access and info through TIERS. I don t have all the impact assessments yet, but we ll get it changed to provide that historical picture. 19 The above indicates that the developers of TIERS originally designed the system architecture for TIERS to work with a data warehouse, meaning a repository of record for detailed historical case transaction Page 12

19 data. The assumption is that the original TIERS developers were following the industry standards for the time period in question. A review by OIG of reference material applicable to the time period in question indicates that the term data warehouse as used in the original TIERS contract meant a separate application that would pull, (i.e., identically copy) the data at issue from TIERS into a separate environment, at the time of or soon after its creation, and under circumstances that would ensure its accuracy. The separate application would also store the captured data in unalterable form, for later retrieval in a readily understandable format, as a routine business record created in the ordinary course of business. There exists no current ability for TIERS users to obtain reliably accurate, readily accessible, detailed case transaction histories, showing the historical decisions with respect to each eligibility criterion for each type of assistance in a TIERS case, in a readily understandable format. (See Appendix 6). It is notable that the lack of ready access to reliably accurate and specific historical case transaction details in TIERS is a significant departure from the capabilities of the SAVERR system. Unlike SAVERR users who have access to GWS, an eligibility worker using TIERS cannot today retrieve and see in snapshot form reliably accurate historical case transaction details, under a format like the GWS. Yet, it is apparent that ready access to such information is often necessary for the worker to make informed eligibility decisions in the routine course of business. TIERS was initially deployed as a pilot in June 2003, in certain eligibility offices in Travis and Hays counties. By November 25, 2003, the need for access to reliably accurate and specific historical case transaction details was officially entered as a system defect: Users need to be able to view a list of transaction that have been completed, the date of the completion, the mode of the transaction. Users should be able to use this list to select a transaction to view. The view should show the information used to determine eligibility/benefits for the transaction. There is currently no way to do this in TIERS. (See Appendix 6) This defect was classified as priority 4, on a scale of 1 5. Priority 4 Page 13

20 defects are the second most serious in nature, and are reserved for situations such as the system is not issuing benefits correctly. Only priority 5 defects are more serious, and are reserved for situations such as the system is down and thus not functioning at all. On February 27, 2004, due to this case history defect having not been addressed, the following entry was made: The user needs to easily access a particular transaction all information entered or changed from the time the transaction was initiated to the disposition of the transaction including verifications used, related [correspondence] and forms, documentation, etc. The user should be able to see a display of a list of transactions and choose the particular transaction to view. The list of transactions should include data such as the person(s) who completed the transaction, the date of the transaction, the case mode of the transaction. After additional months of the case history defect still not being addressed, a series of communications ensued in the defect tracking system, excerpted as follows: 11/8/2004: Requirements are not understood. Please have them clarified with technical requirements to framework before assignment to framework track. 12/28/2004: Please see note from Rob Bennett dated 11/8/2004. He is requesting clarification of the requirements related to this defect. 12/28/2004: Please do not put back in NMI until you can tell me exactly what information you need. Ron s notes look to me like they are directed to Deloitte not DHS since DHS did not assign to the framework track. 5/25/2005: My note was directed at Deloitte. However, it appears to me that the system fulfills the requirements. We have stored all historical data as type 1 or type 2 tables. With RI, there are separate audit tables. All case actions can be tracked through ad hoc reporting. Case notes are also available to supplement system actions. I recommend this be withdrawn. Page 14

21 6/01/05: As per Craig Pauler and Shari Rider, this functionality is not available in TIERS. User has no way to view lists of transactions other than the transaction just completed. There is no way to determine when a specific transaction was completed, who completed the transaction, or what information was used to determine benefits. 6/22/05: Resolution of this defect is critical to the processes being pursued to meet the needs of OIG. High level meetings are underway with TIERS, OIG, Fair Hearings, Quality Control, hosted by Adriana Ramirez Byrnes. sent to Damien Rodriguez (Deloitte) on the critical need to have this resolved. 9/27/2005: Defect is no longer disputed. Leah Burton and Charlie Bertero met with Deloitte and have agreed to not pursue having this item addressed as a defect. Please evaluate this item for a possible CCB or close out defect and reject related requirements. 10/11/2005: Resubmitting CCB (See Appendix 6). Thus, by October 11, 2005, state authorities had decided that the nearly two year old defect would be addressed by a CCB, that is, a Change Control Board item, meaning the state intended to negotiate an agreed Change Order and pay an amount above the original contract price with Deloitte, for them to fix the problem. However, this priority 4 historical case action details defect still has not been resolved to date. Page 15

22 Commitment of State Authorities to TIERS Application The selected funding strategy for IEES set the course that ultimately foreclosed the state from exploring other possible technical solutions. Under HB 2292 that took effect September 1, 2003, HHSC was required to implement a call center model for determining client eligibility if cost effective. HHSC coined the call center effort at this early stage the new Integrated Eligibility (IE) project. In order for federal funding to be available for IE, state authorities needed federal approval. The vehicle for obtaining such federal approval is the Advance Planning Document (APD) process. The APD process involves submitting to federal authorities a series of documents. Typically, the first is a Planning Advance Planning Document (PAPD). The PAPD announces the state s plans and seeks federal approval to move forward. Upon receiving approval, the state then provides an Implementation Advance Planning Document (IAPD), which is a comprehensive and detailed submission seeking further approval to move towards implementation that includes a requirements analysis, a feasibility study, an alternatives analysis, and a cost benefit analysis. If implementation is approved, the state then proceeds to implement and provides a series of Advance Planning Document Updates (APDUs). An APDU is to keep federal authorities informed of the status of the implementation, and to obtain any needed approvals to continue moving forward. As has been noted, TIERS launched as a pilot in June Thus, in regard to TIERS, state authorities had already submitted and received approvals for a PAPD and an IAPD when the law changed to require IE implementation if cost effective. Since TIERS had already been approved and was at the point of the submission of APDUs, state authorities determined that there existed two alternative strategies for obtaining federal approval for IE implementation. Each alternative was explained in a document dated January 30, 2004, which included an explanation of the pros and cons for each alternative, and a recommendation. (See Appendix 7). The first strategy involved piggybacking the new IE program onto the already approved TIERS project, whereas the second strategy involved treating IE as a separate project from TIERS. Under the first strategy, there would be no choice but to use TIERS in implementing the call center model. By contrast, under the second strategy, the option would remain available of whether or not to use TIERS in the implementation of call centers. If the second strategy were pursued, the ultimate decision of whether TIERS would be used Page 16

23 for call centers would depend on the results of certain analyses, including the studying of alternatives, of feasibility, and of costs versus benefits. The document that presented each strategy recommended the first strategy of piggybacking IEE onto TIERS, meaning, TIERS and IEE would be combined for funding purposes. Strategy two, which separated TIERS from IEE listed several cons, including having to conduct a feasibility study, a cost benefit analysis, and a proposed budget in order to receive federal funding. That is, the decision foreclosed the possibility that a private sector solution might be offered that did not employ TIERS to operate the call centers, but that might be demonstrated as less expensive to the state and more effective operationally than if TIERS were used. State authorities settled on the first strategy, and this is the apparent reason why the resulting RFP for the IEES contract that was released in July 2004 required all bidders to present a solution that would employ TIERS as the system of record for determining eligibility under the call center model. Thus, the decision to go with the first strategy had the effect of foreclosing any potential vendor under IEES from offering an alternative to the use of TIERS. That is, the decision foreclosed the possibility that a private sector solution might be offered that did not employ TIERS to operate the call centers, but that might be demonstrated as less expensive to the state and more effective operationally than if TIERS were used. OIG questioned state management as part of the review to determine whether alternatives to TIERS were explored or considered before the decision was made to require the use of TIERS for any call center operation: Q: Was there ever a time that anything else other than TIERS was considered for the IEES project? Were any other applications considered? And we already made a big investment in TIERS. And if TIERS could support [IE], that was the direction we wanted to head. A: We looked we looked at whether TIERS would be able to support it.... If that determination would have said... it s not just not going to get us there then we would have looked at something else.... And we already made a big investment in TIERS. And if TIERS could support [IE], that was the direction we wanted to head.... So really the evaluation was what if what I have is going to be able to support it, then it s good to go, let s go with that. Q: Was so so as I understand you, there was no discussion at that point as to whether SAVERR would be more ef Page 17

24 fective or TIERS or it was just assumed that we ll see if TIERS works first and then we ll go to explore other options? A: That was basically what we did, yes. 20 Thus, state authorities decided that no alternative to TIERS would be explored, unless after it was first determined that TIERS could not work in a call center environment. This decision was driven, at least in part, by the fact that a large amount of taxpayer dollars had already been invested in TIERS. Page 18

25 Decision to Privatize Did Not Use All Relevant Data The analysis by state authorities, that determined privatization would be cost effective, relied only on historical data metrics from cases not worked in TIERS. However, TIERS had been in operations for seven months preceding the release of the analysis. The September 1, 2003 changes in law that called upon the transition from the traditional service delivery model to a call center model and that called upon the privatization of call center operations, for both requirements, imposed the requirements only if cost effective. Accordingly, it was incumbent on state authorities to first determine whether it would be cost effective to transition from the traditional service delivery model to a call center model, and to determine whether it would be cost effective to operate the call center model through a privatization contract. As found in the previous discussion, state authorities made the decision on or before January 2004 that TIERS would be the system used in the call center model. The following month, HHSC issued an Integrated Eligibility Determination Discovery Report (IE Discovery Report), 21 and then in March 2004, HHSC issued an Integrated Eligibility Determination Business Case Analysis (IE Analysis). 22 These issuances marked the end of the first phase of the IE project, with phase two being the issuance of an RFP seeking vendor bids towards determining whether outsourcing components of the call center model would be cost effective. 23 Essentially, the IE Discovery Report and the IE Analysis compared the traditional model for service delivery to a proposed call center model, and each concluded that the proposed call center model would be cost effective. The analysis involved constructing a sound cost allocation model from which to then predict costs and arrive at conclusions. In regard to the building of the cost allocation model, the IE Discovery Report states: The model calculates current costs and, with the addition of assumptions, future costs. The inputs to the model are based, to a large degree, on extensive, statistical work measurement studies performed by the management analysis unit within the Office of Planning, Evaluation, and Project Management in DHS. The studies present a comprehensive list of activities that advisors (workers), supervisors, clerks, and in some cases, nurses, perform in the execution of determining eligibility. One of the outputs of the study is an allocation of time across each of the activities by type of worker. Our model relies heavily on this data, which is pre TIERS data. As necessary, the model may be adjusted in the future if new analyses and data are published. 24 Page 19

26 Thus, the conclusions of the IE Discovery Report and the IE Business Cases Analysis relied heavily on pre TIERS data. Yet, when these documents were produced, TIERS had been in actual use to process cases for a period of months (since June 2003). Given this, it is unclear why the analyses did not also obtain and consider data from cases processed using TIERS. In order to determine whether the failure to use TIERS case data metrics had any material impact, OIG as part of its review performed a test. (See Appendix 3 describing this test and its results in more detail). The objective of this test was to compare the resource intensity of the administration of cases in TIERS versus SAVERR. Table 2 explains how this test was performed. Table 2: Test Methodology Test Methodology: TIERS vs. SAVERR The OIG tested the respective automation systems with two different fictional but typical eligibility cases: A three person household that is applying for food and medical assistance (i.e., Food Stamps and Medicaid); and A four person household that is applying for food, medical, and cash assistance (i.e., Food Stamps, Medicaid, and TANF). The HHSC Associate Commissioner for Eligibility Services selected three highly experienced eligibility workers to act as Testers. Each tester has strong familiarity with and long experience in the use of both TIERS and SAVERR, and strong knowledge of eligibility policy. Each tester entered each fictional case in TIERS, from beginning to end, and also did the same thing in SAVERR using identical case data. Test Scoring: TIERS vs. SAVERR The following information was used to score the test results: the number of minutes required for each worker to work each case in each system; the number of screens for each case that each worker was required to go through in each system; and the accuracy of the eligibility result (according to what the correct outcome should be based on HHSC eligibility policy). The test results, summarized, are as follows: For the first eligibility case, the testers on average required over 45 minutes more time using TIERS than using SAVERR. The testers also were required to go through 155 more screens in TIERS as compared to SAVERR. Page 20

27 For the second eligibility case, the testers on average required over 45 minutes more time using TIERS than using SAVERR. The testers also were required to go through 94 more screens in TIERS as compared to SAVERR. Accordingly, in performing the studies and analyses during phases one and two of the IE Project, state authorities would have had the benefit of materially different and relevant information if data metrics from cases worked in the TIERS system had also been utilized. From a survey administered to the Assistance Response Team (ART) workers of the Office of Eligibility Services, OIG determined the following statistically valid findings: SAVERR: Significantly statistically out performed TIERS in every category except necessity of policy knowledge and system availability. The policy knowledge required and system availability for both systems is statistically the same. Complies with policy and federal regulations more often than TIERS; Determines eligibility correctly more often than TIERS; Is easier for the worker to verify that eligibility was determined correctly, than in TIERS; Calculates benefits correctly more often than TIERS; Is easier for the worker to verify that benefits were correctly calculated, than in TIERS; Completes cases quicker than TIERS; Experiences less down time than TIERS; and Is less prone to issuing overpayments than TIERS. (See Appendix 8.) Accordingly, although it must be acknowledged that SAVERR is an outdated system, both objective testing and survey results from subject matter experts indicate that, as of today, SAVERR out performs TIERS. The 2006 federal Single State Audit, performed by KPMG, noted the following: Page 21

28 TIERS is not appropriately designed or operating effectively to determine eligibility. TIERS is not appropriate designed or operating effectively to authorize the proper amount of benefits. TIERS lacks the level of data integrity required to support court cases. TIERS does not provide an easily accessible history for each client s file. 55% or 6 of 11 TIERS Food Stamp files reviewed contained supplemental benefits that were calculated incorrectly. 38% of 19 or 50 TIERS Food Stamp files reviewed were found to be incomplete or the benefits calculated using unsupported documentation (6% of the 50 SAVERR files were incomplete). 38% or 19 of the 50 TIERS TANF files reviewed were found to be incomplete or the benefits calculated using unsupported information (6% of the 50 SAVERR files were incomplete). 24% or 12 of the TIERS Medicaid files reviewed were found to be incomplete or the benefits calculated using unsupported documentation (6% of the 50 SAVERR files were incomplete). To place this in perspective, based on an extrapolation of the findings of their audit, if TIERS were deployed statewide today, it could increase the state s questioned costs, as found in the recent audit, covering 1 million to 1.85 million more needy Texans, and ranging between $2.6 and $4.9 billion more taxpayer dollars. (See Appendix 2). Page 22

29 Departure from SAO Scoring Methodology Although state authorities initially used the statistically valid SAO scoring methodology, a different methodology was then utilized in the process of awarding the IEES contract. OIG s review found little indication that the state authorities derived maximum benefit from the evaluation data, methods, and tools known to be available at the time of determining which bidder offered Best Value. In a letter to the federal Food and Nutrition Service dated November 22, 2004, and in response to a question concerning the IEES procurement, the executive commissioner explained: HHSC is utilizing the Texas State Auditor s Office s procurement evaluation tool, an automated instrument based on the State Auditor s Statistical Toolbox. The evaluation tool documents the evaluation process, evaluates the statistical validity of evaluation results, and assists in the identification of potential errors in the evaluation process. The tool ensures evaluation and selection is based on best value.... We believe the process employed by the state will provide substantial documentation of the procurement results. (See Appendix 9). OIG found that HHSC did employ the Statistical Toolbox created by the State Auditor s Office (SAO), as a tool to evaluate the final three IEES bidders. But, OIG also found that the SAO tool was one of three tools employed. The remaining two tools that were utilized are not part of the SAO s Statistical Toolbox. OIG could find no indication that federal authorities were later informed that HHSC would use other evaluation methodologies, in addition to the SAO s Statistical Toolbox. Concerning the SAO tool, HHSC reports there were certain evaluators designated, and that these evaluators were cloistered separately to complete the SAO tool. It thus appears the HHSC s use of the tool complied with normal protocols. HHSC s analysis of the results found no statistically significant difference between the respective scores of the final bidders considered for the contract: IBM and Accenture. 25 In addition to the SAO tool completed by a team of evaluators, a separate team of reviewers were called upon to complete a separate tool, coined a negotiation tool. 26 This tool requested of each reviewer an assessment of the strengths and weaknesses of IBM and Accenture s proposals, with respect to various requirements of the RFP for IEES. This tool also sought comments from the reviewer. A third tool was also employed, as the final scoring tool used before the announcement of tentative award to Accenture that was made in February (See Appendix 10). This third tool was apparently Page 23