QUALITY ASSURANCE AND COMPLIANCE MONITORING AND QUALITY IMPROVEMENT MANUAL

QUALITY ASSURANCE AND COMPLIANCE MONITORING AND QUALITY IMPROVEMENT MANUAL FOR I. Utilization Management II. Temporary Assistance for Needy Families (TANF), and III. Substance Abuse Contract Administration and Treatment July 2005, 1 st Edition

Table of Contents Introduction A. Guiding Principles B. Flowchart of Programs, Functions, Activities and Timeframes C. Annual Quality Assurance Monitoring Plan D. Crosswalk of Processes, Tables, and Appendices Page No. 5 6 6 7 Chapter 1 Description of Quality Assurance Program 1-1 Purpose 13 1-2 Scope 13 1-3 Goals and Objectives 14 1-4 Roles and Responsibilities 15 1-5 Subcontractor QA/QI Workgroup 18 1-6 Amending UM, TANF and Substance Abuse Policies and Procedures Chapter 2 Internal Quality Assurance Monitoring of UM, TANF and Substance Abuse Programs 2-1 Frequency of Quality Assurance Monitoring 19 2-2 Scope of the Quality Assurance Monitoring 19 2-3 Methodologies for Conducting Quality Assurance Monitoring 19 2-4 Due Dates for Submitting Quality Assurance Monitoring 19 Reports 2-5 Format of Quality Assurance Monitoring Reports 20 2-6 Reviewing and Approval of Quality Assurance Monitoring 21 Reports 2-7 Tracking of Corrective Action 21 18 2

2-8 Utilization Management 22 2-9 Temporary Assistance Need Families (TANF) 25 2-10 Substance Abuse Administrative 30 Chapter 3 - Quality Assurance Monitoring of Subcontractors 3-1 Frequency of Quality Assurance Monitoring 35 3-2 Scope of the Quality Assurance Monitoring 35 3-3 Methodologies for Conducting Quality Assurance Monitoring 36 3-4 Due Dates for Submitting Quality Assurance Monitoring 39 Reports 3-5 Roles and Responsibilities 39 3-6 Format of Quality Assurance Monitoring Reports 39 3-7 Reviewing and Approval of Quality Assurance Monitoring 41 Reports 3-8 Tracking of Corrective Action 41 3-9 Level 2 Monitoring 42 3-10 Coordination of Monitoring with DCF 46 Chapter 4 Ongoing Risk Determination of Subcontractors 4-1 Purpose 47 4-2 Roles and Responsibilities 47 4-3 Timeframes For Completing Risk Assessment 47 4-4 Risk Factors 48 Chapter 5 - Quality Improvement Standards and Processes 5-1 PLAN 51 5-2 DO 52 5-3 CHECK 53 5-4 ACT 53 Appendix A UM Data System and Hardcopy Client Information Checklist 56 3

Appendix B Data Staff Questionnaire 61 Appendix C Personnel File Checklist 63 Appendix D Human Resource Staff Questionnaire 64 Appendix E UM & TANF Data Collection Questionnaire 66 Appendix F UM, TANF, Fiscal, Contract Management and Contract Monitoring Staff Questionnaire Appendix G DCF Performance Measures Analysis Worksheet 71 Appendix H DCF Performance Measures Document Checklist 72 Appendix I Tracking DCF Performance Measures Questionnaire 73 Appendix J UM & TANF Required Reports Checklist 75 Appendix K UM Required Reports Questionnaire 77 Appendix L TANF and Substance Abuse Data Systems Checklist 78 Appendix M Substance Abuse Record Management Checklist 81 Appendix N Substance Abuse Financial Management Documentation Review Process Descriptions Appendix O SFBHN Checklist of MIS, Accounting, and Contract Management Policies and Procedures Appendix P Monitoring Report Checklist 90 Appendix Q Subcontractor Quality Assurance Policy and 92 Procedure Checklist Appendix R Subcontractor Quality Assurance Staff Qualifications 94 Checklist Appendix S Subcontractor Quality Assurance Report Card 95 Appendix T Substance Abuse Programmatic Quality Assurance 103 Monitoring Protocols and Tools Appendix U Substance Abuse Programmatic Quality Assurance 117 Monitoring Protocols and Tools Appendix V Facilities Checklist 129 Appendix W Subcontractor Self Evaluation Contract Monitoring 130 68 83 87 4

Introduction A. Guiding Principles There are three (3) guiding principles that are the foundation to SFBHN s Quality Assurance Monitoring policies and procedures. 1. The first principle is to create a compliance environment. That is to espouse a corporate cultural among our subcontractors that embraces the processes and procedures and the benefits that working cooperatively and willingly in a partnership with one another and the coalition will produce. An environment where the importance of an effective quality assurance program becomes an accepted and valued component of each organization. And, by forming these individual quality assurance programs into a team that meets regularly and assistance one another, SFBHN leverages their skills and experience. 2. The second principle states that policies and procedures specified in this manual will provide reasonable assurance that both the SFBHN and its subcontractors are free of material weaknesses. A material weakness is one, which could have a direct and material effect on programmatic quality of care, administration, or financial statement amounts, or a condition in which the design or operation of one or more of these processes does not reduce to a relatively low level the risk that noncompliance with the applicable requirements of laws, regulations, contracts, and grants caused by error or fraud that would be material in relation to a major federal program or state financial assistance project being monitored may occur and not be detected within a timely period by employees in the normal course of performing their assigned functions. It is SFBHN s responsibility to conduct quality assurance reviews of it compliance with requirements of laws, regulation, contracts and grants applicable to each of their major federal programs and state financial assistance projects that it is responsible for. Additionally, it is SFBHN s responsibility to monitor its subcontractors by examining, on a test basis, evidence the subcontractor s compliance with those requirements and performing such other procedures as necessary in the circumstances to provide a reasonable basis for an opinion that the subcontractor does, or does not, have any programmatic, administrative, or fiscal material weaknesses. Compliance with requirements of laws, regulation, contracts and grants applicable to each of their major federal programs and state financial assistance projects is the responsibility of the subcontractor s management. 3. The third principle is that insuring a subcontractor s compliance should be as cost effective as possible. This manual is based on the conceptual framework that quality assurance and monitoring are on two ends of the cost-effective spectrum. An effective QA program identifies non-compliance and inefficient processes 5

continuously throughout the year. In this way, corrections can be implemented quickly reducing the risk of liabilities, wastefulness, and poor performance. At the other end of the spectrum, monitoring occurs very infrequently increasing the risk that can undermine financial security, the continuation of programs and services, client dissatisfaction, and failure to achieve contractual outcomes. Consequently, SFBHN has adopted a continuous quality assurance monitoring process for evaluating the policies and procedures that subcontractors are required to comply with. Additionally, the continuous quality assurance monitoring process was designed to identify weaknesses and more serious material weaknesses. This continuous monitoring consists of a three-tiered process. The first tier is to monitor the subcontractor s quality assurance program on a quarterly basis using this manual as the evaluation tool (see Chapter 3). The second tier involves an annual desk review of fiscal and administrative materials (see Chapter 4). If, in the judgment of a prudent person, either of these first two assessments reveals a serious material weakness, a more intensive and thorough monitoring might be justified and a Level 2 monitoring as described in Chapter 4 will be conducted. B. Flowchart of Programs, Functions, Activities, and Timeframes This manual was written so that all of the tasks and activities necessary to carry out the quality assurance and monitoring processes of the Utilization Management, TANF, and Substance Abuse Administrative functions are specified for each one, even though they may be redundant or occur simultaneously. The following flowchart cross-references the staff, processes, and timeframes that SFBHN must perform to comply with its DCF contractual requirements. C. Annual Quality Assurance Monitoring Plan During the first month of each fiscal year, SFBHN will publish and distribute an Annual Quality Assurance Monitoring Plan. The following is a description of the plan. The Annual Quality Assurance Plan is a two-part document that describes the quality assurance monitoring and compliance activities that will be performed during the fiscal year. Part 1 addresses the activities internal to SFBHN and Part 2 specifies the activities related to the subcontractors. Each part will specify: 1. What processes are to be reviewed, e.g. UM Waiting Lists, UM Adult Assessments, UM Authorization, etc.; TANF Outreach, TANF Screening and Referral, TANF Program Survey, TANF eligibility and service documentation, TANF treatment plans, etc; or Substance Abuse service quality, client eligibility, invoice validity and reliability, and performance. 2. An ongoing risk determination of each subcontractor to determine the types of QA and compliance monitoring that will be conducted during the year. 3. The methodology(ies) that will be used; 6

4. The dates, times, and locations of the QA and Compliance Monitoring for SFBHN s UM, TANF, and Substance Abuse Administrative programs and for each subcontractor; 5. A list of times, dates, and locations of the QA/QI workgroup meetings and training events. 6. Information on how to obtain technical assistance. 7. A list of all the changes to be implemented in both the UM Management Program Operating Procedures and the TANF SAMH Program Guidelines. The QA Coordinator will submit a draft the Annual Quality Assurance and Compliance Monitoring Plan to the Contract Manager, Contract Monitor, and Compliance Specialist during the first week of May. When approved by the all three, the QA Coordinator will distribute this second draft to all SFBHN Supervisors specified in Table 1, District 11 CPU and SAMH Offices, and subcontractors for a two-week review and comment period. This should be planned for the last week in May. The QA Coordinator incorporates all accepted comments and submits a final draft through the Compliance Specialist to the Executive Director. Once the Executive Director reviews and approves the plan, it is finalized, copied, and distributed to all SFBHN Supervisors specified in Table 1, District 11 CPU and SAMH Offices, and subcontractors. D. Crosswalk of Programs, Tables and Appendices The following chart is a quick reference guide to the DCF programs and the applicable Tables and Appendices contained in this manual. DCF Programs Tables Appendices 1. Utilization Management Table 3 A, B, C, D, E, F, G, H, I, J, K 2. TANF Table 4 L, B, C, D, E, F, G, H, I, J, K 3. Substance Abuse Administration a. Data and Administrative b. Fiscal 4. Monitoring Subcontractors a. Ongoing Risk Determination b. QA Programs c. Compliance Table 5 Table 6 L, B, F, M F, N, O, P Q, R, S, T, U, V 7

Systems Functions Data Systems SFBHN Quality Assurance Monitoring Flowchart For DCF Programs (Internal) UM TANF S.A. Admin. Documentation Review Timeframes Frequency KIS Download assessment, placement, authorization, and continued stay information, if applicable, between 2 and 5 client records from each subcontractor. (Appendix A) Download target pop. group, placement, outcome, assessment, and service event data. The size of the sample is based on the number of clients based on the population of TANF clients. (Appendix L) Select, randomly, 100 non-tanf, non-residential service events billed to DCF from the KIS, ensuring that all subcontractors are included in the sample. For each client in the sample download demographic, target pop. determination, assessment, placement, service events and outcome data into a worksheet. (Appendix L) 2 days/every quarter One Family Download same info as downloaded from KIS. (Appendix L) Download same information as downloaded from KIS. (Appendix L) 2 days/every quarter TANF Elect. Elig. Notification Sys. KIS Data Flow System not implemented yet. For each data element given in the Documentation Review checklist, list the various staff/offices that are involved in getting it in the database and validating that the database is complete and accurate, beginning with the originating source and progressing in order. This will require an onsite review for TANF and SA Administration data. NA 1 day at each agency/once a year One Family Compare the data flow charts of the SFBHN data unit for the receipt of data from subcontractors and the downloading of that data into One Family with the actual experience since the last review. KIS & One Family Staff Interviews Interview subcontractor data staff using the Data Staff Questionnaire given in Appendix B. 1 day at each agency/once a year 8

Systems Functions Admin. Sys. Personnel Data Collection Record Management Client Records Invoice Records Subcontractor Files SFBHN Quality Assurance Monitoring Flowchart For DCF Programs (Internal) Cont d UM TANF S.A. Admin. Documentation Review Review the personnel file of each staff. (Appendix C). For TANF staff, check timesheets to verify 10% limit on non-tanf activities. Data Flow For each applicable document specified in the checklist given in Appendix C, list the various staff/offices that are involved in completing the form and validating that it is complete and accurate. Staff Interviews Interview Human Resource supervisor using the HR Questionnaire in Appendix D. Data Flow For each applicable document specified in the checklist given in Appendix A for UM and Appendix L for TANF and SA Admin., list the methods and timeframes used to collect the information, e.g., e-mail, on-line, hardcopy by mail, or hardcopy by courier. Staff Interviews Interview staff responsible for transmitting and receiving documents using the Data Collection Questionnaire in Appendix E. Documentation Review Review the hardcopy medical records of clients selected in Data Systems review. Complete the checklist given in Appendices A and L to evaluate the completeness and compare the electronic system information with the client s medical record. This information is obtained on-site. Validate one TANF Invoice from each subcontractor that includes Treatment, Outreach and Prevention, and One-time Payment as applicable. Validate one non-tanf Invoice from each subcontractor. For the same subcontractors selected for review in Contracting with DCF and Subcontracting, function use the checklist in Appendix M to evaluate the completeness of the contract file. Timeframes Frequency 4 hours/year 4 hours/year 1 hour/year 4 hours/year 1 hour/year 2 days/every quarter 1 days/every quarter 50%/quarter Each FileTwice a year 9

Systems Functions Admin. Sys. Cont d Client & Invoice Records Subcontract Files SFBHN Quality Assurance Monitoring Flowchart For DCF Programs (Internal) Cont d UM TANF S.A. Admin. Data Flow For each data element given in Appendix L, list the various staff/offices involved in getting it into the file and validating that it is complete and accurate. If client information is contained on the document, indicate if the data meets DCF data security standards, i.e., in locked and fire proof cabinet with access limited to those with security authorization. For each item specified on Appendix M list the methods and timeframes used to collect the information, e.g., e- mail, on-line, hardcopy by mail, or hardcopy by courier. Timeframes Frequency 2 hours/ subcontractor Quarterly 4 hours/once a year Client & Invoice Records Subcontract Files Outcome Reporting Staff Interviews Interviews staff with the UM, TANF, Fiscal, Contract Management and Contract Monitoring Staff Questionnaire, given in Appendix F. Documentation Review Download outcome information from One Family database by subcontractor and compare to subcontractor contract requirements. Summarize all subcontractor outcome information and compare to SFBHN s DCF Performance Contract. Use the format given in Appendix G for this analysis. In addition, using the checklist in Appendix H, review the source documents from which this database is Data Flow Data Flow For each outcome, list the various staff/offices that are involved in getting collecting the data and validating that the outcome data is complete and accurate, beginning with the originating source and progressing in order. Staff Interviews Interview subcontractor staff using the Outcome Tracking Questionnaire given in Appendix I. 4 hours/once a year 1 day/every quarter 1 hour/ subcontractor/ once a year 1 hour/ subcontractor/ once a year 10

Functions Systems Required Reports SFBHN Quality Assurance Monitoring Flowchart For DCF Programs (Internal) Cont d UM TANF S.A. Admin. Documentation Review Using the checklist in Appendix J, verify that all required reports were submitted to DCF on time, were complete, and approved. Data Flow Verify that the sources of information contained in these reports were the same as the ones specified in SFBHN s policies and procedures. Staff Interviews Interview staff that prepares these reports using the Required Reports Questionnaire in Appendix K. Timeframes Frequency 2 days/every quarter 1 hour/ subcontractor/ once a year 1 hour/ subcontractor/ once a year Policies and Procedures Documentation Review Select 5 percent of the policies from MIS, Accounting, and Contract Management applicable to administration of the SA program and using the checklist in Appendix O, determine their completeness and correctness. 2 days/once a year Data Flow Not Applicable Staff Interviews Not Applicable 11

Functions Systems Fiscal Systems Contracting Financial Management SFBHN Quality Assurance Monitoring Flowchart For DCF Programs (External) UM TANF S.A. Admin. Documentation Review Data Flow Staff Interviews Documentation Review DCF contract file and subcontractor files are reviewed to check that all required documents are present, completed, correct, dated and signed and all required processes were completed in accordance with policy. For each document list the methods and timeframes used to collect the information, e.g., e-mail, on-line, hardcopy by mail, or hardcopy by courier. For each process, review contract manager s notes. Interview SFBHN contract managers and fiscal staff using the Staff Questionnaire given in Appendix F. Reviews 9 financial management processes to insure compliance with state and federal rules and regulations. (Appendix N) Timeframes Frequency 2 days/once a year for DCF file & 1 day/quarter for Subcontractors 2 hour/ subcontractor/ once a year 1 hour/ subcontractor/ once a year 4 hours/once a year Data Flow Staff Interviews Not applicable. Interviews SFBHN contract managers and fiscal staff using the UM, TANF, Fiscal, Contract Management and Contract Monitoring Staff Questionnaire given in Appendix F. 1 hours/once a year 12

Chapter 1 Description of the Quality Assurance Monitoring Program 1-1 Purpose The purpose of this manual is to threefold. First, it will insure that the SFBHN s Utilization Management (UM), TANF Program, and Substance Abuse administrative services responsibilities are being conducted in compliance with the most currently approved Department of Children and Families (DCF) policies and procedures and contractual terms and conditions. Secondly, it will insure that SFBHN s subcontractors Utilization Management (UM), TANF Program, and Substance Abuse administrative services responsibilities are being conducted in compliance with the most currently approved Department of Children and Families (DCF) policies and procedures and contractual terms and conditions. The tasks, activities, and reports that are contained in this manual are designed to monitor SFBHN s and subcontractors performance and provide DCF an assurance that the goals and objectives are being met. Thirdly, it establishes a quality improvement program. While it is essential to follow the established policies and procedures, it is also important to create an internal process that continuously evaluates these policies and procedures to improve their effectiveness and efficiency. 1-2 Scope The scope of this manual is also twofold. First, it applies to the UM, TANF Program, and Substance Abuse administrative services being managed by the SFBHN. SFBHN must insure that each area of responsibility achieves DCF s goals and objectives through the effective and efficient implementation of the established policies and procedures. Additionally, SFBHN is responsible to insure that the subcontractors also are performing in accordance with DCF s requirements. Secondly, it is intended to provide SFBHN the structure and tools necessary to monitor these same programs and services being operated by the subcontractors to insure that they also have adopted quality assurance and improvement processes that both insure compliance and continuous evaluation of performance.

Quality assurance monitoring assesses four aspects of every compliance issue: If the requirement was performed; if it was completed; if it was completed correctly; and, if the performance met the qualitative standard. Given this four dimensional perspective, the extent of the material to be monitored, the number of responsible parties, and the large number of subcontractors, coordination among the various stakeholders is essential to avoid duplication, excessive preoccupation by clinical staff, and unnecessary costs. Consequently, it is essential that the Annual Quality Assurance Monitoring Plan be submitted to District 11 SAMH Office at the beginning of each fiscal year. 1-3 Goals and Objectives The primary goal of the UM program is to promote effective management of District 11 s substance abuse residential services. The UM program was created to make level 1 children s substance abuse residential services and level 2 adult and children substance abuse residential services more accessible to all clients in need of these services. This is achieved by reducing waiting lists, controlling for over-utilization, and insuring that placements are appropriate. The primary goal of the quality assurance (QA) processes is to monitor the UM staff s performance, the processes they follow, and the required documentation, to determine if the processes and documentation are being completed as specified in DCF s Utilization Management Program Operating Procedures, August 1, 2003 or any revisions. The primary goal of the TANF Substance Abuse and Mental Health (SAMH) program is to promote healthy, substance-free lifestyles with improved functionality including economic and family stability. The primary goal of the quality assurance processes is to monitor the TANF SAMH staff s performance, the processes they follow, and the required documentation to determine if the processes and documentation are being completed as specified in the TANF SAMH Program Guidelines, December 2004 (5 th Edition) or any revisions. There are four primary goals of the Substance Abuse administrative services. They are to insure 1. That quality services are provided, 2. That they are provided to eligible clients, 3. That the payments are based on properly documented services, and 4. That performance outputs and outcomes are achieved. The primary goal of the quality assurance processes is to monitor the SA contract management and fiscal staff s performance, the processes they follow, and the required documentation to determine if the processes and documentation are being completed as specified in the DCF SAMH Performance Contract, Chapter 65D-30, Florida 14

Administrative Code (FAC) and the Mental Health and Substance Abuse Measurement and Data, DCF PAM 155-2, March 2005 (6 th Edition, Version 1) or any updated revisions). The second goal of SFBHN s continuous quality assurance monitoring is to reduce or eliminate weaknesses and particularly any serious material weaknesses. An effective Quality Assurance program identifies non-compliance and inefficient processes continuously throughout the year. In this way, corrections can be implemented quickly reducing the risk of liabilities, wastefulness, and poor performance. Traditional, once-ayear or every other year monitoring occurs so infrequently that it significantly increases the risk that a weakness can go undetected long enough to undermine an organization s financial security, damage the quality of programs and services, result in client dissatisfaction, and cause the subcontractor to fail in achieving their contractual outcomes. The objectives associated with these goals are: 1. To provide DCF the assurance that the program is being implemented with fidelity and that no material weaknesses exist; 2. To identify exemplary performance and shore up weaknesses; 3. To strengthen internal consistency: and 4. To propose and implement improvements that increase effectiveness. The second goal of the UM, TANF Program, and Substance Abuse administrative services is to insure that subcontractors also have effective quality assurance programs that monitors their UM, TANF Program, and Substance Abuse staff s performance and the QA processes they follow. The objectives associated with this goal are: 1. To provide subcontractors training and technical assistance on quality assurance practices and processes; 2. To continuously monitor subcontractor QA programs and operations; 3. To create a team of subcontractor QA staff to act as a Peer Review to identify exemplary performance, assist with desk reviews and QA monitoring, and improve the QA methodologies, tools, policies and procedures; 4. To strengthen systemic consistency of the QA processes; and 5. To propose and implement changes. 1-4 Roles and Responsibilities Although the primary responsibility for quality assurance rests with the QA coordinator, it is also necessary that a number of other SFBHN staff must participate and support the 15

efforts of the QA coordinator. Below is a description of the roles and responsibilities of each of these team members. The team will meet at least quarterly to receive status reports, review findings, recommend proposed changes to the annual QA plan, and discuss implementation problems. TABLE 1 Team Roles Members QA Coordinator Monitoring and Technical Assistance for Subcontractors Compliance Specialist Oversight Responsibilities 1. Develops annual QA plan. 2. Coordinates internal monitoring plan with supervisors. 3. Coordinates with DCF SAMH Program Office. 4. Conducts internal QA monitoring. 5. Writes reports of internal monitoring findings, recommendations, and proposes CAPs. 6. Tracks CAPs and reports progress to supervisors. 7. Conducts at least one quality improvement (QI) project per year. 8. Submit proposed changes to policies and procedures based on QI project findings. 9. Conducts monitoring of subcontractor s QA/QI programs. 10. Writes reports of subcontractor QA monitoring findings, recommendations and proposes CAPs. 11. Track subcontractor s implementation of QA/QI CAPs. 12. Coordinates a subcontractor QA/QI Workgroup. 13. Provides at least 2 training workshops QA/QI processes and best practices related to the UM, TANF, SA programs and services. 14. Provides technical assistance as needed. 15. Writes and submits to QA supervisor, Executive Director, and DCF SAMH a summary report of QA/QI internal and subcontractor activities. 1. Reviews the annual plan and submits to Executive Director for approval. 2. Meets with QA Coordinator for regular status reports. 3. Reviews monitoring reports and CAPs and submits to Executive Director for approval. 4. Approves QI project(s). 5. Approves training curriculum. 6. Oversees subcontractor QA/QI workgroup. 16

UM and TANF Supervisors Contract Monitor Contract Manager Supervisor of Operations, MIS Supervisor and Chief Fiscal Officer Executive Director Coordination Coordination with QA Coord., Risk Assessment, & Monitoring Subcontractors Coordination with QA Coord. & Contract Monitor Coordination Oversight and Approval 7. Reviews proposed changes to policies and procedures. 8. Coordinates with DCF SAMH Program Office. 9. Reviews summary report and submits to Executive Director for approval. 1. Review annual plan and submit comments to Supervisor of Operations. 2. Cooperates with QA monitoring of the UM, TANF, SA programs by making staff and records available. 1. Conducts level 2 monitoring. 1. Completes Desk Review. 2. Reviews QA Monitoring Reports and CAP s 3. Assess risk and recommends levels of monitoring for each subcontractor in the Annual QA. Monitoring Plan. 1. Review annual QA plan and submit comments to QA coordinator. 2. Coordinates QA monitoring of the UM T ANF, and SA programs by making staff and records available. 3. Reviews and submits comments to QA coordinator regarding QI project. 4. Monitors implementation of any CAP and/or QI project. 1. Approves the annual plan and QI projects. 2. Reviews and approves internal and subcontractor monitoring reports and CAPs and summary report. 3. Submits annual plan and summary report to DCF SAMH. Peer Review Coordination 1. Assists with desk reviews and QA monitoring. 2. Recommends improvements to QA methodologies, tools, policies and procedures. 3. Identifies best practices. 17

1-5 Subcontractor QA/QI Workgroup The QA coordinator will organize a Subcontractor QA/QI Workgroup. At a minimum, each subcontractor must identify one person to serve on this workgroup. The workgroup will meet at least once a quarter. The purpose of this group is to discuss compliance issues and best practices, provide technical assistance, conduct orientation for new QA staff and training on QA policies and procedures, assess needs, propose changes to policies and procedures, and review QI project(s) status. Additionally, this workgroup will assist the QA coordinator during the quarterly quality assurance monitoring described in Chapter 3 as peer reviewers. They will provide technical assistance to other subcontractors and help SFBHN improve the QA methodologies, tools, policies and procedures. The workgroup during their first meeting of each fiscal year will develop a list of goals and objectives, benchmarks, and outputs and outcome performance measures. Twice a year the workgroup will prepare a status report that will be submitted to the Executive Director of each agency. 1-6 Amending UM, TANF, or Substance Abuse Treatment Policies and Procedures First, neither SFBHN nor any subcontractor is authorized to amend any of the policies or procedures specified in the DCF Utilization Management Program Operating Procedures, TANF SAMH Program Guidelines, or Substance Abuse rules and regulations without prior written approval from DCF. When a change has been approved the QA coordinator will distribute a notification of the change to all SPFC team members and Subcontractor QA/QI workgroup members and at the beginning of each fiscal year, incorporate the change into a new edition of each policy and distribute it to each agency. Generally, the SFBHN team and the Subcontractor QA/QI Workgroup propose amendments. However, the Executive Director of SFBHN or any of the subcontractors may also submit recommendations to the QA coordinator for evaluation by the SFBHN team and the Subcontractor QA/QI Workgroup. All recommended amendments being submitted to DCF for approval must be supported with: 1. A narrative explaining why the amendment is necessary, 2. A cost/benefit analysis, 3. The proposed language of the amendment, and 4. A transmittal letter approving the request signed and dated by the SFBHN Executive Director. 18

Chapter 2 Quality Assurance Monitoring of SFBHN s UM Program, TANF Program, and Substance Abuse Administrative (Internal) 2-1 Frequency of Monitoring The QA coordinator will perform quality assurance monitoring of the UM program, TANF program, and substance abuse administrative services, quarterly. 2-2 Scope of the Monitoring The QA coordinator will monitor all of the systems that implement the UM program, TANF program, and substance abuse administrative services policies and procedures. 2-3 Methodologies for Conducting A Quality Assurance Monitoring This section describes three techniques that the QA coordinator will use to conduct their quality assurance monitoring. The first involves reviewing a sample of documents using a set of checklists. This component of the review must be completed every quarter. The second involves conducting a data flow analysis. The purpose of the analysis is to determine if the information is being properly reviewed and approved, if there are backlogs, if it represents all of data, and if there are any inefficiencies with the flow of the data. The data flow analysis must be performed at least once a year. Additional reviews are only required, if corrective action was necessary. The frequency of the follow up reviews will be based on the corrective action plan. The last technique consists of structured staff interviews. The purpose of staff interviews is to determine if staff is trained both on the procedures and the content of the information they are responsible for collecting, transmitting, or inputting, if they have problems completing their tasks, and if they have suggestions for improving the procedures. These interviews must be performed at least once a year. Additional follow up is only required, if corrective action was necessary. The frequency of the follow up reviews will be based on the corrective action plan. 2-4 Due Dates for Submitting Quality Assurance Monitoring Reports Reports on Quality Assurance Reviews conducted by the Quality Assurance coordinator are due to the Compliance Specialist 10 working days after each quarter and to the Executive Director 15 working days after each quarter. 19

2-5 Format of Quality Assurance Monitoring Report All Quality Assurance Monitoring Reports will be formatted in the same way. The report consists of three parts. The first part is an executive summary of each of the six (6) systems. Each summary will describe the most significant findings in terms of strengths, weaknesses, and most importantly material weaknesses. Weaknesses and material weaknesses will be subdivided into two types: 1) those that require corrective action and 2) those that require a change in policy. The second part is a detailed analysis of each of the six (6) systems. The analysis consists of five (5) sections: Positive and Negative Findings, Conclusions including the identification of weaknesses, material weaknesses, and potential weaknesses or material weaknesses, Recommended Corrective Actions, Responsible Persons/Offices, and Completion Dates. Table 2 describes each of these 5 sections. This analysis is to be organized in accordance with the order of the systems described in Table 3. The third part consists of the documentation generated by the Quality Assurance Monitoring. The checklists, data flow analyses, and interview questionnaires are included in this section. This material is to be organized in accordance with the order of the systems described in Table 3. TABLE 2 Section Findings Conclusions Description a. Findings related to documentation reviews are to be a statement of the facts as expressed in quantitative terms, i.e., counts, percents, averages, medians, modes, moving averages, and both standard statistics and non-parametric statistics, as applicable. Tables, charts, and graphs may also be used to analyze the data. b. Findings related to data flow is primarily expressed in qualitative terms, e.g., no backlog was observed; supervision insured accuracy, completeness, and timeliness; process did not appear to include unnecessary steps, etc. However, quantitative analyses also can provide important findings, e.g., 20% of the records reviewed were misplaced for 5 months. Data from 15 percent of the reporting units did not indicate supervisory review, etc. c. Findings related to staff interviews are expressed in both quantitative and qualitative terms, e.g., 25% of the staff answered question #7 related to his or hers understand of policy X. Conclusions represent a reasoned deduction or inference from the findings. There are three (3) general conclusions: 1) The process reviewed was materially compliant; 2) The process reviewed was materially out of compliance, and if so they 20

Corrective Action Responsible Person/Office Due Dates were weak, materially weak or potentially so, and corrective action is necessary; and 3) The findings were inconclusive and additional review is necessary. Additionally, each conclusion should be accompanied by an explanation of the logic used to arrive at the conclusion. When the conclusion calls for corrective action, the quality assurance coordinator restates what needs to be corrected and proposes actions or changes necessary to effectively correct the problem(s). The quality assurance coordinator specifies the person or office responsible for the corrective action. Many corrective actions require a coordinated effort of a number of individuals and approval from different supervisors. In these cases, the quality assurance coordinator should list all individuals and office but the responsible or lead person/office should be listed first. The quality assurance coordinator proposes the date that the corrective action should be completed. The length of time should be based on the risk of continuing non-compliance. Client and financial risks that result in endangerment to the client or disallowed services and costs are top priorities and should be corrected in no less than 10 working days or less. 2-6 Reviewing and Approval of Quality Assurance Monitoring Reports Once the quality assurance coordinator completes the draft of the UM, TANF, or SA Administrative Quarterly Quality Assurance Monitoring Report, it is given to the Compliance Specialist for review and approval. After being approved by the Compliance Specialist. The report is given to the UM or TANF supervisor, as appropriate, for review and comment. The UM or TANF supervisor and QA coordinator negotiate any changes and the QA coordinator finalizes the report. The final version is submitted to the SFBHN Executive Director for final approval. 2-7 Tracking of Corrective Action The UM or TANF supervisor, as applicable, maintains a notebook that contains documentation that the corrective action has been taken. For example, if the corrective action involved training the notebook would contain copies of the training attendance log and/or copies of the certificates of completion; if the corrective action involved a change in policy, a copy of the revised policy, signed and dated by the authorized staff would be available; etc. Every two weeks or once a month depending on the risk factors on noncompliance, the QA coordinator will review the notebook and compare it to the corrective action plan (CAP). Following this review, the QA coordinator will report to the Compliance Specialist that the corrective action plan is on track, that some unforeseen barriers have been encountered and further review is necessary, or that the unit has failed to comply with the CAP. In the latter case, the Compliance Specialist will immediately notify the SFBHN Executive Director. 21

2-8 Utilization Management In this section, we will be discussing the systems to be reviewed that are applicable to Utilization Management for assessing whether or not the SFBHN s UM program is being implemented in compliance with DCF s Utilization Management Program Operating Procedures. 2-8.1 List of Systems to be Reviewed The following is a list of the systems that SFBHN has established to implement the UM programs. Some of these systems are automated and some are not. In either case, the quality assurance review will include a review of the staff skills, knowledge, and abilities, the correctness and completeness of the data being collected, and the manner in which the data is being maintained. 1) UM Data System For UM the Knight Information System (KIS) contains client assessment, client placement, waiting list, bed capacity, authorization, and continued stay information. 2) Personnel System Personnel records contain information regarding compliance with staffing skills, knowledge and abilities as well as documentation on training, security, and performance. 3) Data Collection Processes This involves an evaluation of the methods used to collect data to insure that it is accurate, complete, and from the required sources. 4) Record Management The DCF UM policy requires that SFBHN maintain: a. A hardcopy file for each client being paid for by the department for substance abuse residential level 2 services, b. The authorization form, c. Initial assessment if not in KIS, d. Treatment plans and reviews, e. Continued stay authorization forms, if applicable, f. The discharge summary g. A waiting list log, h. High-risk logbook, i. A subcontractor contact log, and j. The latest approved version of the Utilization Management Program Operating Procedures. 5) Outcome Reporting The SAMH contract requires supporting documentation for the outputs and outcomes specified in Exhibit D. 6) Required Reporting The DCF UM policy requires a monthly statistical report on client populations served and waiting list statistics, monthly high-risk authorization report and follow up report, monthly high user authorization report and follow up report, and a list of high risk and high user clients authorized to each subcontractor. 22

The manner in which these techniques will be applied to each of the above systems will be explained in the following table. TABLE 3 System Technique Description 1) KIS a. Documentation Review Download assessment, placement, authorization, and continued stay information, if applicable, between 2 and 5 client records from each subcontractor. The size of the sample is based on the number of beds as follows: 2-5 beds = 2 records; 6-10 beds = 3 records; 11-15 beds = 4 records; and 16 beds or more = 5 records. Complete the checklist given in Appendix A to evaluate the completeness of the required data. b. Data Flow For each data element given in the Documentation Review checklist, list the various staff/offices that are involved in getting it in the database and validating that the database is complete and accurate, beginning with the originating source and progressing in order. c. Staff Interview Interview data staff using the Data Staff Questionnaire given in Appendix B. 2) Personnel a. Documentation Review 3) Data Collection Review the personnel file of each UM staff using the checklist in Appendix C. b. Data Flow For each applicable document specified in the checklist given in Appendix C, list the various staff/offices that are involved in completing the form and validating that it is complete and accurate, beginning with the originating source and progressing in order. c. Staff Interview Interview Human Resource supervisor using the HR Questionnaire in Appendix D. a. Documentation Not Applicable Review b. Data Flow For each applicable document specified in the checklist given in Appendix A, list the methods and timeframes used to collect the information, e.g., e- mail, on-line, hardcopy by mail, or hardcopy by courier. c. Staff Interview Interview staff responsible for transmitting and receiving documents using the Data Collection Questionnaire in Appendix E. 23

4) Record Management 5) Outcome Reporting 6) Required Reports a. Documentation Review Review the hardcopy records of clients selected in Data Systems review. Complete the checklist given in Appendix A to evaluate the completeness and compare the electronic system information with the hardcopy. Additionally, the QA coordinator checks that the Utilization Management Program Operating Procedures being used is the most recently approved version. b. Data Flow For each document given in the Documentation Review checklist, list the various staff/offices that are involved in getting it into the file and validating that the document is complete and accurate, beginning with the originating source and progressing in order. If client information is contained on the document, indicate if the data meets DCF data security standards, i.e., in locked and fire proof cabinet with access limited to those with security authorization. c. Staff Interview Interview UM staff using the UM, TANF, Fiscal, Contract Management and Contract Monitoring Staff Questionnaire given in Appendix F. a. Documentation Review Download outcome information from One Family database by subcontractor and compare to subcontractor contract requirements. Summarize all subcontractor outcome information and compare to SFBHN s DCF Performance Contract. Use the format given in Appendix G for this analysis. In addition, using the checklist in Appendix H, review the source documents from which this database is created. The sample size is the same as specified above. b. Data Flow For each outcome, list the various staff/offices that are involved in getting collecting the data and validating that the outcome data is complete and accurate, beginning with the originating source and progressing in order. c. Staff Interview Interview subcontractor staff using the Outcome Tracking Questionnaire given in Appendix I. a. Documentation Review Using the checklist in Appendix J, verify that all required reports were submitted to DCF on time, were complete, and approved. b. Data Flow Verify that the sources of information contained in 24

these reports were the same as the ones specified in SFBHN s policies and procedures. c. Staff Interview Interview staff that prepare these reports using the Required Reports Questionnaire in Appendix K. 2-9 Temporary Assistance for Needy Families (TANF) In this section, we will be discussing the systems to be reviewed that are applicable to TANF for assessing whether or not the SFBHN s TANF SAMH program is being implemented in compliance with DCF s TANF SAMH Program Guidelines. 2-9.1 List of Systems to be Reviewed The following is a list of the systems that SFBHN has established to implement the TANF SAMH program. Some of these systems are automated and some are not. In either case, the quality assurance review will include a review of the staff skills, knowledge, and abilities, the correctness and completeness of the data being collected, and the manner in which the data is being maintained. 1) Knight Information System, TANF Electronic Eligibility Notification System, and One Family The Knight Information System (KIS), which is SFBHN s client information system contains the client s target population determination, assessment, placement, services, and outcome data. The TANF Electronic Eligibility Notification System is not operational at this time. It will contain the information on the TANF EZ Eligibility Form. The One Family system, which is DCF s client and service data contain the same information as on KIS. 2) Personnel System Personnel records contain information regarding compliance with staffing skills, knowledge and abilities as well as documentation on training, security, and performance. 3) Data Collection Processes This involves an evaluation of the methods used to collect data to insure that it is accurate, complete, and from the required sources. 4) Record Management The DCF TANF SAMH Program Guidelines requires that SFBHN maintain: a) A hardcopy or electronic file for each client being paid for by the department and that file contain the new and renewed TANF EZ Eligibility Form (during July), b) Copy of confirmation of receipt of TANF EZ Eligibility Form, c) Tracking log for residential placement if applicable, d) Re-certification of TCA eligibility for each month in residential services, e) TANF SAMH Program Eligibility Confirmation form f) Eligibility Notification form indicating discharge, g) Medical Incapacity form if applicable, h) TANF participant logs, i) Outreach/Prevention Logs, and 25

j) Subcontractor invoice validation worksheets k) TANF SAMH & One-Time Payment l) Copy of latest version of the TANF SAMH Program Guidelines. 5) Outcome Reporting The SAMH contract requires supporting documentation for the outputs and outcomes specified in Exhibit D. 6) Required Reporting The DCF TANF SAMH policy requires surplus/deficit reports, quarterly screening and referral analysis reports, waiting lists and referral time frames reports, participant complaints, monthly contingency fund reports, needs assessment reports every two years. The manner in which these techniques will be applied to each of the above systems will be explained in the following table. TABLE 4 System Technique Description 1) KIS a. Documentation Review Download target population group, placement, outcome, assessment, and service event data. The size of the sample is based on the number of clients based on the population of TANF clients. For subcontractors with a population of 5 to 10 clients the sample size = 100%; if 11 to 20 clients the sample size = 10 records; if 21 to 40 clients the sample size = 50% of the records; if 41 or more clients the sample size = 20 records. Complete the checklist given in Appendix L to evaluate the completeness of the required data. 2) One Family b. Data Flow For each data element given in the Documentation Review checklist, list the various staff/offices that are involved in getting it in the KIS database and validating that the database is complete and accurate, beginning with the originating source and progressing in order. This will require on-site reviews of the subcontractors. c. Staff Interview Interview SFBHN and subcontractor data staff using the Data Staff Questionnaire given in Appendix B. a. Documentation Review Download the same information specified in 1) KIS, above, from the One Family system. Complete the checklist given in Appendix L to evaluate the completeness and compatibility of both systems. b. Data Flow Compare the data flow charts of the SFBHN data unit for the receipt of data from subcontractors and the 26