QUALITY RATING AND IMPROVEMENT SYSTEM LOCAL EVALUATION REQUEST FOR QUALIFICATIONS Application Deadline: 5:00pm, September 16, 2014 APPLICATION INFORMATION: 1. All sections of the application must be completed. Applicants are responsible for ensuring that applications are received by the deadline. 2. Completed applications should be delivered, mailed, or emailed to: Attn: Jane Bernzweig First 5 Alameda County 1115 Atlantic Ave. Alameda, CA 94501 Jane.Bernzweig@first5alameda.org 3. Applications must be received by 5:00pm, September 16, 2014 applications may be mailed, delivered or emailed. PLEASE SUBMIT ONE COPY OF THE APPLICATION INCLUDING NARRATIVE, REFERENCES, RESUMES, REPORT SAMPLE, PROJECT BUDGET, AND BUDGET NARRATIVE. FOR EMAIL SUBMISSIONS, PLEASE SEND ONE PDF DOCUMENT WITH ALL REQUIRED DOCUMENTS INCLUDED. 4. All applicants will be notified by Friday, September 29, 2014. If you have any questions, please contact Jane Bernzweig via email at jane.bernzweig@first5alameda.org QRIS Local Evaluation RFQ 9.4.14 1
SECTION: 1 INTRODUCTION The California Department of Education Child Development Division (CDD) was awarded federal funding through the Race to the Top Early Learning Challenge grant. The goals of this grant are to increase the number of low-income and high need young children who are enrolled in high-quality early learning programs through the design and implementation of a quality rating and improvement system that conforms to national standards. California is in the third cycle of states working on Early Education Systems. First 5 Alameda County (F5AC) is the lead agency for one of 16 county jurisdictions that is developing a local Quality Rating and Improvement System (QRIS). F5AC is coordinating with other Bay Area counties to develop a regional approach. This request seeks proposals to conduct a local evaluation of the first two pilot years of First 5 Alameda County s program. Between 2013 and 2015, Alameda County will be enrolling up to 100 early care and education sites to be part of our Quality Rating and Improvement System (QRIS) pilot. The Alameda County QRIS Advisory group requested a local evaluation of the program to address immediate, mid-term and long-term questions that will not be addressed by the state evaluation. Many of the questions in the local evaluation are related to learning provider and stakeholder perspectives about, What works and what doesn t. The QRIS Advisory group would like to learn about participant motivation, retention, capacity, leadership, systems change and the successes and challenges of implementing a model to promote movement on the tiers of the California Department of Education Quality Continuum Framework. 1 In addition to provider perspectives, First 5 Alameda County would also like to learn about the county s coaching and technical assistance (TA) resources to promote quality improvements. This analysis will address motivation for participation, coaching and technical assistance capacity, progression on the California Department of Education Quality Continuum Framework, site readiness and leadership, the cost of participation and the integration of quality improvements. SECTION 2: BACKGROUND Representatives from F5AC and the Alameda County Local Childcare Planning Council (LCPC) have been engaged with the CDD and other participating counties to develop a consistent state wide approach to California s locally driven QRIS. This work is important because there is a need to coordinate, build on linkages, strengthen partnerships, and ensure effective and accountable use of limited public resources. Ultimately, the process of building a QRIS will result in: A shared definition of quality based on reliable and validated research; A comprehensive and consistent approach to assess quality services 1 http://www.cde.ca.gov/sp/cd/rt/rttelcapproach.asp. QRIS Local Evaluation RFQ 9.4.14 2
Access to a system that captures and supports quality improvement, especially for early childhood education programs serving high need children and families; Consistent language and messaging for the parents and consumers of services that increases awareness and demand for quality early care and education. SECTION 3: SCOPE OF WORK Under the leadership and guidance of F5AC, the local evaluation contractor will be responsible for conducting structured interviews and focus groups with cohorts of providers, stakeholders and service providers, coding, analyzing, summarizing and reporting results to F5AC. Some of the data for QRIS is being collected by technical assistance providers and entered into either F5AC ECC Online data base or the Bay Area counties WELS database. The selected contractor will perform additional quantitative data analysis to augment the understanding and description of the local QRIS system including analyses, summaries and reporting of quality improvement plans and technical assistance logs housed in these databases. The local evaluation contractor will be responsible for combining the results of qualitative and quantitative data into a final written evaluation report for F5AC and other county stakeholders. At the broadest level the local evaluation must address the following questions: Why do providers choose to participate? Does the Framework s definition of quality align with providers definitions of quality? Were incentives meaningful? Were we able to retain sites for 2 years? Do we have the coaching and technical assistance capacity to address site needs? Have participating sites progressed on the framework? How does site readiness and leadership relate to progress? What does it cost a site to participate? What does it cost QRIS to move sites on the tiers? Are providers satisfied with the technical assistance they receive? What dosage and intensity is necessary to facilitate progress? Was QRIS able to integrate different supports into a seamless experience for providers? The scope of work will include: 1. Development of a sampling and analytic plan which integrates multiple sources of data to address evaluation questions (See attachment A for the initial evaluation plan). Work with F5AC evaluation and program staff to finalize this sampling and data analytic plan. 2. Work with F5AC evaluation and program staff to finalize interview and focus group questions (see attachment B for sample interviews). QRIS Local Evaluation RFQ 9.4.14 3
3. Schedule and conduct culturally and linguistically responsive interviews (in English, Spanish, Cantonese, Mandarin or Vietnamese 2 ) over 2 or 3 time periods throughout the 20 month contractual period with a sample (size TBD) of participants. Sample should include all types of participating site and sites who have chosen to discontinue participation. 4. With guidance and direction from F5AC evaluation and program staff, transcribe, code and analyze interviews using qualitative research analytic tools such as N- Vivo, as necessary. 5. Communicate with evaluation specialist and data analyst on access to the required data fields for analysis from ECC Online, QRIS applications and WELS databases. 6. Combine and analyze data from interviews and focus groups with rating and quality improvement tracking data using platforms such as SPSS, Access, SAS, etc 7. Complete progress reports on key results and challenges 8. Analyze and Summarize all data to answer questions for final report 9. Conduct further analyses, if needed 10. Revise summary of findings, if needed 11. Present summary of findings and recommendations to stakeholders 12. Produce Final Report 13. Provide all documents and electronic files for F5AC SECTION 4: TIMELINE, FUNDING AND DELIVERABLES Contract Timeline The term of the contract will be October 1, 2014 through June 30, 2016. Funding First 5 Alameda County has allocated $50,000 toward evaluation costs. Proposals that exceed this amount will not be considered. Contractor payments will be made based on actual approved expenses and satisfactory work performance of identified deliverables. 2 In the initial sample of providers enrolled in QRIS in the county, we learned that approximately 15% speak Spanish, 13% Cantonese, 3% Mandarin and 1% Vietnamese. Therefore, it is possible that providers may prefer to be interviewed in their primary language rather than English. QRIS Local Evaluation RFQ 9.4.14 4
SECTION 5: DELIVERABLES AND TIMELINE The selected Contractor will conduct the following deliverables in the timeline indicated: Deliverables Building on existing evaluation design work, draft sampling and data analytic plans and work with F5AC evaluation and program staff to finalize plans Work with F5AC program and evaluation staff to finalize interviews and focus group questions Work with F5AC program and evaluation staff to translate, schedule, conduct, analyze and summarize interviews including, but not limited to: o One- hour, structured interviews with a sample drawn from 60 center directors/family child care operators over 2-3 time periods. Sample to include different participant types including Family child care, Head Start, Title V, Title 22 and non- retained providers. Timeline By October 30, 2014 By November 15, 2014 Submit midterm and year-end progress reports including key results and challenges by: Dec 31, 2014 July 1, 2015 January 15, 2016 o One-time focus group of TA providers o One-time focus group with QRIS rating staff o Coding and analyzing interviews and focus groups using appropriate qualitative analytic tools Work with F5AC data analyst and evaluation staff to access files from applications, ECC Online and WELS data bases to analyze and summarize data including, but not limited to: QRIS applications, ECC Online TA logs, Quality Improvement Plans and WELS ratings, assessments and tier movement information Develop and package final written report combining data from all sources and present key findings Final report and presentation due June 30, 2016 SECTION 6: ELIGIBILITY, APPLICATION INSTRUCTIONS, AND BUDGET Eligibility F5AC is seeking a consultant or a consulting team that has expertise and demonstrated QRIS Local Evaluation RFQ 9.4.14 5
ability in the following areas: Demonstrated experience working with stakeholders to develop a sampling and analytic plan for complex quantitative and qualitative data from multiple data sources Demonstrated experience and sensitivity to conducting interviews in culturally and linguistically diverse early childhood settings on quality improvement Ability to analyze qualitative data from interviews, possibly using qualitative software analysis tools Qualified interviewers must: o Participate in required meetings o Have experience conducting interviews in early childhood settings o Conduct some interviews in languages other than English (Spanish and Cantonese) o Sign a conflict of interest statement, which discloses any early care settings for which the applicant may have a conflict o Sign a statement of confidentiality o Hold a valid California driver s license in order to conduct on-site interviews o Demonstrate the ability to conduct themselves professionally and respectfully in a wide variety of culturally and linguistically diverse ECE settings Ability to conduct secondary data analyses from exported data using statistical software such as SPSS or SAS Demonstrated experience conducting qualitative and quantitative research and/or evaluation and the ability to synthesize complex qualitative and quantitative results into a comprehensive report and presentation Prior experience working in Alameda County is preferred. SECTION 7: APPLICATION INSTRUCTIONS Project Narrative: Please respond in no more than four (4) total pages of narrative in font size 12. Resumes do not count toward the narrative page limit. a. Knowledge: Describe your knowledge of QRIS initiatives in early childhood settings. b. Qualifications: Describe your qualifications to develop evaluation and sampling plans, organize, conduct and manage interviews in center- based and family child care settings over a 20-month period. Describe your ability to analyze qualitative data. c. Reliability and Validity of Interviews: Describe how you will ensure reliability and validity of interviews. How will you ensure that interviews are conducted in a professional and timely manner? How will you ensure that interviews are conducted in a culturally and linguistically appropriate manner? QRIS Local Evaluation RFQ 9.4.14 6
d. Experience: Describe your experience analyzing and combing complex exported data from a variety of qualitative and quantitative data sources. e. Challenges: What do you anticipate the challenges to be? References: Please provide 3-5 client references Resumes: (2-page limit for each resume): Please provide resumes for researchers and interviewers Report Sample: Please provide a sample of research or evaluation you ve completed on combining complex data from a variety of data sources. Project Budget: Please provide a budget of your proposed expenses, using the attached template. Please include the following: The cost of developing sampling and data analytic plans and participating in preliminary review of interview and focus group tools The cost of translating and conducting a sample out of a possible 60 interviews in different languages, 2-3 times over the course of 20 months The cost of transcribing, coding, analyzing and summarizing the interviews over time The cost of analyzing quantitative data The cost of combining qualitative and quantitative data, summarizing, reporting and presenting results Fixed costs Note: the budget form does not count toward the four page limit. 6. Budget Narrative [one page limit]: Please provide a narrative description of your budget request. SECTION 8: SELECTION CRITERIA & SELECTION PROCESS Selection Criteria The criteria by which each response will be judged are as follows: 1. Demonstration of understanding The extent to which the applicant communicates knowledge of the quality improvement process in early childhood programs. 2. Applicable experience The extent to which the applicant demonstrates experience and expertise in working with complex exported data from multiple data systems. 3. Applicable approach The extent to which the applicant describes a rigorous approach to interviews and qualitative research in early care and education including both validity and reliability and respectful and culturally responsive conduct. 4. Fit with First 5 Alameda County The degree to which the applicant s approach and values are consistent with that of F5AC. 5. Cost Should be reasonable and well justified. QRIS Local Evaluation RFQ 9.4.14 7
Selection Process All applications will be reviewed by an internal staff committee using a ranking tool. Applicants selected for the second phase of the application process may be invited for an interview. References will be consulted. SECTION 9: OTHER INFORMATION 1. No response will be considered for award unless submitted in full, delivered on or before the date specified in the RFQ. 2. Specifications are carefully prepared describing the services desired. An appropriate addendum will be issued to all applicants, if necessary, stating revisions, deletions, or additions to be made to the specifications as a result of any inquiries. 3. Solicitation of bids in no way obligates F5AC to purchase any or all of the described goods or services prior to bid award by the Commission. 4. F5AC shall require indemnification on behalf of itself and Alameda County entities with any agencies or businesses into which it enters into agreements. In addition, F5AC requires all entities to provide evidence of both workers compensation and general liability insurance. QRIS Local Evaluation RFQ 9.4.14 8
Quality Rating and Improvement System Local Evaluation Budget Proposal Form Budget Period: 10/1/2014-6/30/2016 Applicant Name: Attachment 1 QRIS Local Evaluation October 1, 2014 June 30, 2016 Proposed Budget Form STAFF NAME & POSITION Hourly rate Proposed Budget SUBTOTAL GENERAL EXPENSES (include brief description) Proposed Budget SUBTOTAL SUBCONTRACTORS/CONSULTANTS (List separately) Proposed Budget QRIS Local Evaluation RFQ 9.4.14 9
SUBTOTAL TOTAL Proposed Budget SUBTOTAL OF DIRECT EXPENSES (Add subtotals for Personnel, General, Equipment, Subcontractors) ADMINISTRATIVE/INDIRECT COSTS (Maximum is 15%) TOTAL PROJECT EXPENSE QRIS Local Evaluation RFQ 9.4.14 10