2013-14 Program Year. Audit Report. Prepared for the California Community Colleges Chancellor s Office

2013-14 Program Year Audit Report Prepared for the California Community Colleges Chancellor s Office

Purpose The California Community Colleges Chancellor s Office (CCCCO) requested an audit for the 2013-14 fiscal year for its California Partnership for Achieving Success (Cal-PASS Plus) grant. In general, the CCCCO directed Cal- PASS Plus (CPP) to identify funding sources and to assess Cal-PASS expenditures of CCCCO grant funds as related to program objectives and deliverables. Background Cal-PASS, the statewide database established by the CA Community College Chancellor s Office (CCCCO) for the post-secondary community, was initially created to promote student achievement through the collection, analysis, and review of student assessment data. The inaugural version of Cal-PASS was a valuable asset and possessed significant potential to play an important role in closing the achievement gap, increasing college readiness, and ensuring workforce preparedness. In the community college system, the demand for data to guide decision-making exists on many levels, including for the purposes of tracking remediation, transitions, certificate completion, associate degree attainment and completion of baccalaureate degrees. Moreover, throughout the education community, significant demand exists for actionable data related to specific issues, such as Career and Technical Education (CTE), and related to specific populations, including students with disabilities, ethnic minorities, lowincome students, and English Language Learners. However, prior to 2012, the Cal-PASS system was unable to provide big-picture analysis or actionable data reports, lacked gap-identification ability, and interacted with educators primarily through institutional researchers and insulated localized consortia. In particular, users were frustrated by the inaccessibility of data. At the time, Cal-PASS was designed to be a query-based system. In this type of system, data is not regularly accessible, nor is it actionable over multiple systems. Query-based reporting limits analysis to a micro or local level and requires intensive staff support that is not sustainable over a large scale of users. As a result, the system lacked accessibility for widespread use and did not adequately provide for the replication of effective practices required to better inform education policy. In July 2012, administration of the Cal-PASS Plus grant was awarded, through a competitive proposal process, to San Joaquin Delta College (SJDC). A partnership was developed between SJDC, Educational Results Partnership (ERP), and the San Joaquin County Office of Education (SJCOE). This created a team that sought not only to effectively host and manage the existing Cal-PASS system but to expand it. SJDC subcontracted with ERP, to provide some project management, data analysis, and programming expertise. More importantly, the team determined that merging the ERP K-12 actionable data system with Cal- PASS would elevate the functionality, scope, and usability to a new level. SJDC also subcontracted with the San Joaquin County Office of Education (SJCOE) to expand the scope of participation in CPP through outreach to the county office of education infrastructure and by maximizing the already established, but isolated, Professional Learning Councils. Since 2012, the expansion of the original Cal-PASS system to become Cal-PASS Plus (CPP), has created an actionable, statewide system of data from pre-k to employment that promotes collaboration and problemsolving among the faculty and education leaders who comprise 11 Regional Learning Councils (RLCs) and numerous other collaborative partnerships focused on student success. Through these efforts, the new CPP system seeks to develop and support a statewide community of practice that leads the way in closing achievement gaps, increasing college readiness and completion, and ensuring workforce preparedness. Cal-PASS Plus Program and Financial Audit July 2013 June 2014 2

Governance CPP is managed as a partnership between SJDC, SJCOE and ERP, a 501(c)(3) non-profit organization located in Sacramento, California. CPP is also supported by an Advisory Board, which meets quarterly to provide guidance and inform the work. The CPP Advisory includes representatives from each academic segment including: K-12, the California Community Colleges (CCC), California State Universities (CSU), Universities of California (UC), and the private post-secondary community. It also includes representatives from the business and workforce development community. Staffing Plan The CPP staffing plan is as follows: SJDC provides administration and oversight personnel, project management, a technical support team to manage the data servers and equipment, and a Director of Outreach for post-secondary institutions. In addition this position monitors the grants, goals and objectives, and additional integrated projectswith financial support personnel. ERP, as a subcontractor, provides executive, administrative and project management personnel, outreach staff, a data and programming team, technical support, and a administrative support personnel. SJCOE, as a subcontractor, provides a Director of Outreach for K-12 institutions and administrative support personnel. Roles and Responsibilities The primary functions necessary to maintain and expand the work of CPP fall in to 3 categories, the roles and responsibilities of which are shared across the governance structure to maximize the strengths of each partner. These primary functions include: 1) Administration 2) Outreach and Communication 3) Data and Technology Administration SJDC, SJCOE, and ERP jointly manage CPP. SJDC serves as the project monitor, outreach component, fiscal agent, and provides oversight of the project to ensure timelines and deliverables are met, as well as fiscal accountability. SJCOE provides an outreach coordinator and administrative support. ERP oversees the dayto-day operations of CPP, including some project management, data analysis, and programming expertise. Outreach and Communication The administrative team of SJDC, ERP, and SJCOE collaborates, under guidance of the CCCCO, to direct overall outreach and communication efforts, which include: identifying and addressing the needs of CPP users; cultivating relationships with academic and research institutions; managing technical assistance and training; and, coordinating with all major CCC technology related initiatives. In addition, ERP contributes to program-specific outreach and communication efforts. This includes efforts to support specific pathways, such as career technical education and the CTE LaunchBoard, as well as programs intended to serve specific subgroups, such as foster youth. ERP also manages some integrated outreach with private and non-profit research and academic institutions. As noted in the staffing plan, both SJDC and SJCOE have a Director of Outreach. The two Directors of Outreach are primarily responsible for fostering relationships with academic institutions in the K-12, CCC, CSU, and UC segments. Additional roles and responsibilities for the Directors include: facilitating ongoing data acquisition efforts; supporting training, technical assistance to ensure users can maximize the CPP website, system of data, and reporting tools; and supporting the development and expansion of Regional Learning Cal-PASS Plus Program and Financial Audit July 2013 June 2014 3

Councils. While the Director of Outreach for SJDC focuses on higher education segments, SJCOE focuses on expanding the scope of participation of K-12 school districts in CPP through outreach to the county offices of education infrastructure. Data and Technology SJDC provides the back-end technology infrastructure, including servers, network and hosting for CPP. However, ERP provides data management, programming, design and user interface. ERP possesses a twelveyear track record of managing a similar system to Cal-PASS and possesses extensive expertise in data management and analysis, as well as in developing, programming and maintaining the dynamic, open-source CPP system of data. Since acquiring the Cal-PASS grant in 2012, the team has converted CPP to a longitudinally linked system of data. This means that comprehensive student-level data available in the system is linked across data sets, and therefore over multiple institutions, and over many years. This expanded functionality allows for the tracking of student cohorts on many facets, including demographics, achievement, transitions, and mobility. When combined with employment outcomes and college completion data, the system of data can now backward map across multiple institutions to identify higher-performers. A significant and ongoing portion of the work conducted by ERP includes uploading, organizing, storing, and securing institutional data from members of CPP to support the continued expansion of the CPP system of data. ERP is also primarily responsible for the coordination of the technical aspects of website design, creation, and ongoing development to refine and expand the comprehensive reporting tools available to CPP users. Finally, ERP provides all technical support and training efforts for institutional members of CPP. Scope of Audit The scope of information provided includes the 2013-2014 fiscal year. This is the second program year for the CPP grant under the administration of SJDC. Since 2012, CPP has embarked on a series of improvements and expansions to the system of data that support the goals defined in the awarded grant application, which include: 1) Develop a Robust and Actionable K-16 Database 2) Guide Decision-Making Through Big Picture Analysis 3) Expand the Scope of Participation 4) Develop a Statewide Community of Practice The funding details provided in this document correspond directly with the program objectives outlined in the awarded grant application. Recent expansions of the CPP program and activities, supported by additional funding sources, are also described in this document. Funding details are broken down in to five distinct categories, which include: 1) Data and Technology 2) Outreach 3) Travel 4) Communication 5) Administration Budget Summary The allocation of fiscal resources in the second program year of the CPP grant has focused on the creation, development, and implementation of www.calpassplus.org. The CPP website now offers users a variety of features, including data analysis and reporting tools at local, regional, and statewide levels. Across the five funding categories, resources were allocated primarily to outreach, data and technology personnel. Each Cal-PASS Plus Program and Financial Audit July 2013 June 2014 4

category is presented below as a percentage of overall expenditures. Detailed breakdowns of project-based funding allocations are provided. Additional Resources Generated The following represents additional resources (non-ttip funding) generated by each partner to support the objectives of CPP. This includes funding from philanthropic funding obtained via ERP, such as: Stuart Foundation grant funds to support engagement of academic institutions around foster youth college and workplace outcomes; College Access Foundation grant funds to support the analysis of financial aid accessibility and impact on outcomes; and, Rockefeller Philanthropy Advisors grant funds to support the study of Deeper Learning best practices. The following diagram also includes resources leveraged by SJDC through the CCCCO to support the CTE LaunchBoard and the Multiple Measures Assessment Project contracts. Additional Resources by Entity 2013-14 800,000 700,000 600,000 500,000 400,000 300,000 EdResults Delta College SJCOE 200,000 100,000 0 Tech/Data Outreach/Data Collection Travel Cal-PASS Plus Program and Financial Audit July 2013 June 2014 5

Distribution of Resources by Function The graph below represents the total expenditures of CPP by function. These functions support the partners in accomplishing the objectives, projects, and activities of CPP. 550,000 500,000 450,000 400,000 350,000 300,000 250,000 200,000 150,000 100,000 50,000 0 Total Expenditures Total Expenditures Significant Return on Investment As the above figure demonstrates, significant expenditures were made in the areas of outreach and data collection and technology and data management. These areas represent the core of the Cal-PASS Plus operation. Outreach When SJDC took over the Cal-PASS project in 2012, almost all of the MOUs were set to expire between 2013-2015. As a result, Cal-PASS Plus undertook a statewide effort to obtain revised MOUs from all CPP members, and to increase the number of MOU holders overall. As a result of these outreach efforts, the number of CPP users from 970 to 4,162 since 2013. The previous version of the MOU possessed significant limitations on data sharing among institutions. Maximum usability of the CPP system requires open sharing among institutions. CPP embarked on a statewide campaign to build consensus around open sharing and to increase data sharing among institutions. This included the development of new MOU terms to support maximum usability. Through this effort, CPP has increased the number of participating institutions that share data with all members to 25% for K-12 districts and 40% for colleges and universities to the partnership between CPP entities expanded the scope of participation in CPP through outreach to the county office of education infrastructure. The isolated and regionalized nature of the Professional Learning Cal-PASS Plus Program and Financial Audit July 2013 June 2014 6

Councils led the team to redefine consortia into Regional Learning Councils (RLCs). These are more collaborative partnerships initially based out of the 11 CISC regions but ultimately expanded to include, 35 other collaborative partnerships (focused on foster youth, CTE pathways, etc.) focused on student success. The RLCs and other collaborative partnerships are comprised entirely of volunteer administrators, faculty, researchers, and staff who elect to participate in regional and statewide communities of practice. These efforts lead the way in closing achievement gaps, increasing college readiness and completion, and ensuring workforce preparedness. Because no comprehensive data collection mechanism exists for K-12, and no statewide K-20 database previously existed, efforts to expand memberships and collect data have been institution by institution. In a state as large and diverse as California, hands-on outreach efforts require significant resources, including frequent and ongoing travel, to meet the needs of CPP users. Data Data is the transactional currency of Cal-PASS Plus. The system of data and its dynamic user tools are only as useful as its data is comprehensive. The data in the system must be robust, accurate, and accessible to users. Previously, the Cal-PASS system contained data primarily derived from California s Community Colleges (CCC) and individual K-12 school districts. The team s decision to merge data systems to build a more comprehensive structure has led to Cal-PASS Plus (CPP) becoming an actionable, statewide system of data from pre-k to employment. In its new form, CPP boasts a large, complex system of data with a number of participating institutions that is unprecedented on a national scale and includes K-12, CCC, CSU, UC, and increasingly private/independent institutions, as well as labor market and employment data. Specifically, EdResults.org merged 152M new K-12 performance and demographic records from publicly available sources in to the CPP system. This data spans every district, school and grade level/subject over the past 12 years. In addition, the system of data now holds 571M total student-level records, up from 525M in 2013. These records are linked inter-segmentally and longitudinally. Managing a longitudinal system of data of this size and scope requires extensive work and significant staff time to perform a variety of ongoing functions, which include: uploading and processing data, verifying data quality, cleaning data, and system testing. In addition, the data team provides ongoing technical support to staff as well as CPP users. Technology As of the transition in 2012, immediate efforts began to host and manage the existing system. Goals and objectives were created to significantly expand its size, scope, and usability. The workload and time to create the expanded system of data and related tools was extensive and thus represented a significant portion of the budget in 2013-14. In 2013-2014, CPP s new user website, featuring the system of data and a variety of user tools, was successfully built out and launched. The new site allows practitioners direct access to actionable data. This increased usability now supports the daily work of educators while also holding the power to support longitudinal research. In addition to the reports and tools currently available on the website, CPP creates frequent special reports for users including: college enrollment trends, course taking and large unit attainment, college enrollment and success metrics, student athlete outcomes reports, CCC program review and accreditation reporting. In 2014, CPP has continued to develop actionable data reports for K-12 districts and dynamic reporting tools that are now available on www.calpassplus.org. Cal-PASS Plus Program and Financial Audit July 2013 June 2014 7

CPP has also undertaken a number of special projects, including developing the CTE LaunchBoard and partnering with the CCCCO Doing What Matters leadership to support 15 regional consortia focused on CTE program improvement. At the end of the 13-14 project year, the CTE portion of the project was expanding to the California Career Pathways Trust grantees. This effort has required significant investments in the areas of data, outreach, and technology. Cal-PASS Plus Program and Financial Audit July 2013 June 2014 8

Program Objectives, Outcomes, and Budget Details Background: Increasing Participation and Developing a Community of Practice through Communication and Outreach Per the FY2 Work Plan, specific Communication and Outreach goals included: Increased participation within the newly restructured Regional Learning Councils (RLCs) Increased outreach to K-12s, CSUs, and UCs to support data collection efforts Supporting research and analysis of best practices to guide decision-making through the Regional Learning Councils and other targeted conversations Collaborating with other CCC programs, under the direction of the CCCCO, to support program objectives Increased awareness about CPP through presentations at conferences, local meetings, statewide events, and within existing RLCs Responsible Parties: The team directs outreach and awareness through statewide events, summits, conferences and local meetings. ERP contributes to overall outreach and communication efforts, program-specific outreach and communication efforts, as well as outreach with private and non-profit research and academic institutions. SJDC focuses primarily on expanding the scope of participation in CPP within higher education segments, obtaining data and sharing agreements, and facilitating relationships and building RLCs. SJCOE focuses primarily on expanding the scope of K-12 participation in CPP through outreach, relationship building, and presentations to various groups across the spectrum of K-12. Allocations by percent and number of total budget Delta SJCOE EdResults K-12 MOUs & Data 30% N/A $60,497 $122,164 Post-Secondary 25% $87,408 N/A $91,623 MOUs & Data RLCs 20% $43,704 $30,249 $61,082 Communications 10% $14,568 $10,083 $30,540 Travel 7% $13,419 $13,042 25,307 Admin/Reporting 6% $9,564 $22,011 $32,512 Supplies 2% $295 $1,148 $2,244 TOTALS 100% $168,663 $137,030 $365,472 Cal-PASS Plus Program and Financial Audit July 2013 June 2014 9

Data and Technology Budget Allocations Admin/Report ing 6% Travel 7% Outreach Supplies 2% Comm. 10% K-12 MOUs & Data 30% Regional Learning Councils 20% Post- Secondary MOUs & Data 25% Summary of Communication and Outreach Outcomes 1) Increases in MOUs and total schools and data: 536 participating K-12 districts (52%, but representing 80% of all students) up from 489 MOUs in 2012-2013 112 participating Community Colleges (100%) represents all community colleges, but an increase of 5 renewed, expiring MOUs 18 participating CSU Universities (78%) up from 16 in 2012-13 8 participating UC Universities (80%) up from 7 in 2012-13 2) Increased the number of participating institutions that share data with all members to 25% for K-12 districts and 40% for colleges and universities 3) Increased the number of CPP users from 970 to 4,162 since 2013 4) Established 11 RLCs in each CISC region, comprised of volunteer administrators, faculty, researchers, and staff. 5) Convened RLCs in Riverside, Santa Cruz, Santa Barbara, San Diego, East Bay, Butte/Shasta. 6) Convened a statewide summit of approximately 70 volunteer administrators, faculty, researchers, and the business community to improve the education to workforce pipeline. 7) Developing 15 pilot Community Colleges and their feeder K-12 districts to improve placement through the use of multiple measures Cal-PASS Plus Program and Financial Audit July 2013 June 2014 10

Development and Management of the CPP User Platform, including: Data Collection, Analysis, and Reporting Tools Background: Per the FY2 Work Plan, specific Data and Technology goals included: Developing and launching www.calpassplus.org - a more user-friendly, functional website with resources, applications, features, and tools that support actionable data for users Increasing technical assistance and help desk support Developing and launching more comprehensive data analysis and reporting tools to support users Responsible Party: ERP provides data management, programming, design and user interface to support CPP. In addition, ERP is responsible for technical support and training. Allocations by percent and number of total budget: Ed Results Reporting Tools / Data 30% $111,997 Analysis Website Development 30% $111,997 Technology Infrastructure / Data Warehouse 30% $111,997 Help Desk 10% $37,332 Totals 100% $373,323 Cal-PASS Plus Program and Financial Audit July 2013 June 2014 11

Data and Technology Budget Allocations Web Development Tech Infrastructure / Data Warehouse 30% Help Desk 10% Website Development 30% Reporting Tools / Data Analysis 30% Summary of Data and Technology Outcomes 1) Successful build out and launch of new website, www.calpassplus.org. 2) Merged 152M new K-12 performance and demographic records from publicly available sources, that spans every district, school and grade level/subject over 12 years 3) Investigated best practices in 8 higher performing, high poverty districts and schools and provided a Success Factor framework with artifacts to facilitated replication of what works to raise student achievement. 4) 571M total student-level records (up from 525M), linked inter-segmentally and longitudinally. 5) 10 new, dynamic reporting tools linking data across educational segments. 6) Creation of innovative data tools on the CPP website including: Regional and county pipeline reports K-20 and on to the workplace that link 25 performance metrics dynamically, along with benchmarking tools and lists of higher performing schools, colleges and universities to increase sharing of best practices.. K-12 to college and university transition reports that provide institutions with data on the numbers of students transitioning to post-secondary, remediation reports and student completion of certificates, degrees and transfers. Comprehensive longitudinal data tools on every K-12 school/district that allow for research on academic subjects, grade levels, strands of the California Standards Test and subgroup performance over time. Benchmarking tools on all data charts and reports that identify higher performing bright spot schools, districts, colleges and universities and connect peers to learn effective practices greatly expanding the reach and impact of CPP actionable data beyond historically localized and query-based use of the data. New data dashboard concepts looking at special populations such as foster youth, financial aid recipients and student athletes, providing comprehensive data on student enrollment, support, course completion and success, degree/transfer attainment and ultimately workplace outcomes all of which increase CPP s user base and deployment of actionable data to improve student outcomes. Cal-PASS Plus Program and Financial Audit July 2013 June 2014 12

7) Created frequent special reports for members including: college enrollment trends, course taking and large unit attainment, college enrollment and success metrics, student athlete outcomes reports, CCC program review and accreditation reporting. 8) Created the CTE LaunchBoard and partnered with the CCCCO Doing What Matters leadership to support 15 regional consortia focused on CTE program improvement. Observation and Recommendations Following is a table detailing all the observations from the audit along with management s action plan to address these issues. Observation #1 Controls Recommendation Cal-PASS Plus Program and Financial Audit July 2013 June 2014 13