The Importance of a Long-itudinal Data System

Transcription

1 Michigan s P-20 Initiative Executive Summary MCCA s P-20 Taskforce March 2010 (Draft 2) P-20 Taskforce: Nick Baker, Director of Institutional Research, Kirtland Community College Dr. Conway A. Jeffress, President, Schoolcraft College Dr. Reginald Gerlica, Vice President of Academic Affairs Arts and Sciences, Henry Ford Community College Mike Hansen, President, Michigan Community College Association Dr. Randall Hickman, Director of Institutional Research, Macomb Community College Dr. Jim Jacobs, President, Macomb Community College Randall Melton, Executive Directory Information Technology and Institutional Research, Lake Michigan College Dr. Adriana Nichols, Vice President for Public Policy, Michigan Community College Association Dr. Dan Phelan, President, Jackson Community College Howard Shanken, Registrar, Grand Rapids Community College Rob Stirton, Director of Institutional Research, Schoolcraft College Other Contributors: Eileen Brennan, Institutional Research Analyst, Oakland Community College Kristin Buttegig, Director of Institutional Research, Jackson Community College Heath Chelsvig, Institutional Research Analyst, Grand Rapids Community College Gail Ives, Executive Director for Institutional Research, Mott Community College Leslie Kellogg, Coordinator of Career and Technical Education, Henry Ford Community College Doris Lewis, Director of Institutional Research, Kellogg Community College Jeff Noel, Sr. Data Analyst, Lake Michigan College Nancy Showers, Director of Institutional Research, Oakland Community College Mike Wood, Director of Institutional Research, Delta College

2 Background There is a growing trend nationally toward linkage of P-12 and postsecondary data systems. As of 2007, a Data Quality Campaign survey indicated that 22 states had the ability to link student records in the P-12 system to records in the postsecondary system (although according to a NCHEMS survey in 2006, only 11 states had linked postsecondary student records with high school student records). Another 26 states have plans to implement P-20 data systems. While there are doubtless many reasons behind this growing effort to construct longitudinal data systems, the effort to link P-12 and postsecondary data systems originated primarily out of concern to better align elementary, secondary, and postsecondary systems in order to increase the proportion of college ready students entering the postsecondary systems. How high school students become (or do not become) ready for college level work can only be answered by linking information about the curricular pathway and performance of the high school student with information about the pathway and performance of that student in postsecondary institutions. During 2009, Michigan was a recipient of approximately $1.5 billion as part of the State Fiscal Stabilization Fund (SFSF); a new one-time appropriation of $53.6 billion under the American Recovery and Reinvestment Act of 2009 (ARRA), of which the U.S. Department of Education (ED) will award states with approximately $48.6 billion by formula. The federal awards are in exchange for a commitment from governors to advance essential education reforms to benefit students from early learning through post-secondary education, including: college and careerready standards; valid and reliable assessments for all students; development and use of pre-k through postsecondary and career data systems to track student progress and foster continuous improvement(p-20); increasing teacher effectiveness and ensuring an equitable distribution of qualified teachers; and turning around the lowest-performing schools. ARRA, H. R specifically states the following: The State will establish a longitudinal data system (LDS) that includes the elements described in section 6401(e)(2)(D) of the America COMPETES Act (20 U.S.C. 9871). This system must support the following baseline functionality: 1. Provide a unique statewide student identifier (UIC) that does not permit a student to be individually identified by users of the system. 2. Support Student-level enrollment, demographic, and program participation information. 3. Provide Student-level information about the points at which students exit, transfer in, transfer out, drop out, or complete P 20 education programs. 4. Include the capacity to communicate with higher education data systems. 5. Provide a state data audit system assessing data quality, validity, and reliability. The longitudinal data system includes creating a collaborative framework between early learning, K-12 and postsecondary providers to improve student success through the sharing of data and the alignment of policies. 1 Preschool K-12 P-16 K-16 P-20 College Level Graduate Level These efforts are commonly named K-16, P-16 or P-20 systems with a goal of supporting a seamless system of education. One of the many goals of P-20 is to increase student success and to develop a more competitive workforce. The Center for Educational Performance and Information, (CEPI) has been designated as the custodian for implementing P-20 in Michigan. This is a natural extension of their role as the state s data collector and reporting agency for the K-12 public school system. As part of receiving ARRA funds, the state has committed to demonstrating the capacity of a P-20 system by September of 2011 to ED. This capacity will target baseline functionality of a longitudinal data system and could be considered a pilot or proof of concept. A potential secondary project that is contingent upon additional grant funding will target a fully operational framework for P- 20.

3 Value of a Longitudinal Data System The linking of elementary, secondary, and postsecondary data systems in a single, comprehensive longitudinal data system (LDS) provides a powerful tool for policymakers and educators for enhancing educational outcomes. A comprehensive longitudinal data system enables addressing questions about institutional performance, the performance of the educational system as a whole, and the impact of policy at the state level that cannot otherwise be addressed in an effective way. The value of such a data system emerges clearly when it is recognized that educational systems are best understood in terms of the value-added model. In fact, fully reaping the benefits of a comprehensive longitudinal data system requires recognizing the value-added nature of educational systems and designing the data system accordingly. The value-added model recognizes that our educational system should be viewed as follows: The educational system is a pipeline with an intended dominant direction of flow. It contains various types of educational institutions (and multiple institutions within each type) that provide different kinds of experiences for students, i.e. add value to students in different ways and to different degrees. Students flow through this pipeline (though not necessarily in a single direction), exiting at various points (for different reasons) and after different levels of exposure to different kinds of institutions. As students flow through this pipeline, they are exposed to institutions of different types those concerned with primary and secondary education and the postsecondary institutions, the 2-year and 4-year colleges and universities. In the course of their exposure to these institutions they generate various kinds of data, including data concerning attendance patterns, curricular pathways, academic performance, and other outcomes such as transfer and degree completion. There are three overarching questions each institution within the pipeline should ask: 1. How prepared are our learners and at what level? 2. What value are we adding? 3. Where and how are learners progressing with the value we added? The key value of a comprehensive longitudinal data system lies in this fundamental insight: Learners Educational Preparedness: How ready are our learners? Learners Transformation: What value did we add? Learner Progression: Where did our learners go? If an institution has no upstream and downstream measurements on the students that flow through it, it is impossible for that institution to fully understand and manage its performance. In the absence of upstream and downstream measurements, an institution cannot fully understand how its performance (the value it adds) affects the ability of institutions downstream to perform and, therefore, is unable to take systematic, well-informed measures to increase the value that it adds and to thereby assist institutions downstream to improve their performance. It also cannot fully understand how its performance is affected by the performance of institutions upstream (the value they add) and, therefore, is unable to assist upstream institutions with their performance. 2

4 Business Case for P-20 As Michigan continues to focus on meeting the challenges in today s economy, we as educators need to reinvent the way we work together across the different service silos on improving student success and in workforce development so we can effectively compete globally. A robust P-20 framework can assist in optimizing the performance of the state as a whole as we remove barriers and constructively share data. Notwithstanding, the ARRA mandate for a P-20 system, the business case for P-20 can be made on the following grounds: Improve Learner Outcomes. We need to improve student success and develop a highly skilled workforce to meet the emerging challenges in a global and dynamic economy. The linking of elementary, secondary and postsecondary data systems in a single, comprehensive longitudinal data system (LDS) provides a powerful tool for policymakers and educators for enhancing educational outcomes. Collaboration is needed across the Educational Silos to support a continuous improvement model. An effective engagement model must be developed and implemented that supports conversations between the different levels of education concerning student outcomes. Data must be timely shared and acted upon in collaborative spaces that support an instructor to instructor (i2i) dialog and policy maker to policy maker (p2p) dialog. Persistent conversations about student outcome trends and observations must become part of the culture. Social web tools and data portals should be converged to support timedistance discussions between the i2i and p2p stakeholders. Educators must adopt the same social tools that students are using to connect and exchanging ideas between the silos of service. A new analytical system is needed for measuring effectiveness and making data supported decisions. A state level comprehensive data warehouse needs to be developed and implemented that supports a longitudinal data system and business intelligence (BI) services through a P-20 web portal. This should include access to trustworthy data that maximizes insights and supports decisions on the most recent data. To reach this objective, common data standards will need to be developed and adopted. Business intelligence services should be provided through a web portal and include: predefined reports, ad-hoc reporting tools, role-based security, dashboards and benchmark analysis. User training and a dedicated helpdesk should be available for supporting BI services. In addition to BI services, social web tools should be integrated into the P-20 portal for supporting stakeholder collaboration. We need to communicate on how we are doing and provide a framework for accountability. Part of the P-20 system should include a framework for communicating between stakeholders and the public. An annual P-20 report should be produced that outlines observed trends and emerging needs. Improve the process for assessing and defining public policies. The ability to track student outcomes across the different levels of education will provide policy makers insight into student outcomes and institutional performance. This will help substantiate investments and clarify needs for improving learner outcomes. P-20 will provide information to educators and the public to address the individual needs of students and improve teacher performance. Agility is needed in our educational systems to meet new demands. The evolution of learning using digital tools and spaces, and our changing economy is going to require a robust process for managing change. Learners are acquiring new skills through multiple dimensions; virtual communities, online encyclopedias, blogs, e-books, participatory learning. Traditional institutions must be open to adopting and blending new modalities to engage the digital generation and meet new needs in the economy. As learning becomes more of a continuous process and re-skilling becomes standard the role of traditional institutions must evolve. A P-20 system will provide multiple snapshots within an academic year for tracking student enrollment activity and their outcomes. 3

5 While the recent ARRA is a driver for P-20, it is important that we make P-20 a true performance management system for improving student success and collaboration between stakeholders and not another mandate. Concerns to keep in Perspective Reporting for the sake of reporting with no engagement model does not support change. Data can becomes meaningless and have its value lost if it is not timely analyzed and shared with change agents. The value of data is degraded unless it can be acted upon in a collaborative environment to improve outcomes. Preparing students for assessment tests can be detrimental to student success; the long-term perspective must be kept in view. Using data primarily as a punitive tool rather than a feedback mechanism for real change does not promote trust or build vital relationships. This does not preclude accountability; rather it puts the focus on taking ownership of problems and fixing them at the source. Change occurs most effectively when instructors and stakeholders adopt and value feedback mechanisms as a core component of the educational delivery process. Operating without comparative data for key measures can impair an organization. It is critical that the LDS capture key measures that empower decision makers. Objectives of the proposed LDS In order that state policy makers and the elementary, secondary and postsecondary institutions derive full value from the proposed LDS (with features as reflected in Functional Considerations for Michigan s P-20 Initiative ), the LDS should have the following key objectives: Support compliance with mandated state and federal reporting requirements. Support the management of institutional performance by enabling research concerning curricular pathways associated with success. Support an engagement model and collaboration between stakeholders. Support the formation of policy statewide by enabling research focused on evaluating the impact of current policy and identifying changes in policy that could positively impact student outcomes. The Mission of P-20 The mission of P-20 is to improve student success and develop a competitive workforce through the sharing of data and collaboration between service providers, answering the questions of How are we doing?, What should we be doing? and Why?. Key to the success of P-20 is building an engagement model that gets data into the right context for conversations between individuals who can become change agents for improving student success. Data is the connecting point; the real value comes from collaboration between providers within the educational supply chain. K12 Teachers IR (Data Coaches) Administrators Student Success Policy Makers College Faculty 4

6 Recommendations The recommendations of the MCCA task force are made in light of the key objectives identified above and the suggested mission of the LDS. Guiding Principles to Consider: An emerging state level postsecondary data system should: Protect privacy - Student privacy controls should be integrated into the P-20 system to effectively safeguard personal identifying information and foster public trust. This should include the prohibit disclosure of personally identifiable information except as permitted under section 444 of the General Education Provisions Act and any additional limitations set forth in State law. Include a provision for data breaches - keep an accurate accounting of the date, nature, and purpose of each disclosure of personally identifiable information in the statewide P 20 education data system, a description of the information disclosed, and the name and address of the person, agency, institution, or entity to whom the disclosure is made, which accounting shall be made available on request to parents of any student whose information has been disclosed. Link to K-12 using UICs and a centralized data warehouse. Link to labor and workforce development (utilizing wage record matching). Contain metadata including standard definitions. Be flexible enough to expand in the future. As the system evolves it should: Have the capacity to audit data - State data audit system assessing data quality, validity, and reliability. Be formally aligned with state goals - We need to define state based goals as we baseline performance measures within the P-20 system. Incorporate standards for common data elements. Demonstrate usability and sustainability. Measure non-credit activity. A fully realized state data system should: Include support for independent and for-profit institutions. Incorporate relevant policy information. Demonstrate interoperability across multiple sectors. Use Effective Project Management Methodologies An agile project management methodology should be employed in implementing and maintaining the P-20 system; considerations should include: Ensure close cooperation between all stakeholders and implementers. Use self-organizing teams with appropriate representation and solicit feedback from stakeholders before finalizing requirements. Use a phased and prioritized approach for managing deliverables. Use effective rules of engagement and guiding principles for decision making. Use a pilot approach for validating solutions before widespread deployment. Streamline by rapid, iterative and continuous delivery of solutions. Publish project plans and develop communication plans to manage awareness. Address data integration issues across the numerous diverse student information systems. 5

7 Governance Considerations We need appropriate representation and clearly defined objectives that aligns all stakeholders and provides an effective structure for success. State Level Objectives Policies Governance Councils & State Representation The following policies should be considered in the P-20 governance framework: Privacy Policy Data Usage Policy Data Research Policy Data Mining Policy Reporting Requirements Policy All Stakeholders Outlines how the privacy of student data will be maintained in the LDS and in extracted forms. Defines the custodian role and use rights for all stakeholders. Defines the custodian role for researchers. State Level - Provide guidance in how data can be used at the state level as an accountability tool. Reporting Requirements for state-based institutions. Data Governance We need an effective Data Governance policy that promotes trust and defines the responsibilities of each P-20 participant Provide support in an Executive Order for authorizing CEPI as the custodian for collecting and hosting student data in the LDS. Specify in the Executive Order that Student Privacy will be managed using guidelines from the American Competes Act, H.R Specify in the Executive order the responsibility each public institution has for supporting P-20. 6

8 Data Standards and Common measures To enable comparison across the LDS, common definitions will need to be developed. Data standards and business rules will need to be defined to support electronic exchange of information. The LDS should support a meta-data layer that enables each institution to map and transform their data as needed into uniformed data elements. In some cases it may be necessary that the institution harmonize their definitions within their student information with the P-20 system. This will enable the LDS to be used to satisfy selected state and federal reporting requirements. Measures and Dimensions to consider in the P-20 LDS Learner Preparedness Metrics / KPIs College Readiness o Placement o Standardized testing Awards o Degrees o Certificates Credits Completed Advanced Placement Learner Value Added Metrics / KPIs Student Goal Attainment Student Persistence Licensure/Certification Pass Rates Student Satisfaction & Engagement Rates Performance in Transfer Institutions mental Course Success College-level Course Success mental to College-Level Course Success Credits Completed Non-Credit Activity Core Competencies Grade Distribution Diversity Rate Learner Progression Metrics / KPIs Other school attended Credits attempted Non-Credit Activity What additional awards were obtained Degrees Certificates Employment Workforce ment (New skills) Dimensions Student Demographics Cohort Financial Aid Year Term Course Delivery Schools Instructor Award Dimensions Student Demographics Cohort Financial Aid Course Delivery Year Term Major Schools Instructor Award Dimensions Student Demographics Cohort Financial Aid Year Award Schools Industry 7

9 Research Methodologies around the Key Measures A LDS can provide insight to measures of student success, such as persistence and degree completion. This research is useful for identifying the levers available to us that could be used for improving outcomes for students. With an LDS (with an appropriate design) we can take advantage of research that has identified predictors (or leading indicators ) of desirable student outcomes. For example, both number of completed credits and GPA in the first semester has been shown to be a useful predictor of persistence. An LDS would permit institutions to monitor their performance on key predictors and changes in their performance as they attempt to improve outcomes for students by working with feeder high schools to increase the preparedness of incoming students. What is the first semester mean GPA at XYZ college/university of students from my college that have transferred to XYZ college/university? What is the first semester mean number of completed credits at XYZ college/university of students from my college that have transferred to XYZ college/university? What proportion of students from my college transferring to XYZ college/university receives a degree from XYZ within 2 years of transfer? Within 3 years of transfer? Within 4 years of transfer? Does the degree completion rate (per unit time) at XYZ college/university of students from my college transferring to XYZ college/university vary significantly by program at XYZ college/university? Does it vary significantly by program at my college? Does the fall-to-winter persistence of students in an incoming cohort at my college vary by feeder high school? Does the fall-to-fall persistence of students in an incoming cohort at my college vary by feeder high school? Does the degree completion rate (per unit time) at my college vary significantly by feeder high school? Does mean number of completed credits in the first semester at my college vary by feeder high school? Does mean GPA of an incoming cohort in the first semester at my college vary by feeder high school? Is variation in curricular pathway at feeder high schools (e.g., number of math courses, performance in math courses) associated with variation in degree completion rate (per unit time) at my college? Is variation in curricular pathway at feeder high schools associated with variation in first-semester performance at my college (completed credits and GPA)? Is variation in curricular pathway at feeder high schools associated with variation in performance in gateway courses at my college? Is having some college credits prior to graduation from high school an important advantage with respect to degree completion at my college? With respect to first semester academic performance (completed credits and GPA)? State Level LDS Metrics / KPIs High School Graduation Rates College-Going Rates Closing Race / Ethnic Gaps in College participation and completion Increased production by field What additional awards were obtained Retention Rates Awards earned Affordability KPI related to transfer rates (per period of time) of students seeking to transfer Mean (or median) time to degree completion. Dimensions Student Demographics Cohort Financial Aid Year Award Schools Industry Course Delivery 8

10 Operational Considerations There are several operational aspects for implementing and maintaining a P-20 system that should be formalized and developed with appropriate user group representation using a formalized requirements elicitation process. Integration E-Transcripts Support Define Process Data Standards UIC Codes Deployment Hosting Support Business Rules Data Quality Data Management Major Components a Data Quality plan should include support for data standards, documented business rules and an audit process. A P-20 support center should be developed that provides: technical support, analytic support and training on the P-20 reporting tools. P-20 data warehousing hosting services should reside on state managed servers. Data should be uploaded into the data warehouse using extractions scripts rather than from transcripts. ETL scripts should be developed for each major student information system and shared across the state. The UIC business rules should be developed by the MACRAO user community. The following technical solutions should be considered: Possible options for Integrating UICs Batch SIS Self-Service Michigan College Access Network SIS Real-Time SRSD SRSD 9

11 The P-20 system should provide four technical services for all service providers in the state. LDS Operational Data Store Business Intellengence Tool box Social Collaboration Framework CEPI s Summarized High Level Plan for P-20 (Draft; subject to change) Phase I Phase II Phase III - Year 2011 Phase IV Piloted UIC matching processes Selected E- Transcript Service Provider docufide Registered all schools for E- Transcripts with Docufide Recieved ARRA funds and made commitments to ED for P20 E- Transcripts (receiving and sending) P20 Council Baseline data collection process for a pilot P20 LDS with Docufide. Solicit functional requirements from stakeholders for an operational P20 system. RFP for the operational P20 System. Jan 2011; have all schools provide an e-transcript extraction to Docufide to populate the Pilot P20 LDS. UICs at the postsecondary level. Have Docufide provide baseline P20 reports from LDS. September 2011; demonstrate the capacity of the P20 LDS to ED. TBD - Operational P20 System 10

12 Proposed High Level ation Plan Policies Components Stakeholders Category Phases Phase I Phase II Phase III Privacy Policy All Governance Operational Operational Data Usage Policy All Governance Operational Data Research Policy All Governance Operational Data Mining Policy All Governance Operational Reporting Requirements Policy All Governance Operational Electronic Transcript Systems Inbound (receive) electronic transcript processing Outbound (send) electronic transcript processing Post-Secondary Operational Operational Operational K-12 Post-Secondary Registrars Post-Secondary Registrars Baseline Docufide transcript parsing services Provide test extraction. UIC / student identity Post-Secondary management system for postsecondary institutions Registrars /IT Longitudinal Data System (LDS) / Data Analysis System (DAS) LDS extract, transform, load processes IR / IT Operational Operational Operational Operational Technical Operational Technical Technical Student Assessment Data IR Operational Term Enrollment Activity IR Operational Awards Conferred IR Operational NA NA Operational Common Definitions IR Operational Term Outcome Data IR Operational PKI & Benchmarking Metrics Various Operational Error & Audit Reports IR Operational Reporting Portal Other Engagement Framework Various Faculty / IR/ K-12 Operational Technical Operational Support / Helpdesk IR/Registrars/IT Operational Training Various Operational 11

13 a Formal Performance Evaluation Plan for P-20 To maintain the effectiveness of the P-20 system, a formalized performance evaluation plan should be considered that includes the following: 1. Periodic evaluation of the performance of the LDS (and making modifications to the LDS based on the evaluation) is essential to deriving full value from the LDS. (By the way, if we re not prepared to make modifications, there is no point in conducting a performance evaluation.) The departure point for any assessment of the performance of the LDS is obviously the key objectives or goals of the LDS, since they define the performance respects that matter. The key objectives or goals help define the expected (desired or intended) performance, and clearly any performance assessment that is intended to support continuous improvement must in some way compare actual performance with expected performance. (During the development phase, we are really engaging in a prediction of how a LDS with certain features will perform. Performance evaluation, after a period of use, enables us to test those predictions.) That comparison should be made with respect to a number of dimensions, where the dimensions would be those dimensions or aspects of the system relevant or instrumental for achieving those goals or for determining the level of performance on the objectives or goals. Hence there could be a number of gaps between expected and actual performance on the various dimensions. (Which gaps to target for remediation efforts is not a straightforward matter, since that involves feasibility and cost/benefit considerations.) The goals may change over time, but at a given point in time there should be a common understanding concerning the objectives or goals of the LDS. 2. The performance of the LDS needs to be viewed comprehensively. Although the primary goals of the LDS are to provide information that supports compliance with mandated state and federal reporting, information supporting the management of institutional performance, and information informing policy making at the state level, other objectives are instrumental to accomplishing these goals. Hence the evaluation of the performance must also include technical aspects, such as the uploading and downloading of data, the merging of data from more than one institution to support reporting, and performance of the reporting portal. (We don t know yet, of course, exactly what some of these processes will look like because the LDS is still being developed.) It should also be noted that the performance cannot be properly assessed without taking account of the policy and procedures context in which the LDS operates, since they will play a role in determining how the LDS can be used. Thus we should also think about that context as an aspect (or aspects) of the current design of the LDS that is subject to modification in order to improve performance. What is being implemented, in effect, is not simply a database within a certain information technology infrastructure; we are also implementing a policy and procedure context that will affect the performance of the system. Hence we must also evaluate the performance of the policy and procedure context, since it is inextricably connected with the performance of the LDS. If there are aspects of the policy and procedure context that assessment results suggest are limiting the performance of the LDS, we need to be aware of that. 3. The design of a LDS always reflects assumptions about information needs that will be supported by the LDS (i.e. expected usage of the LDS). To the extent that the information needs change, the LDS needs to change (to the extent possible) to ensure a reasonable fit/match between the performance of the LDS (on the information dimension) and the needs. Information needs may also need to be prioritized (with categories such as necessary, desired, and supplemental ) so that improvement efforts can also be prioritized. So there needs to be a periodic information audit to determine how the information needs have changed. (The usual kind of change is an increase in the kinds of information desired, but it is not impossible that certain kinds of information that were desired in the past are no longer needed. I am not sure that that kind of change would lead to a change in the system, since it would imply that some of the data loaded into the LDS would no longer be.) Information on how information provided by the LDS is actually being used, i.e. the decisions that the information supports, should also be a component of the audit. 4. How changes are made to the LDS as a result of the evaluation can be a long story in itself, and I don t know how much space you want to devote to that. It is very common to find a version of Deming s PDSA cycle as a component in continuous improvement models, so you may want to make mention of that. The key idea in incorporating the PDSA cycle (or a version of it) in the process for evaluating system performance and making changes is that change needs to be evidence driven, i.e., based on relevant data allowing us to predict that the anticipated changes will improve system performance in the anticipated 12

14 way. In fact, including a section on periodic evaluation of the performance of the LDS in the Functional Considerations document could be regarded as our recommendation that the PDSA cycle needs to be implemented on a large scale: we are currently in the planning phase of the LDS (the plan phase of the Deming cycle), which will be followed by implementation (the do phase), which will in turn be followed by evaluation of the performance (the study phase) and system modifications based on the results of that evaluation (the act phase). 5. You may want to consider giving a role to benchmarking as an additional source of information concerning what level of performance (and in what dimensions) should be expected from the LDS. The periodic evaluation process is always an opportunity to revisit design issues; in fact, I would take a stronger position, namely that we need the kind of evaluation process that systematically includes revisiting the design issues that resulted in the original specifications. The fact that the design questions were answered in a certain way during the development phase is not in itself a reason for not making changes to the LDS. The past is a sunk cost, and the value of the LDS in its current form always needs to be determined from the standpoint of its present value : given what we now know concerning what we require from the LDS, what changes need to be made to reduce the gap between its current performance and its desired performance. 6. Performance evaluation should not be conducted too soon after implementation and initial use. There will be an initial period of time when users (at institutions and at the State) are still learning how to interact with the LDS in ways that add value to their decision processes. In other words, during this initial period when users are still moving down the learning curve, so to speak, we wont have enough information to estimate what performance levels can be expected from the system in its current design and, hence, will not be able to compare that with the desired performance. 13