Smarter, Balanced Validation: Incorporating Systemic Objectives into a Validity Agenda 1. Stephen G. Sireci. University of Massachusetts Amherst

Transcription

1 Validating Theories of Action 1 Smarter, Balanced Validation: Incorporating Systemic Objectives into a Validity Agenda 1 Stephen G. Sireci University of Massachusetts Amherst 1 Center for Educational Assessment Research Report No Amherst, MA: Center for Educational Assessment, University of Massachusetts. Paper presented at the National Conference on Student Assessment, National Harbor, MD, June 20, 2013.

2 Validating Theories of Action 2 Abstract The Standards for Educational and Psychological Testing (AERA et al., 1999) provide important guidance for test development and for evaluating the use of a test for a particular purpose. However, modern assessment systems, such as the Smarter Balanced Assessment Consortium, have multiple purposes, many of which extend beyond the assessment to broader societal goals. In this paper we discuss how the validation framework provided in the AERA et al. Standards can be extended to incorporate the key ideas in a broad Theory of Action. This extension involves incorporating a program evaluation perspective. An illustration of the approach is provided by describing the comprehensive research agenda developed for the Smarter Balanced Assessment Consortium. Suggestions for future directions in validating assessment systems are provided.

3 Validating Theories of Action 3 Smarter, Balanced Validation: Incorporating Systemic Objectives into a Validity Agenda In educational and psychological testing, validity is often referred to as the most important aspect in determining the quality and appropriateness of a test. This importance can be seen in the definition of validity provided by the Standards for Educational and Psychological Testing (hereafter referred to as the Standards), which is an authoritative publication produced by a joint committee from the American Educational Research Association (AERA), the American Psychological Association (APA) and the National Council on Measurement in Education (NCME; AERA, APA, & NCME, 1999). The Standards define validity as the degree to which evidence and theory support the interpretations of test scores entailed by proposed uses of tests (p. 9). From this definition we can see that the proposed uses of tests must be clearly specified to (a) understand what must be validated, and (b) determine the types of evidence that must be gathered to evaluate the degree to which the test is fulfilling its purposes. In previous writings (e.g., Sireci, 2012, 2013a), I argued that proper validation of educational assessments can follow a three-step process that uses the AERA et al. (1999) Standards as a framework. The first stage is articulating the intended purposes of the testing program; the second stage is considering potential misuses or potential unintended consequences of the testing program; the third step is crossing these purposes and potential problems with the five sources of validity evidence promulgated by the Standards. In Sireci (2012), I used the Massachusetts Adult Proficiency Tests (MAPT) to provide an example of this approach to validation. The MAPT assesses the mathematics and reading knowledge and skills of adult basic education students in Massachusetts and is based on

4 Validating Theories of Action 4 curriculum frameworks for adult learners in Massachusetts. The Technical Manual for the MAPT lists its intended purposes as follows: The purposes of the MAPT are to measure ABE learners knowledge and skills in math and reading so that their progress in meeting educational goals can be evaluated. The MAPT is designed to measure learners educational gains for the purposes of state monitoring and accountability under the [Federal Government s National Reporting System]. Learners MAPT scores and score gains can be aggregated to provide meaningful summative measures of program effectiveness. (Sireci et al, 2008, p. 8). The MAPT is designed to meet Federal and state accountability demands and to evaluate students progress in meeting their academic goals. Potential misuses of MAPT scores are also provided in the Technical Manual. Specifically, MAPT assessments are not designed for diagnosing learners specific strengths and weaknesses within each subject area. Diagnostic evaluation should be based on more than MAPT scores MAPT assessments are also not designed for placing students into instructional programs. As described in the Assessment Policies and Procedures Manual ( students should be in instructional programs for at least two weeks before they are pre-tested with the MAPT. (Sireci et al., 2008, p. 8) Consideration of these testing purposes and potential misuses represents the first step in a traditional validation plan. Table 1 (from Sireci, 2012) illustrates how these purposes and potential misuses are used to create the validation framework. The rows represent (abbreviated) validity questions based on the MAPT purpose statements or potential misuses of MAPT scores. The columns represent the Standards five sources of validity evidence. The check marks ( ) indicate the types of evidence that should be gathered to support the use of the MAPT for the specific purpose listed or for evaluating a specific potential misuse. It is interesting to note that the potential misuse questions, and the general question regarding the MAPT s effect on

5 Validating Theories of Action 5 instruction, require validity evidence based on testing consequences, which is often overlooked in most validation endeavors. Table 1 Illustration of MAPT Validation Plan Validity Question Does the MAPT measure the correct skills? Are the tests congruent with the curriculum frameworks? Content Internal Structure Source of Validity Evidence Relations w/ Ext. Variables Response Processes Are the scores accurate? Do they adequately measure progress? Do they meet Federal requirements? Are they useful for program evaluation? Inappropriate diagnostic use? Inappropriate placement? Positive effect on instruction? Testing Consequences For a brief time, I was content with this framework for evaluating a particular test. However, this framework does not fully address the purposes of several new assessment systems, such as the consortia funded by the Federal Race-to-the-Top program (i.e., Dynamic Learning Maps, National Center and State Collaborative, Partnership for Assessing Readiness for College and Career, and the Smarter Balanced Assessment Consortium). Consider, for example the theory of action (TOA) for Smarter Balanced, which is presented in Figure 1. The TOA

6 Validating Theories of Action 6 specifies three components of the system summative assessments, interim assessments, and formative assessment tools and resources. Figure 1 Overview of SBAC Theory of Action Source: Balanced-Theory-of-Action.pdf A review of Figure 1 immediately reveals purposes that would be difficult to fit into the three-step validation plan I previously proposed. For example, it is hard to fit the goals of Policies and standards are communicated to schools, districts, and policymakers, or teachers are provided with curriculum and instructional materials and given rich professional development into one of the AERA et al. (1999) Standards five sources of validity evidence. These goals go beyond the assessments, even though the assessments are integral to achieving

7 Validating Theories of Action 7 the stated goals of the assessment system. Therefore, contemporary validation efforts must go beyond validating the use of a test for a particular purpose and extend to an evaluation of the degree to which the entire assessment system is meeting its intended purposes. In the remaining sections of this paper, I discuss how a TOA could be incorporated into a validation plan. Next, I illustrate how a comprehensive research agenda was proposed for the Smarter Balanced Assessment Consortium based on its TOA. I conclude with some thoughts about how we can best validate assessment systems that involve comprehensive TOAs. Incorporating a Theory of Action into Validation According to Bennett (2010), the concept of a TOA comes from the field of program evaluation and describes the intended goals of a program. Bennett notes that accountability testing is analogous to an educational program. As he put it, in educational accountability testing, change is intended, it seems appropriate to require a theory of action for such assessment programs, in addition to the more usual scientific evidence in support of instrument technical adequacy. (p. 71) Bennett s point is that accountability testing requires more than what we typically do to validate test score interpretations for individual students. He stated that a TOA for an assessment system might include the following elements: the intended effects of the assessment system the components of the assessment system and a logical and coherent rationale for each component the interpretive claims that will be made from assessment results the action mechanisms designed to cause the intended effects the potential unintended negative effects and what will be done to mitigate them (p. 71).

8 Validating Theories of Action 8 If accountability systems develop TOAs with these elements, they will facilitate validation. Bennett s third bullet has been the focus of traditional validation efforts. However, considering all these elements in the broader sense of program evaluation will provide more complete information for evaluating an entire assessment system. Such an evaluation is consistent with the notion of gathering validity evidence based on testing consequences, which is important in any endeavor involving validation of educational tests. How then can we extend a more traditional validation framework so that we can validate assessment systems that are based on a TOA? The answer lies in adding program evaluation processes to the assessment validation framework. As illustrated in the Smarter Balanced TOA (Figure 1), some purposes related to the assessment, while others relate to the system. I believe we can apply the validation framework based on the AERA et al. (1999) Standards to those aspects of the system that specify intended goals of the assessments, and program evaluation processes to those goals that refer to the system itself. To illustrate how these different goals are addressed in a comprehensive validation effort, I next describe the comprehensive validation plan proposed for Smarter Balanced. A Validity Research Agenda for the Smarter Balanced Assessment Consortium In June (2012), I was charged with helping Smarter Balanced develop a comprehensive research agenda. The goals of this agenda were to, inform Smarter Balanced of research that should be done to evaluate degree to which the Consortium is accomplishing its goals and to demonstrate that the assessment system adheres to professional and Federal guidelines for fair and high quality assessment. The intent is to provide a comprehensive and detailed research agenda for the Consortium that includes suggestions and guidance for both short- and long-term research activities that will support Consortium goals. (Sireci, 2013b, p. 5)

9 Validating Theories of Action 9 In beginning to develop the comprehensive research agenda I began work on a validation plan, attempting to use the three-step process I outlined earlier (Sireci, 2012, 2013a). This threestep plan is based on an argument-based approach to validity (Kane, 1992, 2006, 2013), that essentially describes validation as establishing a scientifically sound argument, supported by empirical evidence, that supports the use of a test for a particular purpose. The first step in developing the validation plan for Smarter Balanced was translating the essential goals of the TOA into explicit purpose statements that would be the focus of validation. This translation was accomplished via a consensus process that involved the leadership of the Consortium, its Technical Advisory Committee, and two work groups made up of state representatives (Test Administration and Design Work Group, Validation and Psychometrics Work Group). This process spanned about five months. The purpose statements that became the focus for the comprehensive research agenda were stratified by the three components of the assessment system. That is, separate purpose statements were articulated for the Summative, Interim, and Formative components. These statements (from Sireci, 2013b) are as follow: The purposes of the Smarter Balanced Summative assessments are to provide valid, reliable and fair information about, 1. students ELA and Mathematics achievement with respect to those CCSS measured by the ELA and Mathematics summative assessments 2. whether students prior to Grade 11 have demonstrated sufficient academic proficiency in ELA and mathematics to be on track for achieving college readiness 3. whether Grade 11 students have sufficient academic proficiency in ELA and Mathematics to be ready to take credit-bearing college courses 4. students annual progress toward college and career readiness in ELA and Mathematics 5. how instruction can be improved at the classroom, school, district, and state level 6. students ELA and Mathematics proficiencies for Federal accountability purposes and potentially for state and local accountability systems 7. students achievement in ELA and Mathematics that is equitable for all students and subgroups of students.

10 Validating Theories of Action 10 The purposes of the Smarter Balanced Interim assessments are to provide valid, reliable, and fair information about 1. student progress toward mastery of the skills measured in ELA and Mathematics by the summative assessment 2. students performance at the content cluster level so teachers and administrators can track student progress throughout the year and adjust instruction accordingly 3. individual and group (e.g., school, district) performance at the claim level in ELA and mathematics to determine whether teaching and learning are on target 4. student progress toward the mastery of skills measured in ELA and Mathematics across all students and subgroups of students The purposes of the Smarter Balanced Formative Assessment Resources are to provide measurement tools and resources to, 1. improve teaching and learning 2. monitor student progress throughout the school year 3. help teachers and other educators align instruction, curricula, and assessment 4. help teachers and other educators use the Summative and Interim assessments to improve instruction at the individual student and classroom levels 5. illustrate how teachers and other educators can use assessment data to engage students in monitoring their own learning Thus, 16 purposes were articulated across the three components of the Smarter Balanced Assessment Consortium. The seven purposes associated with the Summative Assessment system could be accommodated by a traditional argument-based validation framework, such as the three-step process I outlined earlier. The same could be said of the four purpose statements associated with the Interim Assessment system. Illustrations of how these purpose statements were crossed with the AERA et al. (1999) Standards five sources of validity evidence are presented in Tables 2 and 3 for the Summative, and Interim assessment systems, respectively. However, the purposes of the Formative Assessment Resources are more systemic, and so a traditional validity framework is insufficient. Only the second purpose listed, monitor student progress throughout the school year is easily fit into a traditional validation paradigm. The other four purposes require confirming that the intended products were produced, disseminated appropriately, were used and valued by educators and students, and had a positive effect on student learning. The studies that were recommended to evaluate these purposes

11 Validating Theories of Action 11 The purposes of the Smarter Balanced Summative assessments are to provide valid, reliable and fair information about, 1. students ELA and Mathematics achievement with respect to those CCSS measured by the ELA and Mathematics summative assessments 2. whether students prior to Grade 11 have demonstrated sufficient academic proficiency in ELA and mathematics to be on track for achieving college readiness 3. whether Grade 11 students have sufficient academic proficiency in ELA and Mathematics to be ready to take creditbearing college courses 4. students annual progress toward college and career readiness in ELA and Mathematics 5. how instruction can be improved at the classroom, school, district, and state level 6. students ELA and Mathematics proficiencies for Federal accountability purposes and potentially for state and local accountability systems 7. students achievement in ELA and Mathematics that is equitable for all students and subgroups of students. Table 2 Validity Framework for Smarter Balanced Summative Assessments Source of Validity Evidence Internal Relations w/ Response Content Structure Ext. Variables Processes Testing Consequences

12 Validating Theories of Action 12 Table 3 Validity Framework for Smarter Balanced Interim Assessments The purposes of the Smarter Balanced Interim assessments are to provide valid, reliable and fair information about, 1. student progress toward mastery of the skills measured in ELA and Mathematics by the summative assessment 2. students performance at the content cluster level so teachers and administrators can track student progress throughout the year and adjust instruction accordingly 3. individual and group (e.g., school, district) performance at the claim level in ELA and mathematics to determine whether teaching and learning are on target 4. student progress toward the mastery of skills measured in ELA and Mathematics across all subgroups of students Content Internal Structure Source of Validity Evidence Relations w/ Ext. Variables Response Processes Testing Consequences

13 Validating Theories of Action 13 included (a) confirming the development and successful implementation of all planned Formative Assessment Resources, (b) evaluating usage statistics of all tools and other resources, (c) review of all documents supporting the system, (d) comprehensive surveys of the collaborative leadership involved in overseeing the products, and processes, (e) comprehensive surveys of users of the resources (teachers, administrators, students, parents), and (f) case studies of teachers and administrators who are frequent users of the resources (Sireci, 2013b, pp ). Clearly, these activities are more reminiscent of program evaluation activities, than test validation activities. Summary of Smarter Balanced Illustration The comprehensive research agenda developed for the Smarter Balanced Assessment Consortium illustrates how a more traditional, argument-based validation plan needs to be supplemented with additional program evaluation studies to fully evaluate an assessment system. Assessment systems specify TOAs that are designed to affect change not only for individual students, but also for groups of individuals (e.g., teachers) and organizations (e.g., schools, districts). The laudable purposes contained in these TOAs involve more than reliable and valid assessments. They require us to validate more than interpretations and uses of test scores they require us to validate system implementation and impact. The Smarter Balanced Comprehensive Research agenda illustrates this integration of validation and program evaluation research. The validation paradigm involved first articulating system goals and purposes in a way that supports validation and evaluation. It then identified the aspects of the system that pertained to test use and interpretation, and those aspects that pertained to program implementation and impact. Next, studies were identified that would provide evidence regarding test use and interpretation using the AERA et al. (1999) Standards five

14 Validating Theories of Action 14 sources of validity evidence. Other studies were identified to evaluate program implementation and impact. In total, 55 research activities were included in the recommended research agenda, many of which were part of standard test development and validation practices, but some of which were not. A description of these studies is beyond the scope of this paper and so interested readers are referred to Sireci (2013b; see also Martineau, Sireci, McCall, Gallagher, & Mitchell, 2013). Recommendations for Validating 21 st -Century Educational Assessment Systems The Smarter Balanced example described in the previous section illustrates the approach I recommend for validating a comprehensive assessment system that has systemic goals associated with broad educational policy. I put validating in quotes because I sometimes want to use the term evaluating, because some of the goals we are studying go way beyond the assessments themselves and bring us into the realm of program evaluation. This blending of validation and evaluation is consistent with Cronbach s (1988) views where he characterized validation as similar to program evaluation. The recommended steps for validating an assessment system are, (a) Isolate the components of the TOA that should be translated into purpose statements for validation. (b) Determine which purposes relate to test score use and interpretation that fit into a traditional test validation paradigm, and which should be validated using a program evaluation approach. (c) Cross the testing purposes related to test score use and interpretation, and validity questions related to potential test score misuse, with the AERA et al. (1999) Standards

15 Validating Theories of Action 15 five sources of validity evidence. This process will indicate the areas where studies need to be conducted. (d) Design studies to evaluate the purposes that relate to the efficiency, fairness, quality, and success of the system. These studies are those more commonly found in program evaluation, such as audits of procedures, observations, interviews, focus groups, and surveys. (e) Prioritize all studies based on their importance with respect to how much information they will contribute to the validity argument and the relative importance of the purpose(s) they will support. The first, third, and fifth steps are consistent with traditional validation practices. The second and fourth steps illustrate how contemporary validation of educational assessment systems extends the traditional paradigm to ensure the goals of an entire system, as articulated in the TOA, are addressed. As more assessment programs postulate comprehensive theories of action designed to affect systemic change, more validation efforts will require extensions beyond traditional validation efforts. The approach recommended here is one strategy, and Bennet (2010) describes other ideas. Hopefully, over the next few years we will see a great deal of activity in gathering and reporting validity evidence for these systems, and our validation paradigms will evolve accordingly. For educational assessment systems to meet their laudable goals, they will need to be studied and validated. In addition to validating the systems goals are being met, a comprehensive validation plan should provide formative data during the life of the system, so that the validation will facilitate meeting those goals.

16 Validating Theories of Action 16 References American Educational Research Association, American Psychological Association, & National Council on Measurement in Education. (1999). Standards for educational and psychological testing. Washington, DC: American Educational Research Association. Bennett, R. E. (2010). Cognitively based assessment of, for, and as learning (CBAL): A preliminary theory of action for summative and formative assessment. Measurement, 8(2 3), Cronbach, L. J. (1988). Five perspectives on the validity argument. In H. Wainer & H.I. Braun (Eds.), Test validity (pp. 3-17). Hillsdale, New Jersey: Lawrence Erlbaum. Kane, M.T. (1992). An argument-based approach to validity. Psychological Bulletin, 112, Kane, M. (2006). Validation. In R. L. Brennan (Ed). Educational measurement (4 th edition, pp ). Washington, DC: American Council on Education/Praeger. Kane, M. (2013). Validating the interpretations and uses of test scores. Journal of Educational Measurement, 50, Martineau, J. A., Sireci, S. G., McCall, M., Gallagher, C., & Mitchell, C. (2013, April). The current state of the Smarter Balanced research agenda. Paper presented at the annual meeting of the National Council on Measurement in Education, San Francisco. Sireci, S. G. (2012, April). Deconstructing test validation. Paper presented at the annual conference of the National Council on Measurement in Education as part of the symposium Beyond Consensus: The Changing Face of Validity, (P. Newton, Chair), Vancouver. Sireci, S. G. (2013a). Agreeing on validity arguments. Journal of Educational Measurement, 50, Sireci, S. G. (2013b). Smarter Balanced Assessment Consortium comprehensive research agenda. Unpublished manuscript submitted to the Smarter Balanced Assessment Consortium. Sireci, S. G., Baldwin, P., Martone, A., Zenisky, A. L., Kaira, L., Lam, W., Shea, C. L., Han, K., Deng, N., Delton, J., & Hambleton, R. K. (2008). Massachusetts adult proficiency tests technical manual: Version 2: Amherst, MA: Center for Educational Assessment. Available at