Works Cited Index

Transcription

1 Contents Part 1 Nature and Purposes of The Iowa Tests The Iowa Tests Major Purposes of the ITBS Batteries Validity of the Tests Description of the ITBS Batteries Names of the Tests Description of the Test Batteries Nature of the Batteries Nature of the Levels Grade Levels and Test Levels Test Lengths and Times Nature of the Questions Mode of Responding Directions Other Iowa Tests Iowa Writing Assessment Listening Assessment for ITBS Constructed-Response Supplement to The Iowa Tests Other Manuals Part 2 The National Standardization Program.. 7 Planning the National Standardization Program Procedures for Selecting the Standardization Sample Public School Sample Catholic School Sample Private Non-Catholic School Sample... 8 Summary Design for Collecting the Standardization Data Weighting the Samples Racial-Ethnic Representation Participation of Students in Special Groups Empirical Norms Dates School Systems Included in the 2000 Standardization Samples New England and Mideast Southeast Great Lakes and Plains West and Far West Part 3 Validity in the Development and Use of The Iowa Tests Validity in Test Use Criteria for Evaluating Achievement Tests Validity of the Tests Statistical Data to Be Considered Validity of the Tests in the Local School.. 26 Domain Specifications Content Standards and Development Procedures Curriculum Review Preliminary Item Tryout National Item Tryout Fairness Review Development of Individual Tests Critical Thinking Skills Other Validity Considerations Norms Versus Standards Using Tests to Improve Instruction Using Tests to Evaluate Instruction Local Modification of Test Content Predictive Validity Readability Part 4 Scaling, Norming, and Equating The Iowa Tests Frames of Reference for Reporting School Achievement Comparability of Developmental Scores Across Levels: The Growth Model The National Standard Score Scale Development and Monitoring of National Norms for the ITBS Trends in Achievement Test Performance Norms for Special School Populations Equivalence of Forms Relationships of Forms A and B to Previous Forms i

2 Part 5 Reliability of The Iowa Tests Methods of Determining, Reporting, and Using Reliability Data Internal-Consistency Reliability Analysis.. 64 Equivalent-Forms Reliability Analysis Sources of Error in Measurement Standard Errors of Measurement for Selected Score Levels Effects of Individualized Testing on Reliability Stability of Scores on the ITBS Part 6 Item and Test Analysis Difficulty of the Tests Discrimination Ceiling and Floor Effects Completion Rates Other Test Characteristics Part 7 Group Differences in Item and Test Performance Standard Errors of Measurement for Groups Gender Differences in Achievement Racial-Ethnic Differences in Achievement Differential Item Functioning Part 8 Relationships in Test Performance Correlations Among Test Scores for Individuals Structural Relationships Among Content Domains Levels 9 through Levels 7 and Levels 5 and Interpretation of Factors Reliabilities of Differences in Test Performance Correlations Among Building Averages Relations Between Achievement and General Cognitive Ability Predicting Achievement from General Cognitive Ability: Individual Scores Obtained Versus Expected Achievement Predicting Achievement from General Cognitive Ability: Group Averages Part 9 Technical Consideration for Other Iowa Tests Iowa Tests of Basic Skills Survey Battery Description of the Tests Other Scores Test Development Standardization Test Score Characteristics Iowa Early Learning Inventory Description of the Inventory Test Development Standardization Iowa Writing Assessment Description of the Test Test Development Standardization Test Score Characteristics Constructed-Response Supplement to The Iowa Tests Description of the Tests Test Development Joint Scaling with the ITBS Test Score Characteristics Listening Assessment for ITBS Description of the Test Test Development Standardization Test Score Characteristics Predictive Validity Integrated Writing Skills Test Description of the Tests Test Development Standardization Test Score Characteristics Iowa Algebra Aptitude Test Description of the Test Test Development Standardization Test Score Characteristics Works Cited Index ii

3 Tables and Figures Part 1: Nature and Purposes of The Iowa Tests Table 1.1 Test and Grade Level Correspondence Table 1.2 Number of Items and Test Time Limits Part 2: The National Standardization Program Table 2.1 Summary of Standardization Schedule Table 2.2 Sample Size and Percent of Students by Type of School Table 2.3 Percent of Public School Students by Geographic Region Table 2.4 Percent of Public School Students by SES Category Table 2.5 Percent of Public School Students by District Enrollment Table 2.6 Percent of Catholic Students by Diocese Size and Geographic Region Table 2.7 Percent of Private Non-Catholic Students by Geographic Region Table 2.8 Racial-Ethnic Representation Table 2.9 Test Accommodations Special Education and 504 Students Table 2.10 Test Accommodations English Language Learners Part 3: Validity in the Development and Use of The Iowa Tests Figure 3.1 Steps in Development of the Iowa Tests of Basic Skills Table 3.1 Distribution of Skills Objectives for the Iowa Tests of Basic Skills, Forms A and B Table 3.2 Types of Materials Table 3.3 Content/Process Standards Table 3.4 Listening Content/Process Standards Table 3.5 Comparison of Language Tests by Battery Table 3.6 Computational Skill Level Required for Math Problem Solving and Data Interpretation Table 3.7 Summary Data from Predictive Validity Studies Table 3.8 Readability Indices for Selected Tests Part 4: Scaling, Norming, and Equating The Iowa Tests Table 4.1 Comparison of Grade-to-Grade Overlap Table 4.2 Differences Between National Percentile Ranks Figure 4.1 Trends in National Performance Table 4.3 Summary of Median Differences Figure 4.2 Trends in Iowa Performance Table 4.4 Sample Sizes for Equating Forms A and B Part 5: Reliability of The Iowa Tests Table 5.1 Test Summary Statistics Table 5.2 Equivalent-Forms Reliabilities, Levels Table 5.3 Estimates of Equivalent-Forms Reliability Table 5.4 Mean (Grades 3 8) Reliability Coefficients: Reliability Types Analysis by Tests Table 5.5 Test-Retest Reliabilities, Levels Table 5.6 Standard Errors of Measurement for Selected Standard Score Levels Table 5.7 Correlations Between Developmental Standard Scores, Forms A and B Table 5.8 Correlations Between Developmental Standard Scores, Forms K and L iii

4 Part 6: Item and Test Analysis Table 6.1 Word Analysis Content Classifications with Item Norms Table 6.2 Usage and Expression Content Classifications with Item Norms Table 6.3 Distribution of Item Difficulties Table 6.4 Summary of Difficulty (Proportion Correct) and Discrimination (Biserial) Indices Table 6.5 Ceiling Effects, Floor Effects, and Completion Rates Part 7: Group Differences in Item and Test Performance Table 7.1 Standard Errors of Measurement in the Standard Score Metric for ITBS by Level and Gender by Level and Group Table 7.2 Male-Female Effect Sizes for Average Achievement Table 7.3 Descriptive Statistics by Gender Table 7.4 Gender Differences in Achievement over Time Table 7.5 Race Differences in Achievement Table 7.6 Effect Sizes for Racial-Ethnic Differences in Average Achievement Table 7.7 Fairness Reviewers Table 7.8 Number of Items Identified in Category C in National DIF Study Part 8: Relationships in Test Performance Table 8.1 Correlations Among Developmental Standard Scores Table 8.2 Reliabilities of Differences Among Scores for Major Test Areas: Developmental Standard Scores Table 8.3 Reliabilities of Differences Among Tests: Developmental Standard Scores Table 8.4 Correlations Among School Average Developmental Standard Scores Table 8.5 Correlations Between Standard Age Scores and Developmental Standard Scores Table 8.6 Reliabilities of Difference Scores and Standard Deviations of Difference Scores Due to Errors of Measurement Table 8.7 Correlations, Prediction Constants, and Standard Errors of Estimate for School Averages Part 9: Technical Consideration for Other Iowa Tests Table 9.1 Test Summary Statistics Iowa Tests of Basic Skills Survey Battery, Form A Table 9.2 Average Reliability Coefficients, Grades Iowa Writing Assessment Table 9.3 Correlations and Reliability of Differences Iowa Writing Assessment and Iowa Tests of Basic Skills Language Table 9.4 Internal-Consistency Reliability Constructed-Response Supplement Table 9.5 Correlations and Reliabilities of Differences Constructed-Response Supplement and Corresponding ITBS Subtests Table 9.6 Test Summary Statistics Listening Assessment for ITBS Table 9.7 Correlations Between Listening and ITBS Achievement Table 9.8 Correlations Between Listening Grade 2 and ITBS Grade Table 9.9 Test Summary Statistics Integrated Writing Skills Test, Form M Table 9.10 Correlations Between IWST and ITBS and Language Tests Table 9.11 Test Summary Statistics Iowa Algebra Aptitude Test Grade 8 Table 9.12 Correlations Between IAAT and Algebra Grades and Test Scores iv

5 PART 1 Nature and Purposes of The Iowa Tests The Iowa Tests Validity of the Tests The Iowa Tests consist of a variety of educational achievement instruments developed by the faculty and professional staff at Iowa Testing Programs at The University of Iowa. The Iowa Tests of Basic Skills (ITBS ) measure educational achievement in 15 subject areas for kindergarten through grade 8. The Iowa Tests of Educational Development (ITED ) measure educational achievement in nine subject areas for grades 9 through 12. These test batteries share a history of development that has been an integral part of the research program in educational measurement at The University of Iowa for the past 70 years. In addition to these achievement batteries, The Iowa Tests include specialized instruments for specific achievement domains. This Guide to Research and Development is devoted primarily to the ITBS and related assessments. The Guide to Research and Development for the ITED contains technical information about that test battery and related assessments. Major Purposes of the ITBS Batteries The purpose of measurement is to provide information that can be used to improve instruction and learning. Assessment of any kind has value to the extent that it results in better decisions for students. In general, these decisions apply to choosing goals for instruction and learning strategies to achieve those goals, designing effective classroom environments, and meeting the diverse needs and characteristics of students. The Iowa Tests of Basic Skills measure growth in fundamental areas of school achievement: vocabulary, reading comprehension, language, mathematics, social studies, science, and sources of information. The achievement standards represented by the tests are crucial in educational development because they can determine the extent to which students will benefit from later instruction. Periodic assessment in these areas is essential to tailor instruction to individuals and groups, to provide educational guidance, and to evaluate the effectiveness of instruction. The most valid assessment of achievement for a particular school is one that most closely defines that school s education standards and goals for teaching and learning. Ideally, the skills and abilities required for success in assessment should be the same skills and abilities developed through local instruction. Whether this ideal has been attained in the Iowa Tests of Basic Skills is something that must be determined from an itemby-item examination of the test battery early in the decision-making process. Common practices to validate test content have been used to prepare individual items for The Iowa Tests. The content standards were determined through consideration of typical course coverage, current teaching methods, and recommendations of national curriculum groups. Test content has been carefully selected to represent best curriculum practice, to reflect current performance standards, and to represent diverse populations. The arrangement of items into levels within tests follows a scope and sequence appropriate to a particular level of teaching and cognitive development. Items are selected for content relevance from a larger pool of items tried out with a range of students at each grade level. Throughout the battery, efforts have been made to emphasize the functional value of what students learn in school. Students abilities to use what they learn to interpret what they read, to analyze language, and to solve problems are tested in situations that approximate to the extent possible with a paper and pencil test actual situations in which students may use these skills. Ultimately, the validity of information about achievement derived from The Iowa Tests depends on how the information is used to improve instruction and learning. Over the years, the audience for assessment information has grown. Today it represents varied constituencies concerned about educational progress at local, state, and national levels. To make assessment information useful, careful attention must be paid to reporting results to students, to parents and teachers, to 1

6 school administrators and board members, and to the public. Descriptions of the types of score reports provided with The Iowa Tests are included in the Interpretive Guide for Teachers and Counselors and the Interpretive Guide for School Administrators. How to present test results to various audiences is discussed in these guides. Description of the ITBS Batteries Names of the Tests Iowa Tests of Basic Skills (ITBS ) Form A, Level 5; Form A, Level 6; Forms A and B, Levels 7 and 8; Forms A and B, Levels Description of the Test Batteries The ITBS includes three batteries that allow for a variety of testing needs: The Complete Battery consists of five to fifteen subtests, depending on level, and is available at Levels 5 through 14. The Core Battery consists of a subset of tests in the Complete Battery, including all tests that assess reading, language, and math. It is available at Levels 7 through 14. The Survey Battery consists of 30-minute tests on reading, language, and math. Items in the Survey Battery come from tests in the Complete Battery. It is available at Levels 7 through 14. Nature of the Batteries Levels 5 8 Levels 5 and 6 of Form A are published as a Complete Battery; there is no separate Core Battery or Survey Battery for these levels. Levels 7 and 8 of Forms A and B are published as a Complete Battery (twelve tests), a Core Battery (nine tests), and a Survey Battery (three tests). Levels 9 14 Levels 9 through 14 of Forms A and B are published in a Complete Battery (thirteen tests) and a Survey Battery (three tests). At Level 9, two additional tests are available, Word Analysis and Listening. For Level 9 only, a machine-scorable Complete Battery, a Core Battery (eleven tests), and a Survey Battery are available. Levels 10 through 14 have no separate Core Battery booklet; all Core tests are part of the Complete Battery booklet. Nature of the Levels Levels 5 6 (Grades K.1 1.9) The achievement tests included in the Complete Battery are listed below. The Composite score for these levels, Core, includes only the tests preceded by a solid circle ( ). Those included in the Profile are followed by an asterisk (*). Abbreviations used in this Guide appear in parentheses. * (V) Word Analysis* (WA) Listening* (Li) Language (L) Mathematics (M) : Words* (Level 6 only) (RW) : Comprehension* (Level 6 only) (RC) Levels 7 8 (Grades ) The achievement tests included in the Complete Battery and the Core Battery are listed below. Those in the Core Battery are preceded by a solid circle ( ). Those included in the Profile are followed by an asterisk (*). Test abbreviations are given in parentheses. * (V) Word Analysis* (WA) * (RC) Listening* (Li) Spelling* (L1) Language (L) Mathematics Concepts (M1) Mathematics Problems (M2) Mathematics Computation (M3) Social Studies (SS) Science (SC) Sources of Information (SI) Levels 9 14 (Grades ) The achievement tests in the Complete Battery are listed below. Those in the Core Battery are preceded by a solid circle ( ). Those tests included in the Profile for Level 9 are followed by an asterisk (*). * (V) Comprehension* (RC) Word Analysis* (Level 9 only) (WA) Listening* (Level 9 only) (Li) Spelling* (L1) Capitalization (L2) Punctuation (L3) Usage and Expression (L4) 2

7 Math Concepts and Estimation (M1) Math Problem Solving and Data Interpretation (M2) Math Computation (M3) Social Studies (SS) Science (SC) Maps and Diagrams (S1) Reference Materials (S2) Tests in the Survey Battery, Language, and Mathematics comprise items from the Complete Battery. Each test is divided into the parts indicated. (two parts) Comprehension Language Mathematics (three parts) Concepts, Problem Solving and Data Interpretation Estimation Computation Grade Levels and Test Levels Levels 5 through 14 represent a comprehensive assessment program for kindergarten through grade 9. Each level is numbered to correspond roughly to the age of the student for whom it is best suited. A student should be given the level most compatible with his or her level of academic development. Typically, students in kindergarten and grades 1 and 2 would take only three of the Primary Battery s four levels before taking Level 9 in grade 3. Table 1.1 shows how test level corresponds to a student s level of academic development, expressed as a grade range. Decimals in the last column indicate month of the school year. For example, K means the first month of kindergarten through the fifth month of grade 1. Test Lengths and Times For Levels 5 through 8, the number of questions and approximate working time for each test are given in Table 1.2. Tests at these levels are untimed; the actual time required for a test varies somewhat with the skill level of the students. (The administration times in the table are based on average rates reported by teachers in tryout sessions.) The Level 6 test is administered in two sessions. For Levels 9 through 14, all tests are timed; the administration times include time to read directions as well as to take the tests. Table 1.1 Test and Grade Level Correspondence Iowa Tests of Basic Skills, Forms A and B Test Level Age Grade Level 5 5 K K Nature of the Questions For Levels 5 through 8, questions are read aloud except at Level 6 for parts of the test, and at Levels 7 and 8 except for the test and parts of the and Math Computation tests. Questions are multiple choice with three or four response options. Responses are presented in pictures, letters, numerals, or words, depending on the test and level. All questions in Levels 9 through 14 are multiple choice, have four or five options, and are read by the student. Mode of Responding Students who take Levels 5 through 8 mark answers in machine-scorable booklets by filling in a circle. Those who take Levels 9 through 14 mark answers on a separate answer folder (Complete Battery) or answer sheet (Survey Battery). For the machine-scorable booklets at Level 9, students mark answers in the test booklets. Directions A separate Directions for Administration manual is provided for each Complete Battery (Levels 5 through 8) and Core Battery (Levels 7 and 8) level and form. The Survey Battery (Levels 7 and 8) has separate Directions for Administration manuals for each level and form. At Levels 9 through 14, there is one Directions for Administration manual for Forms A and B of the Complete Battery. At these levels, the Survey Battery has a single Directions for Administration manual. The machine-scorable booklets of Level 9 have separate Directions for Administration manuals. 3

8 Table 1.2 Number of Items and Test Time Limits Iowa Tests of Basic Skills, Forms A and B Level 5: Complete Battery Approximate Working Time (Minutes) Number of Items Word Analysis Listening Language Mathematics Core Tests 1 hr., 10 min. 87 Complete Battery 2 hrs. 146 Level 6: Complete Battery Approximate Working Time (Minutes) Number of Items Word Analysis Listening Language Mathematics : Words : Comprehension Core Tests 1 hr., 10 min. 97 Complete Battery 2 hrs., 43 min. 211 Level 7: Complete and Core Battery Approximate Working Time (Minutes) Number of Items Word Analysis Listening Spelling Language Math Concepts Math Problems Math Computation Social Studies Science Sources of Information Core Battery 3 hrs., 5 min. 260 Complete Battery 4 hrs., 20 min. 344 Level 8: Complete and Core Battery Approximate Working Time (Minutes) Number of Items Word Analysis Listening Spelling Language Math Concepts Math Problems Math Computation Social Studies Science Sources of Information Core Battery 3 hrs., 5 min. 284 Complete Battery 4 hrs., 25 min. 374 Level 7: Survey Battery Approximate Working Time (Minutes) Number of Items Language Mathematics Mathematics Computation 8 13 Survey Battery 1 hr., 25 min. 114 Level 8: Survey Battery Approximate Working Time (Minutes) Number of Items Language Mathematics Mathematics Computation 8 17 Survey Battery 1 hr., 25 min

9 Table 1.2 (continued) Number of Items and Test Time Limits Iowa Tests of Basic Skills, Forms A and B Levels 9 14: Number of Items, Complete and Core Battery Working Time (Minutes) Level Comprehension Spelling Capitalization Punctuation Usage and Expression Mathematics Concepts and Estimation Mathematics Problem Solving and Data Interpretation Mathematics Computation Social Studies Science Maps and Diagrams Reference Materials Word Analysis Listening Core Battery (3 hrs., 31 min.) Complete Battery (5 hrs., 26 min.) This test is administered in two parts. 2 This test is untimed. The time given is approximate. 3 With Word Analysis and Listening at Level 9, testing time is 256 min. (4 h., 16 m.) and the number of items is With Word Analysis and Listening at Level 9, testing time is 371 min. (6 h., 11 m.) and the number of items is 428. Levels 9 14: Number of Items, Survey Battery Working Time (Minutes) Level Part 1: Part 2: Comprehension Language Mathematics Part 1: Concepts and Problems Part 2: Estimation Part 3: Computation Survey Battery (1 hr., 30 min.)

10 Other Iowa Tests Iowa Writing Assessment The Iowa Writing Assessment measures a student s ability to generate, organize, and express ideas in writing. This assessment includes four prompts that require students to compose an essay in either narrative, descriptive, persuasive, or expository modes. With norm-referenced evaluation of a student s writing about a specific topic, the Iowa Writing Assessment adds to the information obtained from other language tests and from the writing students do in the classroom. Listening Assessment for ITBS Content specifications for Levels 9 through 14 of the Listening tests are based on current literature in the teaching and assessment of listening comprehension. The main purposes of the Listening Assessment are: (a) to measure strengths and weaknesses in listening so effective instruction can be planned to meet individual and group needs; (b) to monitor listening instruction; and (c) to help make teachers and students aware of the importance of good listening strategies. Other Manuals In addition to this Guide to Research and Development, several other manuals provide information for test users. Each Directions for Administration manual includes a section on preparing for test administration as well as the script needed to administer the tests. The Test Coordinator Guide offers suggestions about policies and procedures associated with testing, advice about planning for and administering the testing program, ideas about preparing students and parents, and details about how to prepare answer documents for the scoring service. The Interpretive Guide for Teachers and Counselors describes test content, score reports, use of test results for instructional purposes, and communication of results to students and parents. The Interpretive Guide for School Administrators offers additional information, including guidance on designing a districtwide assessment program and reporting test results. The Norms and Score Conversions booklets contain directions for hand scoring and norms tables for converting raw scores to derived scores such as standard scores and percentile ranks. Constructed-Response Supplement to The Iowa Tests These tests may be used with the Complete Battery and Survey Battery of the ITBS. The Constructed- Response Supplement measures achievement in reading, language, and math in an open-ended format. Students write answers in the test booklet, and teachers use the scoring guidelines to rate the responses. The results can be used to provide information about achievement to satisfy requirements for multiple measures. 6

11 PART 2 The National Standardization Program Normative data collected at the time of standardization is what distinguishes normreferenced tests from other assessments. It is through the standardization process that scores, scales, and norms are developed. The procedures used in the standardization of The Iowa Tests are designed to make the norming sample reflect the national population as closely as possible, ensuring proportional representation of ethnic and socioeconomic groups. The standardization of the Iowa Tests of Basic Skills (ITBS) Complete Battery and Survey Battery was a cooperative venture. It was planned by the ITBS authors, the publisher, and the authors of the Iowa Tests of Educational Development (ITED) and the Cognitive Abilities Test (CogAT ). Many public and non-public schools cooperated in national item tryouts and standardization activities, which included the 2000 spring and fall test administrations, scaling, and equating studies. Planning the National Standardization Program The standardization of the ITBS, ITED, and CogAT was carried out as a single enterprise. After reviewing previous national standardization programs, the basic principles and conditions of those programs were adapted to the following current needs: The sample should be selected to represent the national population with respect to ability and achievement. It should be large enough to represent the diverse characteristics of the population, but a carefully selected sample of reasonable size would be preferred over a larger but less carefully selected sample. Sampling units should be chosen primarily on the basis of school district size, region of country, and socioeconomic characteristics. A balance between public and non-public schools should be obtained. The sample of attendance centers should be sufficiently large and selected to provide dependable norms for building averages. Attendance centers in each part of the sample should represent the central tendency and variability of the population. To ensure comparability of norms from grade to grade, all grades in a selected attendance center (or a designated fraction thereof) should be tested. To ensure comparability of norms for ability and achievement tests, both the ITBS and the CogAT should be administered to the same students at the appropriate grade level. To ensure comparability of norms for Complete and Survey Batteries, alternate forms of both batteries should be administered at the appropriate grade level to the same students or to equivalent samples of students. To ensure applicability of norms to all students, testing accommodations for students who require them should be a regular part of the standardization design. Procedures for Selecting the Standardization Sample Public School Sample Three stratifying variables were used to classify public school districts across the nation: geographic region, district enrollment, and socioeconomic status (SES) of the school district. Within each geographic region (New England and Mideast, Southeast, Great Lakes and Plains, and West and Far West), school districts were stratified into nine enrollment categories. School district SES was determined with data from the National Education Database (Quality Education Data, 2002). The socioeconomic index is the percent of students in a district falling below the federal government poverty guideline, similar to the Orshansky index used in sampling for the National Assessment of Educational Progress (NAEP). This index was used in each of the four regions to break the nine district-size categories into five strata. 7

12 In each SES category, districts were selected at random and designated as first, second, or third choices. Administrators in the selected districts were contacted by the publisher and invited to participate. If a district declined, the next choice was contacted. Catholic School Sample The primary source for selecting and weighting the Catholic sample was NCEA/Ganley s Catholic Schools in America (NCEA, 2000). Within each geographic region of the public sample, schools were stratified into five categories on the basis of diocesan enrollment. A two-stage random sampling procedure was used to select the sample. In the first stage, dioceses were randomly selected from each of five enrollment categories. Different sampling fractions were used, ranging from 1.0 for dioceses with total student enrollment above 100,000 (all four were selected) to.07 for dioceses with fewer than 10,000 students (seven of 102 were selected). In the second stage, schools were randomly chosen from each diocese selected in the first stage. In all but the smallest enrollment dioceses where only one school was selected two schools were randomly chosen. If the selected school declined to participate, the alternate school was contacted. If neither school agreed to participate, additional schools randomly selected from the diocese were contacted. Private Non-Catholic School Sample The sample of private non-catholic schools was obtained from the QED data file. The schools in each geographic region of the public and Catholic samples were stratified into two types: churchrelated and nonsectarian. Schools were randomly sampled in eight categories (region by type of school) until the target number of students was reached. For each school selected, an alternate school was chosen to be contacted if the selected school declined to participate. Summary These sampling procedures produced (1) a national probability sample representative of students nationwide; (2) a nationwide sample of schools for school building norms; (3) data for Catholic/private and other special norms; and (4) empirical norms for the Complete Battery and the Survey Battery. The authors and publisher of the ITBS are grateful to many people for assistance in preparing test materials and administering tests in item tryouts and special research projects. In particular, gratitude is acknowledged to administrators, teachers, and students in the schools that took part in the national standardization. These schools are listed at the end of this part of the Guide to Research and Development. Schools marked with an asterisk participated in both spring and fall standardizations. Design for Collecting the Standardization Data A timetable for administration of the ITBS and the CogAT is given in Table 2.1. This illustrates how the national standardization study was designed. During the spring standardization, students took the appropriate level of the Complete Battery of the ITBS, Form A. These same students took Form 6 of the CogAT. The design of the fall standardization was more complex. Every student in grades 2 through 8 participated in two units of testing. The order of the two testing units was counterbalanced. In the first testing unit, the student took the Complete Battery of either Form A or Form B of the ITBS. In grades 2 and 3, Forms A and B of the ITBS machine-scorable booklets were used in alternate classrooms. In approximately half of the grade 3 classrooms, alternate forms of the ITBS Level 8 were administered; in the remaining grade 3 classrooms, Forms A and B of Level 9 were administered to every other student. In grades 4 through 8, Forms A and B were administered to every other student in all classrooms. In the second testing unit of the fall standardization, students took Form A or Form B of the Survey Battery. (Students who had taken Form A of the Complete Battery took Form B of the Survey Battery and vice versa.) Weighting the Samples After materials from the spring standardization had been received by the Riverside Scoring Service, the number and percents of students in each sample (public, Catholic, and private non-catholic) and stratification category were determined. The percents were adjusted by weighting to compensate for missing categories and to adjust for schools that tested more or fewer students than required. 8

13 Table 2.1 Summary of Standardization Schedule Time First Unit Second Unit Spring 2000 ITBS, Form A Complete Battery (Levels 5 8, Grades K 2) CogAT, Form 6 (Levels 1 2, Grades K 3) ITBS, Form A Complete Battery (Levels 9 14, Grades 3 8) CogAT, Form 6 (Levels A F, Grades 3 8) Fall 2000 ITBS, Form A/B Complete Battery (Levels 7 8, Grades 2 3) ITBS, Form A/B Complete Battery (Levels 9 14, Grades 3 8) ITBS, Form B/A Survey Battery (Levels 9 14, Grades 3 8) The number of students in the 2000 spring national standardization of the ITBS is given in Table 2.2 for the public, Catholic, and private non-catholic samples. Table 2.2 also shows the unweighted and weighted sample percents and the population percents for each cohort. Optimal weights for these samples were determined by comparing the proportion of students nationally in each cohort to the corresponding sample proportion. Once the optimal weight for each sample was obtained, the stratification variables were simultaneously considered to assign final weights. These weights (integer values 0 through 9, with 3 denoting perfect proportional representation) were assigned to synthesize the characteristics of a missing unit or adjust the frequencies in other units. As a result, the weighted distributions in the three standardization samples closely approximate those of the total student population. Tables 2.3 through 2.7 summarize the unweighted and weighted sample characteristics for the spring 2000 standardization of the ITBS based on the principal stratification variables of the public school sample and other key characteristics of the nonpublic sample. In addition to the regular norms established in the 2000 national standardization, separate norms were established for special populations. These norms and the procedures used to derive them are discussed in Part 4. 9

14 Table 2.2 Sample Size and Percent of Students by Type of School Spring 2000 National Standardization Sample, ITBS, Grades K 8 Public School Sample Catholic School Sample Private Non-Catholic Sample Unweighted Sample Size Unweighted Sample % Weighted Sample % Population % 149,831 10,797 9, , Table 2.3 Percent of Public School Students by Geographic Region Spring 2000 National Standardization Sample, ITBS, Grades K 8 Geographic Region % of Students in Sample % of Students in Weighted Sample % of Students in Population New England and Mideast Southeast Great Lakes and Plains West and Far West Table 2.4 Percent of Public School Students by SES Category Spring 2000 National Standardization Sample, ITBS, Grades K 8 SES Category % of Students in Sample % of Students in Weighted Sample % of Students in Population High High Average Average Low Average Low

15 Table 2.5 Percent of Public School Students by District Enrollment Spring 2000 National Standardization Sample, ITBS, Grades K 8 District K 12 Enrollment % of Students in Sample % of Students in Weighted Sample % of Students in Population 100, ,000 99,999 25,000 49,999 10,000 24,999 5,000 9, ,500 4, ,200 2, , Less than Table 2.6 Percent of Catholic Students by Diocese Size and Geographic Region Spring 2000 National Standardization Sample, ITBS, Grades K 8 % of Students in Sample % of Students in Weighted Sample % of Students in Population Diocese Size 100, ,000 99, ,000 49, ,000 19, Less than 10, Geographic Region New England and Mideast Southeast Great Lakes and Plains West and Far West

16 Table 2.7 Percent of Private Non-Catholic Students by Geographic Region Spring 2000 National Standardization Sample, ITBS, Grades K 8 Geographic Region % of Students in Sample % of Students in Weighted Sample % of Students in Population New England and Mideast Southeast Great Lakes and Plains West and Far West Racial-Ethnic Representation Although not a direct part of a typical sampling plan, the racial-ethnic composition of a national standardization sample should represent that of the school population. The racial-ethnic composition of the 2000 ITBS spring standardization sample was estimated from responses to demographic questions on answer documents. In all grades, all the racialethnic group(s) to which a student belonged was requested. In kindergarten through grade 3, teachers furnished this information. In the remaining grades, students furnished it. The results reported in Table 2.8 include students in Catholic and other private schools. The table also shows estimates of population percents in public schools for each category, according to the National Center for Education Statistics. The response rate for racial-ethnic information was high; 98 percent of the standardization participants indicated membership in one of the groups listed. Although the percents of students in each group fluctuate from grade to grade, differences between sample and population percents were generally within chance error. This was true for all groups except Hispanics or Latinos, who were slightly underrepresented. However, some of this underrepresentation can be attributed to school districts exempting from testing students whose first language is not English. These students are not as likely to be represented in the test-taking population as they are in the school population. Collectively, the results in Table 2.8 provide evidence of the overall quality of the national standardization sample and its representativeness of the racial and ethnic makeup of the U.S. student population. Participation of Students in Special Groups In the spring 2000 national standardization, schools were given detailed instructions for the testing of students with disabilities and English Language Learners. Schools were asked to decide whether students so identified should be tested, and, if so, what modifications in testing procedures were needed. Among students with disabilities, nearly all were identified as eligible for special education services and had an Individualized Education Program (IEP), an Individualized Accommodation Plan (IAP), or a Section 504 Plan. Schools were asked to examine the IEP or other plan for these students, decide whether the student should receive accommodations, and determine the nature of those accommodations. Schools were told an accommodation refers to a change in the procedures for administering the test and that an accommodation is intended to neutralize, as much as possible, the effect of the student s disability on the assessment process. Accommodations should not change the kind of achievement being measured, but change how achievement is measured. When accommodations were used, the test administrator recorded the type of accommodation on each student s answer document. The accommodations most frequently used by students with IEPs or Section 504 Plans were listed on the student answer document. Space for indicating other accommodations was also included. 12

17 Table 2.8 Racial-Ethnic Representation Iowa Tests of Basic Skills Complete Battery, Form A Spring 2000 National Standardization White (62.1 %)* Weighted Grade Number Percent Number Percent K 11, , , , , , , , , , , , , , , , , , , , Black or African American (17.2 %)* Weighted Grade Number Percent Number Percent K 2, , , , , , , , , , , , , , , , , , , , Hispanic or Latino (15.6 %)* Weighted Grade Number Percent Number Percent K 1, , , , , , , , , , , , , , , , , , , , Asian/Pacific Islander (4.0 %)* Weighted Grade Number Percent Number Percent K , , , , , , , , , , , American Indian/Alaskan Native (1.2 %)* Weighted Grade Number Percent Number Percent K , , , , , , , , Native Hawaiian (NA) Weighted Grade Number Percent Number Percent K , , *Population percent (Source: Digest of Education Statistics 2000, public school enrollment) 13

18 For students whose native language was not English and who had been in an English-only classroom for a limited time, two decisions had to be made prior to testing. First, was English language developed sufficiently to warrant testing, and, second, should an accommodation be used? In all instances, the district s instructional guidelines were used in decisions about individual accommodations. The test administrators were told that the use of testing accommodations with English Language Learners is intended to allow the measurement of skills and knowledge in the curriculum without significant interference from a limited opportunity to learn English. Those just beginning instruction in English were not likely to be able to answer many questions no matter what types of accommodations were used. For those in the second or third year of instruction in an English as a Second Language (ESL) program, accommodations might be warranted to reduce the effect of limited English proficiency on test performance. The types of accommodations sometimes used with such students were listed on the student answer document for coding. Table 2.9 summarizes the use of accommodations with students with disabilities during the standardization. While the percents vary somewhat across grades, an average of about 7 percent of the students were identified as special education students or as having a 504 Plan. Of these students, roughly 50 percent received at least one accommodation. The last column in the table shows that in the final distribution of scores from which the national norms were obtained, an average of 3 percent to 4 percent of the students received an accommodation. Table 2.10 reports similar information for English Language Learners. Test administration for the 2000 spring standardization of the ITBS, Form A, took place between March 23 and May 29; it took place for the fall standardization between September 21 and November 11. The spring norming group was a national probability sample of approximately 170,000 students in kindergarten through grade 8; the fall sample was approximately 76,000 students. After answer documents were checked and scored and sampling weights had been assigned to schools, weighted opening and closing dates were determined. These are reference points for the empirical norms dates. The median empirical norms date for spring testing is April 30; for fall testing it is October 22. Regular fall, midyear, and spring norms can be used by school districts that operate on a twelve-month schedule. To do so, testing should be scheduled so the number of instructional days prior to testing corresponds to the median number of instructional days for schools in the national standardization. For example, the fall norms for the 2000 national standardization were established with a median testing date of October 22, on average 40 instructional days from the median start date of schools in the national standardization. If a school year begins on July 15, testing should be scheduled between September 1 and September 21. Doing so places the median testing date at September 10, about 40 instructional days from the July 15 start date. By testing during this period, instructional opportunity is comparable to the norms group and the use of fall norms is therefore appropriate. Testing dates for twelve-month schools can be calculated in a similar way so midyear and spring norms can be used. Empirical Norms Dates To provide more information for schools with alternative school calenders, data were collected from districts on their opening and closing dates. Procedures to analyze these data were altered from those used in the standardization when the Title I program first required empirical norms dates to determine weighted opening and closing dates. The procedures used and the advice given to school districts that do not have a standard 180-day, September-to-May school year are noted below. 14