Designing Assessments of Learning with Technology

Transcription

1 Assessment in Education, Vol. 10, No. 3, November 2003 Designing Assessments of Learning with Technology EDYS S. QUELLMALZ & ROBERT KOZMA SRI International, 333 Ravenswood Avenue, Menlo Park, CA 94025, USA ABSTRACT In this article, we describe an approach designed to assess student skills as they use information and communications technologies (ICT) to solve complex, significant, recurring academic and applied problems. We describe a Coordinated ICT Assessment Framework and its underpinnings in theory and research. We describe studies of ICT use in schools and highlight the recent Second Information Technology in Education Study: Module 2 (SITES M2) international case studies. We then offer an innovative design approach for assessing students performances as they employ a range of reasoning strategies to use technologies to solve academic and applied problems. Introduction Globally, the widespread use of computers in schools has now gone beyond the major industrialised countries as evidenced in a 1999 International Association for the Evaluation of Educational Achievement (IEA) study the Second Information Technology in Education Study: Module 1 (SITES M1) which documented the significant investment in educational information and communication technologies (ICT) around the world (Pelgrum & Anderson, 1999). In the USA, virtually all schools now have computers, and in a recent study from the National Centre for Educational Statistics (2001), the number of computers in US schools has increased to one for every five students. This increase in the educational use of ICT is driven and supported by evidence that new technologies can change schools and improve education (Bracewell et al., 1998; Coley et al., 1999; Means & Olson, 1995; Wenglinski, 1998) and by major shifts in policy at both national and multinational levels. ICT can transform schools and classrooms by bringing in new curricula based on real-world problems, providing scaffolds and tools to enhance learning, giving students and teachers more opportunities for feedback and reflection, and building local and global communities that include students, teachers, parents, practising scientists, and other interested parties (Bransford et al., 2000). The ubiquitous presence of ICT and the major investment by public and private organisations in placing technologies in schools are accompanied by calls for proof of returns on the investment. However, attempts to document the impacts of ICT use generally fail to be captured by traditional ISSN X print; ISSN X online/03/ Taylor & Francis Ltd DOI: /

2 390 E. S. Quellmalz & R. Kozma assessment approaches (Burns & Ungerleider, 2002; McFarlane et al., 2000). New assessments are required that use innovative approaches to capture the new forms of learning associated with ICT use and to dissociate them from students ability simply to manipulate features of the tools. With funding from the US National Science Foundation (NSF), the Centre for Technology in Learning at SRI [1] International has drawn upon its expertise in the development of both performance assessment and technology systems to collaborate with an international group of scholars to accomplish three main goals: (1) to forge a coordinated framework for the assessment of ICT in school subjects which distinguishes tool operation from applied, strategic use; (2) to develop innovative methodologies for assessing student ICT competencies specifically, those competencies related to learning with ICT in science and maths; and (3) to develop a technology infrastructure to deliver ICT assessments. Development of the Coordinated ICT Assessment Framework One component of the project funded by the US National Science Foundation, Coordinated, Innovative Designs for International Information Communication Technology Assessment in Science and Mathematics Education, is the development of the Coordinated ICT Assessment Framework that aligns standards documents and assessment frameworks for technology proficiency with standards and frameworks for the academic domains of science and mathematics. In this article, we describe how our new, integrated framework is grounded in theory, research, and practice drawn from principles of learning and studies of ICT use in schools. Learning with Technology In the highly influential synthesis of research, How People Learn, the authors indicate that an emphasis on learning with understanding is a hallmark of the new science of learning (Bransford et al., 2000). They summarise research on the benefits of new technologies for enhancing student learning, stressing that technologies do not guarantee effective learning, but that technologies can make it easier to create environments that embed research-based principles of learning. In many fields, new technologies allow representation of data in new ways. Technologies such as three-dimensional models of planets or molecular structures help people to visualise difficult-to-understand concepts. This move from static to dynamic models has profoundly changed the nature of inquiry in mathematics and science for researchers, as well as for students. Representational technologies have expanded the phenomena that can be considered and the nature of argumentation and acceptable evidence (Bachelard, 1984; Holland, 1995). Communication technologies enable socially constructed dialogues of learning among distributed collaborators, learners, and experts, set within extended projects. Constructivist views of learning, in contrast to behaviourist paradigms, drive the designs of different goals, standards, curriculum, and assessment related to the role of technology in teaching and learning. The National Research Council report,

3 Designing Assessments of Learning with Technology 391 Knowing What Students Know, brings together the two foundational areas of cognitive science and measurement science to propose new directions for the design and use of assessments (Pellegrino et al., 2001). This report synthesises decades of research and recommends that assessment practices need to move beyond a focus on component skills and discrete bits of knowledge to encompass complex, authentic performances. The authors decry the poor fit of typical standardised test items with the cognitive outcomes targeted by technology-enhanced learning environments. They recommend a principled approach to assessment design that distinguishes the specific skills related to particular technology tools from the reasoning skills and knowledge required to orchestrate technologies to solve complex problems. Our model views cognition and learning as a process distributed among people, tools, and activities (Pea, 1993; Perkins, 1993; Solomon, 1993). This is a particularly appropriate model for the assessment of ICT skills. Solomon et al. (1991) make a distinction between the cognitive effects of technology and the cognitive effects with technology. Effects of technology are those residual changes in students cognitive capacity that result from the use of technology to learn. Effects with technology are those performances that students display while equipped with a cognitive tool, such as a visualiser, analysis package, or a model builder. From the latter perspective, some cognition is performed by the person and some by the technology that they use (Kozma, 1991). Central to our approach to the design of ICT assessments is the measurement of students skills as they use technological tools to address problems in science or mathematics. The prospect is that students are able to capitalise on the capabilities of the technology to engage in complex thinking and solve complex problems. Pea (1993) describes the relationship between people and tools as they interact in the context of activity. Tools serve to reorganise both the task and cognition; what humans do in their activities is transformed by technologies. Our view of ICT assessment is that twenty-first century assessments should incorporate the explicit examination of technologies in supporting, extending, and transforming student learning. New assessment frameworks are required that specify the strategies, procedures, and knowledge to be tested as students use technologies to solve significant, complex problems within subject matter domains. The Coordinated ICT Assessment Framework aims to guide the coherent collection of evidence within and across studies and thereby shape the design of innovative approaches to capture twenty-first century skills. Alignment of Frameworks Goals for student attainment of twenty-first century skills are being set by national and local educational systems and by professional organisations. These documents are meant to shape the intended curriculum. These curriculum frameworks often integrate the use of technology. But rather than focus on technology features and basic operations (e.g. how to save a document, write a Web search query), these frameworks focus on the strategic deployment of technology in school curricula and learning environments (e.g. conducting analyses using graphing tools) to support the

4 392 E. S. Quellmalz & R. Kozma learning of complex skills such as information management, problem solving, analysis, communication, and collaboration, skills needed in the twenty-first century economy and society (European Commission, 2000; OECD, 2001a). In our NSF study, we formed a working group of international experts in ICT representing Chile, Finland, Norway, Singapore, and the USA. The group aligned standards documents that specify important technology proficiencies with those focusing on mathematics and science and the role of technology within the domains. In addition, we examined assessment frameworks that shape major national and international achievement studies in science and mathematics. Our alignment analyses employed crosscutting, general cognitive demands (declarative, procedural, schematic, and strategic knowledge) to classify the knowledge and skills expected according to types of conceptual understanding about technology tools and the subject domain, as well as kinds of proceduralised and strategic reasoning. Relationship to technology proficiency frameworks. The ICT strategies we specify in the Coordinated ICT Assessment Framework appear throughout technology proficiency frameworks and are differentiated from the facts and skills needed to operate the technology tools. The International Society for Technology and Education (ISTE) standards, for example, propose that as a result of ICT integration in the curriculum, students become capable information technology users, information seekers, analysts, evaluators, problem solvers, decision-makers, creative and effective users of productivity tools, and communicators, collaborators, publishers and producers (ISTE, 2000). The US National Educational Technology Standards (NETS) include use of productivity tools, research tools, and communication tools for purposes such as enhancing learning, collaborating, publishing and communicating (ISTE, 2000). The NETS framework also calls for demonstrations of basic operations and concepts. Relationship to subject matter standards and assessment frameworks. The US National Science Education Standards (NSES) describe the science and technology partnership as establishing connections between the natural and designed worlds and providing students with opportunities to develop decision-making abilities (NSES, 1996). In the US National Council of Teachers of Mathematics document, Principles and Standards, technology is one of the six principles. The standards document makes the point that technologies allow students to focus on decision-making, reflection, reasoning and problem solving (NCTM, 2000). The assessment frameworks for the IEA Trends in Maths and Science Study (TIMSS) and OECD Programme for International Student Assessment (PISA) science and mathematics present similar types of knowledge and reasoning categories (OECD, 2001b). From our alignment study, we culled common categories of ICT strategic use that could shape the coherent collection of evidence across studies of students abilities to use ICT. The Coordinated ICT Assessment Framework. Figure 1 presents a model of our Coordinated ICT Assessment Framework. The circle depicts the subject matter domain the content and processes of the

5 Designing Assessments of Learning with Technology 393 FIG. 1.Coordinated ICT assessment framework. discipline, which in our NSF project focused on science and mathematics. The left side represents the declarative knowledge of the domain, that can vary from content-lean, factual knowledge to content-rich, schematic knowledge composed of interrelated concepts and principles (Baxter & Glaser, 1998). The right side represents the process dimension where problem solving demands of an assessment can range from simple, procedural knowledge for routine problems, to complex, strategic knowledge for non-routine problems. Within the problem space, learners use ICT strategies to integrate technologies into problem solving activities. The ICT strategies include taking advantage of the capabilities of technologies to understand and plan how to approach a problem, access and organise information and relevant data; represent and transform data and information; analyse and interpret information and data; critically evaluate the relevance, credibility and appropriateness of information, data, and conclusions; communicate ideas, findings, and arguments; design products within constraints; and collaborate to solve complex problems and manage information. Figure 1 deliberately portrays these ICT strategies as non-linear and iterative. Thus, planning may be needed to find relevant digital information and data at the outset of a task and again to decide what to vary in a modelling tool to test a model. Various technologies can support collaboration throughout the problem solving activities (see O Neil et al., this issue). Technology tools appear in the centre of the problem space. Internet, productivity, and specialised tools may be chosen to accomplish multiple ICT strategies. Factual and procedural knowledge required for the operation of specific tools or classes of tools can vary according to the affordances of particular tools and the basic or more advanced features chosen or required. Our international working group

6 394 E. S. Quellmalz & R. Kozma chose to focus our framework on generalisable ICT strategies, rather than discrete, often changing features of technology tools. Table I presents a more detailed description of the Coordinated ICT Assessment Framework. The table aligns subject domain and ICT knowledge and strategies with general cognitive demands and identifies component strategies derived from analyses of technology use in research and practice. General cognitive demands include: (1) declarative knowledge of factual information knowing what ; (2) procedural knowledge of routines knowing how ; and (3) schematic and strategic knowledge knowing when, and why. Declarative knowledge within a domain might include, for example, information about the habitats of endangered species. Declarative knowledge about technologies might include knowing the names of features of a PowerPoint view menu and the display generated by each view. Procedural knowledge in mathematics might involve performing a calculation or routine steps. Procedural knowledge related to technologies might include knowing how to specify a formula in an Excel spreadsheet. The ICT strategies of interest in the Coordinated ICT Assessment Framework fall in the Schematic and Strategic Knowledge category that identifies organised knowledge structures (schematic) and systematic methods (strategic) to assess where students are taking advantage of the capabilities of technologies to solve complex problems. Planning strategies for a problem on how to save endangered species, for example, could involve analysing the problem presented according to information given in various representational forms, such as visualisations, maps, and tables of endangered species populations, and developing plans for collecting information and data from a range of databases and Web sites. Components of the ICT strategy for critical evaluation might include evaluating the credibility of reports by game preserves and sporting clubs on the need for stabilising animal populations. Students might search the Web for more information about the membership and views of the organisations and also exchange critiques of others reports with remote collaborators. Technology Practices in Schools In this section, we examine international and national studies and curriculum projects that document the ways in which ICT strategies are being implemented in schools, as well as the impacts of technologies on emerging pedagogical practices and student outcomes. We analysed these studies to determine how well our framework represents current and emerging technology use in schools. Our analyses also enabled us to identify recurring types of classroom assignments to represent in our design of assessments of student learning. Technology Surveys Recent national and international technology surveys provide a general picture of technology use in schools. A 1998 national survey of US teachers found that teachers use a wide variety of software with their students (Becker & Anderson,

7 Designing Assessments of Learning with Technology 395 TABLE I. ICT assessment framework General ICT Knowledge & Strategies: Cognitive Use Technology to Solve Sample Component Strategies: Demands Complex Problems Use Technology to: Declarative Knowledge Identify/list required domain information Identify features and functions of technology tools Identify features Identify functions Identify uses of tools For each tool group and specific tool, identify appropriate uses Procedural Perform steps Follow directions Knowledge Operate tools Use algorithm Produce component and complete operations Schematic Plan strategies and procedures Analyse problem and Identify needed and given information Strategic Knowledge Pose questions Specify design for data/information collection Specify analysis plan Choose appropriate tools Specify product form and content Access and organise information & data Represent and transform information & data Analyse and interpret information & data Critically evaluate Communicate ideas, findings, arguments Specify search purpose/topic Navigate directories Generate Web searches Search multiple representational formats Generate representations from data or phenomena Transform data from one form to another Take and record measurements Identify information/data Apply quantitative and qualitative procedures Understand & compare data and information Infer trends/ patterns Produce solutions/findings Use modelling and visualisation tools to investigate, compare, test Evaluate relevance, credibility of information, data, representations Evaluate quality of plan, conduct, analysis, argument, conclusions Express questions, ideas clearly & appropriately Present ideas, findings in alternative formats appropriate for audience Present supported argument/findings continued overleaf

8 396 E. S. Quellmalz & R. Kozma TABLE I. ICT assessment framework continued Design product Collaborate to solve complex problems and manage information Compose product to fit constraints, appropriate for audience, purpose Plan project work and roles Contribute relevant information Fulfil task assignment Incorporate and integrate others information and views 1998). Teachers were apt to see information acquisition and written expression as primary objects of use of computers. Data analysis goals were addressed through use of spreadsheets, databases, and simulations. The SITES Module 1 surveys found that students in most countries were expected to be able to use word processing and spreadsheets by the time they finished secondary school, if not earlier (Pelgrum & Anderson, 1999). In the Integrated Studies of Educational Technology (ISET) survey of 1,300 US teachers, 66% of the secondary mathematics teachers reported having students use computers to solve problems and analyse data. The study found that less traditional teachers were more likely to use technology in the classroom. Teachers reported an increase or slight increase in student achievement in students research skills (76%), breadth and depth of understanding of subjects taught (64%), problem solving skills (57%), and quality of writing (59%) (Adelman et. al., 2002). SITES M2 Case Studies The recently completed Second Information Technology in Education Study Module 2 (SITES M2) provides a much more detailed picture of how ICT is being used to support innovative teaching practices around the globe. SITES M2 was an IEA project that collected and analysed 174 case reports of innovative classroom practices from 28 countries in Europe, North America, Asia Pacific, Africa, and South America. The analysis of these case reports found interesting patterns in how technology was used to support instructional change around the world. Design of the study. Research teams in each of the 28 participating countries formed national panels to identify innovative practices. The panels consisted of researchers, teachers, school administrators, and policy makers. They used a set of standard criteria according to which selected cases were those where: There were changes in teaching, learning, or curricular practices Technology played a significant role in supporting these changes The changes resulted in positive outcomes for students and/or teachers The innovations could be sustained and transferred The innovations were innovative, as defined by the national committees. The panels reviewed over 220 final nominations to select the 174 cases that were submitted for the international study. While these selected cases are in no way

9 Designing Assessments of Learning with Technology 397 representative of what is happening in countries around the world, they can be seen as representing what national panels saw as the best practices in their countries. The research teams collected information on each case from a variety of sources that included interviews of administrators, teachers, students, and parents; classroom observations; and the analysis of documents, (e.g. teacher lesson plans and samples of student work). The researchers used a standard template to write up each case report describing teacher and student classroom practices, the use of ICT, curriculum goals, teacher and student outcomes, and contextual factors, such as school background, administrative support, and national policies. The cases were in turn analysed by a team of researchers from the USA, Canada, and The Netherlands to identify patterns of practices and relate these to reported outcomes, school characteristics, and national policies (Kozma, 2003). We focus here on just one aspect of this study patterns of innovative classroom practices that are supported by technology. Patterns of classroom practice. The 174 cases were quite evenly divided among primary, lower secondary, and upper secondary grades, with about 35% of the cases at each level. Many cases were in the sciences, with 25% in biological or life sciences, 14% in earth sciences, and 13% in physics. Another 21% were in mathematics. Languages accounted for another large group, with 32% in mother tongue and 24% in foreign languages. The social sciences (14% in geography, 16% in history, and 13% in civics) or creative arts (20%) formed a smaller group. This indicates that across countries ICT has become integrated throughout the curriculum, at least in this set of innovative cases. What kinds of technology did teachers and students use and what role did ICT play in supporting these innovations? A large majority of the innovations used productivity tools (78%), Web resources (71%), and (68%). Many (52%) used multimedia software. Some used Web design tools (34%). Very few used specialised educational software such as simulations and microcomputer-based laboratories (13%). In almost all of the cases (94%) computers were used in regular school settings such as the classroom, library, or computer laboratory. In few cases (28%) technology was used outside of the school. Software packages were used to create products or presentations (80%), Web browsers or CD-ROMs were used to search for information (77%), and was used to support communication (55%). In far fewer cases, teachers used ICT to plan or organise instruction (26%) or to monitor or assess student work (22%). In a small number of cases ICT was used to support student collaboration (17%), or simulations or modelling software packages were used for research or experimentation (13%). Beyond the frequency of various classroom practices, we performed a cluster analysis to examine how patterns of teacher, student, and ICT practices occurred together. Seven meaningful clusters were identified, and these are briefly described in Table II. Two of these clusters are particularly noteworthy. In the cases in the Information Management Cluster, all teachers created structure for students (compared to 80% of the cases overall), and they designed materials in 91% of the cases in this cluster (compared to only 58% overall). In all of the cases,

10 398 E. S. Quellmalz & R. Kozma TABLE II. Description of practices for different clusters Practices Student Collaborative Information Teacher Outside Tool Use Research Management Collaboration Communication Product Creation Tutorial (N 14) (N 14) (N 22) (N 19) (N 27) (N 35) (N 12) Teacher Teachers in this cluster Teachers in this group Teachers in this cluster Teachers often created All the teachers in this Teachers often designed Practices were most likely to give most often designed were most likely to structure, advised cluster created structure materials, frequently in lectures and provide materials and created collaborate with students, and monitored and advised students. collaboration with structure for students. structure for students. colleagues, students, and their progress. They also colleagues. They provided advice and They often provided outside actors. They also collaborated with monitored student students with advice and designed materials, colleagues. progress during student monitored their their created structure for activities. They often progress. They often students, provided them designed materials. collaborated with with advice, and colleagues. monitored their progress. Student Students often Students in this cluster Students in this cluster Students in this group Students in this cluster Students in this cluster Students in this group Practices collaborated with each were most likely to were most likely to were most likely to pick were most likely to were most likely to create were most likely to other to search for collaborate with other search for information, their own tasks. They collaborate with others products. They also engage in drill and information and create students to conduct solve problems, publish also collaborated with outside the class. They collaborated with each practice. products. research and analyse data. results, and assess their each other and others also collaborated with other to search for They also searched for own work and that of outside the class to search other students to conduct information and publish information and solved others. They also for information, create research, search for results. problems. collaborated with other products, and publish information, create students to conduct results. products, and publish research and create results. products. ICT Use Students and teachers in Students and teachers in Teachers in this group Teachers and students in Students and teachers in Students and teachers in All the students in this this group were most this group were most were most likely to use this group were most this group were most this group were among group used tutorial likely to use productivity likely to use Web design course management tools likely to use technology likely to use collaborative those who most often packages. tools and . They tools, multimedia, , and to use technology to to create products and to environments and were used technology to create also used multimedia laptops, and local area plan instruction and use simulations. They among the most frequent products. They also used tools and Web resources. networks. They were monitor student progress. also used productivity users. They most Web resources to search They used technology to most likely to use Teachers and students tools, multimedia, and often used technology to for information and used create products, search technology to simulate were most likely to use . They used the communicate. They used productivity and for information, and research and collaborate. Web resources to search Internet to search for Web resources to search multimedia tools. communicate. They also used Web for information and information and for information, and they resources and productivity tools to communicate with others. used productivity tools. productivity tools. They create products. They used technology to also used multimedia, communicate, search for local area networks, and information, and create to communicate. products. Claimed Students in this cluster Students were more Teachers acquired new Out- were most likely to likely to acquire ICT collaborative skills. comes acquire new ICT, skills, communication problem-solving, and and collaboration skills, collaboration skills. and information handling Teachers acquired new and problem solving pedagogical skills. The skills. Teachers acquired curriculum and class day new pedagogical skills. was more likely to be The curriculum was more reorganised. often reorganized.

11 Designing Assessments of Learning with Technology 399 ICT was used by teachers and/or students to plan and organise instruction (compared to 26% overall) and to create products or presentations (80% overall). ICT was used to monitor or assess students in 86% of the cases in this cluster. This was done in only 22% of the cases overall. In 23% of the cases, ICT course management tools were used, more often than in any other cluster; only 6% of the total cases did so. Productivity tools were used in all cases in this cluster, as they were in several other clusters, including all of the cases in the ICT Tools Cluster. Web resources were also used in 95% of the cases. Students searched for information in all these cases, compared to 74% overall. Students frequently published or presented results (95% compared to 66% overall), solved problems (77% versus 33% overall), and assessed their own and/or each others work (54% versus 30% overall). In one illustrative case from this cluster, an upper secondary school in the USA called Future High School was redesigned from the ground up around technology and project-based learning. The school was organised as a high-tech start-up business in which students were given real world projects consisting of complex tasks with long-range due dates for which they had individual and shared responsibility. Students used productivity tools on a daily basis for everything from research on the Internet to a multimedia integrated design project that combined interdisciplinary content from social studies, maths, science, economics, government, and literature. They created business cards, stamps, posters, letterheads, Web pages, and other integrated multimedia products. Students completed an online portfolio that was assessed by a panel of staff and community members. In the Student Collaborative Research Cluster, students collaborated with others in the class (in all of the cases compared to 83% overall), typically doing research (86% versus 39% overall), and occasionally conducting data analysis (36% versus 22% overall), more so than cases in any other cluster. In all cases, teachers supported their students by giving advice, structuring their activities, and monitoring student progress (compared to 90%, 80%, and 76% overall, respectively). Frequently 71% of the cases in this cluster versus 25% overall teachers lectured or otherwise provided content. In all of the cases, students used local area networks and , frequently used multimedia (86% versus 52%), and occasionally Web design tools (50% versus 34%) and laptops (43% versus 16%), more so than cases in any other cluster. ICT practices occasionally included the use of simulations or modelling to support research and students collaboration 50% of the cases for both practices versus 17% and 13% overall, respectively. A case in the Student Collaborative Research Cluster is a microcomputer-based laboratory in an all-girls secondary school in the Philippines. The goals of the project were to enable the students to learn Physics experientially by gathering data as they conducted experiments using Personal Science Laboratory kits that supported the collection and analysis of data. Teachers wanted to enhance the development of critical thinking skills through hands-on investigation, in-depth verification, exploration, and discovery of science concepts and processes. The teacher s role was to design her own modules, guide the students in setting up parameters for the problems to be investigated, and provide direction when necessary. Teams of students used computers and probe-ware to conduct experiments and

12 400 E. S. Quellmalz & R. Kozma solve a hypothetical murder case. Student performance was assessed using such standard measures as quizzes, periodic examinations, participation in class, as well as ability to use probes in the conduct of experiments. The Information Management and Student Collaborative Research clusters were particularly important because they were more likely to be associated with reported student outcomes than other clusters. For example, cases assigned to these clusters were more likely to report student acquisition of ICT skills (100% and 95.5% respectively), problem solving skills (35.7% and 45.5% respectively), and collaboration skills (78.6% and 90.7% respectively), compared to cases overall (75.3% ICT skills, 18.2% problem solving skills, 63.0% collaboration skills). In addition, cases in the Information Management Cluster were more likely to be associated with student acquisition of communication skills (72.7% versus 39.4% overall) and information handling skills (50.0% versus 28.2% overall). While the outcomes in this study were not directly measured by the research teams, the kinds of impact reported by students, teachers, and administrators indicate the need for novel assessment approaches that can show how technology-supported practices are influencing student learning of these advanced skills. Assessments of Student Learning with Technology Our reviews of research on learning with technology, alignments of technology and subject matter standards and frameworks, and the studies of technology implementation in schools substantiate the centrality of the ICT strategies in the Coordinated ICT Assessment Framework. The framework identifies fundamental uses of technologies to assess. In the school studies, claims of the impacts of technologies on student achievement were based primarily on teacher reports. For our NSF ICT assessment project, these implementation studies were of interest primarily for the types of outcomes claimed to result from classroom activities in which ICT strategies and tools were used. Since a goal of our NSF study was to design innovative ICT assessments that could gather evidence of use of ICT strategies in science and mathematics, we also examined a range of student assessments directly testing technology. We found that most of these tests tended to be techno-centric, testing what students know about the technologies and how to operate them (Crawford & Toyama, 2002). Thus, questions might ask students to identify from a list a function of a spreadsheet or to construct a table with two rows and three columns. To design assessments appropriate for testing the ICT strategies in our Coordinated ICT Assessment framework, our international working group looked to performance assessment formats as the most appropriate way to make students thinking and reasoning with technology visible. The Design of Integrated Performance Assessments in Technology We drew upon cognitive learning research and studies of classroom use of technol-

13 Designing Assessments of Learning with Technology 401 ogy to inform our design for assessments of student learning with ICT. In our NSF ICT assessment project, we extended an innovative performance assessment design approach we had developed in previous SRI national and international projects that aimed to measure students twenty-first century skills. As part of an evaluation of the World Links for Development (WorLD) programme funded by the World Bank, SRI had developed sets of performance assessments requiring use of technology in brief, project-based learning activities (Quellmalz, 1987; Quellmalz & Zalles, 1998). The assessments provided evidence of secondary students proficiencies in technology use, reasoning with information, and communication. A field test of one set of the tasks with two hundred WorLD and non-world students in Uganda revealed advantages in reasoning and communication for WorLD students who had opportunities to experience ICT in their schools (Quellmalz & Zalles, 2002b). The field test also provided data supporting the discriminative validity of the performance assessments and acceptable levels of raters agreement on use of the scoring rubrics. In a related project, prototype online performance assessment tasks designed to assess Internet research skills, reasoning, and communication were developed and pilot-tested with middle school students in the USA (Means et al., 2000; Quellmalz & Zalles, 2002a; Zalles & Yarnall, 2000; The tasks were intended to serve as models that could be used or adapted by evaluators or teachers in US Department of Education programmes such as the Education Technology Challenge Grants and Preparing Tomorrow s Teachers with Technology. The pilot data provided evidence of technical qualities of the prototypes. Their use in evaluation or research projects would require collection of technical quality data appropriate for the purposes of the studies. From these projects emerged the Integrated Performance Assessments in Technology (IPAT) design approach, which extended the problem-based, modular design (Quellmalz & Hinojosa, 2000). A generic IPAT task model poses an engaging, authentic problem and presents modules with sets of tasks and questions requiring students to engage in ICT strategies such as planning, analysing, and communicating. The modular structure of the assessment task components allow modules to be tailored to the examinee population by adding or deleting a module (if students are inexperienced with technologies required in a module, for example) or by varying the task complexity, while maintaining the logical flow of the problem solving activities. One IPAT assessment model presents modules in which students conduct research on the Internet, gather relevant information, use reasoning strategies to analyse and interpret the information, use productivity tools such as graphics programs, word processors, and presentation tools, and communicate findings and recommendations citing Web-based evidence. For our NSF ICT assessment study, we applied the IPAT generic design to develop more extensive prototype ICT assessments. Our assessment development efforts addressed two quite different assessment purposes. One purpose was to develop ICT performance assessments that could be administered as one component of the design proposed for SITES Module 3. That design called for school, teacher, and student surveys of technology use as well as administration of an ICT

14 402 E. S. Quellmalz & R. Kozma performance assessment to a small sample of students in classes that had passed a paper-and-pencil technology screening test that would ensure student familiarity with basic operations of technology tools. These problem-driven ICT performance assessments were developed according to the Coordinated ICT Assessment Framework and designed to test ICT strategies of 13-year-old students who had passed the tool screening test. Assessment of ICT strategies and tools was the focus of these assessments, with science and mathematics problems framing the problem, yet drawing on domain knowledge that was well-taught and well-learned. Based on findings from SITES surveys and case studies, these performance assessments were designed to address ICT strategies employing the most widely used technologies, (e.g., Internet, productivity, and communication tools). A few optional assessment modules were developed to tap ICT strategies involving problems where students could use less common tools (e.g. modelling tools). A second purpose of our NSF project was to draw upon the Coordinated ICT Assessment Framework and the modular design approach to fashion prototype performance assessments that would tap transformative uses of technology in advanced science and mathematics (e.g. visualisations, modelling). These prototypes could be used by teachers and evaluators to assess student learning in innovative technology-supported curricula. Figure 2 portrays how the modular design of the ICT performance assessments can vary the requirements of the subject domain and the ICT strategies and tools (Quellmalz & Hinojosa, 2000). The SITES M3 study design called for low domain complexity (familiar science and maths) and medium to low technology challenges (e.g., common uses of common tools), while the NSF prototypes called for high domain complexity and advanced technology use. We illustrate the modular design approach by describing one prototype ICT performance assessment for middle school students. Table III presents an assessment scenario describing a prototype assessment that would be administered electronically. The problem, Should lynx be re-introduced into a park? is an example FIG. 2.Assessment design options for emphasis on domain and ICT competencies.

15 Designing Assessments of Learning with Technology 403 TABLE III. ICT assessment scenario: predator/prey Problem: Parks are being overrun by hares. The government should re-introduce lynx Science & math content: familiar or given Strategy Module Sample questions/tasks ICT Strategy Component Sample tools 1 Given data in text message of Plan strategies & Analyse problem Spreadsheet 4 years of hare/lynx population procedures Choose Table data, describe the problem. appopriate tools Given data for more years by Collaborate to Integrate others collaborators, describe problem. solve problem data 2 Type in a search to find Access Formulate a Web browser how hare and lynx information search query Table populations are related. & data Look through these three sites. Conduct search Search box Take notes and cite sources. Enter information Search results Copy and paste information. Organise in table or notes Web directory Pick which search might information Evaluate quality Web pages be better. & data of search results Table Are these good search results? Critically evaluate Contribute Word Send suggestions to collaborator. Collaborate feedback document 3 Enter the 25 years of population Represent and Display data in Spreadsheet data into a spreadsheet. transform data one format, Table Create another way to look and information convert to a at the pattern. different form Graph What is the relationship in 2003? Analyse and Record and read interpret data data What trends do you see? Identify and What do you predict will happen explain trends in 5 years? Make predictions 4 Run the model with given settings. Analyse data Read graphs Modelling tool What are the populations in 2002 Interpret data Infer trends Word and 2005? Make predictions processor What do you predict will happen Explain in 2008? predictions Increase the lynx population. What do you think will happen? Run the model. Explain. 5 Plan your recommendation and Plan argument Specify position Web form presentation. Identify relevant Word Compose your presentation evidence processor using information and pictures Communicate Present Tables from Web sites, data. findings & recommendation, Graphs Present argument. supported relevant data, and Graphics argument information in Presentation coherent argument tool 6 Critique recommendation from Critically evaluate Critique position, another team (with inaccurate arguments evidence, support Word data) by explaining if you agree explanation, processor with the recommendation, the organisation appropriateness of their data and informantion, their support for the recommendation.

16 404 E. S. Quellmalz & R. Kozma of predator/prey problems that have been addressed in curricula ranging from the upper elementary to university levels. For this prototype, the science and mathematics required were well-taught, well-learned material. Questions and tasks within modules in this prototype are designed to capture student responses dynamically as they employ ICT strategies to accomplish a sub-task using various technologies. First, the problem is presented and hypothetical student team members from another school who will be virtual collaborative partners are introduced. Module 1 assesses ICT planning strategies through questions and tasks for analysing the problem by examining data on hare and lynx populations, selecting from a set of technology tools. Module 1 assesses collaborative planning through tasks and questions where the student uses to examine hare and lynx population data sent by virtual team members. Evidence of skills in operating the technology tools are by-products of students use of the tools in the problem solving tasks. Assessment of strategies for using technology to access and organise information is tested in Module 2 in a series of tasks where the student formulates a search query, gathers information and data from Web pages, and organises them in a table. Critical evaluation, tested throughout the modules, is assessed by questions on the credibility of information from a Web report produced by a fur trading company and by questions on the effectiveness of Web search results. Module 3 assesses the ICT strategies for using technologies to represent and transform information and data through questions and tasks where students convert data sent in an text message by virtual collaborators to data on a spreadsheet, then transform the data into a graph. Module 4 tests the ICT strategies for using technologies for analysis and interpretation of information and data through questions and tasks asking students to read specified data presented in tables and graphs and to interpret trends. Module 5 tests analysis and interpretation using a modelling tool that displays the pattern of hare and lynx populations. Students answer questions about output of the model at specified years, predict trends, and manipulate population values in the model to test predictions. In Module 6, use of ICT strategies and technologies for planning a presentation and communicating findings and results are tested in questions and tasks asking for a plan specifying a recommendation on re-introducing lynx into the park, supported by data and evidence the student gathered from Web searches. Students are asked to prepare a presentation using a word processor or presentation tool and appropriate graphics and charts. The presentation is evaluated according to the quality of the argument and its organisation. Module 6 also assesses critical evaluation by students of the presentation prepared by another group. Students are asked to prepare a critique and send it to the other group. This Predator/Prey scenario describes one prototype that can be developed using the IPAT modular design. Our project will be pilot testing this assessment and

17 Designing Assessments of Learning with Technology 405 others during Modules can vary the complexity of the problem and the cognitive demands of the tasks and the technologies. Each module has been designed to allow students to select appropriate tools. Modules can be combined in varying ways to accommodate time and logistical constraints. Since the prototypes are delivered over the Web or by CD, the configuration of modules for an administration can be tailored. Student responses to questions and tasks can be captured electronically. Because the responses are recorded as students are in the midst of problem solving, the ICT performance assessments can provide a rich set of data on student problem solving and technology proficiency. As in previous IPAT projects (Quellmalz & Zalles, 2002a), scores can be reported for ICT strategies and for tool use. Rubrics have been developed to rate the quality and appropriateness of student responses. Rater training and online scoring using standard procedures will be used to establish and maintain inter-rater reliability. Furthermore, during the pilot testing of the prototypes, we will conduct think-alouds and subsequent cognitive analyses to examine the thinking and reasoning that students employ related to the ICT strategies, domain content and processes, and operation of technology tools. Additional scenarios and prototypes have been developed for the middle school and secondary levels. A more challenging version of the Predator/Prey module using a modelling tool has been developed for the secondary level. We have also developed prototypes assessing more advanced science and mathematics problems in which advanced technologies such as visualisations and modelling tools are integral. Next Steps Understanding the interrelationships of declarative, procedural, schematic, and strategic reasoning related to the subject domain technologies is a significant issue in the design of assessments of twenty-first century skills. In our NSF ICT study, we are conducting research with the secondary-level ICT prototypes for advanced science and mathematics and advanced tools. We are examining the types of knowledge and reasoning related to the subject matter domain and ICT strategies and tools that are elicited by the assessment modules and revealed through cognitive interviews with students. We are comparing evidence of student learning yielded by the types of questions used in traditional paper-and-pencil tests like those that have been used in related research with the evidence gathered as students are actually using the technologies to solve complex problems. Convergent methods for documenting the validity and reliability of such assessments need to be employed and reported. For example, in an assessment project funded by NSF, The Validities of Science Inquiry Assessments, we are triangulating alignment, empirical, and cognitive analysis studies to examine the extent to which test designs are measuring their intended constructs. Significant issues of test and item design are related to the purpose of the assessment, e.g. summative, formative, or exploratory. The use of assessment data to provide evidence of student learning in intended, enacted, and emerging curricula will also have a significant effect on ICT test design.

18 406 E. S. Quellmalz & R. Kozma Performance standards are another area that will require substantial investigation. There are currently few studies of the development of skilled performance in the use of technologies in significant, complex problems to inform setting proficiency levels at different age and grade ranges. As part of our NSF ICT assessment project, SRI plans to convene a workshop of experts in learning science, assessment, science, mathematics, and technology to address issues related to the design of assessments for the twenty-first century. The workshop will focus experts on issues of test purposes, uses, design, implementation, and technical quality. Such efforts are needed to engage the larger community in developing assessments appropriate for twenty-first century learning environments. NOTE [1] SRI was formerly known as Stanford Research Institute. REFERENCES ADELMAN, N., DONNELLY, M.B., DOVE, T., TIFFANY-MORALES, J., WAYNE, A. & ZUCKER, A. (2002) The Integrated Studies of Educational Technology: professional development and teachers use of technology (Washington DC, SRI International). BACHELARD, G. (1984) The New Scientific Spirit (Boston, MA, Beacon). BAXTER, G. P.& GLASER, R. (1998) The cognitive complexity of science performance assessments, Educational Measurement: issues and practice, 17(3), pp BECKER, H. J.& ANDERSON, R. E. (1998) Teacher s Survey: versions 1 4 (Irvine, CA, Center for Research on Information Technology and Organizations). Available at BRACEWELL, R., BREULEUX, A., LAFERRIERE, T., BENOIT, J.& ABDOUS, M. (1998) The Emerging Contribution of Online Resources and Tools to Classroom Learning and Teaching. Available at BRANSFORD, J., BROWN, A.& COCKING, R.(2000) How People Learn: brain, mind, experience, and school (Washington DC, National Academic Press). BURNS, T. C. & UNGERLEIDER, C. S. (2002) Information and communication technologies in elementary and secondary education, International Journal of Educational Policy, Research and Practice, 3(4), pp COLEY, R. J., CRADLER, J.& ENGLE, P. K.(1999) Computers and Classrooms: the status of technology in U.W. schools (Princeton, NJ, Educational Testing Service, Policy Information Center). Retrieved from CRAWFORD, V. & TOYAMA, Y. (2002) Assessment of student technology proficiency and an analysis of the need for technology proficiency assessments: a review of state approaches. Paper presented at the American Educational Research Association Annual Meeting, New Orleans, LA, April. EUROPEAN COMMISSION (2000) eeuopre: an information society for all (Brussels, European Commission). HOLLAND, J.H. (1995) Hidden Order: how adaptation builds complexity (New York, Addison- Wesley). INTERNATIONAL SOCIETY FOR TECHNOLOGY IN EDUCATION (ISTE) (2000) National Educational Technology Standards for Students: connecting curriculum and technology (Eugene, OR, ISTE). KOZMA, R. (2003) Technology, Innovation, and Educational Change: a global perspective (Eugene, OR, International Society for Educational Technology). KOZMA,R.B.(1991) Learning with media, Review of Educational Research, 61(2), pp

19 Designing Assessments of Learning with Technology 407 MCFARLANE, A. E., HARRISON, C., SOMEKH, B., SCRIMSHAW, P., HARRISON, A.(2000) The Impact of ICT on Attainment. Preliminary Study 1 for the Impact 2 Project. Electronically published report to UK Government: MEANS, B.& OLSON, K.(1995) Technology s Role in Education Reform: findings from a national study of innovating schools (Washington DC, US Department of Education, Office of Educational Research and Improvement). MEANS, B., PENUEL, B.& QUELLMALZ, E. (2000) Developing assessments for tomorrow s classrooms, in: W. HEINEKE & L. BLASI (Eds) Research Methods for Educational Technology (Greenwich, CT, Information Age Publishing). NATIONAL CENTER FOR EDUCATIONAL STATISTICS (2001) Internet Access in U.S. Public Schools and Classrooms: (Washington DC, NCES). NATIONAL COUNCIL OF TEACHERS OF MATHEMATICS (NCTM) (2000) Principles and Standards for School Mathematics (Reston, VA, NCTM). National Science Education Standards (NSES) (1996) (Washington DC, National Academy Press). ORGANISATION FOR ECONOMIC COOPERATION AND DEVELOPMENT (OECD) (2001a) Measuring Student Knowledge and Skills: the PISA 2000 assessment of reading, mathematical, and scientific literacy (Paris, OECD). ORGANISATION FOR ECONOMIC COOPERATION AND DEVELOPMENT (OECD) (2001b) Education Policy Analysis (Paris, OECD). PEA, R.D. (1993) Practices of distributed intelligence and designs for education, in: G. S. SOLOMON (Ed.) Distributed Cognitions: psychological and educational considerations (Cambridge, Cambridge University Press). PELGRUM, W.& ANDERSON, R. (1999) ICT and the Emerging Paradigm for Life Long Learning: a worldwide educational assessment of infrastructure, goals, and practices (Amsterdam, IEA). PELLEGRINO, J., CHUDOWSKY, N.& GLASER, R. (2001) Knowing What Students Know: the science and design of educational assessment (Washington DC, National Academy Press). PERKINS, D.N. (1993) Person-plus: a distributed view of thinking and learning, in: G. S. SOLOMON (Ed.) Distributed Cognitions: psychological and educational considerations (Cambridge, Cambridge University Press). QUELLMALZ, E. S. (1987) Developing reasoning skills, in: J.R. BARON & R.J. STERNBERG (Eds) Teaching Thinking Skills: theory and practice (New York, Freedman Press). QUELLMALZ, E. S.& HAERTEL, G.(2002) Validities of Standards-based Science Inquiry Assessments: design study report (Menlo Park, CA, SRI International). QUELLMALZ, E.S. & HINOJOSA, T. (2000) Technology supported assessment of technology proficiency assessment frameworks. Presented at the American Educational Research Association Annual Meeting, New Orleans, LA, April. QUELLMALZ, E. S.& ZALLES, D.(1998) World Links Student Assessment, Report (Menlo Park, CA, SRI International). QUELLMALZ, E. S.& ZALLES, D. (2002a) Designing Technology Assessments Cognitive-based Modular Design. Presented at the American Educational Research Association Annual Meeting, New Orleans, LA, April. QUELLMALZ, E. S.& ZALLES, D. (2002b) World Links for Development: student assessment Uganda field test, Report to the World Links for Development Organization (Menlo Park, CA: SRI International). SOLOMON, G. S. (1993) No distribution without individuals cognition: a dynamic interactional view, in: G. S. SOLOMON (Ed.) Distributed Cognitions: psychological and educational considerations (Cambridge, Cambridge University Press). SOLOMON, G. S., PERKINS, D. N. & GLOBERSON, T. (1991) Partners in cognition: extending human intelligence with intelligent technologies, Educational Researcher, 20(3), pp WENGLINSKI, H.(1998) Does It Compute? The Relationship between Educational Technology and Student Achievement in Mathematics (Princeton, NJ, ETS). ZALLES, D.& YARNALL, L.(2000) Using online tools to assess students research and communication skills. Paper delivered at the Center for Innovative Learning Technologies Conference. Chevy Chase, MD, October.

20