1 294 Int. J. Learning Technology, Vol. 2, No. 4, 2006 How do you know they are learning?: the importance of alignment in higher education Thomas C. Reeves College of Education The University of Georgia 604 Aderhold Hall Athens GA, , USA Abstract: The success of any learning environment is determined by the degree to which there is adequate alignment among eight critical factors: 1) goals, 2) content, 3) instructional design, 4) learner tasks, 5) instructor roles, 6) student roles, 7) technological affordances, and 8) assessment. Evaluations of traditional, online, and blended approaches to higher education teaching indicate that the most commonly misaligned factor is assessment. Simply put, instructors may have lofty goals, high-quality content, and even advanced instructional designs, but most instructors tend to focus their assessment strategies on what is easy to measure rather than on what is important. Adequate assessment should encompass all four learning domains: cognitive, affective, conative, and psychomotor. This paper describes procedures for the development and use of reliable and valid assessments in higher education. Keywords: assessment; higher education; conative domain; learning technology; e-agenda. Reference to this paper should be made as follows: Reeves, T.C. (2006) How do you know they are learning?: the importance of alignment in higher education, Int. J. Learning Technology, Vol. 2, No. 4, pp Biographical notes: Thomas C. Reeves is a Professor in the Department of Educational Psychology and Instructional Technology in the College of Education at The University of Georgia where he teaches design, evaluation, and research courses. Since receiving his PhD at Syracuse University, he has developed and evaluated numerous interactive learning systems for both education and training. In addition to more than 100 presentations and workshops in the USA, he has been an invited speaker in other countries such as Australia, Belgium, Brazil, Bulgaria, Canada, China, England, Finland, Italy, Malaysia, The Netherlands, New Zealand, Peru, Portugal, Russia, Singapore, South Africa, Spain, Sweden, Switzerland, Tunisia, and Taiwan. He is a former Fulbright Lecturer and the co-founder of the Learning and Performance Support Laboratory at UGA (http://lpsl.coe.uga.edu). In 1995, he was selected as one of the Top 100 people in multimedia by Multimedia Producer magazine, and in 2003, he received the first AACE Fellowship Award from the Association for the Advancement of Computing in Education. Copyright 2006 Inderscience Enterprises Ltd.
2 How do you know they are learning? Introduction In a two-hour television documentary titled Declining by Degrees: Higher Education at Risk, first shown in June 2005 on the Public Broadcasting System (PBS) Network in the USA, John Merrow and his collaborators present a compelling case that Americans do not know enough about the outcomes of higher education in the USA, but that it is convenient for everyone involved (students, professors, parents, alumni, legislators, donors, and the tax-paying public at large) to pretend that high quality teaching and learning are occurring. In the companion book by the same name (Hersh and Merrow, 2005), Schneider (2005, p.62) highlights the problem: Americans are increasingly cynical about their public institutions and public leaders. But their skepticism does not extend to the content of a higher education. Most students and the public as a whole assume without question that whatever students choose to study in college, they will learn what they need to know for today s competitive and complex environment. But in practice, college figures in the public imagination as something of a magical mystery tour. It is important to be admitted; it is also important to graduate with a degree. But what one does in between, what students actually learn in college, is largely unknown and largely unchallenged. In the absence of reliable and accurate information about the outcomes of a higher education, undergraduate students and their parents have increasingly come to rely upon commercial ratings of colleges and university provided by media sources such as US News and World Report and the Princeton Review. However, the criteria factored into the ratings provided by these and other commercial services do not include any meaningful data about student learning or academic achievement. Instead, numerous proxy indicators of the quality of a higher education are used such as the average Scholastic Aptitude Test (SAT) scores of entering freshmen and selectivity as measured by the ratio of students admitted to students applying. As Erhenberg (2005) notes, the administrators of most colleges and universities in the USA claim that they pay little attention to such rankings, but in truth they do. After all, a detailed analysis presented by Zemsky et al. (2005) indicates that what the US News and World Report rankings essentially measure is competitive advantage with respect to attracting the best students. Clearly, the rankings are primarily about inputs rather than outputs. 2 What should undergraduate students learn? Student learning outcomes in both K-12 and postsecondary education are traditionally defined in relationship to three primary domains: cognitive, affective, and psychomotor. The cognitive domain relates to the capacity to think or one s mental skills. As originally defined by Bloom et al. (1956) and revised by Anderson et al. (2001), the cognitive domain has six levels ranging from remembering to creating (see Figure 1). The affective domain (Krathwohl et al., 1964) is about emotions and feelings, especially in relationship to a set of values. It ranges from receiving or becoming aware of stimuli that evoke feelings to manifesting behaviour characterised by a set of consistent and predictable values (see Figure 2). The psychomotor domain (Harrow, 1972) is concerned with the mastery of physical skills ranging from reflexive movements to exhibiting appropriate body language (see Figure 3).
3 296 T.C. Reeves Figure 1 Revised taxonomy of the cognitive domain Creating Evaluating Analysing Applying Understanding Remembering Source: Anderson et al. (2001) Figure 2 Taxonomy of the affective domain Characterising by values Organising and conceptualising Valuing Responding Receiving Source: Krathwohl et al. (1964)
4 How do you know they are learning? 297 Figure 3 Taxonomy of the psychomotor domain Non-discursive communication Skilled movement Physical abilities Perceptual skills Fundamental movement Reflective movement Source: Harrow (1972) None of these domains are completely satisfactory. In addition, it is clear that most instruction in higher education is focused on the cognitive domain rather than the affective or psychomotor domains (Sperber, 2005). Regrettably, even within the cognitive domain much more attention is paid to the lower half of the domain (remembering, understating, and applying) than it is to the arguably more important upper half (analysing, evaluating, and creating). This problem stems largely from the relative ease with which the skills encompassed in the lower half can be taught and tested within most fields or disciplines. Teaching and assessing the cognitive skills required for analysis, evaluation, and creation takes more time and effort than many, if not most, university instructors feel they have. Even more regrettable is the fact that an entire domain is ignored by most instructors in academe today. This is the conative domain (Snow et al., 1996). The cognitive domain is concerned with thinking, the affective with valuing, the psychomotor with skilled behaviour, and the conative domain is associated with action. It is clear that while an individual may possess the cognitive capacity, affective values, and physical skills to perform a given task, whether he or she possesses the will, desire, drive, level of effort, mental energy, intention, striving, and self-determination to actually perform at the highest standards possible remains an unanswered question. The conative domain focuses on conation or the act of striving to perform at the highest levels. With rare exceptions, the literature on higher education teaching, learning, and assessment is not informed by consideration of the conative domain. Ironically, the roots of conation can be traced all the way back to Aristotle who used the Greek word orexis to signify striving, desire, or the conative state of mind. Kolbe (1990) contrasted the cognitive, affective, and conative domains as illustrated in Figure 4.
5 298 T.C. Reeves Figure 4 Kolbe s (1990) comparison of cognitive, affective, and conative domains Cognitive Affective Conative To know Thinking Thought Epistemology Knowing To feel Feeling Emotion Esthetics Caring To act Willing Volition Ethics Doing Given the increasingly global nature of competition (Friedman, 2005), the higher education graduate of the 21st century can ill afford to enter the world of work without the opportunity to develop expertise across all four domains of learning. Figure 5 illustrates a comprehensive array of the learning domains that every college graduate should possess. Unfortunately, very few colleges and universities collect evidence that their graduates leave their institutions with comprehensive achievement across the four domains. Indeed, when the National Center for Public Policy and Higher Education (2000), an independent, non-profit, non-partisan organisation, compiled a report titled Measuring Up 2000: The State-by-State Report Card for Higher Education, Centre personnel found that they could find data to allow state-by-state comparisons of higher education institutions with respect to preparation, participation, affordability, completion, and benefits, but not for learning. Figure 5 Comprehensive learning outcomes for 21st century college graduates Cognitive capacity to think, problem-solve, and create Affective capacity to value, appreciate, and care Conative capacity to act, decide, and commit Psychomotor capacity to move, perceive, and apply physical skills
6 How do you know they are learning? Learning assessment should be the driving force in higher education Regardless of the field or discipline, there are a set of meta-outcomes that cut across the four domains. The following list, while not exhaustive, suggests the scope of these outcomes: accessing and using information communication skills using multiple media demonstrating understanding accompanied by deep reflection applying rules and procedures to structured and unstructured problems being creative thinking critically making sound judgments problem solving being committed to life-long learning exhibiting intellectual curiosity proactively seeking to extend knowledge in one s discipline exhibiting ethical behaviour. These kinds of meta-outcomes will not be achieved in higher education unless they are assessed. This conclusion is supported by strong evidence that if something is not assessed in higher education, then it is not learned (Bain, 2004). It may appear to be somewhat cynical to state that assessment drives learning, but it does. Perhaps it is just human nature, but college students choose to focus their study efforts on the things on which they know they will be tested and graded (Napoli and Raymond, 2004). Most students come to recognise that they can get good grades by cramming for tests and then quickly forgetting what they have memorised to allow themselves to focus on other pursuits. Most instructors are familiar with the typical questions asked by undergraduate students in higher education courses such as Are we responsible for knowing that material? or Is this content going to be on the test?. Instructors recognise the desire of students to focus their study efforts, minimal as they are, on accumulating the knowledge that will be tested, and in far too many cases, instructors give into these wishes rather than pushing their students to achieve at higher levels. When John Merrow, the producer of the Declining by Degrees was asked what was the most shocking discovery he found when he was visiting various US institutions of higher education for his television documentary, he responded: NOT that students are binge drinking, NOT that athletics is a business, and NOT that most students don t seem to have to work very hard to get good grades, because we knew those things. What came as a surprise was what one of our experts calls the non-aggression pact between professors and students. It amounts to an unspoken compact: don t ask too much of me, and I won t expect much from you. This allows the faculty members to concentrate on what their institution values: publications, research and getting grants. And it means that students get good grades and can float though college with plenty of
7 300 T.C. Reeves time for socializing, networking and other activities. Few complain, even though to an outsider it s pretty clear that the emperor has no clothes. That came as a shock. Few professors design their learning assessments on the assumption that testing and grading are not incidental acts that come at the end of teaching but powerful aspects of education that have an enormous influence on the entire enterprise of helping and encouraging students to learn (Bain, 2004, p.150). If we want our college graduates to possess the 21st century skills outlined above, assessment must focus on these higher order types of outcomes. Accordingly, university and college teachers must devote much more effort to the task of assessment because it is the lifeblood of good teaching. This is no easy task. According to a recent special supplement in The Chronicle of Higher Education (Blumenstyk, 2006, p.b30), more than 40 percent of entering college students [need] to take at least one remedial course in mathematics, reading, or writing. There is ample evidence that American students are graduating from high school without the academic preparation required to engage in higher order learning and assessment. When this fact is combined with the growing pressure on faculty members in research universities to fund and conduct research and on instructors in teaching universities and colleges to teach more students with fewer resources, higher education certainly does seem to be declining by degrees (Hersh and Merrow, 2005). 4 The millennial learner: myth or reality Interestingly, at a time when people both within and outside academe are beginning to question the assumed high quality of American higher education, several pundits and even a few scholars are predicting that a new kind of student is entering our institutions of higher learning that may save the day. According to Raschke (2002, p.68), Colleges and universities are about to be beset by a new generation of learners whose skills and expectations derive from growing up on the net. Over the past few years, much has been written in both popular and scholarly literature about the generation of students entering higher education today called by various monikers such as Generation Y, Millennials, and the Net Generation (Coomes and DeBard, 2004; Howe and Strauss, 2000; Oblinger, 2003; Oblinger and Oblinger, 2005; Tapscott, 1998). Prognosticators such as Howe and Strauss (2000, p.4) who wrote today s teens are recasting the image of youth from downbeat and alienated to upbeat and engaged have predicted that the Americans born between 1982 and 2000 constitute the next greatest generation that will out achieve previous generations such as the Baby Boomers (born ) and Generation X (born ). However, the evidence for such optimism appears to be largely drawn from surveys and focus groups conducted with young people living in affluent suburbs, sometimes in the presence of their parents. Twenge (2006) uses empirical data collected over the past 50 years to dismiss most of the optimistic claims about Millennial students. Meanwhile, the findings of the National Survey of Student Engagement (NSSE) conducted by Indiana University paint a very different story about the current generation of college undergraduates. Conducted every year since 2000 and involving nearly 500 colleges and universities in the USA, NSSE indicates that undergraduate students are much less engaged in learning activities known to foster academic achievement than expected by their professors (Kuh, 2003). The average professor expects undergraduate
8 How do you know they are learning? 301 students to be engaged in classes or labs hours per week and out-of-class studying for another hours per week. This does not seem like an unreasonable expectation, but the NSSE data shows that 20% of students spend less than five hours per week studying, 25% spend 6 10 hours per week, 48% spend hours per week, and only 7% exceed the 30 hours per week expected by faculty members. The researchers involved in the NSSE studies point to five essential strategies for increasing student engagement (Kuh et al., 2004): 1 increasing student faculty interaction 2 engaging students in active, collaborative learning activities 3 encouraging more achievement-oriented time-on-task among students 4 setting high academic challenge 5 providing continuous timely feedback. All of these strategies have implications for assessment, but especially the last two. Unless professors raise the level of the objectives they are trying to achieve and encompass all four learning domains in their assessments, students will not be compelled to become more academically engaged. In addition, unless professors are willing to become more engaged in high quality teaching themselves, especially with respect to providing continuous timely feedback through better assessment strategies, any increased student interest in academic engagement will diminish quickly. 5 Enhancing assessment in higher education Bain (2004) describes how the best teachers in higher education focus their teaching assessment activities on critical thinking, problem solving, creativity, curiosity, concern for ethical issues as well as breadth and depth of specific knowledge and the methodologies and standards of evidence used to create that knowledge (pp.8 9). In addition, they use assessment to help students learn, not just rate and rank their efforts (p.151). Pellegrino et al. (2001, p.44) describe an assessment triangle (see Figure 6) wherein: the corners of the triangle represent the three key elements underlying any assessment a model of student cognition and learning in the domain, a set of beliefs about the kinds of observations that will provide evidence of students competencies, and an interpretation process for making sense of the evidence. The cognitive corner is based upon a theory of learning underpinning the set of knowledge, skills, attitudes, and habits of mind to be measured. For example, if an instructor subscribes to a constructivist theory of learning (Fosnot, 1996), his or her assessments should focus on cognition related to the production of original representations of knowledge as opposed to the regurgitation of textbook knowledge. The observation corner refers to the tasks or performances that instructors design to provide evidence that students have learned. The nature of this evidence should be linked to the cognitive learning theory in such a way to support the decisions derived from the assessment results. Again, within a constructivist learning theory, an instructor is likely to observe the learner s learning through the analysis of tangible phenomena such as
9 302 T.C. Reeves portfolios, products, or performances. The interpretation corner expresses how the observations derived from a set of assessment tasks constitute evidence about the knowledge and skills being assessed (Pellegrino et al., 2001, p.48). With a constructivist learning theory in mind, an instructor might design a rubric that expresses levels of quality for the portfolio, product, or performance being assessed (Miller, 2005). Figure 6 The assessment triangle Observation Interpretation Source: Pellegrino et al. (2001) Cognition Obviously, no assessments are perfect, all failing to provide completely reliable and valid evidence of student learning. Some inference is required because assessments inevitably oversimplify the full capacity students possess to perform across all four domains of learning. As Pellegrino et al. (2001, p.49) clarify: A crucial point is that each of the three elements of the assessment triangle not only must make sense on its own, but also must connect to each of the other two elements in a meaningful way to lead to an effective assessment and sound inferences. This implies the need for alignment, not just within an assessment but between assessment and the other critical factors that define a learning environment. 6 The importance of alignment The success of any learning environment, including e-learning, is determined by the degree to which there is adequate alignment among eight critical factors: 1) goals, 2) content, 3) instructional design, 4) learner tasks, 5) instructor roles, 6) student roles, 7) technological affordances, and 8) assessment. Evaluations of traditional, online, and blended approaches to postsecondary teaching indicate that the most commonly misaligned factor is assessment (Reeves and Hedberg, 2003). Simply put, instructors may
10 How do you know they are learning? 303 have lofty goals, share high-quality content, and even utilise advanced instructional designs, but most assessment strategies tend to focus on what is easy to measure rather than what is important. Figure 7 Critical factors that must be aligned in any learning environment Lower order, discrete One right answer Direct instruction Academic Focus on teaching Passive orientation Prepackaged static data Nature of objectives Nature of content Pedagogical dimensions Learner tasks Instructor roles Student roles Technology roles Focus of assessment Higher order, general Multiple perspectives Problem-based learning Authentic Focus on learning Active engagement Real world, real time data Measuring lower level cognition Assessing all four domains Figure 7 illustrates the eight critical factors that must be aligned within a learning environment for it to be effective. Alignment cannot be over-emphasised. If an undergraduate course is designed based upon a constructivist learning theory (Fosnot, 1996), the remaining factors must be in alignment with the pedagogical design. A description of these factors follows: 1 Nature of objectives the knowledge, skills, attitudes, and intentions that students should develop as a result of participating in the course. Objectives are ideally stated as measurable outcomes ranging from discrete knowledge (e.g., students will be able to identify distinguishing properties of a phenomenon) to higher order thinking (e.g., students will exhibit a robust mental model of related systems). 2 Nature of content the information and data that encompass the subject matter to be taught, studied, and learned in the course. More often than not in undergraduate courses, content is presented in highly structured formats such as textbooks, but content can also be accessed in ill-structured, real-world formats such as original historical documents or scientific data from remote sensors. 3 Instructional design the overall arrangement of activities, resources, structure, and procedures that an instructor designs to promote learning. Traditional instructional designs employed in higher education are focused on teacher talk (through lecture), static content (through textbooks), and fixed assessment (through tests seeking one right answer). While such designs are ubiquitous, as described above, the evidence
11 304 T.C. Reeves for the efficacy of traditional instructional methods is unacceptably weak. Alternative instructional designs include problem-based learning, project-based learning, authentic tasks, and service learning among others (Fink, 2003). 4 Learner tasks the strategies used to engage students in meaningful learning activities. The NSSE studies (Kuh, 2003) point out that students are often inadequately engaged by traditional academic tasks such as writing term papers or cramming for multiple-choice tests of lower level cognitive knowledge. By contrast, there is great potential for undergraduate learners to be engaged by authentic tasks (Herrington and Oliver, 2000) such as conducting real world inquiry or service learning (Kezar, 2002). 5 Instructor roles the learning support (sometimes called scaffolding) functions the instructor will provide while students are engaged in learning. Instructors accustomed to a didactic teaching approach wherein they deliver prepackaged information to students in the form of lectures and assigned readings may struggle with the necessity of allowing their students to grapple with the inevitable complexities of authentic tasks or service learning. Bain (2004) describes how the best teachers surrender some of their power as experts and become co-learners with their students. 6 Student roles the active cognitive, psychomotor, affective, and conative (Snow et al., 1996) interactions in which students will engage as they grapple with authentic tasks, dynamic content, peers, instructors, and other components of the learning environment. Students accustomed to more passive roles in the college lecture hall may initially resist the active requirements of constructivist pedagogy. The best learning environments often require collaboration and teamwork, and students who resist working in groups may balk. Resistance to changing roles may be especially strong among the students most often rewarded with high grades within the traditional teacher-text-test-centred pedagogy. 7 Technological affordances the cognitive tools, visualisations, simulations, role-playing games, and other interactive resources provided by technologies such as computers with internet access. An affordance is the interaction possibilities posed by objects in the real or cyber worlds. Thinking of technology as a cognitive tool is an especially effective strategy in higher education (Jonassen and Reeves, 1996). 8 Assessment strategies the methods used to estimate student accomplishment of the course objectives. Historically, this has been one of the weakest aspects of both traditional and innovative course design and implementation in higher education (Shipman et al., 2003). In most undergraduate courses, assessment and grading are usually based upon multiple-choice tests or academic essays. In an authentic learning environment (Herrington and Oliver, 2000), assessment is based upon observations of student engagement and analysis of artefacts produced in the process of completing the tasks. Rather than using just one method, robust assessment requires the critical analysis of multiple forms of evidence that learning outcomes have been attained.
12 How do you know they are learning? 305 Failure to align these eight dimensions will undermine the successful design and implementation of an undergraduate course, regardless of whether it is offered in a classroom, online, or via a blended model. The efficacy of any one or a few of these factors cannot be evaluated in isolation from the others. Alignment is essential. If an instructor specifies higher order outcomes for a course, every effort must be made to assess those outcomes in the most reliable, valid, and feasible manner possible. If the learner tasks are appropriately complex and challenging, the instructor must assemble the necessary real world and technological resources to scaffold students as they strive to accomplish those tasks. Although in general higher education practice, innovative approaches to teaching are more the exception than the rule, there are good examples in the higher education literature of undergraduate courses where an appropriate level of alignment has been reached (Bain, 2004). But the weakest component of most designs is assessment, perhaps because both instructors and students are so accustomed to thinking of assessments in traditional ways. 7 Steps toward enhancing assessment in higher education Better assessment and enhanced alignment in college teaching and learning will require a larger investment in assessment and evaluation than most institutions are expending at this time. For starters, administrators and faculty alike must distinguish assessment from evaluation. Although the terms assessment and evaluation are commonly used interchangeably, they have two distinct meanings. Assessment is defined as the activity of measuring student learning and other human characteristics such as aptitude whereas evaluation is focused on judging the effectiveness and worth of educational programmes, practices, and products. It may help to think that people are assessed and things are evaluated. Of course, the terms are often confused because the same data can be used for both assessment and evaluation. For example, final exams are commonly utilised to assess student learning in courses and provide the basis for grading. The same examination results, however, could be used in the evaluation of the effectiveness of a course, e.g., if an institution wanted to compare classroom and online versions of the same course. Enhancing assessment will also require higher education institutions to stop hiding the results of the few national and international evaluations in which they participate. Mathews (2005, p.57) points out that most colleges participating in NSSE have refused to release the results of those surveys, preferring to use them for internal decisions on which courses need to be improved or dropped. This is a risky policy because as Kuh (2001, p.12) points out Sooner or later, colleges and universities are either going to demonstrate what students are learning or some external entity will impose its own approach. Indeed, there is increasing discussion among legislators and policymakers at the federal and state levels in the USA of extending the assessment practices currently used in K-12 education under the No Child Left Behind Act to undergraduate education (Spellings, 2005). Fortunately, if institutions and their adherents sincerely wish to enhance how they conduct and use assessment, there is a body of knowledge that can help (Angelo and Cross, 1997; Huba and Freed, 1999; Wiggins, 1998). Assessment has many purposes in
13 306 T.C. Reeves higher education ranging from narrow, formative ones to broad, summative ones (Quellmalz and Kozma, 2003). Terenzini (1989) recommends that the development of student assessments should be guided by three major questions: 1 What is the purpose of the assessment, e.g., formative (to provide information for helping a student learn) or summative (rendering a judgment of a student s achievement)? 2 What is the scope of the assessment, e.g., a quiz administered to a single class immediately after a lecture, or an external assessment of the graduates of a university programme such as a bar examination for law students? 3 What is being assessed, e.g., knowledge, skills, attitudes, mental models, or higher order thinking capacities? Of course, additional questions must be addressed in any assessment activity related to issues such as reliability, validity, feasibility, and security. Some faculty members may require assistance from instructional designers and assessment experts to free themselves from continuing to over rely on traditional assessments such as multiple-choice tests and term papers, but most can begin to make improvements in their assessment practices on their own or in small groups of faculty from the same or related disciplines. Excellent guidance is available in resources such as Angelo and Cross (1997) and Wiggins (1998). Angelo and Cross (1997, pp.7 11) describe the following important assumptions about assessment: The quality of student learning is directly, although not exclusively, related to the quality of teaching. Therefore, one of the most promising ways to improve learning is to improve teaching. To improve their effectiveness, teachers need first to make their goals and objectives explicit and then to get specific, comprehensive feedback on the extent to which they are achieving those goals and objectives. To improve their learning, students need to receive appropriate and focused feedback early and often; they also need to learn how to assess their own learning. The type of assessment most likely to improve teaching and learning is that conducted by faculty to answer questions they themselves have formulated in response to issues or problems in their own teaching. Systematic inquiry and intellectual challenge are powerful sources of motivation, growth, and renewal for college teachers, and classroom assessment can provide such challenge. Classroom assessment does not require specialised training, it can be carried out by dedicated teachers from all disciplines. By collaborating with colleagues and actively involving students in classroom assessment efforts, faculty (and students) enhance learning and personal satisfaction.
14 How do you know they are learning? Conclusion For the past 50 years, US higher education has been held up as the best in the world, but that ranking is increasingly being questioned both within and outside the USA (Bok, 2005; Hersh and Merrow, 2005). This is nowhere more evident than in the increasing global competition with respect to online learning (Katz, 1999; Pittinsky, 2003; Reeves, 2003). Technology is presenting higher education with opportunities for rapid and radical change, but decisions about making such changes may be made on the basis of politics or expediency rather than careful evaluation and rational decision-making (Reeves and Hedberg, 2003). Already some critics have suggested that universities and colleges may do well to unbundle their assessment functions from their teaching roles (Baer, 2000; Paulson, 2002). It is clear that colleges and universities and their faculty members are being challenged as never before to answer the question How do you know they are learning? If nothing else, hopefully this paper will inspire its readers to address this question through better assessment and enhanced alignment of teaching and learning. References Anderson, L.W. (Ed.), Krathwohl, D.R. (Ed.), Airasian, P.W., Cruikshank, K.A., Mayer, R.E., Pintrich, P.R., Raths, J. and Wittrock, M.C. (2001) A Taxonomy for Learning, Teaching, and Assessing: A Revision of Bloom s Taxonomy of Educational Objectives, Complete edition, New York: Longman. Angelo, T.A. and Cross, K.P. (1997) Classroom Assessment Techniques: A Handbook for College Teachers, 2nd ed., San Francisco: Jossey-Bass. Baer, W.S. (2000) Competition and collaboration in online distance learning, Information, Communication & Society, Vol. 3, No. 4, pp Bain, K. (2004) What the Best College Teachers Do, Cambridge, MA: Harvard University Press. Bloom, B.S. (Ed.), Engelhart, M.D., Furst, E.J., Hill, W.H. and Krathwohl, D.R. (1956) Taxonomy of Educational Objectives: The Classification of Educational Goals. Handbook 1: Cognitive Domain, New York: David McKay. Blumenstyk, G. (2006) Businesses have remedies for sale, but a cure is not guaranteed, The Chronicle of Higher Education, 10 March, pp.b30, B32. Bok, D. (2005) Our Underachieving Colleges: A Candid Look at How Much Students Learn and Why They Should be Learning More, Princeton, NJ: Princeton University Press. Coomes, M.D. and DeBard, R. (Eds.) (2004) Serving the Millennial Generation: New Directions for Student Services, San Francisco: Jossey-Bass, Vol Ehrenberg, R.G. (2005) Method or madness? Inside the USNWR college rankings, Journal of College Admission, Vol. 189, pp Fink, L.D. (2003) Creating Significant Learning Experiences: An Integrated Approach to Designing College Courses, San Francisco: Jossey-Bass. Fosnot, C. (1996) Constructivism: Theory, Perspectives, and Practice, New York: Teachers College Press. Friedman, T.L. (2005) The World is Flat: A Brief History of the Twenty-First Century, New York: Farrar, Straus and Giroux. Harrow, A.J. (1972) A Taxonomy of the Psychomotor Domain: A Guide for Developing Behavioral Objectives, New York: David McKay.
15 308 T.C. Reeves Herrington, J. and Oliver, R. (2000) An instructional design framework for authentic learning environments, Educational Technology Research and Development, Vol. 48, No. 3, pp Hersh, R.H. and Merrow, J. (2005) Declining by Degrees: Higher Education at Risk, New York: Palgrave Macmillan. Howe, N. and Strauss, W. (2000) Millennials Rising: The Next Great Generation, New York: Vintage Books. Huba, M.E. and Freed, J.E. (1999) Learner-Centered Assessment on College Campuses: Shifting the Focus from Teaching to Learning, Boston: Allyn and Bacon. Jonassen, D.H. and Reeves, T.C. (1996) Learning with technology: using computers as cognitive tools, in D.H. Jonassen (Ed.) Handbook of Research on Educational Communications and Technology, New York: Macmillan, pp Katz, R.N. (1999) Dancing with the Devil: Information Technology and the New Competition in Higher Education, San Francisco: Jossey-Bass. Kezar, A. (2002) Assessing community service learning: are we identifying the right outcomes?, About Campus, Vol. 7, No. 2, pp Kolbe, K. (1990) The Conative Connection: Acting on Instinct, Reading, MA: Addison-Wesley. Krathwohl, D.R., Bloom, B.S. and Masia, B.B. (1964) Taxonomy of Educational Objectives: The Classification of Educational Goals. Handbook II: The Affective Domain, New York: David McKay. Kuh, G.D. (2001) Assessing what really matters to student learning, Change, Vol. 33, No. 3, pp.10 17, 66. Kuh, G.D. (2003) What we re learning about student engagement from NSSE, Change, Vol. 35, No. 2, pp Kuh, G.D., Laird, T.N. and Umbach, P. (2004) Aligning faculty activities and student behavior: Realizing the promise of greater expectations, Liberal Education, Vol. 90, No. 4, pp Mathews, J. (2005) Caveat lector: unexamined assumptions about quality in higher education, in R.H. Hersh and J. Merrow (Eds.) Declining by Degrees: Higher Education at Risk, New York: Palgrave Macmillan, pp Miller, R. (2005) Integrative learning and assessment, Peer Review, Vol. 7, No. 4, pp Napoli, A.R. and Raymond, L.A. (2004) How reliable are our assessment data?: a comparison of the reliability of data produced in graded and un-graded conditions, Research in Higher Education, Vol. 45, No. 8, pp National Center for Public Policy and Higher Education (2000) Measuring up 2000: The State-by-State Report Card for Higher Education, San Jose, CA: National Center for Public Policy and Higher Education. Oblinger, D.G. (2003) Boomers, gen-xers, and millennials: understanding the new students, EDUCAUSE Review, Vol. 38, No. 4, pp Oblinger, D. and Oblinger, J. (Eds.) (2005) Educating the Net Gen, Washington, DC: EDUCAUSE. Paulson, K.W. (2002) Reconfiguring faculty roles for virtual settings, Journal of Higher Education, Vol. 73, No. 1, pp Pellegrino, J.W., Chudowsky, N. and Glaser, R. (Eds.) (2001) Knowing what Students Know: The Science and Design of Educational Assessment, Washington, DC: National Academy Press. Pittinsky, M.S. (Ed.) (2003) The Wired Tower: Perspectives on the Impact of the Internet on Higher Education, Upper Saddle River, NJ: Prentice Hall. Quellmalz, E. and Kozma, R. (2003) Implications of technology for learning and assessment, Assessment in Education, Vol. 10, pp Raschke, C. (2002) The Digital Revolution and the Coming of the Postmodern University, New York: RoutledgeFalmer.
16 How do you know they are learning? 309 Reeves, T.C. (2003) Storm clouds on the digital education horizon, Journal of Computing in Higher Education, Vol. 15, No. 1, pp Reeves, T.C. and Hedberg, J.G. (2003) Interactive Learning Systems Evaluation, Englewood Cliffs, NJ: Educational Technology Publications. Schneider, C.G. (2005) Liberal education: slip-sliding away, in R.H. Hersh and J. Merrow (Eds.) Declining by Degrees: Higher Education at Risk, New York: Palgrave Macmillan, pp Shipman, D., Aloi, S.L. and Jones, E.A. (2003) Addressing key challenges in higher education assessment, Journal of General Education, Vol. 52, No. 4, pp Snow, R.E., Corno, L. and Jackson, D. (1996) Individual differences in affective and conative functions, in D.C. Berliner and R.C. Calfee (Eds.) Handbook of Educational Psychology, New York: Macmillan, pp Spellings, M. (2005) A strategic vision for higher education, Presidency, Vol. 8, No. 2, pp Sperber, M. (2005) How undergraduate education became college lite and a personal apology, in R.H. Hersh and J. Merrow (Eds.) Declining by Degrees: Higher Education at Risk, New York: Palgrave Macmillan, pp Tapscott, D. (1998) Growing up Digital: The Rise of the Net Generation, New York: McGraw Hill. Terenzini, P.T. (1989) Assessment with open eyes: pitfalls in studying student outcomes, The Journal of Higher Education, Vol. 60, No. 6, pp Twenge, J.M. (2006) Generation Me: Why Today s Young Americans are More Confident, Assertive, Entitled and More Miserable than Ever Before, New York: Free Press. Wiggins, G. (1998) Educative Assessment: Designing Assessments to Inform and Improve Student Performance, San Francisco: Jossey-Bass. Zemsky, R., Wegner, G.R. and Massy, W.P. (2005) Remaking the American University: Market-Smart and Mission-Centered, Piscataway, NJ: Rutgers University Press.
Applying and Promoting Critical Thinking in Online Education Hisham Al-Mubaid 1 1 University of Houston - Clear Lake, Houston, USA Abstract Critical thinking is a mental process that involves high quality
A Primer on Writing Effective Learning-Centered Course Goals Robert K. Noyd (DFB) & The Staff of The Center for Educational Excellence (CEE) US Air Force Academy A Shift from a Topic-Centered to a Learning-Centered
Curriculum Development in 6 easy steps for busy Med-Ed types Joanne Lynn MD Curriculum Development in MedEd Medical educators are often charged to plan educational experiences without specific training
Integrating Field Research, Problem Solving, and Design with Education for Enhanced Realization Mohamed El-Sayed Kettering University Mechanical Engineering Flint, MI, 48504 USA and Jacqueline El-Sayed
Abstract Limbach, Barbara Chadron State College Waugh, Wendy Chadron State College This paper identifies an interdisciplinary, five-step process, built upon existing theory and best practices in cognitive
Charting Your Course: Instructional Design, Course Planning, and Developing the Syllabus Danielle Mihram, Ph.D. Faculty Fellow and Director USC Center for Excellence in Teaching firstname.lastname@example.org Originally
Learning Objectives: Making Sure it Links Kimberly K. Daugherty, Pharm.D., BCPS Assoicate Professor and Assistant Dean of Academic Affairs and Assessment SU Faculty Retreat November 2012 Objectives Given
Preparing tomorrow s lawyers today: graduate attributes in first year law Dr Michelle Sanson First Year Coordinator, Law School University of Western Sydney Abstract Graduate attributes are not only pedagogically
2 We are all accustomed to periodic summative testing for achievement, but in this chapter the importance of requiring mastery of fundamentals as the baseline for evaluation, regardless of the level of
Improving the Educational Quality of Liberal Arts Colleges by Charles F. Blaich and Kathleen Wise February 2014 Summary A large body of research points to a core set of teaching practices and institutional
Running head: PERSONAL STATEMENT ON LEARNING AND INSTRUCTION 1 Personal Statement on Learning and Instruction Jay A. Bostwick IP&T 620 Principles of Learning PERSONAL STATEMENT ON LEARNING AND INSTRUCTION
Essays on Teaching Excellence Toward the Best in the Academy A Publication of The Professional & Organizational Development Network in Higher Education Vol. 19, No. 5, 2007-2008 Building Assignments that
UDL and Course Design at AL$ workshop Elizabeth Tu Center for Faculty Development June 13 & 14, 2012 Agenda Overview of Agenda The Universal Design for Learning (UDL) Framework and Your Instructional Issue
MISSISSIPPI STATE UNIVERSITY COLLEGE OF EDUCATION DEPARTMENT of LEADERSHIP & FOUNDATIONS COURSE SYLLABUS Course Prefix and Number: CCL 8153 Course Title: Credit hours: Type of course: Catalog description:
Engaging Students for Optimum Learning Online Informing the Design of Online Learning By the Principles of How People Learn What Is Engagement? As early as 1995, student engagement was "the latest buzzword
The affective domain and social networking: definitorial issues and misleading assumptions Christian Voigt 1,2, Kit MacFarlane 1 1 University of South Australia, Adelaide, Australia 2 Centre for Social
Restructuring a Masters Teaching Program Marilyn Koeller National University This article will explain the process that Course Leads used to restructure the Masters in the Arts of Teaching program by working
2010 2011-2012 Edition LoTi Mentor Certification Institute LMC Online Course Syllabus The glass ceiling gets more pliable when you turn up the heat. Pauline R. Kezer American Teacher and Legislator LMC
DEFINING, TEACHING AND ASSESSING LIFELONG LEARNING SKILLS Nikos J. Mourtos Abstract - Lifelong learning skills have always been important in any education and work setting. However, ABET EC recently put
Essays on Teaching Excellence Toward the Best in the Academy Volume 17, Number 3, 2005-06 A publication of The Professional & Organizational Development Network in Higher Education (www.podnetwork.org).
IDEA PAPER #42 Integrated Course Design L. Dee Fink University of Oklahoma Teaching is a complex human action. The many tasks that are involved comprise four general components: Knowledge of the subject
Center for Pre-College Programs New Jersey Institute of Technology Cloud Computing Workshop Standards-Based Lesson Planning Howard Kimmel Kimmel@adm.njit.edu John Carpinelli Carpinelli@adm.njit.edu and
Developing higher level thinking Barbara Limbach Chadron State College Wendy Waugh Chadron State College Abstract This paper identifies an interdisciplinary, five-step process, built upon existing theory
University of Wisconsin-Platteville Course Syllabus for Academic Credit Curriculum and Course Construction #50 Wisconsin Technical College System (WTCS) Course Title: Curriculum and Course Construction
GEORGIA STANDARDS FOR THE APPROVAL OF PROFESSIONAL EDUCATION UNITS AND EDUCATOR PREPARATION PROGRAMS (Effective 9/01/08) Kelly Henson Executive Secretary Table of Contents Standard 1: Candidate Knowledge,
Essays on Teaching Excellence Toward the Best in the Academy Volume 17, Number 7, 2005-06 A publication of The Professional & Organizational Development Network in Higher Education (www.podnetwork.org).
Teaching principles of management through experiential and service learning ABSTRACT Omid Furutan University of La Verne Management faculties often use cases, simulations, and research projects to achieve
Supporting the Implementation of NGSS through Research: Curriculum Materials Janet Carlson, BSCS/Stanford University Elizabeth A. Davis, University of Michigan Cory Buxton, University of Georgia Curriculum
Title: Transforming a traditional lecture-based course to online and hybrid models of learning Author: Susan Marshall, Lecturer, Psychology Department, Dole Human Development Center, University of Kansas.
Bill Cerbin UW La Crosse, Center for Advancing Teaching & Learning Prioritize the subject matter What subject matter topics, ideas, concepts are 1. Essential 2. Important 3. Worth being familiar with Use
Assessment of Graduate Programs: Clear, Simple, and USEFUL Barbara E. Walvoord, Ph.D. Professor Emerita University of Notre Dame Notre Dame IN 46556 Walvoord@nd.edu Mobile: 574-361-3857 1 SACS Standards
Converting a Face-to-Face Course to a Hybrid Course A research paper by Stephanie Delaney For College Teaching EDAD 966 Introduction Hybrid courses blend the best of face-to-face classes with online learning.
Professor Julie Hall email@example.com Napa Valley College Flex Day Fall 2012 Practical Steps & Strategies to Engage Students in Critical Reflective Thinking Samples of Reflective Prompts & Question
Implementing E-Learning Designed Courses in General Education Prasart Nuangchalerm 1, Krissada Sakkumduang 2, Suleepornn Uhwha 3 and Pacharawit Chansirisira 4 1 Department of Curriculum and Instruction,
Competencies and Learning Objectives * The purpose of this paper is to support faculty of schools of public health/public health programs Programs (SPH/PHP) as they consider approaches and make decisions
Essays on Teaching Excellence Toward the Best in the Academy Volume 13, Number 1, 2001-02 A publication of The Professional & Organizational Development Network in Higher Education (www.podnetwork.org).
University Of North Dakota SBHE Policy 403.1. & 404.1 In accordance with SBHE Policy 403.1, Program Approval; and 404.1, Distance Learning Credit activities, UND seeks on-going approval for on-campus and
Implementing E-Learning Designed Courses in General Education Prasart Nuangchalerm 1, Krissada Sukkhamduang 2, Suleeporn Uhwa 2, Pacharawit Chansirisira 3 1 Department of Curriculum and Instruction, Faculty
Learner Centered Education in Online Classes Paula Garcia Center for E-learningE Walter Nolan CTEL and College of Education What is learner centered education? An approach to teaching and learning that
COURSE CATALOG Contents Introduction... 3 Delta Courses... 4 The College Classroom... 4 The College Classroom: International Students, International Faculty... 4 Diversity in the College Classroom... 4
STATE UNIVERSITY OF NEW YORK COLLEGE OF TECHNOLOGY CANTON, NEW YORK COURSE OUTLINE PSYC 350 EDUCATIONAL PSYCHOLOGY Prepared By: John T. Nixon, Ph.D. Reviewed and modified by Edward Boyd April 2015 SCHOOL
0085 LTA Reconceptualising Critical Thinking Critical Thinking and Higher Education The development of critical thinking is widely claimed as a primary goal of higher education (e.g., Barnett, 2000; Halpern,
Assessing Teaching Skills in Higher Education Regulations How the demand for demonstrated teaching skills should be handled when appointing or promoting teachers in higher education is regulated in Higher
Evaluating Students in the Classroom 2007 Faculty Development Conference February 2007 D. P. Shelton, MSN, RN, CNE, EdD (c) Objectives Describe assessment strategies for use in the classroom. Explore the
Degree Program Student Learning Report (rev. 7/14) Fall 2013 Spring 2014 The Department of Applied Technology in the School of Business & Technology Business Information Technology, B.S. Effectively assessing
6th WSEAS International Conference on E-ACTIVITIES, Tenerife, Spain, December 14-16, 2007 263 Implementing Information Technology in the Learning Process SEPPO SIRKEMAA Pori Unit Turku School of Economics
APPENDIX Revised Bloom s Taxonomy In 1956, Benjamin Bloom and his colleagues published the Taxonomy of Educational Objectives: The Classification of Educational Goals, a groundbreaking book that classified
CONSUMERS OF ONLINE INSTRUCTION Dr. Lillie Anderton Robinson, North Carolina A&T State University, firstname.lastname@example.org ABSTRACT There is a great demand for the online delivery system of instruction. It accommodates
Florida Association of Teacher Educators Journal Volume 1 Number 4 2004 1-10 http://www.fate1.org/journals/2004/almerico1.pdf Bloom s Taxonomy Illustrative Verbs: Developing a Comprehensive List for Educator
Guide to Writing MBA Program and Course Learning Outcomes and Assessment that Align with QFEmirates Level 9 Descriptors Commission for Academic Accreditation United Arab Emirates May 2015 1 Contents Part
AC 2011-1304: INTEGRATION OF HUMANITIES AND SOCIAL SCIENCES INTO CIVIL ENGINEERING EDUCATION Jeffrey C. Evans, Bucknell University Jeffrey Evans is Professor and Chair of the Department of Civil and Environmental
p e d a g o g y s t r a t e g y MCEETYA A u s t r a l i a N e w Z e a l a n d Pedagogies that integrate information and communication technologies can engage students in ways not previously possible, enhance
Supporting the General Education Program for Distance and Adult Students Jodene DeKorte, PhD Kaplan University Jdekorte@kaplan.edu Kathleen Ingram, PhD Kaplan University email@example.com Jon Proctor,
Selecting and Scoring Appropriate Assessment Strategies Lynne E. Houtz, Ph.D. Creighton University Beverly A. Doyle, Ph.D. Creighton University Assessment is defined as, the systematic collection, review
UMD Department of Mechanical and Industrial Engineering Indices and Standards for Tenure and Promotion to Professor as Required by Section 7.12 of the Board of Regents Policy on Faculty Tenure (June 10,
National Framework for Values Education in Australian Schools National Framework for Values Education in Australian Schools Commonwealth of Australia 2005 ISBN: 0 642 77496 X ISBN: 0 642 77497 8 (online
f ce.in the SchooLof LiberalArt Public Administration, B.A. Effectively assessing a degree program should address a number of factors: ---------"Lj-Valid_studenLieaming cutcomes shouidbe clearl'i articulated;
Text of article appearing in: Issues in Science and Technology, XIX(2), 48-52. Winter 2002-03. James Pellegrino Knowing What Students Know Recent advances in the cognitive and measurement sciences should
1 of 5 17TH Annual Conference on Distance Teaching and Learning Activities to Engage the Online Learner Rita-Marie Conrad, Ph.D. Assistant Professor Florida State University Ana Donaldson, Ed.D. Assistant
Essays on Teaching Excellence Toward the Best in the Academy Volume 8, Number 2, 1996-97 A publication of The Professional & Organizational Development Network in Higher Education (www.podnetwork.org).
Programme Specification Where appropriate outcome statements have be referenced to the appropriate Benchmarking Statement (BS) 1 Awarding Institution Queen Margaret University 2 Teaching Institution Queen
Clarifying Misconceptions between Technology Education and Educational Technology Only you will assume the responsibility to educate others about the importance of technology education in schools today.
Management Fundamentals in Healthcare Organizations University of Minnesota School of Public Health LEARNING MODEL The learning model underlying the Management Fundamentals Certificate is an application
The Domain of Instructional Design In simple terms, instructional design is the process of asking questions to determine an appropriate solution for an instructional problem. During instructional design
Writing Effective Learning Goals and Objectives Office of Medical Education Creighton University School of Medicine Last Updated July 2015 Session objectives After completing this session the learner should
6th WSEAS International Conference on EDUCATION and EDUCATIONAL TECHNOLOGY, Italy, November 21-23, 2007 110 Using Formative Writing Assignments to Enhance Student Learning in a First Year Chemistry Class
Linking Competencies and Curriculua Presentation to the Faculty and Students of National Tsing Hua University Richard A. Voorhees, Ph.D. Principal, Voorhees Group LLC Littleton, Colorado USA firstname.lastname@example.org
The Fifth International Conference on e-learning (elearning-2014), 22-23 September 2014, Belgrade, Serbia TEACHING AND LEARNING IN COMPETENCY-BASED EDUCATION NEIL O SULLIVAN Innovative Together Ltd, email@example.com
Competencies and Learning Objectives * The purpose of this paper is to support faculty of schools of public health/public health programs (SPH/PHP) as they consider approaches and make decisions for competency-based
Master of Science, Curriculum and Instruction The Master of Science degree in Curriculum and Instruction is a competency-based program and represents a path for K-12 educators and corporate trainers wishing
Eleventh LACCEI Latin American and Caribbean Conference for Engineering and Technology (LACCEI 2013) Innovation in Engineering, Technology and Education for Competitiveness and Prosperity August 14-16,
TABLE OF SPECIFICATION DEVELOPMENT AND USAGE: AN OVERVIEW Dr Aijaz Ahmed Federal Urdu University Aziz-un-Nisa University of Karachi Dr Tayyaba Zarif Newports Institute of Communications and Economics Abstract
MSU Departmental Assessment Plan 2009 2010 Department: Education Department Head: Dr. Joanne Erickson Assessment Coordinator: Bill Freese 214 Reid Hall 994 3072 Degrees/Majors/Options Offered by Department
Study Guide for the Mathematics: Proofs, Models, and Problems, Part I, Test A PUBLICATION OF ETS Table of Contents Study Guide for the Mathematics: Proofs, Models, and Problems, Part I, Test TABLE OF CONTENTS
A Guide to Curriculum Development: Purposes, Practices, Procedures The purpose of this guide is to provide some general instructions to school districts as staff begin to develop or revise their curriculum
Assessment and Evaluation in Teacher Education Renewal Calvin M. Frazier A generalist steps lightly in the testing and assessment arena. As a generalist and one specializing in state policymaking, my tread
1 20th Annual Conference on Distance Teaching and Learning Future of E-learning in Higher Education and Training Environments Curtis J. Bonk, Professor Departments of Educational Psychology and Instructional
Integrating Critical Thinking into the Business Curriculum Linda L. Miles, Ph.D. Dean of the School of Business Colgate Darden Professor of Finance Chowan University Andrew T. Babyak, Ph.D. Assistant Professor
THEORY INTO PRACTICE / Autumn 2002 Revising Bloom s Taxonomy David R. Krathwohl A Revision of Bloom s Taxonomy: An Overview THE TAXONOMY OF EDUCATIONAL OBJECTIVES is a framework for classifying statements
Clarifying Misconceptions between Technology Education and Educational Technology Only you will assume the responsibility to educate others about the importance of technology education in schools today.
Volume 8, Number 10 April, 2005 ISSN 1099-839X Teaching Students with Learning Disabilities: Constructivism or Behaviorism? Marcee M. Steele University of North Carolina Wilmington There is much controversy
IDEA PAPER #49 Effective Classroom Discussions 1 William E. Cashin, professor emeritus Kansas State University The prototypic teaching method for active learning is discussion. Svinicki and McKeachie (2011,
Master of Arts in Teaching (M.A.T.) Application Forms Packet As a part of the application process for admission to the M.A.T. program at the Graduate School of the College of Charleston, we request that
HONORS INDEPENDENT STUDY GUIDELINES: According to the Lake Land College catalog course description, independent study is designed to permit the student to pursue a course of study not typically available
An Examination of Assessment Methods Used in Online MBA Courses Shijuan Liu Richard Magjuka Xiaojing Liu SuengHee Lee Kelly Direct Programs, Indiana University Curtis Bonk Department of Instructional Systems
B.A. in Education Specialization: Early Childhood Education (P-3) Student Handbook Rowan University College of Education Teacher Education Department 1 Table of Contents Program Description 3 ECED Program
Your consent to our cookies if you continue to use this website.