Evaluation of Two Environmental Science Web-based Learning Environments. Margaret Thayer ED TECH Fall 2010 December 9, 2010

Transcription

1 Evaluation of Two Environmental Science Web-based Learning Environments Margaret Thayer ED TECH Fall 2010 December 9, 2010

2 Contents Learning Reflection... 4 Executive Summary... 5 I. Purpose of the Evaluation... 6 A. Introduction and Purpose... 6 B. Evaluation Questions... 6 C. Impact of Evaluation Results... 7 II. Background Information... 8 A. Rationale... 8 B. Goals... 8 C. Existing Instructional Materials... 9 D. Program Personnel and Development... 9 E. Characteristics of the WBLEs III. Description of Evaluation Design A. Evaluation Model B. Evaluation Criteria C. Learner Survey IV. Results V. Discussion of the Results A. Congruence Analysis B. Content Analysis C. Design Analysis D. Feasibility Analysis E. Overall Decision VI. Conclusions and Recommendations A. Immediate Conclusions B. Long-Range Planning C. Evaluation Insights References Appendix A: AP Environmental Science Exam Themes Appendix B: AP Environmental Science Topic Outline Appendix C: Survey Instructions P age

3 Appendix D: WBLE Design Evaluation Criteria: Operation: Resilient Planet Appendix E: WBLE Design Evaluation Criteria: The Habitable Planet Appendix F: WBLE Content Evaluation Criteria: The Habitable Planet Appendix G: Learner Survey Results P age

4 Learning Reflection Through my work in this course, I have developed several insights about evaluation: 1. Every evaluation is a unique situation that requires a unique treatment. Evaluators should glean insight from many different sources and not be locked into a particular process or method. 2. Evaluations must be conducted within the context and purpose of the evaluation situation. My final evaluation project was based on a hypothetical situation, which made it difficult to determine the finer details of the context and purpose that would have so much bearing on how I would design the evaluation. 3. The most productive approach to designing an educational evaluation is to start with the classic instructional design principles. I was most appreciative of Carey and Dick s chapters on formative and summative evaluations, because they do such a terrific job of striking a balance between academic concepts and pragmatic considerations. During the course of my final evaluation project, I had to remind myself that I was performing an assessment, not trying to prove the worth of the products I was evaluating. I really wanted both of the WBLEs in my project to be outstanding and suitable for the purpose of the evaluation, because I like to think that well-designed, interactive, Web-based instructional materials have something to add to education beyond traditional instructional materials. But I had to proceed with a systematic evaluation process and let the evaluation results speak for themselves. The Operation: Resilient Planet WBLE is quite appealing, but I could have rejected it as a candidate for evaluation once I had developed my hypothetical evaluation scenario. However, I realized it was simply not wise to try to conduct an evaluation of two full programs/courses (each containing weeks or even months of material) within the time frame of the project. I would have been much better off limiting my evaluation to The Habitable Planet, which has its virtues but lacks the pizzazz of Operation: Resilient Planet. That s the power of first impressions! I also think that evaluations are best conducted by more than one person. It is easy to get too close to the material and lose sight of the purpose and the next direction. Multiple perspectives would be valuable for any evaluation situation. I m not yet certain what to take from my final project experience of designing and conducting a survey for teenage learners. I required the participants to obtain permission from a parent or guardian to participate in the survey, because I did not know how parents might react to a stranger soliciting their child s participation in this activity. I don t know if the permission requirement had any effect on the low response rate. I am not yet certain how I will use my education in evaluation in my career as an instructional designer. I am now almost two months into my first job as an instructional designer. My company creates Web-based courses for all different types of clients and purposes, but formal evaluations are never discussed. Our work is based on what each client wants and is willing to pay for, and we do our best to combine those parameters with solid instructional design principles. Learning outcomes are discussed only anecdotally. I do think my education in evaluation reinforces the virtue of applying solid instructional design principles to any form of instruction. 4 P age

5 Executive Summary Two Web-based learning environments (WBLEs), Operation: Resilient Planet and The Habitable Planet: A Systems Approach to Environmental Science, were evaluated for their potential use as instructional materials in preparation for the Advanced Placement (AP) environmental science course and exam. The evaluation was initiated by the AP advisory committee at Green Planet Virtual School (GPVS). Operation: Resilient Planet is an interactive multimedia learning environment that sends participants (who are members of Team Argonaut ) on virtual missions, hosted by real-world science researchers, to study the ecological health of selected ecosystems around the world. The program was designed to align with the National Science Education Standards for grades 5 through 8 with an emphasis on the principle of science as inquiry. The Habitable Planet: A Systems Approach to Environmental Science is a linear 13-unit college-level course with multimedia and interactive elements. The course was designed as a professional development learning tool for science educators and for other undergraduatelevel adult learners. The content of both WBLEs is available for free online with optional print/dvd versions available for purchase; The Habitable Planet also requires a small per-student or per-institution licensing fee. The evaluation methodology consisted of two elements: a criterion-referenced evaluation and a survey of some members of the target learner group. The criterion-referenced part of the evaluation assessed the two WBLEs in two key areas: (1) Web-based design/usability characteristics, and (2) applicability to the AP environmental science exam content. The evaluation survey was intended to assess the motivational level of both WBLEs for the target learners. The purpose of the evaluation was to assess both WBLEs for their use as self-study learning tools by high school students who are preparing for the AP environmental study exam with limited assistance by online instructors; the evaluation did not seek to select one WBLE over the other. The results of this evaluation suggest that Operation: Resilient Planet, although extremely appealing and engaging for its intended audience, is unsuitable for the purpose of this evaluation; however, The Habitable Planet course is potentially suitable and motivating for use as instructional material in the AP environmental science program at GPVS. 5 P age

6 I. Purpose of the Evaluation A. Introduction and Purpose The Advanced Placement (AP) advisory committee at Green Planet Virtual School (GPVS) is developing the curriculum for a new AP environmental science program at the school. The committee engaged the services of Thayer Evaluations Inc. (TEI) to conduct a summative evaluation of two interactive Web-based Learning Environments (WBLEs) on the topic of environmental science. The committee initiated this evaluation for the purpose of determining whether one or both of the WBLEs could be recommended for use as instructional materials by high school students who are preparing for the AP environmental science exam. The two WBLEs considered are Operation: Resilient Planet, which was produced by the JASON Project, a nonprofit subsidiary of The National Geographic Society; and The Habitable Planet: A Systems Approach to Environmental Science, produced by Annenberg Media and the Harvard Smithsonian Center for Astrophysics in association with the Harvard University Center for the Environment. The advisory committee selected these two programs for evaluation because both are award winners: Operation: Resilient Planet was the 2009 CODiE winner for Best Science Instructional Solution, and The Habitable Planet was awarded a 2009 Science Magazine/AAAS SPORE (Science Prize for Online Resources in Education). The advisory committee would like to know if one or both of these WBLEs would be suitable for self-study by students who are preparing for the AP exam with limited guidance from online instructors. Instructional design experts Lou Carey and Walter Dick developed guidelines for conducting summative evaluations based on instructional design principles. Their summative evaluation guidelines specifically address situations in which organizations must analyze instructional materials for the purpose of deciding whether to adopt the materials. They suggest that a summative evaluation should include two phases: an expert judgment phase in which the evaluator determines whether candidate materials have the potential to meet the instructional needs of the organization, and a field trial phase in which the evaluator assesses the candidate materials effectiveness with target group members. Carey and Dick note that the expert judgment phase is typically used to select one or two sets of promising materials for evaluation in the field trial phase (Carey & Dick, 1991, pp ). This report covers phase 1, the expert judgment phase, of this evaluation; as such, its goal is to determine whether one or both of the candidate WBLEs should be recommended for phase 2, a field trial evaluation to assess their effectiveness with members of the target learner group. Phase 1 of the evaluation uses a criterion-referenced approach to assess two key aspects of the two WBLEs: (1) their congruence with the subject matter of the AP environmental science exam, and (2) their suitability for use by high school students primarily for self-study in preparation for the AP environmental science exam. This evaluation does not consider the general worth of the two WBLEs as educational materials, but instead focuses on their applicability as self-study materials for the AP exam. B. Evaluation Questions Carey and Dick (1991, p. 281) suggest that the expert judgment phase of a summative evaluation should address the following five decision points: 1. Overall Decision: Do the materials have the potential for meeting this organization s needs? 2. Congruence Analysis: Are the needs and goals of the organization congruent with those in the instruction? 3. Content Analysis: Are the materials complete, accurate, and current? 4. Design Analysis: Are the principles of learning, instruction, and motivation clearly evident in the materials? 5. Feasibility Analysis: Are the materials convenient, durable, cost-effective, and satisfactory for current users? 6 P age

7 By applying the above five decision points to this evaluation, the following key questions emerged for consideration about each of the two candidate WBLEs: 1. Overall Decision: Does the candidate WBLE have the potential for being suitable as self-study materials for high school students who are preparing for the AP exam with limited guidance from online science teachers? 2. Congruence Analysis: Is the WBLE suitable for the target learners? If not, what modifications would be necessary to make the WBLE suitable for these learners? 3. Content Analysis: To what extent does this WBLE cover the concepts and topics included in the AP environmental science exam? 4. Design Analysis: Does the WBLE motivate students to engage in the material? 5. Feasibility Analysis: Is this WBLE suitable for use by individual students with limited guidance from an instructor? C. Impact of Evaluation Results The results of this evaluation are expected to impact the following people: Students at GPVS who are studying for the AP environmental science exam Parents of GPVS environmental science students Members of the AP advisory committee Online AP environmental science instructors and instructional designers at GPVS Administrators and staff at GPVS Students at GPVS will be impacted by this evaluation if Web-based, AP-level environmental science materials are made available to them as an alternative to the traditional printed textbook, particularly if the Web-based materials offer more interactivity; the increase in interactivity may also demand greater initiative on the part of the students and perhaps their parents. If the advisory committee wishes to add either of the WBLEs to the school s AP instructional materials list, the committee will need to seek approval by the College Board, which administers the AP program. Environmental science instructors and instructional designers at GPVS would need to make adjustments to lesson plans and teaching methodologies. Finally, GPVS administrators and staff would need to consider the impact on program costs, resources, and administrative needs. 7 P age

8 II. Background Information A. Rationale The subject of environmental studies has been taught at GPVS for the school s entire (though brief) two-year existence, but the school administrators recently decided to add an AP environmental science course to the school curriculum. The school s AP advisory committee determined that the existing environmental studies materials would be inadequate for use in the AP course, so the committee initiated a process to evaluate instructional materials for the new AP environmental science course. The AP program is administered by the nonprofit College Board ( to provide academically motivated high school students with challenging college-level courses in more than 30 subjects. Students also have the potential to earn college credit by taking the AP exam associated with each AP course, so AP students and instructors tend to focus their learning and instruction around preparation for the exam. The College Board authorizes a school to use the AP course label after the course instructor participates in an AP course audit; AP instructors are required to develop their own AP course curriculum. The AP exams are administered in May every year and graded by college faculty and AP expert teachers. The AP environmental science course is intended to be the equivalent of a one-semester introductory college course that focuses on environmental science, including a laboratory component, rather than environmental studies. Students are more likely to be successful in the course if they have already completed two years of high school laboratory classes (life science and physical science) and one year of algebra. The AP environmental science exam is a three-hour test of 100 multiple choice questions and four free-response/problem-solving questions. The latter four questions include one dataset question, one document-based question, and two synthesis-and-evaluation questions. The College Board established six themes for the environmental science course; these themes are reprinted in Appendix A. In addition, the College Board established a detailed outline of topics that are covered in the environmental science exam; this outline is reprinted in Appendix B. The AP advisory committee at GPVS chose Operation: Resilient Planet and The Habitable Planet WBLEs for evaluation not only for their potential as college-level instructional materials for the AP environmental science course, but also to fulfill the school s mission of providing students with an enriching, technology-supported learning environment. Neither of these WBLEs was developed for use with the AP environmental science course, so the advisory committee was most interested in determining, during this first phase of the evaluation, the extent to which each WBLE covered the AP environmental science course topics and breadth of information. B. Goals In developing the new AP environmental science program at GPVS and selecting suitable instructional materials for the course, the AP advisory committee seeks to meet the College Board s stated goal for the AP environmental science course: The goal of the AP Environmental Science course is to provide students with the scientific principles, concepts, and methodologies required to understand the interrelationships of the natural world, to identify and analyze environmental problems both natural and human-made, to evaluate the relative risks associated with these problems, and to examine alternative solutions for resolving or preventing them. (Reprinted from the College Board s AP environmental science course description, 2010) Additional committee goals are as follows: 8 P age

9 1. To provide AP environmental science students at GPVS with instructional materials that encompass all of the topics outlined by the College Board for the AP environmental science exam. 2. To provide AP environmental science students with an enriching, interactive, technologysupported learning environment in which to study environmental science. 3. To establish the foundational curriculum for the new AP environmental science program at GPVS. C. Existing Instructional Materials As noted in the College Board s AP environmental science course description, environmental science is a multidisciplinary subject that integrates material from many different areas of study, from biology, chemistry, and physical sciences to geography and geology. In addition, the subject is greatly influenced by public policy decision-making processes. The subject of environmental studies concentrates on this latter aspect rather than the scientific components that are the focus of the AP environmental science program. The existing instructional materials used at GPVS for its environmental studies curriculum consist primarily of environmental public policy and current events related to environmental issues, with a cursory overview of the science behind the issues. Thus, they do not include the depth and breadth of the environmental science component demanded of the AP curriculum. The existing materials at GPVS would be suitable for use to highlight the real-world applicability of environmental science. The AP environmental science course description points out the importance of selecting a suitable textbook to cover the course syllabus. It also highlights the importance of using a recently published textbook due to the rapidly changing nature of environmental science and suggests that textbooks cannot be exhaustive (College Board, 2010, p. 5). Nevertheless, it is notable that the 2010 AP environmental science course description does not specifically mention the potential utility of using Webbased resources, which can be updated more frequently than printed textbooks and can provide interactive elements that permit students to practice science. The College Board s AP Central site ( includes a Teacher s Resources section that is searchable by course content and resource medium (book, lesson plan, Website, etc.), and many Websites are listed among the environmental science resources. Although these Web resources may be valuable, none appear to be comprehensive; the sites tend to revolve around particular topics such as oceans or species biodiversity or laboratory exercises. D. Program Personnel and Development The implementation of the AP environmental science program at GPVS is largely under the direction of the AP advisory committee. The committee consists of nine members: five AP instructors from various subject areas, one instructional designer, one member from the school s program administrative staff, and two parents of current AP students. The committee functions in an advisory capacity to school instructors, administrators, and parents, and represents GPVS in communications with the College Board s AP program. The committee solicited recommendations for potential WBLEs for use in the new environmental science course from all the GPVS instructors who might have a stake in the environmental science curriculum. From among these recommendations, Operation: Resilient Planet and The Habitable Planet emerged as the best candidates. The AP environmental science course will be coordinated by one instructor, and the course is expected to enroll approximately 18 students for its inaugural year. However, due to the multidisciplinary nature of environmental science, enrolled students will participate in some of their AP environmental science coursework in conjunction with other classes and instructors. GPVS students and instructors use the Moodle learning management system (LMS) as a portal for all course communications and coordination. 9 P age

10 E. Characteristics of the WBLEs The two WBLEs considered in this evaluation are described below: 1. Operation: Resilient Planet Operation: Resilient Planet is one of several interactive/multimedia science education programs produced by the JASON Project, a nonprofit subsidiary of National Geographic. The JASON Project was founded by Dr. Robert Bob Ballard, an explorer-in-residence at the National Geographic Society. Dr. Ballard is known for his use of remotely operated vehicles, which he used to explore the wreck of the RMS Titanic. The JASON Project is named for the mythological Greek explorer. All Operation: Resilient Planet resources are available for free online at and a print/dvd version is also available for purchase. Operation: Resilient Planet was developed for grades 5 to 8, but the program description indicates that it can be modified for older students. The program provides at least five to nine weeks of material. The program was designed to align with the National Science Education Standards (NSES) for grades 5 through 8, with a special emphasis on Standard A: Science as Inquiry. Operation: Resilient Planet sends students on five missions to investigate the ecological health of selected ecosystems around the world. Each mission is hosted by a real-life science researcher selected from the National Oceanic and Atmospheric Administration (NOAA), NASA, and National Geographic. To participate in the program, students register online to access the JASON Mission Center, the core resource of the program. All registered students and teachers are Argonauts; each year, researchers and students are selected to be a part of Argonauts teams, which lead research activities within the five program missions. The program includes lesson plans and extensions, teacher resources, a 144-page teacher manual, multimedia resources, digital labs and games, an online communication center, a getting started student tutorial, a 120-page student magazine, and the Operation: Resilient Planet Game a 3D underwater adventure game. Onsite training workshops and seminars are available for teachers who wish to use the JASON curriculum in their classrooms. Other teacher resources include a customizable online lesson builder and assignment manager, and a customizable assessment builder. The JASON Project also offers five-week online graduate courses in science and pedagogy. 2. The Habitable Planet: A Systems Approach to Environmental Science The Habitable Planet is an interactive multimedia course produced by Annenberg Media and the Harvard Smithsonian Center for Astrophysics in association with the Harvard University Center for the Environment. All course materials are available at Although the entire course is available for free online, the Annenberg Media Website FAQs indicate that distance learning licensing fees apply if an institution uses a course for credit or non-credit. Flat-fee licenses are offered on a per term, per year, and three-year basis. Individual students can license the material for $45. A print/dvd version of the course is also available for purchase. The Habitable Planet course was developed for high school teachers, undergraduate students, adult learners, and advanced high school students. The course description indicates that the curriculum is not suitable for high school classrooms, but the materials can be used to supplement the regular curriculum. The online course consists of 13 units that each include a half-hour video, an online textbook (PDF files), five interactive labs, animations and simulations, visuals, and a glossary. Each unit was produced by a leading science academic (many from Harvard University) selected by the course producers. Students and instructors could expect to spend at least one week per unit to fully explore all the materials for each unit. The Habitable Planet course is linear in structure with a straightforward menu on the left side of the screen that permits learners to access any unit within the course at any time. Learners can view/read the course materials at their own pace. Each unit includes a list of additional resources. A professional development guide is available as PDF files for instructors. 10 P age

11 III. Description of Evaluation Design A. Evaluation Model This evaluation was conducted based on the Decision-Making Model (Boulmetis & Dutwin, 2005, p. 89). This model, originally developed by Daniel Stuffelbeam and colleagues as the CIPP Evaluation Model, is used in situations that require a decision about the future use of a program. Carey and Dick (1991, p. 270) describe this model as an accountability type of evaluation that they placed among the foundations of summative evaluation. The model does not prescribe a specific data collection methodology, so any number of data collection tools may be used. In this model, CIPP = context, input, process, and product. Each of these evaluation concepts includes a number of variables. The variables used in this evaluation as applied to the decision-making process are shown in Table 1. The CIPP concepts and variables in Table 1 reflect the information needed to answer the evaluation questions presented in Section 1B of this report. Table 1. Decision-Making (CIPP) Model variables for consideration in this evaluation.* CIPP Evaluation Concept Context Input (external and internal) Process Product Purpose Identify needs and set goals Prescribe a course of action by which to make changes Analyze implementation of the product Measure, interpret, and judge outcomes Variables Learner needs Program goals and resources Criteria to judge outcomes Potential approaches System capabilities (equipment and facilities requirements) Cost Accuracy, currency, and durability of materials Appropriateness of materials for target group Time for completion Instructional activities and procedures (group vs. individual) Instructional strategy of product Measurement of outcomes (learner attitudes) Data analysis Comparison of outcomes to program/product needs * Criteria were identified based on discussions of this model in Stuffelbeam (2003) and Carey & Dick (1991). B. Evaluation Criteria The WBLEs in this evaluation were assessed using a criterion-referenced methodology that considered the WBLEs overall design and usability as Web-based resources, and their applicability to the AP environmental science course and exam. The evaluation criteria were developed primarily from two sources described below. 1. Design Criteria. A rubric developed by Baya a, Shehade, and Baya a (2009) was used to evaluate the WBLEs in four main categories: usability, content, educational value, and vividness (a measure of the currency of resources and links). Twenty evaluation criteria were considered within these four main categories. The criteria were used as an overall assessment of each WBLE s presentation and design of the content. 11 P age

12 2. Content Criteria. The outline of topics for the AP environmental science exam, presented in the AP course description (College Board, 2010), was used to assess the extent to which the WBLE content applies to the AP exam content. The evaluation criteria were assessed using a four-item Likert scale. Points were assigned to each scale item where Strongly Agree=4, Somewhat Agree=3, Somewhat Disagree=2, and Strongly Disagree=1. The points were totaled and averaged for all the main categories within the usability and content criteria areas. These main categories of criteria are summarized in Table 2. Table 2. Summary of evaluation criteria and variables. Evaluation Criteria Design Usability Content Educational Value Vividness Content Earth Systems and Resources The Living World Population Land and Water Use Energy Resources and Consumption Pollution Global Change Summary of Variables Considered Purpose, Homepage, Navigation, Design, Enjoyment, Readability Authority, Accuracy, Relevance, Sufficiency, Appropriateness Learning Activities, Activity Plan, Resources, Communication, Feedback, Rubric, Help Tools Links, Currency Earth Science Concepts, The Atmosphere, Global Water Resources and Use, Soil and Soil Dynamics Ecosystem Structure, Energy Flow, Ecosystem Diversity, Natural Ecosystem Change, Natural Biogeochemical Cycles Population Biology Concepts, Human Population Dynamics, Population Size, Impacts of Population Growth Agriculture, Feeding a Growing Population, Controlling Pests, Forestry, Rangelands, Other Land Use, Urban Land Development, Transportation Infrastructure, Public and Federal Lands, Land Conservation Options, Sustainable Land-Use Strategies, Mining, Fishing, Global Economics Energy Concepts, Energy Consumption, History, Present Global Energy Use, Future Energy Needs, Fossil Fuel Resources and Use, Nuclear Energy, Hydroelectric Power, Energy Conservation, Renewable Energy Pollution Types (Air, Noise, Water, Solid Waste), Impacts on the Environment and Human Health, Hazards to Human Health, Hazardous Chemicals in the Environment, Economic Impacts Stratospheric Ozone, Global Warming, Loss of Biodiversity (Habitat Loss; Overuse; Pollution; Introduced Species; Endangered and Extinct Species, Maintenance Through Conservation) To rate both WBLEs on the criteria and variables listed in Table 2, the TEI evaluator visited both sites several times over several weeks from October to December The initial visits served to generate a first impression assessment of each WBLE based on ease of navigation and visual appeal. The AP content criteria required much more purposeful visits to determine the extent to which the content was covered. Since this evaluation was not a competition between the two sites, the evaluation of each WBLE was done entirely in reference to the criteria, goals, purpose, and evaluation questions outlined in this report; the two sites were not compared with each other. C. Learner Survey In addition to the criteria evaluation, TEI gathered survey feedback from members of the target learner group to gauge whether the WBLEs would be motivational to and suitable for the age group. Participation instructions were sent to ten 10th-grade students at an International Baccalaureate (IB) school with a strong AP program, including an environmental science course. Since the survey questions were designed to elicit feedback about the motivational aspects of the two WBLEs for 12 P age

13 academically motivated high school students but not specifically about the WBLEs applicability to the AP exam content, the target learners included some students who were enrolled in the AP environmental science course and some who were not. Since both WBLEs in this evaluation have an extensive amount of content that would require weeks or months of study, the participants were asked to review only certain parts of the content and provide feedback about in an 18-item online survey. Participants were expected to spend a total of about 40 to 50 minutes reviewing the two WBLEs and completing the survey, but they were instructed to spend as much time reviewing additional content as they wanted. The participants were also asked about their typical use of and attitudes toward Web-based educational materials. The survey instructions given to the target learners are shown in Appendix C. 13 P age

14 IV. Results Although the two WBLEs were evaluated from the same initial approach, the two evaluations took somewhat different directions. Operation: Resilient Planet is a dynamic and appealing site, but initial assessments of the content and delivery cast serious doubts about whether the WBLE could pass the Congruence Analysis question: Are the needs and goals of the organization congruent with those in the instruction? Consequently, Operation: Resilient Planet was rejected from contention before a detailed content assessment was performed. In contrast, the detailed assessment of content revealed some important information about The Habitable Planet. The complete results of the WBLE design criteria evaluation for Operation: Resilient Planet are in Appendix D, and the complete results for The Habitable Planet are in Appendix F. A summary of the WBLE design criteria ratings are shown in Table 3. Table 3. Summary of average WBLE design criteria ratings. Operation: Resilient Planet The Habitable Planet Usability Content Educational Value Vividness The low rating for Operation: Resilient Planet in the Content category reflects the fact that the content was prepared for students in grades 5 through 8, and the target learners need to study college-level material. Although the site indicates that the material could be adapted for older learners, the necessary adaptations would be time-consuming. The low rating for The Habitable Planet in the Educational Value category reflects the course s lack of quizzes or other types of assessment. A summary of the content criteria ratings for The Habitable Planet are shown in Table 4. The course attained a low score in category IV, Land and Water Use, because the course tends to take a global view of environmental issues, while the AP environmental science exam includes a large component on land and water use patterns and policy pertinent to the United States. The complete results of the content criteria evaluation for The Habitable Planet are in Appendix G. Table 4. Summary of average content criteria ratings for The Habitable Planet. Content Category Average Rating I. Earth Systems and Resources 4.0 II. The Living World 4.0 III. Population 4.0 IV. Land and Water Use 2.17 V. Energy Resources and Consumption 4.0 VI. Pollution 3.29 VII. Global Change 4.0 The results of the learner survey were a bit disappointing, largely because only two students completed the survey. Several factors may have contributed to the low response rate: 1. The respondents were required to spend up to an hour to review enough content on the two sites to be able to form opinions and respond to the survey. This might be a big time commitment for a 15-year-old. 14 P age

15 2. The timing of the survey was poor since it occurred over the Thanksgiving break and at the end of the semester when students were finishing final projects. 3. The students had no incentive (or disincentive) to participate in the survey. 4. The students were required to get the permission of a parent or guardian to participate, which some teenagers can be loath to do. Both of the respondents are currently enrolled in an AP environmental science course in a brick-andmortar school, and both indicated that they already enjoy studying environmental science; therefore, the survey results cannot indicate whether either of the WBLEs would motivate students to study the subject matter. However, a few insights could be gleaned from the responses. Overall, the two respondents were receptive to the idea of using Web-based sources for learning. Both seemed interested in the engaging aspects of Operation: Resilient Planet, but one respondent remarked that the materials did not seem applicable to his/her academic needs and would not help with studying for a test. Somewhat surprisingly, both respondents responded quite positively to The Habitable Planet despite (or perhaps because of) its linear nature. Neither respondent reported any technical problems with the WBLEs and had no difficulties navigating the sites to access the content. Complete results of the survey are shown in Appendix G. The results of this evaluation present a clear picture of Operation: Resilient Planet and a less clear picture of The Habitable Planet. The results will be analyzed in the next section as they apply to the five evaluation questions presented in Section I. 15 P age

16 A. Congruence Analysis V. Discussion of the Results The Congruence Analysis asks, Is the WBLE suitable for the target learners? As explained in Section IV, the WBLE Operation: Resilient Planet did not pass this first analysis question. Operation: Resilient Planet is a dynamic and appealing site with much content to be admired, but the content is delivered in an eco-adventure format and aligned to the grades 5 through 8 NSES. Therefore, it would not meet the needs of AP-level high school students nor the goals of GPVS new AP environmental science program. Modifications to make the WBLE suitable for the target learners would require too much time to make that approach feasible. However, there is no question that The Habitable Planet is suitable for the learners targeted in this evaluation. The Habitable Planet was designed as an undergraduate-level college course, which aligns exactly with the intent of the AP environmental science program. The two survey respondents indicated that they found the content engaging and interesting. B. Content Analysis The Content Analysis asks, To what extent does this WBLE cover the concepts and topics included in the AP environmental science exam? Operation: Resilient Planet covers many of the concepts and topics of the AP exam, but not to the depth and breadth required of study materials. The Habitable Planet covers a great deal of the content in the AP exam to the depth and breadth required with the exception of category IV, Land and Water Use. Instructors would need to enhance the content for this category and a few other subject areas of the AP material. C. Design Analysis The Design Analysis asks, Does the WBLE motivate students to engage in the material? The answer to this question appears to be yes for both of the evaluated WBLEs. The survey respondents reacted quite positively to both sites. D. Feasibility Analysis The Feasibility Analysis asks, Is this WBLE suitable for use by individual students with limited guidance from an instructor? Since the survey respondents had no difficulties navigating the sites or understanding the content, both WBLE sites appear to be suitable for use by individual students with little guidance from instructors. However, the use of Operation: Resilient Planet would require instructor participation in the program s eco-adventure missions, which would be quite time-consuming. The lack of assessments in The Habitable Planet implies that instructors would need to create assessments for students, which may or may not be feasible for the GPVS instructional staff. E. Overall Decision The above analyses lead to the Overall Decision question: Does the candidate WBLE have the potential for being suitable as self-study materials for high school students who are preparing for the AP exam with limited guidance from online science teachers? The answer to this question for Operation: Resilient Planet is clearly no. Students would likely enjoy the experience of participating in the ecoadventure missions in Operation: Resilient Planet, but this participation is not likely to help them study for the AP environmental science exam to any great extent. The answer to this question for The Habitable Planet is yes, it does have the potential for this use. And The Habitable Planet s motivational factor may 16 P age

17 supersede its content shortfalls (in regards to the AP exam content) and lack of assessments compared to a traditional textbook course. This purpose of this evaluation was to determine whether one or both of the candidate WBLEs should be recommended for phase 2, a field trial evaluation to assess the candidate materials effectiveness with target group members. Carey and Dick (1991, p. 271) indicate that the field trial phase of a summative evaluation should consider learning outcomes: learners performance and attitudes before, immediately after, and at prescribed intervals following the instruction. This type of evaluation requires extensive time and resources more time and resources than could be managed by the staff at GPVS. And it is clear that The Habitable Planet course could not stand alone as the primary source of instruction for AP environmental science students who are preparing for the AP exam. Therefore, a phase 2 evaluation of learning performance for The Habitable Planet would not be advisable. However, a small-scale phase 2 evaluation of learner attitudes for The Habitable Planet may be productive. 17 P age

18 VI. Conclusions and Recommendations A. Immediate Conclusions The results of this evaluation suggest the following immediate conclusions: The Operation: Resilient Planet program is not suitable for the purpose of serving as instructional material in the AP environmental science program at GPVS, so it should be removed from consideration for that purpose. This assessment does not reflect on the general value of the program, but only on the program s utility for the purpose of this evaluation. The Habitable Planet course is potentially suitable for the purpose of serving as instructional material in the AP environmental science program at GPVS. Although The Habitable Planet course cannot serve as the sole source of instructional material for the AP environmental science program at GPVS, its motivational factor may supersede its content shortfalls and lack of assessments. B. Long-Range Planning The results of this evaluation may impact long-range planning at GPVS in the following ways: The AP advisory committee should consider executing a phase 2 evaluation of learner attitudes to assess whether learners would be more motivated by The Habitable Planet interactive Web-based course than a traditional printed textbook. The AP advisory committee must consider the cost of obtaining a license if they wish to use The Habitable Planet course as an official part of the AP environmental science curriculum. Although all the materials for this program are available online, the publisher of the program expects institutions and individual students to pay a small fee for use of the program. If GPVS were to adopt The Habitable Planet for its AP environmental science program, instructors would need to find supplementary material to cover all the content of the AP exam and devise student assessments. C. Evaluation Insights The design and implementation of this evaluation could have been improved by enacting the following changes: The Operation: Resilient Planet WBLE could have been rejected as a candidate for this situation so evaluation resources could be devoted to investigating The Habitable Planet course in more detail. A more complete review of the course could have produced a one-to-one comparison of the WBLE content and the AP content that GPVS instructors might find valuable. The learner survey could have been sent to members of the target learner group earlier in the year to avoid conflicts with the holiday break and end-of-semester projects. Follow-up interviews of the learners who reviewed the two WBLEs and participated in the survey could have added some insight into the elements they found engaging, interesting, and potentially useful for their studies. 18 P age

19 References Baya a, N., Shehade, H.M., & Baya a, A.R. (2009). A rubric for evaluating web-based learning environments. British Journal of Educational Technology, 40(4), doi: /j x Boulmetis, J. & Dutwin, P. (2005). The ABCs of evaluation (2nd Ed.). San Francisco, California: Jossey- Bass. Carey, L.M., & Dick, W. (1991). Summative evaluation. In L.J. Briggs, K.L. Gustafson, & M.H. Tillman (Eds.), Instructional design: Principles and applications (2nd Ed.), Englewood Cliffs, New Jersey: Educational Technology Publications. The College Board. (2010). AP environmental science course description. Effective Fall pp. (PDF). Available at Stuffelbeam, D.L. (2003). The CIPP model for evaluation. In T. Kellaghan & D.L. Stuffelbeam (Eds.), International Handbook of Educational Evaluation, Dordrecht, The Netherlands: Kluwer Academic Publishers. 19 P age

20 Appendix A: AP Environmental Science Exam Themes The College Board established the following six themes for the AP Environmental Science course: 1. Science is a process. Science is a method of learning more about the world. Science constantly changes the way we understand the world. 2. Energy conversions underlie all ecological processes. Energy cannot be created; it must come from somewhere. As energy flows through systems, at each step more of it becomes unusable. 3. The Earth itself is one interconnected system. Natural systems change over time and space. Biogeochemical systems vary in ability to recover from disturbances. 4. Humans alter natural systems. Humans have had an impact on the environment for millions of years. Technology and population growth have enabled humans to increase both the rate and scale of their impact on the environment. 5. Environmental problems have a cultural and social context. Understanding the role of cultural, social, and economic factors is vital to the development of solutions. 6. Human survival depends on developing practices that will achieve sustainable systems. A suitable combination of conservation and development is required. Management of common resources is essential. Reprinted from: The College Board. (2010). AP environmental science course description. Effective Fall pp P age

21 Appendix B: AP Environmental Science Topic Outline The College Board established the following outline of topics to be covered in any AP Environmental Science course and in the AP exam. Each major topic is followed by a percentage range that indicates the approximate proportion of exam questions that will cover that topical area. I. Earth Systems and Resources (10 15%) A. Earth Science Concepts (Geologic time scale; plate tectonics, earthquakes, volcanism; seasons; solar intensity and latitude) B. The Atmosphere (Composition; structure; weather and climate; atmospheric circulation and the Coriolis Effect; atmosphere ocean interactions; ENSO) C. Global Water Resources and Use (Freshwater/saltwater; ocean circulation; agricultural, industrial, and domestic use; surface and groundwater issues; global problems; conservation) D. Soil and Soil Dynamics (Rock cycle; formation; composition; physical and chemical properties; main soil types; erosion and other soil problems; soil conservation) II. The Living World (10 15%) A. Ecosystem Structure (Biological populations and communities; ecological niches; interactions among species; keystone species; species diversity and edge effects; major terrestrial and aquatic biomes) B. Energy Flow (Photosynthesis and cellular respiration; food webs and trophic levels; ecological pyramids) C. Ecosystem Diversity (Biodiversity; natural selection; evolution; ecosystem services) D. Natural Ecosystem Change (Climate shifts; species movement; ecological succession) E. Natural Biogeochemical Cycles (Carbon, nitrogen, phosphorus, sulfur, water, conservation of matter) III. Population (10 15%) A. Population Biology Concepts (Population ecology; carrying capacity; reproductive strategies; survivorship) B. Human Population 1. Human population dynamics (Historical population sizes; distribution; fertility rates; growth rates and doubling times; demographic transition; age-structure diagrams) 2. Population size (Strategies for sustainability; case studies; national policies) 3. Impacts of population growth (Hunger; disease; economic effects; resource use; habitat destruction) IV. Land and Water Use (10 15%) A. Agriculture 1. Feeding a growing population (Human nutritional requirements; types of agriculture; Green Revolution; genetic engineering and crop production; deforestation; irrigation; sustainable agriculture) 2. Controlling pests 21 P age

22 (Types of pesticides; costs and benefits of pesticide use; integrated pest management; relevant laws) B. Forestry (Tree plantations; old growth forests; forest fires; forest management; national forests) C. Rangelands (Overgrazing; deforestation; desertification; rangeland management; federal rangelands) D. Other Land Use 1. Urban land development (Planned development; suburban sprawl; urbanization) 2. Transportation infrastructure (Federal highway system; canals and channels; roadless areas; ecosystem impacts) 3. Public and federal lands (Management; wilderness areas; national parks; wildlife refuges; forests; wetlands) 4. Land conservation options (Preservation; remediation; mitigation; restoration) 5. Sustainable land-use strategies E. Mining (Mineral formation; extraction; global reserves; relevant laws and treaties) F. Fishing (Fishing techniques; overfishing; aquaculture; relevant laws and treaties) G. Global Economics (Globalization; World Bank; Tragedy of the Commons; relevant laws and treaties) V. Energy Resources and Consumption (10 15%) A. Energy Concepts (Energy forms; power; units; conversions; Laws of Thermodynamics) B. Energy Consumption 1. History (Industrial Revolution; exponential growth; energy crisis) 2. Present global energy use 3. Future energy needs C. Fossil Fuel Resources and Use (Formation of coal, oil, and natural gas; extraction/purification methods; world reserves and global demand; synfuels; environmental advantages/disadvantages of sources) D. Nuclear Energy (Nuclear fission process; nuclear fuel; electricity production; nuclear reactor types; environmental advantages/disadvantages; safety issues; radiation and human health; radioactive wastes; nuclear fusion) E. Hydroelectric Power (Dams; flood control; salmon; silting; other impacts) F. Energy Conservation (Energy efficiency; CAFE standards; hybrid electric vehicles; mass transit) G. Renewable Energy (Solar energy; solar electricity; hydrogen fuel cells; biomass; wind energy; small-scale hydroelectric; ocean waves and tidal energy; geothermal; environmental advantages/disadvantages) VI. Pollution (25 30%) A. Pollution Types 1. Air pollution (Sources primary and secondary; major air pollutants; measurement units; smog; acid deposition causes and effects; heat islands and temperature inversions; indoor air pollution; remediation and reduction strategies; Clean Air Act and other relevant laws) 2. Noise pollution (Sources; effects; control measures) 22 P age