ENVS 101: Introduction to Environmental Science Learning Outcomes Assessment Project Executive Summary

Transcription

1 Submitted by Elizabeth Shrader ENVS 101: Introduction to Environmental Science Learning Outcomes Assessment Project Executive Summary Stage 1: Designing and proposing a Learning Outcomes Assessment Project: Project Description: ENVS 101: Introduction to Environmental Science is a first year science course with no college level prerequisites. While some students intending to major in the field do take the course, the largest enrolled population is general education students who must complete science credits for their AA degree or certificate programs. Students can choose to take the laboratory as a related but completely separate course. The ENVS 101 Learning Outcomes Assessment (LOA) project was intended to determine to what extent students in the introductory ENVS lecture course are achieving the stated Common Course Objectives. These objectives are due for revision in Spring 2012 and this information from the LOA project will be used as part of that review. Project Objectives: 1. Comprehend how the earth s natural system operate and interrelate with one another. 2. Describe how human activities impact natural systems. 3. Delineate natural flows of chemicals, minerals, water and energy. 4. Explain how human activities alter or degrade natural flows. 5. Evaluate a wide range of cultural and social approaches and how they apply to environmental issues. 6. Delineate geographic distributions of environmental impacts. 7. Explain concepts of environmental justice. 8. Summarize major environmental policies and regulations. 9. Assess the effect and importance of individual behavior in environmental issues. 10. Classify technologies that prevent, control and reverse environmental harm. Methodology & LOA Instrument Design: This assessment was via a multiple choice pre/post-test which included 30 questions, each linked to a specific course objective. Students were given the pre/post-test (henceforth the LOA Instrument) within the first week of the course to determine the degree of prior knowledge of the content. The LOA Instrument was then given again at the end of the course to determine if the degree of content knowledge as relates to the course objectives was increased through the learning activities of the course. Student data was compiled by matching the SID numbers placed on the answer sheets. Three questions were assigned to each of the 10 listed project objectives. The multiple choice questions were pulled from the test bank questions of the required texts for the course on all three CCBC campuses and developed in collaboration between East and Westside faculty whenever possible. The questions were then reviewed by colleagues at institutions to which CCBC Environmental Science students regularly transfer. Faculty were requested to administer the pre-test component during the first week of classes in Fall 2009 and Fall Additionally, faculty were requested to repeat the administration of the LOA Instrument as a component of their final exam or final in class assessment. The LOA Instrument was to be administered

2 in both face to face and on-line sections of the course. External Validation: External validation occurred in two ways. First, the test was reviewed by two external reviewers. These reviewers are colleagues in the field who teach at Towson University, an institution to which CCBC students frequently transfer and with whom CCBC has articulation agreements for many science courses. In addition the multiple choice questions used in the LOA Instrument were selected and edited as needed from the test bank questions of the required texts for the course on all CCBC campuses and developed in collaboration between East and Westside faculty whenever possible. Therefore most of the questions modeled the language and style of the texts students in the class use as their main learning tools. After the initial administration of the LOA Instrument and the analysis of the Fall 2009 results, concerns were raised by several faculty about the content of several questions. In the Fall 2010 administration of the LOA Instrument, questions of specific concern were changed although all Course Objectives were still represented by three separate questions on the LOA Instrument and all CCBC ENVS faculty were asked for input in the re-wording or other alteration of the questions between the Fall 2009 and the Fall 2010 administration of the LOA Instrument. Stage 2: Implementing the design and collecting data The LOA instrument was administered as a pre/post-test in ENVS 101 sections in the Fall of Seventeen sections of the course ran in the fall of 2010, including two online sections. Eighty percent of students successfully completed the course with a grade of A-D. Most (79%) of the students were in the 18 to 24 year-old age group. Summary of the pre and post-test results On the pre-test students scored an average of 45% (M=13.37) of the items correctly and significantly increased to an average of 55% (M=16.53) correct on the post-test on the Fall 2009 assessment. The Dundalk campus, with the least number of students saw a modest decrease from pre to posttest and all other locations including online sections experienced significant increases in scores (p<.05) Students receiving a grade of A-D experienced large gains in means scores from pre to post with students receiving a B and a D grade showing the largest percentage points increase (13%). Caucasian/White and African-American/Black students experienced significant increases in their scores with African-American/Black students showing a slightly larger gain but they still lagged behind the scores of Caucasian/White students (p<.05) Stage 3: Redesigning the course to improve student learning: Several activities were implemented in Fall 2010 in an attempt to improve student learning and outcomes. Recommendation 1: All sections of ENVS 101 should use the same textbook. Since students appeared to have the largest degree of success using the Withgott & Brennan text, all face-to-face

3 sections of ENVS 101 used the same Withgott & Brennan text. Only the on-line sections used a different text (Cunningham) due to a schedule change problem with the Catonsville bookstore. Recommendation 2: All faculty in ENVS were part of the School of Math & Science Pedagogy Project which used regular formative assessments to address the learning needs of students in an informal way throughout the course. Faculty were required to establish Formative Assessments and to make changes to their classroom teaching as indicated by the student feedback in a timely way throughout the semester. Recommendation 3: The LOA committee, upon reviewing the data, recommended that some of the questions on the Instrument, while well linked to the objectives, were not well matched to the specific content taught in the course. In environmental science, it is possible to address a single objective by using a variety of examples, however if the students are presented with an unfamiliar example, they may not be able to successfully address the details of the question while actually having knowledge of the broader course objective. Changes were made to four of the original 30 questions (please see Appendix 2) although the Instrument continued to measure each of the 10 Course Objectives studied with three questions. Recommendation 4: During the development of the LOA Instrument and during the data analysis of Fall 2009, it became obvious that the Course Objectives as stated in the ENVS 101 Common Course Outline were so vague as to be interpreted very differently by the various fulltime faculty who teach the course. It was strongly recommended that the ENVS 101 Course Committee address and revise the ENVS 101 CCO so that the Course Objectives are more detailed, concrete and measureable. This recommendation was not implemented for the Fall 2010 LOA administration, however, the CCO revision will being in Spring 2012 so that by the next round of LOA implementation the objectives should be much easier to address and analyze in a coherent and consistent way. S Stage 4: Implementing course revisions and reassessing student learning The LOA instrument was administered as a pre/post-test in ENVS 101 sections in the Fall of Nineteen sections of the course ran in the fall of 2010, including two online sections. Sixty-nine percent of students successfully completed the course with a grade of A-C. Most (75%) of the students were in the 18 to 24 year-old age group. Summary of the pre and post-test results On the pre-test students scored an average of 47% (M=13.48) of the items correctly and significantly increased to an average of 65% (M=18.18) correct on the post-test on the Fall 2010 assessment. The Essex campus showed the largest increase of 65% from pre to post (p<.05) Students receiving a grade of A-D experienced large gains in means scores from pre to post with students receiving a D grade showing the largest increase,78%. Interestingly, students obtaining an A in the course showed the smallest (15%) increase from pre to post-test and obtained a 67% correct on the post-test which is slightly lower than the mean score of those obtained a grade of B and D Caucasian/White and African-American/Black students experienced significant increases in their scores (p<.05) Male and female students showed significant increases in mean scores, regardless of race

4 Students enrolled on a fulltime basis showed a larger increase in their scores than students enrolled on a part-time basis; in addition, fulltime students obtained a significantly higher mean score on the post-test, 67% correct compared to 59% for part-time students The majority of the students in this analysis were taught by fulltime faculty and showed a significantly larger increase (55%) in mean scores than those taught by part-time faculty. Online sections experienced a 39% success rate which is considerably lower than the overall course success rate Stage 5: Final analysis and results Major Findings from Comparison between Fall 2009 & Fall 2010 Generally students in Fall 2009 and Fall 2010 achieved the same degree of success as indicated by the grade distributions of students which are similar. The exception was for the online students who saw a marked decrease in successful completion in Fall All campuses saw in increase in mean post-test score in the Fall 2010 administration of the LOA with the Essex campus continuing to have the greatest increase in mean post-test score. Overall students in the course increased their post-test score by 44% over their pre-test score in Fall Achievement of minority students in science continues to be a concern, however the data demonstrated that between Fall 2009 and Fall 2010 minority students closed the achievement gap as measured by percent positive change on post-test score as compared to pre-test score by 6%. Student enrollment status (full-time versus part time) did not significantly change their outcomes in the course. The data from Fall 2010 when a large number of adjuncts were teaching ENVS 101 continues to support the need to full-time faculty to play an active role in the mentoring of adjuncts. Students increased their performance on all measured course objectives as measured by Mean % change in pre- versus post-test scores. The smallest increases were seen for Objectives 5, 8, and 9. The Common Course Outline and all course objectives will be subject to review in Spring These objectives will be specifically addressed. The current objectives are made is such a way that while the overall objective may have been taught, it can be addressed in such a diversity of ways that a single LOA instrument may not accurately assess the level of understanding. Impact of Interventions & Recommendations for the Future From the data presented here, the following conclusions can be drawn about the effectiveness of the recommended interventions. Using a single common text-book appears to have contributed to the success of students in the course. The fact that students in the online sections, which did not use the same text as the face-to-face sections and who had markedly decreased success reinforces this finding. The recommendation of the LOA Committee is that all sections should use the same text and that the text be selected in such a way as to best match the stated course objectives. Regular informal communication with students via multiple Formative Assessments also appears to have increased student performance on the post-test. Perhaps these types of communication in the course enabled faculty to address problems in content knowledge before the issue became overwhelming for students and increased student engagement with the course materials.

5 Changing the four questions on the instrument may have created a more accurate portrayal of our students understanding of the course content. Developing strong course objectives that are clearly linked to measureable outcomes will be an important aspect of the Common Course Outline review process of Plan for Monitoring Future Recommendations The ENVS 101 Course Committee should continue to monitor progress in the course. Clearly the decisions made about revisions to the Common Course Outline and in adopting a high quality common textbook for the course are paramount. Regular ENVS 101 Course Committee meetings should occur. Planning course committee meetings at the start and end of each semester at the time of School of Mathematics & Science meetings may be a way of successfully implementing this monitoring program. Timeline & contact for Future Implementation Spring 2012: Begin revisions of ENVS 101 Common Course Outline Fall 2012: Begin textbook review for ENVS 101 as relates to revised Course Objectives Fall 2013: Revised CCO in place; common text adopted for all sections of ENVS 101 at CCBC.