Annual Report on Degree Program Assessment of Student Learning University Assessment Committee Office of Academic Assessment Purpose: The purpose of the Annual Report on Degree Program Assessment of Student Learning is to provide information about progress in assessment efforts for each degree program within your academic unit. Only one report is requested of each academic unit, as this report will accommodate multiple degree plans. (You can still submit separate reports if you prefer.) The report will be made available publicly at the Office of Academic Assessment website and will be available to appropriate accrediting agencies. It is recommended that your unit use your assessment report and results to celebrate achievements of student learning as well as to identify potential areas for future curriculum improvement. The University Assessment Committee will review your report to provide constructive feedback, as well as to identify particular academic units for potential assessment awards and/or mini-grants to support continuing assessment efforts. Please email this completed form as an attachment to d-oaa@jan.ucc.nau.edu. CONTACT INFO: Academic Unit: Geography, Planning & Recreation Date: November, 2008 Name: Pam Foti, Tom Paradis Title: Chair (Pam Foti) Email: pam.foti@nau.edu, Thomas.paradis@nau.edu Phone: 523-6196 NAU Box: 15016 Degree Program(s) reported here: BS Geography, BS Geographic Information Science ASSESSMENT REPORT: Instructions: Please answer the following five questions to the best of your ability for each degree program offered within your unit. You may use the table provided on the next page, or you may create your own report format. 1. Summarize your assessment activities during the past year for each degree program. (e.g. faculty discussions, new survey design, data collection, revised assessment plans or learning outcomes, etc.). Assigned, collected, and analyzed nine reflective essays from GGR 480C in May, 2008. A total of nine senior students completed the three-week capstone course. Assigned, collected, and scored with rubrics the following GGR 480C projects for both course-level and program-level assessment: o Written analytical essay: Glen Canyon Dam and positions for and against its o removal. Formal oral presentation: Learning from Los Angeles: Successes and Challenges with Redevelopment and Urban Design. Assembled a summer subcommittee with four faculty members and instructors: Jim Sell, Ray Huang, Kathy Thatcher, Tom Paradis. Subcommittee met two times to review
sample projects from GGR 480C portfolios. Each student portfolio included all course work for GGR 480C and two student-chosen projects from other courses. Subcommittee members attempted to complete a comprehensive rubric that included all seven program-level outcomes (see attached to report below). Several conclusions about student success and challenges were made with the input from the entire subcommittee. Assigned a student research poster project, requiring students in GGR 480C to produce, print, and present their team-based poster to the class following the final project. A scoring rubric will need to be designed for this project next year. In this case, the summer subcommittee reviewed the poster submissions, and three of the four subcommittee members watched the students present in front of the class. The poster became a part of their final course portfolios, reviewed by the subcommittee. 2. Describe specific assessment findings related to the learning outcomes assessed for each degree program, including any pertinent context surrounding the findings. Please include the learning outcomes themselves. (e.g. 77% of seniors performed at the proficient level of competency in problem solving, which is where we aimed to be this year using a new scoring rubric ) a. Please attach any tables, graphics, or charts to the end of this report. Please see table below, followed by an Executive Summary of findings. 3. Describe how assessment feedback has been provided to students, faculty, and staff. (e.g. report for faculty, executive summary for the dean, web page for students, alumni newsletter, discussion with students in class or club event, etc.) Poster presentation of 06-07 assessment findings (based on report from Fall 2007) displayed at the spring, 2008 NAU Assessment Fair. Poster then displayed prominently in the Geography office area for several months. Assessment findings and recommendations were discussed and used as input for a year-long curricular revision process, beginning with a presentation of assessment findings at the fall, 2007 annual departmental retreat. Geography summer subcommittee discussed May, 2008 assessment findings from the capstone geography course (see findings below) and concurred on dominant conclusions and future recommendations. Presented this latest report with the faculty at a regular faculty meeting in November, 2008. Conclusions and recommendations discussed and considered for next year. 4. In what ways have you used assessment findings to celebrate student achievements and/or to improve the curriculum this past year? (e.g. prizes to students, hosting student parties, changes to curriculum, student projects, learning goals, assessment strategies, etc.) Using the fall, 2007 program assessment report as one of several inputs, the entire geography faculty met once or twice a month throughout the 2007-2008 academic year to produce a major overhaul of the geography curriculum. This curriculum redesign consisted generally of an analysis of the entire core curriculum and the sequence of courses; several curricular changes within key courses to address various weaknesses; strategic additions and deletions of core courses themselves in an effort to produce a more efficient, comprehensive, and flexible geography program. The year-long process began with the August, 2007 Geography Retreat and continued at the next faculty meeting with the distribution of numerous internal and external reports and related information related to student learning. Two primary assessment documents were distributed and discussed during early fall, 2007: 1) the annual Geography assessment report that incoporated two years worth of assessment data, and 2) a national NSF-funded geography report sponsored by the Association of American Geographers that included national
data from geography alumni and geography employers across the country. This latter report included lists of the most important academic skills and geography knowledge deemed necessary for successful college curricula and ensuing employment. The geography faculty considered the report s conclusions and matched them with our own learning outcomes and assessment system. The national report confirmed our current focus on the real-world skills of writing, oral communication, teamwork, and research methods. The following bulleted list includes the dominant changes to the geography curriculum undertaken during 2007-2008: Assuring a balanced curriculum: roughly equal emphasis on dominant geography sub-disciplines, including cartography/gis, human geography, physical geography, and regional geography. Providing enhanced flexibility for the Extended Geography major. The resulting core curriculum of 47 credits became the standard for both the BS Geography (with Minor) and the Extended BS. Those enrolled in the Extended major can complete a variety of active-learning experiences including travel abroad, internships, and/or independent research. Improving efficiency and updating of Cartography/GIS course content through a revision of GGR 230, 330, and 331, consisting of the primary technical geography sequence. Improved scaffolding of the curriculum from sophomore through senior-year courses, where freshmen/sophomore level of course content serves as a stepping stone for more advanced learning during junior and senior years. Adoption of a four-year sequence of courses reflecting the revised core curriculum, useful as an advising tool. Students are encouraged to take courses in a logical sequence to help prevent past practices of students enrolling in courses largely out of sequence. Revisions to the Capstone geography course, GGR 480C: o Moved course from traditional May intersession slot to spring semester. Will be taught during spring, 2009 for the first time. A full spring course will allow students more time to develop key skill sets prior to graduation, including oral and written communications, cartographic (mapping) production, and competencies with research methods, analyses, and synthesis. o Scheduling the dominant student presentations in conjunction with two departmental or university functions: 1) the annual department student conference in Dubois, and 2) the NAU Undergraduate symposium. Both forums will allow students to present their research to a wider audience in both PowerPoint and poster formats, further enhancing the stakes for development of oral presentation and research skills. o An enhanced focus on in-depth undergraduate research focused on a topic of student s choice. Rather than a series of separate field-based projects during the course, students will more thoroughly develop a particular research project including literature review, development of research questions, methods, and approaches, and reporting of research analyses and findings. Students will participate in a class practice trip to a nearby locale to practice their field research methods. 5. Describe any changes to your assessment plans, or any challenges or educational experiences with the assessment process this past year that you would like to share. In large part based on the recommendations of the UAC, the following improvements have been attempted during 07-08 to the geography assessment process:
A faculty subcommittee was formed to review GGR 480C student portfolios during summer, 2008. This was a first attempt to provide faculty members outside of GGR 480C to inspect and evaluate some student work from the capstone course, in addition to the traditional input from the course instructors. The subcommittee met twice to review a sample of student portfolios that included projects from the capstone course and other student-chosen projects. Though deemed a successful first attempt, time went by quickly at each meeting. The subcommittee gleaned various conclusions, but not a thorough analysis of student work. This process will need be made more efficient, especially should the number of students increase. Developed a comprehensive rubric for use by Geography Subcommittee members (see attachment below) to provide general scores of student work for all seven program learning outcomes. This process was partially successful though provided only a generalized overview. Still, the input from multiple (4) faculty and instructors provided enhanced validity of assessment conclusions. Created a new rubric to assess critical thinking of a specific student field project, applied to a class trip to Page and the Glen Canyon Dam. Rubric includes criteria that apply to program outcomes, including geographical analysis, synthesis, and writing. Rescheduled the geography capstone course from May intersession to spring semester, beginning in spring, 2009. This will allow more thorough assessment of cartographic products that were not possible during the 3-week intersession. It will further allow for student-peer assessment of oral presentations during the annual geography department conference at Dubois. Students from other classes and disciplines fill out simple rubrics to assess every student presenter. This will likely be built into the revised geography assessment plan. Developed a rubric to score writing assignments in the capstone prerequisite, GGR 380W. Instructors of 380W and 480C collaborated on a more standardized rubric for use in both courses, to provide consistent assessments from one course to the next, and to allow for program-level assessment outside of the field course. a. Please submit any revised/updated assessment plans to the Office of Academic Assessment along with this report. ALTERNATIVE TABLE FORMAT: Degree-Program Learning Outcomes Assessment Method and Analysis Assessment Findings: How well are your students achieving the learning outcomes assessed? Geographic Description, Analysis, and Synthesis: differentiate between geographical description, analysis, and synthesis and demonstrate their own cognitive skills by Writing and Oral Rubric Scores (2 projects) Faculty Observations of sample portfolios Synthesis: 9 out of 9 students met standards for proficient or advanced. Analysis: 4 of 9 students scored at Novice level. Essays: Students overall satisfied, felt they made progress throughout course. Portfolio samples: Team ranked samples from Novice to Advanced, with wide disparity.
effectively describing, analyzing, and synthesizing various geographical data and information. Written Geographic Presentation: Students will be able to effectively present physical and human geography content knowledge, description, analyses and synthesis in written form while demonstrating satisfactory writing skills including grammar, paragraphing, organization, and referencing. Writing Rubric Scores Faculty Observations of sample portfolios Mechanics: 4 of 9 students scored at Novice level. 5 of 9 were Proficient or Advanced. Essays: Few comments on writing, though essays themselves demonstrated that 9 or 9 students were Proficient or Advanced with writing. Portfolio samples: Group rated students from Unsatisfactory to Proficient, a wide range. Oral Geographic Presentation: Students will be able to effectively present geographic analysis and synthesis through the use of oral presentations. Students should demonstrate their ability to clearly communicate geographical information in a teaching situation to lay audiences and demonstrate skills with regard to time management, organization, and speaking clarity. Oral Rubric Scores Rubrics: Delivery and Language: 9 of 9 students were Proficient. Intro, Organization, Knowledge, Evidence: 7 of 9 students rated Advanced, 1 Proficient, 1 Novice in one category each. Audience interaction and awareness: 9 of 9 students rated Proficient or Advanced. Documentation of Evidence: 6 of 9 students rated as Novice. Essays: Students overall pleased with progress in oral communication skills. Frequently mentioned in essays, thankful to have the chances to improve presentation abilities. Several students commented about continued nervousness in front of a group. Cartographic Design and Production: design and produce quality reference and thematic maps for appropriate audiences and purposes within at least one computergenerated production Faculty Observations of Sample Portfolios Essays: Students generally desiring more cartographic experience, though realize limitations with 3-week course. Portfolio samples: All samples rated Unsatisfactory. In part due to lack of attention to this outcome within the course.
method available, including software for computerized graphics and/or GIS software systems. Conceptual Knowledge of Physical and Human Geography: Students will be able to demonstrate their knowledge and understanding of how physical processes shape the earth s surface and interact with plant and animal life to create, sustain, and modify ecosystems. Students will likewise be able to demonstrate their knowledge and understanding of how political, cultural, social, and economic activities collectively shape human landscapes and physical environments. Research Methodology and Design: Students will be able to effectively apply the scientific method to conduct and interpret geographical inquiries utilizing a variety of primary and secondary research methods both qualitative and quantitative. Students should be able to conduct research in both individual and teamwork situations through the development of specific research questions and problems. Teamwork and Collaboration: Students will be able to demonstrate effective, professional teamwork and collaborative skills when working in groups toward a common goal. Oral Rubric Scores Faculty Observations of Sample Portfolios Writing and Oral Rubric Scores Faculty Observations of sample portfolios Written and oral rubric scores Oral Rubrics: 9 of 9 students rated Proficient or Advanced. Essays: Students generally pleased with ability to incorporate knowledge into products. No key comments one way or another. Portfolio samples: Students rated on a range from Novice to Advanced. No real consensus. Very subjective. Rubrics: Not yet available in rubric form. Essays: Students provided numerous comments with respect to qualitative and quantitative research: Very pleased with opportunities to improve qualitative research methods and move through the scientific research process. Very displeased with lack of opportunities to conduct quantitative research within the course. Focused more on human rather than physical geography. Portfolio samples: Similar findings: Excellent student work in conducting, analyzing, and reporting qualitative methods and findings. Lack of experiences to practice and assess quantitative research. Final student posters demonstrated thorough knowledge of the research process. Rubrics: 9 of 9 students rated Proficient or Advanced. Essays: Students commented often about teamwork, generally pleased with their own abilities to work within a team. Occasional comments about challenging team members and how they approached the situations.
EXECUTIVE SUMMARY OF FINDINGS: Student Successes: Assessment indicators consistently reveal student success in the areas of oral communication, teamwork, and qualitative research design and implementation. All of these outcomes are emphasized within the capstone course. Areas for Improvement: Assessments revealed mixed results and high disparity among student outcomes with respect to effective writing, knowledge of physical and human geography, and geographic analysis and synthesis. Poster design was acceptable though could be better improved and guided by the instructors. Areas to add experiences and assessments: Assessment findings (or lack of them) reveal a need to include greater opportunities for quantitative research experiences, such as in the area of physical geography. Cartographic techniques should also be further emphasized as a class project (or embedded within other projects) and assessed accordingly.
Degree Program Assessment 2008 Outcomes from Program Assessment Plan (for scoring of Portfolios by Geography Subcommittee) CRITERIA (Check one box at right for each criterion based on your sense of overall student accomplishment.) Unsatisfactory Novice Proficient Advanced 1) OUTCOME 1: Geographic Description, Analysis, and Synthesis: Students will be able to differentiate between geographical description, analysis, and synthesis and demonstrate their own cognitive skills by effectively describing, analyzing, and synthesizing various geographical data and information. 2) OUTCOME 2: Written Geographic Presentation: effectively present physical and human geography content knowledge, description, analyses and synthesis in written form while demonstrating satisfactory writing skills including grammar, paragraphing, organization, and referencing. 3) OUTCOME 3: Oral Geographic Presentation: effectively present geographic analysis and synthesis through the use of oral presentations. Students should demonstrate their ability to clearly communicate geographical information in a teaching situation to lay audiences and demonstrate skills with regard to time management, organization, and speaking clarity.
4) OUTCOME 4: Cartographic Design and Production: design and produce quality reference and thematic maps for appropriate audiences and purposes within at least one computergenerated production method available, including software for computerized graphics and/or GIS software systems. 5) OUTCOME 5: Conceptual Knowledge of Physical and Human Geography: demonstrate their knowledge and understanding of how physical processes shape the earth s surface and interact with plant and animal life to create, sustain, and modify ecosystems. Students will likewise be able to demonstrate their knowledge and understanding of how political, cultural, social, and economic activities collectively shape human landscapes and physical environments. 6) OUTCOME 6: Research Methodology and Design: effectively apply the scientific method to conduct and interpret geographical inquiries utilizing a variety of primary and secondary research methods both qualitative and quantitative. Students should be able to conduct research in both individual and teamwork situations through the development of specific research questions and
problems. 7) OUTCOME 7: Teamwork and Collaboration: demonstrate effective, professional teamwork and collaborative skills when working in groups toward a common goal. WRITING MECHANICS CRITERIA Mechanics Unsatisfactory (D/F) Many or numerous repeated errors in grammar, diction, sentence structure, and/or spelling that detract from writing effectiveness. Novice (B-/C) Proficient (A-/B) Advanced (A/A+) Some errors in Few errors which Few or no errors grammar, diction, do not detract which do not sentence significantly from detract at all from structure, and/or writing writing spelling that may effectiveness; effectiveness; or may not sentence sentences and detract from structure and word choices writing word choice varied and effectiveness. show variety. creative. HIGHLIGHTS OF OUTCOMES THAT APPEAR TO BE SUCCESSFULLY LEARNED SUGGESTIONS FOR IMPROVEMENT FOR LEARNING OUTCOMES