Assessment Report July 2010 June 2011 Columbia College

Transcription

1 Assessment Report July 2010 June 2011 Columbia College Introduction There were four institutional assessment goals for , as approved by the Academic Assessment Committee: 1. Increase MFT scores above the national mean in all areas. Areas above national mean: Campus: Day (52 nd ) Major: Criminal Justice (61 st ) Location: Orlando (54 th ) Freeport (58 th ) Fort Worth (54 th ) Salt Lake City (56 th ) 2. Review current instruments to add value to curricular and faculty development. History has piloted a local senior test. Human Services, CIS and American Studies assessments remain under development. All other instruments, including ETSPP for general education, were deemed satisfactory. 3. Continue to use assessment results to improve curriculum and pedagogy. This is an ongoing goal, measured in this document. 4. Create an assessment website. The Academic Assessment Committee is currently developing a website. Faculty and staff continue to review MFT results and make changes based on what they find. Some of these changes are discussed below. This report will summarize academic assessment activities during the academic year, including summer A special feature of the report will be a review of initiatives driven by assessment outcomes during the last five years. Detailed department reports are available in the Academic Affairs office.

2 General Education General Education is formally assessed with the Educational Testing Services Proficiency Profile (ETSPP). The ETSPP and its predecessors, MAPP and AP, have been administered at the College since Day freshmen also took the ETSPP and scored above the national means overall and in all areas but mathematics. Overall the results are comparable to last year. Day seniors scored about national means in all categories except math, which was less than one point lower than the national mean. Nationwide seniors (including Evening) scored at national means in all categories except critical thinking, math and overall. All scores, except for overall, were close to the national means and there was little change from 2009 scores. Online seniors scored below the national averages in all areas but better than 2009 means in most categories. Potential explanations for inter-venue differences include: Unlike the Major Field Test, ETSPP has no extrinsic incentive. Students in the Nationwide and Online programs may not be taking our general education courses and scores may not be a reflection on our curriculum. Nontraditional students tend to enroll in the Nationwide and Online programs and mathematics is routinely difficult for the nontraditional student. Often, math is the course that scares students away from attending college right out of high school or derails the nontraditional student if they attended college earlier in their adult life. A positive assessment climate is difficult at best in the Online venue. Often, the presence of an instructor, site director, and/or classmates will motivate the student to take a test more seriously than sitting alone taking a test. Columbia College works hard to create a positive assessment climate but the online venue is inherently problematic. The Assessment Committee reviewed ETSPP results and AHE s analysis and recommended no change to the general education curriculum. Faculty Association endorsed the recommendation. Major Field Test Most baccalaureate degree granting majors use the Major Field Test (MFT) to assess learning outcomes. The College administered 1,346 MFTs in ten undergraduate and one graduate disciplines in The average score was 44 th percentile, (43 rd percentile last year). The range among divisions was 52 nd percentile in Day to 39 th percentile in Evening. There was much variation among Nationwide campuses (58 th percentile in Freeport, IL to 21 st percentile in Aurora). There was also much variation among disciplines. Criminal Justice (61 st percentile) was highest and was the only MFT that averaged above the 50 th percentile. Biology and Computer Science showed significant gains from last year. For the second consecutive year, Day seniors scored above the national mean overall, as did several Day disciplines (Business Administration, Computer Science, Criminal Justice, Psychology and Sociology). 2

3 Degree Programs Art The Art Department inaugurated portfolio assessment in In , 17 students were reviewed. In Fall 2010, eight students were reviewed. Three students had strong portfolios and presentations, scoring proficient or accomplished in most categories. Five had average portfolios and presentations, scoring developing or slightly above in six or more categories. No students were considered unsatisfactory in any category. In Spring 2011, nine students were reviewed. Four students had strong portfolios and presentations, scoring proficient or accomplished in most categories. Four students had average portfolios and presentations, scoring developing or slightly above in six or more categories. One student was below average or unsatisfactory in several categories. In , the faculty began a review of the core. It has been suspended to await the input of the new faculty. General topics will include problem-solving skills, aesthetics, and skill comparability between fine art (painting, drawing, printmaking) and design and photography students. Business Administration Undergraduate More than 800 seniors were assessed with the MFT, the Degree Program Assessment Form (DPAF) and case analyses. Overall MFT scores averaged slightly below national means (41 st percentile). Day averaged 53 rd percentile, down from 65 th percentile last year. AHE students averaged 40 th percentile, the same as last year. The range among campuses was smaller than last year (58 th percentile at Freeport, IL to 20 th percentile at Aurora). Aurora is a perennial weak scorer and will receive special attention in Faculty analysis of MFT results and DPAF forms resulted in increased coverage of statistical analysis in preparatory courses, increased coverage of situational strategic planning and more student presentation opportunities. Discussion continues about requiring additional undergraduate finance and quantitative analysis courses. Graduate Home Campus MBA students scored at the 35 th percentile on the MBA MFT after being above average last year. AHE results were 47 th percentile Online and 36 th percentile Nationwide. There were no trends observed that result in proposals for change. 3

4 Computer and Mathematical Sciences The MFT in Computer Science is taken by both Computer Science and Computer Information Systems seniors. The latter are at a disadvantage because of a lack of exposure to computer architecture, programming language theory and discrete mathematics. A departmental exam will be created for CIS majors. Seven Day Computer Science students averaged 64 th percentile. AHE students have steadily scored about one standard deviation below the national mean (13 th percentile). Management Information Systems graduates take the Business MFT and their scores are not separated from Business students. The MIS curriculum is being overhauled and the updated degree will be offered in A department test will likely be created. After a strong performance last year (75 th percentile), five Mathematics seniors scored at the 29 th percentile this year. The new, more demanding curriculum will not be tested until Criminal Justice Administration and Human Services Undergraduate Criminal Justice The MFT was taken by nine Day and 235 AHE students. MFT results remain an institutional success story: CJ seniors scored above the national average, ranging from 71 st percentile in Day to 51 st percentile in Evening. There was very little variation from campus to campus (68 th percentile at Lake County to 54 th percentile at Fort Worth). AHE scored at 61 st percentile overall. Forensic Science Forensic Science students take the Biology or Chemistry MFT. Because a number of undergraduate-level topics covered on these MFTs are not required in forensic science coursework, forensic science seniors do not perform as well as biology majors. Department faculty will seek guidance on assessment during the fall 2012 program review. Graduate Criminal Justice Assessment in the capstone (MSCJ 580) includes critical writing, presentation and completion of a program assessment instrument. The updated course emphasizes leadership. Human Services There is no Human Services MFT. Assessment takes place in the culminating experience course (HUMS 495). Human Services faculty piloted a departmental senior test in but needs more results before exporting it to AHE. Criminal Justice utilizes faculty integration conferences (FIC) and advisory groups especially well for assessment. A Masters in Forensic Science is a potential outcome of working with an advisory group. 4

5 Education Undergraduate Of the 52 students who completed their portfolios, only two needed additional coursework and/or work to complete the program. There was a 98% first time pass rate on the Praxis II test. The Praxis pass rate exceeds statewide averages. Rolla and Lake Ozark seniors are included in these results. Results of Senior Education Student Capstone Experiences Portfolio pass 49/54 50/52 43/45 50/52 Praxis first time pass 94% 100% 98% 98% In surveys graduating seniors were generally positive about their experiences, especially about assessment and professional development. Most first year teachers report they are adequately well or very well prepared. Differentiation, reported as a deficiency by some first year teachers, is being modeled by faculty in the instructional setting. Advanced smartboard techniques were covered in the November 2011 professional development seminar, addressing a reported technology deficiency. Graduate All MAT completers scored above the 90 th percentile on written and oral presentations, meeting the National Board Certification Goals. Exit interviews indicated general support for the program. Surveys reported issues with post-baccalaureate students being placed in MAT classes with seasoned teachers. The new Post-Bac core will address this concern. As a result of the 2010 consultant s report, the department added to the available online MAT courses (EDUC 563, Management of the Classroom Environment and EDUC 516, Reading and Writing across the Curriculum). History and Political Science History After ETS discontinued the History MFT and the ACAT proved unsatisfactory, the History department worked with Norton Publishing to develop the History Assessment Test (HAT). From a bank of 500 questions seniors are tested in the D2L environment (Desire 2 Learn, the college s course management system for online delivery). Seven students took the test in March The mean score was 41% correct answers, with a range from 52% to 33% correct. The test will be implemented in AHE soon and content area results will be available. Few curricular conclusions can be drawn from the first administration of HAT. 5

6 Comments on the DPAF note that many history majors are pursuing education certification. The department will consider how HIST 494 might better meet their needs as future educators (from departmental report). American Studies The department is in the third year of developing a review assessment for American Studies. Political Science Only one Day student took the MFT, scoring poorly; however, political science seniors have scored well over the years. Humanities English Six Day seniors took the Literature in English MFT, scoring at the 28 th percentile and ten Evening seniors, scoring at the 46 th percentile. Evening scored higher than Day in all eight content areas. Faculty are reviewing ENGL 431, the capstone course, for functionality. The writing track is also being upgraded. Philosophy and Religious Studies There is no MFT; assessment is limited to the senior seminar. No one graduated in Speech Communication There is no MFT; the ACAT was used in 2009 but deemed unsuitable. Assessment takes place mostly in the senior seminar (six students). The faculty are adopting the program review recommendation to create a local assessment device. Psychology and Sociology Psychology Psychology was assessed with MFT s, exit interviews and senior projects. The MFT was significantly revised by ETS; however, scores were not dramatically different form last year. Thirteen Day seniors averaged 78 th percentile on the MFT, the highest of any cohort. Fourteen Evening seniors averaged 31 st percentile (a drop from last year), 59 Nationwide seniors averaged 21 st percentile and 109 Online seniors averaged 29 th percentile. Nationwide campuses ranged from 44 th percentile at Crystal Lake (IL) to 19 th percentile at Rolla. Prerequisites for PSYC 495, the capstone, have been increased and the core is being expanded. Also, more Nationwide campuses are offering the more rigorous courses more frequently; however, students can avoid taking several of them and still fulfill degree requirements. 6

7 Adjunct Psychology faculty will be held to a higher standard in the approval process. It is hypothesized that a partial explanation for weak MFT scores in AHE is poorly-prepared instructors. The department is working with the Online Campus to modify online PSYC 101 significantly. The new model will provide rich content modules and weekly assessments. Sociology After several years above national averages, Sociology scores dropped to 44 th overall (53 percentile for six Day seniors; 40 th percentile for 15 AHE, all Online). Faculty had no specific suggestions for improvement other than increasing the quality of adjunct faculty. Nursing Nursing graduates are assessed with their licensure exam (NCLEX). The Missouri State Board of Nursing minimum requirement for the program is an 80% pass rate. Below are recent national, Columbia Campus and Lake Ozark Campus pass rates. Table 3 NCLEX Pass Rates, National Columbia Campus Lake Ozark Campus % 89.1% NA % 80.0% 62.8% % 81.6% 74.5% % 77.05% 95.24% % 90.0% 68.0% % (1 st quarter report) 86.4% (2 nd quarter report) 85.0% (3 rd quarter report) 83.8% (1 st quarter report) 100% (2 nd quarter report) 80.0% (3 rd quarter report) *Not all students from the May, 2011 graduating class have taken the exam. 70.0% (1 st quarter report) 100% (2 nd quarter report) 66.6% (3 rd quarter report) Several recent graduates, especially at the Lake Campus, were barely admissible. Along with those who washed out of the program but were regrettably readmitted, this group failed the NCLEX at a disproportionate rate. With higher admission and readmission standards in place, pass rates should increase. The new curriculum is also fully implemented. Mandatory HESI testing, with minimum scores required, is also having a positive effect. Student survey results show less satisfaction than last year. This can be attributed to increased course rigor, the HESI requirement and faculty turnover (two resignations, one at each campus). 7

8 Faculty are motived by high NCLEX pass rates and have embraced more rigorous standards. They are implementing videoconferencing, which has had its hiccups. They are seeking NLNAC accreditation, which will require better-credentialed clinical instructors. Science Biology MFT scores show improvement in both Day and Evening venues. When accounting for where seniors took the majority of their biology coursework (many Evening seniors took significant coursework in the Day program), Day seniors scored at the 60 th percentile and Evening at the 20 th. Curricular reforms previously implemented may be having a positive impact on scores; e.g., a 200-level cell biology course added to the core as well as a one-semester chemistry course. Also, basic skills such as applying statistics, using literature and writing lab requests as introduced in lower-level courses. A weak area in the MFT is organismal/plant. The department offers little coursework in this area and is considering offering more. Chemistry Three seniors scored at the 28 th percentile on the MFT, down from last year. A potential explanation is that they did not realize they had access to the Periodic Table during the test, which would have increased scores. Over the years chemistry faculty have revised and upgraded the curriculum based in part on MFT results. These reforms include adding a course in instrument analysis, an intro chemistry course for non-majors, a capstone separate from biology, and use of more sophisticated equipment and technology. These reforms continue with more development of basic skills in earlier coursework and more instrumentation. Environmental Science Four seniors completed ENVS 490. DPAFs indicated strengths in accessing and using scientific literature, assembling data and presenting information; weaknesses included critically analyzing literature and interpreting outcomes. Since there is no MFT for environmental science, faculty are considering requiring the Biology MFT and focusing only on the subject questions that are relevant to the degree. They are also considering a local senior exam. 8

9 Conclusion Overall MFT results were stagnant. There were bright spots, some continuing, some new. There were weak spots, some continuing, some new. In general, assessment--and especially using outcomes for improvement--has a higher institutional profile each year. However, the conclusion stated in last year s report remains accurate: The College is likely a model institution in the effort devoted to outcomes assessment. Work remains in closing the feedback loop. Perhaps AHE will follow closely behing the Day program s breakout in MFT scores. Sustained improvement is not likely, however, until there are meaningful incentives for students, faculty and staff to create and maintain an assessment-based culture of teaching and learning. The College has many venues for assessment in addition to MFT results: Faculty Integration Conferences Program Reviews Degree Performance Assessment Forms (DPAF s) Departmental and Institutional Student Surveys The MFT remains, however, the best objective indicator of how much students are learning in their degree programs. The discussions that began last year about making the MFT a higher-stakes test continue. Each of the proposals below has merit but has at least one basic flaw (either the difficulty in implementing it universally, especially Online, or how to apply the requirement to non-mft programs): Make the MFT score a part of a student s capstone class grade. Require a minimum MFT score in order to graduate. Post the MFT score on the transcript. Award an Incomplete Grade in the capstone course to any student who fails to take a required MFT and does not have it waived. Meet with all students with a poor score to ascertain the reason for the weak performance. Require a senior to score at least the national mean on the MFT to be admitted to a Columbia College graduate program. Assessment-driven Initiatives in There were a number of initiatives that emerged from faculty study of assessment results. All increased the rigor and integrity of degrees. Below are highlights: 9

10 Biology: a 200-level cell biology class CIS: development of a local test History: development of a local test Human Services: pilot of a local test MIS: revision of major; development of a local test MSCJ: curricular revisions Nursing: HESI required; admissions and readmission standards tightened Psychology: core strengthened; standards revised for adjunct faculty approval Five Year Review 2005 Assessment Plan approved in 2004 is implemented. Master syllabus revision work begins. MBA and MSCJ capstones are modified for online delivery. Development of departmental CIS exam begins. K-12 Department of Education certification is reinstated. Psychology faculty express need for lab space. MFT language is added to Master Syllabi of all core courses Master syllabus revision work continues. Lead faculty hired in Business Administration at Orlando and St. Louis. Art begins offering web design classes. Business increases coverage of statistical and quantitative analysis, situational strategic planning, and finance. Business requires a C in Accounting I & II. Development of departmental CIS exam continues. Laws of Criminal Evidence removed from CJ core. Psychology faculty express need for lab space. Nurse faculty raise admission requirements to program. Science looks at course numbering and sequencing Master syllabus revision work is completed. The HLC progress report on graduate assessment is accepted. Analysis of MFT scores shows: o No meaningful difference between Day and AHE average scores o Strong positive correlations between MFT score and GPA in major o Most seniors took most major courses with Columbia College o Most seniors took three or fewer years to complete their major requirements Development of a departmental CIS exam continues. CJ MFT scores are the highest in the College. Psychology faculty express need for lab space. 10

11 Graduates who score above the 90 th percentile on the MFT receive a special cord at commencement The MBA assessment metric proves faulty. The overall MFT average remains at 43 rd percentile. AHE begins filling curricular gaps in Psychology. MAPP scores continue to be near national means except for Online students, who score lower. St. Louis campus emerges as a model assessment culture. Development of a departmental CIS exam continues. Development of a departmental Human Services exam continues. Day and Evening psychology seniors continue to score well on the MFT. Psychology faculty express need for lab space. Sociology and Political Science students continue to score well on the MFT. Biology MFT scores improve due in part to the requirement of applied statistics and 200- level cell biology and redesigned research design and senior seminar classes. ETS discontinues the History MFT Business faculty experiment with and abandon the ACAT. History faculty experiment with and abandon the ACAT. Speech Communications faculty experiment with and abandon the ACAT. ETS replaces the MAPP with the ETSPP (same test, new name). St. Louis continues to perform well on Business MFT. Business requires a C in all core courses. Development of a departmental CIS exam continues. Development of a departmental Human Services exam continues. Criminal Justice Day and Evening, Psychology, Political Science and Sociology seniors continue to perform well on the MFT. Nursing students are required to take the HESI. TBS/lfs 12/28/11 11