MEMORANDUM. William E. Knight, Assistant Provost of Institutional Effectiveness

Transcription

1 MILLER COLLEGE OF BUSINESS Muncie, IN DEPARTMENT OF INFORMATION SYSTEMS Phone: AND OPERATIONS MANAGEMENT MEMORANDUM TO: FROM: William E. Knight, Assistant Provost of Institutional Effectiveness Allen D. Truell, Professor DATE: August 1, 2014 RE: Summer 2014 Assessment Grant Final Report I am writing in response to your request for a summer 2014 assessment grant final report. My summer 2014 assessment grant final report is outlined below: Project Goals/Objectives: The Miller College is accredited by The Association to Advance Collegiate Schools of Business (AACSB). As part of the accreditation process, the faculty identified key assurance of learning curricular outcomes for all Miller College graduates. Specifically, these key assurance of learning curricular outcomes are: communication, knowledge, integration, teamwork, and ethics (C-KITE). The key assurance of learning curricular outcome for communication is divided into two sub goals: oral and written. AACSB requires that student learning be assessed and that the loop be closed when deficiencies are discovered. The focus of this project was on the written portion of the key assurance of learning curricular outcome communication. Thus, the goals/objectives of this project were to: (1) Analyze data collected during the fall 2013 and spring 2014 Writing Proficiency Exam cycles for Miller College students. The Writing Proficiency Exam rubric is the data collection instrument for the written communication assurance of learning sub goal. The Writing Proficiency Exam rubric and its analysis coding system is attached. Data collected during the fall 2013 and spring 2014 Writing Proficiency Exam cycles for Miller College students were analyzed and are presented in Excel spreadsheet included with this report. (2) Prepare a report based on the data analysis. The purpose of this report is twofold: (1) to determine the Miller College student Writing Proficiency Exam pass rate

2 and (2) to determine the Writing Proficiency Exam rubric elements where Miller College students perform by assessment level. The Writing Proficiency Exam rubric and its analysis coding system is attached at the end of this report. The pass rates for the fall 2013 and spring 2014 Writing Proficiency Exam cycles were 70.56% (n = 175) and 68.35% (n = 203), respectively. These pass rates are above the four-year average pass rate of 67.99%. Writing Proficiency Exams are scored holistically. As part of this holistic scoring process, evaluators recommend either credit (proficient) or no credit (not proficient). In addition, evaluators provide feedback on rubric items needing improvement when recommending both credit and no credit for individual essays. Rubric items by percentage of reads for all, credit, and no credit student exams is presented in the accompanying Excel spreadsheet. Of most interest are the areas identified by evaluators as needing improvement. Conditional formatting was used to highlight the rubric items needing the most improvement (>10%). The only rubric item identified as needing improvement at >10% for credit student essays during the fall 2013 assessment process was: constructs a well-developed and coherent argument at 10.2% of reads. No rubric items were identified as needing improvement by evaluators at >10% for credit students during the spring 2014 assessment process. Of greater interest are the rubric items needing improvement for no credit student essays. The Writing Proficiency Exam rubric items at >10% needing improvement are noted below by both semester and percentage. Argument and Judgment The essay S1 Addresses the prompt (fall, 28.65%; spring, 40.85% S2 Communicates a focused purpose to an academic audience (fall, 49.44%; spring, 52.34%) S3 Constructs a well-developed and coherent argument (fall, 62.36%; spring, 63.83%) S4 Provides appropriate support for judgments (fall, 30.34%; spring, 23.83%) S7 Engages challenging ideas in the article (fall, 17.98%; spring, 16.60%) S8 Demonstrates how knowledge gained for the article supports judgments (fall, 10.11%) Writing Skills The essay S10 Appropriately introduces the reader to the main idea (fall, 16.85%; spring, 16.17%) Its paragraphs S12 Use content which consistently supports the thesis or main idea (fall, 23.03%; spring, 26.66%)

3 S14 Consistently maintained relationships or connections between ideas they develop (fall, 15.17%; spring, 20.00%) S15 Are developed enough to consistently keep readers focused on the thesis or main ideas (fall, 13.48%; spring, 12.34%) It sentences S19 Use relevant diction and stylistic conventions appropriate for junior-level university writers (fall, 15.73%; spring, 13.19%) S20 Are free of spelling errors which consistently distract the reader (fall, 14.04%; spring, 11.91%) S21 Are free of grammatical errors which consistently distract the reader (fall, 13.48%; spring, 14.89%) (3) Prepare a trend report based on data analyzed over the past four years. The purpose of this trend report is to offer visual picture of student performance changes overtime. This analysis is expected to indicate where improvement has occurred as well as areas needing additional work. The average pass rates for the fall 2010 through spring 2014 Writing Proficiency Exam cycles was 67.99% (n = 1,319). The pass rates for fall 2010 through fall 2010 through spring 2014 ranged from a high of 76.12% to a low of 62.8%. Writing Proficiency Exams are scored holistically. As part of this holistic scoring process, evaluators recommend either credit (proficient) or no credit (not proficient). In addition, evaluators provide feedback on rubric items needing improvement when recommending both credit and no credit for individual essays. Rubric items by percentage of reads for all, credit, and no credit student exams is presented in the Excel spreadsheet. Of most interest are the areas identified as needing improvement. Conditional formatting was used to highlight the rubric items needing the most improvement as identified by evaluators (>10%). No rubric items were identified as needing improvement by evaluators at an average of >10% for credit students during the fall 2010 through spring 2014 assessment cycles. Of greater interest are the rubric items needing improvement for no credit student essays. The specific rubric items averaging needing improvement at >10% over during the fall 2010 through spring 2014 assessment cycles are identified below: Argument and Judgment S1 Addresses the prompt (four-year average, 29.65%) S2 Communicates a focused purpose to an academic audience (four-year average, 43.85%) S3 Constructs a well-developed and coherent argument (four-year average, 57.92%) S4 Provides appropriate support for judgments (four-year average, 32.72%) S7 Engages challenging ideas in the article (four-year average, 20.35%)

4 S8 Demonstrates how knowledge gained for the article supports judgments (four-year average, 18.72%) Writing Skills: S10 Appropriately introduces the reader to the main idea (four-year average, 16.56%) Its paragraphs: S12 Use content which consistently supports the thesis or main idea (four-year average, 17.93%) S14 Consistently maintained relationships or connections between ideas they develop (four-year average, 15.92%) S15 Are developed enough to consistently keep readers focused on the thesis or main ideas (four-year average, 14.01%) Its sentences: S19 Use relevant diction and stylistic conventions appropriate for junior-level university writers (four-year average, 17.21%) S20 Are free of spelling errors which consistently distract the reader (four-year average, 12.30%) S21 Are free of grammatical errors which consistently distract the reader (fouryear average, 15.38%) (4) Prepare a list of possible strategies for closing the loop relative to deficiencies identified during the data analysis. Closing the loop is a critical component of the AACAB accreditation process as it relates directly and significantly to improving student learning. Possible strategies for closing the loop are listed below. Share the results of this analysis with ISOM 249 Foundations of Business Communication faculty. By sharing this information with ISOM 249, they will be in a better position to provide the needed closing of the loop feedback for students. A considerable number of written communication assignments completed during ISOM 249 Foundations of Business Communication so these faculty are uniquely position to help close the loop. The ISOM 249 Foundations of Business Communication faculty developed a common written communication rubric that closely aligns with the Writing Proficiency Exam rubric. Share the results of this analysis with Miller College faculty during scheduled brainstorming sessions. By sharing this information with Miller College faculty, they will be in a better position to assist students and to provide the needed closing of the loop feedback.

5 Review the with Miller College faculty during scheduled brainstorming sessions the administration and purpose of the Writing Proficiency Exam. By sharing this information with the Miller College faculty, they will be in a better position to assist students and to provide the needed closing of the loop feedback for students. Encourage all Miller College faculty to use the common written communication rubric developed by the ISOM 249 Foundations of Business Communication faculty. By using this common written communication rubric to assess writing assignments, the criteria assess by Writing Proficiency Exam evaluators will be reinforced throughout the students academic program.

6 Readers use the following criteria to evaluate Writing Proficiency Exams: ARGUMENT AND JUDGMENT Addresses the prompt (Analysis Item - S1). Communicates a focused purpose to an academic audience (Analysis Item - S2). Constructs a well-developed and coherent argument (Analysis Item - S3). Provides appropriate support for judgments (Analysis Item - S4). Gives appropriate consideration to alternative perspectives or evidence (Analysis Item - S5). Successfully interprets purpose(s) and focus of the article (Analysis Item - S6). Engages challenging ideas in the article (Analysis Item - S7). Demonstrates how knowledge gained from the article supports judgments (Analysis Item - S8). Does not dismiss evidence provided in the article without adequate justification (Analysis Item - S9). WRITING SKILLS: Appropriately introduces the reader to the main idea (Analysis Item - S10). Includes final paragraph(s) that provide(s) closure consistent with the main idea (Analysis Item - S11). Its paragraphs: Use content which consistently supports the thesis or main idea (Analysis Item - S12). Signal new ideas, sub-ideas, or emphases (Analysis Item - S13). Consistently maintain relationships or connections between the ideas they develop (Analysis Item - S14). Are developed enough to consistently keep readers focused on the thesis or main idea (Analysis Item - S15). Its sentences: Within each paragraph connect clearly and consistently to the main emphasis of the paragraph (Analysis Item - S16). Develop relevant ideas by building on content from preceding sentences (Analysis Item - S17). Within each paragraph connect clearly to the meaning of sentences that come before and after (Analysis Item - S18). Use relevant diction and stylistic conventions appropriate for junior-level university writers (Analysis Item - S19). Are free of spelling errors which consistently distract the reader (Analysis Item - S20).

7 Are free of grammatical errors which consistently distract the reader (Analysis Item - S21).