Assessment Plan for the Computer Science Program. All courses are offered in a traditional lecture and independent laboratory format.

Transcription

1 Assessment Plan for the Computer Science Program The mission of the School of Health and Sciences is to foster a learning environment in which students, faculty, staff, and community interact to educate students in the study of the natural sciences, technology, human-environment interaction, and health care through excellence in teaching, research, creative activities, and service. The Computer Science department contributes to the overall mission of East Central University and the School of Mathematics and Sciences by offering a baccalaureate degree in Computer Science. The degree program is especially designed to support students who wish to enter graduate school or the industrial market providing strong skills in programming, system management, and software engineering. To this end, the curriculum was developed following the recommendations of the Association for Computing Machinery and the Institute of Electrical and Electronic Engineers. The key "stakeholders" of the program include alumni, currently enrolled students, employers of our graduates, and the department faculty. Data are being collected which reflect input from all stakeholder groups. This report, and others, are made available to all stakeholders via a link placed on the official ECU Computer Science webpage which is hosted on the website All courses are offered in a traditional lecture and independent laboratory format. The Department of Computer Science has primary responsibility for the review of the program. Under certain circumstances the Dean of the College of Health and Sciences, the Academic Committee, and the Vice President for Academic Affairs may also become involved. Information gained from the assessment process is used to improve the program. If a concern is identified, it is possible to "trace" the origin of the concern to its source. Once the source of a concern is identified it is possible to take remedial action.

2 Primary Program Goal and Student Outcomes The primary goal of the Computer Science Program is to prepare graduates to enter professional fields or graduate schools requiring an undergraduate education in computer science. In the following materials survey questions are paraphrased in order to conserve report space. The sense of each question is preserved. To fulfill this goal, Program graduates should attain the following student outcomes: A. Program graduates should be able to integrate knowledge from computer science, mathematics, and other disciplines to enable students to formulate original computer programs. Criteria for Student Outcome A: 1. Student performance on standardized exams. Assessment instruments: The Major Field of Achievement Test in Computer Science. The report received from ETS concerning the MFAT test administered in April of 2007 contains this quote: Total scores for this test file cannot be compared to testing years prior to 2006 due to changes in the Computer Science Test that were introduced in This statement renders useless all MFAT data collected since 1991, the first year of use by ECU. We therefore removed all data gathered between 1991 and 2006 from this portion of the report. Major Field Achievement Test in Computer Science Scores of East Central Computer Science Graduating Seniors Academic Year Examinees Examinees Scoring at or Above 30% ECU Number ECU Number ECU Percent % % Total %

3 Performance Goal We recognize that the "Major Field Achievement Test" offered by ETS is not a "perfect fit" for our program, but we also recognize the need for a nationally normed instrument. We are willing to accept 30 or more percent of ECU students meeting or exceeding the 30th percentile as indicating that our program is serving its intended purposes. Judgement: The National on this test in was for males, and for females, with standard deviation of ECU students took the test (one was female). The ECU average was 139, within a standard deviation of the National, and again one ECU student scored above the 90 th percentile. We believe the performance goal is being met, but the lack of historical data will impell us to modify the goal as ETS gains experience with this new MFAT. This year 64 institutions utilized the Computer Science MFAT, as opposed to only 33 last year. Comment: The loss of historical data necessitated by the change in the MFAT itself is very troubling, and greatly perturbs the assessment effort of this department. At this writing, no better alternative to the MFAT has been found, although some consideration might be given to the on line version.

4 2. Alumni and employer satisfaction with the quality of the computer science education they received at ECU. Assessment instruments: a) Alumni Survey, item 5. b) Employer Survey, item 10. ECU Computer Science Department Survey Item 5 Survey of Alumni for Student Outcome A Quality of Education in Computer Science Year Number of Insufficient Excellent Respondents Total Percent % 1.09% 28.42% 71.00%

5 ECU Computer Science Department Item 10 Survey of Employers for Student Outcome A Overall Assessment of Occupational Preparation Year Number of Poor Below Above Excellent Respondents Total Percent % 0.00% 1.20% 19.88% 64.47% 11.45% Performance Goal We expect at least 75% of respondents to each question to indicate "average" to "excellent". Judgment: 100% of the alumni surveyed (15) indicated average to excellent on each of the two alumni survey questions. Only 1 employer responded, but that employer indicated reported average to excellent on the relevant questions. All performance goals are being greatly exceeded. No changes in the program are planned at this time. Some effort to improve the response to questionnaires should be contemplated, since responses in previous years have been more numerous. Comment: Over the years only two graduates have been rated as "below average". We regard this as an indication of a problem with one individual, since we believe the same graduate is being reported in multiple years.

6 B. Students should be familiar with the courses and disciplines that encompass a large and appropriate spectrum of the computing sciences. Criteria for Student Outcome B: 1. MFAT Indicators concerning specific subdisciplines Assessment Instruments: MFAT Departmental Summary of Assessment Indicators Year Mean Percent Correct in Each Subdiscipline Programming Fundamentals Number of Responde nts Discrete Structure and Algorithms 11 49% 23% 27% % 26.00% 31.00% Systems: Architecture, Operating Systems, Database Performance Goal: These subdisciplines have now been measured for two years, with essentially consisting results. We would like to see all indicators increase to the vicinity of 50% correct responses. According to the MFAT report, not all students responded to questions in these categories. Judgement: The current scores seem low. However, ETS did not provide standard deviations for these sub scores, so a firm judgement is perhaps premature. The MFAT should be examined again to determine if it is measuring skill sets which are appropriate to non-doctoral granting institutions. 2. Alumni assessment of their job skills in comparison with the job skills of their colleagues. Assessment instruments: Alumni Survey, items 5 & 9.

7 ECU Computer Science Department, Item 5 Survey of Alumni for Student Outcome B Quality of Computer Science Education Year Number of Insufficient Excellent Respondents Total Percent % 1.10% 27.12% 71.78%

8 ECU Computer Science Department, Item 9 Survey of Alumni Skills of ECU Grads Compared to Colleagues Year Number of Less Well About the Well Very Well Respondents Prepared Same Total Percent % 2.47% 15.62% 36.99% 42.19% Performance Goal: We expect that 75% or more of our respondents to any question to indicate "well" or above, or "average" or above, as appropriate. Judgement: 93.33% of 15 alumni surveyed indicated either average to excellent on the question regarding quality of CS Education % of respondents indicated that their job skills compared well to very well with those colleagues from other schools. 1 respondent indicated that their job skills were about the same as colleagues from other schools. One rated themselves lower. The performance goals are being exceeded. No changes are anticipated at this time.

9 3. Employer assessment of alumni job skills in comparison with the job skills of their colleagues. Assessment instruments: Employer Survey, items 1, 7, 10 & 11. ECU Computer Science Department, Item 1 Survey of Employer for Student Outcome B General Education Preparation of ECU Grads Year Number of Poor Below Above Excellent Respondents Total Percent % 0.00% 1.85% 25.93% 63.58% 8.64%

10 ECU Computer Science Department, Item 7 Survey of Employer for Student Outcome B Problem Solving Skills Year Number of Poor Below Above Excellent Respondents Total Percent % 0.00% 3.98% 11.93% 42.61% 41.48%

11 ECU Computer Science Department, Item 10 Survey of Employer for Student Outcome B Overall Occupational Preparation Year Number of Poor Below Above Excellent Respondents Total Percent % 0.00% 1.92% 17.95% 68.59% 12.18%

12 ECU Computer Science Department, Item 11 Survey of Employer for Student Outcome B Preparation of ECU Grads Compared To Other Students Year Number of Less Well About the Same Better Prepared Respondents Prepared Total Percent % 1.83% 75.00% 23.17% Performance Goal: We expect that 75% or more of our respondents to any question to indicate "well" or above, or "average" or above, as appropriate. Judgement: In the most recent survey 100% of 1 employers responding indicated either about the same to very well, or average to excellent as was appropriate for a particular question. The performance goals are being exceeded. No changes are anticipated at this time. Comment: Over the years, there seems to be only a single graduate who is not satisfying his employer. That graduate is reported in multiple years. The employer response to surveys is becoming frustratingly low, perhaps due to concerns that employers have about legal liability. It is possible that such traditional assessment tools have become outmoded, and the Assessment Committee should consider finding means of replacing them.

13 C. Students should be able to communicate effectively in oral and written presentations. Criteria for Student Outcome C: 1. Reported satisfaction by employers of our graduates. Assessment Instruments: Employer survey, items 2, 3, & 4. ECU Computer Science Department, Item 2 Survey of Employer for Student Outcome C Writing Competency Year Number of Poor Below Above Excellent Respondents Total Percent % 1.83% 3.66% 53.05% 33.54% 7.93%

14 ECU Computer Science Department, Item 3 Survey of Employer for Student Outcome C Comprehension of Written Materials Year Number of Poor Below Above Excellent Respondents Total Percent % 0.00% 1.30% 33.77% 50.00% 14.94%

15 ECU Computer Science Department, Item 4 Survey of Employer for Student Outcome C Oral Communication Skills Year Number of Poor Below Above Excellent Respondents Total Percent % 0.00% 4.82% 59.64% 25.30% 10.24% Performance Goal: At least 75% of the respondents should report an "average" or above rating for our graduates on questions. Judgement: The performance goals are being met. Over the years writing competency and comprehension of written materials have been a minor discordant note. Only one employer responded to the survey. Comment: Given the small number of respondents, one individual seems to skew the results disproportionately. 2. Portfolio assessment of ECU student written materials Assessment Instruments: A blind evaluation by ECU CS faculty of two assignments: one at the lower level, and one at the upper level, using a standard rubric developed for the purpose.

16 Rubric for Assessment of Written Materials Target Criteria Points to Assign English Language Presentation Many misspelled words, poor punctuation, bad grammar 0 spelling, grammar, and punctuation are acceptable spelling, grammar, and punctuation are exceptional Logical Organization of paper No organization to paper, difficult to read and understand Organization is acceptable, and understandable Use of technical vocabulary Technical Content Organization is exceptional, paper follows logical development, imminently readable Too much, or too little jargon, terms not defined Acceptable use of jargon, technical vocabulary Jargon and technical vocabulary is used appropriately and exceptionally well Poor or incorrect technical content technical content is adequate technical content is exceptional Year Number of papers Results of Rubric Scoring English Presentation Logical Organization Technical Vocabulary / / / / / / / / / / / /2.0 Ave / / / /2.0 Technical Content

17 Performance Goal: We expect that each category should average 1.0/2.0. Judgement: In 2007 and 2008 the goal was not met in the area of technical content. In particular, the year 2008 seems to have represented a low point. Last year we examined ways to enhance the scores in this area, including investigation of textbooks, and encouragement for students to take the ECU Technical Writing Course. This year, the apparent trend is upward, with all scores for this year meeting the expectations. The three-year average remains low, but the current year has met expectations. We will continue to monitor this area, and continue to encourage the use of writing enhancement materials.

18 Mission Statement Student Outcomes Assessment Plan for the Computer Science Program Criteria & Performance Goals Assessment Results Use of Results The Computer Science department contributes to the overall mission of East Central University by offering a baccalaureate degree in Computer Science. The degree program is especially designed to support students who wish to enter graduate school or the industrial market providing strong skills in programming, system management, and software engineering. To this end, the curriculum was developed following the recommendations of the Association for Computing Machinery and the Institute of Electrical and Electronic Engineers. Goal Statement A. Program graduates should be able to integrate knowledge from computer science, mathematics, and other disciplines to enable students to formulate original computer programs. A1. Student performance on standardized examinations. We are willing to accept the ECU average score on the ETS Major Field of Achievement Test being within one standard deviation of the national average as indicating that our program is serving its intended purposes. A2. Alumni and employer satisfaction with the quality of the computer science education students received at ECU. We expect at least 75% of respondents to the alumni survey to assess the quality of the computer science education they received at ECU as average to excellent. A1. The National on this test in was for males, and for females, with standard deviation of (one female) ECU students took the test. The ECU average was 139.0, within a standard deviation A % of the alumni surveyed (n=15) indicated average to excellent on the relevant question. Only one employer responded to the survey, but indicated 100% average to excellent. A1. Each year the ECU average score has been within one standard deviation of the national average, so no changes in the program are indicated at this time. The MFAT test itself has changed last year, making a comparison of this year's scores with previous years impossible. A2. The performance goal is being met. No changes are planned at this time. The primary goal of the Computer Science Program is to prepare graduates to possess a working knowledge of computer science principles and terminology. A further goal is to support the School of Health and Sciences Mission by offering a strong undergraduate degree. B. Students should be familiar with courses and disciplines that encompass a large and appropriate spectrum of the computing sciences. B1.MFAT Indicators concerning specific subdisciplines. Students should answer 50% or more correct in each category. B2. Alumni assessment of their job skills in comparison with the job skills of their colleagues. We expect at least 75% of respondents to the alumni survey to indicate that their computer science preparation compared well or very well to that of their colleagues. B1.Three subdisciplines : Programming Fundamentals, Discrete Structures, and Systems have been measured, with scores below 50% correct in each instance. B2. Survey results indicated 100% of respondents indicate well to very well on the relevant questions. (N=15). B1. The goal is not being met. Faculty will examine the actual MFAT instrument to determine its appropriateness to a nondoctoral granting institution. B2. The performance goals are being met. No changes are anticipated at this time.

19 C. Students should be able to communicate effectively in oral or written presentations. B3. We expect that at least 75% of respondents to the employer survey will indicate that Computer Science Preparation of ECU graduates is about the same to better prepared. C1. Reported satisfaction by employers of our graduates. At least 75% of employers should report an average or above rating for our graduates on questions concerning oral and written communications. B3. Only one employer responded. 100% of employers have responded about the same to better prepared. C1. 100% of respondents to the relevant questions on the employer survey indicate a rating of average to excellent. Employers of 44 ECU graduates responded to the survey. B3. The performance goals are being met. No changes are anticipated at this time. C1. The performance goals are being met. No changes are anticipated at this time. Some change in the survey methodology is indicated. C2. Portfolio assessment applied to two major assignments, measuring usage of English Language, Logical Organization, Technical Vocabulary, Technical Content. We expect each category to achieve a score of 1.0/2.0. C2. All areas scored at least 1.0/2.0 an improvement from last year. C2. The performance goals are being met. Last year's remediation seems to have worked, and will be continued. Increased referrals to the ECU Writing Center will be made, and students will be advised to take the ECU Technical Writing class.