Performance Assessments in Computer Science. An example of student perceptions

Size: px

Start display at page:

Download "Performance Assessments in Computer Science. An example of student perceptions"

Griselda Quinn
8 years ago
Views:

1 Performance Assessments in Computer Science An example of student perceptions , Falun, Sweden Erik Bergström Helen Pehrsson University of Skövde Informatics Research Center

2 Background (1) Computer science studies aretransforming Specialization of study programs Demand mainly from industry and students Certifications included in many study programs IT certifications dominated by Microsoft and Cisco (McGill & Dixon, 2007) Job advertisements ask for certifications, not necessarily degrees (Morris et al., 2012) Erik Bergström & Helen Pehrsson 2

certifications dominated by Microsoft and Cisco (McGill & Dixon, 2007) Job advertisements ask

3 Background (2) Benefits of certification Greater knowledge and increased productivity, a certain level of expertise and skill, improved support quality, reduced training costs and higher morale and commitment as benefits of employing students that have undertaken certification (Ray & McCoy, 2000) From an educational perspective, the certification examination provides an additional tool for evaluating course and program content (Ray & McCoy, 2000) Attracting students (Brookshire, 2000) Drawbacks of certification Absence of unbiased neutral groups for determining course contents, creating exams and authorizing examiners (Ray & McCoy, 2000) Lack of educational rigor, to focused material, training oriented rather than education oriented, and that it is too market and popularity driven. (Jovanovic et al., 2006) Erik Bergström & Helen Pehrsson 3

evaluating course and program content (Ray & McCoy, 2000) Attracting students (Brookshire, 2000) Drawbacks of certification Absence of unbiased neutral groups for determining course contents,

4 Background (3) Courses in computer science normally contain two parts Theory part Written exam Practical part Written lab report Lab reports are prone to plagiarism Exams are hard to vary Despite the use of practical assignments computer science educators expresses concern for their students lack of practical skills (McCracken et al., 2001, Lister et al., 2004). Why examine programming/system administration/ with a written lab report? Performance assessments can be best of both worlds Erik Bergström & Helen Pehrsson 4

concern for their students lack of practical skills (McCracken et al., 2001, Lister et al., 2004).

5 Background (4) Performance assessments provide an alternative way of examine practical skills Performance assessments can measure students cognitive thinking and reasoning skills and their ability to apply knowledge to solve realistic, meaningful problems (Lane, 2010) They emulate the context or conditions in which the intended knowledge or skills are actually applied (American Educational Research Association, 1999) Erik Bergström & Helen Pehrsson 5

solve realistic, meaningful problems (Lane, 2010) They emulate the context or conditions in which the intended

6 Goal How do the students perceive performance assessments? Literature describe it as a more stressful situation than a test with pen and paper (Wästlund et al Asked for by companies, but do the students perceive it as a positive aspect? Can we have a clear progression in the performance assessments? Erik Bergström & Helen Pehrsson 6

7 Research Questions Do the students perceive the performance assessment as more stressful than examination with a written lab report? Do the students perceive the performance assessment as a positive aspect to highlight when applying for jobs? How do the students experience the progression of performance assessment between courses? Erik Bergström & Helen Pehrsson 7

Do the students perceive the performance assessment as a positive aspect to highlight when

8 Method and Context A soft case study as described by Walsham (1993) Interpretative (understanding) as described by Braa and Vidgen (1999). Case study context Network and Systems Administration study program at University of Skövde Erik Bergström & Helen Pehrsson 8

9 Data Collection Document studies (three courses) Course plans Examination criteria Lab documentation Questionnaires First year students in the NSA study program First year students in the web development and computer science study programs Second and third year students of the NSA program Group discussions Students Educators Erik Bergström & Helen Pehrsson 9

students in the web development and computer science study programs Second and third year

10 Case Study Context Lab Setup One student = two computers, three screens, several virtual machines Own subnet VPN possibilities Teachers can push images, create faults in setups Erik Bergström & Helen Pehrsson 10

subnet VPN possibilities Teachers can push images,

11 Performance Assessment Lab presentation prerequisite ~2.5 4 hours Their own disks and lab systems Open book examination (system documentation, manuals, Internet, ) No conversations in individual assessments Erik Bergström & Helen Pehrsson 11

examination (system documentation, manuals, Internet, ) No

12 Performance Assessments Progression Course: Computer Fundamentals 2 individual performance assessments plus no additional documentation Level 1: Repeat steps performed in the lab Tasks such as: add user, set correct file system permissions, schedule backup Time limit: 2.5 hours Course: Windows administration I 1 individual performance assessment plus system documentation Level 2: Create new things on a prepared system Tasks such as: Install feature, perform backup, restore backup, centrally distribute client settings, shared folders and file system permissions Time limit: 2.5 hours Course: Windows administration II 1 group performance test plus system documentation, system design report Level 3: Prepare a system to work in an environment created by the educators Task: Hardware failure, restore after alert from monitoring system Time limit: 3 4 hours Erik Bergström & Helen Pehrsson 12

5 hours Course: Windows administration I 1 individual performance assessment plus system documentation Level 2: Create new things on a prepared system Tasks such as: Install feature, perform backup,

13 Results (1) 100% 90% 80% 70% 60% 50% 40% a b 30% 20% 10% 0% Yes No Don't know a) Does performance assessments measure your practical skills? b) Do you think it's good to be examined by performance assessments? Erik Bergström & Helen Pehrsson 13

14 Results (2) 70% 60% 50% 40% 30% 20% 10% 0% Yes No Don't know Do you perceive performance assessments as more stressful than submission of a written lab report? Erik Bergström & Helen Pehrsson 14

15 Results (3) 80% 70% 60% 50% 40% 30% (1) (2) (3) 20% 10% 0% Yes No Don't know Do you perceive performance assessments as more stressful than submission of a written lab report? (Group 1=NSA year 1, Group 2= Web and CS year 1 and Group 3= NSA year 2 and 3) Erik Bergström & Helen Pehrsson 15

16 Results (4) 100% 90% 80% 70% 60% 50% 40% 30% 20% 10% 0% Yes No Don't know Do you think that the performance assessments in the NSA program is a positive aspect to point out when you apply for jobs? Erik Bergström & Helen Pehrsson 16

in the NSA program is a positive aspect to point out when

17 Results (5) 90% 80% 70% 60% 50% 40% 30% 20% 10% 0% Yes No Don't know Do you feel that there has been a progression in the performance assessments in the program? Erik Bergström & Helen Pehrsson 17

18 Results (6) Group discussion and free text answers from students Performance assessment in the first course served as a soft start of the studies Proves in a better way the student have learned the course goals than repetitive how to reports based on manuals and Google The only real good argument [educators] have for examining lab reports is that it gives training for the FYP Many companies wish for practical knowledge when applying for jobs add even more performance assessments. I think all courses with a practical part should have performance assessment Progression is important and there are clear repetitions between the courses, but it would be good with even clearer steps Better preparations from the educators so there are no problems with e.g. faulty NICs Erik Bergström & Helen Pehrsson 18

wish for practical knowledge when applying for jobs add even more performance assessments.

19 Results (7) Group discussions with educators Results from performance assessments show that skilled students are normally theoretically skilled as well Measures practical skills Catch students not skilled enough Better than a written lab report Less risk of plagiarism Harder to fail students face to face Need more synchronization to get a progression of lab reports in other courses (to prepare for FYP) Erik Bergström & Helen Pehrsson 19

written lab report Less risk of plagiarism Harder to fail students face to face Need more synchronization to get

20 Conclusion Surprising low differences between student groups Perceived as less stressful than expected Give the students more performance assessments, but not only performance assessments! Takes some time to prepare, set up labs, more hardware dependent, need skilled teachers (new problems might arise during assessment), does not save time Erik Bergström & Helen Pehrsson 20

Takes some time to prepare, set up labs, more hardware dependent, need skilled teachers (new

21 Next step for us To investigate the uncritical approach among teachers and students of how certifications are used and applied within higher education. Literature is uncritical to certification Students accept anything as long as it leads to certification Erik Bergström & Helen Pehrsson 21

22 References Braa, K., & Vidgen, R. (1999). Interpretation, intervention, and reduction in the organizational laboratory: a framework for in context information system research. Accounting, Management and Information Technologies, 9(1), Brookshire, R. G. (2000). Information technology certification: Is this your mission? Information Technology, Learning, and Performance Journal, 18(2), 1 2. Lister, R., Adams, E. S., Fitzgerald, S., Fone, W., Hamer, J., Lindholm, M., McCartney, R., Moström, J. E., Sanders, K., Seppälä, O., Simon, B. & Thomas, L A multinational study of reading and tracing skills in novice programmers. SIGCSE Bull., 36, McGill, T., & Dixon, M. (2007). Information Technology Certification: A Student Perspective Integrating Information & Communications Technologies Into the Classroom (pp ): IGI Global. Morris, G., Fustos, J., & Haga, W. (2012). Preparing for a Career as a Network Engineer. Information Systems Education Journal, 10(1), Ray, C. M., & McCoy, R. (2000). Why Certification in Information Systems? Information Technology, Learning, and Performance Journal, 18(1), 1 4. Walsham, G. (1993). Interpreting Information Systems in Organizations: John Wiley & Sons, Inc. Wästlund, E., Reinikka, H., Norlander, T. & Archer, T Effects of VDT and paper presentation on consumption and production of information: Psychological and physiological factors. Computers in Human Behavior, 21, Erik Bergström & Helen Pehrsson 22

March 19 20 2014, Dalarna University, Falun, Sweden

NGL 2014 NEXT GENERATION LEARNING CONFERENCE March 19 20 2014, Dalarna University, Falun, Sweden Conference Summary ISBN: 978-91-85941-58-2 Foreword On the 19th-20th March, the Next Generation Learning