Predicting Academic Performance in the School of Computing & Information Technology (SCIT)

Predicting Academic Performance in the School of Computing & Information Technology (SCIT) Paul Golding and Sophia McNamarah University of Technology, Jamaica cashmere@cwjamaica.com, smcnamarah@utech.edu.jm Abstract What determines academic performance? Prior research shows that standardized measures such as aptitude (e.g. SAT scores), prior academic performance, effort and motivation explain a significant portion of the variation in class When universities in the United States determine which students to admit, typical criteria include SAT, ACT or other achievement scores and high school GPA. At the University of Technology, Jamaica in the School of Computing & Information Technology, the main admission criteria are appropriate score in an aptitude test and passes in at least five Caribbean Examination Council subjects including Mathematics and English. This study examines the relationship between students demographic attributes, qualification on entry, aptitude test scores, performance in first year courses and their overall performance in the program. The study has implications for the School s admission policy. The results should help us to identify an optimal set of admission indicators, which have the potential of predicting students Index Terms Information System Education, Predicting Academic Performance. INTRODUCTION The School of Computing & Information Technology (SCIT) at the University of Technology, Jamaica (UTech), has had over seven hundred (700) applicants per year to its programs over the last few years. Of this number approximately two hundred and twenty five are accepted. The School offers two programs a Bachelor of Computing & Information Technology (BsCIT) and a Bachelor of Computing with Management Studies (BCMS). The BsCIT course is the more popular of the two attracting approximately two thirds of the applicants. The selection of students for both programs involve assessing the applicants Caribbean Examination Council (CXC) exam and/or General Certificate Ordinary Level (GCE O Level) exam results and the score attained in the SCIT s aptitude test. The current matriculation requirements for the programs are five (5) CXC or GCE O Level subjects inclusive of Mathematics and English and preferably a science subject. With the applicant to admission ratio of approximately 3:1 questions have arisen about the admission criteria for the program. Is SCIT selecting the cream of the crop of applicants and to what extent does the admission decision variables predict student academic performance? In addition, the failure rate in the Introduction to Programming (ITP) course has risen from 30% to just over 40%. This is a cause for concern as ITP is a foundation course and a prerequisite for several other programming courses including Programming using C, Data Structure using C, and Advance Programming using Java. Although specific actions have been taken to address the course failure rates problems [1], [2], the question of whether or not it is possible to predict which student will perform well on the programs has persisted. If we can identify the factors that indicate which students are more likely to succeed in the programs then we will able to optimize the selection process. The objective of this study is to determine the relationship between the students personal attributes (such as age, gender) and other factors (such as qualifications held on entry and scores in the aptitude test) and their performance in SCIT program. Answers to some specific questions are sought, for example: Do age, gender, CXC qualification, aptitude test scores and experience affect performance? Does mathematical reasoning ability and mathematical background determine success in this program? What is the profile of the students with the highest grade point average (GPA)? REVIEW OF LITERATURE Prior research indicates that standardized measures of aptitude (e.g. SAT and ACT scores), prior academic performance (e.g. high school GPA) and effort or motivation explain a significant portion of the variation in student performance [3], [4], [5]. Even though these variables are helpful in predicting success in Computer Science it appears that they could also predict success in other fields [6]. SCIT admission criterion includes an aptitude test, which is a proxy for the SAT exams. The proxy for high school GPA is passes in the CXC and GCE exams. According to [7] prior experience in programming provide a significant indication of how students perform in the computer science program and/or subsequent programming courses. Taylor and Mounfield found that prior computer science course experience of any kind is significant over no prior knowledge. They concluded, (in support of Ramberg [8]), that prior exposure whether at the high school or college level is an important factor to students success in computer science programs. The SCIT admission does not include passes in computer science, as this subject is not taught at the S2H-16

high school level. One factor that will be considered in this study is prior knowledge in information technology both at the ordinary and advanced level. Several of the reviewed studies showed that success in Mathematics was a good predictor of success in computer science [9], [10], [3], [11], [12]. In fact this is a generally accepted notion, and some colleges have specific mathematics requirements for those students doing computer science. There is a belief that the concepts which a student has to comprehend in order to master mathematics problems are similar to those for programming [11]. In [10] and [13] specific reference is made to calculus. They indicate that the best predictor in their study, for success in computer science is success in a calculus course. In [13] the conclusion is that the skills developed in learning and succeeding in calculus are similar to those needed for mathematics, a similar statement to that made by [11]. One of the specific requirements for admission to the SCIT is a pass in math at the ordinary level. This study will also examine predictive capability of passes of math at the advance level. In general the reviewed research found little or no correlation between demographic data and success in computer science, for example [14]. Although earlier studies (especially those prior to 1975) indicated that being male (gender) impacted on computer science success, later studies have found no correlation between gender and computer science success. Other demographic factors and personal background did not prove significant. In [15] twelve factors were reviewed as potentially predictive to success in computer science. They included math background, attribution for success/failure (i.e. how students explained or justified their performance), domain specific efficacy (how knowledgeable a person is in terms of their chosen discipline), encouragement and comfort level in the course, work style preference, previous programming experience, and gender. Two instruments were used in this study: a questionnaire and a self-efficacy scale. A pilot test was conducted to ensure that the method of testing was valid; in addition professionals such as experts in the field of psychology and seasoned professors were involved, all in an effort to add validity to the study. Wilson [14] found that the most important factors that contributed to success were comfort level, mathematics background and attribution for success/failure. These are interesting finding for whilst the correlation to mathematics background is predictable, attribution to failure/success and comfort level adds a new dimension to the research framework and points us in a different direction. One area of variability in the reviewed studies was the data gathering methods. Some researchers relied purely on surveys, others used test scores, some used secondary data, and others used a combination of the above. In [16] the researcher compares his work with that of another, and considered the low return rate of questionnaire and student self-selection as negative features. He therefore sought to eliminate these from his project. He relied heavily on data from test scores, assignments and final exams. We use a similar approach in this study. METHODOLOGY The purpose of this study is to determine what factors predict academic performance in the SCIT programs; this will help us to determine the appropriateness of the matriculation requirements. Based on the literature reviewed the following is posited in the null: Academic performance will not be affected by the individual s gender. Prior academic performance in CXC and GCE will not affect current academic Prior academic performance in mathematics and sciences courses will not affect current academic performance Aptitude test grades will not affect current academic In Jamaica the prevailing view is that the more passes that a high school student have at the ordinary level and or the advanced level the more likely they are to succeed. It is therefore not uncommon for students to have taken and passed up to 10 subjects at the ordinary level. Generally students take a minimum of five subjects at the ordinary level. We therefore posit that The number of passes at the ordinary or advance level will not affect academic As it relates to prior-related studies researchers have found mixed results. Numerous studies have been done in economics showing that prior economics or calculus classes tend to be associated with performance in a college-level economics class [17]. The following is therefore posited. Prior academic performance in information technology courses will not affect current academic performance The research was done as a longitudinal study that tracked students that were accepted to in the Bachelor of Science in Computing and Information Technology (BsCIT) at UTECH in 1999-2000 academic year. Student s grades for information technology and computer science related courses were recorded for the four-year period 1990 2004. The students GPA for each of the four years were recorded. DEMOGRAPHICS The sample consisted of 96 students, 68 male and 28 females. Recall that to matriculate to the SCIT programs students are required to take an aptitude test and to have passed at least five subjects at CXC and/or GCE O Level. The subjects should include English Language, Mathematics and preferably a science subject. Fifty-five (57%) of the matriculated candidates passed between 7 and 10 O Levels and 35 (approximately 37%) had between 4 and 6 subjects, just about six (6%) had between 1 and 3 subjects. SCIT does allow for a small percentage of S2H-17

mature students to enter without the required subjects providing they have the necessary industry experience. See table 1 below. Table 1: Number of O-Levels Valid Frequency Valid Cumulative 1-3 6 4.4 6.3 6.3 4-6 35 25.9 36.5 42.7 7-10 55 40.0 57.2 100.0 Total 96 71.1 100.0 The data from table 2 below indicate that the majority of the matriculated students did not have A Levels. Sixty-four (66.7%) of the students had no passes at A Levels, while twelve (12.5%) had one subject and another twelve (12.5%) had two subjects. Table 2: Number of A-Levels Valid Frequency Valid Cumulative 0 64 47.4 66.7 66.7 1 12 8.9 12.5 79.2 2 12 8.9 12.5 91.7 3 6 4.4 6.3 97.9 4 2 1.5 2.1 100.0 Total 96 71.1 100.0 Regarding the GPA only four students (4.2%) received a first class honors, 18 students or (approximately 19%) received upper second class, 48 students or (50%) received lower second-class honors and three students or (3%) received a pass. Twenty-three students (24%) discontinued. There were no failures. We examined a number of predictor variables; gender, scores in the aptitude test, passes in Chemistry, Physics, Math, additional Math, GCE Advanced Level (A Level) Math, Accounting, Information Technology at both the O Level and A Level, and the number of CXC, O Level and A Level passes. We also included passes in first year first semester computing courses; Introduction to Programming (ITP), and Computer Logics and Digital Design (CLDD). The dependent variable in this study was student s final year GPA. GPA was divided into five categories, first class (3.45 4), upper second class (3.05 3.44), lower second class (2.40 3.04), pass (1.70 2.39) and fail (0 1.69). The data was analyzed using stepwise multiple regression analysis. The stepwise approach eliminates variables already in the model that are no longer significant predictors. RESULTS This section discusses the effects of the specified variables on performance in the SCIT. We first tried to determine to what extent the aptitude test, and science subjects (Chemistry and Physics at O Level and Physics at A Level related to academic The results indicated that neither of these factors had any effects on GPA, as all were removed from the stepwise analysis. The second model examined the predictive value of Mathematics (both O Level and A Level), Additional Mathematics, Principles of Accounting (both O and A-level) and Information Technology (both O and A levels) Table 3: Model 2 Summary Model R R Square Adjusted R Square 1 0.276 a 0.076 0.066 1.47686 2 0.345 b 0.119 0.100 1.44999 a. Predictors: (Constant), Additional Mathematics b. Predictors: (Constant), Additional Mathematics Information Technology A Levels The summary indicates that Additional Mathematics explains only 7.6% of the students performance with a significance of.006 (see table 3) and Information Technology and Additional Mathematics explains approximately 12% of the students performance with a significance of.003. The correlation (R) was weak.276 for additional mathematics and slightly improved.345 for both additional mathematics and information technology. While both Additional Mathematics and Information Technology have significant p-value (0.006 and 0.003 respectively) they are both weak predictors of Table 4: ANOVA for Model 2 Model Sum of df Mean F Sig. 1 Regression 16.935 1 16.935 7.764 0.006 a Residual 205.024 94 20181 Model Sum of df Mean F Sig. 2 Regression 26.427 2 13.214 6.285 0.003 b Residual 195.531 93 2.102 a. Predictors: (Constant), Additional Mathematics b. Predictors: (Constant), Additional Mathematics Information Technology A Levels The third model examined the number of A-Level subjects passed and the number of O-Level subjected passed. The expectations is that persons who have passed more O and A level subjects would perform better than those who did not. The results indicate that these factors were also poor predictors of The number of passes in A-levels could only explain approximately 5% of performance, while the correlation (R) was.220 (see table 4). The p-value was 0.016, which is not significant. S2H-18

Table5: Model 3 Model R R Square Adjusted R Square 1 0.220 a 0.048 0.038 1.49906 a. Predictors: (Constant), Number of A Levels The fourth model examined the result of first year courses in Introduction to Programming (ITP), C-Programming, Computer Logics and Digital Design (CLDD) and Introduction to Networking as a predictor of overall Table 6: Model 4 Model R R Square Adjusted R Square 1.650 a 0.423 0.417 1.16752 2.686 b 0.471 0.460 1.12337 3 706 c 0.499 0.483 1.09943 a. Predictors: (Constant), Computer Logics and Digital Design b. Predictors: (constant), Computer Logic and Digital Design, C Programming c. Predictors: (Constant), Computer Logic and Digital Design, C Programming, Introduction to Networking Table 7: ANOVA Model 4 Model 1 Sum of df Mean F Sig. Regression 93.828 1 93.828 68.834.000 a Residual 128.131 94 1.363 Model 2 Sum of df Mean F Sig. Regression 104.596 2 52.298 41.442.000 b Residual 117.362 93 1.262 Model 3 Sum of df Mean F Sig. Regression 110.753 3 36.918 30.542.000 c Residual 111.205 92 1.209 a. Predictors: (Constant), Computer Logics and Digital Design b. Predictors: (Constant), computer Logics and Digital Design, C Programming c. Predictors: (constant), Computer Logics and Digital Design, C Programming, Introduction to Networking Model 4, first year courses were generally better predictors of CLDD explained 42% of performance at significance level of (.000), with a correlation of.65. CLDD and C-programming explained 47% of performance at a significance level of (.000) with an increased positive correlation of.686 and CLDD, C-Programming and Introduction to Networking explains approximately 50% of performance at a significance level of (.000) with an even to note that introduction to programming was not a factor in determining DISCUSSION The objective of this study was to determine the ability of SCIT s matriculation requirements in determining academic success. The most consistent finding was the low explanatory power of the linear models, which suggest that the matriculation requirement of the SCIT, UTECH program is a weak predictor of academic The first model which included the aptitude test, chemistry and physics both at the O Level and A Level were excluded as having no predictive powers. Prior research [18] and [3] indicated that SAT scores explained a significant portion of the variation in student performance, the aptitude test however was not a predictor. Previous researchers [9] reported that pre-college students who were successful in high school mathematics and science would probably be successful computer science students. This prior result was uncorroborated by our findings. The science courses taken at high school level were not a predictor, neither was O Level mathematics. Additional-Mathematics accounted for only 8% of the variability. Previous studies have found mixed results about the relationship between performance in prior related classes and performance in later classes. In this research the approach to this question was two-fold. The first used information technology (as a prior related course) to test the matriculation predictability. Information technology at the O Levels were not a factor while at the A Levels accounted for only 2% of the variability with a insignificant p-value of.404. This finding is consistent with [19] who found that prior courses in information systems did not predict information systems GPA. When additional-mathematics and a-level information technology was used as a model, it accounted to an improved but unimpressive 12% of the variation and a p- value of.003. The paper also examined the extent to which results of first year college information systems courses were a predictor of GPA. First year information systems subjects particularly CLDD, C-Programming, and Introduction to Networking provided an explanation of student s CLDD accounted for 42% of the variability with a positive correlation of.650, CLDD and C-programming accounted for 47% with a correlation of.686 and CLDD, C-Programming and Introduction to Network accounted for approximately 50% with a correlation of.706. All three models have p-value of.000. These findings are congruent with [17]. While this results would not help to determine matriculation requirements it does help to identify students who might be at risk of failure. An interesting result of this study is that ITP, which there is a concern about the increasing failure rate, is not a factor in predicting This is a curious result as ITP is considered the prerequisite for all the other programming courses. There is a tendency in Jamaica to accept students who have more passes in O Levels and or A Levels, the model higher correlation of.706. (See tables 5 and 6). It is important S2H-19

suggest that passes in A-Levels account for only approximately 5% of the variability in While gender is not one of the matriculation requirements we found that gender was not a factor in predicting This result is similar to [17] and [18]. CONCLUSION AND RECOMMENDATION Several studies have been done attempting to determine what matriculation factors determine academic success in general and success in information systems in particular. Based on our results the answer to this question remains elusive. According to [20] one highly respected school explicitly indicated that in a survey that it has no specific admission rules simply because it cannot relate such rules statistically to academic One revealing result is that the number of passes in O Levels and A Levels were not a factor in predicting academic success, the quality of the passes at (A s, B s or C s) maybe a better predictor of What this may also suggest is that SCIT maybe using the best available quantitative predictors for performance in their admission process. This study has reinforced that success in first year college-level information system courses are associated with overall Lecturers/professors should monitor these signals as they serve as a warning for potential failure. The students sampled were matriculated for Bachelors of Science of Science in Information Technology (BSCIT), which is programming intensive. A number of these students are not good programmers and therefore do not do well in the course. In the evenings, a part-time program the Bachelors of Science in Computing and Management Studies (BCMS), which is not as programming intensive, but includes a range of management studies courses, is offered. The suggestion is that students who have done well in the foundation programming courses should be encouraged to continue in the BSCIT program, while students who have not grasped the concepts should be channeled in the BCMS program, which should also be offered on a full-time basis. The fact that none of the models in this study accounted for more that 12% of the variation in academic performance is notable, what it also indicates is that the task of finding effective predictors of academic performance remains incomplete. There are several limitations of the study. The participants were from one university and one cohort in the SCIT, further research could include participants from both programs and other institutions to rule out program or university bias. The study excluded factors such as personal motivation, socio-economic factors and the quality of the educational experience. These factors could be included in future research. ACKNOWLEDMENT The authors would like to thank our Research Assistant Kimberly Spencer for the gathering and input of the data. REFERENCES [1] Facey-Shaw, L & Golding P., Effects of Peer Tutoring and Attitude on Academic Performance of First Year Introductory Programming Students (Unpublished) [2] McNamarah, S., Pyne, R., Teaching a First Level Programming Course: Strategies for Improving Students Performance Journal of Art Science & Technology, 2004 pp 42 49 [3] Eskew, R.K., & Faley, R.H. Some Determinants of Student s Performance in the First College-Level Financial Accounting Course. The Accounting Review LX111 (1), 137-147 [4] Grabe & Latte, 1981Author's Last name, First initial, Middle initial, "Title", Journal or book (italics), Vol, No #., date, pp. Need to look up these references. [5] Hostetler, T, R, Predicting Student Success in an Introductory Programming Course, ACM SIGCSE Bulletin, Vol. 15, No.# 3, September 1983, pages 40-49 [6] Corman, L, S, Cognitive Style, Personality Type and Learning Ability as Factors in Predicting the Success of the Beginning Programming Student, ACM SIGCSE Bulletin, Vol. 18, No# 4, December 1986, pages 80-89 [7] Taylor, H, G, Mounfield, L, C, The Effect of High School Computer Science, Gender, and Work on Success in College Computer Science, Proceedings of 20 th SIGCSE Technical Symposium in Computer Science Education, Vol 21, Issue 1, February 1989, pages 195-198. [8] Ramberg, P. A New look at an Old Problem: Keys to Success for Computer Science Students, SIGCSE Bulletin, Vol. 18, No# 3, September 1986, pages 36-39. [9] Konvalina, J, Wileman, S, S, Stephens, L, J, Math Proficiency: A key to Success for Computer Science Students, Communications of the ACM, Vol 26, No# 5, May 1983, Pages 377-382 [10] Chowdhury, A, A, Nelson, C, V, etal, Predicting Success of a Beginning Computer Course Using Logistic Regression, Proceedings of the 15th annual conference on Computer Science, February 1987, pages 449 [11] Byrne, P, Lyons, G, The Effect of Student Attributes on Success in Programming, ACM SIGCSE Bulletin, Proceedings of the 6th annual conference on Innovation and technology in computer science education, Volume 33 Issue 3, June 2001, pages 49-52 [12] Campbell. P, F, McCabe, G, P, Predicting the Success of Freshmen in a Computer Science Major, Communications of the ACM, Vol. 27, No#. 11, November 1984, pages 1108-1113 [13] Chapstick, C, K, Gordon, J, D, Salvadori, A, Predicting Performance by University Students in Introductory Computing Courses, ACM SIGCSE Bulletin, Volume 7 Issue 3,, September 1975, pages 21-29 [14] Evans, G, E, Simkin, M, G, What Best Predicts Computer Proficiency, Communications of the ACM, Vol. 32, No#. 11, November 1989, pages 1322-1327 [15] Wilson, B, & C, Shrock, S, Contributing to Success in an Introductory Computer Science Course: A Study of Twelve Factors, Proceedings of the thirty-second SIGCSE technical symposium on Computer Science Education,, Pages 184 188, 2001 [16] Mazlack, L, Identifying Potential to Acquire Programming Skill, Communications of the ACM, Vol 23, No# 1, January 1980 [17] Kruck, S.E., & Lending, D. Predicting Academic Performance in an Introductory College-Level IS Course, Information Technology, Leraning and Performance Journal, Fall 2003, pages 9 15, [18] Camara, W. J., & Echternacht, G., The SAT and High School Grades: Utility in Predicting Success in College, The College Board Research, Research Notes, The College Board Office of Research and Development. [19] Sexton, R.S. Hignite, M. A., Margavio, & T. Satizinger, J Neural Networks Refined: Using a Generic Algorithm to Identify Predictors of IS Students Success Journal of Computer Information Systems, 41 (3) 2001 pages 42 47. [20] Page, A. & West, R.R., Evaluating Student Performance in Graduate Schools of Business, Journal of Business 42, 1, 1969 S2H-20