Does Discussion Promote Learning Outcomes? Analysis of an Online Criminology Class: Research Note

Transcription

1 JOURNAL OF CRIMINAL JUSTICE EDUCATION, 2013 Vol. 24, No. 3, , Does Discussion Promote Learning Outcomes? Analysis of an Online Criminology Class: Research Note Steven Stack The extent to which discussion promotes learning outcomes is a relatively unexplored area in teaching. The present paper contributes to the literature by assessing the extent to which student use of a discussion board is predictive of grades. The sample consists of 341 students who completed an online course in criminology. Learning outcomes are measured in terms of final examination scores. The number of participations or hits (N = 6,934) on a web-based discussion board is used as a measure of the quantity of class participation. Overall, hits on the discussion board did not predict grades. However, among low achievers, the greater the participation in the board, the higher the grades. Using data on a web-based discussion board as a measure, the present study reports a link between discussion and achievement for low achievers a result consistent with a previous study in online physics classes. Future work is needed to explore such moderating effects in the link between the use of other web-based utilities, such as blogs, on student performance. Online instruction has rapidly developed in just the last decade. A majority of students in higher education have now taken at least one online course and their numbers are growing (Allen, Seaman, & Garrett, 2007; Forte & Root, 2011). A survey of 379 criminal justice departments in 2007 determined that 80 offered classes online. Fully 66% of these offered an entire degree program online (Hummer, Sims, Wooditch, & Salley, 2010). In the context of growth in online instruction, it is increasingly important to assess the learning outcomes in online instruction. While online instruction has grown substantially in the last decade, there is relatively little research assessing what technologies and teaching methods are best to promote academic achievement. This is also true in criminal justice. For example, in the period from , the Journal of Criminal Justice Education published 222 articles. However, none performed an empirical test of what factors predict academic performance in online classes. In contrast, 16 studies were published exploring predictors of academic performance in traditional classes (Burruss & Furlow, 2007; Bushway & Flower, Ó 2013 Academy of Criminal Justice Sciences

2 ANALYSIS OF AN ONLINE CRIMINOLOGY CLASS ; Dabney, Green, & Topali, 2006; Gordon, Barnes, & Martin, 2009; Hartman, Bjerregaard, & Lord, 2009; Kessler & Swatt, 2001; Kunkel, 2003; Martin & Hanrahan, 2004; McKee, Mallory, & Campbell, 2001; Proctor, 2002, 2006; Reisig & DeJong, 2005; Schroeder, Grohe, & Pogue, 2008; Stack & Kelley, 2002; Stack,2005; Tibbetts, 2003). The present study contributes to the literature by addressing this gap in the literature. It performs a test of hypotheses concerning the impact of participation in a popular online instructional utility, the discussion board, on final grades in an online course in criminological theory. Background: Predicting Academic Performance The present study deals with the intersection between two bodies of research on academic performance: (1) academic achievement in criminal justice classes, and (2) academic achievement in web-based courses. The latter work has been done entirely in fields outside of criminal justice. A third literature on academic performance by style of delivery (online vs. traditional classes) is briefly reviewed as background on the challenge of online instruction to the traditional mode of delivery, the offline classroom. Research on Academic Performance in Criminal Justice: Traditional Classes A total of 222 articles published in the Journal of Criminal Justice Education during were reviewed for the state of knowledge of factors associated with academic performance in the criminal justice context. All studies had a measure of academic performance (e.g. grades on examinations) and tested at least one hypothesis to explain the variability in performance. All 16 investigations were done using traditional classroom instruction none were based on online classes. However, several did use technological instructional utilities in the context of offline instruction. The results on this subset of studies were mixed. Technological utilities that contributed to better academic performance included the use of web-based writing software in a CJ writing class (Schroeder et al., 2008), automated frequent quizzes in a research methods class (Bushway & Flower, 2002), and computer-assisted instruction, such as power points, in two courses but not criminological theory (Kunkel, 2003). One web-based utility in these traditional courses did not enhance academic performance. A supplemental web-based tutorial on chi square in a research methods class (Burruss & Furlow, 2007) did not create any significant impact on overall exam grades. None of these investigations assessed the effect of a discussion board on academic performance. Eleven investigations assessed non-technological factors on academic performance. These conditions included, as noted in Table 1, tutoring; compulsory attendance; class size; opportunities to rewrite course assignments; test scores

3 376 STACK Table 1 Characteristics and findings of 16 previous JCJE articles on predictors of academic performance: lead author, date, sample size, dependent variable(s), core independent variable, and main findings, (of 222 articles published) Author (year) N Dependent variable, course Core independent variable Main findings Bushway (2002) Burruss (2007) Dabney (2006) Gordon (2009) Hartman (2009) Kessler (2001) Kunkel (2003) 178 Failure rate in research methods 70 Grades on Chi Square Exam, research methods Tutoring, automated quizzes, required class attendance visual, web based tutorial on chi square 176 GPA Member, Learning Community 245 Exam grades in research methods Class Size, Large ( ) vs. small (25) 198 GPA all classes Transfer vs. native student 71 Final exam score, research methods 536 Final exam, crime theory, courts, class/race & Justice courses Opportunity to rewrite course assignments Classes & w/o CAI (computer assisted instruct.) Tutoring and automated quizzes reduced the risk of failure. Required attendance did not. Visual tutorial did not enhance overall exam grade. Not significant after 1st term Exam scores higher in large class (19.6 vs. 18.3/30 questions). No significant difference between transfer & natives The greater the number of rewritten assignments, the higher the exam grade. CAI improves exam grades in 2 CJ courses but not in criminological theory. McKee 94 Graduate GPA GRE, Undg GPA Both predict GPA Martin 129 GPA SAT, Effort Both predict GPA (2004) Proctor (2002) Proctor (2006) Reisig (2005) Schroeder (2008) Stack (2002) Stack (2005) Tibbetts (2003) 22 Final Grade, statistics 49 Exam & Homework scores, statistics 206 GPA Graduate students, overall 91 Grades in CJ writing class 70 GPA graduate students 222 Grades on paper, Final Exam Crim Theory 130 Exam Grade, Intro CJ Exposed to SPSS vs. EXCEL CJ major vs. other majors GRE, UNDG GPA Use of Writing Technology GRE, UNDG GPA Doing Extra credit paper on crim theory Enrollment in Freshman Interest Group (FIG). Students using EXCEL scored higher. Non CJ majors score higher GRE & Undg GPA predict Graduate GPA Classes with writing technology have higher mean grades GRE-V & UNDG GPA predict graduate GPA Extra credit predicts grade on term paper, but not final grade. Students in FIGS had higher grades in CJ class.

4 ANALYSIS OF AN ONLINE CRIMINOLOGY CLASS 377 including the SAT, GRE, and past GPA; type of statistical software used to teach statistics; major field of the student; doing extra credit; and enrollment in a Freshman Interest Group, which provided networks for learning. Online Classes and Academic Performance The literature review could not locate any investigations of academic performance in online criminal justice classes. There are only a few investigations specific to an online course that assess the link between the degree of use of web-based learning tools and scores on independent outcome measures such as final examinations. Several of these have relevance for the use of discussion boards. Armstrong (2010), in an analysis of students in an online English as second language class, found that the number of hits on a discussion board did not predict grades on other assignments. Lu, Yu, and Liu (2003) determined that the number of usages on a discussion board combined with other utilities did not predict academic performance among 76 students in an online graduate course in management information systems. In a similar fashion, perceived effectiveness of the discussion board did not predict academic performance in an online management information systems class (McFarland & Hamilton, 2005). None of these studies explored possible interaction effects. However, one done for physics classes found that the use of online utilities enhanced the grades for one special group of students the low achievers (Shieh, Chang, & Liu, 2011). Academic Performance: Comparisons of Online vs. Traditional Classes A third body of research is devoted to comparative analyses of academic performance between online classes and traditional classes. These generally compare the results of a common final exam, or set of exams, in the same course using an experimental and control group. Herein, the first is the mean score for an online class and the latter is the mean score for the traditional, or live classroom (e.g. Forte & Root, 2011; Hylton, 2006; Parsons-Pollard, Lacks, & Grant, 2008; Sami, 2011; for a review see Shachar & Neumann, 2003). A key question is: do students in online classes perform better, worse or the same as students in the traditional classroom? Nearly all of over 80 such investigations were done in fields outside of criminal justice (an exception is a recent study by Parsons-Pollard et al., 2008). A meta-analysis of 86 investigations performed during the period of determined that the academic performance (grades) of students in online sections of a course was higher than that of students in a matched control group based on traditional offline classrooms. Students in the online section, on average, received grades on a shared final examination that were one third of a standard deviation higher than

5 378 STACK grades received by the students in the control group, the traditional classroom (Shachar & Neumann, 2003). While considerable evidence suggests that academic performance is higher in online classes than offline classes, it is not clear what accounts for this generalization. In addition, it is not clear if the factors that explain variability in academic performance in offline classes will be relevant in an online class. The present study addresses the second issue. We explore the possible impact of one widely used, but understudied, web utility, the discussion board, on academic performance in an online course, criminological theory. While several studies suggest that discussion boards have no apparent effect on academic performance in other fields (Armstrong, 2010; Lu et al., 2003; McFarland & Hamilton, 2005), this may not be the case in criminal justice. In addition, previous investigations generally lump all students together. However, one which sorted out low achievers determined that the quantity of use of a discussion board improved the academic performance of low achievers in the online class (Shieh et al., 2011). The present study also explores this moderating effect. The Nature of Discussion, Online vs. Offline, & Hypotheses One common assumption of traditional classroom teaching is that class discussion facilitates learning outcomes. However, classroom discussions often involve only a small percentage of the students who are present. Importantly, there is not a wealth of empirical data showing that such participation is linked to standard measures of learning such as grades on examinations. As a corollary, there is little information on what subtypes of types of students or special contexts might support a link between participation in discussions and learning. For example, possibly it may work for the more challenged or low achievers in the class, or by categories of multiple intelligences such as those comprised of verbal and visual learners (Johnson & White, 2002). The online learning environment has some advantages for the empirical assessment of a possible link between discussion and learning outcomes. First, web-based discussion boards have the advantage of permanently storing all entries from students. The texts can be retrieved and assessed. Second, the authors of each comment on the board can be identified. Third, the number of hits from each student can be counted and, if desired, the content of each statement can be scrutinized. These key features of discussions on web-based boards are in sharp contrast to discussions in the traditional classroom. Comments from students in the traditional classroom are generally lost and there is generally no permanent record. In the traditional classroom, discussion is often fast-paced, and several students may be talking at once. There are interruptions. The names of students may not be known or recorded incorrectly, and so forth. There are also important differences between the scope of participation in a web-based class and the traditional classroom. In the present analysis,

6 ANALYSIS OF AN ONLINE CRIMINOLOGY CLASS 379 participation in discussion is required. This is generally not the case in traditional classrooms. As such, the more timid or introverted students who would be less apt than the more aggressive extroverted students to participate in discussions in the traditional, live classroom, will be heard from in an online venue. Often, in traditional or live classroom environments, only a few students dominate the typical class discussion. The present investigation tests two hypotheses: H1: The greater the participation in the discussion board, the higher the grades on examinations. Most research on the possible impact of participation in discussion boards on final exam grades has not tested for contextual effects (e.g. Armstrong, 2010; Lu et al., 2003; McFarland & Hamilton, 2005). However, in an analysis of a class in physics, the use of related technological, web-based learning tools, while having no impact on final grades overall, did enhance the academic performance on final examination within the lower third of the class, or the low achievers (Shieh et al., 2011). Following the findings of Shieh et al. (2011) in physics, we also test for the moderator of low vs. high achievement (as measured by examinations) on the relationship between discussion board participation and examination grades. Subjects H2: The relationship between discussion and grades will be stronger for low achievers than high achievers. Methodology All subjects were enrolled as students at a Carnegie Research Extensive University located in the Midwest. All (N = 341) were enrolled in and completed an online, standardized course in criminological theory, taught by the same faculty member between Winter term (January to May) 2006 through Fall term The course did not undergo any changes during calendar years All materials including power point slides, audio files of lectures, and exams remained unchanged. Eleven sections were taught during this time frame. Discussion Board The students were required to participate at least once a week in the discussion board. Such participation counted 20% of a student s final grade in the course. Options for participation included answering any of the 30 standard discussion questions. There were between one and four such questions per

7 380 STACK week. In order to receive a C grade on the discussion board, students had to participate at least once per week. However, students could also receive credit for responding to another student s response to the standard questions. Further, students had the option of addressing an issue of their choosing. Examples of standard questions included: What, in your view, causes crime? and Think of the most delinquent person you knew in high school. Do the risk factors in differential association theory help you to understand the causes of his/her delinquency? If not, what does? The number of hits on the discussion board averaged 19/student. The range in hits was between 0 and 53. The distribution of hits approximated a normal distribution with a skewness index that was near zero (.07). The vast majority of hits addressed the 30 standard questions on the board. It is important to note that given the large size of the class and the number of hits on the discussion board, the quality of the posts on the discussion board was not evaluated beyond minimal requirements. The present study follows previous research which has also measured only the quantity, not the quality, of posts on discussion boards (e.g. Armstrong, 2010). Academic Performance Four measures of learning outcomes or academic performance were available in archival records. The percentage score on each of the three-hour exams and the average of the three-hour exams comprise the measures of academic performance. Given that research has determined a higher rate of cheating in classes taught online than in the traditional offline classroom (Lanier, 2006), it is important to exercise additional caution in measuring academic performance in research based on the online mode of delivery. All exams were given in a web secure environment using Respondus Lock Down Browser (RLDB). Students accessed exams from the course blackboard web page. Exams were accessible using only RLDB. Efforts to access exams with Internet Explorer or Firefox would fail. In RLDB, all other software and web utilities (e.g. , instant messaging) are unavailable to the student. In addition, exam questions were presented to the student in a randomized order. No two students received the exam with the questions in the same order. There was no opportunity for backtracking (going back to previous questions). All students had to sign onto the exam at exactly the same time and end by a designated time. These strategies reduced opportunities for cheating. However, all students had equal opportunity to consult their notes and readings while taking the exams. Nevertheless, they were cautioned that they would have only less than 2 min per question, so that relying on their notes would probably prove unproductive. The present analysis is focused on the final or average grade on the three examinations as the learning outcome.

8 ANALYSIS OF AN ONLINE CRIMINOLOGY CLASS 381 Control Variables Grades on the first-hour exam are used as a control variable. These are taken as a synthetic indicator of several constructs thought to predict performance such as academic ability, organizational study skills, effort, and time spent or available for studying. Data on these specific constructs, captured by first exam grades, were unavailable. In order to gage any trends in learning outcomes over time, a control is incorporated for the semester and year of the class in which the student was enrolled. Ten binary variables (0, 1) were created (e.g. Fall term 2006, Winter term 2007 through Fall term 2011) with the baseline or reference category comprised of Winter term Possibly, grades might increase over time if security procedures failed. The second part of the analysis will explore achievement as a moderating effect on the association between discussion board participation and learning outcomes. Low achievers are defined as those whose final grade was below the class mean (N = 167), and high achievers being those whose final grade is above the class mean (N = 147). Results In results not fully reported here, there was no evidence of a bivariate linkage between discussion board participation and final exam grades (r = 0.072, P >.05). Table 2 presents the results of the multivariate analysis of the relationship between discussion board participation and learning outcomes. Controlling for the other variables, the degree of participation in the discussion board was not a significant predictor of final exam grades. While the sign is in the right direction, the coefficient for discussion is just 1.1 times its standard error. In contrast, grades on exam 1 were strong predictors of final grades. A one unit increase in grades on exam 1 was associated with a.6% point increase Table 2 The effect of discussion board participation and exam 1 scores on final course grades in an online criminology course (controls for term are not shown for the purposes of clarity and brevity). N = 341 students Regression coefficient t- statistic Beta weight Predictors: Discussion Board Participation Exam 1 Score Intercept R-squared.686 F-statistic 54.7 p <.05

9 382 STACK Table 3 The effect of discussion board participation and exam 1 scores on final course grades in an online criminology course. Panel A low achievers (N = 167) and Panel B high achievers (N = 147). (Controls for term are not shown for the purposes of clarity and brevity). Low achievers are students below the class median on final grades PANEL A Low achievers Regression coefficient t- statistic Beta weight Predictors: Discussion Board Participation Exam 1 Score Intercept R-squared.609 F-statistic p <.05 PANEL B High achievers Regression Coefficient t- statistic Beta Weight Predictors: Discussion Board Participation Exam 1 Score Intercept R-squared.517 F-statistic p <.05 in the final grade. There was no evidence of an upward trend found in the eleven binary variables for term of the course. The full model works well and explains 69% of the variance in final grades. Table 3 presents the results for splitting the students into two groups: those above and those below the class mean in learning outcomes. Panel A presents the results for low achievers. For this group, the greater the participation in the discussion board, the higher the final examination grade. However, for the high achievers, the results in panel B are insignificant. The sign of the discussion board participation variable is actually negative for the high achievers. Conclusion There has been substantial growth in the use of online course instruction (Allen et al., 2007; Forte & Root, 2011; Hummer et al., 2010). Academic performance of students in online classes tends to be significantly higher than that of students in traditional, offline classes (Shachar & Neumann, 2003). The reasons for this generalization are not clear, but available research indicates that part of the difference may be due to a corresponding differential in cheating

10 ANALYSIS OF AN ONLINE CRIMINOLOGY CLASS 383 between online and offline instruction (Lanier, 2006). Nevertheless, in terms of educational policy-making, use of the uncorrected differential can serve as a strong rationale for further expansion of online instruction. The work on academic performance in criminal justice has been focused on the traditional classroom to the near total neglect of the online classroom (Burruss & Furlow, 2007; Bushway & Flower, 2002; Dabney et al., 2006; Gordon et al., 2009; Hartman et al., 2009; Kessler & Swatt, 2001; Kunkel, 2003; Martin & Hanrahan, 2004; McKee et al., 2001; Proctor, 2002, 2006; Reisig & DeJong, 2005; Schroeder et al., 2008; Stack, 2005; Stack & Kelley, 2002; Tibbetts, 2003). Research on academic performance in online classes is, however, found in other fields including business administration, English, and physics (Armstrong, 2010; Lu et al., 2003; McFarland & Hamilton, 2005; Shieh et al., 2011). Some of this work suggests that there may be contextual effects such that web-based utilities increase the academic performance of subgroups of students such as low achievers (Shieh et al., 2011). The present study adds to the literature concerning online instruction in criminal justice by performing the first test of the impact of the quantity of discussion board participation on academic performance in a criminological theory class. However, there was no overall association between such participation and academic achievement. This particular result is consistent with the findings of three other investigations from other fields (Armstrong, 2010; Lu et al., 2003; McFarland & Hamilton, 2005). However, following the lead of an investigation from classes in physics (Shieh et al., 2011), the present study explored the association between the quantity of hits on the discussion board for two groups of students: low achievers (below the class mean on exams) and high achievers. For low achievers, the association was significant, the greater the hits on the discussion board, the higher the academic performance. This finding was consistent with the conditional nature of the association found in the work by Shieh et al. (2011). Future work can explore similar moderating effects of the link between the use of technological utilities and academic performance. Perhaps, utilities such as blogs, audio files, and online extra credit projects might boost the performance of some subgroups of students, such as low achievers, while not for other groups of students. In addition, future work is needed to assess the quality of participation on the discussion board and its link with academic performance. This was beyond the scope of the present investigation. Future work on the quality of individual posts on discussion board might use grading strategies such as those in pedagogical sources such as Ko and Rossen (2004). In addition, further work is needed to explain what factors, such as differential rates of cheating (Lanier, 2006), selection effects, and specific webbased utilities, account for the higher reported academic achievement in online classes versus offline classes (Shachar & Neumann, 2003). If the more industrious, hard working, self motivated students disproportionately sign up for online classes, the gap in achievement may be mainly due, at least in part,

11 384 STACK to selection effects. Possibly the superior performance of online sections may be due, in part, to the prudent use of particular web-based utilities. For example, the present investigation would suggest that instructors find ways of encouraging low achievers in online classes to make optimal use of their discussion board. Notes on contributor Steven Stack is a professor in the Departments of Psychiatry and Criminal Justice at Wayne State University in Detroit, MI. He is the author of 288 articles and two books, including (with Barbara Bowman), Suicide Movies: Social Patterns, , Cambridge, MA: Hogrefe, References Allen, I. E., Seaman, J., & Garrett, R. (2007). Blending in: The extent and promise of blended education in the United States. Sloan-C. Retrieved from C.org/publications/survey/pdf/Blending_In.pdf Armstrong, K. M. (2010). Fluency, accuracy, and complexity in graded and ungraded writing. Foreign Language Annals, 43, Burruss, G. W., & Furlow, M. H. (2007). Teaching statistics visually: A quasi experimental evaluation of teaching chi square through computer tutorials. Journal of Criminal Justice Education, 18, Bushway, S. D., & Flower, S. M. (2002). Helping criminal justice students learn statistics: A quasi experimental evaluation of learning assistance. Journal of Criminal Justice Education, 13, Dabney, D. A., Green, L., & Topali, V. (2006). Freshman learning communities in criminology & criminal justice. An effective tool for enhancing student recruitment and learning outcomes. Journal of Criminal Justice Education, 17, Forte, J. A., & Root, V. (2011). To ITV or not to ITV: A comparison of hybrid and web enhanced approaches to teaching a macro-course in human behavior in the social environment. Journal of Human Behavior in the Social Environment, 21, Gordon, J. A., Barnes, C. M., & Martin, K. J. (2009). Undergraduate research methods: Does size matter? A look at the attitudes and outcomes of students in a hybrid class format vs. a traditional class format. Journal of Criminal Justice Education, 20, Hartman, J. L., Bjerregaard, B., & Lord, V. B. (2009). Identifying factors that influence the successful transition of criminal justice transfer students. Journal of Criminal Justice Education, 20, Hummer, D., Sims, B., Wooditch, A., & Salley, K. S. (2010). Considerations for faculty preparing to develop and teach online criminal justice courses at traditional institutions of higher learning. Journal of Criminal Justice Education, 21, Hylton, M. E. (2006). Online vs. classroom based instruction: A comparative study of learning outcomes. The Journal of Baccalaureate Social Work, 11, Johnson, K., & White, J. T. (2002). The use of multiple intelligence in criminal justice education. Journal of Criminal Justice Education, 13, Kessler, D. A., & Swatt, M. (2001). Mastery learning, rewriting assignments and student learning of criminal justice research methods. Journal of Criminal Justice Education, 12, Ko, S., & Rossen, S. (2004). Teaching online: A practical guide. Boston, MA: Houghton Mifflin.

12 ANALYSIS OF AN ONLINE CRIMINOLOGY CLASS 385 Kunkel, K. R. (2003). Assessing computer technology in the criminal justice classroom: A case study. Journal of Criminal Justice Education, 14, Lanier, M. M. (2006). Academic integrity and distance education. Journal of Criminal Justice Education, 17, Lu, J., Yu, C. S., & Liu, C. (2003). Learning style, learning patterns, and learning performance in a WebCT-based MIS course. Information and Management, 40, Martin, J. S., & Hanrahan, K. (2004). Criminology freshman: Preparation, expectations, and college performance. Journal of Criminal Justice Education, 15, McFarland, D., & Hamilton, D. (2005). Factors affecting student performance and satisfaction: Online vs. traditional course delivery. Journal of Computer Information Systems, Winter, McKee, A. J., Mallory, S. L., & Campbell, J. (2001). The graduate record exam and undergraduate grade point averages in a criminal justice graduate program. Journal of Criminal Justice Education, 12, Parsons-Pollard, N., Lacks, T. R. D., & Grant, P. H. (2008). A comparative assessment of student learning outcomes in large online and traditional campus-based introduction to criminal justice courses. Criminal Justice Studies, 21, Proctor, J. L. (2002). SPSS vs. Excel: Computing software, criminal justice students, and statistics. Journal of Criminal Justice Education, 13, Proctor, J. L. (2006). Academic achievement & statistical knowledge: A comparison of criminal justice and non criminal justice majors. Journal of Criminal Justice Education, 17, Reisig, M. D., & DeJong, C. (2005). Using GRE scores and prior GPA to predict academic performance among criminal justice graduate students. Journal of Criminal Justice Education, 16, Sami, F. (2011). Course format effects on learning outcomes in an introductory statistics course. MathAMATYC Educator, 2, Schroeder, J., Grohe, B., & Pogue, B. (2008). The impact of criterion writing evaluation technology on criminal justice student writing skills. Journal of Criminal Justice Education, 19, Shachar, M., & Neumann, Y. (2003). Differences between traditional and distance education academic performances: A Meta analytic approach. The International review of Research in Open and Distance Education, 4 (2). Shieh, R. S., Chang, W., & Liu, E. Z. F. (2011). Technology enabled active learning (TEAL) in introductory physics: Impact on genders and achievement levels. Australasian Journal of Educational Technology, 27, Stack, S. (2005). The effect of extra credit papers on learning. Journal of Criminal Justice Education, 16, Stack, S., & Kelley, T. (2002). The graduate record exam as a predictor of graduate student performance. The case of criminal justice. Journal of Criminal Justice Education, 13, Tibbetts, S. G. (2003). Freshman interest groups in criminal justice education. Journal of Criminal Justice Education, 14,