INNOVATIVE TEACHING. Laboratory assignments and their impact on final examination grades: an online research methods/statistics course 1.

Transcription

1 INNOVATIVE TEACHING 2015, Volume 4, Article 1 ISSN DOI /01.IT.4.1 Steven John Stack 2015 Attribution-NonCommercial- NoDerivs CC-BY-NC-ND Received April 4, 2014 Accepted December 18, 2014 Published January 8, 2015 CITATION Stack, S. J. (2015) Laboratory assignments and their impact on final examination grades: an online research methods/ statistics course. Innovative Teaching, 4, 1. Laboratory assignments and their impact on final examination grades: an online research methods/statistics course 1 Steven John Stack Wayne State University Abstract Over the last decade, enrollment in online courses has tripled. However, the delivery of online sections of courses that typically require laboratory work on campus can present a challenge. The present study has two goals: (1) describe the laboratory assignments for an online version of an undergraduate research methods course, and (2) assess the extent to which learning in the laboratory assignments is related to grades on the final examination. Participants were 108 students completing an online course in research methods/statistics in a Carnegie Research Extensive University. Students chose a sample of 40 offenders from an online public database, the Offender Tracking Information System (OTIS), entered the data into an Excel file, and used the Excel Data Analysis Toolkit to produce both descriptive statistics and to test a series of hypotheses including the predictors of violent offending. The dependent variable is the scores on the final examination, and the independent variable was the grades received on six laboratory assignments. Control variables included grades on a preliminary 1-hr examination and sex. In a bivariate regression, laboratory and grades explained 13% of the variance in final examination grades. However, in a multiple regression analysis, controlling for other variables, the grade on the laboratory assignments was unrelated to final examination grades. The model explained 47% of the variance in final examination grades. Standard laboratory assignments were successfully delivered in an online environment, but grades on laboratory assignments were not an independent predictor of final examination grades. Further work is needed for other fields and educational contexts. Enrollment in online classes has more than tripled in the last decade. In 2002, a total of 1,602,970 students took at least one class online. Nine years later, this figure more than tripled to 6,714,792 students. The percent of online enrollment over total enrollment increased from 9.6% in fall 2002 to 32.0% in fall 2011 ( Allen & Seaman, 2013 ). Work on the amount of learning that transpires in online courses has emphasized comparisons between online and traditional classrooms. This work has presented diverse findings (e.g., Russell, 1999 ; Brown & Leidholm, 2002 ; Harmon & Lambrinos, 2006 ; Bray, Harris, & Major, 2007 ; Parsons-Pollard, Lacks, & Grant, 2008 ; Gratton-LaVoie & Stanley, 2009; Figlio, Rush, & Yin, 2010 ). Some investigations report that examination scores are higher for traditional classes than online classes (e.g., Brown & Leidholm, 2002 ; Parsons-Pollard, Lacks, & Grant, 2008 ; Figlio, Rush, & Yin, 2010 ), while others report the opposite, that student performance is higher in online sections (e.g., Harmon & Lambrinos, 2006 ; Gratton-LaVoie & Stanley, 2009; Means, Toyama, Murphy, Bakia, & Jones, 2010 ). Still others report no significant difference in student performance between online and live classes (for a review, see Russell, 1999 ). Shachar and Neumann (2003 ) searched for studies on the effects of online instruction on academic performance in electronic databases (ProQuest) and using search engines Google, Northern Light, and Wisenut. To be included in the meta-analysis, published or unpublished studies met the following criteria: (1) reported between 1990 and 2002, (2) had both an online group and a control or comparison group, (3) had sufficient quantitative data (sample size, mean, and standard deviation) from which effect sizes could be calculated, and (4) showing no severe methodological flaws. A total of 86 studies met the criteria for inclusion. A meta-analysis of the 86 studies showed that students Ammons Scientific 1 Address correspondence to Steven J. Stack, Department of Criminal Justice, Wayne State University, Detroit, Michigan or ( aa1051@wayne.edu ).

2 in online sections of a course generally scored higher on standardized final examinations than students enrolled in traditional classes ( Shachar & Neumann, 2003 ). The reported difference was large, amounting to a half a standard deviation. A more recent meta-analysis, limited to 50 findings from the relevant research, also confirmed that academic performance was higher in online compared to traditional classes ( Means, et al., 2010 ). However, there were a wide variety of confounders that may have artificially enhanced student performance in online classes. Most research has not controlled these factors, which include greater opportunities for cheating in online classes and self-selection by the better students into online sections vs. the traditional sections of a course ( Lanier, 2006 ; Moten, Fitterer, Brazier, Leonard, & Brown, 2013 ). A recurrent limitation of this body of work on learning outcomes is that there is relatively little information about which factors shape the amount students learn with online classes. In the field of criminology, e.g., a review of 222 journal articles in the Journal of Criminal Justice Education published between 2001 and 2010 showed that only 16 dealt with the predictors of academic performance. None of these dealt with predictors of learning outcomes within online courses ( Stack, 2013 ). The present paper adds to knowledge of online learning using data from a laboratory module from an undergraduate online research methods/statistics course. Research Question Will learning in the laboratory module, measured by grades on the laboratory module, be associated with better grades on the final examination? Method Participants The subjects were all students enrolled in eight sections of a research methods course at a Carnegie research extensive university during Fall 2009 through Fall All sections were taught by the same instructor. Of the 147 students enrolled at the beginning of the semester, a total of 108 completed the course and comprise the sample. A total of 42.9% were men and 57.1% were women. From a hands-up poll during an orientation session, the majority of the students each term were graduating seniors. Students tend to put the course off until their senior year. All eight classes were taught online, with the exception of a mandatory orientation session during the first week of class. Course Content and Assignments All classes had exactly the same reading assignments and examinations. Blackboard technology provided the web page for both sets of classes. All readings were posted on the blackboard page as PDF files. In addition to readings from various textbooks, students were required to review power point slides (prepared by the instructor) which emphasized critical points, clarifications, applications, and extensions of the readings. Course work consisted of required readings and the review of approximately 100 power point slides each week. Each slide had a built-in audio file. The audio file lasted approximately 1 to 5 min. in length. The audio file clarified the concepts under review in each slide. The audio files collectively took approximately 200 min. of listening time each week. This amount of time required in listening to the audio files approximated the amount of lecture time if the course was taught in a traditional classroom. The course was a four-credit class which would meet 3 hr and 50 min if taught live in a campus classroom. The materials for the online course were prepared unit by unit over a period of four years. The eight sections under investigation were taught after the 4 yr. preparation period. In addition, students in all sections were required to respond to at least one question or problem on the Blackboard Discussion Board. On the whole, the course covers both research methods and statistics, with more of an emphasis on the latter than the former. Learning Module The laboratory assignments were designed to take the student through the steps of the scientific method. While these steps and exercises are emphasized through much of the course, the laboratory assignments gave the students hands-on experience in actually doing research, as opposed to just reading about it. This hands-on experience is thought to improve the depth of understanding of constructs in statistics and methods. The students apply what they have learned in the readings to some real data they have gathered themselves. A learning-bydoing approach may engage some students in a way that enhances motivation to do well on other components of the course. Full details of the laboratory assignments are included in a 33-page Laboratory Guide which is provided to all students (the manual is available from the author upon request). Briefly, there were six exercises. (1) Literature Review. The first involved reviewing the literature on a focused class problem: the relationship between tattoos and deviant behavior, including criminality. Each student had to summarize one article in his or her own words and post the summary on the blackboard web page for review by the class. Available electronic search engines in the library were used to locate appropriate articles. These search engines included Criminal Justice Abstracts, Medline, Psychological Abstracts, Social Work Abstracts, and Sociological Abstracts. Students were supplied a list of potential keywords to facilitate their search. These included tattoo, violence, crime, mental illness, alcoholism, and drug addiction. Summaries of each student's article were posted on the course webpage for synthesis and commentary on the discussion board , Volume 4, Article 1

3 Patterns emerged, with most of the research articles reporting an association, especially between the number of visible tattoos and crime/deviant behavior. (2) Sampling. Students were instructed to draw a representative sample of 20 offenders with tattoos from a publicly accessible database, the Michigan Offender Tracking Information System. 2 From the rap sheets on the offenders, the students coded data into an Excel spreadsheet for selected variables including age, race, number of violent offenses, and number of tattoos. (3) Sampling. Students drew a sample of 20 offenders without tattoos from the same database and coded the corresponding data on variables into the Excel spreadsheet. (4) Descriptive Statistics. Students used the Excel Data Analysis Toolkit to calculate basic descriptive statistics including measures of central tendency such as mean, mode, and median, and measures of dispersion including standard deviation, standard error, and range. (5) Hypothesis Testing: Correlation. Students constructed a null hypothesis, research hypothesis, test statistic (Student s t test), and rejection region for the null hypothesis. Four hypotheses were formulated. There were four hypotheses to test: (H1) the greater the number of tattoos, the greater the number of violent offenses; (H2) the greater the age of the offender, the lower the number of tattoos (H3) the greater the age of the offender, the greater the number of tattoos; and (H4) female offenders have fewer tattoos than male offenders. These hypotheses were tested using Pearson's correlation coefficient paired with the Student s t ratio as a means of statistical inference. (6) Regression Analysis. Students performed a multiple regression analysis to apply the concept of statistical control. The primary hypothesis H1 was that, controlling for age, the greater the number of tattoos, the greater the number of violent offenses. The students interpreted the statistics in the Excel output including regression coefficient, standard error, R 2 statistic, and F statistic. In addition, they used the standard error to calculate a 95% confidence interval. The regression equation was used to estimate the number of tattoos that a 50-yr.-old prisoner would have. The latter exercise gave them handson experience in applying regression coefficients to the general scientific purpose of prediction. Measures Examinations. The examinations consisted of multiple choice and true and false questions. The questions were kept the same across all sections of the course. Examinations were given in a controlled environment to minimize cheating. All examinations used commercially available software, Respondus Lockdown Browser, which locks down the computer being used during examination time such that the only utility that can be performed on the machine is taking an examination on 2OTIS, Blackboard. Instant messaging, , and all other utilities are shut down. Furthermore, to promote security all students took the examination on the same date and time. In addition, examination items were provided to each student in a scrambled or randomized order. Importantly, backtracking was blocked. Once an answer was typed in and submitted, users could not go back to change their response. Finally, the examination environment was structured so that students would have approximately 1 min. to answer each objective question. These conditions helped to minimize collaborative or group-oriented taking of examinations. Dependent and Independent Variables. The dependent variable was the grade received on the final examination. The central independent variable was the grade on the six laboratory assignments. Grades on the first 1-hr. examination were used as a proxy independent variable for several constructs thought to predict student achievement: academic ability, amount of academic effort, and the amount of time spent and/or available for studying course material ( Stack, 2013 ). Data on these specific constructs were not available, but it is assumed they are at least partially captured by grades on the first 1-hr. exam. To assess possible gender differences in grades in the course, analyses were controlled for the sex of the student (1 = female, 0 = male). There was a possibility for trending in examination scores over time. For example, if copies of any examination were somehow obtained and distributed to students taking the course in subsequent terms, examination scores would be expected to increase over time. To assess this possibility, a control was included for the semester and term of the course. Seven binary variables (0,1) were created, e.g., Winter Term 2010 (0,1), Fall Term 2010 (0,1) through Fall Term 2013 (0,1), with the baseline or reference category comprised of Fall Term 2009 (0,1). Results The bivariate association between the mean grade on the laboratory assignments and the final examination grade is shown in column 2 of Table 1. At the bivariate level, the grade on laboratory assignments shared 13% of its variance with final examination grades. However, it is plausible that the score on the first examination, a proxy measure of general academic ability and effort, may shape both scores on the final examination and the laboratory grades. If so, the relationship between laboratory grades and the final examination could be spurious. A multivariate analysis was run to control the effect of generalized academic ability and effort (grades on the first exam). The results of the multivariate analysis are provided in Table 1, column 3. Controlling for the other predictors, including the measure of generalized academic ability and effort (first examination grades), grades on the laboratory assignments were not statistically significantly related to final examination scores. Sex was , Volume 4, Article 1

4 TABLE 1 Multiple Regression: Effect of Learning in Laboratory Assignments and Control Variables on Final Examination Grades in an Online Statistics/Research Methods Course ( N = 108 students; average class size across 8 sections was 13.5 students) Variable Model 1 Laboratory Grade Model 2 Laboratory Grade & Control Variables B SE β t p B SE β t p Grade on laboratory assignments 0.15 * < Grade, first hour examination 0.49 * <.001 Sex (female = 1) Term: Fall 2013 section (0,1) Winter 2013 section (0,1) Fall 2012 section (0,1) Fall 2011 section (0,1) Winter 2011 section (0,1) Fall 2010 section (0,1) Winter 2010 section (0,1) Fall 2009 (reference) 1.00 Constant 34.6 * * F * 8.93 * R R Adj R SE of Estimate Δ R Δ F df total df regression 1 10 df residual * p <.05. not predictive of final examination grades (B = 0.88, ns). However, grades on the first 1-hr. examination predicted final examination scores (B = 0.50, p <.05). In order to assess if there was any trending in examination scores over time, the analysis included seven binary variables for year/semester. That is, over time items on the hour examination may have, e.g., been copied and leaked out to subsequent test takers. If so, subsequent test takers would have an advantage over early test takers. However, as shown in Table 1, none of the t ratios for the seven temporal variables were significant. Hence, there was no evidence that examination scores improved (or worsened) over the years of the study. The model as a whole significantly predicted final examination scores ( F = 8.93, p <.05). The model explained 48% of the variance in final examination scores. In results not fully reported here, the model was replicated for male and female students separately. Grades on the laboratory assignments were non-significant in both sex-specific analyses. It is possible that while the laboratory grades did not predict scores on the final examination for the whole group of students, laboratory grades might have an association with final examination grades for a subset of the students. In a previous study, grades on a discussion board predicted final examination scores for low achievers, but not high achievers ( Stack, 2013 ). To assess this possibility, following the previous study ( Stack, 2013 ), the students were divided into two groups: those who scored below the class mean on final examination grades and those scoring above that mean. Following the previous investigation ( Stack, 2013 ), a regression was run on each of these two groups of students. The results are provided in Table 2. For low achievers, controlling for the other predictors, grades on the laboratory module did not predict final examination grades. Controlling for the other variables, grades on the laboratory module did not predict final examination grades for high achievers. In both regressions, grades on the first 1-hr. examination predicted substantial variance in grades on the final examination (18% and 28%, respectively). Discussion The laboratory module was developed to mirror the stages of the scientific method from literature review through multiple regression analysis to test research hypotheses. Students drew samples from a publicly accessible database. Such databases, including sex offender registries, are available in many states besides Michigan and can engage student interest (some students had been in , Volume 4, Article 1

5 TABLE 2 The Effect of Learning in Laboraratory Assignments on Final Examination Grades in Online Statistics/Research Methods. High Achievers ( N = 45) and Low Achievers ( N = 62) Variable High Achievers Low Achievers B t B t Grade on laboratory assignments Grade, first one-hour examination 0.34 * * 2.26 Sex (female = 1) Constant 26.3 * * 9.25 F 5.32 * 4.33 * R R Adj R SE of Estimate df total df regression 3 3 df residual Note. Low achievers are students below the class mean on final grades. For the purposes of brevity and clarity, SE, β, and p of coefficients, and terms for semester are not shown. *p <.05. the OTIS database themselves or had friends and neighbors profiled in it). The use of the statistical tool kit in Excel had strong advantages over alternative such as SPSS or SAS. Students were already familiar with Excel and had the software on their own computers. The combination of OTIS with Excel allowed the students to do the laboratory assignments anywhere where there was an Internet connection. They did not have to travel back to campus to do assignments in a computer lab with specialized databases and/or software. Research on the efficacy of online learning has tended to focus on comparisons of student academic performance in online and traditional classroom instruction (e.g., Russell, 1999 ; Brown & Leidholm, 2002 ; Harmon & Lambrinos, 2006 ; Bray, Harris, & Major, 2007 ; Parsons-Pollard, Lacks, & Grant, 2008 ; Gratton-LaVoie & Stanley, 2009; Figlio, Rush, & Yin, 2010 ). There is relatively little work on assessing learning outcomes after innovations have been introduced to online criminology courses (for an exception, see Stack, 2013 ). The present study contributes to the literature by assessing the efficacy of a laboratory module on final examination grades in an online research method/statistics course. No comparisons were made to traditional course delivery in a classroom. The focus was to assess whether the laboratory module enhanced learning in an online context. The results of the present study were mixed. At the bivariate level, the grades on the laboratory module were correlated with the scores on the final exam. However, this association may be spurious, since the multivariate model controlling for grades on the first 1-hr. examination showed that the grades on the laboratory module did not have an independent effect on the student's performance on the final examination. The first 1-hr. examination grades predicted both grades on the laboratory module and the grades on the final examination. This suggests that students relatively high in academic ability and effort do well in both applied work (the laboratory module), as well as understanding more general theoretical and empirical work in research methods and statistics. The results based on the multivariate analysis suggest that the innovation of online assignments failed to improve learning. This does not mean that the results were unimportant. The larger literature on innovations in criminological classes indicates that well-intentioned innovations do not improve learning outcomes. For example, Burruss and Furlow (2007 ) found that exposure to visual tutorials on statistical significance tests did not improve examination scores on the subject. A study of learning communities determined that participants grades improved in their Freshman year. However, such participation had no effect on grades in subsequent semesters ( Dabney, et al., 2006 ). Requiring class attendance was found not to improve test scores ( Bushway & Flower, 2002 ). However, some studies have found an effect of innovations on examination scores; e.g., students who were taught statistics using Excel scored higher on the final examination than students taught with SPSS ( Proctor, 2006 ). Still other studies reported mixed results: while participation in an online discussion board had no effect for high achievers, for low achievers the greater the participation, the higher the scores on the final examination. The regression coefficient for low achievers ( B = 0.06) was 1.9 times its standard error ( Stack, 2013 ) , Volume 4, Article 1

6 Limitations and Conclusions A limitation is that data were not available to measure the academic ability and study habits of the students. An indirect measure of these latent constructs, grades on the first hour examination, was used. The investigation needs to be replicated in other institutional contexts. In summary, research results on innovations are useful guides for faculty when they are making decisions on which innovations to use in their classes. Results indicating that a particular, intuitive, well-meaning innovation does not improve test scores can save faculty time, since adopting such innovations may not be productive. Some innovations may be more likely to be effective, while others are less likely to improve learning outcomes. Of course, there are other reasons why a faculty member might maintain a laboratory module even if it does not improve final examination grades. For example, the goal of being able to enter data, analyze it with a statistical package, and interpret statistical output constitutes a set of marketable skills. Further work is needed for other courses with laboratory modules such as senior seminars and graduate classes. In addition, work is needed for other disciplines as well as other institutions of higher education at varied Carnegie levels, including master's level universities and liberal arts colleges. Additional work in these other contexts is necessary to examine whether or not there is a link between laboratory assignments and final grades for low achievers vs. high achievers, as well as other subgroups of students (e.g., by sex). The zero order correlation between laboratory grades and final examination scores for low achievers in the present study was r =.24. In a previous study, the correlation between participation in the discussion board and final exam grades for low achievers was r =.11. These correlations are to be interpreted with caution due to differences in sample size, the course being taught (methods/ statistics vs. criminological theory), and the nature of the innovation (laboratory module vs. a discussion board). REFERENCES Allen, I. E., & Seaman, J. (2013 ) Changing course: ten years of tracking online education in the United States. Wellesley, MA: Babson Survey Research Group/Pearson Publishers and Sloan Foundation. Bray, N. J., Harris, M. S., & Major, C. (2007 ) New verse or the same old chorus: looking holistically at distance education research. Research in Higher Education, 48, Brown, B. W., & Leidholm, C. E. (2002 ) Teaching microeconomic principles. American Economic Review, 92, Burruss, G. W., & Furlow, M. H. (2007 ) Teaching statistics visually: a quasi-experimental evaluation of teaching chi square through computer tutorials. Journal of Criminal Justice Education, 18, Bushway, S. D., & Flower, S. M. (2002 ) Helping criminal justice students learn statistics: a quasi-experimental evaluation of learning assistance. Journal of Criminal Justice Education, 13, Dabney, D. A., Green, L., & Topali, V. (2006 ) Freshman learning communities in criminology & criminal justice. Journal of Criminal Justice Education, 17, Figlio, D. N., Rush, M., & Yin, L. (2010 ) Is it live or is it Internet? Experimental estimates of the effects of online instruction on student learning. Working paper Cambridge, MA : National Bureau of Economic Research. Gratton-Lavoie, C., & Stanley, D. (2009 ) Teaching and learning principles of micro economics online: an empirical assessment. Research in Economic Education, 40(1), Grossman, S. (2013 ) American MOOC providers face international competition. Chronicle of Higher Education. Harmon, O. R., & Lambrinos, J. (2006 ) Online format vs. live mode of instruction: do human capital differences or differences in returns to human capital explain the differences in outcomes? Department of Economics Working Paper Storrs, CT : Univer. of Connecticut. Lanier, M. M. (2006 ) Academic integrity in distance education. Journal of Criminal Justice Education, 17, Means, B., Toyama, Y., Murphy, R., Bakia, M., & Jones, K. (2010 ) Evaluation of evidence based practices in online learning: a metaanalysis and review of online learning studies. Washington DC : U.S. Department of Education. Moten, J., Jr., Fitterer, A., Brazier, E., Leonard, J., & Brown, A. ( 2013 ) Examining online college cyber cheating methods and prevention measures. The Electronic Journal of e-learning, 11, Parsons-Pollard, N., Lacks, T. R., & Grant, P. H. (2008 ) A comparative assessment of student learning outcomes in large online and traditional campus based introduction to criminal justice courses. Criminal Justice Studies, 2, Proctor, J. L. (2006 ) SPSS vs. Excel: computing software, criminal justice students, and statistics. Journal of Criminal Justice Education, 13, Russell, T. L. (1999 ) No significant difference: a comparative research bibliography on technology for distance education. Raleigh, NC : North Carolina State Univer. Shachar, M., & Neumann, Y. (2003 ) Differences between traditional and distance education academic performances: a meta-analysis approach. International Review of Research in Open and Distance Education, 4 (2 ). Retrieved October 12, 2012, from Stack, S. (2013 ) Does discussion promote learning outcomes? Analysis of an online criminology class: research note. Journal of Criminal Justice Education, 24, , Volume 4, Article 1