1 Published in Southern Economic Journal, January 2008 Department of Economics Working Paper Series Student Performance in Traditional vs. Online Format: Evidence from Introductory Economics Classes Oskar R. Harmon University of Connecticut James Lambrinos Union University Working Paper R March 2007, revised December Mansfield Road, Unit 1063 Storrs, CT Phone: (860) Fax: (860) This working paper is indexed on RePEc,
2 Abstract This study uses a different approach to testing for a difference in student performance between traditional and online courses than prior studies that compare learning outcomes in economics courses. The study uses exam questions as the unit of observation and a specification that includes indicator variables for each student. These indicator variables capture the effect of differences in unobserved student characteristics on learning outcomes and thereby eliminate omitted variable bias. The study reports the finding that for an MBA introductory economics course taught in hybrid format the students had a significantly greater chance of answering a question correctly if it came from a chapter covered online (p.0075), and that for two undergraduate courses in principles of microeconomics, one online and one traditional, there was a marginally significant result in three different models (p.1023,.0829,.0737). Journal of Economic Literature Classification: A2, A22 Keywords: online, instruction, economics, traditional
3 2 Student Performance in Traditional vs. Online Format: Evidence from Introductory Economics Classes Web instruction increases access to university courses because of significant reductions in commuting time, and flexibility in scheduling online learning sessions around work and family responsibilities. It is not surprising that a recent survey (Coates and Humphreys 2003) of economics departments reported that enrollment for cybereconomics courses.tends to enroll a high proportion of non-traditional students, like working adults and non-degree seeking students. Web instruction, however, faces the challenge of developing equivalence between digital and live communication in student-to-student and student-to-instructor interactions that are fundamental to learning. Factors that contribute to equivalence in communication are students with strong independent learning skills and high levels of self-discipline and motivation. Because these characteristics are typically present in relatively less abundance among traditional than non-traditional undergraduates, Brown and Liedholm (2002) advise that the online format is likely to be far less appropriate than the traditional format for the teaching of principles of economics to typical undergraduates. A factor that might promote equivalence between digital and live communication is the heretofore less studied format of mixing online lectures with traditional lectures in the same course. In a rotating format, the traditional lectures could potentially marry the advantages of the online format, which include a self-paced format,
4 3 flexibility in scheduling, and convenience in viewing, listening, and printing presentations, with the advantages of the traditional format, which include the discipline imposed by attending class at a fixed time, impromptu explanations and examples developed in response to live questions, and greater stimulation than when working alone (Terry, Lewer et al. 2003). All prior studies of the effect of instruction format on learning outcomes have used the approach of regressing a measure of learning outcome on student characteristics. A fundamental disadvantage of this approach is the econometric problem of potential bias from omitted variables arising because many student characteristics that influence learning outcomes are unobservable and/or difficult to measure. In this study we use a different approach. Whereas, all prior studies have used the student as the unit of observation, in this study we use the question as the unit of observation (Marburger 2001, 2006). That is, from a record of exam responses and whether the exam questions were covered in the online format or traditional format we estimate a qualitative choice model in which the probability of a correct response is correlated with the classroom format. An advantage of this approach is that it solves the problem of omitted variables because it uses an indicator variable for each student to capture the effect of unobserved student characteristics. In the next section, we review the literature on the effectiveness of online versus traditional in-class teaching. Section II contains a discussion of our data, empirical model and results while the final section contains our conclusions. I. Online vs. Traditional Teaching Studies on the relative effectiveness of online v traditional teaching are far more common in other disciplines and the predominant finding of these studies is no
5 4 significant difference. Moreover, in the group of studies that do find a significant difference, the predominant finding is that the online format is more effective than the traditional format in promoting learning outcomes. 1 To our knowledge there are only five studies of principles of economics classes that use direct evidence on test scores to compare learning outcomes in the two different instructional formats: Brown and Liedholm (2002); Coates et al. (2004); Anstine and Skidmore (2005); Navarro and Shoemaker (1999); and Terry et al. (2003). 2 Not withstanding major differences in the student characteristics of these samples in these studies 3, the majority of these studies, by the margin of three to two (60%) conclude that learning outcomes in the online format are inferior to the traditional format. Brown and Liedholm (2002) surveyed students in three different instructional formats: online, hybrid, and traditional; in a principles of microeconomics course. The students in their sample were traditional aged undergraduates at a large residential university. 4 They reported that exam scores, after controlling for differences in student characteristics, were approximately 6% percent higher for the traditional format than for 1 Studies in other disciplines of the issue of whether instructional format effects learning outcomes are far more common and thepredominate finding is of no significant difference. The website by T.Russell (http://nosignificantdifference.wcet.info), compiles dozens of studies on distance education (Coates et. al. 2004) shows (as of 2/22//2007) that 131 (73%) studies report no significant difference in outcomes between materials delivered as distance-education or face-to-face, 45 (25%) studies report outcomes improved when delivered as distance-education, 7 report mixed results, and 3 (2%) improved outcomes when delivered face-to-face. 2 These five studies of economics classes we next review are not included in the Russell (2007) survey 3 Two studies are of undergraduate students, which uniformly report that learning outcomes are lower in online sections and the remaining three are of MBA students, to of which report no significant difference for economics classes. 4 The face-to-face course was taught in two large sections averaging 180 students each and did not use online instructional materials. The online course was taught in two small sections averaging 45 students each and used an array of online instructional materials that included access to video of the face-to-face class lectures, PowerPoint lecture slides, and interactive online practice materials. A principal component of the practice materials was the online Excel exercises Principles of Microeconomics (http://www.msu.edu/course/ec/201/brown/pim). The hybrid course was taught in traditional lecture format and the Excel exercises in the online course were a required homework assignment.
6 5 the online format. It was further reported that for questions of concept identification there was no significant difference, but for questions of higher level learning (concept application) the mean was significantly lower for the online classes. They attribute the relatively better performance in the traditional classes to the benefit of in-person instructor-student interactions, and attribute the relatively poorer performance of the students in the online class to the lack of self-discipline necessary for successful independent learning in the online environment. Coates et al. (2004) surveyed three pairs of traditional and online principles of economics courses taught at three different institutions. The students in the online courses were older, had longer commute times and more job responsibilities than the students selecting the traditional courses. 5 The students score on the Test of Understanding College Level Economics (TUCE) administered at the end of the semester is used as the measure of learning outcomes. After controlling for selection bias and differences in student characteristics, they report that the average TUCE scores are about 15% higher for the traditional format than for the online format. 6 Navarro and Shoemaker (1999) surveyed a pair of traditional and online sections of an MBA class in principles of macroeconomics. The students self-selected the instruction format, each section was approximately 30 students, and there was no 5 In the traditional format sample 50% of the students work, and of those working the average weekly hours are 18, 40% live on campus and 90% are in the age range In the online sample 73% of the students have jobs, of those the average work week is 33 hours, 20% live on campus, and 63% are in the age range The sample has 126 observations, 67 in the traditional format class, and 59 in the online format. Two of the section pairs were from a principles of macroeconomics course and the other from a principles of microeconomics course. The traditional and online classes were of similar size ranging from 24 to 37 students each and the response rates range from 85% to 96%. The exams were multiple choice, the exams for the online class were not proctored. 6 Other findings were that freshman and sophomores taking the online format scored lower than upperclassman, and that students who self select the online format did better than a randomly selected student of similar characteristics, a difference they attribute to an unobserved positive interaction of learning styles and instruction format.
7 6 difference in the demographic composition of each section. They used a simple comparison of means on test scores and reported no-significant difference in learning outcomes between the two formats. Terry, Lewer et al. (2003) surveyed 240 MBA students in a program offering courses in the three formats of online, traditional, and hybrid. Approximately seventy students were enrolled in each sequence. Using a standard regression model (without correction for sample selection bias) with final exam score as the dependent variable and student characteristics as independent variables, they report that predicted exam scores for students in the online courses were significantly less than for students in the traditional and the hybrid formats. However, for the comparison of exam scores between students in the hybrid compared to the traditional classes they report no significant difference. Anstine and Skidmore (2005) surveyed two pairs of traditional and online courses, one in statistics, and the other in managerial economics 7 in an MBA program. They reported, after controlling for student characteristics and selection bias 8, students in the online format of the statistics class had exam scores 14.1 percentage points less than in the traditional format, whereas, for the managerial economics class the test scores between the two formats were not significantly different 9. 7 The instructor, class materials and exams, which were take-home, are identical for the traditional and the online sections statistics and managerial economic courses, respectively. 8 In their selection model of choice of learning environment the dependent variable is class format (online = 1) and the independent variables are travel time, reported weekly hours devoted to work and children in the home. Results are not presented but they report that the indicator variable for children in the family was positive and significant. 9 Similar to the (Coates 2004) study they report that of the many variables for student characteristics tested, few were statistically significant. For the statistics class only age was significant, and for the managerial class only GMAT, study time, and foreign student status were statistically significant.
8 7 The two studies of undergraduate students in principles of economics courses (Brown and Liedholm 2002; Coates, Humphreys et al. 2004) reported lower test scores for students in the online class compared to the traditional class 10. In contrast, two of the three studies of MBA students reported no significant difference for courses in economics (Navarro and Shoemaker 1999; Anstine and Skidmore 2005), and only the third study (Terry, Lewer et al. 2003) reported lower exam scores for the online class compared to the traditional class. 11 Compared to the results for undergraduates, the studies of MBA students suggests that unobservable differences (such as motivation and maturity) might have a significant role in explaining MBA students perform relatively better than undergraduates in online courses compared to traditional courses. The two studies: Coates (2002) and Anstine and Skidmore (2005); that correct for bias from self-selection and omitted unobserved variables used the Heckman Correction econometric procedure. These researchers have had, arguably, limited success in eliminating the bias. These researchers report that of the many observable variables (such as GPA, gender, age, and SAT scores) they use (as proxies for unobservable attributes, such as of self-discipline, motivation, and independent learning skills) to estimate the Heckman Correction, few have statistically significant explanatory power. 12 An alternative approach, the one used in this study, is to use a dummy variable for each student that inclusively captures the effects of all the differences in student attributes that effect learning outcomes. Whereas, this approach cannot assess the independent effect of 10 For a hybrid class Brown and Liedholm (2002) report the scores were better than in the online section but lower than in a traditional section. 11 For a hybrid class the study reports the scores were better than in the online section but no different than in a traditional section. 12 For example both Anstine and Skidmore (2005) and Coates (2002) report that they are unable find more than a few student characteristics with statistically significant explanatory power.
9 8 important specific measurable attributes such as GPA, gender, and age, it does avoid the difficult problem of developing measures of attributes that are inherently difficult to measure, such as motivation and independent learning skills, which can potentially bias the estimation results. Another possible factor which could bias some of the studies is the difference in the likelihood that cheating takes place in these courses. For example, if students in the online courses are taking the exams in an unproctored setting, it could be that it is more likely cheating is taking place in these courses. In two of the five studies reviewed here the exams in the online section were not proctored (Coates et al. 2003; Anstine and Skidmore 2005). There is a large literature on cheating, and there is agreement that cheating is a large and growing problem and students have a narrower definition than instructors of cheating. Warnings, honor codes, and the certainty of punishment decrease cheating (Kerkvliet and Sigmund 1999; McCabe, Trevion et al. 2001; Burrus, et al. 2007; Bisping et al. 2008). However, in a course if the exams of one section are proctored and the exams in the other section are not proctored there is the potential for bias due to more cheating opportunities in the section where the exam is not proctored (Harmon and Lambrinos 2008) and (Lanier 2006), for example, report evidence to suggest that more cheating takes place in unproctored online exams, than in proctored exams. However, the direction of the bias it is not clear. Students cheat to improve test scores (Finn and Frone 2004), but studies (Whitley 1998; Dickson et al. 2005) report evidence to suggest that some forms of cheating (e.g. crib sheets) do not improve test scores. Explanations of this counterintuitive result include: the benefit of cheating is out weighed by the effect of under preparation, and if a student considers the course of low
10 9 value they will use cheating as a way to work less hard for a minimal passing grade rather than as a way to achieve a high grade. Our review of five previous studies on the issue of whether the online format is inferior to the traditional format in economics classes shows there are important differences in these studies regarding omitted variable bias and proctored exams. The two studies of undergraduate students both report that the online format is inferior to the traditional format. However, only one study corrects for omitted variable bias (Coates et al. 2003), and in that study the exam for the online section is not proctored. If cheating improves outcomes, this suggests that Coates could have found a larger difference while if cheating results in lower outcomes, the difference could become smaller and perhaps insigificant. Of the three studies of MBA students, two report no difference between the online and traditional format while one reports the online format is inferior. However only one study corrects for omitted variable bias (Anstine and Skidmore 2005), and in that study the exam for the online section is not proctored These five studies and their salient characteristics are summarized Table 1 below.
11 10 TABLE 1: Summary of Literature Review Proctored Exam: Result: Which is the better STUDY Sample: UG* or MBA Correction for Omitted Variables Online Class Traditional Class Online v Traditional Hybrid v Traditional Brown and Liedholm (2002) UG No n.a. Traditional Traditional Coates et al. (2004) UG Yes No Yes Traditional - Navarro and Shoemaker (1999) MBA No Yes Yes No Difference - Trerry et al. (2003) MBA No Yes Yes Traditional No Difference Anstine and Skidmore (2005) MBA Yes No No No Difference - Notes: Abbreviations: UG indicates Undergraduate. n.a. indicates not available. II. Data, Empirical Model and Results Our data are from two sources: a principles of economics course for an MBA program taught in hybrid format, and two sections of an undergraduate principles of microeconomics course one taught in online format, the other in traditional format taught by the same instructor in the same semester.
12 11 Table 1 Descriptive Statistics: Graduate Students MBA Level Introductory Principles: Hybrid Standard Variable N Mean Deviation CORRRECT Percentage Correct Online Percentage Correct Traditional Question-Specific Variables: MID_TERM FINAL_EXAM Question-Difficulty Rating Variables: Hybrid Class,Rating from testbank (1=easy, to 5=hard) Covered Online Covered Traditional Student-Specific Variables: MBA student (1=MBA ) GPA under grad GPA grad Gender (1=Female) Number of Students: 13 The sample of MBA students is from a principles of economics class for students who could not waive the course 13. The course covered both microeconomics and macroeconomics 14 and was taught in hybrid format. Two-thirds of the lectures were taught in traditional lecture format using PowerPoint presentations. One-third of the lectures were taught in online format using PowerPoint presentations annotated by the 13 Two undergraduate courses in economics with grades of B- or better were required to waive this course. 14 All eighteen chapters in the textbook were covered in this course and these consisted of two introductory chapters, eight micro chapters and eight macro chapters. Five of the 8 microeconomics chapters and six of the 8 macroeconomics chapters were taught in a traditional lecture format using PowerPoint presentations, the other 4 chapters were taught in online format.
13 12 instructor with audio clips. Homework problems were assigned for each chapter. For the traditional lectures the answers to the homework problems were reviewed in class as time permitted; for the online chapters, the solutions to the homework problems were made available electronically. There was a midterm covering the two introductory chapters and the eight micro chapters and a final exam covering the eight macro chapters. The exams consisted of two parts Part I were multiple choice questions and Part II were problems. The multiple choice questions were selected from the test bank provided with the textbook. Both exams were administered in a proctored situation. The instructor kept a record of which questions were covered in online format and which questions were covered in traditional format. This sample differs from the five reviewed studies in that our sample permits the comparison of the performance of the same students in different instructional formats, whereas the other studies compared the performance of different students in different instructional formats. As can be seen from Table 2, 75.4% of the questions covered online were answered correctly while only 72.4% of the questions covered in the traditional format were answered correctly. However, without holding constant the effect of the other variables, it is difficult to make any conclusions about the difference in effectiveness in teaching formats just on the basis of the difference in means. The average difficulty level of the questions covered online was 3.6 (on a scale of 1 to 5, with 5 for the most difficult) and the average difficulty level of the questions covered in traditional format was 2.8. From the student specific variables, we see that 62% of the students were in the MBA program (the other students were in the MA program for engineers), the students had an
14 13 average undergraduate GPA of 3.24, a graduate GPA of 3.61 at the time they were taking this class and that 38% of the students were female. To check for the consistency of the test bank rankings we used similar questions from the same test bank in another offering of the course. This course covered the same material, used the same textbook and test bank, was taught in a different semester and was taught entirely in online format. The exam was proctored. The sample consisted of 36 multiple choice questions, and 20 students for a total of 720 observations. For each question we calculated the percent correct and compared that to the test bank ranking. We found that the correlation between percent correct and question difficulty ranking was and this was statistically significant at the 0.04 level. The sample of undergraduate students is from a principles of microeconomics course taught in 2 separate sections one in online format, the other in traditional format by the same instructor in the same semester. The course instructional materials included lecture notes in PowerPoint, homework assignments from the Aplia website (www.aplia.com), reading assignments from a standard principles textbook, and multiple choice quizzes. The online course included required participation in a weekly discussion board. Both sections took an identical cumulative final exam of 50 multiple choice questions at the same time in a proctored situation.
15 14 Table 3 Descriptive Statistics: Undergraduate Students Online Traditional Standard Standard Variable N Mean Deviation N Mean Deviation CORRRECT (percent correct) Question-Difficulty Rating Variables:* Rating from testbank (1=easy, to 3= hard) Rating from student response data (1=easy, to 3=hard) Number of Students: Student-Specific Variables: Level: 1=Fresh, 2=Soph; 3=Junior; 4=Senior; 5=Grad Major (Business or Economics =1) Credits this semester GPA Age Gender (Male=1) * Identical questions were given to each section. Undergraduate Principles of Microeconomics: As reported in Table 3, 68% of the questions covered in the online section were answered correctly while only 54% of the questions covered in the traditional format were answered correctly. Identical questions were used in each section. The average difficulty level of the questions from the test bank publisher s ranking was 2.0 (on a scale of 1 to 3, with 3 for the most difficult). To construct an alternative measure of difficulty we used the same questions in another offering of the course. This course covered the same material, used the same textbook and test bank, was taught in a different semester and was taught entirely in online format. The exam was proctored. These 50 questions were included in a stratified pool of 88 questions from which 59 questions were randomly drawn for a final exam. The exam was administered to 29 students. On average each question was selected 26 times with a standard deviation of 2.6. The difficulty rank of 1 was assigned for a percentage correct of 90 or above, 2 for a percentage correct
16 15 below 90 and equal to or above 70, and 3 to a percentage correct below 70. Using this scale the average difficulty ranking was We found that the correlation between percent correct and test bank question difficulty ranking was and this was statistically significant at the 0.05 level. From the student specific variables, we see that average class rank (on a scale of 1 to 5, with 1=freshman) is a junior in the online section, and a sophomore in the traditional format section. In the online section the average was older (22 compared to 21), the average GPA was higher (3.12 compared to 2.77), and the average credit hours taken during that semester was higher (14.1 compared to 12.4). The percent that were majors in economics of business was similar (11% in the online, 5% in the traditional) and the percent female was also similar (33% in the online, 32% in the traditional). We employ an estimation approach previously introduced by Marburger (2001; Marburger 2006). In this approach the exam questions are the units of observation. The student s answer to the multiple choice question, coded as 1 if correct, 0 if incorrect, is the dependent variable. One independent variable is an indicator variable for whether the question was covered in the online format (coded as equal 1) or in traditional format (coded as 0 ). This variable is used to determine if the probability of a correct response is different for questions covered in the online lecture of the course as compared to the traditional lecture. Another independent variable is a set of indicator variables, one for each student in the sample. These variables capture the effect of differences in unobservable student characteristics. Since the dependent variable is an indicator variable, a logistic regression is estimated. Because the exams are proctored the potential cheating bias is not larger for one instructional format than the other.
17 16 The empirical model for the hybrid class of MBA students takes the following form: MODEL 1: CORRECT i, = B 0 + B 1 ONLINE i, + B 2 FINAL_EXAM i, + B 3k 5 k=2 Q_DIFFICULTY i,k + B 4j m j=1 STUDENT_ID i,j + U ij where CORRECT i equals 1 if the student has the correct answer for question i; 0 otherwise; ONLINE i, = 1 if question i was taken from a chapter that was covered online; 0 otherwise; FINAL EXAM i equals 1 if question i was taken from the final; 0 for the midterm; Q_DIFFICULTY i,k is a dummy variable for difficulty level of question i ranging from k = 2 for easy, incrementing by up to k = 5 for hard; STUDENT_ID i,j is a dummy variable for student identity; it equals 1 if the ith question was being answered by the j th student; 0 otherwise; U i is a random error term for the i th student. The empirical model for the pair of online and traditional format classes with undergraduate students is the same as for the hybrid class: MODEL 2: CORRECT i, = B 0 + B 1 ONLINE i, + B 2 FINAL_EXAM i, + B 3k 3 k=2 Q_DIFF_TB i,k + B 4j m j=1 STUDENT_ID i,j + U ij with the exception that the variable for the question difficulty rating by the test bank (Q_DIFF_TB) variable has 3 levels. For MODEL 3, student response data represented by variable Q_DIFF_SR is substituted for Q_DIFF_TB. Our last model which uses the specification with dummy variables for each question and the empirical model is: MODEL 4: CORRECT i, = B 0 + B 1 ONLINE i, + B 2 FINAL_EXAM i, + B 3k 50 k=2 QUESTION_ID i,k + B 4j m j=1 STUDENT_ID i,j + U ij
18 17 Table 4 Exam Performance and Class Format Graduate Introductory Principles: Hybrid Undergraduate Principles of Microeconomics: Online v Traditional MODEL1 MODEL 2 MODEL 3 MODEL 4 SAMPLE Hybrid Online v F2F OL v F2F Classes OL v F2F Classes Variable Coefficient Pr > ChiSq Coefficient Pr > ChiSq Coefficient Pr > ChiSq Coefficient Pr > ChiSq Intercept ONLINE Q_DIFFICULTY_ Q_DIFFICULTY_ Q_DIFFICULTY_ Q_DIFFICULTY_ Q_DIFF_T_BANK_ Q_DIFF_T_BANK_ Q_DIFF_STU_RESP_ Q_DIFF_STU_RESP_ Student Indicators* YES YES YES YES Question Indicators* YES N Likelihood Ratio *The estimation results for the student and question indicator variables are not reported here.
19 18 As reported in Table 4, the results for the hybrid class (Model 1) show that the covered online variable (ONLINE) is positive and statistically significant at the level indicating that the students did better on the questions which came from the chapters covered online. The coefficient of 0.65 from Model 1 for the covered online variable (ONLINE) indicates that the students had a 65% higher chance of answering the question correctly if it was from a chapter covered online. In the hybrid MBA class, informally, students reported that in the online format they were able to listen to the power point slides at a time that was conducive to good learning and they were able to listen to the slides repeatedly, which they were not able to do with the traditional format. The disadvantage of the online format they reported was not being able to ask questions immediately upon having difficulty understanding a particular concept covered on the power point slide. The exam results for our sample suggest that in the hybrid format the advantages of online lectures offset its disadvantages. Models 2,3, and 4 report the results for the pooled sample of two sections one online and the other traditional format of an undergraduate principles of microeconomics class. Each model uses a dummy variable for each student. Across Models 2, 3 and 4 the coefficient of ONLINE is positive, and though in two of the three models the ONLINE coefficient is significant at the 0.10 level; its level of significance improves from in Model 2 to in Model 3, and to in Model 4. In Model 2 the dummy variables for difficulty rating by the test bank are negative and statistically significant at the level, indicating a reduced probability of correct response as question difficulty increases. The results in Model 3, a specification which differs from Model 2 only by using question difficulty ratings from student response data,
20 19 has similar results the coefficients are negative and statistically significant; but the likelihood ratio is almost doubled, from 180 to 343 indicating a significant boost in the explanatory power of the model. Finally, Model 4, which differs from Models 2 and 3 by virtue of using indicator variables for each question, has the highest explanatory value of the three specifications with a likelihood ratio of 498. An interpretation is that the indicator variables for each question (Model 4) have captured more of the influence on the probably of correct response than do the variables that capture only one dimension of question difference level of difficulty (Models 2 and 3). Other possible dimensions not captured by the question difficulty rankings are, for example, whether the question has been better covered than others in the text or lecture, and whether a similar question has appeared in a quiz, or homework or review sheet, either of which would increase the probably of correct response independent of the inherent difficulty of the question. III. Conclusions The issue of no significant difference between online and traditional instruction formats is especially important for non-traditional undergraduate students and MBA students because web instruction significantly expands their higher education opportunities. The promise of these opportunities would be less appealing if the mode of instruction inherently handicaps learning outcomes. Our study differs from previous research in two significant ways. First we use a different empirical model and second we report different findings. The existing five empirical studies compared matched sections of online and traditional instruction for courses in economics, used the student as the unit of observation and regressed test scores
21 20 on selected student characteristics. Two of these studies addressed the econometric problem of sample selection and omitted variable bias and each used the Hechman correction. Our study uses the exam question as the unit of observation and estimates a qualitative model of the probability of correct response. We address the econometric problem of sample selection and omitted variable bias by using an indicator variable for each student that captures the effect of differences in unobserved student characteristics on the probability of a correct response. Second, our findings differ from previous research on MBA level and undergraduate level students in principles of economics classes. Whereas, two of the three previous studies of MBA economics classes students concluded there was no significant difference in learning outcomes and the third concluded that the online was inferior 15, our finding is that the students performed relatively better in the online format relative to the traditional format. And, whereas, the two previous studies of undergraduate students in principles of economics 16, concluded that learning outcomes were lower in the online format compared to the traditional format, our study (consistent with the predominate finding in the literature in other disciplines) concludes that there is some indication of marginally significant results in favor of the online instruction. 15 Only one of these (Anstine) corrects for sample selection bias. 16 Only one of these (Coates) corrects for sample selection bias.
22 21 REFERENCES Anstine, J. and M. Skidmore (2005). "A Small Sample Study of Traditional and Online Courses with Sample Selection Adjustment." Journal of Economic Education 36(2): Bisping, T. O., Hilde Patron, Kenneth Roskelley (2008). "Modeling Academic Dishonesty: The Role of Student Perceptions and Misconduct Type." Journal of Economic Education 39(1): Brown, B. W. and C. E. Liedholm (2002). "Can Web courses replace the classroom in principles of microeconomics?" The American Economic Review 92(2): Burrus, R. T., K. McGoldrick, et al. (2007). "Self-Reports of Student Cheating: Does a Definition of Cheating Matter?." Journal of Economic Education 38(1): Coates, D. and B. R. Humphreys (2003). "An Inventory of Learning at a Distance in Economcs." Social Science Computer Review 21(2): Coates, D., B. R. Humphreys, et al. (2004). ""No Significant Distance" Between Face-to- Face and Online Instruction: Evidence from Principles of Economics." Economics of Education Review 23(6): Dickson, K. L. and. M. D. Miller. (2005). "Authorized Crib Cards Do Not Improve Exam Performance." Teaching of Psychology 32(4): Finn, K. V. and M. R. Frone (2004). "Academic Performance and Cheating: Moderating Role of School Identification and Self-Efficacy." Journal of Education Research 97(3): Harmon, O. R. and J. Lambrinos (2008). "Are Online Exams an Invitation to Cheat?." Journal of Economic Education 39(2): Kerkvliet, J. and C. Sigmund, L. (1999). "Can we control cheating in the classroom?" Journal of Economic Education 30(4): Lanier, M. M. (2006). "Academic Integrity and Distance Learning*." Journal of Criminal Justice Education 17(2): Marburger, D. R. (2001). "Absenteeism and Undergraduate Exam Performance." Journal of Economic Education 32: Marburger, D. R. (2006). "Does Mandatory Attendance Improve Student Performance?" Journal of Economic Education 37(2): McCabe, D. L., L. K. Trevion, et al. (2001). "Cheating in Academic Institutions: A Decade of Research." Ethics & Behavior 11(3): Navarro, P. and J. Shoemaker (1999). "The Power of Cyberlearning: An Empirical Test." Journal of Computing in Higher Education 1(1): Terry, N., J. Lewer, et al. (2003). "The Efficacy of Alternative Instruction Modes in Economics." Journal of Economics and Economic Education Review 4(1): Whitley, B. E. J. (1998). "Factors Associated with Cheating Among College Students: A Review." Research in Higher Education 39(3):