Tradtonal versus Onlne Courses, Efforts, and Learnng Performance Kuang-Cheng Tseng, Department of Internatonal Trade, Chung-Yuan Chrstan Unversty, Tawan Shan-Yng Chu, Department of Internatonal Trade, Chung-Yuan Chrstan Unversty, Tawan ABSTRACT The ssue of endogenety on learnng efforts has not been addressed n prevous researches dsentanglng on the relatonshp between learnng modes and outcomes. By conductng a two stage least squares procedure to control for ths endogenety bas, we fnd that onlne envronment s crucal for facltatng better student learnng performance. In addton, students who spend more studyng hours n Economcs sgnfcantly outperform ther peers makng fewer efforts. Keywords: Learnng Mode, Learnng Performance, Learnng Efforts, Endogenety INTRODUCTION Owng to the dstnct advantage of flexblty of dstance learnng, onlne courses have been quckly spread among educaton nsttutons over the past ten years. Ths stmulates the development of studes around the effect of learnng modes on learnng performance. In addton, efforts are found to be an mportant determnant of learnng performance. However, study effort s generally treated as an exogenous varable, whch volates the realty and the educaton producton theory. Ths gnorance of endogenous bas of efforts wll lead the classcal ordnary least squares estmators to be nconsstent and subsequently the msleadng nference n educatonal evaluaton. Accordngly, ths paper extends to reconcle the endogenety of efforts nto the relatonshp between learnng modes and learnng outcomes. There are advantages nherent n the onlne envronment (Km et al., 005) over the tradtonal ones. They comprse the convenence of learnng locaton choces and learnng schedule arrangement, flexblty and mass delvery as well as the feasblty of repeated learnng. Due to these advantages, there are more unverstes nvolved n offerng onlne courses. Consequently, t s mportant and necessary for educatonal nsttutons to understand and concern about the effectveness of those onlne courses, as compared wth the tradtonal courses. To respond to ths concern, many studes have been carred out to examne the mpact of learnng modes on learnng performance. However, exstng emprcal fndngs are largely n controversal and yet reach to a consensus. Some papers assert that lttle dfference n learnng performance was found between onlne format and tradtonal manner; see, e.g., Abraham (00) and Kekkonen-Moneta (00) for nstance. Some recent studes, for example, Anstne and Skdmore (005), Sauers and Walker (0004) and Kan and Cheung (007), seem to support that students of tradtonal classes have better performance than those of the onlne classes. Nonetheless, there are also papers showng that onlne mode s superor to tradtonal manner, see, e.g., Scay and Mlman (994) and Raynauld (006), among many others. It s surprsng that, to the best of our knowledge, none of the exstng research ever addressed the potental endogenety of learnng efforts n dsentanglng learnng performance. It s known that Krohn The Journal of Human Resource and Adult Learnng Vol. 6, Num., June 00 5
and Catherne (005) extend the standard educaton producton functon and student tme allocaton analyss to focus on the nteractons between student efforts and performance over the semester. The tme students spent s not exogenous but an equlbrum decson n the optmal process. Based on Krohn and Catherne (005), the optmal tme spent on the course can be wrtten as an mplct functon of the student s ntal ablty and parameters of the model. Ths suggests that the endogenety of learnng efforts should be consdered before formally examnng the relatonshp between learnng modes and performance. Ths paper hence s the frst one approprately addresses the ssue of endogenous learnng effort by dentfyng proper nstruments and employed subsequently n a two stage least squares analyss. Our approach suggests treatng the choce of learnng efforts as endogenous and uses the properly constructed nstrument to reexamne how learnng modes affect performance. We found learnng performance n the onlne envronment are superor to that n the tradtonal mode. Moreover, after controllng the endogenety bas of learnng efforts, the ncrease n weekly hours a student spent tends to mprove learnng performance. These results undoubtedly demonstrate a brand new pcture toward on-lne learnng that s dfferent from the prevous results obtaned n the lterature. We specfy our emprcal models n secton. Secton 3 ntroduces the data collecton and defntons of employed varables. Secton 4 explans the obtaned emprcal results and Secton 5 concludes. EMPIRICAL MODELS We conduct two emprcal models to nvestgate the relatonshp between learnng formats and learnng outcomes. As a benchmark for comparson and followng the typcal setup n prevous lterature n whch learnng effort s assumed to be an exogenous varable, we estmate the frst model usng ordnary least squares. In order to take the endogenous problem of learnng efforts nto consderaton, we use the two stage least squares (TSLS) approach n the second part n evaluatng the learnng performance. Results from both emprcal models are subsequently compared. A Typcal Model of Learnng Performance va OLS We utlze ordnary least squares frst to evaluate learnng performance whle controllng for other determnants of learnng performance as follows. S D H X, = N () where S represents the exam scores for the ndvdual student among N students n the data set. For each student, D captures the learnng mode; D = denotes the onlne envronment whle D =0 reveals the tradtonal manner. H reports the learnng efforts measured by the tme spent on Economcs learnng. α, ψ, and φ are parameters to be estmated.θ s a row vector of parameters measurng the effects of a column vector X of controlled varables on S. The unobserved random component of learnng outcomes s captured byε. The estmators from ordnary least squares wll be unbased and consstent f the error term s ndependent of explanatory varables such as Cov (H,ε )=0. As wll be shown subsequently, a model that overlooks any endogenety arsen n the employed varables suffers substantal estmaton bas and therefore wll mslead the statstcal nference. 6 The Journal of Human Resource and Adult Learnng Vol. 6, Num., June 00
An Endogenuty-Corrected Model va TSLS Exstng papers on the ssue of learnng modes and outcomes gnore the potental problem of endogenety of efforts snce they generally treat efforts as an exogenous varable for learnng performance. However, Krohn and Catherne (005) buld up an educaton producton model where each student optmally chooses hs efforts nput. As such, learnng efforts s hence endogeneously determned by ablty, gradng standards, and learnng experence. It s nature to expect the problem of measurement error would arse due to the endogenety of efforts. Once Cov (H,ε ) 0, estmators form ordnary least squares wll n generally be nconsstent. To solve ths problem, nstrumental varables that are hghly correlated wth efforts but uncorrelated wth performance should be dentfed and employed. To analyze the determnants of efforts, we examned some proxy varables followng the three dmensons (ablty, gradng standards, and learnng experence) suggested from Krohn and Catherne (005). On ablty, we use varables such as famly background factors and entrance exam performance; however, they are not crucal for efforts. As the gradng scheme for Economcs s on the teacher-level whch s dentcal for all students, t s unlkely that a fxed gradng standard wll explan much the dfference n efforts among students. At last, based on results from a correlaton analyss, learnng experence captured by reported studyng hours one perod ahead turned out found to be hgh and can be served as a sutable nstrumental varable for efforts. Moreover, a student s major also provdes a good nstrument for efforts. To properly deal wth the endogenety problem dsclosed n the recent lterature, we adopt a two stage least squares approach. The frst stage s to characterze the endogenety of efforts n the followng equaton (). H M H' X ', = N () where H s the reported weekly studyng hours capturng efforts for the ndvdual student among N students. M denotes student s major; M = f the student majors n Fnancal and Economc Law and otherwse M =0. H reports the tme spent on Economcs learnng one perod ahead. α ψ, and φ are parameters to be estmated.θ s a row vector of parameters measurng the effects of the column vector X' of all determnants on H mentoned n equaton (). The unobserved error term of learnng efforts s captured byε. After the ftted value of efforts s estmated from the frst stage, the second stage s smlar to equaton () except for usng the ftted efforts to substtute the orgnal efforts. DATA COLLECTION AND VARIABLE DEFINITION Dfferent from the extant lterature, we collect weekly data n two learnng modes from the same group of students. Durng the autumn semester n 009, Economcs courses are offered n the tradtonal n-class format before the mdterm exam, whereas they are offered completely onlne after that. Ths desgn avods the potental bas due to sample selecton when dfferent students take dfferent learnng manners. There are about 50 students enrolled n Economcs from two departments. Although the students major are dfferent, the contents of Economcs, ncludng teachng topcs, assgnment questons, and examnaton paper, are fully the same. Economcs s a three credt course taught over the full 6 weeks of the semester. An dentcal professor teaches n both tradtonal and onlne envronment as well as across varous classes. Ths hence enables to elmnate the potental bas that s lkely to be ntroduced when dfferent nstructors teach dfferent classes. The Journal of Human Resource and Adult Learnng Vol. 6, Num., June 00 7
Learnng performance s evaluated by standardzed exam scores termed as S. Course contents focus on the consumer theory before the mdterm and they are concentrated on the frm theory after the mdterm. In other words, students learn the consumer theory n the tradtonal envronment whle they learn the frm theory n the onlne mode. o control for the effect of course contents on learnng performance, standardzed scores are employed nstead of the orgnal exam scores. We compare the standardzed exam scores for the tradtonal and onlne courses holdng other thngs beng equal. An ndcator for learnng modes s represented by D where D = reveals the student n the onlne mode and D = 0 exhbts the student n the tradtonal manner. We examne the relaton between learnng formats and performance by testng the sgnfcance of the coeffcent of the mode dummy. Anstne and Skdmore (005) conclude that efforts postvely mpact performance. Learnng efforts, denoted by H, are measured by reported weekly hours devoted to Economcs. Moreover, based on the optmal studyng tme allocaton derved from the educaton producton theory, we also collect varables that are expected to affect a student s decson of learnng efforts. Followng the normal process n TSLS dealng wth the endogenety problem, at least one nstrumental varable s needed to be dentfed for efforts n the frst stage n addton to determnants of learnng performance n the second stage. A student s major as well as hs reported weekly hours one perod ahead are found to affect learnng efforts but not nfluence learnng performance. Accordngly, these varables become approprate nstruments for efforts. We denote separately these two nstruments by M and H where M = f the student majors n Fnancal and Economc Law but otherwse M =0. Exstng lterature fnd a postve correlaton between the exam score n perod t and that n perod t- for an ndvdual student. Dellana et al. (000) conclude that students prevous academc achevements are postvely assocated wth ther performance afterward n many studes. Accordngly we gather each student s exam scores one perod ahead denoted by S. Gender varable s generally consdered n the lterature and hence we set G = for the male whle G = 0 for the female. Female students were generally found to do better n certan academc subjects, such as language. Male students perform better n other subjects, for example, mathematcs and scence. Another knd of factors requred to be controlled s academc background whch was found to be a strong determnant affectng learnng performance. Pretest exam scores (PT) and entrance exam scores (ES) are controlled as the factors of relevant academc background. A pretest s admnstered at the begnnng of the class to evaluate the pror knowledge of Economcs. Students are asked to take ths pretest serously. Accordng to a theoretcal model of educaton producton functon proposed by Dolton, et al. (003), student characterstcs, such as ablty that are potentally dfferent among students, are crucal for learnng performance. To capture these characterstcs, we collect a student s famly background nformaton ncludng father s educaton level (DAD), together wth mother s educaton level (MOM). There are fve educaton levels, -5, separately representng the level of elementary school, junor hgh school, senor hgh school, unversty, as well as graduate school and above. Summary statstcs and explanaton for all the determnants necessary for the emprcal analyss are tabulated n Table. Table : Varable Defnton and Summary Statstcs Varables Defnton Mean St. Dev S Standardzed exam scores 0.00.00 D Learnng mode, D = for onlne whle D = 0 for tradtonal 0.57 0.50 H Efforts, reported weekly hours devoted to Economcs.45.89 8 The Journal of Human Resource and Adult Learnng Vol. 6, Num., June 00
G Gender, G = for the male whle G = 0 for the female 0.5 0.50 S Exam scores one perod ahead 55. 9.6 PT Pretest scores 39.60 4.3 ES Entrance exam scores 50.05 4.34 DAD Father s educaton level, -5 3.3 0.95 MOM Mother s educaton level, -5.96 0.84 M Major, M = for Fnancal and Economc Law whle M =0 0.46 0.50 otherwse H Reported weekly hours one perod ahead.36.7 EMPIRICAL RESULTS Table summarzes our emprcal results. The second column shows the results from the typcal model overlookng the learnng effort endogenety estmated va OLS. As efforts are assumed to be exogenous, reported weekly hours devoted to Economcs are drectly used to capture efforts. The thrd and the fourth columns demonstrate the estmaton results from a TSLS approach. Due to the endogenety of efforts, nstrumental varables are needed n the frst-stage efforts equaton. As a consequence, the second-stage regresson equaton on learnng performance s specfed as a functon of the ftted efforts. Table : OLS and TSLS Coeffcents for Standardzed Exam Scores OLS TSLS TSLS Second stage Frst stage D 0.07 0.484** -3.539*** (0.08) (.98) (-9.55) H 0.037 (0.96) H ftted 0.69*** (.66) G -0.33*** -0.405*** 0.477 (-.7) (-.88) (.3) S 0.07*** 0.06*** 0.030 (7.5) (5.88) (.) PT -0.0055-0.00338 0.04 (-0.5) (-0.59) (0.79) ES 0.036** 0.05 0.068 (.33) (0.80) (.3) DAD -0.0668-0.00497-0. (-0.84) (-0.05) (-0.95) MOM 0.4 0.0590 0.00 (.) (0.55) (0.7) Constant -3.09*** -.66*** -0.836 (-3.95) (-.94) (-0.36) M.97*** (.89) H 0.38*** (5.9) Number of observatons 5 44 44 Adjusted R 0.339 0.399 0.455 Note: Standard devatons are n parentheses. ** denotes for sgnfcance at the two-tal sze of 0.05 and *** denotes for statstcally sgnfcance at the two-tal sze of 0.0. The Journal of Human Resource and Adult Learnng Vol. 6, Num., June 00 9
OLS wth Exogenous Efforts Results from the ordnary least squares estmatng the determnants of a typcal learnng performance model are presented on the frst column n Table. The results suggest that, on average, boys' exam scores are nferor to those from grls. Exam scores are postvely assocated wth ther scores one perod ahead. Entrance exam grades postvely affect learnng performance n Economcs. Nether father s educaton level nor mother s educaton level sgnfcantly affects learnng performance. Ths arses due to the hghly postve correlaton between these two determnants. After elmnatng the mult-collnearty problem by droppng father s educaton level, a sgnfcantly postve nfluence of mother s educaton level on performance s apparent. Learnng modes show no sgnfcant effect on performance as efforts are treated as an exogenous varable. It s nterestng that Efforts are not sgnfcant determnants of performance as well. It s noted that, however, Schmdt (983) has ponted out that learnng efforts can be expressed as an mplct functon of ablty or other factors whch suggests that efforts are endogenous varables. Ths remnds us that the gnorance of the endogenety of efforts may lead to the potentally based nsgnfcance of efforts and learnng modes on performance. A better econometrc recpe that can accommodate ths ssue s called for. TSLS wth Endogenous Efforts Our nvestgaton verfed the presence of endogenety of learnng efforts n our data. The covarance of two error terms, ε andε, separately n the performance and efforts functon s -0.97 and sgnfcantly dfferent from zero. In addton, the coeffcents of the two nstrumental varables, reported hours one perod ahead and undergraduate majors, are also dfferent from zero at a % sgnfcant level. These results suggest that TSLS s useful n correctng for the endogenety-nduced nconsstency under the typcal OLS approach. Gven that the choce to learnng efforts may be related to learnng performance, we employ a two stage least squares model. A two stage least squares analyss yelds fndngs dfferent from prevous studes usng ordnary least squares estmaton. When other varables are well-controlled, a two stage least squares analyss reveals that learnng performance n the onlne envronment are superor to that n the tradtonal mode. After controllng the endogenety bas of efforts, the ncrease n weekly hours a student spent tends to mprove learnng performance; however, ths relatonshp cannot be shown when OLS s used. CONCLUSIONS As dstance-learnng courses have dramatcally prevaled among conventonal educaton nsttutons over the past ten years, a great number of papers have focused on the mpact of onlne versus tradtonal learnng mode on learnng performance. In these papers, student effort s generally consdered as an exogenous varable. Nonetheless, ths assumpton obvously volates the realty as well as the educatonal producton theores. Ths paper hence ntegrates the endogenety of efforts nto the relatonshp between learnng formats and learnng outcomes. A smple OLS regresson shows that holdng other determnants constant, learnng modes do not sgnfcantly determne learnng performance. After correctng for the endogenety bas, however, TSLS reveals the effect of learnng modes. Whle Economcs s generally vewed as a hghly quanttatve course that students typcally fnd challengng n 0 The Journal of Human Resource and Adult Learnng Vol. 6, Num., June 00
most learnng envronments, ths paper suggests that learnng performance of a quanttatve course can be successfully enhanced n an onlne envronment. REFERENCES Abraham, T. (00), Evaluatng the Vrtual Management Informaton Systems Classroom, Journal of Informaton Systems Educaton, 3 (), 5-33. Anstne J. and M. Skdmore (005), A Small Sample Study of Tradtonal and Onlne Courses wth Sample Adjustment, Journal of Economc Educaton, 36 (), 07. Dellana, S. A., W. H. Collnns, and D. West (000), Onlne Educaton n a Management Scence Course- Effectveness and Performance Factors, Journal of Educaton for Busness, 77 (5), 57-63. Kan, A. C. N. and L. L. M. Cheung (007), Relatve Effects of Dstance versus Tradtonal Course Delvery on Student Performance n Hong Kong, Internatonal Journal of Management, 7 (4), 763-83. Kekkonen-Moneta, S., and G. B. Moneta (00), E-Learnng n Hong Kong: Comparng Learnng Outcomes n Onlne Multmeda and Lecture Versons of an Introductory Computng Course, Brtsh Journal of Educatonal Technology, 33 (4), 43-433. Km K. J., S. Lu and J. Curts (005), Onlne MBA Students Perceptons of Onlne Learnng:Benefts, Challenges, and Suggestons, Internet and Hgh Educaton, 8, 335-344. Krohn G. A. and M. O. Catherne (005), Student Effort and Performance Over the Semester, Journal of Economc Educaton, 36 (), 3-8. Raynauld J. (006), A Comparson of Onlne And Face to Face Learnng n Undergraduate Fnance And Economcs Polcy Courses, Prelmnary, HEC Montréal. Sauers, D. and R. C. Walker(004), A Comparson of Tradtonal and Technology-Asssted Instructonal Methods n the Busness Communcaton classroom, Busness Communcaton Quarterly, 67 (4), 430-44. Scay, R. A. and M. I. Mlman (994), Interactve Televson Instructon: An Assessment of Student Performance and Atttudes n an Upper Dvson Accountng Course, Issues n Accountng Educaton, 9 (), 80-95. Schmdt R. M. (983), Who Maxmzes What? A Study n Student Tme Allocaton, The Amercan Economc Revew. 73 (), 3-8. Acknowledgement Shan-Yng Chu sncerely thanks NSC for the fnancal support (NSC 98-40-H-033-05). The Journal of Human Resource and Adult Learnng Vol. 6, Num., June 00