Online Homework and Student Performance: Evidence from a Natural Experiment

Transcription

1 Online Homework and Student Performance: Evidence from a Natural Experiment Karen Menard, Bridget O Shaughnessy and Abigail Payne

2 Online Homework and Student Performance: Evidence from a Natural Experiment Karen Menard** Bridget O Shaughnessy* Abigail Payne* With Olesya Vovk* and Katherine Swierzewski* May, 2014 Preliminary Draft Do not cite Abstract: Recent research has shown that online homework in economics courses can provide a small but statistically significant improvement in student achievement. During the fall 2010 semester, it was brought to the attention of instructors at McMaster that the provincial government's policies regarding ancillary fees at universities prohibit the mandatory use of online resources linked to students' grades. In Introductory Macroeconomics, the enforcement of the policy had a dramatic impact on how the course was delivered. Prior to fall 2010 online homework, delivered through a website called "Aplia", comprised twenty percent of students' grades. Access to the website came free with the purchase of a new text book. Students who did not wish to purchase a new hard copy book could purchase access to the Aplia website which included an electronic copy of the text. For the winter 2011 semester, the online homework component was dropped, and students were evaluated using only mid-term tests and a final exam. During the academic year, students had the option of doing online homework for grades, or just having their grade made up of the tests and exam. The enforcement of the provincial policy regarding the use of online tools provides a natural experiment that can be used to study the effectiveness of such tools - over 5000 students from across campus are involved in the study. Preliminary results indicate that the online homework improves student learning outcomes, particularly for weaker students. *Department of Economics and Public Economics Data Analysis Laboratory (PEDAL), McMaster University ** Ontario Institute for Cancer Research

3 Online Homework and Student Performance: Evidence from an Online Experiment 1. Introduction With class sizes ever increasing, and teaching resources constrained, instructors would like to use high quality online resources to assess students. The current study will address the following question: are online resources effective as teaching and learning tools? As explained in more detail below, we study this question by taking advantage of the natural experiment that occurred when the provincial policy around the use of online resources was strictly enforced in From 2008 to 2012, all sections of a large enrollment economics course during the regular school year (September-April) were taught by the same instructor, using the same text book and largely the same evaluation methods. Until the Fall 2010 term, students wrote two term tests and a final exam and did mandatory online homework, which was provided by a publishing company for a fee. The fee included an online version of the text book at a lower price than purchasing a new edition of the paper book. During the fall 2010 term, it was brought to the attention of the instructor that the provincial policy regarding ancillary fees prohibits instructors from mandating that students purchase access to such websites for graded work. As a result of this new information, the online homework portion of the grading scheme was dropped for the winter 2011 term. After consultation with administrators, it was determined that the use of such online learning tools was acceptable as long as they were optional to students. Beginning in fall 2011, students could purchase access to the online homework and use it for grades. Students who did not choose to

4 purchase the online homework package had the weight transferred to their final exam. A breakdown of the grading scheme for each term in the study is given in Table 1. The two term tests were not weighted equally the test with the higher score was given more weight in the calculation of a student s final grade. The exact weights are given in parentheses. The online homework included a math test designed to review basic math concepts taught in high school, weekly homework assignments based on lectures and textbook readings, and, when the homework was mandatory, assignments based on online experiments. During the experiments, hundreds of students met online in a simulated market. Each student was given a role as either a buyer or seller, or a lender or borrower, and placed bids through their internet browser. When the computer found a match between two parties, the transaction was recorded. Online discussions followed several rounds of trading in the market. For each of the two experiments in 2010, students were required to complete two homework assignments, one prior to the experiment and one after the experiment was completed. When the online homework system became optional, the experiments were dropped from the course and the only online homework assignments were based on the lectures and textbook readings. In our analysis, we will examine student performance in the course overall as well as their performance on individual tests and the exam. Table 2 shows the timing of the homework assignments relative to the tests and final exam for the two terms with online homework. The remainder of the paper is structured as follows: Section 2 presents a literature review; Section 3 describes the methods used and data set development. Section 4 presents our results, and Section 5 concludes.

5 2. Literature Review There are two bodies of literature that directly inform this study. The first relates to student retention. Finnie and Qiu (2008) find that first year students are more likely to leave university than students who make it into their second year of studies. We know that students who have a connection to faculty tend to perform better than students who do not. Most first year students face large classes and we also know that it is difficult to get this connection to faculty in classes of 400 to 600 students (Nagda et al, 1998, Chickering and Gamson, 1987). Finnie and Qiu also report that the most common reason students leave university is didn t like it/not for me. Chickering and Gamson emphasize time on task as a core principle in good teaching. With very large classes, offering traditional paper homework assignments, graded by the instructor or by teaching assistants, is impractical. Instructors are using online homework both to engage students and to provide students with more time on task, so gathering evidence of its efficacy is an important area of work. Bosshardt (2004) studies the decision of first year economics students to drop the course. Based on student characteristics, he divides the class into groups that are good students or at-risk students, those with a high probability of receiving a grade of D or F. Some of the at-risk students dropped the class while others did not. Bosshardt finds that of the at-risk students who dropped the course, only 16% remained at the university they were attending. One particularly interesting aspect of our project will be whether or not there are differential impacts with online homework does the online homework best serve good students or atrisk students? Does the answer depend on whether or not the homework is required or optional?

6 Another body of work that is relevant pertains to the effectiveness of online homework in economics. In a survey of 700 students conducted by Kennelly and Duffy (2007), students overwhelmingly favoured online homework compared with traditional paper homework, felt that using the publishers online homework tool helped improve their understanding of the course material, and wanted their instructors to use the website in the future. Nguyen and Trimarchi (2010) study two different publishers online homework tools, finding that online homework had a small but statistically significant impact on student grades. Use of such websites led to a 2% increase in grades whether or not the online homework was required. The experiments Nguyen and Trimarchi (2010) conducted involved several sections of the course taught by the same instructor, some sections having online homework and others not having online homework. In one experiment, no variables were included to control for ability or other student characteristics. In the other experiment information about student characteristics was obtained via a student survey. Our study uses administrative data and information from students university applications to determine student characteristics, which are likely to be more reliable than survey responses. Additionally, we are including three distinct cases in our study one with mandatory online homework, one with optional online homework, and one with no online component at all. Lee, Courtney, and Balassi (2010) compare online homework with traditional paper homework graded by a teaching assistant. The core measure of student learning is the difference between pre- and post-test scores on the Test of Understanding in College Economics (TUCE), a commonly used measure in the economic education literature. They find that differing homework methods do not affect students improvement in TUCE scores, despite anecdotal evidence that students prefer online homework. One drawback of the study

7 is the small sample size they consider three sections of a microeconomics principles course with enrollments of between 46 and 77. When we are considering large classes with approximately 1,500 students in each of the three semesters of analysis, the choice is often not between paper-based homework and online homework, but between online homework and no homework at all. Clearly, this ties in with Chickering and Gamson s emphasis on time on task. 3. Methodology and Dataset Development 3.1 Natural Experiment Framework To understand better the effectiveness of online homework on student performance in large enrollment classes, we are utilizing a natural experiment type of methodology. As explained above, prior to the winter term of 2011, the instructor of a large enrollment economics course (five sections over an academic year, 2500 students across these sections) the online homework was mandatory. For the winter term of 2011, no online homework was offered. Starting in the fall term of 2011, online homework was offered but was an optional component. Thus, we can study two natural experiments. Experiment #1 is the comparison of a control group that does not have any online homework (winter 2011) with a treatment group that has required online homework (winter 2010). Experiment #2 is the comparison of a treatment group of students that are offered optional online homework (winter 2012) with two control groups: those with no online homework (winter 2011) and those with required online homework (winter 2010). Note that we are only using students enrolled in the winter terms as we have observed slight differences in student background characteristics across the fall and winter terms. Thus the comparison of the effects no online homework, mandatory

8 online homework, and voluntary online homework regimes are best done only on winter term students. In an ideal setup, we would observe the same student under both the control and the treatment conditions. When it comes to matters such as education and other social science based experiments, however, it is rare to accomplish this. Instead we must compare similar students that are exposed to different treatments. This requires a level of confidence that the students exposed to the different treatments are similar in all ways except for the instrument being tested (the use of online homework). Fortunately, in our set up, we have the students being exposed to the same instructor, the same lectures, the same text book, and the same test bank of multiple choice questions. The students may differ in terms of program of study (e.g. engineering, commerce, economics, social science) and they may differ in terms of the level of preparation. We have developed a rich data set that will help us to control for these differences in our multivariate analysis. 3.2 Data We have constructed the data set by linking the following measures from four main sources of data. The primary source of data was the course data collected by the instructor. The data are collected for all students who registered to take the course in three winter terms: winter 2010, 2011 and We have developed measures to reflect marks received by the student on two term tests, the final exam, as well as marks on each of the 14 homework assignments and math evaluation test for the terms where homework was offered. Additional measures were created to capture the weight of each test and each homework assignment used to calculate the final course grade. Both the term tests and the final exam consisted of

9 multiple choice questions and short answer questions. To ensure consistency, we are using only the multiple choice sections from the term tests and final exam for all terms under study. The second data source was obtained from the university s Office of the Registrar and includes information on other courses taken by the students. From that source, we created measures to capture overall student performance in the fall term preceding the course as well as the winter term of the course. We calculated the number of credits taken (full or partial load), average GPA in each term, and flagged any other economics courses taken by the student, as well as previous attempts at the course under study. Other measures include the level of enrolment (level 1, level 2, etc), and program of study where the student is registered during the term in question (e.g. engineering, commerce, social science, humanities). The Registrar s data also provide us with information on students who withdrew from the university entirely but were listed in the course data as failed the course. The third data source was information on the applications submitted by the students for admission to the university through the Ontario University Application Centre. The data records include applications for students applying directly from an Ontario high school (known as 101 students) and delayed entry and/or non-ontario high school students (known as 105 students). The 101 set of applications capture information on the students performance in level 4 (grade 12) courses in high school and their home postal code. From the 105 set of applicants the high school grade information was more limited. The location of their residence and home postal code (if from Canada), however, were available. Using the high school course information, we calculated the average of the best 6 grade 12 U-level courses, which is equivalent to the general admission requirements for the university.

10 The fourth source of the data was the 2006 Canadian Census. Using the home postal code measures, we added the information on socio-economic characteristics of students neighbourhoods such as average household income, share of immigrants, age and education profiles. The census geography utilized was the dissemination area, a geography that covers roughly 500 households. Table 3 captures basic information on the students who took the economics course in 2010, 2011, or 2012 and compares them with entry level students at the university in these years as well as entry level students across all universities in Ontario. Compared to the university as a whole, the economics course contains a higher share of male students, higher shares of commerce and engineering students, and higher shares of immigrant students. There are lower shares of science/health and humanities students. Overall, however, the students share similar high school GPAs and other characteristics. In Table 4 we report summary statistics that compare students enrolled in the three terms the macroeconomics course was offered. Many of the demographic characteristics of students in the three terms are comparable. About the same proportion of students attended 6 or more years in a Canadian school system, speaks English as their first language, applied for OSAP, and come from low-, middle-, and high-income neighbourhoods. Performance in high school is also similar across students registered in all three terms. The average of the students best six grade 12 courses is virtually identical, at about 86%, and almost all 101 students had at least one grade 12 math course. Another strikingly similar characteristic is the proportion of students carrying a full time course load during the terms under study, 83-84% in each term. There are a few moderate differences across terms. In 2010 the proportion of males taking the economics course was lower while the proportion of Canadian citizens was higher. A

11 smaller proportion of the class was comprised of business and engineering students in 2010 than in the other terms. This is likely due to scheduling issues, and it seems plausible that business and engineering students took the course in the fall 2009 term rather than the winter term in The largest differences can be seen in the students performance on the online math test (for the two semesters the online homework was available) and in the percentage of the class enrolled in level 1. The average in the online math test was 92% in 2010 and 82% in Incentives may matter in 2010 the math test was worth 5% of the students final grade, while in 2012 it was only worth 3%. Level 1 students make up a higher proportion of the class in 2012 than either 2010 or 2011 (80%, 68%, and 72% respectively). Overall, the students appear quite similar in all three of the terms under study. 3.3 Empirical Model Our focus is on student performance on the multiple choice section of two term tests and the final exam. Student i s test score, Ti, depends individual student information obtained from course records, the Registrar and the student s university application, Xi, and information about the dissemination area the student resided in when they applied for university, Ni. (1) T = + H i +X X +N N + Individual characteristics include a dummy variable for online homework, Hi, faculty, gender, level at the university (first year or higher), English as a second language, high school average grade (best six courses), OSAP, years in Canadian school system, age, citizenship, and a dummy variable for 105 students. Neighbourhood characteristics include measures of income and educational attainment, visible minority percentage, percentage of

12 one-parent families, youth unemployment, population, and proportion of population between the ages of 15 and 24. Robust standard errors are clustered by DA level. Standard errors are clustered when members of a group (in this case census dissemination area, approximately 500 households) are likely to share some characteristics that are unobservable. 4. Results 4.1 Mandatory Homework Compared with No Homework Figure 1 shows the average score on the multiple choice section of two term tests and the final exam for the year in which online homework was mandatory and year without online homework. Mean scores are slightly lower on the term tests, dramatically lower on the final exam. When online homework was mandatory, the mean on the final exam MC section was 67.8%, compared with 55.2% in the term with no homework. Distributions of the test and exam grades are given in Figure 2. Again, the difference in term test grade distributions is not large. For the final exam, the shapes of the distributions are strikingly similar, but the entire distribution is shifted to the left in the term with no online homework. Regression results are provided in Tables 5A and 5B. Our key variable of interest, a dummy variable equal to one for students with online homework, is statistically significant for both term tests and the final exam. A student s grade on each term test is likely to be about two percentage points higher in the term with mandatory homework, while the final exam grade is over twelve percentage points higher. Faculty of enrolment has a statistically significant impact on test performance in most cases. Relative to Social Sciences students, Business and Science/Health Science students perform better on both tests and the final exam. The only significant difference for Engineering students is on the final exam, outperforming Social Sciences students by over three percentage points. Humanities students

13 fare worse on all three assessments, but the difference on the first test is not statistically significant. Any grade difference of more than three percentage points is meaningful to the students in the sample. The university under study uses a twelve point grading system to calculate cumulative averages, where an F equals zero, D- equals one, D equals two, up to A+ equalling twelve. For grades in the D, C, and B ranges, an increase in grade of three percentage points is enough to move students up to the next grade level. For example, a student with a grade of 67% has a C+, which translates to six out of twelve, while a student with 70% has a B-, which is seven out of twelve. Gender, high school grades, and being an upper year student all have statistically significant and meaningful impacts on test and exam performance. The largest of these coefficients is on upper year, with upper year students scoring over five percentage points better on the term tests and almost three percentage points better on the final exam. Females perform between one and two percentage points worse than males, and a one percentage point increase in high school average leads to about a one percentage point increase in test and exam performance. Census measures are all numerically small and insignificant, with the exception of income. Students in lower income neighbourhoods have slightly lower scores on both term tests and the final exam. However, only two coefficients are statistically significant: the final exam for the lowest tercile and the first term test for the middle tercile. 4.2 Mandatory Homework Compared with Optional Homework Figures 3 and 4 show the test/exam means and grade distributions for mandatory homework and optional homework. These figures look strikingly like Figures 1 and 2, where we compared mandatory homework with no homework. Performance on the first term test

14 and final exam are lower in the year with optional homework, with the final exam mean dropping by more than eleven percentage points when homework is optional. The mean scores on the second term test are identical in the two years. Optional homework leads to a four percentage point decrease in Test #1 and almost twelve percentage point decrease in the final exam compared with mandatory homework. Performance on test 2 is not affected by the optional nature of the online homework. A student s faculty of enrollment affects test/exam performance, with Business, Engineering, and Science/Health students outperforming Social Science students on all assessments by 1.5 to 5.6 percentage points. All are statistically significant except for Engineering students performance on the second term test. Being an upper year student, male, a native English speaker, and having a higher high school average all contribute to higher scores on the tests and final exam. Living in a low income neighbourhood negatively impacts test/exam performance. None of the other census variables are statistically significant or meaningful in terms of coefficient size. 5. Conclusions Does online homework improve student performance on term tests and exams? When the homework is mandatory, the answer is yes. Students with mandatory online homework did better on term tests by two to four percentage points than students with no homework or with optional homework. Final exam results were even more striking mandatory online homework leads to an eleven to twelve percentage point increase in the final exam score, again compared with no homework or optional homework. These results were obtained using large samples of over 2500 students, controlling for many individual and neighbourhood characteristics.

15 The next step in this research is to compare the performance of students who chose to do the homework with those who did not during the term when homework was optional. We also need to examine the relationship between homework and test performance for weaker students compared with stronger students. We will conduct propensity score matching analysis to further explore the relationship between online homework and test/exam performance.

16 References Bosshardt, W. (2004) "Student drops and failure in principle courses", Journal of Economic Education, 35(2) Chickering, A., Gamson, Z. (1987) Seven principles for good practice in undergraduate education, American Association of Higher Education Bulletin, 39 (7) 3-7. Finnie, R., Qiu, H. (2008) The patterns of persistence in post-secondary education in Canada: evidence from the YITS-B dataset, accessed 19 March 2014 at Kennelly, B., Duffy, D. (2007) Using Aplia software to teach principles of economics, paper presented to Developments in Economic Education Conference in Cambridge, UK, 6-7 September. Lee, W., Courtney, R., and Balassi, S. (2010) Do online homework tools improve student results in principles of microeconomics courses? American Economic Review, 100(2) Nagda, B. A., Gregerman, S. R., Jonides, J., von Hippel, W., Lerner, J. S. (1998) Undergraduate student-faculty research partnerships affect student retention, The Review of Higher Education 22 (1) Retrieved from Nguyen,T. and Trimarchi, A. (2010) Active learning in introductory economics: do MyEconLab and Aplia make any difference?, International Journal for the Scholarship of Teaching and Learning, 4(1)10.

17 Table 1 Winter, 2010 Winter, 2011 Winter, 2012 Term Tests (25%, 15%): Final Exam: Online Math Test: Online Homework: 40% 40% 5% 15% Term Tests (30%, 20%): Final Exam: 50% 50% Term Tests (25%, 20%): Final Exam: Online Math Test: Online Homework: 45% 40% 3% 12% Table 2: Course Schedule, Winter 2010 and Winter 2012 Winter, 2010 Winter, 2012 Week Date Homework Test Week Date Homework Test 2 Jan. 15 Math Test 2 Jan. 13 Ch. 2 Ch. 2 3 Jan. 18 Jan. 22 Experiment #1 Prep Ch. 3 3 Jan. 17 Ch. 3 Math Test 4 Jan. 25 Experiment #1 4 Jan. 24 Ch. 4 Jan. 29 Analysis Ch. 4 5 Feb. 6 Test # 1 (1-4) 5 Jan. 31 Ch. 5 6 Feb. 12 Ch. 5 Ch. 6 (practice) 6 Feb.7 Feb. 11 Ch. 6 Ch. 7 Test # 1 (1-6) 7 Feb Reading Week 7 Feb. 14 Ch. 8 8 Feb. 24 Ch. 7 8 Feb Reading Week 9 Mar. 5 Ch. 9 (practice) 9 Mar. 2 Ch. 9 Ch Mar. 13 Test #2 (5-10) 10 Mar. 6 Mar. 10 Ch. 10 Ch. 11 Test # 2 (7-11) 11 Mar. 19 Ch Mar. 13 Ch. 12 Ch. 12 (practice) 12 Mar. 22 Experiment #2 Prep 12 Mar. 20 Ch. 13 Mar. 26 Ch Mar. 29 Experiment #2 13 Mar. 27 Ch. 14 Analysis 14 Apr. 5 Ch Apr. 3 Ch. 15 Apr. 9 Ch. 15 Apr. 14 Final Exam Apr. 7 Final Exam

18 Table 3: Sample comparison with the student population Sample students All (1) students at university under study All (2) Ontario students (1) (2) (3) Total students in sample 4,295 18, ,633 Delayed or non-ontario students (% 105 students) 11.0% Gender (% male) 63.4% 46.7% 43.9% Immigrant status Canadian Citizen 77.8% 88.4% 91.1% Permanent Resident 11.0% 7.3% 5.7% Other 11.2% 4.3% 3.1% Years in Canadian K-12 school system 6 years or more 76.0% 88.3% 90.8% 3-5 years 13.4% 7.2% 5.1% 2 years or less 10.6% 4.5% 4.1% % with English reported as their primary language 65.6% 75.8% 76.9% Indication on application that an application for financial aid was submitted (% applied) 61.7% 67.6% 65.4% Students with ON postal code (3) 4,019 18, ,933 % living in low income neighbourhood 15.3% 15.3% 20.3% High School GPA (Best 6 University or Mixed Courses) Average Best 6 GPA 86.2% 86.1% 83.3% (standard deviation) (5.1) (5.5) (6.5) (Note: Differences of group means are statistically different from zero at 1% confidence level across all groups) 5th percentile of best % 76.8% 72.5% Median best % 86.0% 83.5% 95th percentile of best % 95.5% 93.8% Students by Program of Registration, Year 1 (4) Commerce (includes Social Sciences) 26.7% 11.9% 14.1% Engineering 28.2% 19.1% 9.0% Science/Health 24.5% 33.2% 28.4% Humanities 13.5% 30.8% 40.0% Other 7.2% 5.0% 8.5% NOTES: (1) All students who applied directly from HS in 2010, 2011 and 2012 and were registered at the university under study. (2) All students who applied directly from HS in 2010, 2011 and 2012 and were registered at any University. (3) OUAC 101 students (direct applicants) may have a non-ontario postal code if they attend an International Ontario approved High School. For schools located in Ontario, if student postal code was invalid, school postal code was used. (4) Program of registration is based on transcript data for column 1 and application data for columns 2 and 3

19 Table 4A: Demographic Characteristics of Sample Students: Comparison Across Terms Absence of Online Homework Presence of Online Homework Winter 2011 Winter 2010 Winter 2012 (1) (2) (3) Total students per term ,454 Gender (% male) 64.73% 59.14% 66.23% Immigrant Status Canadian Citizen 76.40% 80.17% 76.82% Permanent Resident 11.45% 10.87% 10.80% Other 12.15% 8.96% 12.38% Students by number of years in Canadian school system (at admission) 6 years or more 73.95% 77.77% 76.07% 3-5 years 15.08% 12.35% 12.93% <= 2 years 10.96% 9.88% 11.00% % with English reported as their primary language 64.53% 66.97% 65.27% Students by OSAP application status Appied for OSAP (% applied) 60.68% 62.74% 61.62% Students by neighbourhood of residence (if residing in Ontario at admission) Students with ON postal code ,356 % living in low income neighbourhood (bottom tercile) 20.28% 20.95% 18.66% % living in mid-income neighbourhood (middle tercile) 23.25% 23.59% 21.50% % living in high income neighbourhood (top tercile) 56.46% 55.46% 59.84% Median distance to the University, km

20 Table 4B: High School Information of Sample Students: Comparison Across Terms Absence of Online Homework Presence of Online Homework Winter 2011 Winter 2010 Winter 2012 (1) (2) (3) High School GPA (Best 6 University or Mixed Courses) Average Best Standard Deviation of Best 6 (5.0) (5.3) (4.9) Average Grade difference with Winter 2010 students (0.2) (0.2) Average Grade difference with Winter 2011 students 0.2 (0.2) Median Best High School Math (average of all grade 12 University stream courses) # students with at least one Math course (% 101 students) 98.2% 97.5% 97.8% (% of all students) 85.3% 88.9% 86.9% % taking level 12 Math courses (101 students) 0 math courses 1.85% 2.48% 2.24% 1 math course 7.07% 10.53% 5.19% 2 math courses 62.22% 57.78% 61.46% 3 math courses 28.54% 27.27% 30.80% 4 or 5 math courses % 1.94% 0.31% Average of Best Math grade (standard deviation) (8.2) (8.4) (7.9) Median Best Math grade Only 5 students in the entire sample took 5 grade 12 math courses, all of whom were in the economics course in 2010.

21 Table 4C: University Information of Sample Students: Comparison Across Terms Absence of Online Homework Presence of Online Homework Winter 2011 Winter 2010 Winter 2012 (1) (2) (3) Students by Enrolled Faculty At Time of Course Business 27.65% 23.78% 28.27% Social Sciences 12.08% 13.62% 11.83% Engineering 27.72% 24.84% 31.64% Science/Health 29.12% 34.65% 24.97% Humanities 2.72% 2.40% 2.89% (N/A) 0.63% 0.64% 0.41% % students registered in Level % 68.38% 79.78% Concurrent or Past Enrollment in First Year Microeconomics Course Number of students 1,215 1,159 1,305 Mean performance in microeconomics course (scale of 12) (standard deviation) (2.9) (2.6) (3.0) Performance in Course Online Math Test (standard deviation) (14.5) (31.3) Overall Term Performance What was the total # credits taken in the Term Average # of credits (standard deviation) (3.4) (3.2) (3.5) Minimum credits Median credits Max credits Share of students with "full time loads" 83.80% 83.98% 83.22%

22 Figure 1 Percentage on Test Average Term Test and Exam Scores Based on Online Participation Term Test One Term Test Two Final Exam 55.2 Winter 2010 Mandatory Online Homework Winter 2011 No Online Homework

23 Figure 2A Distribution of Midterm Grades by Term Fraction of Students Term Test One Fraction of Students Term Test Two Distribution of Grades Distribution of Grades Winter 2010 Winter 2011 Figure 2B Distribution of Final Exam Grades by Term Fraction of Students Distribution of Grades Winter 2010 Winter 2011

24 Table 5A Winter 2010 (Mandatory Online Homework) and Winter 2011 (No Online Homework) Midterms and Final Exam Regressions VARIABLES (1) (2) (3) Test 1 Grade, Percentage on Multiple Choice Test 2 Grade, Percentage on Multiple Choice Final Exam Grade, Percentage on Multiple Choice Online Homework Mandatory (0.484)** (0.475)** (0.393)** Transcript Measures Business (0.891)** (0.947)** (0.747)* Engineering (0.921) (0.985) (0.758)** Science/ Health (0.999)** (1.083)** (0.873)** Humanities (1.793) (1.729)* (1.340)* OUAC Measures Upper Year Student (0.645)** (0.706)** (0.593)** Distance from University (0.003)* (0.003) (0.002)** Female (0.499)** (0.529)* (0.424)** Non-English First Language (0.628) (0.616) (0.516) Average Best 6 High School Grade (0.056)** (0.055)** (0.047)** OSAP Applicant (0.506) (0.519) (0.409) Years in Canadian School System (0.106) (0.084) (0.076)** Age at time of Course (year taking course - year of birth +1) (0.193) (0.201) (0.186) Canadian Citizen (1.038) (0.970) (0.839) OUAC 105 Students (0.291)* (0.303) (0.261)

25 Table 5B Winter 2010 Mandatory Online Homework and Winter 2011 No Online Homework Midterms and Final Exam Regressions VARIABLES Test 1 Grade, Percentage on Multiple Choice Test 2 Grade, Percentage on Multiple Choice Final Exam Grade, Percentage on Multiple Choice Census Measures (DA Level) % Pop. Visible Minority (0.011) (0.012) (0.009) % Pop. 1 Parent Family (0.030) (0.030) (0.025) % Pop. With High School Diploma (0.049) (0.050) (0.042) % Pop. With Trade Certificate (0.077) (0.083) (0.064) % Pop. With College Certificate or Diploma (0.049) (0.054) (0.043) % Pop with University Certificate of Diploma (0.083) (0.085) (0.071) % Pop with BA or Higher (0.033) (0.034) (0.027)** Bottom Third DA Income Tercile (0.773) (0.835) (0.641)* Middle Third DA Income Tercile (0.641)* (0.644) (0.516) % Unemployed age (0.072) (0.078) (0.064) Total Population in Thousands (0.113) (0.101) (0.094) % Population age (0.059) (0.066) (0.049) Constant (28.670)** (30.031) (25.505) Observations 2,577 2,540 2,602 R-squared Clustered standard errors in parentheses ** p<0.01, * p<0.05

26 Figure 3 Percentage on Test Average Test Scores Based on Online Participation Term Test 1 Term Test 2 Final Exam Winter 2010 Mandatory Online Homework Winter 2012 Optional Online Homework

27 Figure 4A Distribution of Midterm Grades by Term Fraction of Students Term Test One Fraction of Students Term Test Two Distribution of Grades Distribution of Grades Winter 2010 Winter 2012 Figure 4B Fraction of Students Distribution of Final Exam Grades by Term Distribution of Grades Winter 2010 Winter 2012

28 Table 6A Winter 2010 Mandatory Online Homework and Winter 2012 Optional Online Homework Midterm Regressions VARIABLES (1) (2) (3) Test 1 Grade Percentage on Multiple Choice Test 2 Grade Percentage on Multiple Choice Final Exam Grade, Percentage on Multiple Choice Online Homework Optional (0.429)** (0.473) (0.394)** Transcript Measures Business (0.813)** (0.943)* (0.758)** Engineering (0.809)** (0.938) (0.769)** Science/ Health (0.863)** (1.015)** (0.861)** Humanities (1.467) (1.909) (1.379) OUAC Measures Upper Year Student (0.618)** (0.679)** (0.610)** Distance from University (0.003) (0.003)** (0.002)** Female (0.468)** (0.506) (0.441)** Non-English First Language (0.562)* (0.638)** (0.528)* Average Best 6 High School Grade (0.047)** (0.053)** (0.046)** OSAP Applicant (0.452) (0.515) (0.412) Years in Canadian School System (0.100) (0.110) (0.094) Age at time of Course (year taking course - year of birth +1) (0.215) (0.231) (0.209) Canadian Citizen (0.963) (1.151) (0.934) OUAC 105 Students (0.306) (0.352) (0.297)

29 Table 6B Winter 2010 Mandatory Online Homework and Winter 2012 Optional Online Homework Midterm Regressions Census Measures (DA Level) % Pop. Visible Minority (0.010) (0.011)** (0.009) % Pop. 1 Parent Family (0.029) (0.030) (0.025) % Pop. With High School Diploma (0.048) (0.050) (0.042) % Pop. With Trade Certificate (0.070) (0.075) (0.061) % Pop. With College Certificate or Diploma (0.046) (0.051) (0.040) % Pop with University Certificate of Diploma (0.075) (0.083) (0.070) % Pop with BA or Higher (0.031) (0.034) (0.027) Bottom Third DA Income Tercile (0.802) (0.845)* (0.651)* Middle Third DA Income Tercile (0.589) (0.641) (0.516) % Unemployed age (0.069) (0.082) (0.064) Total Population in Thousands (0.110) (0.121) (0.099) % Population age (0.063) (0.067) (0.049) Constant (30.045)* (34.456) (28.905) Observations 2,640 2,523 2,649 R-squared Clustered standard errors in parentheses ** p<0.01, * p<0.05

30 The Effectiveness of Tutorials in Large Classes: Do they matter? Is there a difference between traditional and collaborative learning tutorials? Karen Menard** Bridget O Shaughnessy* A. Abigail Payne* With Olesya Kotlyachkov* and Bradley Minaker* June 2014 *Department of Economics and Public Economics Data Analysis Laboratory (PEDAL), McMaster University **Ontario Institute for Cancer Research

31 I. Introduction Across much of Ontario and Canada, a typical first-year university student experience involves the enrollment in a large class. While there are many benefits for universities to use large classes from a financial and resource perspective, the impact on students, particularly weaker students, tends to be overlooked. Struggling students may not seek help and/or may disengage from their studies. Ultimately this may lead to bad decisions around the program of study, academic choices made in later years and, in many instances, may lead to dropping out. As illustrated in Dooley, Payne, and Robb (2011), high school grades are a strong indicator of success in university and first-year students are more likely to leave university than students who make it into their second year of studies (see also Finnie and Qiu, 2008). In this report we examine the use of tutorials in a large enrollment course and analyze the benefits of tutorials in such courses. It has been demonstrated that students who develop a connection with faculty tend to perform better than students who do not develop such a connection. Studies have shown that it is difficult to develop a student/faculty connection in classes of 400 to 600 students (see Nagda, 1998, Chickering and Gamson, 1987, and Finnie and Qiu, 2008). If students are engaged in their first year, this may have a positive impact on their university progression. However, given the reality of resource constraints on universities and the new practice of large classes, how can students be appropriately engaged in the context of a large class? Admittedly, poor performance in any one class is unlikely to cause a student to leave post-secondary education (PSE), and good performance in any one class is equally unlikely to entice a student to stay in PSE. It seems reasonable to believe, however, that helping a student improve performance in one class could mitigate other factors that might cause the student to leave university. If an approach were successful in improving at-risk student performance in one

32 course, it could possibly be replicated in many first year courses and have an impact on persistence. If tutorials in large enrollment courses are successful at engaging students, does it matter if the tutorial is conducted as a traditional tutorial (i.e., with a teaching assistant working through a problem with the students) or as a collaborative learning tutorial (i.e., groups of students working through a problem together and the teaching assistant providing assistance to the students). Collaborative learning has been extensively studied in science and engineering programs. Felder (1995) and Felder, Felder, and Dietz (1998) found that collaborative learning and active learning improved student outcomes and student satisfaction in a sequence of large (90 to 123 students) chemical engineering courses. The instructional techniques used in the Felder studies, however, did not seem to help the weakest students in the class. A meta-analysis of 39 studies in science, mathematics, engineering, and technology courses showed a positive and statistically significant impact on student achievement, motivation, and attitudes when cooperative learning or collaborative learning methods were used (see, Springer, Stanne, and Donovan, 1999). Two studies have examined collaborative learning in the social sciences but for relatively smaller classes than what is being proposed. Yamarik (2007) studied class sizes of students and focused on students in second/third year of university. He found collaborative learning classes to be more effective in leading toward student success than traditional classes. Huynh, Jacho-Chaves, and Self (2010 and 2011) studied classes of 200 students in first-year studies. They found benefits to collaborative learning but lacked an experimental design that allowed them to compare the benefits to other methods such as traditional tutorials. A key

33 finding of Huynh, Jacho-Chaves and Self (2011) is that collaborative learning had a particularly strong positive impact on students falling in the bottom 40 th percentile. In this report we analyze an intervention conducted during the 2012/2013 academic school year for a large class in economics. This course typically enrolls over 2,500 students each year across five sections. The students in this course represent all faculties across the campus as the course is a pre-requisite for many programs. The backgrounds of students and their level of preparation for the course vary. Previous attempts at low-cost interventions to improve student performance in this course were ineffective. For example, in 2009/2010, a random set of students who performed poorly on the first test were personally ed by the instructor and provided information on academic resources. Final course performance by these students was no different than students who were not ed but performed similarly on the first test. The innovation in our study is to examine the effectiveness of collaborative learning in larger classes (approximately students per class) and to compare and contrast the effectiveness of collaborative learning versus traditional tutorials. In previous studies with smaller class sizes, most of the students participated in the tutorials. With very large classes, students may feel anonymous and attendance at tutorials may be lower than in a small class setting. As many departments face increasingly tight budgets, there is a trend towards eliminating tutorials, partly due to a perceived lack of student participation. Our study will begin to establish participation rates and identify which type of tutorial methods may be more beneficial in engaging students in large enrollment courses. Overall we find a high proportion of students participate in the first tutorial. We also find that close to 70% of the students attend at least 3 tutorials but that less than half the students attend all five tutorials. We find a measurable impact of tutorial attendance on course exams and

34 on the final grade. Students who participated in all five tutorials performed better than those who only attended three tutorials. The traditional tutorials have a stronger (positive) effect on course performance. There is, however, a stronger positive correlation between the collaborative learning tutorials and performance on optional online homework assignments. The following section of the report outlines the course and experimental design. In section III we discuss the data, the selection of students studied and present summary statistics. Finally in sections IV and V we present the analysis and the discussion, respectively. II. Experiment Design Our experiment focused on instruction of introductory macroeconomics at an Ontario university. This course was taught by a single instructor and followed a similar structure for seven years. Each year, five sections of the course were offered: two in the fall (September to December) and three in the winter (January to April). The enrollment in each section ranged between 400 and 600 students, with a total enrollment of approximately 2,400 students each academic year. At the university under study, students taking introductory macroeconomics, however, are registered in all of the faculties as there are several programs across the various faculties for which this course is a requirement (e.g., engineering, commerce). Thus the students taking this course are diverse in their backgrounds, particularly in their academic preparation. This course, as is typical with economics courses, requires strong math and analytical skills. Prior to the year in which tutorials were introduced, students were evaluated based on their performance on two term tests and a final exam. The instructor offered optional online homework. A student who completes the online homework assignments can use her performance on the homework to reduce the weight allocated to her final exam. For a student

35 who chooses not to complete the online homework assignments, the final grade was allocated as follows: 25% of the marks received on the better of two term tests (typically test #1) 20% of the marks received on the other term test 55% of the marks received on the final exam If a student completed the online homework assignment then the weight of the final exam was reduced to 40%. The 15% for the online homework was allocated as 5% of the marks received on a basic math test offered in the first two weeks of the course and 10% of the marks received on weekly homework assignments. The math component of the online homework is designed to review concepts learned in high school. To ensure tutorial attendance, the instructor offered an incentive. The tutorials were not for grades, but students were offered a grade weight shift as an incentive to attend. The grade weight shift allowed students to shift a small portion of the weight from the final exam to their higher term test grade. Students who performed better on the final exam than both tests would not be penalized because their shift would work in the opposite direction. Historically, the course average was higher on Test #1 than on Test #2, and both term test averages were typically higher than the average on the final exam. Usually only five percent of the class performed better on the final exam than both term tests. The percentage shift was as follows: Attend all five tutorials: Attend four tutorials: Attend three tutorials: Shift 5% of final exam weight to highest term test grade. This results in the better term test being worth 30%. Shift 4% of final exam weight to highest term test grade. This results in the better term test being worth 29%. Shift 3% of final exam weight to highest term test grade. This results in the better term test being worth 28%.

36 Attend zero to two tutorials: No weight shift, grade is calculated according to scheme described above. Tutorials were provided for the students in Introductory Macroeconomics during the 2012/2013 academic year for the first time in two decades. Tutorials were held bi-weekly. There were five tutorials held each semester, beginning in Week 4 in the fall and Week 2 in the winter. The schedule for the tutorials and tests and the subjects covered (chapters of a text book) were as follows: Fall, 2012 Winter, 2013 Week Date Tutorial Test Week Date Tutorial Test 2 2 Jan. 17 Jan. 18 Chapter Sept Jan. 23 Sept Sept. 24 Chapter Jan. 30 Chapter Oct. 1 5 Feb. 6 Test 1 (1-6) 6 Oct. 9 Chapter 4-6 Test 1 (1-6) 6 Feb. 13 Chapter 7,8 7 Oct Feb Oct. 22 Chapter Feb. 27 Chapter Oct Mar. 6 Test 2 (7-11) 10 Nov. 5 Chapter 10,11 Test 2 (7-11) 10 Mar. 13 Chapter 12,13 11 Nov Nov. 20 Chapter 12 Mar ,13 13 Nov Apr Dec Apr. 10 Final Exam Final Exam