The Accuracy and Validity of Online Homework Systems by

Size: px
Start display at page:

Download "The Accuracy and Validity of Online Homework Systems by"


1 The Accuracy and Validity of Online Homework Systems by Kathleen Malevich MS Candidate: Program in Applied and Computational Mathematics Advisor: Carmen Latterell Department of Mathematics and Statistics University of Minnesota Duluth

2 Table of Contents Abstract Introduction Purpose Methods Literature Review Statistical Evidence Pros versus Cons Cheating Student Performance Time and Money Saver Mathematically Speaking Grading WebAssign and Postprocessing ALEKS and Knowledge Space Theory Numerical versus Open-ended Answers Limitations with the Research Future Research Conclusion...24 References...26 Appendix A

3 Abstract This study investigated the validity and accuracy of using online homework systems in college mathematics courses. By examining various research articles that were written about the use of web-based homework in mathematics courses compared to paper-based homework, conclusions were reached on whether using one or the other made a significant impact on students learning mathematics. In addition, the grading methods of two online homework systems, ALEKS and WebAssign, were specifically examined. Select mathematics problems from lower-level, mid-level, and upper-level undergraduate as well as graduate-level courses were then considered to determine if they could be successfully graded using online homework systems. 1. Introduction In the last twenty years, many colleges and universities have opted to implement online homework programs for students to complete assignments rather than the traditional paper and pencil method. These programs are rapidly gaining in popularity around the country, partly due to the fact that many book publishers have designed some of the programs. But some of the primary reasons for this expansion are to decrease costs of having to hire homework graders; to ease the burden of correcting on teaching assistants and instructors; to provide immediate feedback to students on their homework; and to eliminate cheating amongst students. While there is research supporting and opposing these pros, there is still much to be considered and researched when it comes to online homework systems because they are still relatively new and they are constantly expanding and adapting. WebAssign, WeBWorK, ALEKS, MyMathLab, and ARIS are just some of the numerous online homework systems available for instructors and professors to use in their classrooms. ARIS is a prime example of a system developed by a book publisher, McGraw-Hill. Pearson Education is the company behind the development of MyMathLab, while WebAssign works closely with many different book publishers. ALEKS also has numerous textbooks available for use. WeBWorK, developed in 1994 by mathematicians, is unique in the sense that it is currently maintained by mathematicians and not by a textbook publisher. WeBWorK is promoted by the Mathematical Association of America (MAA) and can be used with higher-level mathematics classes. According to their website, WeBWorK is currently being used in over 240 colleges, universities, and high schools, while the WebAssign webpage states that their system is used in 1500 institutions worldwide. In addition, the website for ALEKS claims that their system is being used by thousands of K-12 schools and higher education institutions throughout the world. WebAssign can be used for the following disciplines: accounting, astronomy, biology, chemistry, engineering, geoscience, mathematics, physical science, physics, social studies, and statistics (See Appendix A). But, WebAssign is primarily used across the country in mathematics, physics, and chemistry courses. WeBWorK is primarily used for mathematics courses like college algebra, discrete mathematics, probability and statistics, single and multivariable calculus, differential equations, linear algebra, and complex analysis. ALEKS is available to be used for K-12 mathematics as well as college courses in mathematics, statistics, business, behavioral science, chemistry, and physics (See Appendix A). MyMathLab is used 3

4 mainly for different mathematics courses and its affiliate MyStatLab can be used in conjunction with statistics courses. Instructors and students usually enjoy the benefits of having online homework systems to use for their courses. But what are some of the drawbacks and shortcomings of such programs, if there are any? Do such online homework systems benefit student learning? Are such systems able to be an extension of what the instructors are teaching in the classroom? These are all questions that are still being researched by various instructors across the country and they will be addressed in my research. 2. Purpose The purpose of my research was initially to determine the validity and accuracy of how these online homework systems determine whether or not the answers are correct, but after trying to research the various grading rubrics these systems use, I was not able to find the most useful information. Although I will still discuss the grading rubrics to some extent, it will not be in the full depth I would have hoped. Thus, I decided to focus primarily on how well the online homework systems align with the type of mathematics most professors and instructors would like their students to learn and if, in fact, they help or hinder student performance. After working with WebAssign during my time at the University of Connecticut as a teaching assistant, I saw the conflicting benefits and problems with the use of online homework and became motivated to research what others are saying about using online homework systems. Although using online homework systems can be a positive experience for students and instructors, I think it is important for instructors to understand exactly how the systems work so they can use them for the benefit of the student and not make it a hindrance for their learning. 3. Methods Last year working as a graduate teaching assistant at the University of Connecticut (UCONN), I spent a considerable amount of time working with WebAssign. Therefore, I will use many of my past experiences as well as those of fellow teaching assistants as the primary source of my inspiration for researching what other instructors are saying about WebAssign, specifically. The University of Minnesota Duluth currently uses ALEKS in some of their courses and with positive reviews from various instructors I have spoken to, this compelled me to further research this online homework system. The last system I will focus on will be the WeBWorK system, since the MAA and the National Science Foundation (NSF) support it. I will spend the majority of my time discovering what other researchers have determined on the topic of using online homework systems in entry-level college courses and then use their reports to arrive at my own conclusions. I will also take the time to discuss the types of mathematical problems that seem appropriate and capable of being asked using an online homework system. This was determined by considering the possible answers to various lower-level, mid-level, and upper-level undergraduate as well as graduate-level course questions. 4

5 4. Literature Review The effect of online homework systems on student learning was the usual reason for researchers to conduct studies on some of the various systems. The papers I focused on were mainly about the use of WebAssign, WeBWorK, ALEKS, systems designed specifically by the school, and about the use of online homework in general. For example, Bonham, Beichner, and Deardorff (2001) conducted a study by having one large lecture section, approximately 110 students, of an introductory calculus-based physics course use WebAssign for homework while another large lecture section for the same course used the traditional paper-and-pencil method. The study was repeated the next semester using two smaller sections, approximately 60 students, of an algebra-based course. For both semesters, an instructor taught the two sections back-toback and tried to keep everything in the lectures as similar as possible for each section. The only differing factor was how the students were to complete their homework for the course. Homework, quiz, and exam scores were complied and compared between the two different sections. However, they did not account for varying student skills in the different sections and later claimed that the small difference seen in test scores is possibly due to a difference in student ability since the students in the WebAssign section had somewhat higher GPA and SAT mathematics scores. Although they did find that the paper-based section was much better at following directions to circle their final answers when taking tests and quizzes. Basically, it was decided, the method of collecting and grading homework makes little difference to student performance and concluded that technology itself does not improve or harm student learning, but rather the underlying pedagogy is the critical issue (Bonham et al., 2001). The research conducted by Axtell and Curran (2011) questioned the validity and effectiveness of using online homework since some researchers found benefits for students and instructors while others have not found any advantages. For their study, they set up their experiment similar to that of Bonham et al. (2001). Axtell and Curran (2011) also wanted to determine the effectiveness of web-based homework systems compared to traditional paper and pencil by studying the performance of Finite Mathematics students in small sections of approximately 24 pupils. One section was to complete their homework online while the other section turned in paper assignments to their instructor. Both sections had the exact same problems on their assignments, the same instructor, and common exams keeping all things similar with the exception of how homework was to be completed (Axtell & Curran, 2011). In research performed by Hirsch and Weibel (2003), they found no significant results on student performance using a paper-based homework section versus a WeBWorK section. They claimed that the effectiveness of WeBWorK depended dramatically upon how many of the problems were attempted by the students. Although I would venture to say that the effectiveness of any homework, no matter what medium it is being delivered by, is entirely dependent on the amount of effort the students put into completing and mastering their homework. Hirsch and Weibel (2003) arrived at their conclusions by examining how the use of WeBWorK at Rutgers University affected student performance in the general calculus class. WeBWorK was only put into practice in two-thirds of the sections of calculus, so they used the data from the non- WeBWorK sections as the control group in the experiment. At the end of the semester, they looked at the SAT, placement, exam, and other scores for the students to make their comparisons. This experiment was slightly different than the ones conducted by Bonham et al. (2001) and Axtell and Curran (2011) since all sections were required to submit paper and pencil 5

6 homework assignments, but in the study group, approximately 11 problems per week were replaced by online problems using WeBWorK (Hirsch & Weibel, 2003). Similar to research by Hirsch and Weibel (2003), studies conducted by Hauk, Powers, Safer, and Segalla (2004) indicated that they also found no large statistical differences in the performances of students in web-based homework sections versus paper sections. For their study, Hauk et al. (2004) had 12 of 19 college algebra classes used WeBWorK for submitting homework while the remaining 7 classes used the traditional paper and pencil format. Students were evaluated by a pre- and post-test of their algebra skills as well as their overall performance in the course to determine the effectiveness of using online homework. While they found not large differences, Hauk et al. (2004) did discuss the importance of assigning homework regardless of how it is being done. In studies done by Cooper (1989), Cooper, Lindsay, Nye, and Greathouse (1998), Keith and Cool (1992), and Warton (2001), it was found that homework is important for the advanced cognitive development expected in high school and college mathematics (as cited in Hauk et al., 2004). Keith and Benson claim that the key factors in learning are ability, motivation towards mastery, quality instruction, and amount of academic instructional time including time spend on homework (as cited in Hauk et al., 2004). Thus, one can conclude that although homework is considered necessary for student achievement, it is not automatically sufficient (Hauk et al., 2004). Demirci (2006) also supported the idea that having homework versus not having homework plays a significant role in student performance. Because of this reasoning, the studies I read did not question whether or not homework should be given, but rather, in paper or online form. For his experiment, Demirci (2006) also used two identical sections of an introductory physics course that only differed in the form in which the sections submitted their homework. Conclusions by Demirci (2006) are similar to others in that he claims, web-based homework is a viable alternative to the traditional paper-based approach. It does not bring significantly greater benefits to students, but neither does it do much worse than standard methods of collecting and grading homework technology itself does not improve or harm student learning, but somewhat the underlying pedagogy is the vital issue. 4.1 Statistical Evidence In many of the research articles, the authors provided various statistics in support of their findings. Most of the researchers provided results from regression analysis, analysis of variance, and other t-tests they conducted on the scores of student homework and exams in the different sections of online versus paper homework. However, they did not include enough information to be able to replicate the exact procedures that were performed. Therefore, it was hard to say what types of comparisons were actually being made without knowing what the null hypotheses and alternative hypotheses were for each of the experiments. Nevertheless, I will still discuss the statistical results from the research articles. For example, in the paper by Bonham et al. (2001), the results from the sections were complied and compared using a two-tailed t-test, shown in Figure 1 taken from the article. In the calculus course, there was a statistically significant difference in the homework scores of the two different sections as well as the time spent on homework per week. The only statistically significant difference between the two algebra sections was the amount of time the students spent on homework each week. These differences could be attributed to a couple of factors. Bonham et al. (2001) concluded that the large discrepancy in homework scores for the calculus section could have been due to the multiple 6

7 submissions difference or because the students had three short assignments each week instead of one longer one like the paper-based sections. Since a large difference in homework scores is not reflected in the algebra sections, it was assumed that this could have been caused by both sections having one long assignment each week or by the considerable amount of technical difficulties encountered early on in the course, which may have discouraged students (Bonham et al., 2001). The second major difference in sections was the amount of time spent on homework by students. Bonham et al. (2001) did not provide an explanation as to why there is such a huge difference in time spent, but I believe it is due to Figure 1. Table from Bonham et al. (2001) article. the fact that students completing online homework have the opportunity to submit answers to their problems multiple times. If a student gets an answer wrong, they are told immediately so they can fix their answer. On the other hand, students completing assignment on paper do not have the luxury of immediate feedback so they will not spend as much time on each problem as students doing the problems online. Because of these explanations for the statistically significant differences, Bonham et al. (2001) were able to conclude that the method of collecting and grading homework makes little difference to student performance. In the article by Hirsch and Weibel (2003), they found that the students using WeBWorK had a small yet statistically significant improvement of 4% on the final exam in calculus. Even after making adjustments based on student placement exam scores, there was still evidence of improvement. As mentioned earlier, Hirsch and Weibel (2003) concluded, the effectiveness of WeBWorK depended dramatically upon how many of the problems were attempted. This was based on the fact that the students who did not attempt as many web assignments as other students did caused the improvement in the final exam performance to decrease. They found that the correlation between attempts and percentage of problems solved was a remarkable.944 which they thought suggested that students persevered to complete a problem once they had attempted it. Hirsch and Weibel (2003) conducted an analysis of variance which resulted in only 9% of the variability in WeBWorK scores could be attributed to prior skill level which caused them to arrive at the conclusion that the number of problems attempted by students probably has more to do with student effort than ability. The article went on to describe large differences in letter grades, failing to passing, amongst first-year, upper-class, and repeating students based on the amount of WeBWorK problems attempted. Not surprisingly, the more problems attempted, the higher the passing grade while the fewer problems attempted, the lower the grade in the course. 7

8 Hauk et al. (2004) also used a paired t-test analysis similar to Bonham et al. (2001) to compare the results of the pre- and post-test for paper-based homework sections to web-based homework sections. Even though the web-based homework section did slightly better on the post-test than the paper-based sections, the difference was not statistically significant at the 0.05 level. Demirci (2006) also conducted t-tests to determine whether or not there was any difference in performance on homework and tests between paper and online homework sections. Although there was no statistically significant difference between test scores of the two sections as the 0.05 level, there was a slight difference in favor of the paper-based homework section for the homework scores. Again, this could be contributed to a number of different reasons including student ability as well as effort, so the difference is not enough to claim that one method or the other has a larger impact on student performance. 5. Pros versus Cons As stated earlier, I found that many researchers were approaching the topic with the idea that it has to be web-based versus paper and pencil homework for students. Initially, I also thought this was to be an approach I wanted to further research. But, it seems to me that with the prevalent use of technology in our society today, online homework systems will not be going away any time soon, but rather they will continue to expand, develop, and improve. So I ask, why does it have to be one versus the other? Would not time be better spent trying to determine how to make either system the best it can possibly be for learning mathematics? Both ways of having students complete course work have their shortcomings, but instead of trying to decide which method is the lesser of the two evils, I think it would be best to establish how they can be improved upon to satisfy most, if not all, expectations of what homework should accomplish for students. As Titus (2000) claims, When evaluating the pros and cons of a tool, it s important to realize it is a tool! A tool can be used effectively or ineffectively. Therefore, I will discuss the pros and cons associated with online homework systems and suggest how to address the drawbacks to make improvements. The fact that researchers have conflicting opinions on whether or not a certain topic is indeed an advantage for students as opposed to a disadvantage is quite interesting. Quite often, researchers are able to make their findings align with their own preconceived notions and biases. So rather than blindly accepting their claims as fact, one must ask questions to determine the context of the research and what variables, usually related to the students and not being able to control their behavior, were not accounted for in the beginning of the experiment. Some of the pros stated in the research in favor of using online homework systems: P1. They are time savers for instructors who no longer have to collect, grade, and return papers (Bonham et al., 2001; Hauk & Segalla, 2005 as cited in Axtell & Curran, 2011; Titus, 2000; Demirci, 2006; Burch & Kuo, 2010; Pascarella, 2004; Doorn, Janssen, & O Brien, 2010). P2. Immediate feedback on whether or not a student s answer is correct is a benefit of online homework systems compared to waiting a few days or more with the traditional paper-and-pencil option (Bonham et al., 2001; Baron, 2010; Titus, 2000; Demirci, 2006; Pascarella, 2004; Carpenter & Camp, 2008; Doorn et al., 2010). 8

9 P3. Students can practice and engage in the material more which may lead to increased knowledge and skills (Bonham et al., 2001; Axtell & Curran, 2011; Titus, 2000; Demirci, 2006). P4. Online homework systems generate similar problems but with different randomized variables and parameters, eliminating cheating amongst classmates (Bonham et al., 2001; Pascarella, 2004; Carpenter & Camp, 2008; Doorn et al., 2010). P5. A nice feature with many of the online systems is having links to online tutorials or examples in the book that are similar to the given problem so students can access this information when they have trouble solving the problem (Axtell & Curran, 2011; Burch & Kuo, 2010; Pascarella, 2004). P6. They save money for colleges since colleges are no longer required to hire graders to correct student assignments (Axtell & Curran, 2011; Hauk et al., 2004). P7. Students are allowed multiple attempts and submissions for each homework problem, which can help improve their performance in class (Denny & Yackel, 2005 and Zerr, 2007 as cited in Axtell & Curran, 2011; Burch & Kuo, 2010; Pascarella, 2004; Carpenter & Camp, 2008; Doorn et al., 2010). P8. The ability to get feedback for each problem on an assignment, as opposed to only getting feedback on select problems on a larger paper-and-pencil assignment, helps students to know exactly what problems they understand (Zerr, 2007 as cited in Axtell & Curran, 2011; Burch & Kuo, 2010; Carpenter & Camp, 2008). P9. Instructors can interact more usefully with students since they can tell what a student was thinking just by viewing the answer they submitted most recently and what the problem might have been, i.e. conceptual, syntax, or a technical error (Baron, 2010; Carpenter & Camp, 2008). P10. Due to the online nature of the homework, multimedia-focused questions can be presented to students that could not be used in the traditional paper format of homework (Titus, 2000; Demirci, 2006). P11. It is convenient and provides students with flexibility (Pascarella, 2004; Doorn et al., 2010). Some of the cons found in the research against using online homework systems: C1. They usually do not give any reasoning as to why an answer is wrong (Bonham et al., 2001; Titus, 2000; Pascarella, 2004; Carpenter & Camp, 2008). C2. Online homework systems tend to put emphasis on the final answer rather than the process students use to obtain their solution (Bonham et al., 2001; Axtell & Curran, 2011; Pascarella, 2004). Also, instructors cannot see the work students have done on each problem resulting in not being able to give partial credit for work (Axtell & Curran, 2011; Demirci, 2006) or not being able to fully recognize a student s conceptual understanding of the problem (Hauk et al., 2004). C3. Having the ability to submit answers multiple times until students get the problem right can lead the students to adopt a trial-and-error strategy rather than persisting and actually trying to solve the problem by carefully thinking about the problem (Bonham et al., 2001; Axtell & Curran, 2011; Burch & Kuo, 2010; Pascarella, 2004; Carpenter & Camp, 2008). In addition, the level of students awareness of their understanding of the material may be false due to the fact that multiple submissions result in high homework scores. This gives the students a false sense of 9

10 security about their abilities and will result in poor quiz and test scores (Axtell & Curran, 2011). C4. If students are not keeping track of their written work to solve the homework problems, then they will be unable to use their homework as a tool to help them study (Axtell & Curran, 2011). C5. Technical errors can cause difficulties for both students and instructors, i.e. syntax errors, server crashes, programming errors, etc. (Titus, 2000; Demirci, 2006; Carpenter & Camp, 2008; Doorn et al., 2010). C6. There is a learning curve for both instructors and students with online homework systems (Titus, 2000; Carpenter & Camp, 2008; Doorn et al., 2010). C7. Certain types of problems can be challenging to program into the systems, resulting in a less diverse question bank that uses fewer grading methods (Demirci, 2006; Carpenter & Camp, 2008). C8. The issue of cheating amongst classmates can be a problem since there is no way to determine who is actually completing the online homework assignment (Demirci, 2006; Burch & Kuo, 2010). C9. For some of the online homework systems, there can be an extra cost for students since they have to purchase access codes (Doorn et al., 2006). Even though these lists of pros and cons seem quite extensive, they contain only some of the many advantages and disadvantages of using online homework systems. It is interesting though to note some of the conflicting opinions. Although P1 claims the online homework systems save time for instructors since they no longer have to collect and grade assignments, C5 would counter that the time might now be spent dealing with technical difficulties that students encounter. From my own experience, I spent at least 5 hours a week as a teaching assistant answering s about the homework and many times, the problems students were having had to do with syntax errors and other technical glitches in the system. Now P2 says it is beneficial for students to receive immediate feedback on wrong answers, but C1 states that students usually are not told why their answers are wrong. Some might argue that P5 (online tutorials) and P7 (multiple submissions) are remedies to this, but C3 claims the multiple submissions and not knowing why their answers are wrong could lead to just trying answers until something works without taking the time to really figure out what they need to do to solve the problem. Another conflict in opinions is that even though the online homework systems save money for colleges (P6), they might actually cost students more money (C9). Interestingly enough, P10 says more types of problems that include multimedia can be used in homework while C7 says that it can be difficult to incorporate certain types of problems due to the difficulty in programming such problems. Finally, some claim that having randomized variables reduces cheating (P4) while others find that using online homework can make cheating worse since you are not able to determine who actually completed the work (C8). Clearly, there are many differing viewpoints on the topic of advantages and disadvantages of online homework use by students. Often times, these opinions are found to contradict each other. This is just a prime example of researchers being able to meet their own preconceived notions about the usefulness, or lack there of, in using online homework systems in college courses. Therefore, I would like to further analyze some of the more common issues of cheating, student performance, saving time and money, and the mathematical concerns associated with online homework systems in the following sections. 10

11 5.1 Cheating Cheating is, and always will be, a problem for instructors no matter which type of homework, online or paper, is given to students. What I found to be interesting in my research though was that the researchers were so intent on trying to prevent cheating amongst classmates, that they forgot to even mention how much easier it is to use other sources for cheating. That is, with the exception of Carpenter and Camp (2008) who mentioned, in recent years the ready availability of complete textbook solutions manuals via online retailers has further watered down the effectiveness of textbook-based homework assignments. But even with many of the online homework systems, the question banks are based off of textbook questions so using some of these systems would still not deter students from using solutions manuals to get answers. Even with the different numbers for questions, students are usually bright enough to figure out how to substitute their numbers into a solved problem to get an answer for their own problem. This is also how students can usually still cheat off of their classmates using online homework systems with randomized variables and numbers. Cheating using online homework systems is usually easier to detect than cheating on paper, but in my opinion, it is also much easier to cheat using the computer than paper. For most mathematics assignments on paper, students are required to write out all of the steps and show all of their work for each problem. So when a classmate copies someone else s work, they must copy the whole process to get to the final answer. It can be argued that during the process of copying problems by hand that a student might still retain some of the information from the assignment. But when completing homework online, all students need to do is figure out and then enter the correct answers to the problems. Therefore, one can argue that students do not have to think about the process required to solve the problems as much as they would if they were copying it by hand, although it is true you can write without really having to think much about what you are writing. As far as cheating goes online, I found two different articles about how to determine if a student is most likely cheating. Lynch (2010) and Hirsch (2010) both discussed separately how instructors could look at the elapsed time from when a student first starts a problem to when they finish it. If the student has correctly answered a difficult problem on the first try and in a shorter amount of time than most students need to actually read the problem and subsequently solve it, then it is probable that they had gotten the answer beforehand from a classmate. It is quite possible though that the students are able to get the answers although not from a fellow classmate. During my time at UCONN, we had quite a problem with cheating in the Calculus classes which all used WebAssign. It was obvious to us that students were getting answers elsewhere and not even from classmates. Some of the answers they were arriving at, though correct, could not have been a result from the processes they were being taught in class or what was in the book. These answers were so obscure that it was not even likely they were from more advanced graphing calculators that can solve integrals. What we discovered is that students were using the website WolframAlpha, which is the same company that sells the mathematics software program Mathematica. But the great thing about WolframAlpha is that any student can use it because it is a free online website! Now they even have mobile applications for the website that students can download onto their smartphones or ipads. But this website is capable of calculating all sorts of mathematical problems, even hard integrals, and it sometimes even shows all of the steps taken to arrive at the final answer. However, it sometimes does not use the methods taught in class which is why when many students were arriving at a correct answer that had the 11

12 hyperbolic sine function as part of their answer to an integral and no hyperbolic functions were ever mentioned in the whole Calculus II course, we knew there was a problem. Sure enough, the first answer on the WolframAlpha website was the one containing the hyperbolic sine function. Of course students are only hurting themselves by cheating, but with the relative ease in which they can now do so, it makes it much more tempting. It is so easy to open a browser to access the homework assignment and then open up another one to use a website like WolframAlpha. All the student has to do is read the problem, type it into the website, and get the answer to put into their assignment. This can all be done without the help of other classmates, and instructors should not overlook this type of cheating. 5.2 Student Performance The majority of the studies I read found little or no statistical differences between student performances in web-based homework sections compared to paper-based homework sections. As Hauk et al. (2004) stated, using online homework instead of paper and pencil does not appear to harm student performance. Thus, based on all these different researchers findings, one can supposedly conclude that the method in which homework is to be turned into instructors does not have a significant effect on student learning and achievement. Personally, I find little value in directly trying to compare two different groups of students using the different approaches to homework. It is far too difficult to account for varying levels of student ability and motivation in a given course. These two factors alone have much more of an impact on student performance in a class than whether or not the homework is completed online or on paper. One research article even suggested that it would be ideal, but not practical, to use the same group of students and switch between methods halfway through the semester (Burch & Kuo, 2010). Due to the fact that most research has come to the same conclusion of little or no difference in using online homework as opposed to paper-based submissions, but there are still many disadvantages to the technologically advanced option, I would think it would be worthwhile for researchers to take the time to figure out ways to improve the current systems to better meet their expectations. For example, working with programmers for these online homework systems to come up with a way to let students know why they got an answer wrong or a way to give better hints. One of the primary negatives associated with online homework is the fact that being able to attempt a problem multiple times before submitting a final answer can lead to students not fully thinking about the problem and just using a trial-and-error approach (C3). This can have an unconstructive impact on performance and learning in the course if students are not being encouraged to be thoughtful and critical in their endeavor to solve the homework problems. Pascarella (2004) suggests weighting each submission to only receive full credit on the first try to encourage students to be more careful when working on their assignments. This approach to scoring the assignments is similar to partial credit that is usually given on paperbased homework assignments. Nevertheless, the multiple submissions aspect is not without its positives, as listed in the pros section and as Middleton and Spanias stated, such an attemptfeedback-reattempt feature also conveys to students that it is acceptable (and sometimes even expected) to make mistakes as they are learning mathematics, leading to greater confidence later (as cited in Zerr, 2007). 12

13 Overall, student performance should be very important to instructors, but changing the various forms of how assignments are to be completed will not have nearly the same impact as changing student attitudes and motivation. 5.3 Time and Money Saver Online homework systems supposedly save time (P1) and money (P6) for instructors and colleges. But they can cost more for students monetarily (C9) while also sacrificing instructor and student time when there are technical errors (C5) and also when both instructors and students must learn how to use the new system (C6). Colleges benefit the most from the exchange of going from paper-and-pencil homework to online due to the money saved by not having to hire graders, but instructors and students might have to deal with some hardships. For some online homework systems, students will have to pay extra for an access code in addition to buying their textbooks. This extra expense coupled with the possibility that students many encounter many technical difficulties with the system, might very well cause the students to be discouraged about the class in general. But students today tend to be much more technically savvy than in years past and will therefore have a much faster learning curve when it comes to learning how to use a new online homework system. Where many students will have difficulty though is following directions, which always seems to be a problem for students, and in learning any mathematical syntax needed for the program, which I will discuss further in Section 5.4. For instructors, it can almost be double duty when it comes to learning the new system s interface since many programs have different views for students and instructors. Thus, instructors must also become familiar with what students see in order to help them when problems arise. In small classes, the instructors will be the ones contacted via whenever students encounter a technical problem but in large lectures with discussions, that burden will most likely fall on the teaching assistants. As a teaching assistant at UCONN we were expected to handle most of the technical problems on our own, but there were some issues, like transferring students to different classes and sections online, that only the professors could do. Thus, extra time was spent being the middle man in these transactions. Also, at the beginning of the school year, we experienced many problems with not being able to access the WebAssign site due to server overload on their end. They had not anticipated the heavy volume of students and had to add more servers. In the mean time, we had to schedule assignments to be due at odd times in order to help prevent the server from crashing again. Having to deal with students claiming to have technical issues for not being able to finish their homework on time was interesting, especially when it would be only one student saying they could not access the site while everyone else completed the assignment. Carpenter and Camp (2008) claim that students will exhaust all other routes before contacting the instructor if they have a problem. From my own experience, I did not find this to be true, which added more time I spent trying to help with the technicalities of students trying to complete their assignments online. Carpenter and Camp (2008) also said that it is usually sufficient for the instructor to check two to three times during the evening and that would be all that is required to communicate with students on their problems. First of all, I think this depends greatly on how helpful instructors want to be for their students. Usually office hours are held during the day, but online homework usually has deadlines in the evening, which causes students to be asking for help quite often after regular school hours and far beyond typical office 13

14 hours. While this can be considered to take the place of time spent grading homework, it can still be a significant amount of time depending on how accessible instructors wish to be for their students. From my own experience, I found that if you are going to make an assignment due at ten o clock at night, even if you had office hours earlier that day, most students will be trying to finish the assignment closer to the deadline and long after you have left the office. I found myself answering s for hours before assignments were due each week. This takes a lot of time and is not very easy to do when trying to explain where a student might have gone wrong over . Most students do not follow instructions to include details on the problem they are having trouble with and for what section of the book making it very difficult to go into their homework to view their latest submission. Carpenter and Camp (2008) thought that if a faculty member were to give the students clear instructions and if they were to alert students to potential pitfalls ahead of time in class that students would have fewer problems completing their homework and would therefore contact the instructors less. I again found this to be less than true from my experience at UCONN but also as an educator in general. Students do not like to (1) read directions, (2) listen to directions, or (3) follow directions. I had to explain many times that 1.3 is not equal to which is why their answer was not being accepted as correct, but to no avail (this will be further discussed in Section 5.4). In general, I do not believe that online homework is as significant of a time saver as many researchers claim. While there are benefits to using online homework, in my own experiences I found many of the difficulties and frustrations more time consuming for instructors than the articles indicated. Essentially, I believe that paper-based homework and web-based homework can take up equal amounts of time for instructors. 5.4 Mathematically Speaking When assigning homework in mathematics courses, what is the primary objective? What is it that students are to gain from having to complete exercises? I think many mathematicians would agree that apart from the practice aspect, homework in mathematics courses is important for students to learn how to think critically; to show they are able to follow algorithms and procedures but also show they can extend that knowledge and apply it to something new; and to demonstrate various ways to solve the same problem. Mathematics is not always about the answer to the problem but rather, the process taken to reach the conclusion. Of course other mathematicians and instructors may have varying opinions from me and may have additional reasons to add or subtract from my list, but in any case, I think all instructors should carefully consider what general and mathematical knowledge they want their students to gain from having to complete homework assignments and then try to make sure it is achieving their goals. Some of the current disadvantages of online homework are that they tend to put the emphasis on the product over process (C2) and they do not necessarily encourage students to keep track of their written work to use as a study tool later (C4). Usually mathematics instructors will not take off all of the points for a problem if the student does not get the final answer right, but rather, they will check to see if the process the student was using was correct with only small errors made along the way to arrive at the incorrect answers. Instructors will then award partial credit if they can follow the student s thought process and see if they were justified in their work. Although some problems in the ALEKS system and in WeBWorK allow students to input intermediate steps leading to the final answer, many systems do not have this feature. Thus, 14

15 instructors are left to guess as to what a student tried to do only based on their answer, especially if the student is asking questions through and not in person during office hours. A memorable negative experience with online homework while at UCONN was with a particularly frustrating student. On their assignment for integration using partial fraction decomposition, the very first problem allowed every possible answer you could ever think of to be graded as correct. This particular student had found the partial fraction decomposition and entered that as his answer, which was marked as correct; he never integrated, which was the whole point of the assignment. Somehow he proceeded to complete the rest of the assignment successfully by actually integrating the rest of the problems. When it came time for their weekly quiz, the student noticed that the first problem on the quiz was similar to the first problem on the homework and proceeded to just find the partial fraction decomposition and never actually integrated. He later explained this to me before I graded the quizzes and I went into the assignment to look at other students results. Taking this into consideration, I was slightly more generous than I should have been with his score. At the end of the year he came back and was arguing with me and threatening to go to the professor to get his two quiz points back that he had lost since he was only following what the online homework system had told him was correct. I did not give in and explained that the whole chapter was on integration so he should have realized that he needed to integrate to arrive at the final answer, but he was not buying it. This was a prime example of a student putting far too much value on the product over process when solving problems. While not all students will act in this way, instructors should still be mindful of what might happen if students place too much emphasis on a final answer over the process. Since the process is not required for many of the homework problems, students might not keep a detailed record of their written work for assignments. Even if a student does keep track, he or she might not be using mathematical notation correctly. One common mistake I saw in the calculus classes is that when students were solving limit problems, most of them would not carry the lim notation all the way through. It is not mathematically correct to say that equal to just n when taking derivatives or calculating integrals in which students are careless with mathematical notation. I found that when it came to quizzes and tests, students had a hard time remembering to show all of their work and had difficulty using proper mathematical notation. I also had many students constantly asking how much work do I need to show? They understood that they needed to give some justification, but since they were used to only providing an answer online, they wanted to get by with as little writing as possible and had a hard time gauging what would be appropriate. Another mathematical issue students had from time to time was learning the appropriate syntax that went along with the online homework system. There were only certain ways students could correctly enter answers using sine, cosine, e, etc., which is an extra concept they must learn. Part of the problem was that even though there is usually documentation for the system provided somewhere, the students did not search for this information and instead contacted the teaching assistants to figure out why their answers were wrong. After I found this information and sent it to my discussion sections, there were far less problems with syntax errors. However, there were still other mathematical issues that arose. Students had a very difficult time when inputting answers that included numerous parentheses. Even though students have this same hardship when writing out an answer by hand, lim n n n +1 is without including the lim in front of it. There are many other examples 15

16 the problem with an incorrect answer online due to missing or misplaced parentheses is that students may not realize that is their only error. For example, many students do not see the (x + 4) difference in and x + 4 when they are written out in one line like (x + 4)/2 and x + 4/ And although most students can see the difference when written out on paper, it is much harder to type in a string of mathematical commands with parentheses and arrive at an equivalent expression, much like when students have a hard time entering items on a calculator. Thus, many students feel that their answer is in fact correct and will complain that the online system is incorrect for marking their answer wrong. Even though for some students this is only a technical error, for other students they have a conceptual problem understanding the difference between (sin(x) + cos(x))/(x + 7) and sin(x) + cos(x)/x + 7 and legitimately do not know they are different expressions. Therefore, online homework systems can hinder a student s understanding of which expression is correct due to the method of having to enter an answer. Another concept students struggle with is being able to understand the difference between providing an exact answer versus an approximate one. Most students rely so heavily on their calculators that they use decimals for everything. Even after telling students countless times that they should always enter answers in exact form for their online homework unless otherwise noted, I still received many s from students claiming their answer was correct even though the system marked it wrong. This was due to the fact that unless the question asked for a decimal rounded to a certain place, it wanted an exact answer. As long as the decimal was a terminating decimal, this was counted as a correct, exact answer. But as soon as the decimal was a repeating decimal, it was incorrect since most students would round their answer causing it to no longer be exact. This is where I had to explain that 1/3 is not equal to 0.3 since the decimal is not terminating. Moreover, on the exams and quizzes, they were required to provide answers in exact form, but the use of the online homework system was not supporting this requirement. However, using traditional paper-based homework normally enforces this condition since the instructor, teaching assistant, or grader is regularly looking at student homework. With the online homework system, instructors do not have to look at the work of the students unless asked, so from a timesaving aspect, most instructors probably do not look through all of the students homework assignments on a regular basis. In general, I think it is very important for the homework to reflect the caliber of work that is expected of students on quizzes and exams in the course. If students are not practicing and being reminded of how to use proper mathematical notation on a regular basis, then they will more than likely do a poor job of this on quizzes and exams. I feel that it is unfair to students in courses that use only online homework, written exams, and no quizzes for students to get penalized on the written exams because the homework does not require them to show any of their work. Mathematics is so much more than finding a right answer all of the time and if this is an opinion valued by an instructor, then it is only right to determine the best way to be using homework to help students learn and enhance their mathematical ability. 6. Grading From my experiences with using WebAssign at UCONN, I became curious to learn more about how the various online homework systems determine whether or not a student s answer is correct. This task was slightly more difficult than I had anticipated as many of the programs 16

17 students must pay for are not going to want to reveal their secrets. But I was able to learn more about the theory behind both the ALEKS and WebAssign homework systems, which I will discuss in more detail in the following sections. Since WeBWorK uses Perl for programming its problems and since it uses many macros and LaTeX commands, it is much easier for instructors familiar with these programs to design their own problems and grading schemes (Carpenter & Camp, 2008). Therefore, I will be focusing on ALEKS and WebAssign in the following sections. 6.1 WebAssign and Postprocessing According to Knipp and Chaudhury (2009), most online homework systems determine the correctness of a student s answer, x, by comparing it to the key, X, using a specified tolerance, ε. In WebAssign, the default tolerance setting is 2% of X, thus student answers are calculated by determining if X x < 0.02, which is the default ε-value. Although the instructor can adjust this ε-value, it is important for instructors to be aware of this policy for any online homework system being used by the college. I would assume that many instructors would not find an answer of to be acceptable when solving x + 4 = 1 for x, however using a 2% error tolerance causes the system to accept any values for x that fall between and Therefore, it is very important for instructors to do some research on their online homework system to see if there is a tolerance setting for answers and if so, whether or not it can be adjusted. The term postprocessing is used to describe a grading algorithm in which the student s answer is not just compared to the answer key to see if it is sufficiently close (Knipp & Chaudhury, 2009). The most common way that postprocessing is used in online homework systems is by providing hints for students (Knipp & Chaudhury, 2009). An example of this, as discussed by Knipp & Chaudhury (2009), is that if a student is asked to calculate the area of a square with side length L, then using postprocessing will compare the student submission, x, to the answer key X, the perimeter 4L, and the diagonal 2 L. If the student s answer is sufficiently close to say, the perimeter 4L, then a hint will prompt the student to fix their answer by stating, You ve calculated the perimeter, not the area. If a student s answer is not close to any of the programmed options, then the student is told that the system is unsure as to why that answer was entered. This makes the grading process more similar to that of a person actually grading the homework by making detailed notes along the way, with the exception of the student having the opportunity to fix their mistake immediately online as opposed to waiting at least a few days before getting their assignment handed back. Although postprocessing is much better than strictly comparing a student s answer to the answer key, it still cannot include all possible common errors students might make. In addition, when instructors are able to grade the problems by hand, they can also see the common mistakes that keep recurring and can then address the whole class to clarify. This most likely will not happen using online homework unless the instructors go through all the previous submissions of a student leading up to the correct answer. But, in general, postprocessing is an efficient way to try and simulate a real grader s actions and comments. 17

18 6.2 ALEKS and Knowledge Space Theory According to the website for ALEKS (Assessment and LEarning in Knowledge Spaces), this online homework system is the practical realization of Knowledge Space Theory the result of ground-breaking research in mathematical cognitive science initiated by Professor Jean- Claude Falmagne at New York University (NYU) and the University of California, Irvine (UCI) and Professor Jean-Paul Doignon at the University of Brussels. The National Science Foundation provided the professors with grants to conduct research that produced the mathematical theory behind Knowledge Space Theory between 1983 and On the website for ALEKS, it is explained that Knowledge Space Theory applied concepts from Combinatorics and Probability Theory to the modeling and empirical description of particular fields of knowledge. Within this theory, a mathematical language has been developed to delineate the ways in which particular elements of knowledge can be gathered to form distinct knowledge states of individuals. What makes this form of assessment different from standardized tests, is that instead of providing students with a numerical measurement of accomplishment it gives a description of the types of problems the student has mastered or can successfully do as well as a list of topics that the student is ready to learn. This is the whole basis for the ALEKS online homework program. Knowledge Space Theory assumes that a finite set of items, Q, is given for a specific domain of knowledge (Hockemeyer, 1997). A family of subsets of Q, call it K, is a knowledge space if an only if (1) K, contains along with Q and (2) K, is closed under unions. A knowledge state is a set that is an element of K,; that is, K such that K K,. For q Q, we get the family K,q = {K K, : q K}. The basis of K, is the smallest sub-family of a knowledge space from which the complete knowledge space can be reconstructed by closure under union (Hockemeyer, 1997). Now for the items in Q, there is a precedence relation. For example, if Q = {a, b, c, d, e, f} as in Figure 1, then if a student has mastered problem e, this implies the student was also likely able to successfully complete problems a, b, and c (Falmagne, J. C., Doignon, J. P., Cosyn, E., & Thiéry, N.). By considering the precedence relations of Q, it can be determined that there are ten knowledge states that can form K, = {, {a}, {b}, {a, b}, {a, c}, {a, b, c}, {a, b, c, d}, {a, b, c, e}, {a, b, c, d, e}, Q}. On the way to mastering Q, there are many different routes a student can take. These routes are the ways students can progress in understanding the different problems. In this particular Figure 1. example, there are six different routes: Precedence Diagram (1) a b c e d f, (2) a b c d e f, (3) a c b d e f, (4) a c b e d f, (5) b a c d e f, and (6) b a c e d f. In general, this is the main idea behind how ALEKS is used to assess a student s knowledge on various mathematical topics. I signed up for a trial version as student for ALEKS and selected college algebra as the area in which I wanted to assess my knowledge. When taking the assessment, guessing is discouraged in order to be able to get a better idea of what a student actually knows since they might guess the right answer. The next questions chosen for the student are based on the idea in Figure 1 since it determines what they think a student can or cannot do based on correct, incorrect, or skipped questions. After all of the questions are asked that cover the domain space, Q, then students are shown a pie graph of the various items in Q 18

19 that they have mastered and those they still need to work on more. Based on this, the software provides students with questions they should be able to learn next based on the fact they have mastered elements earlier in the precedence relations. From this, it was easy to see that there can be many different routes taken on the way to mastering all of the content in a domain space. The reasoning behind Knowledge Space Theory is very logical, but students could still get lucky with answers, not follow directions, or use outside resources that would result in the program assuming they have already mastered concepts that they have not. But like any test, homework, or form of assessment, nothing is perfect. So while ALEKS does not do anything particularly different in determining the correctness of a student s answer compared to other online homework systems, it does try and provide a better evaluation of student knowledge in general through its design using Knowledge Space Theory, which is similar to what instructors do on a regular basis with their students by trying to determine what students know and what they are ready to learn. 6.3 Numerical versus Open-ended Answers While the dissension between educators over drill and practice type problems versus open-ended problems will probably always continue, it is something that mathematicians and instructors must consider. Usually, lower-level mathematics courses like college algebra and calculus are more computationally heavy and rely on following specific steps to find an exact number for an answer. Alternatively, upper-level mathematics courses like discrete mathematics or abstract algebra are proof-based courses that involve more open-ended questions. Until technology and programming can be further advanced, there is a limit on what types of problems can be asked and graded using online homework systems. While online homework systems can be very beneficial for lower-level mathematics courses in colleges where the classes can hold 100+ students, are they sufficient for grading homework of mathematics students in upper-level classes with less than 30 students? Do online homework systems limit instructors in the type of mathematics they want students to learn and practice? These are questions that I want to explore using the research I found on various online homework systems as well as a few selected homework problems in varying levels of mathematics courses. 2 Problem 1: 2x 1+ x 2 dx is a lower-level undergraduate course question from calculus 0 that could easily be asked using an online homework system. The numerical answer that would 2 be required for a correct submission is 3 ( ). Typically on this type of problem online, it does not ask the student what the substitution would be, unless it is the only section of problems they are attempting so it would be the only method in which they would try to solve the problems. Thus, in an online homework environment, even though it is easy to ask this question and there is only one correct answer, it is at a disadvantage to paper homework where students must show all their work to arrive at the answer. Especially since there are two ways to deal with limits when using u-substitution for integration, it is very easy for students to not properly document their steps when completing the online homework which leads to mathematically incorrect statements on their quizzes and homework. For example, using method 1 of switching back to x s after integrating with u =1+ x 2, we get 19

20 0 2 2x 1+ x 2 x= 2 dx = u 1 2 du = u 3 x= x= 2 x= 0 = (1+ x 2 ) = Using method 2 of changing the bounds to be in terms of u where u ( ) 3 2 = u =1+ x 2, we get 2 2x 1+ x 2 5 dx = u 1 2 du = = = The differences are very subtle, but the methods are different enough that students get mixed up quite frequently. The main problem students have when using the first method is they do not show that the limits are kept in terms of x when switching to u. They then forget to substitute x back in for u after integrating. It is possible that the problem could be programmed to include steps with both options of solving the problem, but it might not be very easy. As with any lower-level undergraduate mathematics course, it is extremely convenient for instructors to be switching their homework from paper to online homework systems, but then I would have to say that instructors might have to adjust their grading methods on quizzes and especially exams by being more lenient with grading. At least until the online homework systems become advanced enough to allow students to show their steps for each question and include the little details that mathematicians can be very particular about. Then again, many mathematicians might feel that the integrity of the mathematics is compromised by not marking students off for failing to pay attention to the types of details that can alter an expression enough to cause the problem to no longer be mathematically correct. Although Problem 1 is a fairly common and easy question to include in an online homework environment, one must wonder about the little proof-writing students usually have to do in calculus; that is, epsilon-delta proofs. Could these easily be graded by online homework systems? I would have to say yes, although for these proofs it would almost have to be submitted in various guided steps. Since there can be slight changes in the language used in each step, it might be hard for an online system to distinguish between right and wrong answers if students were able to answer it as an open-ended question. So for Problem 2 we have: Prove that lim 5x 3 x 1 ( ) = 2. Now if the online system were to have steps similar to the following, it would be possible for students to be able to submit their proof by filling in the blanks: 5 ( ) Proof : Let be given. Let δ =. So whenever < δ =. Then < ε < ε < ε. The only problem with this is if the students are unsuccessful in first determining what δ should be equal to for their problem. It would also be slightly difficult to know how many steps are required to finally prove the problem using the limit definition. Since 5x-5 and 5x-3-2 are mathematically equivalent, would it be acceptable for students to just stop at 5x-5 or just jump straight to 5x-3-2? With these variations, it could be more tedious for students to do the proof. Which makes one inclined to think that having an open-ended question would be best. But would the computer be able to pick up on all the possible variations in the proof? As an example, here are a few correct proofs out of many: Proof 1: Let ε > 0 be given and δ = ε 5. If x 1 < δ = ε 5, then 5 x 1 < ε 5x 5 < ε 5x 3 2 < ε. 20

Is that paper really due today? : differences in first-generation and traditional college students understandings of faculty expectations

Is that paper really due today? : differences in first-generation and traditional college students understandings of faculty expectations DOI 10.1007/s10734-007-9065-5 Is that paper really due today? : differences in first-generation and traditional college students understandings of faculty expectations Peter J. Collier Æ David L. Morgan

More information

Student Learning in Hybrid French and Spanish Courses: An Overview of Language Online 1

Student Learning in Hybrid French and Spanish Courses: An Overview of Language Online 1 N. Ann Chenoweth, Eiko Ushida, and Kimmaree Murday 115 Student Learning in Hybrid French and Spanish Courses: An Overview of Language Online 1 ABSTRACT N. ANN CHENOWETH University of Texas-Pan American

More information



More information

A Usability Study and Critique of Two Password Managers

A Usability Study and Critique of Two Password Managers A Usability Study and Critique of Two Password Managers Sonia Chiasson and P.C. van Oorschot School of Computer Science, Carleton University, Ottawa, Canada Robert Biddle Human

More information

Intellectual Need and Problem-Free Activity in the Mathematics Classroom

Intellectual Need and Problem-Free Activity in the Mathematics Classroom Intellectual Need 1 Intellectual Need and Problem-Free Activity in the Mathematics Classroom Evan Fuller, Jeffrey M. Rabin, Guershon Harel University of California, San Diego Correspondence concerning

More information

GRE. Practice Book for the Paper-based. GRE revised General Test. Second Edition.

GRE. Practice Book for the Paper-based. GRE revised General Test. Second Edition. GRE Practice Book for the Paper-based GRE revised General Test Second Edition 19587 Note to Test Takers: Keep this practice book until you receive your score report. This book contains

More information

A Self-Directed Guide to Designing Courses for Significant Learning

A Self-Directed Guide to Designing Courses for Significant Learning A Self-Directed Guide to Designing Courses for Significant Learning L. Dee Fink, PhD Director, Instructional Development Program University of Oklahoma Author of: Creating Significant Learning Experiences:

More information

Conditions Under Which Assessment Supports Students Learning

Conditions Under Which Assessment Supports Students Learning Learning and Teaching in Higher Education, Issue 1, 2004-05 Conditions Under Which Assessment Supports Students Learning GRAHAM GIBBS 1 & CLAIRE SIMPSON 2 1 Oxford University, 2 Open University, UK ABSTRACT

More information

Assessing Online Learning: Strategies, Challenges and Opportunities

Assessing Online Learning: Strategies, Challenges and Opportunities Assessing Online Learning: Strategies, Challenges and Opportunities Featuring content from A MAGNA PUBLICATION Effective Group Work Strategies for the College Classroom. Assessing

More information

On System Design. Jim Waldo

On System Design. Jim Waldo On System Design Jim Waldo On System Design Jim Waldo Perspectives 2006-6 In an Essay Series Published by Sun Labs December 2006 This work first appeared as part of the OOPSLA 2006 Essays track, October

More information

Advanced Problems. Core Mathematics

Advanced Problems. Core Mathematics Advanced Problems in Core Mathematics Stephen Siklos Fourth edition, October 2008 ABOUT THIS BOOKLET This booklet is intended to help you to prepare for STEP examinations. It should also be useful as preparation

More information

Turning Student Groups into Effective Teams

Turning Student Groups into Effective Teams Copyright 2004, New Forums Press, Inc., P. O. Box 876, Stillwater, OK. All Rights Reserved. Turning Student Groups into Effective Teams Barbara Oakley Oakland University Rebecca Brent Education Designs,

More information

Does Automated White-Box Test Generation Really Help Software Testers?

Does Automated White-Box Test Generation Really Help Software Testers? Does Automated White-Box Test Generation Really Help Software Testers? Gordon Fraser 1 Matt Staats 2 Phil McMinn 1 Andrea Arcuri 3 Frank Padberg 4 1 Department of 2 Division of Web Science 3 Simula Research

More information

Perfect For RTI. Getting the Most out of. STAR Math. Using data to inform instruction and intervention

Perfect For RTI. Getting the Most out of. STAR Math. Using data to inform instruction and intervention Perfect For RTI Getting the Most out of STAR Math Using data to inform instruction and intervention The Accelerated products design, STAR Math, STAR Reading, STAR Early Literacy, Accelerated Math, Accelerated

More information

How Do You Know It? How Can You Show It?

How Do You Know It? How Can You Show It? How Do You Know It? How Can You Show It? Penny Reed Wisconsin Assistive Technology Initiative Gayl Bowser Oregon Technology Access Program Jane Korsten Responsive Centers for Psychology and Learning Wisconsin

More information

Contingent Commitments Bringing Part-Time Faculty Into Focus

Contingent Commitments Bringing Part-Time Faculty Into Focus Contingent Commitments Bringing Part-Time Faculty Into Focus A SPECIAL REPORT from the Center for Community College Student Engagement Acknowledgments Our gratitude is unconditional. For their incalculable

More information

Six Sigma Black Belts: What Do They Need to Know?

Six Sigma Black Belts: What Do They Need to Know? This paper was presented during the Journal of Quality Technology Session at the 45th Annual Fall Technical Conference of the Chemical and Process Industries Division and Statistics Division of the American

More information

Is This a Trick Question? A Short Guide to Writing Effective Test Questions

Is This a Trick Question? A Short Guide to Writing Effective Test Questions Is This a Trick Question? A Short Guide to Writing Effective Test Questions Is This a Trick Question? A Short Guide to Writing Effective Test Questions Designed & Developed by: Ben Clay Kansas Curriculum

More information

Accounting Instructors Report

Accounting Instructors Report Accounting Instructors Report A Journal for Accounting Educators Belverd E. Needles, Jr., Editor ISSN#: 2325-0267 SPRING 2015 AIR NEWSLETTER TRENDS Troubleshooting Those Difficult Classroom Issues Belverd

More information

10 Principles of Effective Online Teaching: Best Practices in Distance Education. Featuring content from

10 Principles of Effective Online Teaching: Best Practices in Distance Education. Featuring content from 10 Principles of Effective Online Teaching: Best Practices in Distance Education Featuring content from AMAGNA PUBLICATION 10 Principles of Effective Online Teaching: Best Practices in Distance Education

More information

developer.* The Independent Magazine for Software Professionals Code as Design: Three Essays by Jack W. Reeves

developer.* The Independent Magazine for Software Professionals Code as Design: Three Essays by Jack W. Reeves developer.* The Independent Magazine for Software Professionals Code as Design: Three Essays by Jack W. Reeves Introduction The following essays by Jack W. Reeves offer three perspectives on a single theme,

More information

Evaluating CALL Software

Evaluating CALL Software Chapter 13 from L. Ducate and N. Arnold (eds) 2006. Calling on CALL: From Theory and Research to New Directions in Foreign Language Teaching. San Marcos: CALICO. Pre-publication copy. Evaluating CALL Software

More information

No Significant Distance Between Face to Face and Online Instruction: Evidence from Principles of Economics

No Significant Distance Between Face to Face and Online Instruction: Evidence from Principles of Economics No Significant Distance Between Face to Face and Online Instruction: Evidence from Principles of Economics Dennis Coates University of Maryland, Baltimore County Brad R. Humphreys University of Maryland,

More information

Evaluation. valuation of any kind is designed to document what happened in a program.

Evaluation. valuation of any kind is designed to document what happened in a program. Using Case Studies to do Program Evaluation E valuation of any kind is designed to document what happened in a program. Evaluation should show: 1) what actually occurred, 2) whether it had an impact, expected

More information

Traditional vs. Modern Teaching Methods: Advantages and Disadvantages of Each Master s Diploma Thesis

Traditional vs. Modern Teaching Methods: Advantages and Disadvantages of Each Master s Diploma Thesis Masaryk University Faculty of Arts Department of English and American Studies English Language and Literature Bc. Viera Boumová Traditional vs. Modern Teaching Methods: Advantages and Disadvantages of

More information

Toward a More Comprehensive Conception of College Readiness

Toward a More Comprehensive Conception of College Readiness Toward a More Comprehensive Conception of College Readiness 2007 David T. Conley Prepared for the Bill & Melinda Gates Foundation, March 2007 Table of Contents Introduction. 5 An Operational Definition

More information

Using assessment to support student learning. By Graham Gibbs

Using assessment to support student learning. By Graham Gibbs Using assessment to support student learning By Graham Gibbs Using assessment to support student learning Contents 1. Introduction 1 2. How assessment influences student learning 2 3. Pedagogic principles

More information

Methods for Understanding Student Learning University of Massachusetts Amherst Contributing Authors:

Methods for Understanding Student Learning University of Massachusetts Amherst Contributing Authors: R Urban Policy Calculus Lively Arts Minute Paper Accounting Syllabus Metaphysics Pre/Post Cervantes Cyberlaw Primary Trait COURSE-Based Review and Assessment Methods for Understanding Student Learning

More information


SAMPLE COPY NOT FOR DISTRIBUTION U.S. CULTURE SERIES: U.S. Classroom Culture MICHAEL SMITHEE SIDNEY L. GREENBLATT ALISA ELAND Funded by a grant from the Bureau of Educational & Cultural Affairs of the U.S. Department of State About NAFSA:

More information

Department of Psychology Montana State University M.S. in Psychological Science Graduate Student Handbook

Department of Psychology Montana State University M.S. in Psychological Science Graduate Student Handbook 1 Department of Psychology Montana State University M.S. in Psychological Science Graduate Student Handbook Last updated: August 2014 2 Table of Contents Program objectives...3 Pearls of wisdom from graduate

More information