IDEA Research Report #3

Transcription

1 IDEA Research Report #3 DECEMBER 2001 Research Report #3 Are Quantitatively-Oriented Courses Different? Donald P. Hoyt and Subashan Perera The IDEA Center This study examined classes within each of four disciplines that differed in the degree to which they emphasized math/quantitative skills. A number of distinctive characteristics of such courses were discovered. Overall, those teaching classes emphasizing math/quantitative skills are more likely than their colleagues in similar disciplines to have poorly motivated students who regard their courses as difficult and demanding and who offer relatively negative evaluations of the course, its instructor, and course outcomes. However, these conclusions vary with discipline. Implications of these findings are discussed. Courses differ in the types of skill or background they emphasize. Some stress writing skills, others computer skills, and others creative/artistic skills. The focus of this study is on classes that stress mathematical/quantitative work. Of course, almost all classes in mathematics and statistics place a heavy emphasis on such work, but courses in other disciplines differ widely in the degree to which quantitative skills are required. In business, the social sciences, the physical sciences, and several other disciplinary groups, the curriculum usually includes courses that have a quantitative emphasis as well as courses that rely on other kinds of academic skills or background. To increase the dependability of the findings, this study was restricted to classes enrolling at least 10 students for which a response rate of at least 75% was obtained. These restrictions ensured minimally acceptable levels of reliability and representativeness. Because response rate is related to student attendance, which, in turn, is related to teaching effectiveness, our sample probably excluded a disproportionate number of classes that were taught with low levels of effectiveness. Faculty members participating in the IDEA program complete a Faculty Information Form and are asked to indicate, on a volunteer basis, the degree to which each of seven academic skills 1 are required in their class (None, Some, Much). During the year (September 1998 through August 1999), approximately 80% of all participants responded to this question. In this study, we first explored three types of potential differences between quantitatively-oriented classes and other kinds of classes not emphasizing quantitative skills. Student motivations for enrolling in the class Course characteristics (types of assignments; difficulty) Global outcome measures (student ratings of teaching effectiveness) Secondly, we made comparisons within each of four broad disciplinary groupings, chosen because IDEA results were available for at least 50 classes at each of the three levels of quantitative involvement (None, Some, Much). Social/Behavioral Sciences (including economics, history, political science, psychology, public administration, social science, social work, and sociology) Business (including general business, business administration and management, accounting, finance, information/data processing, and marketing) Physical Sciences (including chemistry, physics, and other physical sciences) Applied Mathematics (including computer and information sciences, engineering, and engineering technologies) In addition to comparing classes that differed in their emphasis on mathematics/quantitative skills, differences among the four disciplinary groups were also explored. 1 Writing, oral communication, computer applications, group work, mathematical/quantitative work, critical thinking, and creative/artistic/design endeavor.

2 Results Student Motivation Three of the questions on the IDEA form elicited information related to student motivation. I had a strong desire to take this course. I really wanted to take a course from this instructor. I really wanted to take this course regardless of who taught it. A fourth item was indirectly related: I worked harder on this course than on most courses I have taken. For these four questions, a 5-point response scale was used with anchors of Definitely false (scored 1) and Definitely true (scored 5). Summary of Results: Student Motivation Students in classes where math/quantitative skills were emphasized were less motivated but worked harder than did students in classes that placed no emphasis on these skills. In three of the four disciplinary groups, student motivation was similar to that for all classes. However, in Physical Science classes, students not only were significantly less motivated than those in the other three disciplinary groups, but also reported working harder. Details supporting these conclusions are given below. Quantitative Emphasis Subgroup Comparisons. On the three motivation items, statistically significant differences were found among classes differing in their emphasis on math/quantitative skills; averages for Much emphasis classes were lower than for No emphasis classes while those for the Some emphasis classes were between these extremes. Differences between Much and None groups were large enough to be considered practically significant (see Appendix Table 1). On the effort item (I worked harder on this course than on most courses I have taken), the difference between the Much group and the None group was even greater, and the Some group again averaged between these extremes. Additionally, in classes that require Much math/quantitative work, students generally worked harder despite being less motivated than those in classes with No math/quantitative emphasis (see Appendix Table 1). Comparisons of s With All Other Classes. On an overall basis, average ratings on the three motivation items were similar to those in the all classes database for three of the four disciplinary groups. However, for the Physical Science subgroup, average motivation ratings on these items were well below those for the entire database (see Appendix Tables 2, 3, and 4). The Physical Science group also had the highest average rating of effort. The other three disciplinary groups had effort ratings about the same as that for all classes (see Appendix Table 5). Thus, in this study, students in the Physical Science classes were poorly motivated but put forth an above average effort to learn. Those in the other three disciplinary groups displayed typical levels of both academic motivation and effort. Physical science classes were also distinctive when withindiscipline comparisons were made of Much, Some, and No emphasis groups. For this disciplinary group, motivation ratings, while well below the all classes average, did not differ among the three emphasis groups. For the other three disciplinary groups, motivation ratings progressed from low to high as the math/quantitative requirement decreased (see Appendix Tables 2, 3, and 4). On the effort item, however, there were no disciplinary differences in trends; the Much emphasis group averaged significantly higher levels of effort than did the No emphasis group for all four disciplines (see Appendix Table 5). Course Characteristics The IDEA form asked students to describe the course in terms of: Amount of reading, Amount of work in other (non-reading) assignments, and Difficulty of the subject matter. Again a 5-point rating scale was used, anchored by 1=Much less than most courses and 5=Much more than most courses. Average ratings on these items were very different among the four disciplinary groups and among the three groups defined by their math/quantitative requirements. Summary of Results: Course Characteristics In classes with a high emphasis on math/quantitative skills, ratings of both difficulty and amount of other (non-reading) work were much higher than for No emphasis classes. Amount of reading, however, did not differ among emphasis groups except for Social/Behavioral Sciences where the None group averaged well above the Much group. There were large disciplinary differences on all three items. Relative to all classes in the database, Applied Math classes were distinguished by low ratings on amount of reading and high ratings on amount of other work. Similar, but less extreme results were found for Physical Science classes which were most differentiated by a very high difficulty rating. Business classes were rated slightly above average on all three characteristics, while Social/Behavioral Sciences had very high ratings on amount of reading and very low ratings on amount of other work. Details are given below.

3 Quantitative Emphasis Subgroup Comparisons. When disciplines were combined, classes placing Much emphasis on math/quantitative skills required less reading, much more other work, and were regarded as much more difficult than classes with No quantitative emphasis. In each instance, the average rating for Some emphasis classes was between the two extremes (see Appendix Table 6). Comparison of s With All Classes. For Business classes, ratings on all three course characteristic items were slightly above the all classes average. Applied Math classes required much less reading and much more other work than did all classes. Physical Science classes were like those in Applied Math except that the differences from the all classes averages were smaller; this disciplinary group was most distinctive because its average on difficulty of subject matter was especially high. Classes in Social/Behavioral Sciences made high reading demands when compared to classes, but fewer demands in the amount of other (non-reading) work (see Appendix Tables 7, 8, and 9). Previous IDEA Center studies have shown that difficulty is correlated principally with both amount of reading and amount of other work (correlations of.41 and.47). Results for the Social/Behavioral Sciences, Applied Math, and Business groups were consistent with these findings. But the extremely high difficulty rating for Physical Science courses cannot be explained by its combination of below average reading requirements and above average amount of other work requirements. It appears that, while the types of required assignments typically affect the difficulty of a course, difficulty in the Physical Sciences may also be a function of the high level of intellectual challenge posed by the discipline. Global Outcome Measures The IDEA system provides four global ratings of teaching effectiveness. The most important of these is Progress on Relevant Objectives. This measure is obtained by averaging student ratings of progress on objectives the instructor identified as Important or Essential after double-weighting the latter 2. As a comprehensive measure of student learning, it represents a potent proxy for teaching effectiveness. The other three global measures are student ratings on single items: As a result of taking this course, I have more positive feelings toward this field of study. Overall, I rate this instructor an excellent teacher. Overall, I rate this course as excellent. Standard 5-point rating scales are employed for these three items, anchored by 1=Definitely false and 2 To compensate for differences among objectives, averages are converted to standard T Scores (mean=50, standard deviation=10) before being weighted and combined. Because progress ratings for Essential objectives are typically higher than those for Important objectives, the average PRO for all classes is about 52, a little higher than the 50 average for progress ratings on individual learning objectives. 5=Definitely true. Summary of Results: Global Outcome Measures The more emphasis placed on math/quantitative skills, the lower the overall ratings of teaching effectiveness. This was more apparent on measures related to attitude toward the discipline and quality of the course than on quality of the teacher or amount learned. Classes in Social/Behavioral Sciences were distinctive because the lowest ratings of effectiveness were obtained in classes where Some emphasis was placed on math/quantitative skills. Most global measures of teaching effectiveness for these disciplinary groups were not significantly different from those for all classes in the database except for the Physical Sciences; for this disciplinary group, ratings of overall teaching effectiveness were consistently below the all classes average. Details are given below. Quantitative Emphasis Subgroup Comparisons. Global outcome ratings were consistently higher for classes where No emphasis was placed on math/quantitative skills than on those with Much emphasis; Some emphasis classes always averaged in between. Differences were sharper on increased positive attitude and excellent course measures (see Appendix Table 10). Low to high trends for Much, Some, and No emphasis groups, respectively, were found on all four global ratings for three of the disciplinary groups, except for Social/Behavioral Science classes, where, although the No emphasis group had the highest average, the lowest average was consistently obtained by the Some group (see Tables 11, 12, 13, and 14). Further study is needed to determine if this anomaly reflects either inconsistencies among the individual disciplines included in this broad category (economics, sociology, psychology, etc.) or some peculiarities of this sample. Comparisons of s with All Classes. Results for the Physical Sciences were relatively distinctive. Whereas most of the global ratings for other three disciplines were not significantly different from those for all classes, Physical Science classes averages were lower (see Tables 11, 12, 13, and 14) 3. 3 Outcomes are affected by a number of extraneous circumstances -- factors not under the control of the instructor. Chief among these is student motivation. When such factors were controlled statistically, outcomes for these disciplinary groups were more positive, regardless of the degree to which math/quantitative skills were emphasized. However, even after adjustments, outcomes for the Physical Sciences group were significantly below those for all classes and for other disciplinary groups. Whether this reflects the distinctiveness of the disciple or of its teaching processes cannot be inferred from these results.

4 Discussion Those teaching classes which require math/quantitative skills encounter a relatively unique learning environment. This study provided evidence that students in such classes are, on the whole, less motivated than those in other classes. Since motivation is a major determinant of the amount of student learning, teachers face a serious challenge of dealing with student attitudes and fears so that they will not seriously inhibit subject matter mastery. We can only speculate on the reasons motivation ratings were so low. It seems likely that these findings reflect the lack of confidence that students commonly express in their mathematics background. Fear may stimulate effort, but it is not an effective facilitator of learning. The general findings of this study were consistent with this observation. Students in quantitatively-oriented classes worked harder than those in other classes. Nonetheless, student ratings of outcomes were generally less favorable in classes emphasizing math/quantitative skills. Students who are fearful that their skills may be inadequate to meet the demands of the class are likely to be handicapped in learning beyond what would be expected purely on the basis of their academic qualifications. As expected, they rated these classes as having a very high difficulty level. Although, when all classes are considered, difficulty is most closely related to amount of reading and amount of other (non-reading) work. The high difficulty rating of mathoriented courses cannot be explained by these relationships. It seems likely that the high difficulty ratings of math/quantitatively oriented classes reflects both math anxiety and the fact that classes requiring such a background pose especially challenging intellectual demands. If these findings are confirmed in subsequent studies, it may be necessary to take them into account in routine IDEA reports to participating faculty. Since there were many differences among the four disciplinary groups included in this study, it is apparent that many of its implications must be discipline-specific. Additional studies are needed to confirm these findings through replication, to expand both the number of disciplinary groups and the number of objectives, and to study individual disciplines. Such studies should also determine if the effectiveness of specific teaching methods varies with the amount of emphasis given to math/quantitative skills. Appendix Table 1: Student Motivation Quantitative Emphasis Subgroup Comparison: Average Rating (Number of Classes) Student Motivation Item Level of Quantitative Emphasis Much Some None I had a strong desire to take this course (487) 3.53 (527) 3.73 (1236) I really wanted to take a course from this instructor (488) 3.20 (527) 3.32 (1236) I really wanted to take this course regardless of who taught it (488) 3.35 (527) 3.44 (1236) I worked harder on this course than on most courses I have taken (488) 3.61 (527) 3.46 (1236) Table 2: I had a strong desire to take this course. Comparison: Average Rating 1. Applied Mathematics S,N>M Business S,N>M Physical Sciences NS Social/Behavioral Sciences N>M,S Significant Differences 1,2>3,4 1>2,3,4 1>2,3 2,4>3 All Classes

5 Table 3: I really wanted to take a course from this instructor. Comparison: Average Rating 1. Applied Mathematics S,N>M Business S,N>M Physical Sciences NS Social/Behavioral Sciences N>S,M Significant Differences 1>2,3,4 1>2,3,4 1>2,3,4 2>4 2>3,4 2,4>3 All Classes Table 4: I really wanted to take this course regardless of who taught it. Comparison: Average Rating 1. Applied Mathematics NS Business NS Physical Sciences NS Social/Behavioral Sciences N>S,M Significant Differences NS NS 4,2>3 All Classes Table 5: I worked harder on this course than on most courses I have taken. Comparison: Average Rating 1. Applied Mathematics M>N,S Business M,S>N Physical Sciences M>N Social/Behavioral Sciences M>S,N Significant Differences NS 2,3>1,4 NS All Classes

6 Table 6: Course Characteristics Quantitative Emphasis Subgroup Comparison: Average Rating (Number of Classes) Student Motivation Item Level of Quantitative Emphasis Much Some None Amount of reading 3.12 (489) 3.17 (529) 3.43 (1236) Amount of work in other (non-reading) assignments 3.67 (487) 3.55 (527) 3.21 (1236) Difficulty of subject matter 3.92 (487) 3.66 (527) 3.33 (1236) Students rated the items on a 5-point scale: 1=Much less than most courses 2=Less than most courses 3=About average 4=More than most courses 5=Much more than most courses Table 7: Amount of reading Comparison: Average Rating 1. Applied Mathematics NS Business NS Physical Sciences NS Social/Behavioral Sciences N>S>M Significant Differences 2,3,4>1 2,3,4>1 2,3,4>1 2>3 2,4>3 4>2,3 All Classes Students rated the items on a 5-point scale: 1=Much less than most courses 2=Less than most courses 3=About average 4=More than most courses 5=Much more than most courses Table 8: Amount of work in other (non-reading) assignments Comparison: Average Rating 1. Applied Mathematics M>N Business M,S>N Physical Sciences M>S,N Social/Behavioral Sciences M>S,N Significant Differences 1,2,3>4 1,2,3>4 1,2,3>4 1>2 All Classes Students rated the items on a 5-point scale: 1=Much less than most courses 2=Less than most courses 3=About average 4=More than most courses 5=Much more than most courses Table 9: Difficulty of the subject matter Comparison: Average Rating 1. Applied Mathematics M>S>N Business M>S>N Physical Sciences M>S>N Social/Behavioral Sciences M>S>N Significant Differences NS NS 3>1,2,4 2,4>1 All Classes Students rated the items on a 5-point scale: 1=Much less than most courses 2=Less than most courses 3=About average 4=More than most courses 5=Much more than most courses

7 Table 10: Global Outcome Measures Quantitative Emphasis Subgroup Comparison: Average Rating (Number of Classes) Student Motivation Item Level of Quantitative Emphasis Much Some None Progress on relevant objectives (492) (527) (1236) As a result of taking this course, I have more positive feelings toward this 3.53 (487) 3.66 (527) 3.95 (1236) field of study. 2 Overall, I rate this instructor as excellent (487) 3.99 (527) 4.21 (1236) Overall, I rate this course as excellent (487) 3.66 (527) 3.95 (1236) 1 Average of student ratings of progress on objectives identified by the instructor as either important or essential. To compensate for differences among objectives, averages are converted to T Scores, a standardized score with an average of 50 and standard deviation of 10. Essential objectives are double weighted. 2 Table 11: Progress on Relevant Objectives Comparison: Average Rating 1. Applied Mathematics NS Business N>M Physical Sciences S,N>M Social/Behavioral Sciences M,N>S Significant Differences 1,2,4>3 2>4 NS Average of student ratings of progress on objectives identified by the instructor as either important or essential. To compensate for differences among objectives, averages are converted to T Scores, a standardized score with an average of 50 and standard deviation of 10. Essential objectives are double weighted. Because progress ratings for Essential objectives are typically higher than those for Important objectives, the average PRO for all classes is about 52, a little higher than the 50 average for progress ratings on individual learning objectives. Table 12: As a result of taking this course, I have more positive feelings toward this field of study. Comparison: Average Rating 1. Applied Mathematics S,N>M Business N>S>M Physical Sciences N>M Social/Behavioral Sciences N>S,M Significant Differences 1,2,4>3 1,2>3,4 1,2,4>3 All Classes

8 Table 13: Overall, I rate this instructor as excellent. Comparison: Average Rating 1. Applied Mathematics N>M Business N>S,M Physical Sciences NS Social/Behavioral Sciences M,N>S Significant Differences 4>1,2,3 NS 4>2,3 All Classes Table 14: Overall, I rate this course as excellent. Comparison: Average Rating 1. Applied Mathematics S,N>M Business N>S>M Physical Sciences N>M Social/Behavioral Sciences N>S,M Significant Differences 1,2,4>3 1,2>3,4 4>2,3 1>3 All Classes The IDEA Center 211 South Seth Child Road Manhattan, KS or idea@ksu.edu Home Page: Fax: