A Modest Experiment Comparing HBSE Graduate Social Work Classes, On Campus and at a. Distance

Transcription

1 A Modest Experiment 1 Running Head: Comparing On campus and Distance HBSE Courses A Modest Experiment Comparing HBSE Graduate Social Work Classes, On Campus and at a Distance

2 A Modest Experiment 2 Abstract This paper describes a quasi experimental comparison of two master's level classes delivering content on Human Behavior and the Social Environment. One class was delivered in an on campus setting. The other was delivered using distance technologies. The authors hypothesized that students would learn equally well in both classes. A no difference in learning outcome hypothesis was supported, even after accounting for other variables. The authors speculate that student strengths may mask any difference resulting from delivery methods, and future comparisons need to consider statistical differences in the treatment groups.

3 A Modest Experiment 3 A Modest Experiment Comparing HBSE Graduate Social Work Classes, On Campus and at a Distance Distance education became an important issue for social work education in the U.S.A. in the 1990s. By the mid nineties, a survey reported by Siegel, Jennings, Conklin, and Napoletano Flynn (1998) indicated that growth in distance education in social work was evident, and out of the 259 institutions surveyed 41 (16%) programs had distance offerings. These programs indicated that Human Behavior in the Social Environment (HBSE) courses were the most frequently offered curricular area, offered at half of the institutions with distance education. Interestingly, the high number of offerings of HBSE through distance education appeared to continue through 2005 (Moore, 2005). Moore surveyed 56 social work faculty and discovered that within the 56 faculty, HBSE had been taught at a distance a total of 75 times, more that any other social work content area (Moore, 2005). Despite the high number of HBSE courses being offered through distance education technology (interactive television or ITV, web based, electronic communication, listservs, discussion groups), few evaluative studies exist that question if HBSE curriculum lends itself to delivery through distance education technologies. In fact, Rooney, Hollister, Freddolino, and Macy (2001) suggested that further research was needed to examine the effectiveness of distance education within different content areas such as HBSE. HBSE has been a core content area required by the Council on Social Work Education. Social work educators have probably seen HBSE as a knowledge course, in a curriculum that they have traditionally divided into knowledge, values and skills. As a knowledge course, they probably have seen it as more amenable to distance education than the more culturally intense values or the

4 A Modest Experiment 4 more practice intense skills, both which social work educators probably have seen as more amenable to face to face instruction. Literature Review Despite the call for evaluative studies of core curriculum offerings in social work education, few studies have examined the delivery HBSE at a distance (Barnett Queen and Zhu, 1999; Johnson and Huff, 2000; Frey, Yankelov, and Faul, 2003). Unfortunately, when the delivery of HBSE at a distance has been evaluated, the evaluation has tended to center on the students' perceptions of the delivery method, rather than if students learned the necessary content of HBSE. Numerous studies have taken this approach. Barnett Queen and Zhu (1999) and Frey, Yankelov, and Faul (2003) examined attitudes about the methods of delivery of HBSE through distant technologies. Barnett Queen and Zhu found that students in the distance delivery had more favorable attitudes toward the use of distance delivery techniques (posting of syllabus, class notes, class assignments, and discussion questions) than on campus students. In addition, Frey, Yankelov, and Faul focused on comparing the perceptions of distant students taking HBSE and research methods courses on their use of features found in WebCT, a course delivery package. The features included communication, posting of the course syllabus, on line quizzes, and on line discussion groups. Rather than focus on student perceptions of the delivery methods, Johnson and Huff (2000) focused on the use of computer mediated communication ( and listservs) among HBSE students. The authors, however, did not use to disseminate HBSE curriculum content. Instead, they found that and listservs tended to be used for course business with requests for grades being the most addressed item. As the literature has demonstrated, the evaluation of student learning when HBSE has been delivered via distance is needed.

5 A Modest Experiment 5 Despite the lack of examination of student learning in distance offerings of HBSE, other studies have focused on learning outcomes in distance offerings. Coe and Elliot (1999) found no significant differences in course grades. In a meta analysis of 19 studies covering some 40 years of study in distance education other than social work, Matchmes and Asher (2000) concluded that there is no overall difference in learning between distance and traditional learners. They said that two way interaction improved learning over old one way television, that workplace based courses had higher achievement than nonworkplace courses, and that the type of remote site did have an impact. Haga and Heitkamp (2000) reported similar grade point averages between on campus and distance undergraduate students. Faux and Black Hughes (2000) reported an improvement from pre to post multiple choice scores when content was delivered on line. Wolfson, Marsom and Magnuson (2005), though they relied on student perceptions and had an N of just 21, reported greater perceived achievement of learning objectives among on line students than in class students in a field education seminar. A few studies have considered the possibility that predicting student learning outcome variables can be dependent on several variables. For instance, Stocks and Freddolino (1998) compared grades of a research methods course in social work on campus to an internet based course. The authors found that for the internet section, attitudes about computers was the best predictor of course grades (24.6% of the variance explained), while for the on campus section, having an existing account was the best predictor (16.6% variance explained). The study also considered technology use, access to a home computer with a modem, s to the instructor, and s to a list as predictors of final grades. In another example, Harrington (1999) compared a traditional on campus statistics course to one delivered via computer programmed instruction. The author found that ethnicity, gender, age, grade point average, and course type (on campus vs.

6 A Modest Experiment 6 programmed instruction) account for 37% of the variance explained in course grades. In addition, the best predictor of course grades was the course type. In conclusion, the review of the above literature has indicated a need to further examine the delivery of HBSE at a distance. In addition, it has shown that the exploration of course outcomes, such as grades and predictors of such outcomes, is needed. Thus, the research question examined here is twofold. First, we asked if HBSE could be successfully delivered via distance and produce equivalent student outcomes when compared to on campus delivery. Second, we asked what would predict successful learning when delivering HBSE by differing methods. Methods This section will describe the variables, the differences of the two classes, and the statistical analysis used in our study. The statistical analyses included comparison of means, regression and multiple correlation. The variables Variables studied included measures representing achievement prior to the class consisting of average grades for the students in their baccalaureate program (GPA). Also studied was the time since the students had achieved that degree, as well as age, as proxies for experience. A pretest of the basic theoretical knowledge that students should have achieved prior to the course as well as of knowledge the student should gain in the course was administered. Unreliable items were eliminated from this pretest to achieve a reliability of alpha=.70. When this pretest variable proved to be of limited importance in the analysis, five missing values were replaced by mean values of the pretest to allow inclusion of all students in the two classes in the larger study. Student achievement in the class was measured by a total of scores for the various assignments of the class. These assignments consisted of weekly presentations by students, as well as weekly multiple choice quizzes, periodic

7 A Modest Experiment 7 multiple choice exams, and, a paper. All of the above variables were based on some numerical count or sum derived from judgment, the points awarded on an assignment for example. Two variables were dichotomous. One dichotomous variable indicated whether students had gotten their undergraduate degree from the institution where they were presently enrolled. Most important for this study was the kind of class, or the treatment that students received via the differing course offerings, which was also dichotomous. The classes as treatment groups. Two class offerings have been compared here, and the two classes shared many characteristics. The classes originated from eastern North Dakota, beginning in January and ending in May Small populations are scattered over long distances in this prairie landscape. Both classes were part of a master's of social work program. The subject of both classes was HBSE and both classes emphasized theoretical learning about macro social systems. Generally, theoretical concepts which were compatible with an approach to systems theory called complexity theory were emphasized in the courses. All students would have had a prerequisite HBSE course that included some content about systems theory. Both classes used the same student assignments, and students had some control of the content of class activities via their weekly assignments. Both classes used student participation in the form of working in groups to develop examples of concepts assigned for each week. Student interaction outside of class centered on working in these groups, and in class it centered on individual student presentations emerging from their work in the groups. The classes were similar in size and had the same instructor. Both classes used a computer program called Blackboard to supplement instruction, and all standard information from the instructor was posted on Blackboard for both classes. Students who missed classes were required to post their work on Blackboard. Both classes had at least some students who commuted substantial distances to attend classes. The classes thus had much in common, though they also differed substantially.

8 A Modest Experiment 8 The on campus class was offered in a traditional format. These students were considered full time. This class utilized live class interaction in the physical presence of the instructor. It met in this person to person mode for a total of about 30 hours, two hours weekly over the 15 week semester, and students sat in rows of desks with the instructor in the front of the room. Student workgroups were able to meet in person as they chose. The blackboard program was used for sharing established information for the most part, not for student interaction. Thirteen of the 21 students in this group had received their baccalaureate degree from the same institution offering the graduate HBSE course under study, and some of those students had taken several classes from the instructor of the graduate HBSE course. In an end of class student survey, just one of seventeen students responding indicated disagreement with positive statements regarding the course and instructor. This student indicated he or she did not find the required reading helpful or well utilized by the instructor, and found the effort required in the course inappropriate. The distance class consisted of students considered part time students. Just five of the 23 students in this group had received their undergraduate degree from the HBSE offering institution. The students met over interactive television for about 15 hours over the course of the semester, meeting alternate weeks. The students had all met one another in on campus classes during the previous summer. Some of the students in that class attended in the room where the instructor was located, and the remainder were scattered in other interactive television rooms in clusters from one student to half a dozen, over hundreds of miles of territory. Blackboard was utilized extensively, presumably substituting on line interaction for work equivalent to the reduced student meeting time. Students would both organize their group work via Blackboard, and then present work on Blackboard in weeks when there was no interactive television meeting. The Blackboard discussion was carried on in a computer program function called "discussion board," where students typed in

9 A Modest Experiment 9 their assignments and others, including the instructor, typed in responses for all to see, over some extended period. The end of class student survey indicated that at least several students were frustrated with the class processes. While just 12 students in this group completed the end of class survey, several students indicated they did not find the readings useful, found that amount of effort required was inappropriate, and that instructor feedback was inadequate. Five of the twelve responding thought the course was not useful. Clearly, the two classes neither constitute a true experiment with random assignment nor a randomly selected sample. Rather, the selection was determined by who applied and was accepted in an admissions process. Thus, this study cannot assert that all of the important variables are equalized by random assignment to the groups, or that the sample would have variable values similar to any larger population. These two points will therefore receive further attention below. Significance and power In this paper, we follow the convention of setting an acceptable level of significance at point 05. Significance, of course, addresses the probability of making a type I error, that is rejecting a null hypothesis that is true. Power assesses the probability of making a type II error, failing to reject a false null hypothesis. Given that the null hypothesis of no difference in achievement of the two classes is central to our work here, we would want to avoid that error. With our sample of 44, to reject the null hypotheses with 95% certainty we would need to explain 25% or more of the variance in achievement. This paper centers on a null hypothesis of no difference in achievement in the two class types, so we will want to significance of.05 and 25% of the variance explained to reject the null hypothesis.

10 A Modest Experiment 10 Analysis The research design was quasi experimental, comparing the two forms of class offerings, but also utilizing other available variables that might predict learning and interact with the form of class offering. Bivariate analysis, the differences of means, was utilized to describe the differences of the two classes. Bivariate correlation was utilized to look at relationships between variables. In addition, a linear regression model was developed to predict achievement in the two groups. Given the earlier findings of research on such means of delivery, we hypothesized that students would learn equally well in both classes, and that was largely supported. The comparison of means and the regression model which included the student's program as a dummy variable both suggested that the program type had little or no impact on the achievement outcome variable. The means of the available variables for the two classes described above were compared. The statistics are, for the most part, very similar as can be seen in Table 1. The age, baccalaureate GPA, pretest score and total points for the course did not differ much, and the differences were not statistically significant. As can be seen in these statistics, and as an independent t test found, the only sizable difference in the two groups of students was the time since receiving their baccalaureate degree. Insert Table 1 about here The spread of total points for all course work for all 44 students was 125 to 189. As Table 1 shows, the mean total score for the both groups was 156, but the on campus group was more variable, with a standard deviation of 18, while the distance group had a standard deviation of 10. An independent t test was conducted and as might seem obvious, even assuming unequal variance, there was no statistically significant difference between the groups (t=.02, df=30, p=.98). The lack of a difference in outcomes seems to indicate that students' achievements were equal for both HBSE

11 A Modest Experiment 11 classes, supporting our original hypothesis. Of the remaining predictor variables available to the analysis, time since receiving an undergraduate degree was the only variable showing significant difference (t=3.40, df=42, p=.002) between the two class groups, with a mean of 4.3 years on campus, and 12.1 years in the distance program. To investigate the influence of age, baccalaureate GPA, time since baccalaureate degree, location of baccalaureate degree, pretest, and delivery method had on total points, a linear regression model was developed and presented in Table 2. The predictor variables were able to explain 41% (p=.00) of the variance in total scores. This 41% figure is clearly in excess to the 25% power level and therefore powerful enough to analyze further. Table 2 column 2 provides the beta weights for each predictor. The larger the beta weight, the stronger the prediction, limited of course by the total variance explained. For example, baccalaureate GPA had a beta weight of.60 (p=.00) therefore, the conclusion can be drawn that baccalaureate GPA was the strongest predictor of total achievement, with higher baccalaureate GPAs indicating higher achievement, as might be expected. Time since baccalaureate graduation, at.59 (p=.00), is almost as important an explanation of achievement, the greater the time, the greater the achievement. Interestingly, the delivery method, whether on campus or at a distance, had limited influence with a beta of.25 (p=.09). The variance explained by class delivery method would be approximately the square of.25 or about 6%, well below the required power level for rejecting the null hypothesis with 95% certainty. However, this level made us a bit less certain that no difference exists in class outcomes. Insert Table 2 about here The last column of Table 2 lists the bivariate correlation between the total points achieved and the respective independent variables. While the beta coefficients represent the degree of explanation provided by each independent variable while the others are, in effect, held constant, the

12 A Modest Experiment 12 bivariate correlation accounts for just one variable at a time. Given that the one known difference between the two groups was the fact that distance students had received their baccalaureate earlier than the on campus students had, the importance of that difference might be further examined. As can be seen in the last column of Table 2, the bivariate correlation between total points achieved and years since the undergraduate GPA was r=.45, but the beta coefficient was.60. As discussed above, the weak relationship between class type and achievement appears when we account for various independent variables. As we have shown, years since undergraduate degree had a notable relationship with the delivery method (see table 1) as well as with GPA (r=.44). In an ideal experiment, where the difference in time since the degree would have been removed, undergraduate GPA may have a strong impact on achievement and the type of class may have been weakly negative for distance students achievement. Discussion What then can we say about the teaching of this knowledge course in the differing formats? First, our hypothesis that there would be no difference in the outcomes between class types was left standing, or nearly so. In terms of learning outcomes, and consistent with what previous work had suggested, our study suggests that the knowledge content of HBSE in social work education will lend itself to the varying delivery methods we studied, at least so far as students' learning the content and in the context we described. However, other issues may lie behind this conclusion, and those issues may need attention in both the study of, and the delivery of social work education. Differences noted between the two classes in two areas are suggestive, and those areas are the end of class ratings of the classes by the students, and the differences in at least the time of post baccalaureate experience of the students, and perhaps differing selection of students resulting from the amount of that experience.

13 A Modest Experiment 13 Second, student responses to the end of class survey are of interest. Data on students' endof class responses cannot be matched on a student by student basis due to confidentiality of the student responses and was thus lumped into the dichotomous class type variable. Nonetheless, we can guess that the dissatisfaction of some distance students is couched in the distance delivery as well as context that detracts from students' appreciation of the course delivery. Distance students, more than on campus students perhaps, maintain a high level of responsibility to job and family while getting their degrees. Thus, contextual stress may be compounded by the distance student's participation in the degree program, and this makes the experience a less positive one. This contextual stress can be contrasted with the on campus students' possibly lesser stress, as well as their greater experience with the campus and instructor, which when compounded with the in person delivery may have led to a more meaningful and comfortable course delivery. Third, the selection of distant students into the program at a time more distant from their baccalaureate degrees than the on campus group may have also given them different and greater experience. Given that they probably had more opportunity to develop learning skills since graduation, they were probably less dependent on their prior academic experience for their achievement than on campus students. In the language of experimental studies, the distance students had matured more than on campus students prior to program entry. Another way of thinking about this is the language of strengths often used in social work. The on campus students had varying strengths gained from their on campus experience, and were able to utilize the strengths available to achieve in the course. The distance students had varying strengths gained from their post graduate experience, and were able to use those strengths to achieve. Thus, both groups achieved at the same level.

14 A Modest Experiment 14 Fourth, the presence of differing student backgrounds suggests other issues for comparisons of and delivery of varying forms of classes. Researchers need to be aware of the possibility of selection bias. They need to examine that issue by collecting data on background variables that describe possible differences. In our own program, we will seek to expand the list of known variables in future studies. We would further expect relevant variables to vary from one setting to another, for example an urban setting may not select students into programs as they were selected in our study on the basis of time since undergraduate degree. The relationship of the interaction of variables should be further researched, and that research may have interesting implications for selection of students through admission processes. If the time since the degree does reduce the impact of GPA on achievement for example, then programs like our distance program which attract more experienced students might be able to reduce GPA requirements without damage to achievement outcomes. Finally, we recognize that our small study, in a unique region of the country, is not readily generalizable to other areas. Thus, our attention to power and significance probably has more to do with how we will generalize the research into our environment, than it does for generalization into other environments. However, the support of the no difference hypothesis, as well as the notation of differences in the groups, suggests issues that other educators in other settings might well consider as they study educational delivery. We would submit that any transfer of understanding that might be tentatively made to other settings should be done in relation to comparable course content, namely advanced macro HBSE content. Micro practice content, which might be learned through person toperson role play for example, might have considerably different achievement results on campus vs. at a distance.

15 A Modest Experiment 15

16 A Modest Experiment 16 References Barnett Queen, T., & Zhu, E. (1999). Distance education: Analysis of learning preferences in two sections of undergraduate HBSE like human growth and development course: Face to face and Web based distance learning. Paper presented at the 3rd Annual Technology Conference for Social Work Education, Charleston, SC. Coe, J. R. & Elliot, D. (1999). An evaluation of teaching direct practice courses in a distance education program for rural settings. Journal of Social Work Education, 35 (3), Faux, T. L., & Black Hughes, C. (2000). A comparison of using the Internet versus lectures to teach social work history. Research on Social Work Practice, 10(4), Frey, A., Faul, A., & Yankelov, P. (2003). Student perceptions of Web assisted teaching strategies. Journal of Social Work Education, 39(3), Haga, M. & Heitkamp, T. (2000). Bringing social work education to the prairie. Journal of Social Work Education, 36 (2), Harrington, D. (1999). Teaching statistics: A comparison of traditional classroom and programmed instruction/distance learning approaches. Journal of Social Work Education, 35 (3), Johnson, M. M., & Huff, M. T. (2000). Students use of computer mediated communication in a distant education course. Research on Social Work Practice, 10(4), Matchmes, K. & Asher, W. (2000). Meta analysis of telecourses in distance education. American Journal of Distance Education, 14 (1), Moore, B. (2005). Faculty Perceptions of the Effectiveness of Web Based Instruction

17 A Modest Experiment 17 in Social Work Education: A National Study. Journal of Technology in Human Services, 23(1/2), Rooney, R., Hollister, C. D., Freddolino, P., & Macy, J. (2001). Evaluation of distance education programs in social work. Journal of Technology in Human Services, 18(3/4). Siegel, E., Jennings, J G., Conklin, J. & Napoletano Flynn, S. A. (1998). Distance learning in social work education: Results and implications form a national survey. Journal of Social Work Education, 34 (1), Stocks, J. T., & Freddolino, P. P. (1998). Evaluation of a World Wide Web based graduate social work research methods course. Computers in Human Services, 15(2/3), Wolfson, G K., Marsom, G., & Maguson, C. W. (2005). Changing the nature of discourse: teaching field seminars on line. Journal of Social Work Education, 14 (2),

18 A Modest Experiment 18 Table 1: Comparative Statistics of the On Campus and Distance Classes Variable On campus Mean, (SD) Distance Mean, (SD) t 1 P 1 Age 34.8 (12.0) 37.1 (9.1) Undergraduate GPA 3.2 (.40) 3.15 (.43) Pretest score 16.3 (4.21) 15.0 (3.76) Years since degree 4.3 (7.05) 12.1 (8.17) Total course points (18.1) (9.9) Significance levels and t values assuming equal and nonequal variance are nearly identical. Table 2: Regression Model of Student Achievement as Dependent Variable 1 Variable Std. Err. Beta t P r Undergraduate GPA Years since degree Institution (Different=0, Same=1) Pretest score Age Class (On Campus=0, Distance=1) Adjusted r 2 =41%, standard error=11.