Title of the submission: A Modest Experiment Comparing Graduate Social Work Classes, on-campus and at A Distance

Transcription

1 Title of the submission: A Modest Experiment Comparing Graduate Social Work Classes, on-campus and at A Distance Topic area of the submission: Distance Education Presentation format: Paper Names of the authors: Andrew Quinn and Ralph Woehle Department and affiliation: Department of Social Work, University of North Dakota, U.S.A. Mailing address: Department of Social Work, Box 7135, University of North Dakota, Grand Forks, ND addresses: ralphwoehle@mail.und.nodak.edu, andrew quinn@mail.und.nodak.edu Phone numbers: (701) and (701) Fax number: (701) Corresponding author: Ralph Woehle

2 Abstract A Modest Experiment Comparing Graduate Social Work Classes, on-campus and at a Distance Literature on distance education in higher education, particularly that education provided at a distance via technology, suggests that such means can provide learning as well as the traditional face-to-face classroom. The issue of evaluation of such delivery remains an important empirical issue for professional social work education, especially for the "human behavior in the social environment," area described in this paper. This paper will describe a quasi-experimental comparison of two classes that delivered content of masters level social work curriculum. The classes were nearly identical in curricular content, but differed substantially in pedagogy, particularly the means of delivery. One class (N=17) was delivered in an on-campus setting with computer technology assisting that delivery. The other (N=22) was delivered in off-campus settings of varying degrees of distance, and was substantially more dependent on computer technology as well as interactive television. Given the earlier findings of research on such means of delivery, we hypothesized that students would learn equally well in both classes. To aid in the test of the hypothesis, several variables were identified as predictors of learning. These variables include student age, time since graduation, undergraduate grade-point average, a pretest and a measure of achievement during the course. Two statistical methods were used to compare the classes. First, bivariate analysis was used to investigate differences between groups on the predictor variables. Second, regression models were developed for both the on-campus group and the distance group. The variance explained and the magnitude of the influence of the predictor variables were compared. Overall, it appeared that the results were consistent with the hypothesis that distance learning can produce similar results to the in-class experience. The explanation of these findings appears to lie in the differing achievement of students prior to their entry into the academic program, with undergraduate grade point average being an important strength for on-campus students, and experience since achieving their undergraduate degree an important strength for distance students.

3 A Modest Experiment Comparing Graduate Social Work Classes, on-campus and at A Distance Distance education became an important issue for social work education in the U.S.A in the 1990s. By the mid-nineties, a survey reported by Siegel, Jennings, Conklin, and Napoletano Flynn (1998) indicated that growth in distance education in social work was evident, and out of the 259 institutions surveyed 41 (16%) programs had distance offerings. These programs indicated that Human Behavior in the Social Environment (HBSE) courses were the most frequently offered curricular area, being offered at 51% of the institutions. Interestingly, the high number of offering of HBSE through distance education continued through 2005 (Moore, 2005). Moore surveyed 56 social work faculty and discovered that within the 56 faculty surveyed, HBSE had been taught at a distance a total of 75 times which was more that any other social work content area (Moore, 2005). Despite the high number of HBSE courses being offered through distance education technology (interactive television or ITV, web-based, electronic communication, listservs, discussion groups), few evaluative studies exist that question whether or not HBSE curriculum lends itself to delivery through distance education technologies. In fact, Rooney, Hollister, Freddolino, and Macy (2001) suggest that further research is needed to examine the effectiveness of distance education within different content areas such as HBSE. HBSE is a core curriculum course required to be taught by the Council on Social Work Education, which accredits social work education in the U.S.A. Social work educators would probably see HBSE as a knowledge course, in a curriculum that they

4 commonly divide into knowledge, values and skills. As a knowledge course, they would probably see it as more amenable to distance education than the more culturally intense values or the more practice intense skills, which social work educators would see as more amenable to face-to-face instruction. Literature Review Despite the call for evaluative studies of core curriculum offerings in social work education, few studies exist that examine the delivery of human behavior in the social environment (HBSE) at a distance (Barnett-Queen and Zhu, 1999; Johnson and Huff, 2000; Frey, Yankelov, and Faul, 2003). Unfortunately, when the delivery of HBSE at a distance has been evaluated, the evaluation tends to center on the students perceptions of the delivery method, rather than whether students learn the necessary content within HBSE. For example, Barnett-Queen and Zhu (1999) and Frey, Yankelov, and Faul examined attitudes about the methods of delivery of HBSE through distant technologies rather than focusing on student learning. Barnett-Queen and Zhu found that students in the distance delivery had more favorable attitudes towards the use of distance delivery techniques (posting of syllabus, class notes, class assignments, and discussion questions) than on campus students. In addition, Frey, Yankelov, and Faul focused on comparing the perceptions of distant students taking HBSE and research methods courses on their use of features found in WebCT, a course delivery package. The features included communication, posting of the course syllabus, on-line quizzes, and on-line discussion groups. Rather than focus on student perceptions of the delivery methods, Johnson and Huff (2000) focused on the use of computer-mediated communication ( and listservs) among HBSE students. The authors, however, did not use to disseminate

5 HBSE curriculum content. Instead, they found that and listservs tended to be used for course business with requests for grades being the most addressed item. As the literature clearly demonstrates the evaluation of student learning when HBSE is delivered via distance is sorely needed. To date, there appears to be no such studies on HBSE where the outcome measure is student learning. Despite the lack of examination of student learning in distance offerings of HBSE, one common method of evaluating distance offerings is to focus on student learning outcomes. For example, Coe and Elliot (1999) found no significant differences in course grades. In a meta analysis of 19 studies covering some 40 years of study in distance education other than social work, Matchmes and Asher (2000) concluded that there is no overall difference in learning between distance and traditional learners, but that two-way interaction improved learning over old one-way television, that workplacebased courses had higher achievement than non-workplace courses, and that the type of remote site did have an impact. Haga and Heitkamp (2000) reported similar grade-point averages between on campus and distance students. Faux and Black-Hughes (2000) reported an improvement from pre- to post-multiple choice scores when content was delivered on-line. Wolfson, Marsom and Magnuson (2005), though they relied on student perceptions and had an N of just 21, reported greater perceived achievement of learning objectives among on-line students than in-class students in a field education (internship) seminar. Few studies have considered the possibility that predicting student learning outcomes variables can be dependent on several variables. For instance, Stocks and Freddolino (1998) compared grades of a research methods course in social work on

6 campus to an Internet based course. The authors found that for the Internet section, attitudes about computers was the best predictor of course grades (24.6% of the variance explained), while for the on campus section, having an existing account was the best predictor (16.6% variance explained). The study also considered technology use, access to a home computer with a modem, s to the instructor, and s to a list as predictors of final grade. In another example, Harrington (1999) compared an in class statistics course to one delivered via computer programmed instruction. The author found that ethnicity, gender, age, grade point average, and course type (on campus vs. programmed instruction) account for 37% of the variance explained in course grades. In addition, the best predictor of course grades was the course type. In conclusion, the review of the above literature indicates a need to further examine the delivery of HBSE at a distance. In addition, exploration of course outcomes, such as grades and predictors of such outcomes are needed. Thus, the research question to be examined here twofold. First, can HBSE be successfully delivered via distance and produce equivalent student outcomes when compared to on campus delivery? Second, what are predictors of successful learning when delivering HBSE by differing methods? Methods This section will describe the variables, the differences of the two classes, and the statistical analysis of our study. The variables. The researchers had access to a variety of data describing the students. This included measures representing achievement prior to the class consisting of average grades for the students in their baccalaureate program (GPA) and the time since the students had achieved that degree as well as age (proxies for experience). A pretest of

7 the basic theoretical knowledge students should have achieved prior to the course as well as of knowledge the student should gain in the course was administered. Unreliable items were eliminated from this pretest to achieve a reliability of alpha=.70. Student achievement in the class was measured by a total of scores for the various assignments of the class. These assignments consisted of weekly presentations by students, as well as weekly multiple choice quizzes, exams and a paper. Most important for this study was the kind of class, or the treatment that students received via the differing course offerings. The classes as treatment groups. Two class offerings will be compared here, and the two classes shared many characteristics. The classes were offered from eastern North Dakota, U.S.A., beginning in January and ending in May Small populations are scattered over long distances in this prairie landscape. Both classes were part of a Master s of Social Work program. The subject of both classes was HBSE and both classes emphasized theoretical learning about macro social systems. Generally, theoretical concepts which were compatible with an approach to systems theory called complexity theory were emphasized in the course. All students would have a prerequisite HBSE course which included some content about systems theory. The classes used the same student assignments, though students were in control of the content of class activities via their weekly assignments. Both classes used student participation in the form of working in groups to develop examples of concepts assigned for each week. Student interaction outside of class centered on working in these groups, and in class it centered on individual student presentations emerging from their work in the groups.the classes were similar in size and had the same instructor. Both classes used a computer program called Blackboard to supplement instruction, and all standard information from

8 the instructor was posted on Blackboard for both classes. Students who missed classes were required to post their work on Blackboard. Both classes had at least some students who commuted substantial distances to attend classes. While the classes had much in common, they also differed substantially. The on-campus class was offered in a traditional format. These students were considered full time. It utilized live class interaction in the physical presence of the instructor. This class met in this person-to-person mode for a total of about 30 hours, two hours weekly over the 16-week semester, and students sat in small desks aligned in rows with the instructor in the front of the room. Student workgroups were able to meet each other in person weekly if they chose. The blackboard program was used for sharing established information for the most part, not for student interaction. The majority of this group had received their baccalaureate degree form the institution offering the graduate HBSE course, and some of those students had taken several classes from the instructor of the graduate HBSE course. In an end-of class student survey which rates the student's own initiative, the quality of the learning process and the ability of the instructor, students indicated above average ratings. The distance class consisted of students considered part time students. They met over interactive television for about 15 hours over the course of the semester, meeting alternate weeks. The students had all met one another in on-campus classes during the previous summer. Some of the students in that class attended in the room where the instructor was located, and the remainder were scattered in other interactive television rooms in clusters from one student to half a dozen, over hundreds of miles of territory. Blackboard was utilized extensively, presumably substituting on-line interaction for work

9 equivalent to the reduced student meeting time. Students would both organize their group work via Blackboard, and then present it on Blackboard in weeks when there was no interactive television meeting. The Blackboard discussion was carried on in a computer program function called discussion board, where students typed in their assignments and others, including the instructor, typed in responses for all to see, over some extended period of time. The end-of class student survey which rates the student's own initiative, the quality of the learning process and the ability of the instructor rated the class lower than usual ratings. Analysis. The research design was quasi experimental, comparing the two forms of class offerings, but also utilizing other available variables which might predict change. While the on-campus class was slightly smaller with 21 students, not all students completed the pretest resulting in just 17 students available for the study. In the distance class, 22 of 23 students completed the pretest. Bivariate analysis differences of means in the two class groups was utilized to describe the differences of the two classes. In addition, three separate regression models were developed to predict achievement in the two groups. Given the earlier findings of research on such means of delivery, we hypothesized that students would learn equally well in both classes, and that was supported. However, a logistic regression was utilized to examine possible selection bias, and did indeed find such bias. Results The two classes described above were compared, a total sample of 39 students. The statistics are, for the most part, very similar. Their age, undergraduate GPA, pretest score and total points for the course did not differ much, and the differences were not

10 statistically significant. As can be seen in these statistics, and as an independent t-test found, the only sizable difference in the two groups was the time since receiving their undergraduate degree, significant at the.05 level. Table 1: Comparative statistics of the on-campus and distance classes Variable On-campus Mean, (SD) Distance Mean, (SD) Age (12.22) (9.05) Undergraduate GPA 3.22 (.40) 3.17 (.44) Pretest score (4.69) (3.84) Total points for course (18.85) (10.10) Time since undergraduate degree 5.00 (7.71) 7.50 (7.85) In addition to the above discussed variables, the groups were compared on whether their undergraduate degree was completed at the same institution from which the two HBSE classes were delivered, or at another place. Within the on -campus group, ten of the 17 students had received their undergraduate degree at the HBSE-offering institution, but the distance students were much more likely to have received it elsewhere, with just five of the 22 having received their degree at the HBSE offering institution. The spread of total points for all 39 students was 125 to 189, with a mean of As Table 1 shows, the mean total score for the on-campus group was (SD=18.85), while the distance group had a mean score of (SD=10.10). An independent t test was conducted to see if there was a statistically significant difference between the two groups, and despite the slightly higher level of achievement of the distance students and the more limited variation for that group, there was no statistically

11 significant difference between the groups (t=-.52, df=37, p=.61). The lack of statistical significance seems to indicate that students achievements were equal for both HBSE classes, supporting our original hypothesis. To investigate the influence of age, undergraduate GPA, time since undergraduate degree, location of undergraduate degree, pretest, and delivery method had on total points, linear regression models were developed and presented in Table 2. For all 39 students, the predictor variables were able to explain 44% of the variance in total scores. Table 2 column 2 provides the beta weights for each predictor for all 39 students. The larger the beta weight, the stronger the prediction, limited of course by the total variance explained. For example, in the total 39 student group undergraduate GPA had a beta weight of.57 therefore, the conclusion can be drawn that undergraduate GPA was the strongest predictor of total achievement, with higher undergraduate GPAs indicating higher achievement, as might be expected. Time since graduation, at.54, is almost as important an explanation of achievement, and the greater the time, the greater the achievement. Interestingly, the delivery method, whether on-campus or at a distance, had little influence (beta= -.16) on total achievement, again indicating support for our central hypothesis. Table 2: Regression models of student achievement Variable Total Sample Beta On-campus Beta Distance Beta Weight (r 2 =44%) Weight (r 2 =60%) Weight (r 2 =7%) Undergraduate GPA Time since graduation Undergraduate location

12 Pretest score Age Class (Distance =1) Next, the two delivery methods were divided and the predictor variables were regressed against total score for each group, as indicated in the last two columns of Table 2. The predictor variables considered were undergraduate GPA, time since undergraduation, location of undergraduate degree, pretest, and age. Interestingly, the variance explained for the on campus group was 60% while the off campus group had only 7% of the variance explained by the predictors. Thus, we can reach a general conclusion that being in one program as opposed to the other is very important for the students, because the effects of the other variables in the model are changed by the program. On-campus the effects are strong, but at a distance they all but disappear. Due to the difference in the models, a post hoc hypothesis that different types of students were selected into the distance delivery and the on-campus was examined. A logistic regression model was run using the predictor variables of undergraduate GPA, time since graduation, location of undergraduate degree, age, pretest, and total points. The logistic model was designed to examine whether these variables were predictors in whether a student was likely to enroll in the distance delivery or on-campus. Overall, the model was able to predict enrollment accurately 80% of the time. More specifically, for the on campus students (N=17) the logistic model was able to classify 77% of the students (n=13) accurately. For the distance delivery, the logistic model was able to classify 82% of the students (n=18) accurately. Given that the only significant difference

13 in the means of the two groups was the time since graduation, it appears to be the most logical explanation of the fact that effects are very different in the two groups. This is a very important design issue which we will discuss below. Discussion What then can we say about the teaching of this knowledge course in the differing formats? Clearly, our hypothesis that there would be no difference in the outcome was supported. In terms of learning outcomes, and consistent with what previous work had suggested, our study suggests that the knowledge content of HBSE in social work education will lend itself to varying delivery methods, at least so far as students' learning is concerned. However, other issues may lie behind this conclusion, and those issues may need attention in both the study of, and the delivery of social work education. Differences noted between the two classes in two areas are suggestive, and those areas are the ratings of the courses by students, and the differences in experience of the students, particularly those which suggest a selection bias. While the detailed data from student ratings of the courses was not presented here, distance students were not as comfortable with their own class process, the organization of the course, or the instructor as were the on-campus students. While we do not have data which describes the reasons for these differences, we suspect from what we know anecdotally that there are both contextual and course delivery issues at play here. The contextual issues which detract from students' appreciation of the course delivery are probably couched in the non-educational responsibilities which come with the course delivery. Distance students, more than on-campus students perhaps, maintain a high level of responsibility to job and family while getting their degrees. Thus, contextual stress

14 may be compounded by the distance student's participation in the degree program, and makes the experience a less positive one for the student. This contextual stress can be contrasted with the on-campus students' greater experience with the campus and instructor, which when compounded with the in-person delivery probably leads to a more comfortable course delivery. However, the selection of distant students into the program at a time more distant from their undergraduate degrees than the on-campus group may have also changed their experience, and in a positive way. Given that they probably had more professional experience and the opportunity to develop learning skills since graduation, they were probably less dependent on their prior academic experience for their achievement than on-campus students. Another way of thinking about this is the language of strengths often used in social work. The on-campus students had strengths gained from their on-campus experience, and were able to utilize them to achieve in the course. The distance students had strengths gained from their professional experience, and were able to use them to achieve. Thus, both groups achieved, and at about the same level. However, the presence of differing backgrounds suggests another issue for comparisons of delivery with quasi-experimental designs, namely that researchers need to be aware of the possibility of selection bias. They need to examine that issue by collecting data on background variables which describe those differences. Finally, we recognize that our small study, in a unique region of the U.S.A., is not generalizable to other areas. However, the support of the no difference hypothesis, as well as the notation of differences in the groups, suggests issues which other educators in other settings might well consider as they study educational delivery.

15 References Barnett-Queen, T., & Zhu, E. (1999). Distance education: Analysis of learning preferences in two sections of undergraduate HBSE-like human growth and development course: Face-to face and Web-based distance learning. Paper presented at the 3rd Annual Technology Conference for Social Work Education, Charleston, SC. Coe, J. R. & Elliot, D. (1999). An evaluation of teaching direct practice courses in a distance education program for rural settings. Journal of Social Work Education, 35 (3), Faux, T. L., & Black-Hughes, C. (2000). A comparison of using the Internet versus lectures to teach social work history. Research on Social Work Practice, 10(4), Frey, A., Faul, A., & Yankelov, P. (2003). Student perceptions of Web-assisted teaching strategies. Journal of Social Work Education, 39(3), Haga, M. & Heitkamp, T. (2000). Bringing social work education to the prairie. Journal of Social Work Education, 36 (2), Harrington, D. (1999). Teaching statistics: A comparison of traditional classroom and programmed instruction/distance learning approaches. Journal of Social Work Education, 35 (3), Johnson, M. M., & Huff, M. T. (2000). Students use of computer-mediated communication in a distant education course. Research on Social Work Practice, 10(4), Matchmes, K. & Asher, W. (2000). Meta-analysis of telecourses in distance education.

16 American Journal of Distance Education, 14 (1), Moore, B. (2005). Faculty Perceptions of the Effectiveness of Web-Based Instruction in Social Work Education: A National Study. Journal of Technology in Human Services, 23(1/2), Rooney, R., Hollister, C. D., Freddolino, P., & Macy, J. (2001). Evaluation of distance education programs in social work. Journal of Technology in Human Services, 18(3/4). Siegel, E., Jennings, J G., Conklin, J. & Napoletano Flynn, S. A. (1998). Distance learning in social work education: Results and implications form a national survey. Journal of Social Work Education, 34 (1), Stocks, J. T., & Freddolino, P. P. (1998). Evaluation of a World Wide Web-based graduate social work research methods course. Computers in Human Services, 15(2/3), Wolfson, G K., Marsom, G., & Maguson, C. W. (2005). Changing the nature of discourse: teaching field seminars on-line. Journal of Social Work Education, 14 (2),