Online vs. Classroom Instruction: Student Satisfaction and Learning Outcomes in an Undergraduate Allied Health Pharmacology Course

Transcription

1 RESEARCH NOTES vs. Instruction: Student Satisfaction and Learning Outcomes in an Undergraduate Allied Health Pharmacology Course LaDonna S. Hale, PharmD Emily A Mirakian, MPA, PA-C David B. Day, EdS, MPAS, PA-C instruction is frequently utilized in allied health education yet only a small number of controlled comparative studies specifically in healthcare education have been published. Purpose: The purpose of this study was to compare student satisfaction and objective learning outcomes of an undergraduate allied health online pharmacology course using streaming media lectures to traditional classroom instruction. Methods: The online (n=47) and classroom (n=177) courses for fall 2005 and spring 2006 used the same instructor, notes, text, learning objectives, and exams. Student characteristics, learning style preferences, and aptitude for distance learning were measured. A statistically reliable, valid survey measured student satisfaction with elements of the course, instructor, and self-perceived knowledge gains. Learning outcomes were evaluated using withdrawal rates and exam scores. Results: Mean satisfaction scores for both courses were high, generally > 4.0/5.0. Mean scores from the classroom students were significantly higher than online students regarding students ability to share ideas, instructor s ability to establish rapport, self-perceived knowledge gains, fundamental principles and application of material, and overall excellence of course. There were no significant differences in objective exam scores or withdrawal rates. Conclusion: The online and classroom pharmacology courses had similar withdrawal rates and exam scores, indicating similar learning gains. Overall, both courses had high student satisfaction ratings in all criteria measured. However, students in the online course were less satisfied with 8 criteria related to student satisfaction with instructor rapport, course excellence, peer interaction, and self-perceived knowledge gains. J Allied Health. 2009; 38:e36 e42. LaDonna S. Hale is Associate Professor and David B. Day is Assistant Professor, Department of Physician Assistant, Wichita State University, Wichita, KS 67260; Emily A. Mirakian, formerly a student at Wichita State University, is currently a Physician Assistant, Wesley Medical Center, Emergency Department, Wichita, KS RN779: Received December 4, 2007; accepted February 28, Address correspondence and reprint requests to: LaDonna Hale, PharmD, Wichita State University, Department of Physician Assistant, 1845 Fairmount, Box, Wichita, KS Phone (316) ; fax (316) ; ladonna.hale@wichita.edu. ONLINE EDUCATION has become increasingly utilized as a means of instruction. In 2002, 56% of two-year and fouryear institutions offered online courses and the number offering degree or certificate programs, which can be completed entirely through distance instruction, has increased from 8% in 2000 to 19% in , 2 Likewise, an increasing number of allied health programs augment classroom instruction with online modalities and/or offer mostly or completely online degree completion programs. In the allied health professions, students must not only meet certain knowledge competencies, they must effectively integrate and apply that knowledge. 3 Experts have presented arguments both for and against certain aspects of online education. An evidence-based literature review published in 2002 evaluated 35 research articles measuring various aspects of online instruction specifically related to healthcare education. 4 No controlled studies evaluating student satisfaction with online instruction were identified. One controlled study evaluated self-perceived knowledge gains and did show higher ratings with online instruction. Four other controlled studies evaluating objective knowledge gains found no significant difference. However, the multitude of variables which influence learning and cognition may complicate comparative research, making definitive conclusions difficult. 5 Research into online education is still in its infancy and many questions remain unanswered. 4 This study is unique in that it compares two semesters of allied health pharmacology courses taught by the same instructor, using the same text, notes, learning objectives, and exams and measures both actual and self-perceived knowledge gains as well as satisfaction with the instructor and course. Thus, the similarity of the pharmacology courses studied here may help limit the large number of variables usually encountered in comparative research. Study Purpose The purpose of this study is to compare online instruction using streaming media lectures to traditional classroom instruction in an undergraduate allied health pharmacology course taught by the same instructor as measured by: 1) percentage of student withdrawals, 2) mean exam scores, 3) letter grade distribution, and 4) student satisfaction with the instructor, course, and selfperceived knowledge gains. e-36 HALE, MIRAKIAN, AND DAY, v Pharmacology Course

2 Methods SETTING AND STUDY POPULATION This study was conducted in a three-credit hour 16-week undergraduate allied health pharmacology course at University. The University s 15,000-student population is varied with a large number of part-time, nontraditional students. The average age of an undergraduate student is 24 years with over 85% being from the state of. 6 The potential study population included all students enrolled in the online courses, 22 in fall 2005 and 25 in spring 2006 (total n = 47) and all students enrolled in the traditional classroom courses, 95 in fall 2005 and 82 in spring 2006 (total n = 177). SIMILARITIES AND DIFFERENCES BETWEEN COURSES This pharmacology course is required for entry into the dental hygiene and nursing programs and serves as an elective for other allied health students. The course discusses most major classes of drugs regarding mechanism of action, pharmacokinetics, side effects, drug interactions, contraindications, therapeutic use, appropriate drug monitoring, and clinical application. Both courses used the same instructor, text, course notes, and learning objectives, similar grading criteria, and the same or similar paper-pencil exams. Each course had three exams, the last being comprehensive. Exams were 95% multiple choice and 5% short answer. Generally 20% of questions required a higher level of synthesis and application of knowledge and 10% of questions came from text readings. The classroom students took the exam in the classroom with the TABLE 1. Instructional Elements of the and Pharmacology Courses. Similarities/Differences Exams 3 Exams, 100 pts each (last exam being comprehensive) Same 98% of exam questions are identical between the courses. Quizzes None Weekly quizzes, 10 pts each Weekly quizzes are used in the classroom course to reinforce weekly content. Homework Weekly homework assigned, including 3 asynchronous discussion board activities. 2 3 pts each None Due to online testing security concerns, weekly homework were used to reinforce course content. Since these are open-book, the point values are minimal. Learning Objectives Same Same All exam questions are derived from the learning objectives. Practice Questions Available 2 weeks before each exam. Same practice questions were available to both courses. Course Text Same text used for both courses. Same 10% of exam questions come from the text rather than the notes. Course Notes 110 page note packet required for the course. Same Same notes packet used for both courses. Lecture Material Approximately 2.5 hrs of video lecture viewing required each week. Approximately 2.7 hrs spent each week in the class room. Students in the TD course also had access to the same video lectures used in the online class. Enrollment Cap 25 student maximum No maximum Clinical Group Patient Case Assignments Students complete 2 of 4 cases individually. Groups of 4 6 students meet online for a synchronous discussion of answers. Each group posts one set of answers. Students review and comment on each other s work. Students complete 4 cases either individually or in groups as homework. Students form teams of 5 7. Teams use an audience response system to answer questions as each case is discussed. Correctness of answers is not directly graded. Full participation is worth 5 pts. e-37 HALE, MIRAKIAN, AND DAY, v Pharmacology Course

3 instructor present. The online students were required to use a proctored testing facility for exams. Most local students used the testing facilities on campus. Students living farther away had the option of using a proctored testing facility near them. The classroom students received lectures with the instructor present using PowerPoint while the online students viewed previously videotaped lectures through streamed media (either Tegrity TM or MS Producer TM ) accessed either online or Characteristic TABLE 2. Student Characteristics n=26 n=107 p-value Percent female 84.6% 86.1% Age Categories % 75.0% % 12.0% % 8.3% % 4.6% Student Rank Freshman 3.8% 0.9% Sophomore 15.4% 27.8% Junior 38.5% 45.4% Senior 30.8% 13.0% Graduate Student 2.8% 4.6% Other 7.7% 8.3% Self-Reported Grade Point Average % 0.0% % 1.9% % 4.6% %.5% % 24.1% I do not wish to answer 0.0% 25.9% Computer Experience 1 year 3.8% 0.9% years 11.5% 20.4% 5 10 years 42.3% 48.1% 11 years 42.3% 30.6% Previous Courses Yes 80.8% 79.6% No 19.2% 19.4% Distance courses, but not online 0.0% 0.9% No. of Previous Courses 0 courses 19.2% 18.5% course 34.6% 41.7% 2 3 courses 15.4% 32.4% 4 5 courses 23.1% 3.7% 6 courses 7.7% 3.7% Previous Pharmacology Courses Yes 7.7% 14.8% No 80.8% 82.4% No, but have clinical pharm exp 11.5% 2.8% Data were compared using the Chi square test or Fisher s exact test as appropriate. through CDs. This streaming video consists of audio and video of the instructor and the rolling PowerPoint slides. The course is offered through the online course management system, Blackboard TM. The streaming video lectures were also accessible by students in the classroom course as a means of repetition or to make up missed class lectures. Some group work was required in each course. In the online course, the only required face-to-face time with the instructor was a two hour orientation session at the beginning of the course. An estimated 95% of the instructor/student interaction in the online course was asynchronous as compared to less than 20% for the classroom course. There was an enrollment cap of 25 students in each online course; there was no cap in the classroom course. Table 1 shows other similarities and differences between instructional elements of the online and classroom courses. COURSE INSTRUCTOR S BACKGROUND AND TRAINING The course instructor is a registered pharmacist with a Doctor of Pharmacy and 10 years clinical experience and is a tenured Associate Professor with six years teaching experience. The fall 2005 semester was her twenty-first semester teaching the classroom course and fifth semester teaching the online course. Prior to teaching the online course she received specialized training in Blackboard TM management and online teaching methodologies. MEASUREMENTS Because students were not randomly selected for the online course, but rather self-selected it, a distance learning aptitude survey was used to measure aptitude for online instruction and the Canfield s Learning Styles Inventory (CLSI) was used to measure learning style preferences. These tests were used to detect potential differences between the two groups of students. Other measured student characteristics included: age, gender, years in school, prior online courses, and prior pharmacology experience. Student satisfaction with elements of the course, instructor, and self-perceived gains in learning was measured using the Individual Development and Educational Assessment (IDEA) Diagnostic Form Report Survey. Learning outcomes were objectively evaluated using withdrawal rates, mean exam scores and letter grade distribution. All students were included in the analysis of course withdrawal rates. Exam scores and grade distribution were analyzed only for students who completed all three exams. All survey data were included, regardless of the student s course completion status; however, because the IDEA survey was completed following the last exam, it only represents those students who remained enrolled in the course. DESCRIPTIONS OF SURVEY INSTRUMENTS The distance learning aptitude survey is a 12-question survey used by various colleges as a student self assessment. It was originally adapted from Are Telecourses for Me? from PBS- Journal of Allied Health, Summer 2009, Volume 38, Number 2 e-38

4 Question Course N Mean SD p-value Q4: Demonstrated importance and significance of the subject matter Q8: Stimulated students to intellectual effort beyond that required by most courses Q13: Introduced stimulating ideas about the subject Q15: Inspired students to set and achieve goals which really challenged them Q5: Formed teams or discussion groups to facilitated learning Q16: Asked students to share ideas with others whose backgrounds / viewpoints differed Q18: Asked students to help each other understand ideas or concepts Q1: Displayed personal interest in students and their knowledge Q2: Found ways to help students answer their own questions Q7: Explained reasons for criticism of academic performance Q20. Encouraged student-faculty interaction outside of class Q9: Encouraged students to use multiple recourses to improve understanding Q11. Related course material to real life situations Q14: Involved students in hands on projects eg research, case studies, or real life activities Q19. Gave projects, tests, or assignments that required original or creative thinking Q3: Scheduled course work in ways encouraging students to stay up-to-date with work Q6: Made it clear how each topic fit into the course Q10: Explained course material clearly and concisely Q12: Gave tests, projects, etc. that covered the most important points of the course Q17: Provided timely and frequent feedback on tests, projects, etc to help students improve Q21: Gaining factual knowledge (terminology, classifications, methods, trends) Q22: Learning fundamental principles, generalizations, or theories Q23: Learning to apply course material to improve thinking, problem solving, and decisions Q29: Learning how to find and use resources for answering questions or solving problems Q33: Amount of reading Q34: Amount of work in other (non-reading) assignments <0.001 Q35: Difficulty of subject matter Q37: I worked harder on this course than most courses I have taken Q39: I really wanted to take this course regardless of who taught it Q: As a rule, I put forth more effort than other students on academic work Q41: Overall, I rate this instructor an excellent teacher Q42: Overall, I rate this course as excellent Q36: I had a strong desire to take this course Q38: I really wanted to take this course from this instructor Q40: As a result of taking this course, I have more positive feelings toward this field of study Means were compared using the two-tailed Student s t-test and Lavene s test for equality of variance. Stimulating Student Interest Fostering Student Collaboration Establishing Rapport Encouraging Student Involvement Structuring Experiences Learning on Relevant Objectives Course Difficulty Student Description Overall Ratings Misc Questions TABLE 3. Results for IDEA Student Perceptions and Satisfaction Survey e-39 HALE, MIRAKIAN, AND DAY, v Pharmacology Course

5 TABLE 4. Objective Measurements of Learning: Course Grades and Withdrawal Rates Mean exam scores (3 exams)* receiving an A receiving a B receiving a C receiving a D receiving an F passing the course (receiving an A, B, or C) n= % ± 10.6 n= % ± 10.7 p-value % 29% < % 39% 21% 21% 17% 8% 0% 3% 83% 89% Course withdrawal rate 28% 22% *Mean exam scores expressed as mean ± standard deviation. Means were compared using the students t test. Frequency data were compared using the Chi square test or Fisher s exact test as appropriate. Adult Learning Service, The Agenda, Spring Each question has 3 possible answers assigned a point value ranging from 1 to 3. The total response points range from 12 to 36. Students scoring 27 are likely to be successful in an online course; students scoring may succeed in an online course, but may need to make some adjustments in study habits; and students scoring 17 are not likely to be successful in an online course. No data specifically validating this survey could be located. The CLSI is a 30-item, validated and reliable assessment with 21 subscale variables grouped into four major categories that measures preferred learning styles. 8, 9 It is described in detail elsewhere. 9 Both the CLSI and distance learning aptitude survey were taken either online or on paper following either the second or third exam. The IDEA diagnostic form report survey is a standardized, statistically reliable, valid survey developed by IDEA Center and used at a variety of universities nationwide to evaluate student satisfaction. 10 It has been used nationwide to evaluate both online and classroom courses. The IDEA survey is a 5- section assessment with criteria using a Likert-rating scale designed to provide information regarding the instructor s ability to stimulate student interest, foster student collaboration, establish rapport, encourage student involvement, and structure classroom experiences. It also provides information regarding the student s self-perceived progress on relevant course learning objectives, course difficulty, some salient characteristics of the student, and overall satisfaction with the instructor and course. The IDEA survey was administered on paper following the third exam. IDEA does provide summary data with means adjusted for extraneous factors that have been statistically shown to affect student satisfaction and learning that are beyond the control of the instructor. However, individual student responses (raw data) were analyzed for the purposes of this study, because the effect of those factors on an online course are unknown. Currently, IDEA does not consider the course delivery format to be one of those extraneous factors, therefore the decision to use raw data seemed reasonable and less likely to introduce bias. 10 This research project was approved by the university s institutional review board. All course data were de-identified by the course instructor prior to analysis. Survey data were anonymous. DATA ANALYSIS Statistical analysis was performed with SPSS version 13.0 (SPSS Inc., Chicago, IL). The alpha-level was set at Mean data were compared using the independent sample, two-tailed student s t-test and Lavene s test for equality of variance. Frequency data were compared using the chi-square test or Fisher s exact test when appropriate. RESULTS Fifty-five percent (26/47) of online and 60% (107/177) of classroom students completed the first survey and 47% (22/47) vs. 48% (85/177) completed the CLSI. Students in the online course were more likely to have taken a higher number of previous online courses, p = Student characteristics are reported in Table 2. There were no statistical differences in mean scores for the distance learning aptitude survey. However, a lower percentage of online students scored in the top tier, 27. No students scored 17. There were no statistically significant differences between the students with regard to preferred learning styles or expected course grade as measured by the CLSI. The results for the IDEA survey are presented in Table 3. Ninety-one percent (/47) of online and 82% (146/177) of classroom students completed the IDEA survey. Mean satisfaction scores for both courses were high, generally > 4.0/5.0 and found to be above the national and University averages for courses of similar size and type. Statistically significant differences were noted on 11 of the criteria. Eight of these criteria were related to student satisfaction with the course, instructor, and self-perceived knowledge gains, two criteria were related to course difficulty and student self-description, and the relevance of one is unknown. Mean scores from the classroom students were significantly higher than the online students regarding students ability to share ideas; helping students answer their own questions; instructor s level of interest in students; students self-perceived knowledge gains, fundamental principles, and application of knowledge; students positive feelings towards this course of study; and the overall excellence of the course. students had a higher mean score for the question related to the amount of work in the course, but the classroom students had a higher mean in response to how hard they Journal of Allied Health, Summer 2009, Volume 38, Number 2 e-40

6 worked in the course. No significant differences were observed in objective measurements of learning outcomes regarding withdraw rates and mean exam scores, but there were differences noted in the letter grade distribution; see Table 4. DISCUSSION Significant differences were found between the online and classroom courses in 11 of the IDEA survey criteria. students were less satisfied with the instructor s ability to establish rapport as indicated by Q1) displayed personal interest in students and their knowledge and Q2) found ways to help students answer their own questions; and fostering student interaction as indicated by Q16) asked students to share ideas and experiences with others whose backgrounds and viewpoints differed. The classroom had typical impromptu questions and discussion during lecture, while instructor/student interaction in the online class was generally asynchronous via . Students questions were answered within 24 to 48 hours for both courses; however, the instructor s interactions were typically down to business, brief, and likely lacked the kind of personality expressed in face-to-face interaction. Due to the structure of this particular online course, opportunities for peer interaction and instructor interaction were limited. students were also less satisfied with self-perceived knowledge gains, fundamental principles, and application of course material. Yet there were no significant differences in mean exam scores, which did require application of knowledge and fundamental principles. Both courses had similar structured opportunities to practice knowledge application and participate in peer interaction through three sets of case studies. All instructor feedback related to the case studies was immediate for the classroom students (occurring as class discussion) but asynchronous for the online students. The online students completed two cases, discussed answers in a synchronous online small group chat, posted the group answers online, read the other groups answers and commented on them. This required three to seven days for feedback. The online class had only three additional opportunities for peer interaction in the form of asynchronous discussion board activities all occurring at the beginning of the semester. The questions posed were not high level critical thinking questions and therefore students typically posted one answer and did not comment substantially on previous postings. The instructor rarely provided feedback in these asynchronous discussion boards. Thus, the lower satisfaction in the areas of instructor rapport, peer interaction, and knowledge gains may be due, in part, to the limited delayed asynchronous feedback from the instructor in the online course as compared to more frequent immediate synchronous feedback in the classroom and lower quantity and quality of peer interaction. It is unknown how much unstructured student interaction such as study groups or comparing of scores between peers may have occurred. It is reasonable to assume that this occurs less in the online environment, thus online students may have limited ways of measuring themselves against others. Alterations in the structure of this online course with close attention to improving communication, faculty and peer interaction and feedback may have improved student satisfaction in these areas. Instructors should recognize that prompt feedback and interaction with peers is often needed to build student confidence in learning outcomes and establish rapport with students. This may not occur as frequently in online classes as it does in traditional classroom settings. Based on written comments from course evaluations, students found the synchronous chats valuable and enjoyable but felt scheduling was sometimes difficult. When synchronous chats are not feasible, increasing the frequency of asynchronous chats and weekly assignments that require critical thinking and application of course material may be an effective way to improve student perceptions. High quality and quantity, synchronous and asynchronous, instructor and peer interaction can occur in online courses, and asynchronous discussion has been shown to actually increase some students abilities to interact. 11, 12 For example, an asynchronous discussion board can provide shy students with enough time to carefully think out their responses in advance. Providing responses in writing rather than verbally may overcome public speaking barriers. It is important to deliberately focus on providing multiple opportunities for application of knowledge and critical thinking skills with frequent and prompt feedback and opportunities for high quality instructor and peer interaction. Interestingly, online students had a higher mean response to Q34) amount of work in other non-reading assignments; while classroom students had a higher mean response to Q37) I worked harder on this course than most courses I have taken. Indicators of amount of reading, difficulty of subject matter, and amount of effort exerted were similar. Because the classroom students did not have weekly homework assignments but did have weekly quizzes, it is likely that they did not consider study time to be an assignment. Because of the lack of weekly quizzes, the online students likely spent more time studying for exam. However, the pressure of weekly quizzes may have been perceived as more rigorous thus the higher ratings of worked harder. The perception of worked harder may also reflect differences in letter grade distribution. It is unknown how students desire to take the course from this instructor (Q38) is relevant, because only one instructor teaches this course at this university. Although withdrawal rates and mean exam scores did not differ significantly, a difference in the letter grade distribution was noted. Besides the 300 exam points and 15 group case study points in both courses, the online students had 35 points from weekly homework activities; while the classroom students had eight 10-point quizzes (80 points). This indicates that a review of the point assignments for quizzes and activities in these courses may be indicated. STUDY LIMITATIONS Students self-selected either the online or classroom course. It is possible that students self-selected appropriately based on e-41 HALE, MIRAKIAN, AND DAY, v Pharmacology Course

7 their own understanding of their learning style preferences. As noted in other studies, the online students tended to be older and had more experience with online courses. 13 It is also possible that some students selected the online course due to the common misperception that it would be easier or were forced into an online course due to schedule conflicts. A small percentage of students, <2%, were repeating the course, thus some cross contamination may have occurred. The instructor noted that approximately 3% - 5% of classroom students would leave class early each week after the quiz. These students, therefore, did not attend the face-to-face lectures but would view the video lectures online instead; thus these students may represent an overlap between classroom and online methodologies. Students who tend to be most successful in online courses are those with a compelling reason to take the course online and individuals who are self-directed and self-motivated. 7 Several studies suggest that drop rates in online courses are generally higher than classroom courses. 13 This study found no statistically significant difference. Because this is a required prerequisite, it is likely that students were more motivated to succeed, remain engaged, and complete the course regardless of the course format. Thus, these results may not be generalizable to all online courses. CONCLUSION The online and classroom pharmacology courses had similar withdrawal rates and exam scores, indicating similar learning gains. Overall, both courses had high student satisfaction ratings in all criteria measured. However, students in the online course were less satisfied with 8 criteria related to student satisfaction with instructor rapport, course excellence, peer interaction, and self-perceived knowledge gains. REFERENCES 1. The Pocket Guide to U.S. Higher Education Washington, DC: EDU- CAUSE; US Department of Education, National Center for Education Statistics. Distance Education at Degree-Granting Postsecondary Institutions: Washington, DC: NCES , by Waits T and Lewis D. Project Officer: Greene B; Day DB, Hale LS. Promoting critical thinking in online physician assistant courses. Perspective on Physician Assistant Education. 2006;16(11): Chumley-Jones HS, Dobbie A, Alford CL. Web-based learning: sound educational method or hype? A review of the evaluation literature. Academic Medicine. 2002;77(10):S Ramage T. The "No Significant Difference" phenomenon: a literature review. E-Journal of Instructional Science and Technology. 2001;5(1). 6. University's website. Available at: Accessed Aug 18, Are distance-learning courses for me? Center for Independent Learning Web site. Available at: Accessed Oct 2, Tucker S. Distance education: better, worse, or as good as traditional education? Journal of Distance Learning Administration. 2001;4(4). 9. Canfield A. Learning Styles Inventory Manual Ann Arbor, MI: Humanics Media; IDEA Survey. Kansas State University IDEA Web site. Available at: Accessed Oct 25, Christopher MM, Thomas JA, Tallent-Runnels MK. Raising the bar: encouraging high level thinking in online discussion forums. Roeper Review. 2004;26(3): DeLoach SB, Greenlaw SA. Do electronic discussions create critical thinking spillovers? Contemporary Economic Policy. 2005;23(1): Diaz DP. drop rates revisited. The Technology Source Archives. May/June 2002 available at online_drop_rates_revisited. Accessed July 25, Journal of Allied Health, Summer 2009, Volume 38, Number 2 e-42