FACULTY AND STUDENT ASSESSMENT OF TECHNOLOGY: VIDEO-CONFERENCING Institutional Research Fall 2008
INTRODUCTION Fall 2008 marked the second semester of the video conferencing of courses between the University of South Florida Sarasota Manatee (USF SM) main campus and the USF Sarasota Manatee at MCC Venice South County site. 5 courses from the criminology, English, management, and marketing departments were selected to deliver the courses. This semester all courses were live on the USF SM main campus and beamed to the South County site. Each instructor was to hold class live on the remote site four times during the semester. METHOD of ASSESSMENT Instructors were asked to participate in an interview with Institutional Research to provide their insights and perspectives on incorporating the technology in the class room. Interviews were held during the final two weeks of their course. Questions were developed prior to the interview and all faculty members were questioned on the same topics. (To view the questions asked, see Appendix A.) The interviews lasted no more than forty minutes and were conducted face to face and over the telephone. Students were given an in class survey to complete during the final two weeks of the semester. Both scaled questions and open ended questions were used on the surveys. Some demographic information was collected as well. To view the survey, see Appendix B. SUMMARY The purpose of this assessment was to continue to investigate the experiences of the instructors and students participating in the video conferencing of courses through the USF SM campus. Comments on the experience of delivering and taking these courses were nearly the same as those received after the summer semester. Findings indicated that student benefits included (1)saving gas, money, and time by staying at the campus closest to their home/work, (2)more students served with the opening of the additional section, when the course may not have been held otherwise, (3)closer replica of in class instruction than that of online course offerings. Disadvantages included: (1)technology glitches and placement of equipment causing distractions to teaching and learning, (2)instructor difficulties in engaging remote site students, and (3)problems related to the use of different media types such as DVDs and PowerPoints. Fall results showed a drop in student likelihood of recommending these types of courses to others (summer: 74%, fall: 64%) and in the meeting of student expectations of the courses (summer: 93%, fall: 88%). It is evident that improvements to the equipment and a facilitation of instructor comfort with the technology will be needed for these courses to be successful.
FACULTY ASSESSMENT All agreed that video conferencing is a cost effective way of expanding course selection on the South County site, however, they also feel as though technology is not up to par to deliver it effectively. In addition, most faculty agreed that additional practice and training or assistance in the classroom could improve their delivery of the video conferenced course. Faculty Concerns/Comments: Remote Site Engagement The issue of remote site engagement was of major concern to the faculty delivering the videoconferenced courses both this semester and in the summer. Comments made by the faculty suggest that they are finding it difficult to gauge whether remote site students understand lessons because of the loss of eye contact and personal interaction which they usually use to infer student learning. Part of the problem, mentioned by those with classes in room A205, is the camera and television placement. All faculty agree that it is necessary for them to see both classrooms in one glance and less important for the classes to see each other. Also, camera placement should be as such that neither site is left looking at the back of the instructor. Finally, the quality of audio and video are negatively impacting the communication between the campuses. Faculty members also advised that more work needs to be done on their part. The flow, interaction, and timing of a class are changed by this type of technology. They stated that it takes a concerted effort on their part to remember to engage the remote site, to look into the cameras, and to watch the monitors. One faculty member suggested that earlier visits to the remote site could also help to resolve the issue of engagement. Technical Issues Audio quality continues to be an issue for some. Some stated that the sound is often garbled and at times lost completely. It was suggested that two televisions should be placed in B240 as the projected picture is often blurry. As in the summer, faculty were concerned with the difficulty of, and technical glitches they encountered, when attempting to use different media such as power points or movies. A television should be placed in the back of room A205. Faculty Insights on Student Perceptions Faculty agree that students appreciate course offerings closer to their homes/work. Faculty found students to be indifferent to the technology unless issues arose. They found that they quickly lost students attention when glitches were encountered. Faculty felt that students at both sites were on par in learning, as evidenced in their comparable scores. What Is Working Faculty all agree that the support staff and technicians are helpful and responsive. Blackboard is an effective tool for facilitating discussions between the classes. There were no reported problems in the delivery of in class materials or the administering of exams.
Additional Comments Keep these types of courses to one day a week, block schedules as there already inherent barriers to engaging to the students and the flow and timing of these classes. Faculty disagreed over the need and frequency of attending the remote site. One faculty suggested that it was not necessary at all, while another stated that it was important to get the remote site early to become familiar with the students and for the students to feel comfortable with the instructor. STUDENT ASSESSMENT Response Rate 82 of the 160 students enrolled in the course sections completed the survey. Table I Response Rate n Overall 51% 82:160 Remote Site 40% 17:42 Face to Face Site 55% 65:118 Student Profile (For a complete listing of demographic information see Appendix C) The majority of students : Had not previously taken a video conferenced course 77% Were male in gender 55% Were age 21 24 years old 54% Most often listed by students: Attending the Sarasota Manatee Campus, Sarasota as their city of residence 43% Attending the South County Campus, Venice as their city of residence 29% Attending the Sarasota Manatee Campus, Sarasota as the city in which they work. 38% Attending the South County Campus, evenly Englewood and Venice as the city in which they work 24% Student s College Percentage of Respondents CAS 43% COB 54% COE 0% SHRM 3% 34% of the respondents listed themselves as Criminology majors. For a full breakdown of majors, see Appendix C.
Student Perceptions Students were asked to answer the following questions using a scale of 1 4, 0 Strongly Disagree (SD), Disagree (D), Agree (A), Strongly Agree (SA), or Don t know (DK). Student perceptions of video conferencing varied as shown by the standard deviations calculated for the scale question means. However, in all instances, the majority of the students strongly agreed/agreed with the scale statements. Students most strongly agreed, in both summer and fall, that the courses met their expectations and that the instructors were effective at delivering the courses. By review of the scales scores, students in the fall semester appeared to be less satisfied than those in the summer. All items to be rated received lower scores in the fall, except the video quality which did improve slightly. However, only the question regarding technicians resulted in a statistically significant difference from the summer to the fall. Of special concern is the drop from summer to fall in the number of students strongly agreeing/agreeing that they would recommend video conferenced courses. % SA/A SUMMER 2008 Mean (Scale 1 4) St. Dev 86% 3.32.819 76 80% 3.25.802 76 68% 3.07.924 72 71% 3.03.850 73 67% 2.92.924 73 92% 3.50.646 74 84% 3.55.661 66 % SA/A n* % SA/A 1a) I believe I learned as much in this class as I would have in a class that did not use video conferencing. 1b) I felt as free to ask questions and make comments in this class as I do in classes that do not use videoconferencing. 1c) I felt my experience at the videoconference side was equivalent to my experience at the live end. 1d) The video images were clear & comprehensible when the instructor was remote. 1e) The audio was clear & comprehensible when the instructor was remote. 1f) The instructor was effective at delivering this video conferenced course. **1g) All technicians were effective and helpful. FALL 2008 Mean (Scale 1 4) St. Dev n* 81% 3.14.780 79 79% 3.13.795 78 63% 2.82.864 78 74% 3.07.781 71 64% 2.90.936 73 88% 3.30.726 78 85% 3.29.749 75 93% 3.43.681 75 1h) This course met my expectations. 88% 3.23.746 80 1i) I would recommend videoconferenced courses to others. 74% 3.04.879 72 64% 2.89.919 71 *Those that answered Don t Know were not included in mean score or standard deviation. n= those that answered Strongly Disagree through Strongly Agree. **Statistically significant difference between summer and fall mean scale scores. t test for independent samples ρ<.05
The majority of respondents, 66% (54:82), stated that they would prefer to attend a video conferenced course site than drive a further distance to attend the live course site. (Q4) STUDENT COMMENTS Student perceived advantages Being able to take classes close to home saves gas, money, and time. Serve more students in different locations. Retain some of the elements of a live class, as opposed to an online course. Student Perceived Needed Improvements Need better placement of cameras and television screens (room A205). Remote site should not be looking at the class, it is distracting. Also, the professor has to turn to look at the remote students. o Instructor should always be facing the camera. o Instructor should be able to look at the screen and into the camera at the same time. Video needs improvement. o Should be shown in HD o Screen/wall projection is not clear. Watching videos in class or using computer applications seemed to be a hassle. Either make the technology easier to use or train the instructors better. o Quality of the PowerPoints was terrible, very hard to read. o Couldn t communicate during PowerPoints. Audio was not clear, need to improve microphones. o Intermittent and garbled. o Professor should have own microphone to cut down on background noise. o Have a microphone on each table it might sound better than the single mic. o Instructors need to vocalize over the mic better. Volume is an issue. Simplify for the non tech savvy instructor. Instructors also need to be trained better. Have teaching assistants or non student proctors at the remote end. Instructors need to engage the remote end. It needs to be smoother. There are always technical issues. Additional Student Comments This type of technology could also work as an online class. Video tapes of the live class could be viewed by students from home. Allow time for remote students to have private conversations with the instructor before or after class. Otherwise, all students can hear. I feel as though I learn just as well this way. It was a great experience, interesting to experience the technology. Should not have these classes at all. Video conference courses are impersonal and have many issues. Students talk and leave class when the instructor is remote. Instructors should attend the remote site more often.
APPENDIX A. Questions to Instructors B. Student Video-conference Course Survey C. Complete Statistical Analysis D. Improvement made for Spring 2009 semester E. Planned Future Improvements
Appendix A: QUESTIONS TO FACULTY ASSESSMENT OF VIDEO CONFERENCING TECHNOLOGY SUMMER 08 TECHNICAL SUPPORT Did you feel you received adequate technical support? Did you feel you received adequate training? If you do not feel like you received adequate training how could this be improved? Were there any aspects of the technology, such as the video or audio that you identified as needing improvement? THE STUDENTS Did you feel resistance from the students? If so, did you feel this more from the remote site or the main site? Do you feel there are any added benefits to students by using video conferencing? Were student performances on tests, work assignments and/or class participation comparable at both sites? Overall, what do you feel students reactions were to the video conferencing of the course? OVERALL What was your overall comfort level with various technologies, not just video conferencing, before this semester? What is your overall comfort level now after implementing the video conferencing technology? What did you like the most about incorporating video conferencing into your course? How many additional hours of preparation were needed, weekly and before the start of the course? What challenges or barriers did you face when incorporating video conferencing into your course? Is there anything you would change about the way in which you organized the course to incorporate the videoconferencing aspect? Do you have any suggestions for others who may attempt to use this technology? Do you have any suggestions for administrators? Anything else you would like add
APPENDIX B: STUDENT VIDEO CONFERENCE COURSE SURVEY DIRECTIONS: To help the USF Sarasota Manatee Campus better serve you the student, please take a moment to provide feedback on your experience with video conferencing this semester. Though you may have had the majority of your class sessions face to face with the instructor, answer the following based on when you were on the receiving end of the video conference. Thank you. 1. Please indicate whether you AGREE or DISAGREE with the following statements, if you do not have enough information to judge, mark DON T KNOW. 1 2 3 4 0 Strongly Disagree Disagree Agree Strongly Agree Don't Know a. I believe I learned as much in this class as I would have in a face to face class. b. I felt as free to ask questions and make comments in this class as I do in face to face classes. c. I felt my experience in the course was equivalent to the experience of those in the live class. d. The video images were clear & comprehensible when the instructor was remote. e. The audio was clear & comprehensible when the instructor was remote. f. The instructor was effective at delivering this videoconference course. g. All technicians were effective and helpful. h. This course met my expectations. i. I would recommend video conferenced courses to others. 2. Is this the first video-conference course you have taken? Yes No 3. Which is your preferred USF site? South County MCC Venice Sarasota-Manatee Main Campus 4. Which option would you be more likely to do: a. Drive a further distance to attend a face-to-face course OR_ b. Attend a video-conferencedd course closer to your home/work. 5. Your City of Residence: Zip Code: 6. City Where You Work: Zip Code: 7. Gender: Male Female 8. Age: Under 21 21-24 25-29 30-39 40-49 50-59 60+ 9. College: Arts & Sciences Business Education Undergraduate Studies 10. Major: 11. Level: Junior Senior Master s Non-Degree PLEASE TURN OVER
12. What are the advantages of video-conferenced courses? 13. What improvements could be made to video-conference courses? 14. Please feel free to provide any additional comments about video-conferencing.
Appendix C: Complete Statistical Analysis of Student Survey Question % Strongly Agree (4) 1a) I believe I learned as much in this class as I would have in a class that did not use videoconferencing. 1b) I felt as free to ask questions and make comments in this class as I do in classes that do not use video conferencing. 1c) I felt my experience at the video conference side was equivalent to my experience at the live end. 1d) The video images were clear & comprehensible when the instructor was remote. 1e) The audio was clear & comprehensible when the instructor was remote. 1f) The instructor was effective at delivering this videoconferenced course. 1g) All technicians were effective and helpful. 1h) This course met my expectations. 1i) I would recommend videoconferenced courses to others. % Agree (3) % Disagree (2) % Strongly Disagree (1) % Don t Know (0) N w/dk Mean Score (Scale 1 4) St. Dev N w/out DK 33% 48% 12% 4% 3% 81 3.14.780 79 33% 46% 14% 4% 3% 81 3.13.795 78 22% 41% 27% 6% 4% 81 2.82.864 78 27% 47% 12% 4% 10% 79 3.07.781 71 26% 38% 19% 9% 8% 80 2.90.936 73 43% 45% 7% 3% 2% 80 3.30.726 78 40% 45% 5% 4% 6% 80 3.29.749 75 37% 51% 7% 4% 1% 81 3.23.746 80 24% 40% 16% 9% 11% 80 2.89.919 71 *Those that answered Don t Know were not included in mean score or standard deviation. n= those that answered Strongly Disagree through Strongly Agree. Q2. Is this the first video conferenced course you have taken? Yes: 77% No: 23% Q4. Which option would you be more likely to do: a. 28% Drive a further distance to attend a face-to-face course ORb. 66% Attend a video-conferenced course closer to your home/work. c. 6% No Response
Students Attending Sarasota Manatee Campus CITY WORKING IN Bradenton LWDR Longboat Key North Port Osprey Parrish Port Charlotte Sarasota Not Working Bradenton 9 8 4 Holmes Beach 1 Lakewood Ranch 1 1 CITY OF RESIDNECE Myakka City 1 1 Nokomis 1 North Port 1 3 Palmetto 1 1 Parrish 1 Port Charlotte 1 Sarasota 3 1 2 1 13 8 Venice 1 *One respondent did not report city working in, only county Manatee. Residence in Myakka City. Students Attending South County Site CITY WORKING IN Englewood North Port Pt Charlotte Punta Gorda Venice Not Working Cape Haze 1 CITY OF RESIDENCE Englewood 2 Nokomis 1 North Port 2 1 Port Charlotte 2 1 1 Sarasota 1 Venice 4 1
Q7. Gender All Respondents (n=82) SM Campus (n=65) Sth County Campus (n=17) Male 55% 55% 53% Female 44% 43% 47% Not Reported 1% 2% 0% Q8. Age Range All Respondents (n=82) SM Campus (n=65) STH County Campus (n=17) Under 21 18% 17% 24% 21 24 54% 57% 41% 25 29 15% 12% 24% 30 39 6% 8% 0% 40 49 6% 5% 1% 50 59 1% 2% 0% 60+ 0% 0% 0% Q9. Major % of all (n=82) Accounting 11% Criminology 34% Economics 1% Q10. Student s College % of all (n=82) CAS 43% COB 53% SHRM 4% Finance 12% GBA 11% History 1% Int l Business 4% ISM 1% Q11. Level % of all (n=82) Junior 71% Senior 29% ISS 7% Management 5% Marketing 9% SHRM 4%
Appendix D: Improvements made for Spring 2009 semester Tutorial developed to familiarize faculty with use of the technology. A205 multiple programming changing to the touch panels to correct problems specifically with the sending of content within a video conference and the functionality within the video conference controls. A205 reconfigured microphones to reduce the amount of noise carried to the remote site. A205 added a second TV for professors (has been installed should be active by next week, need an s video splitter). A205 on site tech in the room during class. A205 on site tech is using xpanel to control room functions from a laptop (xpanel is a software version of the touch panel in the front of the room). A205 changed setting within the Polycom system (examples: changed monitor layout, changed sending content settings, etc). Venice multiple programming changing to the touch panels to correct problems specifically with the sending of content within a video conference and the functionality within the video conference controls. Venice changed microphone from table top to ceiling mic for better audio. Venice remote tech in Sarasota during class Venice remote tech is using xpanel to control room functions from a laptop (xpanel is a software version of the touch panel in the front of the room). Venice changed setting within the Polycom system (examples: changed monitor layout, changed sending content settings, etc). B240 changed microphone from table top to ceiling mic for better audio B240 replaced blown bulb on projector, now has better picture without needing to turn down lights. B240 changed setting within the Polycom system (examples: changed monitor layout, changed sending content settings, etc). B240 on site tech in the room during class. B240 on site tech is using wireless touch panel to control room function
Appendix E: Planned Future Improvements Locations B240 and Venice need to have the s video output and RGB output switched on the Polycom units, this will involve some programming changes as well as re wiring. This will allow the content to be displayed on the LCD projector in B240 and the Big TV in Venice at optimum quality. Currently the B240 location has the content being sent out of the RGB output which goes to the small TV in front of the room and the Venice location has the content being sent out of the s video to the large TV (quality is not optimal). Additional programming changes need to be made for B240 to correct some issues with touch panel sending the correct information to the Polycom unit and processor in the room. Additional programming needed for A205 and Venice to provide better feedback from button presses on the touch panels. Continued microphone adjustments in all locations to achieve optimal audio quality (no feedback, no tinning of voices, hearing yourself audio loop) this also might entail moving the microphones around (classroom placement). Continued lighting adjustments in B240 to improve classroom conditions (so you don t have to turn the lights down all the way to see the projector because then you can t see what you are writing in your notebook).