FACULTY AND STUDENT ASSESSMENT OF TECHNOLOGY: VIDEO-CONFERENCING. Institutional Research

Transcription

1 FACULTY AND STUDENT ASSESSMENT OF TECHNOLOGY: VIDEO-CONFERENCING Institutional Research Fall

2 INTRODUCTION Summer semester 2008, USF Sarasota Manatee initiated the use of video conferencing technology in the classroom to broadcast courses simultaneously on the main campus in Sarasota and on the South County site in Venice. The Colleges of Arts & Sciences, Business, and Education have all taught courses using the technology. METHOD of ASSESSMENT All students were given the opportunity to complete the paper and pencil questionnaire during class time. Both scaled questions and open ended questions were used on the surveys. Some demographic information was collected as well. To view the questionnaire itself, see Appendix B. Faculty members, teaching the video conferenced courses, were interviewed by the Office of Institutional Research staff. Interviews were conducted face to face and over the telephone. All faculty members were asked the same questions. Topics included: 1) perception of the technology; 2) technical and pedagogical issues; and 3) student responses. SUMMARY The purpose of this assessment was to continue to investigate the experiences of the students enrolled in and the faculty members teaching the video conferencing courses offered by USF Sarasota Manatee. The Fall 2009 assessments showed slight improvements, but overall ratings and comments remained less positive than those collected in All faculty members interviewed stated they had problems with the equipment functioning properly. Specifically, they had issues with audio and video quality and loss of connections between the sites. The majority of student comments collected also showed a need for improvements to equipment, especially the audio equipment. Students commented that they found it difficult to remain alert and attentive when the professor was remote and the instructors stated they had difficulty keeping their remote classes engaged. Students on the South County site were more positive than the Sarasota campus students in their ratings of the technology. 79% of South County students, in comparison to 45% of Sarasota students, stated they would chose to attend a video conferenced course over driving a further distance to attend a live class. The South County students showed significantly higher agreement with the following statements over their Sarasota counterparts: 1) I believe I learned as much in this class as I would have in a class that did not use video conferencing; 2) I felt my experience at the video conferenced end was equivalent to the experience of the live class; 3) The audio was clear & comprehensible; and 4) I would recommend video conference courses to others. When looking at the assessment responses overall it would appear that the perception of the students and faculty members involved in the video conference courses is one of frustration. Almost all seemed to champion the idea of broadcasting courses between the sites, but almost all also shared their aggravations and view of struggling to teach or learn while dealing with frequent technical glitches and failings. 2

3 FACULTY ASSESSMENT All faculty members agreed that video conferencing is a cost effective way of expanding course selections. However, most stated that glitches and issues with the technology hampered the effectiveness of the classroom delivery. There was a strong sense that the faculty members believed that use of the technology should be halted until an upgrade to the system could be made and that all issues involving connections, audio and video quality, and the display of PowerPoint slides can be resolved. Most stated they would attempt to use the technology again but only if the class could run seamlessly and that both ends of the connection could participate in the class as if they were in one room. Technical Issues Dropped Audio/Video, Time to connect/re connect Most frustrating to the faculty was the loss in connections of both audio and video throughout the semester. One instructor stated that he lost connection to the remote classroom at least once during but as many as three times during each of his class periods. Audio Each faculty member interviewed had experienced an issue with the audio equipment including feedback, tinning of voices, and distractions from amplified classroom noises (paper shuffling, chairs moving, etc). Fixes applied to resolve these issues involved the muting of the remote classroom (the room in which the faculty member was not physically present). Though this helped it also put the remote site at a disadvantage as the students in that room had to somehow get the attention of the instructor to take them off mute when they wished to participate in a discussion or to ask a question. One instructor stated that a student in his class drew a sign that read We re Muted in order to get the instructor s attention. In addition, students complained to faculty that classmates were be loud and distracting when the room was muted. Instructors stated they found it difficult to keep both classes engaged when one side could not fully participate. Incorporating Media Most of the instructors interviewed stated their lesson plans include the use of PowerPoint, movies, overhead projections, etc. and that the video conferencing technology inhibited their use of these medias. When displaying PowerPoint slides, the remote class would see PowerPoint slides on one monitor and themselves on the other monitor, the class is unable to view the instructor. When both the PowerPoint slides and the instructor were made visible to the remote site the instructor lost visual of the remote class. Faculty members felt this was disruptive to class discussions and added to the difficulties in engaging the remote site. Training Half of the faculty members interviewed felt as though they could have benefited from additional training before taking on the video conferenced classroom. Most felt that practice time in front of the camera would have helped their comfort level and that they would have liked to have been given one on one time with a peer who had experience in using the technology. The other half felt training was adequate since technicians were present in each class to control the technology. 3

4 Remote Site Engagement Engaging the remote site continues to be a topic of issue for faculty members. All say the onus is on them to ensure the remote site is involved in discussions and is given chances to ask questions. Most of the instructors stated they specifically ask one site or the other to answer a question or to provide comments. Faculty Insights on Student Perceptions The faculty members had differing comments on the aspect of student perception of the technology. Some found that the remote site (the Sarasota campus was the main teaching site for each course taught during the Fall 09 semester, thus the South County site is referenced as the remote site) were not as engaged and did poorer on exams and in discussions. However, others stated they found the remote site more diligent and that there was more of a sense of community amongst the South County students, perhaps, as one instructor commented, because they had to try harder to become involved in the class. Suggestions/Tips for: Other faculty members o o o o o o Do not be afraid of the technology. It isn t as hard as it looks. Do not make heavy use of PowerPoint. Provide these and other media through your Blackboard site. Keep the class interactive to ensure that the remote site stays engaged OR provide straight lecture to avoid having issues with the turning on and off microphones. Do not teach a course for the first time using this technology. Keep to courses you have more experience teaching. Speak with a faculty member that has experience using video conferencing. They will likely have great advice to share. You should materials to South County; mailing is too slow and items may not arrive in time. Administrators o New faculty members should not be selected to conduct classes via video conferencing during their first semesters. The burden of this type of class may be overwhelming during the period of adjustment to a new campus. o Introductory courses may not be the best selection to be delivered through videoconferencing. Students in these classes are often in need of more attention and are at greater risk of losing motivation. Resident faculty should be providing one on one interaction. o Room B240 on the Sarasota campus was found to be inadequate for administering exams. Students had to be seated too close together despite only 15 being in the room. o A procedure for proctoring exams is needed. The faculty member can only be at one campus at one time. Technicians, who are present in the Sarasota classroom every session, had to fill in on some occasions. 4

5 o Don t be afraid to take this as a loss and move on. This is not rocket technology, it should function. A whole new system is needed since tweaks and patches are not working. o The technical glitches, the disconnections, and the poor audio quality are reflecting badly on the campus as a whole. It is embarrassing for the instructors and the institution. o Is course delivery through elluminate to remote sites a possible alternative to videoconferencing? o Be careful of times when scheduling these courses. South County s staff does not start until 9am, thus classes on that site should not start until at least 9:30am. o A better description of the course delivery is needed in the schedules. Though students know that the course is being viewed by a remote site they are often unaware that multiple times throughout the semester they themselves become the remote site. o Instructors did not agree on how many times one should visit the remote site. Some argued that there are usually only a few students attending the South County class and so they should be spending their time at the Sarasota campus where the majority of students are located. Other instructors argued that it is important to split the classes in which the professor is present equally between the sites. They find that they can better engage students by building connections through face to face interactions. POPULATION DESCRIPTION Response Rate 139 of the 187 or 74% of the students enrolled in the Fall 09 course sections completed the survey. Summer 2008 Fall 2008 Spring 2009 Fall 2009 OVERALL 76% (76:100) 51% (82:160) 65% (103:158) 74% (139:187) ATTENDING SARASOTA CAMPUS 78% (62:80) 55% (65:119) 63% (72:115) 77% (111:144) ATTENDING SOUTH COUNTY SITE 67% (14:20) 40% (17:42) 67% (29:43) 65% (28:43) 5

6 Student Profile (For a complete listing of demographic information see Appendix B) The majority of students : Summer 08 Fall 08 Spring 09 Fall 09 Were female in gender 67% 45% 58% 56% Were age years old 57% 54% 51% 44% Attended the Sarasota Manatee Campus 82% 79% 70% 80% Student s College Summer 08 Fall 08 Spring 09 Fall 09 CAS 21% 43% 35% 49% COB 37% 54% 55% 44% COE 41% 0% 6% 1% US 1% 3% 2% 2% Majors Fall 09 top three majors reported were Psychology (31%), General Business Administration (14%), and Criminology (11%). Q2. Is this the first video conference course you have taken? Yes, first video conferenced course taken. 100% 90% 80% 70% 60% 50% 40% 30% 20% 10% 0% 76% 80% 61% All Attending SM Campus Attending Sth. County Summer 08 Fall 08 Spring 09 Fall 09 6

7 Q4. Which option are you more likely to do: a. Drive a further distance to attend a face-to-face course OR b. Attend a video-conferenced course closer to your home/work. Prefer to attend the video conference course. 100% 80% 79% 60% 40% 20% 52% 45% 0% All Attending SM Campus Attending Sth. County Summer 08 Fall 08 Spring 09 Fall 09 Student Perceptions Students were asked to answer the following questions using a scale of 1 4, 0 Strongly Disagree (SD), Disagree (D), Agree (A), Strongly Agree (SA), or Don t Know (DK). Six of the nine statements, in Fall 09, showed an increased in mean scores over Spring 09 scores. However, none of the differences were statistically significant. As in Spring 09, Fall 09 scores remained lower than those calculated in Summer 08 and Fall 08, (see notes for significance). 1a) I believe I learned as much in this class as I would have in a class that did not use video conferencing. % Strongly Agree/Agree Median Mean St. Dev N* Summer 08 86% Fall 08 81% Spring 09 67% Fall 09 73% **A Kruskal Wallis Test revealed a statistically significant difference in degree of agreement levels across the four semesters X 2 (3, n=392) = 11.95, p=

8 1b) I felt as free to ask questions and make comments in this class as I do in classes that do not use videoconferencing. % Strongly Agree/Agree Median Mean St. Dev N* Summer 08 80% Fall 08 79% Spring 09 76% Fall 09 73% **A Kruskal Wallis Test did not reveal a statistically significant difference in degree of agreement levels across the four semesters X 2 (3, n=392) = 7.61, p= c) I felt my experience at the video conference side was equivalent to my experience at the live class. % Strongly Agree/Agree Median Mean St. Dev N* Summer 08 68% Fall08 63% Spring 09 44% Fall 09 48% **A Kruskal Wallis Test revealed a statistically significant difference in degree of agreement levels across the four semesters X 2 (3, n=379) = 21.88, p= d) The video images were clear & comprehensible when the instructor was remote. % Strongly Agree/Agree Median Mean St. Dev N* Summer 08 71% Fall 08 74% Spring 09 54% Fall 09 64% **A Kruskal Wallis Test revealed a statistically significant difference in degree of agreement levels across the four semesters X 2 (3, n=376) = 19.45, p=

9 1e) The audio was clear & comprehensible when the instructor was remote. % Strongly Agree/Agree Median Mean St. Dev N* Summer 08 67% Fall 08 64% Spring 09 43% Fall 09 55% **A Kruskal Wallis Test revealed a statistically significant difference in degree of agreement levels across the four semesters X 2 (3, n=379) = 19.00, p= f) The instructor was effective at delivering this video conference course. % Strongly Agree/Agree Median Mean St. Dev N* Summer 08 92% Fall 08 88% Spring 09 87% Fall 09 90% **A Kruskal Wallis Test revealed a statistically significant difference in degree of agreement levels across the four semesters X 2 (3, n=388) = 9.58, p= g) All technicians were effective and helpful. % Strongly Agree/Agree Median Mean St. Dev N* Summer 08 84% Fall 08 85% Spring 09 87% Fall 09 86% **A Kruskal Wallis Test revealed a statistically significant difference in degree of agreement levels across the four semesters X 2 (3, n=377) = 13.53, p=

10 1h) This course met my expectations. % Strongly Agree/Agree Median Mean St. Dev N* Summer 08 93% Fall 08 88% Spring 09 82% Fall 09 84% **A Kruskal Wallis Test revealed a statistically significant difference in degree of agreement levels across the four semesters X 2 (3, n=390) = 19.04, p= i) I would recommend video conference courses to others. % Strongly Agree/Agree Median Mean St. Dev N* Summer 08 74% Fall 08 64% Spring 09 45% Fall 09 61% **A Kruskal Wallis Test revealed a statistically significant difference in degree of agreement levels across the four semesters X 2 (3, n=367) = 19.96, p=.000. *Those that answered Don t Know were not included in mean score or standard deviation. n= those that answered Strongly Disagree through Strongly Agree. **Statistical significance. Kruskal Wallis, p.05 10

11 FALL SEMESTER 2009 DIFFERENCES BY SITE* Sarasota Mean Score (n=111) Respondent Site South County Mean Score (n=28) 1a) I believe I learned as much in this class as I would have in a class that did not use video conferencing.* b) I felt as free to ask questions and make comments in this class as I do in classes that do not use video conferencing c) I felt my experience at the video conference side was equivalent to my experience at the live class.* d) The video images were clear & comprehensible when the instructor was remote e) The audio was clear & comprehensible when the instructor was remote.* f) The instructor was effective at delivering this video conference course g) All technicians were effective and helpful h) This course met my expectations i) I would recommend video conference courses to others.* * Mann Whitney test revealed statistical significance (p<.05), for 1a) z= 2.114, 1c) z= 2.531, 1e) z= 2.648, 1i) z= STUDENT COMMENTS The majority of comments made by the students involved issues with technical glitches and audio and visual quality. Some stated that despite these issues they really enjoyed their class and that they saw the need for this type of technology. Others, however, we less positive about their experiences and found the technology distracting and frustrating. The following is a list of student responses to the question, What improvements could be made to videoconference courses? Student Perceived Needed Improvements Audio and video quality was inconsistent. It was difficult to hear the other classroom and the instructor they were remote. o The sound quality is very bad. It is sometimes very difficult to hear the professor and most of the time impossible to hear the other students. o We had to turn toward the microphone and yell for the other class to hear us. o Very difficult to see slides when professor was in Venice. Both sites should be able to see the instructor and the PowerPoint slides at the simultaneously. The loss of connections between the sites was a waste of class time and became a distraction to learning. The instructors should be better trained and not have to rely on the technicians so much. Things might go smoother if the instructor was in control. Some techs surfed the internet instead of paying attention to the process of delivering the course. They should be focused on what the professor is 11

12 doing, assessing the quality of sound and video, and should not be waiting for the professor to ask them to do something. Additional Student Comments When instructor is remote it is difficult to pay attention, the class is less enjoyable. We are charged the same amount or more per credit hour and yet receive an inferior course experience. The instructor is unable to present material efficiently and all students are penalized. I appreciated our instructor switching between campuses each week. It made it more equal. If given the choice, I would prefer to watch the class from my home computer than watch a screen in a classroom. I found it difficult to ask the instructor questions before and after class. The IT guys seem to do a nice job. The VC was executed better than I expected, but I think it is easier to pay attention when the instructor is in the room. 12

13 APPENDIX A. Student Video-conference Course Survey B. Complete Statistical Analysis C. Improvement made for Spring 2009 semester D. Planned Future Improvements 13

14 APPENDIX A: STUDENT VIDEO CONFERENCE COURSE SURVEY DIRECTIONS: To help the USF Sarasota Manatee Campus better serve you the student, please take a moment to provide feedback on your experience with video conferencing this semester. Though you may have had the majority of your class sessions face to face with the instructor, answer the following based on when you were on the receiving end of the video conference. Thank you. 1. Please indicate whether you AGREE or DISAGREE with the following statements, if you do not have enough information to judge, mark DON T KNOW. a. I believe I learned as much in this class as I would have in a face to face class. b. I felt as free to ask questions and make comments in this class as I do in face to face classes. c. I felt my experience in the course was equivalent to the experience of those in the live class. d. The video images were clear & comprehensible when the instructor was remote. e. The audio was clear & comprehensible when the instructor was remote. f. The instructor was effective at delivering this videoconference course Strongly Disagree Disagree Agree Strongly Agree Don't Know g. All technicians were effective and helpful. h. This course met my expectations. i. I would recommend video conferenced courses to others. 2. Is this the first video-conference course you have taken? Yes No 3. Which is your preferred USF site? South County MCC Venice Sarasota-Manatee Main Campus 4. Which option would you be more likely to do: a. Drive a further distance to attend a face-to-face course OR_ b. Attend a video-conferenced course closer to your home/work. 5. Your City of Residence: Zip Code: 6. City Where You Work: Zip Code: 7. Gender: Male Female 8. Age: Under Major: 10. Level: Undergraduate Master s Non-Degree PLEASE TURN OVER 14

15 13. What improvements could be made to video-conference courses? 14. Please feel free to provide any additional comments about video-conferencing. 15

16 Appendix B: Complete Statistical Analysis of Student Survey Question % Strongly Agree (4) 1a) I believe I learned as much in this class as I would have in a class that did not use videoconferencing. 1b) I felt as free to ask questions and make comments in this class as I do in classes that do not use video conferencing. 1c) I felt my experience at the video conference side was equivalent to my experience at the live end. 1d) The video images were clear & comprehensible when the instructor was remote. 1e) The audio was clear & comprehensible when the instructor was remote. 1f) The instructor was effective at delivering this videoconferenced course. 1g) All technicians were effective and helpful. 1h) This course met my expectations. 1i) I would recommend videoconferenced courses to others. % Agree (3) % Disagree (2) % Strongly Disagree (1) % Don t Know (0) N w/dk Mean Score (Scale 1 4) St. Dev N w/out DK 27% 46% 20% 4% 3% % 49% 22% 3% 2% % 36% 36% 10% 5% % 46% 27% 5% 4% % 43% 33% 9% 2% % 58% 4% 4% 2% % 54% 7% 4% 3% % 61% 9% 4% 3% % 49% 24% 11% 4% *Those that answered Don t Know were not included in mean score or standard deviation. 16

17 Q2. Is this the first video conferenced course you have taken? All (n=139) SM Campus (n=111) Sth. Cty. Campus (n=28) a. Yes 76% 80% 61% b. No 23% 20% 36% No Response 1% 0% 3% Q4. Which option would you be more likely to do? a. Drive a further distance to attend a face-to-face course OR- b. Attend a video-conferenced course closer to your home/work. All (n=139) SM Campus (n=111) Sth. Cty. Campus (n=28) 46% 54% 14% 52% 45% 79% No Response 2% 1% 7% 17

18 Students Attending Sarasota Manatee Campus CITY WORKING IN Anna Maria Bradenton Ellenton Ft. Myers Holmes Beach Lakewood Ranch Long Boat Key Myakka Osprey Palmetto Parrish Punta Gorda Sarasota St Pete Tampa Venice Wauchula Arcadia Bradenton Fort Myers 1 Myakka 1 1 CITY OF RESIDNECE Nokomis 1 North Port 2 Osprey 1 Palmetto 2 1 Parrish 2 Punta Gorda 2 Sarasota Venice Wauchula 1 Students Attending South County Site CITY WORKING IN Arcadia Bradenton CITY OF RESIDENCE Englewood Nokomis North Port Port Charlotte Punta Gorda Sarasota St. Pete Venice Englewood 2 Ft. Ogden 1 Nokomis 1 1 North Port Port Charlotte 1 Punta Gorda 1 2 Sarasota 1 Venice 4 18

19 Q7. Gender All Respondents (n=139) SM Campus (n=111) Sth County Campus (n=28) Male 43% 44% 39% Female 56% 55% 61% Not Reported 1% 1% 0% Q8. Age Range All Respondents (n=139) SM Campus (n=111) STH County Campus (n=28) Under 21 14% 14% 11% % 49% 25% % 20% 21% % 7% 21% % 9% 14% % 1% 7% 60+ 0% 0% 0% Q9. Major % of all (n=139) Psychology 31% GBA 14% Criminology/Crim Just 13% Accounting 9% Finance 7% Marketing 7% Business Management 4% ISS 4% Hotel Restaurant Mgmt 2% Int l Business 2% Q10. Student s % of all (n=139) College CAS 51% COB 44% COE 1% US 2% Not Reported 2% Q11. Level % of all (n=139) Undergraduate 99% Non Degree 0% No Response 1% Social Work 2% All others (1 reporting) 3% Not Reported 2% 19

20 Appendix C: Improvements made for Spring 2009 semester Tutorial developed to familiarize faculty with use of the technology. A205 multiple programming changing to the touch panels to correct problems specifically with the sending of content within a video conference and the functionality within the video conference controls. A205 reconfigured microphones to reduce the amount of noise carried to the remote site. A205 added a second TV for professors (has been installed should be active by next week, need an s video splitter). A205 on site tech in the room during class. A205 on site tech is using xpanel to control room functions from a laptop (xpanel is a software version of the touch panel in the front of the room). A205 changed setting within the Polycom system (examples: changed monitor layout, changed sending content settings, etc). Venice multiple programming changing to the touch panels to correct problems specifically with the sending of content within a video conference and the functionality within the video conference controls. Venice changed microphone from table top to ceiling mic for better audio. Venice remote tech in Sarasota during class Venice remote tech is using xpanel to control room functions from a laptop (xpanel is a software version of the touch panel in the front of the room). Venice changed setting within the Polycom system (examples: changed monitor layout, changed sending content settings, etc). B240 changed microphone from table top to ceiling mic for better audio B240 replaced blown bulb on projector, now has better picture without needing to turn down lights. B240 changed setting within the Polycom system (examples: changed monitor layout, changed sending content settings, etc). B240 on site tech in the room during class. B240 on site tech is using wireless touch panel to control room function 20

21 Appendix D: Planned Future Improvements Locations B240 and Venice need to have the s video output and RGB output switched on the Polycom units, this will involve some programming changes as well as re wiring. This will allow the content to be displayed on the LCD projector in B240 and the Big TV in Venice at optimum quality. Currently the B240 location has the content being sent out of the RGB output which goes to the small TV in front of the room and the Venice location has the content being sent out of the s video to the large TV (quality is not optimal). Additional programming changes need to be made for B240 to correct some issues with touch panel sending the correct information to the Polycom unit and processor in the room. Additional programming needed for A205 and Venice to provide better feedback from button presses on the touch panels. Continued microphone adjustments in all locations to achieve optimal audio quality (no feedback, no tinning of voices, hearing yourself audio loop) this also might entail moving the microphones around (classroom placement). Continued lighting adjustments in B240 to improve classroom conditions (so you don t have to turn the lights down all the way to see the projector because then you can t see what you are writing in your notebook). 21