STUDENT ASSESSMENT OF TECHNOLOGY: VIDEO-CONFERENCING

Transcription

1 STUDENT ASSESSMENT OF TECHNOLOGY: VIDEO-CONFERENCING Institutional Research Fall

2 INTRODUCTION Summer semester 2008, USF Sarasota Manatee initiated the use of video conferencing technology in the classroom to broadcast courses simultaneously on the main campus in Sarasota and on the South County site in North Port. The Colleges of Arts & Sciences, Business, and Education have all taught courses using the technology. During the Fall 2011 semester all three Colleges conducted courses via video conferencing. METHOD of ASSESSMENT All students were given the opportunity to complete the paper and pencil questionnaire during class time. Both scaled questions and open ended questions were used on the surveys. Some demographic information was collected as well. To view the questionnaire itself, see Appendix B. SUMMARY The purpose of this assessment was to continue to investigate the experiences of the students enrolled in the video conferencing courses offered by USF Sarasota Manatee. Students in Fall 2011 rated the video conferencing technology higher on all items than compared to the previous semester. The following percentages represent those that agreed/strongly agreed with the comments. The comments have been sorted to start with those that received the highest mean score. Mean % Agree/ Strongly Agree All technicians were effective and helpful % The instructor was effective at delivering this video conferenced course % I felt as free to ask questions and make comments in this class as I do in classes that do not use video conferencing % The video images were clear & comprehensible when the instructor was remote % This course met my expectations % I believe I learned as much in this class as I would have in a class that did not use video conferencing % The audio was clear & comprehensible when the instructor was remote % I felt my experience at the video conference side was equivalent to my experience at the live end % I would recommend video conferenced courses to others % 2

3 POPULATION DESCRIPTION Response Rate 103 of the 153 or 67% of the students enrolled in the Fall 11 course sections completed the survey. Fall 2008 Fall 2009 Fall 2010 Fall 2011 OVERALL 51% (82:160) 74% (139:187) 87% (68:78) 67% (103:153) ATTENDING SARASOTA CAMPUS 55% (65:119) 77% (111:144) 97% (59:61) 68% (76:111) ATTENDING SOUTH COUNTY SITE 40% (17:42) 65% (28:43) 53% (9:17) 64% (27:42) Student Profile (For a complete listing of demographic information see Appendix B) The majority of respondents : Fall 08 Fall 09 Fall 10 Fall 11 Gender Female (45%) Female (56%) Male (50%) Female (66%) Age (54%) (44%) (49%) (46%) Attended Sarasota Campus (79%) Sarasota Campus (80%) Sarasota Campus (87%) Sarasota Campus (74%) Student s College Fall 08 Fall 09 Fall 10 Fall 11 CAS 43% 49% 0% 6% COB 54% 44% 94% 68% COE 0% 1% 0% 26% SHRM 3% 2% 5% 0% Majors Fall 11 top three majors reported were General Business Administration (28%), Elementary Education (26%), and Management (16%). 3

4 Q2. Is this the first video conference course you have taken? Yes, first video conferenced course taken. 100% 80% 79% 80% 74% 60% 40% 20% 0% All Attending Sarasota Attending North Port Fall 08 Fall 09 Fall 10 Fall 11 Q4. Which option are you more likely to do: a. Drive a further distance to attend a face-to-face course OR b. Attend a video-conferenced course closer to your home/work. Prefer to attend the video conference course. 100% 80% 67% 68% 63% 60% 40% 20% 0% All Attending Sarasota Attending North Port Fall 08 Fall 09 Fall 10 Fall 11 4

5 Student Perceptions Students were asked to answer the following questions using a scale of 1 4, Strongly Disagree (SD=1), Disagree (D=2), Agree (A=3), Strongly Agree (SA=4), or Don t Know (DK=0). All of the nine statements, In Fall 11, showed an increase in the mean scores over Fall 10. 1a) I believe I learned as much in this class as I would have in a class that did not use video conferencing Fall 08 Fall 09 Fall 10 Fall 11 % Strongly Agree/Agree Median Mean St. Dev N* Fall 08 81% Fall 09 73% Fall 10 72% Fall 11 81% **A Kruskal Wallis Test did not reveal a statistically significant difference in degree of agreement levels across the four semesters X 2 (3, n=383) = 6.593, p=

6 1b) I felt as free to ask questions and make comments in this class as I do in classes that do not use videoconferencing Fall 08 Fall 09 Fall 10 Fall 11 % Strongly Agree/Agree Median Mean St. Dev N* Fall 08 79% Fall 09 73% Fall 10 68% Fall 11 82% **A Kruskal Wallis Test revealed a statistically significant difference in degree of agreement levels across the three semesters X 2 (3, n=382) = , p=.005. Follow up Mann Whitney tests using the Bonferroni adjusted alpha comparing Fall 2011 to previous years revealed statistical significance between Fall 2010 and Fall 2011 and Fall 2009 and Fall 2011 (p<.017). 6

7 1c) I felt my experience at the video conference side was equivalent to my experience at the live class Fall 08 Fall 09 Fall 10 Fall 11 % Strongly Agree/Agree Median Mean St. Dev N* Fall08 63% Fall 09 48% Fall 10 51% Fall 11 63% **A Kruskal Wallis Test revealed a statistically significant difference in degree of agreement levels across the three semesters X 2 (3, n=376) = , p=.003. Follow up Mann Whitney tests using the Bonferroni adjusted alpha comparing Fall 2011 to previous years revealed statistical significance between Fall 2010 and Fall 2011 and Fall 2009 and Fall 2011 (p<.017). 7

8 1d) The video images were clear & comprehensible when the instructor was remote Fall 08 Fall 09 Fall 10 Fall 11 % Strongly Agree/Agree Median Mean St. Dev N* Fall 08 74% Fall 09 64% Fall 10 72% Fall % **A Kruskal Wallis Test revealed a statistically significant difference in degree of agreement levels across the three semesters X 2 (3, n=375) = , p=.000. Follow up Mann Whitney tests using the Bonferroni adjusted alpha comparing Fall 2011 to previous years revealed statistical significance between Fall 2010 and Fall 2011 and Fall 2009 and Fall 2011 (p<.017). 8

9 1e) The audio was clear & comprehensible when the instructor was remote Fall 08 Fall 09 Fall 10 Fall 11 % Strongly Agree/Agree Median Mean St. Dev N* Fall 08 64% Fall 09 55% Fall 10 65% Fall % **A Kruskal Wallis Test revealed a statistically significant difference in degree of agreement levels across the three semesters X 2 (3, n=377) = 9.552, p=.000. Follow up Mann Whitney tests using the Bonferroni adjusted alpha comparing Fall 2011 to previous years revealed statistical significance between Fall 2010 and Fall 2011 and Fall 2009 and Fall 2011 (p<.017). 9

10 1f) The instructor was effective at delivering this video conference course Fall 08 Fall 09 Fall 10 Fall 11 % Strongly Agree/Agree Median Mean St. Dev N* Fall 08 88% Fall 09 90% Fall 10 91% Fall 11 87% **A Kruskal Wallis Test revealed a statistically significant difference in degree of agreement levels across the three semesters X 2 (3, n=381) = 9.552, p=.023. Follow up Mann Whitney tests using the Bonferroni adjusted alpha comparing Fall 2011 to previous years revealed statistical significance between Fall 2010 and Fall 2011 (p<.017). 10

11 1g) All technicians were effective and helpful Fall 08 Fall 09 Fall 10 Fall 11 % Strongly Agree/Agree Median Mean St. Dev N* Fall 08 85% Fall 09 86% Fall 10 96% Fall 11 94% **A Kruskal Wallis Test revealed a statistically significant difference in degree of agreement levels across the three semesters X 2 (3, n=379) = , p=.001. Follow up Mann Whitney tests using the Bonferroni adjusted alpha comparing Fall 2011 to previous years revealed statistical significance between Fall 2009 and Fall 2011 and Fall 2010 and Fall 2011 (p<.017). 11

12 1h) This course met my expectations Fall 08 Fall 09 Fall 10 Fall 11 % Strongly Agree/Agree Median Mean St. Dev N* Fall 08 88% Fall 09 84% Fall 10 85% Fall 11 82% **A Kruskal Wallis Test did not reveal a statistically significant difference in degree of agreement levels across the three semesters X 2 (3, n=382) = 4.443, p=

13 1i) I would recommend video conference courses to others Fall 08 Fall 09 Fall 10 Fall 11 % Strongly Agree/Agree Median Mean St. Dev N* Fall 08 64% Fall 09 61% Fall 10 49% Fall 11 62% **A Kruskal Wallis Test revealed a statistically significant difference in degree of agreement levels across the three semesters X 2 (3, n=360) = , p=.017. Follow up Mann Whitney tests using the Bonferroni adjusted alpha comparing Fall 2011 to previous years revealed statistical significance between Fall 2010 and Fall 2011 (p=.017). *Those that answered Don t Know were not included in mean score or standard deviation. n= those that answered Strongly Disagree through Strongly Agree. **Statistical significance. Kruskal Wallis, p.05 13

14 FALL SEMESTER 2011 DIFFERENCES BY SITE Sarasota Mean Score (n=75) Respondent Site South County Mean Score (n=28) 1a) I believe I learned as much in this class as I would have in a class that did not use video conferencing b) I felt as free to ask questions and make comments in this class as I do in classes that do not use video conferencing.* c) I felt my experience at the video conference side was equivalent to my experience at the live class d) The video images were clear & comprehensible when the instructor was remote e) The audio was clear & comprehensible when the instructor was remote.* f) The instructor was effective at delivering this video conference course g) All technicians were effective and helpful h) This course met my expectations.* i) I would recommend video conference courses to others * Mann Whitney test revealed statistical significance (p<.05), for 1b) z= 2.570, p =.010; 1e) z= 2.138, p=.033; 1h) z= 2.653, p=.008. STUDENT COMMENTS Student comments focused mostly on technical glitches with the audio and video quality. As in the previous semesters, some student were frustrated by the technology, some had concerns about their ability to pay attention to a remote instructor, while some had no issues at all. The following is a summary of their comments. Aspects of Video conferencing Students Liked Class close to home/convenience Good alternative to an online course. Very similar to a face to face classroom experience. The faculty members. More students can take the class without overcrowding the room. Ability to mute other class during individual or group work. Aspects of Video conferencing Students Did Not Like Lack of opportunity to speak with the professor one on one when faculty were remote. Poor audio/video quality. o Sound delay. o Difficult to hear/understand audio. o Microphones pick up too much background noise. o Blurry images. o Delay in video feed when watching DVDs. Can t see the board if you are in the remote location. 14

15 Didn t feel a connection to the class or professor/less personal Distracting Watching a television instead of person in class/prefer face to face classes. Student Perceived Needed Improvements Improve audio and video quality o Adjust audio so student noise can t be heard as much. o Difficult to hear, volume should be louder. o Mic the professor to eliminate background noise. o Need better quality video. o Need bigger televisions. Instructors and Technicians o Train instructors on where cameras and microphones are placed in the room. o The instructors could be better educated on the technology so that the class runs more smoothly. o The technicians need to be more available in the class. There were some tech assistants that didn t do anything but talk on their cell phones. We had to spend a lot of unnecessary time at the beginning of class to get the video conference to work. However, if a technician was there this wouldn t have happened. Bigger classrooms. Have the other class on the large screen not the TV. I would prefer to see the instructor more than the other class. Do not have the class on the front board. There should be an easier way to get the professors attention when you are remote. Maybe the technician could signal the professor with a light. Additional Student Comments o Great class! Great technology! o I enjoyed the class. o I know we need technology but I am more of a hands on learner. o I think VC is fine as long as you make yourself pay attention. For me, I get more out of a face to face class. The way it is set up and the audio& video equipment are good. o It is very distracting to be watching the other class during lecture. o Professor Kosten is an amazing teacher. She is very helpful, respectful, understanding, and caring. She is also a very fun and effective teacher. o The reason this course was so successful was the professor. She is more than an excellent at her profession. Her personality and openness made it succeed. Most professors wouldn t succeed like Professor Gilbertson did. o I think the teacher makes the difference. A great teacher can make the experience just as fulfilling as a live class. 15

16 APPENDIX A. Student Video-conference Course Survey B. Complete Statistical Analysis C. Improvement made for Fall 2011 semester D. Previous Improvements Made 16

17 APPENDIX A: STUDENT VIDEO CONFERENCE COURSE SURVEY DIRECTIONS: To help the USF Sarasota Manatee Campus better serve you the student, please take a moment to provide feedback on your experience with video conferencing this semester. Though you may have had the majority of your class sessions face to face with the instructor, answer the following based on when you were on the receiving end of the video conference. Thank you. 1. Please indicate whether you AGREE or DISAGREE with the following statements, if you do not have enough information to judge, mark DON T KNOW. a. I believe I learned as much in this class as I would have in a face to face class. b. I felt as free to ask questions and make comments in this class as I do in face to face classes. c. I felt my experience in the course was equivalent to the experience of those in the live class. d. The video images were clear & comprehensible when the instructor was remote. e. The audio was clear & comprehensible when the instructor was remote. f. The instructor was effective at delivering this videoconference course Strongly Disagree Disagree Agree Strongly Agree Don't Know g. All technicians were effective and helpful. h. This course met my expectations. i. I would recommend video conferenced courses to others. 2. Is this the first video-conference course you have taken? Yes No 3. Which is your preferred USF site? North Port Sarasota-Manatee 4. Which option would you be more likely to do: a. Drive a further distance to attend a face-to-face course OR_ b. Attend a video-conferenced course closer to your home/work. 5. Your City of Residence: Zip Code: 6. City Where You Work: Zip Code: 7. Gender: Male Female 8. Age: Under Major: 10. Level: Undergraduate Master s Non-Degree PLEASE TURN OVER 17

18 11. What aspects of video-conferencing did you like? 12. What aspects of video-conferencing did you like? 13. What improvements could be made to video-conferencing courses? 14. Please feel free to provide any additional comments about video-conferencing. 18

19 Appendix B: Complete Statistical Analysis of Student Survey All Respondents Question % Strongly Agree (4) 1a) I believe I learned as much in this class as I would have in a class that did not use videoconferencing. 1b) I felt as free to ask questions and make comments in this class as I do in classes that do not use video conferencing. 1c) I felt my experience at the video conference side was equivalent to my experience at the live end. 1d) The video images were clear & comprehensible when the instructor was remote. 1e) The audio was clear & comprehensible when the instructor was remote. 1f) The instructor was effective at delivering this videoconferenced course. 1g) All technicians were effective and helpful. 1h) This course met my expectations. 1i) I would recommend videoconferenced courses to others. % Agree (3) % Disagree (2) % Strongly Disagree (1) % Don t Know (0) N w/dk Mean Score (Scale 1 4) St. Dev N w/out DK 35.9% 44.7% 14.6% 3.9% 1.0% % 40.8% 12.6% 3.9% 1.9% % 35.9% 32% 2.9% 2.0% % 55.3% 9.7% 1.9% 0% % 49.5% 14.6% 3.9% 1.0% % 40.8% 5.8% 3.9% 2.9% % 35.9% 1.9% 2.9% 1.0% % 41.7% 11.7% 5.8% 1.0% % 34.0% 20.4% 12.6% 4.9% *Those that answered Don t Know were not included in mean score or standard deviation. 19

20 Appendix B: Complete Statistical Analysis of Student Survey Majority of Time Remote Students Question % Strongly Agree (4) 1a) I believe I learned as much in this class as I would have in a class that did not use videoconferencing. 1b) I felt as free to ask questions and make comments in this class as I do in classes that do not use video conferencing. 1c) I felt my experience at the video conference side was equivalent to my experience at the live end. 1d) The video images were clear & comprehensible when the instructor was remote. 1e) The audio was clear & comprehensible when the instructor was remote. 1f) The instructor was effective at delivering this videoconferenced course. 1g) All technicians were effective and helpful. 1h) This course met my expectations. 1i) I would recommend videoconferenced courses to others. % Agree (3) % Disagree (2) % Strongly Disagree (1) % Don t Know (0) N w/dk Mean Score (Scale 1 4) St. Dev N w/out DK 37% 47% 16% 0% 0% % 41% 12% 3% 3% % 41% 31% 3% 0% % 47% 12% 0% 0% % 44% 16% 6% 3% % 35% 10% 0% 0% % 28% 3% 3% 3% % 50% 13% 9% 0% % 41% 9% 13% 6%

21 Appendix B: Complete Statistical Analysis of Student Survey Majority of Time Live Students Question % Strongly Agree (4) 1a) I believe I learned as much in this class as I would have in a class that did not use videoconferencing. 1b) I felt as free to ask questions and make comments in this class as I do in classes that do not use video conferencing. 1c) I felt my experience at the video conference side was equivalent to my experience at the live end. 1d) The video images were clear & comprehensible when the instructor was remote. 1e) The audio was clear & comprehensible when the instructor was remote. 1f) The instructor was effective at delivering this videoconferenced course. 1g) All technicians were effective and helpful. 1h) This course met my expectations. 1i) I would recommend videoconferenced courses to others. % Agree (3) % Disagree (2) % Strongly Disagree (1) % Don t Know (0) N w/dk Mean Score (Scale 1 4) St. Dev N w/out DK 35% 44% 14% 6% 1% % 41% 13% 4% 1% % 34% 33% 3% 1% % 59% 8% 3% 0% % 52% 14% 3% 0% % 44% 4% 6% 1% % 40% 1% 3% 0% % 38% 11% 4% 1% % 31% 25% 13% 4% Q2. Is this the first video conferenced course you have taken? All (n=103) SM Campus (n=76) Sth. Cty. Campus (n=27) a. Yes 79% 80% 74% b. No 21% 20% 26% Q4. Which option would you be more likely to do? All (n=103) SM Campus (n=76) Sth. Cty. Campus (n=27) a. Drive a further distance to attend a face-to-face course 29% 30% 26% 21

22 ORb. Attend a video-conferenced course closer to your home/work. 67% 68% 63% No Response 4% 2% 11% 22

23 Students Attending Sarasota Manatee Campus CITY WORKING IN Anna Maria Bradenton Duette Ellenton Lakewood Ranch Oneco Osprey Palmetto Sarasota Tampa Venice Bradenton Bradenton Beach 1 CITY OF RESIDENCE Holmes Beach 1 Palmetto 1 1 Parrish 1 1 Sarasota Tampa 1 Terra Ceia 1 Venice 2 1 Students Attending North Port CITY WORKING IN CITY OF RESIDENCE Arcadia Englewood Nokomis North Port Port Charlotte Punta Gorda Sarasota Venice Arcadia 1 Englewood Nokomis 1 1 North Port Port Charlotte Punta Gorda 1 Venice 3 23

24 Q7. Gender All Respondents (n=103) SM Campus (n=76) North Port Campus (n=27) Male 28% 29% 26% Female 66% 64% 70% Not Reported 3% 7% 4% Q8. Age Range All Respondents (n=103) SM Campus (n=76) North Port Campus (n=27) Under 21 6% 8% 0% % 42% 56% % 22% 18% % 15% 11% % 11% 11% % 1% 4% 60+ 0% 0% 0% Not Reported 1% 1% 0% Q9. Major % of all (n=103) Accounting 13% BSIT 1% Criminology 1% Early Childhood Ed 1% Elementary Ed 26% Finance 4% GBA 28% ISS 3% Q10. Student s % of all (n=103) College CAS 6% COB 68% COE 26% Q11. Level % of all (n=103) Undergraduate 100% Non Degree 0% Management 16% Marketing 8% 24

25 Appendix C: Improvements Made in 2011 B240 has a newer, larger secondary monitor. Mandatory training for faculty instituted. Tampa redesigned the touch screen panels for all VC rooms. Fixed Help Button issue. Appendix D: Previous Improvements Made B240 / A205 both have new Crestron systems. Many of the older components where replaced with identical parts to be used as spares while others where upgraded outright. Addition of a portable Videoconference unit for greater flexibility Upgraded Crestron processor (MC2E upgraded to MPS300) in B240 and A205. This eliminates the need for a Crestron 7x2 switch in the room. Wireless microphone in B240 with wireless receiver/transmitter. Two Polycom VSX 7000E s to replace the VSX 7000E s already in those rooms. This was intended to get extended service contracts; also would have spare units in case of hardware failure. NP has Nexia preamps for better audio control. Choir microphones in both NP VC rooms for improved audio quality. NP was installed with Crestron MPS300 s, updated TPS 6X touchpanels, long life CENUPS 1250 (affords roughly 2 hours of uptime in case of power failure). A205 and B240 also have this unit installed. NP has TANDBERG 990 MXP s for its VC units. Tandberg was picked over Polycom as USF as a whole is moving to Tandberg for its VC standard. A205, and all of NP has upgraded PT F300U Panasonic Projectors with 4,000 lumens. B240 is slated to get its Hitachi CP X445 replaced with the Newer PT F300U during spring break. Tutorial developed to familiarize faculty with use of the technology. A205 multiple programming changing to the touch panels to correct problems specifically with the sending of content within a video conference and the functionality within the video conference controls. A205 reconfigured microphones to reduce the amount of noise carried to the remote site. A205 added a second TV for professors (has been installed should be active by next week, need an s video splitter). A205 on site tech in the room during class. 25

26 A205 on site tech is using xpanel to control room functions from a laptop (xpanel is a software version of the touch panel in the front of the room). A205 changed setting within the Polycom system (examples: changed monitor layout, changed sending content settings, etc). Venice multiple programming changing to the touch panels to correct problems specifically with the sending of content within a video conference and the functionality within the video conference controls. Venice changed microphone from table top to ceiling mic for better audio. Venice remote tech in Sarasota during class Venice remote tech is using xpanel to control room functions from a laptop (xpanel is a software version of the touch panel in the front of the room). Venice changed setting within the Polycom system (examples: changed monitor layout, changed sending content settings, etc). B240 changed microphone from table top to ceiling mic for better audio B240 replaced blown bulb on projector, now has better picture without needing to turn down lights. B240 changed setting within the Polycom system (examples: changed monitor layout, changed sending content settings, etc). B240 on site tech in the room during class. B240 on site tech is using wireless touch panel to control room function 26