Switching to on-line evaluations for courses at UC Berkeley

Transcription

1 Switching to on-line evaluations for courses at UC Berkeley Daphne Ogle, Lead Design, UC Berkeley Aaron Zeckoski, Lead Developer, Unicon June 10-15, 2012 Growing Community; Growing Possibilities

2 Speakers Daphne Ogle Berkeley Design Lead Aaron Zeckoski Unicon Development Lead OAE since 2010, Fluid Project, Sakai since 2004 Evaluation System original lead (2006), Evals software since 2001, Sakai since

3 "#$%"&'(%)*+& Unicon Profile! "#$%&'()*+,-&%#+),.)/0) *+,.#11&,'$2)1#+-&3#1).,+) 45#)#%63$7,')8$+9#4)! :,6'%#%)&');<<=)! >+($'&3$22?).6'%#%)$'%) A#$%B6$+4#+#%)&')) C&2D#+4E)FG)$'%)14$H#%) 45+,6(5,64)45#)IJ) 3

4 Unicon technology & domain expertise 0"+1)%2'! "#$%#&'()*+,-''! (.**,#%/0'1&2&$.+*&#/'! 3#/&45)6.#'7.8,-' *&)+,$,-'! 9:;'! (.#/&#/'85&)6.#')#<'<&$%2&50'! )--&--*&#/'!"#$%&'! =,/A&#68)/&<'+&5-.#)$%B&<' 8.#/&#/'! ;,++.5/')85.--'*,$6+$&'<&2%8&-' )#<'?5.C-&5-'! =++'-,?*%--%.#'-,++.5/'!&($)'! =<<'5%8AD*&<%)'8)+)?%$%6&-' /.G'! 9:;'! H.5/)$ 3(&,/14'),('566&22'!),)-&7&,1'! ;%#4$&'-%4#D.#'! 7&<&5)/&<'31'! :,$6DE)8/.5'),/A&#68)6.#'."%%)#"+)/",'! F*)%$'! ()$&#<)5%#4'! 1.8,*&#/-' 4

5 Unicon service capabilities / offerings "#$%&'(! )*+*,&-&+(!.$/0*#&! 1&2&3$4-&+(! 5-43&-&+*6$+! *+7!.844$#(!! "#$%&'$$( )*%+&,'&-(!.,/*','&-012&( 3*0&&%&+(! )+%*'(3425'$$6( 7'-829$(0&9( '$(! :2&-'&-( ;'<'*2/,'&-(! =-0>(7'&-24%&+(! )//*%5012&( ;'<'*2/,'&-((!?$'4(@A/'4%'&5'(! 324-*'-( ;'<'*2/,'&-(! B#0*%-C( )$$#40&5'(! 3'4D24,0&5'( E'$1&+(!!.&$-0**012&(0&9( :2&F+#4012&(! =C$-',(.&-'+4012&(! E'58&%50*(=#//24-(! 7%+4012&G?/+409'$(! H2$1&+(='4<%5'$(! =C$-',$G;0-0I0$'( )9,%&%$-4012&(! E40%&%&+(! H'*/(;'$J(G(=#//24-( 5

6 Today s topics Intro to UC Berkeley on-line course evaluation project Demo: Updates to CLE Evaluation System for Spring pilot Future work What we learned Questions & Answers 6

7 Introduction to online evaluations project at UC Berkeley Overall project initiative 7

8 Online evaluations for courses New shared questions campus-wide instructional format-based (14 formats) piloted first On-line pilot CLE Evaluation System development to meet Berkeley needs & overall system improvement Berkeley & Unicon collaboration 8

9 Goals of initiative Rapid access to useful information to help faculty improve their courses and their teaching effectiveness Improvements to the quality & integrity of data that the campus uses to understand and recognize teaching contributions 21st century tools for staff to perform their jobs efficiently and free up time for direct service to students and faculty High quality information for students about courses and instructors at Berkeley Cynthia Schrager, Assistant Vice Provost 9

10 Project time line Research new questions & instructional formats Design, analysis & development CLE eval sys (in UCB branch) Pilot CLE eval sys Design & development (intend to move to trunk) Campus buy-in Evaluation systems comparative analysis Pilot new questions & instructional formats Assess pilot Larger 2nd pilot Summer 2010 Fall 2010 Spring 2011 Summer 2011 Fall 2011 Spring 2012 Summer 2012 Fall

11 Introduction to online evaluations project at UC Berkeley Spring Pilot 11

12 Goals of spring pilot Establish baseline response rate Assess CLE evaluation system technical longterm sustainability user experience Build knowledge of CLE evaluation system Build knowledge of infrastructure needs 12

13 Spring pilot success criteria Establish baseline response rate Collect data on function & experience User experience meets or exceeds previous Transparency across campus Elicit higher quality response Gain understanding to inform Fall design & development effort 13

14 Spring pilot makeup 8 departments 61 courses (primary & secondary) enrollments 14

15 Spring pilot Demo 15

16 Spring pilot development Significant time understanding system Overall model & process to follow uber template self-service Focus on student & instructor experience System areas of focus Dashboard Evaluation presentation Administration page Notifications 16

17 Dashboard - student 17

18 Dashboard - administrator 18

19 Evaluation 19

20 Administration 20

21 Notifications 21

22 Bulk import / export configurations 22

23 Live Demo 23

24 Lessons learned Food for thought 24

25 Lessons learned Configurability + flexibility = Complexity documentation not always up to date allow significant time to play around Unfinished functionality notifications options in evaluation creation Global tool acts different than other tools where will it be exposed? 25

26 Lessons learned - cont d Allow a lot of time for QA see 1st bullet: lots of variables QA ing up to the last minute Clean data required setup if not using group provider site roles NOT always evaluation roles Communicate! be transparent remind users it s a pilot 26

27 Lessons learned - cont d Learn from other implementors document models, workflows & implications community project? Culture & legal requirements vary across campus viewing respondents roles, permissions, viewing each others evaluations 27

28 Future plans Evaluation system at Berkeley 28

29 What s next (1 of 2)? Feedback sessions with users on Spring pilot Needs analysis around results Move our work back into trunk Configuration option to hide respondents page UX improvements results page multiple instructors (Berkeley-centric) add respondent link to dashboard make display options clear in context configuration page (information architecture) 29

30 What else is next? Investigation support for instructional formats (Berkeley-centric) give department admins access to results Evalsys tool added to My Workspace automatically group provider & SIS integration partial OAE integration 30

31 Results (early idea) 31

32 Results (early idea) 32

33 Sneak peak: dashboard 33

34 Sneak peak: scale creation 34

35 Sneak peak: scale creation 35

36 Resources Berkeley Confluence Sakai Confluence Sakai Jira 36

37 Thank you! Questions, comments, discussion... 37