Online SEI software recommendation for Faculty Senate Review,

Transcription

1 Online SEI software recommendation for Faculty Senate Review, The 49 th Faculty Senate endorsed the creation of an Ad Hoc Committee (along with the Office of the Provost) to study an alternative software solution for the required student evaluation of instruction, in response to declining student participation in online SEI and an increase in the number of academic departments that were choosing or considering a return to the use of paper SEI forms. In addition, study of alternatives was requested by the Student Senate (also concerned about low student participation) and by ITS. Primary concerns in ITS were the age and cost of replacement for hardware needed to support paper SEI forms, and the lack of available staff time for managing and supporting the use of paper forms. The Ad Hoc Committee membership included faculty, ADAs, and administrative officers. They evaluated an alternative to UWL s own software that had been developed to manage SEI data collection in an online format, on the basis of capability, usability, and cost. After an initial review, they endorsed the creation of a pilot project for SEI data collection during the summer session of 2015, with faculty and ADAs from several departments participating. The software solution that was piloted is referred to as SmartEvals, and was offered for free for the duration of the pilot project. This project concluded at the end of the summer, and the results were summarized in a report from Shawn Frederick and other members of ITS. That report is attached. The Senate Executive Committee has discussed this report with our CIO, Dr. Mohamed Elhindi, and has recommended the following action for consideration by the Faculty Senate. The Senate Executive Committee recommends that the SmartEvals software package become the standard software for the online collection of SEI data by academic departments of the university, to begin in Spring Semester of The context of this recommendation includes a desire to eventually have all SEI data collection done via an online software solution, and to phase out the resource-intensive paper SEI forms and processes. Adoption of the SmartEvals software package as the standard software does not, however, make the use of online SEI collection mandatory at this time. What it accomplishes, we believe, is to reduce costs significantly while providing better tools for online data collection than the software that is currently in use. It may help to make online SEI collection a more palatable solution for ADAs, students, and instructors based on the experience from the pilot project in the summer of While several members of the SEI Ad Hoc Committee from the spring semester, and at least one member of ITS who played an important role in the pilot project itself, are no longer at UWL, we expect to reconvene the committee to provide further advice on full-scale implementation. The cost of the SmartEvals software is projected to be no more than $11,000 per year, including support services. ITS reports that support services from the SmartEvals providers were solid and very responsive over the course of the summer. Cost of paper SEI administration for the small number of departments now using paper forms is estimated to exceed $55,000, including paper and IT support. Those costs will increase significantly if additional departments return to the use of paper forms, and there is uneasiness

2 about the ability to maintain outmoded hardware needed for the use of paper forms even if significant dollars are available. ADAs who were involved in the pilot project report that the use of SmartEvals software is simpler than the use of UWL s current online software. Faculty who were involved in the pilot project were neutral or in favor of changing to the SmartEvals software (no complete survey has been done, however). Student experience of the SmartEvals software is neutral or improved, with a significant improvement being seen in that the SmartEvals software is friendly for mobile devices, and can therefore be used in the classroom setting. Response rates were similar to the current online software, though instructors who took advantage of the ability to use mobile devices for students to respond in the classroom saw very good response rates. Members of the SEI Ad Hoc Committee were as listed below. Kary Auby (Math ADA) Anders Cedergren (HEHP) Shawn Frederick (ITS) Dan Grilley (Chem/Biochem, ATC Committee) Mary Grattan (Economics ADA) Storm Larson (Student Senate) Alex O'Brien (Psychology) Kasi Periyasamy (Computer Science) Brad Seebach (Biology, Faculty Senate) Natalie Solverson (Institutional Research Officer) Robert Hoar (Associate Vice Chancellor of Academic Affairs)

3 SmartEvals SEI Overview Report Background SmartEvals was piloted as a replacement tool for collecting SEI s to increase the survey completion results. The current SEI system is developed in-house and supported in ITS. SmartEvals is a web based tool with strong reporting features. The application was piloted with six departments running summer classes. The point individual for the instructors was Brad Seebach and department ADAs. Results were reviewed with the ADAs to glean ease of use and reporting functionality. Configuration Setup o Does not integrate with D2L directly through LTI, but instructors can hyperlink to SmartEvals. o 3 main tables were exported from WINGS and uploaded to SmartEvals: Student Roster User ID and demographics on the students Courses Course, department, (survey start, end and release dates), custom traits (faculty/instructional staff) and instructor information Registration Course and student ID How students are sent access to SmartEvals o Students are sent s with links to the surveys. o Through a link provided in D2L o The QR code feature is not able to be leveraged due to single sign on setup Single sign on o SSO integration with SmartEvals eliminates the need for maintaining separate passwords. o Daily export of the users, instructors and courses from WINGS will need to be configured to keep the data current. o Advance customizable notifications exist. o For the summer pilot three s were sent but the features exist to send more. o The notifications were manual during the pilot but can be configured to be automated.

4 Response rate results Actual response rates as reported in Smart Evals as broken down by department Department Not Responded Responded Total Percentage BIO CHM CYC ECO MTH PSY Actual response rates as reported in Smart Evals as broken down by course Instructor Dept Num Course Name Sec Type N Tot % Litster, M. BIO 103 Intro Biology 101 LEC % Galbraith, A. (team) BIO 306 Genetics 101 LEC % Osmundson, T. (team) BIO 306 Genetics 101 LEC % Human Anatomy & Seebach, B. BIO 312 Physiology I 101 LEC % Bryan, J. CHM 103 General Chem I 201 LEC % Carmosini, N. CHM 103 General Chem I 21L LAB % Kirsch, J. CHM 103 General Chem I 22L LAB % Tobin, C. CYC 450 Intern Child/Youth 413 FLD % Nunley, J. ECO 110 Microeco & Pub Pol 431 LEC % Babakhani Teimouri, S. ECO 120 Global Macroecon 421 LEC % Baumann, D. MTH 145 Elem Statistics 102 LEC % Allen, R. MTH 145 Elem Statistics 203 LEC % Basten, B. PSY 100 General Psychology 411 LEC % Tobin, C. PSY 204 Abnormal Psychology 421 LEC % Tobin, C. PSY 309 Volunteer Experience in Psyc 411 FLD % Basten, B. PSY 321 Research Methods Psy Minors 411 LEC % McKelley, R. PSY 334 Health Psychology 411 LEC % OBrien, A. PSY 432 Learning and Memory 421 LEC % Tobin, C. PSY 450 Fieldwork Experience Psy 411 FLD % Tobin, C. PSY 450 Fieldwork Experience Psy 412 FLD % Quartiroli, A. PSY 451 Psych Measurement 411 LEC % 2 of 21 classes meet or exceeded the goal of 75% response rate. When the reports were reviewed, each successive sent generated 10% more response.

5 Marketing the survey seems to be the key factor in the higher response rates. Reports What type of information is included on the reports o Fractional median as defined by UWL is included. o List of comments are included in the survey results. o Graphic representations are optional to include for quick assessment. o More comprehensive tools for reporting results are available using the custom reports. The reports can be filtered by: Evaluation period Individual or group class summary Class sections Departments Instructors, individuals or group Custom feature of Faculty or instructional staff What the reports do not include o The calculated fractional median for questions 2-6 would need to be developed by the vendor outside the scope of the pilot. o Does not integrate with TAI s (Teaching Assignment Information). However, the SmartEvals reports can be given in addition to what is provided through WINGS. Summary SmartEvals did not meet the intended goal of 75% response rate. Some factors that might have affected the results were that the QR codes were not able to be used and the frequency was limited to 3 total. SmartEvals has a number of features that the ADAs liked during the survey results review, the ability to generate detailed reports easily and to customize the reports to address different criteria request was emphasized. The option exist for exporting the survey results to many different formats, mainly focusing on excel or PDF. SmartEvals has advance reporting demographic features that were not used in the pilot but could be explored in future implementations. During the pilot the setup was controlled and configured to a large part by IT, for production we would have to confirm that the ADAs were able to configure the classes. SmartEvals covers the same basic information as the current SEI collection system with more advance reporting features. Response rates are similar to the current system which suggest that the marketing of the surveys needs to be addressed.