Evaluation of the Pilot of the Desire2Learn Learning Management System Implemented in Fall 2013

Transcription

1 Evaluation of the Pilot of the DesireLearn Learning Management System Implemented in Fall Mary Piontek, Ph.D - Indiana University Anastasia Morrone, Ph.D Indiana University December 8, Prepared by Center for Evaluation & Education Policy CEEP University Information Technology Services UITS

2 Evaluation of the Pilot of the DesireLearn Learning Management System Implemented in Fall Executive Summary December 8, Overview The Center for Evaluation and Education Policy (CEEP) in partnership with University Information Technology Services (UITS) at Indiana University (IU) conducted an evaluation of the DesireLearn Learning Management System Pilot (DesireLearn LMS) Program being implemented during the - academic year across IU campuses. The purpose of the evaluation was to provide formative feedback to guide the use of new technologies in teaching and learning at Indiana University campuses. Of the 7 faculty members who piloted the DesireLearn LMS in their courses during the fall semester, six responded to the survey (% = /7). Of the students who were enrolled in a course that piloted the DesireLearn LMS during the fall semester, twenty-four responded to the survey (% = /). Faculty and student respondents represented the IU Bloomington, IUPUI, IU East, IU Kokomo, and IUPU Columbus campuses. Slight to Moderate Satisfaction with DesireLearn by Faculty and Student Respondents All six faculty respondents used DesireLearn LMS to post course announcements (News) and upload, organize, and share course files (Manage Files, Content); and were moderately satisfied with those features (mean ratings of.8 and.7 respectively on a scale of -, from not at all satisfied to highly satisfied). Features used by at least % of respondents (i.e., three or more respondents) had means that spanned from slightly satisfied to moderately satisfied range (mean range.-.7). Features not used by any of the faculty respondents included the features of assigning individual or collaborative portfolio or web projects to students (eportfolio); creating and using rubrics to grade student work (Rubrics, Dropbox, Discussions, Quizzes, Grades); conducting online chat sessions (Chat); importing or exporting course content (Import/Export/Copy Components); and integrating an external learning tool or platform with the course, e.g., SoftChalk Cloud, Piazza, etc. (External Learning Tools). Similarly, student respondents were asked to rate the utility (i.e., usefulness) of the features of DesireLearn LMS on a scale of to (not at all useful to highly useful). The highest rated feature was Grades with a mean of. (moderately useful rating). The lowest rated feature was eportfolio with a mean of. (slightly useful rating); however, it should be noted only 9% of respondents (7) rated this item. Faculty and student respondents were also asked to rate their level of agreement with a variety of statements about DesireLearn LMS on a scale of to (strongly disagree to strongly agree). The highest rated statement by faculty respondents was DesireLearn enabled me to do what I wanted for my course(s) with a mean of.8 (agree rating); the lowest rated statement was DesireLearn was a valuable aid to me in my teaching with a mean of.8 (neither agree nor disagree rating). For student respondents, the highest rated statement was DesireLearn helps me to complete course assignments with a mean of.7 (agree rating). The statement with the highest percentage of strongly disagree and disagree ratings from student respondents was DesireLearn helps me to study for exams/tests (%); the statement with the lowest percentage of strongly disagree and disagree ratings was DesireLearn was beneficial to my overall learning in the course (8% of respondents to the question).

3 When asked what they most liked about DesireLearn LMS faculty respondents noted the ability to randomize assessment questions; the overall interface s look and feel; and the drag and drop capability. Faculty respondents disliked the difficulty in navigating the initial interface (Set Up feature) and between students assignments and gradebook feature; and difficulty in accessing students coursework submissions (Dropbox feature). No features or tools in DesireLearn LMS were described as allowing faculty to design their courses and/or teach in a new way. Student respondents most liked about DesireLearn the appearance of the interface; receiving confirmations of assignment submissions and updates on course grades; and ease of access to supplemental videos for a course. Student respondents disliked the difficulty in learning how to use the site s features; confusion in navigating discussion board postings and in locating course content and assignments; and the lack of interface/linkage between the DesireLearn system and the IU system. Similar Faculty and Student Preferences for LMS Systems Thirty-three percent () of faculty respondents prefer DesireLearn over Oncourse LMS, % () prefer Oncourse over DesireLearn LMS, and 7% () have no preference. Twenty-six percent (/) of student respondents prefer DesireLearn over Oncourse LMS, 7% (/) prefer Oncourse over DesireLearn LMS, and 7% (/) have no preference.

4 Evaluation of the Pilot of the DesireLearn Learning Management System Implemented in Fall Summary of Survey December 8, Overview The Center for Evaluation and Education Policy (CEEP) in partnership with University Information Technology Services (UITS) at Indiana University (IU) conducted an evaluation of the DesireLearn Learning Management System Pilot (DesireLearn LMS) Program being implemented during the - academic year across IU campuses (IU Bloomington, IUPUI, IU East, IU Kokomo, IU Northwest, IU South Bend, IU Southeast, and IUPU Columbus). This aggregate report for UITS summarizes the findings of the evaluation activities conducted between August and December. The purpose of the evaluation was to provide formative feedback to guide the use of new technologies in teaching and learning at Indiana University campuses. As part of the evaluation, staff members from CEEP and UITS collected data from faculty members (total n=7) who used DesireLearn LMS in their courses during the fall semester and from students (total n=) enrolled in those courses. The evaluation data collection included online surveys of faculty members and students aimed at capturing stakeholders experiences with DesireLearn LMS and their perceptions of its quality and utility. Dr. Mary Piontek, Senior Research Associate at CEEP, oversaw the design and implementation of the evaluation; created the materials for the Institutional Review Board (IRB) application and submitted the IRB application; conducted the faculty and student survey collection activities; completed the analyses of the survey data; and created the preliminary version of this report on behalf of CEEP. Dr. Anastasia Morrone, Associate Vice President for Learning Technologies and Associate Professor of Educational Psychology, participated in the design and development of the survey instruments and also contributed to the final version of this report.

5 Summary of Faculty Survey In November the 7 faculty members who piloted the DesireLearn LMS in their courses during the fall semester across Indiana University campuses were sent a customized link to an electronic survey (using Qualtrics software) via to their university address by Dr. Mary Piontek at CEEP. The faculty members were invited to voluntarily participate in the survey data collection on November th, with follow-up s sent on December nd and December th to all non-respondents. The survey was closed at pm (EDT) on December,. Six faculty members responded to the survey (% = /7). The survey included closed-ended and open-ended questions related to the faculty members experiences with DesireLearn LMS and their perceptions of its quality and utility. The survey took approximately minutes to complete. No individual received compensation for participating in the survey. Respondents could skip any items on the survey and could terminate their participation at any time. Only individuals 8 years of age or older (as of the date of opening the survey link) were allowed to participate. See Appendix A, Faculty Survey: DesireLearn Pilot Fall Semester, for a copy of the survey questions. All survey responses are confidential and are reported in the aggregate. Individual comments are not reported with any combination of demographics that allows for identification of individuals. Three respondents identified themselves as female and one as male; one respondent did not identify the gender. Table provides the years of experience as a faculty member/instructor in higher education, with all respondents having no more than years of experience. Table provides the campus affiliation of the respondents. Table : Years as Faculty Member/Instructor in Higher Education Years as a Faculty Member/Instructor Total N % year or less % - years 7% - years % - years % - years % More than years % Total % Table : Campus Affiliation Campus Affiliation Total N % IU Bloomington 7% IUPUI 7% IU East % IU Kokomo 7% IU Northwest % IU South Bend % IU Southeast % IUPU Columbus 7% Total % During the fall semester DesireLearn LMS was scheduled to be piloted in nine courses; respondents to the faculty survey represented six courses. Those courses included the academic areas of chemistry, nursing, political science, and psychology [Note: the list of the courses that the faculty

6 respondents taught is not included in this report to provide anonymity to respondents]. Thirty-three percent of the courses were structured as face-to-face or only online with no face-to-face interaction (see Table ). All respondents (%) described themselves as being very comfortable in using different types of technology (see Table ). Respondents used a variety of technology devices to interact with DesireLearn LMS including most often a laptop/netbook computer (%), and a tablet (7%) (see Table ). Table : Course Format Course Format Total N % Face-to-face % In a hybrid format using a blend of face-to-face and online interaction 7% Online with face-to-face interaction only for exams % Only online with no face-to-face interaction % Other 7% Total % Table : Level of Comfort with Technology Level of Comfort in Using Different Types of Technology Total N % Very Uncomfortable % Somewhat Uncomfortable % Somewhat Comfortable % Very Comfortable % Total % Table : Devices Used to Interact with DesireLearn LMS Mobile Devices(s) Used to Interact with DesireLearn Total N % N/ Mobile phone without web access % Mobile phone with web access % Portable media player without web access (e.g., mp player) % Portable media player with web access (e.g., ipod Touch) 7% Ebook reader (e.g., Kindle) % Tablet (e.g., ipad) 7% Laptop/Netbook computer % Other device % The faculty respondents were also asked to rate their level of satisfaction with the features of DesireLearn LMS (see Table ). Not all features were equally used across the courses. All six respondents posted course announcements (News); and uploaded, organized, and shared course files (Manage Files, Content), with mean ratings of.8 and.7 respectively. Features used by at least % of respondents (i.e., three or more respondents) had means in the slightly satisfied to highly satisfied range (mean range.-.7). No respondents used the features of assigning individual or collaborative portfolio or web projects to students (eportfolio); creating and using rubrics to grade student work (Rubrics, Dropbox, Discussions, Quizzes, Grades); conducting online chat sessions (Chat); importing or exporting course content (Import/Export/Copy Components); and integrating an external learning tool or platform with the course, e.g., SoftChalk Cloud, Piazza, etc. (External Learning Tools).

7 Table : Level of Satisfaction with DesireLearn LMS Tasks Please rate your level of satisfaction with the DesireLearn tools and features designed to support the following teaching and course management tasks: Did Not Use Not at all Satisfied Slightly Satisfied Creating and publishing the course syllabus (Content) (%) Creating a course calendar (Calendar) (7%) (7%) Posting course announcements (News) Uploading, organizing, and sharing course files (Manage Files, Content) Posting audio/video lectures or other multimedia (Content) Creating course web pages (Content > Modules > Create a File) Organizing course content, activities, and assessments into a series of modules or lessons (Course Builder, Content) Posting assignments (Dropbox) Using Turnitin originality checking on assignments (Dropbox) Creating and administering online quizzes, tests, and/or surveys (Quizzes, Self- Assessments, Surveys) Assigning individual or collaborative portfolio or web projects to students (eportfolio) Facilitating graded and ungraded discussions (Discussions) Giving feedback on and/or grading student submissions (Dropbox, Discussions, Grades) Giving audio or video feedback (Dropbox, Discussions, Grades) (7%) (7%) (%) (%) (7%) (7%) (7%) (7%) (7%) (%) (%) (%) (7%) (7%) Moderately Satisfied (%) (7%) (%) (7%) (7%) (7%) (7%) Highly Satisfied (7%) (%) (%) (7%) (7%) (%) (%) (7%) (%) (%) (7%) Total N Mean N/A.8.. 7

8 Table : Level of Satisfaction with DesireLearn LMS Tasks (continued) Please rate your level of satisfaction with the DesireLearn tools and features designed to support the following teaching and course management tasks: Did Not Use Not at all Satisfied Slightly Satisfied Moderately Satisfied Creating and using rubrics to grade student work (Rubrics, Dropbox, Discussions, Quizzes, Grades) Setting up and using the gradebook to enter and track student grades (Grades) Monitoring course activity and student progress (View User Progress) Creating course learning outcomes and objectives and aligning course activities with them (Competencies) Sending and receiving messages to and from students and groups ( , Pager) Creating and managing groups for group assignments, group discussions, and/or group projects (Groups) Conducting online chat sessions (Chat) Keeping track of your course tasks (Calendar, Upcoming Events, Notifications) Importing or exporting course content (Import/Export/Copy Components) Integrating an external learning tool or platform with my course, e.g., SoftChalk Cloud, Piazza, etc. (External Learning Tools) Customizing the navigation, look, and feel of your course (Homepages, Navigation and Themes) (%) (7%) (%) (%) (%) (%) (%) (%) (%) (%) (%) (%) (7%) (%) (%) (%) (7%) () (%) (7%) (7%) Highly Satisfied (7%) (%) (%) (7%) (7%) (7%) Total N Mean N/A..... N/A. N/A N/A.7 8

9 Thirty-three percent of faculty respondents () found DesireLearn LMS very easy to use, highly useful for their teaching, and useful for online documentation (see Tables 7 and 8). Table 7: Overall Ease of Use of DesireLearn LMS Please rate the overall ease of use of Difficult DesireLearn. to Use Overall ease of use of DesireLearn (%) Slightly Easy to Use (%) Moderately Easy to Use Very Easy to Use (%) Total N Mean. Table 8: Overall Utility of DesireLearn LMS for Teaching Please rate the overall usefulness of DesireLearn for your teaching. Not at all Slightly Overall usefulness of DesireLearn for your teaching (7%) (7%) Overall usefulness of DesireLearn s online documentation (7%) (%) Moderately (%) Highly (%) (%) Total Mean N.8. Respondents were also asked to rate their level of agreement with a variety of statements about DesireLearn LMS (see Table 9). All of the statements were rated.8 or above on a scale of to (strongly disagree to strongly agree). The highest rated statement was DesireLearn enabled me to do what I wanted for my course(s) with a mean of.8 (agree rating). The lowest rated statement was DesireLearn was a valuable aid to me in my teaching with a mean of.8 (neither agree nor disagree rating). Thirty-three percent () of faculty respondents prefer DesireLearn over Oncourse LMS, % () prefer Oncourse over DesireLearn LMS, and 7% () have no preference. When asked what they most liked about DesireLearn LMS faculty respondents noted the ability to randomize assessment questions; the overall interface s look and feel; and the drag and drop capability. When asked what they liked least about DesireLearn LMS faculty respondents noted difficulty in navigating the initial interface (Set Up feature) and between students assignments and gradebook feature; and difficulty in accessing students coursework submissions (Dropbox feature). No features or tools in DesireLearn LMS were noted that allowed faculty to design their courses and/or teach in a new way. 9

10 Table 9: Level of Agreement with Statements about DesireLearn LMS Please rate your level of agreement with the following statements about DesireLearn. Strongly Disagree Disagree Neither Agree nor Disagree Agree Strongly Agree DesireLearn enabled me to do what I wanted for my course(s). DesireLearn was easy for my students to learn how to use. DesireLearn increased my efficiency as a teacher. DesireLearn increased my effectiveness as a teacher. DesireLearn was a valuable aid to me in my teaching. Using DesireLearn has been beneficial to my students overall learning. (7%) %) (%) (%) (7%) (7%) (%) (7%) (7%) (%) (7%) (7%) (7%) (7%) (7%) (%) (%) (%) (%) (%) (%) (%) (%) (%) Not Applicable Total N Mean

11 Summary of Student Survey In November the students who were enrolled in a course that piloted the DesireLearn LMS during the fall semester across Indiana University campuses were sent a customized link to an electronic survey (using Qualtrics software) via to their university address by Dr. Piontek. The students were invited to voluntarily participate in the survey data collection on November th, with follow-up s sent on December nd and December th to all non-respondents. The survey was closed at pm (EDT) on December,. Twenty-four students responded to the survey (% = /). The survey included closed-ended and open-ended questions related to the students experience with DesireLearn LMS and their perceptions of its quality and utility. The survey took approximately minutes to complete. No individual received compensation for participating in the survey. Respondents could skip any items on the survey and could terminate their participation at any time. Only individuals 8 years of age or older (as of the date of opening the survey link) were allowed to participate. See Appendix B, Student Survey: DesireLearn Pilot Fall Semester, for a copy of the survey questions. All survey responses are confidential and are reported in the aggregate. Individual comments are not reported with any combination of demographics that allows for identification of individuals. Eighteen (7%) respondents identified themselves as female and six (%) as male. The age of students ranged from 8 to 9 years of age with the greatest percentage (%, /) in the 8 to range. Thirty-three percent (8) of student respondents are second-year undergraduates (sophomores) (see Table ). Fifty-eight percent of respondents are affiliated with the IU East campus (see Table ). During the fall semester DesireLearn LMS was scheduled to be piloted in nine courses; respondents to the student survey represented eight courses (see Table ). Fifty-four percent () of the respondents were in courses with a face-to-face format (see Table ). Table : Academic Level Academic Level Total N % First-year undergraduate (Freshman) 8% Second-year undergraduate (Sophomore) 8 % Third-year undergraduate (Junior) 8% Four or more years undergraduate (Senior) % Masters student (MA, MS, MBA, MFA, MSW, MPA, etc.) % Doctoral Student (EdD, PhD, etc.) % Total % Table : Campus Affiliation Campus Affiliation Total N % IU Bloomington % IUPUI % IU East 8% IU Kokomo % IU Northwest % IU South Bend % IU Southeast % IUPU Columbus % Total %

12 Table : Name of Enrolled Course Course Name Total % N CHEM C, Principles of Chemistry I, EA, Section 8% CHEM C, Principles of Chemistry I, EA, Section 7 % EDUC W, Technology Leadership IN K-, CO, Section 98 % INFO B, Clinical Information Systems, IN, Section 7 9% NURS I, Clinical Information Systems, IN, Section 9% NURS T7, Evaluation in Nursing, IN, Section % PHSL P, Basic Human Physiology, EA, Section 9 9% POLS Y9, Pol Principle and Social Justice, BL, Section 8 % PSY P9, Practicum in Psychology, KO, Section 77 % Total % Table : Course Format Course Format Total N % Face-to-face % In a hybrid format using a blend of face-to-face and online interaction 7% Online with face-to-face interaction only for exams % Only online with no face-to-face interaction 7 9% Other % Total % Thirty-eight percent (9) of respondents described themselves as being very comfortable in using different types of technology and % () described themselves as being very uncomfortable (see Table ). Respondents used a variety of technology devices to interact with DesireLearn LMS including most often a laptop/netbook computer (9%) and mobile phone with web access (7%) (see Table ). Most students (%) averaged fewer than five hours per week using DesireLearn for their courses (see Table ). Forty-three percent of student respondents () found DesireLearn LMS moderately easy to use and 8% moderately useful () for online documentation (see Tables 7 and 8). Table : Level of Comfort with Technology Level of Comfort in Using Different Types of Technology Total N % Very Uncomfortable % Somewhat Uncomfortable 8% Somewhat Comfortable 7 9% Very Comfortable 9 8% Total %

13 Table : Devices Used to Interact with DesireLearn LMS Mobile Devices(s) Used to Interact with DesireLearn Total N % (N/9) Mobile phone without web access 8% Mobile phone with web access 8 7% Portable media player without web access (e.g., mp player) 8% Portable media player with web access (e.g., ipod Touch) 7% Ebook reader (e.g., Kindle) % Tablet (e.g., ipad) % Laptop/Netbook computer 9% Other device % Table : Average Number of Hours Per Week Using DesireLearn LMS Average Number of Hours Using DesireLearn for the Course During the fall Total N % Semester Never % Fewer than hours % - hours 7% - hours % - hours 8% More than hours per week % Total % Table 7: Overall Ease of Use of DesireLearn LMS Please rate the overall ease of use Difficult of DesireLearn. to Use Overall ease of use of DesireLearn (7%) Slightly Easy to Use (9%) Moderately Easy to Use (%) Very Easy to Use 7 (%) Total N Mean.87 Table 8: Overall Utility of DesireLearn LMS for Learning Please rate the overall usefulness of DesireLearn for your learning. Not at all Slightly Overall usefulness of DesireLearn s online documentation (%) Moderately Highly (%) Total Mean N.7 The student respondents were asked to rate the utility (i.e., usefulness) of the features of DesireLearn LMS on a scale of to (not at all useful to highly useful) (see Table 9). The highest rated feature was Grades (e.g., potential points for an assignment, your results on a quiz/assignment, overall course grade, etc.) with a mean of.. The lowest rated feature was eportfolio (for personal or collaborative web sites about your accomplishments or other topics) with a mean of. (however, it should be noted only 9% of respondents rated this item). The most frequently used feature was Content (for viewing course materials and completing activities organized into lessons or modules) with 87% of respondents () rating the item. The least frequently used feature was Blog (for creating a web log with public or private posts and optionally receiving comments from other users) with only 7% of respondents () rating the item. Student respondents were also asked to rate their level of agreement with a variety of statements about DesireLearn LMS (see Table ). All of the statements had a mean rating of. or above on a scale of to (strongly disagree to strongly agree). The highest rated statement was DesireLearn helps me to complete course assignments with a mean of.7. The statement with the highest percentage of strongly

14 disagree and disagree ratings was DesireLearn helps me to study for exams/tests (%). The statement with the lowest percentage of strongly disagree and disagree ratings was DesireLearn was beneficial to my overall learning in the course (8% of respondents to the question). Twenty-six percent (/) of student respondents prefer DesireLearn over Oncourse LMS, 7% (/) prefer Oncourse over DesireLearn LMS, and 7% (/) have no preference. When asked what they most liked about DesireLearn LMS student respondents noted the appearance of the overall interface; receiving confirmations of assignment submissions and updates on course grades; and ease of access to supplemental videos for the course. When asked what they liked least about DesireLearn LMS respondents noted difficulty in learning how to use the site s features; confusion in navigating discussion board postings and in locating course content and assignments; and the lack of interface/linkage between the DesireLearn system and the IU system. Table 9: Utility of DesireLearn LMS Features Please rate the usefulness of the following features of DesireLearn in contributing to your learning in this course. Did Not Use This Feature Not at all Blog (for creating a web log with public or private posts and optionally receiving comments from other users) Calendar (for viewing course events and due dates) Chat/Pager (for live text messaging with classmates and other DesireLearn users) Checklist (for completing a list of tasks prepared by the instructor) Classlist (for viewing a list of the other people in the course) Content (for viewing course materials and completing activities organized into lessons or modules) Discussion (for participating in online discussions with the entire class or in small groups) Dropbox (for submitting individual or group assignments) (for sending and receiving messages to and from your instructor and other students) eportfolio (for personal or collaborative web sites about your accomplishments or other topics) Grades (for viewing a list of the graded items in the course and the grades you received) (%) 7 (7%) 7 (9%) 9 (%) (%) (%) (7%) 7 (7%) (%) (%) (%) (%) (%) (%) (7%) (7%) (%) Slightly (7%) (7%) (%) (%) (7%) (%) (%) (%) Moderately 7 (9%) 9 (%) (%) (7%) (%) Highly (%) (%) (%) (%) (%) (%) (%) 9 (%) (%) Total N Mean

15 Table 9: Utility of DesireLearn LMS Features (continued) Please rate the usefulness of the following features of DesireLearn in contributing to your learning in this course. Did Not Use This Feature Not at all Slightly Groups (for collaborating with a specific group of students on assignments, discussions, or projects) Locker (for storing personal files or files belonging to your group) News (read announcements and other timely news and information posted by your instructor) Quizzes/ Self-assessments (for taking and receiving feedback on online quizzes, tests, and self-assessments) Rubrics (for understanding how your work will be or was graded) Surveys (for taking online surveys) User Progress (for tracking your own academic progress in the course) 9 (79%) 9 8 (%) (%) (%) (%) (%) (%) (7%) (%) Moderately (%) (%) (%) (%) Highly (7%) (%) 7 (9%) (%) (%) 7 (9%) Total N Mean Table : Level of Agreement with Statements about DesireLearn LMS Please rate your level of agreement with the following statements about DesireLearn. Strongly Disagree Disagree Neither Agree nor Disagree Agree Strongly Agree DesireLearn helps me to learn the course materials/content. DesireLearn helps me to study for exams/tests. DesireLearn helps me to complete course assignments. DesireLearn helps me to take quizzes/exams. DesireLearn helps me to make efficient use of my time in the course. DesireLearn helps me to be in control of my own learning in the course. (%) (7%) (%) (7%) (%) (%) (9%) (%) (9%) (%) (9%) (%) (%) (%) (7%) (%) 8 (%) 8 (%) 8 (%) (%) (%) (7%) (%) (%) (9%) (7%) (%) (%) (7%) (%) Not Applicable (%) (%) (7%) (%) (%) (%) Total N Mean

16 Table : Level of Agreement with Statements about DesireLearn LMS (continued) Please rate your level of agreement with the following statements about DesireLearn. Strongly Disagree Disagree Neither Agree nor Disagree Agree Strongly Agree Not Applicable DesireLearn helps me to communicate with my professor. DesireLearn expands access to learning materials/resources available to me (e.g., print, audio, video, etc.). DesireLearn was beneficial to my overall learning in the course. (%) (9%) (9%) (9%) (%) (9%) (%) 8 (%) 8 (%) (%) (%) (%) (%) (7%) (9%) (9%) (%) (%) Total N Mean...

17 Appendix A Faculty Survey: DesireLearn Pilot Fall Semester Introduction and Consent In partnership with University Information Technology Services (UITS) at Indiana University (IU), the Center for Evaluation and Education Policy (CEEP) is conducting an evaluation of the DesireLearn Learning Management System (DesireLearn) Pilot Program being conducted during the fall semester. You have been invited to participate in this survey because you are teaching one or more courses in DesireLearn this semester. Participation in this survey is completely voluntary. Should you decide to participate your responses will be kept confidential and will be reported only in aggregate in published reports. Comments will not be reported with demographics that would allow for identification of individuals. If you have any questions about the survey, please contact Dr. Mary Piontek, Senior Research Associate at CEEP, at mepionte@indiana.edu. This survey should take approximately minutes to complete. Data collection will close at pm (EST) on Day, Month, Date,. You may terminate your participation in the survey at any time. Pre-question : If you consent to participate in the survey data collection process, please acknowledge by checking the box below. Then proceed to next page to begin the survey. I consent to participate in this survey. (required check box) Part I: Course/IU Context Q: For how many years have you been an instructor/faculty member in higher education? Scale: year or less, - years, - years, - years, - years, more than years Q: What is your gender? (open text box) Q: At which campus are you a faculty member? (Choose your PRIMARY academic appointment) IU Bloomington, IUPUC, IUPUI, IU East, IU Kokomo, IU Northwest, IU South Bend, IU Southeast Q: In which course during Fall did you use DesireLearn? (If you used DesireLearn in multiple courses, please choose one for the purpose of this survey.) Although we ask you to identify the course in which you piloted DesireLearn, this survey is not an evaluation of the course. We will use this information to report aggregate response rates across the courses and to create categories of types (disciplinary areas) and levels of courses for reporting purposes. (drop down list of DesireLearn pilot courses) 7

18 Q: The course was delivered (Choose one BEST answer) Scale: primarily face-to-face; using a blend of face-to-face and online interaction; online with face-to-face interaction only for exams; exclusively online with no face-to-face interaction; other (please describe) Part II: Use of Technology Q: In terms of my level of comfort in using different types of technology, I am Scale: very uncomfortable, somewhat uncomfortable, somewhat comfortable, very comfortable Q7: Which device(s) do you currently use to interact with DesireLearn? (Choose all that apply) mobile phone; portable media player (e.g., ipod Touch); ebook reader (e.g., Kindle); tablet (e.g., ipad, Nexus, Galaxy); laptop/netbook computer; desktop computer; other device (please describe) Part III: Feedback on DesireLearn Q8: Based on your experience during the fall pilot, please rate your level of satisfaction with the DesireLearn tools and features designed to support the following teaching and course management tasks: Scale: did not use, not at all satisfied, slightly satisfied, moderately satisfied, highly satisfied Creating and publishing the course syllabus (Content) Creating a course calendar (Calendar) Posting course announcements (News) Uploading, organizing, and sharing course files (Manage Files, Content) Posting audio/video lectures or other multimedia (Content) Creating course web pages (Content > Modules > Create a File) Organizing course content, activities, and assessments into a series of modules or lessons (Course Builder, Content) Posting assignments (Dropbox) Using Turnitin originality checking on assignments (Dropbox) Creating and administering online quizzes, tests, and/or surveys (Quizzes, Self- Assessments, Surveys) Assigning individual or collaborative portfolio or web projects to students (eportfolio) Facilitating graded and ungraded discussions (Discussions) Giving feedback on and/or grading student submissions (Dropbox, Discussions, Grades) Giving audio or video feedback (Dropbox, Discussions, Grades) Creating and using rubrics to grade student work (Rubrics, Dropbox, Discussions, Quizzes, Grades) Setting up and using the gradebook to enter and track student grades (Grades) Monitoring course activity and student progress (View User Progress) Creating course learning outcomes and objectives and aligning course activities with them (Competencies) Sending and receiving messages to and from students and groups ( , Pager) 8

19 Creating and managing groups for group assignments, group discussions, and/or group projects (Groups) Conducting online chat sessions (Chat) Keeping track of your course tasks (Calendar, Upcoming Events, Notifications) Importing or exporting course content (Import/Export/Copy Components) Integrating an external learning tool or platform with my course, e.g., SoftChalk Cloud, Piazza, etc. (External Learning Tools) Customizing the navigation, look, and feel of your course (Homepages, Navigation and Themes) Other Q9: Please rate the overall ease of use of DesireLearn. Scale: difficult to use, slightly easy to use, moderately easy to use, very easy to use Q: Please rate the overall usefulness of DesireLearn for your teaching. Scale: not at all useful, slightly useful, moderately useful, highly useful Q: Please rate the overall usefulness of DesireLearn s online documentation. Scale: did not use, not at all useful, slightly useful, moderately useful, highly useful Q: What did you like MOST about DesireLearn? Why? (open text box) Q: What did you like LEAST about DesireLearn? Why? (open text box) Q: Which, if any, features/tools in DesireLearn allowed you to design your course and/or teach in a new way? (open text box) Q Please rate your level of agreement with the following statements about DesireLearn: Scale: strongly disagree, disagree, neither agree nor disagree, agree, strongly agree, not applicable. DesireLearn enabled me to do what I wanted for my course(s).. DesireLearn was easy for my students to use.. DesireLearn increased my efficiency as a teacher.. DesireLearn increased my effectiveness as a teacher.. DesireLearn was a valuable aid to me in my teaching.. Using DesireLearn has been beneficial to my students overall learning. 9

20 Q Overall, compared to Indiana University s current LMS (Oncourse), please select the choice that best describes your preference for Oncourse versus DesireLearn: I prefer DesireLearn over Oncourse. I prefer Oncourse over DesireLearn. I have no preference. Q7 Additional Comments Is there anything else you would like to tell us about your experience using DesireLearn this semester? (open text box)

21 Appendix B Student Survey: DesireLearn Pilot Fall Semester Introduction and Consent In partnership with University Information Technology Services (UITS) at Indiana University (IU), the Center for Evaluation and Education Policy (CEEP) is conducting an evaluation of the DesireLearn Learning Management System (DesireLearn) Pilot Program being conducted during the fall semester. You have been invited to participate in this survey because you are enrolled in one or more courses being taught in DesireLearn this semester. Participation in this survey is completely voluntary. Should you decide to participate your responses will be kept confidential and will be reported only in aggregate in published reports. Comments will not be reported with demographics that would allow for identification of individuals. If you have any questions about the survey, please contact Dr. Mary Piontek, Senior Research Associate at CEEP, at mepionte@indiana.edu. This survey should take approximately minutes to complete. Data collection will close at pm (EST) on Day, Month, Date,. You may terminate your participation in the survey at any time. Pre-Question : If you are 8 years of age or older as of today s date and consent to participate in the survey data collection process, please acknowledge by checking the box below. Then proceed to next page to begin the survey. If you are NOT 8 years of age or older as of today s date or do not consent, please close the link to the survey now. I acknowledge that I am at least 8 years of age or older as of today s date and consent to participate in this survey (will be a required check box) Part I: Student Demographics/Institutional Context Q: What is your current academic level? Scale: first-year undergraduate (freshman), second-year undergraduate (sophomore), third-year undergraduate (junior), four or more years undergraduate (senior), master s degree student (MA, MS, MBA, MFA, etc.), doctoral student (EdD, PhD, etc.) Q: At which campus are you enrolled as a student? (Choose ONE) IU Bloomington, IUPUC, IUPUI, IU East, IU Kokomo, IU Northwest, IU South Bend, IU Southeast Q: What is your age? (open text box) Q: What is your gender? (open text box)

22 Q: Which course were you enrolled in during fall that used DesireLearn? (If you were enrolled in more than one course that used DesireLearn, please choose one for the purpose of this survey) Although we ask you to identify the course in which you were enrolled, this survey is not an evaluation of the course. We will use this information to report aggregate response rates within each course and to create categories of types (disciplinary areas) and levels of courses for reporting purposes. (drop down list of DesireLearn pilot courses) Q: That course was delivered (Choose one BEST answer) primarily face-to-face; using a blend of face-to-face and online interaction; online with face-toface interaction only for exams; exclusively online with no face-to-face interaction; other (please describe) Part II: Use of Technology General Level of Experience/Use: Q7: In terms of my level of comfort in using different types of technology, I am Scale: very uncomfortable, somewhat uncomfortable, somewhat comfortable, very comfortable Q8: What type(s) of networked device(s) do you currently use on a regular basis? mobile phone; portable media player (e.g., ipod Touch); ebook reader (e.g., Kindle); tablet (e.g., ipad, Nexus, Galaxy); laptop/netbook computer; desktop computer; other device (please describe) Q9: On average, how many hours per week have you been spending in DesireLearn for this course? Scale: none, fewer than hours, - hours, - hours, - hours, more than hours Part III: Feedback on DesireLearn Q: Please rate the ness of the following features of DesireLearn in contributing to your learning in this course. Scale: did not use this feature, not at all useful, slightly useful, moderately useful, highly useful Blog (for creating a web log with public or private posts and optionally receiving comments from other users) Calendar (for viewing course events and due dates) Chat/Pager (for live text messaging with classmates and other DesireLearn users) Checklist (for completing a list of tasks prepared by the instructor) Classlist (for viewing a list of the other people in the course) Content (for viewing course materials and completing activities organized into lessons or modules) Discussion (for participating in online discussions with the entire class or in small groups)

23 Dropbox (for submitting individual or group assignments) (for sending and receiving messages to and from your instructor and other students) eportfolio (for personal or collaborative web sites about your accomplishments or other topics) Grades (for viewing a list of the graded items in the course and the grades you received) Groups (for collaborating with a specific group of students on assignments, discussions, or projects) Locker (for storing personal files or files belonging to your group) News (read announcements and other timely news and information posted by your instructor) Quizzes/ Self-assessments (for taking and receiving feedback on online quizzes, tests, and self-assessments) Rubrics (for understanding how your work will be or was graded) Surveys (for taking online surveys) User Progress (for tracking your own academic progress in the course) Q: Please rate the overall ease of use of DesireLearn. Scale: difficult to use, slightly easy to use, moderately easy to use, very easy to use Q: Please rate the overall usefulness of DesireLearn s online documentation for students. Scale: did not use, not at all useful, slightly useful, moderately useful, highly useful Q: What did you like MOST about DesireLearn? Why? (open text box) Q: What did you like LEAST about DesireLearn? Why? (open text box) Q: Perceived impact on learning Scale: strongly disagree, disagree, neither agree nor disagree, agree, strongly agree, not applicable DesireLearn helps me to learn the course materials/content. DesireLearn helps me to study for exams/tests. DesireLearn helps me to complete course assignments. DesireLearn helps me to take quizzes/exams. DesireLearn helps me to make efficient use of my time in the course. DesireLearn helps me to be in control of my own learning in the course. DesireLearn helps me to communicate with my professor. DesireLearn expands access to learning materials/resources available to me (e.g., print, audio, video, etc.). DesireLearn was beneficial to my overall learning in the course.

24 Q Overall, compared to Indiana University s current LMS (Oncourse), please select the choice that best describes your preference for Oncourse versus DesireLearn: I prefer DesireLearn over Oncourse. I prefer Oncourse over DesireLearn. I have no preference. Q7 Additional Comments Is there anything else you would like to tell us about your experience using DesireLearn this semester? (open text box)