Measuring Information Technology Use. at Winona State University

Transcription

1 at Winona State University Hyesung Park, Ph.D. Assessment and Institutional Research

2 ABSTRACT The use of technology in the classroom has grown significantly as information technology (IT) has become more affordable and effective. To increase accessibility to technology in the classroom, institutions of higher education have invested more money in IT. Despite this increase in IT investments, various studies have shown that IT remains underutilized due to users reluctance to adopt new technology. This study investigated the use of information technology at Winona State University to understand faculty- and student- technology use related to general issues. This study reveals IT adoption barriers, perceptions of faculty and students toward technology, and an online success model at Winona State University. 2

3 INTRODUCTION Two cycles of study, faculty surveys and student surveys, were employed to understand the issues surrounding IT use from the perspectives of faculty members and students at Winona State University. Faculty Survey In November 2006, Winona State University participated in the University of Wisconsin- Madison s survey, A Study of Faculty Use of Digital Resources and Libraries. Over 100 Winona State faculty members responded to the online survey. The current study was designed to answer the following questions after assessing the survey s feasibility and its appropriateness to the current study objectives: (1) What are the characteristics of online collections that make them useful or not useful for teaching? (2) How do faculty members employ the materials from useful collections? (3) How can collections, resources, and services be best aligned with a faculty work pattern? Student Assessment Day Survey After the faculty survey, the second cycle of the study was launched with a focus on the student perception of an online course management system. Across MnSCU (Minnesota State Colleges and Universities system), Desire to Learn systems (D2L) were newly selected as the official course management systems for spring This survey was designed to answer the following questions: (1) Do the students see D2L as a useful, effective, and easy-to-use course management tool? (2) What are the students perceptions regarding system quality (e.g., ease of use, course material access, ease of navigation), service quality (e.g., course organization, course 3

4 presentation), information quality, and user satisfaction (e.g., beliefs about D2L s ability enhancing learning success, positive user experience, and future expectation for using D2L)? To collect student evaluations of D2L, a survey was developed and administered on Assessment Day in February of METHODOLOGY Faculty Survey As the survey items measured faculty members use of technology, the data provided a baseline for the increasing awareness of IT capabilities available at WSU. The data were also used to identify barriers that prevent faculty from using the IT resources available throughout the campus. The faculty survey was developed by University of Wisconsin Madison (UW-Madison). They conducted a pre-test with 20 faculty members, and in-depth interviews with six samples. The survey items were continually revised to increase the validity of the survey. The survey concerned faculty members use of a collection of five digital resources and libraries provided by IT: (1) digital images/visual materials and historical documents (e.g., drawings, photographs, digital video, art, posters, and digitized documents); (2) online simulations/animations; (3) online datasets (e.g., online educational, business, and government datasets; scientific research databases); 4

5 Measuring Information Technology Use (4) teaching or learning activities and exercises (e.g., assignments, tutorials, lab procedures, problem sets, and case studies); and (5) online scholarly resources (e.g., online journals, scholarly articles, and other scholarly discussion groups or sites) ). The survey site was open to faculty members from October 2006 to January More than 100 faculty members were responded to the survey. The respondents were tenured faculty (50 %), permanent non-tenured faculty (4 %), non-tenured faculty (27 %), adjunct faculty (7 %), librarian (1 %), instructors (5 %), and others (6%). A demographic chart by position is provided in Figure 1 and a demographic chart by department is provided in Figure 2. Figure 1. Faculty Demographics by Position Adjunct Faculty, 7% Instructor, 5% Librarian, 1% Tenured Faculty, 50% Non-Tenured Faculty, 27% Permanent Non-Tenured Faculty, 4% Other (please specify) ), 6% 5

6 Measuring Information Technology Use Figure 2. Faculty Demographics by Department 16% 14% 12% 10% 8% 6% 4% 2% 0% 1% 2% 4% 3% 3% 5% 10% 10% 10% 12% 13% 13% 14% Student Assessment Day Survey Until fall of 2006, the Blackboard system was the official course management system: the transition period from Blackboard to D2L was lengthy. Students used D2L and Blackboard systemss together in the fall of 2006 during the transition period and D2L replaced Blackboard in the spring of 2007 as the official course management system. Therefore, the spring of 2007 was the most importantt year to measure the students perception of D2L. A group composed of 499 freshmen and sophomore undergraduate students participated in the student assessment survey to provide perspectives on the newly adopted course management tool. A survey was developed based on the literature review and refined based on the feedback of three experts * in the area of academic research and assessment. Survey items were refined to * Acknowledgement: the survey was developed and conducted in collaboration with Dr. Susan Hatfield from the office of Assessment and Institutional Research, Dr. Kihyun Kim from the MISO departments, and Dr. Ken Graetz from the e Learning Center. The author appreciates their excellent research support. 6

7 Measuring Information Technology Use better assess the target measures-based on the purpose of the study. Factor analysis and a reliability test weree conductedd to increasee validity. DATA ANALYSIS AND FINDINGS Faculty Survey Winona State University was one participant of 120 institutions of higher education (4,678 faculty members). WSU s participants made up 2% of the total respondents (100 faculty members were considered as valid sample number) to the survey. Approximately 52% of faculty members reported that they considered digital resources to be of great value to their instruction (Figure 3). The results from other institutions and Winona State University s result were very similar in distribution pattern. Figure 3. Value of Digital Resources Other Institutions WSU Unknown **I do not instruct students. Of no value to my instruction. 3% 2% 2% 2% 1% 1% Of some value to my instruction. 34% 43% Of great value to my instruction. 52% 60% There was limited information to compare dataa of other institutions to WSU s data because we only received WSU s raw data. Therefore, only figures regarding value of digital resources and barriers to using digital resources information from other institutions were requested for comparison with WSU s figures to evaluate similarities in data patterns. 7

8 Table 1. Barriers to Using Digital Resources (Ranked) I would use more digital resources if Rank WSU Other Institutions 1 More time More time 2 More using digital resources More using digital resources 3 More/better training available to use instructional technology More/better training available to use instructional technology 4 Technology were more dependable My institution rewarded me for using them 5 My institution rewarded me for using them 6 More flexibility in the content/curriculum of my classes Technology were more dependable More access to technology 7 More access to technology Greater priority at my institution 8 Greater priority at my institution More flexibility in the content/curriculum of my classes The survey indicated a few of the barriers to faculty members use of digital resources were identified (Table 1). The faculty members of both WSU and other institutions expressed as their top three barriers as: concerns about time, insufficient useful digital resources, and more or better training to use IT. Other barriers identified by faculty members were quality of technology, rewards from institutions, accessibility of technology, and priority issues at their institutions. The data also revealed the importance of digital resources features such as high quality, prompt access of information to organize, the role of teaching materials, and the role of professional development materials. Table 2 shows in a numerical rank how faculty members at Winona State University use digital resources. For example, online scholarly resources were 8

9 measured as the top ranked digital resources, followed by visual materials and historical documents. These findings of how the faculty members use each digital resource were very interesting: when they use online scholarly resources, the faculty members usually use them for their own scholarly work or professional development as a teacher. They use visual materials and historical documents for their presentation during lectures or classes. The faculty members also use teaching/learning activities by posting them on the course management tool with the recommendation that their students use them as study aids or review tools. Online datasets were used for problem-based learning projects; online simulations and animations, though infrequently used by faculty, were used for student guides. Faculty members provided a range of responses regarding questions of how much they developed or modified digital resources. IT (Information Technology) is worth noting that 11- % to 32% of faculty members modified the digital resources and 29- % to 34% of faculty members actually developed their own visual materials and historical documents, as well as teaching and learning activities. Table 2. Use of Digital Resources Based on Frequent and Occasional Use (Ranked) Winona State University Rank 1: Online Scholarly Resources (80 %) 1. For use in scholarly work 2. Professional development as a teacher 3. Recommended to students for review and/or a study aid 4. Problem-based learning assignment 5. Used in tests and quizzes 6. Presentation during lectures/class 7. Online class discussion 8. Post on course website/course management system. 9

10 Rank 2: Digital Images/Visual Material and Historical documents (75 %) 1. Presentation during lectures/classes 2. Professional development as a teacher 3. Post on course website/course management system 4. For use in scholarly work 5. Problem-based learning assignment 6. Used in tests and quizzes 7. Recommended to students for review and/or a study aid. 8. Online discussion Rank 3: Teaching or Learning Activities and Exercises (62 %) 1. Post on course website/course management system. 2. Recommended to students for review and/or a study aid. 3. Presentation during lectures/class 4. Problem-based learning assignment 5. Professional development as a teacher 6. Online class discussion 7. Used in tests and quizzes 8. For use in scholarly work Rank 4: Online Datasets (50 %) 1. Problem-based learning assignment 2. Presentation during lectures/class 3. Recommended to students for review and/or a study aid. 4. Professional development as a teacher 5. For use in scholarly work 6. Post on course website/course management system. 7. Online class discussion 8. Used in tests and quizzes Rank 5: Online Simulations/Animations (39 %) 1. Recommended to students for review and/or a study aid. 2. Presentation during lectures/classes 3. Post on course website/course management system. 4. For use in scholarly work 5. Used in tests and quizzes 6. Online discussion 7. Professional development as a teacher 8. Problem-based learning assignment 10

11 Student Assessment Day Survey The students D2L use survey was completed by freshmen and sophomores only (Table 3) with four global constructs: System quality, service quality, user satisfaction, and information quality. These constructs were developed based on the review of the literatures on web success and e-learning satisfaction (DeLone & McLean, 2003; Wang, Wang, & Shee, 2007) and each item of each construct was developed to understand the value and efficacy of course management system to provide a success model of using IT at WSU based on the faculty survey as a baseline for the current study. This survey measured three dimensions of success and identified that they are interrelated rather than independent. This has important implications for the measurement, assessment and analysis of a course management system as a part of IT. Furthermore, it is important because this study examined the experience of students who are presented with D2L as the course management system in their classes. In examining whether the users are satisfied with the proposed dimensions, it is to understand the issues about the use and non-use of IT at WSU. 11

12 Measuring Information Technology Use Figure 3. Response Rate on How many traditional courses that used D2L have you ever taken, including this semester? Never More than 10 2% 10% 15% 26% 47% Almost 98 percent of students who respondedd to the survey had experience of using D2L (Figure 3) in the traditional class setting (face-to-face). Table 3 shows how many students agree or strongly agree with each question related to system quality, service quality, information quality, and satisfaction. Figure 4 shows which areaa of D2L were the most helpful to students in the class. Content area (80%) and grade tools (76%) were identified as the most helpful area of D2L. After conducting the initial survey, a survey refinement process from the pooled data was conducted. The first step in refining the data was to conduct a reliability test: all items in the instrument had a reliability of (Cronbach s α). For the second step, an exploratory factor analysiss was conducted to examine the factor structure of the survey instrument. The sample data of 499 responses were examined using a principal axis factoring analysis as the extraction 12

13 technique and varimax as the orthogonal rotation method. As a result, three factors-system quality, service quality, and user satisfaction-were able to explain 78.8% of the variance in the dataset: items of each factor are provided in Table 4. Table 3. Student Assessment Day Survey D2L Use (Strongly Agree/Agree Response Rate) Questions Agree/Strongly Agree Once I learned how to use D2L, I was able to do things pretty efficiently. 82 % In my courses that used D2L, I was able to access course materials. 81 % It was easy to learn how to use D2L. 79 % I trust the information I received from the course through D2L. 79 % Overall, my instructors demonstrated good knowledge of the subject matter. 79 % Course materials (e.g., PowerPoint presentation files, document files, movie clips) contributed to my understanding of the course content. 78 % In my courses that used D2L, I was able to navigate easily to find what I needed to complete the course assignments/projects. 76 % D2L is user friendly. 74 % I am satisfied with my courses that used D2L. 74 % In my courses that used D2L, my instructors organized the course material in an effective manner through D2L. 73 % I believe D2L was used successfully in my courses. 72 % In my courses that used D2L, course materials in the course content area provided precise information I needed. 72 % In my courses that used D2L, I was satisfied with my instructors. 71 % In my courses that used D2L, my instructors outlined in reasonable detail course requirements and grading procedures through D2L. 69 % In my courses that used D2L, most materials from D2L contributed to my understanding of the course content. 68 % My instructors' grading in the courses that used D2L were fair and consistent. 68 % Assignments in the courses that used D2L were distributed evenly throughout the semester. 67 % Overall, my instructors could be contacted for consultation. 67 % Course assignments in D2L contributed to my understanding of the course content. 65 % I would like to see more D2L features used by my instructors in future 65 % 13

14 classes. My instructors were able to facilitate student participation in the course activities through D2L. 63 % I enjoyed the learning experience through D2L. 63 % Posted Quizzes in the Quizzes area of D2L contributed to my understanding of the course content. 61 % I was able to be actively engaged in the learning process through D2L. 61 % Graded assignments, test, etc., were returned promptly through D2L. 57 % Posted discussions in D2L contributed to my understanding of the course content. 57 % I was able to complete and submit assignments in less time in my D2L courses. 56 % I was able to use written communication in my D2L courses. 55 % D2L helped me to control when and where to learn. 55 % Courses using D2L stimulated my interest in the subjects. 48 % 14

15 Table 4. Refined Survey Items Service quality (SVQ): 1) My instructors grading in the courses used D2L was fair and consistent. 2) Assignments in courses using D2L were distributed fairly throughout the semester. 3) Graded assignments, test, etc., were returned promptly through D2L. 4) Overall, my instructors could be contacted for consultation. 5) Overall, my instructors demonstrated good knowledge of the subject matter. 6) My instructors were able to facilitate student participation in the course activity through D2L. 7) I trust the information I receive from the course through D2L. 8) In my course that used D2L, course materials in the course content area provided the precise information I needed. System quality (SQ): 1) In my courses that used D2L, I was able to navigate easily to find what I needed to complete the course assignments/projects. 2) In my courses that used D2L, I was able to access course materials. 3) D2L is user friendly. 4) It was easy to learn how to use D2L. 5) Once I learned how to use D2L, I was able to do things pretty efficiently. 6) In my courses that used D2L, overall my instructors organized the course material in an effective manner through D2L. User satisfaction (S): 1) I am satisfied with the course used D2L 2) I believe D2L was successful in the courses. 3) I enjoyed the learning experience through D2L. 4) I would like to see more D2L features used by my instructors in future classes. 15

16 Measuring Information Technology Use Figure 4. Most helpful areas in D2L 80% 76% 36% 15% 1 1% 9% 3% 3% 1% 1% 0% Based on the exploratory factor analysis, a confirmatory factor analysiss was conducted. To interpret the fit of the measurement model, the chi-square test, root-mean-square residuals (RMR), root-mean-square error of approximation (RMSEA), the normal fit index (NFI), and the comparative fit index (CFI) were examined (Table 5). All tests were indicated good fits based on the recommended ranges of each test: (RMSEA 0.08; NFI 0.90; CFI 0.95). As shown in Figure 5, a t-test was also conductedd because the path values were unstandardized values whereas t-values are standardized value. All path values were significant: the path values between service quality and user satisfaction (0.75), and between system quality and user satisfaction (0.19) had a strong and moderate path values, respectively. The structural equation is shown in Figure 5. 16

17 Table 5. Goodness-of-Fit Statistics Minimum fit function chi-square RMSEA NFI NNFI CFI *: good fit * 0.096* 0.98* 0.98* 0.98* Figure 5. Simplified Path Model and Structural Equations (S: User satisfaction, SQ: System quality SVQ: Service quality) SQ Path: 0.19 t-value: 4.98** (p=0.039) S SVQ Path: 0.75 t-value: 16.44** (p=0.045) S = 0.19 * SQ * SVQ, Error variance=0.18 (p=0.017), =0.82 Note: Significant level of t-value (> ±1.96, **p<0.05) The findings are fully supported by the strong positive relationship between service quality of D2L and the students satisfaction in using D2L, as well as the moderate relationship between system quality and students satisfaction in using D2L. In each case, the quality factors are correlated with the corresponding satisfaction factor. 17

18 CONCLUSIONS In this study, a faculty survey was conducted to understand how faculty members use digital resources and what barriers were preventing the use of them. In addition, a student survey was conducted to understand students perceptions of D2L as a placeholder for digital resource and as part of IT. A limitation of this study is the generalization of the model. The data were collected using a self-report survey, so test-retest reliability should be evaluated to increase internal consistency and to increase the stability of the survey instrument over time. As a result of this survey, service quality and system quality of digital resources were identified as two of the most important potential factors which impact the satisfaction of using information technology (in this study, it is D2L). For example, the degree to which the system is easy to navigate, easy to access, and easy to use heavily impacts the use of digital resources. Service qualities such as the instructors grading policy, the instructors knowledge, the organizational skill with digital resources, the providing of precise-information through digital resources and the encouragement of student participation were all extremely important to students for higher satisfaction with digital resources. Based on the result of this study, the positive impact of two factors will increase the satisfaction level of using IT: system quality and service quality in terms of instructors course delivery quality. These are linked to students successful learning in their courses and students having a positive learning experience. This study presented an empirically validated model for measuring effective and successful user satisfaction using IT (e.g., a course management system) by students. In addition faculty members perspectives on using digital resources were shown. A primary contribution of this 18

19 study is the development of a means to measure IT success at WSU. The survey and accompanying model provide not only an overall assessment but also the capability to investigate different aspects of IT resources at WSU. We will need to continuously build upon our understanding of user needs to ensure optimal use of IT. 19

20 REFERENCES DeLone, W. H., & McLean, E. R. (2003). The DeLone and McLean Model of Information System Success: A Ten-Year Update. Journal of Management Information Systems, Spring 2003, Vol. 19 (4) Wang, Y. S., Wang, H. Y., & Shee, D. Y. (2007). Measureing e-learning systems success in an organizational context: Scale development and validation. Computers in Human Behavior 23,