Concentric Model for Evaluating Internet-Based Distance Learning Programs. Elizabeth Reed Osika Director, User Services Purdue University Calumet

Transcription

1 Concentric Model for Evaluating Internet-Based Distance Learning Programs Elizabeth Reed Osika Director, User Services Purdue University Calumet Denise Camin Assistant Professor DeVry University Evaluation in Distance Learning With distance learning becoming more prevalent as an option for obtaining an education, many schools are rushing to get their courses online and available to students (Institute for Higher Education Policy, 2000). However, is anyone stopping to ask if what is in place is a quality program? If so, what is the definition of quality and what aspects of the program are being evaluated? According to the current literature, the majority of articles on the assessment and evaluation of distance learning programs focus on single elements and fail to take into consideration the complex environment necessary for a quality program. Even titles that convey a holistic approach to evaluation only address a specific issue. For example, Benigno & Trentin s (2000) article, The Evaluation of Online Courses focuses on assessing student participation through analyzing the students messages; Gunawardena, Lowe and Carabajal s (2000) article, Evaluating Online Learning: Models and Methods focuses on methods for evaluating discussion groups, and Robson s (2000) article Evaluating Online Teaching focuses only on the issue of human computer interface. Moreover, only a few articles actually provide insight into the complexity involved in evaluating an entire distancelearning program. One of the best reports, in regards to explaining the breadth of the issue is Quality on the Line: Benchmarks for Success in Internet-Based Distance Education by the Institute for Higher Education Policy (2000). This report did a thorough review of the literature, selected six institutions that had reputations as leaders in distance education, and conducted interviews and surveys of students, faculty, and administrators at each of the schools. Other articles that provide insight into the realm of a holistic program evaluation are Benke, Brigham, Jarmon and Paist (2000) article, Rethinking Quality Assurance: Examining Established Practices, Exploring New Strategies, which offers valuable insight into four of the major elements faculty, course development, student support, and technology considerations and Dasher-Alston s (1999) article, Evaluation Criteria for Distance Learning, which focuses on academic planning yet provides additional information on other aspects relating to quality. While each of these articles present valuable information into the aspects required in a quality distance learning program, none provide all the aspects we, as researchers, feel should be addressed and none provide a visual model that illustrated how the elements in question related to one another. Therefore, the Concentric Model for the evaluation of quality distance learning programs was created to meet this need and is now in the process of being validated. Defining a Quality Distance Learning Program For more resources click here -> In order to develop an evaluation model for distance learning programs, a clear definition of what was meant by a quality distance-learning program had to be established. A survey of people interested in the field of distance learning (as determined by membership of listservs dealing with distance learning issues) was conducted to define a quality distance learning program and list the key elements a quality program would contain. After compiling the results and incorporating elements deemed as critical from the researchers point of view the following definition was created and used as the foundation of the model: 1 A quality distance-learning program focuses on and supports the needs of the people it is intended to serve. Therefore, it has Copyright at its core 2005 The the Board interaction of Regents between of the University faculty of and Wisconsin students, System. surrounded by pedagogically appropriate Duplication content or redistribution presented prohibited through without a stable written technology permission of platform the author(s) that is supported,

2 For more resources click here -> both technically and programmatically, to provide knowledge and/or training that is accepted and desired by the larger community. When reviewing the various definitions of a quality distance learning program as provided by the participants of the survey, it was interesting to note that the most frequently listed components focused on the needs required by the faculty and students of a specific course. Within the various definitions, the support needs required to sustain a program were frequently absent. The reason for this can only be hypothesized at this point. Was it because the respondents were primarily faculty and were not aware or choose not to emphasize the larger picture of supporting an entire program? Is it a weakness in the data collection process, in that technical support and administrators did not comprise a large portion of the sample? Could it be that the elements outside the everyday classroom are not important in the assessment of quality in a distance-learning program? Only further analysis and validation of this definition will tell. Development of a Model for Evaluation of Distance Learning Programs Using the above definition as the foundation for a model that will assess and evaluate distance learning programs, the researchers wanted to capture and build upon their opinion, and that of the participants they surveyed, that for a distance learning program to be successful it had to focus and revolve around the people it intended to serve - primarily the students who are learning but also the faculty who are teaching. In addition, the model needed to position the other elements critical to the success of the program around the core of people. The model (see figure 1) begins with a core symbolizing the student and faculty member. Each half of this core is equally important, as the success of the student learning depends on the success of the teacher in providing instruction. Supporting the core is a ring indicating the interaction of the students and teachers with the content of the course. The content is surrounded by a course management system, which enables the content to be managed by the instructor and served up to the students. We are not trying to presuppose that a quality distance learning program needs to have one of the large commercial course management systems; we are only suggesting that the content needs to be managed through a system for consistency and security of the information. The content, faculty, and students need to be supported by a team of technical staff, who are available to assist in troubleshooting problems, providing training and instruction, and attending to the server and other technology. This enables individual courses and students to be successful, at least in terms of technical expertise. This is where the majority of the research ends, in terms of evaluation. However, we suggest that there are two additional levels that need to be considered. First, a quality program will need programmatic support for the faculty and students. This means that clear support from the executive leadership needs to be in place, which will help enable and support faculty needs (e.g. release time for development, importance of distance learning to promotion and tenure, etc.) and the needs of the students (e.g. online services such as registration and payment, research resources, etc.). Finally, there has to be community support for the program. Questions such as is the degree accepted by industry as a valid from of education, can students receive financial aid, and what is the general opinion of the public for online education need to be addressed. 2

3 For more resources click here -> Figure 1: Concentric Model for evaluating distance-learning programs Required Elements for Evaluation of a Distance Learning Program After the various rings of support were established, the researchers reviewed the literature and pulled from their own experience to determine the various aspects within each area that should be evaluated. This initial analysis yielded 42 critical elements (see Appendix A). These elements were initially validated by comparing them to the results provided by the survey participants. Upon review, we noticed a large omission from our initial list in that we failed to indicate the importance of addressing ADA compliance when developing content. Thus, category 14a was created to address this critical issue. Use of the Model The model is intended to help involved parties analyze the quality of their program by indicating whether each element is something in which they excel, need more work, or are lacking. This will provide an indication as to where effort should be placed for improvement, by starting from the inside of the rings and working out. Additionally, the model can be used to help establish a distance-learning program by using the critical elements to assess the feasibility of success, starting at the outer most rings and working their way in. Future Plans and Directions The next phase in the development process will revolve around the validation of the model through the use of expert reviewers and further review of the literature. During the validation process we will be looking to make sure that all the required rings of support are indicated and all critical elements are defined. If you are interested in participating as an expert reviewer, please contact Elizabeth Osika. References Benigno, V., & Trentin, G. (2000). The Evaluation of online courses. Journal of Computer Assisted Learning, 16(3) Benke, M., Brigham, D., Jarmon, C., & Paist, E. (2000). Quality indicators for distance education programs. Rethinking Quality Assurance: Examining Established Practices, Exploring New Strategies. Selected papers presented at the Annual Accreditation on Quality Assurance Conference. Philadelphia, PA: Middle States Commission on Higher Education. Dasher-Alston, R. M., & Patton, G. W. (1998 Fall). Evaluation criteria for distance learning. Planning for Higher Education, 27(1)

4 Gunawardena, C., Lowe, C., & Carabajal, K. (2000). Evaluating online learning: Models and methods. Paper included in the Proceedings of the Eleventh Society for Information Technology and Teacher Education International Conference. San Diego, CA. Institute for Higher Education Policy. (2000). Quality on the line: Benchmarks for success ininternet-based distance education. Washington, D.C: Author Robson, J. (2000). Evaluating on-line teaching. Open Learning, 15(2), Biographical Sketches Elizabeth Reed Osika is the Director of User Services at Purdue University Calumet in Hammond, Indiana. She has been working in the field of distance learning since 1998, when her institution decided to begin offering a course management system to the faculty. From that point in time, she has been heavily involved in the training, support, and administration of the school s distance learning program. Elizabeth is currently working on her doctoral degree in Education Technology at Purdue University in West Lafayette. This presentation is part of her dissertation activities. Address: Purdue University Calumet, th Street, Gyte 134, Hammond, Indiana, osika@calumet.purdue.edu Phone: Fax: Denise Camin is currently an Assistant Professor of English for DeVry University. She has been interested in the field of distance learning since her first encounter with the Faculty Instructional Technology Services (FITS) center at Purdue University Calumet when she was a visiting instructor in the English dept. there. Denise is also pursuing her doctoral degree in Educational Technology at Purdue University. Address: DeVry University, West Creek Drive, Tinley Park, IL dcamin@tp.devry.edu Phone: Fax: For more resources click here -> 4

5 Faculty 1. Technical Competency 2. Knowledge of online pedagogy 3. Behavioral skill set/motivation 4. Technology available Student 5. Technical competency 6. Behavioral skill set / motivation 7. Technology available Appendix A: Listing of Critical Components in the Concentric Model Content 8. Interaction between students and students 9. Interaction built in the class between faculty and students 10. Design criteria (look, navigation) 11. Clear objectives and goals (good instructional design) 12. Assessment / evaluation is consistent with objectives and goals 13. Learner centered vs. content centered (active classroom) 14. Does the content make use of the online capabilities 14a.Adherence to ADA standards for accessibility Course Management System 15. Easy to use for faculty and students 16. Feature set available 17. Allows for interactivity 18. Administrative overhead that is required (money, people, time) 19. Consistent format 20. Visually appealing Technical Support Student 21. Software support (training on Word, PPT, , etc.) 22. Problem resolution - online and in person Faculty 23. Multimedia development assistance 24. Copyright assistance/policy 25. Training on course management system and pedagogy 26. Individual consulting 27. Problem resolution - online and in person Infrastructure 28. System administration support 29. Stable, robust infrastructure to support technology For more resources click here -> 5

6 Programmatic Support Faculty 30. Commitment from executive leadership/clear direction 31. Release time for faculty development 32. Important to promotion and tenure 33. Peer assistance and mentoring 34. Policy on Intellectual property issues Student 35. Online registration 36. Online payment 37. Online support services (advising, etc.) 38. All courses (pre-reqs and required) online or transferable 39. Research resources (library services) Community Support 40. Acceptance of degree by industry (recruitment, job placement) 41. Financial aid for program 42. Public opinion / impression of the validity of online services For more resources click here -> 6