Quality Assessment and Quality Assurance in Online Learning

Quality Assessment and Quality Assurance in Online Learning Michael A. Mariasingam Quality Learning Global USA 28th Annual Conference on Distance Teaching & Learning, Madison August 8-10, 2012 Copyright 2012 The Board of Regents of the University of Wisconsin System 1

Session 1 Topics Current status of quality assessment and assurance in online learning New comprehensive approach to quality assurance Benchmarks and measurements for quality assessment and assurance Quality Frameworks Copyright 2012 The Board of Regents of the University of Wisconsin System 2

Focus on Limited Stakeholders Quality guidelines are based on a quality concept that has focus on limited stakeholders Stakeholders normally included: Learners, faculty, and institution Stakeholders generally not included: Employers, professional associations, society, government, and others Copyright 2012 The Board of Regents of the University of Wisconsin System 4

Limited Perspectives Guidelines have only the functional perspective learning effectiveness, student support, technology support etc Example: Sloan-C Five pillars of quality - - learning effectiveness, cost effectiveness, access, faculty satisfaction, and student satisfaction Copyright 2012 The Board of Regents of the University of Wisconsin System 5

Structure of Quality Frameworks Frameworks are two-level frameworks. First level : Functional categories, like learning effectiveness, as quality assessment categories. Example: Quality assessment category: Course Development Benchmarks Quality On the Line. (IHEP, 2000). Copyright 2012 The Board of Regents of the University of Wisconsin System 6

Structure of Quality Frameworks Second level : Guidelines as criteria for assessment under the quality assessment category. Example: Guideline: Courses are designed to require students to engage themselves in analysis, synthesis, and evaluation as part of their course and program requirements. Quality On the Line. (IHEP, 2000). Copyright 2012 The Board of Regents of the University of Wisconsin System 7

Major Characteristic Feature Quality frameworks, with few exceptions [Mariasingam (2006)], are guidelines, not benchmarks or standards although they are often called benchmarks. Example: The course or program provides for appropriate interaction between faculty and students and among students. (IHEP, 2000) This is just a guideline, not a benchmark or standard. Copyright 2012 The Board of Regents of the University of Wisconsin System 8

Levels of Assessment Guidelines, with a few exceptions, are defined to assess quality at program level Quality must be measured at multiple levels institutional level, program level, and course level. Copyright 2012 The Board of Regents of the University of Wisconsin System 9

Need for an Alternative Accreditation Model Quality assurance in distance learning should be sensitive to both learning principles common to all forms of higher education, and aspects of learning that are distance specific (Marginson 2002). Hence, a need for an alternative accreditation model that has a comprehensive quality assessment and assurance approach. Copyright 2012 The Board of Regents of the University of Wisconsin System 11

Alternative Customer Focused Quality Concept Education now a global economic commodity To be sold, must meet the requirements of all stakeholders Must have a quality assurance approach that has a business orientation Must have a customer focused industry oriented approach. Copyright 2012 The Board of Regents of the University of Wisconsin System 12

Multiple Dimensions of Quality (cf. Ehlers 2002a, 2002b, 2003a) Different meanings of quality Different perspectives Q Different levels of quality Copyright 2012 The Board of Regents of the University of Wisconsin System 13

Multiple Perspectives of Quality Quality, like beauty, is in the eye of the beholder (or the stakeholder) The achievement of quality can be managed and quality itself be credibly measured only when it is clear for what purpose the product or service is to be used and whose quality interests are to be served (Schweiger, 1996). There are multiple stakeholders in education; to be of quality, education should serve the interests of all stakeholders A new approach that meets the requirements from the perspectives of all stakeholders is needed. Copyright 2012 The Board of Regents of the University of Wisconsin System 14

Multiple Levels of Quality To be precise and meaningful quality must be measured at multiple levels institutional level, program level, and course level Quality measures defined for these three levels will be interrelated, will have some overlap but the breadth and depth of these performance measures will vary significantly with levels Quality frameworks for the three different levels will be different Copyright 2012 The Board of Regents of the University of Wisconsin System 15

Criterion Based Negotiated Approach Criterion referenced concept Quality is measured by indicators of the performance of individual institutions or of the worth of the products on their own rather than in comparison with others (Baumgart and Kaluge, 1987) Negotiated approach Quality assessment process includes both quantifiable measures and qualitative intrinsic values Copyright 2012 The Board of Regents of the University of Wisconsin System 16

Outcomes-based Approach In outcomes-based approach learning outcomes that are measured should include learning outcomes like learning skills at Kirkpatrick s behavior level [Kirkpatrick s four-level model: reaction, learning, behavior, and business results ] Education must provide learners the ability and skills necessary to transfer knowledge gained to real life situations. Copyright 2012 The Board of Regents of the University of Wisconsin System 17

Benchmarks A standard against which something can be measured Specify criteria that define the element(s) of quality to be assessed at multiple levels (e.g): Threshold Modal Best Copyright 2012 The Board of Regents of the University of Wisconsin System 19

Selection of Standard The standard selected could be: Absolute standard based on theory Examples: Quality Assurance Agency [UK] subject benchmarks http://www.qaa.ac.uk/ Mariasingam, M. A. (2006) Quality criteria and benchmarks for online degree programs (Paperback). Ann Arbor, MI: ProQuest / UMI Copyright 2012 The Board of Regents of the University of Wisconsin System 20

Selection of Standard Best practice Example: Master of Engineering in Professional Practice [MEPP] program (UW-Madison) Internal To suit the institution s, program s goals, contexts of learners, stakeholders. Given the constraints of the institution, program, the standard selected need not be either the ideal possible or the best practice, it would be internal institution s or program s specific benchmark. Copyright 2012 The Board of Regents of the University of Wisconsin System 21

Benchmarking process Eight step process: Decide what to benchmark Decide the level at which to define the benchmark Select the standard absolute, best practice, or internal Identify the current level of quality Identify the gap in quality Identify and take the necessary steps to improve quality Monitor the implementation process and ensure quality Adopt the culture of CQI Copyright 2012 The Board of Regents of the University of Wisconsin System 22

1: Define Quality Criteria and Benchmarks: They should: address the requirements of all stakeholders address the different perspectives of each stakeholder be measurable include metrics to determine the extent to which the program meets the quality criteria Copyright 2012 The Board of Regents of the University of Wisconsin System 24

2: Organize Specific Program Processes So that the program: Meets the defined quality criteria and benchmarks at all levels Addresses key components of program planning, program implementation, and program quality assessment and assurance Includes multiple perspectives when considering these key components Copyright 2012 The Board of Regents of the University of Wisconsin System 25

3: Establish: A monitoring system to ensure that the steps and processes specified in Step 2 are implemented A system for periodic review to ensure that processes are functioning as intended. Copyright 2012 The Board of Regents of the University of Wisconsin System 26

4: Maintain, Monitor, and Improve: A self-assessment process to determine that all quality assurance steps taken in [1] to [3] are yielding intended program outcomes Processes for correcting any deficiencies that the self-assessment process might reveal. Copyright 2012 The Board of Regents of the University of Wisconsin System 27

Quality Framework at Program Level Quality Framework Mariasingam, M. A. (2006) Quality criteria and benchmarks for online degree programs (Paperback). Ann Arbor, MI: ProQuest / UMI. Copyright 2012 The Board of Regents of the University of Wisconsin System 30

Framework Features Alternative quality concept Wider role for education Multiple dimensions of quality Multiple perspectives of quality Multiple levels of quality Criterion-based approach to quality assessment Negotiated approach to quality assessment Outcomes-based approach Copyright 2012 The Board of Regents of the University of Wisconsin System 31

Quality Framework at Institution Level McKinnon, K R., Walker, S H., & Davis, D. (2000). Benchmarking: A manual for Australian Universities. Canberra, ACT: Higher Education Division, Department of Education, Training and Youth Affairs. February 2000. http://www.dest.gov.au/archive/highered/otherp ub/bench.pdf Copyright 2012 The Board of Regents of the University of Wisconsin System 32

Quality Framework at Institution Level - Another Framework Quality Improvement Framework Inglis, A., Ling, P., and Joosten, V. (2002). Delivering Digitally: managing the transition to the knowledge media (2nd ed.). London: Kogan Page. Copyright 2012 The Board of Regents of the University of Wisconsin System 33

Choosing Between the Frameworks The two frameworks have slightly different purposes: The Benchmarking Framework was intended to enable universities to make comparisons across the full range of institutional functions. The Quality Improvement Framework has a focus on the aspects of an institution s functions that are concerned with teaching, learning, and student support. For more information see: Inglis, A. (2005). Quality Improvement, Quality Assurance, and Benchmarking: Comparing Two Frameworks for Managing Quality Processes in Open and Distance Learning. International Review of Research in Open and Distance Learning, Vol 6, No1 March 2005. http://www.irrodl.org/index.php/irrodl/article/view/221/304 Copyright 2012 The Board of Regents of the University of Wisconsin System 34

Quality Framework at Course Level Michigan Virtual University (2002). Standards for quality online courses. http://standards.mivu.org/overview/ Copyright 2012 The Board of Regents of the University of Wisconsin System 35