1 20th Annual Conference on Distance Teaching and Learning click here -> Evaluating Pedagogical Quality: An Integral Part of Course Development Strategy L Bagley, PhD Head of Learning Research and Development Ufi Ltd Ufi Ltd was created in 1998 to bring about the UK government s vision of a university for industry. It is a unique partnership between government and the private and public sectors. Within a year of becoming fully operational in October 2000, it had created with its partners - learndirect - the largest publicly-funded online learning service in the UK. Today there are over 1.2 million registered learners and 21,000 new course enrolments every week. The vision of Ufi is that, through its learndirect network, the organisation will act as a significant catalyst for change in the lifelong learning market. Its remit is to widen participation in lifelong learning, in particular for those individuals and employers who are impeded by economic, motivational, physical or learning barriers, and improve the employability of individuals and the competitiveness of businesses through the provision of high quality online learning opportunities. Ufi/learndirect offers a fully supported online learning experience through a network of partnerships with its education, industry and community partners staffing over 2000 learning centres throughout England, Wales and Northern Ireland. These learning centres provide attractive learning environments, offering high specification computers, access to the Internet and staff trained to support learners on technical, administration, learning and social matters. In addition there is a network of over 1000 trained online tutors to provide expert course support. A national telephone support line is also available 24 hours a day, seven days a week. Quality Ufi/learndirect is committed to an evidence-based approach to refining its learning model and improving the quality of its courses. Quality can be defined as fitness for purpose as expected by the relevant stakeholders. This definition however appears too simplistic when one attempts to define the quality of learning software. Alley and Jansak (2000) present an integrated approach to quality in online courses as a pyramid with the principles of learning science independent of the delivery medium at the base, practice (in instructional artistry) which dictates the nature of delivery on the next level, and the application or system by which the delivery is achieved at the top. Thus the pedagogic principles should underpin both the medium and the message. Freeman (1993:60) regards quality assurance as encompassing more than the product of the learning, stating the product of training and education is in two distinct parts: process (the experience of learning); product (what you have achieved at the end). For the purposes of this paper, learning materials are accepted as high quality when learners learn what they wanted and expected to learn in a way that enables them to put their learning into practice. In addition, after a quality learning experience learners should be motivated to engage in further similar learning experiences. This paper summarises the ways in which Ufi/learndirect addresses the issue of pedagogic quality, concentrating on one specific learner-centred quality management tool, QUAL-IT, which was developed during 2000-2002 as part of the Ufi Improving Learning Project carried out by Professor S Geertshuis and her team at the University of Wales Bangor and the University College Northampton. The pedagogy of the online courses is currently addressed in various ways during the production cycle. Before the materials are commissioned, prospective suppliers must
2 20th Annual Conference on Distance Teaching and Learning click here -> demonstrate their ability to develop interactive learning materials that enable learners to test their progress as they work through the materials and consolidate their learning. A handbook and templates stipulating all the pedagogical requirements, including those for design, presentation and content, is available online. Materials are tested qualitatively with learners using accompanied surfing techniques before final sign-off. This ensures that the pedagogical approach is appropriate for the learners as well as identifying inaccuracies and inconsistencies in the content. Any amendments required are agreed with the suppliers and incorporated. This methodology is supplemented with a more quantitative technique which enables comparisons to be made between courses by assigning each course a numerical score along several dimensions. This methodology is described below. Practitioners at Ufi/learndirect worked with the academic team led by Professor Geertshuis between 2000 and 2003 to test and further develop a quality tool. The tool was intended to assess the quality of computer-based learning materials thereby informing and empowering commissioners, purchasers and the users of the materials. The explicit focus of the project was the pedagogical perspective, rather than the technical or administrative concerns associated with computer-based learning and online delivery. QUAL-IT is essentially an online questionnaire, comprising eight sections each addressing a different dimension of e-learning. Each section asks learners to respond to between four and seven statements from a database. namely: user friendliness (eg, it was easy to start up the course ); match to learner (eg, the course suited my level of knowledge ); clarity and content quality (eg, the course clearly explained what I would learn ); multimedia presentation qualities (eg, there was an appropriate balance between graphics and text for me ); opportunities for active learning (eg, the course gave me lots of chance to practice what I was learning ); engagement (eg, the course material was interesting ); feedback and assessment (eg, I had the right amount of assessment to tell me how I was getting on ); and learning outcomes (eg, the course increased my knowledge of the subject matter ). (Originally there were 11 sections; three were merged with others after the first pilot study with learndirect learners.) Each statement is positively framed and has a seven-scaled response option plus a not applicable option; for example, under user friendliness, it was easy to start up the course, learners could respond by selecting one of the following options: very strongly agree; strongly agree; agree; neither; disagree; strongly disagree; very strongly disagree; not applicable. Learners are able to make comments at the beginning of the questionnaire regarding technical problems or other generic problems they encountered whilst working through the materials in order to identify any problems outside the scope of the course materials. Once submitted the learner responses are recorded, and the learner receives feedback on the answers and comments given. An opportunity is then given for them to confirm or deny their choice. Once all responses have been received a copy of their final report is delivered to the learner. QUAL-IT includes a secure administrator s page enabling staff to add, edit or delete courses for evaluation. All gathered data, such as overall course scores against the eight dimensions expressed as percentages, general comments, specific course comments, are accessible through the administrator s page.
3 20th Annual Conference on Distance Teaching and Learning click here -> Evaluation Phase 1 (2000-01) The original QUAL-IT instrument was tested in two main studies (Geertshuis et al, 2000). One used a sample of 22 learners who worked through a pre-selected set of learndirect courses. This enabled reliability and validity to be assessed. The results of the reliability and validity testing were positive with pedagogical quality dimensions correlating highly with self-reported learning outcomes. The study was successful in that it confirmed that the instrument was reliable, as assessed by Cronbach alpha scores. The second was an implementation study using an opportunity sample of 150 learndirect learners across a range of learning centres and engaged with a course of their choice. The results of the study indicated that the sample was on average satisfied with the quality of their courses and their learning outcomes. Overall the reliability in study one was maintained despite the diverse sample. The results of both studies served to confirm the importance of capturing quantitative and qualitative data. The former enables direct comparison to be made between courses. The latter provides an insight into how differences in perception arise. Phase I concluded that the QUAL-IT instrument could be used in three ways: during the development of courses; after development, but prior to acceptance by Ufi, or as a means of routinely monitoring learner perceptions. Given the process involved in commissioning online learning materials and the complexity of the technical testing processes involved, Ufi indicated that the most appropriate application of QUAL-IT would be when the developers had signed off the content and any technical bugs and interworking issues had been corrected. In this way the learners experience would not be influenced by issues other than pedagogical ones. The following two recommendations of Phase I were implemented: the instrument was shortened by 15% and the layout was adapted to include a rollover help facility with sound to assist learners completing the instrument without support. Phase 2 (2001-02) Initial in-depth testing with a small number of learners (6) investigated the user friendliness of the package from the learner s viewpoint and the ease of administration from the administrator s viewpoint. The results of this phase led to a series of refinements, like the facility to insert a progress indicator showing the individual s advancement through the QUAL-IT questionnaire. The main study tested the QUAL-IT tool with 200 learners representing a cross-section of Ufi s target audience. Learners worked through one of 23 courses and accessed QUAL-IT either from home, via a learndirect learning centre or from a college. On recruitment, learners were emailed a web address where they could register and be entered into a prize draw. Those failing to complete the questionnaire within the designated period were sent a reminder and a fast track form to reduce completion time.
4 20th Annual Conference on Distance Teaching and Learning click here -> Results The main result was the confirmation not only of the validity of QUAL-IT as a tool to assess the pedagogic quality of learndirect s online learning materials, but also of its suitability for implementation into the organisation s course development cycle. QUAL-IT was straightforward to use and was accessed, completed and submitted remotely by all learners. Significantly, the majority of learners required no additional support to complete the questionnaire. By reducing the number of items included in the instrument, the time taken to complete the questionnaire was similarly reduced to a maximum of 10 minutes. The results also confirmed that the qualitative data received from learners were important in further illuminating the quantitative data and should therefore remain a vital component in the measurement of pedagogic quality. This is true of the comments invited at the beginning of the questionnaire on technical or general concerns as much as those comments relating to specific aspects of the course content. Moreover, the qualitative and quantitative data supported each other with a greater number of comments being attributed to those courses with the lower quantitative ratings. Further enhancements to the administrator s page to ease the interpretation of data have been recommended, for example the inclusion of additional learner identifiers, and have been incorporated. This function will be developed by Ufi on an on-going basis as QUAL-IT continues to be used as a key part of the organisation s piloting process. Implementation The final stage of the research concentrated on the day-to-day integration of QUAL-IT into Ufi s course development cycle. This has been easy to achieve given the inherent flexibility of QUAL-IT; QUAL-IT was designed to measure the pedagogical effectiveness of discrete learning episodes irrespective of length. All courses are evaluated before their release with learners using QUAL-IT, some in their entirety and others in specific chunks, depending on the intended outcomes of the course and the target audience. The feedback to date has been extremely constructive in helping direct suppliers to make improvements to courses based on qualitative and quantitative research evidence from learners. Conclusion Ufi/learndirect is committed to investigate ways in which the pedagogical quality of its online courses can be improved. QUAL-IT provides the organisation with an opportunity to benchmark its learning materials, highlighting particular areas and issues which need further attention from course designers and content specialists. It is clear from the research carried out during both Phase 1 and Phase 2 of this project and its subsequent implementation that QUAL-IT will make a valuable contribution to Ufi s commitment to adopt a rigorous approach to the development of quality standards for e-learning resources, materials and technology platforms (Ufi Ltd 2002). Acknowledgements The author would like to acknowledge the research carried out by Professor Geertshuis and her team at the University of Wales Bangor and the University College Northampton in developing and refining QUAL-IT as part of this Ufi-funded project.
5 20th Annual Conference on Distance Teaching and Learning click here -> References Alley, L. & Jansak, K. (2001) Ten Keys to Quality Assurance and Assessment in Online Learning Worldclass Strategies 20pp http://www.worldclassstrategies.com/papers/keys.htm Freeman, R. (1993) Quality Assurance in Training and Education: How to Apply BS5750 (ISO 9000) Standards London: Kogan Page Geertshuis, S., Geertshuis, G., Willis, R., Bagley, L. (2001) Pedagogical Quality in Online Learning Internal Project Report written by University College Northampton and University of Wales Bangor for Ufi Ltd. http://www.ufi.com Ufi Limited, (2002) Transforming Learning: Changing Lives. Working through learndirect to Expand the Learning Market Strategic Plan (2002-05) http://www.ufi.com Biographical Sketch Dr Lindsey Bagley is currently Head of Learning Research and Development at Ufi Ltd. Before joining the organisation in June 2000 she worked in the Higher Education sector specialising in distance education provision at the University of Manchester and also The Open University. Formerly, she worked in the commercial sector, developing distance learning materials for large blue-chip companies and as a training advisor at Rank Xerox (UK) Ltd where she undertook her PhD in 'The Design, Implementation and Evaluation of Distance Learning Materials'. Dr Bagley also has more recently gained an MA in Online Learning from the Open University. Address: Ufi Ltd 1 Young Street Sheffield S1 4UP E-mail: lbagley@ufi.com URL: www.ufi.com Phone: 00 44 114 291 5304