A Guide to Higher Education and the Audit Team

Transcription

1 Open University MARCH 2004 Institutional audit

2 Published by Quality Assurance Agency for Higher Education Southgate House Southgate Street Gloucester GL1 1UB Tel Fax Web Quality Assurance Agency for Higher Education 2004 ISBN All the Agency's publications are available on our web site Printed copies are available from: Linney Direct Adamsway Mansfield Nottinghamshire NG18 4FN Tel Fax [email protected]

3 Preface The Quality Assurance Agency for Higher Education (the Agency) exists to safeguard the public interest in sound standards of higher education (HE) qualifications and to encourage continuous improvement in the management of the quality of HE. To do this the Agency carries out reviews of individual HE institutions (universities and colleges of HE). In England and Northern Ireland this process is known as institutional audit. The Agency operates similar but separate processes in Scotland and Wales. The purpose of institutional audit The aims of institutional audit are to meet the public interest in knowing that universities and colleges are: providing HE, awards and qualifications of an acceptable quality and an appropriate academic standard; and exercising their legal powers to award degrees in a proper manner. Judgements Institutional audit results in judgements about the institutions being reviewed. Judgements are made about: the confidence that can reasonably be placed in the soundness of the institution's present and likely future management of the quality of its programmes and the academic standards of its awards; the reliance that can reasonably be placed on the accuracy, integrity, completeness and frankness of the information that the institution publishes, and about the quality of its programmes and the standards of its awards. These judgements are expressed as either broad confidence, limited confidence or no confidence and are accompanied by examples of good practice and recommendations for improvement. Nationally agreed standards Institutional audit uses a set of nationally agreed reference points, known as the 'academic infrastructure', to consider an institution's standards and quality. These are published by the Agency and consist of: The framework for higher education qualifications in England, Wales and Northern Ireland (FHEQ), which include descriptions of different HE qualifications; The Code of practice for the assurance of academic quality and standards in higher education; subject benchmark statements, which describe the characteristics of degrees in different subjects; guidelines for preparing programme specifications, which are descriptions of the what is on offer to students in individual programmes of study. They outline the intended knowledge, skills, understanding and attributes of a student completing that programme. They also give details of teaching and assessment methods and link the programme to the FHEQ. The audit process Institutional audits are carried out by teams of academics who review the way in which institutions oversee their academic quality and standards. Because they are evaluating their equals, the process is called 'peer review'. The main elements of institutional audit are: a preliminary visit by the Agency to the institution nine months before the audit visit; a self-evaluation document submitted by the institution four months before the audit visit; a written submission by the student representative body, if they have chosen to do so, four months before the audit visit; a detailed briefing visit to the institution by the audit team five weeks before the audit visit; the audit visit, which lasts five days; the publication of a report on the audit team's judgements and findings 20 weeks after the audit visit. The evidence for the audit In order to obtain the evidence for its judgement, the audit team carries out a number of activities, including: reviewing the institution's own internal procedures and documents, such as regulations, policy statements, codes of practice, recruitment publications and minutes of relevant meetings, as well as the self-evaluation document itself; reviewing the written submission from students; asking questions of relevant staff; talking to students about their experiences; exploring how the institution uses the academic infrastructure. The audit team also gathers evidence by focusing on examples of the institution's internal quality assurance processes at work using 'audit trails'. These trails may focus on a particular programme or programmes offered at that institution, when they are known as a 'discipline audit trail'. In addition, the audit team may focus on a particular theme that runs throughout the institution's management of its standards and quality. This is known as a 'thematic enquiry'. From 2004, institutions will be required to publish information about the quality and standards of their programmes and awards in a format recommended in document 02/15 Information on quality and standards in higher education published by the Higher Education Funding Council for England. The audit team reviews progress towards meeting this requirement.

4 Contents Summary Introduction 1 Outcome of the audit 1 Features of good practice 1 Recommendations for action 1 Programmes leading to the named awards of BA (Hons) Language Studies (with a focus on Spanish and German elements); BSc (Hons) IT and Computing; BEng and MEng; MBA 1 National reference points 2 Main report 4 Section 1: Introduction: the Open University 4 Mission statement 5 Collaborative provision 5 Background information 5 The audit process 6 Developments since the previous academic audit 6 Section 2: The audit investigations: institutional processes 7 The institution's view as expressed in the SED 7 The institution's framework for managing quality and standards 8 The institution's intentions for the enhancement of quality and standards 10 Internal approval, monitoring and review processes 11 External participation in internal review processes 14 External examiners and their reports 14 External reference points 15 Programme-level review and accreditation by external agencies 16 Student representation at operational and institutional level 17 Feedback from students, graduates and employers 17 Progression and completion statistics 18 Assurance of the quality of teaching staff, appointment, appraisal and reward 18 Assurance of the quality of teaching through staff support and development 19 Assurance of the quality of teaching delivered through distributed and distance methods 20 Learning support resources 20 Academic guidance, support and supervision 21 Personal support and guidance 23 Section 3: The audit investigations: discipline trails 24 Discipline audit trails 24 Section 4: The audit investigations: published information 30 The students' experience of published information and other information available to them 30 Reliability, accuracy and completeness of published information 31 Findings 34 The effectiveness of institutional procedures for assuring the quality of programmes 34 The effectiveness of institutional procedures for securing the standards of awards 35 The use made by the institution of the academic infrastructure 36 The effectiveness of institutional procedures for supporting learning 36 Outcomes of discipline audit trails 36 The utility of the SED as an illustration of the institution's capacity to reflect upon its own strengths and limitations, and to act on these to enhance quality and standards 37 Commentary on the institution's intentions for the enhancement of quality and standards 38 Reliability of information 38 Features of good practice 38 Recommendations for action 38 Appendix 40 The Open University's response to the audit report 40

5

6 Institutional Audit Report: summary Summary Introduction A team of auditors from the Quality Assurance Agency for higher education (the Agency) visited the Open University (the University) from 22 to 26 March 2004 to carry out an institutional audit. The purpose of the audit was to provide public information on the quality of the opportunities available to students and on the academic standards of the awards that the University offers. To arrive at its conclusions the audit team spoke to members of staff and current students, at the University's main campus and at four regional centres, and read a wide range of documents relating to the way the University manages the academic aspects of its provision. The words 'academic standards' are used to describe the level of achievement that a student has to reach to gain an academic award (for example, a degree). It should be at a similar level across the UK. Academic quality is a way of describing how well the learning opportunities available to students help them to achieve their award. It is about making sure that appropriate teaching, support, assessment and learning opportunities are provided for them. In institutional audit, both academic standards and academic quality are reviewed. Outcome of the audit As a result of its investigations, the audit team's view of the University is that: broad confidence can be placed in the soundness of the University's current and likely future management of the quality of its programmes and the academic standards of awards. Features of good practice The audit team identified the following areas as being good practice: the way that the University monitors the security of its academic standards through the Results Ratification and Awards Classification Panel; the systematic and comprehensive collection and use of feedback from students; the arrangements for appointing, monitoring and supporting associate lecturers; the proactive stance taken by the University in giving academic guidance and support to students; and the third-party monitoring system for research students. Recommendations for action The audit team also recommends that the University should consider further action in a number of areas to ensure that the academic quality and standards of the awards it offers are maintained. The team advises the University to: take the opportunity offered by the governance review to articulate more clearly, for the benefit of the University community as a whole, the University's approach to assuring and enhancing the quality of provision; review the effectiveness of its procedures for determining whether intended programme learning outcomes will be met through all pathways leading to named awards; systematically include external subject expertise in the procedures for approving programmes leading to named degrees; consider the effectiveness of its present system for gaining a University-level overview of annual review activity at course and programme levels; and reflect on the possible consequences of using different versions of programme specifications for different audiences. The University may also wish to consider the desirability of enhancing its quality management arrangements by: ensuring that research students have clear information about their financial entitlements. Programmes leading to the named awards of BA (Hons) Language Studies (with a focus on Spanish and German elements); BSc (Hons) IT and Computing; BEng and MEng; MBA To arrive at these conclusions, the audit team spoke to staff and students, and was given information about the University as a whole. The team also looked in detail at the programmes listed above to find out how well the University's systems and procedures were working at programme level. The University provided the team with documents, including student work and, here too, the team spoke to staff and students. As well as supporting the overall confidence statement given above, the team was able to state that the standard of student achievement in these programmes was appropriate to the titles of their awards and their place in The framework for higher education qualifications in England, Wales and Northern Ireland. The team considered that the quality of learning opportunities available to students in each of the programmes was suitable for a programme of study leading to the named award. page 1

7 Open University National reference points To provide further evidence to support its findings, the audit team also investigated the use made by the University of the academic infrastructure which the Agency has developed on behalf of the whole of UK higher education. The academic infrastructure is a set of nationally agreed reference points that help to define both good practice and academic standards. The audit found that the University was making effective use of the academic infrastructure to inform its framework for the management of quality and standards. From autumn 2004 the Agency's audit teams will comment on the reliability of the information about academic quality and standards that institutions will be required to publish, and which is listed in the Higher Education Funding Council for England's document 03/51, Information on quality and standards in higher education: Final guidance. The audit found that the University was preparing appropriately for the publication of the required information. page 2

8 Main report

9 Open University Main report 1 An institutional audit of the Open University (the University, or the OU) was undertaken during the period 22 to 26 March The purpose of the audit was to provide public information on the quality of the University's programmes of study and on the discharge of its responsibility as an awarding body. 2 The audit was carried out using a process developed by the Quality Assurance Agency for Higher Education (the Agency) in partnership with the Higher Education Funding Council for England (HEFCE), the Standing Conference of Principals (SCOP) and Universities UK (UUK), and has been endorsed by the Department for Education and Skills. For institutions in England, it replaces the previous processes of continuation audit, undertaken by the Agency at the request of UUK and SCOP, and universal subject review, undertaken by the Agency on behalf of HEFCE, as part of the latter's statutory responsibility for assessing the quality of education that it funds. 3 The audit checked the effectiveness of the University's procedures for establishing and maintaining the standards of academic awards; for reviewing and enhancing the quality of the programmes of study leading to those awards; for publishing reliable information; and for the discharge of its responsibility as an awarding body. As part of the audit process, according to protocols agreed with HEFCE, SCOP and UUK, the audit included consideration of examples of institutional processes at work at the level of the programme, through four discipline audit trails (DATs), together with examples of those processes operating at the level of the institution as a whole. The audit did not include the University's collaborative provision. Section 1: Introduction: the Open University 4 The University was founded by Royal Charter in 1969, and began teaching students in Its administrative headquarters are at Walton Hall in Milton Keynes in Buckinghamshire, and it has 13 regional centres located throughout the United Kingdom (UK). The University was established specifically and distinctively to provide supported open learning opportunities for students to study with the University from a distance on a part-time basis while continuing with other commitments. The University has an extensive geographical reach, operating through out the European Union, Switzerland and Slovenia. It also offers some on-line learning globally. 5 The University has full degree-awarding powers. Undergraduate and most postgraduate taught study is fully modular, and students register for creditrated courses (modules), and accumulate credit via completion of courses towards qualifications. About 600 or so courses, many of which are interdisciplinary, are available across a wide range of subjects. There is an open admission policy for undergraduate taught programmes, and students, the majority of whom are mature and in employment, enter the University with a wide range of study goals and previous educational and general experience. There is an active research community, and the University also offers full-time masters degrees in research methods, and full and part-time external research degrees. 6 The University states that it is the UK's largest university, with some 20 per cent of the UK's parttime higher education (HE) students. In , over 178,000 undergraduate students registered for courses with the University, of whom nearly 24,000 where studying at postgraduate level. However, because students can, and do, study for more than one course in any one year, course registrations totalled around 240,000. Some 5,500 of the total registered students live in Continental and Western Europe, and there are a further 12,000 students studying with other institutions on approved programmes leading to the University's awards. There are over 250 research students studying full-time at Walton Hall, about 475 directly registered research students studying part-time, and nearly 500 research students studying at sponsoring establishments. 7 The University has nine faculties, schools or institutes (in this report the term 'faculty' will be taken to refer generically to these academic units). They are: Arts; Education and Language Studies; Educational Technology; Health and Social Welfare; Management; Mathematics and Computing; Science; Social Sciences; and Technology. Most faculty members are based at Walton Hall. Full-time regional managers and staff tutors based in the regional centres are also members of a relevant faculty, and participate fully in their academic activities. 8 Regional centres provide the services and functions to support course delivery and students' learning. The centres deal with enquiries from applicants and students; arrange tutorial provision and support; manage local study centres; and appoint, develop and support the 7,000 or so part time associate lecturers (ALs) who act as course tutors. Regional centres act as an interface between the University and local and regional institutions and government agencies, helping to develop partnerships page 4

10 Institutional Audit Report: main report and collaborations. The OU in Scotland is Scotland's largest single provider of part-time HE. Although the University receives funding from the HEFCE, the Scottish Higher Education Funding Council (SHEFC) funds the teaching of OU students living in Scotland, and the activity in Scotland is subject to SHEFC policy directives. The North region also has particular responsibility for supporting students based in continental western Europe, and has a network of country coordinators in major European cities. 9 The University has established itself as the dominant provider of open and supported learning in the UK, and it is distinctive in that this form of learning is the basis on which the OU was founded. In its SED, the University defined 'supported open learning' as an integrated 'system of distance learning providing carefully designed, high quality, multimedia teaching materials and services, together with personalised learning and support'. The core component of the study materials is normally a series of specially written course texts augmented by a range of other learning resources and support, including on-line, and via the regional centres. The traditional model of study leads to the University's BA or BSc Open degrees, allowing students to design their own programmes of study to suit their own purposes. In 2000, the University introduced a range of single and joint subjects named degrees whose content is more closely prescribed. Mission statement 10 At the time of the audit visit, the University had just revised its Mission, with the aim of achieving a clearer sense of direction and purpose for those within and outside the University. The revised Mission Statement was not available to the audit team until the end of its visit. The revised Mission affirms the philosophy and values enunciated at the time of the University's inauguration in 1969, in that 'The Open University is open to people, places, methods and ideas'. It 'promotes educational opportunity and social justice by providing highquality university education to all who wish to realise their ambitions and fulfil their potential'. The Mission goes on to state that 'through academic research, pedagogic innovation and collaborative partnership, (the University) seeks to be a world leader in the design, content and delivery of supported open and distance learning'. Collaborative provision 11 OU Validation Services, established in 1993, oversees the University's collaborative provision. This provision does not form part of this institutional audit because the Agency has agreed with the University that there will be a separate audit of the University's collaborative provision in Background information 12 The published information available for this audit included: statistical data provided via the Higher Education Statistics Agency, HEFCE, UCAS, Higher Education Research Opportunities and the OU web sites; the information on the University's web site, including its degree prospectuses and course catalogues; the report of the continuation audit undertaken by the Agency, published in January 1999; reports of reviews by the Agency of provision at subject level, published since the 1998 continuation audit; a wide range of information relating to the management of standards and quality; the reports, confidential to the University and HEFCE, of the developmental engagements undertaken by the Agency in March The University provided the Agency with: an institutional self-evaluation document (SED); prospectuses and course catalogues; a discipline self-evaluation document (DSED) for each of the four areas selected for DATs, and which included the programme specifications for the courses and supporting documentation; the University's Corporate Plan, Plans for Change: The University's Strategic Plan for 2002 to 2012; statistical information, which included a demographic analysis of the student population over three years, student progress and achievement at course and award level and student feedback against key institutional indicators. 14 During the briefing and audit visits, the audit team was given ready access to a range of the University's internal documents in hard copy and on a CD-Rom accompanying the SED. It was also able to access the University's intranet and the documents, committee minutes, systems and material held there. During the audit visit the University provided the team with a range of documentation relating to the selected DATs, including examples of students' assessed work, teaching materials and reports of professional and statutory bodies (PSBs) where relevant. The team also had access to, and was given copies of, internal page 5

11 Open University documents and committee minutes at regional centres. The information included additional published material about the OU in Scotland, and a copy of 'The OU in Scotland's' strategic and development plans 2002 to The audit process 15 Following a preliminary meeting at the University in May 2003 between an Agency officer and representatives of the University and students, the Agency confirmed that four DATs would be conducted during the audit visit. On the basis of the SED and other published information, the audit team confirmed that the DATs would focus on programmes leading to the awards of: BA (Hons) Language Studies (with a focus on Spanish and German elements); BSc (Hons) Information Technology (IT) and Computing; BEng and MEng; MBA. 16 The University provided the Agency with DSEDs in January By agreement with the University, members of the audit team visited four of the University's regional centres to meet staff and students. Each regional visit focused on one of the DATs as follows: North Region - BA (Hons) Language Studies; London Region - BSc (Hons) IT and Computing; Scottish Region - BEng and Meng; South-West Region - MBA. 17 At the preliminary meeting for the audit, the students of the University were invited, through the Open University Students' Association (OUSA), to submit a written submission to the audit expressing views on the student experience at the University, and identifying any matters of concern or commendation with respect to the quality of courses and programmes and the academic standards of awards. They were also invited to give their views on the level of representation afforded to them, and the extent to which their views were noted and acted upon. The OUSA generated a document which was submitted to the Agency, and to the University, in November It was drafted with input from a specially established steering group, and drew on consultation with the student body via the University's on-line conferencing system, a training event for student activists, and articles in the OUSA bulletin and newspaper. The students' written submission (SWS) did not contain any matters which required the audit team to treat them with confidentiality greater than that normally applying to the audit process. The team is grateful to the students of the University for preparing the substantial and helpful written submission. The resolutions of the 2003 OUSA Conference were also made available to the team. 18 The audit team visited the University on 16 and 17 February 2004 for the purpose of exploring with the Vice-Chancellor, senior members of staff of the University and student representatives matters of institutional-level management of quality and standards raised by the University's SED, the SWS, and published documentation. At the close of this briefing visit, a programme of meetings for the audit visit was agreed with the University. The team did not select any area for a thematic enquiry. 19 The audit visit took place from 22 to 26 March Meetings were held during the visit with groups of staff and students based at Walton Hall. In view of the distinctive nature of the University, and the significance of its regional centres, members of the audit team visited four regional centres (see above, paragraph 16) in the period between the briefing and audit visits. During these regional visits, team members met regional directors and other members of full-time University staff based in the centres, some ALs, including those with particular interest in the DATs, and representative students studying courses in the DAT subject areas. The team is grateful to all those staff and students who participated in meetings at the regional centres and at Walton Hall. 20 The audit team comprised Professor D G Bonner, Mr C E J Griffiths, Mrs C Penna; Professor N Pratt; Professor D Timms, Dr M Wing, auditors, and Ms G Simpson, audit secretary. The audit was coordinated for the Agency by Dr D J Buckingham, Assistant Director, Reviews Group. Developments since the previous academic audit 21 The Agency conducted an academic quality continuation audit in May 1998, the report of which was published in January The report commended the University for a number of aspects of its work including the progress made in integrating regional activities with the centre through the creation of the student services structure; the extent to which its staff have a shared commitment to the University's quality ethos; its approach to course design; its approach to considering the implications of the development of named degree programmes; page 6

12 Institutional Audit Report: main report the strengthening of the links between strategic and academic planning through the unit planning process; the work of the Institute for Educational Technology; the use made of feedback from students and staff; its sensitive approach to the development and use of IT-based learning materials; and the quality of its promotional and course materials. 22 Points for further consideration included the necessity of reviewing the approval processes for new courses to ensure regulations are approved before courses/programmes start. The University was also advised to consider continuing to review the effectiveness of deliberative and management structures; increasing external involvement at programme level; ensuring, where practicable, that published materials remain current; keeping the operation of the Career Development and Staff Appraisal Scheme under review; developing a formal complaints procedure; and reviewing the portfolios of the Pro-Vice-Chancellors (PVCs), the Academic Board Committee structures to accord more closely with these. It was suggested that the University may wish to consider the desirability of continuing to review the nature and scope of the internal review process; providing further clarification of the process for approving research projects and supervisors are allocated; introducing formal research methods training for research students; continuing to review the impact of the student support arrangements in the regions; clarifying staff management roles in faculties; evaluating the effectiveness of the Guide to Quality Assurance; and providing induction courses for new committee members. 23 The SED detailed the positive action taken by the University in response to the 1999 Higher Education Quality Council (HEQC) report to implement the requirements and recommendations. New and changed processes and systems were introduced where relevant. These included a new Academic Board substructure, later further defined when PVCs' portfolios changed; a strengthening of the internal review process and new processes for the review of non-course based services at regional level; an evaluation of the Guide to Quality Assurance; and the establishment of a Research School. 24 Since the continuation audit, academic developments have included a stronger focus on programmes, and the introduction of new areas of academic provision; a range of 'named' degrees at undergraduate level (the first awards being made in 2000); the establishment of the University's own qualifications and levels frameworks; and the development of programme specifications. A review of course models is currently under way. Learning support developments include the 'Knowledge Network' and extensive development of on-line teaching, learning and support services to fulfil the requirements for all ALs to be on-line in 2004, and students in The implementation of progress files has also been developed. The University has undertaken major reviews of some of its services, with some organisational changes resulting. 25 There have been some important initiatives to enhance activity and support student learning and respond to external changes. The curriculum and awards strategy, Size and Shape , sets out changes to the curriculum to encourage innovation to respond to growing competition. An institutionwide retention project has been set up alongside the development of the widening participation strategy and partnerships and collaborations at regional level via Partnerships for Progression initiatives. At the time of the audit, the University was in the process of revising its corporate plan to respond to the changing external environment and issues it had identified as impacting upon its activity and future direction, and as part of the change in leadership following the appointment of the current Vice- Chancellor two years ago. 26 The introduction of named awards has presented a new focus for quality management arrangements, and a change in their emphasis and operation. The audit team explored aspects of these developments in relation to their impact on the quality, standards and enhancement of the student experience. It gave particular attention to the integration of named awards, and to the appropriateness of the quality management and enhancement processes for outcomes-focused provision of this type in relation to those for the University's traditional course-based model. Section 2: The audit investigations: institutional processes The institution's view as expressed in the SED 27 The SED referred to the University's Guide to Quality and Standards in the Open University when describing its approach to managing the quality its programmes and the academic standards of its awards. That approach makes use of the University's framework for qualifications and its constituent parts, course approval processes, annual, interim monitoring and periodic review processes, assessment and awards frameworks, and arrangements for student support and guidance. Responsibility for the institutional management of quality and standards is page 7

13 Open University discharged through the University's governance, committee and management structures. In its discussions with staff and students during the briefing and audit visits, the audit team explored how the approaches outlined in the SED for assuring the quality of academic activity and academic standards of awards operated in practice. The institution's framework for managing quality and standards 28 The University's committee structure is strongly participative. The academic authority in the University is its Senate, a body of some 1,300 members. An Academic Board with some 60 members formulates and reviews the academic policies of the University on behalf of the Senate. The SED described the structure of committees, implemented since the last audit, through which the Academic Board formulates the academic policies of the University for approval by the Senate, acting on behalf of its parent body to determine academic priorities and coordinate the development of courses and programmes. The Academic Board is served by six sub-boards, each with its own committee substructure. 29 The SED explained that the primary committee for the oversight of quality assurance processes themselves is the Quality and Standards Board (QSB), which is also responsible for arrangements for setting and reviewing standards, annual review of courses, periodic review of programmes, and maintaining oversight of the University's engagements with external organisations such as the Agency and PSBs. The University explained that QSB is also responsible for the design of quality assurance procedures, and the other boards are accountable to it for their implementation. Other key groups in matters of quality management are: the Curriculum and Awards Board (CAB), which either directly or through its subcommittees is responsible for course and programme approval, annual programme review, and assessment; the Student Policy Board (SPB), which takes oversight of arrangements for student support, including relationships with the ALs who assess and provide face-to-face tuition at regional level; and the Learning and Teaching Board (LTB), which takes oversight of learning resources. The audit team was informed during its visit that this structure was currently under review. 30 The boards that report direct to the University's Academic Board broadly reflect the organisational structure of the University and the responsibilities of the PVCs. There are offices for Learning and Teaching, Curriculum and Awards, and Student Policy. The QSB and the LTB both fall within the remit of the PVC (Learning and Teaching), and the CAB is chaired by the PVC (Curriculum and Awards). The functional area of Student Services is currently being reviewed and restructured to make more effective use of available resources, but it is intended that a senior manager with status similar to that of a PVC will soon take over this area. 31 Each of the nine faculties (where 'faculty' is taken to refer also to 'school' and 'institute', see above, paragraph 7) has a board, structurally equivalent to the Academic Board at the University level, and headed by a dean or director. Some faculties have departments with departmental heads or heads of disciplinary areas. Course team chairs have academic responsibility for courses, and report to their various deans or directors. Until 2000, the University offered the degrees of BA and BSc without subject descriptors, but it has recently devoted considerable effort to 'linking' courses into named awards, and to developing courses in the contexts of named awards. For each such award a 'named award board' is designated. In some instances, groups of named awards are collected together for some academic management purposes into 'programmes', and in such cases a 'programme board' is established to fulfil the collective responsibilities relating to the set of named awards for which it is responsible. For ease of reference in this report, 'programme board' will be understood to refer either to a named degree board or to a programme board. Course and programme teams oversee courses and programmes respectively, and report to their faculty boards. 32 The SED listed a comprehensive collection of printed and internet-based materials detailing the formal policies and procedures of the University with regard to quality management. It also pointed to much of the deliberative material derived from committee and other papers that supports policy and procedure and provides guidance on the practical operation of policy and procedure. Part of this documentation is the Guide to Quality and Standards in the Open University, which takes the form of 'factsheets', covering key aspects of the University's arrangements for managing the quality of provision and the standards of awards, from Governance and Planning (Factsheet 1) through to Accountability to Stakeholders (Factsheet 8). The regional dimension 33 Regions produce their own strategic plans setting out the scope of their activities and their approach. Each of the 13 regions has a regional director, responsible for a group of staff tutors who page 8

14 Institutional Audit Report: main report have affiliations to faculties and to individual courses and programmes. Funding comes to the regions from various sources - some is formula driven, some is open to a bidding process, and some is specific to the region, as in the case of Scotland where government-funded schemes such as the Scottish quality enhancement development apply. There is flexibility and autonomy to vary provision, especially tutorial support, where particular circumstances warrant this. An example of such variation is in the North Region which receives additional funding to support students and staff in continental and western Europe. The Business School has established three 'hubs' and three 'associate hubs', each serving the MBA programmes in more than one region. 34 Regions have regional committees and a substructure of committees that support their work. Regional committees deal with a range of business including: commenting on University strategies feeding views to the University strategic planning committee; the performance of students with a view to identifying any trends which might have relevance to student support and act as pointers for action; operational aspects of the student support arrangements and regional and national policy developments. Regional committees also receive papers of relevant University committees and staff from regional centres are members of relevant University committees. Framework for assessment 35 The University's policy framework for assessment is established by CAB and its Assessment Policy Committee. Both are chaired by the PVC (Curriculum and Awards). Management of assessments submitted by students, and the recording of all matters to do with marks, is the responsibility of the Head of Assessment, Credits and Awards, who reports to the Director of Student Services. 36 The University's approach to assessment is strongly modular or course-based (the University uses the word 'course' to refer to a module of study, and this terminology is used in this report). The overall credit framework is similar to that in UK HE generally, and levels are calibrated to The framework for higher education qualifications in England, Wales and Northern Ireland (FHEQ), published by the Agency. The University is aware of the Bologna declaration and is working to reflect its requirements in the academic provision. Students highlighted in their SWS that there are some difficulties with recognition of the OU degree in parts of Europe, which remain to be resolved. 37 Every course, of whatever size, has at least one designated external examiner. The classification of degrees is based exclusively on students' results in their several courses, and an individual classification is generated by algorithm. Assessment tasks are determined by course teams, and for any one course these tasks may consist of a combination of continuous assessment and examinations. Continuous assessment may include computermarked assessments (CMAs), but tutor-marked assessments (TMAs) remain a very important element of the University's assessment practices. Marking of TMAs is mainly done by the ALs who are employed by the University to provide learning support for courses. The SED described a comprehensive system for training ALs in assessment, and for monitoring the standards to which they mark, mainly using the staff tutors based in the regions. The University's arrangements for monitoring depend on a sophisticated infrastructure for managing and recording assessment items. 38 Responsibility for determining marks rests with examination and assessment boards, within guidelines established with the Senate's authority. Ratification of results is undertaken by a Results Ratification and Awards Classification Panel (RRACP) reporting direct to the Senate, which may require an examination board to revisit its results-list when it appears statistically anomalous, or where there is any other cause for concern. RRACP also undertakes the longitudinal monitoring of results. The arrangements for assessment are explained in several University documents, including one of its factsheets. The arrangements described there in outline are expanded in documentation for individual courses, and in various other documents, both paper-based and electronic. 39 The SED suggested that the University has made a significant re-assessment of its offerings in its strategic document Plans for Change and the associated document on 'Size and shape '. An important element in the development of the University's curriculum and, therefore, in its management of quality and standards, is a general move towards increased provision of named awards, or programmes as opposed to courses. The SED described several new quality assurance practices that had accompanied this shift. For instance, in addition to the annual process for the review of each course, there is now an annual process for the review of programmes. 40 The University indicated that the distinction between the boards for Quality and Standards on the one hand, and for Curriculum and Awards, Learning and Teaching and Student Policy on the other, was that QSB oversaw the development and page 9

15 Open University implementation of the quality management activities of the other three committees. However, the audit team noted that this distinction did not always apply: Faculty reports on annual course monitoring come directly to QSB, for instance, although annual reports on programmes are the business of CAB. The University later emphasised that faculty reports also 'feed into programme board reports'. The team concluded that it may be that the distinction between the work of the committees was not completely explicit in practice. The consequences of the move towards programmes rather than courses might be considered in relation to the context of the quality assurance system and committee responsibilities as a whole. 41 Overall, it appeared to the audit team that, in its move toward adding programme-based provision to its established course-based provision, the University had tended to add new quality management processes to those that already existed. The result was what the SED itself suggested may be 'too much review', perhaps deriving from a laudable wish to be thorough, without considering how the introduction of a new process affects the totality of processes already in place. Nevertheless, the team considered the University's arrangements for assuring the standards of assessments to be thorough and robust, especially in the context of the very large number of individual assessments, and the dispersed locations of those who mark assessments. The team considered that, at all levels, the University takes great pains to assure the standards of student performance, right up to the level of the RRACP, and the University's mechanism for ensuring the reliability of its results through RRACP seemed to the audit team to be a feature of good practice. The institution's intentions for the enhancement of quality and standards 42 The SED stated that 'the University's intentions for the enhancement of quality and standards at institutional level are encapsulated in a number of institutional strategies, in particular: Curriculum and Awards (Size and Shape ); Learning and Teaching; Human Resources; Student Support; Research; Widening Participation; Improved Student Retention; Equal Opportunities; Race Equality; Disabled Students; and Research'. These reflect and feed into the University's overall strategic and development plan, Plans for Change, and translate high-level institutional objectives into development priorities. Academic and administrative units develop operational plans from these and other academic and pedagogical and enabling strategies and planning data. 43 The SED went on to state that 'institutional strategic aims dictate development activities and priorities'. The University's strategic plan is revised every two years in the context of a thorough re-appraisal of the University's recent performance (The Institutional Performance Review report) and an assessment of the impact of the external environment and internal capabilities. The University had identified a levelling of registration rates, a fall or status quo in completion rates, and a decline in progression rates as concerns. It identified three interrelated main challenges for the future: maintaining the quality and vitality of the teaching provision and of the support for students; understanding the future and ensuring the capacity to respond to change; and improving operations in key business areas in the context of the rise in online activity generally in the sector, widening participation, retention and cost effective operation. 44 The University had signalled the need to monitor the external environment, to refresh its mission, and to revise its corporate plan in response. It also targeted improving communication, realigning its Human Resources Strategy and revision of its elearning strategy. To this extent, the SED gave a clear picture of what the University saw as the way forward and its strengths and weaknesses. It also stressed that planning is an iterative process, and that planning underpins the quality assurance processes and influences enhancement. At its meeting with senior staff, the audit team learnt about the challenges perceived to be facing the University, the rationale for change and the benefits of the scenario planning process adopted for the first time in the University. This had identified 'a number of key driving forces' which shaped and influenced the revised strategic plan. The team gained the impression of a significant level of activity, with change as a clear agenda and an understanding of the multiplicity of tasks to be undertaken. 45 Senior staff who met the audit team spoke of quality strategy as aiming to ensure that the processes for evaluating the operations of all services and boards are carried out in an effective and reflective manner, and that they reflect national requirements. However, this was not explicitly stated in the University's documentation and, given that it stated in the SED that a more critical appraisal of activities' cost effectiveness is needed, the University may wish to consider the need for a clearer articulation of its quality strategy. The team also explored with staff the impact of the move to named degrees, and the need to assess the whole learning experience and outputs page 10

16 Institutional Audit Report: main report as a defined programme. With the shift in emphasis from course-based to programme-based provision, it would be reasonable to expect that some adjustments to the frameworks and strategies for quality management and enhancement, including those for quality of learning and teaching, would be required. Given that the University is entering a new phase of its strategic development, it may wish to reflect on the fitness for purpose and cost effectiveness of the frameworks for the enhancement of quality and standards. 46 While it was clear to the audit team that forward planning in the University is a reflective and well-informed process, it was less clear how the priorities for enhancement were determined in practice and how they underpinned University objectives. The University's approach to enhancing quality and standards was stated as being encapsulated in the various institutional strategies. While all of these are important, it was not clear to the team which were of key significance, nor was it clear, in the context of the major challenges identified, where the priorities lay. The team would therefore encourage the University to underpin its approach to quality enhancement with a more explicit sense of priority, and to state its strategy for quality enhancement as clearly as possible. Internal approval, monitoring and review processes Approval of courses and programmes 47 University policy for the approval of courses is determined principally by the CAB, LTB and SPB. All their decisions are subject to the approval of the Senate. The University stated that this arrangement ensures that the Senate retains its statutory authority over the development of the curriculum, and that the curriculum as a whole remains consistent with the University's agreed objectives. The University's fact sheet on its qualifications framework and its Course Management Guide explain arrangements for course approval, including the ways in which they match the requirements of the Agency's academic infrastructure. CAB approves courses in the context of the overall University Curriculum Plan while LTB and SPB determine policy relating to the delivery and presentation of courses. Approval of the detailed plans for individual courses is devolved to faculties after outline approval is granted. One of the key responsibilities of the course team, appointed to undertake the detailed planning, is the design of the course materials that are fundamental to the University's system of supported open learning. The University requires that for each course an external assessor is appointed who reports to the head of the academic unit concerned on the progress of course design, and also on the progress of the course after the first time that it is 'presented'. 48 The approval of programmes leading to named awards has a somewhat different set of requirements. There is a standard pro forma for new award proposals which also forms the basis of a programme specification (including learning outcomes) for awards that are approved. Each award must conform to the University's curriculum strategy and its general guidelines on credit. A new programme of studies leading to an award is sponsored by an academic unit through its faculty board, and is approved by the Academic Board acting on behalf of the Senate, although the audit team noted that some new proposals had required a full Senate discussion. Policy on named awards is considered at University level by the Awards Committee. The academic standard for each degree is overseen by a programme board responsible both to its faculty and to the Awards Committee. After a new programme is approved an 'external adviser' is appointed to the programme board. 49 In general, the audit team found that the procedures for course development and approval were tried and tested, and worked well. The course team, a project team set up expressly for the development and management of a course, is a unit of operation that is well understood within the University, and course team chairs are key individuals in matters of quality management. For understandable reasons to do with the University's history of modular provision and the scale of investment in the establishment of a new course, the University lays great stress on the development of the course materials that form the foundation of its system of supported open learning. The team noted, however, that it was not a rare occurrence for course teams to find that volume of the assessment tasks set for a new course had not been fully estimated in advance, making it necessary to take in-course remedial action, such as varying assessment requirements in number or in timing, in order to respond to students' difficulties. It appeared to the audit team that course teams may sometimes be more alive to the content of courses than to the intended learning outcomes and the challenge that their achievement presents to students. 50 In other matters concerning design and approval, it appeared to the audit team the University had not yet fully come to terms with the challenge of designing programmes as a structured set of courses leading to clear programme outcomes, as opposed to the individual courses page 11

17 Open University themselves. Despite an extensive University-wide project entitled Learning Outcomes and their Assessment, intended to develop tools for mapping intended learning outcomes and how they are assessed, evidence from some of the DATs suggested to the team that there was still much work to do in following through the consequences of a learningoutcomes design approach rather than a contentdriven approach to provision. In some of the programmes covered by the DATs, not all intended learning outcomes could in fact be mapped on to all potential pathways for the designed award (see below, paragraphs 137, 146). These programmes had been approved despite these limitations. 51 The audit team noted that the external adviser is appointed only after approval is granted for the establishment of an award. This means that there is no involvement external to the University equivalent to that performed by external assessors in the early development of courses. The external assessor might provide an effective check where a new course is proposed for linkage to an existing award, but not when the programme is approved for the first time. The University is advised to review its use of external advice in programme approval, to ensure that it can systematically include external subject expertise in the procedures for approving programmes leading to named degrees. Annual monitoring 52 QSB determines and reviews strategy for monitoring and review as a whole, and addresses matters which require attention at University level. In its Internal Review Sub-Group developed the basis of a University-wide pro forma for annual course reports. For a course lasting a year, completion of a standard course report is required annually. The report combines an overview of quantitative information for example, about student registrations, progress and completion rates, with qualitative information about the feedback received and the action taken by the course team, significant changes to the course or the performance of students. The forms are available to course teams electronically, with some sections precompleted with student data held on the University's record systems. Arrangements for consideration of annual course report forms vary slightly between faculties, but in general, the course team chair sends the completed report to the sub-dean with responsibility for quality in teaching and learning, who considers it on behalf of the faculty. The sub-dean prepares a summary report for the faculty confirming that annual reports have been completed, drawing out issues for attention at unit level and, in particular, identifying high and low-performing courses (or those which have changed from year to year in these respects) in relation to student feedback and student performance. In the case of courses that are 'core' to awards, the report is forwarded to the programme board(s) of which the course is a core component. The sub-dean also completes a report on the process and key findings for the QSB at University level. 53 Pre-dating the requirement for annual course review, the University had a process under which a course was reviewed at specified stages in its lifecycle of (normally) eight years: end of first year; mid-life (i.e. usually after six years); and review before extension. This process continues, and has been brought together with annual monitoring, so that in the years in which the 'lifecycle' reviews occur these reports subsume and replace the annual report. The end of first-year review is effectively an enhanced version of the standard annual review. Mid-life review is more comprehensive and involves formal consideration by a panel appointed by the faculty; it may include an external member. Review to extend the life of a course beyond ten years, that is, any further review beyond the mid-life review, is also done by a formally constituted panel, which includes an external member and also a member from the University but outside the faculty. The panel reports to the faculty board or equivalent, which is empowered by the Senate to take action on its recommendations. This practice is longestablished in the University and the audit team saw evidence that it worked effectively. 54 For programmes leading to named awards, programme boards review the performance of students against previous cohorts and a range of feedback on the constituent courses. In addition to the course reports on core courses, programme boards receive information on a range of standards matters, including information from relevant examination boards. Programme boards are required to complete annual reports for consideration at the level of the academic unit and by the University's Awards Committee. These reports take into account course reports and external examiners' reports for compulsory and core courses. The University stated that in their annual reports programme boards are required to comment on a range of matters including the currency of the courses in the award(s) and how the award(s) relates to external reference points such as the FHEQ, subject benchmark statements, relevance to the requirements of PSBs, and so on. Programme boards are also encouraged to identify any features of good practice. 55 The audit team noted that in annual monitoring of both courses and programmes, as much as in other page 12

18 Institutional Audit Report: main report areas, the statistical data made available to those compiling reports was comprehensive. However, commentary on the statistical evidence and on qualitative indicators is variable between course teams' reports, so that some reports are less effective than others in consistently providing information on the health of the course in question. The 'perfunctory' quality of some course reports had also been noted in the report of at least one external subject review. The pattern of variability was also evident in the reports to the QSB, which took faculty overview reports for the first time in May 2003, and had not at the time of the audit visit taken them for the subsequent year. Some overview reports explicitly state that the author had not yet seen all course reports. The minutes of the relevant meeting of the QSB do not indicate whether action was intended to supply the deficiency. The contents of some reports appeared to give insufficient information to enable QSB to take an overview of the performance of faculty courses overall. It was therefore not clear to the team that this final link in the reporting chain was capable reliably of enabling QSB to assure itself that quality assurance policy had been fulfilled at other levels, or to provide it with information to enable it to make University-level judgements on the health of courses. Neither could the team follow the logic for making so complete a separation between QSB and CAB in the process of taking a University-level overview of programmes and courses. The University is advised to consider the effectiveness of its present system for gaining University-level overview of annual review activity at course and programme levels whether it has the most effective mechanism for making an annual assessment at University level of its academic offerings. 56 A separate internal review concerned primarily with performance in the regions is carried out under the authority of SPB. The shape of the resultant reports is much more determined by a template than reports on courses and programmes, and is strongly oriented towards perceived strengths, weaknesses and action. The audit team considered the reports to be an effective means for the University to keep itself informed on key regional activity, and also an effective means for regions to share good practice. The regional dimension 57 Regional staff are involved in institutional academic matters through their full and active membership of course teams. ALs, appointed and supported by the regional staff, and regional staff themselves, deliver courses according to the guidelines established by course teams. The annual process through which course teams monitor academic standards of attainment is also used by regional centres to assess the impact that their support services may have on outcomes. Periodic review 58 The University has recently introduced an additional procedure for periodic review, focusing on programmes leading to named awards. The intention is that the process will be carried out over a six-year cycle, with all taught provision in two or three Joint Academic Coding System subject areas being reviewed each year. At the time of the audit visit one review had been completed, in October 2003, focusing on law. The law review was conducted according to a carefully designed process that had been discussed in various University-level forums. The procedure is clearly documented, and its aims, objectives and outcomes clearly stated. The method is audit-based, and focuses on the ability of the area under review to manage the quality and standards of its awards. The evidence base is a self-evaluation by the area concerned, programme specifications and handbooks, the various documentation generated by routine quality assurance processes including external examiners' reports, annual course review, etc, and meetings with staff and students. The review team includes, and in the case of the law review was chaired by, a senior member, of academic staff of the University outside the faculty engaged in the review, and also at least one senior academic member external to the University, familiar with the subject area under review. In the case of the law review a single external member met this requirement. Those conducting the review and those reviewed had been provided with comprehensive briefing documentation on the process itself, including guidelines for the self-evaluation, for the report, and even for the kinds of questions that review-team members might ask. The guidelines for the SED included a requirement that the area's approach to key aspects of the academic infrastructure should be described. The review report was a comprehensive document, which the audit team considered to be valuable both to the law team and to the University as an instrument for assuring itself of the continuing health of the law programme. 59 The review had been followed up by a detailed consideration not only of the report itself, but of the process, including written comments by both the Chair of the review panel and the external member. The audit team considered that the University, and particularly QSB, had shown considerable care in implementing the process, documenting its requirements, and considering it thoroughly afterwards. The team noted, however, that the final, page 13

19 Open University University-level, recipient of reports from periodic review was, and would normally be, QSB. It is clear that, given its remit and its own description of its function, QSB is the appropriate forum for development and discussion of the process, and the team formed the view that it discharged that function well. However, QSB is not the University-level destination for most routine quality assurance information about programmes, and it seemed to the team that in this matter, as in some others, the University had not achieved the most productive distribution of responsibilities between its key committees. External participation in internal review processes 60 For each course, the University appoints an external assessor who is a senior academic experienced in the relevant field. The external assessor is proposed by the course team after the course outline has been accepted by the University, and is approved by the relevant faculty board on behalf of the Senate. The initial role of the external assessor is to assist with design, and to provide informal advice to the course team throughout the development of the course on overall structure as well as drafts of course units or other aspects of the teaching material. The assessor also provides the faculty with a full and detailed interim report, six to nine months before the first time the course is offered, covering all aspects of the course, including all the material available (in all media). The faculty uses this report as key evidence in deciding whether to proceed with the course as planned. Finally, the assessor supplies a final report at, or near the end of, the design process to confirm satisfaction with all the course/pack materials. The University's arrangements for the involvement of external experts in course design are well-established, and seemed to the audit team to provide views from outside at all points where they might be most effective, particularly at the design stage, where their impact and value for students is likely to be greatest. 61 When a course is reviewed at mid-cycle, or at later points, to consider whether or not it should continue to be offered, the University again makes use of senior external academic expertise, and includes an external member on the panel appointed by a faculty to make an assessment of a course and its currency. This practice, too, is well established and appeared to the audit team to be working for the effective management of quality and standards. 62 At programme/named award level the role of external adviser has been developed to provide appropriate assurances of standards in the context of an external examiner system that operates at course level only. The external adviser does not scrutinise student work or consider individual student marks, but may attend meetings of the programme board (note that 'programme board' is taken here also to include 'named award board', see above, paragraph 31), and advise on the requirements for the award and for the classification of the degree. 63 The only periodic review in the University's new style of review that had been conducted at the time of the audit visit had included as a panel member a senior academic external to the University. The audit team formed the view that the University had used this external member to good effect, not only to undertake the review, but also to comment on the review process. The team also noted that the procedure on paper provided for 'at least one (normally two) senior member of academic or academic-related staff external to the University'. No doubt the focus of the pilot review, narrower than most of those planned for the future, meant that one rather than two external reviewers was sufficient in this instance. The team noted that the Chair of the review, commenting to QSB, suggested that more than one external member would be appropriate to the general run of reviews. The team considered this to be good advice. 64 It appeared to the audit team, therefore, that the University had made strong and scrupulous use of its external member in the single periodic review conducted so far, and that its plans for the future indicated that this would continue to be the case. External examiners and their reports 65 The University has a strong focus on quality assurance and standards at course level, and appoints external examiners for each course. External examiners are appointed by the PVC (Curriculum and Awards) on the recommendation of the University's Assessment Policy Committee, and the criteria for nomination and appointment are monitored by the Assessment Policy Office which issues guidelines on the appointment process and role. External examiners have a written role description and are briefed by the course examination and assessment board. For each course during the development stage an external assessor is appointed who provides advice to the course team and produces an interim and a final report to confirm satisfaction with the quality of the course materials. 66 At the end of each awarding period, external examiners submit a report to the Vice-Chancellor with comments on the appropriateness of the course materials, the course assessment strategy, the administration of the examination process, the page 14

20 Institutional Audit Report: main report quality of the candidates and whether previous issues raised have been addressed by the University. All external examiners' reports are responded to individually by the University's Head of Assessment, Credit and Awards, and the reports are copied to the relevant academic units. Two standard pro formas are used to follow them up. The first requires the course team to comment on the external examiner's report, to state the actions they have taken in relation to each issue and to confirm that the external examiner is satisfied with the outcome of any issue raised in previous reports. The second is completed at faculty level, and includes a number of sections requiring confirmation that the process of considering reports has been fully completed, a summary of issues, actions taken at academic unit level, and the identification of any general issues for the attention of the Assessment Policy Committee. 67 The 1999 report of the HEQC continuation audit advised the University to consider increasing external involvement at the programme level and whether there was a need to separate more clearly the roles of the external assessor and external examiner. Both of these recommendations have been considered, and since 2000, award boards have been established for each named award, and the new role of the external adviser has been developed. The policy to allow an external assessor to become an external examiner was reconsidered, but it is still regarded as an appropriate option. 68 An external adviser is appointed for each named award. The external adviser sees the external examiners' reports for the relevant courses, and has a role which includes advising the University on the requirements for the award and the classification of the degree, and jointly with the degree board evaluating the achievement of courses in contributing to the overall objectives of the named award. External advisers submit annually to the Vice-Chancellor a pro forma-based report on the maintenance of standards and the continued validity of the named award. External advisers are not involved in considering the eligibility of individual students for the award of the degree because external examiners have already determined the student's eligibility for the award of each course they have taken. Reports from external advisers are considered by the award boards at a meeting at which the adviser is normally present. These reports then form part of the report from the award board to the University's Awards Committee. 69 In November 2002, the Awards Committee noted that the reports were mostly encouraging, but the SED identified two examples where the external adviser disagreed with the present policy in relation to the BA Humanities. Any disagreement between external advisers and an award board is logged with the Awards Committee. From the MBA DAT the audit team noted that, while in other named programmes the external adviser process was already in operation, the MBA award board had yet to be set up and the two proposed external advisers to be appointed, although documentation confirmed that this would take place shortly. 70 The audit team was able to study a sample of reports of external examiners for the courses and reports from external advisers on some of the named programmes of the DATs. None of the reports seen by the team raised concerns about the quality of the provision or the academic standards of the named awards. The team saw evidence that the process for dealing with issues raised by external examiners and advisers at course and programme level is effective, and that monitoring takes place at faculty and University level. The team formed the view that the addition of the external adviser to report on named programmes has strengthened the robustness of the external examining process. External reference points 71 The University's approach to the Code of practice for the assurance of academic quality and standards in higher education (Code of practice), published by the Agency, as outlined in the SED, was to amend its own procedures and guidelines, as deemed appropriate, as each section of the Code was published. The process used to achieve this was for the Quality Assurance Office, on behalf of QSB, to request the relevant committee or business area for a commentary on the extent to which its current policy and practice matched the precepts, and to a lesser extent the guidance, in the relevant section of the Code. A statement on each section of the Code was prepared for QSB to satisfy it that there was broad alignment with the precepts or there were in place appropriate alternative arrangements for the management of quality and standards. In the case of the section of the Code for collaborative provision, QSB was not satisfied and initiated a review. Current position statements in relation to sections of the Code are periodically revisited and kept up to date by the Quality Assurance Office, and are available on the University's intranet. The requirements of the sections of the Code for providing information to students, for example, on careers advice, research degree supervision and student support and guidance, are met by providing statements of service on the University's web site. While the University does not expect academic staff to use the page 15

21 Open University Code directly, there was good evidence that the precepts and guidance in the Code were embedded in the University's policies and processes. 72 Similarly, the University's own qualifications framework has been developed to be consistent with the FHEQ. An example of a change made as a result of the University's consideration of the FHEQ is the reduction of credit rating of the bachelor's degree without honours which has been reduced to 300 from 360. The University has developed a levels framework for its provision with generic level indicators that have been informed by the FHEQ. 73 Programme specifications are now produced for new award proposals using a standard template which requires a commentary on how subject benchmark statements, the FHEQ and professional requirements have been taken into account. The SED drew attention to two cases where consideration of subject benchmark statements had led to changes in programmes. When awards are approved, externality is taken account of via the Business Appraisal Process for Curriculum Developments, which is a University-wide process, designed to help faculties make decisions about supporting new proposals. The process ensures alignment with the standards of relevant PSBs, as well as ensuring that the new proposal is coherent with the University's portfolio of awards. The CAB includes four external members, who are able to assess the extent to which external reference points have been used to inform the award proposal. 74 The programme specifications which formed part of the evidence base for the DATs showed that the requirements of the relevant benchmark statements had been considered. Programme specifications are not generally communicated to students, but student versions containing programme intended learning outcomes are being prepared, and will be included in programme information. External advisers are required to report on the validity of the named degree in relation to the appropriate benchmark statements, and the advisers' reports that were available to the audit team confirmed that due attention is being paid to them. The outcomes of accreditation reports from PSBs are reported to the Academic Board, and a log of issues is kept by the Quality Assurance Office, which tracks action taken on them. Overall, the team was able to conclude that the academic infrastructure published by the Agency is being properly addressed by the University. Programme-level review and accreditation by external agencies 75 In the period since the publication of the report of the HEQC continuation audit, the University has participated in 11 reviews by the Agency of provision at subject level. A high aggregate grading was a feature of the reports of these reviews, with the student support and guidance aspect receiving the maximum grade in every case and the learning resources aspect receiving maximum grade in all but one case. While there were no themes common to the recommendations of every subject review, the audit team noted that at least one had commented on work-overload, and several recommended that attention be paid to the consequences of assembling courses into programmes leading to named awards. Developmental engagements were conducted in 2003 in English and history, and both received positive reports. One made no recommendations for improvement. The other, while also being positive, suggested that there are matters where, again, the University might consider further some consequences of its strategy for named awards at the level of the particular programme. These findings are consistent with the views of the findings of the present institutional audit. 76 The University is subject to a number of other external reviews related to programmes with professional content. The SED drew attention to the recent successful recognition of new or changing academic provision in the areas of nursing and midwifery, and social work. Three of the areas covered by DATs involved professional recognition for all or parts of the programme areas; all had successfully achieved recognition. 77 The audit team formed the view that the University takes a thorough approach to responding to external review by external agencies. In addition to action taken at the level of the subject, all reports are forwarded to QSB, as are the responses from subject teams and their reports on action after one year. The team saw evidence of the care taken by QSB in the consideration of these reports. In relation to subject review and developmental engagements, consideration by QSB has included overview reports as well as reports on individual reviews, resulting in action at institutional level to make improvements in relation to specific 'themes' such as retention. To ensure that action is followed up, the Quality Assurance Office maintains an 'issues log' on behalf of QSB. For actions consequent on PSB reports, the Office keeps an 'accreditation log'. The team considered that the University is appropriately responsive to the outcomes of review by external agencies. page 16

22 Institutional Audit Report: main report Student representation at operational and institutional level 78 The OUSA is recognised by the University as the representative body for students. OUSA appoints student representatives to a wide range of central University bodies including the Council, the Senate, the Academic Board and a number of boards which report to Academic Board such as QSB, SPB and CAB. A central Consultative Committee includes students and ALs from each region along with full-time staff. Students are represented on regional committees, which meet three or four times a year and are responsible for considering the University's policies and practices, and for making local recommendations which are consistent with the University policy. Regional committees also include regional staff and representatives of ALs. They are empowered to discuss any matters of interest to the groups represented, and to make recommendations to the Central Consultative Committee on any matter which it is felt merits wider discussion. 79 The University expressed the view that these representative structures are effective; that students are able to shape and contribute to the development of University policy in appropriate cases and are able to take a full part in debates concerning the University's situation and its priorities for the future. In addition to the structured opportunities available to students to express their opinions concerning the operation of the University, Sesame, the OU newspaper, provides a debating platform for matters of concern for all students. 80 OUSA student representatives who met the audit team at Walton Hall, and other students who met the team during the regional visits, confirmed the University's view of the effectiveness of these arrangements. The processes for representation seem to be understood and to work, but, while students are able to make their views known, it is not always clear to them how issues are acted upon. The team formed the view that students appeared to be fully engaged with the University, were clear that their views were both actively sought and listened to, and were generally happy with the arrangements in place. The team recognised that the University's students are characterised by a level of maturity and an ability to clearly articulate their views. These characteristics are reflected in, and are a particular feature of, the dialogue between students and the University. 81 The capture of the student voice by the University appeared to the audit team to be particularly effective at institutional level. Students have many opportunities to contribute to institutional deliberations, and are justly proud of the part they are able to play in the development of policy and in effecting institutional change, citing the move towards named degrees as a significant example of their contributions. At regional level, too, students have the capacity, through the regional committees, to keep themselves informed of and to take part in institutional debates and, within the allowed parameters, to effect operational change. Regional staff confirmed the value of the student contribution through these committees. Students do not, however, appear to have representation on named degree programme boards, although they are represented on all boards of academic units and some academic unit committees. The University might wish to reflect on the merits of student representation on programme boards as named awards become a more substantial part of its overall provision. 82 The University clearly values the views and opinions of its student body in its management of the quality of its provision. It has put in place systems of student representation that facilitate the expression of those views and opinions, and which allow students to contribute effectively to the deliberations and operation of the University and its provision. Feedback from students, graduates and employers 83 The University has a sophisticated system in place for eliciting student feedback by the use of questionnaires. This activity is overseen centrally by the Internal Review Committee at institutional level, and is coordinated by the Student Research Centre, which carries out most surveys, processes data, summaries feedback, and provides guidance on interpretation of data. 84 The end-of-course survey, undertaken annually on a sample of courses, is the major means whereby student feedback is elicited. The Student Research Centre produces an analysis of findings for each academic unit, and Centre staff members meet members of academic units to discuss and interpret the findings for their courses. The annual course review process now requires course teams to comment on how they have addressed this feedback and the means by which they will convey their response back to students. Degree-level surveys are also undertaken, covering students' general experience of studying OU courses, the effects of study on a number of other aspects affecting the individual, including personal development and career development. The University is to review its student feedback processes at award level in the light of sector-wide developments in this area, in particular the introduction of a national student satisfaction page 17

23 Open University survey. In addition to course and award surveys, specific student groups are surveyed, for example, those students with reserved places on courses, new students, or students who have withdrawn. 85 Students who met the audit team affirmed that the University was assiduous in eliciting and responding to student feedback. They noted that questionnaires, as a route for student feedback, were complemented effectively by other channels such as student representation on University committees, meetings with OUSA, and student electronic conferences. However, it was clear that students were not always made aware of the particular actions taken as a result of their feedback, and the team would support the requirement recently introduced into annual course review that course teams make explicit the means by which they will inform students of any responses to student feedback. The team was able to confirm that the University otherwise has a very comprehensive and effective system in place for eliciting and responding to student feedback. 86 Academic units are encouraged to seek employers' views on course and award curriculum during the approval process. In the case of courses with a strong vocational or continuing professional aspect, employer input is often mediated through the involvement of experienced practitioners acting as external assessors and/or the involvement of professional bodies. The University has also initiated a number of projects to elicit employer views, including a study on the employment issues which face graduates in the arts, and a project to establish the needs of the National Health Service market. The University has recently developed an employability strategy, and the audit team was informed that an action plan associated with this strategy is to be implemented this academic year. The University is involved in employability research at national level through the HEFCE Enhancing Student Employability Co-ordination Team in which the University's Centre for Outcomes-Based Education is taking a lead. The team formed the view that the University recognises the importance of engaging employers with the University's academic programmes, and takes steps actively and effectively to obtain employer input into the curriculum development. Progression and completion statistics 87 Progression and completion statistics are provided by the Student Research Centre. At course level, results, ratified by the RRACP (see above, paragraph 38), are provided to course teams in the form of statistical reports. The Centre meets appropriate staff to facilitate interpretation of the data, and provides additional support and guidance where this is required. Course progression and completion data includes longitudinal data suitable for trend analysis. Data is also provided on a demographic basis. 88 Data is presented within course reports and considered as part of the annual reporting and review process, with course reports forming the basis of summary reports to teaching and courses committees, or equivalent, and to the relevant award and/or programme boards. The main objectives of this review activity are to determine how well a course is meeting its intended outcomes, to highlight opportunities for improvement, and to identify and address issues which have arisen in particular courses, or across courses or levels. Course information is also used to inform the deliberations of the University's student retention project, and provide an input into the University's planning processes. 89 The SED pointed to difficulties in presenting and interpreting student data at degree level, attributing these difficulties to a number of factors, including the fact that students are not obliged to register for a named award (although increasing numbers of students are opting to do so), and that the rate of completion can vary considerably as students have considerable flexibility in determining the pace at which they study for a degree. Award completion data is available, including that suitable for trend analysis. This data is considered by the award boards, and in the annual course review report. 90 The audit team found that, at course level, extensive data, encompassing progression, completion and trends for the current and previous years, is used effectively in the monitoring the assurance of quality and achievement of academic standards. The Student Research Centre provides valuable support to this process. Evidence from the DAT studies demonstrated to the team that award completion data is also considered in an appropriate manner. Nevertheless, in view of the increasing numbers of students enrolling for programmes leading to named awards, the team would encourage the University to reflect on the appropriateness of its provision of statistical data to support named awards, where programme-oriented information on such matters as prerequisites and electives will be required over and above course-oriented information. Assurance of the quality of teaching staff, appointment, appraisal and reward 91 The University's academic staffing comprises a core group of full-time staff and a much larger group of part-time ALs. Full-time staff are recruited centrally, and about a quarter of them are employed as staff page 18

24 Institutional Audit Report: main report tutors in the regional centres. Formal responsibility for the appointment of ALs rests with regional directors, advised by members of academic units who are based in the regional centres. The University has well-established recruitment and selection procedures, and all staff receive a formal induction. Full-time academic staff are subject to a four-year probationary period, during which they receive mentor support. Policies are in place for promotion of academic staff and job regrading for academic-related staff. Since 1999, the University has operated a teaching awards scheme which aims to recognise, reward and celebrate excellence in teaching. This scheme is to be extended to ALs this year. 92 The University stated in its SED that all academic and support staff are recruited 'through a rigorous selection procedure based on a fair selection policy which is consistently applied to the recruitment of full and part-time staff'. The SED also recognised the role of regionally-based staff tutors as 'central to quality assurance, fulfilling a bridging function between central academic units, where courses are mostly developed and produced, the regional centres responsible for the implementation of tutorial strategy, and the Associate Lecturers appointed to facilitate student learning'. Staff tutors have two reporting lines: one to the dean of a faculty (or other academic unit) and the other to the regional director. The University acknowledges the 'intrinsic tensions' that the two reporting lines may cause. The audit team explored with senior staff these intrinsic tensions, and was assured that, in practice, the system is sufficiently flexible to avoid any difficulties. Staff tutors also confirmed to the team that generally they were able to work satisfactorily within the current model. The University has recently reviewed the staff tutor role, and the report of that review confirms that the current model should be retained. The recommendations from the report, which are currently being implemented, will address the potential for tension in the reporting lines for staff tutors. The review report also addressed the balance between central and regionally-based staff, and ways of ensuring that the balance remains appropriate to the needs of the University and its students. 93 A guide to effective recruitment and selection prepared by the University's Personnel Division was updated in January The document confirms careful procedures to ensure that the job description and person specification are matched, and that an appropriate person specification is agreed by all parties. The fairness of the appointment policy is indicated, for example, by the special consideration given to disabled candidates, the willingness to conduct interviews using communications technology, and the requirement that all staff involved in the selection process must take the University's formal training course. The rigour of the appointment process was confirmed by ALs, who reported to the audit team that they are subject to the same careful appointment process even if they had previously held an AL post. On the basis of the evidence available, the team was satisfied that the University's appointment policy for full and part-time academic staff is rigorous and fair. 94 A Career Development and Staff Appraisal System is in place for full-time staff, who are appraised at least every two years. Full-time staff who met the audit team expressed the view that the appraisal process is effective, and reported that it is common for staff to seek appraisal annually despite the biannual requirement. ALs are not included in the appraisal system, but the team formed the view that the strong supporting infrastructure for ALs, which includes close monitoring and mentoring during their two-year probationary period, adequately compensates for lack of routine appraisal. Assurance of the quality of teaching through staff support and development 95 The University has a formal staff development policy statement for full-time staff, and it is the role of the Organisational Development and Change Team within the University's Personnel Division to provide support for staff in acquiring the skills and expertise to undertake their duties effectively. The faculties (or other academic units) are responsible for developing course provision in the respective academic disciplines. An Academic and Professional Development Programme has been developed for full-time staff, aimed specifically at supporting staff who are engaged in course design and development for distance learning. Part of this programme is delivered by the University's Institute of Educational Technology (IET), which also undertakes educational research to support faculties' activity in the development and presentation of their courses. All staff, including ALs, may register for OU courses for personal and/or professional development, and their fees are waived. A notable characteristic of the OU is the large number of staff who are also students, and who can therefore provide particular insight for peer review and comment on the student experience. 96 Staff development for ALs is provided by regional staff development working groups which are coordinated by a central staff development team. Terms and conditions of service for ALs were updated in November 2002 to include a requirement for them to undertake the equivalent of two days of page 19

25 Open University pedagogical skills development each year, payment for this time commitment being included within the salary. The Associate Lecturer Development and Accreditation Pathway (ALDAP) has recently been introduced to provide a staff development framework that integrates the regional staff development programmes with personal development plans. Through the ALDAP, ALs are encouraged to become members of the Institute for Learning and Teaching in Higher Education. The range of staff development resources and activities available to ALs includes online material, accessible through the TutorHome web site, such as 'open learning toolkits' which offer good practice on relevant topics such as supporting students with special needs. The TutorHome web site also provides information for ALs on links to courses and faculty sites. A handbook for ALs includes sections ranging from general information about information and communications technologies (ICT) and the AL role, the bullying and harassment policy and data protection information, through to the daily average hours agreement. 97 The University has recently reached the view that its committee structure relating to staff development and training was 'overly complex', with overlap between a number of committees and groups concerned with staff development. Under revised arrangements from September 2003, the Director of Personnel reports annually to the Staff Strategy Committee on staff development policy and provision in the University. Cross-University development activity is coordinated within the Staff Development Providers Group, chaired by the Head of the Organisational Development and Change Team. In order to meet better the evolving training and development expectations within the sector as a whole, the University is in the process of reevaluating its policy and provision for academic staff development, and the Career Development and Staff Appraisal system was under review at the time of the audit visit. An area of staff development which the University regards in its SED as 'most challenging and successful' is the collaboration between staff during the preparation and review of course materials. The audit team noted with interest the way that the University endeavours, in this activity, to ensure that new and more experienced staff work together and engage in the pedagogy of distance-learning teaching and assessment. 98 Staff who met the audit team confirmed their satisfaction with the level of support and development opportunities available to them. ALs who met the team were particularly enthusiastic about the support provided for them, and confirmed that they generally engaged in development activities in excess of the two days provided for in their contract. They were complimentary about the on-line support, especially the 'toolkits', and confirmed the willingness of staff tutors to provide support and advice. The team noted that central oversight of staff development for ALs was active, and reviewed and maintained the University's staff development policy for ALs effectively. The University has recently established a Teaching and Learner Support Sub-Unit which includes in its role the provision of additional support for ALs. On the basis of the evidence available, the team was able to conclude that the University has effective procedures and practices for the support and development of its staff, and that the level of support available to ALs is particularly worthy of note. Assurance of the quality of teaching delivered through distributed and distance methods 99 The audit team considered that, in view of the particular nature of the OU, with all of its provision taking the form of supported open learning, that the whole of this institutional audit was, in effect, addressing the quality of teaching delivered through distributed and distance learning. The team decided, therefore, that it was not appropriate to cover separately this section of the report of an institutional audit. Learning support resources 100 The University delivers programmes of supported open learning that are built around high-quality teaching materials and effective support for learning, provided principally by locally-based ALs, but supplemented by a range of print and electronic resources. Courses are developed by course teams that include academic staff authors with expertise specific to the subject, together with specialist staff with expertise in design, educational technology and audiovisual production. Courses are designed around designated learning outcomes. Texts specially written to support the course may be augmented by other learning materials offered in, and delivered through, various media. In constructing and developing courses, course team members are able to access the experience and expertise of colleagues across the institution through the University's Knowledge Network. Course teams are also able to benefit from research undertaken by the IET and other special research groups in the strategic move towards the enhancement and extension of the use of elearning in courses, in furtherance of the University's Learning and Teaching Strategy 2004 to page 20

26 Institutional Audit Report: main report 101 The University recognises the challenges that it faces, in terms both of the competition from other HE institutions and of its own financial constraints, in continuing to develop and extend its course offerings. In consequence, it has recently instituted a Course Models Review with the aim of setting out a limited range of models of course materials, services, modes of assessment and student support, from which will be selected a model to be applied to the production and presentation of a particular course. The audit team learnt that this is intended to allow a better balance to be effected between defined pedagogic aims and the estimated costs of production and presentation. 102 Students are able to access a number of internet-based and electronic resources through the OU Library web site, and are offered useful guidance on how to find the resources they require for their studies through other local and national libraries in the UK. The Learner Guide offering advice about a range of issues related to study, including choosing a course, essay writing and time-management is available on the internet, and the Lyceum internetbased tutorial tool has the capacity to contribute to effective student-to-student, and student-to-tutor interactions. Some courses provide an 'edesktop' web site containing further learning materials and links to appropriate resources. 103 Students expressed to the audit team their admiration of the high quality, range and effectiveness of the printed and audiovisual materials provided to support their studies. They spoke enthusiastically to the team of their experience of learning with the OU, and confirmed that staff were responsive and attuned to their needs. There was some suggestion that a more staged delivery of materials with a greater degree of initial guidance as to how to work through them, might have enhanced still further what was clearly considered to provide an excellent learning experience. Certain electronic resources such as Lyceum were also considered to give excellent support to learning. Some disquiet was expressed, however, with the perceived move away from hard copy learning materials in some courses towards greater dependence on ICT for the delivery of learning materials, particularly in cases where the choice of media required upgrading of the technological resources available to students. The OU Library did not figure highly in students' comments to the team about learning support, and this low level of awareness of what is potentially a very useful means of offering support for learning was reflected in the University's end-of-course survey for 2002 in which only 27 per cent of respondents commented on information or advice received through the OU Library. 104 Considerable cost and effort are involved in the development and presentation of courses, and the institution of the Course Models Review appeared to the audit team to be a sound way of building on, and rationalising, the evident strengths of current provision. There is generally a good mix of media employed in delivery, a mix that in itself is appreciated by learners. Good internet-based provision allows students to seek learning support at times that suit themselves, although the University perhaps needs to consider ways of ensuring that more students are familiar with and use the resources provided through the University Library and its web site. Overall, the team was satisfied that the University provides an appropriate mix of high-quality learning materials for its particular style of supported open learning. The team noted that the University is actively engaged in considering how best to develop and enhance the provision that it offers in the context of the changing face of HE in the UK and the continuing development of ICT-mediated learning. Academic guidance, support and supervision 105 The University is committed to offering high levels of academic guidance and support through its Student Services staff, ALs and staff tutors and a range of print and on-line guidance. The on-line Learners Guide and the advisers based in regional centres provide students with advice and guidance on choice of courses and planning their studies. Although the University's open-entry policy allows students to start their undergraduate studies at any level, new students who opt for level 3 courses are routinely contacted to confirm that this is the most appropriate choice. Potential students whose first language is not English are advised at least to familiarise themselves with the International English Language Testing System test materials to satisfy themselves that their competence in English is sufficient for them to succeed in study with the University. Students with low previous educational achievements are offered 'Openings' courses, which are designed to introduce them not only to the broad subject area, but also to generic learning skills; the audit team was interested to note some evidence to suggest that those students who take advantage of this opportunity perform significantly better at level 1 than those who do not. 106 Students are supported in their learning by locally-based ALs, who are trained to help in the delivery of specific courses through a variety of means including face-to-face tutorials, day schools, telephone and on-line support. The audit team formed the view that ALs generally provide timely and helpful feedback on student assignments, and are encouraged to be page 21

27 Open University proactive in the identification of students who may be experiencing problems with their studies. 107 The University also facilitates a measure of peer support through the FirstClass electronic conferencing forum, which enables students to share their learning experiences with others in a non-judgemental and supportive environment. The University is currently making available to students an internet-based Personal Development Planning (PDP) framework which will build on existing on-line resource, and which aims to support students as they learn by providing structured opportunities for reflection on learning as well as for the analysis and recording of their achievements. 108 The University monitors the effectiveness of these arrangements through progression statistics, and through end-of-course surveys and the annual Student Services review. It is aware of the difficulties faced by its part-time students in managing their studies in the context of conflicting demands on their time. The University accepts the essential interdependence of its strategies for student support and for teaching and learning, and it aims in the long-term to bring these together. Currently, it is attempting to address separately a number of related issues such as workload, through its Teaching and Learning Strategy, and retention, through its project concerned with proactive student support, which was piloted in three regions in 2003 and is being rolled out across all regions in Students who met the audit team indicated high levels of satisfaction with the quality of guidance available and the degree of academic support provided, especially by the ALs. Face-to-face tutorials and day schools were rated highly by students as effective learning experiences. Maintaining regular attendance at tutorials, because of conflicting demands on students' time is clearly a problem for some, but the clarity of information in course packs and individual support from ALs was generally considered by students to compensate adequately for the occasional absence. The few reported problems concerning ALs and their relationship with individual students, mainly relating to the return of marked assignments, were stated to have been swiftly and satisfactorily dealt with by regional staff, to whom students appear to have good and easy access. 110 The audit team considered that the academic guidance and support provided by the University is of a high quality, and aligned well with the recommendations in the Code of practice. The high quality of the provision derived especially, in the view of the team, from the effective combination of the personal support provided by regional staff and ALs on the one hand, and the paper-based and on-line support, on the other. There appeared to be little chance that an individual student, or prospective student, would not have access to appropriate sources of information, guidance or support. 111 The audit team also formed the view that the University, at institutional level, was committed to improving students' learning experience, and that it was actively engaged in a number of ways in so doing. The commitment and student-centric approach of the members of full and part-time academic and support staff who met the team indicated clearly that institutional initiatives are fully supported and informed by the University community as a whole. The team formed the view that the combination of the quality of the support offered to students through human and material resources, and the capacity for students to put together a mix of support in a way that best suits their own needs is a feature of good practice. Postgraduate research students 112 The University's Research School was established in September 2000, inter alia, to enhance the research environment and create an integrated research community, and to improve the quality of provision for all its research students. The Research School, which describes itself a 'mix of palpable and virtual forms', has a physical presence in the form of the central staffing of the Research School. In other respects it is a virtual entity, drawing on the services of other areas; it is directed by the University's Research Board. 113 Support for research students is defined by the University's code of practice for supervisors and research students. It includes guidelines for the conduct of the relationship between research students and supervisors, defining responsibilities and indicating how progress is monitored. The Research School established the Research Academic and Professional Development Programme in 2002, which includes induction sessions for new staff and students, and supports faculties' discipline-specific development opportunities by sharing expertise and providing a training pack. Resources required by all research students are carefully considered as part of the admission process and assessed at registration and during the regular progress reviews. Course U500, Doing Academic Research, is a self-study pack provided for all research students and their supervisors, designed to help meet the expectation that all students, including external and part-time, have equal access to training. A system of third-party monitoring is in place to address any unresolved problems that impede students' study. page 22

28 Institutional Audit Report: main report 114 The audit team was interested to explore the support arrangements for research students. Both full and part-time students confirmed to the team their awareness and understanding of the University's code of practice for supervisors and research students and its application. They spoke warmly about their engagement with their subject, the supervision and the high level of academic support provided. Progress is regularly and systematically reviewed and full access to appropriate learning resources is available. Students who met the team were aware of formal and informal mechanisms for representation, and for expressing views and concerns. They highlighted the third-party monitoring process as a particularly useful and supportive mechanism, giving examples, but felt it needed to be seen as a truly impartial system. In this regard, the team was reassured that in the recent revision to the process the University had clarified its independence. The team considered that the revision would further strengthen this supportive process, and would encourage its adoption by all faculties. 115 The references in the SWS relating to concerns over the resources and facilities provided for research students were confirmed as referring to the relative lack of social amenities, especially after normal business hours. Discussions with staff confirmed to the audit team that the University has attempted to augment these amenities as far as practicable with available finances. Research students confirmed that the training provision was very comprehensive and appropriate, and that overseas students' needs were catered for. Professional development and teaching and demonstration opportunities are available, although students expressed to the team the view that there is a lack of clarity about the entitlement to financial support for conference attendance and how funds tied to Research Council grants are distributed. The team would encourage the University to ensure that research students have clear information about their financial entitlements to avoid misunderstandings amongst the student body and to enable students to maximise opportunities for professional development. Personal support and guidance 116 Policies for the provision of personal support are developed by the SPB, and are implemented through Student Services. Support is available for a range of targeted groups, including disabled students and students suffering financial hardship. The Disabled Student Services section within Student Services provides a wide range of services to disabled enquirers, applicants and students, and disability and additional requirement teams in regional centres provide advice and support to applicants and students who declare a disability. The University is proactive in supporting disabled learners, and in ensuring a reasonable equality of experience where a disability might otherwise prevent a student from benefiting from a programmed academic activity. The development of the University's Disabled Student Strategy took into account the guidance of the Code of practice. 117 Students suffering financial difficulties are not only directed to HEFCE-funded support, but are also, in the case of students following courses weighted at less than 60 credit points, considered for assistance provided by the University itself. The recently established Careers Advisory Service provides careers advice and guidance, but use of the service remains low according to recent end-ofcourse surveys. There is an awareness of the need to improve the link between information on courses and awards and the career implications, and it is hoped that a new section of the web site, OU Study and Your Career, will help students to match courses to their career aspirations. 118 The University's support for students has always been predicated on the diversity of the student body, but the University is aware that both the widening participation agenda and the growing number of students from younger age groups taking up study with the University will have implications for the nature of the support that needs to be offered in the future. Students who met the audit team were aware of the various aspects of personal support available, but few, if any, reported that they had availed themselves of these particular services. The University believes that its targeted support for disabled students and for students in financial hardship is effective. It recognises the need to enhance its provision for careers advice, even though the vast majority of its students are already in work. It believes, however, that one of the principal benefits of the PDP framework currently being adopted will be to make students much more aware of their achievements and their acquisition of generic skills, and that through being able to record and to articulate them more effectively their employability will be considerably enhanced. 119 The audit team formed the view that the University is responsive to the demands of its students in general and is able to deliver targeted support appropriately and in conformity with legal requirements. Evidence from student surveys suggests that students are already able to appreciate the range of generic skills that are fostered through their academic studies. The team endorsed the University's view that the PDP framework is likely to page 23

29 Open University be especially useful in enhancing the career prospects and employability of those of its students who take advantage of the opportunity that it offers for the more formalised articulation and recording of skills acquisition. The regional dimension 120 With their focus on the delivery of courses and support for the students, regional centres form a key element in the University's student support system. Each region is required to deliver the core student support services determined by the University's student support strategy to a standard set in the University's Statement of Service. The level of support is monitored and assessed through the University's annual monitoring process, supported by extensive statistical data and analyses of student feedback, produced centrally and covering both specific and general issues. Reports are made to SPB. 121 Regional staff carry out multiple roles. The focus is on ensuring the delivery of the student experience at the course level, and the role of the staff tutors is crucial in providing a link between the academic unit and its course team and the regional environment. Despite the logistic challenges, the evidence available to the audit team indicated that the system works effectively and provides essential support to students. There is active communication and interaction between the regional staff, staff in other regions and staff in central units of the University, ensuring that staff form an academic community with a sense of collegiality yet also with a clear regional identity. Section 3: The audit investigations: discipline audit trails Discipline audit trails BA (Hons) Language Studies (with a focus on Spanish and German language elements) 122 The DAT focused on German and Spanish languages provision within the named degrees BA (Hons) Language Studies (B21) and BA (Hons) Modern Language Studies (B30). The DSED was prepared specifically for the audit visit by the Department of Languages within the Faculty of Education and Language Studies. Although it focused on German and Spanish courses, all language groups within the Department were involved in a process of consultation that led to its creation. In the course of this DAT, members of the audit team met staff and students at the North Regional Centre in Newcastle, and also met Faculty staff based at Walton Hall. 123 It is intended that the B30 programme, allowing for the study of either English and a modern foreign language (French, German or Spanish), or a combination of two modern foreign languages (two from French, German or Spanish) will subsume B21 which only allows the study of one modern foreign language combined with English. The latter award will no longer be offered after a suitable period of notice. The University is aware of the implications that this change will have for current students, and has put in place mechanisms to inform students that they will be allowed to remain on the B21 programme, or, in those cases where they have already fulfilled appropriate requirements (specifically the study of 60 credits at level 1) or are willing to do so, to transfer to B In , the Department of Languages undertook a review of courses comprising the Diplomas in French and German in order to bring them into line with the University's qualifications framework and, in consequence, to allow them to count towards the named degree at an appropriate level. The review was undertaken by external assessors, and led to the incorporation in the specified courses of learning objectives that developed more extensively appropriate analytical and critical skills. The courses in Spanish had been developed following the adoption of the University's qualifications framework, to which they conform. 125 The Department of Languages actively engaged with the deliberations leading to the creation of the Subject benchmark statement for languages and related studies, and the audit team found that the programme specifications for both B21 and B30 are compatible with the provisions of the benchmark statement, as well as with those for linguistics and English, where relevant. 126 Given the recent introduction of the named degree (B21), completion information at the time of the audit visit was sparse, although the figures available had been considered by the Modern Languages Studies Degree Board, on which the external adviser sits. Detailed progression statistics for individual language courses are available. Given the nature of study at the University, interpretation of the figures is difficult, but Pass rates and the distribution of grades are considered as part of the process of preparation for the course report. 127 Annual course reports form the principal means of monitoring and review of individual courses. In general the audit team found clear evidence of engagement with both the quantitative data, produced by the statistics section of IET, and page 24

30 Institutional Audit Report: main report qualitative responses from students, staff tutors and ALs, where available. Issues were highlighted and, where appropriate, action to address them was proposed. Course reports for compulsory and optional courses of the language studies programme are also considered at the Modern Languages Studies Degree Board. 128 External examiners' reports are sent to course teams via the Dean and Sub-Dean of the Faculty. Course teams consider the reports, and are required to respond to the Faculty on a pro forma detailing the actions proposed in response to the issues raised by the external examiners, and indicating the planned dates for completion of such actions. From the examples seen during the visit there was clear evidence of timely and responsive action to issues raised. However, the mechanism for ensuring that this was always the case did not appear to be sufficiently formalised, and although external examiners' reports are seen by the Modern Languages Studies Degree Board, it is not clear that course teams' responses are always seen as well. 129 In line with University policy, the Department of Languages has moved towards a greater formalisation of learning outcomes, and is in the process of linking these to assessment guides and marking criteria. Assessment comprises a mixture of TMAs (formative and summative) and timed examinations. In order to assess all four language skills together, special examination arrangements have been developed in consultation with Student Services. In consecutive morning and afternoon sessions students take a written examination and participate in an oral test in the target language. At all levels, the test is conducted in a group situation, and comprises an individual student presentation and participation in a group discussion. From the evidence seen by the audit team, this test allows for a rigorous and meaningful assessment of student competence in the target language. Student feedback indicates a high degree of satisfaction with these arrangements. The team was given to understand that there was some variation in assessment patterns within individual courses between languages, but was assured that there is an intention on the part of the Department to move towards a greater degree of harmonisation in this respect. The team was interested to note that feedback on coursework at levels 2 and 3 is normally given in the target language, on the basis that this further contributes to students' learning. Course guides have been developed by the Department to help students check on progress towards learning targets, and were rated highly by students in course surveys for their clarity and usefulness. 130 Assessed work seen by the audit team was in line with the planned outcomes of the programme specification, and showed evidence of high levels of language competence and of critical engagement with aspects of the culture and societies of the target language. External examiners confirmed that the standards of achievement were appropriate to the designated levels, and that the standards achieved by these students matched those of students in other HE institutions with which they were familiar. 131 The course materials are of good quality and include text, video and audio-cd. Students reported to the audit team that they found the materials appropriate and effective in promoting learning. The materials are supplemented by support from ALs who interact with students in tutorials and via telephone or on-line. Students were warm in their praise of the work of ALs to support their learning, and were generally appreciative of the nature and quality of feedback provided on coursework assignments. The extensive and helpful nature of the feedback was confirmed by the examples of assessed work that the team saw during the visit. It was clear to the team that AL feedback itself was carefully monitored, and that constructive comments were made by monitors on the quality of the feedback provided to students, as part of the ongoing training of ALs. 132 A compulsory one-week residential school for level 2 language courses further supports and develops student learning. The school takes place either in the country of the target language, with appropriate activities to ensure interaction with local native speakers of the target language, or in the UK. In the latter case, a policy of total immersion in the target language ensures an experience comparable with a school located abroad. After each residential school, teams hold debriefing sessions which benefit from comprehensive student feedback in the form of completed questionnaires that facilitate the enhancement of future schools. For those few students unable to attend a residential school, an alternative learning experience is available, involving logged periods of extensive on-line discussion in the target language. 133 Students also benefit from internet-based and on-line support for learning through the University's web site and through FirstClass electronic conferencing. Although many students still rate faceto-face tutorial support more highly than electronic support, the outcomes of surveys and students who met the audit team confirmed the quality and effectiveness of the use of ICT as a means of support for learning. The Lyceum system and edesktop support, available for some course units, also make positive contributions to learning. page 25

31 Open University 134 Student feedback derives from detailed and course-specific surveys undertaken by Student Research Centre and through comments passed on to course teams by ALs. Difficulties raised in qualitative feedback mainly related to delays in returning assignments by ALs and general comments about excessive workload. The audit team saw evidence that any problems with ALs had been followed up quickly. A heavy workload for students appeared to the team to be a systemic feature of OU courses, but course teams in the Department of Languages are sensitive to this issue and ready to adjust the teaching programme in order to facilitate student participation and the completion of target activities. While students are not directly involved in the quality management of programmes at course or departmental level, it was clear to the team that the student voice is heard through extensive surveys, and that it has a role in the enhancement of the provision at all levels. 135 There is a clear progression through the language courses at different levels, broadening and developing linguistic competence and increasingly engaging with the culture and societies of the target language. The languages provision, with its emphasis on communicative skills-orientated courses, in combination with one or more compulsory English language components, forms part of a named degree programme which is distinctive from most other modern languages degree programmes offered by UK institutions, while retaining a shape and academic rationale that will find acceptance in the modern languages academic community at large. The audit team considered that the quality of the learning opportunities is suitable for the programme of study leading to the named award, and that the quality of learning resources and the effectiveness of academic support and guidance further enhance the overall quality of the student learning experience. BSc (Hons) in Information Technology and Computing 136 This programme is offered collaboratively by the Faculty of Technology and the Faculty of Mathematics and Computing, and offers a broadly based, modular, distance-learning degree in the subject area. In the course of this DAT, members of the audit team met staff and students in the London Regional Centre, and also met faculty staff based at Walton Hall. Documentation available to the team included the DSED, which was especially written for the audit, examples of assessed student work, course material, external examiners' and external assessors' reports, annual review reports, Degree Board minutes, and programmes specifications. 137 The programme specification is clearly mapped against the FHEQ, for both England, Wales and Northern Ireland, and for Scotland. Due regard has been taken of the Subject benchmark statement for computing in determining the degree curriculum. The programme specification is generally accurate although, as noted in the minutes of the IT and Computing Degree Board, the current programme curriculum does not necessarily ensure that all students will achieve all the programme learning outcomes as defined in the specification. The DSED explained that to address this matter, a version of the programme specification, showing those learning outcomes that would necessarily be achieved, has been prepared and will be made available to students. Over time, revisions to courses are anticipated, and it was reported to the audit team that these changes will be made to ensure all learning outcomes as defined in the programme specification approved by the Degree Board will necessarily be achieved by graduates of the programme. 138 Extensive progression and completion data exists at course level, encompassing demographic, aggregated and previous years' data, the latter being used to generate trend analysis. Award-level data is more limited, and the DSED pointed to difficulties in presenting progression and retention data at degree level, due to widely varying completion rates and the relatively low numbers of students registering for the named award at the outset of their studies. 139 Data is reviewed as part of the annual course review. In addition to considering progression, retention and completion data, annual review also considers feedback from external examiners and external assessors, and student feedback. The audit team found that annual review was effective in monitoring achievement of standards and in assuring quality, and that significant issues were identified and addressed. This degree programme has yet to undergo the University's recently introduced award-level periodic review process. 140 External examiners' reports on courses are reviewed by course teams, and it was clear that issues raised were given adequate consideration, with full responses provided to the external examiners. External assessors' reports are addressed in an equally thorough manner and, additionally, are reported upon within the annual award report. 141 The assessment strategy for each course is designed by its course team in conformity with the University's assessment policy. Assessment of student work accurately reflected achievement of learning page 26

32 Institutional Audit Report: main report outcomes as defined in the programme specification. This was a view confirmed by the external examiners, who were generally positive about the achievement of students. Feedback to students on TMA assessment was considered by the audit team to be of good quality, and was highly valued by students. The team welcomes the recent initiative, outlined in the DSED, to provide equally helpful feedback on examinations and end-of-course assessments. The team found that the standard of student achievement is appropriate to the title of the award and its location within the FHEQ. 142 Student guidance provided at course level clearly lays out the structure, objectives and requirements of the courses. At award level, a programme web site is provided. This was found to be generally complete and helpful. Staff who met the audit team reported that award-level learning outcomes are to be added to this site. Course learning materials were found to be comprehensive and of excellent quality. Students reported to the team that ALs and staff tutors are approachable and supportive. 143 All courses in their first year of presentation and those due for review are subject to a course survey which reports on a wide range of generic and course-specific matters, including issues related to teaching, learning, assessment, learning resources and support. Course teams are required to comment within course reports on how they have addressed feedback. The audit team found that this requirement had been met, and that feedback had been used effectively to improve course delivery. Given the wide geographical dispersion of students on the programme leading to this award, there is no equivalent of a staff-student liaison committee, but at course level students can access and contribute to discussion in the web-based FirstClass conferencing area. The team noted that these electronic discussions are regularly viewed by academic staff for quality enhancement purposes. 144 The audit team concluded that the programme provides an appropriate learning experience for students, comparable with programmes delivered through more traditional modes in computing at undergraduate honours level. The team was satisfied that the quality of learning opportunities available was suitable for a programme of study leading to an award within the subject area. BEng and MEng 145 The DAT covered the BEng and MEng programmes offered by the Faculty of Technology. These are specific types of degree but do not carry titles defining a particular area studied in engineering. They are, therefore, not fully 'named degrees', but are considered as such in terms of the University's quality processes. Although the Faculty had for a number of years offered profiles of courses leading to open degrees deemed appropriate by some engineering institutions for graduates seeking professional recognition, it was not until 1997 that the MEng was formalised as a 480-credit programme. This followed extensive consultation with representatives of the Engineering Council, the Engineering Employers Federation and the Institutions of Mechanical Engineering and Electrical Engineering. A 360-credit BEng programme was launched in In the course of this DAT, members of the audit team met students and regional staff at the Scottish Regional Centre in Edinburgh, and met faculty staff based at Walton Hall. 146 The DSED was written for the audit rather than being derived from an internal review report, although it made reference to a wide range of internal documents. Programme specifications for each programme were appended to the report.a detailed mapping of the learning outcomes from the array of courses that can contribute to these programmes demonstrated comprehensive coverage of the Subject benchmark statements engineering at honours and masters level. Staff who met the audit team agreed that the next step in the development of the programmes would be to use the mapping data to ensure that a particular set of courses selected by an individual student would enable all the programme learning outcomes to be met. The team noted that the Engineering Degree Board monitors the profiles of courses selected by students, and staff indicated that this could lead to recommendations of particular pathways. Students commented to the team that they would welcome more liaison between the University and the engineering professional bodies so that they could be more certain of the professional acceptability of their selected pathways. Both awards have been considered in relation to the FHEQ. The programme specification for the current MEng programme states that it does not fully comply with the FHEQ descriptors and, as a result, a revised version of the MEng programme will be offered from Progression and achievement data on a course-by-course basis from the past three years were provided to the audit team. The team was able to confirm that these data had been considered by course teams and included in course reports. Staff who met the team cited examples of actions taken with respect to specific courses in response to poor progression data. The Engineering Degree Board reports annually on student achievement at programme level with trend analyses of the class of awards. page 27

33 Open University 148 Monitoring is undertaken primarily at course level, with a course report prepared after each presentation. The course reports seen by the audit team were somewhat perfunctory in nature, and could have benefited from a greater level of evaluation. These reports are considered at a programme board which reports to Awards Committee, and the Faculty reports on all its courses annually to QSB. The team saw examples of programme board minutes and annual faculty reports. It was evident from these that detailed consideration is given to course reports and the associated reports of external examiners. There was also evidence of the sharing of good practice within the Faculty between courses. Course teams receive feedback from ALs and from the external assessor. The team saw evidence that this feedback is considered by course teams, and of action taken as a result of such feedback. 149 Staff who met the audit team confirmed the process for the consideration of external examiners' reports: initial consideration and actions by the course team, followed by a summary report of all external examiners' reports across the faculty prepared by the sub-dean, presented first to the Faculty Board of Studies, then to the Assessment Policy Committee for consideration at University level. Examples of Course Examination and Assessment Board, Faculty Board and Assessment Policy Committee minutes were provided to the team. The team noted that external examiners' reports are specifically excluded from course reports although they might usefully be considered there alongside other sources of feedback. The team also saw evidence that external advisors' reports are considered by the Engineering Degree Board and incorporated in its annual report to Awards Committee. 150 Individual course assessment strategies comply with the University's common assessment framework. Staff who met the audit team stated that course assessment strategies are approved at faculty level by the Faculty Teaching and Curriculum Committee, and at University level. Staff also confirmed the rigorous marking procedures, including double marking, and the subsequent standardisation process involving scrutiny of marks through analysis by question and by marker. Students confirmed to the team their general satisfaction with both the quality and timeliness of feedback from staff, although they commented on some instances of poor legibility in handwritten comments. The team studied examples of students' assessed work. These illustrated the detailed feedback provided by tutors. The team considered, on the available evidence, that the standard of student achievement is appropriate to the named awards and their location within the FHEQ. 151 Details of each course are made available to students through the University's web site, and the audit team was able to study these. Students confirmed to the team their satisfaction with the appropriateness and accuracy of this information. The team saw student versions of the MEng and BEng programme specifications, and noted that the programme learning outcomes in the key skills section of the full programme specification were not replicated in the student version. The student version was, however, more explicit about how skills would be taught and assessed. 152 Students confirmed a high level of satisfaction with tutor support, praising in particular the access to ALs by telephone and . The individual nature of the support was highlighted, with one student stating that the OU offers 'a bespoke experience'. They also confirmed that additional support is provided for students studying OU courses for the first time. They expressed the view that course materials are excellent, although they mentioned that there were instances of materials, particularly videos, becoming dated during the life of a course. Practical facilities are provided at residential schools, and through the provision of home experiment kits. 153 Examples of student feedback on courses were seen by the audit team. Staff who met the team confirmed that course teams receive student feedback data from the IET with aspects of particular satisfaction and dissatisfaction highlighted. The course team responses to these are recorded in course reports. The team heard that student surveys dealing with aspects of regional learning support are also conducted in regional centres, this feedback being considered by regional staff tutors. Students told the team that they were able also to express their views through their ALs, and were confident that their voice was heard, although they were not aware of any specific mechanism for receiving responses to their comments made through this route. Overall, the team was satisfied that the quality of the learning opportunities available to students is suitable for programmes of study leading to the awards of BEng and MEng. MBA 154 The scope of this DAT is the MBA programme offered by the Open University Business School (OUBS). The MBA is a large programme with some 13,500 students having completed it since its inception in The OUBS has approximately page 28

34 Institutional Audit Report: main report 190 permanent academic and support staff, and employs some 750 part-time tutors who deliver the programme through the regional centres. In the course of this DAT, members of the audit team met staff and students at the South West Regional Centre in Bristol, which is also a regional hub for the programme, and met faculty staff based at Walton Hall. 155 The DSED was written specifically for the purposes of the institutional audit. The MBA programme was significantly revised in 2002, with the first delivery of the revised programme commencing in November There is a detailed programme specification which includes aims which are stated in the DSED to relate explicitly to the Subject benchmark statement for masters awards in business and management and to the FHEQ. 156 Monitoring student progression is complicated because students are not required to declare an award intention at first registration, and may study for individual courses before they declare that they intend to complete the MBA. Wastage rates within the MBA courses are stated to be low by the University's norms, but a significant number of students do not complete the entire programme. This proportion increased after the first few years and then stabilised. 157 Internal monitoring and review is currently based mainly on courses, although the audit team was told that it is planned to operate more effectively at the programme level in addition in the near future, following the setting up of the MBA Programme Board and the appointment of external advisers. Course teams produce annual course monitoring reports which draw on statistical information relating to student achievement and student feedback surveys, and comment on various aspects of course delivery, although they exclude any reference to external examiners' reports. In most instances the course reports that the team saw were brief and not very evaluative. Annual course reports are considered by the Masters Programme Committee (shortly to be supplemented by the MBA Named Degree Board), together with the reports of external examiners. The Programme Manager and Programme Leader may require course teams to respond to issues identified by the Programme Committee. The team noted that an issue relating to course delivery, identified by student surveys, was successfully resolved although it had not been referred to in the relevant course report. The team is confident that the overall course monitoring process is effective, but would encourage course teams to produce annual reports which are more evaluative. It also welcomes the development of a programme monitoring process when the MBA Named Degree Board is set up. 158 An external examiner is appointed for each course. They produce a report on each of the two presentations of the course each year, using a standard form which requires comments on a range of issues including the assessment strategy for the course and the quality of candidates compared with previous cohorts and with standards in other institutions. External examiners' reports are considered by the course team and the Programme Committee, and issues thus raised are tracked by the use of a standard form, which records the actions taken. An annual summary of all of the MBA external examiners' reports is produced, again on a standard form which identifies issues of excellence, areas of common concern and actions taken at faculty level, and this summary is considered by the University's Assessment Policy Committee. The audit team saw external examiners' reports for each of the MBA courses for the last three years. These made clear that the external examiners were satisfied that the academic standards were appropriate, and that their comments had been responded to appropriately. The team considered that the most recent annual summary report was an effective mechanism for drawing together all of the issues raised the MBA external examiners. The DSED stated that two external advisers will shortly be appointed to the MBA to oversee the programme as a whole, one academic adviser and one practitioner. The team formed the view that the external examining process is effective, and considered that the addition of external advisers to operate at the programme level was a valuable and timely step. 159 All of the MBA courses have clearly stated assessment processes which are designed to check student progress against the stated learning outcomes. As part of the development of a programme focus for the MBA, the active management of assessment across the MBA programme as a whole has become essential, and an Assessment Working Group has been set up to consider the issues involved. Feedback on continuous assessment is provided by ALs, and is carefully monitored. Feedback is not given on examination performance, but the DSED explained that ways of giving such feedback are being considered. Overall, the audit team came to the view that the assessment strategy of the MBA is effective and in accordance with the Code of practice. 160 The audit team studied examples of assessed work in the form of marked coursework assignments and examination scripts for a range of courses page 29

35 Open University within the MBA programme. Feedback to students on the continuous assessment was detailed, and would have provided a valuable learning resource for those students. On the basis of its study of marked work, the team was able to confirm that the academic standard and content of the work is appropriate to the award of MBA. 161 The MBA programme is currently focused on its constituent courses and, while information about each course is extensive, the audit team did not see much information about the programme as a whole. Although programme learning outcomes are stated in the programme specification, they have not up to now been communicated to students. It was reported to the team, however, that a student version of the programme specification is being prepared and will be included in programme information to students by the end of Students who met the team were positive about the information they received, which they found to be accurate and useful. 162 Each course has an extensive range of learning materials including specially written course texts, audiovisual resources, on-line conferences and software. In line with the University's Teaching and Learning Strategy, new learning media are playing an increasing role in teaching and learning processes, and all MBA students are required to have access to computing and internet facilities. The audit team saw a complete set of materials for one course, and took particular note of the volume and quality of the material, which it found to be appropriate to the level of the course. Students who met the team were positive about the quality of the learning materials and their variety, and confirmed the outcomes of student surveys that the printed material was the most highly valued resource. ALs support students through correspondence, telephone, and face-to-face tuition or on-line tuition for those unable to attend face-to-face tuition. Students were generally positive about the face-toface tuition, although they commented to the team on some considerable variation between tutors. 163 Students are also supported by staff based at Walton Hall and at the MBA regional hubs. Students can access a wide range of information, advice and guidance services through on-line resources or by visiting their regional office. The students who met the audit team commented positively on their ability to communicate with tutors and regional staff, and on the way in which the University tries to link students in study groups and in on-line conferencing. They were also positive about the support they receive if they have personal problems that affect their academic work. 164 Student views are sought via student surveys which include questions on the achievement of objectives. All of the MBA courses have been surveyed over the last three years, with most of them being surveyed in 2003 as part of the review of the MBA. Most courses showed high levels of student satisfaction. For courses where there was a relatively lower level of satisfaction there was evidence of course teams taking corrective action. Overall, the audit team formed the view that there are appropriate and effective mechanisms for gaining feedback from students on the quality of their learning experience, and that appropriate action is taken in response to the outcomes of such feedback. 165 Overall, the audit team was satisfied that the standard of student achievement is appropriate to the award of the MBA, and that the quality of the learning opportunities is suitable for a programme of study leading to the award of the MBA. It considers that the appointment of external advisers and the establishment of the named Programme Board will further strengthen the management of quality and standards of the MBA programme as a whole. Section 4: The audit investigations: published information The students' experience of published information and other information available to them 166 The University provides a wide range of information for the use of students, including promotional materials, prospectuses, induction material, learning materials, learning and study skills guidance, guides to OU services and University regulations. These are provided variously as hard copy, CD-ROM, DVD, and/or via the internet. 167 The SED described how this information undergoes a rigorous approval process. The Student Services Communications Team collates and disseminates the majority of centrally provided student information, with the Group for the Review of Promotional Publications Strategy having a specific responsibility for the University's promotional publications. Course material is approved as part of the course approval process. Market research is used to assess readers' perception of material. The end-of-course survey routinely seeks student views on published materials. 168 Students who met the audit team had broadly positive views of the University's published information. Prospectuses were considered to be informative, full and well presented. The courses and page 30

36 Institutional Audit Report: main report qualifications web site was singled out for particular praise, and was considered by students to contain much useful and accurate information on courses and awards. The SWS described Sesame, the student newspaper provided by the University to all students, as 'invaluable'. These views are corroborated by student feedback; the overall quality of information, advice and support provided was rated highly in the 2002 November end-of-course survey, with more than 95 per cent of respondents being either 'fairly' or 'very' satisfied with the information in the Student Handbook, and 94 per cent of respondents declaring themselves to be 'fairly' or 'very' satisfied with administrative information presented on the University's web site. The team studied a sample of such material from which it was able to draw a similarly favourable view. 169 Students who met the audit team regarded their course material as generally excellent. Students noted that there had been some issues concerning continuing presence of errata in print materials, or learning materials not being available at the commencement of a course. The team did not deny the seriousness of such issues, but accepted the evidence that these problems occurred rarely in a very large portfolio of materials, a reflection of the soundness of the mechanisms the University has in place to ensure the timely production of good quality course materials. 170 Students raised with the audit team the issue of course cancellations. Cancellations have a particular significance when the courses are part of a named award. From its discussions with senior staff of the University it was clear to the team that scrupulous attention is paid to these matters. It was reported to the team that presentation of courses is only delayed or cancelled where quality would otherwise be compromised, and the approval of the PVC (Curriculum and Awards) is always required in such cases. The University has mechanisms to ensure that students are kept informed of changes to intended delivery of courses, and the team formed the view that these are generally effective but not infallible. 171 The University has made clear its intention to produce more than one version of programme specifications, with one version to be publicly available for the use of students. The audit team noted that these 'student-friendly' versions, as presented during the audit, were not always consistent with the definitive programme specifications as approved by award committees. The University may wish to reflect on how best to ensure consistency between the definitive programme specifications as approved by degree boards (or equivalent), and versions produced for other purposes, bearing in mind the possible consequences of using different versions for different audiences. 172 Based on the views of students and an examination of the documents supplied to students, the audit team concluded that the University has effective means of ensuring the accuracy, clarity and utility of information supplied to students. Reliability, accuracy and completeness of published information 173 The audit process included a check on progress towards meeting the requirements of HEFCE's document 03/51, Information on quality and standards in higher education: Final guidance. Responsibility for ensuring the production of a reliable, accurate and complete set of published information ultimately lies with the University's QSB which, through the Monitoring and Feedback Group, mapped the University's provision against the data and information requirements identified in HEFCE's document 02/15 Information on quality and standards in higher education, thereby identifying those areas where further action was required. The University was one of six UK HE institutions to participate in the national teaching quality information (TQI) pilot conducted in 2003, an activity which further informed the University's plans for publishing information on quality and standards. The University's action plan was updated in the light of the publication of HEFCE 03/ The SED noted some difficulties currently faced by the University in meeting all requirements as set out in HEFCE 03/51, due to the particular nature of its provision, with its current focus on the course rather than the award. The University is actively considering how it will address these issues, investigating how best to provide award review reports, and has work in progress concerning the tracking of students who enter the University as part of a named degree. The audit team considered these to be valuable initiatives, and concluded that the University is making appropriate progress towards delivery of the data and information specified in HEFCE 03/51. page 31

37

38 Findings

39 Open University Findings 175 An institutional audit of the University was undertaken during the period 22 to 26 March The purpose of the audit was to provide public information on the quality of the University's programmes of study and on the discharge of its responsibility as an awarding body. As part of the audit process, according to protocols agreed with HEFCE, SCOP and UUK, four DATs were conducted. This section of the report summarises the findings of the audit. It concludes by identifying features of good practice that emerged during the audit, and making recommendations to the University for action to enhance current practice. The effectiveness of institutional procedures for assuring the quality of programmes 176 From the inception of the University, the course or single credit-bearing module has been the fundamental building block of all its taught provision, and the University's strengths in the production of courses and course-materials are recognised not only by its students, but nationally and internationally. The University remains strongly committed to an educational provision characterised by the development of the individual course of study and the materials that support it. The course continues to be the main reference point for the University's procedures for assuring quality and standards, and it was explained to the audit team that the course is the 'unit of account'. 177 Since 2000, the University has additionally been developing programmes leading to named awards, initially through the explicit 'linking' of extant courses by subject focus, but increasingly by designing new courses with the needs of programmes in mind. With the introduction of named programmes, the University has set up new quality assurance mechanisms. The University's quality management system now includes procedures for programme design and approval as well as course design and approval, annual monitoring of programmes as well as annual monitoring of courses, and for periodic review of programme areas. However, it appeared to the audit team that the shift in perspective implied by a change from a course-centred to a programmecentred model was still not fully internalised by the University, and that the quality assurance measures introduced to deal with the demands of programmes tended to be supplementary to those already in place, rather than transformative of them. 178 Despite the University's recent attention to the development and identification of learning outcomes, the audit team formed the view that courses tend to remain content-led rather than outcomes-led, with more focus on what is taught than on what is learned. The procedure for approval of new courses appears often to permit a tendency to include an overload of material and/or assessment, a design fault that is regularly acknowledged by both students and staff. For new programmes, the approval procedure includes no external check on the coherence between intended leaning outcomes and programme content prior to approval being granted for the establishment of the award. The team noted instances where programmes had been approved although not all possible pathways yielded the programme's intended learning outcomes, and considered that this problem might arise from programmes being derived from, and primarily dependent upon, courses, rather than courses being designed to support programmes. 179 The progress and outcomes of the procedures for quality management are considered by a highly participative structure of committees. The Senate is at its head, supported by the Academic Board, which has six University-level boards reporting to it, each with further sub-boards with specific remits. The faculties, schools and institutes of the University have analogous quality management substructures, with reporting lines into the University-level boards. Regions, too, have an extensive committee structure, and full and part-time regional staff are engaged with committee and development work based at Walton Hall. The audit found that regional operations are an effective and appropriate aspect of the University's provision. Regional staff are an integral part of the academic community of the University, and of its quality management structures. The support provided by staff in the regions is valued highly by students, and the commitment and collegiality of staff, and their identity with the mission of the OU, was evident to the audit team. 180 The University's QSB develops quality assurance policy and procedures, while the CAB, LTB and SPB are accountable to it for implementation of the various procedures for both courses and programmes. It appeared to the audit team that the description of the different responsibilities of the various University-level boards does not always match practice, the Quality and Standards Board dealing directly with several non-procedural matters. The team noted that monitoring of courses is overseen by QSB, while the CAB is responsible for the monitoring of programmes. The QSB operates a mechanism for ensuring that the processes are carried out, but there is no point at the senior level page 34

40 Institutional Audit Report: findings where an overview of the quality of provision overall is considered. The team was informed that the University is engaged in a review of its arrangements for academic governance, and considered this will provide a useful opportunity for the University to reflect upon whether it has yet achieved the most effective and efficient means for maintaining its oversight of the quality of courses and programmes. 181 The University has comprehensive systems for gathering student feedback and for its central analysis. Student satisfaction in its broadest sense is a key element in its methods for evaluating its courses and programmes and the materials that support them. The nature of the student population dictates that this feedback is usually by questionnaire or similar means. Less formal channels for enabling student feedback are provided by a range of computer-based conference facilities maintained by the University, and these are well used. A routine gathering of student views about the University's learning support services offered in the regions is reported on annually at the SPB. The University offers extensive representation of the student body on committees at central and regional levels, and has a close relationship with the OUSA. Gathering of student feedback, formally and informally, and using it in the evaluation of courses and programmes and of the effectiveness of learning and student support is taken very seriously by the University, and is one of its particular strengths. 182 The audit team considered that processes for assuring the quality of courses and programmes were operating effectively, although the quality assurance measures introduced to deal with named programmes tended to overlay those already in place for courses. Overall, the findings of the audit confirm that broad confidence can be placed in the soundness of the University's current and future management of the quality of its programmes. The effectiveness of institutional procedures for securing the standards of awards 183 The University's framework for securing academic standards is managed through the committee system, with the most significant committees for this purpose being the QSB and the CAB, both of which report to the Academic Board. The QSB is responsible for approving and reviewing the arrangements for setting and reviewing academic standards while the CAB formulates and keeps under review the institutional strategy and policy for matters relating to awards and assessment, and is responsible for ensuring that courses and awards are consistent in standard and comparable with other UK universities. The audit team considered the division of responsibility for standards between the QSB and the CAB to be complex, but nevertheless effective. 184 Student statistics on progression and achievement are published internally and externally. A range of statistical data is produced at the course level and is used as part of the annual course monitoring process. The RRACP is responsible for the monitoring of standards in approved awards, and does this by reviewing the results of each course with previous results of the course and with the results of similar courses and assesses the consistency of standards. The audit team considered that the RRACP provides an effective additional mechanism for maintaining the academic standards of courses. 185 The system for external scrutiny of the academic standards of awards includes external examiners at the course level and external advisers for named degrees, both of whom are required to produce annual reports. At the end of each awarding period external examiners submit a pro forma-based report to the Vice-Chancellor on the appropriateness of the course materials, the course assessment strategy, the administration of the examination process, the quality of the candidates and whether previous issues raised have been addressed by the University. External advisers report to the Vice-Chancellor on the maintenance of standards and the continued validity of the named award, on the consistency of the aims of the programme with the relevant subject benchmark statements and whether the standard of the award is comparable with the expectations of similar awards in the UK. 186 The audit team considers that the University has appropriate procedures to ensure that the external examining system is working effectively at both course and named programme level. Through the DATs, the team reviewed external examiners' reports for a range of courses and external advisers' reports on some named programmes and was able to confirm that prompt responses to comments had been made. All of the reports seen by the team confirmed the security of the academic standard of the University's awards. 187 The University's mechanisms for securing the academic standards of its awards will be strengthened by the implementation of its procedures relating to the award of named degrees. The findings of the audit confirm that broad confidence can be placed in the in the soundness of the University's current and likely future management of the academic standards of awards. page 35

41 Open University The use made by the institution of the academic infrastructure 188 The University considered the sections of the Code of practice as they were published, and made adjustments to its policies and processes where appropriate. The audit team formed the view that the University had given all sections of the Code appropriate consideration, and had aligned its policies and processes with the Code. The team supports the University's view that its code of practice for supervisors and research students embodies the principles of the Code, and also reflects Research Council requirements. The University's position statements on alignment with the Code are kept up to date by the Quality Assurance Office and are available on the University's web site. 189 The University has its own levels and awards framework, which has been reviewed against the FHEQ and amended, where appropriate, to bring it into line with national norms. The audit found that the University's levels and awards frameworks is consistent with the FHEQ. The framework is used to inform the development of programme specifications for new awards, and provides useful guidance for staff with responsibility for course and programme approval and monitoring. 190 Programme specifications have been produced for all of the named degree programmes and for the BA/BSc (Open) degrees, although these are not generally made available to students. Much of the information in them, including programme learning outcomes, is being rewritten with the student audience in mind, and will be communicated to students with other programme information. The audit team studied programme specifications during the DATs, and considered that the definitive programmes specifications, as approved by programme/award boards, were appropriate in content and scope. 191 Subject benchmark statements are used as reference points in the formation of programme specifications, and the programme specification template requires explicit reference to them. The team saw evidence during the DATs that subject benchmark statements are used effectively and are reflected in programme specifications. Overall, the audit found that the University is making effective use of the academic infrastructure. The effectiveness of institutional procedures for supporting learning 192 The University supports student learning through well designed and well presented teaching materials; academic support and guidance delivered by ALs and monitored by Staff Tutors; extensive internet-based resources including general guidance and advice on learning, and more subject-specific resources accessed through the OU's Library web site. Students commented very positively to the audit team about the quality of the teaching materials. They appreciated the current mix of print and electronic media in which these materials are currently presented, and some expressed a hope that the University's planned move toward more ICT-based delivery would not have a restricting effect on the variety of media used to support learning. 193 Personal support offered to learners recognises the diversity of its student body, and students who met the audit team were aware and appreciative of the various aspects of personal support available. With its agenda for widening participation, and with the growing number of students from younger age groups taking up study with the University, there are implications for an approach to student support that may well be different from that which is suited to the needs of the mature, life-experienced, 'traditional' OU students. The University is aware of this, and is taking action to prepare for an extended range of student support needs. 194 The audit team noted the positive comments it heard from research students about their experience, and about the actions taken to provide and review the support and training mechanisms available to them. It took particular note of University's thirdparty monitoring process, and would encourage the University to ensure the system is universally adopted by all faculties and academic units. 195 The University takes very seriously its responsibility to support and develop its staff. It has in place appropriate academic structures to enable a strategic overview of staff development and provides a comprehensive range of staff development opportunities. Provision of staff support is responsive to the University's policy for the development of its provision, as exemplified by the provision of ICT training opportunities to support the planned move toward more ICT-based course delivery. The arrangements for appointing, monitoring and supporting AL are robust. Outcomes of discipline audit trails BA (Hons) Language Studies (with a focus on Spainish and German language elements) 196 The programme offers learners coherent pathways leading to learning outcomes that are appropriate to the level of the award. The standard page 36

42 Institutional Audit Report: findings of student achievement in languages is appropriate to the title of the award to which it currently contributes (B21) and to which it will contribute (B30). The standards achieved by students are appropriate to the level of award. 197 There is evidence of careful thought in the preparation of the languages component within the named degree programme, and of responsiveness to feedback. Comments from external examiners are generally responded to in a timely and appropriate fashion. Learning outcomes are aligned with the appropriate subject benchmark statements. The range of learning opportunities offered is sufficiently diverse, and their nature is appropriately challenging for the level of award. BSc (Hons) Information Technology and Computing 198 Based on a study of assessed student work, course material, external examiners' and external assessors' reports, annual review reports, Degree Board minutes, programmes specifications, and from discussions with students and staff, the audit team formed the view that the standard of student achievement was appropriate to the title of the award and its location within the FHEQ. The programme specification draws appropriately on subject benchmark statements and the FHEQ. The programme benefits from high-quality learning materials, and students benefit from an effective system of academic support. Student evaluation of the programme was very positive. The team was able to conclude that the quality of the learning opportunities is suitable for programmes of study leading to the named award. BEng and MEng 199 The programme specifications for the programmes leading to the awards of BEng and MEng set out appropriate educational aims and learning outcomes, and link these clearly to the teaching and learning styles. The specification for the award of MEng is currently not consistent with the FHEQ, and is shortly to be replaced. Detailed mapping of course learning outcomes to the Subject benchmark statement for engineering has demonstrated that the range of courses associated with the programmes provide comprehensive coverage of the benchmark statement, although further consideration should be given to whether all the learning outcomes can be met by particular combinations of courses that a student may select. There was evidence that feedback from external examiners, external advisors and students is considered and acted upon. From its study of students' assessed work, and from discussions with students and staff, the audit team formed the view that the standard of student achievement in these programmes is appropriate to the title of the awards. 200 Students who met the audit team expressed their satisfaction with the learning materials and the learning support available to them. Student feedback on courses is generally positive, and external assessors are complimentary about the course structures and teaching learning and assessment strategies. The team concluded that the quality of the learning opportunities available to students is suitable for programmes of study leading to the awards of BEng and MEng. MBA 201 From its study of students' assessed work, and from discussions with students and staff, the audit team formed the view that the standards of student achievement in the MBA was appropriate to the title of the award and its location in the FHEQ. The programme specification demonstrates overall alignment with the FHEQ, and reference is made to the Subject benchmark statement for masters awards in business and management. Opportunities are provided for students to feedback on courses, and there is an annual process whereby courses are evaluated by the Programme Committee. Student feedback on courses and learning resources are positive. The team concluded that the quality of the learning opportunities available to students is suitable for a programme of studies leading to the MBA award. The utility of the SED as an illustration the institution's capacity to reflect upon its own strengths and limitations, and to act upon these to enhance quality and standards 202 The SED prepared for this audit was comprehensive, and while it was generally clear and informative, some inaccuracies in the descriptions of reporting lines and the operation of processes emerged during the audit visit. The detailed and helpful descriptions explained the changes that had taken place since the last audit. The SED identified some strengths, particularly the integrated planning process and the way that process determines priorities for enhancement. It conveyed a sense of some propensity for overload in the context of the internal review process, but did not otherwise engage to any great extent in evaluation of the effectiveness of the University's procedures for the management of its quality and standards in courses and programmes. The impact of the move to named degrees and the implications did not feature prominently, but the SED usefully signalled the University's view of necessary changes in its strategic direction, and clearly page 37

43 Open University articulated the University's significant change agenda, its context and the actions being taken. Commentary on the institution's intention for the enhancement of quality and standards 203 The SED stated that 'the University's intentions for the enhancement of quality and standards at institutional level are encapsulated in a number of institutional strategies' which interpret the institutional high level objectives into development priorities. It further stated that the institutional planning cycle underpins the quality assurance processes, integrates planning at University and unit level and includes regular review. The audit team found that the planning process was rigorous and well evidenced, and that through it, the University had identified and planned to respond to key challenges. Annual review processes were identified as a key vehicle for monitoring standards and disseminating good practice, and the impact of academic and service reviews is clear in the University's revision of its strategic plan. It was less clear, though, how the priorities for enhancement are determined in the planning process, and how these priorities are articulated to the academic community as a whole. 204 In discussions with the audit team, the University's quality strategy was articulated as focusing on ensuring that processes for the evaluating that the operations of all services and boards are carried out in an effective and reflective manner. However, this was not expressed explicitly in the University's formal documents, and given that the University had signalled in the SED the tendency for 'too much review', the audit team would encourage the University to clearly articulate its strategy for the enhancement of quality and standards, and how it is reflected in its frameworks for quality management. This will be particularly important as more named awards are introduced, and as the University addresses other challenges in the external higher education environment. Reliability of information 205 The University provides a wide range of published materials, including teaching resources, associated learning resources and student support materials. The audit team noted the systematic approval process which the University applies to these publications. Course materials are supplied in a variety of media, and attention is given to using media appropriate to both the nature of the material presented and to the resources and capabilities of the student. The team noted that in some cases there were contradictions between the contents of programme specifications given to students and those understood by the University to be the approved versions. Overall, however, the team found the quality of the teaching and learning support materials to be very good, and held in high regard by students. 206 From the autumn of 2004 the audit process will include a check on the reliability of the information set published by institutions in the format recommended in HEFCE 03/51. The University has mapped current provision against the requirements and has well developed plans in place to provide for these requirements. The audit team was satisfied that the information that the University was publishing currently about the quality of its programmes and the standards of its awards was reliable. Features of good practice 207 Of the features of good practice noted in the course of the audit, the audit team noted the following in particular: i the way that the University monitors the security of its academic standards through the RRACP (paragraphs 38; 41); ii the systematic and comprehensive collection and use of feedback from students (paragraph 84); iii the arrangements for appointing, monitoring and supporting ALs (paragraphs 92; 96; 98); iv the proactive stance taken by the University in giving guidance and support to students (paragraphs 110; 118); v the third-party monitoring system for research students (paragraph 113). Recommendations for action 208 The University is advised to: i take the opportunity offered by the governance review to articulate more clearly, for the benefit of the University community as a whole, the University's approach to assuring and enhancing the quality of provision (paragraphs 40; 41; 45; 46; 59); ii review the effectiveness of its procedures for determining whether intended programme learning outcomes will be met through all pathways leading to named awards (paragraphs 50; 137); iii systematically include external subject expertise in the procedures for approving programmes leading to named degrees (paragraph 51); page 38

44 Institutional Audit Report: findings iv consider the effectiveness of its present system for gaining a University-level overview of annual review activity at course and programme levels (paragraph 55); v reflect on the possible consequences of using different versions of programme specifications for different audiences (paragraphs 137; 151; 161; 171). The University is also invited to consider the desirability of: vi ensuring that research students have clear information about their financial entitlements (paragraph 115). page 39

45 Open University Appendix The Open University's response to the audit report The University welcomes this report of its institutional audit held in March 2004 and the judgement of broad confidence in the soundness of the University's current and likely future management of the quality of its programmes and the academic standard of its awards. The quality of learning opportunities available to students on the four audit trails carried out by the team were also judged to be 'suitable for a programme of study leading to that award'. We were pleased that the audit identified a number of areas of good practice relating to the management of quality and standards. We are already responding to the recommendations, and can report that: the University is already engaged in the process of a review of academic governance, and the terms of reference of new or revised boards and committees for academic governance will include clear responsibility for assuring and enhancing the quality of provision; we have acknowledged that our present system for gaining University-level overview of annual review activity at course and programme levels currently involves some duplication of effort and we shall keep it under review as we begin to locate more of our services to students at programme, rather than course, level though this is intended to be quite a gradual process; we are making arrangements to include external subject expertise systematically in the procedures for approving programmes leading to named degrees; whilst we believe that we already have robust procedures in place for determining whether intended programme learning outcomes are met through all pathways leading to named awards, we shall keep these under review; we are continuing to develop 'student-friendly' versions of programme specifications for publication, and will put in place rigorous processes for checking that all versions of these documents are consistent; we are reviewing the information provided for research students about their financial entitlements. This audit came at a time of considerable change for the University, and we welcome the Agency's contribution to that process. page 40

46 RG /04