1 Using survey data to inform and target curriculum improvement Key Contacts: Vicky Marsh / Rebecca Galley
2 Agenda 1 Background to the Project 2 Analysis to Date 3 Satisfaction modelling 4 Next Steps
3 Background to the Project SEaM Student Experience on a Module (SEaM) survey: methodology, exclusions & reliability Annual Quality Review OU Annual Quality Review (AQR) process: Summary & integration of student outcomes Learning Design Learning Design at the OU: Background & practice Cycle joining it all up Current practice at the OU in terms of linking learning design to student outcomes and quality review
4 SEaM Overview Revised Survey in place for modules that started from 2012J+ Every student on every module will be surveyed Integrated survey including the student feedback on their tutor and on their module Three themed sets of questions which cover The module overall Teaching, learning and assessment Feedback on the tutor (Tutor supported modules only) Students are sent an invitation to participate two to three weeks before the end of the module Coverage includes presentations that have ended between 1st August 2013 and 31st July 2014, who had results available by 13th August 2014 There are a selection of short course, residential school and overseas modules included in AQR that were not surveyed, these modules are not included. All students regardless of their completion status are included - all previous SEaM results were based on students who completed only. Reliability Consideration should be given to response rates (34%) and the number of responding students when reviewing SEaM results. Non-response bias there is some evidence of bias, but the impact on survey estimates is minimal 4
5 OU Annual Quality Review process Annual Data Release Programme Committees PVC Prog Review Interim Data Release Internal Prog Review End of modules Annual Data Available September October November December January February March April May June July August Student Intake Student Intake Monthly & daily qualification, module and student data available Annual/interim data release includes: Module pass rates Module SEaM survey results Five year module SEaM survey and pass rate trends Qualification NSS, PTES, PRES, DLHE survey results Qualification progress rates Qualification classification and award data Module VLE & Learning Design information Monthly/Daily real time data includes: Module SEaM survey results Module profile e.g. socio-demographics, assignment submission Module flows e.g. concurrent study & next modules studied Qualification profile e.g. which modules are students studying Module activity charts VLE activity, registrations & assignment submission
6 OU Learning Design The term learning Design describes a field of educational research and development that seeks to enhance the processes involved in designing, describing and sharing learning activities, to support educators design practices. Learning design at the OU is characterised by 3 principles: Mechanisms to encourage design conversations across disciplines and expert roles The use of tools and instruments as a means of describing and sharing design The use of information and data to inform the conceptual tools and frameworks that guide the decision-making process OU Learning Design workshops are seen to support the creative design process: By providing a common language and set of representations so that individuals or teams can better express and discuss their own ideas; By providing activities that frame and focus discussion, and collaborative deign decision-making and; By creating an opportunity for stakeholders to come together and share knowledge and experience, and co-create new, informed and innovative approaches to learning and teaching. To support the learning design workshops, the OU has introduced a toolkit of Learning Design resources and activities which can be used to help stimulate discussion and design thinking around some common design challenges. The OU s online Learning Design tools are designed to support teams engage in collaborative design activity, make their own design more visible, and more easily and openly search for and borrow from examples of good practice. The tools provide: Interactive visual representations of modules and qualifications Functionality that supports collaboration, sharing and reuse of designs; and Mechanisms for flagging, browsing and describing good practice. OU administrative Learning Design data is: Systematically collected across all modules; However, it has only recently been a formal requirement for this information to be completed, so it is sometimes inaccurate and incomplete.
7 Current Practice Annual Quality Review Learning Design Reviews Survey Results Module Changes
8 Student Experience on a Module (SEaM) 2015 Analysis of Overall Satisfaction 1 Background to the Project 2 Analysis to Date 3 Students Satisfaction Modelling 4 Next steps
9 Undergraduate Students Factor Analysis: Key Findings (1)
10 Postgraduate Students Factor Analysis: Key Findings (2)
11 Key Driver Analysis Findings (1) Institutional reporting of SEaM survey results adoption of reporting of the key factors is considered as an alternative to the KPI s with UG/PG results separated. If KPI s remain then KPI-01 (satisfaction with quality) is reconsidered as the main indicator of student satisfaction - KPI-08 is suggested as an alternative. Question 36 (satisfaction with assessment) and Q13 (module contribution to qualification aim) are considered as additional KPIs.
12 Key Driver Analysis Findings (2) The question wording for KPI-07 (clear learning outcomes), Q11 (completing assignments) and Q7 (the library s online resources) is considered as these questions do not clearly group within any of the defined factors. Cognitive interviewing could be used to validate the questions. The quantity of questions is re-visited there is a clear method here for reducing the number of questions that are measuring the same constructs, particularly the tutor support factor.
13 Methodology (Logistic Regression) & Validation Module Presentation Step 1: A descriptive analysis was conducted to discount variables that were unsuitable for satisfaction modelling. Step 2: Each subset of variables was modelled in groups. The variables that were statistically significant from each subset were then combined and modelled to identify the final list of key drivers Step 3 Validation: all models have been verified by using subsets of the whole data to ensure the solutions are robust. A variety of model fit statistics have also been used to identify the optimum solutions. Student Concurrency Overall Satisfaction Study history SEaM Step 1 also identified highly correlated predictors and methodically selected the most appropriate. UG new, UG continuing, PG new and PG continuing students were modelled separately at Step 2. We found at Step 3 that the combined scale provided the simplest and most interpretable solution for PG students and the whole scale for UG students. The solution without the KPI s included was much easier to use in terms of identifying clear priorities for action.
14 Undergraduate Continuing Students % planned life cycle Module: Examinable Component Module: Level of study Q6 Method of delivery 11 Q11 Assignment completion 09 Q23 Tutor knowledge 07 Q3 Advice & guidance Importance to Overall Satisfaction Q13 Qualificatio n aim KPI-05 Teaching materials Module: Credits KPI-06 Workload Q9 Assignment instructions Q14 Career relevance Q5 Integration of materials Q36 Assessment
15 Undergraduate New Students Q14 Career relevance Q3 Advice & guidance KPI-05 Teaching materials 07 Age 05 Importance to Overall Satisfaction Q13 Qualification aim Q5 Integration of materials Q36 Assessment
16 UG Continuing VS New Students Both the models are dominated by the survey questions, confirming that satisfaction with learning design is a better driver for overall satisfaction. The UG continuing model is the only one where level is a significant driver. The drivers of new UG student are very similar to continuing students with the exception of age group. Satisfaction with teaching materials, assessment and advice and guidance are absolutely key for UG students when predicting overall satisfaction.
17 Postgraduate Continuing Students Credit transfer Module: Result 07 Q6 Method of delivery 05 Q36 Assessment 03 KPI-05 Teaching materials Importance to Overall Satisfaction Study motivation Number if tutor groups Q5 Integration of materials Q30 Assessment feedback
18 Postgraduate New Students Q16 Tutor contact at start 05 Q36 Assignment 03 Q11 Assignment completion 01 Importance to Overall Satisfaction Q22 Tutor help with online Q12 Collaborative activities Q6 Method of delivery
19 PG Continuing VS New Students Satisfaction with particular aspects of the student experience remain key for PG students, PG continuing students who are motivated by personal development are most likely to be satisfied with their overall experience. The PG continuing model is much more similar to the UG models than the PG new. This is not surprising as there are a smaller number of students available to model for PG new, therefore, this is the least stable of the four models and would benefit from the addition of another cohort of students to validate it robustly.
20 Key Findings Satisfaction with assessment, teaching materials, integration of materials or method of delivery consistently appear as the most important key drivers of overall student satisfaction. These findings are irrespective of student type e.g. new or continuing and module/ presentation information such as level of study (UG/PG), faculty and subject area. Previous study experience with the OU and socio-demographics such as gender, ethnicity, socio-economic status and disability are also not predictive of overall satisfaction once satisfaction with the learning design of the module is accounted for. The relationship between UG study level and satisfaction is not clear. This needs to be investigated further in order to establish clear recommendations. Ideally with an additional student cohort of data this analysis was based on respondents to SEaM in the 2013/14 academic year.
21 Aspirations Module Learning Design review Refinement of Learning Design benchmarks Monthly review of module activity Macro analysis of activity, feedback and outcomes In presentation adjustments to module where possible Annual review of student outcomes Annual review of student feedback
22 Aspirations How is this information being used? Learning Design benchmarks are being developed Module teams are using the key drivers to identify areas for action Aim is that this information will inform the next PVC priorities & Annual Quality Review cycle
23 Student Experience on a Module (SEaM) 2015 Analysis of Overall Satisfaction Background to the Project Analysis to date Students Satisfaction Modelling Next steps Add the SEaM survey questions to the pass rate model to verify the key drivers of pass compared to satisfaction. Further investigate the linking of systematic learning design data with survey & outcome data. Replicate the analysis for satisfaction, pass and retention with the new cohort of SEaM data for 2014/15 in order to validate these findings. Disseminate the findings more widely.
Standards and Guidelines for Quality Assurance in the European Higher Education Area (ESG) Endorsed by the Bologna Follow-Up Group in September 2014 Subject to approval by the Ministerial Conference in
Standards and Guidelines for Quality Assurance in the European Higher Education Area Standards and Guidelines for Quality Assurance in the European Higher Education Area 2 DG Education and Culture This
Support Materials Contents This resource guide has been developed to support the work of school leaders, teachers and educators who work in and with schools across Australia. guide is designed to enable
Guidance document on programme development and implementation for the Ten Year Framework of Programmes on Sustainable Consumption and Production (10YFP): Criteria, structure and steps to develop and operationalize
Evaluation Guidance Note Series UNIFEM Evaluation Unit October 2009 Guidance Note on Developing Terms of Reference (ToR) for Evaluations Terms of Reference (ToR) What? Why? And How? These guidelines aim
Primary Years Programme Guide to school authorization: Primary Years Programme Primary Years Programme Guide to school authorization: Primary Years Programme Published October 2010 International Baccalaureate
4 Guide to Developing a Quality Improvement Plan September 2013 Copyright The details of the relevant licence conditions are available on the Creative Commons website (accessible using the links provided)
Final report Research into the impact of Project Maths on student achievement, learning and motivation Jennifer Jeffes Emily Jones Máirín Wilson Emily Lamont Suzanne Straw Rebecca Wheater Anneka Dawson
JANUARY 2007 looked after children & young people: Working together to build improvement in the educational outcomes of Scotland s looked after children & young people. looked after children & young people:
The New Zealand Qualifications Framework NEW ZEALAND QUALIFICATIONS AUTHORITY Contents Section 1. The New Zealand Qualifications Framework 2 Purpose of the NZQF 2 Principles underpinning New Zealand qualifications
MISSION AND GOALS 34 3 DEVELOP A MISSION, AND GOALS When organizations or individuals decide to join forces and work as a partnership, they do so with a purpose in mind. While the individual partners may
HOW GOOD ARE WE NOW? HOW GOOD IS OUR SCHOOL? HOW GOOD CAN WE BE? THE JOURNEY TO EXCELLENCE: PART 3 HOW GOOD ARE WE NOW? HOW GOOD IS OUR SCHOOL? HOW GOOD CAN WE BE? THE JOURNEY TO EXCELLENCE: PART 3 Please
Learning Together: Improving teaching, improving learning The roles of continuing professional development, collegiality and chartered teachers in implementing Curriculum for Excellence Learning Together:
What do researchers do? Doctoral graduate destinations and impact three years on 2010 First ever analysis of employment destinations and impact of doctoral graduates three years after graduation Benefits
Climate Surveys: Useful Tools to Help Colleges and Universities in Their Efforts to Reduce and Prevent Sexual Assault Why are we releasing information about climate surveys? Sexual assault is a significant
Research Report DCSF-RR011 Effective Teaching and Learning for Pupils in Low Attaining Groups Máiréad Dunne, Sara Humphreys and Judy Sebba University of Sussex Alan Dyson, Frances Gallannaugh and Daniel
Diploma Programme The Diploma Programme From principles into practice Diploma Programme The Diploma Programme From principles into practice Diploma Programme The Diploma Programme: From principles into
Securing Their Future The New South Wales Government s reforms for the Higher School Certificate Foreword It is three and a half decades since the decision was taken to restructure New South Wales secondary
THE CHILD AT THE CENTRE SELF-EVALUATION IN THE EARLY YEARS THE CHILD AT THE CENTRE SELF-EVALUATION IN THE EARLY YEARS Crown copyright 2007 ISBN: 978-0-7053-1109-0 HM Inspectorate of Education Denholm House
GUIDE TO EVALUATING INSTITUTIONS A Publication of the Accrediting Commission for Community and Junior Colleges Western Association of Schools and Colleges JULY 2013 Edition ACCJC/WASC 10 Commercial Blvd.
Research Report DCSF-RR021 Pilot Scheme for Two Year Old Children Evaluation of Outreach Approaches Anne Kazimirski, Sarah Dickens and Clarissa White National Centre for Social Research Research Report
A review of international evidence on interagency working, to inform the development of Children s Services Committees in Ireland AUGUST 2011 DEPARTMENT OF CHILDREN AND YOUTH AFFAIRS The author of this
Rapport 18 Promoting and assessing value creation in communities and networks: a conceptual framework Etienne Wenger Beverly Trayner Maarten de Laat Ruud de Moor Centrum Ruud de Moor Centrum Open Universiteit
Students experiences of full-time Foundation degrees A Report to fdf Holly Higgins, Jane Artess and Isla Johnstone 1 Students experiences of full-time Foundation degrees Contents Foreword 3 Executive Summary
Time for Training A Review of the impact of the European Working Time Directive on the quality of training Professor Sir John Temple May 2010 Foreword Learning from experience; it takes too long. It often
Evaluating Your Community-Based Program Part II Putting Your Evaluation Plan to Work A publication of the American Academy of Pediatrics A Cooperative Agreement Program of the Federal Maternal and Child
1101 Pennsylvania Ave. NW, Suite 600 Washington, DC 20004 P 202.756.2971 F 866.808.6585 www.hanoverresearch.com Best Practices in Online Teaching Strategies In the following report, The Hanover Research
Creating Pathways to Success AN EDUCATION AND CAREER/LIFE PLANNING PROGRAM FOR ONTARIO SCHOOLS P o l i c y and P r o g r a m R e q u i r e m e n t s, K i n d e r g a rt e n t o Grade 12 2 0 1 3 The Ontario
E U A P U B L I C A T I O N S 2 0 1 0 Examining Quality Culture: Part 1 Quality Assurance Processes in Higher Education Institutions By Tia Loukkola and Thérèse Zhang Copyright by the European University