Using survey data to inform and target curriculum improvement Key Contacts: Vicky Marsh / Rebecca Galley Vicky.Marsh@open.ac.uk Rebecca.Galley@open.ac.uk
Agenda 1 Background to the Project 2 Analysis to Date 3 Satisfaction modelling 4 Next Steps
Background to the Project SEaM Student Experience on a Module (SEaM) survey: methodology, exclusions & reliability Annual Quality Review OU Annual Quality Review (AQR) process: Summary & integration of student outcomes Learning Design Learning Design at the OU: Background & practice Cycle joining it all up Current practice at the OU in terms of linking learning design to student outcomes and quality review
SEaM Overview Revised Survey in place for modules that started from 2012J+ Every student on every module will be surveyed Integrated survey including the student feedback on their tutor and on their module Three themed sets of questions which cover The module overall Teaching, learning and assessment Feedback on the tutor (Tutor supported modules only) Students are sent an invitation to participate two to three weeks before the end of the module Coverage includes presentations that have ended between 1st August 2013 and 31st July 2014, who had results available by 13th August 2014 There are a selection of short course, residential school and overseas modules included in AQR that were not surveyed, these modules are not included. All students regardless of their completion status are included - all previous SEaM results were based on students who completed only. Reliability Consideration should be given to response rates (34%) and the number of responding students when reviewing SEaM results. Non-response bias there is some evidence of bias, but the impact on survey estimates is minimal 4
OU Annual Quality Review process Annual Data Release Programme Committees PVC Prog Review Interim Data Release Internal Prog Review End of modules Annual Data Available September October November December January February March April May June July August Student Intake Student Intake Monthly & daily qualification, module and student data available Annual/interim data release includes: Module pass rates Module SEaM survey results Five year module SEaM survey and pass rate trends Qualification NSS, PTES, PRES, DLHE survey results Qualification progress rates Qualification classification and award data Module VLE & Learning Design information Monthly/Daily real time data includes: Module SEaM survey results Module profile e.g. socio-demographics, assignment submission Module flows e.g. concurrent study & next modules studied Qualification profile e.g. which modules are students studying Module activity charts VLE activity, registrations & assignment submission
OU Learning Design The term learning Design describes a field of educational research and development that seeks to enhance the processes involved in designing, describing and sharing learning activities, to support educators design practices. Learning design at the OU is characterised by 3 principles: Mechanisms to encourage design conversations across disciplines and expert roles The use of tools and instruments as a means of describing and sharing design The use of information and data to inform the conceptual tools and frameworks that guide the decision-making process OU Learning Design workshops are seen to support the creative design process: By providing a common language and set of representations so that individuals or teams can better express and discuss their own ideas; By providing activities that frame and focus discussion, and collaborative deign decision-making and; By creating an opportunity for stakeholders to come together and share knowledge and experience, and co-create new, informed and innovative approaches to learning and teaching. To support the learning design workshops, the OU has introduced a toolkit of Learning Design resources and activities which can be used to help stimulate discussion and design thinking around some common design challenges. The OU s online Learning Design tools are designed to support teams engage in collaborative design activity, make their own design more visible, and more easily and openly search for and borrow from examples of good practice. The tools provide: Interactive visual representations of modules and qualifications Functionality that supports collaboration, sharing and reuse of designs; and Mechanisms for flagging, browsing and describing good practice. OU administrative Learning Design data is: Systematically collected across all modules; However, it has only recently been a formal requirement for this information to be completed, so it is sometimes inaccurate and incomplete.
Current Practice Annual Quality Review Learning Design Reviews Survey Results Module Changes
Student Experience on a Module (SEaM) 2015 Analysis of Overall Satisfaction 1 Background to the Project 2 Analysis to Date 3 Students Satisfaction Modelling 4 Next steps
Undergraduate Students Factor Analysis: Key Findings (1)
Postgraduate Students Factor Analysis: Key Findings (2)
Key Driver Analysis Findings (1) Institutional reporting of SEaM survey results adoption of reporting of the key factors is considered as an alternative to the KPI s with UG/PG results separated. If KPI s remain then KPI-01 (satisfaction with quality) is reconsidered as the main indicator of student satisfaction - KPI-08 is suggested as an alternative. Question 36 (satisfaction with assessment) and Q13 (module contribution to qualification aim) are considered as additional KPIs.
Key Driver Analysis Findings (2) The question wording for KPI-07 (clear learning outcomes), Q11 (completing assignments) and Q7 (the library s online resources) is considered as these questions do not clearly group within any of the defined factors. Cognitive interviewing could be used to validate the questions. The quantity of questions is re-visited there is a clear method here for reducing the number of questions that are measuring the same constructs, particularly the tutor support factor.
Methodology (Logistic Regression) & Validation Module Presentation Step 1: A descriptive analysis was conducted to discount variables that were unsuitable for satisfaction modelling. Step 2: Each subset of variables was modelled in groups. The variables that were statistically significant from each subset were then combined and modelled to identify the final list of key drivers Step 3 Validation: all models have been verified by using subsets of the whole data to ensure the solutions are robust. A variety of model fit statistics have also been used to identify the optimum solutions. Student Concurrency Overall Satisfaction Study history SEaM Step 1 also identified highly correlated predictors and methodically selected the most appropriate. UG new, UG continuing, PG new and PG continuing students were modelled separately at Step 2. We found at Step 3 that the combined scale provided the simplest and most interpretable solution for PG students and the whole scale for UG students. The solution without the KPI s included was much easier to use in terms of identifying clear priorities for action.
Undergraduate Continuing Students % planned life cycle 15 14 Module: Examinable Component Module: Level of study 13 12 Q6 Method of delivery 11 Q11 Assignment completion 09 Q23 Tutor knowledge 07 Q3 Advice & guidance Importance to Overall Satisfaction 10 08 06 05 Q13 Qualificatio n aim 04 03 KPI-05 Teaching materials 01 02 Module: Credits KPI-06 Workload Q9 Assignment instructions Q14 Career relevance Q5 Integration of materials Q36 Assessment
Undergraduate New Students Q14 Career relevance Q3 Advice & guidance KPI-05 Teaching materials 07 Age 05 Importance to Overall Satisfaction 03 01 02 06 04 Q13 Qualification aim Q5 Integration of materials Q36 Assessment
UG Continuing VS New Students Both the models are dominated by the survey questions, confirming that satisfaction with learning design is a better driver for overall satisfaction. The UG continuing model is the only one where level is a significant driver. The drivers of new UG student are very similar to continuing students with the exception of age group. Satisfaction with teaching materials, assessment and advice and guidance are absolutely key for UG students when predicting overall satisfaction.
Postgraduate Continuing Students Credit transfer Module: Result 07 Q6 Method of delivery 05 Q36 Assessment 03 KPI-05 Teaching materials 01 09 Importance to Overall Satisfaction 08 02 Study motivation 06 04 Number if tutor groups Q5 Integration of materials Q30 Assessment feedback
Postgraduate New Students Q16 Tutor contact at start 05 Q36 Assignment 03 Q11 Assignment completion 01 Importance to Overall Satisfaction 02 06 04 Q22 Tutor help with online Q12 Collaborative activities Q6 Method of delivery
PG Continuing VS New Students Satisfaction with particular aspects of the student experience remain key for PG students, PG continuing students who are motivated by personal development are most likely to be satisfied with their overall experience. The PG continuing model is much more similar to the UG models than the PG new. This is not surprising as there are a smaller number of students available to model for PG new, therefore, this is the least stable of the four models and would benefit from the addition of another cohort of students to validate it robustly.
Key Findings Satisfaction with assessment, teaching materials, integration of materials or method of delivery consistently appear as the most important key drivers of overall student satisfaction. These findings are irrespective of student type e.g. new or continuing and module/ presentation information such as level of study (UG/PG), faculty and subject area. Previous study experience with the OU and socio-demographics such as gender, ethnicity, socio-economic status and disability are also not predictive of overall satisfaction once satisfaction with the learning design of the module is accounted for. The relationship between UG study level and satisfaction is not clear. This needs to be investigated further in order to establish clear recommendations. Ideally with an additional student cohort of data this analysis was based on respondents to SEaM in the 2013/14 academic year.
Aspirations Module Learning Design review Refinement of Learning Design benchmarks Monthly review of module activity Macro analysis of activity, feedback and outcomes In presentation adjustments to module where possible Annual review of student outcomes Annual review of student feedback
Aspirations How is this information being used? Learning Design benchmarks are being developed Module teams are using the key drivers to identify areas for action Aim is that this information will inform the next PVC priorities & Annual Quality Review cycle
Student Experience on a Module (SEaM) 2015 Analysis of Overall Satisfaction 1 2 3 4 Background to the Project Analysis to date Students Satisfaction Modelling Next steps Add the SEaM survey questions to the pass rate model to verify the key drivers of pass compared to satisfaction. Further investigate the linking of systematic learning design data with survey & outcome data. Replicate the analysis for satisfaction, pass and retention with the new cohort of SEaM data for 2014/15 in order to validate these findings. Disseminate the findings more widely.
Institute of Educational Technology