Phase 1 pilot 2005/6 Intervention. GCU Caledonian Business School Business Management Page 1 of 8. Overview

Transcription

1 University Department Module Overview Glasgow Caledonian Business school Business Management The Business School has been implementing assessment re-engineering using a variety of technologies, initially within the core modules in six of the eight disciplinary divisions of the Business School, but sharing experiences across the whole Business School over the duration of the project. This will establish models for transformational change across a single faculty within a new university (post-1992). The Caledonian Business School also has a special interest in developing the use of an interactive business simulation or game for students studying core modules. The purpose is to develop students problem-solving and decision-making skills in authentic contexts and to provide enhanced feedback. Level 1 Core Module: Managerial Finance: The Managerial Finance module is a foundation module for non-accountancy students (c. 800) who have an accounting component in their programme. Level 1 Core Module: Marketing Fundamentals: Marketing Fundamentals is a level 1 module with c900 students over 2 semesters. Forty percent of the summative assessment is in the form of a multiple choice question (MCQ) exam. Level 1 Core Module: Economics, Markets and Enterprise: Economics, Markets and Enterprise is a level 1 core module in the Caledonian Business School (CBS) and is taken by 500 students from a wide range of programmes. Phase 1 pilot 2005/6 Intervention Business Management is a level 3 business management module for students undertaking the Optometric Dispensing Diploma BSC. It is a small class with the traditional assessment structured split between one coursework assignment activity (50% of grade weighting) and end of term exam (50%).: Managerial Finance: 10 on-line weekly assessments which had been used previously for formative assessment purposes were converted to low stakes assessment (quizzes). The move from formative and summative assessment was undertaken due to the disappointing participation rates which occurred in previous sessions. The introduction of this initiative led to full engagement by students given that their efforts within the assessment now contribute to their overall aggregate grade. Marketing Fundamentals: Previous optical scanning of paper submissions had been found to be inaccurate, and manual marking was time consuming. An MCQ exam was implemented online through the Blackboard virtual learning environment in January 2006 and again in May On evaluation, students responded very positively to taking the exam online and particularly liked receiving their marks immediately. Economics, Markets and Enterprise: In order to enliven and contextualise the topic, online quizzes, web-based resources and podcasts of lectures had already been developed by the tutor. These additional resources were not fully used by all students however, and some still had difficulty with basic economic concepts. One way of addressing this issue was to employ the use of electronic voting systems in large lectures. Perceived benefits for students included increased levels of formative feedback and a deeper understanding of key economic concepts through more engagement. GCU Caledonian Business School Business Management Page 1 of 8

2 Drivers for change Phase 2 pilot 2006/7 Intervention Glasgow Caledonian university business school undergraduate provision operates within a framework of programmes and core modules taken by all the students (7 or 800) In some cases, this is split over 2 semesters, in others all of the students take the modume in either semester A or B. The lecture theatres can accommodate 500 students only theatre meaning that lectures have to be repeated. Despite the phase 1 interventions, staff noted that the because of the large numbers of students in each lecture the mode of delivery has been somewhat transmissive meaning that the lectures themselves have been problematic. In practice, it was extremely difficult to be interactive with up to 500 students in lectures. Allied to that, there were attendance issues, which had been a problem for a number of years. All of the previous interventions had happened in line with the VLE (Blackboard) being adopted across the university and particularly with the business school because they were the first to adopt it. Since much of the teaching material had been posted on to the VLE there was a perception that students were not attending because they could access the learning materials on-line i.e. either lecture web notes or PowerPoint slides and viewed these as being a substitute for attending the lecture. There was little opportunity for formative feedback in the traditional format and the main problems have been low levels of student engagement, low grades and heavy marking schedules During the session, electronic voting systems (Interwrite PRS) have been introduced into lectures in order to engage students in more active learning in order to focus their attention and deepen their learning experience with the added benefit of being able to monitor attendance. Radio frequency personal voting system handsets were used in all of the modules with the primary aim of providing increased opportunity for self-reflection and peer feedback. The aim of intervention here was to increase student engagement and confidence through promoting greater independence and collaboration on assignments and ultimately to embed a system that will lead to efficiencies in terms of assessment marking time. The focus is on gaining process measures though self and peer-assessment and social construction. Methods included the introduction of an electronic voting system in lectures and as an integral part of group presentations. The assessment structure was altered through the introduction of self/peer assessed PowerPoint presentations. This shifted 20% of the assessment weighting towards the presentation from the course assignment, reducing it to 30%, while the end of term exam remained at 50%. EMPOWERMENT NICOL S 7 PRINCIPLES OF GOOD ASSESSMENT DESIGN Principle 1 (clarify criteria) Activities 1) By designing their own presentations, students had an opportunity to set their own criteria and standards. 2) Students were given a small group task in wk 3 or 4 of the course and were instructed about it at the start of the course. Criteria for presentations was primarily derived from formative feedback from a practice presentation. During this practice session, there was no use of EVS or PowerPoint, however it helped to set the parameters of the assessed presentations. 3) Students were encouraged to critique their peers performances, with criteria provided through verbal tutor prompts at the end of the session, although formal criteria sheets were not provided in order to encourage the students to have to think on their own about what objectives and standards they believed to be important as performance indicators. 4) The generic tutor feedback available in the Blackboard VLE system sets a standard against which students could assess their performance 5) Students could view the class response to EVS questions during the lecture slot or at any time using the Blackboard system, which the EVS responses and graphs are posted on. This enabled them to draw direct comparisons on their own performance against that of their peers, providing a framework for good performance. GCU Caledonian Business School Business Management Page 2 of 8

3 Principle 2 (self-assess, reflect) 1) Increased self-reflection through access to class responses and through discussion. Students had increased opportunities to reflect on their learning through EVS prompted discussion and from being able to access other students responses, which they could compare to their own. 2) Increased opportunity to grade own work through EVS link to Blackboard. Since the EVS responses are sent to the VLE Blackboard system, students could pose themselves the questions from the lectures, attempt to answer them and then compare their answers with the correct responses as well as with the class response. This provided them with an opportunity to be able to grade their own responses in their own time. 3) Increased opportunity to generate explanations and increase self-assessment through debate. Since many of the course leaders have chosen to use the EVS to facilitate debate or discussion, students have an opportunity to generate and defend their own explanations, which enables them to self-assess their responses and knowledge compared to both peer responses and tutor guidance. 1) Students were able to gain autonomy through designing their own group PowerPoint presentations and EVS questions to support it. 2) Students were encouraged to incorporate their own personal examples into the presentations and were encouraged to think as independent business students rather than simply as optometric dispensing students 3) Students engaged in self-appraisal after the presentations Principle 3 (tutor feedback) Principle 4 (peer feedback) 1) Blackboard has generally facilitated the delivery of generic feedback on assignments while EVS is useful for frequent, immediate formative feedback on lecture content. 2) The Blackboard system is available for generic feedback from throughout and outside of the campus, while personal feedback can be accessed through tutors by within the extensive computing facilities or from a computer off campus. 1) Students received more positive, constructive written feedback on coursework than in the traditional format. 2) Students were able to access and act on generic formative feedback posted on the Blackboard VLE system after assessments 3) Students worked together collaboratively to pool information, thus increasing the quality of knowledge obtained 1) Increased peer discussion, peer assessment & formative feedback through electronic handset voting. Ambiguous style EVS MCQs were particularly useful for promoting peer discussion and debate among students. The anonymity provided by the EVS appeared to engender confidence in the students to offer their opinion and engage more in peer dialogue. The EVS system has been used by most of the course leaders to promote increased peer discussion and formative feedback. 2) Increased formative peer feedback on specific answers. Managerial Finance students experienced multiple choice questions used to encourage students to discuss incorrect answers during the lecture, thus providing students with an opportunity to give and receive peer formative feedback on their specific answers. 1) Students engaged in peer discussion in their presentation study groups and were encouraged to form independent study groups 2) Students engaged in peer and tutor dialogue during the critique opportunities after the presentation assignments 3) Students took part in discussion during lectures prompted by the EVS questions. GCU Caledonian Business School Business Management Page 3 of 8

4 Principle 5 (motivation) Principle 6 (close feedback loop) 1) Students were motivated to learn by having more control in the learning design process of the group presentations 2) In the course of viewing their peers giving presentations, students in the audience role had the opportunity to vote with the electronic handsets at the beginning of the session on their self-assessed level of confidence in their knowledge of the presentation content and then again at the end of the presentation. This enabled them to see over a short period of time, a concrete increase in their knowledge and confidence, which could boost their selfesteem. 1) Marketing students were asked a question at the beginning of the lecture that was repeated near the end of the lecture, so that again they would be prompted to attend to and engage with the lecture content. Following the initial response, the student results were graphically displayed so that they could compare their own responses with that of their peers. Peer discussion was then generated from the responses. After the question had been resubmitted, the student responses were again displayed so that the change in class responses could be compared. For these students, EVS was also used as a revision tool at the end of each lecture and each lecture block, using both MCQs and true/false formats and the questions were related to website pointers for further study aids. Tying video and lecture material to EVS questions, encouraging peer discussion. Revision questions at end of each lecture and block can be useful if structured well for the right kind of subject. Managerial Finance students were presented with past paper exam style EVS questions, which served both as a wake-up call and to provide them with the opportunity of some exam technique practice. There was some inter-module staff consensus that ambiguous MCQ style EVS questions were particularly useful in promoting peer discussion and debate where self-reflection and assessment are required. 1) By designing their own assignment and setting their own EVS questions, students had an opportunity to close the gap between learning and performance by learning to think more like a teacher than simply from the view of a learner. 2) Students could use the practice opportunities available before the presentation sessions and MCQ practice sessions to help to enhance congruence between learning and performance. 3) Since the group presentations topics were congruent with 3 of the exam question topics, the learning outcomes of students having engaged with this task provided a significant opportunity for them to link feedback into subsequent assessment. GCU Caledonian Business School Business Management Page 4 of 8

5 Principle 7 (shape teaching) 1) Increased opportunity for staff to receive feedback in order to shape lecture content. Staff, particularly from the managerial finance module, highlighted that EVS was useful as diagnostic tool in order to shape lecture content and to gain some insight to the students comprehension in general. While not quite at a Just in Time lecture level, can go over information about students misconceptions and cover it in next lecture. Questions were included as a consequence of lecture delivery and thus were material driven rather than technology driven 2) Staff recommended that the lecture content be manipulated around the EVS questions rather than them being tagged on to existing material. This way, tutors can use them when they required or are most productive for learning without having to compromise their lecture material. 3) The EVS was considered by staff to be very useful in some cases as a diagnostic tool, providing student feedback to staff in order to facilitate staff action in shaping the lecture content in a more educationally targeted approach. 4) EVS responses can be monitored in some modules/classes but it cannot be used diagnostically in all instances since it is often best used with ambiguous questions, thus cannot provide information correct or incorrect data. However, if tit is used to promote discussion, this in itself can inform the lecturer of the level of content understanding. 5) Although tutor to tutor feedback at the moment relies on direct communication, there has been significant interest among staff in developing an feedback template that can be distributed throughout staff on the module or between modules to maximise continuity and efficiency of targeted content delivery. ENGAGEMENT GIBBS & SIMPSON S 4 CONDITIONS OF TIME & EFFORT ON TASK Condition 1 (in and out of class) Condition 2 (spread evenly) 1) Students provided feedback to staff through EVS responses and class discussion, which has shaped the lecture content focus more towards application of theory than theory alone. 1) Increased access to generic feedback campus wide & beyond. Students had the opportunity to communicate with each other throughout the campus and from outside using the Blackboard VLE system. The specific redesign issues this year have added an extra dimension to this facility by using it to deliver class EVS responses, thereby providing tutor and peer feedback that is available from anywhere in or out of the campus. 1) Although there were only two assessed tasks as well as the final exam, students were enabled to have sufficient opportunity to achieve a regular study routine because of the spread of the presentation task 2) Students also had two formal opportunities over the module duration for self and peer critique on assessment in the course of presentations and practice sessions. 1) Students were required from week 4 of the course until their presentations near the end of the semester to engage in the task by adopting a specific role within the team. They had to fill in weekly activity sheets that recorded their efforts and contributions to the collaborative task for that week, thus ensuring that each student allocated the appropriate amount of time and effort to the most important aspects of the task. GCU Caledonian Business School Business Management Page 5 of 8

6 OUTCOMES Condition 3 (deep not surface) Condition 4 (high expectations Efficiencies 1) Deep learning was promoted through reinforcement of concepts by presentation of EVS questions, cues and testing. 2) Course leaders have adopted several different formats for EVS use. The most effective ones include examples from students in the Economics, Marketing and Enterprise module for whom a theme was identified at the beginning of the lecture. They were then asked some MCQ style questions later on so that during the lecture, the students had to look for cues. This helped to keep their attention and promote a deeper engagement with the lecture material. They were also asked several iterations of one EVS question, which helped to facilitate a deeper understanding on the students behalf through the requirement of increased reflection of the content. Some lectures, were linked to multimedia resources, such as videos in order to quickly illustrate concepts. Cues were then presented during the lecture, again with EVS questions at the end. Contrasting the video scenario with the lecture concepts helped to promote a deeper approach to student reflection about the content. 1) There was an improvement in the quality of learning by students being abler to bring key aspects alive by making the information relevant to real world experiences. This encouraged higher level thinking and application to theory. 1) Students could effectively communicate to each other clear and high expectations of the roles and standards that each was required to have in the team process for the presentation assignments by working closely together and formulating their own criteria. 2) Students had an opportunity to gain knowledge of tutor expectations during the lecture discussions prompted by EVS responses 3) The demands on the students to design their own EVS questions and work collaboratively conveyed the high expectations of the course leader on their efforts. 1) The main efficiency has been in terms of the student learning gains, which may potentially increase retention and progression rates. 1) Although there have been no specific savings made in this semester with regard to staff time or costs, the long-term aim is that once the pedagogical principles have been successfully embedded in the teaching practice, the EVS resources could be used for assessed exercises in order to reduce marking time. GCU Caledonian Business School Business Management Page 6 of 8

7 Informal Learning Gains Anecdotal evidence from staff focus group. 1) There was a feeling among staff that students appeared to significantly benefit from being able to see their peer s responses and receive immediate feedback. 2) Increased tutor engagement. Students appeared to engage better with the tutor through increased opportunities for open discussion 3) Increased peer engagement. Students appeared to engage better with the peers through increased open discussion arising from EVS use 4) Increased lecture material engagement. Students appeared to engage better with the material through increased opportunities for reflection, discussion and peer response comparison 5) Increased enjoyment of lectures. Outcomes of discussion with seminar students suggested that they did generally enjoy the experience although there was a feeling that this may have been particularly the case for the less able students, while the more successful students seemed to become a little more agitated by technical problems and any loss of lecture material as a result. 6) Increased understanding of core concepts. There was a perception among the staff that the use of EVS appears to has deepened understanding of concepts through increased opportunity for discussion and reflection and in some cases where for example several iterations of a single question have been posed, the students are taken beyond a superficial analysis by having to think about alternative perspectives, and layers to a problem. Quantitative evidence from staff & student questionnaires revealed that using the EVS in lectures 1) Increased student understanding of course content - staff (66%) students (65%) 2) Increased student concentration in lectures staff (66%) students (76%) 3) Increased student interest in lecture content staff (67%) students (60%) 4) Increased the opportunity for staff to gain feedback from students about their understanding staff (100%) students (90%) 5) Increased student engagement with staff staff (100%) students (79%) 6) Increased student engagement with peers staff (83%) students (56%) 7) Had been beneficial to students learning staff (100%) students (84%) Qualitative post measures from student interviews revealed that students felt that the introduction of the PowerPoint presentations and EVS had 1) Increased course enjoyment 2) Increased engagement with staff, peers and learning material 3) Increased motivation 4) Increased confidence 5) Resulted in a deeper learning approach Formal Learning Gains Quantitative evidence from student class questionnaires (11 students) revealed that using EVS in lectures 1) Increased opportunities for staff to gain feedback of student understanding (100%) 2) Increased students concentration and interest in lectures (73%) 3) Increased student engagement with staff (100%) 4) Increased confidence (73%) 5) Increased enjoyment (100%) 6) Increased learning (81%) GCU Caledonian Business School Business Management Page 7 of 8

8 GCU Caledonian Business School Business Management Page 8 of 8