Online Quizzes for Enhancing Student Learning in a First Year Engineering Course

Transcription

1 Online Quizzes for Enhancing Student Learning in a First Year Engineering Course Author: S M Aziz, University of South Australia, Mawson Lakes, SA 5095, Australia, mahfuz.aziz@unisa.edu.au Abstract In the context of increasing student disengagement with on campus teaching and learning activities, the author undertook a project in order to stimulate and maintain student interest in a first year engineering course. Online technology has been used in order to provide students with flexible access to self paced interactive study materials in the form of online quizzes. These quizzes were also aimed at providing students with an alternative learning style and a mechanism to self assess their progress. This paper examines the impacts of these online quizzes on student engagement and learning, and seeks to obtain student feedback on the suitability of this medium for enhancing students overall learning experience. Index Terms Flexible delivery, online quiz, self-assessment, student disengagement. INTRODUCTION There are evidences of increasing student disengagement in on campus teaching and learning activities in recent times [1]. While there are many reasons for such disengagement, the most commonly known reason is that an inreasing number of students are now working longer hours than ever before [2]. Other reasons include the perceived difficulty of addressing the diverse learning needs of the student body using traditional lecture/tutorial format especially in large classes. The diversity in the learning needs arise from different learning styles of individual students [3]. Whatever the reason there is no doubt that continuing student disengagemnt and absence have a profound impact on students learning outcomes and progression torugh the program of study [1]. The problem is more compounded for first year undergraduate students many of whom have difficulty transitioning from a high school system of education to a university environment where they have more freedom but are expected to take more responsibility for their own learning. While teaching a large first year engineering class at the University of South Australia I experienced the problem of increasing student absence/disengagement as the semester progressed. Student attendance in the lecture classes often declined to almost half approximately half way through the semester. On the other hand many students who were regularly attedning classes were found not to be able to keep up with the progression of the course despite various active learning strategies [4] implented in the class. A large proportion of these students were found reluctant to spend time at home studying printed material alone for enhancing their understanding of the material covered in the class. This presented a critical teaching problem. This being one of the foundation courses in engineering offered in the very first semester for students entering the university, it was seen as a course to stimulate students motivation to complete a four year engineering program of study. I felt that augmenting the on campus teaching and learning activities with additional (alternative) activities would be useful for stimulating students interest, and for enhancing their learning experience and outcomes in the very first year at university. In the context of declining governent funding in the higher education sector [5]-[7] alternative learning activities not requiring face-to-face interaction could also help reduce the economic hardship on universities. All of the above represented a critical opportunity to engage first year students with online technologies as the tools of the present and to engage them in a stimulating learning environment. Students could benefit from the flexibility of online learning as more students are now working longer hours than ever before and having difficulty attending campus activities due to work commitments. While others could benefit from self paced interactive online learning activities to supplement and reinforce the laerning that occurs in the classroom. Appropriate online learning activities could also provide students with the opporotunity to self assess their progress in the course [8]. The online learning medium could therefore assist in enhancing the quality of learning. The requirement to comply with rigorous quality assurance processes [9] demand increasing amount of staff time. With the establishment of Australian University Quality Agency [10] this demand is likely to grow even further. Academics report of being overwhelmed by administrivia and accountability requirements [11]-[13] that add little value to the quality of teaching. Since online interactive learning activities don t require any face-to-face interaction with academic staff, it could contribute in improving the quality of learning without putting much demand for intercation with academic staff. This is not to say that the development and maintenance of online learning resources do not involve efforts and time on the part of the academic staff. However, most universities now have well developed and supported online 1

2 learning environments. Once an online learning resource is developed for a course it can then be reused for future deliveries of the course without having to go through the laborious development process on each occasion. Surely academics will endevour to make enhancements and additions as appropriate (don t they always do), and modifications may be required from time to time with changes in the context and circumstances in which the course is offered. This is applicable to print based learning resources as well. The idea of using online technologies makes sense as it aims to utilise already existing learning environments to provide students with more choices to enhance their learning. The University of South Australia has been committed [14] to the use of technology for flexible delivery of programs so that students can exercise some choice and self-management in the way they approach their studies and feel supported in so doing. Over the last decade the University has moved in a systematic and planned way to make significant progress toward flexible delivery of academic programs [15]. There is both a practical and logical link between student-centred approaches, a focus on Graduate Qualities, and flexible delivery [16]. It is recognised that flexible delivery is facilitated by technological developments, most notably online technologies. The concept accords with the following definition: Provision of learning resources and the application of technologies to create, store and distribute course content, enrich communication, and provide support and services to enable more effective management of learning by the learner [17]. The University is also determined to be an international institution by recruitment of students from other countries to take its programs, with the option of studying either in Australia or remaining at home. It is envisgaed that by % of the University s total student body will be from overseas and that all students will take part of their program online [15]. For these reasons achieving greater interactivity within the online learning environment has been one of the priorities for the University. Developing interactive online learning resources is therefore totally in tune with the University s future teaching and learnbing framework, and priorities. I therefore decided to develop online quizzes for the course Principles of Computer Systems. This paper presents the processes involved in developing the quizzes and analyses the student responses obtained. It attempts to find answers to some key questions: whether students engage with the online quizzes and why, whether the flexibility of access (time and place) make them attractive alternative, whether students find the quizzes an interesting way of learning, whether they really benefit from engaging with the quizzes, whether the quizzes are useful for deeper learning in a technical engineering course etc. Before presenting the process of development of the online quizzes and the student responses, I would like to summrise the rationale for developing the online quizzes. To introduce students to the use of online technologies for learning in the very first year. This would be a valuable experience for them considering that most university courses use online resources in one form or another. To provide students with more choices for learning using already existing online infrasturcture. To assist students who are unable to regularly attend lectures due to work committments or other reasons by providing flexible access to interactive learning resources. To provide alternative learning resources and cater for the needs of students with diverse learning styles. To enable students to self assess their understading of the concepts and progress in the course. To provide students with reasonable (automated) feedback. While the feedback may not be compreshensive enough to answer all the questions the students might have, it will at least indicate to the students the validity of their concepts and/or expose problem areas. The online resources would be useful for prospective offshore programs the university aims to develop. THINGS TO EXPLORE This was my first experience as a teacher on the development of online quizzes. I was excited and thrilled about it. Nevertheless, I wanted to explore a number of things and learn from the experience, especially to what extent do these quizzes assist students learning, how much do students engage with them and how do they feel about them. So, to me it was not only a task involving the development of some active learning resources, but also an experiment backed up by the abovementioned rationales in order to explore the following, first-hand from my own students: Do the students find the quizzes relevant, interesting and motivating? How many times do students attempt each quiz (once, twice, three times or more)? Do the questions require students to think and do problem solving to get to the answer (deep or surface learning [18])? Do the students think of them as useful tools for assisting learning? Do they find them useful for self-assessment? Are the feedbacks provided appropriate and adequate? Do they cherish the flexibility (time and place) offered by such a system? 2

3 What are the weaknesses, if any, of this system? Would the students want to see more quizzes in future courses? DEVELOPMENT OF THE ONLINE QUIZZES I used University of South Australia s online environment called UniSAnet [19] to develop the online quizzes. Every course offered by the university has a default home page on UniSAnet. I had learning resources for the course titled Principles of Computer Systems under its home page. These included course information, study guide, lecture notes, tutorial and practical laboratory handouts. I had progrssively developed and enhanced these resources over a two year period in order to give students flexible access to course resources. However, these resources were merely a repository of the course materials. Majority of the students printed the materials off the web and used the printed materials for their study. In contrast, the online quizzes were aimed to be truly interactive learning resources. UniSAnet has a quiz wizard that facilitates the creation of multiple choice, true/false and fill in the blanks questionnaire among other types. As this was the first time I was developing online quizzes, I needed to learn how the quiz wizard worked. I consulted the Online Adviser in our campus and had two productive hands on sessions with him. He provided me some printed materials on using the wizard and pointed to some useful links for additional materials. I spent some time playing with the tool until I felt comfortable using it. I developed some sample quizzes and tried them myself the way students would in order to ensure that the system was functioning without too many undesirable distractions and problems. This is something I passionately try to do most of the time I develop some new laerning resources for students, otherwise some problems may go unnoticed leading to frustration among students. Type of Questions The course Principles of Computer Systems requires students to develop in depth (technical) knowledge on the topics covered. A deep learning approach [18] is essential to grasp the underlying concepts so that they can be utilised successfully to design, develop and implement simple digital computer systems. This being a foundation course for first year engineering students of all streams in our school, students performance in future courses in this area would depend on their level of achievement in this course. It is therefore essential to test students knowledge using an appropriate set of questionnaire that would test their understanding of various topics at a deeper level. This meant that I had to design questions that would require students to think and apply the relevant concepts to solve problems before they could answer the questions. I prepared solution to each question and tested them on UniSAnet. As for the format of the questions, I found the following two formats most appropriate for developing the questionnaire: Multiple choice Fill in the blanks In UniSAnet quiz wizard, there are two different ways to create provisions for feedback to students: Question feedback Answer feedback Although the way the two feedback mechanisms have been implemented in UniSAnet makes it difficult to differentiate between the two, they are both useful for providing feedback to students in order to assist them in getting their understanding of a topic clearer. In both cases feedback can be provided for both right and wrong answers. Progressive publication of quizzes Before publishing each quiz on the course web site, it was important to cover the relevant topics in the lecture and tutorial classes. As I taught only half of the course during the second half of the semester, I had about six weeks to get the whole thing done. I decided that I would prepare three quizzes and release them progressively at the beginning of weeks 9, 11 and 13 (week 13 being the last teaching week in a semester). As I published each quiz I sent an to the students alerting them of the availability of the newly released quiz. In addition, I informed the students about it in the lectures. The students wanted these alerts as they provided timely reminders to visit the web site and do the quizzes. Cost As stated earlier, I used University of South Australia s online learning environment called UniSAnet. It already had a web page allocated for the course and a wizrd accessible from the course web page for creation of online quizzes. Therefore, it wasn t necessary to set up a new web page. Neither was it necessary to acquire or develop any software for the task. From that point of view no cost was incurred for creation of the online environment itself. However, there is definitely a cost 3

4 associated with the development, maintenance and upgrade of the university s online environment, an estimate of which is beyond the scope of this work. There is a direct cost though for the development of online quizzes for any course. This is the time required for the development of the quiz questionnaire, feedback and testing. Neither the university nor the school saw this cost as I undertook the project on my own initiative. Nevertheless, I see this as a non-recurring cost in the sense that I would only have to make refinements, additions etc. in future years requiring much less time compared to the original development time. In total I spent 33 hours for developing, testing and releasing all the three quizzes. This included the time I spent in familiarising myself with the online quiz wizard and reading the relevant documentation. So, the total of 33 hours required for developing the three quizzes is possibly on the high side. Another point worth noting is that the time required to develop the quizzes reduced progressively from the first quiz to the last. STUDENT FEEDBACK I sought feedback from the students on a continuing basis especially during my face-to-face interactions with them. Majority of the students expressed strong interest in the online quizzes and said that they were helpful for learning. One student named Tristan Turner sent a spontaneous to me with the following message after the first quiz was released: I do not know how much positive feedback you get from students, but I thought I would send you a quick after doing the quiz to say they are really helpful in learning and applying the material that is being learnt. Applying knowledge is one of University of South Australia s strategic intents [20]. It was encouraging for me to know that the online quizzes assisted students learning and their ability to apply the knowledge gained to the practical project component of the course. I found most of the student feedbacks very encouraging. EVALUATION BY STUDENTS As stated earlier the primary motivation for me to undertake this project was to assist students with their learning, provide them with more choices, and also give them flexible access to intercative learning materials. The details of the rationale have been given in the Introduction. In my opinion one of the best ways to judge the effectiveness of a learning activity is to ask the students provided a democratic atmosphere [21] exists for students to express their views without any hesitation and/or fear. I therefore decided to get student evaluation done anonymously using an appropriate set of questionnaire that would provide answers to the key questions outlined earlier in the section Things To Explore. Questionnaire for evaluation In my view any set of questionnaire for student evaluation must be prepared with care so that it addresses the key issues, and that there is no ambiguity or contradiction. Otherwise one runs the risk of obtaining student responses that may be inconsistent to say the least. The questionnaire in Table I were designed in consultation with academics engaged in teaching and research in the area of education in general, and flexible delivery in particular. In addition they all had plenty of experience in student evaluation. Evaluation process The next question for me was to decide whether to distribute the questionnaire in a lecture class or use online evaluation. Our experience at the University of South Australia indicates that student participation in online evaluations of courses and teaching have been very low. The online instruments used at the university for these evluations are called Course Evaluation Instrument (CEI) and Student Evaluation of Teaching (SET) [22]. The participation of students have been approximately in the range of 15~20% in these evaluations. In contrast, if evaluations are done in a classroom using printed questionnaire, almost all the students present complete and return the questionnaire. This means that even if half of the students enrolled turn up at the lecture, then one can expect to get responses from around 50% of the students doing the course. While one of the objectives of developing the online quizzes was to support students unable to attend campus activities regularly, I chose to do the evaluations in a lecture class using printed questionnaire in order to get maximum response. This is an issue that would require further consideartions on how to motivate students to participate in online evaluation process and to make the evaluation process more inclusive. Out of a total of 140 students doing the course, 70 were present in the class all of whom participated in the evaluation. That is, 50% of the students enrolled in the course took part in the evaluation. The number of students attending the lectures did not exceed this figure for most part of the semester except during the early weeks. This is an indication of student disengagement, examples of which have been well documented by McInnis [1]. 4

5 Out of the 70 students who participated in the evaluation, 23% said that they did not attempt any online quiz. They were not asked to answer any further question apart from why they didn t attempt any quiz. While half of them did not give any reason, the other half said that they were busy with other things including work committments, assignments from other courses etc. and did not have the time to do the quizzes. However, majority of them thought that the quizzes were a good way of learning and would like to do them before the exams. In the remainder of this section, I would analyse the results of evaluation by the 77% of the respondents who attempted the quizzes. Evaluation by respondents who attempted the quizzes Figure I illustrates student responses on the questions of whether online quizzes were useful for testing their learning and for focussing on important topics. At least 70% students felt that way on both questions. Only 2% disagreed on the testing aspect while 6% disagreed on the focussing aspect, others were neutral. Clearly majority of the students who engaged with the quizzes found them useful for testing their knowledge and for focussing on important topics in the course. On the question of whether the quizzes were a more interesting way of learning than simply attending lectures, tutorials and reading the text, 58% agreed, 11% disagreed while 31% were neutral. This provides a reasonable indication that a large number of students found their engagement with the quizzes to be an interesting way to learn. On the question of feedback provided, 71% thought that they were appropriate and adequate, 7% disagreed while 22% were neutral. This shows that the feedback I had created with the questions were useful for most students. Nevertheless there are always rooms for improvements and I am certain that I shall be able to make them even more meaningful. 72% of the students said that the quiz questions required them to think and apply the associated concepts to find an answer, 2% disagreed while 26% were neutral. 67% of the students thought that they could not answer many of the quiz questions without clearly understanding the underlying concepts, no one expressed disagreement, 31% were neutral while 2% said that they were unable to answer the question. These figures are indications that the students had to grasp the underlying concepts in order to feel comfortable with the questions and answer them correctly. Only 37% of the students said that they revisited each quiz multiple times until they clearly understood the underlying concepts. In contrast, in answer to the question on how many times they attempted each quiz, 50% said twice, 11% said three times and 4% said more than three times. These lead to a total of 65%. This means that although many students (65%) attempted the quizzes multiple times, only 37% attempted them enough number of times until they understood the concepts. It was reassuring to me that 65% students attempted the quizzes more than once as it shows that they engaged with the activity in order to improve their understanding of the topics. 72% of the students said that the quizzes helped them to judge their strengths and weaknesses in the course. This in itself is a good outcome as long as the students act to enhance their knowledge by building on their strengths and addressing the weaknesses identified. An overwhelming majority of the students who attempted the quizzes (at least 80%) said that they enjoyed the flexibility offered by online quizzes as they could do them anytime and anywhere, and because there was no restriction on the number of times they could attempt them. This is depicted graphically in Figure II. Would students like to see online quizzes in future courses? An overwhelming 93% said yes. These are surely indications that the students found the quizzes useful and an interesting way to learn, and benefited from the activity. They would surely not want to engage in some activity unless they really thought that it was flexible and at the same time useful for their learning. Student comments Many students commented on why they would like to see online quizzes in future courses. I present some of the interesting comments: They are a good self-testing tool...i think they should be available right from the start of the course. Because of the instant feedback requiring no appointments/meetings. The animosity is a bonus for those not willing to see someone for help. Because it gives students a chance to check their knowledge of individual sections of the course, rather than testing on a large scale (exam). This allows students to have better focus on what they need to concentrate on....i found that many questions were difficult to do and I had to go through the materials, and then redo the quiz. This confirmed that I did not understand the concepts well at the first round. 5

6 Student feedback on user friendliness and other comments Many students commented on various aspects of the online quizzes, ranging from user friendliness of the online quiz environment to the activity not being assessable as a summative [18] one. Following are some of the comments on the user friendliness of the system: I would have liked to attempt a question (that I got wrong) again after studying the text without going through the whole quiz. The student is referring to the fact that if a student wants to revisit a particular question in another session then he/she has to go through the entire quiz all over again. In fact most quizzes are likely to be generated randomly from a repository of a large number of questions. A student may not necessarily come across exactly the same set of questions in his/her subsequent attempts of a quiz. The idea is that a particular quiz will test the student s knowledge of specific topics. Therefore, a welldesigned online quiz is expected to test the student s understanding of the relevant concepts irrespective of the exact set of questions generated in a certain session. However, facility for revisiting the questions a student gets wrong in a quiz immediately after taking the quiz would be useful. This way, the time required to browse through questions answered correctly can be avoided. Another student said: The use of test fields made answering difficult. For example, I answered IMM for Immediate addressing, and got the answer wrong. May be pull down menus. This was one of the fill in the blanks questions. Unfortunately the system looks for an exact match to decide whether the answer given is right or wrong. It does not allow the quiz developer to specify multiple alternate answers. Pull down menus could be a solution, but the software needs to have provision for such facility. Another student commented: Quizzes are overall a good way of testing our knowledge. However, I think the questions should tell the student what type of input is needed... This student is also referring to the fill in the blanks questions, which expect exact match for the correct answer. The suggestion made by the student makes sense. Some explanation and perhaps examples of the type of inputs allowed could be incorporated. Some of the students comments on making the online quizzes assessable are given below: I reckon it should be a policy that every subject has online quiz with gradable marks, eg, 5%. Too optional. Needs to be more compulsory... Perhaps 5% of final mark would encourage more students to do the quiz... There could be an incentive for those who try the quizzes. Clearly students would want to be rewarded for the efforts they put into some activity in a course. Some of them find motivations for engaging in an activity when there are marks allocated for that activity. Perhaps this is not a bad idea. If students benefit from the activity then why not allocate some marks to motivate them. I did not allocate marks because the the online quiz system in UniSAnet does not incorporate automatic scoring facility. In the absence of automatic scoring and notification facilities, it is not possible to make an online task assessable. I see value in incorporating such facilities into the online quiz system. They will make the online quizzes more attrcative for both academics and students. CONCLUSIONS Working on the development of online quizzes has been a very rewarding experience for me, not only because most students who did the quizzes spoke favourably about them, but because the students engaged with them and benefited from them. Online quizzes can be an effective way of engaging students in the context of increasing student disengagements. They can be used to stimulate and maintain the interests of first year undergraduates. The quizzes provide students with supplementary self paced interactive learning resources. Students can atempt the quizzes repeatedly until they have a clear understanding of the underlying concepts. Stuednts appreciate the flexibility and action oriented learning style of the online quizzes. The flexibility is specially useful for students who are unable to attend campus activities regularly due to various committments. 6

7 However, the online quiz system in UniSAnet have some limitations. The lack of automatic scoring and reporting facility is the most crucial one. Student feedback strongly indicate that making the online quizzes a summative assessment task would motivate the students further. In my experience, this is true for any acivity the stuednts are required to undertake in a course. They are reluctant to put serious efforts into something that doesn t count toward their final grades in some way. The other apparent limitation of the current system in UniSAnet is the requirement for an exact match in fill in the blank type of questions. While multiple choice questions provide one solution to this problem, fill in the blank questionnaire could be improved by facilitating provisions for multiple correct answers, for example, long and abbreviated forms of answers. Online quizzes can be a very effective interactive learning tool in the context of University of South Australia s teaching and learning framework for the next decade. References [1] McInnis, C., Signs of disengagement: the changing undergraduate experience in Australian universities, Inaugural Professorial Lecture, Faculty of Education, University of Melbourne, August 2001, pp [2] McInnis, C., James, R. and Hartley R., Trends in the First Year Experience, Publication of the Commonwealth Department of Education Science and Training, [3] Felder, R.M., Matters of Style, ASEE Prism, Vol 6, No 4, December 1996, [4] Carbone, E., Active learning in a large class, in Teaching Large Classes: Tools and Strategies. Thousand Oaks: Sage Publications, 1998, pp [5] Go8, Projection of Australia's GERD/GDP compared with the OECD Average with and without the contribution of the Innovation Action Plan, Group of Eight, April [6] OECD, Education at a Glance: OECD Indicators 2001 Edition, Jun [7] Schubert, J., BCA supports funding, Address at the Annual Dinner of the Business Council of Australia, Sydney, 11 October The Australian, 12 October [8] Laurillard, D., Rethinking university teaching: A framework for effective use of educational technology, London: Routledge, [9] DETYA, Quality Assurance and Accreditation in Australian Higher Education, March [10] Skilbeck, M. and Connell, H., Quality assurance and accreditation in Australian higher education: a national seminar on future arrangements, DETYA, [11] Illing, D., Driving your scholar, The Australian Higher Education, 5 June p. 21. [12] Lloyd, P., Crisis on campus, The Advertiser Insight, Adelaide, 1 April [13] Madden, J., Risk of stress for uni staff, The Australian Higher Education, 4 July [14] University of South Australia, Future learning environments for the University of South Australia: A report, June [15] King, B., Introducing the teaching and learning strategy , Presentation at the Contextual Awareness Seminar, 17 May [16] University of South Australia, Teaching and learning framework 2010, Final discussion paper, [17] King, B., Distance Education in Australia, in Harry, K., (ed.), Higher Education through Open and Distance Learning, World review of distance education and open learning, Vol 1, London and New York: The Commonwealth of Learning, 1999, p [18] Biggs, J., Teaching for quality learning at University, Buckingham: The Society for Research in Higher Education and Open University Press, [19] University of South Australia, UniSAnet: The online learning environment of the University of South Australia, [20] University of South Australia, Statement of strategic intent, [21] Brookfield, S., Becoming critically reflective: A process of learning and change, in Becoming a Critically Reflective Teacher, San Francisco: Jossey- Bass, 1995, p. 44. [22] University of South Australia, Course Evaluation Instrument, FIGURES AND TABLES TABLE I QUESTIONNAIRE FOR STUDENT EVALUATION Likert scale questionnaire: I like to do quizzes frequently as a way of testing my learning in the course. 7

8 The online quizzes helped me to focus my learning on important topics in this course. The quizzes were a more interesting way of learning than simply attending lectures, tutorials and reading the text. The feedbacks provided were appropriate and adequate. The quiz questions required me to think and apply the associated concepts to find an answer. I could not answer many of the quiz questions without clearly understanding the underlying concepts. I revisited each quiz a number of times until I was convinced that I clearly understood the underlying concepts. The online quizzes helped me to judge my strengths and weaknesses in the subject. I liked the online quizzes because I could do them anytime and anywhere. I liked the online quizzes because I could do them as many times as I wanted. Other questions: Approximately how many times did you attempt each quiz on average? Would you like to see more online quizzes in future courses? FIGURE I STUDENT RESPONSE ON ONLINE QUIZZES AS A WAY OF TESTING AND FOCUSSING LEARNING Student response on online quizzes as a way of testing and focussing learning Test learning Focus learning on important topics % of students Agreement (1:Strongly disagree, 2: Disagree, 3:Neutral, 4: Agree, 5: Strongly agree) FIGURE II STUDENT RESPONSE ON FLEXIBILITY OF ONLINE QUIZZES Student response on flexibility of online quizzes 50 % of students Agreement (1:Strongly disagree, 2: Disagree, 3:Neutral, 4: Agree, 5: Strongly agree) Time & location Quizzes taken any number of times 8