Perceptions of Students and Instructors of Online Objective Testing in Undergraduate Accounting Courses

Transcription

1 Perceptions of Students and Instructors of Online Objective Testing in Undergraduate Accounting Courses Pavithra Siriwardhane, Meredith Tharapos and Arabella Volkov School of Accounting, RMIT University Contact author: Meredith Tharapos School of Accounting RMIT University Level 11, 445 Swanston St. Melbourne, Vic. 3000, Australia Phone:

2 Perceptions of Students and Instructors of Online Objective Testing in Undergraduate Accounting Courses Abstract This study examines how students undertaking an introductory management accounting course and a capstone accounting theory course at an Australian university perceive the use of online objective testing. The use of a virtual learning platform, such as WebCT or Blackboard, is increasingly common in tertiary education and this study is a timely extension of previous attempts to improve our understanding of how this environment can be used to enhance the learning experience of undergraduate students. Using a modified version of the survey instrument employed in Apostolou, Blue, & Daigle (2009) it is found that students perceive both positive and negative aspects to online objective testing, however, overall perceptions of the students are positive. The results of the interviews with instructors suggest that instructors have positive perceptions of online objective testing and recognise pedagogical benefits of this type of assessment. 2

3 1. Introduction There has been limited investigation of students perceptions of online testing in undergraduate accounting courses. This study examines how students undertaking an introductory management accounting course and a capstone accounting theory course at an Australian university perceive the use of weekly, online formative and summative objective testing. An objective test is defined as a test where the testee selects one answer from a number of alternatives which are either correct or incorrect (Heywood cited in Bence & Lucas, 1996 pp ). The use of a virtual learning platform, such as WebCT or Blackboard, is increasingly common in tertiary education and this study is a timely extension of previous attempts to improve our understanding of how this environment can be used to enhance the learning experience of undergraduate students. Using a modified version of the survey instrument employed in Apostolou, Blue, & Daigle (2009) the perceptions of undergraduate students with respect to online objective testing (OOT) in an introductory management accounting course and a capstone accounting theory course at an Australian university will be examined. The introduction of weekly OOT to an introductory management accounting course and a capstone accounting theory course was perceived by the team members involved in the course design as having many potential benefits both for the students enrolled in these courses and the instructors undertaking the teaching in these courses. Perceived benefits to the students included the expectation that weekly testing, in a flexible learning environment, would encourage them to steadily progress through the course materials and simultaneously provide them with timely feedback on their comprehension of the course content. It was also envisaged that from an instructor s perspective the provision of OOT would provide timely linkages of assessment to the course content, improve student evaluations and ease the pressure of large assessment marking loads. Further, the use of OOT during an undergraduate course is reflective of the increasing use of online testing by the professional bodies (Petersen & Reider, 2002). There has been very little accounting education research that has investigated the perceptions of both students and instructors with respect to the benefits and weaknesses of using an OOT medium. In particular, prior research has often been focussed on the perception of first year business students who may not intend completing an accounting major. De Lange, Suwardy and Mavondo (2003) found support for online teaching resources enhancing student satisfaction but did not specifically investigate the variables which resulted in negative and/or positive perceptions of online testing. Petersen and Reider (2002) examined the use of online testing in professional accreditation 3

4 programs and, while they found that in general students had a positive experience with OOT, their respondents indicated that the use of OOT reduced their perception of the overall quality of the professional program. The use of OOT in summative assessment at first-year level has been found to impact positively on accounting major students learning, motivation and engagement (Marriott & Lau, 2008). However Apostolou, Blue and Daigle s (2009) research suggests that students perceptions of this form of assessment are more mixed than Marriott and Lau s (2008) research would suggest, and that further research needs to be undertaken to understand how online testing can be implemented to maximise the positive experience of students. This research will contribute to the accounting education literature by expanding our understanding of how students perceive the role of OOT in their university education. Specifically, this research will investigate the perceptions of students undertaking an introductory management accounting course which is a core unit for not only accounting students, but also various other programs within the business discipline, and final year accounting major students with respect to summative and formative OOT tasks. This study will seek to compare and contrast the differences in perceptions between an introductory multi-disciplined, practice based accounting course and a core capstone accounting theory course. This is an area which, to the authors knowledge, has not been studied previously and with this type of assessment task becoming more common in accounting programs, it is crucial that instructors gain insights into how these tasks can be rendered more effective in facilitating students learning and satisfaction with their studies. With the use of OOT becoming more common across undergraduate university programs it is important that instructors are able to understand and articulate the pedagogical benefits of this form of assessment. This study will also endeavour to identify the perceptions of undergraduate students and course instructors with respect to OOT in accounting courses at an Australian university. Identifying the strengths and weaknesses of this form of assessment will assist future improvements to this assessment practice in the undergraduate environment. 2. Review of the literature 2.1 The changing assessment environment Assessment is the process of systematically collecting, interpreting, and using information to improve student learning and satisfaction (Harwood, 1999). According to Biggs and Tang (2007) what and how students learn depends to a major extent on how they think they will be assessed and assessment practices must send the right signals to students about what they should be learning (learning objectives) and how they should be learning it. There should be constructive alignment 4

5 between the learning objectives and the assessment items employed. Assessment has been identified as having a major influence on students perception of learning and how they should learn (Biggs & Tang, 2007; Rust, 2005). This is further supported by Rust (2005) who suggests that assessment practice should be dynamic in order for it to have a positive effect on student learning. This positive effect was recognised by many researchers, who observed an improvement in student learning when the method of assessment was altered. A change in assessment method may result in a shift of the focus of the knowledge, skills, understanding and competencies that is being assessed (CNAA cited in Sangster, 1996 p. 132). Assessment is thought to develop in students a motivation and a commitment to learn and it could be argued that in order to test a wide range of intended learning outcomes, diversity of assessment practice is necessary (Marriott & Lau, 2008). Research undertaken by Galloway (2007 p. 2) for the Scottish Qualifications Authority identified a range of innovative assessment practices that include projects and investigations, multiple choice and other objective tests, oral assessment, realistic or problem-solving tasks, assessment based on simulations or role play, computer-assisted assessment, portfolios and profiles, group assignments and self, peer and co-assessment. Sangster (1996) suggests that student learning approaches can change when innovative assessment practices are utilised and that learning approaches in turn may depend upon the level and nature of the course and the manner in which course materials are presented. Information and communication technology (ICT) is having a significant effect on the academic activities of university faculty (Marriott & Lau, 2008) and Nicol (2006) states that recent advances in ICT are having a large impact on the delivery of student learning in higher education. Traditional paper based methods of assessment are being replaced by, or complemented with, the innovative use of ICT as it is believed that feedback can be improved by the use of technology, which can provide students with immediate automated feedback. However Rebele et al. (1998) identified a lack of research evidence to provide guidance on the integration of technology into the curriculum and called for more research in this area to explore how educational technology can contribute to the continuing evolution and improvement of accounting education.focusing on whether and in what context the application of technology results in more effective and efficient learning. 2.2 Online objective testing (OOT) Boyce (1999) states that the development of computer applications in education and increased accessibility to software and hardware has resulted in the adoption of computer technologies, including computer-assisted learning technologies, in the teaching programs of many university 5

6 accounting departments. Professional accounting bodies and employer groups have increasingly urged that accounting students be equipped with a range of technology skills relevant to their discipline. At the same time many university accounting departments have seen the use of information technologies as a way to make the education of students more efficient and/or more effective, facilitating the creation of innovative assessment practice which increases student engagement and motivation. Such innovation can be achieved without an excessive burden on the staff and students and can therefore be regarded as an efficient assessment option. OOT can be considered as one assessment option, if it is considered to be an appropriate way of encouraging students to engage more fully with the topic and the learning process. As far back as 1994, 65% of the respondents to a survey directed at accredited accounting courses in the United Kingdom indicated that objective testing was used as a summative or formative assessment tool (Bence & Lucas, 1996). An objective test is defined as a test where the testee selects one answer from a number of alternatives which are either correct or incorrect ) (Heywood cited in Bence & Lucas, 1996 pp ). Boyce (1999) predicted that the availability of costeffective computer technologies is a dynamic that has the potential to dramatically change university accounting education. Computer-based test banks, commonly supplied by textbook publishers, were an early innovation involving the use of computers as an instructional technology, offering teachers and learners advantages by flexibly testing the knowledge of baseline course content. These test-banks are essentially a productivity device, allowing more efficient conduct of certain types of examination including objective testing (usually multiple-choice or true/false type questions) where students sit computer-based tests on particular components of a course and are given a mark based on their performance. Computer-aided assessment (CAA), and by implication OOT, can be applied to summative assessments (Marriott & Lau, 2008). According to Biggs and Tang (2007) in summative assessment, the results are used to grade students at the end of a course or accredit at the end of a program as it is carried out after the teaching episode has concluded. Its purpose is to see how well the students have learned what they were supposed to have learned. The compulsory nature of summative assessment means that it can be used as a performance indicator to gauge the learner s success in meeting the assessment criteria. It need not be presented by one piece of coursework or one end-of-year examination, but could be replaced with a series of assessments delivered throughout the course (Marriott & Lau, 2008). The continuous nature of the tasks could take a wide range of forms, including tests, essays and presentations and would allow students to monitor their 6

7 performance, providing them with timely feedback that could be used to improve future performance. Correctly designed and implemented assessments can be both summative and formative in function. OOT provides a potential medium through which students can be provided with timely feedback on their progress within a course, while simultaneously completing summative assessment. Prior studies of students perceptions of OOT found that students identify the formative value of these assessments: They liked the course materials and they recognized the value of the OLAs 1 in keeping up-to-date with the study timetable and identifying their areas of weakness (Aisbitt & Sangster, 2007 p. 392). 2.3 Students and instructors perceptions of online objective testing (OOT) Of particular concern when designing assessment in accounting courses is the finding that students attitudes towards accounting as a profession deteriorates as they progress through the degree program (Marriott & Marriott, 2003), however it is unclear whether this deterioration extends to their attitude towards assessment. Further, their study suggests that there are institutional factors that affect a student s attitude towards the accounting profession as they progress through an accounting degree. Drawing upon the findings of Marriott and Marriott (2003), Biggs and Tang (2007) and Rust (2005) it is proposed that students attitudes and perceptions of assessment practices will change over the progression of their degree program. However, a study conducted by Peterson and Reider (2002) examining student and teacher perceptions of online testing for the Institute of Management Accountants, found that while students perceive computer-based testing as a positive experience, the use of computer-based testing may weaken the community s perception of the quality of qualifications awarded. The current student cohort enrolled at Australian universities have significant differences in their command of the English language and learning backgrounds as a result of the increasing number of international students, and this is all the more so in business degrees. As a result, Waldmann et al (1993) posit that consideration should be given not only to the type of assessment used, but also students attitudes towards different assessment items. The results of their study of first year students enrolled in a Bachelor of Business at an Australian university: support the view that South East Asian and Australian students differ in their attitudes towards tertiary assessment methods. Notably, South East Asian students were concerned with issues relating to examiner discrimination and unfair assessment, and rated multiple choice tests as fairer and more representative of subject knowledge than were written examinations (Waldmann, et al., 1993 p. 206). 1 OLA refers to online assessment 7

8 This suggests that students with English as a second language will perceive OOT positively as subjectivity in marking is minimised. Given the recent prediction that Australian universities: will need to significantly streamline their operations and asset base, at the same time as incorporating new teaching and learning delivery mechanisms, a diffusion of channels to market, and stakeholder expectations for increased impact (Ernst & Young, 2012, p. 4) it is now even more critical that the designers of accounting assessments understand the pedagogical implications of the online teaching environment. The perception of accounting students and accounting instructors with respect to the use of OOT as an assessment tool is one facet of our preparation for this new era in university education. As the literature suggests, students perceptions of different assessment tasks are not predicted to be static during their progression through their degree program. Given the paucity of our understanding of the perceptions of students and instructors regarding OOT the following research questions are proposed: 1. What, if any, are the positive perceptions accounting students have with respect to objective online testing? 2. What, if any, are the negative perceptions accounting students have with respect to objective online testing? 3. Do students perceive objective online testing differently in different accounting courses? 4. What, if any, are the positive perceptions accounting instructors have with respect to objective online testing? 5. What, if any, are the negative perceptions accounting instructors have with respect to objective online testing? 6. Does an instructor s perceptions of objective online testing influence their students perceptions of objective online testing? (as suggested by the Apostolou et al (2009) study) 3. Background At the time of this study, Management Accounting and Business (MAB) was a core first year course in the Bachelor of Business Accountancy (three year degree programme) and the Bachelor of Business - Professional Accountancy (four year degree programme) at a major Australian university. It was also a core unit for a number of different programs within the College of Business, and an elective for various other programs, and as a result had a large and diverse student cohort. It had a combined enrolment of in excess of 1,500 students per semester across all locations (Melbourne, Bairnsdale, Singapore, and Vietnam), making it the largest course in the School of Accounting in the semester the study was conducted. The course was taught over 12 weeks, face-to- 8

9 face, via a weekly two-hour lecture and a weekly one-hour tutorial. The lecture was delivered each week during the day for full-time students and repeated in the evening for part-time students. The weekly tutorials were conducted by a team of full-time and sessional staff in which students were presented with a practical problem to solve during the tutorial. The size of these tutorials was limited to 25 students per tutorial. All the learning materials were available on Blackboard (an online teaching platform), which was the main corresponding mode and were accessible by students at all times. MAB introduces students to the role of the management accountant and management accounting information as it is applied in a variety of business contexts. Each student was required to undertake a weekly OOT for ten weeks throughout the semester. The test comprised both multiple choice questions involving theory concepts and practical calculation questions based on the relevant weekly topic. The questions were selected by the course coordinator from the test bank provided by the publisher of the prescribed text. The test was available on Blackboard for one week only and accessible at all times. Five of the OOTs counted towards students assessment and students were advised in advance as to which were the assessable tests. With the assessable OOTs students were permitted two attempts and once started must be completed within a set time frame (determined by the coordinator). The aggregate of the five assessable OOTs counted towards 10% of the total assessment for the course. With the non-assessable OOTs students were permitted an unlimited number of attempts and an unlimited amount of time in which to complete them. Accounting Theory (AT) is a core course in the Bachelor of Business Accountancy (three year degree program) and Bachelor of Business - Professional Accountancy (four year degree program). The course was taught over 12 weeks, face-to-face, in the final year and was run as a three hour seminar session covering a weekly topic in addition to problem solving activities. All the learning materials were available on Blackboard, which is the main mode for correspondence, and were accessible by students at all times. The aim of this course was to examine various theories that have been put forward to understand the role of financial accounting in decision-making, and to explain why and how entities make decisions concerning disclosure of financial information in the absence of complete regulation. For AT, each student was required to undertake a weekly OOT for eleven weeks throughout the semester. Each test comprised eight multiple choice questions based on the relevant weekly topic and was drawn from the test bank provided by the publisher of the recommended text. The test was 9

10 available on Blackboard for one week only, accessible at all times, and once started was required to be completed within twenty minutes (determined by the coordinator), since multiple attempts were not permitted. The students were not given the option to either save their answer, or to change their answer once it had been submitted. In addition the questions were randomized in order to minimize cheating. Each test carried four marks (0.5 marks per question) and the final mark awarded to the student at the end of the semester was an aggregate of the best five test scores as a mark out of 20, which permitted the students the option of attempting all tests, or of being selective. 3. Research Method An anonymous paper-based survey was conducted among the undergraduate students who were enrolled in MAB and AT at the Melbourne campus to evaluate the students perception of OOT. The survey was administered by a research assistant among 540 MAB students and 225 AT students during week 8 of semester two The questionnaire was distributed to students at the end of their tutorials or seminar sessions. They were then asked to submit the completed questionnaire to the same research assistant during the following week s tutorial or seminar session. A total of 159 completed survey responses were received, resulting in an overall response rate of 20.7%. The response rate for MAB students was 22%, while the response rate for AT students was 16%. The relatively low response rate to the questionnaire was expected by the researchers due to the method of questionnaire administration required as a result of restrictive ethics approval conditions. It is a policy at the university at which the research was conducted that class time not be used for the administration of questionnaires. As a result students had to remember to complete the questionnaire outside class-time and also to bring it with them the following week. Further, the literature suggests that while paper-based questionnaires tend to result in higher response rates than web-based or ed questionnaires (Sax, Gilmartin, & Bryant, 2003), the characteristics of the university at which the questionnaire is conducted also affect response rates (Porter & Umbach, 2006). Several factors at the university at which this research was completed may have reduced the response rate from students, namely its urban location and its traditional teaching focus as opposed to a research focus (Porter & Umbach, 2006). The items in the questionnaire were based on previous literature, in particular the work done by Apostolou, Blue and Daigle (2009) and Peterson and Reider (2002). The modified instrument was submitted to a number of senior colleagues in order to gain feedback on the appropriateness of the 10

11 statements. Based on the feedback received the instrument was then further modified and tested. The questionnaire consisted of three main sections: items relating to the background and demographic data of the students, learning approaches of the students and the perceptions of the students of OOT. In the background and demographic details section there were nine questions including when and where the students attempt the OOT. In section two, the students were given 18 statements to indicate their preference regarding learning approaches on a five-point Likert scale (1= strongly disagree, 5= strongly agree). In section three, students were asked to rate their preference on 19 statements relating to perceptions of OOT on five-point and six-point Likert scales. Section three also included an open-ended question in order to offer students the opportunity to provide any additional comments with respect to OOT. The demographic details of each student cohort are given in Table 3.1. There were 123 respondents from the MAB course and 36 respondents from the AT course. Of the MAB respondents 66% were enrolled in an accounting degree and 61% were in their first year of tertiary study. For the AT course 97% of the respondents were enrolled in an accounting degree while 78% of the respondents were in their third year of tertiary studies. English was the first language for 49% of the MAB respondents and 36% for the AT respondents. Of the students who responded, 95% of the MAB respondents and 86% of the AT respondents were studying full time. In both courses the majority of the respondents were females (55% MAB, 67% AT). Of the MAB respondents, 55% were below 21 years of age while 61% of the AT respondents were in the age group of years. With respect to completion of the OOTs 36% of the MAB respondents and 44%% of the AT respondents attempted the OOTs towards the middle of its availability, while only 8% and 6% of the MAB and AT respondents respectively attempt it as soon as it was available. In both courses most of the students attempt the OOTs at home (79% MAB, 78% AT). Of the MAB respondents only 35% had previously used an OOT in their tertiary studies, compared to 74% of AT respondents. 11

12 Table 3.1: Demographics of the sample MAB AT Both Courses No: of responses % of responses No: of responses % of responses No: of responses % of responses Student responses Year of the degree First Second Third Fourth Other Degree enrolled in Accounting Other First Language English Other Mode of study Full-time Part-time Gender Male Female Age Group Under 21 years years years or over Test Attempted: As soon as it is available Towards the middle of its availability Towards the end of its availability Just before it closes Place Test Attempted: At University At home At your place of work Other On-line testing in previous subject: Yes No

13 Interviews were also conducted with instructors teaching either MAB or AT in semester 2, Two interviews were conducted with AT instructors and five with MAB instructors. All three researchers attended all interviews, which tended to be between minutes duration and were audio recorded. Participants were told the nature of the subject matter to be covered in the interview prior to the interview but were not provided with a copy of the interview questions in advance. 4. Data analysis and Discussion 4.1 Student perceptions of online objective testing (OOT) Independent t-tests were carried out to compare differences between means on activities the students engage in while undertaking the tests. The results are shown in table 4.1 and are divided into five activities. This table highlights the similarities and differences as to how the different student cohorts complete the OOTs. Examining the five activities that students could engage in while completing the tests, it is clear from the results that both cohorts tended to refer to the prescribed text book. In contrast, neither cohort tended to complete the test while at the same time and in the same room as a friend. However there are significant differences between the means of the two student cohorts with regard to the other three activities. MAB students responded that they engaged more in the following activities while completing the test: discussing the options with a friend, using any available online resources and contacting a friend for help when completing the online test. These differences could be due to differences in how the online tests were administered in the two courses. The AT online test was time limited and students could only attempt the tests once, whereas for the MAB tests the students were either given unlimited attempts or two attempts (as described above). As a result the MAB course provided more of an opportunity for students to collaborate on the tests compared to the AT course. 13

14 Table 4.1: Activities undertaken whilst completing the online test Question Brief Description Explanation for a rating of 4 Mean (S.D.) MAB n= Refer to the text book Always 2.75 (.90) Discuss the options with a Always 2.13 friend (.99) Use any available online Always 2.48 resources (.90) Complete it at the same time Always 1.66 and the same room as my friend (.92) Contact a help for friend Always 1.80 (.91) *p< 0.05; **p< 0.01 AT n= (1.0) 1.53 (.84) 1.92 (1.10) 1.39 (.76) 1.39 (.76) t test for equality means t- differences of P mean from zero value ** ** * Table 4.2 shows the difference between the mean perceptions of MAB students and AT students with respect to OOT. The results relate to Research Questions 1-3 as stated above. Of the 14 items presented regarding students perceptions of OOT, MAB students perceive 10 of the statements as being positive aspects of OOT, compared to 11 items for the AT students (a positive perception is interpreted as a mean above the neutral point of 3.0 based on five-point scale (Geiger & Ogilby, 2000)). Both the student cohorts have common positive perceptions on nine of the 14 statements with respect to OOT (Research Question 1). These are: (1) the effectiveness of OOT on depth of material tested, (2) OOT as an accurate measurement of student s level of understanding, (3) flexibility of scheduling and taking tasks, (4) permits faster feedback on test performance than traditional paper-based testing, (5) elimination of essay questions or long-form problems, (6) elimination of judgement in marking allocation, (7) forces students to regularly review the course material throughout the semester, (8) OOTs are better than other forms of assessment during semester, and (9) encouragement by the tutor to undertake the OOTs. Of the 14 items presented, MAB students perceive four items as negative while AT students perceive three of the items as negative (a negative perception being interpreted as a mean under the neutral point of 3.0 based on five-point scale (Geiger & Ogilby, 2000)). Both student cohorts perceive two of the statements to be negatively associated with OOT (Research Question 2): (1) the requirement that the student be knowledgeable about computers, and (2) provides students with the opportunity to cheat. However, out of the different positive and negative perceptions resulting from the two student cohorts only five of the positive perceptions have significant differences in means. 14

15 These significant mean differences need to be further analysed in order to gain insight into how OOT is perceived by the two different student cohorts (Research Question 3). The first significant mean difference was that online testing provides flexibility of scheduling and taking tests (p<0.05). Both student cohorts have positive perceptions with regard to this statement (Q3.3.1). However the AT students have a significantly higher mean positive perception on the above statement compared to MAB students. The reason for this difference could be that MAB students, having recently come from a secondary school environment, are more familiar with a structured learning environment and as such do not view flexibility of scheduling and taking tests as an important factor. As AT is offered in the final year these students may prefer more flexibility in their learning schedules and learning environment due to the pressure of assessments in other final year courses. The second significant mean difference is observed in the item online testing eliminates judgement in marking allocation (Q3.3.5). Here again the positive perception of AT students is stronger than MAB students (p<0.005). As AT is a theory course which requires understanding and application of theories into various real life situations the students may consider the subjectivity of allocating marks a significant factor compared to MAB students, whose course requires a considerable amount of numerical calculations which are less subjective to mark. The third significant mean difference is on the statement that online testing forces students to regularly review the course material throughout the semester (Q3.3.8). This statement was one of the perceived benefits expected by including OOT as an assessment item in both courses. The AT students hold a stronger positive perception than MAB students (p<.05). Due to AT being theory based, AT students may otherwise tend to delay revising the subject matter until required to do so by a forthcoming assessment item and therefore may perceive regular OOT as assisting in preventing this from occurring. MAB, on the other-hand, tends to build on the previous week s subject matter and therefore concepts and principles are revised at the beginning of each of the current week s lectures as a matter of course. The fourth significant mean difference exists on the perception that online tests are better than other forms of assessment during semester (Q 3.3.9). Once again AT students exhibit a stronger positive perception of this form of assessment when compared with MAB students (p<.05). A possible 15

16 reason could be that they appreciate the flexibility of this form of assessment, as compared to other forms of assessment, as outlined earlier. The final significant mean difference is on the encouragement of the tutor to undertake the OOT (Q3.6). The AT students hold a stronger positive perception on the statement compared to MAB students ( p<.01). Interview responses from instructors in MAB indicate that they did not encourage their students on a regular basis to undertake the OOTs. Instructors in AT, on the other-hand, appear to have been extremely vigilant in reminding and encouraging students to complete the OOTs on a weekly basis (refer to comments made by respective instructors in section 4.2). Table 4.2: Perceptions of objective online testing Question Brief Description Explanation for a rating of 5 Mean (S.D.) MAB AT n=123 n= The effectiveness of online Strongly expands testing on depth of material depth (1.09) (1.04) 3.2 Online testing as an accurate Accurately measurement of student s level measures (1.10) (1.18) of understanding understanding Flexibility of scheduling and Strongly Agree taking tests (.99) (1.13) Permits faster feedback on test Strongly Agree performance than traditional (.967) (1.09) paper-based testing Format allows guessing of answers Strongly Agree (2.85) (1.14) Elimination of essay questions or Strongly Agree long-form problems (.992) (.919) Elimination of judgement in Strongly Agree marking allocation (1.093) (.958) Elimination of partial credit in Strongly Agree marking allocation (1.04) (1.12) Requirement that the student be Strongly Agree knowledgeable about computers (.98) (1.21) Forces student to regularly review Strongly Agree course material throughout the (1.02) (1.00) semester Online tests are better than other forms of assessment during semester 3.4 Affects a student s overall stress level 3.5 Provides students with the opportunity to cheat 3.6 Encouragement of the tutor to undertake the online test *p< 0.05; **p< 0.01 Strongly Agree 3.20 (1.24) Strongly reduces 2.96 stress (1.07) Much more 2.72 difficult to cheat (1.02) Strongly encourage 3.21 (1.16) 3.69 (.92) 3.08 (.906) 2.47 (.87) 3.89 (.95) t test for equality means t- differences of P value mean from zero * ** * * ** 16

17 Table 4.3 shows how the students perceive the effect of different factors on their performance while undertaking the OOT. Since none of the differences of the mean perceptions between the two students groups are significant, only the perceptions of individual student cohorts are discussed here. The means are ranked based on their values. All five factors were negatively perceived by MAB students indicating that none of the factors were perceived to positively affect the performance of students while undertaking an OOT. AT students have a mean positive perception on only one factor, namely, required to read a computer while all the other factors were negatively perceived. The mean most negative effect of OOT perceived by both student cohorts was the speed of the computer may impact a student s ability to complete the test efficiently. The perceptions are ranked according to the mean values. Table 4.3: Factors that affect a student s performance while undertaking an objective online testing Question Brief Description Explanation for a rating of Required to read a computer screen Limits the student s ability to make notes on the test Does not allow reading time before commencing the test Speed of the computer may impact a student s ability to complete the test efficiently Size of the computer screen may impact a student s ability to complete the test efficiently *p< 0.05; **p< 0.01 Very positive effect / I do not know Very positive effect / I do not know Very positive effect / I do not know Very positive effect / I do not know Very positive effect / I do not know MAB n=123 AT n=36 Mean Rank Mean (S.D.) (S.D.) 2.95 (1.07) 2.69 (1.11) 2.78 (1.15) 2.65 (1.25) 2.98 (1.26) (.772) (.967) (1.07) (1.27) (1.22) t test for equality means Rank t- differences P of mean from value zero Students were given the opportunity to provide any additional comments they had with regard to OOT. Table 4.4 shows a summary of the comments provided by the students in both courses. From the total responses of MAB students 36% provided an additional comment, while 39% of the respondents in AT provided an additional comment. In both courses students provided more negative comments than positive comments with respect to OOT. The negative comments provided by the MAB student cohort include: lack of detailed workings in the feedback, technical issues in 17

18 the system, time consumption compared to the marks awarded and questions with a higher degree of difficulty compared to the questions discussed in lectures and tutorials. The negative comments provided by the AT student cohort include: the non-availability of the correct answers at the end of the session, technical issues in the system and badly worded questions. Table 4.4 Summary of additional comments provided by students about online testing Response Details MAB AT Positive Responses 10 (23%) 5 (14%) Negative Responses 27 (61%) 7 (50%) Neutral responses 7 (16%) 5 (36%) Total number of responses 44 (100%) 14 (100%) No: of responses as a % of total responses * 28% 9% No: of responses as a % of total responses in each course** 36% 39% * Total responses 159 ** Total MAB responses= 123: Total AT responses= Instructor perceptions of online objective testing (OOT) Of the instructors interviewed all agreed that OOT was a valuable and useful assessment item, with advantages to both instructors and students alike (Research Question 4). Advantages cited by the instructors in the interviews included a substantial reduction in time taken to write and mark assessment items, an additional medium to facilitate increased discussion during tutorials/seminars, an indicator of student s level of understanding of the current week s material which was then used to inform the teaching for the following week, and a valuable tool to assist in keeping students focused during the semester. I have to say when I first heard of it being introduced I would have to say I thought that s a cop out, I really did. But it wasn t until I understood what was going on and had a bit of a look at it that I realised how valuable it could be. (MAB instructor) It (OOT) makes a world of difference. If you can get one or two students coming up with comments or questions. It changes the entire class I believe. (MAB instructor) I use the results from the previous week s test to determine how much detail I need to go into in the next class. Otherwise I would be just guessing as to their level of understanding. I 18

19 mean if they don t say anything in class how would you know how much they had understood? (AT instructor) I think the main value is that it forces the students to go to the textbook, and hopefully the other resources that are there, because one doesn t see a lot of evidence of it otherwise. (MAB instructor) The pressure to undertake research, in addition to teaching commitments, was also a practical consideration in choosing OOT as an assessment item: I would stick with it. If staff s time can be feed up a little bit and put into something else, they are definitely worth it! (MAB instructor) I am under pressure to complete my PhD and so online tests, which mark themselves, cut down on my marking time without disadvantaging the students in any way. (AT instructor) Perceived advantages to students included providing an assessment item directly linked to the material being presented in the current week s lecture/seminar in order to promote understanding and encourage immediate reflection and revision, providing an assessment item in a medium that they were already extremely comfortable with, flexibility with respect to when the assessment was undertaken given part time and full time work commitments, and a reduction in the perceived subjectivity of marking across different instructors in the same course. I think if I were a student I would think it was quite an achievement to get through it and of course these students are far more computer literate than I will ever be and so they can probably find their way around things better. (MAB instructor) It is interesting as I haven t had any problems with students thinking that they have been hard done by with the way it is done. I would have expected perhaps, I mean there is always somebody. You can never satisfy everybody all of the time. But there haven t really been any complaints about the type of feedback. (MAB instructor) When asked about the limitations of OOT as a form of assessment (Research Question 5) AT instructors did mention the possibility of collusion, whereas the MAB instructors stated that even if students did work together, because of the nature of the questions in the course which involved calculations, they would still achieve a benefit from undertaking the test. 19

20 Because there are number calculations you actually have to do it yourself and everybody s numbers are going to be different. And the second attempt you are going to get a different set of numbers all over again. It is possible that someone could sit there and help someone but they are still going to learn while they are doing that so I don t necessarily see that. It is not a rote learned response because of the calculations and you do have a time limit. (MAB instructor) My view when it comes to that sort of thing, as it has always been, is that they are not going to win out in the end. I suppose they could do it together but I haven t heard of it occurring. (MAB instructor) In response to the perceived negative perceptions of students the AT instructors did mention the technological issues that they had encountered early in the semester. MAB instructors interviewed did not cite any perceived negative perceptions of students: When I have asked from time to time how they are going, fine, fine. Nothing really negative. No, I don t think I have had one negative comment to be honest. (MAB instructor) The instructors varied in their responses when asked whether they encouraged students in class to undertake the OOT (Research Question 6). The AT theory instructors who were interviewed were much more diligent in this regard than their MAB counterparts, who openly acknowledged that they had not been forthcoming in this area: And I think perhaps I probably could have put a little bit more into making use of them in class, checking more carefully what was in the questions and perhaps going through some of the issues. I haven t developed it in relation to a class situation as it probably could have been, but I think that there would be potential for doing something. (MAB instructor) 5. Conclusion This study has investigated how students and their respective instructors perceive the use of OOT as an assessment item in accounting undergraduate courses. The results indicate that students perceive both positive and negative aspects but overall their perceptions are positive in both courses investigated. Students in the MAB course were less positive, in general, regarding the OOT when compared to the AT students. This may be due to the different course content, the timing of the courses in the degree program and the differences in administration of the online tests in the two courses investigated. Another possible explanation for differences in student perception between the 20

21 courses could result from the perceived support (or lack thereof) shown by the respective instructors in the different courses. The instructors interviewed were all generally positive regarding the use of OOT but some reflected that they could have more actively encouraged the students to engage with the assessment. In terms of the limitations of this study, the research was conducted at only one Australian university and this limits the generalisability of the results. The normal limitations regarding the use of a survey instrument apply to the results relating to the perceptions of the students. In addition the response rate for the students was fairly low and not all instructors associated with the courses investigated were available for an interview. Nonetheless this exploratory study does provide support for the positive implications of the use of OOT as a form of assessment not only from the students point of view, but also from that of the instructors, and it is hoped will provide the impetus for further research in this area. References Aisbitt, S., & Sangster, A. (2007). Using internet-based on-line assessment: a case study. Accounting Education: An International Journal, 14(4), Apostolou, B., Blue, M., & Daigle, R. (2009). Student Perceptions about Computerized testing in Introductory Managerial Accounting. Journal of Accounting Education, 27, Bence, D., & Lucas, U. (1996). The use of objective testing in first-year undergraduate accounting courses. Accounting Education: An International Journal, 5(1), Biggs, J., & Tang, C. (2007). Teaching for quality learning at university: Open University Press. Boyce, G. (1999). Computer-assisted teaching and learning in accounting: pedagogy or product? Journal of Accounting Education, 17, De Lange, P., Suwardy, T., & Mavondo, F. (2003). Integrating a Virtual Learning Environment into an Introductory Accounting Course: Determinants of Student Evaluation. Accounting Education: An International Journal, 12(1), Ernst & Young. (2012). University of the Future: A thousand year old industry on the cusp of profound change. Retrieved from Public-Sector/UOF_University-of-the-future Galloway, A. (2007). Diversity and innovation in assessment practices in higher education institutions and by employers and training providers. In S. Q. A. Glasgow (Ed.), Research and information services bulletin. Geiger, M., & Ogilby, S. (2000). The first course in accounting: students perceptions and their effect on the decision to major in accounting. Journal of Accounting Education, 18,

22 Harwood, E. (1999). Student perceptions of the effects of classroom assessment techniques (CATs). Journal of Accounting Education, 17, Marriott, P., & Lau, A. (2008). The Use of On-line Summative Assessment in an Undergrauate Financial Accounting Course. Journal of Accounting Education, 26, Marriott, P., & Marriott, N. (2003). Are we turning them on? A longitudinal study of undergraduate accounting students attitudes towards accounting as a profession. Accounting Education: An International Journal, 12(2), Nicol, D. (2006). Increasing success in first-year courses: Assessment re-design, self-regulation and learning technologies. Paper presented at the ASCILITE conference. Petersen, B., & Reider, B. (2002). Perceptions of Computer-Based Testing: a Focus on the CFM Examination. Journal of Accounting Education, 20, Porter, S., & Umbach, P. (2006). Student Survey Response Rates Across Institutions: Why do they vary? Research in Higher Education, 47(2), Rebele, J., Apostolou, B., Buckless, F., Hassell, J., Paquette, L., & Stout, D. (1998). Accounting education literature review ( ), part II: students, educational technology, assessment, and faculty issues. Journal of Accounting Education, 16(2), Rust, C. (2005). Developing a variety of assessment methods, including self an peer-assessment, Mansfield, Quality Assurance Agency for Higher Education. Enhancement themes publications, 1, Sangster, A. (1996). Objective tests, learning to lean and learning styles. Accounting Education: An International Journal, 5(1), Sax, L., Gilmartin, S., & Bryant, A. (2003). Assessing Response Rates and Nonresponse Bias in Web and Paper Surveys. Research in Higher Education, 44(4), Waldmann, E., Romano, C., & Smyrnios, K. (1993). Attitudes towards objective testing: a crosscultural perspective. Accounting Education: An International Journal, 2(3),