Designing Effective Assessment in Postgraduate Construction Project Management

Transcription

1 pp (15) ISSN: (Online) Designing Effective Assessment in Postgraduate Construction Project Management Patrick X. W. Zou: The University Of New South Wales, Australia Abstract It is commonly agreed that assessment is the single most powerful driver on learning in postgraduate education. Effective assessment methods are held to be very important in maximising students learning, motivating students, fairly and reliably evaluating the subject and providing informative feedback to lecturers and to students on their learning progression. Assessment normally consists of individual assignments, group assignments and final exams. Furthermore, group assignments have been commonly used in postgraduate education to allow students to learn about teamwork skills. Assessment of group assignments in a proper and fair manner has been a major concern, however, and a challenge to both students and lecturers. Using a questionnaire survey, students perceptions on the various assessment methods used in a postgraduate construction project management course are investigated. The findings suggest that while the research-based group assignment was considered effective in demonstrating learning of technical content and collaboration skills, the students strongly favoured assessment of individual contributions to group assignments. Furthermore, it was found that group-based in-class oral presentations were also effective in demonstrating learning of technical content and developing presentation and communication skills, and peer assessment of the oral presentations was found to be worthwhile. While group assignment is a common assessment format, individual assignments and the final exam are also a necessary part of the overall assessment of a course. The research findings suggest that well-designed learning activities which correspond to learning objectives and the use of different assessment methods with prescribed assessment criteria are useful strategies contributing to successful learning and teaching in postgraduate construction management studies. Keywords: Assessment, Construction Management Education, Group Assignment, Presentation, Peer Assessment 80 Copyright 2008 CEBE

2 P.X.W. Zou: Designing Effective Assessment in Postgraduate Construction Project Management Introduction Assessment is the single most powerful informative and controlling device in learning and teaching. Brown and Knight (1994) placed assessment at the heart of student experience, and its impact on learning was described by Brown et al. (1997, p. 7) if you want to change student learning then change the method of assessment. The level of emphasis and effort that students place on learning activities are not a function of what the subject objectives are, but of what are going to be assessed (Ramsden, 1992; Williams et al., 2004). Indeed, Williams et al. (2004) list the purposes of assessment as: motivating students; directing and enhancing students learning; providing feedback to students on their strengths and weaknesses and guidance on how they might improve; helping to maintain educational standards; acting as a criterion to check whether course objectives have been achieved; allowing certification for the programme; providing feedback to the lecturers; and identifying both high achievers and students in need of support. A range of assessment instruments, such as assignments, tests, essays, reports and oral presentations, in group or individual format, are commonly used in university education to evaluate students learning and to determine their learning outcomes and grades. Palomba and Banta (1999, p. 11) for example, considered the most important criterion for the selection of the assessment instrument to be the provision of useful information. A good assessment technique is also held to be as one that contributes to learning (Wright, 1997; Kinnick and Walleri, 1995; Marchese, 1998, in Palomba and Banta, 1999). Brown et al. (1997) classified purposes of assessment into three categories, which can be described as beneficial to students (giving feedback, motivation, etc.), beneficial to lecturers (giving feedback, course evaluation, etc.) and also their informative importance for licensing, ranking, and acting as criteria for future courses. Assessment can be classified under two categories: the Criterion Referenced Assessment (CRA) and Norm Referenced Assessment (NRA) (Figure 1). CRAs are assessments designed for the purpose of understanding how well some contents have been learned, and its outcomes would define how well students have achieved the defined learning objectives and contents, while NRAs are those designed for the purpose of comparing students learning with each other. Cordon (2003) argued that in universities, assessment should be CRA and based on clear goals and objectives according to the course s goals and objectives. Likewise, Hardinghaus (2006) argued that assessment criteria should enable students to generate a picture about the values of a certain subject, and, as such, assessment should be criteria-referenced rather than norm-referenced. Alternatively, assessment may be divided into summative, formative and diagnostic forms (Figure 2). Crooks (2001) distinguished between summative assessment and formative assessment by stating that the former tries to summarise students attainment at a particular time, while formative assessment tries to promote further improvement of the students attainment. In other words, summative assessment is assessment of learning while formative 81

3 P. X.W. Zou: Designing Effective Assessment in Postgraduate Construction Project Management assessment is assessment for learning (Crooks, 2001). Diagnostic assessment may be used to identify particular strengths and weaknesses of students (Brown et al., 1997). Moreover, Brown et al. (1997) have also characterised assessment as either judgmental or developmental, where judgmental assessment is concerned with licensing to proceed to the next stage, while developmental assessment is concerned with students continuous learning. Assessment Assessment Criterion Referenced Assessment Norm Referenced Assessment Summative Assessment Formative Assessment Diagnostic Assessment Figure 1 Types of assessment classified in accordance to the reference used (i.e. criterion or norm) Figure 2 An alternative classification of assessment based on the aims of assessment Assessment methods may also be divided into direct and indirect methods. Direct assessment methods are those through which students display their knowledge and skills, such as objective tests (students choose among the responses provided) and performance measures (e.g. essays, presentations and classroom assignments in which students generate their own responses). Indirect assessment methods are those in which students reflect on their learning, such as exit surveys and interviews of graduates (Palomba and Banta, 1999, pp ). A further classification divides assessment methods into quantitative methods, which are mainly based on numerical scores, or qualitative methods, which are mainly based on descriptions (Palomba and Banta, 1999, p. 12). The selection of a suitable assessment method for a specific topic subject should be designed in accordance with the subject s objectives and aims (Cordon, 2003). However, it is not only what is to be assessed but also the type of assessment that influences students learning (Watkins and Hattie, 1985). As well as being important in the learning process, assessment provides lecturers with helpful information regarding how the teaching process is perceived by students. In this regard, assessment should reveal the students understanding of the learning materials. James et al. (2002) contend that old assessment practices may need to be revised or redesigned. This is due to many imperatives, including heightened awareness of the importance of assessment in guiding students learning, the need for developing generic communication and teamwork skills, perceived threat of an increase in plagiarism, finding 82

4 P.X.W. Zou: Designing Effective Assessment in Postgraduate Construction Project Management both cost and time effective methods for assessing large classes, new technological possibilities, and the changing nature of students. Despite all the recommendations available (Williams et al., 2004; Cordon, 2003; and James et al., 2002) on when and where each type of assessment might be used and what constitutes good or poor assessment practice, designing assessment tasks and criteria remains very complicated since it must take into account many different issues. There is no single best assessment design. Factors such as the current standing of students, number of students in the class, time and resources available, and course objectives all affect the choice of appropriate assessment. This raises a number of questions for course designers. This paper first describes the assessment methods used for a postgraduate core course, Computers in Construction Management, and then presents the results of a survey conducted at the end of the course, which aimed to understand the students experience and identify the pros and cons of the assessment methods used in the course. It should be pointed out that despite the fact that this research has used an individual course as a case study, the research findings may well be applicable to other postgraduate courses. This requires empirical evidence which itself would provide for comparative analysis. In addition, it is expected that the assessment forms presented in this paper may be useful for other courses too. Design Learning Activities and Assessment Tasks The course Computers in Construction Management was a core subject in the Master of Construction Management (MCM) Programme (which is now named Master of Construction Project Management Programme). The subject has three contact hours every week, comprising both lectures (two hours per week) and computer laboratory practical classes (one hour per week), during a 14-week semester. A course website using the WebCT platform was developed and used for delivering learning materials and some assessment activities and communications. A number of learning activities have been integrated in the course, including lectures, tutorials, computer laboratories and online group discussions. The following mix of assessment tasks were introduced: three individual tutorial questions; two individual assignments; five online individual quizzes; five online group discussion tasks; one major group assignment; and a final exam. 83

5 P. X.W. Zou: Designing Effective Assessment in Postgraduate Construction Project Management The three individual tutorial questions aim to develop students hands-on skills relating to the three software applications discussed in course, while the two individual assignments aim to enhance the skills learned from the course. The five online quizzes aim to provide students with the opportunities to learn the basic concepts and techniques discussed in the course. The purpose of the online group discussion tasks was to introduce peer-learning and knowledge sharing through an online collaborative environment. For the online discussions, the students were divided into five groups randomly using WebCT. The group members communicated during the course through website or face-to-face contexts, and each member s contributions were posted online so that lecturers and fellow students could assess them. This assessment accounted for 10% of the final mark for five topics (each 2%). The assessment consists of two components: individual contribution and the performance of the group as a whole. The major group assignment is a research report where students need to conduct in-depth research on a given topic. There were four or five students in each group which was formed by the student themselves. The students need to submit a written report and present their research findings to the class. Assessment criteria for both the written report and in-class oral presentation were issued to the students at the time of releasing the assignment. The assessment of the written report was based on the group as a whole (Table 1), but the assessment of the oral presentations consisted of individual performance and groupdynamics (Table 2). It should be noted that the entire class took part in the assessment of oral presentations using the criteria outlined in Table 2. Table 1 Assessment criteria for group written research report Executive summary Relevance to topic given Breadth of contents Depth of analysis Logic of argument Case studies New information Conclusions and Recommendations Source materials and References Signed statements of individual contribution and plagiarism Report presentation (format etc.) Excellent Good Fair Poor To facilitate the students to effectively work and learn from these team-based assessment tasks, group assignment guidelines developed by the author were provided to the students at the beginning of the semester. The 25 page guidelines contain the following sections: why 84

6 P.X.W. Zou: Designing Effective Assessment in Postgraduate Construction Project Management work in groups; how to work in groups; monitoring the processes and making progress; handling difficult situations; presenting group work; reflecting the group learning processes. The aim of the final exam is to allow students to revise and enhance their understandings of the key concepts, techniques of the course and further develop their skills and knowledge of the substantive contents. It also, to some extent, generates pressure for students to learn and pick up the individual elements they may not have learned thoroughly during the semester. The final exam is summative assessment. Table 2 Assessment criteria for oral presentation of research report Individual Presentation Presenter 1/2/3:. Knowledge of content (i.e. how well you know your research, clarity of logic) 20 Communication with audience 20 Voice and language (volume, tone, clarity, speed, use of clear and communicative language) 10 Body language and gesture (i.e. body expression, eye contact, how you capture the audience, etc.) 10 Level of confidence 10 Use of presentation media (easy to read, support to spoken presentation, use of graphics to illustrate the point, facilitating communication) 10 How well did the presenter "sell" the message? 10 Overall preparation and control of the presentation (being well prepared and organised) 10 Group Dynamics Quality of research content as a whole (i.e. relevance of contents to the topic given, clarity of logic and correlation between sections specific and in-depth) 20 Group/team cohesion (how each presenter s section relates to the whole, group interaction and support, communication) 20 Time management (start and finish on time, proper allocation of time between members) 20 Handling questions/comments and providing answers 20 Overall quality of presentation as a group 20 85

7 P. X.W. Zou: Designing Effective Assessment in Postgraduate Construction Project Management Questionnaire Survey A questionnaire survey aimed to investigate the students perceptions on the effectiveness of the assessment structure used in the course, including group/team work, oral presentation, and online collaboration. The survey was conducted at the last lecture of the course. The students were asked to respond honestly and they were assured that their responses were not personally identifiable and would not affect their grades, but would be used to improve the use of assessment methods in the course. The survey contained ten questions, each of which was a statement that students endorsed on a 4-point Likert scale (1 strongly agree, 2 agree, 3 disagree, 4 strongly disagree). A not-sure option was also provided in the survey form. The Student Profiles There were 30 postgraduate students enrolled in the course and at the time of the survey, 29 of them were present and participated in the survey, giving an effective respondent rate of 97%. The students came from 13 different countries: Australia, China, India, Egypt, England, Indonesia, Iran, Malaysia, Pakistan, Saudi Arabia, Sri Lanka, Singapore, and Vietnam. The students professional backgrounds were also different and included architecture, civil engineering, construction management, economics, real estate, materials engineering, systems analysis, building, finance, quantity surveying, interior architecture and industrial design. The students were in different stages of their studies with the majority in their first semester, however, most of them had worked in group assignments in their previous studies. The diversity of the students cultural and technical background provided an opportunity for them to share knowledge and to generate different ideas for group assignments. However, at the same time, managing this diversity posed a challenge to lecturers and students alike, since such diversity may lead to different learning methods and expectations, as well as time and effort commitments among the students. Survey Results and Discussions Table 3 shows the distribution of students responses among different questions and the following sections provide discussions of these results. Group assignment It is clear that the majority of the students (72.4% and 75.9%) believed the group assignment was effective for learning technical content and developing teamwork skills, respectively (Table 3). 86

8 P.X.W. Zou: Designing Effective Assessment in Postgraduate Construction Project Management Table 3 Survey Results Questions Agree (%) Agree (%) Disagree (%) Disagree (%) Not Sure (%) The group-based research report was an effective exercise in learning the technical contents of the course The group-based research report was an effective exercise in learning how to work as a member of a group The oral presentation of the research report was an effective learning process to practice/improve my presentation skills Listening to the other groups oral presentations was effective for learning technical contents of the topic present Listening to the other groups oral presentations was effective for learning presentation and communication skills Peer assessment of the oral presentations was an effective incentive for both the presenters to communicate and listeners to concentrate Assessing peers presentations was a worthwhile exercise The online group discussions have been effective in my learning The Group Assignments Guidelines provided by the lecturer was useful in doing group assignments Assessment of individual contributions to group assignment is necessary

9 P. X.W. Zou: Designing Effective Assessment in Postgraduate Construction Project Management As shown in Table 4, 96% of students experienced some benefit from the research report, claiming it helped them to either learn technical content (15%) or develop group work skills (22%), or more commonly both (59%, shaded cells). In contrast, only one student (4%) believed that he/she did not benefit from the group-based research report neither by learning collaboration skills nor technical content. Table 4 Distribution of students views on learning technical content and teamwork skills in the group-based research report The research report was an effective exercise in learning how to work as a member of a group Total Not Applicable Disagree Disagree Agree Agree The research report was an effective exercise in learning technical content Disagree Disagree Agree Agree Total (Note: the digits in the table represent number of students) Presentation and communication skills are vital for any project manager. Evaluation of group presentations has also been considered as a method to assess teamwork (Palomba and Banta, 1999, p. 233). This study investigated the effectiveness of developing presentation and communication skills through an authentic presentation exercise. The students responses indicated that giving a presentation and listening to fellow students presentations was effective in learning presentation skills, with 96.6% in support of giving a presentation and 93.1% in support of listening to other groups presentations. This is consistent with Palmer s (2000) research with engineering students which showed that students enthusiastically support oral presentations. As can be seen in Table 5, the majority of students fall into the area of agreement for both factors, although the interdependence of the two factors can not be ascertained due to the small sample size. The table shows that students think that listening to others presentations and making their own was effective for presentation skills development. However, the emphasis is more toward giving than listening to presentations (seven participants strongly agreed with the effectiveness of giving presentations, while only two found listening to the presentations highly effective). 88

10 P.X.W. Zou: Designing Effective Assessment in Postgraduate Construction Project Management Peer Assessment Based on this research it appears that there are a number of potential benefits provided by peer assessment: from the presenter s point of view, it reminds him/her that he/she is presenting to the entire class /public (rather than the lecturer only) and from the assessor s point of view, it makes him/her concentrate and listen to the speaker. Peer assessment has the potential to bring a sense of ownership of the process, developing evaluation and judgment skills in students, helping students to become independent learners, and making assessment part of the learning process (Brown et al., 1994). Table 5 Distribution of students responses regarding listening to and delivering oral presentations Listening to other groups oral presentations was effective for learning presentation skills Total Disagree Disagree Agree Agree The oral presentation for the research report was an effective learning process to practise/improve my presentation skills disagree Agree Agree Total (Note: the digits in the table represent number of students) Berg et al. s (2003) and Sher s (2004) research also suggest that when peer assessment is used, students comply with procedures and take it seriously. Peer assessments were used in this course to encourage students to be more involved during the presentations, prevent them from being distracted, encourage them to pay more attention to the presentation and develop assessment skills. In order to make the assessment criterion-based, for each presentation an assessment sheet was distributed to all students. The assessment sheet contained different sections regarding each member separately, as well as the group as a whole. Orsmond and Merry (1996) stated that the provision of evaluation criteria that address multiple dimensions/characteristics of the activity being assessed increases validity and students appreciation of peer assessment. 62% of students considered the peer assessment used in the presentations was suitable and 72.4% of them found it a worthwhile exercise (please see Table 3). However, about one quarter (27.5%) of the students did not consider peer assessment of presentations suitable. It is clear from Table 6 that 75% of students considered peer assessments to be suitable both for assessing presentations and as an instructive activity. Yet there are 17% of students who did not agreed with either and only 8% felt that although peer assessment is instructive 89

11 P. X.W. Zou: Designing Effective Assessment in Postgraduate Construction Project Management and useful, it is not appropriate for assessing presentations. Nevertheless, this remains an issue where further investigation is required. As shown in Table 3 Question 10, 82.1% of students believe that the individual contributions to group work should be assessed. This is confirmed with the findings of Zou and Darvish (2005) that the unequal contribution of individual members to group/team work was a major problem of group assignments. Diamond (1998) proposed dealing with these concerns by allocating specific responsibilities to individuals in group assignments. Therefore it is necessary to conduct further research to investigate and develop a mechanism which integrates the group (team) synergy while at the same time maximise and account for individual contributions. Table 6 Distribution of students views regarding peer assessment of presentations being suitable and a worthwhile exercise Assessing peers presentations was a worthwhile exercise Total Not Applicable Disagree Disagree Agree Agree Peer assessment was suitable for oral presentations assessment Not Applicable Disagree Disagree Agree Agree Total (Note: the digits in the table represent number of students) Online group discussions As shown in Table 3, about half (45%) of students considered the online group discussions offer an effective learning activity. Balfour (2007) finds that computer-based assessment and feedback tools can significantly reduce the effort required to provide individualised, written feedback to large groups of students. While the issue of online discussions requires more research, a number of possible reasons are presented here. The low ratings could be due to the fact that this method was new to the students, that they did not have any previous experience in it and were therefore not used to it. The other possible reason could be that the students were able to communicate to each other in a face-to-face manner since they were on-campus and attended class weekly. The third possible reason is that students worried 90

12 P.X.W. Zou: Designing Effective Assessment in Postgraduate Construction Project Management that they might make a mistake in writing their answer, which would remain on the course website permanently. The fourth possible reason raised by the students was that the weighting of two marks per task was not sufficient to motivate the students to put in the substantial amount of effort required. To verify these propositions, face-to-face interviews with students should be used in further research. Integration of individual assessment task, group-based assessment task, and final exam The students were asked for their views on the most appropriate percentage of combination of individual assignments, group assignments, and final exams, by distributing a total of 100% among the three types of tasks (for example, 20% final exam, 30% individual assignment and 50% group assignment). It is based on the assumption that if a student gives a higher percentage to an assessment method, he/she is putting more emphasis on that assessment method. The mean weight assigned to each assessment task is shown in Table 7. It is clear that the students view both individual and group assignment as well as a final exam as all being important assessment components for their learning. Therefore it is important that all these assessment components/formats be integrated when designing the assessment tasks and criteria for postgraduate studies. Table 7 Students views of weightings among different assessment methods/tasks Preferred proportion(%) towards assessment Mean (SD*) Individual Assignment 45% (14%) Group Assignment 30% (12%) Final Exam 25% (14%) *Standard Deviation Conclusions Assessment is one of the main drivers influencing learning outcomes, and therefore should be carefully designed in accordance with course learning objectives. This research has evaluated an assessment structure designed for a postgraduate construction project management course and it shows that while students considered research-based group assignment effective for learning technical content, developing teamwork skills and communication skills, it is important to include mechanisms that allow the assessment of individual contributions to group assignments. The online group discussion tasks were not well received by the students, however, as more than half of the students did not rate its 91

13 P. X.W. Zou: Designing Effective Assessment in Postgraduate Construction Project Management effectiveness. This may be due to the students were enrolled on-campus with a traditional class room teaching format. Furthermore, the research results suggest that individual assignments should not be forgotten and that integrating a variety of assessment methods of individual and group works, formative and summative assessment (i.e. exams) would be an effective method to help students develop different skills and ultimately drive the students success in learning in postgraduate studies. While the research was focused on a postgraduate construction project management course, which may have limited the data collection, the findings should be applicable to other disciplines, including both undergraduate and postgraduate studies. 92

14 P.X.W. Zou: Designing Effective Assessment in Postgraduate Construction Project Management References Balfour, J. A. D. (2007). Some light at the end of the feedback tunnel? CEBE Transactions, 4 (2), Berg, I., Admiraal, W. & Pilot, A. (2003). Peer assessment in university teaching. An exploration of useful designs. The European Conference on Educational Research, University of Hamburg, September URL: (accessed 5 February 2008). Brown, G., Bull, J. & Pendlebury, M. (1997). Assessing student learning in Higher Education. USA and Canada: Routledge. Brown, S. & Knight, P. (1994). Assessing learners in Higher Education. London: Kogan Page. Brown, S., Rust, C. & Gibbs, G. (1994). Strategies for diversifying assessment in Higher Education. Oxford: Oxford Centre for Staff Development. URL: (accessed 5 July 2008). Cordon, A. (2003). Assessment. URL: (accessed 5 February 2008). Crooks, T. (2001). The validity of formative assessment. British Educational Research Association Annual Conference, University of Leeds, September. URL: (accessed 5 February 2008). Diamond, R. M. (1998). Designing and assessing courses and curricula: A practical guide. San Francisco: Jossey-Bass. Hardinghaus, M. (2006). Learning outcomes and assessment criteria in urban design: A problematisation of spatial thinking. CEBE Transactions, 3 (2), James, R., McInnis, C. & Devlin, M. (2002). Assessing learning in Australian universities: Ideas, strategies and resources for quality in student assessment. University of Melbourne, Australia: Centre for the Study of Higher Education (CSHE). URL: (accessed 10 October 2005). Kinnick, M. K. & Walleri, R. D. (1995). Assessment of behavioural change and performance. In Nichols, J. O. (Ed.). A practitioner s handbook for institutional effectiveness and student outcomes assessment implementation. Edison, N.J.: Agathon Press. pp

15 P. X.W. Zou: Designing Effective Assessment in Postgraduate Construction Project Management Marchese, T. J. (1998). Thinking about learning in relation to assessment: A conversation. Presentation at the American Association for Higher Education Assessment Conference, Cincinnati, Ohio, June Orsmond, P. & Merry, S. (1996). The importance of marking criteria in the use of peer assessment. Assessment & Evaluation in Higher Education, 21 (3), Palmer, S. R. (2000). Student responses to activities designed to develop generic professional skills. Journal of Professional Issues in Engineering Education and Practice, 126, (4), Palomba, C. A. & Banta, T. W. (1999). Assessment essentials: Planning, implementing and improving assessment in Higher Education. 1 st Edn. California: Jossey-Bass Inc. Ramsden, P. (1992). Learning to teach in Higher Education. London: Routledge. Sher W. (2004). Peer assessment in the design and construction of a reinforced concrete lintel. CEBE Transactions, 1 (2), Watkins, D. & Hattie, J. (1985). A longitudinal study of the approaches to learning of Australian territory students. Human Learning, 4, Williams A., Gajendran, T. & Sher, W. (2004). Working in teams: Assessment of performance in building courses. In Chen, S. E., Brew, G., Gajendran, T. & Runeson G. (Eds.). Proceedings of The Australian Universities Building Educators Association (AUBEA) Annual Conference, University of Newcastle, Australia, 7-9 July, pp Wright, B. D. (1997). Evaluating learning in individual courses. In Gaff, J. G., Ratcliff, J. L. & Associates. Handbook of the undergraduate curriculum: A comprehensive guide to purposes, structures, practices, and change. A publication of the Association of American Colleges and Universities. San Francisco: Jossey-Bass. pp Zou, P. & Darvish, H. (2005). Students Perspectives on group assignments in construction management studies. In Sidwell, A. C. (Ed.). Proceedings of the Queensland University of Technology Research Week International Conference, Brisbane, Australia, 4-8 July, pp