The Galileo Thermometer as an analogy for the successful measurement of student achievement.

Transcription

1 The Galileo Thermometer as an analogy for the successful measurement of student achievement. Mr. Glyn Robins University of Wales College Newport School of Computing and Engineering ABSTRACT A new novelty is appearing within the home of those who like gimmicks, this is the Galileo Thermometer, whose origins stem from the 16 th Century when Galileo formulated his principles of thermo-dynamics. The operation depends upon the changing density of a liquid with temperature and the movement of fixed density objects within the liquid. This idea can be used to determine a realistic level of achievement for a student when undertaking an engineering course if the assessment structure is designed to find a point at which the student is about to sink. Many of the traditional approaches to student assessment rely too heavily on the single stage approach to attainment grading which, using the Galileo Thermometer analogy, is similar to throwing the student into the water with a weight around his/her ankle and seeing if they sink or swim. For those who swim this is fine but for those who do not, there is often no route back. 1. INTRODUCTION It is a recognised fact that the number of students wishing to undertake engineering courses within the United Kingdom is reducing year on year. The days when hundreds of applications would be received by a company for a handful of jobs are a now a thing of the past. This is due to a number of diverse factors, which on the whole are beyond the control of educational establishments. What has become crucial both to those wishing to employ engineering students and those undertaking their education, is that individuals are not unnecessarily lost to the

2 engineering sector because of poor evaluation processes which have been used in the past but which may fail to indicate the true level of understanding of learners. In my experience, as a lecturer of twenty plus years, I have found that there exists a large discrepancy between the grades students obtain in longer duration assignment type assessments and those obtained in time constrained tasks and examinations. This has led to some institutions moving away from examinations and yet as the table below indicates there are good educational reasons for retaining time constrained activities and examinations. Table 1. Table 1. Assignment Long duration Examination Time constrained Advantages Time to reflect and research. Mirrors many real engineering scenarios. Integrity and security are easy to monitor. Mirrors some real engineering pressure scenarios. Requirement for many higher level qualifications Disadvantages Difficult to guarantee integrity and security. Requires time management skills Same questions and situations for all levels of ability. Student FEAR factor, If both methods are valid vehicles of assessment then we must pose the question, why are they not generating the same conclusions as to student s knowledge and understanding? The question is not easy to answer but I intend to propose one approach to addressing the situation. 2. THE WAY FORWARD Educational institutions need to address the processes and procedures adopted for the evaluation of performance of engineering students. It is my view that this must be done in two ways: Firstly, the adoption of an assessment material and scenarios that mirror is as closely as possible, the environment within which they are already working or intend to work. This has implications in terms of the required physical resources available as well as lecturer familiarity with the potential employment environment.. The use of real case studies and actual engineering problem in addition to problem-based learning is allowing education to meet the challenge of the first requirement. Secondly the adoption of an assessment process which meets the following requirements: A minimum threshold level, which is clearly defined, understood and achievable. An accurate measurable indication of the student s knowledge for evaluation and award purposes.

3 A supportive system that steadily extracts knowledge and understanding from the student. A realistic formative feedback of present level and a forecast of future potential for the student. A student controlled process of evaluation, which is able to meet their expectations. An assessment procedure in which there is full confidence regarding security and integrity. It is important that each of these points is justified and expanded, as these will form the framework for the new assessment design. It has been recognised by such organisations as the Quality Assurance Agency (Subject Benchmarks) and by EDEXCEL (NQF pass criteria) that there needs to be a minimum level of competence across the full range of areas covered by a module before a student can be awarded that module. If we were to go back even a short time this was not so. A student undertaking an examination with five questions could easily have passed with no knowledge in two of the areas. When you think that the student in question may be servicing the aircraft on which you are about to fly or designing a critical part of a machine on which you will rely, the assessment by the use of a pass mark becomes unacceptable. If we recognise and accept the need for a threshold of knowledge that will indicate module success, then we must also recognise that many students will have knowledge and skills that exceed these minimums. We must therefore ensure that the system is able to extract these enhanced skills from the student and be able to measure them. Whether these are defined as good and excellent, or merit and distinction, or 2.2, 2.1 and 1 or some other grade system is unimportant as long as across a qualification they are consistent. When considering a supportive assessment process this must not be confused with a system that wraps the student in cotton wool and protects them from perceived difficult examinations and allows them easier continual assessment, which can be completed in their own time. What it should meant is that a student is not dropped in at the deep end where they gasp for air and possibly drown but that they are allowed to steadily become accustomed to the water. In my experience, I have seen too many students who have left examinations early due to the fear factor. Someone once used the acronym shown in Figure 1. Many students entering an examination have very negative expectations and these are amplified when they read the first question. We need to ensure that requirements are explicit rather than implicit. If we require a student to demonstrate understanding in area d, and the only way of doing this is by demonstrating understanding in a, b and c first, then why simply ask for the answer d. This means no more work for the student but simply guides them through the question. We may ask, why F.. False E.. Expectations A.. Appearing R.. Real Figure 1. do we need to change? In the past, if we lost a few good students we could afford it; these days we need to ensure that all those who deserve to pass do so. Some may argue that the student

4 should know the steps to take to obtain the answer and to some degree I also agree, but we are attempting to assess more than a single level of understanding. The process of learning is changing. The half-life of an engineer (the time taken for half of his or her knowledge to become redundant) has reduced significantly. A time served and college/university educated employee would have a career long half-life with their knowledge being relevant up to the point where they retired. These days the half-life depends on the discipline and is often in low single figures. The concept of LLL (life long learning) is therefore an important consideration for any engineer. The concept of onward and upward seems to be the norm these days. A larger proportion of the population are entering Further and Higher Education but care must be taken as we often push too hard and have unrealistic expectations and these leads to disillusionment and disappointment. Consider two students A and B. A passes with threshold grades, whilst B obtains much higher grades. Both are successful and deserve their qualifications. The qualification in question is deemed to be sufficient to allow progression to a higher-level qualification. Should both be allowed to proceed? Have A and B been given realistic expectations of their potential? It is my view that the assessment process should provide clear indications to the student of their future potential. If we recognise and accept that students will end up with different levels of competence then we should allow students who recognise their final level to tailor their assessment to meet that level and no more. In my role as an External Verifier (EV) for EDEXCEL, I have seen assignment and examinations which have been designed with three sections, complete the first section and you have met the threshold Pass, the second section Merit and the final Distinction. This certainly allows the student to take control and yet care must be taken as: It is human nature to undertake the minimum amount of work to obtain success it is difficult to convince someone in an examination or undertaking and assignment to do more work. The process of sectionalising questions is analogous to dropping the student in deeper and deeper water panic and drowning can occur. With a move towards continuous assessment via assignments the concerns relating to integrity and security are of paramount concern. Assignment design is too often focused on set questions with unique answers and the temptation to compare and copy work is great. What is required is a system that explores possibilities and understanding rather than generating answers from remembered formulae. With this form of assessment it become obvious to all that plagiarism has occurred and therefore it is less likely that students will risk being caught plagiarising. 3. THE SOLUTION The Galileo Thermometer Model, which I propose, addresses many of the issues raised. Without exploring thermodynamic theory too deeply let us remind ourselves of some simple laws which followed on from Archimedes discoveries.

5 If we have an object in a fluid (normally liquid), if the object is more dense than the fluid it sinks, whilst, if the fluid is more dense than the object, the object floats. A submarine dives by filling some of its tanks with water hence increasing its density and when it wishes to surface if blows the tanks by filling them with air hence reducing its density. Here we have a fixed density fluid and a variable density object. If we consider a thermometer (liquid in glass) then as the temperature rises the liquid expands and the level moves up a graduated scale. This can be read off and indicates the temperature. What is actually happening is that we are increasing the volume of liquid but its mass remains constant. If we use the relationship: Mass Density = Volume then what is actually happening is that as the temperature rises the density drops. If we have an object with a fixed density, which is slightly, less than the fluid then it will float. If the temperature is now increased, then a point will be reached where the temperature has reduced the density of the fluid to a point where it is less dense than the object and the object sinks. If the temperature drops the object rises The Galileo Thermometer uses this concept by having a number of calibrated objects with slightly different densities. As the temperature rises the objects drop in order, with the temperature indicated by the last one to sink. The objects are tagged with a temperature indicator. This is shown in the Figure THE PROCESS Figure 2. The analogy I intend to adopt assumes that the students in the group to be assessed (objects in the thermometer) have different levels of competence/understanding (densities). To assess the students they are placed into an assessment regime (the fluid) and the complexity of the assessment (temperature) is steadily increased. Initially, all the objects bob about on the surface able to cope with the requirements of the assessment. They become accustomed to the setting and confidence levels are good. As the C 23 C 24 C

6 temperature is increased some will hardly notice the change and carry on, others will notice the change but as long as their head is above water they will continue. Some will obviously sink and these remove themselves from the section and move onto the next before they can become too distressed. The further into a question or a task the student progresses the hotter it becomes and therefore the student maintains control over their situation. The student is almost led into hotter and hotter water without them noticing, their knowledge being teased out of them bit by bit. This is shown diagrammatically in Figure 3. COOL WARM COOL HOT HOT HOT Traditional Assessment Sectionalised Assessment Proposed Assessment Figure 3. Those who reach the highest temperatures obtain the highest grades as expected but even they will reach a point where it gets too hot. This point may be artificially high and higher than necessary for the module but it will give a realistic indication of potential to assessing staff. Let us see how this process meets some of the requirements laid down earlier. A minimum expectation can be built into this model, not as the initial entry temperature, as even this may be a jolt to some individuals, but as a challenge to the individual based on the environment they find themselves in. Consider a question requiring the student to demonstrate safe working practices in an engineering environment. A question may lead them by the hand into a virtual environment describing the equipment/machines etc. The students may then be asked to identify potential sources of danger or may be asked about maintenance procedures. The answers to these questions may demonstrate total or partial threshold level for that area. Additional avenues can now be explored, possibly developing previously given answers. Measures of students understanding and knowledge can be gleaned from the answers as a whole, as long as clearly defined grading criteria exist and the question has been written in such a way as to draw out the evidence. I have seen assignments that do not give the students a clear indication of a required pieces of evidence. It is no good to say, well I expected the student to include this. The student is left with a lower grade than they might have obtained. This model, by its design, supports the student and steadily extends them. The strength of the question/task depends entirely upon its design. If well structured, generating a stimulating and realistic environment in which to work the student will feel confident to tackle the requirements

7 of the assessment. Poor design will leave the student floundering, unsure of the parameter in which he/she is operating. See Figure 4, Good Design Poor Design Figure 4. In the world of keep fit anyone who is using a piece of exercise equipment starts with a realistic resistance with which to work against. By the end of their session they know how far they have progressed and they have a realistic view of their fitness. The model described does the same, as the water gets hotter they know their limit. They will be aware of how far they have gone and they will be aware of their shortcomings without tasting failure. They will be aware of the increased heat they will face if they move on to higher qualification and they will be less likely to enter the kitchen if they cannot deal with the heat. There is always the danger with students that they will take the easiest route to a qualification even if their final grades are the minimum. The words I ll be happy with a pass are words I am not normally happy with when uttered by a student. There are times though when a module tutor or programme leader recognises that a student has reached their plateau and that pushing will not lift the student above this level. This is not a problem; the only danger is the possibility that assignment design may sink the student. This has been addressed in schools by streaming and the sitting of papers with a maximum possible grade. For instance, lower streamed Mathematics groups sit papers that permit maximum of a B or C grade at G.C.S.E. Also those more vocational and less academic can now sit vocational A levels or BTEC qualifications. Recognising ones ability and shaping the assessment is critical if we are to get the best result from the student. Examinations tend to be secure and their integrity is not in question. Assignments are by their whole nature different. A simple ongoing process by which students can feedback progress would not only enhance security but would provide valuable encouragement, inspiration and formative feedback to them. A student who is secure in terms of what they are doing will be less tempted to rely upon others. 5. APPLYING THE MODEL. If we see any value in the model explored within this paper we need to recognise that this will not happen without a great deal of effort by those people who are responsible for assessing

8 students. The list below indicates some of the preparatory and ongoing work that needs to be carried out. 1. Realistic grading criteria need to be generated for the modules and this needs to be linked to achievement levels whichever ones are relevant to the qualification. 2. Assessments need to be generated which are clear yet inspire imagination, which are prescriptive and yet not constrictive and which have clear goals but are also open ended. 3. Students need to be very well informed and where possible contribute actively to the whole assessment in terms of possible variable parameters. 4. Consistency, consistency, consistency, will lead to results, results, results. 5. Honesty, in terms of expectations and aspirations on all sides is required. 6. Formative Feedback needs to be rapid and constructive.