Nottingham Trent University. Centre for Academic Development and Quality. Guidance on how to construct and use grading matrices.

Transcription

1 Nottingham Trent University Centre for Academic Development and Quality Guidance on how to construct and use grading matrices September 2014 Thanks are due to the School GBA Leads, School Learning and Teaching Coordinators, colleagues engaged in teaching within the Schools, and colleagues from the professional support services that have provided ideas, resources and feedback on this document. For further information on any of the points covered in the guide, contact Kamilah Jooganah, CADQ. 1

2 Guidance on how to construct and use grading matrices This document aims to support the grading practices at Nottingham Trent University (NTU) 1. The first part of the document covers the 'basics' and touches on designing module learning outcomes, selecting appropriate assessment tasks and writing assessment criteria. The second part, and main focus of the document, provides suggestions on how to construct effective grading matrices in order to enhance the learning assessment process. Grade-based assessment and grading matrices With Grade-based assessment (GBA), student work is assigned a grade (e.g. High 2.1). The grade is allocated through comparing features of student work against qualitative grade descriptors 2. Grading is often supported by an analytic grading matrix which illustrates the relationship between the assessment criteria and the learning outcomes of the module (see Appendix A for an example grading matrix). A typical grading matrix contains qualitative grade descriptors for each assessment criterion 3. According to the University Quality Handbook: The descriptors represent a set of common characteristics expected of work at each of the different grade bands and should be contextualised to disciplines and tasks by Schools and/or Academic Teams. 4 Nottingham Trent University Quality Handbook, Part C: Course design, management and enhancement. Section 15: Assessment (2014: 10) 5 The benefits of having a grading matrix Consultations with colleagues across the University have revealed that grading matrices work best where they are adapted to the assessment task at hand. This approach offers numerous benefits, including: Making it transparent to students what they need to do to achieve a high grade. Increasing students understanding of a module s intended learning outcomes and its relationship to the assessment criteria. Providing a framework for staff to think about the design of the assessments they set. For example, considering whether the assessments set are fit for purpose and are aligned to a module s intended learning outcomes and the teaching and learning activities. 1 The links in the footnotes can be opened when reading this document online. See GBA SharePoint site for an online version and for resources on GBA: 2 For some assessment types, such as examinations where student responses are either right or wrong, a marking scheme may be used to assign the grade. However, qualitative grade descriptors should inform the assessment design and the allocation of marks. See Appendices D and E. 3 Some colleagues at NTU use a holistic grading matrix. This is different from an analytic grading matrix as a grade descriptor is not provided for each assessment criterion (see Appendix B for an example). Holistic grading matrices can be particularly useful when grading examinations and/or when it is difficult to separate assessment criteria. 4 A grading matrix must be provided for every assessment element. Separate matrices could be provided for separate assessment tasks that comprise an element, such as if tasks assess different learning outcomes and/or the tasks have different assessment criteria. 5 NTU assessment guidelines 2

3 Supporting staff in providing feedback to students by, for example, explaining to students why they have achieved a certain grade and what they need to do to improve on their academic performance. Standardising the feedback staff provide to students. Helping staff with the moderation process by identifying what to look for when assessing student work. This can help to ensure parity across module teams in how student work is graded. PART 1: Getting the basics right For a grading matrix to be effective, it is important that the basics are in place and are aligned. Key requirements include: Having clear learning outcomes. Setting assessment tasks that are aligned to the teaching-learning activities and that can best evidence student performance vis-a-vis the learning outcomes. Identifying clear assessment criteria. For help with any of the above areas, as well as support relating to teaching and assessment activities in general, it is recommended that you contact the relevant persons at the Centre for Academic Development and Quality (CADQ) 6 and the Centre for Professional Learning and Development (CPLD) 7. It is recommended that you also contact your School s Learning & Teaching Coordinator or Quality Manager. 1.1 Learning outcomes A grading matrix is informed by a module s intended learning outcomes (ILOs). These are statements that predict what knowledge and skills students will have gained as a result of effective learning. Learning outcome statements are performance-oriented, typically: Begin with a verb (e.g. apply, evidence, demonstrate ). Written in the future tense. Relate to the module context. Specify the desired level of performance. Learning outcomes should be achievable (i.e. written at the pass level and not what you expect from the highest achiever), written at the appropriate level (the learning outcomes for Levels 4, 5 and 6 should be different in terms of expectation/demand) and should relate to the overall learning outcomes of a course. It is essential to make module learning outcomes more specific than the course learning outcomes. It is also important that the number of learning outcomes are manageable, usually between 8 and 12 maximum for most modules. Bloom s Taxonomy can be a useful tool when writing your learning outcomes. The taxonomy covers three domains that relate to Knowledge, Attitude and Skills (see Appendix F for an outline of the different domains). These different domains can act as useful prompts when considering what knowledge, skills, and attributes you wish students to achieve from your module. 6 CADQ staff 7 CPLD staff 3

4 The knowledge domain (or cognitive domain) is divided into six levels. These different levels and the associated verbs can be particularly useful when writing a module s learning outcome statements (see Appendix F for the different cognitive domains and associated verbs). Learning outcome statements should make clear to students what they will be getting from the module. To enable this, it is best to avoid ambiguous verbs. Learning Outcomes Recognise the main methods to conducting sociological research Critically evaluate the main methods to conducting sociological research Know the key principles of employment law Identify the key principles of employment law x x 1.2 Setting assessment tasks Establishing clear and specific module learning outcomes can help you decide what assessment task/s you can utilise to help determine student progress against the intended learning outcomes. The assessment task (e.g. essay, portfolio, examination) provides the mechanism or process through which students demonstrate their learning. This should essentially help you to gather evidence of student progress or achievement in relation to the learning outcomes. Assessment tasks should aim to assess what is significant to the student learning experience and include assessing higher order learning, graduate attributes and, where appropriate, the skills and knowledge students need upon graduation as specified by Professional, Statutory or Regulatory Bodies (PSRB). There are a number of different ways in which students can be assessed (see the guidance documents provided by CADQ relating to assessment 8 ). When considering different assessment formats, and thinking about the pros and cons of each, it is important to judge which one/s can help achieve constructive alignment. In a constructively aligned system, all components intended learning outcomes, teaching/learning activities, assessment tasks and their grading support each other, so the learning is enveloped within a supportive learning system. Biggs and Tang (2011: 109) 9 Learning and assessment should be integrated and fully aligned 10. Assessment should function as more than just a measure of student performance; it should help to enhance student learning. 1.3 Assessment criteria A grading matrix should contain the assessment criteria. Assessment criteria are the properties or dimensions by which student work is judged 11. Assessment criteria are informed by a module s learning outcome statements. Hughes (2007) 12 provides suggestions for academics when writing assessment criteria: 8 Nottingham Trent University. (2011). CADQ Guide. Assessment: Designing for inclusion. Nottingham Trent University. (2013). CADQ Guide to assessment design. 9 Biggs, J., & Tang, C. (2011). Teaching for quality learning at university: what the student does (4th ed.). Maidenhead: Society for Research into Higher Education and Open University Press. 10 See resource developed by The University of Sydney on Aligning learning outcomes to assessment type. 11 For information on assessment criteria, see Sadler, D. R. (1987). Specifying and promulgating achievement standards. Oxford Review of Education, 13, See resource developed by Hughes, Practical guidelines for writing assessment criteria & standards. 4

5 Keep descriptions concise. Assessment criteria need to make transparent to students the aspects that will be used to judge their progress against the learning outcomes. Avoid the use of adjectives or adverbs as these imply a level or standard rather than a criterion. For example, as opposed to fluent expression, this should be quality of expression. Be realistic about how many behaviours students can competently demonstrate in an assessment task and how many criteria assessors can take into consideration when grading student work. Hughes suggests that academics should collaborate with colleagues when designing assessment criteria and assessment standards. Assessment standards are different from assessment criteria 13. An assessment standard refers to the level of aspired or actual achievement (Sadler, 1987). In the context of this document, the qualitative grade descriptors associated with each grade band contained in a matrix outlines the standards against which student performance is judged. As well as collaborating with colleagues in writing assessment criteria, Hughes believes that involving students in this process can be an effective way of promoting a shared understanding of the basis for assessment judgements 14. PART 2: Constructing a grading matrix Now that we have covered some of the fundamental principles underpinning assessment practice, we can move onto the second part of this document which provides suggestions on how colleagues may approach constructing and using a grading matrix. To begin with, the document outlines what exactly qualitative grade descriptors are Qualitative grade descriptors A grading matrix contains the 17 grades used at undergraduate level at NTU (at postgraduate level 14 grades are used). These are usually grouped according to grade bands (there are six grade bands that are grouped by degree classification i.e. First, Upper Second, Lower Second, Third, Fail and Zero). In the First class range, there can be two grade descriptors, one covering Low, Mid and High First and another grade descriptor for Exceptional First. In the Fail class range there are also two grade descriptors, one for Marginal Fail, and another grade descriptor covering Mid and Low Fail. In some matrices the grade descriptor for Zero may be absent. The qualitative grade descriptors articulate the standard(s) that work needs to evidence to fall within a particular grade band, and in some cases to achieve a particular grade (i.e. Exceptional First, Marginal Fail and Zero). Each grade band may represent a qualitatively different level of thinking and/or demonstration of evidence. This is given for each assessment criterion (see GBA SharePoint site for the different formats grading matrices can take 15 ). 13 Sadler notes that there are different interpretations that exist for what is considered to be an assessment criterion and assessment standard. It is worth noting that when some academics and researchers in the field of Higher Education speak of assessment criteria they can be referring to assessment standards as defined in this document. 14 See the GBA SharePoint site for examples of how to engage students with GBA 15 GBA resources to support staff 5

6 NTU s generic grade descriptors should be used when constructing your grading matrix 16. The generic grade descriptors define the standards of performance expected across the grade bands and for certain grades. These are aligned to NTU s generic level descriptors. NTU s generic level descriptors offer a guide to the relative demand, complexity, depth of study and degree of autonomy expected of a learner at different stages of study, irrespective of the subject and context. The generic grade descriptors also include elements of the Structure of Observed Learning Outcomes (SOLO) taxonomy (these are in bold typeface in the descriptors). These can help colleagues to define different levels of understanding across the grade bands in terms of the structural complexity of students responses. Although staff may adapt the generic grade descriptors to their module, it is important that the elements of the SOLO taxonomy are retained How to construct a grading matrix A number of methods can be used for constructing a grading matrix. Some staff, for example, find it useful to conceptualise what they would expect from a piece of work for it to be awarded a grade in the Upper Second grade band. This in fact can be used as a point of reference or benchmark when constructing your matrix, that is where you would expect students to be if they were to, in basic terms, do everything you ask them to do and do it well 17. From this starting point, you may wish to do the following: Write a grade descriptor for the Upper Second band for each assessment criterion. This should be written in the future tense and from the student perspective. In fact this process should involve you asking yourself, Would a student understand what is needed for their work to fall into this grade band? Try to make the aspects or characteristics that are needed for work to be awarded a grade in the Upper Second grade band clear so students understand what is expected from them to achieve an Upper Second. Qualitative grade descriptors should also contain an element of generalised content. For example, grade descriptors should not list exact aspects of what makes a model answer otherwise students would just include these aspects in their work without engaging in any meaningful learning. Use NTU s generic grade descriptors and the SOLO taxonomy as a guide (see the next section on the SOLO taxonomy). Pick out key phrases like beyond prescribed range, demonstrates autonomy in approach to learning and incorporate these (also see section on Language). Incorporate key terms from the module s learning outcome statements and the assessment brief. If applicable, you may also wish to look at the feedback you have given to students in the past and pull together phrases like lacks analysis, referencing is poor as it relies heavily on web resources etc. Once you have constructed your grade descriptors for the Upper Second grade band for each assessment criterion, follow a similar process for the other grade bands and for particular grades. It might be useful to consider work in the First grade band as work that has gone beyond what was asked, and work that is an Exceptional First to go considerably beyond the level the module is taught at. For the grade bands below an 16 NTU generic grade and generic level descriptors 17 The benchmark suggested here, Upper Second, is different from the threshold level. The threshold level establishes a satisfactory level of attainment and no more, such as work within the Third range. 6

7 Upper Second, you may find it useful to consider these as characterising work that does not meet the benchmark of an Upper Second (see section on SOLO taxonomy on suggested methods for distinguishing between grade bands). Other staff have taken a different approach to constructing a grading matrix. Some have found it useful to start from the Exceptional First grade and then work backwards. This approach involves identifying what aspects are needed in student work for it to be graded an Exceptional First. For example, for a Level 6 project this could involve referring to the generic grade descriptors and asking What can a student do to achieve a standard that is close to, or is of publishable standards? Although there is no right or wrong way to approach constructing a grading matrix, Rust (2002) 18 does suggest that there is a danger when you start from the highest grade in that as you move downwards the statements can become more negative. This is worth bearing in mind. Trying to articulate what exactly makes a piece of work an Upper Second, or a Lower Second, Third etc. can be difficult, as the grading process often involves a high level of tacit knowledge. Yet, making explicit the intuitive knowledge involved in grading is of paramount importance for enhancing student learning. Indeed, one key requirement when writing grade descriptors is to make clear to students what is needed of them to achieve good grades. Matrices also play an important role in helping to ensure transparency and consistency in grading. Thus, it is essential when writing grade descriptors to remember that this needs to be constructed in a way to enable the marker to make consistent and fair judgements from the first piece of work assessed to the last. Furthermore, there needs to be consistency between markers. Indeed, the process of constructing the matrix should also involve working with colleagues on your module and course and looking at examples from elsewhere across the University. Some exemplars can be found on the GBA SharePoint site Using the SOLO taxonomy as a guide The SOLO taxonomy offers a model that differentiates five levels of structural complexity of understanding 20 : Prestructural: here students are simply acquiring bits of unconnected information, which have no organisation and make no sense. Unistructural: simple and obvious connections are made, but their significance is not grasped. Multistructural: a number of connections may be made, but the meta-connections between them are missed, as is their significance for the whole. Relational: the student is able to appreciate the significance of the parts in relation to the whole. Extended abstract: the student is making connections not only within the given subject area, but also beyond it, able to generalise and transfer the principles and ideas underlying the specific instance. 18 Rust, C. (2002). Purposes and principles of assessment. 19 GBA resources to support staff 20 GBA and the SOLO taxonomy 7

8 Figure 1 below offers an illustration of these different levels of cognitive activity. These different levels can be useful when considering the differences between certain grade bands. For example, grades across the Fail range could characterise work at a prestructural level. Work awarded a Marginal Fail could display elements of unistructural understanding, although perhaps not to the extent where it can be considered a Third. A Third may indicate that the student s work mainly evidences understanding at a unistructural level. Similarly, when considering the differences between an Upper Second and Lower Second, we may imagine that work awarded an Upper Second demonstrates greater relational understanding. For work in the First range, it is possible that the student evidences understanding at the extended abstract level. Figure 1: Image depicting the SOLO Taxonomy with associated verbs of understanding Language Grade descriptors often contain adjectives, however these should be carefully selected. It is important to minimise the use of vague terms such as good and excellent as on their own they can mean very little 22. Unpacking with colleagues the grade descriptors and particular key phrases such as what exactly excellent critical analysis means on your module and course is a useful exercise when constructing your matrix. Colleagues may have different interpretations of what key words and phrases mean. Such differences need to be minimised to ensure consistency in how matrices are used to grade student work. 21 John Biggs (nd). SOLO Taxonomy [online]. Retrieved 20 August, URL: 22 Grading matrices that draw on terms such as excellent, very good, good etc can be enhanced further through clarifying on what those terms actually mean. There may be instances where using general terms cannot be helped. Such as where students take other subjects as part of the module and the grade descriptors cannot be specific as to accommodate the requirements from these different subjects. 8

9 2.5. Get feedback on the grading matrix As part of the moderation process you should pilot the matrix with your module team to ascertain how effective it is for grading and, for example, how user-friendly it is for assessing student work. As a pre-marking activity it could be used to assess a sample of student work from the current or previous year s cohort. This can help with achieving a standardised approach between markers and may be particularly useful for new members of staff who are unfamiliar with the grading scheme at NTU and the grading culture on a course programme. In most cases, staff have suggested that constructing a matrix is an iterative process. It is unlikely that it will be perfect the first time round and where the form of assessment tasks remain constant it will involve an iterative process of adjustment for the next academic year Engage students with the matrix Sharing matrices with students, such as in a tutorial class, can be a useful exercise in introducing students to, or better acquainting students with GBA and how their work is or will be graded. This process before an assessment is due can also help you to ascertain whether students actually understand how their work will be graded. As with colleagues, you may find that the student interpretation of terms, such as synthesise, to be different to your own. As a result you need to explain to students what is meant by such terms and could involve students in a formative exercise where they assess work using the matrix. In fact, when constructing your matrix, it may be a good idea to share this with a group of students and get their feedback. This collaborative co-design can be part of a pilot exercise Weightings There are a number of different approaches staff take to weighting assessment criteria in grading matrices. On the one hand some staff use percentage weightings and on the other some use no weightings at all. Where percentage weightings are used, staff generally feel this helps to convey to the student which parts of the assessment are of greater significance. The downside with giving percentage weightings is that it can make grading a mechanistic process. Staff who prefer to take a more holistic approach to grading student work may use other ways to convey to students the importance of some assessment criterion over others/or the appropriate amount of effort students should spend on each section of an assessment. For instance, some staff indicate the amount of effort students need to spend on sections of an assessment through varying the indicative word length. In an exam, for example, in which students have to answer three essay questions, the paper may indicate that one essay requires approximately 1,000 words, and the other two require responses of approximately 2,000 words each. Whatever approach is taken to weightings, it is crucial that this is communicated clearly to students. It is also worth remembering that students place high value on consistency in how lecturers approach grading across modules. 9

10 2.8. Matrices for assessments of a numeric nature One of the rationales behind GBA is to ensure parity across the subject disciplines in how students are graded. Thus even work of a numeric nature should use the University s generic qualitative grade descriptors as a reference point when assessing students. Constructing grading matrices for assessments of a numeric nature, such as where the answer is either right or wrong, can be challenging. Some staff have reported that it can be hard to make the grade descriptors specific as this could give away the answer. However, as mentioned earlier, grading matrices are meant to have an element of generalisability. A grading matrix conveys the skills, attributes and knowledge that a module aims to impart to students. Even where a marking scheme is used, for example, where worked answers and the marks made available for each question are detailed, a grading matrix can help to provide additional and complementary information to students. At NTU, module leaders have taken various approaches to constructing matrices for assessments of a numeric nature. In Physics colleagues have constructed a grid for exams to give students an idea of the level of performance expected on different areas. The descriptors do not refer to specific questions, but refer to sections that are contained in the exam paper, such as calculations, graph construction, and the key aspects that are associated with the grade bands (see Appendix C for an example of an exam grading matrix) Grading matrices and exams For exams that involve a mixture of numeric answers and short discursive responses a marking scheme may be more suitable. For a mixed exam that requires long discursive and numeric responses and/or short discursive responses, a grading matrix can be used for the former and a marking scheme for the latter (for further information see Appendix D). In short, where it is possible to grade student work with a grading matrix this is strongly encouraged Grading student work Grading matrices can take different formats. Some staff have constructed a grading matrix for each assessment task whereas others have used one matrix to cover different assessments on a single module element. How a grade is found using a grading matrix can differ. However, the most common use of a grading matrix is where student performance on each assessment criterion is assessed against the standard/s as outlined in the qualitative grade descriptors. In finding the overall grade, staff tend to take different approaches. After highlighting the different sections of the grid, staff then find the grade which best represents the student s work. This can be done using the numerical equivalents for each graded criterion or by identifying what grade best fits student performance from the matrix. For a module element that is made up of more than one assessment 23, staff may take a holistic approach for each individual assessment, then calculate the overall grade for that element by using the numerical equivalents given to each piece of work, weighting these, adding the numbers then converting this to a grade. On the other hand, others 23 Guidance on multi-part assessment 10

11 just make an overall judgement based on a student s grades for each of the assessments that comprise that module element Feedback to students The benefits of using a grading matrix are that it can help to provide feedback to students. Yet, it is important to not rely solely on the grade descriptors to do this. For example, some staff provide personalised feedback on a student s performance in the comments section of a matrix. Where this is the case, the comments provided should align with the language contained in the grade descriptors. Providing a grading matrix for students formative work can be particularly useful in terms of helping them prepare for their summative assessment. Matrices should also be given for the majority of students summative work. Some staff have gone further and even provided an audio commentary with the grading matrix. The University Quality Handbook states that the following commitments apply to the format of feedback: Coursework. Students will receive individual feedback (written or recorded), including an individual grade, on all assessed coursework. Examinations. Feedback will be provided for all examinations, where a balance of individual and cohort feedback may be used. In deciding this balance, the course team must take into account the students experience of assessment across the level and course. Students should receive individual feedback for targeted examinations, to enable them to perform better in later examinations. The course team should clearly specify those examinations for which students can expect to receive individual written feedback and those for which they will receive cohort feedback. Nottingham Trent University Quality Handbook, Part C: Course design, management and enhancement. Section 15: Assessment (2014: 16) 24 The National Student Survey results for the 2013/14 academic year revealed that only 70% of students at NTU agreed or strongly agreed with the statement Feedback on my work has helped me clarify things I did not understand. Providing feedback to students can have a major role in enhancing their academic performance and overall learning experience. High quality feedback is characterised by the following: a. It helps to clarify what good performance is. This can be achieved by explaining the goals of the assessment task and providing clear criteria and standards of performance; b. It facilitates the development of self-assessment and reflection. If students can be helped to recognise the strengths and weaknesses in their performance then they can address these; c. It encourages teacher and peer dialogue around learning; d. It is provided in good time to feed into revisions or further work; 24 NTU assessment guidelines 11

12 e. It guides learning by helping students to understand the principles or ideas underpinning their work; f. It encourages self-belief and positive motivation. Nottingham Trent University Quality Handbook, Part C: Course design, management and enhancement. Section 15: Assessment (2014: 14) 25 For help on how to formulate and deliver effective feedback to students, it is recommended that you contact the relevant persons at the Centre for Academic Development and Quality (CADQ) 26. It is recommended that you also contact your School s Learning & Teaching Coordinator Final word Constructing a grading matrix is a complex task. However, once done, the benefits are numerous. If constructed appropriately, grading matrices can greatly support student learning and can help staff in how they assess students. There are many different approaches that staff may use to constructing grading matrices and this document outlines some of these as well as makes suggestions that may be of use. 25 NTU assessment guidelines 26 CADQ staff 12

13 List of Appendices Appendix A: Appendix B: Appendix C: Appendix D: Appendix E: Appendix F: Appendix G: Example Analytic grading matrix Example Holistic grading matrix Grading matrix for numeric exam Aligning grade descriptors to exam marking scheme Example matrix where grade descriptors are aligned to exam mark distribution Bloom s taxonomy Grading matrices checklist 13

14 Appendix A: Example Analytic grading matrix 27 Course: Module Name: Module code: Assessment type: Research Project Research Project Assessment Task at Level 6 (third year) Assessment Date: Module learning outcomes assessed: a) Critical understanding of chosen topic; Student Feedback Form Student ID: Grade: Graded by: 1 st Assessor 2 nd Assessor Date Graded b) Design and justify a suitable research framework within which to undertake an inquiry into a chosen topic; c) Apply the research framework following the correct procedures; d) Structure a discussion in a coherent and convincing fashion by synthesising the material gathered in the context of the research; e) Present information in an appropriate style, adhering to standard academic and/or professional conventions, giving full details of sources used according to the referencing standard laid down. NB: Final grade determined by how well the criteria have been met overall and by not the sum of the individual aspects of the work. 27 Original matrix developed by Dr Graham Pitcher, Senior Lecturer, Accounting and Finance. Matrix adapted by Alison Stewart (CPLD) and Kamilah Jooganah (CADQ). The matrix should be viewed as an example matrix and not a model matrix. To achieve optimal efficiency the descriptors need to be adapted to module and task specific contexts. 14

15 Assessment Criteria Module Learning Outcomes to which they relate Grading Descriptors Exceptional First First (H/M/L) Upper Second (H/M/L) Lower Second (H/M/L) Third (H/M/L) Marginal Fail Fail (M/L) Level of critical understanding of chosen topic a Demonstrates extensive critical understanding of topic in terms of depth and breadth. Evidence of independent study. Draws on and critiques latest developments in the subject area not covered on the module. Level of critical understanding goes considerably beyond what has been taught. Demonstrates capacity for critical evaluation of new knowledge. Presents a novel approach to the relevance and applicability of theory in the context of the research. The work is of, or close to, publishable standards. Demonstrates critical understanding of topic. Evidence of independent study. Draws on and critiques latest developments in the subject area not covered on the module. Level of critical understanding goes beyond what has been taught. Evidences the ability to relate concepts together and apply these to the context of the research. Demonstrates critical understanding of appropriate theoretical frameworks. Evidence of independent study. Extends beyond the readings covered on the module. Some outside sources are critiqued. Evidences the ability to relate concepts together and apply these to the context of the research. Demonstrates knowledge of appropriate theoretical frameworks. Evidence of independent study, although work is mainly, though not completely, centered on set sources. Level of understanding is balanced towards the descriptive, although some critical insights are made. Meaningful synthesis of theories and its application to the research is limited. Demonstrates an awareness of theoretical frameworks, but level of understanding is mainly descriptive. Work lacks meaningful synthesis of theories and its application to the research. Evidence of independent study is limited. Sources drawn on are constrained to the readings covered on the module. Demonstrates an awareness of some of the key theoretical frameworks with some simple connections made. Significant theories are not covered and/or major mistakes in interpretation made. One clear distinction between a 3 rd and 2.2 is the absence of meaningful synthesis of ideas/concepts in the former. Does not demonstrate an understanding of key theoretical frameworks. Level of understanding is at the word level with facts being reproduced in a disjointed or decontextualised manner. Quality of critical engagement with literature. a & b Critical evaluation of the literature is detailed and robust Critical evaluation of the literature is robust and detailed. Critical evaluation of the literature is robust. Review of the literature is balanced Review of the literature is descriptive. Review is descriptive. Work fails to address some Core readings are absent altogether from the review and/ or theories 15

16 For Exceptional 1 st and 1 st at L6, the student would need to clearly evidence the ability to go beyond the prescribed range. This ability is also evident albeit to a lesser extent at 2.1, and possibly at 2.2. in both breadth and depth. Consolidates and extends on knowledge in a novel/creative way. Key theories are critiqued, challenged and/or interpreted in a novel way. Theories are synthesised and related to the research. Literature used is considerably beyond the prescribed range (e.g. latest articles/policies/dev elopments in subject area are drawn on). Consolidates and extends on knowledge. Key theories are critiqued, synthesised and related to the research. Literature selected is clearly beyond the prescribed range (e.g. latest articles in the subject area are drawn on). Consolidates and extends on knowledge. Key theories are critiqued, synthesised and related to the research. The review draws on some outside sources. towards the descriptive. There is some evidence of the ability to critique, synthesise and apply key theories to the research. Consolidates and extends on knowledge to some extent. The review may draw on some outside sources. However, review is reliant on the prescribed range from the module. Meaningful synthesis and application of the key theories to the research is absent. There is evidence of some ability to evaluate core readings. The literature used is limited to prescribed range from the module. aspects of the brief. Needs to provide some evaluation (e.g. note the strengths and weaknesses) of the core literature to achieve a pass standard. are reproduced in a disjointed manner. Of a standard close to or of publishable work. Choice and justification of research methods b Evidence of innovative/creative research methodology. The justification of methods evidences a logic that goes considerably beyond the level taught. Methods adapted to fit the research context in an imaginative and insightful way. Evidence of welldeveloped research methodology. The justification of methods evidences a logic that goes beyond the level taught. Methods adapted to the research context. Evidence of appropriate methodology. The justification of method choice is critical. Concepts are related together. Methods fit the research context. Appropriate methods selected. Justification is lacking in substance. Mainly descriptive, however there is some evidence of critical engagement. Methods selected are somewhat appropriate. Justification for choice of methods is descriptive and knowledge is limited to basic facts and concepts. Inappropriate choice of methods. Poorly argued/illogical justification given for choice of methods. It is important to retain elements of the SOLO taxonomy to ensure parity of standards. These are in bold typeface in the generic grade descriptors. Inappropriate choice of methods made or choice of methods is unclear. No justification given for choice of methods. Facts are reproduced in a disjointed or decontextualized manner. 16

17 The work is of, or close to, a standard as observed in a published research article. Methods fit the research context. Quality of analysis, and evaluation of findings a, c & d Analysis and evaluation of findings are critically sound in depth and breadth. There is synthesis between research findings and the ideas presented in the literature review. Based on the research findings, key ideas in the literature are challenged in a way that goes markedly beyond the level at which the module is taught at. Conclusions made are warranted and supported by the research. Novel/creative insights are made. Standard is close to or is of publishable standards. Analysis and evaluation of findings are critically sound. There is synthesis between research findings and the ideas presented in the literature review. This is achieved in a manner that goes beyond the level at which the module is taught at. Conclusions made are warranted and supported by the research. Insights made go beyond expectations at this level. Analysis and evaluation of findings are critically sound. There is synthesis between research findings and the ideas presented in the literature review. Conclusions made are warranted and on the whole supported by the research. Discussion is on the whole balanced towards the descriptive rather than the critical. Limited meaningful synthesis is made between findings and literature. One clear distinction between a 2.1 and 2.2 is the balance towards the descriptive rather than the critical in the former. Discussion of findings is descriptive. No meaningful synthesis is made between findings and literature review. Basic conclusions are drawn which are not fully justified by the research as a whole. Discussion of findings is limited and descriptive (e.g. does not reflect on/describe all the research findings). Misses the point. No meaningful synthesis is made between findings and literature review. Conclusions are not coherent/irrelev ant. No/little attempt is made to engage with or describe the findings. Conclusions are absent. Conclusions made are sound although not fully justified by the research as a whole. 17

18 Standard of presentation & structure of work e Written expression is eloquent and coherent making it enjoyable and engaging to read throughout. Structure and navigation of work is elegant and flows seamlessly. References are all listed and cited correctly. Written expression is clear and engaging, although contains the occasional grammar and spelling mistake. Structure and navigation of work flows seamlessly. References are all listed and cited correctly Written expression is clear although contains some grammar and spelling mistakes. Structure and navigation of work is clear and logical. References are cited correctly, with only minor errors. Written expression is clear. Work contains some grammar and spelling mistakes. Structure of work lacks coherency in some places. However, this does not greatly impact on navigation. References cited, but some are incomplete and/or contain errors. Written expression competent but lacks clarity in places due to minor issues with sentence construction. Work contains some spelling and grammar mistakes. Structure of work lacks coherency. This makes navigating the work difficult in places. Referencing is incomplete and/or incorrect. Written expression lacks clarity. Key points are not clearly articulated. Work contains frequent spelling mistakes. There are issues with sentence construction. The structure of work is not coherent. Referencing is mostly incomplete and/or incorrect. Written expression lacks clarity. No attempt is made to articulate key points. Work contains frequent spelling mistakes. There are issues with sentence construction. The structure of work is not coherent. No attempt at referencing is made or most references are typically missing. Areas of strength Areas to improve on 18

19 Appendix B: Example Holistic grading matrix Essay-based exam (5 questions), Level 6 28 Class Scale General Characteristics FIRST (Excellent) Exceptional 1st All questions are answered with no mistakes made in terms of content and written expression. Knowledge and understanding of concepts is critical, accurate, and extensive in both depth and breadth. Evidences the ability to relate concepts together and apply these to different contexts. Some novel insights/interpretations made. Arguments made are coherent, logical and supported with well-considered examples. The work is of a standard that goes considerably beyond the level the module is taught at. High 1 st All questions are answered with only minor mistakes made in terms of written expression, e.g. spelling, grammar. Mid 1 st Knowledge and understanding of concepts is critical and accurate. Evidences the ability to relate concepts together and apply these to different contexts. Low 1 st Arguments made are coherent, logical and supported with well-considered examples. The work is of a standard that goes beyond the level the module is taught at. UPPER SECOND (Very good) LOWER SECOND (Good) High 2.1 All questions are answered with only minor mistakes made in terms of written expression, e.g. spelling, grammar. There may also be some minor omissions/mistakes in terms of content, e.g. some theories could be further critiqued. Mid 2.1 Knowledge and understanding of concepts is critical and accurate. Evidences the ability to relate concepts together and apply these to different contexts. Low 2.1 Arguments are on the whole coherent, logical and supported with examples. High 2.2 Mid 2.2 Most questions are answered and answered relatively well. Some mistakes are made in terms of written expression, e.g. spelling, grammar. There are also some omissions/mistakes in terms of content, e.g. some theories could be further critiqued, some questions are better addressed than others, examples used may be inappropriate etc. Reasonable knowledge and understanding of concepts. Work is more descriptive than critical. Low 2.2 Answers contain some meaningful synthesis between concepts. 28 Matrix developed by Kamilah Jooganah, Learning & Teaching Officer, CADQ. The matrix should be viewed as an example matrix and not a model matrix. To achieve optimal efficiency the descriptors need to be adapted to module and task specific contexts. There may be cases where the descriptors do not accurately reflect the student s performance, e.g. where a student has a variable performance and receives a High 1st for 2 questions and Mid 2.1 for the 3 questions. In such situations the marker may take a number of different approaches to finding the overall grade. See guidance on multipart assessment. Guidance on multi-part assessment 19

20 Class Scale General Characteristics THIRD (Sufficient) High 3 rd Questions are on the whole answered at a basic level. Some questions not answered/attempted (e.g. only 3 questions attempted and answers are at best descriptive). Some mistakes are made in terms of written expression, e.g. Mid 3 rd spelling, grammar, sentence construction. Noticeably omissions/mistakes are made in terms of content, e.g. some theories are not covered, some questions lack examples etc. Knowledge and understanding is sufficient to deal with terminology, basic facts and concepts. Low 3 rd Answers lack meaningful synthesis between concepts. Arguments made are inadequate/flawed. FAIL (Insufficient) Marginal Fail Mid Fail Low Fail Most questions not answered (e.g. only 2 questions attempted and are answered poorly). Frequent mistakes are made in terms of written expression, e.g. spelling, grammar, sentence construction. There are also major omissions/mistakes in terms of content, e.g. key theories are not covered, questions lack examples etc. Knowledge and understanding is insufficient with only simple connections made between theories. Arguments made are inadequate/flawed. Most questions not attempted. Frequent mistakes are made in terms of written expression, e.g. spelling, grammar, sentence construction. There are also major omissions/mistakes in terms of content, e.g. key theories are not covered, questions lack examples etc. Knowledge and understanding is insufficient and at the word level with fact being reproduced in a disjointed way. Little/no attempt is made to construct an argument. ZERO Zero Work is of no merit OR absent, work not submitted, penalty in some misconduct cases. 20

21 Appendix C: Grading matrix for numeric exam Fundamental Forces Exams Level 6 29 Note: This grid is intended to give you an idea of the type and level of work expected on different types of exam questions/question parts. Obviously, questions vary, so not all points listed will be relevant in all cases. FIRST UPPER SECOND LOWER SECOND THIRD FAIL ZERO Except l 1st High 1st Mid 1st Low 1st High 2.1 Mid 2.1 Low 2.1 High 2.2 Mid 2.2 Low 2.2 High 3 rd Mid 3 rd Low 3 rd Marg l Fail Mid-Fail Low Fail Zero Derivation based questions Complete, logically organized derivations, presented clearly, with all steps signposted and explained. All assumptions made are noted, and carefully justified. Where applicable, evidence of creativity and ingenuity in finding the most efficient route to the final result. Complete, well organized derivations, with all steps clearly presented and explained. All assumptions noted with justification. Complete derivation with at most minor omissions or unnecessary additional steps. Organization and explanation may need improvement. Key assumptions noted, but other assumptions and justifications may be missing. Key steps in derivation present with at most minor algebraic errors, but some links and steps between them may be missed, or unnecessary additional steps added. Some key assumptions may not have been noted. Key steps may be omitted, and/or given with significant algebraic errors. Assumptions may not have been noted. Alternatively, derivation presented is not appropriate to the question. Work of no merit, or absent Numerical calculations Calculations performed accurately and efficiently, with results presented clearly and completely. Where appropriate, units and error bars listed accurately, to an appropriate number of significant figures/decimal places. Calculations performed accurately. Where appropriate, units and error bars are listed accurately, to an appropriate number of significant figures/decimal places. At most minor arithmetic errors, and/or final results may be poorly or incompletely presented. Units and error bars listed where appropriate. Correct approach/method used, but with some algebraic errors and/or final results may be poorly or incompletely presented. Units and error bars may be incorrect or missing, even where appropriate. Incorrect method used, and/or significant algebraic errors present. Final results, units and error bars may be absent. Alternatively, calculations presented are not appropriate to the question. Work of no merit, or absent. Graphs Careful, neat graphs, fully labelled and titled, with appropriate axis ranges and correct extraction of gradients and intercepts where appropriate. Graphs present, fully labelled, with appropriate axis ranges and correct extraction of gradients and Graphs with at most minor errors in labelling. Possibly problems with axis ranges or extraction of gradients and Graphs present, but with incorrect and/or limited labelling, and/or poor choice of axis ranges. Possibly errors in extraction of Graphs with incorrect data points, poor/incorrect labelling and/or poor choice of axis Work of no merit, or absent 29 Matrix developed by Dr Christopher Castleton, Senior Lecturer, Physics and Mathematics. 21

22 intercepts where appropriate. intercepts, where appropriate. gradients and intercepts, where appropriate. ranges. Alternatively, graph presented is not appropriate to the question. Figures and Diagrams Clear, appropriate and comprehensible figures and diagrams, neatly and carefully sketched, fully labelled and titled, with appropriate links to accompanying text. Appropriate figures & diagrams sketched clearly, fully labelled, and with appropriate links to accompanying text. Figures and diagrams sketched, but scope for more clarity and better labelling. Some elements of the system illustrated possibly incorrect or absent. Some links made in accompanying text. Rough sketches presented, but may be unclear and/or incomplete, and may contain errors. Labelling may also be incomplete or inaccurate. Links to text may be weak. Rough sketches present, but either inaccurate and unclear, or may not be appropriate to the question. Work of no merit, or absent Descriptions, explanations and mini-essay type questions Accurate, detailed explanations and descriptions of phenomena, written in a clear, lucid scientific style, using relevant diagrams and figures where appropriate (see above). Covers all issues relevant to the question, with evidence of study beyond the material presented in lectures. Clear, accurate explanations and descriptions, with relevant diagrams and figures where appropriate (see above). Covers the main issues relevant to the question. Covers the key issues relevant to the question, though there may be scope for improved communication and explanation. There may be some minor errors or omissions, or perhaps irrelevant comments, diagrams etc. Largely correct explanations and descriptions, covering at least some key issues relevant to the question. Space for improved communication of ideas. Possible errors, omissions and/or irrelevant comments, diagrams and/or figures. Key points not communicated. Likely to be significant errors, omissions and/or irrelevant comments, diagrams and/or figures. Alternatively, answer provided is not relevant to the question. Work of no merit, or absent Physical analysis & interpretation based questions Detailed, extensive discussion, interpretation and application of the material studied. Evidence of study beyond the material presented in lectures. Evidence of thoughtful, critical analysis leading to a deep understanding of physical laws and their effects. Careful discussion, interpretation and application of the material studied. Evidence of careful, independent analysis and understanding of physical laws and their effects. Discussion and application of most of the material studied. In general good understanding of physical laws demonstrated, though possibly some minor misunderstandings. Some limited discussion and perhaps application of key aspects of the material studied. Basic understanding of most physical laws demonstrated, though with some misunderstandings. At most limited discussion of the material studied. Some significant misunderstandings of physical laws demonstrated. Alternatively, comments and discussion which are not relevant to the question. Work of no merit, or absent 22

23 Appendix D: Aligning grade descriptors to exam marking schemes There may be some assessments where using a marking scheme is more appropriate to using a grading matrix to assess student work. For example, a marking scheme is appropriate when assessing examinations that require students to write answers that are either right/wrong or true/false, that require some kind of calculation and/or a short discursive response. A marking scheme illustrates the number of marks available for each question/part. It is important that the generic grade descriptors inform the assessment design and the marking scheme. To illustrate, a Level 6 (year 3) student that achieves 40% on an exam, after the mark has been converted to the NTU scale, should actually evidence, as appropriate to the exam condition, the knowledge and skills as outlined in the grade descriptor for the Third grade band (Level 6 NQF). It may be the case that a student achieves 60% on a MCQ test, yet after mapping this value to the NTU scale, this comes out as a Third, as the test primarily measures knowledge of a descriptive nature. A separate exam or the same exam would need to include questions that test for higher levels of knowledge. Questions that require students to apply and/or critically evaluate allow them to be given the opportunity to achieve the higher end grades 30. Where the work assessed involves a combination of numeric answers and/or requires short discursive responses from students and long discursive responses, a grading matrix can be used for the latter and a marking scheme for the former. A grading matrix helps to drive the grade/s awarded to the discursive part/s. How this is achieved can vary as an Analytic or Holistic or Evolved Analytic approach can be used (see Guidance on multi-part assessment 31 for an explanation of the different approaches). Yet the same approach should be taken by module teams to ensure consistency in how an assessment task is graded. For the parts that are of a numeric nature and/or require short discursive responses from students, a marking scheme can be used. The mark allocated may be a percentage, but need not be. A mark out of N (e.g. an examination or multiple-choice question test out of 60) is equally permissible. It is imperative that all values are appropriately converted to the NTU scale before being aggregated with the weighted numerical equivalent/s from the grade/s assigned to the discursive part/s. Indeed, a grade mapping table, also known as a grade conversion table, should be made available showing how the marks/percentages relate to the grades on the NTU scale. 30 For further information on designing exam questions, see GBA SharePoint site. Also see pg. 235 in Biggs, J., & Tang, C. (2011). Teaching for quality learning at university: what the student does (4th ed.). Maidenhead: Society for Research into Higher Education and Open University Press. 31 Guidance on multi-part assessment 23

24 Appendix E: Example matrix where grade descriptors are aligned to exam mark distribution SCHOOL OF SCIENCE AND TECHNOLOGY ISYS30221 Examination Assessment Criteria The examination is designed according to NTU s general grade descriptors for NQF Level 6. The general expectations of each grade class are given below for each section of the examination. The overall grade for the element will be determined by both the average grade across the sections and NTU s grade descriptors 32 Class/Grade/Mark range Assessment Criteria First Exceptional First Low Mid High Upper Second Low Mid High Lower Second Low Mid High Third Low Mid High Fail Marginal Fail Low Mid 1-11 Section A Natural Language Processing 33% Weighting Exceptional knowledge and understanding is evidenced through demonstrating requirements of a First AND showing high-levels of knowledge and problem-solving ability (grammatical notation, NLP terminology, NLP difficult issues) and some evidence of independent study (e.g. referring to PoS taggers not lectured on, the Viterbi algorithm for HMMs, parsing techniques) All questions relating to core concepts/facts answered correctly. The responses/solutions offered demonstrate understanding of English grammar and computational linguistics techniques. Sound understanding of the taught grammar and techniques evidenced with an ability to analyse an NLP problem using taught techniques (WSD, anaphora resolution, PP-attachment resolution etc) with accuracy. Although some minor errors on the core concepts, student is able to accurately describe the chosen technique. Understanding of higher order techniques is limited (e.g. how a HMM works). Little if any Broad knowledge and understanding of the main NLP issues and techniques. Student has difficulty in accurately applying chosen NLP technique to a given situation. e.g. unclear application of top-down parsing algorithm. Glib answers given to some questions. Some notational errors. Knowledge of basic concepts of grammar and NLP. Knowledge lopsided towards a specific aspect (e.g. PoS tagging). Some application to problem scenario but limited to basic overview of method. Some errors in approach and terminology. Some ability to describe key aspects of NLP problem solving - but limited to bullet point list, keywords and minor descriptions with little depth and/or ability to apply it meaningfully to one or more aspects of the NLP problem being modelled. Frequent errors and very limited knowledge of NLP terminology. At best, some descriptive information may be present about the chosen NLP technique i.e. incomplete or incoherent lists of parts or aspects of technique. Unable to provide detailed information or apply technique to problem in any meaningful way. Frequent errors. Almost no knowledge of NLP terminology. 32 Matrix developed by Dr Jonathan Tepper, Principal Lecturer, Computing and Technology. There may be instances where the grade descriptors do not accurately reflect a student s performance, such as where they perform well on the harder questions yet poorly on the easier questions. 24

25 Section B Traditional A.I. 33% Weighting Section C Machine Learning ques for Data Mining 33% Weighting Exceptional knowledge and understanding is evidenced through demonstrating requirements of a First AND showing high-levels of knowledge and problem-solving ability (Propositional Logic, Blind and Informed Search algorithms and Role Playing Games) and some evidence of independent study (e.g. referring to Algorithm complexities in AI not lectured on, optimal decisions in Games) Exceptional knowledge and understanding is evidenced through demonstrating requirements of a First AND showing high-levels of accuracy (notation, constraints applied) and some evidence of independent study (e.g. referring to methods published in research literature) All questions relating to core concepts/facts answered correctly. The responses/solutions offered demonstrate understanding of the Turing Test, NPC Games, A* algorithm and Breadth-First Search algorithm. All questions relating to core concepts/facts answered correctly. The responses/solutions offered evidences understanding of higher order principles of data mining and machine learning (e.g. robust approaches to model fitting and selection, understanding limitations of machine learning techniques and articulating solutions) All aspects of serious issues with notation. Sound understanding of the A* algorithm and techniques evidenced with an ability to solve AI logic problems using taught techniques (Propositional /Predicate logic etc) with accuracy. Although some minor errors on the core concepts of search algorithms, student is able to accurately describe any of such techniques. Understanding of Informed search algorithm is limited (e.g. A*). Little if any serious issues with Breadth-First search. Sound understanding of the taught KDD and data mining modelling techniques evidenced with an ability to pre-process data and accurately apply a taught machine learning technique to perform either prediction, classification or clustering as required with accuracy. Although some minor errors on the core concepts, student is able to accurately model main features of the Broad knowledge and understanding of the main machine intelligent issues and techniques. Student has difficulty in accurately applying chosen informed search algorithm to a given situation. e.g. unclear application of the A* algorithm. Glib answers given to some questions. Some optimization errors. Broad knowledge and understanding of KDD and data mining process, and of one or machine learning technique lacks depth. Able to discuss different stages of KDD process, review problem scenario and identify some key data pre-processing steps and machine learning components. Student has difficulty in accurately applying chosen approach e.g. unclear/incomplete Knowledge of basic concepts of search and Propositional Logic. Knowledge lopsided towards a specific aspect (e.g. Turing Test and NPC Games). Some application to problem scenario but limited to basic overview of method. Some errors in approach and terminology. Knowledge of basic concepts relating to KDD, data mining process and a machine learning method or technique. Knowledge lopsided towards a specific aspect (e.g. KDD process, a specific machine learning technique such linear regression). Some application to problem scenario but limited to basic describing components rather complex aspects Some ability to describe key aspects of machine intelligence - but limited to bullet point list, keywords and minor descriptions with little depth and/or ability to apply it meaningfully to one or more aspects of the AI problem being modelled. Frequent errors and very limited knowledge of search intelligence. Some ability to describe key aspects of the KDD process, data mining and/or a machine learning technique - limited to bullet point list, keywords and minor descriptions with little depth and/or ability to apply it meaningfully to one or more aspects of the problem scenario being addressed. Frequent errors. At best, some descriptive information may be present about uninformed search intelligence i.e. incomplete or incoherent lists of parts or aspects of the algorithm. Unable to provide detailed information or apply technique to problem in any meaningful way. Almost no knowledge of Logic in AI. At best, some descriptive information may be present about the chosen KDD or data mining technique i.e. incomplete or incoherent lists of parts or aspects of technique. Unable to provide detailed information or apply technique to problem in any meaningful way. Frequent errors. 25

26 Additional comments: problem scenario modelled competently with student s own insight clear. Little or no issues with accuracy and notation. system/problem by integrating various aspects of chosen technique. Understanding of higher order principles or abstractions is limited (e.g. link between machine learning techniques like linear regression, Decision trees, Naive Bayes, Neural Nets). Little if any serious issues with notation application of Naive Bayes, Linear Regression, Decision Trees and/or Neural Nets; Problem partially solved. Some notion of ordering/prioritisatio n. Glib answers given to some questions. Some notational errors. such as application. Little sense of appropriate ordering or priority (e.g. order of KDD process incorrect). Frequent notational errors. Glib answers given to some questions. 26

27 Grade mapping table for example exam matrix A grade mapping table illustrates the relationship between the marking scheme and the NTU scale. This should be made available to students in advance of an assessment. The grade mapping table below is based on the example exam matrix in Appendix E. Class Grade Marks NTU Grade Percentage Ranges Numerical Equivalent First Exceptional 1 st High 1 st Mid 1 st Low 1 st Upper second High Mid Low Lower second High Mid Low Third High 3 rd Mid 3 rd Low 3 rd Fail Marginal fail Mid Fail Low Fail Zero Zero The marks from each section of the exam need to be converted to a percentage scale and appropriately weighted before they can be aggregated 33. For example, the mark that a student receives on section A of the exam should be converted to the percentage numerical value using the third and fifth columns. Example: Section A 14 marks = 42% The percentage value for section A of the exam is weighted x 42 = (2dp) The same is done for section B and C. 33 It is possible to construct a table for the whole exam (102 marks). The example illustrates a simple linear translation between the marking scheme and the NTU scale. This need not be the case depending on the degree of challenge of the assessment. 27

28 Section B 22 marks = 65% 0.33 X 65 = (2dp) Section C 32 marks = 96% 0.33 x 96 = (2dp) The weighted percentage value from each section is then aggregated to give the overall percentage value for the exam as a whole = = 67 34, High 2.1 The corresponding grade should be written on the student s exam script and recorded into Banner (Banner will convert the grade to the numerical equivalent). If the assessment element is comprised of more than the exam, for example comprised of a piece of coursework in addition to the exam, then only the exam grade should be given to the student and the percentage value (which does not have to be the numerical equivalent) recorded onto a spreadsheet. To find the overall grade for the assessment element, using the example, aggregate the weighted percentage value from the exam (which may or may not be the weighted numerical equivalent) with the weighted numerical equivalent from the coursework Significant rounding should only happen when finding the element grade. Approaches to rounding should be consistent across module and course teams. 35 Guidance on multi-part assessment 28

29 Appendix F: Bloom s taxonomy Learning domains 36 Cognitive Affective Psychomotor knowledge attitude skills 1. Recall data 1. Receive (awareness) 1. Imitation (copy) 2. Understand 2. Respond (react) 2. Manipulation (follow instructions) 3. Apply (use) 3. Value (understand and act) 3. Develop Precision 4. Analyse (structure/elements) 5. Synthesize (create/build) 6. Evaluate (assess, judge in relational terms) 4. Organise personal value system 5. Internalize value system (adopt behaviour) 4. Articulation (combine, integrate related skills) 5. Naturalization (automate, become expert) Cognitive domain cognitive domain level category or 'level' behaviour descriptions examples of activity to be trained, or demonstration and evidence to be measured 'key words' (verbs which describe the activity to be trained or measured at each level) 1 Knowledge recall or recognise information 2 Comprehension understand meaning, restate data in one's own words, interpret, extrapolate, translate 3 Application use or apply knowledge, put theory into practice, use knowledge in response to real circumstances 4 Analysis interpret elements, organizational principles, structure, construction, internal relationships; multiple-choice test, recount facts or statistics, recall a process, rules, definitions; quote law or procedure explain or interpret meaning from a given scenario or statement, suggest treatment, reaction or solution to given problem, create examples or metaphors put a theory into practical effect, demonstrate, solve a problem, manage an activity identify constituent parts and functions of a process or concept, or deconstruct a methodology or process, making qualitative arrange, define, describe, label, list, memorise, recognise, relate, reproduce, select, state explain, reiterate, reword, critique, classify, summarise, illustrate, translate, review, report, discuss, rewrite, estimate, interpret, theorise, paraphrase, reference, example use, apply, discover, manage, execute, solve, produce, implement, construct, change, prepare, conduct, perform, react, respond, role-play analyse, break down, catalogue, compare, quantify, measure, test, examine, experiment, relate, graph, diagram, plot, extrapolate, value, divide 36 Businessballs.com. ( ). Bloom s Taxonomy learning domains. Retrieved August 1, URL: 29

30 quality, reliability of individual components assessment of elements, relationships, values and effects; measure requirements or needs 5 Synthesis (create/build) develop new unique structures, systems, models, approaches, ideas; creative thinking, operations develop plans or procedures, design solutions, integrate methods, resources, ideas, parts; create teams or new approaches, write protocols or contingencies develop, plan, build, create, design, organise, revise, formulate, propose, establish, assemble, integrate, rearrange, modify 6 Evaluation assess effectiveness of whole concepts, in relation to values, outputs, efficacy, viability; critical thinking, strategic comparison and review; judgement relating to external criteria review strategic options or plans in terms of efficacy, return on investment or costeffectiveness, practicability; assess sustainability; perform a SWOT analysis in relation to alternatives; produce a financial justification for a proposition or venture, calculate the effects of a plan or strategy; perform a detailed and costed risk analysis with recommendations and justifications review, justify, assess, present a case for, defend, report on, investigate, direct, appraise, argue, projectmanage 30

31 Appendix G: Grading matrices checklist Checklist Done/Yet to do Learning outcome statements Learning outcome statements are appropriately written. They are specific and clearly articulate what students are expected to know and demonstrate on completing the module. Learning outcome statements have been included in the grading matrix. Assessment tasks Assessment tasks have been chosen that will effectively allow students to demonstrate their progress against the learning outcomes. Assessment criteria The assessment criteria are linked to the learning outcomes and assessment task. The assessment criteria has been included in the grading matrix. Qualitative grade descriptors The qualitative grade descriptors for each grade class has been written using the NTU generic grade descriptors as a guide (retaining elements of the SOLO taxonomy). Getting feedback on and piloting the matrix A pre-grading activity has been conducted with markers on the module to develop a shared understanding of the expected standards and to facilitate consistent application. An activity has been undertaken with students to get feedback on their understanding of the matrix. Amending the matrix Based on the piloting exercise, and feedback from colleagues and students, the matrix has been amended. 31