Functional Education in Statistics and Related Disciplines: Innovative and Reflexive Learning, Teaching and Assessment of Business Intelligence

Transcription

1 Functional Education in Statistics and Related Disciplines: Innovative and Reflexive Learning, Teaching and Assessment of Business Intelligence Patrick Ezepue Kassim Mwitondi Business Intelligence and Quantitative Modelling Research Group, Communication and Computing Research Centre, Faculty of Arts, Computing, Engineering and Sciences, Sheffield Hallam University, City Campus, Howard Street, Sheffield S1 1WB, United Kingdom Abstract In delivering a module we want students to become grounded in the core concepts and applications of the module. Particularly, we would want students to have learnt how to continue learning about and critically evaluate situations and problems using the knowledge gained from the module. Based on these images of a perceptive student, this project investigates the background situations and learning experiences of students taking the Business Intelligence module, in the Applied Computing Programme of a university faculty. We wish to ascertain whether or not there are statistically detectable differences in abilities and learning characteristics of students involved in the module. The aim is to use insights from the project to introduce progressive innovations in teaching the module and other modules in Information and Computer Studies. How do we achieve these objectives? Section 3 of the paper qualitatively characterises an integrally excellent higher education graduate and explores how close the Business Intelligence students come to this ideal. This exploration develops images of a perceptive critical thinking and functionally educated ICS graduate. Sections 5 and 6 of the paper identify areas of module delivery and understanding to improve upon via the learning, teaching and assessment process such as lecture notes and lesson designs, embedding critical reflection into the module to facilitate reflexive education and active skills development. Insights underpinning these innovations are obtained from a number of sources including assessment scores, a diary of learning experiences in the module and a survey of students' learning experiences. Key words: Students' Learning Experiences Mixed Ability Teaching, Module Innovations and Pedagogical Challenges 1

2 1. Introduction This paper reports some findings from a project sponsored by the Teaching Quality Enhancement Fund in UK to audit final year students' learning in a Business Intelligence module delivered by the authors. The aim is to use insights from the project to introduce progressive innovations in teaching the module and other modules in the Applied Computing Programme. The project methodology is bifocal. Firstly, it qualitatively characterises an integrally excellent graduate from the programme and explores how close the Business Intelligence students come to this ideal. Secondly, it statistically analyses students' performance records against a number of personal attributes and module learning criteria e.g. gender, degree programme being studied, and distinct aspects of the module such as business understanding, data understanding, data preparation and quality and specific data mining tools. If there are significant differences among groups of students, then module delivery will have to employ techniques of mixed ability teaching. If there are no such differences but rather clear areas of underachievement of the module learning outcomes, then insights from the project results will be used to develop strategies to improve students' performance on the outcomes. Information used in the project includes: insights from educational learning principles, curriculum design; experience mapping based on a diary of student-staff module experiences; students' assessment scores; and end-of-module evaluation. The results would be used to identify areas for improvement in the module delivery. Such innovations in module delivery will focus on five key LTA areas of vital importance to students' learning - planning, assessment, student support, teaching and skills development. This paper summarises the key findings from the project and highlights the main steps in translating the findings into pedagogical practice. The rest of the paper is as follows. Section 2 reviews the background literature on learning, teaching and assessment with a focus on effective learning styles, pedagogical innovations and mixed ability teaching. The review draws together ideas that could be used to improve curriculum design and delivery in order to enhance students' achievement of the module learning outcomes. Section 3 explores desirable images of a perceptive learner who critically understands and can apply the learning achieved in a module to real-life situations. Section 4 discusses the methodology of the project and Section 5 describes and analyses the data collected. The implications of these results for module improvement are discussed in this section. Section 6 concludes the paper. 2. Theoretical Background Learning is about how we perceive and understand the world, about making meaning, Matron and Booth (1997). It can be about principles, factual information, techniques and approaches, ideas, behaviours appropriate to types of situations, recognition and reasoning. Student learning in a module is a function of the discipline involved, the level and presentation of module material, student backgrounds and expectations, their motivation and assessment, among other mediating factors. It is important to understudy the affect of these combinations of factors on the learning going on in a particular module. Such knowledge is a crucial input into curriculum design and delivery plans. Effective learning should be constructive as it enables learners to create their own meaning, constructs and conceptions of reality, Fry et al (1999, p. 21). The aim is to shift the balance in learning from regurgitation of factual information to deep understanding of the fundamental concepts involved and their application to reality. Most of the current notions of learning, including experiential learning, the use of reflection, and so on, are based on constructivism, Kolb (1984). Effective learning should happen at four main domains - cognitive (dealing theory and abstraction), affective (involving feelings and deep experience), interpersonal (involving paired and group work and communication processes) and psychomotor (involving practical skills and techniques). The Business Intelligence module is designed to engage students on these four domains. There is a onehour weekly lecture followed by a one-hour practical laboratory tutorial using data mining software to practise the theoretical topics, concepts, approaches and models covered in the lecture. Students are encouraged to work in pairs during tutorials. There are three pieces of assessment in the 2

3 module. A Block 1 Assignment carried out individually by each student covers the first segment of module work on the key concepts in Business Intelligence, and first three stages of a standard process for data mining e.g. business understanding, data understanding and data preparation before the application of specific data mining models. A Block 2 Assignment assesses student knowledge of specific techniques of data mining in a group work, and an end-of-module examination assesses overall learning in the module. Experiential learning draws from experiences in a learner's life-world, discipline-world and work-world. This learning is encapsulated in the Kolb's CE-RE-AC-AE cycle of concrete experience, reflexive evaluation, abstract conceptualisation, and active experimentation. First, students are involved fully and freely in new experiences of business and organisational decisionmaking based on massive datasets e.g. supermarket customer point of sale data, bank data on customers, and educational data involving large number of pupils (CE). Students are also tasked in tutorials to review websites on business intelligence and note key areas of further applications. Second, students are asked to reflect critically on their learning, using guides drawn up from the eight elements and seven standards of critical reasoning discussed in Nosich (2005). These elements and standards are discussed in detail in Ezepue and Chigbu (2006), a paper presented as background material in the Workshop on Functional Education in Statistics and Related Fields in this conference, and to which this paper belongs. Third, students learn to form new concepts and process their ideas into logical theories (AC). Finally, students learn to actively use theories and models of data mining to solve real problems, and make decisions that potentially add value to organisations from which the data emanate (AE). Hence, the Business Intelligence module design is such that students are offered opportunities for deep learning by accessing these four kinds of abilities. The lectures explore abstract concepts, which students are to apply in the tutorials; the tutorials are developed to test student ability to work on real data, formulate meaningful problems and solutions on the bases of an active engagement with key stages of data analysis, and reflect on their findings. The three assessments test student achievement of these skills, and is the subject of the analyses in this paper. Related to experiential learning is Honey and Mumford (1982)'s four-fold categorisation of learning styles - activist, pragmatist, reflector and theorist. Activists respond most positively to learning experiences that challenge them with new experiences, problems, excitement and freedom in their learning. See Ezepue (2006), which explores approaches to active learning using problembased learning and case studies in statistical modelling. Reflectors respond most positively to structured learning which gives them time to think and reflect on their learning. Theorists respond well to logical structure and clear aims. This objective is met in the lectures which start with clear statements of objectives and develop the objectives logically. Pragmatists appreciate practicallybased work that uses theory. This is offered in tutorials, block assignments and case study. 3

4 3. Images of a Perceptive and Reflexive Learner Following Nosich (2005) we visually represent in Figure 3.1 below the dimensions of critical thinking that students could be encouraged to master as part of their learning in a module. Fig. 3.1: A Model for Critical Thinking in a Module A VISUAL REPRESENTATION OF OF A PERCEPTIVE GRADUATE ELEMENTS STANDARDS THE MODULE DECISION MAKING APPLICATION EVALUATION, COMPARISON AND CONTRASTING In this representation, students pass a problem situation (S) through the elements of critical thinking and evaluate their results using a set of standards of reasoning, ensuring they use knowledge gained from the module, in this case Business Intelligence. The aim is to get them to make effective decisions on the bases of these discipline-focused reflections. These decisions will be grounded in appropriate application of the knowledge gained from the module, the results of which should be critically evaluated, compared and contrasted. For instance, students should be able to model data from the standpoint of different plausible models and compare the results for business sense, understanding and ease of implementation. The following notes taken from the Module Study Guide explain how the students are guided on this process of critical thinking in the module. The 8 elements and 7 standards of critical reasoning which will form the bases of your review of learning are as follows: The Elements of Critical Reasoning 1. Point of view 2. Purpose 3. Questions at Issue 4. Assumptions 5. Implications and Consequences 6. Information 7. Concepts 8. Conclusions and Interpretations. 4

5 These elements are discussed extensively in Nosich (2005) as useful ways to think through a discipline. You should use the elements to ask questions of the lectures, tutorials and assessments in the module. Reading module notes with questions makes your study active and reflexive. Examples of intelligent questioning based on the elements of reasoning are: What is the point of view of the tutor and what are the main points of a lecture or tutorial? What have I learnt from a topic? How has my work in a Study Theme enabled me to achieve the learning outcomes associated with the theme? What is the purpose of a lecture? What is the key purpose of the module? Do I understand the central logic of the module? When I'm given a task, tutorial assignment or block assessment, do I understand the questions at issue i.e. what I'm required to do? What assumptions underlie the techniques used in Data Mining? What assumptions do I make in the way I attempt a business intelligence exercise? What are the implications and consequences of my results for the particular problem I'm solving? What sources and kinds of information are needed for a BI project or an assignment? What information does an examination or module assessment require me to use and report? Can I recall and express in my own words the key concepts covered in a lecture, study theme or the whole module? Based on my understanding of the key concepts in a lecture, a study theme or the module, can I summarise the learning involved in each of these in a concept map? What conclusions and interpretations can I develop from the results of a BI investigation? A review of learning using the elements, as reflected in the above questions, must consider the context of an application e.g. what a client is mainly interested in, what practical constraints on the solution exist as a result of the context. Also your review of BI work, and generally any discipline, should consider alternative ways of doing things. Hence, as you reflect on your learning of the module topic using the elements of critical reasoning, you should also think about the context of an application and the alternatives to a problem which the context makes possible. The Standards of Critical Reasoning The standards by which we assess how well we reflect on our learning based on the elements of reasoning are as follows: 1. Clearness 2. Accuracy 3. Importance and Relevance 4. Sufficiency 5. Depth 6. Breadth 7. Precision. It is good practice to use the elements and standards in combination for a better assessment of your progress in the module. You may want your learning to be such that you achieve the above standards as may be appropriate for each of the elements. For example, you may ask the following questions about your learning: Do I clearly understand the point of view and purpose of a task or an assignment? How clear is my report on assessments? That is, do I write clearly? Is my data analysis accurate enough for the purposes of an investigation? Do I consider the relevance of my analyses to the objectives of an investigation? Hence, have I emphasised results that are important and relevant in my report? Is my understanding of a topic of sufficient depth and breadth to enable me explain the core concepts, apply them in practical problems and recall them in module assessments? 5

6 Putting It All Together At the end of lectures and tutorials you are expected to reflect critically on your learning as follows. 1. Take a moment to write down three key points you have learned from the week's work. 2. Use a concept map to identify the core concepts introduced in the lecture and tutorial work. Explain the concepts in your own words. 3. Using the elements and standards of critical reasoning as prompts, check how well you understood the purpose of the lecture and to what extent you have achieved the learning objectives. Examine the particular module learning outcomes that the learning objectives relate to. Hence reflect on your achievement of the learning outcomes. 4. Note down points you're not fully clear about and be prepared to seek further explanations on them from the module tutors and fellow students. The module tutors will from time to time require you to formally write down such reflections on some module tasks and assessments. It is therefore helpful if you regularly reflect on your learning informally as suggested. You should keep a note of your work on module tasks and your reflections in a progress file. The files will be collected and used to assess how actively you participate in module work. This assessment will form a part of the Block 1 and 2 coursework assignments. 4. Methodology The project was implemented with a combination of insights from educational learning principles, curriculum design and delivery, mapping the LTA experience of students and staff within the session, and active engagement with wider LTA issues and policy backdrops. The research typology was action research, with a mixture of qualitative assessment of students' learning preferences, problems and successes, and hard data on their performances in module assignments. The following actions provided the data for the project. Keeping a diary of students' and tutor LTA experiences on the module throughout the duration of the session. Researching into the theoretical underpinnings of the project in relevant literature, especially as regards pedagogic approaches to mixed ability teaching, improving the learning environment, reflexive learning, student support, curriculum planning, lesson design, and assessments. Conducting a formal student experience survey to evaluate effectiveness of the module delivery for learning, teaching and assessment. Carrying out basic statistical analysis of the project data for the purpose of using resulting insights in pedagogic innovations. The methodology consists in assembling and analysing data and insights from the above sources with a focus on learning, teaching and assessment. The analysis combines qualitative and quantitative approaches. 6

7 5. Data Description and Analysis In this section we present and describe the numeric and non-numeric data drawn from the above sources Diary of students' and tutor LTA experiences on the module A diary of LTA experiences of students and module lecturers was set up with rich notes that document insights from ongoing learning encounters in the module. The encounters were through student discussions in the e-learning forum for the module, s sent by students requesting further help with some aspects of the learning materials or direct observation of students' progress learning during tutorials. We summarise below the following insights from the diary and their implications for improving student learning experiences. The insights will be compared with results from an end-of-year questionnaire survey of students' learning experiences in the module. This will help to map out areas for further improvements in module development and delivery. Summary of insights from the diary 1. Participant observation of student attitude to and understanding of specific topics showed a marked difference between students taking a related Data Mining module and those taking the Business Intelligence module. This is attributed to possible differences in their mathematical backgrounds, since the Data Mining students were doing a Computing and Statistics degree, while the Business Intelligence students were admitted into various courses in applied computing e.g. Business Information Technology, Business Information Technology (Software and Network) options and Applied Computing. The implication of this fact for module development is that there is a need for module notes and delivery to take into consideration differing learner-subject matter interactions and previous mathematical backgrounds. For instance, whilst one could be more mathematical in writing materials for Data Mining offered to computing and statistics students, the safest way to teach the Business Intelligence students is as follows: a) adopt an intuitive and verbally descriptive approach to explaining the core mathematical concepts and illustrating software generated results with graphical displays and schematisations b) place less emphasis on the mathematical foundations without sacrificing practical understanding c) consider giving separate mathematics clinics on the mathematical concepts and d) use simple numerical examples to demonstrate the application of formulae. 2. In the tutorials on the topic of logistic regression a table of workings was used to explain the η η equation p (x) = e [1 + e ] ; η = β0 + β1age. Students declared that while in the lecture the logistic equation sounded 'Greek' to them, the numerical example showed them that the meaning of the model parameters and the fact that the equation required them to make p (x) subject of the formula in the logistic model ln[ p( x) (1 p( x) ] = η. This illustrates the importance of the pedagogical lessons noted in 2 above. The idea is to use small-scale examples already solved down to demonstrate the kind calculations involved in obtaining p (x) from the regression functional η = β 0 + β1x + ε i, with x = age in this example. Following this the software results will be explained in detail. To reiterate earlier remarks, it is important to think about how to include necessary mathematical notations in the module lecture notes and still enable students to intuitively understand the gist of the topic. It is also useful to make some tutorials lag lectures on technical topics by a week or a number of days; this enables students to read the notes and familiarise themselves with the meaning of concepts, equations and parameters before the tutorials. 3. 'My question is this, to what extent am I supposed to understand the CRISP-DM model (the cross-industry process for data mining)? It's quite a substantial document all in all (78pages) so would I be better off spending my time forming a general overview of CRISP-DM or 7

8 specialising in one or more areas of it?' This note from a student shows that students need more guidance on how to understand the CRISP-DM framework, which underpins the module. The following reply from the tutor team points out the nature of such guides for teaching the process in future. 'You'll notice that the CRISP-DM process, as a hugely comprehensive data mining methodology, covers a substantial amount of areas - phases, tasks, outputs and reports, etc. But in actual applications, one only needs a firm working understanding of aspects of the document that will relate to a particular investigation. Depending on the scope of the application, not all aspects of the process will feature in any one application. It's the responsibility of the analyst to determine what to look for in an investigation or what is asked for in a client's project brief, and consequently what modelling tools to use, how to deploy the tools and what outputs and reports need to be produced'. 4. Finally, attention should be given to student support, including care in setting coursework assignments and examinations, and especially in the formative feedback given to students from the assessments. For example, a student noted that 'the feedbacks that was written into the returned assignments were full, considered and helpful' and 'the exam was well thought out and excellent'. Another student ed to thank the tutors 'for injecting enthusiasm into the module and for being friendly, helpful and approachable throughout' Questionnaire survey of students' learning experience A questionnaire was administered to students in order to evaluate their learning experiences on the module. Students were asked to rate each of the following aspects of the module by CIRCLING only one number from 1 (Very Dissatisfied) through 5 (Very Satisfied). The average scores are obtained and ranked in descending order as in Table 5.1 below. Table 5.1 Questionnaire Responses Ranked by Average Scores Item Description Average score 4 Time allowed for assignments The quality of feedback from Block 1 assessment The quality of feedback from the Case Study The helpfulness of module tutors in general The extent to which the assignments improve your learning The stimulation from working with other students The stimulation from Block 1 assessment Your understanding of the purpose of the module The stimulation from Case Study How the subject matter relates to other modules The support and guidance for the examination The balance of lectures and tutorials The relevance of the module to your future career The nature of the examination Your self organisation/time management of the module work Your achievement of the learning outcomes Time spent on lectures Time spent on tutorials and the organisation of learning How the learning from the module relates to your experience Teaching standards appropriate for your background Your ability to read the notes critically and make sense of them The structure and language of the lecture notes Quality of learning materials (lectures and tutorials) Fit between your mathematical background and module Guidance as to the mathematical prerequisites

9 This shows that improvements in module delivery should be directed mostly at the following aspects: Aligning teaching standards to students' background Improving the structure and language of the lecture notes to make them simpler and more readable Continuing to enhance the quality of learning materials (lectures and tutorials) Giving more attention to the mathematical foundations of the module. This means that the mathematical aspects of the topics should be explained intuitively and practically; the notes should be written more clearly and in simpler language appropriate to the level of the students. These facts are generally in line with the insights obtained from the diary of students' learning experiences discussed in Section 5.1 above Further Statistical Analysis of the Project Data The dataset used for analysis is same as appears a related paper, Mwitondi and Ezepue (2006). The two papers are part of the workshop on functional education in statistics, mathematics and related fields presented in this conference. As described in Mwitondi and Ezepue (2006) the data consists of 84 student records on 13 variables, see Table 5.2 below. The numeric variables are percentage scores in a number of class tests, with the variables carefully chosen to measure different aspects of learning capabilities. VARIABLE GENDER JOINEDFROM PREVIOUSCOURSE KEYCONCEPTS ANALYSIS DATAQUALITY DATAMODELLING ASSESSMENT INSIGHT CRISPDM STYLE XSECTIONA XSECTIONB DESCRIPTION Student's gender Institution from which the student comes Course pursued prior to joining our BSc (honours) programme. Measures the student's understanding of the underlying key concepts of business intelligence. Looks at the student's analytical capability. In particular, we look at their choice and combination of variables as well as their interpretation of outputs. Students are assessed according to their understanding of data quality. Measures the student's ability to apply data modelling algorithms to a business risk management problem. It carries 60% of the total marks for the case study. Measures the student's ability to assess and evaluate model outputs. Measures the quality of the student's recommendation to senior management and/or the client. Measures the student's understanding of the whole data mining process. Assesses the quality of the produced report. Measures the student's general understanding of both Business Intelligence and Statistics Measures the student's understanding of the specific Business intelligence Techniques. Table 5.2: The table provides a description of the variables used in data analysis. 9

10 Table 5.3: Correlations of the Data Variables KEYC ONCE PTS ANAL YSIS DATAQU ALITY KEYCONCEPTS 1 DATAMODE LLING ANALYSIS DATAQUALIT Y DATAMODELL ING ASSESS MENT ASSESSMENT INSIGHT INSIG HT CRISPDM CRISP DM STYLE XSECTIONA STY LE XSECTI ONA XSECTIONB XSECTI ONB Table 5.2 shows that the variables are generally weakly correlated, apart from assessment and datamodelling (with correlation coefficient 0.72) and CRISPDM and assessment (with correlation coefficient 0.60). Hence, in general, the variables measure independent dimensions of module performance. The summary statistics for the scores are as presented in Table 5.4 below. Table 5.4: Summary Statistics Variable Mean Median Mode Min Max Std Dev CV KEYCONCEPTS ANALYSIS DATAQUALITY DATAMODELLING ASSESSMENT INSIGHT CRISPDM STYLE XSECTIONA XSECTIONB The results show that the students perform relatively poorly in datamodelling (ability to apply data modelling algorithms to a practical problem in risk management), CRISPDM (which measures student understanding of the data mining process) and XSECTIONB (which measures students' understanding of specific BI techniques). However, a look at the maximum scores of approximately 78%, 80% and 98% in these attributes show that some students attained distinctionlevel performances in the attributes. The comparison of the three measures of average performance (mean, median and mode) across all attributes shows a general normal behaviour in the scores. Hence, the two assignments and examination are indicative of a balanced and fair assessment of key attributes of students' understanding and application knowledge gained from the module. The coefficient of variation is too large for the variables XSECTIONB, XSECTIONA and DATAQUALITY (the XSECTIONB variable being more than two times in relative variability compared to the other two variables). This shows that for these performance attributes there is an unusually wide gap between the weak and bright students. Module improvement should be targeted at giving students more detailed explanation of the topics tested under these attributes. For example, developing and explaining specific business intelligence techniques as implied by the XSECTIONB variable, teaching the basic statistical concepts underpinning business intelligence in a bit more detail as implied by the XSECTIONA variable, and giving students more tutorial practice on data understanding and data preparation including assessment of data quality and handling missing values. 10

11 The data analysis here is basic and aims to identify areas to focus in improving the delivery of the module. Principal component and data clustering analyses were performed on the dataset in Mwitondi and Ezepue (2006). The results show that students in three main clusters performed similarly along the ten performance attributes, the differences between the scores for the different clusters are clearly within one standard deviation about the mean, using the standard deviations obtained in Table 5.4 above. 6. Summary of outcomes and implications for module enhancement Examination of the students' performances and experiences on the module suggest an overlap of learning experiences and abilities among the various clusters of students with different background qualifications. However, there are areas of strengths and weaknesses such as students' approach to sense-based questions, depth of critical understanding of business needs and technical data mining results obtained from software. There were also some weaknesses in the part of the examinations which test knowledge of technical modelling techniques e.g. regression analysis, cluster analysis, neural networks and decision trees. These facts suggest that the way to usefully apply the project results to module work in the current session is to consider approaches to mixed ability teaching that address particular weaknesses and generally improve the students' learning experience. These areas of weaknesses are identified in Sections 5 above. The paper has illustrated an approach to embed critical thinking skills in the module. These skills aim to make the students functionally educated by being able to reason through the module and apply the knowledge gained to real-life situations. These improvements are currently being implemented in this year's cohort of students taking the module. 11

12 References [1] EZEPUE P O (2006) Conversations in Applied Statistical Modelling - Part 1 (Stochastic Models for Operations and Profitability Assessments in Barbershops and Part 2 (Towards a Case- Driven Pedagogy for Statistical Modelling and Consulting), Proceedings of the 2006 Hawaii International Conference on Statistics, Mathematics and Related Fields, Honolulu, Hawaii, USA, January [2] EZEPUE P O and CHIGBU P E (2006) Intellectualising Functional Education: Implications for Teaching Statistics, Mathematics and Related Fields, paper presented as part of a Workshop on Functional Education in Statistics, Mathematics and Related Fields with Emphasis on Developing Economies, Proceedings of the 2006 Hawaii International Conference on Statistics, Mathematics and Related Fields, Honolulu, Hawaii, USA, January [3] FRY H, KETTERIDGE S and MRSHALL S (1999) A Handbook for Teaching and Learning in Higher Education, Kogan Page [4] HONEY P and MUMFORD A (1982) The Manual of Learning Styles, Peter Honey, Maidenhead [5] KOLB D A (1984) Experiential Learning, Prentice-Hall [6] MARTON F and BOOTH S (1997) Learning and Awareness, Lawrence Erlbaum Associates, New Jersey [7] MWITONDI K and EZEPUE P O (2006) Searching for clusters in data: implications for teaching Business Intelligence, paper presented as part of a Workshop on Functional Education in Statistics, Mathematics and Related Fields with Emphasis on Developing Economies, Proceedings of the 2006 Hawaii International Conference on Statistics, Mathematics and Related Fields, Honolulu, Hawaii, USA, January [8] NOSICH G M (2005) Learning to Think Things Through: A Guide to Critical Thinking Across the Curriculum, Second Edition, Pearson Prentice Hall 12