A revalidation of the SET37 questionnaire for student evaluations of teaching

Transcription

1 Educational Studies Vol. 35, No. 5, December 2009, A revalidation of the SET37 questionnaire for student evaluations of teaching Dimitri Mortelmans and Pieter Spooren* Faculty of Political and Social Sciences, University of Antwerp, Sint Jacobstraat 2, B-2000 Antwerp, Belgium CEDS_A_ sgm / Educational Original Taylor PieterSpooren pieter.spooren@ua.ac.be and & Article Francis (print)/ Studies (online) In this study, the authors report on the validity and reliability of a paper-and-pencil instrument called SET37 used for Student Evaluation of Teaching (SET) in higher education. Using confirmatory factor analysis on 2525 questionnaires, a revalidation of the SET37 shows construct and discriminant validity of the 12 dimensions included in the instrument. The retest of the instrument reveals (again) the existence of a second-order factor explaining a substantial amount of the variance in seven dimensions of the instrument. In sum, the results provide strong support for both the relevance of the questionnaire and the hypothesis that the SET37 has a multidimensional structure but is compatible with a general underlying factor called Teacher professionalism. Keywords: student evaluation of teaching; teacher evaluation; validity studies; teaching quality; higher education Introduction Going through the great amount of literature concerning the use and the validity of Student Evaluation of Teaching (SET), four main research topics can be distinguished. First, much attention goes to the reliability of student opinions when mapping educational quality (for extensive reviews on this topic, see among others Marsh 1987). The second question deals with the use of SET. For instance, should students be considered (capable of) being full-fledged stakeholders in the evaluation procedure? Third, does student feedback lead to the improvement of instruction (see among others Kember, Leung, and Kwan 2002)? Fourth, is it possible to develop sound instruments for SET? With respect to the latter question, it must be said that many instruments lack a theoretical foundation and/or information on the validation tests (including the results) these instruments were put through. Taking into account that SET are not only used in a formative way (e.g. What can I do to improve my teaching? ), but on the contrary in many college campuses form part of personnel decisions (e.g. tenure and promotion decisions), it is our opinion that SET instruments should meet a panoply of requirements such as reliability, construct validity, convergent validity etc. On top of that, frequent revalidation procedures on new datasets are desirable since (for instance) educational principles change over time (e.g. from classic lectures towards more active and student-centred education). *Corresponding author. pieter.spooren@ua.ac.be ISSN print/issn online 2009 Taylor & Francis DOI: /

2 548 D. Mortelmans and P. Spooren Objectives In this contribution, we report on a revalidation procedure of our SET37 questionnaire we designed at the University of Antwerp (Belgium) based on both educational theory and empirical testing (Spooren, Mortelmans, and Denekens 2007). The basic idea behind the instrument is that teaching skills should be considered latent constructs and thus cannot be measured by means of a single-item approach. Therefore, we decided to develop a Likert scale-based instrument that consists of item sets relevant to measuring student s attitudes towards several dimensions of teaching (e.g. presentation skills, clarity of course objectives, teacher s help during the learning process, etc.) instead of capturing a particular quality (dimension) through one item. On top of that, the Likert-type scale is very useable and provides the possibility of a quality check on the results by means of the Cronbach s alpha statistic on each dimension every time a course is evaluated by the students. Finally, reliability and validity of these multi-item scales can be extensively tested. The study resulted in a 31-item questionnaire, representing 10 dimensions of teaching. In the meantime, two of these 10 scales were revised and two additional scales were introduced, using the same methodology resulting in the validated 12-scale instrument SET37. Next to this validation study, we looked into the existence of a higher order factor below the various teaching dimensions in the instrument (Spooren and Mortelmans 2006). Such a halo-factor might lead to incorrect interpretations of evaluation results. On the other hand, it is shown that, although student ratings are considered to be multidimensional, students give similar ratings across a lot of evaluation items. In other words, it seems that student ratings have a multidimensional structure but are compatible with a very strong general underlying factor (Apodaca and Grad 2005). We termed this higher order factor Teacher professionalism. Results Instrument We use the same SET37-instrument as in the original validation study (Spooren, Mortelmans, and Denekens 2007), representing 12 balanced scales. All 37-Likert items were used in a paper-and-pencil interview in the students classroom. All items were measured on a six-point scale (going from strongly disagree to strongly agree ). Participants Student evaluation forms were administered during the spring semester of the academic year. A total of 2525 evaluation forms completed by 620 students, enrolled in 118 courses at a medium-sized multidisciplinary university (12,000 students), were analysed in the present study. Evaluations were completed in various educational programmes by both graduate and undergraduate students. Revalidation Confirmatory factor analysis is the best technique to extensively test the reliability and validity of the different scales in the instrument. The results of the present study are summarised in Table 1. First, the analysis (for which we used the Lisrel 8.53 program)

3 Educational Studies 549 Table 1. Revalidation of the evaluation instrument: confirmatory factor analysis. Standardised factor loadings t-test Probability Variance t-test R 2 extracted F1: Clarity of course objectives 0.75 (ρ c = 0.90) Item Item Item F2: Value of subject matter 0.64 (ρ c = 0.84) Item Item Item F3: Build-up of subject matter 0.69 (ρ c = 0.87) Item Item Item F4: Presentation skills (ρ c = 0.94) 0.84 Item Item Item F5: Harmony organisation 0.66 course-learning (ρ c = 0.85) Item Item Item F6: Course materials 0.79 (contribution to understanding the subject matter) (ρ c = 0.94) Item Item Item Item F7: Course difficulty (ρ c = 0.92) 0.80 Item Item Item F8: Help of the teacher during the 0.80 learning process (ρ c = 0.92) Item Item Item

4 550 D. Mortelmans and P. Spooren Table 1. (Continued). Standardised factor loadings t-test Probability Variance t-test R 2 extracted F9: Authenticity of the examination(s) (ρ c = 0.80) Item Item Item F10: Linking-up with foreknowledge (ρ c = 0.90) Item Item Item F11: Content validity of the examination(s) (ρ c = 0.90) Item Item Item F12: Formative evaluation(s) (ρ c = 0.86) Item Item Item Note: For an overview of some items (1 per scale), see Appendix 1. shows a fair fit to the data (RMSEA = 0.04; GFI = 0.98; CFI = 0.96; NNFI = 0.96) although the χ 2 test of exact fit is significant whereas the objective is to achieve a nonsignificant p-value (χ 2 (563) = , p <.05). However, Hatcher (1994) indicates that a significant χ 2 does not make a confirmatory factor analysis model inadequate. Convergence validity of the constructs is shown by the large (between 0.63 and 0.93) and significant factor loadings of the items on their posited indicators. On top of that, none of the correlations between the constructs is high enough to challenge convergent validity (all correlations lower than 0.90). Discriminant validity is evidenced by means of the confidence interval test (the 95% confidence interval around the correlations never equals one) and the χ 2 test (all models wherein the correlation between two constructs was set to zero showed a worse fit to the data). The variance extracted test also demonstrates discriminant validity of the constructs since (with a few exceptions) the explained variance of two constructs is greater than their squared correlation. Finally, internal consistency of all scales is shown because the composite reliability (ρ c in Table 1) is higher than 0.80 every time Second-order factor As we mentioned earlier, there might exist a (unidimensional) second-order factor behind various (multiple) dimensions in SET results. For instance, the general impression students have towards a teacher and his or her course might influence SET scores

5 Educational Studies 551 on dimensions which are considered free-standing at first sight. Since we found such a halo-factor in an earlier study on this instrument, we decided to put it to the test while revalidating the questionnaire. Confirmatory factor analysis, wherein a secondorder factor was assumed behind the same seven scales (namely Factors 1 4 and 6 8 in Table 1) as in the previous study (Spooren and Mortelmans 2006), shows a good fit to the data (RMSEA = 0.05; GFI = 0.98; CFI = 0.96; NNFI = 0.96). Item loadings (e.g. first-factor loadings) on the second-order factor are high (between 0.82 and 0.95), and 82% of the variance in the scales is explained by this halo factor. These results thus again show the existence of a general underlying factor in the SET37 scores. This offers a promising perspective towards the use of SET37 in both a formative and a summative way. When using the SET37 as an instrument for feedback, one could use the results on one or more particular dimensions when working on the improvement of (teaching) a course. On the other hand, an overall score derived from the (weighted if necessary) SET37 scores on dimensions of which we know they belong to a (valid and reliable) general factor, could be used for the evaluation of teaching staff. At least, when sound answers are found considering other SET-related research topics. Notes on contributors Dimitri Mortelmans is an associate professor in sociology at the Faculty of Political and Social Sciences of the University of Antwerp, Belgium. He teaches qualitative research methods, multivariate statistics and advanced study of populations, families and the life course. He is head of the Research Centre for Longitudinal and Life Course Studies (CELLO) and also head of the Innovation and Quality of Education Centre. His main research topics cover divorce, work-life balance and leisure time of youngsters. Pieter Spooren is affiliated as an educational advisor at the Faculty of Political and Social Sciences of the University of Antwerp, Belgium. His particular activities are educational innovation and evaluation of the educational process and of educators. His main research interests focus on student evaluation of teaching (SET), in particular their use and validity. References Apodaca, P., and H. Grad The dimensionality of student ratings of teaching: Integration of uni- and multi-dimensional models. Studies in Higher Education 30: Hatcher, L A step-by-step approach to using the SAS system for factor analysis and structural equation modelling. Cary, NC: SAS Institute. Kember, D., D. Leung, and K. Kwan Does the use of student feedback questionnaires improve the overall quality of teaching? Assessment & Evaluation in Higher Education 27: Marsh, H.W Student s evaluations of university teaching: Research findings, methodological issues, and directions for further research. International Journal of Educational Research 11: Spooren, P., and D. Mortelmans Teacher professionalism and student evaluation of teaching: Will better teachers receive higher ratings and will better students give higher ratings? Educational Studies 32: Spooren, P., D. Mortelmans, and J. Denekens Student evaluation of teaching quality in higher education: Development of an instrument based on 10 Likert scales. Assessment & Evaluation in Higher Education 32:

6 552 D. Mortelmans and P. Spooren Appendix 1. Overview of some items included in the questionnaire (1 per scale) Dimension Item number Description Clarity of course objectives 0103 The information presented by the lecturer at the start of the course clearly specified what I should learn and accomplish Value of subject matter 0202 Some topics covered in this course are, in my opinion, completely redundant (REVERSED) Build-up of subject matter 0302 The different topics covered in this course were completely unrelated (REVERSED) Presentation skills 0401 The lecturer explained the material well Harmony organisation course-learning 0503 The lecturer sometimes presented us with an assignment which forced me to think critically Course materials (contribution to understanding the subject matter) 0602 The study material was not appealing to study (REVERSED) Course difficulty 0702 The level of difficulty of this course was acceptable Help of teacher during learning process 0801 The lecturer helped me with questions and problems which arose during this course Authenticity of the examination(s) 0903 The evaluation procedure consisted of more than plainly reproducing the contents of the course Linking-up with foreknowledge 1001 The contents of this course did not connect with anything we already knew or learned (REVERSED) 1103 The examination was a good reflection of the contents of this course Content validity of examination(s) Formative evaluation(s) 1203 This lecturer did not give any performance feedback throughout the course Note: Translated from Dutch.