A Decision Support System for the Assessment of Higher Education Degrees in Portugal

Transcription

1 A Decision Support System for the Assessment of Higher Education Degrees in Portugal José Paulo Santos, José Fernando Oliveira, Maria Antónia Carravilla, Carlos Costa Faculty of Engineering of the University of Porto (FEUP) Portugal Abstract The Foundation of Portuguese Universities (FUP) is a private institution of public interest, created in 1993 and subscribed both by Public Universities and the Catholic Portuguese University, all of them members of the Council of the Portuguese Universities Rectors (CRUP). The assessment of higher education degrees in Portugal is coordinated by FUP. The degrees are grouped by scientific areas and for each area an evaluation panel is designated. The evaluation panels are composed by representatives of the Portuguese universities and by representatives of independent institutions. The task of the evaluation panels is the assessment of the degrees belonging to a scientific area, considering, on the one hand, the self-evaluation reports presented by each school and, on the other hand, a 2 or 3-day visit to the degree site. The assessment is accomplished considering several pre-defined criteria. Recently, the results of the assessment of higher education in Portugal started to be expressed in a quantitative way, complementing the qualitative evaluation. A first phase of this process intended to obtain, for each group of degrees, assessed by the same evaluation panel, a rating for each evaluation criterion. In a second phase, after the definition of a methodology to weight each one of the evaluation criteria, an absolute ranking of the degrees belonging to a certain scientific area may be obtained. We will present a first approach to a Decision Support System (DSS) that intended to help the Foundation of the Portuguese Universities in the task of managing part of the process. The prototype of this DSS was built on an Excel sheet using VBA and it incorporates a methodology to obtain the rating of the degrees for each criterion. Keywords: DSS; Higher Education Degrees; Rating 1 Introduction 1.1 Objective One of the objectives of the Bologna declaration is the assessment of teaching quality. The process of assessment of higher education degrees in Portugal was built with this objective in mind. This is a process of great dimension and complexity, generator of conflicts regarding fundamental questions of the decision model, such as: what is evaluated, how it is evaluated, who evaluates and when it is done. It is therefore necessary, in an initial phase, to clearly define a series of rules, agreed on by all participating elements of this process, that eliminates or at least reduces the inconsistency of the results. This project aims at building a support system for the decision process related to the assessment of higher education degrees. To obtain this objective a statistically robust Decision Support System (DSS) was created. This system should include support tools for the following phases of the assessment process: Input and handle the information; Methodology to obtain a degree rating within each of the assessment criteria. 1.2 Organization and Summary This document is divided into three sections, including this introduction. Section 2 addresses the assessment support system of higher education degrees (AHED) and it is subdivided in three subsections, beginning with a brief introduction to AHED, followed by a description on how the information is input and the methodologies used in the different phases of the process. The third and last section includes a summary of the results and some future work to be done. 1.3 General Overview The decision support system presented in this paper is based on pair wise comparisons between degrees. Each assessor that visited a pair of institutions just has to say if, for each criterion, degree i is better, similar or worse than degree j. The evaluations of all assessors are merged and a global evaluation is computed based on the most frequent score given by all assessors. To much low levels of agreement among assessors and lack of evaluations for a certain pair of degrees are situations that are automatically detected and tools to aid their resolution are implemented. The final step is the rating of the degrees, i.e., deciding to which group each degree should belong. The decision of accepting or rejecting a degree in a group is based on a test of hypothesis over the distance of a degree to the best of the good groups and to the worst of the bad groups. These tests are performed over a normalized sum of the scores obtained by each degree.

2 2 AHED 2.1 Introduction The support system for the assessment of higher education degrees (AHED) is composed by two modules, which correspond to the two phases of the assessment process. In a first phase, the definition of methodologies to input and handle information supplied by each element of the evaluation panel; in a second phase the definition of methodologies in order to obtain the degree rating within each of the assessment criteria. 2.2 Input and Treatment of Information The Worksheet Setup To start the process it is necessary to insert some initial data. These data define the number of degrees, the criteria and the assessors and their designations. Suppose that you fill in the data sheet with the values shown in Figure 1: Figure 1: Data Sheet To start the process the macro StartUp should be executed. This macro will generate one sheet for each assessor s data, an auxiliary sheet and RS m rating sheets, where m is the number of degrees. RS m is given by the following expression: RS m = m 2 if m even Inserting the Assessor s Data m if m odd (1) The next step is the insertion of the information given by each one of the assessors (Figure 2). Figure 2: Example - Assessor s Data For each criterion, there is a square matrix of dimension equal to the number of degrees. The values to put in the cells of the referred matrix can be 0 (worst), 0.5 (similar) or 1 (best). It is important to refer that it is not necessary to fill in all the cells, as generally an assessor does not visit all the degrees. In this case the assessor should leave the cells blank ( ). In order to explain the meaning of these values, the matrix of criterion1, for assessor1, presented in Figure 2, will be used: the (1) in element (1,2) of the matrix means that degree1 is better than degree2; the (0.5) in element (1,3) of the matrix means that degree1 is similar than degree3; the (0) in element (2,3) of the matrix means that degree2 is worse than degree3. In this same table it can be noticed that it is only necessary to fill in the upper triangular matrix. In fact, when classifying element (1,2) of the matrix with value 1, i.e., classifying degree1 as being better than degree2, then element (2,1) of the same matrix automatically receives the information that degree2 is worse than degree1, i.e, it receives the value 0. In general, if Element ij = 1/0 then Element ji = 0/1, else if Element ij = 0.5 then Element ji = 0.5, else Element ij = Element ji =. 2.3 Obtaining the Degree s Rating for Each Criterion The Vote Count We will use the following notation: d assessor {1,...D} k criterion {1,...K} i degree {1,...I} j degree {1,...I} w score {0, 0.5, 1} The votes of each assessor d are input and for each criterion k, each pair of degrees (i,j) and each score w, the votes are counted V kwij. The total number of votes TV kij are also obtained. These partial sums are represented in Figure 3 of our example. The calculation of V kwij and TV kij is done by the following expressions: 1 if d scored with w the relation ij for k P dkwij = 0 if not kwij V kwij = (2) D P dkwij (3) d=1

3 kij TV kij = V kwij (4) w {0,0.5,1} The level of confidence (c) parameter is related to the statistic tests performed in the rating process will be explained in subsection In Figure 5, R kij, the summary of the assessors scores for each criterion k and for each pair of degrees (i,j) is presented. R kij is calculated by the following expression: kij if TV kij < (a) R kij = V otes? else if w V kwij TV kij (b) R kij = w (5) Figure 3: Example - VoteCount Sheet else R kij = Decision! Tuning the Parameters Before starting the rating process it is necessary to adjust three parameters: (a) - minimum total of votes; (b) - minimum percentage of votes to decide; (c) - level of confidence. Figure 4 shows the values used in the example. Figure 5: Example - Rating1 Sheet - Selection Figure 4: Example - Rating1 Sheet- Setup The level of confidence parameter (c) allows to immediately determine the field Cut Value (right tail), corresponding to the percentage value for the t distribution with (I-1) degrees of freedom (where I is the total number of degrees). The parameter minimum total of votes (a) is directly related to the minimum number of assessors that must be common to each pair of evaluation groups, i.e., the minimum number of assessors that can compare two degrees as they have visited both institutions. The minimum value for this parameter is, of course, 1. The parameter minimum percentage of votes to decide (b) is related to the level of agreement necessary to have a final score for each pair of degrees. For instance, a value of 75% means that at least 3/4 of the assessors have to give the same score for that score to be considered in the final tableau. If it doesn t happen this decision is left to be taken during a general meeting of the assessors. In this phase, if for all k,i,j, R kij V otes? and R kij Decision!, according to Canter [1], the values LineSum (LS ki ) and NormLineSum ( LS ki ) are calculated Figure 5. The expressions used to calculate LS ki, T k and LS ki are respectively: LS ki = T k = I R kij (6) j=1 I LS ki (7) i=1 LS ki = LS ki T k (8) The expression LS ki can also assume the label Lack of Decisions if, in the row related to degree i there are one or more Votes? (insufficiency of votes) or Decision! (decisions to make). In order to proceed to the rating process both situations have to be solved.

4 2.3.3 Insufficiency of Votes When the votes are insufficient it is necessary to verify which pairs of degrees have insufficient votes and for each one of these pairs, a pair of assessors that have visited at least one of these degrees and a third in common need to be found. In fact, not having direct information about how to compare degrees α and γ, because there isn t a common element in the committees that visited the respective institutions, makes it necessary to estimate a relation between theses degrees through an indirect comparison, ie, using a third degree β which serves as a pivot in the transitivity relation: e.g. if α is better than β and β is better than γ then α is better than γ. All possible situations for the relation of order are represented in Table 1, together with the respective result. Value 0 expresses a worst relation, value 0.5 expresses a similar relation and value 1 expresses a better relation. Finally, a * represents a wildcard meaning any value of the relation of order. The wildcard is used when the value placed in that position is indifferent for the resulting relation (α,γ) and has the objective of simplifying the table. (α,β) (β,γ) (α,γ) Table 1: Transitivity Table The use of different pivots can originate different conclusions for the relation (α,γ), given a natural degree of inconsistency in the evaluations. So, whenever there is more than one possible pivot for the relation it is necessary to collect different values in order to generate a single relation. The usage of a measure that results in a value not belonging to set V={0, 0.5, 1} is naturally eliminated. It was decided to use the average of the obtained values, rounded to the closest value of set V. Special attention should be given to values 0.25 and So, assuming that whatever can not be decided may be considered as equivalent, as has been done in the transitivity table for the cases where (α,β) (β,γ), both 0.25 and 0.75 will be rounded to 0.5. In order to unravel this transitivity process the Negotiation macro should be executed. After executing this macro the Negotiation sheet is created, where the pairs of degrees with insufficient votes are represented. This sheet indicates for each criterion, which pair or pairs of assessors visited one of the two degrees involved and a third in common, and presents as a result of the transitivity process, the relation of the degrees. To simulate a situation with insufficiency of votes, admit that, for criterion1, a set of assessors classified the degrees as illustrated in Figure 6(a). These classifications lead to the values presented in Figure 6(b), leading to the conclusion that there is an insufficiency of votes in the relation of degree2 with degree3. Through the analysis of Figure 6(a) it is said that assessor1 and assessor2 visited degree2, assessor3 visited degree3, all visited degree1 and none scored the relation of degree2 with degree3. (a) Votes (b) Selection Figure 6: Example - Insufficiency of Votes Figure 7 shows the result of the application of the transitivity process to the data present in Figure 6. Figure 7: Example - Negotiation Sheet - Insufficiency of Votes In order to determine the values presented in Figure 7, consider that (degree2, degree1, degree3) corresponds to (α, β, γ). Given that assessor1 and assessor2 scored (α, β) value 0 (if (β,α) = 1 then (α,β) = 0), and, assessor3 scored (β, γ) value 1, then, see transitivity table, (α, γ) results in value 0.5. Calculating the average of both values we obtain 0.5, score for the relation of degree2 and degree Decisions to Make When there are decisions to make for any pair of degrees, because there is not a minimum level of agreement among the assessors, then the assessors that visited those pairs of degrees should meet in order to define the relation for the pairs of degrees in question and increase the level of agreement. Now we will simulate a situation where decisions need to be made, considering that, for criterion2, the set of assessors scored the degrees as illustrated in Figure 8(a). Figure 8(b) reflects the results of these classifications, leading to a situation of doubt in the relation of degree1 with degree3. This situation is a consequence of the lack of agreement in the evaluations, given that assessor1, assessor2 and assessor3 classified the relation of degree1 with degree3, respectively with 0, 0.5 and 1. To define the relation for the pairs of degrees in question a meeting among the assessors is needed.

5 (a) Votes (b) Selection Figure 8: Example - Decisions to Make The Rating Process After defining the scores for all pairs of degrees it is possible to start the rating process, by executing the Rating macro. The degrees are ordered both in decreasing and increasing order of LS ki. Considering that degree l is such that LS kl = max i LSki the differences R klj R kij are calculated together with the mean j (R klj R kij )/I and the standard deviation for each i. The same is done for degree u, where LS ku = min i LSki (see Figure 9). Figure 9: Example - Best and Worst (criterion1) Since this situation is dealt with sample pairs of small dimension and according to Guimarães and Cabral [2] we must test the hypothesis of the expected value of a normal population µ = 0. The value t of the t-distribution is calculated by the following expression: t = d µ s m (9) The value t for the best degree is not calculated (because s = 0) and is automatically accepted. The remaining is compared with the cut value (for example ) and if they are inferior to this value they are Accepted, if not they are Rejected. The analysis for the Worst table is similar to the analysis relative to the Best. If a degree is rejected by both groups, i.e., by the Best and the Worst, then it is evaluated in the following Rating sheet. Considering the degree is accepted by one of the groups (Best or Worst) then it will be immediately rated. For each Rating i sheet, the degrees belonging to the Best group are labelled i, while the degrees belonging to the Worst group is labelled m i+1. At a final stage the numbers are converted to classes labelled by letters in alphabetical sequence. It should be printed out that the number of rating classes obtained in this process is dependent on the level of confidence defined for the hypothesis test. A high level of confidence (near 1) means that we want to be very sure when rejecting a degree from a group and so we will end up with just one group: nothing is excluded, all the degrees are equal. On the other hand, a low value for the level of confidence will originate a rating where each group will have just one element, i.e, all the degrees are very different but we may be very wrong. A few trial runs should be performed in order to tune an adequate value for this parameter. 3 Conclusions and Future Work Some important results obtained in this investigation and some future work to be done will now be summarized. The Decision Support System (DSS) for the Assessment of Higher Education Degrees is an extremely important tool to manage the whole assessment process. During the meetings, whenever there are situations of uncertainty, it determines which assessors should meet to complete the rating of degrees for the different criteria. This DDS, which is based on the analysis of sample pairs, was used to obtain the rating of engineering degrees in Portugal, having produced valuable results. As for future work, the search for theoretical results that may eventually lead to more efficient systems emerges, as a way of improving the statistical robustness of the evaluation process. Another topic includes the implementation and testing of methodologies to weight the evaluation criteria, leading to the determination of an absolute ranking. References [1] Larry W. Canter. Environmental Impact Assessment, pages McGraw-Hil, 2nd edition, [2] Rui Campos Guimarães and José A. Sarsfield Cabral. Estatística, pages McGraw-Hill, José Paulo Santos José Paulo Santos graduated in Electrical and Computer Engineering and awarded a MSc in Computational Methods in Sciences and Engineering in the area of Simulation at the Faculty of Engineering of the University of Porto (FEUP), Portugal. Recently he is attending a PhD in Electrical and Computer Engineering at the same faculty. José Fernando Oliveira José Fernando Oliveira is Auxiliary Professor at the Department of Electrical and Computers Engineering of the Faculdade de Engenharia da

6 Universidade do Porto and Senior Researcher in the Manufacturing Systems Engineering Unit at INESC-Porto, in Portugal. He received his PhD by the Faculdade de Engenharia da Universidade do Porto, in His primary area of interest is the application of Decision and Optimization Methods to industrial and organizational problems. The main application areas have been Cutting and Packing Problems and Decisions Support Systems for the government of Higher Education schools. Maria Antónia Carravilla Maria Antónia Carravilla graduated in Electrical Engineering and finished her MSc in Electrical and Computer Engineering in the area of Control Theory, both in the Faculty of Engineering of the University of Porto (FEUP). In 1996 she concluded her PhD in Operations Research and Production Planning at FEUP. She is Auxiliary Professor at the Department of Electrical and Computers Engineering of FEUP and Senior Researcher in the Manufacturing Systems Engineering Unit at INESC-Porto, in Portugal. In the last years, her main research areas have been Logistics, Decision Support Systems and Constraint Logic Programming. Carlos Costa Carlos Costa is a Full Professor of Chemical Engineering at the Faculty of Engineering of the University of Porto, Portugal since He obtained his PhD in Chemical Engineering from the Faculty of Engineering of the University of Porto, in His current research interests centre on Process and Environmental Systems Engineering. He is the author of over 50 publications in refereed journals and presented over 50 communications at international conferences with refereeing. He is, since 2001, Dean of the Faculty of Engineering of the University of Porto.