An analytic hierarchy process for school quality and inspection Model development and application

Transcription

1 An analytic hierarchy process for school quality and inspection Model development and application Research Office

2 Abstract Purpose - The study develops an analytic hierarchy planning (AHP) framework to establish criteria weights and to develop a school performance system commonly called school inspections. Design/methodology/approach - The AHP model uses pairwise comparisons and a measurement scale to generate the weights for the criteria. The validity of the approach is confirmed by comparing the outputs of school inspection and the outputs of the model in a sample of schools. Findings - The proposed framework enables school management to address several issues pertaining to its competitive advantage with other schools, the two most important being establishing its performance ranking in the marketplace and identifying the service elements that most require improvement. This study develops a cohesive approach to identify which quality attributes or dimensions require attention. Research limitations/implications - For school inspections, the data collection and computational problems would increase with the increase in the number of criteria and subcriteria, as well as the number of schools considered in the selection. Although the range of reported AHP applications is extensive in many disciplines, in school quality and inspection there are few examples; as a result, this study could not compare its results with other AHP applications in school inspection or assessment. Practical implications - The AHP method has the distinct advantage that it enables a decision problem to be broken down into its constituent parts, and it builds hierarchies of criteria. AHP enables assessors to capture both subjective and objective evaluation measures of school quality. By providing a useful mechanism for assessing the consistency of the evaluation measures and alternatives, the AHP reduces bias in decision making. Originality/value Over time, the AHP model could become a sustainable component of overall school system quality improvement. The AHP annual scores could be used as realistic and measureable gauges for measuring school improvement. Keywords Analytic hierarchy process, multi-criteria decision models, school quality, school inspection, Abu Dhabi. Introduction The process of conducting school inspections is important to enable decision-makers not only to evaluate the quality of schools, but also to identify their strengths and weaknesses. The results of inspection can be used to support evidence-based policy making. Specific findings can be used by system leaders to maintain areas of strength and to take weaknesses. (Grauwe, 2007; Ehren & Visscher, 2008; Janssens & Van Ahelsroot, 2008; Ehren 2

3 & Honingh, 2011). For Abu Dhabi, the aim is to ensure that all schools offer students a quality education that meets standards of excellence achieved in the most highly educated countries of the world. It provides proprietors, board members, principals and managers with an external evaluation which they can use as a basis for improvements. The Abu Dhabi Education Council (ADEC) policy makers expect that if schools are aware that the most successful schools are the most popular, they will be motivated to improve. ADEC published reports on inspection of private schools for the first time in July ADEC is responsible for establishing a quality assurance system for both public and private schools. ADEC started the inspection process for private schools in 2008, largely following the British school inspection models. The process was extended to include a sample of public schools in the academic year All school inspection standards were weighted equally. Each of the standards evaluated through inspection was assigned a score ranging from 1 to 9 with 1 recognizing outstanding provision, and so on through very good, good, satisfactory and improving, satisfactory-1, satisfactory-2, unsatisfactory, very unsatisfactory down to 9 representing poor. After the third school inspection cycle in , many private schools in Abu Dhabi complained that the standards used in the school inspection system used common standards across the various curriculums and schools. Many schools suggested that different weights should be used for the different standards in the system. The inspection system is based on evidence collected on eight performance standard categories during the inspection visit: students achievement and progress; students personal development; the quality of teaching and learning; the curriculum s meeting of students needs; the protection, care, guidance and support of students; the quality of the school s buildings and premises; the school s resources to support its goals; and the effectiveness of leadership and management. The ADEC inspection process involves schools in completing an initial self-evaluation process, followed by a review by an independent and highly experienced inspection team. The inspection teams spend approximately one week in each school under review, measuring performance against the school s self-evaluation and ADEC s framework of schools measuring overall school quality and effectiveness. Figure 1 summarizes the ADEC school inspection process. Schools are usually notified of their inspection by ADEC s inspection agency approximately 15 days in advance of the inspection. Once the inspection is announced, the lead inspector telephones the school principal to make contact and to answer any questions about the process and organization of the inspection. This explanation includes clarifying the purpose of the initial visit and discussions about the content of that visit. Following the initial notification, the inspection agency provides a parents questionnaire. The current study formulates an analytic hierarchy process (AHP) multiple decision making analysis of implementing a school inspection system. The purpose of the decision criteria is to rank schools according to pre-established requirements. Through this process many factors are considered. The study will describe the overall methodology for the approach 3

4 and present the specific results from the approach. The main research questions when applying the AHP to school inspection and quality are: What are the most critical criteria when inspecting school quality? Which schools are ranked high (or low) based on the criteria used for school inspection? When doing sensitivity analysis, is it possible to compare schools with other target schools which share common characteristics for school improvement purposes? Figure 1. The ADEC school inspection process Collect data on all ADEC schools Notify school (15 days prior to visit) Collect detailed data from school Conduct initial school visit Submit draft inspection plan Issue preinspection brief (PIB) Organize team inspection SCHOOL INSPECTION Issue inspection final report Conduct AHP LITERATURE REVIEW School inspections Research notes that school inspection is considered a special type of evaluation conducted within the formal school setting, and its importance is based upon the understanding that the apparatus yields reliable, pertinent and up-to-date information about school quality and the student performance (London, 2004). The central concern of any school inspection is to ensure that the school attains appropriate educational standards. Usually, the assumption of any school inspection system is that the resultant recommendations and findings will stimulate improvement of the school s quality of education (Ofsted, 1999; Daniels, 1999). 4

5 Many countries arrange for external evaluation of their schools to assess and improve the quality of education (Ehren & Honingh, 2011). Inspectorates of Education assess the quality of education in schools, using standards and procedures to evaluate a broad range of goals related to student achievement, teaching, school organization and leadership (McNamara & O Hara, 2008). School inspections set expectations for school performance, produce evidence as to whether these expectations have been met and expect this evidence to stimulate and guide school improvement (McNamara & O Hara, 2008). Inspection reports are usually produced a basis for stakeholders decisions about the education and schooling enterprise in question, as well as to guide implementation for school reform and improvement (Learmonth, 2000). The process is intended to establish a comprehensive educational assessment framework to enhance schools performance level by creating a competitive environment, increasing awareness among educators of the importance of quality performance in education and providing inspection tools and established standards. Effective inspection is an important process in improving the efficiency and development of education systems (Gurkan & Deveci, 2012). Analytic hierarchy process The analytic hierarchy process (AHP) is a theory of measurement through pair-wise comparisons and relies on the judgments of experts to derive priority scales. Thomas Saaty (1994) developed the analytic hierarchy process approach to hierarchy development and validation at the Wharton School of Business. It is an effective tool for dealing with complex decision making, and may aid the decision maker to set priorities and make the best decision. By reducing complex decisions to a series of pairwise comparisons, and then synthesizing the results, the AHP helps to capture both subjective and objective aspects of a decision. In addition, the AHP incorporates a useful technique for checking the consistency of the decision maker s evaluations, thus reducing the bias in the decision making process. It is a Multi-Attribute Decision Method (MADM). MADM refers to a host of quantitative techniques used to facilitate decisions that involve multiple competing criteria. These scales measure intangibles in relative terms. The comparisons are made using a scale of absolute judgments that represents the degree to which one element dominates another with respect to a given attribute. The judgments may be inconsistent, and one concern of the AHP is how to measure inconsistency and improve the judgments, when possible, to improve reliability and consistency. The derived priority scales are synthesized by multiplying them by the priority of their parent nodes and adding all such nodes. The AHP provides a scientific tool for assigning weights to standards and criteria for school inspection. In addition, it systematically uses those derived weights to assign inspection scores to each school. In general, the AHP is a method of breaking down a complex, unstructured situation into its component parts and arranging those parts, or variables, into a hierarchic order. The method is based on assigning numerical values to subjective judgments on the relative importance of each variable, and synthesizing the judgments to determine which variables have the highest priority. The AHP is built on three principles: constructing hierarchies, establishing priorities and logical consistency. The task of establishing priorities requires that the criteria and sub- 5

6 criteria be layered in the hierarchy so that the elements of each level are comparable among themselves in relation to the elements in the next higher level. A weighting process obtains overall priorities by moving down the hierarchical structure of information and weighting the priorities measured within a hierarchy level with respect to the criterion in the next higher level with the weight of that criterion. For the weighting process, the AHP uses pairwise comparison to create a derived scale for each set of criteria in the hierarchy. The pairwise comparison asks the individual to specify levels of importance. Learning theorists have determined that many learning goals can be considered hierarchical in nature. The hierarchical structure of knowledge appears in diverse subject areas. There is even evidence suggesting that the hierarchical arrangement of information is what separates novices from experts. The hierarchy provides a visual means of defining and assessing a target objective. People are demonstrably inconsistent in making comparative judgments, particularly when they deal with intangibles that have no objective scales of measurement. Thus, we should not expect total consistency. One of the greatest advantages of the AHP is that it has built-in systematic assessments for consistency of judgments (Alonso & Lamata, 2006). We use a test statistic, the consistency index (CI), to measure consistency in decision makers comparison of performance criteria (or elements). The CI is then compared to values of the same index for a randomly generated matrix, RI. The ratio of CI to the average RI for the same order matrix is called the consistency ratio, CR. A consistency ratio of zero means that the comparisons are perfectly consistent; higher ratios indicate lower consistencies. The inconsistency ratio is calculated for each set of judgments. It is important to emphasize that the objective is to make good decisions, not to minimize the inconsistency ratio. When the inconsistency ratio is zero we have complete consistency; when it is greater than zero there is some inconsistency. The larger the value of the inconsistency ratios, the more inconsistent the judgments are, and a ratio of 0.10 or less is generally considered a tolerable inconsistency. The degree of inconsistency that indicates a significant problem depends, of course, on the specific situation where the model is applied, but the number 0.10 serves as a general guideline for a maximum acceptable level (Saaty, 1996). The analytic hierarchy process in education In his classic book about AHP, Saaty (1995, pp. 30, 51) provided an illustration for the use of AHP in school quality and school ranking based on many criteria. However, most of the AHP applications in education have focused on higher education, and few focused on school related decision making. Pecar et al. (2009) developed a self-evaluation model and tool based on the weightings of improvement areas indicated by all the stakeholder groups and that also could be used in all European countries. The authors did not utilize the AHP but only they suggested its use. Wu et al. (2012) established weights for performance evaluation indices for higher education based on the official performance evaluation structure developed by the Taiwan Assessment and Evaluation Association and ranked 12 private universities listed by the Ministry of Education as a case study. They utilized AHP to determine the weights and applied a hybrid multi-criteria decision-making model to accomplish these objectives. They 6

7 maintained that the results of this study could provide a reference point for other universities and similar educational institutions in their efforts to conduct evaluations, improve their performances and formulate educational policies. Hu et al. (2009) developed an integrated multi-criteria decision-making approach for a virtual learning environment system in higher education, combining the analytic hierarchy process (AHP) and quality function deployment to evaluate and select the best system. Badri and Abdulla (2004) examined how an institution of higher education might operationalize faculty performance evaluation on criteria of research, teaching and university and community service, using the analytic hierarchy process. They developed weights for each criterion in the evaluation process for a more objective outcome. Tsinidou et al. (2010) used a multi-criteria decision-making methodology for assessing the relative importance of quality determinants that affect student satisfaction. They employed the analytical hierarchical process to measure the relative weight of each quality factor. Hu and Hung (2004) used AHP to develop a marketing mix for a graduate institute at NCYU in Taiwan. A survey using stratified random sampling was conducted, with 14 universities from four different areas in Taiwan randomly selected for the study. They used two questionnaires, a five-point Likert questionnaire regarding school impressions and an AHP questionnaire regarding school selection factors, both were administered to 640 undergraduate students. Of the total number of questionnaires, 602 (94%) valid school image questionnaires and 570 (89%) valid school selection factors questionnaires were used. Lam et al. (2008) prioritized key total quality management (TQM) Culture (TC) constructs and relationships using AHP based interviews, focus groups and ethnographic studies. They formulated an empirical TC based organizational learning (OL) transformation model for vocational education. Das et al. (2012) used AHP in assessing Indian technical education. They focused on the performance evaluation and ranking of seven Indian Institutes of Technology (IITs) on the basis of stakeholders weightings using an integrated model consisting of fuzzy AHP and Commission on Peer Review and Accreditation (COPRA) guidelines for academic year 2007/2008. Lin (2010) evaluated the relative importance of quality factors across several online learning experience groups. He developed an evolution model that integrates triangular fuzzy numbers and the AHP to develop a fuzzy evaluation model that prioritized the relative weights of course website quality factors. The model and results provided a valuable reference for systems designers seeking to enhance course website effectiveness. Zhang et al., (2009) used the AHP to evaluate teachers teaching work-based AHP and fuzzy theory. The criteria relating to teaching quality were broken down into sub-criteria. The present study develops an AHP based framework to establish weights and evaluate Abu Dhabi schools in a school performance evaluation system based on inspections. Study methods Study objectives 7

8 The study attempted first to develop a valid and reliable set of quality indicators (the inspection instrument) appropriate to the study in hand. Using the AHP, the objective was to provide an unbiased assessment of the candidate private and public schools in Abu Dhabi. The instrument is intended to provide an objective framework for measuring each school s performance. To conform to the AHP methodology and to facilitate the management of the research, the study comprised several phases. The four main phases consisted of hierarchy development, hierarchy validation, assessment instrument development and instrument reliability testing. The hierarchy development process involved a review of various school inspection lists and approaches used by the Abu Dhabi Education Council and similar institutions and the review of school inspection variables in literature. After eliminating certain variables based on selection criteria, the remaining items were clustered and arranged into a hierarchy. Stringent scientific methodologies were used in the other phases. The final outcome of this research will be applied to private school inspection in Abu Dhabi on the basis of the eight performance criteria. This study s purpose is specifically to develop an AHP model for school performance inspection and explore its implementation in the ADEC private school system. Using the AHP, the study designed, analysed, tested and validated the instrument constructed, conducted full field experimentation, involved decision-makers and ensured model validity and acceptance in implementation. The detailed objectives were as follows: 1. Through extensive field research, to identify a) The dimensions of school performance evaluation. b) The criteria of school performance evaluation. 2. Using the AHP, to provide a model to increase the objectivity of the decision-making process regarding school performance evaluation. 3. To produce the AHP-based model, devising a simple and understandable methodology for its development and design. 4. To design an objective model based on the AHP to determine the weighing of the criteria for evaluation and assessment. 5. To provide simple procedures and directions for the easy implementation of the model in Abu Dhabi s private school system. 6. To solicit feedback on user satisfaction with the resulting model developed and with the AHP as a methodology for school inspection. 7. To perform validations of the AHP (a) process and (b) outcomes. This study broadens the causal base for school performance evaluation by considering the design and development of a system based on the AHP for objectivity. Even though the focus of the study was on school inspection, the developed model and the method of analysis should be sufficiently comprehensive for use, direct or adapted, and implementation in other school settings for the same or other purposes. AHP steps for developing the school inspection model For the school inspection process, we followed Saaty s (2008) seven steps: (1) Define the problem (school quality and inspection Abu Dhabi private and public schools). 8

9 (2) Structure the hierarchy (the main eight categories of inspection: students achievement and progress; students personal development; the quality of teaching and learning; the curriculum s meeting of students needs; the protection, care, guidance and support of students; the quality of the school s buildings and premises; the school s resources to support its goals; the effectiveness of leadership and management. (3) Construct a pairwise comparison matrix of the relevant contribution or impact of each element on each governing criterion in the next higher level. (4) Obtain judgments required to develop the set of matrices in step 3. (5) Having collected all the pairwise comparison data, obtain the priorities and test the judgment consistency. (6) Perform steps 3, 4 and 5 for all levels and clusters in the hierarchy. (7) Use hierarchical composition to weight the vectors of priority by the weights of the criteria and sum each weighted priority entry corresponding to those in the next lower level. This study reports that the AHP process is conducted at the end of this process where a lead inspector with the inspection team members use the previously determined weights (Table 1) to evaluate the schools, using pair-wise comparisons (only of same schools inspected by the same team). The inspection standards in ADEC adhere to many total quality management frameworks (Malcolm Baldrige National Quality programme, 2013) or the European Framework of Quality Management (EFQM, 2012). The standards and criteria included here are also covered by other self-evaluation models and tools for school quality (Pecar et al., 209). For ADEC, the framework took almost four years to design, test and develop for full implementation. Hierarchy validation process, design and development To ensure involvement and participation, the final analysis team included ten cluster managers and ten private school assessors in ADEC. The team determined the relative importance of each item in the hierarchy while simultaneously validating the hierarchy. The main purpose for this step was to collect input data to measure the relative importance of any element over other elements, and it involved extensive interviews with each team member. For data collection, the decision makers performed a pairwise comparison, producing a set of comparison matrixes that, in turn, resulted in an assessment instrument that reflected the weight of each item in the hierarchy. (See Figure 3 and its related detailed discussion of this step s results.) As we used several methods to compute the priorities/importance of elements in each matrix, we took as input pairwise comparisons prepared in the previous steps and produced as output the relative weights of elements at each level. We aggregated the weights of elements for various levels computed in the previous phases to produce a vector of composite weights that serves as a rating of decision variables or selection choices. To obtain a precise measurement scale, a nine-point combination scale was assigned to each item in the hierarchy (Saaty, 1994; Badri and Abdulla, 2004). Each item on the scale consisted of nominal, descriptive and ratio values. Figure 1 provides the hierarchical view of the model. 9

10 The AHP models developed for school inspection As Figure 1 illustrates, the application of the AHP comparison of schools begins after completing the inspection cycle. However, no timeline specifies the development period of the AHP hierarchy. It could begin as early as the beginning of the inspection cycle. To better understand the steps performed to implement the AHP model, we performed a prehierarchy development process involving the review of ADEC s current practices related to the school inspection framework. Other related or similar school inspection system standards were also reviewed. As a result, ADEC s framework was modified to fit the AHP process. Several experts on school inspection reviewed the framework. The reviewer function titles included school assessors, division head of school operations, division head of P-12, school assessment supervisors, school principals, and heads of faculties, school viceprincipals, cluster managers and academic consultants. All these persons were highly familiar with the ADEC school inspection processes. We also consulted several external experts on school operations. Establishing the decision hierarchy The main task of school inspection was decomposed into primary elements, which we then grouped into several major categories with levels forming a hierarchy. Thus, the lower hierarchy represented a decomposition of the major category in the higher (or parent) hierarchy. The final items were clustered and arranged into a hierarchy based on Saaty s (2008) hierarchy development process. Figure 2 depicts the detailed AHP decision hierarchies. The first performance standard comprises students achievement and progress, with six indicators: student selection policies, student achievement and progress, variances in progress and achievement, use of standardized tests, development of skills and students destination after graduation. The second standard comprises students personal development, with five broad indicators: overall development and attitude, development in moral behaviour and attitude, internal and external engagement, understanding of safe and healthy living and development in social and personal skills. The third standard comprises quality of teaching and learning, with five indicators: teacher competency, methods and resources utilized management skills, turnover and extent of professional development. The fourth standard comprises four indicators related to meeting student needs through the curriculum: the curriculum s impact on students, its representation of the United Arab Emirate culture, outcomes and expectations and student preparation (i.e., for life and subsequent schooling). The fifth standard, student protection-care-guidance and support, comprises five indicators: safe environment, school records, data and information, protecting students from bad behaviour, student rewards and punishments and religious opportunities. The sixth standard addresses the quality of school facilities, with three main indicators: building size per ADEC standards, an environment reflecting good health and hygiene and staff responsibility for health and safety. The seventh standard comprises school resources to support goals, with five indicators: quality, quantity and range of resources, overall nature of teachers and staff, school facilities, school transportation and the nature of storage of food and related items (hygiene and health). The eighth standard comprises effectiveness of leadership and 10

11 management, with five main indicators: governance and management, school leadership, involvement with stakeholders and the community, understanding the school s SWOT (strengths, weaknesses, opportunities and threats) and school financial resources. Hierarchy validation process The hierarchy validation process involves the validation and weighing of the hierarchy. To ensure involvement and participation, the final nomination team included the same ADEC personnel and 15 assessors from an international firm that regularly conduct school visits in Abu Dhabi before, during and after school inspections. The team determined the relative importance of each item in the hierarchy while simultaneously validating the hierarchy. Each member participated in extensive interviews during this step. This step s main purpose was to collect input data to measure the relative importance of each element in relationship to other elements. The decision-makers performed a pairwise comparison, producing a set of comparison matrixes that, in turn, resulted in an assessment instrument that reflected the weight of each item in the hierarchy. (See Figure 3 and its related detailed discussion of this step s results.)as we used several methods to compute the priorities/importance of elements in each matrix, we took as input pairwise comparisons prepared in the previous steps and produced as output relative weights of elements at each level. Implementation of the Abu Dhabi AHP School Inspection Model We aggregated the weights of elements for various levels computed in the previous phases to produce a vector of composite weights to serve as a rating of decision variables or selection choices. To obtain a precise measurement scale, a nine-point combination scale was assigned to each item in the hierarchy (Saaty, 1994). Each item on the scale consisted of nominal, descriptive and ratio values. The main objective of this study was to develop a comprehensive model that established definitions of criteria for school inspections on the eight main standards. The model could be used for a variety of objectives, as described in the Discussion and implications section. As Figure 2 illustrates, the model s first level identifies the school s overall inspection score and the scores for each of the eight main standards. The second level identifies the main indicators on which the school s performance is evaluated by the inspection. Other criteria could be easily added, if required. The objective may be to calculate the overall performance so that the model produces a score for each school. In addition, if the objective is to select schools as Best Performers on each of the eight standards, a score could also be identified for each standard. Results Unfortunately, decision makers do not normally make perfect judgments, resulting in inconsistencies. If the value of the consistency ratio (CR) is smaller than or equal to 10%, the inconsistency is acceptable. For each of the eight dimensions, the inconsistencies were 0.028, 0.061, 0.036, 0.034, 0.030, 0.033, and 0.025, respectively. All values are clearly below 0.10 indicating acceptable levels of inconsistency. 11

12 Figure 3 depicts the weights for the eight dimensions. Student attainment and progress, with a 30.12% weight, has the highest priority, students personal development has the second priority with a 23.67% weight, and school resources (3.58%) is the least important standard. It should be noted here that the current and actual school inspection system in Abu Dhabi does not assign weights to each of the eight dimensions. Hence, one significant advantage of AHP is the derived weights for the overall dimensions and for each of the individual criteria. These weights are used in subsequent analysis to calculate the ranking of the schools. The AHP analysis provides 38 weights for the model s detailed criteria. Figure 4 reports the ranked individual weights. The top criteria of school quality are student achievement and progress, teacher competency, student development and attitude, student skills development, development of students moral viewpoints and curriculum outcomes and expectations. The individual weights provide some importance or priorities for the schools that are trying to improve their individual school inspection ranking in the future. Correlation among school inspection standard weights The comprehensive literature search conducted while performing this study found no evidence for correlation among school inspection standards. However, many school inspection assessors in ADEC preferred that the standards be as mutually exclusive as possible. Correlation calculation among the school inspection standards weights revealed few significant correlations. Despite a high coefficient of correlation between certain pairs of standards, the great majority were not significant statistically. Table 1 reveals only four significant correlations. The standard student achievement and progress correlates significantly with protection, care, and guidance of students (-0.847). The student personal development standard correlates significantly with school leadership quality (0.676). The quality of teaching and learning standard significantly correlates with the standard of school facilities quality (-0.689). The protection, care, guidance and support standard significantly correlates with the quality of school facilities. 12

13 Figure 2. Analytic hierarchy process model for school performance evaluation in ADEC Performance evaluation of schools in Abu Dhabi Student Achievemen ts and Progress Students personal development Quality of teaching and learning Meeting student needs through the curriculum Protection, care, guidance and support of students Quality of school facilities School resources to support goals Effectiveness of leadership and management Student selection policies Achievement and progress of students Variance in progress and attainment Use of standardized tests Development of student skills Destination after graduation Student development and attitude Development in moral viewpoints Student internal/external engagement Understanding of safe and healthy living Development in social and personal skills Teacher competency Teaching methods and resources Teachers' management skills Teacher turnover Teacher professional development Impact of the curriculum on students Curriculum and UAE culture Curriculum, outcomes, and expectations Curriculum and student preparation Safe environment School records/data and information Protecting students from bad behaviour Students' rewards and punishments Religious opportunities Building size and ADEC standards Buildings reflecting good health and safety Staff responsibility for health and safety Quality, quantity and range of resources Overall nature of teachers and staff School facilities School transportation Storage, hygiene and health Governance and management School leadership Involvement with stakeholders and community Understanding school s SWOT School's financial resources 13

14 Figure 3. AHP results of weights for school inspection standards Quality of school buildings Student guidance, care and protection School resources Leadership and management Curriculum and student needs Quality of teaching and learning Student personal development Student attainment and progress Major Standards Achievement and progress Table 1. Correlation among the standards Personal development Teaching and learning Curriculum/ standards Leadership qualities School resources Protection/ guidance Achievement and progress Personal development Teaching and learning Curriculum and standards Leadership qualities * School resources Protection and guidance ** School facilities * ** * Correlation significant at 0.05 level ** Correlation significant at 0.01 level School facilities 14

15 15

16 Figure 4. Importance of inspection criterion indicators (detailed) Attainment and progress of students Teachers competency Development and attitude Development of student skills Development in moral and viewpoint Curiculum and outcomes and expectations Variance in progress and attainment Development in social and personal skills Teaching methods and resources School leadership qualities Curriculum impact on students Student selection Internal and expernal engagements Teachers management and skills Understanding of safe and healthy living Curriculum and student preperation Building reflecting good health and safety Safe environment Use of standardized tests Quality, quantity and range of resources Schools financial resources Destination of students after graduation Professional development Curriculum and UAE culture Building size and ADEC standards Turnovers of teachers Protecting students and behavior Overall nature of teachers and staff Involvement with stakeholders and community Governance and management Understanding own SWOT Students reward and punishment School facilities School records and information Staff responsibility and health and safety School transportation Religious opportunities Storage, hygene and health

17 Level of agreement among the inspectors Kendall's coefficient of concordance for ranks (W) was used to calculate agreement among the school inspection experts as they provided weights to the eight standards and then to the detailed criteria within each standard. The idea is that (n) subjects are ranked (1 to n) by each of the rankers, and the statistical analysis evaluates the rankers level of mutual agreement. The analysis first converts the primary data of weights into ranks, displays the ranks and performs the calculations. The analysis yielded a highly significant coefficient of 0.798, indicating significant agreement among the school inspection team of experts. It should be noted, though, that one assessor provided scores that were significantly different from the rest. If this assessor s scores are included, the Kendall coefficient would be only This result was also presented to the division manager of school inspections. After he reviewed and discussed this assessor s results, he concluded that they were not consistent with scores of other participating assessors, and so these outlier results were not considered for further AHP analysis. Ranking a sample of schools using AHP weights an illustration Only six schools were selected randomly for evaluation by the weights established in the AHP analysis. For the pair-wise comparison of the six schools, we used a group of three assessors. In a group decision-making manner, the group made pair-wise comparisons between each of the two schools on each sub-criterion developed in the AHP model. We should note that the AHP model contains eight dimensions with a total of 38 individual criteria. As a result, for each pair-wise comparison a total of 38 evaluations were made. However, we should note that for each individual criterion, a total of 15 pair-wise comparisons are made. For example, the first school is compared to five schools in a pairwise manner; the second school is compared to four schools and so on. As a result, the 38 criteria indicators required a total of 570 pair-wise comparisons. Table 2 reports the results, with names of the schools abbreviated for anonymity and confidentiality. School ranking by AHP and by the assessors validity As described, ADEC assigns scores ranging from 1 to 9 for the dimensions and for each school, 1 denoting outstanding and 9 denoting poor). According to these scores, schools are only assigned to three bands (Band A: outstanding; Band B: satisfactory; and Band C: poor). As a result, a number of schools are assigned to the three bands without associated individual scores or individual ranks. One major drawback of the system is that all criteria dimensions (standards) are assigned equal weights. In addition, the school inspection system does not provide individual criteria scores. We should note here that the AHP process was applied to only six randomly selected schools. For validation purposes, the old inspection scores for those six schools were compared with the AHP inspection scores, and the ranking determined for each school relative to other schools in the analysis. For both scores to be consistent, the school inspection scores were reversed. As a result, a score of 9 reflects an outstanding school and 17

18 1 denotes a poor performing school. Table 3 reports both the AHP and the old inspection scores. Table 4 reports the detailed AHP scores for the top five private schools in Abu Dhabi. Table 2. Detailed AHP scores for standards and criteria (relative) Schools ISK AMAD GEMS. AAJS BAM ISAA Overall AHP Scores Student achievement and progress Student selection policies Achievement and progress of students Variance in progress and achievement Use of standardized tests Development of student skills Destination of students after graduation Student personal development Student development and attitude Development in moral viewpoints Student internal/external engagement Understanding of safe and healthy living Development in social and personal skills Quality of teaching and learning Teacher competency Teaching methods and resources Teachers management skills Teacher turnover Professional development Curriculum and student needs Impact of the curriculum on students Curriculum and UAE culture Curriculum, outcomes and expectations Curriculum and student preparation Leadership and management Governance and management School leadership qualities Involvement with stakeholders/community Understanding school s SWOT School's financial resources School resources Quality, quantity and range of resources Overall nature of teachers/staff School facilities School transportation Storage, hygiene and health Protection, care, guidance of students Safe environment School records/data and information Protecting students from bad behaviour

19 7.4 Student's rewards and punishments Religious opportunities Quality of school buildings Building size per ADEC standards Buildings reflecting good health and safety Staff responsibility for health/safety School Table 3. School ranking ADEC and AHP systems ADEC school inspection score School rank AHP inspection relative scores ISK Not available AMAD Not available GEMSAA Not available AAJS Not available BAM Not available ISAA Not available School relative rank It was also important to obtain the feedback of assessors on the time required to perform the weightings for the criteria and sub-criteria and later the assessment of the schools. Most assessors felt that for them to use the pairwise comparisons to calculate the weights for each of the standards was straightforward but required greater judgmental effort. They were relieved that the AHP automatically calculated the judgment scores because this automatic process saved substantial time. The process of pairwise systematic assessment between schools, on the other hand, was straightforward. Most assessors also thought that there was no need for general discussions when assessing schools, resulting in considerable time savings. Further analysis by the AHP The AHP provides many alternative results of high value. Figure 5 depicts the stack bar chart of the eight dimensions for each of the six schools. The height of each school s bar indicates the overall inspection result and hence school quality. Within each bar (school) the chart graphically represents the size (score) of each dimension. The figure reveals the overall superiority of the ISK School. The head-to-head analysis of schools on each dimension of the individual detailed criteria provides a unique perspective that helps schools in their self-improvement efforts. Most of the software that could deal with the AHP can provide head-to-head reports on the alternatives. These reports are part of the sensitivity analysis in most commercial software. For example, it would seem to be most effective if a weaker school were compared directly to the best school in its class (or in its band). The report could indicate specific areas for action which would improve the weaker school. Figure 6 reports the head-to-head scores of two schools in Abu Dhabi. The scores could also serve as targets (or benchmarks) for the 19

20 schools in the analysis. For public schools, the head-to-head report informs the design of professional development initiatives for school improvement. Use of data in this way enables schools to work in partnership on improvement with the strongest performing school taking a lead but also learning from the improvement journey of its partner school Figure 5. School performance on each inspection criterion Effectiveness of leadership School resources/support Quality of school facilities Protection/care/guidance Meeting student needs/curriculum Quality of teaching/learning Student personal development Student Attainment/progress Discussions This study developed an integrated AHP decision-making approach to measure the school performance in the context of school inspections. A case study conducted in Abu Dhabi schools was used to demonstrate how the process works. In this approach, ADEC school inspection criteria were used to translate the school performance measures into an integrated system of hierarchies of criteria or standards. The AHP was used to determine the 20

21 relationship weightings consistently. For school inspection assessors, the AHP served as a decision-making technique designed to cope with both the rational and the intuitive to select the best alternative from a set of alternatives evaluated by several criteria (Saaty & Vargas, 2001). In this technique, the decision maker performs simple pairwise comparison judgments, which are then used to develop overall priorities for ranking alternatives. The major advantage of this approach is that evaluation criteria are presented in a way that makes them accessible and of interest to ADEC, the schools, parents and other stakeholders. Another advantage is that the approach can reduce the subjectivity of qualitative and quantitative judgement of a school s overall performance. Figure 6. Head-to-head comparison of two schools ISK AMAD Effectiveness of leadership (0.29) 0.22 School resources/support (0.24) 0.19 Quality of school facilities (0.23) 0.16 Protection/care/guidance (0.22) 0.20 Meeting student needs/curriculum (0.29) 0.17 Quality of teaching/learning (0.26) 0.19 Student personal development (0.24) 0.16 Student Attainment/progress (0.39) 0.20 In addition to developing a more objective approach to the school inspection process, indepth analyses were performed to examine the validity of the selected system by comparing its results with those obtained from current system outcomes of school inspection in ADEC. The AHP system provides an opportunity for ADEC decision-makers to update their school inspection standards, criteria and weights continuously. This is possible because the 21

22 developed standards, criteria and weights are developed in-house and not merely borrowed from other systems. These dynamic standards therefore reflect the specific Abu Dhabi context and enable progressive realignment. ADEC now has an opportunity to review its existing system in the light of the findings about the AHP alternative school inspection approach and reflect on whatever its existing model and framework might be modified to evaluate its efficiency. The proposed AHP approach can support school inspections by producing a scoring mechanism easily understood by parents and the wider Abu Dhabi community. The scores obtained for each of the standards and criteria could be presented in accessible forms and available to all interested stakeholders. One implication for ADEC is that the proposed AHP model can assist policy makers and managers to identify all information sources for required input data. Through an organized hierarchical structure, the AHP model can provide readily-understood data. Usually, managers find it difficult to consistently evaluate and compare standards or more detailed qualitative factors. They commonly complain that it is nearly impossible for an individual to offer a consistent judgment or comparison of those qualitative factors because of their subjective nature. The proposed AHP model is attractive to managers because its pairwise comparison process enables managers to offer relative (rather than absolute) preference assessments, one at a time, on those qualitative factors (Yang & Shi, 2002). The process then scales these relative comparisons uniquely to ensure consistency of these values. In addition, a built-in inconsistency assessment mechanism established within the AHP identifies all inconsistencies at very early stages of the solution process (see Saaty, 2008). The methodological approach of organizing all standards and criteria (or sub-criteria) into a hierarchy in the proposed AHP model has also received positive confirmation from the practice. One advantage of such a hierarchical structure is to provide a framework in which it is critical to seek input about the standards and criteria within the hierarchy from different levels of managers (assessors, or school inspectors) in the organization. In addition, the AHP hierarchical structure can easily be incorporated into an interactive solution procedure that allows more active participation of other policy makers in the solution process. using the AHP procedure to estimate the initial relative weights or preferences for related criteria requires very little training, and the availability of AHP software can even enable managers to make pairwise comparisons (and what-if analysis) in real time (while computers perform complex mathematical manipulations). The AHP allowed school inspectors (or assessors) to provide their own weights to the eight criteria indicating their relative importance in assessing schools. Agreement among the inspectors is highly desirable. Each assessor could understand and articulate that the results of the pairwise comparisons (between criteria or between schools) after the analysis is completed relates to the ability to capture the comments provided by various other discipline experts and decision-makers during the input process. These comments can serve later to explain to the stakeholders the reason for rating certain pairwise comparisons in a particular manner. 22