Stakeholder Survey Analysis on Quality Assurance Process in Georgia

1 Stakeholder Survey Analysis on Quality Assurance Process in Georgia National Center for Educational Quality Enhancement (NCEQE) Nino Kopaleishvili Tamar Lortkipanidze Tbilisi, 2013

2 Table of Contents I. Objectives, Survey Planning and Implementation... 3 II. Survey Findings Students... 4 Employers... 6 Professors and Quality Assurance Staff... 9 III. Recommendations and Conclusion... 11 Acknowledgements... 12

3 I. Objectives, Survey Planning and Implementation The following survey was undertaken within the framework of the project on Promoting Internationalization and Comparability of Quality Assurance in Higher Education in the timeframe of February- March 2013.The aim of the survey was to receive feedback on existing external and internal quality assurance mechanisms in Georgia. The survey was conducted among project partners (four Georgian universities) to measure the effectiveness of the project activities and provide situational analysis on existing practices. The questionnaires were distributed with the assistance of coordinators at Georgian partner universities - Ilia State University (ISU), Ivane Javakhishvili Tbilisi State University (TSU), Shota Rustaveli Batumi State University (RSU) and Kutaisi University (KU). Several questionnaires were developed. The target group consisted of students, academic and quality assurance staff, and employers. The questionnaires were developed together with the EU partner agencies and later, translated into Georgian for further revision by local partners. The final questionnaires incorporated opinions of all stakeholders. Before launching the survey, piloting of questionnaires was also implemented. The questionnaires were distributed with the assistance of local coordinators to relevant stakeholders of higher education area professors, quality assurance staff, students and employers. The questionnaires included open and close-ended questions. The data was collected and analyzed in the statistical programme SPSS. Descriptive statistics was used to analyze quantitative data and coding was employed to analyze the qualitative data. Random sampling was used to select the sample. Due to moderate resources, the sample size was restricted, which is one of the limitations of the survey. There were three types of questionnaires considering stakeholdes (employers, quality assurance and academic staff and students). The questionnaires focused on two issues: first, external quality assurance mechanisms, existing external quality assurance standards, their importance and effectiveness. Second, it also included questions on internal quality assurance mechanisms, involvement of survey participants in the procedures and their views on improving effectiveness. The questionnaire for students mainly focused on how they interpreted quality assurance processes and what was the level of their involvement. The questionnaires for quality assurance and academic staff enquired about their assessment of current standards and procedures. The employer questionnaires mainly focused on how employers perceive authorization and accreditation processes, what benefits they see in relation to the labour market and in what ways they might be involved in the external quality assurance process.

4 II. Survey Findings Students Overall, 140 students were surveyed in all 4 partner HEIs. The student questionnaires focused on student experience regarding accredited programmes, their expectations and involvement in quality assurance process at a HEI. Bachelor students (67.4%), Master students (18.8%), Diplomized medical students (3.6%) and PhD students (10.1%) were surveyed. 97.8 percent of respondents were involved in accredited HE programmes. 84.7 percent of students indicated that they are well aware of programme accreditation and see the advantages of studying on an accredited programme, while 15.3 percent indicated they were not aware of programme accreditation. The survey showed that students have clear understanding of programme accreditation and its advantages, 74.3% of students are aware of authorization of educational institutions, while 23.6% of respondents indicated they don t have information about authorization. 83.1% of respondents indicated that authorization is highly significant, while 10.3 % considered is significant and less than 3% said it was insignificant. Significance of Authorization 83,10% 10,30% 3% highly signficant significant insignificant Figure 1. Significance of authorization The majority of respondents supported the idea that standards for educational institutions should be defined by the State (78.3%), while only 13.8% indicated, they did not have a clear idea who should define the standards and 8 % opposed the practice that the State should define the standards. As to the student involvement in quality assurance process, the results showed the low involvement of students. 12.5% of students indicated they are actively involved and 87.5% responded they have never been involved.

5 Student involvement in quality assurance 87,50% 12,50% actively involved no involvement Figure 2. Student involvement in quality assurance Qualitative Data Accreditation and its benefits to students Students were asked to analyse how they interpreted advantages of programme accreditation. The majority of students (26.3%) stated that the accreditation shall aim at the establishment of regular self-evaluation of educational institutions for improvement of educational quality and promotion of QA mechanisms development through establishment of comparability of an educational programme with standards. 17.1% of students consider that studying on an accredited programme opens up a path to better employment. 7.9% of students think that accreditation shows that an institution has successfully undergone monitoring and evaluation process and opens up a possibility to receive quality education. Some of the explanations included that accreditation means the standards set by the government are met (14.5%) or it s a process for obtaining a status of an educational institution (5.3%). The same percentage (5.3%) link accreditation to student grants. The 26.3% of students consider that accreditation is the basis for recognition of diplomas across the country. Students also see accreditation as a way to ensure compliance of an educational programme with its goals (7.9%), or as a way of obtaining high quality education (9.2%), or state-recognized form of teaching (6.7%). Also, programme accreditation means to satisfy requirements of an institution 2.6%. Authorization and its benefits to students While asking to find appropriate definition for authorization, it was clear that the majority of respondents saw very thin line between authorization and accreditation and some of them were incapable of drawing a difference between them.

6 38.1% of students define authorization as the compliance of formulated standards with selfassessment results. For 4.8% authorization is registration of an educational institution to obtain a status. 6.3% of respondents saw it as an external evaluation of an educational institution and precondition to accreditation (4.8%). The Standards which an educational institution should meet The questionnaire also focused on what standards educational institutions should comply with in the view of stakeholders. 67.4% of students said that relevant environment for teaching, high quality resources. 42.6% stated highly qualified human resources were important. 36.4% of respondents said high quality education programmes, syllabus, curricula, literature were important. 6.2% opted for student-oriented, flexible system and 4.7% gave importance to internalization and mobility. 3.9% said that tuition fees should be accessible for groups with different socio-economic background. Employers 44 employers who cooperate closely with project partner universities were surveyed. To the question whether they have information about external quality assurance, 63.6% of respondents said they have certain knowledge, 29.5% rated their knowledge as profound and 6.8% indicated they do not possess any information at all. 54.5% out of 44 respondents have been involved in quality assurance process of higher education. 4.5 % of respondents consider external quality assurance mechanisms are insignificant to maintain quality culture, 29.5% of respondents consider it more or less significant, 25 % of respondents say it is significant and 40.9 % identify it as highly significant. Quality assurance mechanisms insignificant 4% highly significant 41% more or less significant 30% significant 25% Figure 3: Describes significance of quality assurance mechanisms assessed by employers Analyzing the role of accreditation standards in developing HE programmes to prepare competitive graduates on the labour market, 50% considered the accreditation standards meet this criteria, while another half (50%) were more skeptical, saying it more or less meets the criteria.

7 16.3 % of the surveyed said they did not have any knowledge about programme accreditation. The majority of respondents (60.5%) said they have knowledge and 23.3 % identified their knowledge as profound. As to the accreditation procedures, 58.5% consider accreditation procedures more or less meet the requirements and 41.5% said it meets the requirement. The Table 1. given below identifies how intensively employers are collaborating with educational institutions. Valid Percent Valid I don't cooperate at all (have not attended 13.6 any of the QA meetings) I cooperate very rarely (I have 36.4 attended one or two QA meetings) rarely (I attend QA meetings from time to 34.1 time) actively (I attend all 13.6 the QA meetings) very intensively (I define myself to an educational institution 2.3 the format of cooperation) Total 100.0 Qualitative Data To analyze data coding was used. To the question if you have ever been involved in quality assurance process of higher education, 8 employers answered that they had participated as employers in development and discussion of programmes; 6 employers said that they had enrolled students in practical trainings and 4 employers indicated they had participated as juries in student competitions. The employers who were identified by the partner universities as active stakeholders cooperating with the universities, were also asked about their knowledge of existing standards and particularly which standard they would consider the most significant to the employer organization and why?

8 13 employers indicated that the standard they would address first was the standard on the goals of an educational programme, learning outcomes and compatibility of the programme with them. The goal should be labour-market oriented. 10 employers considered that the standard on learning outcomes of a programme to ensure competitiveness of graduates on educational and labour markets was significant. 9 employers said all standards were equally significant. 5 employers stressed that the standard which focuses on the compatibility of the programme goals with the mission of the institution, that they are clearly defined and employment, market-oriented and attainable is the most significant. 3 employers said they are not familiar with the accreditation standards. The questionnaire also posed the question which standard needs further improvements from the point of view of employers. Six employers said that the standard which deals with the mechanism of stakeholder participation (employers, vocational education teachers, vocational students and graduates) in defining learning outcomes and programme development process, sounds as a formality and needs some precision. It was stressed that if employers will really get involved in designing educational programmes, this will have positive impact on programme outcomes. At the same time, it is not clearly defined how the involvement should take place. 5 employers said that there is no need for improvement. Another question dealt with what kind of standard/indicator should be added to the accreditation standards. 13 employers answered they do not see any need to add indicators and it was rather difficult for them to identify all the criteria. To the question if they are aware of accreditation procedures, 4 employers indicated they would favor reduction of formalities and bureaucracy and simplifying procedures. To the question whether the accreditation procedures satisfy the quality assurance requirements, 9 respondents stated that they would simplify accreditation and self-assessment procedures which on the other hand, will reduce its formal and bureaucratic character. 4 employers considered that the role of employers in this process is not clear and more involvement is needed. Two employers said that accreditation should be implemented by an independent agency. Six employers stated no changes are needed to the accreditation procedures, and seven employers found it difficult to answer the question due to lack of information. To the question (Q.13) in what ways employer organizations should be involved in quality assurance process? Ten employers said view of employers and employer organizations should be considered in the process of standard development in a specific field. Five employers said that thematic councils should be formed by the field with participation of employers and potential employers. Twenty-six employers said that employers should be involved in the programme development process and participate in expert groups. Three employers indicated they might participate in the practical component and assessment of specific cases.

9 Professors and Quality Assurance Staff Overall, 70 professors and staff from quality assurance departments have participated in the survey from four partner universities. It looked that professors and quality assurance staff were the most optimistic about the impact of quality assurance (internal and external). To the question, how effective existing accreditation standards were sixty-five professors and quality assurance staff answered out of seventy. The results showed that 33.8% of respondents said they found accreditation standards very effective, 44.6% said it was effective, 13.8% indicated it was more or less effective, 6.2% stated it was less effective and 1.5% said it was ineffective. more or less effective 14% less ineffective effective 1% 6% Very effective 34% effective 45% Figure 4.shows how quality assurance and academic staff assess programme accreditation To the question, do you consider internal quality assurance mechanisms effective at your institution, 95.3% of respondents answered positively, while 4.7 % gave negative evaluation. 36.8% of respondents said they have good knowledge on external mechanisms of quality assurance, 54.4% said they have profound knowledge, 5.9% indicated they have basic knowledge and 2.9% said they hardly have any knowledge. 54% 3% 6% 37% good profound basic no knowledge Figure 5. shows how quality assurance and academic staff assess their knowledge on external quality assurance When asked to evaluate their knowledge on the level of programme accreditation, 31.4% said they are aware of accreditation standards and 64.3% said they are very well aware. Out of seventy respondents sixty-seven answered the question, three responses were missing. To the question which standard is the most significant and why, 15.7% said the standard on teaching methodology and organization was significant, 18.6% indicated that provision with

10 learning resources (human, financial, physical, technical and informational) was significant, 44.3% indicated that all standards were important. 22.9% considered that the standard on aim of an educational programme, learning outcomes and relevance of the programme with outcomes was significant. 15.7 % said that enhancement of teaching quality was significant as improvement of learning outcomes of an educational programme was impossible without effective external and internal assessment mechanisms, and 10% gave high importance to the standard which deals with the Student Achievement and Individual work with students. Most of the respondents were positively disposed to internal quality assurance mechanisms (95.3%) and only 4.7 % gave negative assessment. Qualitative Data To the question which standard needs improvement and why, eighteen respondents said none of the standards need improvement, however, procedures and human resources involved needed some revisions. Nine respondents indicated that based on specific experience it s desirable to change some standards which have more general and abstract character. Twelve respondents said that standards and criteria which deal with financial provision of programmes, teaching and material resources, enhancement of the quality of teaching should be upgraded. Eight respondents indicated that the standard on assessment of student achievements and teaching methodology needs to be further developed. Four respondents said that it would be desirable if the criteria on internationalization of teaching, research and graduate employability are further defined. Three respondents indicated that the standards and criteria dealing with the graduate employment assessment, estimating the relevance of the labour market requirements, needs improvement. To the question if there is any necessity of adding new indicators to the accreditation standards, 86% of respondents said they do not support addition of new indicators. It was indicated that the indicators are sufficient at this stage and they shall be enhanced according to the local demand or international context.

11 III. Recommendations and Conclusion The survey outlined that stakeholders have information on external quality assurance, however, more awareness raising is needed to give better understanding about existing standards and procedures. It was also stressed that existing standards and procedures are considered satisfactory at this stage and periodically should undergo changes based on the labour market requirements and international context. Due to the survey results, students as one of the key stakeholders gave more significance to programme accreditation rather than authorization. On the other hand, this is easy to explain, as the programme accreditation is directly connected with student grants and remains the major marker of quality. It was also interesting that the majority of respondents supported the idea that the State should finally define the standards and procedures of external quality assurance, which highlighted the need to inform stakeholders about the importance of cooperation with the state to define common standards in line with the ESG. The survey results revealed that not enough efforts were made to involve employers and students in quality assurance process. The results suggest that the composition of expert groups and thematic councils could be reviewed in terms of composition and increase representation of employers as well as students. Based on all above mentioned, this leads to the fact that stakeholders have little information about their ownership in the quality assurance process and ways to contribute to it. Recommendations on schemes, mechanisms to increase stakeholder involvement should be developed.

12 Acknowledgements Special thanks to Nino Zhvania, Tea Kordzadze, Lela Kelbakiani, Maia Shukhoshvili, Lela Turmanidze, Mariam Shoshiashvili, Jana Moehren, CIEP for their contributions.