OECD PROGRAMME FOR INTERNATIONAL STUDENT ASSESSMENT (PISA) Sixth meeting of the Board of Participating Countries 1-3 March 1999, Tokyo

Transcription

1 OECD PROGRAMME FOR INTERNATIONAL STUDENT ASSESSMENT (PISA) Sixth meeting of the Board of Participating Countries 1-3 March 1999, Tokyo SUMMARY OF MAIN OUTCOMES Introduction 1. The sixth meeting of the Board of Participating Countries (BPC) supervising the OECD Programme for International Student Assessment (PISA), was held 1-3 March 1999 in Tokyo, hosted by the Ministry of Education, Science Sports and Culture in Japan. 2. The primary objectives of this meeting were to: REVIEW progress in the implementation of PISA since the last meeting of the BPC in October DEVELOP a strategy for the effective dissemination of the outcomes from PISA. REVIEW the development of the analytic framework and instrumentation for the PISA student and school context questionnaires. REVIEW preparations for the implementation of the PISA field trial. REVIEW the establishment of PISA population definitions, sampling standards and sampling procedures as well as progress in the multilateral co-ordination of PISA national options. 3. The meeting was attended by delegates from Australia, Austria, Belgium, Canada, the Czech Republic, Denmark, Finland, Germany, Greece, Hungary, Iceland, Ireland, Italy, Japan, Korea, Luxembourg, Mexico, the Netherlands, New Zealand, Norway, Poland, Portugal, Sweden, Switzerland and the United States. The meeting was chaired by Mr. Owen from the United States. 4. Following the opening of the meeting by Mr. Owen, the adoption of the agenda [doc. ref. DEELSA/PISA/BPC(99)A1] and the adoption of the summary record from the fifth meeting of the BPC [doc. ref. DEELSA/PISA/BPC(98)32], Mr. Inoue welcomed participants on behalf of the Ministry of Education, Science Sports and Culture in Japan, highlighting the importance of PISA for informing policy development and the educational reform in Japan. 1

2 DEELSA/PISA/BPC(98)32 5. Afterwards, Mr. Alexander: Reported on progress in the implementation of PISA, with reference to the finalisation of the PISA assessment frameworks, the development of the field trial tests, the implementation of the procedures for ensuring the cultural appropriateness and linguistic and psychometric equivalence of the PISA instruments, and the development of the PISA context questionnaires. Outlined proposals for a strategy for the dissemination of PISA outcomes. Reported on progress in the development of the International Life Skills Survey (ILSS) as well as on work undertaken under the programme on the Definition and Selection of Competencies (DESECO), emphasising the importance to capitalise on synergies between such work and the further development of PISA. Outlined possibilities for the future development of PISA, highlighting the perspectives longitudinal survey components would open to PISA for examining: key transitions within education systems and between education and employment, the incidence and effects of early school leaving, the contribution of work experience to skill development and the attitudes, aspirations and behavior of youth entering the labor market. 6. Afterwards, Mr. Schleicher reported on how the issues that had been raised by the BPC at its last meeting had been addressed by the Secretariat and the consortium and led participants through each of the areas requiring further resolution by the BPC. 7. The BPC: WELCOMED the progress made in the implementation of PISA to date and the ways in which the decisions and recommendations made by the BPC at its last meeting had been implemented. WELCOMED the procedures and protocol for the visits of the consortium s National Centre Quality Monitors to participating countries [doc. Ref. DEELSA/PISA/ BPC(99)7] and NOTED the opportunity these visits provide for seeking feedback from national project teams on ways to improve PISA implementation procedures for the main study. REVIEWED the proposed changes to the procedures for the marking of open-ended test items and NOTED that these changes would not lead to overall increases in resource requirements at the national level. ASKED the Secretariat and the consortium to seek evidence from the field trial data that the PISA test items are indeed calculator neutral in the sense that students who do not usually use calculators in mathematics and science instruction will not be disadvantaged. NOTED the importance of validating, over the medium term, the effectiveness of PISA in measuring the preparedness of students for adult life from both national and international levels. 2

3 Dissemination strategy for PISA outcomes 8. Mr. Alexander introduced a first proposal for the dissemination of PISA outcomes, including both conceptual products as well as analytical reports and data products [doc. Ref. DEELSA/PISA/BPC(99)3]. Mr. Alexander noted that a sound method for conveying the results of PISA to key stakeholders in education and the general public was central to the success of PISA and needed to be planned for at an early stage. Such a dissemination strategy would need to provide for an effective international presentation of the comparative outcomes of PISA, paralleled by national analysis and interpretation of the results as developed by each country. The PISA dissemination strategy would need to meet the needs of a wide range of users - from governments wanting to learn policy lessons, through academics requiring data for further analysis, to the general public wanting to track how their nation s schools are progressing in producing world-class students. 9. Mr. Alexander noted that while the proposed publications would mainly draw on PISA data, a distinctive advantage of the OECD was that it could analyse PISA findings in a much broader policy context, drawing on an extensive programme of work in the area of education and training. 10. The BPC: WELCOMED the opportunity to begin working on matters of dissemination in a collaborative fashion from an early point onwards. WELCOMED the proposed dissemination strategy as an appropriate basis for the development of PISA reports and publications that would allow: i) to obtain the attention of policy makers through a first report on key results, ii) to sustain that attention through the provision of a broad range of indicators and analyses and iii) to provide a foundation for national policy review and development through a strong series of thematic reports. NOTED the significant resource implications for the implementation of the strategy at both national and international levels but considered the scope of proposals appropriate for ensuring that the investment currently made in the development of the PISA instruments would pay adequate returns for policy development. 11. For the further development of the first report on PISA results, the BPC: RECOMMENDED to focus the presentation of the initial report on a limited selection of outcomes (perhaps only 10 data tables) that would be well presented and communicated for a non-technical audience. Data shown in the first report would describe the distribution of results across countries using a numerical scale as well as proficiency levels, presented in the context of other national statistics. They could also describe the distribution of outcomes within countries, giving emphasis to the relative dispersion and location of the percentiles and highlight differences in performance between different demographic and social groups. EMPHASISED that the first report should highlight important policy messages emerging from PISA and therefore go beyond the ranking of countries, while recognising the need to also explain the relative standing of countries well in order to avoid that scientifically inappropriate inferences on country rankings would be made in the absence of clear explanations. 3

4 DEELSA/PISA/BPC(98)32 NOTED the importance to base the selection and presentation of data on careful analysis, even if the nature of the first report would not allow for an extensive discussion of data and analyses in the report itself. NOTED the importance to contextualise the findings in the international reports through national analysis and interpretation as determined by each country in ways that respond to the specific policy and educational context. The objective would be that countries would publish national reports simultaneously with the first international report. ASKED the Secretariat to work with participating countries on the development of a common reporting framework. Such a framework could guide both the preparation of the national reports as well as the selection of an initial set of international data and indicators which would be made available to participating countries in advance in order to allow for an international dimension in the national reports. NOTED the need to identify and devote significant resources to the production of national reports within the very limited time available between the release of national data and the publication of the first international report. ASKED the Secretariat to take into account the need of some countries to translate national and international reports into multiple languages when establishing the schedule for the production of international PISA publications. NOTED the importance to provide sufficient explanatory information about PISA in the first report to help the lay person understand what PISA is measuring. SUPPORTED the proposal to limit the first report to around 50 pages. ASKED the Secretariat to clarify the use of proficiency levels in PISA reports and publications. NOTED that the reports should recognise the different nature of the PISA major and minor domains while not diminishing the minor domains for which PISA would still be able to report reliable outcomes. ASKED the Secretariat to organise a seminar with the BPC on matters of dissemination and co-operation with the press at an appropriate point in time and to consider the preparation of audio/visual aids for the release of the reports. DECIDED not to use the term headline reports so as to avoid any impression of superficiality in the presentation of the results. 12. In relation to the further development of the indicators report, the BPC: SUGGESTED that the presentation should go beyond a basic display of the data as currently foreseen and include explanations and interpretations that would guide readers on how to draw valid conclusions from the indicators and to correctly interpret country differences in PISA results. 4

5 NOTED the importance to make PISA data widely accessible, as soon after the release of the first report as practicable, in order to support an opportunistic use of the data by a wide range of constituencies. NOTED that the indicators report should provide for measures of the curricular and cultural coverage of the PISA tests in participating countries. NOTED that the OECD publications Education at a Glance and Education Policy Analysis would continue to analyse and report data emerging from PISA. 13. In relation to the development of proposals for thematic reports, the BPC: WELCOMED the proposals for the thematic reports entitled Learning for Life, A Profile of Under-Achievement and The School Effect. ASKED the Secretariat to develop further topics for thematic reports, addressing: - equity related issues that could be dealt with through analysing the relationship between student and school performance, on the one hand, and the economic, social and cultural capital of students and their families on the other; - the relative importance of student and school factors in determining learning achievement and the roles of schools in moderating the relations between social and economic status and student performance; - parental involvement and support in student and school life; - gender differences in student performance; - the contribution of PISA to assessing life skills and the analysis of students abilities to organise and regulate their own learning, to learn independently and in groups, and to overcome difficulties in the learning process; - policy options for school improvement in cross-curricular and non-curricular areas; - relationships between learning outcomes and curricular differentiation and career tracks; - evidence of important relationships between inputs/processes and outcomes; NOTED the importance of thematic analyses and reports also at the national level. ASKED for an executive summary with main findings and their implications for each of the thematic reports. 14. In relation to the development of ongoing dissemination products, the BPC: NOTED the importance of an ongoing dissemination process that would facilitate an active international exchange among the stakeholders in PISA, both at the policy and scientific levels and also among schools, teachers and students involved in the PISA assessment. 5

6 DEELSA/PISA/BPC(98)32 WELCOMED the production of a PISA brochure that would provide a concise description of what PISA consists of and what it is for, including what is being tested, how the results are expressed and who the key stakeholders involved are. WELCOMED the production of a PISA newsletter keeping all in touch with what has been produced so far and what is coming up, on a twice-yearly basis. WELCOMED the development of a PISA Web Site that would provide entry points to important information on PISA for different types of users, including the general public, and ASKED the Secretariat to ensure that such a Web site would provide links to the national PISA Web sites established by participating countries. EMPHASISED the importance of engaging educators in the PISA process but NOTED that security and legal concerns would not allow for a centrally administered Web site connecting student, teachers and schools participating in PISA at this point. EMPHASISED the importance of developing other instruments that would make PISA accessible to educators, such as a PISA toolbox with international and national components. ASKED the Secretariat and the consortium to consider options for the development of software that would allow participating countries to report school-level results of PISA to participating schools. Assessment framework and instruments 15. Mr. Schleicher introduced the revised PISA assessment frameworks and the plans for their publication. 16. The BPC WELCOMED the manuscript with the revised PISA assessment frameworks [doc. Ref. DEELSA/PISA/BPC(99)8] as an adequate basis for a public product. 17. For the further preparation of the manuscript the BPC: SUGGESTED to prepare a summary that would explain the design and policy relevance of PISA and that would make the concepts and methodology developed by the PISA expert groups more widely accessible for users both in educational policy and practice. This summary should be made available to Member countries as soon as possible for translation into their national languages. ASKED to distinguish those aspects of PISA which are being addressed in the first PISA survey cycle from longer-term objectives for which specific measures would only be developed in future survey cycles. ASKED to use the term assessment instead of test and the term life skills instead of core skills and key skills. AGREED to submit further comments on the manuscript to the Secretariat before 19 March 1999 and to review the final manuscript between 24 and 31 March

7 18. Mr. Owen drew the attention of participants to the importance of ensuring the strict confidentiality of all PISA assessment instruments. 19. The BPC: NOTED that the need to maintain the security of the PISA assessment instruments would not allow an advance release of any of the PISA test items, even for illustrative purposes. NOTED that the Secretariat and the consortium will, after the field trial, identify a set of test items that will provide a balanced and appropriate representation of all aspects of the PISA instruments and that can then be used by participating countries to describe and illustrate the PISA instruments. ASKED the Secretariat to develop a proposal for a longer term policy for the release of items and instruments for discussion by the BPC at its next meeting. Context questionnaires and analysis plans 20. Mr. Adams reported on progress in the development of the PISA context questionnaires and on efforts underway to better focus the context questionnaires and to achieve greater depth and clarity in areas of particular relevance to the BPC s policy priorities. Mr. Adams also outlined the constructs and instruments that are being proposed to address the BPC s priority domains [doc. Ref. DEELSA/PISA/BPC(99)4]. 21. Afterwards Mr. Owen reported from a meeting of a Network A subgroup held January 1999 at which the group had: reviewed the current version of the PISA context questionnaires and their relationship to both the BPC s policy priorities and Network A s plan for the analysis and presentation of outcome indicators; assessed the potential and the limitations of the current PISA context questionnaires for policy oriented analysis and indicator development; and explored options for the further improvement of the questionnaires [doc. Ref. DEELSA/PISA/BPC(99)5]. 22. The BPC: WELCOMED progress made in rationalising the questionnaires but NOTED that further work was necessary before these instruments would address the analytic needs of PISA. WELCOMED the criteria proposed by the consortium for the construction of the main study questionnaires (the questionnaires should contribute to indicators of enduring cross-national relevance, focus on areas that are amenable to policy development, be applicable in most participating countries and be based on standardised cross-national coding schemes). REQUESTED that a framework for the context questionnaires be developed as quickly as possible that would include information on how the information collected through the context questionnaires would be utilised for analytic purposes. WELCOMED the conceptual work on the definition and measurement of the socioeconomic status and social and cultural capital [Appendices 3 and 4 of document DEELSA/PISA/BPC(99)4] and NOTED the importance to ensure that this work was adequately addressed in the PISA context questionnaires. 7

8 DEELSA/PISA/BPC(98)32 NOTED that further work was needed to reflect the subjective personality of students in the student context questionnaire (attitudes, values and judgements). CONSIDERED that the concept of the school in the PISA school questionnaire would need to better reflect the full range of institutional structures that exist in participating countries. INVITED the INES Network A to continue work towards a recommendation of priorities for the different components in the context questionnaires. WELCOMED the consortium s plans for strengthening the developmental process and the international expertise involved in the preparation of the questionnaires. AGREED to submit comments on the framework document DEELSA/PISA/BPC(99)4 to the Secretariat before 22 March EMPHASISED the need for validity studies at both national and international levels, in particular to corroborate the measures of economic, social and cultural capital as well as the information obtained on family structure and home engagement. Population coverage and survey and sampling standards 23. Mr. Schleicher reported on progress in the implementation of PISA standards for population coverage and sampling [doc. ref. DEELSA/PISA/BPC(99)4]. 24. The BPC: NOTED that the implementation of the PISA population definitions and sampling standards was generally proceeding well and that the impact of differences in the coverage of 15 yearolds on mean country performance was, in most countries, expected to remain well within the tolerable margins of error for PISA. SUPPORTED the stringent quality standards that had been established for PISA. SUPPORTED the principles for the treatment of students with special educational needs but NOTED the difficulties some countries were facing in administering PISA instruments in special education institutions. ASKED the Secretariat and consortium to consider the development of a screening test comprising a single booklet with very easy test items that could be administered to assess students that could not meaningfully participate in the full PISA assessment. ASKED the Secretariat and the consortium to document the categories for population exclusions and to report exclusions in the different subcategories on a comparable countryby-country basis for review by the BPC at its meeting in October NOTED that no special accommodations for the administration of PISA tests to students with physical disabilities were planned for PISA at the international level but that countries could pursue this at the national level. 8

9 Evaluation and development of the terms of reference for the second PISA survey cycle 25. Mr. Alexander reported on discussions within the INES Steering Group concerning objectives and procedures for assessing the extent to which PISA is delivering results that are useful, relevant and valid. He also reported on plans for a meeting of national and international experts and administrators involved in the implementation of PISA (including key stakeholders in the consortium and the functional expert groups, representatives from the OECD and a selection of National Project Managers) that would be held 4-6 June 1999 with the objective to identify strengths and weaknesses in: the item and test development process, the PISA survey operations, quality standards and procedures for quality assurance, and the PISA communication mechanisms. 26. The BPC: NOTED both the importance and complexity involved in evaluating whether PISA is delivering results that are useful, relevant and valid and whether the process of implementation is efficient, timely, and cost-effective. EMPHASISED that the evaluation process must be undertaken under the auspices of the BPC and endorsed: i) the involvement of all of the key stakeholders and available expertise in the evaluation process at both international and national levels as well as ii) the role of the OECD Secretariat in reflecting the various views in reporting to the BPC. NOTED that the Secretariat had been asked by the INES Steering Group to develop an evaluation plan for PISA, in consultation with the BPC, the INES Steering Group and Network A and ASKED the Secretariat to consult the BPC on the proposed means and methodology by April NOTED the meeting in June 1999 to assess the PISA field trials. 27. Mr. Owen explained Network A s plans for the development of the terms of reference for the second survey cycle of PISA. He indicated that a first meeting of a Network A subgroup on this would be taking place at the beginning of May, followed by the development of a first draft for the terms of reference for discussion by the BPC at its meeting in October. Afterwards, a second draft would then be prepared for the BPC at its meeting in March 2000 for final review. 28. The BPC: ASKED to be informed about the substantive development of the terms of reference, the proposed tendering procedure and the estimated costs for the implementation of the second PISA survey cycle at the earliest point possible. NOTED that the invitations to OECD Member countries for participation in the second survey cycle would be issued as soon as the framework for the terms of reference for the second survey cycle had been adopted by the BPC and cost estimates had been established. WELCOMED the development of a sound methodology and cost estimates for a longitudinal component for PISA that would allow to relate PISA skill measures to, education and labour market pathways, the incidence and effects of early school leaving, the contribution of work experience to skill development and the attitudes, aspirations and behaviour of youth entering the labour market. Developmental work could draw on the Canadian experience with linking 9

10 DEELSA/PISA/BPC(98)32 the national Youth In Transition Survey with PISA. The BPC would DECIDE on the development of an international option for a longitudinal survey component when finalising the terms of reference for the second survey cycle. ASKED Network A and the Secretariat to clarify options and resource implications for the development of a second PISA target population in the terms of reference for the second PISA survey cycle. NOTED the importance to start with the development of the assessment frameworks for the second survey cycle as early as possible. Programme of work and cycle of meetings 29. Mr. Owen introduced the programme of work for the BPC during the period March 1999 to November 2001, including the proposed cycle of meetings of the BPC [doc. Ref. DEELSA/PISA/ BPC(99)1]. 30. The BPC: WELCOMED the proposed programme of work but suggested to place greater emphasis on written consultations of members of the Board in order to reduce the number of meetings of the BPC. In particular, the BPC would aim to hold only one more meeting in October 1999 at which the directions for the development of the main study instruments would be determined. The BPC Executive Group was mandated to assess, in December 1999, whether these directions were adhered to and to report its deliberations to the BPC in writing. If important questions remained for members of the BPC, the chair would call a special meeting of the BPC either at the end of December 1999 or at the beginning of January ASKED the Secretariat to prepare a schedule that would relate the meetings and programme of work of the BPC to the programme of work of PISA National Project Managers, Network A and the various developmental groups. ASKED to be informed on developments in Network A on a regular basis. 31. The next meeting of the BPC will be held 4-5 October 1999 at the OECD Headquarters in Paris with the objectives to: Assess the implementation of the PISA field trial (quality control and operations). Assess the implementation of the PISA procedures for ensuring the cultural appropriateness and the linguistic and psychometric equivalence of the PISA tests. Review the preliminary assessment results from the PISA field trial and decide on the final shape of the main study instruments. Establish directions for the finalisation of the PISA context questionnaires for the main study. Review the options and budgetary implications established by Network A for the terms of reference for the second PISA survey cycle. 10

11 Adopt the PISA budget for the year Elect the PISA officers. 11