WHAT DO THEY KNOW? A summary of India s National Achievement Survey, Class V, Cycle 3, 2010/11. Supported by SSA TECHNICAL COOPERATION FUND

Transcription

1 WHAT DO THEY KNOW? A summary of India s National Achievement Survey, Class V, Cycle 3, 2010/11 Supported by SSA TECHNICAL COOPERATION FUND

2 Contents Introduction 3 States of change why we evaluate 4 The history of NAS 4 Measuring progress in education 4 Turning test results into trends methodology 5 Developing Class V tests 5 Selecting samples and participants 6 Analysing the results 6 Snapshot of a nation the results 7 National trends 7 Achievement by subject 8 Background factors survey results 14 Data is just the start what next? 15 Making the most of the results 15 Ongoing improvements 15 2

3 Introduction The 2010/11 National Achievement Survey (NAS) of Class V students has been significantly updated in line with international best practice in student assessment. Conducted by the National Council of Educational Research and Training (NCERT), this marks the most comprehensive educational assessment survey in India to date. Between March and May 2010, the survey tested over one hundred thousand students in language, mathematics and environmental studies. In addition, questionnaires were used to collect information on schools, teachers, and pupils school experiences and home backgrounds. This booklet summarises the NAS methodology and findings in order to: explain improvements made in assessment techniques provide a snapshot of achievement levels in Class V highlight areas for further investigation Supporting the National Council of Educational Research and Training The UK government-funded Technical Cooperation Fund is supporting NCERT to reach international professional standards in National Achievement Surveys. To achieve this, technical assistance has been provided in the following four areas: Theory, research, statistics and international experience relating to learning assessment systems Test development and psychometric analysis. Administering, processing and analysing large-scale assessments Data management and reporting 3

4 States of change why we evaluate India has made a significant investment in its education. The government s flagship programme Sarva Shiksha Abhiyan (SSA) is designed to ensure access, equity and quality in elementary education. The nation now needs reliable information about students achievement in order to judge the quality of education provided. The history of NAS Carried out as part of SSA, the NAS aims to collect reliable information about the achievement levels of students in government and government-aided elementary schools. In 2000, NCERT s NAS programme was incorporated into the SSA programme. The plan was to carry out three NAS cycles, each cycle covering three key grades: Class III, Class V, Class VII/VIII. The Baseline Achievement Survey (BAS) was carried out in , followed by the Midterm Achievement Survey (MAS) in The experience gained through these initial two cycles made the value of the NAS clear, and the surveys were made an ongoing feature of the national education system. To mark this shift from stand-alone surveys to continuous assessment, the Terminal Achievement Survey (TAS) has been renamed Cycle 3. Measuring progress in education The NAS is a useful tool for teachers and policymakers alike to establish what students are achieving in core subjects and to identify any areas of concern. By repeating the NAS at regular intervals, the data can be used to measure trends in education achievement levels and measure progress made by SSA and other education reforms. All three Classes are tested in mathematics and language. In Class V, students are also tested in environmental studies (EVS), while Class VII/VIII completes tests in science and social science. Sarva Shiksha Abhiyan towards universal elementary education The 86th amendment to the Constitution of India made free and compulsory education for children aged 6-14 a fundamental right. The government s SSA programme is providing support to achieve this goal. Spanning the whole country the programme addresses the needs of nearly 200 million children. In areas with no or inadequate schooling facilities, SSA is building new schools or improving existing infrastructure. Teaching capacity is being improved through recruitment and extensive teacher training and many other inputs. See website for more details: 4

5 Turning test results into trends methodology For the first time in India, the Class V NAS has established robust baselines for achievement in reading comprehension, mathematics and environmental studies. The introduction of sophisticated, internationally-recognised assessment methods means that future trends in educational standards can be judged with confidence against these new baselines. Developing instruments is a crucial activity in any survey. In preparation for NAS Class V, NCERT developed test booklets in language, maths and environmental studies. In addition, questionnaires were designed for students, teachers and schools to capture information on school environment, teacher qualifications and status, and student background. Developing Class V tests The Class V NAS differs from previous cycles of the survey as it uses Item Response Theory (IRT) to place results on meaningful scales, which are independent of the particular tests used. This method is in line with international best practice. What is IRT? Item Response Theory, or IRT, uses a mathematical model to calculate the probability that a student will respond correctly to a specific test question, based on the student s level of ability and the degree of difficulty of the questions concerned. Previous Class V survey results were reported as the percentage of correct answers. This meant that results were specific to that year s test and could not be compared to results from other tests or survey cycles. The scales created using IRT, however, permit future comparison in each subject. For each of the three subjects, the Class V syllabus and textbooks from across the country were used to develop assessment frameworks, which described the knowledge and skills to be covered in the tests. Once developed, the test booklets were translated into 15 regional languages. The tests were then piloted in 21 states to assess how the items worked across the different languages. Unlike previous surveys, this round of the Class V survey used three test booklets per subject rather than one. This allowed assessment in greater depth. To enable comparison across all three tests, a block of common questions or anchor items was used in all booklets. For each subject the NAS presents item maps which arrange items according to their level of difficulty. The scale score of an item on the map indicates the ability level of students who, on average, are likely to answer it correctly. Hence, the maps can be used to describe what students at different levels of ability know and can do in each subject. 5

6 Selecting samples and participants Cycle 3 of NAS Class V used tests and questionnaires to gather information from 122,543 students in 6,602 schools across 31 states and union territories (UTs). In each state and UT, the sample was generally selected in three stages. Firstly, districts were selected based on the number of Class V students enrolled. Secondly, the requisite number of schools was selected. Finally, the required number of Class V students in the school was selected at random. This summary document concentrates on results from 20 states where the sample size was higher than 80% and students were tested while still in Class V. In some states, low coverage and delays in carrying out tests mean that the results should be read with caution. Analysing the results Following IRT analysis, the Class V test results have been placed on a scale, which was normalised with a mid-point of 250. Achievement scores on this scale cannot be interpreted as being equivalent to any particular score on any particular test. Instead, they can be used to make meaningful comparisons between states. The results can identify relatively high achieving states and those states in which achievement is relatively weak. Not only does the survey report the average performance in each state, it also presents the range of students performance across the ability range. This range of achievement is presented as five key percentiles : 10th, 25th, 50th, 75th and 90th. For example, the score at the 25th percentile is that which 75% of students achieve or surpass, and the score at the 90th percentile is that which 10% of students achieve or surpass. The range between the 25th and 75th percentile, or the inter-quartile range, represents the middle 50% of students. The smaller the inter-quartile range, the greater the homogeneity within the state s student cohort. This allows us to make judgements about issues of equity. 6

7 Snapshot of a nation the results Overall the NAS reveals significant differences in the average achievement levels of students between states. Some difference may be accounted for by contextual factors, but on balance the results suggest that the quality of educational outcomes is far from equal across the country. The survey s findings in reading comprehension and mathematics are of particular concern. While the highest achieving students can read with understanding and can successfully solve more complex mathematical problems, the bottom 25% of students appear to function at low levels of literacy and numeracy. For example, they cannot read with understanding if the context is unclear and they cannot solve calculations involving basic mathematical operations. NAS results show great diversity in achievement between the highest and lowest performing students in some states. Inequity of outcomes suggests that, in some states, more able students are receiving support while their less able peers may not be enjoying adequate opportunities to reach more acceptable levels of learning. National trends The NAS found the following national trends in Class V students: There was no significant difference between boys and girls achievement levels in reading, maths and environmental studies. There was no significant difference in achievement between urban and rural students. But Students from minority groups, comprising Scheduled Castes, Scheduled Tribes and Other Backward Categories, scored significantly lower than students in the General category 7

8 Achievement by subject: Reading Jammu & Kashmir Himachal Pradesh Chandigarh Haryana Punjab Uttarakhand Delhi Rajasthan Uttar Pradesh Bihar Gujarat Madhya Pradesh Tripura Orissa Chhattisgarh Andhra Pradesh Karnataka Tamil Nadu Puducherry Andaman & Nicobar Islands State's average score is not significantly different to group average State's average score is significantly above group average State's average score is significantly below group average Not reported 8

9 Reading comprehension State or Union Territory Interquartile range 75th-25th Average score A & N Islands Andhra Pradesh Bihar Chandigarh Chhattisgarh Delhi Gujarat Haryana Himachal Pradesh Jammu & Kashmir Karnataka Madhya Pradesh Orissa Puducherry Punjab Rajasthan Tamil Nadu Tripura Uttar Pradesh Uttrakhand Group distribution/ average Scale score 10th 25th Percentiles of performance 50th 75th 90th The language exam was designed to test a range of reading skills: locating information, identifying and interpreting ideas, and demonstrating understanding of information that may not be explicit in the text. There was a great variation in students ability in reading comprehension. At the lower end of the scale, about 65% of students were able to locate information given in a table. Intermediate students could do more, for example, around 50% could identify the cause and frequency of an event, and draw conclusions about characters from information given in the text. The relatively small proportion of students, approximately 32%, at the higher end of the scale could read with understanding and infer things alluded to, but not given explicitly in the text. 9

10 Achievement by subject: Mathematics Maths Jammu & Kashmir Himachal Pradesh Chandigarh Haryana Punjab Uttarakhand Delhi Rajasthan Uttar Pradesh Bihar Gujarat Madhya Pradesh Tripura Orissa Chhattisgarh Andhra Pradesh Karnataka Tamil Nadu Puducherry Andaman & Nicobar Islands State's average score is not significantly different to group average State's average score is significantly above group average State's average score is significantly below group average Not reported 10

11 Mathematics State or Union Territory Interquartile range 75th-25th Average score A & N Islands Andhra Pradesh Bihar Chandigarh Chhattisgarh Delhi Gujarat Haryana Himachal Pradesh Jammu & Kashmir Karnataka Madhya Pradesh Orissa Puducherry Punjab Rajasthan Tamil Nadu Tripura Uttar Pradesh Uttrakhand Group distribution/ average Scale score 10th 25th Percentiles of performance 50th 75th 90th The Class V mathematics exam tested students achievement levels in arithmetic, algebra and geometry. In all three domains, students demonstrated a very wide range of ability =? About three-quarters of students successfully found the answer to the sum However, fewer than half could answer the question How much greater is 555 than 198? Disappointingly, only about a third of students could find the difference between two numbers given in decimal notation even when four alternative values were given. 11

12 Achievement by subject: Environmental studies EVS Jammu & Kashmir Himachal Pradesh Chandigarh Haryana Punjab Uttarakhand Delhi Rajasthan Uttar Pradesh Bihar Gujarat Madhya Pradesh Tripura Orissa Chhattisgarh Andhra Pradesh Karnataka Tamil Nadu Puducherry Andaman & Nicobar Islands State's average score is not significantly different to group average State's average score is significantly above group average State's average score is significantly below group average Not reported 12

13 Environmental studies State or Union Territory Interquartile range 75th-25th Average score A & N Islands Andhra Pradesh Bihar Chandigarh Chhattisgarh Delhi Gujarat Haryana Himachal Pradesh Jammu & Kashmir Karnataka Madhya Pradesh Orissa Puducherry Punjab Rajasthan Tamil Nadu Tripura Uttar Pradesh Uttrakhand Group distribution/ average Scale score 10th 25th Percentiles of performance 50th 75th 90th The environmental studies (EVS) exam tested students knowledge of social studies and science. In EVS, students of average ability and above could extract information from a basic bar-chart and could interpret a simple family tree. However, only about a half of students appeared to know that the part of a carrot we eat is, in fact, the root. And only a third of the students successfully identified energy sources and their by-products. 13

14 Background factors survey results The NAS survey used three questionnaires designed for schools, teachers and pupils, to collect information on background factors that could potentially influence learning outcomes. The results suggest that coming from a larger family is related to lower than average performance. Students with only one sibling outperform by a small, but statistically significant amount, those who have two or more brothers and sisters. This effect is lowered, but does not disappear entirely, when key factors, including socio-economic background, are accounted for. Average performance Average performance Incentive schemes such as free uniforms and full scholarships appear to be related to higher achievement. However, since these interventions target disadvantaged groups rather than the whole student population, these findings should be treated with some caution until further investigation is carried out. Students who are given homework every night and have it checked by their teacher do better than their peers. Perhaps surprisingly, after allowing for key background variables, pupils who have a tutor do worse than those who do not. One plausible explanation for this is that families are more likely to hire a tutor if their child is already struggling. Activities such as watching TV, hanging out with friends, using the internet and doing homework all appear to be related to higher attainment. This effect is reduced after allowing for the key variables, suggesting that this may be due, to some extent, to the availability of resources. Playing on the computer is not associated with higher attainment and, after allowing for the key background variables, appears to have a negative impact on achievement levels. Nearly 90% of the students who took part in the Class V survey reported that they like being in school 14

15 Data is just the start what next? The Class V NAS is important in its own right, but it cannot answer every question on the quality of India s elementary education. National Achievement Surveys provide reliable information about levels of educational outcome and also give quantitative data about the relationships between background variables and outcomes. This objective information is an important starting point for identifying challenges within teaching and learning and deciding on the best way to improve educational processes. For example, NAS results can identify the questions that students at different levels of ability find most difficult. It is then for curriculum and teaching specialists to decide how such difficulties can be resolved by, for example, changing teaching practice and improving textbooks and other materials. Making the most of the results Each NAS cycle should be followed by further analysis and action by national and state level decision makers. For example, state educational authorities should review findings within their local context and against local priorities. It is recommended that state-specific reports and corresponding action plans are produced. Ongoing improvements The Class V Cycle 3 NAS was more sophisticated than earlier surveys, however conducting large-scale assessments in India remains technically challenging. Three lessons learnt during this latest survey offer the potential to improve future NAS cycles: Increasing the quality of quantitative information about schools and enrolment will help eliminate current difficulties in drawing truly representative samples. Investment in national and local education information management systems would be of great benefit. Development of future survey tests should be based on a national debate about which curriculum domains should be given priority should surveys focus on subject curricula or on a limited number of critical competencies such as numeracy and reading literacy? To maximise the value of National Achievement Surveys the findings and implications need to be communicated to a wider public, not just education specialists. The diagram below shows how data from NAS is the starting point in a continuous cycle of quality improvement in education. Remedial Action National Assessment Diagnosis Analysis Find out more: The full NAS Class V report will be available online. To read this and other evaluation reports relating to the SSA, please visit the SSA-TCF website: 15

16 WHAT DO THEY KNOW? A summary of India s National Achievement Survey, Class V, Cycle 3, 2010/11 SSA Supported TC FUND by SSA TECHNICAL TECHNICAL SERVICES COOPERATION AGENCYFUND G.B. Pant Block NCERT Sri Aurobindo Marg New Delhi T: E: