The validity of value-added measures in secondary schools

Abstract

The issue of quality education is a critical topic of discussion, for South Africa facing the challenge of implementation amidst a plethora of progressive policies. This research project is undertaken in collaboration with the Curriculum, Evaluation, and Management Centre (CEM) at Durham University in the United Kingdom. The Middle Years Information System (MidYIS) project was developed with the aim of providing schools with information on how learners would perform at the end of two national examinations namely Key Stage 3 and General Certificate in Secondary Education, in addition to providing value-added information. The purpose of the research reported here is to describe the procedures undertaken to explore the feasibility of implementing the MidYIS system in the South African context. The research was guided by two main research questions. The first research main research question is how appropriate is the Middle Years Information System (MidYIS) as a monitoring system in the South African context? The word “appropriate” here interrogates the suitability of the MidYIS system for South Africa looking specifically at validity and reliability issues. This non-experimental study used a mixed methods design, rooted in pragmatism, to explore validity and reliability issues of using MidYIS as a possible monitoring system that would provide a balanced view of the school’s contribution to academic gains made by learners. The sample included in the study ranged from National Department of Education officials (two officials from curriculum and assessment), Provincial Department of Education officials (one mathematics specialist, one language specialist and one specialist from the Gauteng Department of Education Office for Standards in Education), specialists in the field of language, mathematics, and psychology as well as 11 schools. In particular content-related validity (including curriculum validity), construct-related validity, and predictive validity were examined while inferences drawn with regard to reliability were done by means of internal consistency reliability. From a curriculum perspective for content-related validity, it was found that there was moderate curriculum validity for language while inferences drawn for mathematics were substantially stronger. For content-related validity from a psychometric perspective, it was found that there was overlap between the domain of developed abilities and the MidYIS assessment. Construct-related validity was explored by means of Rasch analysis and it was found that items in the MidYIS assessment tend to form well-defined constructs. Predictive validity was explored by means of correlation analysis between the MidYIS assessment and school-based results in language and mathematics. The analysis shows that it could be possible to use the MidYIS assessment for prediction purposes. However, additional research would be needed to explore this facet of validity further with a larger sample and using standardised school-based results. The MidYIS assessment was found to be reliable for the sample as a whole as well as for population groups within the sample. The second main research question extends the first research question. If MidYIS is valid, with South African adaptations, and reliable, then what factors on a school, classroom, and learner-level could have an effect on learner performance. Thus, the second main research question is which factors could have an effect on learner performance and therefore inform the design of the monitoring system? In order to explore factors, multilevel analysis was undertaken on the various levels within the school system namely the principals, mathematics and language educators, as well as learners who completed questionnaires. It was found that four learner-level factors (with whom learners live, mother’s level of education, importance of mathematics and importance of English), one educator level factor (challenges to assessment due to lack of in-service training) and two school-level factors (educators make use of monitoring systems and encouraging academic achievement) seem to have an effect on the performance of learners.