The second phase of the Mpumalanga Secondary Science Initiative (MSSI) was launched in Mpumalanga Province, South Africa in 2003. The MSSI seeks to improve the teaching and learning of mathematics and science in all secondary schools in the province over a period of three years. To achieve this goal an in-service system has been developed. The long-term research of the MSSI is aimed at examining the effect of the intervention on the three science learning outcomes as defined by the new curriculum. Tests were developed to assess 10 learners in each of Grades 8 and 9 in 40 schools, representing a 6% sample. This paper reports on the methodology of the baseline assessments undertaken and the extent of learners’ attainment levels in the three Natural Science outcomes. It then looks at the results in terms of learners’ socio-economic backgrounds. Finally it discusses some of the dilemmas encountered in this research, such as the issue of validity in the light of possible misalignments between the instrument, practice and policy.