Bridging the gap : optimising a feedback system for monitoring learner performance

Abstract

Globally, a wealth of educational data has been collected on learner performance in a bid to improve and monitor the quality of education. Unfortunately, the data seem to have had only limited influence on learning and teaching in classrooms. This thesis aimed to bridge this gap between the availability of learner performance data and their use in informing planning and action in schools. A design research approach was used to optimise the feedback system for the South African Monitoring system for Primary schools (SAMP). Design research aims to produce both an intervention to address a complex real-world challenge and to develop design guidelines to support other designers faced with similar challenges in their own context. In this research, the process of developing and improving the feedback system was also used to examine ways of facilitating the use of the feedback. Multiple cycles of design, implementation and evaluation of four different prototypes of the feedback system were conducted, employing evaluations from both experts (e.g. Dutch and South African academics, research and educational psychologists, instrument designers and teacher trainers) as well as school users (teachers, principals and HoDs). Mixed methods were employed throughout the study, with different sub-samples of school users sampled from the population of 22 schools (English, Afrikaans and Sepedi) in the Tshwane region participating in SAMP. The various research cycles incorporated interviews, observations, journals, questionnaires, the Delphi technique and expert evaluations to examine not only data-use, but also aspects such as problem-solving, planning, data-literacy and attitudes towards evidence-based practice in the schools. Data was analysed using Rasch Modelling, descriptive statistics and computer-aided qualitative data analysis. The study showed that an effective feedback system facilitates appropriate use through a gradual process of enlightenment, is flexible and responsive to user inputs, values collaboration and includes instrument, reporting and support components in its design. An optimum feedback system also positively influences school feedback and monitoring culture by providing opportunities for positive experiences with feedback and increasing data-literacy. This improves the chances of feedback being used for planning, decision-making and action in the schools. An effective feedback system must also offer a comprehensive package to accommodate different users, with various levels of data sophistication, functioning in diverse contexts. The research also showed that an effective feedback system mediates thinking about educational instruction and curriculum and can therefore be a potent change agent. Use of clear, simple, intuitive data presentation in the feedback system allows for experiential learning to increase user data-literacy. The design research approach employed in this study offers an appropriate and powerful approach to adapting, developing and optimising a feedback system. User involvement in design research ensures greater contextualisation and familiarity with the system, while engendering trust and a greater sense of ownership, all of which increase the receptiveness and responsiveness of users to feedback. Finally, the research also contributed design guidelines for other developers of feedback systems, an integrated conceptual framework for use of monitoring feedback and a functioning feedback system employed by 22 schools in the Tshwane region.