An evaluation of an intervention programme on Automotive Service Technicians using Kirkpatrick’s framework

Abstract

This dissertation reports an evaluation study which was done with an educational programme for Automotive Service Technicians which was adapted for South African conditions and derived from a programme used internationally, and originally developed in Schweinfurt, Germany in 2005. The programme was designed to answer to particular problems experienced during automotive driveline-component installations. Since the inception of this programme, ZF Germany had been training representatives from their different subsidiaries over the world on the essential elements of automotive driveline installation protocol. The representatives were trained to adapt the core programme in accordance with the particulars of the vehicle populations in each respective country, and the researcher has performed this task The aim of this research was to evaluate the effectiveness of one particular module known as “Guidelines to clutch replacement” with regard to bringing about the desired changes in knowledge, attitude and behaviour within the trainees. Previous anecdotal feedback from the industry had suggested that the programme had been helpful in the reduction of installation errors, but the extent of the successes and failures of the programme had been unknown until this study. The training department at ZF South Africa was tasked to design further training modules based on the findings of the module under study in order to establish the successes and failures of the core concept for improvement of successive programmes. The approach for this programme evaluation was utilization-focused which allowed the researcher to choose from and combine a variety of data collection strategies over the complete range of summative and formative evaluation approaches. However, in keeping with the stated aim of this study, this study had been limited to a summative inquiry by employing a quantitative data collection strategy at the hand of a quasi-experimental research design. This research report presents the findings of a one-day intervention programme that was offered to Automotive Service Technicians in the Gauteng area. The conceptual framework that was adopted for the research was based on the four level evaluation framework of Kirkpatrick (1998) with the first three levels having been empirically tested and the fourth level discussed on the basis of empirical information. Findings suggest that although the levels of knowledge-acquisition could not to be considered as high, behaviour modification had indeed been observed to be in alignment with the clutch-installation-protocol and almost all the respondents had adopted the protocol as their preferred way of executing clutch installations. In addition, most respondents found the programme to be pleasant and of a high utility value. Certain problems with the programme became evident, such as the pace having been too fast; printed hand-outs were not considered to have high utility value, and sensitivity to personal and cultural differences were found to be lacking. The low levels recorded for knowledge acquisition may be language related which possibly relates to the fast pace of the course. The research findings suggest that the course should be spread over two days instead of one day and be augmented with practical demonstrations and re-designed printed hand-outs. In order to effectively measure level four of the Kirkpatrick framework, criteria of concern should be negotiated with participating organisations in order to provide relative data for answering research questions on this level. Procedures for collecting data over the course of several years need to be established and agreed upon by all stakeholders for such data to be reliable and valid in the inclusion of a time-series study. Regarding a relatively simple programme such as the programme under study with programme objectives that have a predominant procedural-knowledge focus, the Kirkpatrick framework has been found to be effective and its procedures may be applied in other industry-based training programmes. An added academic contribution to the previous one is that the Kirkpatrick framework as utilised in this study has shown that the framework offers a high utility value for fast-paced short courses where contact time with trainees are limited and evaluation designs need to fit in with the practical limitations. The high utility value of the Kirkpatrick framework became evident in the findings of this study where transfer of learning had evidently taken place regardless of possible learning problems such as language barriers.