Exploring the construct validity and reliability of the English Comprehension Test

Abstract

The consideration of a multi-cultural and multi-lingual context makes developing a test not only a challenge but a worthwhile venture. The empirically designed English Comprehension Test (ECT) is theorized to be measuring Verbal Reasoning and thus the construct validity and reliability needed to be established. The ECT was initially developed with the intention of screening for English comprehension, but this has since shifted to creating a screening tool for Verbal Reasoning. The ECT consisted of two test versions which had sample sizes of 597 for the ECT version 1.2 and 882 for the ECT version 1.3. This quantitative study aimed to ascertain whether the construct of Verbal Reasoning was measured by the ECT and in doing so, it required the use of five specific objectives. Firstly, the unidimensionality of items was explored through Rasch analyses. The results revealed that the performance of the persons was consistently misfitting across the two test version which caused deviation from the model because they were either careless, guessing or not paying attention to the questions and thus irregular answering patterns occurred. The item performance however improved across the two test versions and only contained a few items which caused differential performance which was either due to difficulty or issues related to the item content. Additionally, Rasch analyses allowed the dimensionality of the ECT to be assessed which indicated that the test was multidimensional and there was some redundancy in the items and persons which limited the variance explained. Secondly, Confirmatory Factor Analysis (CFA) was used to confirm the dimensionality of the ECT by conducting Structural Equation Modelling. The model created was guided by the Exploratory Factor Analyses conducted on the ECT version 1.2 and performed on the data of the ECT version 1.3. The model demonstrated that the construct of Verbal Reasoning was subdivided into factors of Reasoning, Education, Vocabulary, Deduction and Plurals. The results of the SEM indicated that the model was a good fit and the factors were indeed related to the underlying construct of Verbal Reasoning. It also indicated that the strongest factors were Reasoning and Education. This served to indicate that the different factors of the model were acceptable as it explained the construct of Verbal Reasoning. Thirdly, supportive evidence of construct validity was attainted by conducting a Multi-trait Multi-method (MTMM) analysis. This analysis did not conform to the traditional aspects of a MTMM analysis due to the fact that some information was not available. This is however a limitation of using secondary data. These results indicated that the strongest correlations with the ECT were observed amoung the following constructs: Reading Comprehension, Vocabulary, Verbal Reasoning and long-term Memory. Moreover, the ECT was correlated with constructs that were not hypothesized to relate, such as Calculations, Mathematical Comprehension, Mechanical Insight, Spatial 2D and Spatial 3D. This emphasized that Verbal Reasoning was observed in the correlations across all constructs. Fourthly, measurement invariance was explored using Differential Test Functioning which focused on the comparison of gender and racial groups. The DTF results for the gender comparison revealed that there were a few items which were possibly biased across the genders. The DTF results for the different racial groups (African and White & African and Coloured) indicated more possibly biased items across the racial groups. The items that were identified are a cause of concern and require further investigation. It is however worth noting that the majority of the items were considered to be appropriate for both gender and the different racial groups.