Comparing approaches for combining data collected from multiple complex surveys, adjusting for clustering and stratification

Abstract

Even though there is substantial literature on studies which pool survey data, it is still not clear which are the most efficient methodologies for pooling data from different surveys. For example, it is important to know whether the surveys involved should be given equal importance in the calculation of the combined statistics or not. If they are not given equal importance, then it should be clear how they should be weighted and why. In this research project, alternative methods used to combine survey data were evaluated and new methods proposed.

A literature review of methods that are currently being used in combining repeated and multiple surveys was presented. New methods were proposed or adapted from meta-analysis methodology to try and improve the calculation of weights and precision measures when multiple surveys are combined. Different variance estimators for the proposed point estimators were evaluated through simulation. Only the separate approach was considered in this study. Simple random samples and complex samples were drawn from simulated finite population data and used to evaluate current and proposed methods of combining surveys. Simple super-population models were used to simulate finite population data. The South African Community Survey of 2016 and the General Household Survey of 2016 were used to simulate finite populations which were then used for evaluating the different methods of combining simple random sampling and stratified surveys respectively.

Our results suggest that the choice of weighting method when combining surveys should depend on the super-population model assumed to have generated the finite population. The sample size used appeared to influence the choice of the method used to combine surveys, but the variance of the super-population did not influence the choice. Under simple random sampling, the strength of the skewness and kurtosis also appeared to affect the performance of the weighting strategies. Weighting by the inverse of the sample size, the inverse of variance and the inverse of the coefficient of variation appeared to work for most super-population models. Combining samples appeared to yield better estimates with lower mean square errors compared to single sample estimates. The number of samples combined appeared not to influence the choice of weighting strategy although the mean square errors decreased with increased number of samples combined. Under simple random sampling, the meta-analysis variance estimator appeared to work the best with the inverse of variance weighting method as expected.