Alternative methods to parametric significance testing in linear regression and ANOVA

Abstract

The aim of the study was to survey permutation tests, bootstrapping and jackknife methods and their application to significance testing of regression coefficients in linear regression analysis. A Monte Carlo simulation study was performed in order to compare the different methods in terms of empirical probability of type 1 error, power of a test and confidence interval where coverage and average length of confidence interval were used as measures of comparison. The empirical probability of type 1 error and power of a test were used to compare permutation tests, bootstrapping and parametric methods, while the confidence intervals were used to compare jackknife, bootstrap as well as the parametric method. These comparisons were performed in order to investigate the effect of (1) sample size (2) when errors are normally, uniformly and lognormally distributed (3) when the number of explanatory variables is 1, 2 and 5. (4) When the correlation coefficient between the explanatory variables is 0, 0.5 and 0.9. The results obtained from the Monte Carlo simulation study showed that permutation and bootstrap methods produced similar probability of type 1 error results while the parametric methods understated probability of type 1 error when errors are lognormally distributed. In the absence of multicollinearity all the methods were almost equally powerful and in presence of multicollinearity they all suffered equally in terms of power. The jackknife produced poor result in terms of 100(1???)% confidence interval while the bootstrap produced reasonable results especially for larger sample sizes. The improvement was observed under the jackknife method when percentile based intervals were applied. It was concluded that permutation tests as well as bootstrap methods are good alternative methods to use in significance testing in regression and ANOVA.