Sample size and statistical conclusions from tests of fit to the Rasch measurement model according to the RUMM2030 program

Sample size is a major contributor to statistical null hypothesis testing, which is the basis for many approaches to testing Rasch model fit. To allow for taking this into account, the RUMM2030 Rasch analysis software has the ability to adjust n in the calculation of its chi-2 based fit statistics. This paper examines the effects of such post-hoc adjustments on the statistical conclusions, and explores the occurrence of type I errors with Rasch model fit statistics implemented in RUMM2030. Data simulations of Rasch model fitting 25-item dichotomous scales with sample sizes ranging from n=50-2500 were generated an analysed regarding fit with and without adjusted sample sizes corresponding to the same n values as those simulated. Results suggest that post-hoc downward sample size adjustment is a useful procedure to avoid type I errors when working with relatively large data sets (n≥500). The value of upward adjustment with small data sets is less clear, particularly regarding the total item-trait chi-2 test, which tends to falsely signal misfit. Under the assumption of Rasch model fit, our observations suggest that a sample size around 250 (up to about 500) provides a good balance for the statistical interpretation of RUMM2030 fit statistics.