Limits of data analysis in scientific inference: Reply to Sleep et al.

Statistical inference is an important element of science, but these inferences are constrained within the framework established by the objectives and design of a study. The choice of approach to data analysis, while important, has far less consequence on scientific inference than claimed by Sleep et al. (2007). Their principal assertion - that when model selection is used as the approach to data analysis, all studies provide a reliable foundation for distinguishing among mechanistic explanatory hypotheses - is incorrect and encourages faulty inferences. Sleep et al. (2007) overlook the critical distinction between inferences that result from studies designed a priori to discriminate among a set of candidate explanations versus inferences that result from exploring data post hoc from studies designed originally to meet pattern-based objectives. No approach to data analysis, including model selection, has the power to overcome fundamental limitations on inferences imposed by study design. The comments by Sleep et al. (2007) reinforce the need for scientists to understand clearly the inferential basis for their scientific claims, including the roles and limitations of data analysis.