How certain are our uncertainty bounds? Accounting for sample variability in Monte Carlo-based uncertainty estimates

It is common for model-based simulations to be reported using prediction interval estimates that characterize the lack of precision associated with the simulated values. When based on Monte-Carlo sampling to approximate the relevant probability density function(s), such estimates can significantly underestimate the width of the prediction intervals, unless the sample size is sufficiently large. Using theoretical arguments supported by numerical experiments, we discuss the nature and severity of this problem, and demonstrate how better estimates of prediction intervals can be achieved by adjusting the interval width to account for the size of the sample used in its construction. Our method is generally applicable regardless of the form of the underlying probability density function, and can be particularly useful when the model is expensive to run and large samples are not available. We illustrate its use via a simple example involving conceptual modeling of the rainfall-runoff response of a catchment.