TY - JOUR

T1 - Relative model score

T2 - a scoring rule for evaluating ensemble simulations with application to microbial soil respiration modeling

AU - Elshall, Ahmed S.

AU - Ye, Ming

AU - Pei, Yongzhen

AU - Zhang, Fan

AU - Niu, Guo Yue

AU - Barron-Gafford, Greg A.

N1 - Funding Information:
Acknowledgements This work was supported by the Department of Energy Early Career Award DE-SC0008272 and NSF-EAR Grant 1552329.
Publisher Copyright:
© 2018, Springer-Verlag GmbH Germany, part of Springer Nature.

PY - 2018/10/1

Y1 - 2018/10/1

N2 - This paper defines a new scoring rule, namely relative model score (RMS), for evaluating ensemble simulations of environmental models. RMS implicitly incorporates the measures of ensemble mean accuracy, prediction interval precision, and prediction interval reliability for evaluating the overall model predictive performance. RMS is numerically evaluated from the probability density functions of ensemble simulations given by individual models or several models via model averaging. We demonstrate the advantages of using RMS through an example of soil respiration modeling. The example considers two alternative models with different fidelity, and for each model Bayesian inverse modeling is conducted using two different likelihood functions. This gives four single-model ensembles of model simulations. For each likelihood function, Bayesian model averaging is applied to the ensemble simulations of the two models, resulting in two multi-model prediction ensembles. Predictive performance for these ensembles is evaluated using various scoring rules. Results show that RMS outperforms the commonly used scoring rules of log-score, pseudo Bayes factor based on Bayesian model evidence (BME), and continuous ranked probability score (CRPS). RMS avoids the problem of rounding error specific to log-score. Being applicable to any likelihood functions, RMS has broader applicability than BME that is only applicable to the same likelihood function of multiple models. By directly considering the relative score of candidate models at each cross-validation datum, RMS results in more plausible model ranking than CRPS. Therefore, RMS is considered as a robust scoring rule for evaluating predictive performance of single-model and multi-model prediction ensembles.

AB - This paper defines a new scoring rule, namely relative model score (RMS), for evaluating ensemble simulations of environmental models. RMS implicitly incorporates the measures of ensemble mean accuracy, prediction interval precision, and prediction interval reliability for evaluating the overall model predictive performance. RMS is numerically evaluated from the probability density functions of ensemble simulations given by individual models or several models via model averaging. We demonstrate the advantages of using RMS through an example of soil respiration modeling. The example considers two alternative models with different fidelity, and for each model Bayesian inverse modeling is conducted using two different likelihood functions. This gives four single-model ensembles of model simulations. For each likelihood function, Bayesian model averaging is applied to the ensemble simulations of the two models, resulting in two multi-model prediction ensembles. Predictive performance for these ensembles is evaluated using various scoring rules. Results show that RMS outperforms the commonly used scoring rules of log-score, pseudo Bayes factor based on Bayesian model evidence (BME), and continuous ranked probability score (CRPS). RMS avoids the problem of rounding error specific to log-score. Being applicable to any likelihood functions, RMS has broader applicability than BME that is only applicable to the same likelihood function of multiple models. By directly considering the relative score of candidate models at each cross-validation datum, RMS results in more plausible model ranking than CRPS. Therefore, RMS is considered as a robust scoring rule for evaluating predictive performance of single-model and multi-model prediction ensembles.

KW - Bayes factor

KW - Continuous ranked probability score

KW - Dispersion

KW - Log-score

KW - Reliability

KW - Scoring rule

UR - http://www.scopus.com/inward/record.url?scp=85051490221&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=85051490221&partnerID=8YFLogxK

U2 - 10.1007/s00477-018-1592-3

DO - 10.1007/s00477-018-1592-3

M3 - Article

AN - SCOPUS:85051490221

SN - 1436-3240

VL - 32

SP - 2809

EP - 2819

JO - Stochastic Environmental Research and Risk Assessment

JF - Stochastic Environmental Research and Risk Assessment

IS - 10

ER -