TY - JOUR
T1 - Relative model score
T2 - a scoring rule for evaluating ensemble simulations with application to microbial soil respiration modeling
AU - Elshall, Ahmed S.
AU - Ye, Ming
AU - Pei, Yongzhen
AU - Zhang, Fan
AU - Niu, Guo Yue
AU - Barron-Gafford, Greg A.
N1 - Funding Information:
Acknowledgements This work was supported by the Department of Energy Early Career Award DE-SC0008272 and NSF-EAR Grant 1552329.
Publisher Copyright:
© 2018, Springer-Verlag GmbH Germany, part of Springer Nature.
PY - 2018/10/1
Y1 - 2018/10/1
N2 - This paper defines a new scoring rule, namely relative model score (RMS), for evaluating ensemble simulations of environmental models. RMS implicitly incorporates the measures of ensemble mean accuracy, prediction interval precision, and prediction interval reliability for evaluating the overall model predictive performance. RMS is numerically evaluated from the probability density functions of ensemble simulations given by individual models or several models via model averaging. We demonstrate the advantages of using RMS through an example of soil respiration modeling. The example considers two alternative models with different fidelity, and for each model Bayesian inverse modeling is conducted using two different likelihood functions. This gives four single-model ensembles of model simulations. For each likelihood function, Bayesian model averaging is applied to the ensemble simulations of the two models, resulting in two multi-model prediction ensembles. Predictive performance for these ensembles is evaluated using various scoring rules. Results show that RMS outperforms the commonly used scoring rules of log-score, pseudo Bayes factor based on Bayesian model evidence (BME), and continuous ranked probability score (CRPS). RMS avoids the problem of rounding error specific to log-score. Being applicable to any likelihood functions, RMS has broader applicability than BME that is only applicable to the same likelihood function of multiple models. By directly considering the relative score of candidate models at each cross-validation datum, RMS results in more plausible model ranking than CRPS. Therefore, RMS is considered as a robust scoring rule for evaluating predictive performance of single-model and multi-model prediction ensembles.
AB - This paper defines a new scoring rule, namely relative model score (RMS), for evaluating ensemble simulations of environmental models. RMS implicitly incorporates the measures of ensemble mean accuracy, prediction interval precision, and prediction interval reliability for evaluating the overall model predictive performance. RMS is numerically evaluated from the probability density functions of ensemble simulations given by individual models or several models via model averaging. We demonstrate the advantages of using RMS through an example of soil respiration modeling. The example considers two alternative models with different fidelity, and for each model Bayesian inverse modeling is conducted using two different likelihood functions. This gives four single-model ensembles of model simulations. For each likelihood function, Bayesian model averaging is applied to the ensemble simulations of the two models, resulting in two multi-model prediction ensembles. Predictive performance for these ensembles is evaluated using various scoring rules. Results show that RMS outperforms the commonly used scoring rules of log-score, pseudo Bayes factor based on Bayesian model evidence (BME), and continuous ranked probability score (CRPS). RMS avoids the problem of rounding error specific to log-score. Being applicable to any likelihood functions, RMS has broader applicability than BME that is only applicable to the same likelihood function of multiple models. By directly considering the relative score of candidate models at each cross-validation datum, RMS results in more plausible model ranking than CRPS. Therefore, RMS is considered as a robust scoring rule for evaluating predictive performance of single-model and multi-model prediction ensembles.
KW - Bayes factor
KW - Continuous ranked probability score
KW - Dispersion
KW - Log-score
KW - Reliability
KW - Scoring rule
UR - http://www.scopus.com/inward/record.url?scp=85051490221&partnerID=8YFLogxK
UR - http://www.scopus.com/inward/citedby.url?scp=85051490221&partnerID=8YFLogxK
U2 - 10.1007/s00477-018-1592-3
DO - 10.1007/s00477-018-1592-3
M3 - Article
AN - SCOPUS:85051490221
SN - 1436-3240
VL - 32
SP - 2809
EP - 2819
JO - Stochastic Environmental Research and Risk Assessment
JF - Stochastic Environmental Research and Risk Assessment
IS - 10
ER -