Evaluating probabilistic forecasts with scoringRules

Comparing predictive performance

Forecasts are generally surrounded by uncertainty and being able to quantify this uncertainty is key to good decision making. Accordingly, probabilistic forecasts in the form of predictive probability distributions over future quantities or events have become popular over the last decades in various fields including meteorology, climate science, hydrology, seismology, economics, finance, demography, and political science.

In typical applications, many alternative statistical models and data sources can be used to produce probabilistic forecasts. Hence, evaluating and selecting among competing methods is an important task. The scoringRules package for R provides functionality for comparative evaluation of probabilistic models based on proper scoring rules, covering a wide range of situations in applied work.

The scoringRules package aims to be a comprehensive library for computing scoring rules. We offer implementations of several known (but not routinely applied) formulas and implement some closed-form expressions that were previously unavailable. Whenever more than one implementation variant exists, we offer statistically principled default choices. The package contains the continuous ranked probability score (CRPS) and the logarithmic score, as well as the multivariate energy score and variogram score. All of these scoring rules are proper, which means that forecasters have an incentive to state their true belief.

scoringRules was developed in the Computational Statistics group at HITS is available on CRAN, and developed on GitHub.

References:

Alexander Jordan, Fabian Krüger, Sebastian Lerch
Evaluating Probabilistic Forecasts with scoringRules
J. Stat. Softw. (2019) 90(12):1–37, DOI: 10.18637/jss.v090.i12