Evaluating Probability Forecasts

Forecasting is of utmost importance in statistics. Besides the very popular point forecasts, there are probability forecasts which naturally incorporate the uncertainty associated with a prediction. Probability forecasts have first been used in meteorology, however, applications are manifold, e.g. in economics the Survey of Professional Forecasters (SPF). In order to determine "good" probability forecasts, the concepts of sharpness and calibration are introduced. Calibration which is the statistical consistency between the forecasts and the observations, is assessed by the Probability Integral Transform (PIT). Sharpness measures the extent to which a probabilistic forecast is spread out and is assessed on the basis of so called scoring rules. These scoring rules are introduced along with some properties, the most important being (strict) propriety. Different scoring rules are introduced such as the quadratic score, the log score or the Ranked Probability Score. The empirical analysis is based on the SPF dataset. In addition to the single forecasts of the participants to the SPF, linear and logarithmic combinations with different weighting schemes of these single forecasts are analysed. Finally, statistical tests for equal predictice power in the sense of the well-known Diebold-Mariano test of these combinations are introduced and used to search for the best combination scheme.