Measuring and adjusting for overconfidence

To evaluate density forecasts, the applied scoring rule is often arbitrarily chosen. The selection of the scoring rule strongly influences the ranking of forecasts. This paper identifies overconfidence as the main driver for scoring differences. A novel approach to measure overconfidence is proposed. Based on a non-proper scoring rule, the forecasts can be individually adjusted toward a calibrated forecast. Applying the adjustment procedure to the survey of professional forecasters, it can be shown that out-of-sample forecasts can be significantly improved. Also the ranking of the adjusted forecasts becomes less sensitive to the selection of scoring rules.