Benchmarking robustness of load forecasting models under data integrity attacks

As the internet’s footprint continues to expand, cybersecurity is becoming a major concern for both governments and the private sector. One such cybersecurity issue relates to data integrity attacks. This paper focuses on the power industry, where the forecasting processes rely heavily on the quality of the data. Data integrity attacks are expected to harm the performances of forecasting systems, which will have a major impact on both the financial bottom line of power companies and the resilience of power grids. This paper reveals the effect of data integrity attacks on the accuracy of four representative load forecasting models (multiple linear regression, support vector regression, artificial neural networks, and fuzzy interaction regression). We begin by simulating some data integrity attacks through the random injection of some multipliers that follow a normal or uniform distribution into the load series. Then, the four aforementioned load forecasting models are used to generate one-year-ahead ex post point forecasts in order to provide a comparison of their forecast errors. The results show that the support vector regression model is most robust, followed closely by the multiple linear regression model, while the fuzzy interaction regression model is the least robust of the four. Nevertheless, all four models fail to provide satisfying forecasts when the scale of the data integrity attacks becomes large. This presents a serious challenge to both load forecasters and the broader forecasting community: the generation of accurate forecasts under data integrity attacks. We construct our case study using the publicly-available data from Global Energy Forecasting Competition 2012. At the end, we also offer an overview of potential research topics for future studies.     

Forecasting from time series subject to sporadic perturbations: Effectiveness of different types of forecasting support

How effective are different approaches for the provision of forecasting support? Forecasts may be either unaided or made with the help of statistical forecasts. In practice, the latter are often crude forecasts that do not take sporadic perturbations into account. Most research considers forecasts based on series that have been cleansed of perturbation effects. This paper considers an experiment in which people made forecasts from time series that were disturbed by promotions. In all conditions, under-forecasting occurred during promotional periods and over-forecasting during normal ones. The relative sizes of these effects depended on the proportions of periods in the data series that contained promotions. The statistical forecasts improved the forecasting accuracy, not because they reduced these biases, but because they decreased the random error (scatter). The performance improvement did not depend on whether the forecasts were based on cleansed series. Thus, the effort invested in producing cleansed time series from which to forecast may not be warranted: companies may benefit from giving their forecasters even crude statistical forecasts. In a second experiment, forecasters received optimal statistical forecasts that took the effects of promotions into account fully. This increased the accuracy because the biases were almost eliminated and the random error was reduced by 20%. Thus, the additional effort required to produce forecasts that take promotional effects into account is worthwhile.     

Comparing trained and untrained probabilistic ensemble forecasts of COVID-19 cases and deaths in the United States

The U.S. COVID-19 Forecast Hub aggregates forecasts of the short-term burden of COVID-19 in the United States from many contributing teams. We study methods for building an ensemble that combines forecasts from these teams. These experiments have informed the ensemble methods used by the Hub. To be most useful to policymakers, ensemble forecasts must have stable performance in the presence of two key characteristics of the component forecasts: (1) occasional misalignment with the reported data, and (2) instability in the relative performance of component forecasters over time. Our results indicate that in the presence of these challenges, an untrained and robust approach to ensembling using an equally weighted median of all component forecasts is a good choice to support public health decision-makers. In settings where some contributing forecasters have a stable record of good performance, trained ensembles that give those forecasters higher weight can also be helpful.     

Are forecasting competitions data representative of the reality?

Forecasters typically evaluate the performances of new forecasting methods by exploiting data from past forecasting competitions. Over the years, numerous studies have based their conclusions on such datasets, with mis-performing methods being unlikely to receive any further attention. However, it has been reported that these datasets might not be indicative, as they display many limitations. Since forecasting research is driven somewhat by data from forecasting competitions, it becomes vital to determine whether they are indeed representative of the reality or whether forecasters tend to over-fit their methods on a random sample of series. This paper uses the data from M4 as proportionate to the real world and compares its properties with those of past datasets commonly used in the literature as benchmarks in order to provide evidence on that question. The results show that many popular benchmarks of the past may indeed deviate from reality, and ways forward are discussed in response.     

Accuracy, unbiasedness and efficiency of professional macroeconomic forecasts: An empirical comparison for the G7

In this paper, we use survey data to analyze the accuracy, unbiasedness and efficiency of professional macroeconomic forecasts. We analyze a large panel of individual forecasts that has not previously been analyzed in the literature. We provide evidence on the properties of forecasts for all G7-countries and for four different macroeconomic variables. Our results show a high degree of dispersion of forecast accuracy across forecasters. We also find that there are large differences in the performances of forecasters, not only across countries but also across different macroeconomic variables. In general, the forecasts tend to be biased in situations where the forecasters have to learn about large structural shocks or gradual changes in the trend of a variable. Furthermore, while a sizable fraction of forecasters seem to smooth their GDP forecasts significantly, this does not apply to forecasts made for other macroeconomic variables.     

组合负荷预测方法综述

电力系统负荷具有很多不确定因素,针对单一模型进行负荷预测时,预测精度不高这一问题,可采用组合预测法将多种预测方法所得的预测值进行加权平均而得到最终预测结果,以满足现代电力对负荷预测结果的准确性、快速性和智能化的要求。该文首先简要介绍了几种常用的负荷预测方法,接着详细介绍了组合负荷预测的研究现状及确定组合预测中各模型最优权重的几种方法,最后介绍了组合负荷预测模型的误差修正方法,对提高负荷预测的准确性有一定的现实意义。     

Use and misuse of information in supply chain forecasting of promotion effects

Demand forecasting is critical to sales and operations planning (S&OP), but the effects of sales promotions can be difficult to forecast. Typically, a baseline statistical forecast is judgmentally adjusted on receipt of information from different departments. However, much of this information either has no predictive value or its value is unknown. Research into base rate discounting has suggested that such information may distract forecasters from the average uplift and reduce accuracy. This has been investigated in situations in which forecasters were able to adjust the statistical forecasts for promotions via a forecasting support system (FSS). In two ecologically valid experiments, forecasters were provided with the mean level of promotion uplift, a baseline statistical forecast, and quantitative and qualitative information. However, the forecasters were distracted from the base rate and misinterpreted the information available to them. These findings have important implications for the design of organizational S&OP processes, and for the implementation of FSSs.     

A Kalman filter formulation for noisy regional job data

A problem of great concern to regional economic forecasters is how to know the current local economic status given the delays and noise in provisional data. In this paper the structure of revisions to one of the most important regional data sets, local jobs by industry, is analyzed. A Kalman filter formulation is presented that can be used to improve the monthly estimates of jobs in local industries where large subsequent data revisions are most likely. Two data sets, one for the State of Virginia and one for the Louisville MSA, are used to illustrate the technique. The results, while not conclusive, suggest that the technique may be of practical importance in the monitoring and forecasting of employment activity in mining, construction and other industries for which provisional estimates are subject to the greatest later revision.     

Gauging the uncertainty of the economic outlook using historical forecasting errors: The Federal Reserve’s approach

The Federal Open Market Committee (FOMC) of the U.S. Federal Reserve regularly publishes participants’ qualitative assessments of forecast uncertainty, expressed relative to that seen on average in the past. The benchmarks used for these historical comparisons are the average root mean squared forecast errors (RMSEs) made by various private and government forecasters over the past twenty years. This paper documents how these benchmarks are constructed and discusses some of their properties. We draw several conclusions. First, if past performance is a reasonable guide to future accuracy, considerable uncertainty surrounds macroeconomic projections. Second, different forecasters have similar accuracy. Third, estimates of uncertainty about future real activity and interest rates are now considerably greater than prior to the financial crisis; in contrast, estimates of inflation accuracy have changed little. Finally, fan charts, constructed under certain assumptions and viewed in conjunction with the FOMC’s qualitative assessments, provide a reasonable approximation to future uncertainty.     

How reliable are forecasts of GDP growth and inflation for countries with limited coverage?

This paper analyses the performance of GDP growth and inflation forecasts for 25 transition countries between 1994 and 2007, as provided by 13 international institutions, including multilateral, private and academic forecasters. The empirical results show that there is a positive correlation between the number of forecasters covering a given country and the forecast accuracy. Simple combined forecasts are shown to be unbiased and more accurate than most of the individual forecasters, although also inefficient. However, only a few institutions provide efficient and unbiased forecasts, with just one out of 13 forecasters providing both unbiased and efficient forecasts of both GDP growth and inflation in the observed period. The directional analysis shows a correct forecast of the change in the forecast indicator in over two thirds of cases. However, the eventual outcome is within the range of available forecasts in less than half of the cases, with more than 40% of outcomes for GDP growth above the highest forecast. Encouragingly, forecasts are shown to be improving over time and becoming more accurate with the increase in the number of forecasting institutions – forecast accuracy measured by mean absolute error improves by 0.3 percentage points for growth and by 0.2 percentage points for inflation for each additional institution providing forecasts.     

Can idiosyncratic volatility help forecast stock market volatility?

This paper examines the predictive power of idiosyncratic volatility in the context of daily stock market volatility dynamics. Specifically, the relative performance of various models of market volatility is considered with respect to whether idiosyncratic volatility is excluded or included as an explanatory variable in such models. Using high frequency data covering the thirty stocks within the Dow Jones Industrial Average (DJIA) index, the results indicate that the inclusion of idiosyncratic volatility leads to significant in-sample and out-of-sample improvements in the fit of all the volatility models considered. These results are shown to be relatively robust to the loss function adopted by the forecaster, with reasonable forecast accuracy improvements available to such forecasters.     

Ranking professional forecasters by the predictive power of their narratives

This article presents the first ever ranking of professional forecasters based on the predictive power of the narrative of their regular research reports. The ranking is generated by applying the fully automated four-step procedure – called NLP-ForRank – developed in this article. The four steps are data scraping from the internet; data preparation; application of the natural language processing (NLP) models; and evaluation of the predictive power of the NLP indexes with linear regression, Granger causality, vector autoregression (VAR), and random forest forecasting models. Applying this procedure to five large Polish banks and to many time series shows that including the constructed NLP indexes in the forecasting models lowers the forecast errors, and that the optimal model almost always contains the NLP index. The financial news agencies could consider publishing this type of ranking on a regular basis as it would foster accountability, transparency, and a more competitive environment in the professional forecasting industry.     

Data preprocessing and quantile regression for probabilistic load forecasting in the GEFCom2017 final match

Team QUINKAN competed in the GEFCom2017 final match of hierarchical probabilistic load forecasting by adopting the quantile regression method using the R package quantreg. The weather stations were clustered into 11 groups, from which an optimal one was chosen for each load meter using the boosting method. The load meter records were cleaned and/or supplemented by various methods in order to secure robust quantile predictions. The variation in the regression formulas was kept as small as possible by introducing measures for suppressing prediction instability, although special formulas were employed for loading meters that were of an industrial nature. Several procedures were applied to help improve the accuracy, such as the smoothing of season transitions, coarse graining of the relative humidity, the use of load-oriented day-type definition, the averaging of weather data, and outlier removal.     

ARE EMPIRICAL MEASURES OF MACROECONOMIC UNCERTAINTY ALIKE?

Abstract There is a plethora of time series measures of uncertainty for inflation and real output growth in empirical studies but little is known whether they are comparable to the uncertainty measure reported by individual forecasters in the survey of professional forecasters. Are these two measures of uncertainty inherently distinct? This paper shows that, compared with many uncertainty proxies produced by time series models, the use of real‐time data with fixed‐sample recursive estimation of an asymmetric bivariate generalized autoregressive conditional heteroskedasticity model yields inflation uncertainty estimates which resemble the survey measure. There is, however, overwhelming evidence that many of the time series measures of growth uncertainty exceed the level of uncertainty obtained from survey measure. Our results highlight the relative merits of using different methods in modelling macroeconomic uncertainty which are useful for empirical researchers.     

Rethinking weather station selection for electric load forecasting using genetic algorithms

Demand forecasting is and has been for years a topic of great interest in the electricity sector, being the temperature one of its major drivers. Indeed, one of the challenges when modelling the load is to choose the right weather station, or set of stations, for a given load time series. However, only a few research papers have been devoted to this topic. This paper reviews the most relevant methods that were applied during the Global Energy Forecasting Competition of 2014 (GEFCom2014) and presents a new approach to weather station selection, based on Genetic Algorithms (GA), which allows finding the best set of stations for any demand forecasting model, and outperforms the results of existing methods. Furthermore its performance has also been tested using GEFCom2012 data, providing significant error improvements. Finally, the possibility of combining the weather stations selected by the proposed GA using the BFGS algorithm is briefly tested, providing promising results.     

DSGE forecasts of the lost recovery

The years following the Great Recession were challenging for forecasters. Unlike other deep downturns, this recession was not followed by a swift recovery, but instead generated a sizable and persistent output gap that was not accompanied by deflation as a traditional Phillips curve relationship would have predicted. Moreover, the zero lower bound and unconventional monetary policy generated an unprecedented policy environment. We document the actual real-time forecasting performance of the New York Fed dynamic stochastic general equilibrium (DSGE) model during this period and explain the results using the pseudo real-time forecasting performance results from a battery of DSGE models. We find the New York Fed DSGE model’s forecasting accuracy to be comparable to that of private forecasters, and notably better for output growth than the median forecasts from the FOMC’s Summary of Economic Projections. The model’s financial frictions were key in obtaining these results, as they implied a slow recovery following the financial crisis.     

EFFECTS OF THE 2011 GREAT EAST JAPAN EARTHQUAKE ON ELECTRICITY LOAD DEMAND: STRUCTURAL CHANGES DURING RECOVERY PERIODS

In response to the Fukushima Daiichi Nuclear Power Plant's severe crisis, the Tokyo Electric Power Company planned rolling blackouts, and the Japanese government encouraged companies and residents to conserve electricity by adopting self‐restriction plans. We examine the structural changes caused by the disaster in Japan and the effects of the power blackouts and self‐restriction plans on the magnitude and pattern of load demand. The results show that the total demand decreased after the disaster and changed from weekdays to weekends and holidays. In addition, the effect of temperature on load demand changed after the disaster.     

What do forecasting rationales reveal about thinking patterns of top geopolitical forecasters?

Geopolitical forecasting tournaments have stimulated the development of methods for improving probability judgments of real-world events. But these innovations have focused on easier-to-quantify variables, like personnel selection, training, teaming, and crowd aggregation—bypassing messier constructs, like qualitative properties of forecasters’ rationales. Here, we adapt methods from natural language processing (NLP) and computational text analysis to identify distinctive reasoning strategies in the rationales of top forecasters, including: (a) cognitive styles, such as dialectical complexity, that gauge tolerance of clashing perspectives and efforts to blend them into coherent conclusions and (b) the use of comparison classes or base rates to inform forecasts. In addition to these core metrics, we explore metrics derived from the Linguistic Inquiry and Word Count (LIWC) program. Applying these tools to multiple tournaments and to forecasters of widely varying skill (from Mechanical Turkers to carefully culled “superforecasters”) revealed that: (a) top forecasters show higher dialectical complexity in their rationales and use more comparison classes; (b) experimental interventions, like training and teaming, that boost accuracy also influence NLP profiles of rationales, nudging them in a “superforecaster” direction.     

Rounding behaviour of professional macro-forecasters

The rounding of point forecasts of CPI inflation and the unemployment rate by U.S. Professional Forecasters is modest. There is little evidence that forecasts are rounded to a greater extent in response to higher perceived uncertainty surrounding future outcomes. There is clear evidence that the probability of decline forecasts are rounded: over half of the forecast probabilities of decline in the current quarter are multiples of ten. It is found here that the rounding of these probabilities correlates with worse accuracy, although it is also of note here that worse (less accurate) forecasters might round more as opposed to the degree of rounding per se worsening accuracy. By simulating the loss from rounding for a set of efficient forecasters, it is demonstrated that the explanation that respondents round otherwise efficient forecasts is implausible, and that the contribution of rounding is of minor importance.     

EFFECTS OF COUNTERTRADE-DIVERGENT PERCEPTIONS BETWEEN PRACTITIONERS AND NON-PARTICIPANTS[1]

This article provides a theoretical discussion of countertrade benefits and problems. It then comments on data collected in a 1985 study of large British firms which demonstrate that actual countertrades have perceptions of the effects of this practice which differ markedly from the perceptions of companies that do not countertrade. Throughout, the findings on benefits and problems are compared as between international and domestic cotmtertraders and non-countertraders. It is concluded that many more UK firms could benefit from participation in countertrade.