Study Finds Average 13.2 Bcf Error in Analysts' Weekly Storage Predictions
Over the last decade of weekly natural gas storage reports, industry prognosticators have turned withdrawal/injection forecasting into a science, but since the Energy Information Administration (EIA) took over the survey from the American Gas Association in May 2002, the storage forecasts of 34 top firms, including five energy companies, 28 financial houses and one energy consulting firm, still have been off by about 13.2 Bcf/week on average, according to a new study.
A draft paper circulated earlier this month by Gerald D. Gay of Georgia State University and Betty J. Simkins and Marian Turac, both of Oklahoma State University, made a careful examination of the accuracy of the weekly storage predictions of the 34 firms and those in the Bloomberg weekly survey of analysts' predictions between 1997 and 2004. It found that although analysts providing storage forecasts assisted "significantly in the price discovery" on the natural gas futures market by providing important information, there also were "significant cross-sectional differences in analyst forecasting ability."
Last week was a perfect example of how the storage report sometimes can take everyone by surprise. The 49 Bcf withdrawal reported by the EIA was well outside the range of market expectations, which were between a 25 Bcf withdrawal and a 15 Bcf injection. As a result, the near-month natural gas futures contract soared more than $1 on its last day of trading before expiring and despite being already at a substantial premium to the natural gas cash market.
One of the findings of the study was that analysts' forecasts are less accurate and more widely dispersed during the critical withdrawal season than during the injection season. "Individual analyst forecast errors...and the weekly Bloomberg consensus forecast errors...are significantly larger (all at the 1% significance level) during the withdrawal seasons," the paper said. "In addition, the dispersion of analysts' forecasts is also larger during withdrawal seasons."
Looking at weekly forecasts of the 34 firms since 1997 overall, mean weekly averages ranged from an error as low as 9.49 Bcf (in the case of Citigroup) to as high as 23.13 Bcf (Cresvale).
Although the 13.2 Bcf average error since EIA started the survey might be considered a fairly large number, forecasting success used to be even further off the mark (an average error of 16.4 Bcf/week) when the American Gas Association was in charge of the storage report, according to the paper.
Furthermore, using a "standardized analysts forecast error" calculation, which compares the forecaster's weekly error to the actual working gas level for the week, the paper found that performance between 1997 and 2004 ranged from a weekly average error of only about 0.47% (in the case of Southwest Securities) to a high of 1.3% (Cresvale). Such a mistake even on a weekly basis could hardly be characterized as substantial.
Energy companies typically performed the best with their forecasts compared to financial companies, the report found. However after May 2002, the performance differences between the two groups was insignificant unless examined seasonally. Energy firm analysts still performed better than financial firms during the November through March withdrawal cycle. There were insignificant differences during injection cycles (April through October).
Overall, Salomon Smith Barney Futures was the most accurate forecaster compared to its peers on average over the period, while Cresvale International was the least accurate over the time period.
"Their forecast errors are smaller than alternatives," Gay said in an interview with NGI. "We also provide evidence that the market is embedding these analysts' forecasts into prices given the way that prices react when these weekly reports by the EIA are released. How good they are I don't know, but they are certainly better than a lot of these alternative approaches."
For more details from the paper contact Gay at the Department of Finance, J. Mack Robinson College of Business, Georgia State University in Atlanta at (404) 651-1889 or firstname.lastname@example.org.