Predictive Evaluation of Econometric Forecasting Models - Title

Predictive Evaluation of Econometric Forecasting Models
in Commodity Futures Markets

Tian Zeng
Aeltus Investment Management, Inc.

Norman R. Swanson
Department of Economics
Pennsylvania State University


Pages 159-177

Abstract

The predictive accuracy of various econometric models, including random walks, vector-autoregressive and vector-error-correction models, are investigated using daily futures prices of four commodities (the S&P 500 index, treasury bonds, gold, and crude oil). All models are estimated using a rolling-window approach, and evaluated by both in-sample and out-of-sample performance measures. The criteria considered include system criteria, where we evaluate multiequation forecasting models, and univariate forecast-accuracy criteria. The five univariate criteria are root mean square error (RMSE), mean absolute deviation (MAD), mean absolute percentage error (MAPE), confusion matrix (CM), and confusion rate (CR). The five system criteria used include the trace of second-moment matrix of the forecast-errors matrix (TMSE), the trace of second-moment matrix of percentage-forecast errors (TMAPE), the generalized forecast-error second-moment matrix (GFESM), and a trading-rule profit criterion (TPC) based on a maximum-spread trading strategy. An in-sample criterion, the mean Schwarz information criteria (MSIC), is also computed. Our results suggest that error-correction models perform better in shorter forecast horizons, when models are compared based on quadratic loss measures and confusion matrices. However, the error-correction models which we consider perform better at all forecast horizons (one to five steps ahead) when models are compared based on a profit-maximization loss function. Further, our error-correction model, where the error-correction term is constructed according to a cost-of-carry equilibrium condition, outperforms our alternative error-correction model, which uses the price spreads as the error-correction term.

Bibliography

  • Abken, P. A. (1989). "An analysis of intra-market spreads in heating oil futures." Journal of Futures Market , 9(1).
  • Akaike, H. (1973). "Information theory and an extension of the maximum likelihood principle." In B. N. Petrov and F. Csaki (eds.), Second International Symposium on Information Theory . Budapest: Akademiai Kiado, pp. 267-281.
  • Akaike, H. (1974). "A new look at the statistical model identification." IEEE Transactions on Automatic Control 19:716-723.
  • Bailey, W., and K. C. Chan (1993). "Macroeconomic influences and the variability of the commodity futures basis." Journal of Finance , 48:555-573.
  • Bessembinder, H., J. F. Coughenour, P. J. Seguin, and M. M. Smoller (1995). "Mean reversion in equilibrium asset prices: Evidence from the futures term structure." Journal of Finance , 50:361-375.
  • Breeden, Douglas (1980). "Consumption risk in futures markets." Journal of Finance , 35:520-530.
  • Brennan, M. J. (1958). "The Theory of Storage." American Economic Review , 48:50-72.
  • Cecchetti, S., P. Lam, and N. Mark (1990). "Mean reversion in equilibrium asset prices." American Economic Review , 80:399-418.
  • Clements, M. P., and D. F. Hendry (1993). "On the limitations of comparing mean square forecast errors." Journal of Forecasting , 12:617-676.
  • Clements, M. P., and D. F. Hendry (1995). "Forecasting in cointegrated systems." Journal of Applied Econometrics , 10:127-146.
  • Dusak, Katherine (1973). "Futures trading and investor return: The investigation of commodity market risk premium." Journal of Political Economy , 81: 1387-1406.
  • Dusak, K. C. (1990). "Futures trading and investor returns: An investigation of commodity market risk premiums." Journal of Political Economy , 81:1387-1406.
  • Diebold, Francis X., and Roberto S. Mariano (1995). "Comparing predictive accuracy." Journal of Business and Economic Statistics , 13:253-263.
  • Enders, Walter (1995). Applied Econometric Time Series . New York: Wiley.
  • Engle, R. F., and S. J. Brown (1986). "Model selection for forecasting." Applied Mathematics and Computation , 20.
  • Engle, R. F., and C. W. J. Granger (1987). "Co-integration and error correction: Representation, estimation, and testing." Econometrica , 55:251-276.
  • Fama, E. F., and K. R. French (1987). "Commodity futures prices: Some evidence on forecast power, premiums, and the theory of storage." Journal of Business , 60:55-73.
  • Fama, E. F., and K. R. French (1988). "Business cycles and the behavior of metals prices." Journal of Finance , 43:1075-1093.
  • French, K. R. (1986). "Detecting spot price forecasts in futures prices." Journal of Business , 59:39-54.
  • Granger, C. W. J., and N. Swanson (1997). "Stochastic cointegration." Oxford Bulletin of Economics and Statistics , 59(2):23-35.
  • Hazuka, T. B. (1984). "Consumption betas and normal backwardation in commodity markets." Journal of Finance , 39:647-655.
  • Henriksson, R. D., and R. C. Merton. (1981). "On market timing and investment performance II: Statistical procedures for evaluating forecast skills." Journal of Business , 54:513-533.
  • Hoffman, D., and R. Rasche (1996). "Assessing forecast performance in a cointegration system." Journal of Applied Econometrics , 11:495-517.
  • Kaldor, N. (1939). "Speculation and economic stability." Review of Economic Studies , 7:1-27.
  • Kim, M. J., C. R. Nelson, and R. Startz (1991). "Mean reversion in stock prices? A reappraisal of the empirical evidence." Review of Economic Studies , 58:515-528.
  • Leitch, G., and J. E. Tanner (1991). "Economic forecast evaluation: Profits versus the conventional error measures." American Economic Review , 81:580-590.
  • Lin, J., and R. S. Tsay (1996). "Co-integration constraint and forecasting: An empirical examination." Journal of Applied Econometrics , 11:519-538.
  • Lu, R., and R. M. Leuthold (1994). "Cointegration relations between spot and futures prices for storable commodities: Implications for hedging and forecasting." OFOR Working Paper 94-12, University of Illinois at Urbana-Champaign.
  • Ma, C., J. Mercer, and M. A. Walker (1992). "Rolling over futures contracts: A note." Journal of Futures Market , 12(12).
  • Miller, M. H., J. Muthuswamy, and R. E. Whaley (1994). "Mean reversion of Standard & Poor's 500 index based changes: Arbitrage-induced or statistical illusion." Journal of Finance , 49:479-513.
  • Newey, W. K., and K. D. West (1987). "A simple positive semi-definite, heteroskedasticity and autocorrelation consistent covariance matrix." Econometrica , 55:703-708.
  • Pesaran, M. H., and A. G. Timmermann (1994). "A generalization of the non-parametric Henriksson-Merton test of market timing." Economics Letters , 44:1-7.
  • Rissanen, Jorma (1978). "Modeling by shortest data description." Automatica , 14:465-471.
  • Schnader, M. H., and H. O. Stekler (1990). "Evaluating predictions of change." Journal of Business , 63:99-107.
  • Schwarz, Gideon (1978). "Estimating the dimension of a model." Annals of Statistics , 6:461-464.
  • Sim, Christopher (1980). "Macroeconomics and reality." Econometrica , 48(Jan.):1-49.
  • Stekler, H. O. (1991). "Macroeconomic forecast evaluation techniques." International Journal of Forecasting , 7:375-384.
  • Stekler, H. O. (1994). "Are economic forecasts valuable?" Journal of Forecasting , 13:495-505.
  • Swanson, N. R., and H. White (1995). "A model selection approach to assessing the information in the term structure using linear models and artificial neural networks." Journal of Business and Economic Statistics , 13:265-275.
  • Swanson, N. R., and H. White (1997). "A model selection approach to real-time macroeconomic forecasting using linear models and artificial neural networks." Review of Economics and Statistics , 79(4).
  • Swanson, N. R., and T. Zeng (1996). "Addressing colinearity among competing econometric forecasts: Regression-based forecast combination using model selection." Working Paper, Pennsylvania State University.
  • Swanson, N. R., T. Zeng, and A. E. Kocagil (1996). "The probability of mean reversion in equilibrium asset prices and returns." Working Paper, Pennsylvania State University.
  • Telser, L. G. (1958). "Futures trading and the storage of cotton and wheat." Journal of Political Economy , 66:233-255.
  • Wahab, M., R. Cohn, and M. Lashgari (1994). "Gold-silver spread: Integration, cointegration, predictability, ex ante arbitrage." Journal of Futures Market , 14(6).
  • West, K. D. (1996): "Asymptotic inference about predictive ability." Econometrica , 64(5):1067-1084.
  • Working, H. (1949). "The theory of the price of storage." American Economic Review , 39:1254-1262.