Wednesday, December 14, 2011

If Your Forecasts are Always Wrong, Should You Stop Making Forecasts?

Engineers and planners are regularly tasked with predicting the future. Prediction exercises are useful for thinking about what facilities and investment we need in the future. Prediction exercises are also invariably wrong. Errors occur because trends change, preferences evolve, external forces intervene and/or models are flawed. In transportation and land use planning, these errors lead to permanent changes in our cities, often for the worse, yet no one is held accountable for these errors. Prediction error is not limited to transportation, however. Here is a link to an engineer who laments the limits of his work. (h/t David Levinson)  And here is an op-ed in New Scientist arguing that economic prediction models have much to learn from meteorology. This sounds promising as the weatherman is never wrong.

But here is something we can learn from weather forecasters. Two prominent hurricane forecasters have quit making long range forecasts because the forecasts are always wrong. They will only do near term forecasts. From the Ottawa Citizen:

OTTAWA — Two top U.S. hurricane forecasters, famous across Deep South hurricane country, are quitting the practice of making a seasonal forecast in December because it doesn’t work.
William Gray and Phil Klotzbach say a look back shows their past 20 years of forecasts had no predictive value.
The two scientists from Colorado State University will still discuss different probabilities of hurricane seasons in December. But the shift signals how far humans are, even with supercomputers, from truly knowing what our weather will do in the long run.
Colorado State has been known for decades for forecasts of how many named storms and hurricanes can be expected each official hurricane season (which runs from June to November.)
Last week, the pair made this announcement:
“We are discontinuing our early December quantitative hurricane forecast for the next year … Our early December Atlantic basin seasonal hurricane forecasts of the last 20 years have not shown real-time forecast skill even though the hindcast studies on which they were based had considerable skill.”
The two will still make the traditional forecasts closer to hurricane season.
An earlier version of this story incorrectly said they were stopping all forecasts.

I think this is a promising approach and can be applied to transportation systems. If your long-range forecasts are always flawed, the best solution may be to stop making long-range forecasts.


Shammee SJ said...

Long term forecast about Transportation is far more different from Weather forecast as it is less related to nature. Transportation forecast is more related to economy, government policies and peoples choices than whimsical events of nature.

I think an extensive research can be done on William Gray and Phil Klotzbach.
Why their past 20 years of eorecasts had no predictive value? What are the flaws behind this failure?
After identifying the defect, more effective ways to forecast weather might be revealed.

automobilious said...

@ShammeeSJ: Government policies, the economy, and people's choices are also a little whimsical. No one, for example, predicted the great recession. MTC in the bay area has revised their growth forecasts in light of recent trends so that the lowest income groups comprise a greater proportion of the total population in 2035 than they do currently.

It seems that long-range transportation forecasting can be useful for identifying tradeoffs between projects - anything more precise than this is a worthless exercise. Compliance with California's statewide greenhouse gas targets resulting from, ostensibly, VMT reductions, is demonstrated using a travel demand model. We're kidding ourselves if we think we can _actually_ hit these targets without serious policy change. The model gives us false hope.

The best transportation policy is intuitive - this is what a heavy reliance on models causes us to miss.

Unknown said...

Thank you giving Good Information via this Blog

Carmellimo is one of the best New York Limousines service in the US, We offer World-class NYC Airport Limousine vehicles or New York Limo for promos, parties, sports, events, birthdays, theaters, parties, and any special occasions safely at pandemic.