Several times each year, the Tropical Meteorology Project team at Colorado State University issues hurricane forecasts.  The April forecast just came out:  this year, there will be 16 named storms and 9 hurricanes.  Governments and insurance companies use these forecasts for budgetary planning purposes.  Nervous residents in West Palm use them to decide how many sheets of plywood to buy.

I’m glad we have a team of capable scientists at CSU who are dedicated to forecasting these things.  These folks use the best available computer models to aggregate thousands of climatological data points and process them with the soundest theoretical algorithms.  But let’s see how the models are actually performing.  The figures below show the April hurricane and named storm forecasts compared with the actual number of hurricanes and named storms that occurred.  Each point represents one year from 1984 through 2010.  If the models were perfectly predicting the number of storms, every point would lie exactly on the 45-degree line.  Since models are not perfect, we would expect some scatter around that line.  But if the scatter is too great, then the model loses its utility.  For the April forecasts, the data points are scattered more or less randomly.  There is literally zero correlation between the predicted number of storms in April and the actual number that occurred.  You would be as well off to roll a pair of dice than to believe the forecasts.  It is frankly a waste of CSU’s time and money to put these out.

Surely the forecasts must be better the closer the hurricane season approaches.  The CSU team also issues a forecast in June, as illustrated in the figures below.

As expected, the predictions are slightly more accurate in June, but still the scatter is far too much for the forecasts to be meaningful for any realistic purpose.  For example, suppose 8 hurricanes are predicted.  This tells you almost nothing about the actual number of hurricanes which will occur, which could range from 4 to 15.

Each year when August rolls around the CSU team issues another forecast.  This one, coming on the cusp of the hurricane season, must be highly accurate, right?  The figures below show the data.

Here we see a definite correlation in the data which approaches a level which may be meaningful.  Still, the scatter is large.  Scientists use a number called “r-squared” to quantify the strength of the relationship between two sets of data, such as predicted and observed hurricane numbers:  an r-squared of 1.0 means perfect correlation, and an r-squared of 0 means no correlation.  In both of the above plots, the r-squared value is 0.42.  Suppose 8 hurricanes were predicted.  The actual number of storms could range from 4 to 12.  A narrower range than complete randomness, but would you bank on it?

A significant slice of the public tends to trust climatological analyses far more than reality warrants.  Climate circulation patterns are simply far too complex to accurately predict, and this will always be the case.  Even if the density of our climate station network were to increase tenfold, there would still be 60% of the earth that could never be measured and could never feed data to the model.  Even if we could somehow solve that problem with an ultra-accurate system of remote-sensing satellites, the circulation patterns themselves would still not be fully tractable by the differential equations which the models solve.

It’s a giant advance in thinking to simply accept this fact, accept that risk exists, and not expect science to solve every problem and predict every catastrophe.

Advertisements