“Prediction is very difficult, especially if it’s about the future” – Niels Bohr (attrib.)

Like everybody else, I had no idea who Nate Silver was until I started following his 538 column in the run-up to the 2008 US Presidential election. Like everyone else, I took some solace from his confident, probabilistic forecasts that the McCain/Palin ticket was doomed. And like everyone else, I bought The Signal And The Noise and left it in a pile for months, expecting it to be a heavy read.

I should, of course, have known better: Silver wouldn’t have made any sort of name for himself had he been an impenetrable mathematical writer. Instead, 538 was (and is) at pains to make everything as clear as possible, to the point that anyone with the patience to amalgamate the data and write some code could have got, let’s say, 80% of the way to the same predictions.

The Signal and the Noise: The Art and Science of Prediction follows in the same tradition, but going wider than just politics: exploring topics from baseball and epidemics to climate change and terrorism, politics to hurricanes and earthquakes. On its way, it discusses the various challenges of making predictions – sifting the signal from the noise, making sure there’s a reason for your model, and the possibility of your forecasts affecting the outcome – especially in (for instance) epidemiology, saying there’s a high chance of an outbreak allows authorities to prepare to prevent it… which means the forecast will turn out to be almost completely wrong (but extremely useful).

He looks at the perverse incentives for some forecasters (if you’re making a weather forecast, sometime it pays to be pessimistic; if you’re making a political forecast, partisanship usually trumps accuracy) and some of the pitfalls involved. If I had to find a fault, it would be that it’s a tiny bit self-indulgent in places – but I’d say that he’s earned the right. (Compared to Villani, it’s the model of modesty.)

There’s a reasonable bit of maths in there – while Silver sidesteps the gory details of regression (sensibly), he does hammer the idea of how useful a Bayesian approach to prediction in is. He’s also not afraid to use logarithmic axes, which makes me unusually happy.

Knowing Silver, he probably expected that.

It’s the kind of book that had me itching to find my own data sets to play with and make probabilistic forecasts. Recommended.