Nate Silver: The DSCC Can’t Have It Both Ways

Nate Silver reports:

There was a lot of discussion Monday about the rebuttal by the Democratic Senatorial Campaign Committee (DSCC) to FiveThirtyEight’s latest Senate forecast. That forecast identified the Republicans as more likely than not to take over the Senate. But the GOP’s advantage is slim, and there’s a lot of uncertainty in both the individual races and the overall forecast.

The DSCC’s memo pointed to past forecasts by FiveThirtyEight that were off the mark. It’s a fun story for news outlets, but public statements by partisan groups won’t usually say anything that you didn’t already know.

Indeed, it’s not news that forecasts are sometimes wrong — as our Senate forecasts were in Montana and North Dakota in 2012. (Democrats won both races when Republicans were favored in our model.) Furthermore, the margin of error is larger at earlier stages of the campaign.

That’s why our forecasts are expressed in terms of probabilities. For example, our NCAA tournament model gave Mercer just a 7 percent chance of defeating Duke on Friday. Mercer won. Upsets are supposed to happen sometimes. Specifically, out of all forecasts in which we say the underdog has a 7 percent chance of prevailing, the underdog is supposed to win about seven times out of 100 over the long run — no more and no less. This property is called calibration, and it’s one of the best ways to assess probabilistic forecasts. (We’ll be conducting a test of the historical calibration of our NCAA forecasts this week.)

The conclusion:

At the same time the DSCC is criticizing our forecasts publicly, it’s sending out email pitches that cite Nate Silver’s “shocking, scary” forecasts to compel Democrats into donating.

You’d do well to shut out the noise the next time the DSCC writes a polling memo.

Ouch.