Note to NYT: Simple math ain't simple

Last week, the New York Times drew attention to the CTE problem in American football (link; paywall). The headline was:

Study’s Alarming Finding: At Least 1 in 4 N.F.L. Players Gets Brain Disease

For some reason, the reporting downplayed the message.

The Times told readers:

The study’s main finding was simple math: At least 215 of those 878 former players — 24.5 percent — had C.T.E.

Simple math, apparently, ain't simple.

Wow. What a mess we've found on our hands.

The study did not look at 878 "former players". Rather, the study covers 878 dead former players. If they intended to speak about "former players", then they are making an assumption that the prevalence of CTE is the same amongst living former players as those who passed away.

The study didn't even examine all 878 dead former players. CTE is only diagnosed after death by inspecting the brain. The study actually concerned 235 dead former NFL players whose brains were donated for research. Over 90% of the 235 brains showed signs of CTE.

Instead of informing readers that 215 of 235 brains were posthumously diagnosed with CTE, the reporter gave us the silly ratio of 215 of 878.

This is a particularly egregious case of using "raw data," not making up any numbers, not making any assumptions.

You know what, not making an assumption makes the silliest assumption of them all! Literally, the reporter's "simple math" simply assumes that the dead players whose brains were not donated did not have CTE. Not a single one of them had CTE.

If that's not an assumption, I don't know the meaning of the a-word.

But the reporter didn't see it this way. He freely admitted:

The roughly 25 percent rate does not consider the 643 brains that were not examined.

He's telling his "truth". The 215/878 ratio does not "consider" the 643 brains that were not examined.

Except. Not "considering" is the same as assuming the prevalence is zero. The two paths lead to the same end point, which is the 25% headline figure.

The researchers who wrote the underlying paper are the true culprits. They bask in the glory of "the most conservative assumption". Conservatism is taken as an automatic virtue. One cannot be wrong if one is "conservative." Statistics is about making judgement calls. Conservative calls are as bad as extreme calls. The most conservative assumption is categorically an extreme assumption.

It's really weird to see the next sentence:

Some of those, maybe many of them, likely had C.T.E. too.

If he believes this bit, then how can he justify plugging in a big fat zero for the estimated number of CTE cases amongst the dead players who did not donate their brains?

There's more. The report now generalizes from dead, former players to living, active players.

If the new study’s statistics were applied to today’s players — 1,696 on the league’s 32 regular-season rosters, not including more than 500 members of practice squads and those ineligible due to injury — at least 400 of them would be diagnosed with C.T.E.

I can understand the spirit of this extrapolation. Surely, we can all agree that CTE is a hidden menace amongst active NFL players.

The above statement extends the study's finding in two separate directions: from the dead to the living, and from former to active.

By not adjusting the statistics, the analyst makes key assumptions. By evidently making "no" assumptions, the analyst makes the worst possible assumptions.

First, the analysis assumes that the prevalence of CTE amongst living players is the same as that amongst dead players. Second, the analysis assumes that the prevalence of CTE amongst former players is the same as that amongst current players.