It's not about the bias per se
It's about finding real drivers
Statisticians have an eye for bias in unlikely places.
I came across a nice example on the Linkedin feed via Mary Kate Stimmler. She showed the chart shown above.
The proportion of U.S. adults who have college or higher degrees rose from 14% in 1975 to almost 40% in 2023.
The linear trend is so very clear. How can a statistician nitpick this?
Well, we can and we do.
The issue with this analysis is that it didn't adjust for the change in the age distribution of the underlying population. Between 1975 and 2023, the proportion of the population who's 25 and over probably increased substantially, as is the case in most developed nations.
Based on some feedback, she made the following alternative chart:
She concluded that the adjustment didn't make a difference in this dataset.
Instead of an adjustment, she applied what is called "conditioning". The revised chart only concerns the age group 25-29. It shows the proportion of this age group that has college degrees or higher.
We have to be careful when comparing the two charts as the first one started around 1975 while the bottom one initialized in 1940. If we focus only on the period of the first chart (1975-2023), the increase was from 20% to 40%.
Note that the "adjustment" was much larger (14% --> 20%) on the earlier time than the latter time (38% --> 40%).
Why does the underlying age distribution matter? Here is the demographic trend during those decades:
The population 25 and over increased from 55% to about 70%. Almost all of the increase is due to the 55-and-over group. The 25-to-54 group went up then came back down.
The issue is that college and grad school degrees are usually obtained at specific ages, on the younger side of the 25-and-over range. If over time, the proportion of younger people rises, then on average, we expect a higher proportion of college educated (even if young people went to college at the same rate).
The U.S. population is not getting younger. We have the opposite situation: even if the rate of going to college did not change, the overall percent of college educated should go down as population ages.
What we see though is that the 25-to-29 group are getting degrees at a greater clip over time. The reason why on the right side of the first chart, the proportion went from 40% to 38% after mixing in older folks is precisely a reflection of this trend.
How do we explain the large gap in college educated proportion between the 25-to-29 age group and the 25-and-above age group on the left side of the first chart? The younger cohort showed a 20% proportion; after adding in people 30 or older, the proportion dropped to 14%.
The explanation is hiding in plain sight. It's on the second chart. Look to the left of 1975. Those are the people 30 or older in 1975. Their college educated proportion is quite dramatically lower than 20%. They are dragging the average down.
Such an analysis of bias is really a misnomer. The point is not to identify biases. The point is to find the real drivers of the metrics we're looking at.
