When life calls for causal analysis

Long-time reader Antonio R. sent me to this thread about plummeting subscriptions suffered by a Substack user.

Her last sentence was: "what good are followers when follower growth seems correlated with losing paid subs??"

While stated as a "correlation," the wording reflects a mild form of causal creep. Her mental model underlines a causal model – followers cause paid subscribers. This mental model is widespread in marketing circles. It's got a name: the "marketing funnel". Marketers believe that if they stuff the funnel from one end with followers (substitute eyeballs, free users, free trialers, etc.), they should find more paid members out the other end. The process is called "conversion," endowed with a metric called "conversion rate".

Conversion can be organic, as in self-motivated, but surely, some users are induced by a marketing action, such as a conversion offer (20% off if you sign up now type of thing). Data scientists (I'm one of those) may even devise scientific experiments to prove that these offers generate incremental subscriptions, thus proving causality in a statistical sense.

In our headline example, the causal relationship has been stretched beyond recognition. She observed two phenomena: number of followers skyrocketing; number of paid subscribers falling off a cliff. She wanted to draw an arrow through them.

The situation does not require any direct cause-effect relationship between the two metrics. A likely reason is an underlying process which affects both followers and subscribers in opposite directions.

For example, the newsletter might be designed to drive paid subscriptions by making the bottom half of an article available only to paid subs, typically via a Subscribe Now button. Imagine that this button is replaced by a Follow Free button, everything else unchanged. This switch of the "call to action" surely turns some would-be paid users to followers. The number of followers increases while the number of subscriptions decreases but one doesn't cause the other.

Another example: a viral post flooded the newsletter with lots of readers, causing the follower count to jump. This post covers a topic that is not central to the newsletter, and so most of these new followers do not find value in paying for a long-term subscription. The conversion rate drops as the denominator grows without adding proportionally to the numerator. Meanwhile, the attrition pattern of existing paid subscribers holds steady, meaning that the system loses subs every month. Again, we should observe the two metrics moving in opposite directions but the causal mechanism is not direct.

Causal inference is, in my view, an important area of business concern. Frequently, business metrics move in unexpected directions. Business managers want to take action to reverse the undesirable trends. Such action works only if the cause-effect relationship has been correctly diagnosed. People already do causal reasoning in simple ways; they can benefit from more sophisticated analytical machinery.