Causal vs. correlational: how to tell if your tool actually knows why

Here's a claim you've probably seen in an analytics product:

Customers who use Feature X are 3× more likely to renew.

It's almost certainly true, in the sense that the arithmetic is correct. It's also nearly useless, and acting on it is how teams spend a quarter driving adoption of a feature that changes nothing.

The reason is that the sentence conflates two very different things, and the tool that produced it usually can't tell them apart.

The two possible worlds

World A: Feature X delivers real value. Using it makes customers more successful, so they renew. Driving adoption would increase renewals.

World B: Customers who were already succeeding — bigger teams, better fit, more invested — explore more of the product and encounter Feature X along the way. The feature is a symptom of success, not a cause. Driving adoption would produce nothing except a nicer-looking adoption chart.

Both worlds generate identical data. The 3× is the same number in each. No amount of additional observational data distinguishes them, because the difference isn't in the data — it's in the causal structure that produced it.

Causal finding this is a mathematical fact about observational inference, not an empirical observation of ours. Correlation is compatible with multiple causal structures, and choosing between them requires information that correlation alone doesn't carry.

What a causal claim actually requires

Two things. Both. Neither is optional.

1. A dated change event

Something must have happened, at a specific time, that you can point to.

A deploy on the 14th. A price change effective the 1st. A supplier switch in week 32. A policy update announced Tuesday.

Without a date, you have a state of the world and no intervention to attribute it to. "Customers who use Feature X" is not an event — it's a description of a group. "We shipped Feature X to these accounts on March 3rd" is an event.

This sounds trivial and it's where most organizations fail. Deploy logs live in one system, price changes in a spreadsheet, policy changes in someone's inbox, supplier changes in a procurement tool. Assembling one timeline of dated change events is genuinely the hardest manual step in causal work, and it's why we invested in building it as infrastructure rather than expecting analysts to reconstruct it each time.

2. A comparison group

Some customers must have experienced the change, and some must not have.

The comparison can come from:

• Randomization — the gold standard. You assigned it.
• Staged rollout — feature flags, percentage releases, region-by-region deployment.
• Natural boundaries — the change hit iOS before Android, or one plan tier before another.
• Time adjacency — the cohort immediately before against the cohort immediately after. Weakest of the four, because anything else that changed in the same window is confounded with your event.

Without a comparison group, you can see that something moved after the change. You cannot see whether it would have moved anyway. Seasonality, market conditions, a competitor's outage, a viral moment — all of them produce step changes that look exactly like your intervention working.

The honest labels

This is why Flolyt grades every claim rather than presenting one confidence level for everything:

The labels aren't hedging. They're the difference between a finding you should act on immediately and one you should investigate further, and collapsing them into a single confident voice destroys information the reader needs.

Five questions to ask any analytics vendor

Take these into your next evaluation.

1. "Show me a claim your product made and tell me what would make it false." A vendor who can't answer is describing a system that never produces falsifiable claims. Everything is an "insight," and insights can't be wrong.

2. "When you say something drove an outcome, what was the comparison group?" Watch for "we use a multi-touch model" or "our AI weighs many signals." Neither is a comparison group. A comparison group is a set of customers who didn't get the thing.

3. "What does your product do when it can't determine the cause?" If the answer is that it always surfaces something, that's a design that cannot express uncertainty. Every question gets an answer whether or not one exists.

4. "Does the product distinguish predictive from causal?" A churn risk score is predictive — it says who will leave. It doesn't say why or what would change it. Both are useful; conflating them means teams act on scores as if they were diagnoses.

5. "How do you handle a change that shipped to everyone at once?" The honest answer is "we compare before and after, flag confounders, and call it association rather than causation." Anything more confident is overclaiming, and now you know it.

Where predictive is the right tool

I don't want to overcorrect. Prediction is genuinely valuable and doesn't need causal grounding to be useful.

Triage. A churn risk score that's 70% accurate is a perfectly good queue-ordering tool, even if you have no idea what drives the score. You're deciding who to look at, not what to do.

Forecasting. Projecting next quarter's revenue doesn't require understanding causes. It requires the underlying process to stay stable.

Anomaly detection. Noticing something is off is the trigger for causal work, not a substitute for it. Our agents are predictive at this stage — they detect, then the causal work begins.

The failure isn't using prediction. It's using a prediction as though it were a diagnosis. "This account will churn" tells you to look. It doesn't tell you what to do, and a team that treats it as an instruction will run a retention play against a customer whose problem was a failed payment.

What this looks like in practice

The complete chain, using the checkout example:

1. Signal. Second-order rate steps down in the week of March 3rd. Detected by monitoring cohort curves, not by someone asking.
2. Candidate events. Pull every dated change in that window: three deploys, one shipping-rate update, one supplier change.
3. Comparison group. One deploy went out behind a flag at 50%. That's the usable one.
4. Effect. Exposed cohort down 11 points. Unexposed cohort flat.
5. Mechanism. The deploy moved the shipping fee from the product page to the final step. Support tickets in the same window mention unexpected charges at a rate consistent with the change.
6. Label. Causal finding — dated event, comparison group, coherent mechanism, corroborating evidence.

Now compare that to what you'd have had with correlation alone: "customers who abandoned checkout are more likely to lapse." True, useless, and pointing at the symptom.

What we can't tell you yet

Causal inference on observational data is an active research field and we're not claiming to have solved it. Our approach — require a dated event and a comparison group, otherwise downgrade the label — is conservative by design. It means we say "strong association" in cases where a sophisticated method might legitimately reach a causal conclusion.

We think that trade is correct for a product that teams make money decisions on. But it is a trade, and someone with a strong argument for a better method should tell us. My email is on the author page.

We also can't fix the timeline problem for you. If your organization doesn't record dated change events in a form that maps to customer-facing behaviour, causal analysis is limited no matter whose tool you run. Start logging deploys with a customer-impact flag. Future-you will find it worth the ten minutes.