The problem with control groups in the real world

February 7, 2012

A problem I occasionally encounter when trying to improve behaviour in the real world is the fairness of control groups.

Imagine you have a potential cure for a problem, but you have no proof that it actually works. A standard experiment would randomly allocate individuals to treatment and control conditions, run the experiment for a set period of time, and compare outcome variables. You’d then know if there was a strong effect.

paths

However, suppose the treatment turns out to be effective. Those individuals who did not receive it- control group - could claim that they had been treated unfairly, as others received a potential benefit which they did not.

The fact that the treatment was suspected to improve outcomes is sufficient to indicate that an unfair advantage was being conferred, despite the fact that it was not yet known to be effective.

Sometimes, not always, this fear works against our ability to create new knowledge, and definitively improve outcomes.

A business confronted by such a prospect might have the following thoughts:

This might improve our clients’ outcomes, but we have no proof.
If we run an experiment, those people in the control might feel they have been treated unfairly, and either stop being customers, or even worse, sue us.
We’d therefore have to give it to everyone.
But then we’d have no proof that it helped.
Let’s just not do it.

Note that in this case, no-one benefits from the potential improvement. I’ve been surprised how often this chain of thought is worked through, and the odds seem 50/50 that they come out this way. It takes more courage to find the truth than to continue with the status quo.