Statistics and Lies

I see from today’s Volkskrant that Lucia de Berk has at last won the right to a retrial of her case. De Berk is a nurse who was convicted in 2003 for the supposed four murders and three attempted murders of patients in her care.

The history of the case makes chilling reading, not because of anything that de Berk may have done, but because of the web of statistical “proof” that the prosecution used to put her behind bars. It is perfectly clear that the statistical evidence was deeply flawed from the start, but here we are in 2008, and she has spent almost six years in jail for “crimes” that never existed in the first case.

The judgement against her was based largely on the claim (from the prosecution’s statistician) that the chances of so many people dying on the wards where she was on shift were “one in 342 million to one against”. But, as Ben Goldacre makes clear in his excellent book Bad Science, the fundamental flaw about this claim is twofold. First, the data was selected to make the hypothesis, and then the prosecution’s statistician made a simple, rudimentary error: he combined individual statistical tests by multiplying p-values (the mathematical description of chance, or statistical significance). As Goldacre points out in respect of the first part of the claim:

A huge amount of corollary statistical information was almost completely ignored. In the three years before Lucia worked on the ward in question, there were seven deaths. In the three years that she did work on the ward, there were six deaths. Here’s a thought: it seems odd that the death rate should go down on a ward at the precise moment that a serial killer – on a killing spree – arrives. If Lucia killed them all, then there must have been no natural deaths on that ward at all in the whole of the three years that she worked there.

And in respect of the second flaw, Goldacre points out:

If you multiply p-values together, then harmless and probable incidents rapidly appear vanishingly unlikely. Let’s say you worked in twenty hospitals, each with a harmless incident pattern: say p=0.5. If you multiply those harmless p-values, of entirely chance findings, you end up with a final p-value of 0.5 to the power of twenty, which is p < 0.000001, which is extremely, very, highly, statistically significant. With this mathematical error, by his reasoning, if you change hospitals a lot, you automatically become a suspect. Have you worked in twenty hospitals? For God’s sake don’t tell the Dutch police if you have.

It’s a very cautionary tale of statistics gone horribly wrong, and very reminiscent of the Sally Clark case in the UK (which Goldacre also dissects). Clark was put on trial in 1999, and convicted, for murdering her two babies. At the trial, child expert Professor Sir Roy Meadows stated that the chance of two children in the same family dying of Sudden Infant Death Syndrome (SIDS) was “one in seventy-three million”. It was another case of statistics wielded in error, and Clark spent three years in jail (where she was targeted by other prisoners as a supposed baby-murderer) before her conviction was quashed by the Court of Appeal. She emerged a broken woman and died in March 2007. I fervently hope that that will not be the fate of Lucia de Berk.