I Think You'll Find It's a Bit More Complicated Than That (12 page)

There are people in my profession who think they can ignore this problem. Some are murmuring that this mess is like MMR, a public misunderstanding to be corrected with better PR. They are wrong: it’s like nuclear power. Medical data, rarefied and condensed, presents huge power to do good, but it also presents huge risks. When leaked, it cannot be unleaked; when lost, public trust will take decades to regain.

This breaks my heart. I love big medical datasets, I work on them in my day job, and I can think of a hundred life-saving uses for better ones. But patients’ medical records contain secrets, and we owe them our highest protection. Where we use them – and we have used them, as researchers, for decades without a leak – this must be done safely, accountably and transparently. New primary legislation, governing who has access to what, must be written: but that’s not enough. We also need vicious penalties for anyone leaking medical records; and the HSCIC needs to regain trust, by releasing all documentation on all past releases, urgently. Care.data needs to work: in medicine, data saves lives.

The care.data programme was suspended shortly after this piece was published, with the promise that they’d have a think and relaunch in six months. Six months have already passed, and there has been no relaunch. I’m on their Advisory Group and continue to shout about the issues raised above, indoors and out. Medical data can save lives, but if the single biggest project ever conceived on patient records is not handled properly, we risk destroying public trust for all such projects, not just care.data.

SURVEYS

The Huff

Guardian
, 19 January 2008

In 1954 a man called Darrell Huff published a book called
How to Lie with Statistics
. Chapter 1 is called ‘The Sample With Built-In Bias’, and it reads exactly like this column, which I’m about to write, on a
Daily Telegraph
story in 2008.

Huff sets up his headline: ‘The Average Yaleman, Class of 1924, Makes $25,111 a Year!’ said
Time
magazine, half a century ago. That figure sounded pretty high: Huff chases it, and points out the flaws. How did they find all these people they asked? Who did they miss? Losers tend to drop off the alma mater radar, whereas successful people are in
Who’s Who
and the
College Record
. Did this introduce ‘selection bias’ into the sample? And how did they pose the question? Can that really be salary rather than investment income? Can you trust people when they self-declare their income? Is the figure spuriously precise? And so on.

In the intervening fifty years this book has sold one and a half million copies. It’s the greatest-selling stats book of all time (tough market), and it
remains in print
, at just £8.99.

Meanwhile, ‘
Doctors say no to abortions
in their surgeries’ is the headline in the
Daily Telegraph
. ‘Family doctors are threatening a revolt against Government plans to allow them to perform abortions in their surgeries, the
Daily Telegraph
can disclose.’ A revolt? ‘Four out of five GPs do not want to carry out terminations even though the idea is being tested in NHS pilot schemes, a survey has revealed.’

Channelling Huff through my fingers, in a trancelike state, I went in search of the figures. Was this a systematic survey of all GPs, with lots of chasing to catch the non-responders? Telephoning them at work? A postal survey, at least? No. It was an informal poll through doctors.net.uk, an online chat site for doctors, producing this major news story about a profession threatening a revolt.

The statement to which doctors were invited to respond was this: ‘GPs should carry out abortions in their surgeries’. You can ‘strongly agree’, ‘agree’, ‘don’t know’, ‘disagree’ or ‘strongly disagree’.

I might be slow, but I myself do not fully understand the statement. Is that ‘should’ as in ‘should’, as in, ‘ought to’, as in ‘coerced’? And in what circumstances? With extra training, time, and money? With extra systems in place for adverse outcomes? This is a chat website where doctors go to grumble, cynically, in good company. Are they saying ‘no’ because this new responsibility would involve more work and lower morale? Would you even click the ‘abortion’ link in the chat pages index if you didn’t already have an interest in abortion?

And stepping bravely beyond the second word ‘should’, what does ‘carry out abortions in their surgeries’ mean? Looking at the comments in the chat forum – as I am doing right now – plenty of the doctors seemed to think the question referred to surgical abortions, not the relatively safe oral pill for termination of early pregnancy. Doctors aren’t all that bright, you see, and questionnaire respondents in general may not necessarily know what you’re thinking about if you don’t write a proper question.

Here are some quotes from the doctors in the discussion underneath this poll. ‘This is a preposterous idea. How can GPs ever carry out abortions in their own surgeries. What if there was a major complication like uterine and bowel perforation?’ ‘The only way it would or rather should happen is if GP practices have a surgical day care facility as part of their premises which is staffed by appropriately trained staff, i.e. theatre staff, anaesthetist and gynaecologist … any surgical operation is not without its risks, and presumably [we] will undergo gynaecological surgical training in order to perform.’ ‘What are we all going on about? Let’s all carry out abortions in our surgeries, living rooms, kitchens, garages, corner shops, you know, just like in the old days.’

But my favourite is this: ‘I think that the question is poorly worded and
I hope that DNUK
do not release the results of this poll to the
Daily Telegraph
.’

A New and Interesting
Form of Wrong

Guardian
, 27 November 2010

Wrong isn’t enough: we need interestingly wrong, and this week that came in some research from Stonewall, an organisation for which I generally have great respect, which was
reported in the
Guardian
. Stonewall has conducted a survey, and
its press release
says it shows that ‘the average coming-out age has fallen by over twenty years’.

People may well be coming out earlier than before – intuitively, that seems plausible – but Stonewall’s survey is flawed by design, and contains some interesting statistical traps.

Through social networking sites, Stonewall asked 1,536 people – who were already out – how old they were when they came out. Among the over-sixties, the average age was thirty-seven; those in their thirties had come out at an average age of twenty-one; in the group aged eighteen to twenty-four, the average age for coming out was seventeen.

Why is the age coming down? Here’s one reason. Obviously, there are no out gay people in the eighteen-to-twenty-four group who came out at an age later than twenty-four; so the average age at which people in the eighteen-to-twenty-four group came out cannot possibly be greater than the average age of that group, and certainly it will be lower than, say, thirty-seven, the average age at which people in their sixties came out.

For the same reason, it’s very likely indeed that the average age of coming out will increase as the average age of each age group rises. In fact, if we assume (in formal terms we could call this a ‘statistical model’) that at any time, all the people who are out have always come out at a uniform rate between the age of ten and their current age, you would get almost exactly the same figures (you’d get fifteen, twenty-three and thirty-five, instead of seventeen, twenty-one and thirty-seven). This is almost certainly why ‘the average coming-out age has fallen by over twenty years’: in fact you could say that Stonewall’s survey has found that on average, as people get older, they get older.

But there is also
an interesting problem
around whether, with the data it collected, Stonewall could ever have created a meaningful answer to the question ‘Have people started coming out earlier?’ It’s a difficult analysis to design, because in each age band there is no information on gay people who are not yet out, but may come out later, and also it’s hard to compare each age band with the others.

You could try to fix this by restricting all the data to include only those people who came out under the age of twenty-four, and then measure the mean age of coming out for each age group (eighteen-to-twenty-four, thirties, sixty plus) in this subgroup alone. That would give you some kind of answer for this very narrow age band, but even that makes some very dubious statistical assumptions. And if we allowed ourselves this move, we’d then be working with an extremely small set of data: only thirty-three respondents aged over sixty, for example
.

Even then, the discussion of this poll also assumes that the age at which people know their sexuality has remained unchanged. Some believe that everyone’s sexuality is fixed and known from birth – I may be walking into a minefield here – but if the age at which people recognise their own sexuality is changing, then a more relevant figure by which to measure discomfort at coming out might be the delay, rather than the absolute age.

I thought I’d already covered all the ways that a survey could get things wrong, but this one brought something new. Maybe we should accept that all research of this kind is only produced as a hook for a news story about a political issue, and isn’t ever supposed to be taken seriously. In any case, my intuition is that a well-constructed study would probably confirm Stonewall’s original hypothesis. But it’s still fun to dig.

‘
Hello Madam
, Would You Like Your Children to Be Unemployed?’

Guardian
, 20 November 2010

Obviously I like nerdy days out: like Kelvedon Hatch
secret nuclear bunker
, maybe, with its sign on the A128 pointing the way to the ‘Secret Nuclear Bunker’. Last month eight of us commissioned a boat to get onto a rotting man-made World War II sea-fort in the middle of the ocean through
Project Redsand
(we genuinely thought we might die climbing the ladders), and a couple of weeks earlier, myself and Mrs Bad Science
travelled to Dungeness
, where a
toytown narrow-gauge railway
takes you through amusement parks and back gardens, past
Derek Jarman’s house
, then into barren wasteland, before depositing you incongruously at the base of a magnificent, enormous, and
terrifying nuclear power station
.
¹

I tell you this, because I should declare an interest: I quite like nuclear power stations, not just because they’re clever, or even because I regretfully concede they might be one of our least bad options for power. I secretly like nuclear power stations because they remind me, in the way nostalgia makes us pine for things we disliked at the time, of a childhood in the early 1980s when I knew that I would definitely die in a nuclear holocaust.