I Think You'll Find It's a Bit More Complicated Than That (23 page)

Mbeki’s Health Minister would appear on television to talk up the risks of antiretroviral medication and talk down its benefits, promoting garlic and sweet African potato as effective treatments for Aids. The South African government’s stall at the 2006 World Aids Conference in Toronto was described by other delegates as ‘the salad stall’, because that’s all it contained. It has been estimated that between 2000 and 2005, around 350,000 people with Aids died unnecessarily in South Africa as a result of these ideas. That’s quite a death toll, for some ideas.

How does this happen? Perhaps Aids is just too big to think about clearly. Twenty-five million people have died of it so far, three million in the past year: these figures are so vast, so overwhelming, that it’s hard to mount an appropriate emotional response.

Perhaps the undeniable crimes of the pharmaceutical industry make conspiracy theories about its effective products believable, lending them a kind of poetic truth. It does, after all, adhere to cruel and murderous pricing policies, and only this year, regulations were explicitly changed so that clinical trials conducted by American companies on people in the developing world are no longer subject to the same high ethical standards as those conducted on US citizens.

Pharmaceutical companies, of course, are not all bad, just as there are many good people in the Catholic Church (the overwhelming majority, I would imagine). And a cheap, single dose of the drug Nevirapine, we should remember, has been shown to reduce the risk of a pregnant mother passing on HIV to her baby by half.

But we are mistaken if we imagine that medicine moves forward through technology. In the past that was probably true: antibiotics, intensive care units, and all the tools and technologies of modern medicine are dazzling. But much of the power lies with simple ideas.

So, let me tell you about diarrhoea. What are our two biggest weapons against torrential watery stool? One is telling people to wash their hands: it’s been demonstrated that this can halve the spread of diarrhoea, and so it could save a million lives a year. The other is even simpler, and even more powerful: telling people to rehydrate, using water with added sugar and salt. This is new, it has been carefully researched and refined, and despite being a simple idea which anyone can follow, it has caused deaths from diarrhoea to plummet, saving at least fifty million lives since its universal adoption in the 1980s.

We can go higher. A hundred million people died in the last century from smoking. Around a billion are expected to die this century (because the cigarette industry has been so successful in China). But equally, tens of millions of lives will be saved, because Richard Doll and his colleagues diligently collected and analysed data on the smoking habits and deaths of a few thousand doctors fifty years ago to pull out just one key fact: smoking kills.

With this idea, with handwashing, with rehydration fluid, and with the methods and principles that gave us these ideas, a small group of softly-spoken saints have saved more people than you would meet in a thousand lifetimes.

The royalties from
The Atheist’s Guide to Christmas
will go to the Terrence Higgins Trust. I have seen the work they do up close, working in an HIV clinic in South London. THT offers practical support, but the most powerful work their staff do, to my mind, is in sharing information, destigmatising, informing, preventing infections and improving compliance with treatment regimes. Through the application of common sense, wit, compassion and evidence, they save lives. This is the future of medicine.

And it’s eight teaspoons of sugar and one of salt in a litre of water, if you ever need to know.

Merry Xmas.

‘Exams Are Getting Easier’

Guardian
, 21 August 2010

Pass rates are at 98 per cent. A quarter of grades are an A or higher. This week, every newspaper in the country was filled with people asserting that exams are definitely getting easier, and other people asserting that exams are definitely not getting easier. The question is always simple: how do you know?

Firstly, the idea of kids getting cleverer is not ludicrous.
‘The Flynn Effect’
is a term coined to describe the gradual improvement in IQ scores. This has been an important problem for IQ researchers, since IQ tests are peer referenced: that is, your performance is compared against everyone else, and the scores are rejigged so that the average IQ is always 100. Because of the trend to higher scores, year on year, you have to be careful not to use older tests on current populations, or their scores come out spuriously high, by the standards of the weaker average population of the past. Regardless of what you think about IQ tests, the tasks in them are at least relatively consistent. That said, there’s also some evidence that the Flynn effect has slowed in developed countries recently.

But ideally, we want research that addresses exams directly. One approach would be to measure current kids’ performance on the exams of the past. This is what the Royal Society of Chemistry did in its report
‘The Five Decade Challenge’
in 2008, running the project as a competition for sixteen-year-olds, which netted them 1,300 self-selecting higher-ability kids. They sat tests taken from the numerical and analytical components of O-level and GCSE exams over the past half-century, and it was found that performance against each decade rose over time: the average score for the 1960s questions was 15 per cent, rising to 35 per cent for the current exams (though with a giant leap around the introduction of GCSEs, after which the score remained fairly stable).

There are often many possible explanations for a finding. These results could mean that exams have got easier, but it’s also possible that syllabuses have changed, so modern kids are less prepared for older-style questions. When the researchers looked at specific questions, they found that some things had been removed from the GCSE syllabus – because they’d moved up to A-level – but that’s drifting unwelcomely towards anecdote.

Another approach would be to compare performance on a consistent test, over the years, against performance on A-levels. Robert Coe at Durham University produced a
study of just this
for the Office of National Statistics in 2007. Every year since 1988 a few thousand children have been given the Test of Developed Abilities, a consistent test (with a blip in 2002) of general maths and verbal reasoning skills. The scores saw a modest decline over the 1990s, and have been fairly flat for the past decade. But the clever thing is what the researchers did next: they worked out the A-level scores for children, accounting for their TDA scores, and found that children with the same TDA score were getting higher and higher exam results. From 1988 to 2006, for the same TDA score, A-level results rose by an average of two grades in each subject.

It could be that exams are easier. It could be that teaching and learning have improved, or that teaching has become more exam-focused, so kids at the same TDA level do better in A-levels: this is hard to measure. It could be that TDA scores are as irrelevant as shoe size, so the finding is spurious.

Alternatively, it could be that exams are different: they might be easier, say, with respect to verbal reasoning and maths, but harder with respect to something else. This, again, is hard to quantify. If the content and goals of exams change, then that poses difficulties for measuring their consistency over time, and it might be something to declare loudly (or to consult employers and the public about, since they seem concerned).

Our last study thinks more clearly along those lines: some people do have clear goals from education, and they can measure students against this yardstick, over time.
‘Measuring the Mathematics Problem’
is a report done for the Engineering Council and other august bodies in 2000, analysing data from sixty university maths, physics and engineering departments which gave diagnostic tests on basic maths skills to their new undergraduates each year. These were tests on things that mattered to university educators, and if something educational matters to them, we might think it matters overall. They found strong evidence of a steady decline in scores on their own tests, over the preceding decade, among students accepted onto degree courses where they would need good maths skills.

Sadly they didn’t control for A-level grade, so we can’t be sure how far they were comparing like with like, but there are various plausible explanations for their finding. Maybe maths syllabuses changed, and were less useful for maths and engineering degrees. Maybe the cleverest kids are doing English these days, or becoming lawyers instead. Or maybe exams got easier.

If you know of more research, I’d be interested to see it, but the main thing that strikes me here is the paucity of work in the field. There’s a man called Rupert Sheldrake who believes that pets are psychic: they know when their owners are coming home, that sort of thing. Obviously we disagree on a lot, but we chat, and are friendly, and once when we were talking he came out with an excellent suggestion: maybe 1 per cent – or even 0.01 per cent – of the total UK research budget could be given to the public, so that they could decide what their research obsessions were. Maybe most of this money would get spent on psychic pets, or research into which vegetables cure cancer, but since we’re all clearly preoccupied with the idea, I’d like to think that some of it, possibly, might get spent on good-quality, robust research to find out whether exams are getting easier.

Over There!
An Eight-Mile-High Distraction Made of Posh Chocolate!

Guardian
, 1 August 2009

This week the Food Standards Agency published
two review papers
showing that organic food is no better than normal food,
in terms of composition
,
or health benefits
. The Soil Association’s response has been swift, and it has been given prominent,
blanket right of reply
throughout the media. That is testament to the lobbying power of a £2 billion industry, and the cultural values of journalists. I don’t care about organic food, but I am interested in bad arguments. The Soil Association has made three.

Firstly, it says that the important issue with organic food is not the personal health benefit (there is none to be found in this data), but rather the benefit to the environment. This strategy – ‘
Don’t talk about that
, talk about this’ – is a popular one, but it is cheap: we can talk separately about the environmental issues with organic food, but right now we are talking about the health effects.

Secondly, it says that the health benefits of organic food are related to pesticides, and cannot be measured by the evidence that has been identified and summarised in the FSA paper. This, once more, is gamesmanship. There was no evidence of health benefits to individuals. Possibly the Soil Association is proposing that there are health benefits which somehow cannot be measured: it is hard to disentangle the health benefits of eating organic food from other beneficial features of people’s lives, for example, when you measure their health in a thirty-year study. In this case it is expressing a position of faith, not evidence. Or it is proposing that there are health benefits which could be measured, but have not been measured yet. In that case, again, this is faith rather than evidence, but it could start recruiting researchers now, using some small portion of its industry’s £2 billion revenue to investigate these beliefs with fair tests.

Lastly, like many vitamin-pill peddlers and
pharmaceutical companies before it
, the Soil Association seeks to undermine the public’s understanding of what a ‘systematic review’ is. It says that the report has deliberately excluded evidence to produce the answer that organic food is no better. The accusation is one of cherry-picking, and it is hard to see how it can be valid in the kind of studies that have been published here. These are ‘systematic reviews’: before you begin collecting papers to include, you specify how you will search for evidence, what databases you will use, what types of studies you will use, how you will grade the quality of the evidence (to see if it was a ‘fair test’), and so on.

What does the Soil Association think these systematic reviews have ignored? As an
example, from its press release
, the industry body is ‘disappointed that the FSA failed to include the results of a major European Union-funded study involving thirty-one research and university institutes and the publication, so far, of more than one hundred scientific papers, at a cost of 18 million Euros, which ended in April this year’. It gave the link to
www.qlif.org
.

On this website, you will find the QLIF
list of 120 papers
. Almost all are irrelevant. The first fourteen are on ‘consumer expectations and attitudes’, which are correctly not included in a systematic review of the evidence on food composition and health. Then there are twenty-two on ‘effects of production methods’. Here you might expect to find more relevant research, but no.

The first paper (‘The effect of medium term feeding with organic, low input and conventional diet on selected
immune parameters in rat
’), while interesting, will plainly not be relevant to a systematic review on nutrient content. The same is true of the next paper, ‘
Salmonella Infection Level
in Danish Indoor and Outdoor Pig Production Systems measured by Antibodies in Meat Juice and Faecal Shedding on-farm and at Slaughter’: you might love its results, you might hate them; either way, it doesn’t matter. This paper is simply not relevant to a review on nutrient content.

What’s more, the overwhelming majority of these studies are unpublished conference papers, and some are just brief descriptions of the fact that somebody made an oral presentation at a meeting. The systematic review correctly looked only at good-quality data published in peer-reviewed academic journals, and with good reason: we know that conference papers are unreliable sources of information, that they often change between conference and publication, and that they are often never published at all.