« The secret to great comedy | Main | The Arnebeck mystery solved »

December 13, 2004

Abrogation of duty

What I'm about to discuss is kind of old news. But it's been weighing on me, and I have to get it off my chest. I find it deeply disturbing.

Most jobs are just jobs. But certain professions carry with them a duty, a sacred obligation that one must uphold in order to meet basic ethical standards. Doctors have a duty never to harm their patients. Lawyers have a duty to always represent the best interests of their clients. Journalists have a duty to report the truth. And academic researchers have a duty too: the purpose of academic inquiry must be to broaden our knowledge, not to hinder it. The prestige of a reputable journal must never be used to mislead.

Which brings me to Mortality before and after the 2003 invasion of Iraq: cluster sample survey, by Les Roberts, Riyadh Lafta, Jamal Khudhairi, Richard Garfield, and Gilbert Burnham, hereinafter RLKGB. This study, out of Johns Hopkins University, was published in the British medical journal The Lancet, and purported to investigate, well, mortality before and after the 2003 invasion of Iraq.

What did they find? Why, that 98,000 additional Iraqis were killed as a result of the war. And that's how the study was sort of reported in the news media, which, as I've said, has a duty of its own. Take a gander:

Study puts civilian toll in Iraq at over 100,000, International Herald Tribune
More than 100,000 civilians have probably died as direct or indirect consequences of the U.S.-led invasion of Iraq, according to a study by a research team at Johns Hopkins University's Bloomberg School of Public Health in Baltimore.

100,000 Civilian Deaths Estimated in Iraq, Washington Post
One of the first attempts to independently estimate the loss of civilian life from the Iraqi war has concluded that at least 100,000 Iraqi civilians may have died because of the U.S. invasion.

Iraq death toll 'soared post-war', BBC
Poor planning, air strikes by coalition forces and a "climate of violence" have led to more than 100,000 extra deaths in Iraq, scientists claim.

Note the phraseology in each of these: "More than 100,000 civilians", "at least 100,000 civilians", "more than 100,000 extra deaths". Never mind the rounding up from 98,000 to 100,000, note the comparatives setting 100,000 as a minimum threshold. We'll get to that.

So, this academic study found that 100,000 or more civilians died as the result of the war. Fine. What was their methodology?

They didn't go around and count body bags, death certificates, or graves or anything like that. Admittedly, these methods would be difficult, especially in a war-torn country. So instead, they sampled. They interviewed a subset of the population and extracted the results.

Now, this is perfectly valid. Contrary to popular belief, sampling -- polling -- isn't guessing and it isn't voodoo, it's science. If you randomly sample a population, you can make definitive probabilistic statements about that population. But it's tricky. Things can go wrong in polling, which can mess up your results.

For starters, there's the quality of the sample. The ideal sample is totally random, meaning each member of the population has an equal probability of being sampled. In thought experiments, like pulling marbles out of a jar, this ideal can be achieved. In the real world, we never come close. We do the best we can, but we recognize that sampling error can skew the results. For example, public opinion polls in the United States are generally conducted by telephone. But some people do not have telephones, or do not have telephones that can be legally called for polling purposes. Some people are rarely home at times when pollsters call, making them less likely to be sampled. Less obvious but also a factor is that some people have more than one telephone, making them more likely to be sampled. And some people, gosh darn them, just don't like to answer pollsters. Unless all of these subpopulations are exactly representative of the main population -- and they're not -- the results are less accurate than they could be.

So how was the quality of the sample in the RLKGB study? Well, maybe not so good. Rather than attempting to randomly sample the entire population, they used a technique called a "cluster survey", where certain areas were chosen in the hopes that they were representative, and those areas were sampled exclusively. Door-to-door interviews were used for data collection, and households were interviewed until enough samples were collected for that cluster. Were the clusters truly representative? Possibly, possibly not. Were the households that were not home when the interviewers came calling and the households where interviews were refused representative of the whole? Possibly, possibly not.

But this isn't really my point.

Another potential pitfall preventing polling precision is that of recall bias, which is also known to pollsters by the more technical term "lying". This happens whenever sampled individuals perceive some benefit from answering the poll a certain way and skew their answers accordingly. This perceived benefit need not be very large, and in fact many lie to pollsters simply to feel good about themselves. For example, according to author Michael Fumento, a poll was taken during the energy crisis of the 1970s to measure public response. The poll found that many Americans, consciencious of the need to be good citizens by conserving energy, had taken a number of steps to curb consumption. Good for them. Except among other things, repondents were asked if they'd installed a thermidor in their cars to save fuel, and a number responded in the affirmative. Again, good for them, except "Thermidor" is either the eleventh month of the briefly-used French Revolutionary calendar or a lobster dish which takes its name from the month, neither of which does much for fuel economy.

Pollsters are aware that respondents may lie if they will either feel good about the false answer or bad about the true answer, and for other motives as well. RLKGB discuss recall bias, which is to their credit, but only to dismiss it:

We believe it unlikely that recall bias existed in the reporting of non-infant deaths, because of the certainty and precision with which these deaths were reported, and the importance of burial ceremonies in the Iraqi culture.

No? I can think of a number of reasons why someone might misreport a death. The Coalition invasion of Iraq enjoys wide but not universal support among Iraqis. Iraqis are not stupid, and an Iraqi who is polled regarding civilian fatalities in his household can deduce that if the results of the poll show a high death toll, that would not be helpful to the Coalition. An Iraqi opposed to the Coalition's aims might easily make the leap to indulging in a little "recall bias".

And then there are the fatalities in the pre-Coalition period. Is it not possible that a household which had a member "disappeared" might be reluctant to disclose the fact to a stranger with a clipboard? Any underreporting of fatalities during the reign of Saddam's Ba'athists would tend to skew the sample in favor of a greater increase of casualties after his fall.

But this, again, is not my point, which I will now finally get to.

Let's for the moment stipulate that RLKGB's methodology was ironclad. Let's say they got a perfectly representative statistical sample, as unlikely as that seems, and let's say their fatality count was 100% accurate.

This brings me to the concept of margin of error. Sampling is a statistical process, and as such it results in not certainties but probabilities. After collecting your sample and doing your math, you wind up with a set of probabilities that follows a normal distribution, better known as a bell curve. You can't say that certainly 55% of the population favors Candidate X, you can only say that the probability is, say, 60% that between 54% and 56% of the population favors him. In technical terms, 55% of the population favors Candidate X with a 60% confidence interval of ±1%.

The margin necessary to achieve a confidence interval of 95% is what pollsters usually mean when they refer to margin of error. Odds are nineteen out of twenty that a poll is accurate within the margin of error. That twentieth poll won't be.

As I said, the probabilities form a bell curve. The area under the whole curve is exactly 1.0, and the area under the curve for a given interval is the probability that the characteristics of the population fall within that range. The peak of the curve is at the distribution of the sample, representing the fact that (in the example) 55% support for Candidate X is more likely than any other level. The smaller the margin of error, the sharper the peak of the bell curve, representing the fact that a narrower range is necessary to cover an area of 0.95. Polls with a wider margin of error have flatter, gentler peaks, representing the large range necessary to achieve that area of 0.95.

As it happens, the key factor in determining margin of error is sample size. Population size matters too, but not nearly as much as sample size, which is why a sample of 1,000 individuals suffices to establish a margin of error of only ±3% over the entire United States. So, again, let's assume that RLKGB's sample is the theoretical ideal. As it happens, they sampled a grand total of 142 post-invasion fatalities. That's an exceedingly small sample for a population in excess of 25 million. What's the margin of error it produces?

Well, uh, ±92%. Yes, that's right, ±92%. That's not a typo, there's not an extra digit in there, that's their margin of error. I'll spell it out just to make sure: ninety-two percent. The actual number of excess fatalities might have been as many as 194,000... or as few as 8,000. Yes, that's right, they might have overstated the number of fatalities by a factor of twelve.

The graph of this thing wouldn't look like a bell. It'd look like a nearly perfectly horizontal line, with a very, very slight hump in the middle. In a nutshell, and not to put too fine a point on it, this study is absolutely worthless. A wild guess would be about as accurate.

Now, getting back to the reports of this study in the media, where it was trumpeted as finding "more than" or "at least" 100,000 extra fatalities as a result of the invasion? Those reports are baldfaced falsehoods. The study found nothing of the sort. But as much as I'd like to blame the media for this -- and I would, and I do -- RLKGB deserve a big chunk of it themselves, for the papers were just reporting what they were told. From the study itself, right after they admit their absurd margin of error, the authors claim:

Making conservative assumptions, we think that about 100000 excess deaths, or more have happened since the 2003 invasion of Iraq.


The media still shares the blame for this, because if they'd done their jobs and actually read the study, they never would have swallowed this crap whole, and I suspect their own bias and eagerness to embarrass the Coalition played a part in their inaction.

But what really disturbs me about this is that this garbage was done by researchers at an extremely reputable University, published by an extremely reputable journal, and presumably reviewed by extremely reputable peers. As I began this rant, those who produce research have a duty. The purpose of research is supposed to be to educate. This study, which makes me shudder even to dignify it with the name "research", had one purpose: to deceive. The so-called academicians who produced it intended that those who learned of it would come away with serious misconceptions about the world. And that, to me, is utterly inexcusable. May they all be damned.

December 13, 2004 in Current Affairs | Permalink


TrackBack URL for this entry:

Listed below are links to weblogs that reference Abrogation of duty:


The comments to this entry are closed.