It seems to me that the government of the US having decided not to count casualties in Iraq has abandoned the right to criticise other people who think this an area worthy of study. [Similarly the UK government for its support of the US policy - whether formally stated or merely tacit.]

I recall reading that the Conservative government in the 1950s, even without the benefit of an ethical foreign policy, did undertake a count of the casualties it had caused in Suez.

Isn’t it odd that our government - which elsewhere is prepared to set and monitor targets for virtually everything - from the number of song birds to the number of obese children in the UK - is not prepared to monitor the impact of its policies on the life expectancy of the Iraqi population?

What an excellent blog post- thank you very much. I might have to start doing some YouGov surveys again to pay you for it!

Many thanks Anthony for your thoughtful and useful comments on this. Now if only someone in the media would go through the pro’s and con’s of the survey like that, we might get somewhere

Bloggers like Drinking From Home have looked at chunks of the data. The 30 deaths from car bombs in a 1 year period extrapolate to 60,000-plus car bomb deaths a year. According to Iraq Body Count the average (reported) car bomb kills 7 or 8 people.

Is it possible that 1200 a week are being killed in about 150 car-bombs, of which only a fraction are reported ? I can see how gunshot deaths or sickness goes unreported, but car-bombs ?

It could be - I don’t think any of us have a particularly good idea of what proportion of events in Iraq get anywhere near journalists, to what extent journalists are out and about in the smaller towns and cities of Iraq, and in the areas of high insurgency, and to what extent they are safely locked away in the green zone. If a car bomb goes on in the equivalent of Hicksville, Falluja I doubt there is a a journalist there to report it.

The average number of people killed in a reported car bomb is 7 to 8, but that is, as you say, reported ones. Perhaps car bombings that kill just one or two people are so ten a penny no one bothers to report them any more. That’s probably the case with shootings, and whether it is a gun or bomb doesn’t make it less of a human tragedy.

In the same way there are no independent figures to calibrate this with, there are no figures with which to judge what proportion of deaths are being reporting in media outlets (or more to the point, media outlets which meet IBC criteria).

What a fantastic blog post - I’d not bothered really to look at the methodology of that before - really good stuff.

It would be possible to estimate the size of the urban areas bias effect without further fieldwork, by digitising some roadmaps (or sufficiently high-resolution satellite images) and simulating the distribution of points identified by methods they state, then comparing it to the uniform distribution that would be expected from visiting randomly-generated coordinates with the aid of a GPS unit.

Here’s the description from the 2004 paper:
“Once a town, village, or urban neighbourhood was selected, the team drove to the edges of the area and stored the site coordinates in a global positioning system (GPS) unit. We assumed the population was living within a rectangle, with the dimensions corresponding to the distances spanned between the site coordinates stored in the GPS unit. The area was drawn as a map subdivided by increments of 100m. A pair of random numbers was selected between zero and the number of 100 m increments on each axis, corresponding to some point in the village. The GPS unit was used to guide interviewers to the selected point. Once at that point, the nearest 30 households were visited.”

And from the 2006 paper:
“The third stage [of sampling] consisted of random selection of a main street within the administrative unit from a list of all main streets. A residential street was then randomly selected from a list of residential streets crossing the main street. On the residential street, houses were numbered and a start household was randomly selected. From this start household, the team proceeded to the adjacent residence until 40 households were surveyed.”

Note in both cases all the samples were taken in settlements large enough to appear in the population statistics — isolated farmhouses etc. were not sampled. Settlements were selected by random sample biased by population (so that a town of 100,000 would be 100 times more likely to be selected than a village of 1,000). Once a settlement was selected the difference between the two sampling methods takes effect. Without actually doing the simulation it’s hard to judge what the effect of their selection of a thoroughfare, then a crossing street, then a house within it, followed by visiting adjacent houses, would be compared to selection of a random point followed by visiting the nearest households.

There are high-res satellite images of some towns in Iraq on Google Maps, so it would certainly be possible to test this without spending money on additional satellite imagery if you were so minded. Nevertheless, as Anthony says, the largest possible bias that would arise from this problem is still small compared to the size of the effect they’ve detected.

Though I disagree with you that the survey is sound it’s good to read a well thought out analysis of it.

A survey might be honestly done, and have a sound methodology but still, like this one, produce results which aren’t logical. The possibility that the media has failed to notice the deaths of circa 7% of the Iraqi male population is unlikely.

You have put some valid points across here, a very interesting read and has opened my eyes to the situation.

Excellent, stilumating post. On the two provinces missed - it’s my understanding that the final figures are based on these provinces reporting no excess deaths. This would surely depress rather than exaggerate the total number. No?

Anthony,

Firstly thanks for taking it on.

Secondly, is it possible that there could be a corruption element, where people pay doctors to issue phoney death certificates, be it to allow someone to dissappear or to claim some sort of state death benefit.

I think the study is sound and a figure near 500,000 seems to me about right, but for want of a better term “death Fraud” could be an issue.

Peter.

One interesting fact is that the 2006 survey seems to support the 2004 survey by coming up with a figure of 112,000 extra deaths in 2003/2004.

I know that the survey was carried out in extremely difficult circumstances but one note in the survey states that:

“Once in the clusters, the teams faced suspicion initially, especially at the first house selected in the random process. Lengthy explanations of the purposes of the survey—and that it would help the Iraqi people—were necessary to allay fears. In some areas, people were more welcoming, and all but a very few of the entire sample were eventually very cooperative.”

Could this have influenced the replies received?

Excellent blog piece Anthony and thanks for explaining matters so clearly.

Lancet post number 100

Anthony Wells: So, what could have gone wrong? The more excitable fringes of the US blogosphere have come out with some interesting stuff. Let’s look at criticisms that don’t hold water first. Firstly, the turnout is unbelievably high. The report…

[…] The immediate problem with this charge is that, as it turns out, phenomenally high response rates are apparently very common in Iraq, and not just in this survey. UK Polling Report says the following: The report suggests that over 98% of people contacted agreed to be interviewed. For anyone involved in market research in this country the figure just sounds stupid. Phone polls here tend to get a response rate of something like 1 in 6. However, the truth is that – incredibly – response rates this high are the norm in Iraq. Earlier this year Johnny Heald of ORB gave a paper at the ESOMAR conference about his company’s experience of polling in Iraq – they’ve done over 150 polls since the invasion, and get response rates in the region of 95%. In November 2003 they did a poll that got a response rate of 100%. That isn’t rounding up. They contacted 1067 people, and 1067 agreed to be interviewed. […]

‘it still seems unlikely that a systemic bias of this sort could overall lead to a upwards bias in the number of deaths.’

The surveyors were Arabic-speaking doctors from Baghdad, who in the previous survey managed to sample Fallujah. Any areas they skipped would presumably be ones that represented a danger to them, which is not necessarily the same as being dangerous to the residents.

It would be interesting to see the ethnic breakdown of respondents, if it was recorded.

Failing that, shouldn’t there be another UN survey due soon?

Why would oversampling urban areas lead to an *unrepresentative* survey? The top three cities account for a third of the total population, and the top ten a fair majority. The violence is in the cities, because there’s either no-one to fight or nothing to fight over outside them.

“It seems to be that this approach should skew the location of clusters towards urban areas, and make it less likely that rural areas and areas with informal housing like refugee camps would be selected. If violence is concentrated in urban areas this could skew the sample and give an artifically high number of casualties. ”

Not sampling refugee camps should lead to an undercount of deaths, under the reasonable assumption that refugees are more likely to have suffered violence than those who stayed in place.

Similarly, the fact that 19 (?) households were not sampled due to the buildings being empty would be a bias-down factor.

Ralph: “The possibility that the media has failed to notice the deaths of circa 7% of the Iraqi male population is unlikely. ”

First, the media has written many, many, many articles on death and violence in Iraq. And that’s what they’ve seen under stringent constraints on reporting. If foreign and native journalists were more free to move around and see things, it’d probably look worse. So that’s wrong.

Second, the central government of Iraq is not that functional. Most of the ministry buildings were burned and looted in 2003. Most of the civil servants were then purged under de-baathification.
There’s at least one US newspaper article from Dec 2003 stating that the CPA forbade the counting of civilian deaths. On top of that, the Iraqi and US governments have every incentive to downplay the violence. Currently, Sadr’s militias run the ministry of Health, and many hospitals. Where they seem to feel that Sunnism can be cured with a shot - from an AK-47.

To me, the idea that Iraqi government statistics are trustworthy is a huge assumption.

[…] Among the heavy coverage of the Lancet mortality figures by bloggers (see Tim Lambert for some of the best), this take, by “UK Polling Report” is particularly interesting: Firstly, the turnout is unbelievably high. The report suggests that over 98% of people contacted agreed to be interviewed. For anyone involved in market research in this country the figure just sounds stupid. Phone polls here tend to get a response rate of something like 1 in 6. However, the truth is that - incredibly - response rates this high are the norm in Iraq. Earlier this year Johnny Heald of ORB gave a paper at the ESOMAR conference about his company’s experience of polling in Iraq - they’ve done over 150 polls since the invasion, and get response rates in the region of 95%. In November 2003 they did a poll that got a response rate of 100%. That isn’t rounding up. They contacted 1067 people, and 1067 agreed to be interviewed. […]

Gary Gatter: ‘One interesting fact is that the 2006 survey seems to support the 2004 survey by coming up with a figure of 112,000 extra deaths in 2003/2004.’

Another interesting fact is that the 2004 survey said that 40% of those deaths were non-violent, whereas the 2006 survey says that in the first eighteen months, the period covered by the 2004 survey, the non-violent death rate was actually lower than before the war. One of these assertions must be untrue. See link

Found this interesting bit of comment:

“However, the key to the validity of cluster sampling is to use enough cluster points. In their 2006 report, “Mortality after the 2003 invasion of Iraq: a cross-sectional sample survey,” the Johns Hopkins team says it used 47 cluster points for their sample of 1,849 interviews. This is astonishing: I wouldn’t survey a junior high school, no less an entire country, using only 47 cluster points.

Neither would anyone else. For its 2004 survey of Iraq, the United Nations Development Program (UNDP) used 2,200 cluster points of 10 interviews each for a total sample of 21,688. True, interviews are expensive and not everyone has the U.N.’s bank account. However, even for a similarly sized sample, that is an extraordinarily small number of cluster points. A 2005 survey conducted by ABC News, Time magazine, the BBC, NHK and Der Spiegel used 135 cluster points with a sample size of 1,711–almost three times that of the Johns Hopkins team for 93% of the sample size.”

There was a lot more on the webite, were there too few clusters?

[…] The immediate problem with this charge is that, as it turns out, phenomenally high response rates are apparently very common in Iraq, and not just in this survey. UK Polling Report says the following: The report suggests that over 98% of people contacted agreed to be interviewed. For anyone involved in market research in this country the figure just sounds stupid. Phone polls here tend to get a response rate of something like 1 in 6. However, the truth is that – incredibly – response rates this high are the norm in Iraq. Earlier this year Johnny Heald of ORB gave a paper at the ESOMAR conference about his company’s experience of polling in Iraq – they’ve done over 150 polls since the invasion, and get response rates in the region of 95%. In November 2003 they did a poll that got a response rate of 100%. That isn’t rounding up. They contacted 1067 people, and 1067 agreed to be interviewed. […]

Thank you Anthony for such detailed and mature analysis of such a controversial topic. Its nice to finally see some analysis of the report which is sensible and not pushing an agenda one way or another.

Thanks to Lopakhin for pointing out the discrepancy, I only looked at the excess deaths not the proportions of violent/non violet deaths.

Soru:

Had the 2004 Lancet report included areas with exceptionally high violent death rates, such as Fallujah, it would have skewed the figures, considering the US military has pretty much destroyed it twice and soldiers shot or blew up anything that moved including dogs while there.

Except Fallujah was not included in the 2004:

“Perhaps most damning in Roberts’ reply - in light of media criticism of the Lancet‘s alleged exaggeration of civilian deaths - was his refutation of the claim that the uneven levels of violent unrest in Iraq compromised the accuracy of the figures. In fact the study not only accounted for this variability, it erred on the side of caution by excluding data from Fallujah where deaths were unusually high. Moreover, other violent hotspots - such as Ramadi, Tallafar and Najaf - were all passed over in the sample by random chance. This suggests that the actual total of civilian deaths is likely to be higher than 100,000. Indeed, it would make far more sense for the media to be criticising the report authors for under-estimating the number of deaths.”

From Medialens, Burying the Lancet Part1

This is confirmed by the Findings section of the 2004 Lancet report, in both the summary and within the full text:

“Findings

The risk of death was estimated to be 2·5-fold (95% CI 1·6–4·2) higher after the invasion when compared with the preinvasion period. Two-thirds of all violent deaths were reported in one cluster in the city of Falluja. If we exclude the Falluja data, the risk of death is 1·5-fold (1·1–2·3) higher after the invasion. We estimate that 98000 more deaths than expected (8000–194000) happened after the invasion outside of Falluja and far more if the outlier Falluja cluster is included.

“The possibility that the media has failed to notice the deaths of circa 7% of the Iraqi male population is unlikely.”

Are you having a laugh? Is he having a laugh?

i thing US having a serious problems in IRAQ and thats ofcourse because they delete the 1st army and all of them were trained good so they can’t exit from tats truples if they didn’t solve the ARMY problems … and the new army is very bad and fight the original things that the IRAQy People believed in … so they have to sole the Army problem

In the light of this discussion, this may be of interest:
http://www.rhul.ac.uk/Resources/Helper_apps/Message.asp?ref_no=367

“The report suggests that over 98% of people contacted agreed to be interviewed. For anyone involved in market research in this country the figure just sounds stupid. Phone polls here tend to get a response rate of something like 1 in 6.”

Several years ago, when living in Stornoway I was telephoned for BBC poll about Sunday Observance. Even before the polling was complete, two things were already clear: there was a distinct age bias, and everyone, whether they were for or against Sunday travel, wanted their views to be recorded in the poll.

The response rate tells us a lot about the significance of the topic to those questioned. Death and bereavement are important matters, how satisfied you are with your toothpaste or toilet cleaner is different in nature and not to be compared.

One in 6, and 98% are both “normal”.

There’s a passage in the Iraq Survey Group report that would seem to support the high mortality figures of the recent Lancet study.

“In addition, there is significant underreporting of the violence in Iraq. The standard for recording attacks acts as a filter to keep events out of reports and databases. A murder of an Iraqi is not necessarily counted as an attack. If we cannot determine the source of a sectarian attack, that assault does not make it into the database. A roadside bomb or a rocket or mortar attack that doesn’t hurt U.S. personnel doesn’t count. For example, on one day in July 2006 there were 93 attacks or significant
acts of violence reported. Yet a careful review of the reports for that single day brought to light 1,100 acts of violence. Good policy is difficult to make when information is systematically collected in a way that minimizes its discrepancy with policy goals.” (p. 94-95)

And June was not, AFAIK, significantly more violent than other months. If there could be 1,100 “acts of violence” on one day in a ‘normal’ month, 600,000 dead over the course of three years does not seem implausible at all.

i think the iraq was is stupid and its not fair

Bear in mind that the survey conducted by ORB was multi-stage. The Lancet polls were conducted on a single attempt. There is a huge difference.