andrew_gelman_stats andrew_gelman_stats-2010 andrew_gelman_stats-2010-5 knowledge-graph by maker-knowledge-mining
Source: html
Introduction: Michael Spagat notifies me that his article criticizing the 2006 study of Burnham, Lafta, Doocy and Roberts has just been published . The Burnham et al. paper (also called, to my irritation (see the last item here ), “the Lancet survey”) used a cluster sample to estimate the number of deaths in Iraq in the three years following the 2003 invasion. In his newly-published paper, Spagat writes: [The Spagat article] presents some evidence suggesting ethical violations to the survey’s respondents including endangerment, privacy breaches and violations in obtaining informed consent. Breaches of minimal disclosure standards examined include non-disclosure of the survey’s questionnaire, data-entry form, data matching anonymised interviewer identifications with households and sample design. The paper also presents some evidence relating to data fabrication and falsification, which falls into nine broad categories. This evidence suggests that this survey cannot be considered a reliable or
sentIndex sentText sentNum sentScore
1 ” Also some contradictory descriptions of sampling methods, which are interesting enough that I will copy them here (it’s from pages 11-12 of Spagat’s article): The L2 authors [Burnham et al. [sent-24, score-0.323]
2 ] have often dismissed the possibility of sampling bias by stating that they did not actually follow the sampling procedures that they claimed to have followed in their Lancet publication. [sent-25, score-0.327]
3 (Burnham and Roberts, 2006a) Thus [according to Spagat], this part of the description of sampling methodology should have read: The third stage consisted of random selection of a main street within the administrative unit from a list of all main streets. [sent-27, score-0.847]
4 A residential street was then randomly selected from a list of residential streets crossing the main street. [sent-28, score-1.164]
5 As far as selection of the start houses, in areas where there were residential streets that did not cross the main avenues in the area selected, these were included in the random street selection process, in an effort to reduce the selection bias that more busy streets would have. [sent-29, score-2.191]
6 , 2008, with new text italicised) Combining this with Gilbert Burnham’s New Scientist interview already quoted (Biever, 2007) would imply that at each location: (1) Field teams wrote names of main streets on pieces of paper and selected one street at random. [sent-31, score-1.188]
7 (2) The field teams then walked down this street writing down names of cross streets on pieces of paper and selected one of these at random. [sent-32, score-1.245]
8 (3) The field teams then became aware of all other streets in the area that did not cross the main avenues and may have selected one of these instead of one of the cross streets written on pieces of paper. [sent-33, score-1.864]
9 If other types of streets, beyond those that would be covered by the published methodology, were included in the sampling procedures then the authors need to specify how these streets were included. [sent-36, score-0.75]
10 More fundamentally, how did the field teams discover the existence of such streets that could not be seen by walking down principal streets as described by Burnham in Biever (2007)? [sent-37, score-1.256]
11 The L2 field teams would not have brought detailed street maps with them into each selected area or else it would not have been necessary to walk down selected principal streets writing down names of surrounding streets on pieces of paper. [sent-38, score-1.947]
12 We can also rule out the possibility that the teams completely canvassed entire neighbourhoods and built up detailed street maps from scratch in each location. [sent-39, score-0.486]
13 Developing such detailed street maps would have been very time consuming and the L2 field teams had to follow an extremely compressed schedule that required them to perform 40 interviews in a day (Hicks, 2006). [sent-40, score-0.532]
14 Indeed, involving local residents in selecting the streets to be sampled would seem to be at odds with the random selection of households. [sent-44, score-0.71]
15 Locals could, for example, lead the survey teams to particularly violent areas. [sent-45, score-0.313]
16 The sites were selected entirely at random, so all households had an equal chance of being included. [sent-47, score-0.357]
17 we had an equal chance of picking a main street as a back street. [sent-54, score-0.414]
18 Some streets are much more densely populated than others. [sent-57, score-0.656]
19 If, for example, every street block had an equal chance of selection then households on densely populated street blocks would have lower selection probabilities than households on a sparsely populated street block. [sent-59, score-1.506]
20 If main streets are more densely populated on average than are back streets and main streets and back streets have equal selection probabilities then households on main streets would have lower selection probabilities than households on back streets. [sent-60, score-3.706]
wordName wordTfidf (topN-words)
[('streets', 0.513), ('burnham', 0.427), ('spagat', 0.229), ('street', 0.214), ('roberts', 0.189), ('selection', 0.149), ('selected', 0.143), ('sampling', 0.142), ('teams', 0.141), ('households', 0.132), ('main', 0.118), ('iraq', 0.11), ('survey', 0.098), ('field', 0.089), ('residential', 0.088), ('cross', 0.086), ('lancet', 0.085), ('biever', 0.083), ('ifhs', 0.083), ('iraqi', 0.083), ('equal', 0.082), ('populated', 0.077), ('locals', 0.076), ('et', 0.075), ('violent', 0.074), ('avenues', 0.072), ('deaths', 0.067), ('densely', 0.066), ('pieces', 0.059), ('contradictory', 0.059), ('methodology', 0.058), ('giles', 0.056), ('cluster', 0.051), ('doocy', 0.051), ('fabrication', 0.051), ('lafta', 0.051), ('lambert', 0.051), ('evidence', 0.048), ('included', 0.048), ('random', 0.048), ('breaches', 0.048), ('pockets', 0.048), ('authors', 0.047), ('gilbert', 0.046), ('invasion', 0.044), ('violations', 0.044), ('detailed', 0.044), ('maps', 0.044), ('area', 0.044), ('possibility', 0.043)]
simIndex simValue blogId blogTitle
same-blog 1 1.0 5 andrew gelman stats-2010-04-27-Ethical and data-integrity problems in a study of mortality in Iraq
Introduction: Michael Spagat notifies me that his article criticizing the 2006 study of Burnham, Lafta, Doocy and Roberts has just been published . The Burnham et al. paper (also called, to my irritation (see the last item here ), “the Lancet survey”) used a cluster sample to estimate the number of deaths in Iraq in the three years following the 2003 invasion. In his newly-published paper, Spagat writes: [The Spagat article] presents some evidence suggesting ethical violations to the survey’s respondents including endangerment, privacy breaches and violations in obtaining informed consent. Breaches of minimal disclosure standards examined include non-disclosure of the survey’s questionnaire, data-entry form, data matching anonymised interviewer identifications with households and sample design. The paper also presents some evidence relating to data fabrication and falsification, which falls into nine broad categories. This evidence suggests that this survey cannot be considered a reliable or
Introduction: Mike Spagat points to this interview , which, he writes, covers themes that are discussed on the blog such as wrong ideas that don’t die, peer review and the statistics of conflict deaths. I agree. It’s good stuff. Here are some of the things that Spagat says (he’s being interviewed by Joel Wing): In fact, the standard excess-deaths concept leads to an interesting conundrum when combined with an interesting fact exposed in the next-to-latest Human Security Report ; in most countries child mortality rates decline during armed conflict (chapter 6). So if you believe the usual excess-death causality story then you’re forced to conclude that many conflicts actually save the lives of many children. Of course, the idea of wars savings lives is pretty hard to swallow. A much more sensible understanding is that there are a variety of factors that determine child deaths and that in many cases the factors that save the lives of children are stronger than the negative effects that confli
3 0.23159829 1371 andrew gelman stats-2012-06-07-Question 28 of my final exam for Design and Analysis of Sample Surveys
Introduction: This is it, the last question on the exam! 28. A telephone survey was conducted several years ago, asking people how often they were polled in the past year. I can’t recall the responses, but suppose that 40% of the respondents said they participated in zero surveys in the previous year, 30% said they participated in one survey, 15% said two surveys, 10% said three, and 5% said four. From this it is easy to estimate an average, but there is a worry that this survey will itself overrepresent survey participants and thus overestimate the rate at which the average person is surveyed. Come up with a procedure to use these data to get an improved estimate of the average number of surveys that a randomly-sampled American is polled in a year. Solution to question 27 From yesterday : 27. Which of the following problems were identified with the Burnham et al. survey of Iraq mortality? (Indicate all that apply.) (a) The survey used cluster sampling, which is inappropriate for estim
4 0.1938352 1368 andrew gelman stats-2012-06-06-Question 27 of my final exam for Design and Analysis of Sample Surveys
Introduction: 27. Which of the following problems were identified with the Burnham et al. survey of Iraq mortality? (Indicate all that apply.) (a) The survey used cluster sampling, which is inappropriate for estimating individual out- comes such as death. (b) In their report, Burnham et al. did not identify their primary sampling units. (c) The second-stage sampling was not a probability sample. (d) Survey materials supplied by the authors are incomplete and inconsistent with published descriptions of the survey. Solution to question 26 From yesterday : 26. You have just graded an an exam with 28 questions and 15 students. You fit a logistic item- response model estimating ability, difficulty, and discrimination parameters. Which of the following statements are basically true? (Indicate all that apply.) (a) If a question is answered correctly by students with very low and very high ability, but is missed by students in the middle, it will have a high value for its discrimination
5 0.15289479 141 andrew gelman stats-2010-07-12-Dispute over counts of child deaths in Iraq due to sanctions
Introduction: Mike Spagat writes: Here is yet another debunking article I’ve written, this one in the latest issue of Significance. It shows the Lancet once again publishing spectacularly wrong information that has misinformed public discussion on a crucial issue with ongoing reverberations. For example, there is Tony Blair’s recent justification for the Iraq war offered in front of the Chilcot Inquiry in the UK quoted at the beginning of the paper. I haven’t had a chance to look at this one, but here’s a link to some related Spagat work.
7 0.13737956 107 andrew gelman stats-2010-06-24-PPS in Georgia
8 0.13349298 367 andrew gelman stats-2010-10-25-In today’s economy, the rich get richer
9 0.12650488 435 andrew gelman stats-2010-11-29-Panel Thurs 2 Dec on politics and deficit reduction in NYC
10 0.10799195 784 andrew gelman stats-2011-07-01-Weighting and prediction in sample surveys
11 0.10162979 1628 andrew gelman stats-2012-12-17-Statistics in a world where nothing is random
12 0.10140009 694 andrew gelman stats-2011-05-04-My talk at Hunter College on Thurs
13 0.094837815 1017 andrew gelman stats-2011-11-18-Lack of complete overlap
14 0.089896843 1455 andrew gelman stats-2012-08-12-Probabilistic screening to get an approximate self-weighted sample
15 0.088708073 749 andrew gelman stats-2011-06-06-“Sampling: Design and Analysis”: a course for political science graduate students
16 0.086198553 761 andrew gelman stats-2011-06-13-A survey’s not a survey if they don’t tell you how they did it
17 0.082939096 85 andrew gelman stats-2010-06-14-Prior distribution for design effects
18 0.082853794 2337 andrew gelman stats-2014-05-18-Never back down: The culture of poverty and the culture of journalism
19 0.078306533 1621 andrew gelman stats-2012-12-13-Puzzles of criminal justice
20 0.077852316 12 andrew gelman stats-2010-04-30-More on problems with surveys estimating deaths in war zones
topicId topicWeight
[(0, 0.156), (1, -0.019), (2, 0.037), (3, -0.074), (4, 0.03), (5, 0.039), (6, -0.026), (7, 0.005), (8, -0.02), (9, -0.055), (10, 0.043), (11, -0.083), (12, -0.025), (13, 0.072), (14, -0.023), (15, -0.012), (16, 0.041), (17, 0.017), (18, 0.033), (19, -0.014), (20, -0.031), (21, 0.004), (22, -0.042), (23, 0.028), (24, -0.004), (25, 0.035), (26, -0.012), (27, 0.048), (28, 0.055), (29, -0.004), (30, -0.008), (31, 0.018), (32, -0.044), (33, 0.014), (34, -0.091), (35, 0.007), (36, 0.027), (37, -0.004), (38, -0.018), (39, 0.049), (40, 0.002), (41, 0.028), (42, 0.051), (43, -0.021), (44, 0.007), (45, -0.006), (46, -0.01), (47, 0.016), (48, 0.02), (49, -0.004)]
simIndex simValue blogId blogTitle
same-blog 1 0.96337265 5 andrew gelman stats-2010-04-27-Ethical and data-integrity problems in a study of mortality in Iraq
Introduction: Michael Spagat notifies me that his article criticizing the 2006 study of Burnham, Lafta, Doocy and Roberts has just been published . The Burnham et al. paper (also called, to my irritation (see the last item here ), “the Lancet survey”) used a cluster sample to estimate the number of deaths in Iraq in the three years following the 2003 invasion. In his newly-published paper, Spagat writes: [The Spagat article] presents some evidence suggesting ethical violations to the survey’s respondents including endangerment, privacy breaches and violations in obtaining informed consent. Breaches of minimal disclosure standards examined include non-disclosure of the survey’s questionnaire, data-entry form, data matching anonymised interviewer identifications with households and sample design. The paper also presents some evidence relating to data fabrication and falsification, which falls into nine broad categories. This evidence suggests that this survey cannot be considered a reliable or
2 0.82468581 1371 andrew gelman stats-2012-06-07-Question 28 of my final exam for Design and Analysis of Sample Surveys
Introduction: This is it, the last question on the exam! 28. A telephone survey was conducted several years ago, asking people how often they were polled in the past year. I can’t recall the responses, but suppose that 40% of the respondents said they participated in zero surveys in the previous year, 30% said they participated in one survey, 15% said two surveys, 10% said three, and 5% said four. From this it is easy to estimate an average, but there is a worry that this survey will itself overrepresent survey participants and thus overestimate the rate at which the average person is surveyed. Come up with a procedure to use these data to get an improved estimate of the average number of surveys that a randomly-sampled American is polled in a year. Solution to question 27 From yesterday : 27. Which of the following problems were identified with the Burnham et al. survey of Iraq mortality? (Indicate all that apply.) (a) The survey used cluster sampling, which is inappropriate for estim
Introduction: A reporter emailed me the other day with a question about a case I’d never heard of before, a company called Herbalife that is being accused of being a pyramid scheme. The reporter pointed me to this document which describes a survey conducted by “a third party firm called Lieberman Research”: Two independent studies took place using real time (aka “river”) sampling, in which respondents were intercepted across a wide array of websites Sample size of 2,000 adults 18+ matched to U.S. census on age, gender, income, region and ethnicity “River sampling” in this case appears to mean, according to the reporter, that “people were invited into it through online ads.” The survey found that 5% of U.S. households had purchased Herbalife products during the past three months (with a “0.8% margin of error,” ha ha ha). They they did a multiplication and a division to estimate that only 8% of households who bought these products were Herbalife distributors: 480,000 active distributor
4 0.806355 385 andrew gelman stats-2010-10-31-Wacky surveys where they don’t tell you the questions they asked
Introduction: Maria Wolters writes: The parenting club Bounty, which distributes their packs through midwives, hospitals, and large UK supermarket and pharmacy chains, commissioned a fun little survey for Halloween from the company OnePoll . Theme: Mothers as tricksters – tricking men into fathering their babies. You can find a full smackdown courtesy of UK-based sex educator and University College London psychologist Petra Boynton here . (One does wonder how a parenting club with such close links to the UK National Health Service thought a survey on this topic was at all appropriate, but that’s another rant.) So far, so awful, but what I [Wolters] thought might grab your attention was the excuse OnePoll offered for their work in their email to Petra. (Petra is very well known in the UK, and so was able to get a statement from the polling company.) Here it is in its full glory, taken from Petra’s post: As the agency which commissioned this research and distributed the resulting new
5 0.79707366 784 andrew gelman stats-2011-07-01-Weighting and prediction in sample surveys
Introduction: A couple years ago Rod Little was invited to write an article for the diamond jubilee of the Calcutta Statistical Association Bulletin. His article was published with discussions from Danny Pfefferman, J. N. K. Rao, Don Rubin, and myself. Here it all is . I’ll paste my discussion below, but it’s worth reading the others’ perspectives too. Especially the part in Rod’s rejoinder where he points out a mistake I made. Survey weights, like sausage and legislation, are designed and best appreciated by those who are placed a respectable distance from their manufacture. For those of us working inside the factory, vigorous discussion of methods is appreciated. I enjoyed Rod Little’s review of the connections between modeling and survey weighting and have just a few comments. I like Little’s discussion of model-based shrinkage of post-stratum averages, which, as he notes, can be seen to correspond to shrinkage of weights. I would only add one thing to his formula at the end of his
6 0.7871682 405 andrew gelman stats-2010-11-10-Estimation from an out-of-date census
7 0.78291154 1430 andrew gelman stats-2012-07-26-Some thoughts on survey weighting
8 0.76787019 107 andrew gelman stats-2010-06-24-PPS in Georgia
9 0.76591313 1288 andrew gelman stats-2012-04-29-Clueless Americans think they’ll never get sick
10 0.76527369 1455 andrew gelman stats-2012-08-12-Probabilistic screening to get an approximate self-weighted sample
12 0.7559492 730 andrew gelman stats-2011-05-25-Rechecking the census
13 0.74886703 1940 andrew gelman stats-2013-07-16-A poll that throws away data???
14 0.73622674 705 andrew gelman stats-2011-05-10-Some interesting unpublished ideas on survey weighting
15 0.73126453 761 andrew gelman stats-2011-06-13-A survey’s not a survey if they don’t tell you how they did it
16 0.72468287 142 andrew gelman stats-2010-07-12-God, Guns, and Gaydar: The Laws of Probability Push You to Overestimate Small Groups
17 0.71012372 1551 andrew gelman stats-2012-10-28-A convenience sample and selected treatments
18 0.69728434 1437 andrew gelman stats-2012-07-31-Paying survey respondents
19 0.69164658 2167 andrew gelman stats-2014-01-10-Do you believe that “humans and other living things have evolved over time”?
20 0.67887861 1320 andrew gelman stats-2012-05-14-Question 4 of my final exam for Design and Analysis of Sample Surveys
topicId topicWeight
[(2, 0.049), (9, 0.051), (15, 0.027), (16, 0.065), (21, 0.024), (24, 0.107), (26, 0.027), (37, 0.166), (53, 0.011), (86, 0.027), (95, 0.029), (99, 0.218)]
simIndex simValue blogId blogTitle
1 0.92770398 1601 andrew gelman stats-2012-12-01-A lifetime supply of . . .
Introduction: This story reminded me of when I was planning my short course for Procter & Gamble: The question in my mind was: should I take the consulting fee they offered or should I try to get a voucher for a lifetime supply of P&G; products (assuming, of course, that such a voucher exists)? Wouldn’t that just be the coolest thing, to go into the supermarket, fill up the cart with Tide and Pampers, then flash the P&G; card? P.S. On an unrelated note, this is just sad. Uncle Woody is spinning.
same-blog 2 0.92129678 5 andrew gelman stats-2010-04-27-Ethical and data-integrity problems in a study of mortality in Iraq
Introduction: Michael Spagat notifies me that his article criticizing the 2006 study of Burnham, Lafta, Doocy and Roberts has just been published . The Burnham et al. paper (also called, to my irritation (see the last item here ), “the Lancet survey”) used a cluster sample to estimate the number of deaths in Iraq in the three years following the 2003 invasion. In his newly-published paper, Spagat writes: [The Spagat article] presents some evidence suggesting ethical violations to the survey’s respondents including endangerment, privacy breaches and violations in obtaining informed consent. Breaches of minimal disclosure standards examined include non-disclosure of the survey’s questionnaire, data-entry form, data matching anonymised interviewer identifications with households and sample design. The paper also presents some evidence relating to data fabrication and falsification, which falls into nine broad categories. This evidence suggests that this survey cannot be considered a reliable or
Introduction: Rajiv Sethi writes the above in a discussion of a misunderstanding of the economics of Keynes. The discussion is interesting. According to Sethi, Keynes wrote that, in a depression, nominal wages might be sticky but in any case a decline in wages would not do the trick to increase hiring. But many modern economics writers have missed this. For example, Gary Becker writes, “Keynes and many earlier economists emphasized that unemployment rises during recessions because nominal wage rates tend to be inflexible in the downward direction. . . . A fall in price stimulates demand and reduces supply until they are brought back to rough equality.” Whether Becker is empirically correct is another story, but in any case he is misinterpreting Keynes. But the actual reason I’m posting here is in reaction to Sethi’s remark quoted in the title above, in which he endorses a 1975 paper by James Tobin on wages and employment but remarks that Tobin’s paper did not include the individual-level de
4 0.87708956 1645 andrew gelman stats-2012-12-31-Statistical modeling, causal inference, and social science
Introduction: Interesting discussion by Berk Ozler (which I found following links from Tyler Cowen) of a study by Erwin Bulte, Lei Pan, Joseph Hella, Gonne Beekman, and Salvatore di Falco that compares two agricultural experiments, one blinded and one unblinded. Bulte et al. find much different results in the two experiments and attribute the difference to expectation effects (when people know they’re receiving an experiment they behave differently); Ozler is skeptical and attributes the different outcomes to various practical differences in implementation of the two experiments. I’m reminded somehow of the notorious sham experiment on the dead chickens, a story that was good for endless discussion in my Bayesian statistics class last semester. I think we can all agree that dead chickens won’t exhibit a placebo effect. Live farmers, though, that’s another story. I don’t have any stake in this particular fight, but on quick reading I’m sympathetic to Ozler’s argument that this all is wel
Introduction: Conservative data cruncher Charles Murray asks , “Why aren’t Asians Republicans?”: Asians are only half as likely to identify themselves as “conservative” or “very conservative” as whites, and less than half as likely to identify themselves as Republicans. . . . 70% of Asians voted for Barack Obama in the last presidential election. Something’s wrong with this picture. . . . Everyday observation of Asians around the world reveal them to be conspicuously entrepreneurial, industrious, family-oriented, and self-reliant. If you’re looking for a natural Republican constituency, Asians should define “natural.” . . . Asian immigrants overwhelmingly succeeded, another experience that tends to produce conservative immigrants. Beyond that, Asian minorities everywhere in the world, including America, tend to be underrepresented in politics—they’re more interested in getting ahead commercially or in non-political professions than in running for office or organizing advocacy groups. La
6 0.86346376 157 andrew gelman stats-2010-07-21-Roller coasters, charity, profit, hmmm
7 0.86102474 81 andrew gelman stats-2010-06-12-Reputational Capital and Incentives in Organizations
8 0.84719455 1774 andrew gelman stats-2013-03-22-Likelihood Ratio ≠ 1 Journal
9 0.83819747 1823 andrew gelman stats-2013-04-24-The Tweets-Votes Curve
10 0.83274221 2195 andrew gelman stats-2014-02-02-Microfoundations of macroeconomics
11 0.83118248 1305 andrew gelman stats-2012-05-07-Happy news on happiness; what can we believe?
12 0.83115476 1371 andrew gelman stats-2012-06-07-Question 28 of my final exam for Design and Analysis of Sample Surveys
13 0.8310886 1231 andrew gelman stats-2012-03-27-Attention pollution
14 0.82853377 1254 andrew gelman stats-2012-04-09-In the future, everyone will publish everything.
16 0.82395172 1171 andrew gelman stats-2012-02-16-“False-positive psychology”
18 0.82123154 1280 andrew gelman stats-2012-04-24-Non-Bayesian analysis of Bayesian agents?
19 0.82010198 2352 andrew gelman stats-2014-05-29-When you believe in things that you don’t understand
20 0.81833518 675 andrew gelman stats-2011-04-22-Arrow’s other theorem