andrew_gelman_stats andrew_gelman_stats-2014 andrew_gelman_stats-2014-2303 knowledge-graph by maker-knowledge-mining
Source: html
Introduction: Someone wrote in: We are about to conduct a voting list experiment. We came across your comment recommending that each item be removed from the list. Would greatly appreciate it if you take a few minutes to spell out your recommendation in a little more detail. In particular: (a) Why are you “uneasy” about list experiments? What would strengthen your confidence in list experiments? (b) What do you mean by “each item be removed”? As you know, there are several non-sensitive items and one sensitive item in a list experiment. Do you mean that the non-sensitive items should be removed one-by-one for the control group or are you suggesting a multiple arm design in which each arm of the experiment has one non-sensitive item removed. What would be achieved by this design? I replied: I’ve always been a bit skeptical about list experiments, partly because I worry that the absolute number of items on the list could itself affect the response. For example, someone might not want to che
sentIndex sentText sentNum sentScore
1 As you know, there are several non-sensitive items and one sensitive item in a list experiment. [sent-7, score-1.353]
2 Do you mean that the non-sensitive items should be removed one-by-one for the control group or are you suggesting a multiple arm design in which each arm of the experiment has one non-sensitive item removed. [sent-8, score-0.976]
3 I replied: I’ve always been a bit skeptical about list experiments, partly because I worry that the absolute number of items on the list could itself affect the response. [sent-10, score-1.188]
4 For example, someone might not want to check off 6 items out of 6 but would have no problem checking off 6 items out of 10: even if 4 items on that latter list were complete crap, their presence on the list might make the original 6 items look better by comparison. [sent-11, score-2.229]
5 So this has made me think that a list experiment should really have some sort of active control. [sent-12, score-0.597]
6 Here’s what they said: Macartan Humphreys: I have had mixed experiences with list experiments. [sent-17, score-0.494]
7 Enumerators are sometimes confused by them and so are subjects and sometimes we have found enumerators implementing them badly, eg sometimes getting the subjects to count out as they go along reading the list that kind of thing. [sent-18, score-0.798]
8 In one implementation that we thought went quite well we cleverly did two list experiments with the same sensitive item and different nonsensitive items, but got very different results. [sent-20, score-1.338]
9 The length of list issue I think is not the biggest. [sent-22, score-0.475]
10 pdf A bigger issue though is that list experiments don’t incentivize people to give you information that they don’t want you to have. [sent-27, score-0.787]
11 eg if people do not want you to know that there was fraud, and if they understand the list experiment, you should not get evidence of fraud. [sent-28, score-0.612]
12 Lynn Vavreck: Simon Jackman and I ran a number of list experiments in the 2008 Cooperative Campaign Analysis Project. [sent-30, score-0.672]
13 I will let you make of those patterns what you will — but, it seemed to us to echo what Macartan writes below — if it’s truly a sensitive item, people seem to figure out what is going on and won’t comply with the “treatment. [sent-39, score-0.494]
14 If they “figure out” what you’re up to in the list experiment they may be less likely to give you honest answers to other questions down the line. [sent-45, score-0.544]
15 There’s also the work that Jason Lyall and coauthors have done using both list experiments and endorsement experiments in Afghanistan. [sent-52, score-0.828]
16 This gives rise to the “top-coding” problem: if all J items are things I’ve done, including the sensitive item, then I’m going to respond “J” only if I’m ok revealing myself as someone who would respond “yes” to the sensitive item. [sent-62, score-1.172]
17 Then you’ve got to figure out how to have J items, including your sensitive item, such that J-1 might be the plausible upper bound on the item count. [sent-63, score-0.816]
18 Sure, modeling (a la Imai) can get you estimates of the correlates of the sensitive item and stratification lets you estimate rates in sub-populations. [sent-68, score-0.634]
19 And Brendan Nyhan adds: I suspect there’s a significant file drawer problem on list experiments. [sent-74, score-0.489]
20 They have low power and are highly sensitive to design quirks and respondent compliance as others mentioned. [sent-76, score-0.509]
wordName wordTfidf (topN-words)
[('list', 0.418), ('sensitive', 0.342), ('items', 0.301), ('item', 0.292), ('experiments', 0.205), ('macartan', 0.161), ('enumerators', 0.152), ('mccain', 0.143), ('lynn', 0.131), ('experiment', 0.126), ('lists', 0.12), ('respondent', 0.111), ('removed', 0.093), ('technique', 0.09), ('obama', 0.089), ('jackman', 0.087), ('desirability', 0.083), ('got', 0.081), ('experiences', 0.076), ('eg', 0.076), ('problem', 0.071), ('jason', 0.07), ('respond', 0.068), ('simon', 0.067), ('want', 0.067), ('clinton', 0.062), ('length', 0.057), ('design', 0.056), ('arm', 0.054), ('active', 0.053), ('figure', 0.053), ('validity', 0.052), ('kind', 0.052), ('would', 0.051), ('skeptical', 0.051), ('people', 0.051), ('subjects', 0.05), ('ran', 0.049), ('survey', 0.049), ('seem', 0.048), ('plausible', 0.048), ('ve', 0.047), ('uneasy', 0.046), ('attentive', 0.046), ('cheerleading', 0.046), ('sacrificing', 0.046), ('cooperative', 0.046), ('incentivize', 0.046), ('tingley', 0.046), ('reconsider', 0.046)]
simIndex simValue blogId blogTitle
same-blog 1 1.0000005 2303 andrew gelman stats-2014-04-23-Thinking of doing a list experiment? Here’s a list of reasons why you should think again
Introduction: Someone wrote in: We are about to conduct a voting list experiment. We came across your comment recommending that each item be removed from the list. Would greatly appreciate it if you take a few minutes to spell out your recommendation in a little more detail. In particular: (a) Why are you “uneasy” about list experiments? What would strengthen your confidence in list experiments? (b) What do you mean by “each item be removed”? As you know, there are several non-sensitive items and one sensitive item in a list experiment. Do you mean that the non-sensitive items should be removed one-by-one for the control group or are you suggesting a multiple arm design in which each arm of the experiment has one non-sensitive item removed. What would be achieved by this design? I replied: I’ve always been a bit skeptical about list experiments, partly because I worry that the absolute number of items on the list could itself affect the response. For example, someone might not want to che
2 0.18419106 577 andrew gelman stats-2011-02-16-Annals of really really stupid spam
Introduction: This came in the inbox today: Dear Dr. Gelman, GenWay recently found your article titled “Multiple imputation for model checking: completed-data plots with missing and latent data.” (Biometrics. 2005 Mar;61(1):74-85.) and thought you might be interested in learning about our superior quality signaling proteins. GenWay prides itself on being a leader in customer service aiming to exceed your expectations with the quality and price of our products. With more than 60,000 reagents backed by our outstanding guarantee you are sure to find the products you have been searching for. Please feel free to visit the following resource pages: * Apoptosis Pathway (product list) * Adipocytokine (product list) * Cell Cycle Pathway (product list) * Jak STAT (product list) * GnRH (product list) * MAPK (product list) * mTOR (product list) * T Cell Receptor (product list) * TGF-beta (product list) * Wnt (product list) * View All Pathways
Introduction: Leading theoretical statistician Larry Wassserman in 2008 : Some of the greatest contributions of statistics to science involve adding additional randomness and leveraging that randomness. Examples are randomized experiments, permutation tests, cross-validation and data-splitting. These are unabashedly frequentist ideas and, while one can strain to fit them into a Bayesian framework, they don’t really have a place in Bayesian inference. The fact that Bayesian methods do not naturally accommodate such a powerful set of statistical ideas seems like a serious deficiency. To which I responded on the second-to-last paragraph of page 8 here . Larry Wasserman in 2013 : Some people say that there is no role for randomization in Bayesian inference. In other words, the randomization mechanism plays no role in Bayes’ theorem. But this is not really true. Without randomization, we can indeed derive a posterior for theta but it is highly sensitive to the prior. This is just a restat
4 0.14962292 1951 andrew gelman stats-2013-07-22-Top 5 stat papers since 2000?
Introduction: Jeff Leek writes: I just wrote this post about what the 5 most influential papers in statistics from 2000-2010. I would be really curious to know your list too? Scarily enough I can’t think of any truly influential papers from that decade. I suppose this means I’m getting old! P.S. I did once make a list of the top 5 unpublished papers in statistics .
5 0.13313483 1146 andrew gelman stats-2012-01-30-Convenient page of data sources from the Washington Post
Introduction: Wayne Folta points us to this list .
6 0.12417171 311 andrew gelman stats-2010-10-02-Where do our taxes go?
8 0.11889687 1070 andrew gelman stats-2011-12-19-The scope for snooping
9 0.11814687 1933 andrew gelman stats-2013-07-10-Please send all comments to -dev-ripley
10 0.11808459 2035 andrew gelman stats-2013-09-23-Scalable Stan
11 0.11793885 2255 andrew gelman stats-2014-03-19-How Americans vote
12 0.11787011 1675 andrew gelman stats-2013-01-15-“10 Things You Need to Know About Causal Effects”
13 0.11566499 2011 andrew gelman stats-2013-09-07-Here’s what happened when I finished my PhD thesis
14 0.11477687 283 andrew gelman stats-2010-09-17-Vote Buying: Evidence from a List Experiment in Lebanon
15 0.11411338 481 andrew gelman stats-2010-12-22-The Jumpstart financial literacy survey and the different purposes of tests
16 0.10860369 25 andrew gelman stats-2010-05-10-Two great tastes that taste great together
17 0.10606062 888 andrew gelman stats-2011-09-03-A psychology researcher asks: Is Anova dead?
18 0.10440853 104 andrew gelman stats-2010-06-22-Seeking balance
19 0.1023246 1574 andrew gelman stats-2012-11-12-How to Lie With Statistics example number 12,498,122
20 0.10180227 1807 andrew gelman stats-2013-04-17-Data problems, coding errors…what can be done?
topicId topicWeight
[(0, 0.215), (1, -0.053), (2, 0.044), (3, -0.025), (4, 0.028), (5, 0.022), (6, 0.02), (7, -0.047), (8, 0.022), (9, -0.031), (10, -0.011), (11, -0.017), (12, 0.031), (13, -0.009), (14, 0.003), (15, 0.013), (16, -0.014), (17, -0.022), (18, -0.027), (19, 0.037), (20, 0.015), (21, -0.009), (22, 0.026), (23, 0.035), (24, -0.03), (25, -0.016), (26, 0.022), (27, -0.014), (28, -0.012), (29, 0.02), (30, 0.004), (31, -0.032), (32, -0.0), (33, 0.051), (34, -0.023), (35, -0.04), (36, 0.012), (37, 0.031), (38, -0.027), (39, 0.013), (40, 0.025), (41, -0.03), (42, 0.051), (43, 0.028), (44, -0.035), (45, -0.023), (46, -0.042), (47, 0.002), (48, -0.046), (49, 0.03)]
simIndex simValue blogId blogTitle
same-blog 1 0.97109193 2303 andrew gelman stats-2014-04-23-Thinking of doing a list experiment? Here’s a list of reasons why you should think again
Introduction: Someone wrote in: We are about to conduct a voting list experiment. We came across your comment recommending that each item be removed from the list. Would greatly appreciate it if you take a few minutes to spell out your recommendation in a little more detail. In particular: (a) Why are you “uneasy” about list experiments? What would strengthen your confidence in list experiments? (b) What do you mean by “each item be removed”? As you know, there are several non-sensitive items and one sensitive item in a list experiment. Do you mean that the non-sensitive items should be removed one-by-one for the control group or are you suggesting a multiple arm design in which each arm of the experiment has one non-sensitive item removed. What would be achieved by this design? I replied: I’ve always been a bit skeptical about list experiments, partly because I worry that the absolute number of items on the list could itself affect the response. For example, someone might not want to che
2 0.79453617 1933 andrew gelman stats-2013-07-10-Please send all comments to -dev-ripley
Introduction: Trey Causey asks , Has R-help gotten meaner over time?: I began by using Scrapy to download all the e-mails sent to R-help between April 1997 (the earliest available archive) and December 2012. . . . We each read 500 messages and coded them in the following categories: -2 Negative and unhelpful -1 Negative but helpful 0 No obviously valence or request for additional information 1 Positive or helpful 2 Not a response An example of a response coded -2 would be responses that do not answer the question, along with simply telling the user to RTFM, that they have violated the posting guidelines, or offer “?lm” as the only text when the question is about lm(). . . . Proportions of emails in each category in the test set were estimated on a monthly basis. Much to my surprise, R-help appears to be getting less mean over time! The proportion of “negative and unhelpful” messages has fallen steadily over time, from a high of 0.20 in October of 1997 to a low of 0.015 in January
3 0.78549612 2158 andrew gelman stats-2014-01-03-Booze: Been There. Done That.
Introduction: Our research assistants have unearthed the following guest column by H. L. Mencken which appeared in the New York Times of 5 Nov 1933, the date at which Prohibition ended in the United States. As a public service we are reprinting it here. I’m particularly impressed at how the Sage of Baltimore buttressed his article with references to the latest scientific literature of the time. I think you’ll all agree that Mencken’s column, in which he took a stand against the legality of alcohol consumption, has contemporary relevance , more than 80 years later. Because of the challenge of interpreting decades-old references, we have asked a leading scholar of Mencken’s writings to add notes where appropriate, to clarify any points of confusion. And now here’s Mencken’s column (with notes added in brackets), in its entirety: For a little while in my teenage years, my friends and I drank alcohol. It was fun. I have some fond memories of us all being silly together. I think those moments of
4 0.77665687 777 andrew gelman stats-2011-06-23-Combining survey data obtained using different modes of sampling
Introduction: I’m involved (with Irv Garfinkel and others) in a planned survey of New York City residents. It’s hard to reach people in the city–not everyone will answer their mail or phone, and you can’t send an interviewer door-to-door in a locked apartment building. (I think it violates IRB to have a plan of pushing all the buzzers by the entrance and hoping someone will let you in.) So the plan is to use multiple modes, including phone, in person household, random street intercepts and mail. The question then is how to combine these samples. My suggested approach is to divide the population into poststrata based on various factors (age, ethnicity, family type, housing type, etc), then to pool responses within each poststratum, then to runs some regressions including postratsta and also indicators for mode, to understand how respondents from different modes differ, after controlling for the demographic/geographic adjustments. Maybe this has already been done and written up somewhere? P.
5 0.77074367 1717 andrew gelman stats-2013-02-10-Psychology can be improved by adding some economics
Introduction: On this blog I’ve occasionally written about the problems that arise when economists act as amateur psychologists. But the problem can go the other way, too. For example, consider this blog by Berit Brogaard and Kristian Marlow ( link from Abbas Raza). Brogaard and Marlow give several amusing stories about ripoffs (a restaurant that scams customers into buying expensive bottles of wine, a hairdresser that sucks customers into unnecessary treatments, a ghostwriter who takes thousands of dollars in payments and doesn’t do the job, etc.). Then they ask, “How did it happen? Why did you act in this impulsive way? Why didn’t you learn your lesson the first time around? Do you have some kind of brain damage?” They continue with some discussion of the ventromedial prefrontal cortex, the anterior insula, etc etc etc., and then conclude with the following advice: Is there anything we can do to avoid these moments of crazy decision-making? Yes but only by intentionally turning on our
6 0.76910466 1191 andrew gelman stats-2012-03-01-Hoe noem je?
7 0.76518822 969 andrew gelman stats-2011-10-22-Researching the cost-effectiveness of political lobbying organisations
8 0.76269376 2080 andrew gelman stats-2013-10-28-Writing for free
9 0.75945467 458 andrew gelman stats-2010-12-08-Blogging: Is it “fair use”?
10 0.75357026 668 andrew gelman stats-2011-04-19-The free cup and the extra dollar: A speculation in philosophy
11 0.75264174 1707 andrew gelman stats-2013-02-05-Glenn Hubbard and I were on opposite sides of a court case and I didn’t even know it!
12 0.75008523 889 andrew gelman stats-2011-09-04-The acupuncture paradox
13 0.74580586 1882 andrew gelman stats-2013-06-03-The statistical properties of smart chains (and referral chains more generally)
14 0.74379164 418 andrew gelman stats-2010-11-17-ff
15 0.74097008 1410 andrew gelman stats-2012-07-09-Experimental work on market-based or non-market-based incentives
16 0.73585773 2172 andrew gelman stats-2014-01-14-Advice on writing research articles
17 0.7356953 1797 andrew gelman stats-2013-04-10-“Proposition and experiment”
18 0.73484218 693 andrew gelman stats-2011-05-04-Don’t any statisticians work for the IRS?
19 0.73157829 219 andrew gelman stats-2010-08-20-Some things are just really hard to believe: more on choosing your facts.
topicId topicWeight
[(6, 0.024), (9, 0.067), (16, 0.1), (19, 0.01), (21, 0.023), (24, 0.134), (31, 0.012), (36, 0.047), (38, 0.015), (43, 0.014), (48, 0.017), (66, 0.014), (68, 0.019), (76, 0.039), (81, 0.015), (86, 0.017), (95, 0.044), (99, 0.273)]
simIndex simValue blogId blogTitle
same-blog 1 0.96450633 2303 andrew gelman stats-2014-04-23-Thinking of doing a list experiment? Here’s a list of reasons why you should think again
Introduction: Someone wrote in: We are about to conduct a voting list experiment. We came across your comment recommending that each item be removed from the list. Would greatly appreciate it if you take a few minutes to spell out your recommendation in a little more detail. In particular: (a) Why are you “uneasy” about list experiments? What would strengthen your confidence in list experiments? (b) What do you mean by “each item be removed”? As you know, there are several non-sensitive items and one sensitive item in a list experiment. Do you mean that the non-sensitive items should be removed one-by-one for the control group or are you suggesting a multiple arm design in which each arm of the experiment has one non-sensitive item removed. What would be achieved by this design? I replied: I’ve always been a bit skeptical about list experiments, partly because I worry that the absolute number of items on the list could itself affect the response. For example, someone might not want to che
2 0.95554465 675 andrew gelman stats-2011-04-22-Arrow’s other theorem
Introduction: I received the following email from someone who’d like to remain anonymous: Lately I [the anonymous correspondent] witnessed that Bruno Frey has published two articles in two well known referreed journals on the Titanic disaster that try to explain survival rates of passenger on board. The articles were published in the Journal of Economic Perspectives and Rationality & Society . While looking up the name of the second journal where I stumbled across the article I even saw that they put the message in a third journal, the Proceedings of the National Academy of Sciences United States of America . To say it in Sopranos like style – with all due respect, I know Bruno Frey from conferences, I really appreciate his take on economics as a social science and he has really published more interesting stuff that most economists ever will. But putting the same message into three journals gives me headaches for at least two reasons: 1) When building a track record and scientific rep
3 0.95469213 32 andrew gelman stats-2010-05-14-Causal inference in economics
Introduction: Aaron Edlin points me to this issue of the Journal of Economic Perspectives that focuses on statistical methods for causal inference in economics. (Michael Bishop’s page provides some links .) To quickly summarize my reactions to Angrist and Pischke’s book: I pretty much agree with them that the potential-outcomes or natural-experiment approach is the most useful way to think about causality in economics and related fields. My main amendments to Angrist and Pischke would be to recognize that: 1. Modeling is important, especially modeling of interactions . It’s unfortunate to see a debate between experimentalists and modelers. Some experimenters (not Angrist and Pischke) make the mistake of avoiding models: Once they have their experimental data, they check their brains at the door and do nothing but simple differences, not realizing how much more can be learned. Conversely, some modelers are unduly dismissive of experiments and formal observational studies, forgetting t
Introduction: In response to my most recent post expressing bafflement over the Erving Goffman mystique, several commenters helped out by suggesting classic Goffman articles for me to read. Naturally, I followed the reference that had a link attached–it was for an article called Cooling the Mark Out, which analogized the frustrations of laid-off and set-aside white-collar workers to the reactions to suckers after being bilked by con artists. Goffman’s article was fascinating, but I was bothered by a tone of smugness. Here’s a quote from Cooling the Mark Out that starts on the cute side but is basically ok: In organizations patterned after a bureaucratic model, it is customary for personnel to expect rewards of a specified kind upon fulfilling requirements of a specified nature. Personnel come to define their career line in terms of a sequence of legitimate expectations and to base their self-conceptions on the assumption that in due course they will be what the institution allows persons t
5 0.94964004 1226 andrew gelman stats-2012-03-22-Story time meets the all-else-equal fallacy and the fallacy of measurement
Introduction: Alex Tabarrok with a good catch : In Why Don’t Women Patent? , a recent NBER paper, Jennifer Hunt et al. [Jean-Philippe Garant, Hannah Herman, and David Munroe] present a stark fact: Only 5.5% of the holders of commercialized patents are women. One might think that this is explained by the relative lack of women with science and engineering degrees but Hunt et al. find that “women with such a degree are scarcely more likely to patent than women without.” Instead, most of the difference is “accounted for by differences among those with a science or engineering degree” especially the fact that women are underrepresented in patent-intensive fields such as electrical and mechanical engineering and in development and design. Predictably, the authors do not ask why women might self-selection into non patent-intensive fields, perhaps because this would require at least a discussion of politically incorrect questions . The failure to investigate these questions leads to some dubious co
6 0.94881141 560 andrew gelman stats-2011-02-06-Education and Poverty
7 0.94565558 1422 andrew gelman stats-2012-07-20-Likelihood thresholds and decisions
8 0.94560283 1715 andrew gelman stats-2013-02-09-Thomas Hobbes would be spinning in his grave
9 0.94492996 1110 andrew gelman stats-2012-01-10-Jobs in statistics research! In New Jersey!
10 0.94388974 2137 andrew gelman stats-2013-12-17-Replication backlash
11 0.94356734 571 andrew gelman stats-2011-02-13-A departmental wiki page?
12 0.94339836 599 andrew gelman stats-2011-03-03-Two interesting posts elsewhere on graphics
13 0.94277203 18 andrew gelman stats-2010-05-06-$63,000 worth of abusive research . . . or just a really stupid waste of time?
14 0.94265109 1371 andrew gelman stats-2012-06-07-Question 28 of my final exam for Design and Analysis of Sample Surveys
15 0.94242764 982 andrew gelman stats-2011-10-30-“There’s at least as much as an 80 percent chance . . .”
16 0.94241726 288 andrew gelman stats-2010-09-21-Discussion of the paper by Girolami and Calderhead on Bayesian computation
17 0.94222283 1910 andrew gelman stats-2013-06-22-Struggles over the criticism of the “cannabis users and IQ change” paper
18 0.94183016 586 andrew gelman stats-2011-02-23-A statistical version of Arrow’s paradox
19 0.94137788 2288 andrew gelman stats-2014-04-10-Small multiples of lineplots > maps (ok, not always, but yes in this case)