andrew_gelman_stats andrew_gelman_stats-2011 andrew_gelman_stats-2011-777 knowledge-graph by maker-knowledge-mining
Source: html
Introduction: I’m involved (with Irv Garfinkel and others) in a planned survey of New York City residents. It’s hard to reach people in the city–not everyone will answer their mail or phone, and you can’t send an interviewer door-to-door in a locked apartment building. (I think it violates IRB to have a plan of pushing all the buzzers by the entrance and hoping someone will let you in.) So the plan is to use multiple modes, including phone, in person household, random street intercepts and mail. The question then is how to combine these samples. My suggested approach is to divide the population into poststrata based on various factors (age, ethnicity, family type, housing type, etc), then to pool responses within each poststratum, then to runs some regressions including postratsta and also indicators for mode, to understand how respondents from different modes differ, after controlling for the demographic/geographic adjustments. Maybe this has already been done and written up somewhere? P.
sentIndex sentText sentNum sentScore
1 You could proceed efficiently by first applying mode A to the sample, and then applying mode B to those who did not respond with mode A. [sent-20, score-1.479]
2 At the end, you would have outcomes for types I, II, and III units, and you’d have an estimate of the rate of type IV units in the population. [sent-21, score-0.817]
3 You could content yourself with an estimate for the average response on the type I, II, and III subpopulation. [sent-22, score-0.736]
4 If you wanted to recover an estimate of the average response for the full population (including type IV’s), you would effectively have to impute values for type IV respondents. [sent-23, score-1.259]
5 This could be done by using auxiliary information either to genuinely impute or (in a manner that is pretty much equivalent) to determine which type I, II, or III units resemble the missing type IV units, and up-weight. [sent-24, score-1.017]
6 In any case, if the response of interest has finite support, one could also compute “worst case” (Manski-type) bounds on the average response by imputing maximum and minimum values to type IV units. [sent-25, score-1.229]
7 Mode of contact affects response This might be relevant if, for example, the modes of contact are phone call versus face-to-face interview, and outcomes being measured vary depending on whether the respondent feels more or less exposed in the interview situation. [sent-26, score-1.287]
8 In this case, each unit is characterized by a response under mode A and another under mode B (that is, two potential outcomes). [sent-28, score-1.453]
9 A design that applied both mode A and mode B to the complete sample would mechanically reveal the proportion of type I units in the population, and by implication would identify the proportion of type II, III, and IV units. [sent-36, score-2.042]
10 For type II units we could use mode A responses to improve imputations for mode B responses, and vice versa for type III respondents. [sent-37, score-2.005]
11 Again, one could construct worst case bounds by imputing maximum and minimum response values for each of the missing response types. [sent-39, score-0.81]
12 One wrinkle that I ignored above was that the order of modes of contact may affect either response behavior or outcomes reported. [sent-40, score-0.933]
13 This multiplies the number potential response behaviors and the number of potential outcome responses given that the unit is interviewed. [sent-41, score-0.681]
14 Andy’s proposal to use post-stratification and regressions relies (according to my understanding) on the assumption potential outcomes are independent of mode of contact conditional on covariates. [sent-51, score-0.995]
15 Formally, if the mode of contact is taking on values or , potential outcomes under mode of contact is , is principal stratum, and is a covariate, then implies that, . [sent-52, score-1.569]
16 As discussed above, the design that applies modes A and B to all units in the sample can determine principal stratum membership, and so these covariate- and principal-stratum specific imputations can be applied. [sent-53, score-0.805]
17 A worthwhile type of analysis would be to study evidence of mode-of-contact as well as ordering effects among the type I (always-responder) units. [sent-55, score-0.724]
18 Now, it may be that mode of contact affects response but units are contacted via either mode A or B. [sent-56, score-0.859]
19 Response monotonicity would mean that either type II or type III responders didn’t exist. [sent-59, score-0.853]
20 The common one would be that principal stratum membership is independent of potential responses conditional on covariates. [sent-61, score-0.654]
wordName wordTfidf (topN-words)
[('mode', 0.479), ('type', 0.308), ('contact', 0.258), ('response', 0.255), ('iv', 0.239), ('modes', 0.227), ('units', 0.191), ('iii', 0.189), ('ii', 0.162), ('principal', 0.149), ('responses', 0.135), ('stratum', 0.126), ('types', 0.114), ('outcomes', 0.103), ('potential', 0.101), ('unit', 0.089), ('membership', 0.087), ('average', 0.086), ('population', 0.082), ('affects', 0.074), ('responders', 0.068), ('finite', 0.066), ('phone', 0.066), ('monotonicity', 0.065), ('imputations', 0.063), ('auxiliary', 0.061), ('values', 0.06), ('impose', 0.059), ('impute', 0.059), ('identifiable', 0.058), ('strata', 0.058), ('proportion', 0.058), ('interviewer', 0.056), ('imputing', 0.056), ('would', 0.056), ('assumption', 0.054), ('bounds', 0.054), ('ordering', 0.052), ('characterized', 0.05), ('sample', 0.049), ('either', 0.048), ('minimum', 0.047), ('combine', 0.047), ('assumptions', 0.047), ('depending', 0.046), ('estimate', 0.045), ('target', 0.043), ('could', 0.042), ('order', 0.042), ('worst', 0.041)]
simIndex simValue blogId blogTitle
same-blog 1 0.99999988 777 andrew gelman stats-2011-06-23-Combining survey data obtained using different modes of sampling
Introduction: I’m involved (with Irv Garfinkel and others) in a planned survey of New York City residents. It’s hard to reach people in the city–not everyone will answer their mail or phone, and you can’t send an interviewer door-to-door in a locked apartment building. (I think it violates IRB to have a plan of pushing all the buzzers by the entrance and hoping someone will let you in.) So the plan is to use multiple modes, including phone, in person household, random street intercepts and mail. The question then is how to combine these samples. My suggested approach is to divide the population into poststrata based on various factors (age, ethnicity, family type, housing type, etc), then to pool responses within each poststratum, then to runs some regressions including postratsta and also indicators for mode, to understand how respondents from different modes differ, after controlling for the demographic/geographic adjustments. Maybe this has already been done and written up somewhere? P.
2 0.2407003 1018 andrew gelman stats-2011-11-19-Tempering and modes
Introduction: Gustavo writes: Tempering should always be done in the spirit of *searching* for important modes of the distribution. If we assume that we know where they are, then there is no point to tempering. Now, tempering is actually a *bad* way of searching for important modes, it just happens to be easy to program. As always, my [Gustavo's] prescription is to FIRST find the important modes (as a pre-processing step); THEN sample from each mode independently; and FINALLY weight the samples appropriately, based on the estimated probability mass of each mode, though things might get messy if you end up jumping between modes. My reply: 1. Parallel tempering has always seemed like a great idea, but I have to admit that the only time I tried it (with Matt2 on the tree-ring example), it didn’t work for us. 2. You say you’d rather sample from the modes and then average over them. But that won’t work if if you have a zillion modes. Also, if you know where the modes are, the quickest w
3 0.18403108 368 andrew gelman stats-2010-10-25-Is instrumental variables analysis particularly susceptible to Type M errors?
Introduction: Hendrik Juerges writes: I am an applied econometrician. The reason I am writing is that I am pondering a question for some time now and I am curious whether you have any views on it. One problem the practitioner of instrumental variables estimation faces is large standard errors even with very large samples. Part of the problem is of course that one estimates a ratio. Anyhow, more often than not, I and many other researchers I know end up with large point estimates and standard errors when trying IV on a problem. Sometimes some of us are lucky and get a statistically significant result. Those estimates that make it beyond the 2 standard error threshold are often ridiculously large (one famous example in my line of research being Lleras-Muney’s estimates of the 10% effect of one year of schooling on mortality). The standard defense here is that IV estimates the complier-specific causal effect (which is mathematically correct). But still, I find many of the IV results (including my
4 0.16196409 779 andrew gelman stats-2011-06-25-Avoiding boundary estimates using a prior distribution as regularization
Introduction: For awhile I’ve been fitting most of my multilevel models using lmer/glmer, which gives point estimates of the group-level variance parameters (maximum marginal likelihood estimate for lmer and an approximation for glmer). I’m usually satisfied with this–sure, point estimation understates the uncertainty in model fitting, but that’s typically the least of our worries. Sometimes, though, lmer/glmer estimates group-level variances at 0 or estimates group-level correlation parameters at +/- 1. Typically, when this happens, it’s not that we’re so sure the variance is close to zero or that the correlation is close to 1 or -1; rather, the marginal likelihood does not provide a lot of information about these parameters of the group-level error distribution. I don’t want point estimates on the boundary. I don’t want to say that the unexplained variance in some dimension is exactly zero. One way to handle this problem is full Bayes: slap a prior on sigma, do your Gibbs and Metropolis
5 0.14755808 2041 andrew gelman stats-2013-09-27-Setting up Jitts online
Introduction: I use just-in-time teaching assignments in all my classes now. Vince helpfully sent along these instructions for setting these up on Google. See below. I think Jitts are just wonderful, and they’re so easy to set up, you should definitely be doing them in your classes too. I’ve had more difficulty with Peer Instruction (the companion tool to just-in-time teaching) as it requires questions at just the right level for the class. I do have students frequently work in pairs, though, so I think I get some of the benefit of that. P.S. I’d love to share all the Jitts with you for Bayesian Data Analysis, but I’m afraid this would poison the well and future students would not have the opportunity to be surprised by them. Yes, I know, I should just come up with new ones every year—but I’m not quite ready to do that! Perhaps soon I will. In the meantime, a commenter asked for some Jitts, so here are the ones for the first and last weeks of class: Jitt questions for Bayesian Data
7 0.1219416 725 andrew gelman stats-2011-05-21-People kept emailing me this one so I think I have to blog something
8 0.12103468 535 andrew gelman stats-2011-01-24-Bleg: Automatic Differentiation for Log Prob Gradients?
9 0.12037399 2093 andrew gelman stats-2013-11-07-I’m negative on the expression “false positives”
10 0.11253694 1362 andrew gelman stats-2012-06-03-Question 24 of my final exam for Design and Analysis of Sample Surveys
11 0.10201602 735 andrew gelman stats-2011-05-28-New app for learning intro statistics
12 0.10058557 2102 andrew gelman stats-2013-11-15-“Are all significant p-values created equal?”
13 0.097581029 553 andrew gelman stats-2011-02-03-is it possible to “overstratify” when assigning a treatment in a randomized control trial?
14 0.095319234 784 andrew gelman stats-2011-07-01-Weighting and prediction in sample surveys
16 0.090326861 2330 andrew gelman stats-2014-05-12-Historical Arc of Universities
17 0.090259738 1722 andrew gelman stats-2013-02-14-Statistics for firefighters: update
18 0.089276671 7 andrew gelman stats-2010-04-27-Should Mister P be allowed-encouraged to reside in counter-factual populations?
19 0.087632194 1344 andrew gelman stats-2012-05-25-Question 15 of my final exam for Design and Analysis of Sample Surveys
20 0.085910723 1133 andrew gelman stats-2012-01-21-Judea Pearl on why he is “only a half-Bayesian”
topicId topicWeight
[(0, 0.143), (1, 0.035), (2, 0.079), (3, -0.072), (4, 0.051), (5, 0.034), (6, 0.011), (7, 0.008), (8, 0.004), (9, -0.032), (10, -0.022), (11, -0.044), (12, 0.017), (13, 0.024), (14, -0.034), (15, 0.02), (16, -0.006), (17, -0.025), (18, -0.013), (19, 0.034), (20, -0.017), (21, -0.011), (22, 0.043), (23, 0.011), (24, -0.004), (25, 0.009), (26, 0.017), (27, -0.007), (28, 0.017), (29, 0.019), (30, 0.028), (31, -0.013), (32, -0.019), (33, 0.049), (34, 0.017), (35, -0.046), (36, 0.015), (37, -0.001), (38, 0.002), (39, 0.017), (40, 0.009), (41, -0.04), (42, -0.009), (43, -0.009), (44, -0.048), (45, 0.009), (46, 0.02), (47, 0.018), (48, -0.014), (49, 0.022)]
simIndex simValue blogId blogTitle
same-blog 1 0.97457129 777 andrew gelman stats-2011-06-23-Combining survey data obtained using different modes of sampling
Introduction: I’m involved (with Irv Garfinkel and others) in a planned survey of New York City residents. It’s hard to reach people in the city–not everyone will answer their mail or phone, and you can’t send an interviewer door-to-door in a locked apartment building. (I think it violates IRB to have a plan of pushing all the buzzers by the entrance and hoping someone will let you in.) So the plan is to use multiple modes, including phone, in person household, random street intercepts and mail. The question then is how to combine these samples. My suggested approach is to divide the population into poststrata based on various factors (age, ethnicity, family type, housing type, etc), then to pool responses within each poststratum, then to runs some regressions including postratsta and also indicators for mode, to understand how respondents from different modes differ, after controlling for the demographic/geographic adjustments. Maybe this has already been done and written up somewhere? P.
Introduction: David Radwin asks a question which comes up fairly often in one form or another: How should one respond to requests for statistical hypothesis tests for population (or universe) data? I [Radwin] first encountered this issue as an undergraduate when a professor suggested a statistical significance test for my paper comparing roll call votes between freshman and veteran members of Congress. Later I learned that such tests apply only to samples because their purpose is to tell you whether the difference in the observed sample is likely to exist in the population. If you have data for the whole population, like all members of the 103rd House of Representatives, you do not need a test to discern the true difference in the population. Sometimes researchers assume some sort of superpopulation like “all possible Congresses” or “Congresses across all time” and that the members of any given Congress constitute a sample. In my current work in education research, it is sometimes asserted t
3 0.71605545 1330 andrew gelman stats-2012-05-19-Cross-validation to check missing-data imputation
Introduction: Aureliano Crameri writes: I have questions regarding one technique you and your colleagues described in your papers: the cross validation (Multiple Imputation with Diagnostics (mi) in R: Opening Windows into the Black Box, with reference to Gelman, King, and Liu, 1998). I think this is the technique I need for my purpose, but I am not sure I understand it right. I want to use the multiple imputation to estimate the outcome of psychotherapies based on longitudinal data. First I have to demonstrate that I am able to get unbiased estimates with the multiple imputation. The expected bias is the overestimation of the outcome of dropouts. I will test my imputation strategies by means of a series of simulations (delete values, impute, compare with the original). Due to the complexity of the statistical analyses I think I need at least 200 cases. Now I don’t have so many cases without any missings. My data have missing values in different variables. The proportion of missing values is
4 0.70518965 1940 andrew gelman stats-2013-07-16-A poll that throws away data???
Introduction: Mark Blumenthal writes: What do you think about the “random rejection” method used by PPP that was attacked at some length today by a Republican pollster. Our just published post on the debate includes all the details as I know them. The Storify of Martino’s tweets has some additional data tables linked to toward the end. Also, more specifically, setting aside Martino’s suggestion of manipulation (which is also quite possible with post-stratification weights), would the PPP method introduce more potential random error than weighting? From Blumenthal’s blog: B.J. Martino, a senior vice president at the Republican polling firm The Tarrance Group, went on an 30-minute Twitter rant on Tuesday questioning the unorthodox method used by PPP [Public Policy Polling] to select samples and weight data: “Looking at @ppppolls new VA SW. Wondering how many interviews they discarded to get down to 601 completes? Because @ppppolls discards a LOT of interviews. Of 64,811 conducted
Introduction: A reporter emailed me the other day with a question about a case I’d never heard of before, a company called Herbalife that is being accused of being a pyramid scheme. The reporter pointed me to this document which describes a survey conducted by “a third party firm called Lieberman Research”: Two independent studies took place using real time (aka “river”) sampling, in which respondents were intercepted across a wide array of websites Sample size of 2,000 adults 18+ matched to U.S. census on age, gender, income, region and ethnicity “River sampling” in this case appears to mean, according to the reporter, that “people were invited into it through online ads.” The survey found that 5% of U.S. households had purchased Herbalife products during the past three months (with a “0.8% margin of error,” ha ha ha). They they did a multiplication and a division to estimate that only 8% of households who bought these products were Herbalife distributors: 480,000 active distributor
6 0.6976217 2152 andrew gelman stats-2013-12-28-Using randomized incentives as an instrument for survey nonresponse?
8 0.6790601 784 andrew gelman stats-2011-07-01-Weighting and prediction in sample surveys
9 0.67884493 368 andrew gelman stats-2010-10-25-Is instrumental variables analysis particularly susceptible to Type M errors?
10 0.67626143 2303 andrew gelman stats-2014-04-23-Thinking of doing a list experiment? Here’s a list of reasons why you should think again
11 0.67553645 960 andrew gelman stats-2011-10-15-The bias-variance tradeoff
12 0.67456788 212 andrew gelman stats-2010-08-17-Futures contracts, Granger causality, and my preference for estimation to testing
14 0.67378426 1933 andrew gelman stats-2013-07-10-Please send all comments to -dev-ripley
15 0.67280239 923 andrew gelman stats-2011-09-24-What is the normal range of values in a medical test?
16 0.67252493 1362 andrew gelman stats-2012-06-03-Question 24 of my final exam for Design and Analysis of Sample Surveys
17 0.66607869 553 andrew gelman stats-2011-02-03-is it possible to “overstratify” when assigning a treatment in a randomized control trial?
18 0.66580349 1455 andrew gelman stats-2012-08-12-Probabilistic screening to get an approximate self-weighted sample
19 0.66432154 820 andrew gelman stats-2011-07-25-Design of nonrandomized cluster sample study
20 0.65851682 142 andrew gelman stats-2010-07-12-God, Guns, and Gaydar: The Laws of Probability Push You to Overestimate Small Groups
topicId topicWeight
[(2, 0.015), (15, 0.023), (16, 0.088), (21, 0.048), (24, 0.118), (29, 0.014), (43, 0.023), (45, 0.05), (53, 0.028), (70, 0.046), (73, 0.02), (76, 0.032), (86, 0.065), (89, 0.029), (96, 0.016), (97, 0.022), (99, 0.196)]
simIndex simValue blogId blogTitle
same-blog 1 0.96480632 777 andrew gelman stats-2011-06-23-Combining survey data obtained using different modes of sampling
Introduction: I’m involved (with Irv Garfinkel and others) in a planned survey of New York City residents. It’s hard to reach people in the city–not everyone will answer their mail or phone, and you can’t send an interviewer door-to-door in a locked apartment building. (I think it violates IRB to have a plan of pushing all the buzzers by the entrance and hoping someone will let you in.) So the plan is to use multiple modes, including phone, in person household, random street intercepts and mail. The question then is how to combine these samples. My suggested approach is to divide the population into poststrata based on various factors (age, ethnicity, family type, housing type, etc), then to pool responses within each poststratum, then to runs some regressions including postratsta and also indicators for mode, to understand how respondents from different modes differ, after controlling for the demographic/geographic adjustments. Maybe this has already been done and written up somewhere? P.
2 0.93193889 1266 andrew gelman stats-2012-04-16-Another day, another plagiarist
Introduction: This one isn’t actually new, but it’s new to me. It involves University of Michigan business school professor Karl Weick. Here’s the relevant paragraph of Weick’s Wikipedia entry (as of 13 Apr 2012): In several published articles, Weick related a story that originally appeared in a poem by Miroslav Holub that was published in the Times Literary Supplement. Weick plagiarized Holub in that he republished the poem (with some minor differences, including removing line breaks and making small changes in a few words) without quotation or attribution. Some of Weick’s articles included the material with no reference to Holub; others referred to Holub but without indicating that Weick had essentially done a direct copy of Holub’s writing. The plagiarism was detailed in an article by Thomas Basbøll and Henrik Graham. [5] In a response, Weick disputed the claim of plagiarism, writing, “By the time I began to see the Alps story as an example of cognition in the path of the action, I had lo
3 0.91927916 2246 andrew gelman stats-2014-03-13-An Economist’s Guide to Visualizing Data
Introduction: Stephen Jenkins wrote: I was thinking that you and your blog readers might be interested in “ An Economist’s Guide to Visualizing Data ” by Jonathan Schwabish, in the most recent Journal of Economic Perspectives (which is the American Economic Association’s main “outreach” journal in some ways). I replied: Ooh, I hate this so much! This seems to represent a horrible example of economists not recognizing that outsiders can help them. We do much much better in political science. To which Jenkins wrote: Ha! I guessed as much — hence sent it. And I’ll now admit I was surprised that JEP took the piece without getting Schwabisch to widen his reference points. To elaborate a bit: I agree with Schwabish’s general advice (“show the data,” “reduce the clutter,” and “integrate the text and the graph”). But then he illustrates with 8 before-and-after stories in which he shows an existing graph and then gives his improvements. My problem is that I don’t like most of his “afte
4 0.91723382 1019 andrew gelman stats-2011-11-19-Validation of Software for Bayesian Models Using Posterior Quantiles
Introduction: I love this stuff : This article presents a simulation-based method designed to establish the computational correctness of software developed to fit a specific Bayesian model, capitalizing on properties of Bayesian posterior distributions. We illustrate the validation technique with two examples. The validation method is shown to find errors in software when they exist and, moreover, the validation output can be informative about the nature and location of such errors. We also compare our method with that of an earlier approach. I hope we can put it into Stan.
5 0.9163143 2281 andrew gelman stats-2014-04-04-The Notorious N.H.S.T. presents: Mo P-values Mo Problems
Introduction: A recent discussion between commenters Question and Fernando captured one of the recurrent themes here from the past year. Question: The problem is simple, the researchers are disproving always false null hypotheses and taking this disproof as near proof that their theory is correct. Fernando: Whereas it is probably true that researchers misuse NHT, the problem with tabloid science is broader and deeper. It is systemic. Question: I do not see how anything can be deeper than replacing careful description, prediction, falsification, and independent replication with dynamite plots, p-values, affirming the consequent, and peer review. From my own experience I am confident in saying that confusion caused by NHST is at the root of this problem. Fernando: Incentives? Impact factors? Publish or die? “Interesting” and “new” above quality and reliability, or actually answering a research question, and a silly and unbecoming obsession with being quoted in NYT, etc. . . . Giv
6 0.91600186 1278 andrew gelman stats-2012-04-23-“Any old map will do” meets “God is in every leaf of every tree”
7 0.91311562 807 andrew gelman stats-2011-07-17-Macro causality
8 0.91283649 116 andrew gelman stats-2010-06-29-How to grab power in a democracy – in 5 easy non-violent steps
9 0.91205692 1980 andrew gelman stats-2013-08-13-Test scores and grades predict job performance (but maybe not at Google)
10 0.91026831 2182 andrew gelman stats-2014-01-22-Spell-checking example demonstrates key aspects of Bayesian data analysis
11 0.90968609 982 andrew gelman stats-2011-10-30-“There’s at least as much as an 80 percent chance . . .”
12 0.90918934 994 andrew gelman stats-2011-11-06-Josh Tenenbaum presents . . . a model of folk physics!
14 0.90645415 1012 andrew gelman stats-2011-11-16-Blog bribes!
15 0.90584922 1121 andrew gelman stats-2012-01-15-R-squared for multilevel models
16 0.9055258 2121 andrew gelman stats-2013-12-02-Should personal genetic testing be regulated? Battle of the blogroll
17 0.9048847 2058 andrew gelman stats-2013-10-11-Gladwell and Chabris, David and Goliath, and science writing as stone soup
18 0.90436286 2201 andrew gelman stats-2014-02-06-Bootstrap averaging: Examples where it works and where it doesn’t work
19 0.90422827 1163 andrew gelman stats-2012-02-12-Meta-analysis, game theory, and incentives to do replicable research
20 0.90421581 2149 andrew gelman stats-2013-12-26-Statistical evidence for revised standards