andrew_gelman_stats andrew_gelman_stats-2013 andrew_gelman_stats-2013-2152 knowledge-graph by maker-knowledge-mining
Source: html
Introduction: I received the following question: Is there a classic paper on instrumenting for survey non-response? some colleagues in public health are going to carry out a survey and I wonder about suggesting that they build in a randomization of response-encouragement (e.g. offering additional $ to a subset of those who don’t respond initially). Can you recommend a basic treatment of this, and why it might or might not make sense compared to IPW using covariates (without an instrument)? My reply: Here’s the best analysis I know of on the effects of incentives for survey response. There have been several survey-experiments on the subject. The short answer is that the effect on nonresponse is small and the outcome is highly variable, hence you can’t very well use it as an instrument in any particular survey. My recommended approach to dealing with nonresponse is to use multilevel regression and poststratification; an example is here . Inverse-probability weighting doesn’t really w
sentIndex sentText sentNum sentScore
1 I received the following question: Is there a classic paper on instrumenting for survey non-response? [sent-1, score-0.667]
2 some colleagues in public health are going to carry out a survey and I wonder about suggesting that they build in a randomization of response-encouragement (e. [sent-2, score-1.139]
3 offering additional $ to a subset of those who don’t respond initially). [sent-4, score-0.499]
4 Can you recommend a basic treatment of this, and why it might or might not make sense compared to IPW using covariates (without an instrument)? [sent-5, score-0.678]
5 My reply: Here’s the best analysis I know of on the effects of incentives for survey response. [sent-6, score-0.507]
6 There have been several survey-experiments on the subject. [sent-7, score-0.07]
7 The short answer is that the effect on nonresponse is small and the outcome is highly variable, hence you can’t very well use it as an instrument in any particular survey. [sent-8, score-1.497]
8 My recommended approach to dealing with nonresponse is to use multilevel regression and poststratification; an example is here . [sent-9, score-1.074]
9 Inverse-probability weighting doesn’t really work because you don’t know the probability of nonresponse; all you really can do, usually, is poststratify. [sent-10, score-0.434]
wordName wordTfidf (topN-words)
[('nonresponse', 0.488), ('instrument', 0.321), ('survey', 0.236), ('instrumenting', 0.224), ('randomization', 0.163), ('initially', 0.161), ('poststratification', 0.147), ('carry', 0.146), ('covariates', 0.146), ('offering', 0.143), ('subset', 0.14), ('weighting', 0.139), ('dealing', 0.139), ('incentives', 0.129), ('suggesting', 0.126), ('build', 0.125), ('recommended', 0.124), ('classic', 0.112), ('respond', 0.11), ('hence', 0.108), ('additional', 0.106), ('outcome', 0.105), ('recommend', 0.098), ('highly', 0.097), ('treatment', 0.095), ('received', 0.095), ('colleagues', 0.094), ('multilevel', 0.093), ('health', 0.093), ('variable', 0.091), ('basic', 0.091), ('compared', 0.088), ('short', 0.087), ('use', 0.087), ('discuss', 0.085), ('usually', 0.085), ('wonder', 0.084), ('issues', 0.08), ('might', 0.08), ('really', 0.076), ('public', 0.072), ('regression', 0.072), ('know', 0.072), ('approach', 0.071), ('probability', 0.071), ('answer', 0.071), ('several', 0.07), ('effects', 0.07), ('small', 0.067), ('effect', 0.066)]
simIndex simValue blogId blogTitle
same-blog 1 1.0 2152 andrew gelman stats-2013-12-28-Using randomized incentives as an instrument for survey nonresponse?
Introduction: I received the following question: Is there a classic paper on instrumenting for survey non-response? some colleagues in public health are going to carry out a survey and I wonder about suggesting that they build in a randomization of response-encouragement (e.g. offering additional $ to a subset of those who don’t respond initially). Can you recommend a basic treatment of this, and why it might or might not make sense compared to IPW using covariates (without an instrument)? My reply: Here’s the best analysis I know of on the effects of incentives for survey response. There have been several survey-experiments on the subject. The short answer is that the effect on nonresponse is small and the outcome is highly variable, hence you can’t very well use it as an instrument in any particular survey. My recommended approach to dealing with nonresponse is to use multilevel regression and poststratification; an example is here . Inverse-probability weighting doesn’t really w
2 0.17772323 1315 andrew gelman stats-2012-05-12-Question 2 of my final exam for Design and Analysis of Sample Surveys
Introduction: 2. Which of the following are useful goals in a pilot study? (Indicate all that apply.) (a) You can search for statistical significance, then from that decide what to look for in a confirmatory analysis of your full dataset. (b) You can see if you find statistical significance in a pre-chosen comparison of interest. (c) You can examine the direction (positive or negative, even if not statistically significant) of comparisons of interest. (d) With a small sample size, you cannot hope to learn anything conclusive, but you can get a crude estimate of effect size and standard deviation which will be useful in a power analysis to help you decide how large your full study needs to be. (e) You can talk with survey respondents and get a sense of how they perceived your questions. (f) You get a chance to learn about practical difficulties with sampling, nonresponse, and question wording. (g) You can check if your sample is approximately representative of your population. Soluti
3 0.15204144 1430 andrew gelman stats-2012-07-26-Some thoughts on survey weighting
Introduction: From a comment I made in an email exchange: My work on survey adjustments has very much been inspired by the ideas of Rod Little. Much of my efforts have gone toward the goal of integrating hierarchical modeling (which is so helpful for small-area estimation) with post stratification (which adjusts for known differences between sample and population). In the surveys I’ve dealt with, nonresponse/nonavailability can be a big issue, and I’ve always tried to emphasize that (a) the probability of a person being included in the sample is just about never known, and (b) even if this probability were known, I’d rather know the empirical n/N than the probability p (which is only valid in expectation). Regarding nonparametric modeling: I haven’t done much of that (although I hope to at some point) but Rod and his students have. As I wrote in the first sentence of the above-linked paper, I do think the current theory and practice of survey weighting is a mess, in that much depends on so
Introduction: Leading theoretical statistician Larry Wassserman in 2008 : Some of the greatest contributions of statistics to science involve adding additional randomness and leveraging that randomness. Examples are randomized experiments, permutation tests, cross-validation and data-splitting. These are unabashedly frequentist ideas and, while one can strain to fit them into a Bayesian framework, they don’t really have a place in Bayesian inference. The fact that Bayesian methods do not naturally accommodate such a powerful set of statistical ideas seems like a serious deficiency. To which I responded on the second-to-last paragraph of page 8 here . Larry Wasserman in 2013 : Some people say that there is no role for randomization in Bayesian inference. In other words, the randomization mechanism plays no role in Bayes’ theorem. But this is not really true. Without randomization, we can indeed derive a posterior for theta but it is highly sensitive to the prior. This is just a restat
5 0.14399891 352 andrew gelman stats-2010-10-19-Analysis of survey data: Design based models vs. hierarchical modeling?
Introduction: Alban Zeber writes: Suppose I have survey data from say 10 countries where by each country collected the data based on different sampling routines – the results of this being that each country has its own weights for the data that can be used in the analyses. If I analyse the data of each country separately then I can incorporate the survey design in the analyses e.g in Stata once can use svyset ….. But what happens when I want to do a pooled analysis of the all the data from the 10 countries: Presumably either 1. I analyse the data from each country separately (using multiple or logistic regression, …) accounting for the survey design and then combine the estimates using a meta analysis (fixed or random) OR 2. Assume that the data from each country is a simple random sample from the population, combine the data from the 10 countries and then use multilevel or hierarchical models My question is which of the methods is likely to give better estimates? Or is the
6 0.14143488 761 andrew gelman stats-2011-06-13-A survey’s not a survey if they don’t tell you how they did it
7 0.13737217 1371 andrew gelman stats-2012-06-07-Question 28 of my final exam for Design and Analysis of Sample Surveys
8 0.13709694 1900 andrew gelman stats-2013-06-15-Exploratory multilevel analysis when group-level variables are of importance
9 0.13498829 1344 andrew gelman stats-2012-05-25-Question 15 of my final exam for Design and Analysis of Sample Surveys
10 0.13105166 1341 andrew gelman stats-2012-05-24-Question 14 of my final exam for Design and Analysis of Sample Surveys
11 0.12001525 1317 andrew gelman stats-2012-05-13-Question 3 of my final exam for Design and Analysis of Sample Surveys
12 0.11802901 85 andrew gelman stats-2010-06-14-Prior distribution for design effects
13 0.11685093 784 andrew gelman stats-2011-07-01-Weighting and prediction in sample surveys
15 0.10845664 705 andrew gelman stats-2011-05-10-Some interesting unpublished ideas on survey weighting
16 0.10827607 2359 andrew gelman stats-2014-06-04-All the Assumptions That Are My Life
18 0.10050061 936 andrew gelman stats-2011-10-02-Covariate Adjustment in RCT - Model Overfitting in Multilevel Regression
19 0.099821381 1814 andrew gelman stats-2013-04-20-A mess with which I am comfortable
20 0.098503172 2351 andrew gelman stats-2014-05-28-Bayesian nonparametric weighted sampling inference
topicId topicWeight
[(0, 0.154), (1, 0.051), (2, 0.094), (3, -0.104), (4, 0.078), (5, 0.056), (6, -0.016), (7, -0.012), (8, 0.067), (9, -0.011), (10, 0.046), (11, -0.092), (12, 0.043), (13, 0.078), (14, 0.025), (15, -0.013), (16, -0.036), (17, 0.004), (18, 0.017), (19, 0.035), (20, -0.043), (21, 0.003), (22, -0.01), (23, 0.071), (24, -0.057), (25, 0.004), (26, 0.023), (27, -0.019), (28, -0.03), (29, 0.004), (30, 0.011), (31, 0.033), (32, -0.023), (33, 0.038), (34, -0.076), (35, -0.049), (36, 0.026), (37, 0.001), (38, -0.019), (39, 0.015), (40, 0.018), (41, 0.03), (42, 0.043), (43, -0.07), (44, 0.013), (45, 0.058), (46, 0.043), (47, 0.014), (48, -0.048), (49, 0.042)]
simIndex simValue blogId blogTitle
same-blog 1 0.98297131 2152 andrew gelman stats-2013-12-28-Using randomized incentives as an instrument for survey nonresponse?
Introduction: I received the following question: Is there a classic paper on instrumenting for survey non-response? some colleagues in public health are going to carry out a survey and I wonder about suggesting that they build in a randomization of response-encouragement (e.g. offering additional $ to a subset of those who don’t respond initially). Can you recommend a basic treatment of this, and why it might or might not make sense compared to IPW using covariates (without an instrument)? My reply: Here’s the best analysis I know of on the effects of incentives for survey response. There have been several survey-experiments on the subject. The short answer is that the effect on nonresponse is small and the outcome is highly variable, hence you can’t very well use it as an instrument in any particular survey. My recommended approach to dealing with nonresponse is to use multilevel regression and poststratification; an example is here . Inverse-probability weighting doesn’t really w
2 0.86543763 1430 andrew gelman stats-2012-07-26-Some thoughts on survey weighting
Introduction: From a comment I made in an email exchange: My work on survey adjustments has very much been inspired by the ideas of Rod Little. Much of my efforts have gone toward the goal of integrating hierarchical modeling (which is so helpful for small-area estimation) with post stratification (which adjusts for known differences between sample and population). In the surveys I’ve dealt with, nonresponse/nonavailability can be a big issue, and I’ve always tried to emphasize that (a) the probability of a person being included in the sample is just about never known, and (b) even if this probability were known, I’d rather know the empirical n/N than the probability p (which is only valid in expectation). Regarding nonparametric modeling: I haven’t done much of that (although I hope to at some point) but Rod and his students have. As I wrote in the first sentence of the above-linked paper, I do think the current theory and practice of survey weighting is a mess, in that much depends on so
3 0.80878001 784 andrew gelman stats-2011-07-01-Weighting and prediction in sample surveys
Introduction: A couple years ago Rod Little was invited to write an article for the diamond jubilee of the Calcutta Statistical Association Bulletin. His article was published with discussions from Danny Pfefferman, J. N. K. Rao, Don Rubin, and myself. Here it all is . I’ll paste my discussion below, but it’s worth reading the others’ perspectives too. Especially the part in Rod’s rejoinder where he points out a mistake I made. Survey weights, like sausage and legislation, are designed and best appreciated by those who are placed a respectable distance from their manufacture. For those of us working inside the factory, vigorous discussion of methods is appreciated. I enjoyed Rod Little’s review of the connections between modeling and survey weighting and have just a few comments. I like Little’s discussion of model-based shrinkage of post-stratum averages, which, as he notes, can be seen to correspond to shrinkage of weights. I would only add one thing to his formula at the end of his
4 0.80537176 1900 andrew gelman stats-2013-06-15-Exploratory multilevel analysis when group-level variables are of importance
Introduction: Steve Miller writes: Much of what I do is cross-national analyses of survey data (largely World Values Survey). . . . My big question pertains to (what I would call) exploratory analysis of multilevel data, especially when the group-level predictors are of theoretical importance. A lot of what I do involves analyzing cross-national survey items of citizen attitudes, typically of political leadership. These survey items are usually yes/no responses, or four-part responses indicating a level of agreement (strongly agree, agree, disagree, strongly disagree) that can be condensed into a binary variable. I believe these can be explained by reference to country-level factors. Much of the group-level variables of interest are count variables with a modal value of 0, which can be quite messy. How would you recommend exploring the variation in the dependent variable as it could be explained by the group-level count variable of interest, before fitting the multilevel model itself? When
5 0.77829307 1814 andrew gelman stats-2013-04-20-A mess with which I am comfortable
Introduction: Having established that survey weighting is a mess, I should also acknowledge that, by this standard, regression modeling is also a mess, involving many arbitrary choices of variable selection, transformations and modeling of interaction. Nonetheless, regression modeling is a mess with which I am comfortable and, perhaps more relevant to the discussion, can be extended using multilevel models to get inference for small cross-classifications or small areas. We’re working on it.
6 0.72300744 1371 andrew gelman stats-2012-06-07-Question 28 of my final exam for Design and Analysis of Sample Surveys
7 0.72278082 1455 andrew gelman stats-2012-08-12-Probabilistic screening to get an approximate self-weighted sample
8 0.69502085 1320 andrew gelman stats-2012-05-14-Question 4 of my final exam for Design and Analysis of Sample Surveys
9 0.69496346 761 andrew gelman stats-2011-06-13-A survey’s not a survey if they don’t tell you how they did it
10 0.6938706 352 andrew gelman stats-2010-10-19-Analysis of survey data: Design based models vs. hierarchical modeling?
11 0.68561971 405 andrew gelman stats-2010-11-10-Estimation from an out-of-date census
12 0.68435395 14 andrew gelman stats-2010-05-01-Imputing count data
13 0.6789993 1362 andrew gelman stats-2012-06-03-Question 24 of my final exam for Design and Analysis of Sample Surveys
15 0.67239237 1294 andrew gelman stats-2012-05-01-Modeling y = a + b + c
16 0.66980845 385 andrew gelman stats-2010-10-31-Wacky surveys where they don’t tell you the questions they asked
17 0.66936803 1940 andrew gelman stats-2013-07-16-A poll that throws away data???
18 0.66678458 777 andrew gelman stats-2011-06-23-Combining survey data obtained using different modes of sampling
19 0.66402525 1345 andrew gelman stats-2012-05-26-Question 16 of my final exam for Design and Analysis of Sample Surveys
20 0.65832245 1691 andrew gelman stats-2013-01-25-Extreem p-values!
topicId topicWeight
[(2, 0.033), (16, 0.081), (24, 0.113), (63, 0.019), (77, 0.017), (78, 0.043), (86, 0.075), (89, 0.039), (99, 0.463)]
simIndex simValue blogId blogTitle
same-blog 1 0.99515676 2152 andrew gelman stats-2013-12-28-Using randomized incentives as an instrument for survey nonresponse?
Introduction: I received the following question: Is there a classic paper on instrumenting for survey non-response? some colleagues in public health are going to carry out a survey and I wonder about suggesting that they build in a randomization of response-encouragement (e.g. offering additional $ to a subset of those who don’t respond initially). Can you recommend a basic treatment of this, and why it might or might not make sense compared to IPW using covariates (without an instrument)? My reply: Here’s the best analysis I know of on the effects of incentives for survey response. There have been several survey-experiments on the subject. The short answer is that the effect on nonresponse is small and the outcome is highly variable, hence you can’t very well use it as an instrument in any particular survey. My recommended approach to dealing with nonresponse is to use multilevel regression and poststratification; an example is here . Inverse-probability weighting doesn’t really w
Introduction: Theodore Vasiloudis writes: I came upon this article by Laura Hamilton, an assistant professor in the University of California at Merced, that claims that “The more money that parents provide for higher education, the lower the grades their children earn.” I can’t help but feel that there something wrong with the basis of the study or a confounding factor causing this apparent correlation, and since you often comment on studies on your blog I thought you might find this study interesting. My reply: I have to admit that the description above made me suspicious of the study before I even looked at it. On first thought, I’d expect the effect of parent’s financial contributions to be positive (as they free the student from the need to get a job during college), but not negative. Hamilton argues that “parental investments create a disincentive for student achievement,” which may be—but I’m generally suspicious of arguments in which the rebound is bigger than the main effect.
3 0.98589355 263 andrew gelman stats-2010-09-08-The China Study: fact or fallacy?
Introduction: Alex Chernavsky writes: I recently came across an interesting blog post , written by someone who is self-taught in statistics (not that there’s anything wrong with that). I have no particular expertise in statistics, but her analysis looks impressive to me. I’d be very interested to find out the opinion of a professional statistician. Do you have any interest in blogging about this subject? My (disappointing, I’m sure) reply: This indeed looks interesting. I don’t have the time/energy to look at it more right now, and it’s too far from any areas of my expertise for me to give any kind of quick informed opinion. It would be good for this sort of discussion to appear in a nutrition journal where the real experts could get at it. I expect there are some strong statisticians who work in that field, although I don’t really know for sure. P.S. I suppose I really should try to learn more about this sort of thing, as it could well affect my life more than a lot of other subje
4 0.98512352 822 andrew gelman stats-2011-07-26-Any good articles on the use of error bars?
Introduction: Hadley Wickham asks: I was wondering if you knew of any good articles on the use of error bars. I’m particularly looking for articles that discuss the difference between error of means and error of difference in the context of models (e.g. mixed models) where they are very different. I suspect every applied field has a couple of good articles, but it’s really hard to search for them. Can anyone help on this? My only advice is to get rid of those horrible crossbars at the ends of the error bars. The crossbars draw attention to the error bars’ endpoints, which are generally not important at all. See, for example, my Anova paper , for some examples of how I like error bars to look.
5 0.98387152 2009 andrew gelman stats-2013-09-05-A locally organized online BDA course on G+ hangout?
Introduction: Eoin Lawless wrote me: I’ve been reading your blog (and John Kruschke ‘s) for several months now, as a result of starting to learn Bayesian methods from Doing Bayesian Data Analysis [I love the title of that book! --- ed.]. More recently I completed a Coursera course on Data Science. I found learning through the medium of a online course to be an amazing experience. It does not replace books, but learning new material at the same time as other people and discussing it in the forums is very motivational. Additionally it is much easier to work through exercises and projects when there is a deadline and some element of competition than to plow through the end of chapter exercises in a book. This is especially true, I believe, when the learning is for a long term goal, rather than to be used immediately in work, for example. My question: you are obviously evangelical about the benefits that Bayesian statistics brings, have you ever considered producing a Coursera (or similar) cour
6 0.9837321 1340 andrew gelman stats-2012-05-23-Question 13 of my final exam for Design and Analysis of Sample Surveys
7 0.98321247 773 andrew gelman stats-2011-06-18-Should we always be using the t and robit instead of the normal and logit?
8 0.98259437 2255 andrew gelman stats-2014-03-19-How Americans vote
9 0.9824701 1807 andrew gelman stats-2013-04-17-Data problems, coding errors…what can be done?
11 0.98199576 2258 andrew gelman stats-2014-03-21-Random matrices in the news
12 0.98190117 1740 andrew gelman stats-2013-02-26-“Is machine learning a subset of statistics?”
13 0.98184884 1095 andrew gelman stats-2012-01-01-Martin and Liu: Probabilistic inference based on consistency of model with data
14 0.98181814 1517 andrew gelman stats-2012-10-01-“On Inspiring Students and Being Human”
15 0.98157334 2084 andrew gelman stats-2013-11-01-Doing Data Science: What’s it all about?
16 0.9814285 2072 andrew gelman stats-2013-10-21-The future (and past) of statistical sciences
17 0.98119885 1173 andrew gelman stats-2012-02-17-Sports examples in class
18 0.98092568 1289 andrew gelman stats-2012-04-29-We go to war with the data we have, not the data we want
19 0.98064661 462 andrew gelman stats-2010-12-10-Who’s holding the pen?, The split screen, and other ideas for one-on-one instruction
20 0.98044002 1656 andrew gelman stats-2013-01-05-Understanding regression models and regression coefficients