andrew_gelman_stats andrew_gelman_stats-2012 andrew_gelman_stats-2012-1628 knowledge-graph by maker-knowledge-mining
Source: html
Introduction: Rama Ganesan writes: I think I am having an existential crisis. I used to work with animals (rats, mice, gerbils etc.) Then I started to work in marketing research where we did have some kind of random sampling procedure. So up until a few years ago, I was sort of okay. Now I am teaching marketing research, and I feel like there is no real random sampling anymore. I take pains to get students to understand what random means, and then the whole lot of inferential statistics. Then almost anything they do – the sample is not random. They think I am contradicting myself. They use convenience samples at every turn – for their school work, and the enormous amount on online surveying that gets done. Do you have any suggestions for me? Other than say, something like this . My reply: Statistics does not require randomness. The three essential elements of statistics are measurement, comparison, and variation. Randomness is one way to supply variation, and it’s one way to model
sentIndex sentText sentNum sentScore
1 Rama Ganesan writes: I think I am having an existential crisis. [sent-1, score-0.125]
2 I used to work with animals (rats, mice, gerbils etc. [sent-2, score-0.174]
3 ) Then I started to work in marketing research where we did have some kind of random sampling procedure. [sent-3, score-0.956]
4 Now I am teaching marketing research, and I feel like there is no real random sampling anymore. [sent-5, score-0.87]
5 I take pains to get students to understand what random means, and then the whole lot of inferential statistics. [sent-6, score-0.606]
6 Then almost anything they do – the sample is not random. [sent-7, score-0.084]
7 They use convenience samples at every turn – for their school work, and the enormous amount on online surveying that gets done. [sent-9, score-0.424]
8 The three essential elements of statistics are measurement, comparison, and variation. [sent-13, score-0.248]
9 Nor is it necessary to have “true” randomness (of the dice-throwing or urn-sampling variety) in order to have a useful probability model. [sent-15, score-0.513]
10 For example, consider our work in Red State Blue State, looking at patterns of voting given income and religious attendance by state. [sent-16, score-0.253]
11 Here we did have random sampling—we were working with survey data—but even if we’d had no sampling at all, if we’d had a census of opinions of all voters, we’d still have statistics problems. [sent-17, score-0.932]
12 So I don’t think random sampling is necessary for statistics. [sent-18, score-0.829]
13 To answer your question about nonrepresentative samples, there I think it’s best to adjust for known and modeled differences between sample and population. [sent-19, score-0.363]
14 Here the idea of random sampling is a useful start and a useful comparison point. [sent-20, score-1.059]
15 Ganesan writes back: Yes but all we seem to teach students is significance testing where randomness is assumed. [sent-21, score-0.425]
16 How far can I get away with saying that t-tests, ANOVAs are ‘robust’ to violations of this assumption? [sent-22, score-0.109]
17 My reply: One approach is to forget the t tests, F tests, etc. [sent-24, score-0.069]
18 and instead frame problems as quantitative comparisons, predictions, and causal inferences (which are a form of prediction of potential outcomes). [sent-25, score-0.081]
19 ’s, etc from a random sampling model that you recognize is an approximation. [sent-28, score-0.711]
20 This all loops back to Phil’s recent discussion . [sent-29, score-0.169]
wordName wordTfidf (topN-words)
[('sampling', 0.379), ('random', 0.332), ('randomness', 0.276), ('ganesan', 0.25), ('marketing', 0.159), ('samples', 0.129), ('rama', 0.125), ('surveying', 0.125), ('existential', 0.125), ('contradicting', 0.125), ('nonrepresentative', 0.125), ('useful', 0.119), ('anovas', 0.118), ('necessary', 0.118), ('variation', 0.112), ('comparison', 0.11), ('tests', 0.109), ('violations', 0.109), ('pains', 0.109), ('mice', 0.109), ('rats', 0.106), ('conf', 0.106), ('loops', 0.103), ('attendance', 0.095), ('elements', 0.089), ('animals', 0.088), ('enormous', 0.086), ('work', 0.086), ('convenience', 0.084), ('sample', 0.084), ('state', 0.084), ('students', 0.083), ('inferential', 0.082), ('essential', 0.081), ('frame', 0.081), ('modeled', 0.08), ('statistics', 0.078), ('census', 0.076), ('adjust', 0.074), ('reply', 0.073), ('religious', 0.072), ('robust', 0.071), ('forget', 0.069), ('phil', 0.067), ('supply', 0.067), ('variety', 0.067), ('opinions', 0.067), ('intervals', 0.066), ('back', 0.066), ('teach', 0.066)]
simIndex simValue blogId blogTitle
same-blog 1 0.99999988 1628 andrew gelman stats-2012-12-17-Statistics in a world where nothing is random
Introduction: Rama Ganesan writes: I think I am having an existential crisis. I used to work with animals (rats, mice, gerbils etc.) Then I started to work in marketing research where we did have some kind of random sampling procedure. So up until a few years ago, I was sort of okay. Now I am teaching marketing research, and I feel like there is no real random sampling anymore. I take pains to get students to understand what random means, and then the whole lot of inferential statistics. Then almost anything they do – the sample is not random. They think I am contradicting myself. They use convenience samples at every turn – for their school work, and the enormous amount on online surveying that gets done. Do you have any suggestions for me? Other than say, something like this . My reply: Statistics does not require randomness. The three essential elements of statistics are measurement, comparison, and variation. Randomness is one way to supply variation, and it’s one way to model
2 0.21879981 2359 andrew gelman stats-2014-06-04-All the Assumptions That Are My Life
Introduction: Statisticians take tours in other people’s data. All methods of statistical inference rest on statistical models. Experiments typically have problems with compliance, measurement error, generalizability to the real world, and representativeness of the sample. Surveys typically have problems of undercoverage, nonresponse, and measurement error. Real surveys are done to learn about the general population. But real surveys are not random samples. For another example, consider educational tests: what are they exactly measuring? Nobody knows. Medical research: even if it’s a randomized experiment, the participants in the study won’t be a random sample from the population for whom you’d recommend treatment. You don’t need random sampling to generalize the results of a medical experiment to the general population but you need some substantive theory to make the assumption that effects in your nonrepresentative sample of people will be similar to effects in the population of interest. Ve
3 0.20258197 1144 andrew gelman stats-2012-01-29-How many parameters are in a multilevel model?
Introduction: Stephen Collins writes: I’m reading your Multilevel modeling book and am trying to apply it to my work. I’m concerned with how to estimate a random intercept model if there are hundreds/thousands of levels. In the Gibbs sampling, am I sampling a parameter for each level? Or, just the hyper-parameters? In other words, say I had 500 zipcode intercepts modeled as ~ N(m,s). Would my posterior be two dimensional, sampling for “m” and “s,” or would it have 502 dimensions? My reply: Indeed you will have hundreds or thousands of parameters—or, in classical terms, hundreds or thousands of predictive quantities. But that’s ok. Even if none of those predictions is precise, you’re learning about the model. See page 526 of the book for more discussion of the number of parameters in a multilevel model.
Introduction: Lots of good statistical methods make use of two models. For example: - Classical statistics: estimates and standard errors using the likelihood function; tests and p-values using the sampling distribution. (The sampling distribution is not equivalent to the likelihood, as has been much discussed, for example in sequential stopping problems.) - Bayesian data analysis: inference using the posterior distribution; model checking using the predictive distribution (which, again, depends on the data-generating process in a way that the likelihood does not). - Machine learning: estimation using the data; evaluation using cross-validation (which requires some rule for partitioning the data, a rule that stands outside of the data themselves). - Bootstrap, jackknife, etc: estimation using an “estimator” (which, I would argue, is based in some sense on a model for the data), uncertainties using resampling (which, I would argue, is close to the idea of a “sampling distribution” in
Introduction: Leading theoretical statistician Larry Wassserman in 2008 : Some of the greatest contributions of statistics to science involve adding additional randomness and leveraging that randomness. Examples are randomized experiments, permutation tests, cross-validation and data-splitting. These are unabashedly frequentist ideas and, while one can strain to fit them into a Bayesian framework, they don’t really have a place in Bayesian inference. The fact that Bayesian methods do not naturally accommodate such a powerful set of statistical ideas seems like a serious deficiency. To which I responded on the second-to-last paragraph of page 8 here . Larry Wasserman in 2013 : Some people say that there is no role for randomization in Bayesian inference. In other words, the randomization mechanism plays no role in Bayes’ theorem. But this is not really true. Without randomization, we can indeed derive a posterior for theta but it is highly sensitive to the prior. This is just a restat
6 0.16654736 749 andrew gelman stats-2011-06-06-“Sampling: Design and Analysis”: a course for political science graduate students
7 0.16083558 1015 andrew gelman stats-2011-11-17-Good examples of lurking variables?
8 0.15825619 1289 andrew gelman stats-2012-04-29-We go to war with the data we have, not the data we want
9 0.15425889 107 andrew gelman stats-2010-06-24-PPS in Georgia
10 0.15217531 85 andrew gelman stats-2010-06-14-Prior distribution for design effects
11 0.14998399 405 andrew gelman stats-2010-11-10-Estimation from an out-of-date census
12 0.14538337 1267 andrew gelman stats-2012-04-17-Hierarchical-multilevel modeling with “big data”
14 0.13765217 2115 andrew gelman stats-2013-11-27-Three unblinded mice
15 0.13736227 603 andrew gelman stats-2011-03-07-Assumptions vs. conditions, part 2
16 0.13678917 1455 andrew gelman stats-2012-08-12-Probabilistic screening to get an approximate self-weighted sample
17 0.13148063 1551 andrew gelman stats-2012-10-28-A convenience sample and selected treatments
18 0.12999874 51 andrew gelman stats-2010-05-26-If statistics is so significantly great, why don’t statisticians use statistics?
19 0.12867011 1943 andrew gelman stats-2013-07-18-Data to use for in-class sampling exercises?
20 0.1258256 662 andrew gelman stats-2011-04-15-Bayesian statistical pragmatism
topicId topicWeight
[(0, 0.208), (1, 0.054), (2, 0.075), (3, -0.053), (4, 0.048), (5, 0.112), (6, -0.048), (7, 0.065), (8, 0.042), (9, -0.076), (10, 0.026), (11, -0.037), (12, -0.013), (13, 0.017), (14, -0.01), (15, -0.081), (16, -0.046), (17, 0.022), (18, -0.011), (19, -0.014), (20, 0.014), (21, -0.076), (22, -0.033), (23, 0.072), (24, -0.042), (25, 0.016), (26, -0.059), (27, 0.11), (28, 0.062), (29, 0.097), (30, 0.025), (31, -0.038), (32, -0.052), (33, 0.048), (34, -0.087), (35, 0.039), (36, -0.009), (37, 0.015), (38, -0.109), (39, 0.053), (40, 0.04), (41, -0.002), (42, 0.052), (43, -0.032), (44, -0.063), (45, -0.052), (46, -0.072), (47, 0.031), (48, 0.011), (49, -0.052)]
simIndex simValue blogId blogTitle
same-blog 1 0.97313732 1628 andrew gelman stats-2012-12-17-Statistics in a world where nothing is random
Introduction: Rama Ganesan writes: I think I am having an existential crisis. I used to work with animals (rats, mice, gerbils etc.) Then I started to work in marketing research where we did have some kind of random sampling procedure. So up until a few years ago, I was sort of okay. Now I am teaching marketing research, and I feel like there is no real random sampling anymore. I take pains to get students to understand what random means, and then the whole lot of inferential statistics. Then almost anything they do – the sample is not random. They think I am contradicting myself. They use convenience samples at every turn – for their school work, and the enormous amount on online surveying that gets done. Do you have any suggestions for me? Other than say, something like this . My reply: Statistics does not require randomness. The three essential elements of statistics are measurement, comparison, and variation. Randomness is one way to supply variation, and it’s one way to model
2 0.86778384 107 andrew gelman stats-2010-06-24-PPS in Georgia
Introduction: Lucy Flynn writes: I’m working at a non-profit organization called CRRC in the Republic of Georgia. I’m having a methodological problem and I saw the syllabus for your sampling class online and thought I might be able to ask you about it? We do a lot of complex surveys nationwide; our typical sample design is as follows: - stratify by rural/urban/capital - sub-stratify the rural and urban strata into NE/NW/SE/SW geographic quadrants - select voting precincts as PSUs - select households as SSUs - select individual respondents as TSUs I’m relatively new here, and past practice has been to sample voting precincts with probability proportional to size. It’s desirable because it’s not logistically feasible for us to vary the number of interviews per precinct with precinct size, so it makes the selection probabilities for households more even across precinct sizes. However, I have a complex sampling textbook (Lohr 1999), and it explains how complex it is to calculate sel
3 0.82681495 2359 andrew gelman stats-2014-06-04-All the Assumptions That Are My Life
Introduction: Statisticians take tours in other people’s data. All methods of statistical inference rest on statistical models. Experiments typically have problems with compliance, measurement error, generalizability to the real world, and representativeness of the sample. Surveys typically have problems of undercoverage, nonresponse, and measurement error. Real surveys are done to learn about the general population. But real surveys are not random samples. For another example, consider educational tests: what are they exactly measuring? Nobody knows. Medical research: even if it’s a randomized experiment, the participants in the study won’t be a random sample from the population for whom you’d recommend treatment. You don’t need random sampling to generalize the results of a medical experiment to the general population but you need some substantive theory to make the assumption that effects in your nonrepresentative sample of people will be similar to effects in the population of interest. Ve
4 0.76876438 603 andrew gelman stats-2011-03-07-Assumptions vs. conditions, part 2
Introduction: In response to the discussion of his remarks on assumptions vs. conditions, Jeff Witmer writes : If [certain conditions hold] , then the t-test p-value gives a remarkably good approximation to “the real thing” — namely the randomization reference p-value. . . . I [Witmer] make assumptions about conditions that I cannot check, e.g., that the data arose from a random sample. Of course, just as there is no such thing as a normal population, there is no such thing as a random sample. I disagree strongly with both the above paragraphs! I say this not to pick a fight with Jeff Witmer but to illustrate how, in statistics, even the most basic points that people take for granted, can’t be. Let’s take the claims in order: 1. The purpose of a t test is to approximate the randomization p-value. Not to me. In my world, the purpose of t tests and intervals is to summarize uncertainty in estimates and comparisons. I don’t care about a p-value and almost certainly don’t care a
5 0.76396888 405 andrew gelman stats-2010-11-10-Estimation from an out-of-date census
Introduction: Suguru Mizunoya writes: When we estimate the number of people from a national sampling survey (such as labor force survey) using sampling weights, don’t we obtain underestimated number of people, if the country’s population is growing and the sampling frame is based on an old census data? In countries with increasing populations, the probability of inclusion changes over time, but the weights can’t be adjusted frequently because census takes place only once every five or ten years. I am currently working for UNICEF for a project on estimating number of out-of-school children in developing countries. The project leader is comfortable to use estimates of number of people from DHS and other surveys. But, I am concerned that we may need to adjust the estimated number of people by the population projection, otherwise the estimates will be underestimated. I googled around on this issue, but I could not find a right article or paper on this. My reply: I don’t know if there’s a pa
6 0.73308706 1165 andrew gelman stats-2012-02-13-Philosophy of Bayesian statistics: my reactions to Wasserman
7 0.7312035 1455 andrew gelman stats-2012-08-12-Probabilistic screening to get an approximate self-weighted sample
8 0.69806504 5 andrew gelman stats-2010-04-27-Ethical and data-integrity problems in a study of mortality in Iraq
9 0.69771194 1289 andrew gelman stats-2012-04-29-We go to war with the data we have, not the data we want
10 0.69717938 972 andrew gelman stats-2011-10-25-How do you interpret standard errors from a regression fit to the entire population?
11 0.67869294 1551 andrew gelman stats-2012-10-28-A convenience sample and selected treatments
13 0.66964477 51 andrew gelman stats-2010-05-26-If statistics is so significantly great, why don’t statisticians use statistics?
14 0.66851842 1943 andrew gelman stats-2013-07-18-Data to use for in-class sampling exercises?
15 0.66509145 85 andrew gelman stats-2010-06-14-Prior distribution for design effects
16 0.66435623 784 andrew gelman stats-2011-07-01-Weighting and prediction in sample surveys
17 0.65612721 1940 andrew gelman stats-2013-07-16-A poll that throws away data???
18 0.64451677 749 andrew gelman stats-2011-06-06-“Sampling: Design and Analysis”: a course for political science graduate students
19 0.63410741 1430 andrew gelman stats-2012-07-26-Some thoughts on survey weighting
20 0.63098836 288 andrew gelman stats-2010-09-21-Discussion of the paper by Girolami and Calderhead on Bayesian computation
topicId topicWeight
[(15, 0.023), (16, 0.027), (21, 0.03), (24, 0.175), (42, 0.012), (45, 0.062), (66, 0.01), (89, 0.213), (95, 0.016), (98, 0.014), (99, 0.294)]
simIndex simValue blogId blogTitle
1 0.97773182 1215 andrew gelman stats-2012-03-16-The “hot hand” and problems with hypothesis testing
Introduction: Gur Yaari writes : Anyone who has ever watched a sports competition is familiar with expressions like “on fire”, “in the zone”, “on a roll”, “momentum” and so on. But what do these expressions really mean? In 1985 when Thomas Gilovich, Robert Vallone and Amos Tversky studied this phenomenon for the first time, they defined it as: “. . . these phrases express a belief that the performance of a player during a particular period is significantly better than expected on the basis of the player’s overall record”. Their conclusion was that what people tend to perceive as a “hot hand” is essentially a cognitive illusion caused by a misperception of random sequences. Until recently there was little, if any, evidence to rule out their conclusion. Increased computing power and new data availability from various sports now provide surprising evidence of this phenomenon, thus reigniting the debate. Yaari goes on to some studies that have found time dependence in basketball, baseball, voll
2 0.97056788 1756 andrew gelman stats-2013-03-10-He said he was sorry
Introduction: Yes, it can be done : Hereby I contact you to clarify the situation that occurred with the publication of the article entitled *** which was published in Volume 11, Issue 3 of *** and I made the mistake of declaring as an author. This chapter is a plagiarism of . . . I wish to express and acknowledge that I am solely responsible for this . . . I recognize the gravity of the offense committed, since there is no justification for so doing. Therefore, and as a sign of shame and regret I feel in this situation, I will publish this letter, in order to set an example for other researchers do not engage in a similar error. No more, and to please accept my apologies, Sincerely, *** P.S. Since we’re on Retraction Watch already, I’ll point you to this unrelated story featuring a hilarious photo of a fraudster, who in this case was a grad student in psychology who faked his data and “has agreed to submit to a three-year supervisory period for any work involving funding from the
3 0.96444929 2243 andrew gelman stats-2014-03-11-The myth of the myth of the myth of the hot hand
Introduction: Phil pointed me to this paper so I thought I probably better repeat what I wrote a couple years ago: 1. The effects are certainly not zero. We are not machines, and anything that can affect our expectations (for example, our success in previous tries) should affect our performance. 2. The effects I’ve seen are small, on the order of 2 percentage points (for example, the probability of a success in some sports task might be 45% if you’re “hot” and 43% otherwise). 3. There’s a huge amount of variation, not just between but also among players. Sometimes if you succeed you will stay relaxed and focused, other times you can succeed and get overconfidence. 4. Whatever the latest results on particular sports, I can’t see anyone overturning the basic finding of Gilovich, Vallone, and Tversky that players and spectators alike will perceive the hot hand even when it does not exist and dramatically overestimate the magnitude and consistency of any hot-hand phenomenon that does exist.
4 0.96332568 833 andrew gelman stats-2011-07-31-Untunable Metropolis
Introduction: Michael Margolis writes: What are we to make of it when a Metropolis-Hastings step just won’t tune? That is, the acceptance rate is zero at expected-jump-size X, and way above 1/2 at X-exp(-16) (i.e., machine precision ). I’ve solved my practical problem by writing that I would have liked to include results from a diffuse prior, but couldn’t. But I’m bothered by the poverty of my intuition. And since everything I’ve read says this is an issue of efficiency, rather than accuracy, I wonder if I could solve it just by running massive and heavily thinned chains. My reply: I can’t see how this could happen in a well-specified problem! I suspect it’s a bug. Otherwise try rescaling your variables so that your parameters will have values on the order of magnitude of 1. To which Margolis responded: I hardly wrote any of the code, so I can’t speak to the bug question — it’s binomial kriging from the R package geoRglm. And there are no covariates to scale — just the zero and one
Introduction: I remember in 4th grade or so, the teacher would give us a list of vocabulary words each week and we’d have to show we learned them by using each in a sentence. We quickly got bored and decided to do the assignment by writing a single sentence using all ten words. (Which the teacher hated, of course.) The above headline is in that spirit, combining blog posts rather than vocabulary words. But that only uses two of the entries. To really do the job, I’d need to throw in bivariate associations, ecological fallacies, high-dimensional feature selection, statistical significance, the suddenly unpopular name Hilary, snotty reviewers, the contagion of obesity, and milk-related spam. Or we could bring in some of the all-time favorites, such as Bayesians, economists, Finland, beautiful parents and their daughters, goofy graphics, red and blue states, essentialism in children’s reasoning, chess running, and zombies. Putting 8 of these in a single sentence (along with Glenn Hubbard
6 0.94901145 407 andrew gelman stats-2010-11-11-Data Visualization vs. Statistical Graphics
7 0.94455123 459 andrew gelman stats-2010-12-09-Solve mazes by starting at the exit
8 0.93955022 1991 andrew gelman stats-2013-08-21-BDA3 table of contents (also a new paper on visualization)
9 0.93935311 1160 andrew gelman stats-2012-02-09-Familial Linkage between Neuropsychiatric Disorders and Intellectual Interests
same-blog 10 0.93457925 1628 andrew gelman stats-2012-12-17-Statistics in a world where nothing is random
11 0.93356609 566 andrew gelman stats-2011-02-09-The boxer, the wrestler, and the coin flip, again
12 0.9312104 1320 andrew gelman stats-2012-05-14-Question 4 of my final exam for Design and Analysis of Sample Surveys
13 0.92639083 1855 andrew gelman stats-2013-05-13-Stan!
14 0.92633194 1477 andrew gelman stats-2012-08-30-Visualizing Distributions of Covariance Matrices
15 0.92531931 231 andrew gelman stats-2010-08-24-Yet another Bayesian job opportunity
16 0.91543615 1572 andrew gelman stats-2012-11-10-I don’t like this cartoon
17 0.91506827 1702 andrew gelman stats-2013-02-01-Don’t let your standard errors drive your research agenda
18 0.9141739 623 andrew gelman stats-2011-03-21-Baseball’s greatest fielders
19 0.90899456 850 andrew gelman stats-2011-08-11-Understanding how estimates change when you move to a multilevel model
20 0.9081971 1903 andrew gelman stats-2013-06-17-Weak identification provides partial information