andrew_gelman_stats andrew_gelman_stats-2011 andrew_gelman_stats-2011-603 knowledge-graph by maker-knowledge-mining
Source: html
Introduction: In response to the discussion of his remarks on assumptions vs. conditions, Jeff Witmer writes : If [certain conditions hold] , then the t-test p-value gives a remarkably good approximation to “the real thing” — namely the randomization reference p-value. . . . I [Witmer] make assumptions about conditions that I cannot check, e.g., that the data arose from a random sample. Of course, just as there is no such thing as a normal population, there is no such thing as a random sample. I disagree strongly with both the above paragraphs! I say this not to pick a fight with Jeff Witmer but to illustrate how, in statistics, even the most basic points that people take for granted, can’t be. Let’s take the claims in order: 1. The purpose of a t test is to approximate the randomization p-value. Not to me. In my world, the purpose of t tests and intervals is to summarize uncertainty in estimates and comparisons. I don’t care about a p-value and almost certainly don’t care a
sentIndex sentText sentNum sentScore
1 In response to the discussion of his remarks on assumptions vs. [sent-1, score-0.218]
2 conditions, Jeff Witmer writes : If [certain conditions hold] , then the t-test p-value gives a remarkably good approximation to “the real thing” — namely the randomization reference p-value. [sent-2, score-1.113]
3 I [Witmer] make assumptions about conditions that I cannot check, e. [sent-6, score-0.355]
4 Of course, just as there is no such thing as a normal population, there is no such thing as a random sample. [sent-9, score-0.628]
5 I disagree strongly with both the above paragraphs! [sent-10, score-0.138]
6 I say this not to pick a fight with Jeff Witmer but to illustrate how, in statistics, even the most basic points that people take for granted, can’t be. [sent-11, score-0.449]
7 The purpose of a t test is to approximate the randomization p-value. [sent-13, score-0.679]
8 In my world, the purpose of t tests and intervals is to summarize uncertainty in estimates and comparisons. [sent-15, score-0.462]
9 I don’t care about a p-value and almost certainly don’t care about a randomization distribution. [sent-16, score-0.73]
10 I’m not saying this isn’t important, I just don’t think it’s particularly fundamental. [sent-17, score-0.057]
11 One might as well say that the randomization p-value is a way of approximating the ultimate goal which is the confidence interval. [sent-18, score-0.921]
12 Well, actually it was a few months ago, but still. [sent-22, score-0.066]
13 It was a sample of records to examine for a court case. [sent-23, score-0.392]
wordName wordTfidf (topN-words)
[('witmer', 0.453), ('randomization', 0.437), ('random', 0.286), ('conditions', 0.223), ('drew', 0.219), ('purpose', 0.156), ('jeff', 0.155), ('approximating', 0.136), ('thing', 0.136), ('assumptions', 0.132), ('granted', 0.127), ('remarkably', 0.119), ('care', 0.118), ('namely', 0.103), ('sample', 0.102), ('court', 0.1), ('fight', 0.098), ('ultimate', 0.096), ('approximation', 0.096), ('examine', 0.095), ('records', 0.095), ('summarize', 0.094), ('arose', 0.092), ('paragraphs', 0.091), ('remarks', 0.086), ('approximate', 0.086), ('illustrate', 0.085), ('intervals', 0.08), ('take', 0.077), ('reference', 0.076), ('hold', 0.075), ('confidence', 0.072), ('disagree', 0.071), ('pick', 0.07), ('normal', 0.07), ('strongly', 0.067), ('months', 0.066), ('uncertainty', 0.066), ('hey', 0.066), ('tests', 0.066), ('well', 0.063), ('certain', 0.061), ('basic', 0.061), ('check', 0.06), ('goal', 0.059), ('gives', 0.059), ('order', 0.058), ('say', 0.058), ('particularly', 0.057), ('certainly', 0.057)]
simIndex simValue blogId blogTitle
same-blog 1 0.99999994 603 andrew gelman stats-2011-03-07-Assumptions vs. conditions, part 2
Introduction: In response to the discussion of his remarks on assumptions vs. conditions, Jeff Witmer writes : If [certain conditions hold] , then the t-test p-value gives a remarkably good approximation to “the real thing” — namely the randomization reference p-value. . . . I [Witmer] make assumptions about conditions that I cannot check, e.g., that the data arose from a random sample. Of course, just as there is no such thing as a normal population, there is no such thing as a random sample. I disagree strongly with both the above paragraphs! I say this not to pick a fight with Jeff Witmer but to illustrate how, in statistics, even the most basic points that people take for granted, can’t be. Let’s take the claims in order: 1. The purpose of a t test is to approximate the randomization p-value. Not to me. In my world, the purpose of t tests and intervals is to summarize uncertainty in estimates and comparisons. I don’t care about a p-value and almost certainly don’t care a
Introduction: Leading theoretical statistician Larry Wassserman in 2008 : Some of the greatest contributions of statistics to science involve adding additional randomness and leveraging that randomness. Examples are randomized experiments, permutation tests, cross-validation and data-splitting. These are unabashedly frequentist ideas and, while one can strain to fit them into a Bayesian framework, they don’t really have a place in Bayesian inference. The fact that Bayesian methods do not naturally accommodate such a powerful set of statistical ideas seems like a serious deficiency. To which I responded on the second-to-last paragraph of page 8 here . Larry Wasserman in 2013 : Some people say that there is no role for randomization in Bayesian inference. In other words, the randomization mechanism plays no role in Bayes’ theorem. But this is not really true. Without randomization, we can indeed derive a posterior for theta but it is highly sensitive to the prior. This is just a restat
3 0.17499557 602 andrew gelman stats-2011-03-06-Assumptions vs. conditions
Introduction: Jeff Witmer writes: I noticed that you continue the standard practice in statistics of referring to assumptions; e.g. a blog entry on 2/4/11 at 10:54: “Our method, just like any model, relies on assumptions which we have the duty to state and to check.” I’m in the 6th year of a three-year campaign to get statisticians to drop the word “assumptions” and replace it with “conditions.” The problem, as I see it, is that people tend to think that an assumption is something that one assumes, as in “assuming that we have a right triangle…” or “assuming that k is even…” when constructing a mathematical proof. But in statistics we don’t assume things — unless we have to. Instead, we know that, for example, the validity of a t-test depends on normality, which is a condition that can and should be checked. Let’s not call normality an assumption, lest we imply that it is something that can be assumed. Let’s call it a condition. What do you all think?
4 0.1548638 1241 andrew gelman stats-2012-04-02-Fixed effects and identification
Introduction: Tom Clark writes: Drew Linzer and I [Tom] have been working on a paper about the use of modeled (“random”) and unmodeled (“fixed”) effects. Not directly in response to the paper, but in conversations about the topic over the past few months, several people have said to us things to the effect of “I prefer fixed effects over random effects because I care about identification.” Neither Drew nor I has any idea what this comment is supposed to mean. Have you come across someone saying something like this? Do you have any thoughts about what these people could possibly mean? I want to respond to this concern when people raise it, but I have failed thus far to inquire what is meant and so do not know what to say. My reply: I have a “cultural” reply, which is that so-called fixed effects are thought to make fewer assumptions, and making fewer assumptions is considered a generally good thing that serious people do, and identification is considered a concern of serious people, so they g
5 0.13736227 1628 andrew gelman stats-2012-12-17-Statistics in a world where nothing is random
Introduction: Rama Ganesan writes: I think I am having an existential crisis. I used to work with animals (rats, mice, gerbils etc.) Then I started to work in marketing research where we did have some kind of random sampling procedure. So up until a few years ago, I was sort of okay. Now I am teaching marketing research, and I feel like there is no real random sampling anymore. I take pains to get students to understand what random means, and then the whole lot of inferential statistics. Then almost anything they do – the sample is not random. They think I am contradicting myself. They use convenience samples at every turn – for their school work, and the enormous amount on online surveying that gets done. Do you have any suggestions for me? Other than say, something like this . My reply: Statistics does not require randomness. The three essential elements of statistics are measurement, comparison, and variation. Randomness is one way to supply variation, and it’s one way to model
6 0.13558172 2359 andrew gelman stats-2014-06-04-All the Assumptions That Are My Life
7 0.11406814 317 andrew gelman stats-2010-10-04-Rob Kass on statistical pragmatism, and my reactions
9 0.10505033 2258 andrew gelman stats-2014-03-21-Random matrices in the news
10 0.10459013 662 andrew gelman stats-2011-04-15-Bayesian statistical pragmatism
11 0.096483812 1203 andrew gelman stats-2012-03-08-John Dalton’s Stroop test
12 0.087346919 427 andrew gelman stats-2010-11-23-Bayesian adaptive methods for clinical trials
13 0.087042958 1267 andrew gelman stats-2012-04-17-Hierarchical-multilevel modeling with “big data”
14 0.085920356 1206 andrew gelman stats-2012-03-10-95% intervals that I don’t believe, because they’re from a flat prior I don’t believe
15 0.085684717 1383 andrew gelman stats-2012-06-18-Hierarchical modeling as a framework for extrapolation
16 0.085554019 972 andrew gelman stats-2011-10-25-How do you interpret standard errors from a regression fit to the entire population?
17 0.085117504 480 andrew gelman stats-2010-12-21-Instead of “confidence interval,” let’s say “uncertainty interval”
18 0.082876571 1726 andrew gelman stats-2013-02-18-What to read to catch up on multivariate statistics?
19 0.082696937 2152 andrew gelman stats-2013-12-28-Using randomized incentives as an instrument for survey nonresponse?
20 0.080529064 269 andrew gelman stats-2010-09-10-R vs. Stata, or, Different ways to estimate multilevel models
topicId topicWeight
[(0, 0.129), (1, 0.026), (2, 0.043), (3, -0.04), (4, 0.007), (5, 0.007), (6, -0.001), (7, 0.042), (8, 0.047), (9, -0.066), (10, -0.022), (11, -0.026), (12, 0.009), (13, 0.002), (14, -0.019), (15, -0.005), (16, -0.059), (17, 0.001), (18, 0.008), (19, -0.02), (20, 0.01), (21, -0.019), (22, 0.024), (23, 0.047), (24, 0.003), (25, -0.013), (26, -0.054), (27, 0.065), (28, 0.012), (29, 0.101), (30, 0.005), (31, -0.044), (32, -0.035), (33, -0.016), (34, -0.014), (35, 0.022), (36, -0.071), (37, 0.025), (38, -0.013), (39, 0.009), (40, 0.095), (41, -0.063), (42, 0.052), (43, -0.02), (44, -0.034), (45, -0.003), (46, -0.021), (47, 0.004), (48, 0.018), (49, -0.019)]
simIndex simValue blogId blogTitle
same-blog 1 0.9752053 603 andrew gelman stats-2011-03-07-Assumptions vs. conditions, part 2
Introduction: In response to the discussion of his remarks on assumptions vs. conditions, Jeff Witmer writes : If [certain conditions hold] , then the t-test p-value gives a remarkably good approximation to “the real thing” — namely the randomization reference p-value. . . . I [Witmer] make assumptions about conditions that I cannot check, e.g., that the data arose from a random sample. Of course, just as there is no such thing as a normal population, there is no such thing as a random sample. I disagree strongly with both the above paragraphs! I say this not to pick a fight with Jeff Witmer but to illustrate how, in statistics, even the most basic points that people take for granted, can’t be. Let’s take the claims in order: 1. The purpose of a t test is to approximate the randomization p-value. Not to me. In my world, the purpose of t tests and intervals is to summarize uncertainty in estimates and comparisons. I don’t care about a p-value and almost certainly don’t care a
2 0.73214364 2359 andrew gelman stats-2014-06-04-All the Assumptions That Are My Life
Introduction: Statisticians take tours in other people’s data. All methods of statistical inference rest on statistical models. Experiments typically have problems with compliance, measurement error, generalizability to the real world, and representativeness of the sample. Surveys typically have problems of undercoverage, nonresponse, and measurement error. Real surveys are done to learn about the general population. But real surveys are not random samples. For another example, consider educational tests: what are they exactly measuring? Nobody knows. Medical research: even if it’s a randomized experiment, the participants in the study won’t be a random sample from the population for whom you’d recommend treatment. You don’t need random sampling to generalize the results of a medical experiment to the general population but you need some substantive theory to make the assumption that effects in your nonrepresentative sample of people will be similar to effects in the population of interest. Ve
3 0.73213583 1628 andrew gelman stats-2012-12-17-Statistics in a world where nothing is random
Introduction: Rama Ganesan writes: I think I am having an existential crisis. I used to work with animals (rats, mice, gerbils etc.) Then I started to work in marketing research where we did have some kind of random sampling procedure. So up until a few years ago, I was sort of okay. Now I am teaching marketing research, and I feel like there is no real random sampling anymore. I take pains to get students to understand what random means, and then the whole lot of inferential statistics. Then almost anything they do – the sample is not random. They think I am contradicting myself. They use convenience samples at every turn – for their school work, and the enormous amount on online surveying that gets done. Do you have any suggestions for me? Other than say, something like this . My reply: Statistics does not require randomness. The three essential elements of statistics are measurement, comparison, and variation. Randomness is one way to supply variation, and it’s one way to model
Introduction: David Radwin asks a question which comes up fairly often in one form or another: How should one respond to requests for statistical hypothesis tests for population (or universe) data? I [Radwin] first encountered this issue as an undergraduate when a professor suggested a statistical significance test for my paper comparing roll call votes between freshman and veteran members of Congress. Later I learned that such tests apply only to samples because their purpose is to tell you whether the difference in the observed sample is likely to exist in the population. If you have data for the whole population, like all members of the 103rd House of Representatives, you do not need a test to discern the true difference in the population. Sometimes researchers assume some sort of superpopulation like “all possible Congresses” or “Congresses across all time” and that the members of any given Congress constitute a sample. In my current work in education research, it is sometimes asserted t
5 0.66951311 1881 andrew gelman stats-2013-06-03-Boot
Introduction: Joshua Hartshorne writes: I ran several large-N experiments (separate participants) and looked at performance against age. What we want to do is compare age-of-peak-performance across the different tasks (again, different participants). We bootstrapped age-of-peak-performance. On each iteration, we sampled (with replacement) the X scores at each age, where X=num of participants at that age, and recorded the age at which performance peaked on that task. We then recorded the age at which performance was at peak and repeated. Once we had distributions of age-of-peak-performance, we used the means and SDs to calculate t-statistics to compare the results across different tasks. For graphical presentation, we used medians, interquartile ranges, and 95% confidence intervals (based on the distributions: the range within which 75% and 95% of the bootstrapped peaks appeared). While a number of people we consulted with thought this made a lot of sense, one reviewer of the paper insist
6 0.65718502 1165 andrew gelman stats-2012-02-13-Philosophy of Bayesian statistics: my reactions to Wasserman
7 0.64057833 1289 andrew gelman stats-2012-04-29-We go to war with the data we have, not the data we want
8 0.6277554 602 andrew gelman stats-2011-03-06-Assumptions vs. conditions
9 0.60694635 2155 andrew gelman stats-2013-12-31-No on Yes-No decisions
10 0.59796345 944 andrew gelman stats-2011-10-05-How accurate is your gaydar?
12 0.59711194 51 andrew gelman stats-2010-05-26-If statistics is so significantly great, why don’t statisticians use statistics?
13 0.59480888 2128 andrew gelman stats-2013-12-09-How to model distributions that have outliers in one direction
14 0.59020001 85 andrew gelman stats-2010-06-14-Prior distribution for design effects
15 0.5896042 269 andrew gelman stats-2010-09-10-R vs. Stata, or, Different ways to estimate multilevel models
16 0.58108336 107 andrew gelman stats-2010-06-24-PPS in Georgia
17 0.57951415 1206 andrew gelman stats-2012-03-10-95% intervals that I don’t believe, because they’re from a flat prior I don’t believe
18 0.57254112 1455 andrew gelman stats-2012-08-12-Probabilistic screening to get an approximate self-weighted sample
19 0.56932789 1551 andrew gelman stats-2012-10-28-A convenience sample and selected treatments
20 0.5680756 1527 andrew gelman stats-2012-10-10-Another reason why you can get good inferences from a bad model
topicId topicWeight
[(15, 0.024), (16, 0.051), (20, 0.021), (24, 0.215), (27, 0.014), (53, 0.015), (72, 0.019), (81, 0.017), (84, 0.027), (86, 0.014), (88, 0.146), (99, 0.317)]
simIndex simValue blogId blogTitle
1 0.96941566 569 andrew gelman stats-2011-02-12-Get the Data
Introduction: At GetTheData , you can ask and answer data related questions. Here’s a preview: I’m not sure a Q&A; site is the best way to do this. My pipe dream is to create a taxonomy of variables and instances, and collect spreadsheets annotated this way. Imagine doing a search of type: “give me datasets, where an instance is a person, the variables are age, gender and weight” – and out would come datasets, each one tagged with the descriptions of the variables that were held constant for the whole dataset (person_type=student, location=Columbia, time_of_study=1/1/2009, study_type=longitudinal). It would even be possible to automatically convert one variable into another, if it was necessary (like age = time_of_measurement-time_of_birth). Maybe the dream of Semantic Web will actually be implemented for relatively structured statistical data rather than much fuzzier “knowledge”, just consider the difficulties of developing a universal Freebase . Wolfram|Alpha is perhaps currently clos
same-blog 2 0.96348482 603 andrew gelman stats-2011-03-07-Assumptions vs. conditions, part 2
Introduction: In response to the discussion of his remarks on assumptions vs. conditions, Jeff Witmer writes : If [certain conditions hold] , then the t-test p-value gives a remarkably good approximation to “the real thing” — namely the randomization reference p-value. . . . I [Witmer] make assumptions about conditions that I cannot check, e.g., that the data arose from a random sample. Of course, just as there is no such thing as a normal population, there is no such thing as a random sample. I disagree strongly with both the above paragraphs! I say this not to pick a fight with Jeff Witmer but to illustrate how, in statistics, even the most basic points that people take for granted, can’t be. Let’s take the claims in order: 1. The purpose of a t test is to approximate the randomization p-value. Not to me. In my world, the purpose of t tests and intervals is to summarize uncertainty in estimates and comparisons. I don’t care about a p-value and almost certainly don’t care a
3 0.95623696 629 andrew gelman stats-2011-03-26-Is it plausible that 1% of people pick a career based on their first name?
Introduction: In my discussion of dentists-named-Dennis study, I referred to my back-of-the-envelope calculation that the effect (if it indeed exists) corresponds to an approximate 1% aggregate chance that you’ll pick a profession based on your first name. Even if there are nearly twice as many dentist Dennises as would be expected from chance alone, the base rate is so low that a shift of 1% of all Dennises would be enough to do this. My point was that (a) even a small effect could show up when looking at low-frequency events such as the choice to pick a particular career or live in a particular city, and (b) any small effects will inherently be difficult to detect in any direct way. Uri Simonsohn (the author of the recent rebuttal of the original name-choice article by Brett Pelham et al.) wrote: In terms of the effect size. I [Simonsohn] think of it differently and see it as too big to be believable. I don’t find it plausible that I can double the odds that my daughter will marry an
4 0.95596886 1174 andrew gelman stats-2012-02-18-Not as ugly as you look
Introduction: Kaiser asks the interesting question: How do you measure what restaurants are “overrated”? You can’t just ask people, right? There’s some sort of social element here, that “overrated” implies that someone’s out there doing the rating.
5 0.95139176 1403 andrew gelman stats-2012-07-02-Moving beyond hopeless graphics
Introduction: I was at a talk awhile ago where the speaker presented tables with 4, 5, 6, even 8 significant digits even though, as is usual, only the first or second digit of each number conveyed any useful information. A graph would be better, but even if you’re too lazy to make a plot, a bit of rounding would seem to be required. I mentioned this to a colleague, who responded: I don’t know how to stop this practice. Logic doesn’t work. Maybe ridicule? Best hope is the departure from field who do it. (Theories don’t die, but the people who follow those theories retire.) Another possibility, I think, is helpful software defaults. If we can get to the people who write the software, maybe we could have some impact. Once the software is written, however, it’s probably too late. I’m not far from the center of the R universe, but I don’t know if I’ll ever succeed in my goals of increasing the default number of histogram bars or reducing the default number of decimal places in regression
6 0.95131195 784 andrew gelman stats-2011-07-01-Weighting and prediction in sample surveys
8 0.94178742 1992 andrew gelman stats-2013-08-21-Workshop for Women in Machine Learning
9 0.93954426 400 andrew gelman stats-2010-11-08-Poli sci plagiarism update, and a note about the benefits of not caring
10 0.93805146 1098 andrew gelman stats-2012-01-04-Bayesian Page Rank?
11 0.93336803 1414 andrew gelman stats-2012-07-12-Steven Pinker’s unconvincing debunking of group selection
12 0.93087256 2365 andrew gelman stats-2014-06-09-I hate polynomials
13 0.92823935 2208 andrew gelman stats-2014-02-12-How to think about “identifiability” in Bayesian inference?
14 0.92735088 2109 andrew gelman stats-2013-11-21-Hidden dangers of noninformative priors
15 0.92729962 247 andrew gelman stats-2010-09-01-How does Bayes do it?
16 0.92614448 1241 andrew gelman stats-2012-04-02-Fixed effects and identification
17 0.92567015 970 andrew gelman stats-2011-10-24-Bell Labs
18 0.92525375 1941 andrew gelman stats-2013-07-16-Priors
20 0.92408937 2129 andrew gelman stats-2013-12-10-Cross-validation and Bayesian estimation of tuning parameters