andrew_gelman_stats andrew_gelman_stats-2014 andrew_gelman_stats-2014-2359 knowledge-graph by maker-knowledge-mining
Source: html
Introduction: Statisticians take tours in other people’s data. All methods of statistical inference rest on statistical models. Experiments typically have problems with compliance, measurement error, generalizability to the real world, and representativeness of the sample. Surveys typically have problems of undercoverage, nonresponse, and measurement error. Real surveys are done to learn about the general population. But real surveys are not random samples. For another example, consider educational tests: what are they exactly measuring? Nobody knows. Medical research: even if it’s a randomized experiment, the participants in the study won’t be a random sample from the population for whom you’d recommend treatment. You don’t need random sampling to generalize the results of a medical experiment to the general population but you need some substantive theory to make the assumption that effects in your nonrepresentative sample of people will be similar to effects in the population of interest. Ve
sentIndex sentText sentNum sentScore
1 All methods of statistical inference rest on statistical models. [sent-2, score-0.335]
2 Experiments typically have problems with compliance, measurement error, generalizability to the real world, and representativeness of the sample. [sent-3, score-0.628]
3 Surveys typically have problems of undercoverage, nonresponse, and measurement error. [sent-4, score-0.301]
4 Real surveys are done to learn about the general population. [sent-5, score-0.263]
5 For another example, consider educational tests: what are they exactly measuring? [sent-7, score-0.179]
6 Medical research: even if it’s a randomized experiment, the participants in the study won’t be a random sample from the population for whom you’d recommend treatment. [sent-9, score-0.599]
7 You don’t need random sampling to generalize the results of a medical experiment to the general population but you need some substantive theory to make the assumption that effects in your nonrepresentative sample of people will be similar to effects in the population of interest. [sent-10, score-1.738]
8 Very rarely, the assumptions of a statistical model will be known to be correct. [sent-11, score-0.355]
9 For example, we had a spreadsheet with a list of a few thousand legal files and we took a random sample of 600. [sent-13, score-0.812]
10 The sample files were examined and then we used these to get inference for the full population. [sent-14, score-0.529]
11 This doesn’t happen in surveys of people because we have nonavailability, nonresponse, and shifting sampling frames. [sent-15, score-0.606]
12 But in rare cases we are sampling documents and the statistical theory is exactly correct. [sent-16, score-0.692]
13 Textbook statistical theory is like the physics in an introductory mechanics text that assumes zero friction etc. [sent-17, score-0.838]
14 Friction can be modeled but that turns out to be a bit “phenomenological,” that is approximate. [sent-18, score-0.084]
15 Models are great and there’s no reason to be embarrassed about them. [sent-19, score-0.095]
16 Assumptions are the levers that allow us to move the world. [sent-20, score-0.131]
17 Statisticians take tours in other people’s data. [sent-21, score-0.288]
18 Assumptions about the underlying world + assumptions about the data collection process + the data themselves -> inferences about the world. [sent-22, score-0.42]
wordName wordTfidf (topN-words)
[('friction', 0.288), ('tours', 0.288), ('surveys', 0.263), ('assumptions', 0.229), ('random', 0.199), ('nonresponse', 0.19), ('files', 0.181), ('sample', 0.177), ('sampling', 0.17), ('population', 0.148), ('theory', 0.133), ('levers', 0.131), ('nonrepresentative', 0.131), ('measurement', 0.129), ('statistical', 0.126), ('medical', 0.121), ('experiment', 0.119), ('world', 0.119), ('phenomenological', 0.118), ('real', 0.111), ('representativeness', 0.111), ('compliance', 0.111), ('statisticians', 0.107), ('mechanics', 0.105), ('generalizability', 0.105), ('assumes', 0.103), ('exactly', 0.101), ('shifting', 0.101), ('typically', 0.1), ('embarrassed', 0.095), ('documents', 0.093), ('spreadsheet', 0.093), ('examined', 0.088), ('modeled', 0.084), ('generalize', 0.084), ('inference', 0.083), ('introductory', 0.083), ('thousand', 0.082), ('effects', 0.081), ('legal', 0.08), ('measuring', 0.079), ('educational', 0.078), ('textbook', 0.077), ('rarely', 0.075), ('randomized', 0.075), ('substantive', 0.074), ('collection', 0.072), ('problems', 0.072), ('people', 0.072), ('rare', 0.069)]
simIndex simValue blogId blogTitle
same-blog 1 1.0000001 2359 andrew gelman stats-2014-06-04-All the Assumptions That Are My Life
Introduction: Statisticians take tours in other people’s data. All methods of statistical inference rest on statistical models. Experiments typically have problems with compliance, measurement error, generalizability to the real world, and representativeness of the sample. Surveys typically have problems of undercoverage, nonresponse, and measurement error. Real surveys are done to learn about the general population. But real surveys are not random samples. For another example, consider educational tests: what are they exactly measuring? Nobody knows. Medical research: even if it’s a randomized experiment, the participants in the study won’t be a random sample from the population for whom you’d recommend treatment. You don’t need random sampling to generalize the results of a medical experiment to the general population but you need some substantive theory to make the assumption that effects in your nonrepresentative sample of people will be similar to effects in the population of interest. Ve
2 0.21879981 1628 andrew gelman stats-2012-12-17-Statistics in a world where nothing is random
Introduction: Rama Ganesan writes: I think I am having an existential crisis. I used to work with animals (rats, mice, gerbils etc.) Then I started to work in marketing research where we did have some kind of random sampling procedure. So up until a few years ago, I was sort of okay. Now I am teaching marketing research, and I feel like there is no real random sampling anymore. I take pains to get students to understand what random means, and then the whole lot of inferential statistics. Then almost anything they do – the sample is not random. They think I am contradicting myself. They use convenience samples at every turn – for their school work, and the enormous amount on online surveying that gets done. Do you have any suggestions for me? Other than say, something like this . My reply: Statistics does not require randomness. The three essential elements of statistics are measurement, comparison, and variation. Randomness is one way to supply variation, and it’s one way to model
Introduction: Elias Bareinboim asked what I thought about his comment on selection bias in which he referred to a paper by himself and Judea Pearl, “Controlling Selection Bias in Causal Inference.” I replied that I have no problem with what he wrote, but that from my perspective I find it easier to conceptualize such problems in terms of multilevel models. I elaborated on that point in a recent post , “Hierarchical modeling as a framework for extrapolation,” which I think was read by only a few people (I say this because it received only two comments). I don’t think Bareinboim objected to anything I wrote, but like me he is comfortable working within his own framework. He wrote the following to me: In some sense, “not ad hoc” could mean logically consistent. In other words, if one agrees with the assumptions encoded in the model, one must also agree with the conclusions entailed by these assumptions. I am not aware of any other way of doing mathematics. As it turns out, to get causa
Introduction: This material should be familiar to many of you but could be helpful to newcomers. Pearl writes: ALL causal conclusions in nonexperimental settings must be based on untested, judgmental assumptions that investigators are prepared to defend on scientific grounds. . . . To understand what the world should be like for a given procedure to work is of no lesser scientific value than seeking evidence for how the world works . . . Assumptions are self-destructive in their honesty. The more explicit the assumption, the more criticism it invites . . . causal diagrams invite the harshest criticism because they make assumptions more explicit and more transparent than other representation schemes. As regular readers know (for example, search this blog for “Pearl”), I have not got much out of the causal-diagrams approach myself, but in general I think that when there are multiple, mathematically equivalent methods of getting the same answer, we tend to go with the framework we are used
5 0.15534404 1165 andrew gelman stats-2012-02-13-Philosophy of Bayesian statistics: my reactions to Wasserman
Introduction: Continuing with my discussion of the articles in the special issue of the journal Rationality, Markets and Morals on the philosophy of Bayesian statistics: Larry Wasserman, “Low Assumptions, High Dimensions”: This article was refreshing to me because it was so different from anything I’ve seen before. Larry works in a statistics department and I work in a statistics department but there’s so little overlap in what we do. Larry and I both work in high dimesions (maybe his dimensions are higher than mine, but a few thousand dimensions seems like a lot to me!), but there the similarity ends. His article is all about using few to no assumptions, while I use assumptions all the time. Here’s an example. Larry writes: P. Laurie Davies (and his co-workers) have written several interesting papers where probability models, at least in the sense that we usually use them, are eliminated. Data are treated as deterministic. One then looks for adequate models rather than true mode
7 0.15460964 662 andrew gelman stats-2011-04-15-Bayesian statistical pragmatism
8 0.14691924 1149 andrew gelman stats-2012-02-01-Philosophy of Bayesian statistics: my reactions to Cox and Mayo
9 0.14186493 1289 andrew gelman stats-2012-04-29-We go to war with the data we have, not the data we want
10 0.13725501 1315 andrew gelman stats-2012-05-12-Question 2 of my final exam for Design and Analysis of Sample Surveys
11 0.13558172 603 andrew gelman stats-2011-03-07-Assumptions vs. conditions, part 2
12 0.12877809 1898 andrew gelman stats-2013-06-14-Progress! (on the understanding of the role of randomization in Bayesian inference)
13 0.12746224 1371 andrew gelman stats-2012-06-07-Question 28 of my final exam for Design and Analysis of Sample Surveys
14 0.12743649 1469 andrew gelman stats-2012-08-25-Ways of knowing
16 0.12704009 820 andrew gelman stats-2011-07-25-Design of nonrandomized cluster sample study
17 0.12653206 1455 andrew gelman stats-2012-08-12-Probabilistic screening to get an approximate self-weighted sample
18 0.12383892 85 andrew gelman stats-2010-06-14-Prior distribution for design effects
19 0.12208606 383 andrew gelman stats-2010-10-31-Analyzing the entire population rather than a sample
20 0.12031886 107 andrew gelman stats-2010-06-24-PPS in Georgia
topicId topicWeight
[(0, 0.189), (1, 0.085), (2, 0.039), (3, -0.113), (4, 0.019), (5, 0.055), (6, -0.076), (7, 0.041), (8, 0.068), (9, -0.064), (10, -0.048), (11, -0.075), (12, 0.009), (13, 0.021), (14, -0.042), (15, -0.039), (16, -0.065), (17, -0.017), (18, 0.022), (19, 0.003), (20, -0.029), (21, -0.099), (22, -0.008), (23, 0.108), (24, -0.056), (25, 0.045), (26, -0.053), (27, 0.109), (28, 0.034), (29, 0.087), (30, -0.035), (31, -0.03), (32, -0.035), (33, 0.053), (34, -0.036), (35, 0.021), (36, -0.069), (37, -0.012), (38, -0.082), (39, 0.047), (40, 0.051), (41, 0.007), (42, 0.044), (43, 0.007), (44, -0.023), (45, -0.023), (46, -0.041), (47, -0.002), (48, -0.008), (49, -0.014)]
simIndex simValue blogId blogTitle
same-blog 1 0.98071843 2359 andrew gelman stats-2014-06-04-All the Assumptions That Are My Life
Introduction: Statisticians take tours in other people’s data. All methods of statistical inference rest on statistical models. Experiments typically have problems with compliance, measurement error, generalizability to the real world, and representativeness of the sample. Surveys typically have problems of undercoverage, nonresponse, and measurement error. Real surveys are done to learn about the general population. But real surveys are not random samples. For another example, consider educational tests: what are they exactly measuring? Nobody knows. Medical research: even if it’s a randomized experiment, the participants in the study won’t be a random sample from the population for whom you’d recommend treatment. You don’t need random sampling to generalize the results of a medical experiment to the general population but you need some substantive theory to make the assumption that effects in your nonrepresentative sample of people will be similar to effects in the population of interest. Ve
2 0.83763117 1628 andrew gelman stats-2012-12-17-Statistics in a world where nothing is random
Introduction: Rama Ganesan writes: I think I am having an existential crisis. I used to work with animals (rats, mice, gerbils etc.) Then I started to work in marketing research where we did have some kind of random sampling procedure. So up until a few years ago, I was sort of okay. Now I am teaching marketing research, and I feel like there is no real random sampling anymore. I take pains to get students to understand what random means, and then the whole lot of inferential statistics. Then almost anything they do – the sample is not random. They think I am contradicting myself. They use convenience samples at every turn – for their school work, and the enormous amount on online surveying that gets done. Do you have any suggestions for me? Other than say, something like this . My reply: Statistics does not require randomness. The three essential elements of statistics are measurement, comparison, and variation. Randomness is one way to supply variation, and it’s one way to model
3 0.77454841 603 andrew gelman stats-2011-03-07-Assumptions vs. conditions, part 2
Introduction: In response to the discussion of his remarks on assumptions vs. conditions, Jeff Witmer writes : If [certain conditions hold] , then the t-test p-value gives a remarkably good approximation to “the real thing” — namely the randomization reference p-value. . . . I [Witmer] make assumptions about conditions that I cannot check, e.g., that the data arose from a random sample. Of course, just as there is no such thing as a normal population, there is no such thing as a random sample. I disagree strongly with both the above paragraphs! I say this not to pick a fight with Jeff Witmer but to illustrate how, in statistics, even the most basic points that people take for granted, can’t be. Let’s take the claims in order: 1. The purpose of a t test is to approximate the randomization p-value. Not to me. In my world, the purpose of t tests and intervals is to summarize uncertainty in estimates and comparisons. I don’t care about a p-value and almost certainly don’t care a
4 0.74913663 107 andrew gelman stats-2010-06-24-PPS in Georgia
Introduction: Lucy Flynn writes: I’m working at a non-profit organization called CRRC in the Republic of Georgia. I’m having a methodological problem and I saw the syllabus for your sampling class online and thought I might be able to ask you about it? We do a lot of complex surveys nationwide; our typical sample design is as follows: - stratify by rural/urban/capital - sub-stratify the rural and urban strata into NE/NW/SE/SW geographic quadrants - select voting precincts as PSUs - select households as SSUs - select individual respondents as TSUs I’m relatively new here, and past practice has been to sample voting precincts with probability proportional to size. It’s desirable because it’s not logistically feasible for us to vary the number of interviews per precinct with precinct size, so it makes the selection probabilities for households more even across precinct sizes. However, I have a complex sampling textbook (Lohr 1999), and it explains how complex it is to calculate sel
5 0.73933893 1455 andrew gelman stats-2012-08-12-Probabilistic screening to get an approximate self-weighted sample
Introduction: Sharad had a survey sampling question: We’re trying to use mechanical turk to conduct some surveys, and have quickly discovered that turkers tend to be quite young. We’d really like a representative sample of the U.S., or at the least be able to recruit a diverse enough sample from turk that we can post-stratify to adjust the estimates. The approach we ended up taking is to pay turkers a small amount to answer a couple of screening questions (age & sex), and then probabilistically recruit individuals to complete the full survey (for more money) based on the estimated turk population parameters and our desired target distribution. We use rejection sampling, so the end result is that individuals who are invited to take the full survey look as if they came from a representative sample, at least in terms of age and sex. I’m wondering whether this sort of technique—a two step design in which participants are first screened and then probabilistically selected to mimic a target distributio
6 0.72936076 1289 andrew gelman stats-2012-04-29-We go to war with the data we have, not the data we want
7 0.72094077 1551 andrew gelman stats-2012-10-28-A convenience sample and selected treatments
8 0.71996844 1165 andrew gelman stats-2012-02-13-Philosophy of Bayesian statistics: my reactions to Wasserman
9 0.7161448 85 andrew gelman stats-2010-06-14-Prior distribution for design effects
10 0.70951033 820 andrew gelman stats-2011-07-25-Design of nonrandomized cluster sample study
11 0.69284576 972 andrew gelman stats-2011-10-25-How do you interpret standard errors from a regression fit to the entire population?
12 0.68546975 1979 andrew gelman stats-2013-08-13-Convincing Evidence
13 0.67504066 405 andrew gelman stats-2010-11-10-Estimation from an out-of-date census
14 0.67035365 1317 andrew gelman stats-2012-05-13-Question 3 of my final exam for Design and Analysis of Sample Surveys
16 0.64287043 1940 andrew gelman stats-2013-07-16-A poll that throws away data???
17 0.63361442 5 andrew gelman stats-2010-04-27-Ethical and data-integrity problems in a study of mortality in Iraq
18 0.62937814 51 andrew gelman stats-2010-05-26-If statistics is so significantly great, why don’t statisticians use statistics?
19 0.62782991 1362 andrew gelman stats-2012-06-03-Question 24 of my final exam for Design and Analysis of Sample Surveys
20 0.62733775 784 andrew gelman stats-2011-07-01-Weighting and prediction in sample surveys
topicId topicWeight
[(8, 0.012), (16, 0.041), (17, 0.096), (21, 0.015), (24, 0.195), (35, 0.022), (36, 0.011), (53, 0.013), (55, 0.013), (77, 0.019), (83, 0.015), (86, 0.044), (89, 0.058), (95, 0.015), (99, 0.335)]
simIndex simValue blogId blogTitle
1 0.97891343 1230 andrew gelman stats-2012-03-26-Further thoughts on nonparametric correlation measures
Introduction: Malka Gorfine, Ruth Heller, and Yair Heller write a comment on the paper of Reshef et al. that we discussed a few months ago. Just to remind you what’s going on here, here’s my quick summary from December: Reshef et al. propose a new nonlinear R-squared-like measure. Unlike R-squared, this new method depends on a tuning parameter that controls the level of discretization, in a “How long is the coast of Britain” sort of way. The dependence on scale is inevitable for such a general method. Just consider: if you sample 1000 points from the unit bivariate normal distribution, (x,y) ~ N(0,I), you’ll be able to fit them perfectly by a 999-degree polynomial fit to the data. So the scale of the fit matters. The clever idea of the paper is that, instead of going for an absolute measure (which, as we’ve seen, will be scale-dependent), they focus on the problem of summarizing the grid of pairwise dependences in a large set of variables. As they put it: “Imagine a data set with hundreds
same-blog 2 0.97779864 2359 andrew gelman stats-2014-06-04-All the Assumptions That Are My Life
Introduction: Statisticians take tours in other people’s data. All methods of statistical inference rest on statistical models. Experiments typically have problems with compliance, measurement error, generalizability to the real world, and representativeness of the sample. Surveys typically have problems of undercoverage, nonresponse, and measurement error. Real surveys are done to learn about the general population. But real surveys are not random samples. For another example, consider educational tests: what are they exactly measuring? Nobody knows. Medical research: even if it’s a randomized experiment, the participants in the study won’t be a random sample from the population for whom you’d recommend treatment. You don’t need random sampling to generalize the results of a medical experiment to the general population but you need some substantive theory to make the assumption that effects in your nonrepresentative sample of people will be similar to effects in the population of interest. Ve
3 0.97128308 2314 andrew gelman stats-2014-05-01-Heller, Heller, and Gorfine on univariate and multivariate information measures
Introduction: Malka Gorfine writes: We noticed that the important topic of association measures and tests came up again in your blog, and we have few comments in this regard. It is useful to distinguish between the univariate and multivariate methods. A consistent multivariate method can recognise dependence between two vectors of random variables, while a univariate method can only loop over pairs of components and check for dependency between them. There are very few consistent multivariate methods. To the best of our knowledge there are three practical methods: 1) HSIC by Gretton et al. (http://www.gatsby.ucl.ac.uk/~gretton/papers/GreBouSmoSch05.pdf) 2) dcov by Szekely et al. (http://projecteuclid.org/euclid.aoas/1267453933) 3) A method we introduced in Heller et al (Biometrika, 2013, 503—510, http://biomet.oxfordjournals.org/content/early/2012/12/04/biomet.ass070.full.pdf+html, and an R package, HHG, is available as well http://cran.r-project.org/web/packages/HHG/index.html). A
4 0.96938622 1136 andrew gelman stats-2012-01-23-Fight! (also a bit of reminiscence at the end)
Introduction: Martin Lindquist and Michael Sobel published a fun little article in Neuroimage on models and assumptions for causal inference with intermediate outcomes. As their subtitle indicates (“A response to the comments on our comment”), this is a topic of some controversy. Lindquist and Sobel write: Our original comment (Lindquist and Sobel, 2011) made explicit the types of assumptions neuroimaging researchers are making when directed graphical models (DGMs), which include certain types of structural equation models (SEMs), are used to estimate causal effects. When these assumptions, which many researchers are not aware of, are not met, parameters of these models should not be interpreted as effects. . . . [Judea] Pearl does not disagree with anything we stated. However, he takes exception to our use of potential outcomes notation, which is the standard notation used in the statistical literature on causal inference, and his comment is devoted to promoting his alternative conventions. [C
5 0.96491039 1467 andrew gelman stats-2012-08-23-The pinch-hitter syndrome again
Introduction: Here . (Background here .) P.S. Just a thought: is it possible that one reason for the effectiveness of relief pitchers is that, by the end of the game, the starting players (that is, the hitters who have been playing all game) are getting tired? I’m pretty sure that lots of baseball-statistics experts will know the answer to this.
6 0.96456414 309 andrew gelman stats-2010-10-01-Why Development Economics Needs Theory?
7 0.96282053 2315 andrew gelman stats-2014-05-02-Discovering general multidimensional associations
8 0.96032858 259 andrew gelman stats-2010-09-06-Inbox zero. Really.
9 0.9595325 1557 andrew gelman stats-2012-11-01-‘Researcher Degrees of Freedom’
10 0.95885712 2089 andrew gelman stats-2013-11-04-Shlemiel the Software Developer and Unknown Unknowns
11 0.95821005 1702 andrew gelman stats-2013-02-01-Don’t let your standard errors drive your research agenda
12 0.9577167 1422 andrew gelman stats-2012-07-20-Likelihood thresholds and decisions
13 0.95758641 2324 andrew gelman stats-2014-05-07-Once more on nonparametric measures of mutual information
15 0.95688939 1903 andrew gelman stats-2013-06-17-Weak identification provides partial information
16 0.95613396 2090 andrew gelman stats-2013-11-05-How much do we trust a new claim that early childhood stimulation raised earnings by 42%?
17 0.95536864 1946 andrew gelman stats-2013-07-19-Prior distributions on derived quantities rather than on parameters themselves
19 0.9536913 1661 andrew gelman stats-2013-01-08-Software is as software does
20 0.95349777 1473 andrew gelman stats-2012-08-28-Turing chess run update