andrew_gelman_stats andrew_gelman_stats-2011 andrew_gelman_stats-2011-738 knowledge-graph by maker-knowledge-mining
Source: html
Introduction: John Cook discusses the John Tukey quote, “The test of a good procedure is how well it works, not how well it is understood.” Cook writes: At some level, it’s hard to argue against this. Statistical procedures operate on empirical data, so it makes sense that the procedures themselves be evaluated empirically. But I [Cook] question whether we really know that a statistical procedure works well if it isn’t well understood. Specifically, I’m skeptical of complex statistical methods whose only credentials are a handful of simulations. “We don’t have any theoretical results, buy hey, it works well in practice. Just look at the simulations.” Every method works well on the scenarios its author publishes, almost by definition. If the method didn’t handle a scenario well, the author would publish a different scenario. I agree with Cook but would give a slightly different emphasis. I’d say that a lot of methods can work when they are done well. See the second meta-principle liste
sentIndex sentText sentNum sentScore
1 John Cook discusses the John Tukey quote, “The test of a good procedure is how well it works, not how well it is understood. [sent-1, score-0.571]
2 Statistical procedures operate on empirical data, so it makes sense that the procedures themselves be evaluated empirically. [sent-3, score-0.67]
3 But I [Cook] question whether we really know that a statistical procedure works well if it isn’t well understood. [sent-4, score-0.856]
4 Specifically, I’m skeptical of complex statistical methods whose only credentials are a handful of simulations. [sent-5, score-0.58]
5 “We don’t have any theoretical results, buy hey, it works well in practice. [sent-6, score-0.539]
6 ” Every method works well on the scenarios its author publishes, almost by definition. [sent-8, score-0.829]
7 If the method didn’t handle a scenario well, the author would publish a different scenario. [sent-9, score-0.459]
8 I’d say that a lot of methods can work when they are done well. [sent-11, score-0.227]
9 See the second meta-principle listed in my discussion of Efron from last year. [sent-12, score-0.159]
10 The short story is: lots of methods can work well if you’re Tukey. [sent-13, score-0.445]
11 I also think statisticians are overly impressed by the appreciation of their scientific collaborators. [sent-16, score-0.282]
12 Just cos a Nobel-winning biologist or physicist or whatever thinks your method is great, it doesn’t mean your method is in itself great. [sent-17, score-0.735]
13 If Brad Efron or Don Rubin had come through the door bringing their methods, Mister Nobel Prize would probably have loved them too. [sent-18, score-0.277]
14 Second, and back to the original quote above, Tukey was notorious for developing methods that were based on theoretical models and then rubbing out the traces of the theory and presenting the methods alone. [sent-19, score-0.98]
15 For example, the hanging rootogram makes some sense–if you think of counts as following Poisson distributions. [sent-20, score-0.284]
16 This predilection of Tukey’s makes a certain philosophical sense (see my argument a few months ago) but I still find it a bit irritating to hide one’s traces even for the best of reasons. [sent-21, score-0.834]
wordName wordTfidf (topN-words)
[('cook', 0.333), ('tukey', 0.267), ('traces', 0.227), ('methods', 0.227), ('well', 0.218), ('works', 0.21), ('efron', 0.202), ('method', 0.19), ('procedures', 0.142), ('procedure', 0.135), ('predilection', 0.131), ('quote', 0.116), ('author', 0.112), ('theoretical', 0.111), ('operate', 0.11), ('appreciation', 0.105), ('hanging', 0.103), ('handful', 0.103), ('biologist', 0.103), ('credentials', 0.103), ('hide', 0.101), ('makes', 0.1), ('scenarios', 0.099), ('irritating', 0.099), ('nobel', 0.099), ('overly', 0.098), ('door', 0.096), ('publishes', 0.096), ('loved', 0.095), ('cos', 0.095), ('sense', 0.093), ('john', 0.092), ('brad', 0.092), ('mister', 0.091), ('poisson', 0.089), ('prize', 0.086), ('bringing', 0.086), ('physicist', 0.083), ('scenario', 0.083), ('evaluated', 0.083), ('philosophical', 0.083), ('second', 0.082), ('counts', 0.081), ('impressed', 0.079), ('listed', 0.077), ('statistical', 0.075), ('mean', 0.074), ('handle', 0.074), ('skeptical', 0.072), ('presenting', 0.072)]
simIndex simValue blogId blogTitle
same-blog 1 0.99999994 738 andrew gelman stats-2011-05-30-Works well versus well understood
Introduction: John Cook discusses the John Tukey quote, “The test of a good procedure is how well it works, not how well it is understood.” Cook writes: At some level, it’s hard to argue against this. Statistical procedures operate on empirical data, so it makes sense that the procedures themselves be evaluated empirically. But I [Cook] question whether we really know that a statistical procedure works well if it isn’t well understood. Specifically, I’m skeptical of complex statistical methods whose only credentials are a handful of simulations. “We don’t have any theoretical results, buy hey, it works well in practice. Just look at the simulations.” Every method works well on the scenarios its author publishes, almost by definition. If the method didn’t handle a scenario well, the author would publish a different scenario. I agree with Cook but would give a slightly different emphasis. I’d say that a lot of methods can work when they are done well. See the second meta-principle liste
2 0.21459849 496 andrew gelman stats-2011-01-01-Tukey’s philosophy
Introduction: The great statistician John Tukey, in his writings from the 1970s onward (and maybe earlier) was time and again making the implicit argument that you should evaluate a statistical method based on what it does; you should {\em not} be staring at the model that purportedly underlies the method, trying to determine if the model is “true” (or “true enough”). Tukey’s point was that models can be great to inspire methods, but the model is the scaffolding; it is the method that is the building you have to live in. I don’t fully agree with this philosophy–I think models are a good way to understand data and also often connect usefully to scientific models (although not as cleanly as is thought by our friends who work in economics or statistical hypothesis testing). To put it another way: What makes a building good? A building is good if it is useful. If a building is useful, people will use it. Eventually improvements will be needed, partly because the building will get worn down, part
3 0.19645916 1527 andrew gelman stats-2012-10-10-Another reason why you can get good inferences from a bad model
Introduction: John Cook considers how people justify probability distribution assumptions: Sometimes distribution assumptions are not justified. Sometimes distributions can be derived from fundamental principles [or] . . . on theoretical grounds. For example, large samples and the central limit theorem together may justify assuming that something is normally distributed. Often the choice of distribution is somewhat arbitrary, chosen by intuition or for convenience, and then empirically shown to work well enough. Sometimes a distribution can be a bad fit and still work well, depending on what you’re asking of it. Cook continues: The last point is particularly interesting. It’s not hard to imagine that a poor fit would produce poor results. It’s surprising when a poor fit produces good results. And then he gives an example of an effective but inaccurate model used to model survival times in a clinical trial. Cook explains: The [poorly-fitting] method works well because of the q
4 0.14232753 39 andrew gelman stats-2010-05-18-The 1.6 rule
Introduction: In ARM we discuss how you can go back and forth between logit and probit models by dividing by 1.6. Or, to put it another way, logistic regression corresponds to a latent-variable model with errors that are approximately normally distributed with mean 0 and standard deviation 1.6. (This is well known, it’s nothing original with our book.) Anyway, John Cook discusses the approximation here .
Introduction: Well put.
6 0.13777176 534 andrew gelman stats-2011-01-24-Bayes at the end
7 0.13450775 2151 andrew gelman stats-2013-12-27-Should statistics have a Nobel prize?
8 0.13442193 1469 andrew gelman stats-2012-08-25-Ways of knowing
9 0.13059679 1520 andrew gelman stats-2012-10-03-Advice that’s so eminently sensible but so difficult to follow
10 0.1287156 291 andrew gelman stats-2010-09-22-Philosophy of Bayes and non-Bayes: A dialogue with Deborah Mayo
11 0.1209963 109 andrew gelman stats-2010-06-25-Classics of statistics
12 0.12073404 1140 andrew gelman stats-2012-01-27-Educational monoculture
13 0.11181631 1560 andrew gelman stats-2012-11-03-Statistical methods that work in some settings but not others
14 0.11171425 1205 andrew gelman stats-2012-03-09-Coming to agreement on philosophy of statistics
15 0.1076897 1582 andrew gelman stats-2012-11-18-How to teach methods we don’t like?
17 0.10341382 1827 andrew gelman stats-2013-04-27-Continued fractions!!
18 0.099820927 1690 andrew gelman stats-2013-01-23-When are complicated models helpful in psychology research and when are they overkill?
20 0.098465934 1695 andrew gelman stats-2013-01-28-Economists argue about Bayes
topicId topicWeight
[(0, 0.192), (1, 0.019), (2, -0.052), (3, -0.001), (4, -0.04), (5, -0.012), (6, -0.033), (7, 0.023), (8, 0.054), (9, -0.021), (10, -0.008), (11, -0.031), (12, -0.007), (13, -0.036), (14, -0.019), (15, -0.02), (16, -0.03), (17, -0.001), (18, 0.004), (19, -0.009), (20, 0.022), (21, -0.029), (22, 0.016), (23, 0.061), (24, 0.038), (25, 0.008), (26, -0.012), (27, 0.029), (28, 0.002), (29, 0.038), (30, 0.035), (31, 0.079), (32, 0.092), (33, -0.014), (34, 0.031), (35, -0.034), (36, -0.0), (37, 0.03), (38, -0.045), (39, -0.021), (40, 0.011), (41, 0.034), (42, 0.012), (43, 0.007), (44, 0.058), (45, 0.027), (46, -0.014), (47, -0.012), (48, 0.026), (49, 0.064)]
simIndex simValue blogId blogTitle
same-blog 1 0.97633827 738 andrew gelman stats-2011-05-30-Works well versus well understood
Introduction: John Cook discusses the John Tukey quote, “The test of a good procedure is how well it works, not how well it is understood.” Cook writes: At some level, it’s hard to argue against this. Statistical procedures operate on empirical data, so it makes sense that the procedures themselves be evaluated empirically. But I [Cook] question whether we really know that a statistical procedure works well if it isn’t well understood. Specifically, I’m skeptical of complex statistical methods whose only credentials are a handful of simulations. “We don’t have any theoretical results, buy hey, it works well in practice. Just look at the simulations.” Every method works well on the scenarios its author publishes, almost by definition. If the method didn’t handle a scenario well, the author would publish a different scenario. I agree with Cook but would give a slightly different emphasis. I’d say that a lot of methods can work when they are done well. See the second meta-principle liste
2 0.78046685 241 andrew gelman stats-2010-08-29-Ethics and statistics in development research
Introduction: From Bannerjee and Duflo, “The Experimental Approach to Development Economics,” Annual Review of Economics (2009): One issue with the explicit acknowledgment of randomization as a fair way to allocate the program is that implementers may find that the easiest way to present it to the community is to say that an expansion of the program is planned for the control areas in the future (especially when such is indeed the case, as in phased-in design). I can’t quite figure out whether Bannerjee and Duflo are saying that they would lie and tell people that an expansion is planned when it isn’t, or whether they’re deploring that other people do it. I’m not bothered by a lot of the deception in experimental research–for example, I think the Milgram obedience experiment was just fine–but somehow the above deception bothers me. It just seems wrong to tell people that an expansion is planned if it’s not. P.S. Overall the article is pretty good. My only real problem with it is that
Introduction: For the past several months I’ve been circling around and around some questions related to the issue of how we build trust in statistical methods and statistical results. There are lots of examples but let me start with my own career. My most cited publications are my books and my methods papers, but I think that much of my credibility as a statistical researcher comes from my applied work. It somehow matters, I think, when judging my statistical work, that I’ve done (and continue to do) real research in social and environmental science. Why is this? It’s not just that my applied work gives me good examples for my textbooks. It’s also that the applied work motivated the new methods. Most of the successful theory and methods that my collaborators and I have developed, we developed in the context of trying to solve active applied problems. We weren’t trying to shave a half a point off the predictive error in the Boston housing data; rather, we were attacking new problems that we
4 0.74957001 2151 andrew gelman stats-2013-12-27-Should statistics have a Nobel prize?
Introduction: Xiao-Li says yes: The most compelling reason for having highly visible awards in any field is to enhance its ability to attract future talent. Virtually all the media and public attention our profession received in recent years has been on the utility of statistics in all walks of life. We are extremely happy for and proud of this recognition—it is long overdue. However, the media and public have given much more attention to the Fields Medal than to the COPSS Award, even though the former has hardly been about direct or even indirect impact on everyday life. Why this difference? . . . these awards arouse media and public interest by featuring how ingenious the awardees are and how difficult the problems they solved, much like how conquering Everest bestows admiration not because the admirers care or even know much about Everest itself but because it represents the ultimate physical feat. In this sense, the biggest winner of the Fields Medal is mathematics itself: enticing the brig
5 0.74475646 744 andrew gelman stats-2011-06-03-Statistical methods for healthcare regulation: rating, screening and surveillance
Introduction: Here is my discussion of a recent article by David Spiegelhalter, Christopher Sherlaw-Johnson, Martin Bardsley, Ian Blunt, Christopher Wood and Olivia Grigg, that is scheduled to appear in the Journal of the Royal Statistical Society: I applaud the authors’ use of a mix of statistical methods to attack an important real-world problem. Policymakers need results right away, and I admire the authors’ ability and willingness to combine several different modeling and significance testing ideas for the purposes of rating and surveillance. That said, I am uncomfortable with the statistical ideas here, for three reasons. First, I feel that the proposed methods, centered as they are around data manipulation and corrections for uncertainty, has serious defects compared to a more model-based approach. My problem with methods based on p-values and z-scores–however they happen to be adjusted–is that they draw discussion toward error rates, sequential analysis, and other technical statistical
6 0.73886263 1560 andrew gelman stats-2012-11-03-Statistical methods that work in some settings but not others
9 0.7149654 1859 andrew gelman stats-2013-05-16-How do we choose our default methods?
10 0.70870799 506 andrew gelman stats-2011-01-06-That silly ESP paper and some silliness in a rebuttal as well
11 0.70829654 1769 andrew gelman stats-2013-03-18-Tibshirani announces new research result: A significance test for the lasso
12 0.70753491 1645 andrew gelman stats-2012-12-31-Statistical modeling, causal inference, and social science
13 0.70666081 2127 andrew gelman stats-2013-12-08-The never-ending (and often productive) race between theory and practice
14 0.70201653 155 andrew gelman stats-2010-07-19-David Blackwell
15 0.69621789 496 andrew gelman stats-2011-01-01-Tukey’s philosophy
16 0.69321215 1861 andrew gelman stats-2013-05-17-Where do theories come from?
17 0.69311941 1848 andrew gelman stats-2013-05-09-A tale of two discussion papers
19 0.68544817 1564 andrew gelman stats-2012-11-06-Choose your default, or your default will choose you (election forecasting edition)
topicId topicWeight
[(1, 0.121), (2, 0.012), (8, 0.011), (16, 0.046), (24, 0.13), (27, 0.015), (30, 0.046), (36, 0.01), (38, 0.011), (53, 0.02), (65, 0.017), (76, 0.016), (84, 0.04), (86, 0.061), (99, 0.335)]
simIndex simValue blogId blogTitle
Introduction: We were having so much fun on this thread that I couldn’t resist linking to this news item by Adrian Chen. The good news is that Scott Adams (creater of the Dilbert comic strip) “has a certified genius IQ” and that he “can open jars with [his] bare hands.” He is also “able to lift heavy objects.” Cool! In all seriousness, I knew nothing about this aspect of Adams when I wrote the earlier blog. I was just surprised (and remain surprised) that he was so impressed with Charlie Sheen for being good-looking and being able to remember his lines. At the time I thought it was just a matter of Adams being overly-influenced by his direct experience, along with some satisfaction in separating himself from the general mass of Sheen-haters out there. But now I wonder if something more is going on, that maybe he feels that he and Sheen are on the same side in a culture war. In any case, the ultimate topic of interest here is not Sheen or Adams but rather more general questions of what
2 0.95757687 697 andrew gelman stats-2011-05-05-A statistician rereads Bill James
Introduction: Ben Lindbergh invited me to write an article for Baseball Prospectus. I first sent him this item on the differences between baseball and politics but he said it was too political for them. I then sent him this review of a book on baseball’s greatest fielders but he said they already had someone slotted to review that book. Then I sent him some reflections on the great Bill James and he published it ! If anybody out there knows Bill James, please send this on to him: I have some questions at the end that I’m curious about. Here’s how it begins: I read my first Bill James book in 1984, took my first statistics class in 1985, and began graduate study in statistics the next year. Besides giving me the opportunity to study with the best applied statistician of the late 20th century (Don Rubin) and the best theoretical statistician of the early 21st (Xiao-Li Meng), going to graduate school at Harvard in 1986 gave me the opportunity to sit in a basement room one evening that
3 0.95599383 973 andrew gelman stats-2011-10-26-Antman again courts controversy
Introduction: Commenter Zbicyclist links to a fun article by Howard French on biologist E. O. Wilson: Wilson announced that his new book may be his last. It is not limited to the discussion of evolutionary biology, but ranges provocatively through the humanities, as well. . . . Generation after generation of students have suffered trying to “puzzle out” what great thinkers like Socrates, Plato, and Descartes had to say on the great questions of man’s nature, Wilson said, but this was of little use, because philosophy has been based on “failed models of the brain.” This reminds me of my recent remarks on the use of crude folk-psychology models as microfoundations for social sciences. The article also discusses Wilson’s recent crusade against selfish-gene-style simplifications of human and animal nature. I’m with Wilson 100% on this one. “Two brothers or eight cousins” is a cute line but it doesn’t seem to come close to describing how species or societies work, and it’s always seemed a
4 0.95590222 1154 andrew gelman stats-2012-02-04-“Turn a Boring Bar Graph into a 3D Masterpiece”
Introduction: Jimmy sends in this . Steps include “Make whimsical sparkles by drawing an ellipse using the Ellipse Tool,” “Rotate the sparkles . . . Give some sparkles less Opacity by using the Transparency Palette,” and “Add a haze around each sparkle by drawing a white ellipse using the Ellipse Tool.” The punchline: Now, the next time you need to include a boring graph in one of your designs you’ll be able to add some extra emphasis and get people to really pay attention to those numbers! P.S. to all the commenters: Yeah, yeah, do your contrarian best and tell me why chartjunk is actually a good thing, how I’m just a snob, etc etc.
same-blog 5 0.95539308 738 andrew gelman stats-2011-05-30-Works well versus well understood
Introduction: John Cook discusses the John Tukey quote, “The test of a good procedure is how well it works, not how well it is understood.” Cook writes: At some level, it’s hard to argue against this. Statistical procedures operate on empirical data, so it makes sense that the procedures themselves be evaluated empirically. But I [Cook] question whether we really know that a statistical procedure works well if it isn’t well understood. Specifically, I’m skeptical of complex statistical methods whose only credentials are a handful of simulations. “We don’t have any theoretical results, buy hey, it works well in practice. Just look at the simulations.” Every method works well on the scenarios its author publishes, almost by definition. If the method didn’t handle a scenario well, the author would publish a different scenario. I agree with Cook but would give a slightly different emphasis. I’d say that a lot of methods can work when they are done well. See the second meta-principle liste
6 0.95427036 906 andrew gelman stats-2011-09-14-Another day, another stats postdoc
7 0.95140415 272 andrew gelman stats-2010-09-13-Ross Ihaka to R: Drop Dead
9 0.94741356 2190 andrew gelman stats-2014-01-29-Stupid R Tricks: Random Scope
11 0.94632709 525 andrew gelman stats-2011-01-19-Thiel update
12 0.94560748 541 andrew gelman stats-2011-01-27-Why can’t I be more like Bill James, or, The use of default and default-like models
15 0.94060469 2078 andrew gelman stats-2013-10-26-“The Bayesian approach to forensic evidence”
16 0.93971217 315 andrew gelman stats-2010-10-03-He doesn’t trust the fit . . . r=.999
17 0.93831635 1740 andrew gelman stats-2013-02-26-“Is machine learning a subset of statistics?”
18 0.93829507 581 andrew gelman stats-2011-02-19-“The best living writer of thrillers”
20 0.93579698 1497 andrew gelman stats-2012-09-15-Our blog makes connections!