andrew_gelman_stats andrew_gelman_stats-2013 andrew_gelman_stats-2013-2133 knowledge-graph by maker-knowledge-mining
Source: html
Introduction: If I made a separate post for each interesting blog discussion, we’d get overwhelmed. That’s why I often leave detailed responses in the comments section, even though I’m pretty sure that most readers don’t look in the comments at all. Sometimes, though, I think it’s good to bring such discussions to light. Here’s a recent example. Michael wrote : Poor predictive performance usually indicates that the model isn’t sufficiently flexible to explain the data, and my understanding of the proper Bayesian strategy is to feed that back into your original model and try again until you achieve better performance. Corey replied : It was my impression that — in ML at least — poor predictive performance is more often due to the model being too flexible and fitting noise. And Rahul agreed : Good point. A very flexible model will describe your training data perfectly and then go bonkers when unleashed on wild data. But I wrote : Overfitting comes from a model being flex
sentIndex sentText sentNum sentScore
1 That’s why I often leave detailed responses in the comments section, even though I’m pretty sure that most readers don’t look in the comments at all. [sent-2, score-0.692]
2 Michael wrote : Poor predictive performance usually indicates that the model isn’t sufficiently flexible to explain the data, and my understanding of the proper Bayesian strategy is to feed that back into your original model and try again until you achieve better performance. [sent-5, score-2.026]
3 Corey replied : It was my impression that — in ML at least — poor predictive performance is more often due to the model being too flexible and fitting noise. [sent-6, score-1.245]
4 A very flexible model will describe your training data perfectly and then go bonkers when unleashed on wild data. [sent-8, score-1.15]
5 But I wrote : Overfitting comes from a model being flexible and unregularized. [sent-9, score-0.809]
6 Making a model inflexible is a very crude form of regularization. [sent-10, score-0.593]
7 This is consistent with Michael’s original comment and also with my favorite Radford Neal quote: Sometimes a simple model will outperform a more complex model . [sent-12, score-1.349]
8 Nevertheless, I believe that deliberately limiting the complexity of the model is not fruitful when the problem is evidently complex. [sent-15, score-0.877]
9 Instead, if a simple model is found that outperforms some particular complex model, the appropriate response is to define a different complex model that captures whatever aspect of the problem led to the simple model performing well. [sent-16, score-2.281]
10 I’ll give Radford the last word for now (until anyone responds in the comments). [sent-17, score-0.096]
wordName wordTfidf (topN-words)
[('flexible', 0.371), ('model', 0.366), ('radford', 0.225), ('complex', 0.199), ('corey', 0.143), ('unleashed', 0.143), ('comments', 0.136), ('inflexible', 0.135), ('poor', 0.134), ('simple', 0.133), ('predictive', 0.13), ('performance', 0.13), ('rahul', 0.129), ('michael', 0.128), ('ml', 0.124), ('fruitful', 0.115), ('often', 0.114), ('outperforms', 0.113), ('sufficiently', 0.113), ('wild', 0.11), ('overfitting', 0.11), ('outperform', 0.11), ('neal', 0.109), ('deliberately', 0.105), ('feed', 0.105), ('original', 0.103), ('captures', 0.102), ('limiting', 0.102), ('nevertheless', 0.102), ('evidently', 0.101), ('responds', 0.096), ('achieve', 0.096), ('sometimes', 0.093), ('crude', 0.092), ('complexity', 0.088), ('proper', 0.088), ('indicates', 0.086), ('performing', 0.085), ('agreed', 0.084), ('detailed', 0.083), ('perfectly', 0.081), ('training', 0.079), ('responses', 0.076), ('aspect', 0.075), ('though', 0.074), ('define', 0.073), ('leave', 0.073), ('favorite', 0.072), ('wrote', 0.072), ('led', 0.071)]
simIndex simValue blogId blogTitle
same-blog 1 0.99999988 2133 andrew gelman stats-2013-12-13-Flexibility is good
Introduction: If I made a separate post for each interesting blog discussion, we’d get overwhelmed. That’s why I often leave detailed responses in the comments section, even though I’m pretty sure that most readers don’t look in the comments at all. Sometimes, though, I think it’s good to bring such discussions to light. Here’s a recent example. Michael wrote : Poor predictive performance usually indicates that the model isn’t sufficiently flexible to explain the data, and my understanding of the proper Bayesian strategy is to feed that back into your original model and try again until you achieve better performance. Corey replied : It was my impression that — in ML at least — poor predictive performance is more often due to the model being too flexible and fitting noise. And Rahul agreed : Good point. A very flexible model will describe your training data perfectly and then go bonkers when unleashed on wild data. But I wrote : Overfitting comes from a model being flex
2 0.2959238 1392 andrew gelman stats-2012-06-26-Occam
Introduction: Cosma Shalizi and Larry Wasserman discuss some papers from a conference on Ockham’s Razor. I don’t have anything new to add on this so let me link to past blog entries on the topic and repost the following from 2004 : A lot has been written in statistics about “parsimony”—that is, the desire to explain phenomena using fewer parameters–but I’ve never seen any good general justification for parsimony. (I don’t count “Occam’s Razor,” or “Ockham’s Razor,” or whatever, as a justification. You gotta do better than digging up a 700-year-old quote.) Maybe it’s because I work in social science, but my feeling is: if you can approximate reality with just a few parameters, fine. If you can use more parameters to fold in more information, that’s even better. In practice, I often use simple models—because they are less effort to fit and, especially, to understand. But I don’t kid myself that they’re better than more complicated efforts! My favorite quote on this comes from Rad
3 0.22036549 1041 andrew gelman stats-2011-12-04-David MacKay and Occam’s Razor
Introduction: In my comments on David MacKay’s 2003 book on Bayesian inference, I wrote that I hate all the Occam-factor stuff that MacKay talks about, and I linked to this quote from Radford Neal: Sometimes a simple model will outperform a more complex model . . . Nevertheless, I believe that deliberately limiting the complexity of the model is not fruitful when the problem is evidently complex. Instead, if a simple model is found that outperforms some particular complex model, the appropriate response is to define a different complex model that captures whatever aspect of the problem led to the simple model performing well. MacKay replied as follows: When you said you disagree with me on Occam factors I think what you meant was that you agree with me on them. I’ve read your post on the topic and completely agreed with you (and Radford) that we should be using models the size of a house, models that we believe in, and that anyone who thinks it is a good idea to bias the model toward
Introduction: In response to my remarks on his online book, Think Bayes, Allen Downey wrote: I [Downey] have a question about one of your comments: My [Gelman's] main criticism with both books is that they talk a lot about inference but not so much about model building or model checking (recall the three steps of Bayesian data analysis). I think it’s ok for an introductory book to focus on inference, which of course is central to the data-analytic process—but I’d like them to at least mention that Bayesian ideas arise in model building and model checking as well. This sounds like something I agree with, and one of the things I tried to do in the book is to put modeling decisions front and center. But the word “modeling” is used in lots of ways, so I want to see if we are talking about the same thing. For example, in many chapters, I start with a simple model of the scenario, do some analysis, then check whether the model is good enough, and iterate. Here’s the discussion of modeling
5 0.17293997 391 andrew gelman stats-2010-11-03-Some thoughts on election forecasting
Introduction: I’ve written a lot on polls and elections (“a poll is a snapshot, not a forecast,” etc., or see here for a more technical paper with Kari Lock) but had a few things to add in light of Sam Wang’s recent efforts . As a biologist with a physics degree, Wang brings an outsider’s perspective to political forecasting, which can be a good thing. (I’m a bit of an outsider to political science myself, as is my sometime collaborator Nate Silver, who’s done a lot of good work in the past few years.) But there are two places where Wang misses the point, I think. He refers to his method as a “transparent, low-assumption calculation” and compares it favorably to “fancy modeling” and “assumption-laden models.” Assumptions are a bad thing, right? Well, no, I don’t think so. Bad assumptions are a bad thing. Good assumptions are just fine. Similarly for fancy modeling. I don’t see why a model should get credit for not including a factor that might be important. Let me clarify. I
6 0.16303314 1999 andrew gelman stats-2013-08-27-Bayesian model averaging or fitting a larger model
7 0.15100577 1431 andrew gelman stats-2012-07-27-Overfitting
8 0.14874534 780 andrew gelman stats-2011-06-27-Bridges between deterministic and probabilistic models for binary data
9 0.14549117 754 andrew gelman stats-2011-06-09-Difficulties with Bayesian model averaging
10 0.14227569 639 andrew gelman stats-2011-03-31-Bayes: radical, liberal, or conservative?
11 0.13594164 2007 andrew gelman stats-2013-09-03-Popper and Jaynes
12 0.13400351 1141 andrew gelman stats-2012-01-28-Using predator-prey models on the Canadian lynx series
13 0.13330616 781 andrew gelman stats-2011-06-28-The holes in my philosophy of Bayesian data analysis
14 0.12766443 1817 andrew gelman stats-2013-04-21-More on Bayesian model selection in high-dimensional settings
15 0.1269947 1960 andrew gelman stats-2013-07-28-More on that machine learning course
16 0.12438065 1004 andrew gelman stats-2011-11-11-Kaiser Fung on how not to critique models
17 0.12305146 1197 andrew gelman stats-2012-03-04-“All Models are Right, Most are Useless”
18 0.12145434 1510 andrew gelman stats-2012-09-25-Incoherence of Bayesian data analysis
19 0.12120276 1527 andrew gelman stats-2012-10-10-Another reason why you can get good inferences from a bad model
20 0.12071382 24 andrew gelman stats-2010-05-09-Special journal issue on statistical methods for the social sciences
topicId topicWeight
[(0, 0.189), (1, 0.134), (2, -0.003), (3, 0.068), (4, 0.01), (5, 0.008), (6, 0.009), (7, -0.046), (8, 0.159), (9, 0.033), (10, 0.06), (11, 0.11), (12, -0.076), (13, -0.006), (14, -0.111), (15, 0.0), (16, 0.055), (17, -0.031), (18, -0.051), (19, 0.026), (20, 0.028), (21, -0.07), (22, -0.063), (23, -0.105), (24, -0.034), (25, 0.027), (26, -0.02), (27, 0.003), (28, 0.035), (29, -0.026), (30, -0.063), (31, -0.031), (32, -0.017), (33, 0.005), (34, 0.032), (35, 0.078), (36, 0.03), (37, -0.019), (38, 0.055), (39, -0.039), (40, -0.012), (41, -0.035), (42, -0.005), (43, 0.02), (44, 0.019), (45, -0.008), (46, -0.035), (47, -0.046), (48, 0.015), (49, 0.044)]
simIndex simValue blogId blogTitle
same-blog 1 0.99139899 2133 andrew gelman stats-2013-12-13-Flexibility is good
Introduction: If I made a separate post for each interesting blog discussion, we’d get overwhelmed. That’s why I often leave detailed responses in the comments section, even though I’m pretty sure that most readers don’t look in the comments at all. Sometimes, though, I think it’s good to bring such discussions to light. Here’s a recent example. Michael wrote : Poor predictive performance usually indicates that the model isn’t sufficiently flexible to explain the data, and my understanding of the proper Bayesian strategy is to feed that back into your original model and try again until you achieve better performance. Corey replied : It was my impression that — in ML at least — poor predictive performance is more often due to the model being too flexible and fitting noise. And Rahul agreed : Good point. A very flexible model will describe your training data perfectly and then go bonkers when unleashed on wild data. But I wrote : Overfitting comes from a model being flex
2 0.91443431 780 andrew gelman stats-2011-06-27-Bridges between deterministic and probabilistic models for binary data
Introduction: For the analysis of binary data, various deterministic models have been proposed, which are generally simpler to fit and easier to understand than probabilistic models. We claim that corresponding to any deterministic model is an implicit stochastic model in which the deterministic model fits imperfectly, with errors occurring at random. In the context of binary data, we consider a model in which the probability of error depends on the model prediction. We show how to fit this model using a stochastic modification of deterministic optimization schemes. The advantages of fitting the stochastic model explicitly (rather than implicitly, by simply fitting a deterministic model and accepting the occurrence of errors) include quantification of uncertainty in the deterministic model’s parameter estimates, better estimation of the true model error rate, and the ability to check the fit of the model nontrivially. We illustrate this with a simple theoretical example of item response data and w
3 0.91238457 1004 andrew gelman stats-2011-11-11-Kaiser Fung on how not to critique models
Introduction: In the context of a debate between economists Brad DeLong and Tyler Cowen on the “IS-LM model” [no, I don't know what it is, either!], Kaiser writes : Since a model is an abstraction, a simplification of reality, no model is above critique. I [Kaiser] consider the following types of critique not deserving: 1) The critique that the modeler makes an assumption 2) The critique that the modeler makes an assumption for mathematical convenience 3) The critique that the model omits some feature 4) The critique that the model doesn’t fit one’s intuition 5) The critique that the model fails to make a specific prediction Above all, a serious critique must include an alternative model that is provably better than the one it criticises. It is not enough to show that the alternative solves the problems being pointed out; the alternative must do so while preserving the useful aspects of the model being criticized. I have mixed feelings about Kaiser’s rules. On one hand, I agree wit
4 0.9072513 448 andrew gelman stats-2010-12-03-This is a footnote in one of my papers
Introduction: In the annals of hack literature, it is sometimes said that if you aim to write best-selling crap, all you’ll end up with is crap. To truly produce best-selling crap, you have to have a conviction, perhaps misplaced, that your writing has integrity. Whether or not this is a good generalization about writing, I have seen an analogous phenomenon in statistics: If you try to do nothing but model the data, you can be in for a wild and unpleasant ride: real data always seem to have one more twist beyond our ability to model (von Neumann’s elephant’s trunk notwithstanding). But if you model the underlying process, sometimes your model can fit surprisingly well as well as inviting openings for future research progress.
5 0.90481013 1141 andrew gelman stats-2012-01-28-Using predator-prey models on the Canadian lynx series
Introduction: The “Canadian lynx data” is one of the famous examples used in time series analysis. And the usual models that are fit to these data in the statistics time-series literature, don’t work well. Cavan Reilly and Angelique Zeringue write : Reilly and Zeringue then present their analysis. Their simple little predator-prey model with a weakly informative prior way outperforms the standard big-ass autoregression models. Check this out: Or, to put it into numbers, when they fit their model to the first 80 years and predict to the next 34, their root mean square out-of-sample error is 1480 (see scale of data above). In contrast, the standard model fit to these data (the SETAR model of Tong, 1990) has more than twice as many parameters but gets a worse-performing root mean square error of 1600, even when that model is fit to the entire dataset. (If you fit the SETAR or any similar autoregressive model to the first 80 years and use it to predict the next 34, the predictions
6 0.89365339 24 andrew gelman stats-2010-05-09-Special journal issue on statistical methods for the social sciences
7 0.88646924 1817 andrew gelman stats-2013-04-21-More on Bayesian model selection in high-dimensional settings
9 0.85213763 1392 andrew gelman stats-2012-06-26-Occam
10 0.84365332 2007 andrew gelman stats-2013-09-03-Popper and Jaynes
11 0.83750159 1431 andrew gelman stats-2012-07-27-Overfitting
12 0.83448708 1234 andrew gelman stats-2012-03-28-The Supreme Court’s Many Median Justices
13 0.8302874 754 andrew gelman stats-2011-06-09-Difficulties with Bayesian model averaging
14 0.82788992 1521 andrew gelman stats-2012-10-04-Columbo does posterior predictive checks
15 0.82702684 328 andrew gelman stats-2010-10-08-Displaying a fitted multilevel model
16 0.82540721 1162 andrew gelman stats-2012-02-11-Adding an error model to a deterministic model
17 0.81520981 1041 andrew gelman stats-2011-12-04-David MacKay and Occam’s Razor
18 0.81495762 935 andrew gelman stats-2011-10-01-When should you worry about imputed data?
19 0.81459391 82 andrew gelman stats-2010-06-12-UnConMax – uncertainty consideration maxims 7 +-- 2
20 0.81274503 823 andrew gelman stats-2011-07-26-Including interactions or not
topicId topicWeight
[(8, 0.013), (16, 0.11), (21, 0.034), (22, 0.013), (24, 0.214), (29, 0.123), (37, 0.024), (53, 0.033), (55, 0.016), (58, 0.016), (61, 0.015), (99, 0.284)]
simIndex simValue blogId blogTitle
1 0.96637905 1940 andrew gelman stats-2013-07-16-A poll that throws away data???
Introduction: Mark Blumenthal writes: What do you think about the “random rejection” method used by PPP that was attacked at some length today by a Republican pollster. Our just published post on the debate includes all the details as I know them. The Storify of Martino’s tweets has some additional data tables linked to toward the end. Also, more specifically, setting aside Martino’s suggestion of manipulation (which is also quite possible with post-stratification weights), would the PPP method introduce more potential random error than weighting? From Blumenthal’s blog: B.J. Martino, a senior vice president at the Republican polling firm The Tarrance Group, went on an 30-minute Twitter rant on Tuesday questioning the unorthodox method used by PPP [Public Policy Polling] to select samples and weight data: “Looking at @ppppolls new VA SW. Wondering how many interviews they discarded to get down to 601 completes? Because @ppppolls discards a LOT of interviews. Of 64,811 conducted
same-blog 2 0.96631283 2133 andrew gelman stats-2013-12-13-Flexibility is good
Introduction: If I made a separate post for each interesting blog discussion, we’d get overwhelmed. That’s why I often leave detailed responses in the comments section, even though I’m pretty sure that most readers don’t look in the comments at all. Sometimes, though, I think it’s good to bring such discussions to light. Here’s a recent example. Michael wrote : Poor predictive performance usually indicates that the model isn’t sufficiently flexible to explain the data, and my understanding of the proper Bayesian strategy is to feed that back into your original model and try again until you achieve better performance. Corey replied : It was my impression that — in ML at least — poor predictive performance is more often due to the model being too flexible and fitting noise. And Rahul agreed : Good point. A very flexible model will describe your training data perfectly and then go bonkers when unleashed on wild data. But I wrote : Overfitting comes from a model being flex
Introduction: Amanda Martinez, a writer for The Atlantic and others, advised attendees that her favorite writing “accorded me the basic human dignity of allowing me to draw my own conclusions.” I really like that way of putting it, and this is something we tried hard to do with Red State Blue State, to put the information and our reasoning right there in front of the reader, rather than hiding behind a bunch of statistically-significant regression coefficients. This is related to the idea of presenting research findings quantitatively (which, I think, lends itself to clearer statements of uncertainty and variation) rather than qualitatively (which seems to come out more deterministically, as “X causes Y” or “when A happens, B happens”). The above quote comes from a conference of students organized by Nathan Sanders, who writes: Thanks so much for posting an announcement about the Communicating Science workshop (ComSciCon) back in January! With the help of your blog, we received more than
4 0.95179427 639 andrew gelman stats-2011-03-31-Bayes: radical, liberal, or conservative?
Introduction: Radford writes : The word “conservative” gets used many ways, for various political purposes, but I would take it’s basic meaning to be someone who thinks there’s a lot of wisdom in traditional ways of doing things, even if we don’t understand exactly why those ways are good, so we should be reluctant to change unless we have a strong argument that some other way is better. This sounds very Bayesian, with a prior reducing the impact of new data. I agree completely, and I think Radford will very much enjoy my article with Aleks Jakulin , “Bayes: radical, liberal, or conservative?” Radford’s comment also fits with my increasing inclination to use informative prior distributions.
5 0.94831353 1392 andrew gelman stats-2012-06-26-Occam
Introduction: Cosma Shalizi and Larry Wasserman discuss some papers from a conference on Ockham’s Razor. I don’t have anything new to add on this so let me link to past blog entries on the topic and repost the following from 2004 : A lot has been written in statistics about “parsimony”—that is, the desire to explain phenomena using fewer parameters–but I’ve never seen any good general justification for parsimony. (I don’t count “Occam’s Razor,” or “Ockham’s Razor,” or whatever, as a justification. You gotta do better than digging up a 700-year-old quote.) Maybe it’s because I work in social science, but my feeling is: if you can approximate reality with just a few parameters, fine. If you can use more parameters to fold in more information, that’s even better. In practice, I often use simple models—because they are less effort to fit and, especially, to understand. But I don’t kid myself that they’re better than more complicated efforts! My favorite quote on this comes from Rad
6 0.94714499 1024 andrew gelman stats-2011-11-23-Of hypothesis tests and Unitarians
8 0.94327033 466 andrew gelman stats-2010-12-13-“The truth wears off: Is there something wrong with the scientific method?”
9 0.94064796 1687 andrew gelman stats-2013-01-21-Workshop on science communication for graduate students
10 0.93937713 1421 andrew gelman stats-2012-07-19-Alexa, Maricel, and Marty: Three cellular automata who got on my nerves
11 0.93910545 1344 andrew gelman stats-2012-05-25-Question 15 of my final exam for Design and Analysis of Sample Surveys
12 0.9389047 1034 andrew gelman stats-2011-11-29-World Class Speakers and Entertainers
13 0.93450689 1491 andrew gelman stats-2012-09-10-Update on Levitt paper on child car seats
14 0.93331152 898 andrew gelman stats-2011-09-10-Fourteen magic words: an update
15 0.93276912 799 andrew gelman stats-2011-07-13-Hypothesis testing with multiple imputations
16 0.93242824 1155 andrew gelman stats-2012-02-05-What is a prior distribution?
17 0.93211532 1080 andrew gelman stats-2011-12-24-Latest in blog advertising
18 0.93161136 2246 andrew gelman stats-2014-03-13-An Economist’s Guide to Visualizing Data
19 0.93035173 807 andrew gelman stats-2011-07-17-Macro causality
20 0.93008327 1881 andrew gelman stats-2013-06-03-Boot