andrew_gelman_stats andrew_gelman_stats-2013 andrew_gelman_stats-2013-1972 knowledge-graph by maker-knowledge-mining
Source: html
Introduction: In response to my remarks on his online book, Think Bayes, Allen Downey wrote: I [Downey] have a question about one of your comments: My [Gelman's] main criticism with both books is that they talk a lot about inference but not so much about model building or model checking (recall the three steps of Bayesian data analysis). I think it’s ok for an introductory book to focus on inference, which of course is central to the data-analytic process—but I’d like them to at least mention that Bayesian ideas arise in model building and model checking as well. This sounds like something I agree with, and one of the things I tried to do in the book is to put modeling decisions front and center. But the word “modeling” is used in lots of ways, so I want to see if we are talking about the same thing. For example, in many chapters, I start with a simple model of the scenario, do some analysis, then check whether the model is good enough, and iterate. Here’s the discussion of modeling
sentIndex sentText sentNum sentScore
1 I think it’s ok for an introductory book to focus on inference, which of course is central to the data-analytic process—but I’d like them to at least mention that Bayesian ideas arise in model building and model checking as well. [sent-2, score-1.342]
2 This sounds like something I agree with, and one of the things I tried to do in the book is to put modeling decisions front and center. [sent-3, score-0.458]
3 For example, in many chapters, I start with a simple model of the scenario, do some analysis, then check whether the model is good enough, and iterate. [sent-5, score-0.891]
4 html#toc2 Most chapters in this book are motivated by a real-world problem, so most chapters involve some degree of modeling. [sent-9, score-0.48]
5 I model goal-scoring as a Poisson process, which implies that a goal is equally likely at any point in the game. [sent-12, score-0.592]
6 That is not exactly true, but it is probably a good enough model for most purposes. [sent-13, score-0.485]
7 I pretend, temporarily, that all SAT questions are equally difficult. [sent-19, score-0.274]
8 Actually, the designers of the SAT choose questions with a range of difficulty, because that improves the ability to measure statistical differences between test-takers. [sent-20, score-0.322]
9 But if we choose a model where all questions are equally difficult, we can define a characteristic, p_correct, for each test-taker, which is the probability of answering any question correctly. [sent-21, score-0.908]
10 Is this the kind of model building and model checking you are talking about? [sent-23, score-1.368]
11 I replied: Yes, this is the sort of model building I was talking about. [sent-24, score-0.777]
12 But when I was talking about model checking, I was going a step further. [sent-25, score-0.59]
13 It seems to me that what you are proposing (and I agree with this 100%) is that when you’re planning on fitting a model, you build up to it by fitting simpler models first. [sent-26, score-0.343]
14 But what I’m saying is that, once you get to the serious model that you like, you then test it by using the model to make lots of predictions (within-sample as well as out-of-sample) and seeing if the predictions look like the data. [sent-29, score-1.083]
15 I’m not talking here about error rates but rather about graphical checks to see if the model can reproduce the look of the data. [sent-30, score-0.657]
16 He then wrote: I reviewed Chapter 6 of your book, and I have a good idea now what you mean by model checking. [sent-32, score-0.414]
17 One example: in Chapter 7 I had to make some guesses about the distribution of difficulty for SAT questions. [sent-34, score-0.28]
18 I don’t have any direct measurements of difficulty, so I use a model based on item response theory to generate simulated test scores, then compare to the actual distribution of scores. [sent-35, score-0.742]
19 html#toc100 The data and the simulated data agree pretty well, but the residuals are not independent. [sent-39, score-0.285]
20 I suspect there is a better model that would capture a functional form I am missing, but I concluded that the simple model is good enough for the intended purpose. [sent-40, score-1.029]
wordName wordTfidf (topN-words)
[('model', 0.414), ('chapter', 0.242), ('building', 0.187), ('downey', 0.18), ('equally', 0.178), ('checking', 0.177), ('talking', 0.176), ('simplification', 0.166), ('chapters', 0.165), ('sat', 0.158), ('book', 0.15), ('difficulty', 0.137), ('modeling', 0.133), ('http', 0.129), ('simulated', 0.127), ('bayesian', 0.099), ('questions', 0.096), ('predictions', 0.092), ('decisions', 0.091), ('pretend', 0.09), ('hockey', 0.09), ('fitting', 0.089), ('preface', 0.086), ('choose', 0.084), ('agree', 0.084), ('methods', 0.083), ('analysis', 0.082), ('temporarily', 0.081), ('proposing', 0.081), ('guesses', 0.077), ('improves', 0.075), ('motivating', 0.074), ('residuals', 0.074), ('process', 0.072), ('test', 0.071), ('enough', 0.071), ('allen', 0.07), ('characteristic', 0.069), ('answering', 0.068), ('question', 0.068), ('designers', 0.067), ('reproduce', 0.067), ('distribution', 0.066), ('concluded', 0.066), ('poisson', 0.065), ('functional', 0.064), ('response', 0.064), ('winner', 0.064), ('example', 0.063), ('start', 0.063)]
simIndex simValue blogId blogTitle
same-blog 1 1.0000001 1972 andrew gelman stats-2013-08-07-When you’re planning on fitting a model, build up to it by fitting simpler models first. Then, once you have a model you like, check the hell out of it
Introduction: In response to my remarks on his online book, Think Bayes, Allen Downey wrote: I [Downey] have a question about one of your comments: My [Gelman's] main criticism with both books is that they talk a lot about inference but not so much about model building or model checking (recall the three steps of Bayesian data analysis). I think it’s ok for an introductory book to focus on inference, which of course is central to the data-analytic process—but I’d like them to at least mention that Bayesian ideas arise in model building and model checking as well. This sounds like something I agree with, and one of the things I tried to do in the book is to put modeling decisions front and center. But the word “modeling” is used in lots of ways, so I want to see if we are talking about the same thing. For example, in many chapters, I start with a simple model of the scenario, do some analysis, then check whether the model is good enough, and iterate. Here’s the discussion of modeling
2 0.28975996 1948 andrew gelman stats-2013-07-21-Bayes related
Introduction: Dave Decker writes: I’ve seen some Bayes related things recently that might make for interesting fodder on your blog. There are two books, teaching Bayesian analysis from a programming perspective. And also a “web application for data analysis using powerful Bayesian statistical methods.” I took a look. The first book is Think Bayes: Bayesian Statistics Made Simple, by Allen B. Downey . It’s super readable and, amazingly, has approximately zero overlap with Bayesian Data Analysis. Downey discusses lots of little problems in a conversational way. In some ways it’s like an old-style math stat textbook (although with a programming rather than mathematical flavor) in that the examples are designed for simplicity rather than realism. I like it! Our book already exists; it’s good to have something else for people to read, coming from an entirely different perspective. The second book is Probabilistic Programming and Bayesian Methods for Hackers , by Cameron Davidson-P
3 0.25563118 781 andrew gelman stats-2011-06-28-The holes in my philosophy of Bayesian data analysis
Introduction: I’ve been writing a lot about my philosophy of Bayesian statistics and how it fits into Popper’s ideas about falsification and Kuhn’s ideas about scientific revolutions. Here’s my long, somewhat technical paper with Cosma Shalizi. Here’s our shorter overview for the volume on the philosophy of social science. Here’s my latest try (for an online symposium), focusing on the key issues. I’m pretty happy with my approach–the familiar idea that Bayesian data analysis iterates the three steps of model building, inference, and model checking–but it does have some unresolved (maybe unresolvable) problems. Here are a couple mentioned in the third of the above links. Consider a simple model with independent data y_1, y_2, .., y_10 ~ N(θ,σ^2), with a prior distribution θ ~ N(0,10^2) and σ known and taking on some value of approximately 10. Inference about μ is straightforward, as is model checking, whether based on graphs or numerical summaries such as the sample variance and skewn
4 0.25140733 754 andrew gelman stats-2011-06-09-Difficulties with Bayesian model averaging
Introduction: In response to this article by Cosma Shalizi and myself on the philosophy of Bayesian statistics, David Hogg writes: I [Hogg] agree–even in physics and astronomy–that the models are not “True” in the God-like sense of being absolute reality (that is, I am not a realist); and I have argued (a philosophically very naive paper, but hey, I was new to all this) that for pretty fundamental reasons we could never arrive at the True (with a capital “T”) model of the Universe. The goal of inference is to find the “best” model, where “best” might have something to do with prediction, or explanation, or message length, or (horror!) our utility. Needless to say, most of my physics friends *are* realists, even in the face of “effective theories” as Newtonian mechanics is an effective theory of GR and GR is an effective theory of “quantum gravity” (this plays to your point, because if you think any theory is possibly an effective theory, how could you ever find Truth?). I also liked the i
5 0.23370667 244 andrew gelman stats-2010-08-30-Useful models, model checking, and external validation: a mini-discussion
Introduction: I sent a copy of my paper (coauthored with Cosma Shalizi) on Philosophy and the practice of Bayesian statistics in the social sciences to Richard Berk , who wrote: I read your paper this morning. I think we are pretty much on the same page about all models being wrong. I like very much the way you handle this in the paper. Yes, Newton’s work is wrong, but surely useful. I also like your twist on Bayesian methods. Makes good sense to me. Perhaps most important, your paper raises some difficult issues I have been trying to think more carefully about. 1. If the goal of a model is to be useful, surely we need to explore that “useful” means. At the very least, usefulness will depend on use. So a model that is useful for forecasting may or may not be useful for causal inference. 2. Usefulness will be a matter of degree. So that for each use we will need one or more metrics to represent how useful the model is. In what looks at first to be simple example, if the use is forecasting,
6 0.20905301 2133 andrew gelman stats-2013-12-13-Flexibility is good
7 0.20905207 2007 andrew gelman stats-2013-09-03-Popper and Jaynes
8 0.20608823 1247 andrew gelman stats-2012-04-05-More philosophy of Bayes
9 0.20537472 1392 andrew gelman stats-2012-06-26-Occam
10 0.20031275 1554 andrew gelman stats-2012-10-31-It not necessary that Bayesian methods conform to the likelihood principle
12 0.19828033 320 andrew gelman stats-2010-10-05-Does posterior predictive model checking fit with the operational subjective approach?
13 0.18853104 780 andrew gelman stats-2011-06-27-Bridges between deterministic and probabilistic models for binary data
14 0.18846492 1431 andrew gelman stats-2012-07-27-Overfitting
15 0.1883302 1205 andrew gelman stats-2012-03-09-Coming to agreement on philosophy of statistics
16 0.18500359 1112 andrew gelman stats-2012-01-11-A blog full of examples for your statistics class
17 0.18428317 1004 andrew gelman stats-2011-11-11-Kaiser Fung on how not to critique models
18 0.18303075 773 andrew gelman stats-2011-06-18-Should we always be using the t and robit instead of the normal and logit?
19 0.17940874 1999 andrew gelman stats-2013-08-27-Bayesian model averaging or fitting a larger model
20 0.17509219 1482 andrew gelman stats-2012-09-04-Model checking and model understanding in machine learning
topicId topicWeight
[(0, 0.318), (1, 0.229), (2, -0.049), (3, 0.115), (4, 0.001), (5, 0.068), (6, -0.037), (7, 0.001), (8, 0.221), (9, 0.017), (10, 0.058), (11, 0.089), (12, -0.067), (13, -0.008), (14, -0.038), (15, -0.012), (16, 0.077), (17, -0.01), (18, 0.01), (19, -0.012), (20, 0.026), (21, -0.042), (22, -0.003), (23, -0.075), (24, -0.049), (25, 0.023), (26, 0.004), (27, -0.031), (28, 0.04), (29, -0.015), (30, -0.097), (31, -0.049), (32, 0.002), (33, 0.065), (34, 0.006), (35, 0.093), (36, 0.022), (37, -0.053), (38, -0.018), (39, -0.042), (40, -0.006), (41, -0.022), (42, -0.029), (43, 0.049), (44, 0.01), (45, 0.023), (46, 0.019), (47, -0.069), (48, -0.03), (49, 0.096)]
simIndex simValue blogId blogTitle
same-blog 1 0.99141264 1972 andrew gelman stats-2013-08-07-When you’re planning on fitting a model, build up to it by fitting simpler models first. Then, once you have a model you like, check the hell out of it
Introduction: In response to my remarks on his online book, Think Bayes, Allen Downey wrote: I [Downey] have a question about one of your comments: My [Gelman's] main criticism with both books is that they talk a lot about inference but not so much about model building or model checking (recall the three steps of Bayesian data analysis). I think it’s ok for an introductory book to focus on inference, which of course is central to the data-analytic process—but I’d like them to at least mention that Bayesian ideas arise in model building and model checking as well. This sounds like something I agree with, and one of the things I tried to do in the book is to put modeling decisions front and center. But the word “modeling” is used in lots of ways, so I want to see if we are talking about the same thing. For example, in many chapters, I start with a simple model of the scenario, do some analysis, then check whether the model is good enough, and iterate. Here’s the discussion of modeling
2 0.89849144 1041 andrew gelman stats-2011-12-04-David MacKay and Occam’s Razor
Introduction: In my comments on David MacKay’s 2003 book on Bayesian inference, I wrote that I hate all the Occam-factor stuff that MacKay talks about, and I linked to this quote from Radford Neal: Sometimes a simple model will outperform a more complex model . . . Nevertheless, I believe that deliberately limiting the complexity of the model is not fruitful when the problem is evidently complex. Instead, if a simple model is found that outperforms some particular complex model, the appropriate response is to define a different complex model that captures whatever aspect of the problem led to the simple model performing well. MacKay replied as follows: When you said you disagree with me on Occam factors I think what you meant was that you agree with me on them. I’ve read your post on the topic and completely agreed with you (and Radford) that we should be using models the size of a house, models that we believe in, and that anyone who thinks it is a good idea to bias the model toward
3 0.88254285 1431 andrew gelman stats-2012-07-27-Overfitting
Introduction: Ilya Esteban writes: In traditional machine learning and statistical learning techniques, you spend a lot of time selecting your input features, fiddling with model parameter values, etc., all of which leads to the problem of overfitting the data and producing overly optimistic estimates for how good the model really is. You can use techniques such as cross-validation and out-of-sample validation data to try to limit the damage, but they are imperfect solutions at best. While Bayesian models have the great advantage of not forcing you to manually select among the various weights and input features, you still often end up trying different priors and model structures (especially with hierarchical models), before coming up with a “final” model. When applying Bayesian modeling to real world data sets, how does should you evaluate alternate priors and topologies for the model without falling into the same overfitting trap as you do with non-Bayesian models? If you try several different
4 0.88225687 2133 andrew gelman stats-2013-12-13-Flexibility is good
Introduction: If I made a separate post for each interesting blog discussion, we’d get overwhelmed. That’s why I often leave detailed responses in the comments section, even though I’m pretty sure that most readers don’t look in the comments at all. Sometimes, though, I think it’s good to bring such discussions to light. Here’s a recent example. Michael wrote : Poor predictive performance usually indicates that the model isn’t sufficiently flexible to explain the data, and my understanding of the proper Bayesian strategy is to feed that back into your original model and try again until you achieve better performance. Corey replied : It was my impression that — in ML at least — poor predictive performance is more often due to the model being too flexible and fitting noise. And Rahul agreed : Good point. A very flexible model will describe your training data perfectly and then go bonkers when unleashed on wild data. But I wrote : Overfitting comes from a model being flex
5 0.87808597 935 andrew gelman stats-2011-10-01-When should you worry about imputed data?
Introduction: Majid Ezzati writes: My research group is increasingly focusing on a series of problems that involve data that either have missingness or measurements that may have bias/error. We have at times developed our own approaches to imputation (as simple as interpolating a missing unit and as sophisticated as a problem-specific Bayesian hierarchical model) and at other times, other groups impute the data. The outputs are being used to investigate the basic associations between pairs of variables, Xs and Ys, in regressions; we may or may not interpret these as causal. I am contacting colleagues with relevant expertise to suggest good references on whether having imputed X and/or Y in a subsequent regression is correct or if it could somehow lead to biased/spurious associations. Thinking about this, we can have at least the following situations (these could all be Bayesian or not): 1) X and Y both measured (perhaps with error) 2) Y imputed using some data and a model and X measur
6 0.8745507 2007 andrew gelman stats-2013-09-03-Popper and Jaynes
7 0.87388748 1521 andrew gelman stats-2012-10-04-Columbo does posterior predictive checks
9 0.86945254 1392 andrew gelman stats-2012-06-26-Occam
10 0.86729044 1406 andrew gelman stats-2012-07-05-Xiao-Li Meng and Xianchao Xie rethink asymptotics
11 0.86297947 1141 andrew gelman stats-2012-01-28-Using predator-prey models on the Canadian lynx series
12 0.86186802 754 andrew gelman stats-2011-06-09-Difficulties with Bayesian model averaging
13 0.85999894 1004 andrew gelman stats-2011-11-11-Kaiser Fung on how not to critique models
14 0.85609245 1510 andrew gelman stats-2012-09-25-Incoherence of Bayesian data analysis
15 0.85595739 781 andrew gelman stats-2011-06-28-The holes in my philosophy of Bayesian data analysis
16 0.85312539 1999 andrew gelman stats-2013-08-27-Bayesian model averaging or fitting a larger model
17 0.84734702 328 andrew gelman stats-2010-10-08-Displaying a fitted multilevel model
18 0.84675074 773 andrew gelman stats-2011-06-18-Should we always be using the t and robit instead of the normal and logit?
19 0.84537715 448 andrew gelman stats-2010-12-03-This is a footnote in one of my papers
20 0.84468228 244 andrew gelman stats-2010-08-30-Useful models, model checking, and external validation: a mini-discussion
topicId topicWeight
[(9, 0.01), (15, 0.051), (16, 0.135), (21, 0.026), (24, 0.142), (36, 0.02), (47, 0.014), (50, 0.065), (53, 0.016), (86, 0.034), (94, 0.015), (99, 0.363)]
simIndex simValue blogId blogTitle
1 0.98303413 2137 andrew gelman stats-2013-12-17-Replication backlash
Introduction: Raghuveer Parthasarathy pointed me to an article in Nature by Mina Bissell, who writes , “The push to replicate findings could shelve promising research and unfairly damage the reputations of careful, meticulous scientists.” I can see where she’s coming from: if you work hard day after day in the lab, it’s gotta be a bit frustrating to find all your work questioned, for the frauds of the Dr. Anil Pottis and Diederik Stapels to be treated as a reason for everyone else’s work to be considered guilty until proven innocent. That said, I pretty much disagree with Bissell’s article, and really the best thing I can say about it is that I think it’s a good sign that the push for replication is so strong that now there’s a backlash against it. Traditionally, leading scientists have been able to simply ignore the push for replication. If they are feeling that the replication movement is strong enough that they need to fight it, that to me is good news. I’ll explain a bit in the conte
2 0.98053157 1878 andrew gelman stats-2013-05-31-How to fix the tabloids? Toward replicable social science research
Introduction: This seems to be the topic of the week. Yesterday I posted on the sister blog some further thoughts on those “Psychological Science” papers on menstrual cycles, biceps size, and political attitudes, tied to a horrible press release from the journal Psychological Science hyping the biceps and politics study. Then I was pointed to these suggestions from Richard Lucas and M. Brent Donnellan have on improving the replicability and reproducibility of research published in the Journal of Research in Personality: It goes without saying that editors of scientific journals strive to publish research that is not only theoretically interesting but also methodologically rigorous. The goal is to select papers that advance the field. Accordingly, editors want to publish findings that can be reproduced and replicated by other scientists. Unfortunately, there has been a recent “crisis in confidence” among psychologists about the quality of psychological research (Pashler & Wagenmakers, 2012)
3 0.97925878 232 andrew gelman stats-2010-08-25-Dodging the diplomats
Introduction: The usually-reasonable-even-if-you-disagree-with-him Tyler Cowen writes : Presumably diplomats either enjoy serving their country or they enjoy the ego rents of being a diplomat or both. It is a false feeling of power, borrowed power from one’s country of origin rather than from one’s personal achievements. Huh? I’d hardly think this needs to be explained, but here goes: 1. Diplomats may feel the duty to serve their country, which is not the same as “enjoying” it. Sometimes people take on jobs that are challenging and not well-paid because they feel that it is their duty to do their best at it. 2. Some diplomats are very accomplished individuals, and that is why they are chosen to represent their country. Consider an analogy: Yes, Tyler Cowen borrows some power from George Mason University. But it goes the other way too: GMU borrows power from TC. Beyond all this, and returning to more selfish goals, being a diplomat can be fun–you get to live in a foreign c
Introduction: Jeff Ratto points me to this news article by Dean Baker reporting the work of three economists, Thomas Herndon, Michael Ash, and Robert Pollin, who found errors in a much-cited article by Carmen Reinhart and Kenneth Rogoff analyzing historical statistics of economic growth and public debt. Mike Konczal provides a clear summary; that’s where I got the above image. Errors in data processing and data analysis It turns out that Reinhart and Rogoff flubbed it. Herndon et al. write of “spreadsheet errors, omission of available data, weighting, and transcription.” The spreadsheet errors are the most embarrassing, but the other choices in data analysis seem pretty bad too. It can be tough to work with small datasets, so I have sympathy for Reinhart and Rogoff, but it does look like they were jumping to conclusions in their paper. Perhaps the urgency of the topic moved them to publish as fast as possible rather than carefully considering the impact of their data-analytic choi
Introduction: I was thinking more about David Brooks’s anti-data column from yesterday, and I realized what is really bothering me. Brooks expresses skepticism about numbers, about the limitations of raw data, about the importance of human thinking. Fine, I agree with all of this, to some extent. But then Brooks turns around uses numbers and unquestioningly and uncritically (OK, not completely uncritically; see P.S. below). In a notorious recent case, Brooks wrote, in the context of college admissions: You’re going to want to argue with Unz’s article all the way along, especially for its narrow, math-test-driven view of merit. But it’s potentially ground-shifting. Unz’s other big point is that Jews are vastly overrepresented at elite universities and that Jewish achievement has collapsed. In the 1970s, for example, 40 percent of top scorers in the Math Olympiad had Jewish names. Now 2.5 percent do. But these numbers are incorrect, as I learned from a professor of oncology at the Univ
6 0.97553992 711 andrew gelman stats-2011-05-14-Steven Rhoads’s book, “The Economist’s View of the World”
7 0.97489661 2227 andrew gelman stats-2014-02-27-“What Can we Learn from the Many Labs Replication Project?”
same-blog 8 0.97467446 1972 andrew gelman stats-2013-08-07-When you’re planning on fitting a model, build up to it by fitting simpler models first. Then, once you have a model you like, check the hell out of it
9 0.97418261 110 andrew gelman stats-2010-06-26-Philosophy and the practice of Bayesian statistics
10 0.97315592 1163 andrew gelman stats-2012-02-12-Meta-analysis, game theory, and incentives to do replicable research
11 0.97292328 2280 andrew gelman stats-2014-04-03-As the boldest experiment in journalism history, you admit you made a mistake
14 0.97264475 154 andrew gelman stats-2010-07-18-Predictive checks for hierarchical models
15 0.97173101 120 andrew gelman stats-2010-06-30-You can’t put Pandora back in the box
17 0.97125667 2245 andrew gelman stats-2014-03-12-More on publishing in journals
18 0.97119212 2177 andrew gelman stats-2014-01-19-“The British amateur who debunked the mathematics of happiness”
19 0.9711408 1760 andrew gelman stats-2013-03-12-Misunderstanding the p-value
20 0.97110331 1435 andrew gelman stats-2012-07-30-Retracted articles and unethical behavior in economics journals?