andrew_gelman_stats andrew_gelman_stats-2013 andrew_gelman_stats-2013-1975 knowledge-graph by maker-knowledge-mining
Source: html
Introduction: Jessy, Aki, and I write : We review the Akaike, deviance, and Watanabe-Akaike information criteria from a Bayesian perspective, where the goal is to estimate expected out-of-sample-prediction error using a bias-corrected adjustment of within-sample error. We focus on the choices involved in setting up these measures, and we compare them in three simple examples, one theoretical and two applied. The contribution of this review is to put all these information criteria into a Bayesian predictive context and to better understand, through small examples, how these methods can apply in practice. I like this paper. It came about as a result of preparing Chapter 7 for the new BDA . I had difficulty understanding AIC, DIC, WAIC, etc., but I recognized that these methods served a need. My first plan was to just apply DIC and WAIC on a couple of simple examples (a linear regression and the 8 schools) and leave it at that. But when I did the calculations, I couldn’t understand the resu
sentIndex sentText sentNum sentScore
1 Jessy, Aki, and I write : We review the Akaike, deviance, and Watanabe-Akaike information criteria from a Bayesian perspective, where the goal is to estimate expected out-of-sample-prediction error using a bias-corrected adjustment of within-sample error. [sent-1, score-0.677]
2 We focus on the choices involved in setting up these measures, and we compare them in three simple examples, one theoretical and two applied. [sent-2, score-0.563]
3 The contribution of this review is to put all these information criteria into a Bayesian predictive context and to better understand, through small examples, how these methods can apply in practice. [sent-3, score-0.964]
4 It came about as a result of preparing Chapter 7 for the new BDA . [sent-5, score-0.121]
5 I had difficulty understanding AIC, DIC, WAIC, etc. [sent-6, score-0.075]
6 , but I recognized that these methods served a need. [sent-7, score-0.295]
7 My first plan was to just apply DIC and WAIC on a couple of simple examples (a linear regression and the 8 schools) and leave it at that. [sent-8, score-0.723]
8 But when I did the calculations, I couldn’t understand the results. [sent-9, score-0.092]
9 Hence more effort working all these out in some simple examples, and further thought into the ultimate motivations for all these methods. [sent-10, score-0.411]
10 When introducing AIC, Akaike called it An Information Criterion. [sent-13, score-0.419]
11 When introducing WAIC, Watanabe called it the Widely Applicable Information Criterion. [sent-14, score-0.419]
12 Aki and I are hoping to come up with something called the Very Good Information Criterion. [sent-15, score-0.255]
wordName wordTfidf (topN-words)
[('waic', 0.396), ('dic', 0.258), ('introducing', 0.252), ('akaike', 0.246), ('aic', 0.246), ('aki', 0.218), ('examples', 0.214), ('information', 0.206), ('criteria', 0.192), ('called', 0.167), ('watanabe', 0.147), ('simple', 0.146), ('apply', 0.138), ('applicable', 0.129), ('deviance', 0.123), ('preparing', 0.121), ('review', 0.116), ('served', 0.11), ('criterion', 0.108), ('motivations', 0.102), ('ultimate', 0.1), ('bda', 0.097), ('adjustment', 0.096), ('recognized', 0.094), ('understand', 0.092), ('methods', 0.091), ('calculations', 0.09), ('contribution', 0.088), ('hoping', 0.088), ('widely', 0.084), ('bayesian', 0.081), ('choices', 0.081), ('leave', 0.08), ('schools', 0.079), ('measures', 0.078), ('hence', 0.075), ('plan', 0.075), ('difficulty', 0.075), ('compare', 0.072), ('predictive', 0.071), ('couldn', 0.071), ('linear', 0.07), ('setting', 0.068), ('involved', 0.068), ('expected', 0.067), ('theoretical', 0.066), ('chapter', 0.066), ('effort', 0.063), ('context', 0.062), ('focus', 0.062)]
simIndex simValue blogId blogTitle
same-blog 1 1.0 1975 andrew gelman stats-2013-08-09-Understanding predictive information criteria for Bayesian models
Introduction: Jessy, Aki, and I write : We review the Akaike, deviance, and Watanabe-Akaike information criteria from a Bayesian perspective, where the goal is to estimate expected out-of-sample-prediction error using a bias-corrected adjustment of within-sample error. We focus on the choices involved in setting up these measures, and we compare them in three simple examples, one theoretical and two applied. The contribution of this review is to put all these information criteria into a Bayesian predictive context and to better understand, through small examples, how these methods can apply in practice. I like this paper. It came about as a result of preparing Chapter 7 for the new BDA . I had difficulty understanding AIC, DIC, WAIC, etc., but I recognized that these methods served a need. My first plan was to just apply DIC and WAIC on a couple of simple examples (a linear regression and the 8 schools) and leave it at that. But when I did the calculations, I couldn’t understand the resu
2 0.42952365 2349 andrew gelman stats-2014-05-26-WAIC and cross-validation in Stan!
Introduction: Aki and I write : The Watanabe-Akaike information criterion (WAIC) and cross-validation are methods for estimating pointwise out-of-sample prediction accuracy from a fitted Bayesian model. WAIC is based on the series expansion of leave-one-out cross-validation (LOO), and asymptotically they are equal. With finite data, WAIC and cross-validation address different predictive questions and thus it is useful to be able to compute both. WAIC and an importance-sampling approximated LOO can be estimated directly using the log-likelihood evaluated at the posterior simulations of the parameter values. We show how to compute WAIC, IS-LOO, K-fold cross-validation, and related diagnostic quantities in the Bayesian inference package Stan as called from R. This is important, I think. One reason the deviance information criterion (DIC) has been so popular is its implementation in Bugs. We think WAIC and cross-validation make more sense than DIC, especially from a Bayesian perspective in whic
3 0.31877893 778 andrew gelman stats-2011-06-24-New ideas on DIC from Martyn Plummer and Sumio Watanabe
Introduction: Martyn Plummer replied to my recent blog on DIC with information that was important enough that I thought it deserved its own blog entry. Martyn wrote: DIC has been around for 10 years now and despite being immensely popular with applied statisticians it has generated very little theoretical interest. In fact, the silence has been deafening. I [Martyn] hope my paper added some clarity. As you say, DIC is (an approximation to) a theoretical out-of-sample predictive error. When I finished the paper I was a little embarrassed to see that I had almost perfectly reconstructed the justification of AIC as approximate cross-validation measure by Stone (1977), with a Bayesian spin of course. But even this insight leaves a lot of choices open. You need to choose the right loss function and also which level of the model you want to replicate from. David Spiegelhalter and colleagues called this the “focus”. In practice the focus is limited to the lowest level of the model. You generall
Introduction: Aki Vehtari and Janne Ojanen just published a long paper that begins: To date, several methods exist in the statistical literature for model assessment, which purport themselves specifically as Bayesian predictive methods. The decision theoretic assumptions on which these methods are based are not always clearly stated in the original articles, however. The aim of this survey is to provide a unified review of Bayesian predictive model assessment and selection methods, and of methods closely related to them. We review the various assumptions that are made in this context and discuss the connections between different approaches, with an emphasis on how each method approximates the expected utility of using a Bayesian model for the purpose of predicting future data. AIC (which Akaike called “An Information Criterion”) is the starting point for all these methods. More recently, Watanabe came up with WAIC (which he called the “Widely Available Information Criterion”). In between t
5 0.28915292 776 andrew gelman stats-2011-06-22-Deviance, DIC, AIC, cross-validation, etc
Introduction: The deviance information criterion (or DIC) is an idea of Brad Carlin and others for comparing the fits of models estimated using Bayesian simulation (for more information, see this article by Angelika van der Linde). I don’t really ever know what to make of DIC. On one hand, it seems sensible, it handles uncertainty in inferences within each model, and it does not depend on aspects of the models that don’t affect inferences within each model (unlike Bayes factors; see discussion here ). On the other hand, I don’t really have any idea what I would do with DIC in any real example. In our book we included an example of DIC–people use it and we don’t have any great alternatives–but I had to be pretty careful that the example made sense. Unlike the usual setting where we use a method and that gives us insight into a problem, here we used our insight into the problem to make sure that in this particular case the method gave a reasonable answer. One of my practical problems with D
6 0.20297925 1377 andrew gelman stats-2012-06-13-A question about AIC
7 0.17880434 1983 andrew gelman stats-2013-08-15-More on AIC, WAIC, etc
8 0.1117551 2348 andrew gelman stats-2014-05-26-On deck this week
9 0.10066395 1149 andrew gelman stats-2012-02-01-Philosophy of Bayesian statistics: my reactions to Cox and Mayo
10 0.09383674 696 andrew gelman stats-2011-05-04-Whassup with glm()?
11 0.088499792 1516 andrew gelman stats-2012-09-30-Computational problems with glm etc.
12 0.08824192 729 andrew gelman stats-2011-05-24-Deviance as a difference
14 0.080625951 1695 andrew gelman stats-2013-01-28-Economists argue about Bayes
15 0.079330452 2109 andrew gelman stats-2013-11-21-Hidden dangers of noninformative priors
16 0.07708361 1205 andrew gelman stats-2012-03-09-Coming to agreement on philosophy of statistics
17 0.076890141 1469 andrew gelman stats-2012-08-25-Ways of knowing
18 0.076038286 2017 andrew gelman stats-2013-09-11-“Informative g-Priors for Logistic Regression”
19 0.075670719 1859 andrew gelman stats-2013-05-16-How do we choose our default methods?
topicId topicWeight
[(0, 0.131), (1, 0.082), (2, -0.029), (3, 0.024), (4, -0.0), (5, 0.006), (6, -0.026), (7, 0.009), (8, 0.005), (9, -0.014), (10, 0.028), (11, -0.007), (12, -0.027), (13, 0.017), (14, -0.023), (15, 0.008), (16, 0.016), (17, -0.006), (18, 0.009), (19, 0.019), (20, 0.027), (21, 0.049), (22, 0.071), (23, -0.005), (24, 0.067), (25, 0.015), (26, 0.017), (27, 0.0), (28, 0.009), (29, -0.002), (30, -0.016), (31, 0.062), (32, 0.076), (33, -0.012), (34, 0.042), (35, -0.017), (36, -0.01), (37, -0.043), (38, 0.032), (39, -0.024), (40, -0.043), (41, 0.014), (42, -0.039), (43, -0.013), (44, 0.034), (45, 0.027), (46, 0.017), (47, -0.044), (48, -0.04), (49, 0.019)]
simIndex simValue blogId blogTitle
same-blog 1 0.96137202 1975 andrew gelman stats-2013-08-09-Understanding predictive information criteria for Bayesian models
Introduction: Jessy, Aki, and I write : We review the Akaike, deviance, and Watanabe-Akaike information criteria from a Bayesian perspective, where the goal is to estimate expected out-of-sample-prediction error using a bias-corrected adjustment of within-sample error. We focus on the choices involved in setting up these measures, and we compare them in three simple examples, one theoretical and two applied. The contribution of this review is to put all these information criteria into a Bayesian predictive context and to better understand, through small examples, how these methods can apply in practice. I like this paper. It came about as a result of preparing Chapter 7 for the new BDA . I had difficulty understanding AIC, DIC, WAIC, etc., but I recognized that these methods served a need. My first plan was to just apply DIC and WAIC on a couple of simple examples (a linear regression and the 8 schools) and leave it at that. But when I did the calculations, I couldn’t understand the resu
Introduction: Aki Vehtari and Janne Ojanen just published a long paper that begins: To date, several methods exist in the statistical literature for model assessment, which purport themselves specifically as Bayesian predictive methods. The decision theoretic assumptions on which these methods are based are not always clearly stated in the original articles, however. The aim of this survey is to provide a unified review of Bayesian predictive model assessment and selection methods, and of methods closely related to them. We review the various assumptions that are made in this context and discuss the connections between different approaches, with an emphasis on how each method approximates the expected utility of using a Bayesian model for the purpose of predicting future data. AIC (which Akaike called “An Information Criterion”) is the starting point for all these methods. More recently, Watanabe came up with WAIC (which he called the “Widely Available Information Criterion”). In between t
3 0.76245296 778 andrew gelman stats-2011-06-24-New ideas on DIC from Martyn Plummer and Sumio Watanabe
Introduction: Martyn Plummer replied to my recent blog on DIC with information that was important enough that I thought it deserved its own blog entry. Martyn wrote: DIC has been around for 10 years now and despite being immensely popular with applied statisticians it has generated very little theoretical interest. In fact, the silence has been deafening. I [Martyn] hope my paper added some clarity. As you say, DIC is (an approximation to) a theoretical out-of-sample predictive error. When I finished the paper I was a little embarrassed to see that I had almost perfectly reconstructed the justification of AIC as approximate cross-validation measure by Stone (1977), with a Bayesian spin of course. But even this insight leaves a lot of choices open. You need to choose the right loss function and also which level of the model you want to replicate from. David Spiegelhalter and colleagues called this the “focus”. In practice the focus is limited to the lowest level of the model. You generall
4 0.73068058 776 andrew gelman stats-2011-06-22-Deviance, DIC, AIC, cross-validation, etc
Introduction: The deviance information criterion (or DIC) is an idea of Brad Carlin and others for comparing the fits of models estimated using Bayesian simulation (for more information, see this article by Angelika van der Linde). I don’t really ever know what to make of DIC. On one hand, it seems sensible, it handles uncertainty in inferences within each model, and it does not depend on aspects of the models that don’t affect inferences within each model (unlike Bayes factors; see discussion here ). On the other hand, I don’t really have any idea what I would do with DIC in any real example. In our book we included an example of DIC–people use it and we don’t have any great alternatives–but I had to be pretty careful that the example made sense. Unlike the usual setting where we use a method and that gives us insight into a problem, here we used our insight into the problem to make sure that in this particular case the method gave a reasonable answer. One of my practical problems with D
5 0.68552268 2349 andrew gelman stats-2014-05-26-WAIC and cross-validation in Stan!
Introduction: Aki and I write : The Watanabe-Akaike information criterion (WAIC) and cross-validation are methods for estimating pointwise out-of-sample prediction accuracy from a fitted Bayesian model. WAIC is based on the series expansion of leave-one-out cross-validation (LOO), and asymptotically they are equal. With finite data, WAIC and cross-validation address different predictive questions and thus it is useful to be able to compute both. WAIC and an importance-sampling approximated LOO can be estimated directly using the log-likelihood evaluated at the posterior simulations of the parameter values. We show how to compute WAIC, IS-LOO, K-fold cross-validation, and related diagnostic quantities in the Bayesian inference package Stan as called from R. This is important, I think. One reason the deviance information criterion (DIC) has been so popular is its implementation in Bugs. We think WAIC and cross-validation make more sense than DIC, especially from a Bayesian perspective in whic
6 0.6683194 1516 andrew gelman stats-2012-09-30-Computational problems with glm etc.
7 0.66349405 2311 andrew gelman stats-2014-04-29-Bayesian Uncertainty Quantification for Differential Equations!
8 0.63350546 2001 andrew gelman stats-2013-08-29-Edgar Allan Poe was a statistician
9 0.62901312 1377 andrew gelman stats-2012-06-13-A question about AIC
10 0.61271948 291 andrew gelman stats-2010-09-22-Philosophy of Bayes and non-Bayes: A dialogue with Deborah Mayo
11 0.61067712 449 andrew gelman stats-2010-12-04-Generalized Method of Moments, whatever that is
12 0.60991848 1571 andrew gelman stats-2012-11-09-The anti-Bayesian moment and its passing
13 0.6032415 1560 andrew gelman stats-2012-11-03-Statistical methods that work in some settings but not others
15 0.59272081 2254 andrew gelman stats-2014-03-18-Those wacky anti-Bayesians used to be intimidating, but now they’re just pathetic
16 0.59227198 247 andrew gelman stats-2010-09-01-How does Bayes do it?
17 0.5854187 1469 andrew gelman stats-2012-08-25-Ways of knowing
18 0.58500171 214 andrew gelman stats-2010-08-17-Probability-processing hardware
19 0.57960618 1374 andrew gelman stats-2012-06-11-Convergence Monitoring for Non-Identifiable and Non-Parametric Models
20 0.57650208 1019 andrew gelman stats-2011-11-19-Validation of Software for Bayesian Models Using Posterior Quantiles
topicId topicWeight
[(15, 0.012), (16, 0.089), (24, 0.134), (30, 0.06), (34, 0.013), (41, 0.017), (61, 0.229), (63, 0.024), (86, 0.019), (89, 0.016), (99, 0.275)]
simIndex simValue blogId blogTitle
1 0.9562903 1558 andrew gelman stats-2012-11-02-Not so fast on levees and seawalls for NY harbor?
Introduction: I was talking with June Williamson and mentioned offhand that I’d seen something in the paper saying that if only we’d invested a few billion dollars in levees we would’ve saved zillions in economic damage from the flood. (A quick search also revealed this eerily prescient article from last month and, more recently, this online discussion.) June said, No, no, no: levees are not the way to go: Here and here are the articles on “soft infrastructure” for the New York-New Jersey Harbor I was mentioning, summarizing work that is more extensively published in two books, “Rising Currents” and “On the Water: Palisade Bay”: The hazards posed by climate change, sea level rise, and severe storm surges make this the time to transform our coastal cities through adaptive design. The conventional response to flooding, in recent history, has been hard engineering — fortifying the coastal infrastructure with seawalls and bulkheads to protect real estate at the expense of natural t
2 0.94259965 16 andrew gelman stats-2010-05-04-Burgess on Kipling
Introduction: This is my last entry derived from Anthony Burgess’s book reviews , and it’ll be short. His review of Angus Wilson’s “The Strange Ride of Rudyard Kipling: His Life and Works” is a wonderfully balanced little thing. Nothing incredibly deep–like most items in the collection, the review is only two pages long–but I give it credit for being a rare piece of Kipling criticism I’ve seen that (a) seriously engages with the politics, without (b) congratulating itself on bravely going against the fashions of the politically incorrect chattering classes by celebrating Kipling’s magnificent achievement blah blah blah. Instead, Burgess shows respect for Kipling’s work and puts it in historical, biographical, and literary context. Burgess concludes that Wilson’s book “reminds us, in John Gross’s words, that Kipling ‘remains a haunting, unsettling presence, with whom we still have to come to terms.’ Still.” Well put, and generous of Burgess to end his review with another’s quote. Other cri
same-blog 3 0.93519533 1975 andrew gelman stats-2013-08-09-Understanding predictive information criteria for Bayesian models
Introduction: Jessy, Aki, and I write : We review the Akaike, deviance, and Watanabe-Akaike information criteria from a Bayesian perspective, where the goal is to estimate expected out-of-sample-prediction error using a bias-corrected adjustment of within-sample error. We focus on the choices involved in setting up these measures, and we compare them in three simple examples, one theoretical and two applied. The contribution of this review is to put all these information criteria into a Bayesian predictive context and to better understand, through small examples, how these methods can apply in practice. I like this paper. It came about as a result of preparing Chapter 7 for the new BDA . I had difficulty understanding AIC, DIC, WAIC, etc., but I recognized that these methods served a need. My first plan was to just apply DIC and WAIC on a couple of simple examples (a linear regression and the 8 schools) and leave it at that. But when I did the calculations, I couldn’t understand the resu
4 0.92437899 1028 andrew gelman stats-2011-11-26-Tenure lets you handle students who cheat
Introduction: The other day, a friend of mine who is an untenured professor (not in statistics or political science) was telling me about a class where many of the students seemed to be resubmitting papers that they had already written for previous classes. (The supposition was based on internal evidence of the topics of the submitted papers.) It would be possible to check this and then kick the cheating students out of the program—but why do it? It would be a lot of work, also some of the students who are caught might complain, then word would get around that my friend is a troublemaker. And nobody likes a troublemaker. Once my friend has tenure it would be possible to do the right thing. But . . . here’s the hitch: most college instructors do not have tenure, and one result, I suspect, is a decline in ethical standards. This is something I hadn’t thought of in our earlier discussion of job security for teachers: tenure gives you the freedom to kick out cheating students.
5 0.91748834 2349 andrew gelman stats-2014-05-26-WAIC and cross-validation in Stan!
Introduction: Aki and I write : The Watanabe-Akaike information criterion (WAIC) and cross-validation are methods for estimating pointwise out-of-sample prediction accuracy from a fitted Bayesian model. WAIC is based on the series expansion of leave-one-out cross-validation (LOO), and asymptotically they are equal. With finite data, WAIC and cross-validation address different predictive questions and thus it is useful to be able to compute both. WAIC and an importance-sampling approximated LOO can be estimated directly using the log-likelihood evaluated at the posterior simulations of the parameter values. We show how to compute WAIC, IS-LOO, K-fold cross-validation, and related diagnostic quantities in the Bayesian inference package Stan as called from R. This is important, I think. One reason the deviance information criterion (DIC) has been so popular is its implementation in Bugs. We think WAIC and cross-validation make more sense than DIC, especially from a Bayesian perspective in whic
6 0.91274023 1370 andrew gelman stats-2012-06-07-Duncan Watts and the Titanic
7 0.91252148 714 andrew gelman stats-2011-05-16-NYT Labs releases Openpaths, a utility for saving your iphone data
9 0.9030627 9 andrew gelman stats-2010-04-28-But it all goes to pay for gas, car insurance, and tolls on the turnpike
10 0.89377171 21 andrew gelman stats-2010-05-07-Environmentally induced cancer “grossly underestimated”? Doubtful.
11 0.88541305 827 andrew gelman stats-2011-07-28-Amusing case of self-defeating science writing
12 0.87086129 1433 andrew gelman stats-2012-07-28-LOL without the CATS
13 0.86720061 1714 andrew gelman stats-2013-02-09-Partial least squares path analysis
14 0.86315262 1739 andrew gelman stats-2013-02-26-An AI can build and try out statistical models using an open-ended generative grammar
15 0.85784805 561 andrew gelman stats-2011-02-06-Poverty, educational performance – and can be done about it
16 0.8574751 776 andrew gelman stats-2011-06-22-Deviance, DIC, AIC, cross-validation, etc
17 0.85435545 2134 andrew gelman stats-2013-12-14-Oswald evidence
19 0.8493973 729 andrew gelman stats-2011-05-24-Deviance as a difference
20 0.84696048 2033 andrew gelman stats-2013-09-23-More on Bayesian methods and multilevel modeling