andrew_gelman_stats andrew_gelman_stats-2014 andrew_gelman_stats-2014-2349 knowledge-graph by maker-knowledge-mining
Source: html
Introduction: Aki and I write : The Watanabe-Akaike information criterion (WAIC) and cross-validation are methods for estimating pointwise out-of-sample prediction accuracy from a fitted Bayesian model. WAIC is based on the series expansion of leave-one-out cross-validation (LOO), and asymptotically they are equal. With finite data, WAIC and cross-validation address different predictive questions and thus it is useful to be able to compute both. WAIC and an importance-sampling approximated LOO can be estimated directly using the log-likelihood evaluated at the posterior simulations of the parameter values. We show how to compute WAIC, IS-LOO, K-fold cross-validation, and related diagnostic quantities in the Bayesian inference package Stan as called from R. This is important, I think. One reason the deviance information criterion (DIC) has been so popular is its implementation in Bugs. We think WAIC and cross-validation make more sense than DIC, especially from a Bayesian perspective in whic
sentIndex sentText sentNum sentScore
1 Aki and I write : The Watanabe-Akaike information criterion (WAIC) and cross-validation are methods for estimating pointwise out-of-sample prediction accuracy from a fitted Bayesian model. [sent-1, score-0.659]
2 WAIC is based on the series expansion of leave-one-out cross-validation (LOO), and asymptotically they are equal. [sent-2, score-0.246]
3 With finite data, WAIC and cross-validation address different predictive questions and thus it is useful to be able to compute both. [sent-3, score-0.361]
4 WAIC and an importance-sampling approximated LOO can be estimated directly using the log-likelihood evaluated at the posterior simulations of the parameter values. [sent-4, score-0.564]
5 We show how to compute WAIC, IS-LOO, K-fold cross-validation, and related diagnostic quantities in the Bayesian inference package Stan as called from R. [sent-5, score-0.496]
6 One reason the deviance information criterion (DIC) has been so popular is its implementation in Bugs. [sent-7, score-0.574]
7 We think WAIC and cross-validation make more sense than DIC, especially from a Bayesian perspective in which inference comes as a posterior distribution rather than a point estimate, and we hope that this and future Stan implementations will allow users to become more familiar with these tools. [sent-8, score-0.716]
8 In addition to the implementation, the paper discusses some challenges of interpretation with hierarchical models, demonstrating with the canonical 8 schools example. [sent-9, score-0.602]
wordName wordTfidf (topN-words)
[('waic', 0.676), ('loo', 0.293), ('dic', 0.22), ('criterion', 0.184), ('implementation', 0.159), ('compute', 0.149), ('pointwise', 0.133), ('approximated', 0.126), ('stan', 0.116), ('canonical', 0.113), ('implementations', 0.11), ('posterior', 0.11), ('diagnostic', 0.105), ('deviance', 0.105), ('bayesian', 0.104), ('asymptotically', 0.103), ('demonstrating', 0.093), ('aki', 0.093), ('expansion', 0.087), ('quantities', 0.087), ('evaluated', 0.085), ('inference', 0.085), ('finite', 0.082), ('simulations', 0.076), ('challenges', 0.075), ('fitted', 0.075), ('discusses', 0.072), ('users', 0.072), ('accuracy', 0.072), ('information', 0.07), ('package', 0.07), ('address', 0.069), ('schools', 0.067), ('familiar', 0.065), ('prediction', 0.064), ('interpretation', 0.063), ('allow', 0.062), ('addition', 0.062), ('estimating', 0.061), ('predictive', 0.061), ('estimated', 0.059), ('parameter', 0.058), ('hierarchical', 0.057), ('series', 0.056), ('popular', 0.056), ('future', 0.055), ('become', 0.054), ('hope', 0.052), ('perspective', 0.051), ('directly', 0.05)]
simIndex simValue blogId blogTitle
same-blog 1 1.0000001 2349 andrew gelman stats-2014-05-26-WAIC and cross-validation in Stan!
Introduction: Aki and I write : The Watanabe-Akaike information criterion (WAIC) and cross-validation are methods for estimating pointwise out-of-sample prediction accuracy from a fitted Bayesian model. WAIC is based on the series expansion of leave-one-out cross-validation (LOO), and asymptotically they are equal. With finite data, WAIC and cross-validation address different predictive questions and thus it is useful to be able to compute both. WAIC and an importance-sampling approximated LOO can be estimated directly using the log-likelihood evaluated at the posterior simulations of the parameter values. We show how to compute WAIC, IS-LOO, K-fold cross-validation, and related diagnostic quantities in the Bayesian inference package Stan as called from R. This is important, I think. One reason the deviance information criterion (DIC) has been so popular is its implementation in Bugs. We think WAIC and cross-validation make more sense than DIC, especially from a Bayesian perspective in whic
2 0.42952365 1975 andrew gelman stats-2013-08-09-Understanding predictive information criteria for Bayesian models
Introduction: Jessy, Aki, and I write : We review the Akaike, deviance, and Watanabe-Akaike information criteria from a Bayesian perspective, where the goal is to estimate expected out-of-sample-prediction error using a bias-corrected adjustment of within-sample error. We focus on the choices involved in setting up these measures, and we compare them in three simple examples, one theoretical and two applied. The contribution of this review is to put all these information criteria into a Bayesian predictive context and to better understand, through small examples, how these methods can apply in practice. I like this paper. It came about as a result of preparing Chapter 7 for the new BDA . I had difficulty understanding AIC, DIC, WAIC, etc., but I recognized that these methods served a need. My first plan was to just apply DIC and WAIC on a couple of simple examples (a linear regression and the 8 schools) and leave it at that. But when I did the calculations, I couldn’t understand the resu
3 0.23865055 776 andrew gelman stats-2011-06-22-Deviance, DIC, AIC, cross-validation, etc
Introduction: The deviance information criterion (or DIC) is an idea of Brad Carlin and others for comparing the fits of models estimated using Bayesian simulation (for more information, see this article by Angelika van der Linde). I don’t really ever know what to make of DIC. On one hand, it seems sensible, it handles uncertainty in inferences within each model, and it does not depend on aspects of the models that don’t affect inferences within each model (unlike Bayes factors; see discussion here ). On the other hand, I don’t really have any idea what I would do with DIC in any real example. In our book we included an example of DIC–people use it and we don’t have any great alternatives–but I had to be pretty careful that the example made sense. Unlike the usual setting where we use a method and that gives us insight into a problem, here we used our insight into the problem to make sure that in this particular case the method gave a reasonable answer. One of my practical problems with D
4 0.20228778 778 andrew gelman stats-2011-06-24-New ideas on DIC from Martyn Plummer and Sumio Watanabe
Introduction: Martyn Plummer replied to my recent blog on DIC with information that was important enough that I thought it deserved its own blog entry. Martyn wrote: DIC has been around for 10 years now and despite being immensely popular with applied statisticians it has generated very little theoretical interest. In fact, the silence has been deafening. I [Martyn] hope my paper added some clarity. As you say, DIC is (an approximation to) a theoretical out-of-sample predictive error. When I finished the paper I was a little embarrassed to see that I had almost perfectly reconstructed the justification of AIC as approximate cross-validation measure by Stone (1977), with a Bayesian spin of course. But even this insight leaves a lot of choices open. You need to choose the right loss function and also which level of the model you want to replicate from. David Spiegelhalter and colleagues called this the “focus”. In practice the focus is limited to the lowest level of the model. You generall
Introduction: Aki Vehtari and Janne Ojanen just published a long paper that begins: To date, several methods exist in the statistical literature for model assessment, which purport themselves specifically as Bayesian predictive methods. The decision theoretic assumptions on which these methods are based are not always clearly stated in the original articles, however. The aim of this survey is to provide a unified review of Bayesian predictive model assessment and selection methods, and of methods closely related to them. We review the various assumptions that are made in this context and discuss the connections between different approaches, with an emphasis on how each method approximates the expected utility of using a Bayesian model for the purpose of predicting future data. AIC (which Akaike called “An Information Criterion”) is the starting point for all these methods. More recently, Watanabe came up with WAIC (which he called the “Widely Available Information Criterion”). In between t
6 0.17956018 2348 andrew gelman stats-2014-05-26-On deck this week
7 0.1684521 1983 andrew gelman stats-2013-08-15-More on AIC, WAIC, etc
9 0.10204703 2208 andrew gelman stats-2014-02-12-How to think about “identifiability” in Bayesian inference?
10 0.10165203 1779 andrew gelman stats-2013-03-27-“Two Dogmas of Strong Objective Bayesianism”
11 0.093414083 1363 andrew gelman stats-2012-06-03-Question about predictive checks
12 0.091319382 1748 andrew gelman stats-2013-03-04-PyStan!
13 0.090316951 2368 andrew gelman stats-2014-06-11-Bayes in the research conversation
14 0.089181095 1221 andrew gelman stats-2012-03-19-Whassup with deviance having a high posterior correlation with a parameter in the model?
15 0.086108953 1475 andrew gelman stats-2012-08-30-A Stan is Born
16 0.085191339 1469 andrew gelman stats-2012-08-25-Ways of knowing
17 0.084385172 2124 andrew gelman stats-2013-12-05-Stan (quietly) passes 512 people on the users list
19 0.080345549 1377 andrew gelman stats-2012-06-13-A question about AIC
20 0.079576656 2161 andrew gelman stats-2014-01-07-My recent debugging experience
topicId topicWeight
[(0, 0.11), (1, 0.137), (2, -0.028), (3, 0.053), (4, -0.008), (5, 0.031), (6, -0.032), (7, -0.066), (8, -0.042), (9, -0.072), (10, -0.045), (11, -0.007), (12, -0.054), (13, 0.016), (14, 0.02), (15, 0.002), (16, 0.009), (17, 0.023), (18, -0.013), (19, 0.033), (20, -0.002), (21, 0.029), (22, 0.031), (23, -0.0), (24, 0.07), (25, 0.009), (26, 0.006), (27, 0.016), (28, -0.001), (29, 0.021), (30, -0.007), (31, 0.038), (32, 0.052), (33, -0.018), (34, 0.01), (35, 0.016), (36, 0.016), (37, -0.038), (38, 0.02), (39, -0.019), (40, -0.029), (41, 0.036), (42, -0.054), (43, -0.022), (44, 0.002), (45, -0.02), (46, 0.028), (47, 0.002), (48, -0.021), (49, -0.004)]
simIndex simValue blogId blogTitle
same-blog 1 0.95993054 2349 andrew gelman stats-2014-05-26-WAIC and cross-validation in Stan!
Introduction: Aki and I write : The Watanabe-Akaike information criterion (WAIC) and cross-validation are methods for estimating pointwise out-of-sample prediction accuracy from a fitted Bayesian model. WAIC is based on the series expansion of leave-one-out cross-validation (LOO), and asymptotically they are equal. With finite data, WAIC and cross-validation address different predictive questions and thus it is useful to be able to compute both. WAIC and an importance-sampling approximated LOO can be estimated directly using the log-likelihood evaluated at the posterior simulations of the parameter values. We show how to compute WAIC, IS-LOO, K-fold cross-validation, and related diagnostic quantities in the Bayesian inference package Stan as called from R. This is important, I think. One reason the deviance information criterion (DIC) has been so popular is its implementation in Bugs. We think WAIC and cross-validation make more sense than DIC, especially from a Bayesian perspective in whic
2 0.80274391 1749 andrew gelman stats-2013-03-04-Stan in L.A. this Wed 3:30pm
Introduction: Michael Betancourt will be speaking at UCLA: The location for refreshment is in room 51-254 CHS at 3:00 PM. The place for the seminar is at CHS 33-105A at 3:30pm – 4:30pm, Wed 6 Mar. ["CHS" stands for Center for Health Sciences, the building of the UCLA schools of medicine and public health. Here's a map with directions .] Title of talk: Stan : Practical Bayesian Inference with Hamiltonian Monte Carlo Abstract: Practical implementations of Bayesian inference are often limited to approximation methods that only slowly explore the posterior distribution. By taking advantage of the curvature of the posterior, however, Hamiltonian Monte Carlo (HMC) efficiently explores even the most highly contorted distributions. In this talk I will review the foundations of and recent developments within HMC, concluding with a discussion of Stan, a powerful inference engine that utilizes HMC, automatic differentiation, and adaptive methods to minimize user input. This is cool stuff.
3 0.74578154 1019 andrew gelman stats-2011-11-19-Validation of Software for Bayesian Models Using Posterior Quantiles
Introduction: I love this stuff : This article presents a simulation-based method designed to establish the computational correctness of software developed to fit a specific Bayesian model, capitalizing on properties of Bayesian posterior distributions. We illustrate the validation technique with two examples. The validation method is shown to find errors in software when they exist and, moreover, the validation output can be informative about the nature and location of such errors. We also compare our method with that of an earlier approach. I hope we can put it into Stan.
4 0.72305501 1772 andrew gelman stats-2013-03-20-Stan at Google this Thurs and at Berkeley this Fri noon
Introduction: Michael Betancourt will be speaking at Google and at the University of California, Berkeley. The Google talk is closed to outsiders (but if you work at Google, you should go!); the Berkeley talk is open to all: Friday March 22, 12:10 pm, Evans Hall 1011. Title of talk: Stan : Practical Bayesian Inference with Hamiltonian Monte Carlo Abstract: Practical implementations of Bayesian inference are often limited to approximation methods that only slowly explore the posterior distribution. By taking advantage of the curvature of the posterior, however, Hamiltonian Monte Carlo (HMC) efficiently explores even the most highly contorted distributions. In this talk I will review the foundations of and recent developments within HMC, concluding with a discussion of Stan, a powerful inference engine that utilizes HMC, automatic differentiation, and adaptive methods to minimize user input. This is cool stuff. And he’ll be showing the whirlpool movie!
5 0.71697938 1528 andrew gelman stats-2012-10-10-My talk at MIT on Thurs 11 Oct
Introduction: Stan: open-source Bayesian inference Speaker: Andrew Gelman, Columbia University Date: Thursday, October 11 2012 Time: 4:00PM to 5:00PM Location: 32-D507 Host: Polina Golland, CSAIL Contact: Polina Golland, 6172538005, polina@csail.mit.edu Stan ( mc-stan.org ) is an open-source package for obtaining Bayesian inference using the No-U-Turn sampler, a variant of Hamiltonian Monte Carlo. We discuss how Stan works and what it can do, the problems that motivated us to write Stan, current challenges, and areas of planned development, including tools for improved generality and usability, more efficient sampling algorithms, and fuller integration of model building, model checking, and model understanding in Bayesian data analysis. P.S. Here’s the talk .
6 0.70034724 1975 andrew gelman stats-2013-08-09-Understanding predictive information criteria for Bayesian models
7 0.69239789 2003 andrew gelman stats-2013-08-30-Stan Project: Continuous Relaxations for Discrete MRFs
9 0.67048413 2231 andrew gelman stats-2014-03-03-Running into a Stan Reference by Accident
10 0.66988903 2020 andrew gelman stats-2013-09-12-Samplers for Big Science: emcee and BAT
11 0.66916066 778 andrew gelman stats-2011-06-24-New ideas on DIC from Martyn Plummer and Sumio Watanabe
12 0.66722542 1036 andrew gelman stats-2011-11-30-Stan uses Nuts!
14 0.63251501 1627 andrew gelman stats-2012-12-17-Stan and RStan 1.1.0
15 0.62374032 1710 andrew gelman stats-2013-02-06-The new Stan 1.1.1, featuring Gaussian processes!
16 0.62344062 2360 andrew gelman stats-2014-06-05-Identifying pathways for managing multiple disturbances to limit plant invasions
17 0.62261993 1868 andrew gelman stats-2013-05-23-Validation of Software for Bayesian Models Using Posterior Quantiles
18 0.62036431 449 andrew gelman stats-2010-12-04-Generalized Method of Moments, whatever that is
19 0.61901873 2208 andrew gelman stats-2014-02-12-How to think about “identifiability” in Bayesian inference?
20 0.61388034 2150 andrew gelman stats-2013-12-27-(R-Py-Cmd)Stan 2.1.0
topicId topicWeight
[(13, 0.015), (16, 0.056), (20, 0.074), (24, 0.11), (37, 0.019), (59, 0.016), (61, 0.235), (82, 0.046), (86, 0.041), (99, 0.262)]
simIndex simValue blogId blogTitle
same-blog 1 0.93548036 2349 andrew gelman stats-2014-05-26-WAIC and cross-validation in Stan!
Introduction: Aki and I write : The Watanabe-Akaike information criterion (WAIC) and cross-validation are methods for estimating pointwise out-of-sample prediction accuracy from a fitted Bayesian model. WAIC is based on the series expansion of leave-one-out cross-validation (LOO), and asymptotically they are equal. With finite data, WAIC and cross-validation address different predictive questions and thus it is useful to be able to compute both. WAIC and an importance-sampling approximated LOO can be estimated directly using the log-likelihood evaluated at the posterior simulations of the parameter values. We show how to compute WAIC, IS-LOO, K-fold cross-validation, and related diagnostic quantities in the Bayesian inference package Stan as called from R. This is important, I think. One reason the deviance information criterion (DIC) has been so popular is its implementation in Bugs. We think WAIC and cross-validation make more sense than DIC, especially from a Bayesian perspective in whic
2 0.93074107 1558 andrew gelman stats-2012-11-02-Not so fast on levees and seawalls for NY harbor?
Introduction: I was talking with June Williamson and mentioned offhand that I’d seen something in the paper saying that if only we’d invested a few billion dollars in levees we would’ve saved zillions in economic damage from the flood. (A quick search also revealed this eerily prescient article from last month and, more recently, this online discussion.) June said, No, no, no: levees are not the way to go: Here and here are the articles on “soft infrastructure” for the New York-New Jersey Harbor I was mentioning, summarizing work that is more extensively published in two books, “Rising Currents” and “On the Water: Palisade Bay”: The hazards posed by climate change, sea level rise, and severe storm surges make this the time to transform our coastal cities through adaptive design. The conventional response to flooding, in recent history, has been hard engineering — fortifying the coastal infrastructure with seawalls and bulkheads to protect real estate at the expense of natural t
3 0.91106045 16 andrew gelman stats-2010-05-04-Burgess on Kipling
Introduction: This is my last entry derived from Anthony Burgess’s book reviews , and it’ll be short. His review of Angus Wilson’s “The Strange Ride of Rudyard Kipling: His Life and Works” is a wonderfully balanced little thing. Nothing incredibly deep–like most items in the collection, the review is only two pages long–but I give it credit for being a rare piece of Kipling criticism I’ve seen that (a) seriously engages with the politics, without (b) congratulating itself on bravely going against the fashions of the politically incorrect chattering classes by celebrating Kipling’s magnificent achievement blah blah blah. Instead, Burgess shows respect for Kipling’s work and puts it in historical, biographical, and literary context. Burgess concludes that Wilson’s book “reminds us, in John Gross’s words, that Kipling ‘remains a haunting, unsettling presence, with whom we still have to come to terms.’ Still.” Well put, and generous of Burgess to end his review with another’s quote. Other cri
4 0.90071541 1028 andrew gelman stats-2011-11-26-Tenure lets you handle students who cheat
Introduction: The other day, a friend of mine who is an untenured professor (not in statistics or political science) was telling me about a class where many of the students seemed to be resubmitting papers that they had already written for previous classes. (The supposition was based on internal evidence of the topics of the submitted papers.) It would be possible to check this and then kick the cheating students out of the program—but why do it? It would be a lot of work, also some of the students who are caught might complain, then word would get around that my friend is a troublemaker. And nobody likes a troublemaker. Once my friend has tenure it would be possible to do the right thing. But . . . here’s the hitch: most college instructors do not have tenure, and one result, I suspect, is a decline in ethical standards. This is something I hadn’t thought of in our earlier discussion of job security for teachers: tenure gives you the freedom to kick out cheating students.
5 0.89791203 1975 andrew gelman stats-2013-08-09-Understanding predictive information criteria for Bayesian models
Introduction: Jessy, Aki, and I write : We review the Akaike, deviance, and Watanabe-Akaike information criteria from a Bayesian perspective, where the goal is to estimate expected out-of-sample-prediction error using a bias-corrected adjustment of within-sample error. We focus on the choices involved in setting up these measures, and we compare them in three simple examples, one theoretical and two applied. The contribution of this review is to put all these information criteria into a Bayesian predictive context and to better understand, through small examples, how these methods can apply in practice. I like this paper. It came about as a result of preparing Chapter 7 for the new BDA . I had difficulty understanding AIC, DIC, WAIC, etc., but I recognized that these methods served a need. My first plan was to just apply DIC and WAIC on a couple of simple examples (a linear regression and the 8 schools) and leave it at that. But when I did the calculations, I couldn’t understand the resu
6 0.88938332 1370 andrew gelman stats-2012-06-07-Duncan Watts and the Titanic
7 0.88744444 9 andrew gelman stats-2010-04-28-But it all goes to pay for gas, car insurance, and tolls on the turnpike
9 0.8780452 714 andrew gelman stats-2011-05-16-NYT Labs releases Openpaths, a utility for saving your iphone data
10 0.86806208 21 andrew gelman stats-2010-05-07-Environmentally induced cancer “grossly underestimated”? Doubtful.
11 0.86401927 827 andrew gelman stats-2011-07-28-Amusing case of self-defeating science writing
12 0.85414439 1714 andrew gelman stats-2013-02-09-Partial least squares path analysis
13 0.84272981 1433 andrew gelman stats-2012-07-28-LOL without the CATS
14 0.83579624 776 andrew gelman stats-2011-06-22-Deviance, DIC, AIC, cross-validation, etc
15 0.82958555 561 andrew gelman stats-2011-02-06-Poverty, educational performance – and can be done about it
16 0.82653177 2134 andrew gelman stats-2013-12-14-Oswald evidence
17 0.82421017 1739 andrew gelman stats-2013-02-26-An AI can build and try out statistical models using an open-ended generative grammar
19 0.81676334 2033 andrew gelman stats-2013-09-23-More on Bayesian methods and multilevel modeling
20 0.80745769 729 andrew gelman stats-2011-05-24-Deviance as a difference