andrew_gelman_stats andrew_gelman_stats-2011 andrew_gelman_stats-2011-778 knowledge-graph by maker-knowledge-mining
Source: html
Introduction: Martyn Plummer replied to my recent blog on DIC with information that was important enough that I thought it deserved its own blog entry. Martyn wrote: DIC has been around for 10 years now and despite being immensely popular with applied statisticians it has generated very little theoretical interest. In fact, the silence has been deafening. I [Martyn] hope my paper added some clarity. As you say, DIC is (an approximation to) a theoretical out-of-sample predictive error. When I finished the paper I was a little embarrassed to see that I had almost perfectly reconstructed the justification of AIC as approximate cross-validation measure by Stone (1977), with a Bayesian spin of course. But even this insight leaves a lot of choices open. You need to choose the right loss function and also which level of the model you want to replicate from. David Spiegelhalter and colleagues called this the “focus”. In practice the focus is limited to the lowest level of the model. You generall
sentIndex sentText sentNum sentScore
1 Martyn Plummer replied to my recent blog on DIC with information that was important enough that I thought it deserved its own blog entry. [sent-1, score-0.201]
2 When I finished the paper I was a little embarrassed to see that I had almost perfectly reconstructed the justification of AIC as approximate cross-validation measure by Stone (1977), with a Bayesian spin of course. [sent-6, score-0.13]
3 You need to choose the right loss function and also which level of the model you want to replicate from. [sent-8, score-0.24]
4 You generally can’t calculate the log likelihood (the default penalty) for higher level parameters. [sent-11, score-0.146]
5 For example, in disease mapping DIC answers the question “What model yield the disease map that best captures the features of the observed incidence data during this period? [sent-13, score-0.246]
6 ” But people are often asking more fundamental questions about their models, like “Is there spatial aggregation in disease X? [sent-14, score-0.179]
7 It can be calculated from 2 or more parallel chains and so its sample variance can be estimated using standard MCMC diagnostics. [sent-18, score-0.238]
8 The steps are (or should be): - Compile a model with at least 2 parallel chains - Load the dic module - Set a trace monitor for “pD”. [sent-20, score-0.736]
9 samples function from the rjags package will give you this in a nice R object wrapper. [sent-22, score-0.128]
10 Aki Vehtari commented too, with a link to a recent article by Sumio Watanabe on something called the widely applicable information criterion. [sent-24, score-0.605]
11 Watanabe’s article begins: In regular statistical models, the leave-one-out cross-validation is asymptotically equivalent to the Akaike information criterion. [sent-25, score-0.476]
12 However, since many learning machines are singular statistical models, the asymptotic behavior of the cross-validation remains unknown. [sent-26, score-0.206]
13 In previous studies, we established the singular learning theory and proposed a widely applicable information criterion, the expectation value of which is asymptotically equal to the average Bayes generalization loss. [sent-27, score-1.289]
14 In the present paper, we theoretically compare the Bayes cross-validation loss and the widely applicable information criterion and prove two theorems. [sent-28, score-0.86]
15 First, the Bayes cross-validation loss is asymptotically equivalent to the widely applicable information criterion as a random variable. [sent-29, score-1.193]
16 Therefore, model selection and hyperparameter optimization using these two values are asymptotically equivalent. [sent-30, score-0.321]
17 Second, the sum of the Bayes generalization error and the Bayes cross-validation error is asymptotically equal to 2λ/n, where λ is the real log canonical threshold and n is the number of training samples. [sent-31, score-0.808]
18 Therefore the relation between the cross-validation error and the generalization error is determined by the algebraic geometrical structure of a learning machine. [sent-32, score-0.552]
19 We also clarify that the deviance information criteria are different from the Bayes cross-validation and the widely applicable information criterion. [sent-33, score-0.748]
20 I think that Spiegelhalter, Best, Carlin, and Van der Linde made an important contribution with their DIC paper ten years ago. [sent-36, score-0.132]
wordName wordTfidf (topN-words)
[('dic', 0.448), ('applicable', 0.28), ('asymptotically', 0.263), ('martyn', 0.193), ('spiegelhalter', 0.184), ('widely', 0.182), ('bayes', 0.18), ('pd', 0.15), ('information', 0.143), ('criterion', 0.141), ('generalization', 0.141), ('watanabe', 0.128), ('disease', 0.123), ('singular', 0.119), ('loss', 0.114), ('error', 0.096), ('chains', 0.089), ('parallel', 0.088), ('learning', 0.087), ('log', 0.08), ('equal', 0.074), ('paper', 0.071), ('equivalent', 0.07), ('therefore', 0.069), ('geometrical', 0.068), ('coda', 0.068), ('rjags', 0.068), ('level', 0.066), ('algebraic', 0.064), ('linde', 0.064), ('plummer', 0.064), ('variance', 0.061), ('der', 0.061), ('function', 0.06), ('reconstructed', 0.059), ('bic', 0.059), ('vehtari', 0.059), ('theoretical', 0.058), ('coordinate', 0.058), ('load', 0.058), ('variants', 0.058), ('canonical', 0.058), ('hyperparameter', 0.058), ('deserved', 0.058), ('trace', 0.056), ('aggregation', 0.056), ('silence', 0.055), ('stone', 0.055), ('module', 0.055), ('focus', 0.054)]
simIndex simValue blogId blogTitle
same-blog 1 1.0000001 778 andrew gelman stats-2011-06-24-New ideas on DIC from Martyn Plummer and Sumio Watanabe
Introduction: Martyn Plummer replied to my recent blog on DIC with information that was important enough that I thought it deserved its own blog entry. Martyn wrote: DIC has been around for 10 years now and despite being immensely popular with applied statisticians it has generated very little theoretical interest. In fact, the silence has been deafening. I [Martyn] hope my paper added some clarity. As you say, DIC is (an approximation to) a theoretical out-of-sample predictive error. When I finished the paper I was a little embarrassed to see that I had almost perfectly reconstructed the justification of AIC as approximate cross-validation measure by Stone (1977), with a Bayesian spin of course. But even this insight leaves a lot of choices open. You need to choose the right loss function and also which level of the model you want to replicate from. David Spiegelhalter and colleagues called this the “focus”. In practice the focus is limited to the lowest level of the model. You generall
2 0.5190841 776 andrew gelman stats-2011-06-22-Deviance, DIC, AIC, cross-validation, etc
Introduction: The deviance information criterion (or DIC) is an idea of Brad Carlin and others for comparing the fits of models estimated using Bayesian simulation (for more information, see this article by Angelika van der Linde). I don’t really ever know what to make of DIC. On one hand, it seems sensible, it handles uncertainty in inferences within each model, and it does not depend on aspects of the models that don’t affect inferences within each model (unlike Bayes factors; see discussion here ). On the other hand, I don’t really have any idea what I would do with DIC in any real example. In our book we included an example of DIC–people use it and we don’t have any great alternatives–but I had to be pretty careful that the example made sense. Unlike the usual setting where we use a method and that gives us insight into a problem, here we used our insight into the problem to make sure that in this particular case the method gave a reasonable answer. One of my practical problems with D
3 0.31877893 1975 andrew gelman stats-2013-08-09-Understanding predictive information criteria for Bayesian models
Introduction: Jessy, Aki, and I write : We review the Akaike, deviance, and Watanabe-Akaike information criteria from a Bayesian perspective, where the goal is to estimate expected out-of-sample-prediction error using a bias-corrected adjustment of within-sample error. We focus on the choices involved in setting up these measures, and we compare them in three simple examples, one theoretical and two applied. The contribution of this review is to put all these information criteria into a Bayesian predictive context and to better understand, through small examples, how these methods can apply in practice. I like this paper. It came about as a result of preparing Chapter 7 for the new BDA . I had difficulty understanding AIC, DIC, WAIC, etc., but I recognized that these methods served a need. My first plan was to just apply DIC and WAIC on a couple of simple examples (a linear regression and the 8 schools) and leave it at that. But when I did the calculations, I couldn’t understand the resu
4 0.20228778 2349 andrew gelman stats-2014-05-26-WAIC and cross-validation in Stan!
Introduction: Aki and I write : The Watanabe-Akaike information criterion (WAIC) and cross-validation are methods for estimating pointwise out-of-sample prediction accuracy from a fitted Bayesian model. WAIC is based on the series expansion of leave-one-out cross-validation (LOO), and asymptotically they are equal. With finite data, WAIC and cross-validation address different predictive questions and thus it is useful to be able to compute both. WAIC and an importance-sampling approximated LOO can be estimated directly using the log-likelihood evaluated at the posterior simulations of the parameter values. We show how to compute WAIC, IS-LOO, K-fold cross-validation, and related diagnostic quantities in the Bayesian inference package Stan as called from R. This is important, I think. One reason the deviance information criterion (DIC) has been so popular is its implementation in Bugs. We think WAIC and cross-validation make more sense than DIC, especially from a Bayesian perspective in whic
Introduction: Aki Vehtari and Janne Ojanen just published a long paper that begins: To date, several methods exist in the statistical literature for model assessment, which purport themselves specifically as Bayesian predictive methods. The decision theoretic assumptions on which these methods are based are not always clearly stated in the original articles, however. The aim of this survey is to provide a unified review of Bayesian predictive model assessment and selection methods, and of methods closely related to them. We review the various assumptions that are made in this context and discuss the connections between different approaches, with an emphasis on how each method approximates the expected utility of using a Bayesian model for the purpose of predicting future data. AIC (which Akaike called “An Information Criterion”) is the starting point for all these methods. More recently, Watanabe came up with WAIC (which he called the “Widely Available Information Criterion”). In between t
6 0.16746072 1377 andrew gelman stats-2012-06-13-A question about AIC
7 0.16444996 1983 andrew gelman stats-2013-08-15-More on AIC, WAIC, etc
8 0.11142557 1045 andrew gelman stats-2011-12-07-Martyn Plummer’s Secret JAGS Blog
9 0.10963663 2140 andrew gelman stats-2013-12-19-Revised evidence for statistical standards
10 0.10750656 2231 andrew gelman stats-2014-03-03-Running into a Stan Reference by Accident
11 0.10345529 1221 andrew gelman stats-2012-03-19-Whassup with deviance having a high posterior correlation with a parameter in the model?
12 0.095965385 1695 andrew gelman stats-2013-01-28-Economists argue about Bayes
13 0.094581448 754 andrew gelman stats-2011-06-09-Difficulties with Bayesian model averaging
14 0.090217642 1392 andrew gelman stats-2012-06-26-Occam
15 0.089886189 846 andrew gelman stats-2011-08-09-Default priors update?
16 0.089726254 3 andrew gelman stats-2010-04-26-Bayes in the news…in a somewhat frustrating way
17 0.088122144 780 andrew gelman stats-2011-06-27-Bridges between deterministic and probabilistic models for binary data
18 0.087592363 427 andrew gelman stats-2010-11-23-Bayesian adaptive methods for clinical trials
19 0.086118706 1431 andrew gelman stats-2012-07-27-Overfitting
topicId topicWeight
[(0, 0.169), (1, 0.11), (2, 0.004), (3, 0.006), (4, 0.021), (5, 0.009), (6, -0.011), (7, -0.049), (8, -0.013), (9, -0.012), (10, -0.004), (11, 0.001), (12, -0.045), (13, 0.012), (14, -0.036), (15, -0.007), (16, 0.004), (17, 0.009), (18, -0.014), (19, 0.003), (20, 0.029), (21, 0.0), (22, 0.036), (23, -0.013), (24, 0.044), (25, 0.03), (26, -0.021), (27, 0.054), (28, 0.053), (29, 0.011), (30, -0.036), (31, 0.064), (32, 0.081), (33, -0.042), (34, 0.031), (35, -0.042), (36, -0.005), (37, -0.04), (38, 0.032), (39, -0.011), (40, -0.066), (41, -0.033), (42, -0.025), (43, 0.029), (44, 0.023), (45, 0.001), (46, 0.024), (47, -0.029), (48, -0.021), (49, -0.002)]
simIndex simValue blogId blogTitle
same-blog 1 0.95063472 778 andrew gelman stats-2011-06-24-New ideas on DIC from Martyn Plummer and Sumio Watanabe
Introduction: Martyn Plummer replied to my recent blog on DIC with information that was important enough that I thought it deserved its own blog entry. Martyn wrote: DIC has been around for 10 years now and despite being immensely popular with applied statisticians it has generated very little theoretical interest. In fact, the silence has been deafening. I [Martyn] hope my paper added some clarity. As you say, DIC is (an approximation to) a theoretical out-of-sample predictive error. When I finished the paper I was a little embarrassed to see that I had almost perfectly reconstructed the justification of AIC as approximate cross-validation measure by Stone (1977), with a Bayesian spin of course. But even this insight leaves a lot of choices open. You need to choose the right loss function and also which level of the model you want to replicate from. David Spiegelhalter and colleagues called this the “focus”. In practice the focus is limited to the lowest level of the model. You generall
2 0.80328035 776 andrew gelman stats-2011-06-22-Deviance, DIC, AIC, cross-validation, etc
Introduction: The deviance information criterion (or DIC) is an idea of Brad Carlin and others for comparing the fits of models estimated using Bayesian simulation (for more information, see this article by Angelika van der Linde). I don’t really ever know what to make of DIC. On one hand, it seems sensible, it handles uncertainty in inferences within each model, and it does not depend on aspects of the models that don’t affect inferences within each model (unlike Bayes factors; see discussion here ). On the other hand, I don’t really have any idea what I would do with DIC in any real example. In our book we included an example of DIC–people use it and we don’t have any great alternatives–but I had to be pretty careful that the example made sense. Unlike the usual setting where we use a method and that gives us insight into a problem, here we used our insight into the problem to make sure that in this particular case the method gave a reasonable answer. One of my practical problems with D
3 0.7556538 1975 andrew gelman stats-2013-08-09-Understanding predictive information criteria for Bayesian models
Introduction: Jessy, Aki, and I write : We review the Akaike, deviance, and Watanabe-Akaike information criteria from a Bayesian perspective, where the goal is to estimate expected out-of-sample-prediction error using a bias-corrected adjustment of within-sample error. We focus on the choices involved in setting up these measures, and we compare them in three simple examples, one theoretical and two applied. The contribution of this review is to put all these information criteria into a Bayesian predictive context and to better understand, through small examples, how these methods can apply in practice. I like this paper. It came about as a result of preparing Chapter 7 for the new BDA . I had difficulty understanding AIC, DIC, WAIC, etc., but I recognized that these methods served a need. My first plan was to just apply DIC and WAIC on a couple of simple examples (a linear regression and the 8 schools) and leave it at that. But when I did the calculations, I couldn’t understand the resu
4 0.7528671 2311 andrew gelman stats-2014-04-29-Bayesian Uncertainty Quantification for Differential Equations!
Introduction: Mark Girolami points us to this paper and software (with Oksana Chkrebtii, David Campbell, and Ben Calderhead). They write: We develop a general methodology for the probabilistic integration of differential equations via model based updating of a joint prior measure on the space of functions and their temporal and spatial derivatives. This results in a posterior measure over functions reflecting how well they satisfy the system of differential equations and corresponding initial and boundary values. We show how this posterior measure can be naturally incorporated within the Kennedy and O’Hagan framework for uncertainty quantification and provides a fully Bayesian approach to model calibration. . . . A broad variety of examples are provided to illustrate the potential of this framework for characterising discretization uncertainty, including initial value, delay, and boundary value differential equations, as well as partial differential equations. We also demonstrate our methodolo
5 0.75154436 1374 andrew gelman stats-2012-06-11-Convergence Monitoring for Non-Identifiable and Non-Parametric Models
Introduction: Becky Passonneau and colleagues at the Center for Computational Learning Systems (CCLS) at Columbia have been working on a project for ConEd (New York’s major electric utility) to rank structures based on vulnerability to secondary events (e.g., transformer explosions, cable meltdowns, electrical fires). They’ve been using the R implementation BayesTree of Chipman, George and McCulloch’s Bayesian Additive Regression Trees (BART). BART is a Bayesian non-parametric method that is non-identifiable in two ways. Firstly, it is an additive tree model with a fixed number of trees, the indexes of which aren’t identified (you get the same predictions in a model swapping the order of the trees). This is the same kind of non-identifiability you get with any mixture model (additive or interpolated) with an exchangeable prior on the mixture components. Secondly, the trees themselves have varying structure over samples in terms of number of nodes and their topology (depth, branching, etc
7 0.72441953 1983 andrew gelman stats-2013-08-15-More on AIC, WAIC, etc
9 0.70538861 1377 andrew gelman stats-2012-06-13-A question about AIC
10 0.68921298 674 andrew gelman stats-2011-04-21-Handbook of Markov Chain Monte Carlo
11 0.68164843 1991 andrew gelman stats-2013-08-21-BDA3 table of contents (also a new paper on visualization)
12 0.67723614 1739 andrew gelman stats-2013-02-26-An AI can build and try out statistical models using an open-ended generative grammar
13 0.67307454 810 andrew gelman stats-2011-07-20-Adding more information can make the variance go up (depending on your model)
14 0.66302097 2349 andrew gelman stats-2014-05-26-WAIC and cross-validation in Stan!
15 0.65702206 2277 andrew gelman stats-2014-03-31-The most-cited statistics papers ever
16 0.65557998 964 andrew gelman stats-2011-10-19-An interweaving-transformation strategy for boosting MCMC efficiency
17 0.63869572 822 andrew gelman stats-2011-07-26-Any good articles on the use of error bars?
18 0.63841599 1825 andrew gelman stats-2013-04-25-It’s binless! A program for computing normalizing functions
19 0.63838112 2315 andrew gelman stats-2014-05-02-Discovering general multidimensional associations
20 0.63691664 1363 andrew gelman stats-2012-06-03-Question about predictive checks
topicId topicWeight
[(13, 0.014), (16, 0.065), (21, 0.019), (24, 0.127), (41, 0.098), (53, 0.012), (56, 0.028), (59, 0.03), (61, 0.073), (86, 0.08), (95, 0.027), (99, 0.269)]
simIndex simValue blogId blogTitle
same-blog 1 0.96306384 778 andrew gelman stats-2011-06-24-New ideas on DIC from Martyn Plummer and Sumio Watanabe
Introduction: Martyn Plummer replied to my recent blog on DIC with information that was important enough that I thought it deserved its own blog entry. Martyn wrote: DIC has been around for 10 years now and despite being immensely popular with applied statisticians it has generated very little theoretical interest. In fact, the silence has been deafening. I [Martyn] hope my paper added some clarity. As you say, DIC is (an approximation to) a theoretical out-of-sample predictive error. When I finished the paper I was a little embarrassed to see that I had almost perfectly reconstructed the justification of AIC as approximate cross-validation measure by Stone (1977), with a Bayesian spin of course. But even this insight leaves a lot of choices open. You need to choose the right loss function and also which level of the model you want to replicate from. David Spiegelhalter and colleagues called this the “focus”. In practice the focus is limited to the lowest level of the model. You generall
Introduction: Wayne Folta points me to “EigenBracket 2012: Using Graph Theory to Predict NCAA March Madness Basketball” and writes, “I [Folta] have got to believe that he’s simply re-invented a statistical method in a graph-ish context, but don’t know enough to judge.” I have not looked in detail at the method being presented here—I’m not much of college basketball fan—but I’d like to use this as an excuse to make one of my favorite general point, which is that a good way to characterize any statistical method is by what information it uses. The basketball ranking method here uses score differentials between teams in the past season. On the plus side, that is better than simply using one-loss records (which (a) discards score differentials and (b) discards information on who played whom). On the minus side, the method appears to be discretizing the scores (thus throwing away information on the exact score differential) and doesn’t use any external information such as external ratings. A
3 0.93816346 1626 andrew gelman stats-2012-12-16-The lamest, grudgingest, non-retraction retraction ever
Introduction: In politics we’re familiar with the non-apology apology (well described in Wikipedia as “a statement that has the form of an apology but does not express the expected contrition”). Here’s the scientific equivalent: the non-retraction retraction. Sanjay Srivastava points to an amusing yet barfable story of a pair of researchers who (inadvertently, I assume) made a data coding error and were eventually moved to issue a correction notice, but even then refused to fully admit their error. As Srivastava puts it, the story “ended up with Lew [Goldberg] and colleagues [Kibeom Lee and Michael Ashton] publishing a comment on an erratum – the only time I’ve ever heard of that happening in a scientific journal.” From the comment on the erratum: In their “erratum and addendum,” Anderson and Ones (this issue) explained that we had brought their attention to the “potential” of a “possible” misalignment and described the results computed from re-aligned data as being based on a “post-ho
4 0.93402481 827 andrew gelman stats-2011-07-28-Amusing case of self-defeating science writing
Introduction: We’re all familiar with the gee-whiz style of science and technology writing in which hardly a day dawns without a cure for cancer, or a new pollution-free energy source, or some other amazing breakthrough. We don’t always get the privilege of seeing such reporting shot down the moment it hits the presses. Here’s journalist Matthew Philips: What does it take for an idea to spread from one to many? For a minority opinion to become the majority belief? According to a new study by scientists at the Rensselaer Polytechnic Institute, the answer is 10%. Once 10% of a population is committed to an idea, it’s inevitable that it will eventually become the prevailing opinion of the entire group. The key is to remain committed. . . . The research actually validates the entrenched strategy of the handful of House Republicans threatening to sink John Boehner‘s budget proposal. Turns out if you’re in the minority, you have less of an incentive to compromise than the majority does. Because if
5 0.93170625 303 andrew gelman stats-2010-09-28-“Genomics” vs. genetics
Introduction: John Cook and Joseph Delaney point to an article by Yurii Aulchenko et al., who write: 54 loci showing strong statistical evidence for association to human height were described, providing us with potential genomic means of human height prediction. In a population-based study of 5748 people, we find that a 54-loci genomic profile explained 4-6% of the sex- and age-adjusted height variance, and had limited ability to discriminate tall/short people. . . . In a family-based study of 550 people, with both parents having height measurements, we find that the Galtonian mid-parental prediction method explained 40% of the sex- and age-adjusted height variance, and showed high discriminative accuracy. . . . The message is that the simple approach of predicting child’s height using a regression model given parents’ average height performs much better than the method they have based on combining 54 genes. They also find that, if you start with the prediction based on parents’ heigh
6 0.92957926 516 andrew gelman stats-2011-01-14-A new idea for a science core course based entirely on computer simulation
7 0.92664695 776 andrew gelman stats-2011-06-22-Deviance, DIC, AIC, cross-validation, etc
8 0.92600977 1975 andrew gelman stats-2013-08-09-Understanding predictive information criteria for Bayesian models
9 0.92531896 2311 andrew gelman stats-2014-04-29-Bayesian Uncertainty Quantification for Differential Equations!
10 0.92472053 447 andrew gelman stats-2010-12-03-Reinventing the wheel, only more so.
11 0.92405546 454 andrew gelman stats-2010-12-07-Diabetes stops at the state line?
12 0.92313659 2204 andrew gelman stats-2014-02-09-Keli Liu and Xiao-Li Meng on Simpson’s paradox
13 0.92270684 1019 andrew gelman stats-2011-11-19-Validation of Software for Bayesian Models Using Posterior Quantiles
16 0.92070383 1980 andrew gelman stats-2013-08-13-Test scores and grades predict job performance (but maybe not at Google)
17 0.92047369 1370 andrew gelman stats-2012-06-07-Duncan Watts and the Titanic
18 0.92032617 1669 andrew gelman stats-2013-01-12-The power of the puzzlegraph
19 0.91813052 1983 andrew gelman stats-2013-08-15-More on AIC, WAIC, etc
20 0.91731012 1278 andrew gelman stats-2012-04-23-“Any old map will do” meets “God is in every leaf of every tree”