andrew_gelman_stats andrew_gelman_stats-2013 andrew_gelman_stats-2013-1868 knowledge-graph by maker-knowledge-mining
Source: html
Introduction: Every once in awhile I get a question that I can directly answer from my published research. When that happens it makes me so happy. Here’s an example. Patrick Lam wrote, Suppose one develops a Bayesian model to estimate a parameter theta. Now suppose one wants to evaluate the model via simulation by generating fake data where you know the value of theta and see how well you recover theta with your model, assuming that you use the posterior mean as the estimate. The traditional frequentist way of evaluating it might be to generate many datasets and see how well your estimator performs each time in terms of unbiasedness or mean squared error or something. But given that unbiasedness means nothing to a Bayesian and there is no repeated sampling interpretation in a Bayesian model, how would you suggest one would evaluate a Bayesian model? My reply: I actually have a paper on this ! It is by Cook, Gelman, and Rubin. The idea is to draw theta from the prior distribution.
sentIndex sentText sentNum sentScore
1 Every once in awhile I get a question that I can directly answer from my published research. [sent-1, score-0.266]
2 Patrick Lam wrote, Suppose one develops a Bayesian model to estimate a parameter theta. [sent-4, score-0.414]
3 Now suppose one wants to evaluate the model via simulation by generating fake data where you know the value of theta and see how well you recover theta with your model, assuming that you use the posterior mean as the estimate. [sent-5, score-2.334]
4 The traditional frequentist way of evaluating it might be to generate many datasets and see how well your estimator performs each time in terms of unbiasedness or mean squared error or something. [sent-6, score-1.761]
5 But given that unbiasedness means nothing to a Bayesian and there is no repeated sampling interpretation in a Bayesian model, how would you suggest one would evaluate a Bayesian model? [sent-7, score-1.099]
6 My reply: I actually have a paper on this ! [sent-8, score-0.077]
7 The idea is to draw theta from the prior distribution. [sent-10, score-0.431]
8 You can find the paper in the published papers section on my website. [sent-11, score-0.328]
9 Although unbiasedness doesn’t mean much to a Bayesian, calibration does. [sent-14, score-0.758]
10 We’re planning on implementing this in Stan at some point. [sent-15, score-0.233]
wordName wordTfidf (topN-words)
[('unbiasedness', 0.469), ('theta', 0.337), ('bayesian', 0.24), ('evaluate', 0.199), ('model', 0.182), ('mean', 0.158), ('develops', 0.152), ('generating', 0.143), ('patrick', 0.14), ('recover', 0.138), ('suppose', 0.137), ('squared', 0.136), ('estimator', 0.131), ('calibration', 0.131), ('implementing', 0.129), ('performs', 0.127), ('repeated', 0.119), ('cook', 0.118), ('fake', 0.116), ('evaluating', 0.113), ('datasets', 0.109), ('published', 0.107), ('simulation', 0.105), ('generate', 0.105), ('frequentist', 0.104), ('planning', 0.104), ('traditional', 0.094), ('draw', 0.094), ('wants', 0.091), ('awhile', 0.089), ('assuming', 0.089), ('interpretation', 0.088), ('gelman', 0.082), ('via', 0.081), ('section', 0.081), ('stan', 0.08), ('sampling', 0.08), ('parameter', 0.08), ('happens', 0.079), ('suggest', 0.078), ('well', 0.077), ('paper', 0.077), ('posterior', 0.076), ('terms', 0.073), ('directly', 0.07), ('value', 0.068), ('means', 0.066), ('although', 0.066), ('error', 0.065), ('papers', 0.063)]
simIndex simValue blogId blogTitle
same-blog 1 1.0000001 1868 andrew gelman stats-2013-05-23-Validation of Software for Bayesian Models Using Posterior Quantiles
Introduction: Every once in awhile I get a question that I can directly answer from my published research. When that happens it makes me so happy. Here’s an example. Patrick Lam wrote, Suppose one develops a Bayesian model to estimate a parameter theta. Now suppose one wants to evaluate the model via simulation by generating fake data where you know the value of theta and see how well you recover theta with your model, assuming that you use the posterior mean as the estimate. The traditional frequentist way of evaluating it might be to generate many datasets and see how well your estimator performs each time in terms of unbiasedness or mean squared error or something. But given that unbiasedness means nothing to a Bayesian and there is no repeated sampling interpretation in a Bayesian model, how would you suggest one would evaluate a Bayesian model? My reply: I actually have a paper on this ! It is by Cook, Gelman, and Rubin. The idea is to draw theta from the prior distribution.
2 0.23708378 1941 andrew gelman stats-2013-07-16-Priors
Introduction: Nick Firoozye writes: While I am absolutely sympathetic to the Bayesian agenda I am often troubled by the requirement of having priors. We must have priors on the parameter of an infinite number of model we have never seen before and I find this troubling. There is a similarly troubling problem in economics of utility theory. Utility is on consumables. To be complete a consumer must assign utility to all sorts of things they never would have encountered. More recent versions of utility theory instead make consumption goods a portfolio of attributes. Cadillacs are x many units of luxury y of transport etc etc. And we can automatically have personal utilities to all these attributes. I don’t ever see parameters. Some model have few and some have hundreds. Instead, I see data. So I don’t know how to have an opinion on parameters themselves. Rather I think it far more natural to have opinions on the behavior of models. The prior predictive density is a good and sensible notion. Also
3 0.23525733 1610 andrew gelman stats-2012-12-06-Yes, checking calibration of probability forecasts is part of Bayesian statistics
Introduction: Yes, checking calibration of probability forecasts is part of Bayesian statistics. At the end of this post are three figures from Chapter 1 of Bayesian Data Analysis illustrating empirical evaluation of forecasts. But first the background. Why am I bringing this up now? It’s because of something Larry Wasserman wrote the other day : One of the striking facts about [baseball/political forecaster Nate Silver's recent] book is the emphasis the Silver places on frequency calibration. . . . Have no doubt about it: Nate Silver is a frequentist. For example, he says: One of the most important tests of a forecast — I would argue that it is the single most important one — is called calibration. Out of all the times you said there was a 40 percent chance of rain, how often did rain actually occur? If over the long run, it really did rain about 40 percent of the time, that means your forecasts were well calibrated. I had some discussion with Larry in the comments section of h
4 0.23501486 1089 andrew gelman stats-2011-12-28-Path sampling for models of varying dimension
Introduction: Somebody asks: I’m reading your paper on path sampling. It essentially solves the problem of computing the ratio \int q0(omega)d omega/\int q1(omega) d omega. I.e the arguments in q0() and q1() are the same. But this assumption is not always true in Bayesian model selection using Bayes factor. In general (for BF), we have this problem, t1 and t2 may have no relation at all. \int f1(y|t1)p1(t1) d t1 / \int f2(y|t2)p2(t2) d t2 As an example, suppose that we want to compare two sets of normally distributed data with known variance whether they have the same mean (H0) or they are not necessarily have the same mean (H1). Then the dummy variable should be mu in H0 (which is the common mean of both set of samples), and should be (mu1, mu2) (which are the means for each set of samples). One straight method to address my problem is to preform path integration for the numerate and the denominator, as both the numerate and the denominator are integrals. Each integral can be rewrit
Introduction: Some things I respect When it comes to meta-models of statistics, here are two philosophies that I respect: 1. (My) Bayesian approach, which I associate with E. T. Jaynes, in which you construct models with strong assumptions, ride your models hard, check their fit to data, and then scrap them and improve them as necessary. 2. At the other extreme, model-free statistical procedures that are designed to work well under very weak assumptions—for example, instead of assuming a distribution is Gaussian, you would just want the procedure to work well under some conditions on the smoothness of the second derivative of the log density function. Both the above philosophies recognize that (almost) all important assumptions will be wrong, and they resolve this concern via aggressive model checking or via robustness. And of course there are intermediate positions, such as working with Bayesian models that have been shown to be robust, and then still checking them. Or, to flip it arou
6 0.18136197 899 andrew gelman stats-2011-09-10-The statistical significance filter
7 0.17923489 547 andrew gelman stats-2011-01-31-Using sample size in the prior distribution
8 0.17699793 1695 andrew gelman stats-2013-01-28-Economists argue about Bayes
9 0.17494076 1999 andrew gelman stats-2013-08-27-Bayesian model averaging or fitting a larger model
10 0.17441663 291 andrew gelman stats-2010-09-22-Philosophy of Bayes and non-Bayes: A dialogue with Deborah Mayo
11 0.17393009 1476 andrew gelman stats-2012-08-30-Stan is fast
12 0.16150381 961 andrew gelman stats-2011-10-16-The “Washington read” and the algebra of conditional distributions
13 0.15843336 1554 andrew gelman stats-2012-10-31-It not necessary that Bayesian methods conform to the likelihood principle
14 0.15175928 2351 andrew gelman stats-2014-05-28-Bayesian nonparametric weighted sampling inference
15 0.15161353 534 andrew gelman stats-2011-01-24-Bayes at the end
16 0.14956036 1205 andrew gelman stats-2012-03-09-Coming to agreement on philosophy of statistics
17 0.14749728 1438 andrew gelman stats-2012-07-31-What is a Bayesian?
18 0.14574136 1779 andrew gelman stats-2013-03-27-“Two Dogmas of Strong Objective Bayesianism”
20 0.14466396 244 andrew gelman stats-2010-08-30-Useful models, model checking, and external validation: a mini-discussion
topicId topicWeight
[(0, 0.204), (1, 0.229), (2, -0.03), (3, 0.042), (4, -0.078), (5, -0.006), (6, 0.039), (7, -0.016), (8, 0.019), (9, -0.111), (10, 0.017), (11, -0.02), (12, -0.038), (13, 0.022), (14, -0.007), (15, -0.014), (16, 0.018), (17, 0.027), (18, -0.009), (19, 0.015), (20, 0.022), (21, 0.058), (22, 0.015), (23, -0.076), (24, 0.03), (25, -0.017), (26, -0.053), (27, -0.025), (28, 0.034), (29, 0.033), (30, -0.01), (31, 0.014), (32, -0.045), (33, 0.005), (34, 0.005), (35, 0.038), (36, -0.014), (37, -0.005), (38, -0.038), (39, -0.029), (40, 0.058), (41, 0.071), (42, -0.096), (43, -0.043), (44, -0.049), (45, -0.014), (46, 0.136), (47, 0.07), (48, -0.027), (49, 0.062)]
simIndex simValue blogId blogTitle
same-blog 1 0.95410198 1868 andrew gelman stats-2013-05-23-Validation of Software for Bayesian Models Using Posterior Quantiles
Introduction: Every once in awhile I get a question that I can directly answer from my published research. When that happens it makes me so happy. Here’s an example. Patrick Lam wrote, Suppose one develops a Bayesian model to estimate a parameter theta. Now suppose one wants to evaluate the model via simulation by generating fake data where you know the value of theta and see how well you recover theta with your model, assuming that you use the posterior mean as the estimate. The traditional frequentist way of evaluating it might be to generate many datasets and see how well your estimator performs each time in terms of unbiasedness or mean squared error or something. But given that unbiasedness means nothing to a Bayesian and there is no repeated sampling interpretation in a Bayesian model, how would you suggest one would evaluate a Bayesian model? My reply: I actually have a paper on this ! It is by Cook, Gelman, and Rubin. The idea is to draw theta from the prior distribution.
2 0.78694904 1089 andrew gelman stats-2011-12-28-Path sampling for models of varying dimension
Introduction: Somebody asks: I’m reading your paper on path sampling. It essentially solves the problem of computing the ratio \int q0(omega)d omega/\int q1(omega) d omega. I.e the arguments in q0() and q1() are the same. But this assumption is not always true in Bayesian model selection using Bayes factor. In general (for BF), we have this problem, t1 and t2 may have no relation at all. \int f1(y|t1)p1(t1) d t1 / \int f2(y|t2)p2(t2) d t2 As an example, suppose that we want to compare two sets of normally distributed data with known variance whether they have the same mean (H0) or they are not necessarily have the same mean (H1). Then the dummy variable should be mu in H0 (which is the common mean of both set of samples), and should be (mu1, mu2) (which are the means for each set of samples). One straight method to address my problem is to preform path integration for the numerate and the denominator, as both the numerate and the denominator are integrals. Each integral can be rewrit
Introduction: Yes, checking calibration of probability forecasts is part of Bayesian statistics. At the end of this post are three figures from Chapter 1 of Bayesian Data Analysis illustrating empirical evaluation of forecasts. But first the background. Why am I bringing this up now? It’s because of something Larry Wasserman wrote the other day : One of the striking facts about [baseball/political forecaster Nate Silver's recent] book is the emphasis the Silver places on frequency calibration. . . . Have no doubt about it: Nate Silver is a frequentist. For example, he says: One of the most important tests of a forecast — I would argue that it is the single most important one — is called calibration. Out of all the times you said there was a 40 percent chance of rain, how often did rain actually occur? If over the long run, it really did rain about 40 percent of the time, that means your forecasts were well calibrated. I had some discussion with Larry in the comments section of h
4 0.75212026 2349 andrew gelman stats-2014-05-26-WAIC and cross-validation in Stan!
Introduction: Aki and I write : The Watanabe-Akaike information criterion (WAIC) and cross-validation are methods for estimating pointwise out-of-sample prediction accuracy from a fitted Bayesian model. WAIC is based on the series expansion of leave-one-out cross-validation (LOO), and asymptotically they are equal. With finite data, WAIC and cross-validation address different predictive questions and thus it is useful to be able to compute both. WAIC and an importance-sampling approximated LOO can be estimated directly using the log-likelihood evaluated at the posterior simulations of the parameter values. We show how to compute WAIC, IS-LOO, K-fold cross-validation, and related diagnostic quantities in the Bayesian inference package Stan as called from R. This is important, I think. One reason the deviance information criterion (DIC) has been so popular is its implementation in Bugs. We think WAIC and cross-validation make more sense than DIC, especially from a Bayesian perspective in whic
Introduction: Leading theoretical statistician Larry Wassserman in 2008 : Some of the greatest contributions of statistics to science involve adding additional randomness and leveraging that randomness. Examples are randomized experiments, permutation tests, cross-validation and data-splitting. These are unabashedly frequentist ideas and, while one can strain to fit them into a Bayesian framework, they don’t really have a place in Bayesian inference. The fact that Bayesian methods do not naturally accommodate such a powerful set of statistical ideas seems like a serious deficiency. To which I responded on the second-to-last paragraph of page 8 here . Larry Wasserman in 2013 : Some people say that there is no role for randomization in Bayesian inference. In other words, the randomization mechanism plays no role in Bayes’ theorem. But this is not really true. Without randomization, we can indeed derive a posterior for theta but it is highly sensitive to the prior. This is just a restat
6 0.7451371 2208 andrew gelman stats-2014-02-12-How to think about “identifiability” in Bayesian inference?
7 0.73569196 1182 andrew gelman stats-2012-02-24-Untangling the Jeffreys-Lindley paradox
9 0.73155546 961 andrew gelman stats-2011-10-16-The “Washington read” and the algebra of conditional distributions
10 0.72986233 1438 andrew gelman stats-2012-07-31-What is a Bayesian?
11 0.72458822 1560 andrew gelman stats-2012-11-03-Statistical methods that work in some settings but not others
12 0.71633601 1091 andrew gelman stats-2011-12-29-Bayes in astronomy
13 0.7152881 1779 andrew gelman stats-2013-03-27-“Two Dogmas of Strong Objective Bayesianism”
14 0.70767778 1221 andrew gelman stats-2012-03-19-Whassup with deviance having a high posterior correlation with a parameter in the model?
15 0.69622451 342 andrew gelman stats-2010-10-14-Trying to be precise about vagueness
16 0.69317174 1829 andrew gelman stats-2013-04-28-Plain old everyday Bayesianism!
17 0.68974453 1554 andrew gelman stats-2012-10-31-It not necessary that Bayesian methods conform to the likelihood principle
18 0.68815583 1156 andrew gelman stats-2012-02-06-Bayesian model-building by pure thought: Some principles and examples
19 0.68704122 291 andrew gelman stats-2010-09-22-Philosophy of Bayes and non-Bayes: A dialogue with Deborah Mayo
20 0.68685144 1228 andrew gelman stats-2012-03-25-Continuous variables in Bayesian networks
topicId topicWeight
[(16, 0.051), (21, 0.014), (24, 0.21), (29, 0.016), (76, 0.015), (86, 0.051), (87, 0.214), (99, 0.317)]
simIndex simValue blogId blogTitle
Introduction: John Kastellec, Jeff Lax, and Justin Phillips write : Do senators respond to the preferences of their states’ median voters or only to the preferences of their co-partisans? We [Kastellec et al.] study responsiveness using roll call votes on ten recent Supreme Court nominations. We develop a method for estimating state-level public opinion broken down by partisanship. We find that senators respond more powerfully to their partisan base when casting such roll call votes. Indeed, when their state median voter and party median voter disagree, senators strongly favor the latter. [emphasis added] This has significant implications for the study of legislative responsiveness, the role of public opinion in shaping the personnel of the nations highest court, and the degree to which we should expect the Supreme Court to be counter-majoritarian. Our method can be applied elsewhere to estimate opinion by state and partisan group, or by many other typologies, so as to study other important qu
2 0.95985907 355 andrew gelman stats-2010-10-20-Andy vs. the Ideal Point Model of Voting
Introduction: Last week, as I walked into Andrew’s office for a meeting, he was formulating some misgivings about applying an ideal-point model to budgetary bills in the U.S. Senate. Andrew didn’t like that the model of a senator’s position was an indifference point rather than at their optimal point, and that the effect of moving away from a position was automatically modeled as increasing in one direction and decreasing in the other. Executive Summary The monotonicity of inverse logit entails that the expected vote for a bill among any fixed collection of senators’ ideal points is monotonically increasing (or decreasing) with the bill’s position, with direction determined by the outcome coding. The Ideal-Point Model The ideal-point model’s easy to write down, but hard to reason about because of all the polarity shifting going on. To recapitulate from Gelman and Hill’s Regression book (p. 317), using the U.S. Senate instead of the Supreme Court, and ignoring the dis
3 0.95245916 918 andrew gelman stats-2011-09-21-Avoiding boundary estimates in linear mixed models
Introduction: Pablo Verde sends in this letter he and Daniel Curcio just published in the Journal of Antimicrobial Chemotherapy. They had published a meta-analysis with a boundary estimate which, he said, gave nonsense results. Here’s Curcio and Verde’s key paragraph: The authors [of the study they are criticizing] performed a test of heterogeneity between studies. Given that the test result was not significant at 5%, they decided to pool all the RRs by using a fixed-effect meta-analysis model. Unfortunately, this is a common practice in meta-analysis, which usually leads to very misleading results. First of all, the pooled RR as well as its standard error are sensitive to 2 the estimation of the between-studies standard deviation (SD). SD is difficult to estimate with a small number of studies. On the other hand, it is very well known that the significant test of hetero- geneity lacks statistical power to detect values of SD greater than zero. In addition, the statistically non-significant re
4 0.94848716 233 andrew gelman stats-2010-08-25-Lauryn Hill update
Introduction: Juli thought this might answer some of my questions . To me, though, it seemed a bit of a softball interview, didn’t really go into the theory that the reason she’s stopped recording is that she didn’t really write most of the material herself.
5 0.94632614 1773 andrew gelman stats-2013-03-21-2.15
Introduction: Jake Hofman writes that he saw my recent newspaper article on running (“How fast do we slow down? . . . For each doubling of distance, the world record time is multiplied by about 2.15. . . . for sprints of 200 meters to 1,000 meters, a doubling of distance corresponds to an increase of a factor of 2.3 in world record running times; for longer distances from 1,000 meters to the marathon, a doubling of distance increases the time by a factor of 2.1. . . . similar patterns for men and women, and for swimming as well as running”) and writes: If you’re ever interested in getting or playing with Olympics data, I [Jake] wrote some code to scrape it all from sportsreference.com this past summer for a blog post . Enjoy!
same-blog 6 0.94490325 1868 andrew gelman stats-2013-05-23-Validation of Software for Bayesian Models Using Posterior Quantiles
7 0.94490314 225 andrew gelman stats-2010-08-23-Getting into hot water over hot graphics
10 0.92257977 783 andrew gelman stats-2011-06-30-Don’t stop being a statistician once the analysis is done
11 0.91942847 183 andrew gelman stats-2010-08-04-Bayesian models for simultaneous equation systems?
12 0.91081274 127 andrew gelman stats-2010-07-04-Inequality and health
13 0.89935946 548 andrew gelman stats-2011-02-01-What goes around . . .
14 0.89885169 1788 andrew gelman stats-2013-04-04-When is there “hidden structure in data” to be discovered?
15 0.89802366 2351 andrew gelman stats-2014-05-28-Bayesian nonparametric weighted sampling inference
17 0.88838172 583 andrew gelman stats-2011-02-21-An interesting assignment for statistical graphics
18 0.87597907 2086 andrew gelman stats-2013-11-03-How best to compare effects measured in two different time periods?
19 0.87173474 342 andrew gelman stats-2010-10-14-Trying to be precise about vagueness
20 0.87166369 1191 andrew gelman stats-2012-03-01-Hoe noem je?