andrew_gelman_stats andrew_gelman_stats-2012 andrew_gelman_stats-2012-1560 knowledge-graph by maker-knowledge-mining
Source: html
Introduction: David Hogg pointed me to this post by Larry Wasserman: 1. The Horwitz-Thompson estimator satisfies the following condition: for every , where — the parameter space — is the set of all functions . (There are practical improvements to the Horwitz-Thompson estimator that we discussed in our earlier posts but we won’t revisit those here.) 2. A Bayes estimator requires a prior for . In general, if is not a function of then (1) will not hold. . . . 3. If you let be a function if , (1) still, in general, does not hold. 4. If you make a function if in just the right way, then (1) will hold. . . . There is nothing wrong with doing this, but in our opinion this is not in the spirit of Bayesian inference. . . . 7. This example is only meant to show that Bayesian estimators do not necessarily have good frequentist properties. This should not be surprising. There is no reason why we should in general expect a Bayesian method to have a frequentist property
sentIndex sentText sentNum sentScore
1 (There are practical improvements to the Horwitz-Thompson estimator that we discussed in our earlier posts but we won’t revisit those here. [sent-3, score-0.233]
2 This example is only meant to show that Bayesian estimators do not necessarily have good frequentist properties. [sent-22, score-0.26]
3 There is no reason why we should in general expect a Bayesian method to have a frequentist property like (1). [sent-24, score-0.404]
4 One day, professor X showed LW an example where maximum likelihood does not do well. [sent-27, score-0.674]
5 LW’s response was to shrug his shoulders and say: “that’s interesting. [sent-28, score-0.318]
6 He felt that by showing one example where maximum likelihood fails, he had discredited maximum likelihood. [sent-31, score-0.916]
7 We use maximum likelihood when it works well and we don’t use maximum likelihood when it doesn’t work well. [sent-33, score-1.202]
8 When Bayesians see the Robins-Ritov example (or other similar examples) why don’t they just shrug their shoulders and say: “that’s interesting. [sent-34, score-0.447]
9 But some feel that if Bayes fails in one example then their whole world comes crashing down. [sent-37, score-0.892]
10 Like all statistical procedures, sometimes it works well and sometimes it doesn’t. [sent-41, score-0.296]
11 I agree with Larry that not every method works on every problem. [sent-42, score-0.392]
12 I’ve heard that biostatisticians and software companies also use Bayesian methods, maximum likelihoods, chi-squared tests, etc. [sent-45, score-0.461]
13 Larry writes, “But some feel that if Bayes fails in one example then their whole world comes crashing down. [sent-50, score-0.892]
14 ” I would rephrase this to say: “Some feel that if Bayes fails in one example then it would be good to understand what aspects of the model are causing problems. [sent-52, score-0.516]
15 And for some problems, classical statisticians give up (for example, giving a conf interval but not a point estimate in the y/n case). [sent-59, score-0.286]
16 Some problems are essentially ill-posed (for example, the problem of estimating a ratio whose denominator could be either positive or negative , sometimes called the Fieller-Creasey problem). [sent-60, score-0.365]
17 We’ll give up on theoretical principles (such as Larry’s (1) above) but we don’t like to give up on getting inferences for any quantity of interest. [sent-62, score-0.325]
18 In contrast, non-Bayesians often feel strongly about principles such as unbiasedness and confidence coverage but are willing to give up on producing an estimate if a parameter is nonidentified. [sent-63, score-0.528]
19 So, I think Larry has identified a real difference in attitudes, but I don’t agree with his characterization that Bayesians think “their whole world comes crashing down. [sent-64, score-0.575]
20 ” We’re just more bothered by not being able to come up with an estimate of a probability, rather than not being able to satisfy a minimax property. [sent-65, score-0.232]
wordName wordTfidf (topN-words)
[('maximum', 0.308), ('larry', 0.306), ('bayesians', 0.263), ('fails', 0.225), ('crashing', 0.21), ('likelihood', 0.171), ('lw', 0.171), ('shoulders', 0.171), ('estimator', 0.165), ('shrug', 0.147), ('bayes', 0.137), ('minimax', 0.135), ('frequentist', 0.131), ('example', 0.129), ('give', 0.123), ('bayesian', 0.121), ('property', 0.108), ('function', 0.102), ('sometimes', 0.101), ('estimate', 0.097), ('feel', 0.096), ('works', 0.094), ('methods', 0.09), ('whole', 0.085), ('method', 0.085), ('won', 0.085), ('proportion', 0.083), ('general', 0.08), ('principles', 0.079), ('biostatisticians', 0.078), ('comes', 0.077), ('use', 0.075), ('mystique', 0.073), ('every', 0.072), ('estimating', 0.071), ('world', 0.07), ('thompson', 0.07), ('agree', 0.069), ('problem', 0.068), ('revisit', 0.068), ('parameter', 0.067), ('professor', 0.066), ('rephrase', 0.066), ('satisfies', 0.066), ('conf', 0.066), ('unbiasedness', 0.066), ('african', 0.064), ('characterization', 0.064), ('problems', 0.064), ('positive', 0.061)]
simIndex simValue blogId blogTitle
same-blog 1 1.0 1560 andrew gelman stats-2012-11-03-Statistical methods that work in some settings but not others
Introduction: David Hogg pointed me to this post by Larry Wasserman: 1. The Horwitz-Thompson estimator satisfies the following condition: for every , where — the parameter space — is the set of all functions . (There are practical improvements to the Horwitz-Thompson estimator that we discussed in our earlier posts but we won’t revisit those here.) 2. A Bayes estimator requires a prior for . In general, if is not a function of then (1) will not hold. . . . 3. If you let be a function if , (1) still, in general, does not hold. 4. If you make a function if in just the right way, then (1) will hold. . . . There is nothing wrong with doing this, but in our opinion this is not in the spirit of Bayesian inference. . . . 7. This example is only meant to show that Bayesian estimators do not necessarily have good frequentist properties. This should not be surprising. There is no reason why we should in general expect a Bayesian method to have a frequentist property
2 0.23954025 247 andrew gelman stats-2010-09-01-How does Bayes do it?
Introduction: I received the following message from a statistician working in industry: I am studying your paper, A Weakly Informative Default Prior Distribution for Logistic and Other Regression Models . I am not clear why the Bayesian approaches with some priors can usually handle the issue of nonidentifiability or can get stable estimates of parameters in model fit, while the frequentist approaches cannot. My reply: 1. The term “frequentist approach” is pretty general. “Frequentist” refers to an approach for evaluating inferences, not a method for creating estimates. In particular, any Bayes estimate can be viewed as a frequentist inference if you feel like evaluating its frequency properties. In logistic regression, maximum likelihood has some big problems that are solved with penalized likelihood–equivalently, Bayesian inference. A frequentist can feel free to consider the prior as a penalty function rather than a probability distribution of parameters. 2. The reason our approa
3 0.23093338 1389 andrew gelman stats-2012-06-23-Larry Wasserman’s statistics blog
Introduction: Larry Wasserman, a leading theoretical statistician and generally thoughtful guy, has started a blog on . . . theoretical statistics! Good stuff, and readers of this blog should enjoy the different perspective that Larry offers. Here are some earlier references to Larry on this blog, and here’s a discussion that gives a sense of our different (but not extremely different) attitudes about statistical methods.
4 0.22732671 291 andrew gelman stats-2010-09-22-Philosophy of Bayes and non-Bayes: A dialogue with Deborah Mayo
Introduction: I sent Deborah Mayo a link to my paper with Cosma Shalizi on the philosophy of statistics, and she sent me the link to this conference which unfortunately already occurred. (It’s too bad, because I’d have liked to have been there.) I summarized my philosophy as follows: I am highly sympathetic to the approach of Lakatos (or of Popper, if you consider Lakatos’s “Popper_2″ to be a reasonable simulation of the true Popperism), in that (a) I view statistical models as being built within theoretical structures, and (b) I see the checking and refutation of models to be a key part of scientific progress. A big problem I have with mainstream Bayesianism is its “inductivist” view that science can operate completely smoothly with posterior updates: the idea that new data causes us to increase the posterior probability of good models and decrease the posterior probability of bad models. I don’t buy that: I see models as ever-changing entities that are flexible and can be patched and ex
5 0.21691212 1610 andrew gelman stats-2012-12-06-Yes, checking calibration of probability forecasts is part of Bayesian statistics
Introduction: Yes, checking calibration of probability forecasts is part of Bayesian statistics. At the end of this post are three figures from Chapter 1 of Bayesian Data Analysis illustrating empirical evaluation of forecasts. But first the background. Why am I bringing this up now? It’s because of something Larry Wasserman wrote the other day : One of the striking facts about [baseball/political forecaster Nate Silver's recent] book is the emphasis the Silver places on frequency calibration. . . . Have no doubt about it: Nate Silver is a frequentist. For example, he says: One of the most important tests of a forecast — I would argue that it is the single most important one — is called calibration. Out of all the times you said there was a 40 percent chance of rain, how often did rain actually occur? If over the long run, it really did rain about 40 percent of the time, that means your forecasts were well calibrated. I had some discussion with Larry in the comments section of h
7 0.20448323 788 andrew gelman stats-2011-07-06-Early stopping and penalized likelihood
8 0.19890803 1469 andrew gelman stats-2012-08-25-Ways of knowing
9 0.19122118 1205 andrew gelman stats-2012-03-09-Coming to agreement on philosophy of statistics
10 0.17304921 1695 andrew gelman stats-2013-01-28-Economists argue about Bayes
11 0.16836092 244 andrew gelman stats-2010-08-30-Useful models, model checking, and external validation: a mini-discussion
12 0.16817874 1165 andrew gelman stats-2012-02-13-Philosophy of Bayesian statistics: my reactions to Wasserman
13 0.16383128 1779 andrew gelman stats-2013-03-27-“Two Dogmas of Strong Objective Bayesianism”
14 0.16274467 2254 andrew gelman stats-2014-03-18-Those wacky anti-Bayesians used to be intimidating, but now they’re just pathetic
15 0.15745482 586 andrew gelman stats-2011-02-23-A statistical version of Arrow’s paradox
16 0.15065785 2208 andrew gelman stats-2014-02-12-How to think about “identifiability” in Bayesian inference?
17 0.1487928 779 andrew gelman stats-2011-06-25-Avoiding boundary estimates using a prior distribution as regularization
19 0.14320073 1438 andrew gelman stats-2012-07-31-What is a Bayesian?
20 0.14091213 117 andrew gelman stats-2010-06-29-Ya don’t know Bayes, Jack
topicId topicWeight
[(0, 0.235), (1, 0.141), (2, -0.021), (3, 0.048), (4, -0.105), (5, -0.014), (6, 0.011), (7, 0.069), (8, 0.023), (9, -0.151), (10, -0.035), (11, -0.067), (12, 0.003), (13, 0.023), (14, -0.01), (15, -0.007), (16, -0.003), (17, 0.003), (18, -0.026), (19, -0.004), (20, 0.032), (21, -0.007), (22, 0.04), (23, 0.058), (24, 0.073), (25, -0.022), (26, -0.017), (27, 0.024), (28, 0.012), (29, 0.054), (30, 0.011), (31, 0.041), (32, 0.043), (33, -0.015), (34, 0.05), (35, -0.049), (36, -0.047), (37, 0.006), (38, -0.013), (39, -0.009), (40, 0.039), (41, 0.037), (42, -0.045), (43, 0.023), (44, 0.018), (45, 0.025), (46, 0.053), (47, 0.048), (48, -0.066), (49, 0.048)]
simIndex simValue blogId blogTitle
same-blog 1 0.97428393 1560 andrew gelman stats-2012-11-03-Statistical methods that work in some settings but not others
Introduction: David Hogg pointed me to this post by Larry Wasserman: 1. The Horwitz-Thompson estimator satisfies the following condition: for every , where — the parameter space — is the set of all functions . (There are practical improvements to the Horwitz-Thompson estimator that we discussed in our earlier posts but we won’t revisit those here.) 2. A Bayes estimator requires a prior for . In general, if is not a function of then (1) will not hold. . . . 3. If you let be a function if , (1) still, in general, does not hold. 4. If you make a function if in just the right way, then (1) will hold. . . . There is nothing wrong with doing this, but in our opinion this is not in the spirit of Bayesian inference. . . . 7. This example is only meant to show that Bayesian estimators do not necessarily have good frequentist properties. This should not be surprising. There is no reason why we should in general expect a Bayesian method to have a frequentist property
2 0.81515861 1572 andrew gelman stats-2012-11-10-I don’t like this cartoon
Introduction: Some people pointed me to this : I am happy to see statistical theory and methods be a topic in popular culture, and of course I’m glad that, contra Feller , the Bayesian is presented as the hero this time, but . . . . I think the lower-left panel of the cartoon unfairly misrepresents frequentist statisticians. Frequentist statisticians recognize many statistical goals. Point estimates trade off bias and variance. Interval estimates have the goal of achieving nominal coverage and the goal of being informative. Tests have the goals of calibration and power. Frequentists know that no single principle applies in all settings, and this is a setting where this particular method is clearly inappropriate. All statisticians use prior information in their statistical analysis. Non-Bayesians express their prior information not through a probability distribution on parameters but rather through their choice of methods. I think this non-Bayesian attitude is too restrictive, but in
3 0.80544025 586 andrew gelman stats-2011-02-23-A statistical version of Arrow’s paradox
Introduction: Unfortunately, when we deal with scientists, statisticians are often put in a setting reminiscent of Arrow’s paradox, where we are asked to provide estimates that are informative and unbiased and confidence statements that are correct conditional on the data and also on the underlying true parameter. [It's not generally possible for an estimate to do all these things at the same time -- ed.] Larry Wasserman feels that scientists are truly frequentist, and Don Rubin has told me how he feels that scientists interpret all statistical estimates Bayesianly. I have no doubt that both Larry and Don are correct. Voters want lower taxes and more services, and scientists want both Bayesian and frequency coverage; as the saying goes, everybody wants to go to heaven but nobody wants to die.
4 0.78681922 1610 andrew gelman stats-2012-12-06-Yes, checking calibration of probability forecasts is part of Bayesian statistics
Introduction: Yes, checking calibration of probability forecasts is part of Bayesian statistics. At the end of this post are three figures from Chapter 1 of Bayesian Data Analysis illustrating empirical evaluation of forecasts. But first the background. Why am I bringing this up now? It’s because of something Larry Wasserman wrote the other day : One of the striking facts about [baseball/political forecaster Nate Silver's recent] book is the emphasis the Silver places on frequency calibration. . . . Have no doubt about it: Nate Silver is a frequentist. For example, he says: One of the most important tests of a forecast — I would argue that it is the single most important one — is called calibration. Out of all the times you said there was a 40 percent chance of rain, how often did rain actually occur? If over the long run, it really did rain about 40 percent of the time, that means your forecasts were well calibrated. I had some discussion with Larry in the comments section of h
5 0.7853272 291 andrew gelman stats-2010-09-22-Philosophy of Bayes and non-Bayes: A dialogue with Deborah Mayo
Introduction: I sent Deborah Mayo a link to my paper with Cosma Shalizi on the philosophy of statistics, and she sent me the link to this conference which unfortunately already occurred. (It’s too bad, because I’d have liked to have been there.) I summarized my philosophy as follows: I am highly sympathetic to the approach of Lakatos (or of Popper, if you consider Lakatos’s “Popper_2″ to be a reasonable simulation of the true Popperism), in that (a) I view statistical models as being built within theoretical structures, and (b) I see the checking and refutation of models to be a key part of scientific progress. A big problem I have with mainstream Bayesianism is its “inductivist” view that science can operate completely smoothly with posterior updates: the idea that new data causes us to increase the posterior probability of good models and decrease the posterior probability of bad models. I don’t buy that: I see models as ever-changing entities that are flexible and can be patched and ex
7 0.75248748 2254 andrew gelman stats-2014-03-18-Those wacky anti-Bayesians used to be intimidating, but now they’re just pathetic
8 0.75109881 449 andrew gelman stats-2010-12-04-Generalized Method of Moments, whatever that is
9 0.74818516 1469 andrew gelman stats-2012-08-25-Ways of knowing
10 0.7469511 1438 andrew gelman stats-2012-07-31-What is a Bayesian?
11 0.73154193 1182 andrew gelman stats-2012-02-24-Untangling the Jeffreys-Lindley paradox
12 0.72627163 247 andrew gelman stats-2010-09-01-How does Bayes do it?
13 0.72496766 1781 andrew gelman stats-2013-03-29-Another Feller theory
14 0.7198444 1205 andrew gelman stats-2012-03-09-Coming to agreement on philosophy of statistics
15 0.7190215 1779 andrew gelman stats-2013-03-27-“Two Dogmas of Strong Objective Bayesianism”
16 0.71243531 1157 andrew gelman stats-2012-02-07-Philosophy of Bayesian statistics: my reactions to Hendry
18 0.70462012 117 andrew gelman stats-2010-06-29-Ya don’t know Bayes, Jack
19 0.70067227 2180 andrew gelman stats-2014-01-21-Everything I need to know about Bayesian statistics, I learned in eight schools.
20 0.70034361 1975 andrew gelman stats-2013-08-09-Understanding predictive information criteria for Bayesian models
topicId topicWeight
[(15, 0.077), (16, 0.085), (24, 0.202), (49, 0.077), (55, 0.018), (84, 0.024), (86, 0.046), (89, 0.011), (99, 0.333)]
simIndex simValue blogId blogTitle
same-blog 1 0.97775185 1560 andrew gelman stats-2012-11-03-Statistical methods that work in some settings but not others
Introduction: David Hogg pointed me to this post by Larry Wasserman: 1. The Horwitz-Thompson estimator satisfies the following condition: for every , where — the parameter space — is the set of all functions . (There are practical improvements to the Horwitz-Thompson estimator that we discussed in our earlier posts but we won’t revisit those here.) 2. A Bayes estimator requires a prior for . In general, if is not a function of then (1) will not hold. . . . 3. If you let be a function if , (1) still, in general, does not hold. 4. If you make a function if in just the right way, then (1) will hold. . . . There is nothing wrong with doing this, but in our opinion this is not in the spirit of Bayesian inference. . . . 7. This example is only meant to show that Bayesian estimators do not necessarily have good frequentist properties. This should not be surprising. There is no reason why we should in general expect a Bayesian method to have a frequentist property
2 0.97128141 792 andrew gelman stats-2011-07-08-The virtues of incoherence?
Introduction: Kent Osband writes: I just read your article The holes in my philosophy of Bayesian data analysis . I agree on the importance of what you flagged as “comparable incoherence in all other statistical philosophies”. The problem arises when a string of unexpected observations persuades that one’s original structural hypothesis (which might be viewed as a parameter describing the type of statistical relationship) was false. However, I would phrase this more positively. Your Bayesian prior actually cedes alternative structural hypotheses, albeit with tiny epsilon weights. Otherwise you would never change your mind. However, these epsilons are so difficult to measure, and small differences can have such a significant impact on speed of adjustment (as in the example in Chapter 7 of Pandora’s Risk), that effectively we all look incoherent. This is a prime example of rational turbulence. Rational turbulence can arise even without a structural break. Any time new evidence arrives that
3 0.97021317 2244 andrew gelman stats-2014-03-11-What if I were to stop publishing in journals?
Introduction: In our recent discussion of modes of publication, Joseph Wilson wrote, “The single best reform science can make right now is to decouple publication from career advancement, thereby reducing the number of publications by an order of magnitude and then move to an entirely disjointed, informal, online free-for-all communication system for research results.” My first thought on this was: Sure, yeah, that makes sense. But then I got to thinking: what would it really mean to decouple publication from career advancement? This is too late for me—I’m middle-aged and have no career advancement in my future—but it got me thinking more carefully about the role of publication in the research process, and this seemed worth a blog (the simplest sort of publication available to me). However, somewhere between writing the above paragraphs and writing the blog entry, I forgot exactly what I was going to say! I guess I should’ve just typed it all in then. In the old days I just wouldn’t run this
4 0.96737707 2140 andrew gelman stats-2013-12-19-Revised evidence for statistical standards
Introduction: X and I heard about this much-publicized recent paper by Val Johnson, who suggests changing the default level of statistical significance from z=2 to z=3 (or, as he puts it, going from p=.05 to p=.005 or .001). Val argues that you need to go out to 3 standard errors to get a Bayes factor of 25 or 50 in favor of the alternative hypothesis. I don’t really buy this, first because Val’s model is a weird (to me) mixture of two point masses, which he creates in order to make a minimax argument, and second because I don’t see why you need a Bayes factor of 25 to 50 in order to make a claim. I’d think that a factor of 5:1, say, provides strong information already—if you really believe those odds. The real issue, as I see it, is that we’re getting Bayes factors and posterior probabilities we don’t believe, because we’re assuming flat priors that don’t really make sense. This is a topic that’s come up over and over in recent months on this blog, for example in this discussion of why I d
5 0.9671396 2299 andrew gelman stats-2014-04-21-Stan Model of the Week: Hierarchical Modeling of Supernovas
Introduction: The Stan Model of the Week showcases research using Stan to push the limits of applied statistics. If you have a model that you would like to submit for a future post then send us an email . Our inaugural post comes from Nathan Sanders, a graduate student finishing up his thesis on astrophysics at Harvard. Nathan writes, “Core-collapse supernovae, the luminous explosions of massive stars, exhibit an expansive and meaningful diversity of behavior in their brightness evolution over time (their “light curves”). Our group discovers and monitors these events using the Pan-STARRS1 telescope in Hawaii, and we’ve collected a dataset of about 20,000 individual photometric observations of about 80 Type IIP supernovae, the class my work has focused on. While this dataset provides one of the best available tools to infer the explosion properties of these supernovae, due to the nature of extragalactic astronomy (observing from distances 1 billion light years), these light curves typicall
6 0.9669714 2161 andrew gelman stats-2014-01-07-My recent debugging experience
7 0.96671599 1247 andrew gelman stats-2012-04-05-More philosophy of Bayes
8 0.96598375 899 andrew gelman stats-2011-09-10-The statistical significance filter
9 0.96594781 902 andrew gelman stats-2011-09-12-The importance of style in academic writing
10 0.96588111 1162 andrew gelman stats-2012-02-11-Adding an error model to a deterministic model
11 0.96563041 1435 andrew gelman stats-2012-07-30-Retracted articles and unethical behavior in economics journals?
12 0.96559381 2055 andrew gelman stats-2013-10-08-A Bayesian approach for peer-review panels? and a speculation about Bruno Frey
13 0.96539831 511 andrew gelman stats-2011-01-11-One more time on that ESP study: The problem of overestimates and the shrinkage solution
14 0.96522784 1007 andrew gelman stats-2011-11-13-At last, treated with the disrespect that I deserve
15 0.96514231 1848 andrew gelman stats-2013-05-09-A tale of two discussion papers
16 0.96482956 1713 andrew gelman stats-2013-02-08-P-values and statistical practice
18 0.96475589 2201 andrew gelman stats-2014-02-06-Bootstrap averaging: Examples where it works and where it doesn’t work
19 0.9647249 2353 andrew gelman stats-2014-05-30-I posted this as a comment on a sociology blog