andrew_gelman_stats andrew_gelman_stats-2013 andrew_gelman_stats-2013-2140 knowledge-graph by maker-knowledge-mining
Source: html
Introduction: X and I heard about this much-publicized recent paper by Val Johnson, who suggests changing the default level of statistical significance from z=2 to z=3 (or, as he puts it, going from p=.05 to p=.005 or .001). Val argues that you need to go out to 3 standard errors to get a Bayes factor of 25 or 50 in favor of the alternative hypothesis. I don’t really buy this, first because Val’s model is a weird (to me) mixture of two point masses, which he creates in order to make a minimax argument, and second because I don’t see why you need a Bayes factor of 25 to 50 in order to make a claim. I’d think that a factor of 5:1, say, provides strong information already—if you really believe those odds. The real issue, as I see it, is that we’re getting Bayes factors and posterior probabilities we don’t believe, because we’re assuming flat priors that don’t really make sense. This is a topic that’s come up over and over in recent months on this blog, for example in this discussion of why I d
sentIndex sentText sentNum sentScore
1 X and I heard about this much-publicized recent paper by Val Johnson, who suggests changing the default level of statistical significance from z=2 to z=3 (or, as he puts it, going from p=. [sent-1, score-0.299]
2 Val argues that you need to go out to 3 standard errors to get a Bayes factor of 25 or 50 in favor of the alternative hypothesis. [sent-5, score-0.366]
3 I don’t really buy this, first because Val’s model is a weird (to me) mixture of two point masses, which he creates in order to make a minimax argument, and second because I don’t see why you need a Bayes factor of 25 to 50 in order to make a claim. [sent-6, score-0.428]
4 I’d think that a factor of 5:1, say, provides strong information already—if you really believe those odds. [sent-7, score-0.242]
5 The real issue, as I see it, is that we’re getting Bayes factors and posterior probabilities we don’t believe, because we’re assuming flat priors that don’t really make sense. [sent-8, score-0.44]
6 05 standard for significance with the more stringent p = 0. [sent-14, score-0.316]
7 Ultimately such decisions should depend on costs, benefits, and probabilities of all outcomes. [sent-17, score-0.147]
8 Johnson’s minimax prior is not intended to correspond to any distribution of effect sizes; rather it represents a worst-case scenario under some mathematical assumptions. [sent-18, score-0.565]
9 Johnson’s evidence threshold is chosen relative to a conventional value, namely Jeffreys’ target Bayes factor of 1/25 or 1/50, for which we do not see any particular justification except with reference to the tail-area probability of 0. [sent-20, score-0.588]
10 To understand the difficulty of this approach, consider the hypothetical scenario in which R. [sent-22, score-0.14]
11 In this alternative history, the discrepancy between p-values and Bayes factors remains and Johnson could have written a paper noting that the accepted 0. [sent-27, score-0.324]
12 005 standard fails to correspond to 200-to-1 evidence against the null. [sent-28, score-0.28]
13 Indeed, a 200:1 evidence in a minimax sense gets processed by his fixed-point equation γ = exp[z*sqrt(2 log(γ)) − log(γ)] at the value γ = 0. [sent-29, score-0.408]
14 Moreover, the proposition approximately divides any small initial p-level by a factor of sqrt(−4π log p), roughly equal to 10 for the p’s of interest. [sent-35, score-0.551]
15 005 stems from taking 1/20 as a starting point; p = 0. [sent-37, score-0.069]
16 005 has no justification on its own (any more than does the p = 0. [sent-38, score-0.102]
17 0005 threshold derived from the alternative default standard of 1/200). [sent-39, score-0.403]
18 05 rule that has caused so many problems in later decades? [sent-41, score-0.068]
19 We would argue that the appropriate significance level depends on the scenario, and that what worked well for agricultural experiments in the 1920s might not be so appropriate for many applications in modern biosciences. [sent-42, score-0.343]
20 Thus, Johnson’s recommendation to rethink significance thresholds seems like a good idea that needs to include assessments of actual costs, benefits, and probabilities, rather than being based on an abstract calculation. [sent-43, score-0.284]
wordName wordTfidf (topN-words)
[('johnson', 0.364), ('val', 0.283), ('minimax', 0.255), ('bayes', 0.225), ('factor', 0.173), ('log', 0.172), ('significance', 0.162), ('sqrt', 0.153), ('probabilities', 0.147), ('scenario', 0.14), ('threshold', 0.134), ('flat', 0.133), ('justification', 0.102), ('alternative', 0.101), ('factors', 0.1), ('correspond', 0.095), ('evidence', 0.093), ('standard', 0.092), ('fisher', 0.09), ('chosen', 0.086), ('approximately', 0.08), ('costs', 0.079), ('default', 0.076), ('prior', 0.075), ('valen', 0.073), ('benefits', 0.072), ('stems', 0.069), ('believe', 0.069), ('rule', 0.068), ('proposition', 0.066), ('stimulation', 0.066), ('masses', 0.066), ('jeffreys', 0.066), ('rethink', 0.064), ('discrepancy', 0.062), ('exp', 0.062), ('stringent', 0.062), ('appropriate', 0.061), ('paper', 0.061), ('settle', 0.06), ('processed', 0.06), ('divides', 0.06), ('posterior', 0.06), ('agricultural', 0.059), ('horribly', 0.059), ('earnings', 0.059), ('suffers', 0.059), ('expresses', 0.059), ('usual', 0.059), ('thresholds', 0.058)]
simIndex simValue blogId blogTitle
same-blog 1 0.99999982 2140 andrew gelman stats-2013-12-19-Revised evidence for statistical standards
Introduction: X and I heard about this much-publicized recent paper by Val Johnson, who suggests changing the default level of statistical significance from z=2 to z=3 (or, as he puts it, going from p=.05 to p=.005 or .001). Val argues that you need to go out to 3 standard errors to get a Bayes factor of 25 or 50 in favor of the alternative hypothesis. I don’t really buy this, first because Val’s model is a weird (to me) mixture of two point masses, which he creates in order to make a minimax argument, and second because I don’t see why you need a Bayes factor of 25 to 50 in order to make a claim. I’d think that a factor of 5:1, say, provides strong information already—if you really believe those odds. The real issue, as I see it, is that we’re getting Bayes factors and posterior probabilities we don’t believe, because we’re assuming flat priors that don’t really make sense. This is a topic that’s come up over and over in recent months on this blog, for example in this discussion of why I d
Introduction: As regular readers of this blog are aware, a few months ago Val Johnson published an article, “Revised standards for statistical evidence,” making a Bayesian argument that researchers and journals should use a p=0.005 publication threshold rather than the usual p=0.05. Christian Robert and I were unconvinced by Val’s reasoning and wrote a response , “Revised evidence for statistical standards,” in which we wrote: Johnson’s minimax prior is not intended to correspond to any distribution of effect sizes; rather, it represents a worst case scenario under some mathematical assumptions. Minimax and tradeoffs do well together, and it is hard for us to see how any worst case procedure can supply much guidance on how to balance between two different losses. . . . We would argue that the appropriate significance level depends on the scenario and that what worked well for agricultural experiments in the 1920s might not be so appropriate for many applications in modern biosciences . . .
3 0.33157128 2149 andrew gelman stats-2013-12-26-Statistical evidence for revised standards
Introduction: In response to the discussion of X and me of his recent paper , Val Johnson writes: I would like to thank Andrew for forwarding his comments on uniformly most powerful Bayesian tests (UMPBTs) to me and his invitation to respond to them. I think he (and also Christian Robert) raise a number of interesting points concerning this new class of Bayesian tests, but I think that they may have confounded several issues that might more usefully be examined separately. The first issue involves the choice of the Bayesian evidence threshold, gamma, used in rejecting a null hypothesis in favor of an alternative hypothesis. Andrew objects to the higher values of gamma proposed in my recent PNAS article on grounds that too many important scientific effects would be missed if thresholds of 25-50 were routinely used. These evidence thresholds correspond roughly to p-values of 0.005; Andrew suggests that evidence thresholds around 5 should continue to be used (gamma=5 corresponds approximate
4 0.25233898 1723 andrew gelman stats-2013-02-15-Wacky priors can work well?
Introduction: Dave Judkins writes: I would love to see a blog entry on this article , Bayesian Model Selection in High-Dimensional Settings, by Valen Johnson and David Rossell. The simulation results are very encouraging although the choice of colors for some of the graphics is unfortunate. Unless I am colorblind in some way that I am unaware of, they have two thin charcoal lines that are indistinguishable. When Dave Judkins puts in a request, I’ll respond. Also, I’m always happy to see a new Val Johnson paper. Val and I are contemporaries—he and I got our PhD’s at around the same time, with both of us working on Bayesian image reconstruction, then in the early 1990s Val was part of the legendary group at Duke’s Institute of Statistics and Decision Sciences—a veritable ’27 Yankees featuring Mike West, Merlise Clyde, Michael Lavine, Dave Higdon, Peter Mueller, Val, and a bunch of others. I always thought it was too bad they all had to go their separate ways. Val also wrote two classic p
5 0.23050542 703 andrew gelman stats-2011-05-10-Bringing Causal Models Into the Mainstream
Introduction: John Johnson writes at the Statistics Forum.
6 0.16101253 1941 andrew gelman stats-2013-07-16-Priors
7 0.16067484 2223 andrew gelman stats-2014-02-24-“Edlin’s rule” for routinely scaling down published estimates
8 0.15017581 1817 andrew gelman stats-2013-04-21-More on Bayesian model selection in high-dimensional settings
9 0.1382369 2109 andrew gelman stats-2013-11-21-Hidden dangers of noninformative priors
10 0.13565713 1974 andrew gelman stats-2013-08-08-Statistical significance and the dangerous lure of certainty
11 0.12860514 2312 andrew gelman stats-2014-04-29-Ken Rice presents a unifying approach to statistical inference and hypothesis testing
12 0.1277013 754 andrew gelman stats-2011-06-09-Difficulties with Bayesian model averaging
13 0.12533185 1695 andrew gelman stats-2013-01-28-Economists argue about Bayes
14 0.12452625 1586 andrew gelman stats-2012-11-21-Readings for a two-week segment on Bayesian modeling?
15 0.11202852 511 andrew gelman stats-2011-01-11-One more time on that ESP study: The problem of overestimates and the shrinkage solution
16 0.11193901 291 andrew gelman stats-2010-09-22-Philosophy of Bayes and non-Bayes: A dialogue with Deborah Mayo
17 0.11114446 1560 andrew gelman stats-2012-11-03-Statistical methods that work in some settings but not others
18 0.11113585 3 andrew gelman stats-2010-04-26-Bayes in the news…in a somewhat frustrating way
19 0.11093384 801 andrew gelman stats-2011-07-13-On the half-Cauchy prior for a global scale parameter
20 0.11017384 1247 andrew gelman stats-2012-04-05-More philosophy of Bayes
topicId topicWeight
[(0, 0.236), (1, 0.082), (2, 0.018), (3, -0.055), (4, -0.059), (5, -0.067), (6, 0.067), (7, 0.048), (8, -0.063), (9, -0.063), (10, -0.037), (11, 0.005), (12, 0.026), (13, -0.013), (14, 0.011), (15, -0.026), (16, -0.03), (17, 0.039), (18, -0.0), (19, -0.012), (20, 0.021), (21, 0.015), (22, -0.041), (23, 0.012), (24, -0.014), (25, 0.004), (26, 0.037), (27, 0.018), (28, 0.009), (29, -0.025), (30, -0.027), (31, 0.035), (32, 0.024), (33, 0.014), (34, -0.028), (35, -0.047), (36, 0.023), (37, -0.015), (38, 0.029), (39, 0.01), (40, -0.098), (41, -0.062), (42, -0.01), (43, 0.08), (44, 0.035), (45, -0.006), (46, -0.049), (47, -0.043), (48, 0.002), (49, 0.016)]
simIndex simValue blogId blogTitle
same-blog 1 0.94627988 2140 andrew gelman stats-2013-12-19-Revised evidence for statistical standards
Introduction: X and I heard about this much-publicized recent paper by Val Johnson, who suggests changing the default level of statistical significance from z=2 to z=3 (or, as he puts it, going from p=.05 to p=.005 or .001). Val argues that you need to go out to 3 standard errors to get a Bayes factor of 25 or 50 in favor of the alternative hypothesis. I don’t really buy this, first because Val’s model is a weird (to me) mixture of two point masses, which he creates in order to make a minimax argument, and second because I don’t see why you need a Bayes factor of 25 to 50 in order to make a claim. I’d think that a factor of 5:1, say, provides strong information already—if you really believe those odds. The real issue, as I see it, is that we’re getting Bayes factors and posterior probabilities we don’t believe, because we’re assuming flat priors that don’t really make sense. This is a topic that’s come up over and over in recent months on this blog, for example in this discussion of why I d
Introduction: As regular readers of this blog are aware, a few months ago Val Johnson published an article, “Revised standards for statistical evidence,” making a Bayesian argument that researchers and journals should use a p=0.005 publication threshold rather than the usual p=0.05. Christian Robert and I were unconvinced by Val’s reasoning and wrote a response , “Revised evidence for statistical standards,” in which we wrote: Johnson’s minimax prior is not intended to correspond to any distribution of effect sizes; rather, it represents a worst case scenario under some mathematical assumptions. Minimax and tradeoffs do well together, and it is hard for us to see how any worst case procedure can supply much guidance on how to balance between two different losses. . . . We would argue that the appropriate significance level depends on the scenario and that what worked well for agricultural experiments in the 1920s might not be so appropriate for many applications in modern biosciences . . .
3 0.86851066 2149 andrew gelman stats-2013-12-26-Statistical evidence for revised standards
Introduction: In response to the discussion of X and me of his recent paper , Val Johnson writes: I would like to thank Andrew for forwarding his comments on uniformly most powerful Bayesian tests (UMPBTs) to me and his invitation to respond to them. I think he (and also Christian Robert) raise a number of interesting points concerning this new class of Bayesian tests, but I think that they may have confounded several issues that might more usefully be examined separately. The first issue involves the choice of the Bayesian evidence threshold, gamma, used in rejecting a null hypothesis in favor of an alternative hypothesis. Andrew objects to the higher values of gamma proposed in my recent PNAS article on grounds that too many important scientific effects would be missed if thresholds of 25-50 were routinely used. These evidence thresholds correspond roughly to p-values of 0.005; Andrew suggests that evidence thresholds around 5 should continue to be used (gamma=5 corresponds approximate
4 0.77465904 1713 andrew gelman stats-2013-02-08-P-values and statistical practice
Introduction: From my new article in the journal Epidemiology: Sander Greenland and Charles Poole accept that P values are here to stay but recognize that some of their most common interpretations have problems. The casual view of the P value as posterior probability of the truth of the null hypothesis is false and not even close to valid under any reasonable model, yet this misunderstanding persists even in high-stakes settings (as discussed, for example, by Greenland in 2011). The formal view of the P value as a probability conditional on the null is mathematically correct but typically irrelevant to research goals (hence, the popularity of alternative—if wrong—interpretations). A Bayesian interpretation based on a spike-and-slab model makes little sense in applied contexts in epidemiology, political science, and other fields in which true effects are typically nonzero and bounded (thus violating both the “spike” and the “slab” parts of the model). I find Greenland and Poole’s perspective t
5 0.72326201 643 andrew gelman stats-2011-04-02-So-called Bayesian hypothesis testing is just as bad as regular hypothesis testing
Introduction: Steve Ziliak points me to this article by the always-excellent Carl Bialik, slamming hypothesis tests. I only wish Carl had talked with me before so hastily posting, though! I would’ve argued with some of the things in the article. In particular, he writes: Reese and Brad Carlin . . . suggest that Bayesian statistics are a better alternative, because they tackle the probability that the hypothesis is true head-on, and incorporate prior knowledge about the variables involved. Brad Carlin does great work in theory, methods, and applications, and I like the bit about the prior knowledge (although I might prefer the more general phrase “additional information”), but I hate that quote! My quick response is that the hypothesis of zero effect is almost never true! The problem with the significance testing framework–Bayesian or otherwise–is in the obsession with the possibility of an exact zero effect. The real concern is not with zero, it’s with claiming a positive effect whe
6 0.71471864 1355 andrew gelman stats-2012-05-31-Lindley’s paradox
7 0.70591122 1883 andrew gelman stats-2013-06-04-Interrogating p-values
8 0.69840342 2281 andrew gelman stats-2014-04-04-The Notorious N.H.S.T. presents: Mo P-values Mo Problems
9 0.69754088 314 andrew gelman stats-2010-10-03-Disconnect between drug and medical device approval
10 0.69695979 1760 andrew gelman stats-2013-03-12-Misunderstanding the p-value
12 0.68814099 331 andrew gelman stats-2010-10-10-Bayes jumps the shark
13 0.67928624 2223 andrew gelman stats-2014-02-24-“Edlin’s rule” for routinely scaling down published estimates
14 0.67704993 2312 andrew gelman stats-2014-04-29-Ken Rice presents a unifying approach to statistical inference and hypothesis testing
15 0.67350525 506 andrew gelman stats-2011-01-06-That silly ESP paper and some silliness in a rebuttal as well
16 0.66812336 721 andrew gelman stats-2011-05-20-Non-statistical thinking in the US foreign policy establishment
18 0.65487963 1575 andrew gelman stats-2012-11-12-Thinking like a statistician (continuously) rather than like a civilian (discretely)
19 0.65377474 2183 andrew gelman stats-2014-01-23-Discussion on preregistration of research studies
20 0.65356958 1572 andrew gelman stats-2012-11-10-I don’t like this cartoon
topicId topicWeight
[(7, 0.019), (15, 0.032), (16, 0.095), (21, 0.02), (24, 0.183), (34, 0.064), (42, 0.018), (49, 0.024), (53, 0.035), (65, 0.025), (69, 0.013), (84, 0.022), (86, 0.064), (95, 0.013), (99, 0.254)]
simIndex simValue blogId blogTitle
Introduction: As regular readers of this blog are aware, a few months ago Val Johnson published an article, “Revised standards for statistical evidence,” making a Bayesian argument that researchers and journals should use a p=0.005 publication threshold rather than the usual p=0.05. Christian Robert and I were unconvinced by Val’s reasoning and wrote a response , “Revised evidence for statistical standards,” in which we wrote: Johnson’s minimax prior is not intended to correspond to any distribution of effect sizes; rather, it represents a worst case scenario under some mathematical assumptions. Minimax and tradeoffs do well together, and it is hard for us to see how any worst case procedure can supply much guidance on how to balance between two different losses. . . . We would argue that the appropriate significance level depends on the scenario and that what worked well for agricultural experiments in the 1920s might not be so appropriate for many applications in modern biosciences . . .
same-blog 2 0.96966761 2140 andrew gelman stats-2013-12-19-Revised evidence for statistical standards
Introduction: X and I heard about this much-publicized recent paper by Val Johnson, who suggests changing the default level of statistical significance from z=2 to z=3 (or, as he puts it, going from p=.05 to p=.005 or .001). Val argues that you need to go out to 3 standard errors to get a Bayes factor of 25 or 50 in favor of the alternative hypothesis. I don’t really buy this, first because Val’s model is a weird (to me) mixture of two point masses, which he creates in order to make a minimax argument, and second because I don’t see why you need a Bayes factor of 25 to 50 in order to make a claim. I’d think that a factor of 5:1, say, provides strong information already—if you really believe those odds. The real issue, as I see it, is that we’re getting Bayes factors and posterior probabilities we don’t believe, because we’re assuming flat priors that don’t really make sense. This is a topic that’s come up over and over in recent months on this blog, for example in this discussion of why I d
3 0.96336782 994 andrew gelman stats-2011-11-06-Josh Tenenbaum presents . . . a model of folk physics!
Introduction: Josh Tenenbaum describes some new work modeling people’s physical reasoning as probabilistic inferences over intuitive theories of mechanics. A general-purpose capacity for “physical intelligence”—inferring physical properties of objects and predicting future states in complex dynamical scenes—is central to how humans interpret their environment and plan safe and effective actions. The computations and representations underlying physical intelligence remain unclear, however. Cognitive studies have focused on mapping out judgment biases and errors, or on testing simple heuristic models suitable only for highly specific cases; they have not attempted to give general-purpose unifying models. In computer science, artificial intelligence and robotics researchers have long sought to formalize common-sense physical reasoning but without success in approaching human-level competence. Here we show that a wide range of human physical judgments can be explained by positing an “intuitive me
4 0.96324182 135 andrew gelman stats-2010-07-09-Rasmussen sez: “108% of Respondents Say . . .”
Introduction: The recent discussion of pollsters reminded me of a story from a couple years ago that perhaps is still relevant . . . I was looking up the governors’ popularity numbers on the web, and came across this page from Rasmussen Reports which shows Sarah Palin as the 3rd-most-popular governor. But then I looked more carefully. Janet Napolitano of Arizona was viewed as Excellent by 28% of respondents, Good by 27%, Fair by 26%, and Poor by 27%. That adds up to 108%! What’s going on? I’d think they would have a computer program to pipe the survey results directly into the spreadsheet. But I guess not, someone must be typing in these numbers one at a time. Another possibility is that they are altering their numbers by hand, and someone made a mistake with the Napolitano numbers, adding a few percent in one place and forgetting to subtract elsewhere. Or maybe there’s another explanation? P.S. Here are some thoughts from Mark Blumenthal P.P.S. I checked the Rasmussen link toda
Introduction: Makoto Hanita writes: We have been discussing the following two issues amongst ourselves, then with our methodological consultant for several days. However, we have not been able to arrive at a consensus. Consequently, we decided to seek an opinion from nationally known experts. FYI, we sent a similar inquiry to Larry Hedges and David Rogosa . . . 1) We are wondering if a post-hoc covariate adjustment is a good practice in the context of RCTs [randomized clinical trials]. We have a situation where we found a significant baseline difference between the treatment and the control groups in 3 variables. Some of us argue that adding those three variables to the original impact analysis model is a good idea, as that would remove the confound from the impact estimate. Others among us, on the other hand, argue that a post-hoc covariate adjustment should never be done, on the ground that those covariates are correlated with the treatment, which makes the analysis model that of quasi
6 0.96231627 2149 andrew gelman stats-2013-12-26-Statistical evidence for revised standards
7 0.9567256 1883 andrew gelman stats-2013-06-04-Interrogating p-values
8 0.95433187 899 andrew gelman stats-2011-09-10-The statistical significance filter
9 0.95413548 2299 andrew gelman stats-2014-04-21-Stan Model of the Week: Hierarchical Modeling of Supernovas
11 0.95214754 2161 andrew gelman stats-2014-01-07-My recent debugging experience
12 0.95187485 2201 andrew gelman stats-2014-02-06-Bootstrap averaging: Examples where it works and where it doesn’t work
13 0.95104092 1367 andrew gelman stats-2012-06-05-Question 26 of my final exam for Design and Analysis of Sample Surveys
14 0.95081127 2288 andrew gelman stats-2014-04-10-Small multiples of lineplots > maps (ok, not always, but yes in this case)
15 0.9496218 2055 andrew gelman stats-2013-10-08-A Bayesian approach for peer-review panels? and a speculation about Bruno Frey
16 0.94955969 42 andrew gelman stats-2010-05-19-Updated solutions to Bayesian Data Analysis homeworks
17 0.94931072 1240 andrew gelman stats-2012-04-02-Blogads update
18 0.94906962 1206 andrew gelman stats-2012-03-10-95% intervals that I don’t believe, because they’re from a flat prior I don’t believe
19 0.94902611 1162 andrew gelman stats-2012-02-11-Adding an error model to a deterministic model
20 0.94900215 898 andrew gelman stats-2011-09-10-Fourteen magic words: an update