andrew_gelman_stats andrew_gelman_stats-2013 andrew_gelman_stats-2013-2149 knowledge-graph by maker-knowledge-mining
Source: html
Introduction: In response to the discussion of X and me of his recent paper , Val Johnson writes: I would like to thank Andrew for forwarding his comments on uniformly most powerful Bayesian tests (UMPBTs) to me and his invitation to respond to them. I think he (and also Christian Robert) raise a number of interesting points concerning this new class of Bayesian tests, but I think that they may have confounded several issues that might more usefully be examined separately. The first issue involves the choice of the Bayesian evidence threshold, gamma, used in rejecting a null hypothesis in favor of an alternative hypothesis. Andrew objects to the higher values of gamma proposed in my recent PNAS article on grounds that too many important scientific effects would be missed if thresholds of 25-50 were routinely used. These evidence thresholds correspond roughly to p-values of 0.005; Andrew suggests that evidence thresholds around 5 should continue to be used (gamma=5 corresponds approximate
sentIndex sentText sentNum sentScore
1 The first issue involves the choice of the Bayesian evidence threshold, gamma, used in rejecting a null hypothesis in favor of an alternative hypothesis. [sent-3, score-0.806]
2 Andrew objects to the higher values of gamma proposed in my recent PNAS article on grounds that too many important scientific effects would be missed if thresholds of 25-50 were routinely used. [sent-4, score-0.541]
3 005; Andrew suggests that evidence thresholds around 5 should continue to be used (gamma=5 corresponds approximately to a 5% significance test). [sent-6, score-0.544]
4 My personal experience at a large cancer center suggests that many more than one-half of tested null hypotheses are true (i. [sent-8, score-0.464]
5 And if more than 50% of tested null hypotheses are true, the use of an evidence threshold of 5 is guaranteed to lead to false positive rates that are even higher than 20%. [sent-15, score-1.3]
6 Of course, the evidence threshold that one chooses for declaring a scientific discovery is an inherently subjective choice, and arguments that higher false positive rates should be accepted in order to avoid missing true effects can certainly be made. [sent-16, score-1.074]
7 Regardless of the evidence threshold that one chooses, I think it is important to separate the choice of this threshold from the method that is used to determine the form of the alternative hypothesis that is specified. [sent-19, score-1.241]
8 Instead of classifying these tests as minimax procedures, I think it is more appropriate to regard them as Bayesian analogs of most powerful tests (hence their name). [sent-29, score-0.462]
9 Simply put, UMPBTs are the class of objective Bayesian hypothesis tests that maximize the probability that the Bayes factor in favor of the alternative hypothesis exceeds a specified threshold. [sent-30, score-0.875]
10 Given that an evidence threshold has been agreed upon, the objection to UMPBTs on the basis that they lead to high rates of false negatives therefore seems misguided. [sent-31, score-0.857]
11 If one decides to use an evidence threshold of, say 5, then the UMPBT based on gamma=5 is exactly the Bayesian test that maximizes the probability that an effect is detected. [sent-32, score-0.617]
12 No other test, based on either a subjectively or objectively specified alternative hypothesis, is as likely to produce a Bayes factor that exceeds the specified evidence threshold. [sent-33, score-0.728]
13 The only other way to increase the posterior probability of the alternative hypothesis is to adjust the prior probability assigned to it. [sent-35, score-0.46]
14 When a scientist designs an experiment in which a very large number of items are sampled, she is attempting to either (a) detect a very small effect size, or (b) gain overwhelming evidence in favor of one of the tested hypotheses. [sent-40, score-0.485]
15 In case (a), it is reasonable to regard the evidence threshold as being fixed at a commonly used value (i. [sent-41, score-0.611]
16 On the other hand, the assumptions inherent to case (b) imply that a higher evidence threshold has been specified and that the sample size was increased to accommodate this higher threshold. [sent-48, score-1.095]
17 When the sample size is increased to obtain a higher degree of certainty regarding the validity of competing hypotheses, then the apparent paradox again disappears because the alternative hypothesis specified under the UMPBT does not collapse onto the null value. [sent-49, score-1.007]
18 I am not “suggesting that evidence thresholds around 5 should continue to be used. [sent-53, score-0.492]
19 Second, evidence thresholds based on conventional priors do not represent posterior distributions that I’d want to use for decisions. [sent-56, score-0.543]
20 As Val notes in his original paper, changing the threshold makes it only a little bit more difficult to identify large real effects, while putting a much greater burden on those researchers who are shuffling noise around. [sent-71, score-0.456]
wordName wordTfidf (topN-words)
[('threshold', 0.333), ('val', 0.3), ('umpbts', 0.272), ('evidence', 0.223), ('thresholds', 0.214), ('null', 0.175), ('umpbt', 0.171), ('minimax', 0.169), ('specified', 0.15), ('alternative', 0.143), ('hypothesis', 0.142), ('hypotheses', 0.134), ('loss', 0.13), ('tests', 0.119), ('gamma', 0.116), ('paradox', 0.115), ('false', 0.107), ('higher', 0.107), ('scientific', 0.104), ('bayesian', 0.101), ('andrew', 0.1), ('tested', 0.097), ('patients', 0.077), ('rates', 0.071), ('negatives', 0.07), ('declaring', 0.068), ('invitation', 0.068), ('choice', 0.067), ('decision', 0.067), ('greater', 0.065), ('functions', 0.064), ('size', 0.063), ('exceeds', 0.062), ('probability', 0.061), ('chooses', 0.061), ('commentary', 0.06), ('increased', 0.06), ('concerning', 0.059), ('large', 0.058), ('favor', 0.056), ('continue', 0.055), ('regard', 0.055), ('priors', 0.054), ('lead', 0.053), ('prior', 0.053), ('examined', 0.052), ('sample', 0.052), ('distributions', 0.052), ('significance', 0.052), ('detect', 0.051)]
simIndex simValue blogId blogTitle
same-blog 1 0.99999994 2149 andrew gelman stats-2013-12-26-Statistical evidence for revised standards
Introduction: In response to the discussion of X and me of his recent paper , Val Johnson writes: I would like to thank Andrew for forwarding his comments on uniformly most powerful Bayesian tests (UMPBTs) to me and his invitation to respond to them. I think he (and also Christian Robert) raise a number of interesting points concerning this new class of Bayesian tests, but I think that they may have confounded several issues that might more usefully be examined separately. The first issue involves the choice of the Bayesian evidence threshold, gamma, used in rejecting a null hypothesis in favor of an alternative hypothesis. Andrew objects to the higher values of gamma proposed in my recent PNAS article on grounds that too many important scientific effects would be missed if thresholds of 25-50 were routinely used. These evidence thresholds correspond roughly to p-values of 0.005; Andrew suggests that evidence thresholds around 5 should continue to be used (gamma=5 corresponds approximate
Introduction: As regular readers of this blog are aware, a few months ago Val Johnson published an article, “Revised standards for statistical evidence,” making a Bayesian argument that researchers and journals should use a p=0.005 publication threshold rather than the usual p=0.05. Christian Robert and I were unconvinced by Val’s reasoning and wrote a response , “Revised evidence for statistical standards,” in which we wrote: Johnson’s minimax prior is not intended to correspond to any distribution of effect sizes; rather, it represents a worst case scenario under some mathematical assumptions. Minimax and tradeoffs do well together, and it is hard for us to see how any worst case procedure can supply much guidance on how to balance between two different losses. . . . We would argue that the appropriate significance level depends on the scenario and that what worked well for agricultural experiments in the 1920s might not be so appropriate for many applications in modern biosciences . . .
3 0.33157128 2140 andrew gelman stats-2013-12-19-Revised evidence for statistical standards
Introduction: X and I heard about this much-publicized recent paper by Val Johnson, who suggests changing the default level of statistical significance from z=2 to z=3 (or, as he puts it, going from p=.05 to p=.005 or .001). Val argues that you need to go out to 3 standard errors to get a Bayes factor of 25 or 50 in favor of the alternative hypothesis. I don’t really buy this, first because Val’s model is a weird (to me) mixture of two point masses, which he creates in order to make a minimax argument, and second because I don’t see why you need a Bayes factor of 25 to 50 in order to make a claim. I’d think that a factor of 5:1, say, provides strong information already—if you really believe those odds. The real issue, as I see it, is that we’re getting Bayes factors and posterior probabilities we don’t believe, because we’re assuming flat priors that don’t really make sense. This is a topic that’s come up over and over in recent months on this blog, for example in this discussion of why I d
4 0.2431225 2295 andrew gelman stats-2014-04-18-One-tailed or two-tailed?
Introduction: Someone writes: Suppose I have two groups of people, A and B, which differ on some characteristic of interest to me; and for each person I measure a single real-valued quantity X. I have a theory that group A has a higher mean value of X than group B. I test this theory by using a t-test. Am I entitled to use a *one-tailed* t-test? Or should I use a *two-tailed* one (thereby giving a p-value that is twice as large)? I know you will probably answer: Forget the t-test; you should use Bayesian methods instead. But what is the standard frequentist answer to this question? My reply: The quick answer here is that different people will do different things here. I would say the 2-tailed p-value is more standard but some people will insist on the one-tailed version, and it’s hard to make a big stand on this one, given all the other problems with p-values in practice: http://www.stat.columbia.edu/~gelman/research/unpublished/p_hacking.pdf http://www.stat.columbia.edu/~gelm
Introduction: Ken Rice writes: In the recent discussion on stopping rules I saw a comment that I wanted to chip in on, but thought it might get a bit lost, in the already long thread. Apologies in advance if I misinterpreted what you wrote, or am trying to tell you things you already know. The comment was: “In Bayesian decision making, there is a utility function and you choose the decision with highest expected utility. Making a decision based on statistical significance does not correspond to any utility function.” … which immediately suggests this little 2010 paper; A Decision-Theoretic Formulation of Fisher’s Approach to Testing, The American Statistician, 64(4) 345-349. It contains utilities that lead to decisions that very closely mimic classical Wald tests, and provides a rationale for why this utility is not totally unconnected from how some scientists think. Some (old) slides discussing it are here . A few notes, on things not in the paper: * I know you don’t like squared-
6 0.20619024 1355 andrew gelman stats-2012-05-31-Lindley’s paradox
7 0.19234157 1723 andrew gelman stats-2013-02-15-Wacky priors can work well?
8 0.18244739 2263 andrew gelman stats-2014-03-24-Empirical implications of Empirical Implications of Theoretical Models
9 0.16541202 495 andrew gelman stats-2010-12-31-“Threshold earners” and economic inequality
11 0.16472875 2281 andrew gelman stats-2014-04-04-The Notorious N.H.S.T. presents: Mo P-values Mo Problems
12 0.16250585 758 andrew gelman stats-2011-06-11-Hey, good news! Your p-value just passed the 0.05 threshold!
13 0.15501757 506 andrew gelman stats-2011-01-06-That silly ESP paper and some silliness in a rebuttal as well
15 0.15311992 291 andrew gelman stats-2010-09-22-Philosophy of Bayes and non-Bayes: A dialogue with Deborah Mayo
16 0.15239222 1695 andrew gelman stats-2013-01-28-Economists argue about Bayes
17 0.1518503 1247 andrew gelman stats-2012-04-05-More philosophy of Bayes
18 0.14939621 1974 andrew gelman stats-2013-08-08-Statistical significance and the dangerous lure of certainty
19 0.14637716 1817 andrew gelman stats-2013-04-21-More on Bayesian model selection in high-dimensional settings
20 0.14553943 1605 andrew gelman stats-2012-12-04-Write This Book
topicId topicWeight
[(0, 0.254), (1, 0.126), (2, 0.006), (3, -0.119), (4, -0.126), (5, -0.081), (6, 0.031), (7, 0.081), (8, -0.064), (9, -0.127), (10, -0.142), (11, -0.003), (12, 0.045), (13, -0.093), (14, 0.04), (15, -0.006), (16, 0.006), (17, -0.025), (18, -0.025), (19, -0.011), (20, 0.032), (21, 0.054), (22, -0.062), (23, -0.006), (24, -0.08), (25, -0.052), (26, 0.073), (27, 0.03), (28, 0.061), (29, -0.005), (30, 0.001), (31, 0.007), (32, 0.057), (33, 0.09), (34, -0.056), (35, -0.069), (36, 0.075), (37, -0.055), (38, 0.078), (39, -0.006), (40, -0.112), (41, -0.076), (42, -0.018), (43, 0.07), (44, -0.003), (45, 0.043), (46, -0.036), (47, -0.051), (48, 0.043), (49, 0.001)]
simIndex simValue blogId blogTitle
same-blog 1 0.96893179 2149 andrew gelman stats-2013-12-26-Statistical evidence for revised standards
Introduction: In response to the discussion of X and me of his recent paper , Val Johnson writes: I would like to thank Andrew for forwarding his comments on uniformly most powerful Bayesian tests (UMPBTs) to me and his invitation to respond to them. I think he (and also Christian Robert) raise a number of interesting points concerning this new class of Bayesian tests, but I think that they may have confounded several issues that might more usefully be examined separately. The first issue involves the choice of the Bayesian evidence threshold, gamma, used in rejecting a null hypothesis in favor of an alternative hypothesis. Andrew objects to the higher values of gamma proposed in my recent PNAS article on grounds that too many important scientific effects would be missed if thresholds of 25-50 were routinely used. These evidence thresholds correspond roughly to p-values of 0.005; Andrew suggests that evidence thresholds around 5 should continue to be used (gamma=5 corresponds approximate
Introduction: As regular readers of this blog are aware, a few months ago Val Johnson published an article, “Revised standards for statistical evidence,” making a Bayesian argument that researchers and journals should use a p=0.005 publication threshold rather than the usual p=0.05. Christian Robert and I were unconvinced by Val’s reasoning and wrote a response , “Revised evidence for statistical standards,” in which we wrote: Johnson’s minimax prior is not intended to correspond to any distribution of effect sizes; rather, it represents a worst case scenario under some mathematical assumptions. Minimax and tradeoffs do well together, and it is hard for us to see how any worst case procedure can supply much guidance on how to balance between two different losses. . . . We would argue that the appropriate significance level depends on the scenario and that what worked well for agricultural experiments in the 1920s might not be so appropriate for many applications in modern biosciences . . .
3 0.87021822 1355 andrew gelman stats-2012-05-31-Lindley’s paradox
Introduction: Sam Seaver writes: I [Seaver] happened to be reading an ironic article by Karl Friston when I learned something new about frequentist vs bayesian, namely Lindley’s paradox, on page 12. The text is as follows: So why are we worried about trivial effects? They are important because the probability that the true effect size is exactly zero is itself zero and could cause us to reject the null hypothesis inappropriately. This is a fallacy of classical inference and is not unrelated to Lindley’s paradox (Lindley 1957). Lindley’s paradox describes a counterintuitive situation in which Bayesian and frequentist approaches to hypothesis testing give opposite results. It occurs when; (i) a result is significant by a frequentist test, indicating sufficient evidence to reject the null hypothesis d=0 and (ii) priors render the posterior probability of d=0 high, indicating strong evidence that the null hypothesis is true. In his original treatment, Lindley (1957) showed that – under a parti
4 0.82092822 2295 andrew gelman stats-2014-04-18-One-tailed or two-tailed?
Introduction: Someone writes: Suppose I have two groups of people, A and B, which differ on some characteristic of interest to me; and for each person I measure a single real-valued quantity X. I have a theory that group A has a higher mean value of X than group B. I test this theory by using a t-test. Am I entitled to use a *one-tailed* t-test? Or should I use a *two-tailed* one (thereby giving a p-value that is twice as large)? I know you will probably answer: Forget the t-test; you should use Bayesian methods instead. But what is the standard frequentist answer to this question? My reply: The quick answer here is that different people will do different things here. I would say the 2-tailed p-value is more standard but some people will insist on the one-tailed version, and it’s hard to make a big stand on this one, given all the other problems with p-values in practice: http://www.stat.columbia.edu/~gelman/research/unpublished/p_hacking.pdf http://www.stat.columbia.edu/~gelm
Introduction: Ken Rice writes: In the recent discussion on stopping rules I saw a comment that I wanted to chip in on, but thought it might get a bit lost, in the already long thread. Apologies in advance if I misinterpreted what you wrote, or am trying to tell you things you already know. The comment was: “In Bayesian decision making, there is a utility function and you choose the decision with highest expected utility. Making a decision based on statistical significance does not correspond to any utility function.” … which immediately suggests this little 2010 paper; A Decision-Theoretic Formulation of Fisher’s Approach to Testing, The American Statistician, 64(4) 345-349. It contains utilities that lead to decisions that very closely mimic classical Wald tests, and provides a rationale for why this utility is not totally unconnected from how some scientists think. Some (old) slides discussing it are here . A few notes, on things not in the paper: * I know you don’t like squared-
6 0.76980162 2281 andrew gelman stats-2014-04-04-The Notorious N.H.S.T. presents: Mo P-values Mo Problems
7 0.76719356 2140 andrew gelman stats-2013-12-19-Revised evidence for statistical standards
8 0.76130718 331 andrew gelman stats-2010-10-10-Bayes jumps the shark
9 0.7443164 1883 andrew gelman stats-2013-06-04-Interrogating p-values
10 0.74428791 1024 andrew gelman stats-2011-11-23-Of hypothesis tests and Unitarians
11 0.73661423 643 andrew gelman stats-2011-04-02-So-called Bayesian hypothesis testing is just as bad as regular hypothesis testing
13 0.72252756 314 andrew gelman stats-2010-10-03-Disconnect between drug and medical device approval
14 0.72156155 2272 andrew gelman stats-2014-03-29-I agree with this comment
16 0.70889997 1713 andrew gelman stats-2013-02-08-P-values and statistical practice
17 0.70818675 2183 andrew gelman stats-2014-01-23-Discussion on preregistration of research studies
18 0.70616168 2102 andrew gelman stats-2013-11-15-“Are all significant p-values created equal?”
19 0.68432987 2078 andrew gelman stats-2013-10-26-“The Bayesian approach to forensic evidence”
20 0.67812854 1869 andrew gelman stats-2013-05-24-In which I side with Neyman over Fisher
topicId topicWeight
[(6, 0.017), (15, 0.047), (16, 0.079), (21, 0.045), (24, 0.212), (49, 0.015), (53, 0.081), (55, 0.037), (84, 0.042), (86, 0.044), (89, 0.015), (99, 0.244)]
simIndex simValue blogId blogTitle
same-blog 1 0.97364986 2149 andrew gelman stats-2013-12-26-Statistical evidence for revised standards
Introduction: In response to the discussion of X and me of his recent paper , Val Johnson writes: I would like to thank Andrew for forwarding his comments on uniformly most powerful Bayesian tests (UMPBTs) to me and his invitation to respond to them. I think he (and also Christian Robert) raise a number of interesting points concerning this new class of Bayesian tests, but I think that they may have confounded several issues that might more usefully be examined separately. The first issue involves the choice of the Bayesian evidence threshold, gamma, used in rejecting a null hypothesis in favor of an alternative hypothesis. Andrew objects to the higher values of gamma proposed in my recent PNAS article on grounds that too many important scientific effects would be missed if thresholds of 25-50 were routinely used. These evidence thresholds correspond roughly to p-values of 0.005; Andrew suggests that evidence thresholds around 5 should continue to be used (gamma=5 corresponds approximate
2 0.97140002 547 andrew gelman stats-2011-01-31-Using sample size in the prior distribution
Introduction: Mike McLaughlin writes: Consider the Seeds example in vol. 1 of the BUGS examples. There, a binomial likelihood has a p parameter constructed, via logit, from two covariates. What I am wondering is: Would it be legitimate, in a binomial + logit problem like this, to allow binomial p[i] to be a function of the corresponding n[i] or would that amount to using the data in the prior? In other words, in the context of the Seeds example, is r[] the only data or is n[] data as well and therefore not permissible in a prior formulation? I [McLaughlin] currently have a model with a common beta prior for all p[i] but would like to mitigate this commonality (a kind of James-Stein effect) when there are lots of observations for some i. But this seems to feed the data back into the prior. Does it really? It also occurs to me [McLaughlin] that, perhaps, a binomial likelihood is not the one to use here (not flexible enough). My reply: Strictly speaking, “n” is data, and so what you wa
Introduction: As regular readers of this blog are aware, a few months ago Val Johnson published an article, “Revised standards for statistical evidence,” making a Bayesian argument that researchers and journals should use a p=0.005 publication threshold rather than the usual p=0.05. Christian Robert and I were unconvinced by Val’s reasoning and wrote a response , “Revised evidence for statistical standards,” in which we wrote: Johnson’s minimax prior is not intended to correspond to any distribution of effect sizes; rather, it represents a worst case scenario under some mathematical assumptions. Minimax and tradeoffs do well together, and it is hard for us to see how any worst case procedure can supply much guidance on how to balance between two different losses. . . . We would argue that the appropriate significance level depends on the scenario and that what worked well for agricultural experiments in the 1920s might not be so appropriate for many applications in modern biosciences . . .
4 0.95382035 1858 andrew gelman stats-2013-05-15-Reputations changeable, situations tolerable
Introduction: David Kessler, Peter Hoff, and David Dunson write : Marginally specified priors for nonparametric Bayesian estimation Prior specification for nonparametric Bayesian inference involves the difficult task of quantifying prior knowledge about a parameter of high, often infinite, dimension. Realistically, a statistician is unlikely to have informed opinions about all aspects of such a parameter, but may have real information about functionals of the parameter, such the population mean or variance. This article proposes a new framework for nonparametric Bayes inference in which the prior distribution for a possibly infinite-dimensional parameter is decomposed into two parts: an informative prior on a finite set of functionals, and a nonparametric conditional prior for the parameter given the functionals. Such priors can be easily constructed from standard nonparametric prior distributions in common use, and inherit the large support of the standard priors upon which they are based. Ad
5 0.95096004 687 andrew gelman stats-2011-04-29-Zero is zero
Introduction: Nathan Roseberry writes: I thought I had read on your blog that bar charts should always include zero on the scale, but a search of your blog (or google) didn’t return what I was looking for. Is it considered a best practice to always include zero on the axis for bar charts? Has this been written in a book? My reply: The idea is that the area of the bar represents “how many” or “how much.” The bar has to go down to 0 for that to work. You don’t have to have your y-axis go to zero, but if you want the axis to go anywhere else, don’t use a bar graph, use a line graph. Usually line graphs are better anyway. I’m sure this is all in a book somewhere.
6 0.94924682 2299 andrew gelman stats-2014-04-21-Stan Model of the Week: Hierarchical Modeling of Supernovas
7 0.94753033 1240 andrew gelman stats-2012-04-02-Blogads update
8 0.94684052 1047 andrew gelman stats-2011-12-08-I Am Too Absolutely Heteroskedastic for This Probit Model
9 0.94608921 488 andrew gelman stats-2010-12-27-Graph of the year
10 0.94515532 898 andrew gelman stats-2011-09-10-Fourteen magic words: an update
11 0.94475967 1883 andrew gelman stats-2013-06-04-Interrogating p-values
12 0.94263983 899 andrew gelman stats-2011-09-10-The statistical significance filter
13 0.94211996 2029 andrew gelman stats-2013-09-18-Understanding posterior p-values
14 0.94123137 42 andrew gelman stats-2010-05-19-Updated solutions to Bayesian Data Analysis homeworks
15 0.94109714 1080 andrew gelman stats-2011-12-24-Latest in blog advertising
16 0.94073308 846 andrew gelman stats-2011-08-09-Default priors update?
17 0.94019794 1155 andrew gelman stats-2012-02-05-What is a prior distribution?
18 0.93923855 1713 andrew gelman stats-2013-02-08-P-values and statistical practice
19 0.93910694 1367 andrew gelman stats-2012-06-05-Question 26 of my final exam for Design and Analysis of Sample Surveys
20 0.93895209 446 andrew gelman stats-2010-12-03-Is 0.05 too strict as a p-value threshold?