andrew_gelman_stats andrew_gelman_stats-2013 andrew_gelman_stats-2013-1826 knowledge-graph by maker-knowledge-mining
Source: html
Introduction: Erin Jonaitis points us to this article by Christopher Ferguson and Moritz Heene, who write: Publication bias remains a controversial issue in psychological science. . . . that the field often constructs arguments to block the publication and interpretation of null results and that null results may be further extinguished through questionable researcher practices. Given that science is dependent on the process of falsification, we argue that these problems reduce psychological science’s capability to have a proper mechanism for theory falsification, thus resulting in the promulgation of numerous “undead” theories that are ideologically popular but have little basis in fact. They mention the infamous Daryl Bem article. It is pretty much only because Bem’s claims are (presumably) false that they got published in a major research journal. Had the claims been true—that is, had Bem run identical experiments, analyzed his data more carefully and objectively, and reported that the r
sentIndex sentText sentNum sentScore
1 Erin Jonaitis points us to this article by Christopher Ferguson and Moritz Heene, who write: Publication bias remains a controversial issue in psychological science. [sent-1, score-0.144]
2 that the field often constructs arguments to block the publication and interpretation of null results and that null results may be further extinguished through questionable researcher practices. [sent-5, score-1.191]
3 It is pretty much only because Bem’s claims are (presumably) false that they got published in a major research journal. [sent-8, score-0.177]
4 Had the claims been true—that is, had Bem run identical experiments, analyzed his data more carefully and objectively, and reported that the results were consistent with the null hypothesis—then the result would be entirely unpublishable. [sent-9, score-0.424]
5 Without making this computational error, the FSN turns out to be a gross overestimate of the number of unpublished studies required to bring the mean Z value of published studies to an insignificant level. [sent-23, score-0.59]
6 The FSN thus gives the meta-analytic researcher a false sense of security. [sent-24, score-0.241]
7 The false sense of security persists: Although this fundamental flaw had been spotted early, the number of applications of the FSN has grown exponentially since its publication. [sent-25, score-0.162]
8 Problems with meta-analysis Ferguson and Heene continue: Meta-analyses should be more objective arbiters of review for a field than are narrative reviews, but we argue that this is not the case in practice. [sent-29, score-0.145]
9 The selection and interpretation of effect sizes from individual studies requires decisions that may be susceptible to researcher biases. [sent-33, score-0.383]
10 meta-analyses may be used in such debates to essentially confound the process of replication and falsification. [sent-38, score-0.28]
11 Thus: The average effect size may be largely meaningless and spurious due to the avoidance of null findings in the published literature. [sent-39, score-0.609]
12 This aversion to the null is arguably one of the most pernicious and unscientific aspects of modern social science. [sent-40, score-0.599]
13 I think it’s important to separate the statistical from the scientific null hypothesis. [sent-42, score-0.454]
14 - The statistical null hypothesis is typically that a particular comparison is exactly zero in the population. [sent-43, score-0.491]
15 - The scientific null hypothesis is typically that a certain effect is nonexistent or, more generally, that the effect depends so much on situation as to be unreplicable in general. [sent-44, score-0.773]
16 I might well believe in the scientific null but not in the statistical null. [sent-45, score-0.454]
17 Virtually unkillable Ferguson and Heene continue: The aversion to the null and the persistence of publication bias and denial of the same, renders a situation in which psychological theories are virtually unkillable. [sent-46, score-0.909]
18 Instead of rigid adherence to an objective process of replication and falsification, debates within psychology too easily degenerate into ideological snowball fights, the end result of which is to allow poor quality theories to survive indefinitely. [sent-47, score-0.465]
19 We see this reversal of the burden of proof all the time. [sent-50, score-0.222]
20 do not and cannot provide irrefutable proof of the alleged clerical errors. [sent-56, score-0.14]
wordName wordTfidf (topN-words)
[('fsn', 0.414), ('null', 0.364), ('heene', 0.269), ('ferguson', 0.173), ('proof', 0.14), ('aversion', 0.133), ('unpublished', 0.121), ('publication', 0.119), ('falsification', 0.117), ('bem', 0.103), ('unscientific', 0.102), ('false', 0.099), ('rosenthal', 0.097), ('effect', 0.096), ('studies', 0.092), ('scientific', 0.09), ('theories', 0.087), ('insignificant', 0.085), ('field', 0.084), ('burden', 0.082), ('process', 0.082), ('psychological', 0.079), ('published', 0.078), ('continue', 0.077), ('thus', 0.073), ('virtually', 0.073), ('may', 0.071), ('slate', 0.071), ('researcher', 0.069), ('flawed', 0.068), ('debates', 0.066), ('hypothesis', 0.065), ('controversial', 0.065), ('number', 0.063), ('typically', 0.062), ('objective', 0.061), ('replication', 0.061), ('results', 0.06), ('value', 0.059), ('magazine', 0.058), ('sizes', 0.055), ('theory', 0.055), ('adherence', 0.054), ('erin', 0.054), ('renders', 0.054), ('undead', 0.054), ('claimant', 0.054), ('onus', 0.054), ('antagonists', 0.054), ('degenerate', 0.054)]
simIndex simValue blogId blogTitle
same-blog 1 0.99999988 1826 andrew gelman stats-2013-04-26-“A Vast Graveyard of Undead Theories: Publication Bias and Psychological Science’s Aversion to the Null”
Introduction: Erin Jonaitis points us to this article by Christopher Ferguson and Moritz Heene, who write: Publication bias remains a controversial issue in psychological science. . . . that the field often constructs arguments to block the publication and interpretation of null results and that null results may be further extinguished through questionable researcher practices. Given that science is dependent on the process of falsification, we argue that these problems reduce psychological science’s capability to have a proper mechanism for theory falsification, thus resulting in the promulgation of numerous “undead” theories that are ideologically popular but have little basis in fact. They mention the infamous Daryl Bem article. It is pretty much only because Bem’s claims are (presumably) false that they got published in a major research journal. Had the claims been true—that is, had Bem run identical experiments, analyzed his data more carefully and objectively, and reported that the r
Introduction: Masanao sends this one in, under the heading, “another incident of misunderstood p-value”: Warren Davies, a positive psychology MSc student at UEL, provides the latest in our ongoing series of guest features for students. Warren has just released a Psychology Study Guide, which covers information on statistics, research methods and study skills for psychology students. Despite the myriad rules and procedures of science, some research findings are pure flukes. Perhaps you’re testing a new drug, and by chance alone, a large number of people spontaneously get better. The better your study is conducted, the lower the chance that your result was a fluke – but still, there is always a certain probability that it was. Statistical significance testing gives you an idea of what this probability is. In science we’re always testing hypotheses. We never conduct a study to ‘see what happens’, because there’s always at least one way to make any useless set of data look important. We take
3 0.18796988 1626 andrew gelman stats-2012-12-16-The lamest, grudgingest, non-retraction retraction ever
Introduction: In politics we’re familiar with the non-apology apology (well described in Wikipedia as “a statement that has the form of an apology but does not express the expected contrition”). Here’s the scientific equivalent: the non-retraction retraction. Sanjay Srivastava points to an amusing yet barfable story of a pair of researchers who (inadvertently, I assume) made a data coding error and were eventually moved to issue a correction notice, but even then refused to fully admit their error. As Srivastava puts it, the story “ended up with Lew [Goldberg] and colleagues [Kibeom Lee and Michael Ashton] publishing a comment on an erratum – the only time I’ve ever heard of that happening in a scientific journal.” From the comment on the erratum: In their “erratum and addendum,” Anderson and Ones (this issue) explained that we had brought their attention to the “potential” of a “possible” misalignment and described the results computed from re-aligned data as being based on a “post-ho
4 0.18291174 762 andrew gelman stats-2011-06-13-How should journals handle replication studies?
Introduction: Sanjay Srivastava reports : Recently Ben Goldacre wrote about a group of researchers (Stuart Ritchie, Chris French, and Richard Wiseman) whose null replication of 3 experiments from the infamous Bem ESP paper was rejected by JPSP – the same journal that published Bem’s paper. Srivastava recognizes that JPSP does not usually publish replications but this is a different story because it’s an anti-replication. Here’s the paradox: - From a scientific point of view, the Ritchie et al. results are boring. To find out that there’s no evidence for ESP . . . that adds essentially zero to our scientific understanding. What next, a paper demonstrating that pigeons can fly higher than chickens? Maybe an article in the Journal of the Materials Research Society demonstrating that diamonds can scratch marble but not the reverse?? - But from a science-communication perspective, the null replication is a big deal because it adds credence to my hypothesis that the earlier ESP claims
5 0.17467009 2281 andrew gelman stats-2014-04-04-The Notorious N.H.S.T. presents: Mo P-values Mo Problems
Introduction: A recent discussion between commenters Question and Fernando captured one of the recurrent themes here from the past year. Question: The problem is simple, the researchers are disproving always false null hypotheses and taking this disproof as near proof that their theory is correct. Fernando: Whereas it is probably true that researchers misuse NHT, the problem with tabloid science is broader and deeper. It is systemic. Question: I do not see how anything can be deeper than replacing careful description, prediction, falsification, and independent replication with dynamite plots, p-values, affirming the consequent, and peer review. From my own experience I am confident in saying that confusion caused by NHST is at the root of this problem. Fernando: Incentives? Impact factors? Publish or die? “Interesting” and “new” above quality and reliability, or actually answering a research question, and a silly and unbecoming obsession with being quoted in NYT, etc. . . . Giv
6 0.17353815 1272 andrew gelman stats-2012-04-20-More proposals to reform the peer-review system
7 0.17044693 1607 andrew gelman stats-2012-12-05-The p-value is not . . .
8 0.17016229 2295 andrew gelman stats-2014-04-18-One-tailed or two-tailed?
9 0.16991334 1869 andrew gelman stats-2013-05-24-In which I side with Neyman over Fisher
10 0.16442236 2272 andrew gelman stats-2014-03-29-I agree with this comment
11 0.16031869 1355 andrew gelman stats-2012-05-31-Lindley’s paradox
12 0.15401563 2149 andrew gelman stats-2013-12-26-Statistical evidence for revised standards
13 0.14880246 1998 andrew gelman stats-2013-08-25-A new Bem theory
14 0.14261925 1883 andrew gelman stats-2013-06-04-Interrogating p-values
15 0.14130598 1921 andrew gelman stats-2013-07-01-Going meta on Niall Ferguson
16 0.14020213 1760 andrew gelman stats-2013-03-12-Misunderstanding the p-value
17 0.13885653 1291 andrew gelman stats-2012-04-30-Systematic review of publication bias in studies on publication bias
18 0.13723898 1139 andrew gelman stats-2012-01-26-Suggested resolution of the Bem paradox
19 0.13173579 1493 andrew gelman stats-2012-09-12-Niall Ferguson, the John Yoo line, and the paradox of influence
20 0.1303716 2183 andrew gelman stats-2014-01-23-Discussion on preregistration of research studies
topicId topicWeight
[(0, 0.191), (1, -0.013), (2, -0.011), (3, -0.195), (4, -0.102), (5, -0.088), (6, -0.0), (7, -0.02), (8, -0.013), (9, -0.036), (10, -0.067), (11, 0.01), (12, -0.041), (13, -0.053), (14, 0.014), (15, -0.062), (16, -0.074), (17, -0.12), (18, -0.05), (19, -0.076), (20, -0.025), (21, 0.081), (22, -0.025), (23, -0.027), (24, -0.073), (25, -0.009), (26, 0.065), (27, 0.058), (28, 0.009), (29, -0.019), (30, -0.023), (31, -0.013), (32, 0.057), (33, 0.023), (34, -0.057), (35, -0.031), (36, 0.023), (37, -0.059), (38, 0.023), (39, -0.028), (40, -0.07), (41, 0.005), (42, 0.003), (43, 0.008), (44, 0.034), (45, 0.065), (46, -0.011), (47, -0.007), (48, 0.052), (49, 0.014)]
simIndex simValue blogId blogTitle
same-blog 1 0.97002393 1826 andrew gelman stats-2013-04-26-“A Vast Graveyard of Undead Theories: Publication Bias and Psychological Science’s Aversion to the Null”
Introduction: Erin Jonaitis points us to this article by Christopher Ferguson and Moritz Heene, who write: Publication bias remains a controversial issue in psychological science. . . . that the field often constructs arguments to block the publication and interpretation of null results and that null results may be further extinguished through questionable researcher practices. Given that science is dependent on the process of falsification, we argue that these problems reduce psychological science’s capability to have a proper mechanism for theory falsification, thus resulting in the promulgation of numerous “undead” theories that are ideologically popular but have little basis in fact. They mention the infamous Daryl Bem article. It is pretty much only because Bem’s claims are (presumably) false that they got published in a major research journal. Had the claims been true—that is, had Bem run identical experiments, analyzed his data more carefully and objectively, and reported that the r
2 0.78015447 1883 andrew gelman stats-2013-06-04-Interrogating p-values
Introduction: This article is a discussion of a paper by Greg Francis for a special issue, edited by E. J. Wagenmakers, of the Journal of Mathematical Psychology. Here’s what I wrote: Much of statistical practice is an effort to reduce or deny variation and uncertainty. The reduction is done through standardization, replication, and other practices of experimental design, with the idea being to isolate and stabilize the quantity being estimated and then average over many cases. Even so, however, uncertainty persists, and statistical hypothesis testing is in many ways an endeavor to deny this, by reporting binary accept/reject decisions. Classical statistical methods produce binary statements, but there is no reason to assume that the world works that way. Expressions such as Type 1 error, Type 2 error, false positive, and so on, are based on a model in which the world is divided into real and non-real effects. To put it another way, I understand the general scientific distinction of real vs
3 0.74566978 2272 andrew gelman stats-2014-03-29-I agree with this comment
Introduction: The anonymous commenter puts it well : The problem is simple, the researchers are disproving always false null hypotheses and taking this disproof as near proof that their theory is correct.
Introduction: Masanao sends this one in, under the heading, “another incident of misunderstood p-value”: Warren Davies, a positive psychology MSc student at UEL, provides the latest in our ongoing series of guest features for students. Warren has just released a Psychology Study Guide, which covers information on statistics, research methods and study skills for psychology students. Despite the myriad rules and procedures of science, some research findings are pure flukes. Perhaps you’re testing a new drug, and by chance alone, a large number of people spontaneously get better. The better your study is conducted, the lower the chance that your result was a fluke – but still, there is always a certain probability that it was. Statistical significance testing gives you an idea of what this probability is. In science we’re always testing hypotheses. We never conduct a study to ‘see what happens’, because there’s always at least one way to make any useless set of data look important. We take
5 0.73027128 762 andrew gelman stats-2011-06-13-How should journals handle replication studies?
Introduction: Sanjay Srivastava reports : Recently Ben Goldacre wrote about a group of researchers (Stuart Ritchie, Chris French, and Richard Wiseman) whose null replication of 3 experiments from the infamous Bem ESP paper was rejected by JPSP – the same journal that published Bem’s paper. Srivastava recognizes that JPSP does not usually publish replications but this is a different story because it’s an anti-replication. Here’s the paradox: - From a scientific point of view, the Ritchie et al. results are boring. To find out that there’s no evidence for ESP . . . that adds essentially zero to our scientific understanding. What next, a paper demonstrating that pigeons can fly higher than chickens? Maybe an article in the Journal of the Materials Research Society demonstrating that diamonds can scratch marble but not the reverse?? - But from a science-communication perspective, the null replication is a big deal because it adds credence to my hypothesis that the earlier ESP claims
6 0.7101683 2281 andrew gelman stats-2014-04-04-The Notorious N.H.S.T. presents: Mo P-values Mo Problems
7 0.70747113 2149 andrew gelman stats-2013-12-26-Statistical evidence for revised standards
8 0.69925201 2183 andrew gelman stats-2014-01-23-Discussion on preregistration of research studies
9 0.69785076 2093 andrew gelman stats-2013-11-07-I’m negative on the expression “false positives”
10 0.69469059 1760 andrew gelman stats-2013-03-12-Misunderstanding the p-value
11 0.68042088 1171 andrew gelman stats-2012-02-16-“False-positive psychology”
13 0.66772521 1355 andrew gelman stats-2012-05-31-Lindley’s paradox
14 0.66314667 2295 andrew gelman stats-2014-04-18-One-tailed or two-tailed?
16 0.65480739 1024 andrew gelman stats-2011-11-23-Of hypothesis tests and Unitarians
17 0.6526643 2289 andrew gelman stats-2014-04-11-“More research from the lunatic fringe”
18 0.65236372 1959 andrew gelman stats-2013-07-28-50 shades of gray: A research story
19 0.64903677 1272 andrew gelman stats-2012-04-20-More proposals to reform the peer-review system
topicId topicWeight
[(2, 0.026), (15, 0.056), (16, 0.079), (21, 0.257), (24, 0.122), (41, 0.025), (42, 0.016), (63, 0.012), (86, 0.021), (99, 0.209)]
simIndex simValue blogId blogTitle
1 0.96242082 672 andrew gelman stats-2011-04-20-The R code for those time-use graphs
Introduction: By popular demand, here’s my R script for the time-use graphs : # The data a1 <- c(4.2,3.2,11.1,1.3,2.2,2.0) a2 <- c(3.9,3.2,10.0,0.8,3.1,3.1) a3 <- c(6.3,2.5,9.8,0.9,2.2,2.4) a4 <- c(4.4,3.1,9.8,0.8,3.3,2.7) a5 <- c(4.8,3.0,9.9,0.7,3.3,2.4) a6 <- c(4.0,3.4,10.5,0.7,3.3,2.1) a <- rbind(a1,a2,a3,a4,a5,a6) avg <- colMeans (a) avg.array <- t (array (avg, rev(dim(a)))) diff <- a - avg.array country.name <- c("France", "Germany", "Japan", "Britain", "USA", "Turkey") # The line plots par (mfrow=c(2,3), mar=c(4,4,2,.5), mgp=c(2,.7,0), tck=-.02, oma=c(3,0,4,0), bg="gray96", fg="gray30") for (i in 1:6){ plot (c(1,6), c(-1,1.7), xlab="", ylab="", xaxt="n", yaxt="n", bty="l", type="n") lines (1:6, diff[i,], col="blue") points (1:6, diff[i,], pch=19, col="black") if (i>3){ axis (1, c(1,3,5), c ("Work,\nstudy", "Eat,\nsleep", "Leisure"), mgp=c(2,1.5,0), tck=0, cex.axis=1.2) axis (1, c(2,4,6), c ("Unpaid\nwork", "Personal\nCare", "Other"), mgp=c(2,1.5,0),
same-blog 2 0.94259238 1826 andrew gelman stats-2013-04-26-“A Vast Graveyard of Undead Theories: Publication Bias and Psychological Science’s Aversion to the Null”
Introduction: Erin Jonaitis points us to this article by Christopher Ferguson and Moritz Heene, who write: Publication bias remains a controversial issue in psychological science. . . . that the field often constructs arguments to block the publication and interpretation of null results and that null results may be further extinguished through questionable researcher practices. Given that science is dependent on the process of falsification, we argue that these problems reduce psychological science’s capability to have a proper mechanism for theory falsification, thus resulting in the promulgation of numerous “undead” theories that are ideologically popular but have little basis in fact. They mention the infamous Daryl Bem article. It is pretty much only because Bem’s claims are (presumably) false that they got published in a major research journal. Had the claims been true—that is, had Bem run identical experiments, analyzed his data more carefully and objectively, and reported that the r
3 0.93910694 151 andrew gelman stats-2010-07-16-Wanted: Probability distributions for rank orderings
Introduction: Dietrich Stoyan writes: I asked the IMS people for an expert in statistics of voting/elections and they wrote me your name. I am a statistician, but never worked in the field voting/elections. It was my son-in-law who asked me for statistical theories in that field. He posed in particular the following problem: The aim of the voting is to come to a ranking of c candidates. Every vote is a permutation of these c candidates. The problem is to have probability distributions in the set of all permutations of c elements. Are there theories for such distributions? I should be very grateful for a fast answer with hints to literature. (I confess that I do not know your books.) My reply: Rather than trying to model the ranks directly, I’d recommend modeling a latent continuous outcome which then implies a distribution on ranks, if the ranks are of interest. There are lots of distributions of c-dimensional continuous outcomes. In political science, the usual way to start is
4 0.92572409 1232 andrew gelman stats-2012-03-27-Banned in NYC school tests
Introduction: The list includes “hunting” but not “fishing,” so that’s cool. I wonder how they’d feel about a question involving different cuts of meat. In any case, I’m happy to see that “Bayes” is not on the banned list. P.S. Russell explains .
5 0.92167276 1275 andrew gelman stats-2012-04-22-Please stop me before I barf again
Introduction: Pointing to some horrible graphs, Kaiser writes, “The Earth Institute needs a graphics adviser.” I agree. The graphs are corporate standard, neither pretty or innovative enough to qualify as infographics, not informational enough to be good statistical data displays. Some examples include the above exploding pie chart, which, as Kaiser notes, is not merely ugly and ridiculously difficult to read (given that it is conveying only nine data points) but also invites suspicion of its numbers, and pages and pages of graphs that could be better compressed into a compact displays (see pages 25-65 of the report). Yes, this is all better than tables of numbers, but I don’t see that much thought went into displaying patterns of information or telling a story. It’s more graph-as-data-dump. To be fair, the report does have some a clean scatterplot (on page 65). But, overall, the graphs are not well-integrated with the messages in the text. I feel a little bit bad about this, beca
6 0.91772735 432 andrew gelman stats-2010-11-27-Neumann update
8 0.91370553 2298 andrew gelman stats-2014-04-21-On deck this week
9 0.90420121 62 andrew gelman stats-2010-06-01-Two Postdoc Positions Available on Bayesian Hierarchical Modeling
10 0.90325511 894 andrew gelman stats-2011-09-07-Hipmunk FAIL: Graphics without content is not enough
11 0.90034962 1401 andrew gelman stats-2012-06-30-David Hogg on statistics
12 0.89441299 1857 andrew gelman stats-2013-05-15-Does quantum uncertainty have a place in everyday applied statistics?
13 0.89372492 1675 andrew gelman stats-2013-01-15-“10 Things You Need to Know About Causal Effects”
15 0.88699567 854 andrew gelman stats-2011-08-15-A silly paper that tries to make fun of multilevel models
16 0.88547844 514 andrew gelman stats-2011-01-13-News coverage of statistical issues…how did I do?
18 0.86870587 659 andrew gelman stats-2011-04-13-Jim Campbell argues that Larry Bartels’s “Unequal Democracy” findings are not robust
19 0.8581481 2272 andrew gelman stats-2014-03-29-I agree with this comment
20 0.85214186 810 andrew gelman stats-2011-07-20-Adding more information can make the variance go up (depending on your model)