andrew_gelman_stats andrew_gelman_stats-2013 andrew_gelman_stats-2013-2006 knowledge-graph by maker-knowledge-mining
Source: html
Introduction: Following up on my entry the other day on post-publication peer review, Dan Kahan writes: You give me credit, I think, for merely participating in what I think is a systemic effect in the practice of empirical inquiry that conduces to quality control & hence the advance of knowledge by such means (likely the title conveys that!). I’d say: (a) by far the greatest weakness in the “publication regime” in social sciences today is the systematic disregard for basic principles of valid causal inference, a deficiency either in comprehension or craft that is at the root of scholars’ resort to (and journals’ tolerance for) invalid samples, the employment of designs that don’t generate observations more consistent with a hypothesis than with myriad rival ones, and the resort to deficient statistical modes of analysis that treat detection of “statististically significant difference” rather than “practical corroboration of practical meaningful effect” as the goal of such analysis (especial
sentIndex sentText sentNum sentScore
1 5 or 3 in rare instances 4 people read a paper beforehand & many times that many after; didn’t someone write a paper recently on need to apply to knowledge on validity & reliability to the procedures we use for producing/disseminating/teaching such knowledge? [sent-6, score-0.364]
2 To me this makes the impact of *bad” papers self-limiting. [sent-9, score-0.207]
3 So I don’t worry that much about publication of bad papers. [sent-12, score-0.168]
4 I don’t know if I agree with Kahan’s claim that the impact of *bad” papers will be “self-limiting. [sent-15, score-0.207]
5 Once something has passed pre-publication peer review, the scientific community mostly either accepts it uncritically, ignores it entirely, or else miscites it as supporting whatever conclusion the citing author prefers. [sent-17, score-0.433]
6 Pre-publication peer review is an institutionalized practice that gets around this very human desire to want to think well of one’s peers, and to have them think well of you. [sent-27, score-0.657]
7 That’s why, as frustrated as I (and probably all of you) often get with pre-publication peer review, I’d like to see it reformed rather than replaced. [sent-28, score-0.216]
8 Fox is talking about biology and ecology, but I suspect these problems are going on in other scientific fields as well, and Fox’s perspective seems similar to that of Nicolas Chopin, Kerrie Mengersen, and Christian Robert in our article, In praise of the referee . [sent-32, score-0.109]
9 But I brought this up right now not to discuss peer review but to emphasize that once a mistake is published, it’s hard to dislodge it. [sent-33, score-0.357]
10 Anyway, to continue with the main thread, here’s Kahan again: I think the “WTF” findings are more likely to get “pounded in” than bad studies on things that actually matter. [sent-34, score-0.318]
11 The things that matter are issues of consequence for knowledge or practice that usually admit of multiple competing explanations– the ones in the EOOOYKTA – “everything-is-obvious-once-you-know-the-answer” — set, which is where you will find *serious* social scientists laboring. [sent-35, score-0.297]
12 There I think the life of an invalid study is likely to be short, even if it starts out w/ much fanfare. [sent-36, score-0.26]
13 It is short, moreover, b/c it *lives* in the minds of serious people, who really want to know what’s going on. [sent-37, score-0.18]
14 And, via the sort of science journalism you criticized in your Symposium article , gets pounded “deeply and perhaps irretrievably into the recursive pathways of knowledge transmission associated with the internet. [sent-39, score-0.426]
15 He or she can’t possibly be *telling the story* that a person who is genuinely intrested in scientific discovery cares about. [sent-44, score-0.243]
16 A journalist ought to do that — just b/c he or she ought to; it’s the craft of the profession that person is in. [sent-46, score-0.357]
17 For one thing, when I first encountered the “dentists named Dennis and lawyers named Laura” paper, I simply took it as true . [sent-49, score-0.176]
18 Even now, after the paper has been subject to serious criticism, I still don’t know what to believe . [sent-50, score-0.182]
19 I think it’s fine for science reporters to read scientific papers, but I think it’s hard for any outsider to spot the flaws. [sent-53, score-0.426]
20 If I can’t reliably do it and Steven Levitt can’t reliably do it, I think journalists will have trouble with this task too. [sent-54, score-0.245]
wordName wordTfidf (topN-words)
[('peer', 0.216), ('kahan', 0.178), ('wtf', 0.153), ('review', 0.141), ('knowledge', 0.136), ('genuinely', 0.134), ('papers', 0.129), ('craft', 0.119), ('pounded', 0.119), ('transmitting', 0.119), ('proven', 0.112), ('fox', 0.11), ('scientific', 0.109), ('conclusion', 0.108), ('science', 0.105), ('serious', 0.103), ('invalid', 0.102), ('comprehension', 0.098), ('resort', 0.098), ('bad', 0.096), ('conclusive', 0.094), ('hurricane', 0.094), ('journalist', 0.094), ('researcher', 0.093), ('victims', 0.092), ('observations', 0.089), ('named', 0.088), ('valid', 0.088), ('reliably', 0.087), ('causal', 0.087), ('likely', 0.087), ('inference', 0.086), ('consequence', 0.082), ('well', 0.079), ('paper', 0.079), ('ones', 0.079), ('impact', 0.078), ('worthy', 0.078), ('minds', 0.077), ('attention', 0.075), ('journals', 0.074), ('ought', 0.072), ('publication', 0.072), ('think', 0.071), ('production', 0.07), ('proof', 0.07), ('read', 0.07), ('evidence', 0.069), ('journalism', 0.066), ('findings', 0.064)]
simIndex simValue blogId blogTitle
same-blog 1 0.99999934 2006 andrew gelman stats-2013-09-03-Evaluating evidence from published research
Introduction: Following up on my entry the other day on post-publication peer review, Dan Kahan writes: You give me credit, I think, for merely participating in what I think is a systemic effect in the practice of empirical inquiry that conduces to quality control & hence the advance of knowledge by such means (likely the title conveys that!). I’d say: (a) by far the greatest weakness in the “publication regime” in social sciences today is the systematic disregard for basic principles of valid causal inference, a deficiency either in comprehension or craft that is at the root of scholars’ resort to (and journals’ tolerance for) invalid samples, the employment of designs that don’t generate observations more consistent with a hypothesis than with myriad rival ones, and the resort to deficient statistical modes of analysis that treat detection of “statististically significant difference” rather than “practical corroboration of practical meaningful effect” as the goal of such analysis (especial
Introduction: Mike Spagat points to this interview , which, he writes, covers themes that are discussed on the blog such as wrong ideas that don’t die, peer review and the statistics of conflict deaths. I agree. It’s good stuff. Here are some of the things that Spagat says (he’s being interviewed by Joel Wing): In fact, the standard excess-deaths concept leads to an interesting conundrum when combined with an interesting fact exposed in the next-to-latest Human Security Report ; in most countries child mortality rates decline during armed conflict (chapter 6). So if you believe the usual excess-death causality story then you’re forced to conclude that many conflicts actually save the lives of many children. Of course, the idea of wars savings lives is pretty hard to swallow. A much more sensible understanding is that there are a variety of factors that determine child deaths and that in many cases the factors that save the lives of children are stronger than the negative effects that confli
3 0.21717755 2245 andrew gelman stats-2014-03-12-More on publishing in journals
Introduction: I’m postponing today’s scheduled post (“Empirical implications of Empirical Implications of Theoretical Models”) to continue the lively discussion from yesterday, What if I were to stop publishing in journals? . An example: my papers with Basbøll Thomas Basbøll and I got into a long discussion on our blogs about business school professor Karl Weick and other cases of plagiarism copying text without attribution. We felt it useful to take our ideas to the next level and write them up as a manuscript, which ended up being logical to split into two papers. At that point I put some effort into getting these papers published, which I eventually did: To throw away data: Plagiarism as a statistical crime went into American Scientist and When do stories work? Evidence and illustration in the social sciences will appear in Sociological Methods and Research. The second paper, in particular, took some effort to place; I got some advice from colleagues in sociology as to where
4 0.18821217 1928 andrew gelman stats-2013-07-06-How to think about papers published in low-grade journals?
Introduction: We’ve had lots of lively discussions of fatally-flawed papers that have been published in top, top journals such as the American Economic Review or the Journal of Personality and Social Psychology or the American Sociological Review or the tabloids . And we also know about mistakes that make their way into mid-ranking outlets such as the Journal of Theoretical Biology. But what about results that appear in the lower tier of legitimate journals? I was thinking about this after reading a post by Dan Kahan slamming a paper that recently appeared in PLOS-One. I won’t discuss the paper itself here because that’s not my point. Rather, I had some thoughts regarding Kahan’s annoyance that a paper with fatal errors was published at all. I commented as follows: Read between the lines. The paper originally was released in 2009 and was published in 2013 in PLOS-One, which is one step above appearing on Arxiv. PLOS-One publishes some good things (so does Arxiv) but it’s the place
Introduction: It all began with this message from Dan Kahan, a law professor who does psychology experiments: My graphs– what do you think?? I guess what do you think of the result too, but the answer is, “That’s obvious!” If it hadn’t been, then it would have been suspicious in my book. Of course, if we had found the opposite result, that would have been “obvious!” too. We are submitting to LR ≠1 Journa l This is the latest study in series looking at relationship between critical reasoning capacities and “cultural cognition” — the tendency of individuals to conform their perceptions of risk & other policy-relevant facts to their group commitments. The first installment was an observational study that found that cultural polarization ( political too ; the distinction relate not to the mechanism for polarization over decision-relevant science but only about how to measure what is hypothesized to be driving it) increases as people become more science literate. This paper and ano
6 0.17031756 1139 andrew gelman stats-2012-01-26-Suggested resolution of the Bem paradox
8 0.15842022 1878 andrew gelman stats-2013-05-31-How to fix the tabloids? Toward replicable social science research
10 0.15055375 1844 andrew gelman stats-2013-05-06-Against optimism about social science
11 0.14824495 2004 andrew gelman stats-2013-09-01-Post-publication peer review: How it (sometimes) really works
12 0.14636919 2361 andrew gelman stats-2014-06-06-Hurricanes vs. Himmicanes
13 0.14464599 1683 andrew gelman stats-2013-01-19-“Confirmation, on the other hand, is not sexy”
14 0.14364095 2269 andrew gelman stats-2014-03-27-Beyond the Valley of the Trolls
15 0.14271303 2301 andrew gelman stats-2014-04-22-Ticket to Baaaaarf
16 0.13915816 2353 andrew gelman stats-2014-05-30-I posted this as a comment on a sociology blog
17 0.13821723 1435 andrew gelman stats-2012-07-30-Retracted articles and unethical behavior in economics journals?
19 0.13528392 2244 andrew gelman stats-2014-03-11-What if I were to stop publishing in journals?
20 0.13429266 1860 andrew gelman stats-2013-05-17-How can statisticians help psychologists do their research better?
topicId topicWeight
[(0, 0.283), (1, -0.083), (2, -0.085), (3, -0.157), (4, -0.11), (5, -0.059), (6, -0.001), (7, -0.057), (8, 0.014), (9, 0.01), (10, 0.028), (11, 0.026), (12, -0.028), (13, -0.012), (14, 0.017), (15, -0.036), (16, -0.005), (17, 0.038), (18, -0.001), (19, -0.012), (20, -0.026), (21, -0.05), (22, -0.003), (23, 0.001), (24, 0.013), (25, 0.013), (26, 0.076), (27, -0.022), (28, 0.012), (29, 0.013), (30, 0.01), (31, 0.005), (32, -0.01), (33, -0.007), (34, -0.047), (35, -0.007), (36, 0.003), (37, -0.015), (38, 0.011), (39, -0.015), (40, 0.012), (41, -0.065), (42, -0.031), (43, -0.011), (44, 0.007), (45, 0.033), (46, 0.006), (47, 0.046), (48, 0.026), (49, -0.01)]
simIndex simValue blogId blogTitle
same-blog 1 0.97307783 2006 andrew gelman stats-2013-09-03-Evaluating evidence from published research
Introduction: Following up on my entry the other day on post-publication peer review, Dan Kahan writes: You give me credit, I think, for merely participating in what I think is a systemic effect in the practice of empirical inquiry that conduces to quality control & hence the advance of knowledge by such means (likely the title conveys that!). I’d say: (a) by far the greatest weakness in the “publication regime” in social sciences today is the systematic disregard for basic principles of valid causal inference, a deficiency either in comprehension or craft that is at the root of scholars’ resort to (and journals’ tolerance for) invalid samples, the employment of designs that don’t generate observations more consistent with a hypothesis than with myriad rival ones, and the resort to deficient statistical modes of analysis that treat detection of “statististically significant difference” rather than “practical corroboration of practical meaningful effect” as the goal of such analysis (especial
Introduction: Responding to a proposal to move the journal Political Analysis from double-blind to single-blind reviewing (that is, authors would not know who is reviewing their papers but reviewers would know the authors’ names), Tom Palfrey writes: I agree with the editors’ recommendation. I have served on quite a few editorial boards of journals with different blinding policies, and have seen no evidence that double blind procedures are a useful way to improve the quality of articles published in a journal. Aside from the obvious administrative nuisance and the fact that authorship anonymity is a thing of the past in our discipline, the theoretical and empirical arguments in both directions lead to an ambiguous conclusion. Also keep in mind that the editors know the identity of the authors (they need to know for practical reasons), their identity is not hidden from authors, and ultimately it is they who make the accept/reject decision, and also lobby their friends and colleagues to submit “the
Introduction: Jeff Leek points to a post by Alex Holcombe, who disputes the idea that science is self-correcting. Holcombe writes [scroll down to get to his part]: The pace of scientific production has quickened, and self-correction has suffered. Findings that might correct old results are considered less interesting than results from more original research questions. Potential corrections are also more contested. As the competition for space in prestigious journals has become increasingly frenzied, doing and publishing studies that would confirm the rapidly accumulating new discoveries, or would correct them, became a losing proposition. Holcombe picks up on some points that we’ve discussed a lot here in the past year. Here’s Holcombe: In certain subfields, almost all new work appears in only a very few journals, all associated with a single professional society. There is then no way around the senior gatekeepers, who may then suppress corrections with impunity. . . . The bias agai
4 0.85051715 1878 andrew gelman stats-2013-05-31-How to fix the tabloids? Toward replicable social science research
Introduction: This seems to be the topic of the week. Yesterday I posted on the sister blog some further thoughts on those “Psychological Science” papers on menstrual cycles, biceps size, and political attitudes, tied to a horrible press release from the journal Psychological Science hyping the biceps and politics study. Then I was pointed to these suggestions from Richard Lucas and M. Brent Donnellan have on improving the replicability and reproducibility of research published in the Journal of Research in Personality: It goes without saying that editors of scientific journals strive to publish research that is not only theoretically interesting but also methodologically rigorous. The goal is to select papers that advance the field. Accordingly, editors want to publish findings that can be reproduced and replicated by other scientists. Unfortunately, there has been a recent “crisis in confidence” among psychologists about the quality of psychological research (Pashler & Wagenmakers, 2012)
5 0.84736127 1139 andrew gelman stats-2012-01-26-Suggested resolution of the Bem paradox
Introduction: There has been an increasing discussion about the proliferation of flawed research in psychology and medicine, with some landmark events being John Ioannides’s article , “Why most published research findings are false” (according to Google Scholar, cited 973 times since its appearance in 2005), the scandals of Marc Hauser and Diederik Stapel, two leading psychology professors who resigned after disclosures of scientific misconduct, and Daryl Bem’s dubious recent paper on ESP, published to much fanfare in Journal of Personality and Social Psychology, one of the top journals in the field. Alongside all this are the plagiarism scandals, which are uninteresting from a scientific context but are relevant in that, in many cases, neither the institutions housing the plagiarists nor the editors and publishers of the plagiarized material seem to care. Perhaps these universities and publishers are more worried about bad publicity (and maybe lawsuits, given that many of the plagiarism cas
6 0.84491771 2353 andrew gelman stats-2014-05-30-I posted this as a comment on a sociology blog
7 0.83536196 2361 andrew gelman stats-2014-06-06-Hurricanes vs. Himmicanes
8 0.83521622 1844 andrew gelman stats-2013-05-06-Against optimism about social science
9 0.83152002 2137 andrew gelman stats-2013-12-17-Replication backlash
10 0.83148903 1435 andrew gelman stats-2012-07-30-Retracted articles and unethical behavior in economics journals?
11 0.82206321 2179 andrew gelman stats-2014-01-20-The AAA Tranche of Subprime Science
13 0.81273723 2269 andrew gelman stats-2014-03-27-Beyond the Valley of the Trolls
14 0.80812639 2245 andrew gelman stats-2014-03-12-More on publishing in journals
16 0.79834175 2233 andrew gelman stats-2014-03-04-Literal vs. rhetorical
17 0.79806417 2177 andrew gelman stats-2014-01-19-“The British amateur who debunked the mathematics of happiness”
19 0.79427469 1860 andrew gelman stats-2013-05-17-How can statisticians help psychologists do their research better?
topicId topicWeight
[(2, 0.032), (15, 0.07), (16, 0.063), (21, 0.034), (24, 0.121), (30, 0.012), (43, 0.032), (45, 0.017), (53, 0.029), (55, 0.015), (63, 0.016), (66, 0.01), (69, 0.011), (80, 0.018), (82, 0.014), (86, 0.039), (99, 0.304)]
simIndex simValue blogId blogTitle
same-blog 1 0.98368126 2006 andrew gelman stats-2013-09-03-Evaluating evidence from published research
Introduction: Following up on my entry the other day on post-publication peer review, Dan Kahan writes: You give me credit, I think, for merely participating in what I think is a systemic effect in the practice of empirical inquiry that conduces to quality control & hence the advance of knowledge by such means (likely the title conveys that!). I’d say: (a) by far the greatest weakness in the “publication regime” in social sciences today is the systematic disregard for basic principles of valid causal inference, a deficiency either in comprehension or craft that is at the root of scholars’ resort to (and journals’ tolerance for) invalid samples, the employment of designs that don’t generate observations more consistent with a hypothesis than with myriad rival ones, and the resort to deficient statistical modes of analysis that treat detection of “statististically significant difference” rather than “practical corroboration of practical meaningful effect” as the goal of such analysis (especial
2 0.97833997 1435 andrew gelman stats-2012-07-30-Retracted articles and unethical behavior in economics journals?
Introduction: Stan Liebowitz writes: Have you ever heard of an article being retracted in economics? I know you have only been doing this for a few years but I suspect that the answer is that none or very few are retracted. No economist would ever deceive another. There is virtually no interest in detecting cheating. And what good would that do if there is no form of punishment? I say this because I think I have found a case in one of our top journals but the editor allowed the authors of the original article to write an anonymous referee report defending themselves and used this report to reject my comment even though an independent referee recommended publication. My reply: I wonder how this sort of thing will change in the future as journals become less important. My impression is that, on one side, researchers are increasingly citing NBER reports, Arxiv preprints, and the like; while, from the other direction, journals such as Science and Nature are developing the reputations of being “t
3 0.9776746 1860 andrew gelman stats-2013-05-17-How can statisticians help psychologists do their research better?
Introduction: I received two emails yesterday on related topics. First, Stephen Olivier pointed me to this post by Daniel Lakens, who wrote the following open call to statisticians: You would think that if you are passionate about statistics, then you want to help people to calculate them correctly in any way you can. . . . you’d think some statisticians would be interested in helping a poor mathematically challenged psychologist out by offering some practical advice. I’m the right person to ask this question, since I actually have written a lot of material that helps psychologists (and others) with their data analysis. But there clearly are communication difficulties, in that my work and that of other statisticians hasn’t reached Lakens. Sometimes the contributions of statisticians are made indirectly. For example, I wrote Bayesian Data Analysis, and then Kruschke wrote Doing Bayesian Data Analysis. Our statistics book made it possible for Kruschke to write his excellent book for psycholo
4 0.97600228 1848 andrew gelman stats-2013-05-09-A tale of two discussion papers
Introduction: Over the years I’ve written a dozen or so journal articles that have appeared with discussions, and I’ve participated in many published discussions of others’ articles as well. I get a lot out of these article-discussion-rejoinder packages, in all three of my roles as reader, writer, and discussant. Part 1: The story of an unsuccessful discussion The first time I had a discussion article was the result of an unfortunate circumstance. I had a research idea that resulted in an article with Don Rubin on monitoring the mixing of Markov chain simulations. I new the idea was great, but back then we worked pretty slowly so it was awhile before we had a final version to submit to a journal. (In retrospect I wish I’d just submitted the draft version as it was.) In the meantime I presented the paper at a conference. Our idea was very well received (I had a sheet of paper so people could write their names and addresses to get preprints, and we got either 50 or 150 (I can’t remembe
5 0.97501045 902 andrew gelman stats-2011-09-12-The importance of style in academic writing
Introduction: In my comments on academic cheating , I briefly discussed the question of how some of these papers could’ve been published in the first place, given that they tend to be of low quality. (It’s rare that people plagiarize the good stuff, and, when they do—for example when a senior scholar takes credit for a junior researcher’s contributions without giving proper credit—there’s not always a paper trail, and there can be legitimate differences of opinion about the relative contributions of the participants.) Anyway, to get back to the cases at hand: how did these rulebreakers get published in the first place? The question here is not how did they get away with cheating but how is it that top journals were publishing mediocre research? In the case of the profs who falsified data (Diederik Stapel) or did not follow scientific protocol (Mark Hauser), the answer is clear: By cheating, they were able to get the sort of too-good-to-be-true results which, if they were true, would be
6 0.97417283 2244 andrew gelman stats-2014-03-11-What if I were to stop publishing in journals?
7 0.97351962 384 andrew gelman stats-2010-10-31-Two stories about the election that I don’t believe
8 0.97329926 1395 andrew gelman stats-2012-06-27-Cross-validation (What is it good for?)
9 0.97275561 1273 andrew gelman stats-2012-04-20-Proposals for alternative review systems for scientific work
10 0.97248417 1139 andrew gelman stats-2012-01-26-Suggested resolution of the Bem paradox
11 0.97240251 1779 andrew gelman stats-2013-03-27-“Two Dogmas of Strong Objective Bayesianism”
14 0.97160053 1253 andrew gelman stats-2012-04-08-Technology speedup graph
17 0.97140205 2337 andrew gelman stats-2014-05-18-Never back down: The culture of poverty and the culture of journalism