andrew_gelman_stats andrew_gelman_stats-2013 andrew_gelman_stats-2013-1928 knowledge-graph by maker-knowledge-mining
Source: html
Introduction: We’ve had lots of lively discussions of fatally-flawed papers that have been published in top, top journals such as the American Economic Review or the Journal of Personality and Social Psychology or the American Sociological Review or the tabloids . And we also know about mistakes that make their way into mid-ranking outlets such as the Journal of Theoretical Biology. But what about results that appear in the lower tier of legitimate journals? I was thinking about this after reading a post by Dan Kahan slamming a paper that recently appeared in PLOS-One. I won’t discuss the paper itself here because that’s not my point. Rather, I had some thoughts regarding Kahan’s annoyance that a paper with fatal errors was published at all. I commented as follows: Read between the lines. The paper originally was released in 2009 and was published in 2013 in PLOS-One, which is one step above appearing on Arxiv. PLOS-One publishes some good things (so does Arxiv) but it’s the place
sentIndex sentText sentNum sentScore
1 We’ve had lots of lively discussions of fatally-flawed papers that have been published in top, top journals such as the American Economic Review or the Journal of Personality and Social Psychology or the American Sociological Review or the tabloids . [sent-1, score-1.306]
2 Rather, I had some thoughts regarding Kahan’s annoyance that a paper with fatal errors was published at all. [sent-6, score-0.689]
3 The paper originally was released in 2009 and was published in 2013 in PLOS-One, which is one step above appearing on Arxiv. [sent-8, score-0.517]
4 We can deduce that the paper was rejected by Science, Nature, various other biology journals, and maybe some political science journals as well. [sent-10, score-0.955]
5 I’m not saying you shouldn’t criticize the paper in question, but you can’t really demand better from a paper published in a bottom-feeder journal. [sent-11, score-0.637]
6 Again, just because something’s in a crap journal, doesn’t mean it’s crap; I’ve published lots of papers in unselective, low-prestige outlets. [sent-12, score-0.79]
7 But it’s certainly no surprise if a paper published in a low-grade journal happens to be crap. [sent-13, score-0.724]
8 I think it’s safe to assume that the paper being discussed above was rejected by the tabloids before appearing in PLOS-One. [sent-20, score-0.742]
9 That’s not a surprise: People submit a lot of weak papers to the tabloids and most of them get rejected. [sent-21, score-0.672]
10 When a weak paper appears in a tabloid, lots of people (including me) get annoyed, and we say things like, How could Science (or Nature or Psych Science) publish that crap! [sent-22, score-0.751]
11 Even non-tabloid serious journals publish seriously flawed papers; in recent months we’ve discussed examples from the American Economic Review and the American Sociological Review. [sent-23, score-0.561]
12 Again, in these cases we can get annoyed that major errors slipped past the review process. [sent-24, score-0.379]
13 But when a weak paper appears in PLOS-One, we cannot be shocked. [sent-25, score-0.529]
14 PLOS-One by design will regularly publish weak papers along with the good stuff. [sent-26, score-0.596]
15 Many commenters remarked on the high quality of much of what appears in PLOS-One and urged us all not to restrict our reading to the so-called top journals. [sent-27, score-0.503]
16 Many, maybe most, of my most influential papers over the years have not appear in top journals, but they’re still good papers and represent important research! [sent-29, score-0.791]
17 When we encounter crap in leading journals, we know how to react—with annoyance. [sent-31, score-0.404]
18 To the extent we go beyond the top journals, we will still find a lot of crap even when not looking for it (with an extreme case being Dan Kahan’s encounter with the paper described above), and we need a way of thinking about it. [sent-32, score-0.795]
19 To say it again: the point of this post is not to bash PLOS-One or the Journal of Theoretical Biology or other low-ranked but legitimate journals, it’s to explore how we should think about the papers that appear in such places. [sent-34, score-0.55]
20 The traditional approach is to assume that everything published in a peer-reviewed journal is correct, but that’s not right. [sent-35, score-0.338]
wordName wordTfidf (topN-words)
[('journals', 0.35), ('crap', 0.28), ('papers', 0.262), ('paper', 0.232), ('tabloids', 0.223), ('weak', 0.187), ('published', 0.173), ('kahan', 0.168), ('journal', 0.165), ('top', 0.159), ('publish', 0.147), ('annoyance', 0.145), ('review', 0.127), ('encounter', 0.124), ('american', 0.124), ('appearing', 0.112), ('sociological', 0.112), ('rejected', 0.111), ('appears', 0.11), ('appear', 0.108), ('annoyed', 0.106), ('legitimate', 0.103), ('biology', 0.098), ('dan', 0.091), ('surprise', 0.089), ('urged', 0.086), ('unselective', 0.086), ('reading', 0.083), ('science', 0.083), ('abstract', 0.082), ('tier', 0.081), ('deduce', 0.081), ('bash', 0.077), ('slipped', 0.077), ('lots', 0.075), ('nature', 0.074), ('theoretical', 0.073), ('appropriate', 0.072), ('fatal', 0.07), ('tabloid', 0.07), ('errors', 0.069), ('react', 0.067), ('psych', 0.067), ('outlets', 0.066), ('remarked', 0.065), ('certainly', 0.065), ('lively', 0.064), ('discussed', 0.064), ('publishes', 0.063), ('economic', 0.063)]
simIndex simValue blogId blogTitle
same-blog 1 1.0000002 1928 andrew gelman stats-2013-07-06-How to think about papers published in low-grade journals?
Introduction: We’ve had lots of lively discussions of fatally-flawed papers that have been published in top, top journals such as the American Economic Review or the Journal of Personality and Social Psychology or the American Sociological Review or the tabloids . And we also know about mistakes that make their way into mid-ranking outlets such as the Journal of Theoretical Biology. But what about results that appear in the lower tier of legitimate journals? I was thinking about this after reading a post by Dan Kahan slamming a paper that recently appeared in PLOS-One. I won’t discuss the paper itself here because that’s not my point. Rather, I had some thoughts regarding Kahan’s annoyance that a paper with fatal errors was published at all. I commented as follows: Read between the lines. The paper originally was released in 2009 and was published in 2013 in PLOS-One, which is one step above appearing on Arxiv. PLOS-One publishes some good things (so does Arxiv) but it’s the place
2 0.35351253 2245 andrew gelman stats-2014-03-12-More on publishing in journals
Introduction: I’m postponing today’s scheduled post (“Empirical implications of Empirical Implications of Theoretical Models”) to continue the lively discussion from yesterday, What if I were to stop publishing in journals? . An example: my papers with Basbøll Thomas Basbøll and I got into a long discussion on our blogs about business school professor Karl Weick and other cases of plagiarism copying text without attribution. We felt it useful to take our ideas to the next level and write them up as a manuscript, which ended up being logical to split into two papers. At that point I put some effort into getting these papers published, which I eventually did: To throw away data: Plagiarism as a statistical crime went into American Scientist and When do stories work? Evidence and illustration in the social sciences will appear in Sociological Methods and Research. The second paper, in particular, took some effort to place; I got some advice from colleagues in sociology as to where
Introduction: The other day we discussed that paper on ovulation and voting (you may recall that the authors reported a scattered bunch of comparisons, significance tests, and p-values, and I recommended that they would’ve done better to simply report complete summaries of their data, so that readers could see the comparisons of interest in full context), and I was thinking a bit more about why I was so bothered that it was published in Psychological Science, which I’d thought of as a serious research journal. My concern isn’t just that that the paper is bad—after all, lots of bad papers get published—but rather that it had nothing really going for it, except that it was headline bait. It was a survey done on Mechanical Turk, that’s it. No clever design, no clever questions, no care in dealing with nonresponse problems, no innovative data analysis, no nothing. The paper had nothing to offer, except that it had no obvious flaws. Psychology is a huge field full of brilliant researchers.
4 0.30692983 371 andrew gelman stats-2010-10-26-Musical chairs in econ journals
Introduction: Tyler Cowen links to a paper by Bruno Frey on the lack of space for articles in economics journals. Frey writes: To further their careers, [academic economists] are required to publish in A-journals, but for the vast majority this is impossible because there are few slots open in such journals. Such academic competition maybe useful to generate hard work, however, there may be serious negative consequences: the wrong output may be produced in an inefficient way, the wrong people may be selected, and losers may react in a harmful way. According to Frey, the consensus is that there are only five top economics journals–and one of those five is Econometrica, which is so specialized that I’d say that, for most academic economists, there are only four top places they can publish. The difficulty is that demand for these slots outpaces supply: for example, in 2007 there were only 275 articles in all these journals combined (or 224 if you exclude Econometrica), while “a rough estim
5 0.29611352 1393 andrew gelman stats-2012-06-26-The reverse-journal-submission system
Introduction: I’ve whined before in this space that some of my most important, innovative, and influential papers are really hard to get published. I’ll go through endless hassle with a journal or sometimes several journals until I find some place willing to publish. It’s just irritating. I was thinking about this recently because a colleague and I just finished a paper that I love love love. But I can’t figure out where to submit it. This is a paper for which I would prefer the so-called reverse-journal-submission approach. Instead of sending the paper to journal after journal after journal, waiting years until an acceptance (recall that, unless you’re Bruno Frey, you’re not allowed to submit the same paper to multiple journals simultaneously), you post the paper on a public site, and then journals compete to see who gets to publish it. I think that system would work well with a paper like this which is offbeat but has a nontrivial chance of becoming highly influential. P.S. Just to clar
6 0.27541614 1878 andrew gelman stats-2013-05-31-How to fix the tabloids? Toward replicable social science research
7 0.27047625 2004 andrew gelman stats-2013-09-01-Post-publication peer review: How it (sometimes) really works
8 0.25201169 1435 andrew gelman stats-2012-07-30-Retracted articles and unethical behavior in economics journals?
9 0.23886032 834 andrew gelman stats-2011-08-01-I owe it all to the haters
10 0.21568319 2244 andrew gelman stats-2014-03-11-What if I were to stop publishing in journals?
11 0.20798424 838 andrew gelman stats-2011-08-04-Retraction Watch
12 0.2056751 1272 andrew gelman stats-2012-04-20-More proposals to reform the peer-review system
13 0.20402528 902 andrew gelman stats-2011-09-12-The importance of style in academic writing
14 0.20251326 675 andrew gelman stats-2011-04-22-Arrow’s other theorem
16 0.19859788 1139 andrew gelman stats-2012-01-26-Suggested resolution of the Bem paradox
17 0.19310759 2353 andrew gelman stats-2014-05-30-I posted this as a comment on a sociology blog
18 0.18821217 2006 andrew gelman stats-2013-09-03-Evaluating evidence from published research
19 0.17232054 2269 andrew gelman stats-2014-03-27-Beyond the Valley of the Trolls
20 0.16935284 2233 andrew gelman stats-2014-03-04-Literal vs. rhetorical
topicId topicWeight
[(0, 0.222), (1, -0.107), (2, -0.122), (3, -0.189), (4, -0.142), (5, -0.114), (6, 0.021), (7, -0.203), (8, -0.046), (9, 0.01), (10, 0.245), (11, 0.041), (12, -0.124), (13, 0.058), (14, 0.045), (15, -0.097), (16, 0.012), (17, 0.07), (18, -0.057), (19, -0.044), (20, 0.033), (21, 0.02), (22, 0.019), (23, -0.01), (24, -0.009), (25, 0.004), (26, -0.055), (27, 0.012), (28, 0.015), (29, 0.007), (30, -0.032), (31, -0.052), (32, 0.004), (33, -0.05), (34, -0.086), (35, -0.032), (36, 0.033), (37, 0.062), (38, -0.036), (39, 0.024), (40, 0.007), (41, -0.011), (42, -0.03), (43, -0.006), (44, -0.048), (45, 0.028), (46, 0.049), (47, 0.025), (48, -0.014), (49, 0.022)]
simIndex simValue blogId blogTitle
same-blog 1 0.99098849 1928 andrew gelman stats-2013-07-06-How to think about papers published in low-grade journals?
Introduction: We’ve had lots of lively discussions of fatally-flawed papers that have been published in top, top journals such as the American Economic Review or the Journal of Personality and Social Psychology or the American Sociological Review or the tabloids . And we also know about mistakes that make their way into mid-ranking outlets such as the Journal of Theoretical Biology. But what about results that appear in the lower tier of legitimate journals? I was thinking about this after reading a post by Dan Kahan slamming a paper that recently appeared in PLOS-One. I won’t discuss the paper itself here because that’s not my point. Rather, I had some thoughts regarding Kahan’s annoyance that a paper with fatal errors was published at all. I commented as follows: Read between the lines. The paper originally was released in 2009 and was published in 2013 in PLOS-One, which is one step above appearing on Arxiv. PLOS-One publishes some good things (so does Arxiv) but it’s the place
2 0.93355322 1321 andrew gelman stats-2012-05-15-A statistical research project: Weeding out the fraudulent citations
Introduction: John Mashey points me to a blog post by Phil Davis on “the emergence of a citation cartel.” Davis tells the story: Cell Transplantation is a medical journal published by the Cognizant Communication Corporation of Putnam Valley, New York. In recent years, its impact factor has been growing rapidly. In 2006, it was 3.482 [I think he means "3.5"---ed.]. In 2010, it had almost doubled to 6.204. When you look at which journals cite Cell Transplantation, two journals stand out noticeably: the Medical Science Monitor, and The Scientific World Journal. According to the JCR, neither of these journals cited Cell Transplantation until 2010. Then, in 2010, a review article was published in the Medical Science Monitor citing 490 articles, 445 of which were to papers published in Cell Transplantation. All 445 citations pointed to papers published in 2008 or 2009 — the citation window from which the journal’s 2010 impact factor was derived. Of the remaining 45 citations, 44 cited the Me
Introduction: The other day we discussed that paper on ovulation and voting (you may recall that the authors reported a scattered bunch of comparisons, significance tests, and p-values, and I recommended that they would’ve done better to simply report complete summaries of their data, so that readers could see the comparisons of interest in full context), and I was thinking a bit more about why I was so bothered that it was published in Psychological Science, which I’d thought of as a serious research journal. My concern isn’t just that that the paper is bad—after all, lots of bad papers get published—but rather that it had nothing really going for it, except that it was headline bait. It was a survey done on Mechanical Turk, that’s it. No clever design, no clever questions, no care in dealing with nonresponse problems, no innovative data analysis, no nothing. The paper had nothing to offer, except that it had no obvious flaws. Psychology is a huge field full of brilliant researchers.
4 0.91443777 1393 andrew gelman stats-2012-06-26-The reverse-journal-submission system
Introduction: I’ve whined before in this space that some of my most important, innovative, and influential papers are really hard to get published. I’ll go through endless hassle with a journal or sometimes several journals until I find some place willing to publish. It’s just irritating. I was thinking about this recently because a colleague and I just finished a paper that I love love love. But I can’t figure out where to submit it. This is a paper for which I would prefer the so-called reverse-journal-submission approach. Instead of sending the paper to journal after journal after journal, waiting years until an acceptance (recall that, unless you’re Bruno Frey, you’re not allowed to submit the same paper to multiple journals simultaneously), you post the paper on a public site, and then journals compete to see who gets to publish it. I think that system would work well with a paper like this which is offbeat but has a nontrivial chance of becoming highly influential. P.S. Just to clar
5 0.90673858 834 andrew gelman stats-2011-08-01-I owe it all to the haters
Introduction: Sometimes when I submit an article to a journal it is accepted right away or with minor alterations. But many of my favorite articles were rejected or had to go through an exhausting series of revisions. For example, this influential article had a very hostile referee and we had to seriously push the journal editor to accept it. This one was rejected by one or two journals before finally appearing with discussion. This paper was rejected by the American Political Science Review with no chance of revision and we had to publish it in the British Journal of Political Science, which was a bit odd given that the article was 100% about American politics. And when I submitted this instant classic (actually at the invitation of the editor), the referees found it to be trivial, and the editor did me the favor of publishing it but only by officially labeling it as a discussion of another article that appeared in the same issue. Some of my most influential papers were accepted right
6 0.90294582 1122 andrew gelman stats-2012-01-16-“Groundbreaking or Definitive? Journals Need to Pick One”
7 0.87608957 883 andrew gelman stats-2011-09-01-Arrow’s theorem update
8 0.85754085 371 andrew gelman stats-2010-10-26-Musical chairs in econ journals
9 0.85618865 1118 andrew gelman stats-2012-01-14-A model rejection letter
10 0.85050392 1137 andrew gelman stats-2012-01-24-Difficulties in publishing non-replications of implausible findings
11 0.84676349 2233 andrew gelman stats-2014-03-04-Literal vs. rhetorical
12 0.84376854 2353 andrew gelman stats-2014-05-30-I posted this as a comment on a sociology blog
13 0.82791263 762 andrew gelman stats-2011-06-13-How should journals handle replication studies?
14 0.82313424 1429 andrew gelman stats-2012-07-26-Our broken scholarly publishing system
15 0.8204115 2004 andrew gelman stats-2013-09-01-Post-publication peer review: How it (sometimes) really works
17 0.81358606 1435 andrew gelman stats-2012-07-30-Retracted articles and unethical behavior in economics journals?
18 0.80305094 675 andrew gelman stats-2011-04-22-Arrow’s other theorem
19 0.80260366 1139 andrew gelman stats-2012-01-26-Suggested resolution of the Bem paradox
20 0.79676992 2245 andrew gelman stats-2014-03-12-More on publishing in journals
topicId topicWeight
[(15, 0.03), (16, 0.495), (24, 0.059), (86, 0.02), (99, 0.287)]
simIndex simValue blogId blogTitle
1 0.99343908 1115 andrew gelman stats-2012-01-12-Where are the larger-than-life athletes?
Introduction: Jonathan Cantor points to this poll estimating rifle-armed QB Tim Tebow as America’s favorite pro athlete: In an ESPN survey of 1,502 Americans age 12 or older, three percent identified Tebow as their favorite professional athlete. Tebow finished in front of Kobe Bryant (2 percent), Aaron Rodgers (1.9 percent), Peyton Manning (1.8 percent), and Tom Brady (1.5 percent). Amusing. What this survey says to me is that there are no super-popular athletes who are active in America today. Which actually sounds about right. No Tiger Woods, no Magic Johnson, Muhammed Ali, John Elway, Pete Rose, Billie Jean King, etc etc. Tebow is an amusing choice, people might as well pick him now while he’s still on top. As a sports celeb, he’s like Bill Lee or the Refrigerator: colorful and a solid pro athlete, but no superstar. When you think about all the colorful superstar athletes of times gone by, it’s perhaps surprising that there’s nobody out there right now to play the role. I supp
2 0.98991561 528 andrew gelman stats-2011-01-21-Elevator shame is a two-way street
Introduction: Tyler Cowen links a blog by Samuel Arbesman mocking people who are so lazy that they take the elevator from 1 to 2. This reminds me of my own annoyance about a guy who worked in my building and did not take the elevator. (For the full story, go here and search on “elevator.”)
3 0.9884721 1304 andrew gelman stats-2012-05-06-Picking on Stephen Wolfram
Introduction: Shalizi . But this one is still my favorite.
4 0.98818326 1014 andrew gelman stats-2011-11-16-Visualizations of NYPD stop-and-frisk data
Introduction: Cathy O’Neil organized this visualization project with NYPD stop-and-frisk data. It’s part of the Data Without Borders project. Unfortunately, because of legal restrictions I couldn’t send them the data Jeff, Alex, and I used in our project several years ago.
5 0.98725945 1659 andrew gelman stats-2013-01-07-Some silly things you (didn’t) miss by not reading the sister blog
Introduction: 1. I have the least stressful job in America (duh) 2. B-school prof in a parody of short-term thinking 3. The academic clock 4. I guessed wrong 5. 2012 Conceptual Development Lab Newsletter
6 0.9858858 1180 andrew gelman stats-2012-02-22-I’m officially no longer a “rogue”
7 0.98485982 572 andrew gelman stats-2011-02-14-Desecration of valuable real estate
9 0.97821283 1279 andrew gelman stats-2012-04-24-ESPN is looking to hire a research analyst
10 0.97762811 1487 andrew gelman stats-2012-09-08-Animated drought maps
11 0.97192061 1330 andrew gelman stats-2012-05-19-Cross-validation to check missing-data imputation
12 0.96503234 445 andrew gelman stats-2010-12-03-Getting a job in pro sports… as a statistician
13 0.96286905 1025 andrew gelman stats-2011-11-24-Always check your evidence
14 0.96269023 1598 andrew gelman stats-2012-11-30-A graphics talk with no visuals!
15 0.95047367 700 andrew gelman stats-2011-05-06-Suspicious pattern of too-strong replications of medical research
16 0.94073653 1156 andrew gelman stats-2012-02-06-Bayesian model-building by pure thought: Some principles and examples
17 0.93595576 398 andrew gelman stats-2010-11-06-Quote of the day
18 0.93299484 1168 andrew gelman stats-2012-02-14-The tabloids strike again
same-blog 20 0.92483425 1928 andrew gelman stats-2013-07-06-How to think about papers published in low-grade journals?