andrew_gelman_stats andrew_gelman_stats-2012 andrew_gelman_stats-2012-1435 knowledge-graph by maker-knowledge-mining
Source: html
Introduction: Stan Liebowitz writes: Have you ever heard of an article being retracted in economics? I know you have only been doing this for a few years but I suspect that the answer is that none or very few are retracted. No economist would ever deceive another. There is virtually no interest in detecting cheating. And what good would that do if there is no form of punishment? I say this because I think I have found a case in one of our top journals but the editor allowed the authors of the original article to write an anonymous referee report defending themselves and used this report to reject my comment even though an independent referee recommended publication. My reply: I wonder how this sort of thing will change in the future as journals become less important. My impression is that, on one side, researchers are increasingly citing NBER reports, Arxiv preprints, and the like; while, from the other direction, journals such as Science and Nature are developing the reputations of being “t
sentIndex sentText sentNum sentScore
1 I say this because I think I have found a case in one of our top journals but the editor allowed the authors of the original article to write an anonymous referee report defending themselves and used this report to reject my comment even though an independent referee recommended publication. [sent-6, score-0.764]
2 My reply: I wonder how this sort of thing will change in the future as journals become less important. [sent-7, score-0.225]
3 For example, on his blog, journal editor Steven Levitt wrote, “Is it surprising that scientists would try to keep work that disagrees with their findings out of journals? [sent-13, score-0.271]
4 The first is the paper by Emily Oster , a young economist who engaged in a several-years-long battle with public-health researchers regarding a theory of sex-ratio changes in developing countries. [sent-21, score-0.429]
5 She eventually gave up and admitted that the public-heatlh researchers were right and she was wrong. [sent-22, score-0.193]
6 This was a bit of a scandal; he was reprimanded by the editor of a journal that had published one of the redundant papers. [sent-25, score-0.444]
7 Many years ago I had a colleague who showed me a manuscript of a paper he had submitted to a top journal in his applied field. [sent-28, score-0.319]
8 I was also more peripherally involved in a project with experimental data where the researchers had a before-after comparison with the treatment group and a before-after comparison with the control group. [sent-39, score-0.283]
9 There was a large and statistically-significant improvement in the treatment group—and also a large and stat sig improvement for the controls. [sent-40, score-0.325]
10 My impression is that the biggest concerns regarding retractions right now involve three issues: 1. [sent-46, score-0.201]
11 Here I’m thinking about professional researchers who don’t really do research, but they manage to publish their papers somewhere or another, and those journals and conferences which have nothing to do with science but exist only to make money off of publication fees etc. [sent-51, score-0.49]
12 You might feel you could ignore these because they’re not prestigious journals, but can an outsider really distinguish between the (legit) Journal of Money and Banking and some fake journal out there? [sent-52, score-0.238]
13 Also we hear about researchers in other countries who spend their time swamping legitimate journals with papers that are made up, plagiarized, etc. [sent-53, score-0.49]
14 Errors arising in serious research due to selection bias, confirmation bias, etc. [sent-55, score-0.328]
15 Psychology researchers are particularly worried about this one: the idea is that if you look hard enough you can find confirmation of just about anything. [sent-56, score-0.292]
16 That sounds pretty impressive (and it impressed the journal editors enough that they published it), but it turns out that, yes, if you’re looking for an effect, you can indeed find it, wrongly, in 9 different ways. [sent-58, score-0.261]
17 I don’t consider self plagiarism, or even pure plagiarism as serious a “crime” as making stuff up. [sent-61, score-0.41]
18 I have trouble believing he will show up, although I am always surprised at the chutzpa that some people have to be able ignore problems that would make me ashamed to show my face. [sent-66, score-0.221]
19 One of the two authors that I criticize about impropriety in piracy research was Frey’s student and one of the self-plagiarized papers was coauthored by the two of them, although the fellow I believe to have made stuff up was too junior to be given credit for the deceit with Frey. [sent-67, score-0.313]
20 (In many cases, though, plagiarism does involve fraud, for example a secondhand story presented as factual, or a plagiarized passage that is so badly garbled that it loses its original meaning. [sent-70, score-0.494]
wordName wordTfidf (topN-words)
[('journals', 0.225), ('researchers', 0.193), ('plagiarism', 0.167), ('journal', 0.159), ('frey', 0.15), ('liebowitz', 0.123), ('retractions', 0.119), ('editor', 0.112), ('published', 0.102), ('referee', 0.101), ('plagiarized', 0.101), ('confirmation', 0.099), ('esp', 0.094), ('treatment', 0.09), ('paper', 0.088), ('econ', 0.084), ('research', 0.084), ('improvement', 0.082), ('stuff', 0.082), ('involve', 0.082), ('serious', 0.081), ('pure', 0.08), ('ignore', 0.079), ('developing', 0.077), ('original', 0.077), ('although', 0.075), ('report', 0.074), ('papers', 0.072), ('colleague', 0.072), ('reprimanded', 0.071), ('sig', 0.071), ('brazen', 0.071), ('deceive', 0.071), ('gallo', 0.071), ('economist', 0.071), ('publishing', 0.069), ('fit', 0.068), ('secondhand', 0.067), ('ashamed', 0.067), ('nber', 0.067), ('preprints', 0.067), ('renewed', 0.067), ('selection', 0.064), ('suspect', 0.064), ('wrongly', 0.064), ('hiv', 0.064), ('legit', 0.064), ('withdraw', 0.064), ('fulfilling', 0.064), ('chutzpah', 0.064)]
simIndex simValue blogId blogTitle
same-blog 1 0.99999964 1435 andrew gelman stats-2012-07-30-Retracted articles and unethical behavior in economics journals?
Introduction: Stan Liebowitz writes: Have you ever heard of an article being retracted in economics? I know you have only been doing this for a few years but I suspect that the answer is that none or very few are retracted. No economist would ever deceive another. There is virtually no interest in detecting cheating. And what good would that do if there is no form of punishment? I say this because I think I have found a case in one of our top journals but the editor allowed the authors of the original article to write an anonymous referee report defending themselves and used this report to reject my comment even though an independent referee recommended publication. My reply: I wonder how this sort of thing will change in the future as journals become less important. My impression is that, on one side, researchers are increasingly citing NBER reports, Arxiv preprints, and the like; while, from the other direction, journals such as Science and Nature are developing the reputations of being “t
2 0.28463194 371 andrew gelman stats-2010-10-26-Musical chairs in econ journals
Introduction: Tyler Cowen links to a paper by Bruno Frey on the lack of space for articles in economics journals. Frey writes: To further their careers, [academic economists] are required to publish in A-journals, but for the vast majority this is impossible because there are few slots open in such journals. Such academic competition maybe useful to generate hard work, however, there may be serious negative consequences: the wrong output may be produced in an inefficient way, the wrong people may be selected, and losers may react in a harmful way. According to Frey, the consensus is that there are only five top economics journals–and one of those five is Econometrica, which is so specialized that I’d say that, for most academic economists, there are only four top places they can publish. The difficulty is that demand for these slots outpaces supply: for example, in 2007 there were only 275 articles in all these journals combined (or 224 if you exclude Econometrica), while “a rough estim
Introduction: As regular readers of this blog are aware, I am fascinated by academic and scientific cheating and the excuses people give for it. Bruno Frey and colleagues published a single article (with only minor variants) in five different major journals, and these articles did not cite each other. And there have been several other cases of his self-plagiarism (see this review from Olaf Storbeck). I do not mind the general practice of repeating oneself for different audiences—in the social sciences, we call this Arrow’s Theorem —but in this case Frey seems to have gone a bit too far. Blogger Economic Logic has looked into this and concluded that this sort of common practice is standard in “the context of the German(-speaking) academic environment,” and what sets Frey apart is not his self-plagiarism or even his brazenness but rather his practice of doing it in high-visibility journals. Economic Logic writes that “[Frey's] contribution is pedagogical, he found a good and interesting
4 0.26350254 2245 andrew gelman stats-2014-03-12-More on publishing in journals
Introduction: I’m postponing today’s scheduled post (“Empirical implications of Empirical Implications of Theoretical Models”) to continue the lively discussion from yesterday, What if I were to stop publishing in journals? . An example: my papers with Basbøll Thomas Basbøll and I got into a long discussion on our blogs about business school professor Karl Weick and other cases of plagiarism copying text without attribution. We felt it useful to take our ideas to the next level and write them up as a manuscript, which ended up being logical to split into two papers. At that point I put some effort into getting these papers published, which I eventually did: To throw away data: Plagiarism as a statistical crime went into American Scientist and When do stories work? Evidence and illustration in the social sciences will appear in Sociological Methods and Research. The second paper, in particular, took some effort to place; I got some advice from colleagues in sociology as to where
5 0.25201169 1928 andrew gelman stats-2013-07-06-How to think about papers published in low-grade journals?
Introduction: We’ve had lots of lively discussions of fatally-flawed papers that have been published in top, top journals such as the American Economic Review or the Journal of Personality and Social Psychology or the American Sociological Review or the tabloids . And we also know about mistakes that make their way into mid-ranking outlets such as the Journal of Theoretical Biology. But what about results that appear in the lower tier of legitimate journals? I was thinking about this after reading a post by Dan Kahan slamming a paper that recently appeared in PLOS-One. I won’t discuss the paper itself here because that’s not my point. Rather, I had some thoughts regarding Kahan’s annoyance that a paper with fatal errors was published at all. I commented as follows: Read between the lines. The paper originally was released in 2009 and was published in 2013 in PLOS-One, which is one step above appearing on Arxiv. PLOS-One publishes some good things (so does Arxiv) but it’s the place
6 0.24522342 675 andrew gelman stats-2011-04-22-Arrow’s other theorem
8 0.24493517 902 andrew gelman stats-2011-09-12-The importance of style in academic writing
9 0.24089664 1139 andrew gelman stats-2012-01-26-Suggested resolution of the Bem paradox
10 0.22421056 883 andrew gelman stats-2011-09-01-Arrow’s theorem update
11 0.2202372 1878 andrew gelman stats-2013-05-31-How to fix the tabloids? Toward replicable social science research
12 0.21428195 1393 andrew gelman stats-2012-06-26-The reverse-journal-submission system
13 0.21247648 1588 andrew gelman stats-2012-11-23-No one knows what it’s like to be the bad man
16 0.19001456 2353 andrew gelman stats-2014-05-30-I posted this as a comment on a sociology blog
17 0.18462224 838 andrew gelman stats-2011-08-04-Retraction Watch
18 0.18084936 2244 andrew gelman stats-2014-03-11-What if I were to stop publishing in journals?
19 0.17394491 834 andrew gelman stats-2011-08-01-I owe it all to the haters
20 0.16824757 728 andrew gelman stats-2011-05-24-A (not quite) grand unified theory of plagiarism, as applied to the Wegman case
topicId topicWeight
[(0, 0.335), (1, -0.119), (2, -0.108), (3, -0.197), (4, -0.109), (5, -0.095), (6, 0.051), (7, -0.172), (8, -0.03), (9, 0.038), (10, 0.122), (11, -0.004), (12, -0.11), (13, 0.012), (14, -0.031), (15, -0.101), (16, 0.045), (17, 0.048), (18, 0.012), (19, -0.004), (20, -0.011), (21, 0.01), (22, 0.032), (23, -0.012), (24, 0.012), (25, -0.059), (26, -0.053), (27, -0.033), (28, -0.029), (29, 0.016), (30, 0.037), (31, 0.003), (32, 0.005), (33, 0.014), (34, -0.026), (35, 0.04), (36, -0.005), (37, 0.002), (38, -0.026), (39, 0.056), (40, -0.031), (41, 0.004), (42, -0.012), (43, -0.024), (44, 0.019), (45, 0.016), (46, 0.091), (47, -0.001), (48, 0.008), (49, 0.022)]
simIndex simValue blogId blogTitle
same-blog 1 0.97590637 1435 andrew gelman stats-2012-07-30-Retracted articles and unethical behavior in economics journals?
Introduction: Stan Liebowitz writes: Have you ever heard of an article being retracted in economics? I know you have only been doing this for a few years but I suspect that the answer is that none or very few are retracted. No economist would ever deceive another. There is virtually no interest in detecting cheating. And what good would that do if there is no form of punishment? I say this because I think I have found a case in one of our top journals but the editor allowed the authors of the original article to write an anonymous referee report defending themselves and used this report to reject my comment even though an independent referee recommended publication. My reply: I wonder how this sort of thing will change in the future as journals become less important. My impression is that, on one side, researchers are increasingly citing NBER reports, Arxiv preprints, and the like; while, from the other direction, journals such as Science and Nature are developing the reputations of being “t
2 0.92626542 883 andrew gelman stats-2011-09-01-Arrow’s theorem update
Introduction: Someone pointed me to this letter to Bruno Frey from the editor of the Journal of Economic Perspectives. ( Background here , also more here from Olaf Storbeck.) The journal editor was upset about Frey’s self-plagiarism, and Frey responded with an apology: It was a grave mistake on our part for which we deeply apologize. It should never have happened. This is deplorable. . . . Please be assured that we take all precautions and measures that this unfortunate event does not happen again, with any journal. What I wonder is: How “deplorable” does Frey really think this is? You don’t publish a paper in 5 different places by accident! Is Frey saying that he knew this was deplorable back then and he did it anyway, based on calculation balancing the gains from multiple publications vs. the potential losses if he got caught? Or is he saying that the conduct is deplorable, but he didn’t realize it was deplorable when he did it? My guess is that Frey does not actually think the r
3 0.92083251 902 andrew gelman stats-2011-09-12-The importance of style in academic writing
Introduction: In my comments on academic cheating , I briefly discussed the question of how some of these papers could’ve been published in the first place, given that they tend to be of low quality. (It’s rare that people plagiarize the good stuff, and, when they do—for example when a senior scholar takes credit for a junior researcher’s contributions without giving proper credit—there’s not always a paper trail, and there can be legitimate differences of opinion about the relative contributions of the participants.) Anyway, to get back to the cases at hand: how did these rulebreakers get published in the first place? The question here is not how did they get away with cheating but how is it that top journals were publishing mediocre research? In the case of the profs who falsified data (Diederik Stapel) or did not follow scientific protocol (Mark Hauser), the answer is clear: By cheating, they were able to get the sort of too-good-to-be-true results which, if they were true, would be
4 0.9074738 1139 andrew gelman stats-2012-01-26-Suggested resolution of the Bem paradox
Introduction: There has been an increasing discussion about the proliferation of flawed research in psychology and medicine, with some landmark events being John Ioannides’s article , “Why most published research findings are false” (according to Google Scholar, cited 973 times since its appearance in 2005), the scandals of Marc Hauser and Diederik Stapel, two leading psychology professors who resigned after disclosures of scientific misconduct, and Daryl Bem’s dubious recent paper on ESP, published to much fanfare in Journal of Personality and Social Psychology, one of the top journals in the field. Alongside all this are the plagiarism scandals, which are uninteresting from a scientific context but are relevant in that, in many cases, neither the institutions housing the plagiarists nor the editors and publishers of the plagiarized material seem to care. Perhaps these universities and publishers are more worried about bad publicity (and maybe lawsuits, given that many of the plagiarism cas
Introduction: The other day we discussed that paper on ovulation and voting (you may recall that the authors reported a scattered bunch of comparisons, significance tests, and p-values, and I recommended that they would’ve done better to simply report complete summaries of their data, so that readers could see the comparisons of interest in full context), and I was thinking a bit more about why I was so bothered that it was published in Psychological Science, which I’d thought of as a serious research journal. My concern isn’t just that that the paper is bad—after all, lots of bad papers get published—but rather that it had nothing really going for it, except that it was headline bait. It was a survey done on Mechanical Turk, that’s it. No clever design, no clever questions, no care in dealing with nonresponse problems, no innovative data analysis, no nothing. The paper had nothing to offer, except that it had no obvious flaws. Psychology is a huge field full of brilliant researchers.
6 0.89760852 675 andrew gelman stats-2011-04-22-Arrow’s other theorem
7 0.88873595 2353 andrew gelman stats-2014-05-30-I posted this as a comment on a sociology blog
8 0.88192201 371 andrew gelman stats-2010-10-26-Musical chairs in econ journals
9 0.87989932 834 andrew gelman stats-2011-08-01-I owe it all to the haters
11 0.86771113 1928 andrew gelman stats-2013-07-06-How to think about papers published in low-grade journals?
13 0.86041141 1321 andrew gelman stats-2012-05-15-A statistical research project: Weeding out the fraudulent citations
14 0.85707009 2233 andrew gelman stats-2014-03-04-Literal vs. rhetorical
16 0.84230345 901 andrew gelman stats-2011-09-12-Some thoughts on academic cheating, inspired by Frey, Wegman, Fischer, Hauser, Stapel
17 0.841851 1137 andrew gelman stats-2012-01-24-Difficulties in publishing non-replications of implausible findings
18 0.84022659 1585 andrew gelman stats-2012-11-20-“I know you aren’t the plagiarism police, but . . .”
19 0.83967841 2177 andrew gelman stats-2014-01-19-“The British amateur who debunked the mathematics of happiness”
20 0.83845121 2269 andrew gelman stats-2014-03-27-Beyond the Valley of the Trolls
topicId topicWeight
[(2, 0.017), (7, 0.054), (9, 0.03), (15, 0.099), (16, 0.089), (21, 0.013), (24, 0.146), (36, 0.012), (45, 0.01), (65, 0.012), (86, 0.058), (98, 0.025), (99, 0.28)]
simIndex simValue blogId blogTitle
same-blog 1 0.97465342 1435 andrew gelman stats-2012-07-30-Retracted articles and unethical behavior in economics journals?
Introduction: Stan Liebowitz writes: Have you ever heard of an article being retracted in economics? I know you have only been doing this for a few years but I suspect that the answer is that none or very few are retracted. No economist would ever deceive another. There is virtually no interest in detecting cheating. And what good would that do if there is no form of punishment? I say this because I think I have found a case in one of our top journals but the editor allowed the authors of the original article to write an anonymous referee report defending themselves and used this report to reject my comment even though an independent referee recommended publication. My reply: I wonder how this sort of thing will change in the future as journals become less important. My impression is that, on one side, researchers are increasingly citing NBER reports, Arxiv preprints, and the like; while, from the other direction, journals such as Science and Nature are developing the reputations of being “t
2 0.96472716 902 andrew gelman stats-2011-09-12-The importance of style in academic writing
Introduction: In my comments on academic cheating , I briefly discussed the question of how some of these papers could’ve been published in the first place, given that they tend to be of low quality. (It’s rare that people plagiarize the good stuff, and, when they do—for example when a senior scholar takes credit for a junior researcher’s contributions without giving proper credit—there’s not always a paper trail, and there can be legitimate differences of opinion about the relative contributions of the participants.) Anyway, to get back to the cases at hand: how did these rulebreakers get published in the first place? The question here is not how did they get away with cheating but how is it that top journals were publishing mediocre research? In the case of the profs who falsified data (Diederik Stapel) or did not follow scientific protocol (Mark Hauser), the answer is clear: By cheating, they were able to get the sort of too-good-to-be-true results which, if they were true, would be
3 0.96447814 2353 andrew gelman stats-2014-05-30-I posted this as a comment on a sociology blog
Introduction: I discussed two problems: 1. An artificial scarcity applied to journal publication, a scarcity which I believe is being enforced based on a monetary principle of not wanting to reduce the value of publication. The problem is that journals don’t just spread information and improve communication, they also represent chits for hiring and promotion. I’d prefer to separate these two aspects of publication. To keep these functions tied together seems to me like a terrible mistake. It would be as if, instead of using dollar bills as currency, we were to just use paper , and then if the government kept paper artificially scarce to retain the value of money, so that we were reduced to scratching notes to each other on walls and tables. 2. The discontinuous way in which unpublished papers and submissions to journals are taken as highly suspect and requiring a strong justification of all methods and assumptions, but once a paper becomes published its conclusions are taken as true unless
4 0.95935297 1779 andrew gelman stats-2013-03-27-“Two Dogmas of Strong Objective Bayesianism”
Introduction: Prasanta Bandyopadhyay and Gordon Brittan write : We introduce a distinction, unnoticed in the literature, between four varieties of objective Bayesianism. What we call ‘strong objective Bayesianism’ is characterized by two claims, that all scientific inference is ‘logical’ and that, given the same background information two agents will ascribe a unique probability to their priors. We think that neither of these claims can be sustained; in this sense, they are ‘dogmatic’. The first fails to recognize that some scientific inference, in particular that concerning evidential relations, is not (in the appropriate sense) logical, the second fails to provide a non-question-begging account of ‘same background information’. We urge that a suitably objective Bayesian account of scientific inference does not require either of the claims. Finally, we argue that Bayesianism needs to be fine-grained in the same way that Bayesians fine-grain their beliefs. I have not read their paper in detai
5 0.95813882 2244 andrew gelman stats-2014-03-11-What if I were to stop publishing in journals?
Introduction: In our recent discussion of modes of publication, Joseph Wilson wrote, “The single best reform science can make right now is to decouple publication from career advancement, thereby reducing the number of publications by an order of magnitude and then move to an entirely disjointed, informal, online free-for-all communication system for research results.” My first thought on this was: Sure, yeah, that makes sense. But then I got to thinking: what would it really mean to decouple publication from career advancement? This is too late for me—I’m middle-aged and have no career advancement in my future—but it got me thinking more carefully about the role of publication in the research process, and this seemed worth a blog (the simplest sort of publication available to me). However, somewhere between writing the above paragraphs and writing the blog entry, I forgot exactly what I was going to say! I guess I should’ve just typed it all in then. In the old days I just wouldn’t run this
6 0.95712978 1273 andrew gelman stats-2012-04-20-Proposals for alternative review systems for scientific work
9 0.95534605 2277 andrew gelman stats-2014-03-31-The most-cited statistics papers ever
10 0.95512748 1848 andrew gelman stats-2013-05-09-A tale of two discussion papers
12 0.95416731 2177 andrew gelman stats-2014-01-19-“The British amateur who debunked the mathematics of happiness”
13 0.95311111 721 andrew gelman stats-2011-05-20-Non-statistical thinking in the US foreign policy establishment
15 0.95285743 1774 andrew gelman stats-2013-03-22-Likelihood Ratio ≠ 1 Journal
16 0.95248592 226 andrew gelman stats-2010-08-23-More on those L.A. Times estimates of teacher effectiveness
17 0.95175236 2137 andrew gelman stats-2013-12-17-Replication backlash
18 0.95164871 2304 andrew gelman stats-2014-04-24-An open site for researchers to post and share papers
19 0.95121652 167 andrew gelman stats-2010-07-27-Why don’t more medical discoveries become cures?
20 0.95051694 1637 andrew gelman stats-2012-12-24-Textbook for data visualization?