andrew_gelman_stats andrew_gelman_stats-2014 andrew_gelman_stats-2014-2244 knowledge-graph by maker-knowledge-mining
Source: html
Introduction: In our recent discussion of modes of publication, Joseph Wilson wrote, “The single best reform science can make right now is to decouple publication from career advancement, thereby reducing the number of publications by an order of magnitude and then move to an entirely disjointed, informal, online free-for-all communication system for research results.” My first thought on this was: Sure, yeah, that makes sense. But then I got to thinking: what would it really mean to decouple publication from career advancement? This is too late for me—I’m middle-aged and have no career advancement in my future—but it got me thinking more carefully about the role of publication in the research process, and this seemed worth a blog (the simplest sort of publication available to me). However, somewhere between writing the above paragraphs and writing the blog entry, I forgot exactly what I was going to say! I guess I should’ve just typed it all in then. In the old days I just wouldn’t run this
sentIndex sentText sentNum sentScore
1 But then I got to thinking: what would it really mean to decouple publication from career advancement? [sent-3, score-0.759]
2 This is too late for me—I’m middle-aged and have no career advancement in my future—but it got me thinking more carefully about the role of publication in the research process, and this seemed worth a blog (the simplest sort of publication available to me). [sent-4, score-1.348]
3 However, somewhere between writing the above paragraphs and writing the blog entry, I forgot exactly what I was going to say! [sent-5, score-0.38]
4 In the old days I just wouldn’t run this post, or I’d postpone it, but my “on deck this week” feature is serving as a bit of a preregistration , so I feel I should write something on the topic even if it’s not as thoughtful or elegant as I was hoping for. [sent-7, score-0.227]
5 I don’t need publication for my career but I like to publish things anyway. [sent-11, score-0.703]
6 Most blog hits happen on the very first day, and if someone doesn’t check the blog on the day that some wonderful paper appears , they’re out of luck. [sent-18, score-0.436]
7 A paper in a journal is sitting there for a long time, it gets cited, linked to on Google scholar, etc etc. [sent-19, score-0.258]
8 When I write a paper I’m aiming for lasting impact, not instant attention. [sent-20, score-0.175]
9 On another blog, organizational theorist Balazs Kovacs was derided for writing that “The main reason that I love getting a paper published is that then I can close the process and move on to other new and exciting projects. [sent-23, score-0.473]
10 When I look back on various 5-year-old or 10-year-old or 20-year-old projects that I just wanted to get done, just to get the projects out of the way, I realize that the final products are not so wonderful and in retrospect maybe they could’ve been left unpublished. [sent-27, score-0.435]
11 OK, the vetting is not always so great—as we all know, mistakes do get into the published literature. [sent-32, score-0.218]
12 I don’t need career advancement but students and postdocs do. [sent-45, score-0.491]
13 If you publish a discussion paper in a journal you can get thoughtful comments from leaders in the field. [sent-50, score-0.598]
14 Nowadays I can do this one more easily by blogging, but in past years, back when journals were printed and appeared in everybody’s mailbox on a regular basis, the right article at the right time could get a lot of attention. [sent-54, score-0.543]
15 If we see something published that’s interesting or bothersome, and we have somewhere to go from there (whether it be a criticism, a refutation, or an improvement), we’d like to publish this in the same place, to reach the people who are (or should) be most interested in the topic. [sent-59, score-0.457]
16 The requirements of publication generally improve a paper. [sent-62, score-0.336]
17 Sometimes there is awkwardness in writing for the reviewers rather than for the readers. [sent-64, score-0.249]
18 Are those reasons enough to go through the bother of publication? [sent-66, score-0.186]
19 It was not my intent to defend journal publication or the current system of journals. [sent-73, score-0.636]
20 Rather, I was just trying to lay out, as clearly as possible, my own motivation for continuing to bother submitting papers to journals, given the nontrivial efforts of submissions and revisions and the unclear gains from publication. [sent-74, score-0.321]
wordName wordTfidf (topN-words)
[('publication', 0.336), ('advancement', 0.288), ('career', 0.203), ('publish', 0.164), ('decouple', 0.153), ('arxiv', 0.15), ('journal', 0.147), ('published', 0.132), ('blog', 0.118), ('paper', 0.111), ('reasons', 0.095), ('writing', 0.092), ('reviewers', 0.091), ('bother', 0.091), ('thoughtful', 0.09), ('wonderful', 0.089), ('publications', 0.089), ('projects', 0.087), ('system', 0.087), ('right', 0.086), ('get', 0.086), ('reach', 0.083), ('motivation', 0.083), ('algorithm', 0.08), ('papers', 0.078), ('somewhere', 0.078), ('mailbox', 0.076), ('rhmc', 0.076), ('disjointed', 0.076), ('balazs', 0.076), ('kovacs', 0.076), ('calderhead', 0.076), ('appeared', 0.076), ('feel', 0.071), ('attention', 0.071), ('journals', 0.069), ('getting', 0.069), ('stream', 0.069), ('girolami', 0.069), ('theorist', 0.069), ('revisions', 0.069), ('unfortunately', 0.068), ('got', 0.067), ('awkwardness', 0.066), ('intent', 0.066), ('deck', 0.066), ('stan', 0.066), ('operate', 0.064), ('printed', 0.064), ('lasting', 0.064)]
simIndex simValue blogId blogTitle
same-blog 1 1.0000001 2244 andrew gelman stats-2014-03-11-What if I were to stop publishing in journals?
Introduction: In our recent discussion of modes of publication, Joseph Wilson wrote, “The single best reform science can make right now is to decouple publication from career advancement, thereby reducing the number of publications by an order of magnitude and then move to an entirely disjointed, informal, online free-for-all communication system for research results.” My first thought on this was: Sure, yeah, that makes sense. But then I got to thinking: what would it really mean to decouple publication from career advancement? This is too late for me—I’m middle-aged and have no career advancement in my future—but it got me thinking more carefully about the role of publication in the research process, and this seemed worth a blog (the simplest sort of publication available to me). However, somewhere between writing the above paragraphs and writing the blog entry, I forgot exactly what I was going to say! I guess I should’ve just typed it all in then. In the old days I just wouldn’t run this
2 0.35203713 2232 andrew gelman stats-2014-03-03-What is the appropriate time scale for blogging—the day or the week?
Introduction: I post (approximately) once a day and don’t plan to change that. I have enough material to post more often—for example, I could intersperse existing blog posts with summaries of my published papers or of other work that I like; and, beyond this, we currently have a one-to-two-month backlog of posts—but I’m afraid that if the number of posts were doubled, the attention given to each would be roughly halved. Looking at it the other way, I certainly don’t want to reduce my level of posting. Sure, it takes time to blog, but these are things that are important for me to say. If I were to blog less frequently, it would only be because I was pouring all these words into a different vessel, for example a book. For now, though, I think it makes sense to blog and then collect the words later as appropriate. With blogging I get comments, and many of these comments are helpful—either directly (by pointing out errors in my thinking or linking to relevant software or literature) or indirec
3 0.28752166 2245 andrew gelman stats-2014-03-12-More on publishing in journals
Introduction: I’m postponing today’s scheduled post (“Empirical implications of Empirical Implications of Theoretical Models”) to continue the lively discussion from yesterday, What if I were to stop publishing in journals? . An example: my papers with Basbøll Thomas Basbøll and I got into a long discussion on our blogs about business school professor Karl Weick and other cases of plagiarism copying text without attribution. We felt it useful to take our ideas to the next level and write them up as a manuscript, which ended up being logical to split into two papers. At that point I put some effort into getting these papers published, which I eventually did: To throw away data: Plagiarism as a statistical crime went into American Scientist and When do stories work? Evidence and illustration in the social sciences will appear in Sociological Methods and Research. The second paper, in particular, took some effort to place; I got some advice from colleagues in sociology as to where
4 0.27674574 1291 andrew gelman stats-2012-04-30-Systematic review of publication bias in studies on publication bias
Introduction: Via Yalda Afshar , a 2005 paper by Hans-Hermann Dubben and Hans-Peter Beck-Bornholdt: Publication bias is a well known phenomenon in clinical literature, in which positive results have a better chance of being published, are published earlier, and are published in journals with higher impact factors. Conclusions exclusively based on published studies, therefore, can be misleading. Selective under-reporting of research might be more widespread and more likely to have adverse consequences for patients than publication of deliberately falsified data. We investigated whether there is preferential publication of positive papers on publication bias. They conclude, “We found no evidence of publication bias in reports on publication bias.” But of course that’s the sort of finding regarding publication bias of findings on publication bias that you’d expect would get published. What we really need is a careful meta-analysis to estimate the level of publication bias in studies of publi
5 0.22245838 2233 andrew gelman stats-2014-03-04-Literal vs. rhetorical
Introduction: Thomas Basbøll pointed me to a discussion on the orgtheory blog in which Jerry Davis, the editor of a journal of business management argued that it is difficult for academic researchers to communicate with the public because “the public prefers Cheetos to a healthy salad” and when serious papers are discussed on the internet, “everyone is a methodologist.” The discussion heated up when an actual methodologist, Steve Morgan, joined in to argue that the salad in question was not so healthy and that the much-derided internet commenters made some valuable points. The final twist was that one of the orgtheory bloggers deleted a comment and then closed the thread entirely when the discussion got too conflictual. In a few days I’ll return to the meta-topic of the discussion, but right now I want to focus on one thing Davis wrote, a particular statement that illustrates to me the gap between the rhetorical and the literal, the way in which a statement can sound good but make no sense. He
6 0.21568319 1928 andrew gelman stats-2013-07-06-How to think about papers published in low-grade journals?
8 0.2053566 1393 andrew gelman stats-2012-06-26-The reverse-journal-submission system
9 0.2046091 2353 andrew gelman stats-2014-05-30-I posted this as a comment on a sociology blog
11 0.18084936 1435 andrew gelman stats-2012-07-30-Retracted articles and unethical behavior in economics journals?
12 0.17495009 1139 andrew gelman stats-2012-01-26-Suggested resolution of the Bem paradox
13 0.169742 120 andrew gelman stats-2010-06-30-You can’t put Pandora back in the box
14 0.16691968 1878 andrew gelman stats-2013-05-31-How to fix the tabloids? Toward replicable social science research
15 0.15732288 1273 andrew gelman stats-2012-04-20-Proposals for alternative review systems for scientific work
16 0.1557458 1254 andrew gelman stats-2012-04-09-In the future, everyone will publish everything.
17 0.1534459 1844 andrew gelman stats-2013-05-06-Against optimism about social science
19 0.14980929 834 andrew gelman stats-2011-08-01-I owe it all to the haters
20 0.14968163 2137 andrew gelman stats-2013-12-17-Replication backlash
topicId topicWeight
[(0, 0.292), (1, -0.123), (2, -0.128), (3, -0.084), (4, -0.038), (5, -0.062), (6, 0.092), (7, -0.187), (8, -0.022), (9, -0.051), (10, 0.169), (11, 0.02), (12, -0.063), (13, 0.032), (14, -0.012), (15, -0.052), (16, -0.027), (17, 0.019), (18, -0.059), (19, 0.042), (20, 0.053), (21, 0.031), (22, -0.014), (23, 0.045), (24, -0.005), (25, 0.004), (26, -0.023), (27, 0.024), (28, -0.034), (29, 0.022), (30, -0.0), (31, -0.088), (32, 0.019), (33, 0.03), (34, -0.021), (35, -0.038), (36, 0.003), (37, 0.058), (38, -0.038), (39, 0.008), (40, -0.025), (41, 0.011), (42, -0.025), (43, -0.001), (44, 0.022), (45, 0.005), (46, -0.007), (47, -0.004), (48, -0.041), (49, 0.005)]
simIndex simValue blogId blogTitle
same-blog 1 0.98075986 2244 andrew gelman stats-2014-03-11-What if I were to stop publishing in journals?
Introduction: In our recent discussion of modes of publication, Joseph Wilson wrote, “The single best reform science can make right now is to decouple publication from career advancement, thereby reducing the number of publications by an order of magnitude and then move to an entirely disjointed, informal, online free-for-all communication system for research results.” My first thought on this was: Sure, yeah, that makes sense. But then I got to thinking: what would it really mean to decouple publication from career advancement? This is too late for me—I’m middle-aged and have no career advancement in my future—but it got me thinking more carefully about the role of publication in the research process, and this seemed worth a blog (the simplest sort of publication available to me). However, somewhere between writing the above paragraphs and writing the blog entry, I forgot exactly what I was going to say! I guess I should’ve just typed it all in then. In the old days I just wouldn’t run this
2 0.91285831 2233 andrew gelman stats-2014-03-04-Literal vs. rhetorical
Introduction: Thomas Basbøll pointed me to a discussion on the orgtheory blog in which Jerry Davis, the editor of a journal of business management argued that it is difficult for academic researchers to communicate with the public because “the public prefers Cheetos to a healthy salad” and when serious papers are discussed on the internet, “everyone is a methodologist.” The discussion heated up when an actual methodologist, Steve Morgan, joined in to argue that the salad in question was not so healthy and that the much-derided internet commenters made some valuable points. The final twist was that one of the orgtheory bloggers deleted a comment and then closed the thread entirely when the discussion got too conflictual. In a few days I’ll return to the meta-topic of the discussion, but right now I want to focus on one thing Davis wrote, a particular statement that illustrates to me the gap between the rhetorical and the literal, the way in which a statement can sound good but make no sense. He
3 0.90589958 834 andrew gelman stats-2011-08-01-I owe it all to the haters
Introduction: Sometimes when I submit an article to a journal it is accepted right away or with minor alterations. But many of my favorite articles were rejected or had to go through an exhausting series of revisions. For example, this influential article had a very hostile referee and we had to seriously push the journal editor to accept it. This one was rejected by one or two journals before finally appearing with discussion. This paper was rejected by the American Political Science Review with no chance of revision and we had to publish it in the British Journal of Political Science, which was a bit odd given that the article was 100% about American politics. And when I submitted this instant classic (actually at the invitation of the editor), the referees found it to be trivial, and the editor did me the favor of publishing it but only by officially labeling it as a discussion of another article that appeared in the same issue. Some of my most influential papers were accepted right
4 0.87740582 2353 andrew gelman stats-2014-05-30-I posted this as a comment on a sociology blog
Introduction: I discussed two problems: 1. An artificial scarcity applied to journal publication, a scarcity which I believe is being enforced based on a monetary principle of not wanting to reduce the value of publication. The problem is that journals don’t just spread information and improve communication, they also represent chits for hiring and promotion. I’d prefer to separate these two aspects of publication. To keep these functions tied together seems to me like a terrible mistake. It would be as if, instead of using dollar bills as currency, we were to just use paper , and then if the government kept paper artificially scarce to retain the value of money, so that we were reduced to scratching notes to each other on walls and tables. 2. The discontinuous way in which unpublished papers and submissions to journals are taken as highly suspect and requiring a strong justification of all methods and assumptions, but once a paper becomes published its conclusions are taken as true unless
5 0.87342924 1321 andrew gelman stats-2012-05-15-A statistical research project: Weeding out the fraudulent citations
Introduction: John Mashey points me to a blog post by Phil Davis on “the emergence of a citation cartel.” Davis tells the story: Cell Transplantation is a medical journal published by the Cognizant Communication Corporation of Putnam Valley, New York. In recent years, its impact factor has been growing rapidly. In 2006, it was 3.482 [I think he means "3.5"---ed.]. In 2010, it had almost doubled to 6.204. When you look at which journals cite Cell Transplantation, two journals stand out noticeably: the Medical Science Monitor, and The Scientific World Journal. According to the JCR, neither of these journals cited Cell Transplantation until 2010. Then, in 2010, a review article was published in the Medical Science Monitor citing 490 articles, 445 of which were to papers published in Cell Transplantation. All 445 citations pointed to papers published in 2008 or 2009 — the citation window from which the journal’s 2010 impact factor was derived. Of the remaining 45 citations, 44 cited the Me
7 0.87228417 2245 andrew gelman stats-2014-03-12-More on publishing in journals
8 0.86578298 1393 andrew gelman stats-2012-06-26-The reverse-journal-submission system
9 0.86551088 1137 andrew gelman stats-2012-01-24-Difficulties in publishing non-replications of implausible findings
10 0.86341953 883 andrew gelman stats-2011-09-01-Arrow’s theorem update
11 0.86144066 1928 andrew gelman stats-2013-07-06-How to think about papers published in low-grade journals?
13 0.85743725 1254 andrew gelman stats-2012-04-09-In the future, everyone will publish everything.
15 0.84750479 1429 andrew gelman stats-2012-07-26-Our broken scholarly publishing system
16 0.84738386 1273 andrew gelman stats-2012-04-20-Proposals for alternative review systems for scientific work
17 0.8450312 675 andrew gelman stats-2011-04-22-Arrow’s other theorem
18 0.84365946 1435 andrew gelman stats-2012-07-30-Retracted articles and unethical behavior in economics journals?
19 0.83199435 2269 andrew gelman stats-2014-03-27-Beyond the Valley of the Trolls
20 0.82407868 902 andrew gelman stats-2011-09-12-The importance of style in academic writing
topicId topicWeight
[(2, 0.046), (9, 0.01), (15, 0.096), (16, 0.096), (20, 0.012), (21, 0.019), (22, 0.023), (24, 0.198), (28, 0.011), (55, 0.041), (70, 0.023), (77, 0.016), (86, 0.012), (99, 0.279)]
simIndex simValue blogId blogTitle
same-blog 1 0.97210228 2244 andrew gelman stats-2014-03-11-What if I were to stop publishing in journals?
Introduction: In our recent discussion of modes of publication, Joseph Wilson wrote, “The single best reform science can make right now is to decouple publication from career advancement, thereby reducing the number of publications by an order of magnitude and then move to an entirely disjointed, informal, online free-for-all communication system for research results.” My first thought on this was: Sure, yeah, that makes sense. But then I got to thinking: what would it really mean to decouple publication from career advancement? This is too late for me—I’m middle-aged and have no career advancement in my future—but it got me thinking more carefully about the role of publication in the research process, and this seemed worth a blog (the simplest sort of publication available to me). However, somewhere between writing the above paragraphs and writing the blog entry, I forgot exactly what I was going to say! I guess I should’ve just typed it all in then. In the old days I just wouldn’t run this
2 0.96391487 1171 andrew gelman stats-2012-02-16-“False-positive psychology”
Introduction: Everybody’s talkin bout this paper by Joseph Simmons, Leif Nelson and Uri Simonsohn, who write : Despite empirical psychologists’ nominal endorsement of a low rate of false-positive findings (≤ .05), flexibility in data collection, analysis, and reporting dramatically increases actual false-positive rates. In many cases, a researcher is more likely to falsely find evidence that an effect exists than to correctly find evidence that it does not. We [Simmons, Nelson, and Simonsohn] present computer simulations and a pair of actual experiments that demonstrate how unacceptably easy it is to accumulate (and report) statistically significant evidence for a false hypothesis. Second, we suggest a simple, low-cost, and straightforwardly effective disclosure-based solution to this problem. The solution involves six concrete requirements for authors and four guidelines for reviewers, all of which impose a minimal burden on the publication process. Whatever you think about these recommend
3 0.96385604 2353 andrew gelman stats-2014-05-30-I posted this as a comment on a sociology blog
Introduction: I discussed two problems: 1. An artificial scarcity applied to journal publication, a scarcity which I believe is being enforced based on a monetary principle of not wanting to reduce the value of publication. The problem is that journals don’t just spread information and improve communication, they also represent chits for hiring and promotion. I’d prefer to separate these two aspects of publication. To keep these functions tied together seems to me like a terrible mistake. It would be as if, instead of using dollar bills as currency, we were to just use paper , and then if the government kept paper artificially scarce to retain the value of money, so that we were reduced to scratching notes to each other on walls and tables. 2. The discontinuous way in which unpublished papers and submissions to journals are taken as highly suspect and requiring a strong justification of all methods and assumptions, but once a paper becomes published its conclusions are taken as true unless
Introduction: Chris Masse points me to this response by Daryl Bem and two statisticians (Jessica Utts and Wesley Johnson) to criticisms by Wagenmakers et.al. of Bem’s recent ESP study. I have nothing to add but would like to repeat a couple bits of my discussions of last month, of here : Classical statistical methods that work reasonably well when studying moderate or large effects (see the work of Fisher, Snedecor, Cochran, etc.) fall apart in the presence of small effects. I think it’s naive when people implicitly assume that the study’s claims are correct, or the study’s statistical methods are weak. Generally, the smaller the effects you’re studying, the better the statistics you need. ESP is a field of small effects and so ESP researchers use high-quality statistics. To put it another way: whatever methodological errors happen to be in the paper in question, probably occur in lots of researcher papers in “legitimate” psychology research. The difference is that when you’re studying a
5 0.96220326 902 andrew gelman stats-2011-09-12-The importance of style in academic writing
Introduction: In my comments on academic cheating , I briefly discussed the question of how some of these papers could’ve been published in the first place, given that they tend to be of low quality. (It’s rare that people plagiarize the good stuff, and, when they do—for example when a senior scholar takes credit for a junior researcher’s contributions without giving proper credit—there’s not always a paper trail, and there can be legitimate differences of opinion about the relative contributions of the participants.) Anyway, to get back to the cases at hand: how did these rulebreakers get published in the first place? The question here is not how did they get away with cheating but how is it that top journals were publishing mediocre research? In the case of the profs who falsified data (Diederik Stapel) or did not follow scientific protocol (Mark Hauser), the answer is clear: By cheating, they were able to get the sort of too-good-to-be-true results which, if they were true, would be
6 0.95853817 803 andrew gelman stats-2011-07-14-Subtleties with measurement-error models for the evaluation of wacky claims
7 0.95722139 1713 andrew gelman stats-2013-02-08-P-values and statistical practice
8 0.95627826 1578 andrew gelman stats-2012-11-15-Outta control political incorrectness
9 0.95468652 2149 andrew gelman stats-2013-12-26-Statistical evidence for revised standards
10 0.95297354 2299 andrew gelman stats-2014-04-21-Stan Model of the Week: Hierarchical Modeling of Supernovas
11 0.95277619 2208 andrew gelman stats-2014-02-12-How to think about “identifiability” in Bayesian inference?
12 0.9526599 2227 andrew gelman stats-2014-02-27-“What Can we Learn from the Many Labs Replication Project?”
13 0.95253605 1240 andrew gelman stats-2012-04-02-Blogads update
14 0.95204353 511 andrew gelman stats-2011-01-11-One more time on that ESP study: The problem of overestimates and the shrinkage solution
15 0.9516952 2177 andrew gelman stats-2014-01-19-“The British amateur who debunked the mathematics of happiness”
16 0.95161146 1435 andrew gelman stats-2012-07-30-Retracted articles and unethical behavior in economics journals?
17 0.95097458 898 andrew gelman stats-2011-09-10-Fourteen magic words: an update
18 0.95087111 2179 andrew gelman stats-2014-01-20-The AAA Tranche of Subprime Science
19 0.95071769 1779 andrew gelman stats-2013-03-27-“Two Dogmas of Strong Objective Bayesianism”
20 0.95071608 2248 andrew gelman stats-2014-03-15-Problematic interpretations of confidence intervals