andrew_gelman_stats andrew_gelman_stats-2011 andrew_gelman_stats-2011-908 knowledge-graph by maker-knowledge-mining
Source: html
Introduction: Jeff points us to this news article by Asher Mullard: Bayer halts nearly two-thirds of its target-validation projects because in-house experimental findings fail to match up with published literature claims, finds a first-of-a-kind analysis on data irreproducibility. An unspoken industry rule alleges that at least 50% of published studies from academic laboratories cannot be repeated in an industrial setting, wrote venture capitalist Bruce Booth in a recent blog post. A first-of-a-kind analysis of Bayer’s internal efforts to validate ‘new drug target’ claims now not only supports this view but suggests that 50% may be an underestimate; the company’s in-house experimental data do not match literature claims in 65% of target-validation projects, leading to project discontinuation. . . . Khusru Asadullah, Head of Target Discovery at Bayer, and his colleagues looked back at 67 target-validation projects, covering the majority of Bayer’s work in oncology, women’s health and cardiov
sentIndex sentText sentNum sentScore
1 Jeff points us to this news article by Asher Mullard: Bayer halts nearly two-thirds of its target-validation projects because in-house experimental findings fail to match up with published literature claims, finds a first-of-a-kind analysis on data irreproducibility. [sent-1, score-1.077]
2 An unspoken industry rule alleges that at least 50% of published studies from academic laboratories cannot be repeated in an industrial setting, wrote venture capitalist Bruce Booth in a recent blog post. [sent-2, score-0.975]
3 Khusru Asadullah, Head of Target Discovery at Bayer, and his colleagues looked back at 67 target-validation projects, covering the majority of Bayer’s work in oncology, women’s health and cardiovascular medicine over the past 4 years. [sent-7, score-0.252]
4 Of these, results from internal experiments matched up with the published findings in only 14 projects, but were highly inconsistent in 43 (in a further 10 projects, claims were rated as mostly reproducible, partially reproducible or not applicable . [sent-8, score-1.208]
5 High-impact journals did not seem to publish more robust claims, and, surprisingly, the confirmation of any given finding by another academic group did not improve data reliability. [sent-11, score-0.243]
6 “We didn’t see that a target is more likely to be validated if it was reported in ten publications or in two publications,” says Asadullah. [sent-12, score-0.507]
7 There’s also this amusing bit: The analysis is limited by a small sample size, and cannot itself be checked because of company confidentiality concerns . [sent-16, score-0.426]
wordName wordTfidf (topN-words)
[('bayer', 0.515), ('projects', 0.334), ('claims', 0.217), ('target', 0.208), ('reproducible', 0.185), ('internal', 0.157), ('publications', 0.136), ('match', 0.132), ('company', 0.118), ('alleges', 0.117), ('laboratories', 0.117), ('unspoken', 0.111), ('experimental', 0.109), ('rated', 0.106), ('capitalist', 0.102), ('cardiovascular', 0.102), ('oncology', 0.102), ('published', 0.102), ('booth', 0.099), ('validated', 0.099), ('bruce', 0.096), ('applicable', 0.096), ('underestimate', 0.096), ('venture', 0.094), ('confidentiality', 0.094), ('academic', 0.094), ('findings', 0.093), ('validate', 0.092), ('industrial', 0.089), ('inconsistent', 0.088), ('literature', 0.087), ('covering', 0.084), ('matched', 0.083), ('confirmation', 0.082), ('partially', 0.081), ('supports', 0.08), ('surprisingly', 0.075), ('repeated', 0.075), ('analysis', 0.075), ('finds', 0.075), ('industry', 0.074), ('fail', 0.07), ('amusing', 0.07), ('checked', 0.069), ('discovery', 0.069), ('head', 0.068), ('robust', 0.067), ('medicine', 0.066), ('drug', 0.066), ('ten', 0.064)]
simIndex simValue blogId blogTitle
same-blog 1 1.0000002 908 andrew gelman stats-2011-09-14-Type M errors in the lab
Introduction: Jeff points us to this news article by Asher Mullard: Bayer halts nearly two-thirds of its target-validation projects because in-house experimental findings fail to match up with published literature claims, finds a first-of-a-kind analysis on data irreproducibility. An unspoken industry rule alleges that at least 50% of published studies from academic laboratories cannot be repeated in an industrial setting, wrote venture capitalist Bruce Booth in a recent blog post. A first-of-a-kind analysis of Bayer’s internal efforts to validate ‘new drug target’ claims now not only supports this view but suggests that 50% may be an underestimate; the company’s in-house experimental data do not match literature claims in 65% of target-validation projects, leading to project discontinuation. . . . Khusru Asadullah, Head of Target Discovery at Bayer, and his colleagues looked back at 67 target-validation projects, covering the majority of Bayer’s work in oncology, women’s health and cardiov
2 0.10655738 2227 andrew gelman stats-2014-02-27-“What Can we Learn from the Many Labs Replication Project?”
Introduction: Aki points us to this discussion from Rolf Zwaan: The first massive replication project in psychology has just reached completion (several others are to follow). . . . What can we learn from the ManyLabs project? The results here show the effect sizes for the replication efforts (in green and grey) as well as the original studies (in blue). The 99% confidence intervals are for the meta-analysis of the effect size (the green dots); the studies are ordered by effect size. Let’s first consider what we canNOT learn from these data. Of the 13 replication attempts (when the first four are taken together), 11 succeeded and 2 did not (in fact, at some point ManyLabs suggests that a third one, Imagined Contact also doesn’t really replicate). We cannot learn from this that the vast majority of psychological findings will replicate . . . But even if we had an accurate estimate of the percentage of findings that replicate, how useful would that be? Rather than trying to arrive at a mo
3 0.088453777 2244 andrew gelman stats-2014-03-11-What if I were to stop publishing in journals?
Introduction: In our recent discussion of modes of publication, Joseph Wilson wrote, “The single best reform science can make right now is to decouple publication from career advancement, thereby reducing the number of publications by an order of magnitude and then move to an entirely disjointed, informal, online free-for-all communication system for research results.” My first thought on this was: Sure, yeah, that makes sense. But then I got to thinking: what would it really mean to decouple publication from career advancement? This is too late for me—I’m middle-aged and have no career advancement in my future—but it got me thinking more carefully about the role of publication in the research process, and this seemed worth a blog (the simplest sort of publication available to me). However, somewhere between writing the above paragraphs and writing the blog entry, I forgot exactly what I was going to say! I guess I should’ve just typed it all in then. In the old days I just wouldn’t run this
4 0.08823546 1683 andrew gelman stats-2013-01-19-“Confirmation, on the other hand, is not sexy”
Introduction: Mark Palko writes : I can understand the appeal of the cutting edge. The new stuff is sexier. It gets people’s attention. The trouble is, those cutting edge studies often collapse under scrutiny. Some can’t be replicated. Others prove to be not that important. Confirmation, on the other hand, is not sexy. It doesn’t drive traffic. It’s harder to fit into a paragraph. In a way, though, it’s more interesting because it has a high likelihood of being true and fills in the gaps in big, important questions. The interaction between the ideas is usually the interesting part. In this particular example, Palko is telling the story of a journalist who reports a finding as new when it is essentially a replication of decades-old work. Palko’s point is not that there’s anything wrong with replication but rather that the journalist seems to feel that it is necessary to report the idea as new and cutting-edge, even if it falls within a long tradition. (Also, Palko is not claiming that this
5 0.087402225 637 andrew gelman stats-2011-03-29-Unfinished business
Introduction: This blog by J. Robert Lennon on abandoned novels made me think of the more general topic of abandoned projects. I seem to recall George V. Higgins writing that he’d written and discarded 14 novels or so before publishing The Friends of Eddie Coyle. I haven’t abandoned any novels but I’ve abandoned lots of research projects (and also have started various projects that there’s no way I’ll finish). If you think about the decisions involved, it really has to be that way. You learn while you’re working on a project whether it’s worth continuing. Sometimes I’ve put in the hard work and pushed a project to completion, published the article, and then I think . . . what was the point? The modal number of citations of our articles is zero, etc.
6 0.083192073 723 andrew gelman stats-2011-05-21-Literary blurb translation guide
7 0.079337962 167 andrew gelman stats-2010-07-27-Why don’t more medical discoveries become cures?
8 0.079011776 1139 andrew gelman stats-2012-01-26-Suggested resolution of the Bem paradox
9 0.077086851 1435 andrew gelman stats-2012-07-30-Retracted articles and unethical behavior in economics journals?
10 0.07661286 213 andrew gelman stats-2010-08-17-Matching at two levels
11 0.076161414 423 andrew gelman stats-2010-11-20-How to schedule projects in an introductory statistics course?
12 0.074799418 1671 andrew gelman stats-2013-01-13-Preregistration of Studies and Mock Reports
13 0.074537128 1998 andrew gelman stats-2013-08-25-A new Bem theory
14 0.07443656 714 andrew gelman stats-2011-05-16-NYT Labs releases Openpaths, a utility for saving your iphone data
15 0.070620224 2032 andrew gelman stats-2013-09-20-“Six red flags for suspect work”
16 0.068567514 2137 andrew gelman stats-2013-12-17-Replication backlash
17 0.066290095 361 andrew gelman stats-2010-10-21-Tenure-track statistics job at Teachers College, here at Columbia!
18 0.064813249 2245 andrew gelman stats-2014-03-12-More on publishing in journals
19 0.06379579 1860 andrew gelman stats-2013-05-17-How can statisticians help psychologists do their research better?
20 0.063495517 202 andrew gelman stats-2010-08-12-Job openings in multilevel modeling in Bristol, England
topicId topicWeight
[(0, 0.107), (1, -0.035), (2, -0.018), (3, -0.096), (4, -0.016), (5, -0.003), (6, -0.02), (7, -0.039), (8, -0.049), (9, 0.021), (10, 0.009), (11, -0.002), (12, -0.012), (13, 0.002), (14, -0.012), (15, 0.02), (16, 0.009), (17, 0.004), (18, 0.03), (19, -0.013), (20, -0.008), (21, 0.023), (22, -0.035), (23, 0.002), (24, -0.033), (25, 0.003), (26, -0.023), (27, -0.01), (28, 0.013), (29, 0.016), (30, -0.028), (31, -0.046), (32, 0.017), (33, -0.0), (34, 0.021), (35, 0.044), (36, -0.029), (37, 0.003), (38, -0.0), (39, 0.0), (40, 0.009), (41, -0.019), (42, 0.0), (43, 0.015), (44, 0.037), (45, 0.006), (46, -0.019), (47, 0.036), (48, -0.008), (49, 0.024)]
simIndex simValue blogId blogTitle
same-blog 1 0.96703982 908 andrew gelman stats-2011-09-14-Type M errors in the lab
Introduction: Jeff points us to this news article by Asher Mullard: Bayer halts nearly two-thirds of its target-validation projects because in-house experimental findings fail to match up with published literature claims, finds a first-of-a-kind analysis on data irreproducibility. An unspoken industry rule alleges that at least 50% of published studies from academic laboratories cannot be repeated in an industrial setting, wrote venture capitalist Bruce Booth in a recent blog post. A first-of-a-kind analysis of Bayer’s internal efforts to validate ‘new drug target’ claims now not only supports this view but suggests that 50% may be an underestimate; the company’s in-house experimental data do not match literature claims in 65% of target-validation projects, leading to project discontinuation. . . . Khusru Asadullah, Head of Target Discovery at Bayer, and his colleagues looked back at 67 target-validation projects, covering the majority of Bayer’s work in oncology, women’s health and cardiov
2 0.84391654 1671 andrew gelman stats-2013-01-13-Preregistration of Studies and Mock Reports
Introduction: The traditional system of scientific and scholarly publishing is breaking down in two different directions. On one hand, we are moving away from relying on a small set of journals as gatekeepers: the number of papers and research projects is increasing, the number of publication outlets is increasing, and important manuscripts are being posted on SSRN, Arxiv, and other nonrefereed sites. At the same time, many researchers are worried about the profusion of published claims that turn out to not replicate or in plain language, to be false. This concern is not new–some prominent discussions include Rosenthal (1979), Ioannidis (2005), and Vul et al. (2009)–but there is a growing sense that the scientific signal is being swamped by noise. I recently had the opportunity to comment in the journal Political Analysis on two papers, one by Humphreys, Sierra, and Windt, and one by Monogan, on the preregistration of studies and mock reports. Here’s the issue of the journal. Given the hi
3 0.83029968 2301 andrew gelman stats-2014-04-22-Ticket to Baaaaarf
Introduction: A link from the comments here took me to the wonderfully named Barfblog and a report by Don Schaffner on some reporting. First, the background: A university in England issued a press release saying that “Food picked up just a few seconds after being dropped is less likely to contain bacteria than if it is left for longer periods of time . . . The findings suggest there may be some scientific basis to the ‘5 second rule’ – the urban myth about it being fine to eat food that has only had contact with the floor for five seconds or less. Although people have long followed the 5 second rule, until now it was unclear whether it actually helped.” According to the press release, the study was “undertaken by final year Biology students” and led by a professor of microbiology. The press release hit the big time, hitting NPR, Slate, Forbes, the Daily News, etc etc. Some typical headlines: “5-second rule backed up by science” — Atlanta Journal Constitution “Eating food off the floo
4 0.78181314 2179 andrew gelman stats-2014-01-20-The AAA Tranche of Subprime Science
Introduction: In our new ethics column for Chance , Eric Loken and I write about our current favorite topic: One of our ongoing themes when discussing scientific ethics is the central role of statistics in recognizing and communicating uncer- tainty. Unfortunately, statistics—and the scientific process more generally—often seems to be used more as a way of laundering uncertainty, processing data until researchers and consumers of research can feel safe acting as if various scientific hypotheses are unquestionably true. . . . We have in mind an analogy with the notorious AAA-class bonds created during the mid-2000s that led to the subprime mortgage crisis. Lower-quality mortgages—that is, mortgages with high probability of default and, thus, high uncertainty—were packaged and transformed into financial instruments that were (in retrospect, falsely) characterized as low risk. There was a tremendous interest in these securities, not just among the most unscrupulous market manipulators, but in a
5 0.77946895 1878 andrew gelman stats-2013-05-31-How to fix the tabloids? Toward replicable social science research
Introduction: This seems to be the topic of the week. Yesterday I posted on the sister blog some further thoughts on those “Psychological Science” papers on menstrual cycles, biceps size, and political attitudes, tied to a horrible press release from the journal Psychological Science hyping the biceps and politics study. Then I was pointed to these suggestions from Richard Lucas and M. Brent Donnellan have on improving the replicability and reproducibility of research published in the Journal of Research in Personality: It goes without saying that editors of scientific journals strive to publish research that is not only theoretically interesting but also methodologically rigorous. The goal is to select papers that advance the field. Accordingly, editors want to publish findings that can be reproduced and replicated by other scientists. Unfortunately, there has been a recent “crisis in confidence” among psychologists about the quality of psychological research (Pashler & Wagenmakers, 2012)
6 0.76815218 2220 andrew gelman stats-2014-02-22-Quickies
7 0.75707704 1959 andrew gelman stats-2013-07-28-50 shades of gray: A research story
8 0.75614047 2268 andrew gelman stats-2014-03-26-New research journal on observational studies
9 0.75254154 1055 andrew gelman stats-2011-12-13-Data sharing update
10 0.75126421 1054 andrew gelman stats-2011-12-12-More frustrations trying to replicate an analysis published in a reputable journal
11 0.74977666 2137 andrew gelman stats-2013-12-17-Replication backlash
12 0.74664694 1774 andrew gelman stats-2013-03-22-Likelihood Ratio ≠ 1 Journal
13 0.74199212 1163 andrew gelman stats-2012-02-12-Meta-analysis, game theory, and incentives to do replicable research
15 0.73589903 1844 andrew gelman stats-2013-05-06-Against optimism about social science
16 0.71666789 1954 andrew gelman stats-2013-07-24-Too Good To Be True: The Scientific Mass Production of Spurious Statistical Significance
17 0.71587455 1139 andrew gelman stats-2012-01-26-Suggested resolution of the Bem paradox
18 0.71386629 1273 andrew gelman stats-2012-04-20-Proposals for alternative review systems for scientific work
19 0.71237499 2227 andrew gelman stats-2014-02-27-“What Can we Learn from the Many Labs Replication Project?”
20 0.70877093 2215 andrew gelman stats-2014-02-17-The Washington Post reprints university press releases without editing them
topicId topicWeight
[(2, 0.01), (15, 0.411), (16, 0.05), (24, 0.068), (35, 0.011), (41, 0.012), (48, 0.013), (53, 0.035), (68, 0.012), (76, 0.011), (82, 0.01), (95, 0.013), (98, 0.013), (99, 0.209)]
simIndex simValue blogId blogTitle
1 0.98843336 439 andrew gelman stats-2010-11-30-Of psychology research and investment tips
Introduction: A few days after “ Dramatic study shows participants are affected by psychological phenomena from the future ,” (see here ) the British Psychological Society follows up with “ Can psychology help combat pseudoscience? .” Somehow I’m reminded of that bit of financial advice which says, if you want to save some money, your best investment is to pay off your credit card bills.
2 0.92999756 1394 andrew gelman stats-2012-06-27-99!
Introduction: Those of you who know what I’m talking about, know what I’m talking about.
same-blog 3 0.92724895 908 andrew gelman stats-2011-09-14-Type M errors in the lab
Introduction: Jeff points us to this news article by Asher Mullard: Bayer halts nearly two-thirds of its target-validation projects because in-house experimental findings fail to match up with published literature claims, finds a first-of-a-kind analysis on data irreproducibility. An unspoken industry rule alleges that at least 50% of published studies from academic laboratories cannot be repeated in an industrial setting, wrote venture capitalist Bruce Booth in a recent blog post. A first-of-a-kind analysis of Bayer’s internal efforts to validate ‘new drug target’ claims now not only supports this view but suggests that 50% may be an underestimate; the company’s in-house experimental data do not match literature claims in 65% of target-validation projects, leading to project discontinuation. . . . Khusru Asadullah, Head of Target Discovery at Bayer, and his colleagues looked back at 67 target-validation projects, covering the majority of Bayer’s work in oncology, women’s health and cardiov
4 0.8841725 834 andrew gelman stats-2011-08-01-I owe it all to the haters
Introduction: Sometimes when I submit an article to a journal it is accepted right away or with minor alterations. But many of my favorite articles were rejected or had to go through an exhausting series of revisions. For example, this influential article had a very hostile referee and we had to seriously push the journal editor to accept it. This one was rejected by one or two journals before finally appearing with discussion. This paper was rejected by the American Political Science Review with no chance of revision and we had to publish it in the British Journal of Political Science, which was a bit odd given that the article was 100% about American politics. And when I submitted this instant classic (actually at the invitation of the editor), the referees found it to be trivial, and the editor did me the favor of publishing it but only by officially labeling it as a discussion of another article that appeared in the same issue. Some of my most influential papers were accepted right
5 0.86029863 1081 andrew gelman stats-2011-12-24-Statistical ethics violation
Introduction: A colleague writes: When I was in NYC I went to this party by group of Japanese bio-scientists. There, one guy told me about how the biggest pharmaceutical company in Japan did their statistics. They ran 100 different tests and reported the most significant one. (This was in 2006 and he said they stopped doing this few years back so they were doing this until pretty recently…) I’m not sure if this was 100 multiple comparison or 100 different kinds of test but I’m sure they wouldn’t want to disclose their data… Ouch!
6 0.83846283 1624 andrew gelman stats-2012-12-15-New prize on causality in statstistics education
7 0.83725947 2278 andrew gelman stats-2014-04-01-Association for Psychological Science announces a new journal
9 0.82623124 1541 andrew gelman stats-2012-10-19-Statistical discrimination again
10 0.79968035 945 andrew gelman stats-2011-10-06-W’man < W’pedia, again
11 0.79152822 133 andrew gelman stats-2010-07-08-Gratuitous use of “Bayesian Statistics,” a branding issue?
12 0.79112172 1794 andrew gelman stats-2013-04-09-My talks in DC and Baltimore this week
13 0.77881676 1908 andrew gelman stats-2013-06-21-Interpreting interactions in discrete-data regression
14 0.76730216 1800 andrew gelman stats-2013-04-12-Too tired to mock
15 0.74779743 762 andrew gelman stats-2011-06-13-How should journals handle replication studies?
16 0.73142254 1833 andrew gelman stats-2013-04-30-“Tragedy of the science-communication commons”
17 0.72380853 1998 andrew gelman stats-2013-08-25-A new Bem theory
18 0.70238012 1499 andrew gelman stats-2012-09-16-Uri Simonsohn is speaking at Columbia tomorrow (Mon)
19 0.69931448 1393 andrew gelman stats-2012-06-26-The reverse-journal-submission system
20 0.69841444 1888 andrew gelman stats-2013-06-08-New Judea Pearl journal of causal inference