andrew_gelman_stats andrew_gelman_stats-2013 andrew_gelman_stats-2013-1844 knowledge-graph by maker-knowledge-mining
Source: html
Introduction: Social science research has been getting pretty bad press recently, what with the Excel buccaneers who didn’t know how to handle data with different numbers of observations per country, and the psychologist who published dozens of papers based on fabricated data, and the Evilicious guy who wouldn’t let people review his data tapes, etc etc. And that’s not even considering Dr. Anil Potti. On the other hand, the revelation of all these problems can be taken as evidence that things are getting better. Psychology researcher Gary Marcus writes : There is something positive that has come out of the crisis of replicability—something vitally important for all experimental sciences. For years, it was extremely difficult to publish a direct replication, or a failure to replicate an experiment, in a good journal. . . . Now, happily, the scientific culture has changed. . . . The Reproducibility Project, from the Center for Open Science is now underway . . . And sociologist Fabio Rojas
sentIndex sentText sentNum sentScore
1 For years, it was extremely difficult to publish a direct replication, or a failure to replicate an experiment, in a good journal. [sent-6, score-0.251]
2 I agree with Marcus and Rojas that attention to problems of replication is a good thing. [sent-25, score-0.356]
3 It’s bad that people are running incompetent analysis or faking data all over the place, but it’s good that they’re getting caught. [sent-26, score-0.238]
4 And, to the extent that scientific practices are improving to help detect error and fraud, and to reduce the incentives for publishing erroneous and fradulent results in the first place, that’s good too. [sent-27, score-0.275]
5 I’m sure my letter was indeed not in the top 10% of submissions, but the journal’s attitude presents a serious problem, if the bar to publication of a correction is so high. [sent-35, score-0.383]
6 That’s a disincentive for the journal to publish corrections, a disincentive for outsiders such as myself to write corrections, and a disincentive for researchers to be careful in the first place. [sent-36, score-0.636]
7 Just to be clear: I’m not complaining how I was treated here; rather, I’m griping about the system in which a known error can stand uncorrected in a top journal, just because nobody managed to send in a correction that’s in the top 10% of journal submissions. [sent-37, score-0.429]
8 ” Not so fast: It was over two years before those economists shared the data that allowed people to find the problems in their study. [sent-40, score-0.328]
9 If the system really worked, people wouldn’t have had to struggle for years to try to replicate an unreplicable analysis. [sent-41, score-0.248]
10 Reinhardt and Rogoff also made serious mistakes handling their time-series cross-sectional data. [sent-43, score-0.165]
11 Thomas Basbøll analogizes the difficulties of publishing scientific criticism to problems with the subprime mortgage market before the crash. [sent-53, score-0.805]
12 You could buy them or not buy them but you couldn’t bet explicitly against them; the market for subprime mortages simply had no place for people in it who took a dim view of them. [sent-55, score-0.921]
13 You might know with certainty that the entire mortgage bond market was doomed, but you could do nothing about it. [sent-56, score-0.449]
14 I’ve been trying to “bet against” a number of stories that have been told in the organization studies literature for years now, and the thing I’m learning is that there’s no place in the literature for people who take a dim view of them. [sent-58, score-0.332]
15 In a sense, you can buy the stories people are telling you or not buy them but you can’t criticize them. [sent-61, score-0.314]
16 The mortgage bond market was an evangelical environment in which to hold beliefs about housing prices, default rates, and credit ratings on CDOs. [sent-65, score-0.542]
17 Eventually, as Lewis reports, people were able to bet against the subprime mortgage market, but it wasn’t easy. [sent-69, score-0.495]
18 Marcus’s suggestions on cleaning up science are good ones, and we have a ways to go before they are generally implemented. [sent-72, score-0.175]
19 Leek is making the valid point that the sort of doomsaying that has been needed to draw attention to problems in scientific communication and to motivate improvements, can also be used, in guilt-by-association sense, to disparage good science. [sent-79, score-0.357]
20 Sure, vaccine deniers and global warming deniers and all the other deniers are out there, but it’s not like the 70s when people were buying millions of copies of Chariots of the Gods, The Jupiter Effect, and The Bermuda Triangle, right? [sent-81, score-0.63]
wordName wordTfidf (topN-words)
[('marcus', 0.253), ('leek', 0.207), ('deniers', 0.176), ('problems', 0.161), ('mortgage', 0.16), ('rojas', 0.16), ('bond', 0.145), ('disincentive', 0.145), ('market', 0.144), ('subprime', 0.136), ('scientific', 0.125), ('letter', 0.119), ('publish', 0.115), ('corrections', 0.111), ('jager', 0.107), ('buy', 0.106), ('science', 0.104), ('people', 0.102), ('correction', 0.102), ('reinhardt', 0.101), ('basb', 0.098), ('bet', 0.097), ('evangelical', 0.093), ('lewis', 0.088), ('dim', 0.088), ('rogoff', 0.086), ('journal', 0.086), ('culture', 0.086), ('mistakes', 0.083), ('serious', 0.082), ('faith', 0.081), ('system', 0.081), ('top', 0.08), ('publishing', 0.079), ('place', 0.078), ('submissions', 0.075), ('excel', 0.073), ('uri', 0.071), ('simonsohn', 0.071), ('good', 0.071), ('obtained', 0.067), ('data', 0.065), ('replicate', 0.065), ('view', 0.064), ('agree', 0.063), ('social', 0.061), ('replication', 0.061), ('held', 0.06), ('current', 0.059), ('systematic', 0.059)]
simIndex simValue blogId blogTitle
same-blog 1 1.0000006 1844 andrew gelman stats-2013-05-06-Against optimism about social science
Introduction: Social science research has been getting pretty bad press recently, what with the Excel buccaneers who didn’t know how to handle data with different numbers of observations per country, and the psychologist who published dozens of papers based on fabricated data, and the Evilicious guy who wouldn’t let people review his data tapes, etc etc. And that’s not even considering Dr. Anil Potti. On the other hand, the revelation of all these problems can be taken as evidence that things are getting better. Psychology researcher Gary Marcus writes : There is something positive that has come out of the crisis of replicability—something vitally important for all experimental sciences. For years, it was extremely difficult to publish a direct replication, or a failure to replicate an experiment, in a good journal. . . . Now, happily, the scientific culture has changed. . . . The Reproducibility Project, from the Center for Open Science is now underway . . . And sociologist Fabio Rojas
Introduction: Jeff Leek points to a post by Alex Holcombe, who disputes the idea that science is self-correcting. Holcombe writes [scroll down to get to his part]: The pace of scientific production has quickened, and self-correction has suffered. Findings that might correct old results are considered less interesting than results from more original research questions. Potential corrections are also more contested. As the competition for space in prestigious journals has become increasingly frenzied, doing and publishing studies that would confirm the rapidly accumulating new discoveries, or would correct them, became a losing proposition. Holcombe picks up on some points that we’ve discussed a lot here in the past year. Here’s Holcombe: In certain subfields, almost all new work appears in only a very few journals, all associated with a single professional society. There is then no way around the senior gatekeepers, who may then suppress corrections with impunity. . . . The bias agai
Introduction: Jeff Leek just posted the discussions of his paper (with Leah Jager), “An estimate of the science-wise false discovery rate and application to the top medical literature,” along with some further comments of his own. Here are my original thoughts on an earlier version of their article. Keith O’Rourke and I expanded these thoughts into a formal comment for the journal. We’re pretty much in agreement with John Ioannidis (you can find his discussion in the top link above). In quick summary, I agree with Jager and Leek that this is an important topic. I think there are two key places where Keith and I disagree with them: 1. They take published p-values at face value whereas we consider them as the result of a complicated process of selection. This is something I didn’t used to think much about, but now I’ve become increasingly convinced that the problems with published p-values is not a simple file-drawer effect or the case of a few p=0.051 values nudged toward p=0.049, bu
4 0.21661577 1878 andrew gelman stats-2013-05-31-How to fix the tabloids? Toward replicable social science research
Introduction: This seems to be the topic of the week. Yesterday I posted on the sister blog some further thoughts on those “Psychological Science” papers on menstrual cycles, biceps size, and political attitudes, tied to a horrible press release from the journal Psychological Science hyping the biceps and politics study. Then I was pointed to these suggestions from Richard Lucas and M. Brent Donnellan have on improving the replicability and reproducibility of research published in the Journal of Research in Personality: It goes without saying that editors of scientific journals strive to publish research that is not only theoretically interesting but also methodologically rigorous. The goal is to select papers that advance the field. Accordingly, editors want to publish findings that can be reproduced and replicated by other scientists. Unfortunately, there has been a recent “crisis in confidence” among psychologists about the quality of psychological research (Pashler & Wagenmakers, 2012)
Introduction: I had a brief email exchange with Jeff Leek regarding our recent discussions of replication, criticism, and the self-correcting process of science. Jeff writes: (1) I can see the problem with serious, evidence-based criticisms not being published in the same journal (and linked to) studies that are shown to be incorrect. I have been mostly seeing these sorts of things show up in blogs. But I’m not sure that is a bad thing. I think people read blogs more than they read the literature. I wonder if this means that blogs will eventually be a sort of “shadow literature”? (2) I think there is a ton of bad literature out there, just like there is a ton of bad stuff on Google. If we focus too much on the bad stuff we will be paralyzed. I still manage to find good papers despite all the bad papers. (3) I think one positive solution to this problem is to incentivize/publish referee reports and give people credit for a good referee report just like they get credit for a good paper. T
6 0.18561548 1842 andrew gelman stats-2013-05-05-Cleaning up science
7 0.1845403 2179 andrew gelman stats-2014-01-20-The AAA Tranche of Subprime Science
8 0.18440883 2245 andrew gelman stats-2014-03-12-More on publishing in journals
10 0.18103032 1139 andrew gelman stats-2012-01-26-Suggested resolution of the Bem paradox
11 0.17898372 2353 andrew gelman stats-2014-05-30-I posted this as a comment on a sociology blog
12 0.17343237 1282 andrew gelman stats-2012-04-26-Bad news about (some) statisticians
13 0.17075098 2269 andrew gelman stats-2014-03-27-Beyond the Valley of the Trolls
14 0.15970132 2137 andrew gelman stats-2013-12-17-Replication backlash
15 0.1534459 2244 andrew gelman stats-2014-03-11-What if I were to stop publishing in journals?
16 0.15127538 1435 andrew gelman stats-2012-07-30-Retracted articles and unethical behavior in economics journals?
17 0.15055375 2006 andrew gelman stats-2013-09-03-Evaluating evidence from published research
18 0.14940064 1928 andrew gelman stats-2013-07-06-How to think about papers published in low-grade journals?
19 0.13231173 1272 andrew gelman stats-2012-04-20-More proposals to reform the peer-review system
topicId topicWeight
[(0, 0.297), (1, -0.101), (2, -0.078), (3, -0.136), (4, -0.087), (5, -0.066), (6, 0.002), (7, -0.09), (8, -0.033), (9, 0.023), (10, 0.024), (11, 0.006), (12, -0.072), (13, 0.003), (14, -0.045), (15, -0.036), (16, 0.008), (17, -0.021), (18, 0.004), (19, -0.026), (20, -0.031), (21, 0.026), (22, -0.049), (23, 0.026), (24, -0.081), (25, 0.015), (26, 0.034), (27, 0.011), (28, -0.007), (29, 0.04), (30, -0.01), (31, -0.01), (32, -0.013), (33, -0.027), (34, 0.034), (35, 0.002), (36, -0.045), (37, -0.003), (38, -0.021), (39, 0.002), (40, 0.017), (41, 0.006), (42, -0.028), (43, 0.016), (44, -0.029), (45, 0.056), (46, -0.016), (47, 0.041), (48, 0.004), (49, -0.004)]
simIndex simValue blogId blogTitle
same-blog 1 0.97558385 1844 andrew gelman stats-2013-05-06-Against optimism about social science
Introduction: Social science research has been getting pretty bad press recently, what with the Excel buccaneers who didn’t know how to handle data with different numbers of observations per country, and the psychologist who published dozens of papers based on fabricated data, and the Evilicious guy who wouldn’t let people review his data tapes, etc etc. And that’s not even considering Dr. Anil Potti. On the other hand, the revelation of all these problems can be taken as evidence that things are getting better. Psychology researcher Gary Marcus writes : There is something positive that has come out of the crisis of replicability—something vitally important for all experimental sciences. For years, it was extremely difficult to publish a direct replication, or a failure to replicate an experiment, in a good journal. . . . Now, happily, the scientific culture has changed. . . . The Reproducibility Project, from the Center for Open Science is now underway . . . And sociologist Fabio Rojas
Introduction: Jeff Leek points to a post by Alex Holcombe, who disputes the idea that science is self-correcting. Holcombe writes [scroll down to get to his part]: The pace of scientific production has quickened, and self-correction has suffered. Findings that might correct old results are considered less interesting than results from more original research questions. Potential corrections are also more contested. As the competition for space in prestigious journals has become increasingly frenzied, doing and publishing studies that would confirm the rapidly accumulating new discoveries, or would correct them, became a losing proposition. Holcombe picks up on some points that we’ve discussed a lot here in the past year. Here’s Holcombe: In certain subfields, almost all new work appears in only a very few journals, all associated with a single professional society. There is then no way around the senior gatekeepers, who may then suppress corrections with impunity. . . . The bias agai
3 0.91147017 2137 andrew gelman stats-2013-12-17-Replication backlash
Introduction: Raghuveer Parthasarathy pointed me to an article in Nature by Mina Bissell, who writes , “The push to replicate findings could shelve promising research and unfairly damage the reputations of careful, meticulous scientists.” I can see where she’s coming from: if you work hard day after day in the lab, it’s gotta be a bit frustrating to find all your work questioned, for the frauds of the Dr. Anil Pottis and Diederik Stapels to be treated as a reason for everyone else’s work to be considered guilty until proven innocent. That said, I pretty much disagree with Bissell’s article, and really the best thing I can say about it is that I think it’s a good sign that the push for replication is so strong that now there’s a backlash against it. Traditionally, leading scientists have been able to simply ignore the push for replication. If they are feeling that the replication movement is strong enough that they need to fight it, that to me is good news. I’ll explain a bit in the conte
4 0.90421307 1139 andrew gelman stats-2012-01-26-Suggested resolution of the Bem paradox
Introduction: There has been an increasing discussion about the proliferation of flawed research in psychology and medicine, with some landmark events being John Ioannides’s article , “Why most published research findings are false” (according to Google Scholar, cited 973 times since its appearance in 2005), the scandals of Marc Hauser and Diederik Stapel, two leading psychology professors who resigned after disclosures of scientific misconduct, and Daryl Bem’s dubious recent paper on ESP, published to much fanfare in Journal of Personality and Social Psychology, one of the top journals in the field. Alongside all this are the plagiarism scandals, which are uninteresting from a scientific context but are relevant in that, in many cases, neither the institutions housing the plagiarists nor the editors and publishers of the plagiarized material seem to care. Perhaps these universities and publishers are more worried about bad publicity (and maybe lawsuits, given that many of the plagiarism cas
5 0.88591063 1878 andrew gelman stats-2013-05-31-How to fix the tabloids? Toward replicable social science research
Introduction: This seems to be the topic of the week. Yesterday I posted on the sister blog some further thoughts on those “Psychological Science” papers on menstrual cycles, biceps size, and political attitudes, tied to a horrible press release from the journal Psychological Science hyping the biceps and politics study. Then I was pointed to these suggestions from Richard Lucas and M. Brent Donnellan have on improving the replicability and reproducibility of research published in the Journal of Research in Personality: It goes without saying that editors of scientific journals strive to publish research that is not only theoretically interesting but also methodologically rigorous. The goal is to select papers that advance the field. Accordingly, editors want to publish findings that can be reproduced and replicated by other scientists. Unfortunately, there has been a recent “crisis in confidence” among psychologists about the quality of psychological research (Pashler & Wagenmakers, 2012)
7 0.8798694 2179 andrew gelman stats-2014-01-20-The AAA Tranche of Subprime Science
9 0.86508298 2269 andrew gelman stats-2014-03-27-Beyond the Valley of the Trolls
10 0.86483032 2006 andrew gelman stats-2013-09-03-Evaluating evidence from published research
11 0.85764825 2353 andrew gelman stats-2014-05-30-I posted this as a comment on a sociology blog
12 0.8542695 1273 andrew gelman stats-2012-04-20-Proposals for alternative review systems for scientific work
13 0.8395279 1435 andrew gelman stats-2012-07-30-Retracted articles and unethical behavior in economics journals?
14 0.83196098 2177 andrew gelman stats-2014-01-19-“The British amateur who debunked the mathematics of happiness”
15 0.8313843 1683 andrew gelman stats-2013-01-19-“Confirmation, on the other hand, is not sexy”
16 0.82975399 1626 andrew gelman stats-2012-12-16-The lamest, grudgingest, non-retraction retraction ever
17 0.82876635 2361 andrew gelman stats-2014-06-06-Hurricanes vs. Himmicanes
18 0.82680827 1998 andrew gelman stats-2013-08-25-A new Bem theory
19 0.82177705 1860 andrew gelman stats-2013-05-17-How can statisticians help psychologists do their research better?
20 0.81359529 2215 andrew gelman stats-2014-02-17-The Washington Post reprints university press releases without editing them
topicId topicWeight
[(9, 0.017), (12, 0.016), (15, 0.051), (16, 0.102), (21, 0.03), (24, 0.116), (34, 0.017), (42, 0.144), (47, 0.013), (59, 0.014), (63, 0.02), (76, 0.022), (86, 0.018), (99, 0.287)]
simIndex simValue blogId blogTitle
1 0.96985728 60 andrew gelman stats-2010-05-30-What Auteur Theory and Freshwater Economics have in common
Introduction: Mark Palko writes : We’ll define freshwater economics as the theory that economic behavior (and perhaps most non-economic behavior) can be explained using the concepts of rational actors and efficient markets and auteur theory as the idea that most films (particularly great films) represent the artistic vision of a single author (almost always the director) and the best way to approach one of those films is through the body of work of its author. Both of these definitions are oversimplified and a bit unfair but they will get the discussion started. . . . Compared to their nearest neighbors, film criticism and economics (particularly macroeconomics) are both difficult, messy fields. Films are collaborative efforts where individual contributions defy attribution and creative decisions often can’t be distinguished from accidents of filming. Worse yet, most films are the product of large corporations which means that dozens of VPs and executives might have played a role (sometimes
2 0.96588039 1775 andrew gelman stats-2013-03-23-In which I disagree with John Maynard Keynes
Introduction: In his review in 1938 of Historical Development of the Graphical Representation of Statistical Data , by H. Gray Funkhauser, for The Economic Journal , the great economist writes: Perhaps the most striking outcome of Mr. Funkhouser’s researches is the fact of the very slow progress which graphical methods made until quite recently. . . . In the first fifty volumes of the Statistical Journal, 1837-87, only fourteen graphs are printed altogether. It is surprising to be told that Laplace never drew a graph of the normal law of error . . . Edgeworth made no use of statistical charts as distinct from mathematical diagrams. Apart from Quetelet and Jevons, the most important influences were probably those of Galton and of Mulhall’s Dictionary, first published in 1884. Galton was indeed following his father and grandfather in this field, but his pioneer work was mainly restricted to meteorological maps, and he did not contribute to the development of the graphical representation of ec
3 0.96541113 492 andrew gelman stats-2010-12-30-That puzzle-solving feeling
Introduction: Since this blog in November, I’ve given my talk on infovis vs. statistical graphics about five times: once in person (at the visualization meetup in NYC, a blog away from Num Pang!) and the rest via telephone conferencing or skype. The live presentation was best, but the remote talks have been improving, and I’m looking forward to doing more of these in the future to save time and reduce pollution. Here are the powerpoints of the talk. Now that I’ve got it working well (mostly by cutting lots of words on the slides), my next step will be to improve the interactive experience. At the very least, I need to allocate time after the talk for discussion. People usually don’t ask a lot of questions when I speak, so maybe the best strategy is to allow a half hour following the talk for people to speak with me individually. It could be set up so that I’m talking with one person but the others who are hanging out could hear the conversation too. Anyway, one of the times I gave th
4 0.96390629 483 andrew gelman stats-2010-12-23-Science, ideology, and human origins
Introduction: A link from Tyler Cowen led me to this long blog article by Razib Khan, discussing some recent genetic findings on human origins in the context of the past twenty-five years of research and popularization of science. I don’t know much about human origins (beyond my ooh-that’s-cool reactions to exhibits at the Natural History Museum, my general statistician’s skepticism at various over-the-top claims I’ve heard over the years about “mitochondrial Eve” and the like, and various bits I’ve read over the years regarding when people came over to Australia, America, etc.), but what particularly interested me about Khan’s article was his discussion about the various controversies among scientists, his own reactions when reading and thinking about these issues as they were happening (Khan was a student at the time), and the interaction between science and political ideology. There’s a limit to how far you can go with this sort of cultural criticism of science, and Khan realizes this
5 0.96190077 713 andrew gelman stats-2011-05-15-1-2 social scientist + 1-2 politician = ???
Introduction: A couple things in this interview by Andrew Goldman of Larry Summers currently irritated me. I’ll give the quotes and then explain my annoyance. 1. Goldman: What would the economy look like now if $1.2 trillion had been spent? Summers: I think it’s an artificial question because there would have been all kinds of problems in actually moving $1.2 trillion dollars through the system — finding enough bridge projects that were ready to go and the like. But the recovery probably would have proceeded more rapidly if the fiscal program had been larger. . . . 2. Goldman: You’re aware of — and were making light of — the fact that you occasionally rub people the wrong way. Summers: In meetings, I’m more focused on trying to figure out what the right answer is than making everybody feel validated. In Washington and at Harvard, that sometimes rubs people the wrong way. OK, now my reactions: 1. Not enough bridge projects, huh? I don’t believe it. We’ve been hearing fo
6 0.96182144 1060 andrew gelman stats-2011-12-15-Freakonomics: What went wrong?
8 0.95686448 746 andrew gelman stats-2011-06-05-An unexpected benefit of Arrow’s other theorem
same-blog 10 0.95361936 1844 andrew gelman stats-2013-05-06-Against optimism about social science
11 0.95081943 1936 andrew gelman stats-2013-07-13-Economic policy does not occur in a political vacuum
12 0.9465546 590 andrew gelman stats-2011-02-25-Good introductory book for statistical computation?
13 0.94651157 2323 andrew gelman stats-2014-05-07-Cause he thinks he’s so-phisticated
14 0.94385284 1535 andrew gelman stats-2012-10-16-Bayesian analogue to stepwise regression?
15 0.94293636 111 andrew gelman stats-2010-06-26-Tough love as a style of writing
16 0.94285709 117 andrew gelman stats-2010-06-29-Ya don’t know Bayes, Jack
17 0.94139159 2179 andrew gelman stats-2014-01-20-The AAA Tranche of Subprime Science
18 0.94098729 943 andrew gelman stats-2011-10-04-Flip it around
19 0.94024086 1921 andrew gelman stats-2013-07-01-Going meta on Niall Ferguson
20 0.93964869 1138 andrew gelman stats-2012-01-25-Chris Schmid on Evidence Based Medicine