andrew_gelman_stats andrew_gelman_stats-2013 andrew_gelman_stats-2013-1860 knowledge-graph by maker-knowledge-mining
Source: html
Introduction: I received two emails yesterday on related topics. First, Stephen Olivier pointed me to this post by Daniel Lakens, who wrote the following open call to statisticians: You would think that if you are passionate about statistics, then you want to help people to calculate them correctly in any way you can. . . . you’d think some statisticians would be interested in helping a poor mathematically challenged psychologist out by offering some practical advice. I’m the right person to ask this question, since I actually have written a lot of material that helps psychologists (and others) with their data analysis. But there clearly are communication difficulties, in that my work and that of other statisticians hasn’t reached Lakens. Sometimes the contributions of statisticians are made indirectly. For example, I wrote Bayesian Data Analysis, and then Kruschke wrote Doing Bayesian Data Analysis. Our statistics book made it possible for Kruschke to write his excellent book for psycholo
sentIndex sentText sentNum sentScore
1 The alternative conception of psychology is the “ everything is obvious – once you know the answer . [sent-27, score-0.242]
2 ” The relevance for the present discussion is that this paper was published in Psychological Science, a top journal in psychology. [sent-43, score-0.333]
3 Here’s the abstract: Each month many women experience an ovulatory cycle that regulates fertility. [sent-44, score-0.805]
4 Whereas research finds that this cycle influences women’s mating preferences, we propose that it might also change women’s political and religious views. [sent-45, score-0.508]
5 Building on theory suggesting that political and religious orientation are linked to reproductive goals, we tested how fertility influenced women’s politics, religiosity, and voting in the 2012 U. [sent-46, score-0.332]
6 In two studies with large and diverse samples, ovulation had drastically different effects on single versus married women. [sent-49, score-0.464]
7 Ovulation led single women to become more liberal, less religious, and more likely to vote for Barack Obama. [sent-50, score-0.336]
8 In contrast, ovulation led married women to become more conservative, more religious, and more likely to vote for Mitt Romney. [sent-51, score-0.672]
9 Overall, the ovulatory cycle not only influences women’s politics, but appears to do so differently for single versus married women. [sent-53, score-0.775]
10 I took a look at the paper , and what I found was a bunch of comparisons and p-values, some of which were statistically significant, and then lots of stories. [sent-54, score-0.26]
11 For example, they report that, among women in relationships, 40% in the ovulation period supported Romney, compared to 23% in the non-fertile part of their cycle. [sent-59, score-0.492]
12 A statistician offers helpful advice for psychology researchers My real goal here is to address the question that was brought up at the beginning of this post: What recommendations can I, as a statistician, give to psychology researchers? [sent-68, score-0.513]
13 Here are a few, presented in the context of the paper on ovulation and political attitudes: 1. [sent-69, score-0.308]
14 (“We also did not include women at the beginning of the ovulatory cycle (cycle days 1–6) or at the very end of the ovulatory cycle (cycle days 26–28) to avoid potential confounds due to premenstrual or menstrual symptoms. [sent-72, score-1.357]
15 For example, I would’ve liked to see a comparison of respondents in different parts of their cycle on variables such as birth year, party identification, marital status, etc etc. [sent-80, score-0.329]
16 The researchers asked a few survey questions to a bunch of people on Mechanical Turk, then did who knows how many comparisons and significance tests, reported some subset of the results, then went on to story time. [sent-95, score-0.357]
17 ” As a statistician, my advice is: if a paper is nothing special, you don’t have to publish it in your flagship journal. [sent-104, score-0.36]
18 In this as with the notorious Daryl Bem article, I feel that the journal almost seemed to feel an obligation to publish a dubious claim, just because some referees didn’t happen to find any flaws in the data collection or analysis. [sent-105, score-0.492]
19 I’m giving some very specific suggestions that you, the psychology researcher, can use in your next research project (or, if you’re a journal editor, in your next publication decision). [sent-111, score-0.264]
20 There is, of course, lots and lots of additional advice that I and other statisticians could give. [sent-112, score-0.307]
wordName wordTfidf (topN-words)
[('cycle', 0.329), ('women', 0.253), ('ovulation', 0.239), ('ovulatory', 0.223), ('olivier', 0.142), ('journal', 0.136), ('wtf', 0.134), ('psychology', 0.128), ('conception', 0.114), ('religious', 0.108), ('publish', 0.101), ('advice', 0.101), ('researchers', 0.1), ('married', 0.097), ('statisticians', 0.096), ('psychological', 0.092), ('voting', 0.09), ('hormones', 0.089), ('flagship', 0.089), ('bogus', 0.085), ('kruschke', 0.085), ('durante', 0.085), ('vote', 0.083), ('comparisons', 0.08), ('noise', 0.078), ('sea', 0.078), ('data', 0.077), ('effects', 0.073), ('influences', 0.071), ('top', 0.07), ('paper', 0.069), ('orientation', 0.068), ('influenced', 0.066), ('referees', 0.062), ('politics', 0.062), ('reported', 0.062), ('tests', 0.062), ('kahan', 0.062), ('female', 0.059), ('subset', 0.059), ('feel', 0.058), ('present', 0.058), ('preferences', 0.057), ('bunch', 0.056), ('statistician', 0.056), ('lots', 0.055), ('versus', 0.055), ('stephen', 0.055), ('psychologists', 0.054), ('study', 0.053)]
simIndex simValue blogId blogTitle
same-blog 1 0.99999952 1860 andrew gelman stats-2013-05-17-How can statisticians help psychologists do their research better?
Introduction: I received two emails yesterday on related topics. First, Stephen Olivier pointed me to this post by Daniel Lakens, who wrote the following open call to statisticians: You would think that if you are passionate about statistics, then you want to help people to calculate them correctly in any way you can. . . . you’d think some statisticians would be interested in helping a poor mathematically challenged psychologist out by offering some practical advice. I’m the right person to ask this question, since I actually have written a lot of material that helps psychologists (and others) with their data analysis. But there clearly are communication difficulties, in that my work and that of other statisticians hasn’t reached Lakens. Sometimes the contributions of statisticians are made indirectly. For example, I wrote Bayesian Data Analysis, and then Kruschke wrote Doing Bayesian Data Analysis. Our statistics book made it possible for Kruschke to write his excellent book for psycholo
2 0.285945 2236 andrew gelman stats-2014-03-07-Selection bias in the reporting of shaky research
Introduction: I’ll reorder this week’s posts a bit in order to continue on a topic that came up yesterday. A couple days ago a reporter wrote to me asking what I thought of this paper on Money, Status, and the Ovulatory Cycle. I responded: Given the quality of the earlier paper by these researchers, I’m not inclined to believe anything these people write. But, to be specific, I can point out some things: - The authors define low fertility as days 8-14. Oddly enough, these authors in their earlier paper used days 7-14. But according to womenshealth.gov, the most fertile days are between days 10 and 17. The choice of these days affects their analysis, and it is not a good sign that they use different days in different papers. (see more on this point in sections 2.3 and 3.1 of this paper: http://www.stat.columbia.edu/~gelman/research/unpublished/p_hacking.pdf) - They perform a lot of different analyses, and many others could be performed. For example, “Study 1 indicates that ovul
Introduction: In my article about implausible p-values in psychology studies, I wrote: “Women Are More Likely to Wear Red or Pink at Peak Fertility,” by Alec Beall and Jessica Tracy, is based on two samples: a self-selected sample of 100 women from the Internet, and 24 undergraduates at the University of British Columbia. . . . [There is a problem with] representativeness. What color clothing you wear has a lot to do with where you live and who you hang out with. Participants in an Internet survey and University of British Columbia students aren’t particularly representative of much more than … participants in an Internet survey and University of British Columbia students. In response, I received this in an email from a prominent psychology researcher (not someone I know personally): Complaining that subjects in an experiment were not randomly sampled is what freshmen do before they take their first psychology class. I really *hope* you why that is an absurd criticism – especially of au
Introduction: The other day we discussed that paper on ovulation and voting (you may recall that the authors reported a scattered bunch of comparisons, significance tests, and p-values, and I recommended that they would’ve done better to simply report complete summaries of their data, so that readers could see the comparisons of interest in full context), and I was thinking a bit more about why I was so bothered that it was published in Psychological Science, which I’d thought of as a serious research journal. My concern isn’t just that that the paper is bad—after all, lots of bad papers get published—but rather that it had nothing really going for it, except that it was headline bait. It was a survey done on Mechanical Turk, that’s it. No clever design, no clever questions, no care in dealing with nonresponse problems, no innovative data analysis, no nothing. The paper had nothing to offer, except that it had no obvious flaws. Psychology is a huge field full of brilliant researchers.
Introduction: I had a brief email exchange with Jeff Leek regarding our recent discussions of replication, criticism, and the self-correcting process of science. Jeff writes: (1) I can see the problem with serious, evidence-based criticisms not being published in the same journal (and linked to) studies that are shown to be incorrect. I have been mostly seeing these sorts of things show up in blogs. But I’m not sure that is a bad thing. I think people read blogs more than they read the literature. I wonder if this means that blogs will eventually be a sort of “shadow literature”? (2) I think there is a ton of bad literature out there, just like there is a ton of bad stuff on Google. If we focus too much on the bad stuff we will be paralyzed. I still manage to find good papers despite all the bad papers. (3) I think one positive solution to this problem is to incentivize/publish referee reports and give people credit for a good referee report just like they get credit for a good paper. T
7 0.17954127 1139 andrew gelman stats-2012-01-26-Suggested resolution of the Bem paradox
8 0.1636577 1435 andrew gelman stats-2012-07-30-Retracted articles and unethical behavior in economics journals?
9 0.16212758 2245 andrew gelman stats-2014-03-12-More on publishing in journals
10 0.16065229 1954 andrew gelman stats-2013-07-24-Too Good To Be True: The Scientific Mass Production of Spurious Statistical Significance
12 0.15989186 2336 andrew gelman stats-2014-05-16-How much can we learn about individual-level causal claims from state-level correlations?
13 0.15740111 1878 andrew gelman stats-2013-05-31-How to fix the tabloids? Toward replicable social science research
15 0.15196691 2278 andrew gelman stats-2014-04-01-Association for Psychological Science announces a new journal
16 0.15172341 2255 andrew gelman stats-2014-03-19-How Americans vote
17 0.13966148 1226 andrew gelman stats-2012-03-22-Story time meets the all-else-equal fallacy and the fallacy of measurement
19 0.13429266 2006 andrew gelman stats-2013-09-03-Evaluating evidence from published research
20 0.13412596 1928 andrew gelman stats-2013-07-06-How to think about papers published in low-grade journals?
topicId topicWeight
[(0, 0.311), (1, -0.087), (2, 0.02), (3, -0.145), (4, -0.073), (5, -0.06), (6, -0.07), (7, -0.051), (8, -0.042), (9, -0.011), (10, 0.052), (11, 0.023), (12, -0.011), (13, -0.03), (14, 0.057), (15, -0.014), (16, 0.01), (17, -0.014), (18, 0.007), (19, -0.007), (20, -0.023), (21, -0.026), (22, -0.006), (23, -0.037), (24, -0.025), (25, 0.016), (26, 0.005), (27, -0.049), (28, -0.002), (29, -0.035), (30, -0.035), (31, -0.007), (32, -0.02), (33, -0.023), (34, 0.012), (35, 0.002), (36, -0.014), (37, 0.019), (38, -0.126), (39, -0.011), (40, -0.013), (41, 0.006), (42, 0.025), (43, -0.015), (44, -0.011), (45, -0.023), (46, 0.012), (47, 0.034), (48, -0.031), (49, 0.011)]
simIndex simValue blogId blogTitle
same-blog 1 0.9699477 1860 andrew gelman stats-2013-05-17-How can statisticians help psychologists do their research better?
Introduction: I received two emails yesterday on related topics. First, Stephen Olivier pointed me to this post by Daniel Lakens, who wrote the following open call to statisticians: You would think that if you are passionate about statistics, then you want to help people to calculate them correctly in any way you can. . . . you’d think some statisticians would be interested in helping a poor mathematically challenged psychologist out by offering some practical advice. I’m the right person to ask this question, since I actually have written a lot of material that helps psychologists (and others) with their data analysis. But there clearly are communication difficulties, in that my work and that of other statisticians hasn’t reached Lakens. Sometimes the contributions of statisticians are made indirectly. For example, I wrote Bayesian Data Analysis, and then Kruschke wrote Doing Bayesian Data Analysis. Our statistics book made it possible for Kruschke to write his excellent book for psycholo
2 0.88738346 2236 andrew gelman stats-2014-03-07-Selection bias in the reporting of shaky research
Introduction: I’ll reorder this week’s posts a bit in order to continue on a topic that came up yesterday. A couple days ago a reporter wrote to me asking what I thought of this paper on Money, Status, and the Ovulatory Cycle. I responded: Given the quality of the earlier paper by these researchers, I’m not inclined to believe anything these people write. But, to be specific, I can point out some things: - The authors define low fertility as days 8-14. Oddly enough, these authors in their earlier paper used days 7-14. But according to womenshealth.gov, the most fertile days are between days 10 and 17. The choice of these days affects their analysis, and it is not a good sign that they use different days in different papers. (see more on this point in sections 2.3 and 3.1 of this paper: http://www.stat.columbia.edu/~gelman/research/unpublished/p_hacking.pdf) - They perform a lot of different analyses, and many others could be performed. For example, “Study 1 indicates that ovul
Introduction: Last week I published in Slate a critique of a paper that appeared in the journal Psychological Science. That paper, by Alec Beall and Jessica Tracy, found that women who were at peak fertility were three times more likely to wear red or pink shirts, compared to women at other points in their menstrual cycles. The study was based an 100 participants on the internet and 24 college students. In my critique, I argued that we had no reason to believe the results generalized to the larger population, because (1) the samples were not representative, (2) the measurements were noisy, (3) the researchers did not use the correct dates of peak fertility, and (4) there were many different comparisons that could have been reported in the data, so there was nothing special about a particular comparison being statistically significant. I likened their paper to other work which I considered flawed for multiple comparisons (too many researcher degrees of freedom), including a claimed relation bet
Introduction: Paul Alper writes: Unless I missed it, you haven’t commented on the recent article of Michael Bang Peterson [with Daniel Sznycer, Aaron Sell, Leda Cosmides, and John Tooby]. It seems to have been reviewed extensively in the lay press. A typical example is here . This review begins with “If you are physically strong, social science scholars believe they can predict whether or not you are more conservative than other men…Men’s upper-body strength predicts their political opinions on economic redistribution, they write, and they believe that the link may reflect psychological traits that evolved in response to our early ancestral environments and continue to influence behavior today. . . . they surveyed hundreds of people in America, Denmark and Argentina about bicep size, socioeconomic status, and support for economic redistribution.” Further, “Despite the fact that the United States, Denmark and Argentina have very different welfare systems, we still see that — at the psychol
5 0.85961974 1878 andrew gelman stats-2013-05-31-How to fix the tabloids? Toward replicable social science research
Introduction: This seems to be the topic of the week. Yesterday I posted on the sister blog some further thoughts on those “Psychological Science” papers on menstrual cycles, biceps size, and political attitudes, tied to a horrible press release from the journal Psychological Science hyping the biceps and politics study. Then I was pointed to these suggestions from Richard Lucas and M. Brent Donnellan have on improving the replicability and reproducibility of research published in the Journal of Research in Personality: It goes without saying that editors of scientific journals strive to publish research that is not only theoretically interesting but also methodologically rigorous. The goal is to select papers that advance the field. Accordingly, editors want to publish findings that can be reproduced and replicated by other scientists. Unfortunately, there has been a recent “crisis in confidence” among psychologists about the quality of psychological research (Pashler & Wagenmakers, 2012)
6 0.85179073 1139 andrew gelman stats-2012-01-26-Suggested resolution of the Bem paradox
7 0.84882277 2220 andrew gelman stats-2014-02-22-Quickies
8 0.84659219 2241 andrew gelman stats-2014-03-10-Preregistration: what’s in it for you?
11 0.83020431 1954 andrew gelman stats-2013-07-24-Too Good To Be True: The Scientific Mass Production of Spurious Statistical Significance
12 0.82516813 2137 andrew gelman stats-2013-12-17-Replication backlash
13 0.8207413 2361 andrew gelman stats-2014-06-06-Hurricanes vs. Himmicanes
14 0.81494313 1226 andrew gelman stats-2012-03-22-Story time meets the all-else-equal fallacy and the fallacy of measurement
15 0.8137508 1128 andrew gelman stats-2012-01-19-Sharon Begley: Worse than Stephen Jay Gould?
17 0.81065142 2004 andrew gelman stats-2013-09-01-Post-publication peer review: How it (sometimes) really works
18 0.80958974 1055 andrew gelman stats-2011-12-13-Data sharing update
topicId topicWeight
[(2, 0.013), (9, 0.017), (15, 0.07), (16, 0.058), (21, 0.036), (24, 0.161), (43, 0.12), (45, 0.02), (47, 0.011), (63, 0.012), (82, 0.01), (86, 0.019), (98, 0.012), (99, 0.293)]
simIndex simValue blogId blogTitle
Introduction: Matt Taibbi writes : Glenn Hubbard, Leading Academic and Mitt Romney Advisor, Took $1200 an Hour to Be Countrywide’s Expert Witness . . . Hidden among the reams of material recently filed in connection with the lawsuit of monoline insurer MBIA against Bank of America and Countrywide is a deposition of none other than Columbia University’s Glenn Hubbard. . . . Hubbard testified on behalf of Countrywide in the MBIA suit. He conducted an “analysis” that essentially concluded that Countrywide’s loans weren’t any worse than the loans produced by other mortgage originators, and that therefore the monstrous losses that investors in those loans suffered were due to other factors related to the economic crisis – and not caused by the serial misrepresentations and fraud in Countrywide’s underwriting. That’s interesting, because I worked on the other side of this case! I was hired by MBIA’s lawyers. It wouldn’t be polite of me to reveal my consulting rate, and I never actually got depose
2 0.97057641 1253 andrew gelman stats-2012-04-08-Technology speedup graph
Introduction: Dan Kahan sends along this awesome graph (click on the image to see the whole thing): and writes: I [Kahan] saw it at http://www.theatlantic.com/technology/archive/2012/04/the-100-year-march-of-technology-in-1-graph/255573/ , which misidentified the source (not “visual economics”; visualizingeconomics .com , which attributes it to Nicholas Felton , who apparently condensed this version , which I worry could cause a stroke). But it did have a good write-up that (I’m glad) caught my attention. It made me [Kahan] start to wonder about what sorts of qualities of a technology will influence its dissemination & also about the availability of benchmarks for proliferation of various sorts of things (e.g, fads & trends, health-promoting behaviors, knowledge of a scientific discovery) that one could use to gauge how meaningful the apparent increase in rates of proliferation of these technologies has been over time. That in turn made me wonder whether — indeed, suspect th
3 0.96871471 314 andrew gelman stats-2010-10-03-Disconnect between drug and medical device approval
Introduction: Sanjay Kaul wrotes: By statute (“the least burdensome” pathway), the approval standard for devices by the US FDA is lower than for drugs. Before a new drug can be marketed, the sponsor must show “substantial evidence of effectiveness” as based on two or more well-controlled clinical studies (which literally means 2 trials, each with a p value of <0.05, or 1 large trial with a robust p value <0.00125). In contrast, the sponsor of a new device, especially those that are designated as high-risk (Class III) device, need only demonstrate "substantial equivalence" to an FDA-approved device via the 510(k) exemption or a "reasonable assurance of safety and effectiveness", evaluated through a pre-market approval and typically based on a single study. What does “reasonable assurance” or “substantial equivalence” imply to you as a Bayesian? These are obviously qualitative constructs, but if one were to quantify them, how would you go about addressing it? The regulatory definitions for
same-blog 4 0.9652366 1860 andrew gelman stats-2013-05-17-How can statisticians help psychologists do their research better?
Introduction: I received two emails yesterday on related topics. First, Stephen Olivier pointed me to this post by Daniel Lakens, who wrote the following open call to statisticians: You would think that if you are passionate about statistics, then you want to help people to calculate them correctly in any way you can. . . . you’d think some statisticians would be interested in helping a poor mathematically challenged psychologist out by offering some practical advice. I’m the right person to ask this question, since I actually have written a lot of material that helps psychologists (and others) with their data analysis. But there clearly are communication difficulties, in that my work and that of other statisticians hasn’t reached Lakens. Sometimes the contributions of statisticians are made indirectly. For example, I wrote Bayesian Data Analysis, and then Kruschke wrote Doing Bayesian Data Analysis. Our statistics book made it possible for Kruschke to write his excellent book for psycholo
5 0.9642452 1347 andrew gelman stats-2012-05-27-Macromuddle
Introduction: More and more I feel like economics reporting is based on crude principles of adding up “good news” and “bad news.” Sometimes this makes sense: by almost any measure, an unemployment rate of 10% is bad news compared to an unemployment rate of 5%. Other times, though, the good/bad news framework seems so tangled. For example: house prices up is considered good news but inflation is considered bad news. A strong dollar is considered good news but it’s also an unfavorable exchange rate, which is bad news. When facebook shares go down, that’s bad news, but if they automatically go up, that means they were underpriced which doesn’t seem so good either. Pundits are torn between rooting for the euro to fail (which means our team (the U.S.) is better than Europe (their team)) and rooting for it to survive (because a collapse in Europe is bad news for the U.S. economy). China’s economy doing well is bad news—but if their economy slips, that’s bad news too. I think you get the picture
6 0.96210909 2330 andrew gelman stats-2014-05-12-Historical Arc of Universities
7 0.95865983 481 andrew gelman stats-2010-12-22-The Jumpstart financial literacy survey and the different purposes of tests
8 0.95853531 1920 andrew gelman stats-2013-06-30-“Non-statistical” statistics tools
10 0.95111394 1882 andrew gelman stats-2013-06-03-The statistical properties of smart chains (and referral chains more generally)
12 0.94550586 538 andrew gelman stats-2011-01-25-Postdoc Position #2: Hierarchical Modeling and Statistical Graphics
13 0.94152629 770 andrew gelman stats-2011-06-15-Still more Mr. P in public health
14 0.94111347 70 andrew gelman stats-2010-06-07-Mister P goes on a date
15 0.93912637 803 andrew gelman stats-2011-07-14-Subtleties with measurement-error models for the evaluation of wacky claims
16 0.93824732 902 andrew gelman stats-2011-09-12-The importance of style in academic writing
17 0.93558323 392 andrew gelman stats-2010-11-03-Taleb + 3.5 years
19 0.93514955 2244 andrew gelman stats-2014-03-11-What if I were to stop publishing in journals?
20 0.93478799 2353 andrew gelman stats-2014-05-30-I posted this as a comment on a sociology blog