andrew_gelman_stats andrew_gelman_stats-2014 andrew_gelman_stats-2014-2355 knowledge-graph by maker-knowledge-mining
Source: html
Introduction: Jessica Tracy and Alec Beall, authors of that paper that claimed that women at peak fertility were more likely to wear red or pink shirts (see further discussion here and here ), and then a later paper that claimed that this happens in some weather but not others, just informed me that they have posted a note in disagreement with an paper by Eric Loken and myself. Our paper is unpublished, but I do have the megaphone of this blog, and Tracy and Beall do not, so I think it’s only fair to link to their note right away. I’ll quote from their note (but if you’re interested, please follow the link and read the whole thing ) and then give some background and my own reaction. Tracy and Beall write: Although Gelman and Loken are using our work as an example of a broader problem that pervades the field–a problem we generally agree about–we are concerned that readers will take their speculations about our methods and analyses as factual claims about our scientific integrity. Fu
sentIndex sentText sentNum sentScore
1 The data In their email, Tracy and Beall asked us to put them in contact with the editor of the journal who is handling our paper, and they told us that if we were to amend our paper to take into account of their new analyses, they would share their raw data with us if we would like. [sent-76, score-0.409]
2 Beall and Tracy define peak fertility as days 6-14 of the menstrual cycle. [sent-90, score-0.613]
3 Under the Calendar Method, they recommend you compute the first day when you’re likely to be fertile as “Subtract 18 days from the total days of your shortest cycle,” and for the last day they say, “Subtract 11 days from the total days of your longest cycle. [sent-99, score-0.536]
4 But I don’t think anyone out there is including days 6 and 7 in the peak times for fertility. [sent-106, score-0.373]
5 So where did Beall and Tracy’s fertility days come from? [sent-107, score-0.417]
6 Consider the following statement of Tracy and Beall: It doesn’t particularly matter which window researchers use, as long as they make an a-priori decision about which to use and then run analyses for that window only. [sent-123, score-0.317]
7 It might be that the expert told them that the days of peak fertility were days 10-17 or that they looked at the U. [sent-136, score-0.778]
8 Or it's possible they talked with an gynecological expert who told them that days 6-14 were peak fertility or that they happened to encounter a website that gave those dates. [sent-139, score-0.644]
9 Don’t Beall and Tracy care [just to be clear: I have no doubt that they care about the degree of truth of their research hypotheses; the thing I'm asking is if they care that they might have gotten the dates of peak fertility wrong]? [sent-141, score-0.674]
10 Even so, under that best of all possible worlds (a world which did not exist), they wouldn’t have a finding about peak fertility, they’d have a finding about days 6-14. [sent-145, score-0.33]
11 Upon learning they got the dates of peak fertility wrong, Tracy and Beall’s response should not be: Hey, it doesn’t matter. [sent-147, score-0.526]
12 In particular, they report in their recent note that repeating their analysis using the days 7-14 window (recall that they originally used days 6-14) yields an odds ratio of 1. [sent-157, score-0.453]
13 ] In any case, the point is that if you want to study peak fertility, you should study peak fertility, which, according to the most authoritative source I can find, goes from days 10-17. [sent-164, score-0.624]
14 I’m going into detail about this peak fertility thing because I want to emphasize my statistician’s perspective that, if you care about the science, you should care about the measurement. [sent-168, score-0.549]
15 I don’t criticize Beall and Tracy for getting the dates of peak fertility wrong—it’s natural to trust the literature in your subfield, and we all make mistakes. [sent-169, score-0.526]
16 Department of Health and Human Services really does say the dates of peak fertility are days 10-17, then it’s time for Beall and Tracy to take a deep think. [sent-176, score-0.696]
17 If my only interest were to make an airtight case against the Tracy and Beall claims, I might not even bother with the peak fertility thing, as we have enough other strong points. [sent-180, score-0.52]
18 Finally I don’t particularly care about fertility and choices of clothing, and I do feel that these researchers are shooting in the dark. [sent-213, score-0.458]
19 Meanwhile, if you’re interested in scientific misunderstandings but you don’t want to read more about fertility and clothing choice, I recommend you go to Dan Kahan’s cultural cognition blog . [sent-221, score-0.35]
20 ” Their take on peak fertility is different from mine, but I can see how it makes sense for them to follow the literature in their subfield, even if from my perspective these are not the most authoritative sources on the topic. [sent-248, score-0.633]
wordName wordTfidf (topN-words)
[('beall', 0.57), ('tracy', 0.566), ('fertility', 0.283), ('peak', 0.196), ('days', 0.134), ('window', 0.082), ('paper', 0.078), ('loken', 0.074), ('eric', 0.073), ('fishing', 0.066), ('data', 0.063), ('researchers', 0.063), ('hypotheses', 0.059), ('analyses', 0.056), ('note', 0.053), ('analysis', 0.05), ('comparisons', 0.049), ('dates', 0.047), ('expedition', 0.045), ('feel', 0.044), ('think', 0.043), ('different', 0.043), ('research', 0.043), ('even', 0.041), ('contingent', 0.04), ('multiple', 0.039), ('regan', 0.036), ('take', 0.036), ('us', 0.036), ('care', 0.035), ('work', 0.035), ('moderator', 0.035), ('scientific', 0.034), ('matter', 0.034), ('authoritative', 0.034), ('choices', 0.033), ('services', 0.033), ('clothing', 0.033), ('study', 0.032), ('re', 0.032), ('wrong', 0.032), ('forking', 0.032), ('women', 0.031), ('told', 0.031), ('measuring', 0.031), ('studies', 0.031), ('papers', 0.031), ('subfield', 0.03), ('garden', 0.03), ('new', 0.03)]
simIndex simValue blogId blogTitle
same-blog 1 0.99999982 2355 andrew gelman stats-2014-05-31-Jessica Tracy and Alec Beall (authors of the fertile-women-wear-pink study) comment on our Garden of Forking Paths paper, and I comment on their comments
Introduction: Jessica Tracy and Alec Beall, authors of that paper that claimed that women at peak fertility were more likely to wear red or pink shirts (see further discussion here and here ), and then a later paper that claimed that this happens in some weather but not others, just informed me that they have posted a note in disagreement with an paper by Eric Loken and myself. Our paper is unpublished, but I do have the megaphone of this blog, and Tracy and Beall do not, so I think it’s only fair to link to their note right away. I’ll quote from their note (but if you’re interested, please follow the link and read the whole thing ) and then give some background and my own reaction. Tracy and Beall write: Although Gelman and Loken are using our work as an example of a broader problem that pervades the field–a problem we generally agree about–we are concerned that readers will take their speculations about our methods and analyses as factual claims about our scientific integrity. Fu
Introduction: Last week I published in Slate a critique of a paper that appeared in the journal Psychological Science. That paper, by Alec Beall and Jessica Tracy, found that women who were at peak fertility were three times more likely to wear red or pink shirts, compared to women at other points in their menstrual cycles. The study was based an 100 participants on the internet and 24 college students. In my critique, I argued that we had no reason to believe the results generalized to the larger population, because (1) the samples were not representative, (2) the measurements were noisy, (3) the researchers did not use the correct dates of peak fertility, and (4) there were many different comparisons that could have been reported in the data, so there was nothing special about a particular comparison being statistically significant. I likened their paper to other work which I considered flawed for multiple comparisons (too many researcher degrees of freedom), including a claimed relation bet
3 0.19661534 2241 andrew gelman stats-2014-03-10-Preregistration: what’s in it for you?
Introduction: Chris Chambers pointed me to a blog by someone called Neuroskeptic who suggested that I preregister my political science studies: So when Andrew Gelman (let’s say) is going to start using a new approach, he goes on Twitter, or on his blog, and posts a bare-bones summary of what he’s going to do. Then he does it. If he finds something interesting, he writes it up as a paper, citing that tweet or post as his preregistration. . . . I think this approach has some benefits but doesn’t really address the issues of preregistration that concern me—but I’d like to spend an entire blog post explaining why. I have two key points: 1. If your study is crap, preregistration might fix it. Preregistration is fine—indeed, the wide acceptance of preregistration might well motivate researchers to not do so many crap studies—but it doesn’t solve fundamental problems of experimental design. 2. “Preregistration” seems to mean different things in different scenarios: A. When the concern is
Introduction: In my article about implausible p-values in psychology studies, I wrote: “Women Are More Likely to Wear Red or Pink at Peak Fertility,” by Alec Beall and Jessica Tracy, is based on two samples: a self-selected sample of 100 women from the Internet, and 24 undergraduates at the University of British Columbia. . . . [There is a problem with] representativeness. What color clothing you wear has a lot to do with where you live and who you hang out with. Participants in an Internet survey and University of British Columbia students aren’t particularly representative of much more than … participants in an Internet survey and University of British Columbia students. In response, I received this in an email from a prominent psychology researcher (not someone I know personally): Complaining that subjects in an experiment were not randomly sampled is what freshmen do before they take their first psychology class. I really *hope* you why that is an absurd criticism – especially of au
5 0.17832175 2236 andrew gelman stats-2014-03-07-Selection bias in the reporting of shaky research
Introduction: I’ll reorder this week’s posts a bit in order to continue on a topic that came up yesterday. A couple days ago a reporter wrote to me asking what I thought of this paper on Money, Status, and the Ovulatory Cycle. I responded: Given the quality of the earlier paper by these researchers, I’m not inclined to believe anything these people write. But, to be specific, I can point out some things: - The authors define low fertility as days 8-14. Oddly enough, these authors in their earlier paper used days 7-14. But according to womenshealth.gov, the most fertile days are between days 10 and 17. The choice of these days affects their analysis, and it is not a good sign that they use different days in different papers. (see more on this point in sections 2.3 and 3.1 of this paper: http://www.stat.columbia.edu/~gelman/research/unpublished/p_hacking.pdf) - They perform a lot of different analyses, and many others could be performed. For example, “Study 1 indicates that ovul
6 0.13659705 1860 andrew gelman stats-2013-05-17-How can statisticians help psychologists do their research better?
7 0.11723479 1695 andrew gelman stats-2013-01-28-Economists argue about Bayes
8 0.11517893 2263 andrew gelman stats-2014-03-24-Empirical implications of Empirical Implications of Theoretical Models
9 0.10517461 2245 andrew gelman stats-2014-03-12-More on publishing in journals
13 0.10013139 1878 andrew gelman stats-2013-05-31-How to fix the tabloids? Toward replicable social science research
14 0.098637335 1435 andrew gelman stats-2012-07-30-Retracted articles and unethical behavior in economics journals?
15 0.096315086 1139 andrew gelman stats-2012-01-26-Suggested resolution of the Bem paradox
17 0.094367802 2137 andrew gelman stats-2013-12-17-Replication backlash
19 0.092056885 1588 andrew gelman stats-2012-11-23-No one knows what it’s like to be the bad man
20 0.090334088 888 andrew gelman stats-2011-09-03-A psychology researcher asks: Is Anova dead?
topicId topicWeight
[(0, 0.241), (1, -0.038), (2, -0.021), (3, -0.113), (4, -0.014), (5, -0.051), (6, -0.001), (7, -0.026), (8, -0.028), (9, -0.017), (10, -0.01), (11, 0.025), (12, 0.021), (13, -0.038), (14, 0.022), (15, 0.025), (16, 0.002), (17, -0.028), (18, 0.002), (19, 0.007), (20, 0.007), (21, 0.003), (22, -0.007), (23, -0.014), (24, -0.041), (25, 0.019), (26, 0.0), (27, 0.003), (28, 0.033), (29, -0.022), (30, -0.013), (31, -0.014), (32, 0.007), (33, -0.007), (34, 0.005), (35, -0.001), (36, -0.0), (37, 0.007), (38, -0.05), (39, -0.034), (40, -0.021), (41, -0.006), (42, 0.021), (43, -0.03), (44, -0.015), (45, -0.014), (46, -0.019), (47, -0.038), (48, -0.011), (49, -0.0)]
simIndex simValue blogId blogTitle
Introduction: Last week I published in Slate a critique of a paper that appeared in the journal Psychological Science. That paper, by Alec Beall and Jessica Tracy, found that women who were at peak fertility were three times more likely to wear red or pink shirts, compared to women at other points in their menstrual cycles. The study was based an 100 participants on the internet and 24 college students. In my critique, I argued that we had no reason to believe the results generalized to the larger population, because (1) the samples were not representative, (2) the measurements were noisy, (3) the researchers did not use the correct dates of peak fertility, and (4) there were many different comparisons that could have been reported in the data, so there was nothing special about a particular comparison being statistically significant. I likened their paper to other work which I considered flawed for multiple comparisons (too many researcher degrees of freedom), including a claimed relation bet
same-blog 2 0.95359772 2355 andrew gelman stats-2014-05-31-Jessica Tracy and Alec Beall (authors of the fertile-women-wear-pink study) comment on our Garden of Forking Paths paper, and I comment on their comments
Introduction: Jessica Tracy and Alec Beall, authors of that paper that claimed that women at peak fertility were more likely to wear red or pink shirts (see further discussion here and here ), and then a later paper that claimed that this happens in some weather but not others, just informed me that they have posted a note in disagreement with an paper by Eric Loken and myself. Our paper is unpublished, but I do have the megaphone of this blog, and Tracy and Beall do not, so I think it’s only fair to link to their note right away. I’ll quote from their note (but if you’re interested, please follow the link and read the whole thing ) and then give some background and my own reaction. Tracy and Beall write: Although Gelman and Loken are using our work as an example of a broader problem that pervades the field–a problem we generally agree about–we are concerned that readers will take their speculations about our methods and analyses as factual claims about our scientific integrity. Fu
3 0.90250403 2241 andrew gelman stats-2014-03-10-Preregistration: what’s in it for you?
Introduction: Chris Chambers pointed me to a blog by someone called Neuroskeptic who suggested that I preregister my political science studies: So when Andrew Gelman (let’s say) is going to start using a new approach, he goes on Twitter, or on his blog, and posts a bare-bones summary of what he’s going to do. Then he does it. If he finds something interesting, he writes it up as a paper, citing that tweet or post as his preregistration. . . . I think this approach has some benefits but doesn’t really address the issues of preregistration that concern me—but I’d like to spend an entire blog post explaining why. I have two key points: 1. If your study is crap, preregistration might fix it. Preregistration is fine—indeed, the wide acceptance of preregistration might well motivate researchers to not do so many crap studies—but it doesn’t solve fundamental problems of experimental design. 2. “Preregistration” seems to mean different things in different scenarios: A. When the concern is
4 0.90183967 2236 andrew gelman stats-2014-03-07-Selection bias in the reporting of shaky research
Introduction: I’ll reorder this week’s posts a bit in order to continue on a topic that came up yesterday. A couple days ago a reporter wrote to me asking what I thought of this paper on Money, Status, and the Ovulatory Cycle. I responded: Given the quality of the earlier paper by these researchers, I’m not inclined to believe anything these people write. But, to be specific, I can point out some things: - The authors define low fertility as days 8-14. Oddly enough, these authors in their earlier paper used days 7-14. But according to womenshealth.gov, the most fertile days are between days 10 and 17. The choice of these days affects their analysis, and it is not a good sign that they use different days in different papers. (see more on this point in sections 2.3 and 3.1 of this paper: http://www.stat.columbia.edu/~gelman/research/unpublished/p_hacking.pdf) - They perform a lot of different analyses, and many others could be performed. For example, “Study 1 indicates that ovul
5 0.89545298 1860 andrew gelman stats-2013-05-17-How can statisticians help psychologists do their research better?
Introduction: I received two emails yesterday on related topics. First, Stephen Olivier pointed me to this post by Daniel Lakens, who wrote the following open call to statisticians: You would think that if you are passionate about statistics, then you want to help people to calculate them correctly in any way you can. . . . you’d think some statisticians would be interested in helping a poor mathematically challenged psychologist out by offering some practical advice. I’m the right person to ask this question, since I actually have written a lot of material that helps psychologists (and others) with their data analysis. But there clearly are communication difficulties, in that my work and that of other statisticians hasn’t reached Lakens. Sometimes the contributions of statisticians are made indirectly. For example, I wrote Bayesian Data Analysis, and then Kruschke wrote Doing Bayesian Data Analysis. Our statistics book made it possible for Kruschke to write his excellent book for psycholo
6 0.86157858 2223 andrew gelman stats-2014-02-24-“Edlin’s rule” for routinely scaling down published estimates
7 0.85389471 1074 andrew gelman stats-2011-12-20-Reading a research paper != agreeing with its claims
10 0.83605433 2137 andrew gelman stats-2013-12-17-Replication backlash
11 0.83593154 1171 andrew gelman stats-2012-02-16-“False-positive psychology”
13 0.82850218 511 andrew gelman stats-2011-01-11-One more time on that ESP study: The problem of overestimates and the shrinkage solution
14 0.82639617 1054 andrew gelman stats-2011-12-12-More frustrations trying to replicate an analysis published in a reputable journal
16 0.82351846 2361 andrew gelman stats-2014-06-06-Hurricanes vs. Himmicanes
17 0.8227632 1959 andrew gelman stats-2013-07-28-50 shades of gray: A research story
18 0.81865847 1878 andrew gelman stats-2013-05-31-How to fix the tabloids? Toward replicable social science research
19 0.81829309 1883 andrew gelman stats-2013-06-04-Interrogating p-values
20 0.81608504 2352 andrew gelman stats-2014-05-29-When you believe in things that you don’t understand
topicId topicWeight
[(2, 0.015), (15, 0.021), (16, 0.078), (21, 0.051), (24, 0.135), (47, 0.015), (57, 0.01), (63, 0.022), (72, 0.01), (80, 0.011), (82, 0.15), (86, 0.031), (99, 0.301)]
simIndex simValue blogId blogTitle
1 0.98856395 940 andrew gelman stats-2011-10-03-It depends upon what the meaning of the word “firm” is.
Introduction: David Hogg pointed me to this news article by Angela Saini: It’s not often that the quiet world of mathematics is rocked by a murder case. But last summer saw a trial that sent academics into a tailspin, and has since swollen into a fevered clash between science and the law. At its heart, this is a story about chance. And it begins with a convicted killer, “T”, who took his case to the court of appeal in 2010. Among the evidence against him was a shoeprint from a pair of Nike trainers, which seemed to match a pair found at his home. While appeals often unmask shaky evidence, this was different. This time, a mathematical formula was thrown out of court. The footwear expert made what the judge believed were poor calculations about the likelihood of the match, compounded by a bad explanation of how he reached his opinion. The conviction was quashed. . . . “The impact will be quite shattering,” says Professor Norman Fenton, a mathematician at Queen Mary, University of London.
2 0.97219574 326 andrew gelman stats-2010-10-07-Peer pressure, selection, and educational reform
Introduction: Partly in response to my blog on the Harlem Children’s Zone study, Mark Palko wrote this : Talk of education reform always makes me [Palko] deeply nervous. Part of the anxiety comes having spent a number of years behind the podium and having seen the disparity between the claims and the reality of previous reforms. The rest comes from being a statistician and knowing what things like convergence can do to data. Convergent behavior violates the assumption of independent observations used in most simple analyses, but educational studies commonly, perhaps even routinely ignore the complex ways that social norming can cause the nesting of student performance data. In other words, educational research is often based of the idea that teenagers do not respond to peer pressure. . . . and this : When you isolate a group of students, they will quickly arrive at a consensus of what constitutes normal behavior. It is a complex and somewhat unpredictable process driven by personali
3 0.97191095 178 andrew gelman stats-2010-08-03-(Partisan) visualization of health care legislation
Introduction: Congressman Kevin Brady from Texas distributes this visualization of reformed health care in the US (click for a bigger picture): Here’s a PDF at Brady’s page, and a local copy of it. Complexity has its costs. Beyond the cost of writing it, learning it, following it, there’s also the cost of checking it. John Walker has some funny examples of what’s hidden in the almost 8000 pages of IRS code. Text mining and applied statistics will solve all that, hopefully. Anyone interested in developing a pork detection system for the legislation? Or an analysis of how much entropy to the legal code did each congressman contribute? There are already spin detectors , that help you detect whether the writer is a Democrat (“stimulus”, “health care”) or a Republican (“deficit spending”, “ObamaCare”). D+0.1: Jared Lander points to versions by Rep. Boehner and Robert Palmer .
4 0.97179008 1440 andrew gelman stats-2012-08-02-“A Christmas Carol” as applied to plagiarism
Introduction: John Mashey sends me this delightful video (not in English but it has subtitles) from the University of Bergen (link comes from this page from Elsevier but I don’t see any direct connection between the controversial academic publisher and the Bergen group). Part of me believes, deep down, that if someone were to send this link to Edward Wegman , he will repent, that he’ll just break down, confess, and apologize to everybody involved. I can’t understand the psychology of such people. I mean, I can understand someone being lazy enough to plagiarize and to deny if accused. But to keep denying after you’ve been caught and everyone knows you did it—I simply can’t see how someone can do that. But this surely reflects my nerd-like lack of understanding of human nature, more than anything else. It’s a bit scary that someone such as myself who has such poor intuitions about human behavior can become a prominent social scientist, but I suppose it takes all kinds. P.S. At least I’m
5 0.97173858 335 andrew gelman stats-2010-10-11-How to think about Lou Dobbs
Introduction: I was unsurprised to read that Lou Dobbs, the former CNN host who crusaded against illegal immigrants, had actually hired a bunch of them himself to maintain his large house and his horse farm. (OK, I have to admit I was surprised by the part about the horse farm.) But I think most of the reactions to this story missed the point. Isabel Macdonald’s article that broke the story was entitled, “Lou Dobbs, American Hypocrite,” and most of the discussion went from there, with some commenters piling on Dobbs and others defending him by saying that Dobbs hired his laborers through contractors and may not have known they were in the country illegally. To me, though, the key issue is slightly different. And Macdonald’s story is relevant whether or not Dobbs knew he was hiring illegals. My point is not that Dobbs is a bad guy, or a hypocrite, or whatever. My point is that, in his setting, it would take an extraordinary effort to not hire illegal immigrants to take care of his house
6 0.97135949 67 andrew gelman stats-2010-06-03-More on that Dartmouth health care study
7 0.96956509 340 andrew gelman stats-2010-10-13-Randomized experiments, non-randomized experiments, and observational studies
10 0.96430492 1553 andrew gelman stats-2012-10-30-Real rothko, fake rothko
11 0.96241522 2003 andrew gelman stats-2013-08-30-Stan Project: Continuous Relaxations for Discrete MRFs
12 0.95979643 1134 andrew gelman stats-2012-01-21-Lessons learned from a recent R package submission
13 0.95719421 699 andrew gelman stats-2011-05-06-Another stereotype demolished
14 0.95485693 193 andrew gelman stats-2010-08-09-Besag
15 0.95208406 1725 andrew gelman stats-2013-02-17-“1.7%” ha ha ha
same-blog 16 0.95199078 2355 andrew gelman stats-2014-05-31-Jessica Tracy and Alec Beall (authors of the fertile-women-wear-pink study) comment on our Garden of Forking Paths paper, and I comment on their comments
17 0.95022219 1039 andrew gelman stats-2011-12-02-I just flew in from the econ seminar, and boy are my arms tired
18 0.95002919 1488 andrew gelman stats-2012-09-08-Annals of spam
19 0.95001262 1836 andrew gelman stats-2013-05-02-Culture clash
20 0.9496637 1772 andrew gelman stats-2013-03-20-Stan at Google this Thurs and at Berkeley this Fri noon