andrew_gelman_stats andrew_gelman_stats-2013 andrew_gelman_stats-2013-1963 knowledge-graph by maker-knowledge-mining
Source: html
Introduction: Last week I published in Slate a critique of a paper that appeared in the journal Psychological Science. That paper, by Alec Beall and Jessica Tracy, found that women who were at peak fertility were three times more likely to wear red or pink shirts, compared to women at other points in their menstrual cycles. The study was based an 100 participants on the internet and 24 college students. In my critique, I argued that we had no reason to believe the results generalized to the larger population, because (1) the samples were not representative, (2) the measurements were noisy, (3) the researchers did not use the correct dates of peak fertility, and (4) there were many different comparisons that could have been reported in the data, so there was nothing special about a particular comparison being statistically significant. I likened their paper to other work which I considered flawed for multiple comparisons (too many researcher degrees of freedom), including a claimed relation bet
sentIndex sentText sentNum sentScore
1 That paper, by Alec Beall and Jessica Tracy, found that women who were at peak fertility were three times more likely to wear red or pink shirts, compared to women at other points in their menstrual cycles. [sent-2, score-1.074]
2 a) Gelman suggests that we might have benefited from researcher degrees of freedom by asking participants to report the color of each item of clothing they wore, then choosing to report results for shirt color only. [sent-17, score-0.639]
3 b) We categorized shirts that were red and pink together because pink is a shade of red; it is light red. [sent-19, score-0.661]
4 The theory we were testing is based on the idea that red and shades of red (such as the pinkish swellings seen in ovulating chimpanzees, or the pinkish skin tone observed in attractive and healthy human faces) are associated with sexual interest and attractiveness (e. [sent-20, score-0.512]
5 Thus, our decision to combine red and pink in our analyses was a theoretical one. [sent-27, score-0.424]
6 If any of these analyses other than those of pink and red had produced significant differences, we would have failed to support our hypothesis. [sent-40, score-0.499]
7 We conducted these studies with the sole purpose of testing one specific hypothesis: that conception risk would increase women’s tendency to dress in red or pink. [sent-43, score-0.519]
8 Gelman is right that there is a good deal of debate about which days best reflect a high conception risk period, and this is a legitimate criticism of all research that assesses fertility without directly measuring hormone levels. [sent-49, score-0.568]
9 We adopted the Day 6-14 categorization period after finding that this is the categorization used by a large body of previously published, well-run studies on conception risk (e. [sent-51, score-0.492]
10 Rather, we adopted it a priori and used it and only it in analyzing our data; no researcher degrees of freedom came into play. [sent-58, score-0.477]
11 e) In any study that assesses conception risk using a self-report measure, certain women must be excluded to ensure that those for whom risk was not accurately captured do not erroneously influence results. [sent-59, score-0.601]
12 The decision of whether to exclude women on the basis of these gray-area criteria does lead to the possibility of researcher degrees of freedom. [sent-64, score-0.555]
13 First, like any published set of empirical studies, our article should not be viewed as the ultimate conclusion on the question of whether women are more likely to wear red or pink when at high risk for conception. [sent-73, score-0.877]
14 Indeed, a statistician like Gelman could go well beyond simply mentioning possible places where additional degrees of freedom might have come into play and then making assumptions about the validity of our findings on that basis. [sent-82, score-0.496]
15 He could, and should, instead find out exactly the places where researcher degrees of freedom did come into play, then calculate the precise likelihood that they would have resulted in the two significant effects that emerged in our studies if these effects were not in fact true. [sent-83, score-0.681]
16 In other words, additional researcher degrees of freedom increase the chance that we will find a significant effect where none exists. [sent-84, score-0.62]
17 How many researcher degrees of freedom would it take for this to become a figure that would reasonably allow Gelman to suggest that our effect is most likely a false positive? [sent-89, score-0.589]
18 Researchers do have certain responsibilities—such as avoiding, to whatever extent possible, taking advantage of researcher degrees of freedom and being honest about it when they do– but critics of research have certain responsibilities too. [sent-92, score-0.666]
19 Tracy and Beall write, “If any of these analyses other than those of pink and red had produced significant differences, we would have failed to support our hypothesis. [sent-108, score-0.499]
20 The point is that there are many degrees of freedom available, even if with the particular data that happened to occur , the researchers did only one particular analysis. [sent-113, score-0.421]
wordName wordTfidf (topN-words)
[('perrett', 0.275), ('beall', 0.248), ('tracy', 0.232), ('pink', 0.21), ('degrees', 0.199), ('fertility', 0.193), ('women', 0.179), ('gelman', 0.162), ('freedom', 0.16), ('stephen', 0.159), ('red', 0.155), ('coetzee', 0.151), ('risk', 0.128), ('researcher', 0.118), ('et', 0.116), ('conception', 0.11), ('jones', 0.108), ('elliot', 0.101), ('gu', 0.101), ('guen', 0.101), ('pazda', 0.101), ('pinkish', 0.101), ('categorization', 0.092), ('slate', 0.09), ('wear', 0.088), ('shirts', 0.086), ('significant', 0.078), ('research', 0.077), ('colors', 0.076), ('debruine', 0.075), ('whitehead', 0.075), ('hypothesis', 0.074), ('findings', 0.072), ('studies', 0.07), ('peak', 0.07), ('excluding', 0.066), ('additional', 0.065), ('researchers', 0.062), ('gray', 0.061), ('days', 0.06), ('published', 0.059), ('decision', 0.059), ('clothing', 0.058), ('article', 0.058), ('certain', 0.056), ('would', 0.056), ('practices', 0.055), ('prior', 0.054), ('paper', 0.052), ('color', 0.052)]
simIndex simValue blogId blogTitle
Introduction: Last week I published in Slate a critique of a paper that appeared in the journal Psychological Science. That paper, by Alec Beall and Jessica Tracy, found that women who were at peak fertility were three times more likely to wear red or pink shirts, compared to women at other points in their menstrual cycles. The study was based an 100 participants on the internet and 24 college students. In my critique, I argued that we had no reason to believe the results generalized to the larger population, because (1) the samples were not representative, (2) the measurements were noisy, (3) the researchers did not use the correct dates of peak fertility, and (4) there were many different comparisons that could have been reported in the data, so there was nothing special about a particular comparison being statistically significant. I likened their paper to other work which I considered flawed for multiple comparisons (too many researcher degrees of freedom), including a claimed relation bet
Introduction: Jessica Tracy and Alec Beall, authors of that paper that claimed that women at peak fertility were more likely to wear red or pink shirts (see further discussion here and here ), and then a later paper that claimed that this happens in some weather but not others, just informed me that they have posted a note in disagreement with an paper by Eric Loken and myself. Our paper is unpublished, but I do have the megaphone of this blog, and Tracy and Beall do not, so I think it’s only fair to link to their note right away. I’ll quote from their note (but if you’re interested, please follow the link and read the whole thing ) and then give some background and my own reaction. Tracy and Beall write: Although Gelman and Loken are using our work as an example of a broader problem that pervades the field–a problem we generally agree about–we are concerned that readers will take their speculations about our methods and analyses as factual claims about our scientific integrity. Fu
Introduction: In my article about implausible p-values in psychology studies, I wrote: “Women Are More Likely to Wear Red or Pink at Peak Fertility,” by Alec Beall and Jessica Tracy, is based on two samples: a self-selected sample of 100 women from the Internet, and 24 undergraduates at the University of British Columbia. . . . [There is a problem with] representativeness. What color clothing you wear has a lot to do with where you live and who you hang out with. Participants in an Internet survey and University of British Columbia students aren’t particularly representative of much more than … participants in an Internet survey and University of British Columbia students. In response, I received this in an email from a prominent psychology researcher (not someone I know personally): Complaining that subjects in an experiment were not randomly sampled is what freshmen do before they take their first psychology class. I really *hope* you why that is an absurd criticism – especially of au
4 0.20528084 1860 andrew gelman stats-2013-05-17-How can statisticians help psychologists do their research better?
Introduction: I received two emails yesterday on related topics. First, Stephen Olivier pointed me to this post by Daniel Lakens, who wrote the following open call to statisticians: You would think that if you are passionate about statistics, then you want to help people to calculate them correctly in any way you can. . . . you’d think some statisticians would be interested in helping a poor mathematically challenged psychologist out by offering some practical advice. I’m the right person to ask this question, since I actually have written a lot of material that helps psychologists (and others) with their data analysis. But there clearly are communication difficulties, in that my work and that of other statisticians hasn’t reached Lakens. Sometimes the contributions of statisticians are made indirectly. For example, I wrote Bayesian Data Analysis, and then Kruschke wrote Doing Bayesian Data Analysis. Our statistics book made it possible for Kruschke to write his excellent book for psycholo
5 0.19894226 2236 andrew gelman stats-2014-03-07-Selection bias in the reporting of shaky research
Introduction: I’ll reorder this week’s posts a bit in order to continue on a topic that came up yesterday. A couple days ago a reporter wrote to me asking what I thought of this paper on Money, Status, and the Ovulatory Cycle. I responded: Given the quality of the earlier paper by these researchers, I’m not inclined to believe anything these people write. But, to be specific, I can point out some things: - The authors define low fertility as days 8-14. Oddly enough, these authors in their earlier paper used days 7-14. But according to womenshealth.gov, the most fertile days are between days 10 and 17. The choice of these days affects their analysis, and it is not a good sign that they use different days in different papers. (see more on this point in sections 2.3 and 3.1 of this paper: http://www.stat.columbia.edu/~gelman/research/unpublished/p_hacking.pdf) - They perform a lot of different analyses, and many others could be performed. For example, “Study 1 indicates that ovul
7 0.15854912 2183 andrew gelman stats-2014-01-23-Discussion on preregistration of research studies
8 0.15853772 2187 andrew gelman stats-2014-01-26-Twitter sucks, and people are gullible as f…
9 0.15839615 739 andrew gelman stats-2011-05-31-When Did Girls Start Wearing Pink?
11 0.14923668 2241 andrew gelman stats-2014-03-10-Preregistration: what’s in it for you?
12 0.13816063 1695 andrew gelman stats-2013-01-28-Economists argue about Bayes
14 0.13463356 511 andrew gelman stats-2011-01-11-One more time on that ESP study: The problem of overestimates and the shrinkage solution
15 0.13398996 2263 andrew gelman stats-2014-03-24-Empirical implications of Empirical Implications of Theoretical Models
16 0.13338186 1883 andrew gelman stats-2013-06-04-Interrogating p-values
17 0.13035837 1139 andrew gelman stats-2012-01-26-Suggested resolution of the Bem paradox
18 0.12812831 1878 andrew gelman stats-2013-05-31-How to fix the tabloids? Toward replicable social science research
20 0.12608379 1114 andrew gelman stats-2012-01-12-Controversy about average personality differences between men and women
topicId topicWeight
[(0, 0.273), (1, -0.022), (2, 0.015), (3, -0.205), (4, -0.054), (5, -0.079), (6, -0.033), (7, -0.023), (8, -0.093), (9, 0.002), (10, -0.014), (11, 0.017), (12, 0.024), (13, -0.029), (14, 0.063), (15, 0.026), (16, 0.021), (17, -0.03), (18, 0.015), (19, -0.011), (20, -0.012), (21, 0.001), (22, -0.006), (23, -0.056), (24, -0.011), (25, 0.005), (26, 0.0), (27, -0.003), (28, 0.062), (29, -0.064), (30, -0.032), (31, -0.011), (32, -0.005), (33, 0.021), (34, 0.005), (35, -0.015), (36, -0.005), (37, 0.006), (38, -0.07), (39, -0.071), (40, -0.056), (41, -0.028), (42, 0.051), (43, -0.018), (44, -0.024), (45, 0.009), (46, -0.022), (47, -0.043), (48, -0.014), (49, 0.007)]
simIndex simValue blogId blogTitle
same-blog 1 0.97332394 1963 andrew gelman stats-2013-07-31-Response by Jessica Tracy and Alec Beall to my critique of the methods in their paper, “Women Are More Likely to Wear Red or Pink at Peak Fertility”
Introduction: Last week I published in Slate a critique of a paper that appeared in the journal Psychological Science. That paper, by Alec Beall and Jessica Tracy, found that women who were at peak fertility were three times more likely to wear red or pink shirts, compared to women at other points in their menstrual cycles. The study was based an 100 participants on the internet and 24 college students. In my critique, I argued that we had no reason to believe the results generalized to the larger population, because (1) the samples were not representative, (2) the measurements were noisy, (3) the researchers did not use the correct dates of peak fertility, and (4) there were many different comparisons that could have been reported in the data, so there was nothing special about a particular comparison being statistically significant. I likened their paper to other work which I considered flawed for multiple comparisons (too many researcher degrees of freedom), including a claimed relation bet
2 0.82027161 1860 andrew gelman stats-2013-05-17-How can statisticians help psychologists do their research better?
Introduction: I received two emails yesterday on related topics. First, Stephen Olivier pointed me to this post by Daniel Lakens, who wrote the following open call to statisticians: You would think that if you are passionate about statistics, then you want to help people to calculate them correctly in any way you can. . . . you’d think some statisticians would be interested in helping a poor mathematically challenged psychologist out by offering some practical advice. I’m the right person to ask this question, since I actually have written a lot of material that helps psychologists (and others) with their data analysis. But there clearly are communication difficulties, in that my work and that of other statisticians hasn’t reached Lakens. Sometimes the contributions of statisticians are made indirectly. For example, I wrote Bayesian Data Analysis, and then Kruschke wrote Doing Bayesian Data Analysis. Our statistics book made it possible for Kruschke to write his excellent book for psycholo
Introduction: Jessica Tracy and Alec Beall, authors of that paper that claimed that women at peak fertility were more likely to wear red or pink shirts (see further discussion here and here ), and then a later paper that claimed that this happens in some weather but not others, just informed me that they have posted a note in disagreement with an paper by Eric Loken and myself. Our paper is unpublished, but I do have the megaphone of this blog, and Tracy and Beall do not, so I think it’s only fair to link to their note right away. I’ll quote from their note (but if you’re interested, please follow the link and read the whole thing ) and then give some background and my own reaction. Tracy and Beall write: Although Gelman and Loken are using our work as an example of a broader problem that pervades the field–a problem we generally agree about–we are concerned that readers will take their speculations about our methods and analyses as factual claims about our scientific integrity. Fu
4 0.79755288 1074 andrew gelman stats-2011-12-20-Reading a research paper != agreeing with its claims
Introduction: A journalist wrote to me recently: I was going to include your deconstruction of the beautiful daughters paper, but ran out of space. The author, incidentally, stands by that paper — and emailed me that you’d advised him on a later paper, implying that meant you now accepted the thesis! I responded: I know that Kanazawa stands by his earlier claim. Unfortunately, in this case, “stands by” means believing something with no evidence. And, yes, he send me a copy of his other paper, and I responded by telling him I thought his sample size is too small. I did advise him, and my advice was to not do it! To avoid any future confusion, I thought I’d post my half of my emails with Kanazawa over the years: 28 Aug 2006: Dear Dr. Kanazawa, I read with interest your papers on sex ratios in the Journal of Theoretical Biology. Sex ratios are an inherently interesting topic and also a favorite example for people such as myself who teach probability and statistics. In the cour
5 0.7930612 2236 andrew gelman stats-2014-03-07-Selection bias in the reporting of shaky research
Introduction: I’ll reorder this week’s posts a bit in order to continue on a topic that came up yesterday. A couple days ago a reporter wrote to me asking what I thought of this paper on Money, Status, and the Ovulatory Cycle. I responded: Given the quality of the earlier paper by these researchers, I’m not inclined to believe anything these people write. But, to be specific, I can point out some things: - The authors define low fertility as days 8-14. Oddly enough, these authors in their earlier paper used days 7-14. But according to womenshealth.gov, the most fertile days are between days 10 and 17. The choice of these days affects their analysis, and it is not a good sign that they use different days in different papers. (see more on this point in sections 2.3 and 3.1 of this paper: http://www.stat.columbia.edu/~gelman/research/unpublished/p_hacking.pdf) - They perform a lot of different analyses, and many others could be performed. For example, “Study 1 indicates that ovul
6 0.79199052 2223 andrew gelman stats-2014-02-24-“Edlin’s rule” for routinely scaling down published estimates
8 0.7853561 1128 andrew gelman stats-2012-01-19-Sharon Begley: Worse than Stephen Jay Gould?
9 0.7843501 2241 andrew gelman stats-2014-03-10-Preregistration: what’s in it for you?
12 0.77676427 897 andrew gelman stats-2011-09-09-The difference between significant and not significant…
13 0.77237219 1959 andrew gelman stats-2013-07-28-50 shades of gray: A research story
14 0.76987529 898 andrew gelman stats-2011-09-10-Fourteen magic words: an update
16 0.76657039 1883 andrew gelman stats-2013-06-04-Interrogating p-values
17 0.76485902 1954 andrew gelman stats-2013-07-24-Too Good To Be True: The Scientific Mass Production of Spurious Statistical Significance
18 0.75780022 2220 andrew gelman stats-2014-02-22-Quickies
19 0.75688678 1878 andrew gelman stats-2013-05-31-How to fix the tabloids? Toward replicable social science research
topicId topicWeight
[(2, 0.024), (15, 0.035), (16, 0.09), (21, 0.046), (24, 0.147), (45, 0.014), (63, 0.013), (82, 0.177), (86, 0.013), (95, 0.015), (99, 0.262)]
simIndex simValue blogId blogTitle
1 0.98483944 940 andrew gelman stats-2011-10-03-It depends upon what the meaning of the word “firm” is.
Introduction: David Hogg pointed me to this news article by Angela Saini: It’s not often that the quiet world of mathematics is rocked by a murder case. But last summer saw a trial that sent academics into a tailspin, and has since swollen into a fevered clash between science and the law. At its heart, this is a story about chance. And it begins with a convicted killer, “T”, who took his case to the court of appeal in 2010. Among the evidence against him was a shoeprint from a pair of Nike trainers, which seemed to match a pair found at his home. While appeals often unmask shaky evidence, this was different. This time, a mathematical formula was thrown out of court. The footwear expert made what the judge believed were poor calculations about the likelihood of the match, compounded by a bad explanation of how he reached his opinion. The conviction was quashed. . . . “The impact will be quite shattering,” says Professor Norman Fenton, a mathematician at Queen Mary, University of London.
2 0.97423935 178 andrew gelman stats-2010-08-03-(Partisan) visualization of health care legislation
Introduction: Congressman Kevin Brady from Texas distributes this visualization of reformed health care in the US (click for a bigger picture): Here’s a PDF at Brady’s page, and a local copy of it. Complexity has its costs. Beyond the cost of writing it, learning it, following it, there’s also the cost of checking it. John Walker has some funny examples of what’s hidden in the almost 8000 pages of IRS code. Text mining and applied statistics will solve all that, hopefully. Anyone interested in developing a pork detection system for the legislation? Or an analysis of how much entropy to the legal code did each congressman contribute? There are already spin detectors , that help you detect whether the writer is a Democrat (“stimulus”, “health care”) or a Republican (“deficit spending”, “ObamaCare”). D+0.1: Jared Lander points to versions by Rep. Boehner and Robert Palmer .
3 0.96848571 335 andrew gelman stats-2010-10-11-How to think about Lou Dobbs
Introduction: I was unsurprised to read that Lou Dobbs, the former CNN host who crusaded against illegal immigrants, had actually hired a bunch of them himself to maintain his large house and his horse farm. (OK, I have to admit I was surprised by the part about the horse farm.) But I think most of the reactions to this story missed the point. Isabel Macdonald’s article that broke the story was entitled, “Lou Dobbs, American Hypocrite,” and most of the discussion went from there, with some commenters piling on Dobbs and others defending him by saying that Dobbs hired his laborers through contractors and may not have known they were in the country illegally. To me, though, the key issue is slightly different. And Macdonald’s story is relevant whether or not Dobbs knew he was hiring illegals. My point is not that Dobbs is a bad guy, or a hypocrite, or whatever. My point is that, in his setting, it would take an extraordinary effort to not hire illegal immigrants to take care of his house
4 0.9579165 1440 andrew gelman stats-2012-08-02-“A Christmas Carol” as applied to plagiarism
Introduction: John Mashey sends me this delightful video (not in English but it has subtitles) from the University of Bergen (link comes from this page from Elsevier but I don’t see any direct connection between the controversial academic publisher and the Bergen group). Part of me believes, deep down, that if someone were to send this link to Edward Wegman , he will repent, that he’ll just break down, confess, and apologize to everybody involved. I can’t understand the psychology of such people. I mean, I can understand someone being lazy enough to plagiarize and to deny if accused. But to keep denying after you’ve been caught and everyone knows you did it—I simply can’t see how someone can do that. But this surely reflects my nerd-like lack of understanding of human nature, more than anything else. It’s a bit scary that someone such as myself who has such poor intuitions about human behavior can become a prominent social scientist, but I suppose it takes all kinds. P.S. At least I’m
5 0.95383006 1772 andrew gelman stats-2013-03-20-Stan at Google this Thurs and at Berkeley this Fri noon
Introduction: Michael Betancourt will be speaking at Google and at the University of California, Berkeley. The Google talk is closed to outsiders (but if you work at Google, you should go!); the Berkeley talk is open to all: Friday March 22, 12:10 pm, Evans Hall 1011. Title of talk: Stan : Practical Bayesian Inference with Hamiltonian Monte Carlo Abstract: Practical implementations of Bayesian inference are often limited to approximation methods that only slowly explore the posterior distribution. By taking advantage of the curvature of the posterior, however, Hamiltonian Monte Carlo (HMC) efficiently explores even the most highly contorted distributions. In this talk I will review the foundations of and recent developments within HMC, concluding with a discussion of Stan, a powerful inference engine that utilizes HMC, automatic differentiation, and adaptive methods to minimize user input. This is cool stuff. And he’ll be showing the whirlpool movie!
6 0.94302082 67 andrew gelman stats-2010-06-03-More on that Dartmouth health care study
7 0.94189107 340 andrew gelman stats-2010-10-13-Randomized experiments, non-randomized experiments, and observational studies
8 0.94165683 326 andrew gelman stats-2010-10-07-Peer pressure, selection, and educational reform
9 0.94048017 1488 andrew gelman stats-2012-09-08-Annals of spam
same-blog 10 0.9397409 1963 andrew gelman stats-2013-07-31-Response by Jessica Tracy and Alec Beall to my critique of the methods in their paper, “Women Are More Likely to Wear Red or Pink at Peak Fertility”
12 0.93351811 1553 andrew gelman stats-2012-10-30-Real rothko, fake rothko
13 0.93013972 1749 andrew gelman stats-2013-03-04-Stan in L.A. this Wed 3:30pm
14 0.92865098 1958 andrew gelman stats-2013-07-27-Teaching is hard
15 0.92598259 193 andrew gelman stats-2010-08-09-Besag
16 0.92512459 2003 andrew gelman stats-2013-08-30-Stan Project: Continuous Relaxations for Discrete MRFs
17 0.92361987 699 andrew gelman stats-2011-05-06-Another stereotype demolished
18 0.92336047 1134 andrew gelman stats-2012-01-21-Lessons learned from a recent R package submission
19 0.91544694 1039 andrew gelman stats-2011-12-02-I just flew in from the econ seminar, and boy are my arms tired
20 0.91454458 366 andrew gelman stats-2010-10-24-Mankiw tax update