andrew_gelman_stats andrew_gelman_stats-2013 andrew_gelman_stats-2013-2008 knowledge-graph by maker-knowledge-mining
Source: html
Introduction: In my article about implausible p-values in psychology studies, I wrote: “Women Are More Likely to Wear Red or Pink at Peak Fertility,” by Alec Beall and Jessica Tracy, is based on two samples: a self-selected sample of 100 women from the Internet, and 24 undergraduates at the University of British Columbia. . . . [There is a problem with] representativeness. What color clothing you wear has a lot to do with where you live and who you hang out with. Participants in an Internet survey and University of British Columbia students aren’t particularly representative of much more than … participants in an Internet survey and University of British Columbia students. In response, I received this in an email from a prominent psychology researcher (not someone I know personally): Complaining that subjects in an experiment were not randomly sampled is what freshmen do before they take their first psychology class. I really *hope* you why that is an absurd criticism – especially of au
sentIndex sentText sentNum sentScore
1 Participants in an Internet survey and University of British Columbia students aren’t particularly representative of much more than … participants in an Internet survey and University of British Columbia students. [sent-7, score-0.247]
2 In response, I received this in an email from a prominent psychology researcher (not someone I know personally): Complaining that subjects in an experiment were not randomly sampled is what freshmen do before they take their first psychology class. [sent-8, score-0.812]
3 (And please spare me “but they said men and didn’t say THESE men” because you said there were problems in social psychology and didn’t mention that you had failed to randomly sample the field. [sent-10, score-0.451]
4 Everyone who understands English understands their claims are about their data and that your claims are about the parts of psychology you happen to know about). [sent-11, score-0.475]
5 Just because a freshman might raise a question, that does not make the issue irrelevant! [sent-12, score-0.259]
6 And I hope they remain skeptical of these studies even after they take their first psychology class. [sent-14, score-0.4]
7 Like these freshmen, I am skeptical about generalizing to the general population based on 100 people from the internet and 24 undergraduates. [sent-15, score-0.344]
8 There is no doubt in my mind that the authors and anyone else who found this study to be worth noting) is interested in some generalization to a larger population. [sent-16, score-0.222]
9 Certainly not “all humans” (as claimed by my correspondent), but some large subset of women of childbearing age, some subset that includes college students in Canada and women of various ages who are on Mechanical Turk. [sent-17, score-1.476]
10 Let B be the parameter of interest, in this case the difference in the probability of wearing red or pink shirts, comparing women in two different parts of their menstrual cycle, among the women who are wearing shirts and have regular menstrual periods. [sent-22, score-1.547]
11 For example, perhaps B is a different sign for college students (who typically don’t want to get pregnant), as compared to married women who are trying to have kids. [sent-24, score-0.804]
12 Perhaps B is much different for single women than women with partners (similarly to what was argued in a different recent paper in Psychological Science). [sent-25, score-0.858]
13 If the parameter B is large and pretty much the same sign everywhere, then a sample of college students or internet participants is just fine (measurement issues aside). [sent-36, score-0.917]
14 Now, in this particular case, I expect the authors gained confidence in their results because they appeared in two very different populations. [sent-39, score-0.219]
15 They saw a large B in the group of internet participants and a large B in the college students, hence this is some evidence that B is large in general. [sent-40, score-0.903]
16 This is a good idea in general—two case studies is a good way to get started in looking at variation—but in this particular case I don’t trust it because the sample sizes are so small and the data analysis rules were somewhat flexible (see Eric Loken’s comment here ). [sent-41, score-0.437]
17 In their use of phrases such as “local average treatment effect,” they recognize that treatment effects vary, and they are interested in looking at where an intervention works and where it doesn’t (as in this paper by Rajeev Dehehia). [sent-43, score-0.266]
18 Researchers in medicine and public health are also acutely aware of variation in treatment effects and the need to consider what population is being studied when an effect is being estimated. [sent-44, score-0.336]
19 This is a case where professional expertise can be a bad thing, a case where the intuition of a college freshman can be more valid than the experience of an experienced and much-published researcher. [sent-50, score-0.543]
20 I hope that the next time a freshman comes to my correspondent with a complaint about subjects in an experiment not being randomly sampled, that he (my correspondent) not merely dismiss the complaint but instead discuss with the student its relevance under scenarios 1, 2, and 3. [sent-51, score-0.899]
wordName wordTfidf (topN-words)
[('women', 0.367), ('internet', 0.204), ('freshman', 0.197), ('psychology', 0.194), ('college', 0.184), ('correspondent', 0.164), ('freshmen', 0.164), ('treatment', 0.133), ('large', 0.131), ('generalization', 0.13), ('students', 0.125), ('participants', 0.122), ('british', 0.12), ('shirts', 0.118), ('subset', 0.117), ('understands', 0.109), ('randomly', 0.108), ('representativeness', 0.106), ('clothing', 0.106), ('wearing', 0.101), ('menstrual', 0.099), ('complaint', 0.097), ('wear', 0.097), ('scenarios', 0.095), ('authors', 0.092), ('pink', 0.089), ('sample', 0.085), ('sampled', 0.084), ('case', 0.081), ('scenario', 0.08), ('hope', 0.073), ('population', 0.071), ('medicine', 0.071), ('skeptical', 0.069), ('claimed', 0.068), ('subjects', 0.068), ('vary', 0.067), ('university', 0.067), ('sign', 0.066), ('particular', 0.065), ('samples', 0.065), ('men', 0.064), ('studies', 0.064), ('parts', 0.063), ('fabled', 0.063), ('issue', 0.062), ('different', 0.062), ('measurement', 0.062), ('trust', 0.061), ('aware', 0.061)]
simIndex simValue blogId blogTitle
same-blog 1 0.99999976 2008 andrew gelman stats-2013-09-04-Does it matter that a sample is unrepresentative? It depends on the size of the treatment interactions
Introduction: In my article about implausible p-values in psychology studies, I wrote: “Women Are More Likely to Wear Red or Pink at Peak Fertility,” by Alec Beall and Jessica Tracy, is based on two samples: a self-selected sample of 100 women from the Internet, and 24 undergraduates at the University of British Columbia. . . . [There is a problem with] representativeness. What color clothing you wear has a lot to do with where you live and who you hang out with. Participants in an Internet survey and University of British Columbia students aren’t particularly representative of much more than … participants in an Internet survey and University of British Columbia students. In response, I received this in an email from a prominent psychology researcher (not someone I know personally): Complaining that subjects in an experiment were not randomly sampled is what freshmen do before they take their first psychology class. I really *hope* you why that is an absurd criticism – especially of au
Introduction: Last week I published in Slate a critique of a paper that appeared in the journal Psychological Science. That paper, by Alec Beall and Jessica Tracy, found that women who were at peak fertility were three times more likely to wear red or pink shirts, compared to women at other points in their menstrual cycles. The study was based an 100 participants on the internet and 24 college students. In my critique, I argued that we had no reason to believe the results generalized to the larger population, because (1) the samples were not representative, (2) the measurements were noisy, (3) the researchers did not use the correct dates of peak fertility, and (4) there were many different comparisons that could have been reported in the data, so there was nothing special about a particular comparison being statistically significant. I likened their paper to other work which I considered flawed for multiple comparisons (too many researcher degrees of freedom), including a claimed relation bet
3 0.24709664 1860 andrew gelman stats-2013-05-17-How can statisticians help psychologists do their research better?
Introduction: I received two emails yesterday on related topics. First, Stephen Olivier pointed me to this post by Daniel Lakens, who wrote the following open call to statisticians: You would think that if you are passionate about statistics, then you want to help people to calculate them correctly in any way you can. . . . you’d think some statisticians would be interested in helping a poor mathematically challenged psychologist out by offering some practical advice. I’m the right person to ask this question, since I actually have written a lot of material that helps psychologists (and others) with their data analysis. But there clearly are communication difficulties, in that my work and that of other statisticians hasn’t reached Lakens. Sometimes the contributions of statisticians are made indirectly. For example, I wrote Bayesian Data Analysis, and then Kruschke wrote Doing Bayesian Data Analysis. Our statistics book made it possible for Kruschke to write his excellent book for psycholo
4 0.22731262 2236 andrew gelman stats-2014-03-07-Selection bias in the reporting of shaky research
Introduction: I’ll reorder this week’s posts a bit in order to continue on a topic that came up yesterday. A couple days ago a reporter wrote to me asking what I thought of this paper on Money, Status, and the Ovulatory Cycle. I responded: Given the quality of the earlier paper by these researchers, I’m not inclined to believe anything these people write. But, to be specific, I can point out some things: - The authors define low fertility as days 8-14. Oddly enough, these authors in their earlier paper used days 7-14. But according to womenshealth.gov, the most fertile days are between days 10 and 17. The choice of these days affects their analysis, and it is not a good sign that they use different days in different papers. (see more on this point in sections 2.3 and 3.1 of this paper: http://www.stat.columbia.edu/~gelman/research/unpublished/p_hacking.pdf) - They perform a lot of different analyses, and many others could be performed. For example, “Study 1 indicates that ovul
Introduction: Jessica Tracy and Alec Beall, authors of that paper that claimed that women at peak fertility were more likely to wear red or pink shirts (see further discussion here and here ), and then a later paper that claimed that this happens in some weather but not others, just informed me that they have posted a note in disagreement with an paper by Eric Loken and myself. Our paper is unpublished, but I do have the megaphone of this blog, and Tracy and Beall do not, so I think it’s only fair to link to their note right away. I’ll quote from their note (but if you’re interested, please follow the link and read the whole thing ) and then give some background and my own reaction. Tracy and Beall write: Although Gelman and Loken are using our work as an example of a broader problem that pervades the field–a problem we generally agree about–we are concerned that readers will take their speculations about our methods and analyses as factual claims about our scientific integrity. Fu
7 0.16844174 2187 andrew gelman stats-2014-01-26-Twitter sucks, and people are gullible as f…
9 0.15120162 1114 andrew gelman stats-2012-01-12-Controversy about average personality differences between men and women
10 0.1510078 1226 andrew gelman stats-2012-03-22-Story time meets the all-else-equal fallacy and the fallacy of measurement
12 0.12488847 1864 andrew gelman stats-2013-05-20-Evaluating Columbia University’s Frontiers of Science course
13 0.12225498 963 andrew gelman stats-2011-10-18-Question on Type M errors
14 0.12213167 1352 andrew gelman stats-2012-05-29-Question 19 of my final exam for Design and Analysis of Sample Surveys
15 0.12210616 511 andrew gelman stats-2011-01-11-One more time on that ESP study: The problem of overestimates and the shrinkage solution
16 0.1190984 2210 andrew gelman stats-2014-02-13-Stopping rules and Bayesian analysis
17 0.1176966 1435 andrew gelman stats-2012-07-30-Retracted articles and unethical behavior in economics journals?
18 0.11709152 716 andrew gelman stats-2011-05-17-Is the internet causing half the rapes in Norway? I wanna see the scatterplot.
19 0.11688279 820 andrew gelman stats-2011-07-25-Design of nonrandomized cluster sample study
20 0.11679808 361 andrew gelman stats-2010-10-21-Tenure-track statistics job at Teachers College, here at Columbia!
topicId topicWeight
[(0, 0.259), (1, -0.049), (2, 0.067), (3, -0.191), (4, 0.014), (5, 0.041), (6, -0.004), (7, 0.062), (8, -0.062), (9, -0.001), (10, -0.037), (11, 0.011), (12, 0.007), (13, -0.009), (14, 0.033), (15, 0.001), (16, 0.019), (17, -0.009), (18, -0.012), (19, 0.047), (20, -0.031), (21, -0.054), (22, -0.037), (23, -0.041), (24, 0.018), (25, 0.023), (26, -0.05), (27, 0.009), (28, -0.014), (29, -0.001), (30, -0.08), (31, 0.016), (32, -0.03), (33, -0.004), (34, -0.017), (35, 0.02), (36, -0.014), (37, -0.036), (38, -0.097), (39, -0.027), (40, -0.017), (41, -0.03), (42, 0.071), (43, -0.035), (44, 0.01), (45, -0.012), (46, 0.019), (47, 0.005), (48, -0.021), (49, 0.069)]
simIndex simValue blogId blogTitle
same-blog 1 0.97465324 2008 andrew gelman stats-2013-09-04-Does it matter that a sample is unrepresentative? It depends on the size of the treatment interactions
Introduction: In my article about implausible p-values in psychology studies, I wrote: “Women Are More Likely to Wear Red or Pink at Peak Fertility,” by Alec Beall and Jessica Tracy, is based on two samples: a self-selected sample of 100 women from the Internet, and 24 undergraduates at the University of British Columbia. . . . [There is a problem with] representativeness. What color clothing you wear has a lot to do with where you live and who you hang out with. Participants in an Internet survey and University of British Columbia students aren’t particularly representative of much more than … participants in an Internet survey and University of British Columbia students. In response, I received this in an email from a prominent psychology researcher (not someone I know personally): Complaining that subjects in an experiment were not randomly sampled is what freshmen do before they take their first psychology class. I really *hope* you why that is an absurd criticism – especially of au
2 0.80572975 1114 andrew gelman stats-2012-01-12-Controversy about average personality differences between men and women
Introduction: Blogger Echidne pointed me to a recent article , “The Distance Between Mars and Venus: Measuring Global Sex Differences in Personality,” by Marco Del Giudice, Tom Booth, and Paul Irwing, who find: Sex differences in personality are believed to be comparatively small. However, research in this area has suffered from significant methodological limitations. We advance a set of guidelines for overcoming those limitations: (a) measure personality with a higher resolution than that afforded by the Big Five; (b) estimate sex differences on latent factors; and (c) assess global sex differences with multivariate effect sizes. . . . We found a global effect size D = 2.71, corresponding to an overlap of only 10% between the male and female distributions. Even excluding the factor showing the largest univariate ES [effect size], the global effect size was D = 1.71 (24% overlap). Echidne quotes a news article in which one of the study’s authors going overboard: “Psychologically, men a
3 0.80431402 1053 andrew gelman stats-2011-12-11-This one is so dumb it makes me want to barf
Introduction: Dan Kahan sends in this horror story: A new study finds that atheists are among society’s most distrusted group, comparable even to rapists in certain circumstances. Psychologists at the University of British Columbia and the University of Oregon say that their study demonstrates that anti-atheist prejudice stems from moral distrust, not dislike, of nonbelievers. “It’s pretty remarkable,” said Azim Shariff, an assistant professor of psychology at the University of Oregon and a co-author of the study, which appears in the current issue of Journal of Personality and Social Psychology. The study, conducted among 350 Americans adults and 420 Canadian college students, asked participants to decide if a fictional driver damaged a parked car and left the scene, then found a wallet and took the money, was the driver more likely to be a teacher, an atheist teacher, or a rapist teacher? The participants, who were from religious and nonreligious backgrounds, most often chose the athe
4 0.78869659 1910 andrew gelman stats-2013-06-22-Struggles over the criticism of the “cannabis users and IQ change” paper
Introduction: Ole Rogeberg points me to a discussion of a discussion of a paper: Did pre-release of my [Rogeberg's] PNAS paper on methodological problems with Meier et al’s 2012 paper on cannabis and IQ reduce the chances that it will have its intended effect? In my case, serious methodological issues related to causal inference from non-random observational data became framed as a conflict over conclusions, forcing the original research team to respond rapidly and insufficiently to my concerns, and prompting them to defend their conclusions and original paper in a way that makes a later, more comprehensive reanalysis of their data less likely. This fits with a recurring theme on this blog: the defensiveness of researchers who don’t want to admit they were wrong. Setting aside cases of outright fraud and plagiarism, I think the worst case remains that of psychologists Neil Anderson and Deniz Ones, who denied any problems even in the presence of a smoking gun of a graph revealing their data
Introduction: Paul Alper writes: Unless I missed it, you haven’t commented on the recent article of Michael Bang Peterson [with Daniel Sznycer, Aaron Sell, Leda Cosmides, and John Tooby]. It seems to have been reviewed extensively in the lay press. A typical example is here . This review begins with “If you are physically strong, social science scholars believe they can predict whether or not you are more conservative than other men…Men’s upper-body strength predicts their political opinions on economic redistribution, they write, and they believe that the link may reflect psychological traits that evolved in response to our early ancestral environments and continue to influence behavior today. . . . they surveyed hundreds of people in America, Denmark and Argentina about bicep size, socioeconomic status, and support for economic redistribution.” Further, “Despite the fact that the United States, Denmark and Argentina have very different welfare systems, we still see that — at the psychol
6 0.78092855 1741 andrew gelman stats-2013-02-27-Thin scientists say it’s unhealthy to be fat
9 0.77147025 161 andrew gelman stats-2010-07-24-Differences in color perception by sex, also the Bechdel test for women in movies
10 0.76790273 1128 andrew gelman stats-2012-01-19-Sharon Begley: Worse than Stephen Jay Gould?
11 0.76589721 898 andrew gelman stats-2011-09-10-Fourteen magic words: an update
12 0.75836384 35 andrew gelman stats-2010-05-16-Another update on the spam email study
13 0.75285339 2196 andrew gelman stats-2014-02-03-One-way street fallacy again! in reporting of research on brothers and sisters
16 0.74382496 2223 andrew gelman stats-2014-02-24-“Edlin’s rule” for routinely scaling down published estimates
17 0.7422691 7 andrew gelman stats-2010-04-27-Should Mister P be allowed-encouraged to reside in counter-factual populations?
20 0.73819947 1793 andrew gelman stats-2013-04-08-The Supreme Court meets the fallacy of the one-sided bet
topicId topicWeight
[(1, 0.012), (9, 0.021), (15, 0.046), (16, 0.093), (24, 0.133), (27, 0.012), (41, 0.026), (47, 0.026), (53, 0.038), (55, 0.018), (63, 0.011), (82, 0.054), (86, 0.035), (88, 0.01), (95, 0.019), (99, 0.349)]
simIndex simValue blogId blogTitle
same-blog 1 0.9829644 2008 andrew gelman stats-2013-09-04-Does it matter that a sample is unrepresentative? It depends on the size of the treatment interactions
Introduction: In my article about implausible p-values in psychology studies, I wrote: “Women Are More Likely to Wear Red or Pink at Peak Fertility,” by Alec Beall and Jessica Tracy, is based on two samples: a self-selected sample of 100 women from the Internet, and 24 undergraduates at the University of British Columbia. . . . [There is a problem with] representativeness. What color clothing you wear has a lot to do with where you live and who you hang out with. Participants in an Internet survey and University of British Columbia students aren’t particularly representative of much more than … participants in an Internet survey and University of British Columbia students. In response, I received this in an email from a prominent psychology researcher (not someone I know personally): Complaining that subjects in an experiment were not randomly sampled is what freshmen do before they take their first psychology class. I really *hope* you why that is an absurd criticism – especially of au
2 0.97578084 2313 andrew gelman stats-2014-04-30-Seth Roberts
Introduction: I met Seth back in the early 1990s when we were both professors at the University of California. He sometimes came to the statistics department seminar and we got to talking about various things; in particular we shared an interest in statistical graphics. Much of my work in this direction eventually went toward the use of graphical displays to understand fitted models. Seth went in another direction and got interested in the role of exploratory data analysis in science, the idea that we could use graphs not just to test or even understand a model but also as the source of new hypotheses. We continued to discuss these issues over the years; see here , for example. At some point when we were at Berkeley the administration was encouraging the faculty to teach freshman seminars, and I had the idea of teaching a course on left-handedness. I’d just read the book by Stanley Coren and thought it would be fun to go through it with a class, chapter by chapter. But my knowledge of psych
Introduction: Last week I published in Slate a critique of a paper that appeared in the journal Psychological Science. That paper, by Alec Beall and Jessica Tracy, found that women who were at peak fertility were three times more likely to wear red or pink shirts, compared to women at other points in their menstrual cycles. The study was based an 100 participants on the internet and 24 college students. In my critique, I argued that we had no reason to believe the results generalized to the larger population, because (1) the samples were not representative, (2) the measurements were noisy, (3) the researchers did not use the correct dates of peak fertility, and (4) there were many different comparisons that could have been reported in the data, so there was nothing special about a particular comparison being statistically significant. I likened their paper to other work which I considered flawed for multiple comparisons (too many researcher degrees of freedom), including a claimed relation bet
4 0.9743126 702 andrew gelman stats-2011-05-09-“Discovered: the genetic secret of a happy life”
Introduction: I took the above headline from a news article in the (London) Independent by Jeremy Laurance reporting a study by Jan-Emmanuel De Neve, James Fowler, and Bruno Frey that reportedly just appeared in the Journal of Human Genetics. One of the pleasures of blogging is that I can go beyond the usual journalistic approaches to such a story: (a) puffing it, (b) debunking it, (c) reporting it completely flatly. Even convex combinations of (a), (b), (c) do not allow what I’d like to do, which is to explore the claims and follow wherever my exploration takes me. (And one of the pleasures of building my own audience is that I don’t need to endlessly explain background detail as was needed on a general-public site such as 538.) OK, back to the genetic secret of a happy life. Or, in the words the authors of the study, a gene that “explains less than one percent of the variation in life satisfaction.” “The genetic secret” or “less than one percent of the variation”? Perhaps the secre
Introduction: Jessica Tracy and Alec Beall, authors of that paper that claimed that women at peak fertility were more likely to wear red or pink shirts (see further discussion here and here ), and then a later paper that claimed that this happens in some weather but not others, just informed me that they have posted a note in disagreement with an paper by Eric Loken and myself. Our paper is unpublished, but I do have the megaphone of this blog, and Tracy and Beall do not, so I think it’s only fair to link to their note right away. I’ll quote from their note (but if you’re interested, please follow the link and read the whole thing ) and then give some background and my own reaction. Tracy and Beall write: Although Gelman and Loken are using our work as an example of a broader problem that pervades the field–a problem we generally agree about–we are concerned that readers will take their speculations about our methods and analyses as factual claims about our scientific integrity. Fu
6 0.97403204 1553 andrew gelman stats-2012-10-30-Real rothko, fake rothko
7 0.97346091 67 andrew gelman stats-2010-06-03-More on that Dartmouth health care study
8 0.97328413 1169 andrew gelman stats-2012-02-15-Charles Murray on the new upper class
11 0.97287077 2177 andrew gelman stats-2014-01-19-“The British amateur who debunked the mathematics of happiness”
12 0.97286135 2245 andrew gelman stats-2014-03-12-More on publishing in journals
13 0.97269619 983 andrew gelman stats-2011-10-31-Skepticism about skepticism of global warming skepticism skepticism
14 0.97254026 2114 andrew gelman stats-2013-11-26-“Please make fun of this claim”
15 0.97251606 110 andrew gelman stats-2010-06-26-Philosophy and the practice of Bayesian statistics
16 0.97245431 532 andrew gelman stats-2011-01-23-My Wall Street Journal story
17 0.97233075 1878 andrew gelman stats-2013-05-31-How to fix the tabloids? Toward replicable social science research
18 0.97219491 94 andrew gelman stats-2010-06-17-SAT stories
19 0.97183961 2137 andrew gelman stats-2013-12-17-Replication backlash
20 0.97178781 1435 andrew gelman stats-2012-07-30-Retracted articles and unethical behavior in economics journals?