andrew_gelman_stats andrew_gelman_stats-2014 andrew_gelman_stats-2014-2336 knowledge-graph by maker-knowledge-mining
Source: html
Introduction: Hey, we all know the answer: “correlation does not imply causation”—but of course life is more complicated than that. As philosophers, economists, statisticians, and others have repeatedly noted, most of our information about the world is observational not experimental, yet we manage to draw causal conclusions all the time. Sure, some of these conclusions are wrong (more often than 5% of the time, I’m sure) but that’s an accepted part of life. Challenges in this regard arise in the design of a study, in the statistical analysis, in how you write it up for a peer-reviewed journal, and finally in how you present it to the world. School sports and life outcomes An interesting case of all this came up recently in a post on Freakonomics that pointed to a post on Deadspin that pointed to a research article . The claim was that “sports participation [in high school] causes women to be less likely to be religious . . . more likely to have children . . . more likely to be singl
sentIndex sentText sentNum sentScore
1 School sports and life outcomes An interesting case of all this came up recently in a post on Freakonomics that pointed to a post on Deadspin that pointed to a research article . [sent-5, score-0.475]
2 The claim was that “sports participation [in high school] causes women to be less likely to be religious . [sent-6, score-0.922]
3 But, as the authors of the paper (Phoebe Clarke and Ian Ayres) explain in their blog posts and the scholarly article, they’re not measuring the effects of sports participation directly. [sent-15, score-0.937]
4 in which variation in rates of boys’ athletic participation across states before the passage of Title IX is used to instrument for changes in girls’ athletic participation following its passage . [sent-22, score-1.304]
5 More specifically, their analysis is “comparing women in states with greater levels of 1971 male [high school] sports participation . [sent-26, score-1.289]
6 to women in states with lower levels of 1971 male sports participation. [sent-29, score-0.862]
7 ” So they’re doing their best to a target their analysis on the group of women who’d be affected by the treatment. [sent-31, score-0.28]
8 Then they run individual-level regressions on binary variables (just as a minor point, I’d prefer to keep the original ordered responses; not a big deal but it can’t hurt), but the action is all coming from the state-level predictor (the measure of male athletic participation in 1971, by state). [sent-32, score-0.579]
9 Suppose some states were randomly selected to get the Title IX treatment and some were not. [sent-36, score-0.217]
10 Finally, of course the interaction being studied is not random; there are systematic differences between states with higher and lower boys’ high school sports participation in 1971. [sent-41, score-1.165]
11 Clarke and Ayres are measuring correlations and giving them a causal interpretation. [sent-44, score-0.262]
12 As noted above, if a ten percentage-point increase in state-level female sports participation is associated with a five percentage-point increase in the proportion of women who are mothers, there’s no way that most of this can be coming from a direct effect. [sent-60, score-1.343]
13 The implication would be that there’s this huge group of girls who (i) will have children if they do sports, and (ii) will not have children if they do not do sports. [sent-61, score-0.395]
14 Clearly these estimated elasticities have to be driven by big differences between states that possibly have nothing to do with high school sports. [sent-62, score-0.404]
15 Finally, if you’re interested in the substantive questions about the effects of sports participation, I think it’s essential to make a connection to whatever is already known in this field. [sent-69, score-0.461]
16 Sure, survey data have their limitations: as Clarke and Ayres note, kids select into sports participation. [sent-70, score-0.397]
17 The article includes the following footnote: It is true that many successful women with professional careers, such as Sheryl Sandberg and Brandi Chastain, are married. [sent-75, score-0.201]
18 Women who participate in sports may “reject marriage” by getting divorces when they find themselves in unhappy marriages. [sent-77, score-0.397]
19 It may well be true that, at the individual level, “sports participation causes women to be less religious, more likely to have children, and, if they do have children, more likely to be single mothers,” even if the actual effects are an order of magnitude lower than claimed. [sent-85, score-1.122]
20 Recall that if we were studying state-level correlations of income and voting, we’d come to the false conclusion that poor people are more likely to vote Republican. [sent-87, score-0.193]
wordName wordTfidf (topN-words)
[('participation', 0.401), ('sports', 0.397), ('ix', 0.226), ('women', 0.201), ('ayres', 0.195), ('clarke', 0.165), ('mothers', 0.149), ('children', 0.143), ('states', 0.13), ('boys', 0.111), ('school', 0.111), ('likely', 0.105), ('limitations', 0.103), ('causal', 0.099), ('athletic', 0.097), ('elasticities', 0.09), ('sandberg', 0.09), ('sheryl', 0.09), ('title', 0.09), ('correlations', 0.088), ('treatment', 0.087), ('male', 0.081), ('analysis', 0.079), ('outcomes', 0.078), ('female', 0.078), ('measuring', 0.075), ('variation', 0.074), ('high', 0.073), ('secular', 0.072), ('causes', 0.071), ('increase', 0.071), ('religious', 0.071), ('dakota', 0.07), ('proportion', 0.066), ('effects', 0.064), ('individual', 0.061), ('single', 0.061), ('scatterplot', 0.059), ('five', 0.058), ('speculation', 0.056), ('girls', 0.056), ('instrumental', 0.054), ('lower', 0.053), ('huge', 0.053), ('rise', 0.052), ('passage', 0.052), ('freakonomics', 0.051), ('aggregate', 0.05), ('finally', 0.05), ('sure', 0.049)]
simIndex simValue blogId blogTitle
same-blog 1 1.0 2336 andrew gelman stats-2014-05-16-How much can we learn about individual-level causal claims from state-level correlations?
Introduction: Hey, we all know the answer: “correlation does not imply causation”—but of course life is more complicated than that. As philosophers, economists, statisticians, and others have repeatedly noted, most of our information about the world is observational not experimental, yet we manage to draw causal conclusions all the time. Sure, some of these conclusions are wrong (more often than 5% of the time, I’m sure) but that’s an accepted part of life. Challenges in this regard arise in the design of a study, in the statistical analysis, in how you write it up for a peer-reviewed journal, and finally in how you present it to the world. School sports and life outcomes An interesting case of all this came up recently in a post on Freakonomics that pointed to a post on Deadspin that pointed to a research article . The claim was that “sports participation [in high school] causes women to be less likely to be religious . . . more likely to have children . . . more likely to be singl
2 0.29776913 1173 andrew gelman stats-2012-02-17-Sports examples in class
Introduction: Karl Broman writes : I [Karl] personally would avoid sports entirely, as I view the subject to be insufficiently serious. . . . Certainly lots of statisticians are interested in sports. . . . And I’m not completely uninterested in sports: I like to watch football, particularly Nebraska, Green Bay, and Baltimore, and to see Notre Dame or any team from Florida or Texas lose. But statistics about sports? Yawn. As a person who loves sports, statistics, and sports statistics, I have a few thoughts: 1. Not everyone likes sports, and even fewer are interested in any particular sport. It’s ok to use sports examples, but don’t delude yourself into thinking that everyone in the class cares about it. 2. Don’t forget foreign students. A lot of them don’t even know the rules of kickball, fer chrissake! 3. Of the students who care about a sport, there will be a minority who really care. We had some serious basketball fans in our class last year. 4. I think the best solution
3 0.15989186 1860 andrew gelman stats-2013-05-17-How can statisticians help psychologists do their research better?
Introduction: I received two emails yesterday on related topics. First, Stephen Olivier pointed me to this post by Daniel Lakens, who wrote the following open call to statisticians: You would think that if you are passionate about statistics, then you want to help people to calculate them correctly in any way you can. . . . you’d think some statisticians would be interested in helping a poor mathematically challenged psychologist out by offering some practical advice. I’m the right person to ask this question, since I actually have written a lot of material that helps psychologists (and others) with their data analysis. But there clearly are communication difficulties, in that my work and that of other statisticians hasn’t reached Lakens. Sometimes the contributions of statisticians are made indirectly. For example, I wrote Bayesian Data Analysis, and then Kruschke wrote Doing Bayesian Data Analysis. Our statistics book made it possible for Kruschke to write his excellent book for psycholo
Introduction: In my article about implausible p-values in psychology studies, I wrote: “Women Are More Likely to Wear Red or Pink at Peak Fertility,” by Alec Beall and Jessica Tracy, is based on two samples: a self-selected sample of 100 women from the Internet, and 24 undergraduates at the University of British Columbia. . . . [There is a problem with] representativeness. What color clothing you wear has a lot to do with where you live and who you hang out with. Participants in an Internet survey and University of British Columbia students aren’t particularly representative of much more than … participants in an Internet survey and University of British Columbia students. In response, I received this in an email from a prominent psychology researcher (not someone I know personally): Complaining that subjects in an experiment were not randomly sampled is what freshmen do before they take their first psychology class. I really *hope* you why that is an absurd criticism – especially of au
5 0.14213163 1675 andrew gelman stats-2013-01-15-“10 Things You Need to Know About Causal Effects”
Introduction: Macartan Humphreys pointed me to this excellent guide . Here are the 10 items: 1. A causal claim is a statement about what didn’t happen. 2. There is a fundamental problem of causal inference. 3. You can estimate average causal effects even if you cannot observe any individual causal effects. 4. If you know that, on average, A causes B and that B causes C, this does not mean that you know that A causes C. 5. The counterfactual model is all about contribution, not attribution. 6. X can cause Y even if there is no “causal path” connecting X and Y. 7. Correlation is not causation. 8. X can cause Y even if X is not a necessary condition or a sufficient condition for Y. 9. Estimating average causal effects does not require that treatment and control groups are identical. 10. There is no causation without manipulation. The article follows with crisp discussions of each point. My favorite is item #6, not because it’s the most important but because it brings in some real s
6 0.14002207 2236 andrew gelman stats-2014-03-07-Selection bias in the reporting of shaky research
7 0.13891438 1215 andrew gelman stats-2012-03-16-The “hot hand” and problems with hypothesis testing
8 0.13870949 2243 andrew gelman stats-2014-03-11-The myth of the myth of the myth of the hot hand
9 0.12565672 1114 andrew gelman stats-2012-01-12-Controversy about average personality differences between men and women
10 0.12481177 263 andrew gelman stats-2010-09-08-The China Study: fact or fallacy?
12 0.11613303 1226 andrew gelman stats-2012-03-22-Story time meets the all-else-equal fallacy and the fallacy of measurement
14 0.11295539 1893 andrew gelman stats-2013-06-11-Folic acid and autism
16 0.10971519 755 andrew gelman stats-2011-06-09-Recently in the award-winning sister blog
17 0.10898238 797 andrew gelman stats-2011-07-11-How do we evaluate a new and wacky claim?
18 0.10846644 803 andrew gelman stats-2011-07-14-Subtleties with measurement-error models for the evaluation of wacky claims
19 0.10721587 32 andrew gelman stats-2010-05-14-Causal inference in economics
20 0.10581428 2255 andrew gelman stats-2014-03-19-How Americans vote
topicId topicWeight
[(0, 0.248), (1, -0.045), (2, 0.087), (3, -0.107), (4, 0.024), (5, -0.004), (6, -0.016), (7, 0.036), (8, 0.035), (9, 0.049), (10, -0.066), (11, 0.022), (12, 0.007), (13, -0.011), (14, 0.03), (15, 0.028), (16, 0.018), (17, 0.063), (18, -0.034), (19, 0.061), (20, -0.056), (21, -0.042), (22, -0.002), (23, -0.035), (24, 0.06), (25, 0.064), (26, -0.045), (27, -0.024), (28, -0.015), (29, -0.019), (30, 0.004), (31, -0.016), (32, -0.007), (33, 0.014), (34, 0.021), (35, 0.02), (36, 0.037), (37, -0.024), (38, -0.04), (39, -0.016), (40, -0.044), (41, 0.013), (42, -0.002), (43, -0.055), (44, -0.005), (45, 0.011), (46, -0.024), (47, -0.009), (48, -0.014), (49, 0.005)]
simIndex simValue blogId blogTitle
same-blog 1 0.96462345 2336 andrew gelman stats-2014-05-16-How much can we learn about individual-level causal claims from state-level correlations?
Introduction: Hey, we all know the answer: “correlation does not imply causation”—but of course life is more complicated than that. As philosophers, economists, statisticians, and others have repeatedly noted, most of our information about the world is observational not experimental, yet we manage to draw causal conclusions all the time. Sure, some of these conclusions are wrong (more often than 5% of the time, I’m sure) but that’s an accepted part of life. Challenges in this regard arise in the design of a study, in the statistical analysis, in how you write it up for a peer-reviewed journal, and finally in how you present it to the world. School sports and life outcomes An interesting case of all this came up recently in a post on Freakonomics that pointed to a post on Deadspin that pointed to a research article . The claim was that “sports participation [in high school] causes women to be less likely to be religious . . . more likely to have children . . . more likely to be singl
2 0.84925151 1910 andrew gelman stats-2013-06-22-Struggles over the criticism of the “cannabis users and IQ change” paper
Introduction: Ole Rogeberg points me to a discussion of a discussion of a paper: Did pre-release of my [Rogeberg's] PNAS paper on methodological problems with Meier et al’s 2012 paper on cannabis and IQ reduce the chances that it will have its intended effect? In my case, serious methodological issues related to causal inference from non-random observational data became framed as a conflict over conclusions, forcing the original research team to respond rapidly and insufficiently to my concerns, and prompting them to defend their conclusions and original paper in a way that makes a later, more comprehensive reanalysis of their data less likely. This fits with a recurring theme on this blog: the defensiveness of researchers who don’t want to admit they were wrong. Setting aside cases of outright fraud and plagiarism, I think the worst case remains that of psychologists Neil Anderson and Deniz Ones, who denied any problems even in the presence of a smoking gun of a graph revealing their data
3 0.80998689 1793 andrew gelman stats-2013-04-08-The Supreme Court meets the fallacy of the one-sided bet
Introduction: Doug Hartmann writes ( link from Jay Livingston): Justice Antonin Scalia’s comment in the Supreme Court hearings on the U.S. law defining marriage that “there’s considerable disagreement among sociologists as to what the consequences of raising a child in a single-sex family, whether that is harmful to the child or not.” Hartman argues that Scalia is factually incorrect—there is not actually “considerable disagreement among sociologists” on this issue—and quotes a recent report from the American Sociological Association to this effect. Assuming there’s no other considerable group of sociologists (Hartman knows of only one small group) arguing otherwise, it seems that Hartman has a point. Scalia would’ve been better off omitting the phrase “among sociologists”—then he’d have been on safe ground, because you can always find somebody to take a position on the issue. Jerry Falwell’s no longer around but there’s a lot more where he came from. Even among scientists, there’s
Introduction: In my article about implausible p-values in psychology studies, I wrote: “Women Are More Likely to Wear Red or Pink at Peak Fertility,” by Alec Beall and Jessica Tracy, is based on two samples: a self-selected sample of 100 women from the Internet, and 24 undergraduates at the University of British Columbia. . . . [There is a problem with] representativeness. What color clothing you wear has a lot to do with where you live and who you hang out with. Participants in an Internet survey and University of British Columbia students aren’t particularly representative of much more than … participants in an Internet survey and University of British Columbia students. In response, I received this in an email from a prominent psychology researcher (not someone I know personally): Complaining that subjects in an experiment were not randomly sampled is what freshmen do before they take their first psychology class. I really *hope* you why that is an absurd criticism – especially of au
5 0.80218238 1226 andrew gelman stats-2012-03-22-Story time meets the all-else-equal fallacy and the fallacy of measurement
Introduction: Alex Tabarrok with a good catch : In Why Don’t Women Patent? , a recent NBER paper, Jennifer Hunt et al. [Jean-Philippe Garant, Hannah Herman, and David Munroe] present a stark fact: Only 5.5% of the holders of commercialized patents are women. One might think that this is explained by the relative lack of women with science and engineering degrees but Hunt et al. find that “women with such a degree are scarcely more likely to patent than women without.” Instead, most of the difference is “accounted for by differences among those with a science or engineering degree” especially the fact that women are underrepresented in patent-intensive fields such as electrical and mechanical engineering and in development and design. Predictably, the authors do not ask why women might self-selection into non patent-intensive fields, perhaps because this would require at least a discussion of politically incorrect questions . The failure to investigate these questions leads to some dubious co
8 0.78803408 561 andrew gelman stats-2011-02-06-Poverty, educational performance – and can be done about it
9 0.78214544 1114 andrew gelman stats-2012-01-12-Controversy about average personality differences between men and women
10 0.77659917 797 andrew gelman stats-2011-07-11-How do we evaluate a new and wacky claim?
11 0.77346915 1128 andrew gelman stats-2012-01-19-Sharon Begley: Worse than Stephen Jay Gould?
16 0.75340331 2196 andrew gelman stats-2014-02-03-One-way street fallacy again! in reporting of research on brothers and sisters
17 0.75262308 1996 andrew gelman stats-2013-08-24-All inference is about generalizing from sample to population
18 0.7498619 2204 andrew gelman stats-2014-02-09-Keli Liu and Xiao-Li Meng on Simpson’s paradox
19 0.7493549 393 andrew gelman stats-2010-11-04-Estimating the effect of A on B, and also the effect of B on A
20 0.74432778 1893 andrew gelman stats-2013-06-11-Folic acid and autism
topicId topicWeight
[(2, 0.128), (14, 0.039), (15, 0.015), (16, 0.065), (21, 0.039), (24, 0.135), (44, 0.018), (67, 0.011), (72, 0.041), (86, 0.011), (88, 0.023), (95, 0.045), (99, 0.316)]
simIndex simValue blogId blogTitle
1 0.97473538 1663 andrew gelman stats-2013-01-09-The effects of fiscal consolidation
Introduction: José Iparraguirre writes: I’ve read a recent paper by the International Monetary Fund on the effects of fiscal consolidation measures on income inequality (Fiscal Monitor October 2012, Appendix 1). They run a panel regression with 48 countries and 30 years (annual data) of a measure of income inequality (Gini coefficient) on a number of covariates, including a measure of fiscal consolidation. Footnote 39 (page 51) informs that they’ve employed seemingly unrelated regression and panel-corrected standard errors, and that to double-check they’ve also run ordinary least squares and fixed-effects panel regressions—all with similar results. So far, so good. However, the footnote goes on to explain that “Some of the results (e.g. the causal relationship between consolidation and inequality) may be subject to endogeneity and should be interpreted with caution”. (Italics are mine). Therefore, it seems that the crux of the exercise—i.e. estimating the relationship between fiscal con
same-blog 2 0.97473115 2336 andrew gelman stats-2014-05-16-How much can we learn about individual-level causal claims from state-level correlations?
Introduction: Hey, we all know the answer: “correlation does not imply causation”—but of course life is more complicated than that. As philosophers, economists, statisticians, and others have repeatedly noted, most of our information about the world is observational not experimental, yet we manage to draw causal conclusions all the time. Sure, some of these conclusions are wrong (more often than 5% of the time, I’m sure) but that’s an accepted part of life. Challenges in this regard arise in the design of a study, in the statistical analysis, in how you write it up for a peer-reviewed journal, and finally in how you present it to the world. School sports and life outcomes An interesting case of all this came up recently in a post on Freakonomics that pointed to a post on Deadspin that pointed to a research article . The claim was that “sports participation [in high school] causes women to be less likely to be religious . . . more likely to have children . . . more likely to be singl
3 0.96954364 1893 andrew gelman stats-2013-06-11-Folic acid and autism
Introduction: Aurelian Muntean writes: I have read an article on NPR and the journal article that spun this news. What draw my attention was the discussion in terms of causation implied by one of the authors of the article interviewed in the NPR news, and also by the conclusions of the article itself claiming large effects. Although the total sample (self-selecting pregnant women) seems very large (85,176) the subsamples (270 out of which 114 were in the sub-subsample revealing statistically significant association) used to support the analysis seem to be too small. Or not? My response: The different sources of information do seem to be in some conflict: - The JAMA article reports the autism rate of 1 per 1000 for children of mothers taking folic acid and 2 per 1000 for children of mothers not taking folic acid. (They also report the adjusted odds ratio as 0.6 rather than 0.5, indicted that the two groups differ a bit in some background variables.) - The NPR article has this q
4 0.96827769 1254 andrew gelman stats-2012-04-09-In the future, everyone will publish everything.
Introduction: Bob told me the other day (the other week, actually, as I’m stacking up posts here with a roughly one-month delay) that I shouldn’t try to compete with the electrical engineers when it comes to length of C.V.: according to Bob, these dudes can have over two thousand publications! How do they do it? First, an EE prof will have tons of graduate students and postdocs, they’re all writing papers and presenting at conferences, and they all stick his name on the author list. Second, these students and postdocs write up and publish every experiment they do . Including (especially!) computer experiments. And . . . all these people writing paper cite each other, so they quickly rack up thousands of citations. Upon hearing this, my first reaction to this was fear, plain and simple. One of the distinguishing characteristics of my own research record is that I have so many publications and citations. Those electrical engineers . . . how dare they go around devaluing my currency!
5 0.96329749 1171 andrew gelman stats-2012-02-16-“False-positive psychology”
Introduction: Everybody’s talkin bout this paper by Joseph Simmons, Leif Nelson and Uri Simonsohn, who write : Despite empirical psychologists’ nominal endorsement of a low rate of false-positive findings (≤ .05), flexibility in data collection, analysis, and reporting dramatically increases actual false-positive rates. In many cases, a researcher is more likely to falsely find evidence that an effect exists than to correctly find evidence that it does not. We [Simmons, Nelson, and Simonsohn] present computer simulations and a pair of actual experiments that demonstrate how unacceptably easy it is to accumulate (and report) statistically significant evidence for a false hypothesis. Second, we suggest a simple, low-cost, and straightforwardly effective disclosure-based solution to this problem. The solution involves six concrete requirements for authors and four guidelines for reviewers, all of which impose a minimal burden on the publication process. Whatever you think about these recommend
6 0.96320313 1872 andrew gelman stats-2013-05-27-More spam!
7 0.96313131 1189 andrew gelman stats-2012-02-28-Those darn physicists
8 0.96242094 1196 andrew gelman stats-2012-03-04-Piss-poor monocausal social science
9 0.96104562 1508 andrew gelman stats-2012-09-23-Speaking frankly
10 0.95832622 1017 andrew gelman stats-2011-11-18-Lack of complete overlap
11 0.95753312 27 andrew gelman stats-2010-05-11-Update on the spam email study
12 0.95580339 1954 andrew gelman stats-2013-07-24-Too Good To Be True: The Scientific Mass Production of Spurious Statistical Significance
13 0.95571375 549 andrew gelman stats-2011-02-01-“Roughly 90% of the increase in . . .” Hey, wait a minute!
14 0.954274 17 andrew gelman stats-2010-05-05-Taking philosophical arguments literally
16 0.9428224 542 andrew gelman stats-2011-01-28-Homework and treatment levels
17 0.94215173 352 andrew gelman stats-2010-10-19-Analysis of survey data: Design based models vs. hierarchical modeling?
18 0.93953741 1430 andrew gelman stats-2012-07-26-Some thoughts on survey weighting