andrew_gelman_stats andrew_gelman_stats-2013 andrew_gelman_stats-2013-1910 knowledge-graph by maker-knowledge-mining
Source: html
Introduction: Ole Rogeberg points me to a discussion of a discussion of a paper: Did pre-release of my [Rogeberg's] PNAS paper on methodological problems with Meier et al’s 2012 paper on cannabis and IQ reduce the chances that it will have its intended effect? In my case, serious methodological issues related to causal inference from non-random observational data became framed as a conflict over conclusions, forcing the original research team to respond rapidly and insufficiently to my concerns, and prompting them to defend their conclusions and original paper in a way that makes a later, more comprehensive reanalysis of their data less likely. This fits with a recurring theme on this blog: the defensiveness of researchers who don’t want to admit they were wrong. Setting aside cases of outright fraud and plagiarism, I think the worst case remains that of psychologists Neil Anderson and Deniz Ones, who denied any problems even in the presence of a smoking gun of a graph revealing their data
sentIndex sentText sentNum sentScore
1 Ole Rogeberg points me to a discussion of a discussion of a paper: Did pre-release of my [Rogeberg's] PNAS paper on methodological problems with Meier et al’s 2012 paper on cannabis and IQ reduce the chances that it will have its intended effect? [sent-1, score-0.571]
2 The researchers sorted people into groups according to whether or not they had used cannabis and according to the number of times they had been scored as dependent. [sent-10, score-0.819]
3 The cannabis-and-IQ analysis basically compares average changes in IQ across groups with different cannabis use patterns. [sent-19, score-0.655]
4 The central assumption, in other words, is that the groups would have had the same IQ-development if their cannabis use had been similar. [sent-21, score-0.705]
5 In the Dunedin study, they measured IQ at a number of ages, and average IQ changes in various periods could be shown for each group of cannabis users. [sent-27, score-0.64]
6 If the effect estimate of cannabis on IQ changes a lot, then this suggests that selection issues are important- and that confounders (both known and unknown) must be taken seriously. [sent-32, score-0.718]
7 Another example is the linear trend variable they use for cannabis exposure, which presumably gives a score of 1 to never users, 2 to users who were never dependent, 3 to those scored as dependent once and so on. [sent-39, score-0.732]
8 This is the variable that they check for significance – and it would be Provide other diagnostic analyses, for instance by considering the variance of the outcome variable within each treatment group (how much did IQ change differ within each treatment group? [sent-40, score-0.563]
9 For instance, let us say you identify what seems to be a causal effect of cannabis use and dependency, but its magnitude is strongly reduced (but not eliminated) when you add in various potential confounders. [sent-45, score-0.834]
10 As the authors of the original paper note (when education turns out to affect the effect size), education could be a mediating factor in the causal process whereby cannabis affects IQ. [sent-47, score-1.026]
11 However, this would mean that the permanent, neurotoxic effect they are most concerned with would be smaller, because part of the measured effect would be due to the effect of cannabis on education multiplied by the effect of this education on IQ. [sent-48, score-1.558]
12 It also suggests that we might want to look for evidence to assess how strongly cannabis use causally affects education, to better understand the determinants of this process. [sent-50, score-0.609]
13 For instance, even if there was only a temporary effect of cannabis on cognition, ongoing smoking would do more poorly in school or college, which might then influence later job prospects and long term IQ. [sent-51, score-0.857]
14 Put differently: If the mechanism is via school, then even transitory effects of cannabis becomes important when they coincide with the period of education. [sent-53, score-0.561]
15 Early onset cannabis use appeared to be correlated with a number of risk factors, and these risk factors were also correlated with poor life outcomes (low and poor education, crime, income etc. [sent-60, score-1.141]
16 Since the risk factors mentioned above would serve to push you away from such cognitively demanding environments, it seemed plausible that they would affect long term IQ negatively by pushing you into poorer environments than your initial IQ would have suggested. [sent-67, score-0.706]
17 Putting these pieces together, the risk factors that make you more likely to take up pot smoking in adolescence, and that raise your risk of becoming dependent, also shift you into poorer environments than your initial IQ would predict in isolation. [sent-76, score-0.782]
18 Additionally, these shifts are more likely for kids in lower-SES groups (since the risk factors are correlated with SES), and these also have an IQ more sensitive to environmental changes. [sent-77, score-0.596]
19 However, it should be obvious from my description of this mechanism that we should expect the mechanism to work even within a socioeconomic group: Even within this group, those with high levels of risk factors will experience poorer life outcomes, which may reduce their IQs. [sent-80, score-0.655]
20 This, of course, does not prove or establish that there is no effect of cannabis on IQ. [sent-85, score-0.743]
wordName wordTfidf (topN-words)
[('cannabis', 0.499), ('iq', 0.421), ('rogeberg', 0.175), ('effect', 0.171), ('dunedin', 0.166), ('ses', 0.149), ('factors', 0.137), ('environment', 0.12), ('risk', 0.118), ('environments', 0.11), ('instance', 0.099), ('groups', 0.098), ('poorer', 0.09), ('smoking', 0.088), ('education', 0.085), ('group', 0.082), ('original', 0.082), ('establish', 0.073), ('childhood', 0.072), ('methodological', 0.072), ('pot', 0.071), ('kids', 0.065), ('environmental', 0.064), ('dependent', 0.064), ('sensitive', 0.062), ('socioeconomic', 0.062), ('mechanism', 0.062), ('within', 0.062), ('scored', 0.06), ('number', 0.059), ('use', 0.058), ('study', 0.056), ('magnitude', 0.056), ('jailtime', 0.055), ('neurotoxic', 0.055), ('researchers', 0.053), ('treatment', 0.053), ('affect', 0.052), ('correlated', 0.052), ('affects', 0.052), ('variable', 0.051), ('sorted', 0.05), ('owes', 0.05), ('meier', 0.05), ('potential', 0.05), ('would', 0.05), ('seemed', 0.049), ('school', 0.049), ('outcomes', 0.048), ('confounders', 0.048)]
simIndex simValue blogId blogTitle
same-blog 1 1.0000004 1910 andrew gelman stats-2013-06-22-Struggles over the criticism of the “cannabis users and IQ change” paper
Introduction: Ole Rogeberg points me to a discussion of a discussion of a paper: Did pre-release of my [Rogeberg's] PNAS paper on methodological problems with Meier et al’s 2012 paper on cannabis and IQ reduce the chances that it will have its intended effect? In my case, serious methodological issues related to causal inference from non-random observational data became framed as a conflict over conclusions, forcing the original research team to respond rapidly and insufficiently to my concerns, and prompting them to defend their conclusions and original paper in a way that makes a later, more comprehensive reanalysis of their data less likely. This fits with a recurring theme on this blog: the defensiveness of researchers who don’t want to admit they were wrong. Setting aside cases of outright fraud and plagiarism, I think the worst case remains that of psychologists Neil Anderson and Deniz Ones, who denied any problems even in the presence of a smoking gun of a graph revealing their data
2 0.25297454 561 andrew gelman stats-2011-02-06-Poverty, educational performance – and can be done about it
Introduction: Andrew has pointed to Jonathan Livengood’s analysis of the correlation between poverty and PISA results, whereby schools with poorer students get poorer test results. I’d have written a comment, but then I couldn’t have inserted a chart. Andrew points out that a causal analysis is needed. This reminds me of an intervention that has been done before: take a child out of poverty, and bring him up in a better-off family. What’s going to happen? There have been several studies examining correlations between adoptive and biological parents’ IQ (assuming IQ is a test analogous to the math and verbal tests, and that parent IQ is analogous to the quality of instruction – but the point is in the analysis not in the metric). This is the result (from Adoption Strategies by Robin P Corley in Encyclopedia of Life Sciences): So, while it did make a difference at an early age, with increasing age of the adopted child, the intelligence of adoptive parents might not be making any difference
3 0.19026634 1620 andrew gelman stats-2012-12-12-“Teaching effectiveness” as another dimension in cognitive ability
Introduction: I’m not a great teacher. I can get by because I work hard and I know a lot, and for some students my classes are just great, but it’s not a natural talent of mine. I know people who are amazing teachers, and they have something that I just don’t have. I wrote that book, Teaching Statistics: A Bag of Tricks (with Deb Nolan) because I’m not a good teacher and hence need to develop all sorts of techniques to be able to do what good teachers can do without even trying. I’m not proud of being mediocre at teaching. I don’t think that low teaching skill is some sort of indicator that I’m a great researcher. The other think about teaching ability is that I think it’s hard to detect without actually seeing someone teach a class. If you see me give a seminar presentation or even a guest lecture, you’d think I’m an awesome teacher. But, actually, no. I’m an excellent speaker, not such a great teacher. This all came to mind when I received the following email from anthropologist Hen
4 0.18378174 2363 andrew gelman stats-2014-06-07-“Does researching casual marijuana use cause brain abnormalities?”
Introduction: David Austin points me to a wonderfully-titled post by Lior Pachter criticizing a recent paper on the purported effects of cannabis use. Not the paper criticized here . Someone should send this all to David Brooks. I’ve heard he’s interested in the latest scientific findings, and I know he’s interested in marijuana.
5 0.15990552 257 andrew gelman stats-2010-09-04-Question about standard range for social science correlations
Introduction: Andrew Eppig writes: I’m a physicist by training who is transitioning to the social sciences. I recently came across a reference in the Economist to a paper on IQ and parasites which I read as I have more than a passing interest in IQ research (having read much that you and others (e.g., Shalizi, Wicherts) have written). In this paper I note that the authors find a very high correlation between national IQ and parasite prevalence. The strength of the correlation (-0.76 to -0.82) surprised me, as I’m used to much weaker correlations in the social sciences. To me, it’s a bit too high, suggesting that there are other factors at play or that one of the variables is merely a proxy for a large number of other variables. But I have no basis for this other than a gut feeling and a memory of a plot on Language Log about the distribution of correlation coefficients in social psychology. So my question is this: Is a correlation in the range of (-0.82,-0.76) more likely to be a correlatio
6 0.15980411 1184 andrew gelman stats-2012-02-25-Facebook Profiles as Predictors of Job Performance? Maybe…but not yet.
7 0.13522655 803 andrew gelman stats-2011-07-14-Subtleties with measurement-error models for the evaluation of wacky claims
8 0.12855315 797 andrew gelman stats-2011-07-11-How do we evaluate a new and wacky claim?
10 0.12363604 7 andrew gelman stats-2010-04-27-Should Mister P be allowed-encouraged to reside in counter-factual populations?
11 0.11699984 888 andrew gelman stats-2011-09-03-A psychology researcher asks: Is Anova dead?
12 0.11080459 2223 andrew gelman stats-2014-02-24-“Edlin’s rule” for routinely scaling down published estimates
13 0.1106941 709 andrew gelman stats-2011-05-13-D. Kahneman serves up a wacky counterfactual
15 0.10674002 1957 andrew gelman stats-2013-07-26-“The Inside Story Of The Harvard Dissertation That Became Too Racist For Heritage”
16 0.10487413 2328 andrew gelman stats-2014-05-10-What property is important in a risk prediction model? Discrimination or calibration?
17 0.10373465 1086 andrew gelman stats-2011-12-27-The most dangerous jobs in America
18 0.10369228 1209 andrew gelman stats-2012-03-12-As a Bayesian I want scientists to report their data non-Bayesianly
20 0.10263354 388 andrew gelman stats-2010-11-01-The placebo effect in pharma
topicId topicWeight
[(0, 0.248), (1, 0.009), (2, 0.084), (3, -0.149), (4, 0.016), (5, -0.001), (6, 0.004), (7, 0.013), (8, -0.015), (9, 0.062), (10, -0.063), (11, 0.037), (12, 0.02), (13, -0.037), (14, 0.053), (15, 0.045), (16, 0.04), (17, 0.013), (18, -0.016), (19, 0.041), (20, -0.037), (21, 0.019), (22, -0.002), (23, -0.041), (24, 0.019), (25, 0.038), (26, -0.005), (27, 0.022), (28, -0.016), (29, 0.0), (30, -0.017), (31, 0.001), (32, 0.011), (33, 0.029), (34, 0.017), (35, 0.02), (36, 0.035), (37, -0.012), (38, 0.045), (39, -0.005), (40, -0.014), (41, -0.017), (42, 0.002), (43, -0.03), (44, 0.004), (45, -0.042), (46, -0.014), (47, -0.005), (48, 0.015), (49, 0.036)]
simIndex simValue blogId blogTitle
same-blog 1 0.97123092 1910 andrew gelman stats-2013-06-22-Struggles over the criticism of the “cannabis users and IQ change” paper
Introduction: Ole Rogeberg points me to a discussion of a discussion of a paper: Did pre-release of my [Rogeberg's] PNAS paper on methodological problems with Meier et al’s 2012 paper on cannabis and IQ reduce the chances that it will have its intended effect? In my case, serious methodological issues related to causal inference from non-random observational data became framed as a conflict over conclusions, forcing the original research team to respond rapidly and insufficiently to my concerns, and prompting them to defend their conclusions and original paper in a way that makes a later, more comprehensive reanalysis of their data less likely. This fits with a recurring theme on this blog: the defensiveness of researchers who don’t want to admit they were wrong. Setting aside cases of outright fraud and plagiarism, I think the worst case remains that of psychologists Neil Anderson and Deniz Ones, who denied any problems even in the presence of a smoking gun of a graph revealing their data
2 0.84742314 561 andrew gelman stats-2011-02-06-Poverty, educational performance – and can be done about it
Introduction: Andrew has pointed to Jonathan Livengood’s analysis of the correlation between poverty and PISA results, whereby schools with poorer students get poorer test results. I’d have written a comment, but then I couldn’t have inserted a chart. Andrew points out that a causal analysis is needed. This reminds me of an intervention that has been done before: take a child out of poverty, and bring him up in a better-off family. What’s going to happen? There have been several studies examining correlations between adoptive and biological parents’ IQ (assuming IQ is a test analogous to the math and verbal tests, and that parent IQ is analogous to the quality of instruction – but the point is in the analysis not in the metric). This is the result (from Adoption Strategies by Robin P Corley in Encyclopedia of Life Sciences): So, while it did make a difference at an early age, with increasing age of the adopted child, the intelligence of adoptive parents might not be making any difference
Introduction: Hey, we all know the answer: “correlation does not imply causation”—but of course life is more complicated than that. As philosophers, economists, statisticians, and others have repeatedly noted, most of our information about the world is observational not experimental, yet we manage to draw causal conclusions all the time. Sure, some of these conclusions are wrong (more often than 5% of the time, I’m sure) but that’s an accepted part of life. Challenges in this regard arise in the design of a study, in the statistical analysis, in how you write it up for a peer-reviewed journal, and finally in how you present it to the world. School sports and life outcomes An interesting case of all this came up recently in a post on Freakonomics that pointed to a post on Deadspin that pointed to a research article . The claim was that “sports participation [in high school] causes women to be less likely to be religious . . . more likely to have children . . . more likely to be singl
4 0.82761461 1929 andrew gelman stats-2013-07-07-Stereotype threat!
Introduction: Colleen Ganley, Leigh Mingle, Allison Ryan, Katherine Ryan, Marian Vasilyeva, and Michelle Perry write : Stereotype threat has been proposed as 1 potential explanation for the gender difference in standardized mathematics test performance among high-performing students. At present, it is not entirely clear how susceptibility to stereotype threat develops, as empirical evidence for stereotype threat effects across the school years is inconsistent. In a series of 3 studies, with a total sample of 931 students, we investigated stereotype threat effects during childhood and adolescence. Three activation methods were used, ranging from implicit to explicit. Across studies, we found no evidence that the mathematics performance of school-age girls was impacted by stereotype threat. In 2 of the studies, there were gender differences on the mathematics assessment regardless of whether stereotype threat was activated. Potential reasons for these findings are discussed, including the possibil
5 0.82390553 1793 andrew gelman stats-2013-04-08-The Supreme Court meets the fallacy of the one-sided bet
Introduction: Doug Hartmann writes ( link from Jay Livingston): Justice Antonin Scalia’s comment in the Supreme Court hearings on the U.S. law defining marriage that “there’s considerable disagreement among sociologists as to what the consequences of raising a child in a single-sex family, whether that is harmful to the child or not.” Hartman argues that Scalia is factually incorrect—there is not actually “considerable disagreement among sociologists” on this issue—and quotes a recent report from the American Sociological Association to this effect. Assuming there’s no other considerable group of sociologists (Hartman knows of only one small group) arguing otherwise, it seems that Hartman has a point. Scalia would’ve been better off omitting the phrase “among sociologists”—then he’d have been on safe ground, because you can always find somebody to take a position on the issue. Jerry Falwell’s no longer around but there’s a lot more where he came from. Even among scientists, there’s
9 0.79734683 2223 andrew gelman stats-2014-02-24-“Edlin’s rule” for routinely scaling down published estimates
10 0.77973795 1150 andrew gelman stats-2012-02-02-The inevitable problems with statistical significance and 95% intervals
11 0.77623028 1114 andrew gelman stats-2012-01-12-Controversy about average personality differences between men and women
12 0.77272034 2114 andrew gelman stats-2013-11-26-“Please make fun of this claim”
13 0.77071977 1893 andrew gelman stats-2013-06-11-Folic acid and autism
14 0.76973909 303 andrew gelman stats-2010-09-28-“Genomics” vs. genetics
15 0.76798952 94 andrew gelman stats-2010-06-17-SAT stories
19 0.76090056 32 andrew gelman stats-2010-05-14-Causal inference in economics
topicId topicWeight
[(9, 0.032), (13, 0.021), (15, 0.035), (16, 0.074), (21, 0.04), (22, 0.013), (24, 0.128), (47, 0.01), (48, 0.045), (55, 0.017), (60, 0.014), (61, 0.036), (84, 0.02), (86, 0.063), (89, 0.011), (95, 0.021), (99, 0.266)]
simIndex simValue blogId blogTitle
same-blog 1 0.97788954 1910 andrew gelman stats-2013-06-22-Struggles over the criticism of the “cannabis users and IQ change” paper
Introduction: Ole Rogeberg points me to a discussion of a discussion of a paper: Did pre-release of my [Rogeberg's] PNAS paper on methodological problems with Meier et al’s 2012 paper on cannabis and IQ reduce the chances that it will have its intended effect? In my case, serious methodological issues related to causal inference from non-random observational data became framed as a conflict over conclusions, forcing the original research team to respond rapidly and insufficiently to my concerns, and prompting them to defend their conclusions and original paper in a way that makes a later, more comprehensive reanalysis of their data less likely. This fits with a recurring theme on this blog: the defensiveness of researchers who don’t want to admit they were wrong. Setting aside cases of outright fraud and plagiarism, I think the worst case remains that of psychologists Neil Anderson and Deniz Ones, who denied any problems even in the presence of a smoking gun of a graph revealing their data
Introduction: Etienne LeBel writes: You’ve probably already seen it, but I thought you could have a lot of fun with this one!! The article , with the admirably clear title given above, is by James McNulty, Michael Olson, Andrea Meltzer, Matthew Shaffer, and begins as follows: For decades, social psychological theories have posited that the automatic processes captured by implicit measures have implications for social outcomes. Yet few studies have demonstrated any long-term implications of automatic processes, and some scholars have begun to question the relevance and even the validity of these theories. At baseline of our longitudinal study, 135 newlywed couples (270 individuals) completed an explicit measure of their conscious attitudes toward their relationship and an implicit measure of their automatic attitudes toward their partner. They then reported their marital satisfaction every 6 months for the next 4 years. We found no correlation between spouses’ automatic and conscious attitu
3 0.96468139 1980 andrew gelman stats-2013-08-13-Test scores and grades predict job performance (but maybe not at Google)
Introduction: Eric Loken writes : If you’re used to Google upending conventional wisdom, then yesterday’s interview with Laszlo Bock in the New York Times did not disappoint. Google has determined that test scores and transcripts are useless because they don’t predict performance among its employees. . . . I [Loken] am going to assume they’re well aware of the limits of their claim, and instead I’m going say that as readers of the interview we should not lose sight of a fundamental fact - Across a wide variety of employment settings, one of the most robust findings in organizational psychology is that tests of cognitive ability are strong predictors of job performance. If Google has found otherwise, what they have found is that grades and test scores are not predictive of performance at Google. In general, in the workplace tests are still highly predictive of success. If all the research says that test scores and grades predict performance, why would the people at Google want to igno
4 0.96110612 1518 andrew gelman stats-2012-10-02-Fighting a losing battle
Introduction: Following a recent email exchange regarding path sampling and thermodynamic integration (sadly, I’ve gotten rusty and haven’t thought seriously about these challenges for many years), a correspondent referred to the marginal distribution of the data under a model as “the evidence.” I hate that expression! As we discuss in chapter 6 of BDA, for continuous-parametered models, this quantity can be completely sensitive to aspects of the prior that have essentially no impact on the posterior. In the examples I’ve seen, this marginal probability is not “evidence” in any useful sense of the term. When I told this to my correspondent, he replied, I actually don’t find “the evidence” too bothersome. I don’t have BDA at home where I’m working from at the moment, so I’ll read up on chapter 6 later, but I assume you refer to the problem of the marginal likelihood being strongly sensitive to the prior in a way that the posterior typically isn’t, thereby diminishing the value of the margi
5 0.95975459 1278 andrew gelman stats-2012-04-23-“Any old map will do” meets “God is in every leaf of every tree”
Introduction: As a statistician I am particularly worried about the rhetorical power of anecdotes (even though I use them in my own reasoning; see discussion below). But much can be learned from a true anecdote. The rough edges—the places where the anecdote doesn’t fit your thesis—these are where you learn. We have recently had a discussion ( here and here ) of Karl Weick, a prominent scholar of business management who plagiarized a story and then went on to draw different lessons from the pilfered anecdote in several different publications published over many years. Setting aside an issues of plagiarism and rulebreaking, I argue that, by hiding the source of the story and changing its form, Weick and his management-science audience are losing their ability to get anything out of it beyond empty confirmation. A full discussion follows. 1. The lost Hungarian soldiers Thomas Basbøll (who has the unusual (to me) job of “writing consultant” at the Copenhagen Business School) has been
6 0.95901072 2281 andrew gelman stats-2014-04-04-The Notorious N.H.S.T. presents: Mo P-values Mo Problems
7 0.95877075 781 andrew gelman stats-2011-06-28-The holes in my philosophy of Bayesian data analysis
8 0.95875788 1266 andrew gelman stats-2012-04-16-Another day, another plagiarist
9 0.95760566 1435 andrew gelman stats-2012-07-30-Retracted articles and unethical behavior in economics journals?
10 0.95700115 2137 andrew gelman stats-2013-12-17-Replication backlash
11 0.95695603 1630 andrew gelman stats-2012-12-18-Postdoc positions at Microsoft Research – NYC
12 0.95688915 776 andrew gelman stats-2011-06-22-Deviance, DIC, AIC, cross-validation, etc
13 0.95635206 2058 andrew gelman stats-2013-10-11-Gladwell and Chabris, David and Goliath, and science writing as stone soup
14 0.95627218 2201 andrew gelman stats-2014-02-06-Bootstrap averaging: Examples where it works and where it doesn’t work
15 0.95604897 1162 andrew gelman stats-2012-02-11-Adding an error model to a deterministic model
16 0.95588434 1205 andrew gelman stats-2012-03-09-Coming to agreement on philosophy of statistics
17 0.95588386 797 andrew gelman stats-2011-07-11-How do we evaluate a new and wacky claim?
18 0.95569599 2057 andrew gelman stats-2013-10-10-Chris Chabris is irritated by Malcolm Gladwell
19 0.95537794 1247 andrew gelman stats-2012-04-05-More philosophy of Bayes
20 0.95525593 18 andrew gelman stats-2010-05-06-$63,000 worth of abusive research . . . or just a really stupid waste of time?