andrew_gelman_stats andrew_gelman_stats-2011 andrew_gelman_stats-2011-1074 knowledge-graph by maker-knowledge-mining
Source: html
Introduction: A journalist wrote to me recently: I was going to include your deconstruction of the beautiful daughters paper, but ran out of space. The author, incidentally, stands by that paper — and emailed me that you’d advised him on a later paper, implying that meant you now accepted the thesis! I responded: I know that Kanazawa stands by his earlier claim. Unfortunately, in this case, “stands by” means believing something with no evidence. And, yes, he send me a copy of his other paper, and I responded by telling him I thought his sample size is too small. I did advise him, and my advice was to not do it! To avoid any future confusion, I thought I’d post my half of my emails with Kanazawa over the years: 28 Aug 2006: Dear Dr. Kanazawa, I read with interest your papers on sex ratios in the Journal of Theoretical Biology. Sex ratios are an inherently interesting topic and also a favorite example for people such as myself who teach probability and statistics. In the cour
sentIndex sentText sentNum sentScore
1 The author, incidentally, stands by that paper — and emailed me that you’d advised him on a later paper, implying that meant you now accepted the thesis! [sent-2, score-0.425]
2 I responded: I know that Kanazawa stands by his earlier claim. [sent-3, score-0.18]
3 And, yes, he send me a copy of his other paper, and I responded by telling him I thought his sample size is too small. [sent-5, score-0.447]
4 To avoid any future confusion, I thought I’d post my half of my emails with Kanazawa over the years: 28 Aug 2006: Dear Dr. [sent-7, score-0.174]
5 Kanazawa, I read with interest your papers on sex ratios in the Journal of Theoretical Biology. [sent-8, score-0.305]
6 Sex ratios are an inherently interesting topic and also a favorite example for people such as myself who teach probability and statistics. [sent-9, score-0.17]
7 In the course of reading the articles, I noticed some potential statistical problems. [sent-10, score-0.155]
8 These problems do not necessarily invalidate your research findiings but they suggest some potential difficulties with the statistical interpretation. [sent-11, score-0.149]
9 I hope these comments will be useful to you in your future research. [sent-13, score-0.177]
10 I’m also cc-ing Andrew Oswald because I had some correspondence with him on the general issue of evaluating the statistics in speculative research articles, and I saw in the acknowledgments to your last paper that you had talked with him. [sent-16, score-0.333]
11 The topic is interesting; however, I still think your sample size is too small to detect any real patterns that might be there. [sent-20, score-0.396]
12 With n=4400 and 85% of the parents characterized as “attractive”, you have only 660 parents in your control group. [sent-21, score-0.188]
13 02, which tells us that you cannot distinguish effects of less than 2 percentage points, which are much less than would be expected to be found, based on my reading of the literature. [sent-24, score-0.167]
14 pdf I suppose that large effects of 2 percentage points or more are possible in your case, but they seem unlikely given my reading of the literature. [sent-32, score-0.167]
15 And, if effect sizes are small, it’s hard to learn much without a large sample size. [sent-33, score-0.292]
16 But you can rest assured that I did not cite the People data as a refutation of your hypothesis. [sent-36, score-0.137]
17 Rather, I stated that I thought the sample size from your studies was too small given the plausible effect sizes. [sent-37, score-0.539]
18 I used the People data as an amusing example of another dataset where the sample size was too small to find anything conclusive. [sent-38, score-0.396]
19 (Based on the People data we collected, the %girls was statistically indistinguishable from the population average. [sent-39, score-0.15]
20 However, I think it’s been clear for several years that his work on sex ratios is going nowhere. [sent-45, score-0.305]
wordName wordTfidf (topN-words)
[('kanazawa', 0.383), ('aug', 0.306), ('dear', 0.24), ('andrew', 0.19), ('stands', 0.18), ('satoshi', 0.176), ('ratios', 0.17), ('sample', 0.148), ('size', 0.143), ('sex', 0.135), ('paper', 0.11), ('future', 0.109), ('small', 0.105), ('parents', 0.094), ('responded', 0.091), ('acknowledgments', 0.088), ('paywall', 0.088), ('reading', 0.085), ('percentage', 0.082), ('invalidate', 0.079), ('effect', 0.078), ('gelman', 0.078), ('indistinguishable', 0.077), ('received', 0.074), ('advised', 0.074), ('statistically', 0.073), ('correspondence', 0.072), ('suppress', 0.072), ('refutation', 0.072), ('incidentally', 0.072), ('oswald', 0.072), ('potential', 0.07), ('hope', 0.068), ('nov', 0.068), ('advise', 0.067), ('daughters', 0.067), ('learn', 0.066), ('merit', 0.066), ('thought', 0.065), ('articles', 0.065), ('assured', 0.065), ('referee', 0.063), ('speculative', 0.063), ('luck', 0.062), ('mar', 0.062), ('calculated', 0.062), ('believing', 0.062), ('returned', 0.061), ('declined', 0.061), ('implying', 0.061)]
simIndex simValue blogId blogTitle
same-blog 1 1.0000002 1074 andrew gelman stats-2011-12-20-Reading a research paper != agreeing with its claims
Introduction: A journalist wrote to me recently: I was going to include your deconstruction of the beautiful daughters paper, but ran out of space. The author, incidentally, stands by that paper — and emailed me that you’d advised him on a later paper, implying that meant you now accepted the thesis! I responded: I know that Kanazawa stands by his earlier claim. Unfortunately, in this case, “stands by” means believing something with no evidence. And, yes, he send me a copy of his other paper, and I responded by telling him I thought his sample size is too small. I did advise him, and my advice was to not do it! To avoid any future confusion, I thought I’d post my half of my emails with Kanazawa over the years: 28 Aug 2006: Dear Dr. Kanazawa, I read with interest your papers on sex ratios in the Journal of Theoretical Biology. Sex ratios are an inherently interesting topic and also a favorite example for people such as myself who teach probability and statistics. In the cour
Introduction: The other day we discussed that paper on ovulation and voting (you may recall that the authors reported a scattered bunch of comparisons, significance tests, and p-values, and I recommended that they would’ve done better to simply report complete summaries of their data, so that readers could see the comparisons of interest in full context), and I was thinking a bit more about why I was so bothered that it was published in Psychological Science, which I’d thought of as a serious research journal. My concern isn’t just that that the paper is bad—after all, lots of bad papers get published—but rather that it had nothing really going for it, except that it was headline bait. It was a survey done on Mechanical Turk, that’s it. No clever design, no clever questions, no care in dealing with nonresponse problems, no innovative data analysis, no nothing. The paper had nothing to offer, except that it had no obvious flaws. Psychology is a huge field full of brilliant researchers.
3 0.16365501 963 andrew gelman stats-2011-10-18-Question on Type M errors
Introduction: Inti Pedroso writes: Today during the group meeting at my new job we were revising a paper whose main conclusions were sustained by an ANOVA. One of the first observations is that the experiment had a small sample size. Interestingly (may not so), some of the reported effects (most of them interactions) were quite large. One of the experience group members said that “there is a common wisdom that one should not believe effects from small sample sizes but [he thinks] if they [the effects] are large enough to be picked on a small study they must be real large effects”. I argued that if the sample size is small one could incur on a M-type error in which the magnitude of the effect is being over-estimated and that if larger samples are evaluated the magnitude may become smaller and also the confidence intervals. The concept of M-type error is completely new to all other members of the group (on which I am in my second week) and I was given the job of finding a suitable ref to explain
4 0.1511623 2157 andrew gelman stats-2014-01-02-2013
Introduction: There’s lots of overlap but I put each paper into only one category. Also, I’ve included work that has been published in 2013 as well as work that has been completed this year and might appear in 2014 or later. So you can can think of this list as representing roughly two years’ work. Political science: [2014] The twentieth-century reversal: How did the Republican states switch to the Democrats and vice versa? {\em Statistics and Public Policy}. (Andrew Gelman) [2013] Hierarchical models for estimating state and demographic trends in U.S. death penalty public opinion. {\em Journal of the Royal Statistical Society A}. (Kenneth Shirley and Andrew Gelman) [2013] Deep interactions with MRP: Election turnout and voting patterns among small electoral subgroups. {\em American Journal of Political Science}. (Yair Ghitza and Andrew Gelman) [2013] Charles Murray’s {\em Coming Apart} and the measurement of social and political divisions. {\em Statistics, Politics and Policy}.
Introduction: We recently considered a pair of studies that came out awhile ago involving children and political orientation: Andrew Oswald and Nattavudh Powdthavee found that, in Great Britain, parents of girls were more likely to support left-wing parties, compared to parents of boys. And, in the other direction, Dalton Conley and Emily Rauscher found with survey data from the United States that parents of girls were more likely to support the Republican party, compared to parents of boys. As I discussed the other day, the latest version of the Conley and Raucher study came with some incoherent evolutionary theorizing. There was also some discussion regarding the differences between the two studies. Oswald sent me some relevant comments: It will be hard in cross-sections like the GSS to solve the problem of endogeneity bias. Conservative families may want to have boys, and may use that as a stopping rule. If so, they will end up with disproportionately large number of girls. Without l
7 0.12331016 1878 andrew gelman stats-2013-05-31-How to fix the tabloids? Toward replicable social science research
8 0.12073164 2034 andrew gelman stats-2013-09-23-My talk Tues 24 Sept at 12h30 at Université de Technologie de Compiègne
9 0.11827454 2141 andrew gelman stats-2013-12-20-Don’t douthat, man! Please give this fallacy a name.
10 0.11761799 1880 andrew gelman stats-2013-06-02-Flame bait
11 0.1155633 695 andrew gelman stats-2011-05-04-Statistics ethics question
12 0.11349923 1317 andrew gelman stats-2012-05-13-Question 3 of my final exam for Design and Analysis of Sample Surveys
13 0.11281679 1315 andrew gelman stats-2012-05-12-Question 2 of my final exam for Design and Analysis of Sample Surveys
14 0.11218999 1695 andrew gelman stats-2013-01-28-Economists argue about Bayes
15 0.11036941 1941 andrew gelman stats-2013-07-16-Priors
17 0.10639074 1435 andrew gelman stats-2012-07-30-Retracted articles and unethical behavior in economics journals?
18 0.10431117 2245 andrew gelman stats-2014-03-12-More on publishing in journals
19 0.10401276 1139 andrew gelman stats-2012-01-26-Suggested resolution of the Bem paradox
20 0.10397204 1400 andrew gelman stats-2012-06-29-Decline Effect in Linguistics?
topicId topicWeight
[(0, 0.229), (1, -0.027), (2, -0.001), (3, -0.144), (4, -0.008), (5, -0.019), (6, -0.014), (7, -0.034), (8, -0.039), (9, -0.07), (10, 0.039), (11, -0.025), (12, 0.057), (13, 0.019), (14, 0.04), (15, 0.012), (16, -0.017), (17, 0.015), (18, -0.01), (19, 0.04), (20, -0.01), (21, 0.013), (22, 0.05), (23, -0.038), (24, 0.009), (25, 0.005), (26, -0.034), (27, 0.033), (28, 0.027), (29, -0.051), (30, -0.006), (31, -0.007), (32, -0.068), (33, 0.049), (34, 0.019), (35, 0.028), (36, 0.013), (37, 0.016), (38, -0.02), (39, -0.022), (40, 0.047), (41, -0.012), (42, 0.003), (43, -0.016), (44, -0.01), (45, 0.047), (46, -0.009), (47, -0.077), (48, 0.049), (49, 0.015)]
simIndex simValue blogId blogTitle
same-blog 1 0.96655321 1074 andrew gelman stats-2011-12-20-Reading a research paper != agreeing with its claims
Introduction: A journalist wrote to me recently: I was going to include your deconstruction of the beautiful daughters paper, but ran out of space. The author, incidentally, stands by that paper — and emailed me that you’d advised him on a later paper, implying that meant you now accepted the thesis! I responded: I know that Kanazawa stands by his earlier claim. Unfortunately, in this case, “stands by” means believing something with no evidence. And, yes, he send me a copy of his other paper, and I responded by telling him I thought his sample size is too small. I did advise him, and my advice was to not do it! To avoid any future confusion, I thought I’d post my half of my emails with Kanazawa over the years: 28 Aug 2006: Dear Dr. Kanazawa, I read with interest your papers on sex ratios in the Journal of Theoretical Biology. Sex ratios are an inherently interesting topic and also a favorite example for people such as myself who teach probability and statistics. In the cour
Introduction: Last week I published in Slate a critique of a paper that appeared in the journal Psychological Science. That paper, by Alec Beall and Jessica Tracy, found that women who were at peak fertility were three times more likely to wear red or pink shirts, compared to women at other points in their menstrual cycles. The study was based an 100 participants on the internet and 24 college students. In my critique, I argued that we had no reason to believe the results generalized to the larger population, because (1) the samples were not representative, (2) the measurements were noisy, (3) the researchers did not use the correct dates of peak fertility, and (4) there were many different comparisons that could have been reported in the data, so there was nothing special about a particular comparison being statistically significant. I likened their paper to other work which I considered flawed for multiple comparisons (too many researcher degrees of freedom), including a claimed relation bet
3 0.76521146 2223 andrew gelman stats-2014-02-24-“Edlin’s rule” for routinely scaling down published estimates
Introduction: A few months ago I reacted (see further discussion in comments here ) to a recent study on early childhood intervention, in which researchers Paul Gertler, James Heckman, Rodrigo Pinto, Arianna Zanolini, Christel Vermeerch, Susan Walker, Susan M. Chang, and Sally Grantham-McGregor estimated that a particular intervention on young children had raised incomes on young adults by 42%. I wrote: Major decisions on education policy can turn on the statistical interpretation of small, idiosyncratic data sets — in this case, a study of 129 Jamaican children. . . . Overall, I have no reason to doubt the direction of the effect, namely, that psychosocial stimulation should be good. But I’m skeptical of the claim that income differed by 42%, due to the reason of the statistical significance filter . In section 2.3, the authors are doing lots of hypothesizing based on some comparisons being statistically significant and others being non-significant. There’s nothing wrong with speculation, b
4 0.75097358 963 andrew gelman stats-2011-10-18-Question on Type M errors
Introduction: Inti Pedroso writes: Today during the group meeting at my new job we were revising a paper whose main conclusions were sustained by an ANOVA. One of the first observations is that the experiment had a small sample size. Interestingly (may not so), some of the reported effects (most of them interactions) were quite large. One of the experience group members said that “there is a common wisdom that one should not believe effects from small sample sizes but [he thinks] if they [the effects] are large enough to be picked on a small study they must be real large effects”. I argued that if the sample size is small one could incur on a M-type error in which the magnitude of the effect is being over-estimated and that if larger samples are evaluated the magnitude may become smaller and also the confidence intervals. The concept of M-type error is completely new to all other members of the group (on which I am in my second week) and I was given the job of finding a suitable ref to explain
Introduction: There’s a paradigm in applied statistics that goes something like this: 1. There is a scientific or policy question of some theoretical or practical importance. 2. Researchers gather data on relevant outcomes and perform a statistical analysis, ideally leading to a clear conclusion (p less than 0.05, or a strong posterior distribution, or good predictive performance, or high reliability and validity, whatever). 3. This conclusion informs policy. This paradigm has room for positive findings (for example, that a new program is statistically significantly better, or statistically significantly worse than what came before) or negative findings (data are inconclusive, further study is needed), even if negative findings seem less likely to make their way into the textbooks. But what happens when step 2 simply isn’t possible. This came up a few years ago—nearly 10 years ago, now!—with the excellent paper by Donohue and Wolfers which explained why it’s just about impossible to
7 0.73454154 1400 andrew gelman stats-2012-06-29-Decline Effect in Linguistics?
11 0.70677912 1860 andrew gelman stats-2013-05-17-How can statisticians help psychologists do their research better?
12 0.70331341 629 andrew gelman stats-2011-03-26-Is it plausible that 1% of people pick a career based on their first name?
14 0.69887996 1959 andrew gelman stats-2013-07-28-50 shades of gray: A research story
15 0.69669574 898 andrew gelman stats-2011-09-10-Fourteen magic words: an update
16 0.69482744 2090 andrew gelman stats-2013-11-05-How much do we trust a new claim that early childhood stimulation raised earnings by 42%?
17 0.69410002 490 andrew gelman stats-2010-12-29-Brain Structure and the Big Five
18 0.69353008 797 andrew gelman stats-2011-07-11-How do we evaluate a new and wacky claim?
19 0.69231558 2241 andrew gelman stats-2014-03-10-Preregistration: what’s in it for you?
topicId topicWeight
[(2, 0.035), (9, 0.015), (13, 0.011), (15, 0.049), (16, 0.06), (18, 0.132), (24, 0.146), (36, 0.036), (49, 0.02), (77, 0.011), (86, 0.02), (99, 0.338)]
simIndex simValue blogId blogTitle
1 0.97999752 969 andrew gelman stats-2011-10-22-Researching the cost-effectiveness of political lobbying organisations
Introduction: Sally Murray from Giving What We Can writes: We are an organisation that assesses different charitable (/fundable) interventions, to estimate which are the most cost-effective (measured in terms of the improvement of life for people in developing countries gained for every dollar invested). Our research guides and encourages greater donations to the most cost-effective charities we thus identify, and our members have so far pledged a total of $14m to these causes, with many hundreds more relying on our advice in a less formal way. I am specifically researching the cost-effectiveness of political lobbying organisations. We are initially focusing on organisations that lobby for ‘big win’ outcomes such as increased funding of the most cost-effective NTD treatments/ vaccine research, changes to global trade rules (potentially) and more obscure lobbies such as “Keep Antibiotics Working”. We’ve a great deal of respect for your work and the superbly rational way you go about it, and
2 0.97546649 1967 andrew gelman stats-2013-08-04-What are the key assumptions of linear regression?
Introduction: Andy Cooper writes: A link to an article , “Four Assumptions Of Multiple Regression That Researchers Should Always Test”, has been making the rounds on Twitter. Their first rule is “Variables are Normally distributed.” And they seem to be talking about the independent variables – but then later bring in tests on the residuals (while admitting that the normally-distributed error assumption is a weak assumption). I thought we had long-since moved away from transforming our independent variables to make them normally distributed for statistical reasons (as opposed to standardizing them for interpretability, etc.) Am I missing something? I agree that leverage in a influence is important, but normality of the variables? The article is from 2002, so it might be dated, but given the popularity of the tweet, I thought I’d ask your opinion. My response: There’s some useful advice on that page but overall I think the advice was dated even in 2002. In section 3.6 of my book wit
3 0.97429401 1292 andrew gelman stats-2012-05-01-Colorless green facts asserted resolutely
Introduction: Thomas Basbøll [yes, I've learned how to smoothly do this using alt-o] gives some writing advice : What gives a text presence is our commitment to asserting facts. We have to face the possibility that we may be wrong about them resolutely, and we do this by writing about them as though we are right. This and an earlier remark by Basbøll are closely related in my mind to predictive model checking and to Bayesian statistics : we make strong assumptions and then engage the data and the assumptions in a dialogue: assumptions + data -> inference, and we can then compare the inference to the data which can reveal problems with our model (or problems with the data, but that’s really problems with the model too, in this case problems with the model for the data). I like the idea that a condition for a story to be useful is that we put some belief into it. (One doesn’t put belief into a joke.) And also the converse, that thnking hard about a story and believing it can be the pre
4 0.97406685 698 andrew gelman stats-2011-05-05-Shocking but not surprising
Introduction: Much-honored playwright Tony Kushner was set to receive one more honor–a degree from John Jay College–but it was suddenly taken away from him on an 11-1 vote of the trustees of the City University of New York. This was the first rejection of an honorary degree nomination since 1961. The news article focuses on one trustee, Jeffrey Wiesenfeld, an investment adviser and onetime political aide, who opposed Kushner’s honorary degree, but to me the relevant point is that the committee as a whole voted 11-1 to ding him. Kusnher said, “I’m sickened,” he added, “that this is happening in New York City. Shocked, really.” I can see why he’s shocked, but perhaps it’s not so surprising that it’s happening in NYC. Recall the famous incident from 1940 in which Bertrand Russell was invited and then uninvited to teach at City College. The problem that time was Russell’s views on free love (as they called it back then). There seems to be a long tradition of city college officials being will
5 0.96914536 1319 andrew gelman stats-2012-05-14-I hate to get all Gerd Gigerenzer on you here, but . . .
Introduction: Jonathan Cantor points me to an opinion piece by psychologist Reid Hastie, “Our Gift for Good Stories Blinds Us to the Truth.” I have mixed feelings about Hastie’s article. On one hand I do think his point is important. It’s not new to me, but presumably it’s new to many readers of bloomberg.com. I like Hastie’s book (with Robyn Dawes), Rational Choice in an Uncertain World, and I’m predisposed to like anything new that he writes. On the other hand, there’s something about Hastie’s article that bothered me. It seemed a bit smug, as if he thinks he understands the world and wants to just explain it to the rest of us. That could be fine—after all, Hastie is a distinguished psychology researcher—but I wasn’t so clear that he’s so clear on what he’s saying. For example: The human brain is designed to support two modes of thought: visual and narrative. These forms of thinking are universal across human societies throughout history, develop reliably early in individuals’ lives
same-blog 6 0.96807158 1074 andrew gelman stats-2011-12-20-Reading a research paper != agreeing with its claims
7 0.9674046 1691 andrew gelman stats-2013-01-25-Extreem p-values!
8 0.96723372 588 andrew gelman stats-2011-02-24-In case you were wondering, here’s the price of milk
9 0.96589148 718 andrew gelman stats-2011-05-18-Should kids be able to bring their own lunches to school?
10 0.96238488 2046 andrew gelman stats-2013-10-01-I’ll say it again
11 0.95771188 114 andrew gelman stats-2010-06-28-More on Bayesian deduction-induction
12 0.95541906 456 andrew gelman stats-2010-12-07-The red-state, blue-state war is happening in the upper half of the income distribution
13 0.95532131 1922 andrew gelman stats-2013-07-02-They want me to send them free material and pay for the privilege
14 0.94816089 621 andrew gelman stats-2011-03-20-Maybe a great idea in theory, didn’t work so well in practice
15 0.94652122 1204 andrew gelman stats-2012-03-08-The politics of economic and statistical models
16 0.9456023 2338 andrew gelman stats-2014-05-19-My short career as a Freud expert
17 0.94082153 2239 andrew gelman stats-2014-03-09-Reviewing the peer review process?
18 0.93975079 2148 andrew gelman stats-2013-12-25-Spam!
19 0.93963623 2181 andrew gelman stats-2014-01-21-The Commissar for Traffic presents the latest Five-Year Plan
20 0.93649119 2075 andrew gelman stats-2013-10-23-PubMed Commons: A system for commenting on articles in PubMed