andrew_gelman_stats andrew_gelman_stats-2014 andrew_gelman_stats-2014-2241 knowledge-graph by maker-knowledge-mining
Source: html
Introduction: Chris Chambers pointed me to a blog by someone called Neuroskeptic who suggested that I preregister my political science studies: So when Andrew Gelman (let’s say) is going to start using a new approach, he goes on Twitter, or on his blog, and posts a bare-bones summary of what he’s going to do. Then he does it. If he finds something interesting, he writes it up as a paper, citing that tweet or post as his preregistration. . . . I think this approach has some benefits but doesn’t really address the issues of preregistration that concern me—but I’d like to spend an entire blog post explaining why. I have two key points: 1. If your study is crap, preregistration might fix it. Preregistration is fine—indeed, the wide acceptance of preregistration might well motivate researchers to not do so many crap studies—but it doesn’t solve fundamental problems of experimental design. 2. “Preregistration” seems to mean different things in different scenarios: A. When the concern is
sentIndex sentText sentNum sentScore
1 If he finds something interesting, he writes it up as a paper, citing that tweet or post as his preregistration. [sent-3, score-0.085]
2 I think this approach has some benefits but doesn’t really address the issues of preregistration that concern me—but I’d like to spend an entire blog post explaining why. [sent-7, score-0.864]
3 If your study is crap, preregistration might fix it. [sent-9, score-0.831]
4 Preregistration is fine—indeed, the wide acceptance of preregistration might well motivate researchers to not do so many crap studies—but it doesn’t solve fundamental problems of experimental design. [sent-10, score-0.951]
5 When the concern is the file drawer (the idea that Gelman or some other researcher performs lots of studies, and we want to hear about the failures as well as successes), “preregistration” is a “bare bones summary” that can be tweeted. [sent-15, score-0.176]
6 Sure, preregistration doesn’t solve problems of experimental design. [sent-19, score-0.88]
7 But I think it’s important to be clear what sort of preregistration we’re talking about in any particular case, and what we expect preregistration to be doing. [sent-21, score-1.45]
8 Given that political science is (I assume) not among Neuroskeptic’s areas of expertise, I respect his or her decision to hold off judgment on the issue. [sent-24, score-0.058]
9 Of this study and a recent nonreplication, Neuroskeptic writes: I know of only one way to put a stop to all this uncertainty: preregistration of studies of all kinds. [sent-26, score-0.923]
10 In the case of the ovulation-and-voting study, the authors have large measurement error, high levels of variation, and they’re studying small effects. [sent-29, score-0.117]
11 And all this is made even worse because they are studying within-person effects using a between-person design. [sent-30, score-0.103]
12 That is, the design has a high Type S error rate and a high Type M error rate . [sent-32, score-0.372]
13 As I wrote above, the question is not just if I should preregister but also how I should do it. [sent-35, score-0.153]
14 I do think that blogging my research directions ahead of time (or archiving time-stamped statements, if I want to let my ideas develop in secret for awhile) would be a good idea. [sent-36, score-0.402]
15 It would give people a sense of what worked, what didn’t work, and what I’m still working on. [sent-37, score-0.048]
16 But this sort of blogging would not address multiple comparisons problems. [sent-39, score-0.342]
17 In both cases, there were just too many data-processing and data-analysis choices that would not fit in that research plan. [sent-41, score-0.112]
18 Not what goes in a tweet but a complete step-by-step plan of how I would choose, code, and analyze the data? [sent-43, score-0.133]
19 That might have been a good idea for Bem, Beall, and Tracy, or maybe not, but it would certainly not work for me in my political science research (or, for that matter, in my research in statistical methods). [sent-44, score-0.234]
20 He or she suggested that I do tweet-like announcements of my research plans, and I agree that’s a good idea. [sent-47, score-0.167]
wordName wordTfidf (topN-words)
[('preregistration', 0.725), ('neuroskeptic', 0.271), ('preregister', 0.153), ('ahead', 0.111), ('tweeted', 0.108), ('study', 0.106), ('bem', 0.104), ('beall', 0.098), ('studies', 0.092), ('comparisons', 0.092), ('tracy', 0.092), ('tweet', 0.085), ('blogging', 0.084), ('concern', 0.083), ('doesn', 0.082), ('solve', 0.079), ('experimental', 0.076), ('esp', 0.072), ('crap', 0.071), ('high', 0.064), ('research', 0.064), ('multiple', 0.062), ('expertise', 0.061), ('political', 0.058), ('error', 0.057), ('address', 0.056), ('neither', 0.055), ('suggested', 0.054), ('nonreplication', 0.054), ('studying', 0.053), ('hormonal', 0.051), ('bare', 0.051), ('archiving', 0.051), ('effects', 0.05), ('announcements', 0.049), ('variation', 0.049), ('gelman', 0.048), ('would', 0.048), ('bones', 0.047), ('chambers', 0.047), ('exclusion', 0.047), ('summary', 0.047), ('type', 0.046), ('fertile', 0.046), ('drawer', 0.046), ('worries', 0.045), ('time', 0.044), ('data', 0.044), ('design', 0.044), ('rate', 0.043)]
simIndex simValue blogId blogTitle
same-blog 1 0.99999988 2241 andrew gelman stats-2014-03-10-Preregistration: what’s in it for you?
Introduction: Chris Chambers pointed me to a blog by someone called Neuroskeptic who suggested that I preregister my political science studies: So when Andrew Gelman (let’s say) is going to start using a new approach, he goes on Twitter, or on his blog, and posts a bare-bones summary of what he’s going to do. Then he does it. If he finds something interesting, he writes it up as a paper, citing that tweet or post as his preregistration. . . . I think this approach has some benefits but doesn’t really address the issues of preregistration that concern me—but I’d like to spend an entire blog post explaining why. I have two key points: 1. If your study is crap, preregistration might fix it. Preregistration is fine—indeed, the wide acceptance of preregistration might well motivate researchers to not do so many crap studies—but it doesn’t solve fundamental problems of experimental design. 2. “Preregistration” seems to mean different things in different scenarios: A. When the concern is
2 0.30952764 2183 andrew gelman stats-2014-01-23-Discussion on preregistration of research studies
Introduction: Chris Chambers and I had an enlightening discussion the other day at the blog of Rolf Zwaan, regarding the Garden of Forking Paths ( go here and scroll down through the comments). Chris sent me the following note: I’m writing a book at the moment about reforming practices in psychological research (focusing on various bad practices such as p-hacking, HARKing, low statistical power, publication bias, lack of data sharing etc. – and posing solutions such as pre-registration, Bayesian hypothesis testing, mandatory data archiving etc.) and I am arriving at rather unsettling conclusion: that null hypothesis significance testing (NHST) simply isn’t valid for observational research. If this is true then most of the psychological literature is statistically flawed. I was wonder what your thoughts were on this, both from a statistical point of view and from your experience working in an observational field. We all know about the dangers of researcher degrees of freedom. We also know
3 0.21992706 2232 andrew gelman stats-2014-03-03-What is the appropriate time scale for blogging—the day or the week?
Introduction: I post (approximately) once a day and don’t plan to change that. I have enough material to post more often—for example, I could intersperse existing blog posts with summaries of my published papers or of other work that I like; and, beyond this, we currently have a one-to-two-month backlog of posts—but I’m afraid that if the number of posts were doubled, the attention given to each would be roughly halved. Looking at it the other way, I certainly don’t want to reduce my level of posting. Sure, it takes time to blog, but these are things that are important for me to say. If I were to blog less frequently, it would only be because I was pouring all these words into a different vessel, for example a book. For now, though, I think it makes sense to blog and then collect the words later as appropriate. With blogging I get comments, and many of these comments are helpful—either directly (by pointing out errors in my thinking or linking to relevant software or literature) or indirec
Introduction: Jessica Tracy and Alec Beall, authors of that paper that claimed that women at peak fertility were more likely to wear red or pink shirts (see further discussion here and here ), and then a later paper that claimed that this happens in some weather but not others, just informed me that they have posted a note in disagreement with an paper by Eric Loken and myself. Our paper is unpublished, but I do have the megaphone of this blog, and Tracy and Beall do not, so I think it’s only fair to link to their note right away. I’ll quote from their note (but if you’re interested, please follow the link and read the whole thing ) and then give some background and my own reaction. Tracy and Beall write: Although Gelman and Loken are using our work as an example of a broader problem that pervades the field–a problem we generally agree about–we are concerned that readers will take their speculations about our methods and analyses as factual claims about our scientific integrity. Fu
5 0.1855136 2240 andrew gelman stats-2014-03-10-On deck this week: Things people sent me
Introduction: Mon: Preregistration: what’s in it for you? Tues: What if I were to stop publishing in journals? Wed: Empirical implications of Empirical Implications of Theoretical Models Thurs: An Economist’s Guide to Visualizing Data Fri: The maximal information coefficient Sat: Problematic interpretations of confidence intervals Sun: The more you look, the more you find
6 0.15654081 1671 andrew gelman stats-2013-01-13-Preregistration of Studies and Mock Reports
9 0.119532 1273 andrew gelman stats-2012-04-20-Proposals for alternative review systems for scientific work
10 0.11759111 1989 andrew gelman stats-2013-08-20-Correcting for multiple comparisons in a Bayesian regression model
11 0.11704412 1998 andrew gelman stats-2013-08-25-A new Bem theory
12 0.11596508 506 andrew gelman stats-2011-01-06-That silly ESP paper and some silliness in a rebuttal as well
13 0.10670194 1139 andrew gelman stats-2012-01-26-Suggested resolution of the Bem paradox
14 0.10623029 511 andrew gelman stats-2011-01-11-One more time on that ESP study: The problem of overestimates and the shrinkage solution
15 0.10519243 1860 andrew gelman stats-2013-05-17-How can statisticians help psychologists do their research better?
18 0.10265497 1878 andrew gelman stats-2013-05-31-How to fix the tabloids? Toward replicable social science research
19 0.10079317 1955 andrew gelman stats-2013-07-25-Bayes-respecting experimental design and other things
20 0.10033979 1695 andrew gelman stats-2013-01-28-Economists argue about Bayes
topicId topicWeight
[(0, 0.205), (1, -0.03), (2, 0.001), (3, -0.126), (4, -0.025), (5, -0.041), (6, -0.024), (7, -0.023), (8, -0.033), (9, -0.006), (10, -0.014), (11, 0.042), (12, 0.032), (13, -0.038), (14, 0.006), (15, 0.003), (16, -0.041), (17, -0.019), (18, -0.011), (19, 0.009), (20, 0.011), (21, -0.002), (22, -0.037), (23, 0.02), (24, -0.024), (25, -0.016), (26, 0.006), (27, -0.003), (28, 0.013), (29, -0.023), (30, -0.021), (31, -0.019), (32, -0.001), (33, -0.001), (34, 0.021), (35, 0.007), (36, -0.025), (37, 0.003), (38, -0.046), (39, -0.027), (40, 0.0), (41, 0.017), (42, -0.004), (43, -0.031), (44, -0.015), (45, 0.001), (46, -0.016), (47, 0.027), (48, -0.016), (49, -0.001)]
simIndex simValue blogId blogTitle
same-blog 1 0.96343589 2241 andrew gelman stats-2014-03-10-Preregistration: what’s in it for you?
Introduction: Chris Chambers pointed me to a blog by someone called Neuroskeptic who suggested that I preregister my political science studies: So when Andrew Gelman (let’s say) is going to start using a new approach, he goes on Twitter, or on his blog, and posts a bare-bones summary of what he’s going to do. Then he does it. If he finds something interesting, he writes it up as a paper, citing that tweet or post as his preregistration. . . . I think this approach has some benefits but doesn’t really address the issues of preregistration that concern me—but I’d like to spend an entire blog post explaining why. I have two key points: 1. If your study is crap, preregistration might fix it. Preregistration is fine—indeed, the wide acceptance of preregistration might well motivate researchers to not do so many crap studies—but it doesn’t solve fundamental problems of experimental design. 2. “Preregistration” seems to mean different things in different scenarios: A. When the concern is
Introduction: Last week I published in Slate a critique of a paper that appeared in the journal Psychological Science. That paper, by Alec Beall and Jessica Tracy, found that women who were at peak fertility were three times more likely to wear red or pink shirts, compared to women at other points in their menstrual cycles. The study was based an 100 participants on the internet and 24 college students. In my critique, I argued that we had no reason to believe the results generalized to the larger population, because (1) the samples were not representative, (2) the measurements were noisy, (3) the researchers did not use the correct dates of peak fertility, and (4) there were many different comparisons that could have been reported in the data, so there was nothing special about a particular comparison being statistically significant. I likened their paper to other work which I considered flawed for multiple comparisons (too many researcher degrees of freedom), including a claimed relation bet
Introduction: Jessica Tracy and Alec Beall, authors of that paper that claimed that women at peak fertility were more likely to wear red or pink shirts (see further discussion here and here ), and then a later paper that claimed that this happens in some weather but not others, just informed me that they have posted a note in disagreement with an paper by Eric Loken and myself. Our paper is unpublished, but I do have the megaphone of this blog, and Tracy and Beall do not, so I think it’s only fair to link to their note right away. I’ll quote from their note (but if you’re interested, please follow the link and read the whole thing ) and then give some background and my own reaction. Tracy and Beall write: Although Gelman and Loken are using our work as an example of a broader problem that pervades the field–a problem we generally agree about–we are concerned that readers will take their speculations about our methods and analyses as factual claims about our scientific integrity. Fu
4 0.87533343 1860 andrew gelman stats-2013-05-17-How can statisticians help psychologists do their research better?
Introduction: I received two emails yesterday on related topics. First, Stephen Olivier pointed me to this post by Daniel Lakens, who wrote the following open call to statisticians: You would think that if you are passionate about statistics, then you want to help people to calculate them correctly in any way you can. . . . you’d think some statisticians would be interested in helping a poor mathematically challenged psychologist out by offering some practical advice. I’m the right person to ask this question, since I actually have written a lot of material that helps psychologists (and others) with their data analysis. But there clearly are communication difficulties, in that my work and that of other statisticians hasn’t reached Lakens. Sometimes the contributions of statisticians are made indirectly. For example, I wrote Bayesian Data Analysis, and then Kruschke wrote Doing Bayesian Data Analysis. Our statistics book made it possible for Kruschke to write his excellent book for psycholo
Introduction: I pointed Steven Pinker to my post, How much time (if any) should we spend criticizing research that’s fraudulent, crappy, or just plain pointless? , and he responded: Clearly it *is* important to call out publicized research whose conclusions are likely to be false. The only danger is that it’s so easy and fun to criticize, with all the perks of intellectual and moral superiority for so little cost, that there is a moral hazard to go overboard and become a professional slasher and snarker. (That’s a common phenomenon among literary critics, especially in the UK.) There’s also the risk of altering the incentive structure for innovative research, so that researchers stick to the safest kinds of paradigm-twiddling. I think these two considerations were what my late colleague Dan Wegner had in mind when he made the bumbler-pointer contrast — he himself was certainly a discerning critic of social science research. [Just to clarify: Wegner is the person who talked about bumblers and po
6 0.86431009 1878 andrew gelman stats-2013-05-31-How to fix the tabloids? Toward replicable social science research
7 0.86309171 2093 andrew gelman stats-2013-11-07-I’m negative on the expression “false positives”
8 0.86101902 1959 andrew gelman stats-2013-07-28-50 shades of gray: A research story
9 0.84817207 2361 andrew gelman stats-2014-06-06-Hurricanes vs. Himmicanes
11 0.84438884 511 andrew gelman stats-2011-01-11-One more time on that ESP study: The problem of overestimates and the shrinkage solution
12 0.84338826 1171 andrew gelman stats-2012-02-16-“False-positive psychology”
14 0.83918607 1054 andrew gelman stats-2011-12-12-More frustrations trying to replicate an analysis published in a reputable journal
15 0.83655691 898 andrew gelman stats-2011-09-10-Fourteen magic words: an update
16 0.83159739 2236 andrew gelman stats-2014-03-07-Selection bias in the reporting of shaky research
17 0.8248964 1883 andrew gelman stats-2013-06-04-Interrogating p-values
20 0.82206202 2223 andrew gelman stats-2014-02-24-“Edlin’s rule” for routinely scaling down published estimates
topicId topicWeight
[(15, 0.062), (16, 0.056), (21, 0.016), (24, 0.125), (47, 0.116), (81, 0.035), (82, 0.043), (84, 0.022), (99, 0.406)]
simIndex simValue blogId blogTitle
1 0.98532724 1050 andrew gelman stats-2011-12-10-Presenting at the econ seminar
Introduction: Jim Savage saw this and pointed me to this video. I didn’t actually look at it, but given that it is labeled, “For new econ Ph.D.’s about to look for a job . . . what you might expect when you give your first talk presenting your research,” I can pretty much guess what it’ll look like.
2 0.98314273 1143 andrew gelman stats-2012-01-29-G+ > Skype
Introduction: I spoke at the University of Kansas the other day. Kansas is far away so I gave the talk by video. We did it using a G+ hangout, and it worked really well, much much better than when I gave a talk via Skype . With G+, I could see and hear the audience clearly, and they could hear me just fine while seeing my slides (or my face, I went back and forth). Not as good as a live presentation but pretty good, considering. P.S. And here’s how to do it! Conflict of interest disclaimer: I was paid by Google last year to give a short course.
same-blog 3 0.98276168 2241 andrew gelman stats-2014-03-10-Preregistration: what’s in it for you?
Introduction: Chris Chambers pointed me to a blog by someone called Neuroskeptic who suggested that I preregister my political science studies: So when Andrew Gelman (let’s say) is going to start using a new approach, he goes on Twitter, or on his blog, and posts a bare-bones summary of what he’s going to do. Then he does it. If he finds something interesting, he writes it up as a paper, citing that tweet or post as his preregistration. . . . I think this approach has some benefits but doesn’t really address the issues of preregistration that concern me—but I’d like to spend an entire blog post explaining why. I have two key points: 1. If your study is crap, preregistration might fix it. Preregistration is fine—indeed, the wide acceptance of preregistration might well motivate researchers to not do so many crap studies—but it doesn’t solve fundamental problems of experimental design. 2. “Preregistration” seems to mean different things in different scenarios: A. When the concern is
4 0.98218238 1261 andrew gelman stats-2012-04-12-The Naval Research Lab
Introduction: I worked at the U.S. Naval Research Laboratory for four summers during high school and college. I spent much of my time writing a computer program to do thermal analysis for an experiment that we put on the space shuttle. The facility I developed with the finite-element method came in handy in my job at Bell Labs the following summers. I was working for C. H. Tsao and Jim Adams in the Laboratory for Cosmic Ray Physics. We were estimating the distribution of isotopes in cosmic rays using a pile of track detectors. To get accurate measurements, you want these plastic disks to be as close as possible to a constant temperature, so we designed an elaborate wrapping of thermal blankets. My program computed the temperature of the detectors during the year that the Long Duration Exposure Facility (including our experiment and a bunch of others) was scheduled to be in orbit. The input is the heat from solar radiation (easy enough to compute given the trajectory). On the computer I tr
5 0.97982818 95 andrew gelman stats-2010-06-17-“Rewarding Strivers: Helping Low-Income Students Succeed in College”
Introduction: Several years ago, I heard about a project at the Educational Testing Service to identify “strivers”: students from disadvantaged backgrounds who did unexpectedly well on the SAT (the college admissions exam formerly known as the “Scholastic Aptitude Test” but apparently now just “the SAT,” in the same way that Exxon is just “Exxon” and that Harry Truman’s middle name is just “S”), at least 200 points above a predicted score based on demographic and neighborhood information. My ETS colleague and I agreed that this was a silly idea: From a statistical point of view, if student A is expected ahead of time to do better than student B, and then they get identical test scores, then you’d expect student A (the non-”striver”) to do better than student B (the “striver”) later on. Just basic statistics: if a student does much better than expected, then probably some of that improvement is noise. The idea of identifying these “strivers” seemed misguided and not the best use of the SAT.
6 0.9793793 1055 andrew gelman stats-2011-12-13-Data sharing update
8 0.97403264 2275 andrew gelman stats-2014-03-31-Just gave a talk
9 0.97272003 1273 andrew gelman stats-2012-04-20-Proposals for alternative review systems for scientific work
10 0.97185242 2322 andrew gelman stats-2014-05-06-Priors I don’t believe
11 0.97114182 1349 andrew gelman stats-2012-05-28-Question 18 of my final exam for Design and Analysis of Sample Surveys
12 0.96906364 438 andrew gelman stats-2010-11-30-I just skyped in from Kentucky, and boy are my arms tired
13 0.96902704 1897 andrew gelman stats-2013-06-13-When’s that next gamma-ray blast gonna come, already?
14 0.96895921 1526 andrew gelman stats-2012-10-09-Little Data: How traditional statistical ideas remain relevant in a big-data world
15 0.96833181 1730 andrew gelman stats-2013-02-20-Unz on Unz
16 0.96562409 1668 andrew gelman stats-2013-01-11-My talk at the NY data visualization meetup this Monday!
18 0.96118152 1450 andrew gelman stats-2012-08-08-My upcoming talk for the data visualization meetup
20 0.95902669 2270 andrew gelman stats-2014-03-28-Creating a Lenin-style democracy