andrew_gelman_stats andrew_gelman_stats-2011 andrew_gelman_stats-2011-1055 knowledge-graph by maker-knowledge-mining
Source: html
Introduction: Fred Oswald reports that Sian Beilock sent him sufficient amounts of raw data from her research study so allow him to answer his questions about the large effects that were observed. This sort of collegiality is central to the collective scientific enterprise. The bad news is that IRB’s are still getting in the way. Beilock was very helpful but she had to work within the constraints of her IRB, which apparently advised her not to share data—even if de-identified—without getting lots more permissions. Oswald writes: It is a little concerning that the IRB bars the sharing of de-identified data, particularly in light of the specific guidelines of the journal Science, which appears to say that when you submit a study to the journal for publication, you are allowing for the sharing of de-identified data — unless you expressly say otherwise at the point that you submit the paper for consideration. Again, I don’t blame Beilock and Ramirez—they appear to have been as helpful as
sentIndex sentText sentNum sentScore
1 Fred Oswald reports that Sian Beilock sent him sufficient amounts of raw data from her research study so allow him to answer his questions about the large effects that were observed. [sent-1, score-0.382]
2 This sort of collegiality is central to the collective scientific enterprise. [sent-2, score-0.076]
3 The bad news is that IRB’s are still getting in the way. [sent-3, score-0.094]
4 Beilock was very helpful but she had to work within the constraints of her IRB, which apparently advised her not to share data—even if de-identified—without getting lots more permissions. [sent-4, score-0.342]
5 Again, I don’t blame Beilock and Ramirez—they appear to have been as helpful as can be given their IRB. [sent-6, score-0.159]
6 Maybe once the university realizes that they won’t be getting articles in Science, they’ll curb the IRB’s arbitrary exercises of power. [sent-8, score-0.427]
7 Russell makes a good comment : I think the problem is not the IRB, but rather the terms of the consent agreement. [sent-11, score-0.377]
8 The IRB is objecting not to the sharing of data, but rather to the fact that the participants in the study did not consent to be included in the database. [sent-12, score-0.782]
9 , somebody might vote against me for purely political reasons). [sent-16, score-0.246]
10 The cure is not to change IRBs, but rather to change our consent letters so that we talk about building the database in addition to the journal publications. [sent-17, score-0.942]
11 This is what we need to raise the consciousness of researchers about. [sent-18, score-0.061]
wordName wordTfidf (topN-words)
[('irb', 0.575), ('consent', 0.313), ('beilock', 0.313), ('sharing', 0.195), ('oswald', 0.172), ('database', 0.144), ('journal', 0.134), ('submit', 0.126), ('study', 0.116), ('irbs', 0.104), ('ramirez', 0.104), ('sian', 0.104), ('views', 0.104), ('somebody', 0.103), ('helpful', 0.097), ('getting', 0.094), ('curb', 0.094), ('objecting', 0.094), ('advised', 0.088), ('realizes', 0.088), ('exercises', 0.086), ('political', 0.084), ('fred', 0.081), ('cure', 0.079), ('concerning', 0.079), ('collective', 0.076), ('amounts', 0.075), ('russell', 0.074), ('change', 0.073), ('guidelines', 0.072), ('tenure', 0.071), ('data', 0.071), ('bars', 0.07), ('chances', 0.07), ('risks', 0.065), ('allowing', 0.065), ('arbitrary', 0.065), ('rather', 0.064), ('participate', 0.064), ('constraints', 0.063), ('sufficient', 0.063), ('blame', 0.062), ('letters', 0.062), ('identification', 0.061), ('raise', 0.061), ('purely', 0.059), ('raw', 0.057), ('widely', 0.056), ('affect', 0.054), ('light', 0.053)]
simIndex simValue blogId blogTitle
same-blog 1 1.0 1055 andrew gelman stats-2011-12-13-Data sharing update
Introduction: Fred Oswald reports that Sian Beilock sent him sufficient amounts of raw data from her research study so allow him to answer his questions about the large effects that were observed. This sort of collegiality is central to the collective scientific enterprise. The bad news is that IRB’s are still getting in the way. Beilock was very helpful but she had to work within the constraints of her IRB, which apparently advised her not to share data—even if de-identified—without getting lots more permissions. Oswald writes: It is a little concerning that the IRB bars the sharing of de-identified data, particularly in light of the specific guidelines of the journal Science, which appears to say that when you submit a study to the journal for publication, you are allowing for the sharing of de-identified data — unless you expressly say otherwise at the point that you submit the paper for consideration. Again, I don’t blame Beilock and Ramirez—they appear to have been as helpful as
Introduction: The story starts in September, when psychology professor Fred Oswald wrote me: I [Oswald] wanted to point out this paper in Science (Ramirez & Beilock, 2010) examining how students’ emotional writing improves their test performance in high-pressure situations. Although replication is viewed as the hallmark of research, this paper replicates implausibly large d-values and correlations across studies, leading me to be more suspicious of the findings (not less, as is generally the case). He also pointed me to this paper: Experimental disclosure and its moderators: A meta-analysis. Frattaroli, Joanne Psychological Bulletin, Vol 132(6), Nov 2006, 823-865. Disclosing information, thoughts, and feelings about personal and meaningful topics (experimental disclosure) is purported to have various health and psychological consequences (e.g., J. W. Pennebaker, 1993). Although the results of 2 small meta-analyses (P. G. Frisina, J. C. Borod, & S. J. Lepore, 2004; J. M. Smyth
3 0.32400325 1539 andrew gelman stats-2012-10-18-IRB nightmares
Introduction: Andrew Perrin nails it : Twice a year, like clockwork, the ethics cops at the IRB [institutional review board, the group on campus that has to approve research involving human subjects] take a break from deciding whether or not radioactive isotopes can be administered to prison populations to cure restless-leg syndrome to dream up some fancy new way in which participating in an automated telephone poll might cause harm. Perrin adds: The list of exemptions to IRB review is too short and, more importantly, contains no guiding principle as to what makes exempt. . . . [and] Even exemptions require approval by the IRB. He also voices a thought I’ve had many times, which is that there are all sorts of things you or I or anyone else can do on the street (for example, go up to people and ask them personal questions, drop objects and see if people pick them up, stage fights with our friends to see the reactions of bystanders, etc etc etc) but for which we have to go through an IRB
4 0.268534 35 andrew gelman stats-2010-05-16-Another update on the spam email study
Introduction: I think youall are probably getting sick of this by now so I’ll put it all below the fold. Akinola Modupe and Katherine Milkman responded to my email about their study : We want to clarify the reason we believe that the use of deception and a lack of informed consent were appropriate and ethical for this research study. In this project, we were studying how the timing of a decision affects discrimination based on race and/or gender. The emails all participants in our study received were identical except for a) the sender’s name (we used 20 names that pretesting revealed were strongly associated with being either Caucasian, Black, Indian, Chinese or Hispanic, as well as associated with being male or female) and b) whether the meeting requested was for today or for a week from today. Recipients were randomly selected and were randomly assigned to one of the race/gender/timing conditions. This study design will allow us to test for baseline levels of discrimination in acade
5 0.14454804 18 andrew gelman stats-2010-05-06-$63,000 worth of abusive research . . . or just a really stupid waste of time?
Introduction: As someone who relies strongly on survey research, it’s good for me to be reminded that some surveys are useful, some are useless, but one thing they almost all have in common is . . . they waste the respondents’ time. I thought of this after receiving the following email, which I shall reproduce here. My own comments appear after. Recently, you received an email from a student asking for 10 minutes of your time to discuss your Ph.D. program (the body of the email appears below). We are emailing you today to debrief you on the actual purpose of that email, as it was part of a research study. We sincerely hope our study did not cause you any disruption and we apologize if you were at all inconvenienced. Our hope is that this letter will provide a sufficient explanation of the purpose and design of our study to alleviate any concerns you may have about your involvement. We want to thank you for your time and for reading further if you are interested in understanding why you rece
6 0.1298808 1866 andrew gelman stats-2013-05-21-Recently in the sister blog
9 0.098736979 882 andrew gelman stats-2011-08-31-Meanwhile, on the sister blog . . .
10 0.09286388 1393 andrew gelman stats-2012-06-26-The reverse-journal-submission system
12 0.088740081 2134 andrew gelman stats-2013-12-14-Oswald evidence
14 0.083132677 1860 andrew gelman stats-2013-05-17-How can statisticians help psychologists do their research better?
15 0.076626167 2233 andrew gelman stats-2014-03-04-Literal vs. rhetorical
16 0.076492235 2301 andrew gelman stats-2014-04-22-Ticket to Baaaaarf
17 0.076236062 2244 andrew gelman stats-2014-03-11-What if I were to stop publishing in journals?
18 0.075828485 2245 andrew gelman stats-2014-03-12-More on publishing in journals
19 0.07342536 120 andrew gelman stats-2010-06-30-You can’t put Pandora back in the box
20 0.07233984 2055 andrew gelman stats-2013-10-08-A Bayesian approach for peer-review panels? and a speculation about Bruno Frey
topicId topicWeight
[(0, 0.133), (1, -0.048), (2, -0.01), (3, -0.091), (4, -0.038), (5, -0.001), (6, -0.023), (7, -0.052), (8, -0.057), (9, -0.006), (10, 0.043), (11, 0.014), (12, -0.005), (13, -0.011), (14, 0.001), (15, -0.003), (16, 0.028), (17, -0.012), (18, -0.028), (19, 0.036), (20, -0.018), (21, -0.0), (22, -0.011), (23, -0.018), (24, -0.01), (25, 0.004), (26, 0.005), (27, -0.03), (28, 0.027), (29, -0.004), (30, -0.062), (31, -0.041), (32, -0.024), (33, 0.042), (34, -0.002), (35, 0.046), (36, -0.002), (37, 0.002), (38, -0.023), (39, 0.012), (40, 0.073), (41, -0.011), (42, -0.019), (43, -0.004), (44, -0.025), (45, 0.012), (46, 0.045), (47, 0.054), (48, 0.018), (49, 0.011)]
simIndex simValue blogId blogTitle
same-blog 1 0.95772171 1055 andrew gelman stats-2011-12-13-Data sharing update
Introduction: Fred Oswald reports that Sian Beilock sent him sufficient amounts of raw data from her research study so allow him to answer his questions about the large effects that were observed. This sort of collegiality is central to the collective scientific enterprise. The bad news is that IRB’s are still getting in the way. Beilock was very helpful but she had to work within the constraints of her IRB, which apparently advised her not to share data—even if de-identified—without getting lots more permissions. Oswald writes: It is a little concerning that the IRB bars the sharing of de-identified data, particularly in light of the specific guidelines of the journal Science, which appears to say that when you submit a study to the journal for publication, you are allowing for the sharing of de-identified data — unless you expressly say otherwise at the point that you submit the paper for consideration. Again, I don’t blame Beilock and Ramirez—they appear to have been as helpful as
2 0.78749073 1774 andrew gelman stats-2013-03-22-Likelihood Ratio ≠ 1 Journal
Introduction: Dan Kahan writes : The basic idea . . . is to promote identification of study designs that scholars who disagree about a proposition would agree would generate evidence relevant to their competing conjectures—regardless of what studies based on such designs actually find. Articles proposing designs of this sort would be selected for publication and only then be carried out, by the proposing researchers with funding from the journal, which would publish the results too. Now I [Kahan] am aware of a set of real journals that have a similar motivation. One is the Journal of Articles in Support of the Null Hypothesis, which as its title implies publishes papers reporting studies that fail to “reject” the null. Like JASNH, LR ≠1J would try to offset the “file drawer” bias and like bad consequences associated with the convention of publishing only findings that are “significant at p < 0.05." But it would try to do more. By publishing studies that are deemed to have valid designs an
3 0.77113962 2268 andrew gelman stats-2014-03-26-New research journal on observational studies
Introduction: Dylan Small writes: I am starting an observational studies journal that aims to publish papers on all aspects of observational studies, including study protocols for observational studies, methodologies for observational studies, descriptions of data sets for observational studies, software for observational studies and analyses of observational studies. One of the goals of the journal is to promote the planning of observational studies and to publish study plans for observational studies, like study plans are published for major clinical trials. Regular readers will know my suggestion that scientific journals move away from the idea of being unique publishers of new material and move toward a “newsletter” approach, recommending papers from Arxiv, SSRN, etc. So, instead of going through exhausting review and revision processes, the journal editors would read and review recent preprints on observational studies and then, each month or quarter or whatever, produce a list of pap
4 0.76971281 2301 andrew gelman stats-2014-04-22-Ticket to Baaaaarf
Introduction: A link from the comments here took me to the wonderfully named Barfblog and a report by Don Schaffner on some reporting. First, the background: A university in England issued a press release saying that “Food picked up just a few seconds after being dropped is less likely to contain bacteria than if it is left for longer periods of time . . . The findings suggest there may be some scientific basis to the ‘5 second rule’ – the urban myth about it being fine to eat food that has only had contact with the floor for five seconds or less. Although people have long followed the 5 second rule, until now it was unclear whether it actually helped.” According to the press release, the study was “undertaken by final year Biology students” and led by a professor of microbiology. The press release hit the big time, hitting NPR, Slate, Forbes, the Daily News, etc etc. Some typical headlines: “5-second rule backed up by science” — Atlanta Journal Constitution “Eating food off the floo
Introduction: The story starts in September, when psychology professor Fred Oswald wrote me: I [Oswald] wanted to point out this paper in Science (Ramirez & Beilock, 2010) examining how students’ emotional writing improves their test performance in high-pressure situations. Although replication is viewed as the hallmark of research, this paper replicates implausibly large d-values and correlations across studies, leading me to be more suspicious of the findings (not less, as is generally the case). He also pointed me to this paper: Experimental disclosure and its moderators: A meta-analysis. Frattaroli, Joanne Psychological Bulletin, Vol 132(6), Nov 2006, 823-865. Disclosing information, thoughts, and feelings about personal and meaningful topics (experimental disclosure) is purported to have various health and psychological consequences (e.g., J. W. Pennebaker, 1993). Although the results of 2 small meta-analyses (P. G. Frisina, J. C. Borod, & S. J. Lepore, 2004; J. M. Smyth
6 0.76702732 1878 andrew gelman stats-2013-05-31-How to fix the tabloids? Toward replicable social science research
7 0.75154757 1671 andrew gelman stats-2013-01-13-Preregistration of Studies and Mock Reports
9 0.73299491 1860 andrew gelman stats-2013-05-17-How can statisticians help psychologists do their research better?
10 0.73065382 1273 andrew gelman stats-2012-04-20-Proposals for alternative review systems for scientific work
11 0.72468626 2361 andrew gelman stats-2014-06-06-Hurricanes vs. Himmicanes
12 0.72425139 1959 andrew gelman stats-2013-07-28-50 shades of gray: A research story
13 0.7233845 35 andrew gelman stats-2010-05-16-Another update on the spam email study
14 0.71497893 2137 andrew gelman stats-2013-12-17-Replication backlash
15 0.71412408 2220 andrew gelman stats-2014-02-22-Quickies
16 0.71308655 1053 andrew gelman stats-2011-12-11-This one is so dumb it makes me want to barf
18 0.70840997 1122 andrew gelman stats-2012-01-16-“Groundbreaking or Definitive? Journals Need to Pick One”
19 0.70836961 908 andrew gelman stats-2011-09-14-Type M errors in the lab
20 0.69878894 1291 andrew gelman stats-2012-04-30-Systematic review of publication bias in studies on publication bias
topicId topicWeight
[(9, 0.016), (15, 0.053), (16, 0.043), (21, 0.033), (24, 0.109), (32, 0.011), (47, 0.235), (56, 0.094), (72, 0.025), (76, 0.012), (95, 0.015), (99, 0.215)]
simIndex simValue blogId blogTitle
1 0.91698503 275 andrew gelman stats-2010-09-14-Data visualization at the American Evaluation Association
Introduction: Stephanie Evergreen writes: Media, web design, and marketing have all created an environment where stakeholders – clients, program participants, funders – all expect high quality graphics and reporting that effectively conveys the valuable insights from evaluation work. Some in statistics and mathematics have used data visualization strategies to support more useful reporting of complex ideas. Global growing interest in improving communications has begun to take root in the evaluation field as well. But as anyone who has sat through a day’s worth of a conference or had to endure a dissertation-worthy evaluation report knows, evaluators still have a long way to go. To support the development of researchers and evaluators, some members of the American Evaluation Association are proposing a new TIG (Topical Interest Group) on Data Visualization and Reporting. If you are a member of AEA (or want to be) and you are interested in joining this TIG, contact Stephanie Evergreen.
same-blog 2 0.89565647 1055 andrew gelman stats-2011-12-13-Data sharing update
Introduction: Fred Oswald reports that Sian Beilock sent him sufficient amounts of raw data from her research study so allow him to answer his questions about the large effects that were observed. This sort of collegiality is central to the collective scientific enterprise. The bad news is that IRB’s are still getting in the way. Beilock was very helpful but she had to work within the constraints of her IRB, which apparently advised her not to share data—even if de-identified—without getting lots more permissions. Oswald writes: It is a little concerning that the IRB bars the sharing of de-identified data, particularly in light of the specific guidelines of the journal Science, which appears to say that when you submit a study to the journal for publication, you are allowing for the sharing of de-identified data — unless you expressly say otherwise at the point that you submit the paper for consideration. Again, I don’t blame Beilock and Ramirez—they appear to have been as helpful as
Introduction: Adam Marcus at Retraction Watch reports on a physicist at the University of Toronto who had this unfortunate thing happen to him: This article has been retracted at the request of the Editor-in-Chief and first and corresponding author. The article was largely a duplication of a paper that had already appeared in ACS Nano, 4 (2010) 3374–3380, http://dx.doi.org/10.1021/nn100335g. The first and the corresponding authors (Kramer and Sargent) would like to apologize for this administrative error on their part . . . “Administrative error” . . . I love that! Is that what the robber says when he knocks over a liquor store and gets caught? As Marcus points out, the two papers have different titles and a different order of authors, which makes it less plausible that this was an administrative mistake (as could happen, for example, if a secretary was given a list of journals to submit the paper to, and accidentally submitted it to the second journal on the list without realizing it
4 0.84513271 95 andrew gelman stats-2010-06-17-“Rewarding Strivers: Helping Low-Income Students Succeed in College”
Introduction: Several years ago, I heard about a project at the Educational Testing Service to identify “strivers”: students from disadvantaged backgrounds who did unexpectedly well on the SAT (the college admissions exam formerly known as the “Scholastic Aptitude Test” but apparently now just “the SAT,” in the same way that Exxon is just “Exxon” and that Harry Truman’s middle name is just “S”), at least 200 points above a predicted score based on demographic and neighborhood information. My ETS colleague and I agreed that this was a silly idea: From a statistical point of view, if student A is expected ahead of time to do better than student B, and then they get identical test scores, then you’d expect student A (the non-”striver”) to do better than student B (the “striver”) later on. Just basic statistics: if a student does much better than expected, then probably some of that improvement is noise. The idea of identifying these “strivers” seemed misguided and not the best use of the SAT.
Introduction: Remember How to Lie With Statistics? It turns out that the author worked for the cigarette companies. John Mashey points to this, from Robert Proctor’s book, “Golden Holocaust: Origins of the Cigarette Catastrophe and the Case for Abolition”: Darrell Huff, author of the wildly popular (and aptly named) How to Lie With Statistics, was paid to testify before Congress in the 1950s and then again in the 1960s, with the assigned task of ridiculing any notion of a cigarette-disease link. On March 22, 1965, Huff testified at hearings on cigarette labeling and advertising, accusing the recent Surgeon General’s report of myriad failures and “fallacies.” Huff peppered his attack with with amusing asides and anecdotes, lampooning spurious correlations like that between the size of Dutch families and the number of storks nesting on rooftops–which proves not that storks bring babies but rather that people with large families tend to have larger houses (which therefore attract more storks).
6 0.82561082 1050 andrew gelman stats-2011-12-10-Presenting at the econ seminar
7 0.81686234 1261 andrew gelman stats-2012-04-12-The Naval Research Lab
8 0.81584364 2275 andrew gelman stats-2014-03-31-Just gave a talk
9 0.8112843 2240 andrew gelman stats-2014-03-10-On deck this week: Things people sent me
10 0.80775261 1668 andrew gelman stats-2013-01-11-My talk at the NY data visualization meetup this Monday!
11 0.80699772 1143 andrew gelman stats-2012-01-29-G+ > Skype
12 0.79617739 1897 andrew gelman stats-2013-06-13-When’s that next gamma-ray blast gonna come, already?
13 0.77945101 548 andrew gelman stats-2011-02-01-What goes around . . .
14 0.77650893 716 andrew gelman stats-2011-05-17-Is the internet causing half the rapes in Norway? I wanna see the scatterplot.
15 0.77413094 2290 andrew gelman stats-2014-04-14-On deck this week
16 0.77017033 1730 andrew gelman stats-2013-02-20-Unz on Unz
17 0.76896173 2131 andrew gelman stats-2013-12-12-My talk at Leuven, Sat 14 Dec
18 0.76686281 1273 andrew gelman stats-2012-04-20-Proposals for alternative review systems for scientific work
19 0.76156896 1349 andrew gelman stats-2012-05-28-Question 18 of my final exam for Design and Analysis of Sample Surveys
20 0.75933802 2068 andrew gelman stats-2013-10-18-G+ hangout for Bayesian Data Analysis course now! (actually, in 5 minutes)