andrew_gelman_stats andrew_gelman_stats-2010 andrew_gelman_stats-2010-35 knowledge-graph by maker-knowledge-mining
Source: html
Introduction: I think youall are probably getting sick of this by now so I’ll put it all below the fold. Akinola Modupe and Katherine Milkman responded to my email about their study : We want to clarify the reason we believe that the use of deception and a lack of informed consent were appropriate and ethical for this research study. In this project, we were studying how the timing of a decision affects discrimination based on race and/or gender. The emails all participants in our study received were identical except for a) the sender’s name (we used 20 names that pretesting revealed were strongly associated with being either Caucasian, Black, Indian, Chinese or Hispanic, as well as associated with being male or female) and b) whether the meeting requested was for today or for a week from today. Recipients were randomly selected and were randomly assigned to one of the race/gender/timing conditions. This study design will allow us to test for baseline levels of discrimination in acade
sentIndex sentText sentNum sentScore
1 Akinola Modupe and Katherine Milkman responded to my email about their study : We want to clarify the reason we believe that the use of deception and a lack of informed consent were appropriate and ethical for this research study. [sent-2, score-0.929]
2 This study design will allow us to test for baseline levels of discrimination in academia (e. [sent-6, score-0.629]
3 To study discrimination, it is crucial that study participants are not aware they are being observed, as this knowledge has been shown to dramatically change peoples’ behavior. [sent-14, score-0.776]
4 Thus, we obtained permission from our IRBs to employ deception. [sent-15, score-0.312]
5 To study discrimination, it is also crucial that informed consent be waived. [sent-16, score-0.655]
6 If participants were offered an opportunity to opt into our research study even without knowing its purpose, it would likely have biased our sample towards including only those professors who would agree to a meeting. [sent-17, score-0.615]
7 “Audit” research employing deception and a waiver of informed consent has been conducted many times before, and in fact, typically with far more costs to participants. [sent-18, score-0.537]
8 Past research of this type has examined discrimination in housing markets, employment markets, car purchases, and mortgage applications (see for example Ayres & Siegelman (1995), Bertrand & Mullainathan (2004), Kenney & Wissoker (1994), Neumark et al. [sent-19, score-0.33]
9 There were two potential downsides to our study that some of the postings on your blog have highlighted and that we attempted to mitigate with our experimental design. [sent-21, score-0.423]
10 In some cases, our study (1) absorbed faculty time and (2) created a potential negative externality (our study may have reduced faculty members’ likelihood of responding to future messages from prospective PhD students). [sent-22, score-1.486]
11 We believe that the importance of gaining an understanding of discrimination in academia is worth the sacrifice of faculty time imposed by our request for a quickly-cancelled 10-minute meeting. [sent-24, score-0.673]
12 Regarding the latter issue (a negative externality), we hope faculty will not neglect emails from prospective PhD students in the future as a result of our study. [sent-25, score-0.439]
13 As I noted in one of the comment threads, I can respect the view that responding to an email is a minor effort that should not be worth $10–but I’d like to be the judge of that! [sent-36, score-0.271]
14 In general, I’d think you’d get happier participants if you asked permission ahead of time or compensated afterward. [sent-37, score-0.492]
15 As also discussed, it would not be hard to ask for permission in a study that involves deception, for example by sending an email several weeks in advance saying that they are b-school professors doing a research project and that they will contact me in a few weeks by email. [sent-40, score-1.041]
16 Then, if I agree to participate, the rest of the study goes as planned, and I’d have no idea that the purported student’s email is fake. [sent-42, score-0.392]
17 Or the researchers could do the study as is, and then offer some compensation at the end. [sent-43, score-0.291]
18 Again, if the study is worth bothering 6300 people (without asking for their permission), I think it’s worth compensating these people a bit. [sent-44, score-0.665]
19 To return to point 1, I certainly don’t feel the people who did this study should be punished in any way, nor do I feel that the IRB’s were asleep at the switch. [sent-46, score-0.358]
20 ) I think it’s fine to approve this sort of study, and I also favor ethics guidelines under which it would be expected to ask the permission of research subjects, or to compensate them afterward (or both). [sent-49, score-0.549]
wordName wordTfidf (topN-words)
[('permission', 0.312), ('study', 0.291), ('discrimination', 0.257), ('irb', 0.242), ('consent', 0.184), ('deception', 0.174), ('faculty', 0.167), ('lamar', 0.135), ('externality', 0.123), ('participants', 0.12), ('informed', 0.106), ('compensate', 0.104), ('future', 0.102), ('absorbed', 0.101), ('email', 0.101), ('prospective', 0.09), ('worth', 0.09), ('anderson', 0.089), ('brad', 0.087), ('uncomfortable', 0.082), ('academia', 0.081), ('emails', 0.08), ('responding', 0.08), ('request', 0.078), ('phd', 0.076), ('meeting', 0.075), ('markets', 0.074), ('crucial', 0.074), ('potential', 0.074), ('research', 0.073), ('entries', 0.071), ('randomly', 0.07), ('professors', 0.07), ('washington', 0.069), ('contact', 0.068), ('people', 0.067), ('weeks', 0.063), ('akinola', 0.061), ('milkman', 0.061), ('modupe', 0.061), ('irbs', 0.061), ('opt', 0.061), ('sender', 0.061), ('think', 0.06), ('extremely', 0.06), ('bertrand', 0.058), ('mitigate', 0.058), ('pager', 0.058), ('peoples', 0.058), ('purchases', 0.058)]
simIndex simValue blogId blogTitle
same-blog 1 1.0000004 35 andrew gelman stats-2010-05-16-Another update on the spam email study
Introduction: I think youall are probably getting sick of this by now so I’ll put it all below the fold. Akinola Modupe and Katherine Milkman responded to my email about their study : We want to clarify the reason we believe that the use of deception and a lack of informed consent were appropriate and ethical for this research study. In this project, we were studying how the timing of a decision affects discrimination based on race and/or gender. The emails all participants in our study received were identical except for a) the sender’s name (we used 20 names that pretesting revealed were strongly associated with being either Caucasian, Black, Indian, Chinese or Hispanic, as well as associated with being male or female) and b) whether the meeting requested was for today or for a week from today. Recipients were randomly selected and were randomly assigned to one of the race/gender/timing conditions. This study design will allow us to test for baseline levels of discrimination in acade
2 0.32705829 18 andrew gelman stats-2010-05-06-$63,000 worth of abusive research . . . or just a really stupid waste of time?
Introduction: As someone who relies strongly on survey research, it’s good for me to be reminded that some surveys are useful, some are useless, but one thing they almost all have in common is . . . they waste the respondents’ time. I thought of this after receiving the following email, which I shall reproduce here. My own comments appear after. Recently, you received an email from a student asking for 10 minutes of your time to discuss your Ph.D. program (the body of the email appears below). We are emailing you today to debrief you on the actual purpose of that email, as it was part of a research study. We sincerely hope our study did not cause you any disruption and we apologize if you were at all inconvenienced. Our hope is that this letter will provide a sufficient explanation of the purpose and design of our study to alleviate any concerns you may have about your involvement. We want to thank you for your time and for reading further if you are interested in understanding why you rece
3 0.268534 1055 andrew gelman stats-2011-12-13-Data sharing update
Introduction: Fred Oswald reports that Sian Beilock sent him sufficient amounts of raw data from her research study so allow him to answer his questions about the large effects that were observed. This sort of collegiality is central to the collective scientific enterprise. The bad news is that IRB’s are still getting in the way. Beilock was very helpful but she had to work within the constraints of her IRB, which apparently advised her not to share data—even if de-identified—without getting lots more permissions. Oswald writes: It is a little concerning that the IRB bars the sharing of de-identified data, particularly in light of the specific guidelines of the journal Science, which appears to say that when you submit a study to the journal for publication, you are allowing for the sharing of de-identified data — unless you expressly say otherwise at the point that you submit the paper for consideration. Again, I don’t blame Beilock and Ramirez—they appear to have been as helpful as
4 0.1902055 27 andrew gelman stats-2010-05-11-Update on the spam email study
Introduction: A few days ago I reported on the spam email that I received from two business school professors (one at Columbia)! As noted on the blog, I sent an email directly to the study’s authors at the time of reading the email, but they have yet to respond. This surprises me a bit. Certainly if 6300 faculty each have time to respond to one email on this study, the two faculty have time to respond to 6300 email replies, no? I was actually polite enough to respond to both of their emails! If I do hear back, I’ll let youall know! P.S. Paul Basken interviewed me briefly for a story in the Chronicle of Higher Education on the now-notorious spam email study. Basken’s article is reasonable–he points out that (a) the study irritated a lot of people, but (b) is ultimately no big deal. One interesting thing about the article is that, although some people felt that the spam email study was ethical, nobody came forth with an argument that the study was actually worth doing. P.P.S. In
5 0.17978682 1539 andrew gelman stats-2012-10-18-IRB nightmares
Introduction: Andrew Perrin nails it : Twice a year, like clockwork, the ethics cops at the IRB [institutional review board, the group on campus that has to approve research involving human subjects] take a break from deciding whether or not radioactive isotopes can be administered to prison populations to cure restless-leg syndrome to dream up some fancy new way in which participating in an automated telephone poll might cause harm. Perrin adds: The list of exemptions to IRB review is too short and, more importantly, contains no guiding principle as to what makes exempt. . . . [and] Even exemptions require approval by the IRB. He also voices a thought I’ve had many times, which is that there are all sorts of things you or I or anyone else can do on the street (for example, go up to people and ask them personal questions, drop objects and see if people pick them up, stage fights with our friends to see the reactions of bystanders, etc etc etc) but for which we have to go through an IRB
6 0.16842122 458 andrew gelman stats-2010-12-08-Blogging: Is it “fair use”?
7 0.12945178 1053 andrew gelman stats-2011-12-11-This one is so dumb it makes me want to barf
8 0.12139026 1437 andrew gelman stats-2012-07-31-Paying survey respondents
10 0.11521972 1643 andrew gelman stats-2012-12-29-Sexism in science (as elsewhere)
11 0.11323661 695 andrew gelman stats-2011-05-04-Statistics ethics question
13 0.10444173 1368 andrew gelman stats-2012-06-06-Question 27 of my final exam for Design and Analysis of Sample Surveys
14 0.10220477 2301 andrew gelman stats-2014-04-22-Ticket to Baaaaarf
15 0.10085838 1054 andrew gelman stats-2011-12-12-More frustrations trying to replicate an analysis published in a reputable journal
17 0.10028718 571 andrew gelman stats-2011-02-13-A departmental wiki page?
18 0.099894702 2210 andrew gelman stats-2014-02-13-Stopping rules and Bayesian analysis
19 0.095178783 1289 andrew gelman stats-2012-04-29-We go to war with the data we have, not the data we want
topicId topicWeight
[(0, 0.198), (1, -0.077), (2, 0.012), (3, -0.113), (4, 0.013), (5, 0.051), (6, 0.026), (7, 0.021), (8, -0.061), (9, -0.027), (10, -0.027), (11, -0.003), (12, 0.058), (13, 0.003), (14, -0.024), (15, 0.036), (16, 0.07), (17, -0.048), (18, -0.016), (19, 0.069), (20, -0.013), (21, -0.025), (22, -0.032), (23, -0.056), (24, 0.006), (25, -0.003), (26, 0.027), (27, -0.018), (28, 0.012), (29, 0.011), (30, -0.083), (31, -0.014), (32, -0.025), (33, 0.084), (34, -0.021), (35, 0.006), (36, 0.035), (37, -0.016), (38, -0.045), (39, 0.008), (40, 0.096), (41, -0.002), (42, -0.021), (43, -0.044), (44, 0.02), (45, -0.056), (46, 0.012), (47, 0.014), (48, 0.035), (49, 0.012)]
simIndex simValue blogId blogTitle
same-blog 1 0.96694362 35 andrew gelman stats-2010-05-16-Another update on the spam email study
Introduction: I think youall are probably getting sick of this by now so I’ll put it all below the fold. Akinola Modupe and Katherine Milkman responded to my email about their study : We want to clarify the reason we believe that the use of deception and a lack of informed consent were appropriate and ethical for this research study. In this project, we were studying how the timing of a decision affects discrimination based on race and/or gender. The emails all participants in our study received were identical except for a) the sender’s name (we used 20 names that pretesting revealed were strongly associated with being either Caucasian, Black, Indian, Chinese or Hispanic, as well as associated with being male or female) and b) whether the meeting requested was for today or for a week from today. Recipients were randomly selected and were randomly assigned to one of the race/gender/timing conditions. This study design will allow us to test for baseline levels of discrimination in acade
2 0.90359199 18 andrew gelman stats-2010-05-06-$63,000 worth of abusive research . . . or just a really stupid waste of time?
Introduction: As someone who relies strongly on survey research, it’s good for me to be reminded that some surveys are useful, some are useless, but one thing they almost all have in common is . . . they waste the respondents’ time. I thought of this after receiving the following email, which I shall reproduce here. My own comments appear after. Recently, you received an email from a student asking for 10 minutes of your time to discuss your Ph.D. program (the body of the email appears below). We are emailing you today to debrief you on the actual purpose of that email, as it was part of a research study. We sincerely hope our study did not cause you any disruption and we apologize if you were at all inconvenienced. Our hope is that this letter will provide a sufficient explanation of the purpose and design of our study to alleviate any concerns you may have about your involvement. We want to thank you for your time and for reading further if you are interested in understanding why you rece
3 0.76409072 1191 andrew gelman stats-2012-03-01-Hoe noem je?
Introduction: Gerrit Storms reports on an interesting linguistic research project in which you can participate! Here’s the description: Over the past few weeks, we have been trying to set up a scientific study that is important for many researchers interested in words, word meaning, semantics, and cognitive science in general. It is a huge word association project, in which people are asked to participate in a small task that doesn’t last longer than 5 minutes. Our goal is to build a global word association network that contains connections between about 40,000 words, the size of the lexicon of an average adult. Setting up such a network might learn us a lot about semantic memory, how it develops, and maybe also about how it can deteriorate (like in Alzheimer’s disease). Most people enjoy doing the task, but we need thousands of participants to succeed. Up till today, we found about 53,000 participants willing to do the little task, but we need more subjects. That is why we address you. Would
4 0.76327085 1539 andrew gelman stats-2012-10-18-IRB nightmares
Introduction: Andrew Perrin nails it : Twice a year, like clockwork, the ethics cops at the IRB [institutional review board, the group on campus that has to approve research involving human subjects] take a break from deciding whether or not radioactive isotopes can be administered to prison populations to cure restless-leg syndrome to dream up some fancy new way in which participating in an automated telephone poll might cause harm. Perrin adds: The list of exemptions to IRB review is too short and, more importantly, contains no guiding principle as to what makes exempt. . . . [and] Even exemptions require approval by the IRB. He also voices a thought I’ve had many times, which is that there are all sorts of things you or I or anyone else can do on the street (for example, go up to people and ask them personal questions, drop objects and see if people pick them up, stage fights with our friends to see the reactions of bystanders, etc etc etc) but for which we have to go through an IRB
5 0.76296097 1053 andrew gelman stats-2011-12-11-This one is so dumb it makes me want to barf
Introduction: Dan Kahan sends in this horror story: A new study finds that atheists are among society’s most distrusted group, comparable even to rapists in certain circumstances. Psychologists at the University of British Columbia and the University of Oregon say that their study demonstrates that anti-atheist prejudice stems from moral distrust, not dislike, of nonbelievers. “It’s pretty remarkable,” said Azim Shariff, an assistant professor of psychology at the University of Oregon and a co-author of the study, which appears in the current issue of Journal of Personality and Social Psychology. The study, conducted among 350 Americans adults and 420 Canadian college students, asked participants to decide if a fictional driver damaged a parked car and left the scene, then found a wallet and took the money, was the driver more likely to be a teacher, an atheist teacher, or a rapist teacher? The participants, who were from religious and nonreligious backgrounds, most often chose the athe
7 0.75029504 2301 andrew gelman stats-2014-04-22-Ticket to Baaaaarf
8 0.73693466 27 andrew gelman stats-2010-05-11-Update on the spam email study
11 0.72031373 411 andrew gelman stats-2010-11-13-Ethical concerns in medical trials
12 0.71758282 1163 andrew gelman stats-2012-02-12-Meta-analysis, game theory, and incentives to do replicable research
14 0.71598786 350 andrew gelman stats-2010-10-18-Subtle statistical issues to be debated on TV.
15 0.71402323 1618 andrew gelman stats-2012-12-11-The consulting biz
16 0.69528443 1184 andrew gelman stats-2012-02-25-Facebook Profiles as Predictors of Job Performance? Maybe…but not yet.
17 0.69075507 2367 andrew gelman stats-2014-06-10-Spring forward, fall back, drop dead?
18 0.69065511 568 andrew gelman stats-2011-02-11-Calibration in chess
19 0.68727249 326 andrew gelman stats-2010-10-07-Peer pressure, selection, and educational reform
20 0.68570471 48 andrew gelman stats-2010-05-23-The bane of many causes
topicId topicWeight
[(2, 0.016), (13, 0.012), (15, 0.011), (16, 0.063), (21, 0.025), (24, 0.171), (42, 0.014), (47, 0.038), (57, 0.099), (63, 0.013), (81, 0.018), (86, 0.051), (95, 0.012), (97, 0.024), (99, 0.259)]
simIndex simValue blogId blogTitle
1 0.97176218 1460 andrew gelman stats-2012-08-16-“Real data can be a pain”
Introduction: Michael McLaughlin sent me the following query with the above title. Some time ago, I [McLaughlin] was handed a dataset that needed to be modeled. It was generated as follows: 1. Random navigation errors, historically a binary mixture of normal and Laplace with a common mean, were collected by observation. 2. Sadly, these data were recorded with too few decimal places so that the resulting quantization is clearly visible in a scatterplot. 3. The quantized data were then interpolated (to an unobserved location). The final result looks like fuzzy points (small scale jitter) at quantized intervals spanning a much larger scale (the parent mixture distribution). This fuzziness, likely ~normal or ~Laplace, results from the interpolation. Otherwise, the data would look like a discrete analogue of the normal/Laplace mixture. I would like to characterize the latent normal/Laplace mixture distribution but the quantization is “getting in the way”. When I tried MCMC on this proble
2 0.96532452 1101 andrew gelman stats-2012-01-05-What are the standards for reliability in experimental psychology?
Introduction: An experimental psychologist was wondering about the standards in that field for “acceptable reliability” (when looking at inter-rater reliability in coding data). He wondered, for example, if some variation on signal detectability theory might be applied to adjust for inter-rater differences in criteria for saying some code is present. What about Cohen’s kappa? The psychologist wrote: Cohen’s kappa does adjust for “guessing,” but its assumptions are not well motivated, perhaps not any more than adjustments for guessing versus the application of signal detectability theory where that can be applied. But one can’t do a straightforward application of signal detectability theory for reliability in that you don’t know whether the signal is present or not. I think measurement issues are important but I don’t have enough experience in this area to answer the question without knowing more about the problem that this researcher is working on. I’m posting it here because I imagine t
3 0.96302843 1036 andrew gelman stats-2011-11-30-Stan uses Nuts!
Introduction: We interrupt our usual program of Ed Wegman Gregg Easterbrook Niall Ferguson mockery to deliver a serious update on our statistical computing project. Stan (“Sampling Through Adaptive Neighborhoods”) is our new C++ program (written mostly by Bob Carpenter) that draws samples from Bayesian models. Stan can take different sorts of inputs: you can write the model in a Bugs-like syntax and it goes from there, or you can write the log-posterior directly as a C++ function. Most of the computation is done using Hamiltonian Monte Carlo. HMC requires some tuning, so Matt Hoffman up and wrote a new algorithm, Nuts (the “No-U-Turn Sampler”) which optimizes HMC adaptively. In many settings, Nuts is actually more computationally efficient than the optimal static HMC! When the the Nuts paper appeared on Arxiv, Christian Robert noticed it and had some reactions . In response to Xian’s comments, Matt writes: Christian writes: I wonder about the computing time (and the “una
same-blog 4 0.95956659 35 andrew gelman stats-2010-05-16-Another update on the spam email study
Introduction: I think youall are probably getting sick of this by now so I’ll put it all below the fold. Akinola Modupe and Katherine Milkman responded to my email about their study : We want to clarify the reason we believe that the use of deception and a lack of informed consent were appropriate and ethical for this research study. In this project, we were studying how the timing of a decision affects discrimination based on race and/or gender. The emails all participants in our study received were identical except for a) the sender’s name (we used 20 names that pretesting revealed were strongly associated with being either Caucasian, Black, Indian, Chinese or Hispanic, as well as associated with being male or female) and b) whether the meeting requested was for today or for a week from today. Recipients were randomly selected and were randomly assigned to one of the race/gender/timing conditions. This study design will allow us to test for baseline levels of discrimination in acade
5 0.9493438 1120 andrew gelman stats-2012-01-15-Fun fight over the Grover search algorithm
Introduction: Joshua Vogelstein points me to this blog entry by Robert Tucci, diplomatically titled “Unethical or Really Dumb (or both) Scientists from University of Adelaide ‘Rediscover’ My Version of Grover’s Algorithm”: The Chappell et al. paper has 24 references but does not refer to my paper, even though their paper and mine are eerily similar. Compare them yourself. With the excellent Google and ArXiv search engines, I [Tucci] would say there is zero probability that none of its five authors knew about my paper before they wrote theirs. Chappell responds in the comments: Your paper is timestamped 2010; however the results of our paper was initially presented at the Cairns CQIQC conference in July 2008. . . . The intention of our paper is not a research article. It is a tutorial paper. . . . We had not seen your paper before. Our paper is based on the standard Grover search, not a fixed point search. Hence, your paper did not come to our attention, as we were not concerned with
7 0.94451362 2299 andrew gelman stats-2014-04-21-Stan Model of the Week: Hierarchical Modeling of Supernovas
8 0.94259304 2318 andrew gelman stats-2014-05-04-Stan (& JAGS) Tutorial on Linear Mixed Models
10 0.94162786 2055 andrew gelman stats-2013-10-08-A Bayesian approach for peer-review panels? and a speculation about Bruno Frey
11 0.94103825 861 andrew gelman stats-2011-08-19-Will Stan work well with 40×40 matrices?
12 0.94073725 1486 andrew gelman stats-2012-09-07-Prior distributions for regression coefficients
14 0.93879628 1713 andrew gelman stats-2013-02-08-P-values and statistical practice
15 0.93865848 1218 andrew gelman stats-2012-03-18-Check your missing-data imputations using cross-validation
17 0.93753231 2015 andrew gelman stats-2013-09-10-The ethics of lying, cheating, and stealing with data: A case study
18 0.93737555 899 andrew gelman stats-2011-09-10-The statistical significance filter
19 0.93588376 1117 andrew gelman stats-2012-01-13-What are the important issues in ethics and statistics? I’m looking for your input!
20 0.9351455 2183 andrew gelman stats-2014-01-23-Discussion on preregistration of research studies