andrew_gelman_stats andrew_gelman_stats-2011 andrew_gelman_stats-2011-695 knowledge-graph by maker-knowledge-mining
Source: html
Introduction: A graduate student in public health writes: I have been asked to do the statistical analysis for a medical unit that is delivering a pilot study of a program to [details redacted to prevent identification]. They are using a prospective, nonrandomized, cohort-controlled trial study design. The investigator thinks they can recruit only a small number of treatment and control cases, maybe less than 30 in total. After I told the Investigator that I cannot do anything statistically with a sample size that small, he responded that small sample sizes are common in this field, and he send me an example of analysis that someone had done on a similar study. So he still wants me to come up with a statistical plan. Is it unethical for me to do anything other than descriptive statistics? I think he should just stick to qualitative research. But the study she mentions above has 40 subjects and apparently had enough power to detect some effects. This is a pilot study after all so the n does n
sentIndex sentText sentNum sentScore
1 A graduate student in public health writes: I have been asked to do the statistical analysis for a medical unit that is delivering a pilot study of a program to [details redacted to prevent identification]. [sent-1, score-1.413]
2 They are using a prospective, nonrandomized, cohort-controlled trial study design. [sent-2, score-0.268]
3 The investigator thinks they can recruit only a small number of treatment and control cases, maybe less than 30 in total. [sent-3, score-0.527]
4 After I told the Investigator that I cannot do anything statistically with a sample size that small, he responded that small sample sizes are common in this field, and he send me an example of analysis that someone had done on a similar study. [sent-4, score-0.992]
5 So he still wants me to come up with a statistical plan. [sent-5, score-0.129]
6 Is it unethical for me to do anything other than descriptive statistics? [sent-6, score-0.255]
7 I think he should just stick to qualitative research. [sent-7, score-0.338]
8 But the study she mentions above has 40 subjects and apparently had enough power to detect some effects. [sent-8, score-0.443]
9 This is a pilot study after all so the n does not have to be large. [sent-9, score-0.67]
10 It’s not randomized though so I would think it would need a larger n because of the weak design. [sent-10, score-0.323]
11 My reply: My first, general, recommendation is that it always makes sense to talk with any person as if he is completely ethical. [sent-11, score-0.074]
12 If he is ethical, this is a good idea, and if he is not, you don’t want him to think you think badly of him. [sent-12, score-0.247]
13 If you are worried about a serious ethical problem, you can ask about it by saying something like, “From the outside, this could look pretty bad. [sent-13, score-0.239]
14 An outsider, seeing this plan, might think we are being dishonest etc. [sent-14, score-0.185]
15 To get to your specific question, there is really no such thing as a minimum acceptable sample size. [sent-18, score-0.479]
16 You can get statistical significance with n=5 if your signal is strong enough. [sent-19, score-0.402]
17 Generally, though, the purpose of a pilot study is not to get statistical significance but rather to get experience with the intervention and the measurements. [sent-20, score-1.155]
18 It’s ok to do a pilot analysis, recognizing that it probably won’t reach statistical significance. [sent-21, score-0.773]
19 Also, regardless of sample size, qualitative analysis is appropriate and necessary in any pilot study. [sent-22, score-1.099]
20 Finally, of course they should not imply that they can collect a larger sample size than they can actually do. [sent-23, score-0.653]
wordName wordTfidf (topN-words)
[('pilot', 0.484), ('investigator', 0.226), ('sample', 0.225), ('study', 0.186), ('qualitative', 0.182), ('ethical', 0.164), ('size', 0.163), ('redacted', 0.134), ('statistical', 0.129), ('nonrandomized', 0.126), ('small', 0.119), ('significance', 0.118), ('analysis', 0.115), ('delivering', 0.113), ('dishonest', 0.113), ('recruit', 0.11), ('badly', 0.103), ('larger', 0.102), ('prevent', 0.1), ('prospective', 0.098), ('unethical', 0.098), ('mentions', 0.097), ('regardless', 0.093), ('outsider', 0.093), ('acceptable', 0.092), ('detect', 0.088), ('intervention', 0.088), ('minimum', 0.087), ('collect', 0.087), ('recognizing', 0.087), ('stick', 0.084), ('unit', 0.082), ('trial', 0.082), ('descriptive', 0.081), ('signal', 0.08), ('identification', 0.079), ('imply', 0.076), ('randomized', 0.076), ('anything', 0.076), ('get', 0.075), ('worried', 0.075), ('recommendation', 0.074), ('weak', 0.073), ('reach', 0.073), ('think', 0.072), ('subjects', 0.072), ('thinks', 0.072), ('express', 0.07), ('graduate', 0.07), ('responded', 0.069)]
simIndex simValue blogId blogTitle
same-blog 1 1.0 695 andrew gelman stats-2011-05-04-Statistics ethics question
Introduction: A graduate student in public health writes: I have been asked to do the statistical analysis for a medical unit that is delivering a pilot study of a program to [details redacted to prevent identification]. They are using a prospective, nonrandomized, cohort-controlled trial study design. The investigator thinks they can recruit only a small number of treatment and control cases, maybe less than 30 in total. After I told the Investigator that I cannot do anything statistically with a sample size that small, he responded that small sample sizes are common in this field, and he send me an example of analysis that someone had done on a similar study. So he still wants me to come up with a statistical plan. Is it unethical for me to do anything other than descriptive statistics? I think he should just stick to qualitative research. But the study she mentions above has 40 subjects and apparently had enough power to detect some effects. This is a pilot study after all so the n does n
2 0.40057597 1317 andrew gelman stats-2012-05-13-Question 3 of my final exam for Design and Analysis of Sample Surveys
Introduction: 3. We discussed in class the best currently available method for estimating the proportion of military servicemembers who are gay. What is that method? (Recall the problems with the direct approach: there is no simple way to survey servicemembers at random, nor is it likely that they would answer such a question honestly.) Solution to question 2 From yesterday : 2. Which of the following are useful goals in a pilot study? (Indicate all that apply.) (a) You can search for statistical significance, then from that decide what to look for in a confirmatory analysis of your full dataset. (b) You can see if you find statistical significance in a pre-chosen comparison of interest. (c) You can examine the direction (positive or negative, even if not statistically significant) of comparisons of interest. (d) With a small sample size, you cannot hope to learn anything conclusive, but you can get a crude estimate of effect size and standard deviation which will be useful in a po
3 0.19834891 1315 andrew gelman stats-2012-05-12-Question 2 of my final exam for Design and Analysis of Sample Surveys
Introduction: 2. Which of the following are useful goals in a pilot study? (Indicate all that apply.) (a) You can search for statistical significance, then from that decide what to look for in a confirmatory analysis of your full dataset. (b) You can see if you find statistical significance in a pre-chosen comparison of interest. (c) You can examine the direction (positive or negative, even if not statistically significant) of comparisons of interest. (d) With a small sample size, you cannot hope to learn anything conclusive, but you can get a crude estimate of effect size and standard deviation which will be useful in a power analysis to help you decide how large your full study needs to be. (e) You can talk with survey respondents and get a sense of how they perceived your questions. (f) You get a chance to learn about practical difficulties with sampling, nonresponse, and question wording. (g) You can check if your sample is approximately representative of your population. Soluti
4 0.16174707 963 andrew gelman stats-2011-10-18-Question on Type M errors
Introduction: Inti Pedroso writes: Today during the group meeting at my new job we were revising a paper whose main conclusions were sustained by an ANOVA. One of the first observations is that the experiment had a small sample size. Interestingly (may not so), some of the reported effects (most of them interactions) were quite large. One of the experience group members said that “there is a common wisdom that one should not believe effects from small sample sizes but [he thinks] if they [the effects] are large enough to be picked on a small study they must be real large effects”. I argued that if the sample size is small one could incur on a M-type error in which the magnitude of the effect is being over-estimated and that if larger samples are evaluated the magnitude may become smaller and also the confidence intervals. The concept of M-type error is completely new to all other members of the group (on which I am in my second week) and I was given the job of finding a suitable ref to explain
5 0.14257419 820 andrew gelman stats-2011-07-25-Design of nonrandomized cluster sample study
Introduction: Rhoderick Machekano writes: I have a design question which has been bothering me and wonder if you can clear for me. In my line of work, we often conveniently select health centers and from those sample patients. When I am doing sample size estimation under this design do I account for the design effect – since I expect outcomes in patients from the same health center to be correlated? Given that I didn’t random sample the health facilities, is my only limitation that I cannot generalize the results and make group level comparisons in the analysis? My response: You can generalize the results even if you didn’t randomly sample the health facilities. The only thing is that your generalization applies to the implicit population of facilities to which your sample is representative. You could try to move further on this by considering facility-level predictors. Regarding sample size estimation, see chapter 20 .
7 0.12550105 2366 andrew gelman stats-2014-06-09-On deck this week
8 0.12243699 1966 andrew gelman stats-2013-08-03-Uncertainty in parameter estimates using multilevel models
9 0.11667291 107 andrew gelman stats-2010-06-24-PPS in Georgia
10 0.1155633 1074 andrew gelman stats-2011-12-20-Reading a research paper != agreeing with its claims
11 0.11323661 35 andrew gelman stats-2010-05-16-Another update on the spam email study
12 0.10747575 446 andrew gelman stats-2010-12-03-Is 0.05 too strict as a p-value threshold?
13 0.10581456 1878 andrew gelman stats-2013-05-31-How to fix the tabloids? Toward replicable social science research
14 0.10529792 2210 andrew gelman stats-2014-02-13-Stopping rules and Bayesian analysis
15 0.10489032 1671 andrew gelman stats-2013-01-13-Preregistration of Studies and Mock Reports
16 0.10307505 18 andrew gelman stats-2010-05-06-$63,000 worth of abusive research . . . or just a really stupid waste of time?
17 0.10265313 411 andrew gelman stats-2010-11-13-Ethical concerns in medical trials
18 0.10233808 1695 andrew gelman stats-2013-01-28-Economists argue about Bayes
19 0.1023258 2223 andrew gelman stats-2014-02-24-“Edlin’s rule” for routinely scaling down published estimates
20 0.10111214 1150 andrew gelman stats-2012-02-02-The inevitable problems with statistical significance and 95% intervals
topicId topicWeight
[(0, 0.203), (1, -0.011), (2, 0.029), (3, -0.133), (4, 0.031), (5, 0.025), (6, -0.014), (7, 0.059), (8, -0.017), (9, -0.071), (10, -0.034), (11, -0.027), (12, 0.076), (13, 0.001), (14, 0.0), (15, -0.049), (16, -0.014), (17, -0.015), (18, 0.032), (19, 0.022), (20, -0.037), (21, -0.021), (22, 0.001), (23, 0.031), (24, -0.06), (25, 0.026), (26, -0.008), (27, -0.018), (28, -0.03), (29, -0.029), (30, -0.001), (31, 0.0), (32, -0.007), (33, 0.091), (34, 0.011), (35, 0.087), (36, -0.044), (37, -0.027), (38, -0.04), (39, 0.009), (40, 0.071), (41, -0.021), (42, -0.036), (43, 0.062), (44, 0.066), (45, -0.072), (46, 0.003), (47, -0.041), (48, 0.021), (49, -0.019)]
simIndex simValue blogId blogTitle
same-blog 1 0.97178257 695 andrew gelman stats-2011-05-04-Statistics ethics question
Introduction: A graduate student in public health writes: I have been asked to do the statistical analysis for a medical unit that is delivering a pilot study of a program to [details redacted to prevent identification]. They are using a prospective, nonrandomized, cohort-controlled trial study design. The investigator thinks they can recruit only a small number of treatment and control cases, maybe less than 30 in total. After I told the Investigator that I cannot do anything statistically with a sample size that small, he responded that small sample sizes are common in this field, and he send me an example of analysis that someone had done on a similar study. So he still wants me to come up with a statistical plan. Is it unethical for me to do anything other than descriptive statistics? I think he should just stick to qualitative research. But the study she mentions above has 40 subjects and apparently had enough power to detect some effects. This is a pilot study after all so the n does n
2 0.88926375 1317 andrew gelman stats-2012-05-13-Question 3 of my final exam for Design and Analysis of Sample Surveys
Introduction: 3. We discussed in class the best currently available method for estimating the proportion of military servicemembers who are gay. What is that method? (Recall the problems with the direct approach: there is no simple way to survey servicemembers at random, nor is it likely that they would answer such a question honestly.) Solution to question 2 From yesterday : 2. Which of the following are useful goals in a pilot study? (Indicate all that apply.) (a) You can search for statistical significance, then from that decide what to look for in a confirmatory analysis of your full dataset. (b) You can see if you find statistical significance in a pre-chosen comparison of interest. (c) You can examine the direction (positive or negative, even if not statistically significant) of comparisons of interest. (d) With a small sample size, you cannot hope to learn anything conclusive, but you can get a crude estimate of effect size and standard deviation which will be useful in a po
Introduction: Brendan Nyhan sends me this article from the research-methods all-star team of Katherine Button, John Ioannidis, Claire Mokrysz, Brian Nosek , Jonathan Flint, Emma Robinson, and Marcus Munafo: A study with low statistical power has a reduced chance of detecting a true effect, but it is less well appreciated that low power also reduces the likelihood that a statistically significant result reflects a true effect. Here, we show that the average statistical power of studies in the neurosciences is very low. The consequences of this include overestimates of effect size and low reproducibility of results. There are also ethical dimensions to this problem, as unreliable research is inefficient and wasteful. Improving reproducibility in neuroscience is a key priority and requires attention to well-established but often ignored methodological principles. I agree completely. In my terminology, with small sample size, the classical approach of looking for statistical significance leads
4 0.80677229 1315 andrew gelman stats-2012-05-12-Question 2 of my final exam for Design and Analysis of Sample Surveys
Introduction: 2. Which of the following are useful goals in a pilot study? (Indicate all that apply.) (a) You can search for statistical significance, then from that decide what to look for in a confirmatory analysis of your full dataset. (b) You can see if you find statistical significance in a pre-chosen comparison of interest. (c) You can examine the direction (positive or negative, even if not statistically significant) of comparisons of interest. (d) With a small sample size, you cannot hope to learn anything conclusive, but you can get a crude estimate of effect size and standard deviation which will be useful in a power analysis to help you decide how large your full study needs to be. (e) You can talk with survey respondents and get a sense of how they perceived your questions. (f) You get a chance to learn about practical difficulties with sampling, nonresponse, and question wording. (g) You can check if your sample is approximately representative of your population. Soluti
5 0.78479218 820 andrew gelman stats-2011-07-25-Design of nonrandomized cluster sample study
Introduction: Rhoderick Machekano writes: I have a design question which has been bothering me and wonder if you can clear for me. In my line of work, we often conveniently select health centers and from those sample patients. When I am doing sample size estimation under this design do I account for the design effect – since I expect outcomes in patients from the same health center to be correlated? Given that I didn’t random sample the health facilities, is my only limitation that I cannot generalize the results and make group level comparisons in the analysis? My response: You can generalize the results even if you didn’t randomly sample the health facilities. The only thing is that your generalization applies to the implicit population of facilities to which your sample is representative. You could try to move further on this by considering facility-level predictors. Regarding sample size estimation, see chapter 20 .
6 0.73556232 1150 andrew gelman stats-2012-02-02-The inevitable problems with statistical significance and 95% intervals
7 0.73044771 898 andrew gelman stats-2011-09-10-Fourteen magic words: an update
8 0.7285068 1776 andrew gelman stats-2013-03-25-The harm done by tests of significance
11 0.7256977 963 andrew gelman stats-2011-10-18-Question on Type M errors
12 0.724666 1551 andrew gelman stats-2012-10-28-A convenience sample and selected treatments
13 0.70602661 957 andrew gelman stats-2011-10-14-Questions about a study of charter schools
14 0.70174634 2223 andrew gelman stats-2014-02-24-“Edlin’s rule” for routinely scaling down published estimates
16 0.69785964 2049 andrew gelman stats-2013-10-03-On house arrest for p-hacking
18 0.69257033 1557 andrew gelman stats-2012-11-01-‘Researcher Degrees of Freedom’
19 0.69255257 1702 andrew gelman stats-2013-02-01-Don’t let your standard errors drive your research agenda
topicId topicWeight
[(9, 0.022), (15, 0.02), (16, 0.093), (19, 0.057), (21, 0.033), (24, 0.162), (48, 0.011), (65, 0.01), (82, 0.022), (86, 0.022), (89, 0.018), (95, 0.016), (98, 0.078), (99, 0.336)]
simIndex simValue blogId blogTitle
1 0.97416222 2234 andrew gelman stats-2014-03-05-Plagiarism, Arizona style
Introduction: Last month a history professor sent me a note regarding plagiarism at Arizona State University: Matthew Whitaker, who had received an expedited promotion to full professor and was made Director of a new Center for the Study of Race and Democracy by Provost Elizabeth Capaldi and President Michael Crow, was charged by most of the full professors in the History Faculty with having plagiarized throughout his corpus of work, copying from regular works of scholarship and from web sources. Indeed, in his response, which claimed that the petitioners were racist, Whitaker admitted to plagiarism in his work, defending himself in part by stating that he had not reviewed carefully the research and writing he had hired others to do. . . . What bothered my correspondent was that Whitaker remains an ASU Foundation Professor of History despite all the plaig. According to Whitaker’s webpage , he “is also a highly sought after speaker, having offered commentaries on NPR, PBS, . . . and other medi
2 0.97262383 1867 andrew gelman stats-2013-05-22-To Throw Away Data: Plagiarism as a Statistical Crime
Introduction: I’ve been blogging a lot lately about plagiarism (sorry, Bob!), and one thing that’s been bugging me is, why does it bother me so much. Part of the story is simple: much of my reputation comes from the words I write, so I bristle at any attempt to devalue words. I feel the same way about plagiarism that a rich person would feel about counterfeiting: Don’t debase my currency! But it’s more than that. After discussing this a bit with Thomas Basbøll, I realized that I’m bothered by the way that plagiarism interferes with the transmission of information: Much has been written on the ethics of plagiarism. One aspect that has received less notice is plagiarism’s role in corrupting our ability to learn from data: We propose that plagiarism is a statistical crime. It involves the hiding of important information regarding the source and context of the copied work in its original form. Such information can dramatically alter the statistical inferences made about the work. In statisti
same-blog 3 0.97176182 695 andrew gelman stats-2011-05-04-Statistics ethics question
Introduction: A graduate student in public health writes: I have been asked to do the statistical analysis for a medical unit that is delivering a pilot study of a program to [details redacted to prevent identification]. They are using a prospective, nonrandomized, cohort-controlled trial study design. The investigator thinks they can recruit only a small number of treatment and control cases, maybe less than 30 in total. After I told the Investigator that I cannot do anything statistically with a sample size that small, he responded that small sample sizes are common in this field, and he send me an example of analysis that someone had done on a similar study. So he still wants me to come up with a statistical plan. Is it unethical for me to do anything other than descriptive statistics? I think he should just stick to qualitative research. But the study she mentions above has 40 subjects and apparently had enough power to detect some effects. This is a pilot study after all so the n does n
4 0.96832365 625 andrew gelman stats-2011-03-23-My last post on albedo, I promise
Introduction: After seeing my recent blogs on Nathan Myhrvold, a friend told me that, in the tech world, the albedo-obsessed genius is known as a patent troll. Really? Yup. My friend writes: It’s perhaps indicative that Myhrvold comes up in the top-ten hits on Google for [patent troll]. These blog posts lay it out pretty clearly: http://www.techdirt.com/articles/20100217/1853298215.shtml http://blogs.seattleweekly.com/dailyweekly/2010/12/giant_patent_troll_awakens_as.php http://bits.blogs.nytimes.com/2010/12/08/intellectual-ventures-goes-to-court Just about anyone’s that’s been in the tech game thinks patents are ridiculous. The lab where I used to work wanted us to create an “intellectual mine field” in our field so the companycould block anyone from entering the space. Yes, we made stuff, but the patents were for totally obvious ideas that anyone would have. Even Google’s PageRank was just a simple application of standard social network analysis models of authorities in netw
Introduction: There’s a paradigm in applied statistics that goes something like this: 1. There is a scientific or policy question of some theoretical or practical importance. 2. Researchers gather data on relevant outcomes and perform a statistical analysis, ideally leading to a clear conclusion (p less than 0.05, or a strong posterior distribution, or good predictive performance, or high reliability and validity, whatever). 3. This conclusion informs policy. This paradigm has room for positive findings (for example, that a new program is statistically significantly better, or statistically significantly worse than what came before) or negative findings (data are inconclusive, further study is needed), even if negative findings seem less likely to make their way into the textbooks. But what happens when step 2 simply isn’t possible. This came up a few years ago—nearly 10 years ago, now!—with the excellent paper by Donohue and Wolfers which explained why it’s just about impossible to
6 0.96669322 765 andrew gelman stats-2011-06-14-How the ignorant idiots win, explained. Maybe.
8 0.96455431 955 andrew gelman stats-2011-10-12-Why it doesn’t make sense to chew people out for not reading the help page
9 0.96333301 2160 andrew gelman stats-2014-01-06-Spam names
10 0.9626683 288 andrew gelman stats-2010-09-21-Discussion of the paper by Girolami and Calderhead on Bayesian computation
11 0.96235383 2223 andrew gelman stats-2014-02-24-“Edlin’s rule” for routinely scaling down published estimates
12 0.96222037 2334 andrew gelman stats-2014-05-14-“The subtle funk of just a little poultry offal”
13 0.96216065 1587 andrew gelman stats-2012-11-21-Red state blue state, or, states and counties are not persons
14 0.96184617 1853 andrew gelman stats-2013-05-12-OpenData Latinoamerica
15 0.96125978 18 andrew gelman stats-2010-05-06-$63,000 worth of abusive research . . . or just a really stupid waste of time?
16 0.96117407 1444 andrew gelman stats-2012-08-05-Those darn conservative egalitarians
17 0.96103621 1760 andrew gelman stats-2013-03-12-Misunderstanding the p-value
18 0.96084595 1910 andrew gelman stats-2013-06-22-Struggles over the criticism of the “cannabis users and IQ change” paper
20 0.96051079 966 andrew gelman stats-2011-10-20-A qualified but incomplete thanks to Gregg Easterbrook’s editor at Reuters