andrew_gelman_stats andrew_gelman_stats-2012 andrew_gelman_stats-2012-1527 knowledge-graph by maker-knowledge-mining
Source: html
Introduction: John Cook considers how people justify probability distribution assumptions: Sometimes distribution assumptions are not justified. Sometimes distributions can be derived from fundamental principles [or] . . . on theoretical grounds. For example, large samples and the central limit theorem together may justify assuming that something is normally distributed. Often the choice of distribution is somewhat arbitrary, chosen by intuition or for convenience, and then empirically shown to work well enough. Sometimes a distribution can be a bad fit and still work well, depending on what you’re asking of it. Cook continues: The last point is particularly interesting. It’s not hard to imagine that a poor fit would produce poor results. It’s surprising when a poor fit produces good results. And then he gives an example of an effective but inaccurate model used to model survival times in a clinical trial. Cook explains: The [poorly-fitting] method works well because of the q
sentIndex sentText sentNum sentScore
1 John Cook considers how people justify probability distribution assumptions: Sometimes distribution assumptions are not justified. [sent-1, score-0.861]
2 For example, large samples and the central limit theorem together may justify assuming that something is normally distributed. [sent-6, score-0.805]
3 Often the choice of distribution is somewhat arbitrary, chosen by intuition or for convenience, and then empirically shown to work well enough. [sent-7, score-0.439]
4 Sometimes a distribution can be a bad fit and still work well, depending on what you’re asking of it. [sent-8, score-0.477]
5 It’s not hard to imagine that a poor fit would produce poor results. [sent-10, score-0.397]
6 And then he gives an example of an effective but inaccurate model used to model survival times in a clinical trial. [sent-12, score-0.782]
7 Cook explains: The [poorly-fitting] method works well because of the question being asked. [sent-13, score-0.398]
8 The method is not being asked to accurately model the distribution of survival times for patients in the trial. [sent-14, score-1.013]
9 As the simulations in this paper show, the method makes the right decision with high probability, even when the actual survival times are not exponentially distributed. [sent-16, score-0.526]
10 An example where a bad model works well because of its implicit assumptions In Section 9. [sent-18, score-0.807]
11 3 of Bayesian Data Analysis (second edition) , we compare several different methods for estimating a population total from a random sample in an artificial problem in which the population is the set of all cities and towns in a state. [sent-19, score-0.567]
12 The data are skewed—some cities have much more population than others—but if you use standard survey-sampling estimates and standard errors, you get OK inferences. [sent-20, score-0.395]
13 The inferences are not perfect—in particular, the confidence interval can include negative values because the brute-force approach doesn’t “know” that the data (city populations) are all positive—but the intervals make sense and have reasonable coverage properties. [sent-21, score-0.619]
14 In contrast, as Don Rubin showed when he first considered this example, comparable analyses applying the normal distribution to log or power-transformed data can give horrible answers. [sent-22, score-0.449]
15 How come the interval estimates based on these skewed data have reasonable coverage we use the normal distribution, while inferences based on the much more sensible lognormal or power-transformed models are so disastrous? [sent-24, score-1.081]
16 A quick answer is that the normal-theory method makes implicit use of the central limit theorem, but then this just pushes the question back one step: Why should the central limit theorem apply here? [sent-25, score-1.136]
17 The theorem applies for this finite sample (n=100, in this case) because, although the underlying distribution is skewed, there are no extreme outliers. [sent-27, score-0.619]
18 By using the normal-based interval, we are implicitly assuming a reasonable upper bound in the population. [sent-28, score-0.435]
19 And, in fact, if we put an upper bound into the power-transformed model, it works even better. [sent-29, score-0.397]
20 Sometimes an ill-fitting model works well because, although it doesn’t fit much of the data, it includes some assumption that is relevant to inferences, some aspect of the model that would be difficult to ascertain from the data alone. [sent-31, score-1.078]
wordName wordTfidf (topN-words)
[('distribution', 0.267), ('skewed', 0.224), ('theorem', 0.219), ('survival', 0.205), ('cook', 0.185), ('model', 0.171), ('limit', 0.166), ('interval', 0.165), ('works', 0.156), ('inferences', 0.141), ('method', 0.141), ('poor', 0.136), ('justify', 0.133), ('bound', 0.13), ('central', 0.13), ('cities', 0.128), ('assumptions', 0.126), ('sometimes', 0.125), ('fit', 0.125), ('coverage', 0.12), ('upper', 0.111), ('population', 0.109), ('implicit', 0.104), ('well', 0.101), ('reasonable', 0.101), ('assumption', 0.097), ('ascertain', 0.097), ('disastrous', 0.097), ('assuming', 0.093), ('times', 0.093), ('data', 0.092), ('normal', 0.09), ('towns', 0.087), ('exponentially', 0.087), ('bad', 0.085), ('lognormal', 0.082), ('pushes', 0.08), ('inaccurate', 0.078), ('asked', 0.072), ('empirically', 0.071), ('artificial', 0.069), ('although', 0.068), ('considers', 0.068), ('estimates', 0.066), ('populations', 0.065), ('convenience', 0.065), ('sample', 0.065), ('example', 0.064), ('normally', 0.064), ('patients', 0.064)]
simIndex simValue blogId blogTitle
same-blog 1 1.0000002 1527 andrew gelman stats-2012-10-10-Another reason why you can get good inferences from a bad model
Introduction: John Cook considers how people justify probability distribution assumptions: Sometimes distribution assumptions are not justified. Sometimes distributions can be derived from fundamental principles [or] . . . on theoretical grounds. For example, large samples and the central limit theorem together may justify assuming that something is normally distributed. Often the choice of distribution is somewhat arbitrary, chosen by intuition or for convenience, and then empirically shown to work well enough. Sometimes a distribution can be a bad fit and still work well, depending on what you’re asking of it. Cook continues: The last point is particularly interesting. It’s not hard to imagine that a poor fit would produce poor results. It’s surprising when a poor fit produces good results. And then he gives an example of an effective but inaccurate model used to model survival times in a clinical trial. Cook explains: The [poorly-fitting] method works well because of the q
2 0.20369205 2176 andrew gelman stats-2014-01-19-Transformations for non-normal data
Introduction: Steve Peterson writes: I recently submitted a proposal on applying a Bayesian analysis to gender comparisons on motivational constructs. I had an idea on how to improve the model I used and was hoping you could give me some feedback. The data come from a survey based on 5-point Likert scales. Different constructs are measured for each student as scores derived from averaging a student’s responses on particular subsets of survey questions. (I suppose it is not uncontroversial to treat these scores as interval measures and would be interested to hear if you have any objections.) I am comparing genders on each construct. Researchers typically use t-tests to do so. To use a Bayesian approach I applied the programs written in R and JAGS by John Kruschke for estimating the difference of means: http://www.indiana.edu/~kruschke/BEST/ An issue in that analysis is that the distributions of student scores are not normal. There was skewness in some of the distributions and not always in
Introduction: I’m reposing this classic from 2011 . . . Peter Bergman pointed me to this discussion from Cyrus of a presentation by Guido Imbens on design of randomized experiments. Cyrus writes: The standard analysis that Imbens proposes includes (1) a Fisher-type permutation test of the sharp null hypothesis–what Imbens referred to as “testing”–along with a (2) Neyman-type point estimate of the sample average treatment effect and confidence interval–what Imbens referred to as “estimation.” . . . Imbens claimed that testing and estimation are separate enterprises with separate goals and that the two should not be confused. I [Cyrus] took it as a warning against proposals that use “inverted” tests in order to produce point estimates and confidence intervals. There is no reason that such confidence intervals will have accurate coverage except under rather dire assumptions, meaning that they are not “confidence intervals” in the way that we usually think of them. I agree completely. T
4 0.19645916 738 andrew gelman stats-2011-05-30-Works well versus well understood
Introduction: John Cook discusses the John Tukey quote, “The test of a good procedure is how well it works, not how well it is understood.” Cook writes: At some level, it’s hard to argue against this. Statistical procedures operate on empirical data, so it makes sense that the procedures themselves be evaluated empirically. But I [Cook] question whether we really know that a statistical procedure works well if it isn’t well understood. Specifically, I’m skeptical of complex statistical methods whose only credentials are a handful of simulations. “We don’t have any theoretical results, buy hey, it works well in practice. Just look at the simulations.” Every method works well on the scenarios its author publishes, almost by definition. If the method didn’t handle a scenario well, the author would publish a different scenario. I agree with Cook but would give a slightly different emphasis. I’d say that a lot of methods can work when they are done well. See the second meta-principle liste
Introduction: Peter Bergman points me to this discussion from Cyrus of a presentation by Guido Imbens on design of randomized experiments. Cyrus writes: The standard analysis that Imbens proposes includes (1) a Fisher-type permutation test of the sharp null hypothesis–what Imbens referred to as “testing”–along with a (2) Neyman-type point estimate of the sample average treatment effect and confidence interval–what Imbens referred to as “estimation.” . . . Imbens claimed that testing and estimation are separate enterprises with separate goals and that the two should not be confused. I [Cyrus] took it as a warning against proposals that use “inverted” tests in order to produce point estimates and confidence intervals. There is no reason that such confidence intervals will have accurate coverage except under rather dire assumptions, meaning that they are not “confidence intervals” in the way that we usually think of them. I agree completely. This is something I’ve been saying for a long
6 0.1947953 2258 andrew gelman stats-2014-03-21-Random matrices in the news
7 0.1696008 2128 andrew gelman stats-2013-12-09-How to model distributions that have outliers in one direction
8 0.16615921 2201 andrew gelman stats-2014-02-06-Bootstrap averaging: Examples where it works and where it doesn’t work
9 0.16591814 773 andrew gelman stats-2011-06-18-Should we always be using the t and robit instead of the normal and logit?
10 0.1633507 781 andrew gelman stats-2011-06-28-The holes in my philosophy of Bayesian data analysis
11 0.15518583 1247 andrew gelman stats-2012-04-05-More philosophy of Bayes
14 0.14691548 2072 andrew gelman stats-2013-10-21-The future (and past) of statistical sciences
16 0.1439178 480 andrew gelman stats-2010-12-21-Instead of “confidence interval,” let’s say “uncertainty interval”
19 0.14182201 972 andrew gelman stats-2011-10-25-How do you interpret standard errors from a regression fit to the entire population?
20 0.14156926 669 andrew gelman stats-2011-04-19-The mysterious Gamma (1.4, 0.4)
topicId topicWeight
[(0, 0.264), (1, 0.182), (2, 0.065), (3, 0.009), (4, 0.018), (5, -0.007), (6, 0.021), (7, 0.033), (8, 0.076), (9, -0.053), (10, -0.019), (11, 0.011), (12, -0.074), (13, -0.019), (14, -0.099), (15, -0.017), (16, 0.01), (17, -0.025), (18, 0.04), (19, -0.057), (20, 0.085), (21, -0.045), (22, 0.012), (23, -0.034), (24, 0.094), (25, 0.018), (26, -0.079), (27, -0.014), (28, 0.052), (29, 0.1), (30, -0.033), (31, -0.043), (32, -0.017), (33, 0.038), (34, 0.046), (35, 0.041), (36, -0.057), (37, 0.068), (38, -0.052), (39, -0.008), (40, 0.095), (41, -0.052), (42, 0.02), (43, -0.041), (44, 0.037), (45, 0.003), (46, 0.026), (47, 0.021), (48, 0.027), (49, 0.035)]
simIndex simValue blogId blogTitle
same-blog 1 0.96141535 1527 andrew gelman stats-2012-10-10-Another reason why you can get good inferences from a bad model
Introduction: John Cook considers how people justify probability distribution assumptions: Sometimes distribution assumptions are not justified. Sometimes distributions can be derived from fundamental principles [or] . . . on theoretical grounds. For example, large samples and the central limit theorem together may justify assuming that something is normally distributed. Often the choice of distribution is somewhat arbitrary, chosen by intuition or for convenience, and then empirically shown to work well enough. Sometimes a distribution can be a bad fit and still work well, depending on what you’re asking of it. Cook continues: The last point is particularly interesting. It’s not hard to imagine that a poor fit would produce poor results. It’s surprising when a poor fit produces good results. And then he gives an example of an effective but inaccurate model used to model survival times in a clinical trial. Cook explains: The [poorly-fitting] method works well because of the q
2 0.80809009 2176 andrew gelman stats-2014-01-19-Transformations for non-normal data
Introduction: Steve Peterson writes: I recently submitted a proposal on applying a Bayesian analysis to gender comparisons on motivational constructs. I had an idea on how to improve the model I used and was hoping you could give me some feedback. The data come from a survey based on 5-point Likert scales. Different constructs are measured for each student as scores derived from averaging a student’s responses on particular subsets of survey questions. (I suppose it is not uncontroversial to treat these scores as interval measures and would be interested to hear if you have any objections.) I am comparing genders on each construct. Researchers typically use t-tests to do so. To use a Bayesian approach I applied the programs written in R and JAGS by John Kruschke for estimating the difference of means: http://www.indiana.edu/~kruschke/BEST/ An issue in that analysis is that the distributions of student scores are not normal. There was skewness in some of the distributions and not always in
3 0.8003791 1881 andrew gelman stats-2013-06-03-Boot
Introduction: Joshua Hartshorne writes: I ran several large-N experiments (separate participants) and looked at performance against age. What we want to do is compare age-of-peak-performance across the different tasks (again, different participants). We bootstrapped age-of-peak-performance. On each iteration, we sampled (with replacement) the X scores at each age, where X=num of participants at that age, and recorded the age at which performance peaked on that task. We then recorded the age at which performance was at peak and repeated. Once we had distributions of age-of-peak-performance, we used the means and SDs to calculate t-statistics to compare the results across different tasks. For graphical presentation, we used medians, interquartile ranges, and 95% confidence intervals (based on the distributions: the range within which 75% and 95% of the bootstrapped peaks appeared). While a number of people we consulted with thought this made a lot of sense, one reviewer of the paper insist
4 0.79818171 996 andrew gelman stats-2011-11-07-Chi-square FAIL when many cells have small expected values
Introduction: William Perkins, Mark Tygert, and Rachel Ward write : If a discrete probability distribution in a model being tested for goodness-of-fit is not close to uniform, then forming the Pearson χ2 statistic can involve division by nearly zero. This often leads to serious trouble in practice — even in the absence of round-off errors . . . The problem is not merely that the chi-squared statistic doesn’t have the advertised chi-squared distribution —a reference distribution can always be computed via simulation, either using the posterior predictive distribution or by conditioning on a point estimate of the cell expectations and then making a degrees-of-freedom sort of adjustment. Rather, the problem is that, when there are lots of cells with near-zero expectation, the chi-squared test is mostly noise. And this is not merely a theoretical problem. It comes up in real examples. Here’s one, taken from the classic 1992 genetics paper of Guo and Thomspson: And here are the e
5 0.77860135 1363 andrew gelman stats-2012-06-03-Question about predictive checks
Introduction: Klaas Metselaar writes: I [Metselaar] am currently involved in a discussion about the use of the notion “predictive” as used in “posterior predictive check”. I would argue that the notion “predictive” should be reserved for posterior checks using information not used in the determination of the posterior. I quote from the discussion: “However, the predictive uncertainty in a Bayesian calculation requires sampling from all the random variables, and this includes both the model parameters and the residual error”. My [Metselaar's] comment: This may be exactly the point I am worried about: shouldn’t the predictive uncertainty be defined as sampling from the posterior parameter distribution + residual error + sampling from the prediction error distribution? Residual error reduces to measurement error in the case of a model which is perfect for the sample of experiments. Measurement error could be reduced to almost zero by ideal and perfect measurement instruments. I would h
6 0.77469337 2128 andrew gelman stats-2013-12-09-How to model distributions that have outliers in one direction
8 0.7720713 1460 andrew gelman stats-2012-08-16-“Real data can be a pain”
9 0.76803041 1509 andrew gelman stats-2012-09-24-Analyzing photon counts
12 0.75032747 2201 andrew gelman stats-2014-02-06-Bootstrap averaging: Examples where it works and where it doesn’t work
13 0.74275517 1346 andrew gelman stats-2012-05-27-Average predictive comparisons when changing a pair of variables
15 0.72951138 2029 andrew gelman stats-2013-09-18-Understanding posterior p-values
16 0.71471435 773 andrew gelman stats-2011-06-18-Should we always be using the t and robit instead of the normal and logit?
18 0.70245218 2311 andrew gelman stats-2014-04-29-Bayesian Uncertainty Quantification for Differential Equations!
19 0.70216954 1401 andrew gelman stats-2012-06-30-David Hogg on statistics
20 0.69956213 1476 andrew gelman stats-2012-08-30-Stan is fast
topicId topicWeight
[(2, 0.012), (5, 0.021), (6, 0.032), (9, 0.011), (15, 0.039), (16, 0.067), (21, 0.062), (24, 0.127), (29, 0.046), (65, 0.024), (76, 0.01), (84, 0.026), (93, 0.016), (99, 0.388)]
simIndex simValue blogId blogTitle
same-blog 1 0.98951733 1527 andrew gelman stats-2012-10-10-Another reason why you can get good inferences from a bad model
Introduction: John Cook considers how people justify probability distribution assumptions: Sometimes distribution assumptions are not justified. Sometimes distributions can be derived from fundamental principles [or] . . . on theoretical grounds. For example, large samples and the central limit theorem together may justify assuming that something is normally distributed. Often the choice of distribution is somewhat arbitrary, chosen by intuition or for convenience, and then empirically shown to work well enough. Sometimes a distribution can be a bad fit and still work well, depending on what you’re asking of it. Cook continues: The last point is particularly interesting. It’s not hard to imagine that a poor fit would produce poor results. It’s surprising when a poor fit produces good results. And then he gives an example of an effective but inaccurate model used to model survival times in a clinical trial. Cook explains: The [poorly-fitting] method works well because of the q
Introduction: We had a discussion the other day of a paper, “The Economic Effects of Climate Change,” by economist Richard Tol. The paper came to my attention after I saw a notice from Adam Marcus that it was recently revised because of data errors. But after looking at the paper more carefully, I see a bunch of other problems that, to me, make the whole analysis close to useless as it stands. I think this is worth discussing because the paper has been somewhat influential (so far cited 328 times, according to Google Scholar) and has even been cited in the popular press as evidence that “Climate change has done more good than harm so far and is likely to continue doing so for most of this century . . . There are many likely effects of climate change: positive and negative, economic and ecological, humanitarian and financial. And if you aggregate them all, the overall effect is positive today — and likely to stay positive until around 2080. That was the conclusion of Professor Richard Tol
Introduction: This material should be familiar to many of you but could be helpful to newcomers. Pearl writes: ALL causal conclusions in nonexperimental settings must be based on untested, judgmental assumptions that investigators are prepared to defend on scientific grounds. . . . To understand what the world should be like for a given procedure to work is of no lesser scientific value than seeking evidence for how the world works . . . Assumptions are self-destructive in their honesty. The more explicit the assumption, the more criticism it invites . . . causal diagrams invite the harshest criticism because they make assumptions more explicit and more transparent than other representation schemes. As regular readers know (for example, search this blog for “Pearl”), I have not got much out of the causal-diagrams approach myself, but in general I think that when there are multiple, mathematically equivalent methods of getting the same answer, we tend to go with the framework we are used
4 0.98095143 2236 andrew gelman stats-2014-03-07-Selection bias in the reporting of shaky research
Introduction: I’ll reorder this week’s posts a bit in order to continue on a topic that came up yesterday. A couple days ago a reporter wrote to me asking what I thought of this paper on Money, Status, and the Ovulatory Cycle. I responded: Given the quality of the earlier paper by these researchers, I’m not inclined to believe anything these people write. But, to be specific, I can point out some things: - The authors define low fertility as days 8-14. Oddly enough, these authors in their earlier paper used days 7-14. But according to womenshealth.gov, the most fertile days are between days 10 and 17. The choice of these days affects their analysis, and it is not a good sign that they use different days in different papers. (see more on this point in sections 2.3 and 3.1 of this paper: http://www.stat.columbia.edu/~gelman/research/unpublished/p_hacking.pdf) - They perform a lot of different analyses, and many others could be performed. For example, “Study 1 indicates that ovul
5 0.97994232 1539 andrew gelman stats-2012-10-18-IRB nightmares
Introduction: Andrew Perrin nails it : Twice a year, like clockwork, the ethics cops at the IRB [institutional review board, the group on campus that has to approve research involving human subjects] take a break from deciding whether or not radioactive isotopes can be administered to prison populations to cure restless-leg syndrome to dream up some fancy new way in which participating in an automated telephone poll might cause harm. Perrin adds: The list of exemptions to IRB review is too short and, more importantly, contains no guiding principle as to what makes exempt. . . . [and] Even exemptions require approval by the IRB. He also voices a thought I’ve had many times, which is that there are all sorts of things you or I or anyone else can do on the street (for example, go up to people and ask them personal questions, drop objects and see if people pick them up, stage fights with our friends to see the reactions of bystanders, etc etc etc) but for which we have to go through an IRB
7 0.97951126 2180 andrew gelman stats-2014-01-21-Everything I need to know about Bayesian statistics, I learned in eight schools.
8 0.97938126 1832 andrew gelman stats-2013-04-29-The blogroll
9 0.97902447 2091 andrew gelman stats-2013-11-06-“Marginally significant”
10 0.97900039 1491 andrew gelman stats-2012-09-10-Update on Levitt paper on child car seats
11 0.97880912 1742 andrew gelman stats-2013-02-27-What is “explanation”?
12 0.97815895 868 andrew gelman stats-2011-08-24-Blogs vs. real journalism
13 0.9781009 563 andrew gelman stats-2011-02-07-Evaluating predictions of political events
15 0.97776443 785 andrew gelman stats-2011-07-02-Experimental reasoning in social science
16 0.97771382 2245 andrew gelman stats-2014-03-12-More on publishing in journals
17 0.97765231 1341 andrew gelman stats-2012-05-24-Question 14 of my final exam for Design and Analysis of Sample Surveys
18 0.97754872 1634 andrew gelman stats-2012-12-21-Two reviews of Nate Silver’s new book, from Kaiser Fung and Cathy O’Neil
20 0.97746789 2368 andrew gelman stats-2014-06-11-Bayes in the research conversation