andrew_gelman_stats andrew_gelman_stats-2010 andrew_gelman_stats-2010-310 knowledge-graph by maker-knowledge-mining
Source: html
Introduction: If an estimate is statistically significant, it’s probably an overestimate of the magnitude of your effect. P.S. I think youall know what I mean here. But could someone rephrase it in a more pithy manner? I’d like to include it in our statistical lexicon.
sentIndex sentText sentNum sentScore
1 If an estimate is statistically significant, it’s probably an overestimate of the magnitude of your effect. [sent-1, score-1.106]
2 But could someone rephrase it in a more pithy manner? [sent-5, score-0.617]
3 I’d like to include it in our statistical lexicon. [sent-6, score-0.318]
wordName wordTfidf (topN-words)
[('youall', 0.438), ('rephrase', 0.41), ('lexicon', 0.39), ('overestimate', 0.331), ('manner', 0.304), ('magnitude', 0.269), ('statistically', 0.201), ('significant', 0.183), ('include', 0.172), ('probably', 0.154), ('estimate', 0.151), ('mean', 0.138), ('someone', 0.137), ('statistical', 0.093), ('know', 0.078), ('could', 0.07), ('like', 0.053), ('think', 0.053)]
simIndex simValue blogId blogTitle
same-blog 1 1.0 310 andrew gelman stats-2010-10-02-The winner’s curse
Introduction: If an estimate is statistically significant, it’s probably an overestimate of the magnitude of your effect. P.S. I think youall know what I mean here. But could someone rephrase it in a more pithy manner? I’d like to include it in our statistical lexicon.
2 0.24066485 899 andrew gelman stats-2011-09-10-The statistical significance filter
Introduction: I’ve talked about this a bit but it’s never had its own blog entry (until now). Statistically significant findings tend to overestimate the magnitude of effects. This holds in general (because E(|x|) > |E(x)|) but even more so if you restrict to statistically significant results. Here’s an example. Suppose a true effect of theta is unbiasedly estimated by y ~ N (theta, 1). Further suppose that we will only consider statistically significant results, that is, cases in which |y| > 2. The estimate “|y| conditional on |y|>2″ is clearly an overestimate of |theta|. First off, if |theta|<2, the estimate |y| conditional on statistical significance is not only too high in expectation, it's always too high. This is a problem, given that |theta| is in reality probably is less than 2. (The low-hangning fruit have already been picked, remember?) But even if |theta|>2, the estimate |y| conditional on statistical significance will still be too high in expectation. For a discussion o
3 0.16159187 1072 andrew gelman stats-2011-12-19-“The difference between . . .”: It’s not just p=.05 vs. p=.06
Introduction: The title of this post by Sanjay Srivastava illustrates an annoying misconception that’s crept into the (otherwise delightful) recent publicity related to my article with Hal Stern, he difference between “significant” and “not significant” is not itself statistically significant. When people bring this up, they keep referring to the difference between p=0.05 and p=0.06, making the familiar (and correct) point about the arbitrariness of the conventional p-value threshold of 0.05. And, sure, I agree with this, but everybody knows that already. The point Hal and I were making was that even apparently large differences in p-values are not statistically significant. For example, if you have one study with z=2.5 (almost significant at the 1% level!) and another with z=1 (not statistically significant at all, only 1 se from zero!), then their difference has a z of about 1 (again, not statistically significant at all). So it’s not just a comparison of 0.05 vs. 0.06, even a differenc
Introduction: Type S error: When your estimate is the wrong sign, compared to the true value of the parameter Type M error: When the magnitude of your estimate is far off, compared to the true value of the parameter More here.
5 0.11024877 466 andrew gelman stats-2010-12-13-“The truth wears off: Is there something wrong with the scientific method?”
Introduction: Gur Huberman asks what I think of this magazine article by Johah Lehrer (see also here ). My reply is that it reminds me a bit of what I wrote here . Or see here for the quick powerpoint version: The short story is that if you screen for statistical significance when estimating small effects, you will necessarily overestimate the magnitudes of effects, sometimes by a huge amount. I know that Dave Krantz has thought about this issue for awhile; it came up when Francis Tuerlinckx and I wrote our paper on Type S errors, ten years ago. My current thinking is that most (almost all?) research studies of the sort described by Lehrer should be accompanied by retrospective power analyses, or informative Bayesian inferences. Either of these approaches–whether classical or Bayesian, the key is that they incorporate real prior information, just as is done in a classical prospective power analysis–would, I think, moderate the tendency to overestimate the magnitude of effects. In answ
6 0.10716295 255 andrew gelman stats-2010-09-04-How does multilevel modeling affect the estimate of the grand mean?
7 0.10179454 1866 andrew gelman stats-2013-05-21-Recently in the sister blog
8 0.096446976 146 andrew gelman stats-2010-07-14-The statistics and the science
9 0.090974644 689 andrew gelman stats-2011-05-01-Is that what she said?
10 0.082209595 920 andrew gelman stats-2011-09-22-Top 10 blog obsessions
11 0.078987151 2243 andrew gelman stats-2014-03-11-The myth of the myth of the myth of the hot hand
12 0.078011714 401 andrew gelman stats-2010-11-08-Silly old chi-square!
13 0.076489121 2178 andrew gelman stats-2014-01-20-Mailing List Degree-of-Difficulty Difficulty
14 0.076181367 803 andrew gelman stats-2011-07-14-Subtleties with measurement-error models for the evaluation of wacky claims
15 0.075343214 511 andrew gelman stats-2011-01-11-One more time on that ESP study: The problem of overestimates and the shrinkage solution
18 0.067009203 63 andrew gelman stats-2010-06-02-The problem of overestimation of group-level variance parameters
20 0.065992087 1150 andrew gelman stats-2012-02-02-The inevitable problems with statistical significance and 95% intervals
topicId topicWeight
[(0, 0.074), (1, 0.013), (2, 0.032), (3, -0.073), (4, 0.015), (5, -0.03), (6, 0.005), (7, 0.01), (8, 0.002), (9, -0.037), (10, -0.023), (11, -0.027), (12, 0.05), (13, -0.023), (14, 0.019), (15, 0.01), (16, -0.042), (17, 0.002), (18, 0.015), (19, 0.013), (20, 0.011), (21, 0.01), (22, 0.038), (23, 0.0), (24, 0.008), (25, 0.003), (26, 0.009), (27, -0.037), (28, -0.04), (29, -0.07), (30, 0.052), (31, 0.049), (32, -0.017), (33, -0.023), (34, 0.03), (35, 0.024), (36, -0.04), (37, -0.04), (38, -0.013), (39, -0.052), (40, -0.001), (41, -0.001), (42, -0.074), (43, 0.054), (44, 0.017), (45, -0.026), (46, -0.012), (47, -0.005), (48, -0.014), (49, -0.023)]
simIndex simValue blogId blogTitle
same-blog 1 0.95878428 310 andrew gelman stats-2010-10-02-The winner’s curse
Introduction: If an estimate is statistically significant, it’s probably an overestimate of the magnitude of your effect. P.S. I think youall know what I mean here. But could someone rephrase it in a more pithy manner? I’d like to include it in our statistical lexicon.
2 0.78703475 1072 andrew gelman stats-2011-12-19-“The difference between . . .”: It’s not just p=.05 vs. p=.06
Introduction: The title of this post by Sanjay Srivastava illustrates an annoying misconception that’s crept into the (otherwise delightful) recent publicity related to my article with Hal Stern, he difference between “significant” and “not significant” is not itself statistically significant. When people bring this up, they keep referring to the difference between p=0.05 and p=0.06, making the familiar (and correct) point about the arbitrariness of the conventional p-value threshold of 0.05. And, sure, I agree with this, but everybody knows that already. The point Hal and I were making was that even apparently large differences in p-values are not statistically significant. For example, if you have one study with z=2.5 (almost significant at the 1% level!) and another with z=1 (not statistically significant at all, only 1 se from zero!), then their difference has a z of about 1 (again, not statistically significant at all). So it’s not just a comparison of 0.05 vs. 0.06, even a differenc
3 0.78102171 899 andrew gelman stats-2011-09-10-The statistical significance filter
Introduction: I’ve talked about this a bit but it’s never had its own blog entry (until now). Statistically significant findings tend to overestimate the magnitude of effects. This holds in general (because E(|x|) > |E(x)|) but even more so if you restrict to statistically significant results. Here’s an example. Suppose a true effect of theta is unbiasedly estimated by y ~ N (theta, 1). Further suppose that we will only consider statistically significant results, that is, cases in which |y| > 2. The estimate “|y| conditional on |y|>2″ is clearly an overestimate of |theta|. First off, if |theta|<2, the estimate |y| conditional on statistical significance is not only too high in expectation, it's always too high. This is a problem, given that |theta| is in reality probably is less than 2. (The low-hangning fruit have already been picked, remember?) But even if |theta|>2, the estimate |y| conditional on statistical significance will still be too high in expectation. For a discussion o
4 0.71496439 146 andrew gelman stats-2010-07-14-The statistics and the science
Introduction: Yesterday I posted a review of a submitted manuscript where I first wrote that I read the paper only shallowly and then followed up with some suggestions on the statistical analysis, recommending that overdispersion be added to a fitted Posson regression and that the table of regression results be supplemented with a graph showing data and fitted lines. A commenter asked why I wrote such an apparently shallow review, and I realized that some of the implications of my review were not as clear as I’d thought. So let me clarify. There is a connection between my general reaction and my statistical comments. My statistical advice here is relevant for (at least) two reasons. First, a Poisson regression without overdispersion will give nearly-uninterpretable standard errors, which means that I have no sense if the results are statistically significant as claimed. Second, with a time series plot and regression table, but no graph showing the estimated treatment effect, it is very dif
5 0.69108438 156 andrew gelman stats-2010-07-20-Burglars are local
Introduction: This makes sense: In the land of fiction, it’s the criminal’s modus operandi – his method of entry, his taste for certain jewellery and so forth – that can be used by detectives to identify his handiwork. The reality according to a new analysis of solved burglaries in the Northamptonshire region of England is that these aspects of criminal behaviour are on their own unreliable as identifying markers, most likely because they are dictated by circumstances rather than the criminal’s taste and style. However, the geographical spread and timing of a burglar’s crimes are distinctive, and could help with police investigations. And, as a bonus, more Tourette’s pride! P.S. On yet another unrelated topic from the same blog, I wonder if the researchers in this study are aware that the difference between “significant” and “not significant” is not itself statistically significant .
7 0.65527964 918 andrew gelman stats-2011-09-21-Avoiding boundary estimates in linear mixed models
8 0.63356578 368 andrew gelman stats-2010-10-25-Is instrumental variables analysis particularly susceptible to Type M errors?
10 0.61847395 803 andrew gelman stats-2011-07-14-Subtleties with measurement-error models for the evaluation of wacky claims
11 0.6115976 466 andrew gelman stats-2010-12-13-“The truth wears off: Is there something wrong with the scientific method?”
12 0.61143774 593 andrew gelman stats-2011-02-27-Heat map
13 0.60708404 1971 andrew gelman stats-2013-08-07-I doubt they cheated
14 0.59968114 1171 andrew gelman stats-2012-02-16-“False-positive psychology”
15 0.59584421 643 andrew gelman stats-2011-04-02-So-called Bayesian hypothesis testing is just as bad as regular hypothesis testing
16 0.58750206 2090 andrew gelman stats-2013-11-05-How much do we trust a new claim that early childhood stimulation raised earnings by 42%?
17 0.58539385 1315 andrew gelman stats-2012-05-12-Question 2 of my final exam for Design and Analysis of Sample Surveys
19 0.58207405 695 andrew gelman stats-2011-05-04-Statistics ethics question
20 0.58084303 963 andrew gelman stats-2011-10-18-Question on Type M errors
topicId topicWeight
[(16, 0.039), (24, 0.213), (45, 0.394), (99, 0.147)]
simIndex simValue blogId blogTitle
1 0.8744747 1407 andrew gelman stats-2012-07-06-Statistical inference and the secret ballot
Introduction: Ring Lardner, Jr.: [In 1936] I was already settled in Southern California, and it may have been that first exercise of the franchise that triggered the FBI surveillance of me that would last for decades. I had assumed, of course, that I was enjoying the vaunted American privilege of the secret ballot. On a wall outside my polling place on Wilshire Boulevard, however, was a compilation of the district’s registered voters: Democrats, a long list of names; Republicans, a somewhat lesser number; and “Declines to State,” one, “Ring W. Lardner, Jr.” The day after the election, alongside those lists were published the results: Roosevelt, so many; Landon, so many; Browder, one.
2 0.83321732 1873 andrew gelman stats-2013-05-28-Escalatingly uncomfortable
Introduction: Aggressive, fizzing nonconformity .
3 0.82293582 999 andrew gelman stats-2011-11-09-I was at a meeting a couple months ago . . .
Introduction: . . . and I decided to amuse myself by writing down all the management-speak words I heard: “grappling” “early prototypes” “technology platform” “building block” “machine learning” “your team” “workspace” “tagging” “data exhaust” “monitoring a particular population” “collective intelligence” “communities of practice” “hackathon” “human resources . . . technologies” Any one or two or three of these phrases might be fine, but put them all together and what you have is a festival of jargon. A hackathon, indeed.
same-blog 4 0.75791442 310 andrew gelman stats-2010-10-02-The winner’s curse
Introduction: If an estimate is statistically significant, it’s probably an overestimate of the magnitude of your effect. P.S. I think youall know what I mean here. But could someone rephrase it in a more pithy manner? I’d like to include it in our statistical lexicon.
5 0.74542105 543 andrew gelman stats-2011-01-28-NYT shills for personal DNA tests
Introduction: Kaiser nails it . The offending article , by John Tierney, somehow ended up in the Science section rather than the Opinion section. As an opinion piece (or, for that matter, a blog), Tierney’s article would be nothing special. But I agree with Kaiser that it doesn’t work as a newspaper article. As Kaiser notes, this story involves a bunch of statistical and empirical claims that are not well resolved by P.R. and rhetoric.
6 0.72988665 791 andrew gelman stats-2011-07-08-Censoring on one end, “outliers” on the other, what can we do with the middle?
7 0.69990933 1325 andrew gelman stats-2012-05-17-More on the difficulty of “preaching what you practice”
8 0.69323194 1089 andrew gelman stats-2011-12-28-Path sampling for models of varying dimension
9 0.69113195 2225 andrew gelman stats-2014-02-26-A good comment on one of my papers
10 0.69074935 612 andrew gelman stats-2011-03-14-Uh-oh
11 0.68274736 206 andrew gelman stats-2010-08-13-Indiemapper makes thematic mapping easy
12 0.67800188 1031 andrew gelman stats-2011-11-27-Richard Stallman and John McCarthy
13 0.65427208 69 andrew gelman stats-2010-06-04-A Wikipedia whitewash
14 0.65328979 1121 andrew gelman stats-2012-01-15-R-squared for multilevel models
15 0.64778382 2189 andrew gelman stats-2014-01-28-History is too important to be left to the history professors
16 0.6328851 1504 andrew gelman stats-2012-09-20-Could someone please lock this guy and Niall Ferguson in a room together?
17 0.63042396 1012 andrew gelman stats-2011-11-16-Blog bribes!
18 0.62809861 1658 andrew gelman stats-2013-01-07-Free advice from an academic writing coach!
19 0.62743032 573 andrew gelman stats-2011-02-14-Hipmunk < Expedia, again
20 0.62408227 673 andrew gelman stats-2011-04-20-Upper-income people still don’t realize they’re upper-income