andrew_gelman_stats andrew_gelman_stats-2012 andrew_gelman_stats-2012-1310 knowledge-graph by maker-knowledge-mining
Source: html
Introduction: This time from Bernard Fraga and Eitan Hersh. Once you think about it, it’s hard to imagine any nonzero treatment effects that don’t vary. I’m glad to see this area of research becoming more prominent. ( Here ‘s a discussion of another political science example, also of voter turnout, from a few years ago, from Avi Feller and Chris Holmes.) Some of my fragmentary work on varying treatment effects is here (Treatment Effects in Before-After Data) and here (Estimating Incumbency Advantage and Its Variation, as an Example of a Before–After Study).
sentIndex sentText sentNum sentScore
1 Once you think about it, it’s hard to imagine any nonzero treatment effects that don’t vary. [sent-2, score-1.213]
2 I’m glad to see this area of research becoming more prominent. [sent-3, score-0.634]
3 ( Here ‘s a discussion of another political science example, also of voter turnout, from a few years ago, from Avi Feller and Chris Holmes. [sent-4, score-0.641]
4 ) Some of my fragmentary work on varying treatment effects is here (Treatment Effects in Before-After Data) and here (Estimating Incumbency Advantage and Its Variation, as an Example of a Before–After Study). [sent-5, score-0.945]
wordName wordTfidf (topN-words)
[('treatment', 0.411), ('effects', 0.301), ('avi', 0.291), ('bernard', 0.26), ('feller', 0.245), ('nonzero', 0.245), ('turnout', 0.231), ('incumbency', 0.225), ('voter', 0.208), ('becoming', 0.195), ('glad', 0.191), ('varying', 0.178), ('advantage', 0.161), ('chris', 0.161), ('estimating', 0.147), ('variation', 0.145), ('area', 0.141), ('imagine', 0.128), ('example', 0.107), ('hard', 0.093), ('ago', 0.092), ('study', 0.09), ('political', 0.087), ('discussion', 0.079), ('science', 0.078), ('another', 0.077), ('years', 0.07), ('research', 0.064), ('work', 0.055), ('time', 0.053), ('data', 0.044), ('see', 0.043), ('also', 0.042), ('think', 0.035)]
simIndex simValue blogId blogTitle
same-blog 1 0.99999994 1310 andrew gelman stats-2012-05-09-Varying treatment effects, again
Introduction: This time from Bernard Fraga and Eitan Hersh. Once you think about it, it’s hard to imagine any nonzero treatment effects that don’t vary. I’m glad to see this area of research becoming more prominent. ( Here ‘s a discussion of another political science example, also of voter turnout, from a few years ago, from Avi Feller and Chris Holmes.) Some of my fragmentary work on varying treatment effects is here (Treatment Effects in Before-After Data) and here (Estimating Incumbency Advantage and Its Variation, as an Example of a Before–After Study).
Introduction: Avi sent along this old paper from Bryk and Raudenbush, who write: The presence of heterogeneity of variance across groups indicates that the standard statistical model for treatment effects no longer applies. Specifically, the assumption that treatments add a constant to each subject’s development fails. An alternative model is required to represent how treatment effects are distributed across individuals. We develop in this article a simple statistical model to demonstrate the link between heterogeneity of variance and random treatment effects. Next, we illustrate with results from two previously published studies how a failure to recognize the substantive importance of heterogeneity of variance obscured significant results present in these data. The article concludes with a review and synthesis of techniques for modeling variances. Although these methods have been well established in the statistical literature, they are not widely known by social and behavioral scientists. T
3 0.19248335 2 andrew gelman stats-2010-04-23-Modeling heterogenous treatment effects
Introduction: Don Green and Holger Kern write on one of my favorite topics , treatment interactions (see also here ): We [Green and Kern] present a methodology that largely automates the search for systematic treatment effect heterogeneity in large-scale experiments. We introduce a nonparametric estimator developed in statistical learning, Bayesian Additive Regression Trees (BART), to model treatment effects that vary as a function of covariates. BART has several advantages over commonly employed parametric modeling strategies, in particular its ability to automatically detect and model relevant treatment-covariate interactions in a flexible manner. To increase the reliability and credibility of the resulting conditional treatment effect estimates, we suggest the use of a split sample design. The data are randomly divided into two equally-sized parts, with the first part used to explore treatment effect heterogeneity and the second part used to confirm the results. This approach permits a re
4 0.17056434 1644 andrew gelman stats-2012-12-30-Fixed effects, followed by Bayes shrinkage?
Introduction: Stuart Buck writes: I have a question about fixed effects vs. random effects . Amongst economists who study teacher value-added, it has become common to see people saying that they estimated teacher fixed effects (via least squares dummy variables, so that there is a parameter for each teacher), but that they then applied empirical Bayes shrinkage so that the teacher effects are brought closer to the mean. (See this paper by Jacob and Lefgren, for example.) Can that really be what they are doing? Why wouldn’t they just run random (modeled) effects in the first place? I feel like there’s something I’m missing. My reply: I don’t know the full story here, but I’m thinking there are two goals, first to get an unbiased estimate of an overall treatment effect (and there the econometricians prefer so-called fixed effects; I disagree with them on this but I know where they’re coming from) and second to estimate individual teacher effects (and there it makes sense to use so-called
5 0.16767716 518 andrew gelman stats-2011-01-15-Regression discontinuity designs: looking for the keys under the lamppost?
Introduction: Jas sends along this paper (with Devin Caughey), entitled Regression-Discontinuity Designs and Popular Elections: Implications of Pro-Incumbent Bias in Close U.S. House Races, and writes: The paper shows that regression discontinuity does not work for US House elections. Close House elections are anything but random. It isn’t election recounts or something like that (we collect recount data to show that it isn’t). We have collected much new data to try to hunt down what is going on (e.g., campaign finance data, CQ pre-election forecasts, correct many errors in the Lee dataset). The substantive implications are interesting. We also have a section that compares in details Gelman and King versus the Lee estimand and estimator. I had a few comments: David Lee is not estimating the effect of incumbency; he’s estimating the effect of the incumbent party, which is a completely different thing. The regression discontinuity design is completely inappropriate for estimating the
6 0.13021904 1201 andrew gelman stats-2012-03-07-Inference = data + model
7 0.12903504 1150 andrew gelman stats-2012-02-02-The inevitable problems with statistical significance and 95% intervals
8 0.12006563 1675 andrew gelman stats-2013-01-15-“10 Things You Need to Know About Causal Effects”
9 0.11930387 1571 andrew gelman stats-2012-11-09-The anti-Bayesian moment and its passing
10 0.11904499 898 andrew gelman stats-2011-09-10-Fourteen magic words: an update
11 0.11901044 1869 andrew gelman stats-2013-05-24-In which I side with Neyman over Fisher
12 0.1187346 86 andrew gelman stats-2010-06-14-“Too much data”?
13 0.11176059 1194 andrew gelman stats-2012-03-04-Multilevel modeling even when you’re not interested in predictions for new groups
14 0.10758355 936 andrew gelman stats-2011-10-02-Covariate Adjustment in RCT - Model Overfitting in Multilevel Regression
16 0.10421233 678 andrew gelman stats-2011-04-25-Democrats do better among the most and least educated groups
17 0.10353823 388 andrew gelman stats-2010-11-01-The placebo effect in pharma
18 0.10261284 1267 andrew gelman stats-2012-04-17-Hierarchical-multilevel modeling with “big data”
19 0.102281 485 andrew gelman stats-2010-12-25-Unlogging
20 0.10191616 433 andrew gelman stats-2010-11-27-One way that psychology research is different than medical research
topicId topicWeight
[(0, 0.121), (1, 0.003), (2, 0.049), (3, -0.107), (4, -0.009), (5, -0.011), (6, -0.049), (7, 0.001), (8, 0.032), (9, 0.038), (10, -0.05), (11, 0.017), (12, 0.091), (13, -0.053), (14, 0.068), (15, 0.015), (16, -0.055), (17, 0.015), (18, -0.048), (19, 0.08), (20, -0.052), (21, -0.001), (22, -0.034), (23, -0.015), (24, -0.014), (25, 0.015), (26, -0.105), (27, 0.061), (28, -0.072), (29, 0.023), (30, -0.065), (31, -0.006), (32, -0.094), (33, -0.041), (34, -0.011), (35, -0.028), (36, -0.04), (37, -0.023), (38, 0.005), (39, -0.013), (40, 0.011), (41, 0.074), (42, 0.006), (43, -0.035), (44, -0.003), (45, 0.005), (46, 0.02), (47, -0.057), (48, -0.009), (49, 0.059)]
simIndex simValue blogId blogTitle
same-blog 1 0.97299063 1310 andrew gelman stats-2012-05-09-Varying treatment effects, again
Introduction: This time from Bernard Fraga and Eitan Hersh. Once you think about it, it’s hard to imagine any nonzero treatment effects that don’t vary. I’m glad to see this area of research becoming more prominent. ( Here ‘s a discussion of another political science example, also of voter turnout, from a few years ago, from Avi Feller and Chris Holmes.) Some of my fragmentary work on varying treatment effects is here (Treatment Effects in Before-After Data) and here (Estimating Incumbency Advantage and Its Variation, as an Example of a Before–After Study).
2 0.81800544 433 andrew gelman stats-2010-11-27-One way that psychology research is different than medical research
Introduction: Medical researchers care about main effects, psychologists care about interactions. In psychology, the main effects are typically obvious, and it’s only the interactions that are worth studying.
3 0.79392469 1644 andrew gelman stats-2012-12-30-Fixed effects, followed by Bayes shrinkage?
Introduction: Stuart Buck writes: I have a question about fixed effects vs. random effects . Amongst economists who study teacher value-added, it has become common to see people saying that they estimated teacher fixed effects (via least squares dummy variables, so that there is a parameter for each teacher), but that they then applied empirical Bayes shrinkage so that the teacher effects are brought closer to the mean. (See this paper by Jacob and Lefgren, for example.) Can that really be what they are doing? Why wouldn’t they just run random (modeled) effects in the first place? I feel like there’s something I’m missing. My reply: I don’t know the full story here, but I’m thinking there are two goals, first to get an unbiased estimate of an overall treatment effect (and there the econometricians prefer so-called fixed effects; I disagree with them on this but I know where they’re coming from) and second to estimate individual teacher effects (and there it makes sense to use so-called
4 0.79375881 1744 andrew gelman stats-2013-03-01-Why big effects are more important than small effects
Introduction: The title of this post is silly but I have an important point to make, regarding an implicit model which I think many people assume even though it does not really make sense. Following a link from Sanjay Srivastava, I came across a post from David Funder saying that it’s useful to talk about the sizes of effects (I actually prefer the term “comparisons” so as to avoid the causal baggage) rather than just their signs. I agree , and I wanted to elaborate a bit on a point that comes up in Funder’s discussion. He quotes an (unnamed) prominent social psychologist as writing: The key to our research . . . [is not] to accurately estimate effect size. If I were testing an advertisement for a marketing research firm and wanted to be sure that the cost of the ad would produce enough sales to make it worthwhile, effect size would be crucial. But when I am testing a theory about whether, say, positive mood reduces information processing in comparison with negative mood, I am worried abou
5 0.78776366 2 andrew gelman stats-2010-04-23-Modeling heterogenous treatment effects
Introduction: Don Green and Holger Kern write on one of my favorite topics , treatment interactions (see also here ): We [Green and Kern] present a methodology that largely automates the search for systematic treatment effect heterogeneity in large-scale experiments. We introduce a nonparametric estimator developed in statistical learning, Bayesian Additive Regression Trees (BART), to model treatment effects that vary as a function of covariates. BART has several advantages over commonly employed parametric modeling strategies, in particular its ability to automatically detect and model relevant treatment-covariate interactions in a flexible manner. To increase the reliability and credibility of the resulting conditional treatment effect estimates, we suggest the use of a split sample design. The data are randomly divided into two equally-sized parts, with the first part used to explore treatment effect heterogeneity and the second part used to confirm the results. This approach permits a re
8 0.73959404 1400 andrew gelman stats-2012-06-29-Decline Effect in Linguistics?
9 0.72916603 1186 andrew gelman stats-2012-02-27-Confusion from illusory precision
10 0.70584589 7 andrew gelman stats-2010-04-27-Should Mister P be allowed-encouraged to reside in counter-factual populations?
11 0.68494719 1241 andrew gelman stats-2012-04-02-Fixed effects and identification
12 0.6808449 963 andrew gelman stats-2011-10-18-Question on Type M errors
13 0.66579521 803 andrew gelman stats-2011-07-14-Subtleties with measurement-error models for the evaluation of wacky claims
14 0.65310246 2185 andrew gelman stats-2014-01-25-Xihong Lin on sparsity and density
15 0.65262198 797 andrew gelman stats-2011-07-11-How do we evaluate a new and wacky claim?
16 0.64369434 1513 andrew gelman stats-2012-09-27-Estimating seasonality with a data set that’s just 52 weeks long
17 0.63887417 64 andrew gelman stats-2010-06-03-Estimates of war deaths: Darfur edition
18 0.63533282 2097 andrew gelman stats-2013-11-11-Why ask why? Forward causal inference and reverse causal questions
19 0.63076758 1929 andrew gelman stats-2013-07-07-Stereotype threat!
20 0.62433171 2243 andrew gelman stats-2014-03-11-The myth of the myth of the myth of the hot hand
topicId topicWeight
[(16, 0.034), (24, 0.302), (63, 0.066), (69, 0.19), (99, 0.248)]
simIndex simValue blogId blogTitle
same-blog 1 0.93919373 1310 andrew gelman stats-2012-05-09-Varying treatment effects, again
Introduction: This time from Bernard Fraga and Eitan Hersh. Once you think about it, it’s hard to imagine any nonzero treatment effects that don’t vary. I’m glad to see this area of research becoming more prominent. ( Here ‘s a discussion of another political science example, also of voter turnout, from a few years ago, from Avi Feller and Chris Holmes.) Some of my fragmentary work on varying treatment effects is here (Treatment Effects in Before-After Data) and here (Estimating Incumbency Advantage and Its Variation, as an Example of a Before–After Study).
2 0.90715325 1376 andrew gelman stats-2012-06-12-Simple graph WIN: the example of birthday frequencies
Introduction: From Chris Mulligan: The data come from the Center for Disease Control and cover the years 1969-1988. Chris also gives instructions for how to download the data and plot them in R from scratch (in 30 lines of R code)! And now, the background A few months ago I heard about a study reporting that, during a recent eleven-year period, more babies were born on Valentine’s Day and fewer on Halloween compared to neighboring days: I wrote , What I’d really like to see is a graph with all 366 days of the year. It would be easy enough to make. That way we could put the Valentine’s and Halloween data in the context of other possible patterns. While they’re at it, they could also graph births by day of the week and show Thanksgiving, Easter, and other holidays that don’t have fixed dates. It’s so frustrating when people only show part of the story. I was pointed to some tables: and a graph from Matt Stiles: The heatmap is cute but I wanted to se
3 0.89261603 1787 andrew gelman stats-2013-04-04-Wanna be the next Tyler Cowen? It’s not as easy as you might think!
Introduction: Someone told me he ran into someone who said his goal was to be Tyler Cowen. OK, fine, it’s a worthy goal, but I don’t think it’s so easy .
4 0.89000666 898 andrew gelman stats-2011-09-10-Fourteen magic words: an update
Introduction: In the discussion of the fourteen magic words that can increase voter turnout by over 10 percentage points , questions were raised about the methods used to estimate the experimental effects. I sent these on to Chris Bryan, the author of the study, and he gave the following response: We’re happy to address the questions that have come up. It’s always noteworthy when a precise psychological manipulation like this one generates a large effect on a meaningful outcome. Such findings illustrate the power of the underlying psychological process. I’ve provided the contingency tables for the two turnout experiments below. As indicated in the paper, the data are analyzed using logistic regressions. The change in chi-squared statistic represents the significance of the noun vs. verb condition variable in predicting turnout; that is, the change in the model’s significance when the condition variable is added. This is a standard way to analyze dichotomous outcomes. Four outliers were excl
5 0.88850683 278 andrew gelman stats-2010-09-15-Advice that might make sense for individuals but is negative-sum overall
Introduction: There’s a lot of free advice out there. As I wrote a couple years ago, it’s usually presented as advice to individuals, but it’s also interesting to consider the possible total effects if the advice is taken. For example, Nassim Taleb has a webpage that includes a bunch of one-line bits of advice (scroll to item 132 on the linked page). Here’s his final piece of advice: If you dislike someone, leave him alone or eliminate him; don’t attack him verbally. I’m a big Taleb fan (search this blog to see), but this seems like classic negative-sum advice. I can see how it can be a good individual strategy to keep your mouth shut, bide your time, and then sandbag your enemies. But it can’t be good if lots of people are doing this. Verbal attacks are great, as long as there’s a chance to respond. I’ve been in environments where people follow Taleb’s advice, saying nothing and occasionally trying to “eliminate” people, and it’s not pretty. I much prefer for people to be open
6 0.88812613 743 andrew gelman stats-2011-06-03-An argument that can’t possibly make sense
7 0.88777721 1999 andrew gelman stats-2013-08-27-Bayesian model averaging or fitting a larger model
8 0.88771844 1455 andrew gelman stats-2012-08-12-Probabilistic screening to get an approximate self-weighted sample
9 0.88740242 197 andrew gelman stats-2010-08-10-The last great essayist?
10 0.88637108 1167 andrew gelman stats-2012-02-14-Extra babies on Valentine’s Day, fewer on Halloween?
11 0.88572085 1224 andrew gelman stats-2012-03-21-Teaching velocity and acceleration
13 0.88464785 2143 andrew gelman stats-2013-12-22-The kluges of today are the textbook solutions of tomorrow.
14 0.8843708 482 andrew gelman stats-2010-12-23-Capitalism as a form of voluntarism
16 0.88252544 1092 andrew gelman stats-2011-12-29-More by Berger and me on weakly informative priors
17 0.88220978 414 andrew gelman stats-2010-11-14-“Like a group of teenagers on a bus, they behave in public as if they were in private”
19 0.88170302 938 andrew gelman stats-2011-10-03-Comparing prediction errors
20 0.8813554 1072 andrew gelman stats-2011-12-19-“The difference between . . .”: It’s not just p=.05 vs. p=.06