andrew_gelman_stats andrew_gelman_stats-2011 andrew_gelman_stats-2011-936 knowledge-graph by maker-knowledge-mining
Source: html
Introduction: Makoto Hanita writes: We have been discussing the following two issues amongst ourselves, then with our methodological consultant for several days. However, we have not been able to arrive at a consensus. Consequently, we decided to seek an opinion from nationally known experts. FYI, we sent a similar inquiry to Larry Hedges and David Rogosa . . . 1) We are wondering if a post-hoc covariate adjustment is a good practice in the context of RCTs [randomized clinical trials]. We have a situation where we found a significant baseline difference between the treatment and the control groups in 3 variables. Some of us argue that adding those three variables to the original impact analysis model is a good idea, as that would remove the confound from the impact estimate. Others among us, on the other hand, argue that a post-hoc covariate adjustment should never be done, on the ground that those covariates are correlated with the treatment, which makes the analysis model that of quasi
sentIndex sentText sentNum sentScore
1 Makoto Hanita writes: We have been discussing the following two issues amongst ourselves, then with our methodological consultant for several days. [sent-1, score-0.164]
2 1) We are wondering if a post-hoc covariate adjustment is a good practice in the context of RCTs [randomized clinical trials]. [sent-7, score-0.484]
3 We have a situation where we found a significant baseline difference between the treatment and the control groups in 3 variables. [sent-8, score-0.551]
4 Some of us argue that adding those three variables to the original impact analysis model is a good idea, as that would remove the confound from the impact estimate. [sent-9, score-0.702]
5 Others among us, on the other hand, argue that a post-hoc covariate adjustment should never be done, on the ground that those covariates are correlated with the treatment, which makes the analysis model that of quasi-experimental. [sent-10, score-1.002]
6 For your information, we are all in agreement as to the use of covariates for RCTs that are pre-specified, for the purpose of variance reduction. [sent-11, score-0.368]
7 What we cannot agree on is the use of covariates for the purpose of equating two groups in the context of RCTs. [sent-12, score-0.633]
8 2) Despite our disagreement on #1, we went ahead and tried fitting the model including those 3 additional covariates to our data. [sent-13, score-0.614]
9 Some of us are suspicious of the results of this analysis because of the possible model overfitting. [sent-14, score-0.323]
10 The situation involves fitting the below model to the sample of 14 schools and 800 students. [sent-15, score-0.354]
11 As you can see, there are 5 fixed effects at the school level to be estimated. [sent-16, score-0.156]
12 The analysis was done on multiply-imputed data (5 imputations were made). [sent-17, score-0.175]
13 From what I have read, assessing model overfitting is rather difficult for multilevel models. [sent-18, score-0.307]
14 In our case, we have an additional complexity of multiply-imputed data: PostTest ij = b 0 + b 1 (Treatment j ) + b 2 (School Cov1 j ) + b 3 (School Cov2 j ) + b 4 (School Cov3 j ) + b 5 (PreTest ij ) + b 6 (FRL% j ) + u j + e ij My reply: 1. [sent-19, score-0.957]
15 Ultimately I think the right approach would be to use informative priors on the coefficients, to partially pool them toward zero and thus compromise between the two alternatives of no adjustment and simple least-squares adjustment. [sent-22, score-0.725]
16 I doubt you’d want to do that, though, and I imagine you’d also be uncomfortable with an approximation such as to fit the adjusted model and then divide the coefficients for the adjustment by 2 (that’s a simple compromise). [sent-23, score-0.766]
17 In your case my inclination would be to adjust for the 3 variables. [sent-24, score-0.196]
18 As noted above, I might consider informative priors on the b’s. [sent-28, score-0.188]
19 Possibly more importantly, I’d suggest allowing the coefficient for pre-test to interact with treatment vary by school. [sent-29, score-0.293]
20 And, in my experience , pre-test coefficients are systematically different in treatment and control groups. [sent-31, score-0.451]
wordName wordTfidf (topN-words)
[('ij', 0.287), ('covariates', 0.263), ('adjustment', 0.249), ('hedges', 0.223), ('rogosa', 0.223), ('treatment', 0.215), ('rcts', 0.191), ('model', 0.16), ('school', 0.156), ('covariate', 0.154), ('coefficients', 0.154), ('compromise', 0.152), ('adjust', 0.121), ('purpose', 0.105), ('fyi', 0.101), ('situation', 0.099), ('impact', 0.097), ('additional', 0.096), ('equating', 0.096), ('confound', 0.096), ('posttest', 0.096), ('fitting', 0.095), ('priors', 0.095), ('informative', 0.093), ('pretest', 0.092), ('argue', 0.089), ('imputations', 0.088), ('groups', 0.088), ('analysis', 0.087), ('consequently', 0.084), ('arrive', 0.084), ('inquiry', 0.082), ('consultant', 0.082), ('nationally', 0.082), ('amongst', 0.082), ('control', 0.082), ('context', 0.081), ('interact', 0.078), ('overfitting', 0.078), ('us', 0.076), ('inclination', 0.075), ('hand', 0.072), ('partially', 0.07), ('importantly', 0.07), ('adjusted', 0.069), ('assessing', 0.069), ('uncomfortable', 0.068), ('baseline', 0.067), ('divide', 0.066), ('pool', 0.066)]
simIndex simValue blogId blogTitle
same-blog 1 1.0000001 936 andrew gelman stats-2011-10-02-Covariate Adjustment in RCT - Model Overfitting in Multilevel Regression
Introduction: Makoto Hanita writes: We have been discussing the following two issues amongst ourselves, then with our methodological consultant for several days. However, we have not been able to arrive at a consensus. Consequently, we decided to seek an opinion from nationally known experts. FYI, we sent a similar inquiry to Larry Hedges and David Rogosa . . . 1) We are wondering if a post-hoc covariate adjustment is a good practice in the context of RCTs [randomized clinical trials]. We have a situation where we found a significant baseline difference between the treatment and the control groups in 3 variables. Some of us argue that adding those three variables to the original impact analysis model is a good idea, as that would remove the confound from the impact estimate. Others among us, on the other hand, argue that a post-hoc covariate adjustment should never be done, on the ground that those covariates are correlated with the treatment, which makes the analysis model that of quasi
Introduction: Avi sent along this old paper from Bryk and Raudenbush, who write: The presence of heterogeneity of variance across groups indicates that the standard statistical model for treatment effects no longer applies. Specifically, the assumption that treatments add a constant to each subject’s development fails. An alternative model is required to represent how treatment effects are distributed across individuals. We develop in this article a simple statistical model to demonstrate the link between heterogeneity of variance and random treatment effects. Next, we illustrate with results from two previously published studies how a failure to recognize the substantive importance of heterogeneity of variance obscured significant results present in these data. The article concludes with a review and synthesis of techniques for modeling variances. Although these methods have been well established in the statistical literature, they are not widely known by social and behavioral scientists. T
3 0.15150981 213 andrew gelman stats-2010-08-17-Matching at two levels
Introduction: Steve Porter writes with a question about matching for inferences in a hierarchical data structure. I’ve never thought about this particular issue, but it seems potentially important. Maybe one or more of you have some useful suggestions? Porter writes: After immersing myself in the relatively sparse literature on propensity scores with clustered data, it seems as if people take one of two approaches. If the treatment is at the cluster-level (like school policies), they match on only the cluster-level covariates. If the treatment is at the individual level, they match on individual-level covariates. (I have also found some papers that match on individual-level covariates when it seems as if the treatment is really at the cluster-level.) But what if there is a selection process at both levels? For my research question (effect of tenure systems on faculty behavior) there is a two-step selection process: first colleges choose whether to have a tenure system for faculty; then f
4 0.15130389 850 andrew gelman stats-2011-08-11-Understanding how estimates change when you move to a multilevel model
Introduction: Ramu Sudhagoni writes: I am working on combining three longitudinal studies using Bayesian hierarchical technique. In each study, I have at least 70 subjects follow up on 5 different visit months. My model consists of 10 different covariates including longitudinal and cross-sectional effects. Mixed models are used to fit the three studies individually using Bayesian approach and I noticed that few covariates were significant. When I combined using three level hierarchical approach, all the covariates became non-significant at the population level, and large estimates were found for variance parameters at the population level. I am struggling to understand why I am getting large variances at population level and wider credible intervals. I assumed non-informative normal priors for all my cross sectional and longitudinal effects, and non-informative inverse-gamma priors for variance parameters. I followed the approach explained by Inoue et al. (Title: Combining Longitudinal Studie
5 0.14753681 1486 andrew gelman stats-2012-09-07-Prior distributions for regression coefficients
Introduction: Eric Brown writes: I have come across a number of recommendations over the years about best practices for multilevel regression modeling. For example, the use of t-distributed priors for coefficients in logistic regression and standardizing input variables from one of your 2008 Annals of Applied Statistics papers; or recommendations for priors on variance parameters from your 2006 Bayesian Analysis paper. I understand that these are often of varied opinion of people in the field, but I was wondering if you have a reference that you point people to for a place to get started? I’ve tried looking through your blog posts but couldn’t find any summaries. For example, what are some examples of when I should use more than a two-level hierarchical model? Can I use a spike-slab coefficient model with a t-distributed prior for the slab rather than a normal? If I assume that my model is a priori wrong (but still useful), what are some recommended ways to choose how many interactions to u
6 0.14623556 1196 andrew gelman stats-2012-03-04-Piss-poor monocausal social science
7 0.12709519 823 andrew gelman stats-2011-07-26-Including interactions or not
8 0.12057535 1150 andrew gelman stats-2012-02-02-The inevitable problems with statistical significance and 95% intervals
9 0.12030571 1392 andrew gelman stats-2012-06-26-Occam
10 0.11981472 472 andrew gelman stats-2010-12-17-So-called fixed and random effects
12 0.11858758 753 andrew gelman stats-2011-06-09-Allowing interaction terms to vary
13 0.1183916 1723 andrew gelman stats-2013-02-15-Wacky priors can work well?
14 0.11706369 86 andrew gelman stats-2010-06-14-“Too much data”?
15 0.11471098 1350 andrew gelman stats-2012-05-28-Value-added assessment: What went wrong?
17 0.11383853 1092 andrew gelman stats-2011-12-29-More by Berger and me on weakly informative priors
18 0.11161054 1209 andrew gelman stats-2012-03-12-As a Bayesian I want scientists to report their data non-Bayesianly
19 0.10918744 2 andrew gelman stats-2010-04-23-Modeling heterogenous treatment effects
20 0.10758355 1310 andrew gelman stats-2012-05-09-Varying treatment effects, again
topicId topicWeight
[(0, 0.197), (1, 0.125), (2, 0.068), (3, -0.044), (4, 0.055), (5, 0.025), (6, 0.023), (7, 0.013), (8, 0.046), (9, 0.12), (10, 0.028), (11, 0.055), (12, 0.016), (13, -0.012), (14, -0.009), (15, 0.008), (16, 0.011), (17, 0.011), (18, -0.025), (19, 0.042), (20, -0.035), (21, 0.006), (22, -0.01), (23, -0.033), (24, -0.016), (25, 0.003), (26, -0.022), (27, -0.006), (28, -0.081), (29, 0.044), (30, -0.023), (31, -0.015), (32, 0.01), (33, 0.064), (34, -0.01), (35, 0.023), (36, -0.013), (37, 0.003), (38, 0.074), (39, 0.022), (40, -0.011), (41, -0.04), (42, 0.034), (43, 0.017), (44, 0.017), (45, 0.006), (46, -0.01), (47, -0.065), (48, -0.028), (49, 0.053)]
simIndex simValue blogId blogTitle
same-blog 1 0.9592849 936 andrew gelman stats-2011-10-02-Covariate Adjustment in RCT - Model Overfitting in Multilevel Regression
Introduction: Makoto Hanita writes: We have been discussing the following two issues amongst ourselves, then with our methodological consultant for several days. However, we have not been able to arrive at a consensus. Consequently, we decided to seek an opinion from nationally known experts. FYI, we sent a similar inquiry to Larry Hedges and David Rogosa . . . 1) We are wondering if a post-hoc covariate adjustment is a good practice in the context of RCTs [randomized clinical trials]. We have a situation where we found a significant baseline difference between the treatment and the control groups in 3 variables. Some of us argue that adding those three variables to the original impact analysis model is a good idea, as that would remove the confound from the impact estimate. Others among us, on the other hand, argue that a post-hoc covariate adjustment should never be done, on the ground that those covariates are correlated with the treatment, which makes the analysis model that of quasi
2 0.82568038 753 andrew gelman stats-2011-06-09-Allowing interaction terms to vary
Introduction: Zoltan Fazekas writes: I am a 2nd year graduate student in political science at the University of Vienna. In my empirical research I often employ multilevel modeling, and recently I came across a situation that kept me wondering for quite a while. As I did not find much on this in the literature and considering the topics that you work on and blog about, I figured I will try to contact you. The situation is as follows: in a linear multilevel model, there are two important individual level predictors (x1 and x2) and a set of controls. Let us assume that there is a theoretically grounded argument suggesting that an interaction between x1 and x2 should be included in the model (x1 * x2). Both x1 and x2 are let to vary randomly across groups. Would this directly imply that the coefficient of the interaction should also be left to vary across country? This is even more burning if there is no specific hypothesis on the variance of the conditional effect across countries. And then i
3 0.81759757 823 andrew gelman stats-2011-07-26-Including interactions or not
Introduction: Liz Sanders writes: I viewed your 2005 presentation “Interactions in multilevel models” and was hoping you or one of your students/colleagues could point me to some readings about the issue of using all possible vs. only particular interaction terms in regression models with continuous covariates (I think “functional form validity” is the term I have encountered in the past). In particular, I am trying to understand whether I would be mis-specifying a model if I deleted two of its interaction terms (in favor of using only 2-way treatment interaction terms). The general full model, for example, is: Y = intercept + txt + pre1 + pre2 + txt*pre1 + txt*pre2 + pre1*pre2 + txt*pre1*pre2, where txt is effect coded (1=treatment, -1=control) and pre1 and pre2 are two different pretests that are assumed normally distributed. (The model is actually a multilevel model; the error terms are not listed for brevity.) The truncated model, on the other hand, would only test 2-way treatment inte
4 0.80485183 213 andrew gelman stats-2010-08-17-Matching at two levels
Introduction: Steve Porter writes with a question about matching for inferences in a hierarchical data structure. I’ve never thought about this particular issue, but it seems potentially important. Maybe one or more of you have some useful suggestions? Porter writes: After immersing myself in the relatively sparse literature on propensity scores with clustered data, it seems as if people take one of two approaches. If the treatment is at the cluster-level (like school policies), they match on only the cluster-level covariates. If the treatment is at the individual level, they match on individual-level covariates. (I have also found some papers that match on individual-level covariates when it seems as if the treatment is really at the cluster-level.) But what if there is a selection process at both levels? For my research question (effect of tenure systems on faculty behavior) there is a two-step selection process: first colleges choose whether to have a tenure system for faculty; then f
Introduction: Chris Che-Castaldo writes: I am trying to compute variance components for a hierarchical model where the group level has two binary predictors and their interaction. When I model each of these three predictors as N(0, tau) the model will not converge, perhaps because the number of coefficients in each batch is so small (2 for the main effects and 4 for the interaction). Although I could simply leave all these as predictors as unmodeled fixed effects, the last sentence of section 21.2 on page 462 of Gelman and Hill (2007) suggests this would not be a wise course of action: For example, it is not clear how to define the (finite) standard deviation of variables that are included in interactions. I am curious – is there still no clear cut way to directly compute the finite standard deviation for binary unmodeled variables that are also part of an interaction as well as the interaction itself? My reply: I’d recommend including these in your model (it’s probably easiest to do so
6 0.78591216 251 andrew gelman stats-2010-09-02-Interactions of predictors in a causal model
7 0.78340369 315 andrew gelman stats-2010-10-03-He doesn’t trust the fit . . . r=.999
8 0.77807182 86 andrew gelman stats-2010-06-14-“Too much data”?
9 0.75830704 1294 andrew gelman stats-2012-05-01-Modeling y = a + b + c
10 0.75596666 888 andrew gelman stats-2011-09-03-A psychology researcher asks: Is Anova dead?
11 0.75185513 1070 andrew gelman stats-2011-12-19-The scope for snooping
13 0.73521823 1267 andrew gelman stats-2012-04-17-Hierarchical-multilevel modeling with “big data”
14 0.7344557 250 andrew gelman stats-2010-09-02-Blending results from two relatively independent multi-level models
15 0.73016018 77 andrew gelman stats-2010-06-09-Sof[t]
16 0.72699779 1248 andrew gelman stats-2012-04-06-17 groups, 6 group-level predictors: What to do?
17 0.72670788 1468 andrew gelman stats-2012-08-24-Multilevel modeling and instrumental variables
18 0.72046095 1966 andrew gelman stats-2013-08-03-Uncertainty in parameter estimates using multilevel models
19 0.71923542 375 andrew gelman stats-2010-10-28-Matching for preprocessing data for causal inference
20 0.71701014 1383 andrew gelman stats-2012-06-18-Hierarchical modeling as a framework for extrapolation
topicId topicWeight
[(4, 0.018), (15, 0.04), (16, 0.077), (21, 0.016), (24, 0.211), (34, 0.184), (57, 0.014), (76, 0.017), (82, 0.033), (84, 0.019), (86, 0.024), (99, 0.225)]
simIndex simValue blogId blogTitle
1 0.9570502 929 andrew gelman stats-2011-09-27-Visual diagnostics for discrete-data regressions
Introduction: Jeff asked me what I thought of this recent AJPS article by Brian Greenhill, Michael Ward, and Audrey Sacks, “The Separation Plot: A New Visual Method for Evaluating the Fit of Binary Models.” It’s similar to a graph of observed vs. predicted values, but using color rather than the y-axis to display the observed values. It seems like it could be useful, also could be applied more generally to discrete-data regressions with more than two categories. When it comes to checking the model fit, I recommend binned residual plots, as discussed in this 2000 article with Yuri Goegebeur, Francis Tuerlinckx, and Iven Van Mechelen.
2 0.94785941 1144 andrew gelman stats-2012-01-29-How many parameters are in a multilevel model?
Introduction: Stephen Collins writes: I’m reading your Multilevel modeling book and am trying to apply it to my work. I’m concerned with how to estimate a random intercept model if there are hundreds/thousands of levels. In the Gibbs sampling, am I sampling a parameter for each level? Or, just the hyper-parameters? In other words, say I had 500 zipcode intercepts modeled as ~ N(m,s). Would my posterior be two dimensional, sampling for “m” and “s,” or would it have 502 dimensions? My reply: Indeed you will have hundreds or thousands of parameters—or, in classical terms, hundreds or thousands of predictive quantities. But that’s ok. Even if none of those predictions is precise, you’re learning about the model. See page 526 of the book for more discussion of the number of parameters in a multilevel model.
3 0.94780326 1911 andrew gelman stats-2013-06-23-AI Stats conference on Stan etc.
Introduction: Jaakko Peltonen writes: The Seventeenth International Conference on Artificial Intelligence and Statistics (http://www.aistats.org) will be next April in Reykjavik, Iceland. AISTATS is an interdisciplinary conference at the intersection of computer science, artificial intelligence, machine learning, statistics, and related areas. ============================================================================== AISTATS 2014 Call for Papers Seventeenth International Conference on Artificial Intelligence and Statistics April 22 – 25, 2014, Reykjavik, Iceland http://www.aistats.org Colocated with a MLSS Machine Learning Summer School ============================================================================== AISTATS is an interdisciplinary gathering of researchers at the intersection of computer science, artificial intelligence, machine learning, statistics, and related areas. Since its inception in 1985, the primary goal of AISTATS has been to broaden research in the
4 0.93285334 1501 andrew gelman stats-2012-09-18-More studies on the economic effects of climate change
Introduction: After writing yesterday’s post , I was going through Solomon Hsiang’s blog and found a post pointing to three studies from researchers at business schools: Severe Weather and Automobile Assembly Productivity Gérard P. Cachon, Santiago Gallino and Marcelo Olivares Abstract: It is expected that climate change could lead to an increased frequency of severe weather. In turn, severe weather intuitively should hamper the productivity of work that occurs outside. But what is the effect of rain, snow, fog, heat and wind on work that occurs indoors, such as the production of automobiles? Using weekly production data from 64 automobile plants in the United States over a ten-year period, we find that adverse weather conditions lead to a significant reduction in production. For example, one additional day of high wind advisory by the National Weather Service (i.e., maximum winds generally in excess of 44 miles per hour) reduces production by 26%, which is comparable in order of magnitude t
same-blog 5 0.93086398 936 andrew gelman stats-2011-10-02-Covariate Adjustment in RCT - Model Overfitting in Multilevel Regression
Introduction: Makoto Hanita writes: We have been discussing the following two issues amongst ourselves, then with our methodological consultant for several days. However, we have not been able to arrive at a consensus. Consequently, we decided to seek an opinion from nationally known experts. FYI, we sent a similar inquiry to Larry Hedges and David Rogosa . . . 1) We are wondering if a post-hoc covariate adjustment is a good practice in the context of RCTs [randomized clinical trials]. We have a situation where we found a significant baseline difference between the treatment and the control groups in 3 variables. Some of us argue that adding those three variables to the original impact analysis model is a good idea, as that would remove the confound from the impact estimate. Others among us, on the other hand, argue that a post-hoc covariate adjustment should never be done, on the ground that those covariates are correlated with the treatment, which makes the analysis model that of quasi
6 0.9243744 135 andrew gelman stats-2010-07-09-Rasmussen sez: “108% of Respondents Say . . .”
7 0.91419673 292 andrew gelman stats-2010-09-23-Doug Hibbs on the fundamentals in 2010
8 0.91130912 1734 andrew gelman stats-2013-02-23-Life in the C-suite: A graph that is both ugly and bad, and an unrelated story
10 0.88790679 388 andrew gelman stats-2010-11-01-The placebo effect in pharma
12 0.88426399 1723 andrew gelman stats-2013-02-15-Wacky priors can work well?
13 0.88245386 1842 andrew gelman stats-2013-05-05-Cleaning up science
14 0.87573838 1111 andrew gelman stats-2012-01-10-The blog of the Cultural Cognition Project
15 0.87184215 956 andrew gelman stats-2011-10-13-Hey, you! Don’t take that class!
16 0.87120593 994 andrew gelman stats-2011-11-06-Josh Tenenbaum presents . . . a model of folk physics!
17 0.86843908 2140 andrew gelman stats-2013-12-19-Revised evidence for statistical standards
18 0.86566412 42 andrew gelman stats-2010-05-19-Updated solutions to Bayesian Data Analysis homeworks
19 0.86226928 1612 andrew gelman stats-2012-12-08-The Case for More False Positives in Anti-doping Testing
20 0.86173713 2288 andrew gelman stats-2014-04-10-Small multiples of lineplots > maps (ok, not always, but yes in this case)