andrew_gelman_stats andrew_gelman_stats-2011 andrew_gelman_stats-2011-553 knowledge-graph by maker-knowledge-mining
Source: html
Introduction: Peter Bergman writes: is it possible to “overstratify” when assigning a treatment in a randomized control trial? I [Bergman] have a sample size of roughly 400 people, and several binary variables correlate strongly with the outcome of interest and would also define interesting subgroups for analysis. The problem is, stratifying over all of these (five or six) variables leaves me with strata that have only 1 person in them. I have done some background reading on whether there is a rule of thumb for the maximum number of variables to stratify. There does not seem to be much agreement (some say there should be between N/50-N/100 strata, others say as few as possible). In economics, the paper I looked to is here, which seems to summarize literature related to clinical trials. In short, my question is: is it bad to have several strata with 1 person in them? Should I group these people in with another stratum? P.S. In the paper I mention above, they also say it is important to inc
sentIndex sentText sentNum sentScore
1 Peter Bergman writes: is it possible to “overstratify” when assigning a treatment in a randomized control trial? [sent-1, score-0.362]
2 I [Bergman] have a sample size of roughly 400 people, and several binary variables correlate strongly with the outcome of interest and would also define interesting subgroups for analysis. [sent-2, score-0.605]
3 The problem is, stratifying over all of these (five or six) variables leaves me with strata that have only 1 person in them. [sent-3, score-1.023]
4 I have done some background reading on whether there is a rule of thumb for the maximum number of variables to stratify. [sent-4, score-0.284]
5 There does not seem to be much agreement (some say there should be between N/50-N/100 strata, others say as few as possible). [sent-5, score-0.2]
6 In economics, the paper I looked to is here, which seems to summarize literature related to clinical trials. [sent-6, score-0.135]
7 In short, my question is: is it bad to have several strata with 1 person in them? [sent-7, score-0.647]
8 In the paper I mention above, they also say it is important to include stratum indicators in the regression analysis to ensure the appropriate sized type-I error in the final analysis (i. [sent-11, score-1.012]
9 They demonstrate this through simulation, but is there a reference (or intuition) that shows why these indicators are important theoretically? [sent-14, score-0.251]
10 My reply: I doubt it matters so much exactly how you do this. [sent-15, score-0.063]
11 If you want, there are techniques to ensure balance over many predictors. [sent-16, score-0.237]
12 In balanced setups, you have ideas such as latin squares, and similar methods can be developed in unbalanced scenarios. [sent-17, score-0.279]
13 It’s ok to have strata with one person in them, but if you think people won’t like it, then you should feel free to use larger strata. [sent-18, score-0.576]
14 In answer to your other question about references: Yes, it’s standard advice to include all design information as regression predictors. [sent-19, score-0.154]
15 Bergman actually wrote “dummies,” but I couldn’t bear to see that term so I changed it to “ïndicators. [sent-25, score-0.08]
wordName wordTfidf (topN-words)
[('strata', 0.457), ('bergman', 0.321), ('indicators', 0.251), ('stratifying', 0.249), ('stratum', 0.198), ('ensure', 0.165), ('outcome', 0.159), ('treatment', 0.144), ('variables', 0.124), ('person', 0.119), ('setups', 0.114), ('dummies', 0.107), ('unbalanced', 0.102), ('sized', 0.099), ('thumb', 0.096), ('correlate', 0.096), ('latin', 0.096), ('imbalance', 0.093), ('subgroups', 0.091), ('regress', 0.089), ('sorts', 0.089), ('assigning', 0.084), ('theoretically', 0.084), ('balanced', 0.081), ('include', 0.081), ('bear', 0.08), ('error', 0.08), ('pearl', 0.078), ('damn', 0.076), ('leaves', 0.074), ('regression', 0.073), ('causality', 0.072), ('balance', 0.072), ('several', 0.071), ('squares', 0.071), ('summarize', 0.071), ('bda', 0.071), ('agreement', 0.07), ('trial', 0.07), ('possible', 0.069), ('adjust', 0.068), ('intuition', 0.066), ('say', 0.065), ('randomized', 0.065), ('simulation', 0.065), ('maximum', 0.064), ('clinical', 0.064), ('binary', 0.064), ('peter', 0.064), ('matters', 0.063)]
simIndex simValue blogId blogTitle
same-blog 1 0.99999994 553 andrew gelman stats-2011-02-03-is it possible to “overstratify” when assigning a treatment in a randomized control trial?
Introduction: Peter Bergman writes: is it possible to “overstratify” when assigning a treatment in a randomized control trial? I [Bergman] have a sample size of roughly 400 people, and several binary variables correlate strongly with the outcome of interest and would also define interesting subgroups for analysis. The problem is, stratifying over all of these (five or six) variables leaves me with strata that have only 1 person in them. I have done some background reading on whether there is a rule of thumb for the maximum number of variables to stratify. There does not seem to be much agreement (some say there should be between N/50-N/100 strata, others say as few as possible). In economics, the paper I looked to is here, which seems to summarize literature related to clinical trials. In short, my question is: is it bad to have several strata with 1 person in them? Should I group these people in with another stratum? P.S. In the paper I mention above, they also say it is important to inc
Introduction: I received the following note from someone who’d like to remain anonymous: I read your post on ethics and statistics, and the comments therein, with much interest. I did notice, however, that most of the dialogue was about ethical behavior of scientists. Herein I’d like to suggest a different take, one that focuses on the statistical methods of scientists. For example, fitting a line to a scatter plot of data using OLS [linear regression] gives more weight to outliers. If each data point represents a person we are weighting people differently. And surely the ethical implications are different if we use a least absolute deviation estimator. Recently I reviewed a paper where the authors claimed one advantage of non-parametric rank-based tests is their robustness to outliers. Again, maybe that outlier is the 10th person who dies from an otherwise beneficial medicine. Should we ignore him in assessing the effect of the medicine? I guess this gets me partly into loss f
3 0.11650057 2296 andrew gelman stats-2014-04-19-Index or indicator variables
Introduction: Someone who doesn’t want his name shared (for the perhaps reasonable reason that he’ll “one day not be confused, and would rather my confusion not live on online forever”) writes: I’m exploring HLMs and stan, using your book with Jennifer Hill as my field guide to this new territory. I think I have a generally clear grasp on the material, but wanted to be sure I haven’t gone astray. The problem in working on involves a multi-nation survey of students, and I’m especially interested in understanding the effects of country, religion, and sex, and the interactions among those factors (using IRT to estimate individual-level ability, then estimating individual, school, and country effects). Following the basic approach laid out in chapter 13 for such interactions between levels, I think I need to create a matrix of indicator variables for religion and sex. Elsewhere in the book, you recommend against indicator variables in favor of a single index variable. Am I right in thinking t
4 0.11237784 451 andrew gelman stats-2010-12-05-What do practitioners need to know about regression?
Introduction: Fabio Rojas writes: In much of the social sciences outside economics, it’s very common for people to take a regression course or two in graduate school and then stop their statistical education. This creates a situation where you have a large pool of people who have some knowledge, but not a lot of knowledge. As a result, you have a pretty big gap between people like yourself, who are heavily invested in the cutting edge of applied statistics, and other folks. So here is the question: What are the major lessons about good statistical practice that “rank and file” social scientists should know? Sure, most people can recite “Correlation is not causation” or “statistical significance is not substantive significance.” But what are the other big lessons? This question comes from my own experience. I have a math degree and took regression analysis in graduate school, but I definitely do not have the level of knowledge of a statistician. I also do mixed method research, and field wor
5 0.10653692 888 andrew gelman stats-2011-09-03-A psychology researcher asks: Is Anova dead?
Introduction: A research psychologist writes in with a question that’s so long that I’ll put my answer first, then put the question itself below the fold. Here’s my reply: As I wrote in my Anova paper and in my book with Jennifer Hill, I do think that multilevel models can completely replace Anova. At the same time, I think the central idea of Anova should persist in our understanding of these models. To me the central idea of Anova is not F-tests or p-values or sums of squares, but rather the idea of predicting an outcome based on factors with discrete levels, and understanding these factors using variance components. The continuous or categorical response thing doesn’t really matter so much to me. I have no problem using a normal linear model for continuous outcomes (perhaps suitably transformed) and a logistic model for binary outcomes. I don’t want to throw away interactions just because they’re not statistically significant. I’d rather partially pool them toward zero using an inform
6 0.10616235 1070 andrew gelman stats-2011-12-19-The scope for snooping
7 0.098172858 86 andrew gelman stats-2010-06-14-“Too much data”?
8 0.097581029 777 andrew gelman stats-2011-06-23-Combining survey data obtained using different modes of sampling
12 0.094880551 7 andrew gelman stats-2010-04-27-Should Mister P be allowed-encouraged to reside in counter-factual populations?
13 0.094605148 1900 andrew gelman stats-2013-06-15-Exploratory multilevel analysis when group-level variables are of importance
14 0.093637742 936 andrew gelman stats-2011-10-02-Covariate Adjustment in RCT - Model Overfitting in Multilevel Regression
16 0.08784499 1341 andrew gelman stats-2012-05-24-Question 14 of my final exam for Design and Analysis of Sample Surveys
17 0.086644903 1506 andrew gelman stats-2012-09-21-Building a regression model . . . with only 27 data points
18 0.084301561 107 andrew gelman stats-2010-06-24-PPS in Georgia
20 0.082149774 1133 andrew gelman stats-2012-01-21-Judea Pearl on why he is “only a half-Bayesian”
topicId topicWeight
[(0, 0.162), (1, 0.035), (2, 0.055), (3, -0.063), (4, 0.047), (5, 0.016), (6, 0.01), (7, -0.003), (8, 0.073), (9, 0.037), (10, 0.006), (11, -0.018), (12, 0.044), (13, 0.002), (14, 0.04), (15, -0.007), (16, -0.013), (17, 0.007), (18, 0.023), (19, 0.022), (20, -0.021), (21, 0.005), (22, 0.063), (23, 0.018), (24, 0.029), (25, 0.067), (26, 0.038), (27, -0.053), (28, -0.036), (29, 0.02), (30, 0.021), (31, 0.048), (32, 0.002), (33, 0.039), (34, -0.002), (35, -0.037), (36, 0.009), (37, 0.021), (38, 0.002), (39, 0.007), (40, -0.028), (41, -0.053), (42, 0.001), (43, 0.018), (44, 0.029), (45, 0.021), (46, 0.022), (47, -0.045), (48, -0.001), (49, 0.043)]
simIndex simValue blogId blogTitle
same-blog 1 0.96486682 553 andrew gelman stats-2011-02-03-is it possible to “overstratify” when assigning a treatment in a randomized control trial?
Introduction: Peter Bergman writes: is it possible to “overstratify” when assigning a treatment in a randomized control trial? I [Bergman] have a sample size of roughly 400 people, and several binary variables correlate strongly with the outcome of interest and would also define interesting subgroups for analysis. The problem is, stratifying over all of these (five or six) variables leaves me with strata that have only 1 person in them. I have done some background reading on whether there is a rule of thumb for the maximum number of variables to stratify. There does not seem to be much agreement (some say there should be between N/50-N/100 strata, others say as few as possible). In economics, the paper I looked to is here, which seems to summarize literature related to clinical trials. In short, my question is: is it bad to have several strata with 1 person in them? Should I group these people in with another stratum? P.S. In the paper I mention above, they also say it is important to inc
Introduction: Last year we discussed an important challenge in causal inference: The standard advice (given in many books, including ours) for causal inference is to control for relevant pre-treatment variables as much as possible. But, as Judea Pearl has pointed out, instruments (as in “instrumental variables”) are pre-treatment variables that we would not want to “control for” in a matching or regression sense. At first, this seems like a minor modification, with the new recommendation being to apply instrumental variables estimation using all pre-treatment instruments, and to control for all other pre-treatment variables. But that can’t really work as general advice. What about weak instruments or covariates that have some instrumental aspects? I asked Paul Rosenbaum for his thoughts on the matter, and he wrote the following: In section 18.2 of Design of Observational Studies (DOS), I [Rosenbaum] discuss “seemingly innocuous confounding” defined to be a covariate that predicts a su
3 0.77152336 1967 andrew gelman stats-2013-08-04-What are the key assumptions of linear regression?
Introduction: Andy Cooper writes: A link to an article , “Four Assumptions Of Multiple Regression That Researchers Should Always Test”, has been making the rounds on Twitter. Their first rule is “Variables are Normally distributed.” And they seem to be talking about the independent variables – but then later bring in tests on the residuals (while admitting that the normally-distributed error assumption is a weak assumption). I thought we had long-since moved away from transforming our independent variables to make them normally distributed for statistical reasons (as opposed to standardizing them for interpretability, etc.) Am I missing something? I agree that leverage in a influence is important, but normality of the variables? The article is from 2002, so it might be dated, but given the popularity of the tweet, I thought I’d ask your opinion. My response: There’s some useful advice on that page but overall I think the advice was dated even in 2002. In section 3.6 of my book wit
4 0.76411366 451 andrew gelman stats-2010-12-05-What do practitioners need to know about regression?
Introduction: Fabio Rojas writes: In much of the social sciences outside economics, it’s very common for people to take a regression course or two in graduate school and then stop their statistical education. This creates a situation where you have a large pool of people who have some knowledge, but not a lot of knowledge. As a result, you have a pretty big gap between people like yourself, who are heavily invested in the cutting edge of applied statistics, and other folks. So here is the question: What are the major lessons about good statistical practice that “rank and file” social scientists should know? Sure, most people can recite “Correlation is not causation” or “statistical significance is not substantive significance.” But what are the other big lessons? This question comes from my own experience. I have a math degree and took regression analysis in graduate school, but I definitely do not have the level of knowledge of a statistician. I also do mixed method research, and field wor
5 0.76372671 1663 andrew gelman stats-2013-01-09-The effects of fiscal consolidation
Introduction: José Iparraguirre writes: I’ve read a recent paper by the International Monetary Fund on the effects of fiscal consolidation measures on income inequality (Fiscal Monitor October 2012, Appendix 1). They run a panel regression with 48 countries and 30 years (annual data) of a measure of income inequality (Gini coefficient) on a number of covariates, including a measure of fiscal consolidation. Footnote 39 (page 51) informs that they’ve employed seemingly unrelated regression and panel-corrected standard errors, and that to double-check they’ve also run ordinary least squares and fixed-effects panel regressions—all with similar results. So far, so good. However, the footnote goes on to explain that “Some of the results (e.g. the causal relationship between consolidation and inequality) may be subject to endogeneity and should be interpreted with caution”. (Italics are mine). Therefore, it seems that the crux of the exercise—i.e. estimating the relationship between fiscal con
6 0.74914235 86 andrew gelman stats-2010-06-14-“Too much data”?
7 0.74762905 14 andrew gelman stats-2010-05-01-Imputing count data
8 0.74609607 796 andrew gelman stats-2011-07-10-Matching and regression: two great tastes etc etc
9 0.74470836 2204 andrew gelman stats-2014-02-09-Keli Liu and Xiao-Li Meng on Simpson’s paradox
10 0.74385375 375 andrew gelman stats-2010-10-28-Matching for preprocessing data for causal inference
11 0.73933095 1656 andrew gelman stats-2013-01-05-Understanding regression models and regression coefficients
12 0.73189247 888 andrew gelman stats-2011-09-03-A psychology researcher asks: Is Anova dead?
14 0.72844923 251 andrew gelman stats-2010-09-02-Interactions of predictors in a causal model
15 0.71530277 770 andrew gelman stats-2011-06-15-Still more Mr. P in public health
16 0.71409464 257 andrew gelman stats-2010-09-04-Question about standard range for social science correlations
17 0.71210319 2274 andrew gelman stats-2014-03-30-Adjudicating between alternative interpretations of a statistical interaction?
18 0.7027604 248 andrew gelman stats-2010-09-01-Ratios where the numerator and denominator both change signs
19 0.70046496 1462 andrew gelman stats-2012-08-18-Standardizing regression inputs
20 0.70026618 708 andrew gelman stats-2011-05-12-Improvement of 5 MPG: how many more auto deaths?
topicId topicWeight
[(15, 0.024), (16, 0.065), (21, 0.015), (24, 0.154), (53, 0.029), (63, 0.011), (73, 0.01), (86, 0.035), (95, 0.021), (97, 0.221), (98, 0.011), (99, 0.281)]
simIndex simValue blogId blogTitle
1 0.9602949 1573 andrew gelman stats-2012-11-11-Incredibly strange spam
Introduction: Unsolicited (of course) in the email the other day: Just wanted to touch base with you to see if you needed any quotes on Parking lot lighting or Garage Lighting? (Induction, LED, Canopy etc…) We help retrofit 1000′s of garages around the country. Let me know your specs and ill send you a quote in 24 hours. ** Owner Emergency Lights Co. Ill indeed. . . .
2 0.94900739 996 andrew gelman stats-2011-11-07-Chi-square FAIL when many cells have small expected values
Introduction: William Perkins, Mark Tygert, and Rachel Ward write : If a discrete probability distribution in a model being tested for goodness-of-fit is not close to uniform, then forming the Pearson χ2 statistic can involve division by nearly zero. This often leads to serious trouble in practice — even in the absence of round-off errors . . . The problem is not merely that the chi-squared statistic doesn’t have the advertised chi-squared distribution —a reference distribution can always be computed via simulation, either using the posterior predictive distribution or by conditioning on a point estimate of the cell expectations and then making a degrees-of-freedom sort of adjustment. Rather, the problem is that, when there are lots of cells with near-zero expectation, the chi-squared test is mostly noise. And this is not merely a theoretical problem. It comes up in real examples. Here’s one, taken from the classic 1992 genetics paper of Guo and Thomspson: And here are the e
3 0.94763774 160 andrew gelman stats-2010-07-23-Unhappy with improvement by a factor of 10^29
Introduction: I have an optimization problem: I have a complicated physical model that predicts energy and thermal behavior of a building, given the values of a slew of parameters, such as insulation effectiveness, window transmissivity, etc. I’m trying to find the parameter set that best fits several weeks of thermal and energy use data from the real building that we modeled. (Of course I would rather explore parameter space and come up with probability distributions for the parameters, and maybe that will come later, but for now I’m just optimizing). To do the optimization, colleagues and I implemented a “particle swarm optimization” algorithm on a massively parallel machine. This involves giving each of about 120 “particles” an initial position in parameter space, then letting them move around, trying to move to better positions according to a specific algorithm. We gave each particle an initial position sampled from our prior distribution for each parameter. So far we’ve run about 140 itera
4 0.94530928 882 andrew gelman stats-2011-08-31-Meanwhile, on the sister blog . . .
Introduction: NYT columnist Douthat asks: Should we be disturbed that a leading presidential candidate endorses a pro-slavery position? Who’s on the web? And where are they? Sowell, Carlson, Barone: fools, knaves, or simply victims of a cognitive illusion? Don’t blame the American public for the D.C. deadlock Calvin College update Help reform the Institutional Review Board (IRB) system! Powerful credit-rating agencies are a creation of the government . . . what does it mean when they bite the hand that feeds them? “Waiting for a landslide” A simple theory of why Obama didn’t come out fighting in 2009 A modest proposal Noooooooooooooooo!!!!!!!!!!!!!!! The Family Research Council and the Barnard Center for Research on Women Sleazy data miners Genetic essentialism is in our genes Wow, that was a lot! No wonder I don’t get any research done…
Introduction: Earlier today, Nate criticized a U.S. military survey that asks troops the question, “Do you currently serve with a male or female Service member you believe to be homosexual.” [emphasis added] As Nate points out, by asking this question in such a speculative way, “it would seem that you’ll be picking up a tremendous number of false positives–soldiers who are believed to be gay, but aren’t–and that these false positives will swamp any instances in which soldiers (in spite of DADT) are actually somewhat open about their same-sex attractions.” This is a general problem in survey research. In an article in Chance magazine in 1997, “The myth of millions of annual self-defense gun uses: a case study of survey overestimates of rare events” [see here for related references], David Hemenway uses the false-positive, false-negative reasoning to explain this bias in terms of probability theory. Misclassifications that induce seemingly minor biases in estimates of certain small probab
same-blog 6 0.92837709 553 andrew gelman stats-2011-02-03-is it possible to “overstratify” when assigning a treatment in a randomized control trial?
8 0.90689152 1001 andrew gelman stats-2011-11-10-Three hours in the life of a statistician
9 0.89678049 13 andrew gelman stats-2010-04-30-Things I learned from the Mickey Kaus for Senate campaign
10 0.88539064 526 andrew gelman stats-2011-01-19-“If it saves the life of a single child…” and other nonsense
11 0.87939942 820 andrew gelman stats-2011-07-25-Design of nonrandomized cluster sample study
12 0.87658238 1812 andrew gelman stats-2013-04-19-Chomsky chomsky chomsky chomsky furiously
13 0.87298292 1694 andrew gelman stats-2013-01-26-Reflections on ethicsblogging
14 0.87290955 2118 andrew gelman stats-2013-11-30-???
15 0.86110312 1335 andrew gelman stats-2012-05-21-Responding to a bizarre anti-social-science screed
16 0.85657245 18 andrew gelman stats-2010-05-06-$63,000 worth of abusive research . . . or just a really stupid waste of time?
17 0.85519218 112 andrew gelman stats-2010-06-27-Sampling rate of human-scaled time series
18 0.84549665 115 andrew gelman stats-2010-06-28-Whassup with those crappy thrillers?
19 0.84524089 2121 andrew gelman stats-2013-12-02-Should personal genetic testing be regulated? Battle of the blogroll
20 0.84457815 2246 andrew gelman stats-2014-03-13-An Economist’s Guide to Visualizing Data