andrew_gelman_stats andrew_gelman_stats-2014 andrew_gelman_stats-2014-2294 knowledge-graph by maker-knowledge-mining
Source: html
Introduction: Nelson Villoria writes: I find the multilevel approach very useful for a problem I am dealing with, and I was wondering whether you could point me to some references about poolability tests for multilevel models. I am working with time series of cross sectional data and I want to test whether the data supports cross sectional and/or time pooling. In a standard panel data setting I do this with Chow tests and/or CUSUM. Are these ideas directly transferable to the multilevel setting? My reply: I think you should do partial pooling. Once the question arises, just do it. Other models are just special cases. I don’t see the need for any test. That said, if you do a group-level model, you need to consider including group-level averages of individual predictors (see here ). And if the number of groups is small, there can be real gains from using an informative prior distribution on the hierarchical variance parameters. This is something that Jennifer and I do not discuss in our
sentIndex sentText sentNum sentScore
1 Nelson Villoria writes: I find the multilevel approach very useful for a problem I am dealing with, and I was wondering whether you could point me to some references about poolability tests for multilevel models. [sent-1, score-1.411]
2 I am working with time series of cross sectional data and I want to test whether the data supports cross sectional and/or time pooling. [sent-2, score-2.345]
3 In a standard panel data setting I do this with Chow tests and/or CUSUM. [sent-3, score-0.705]
4 Are these ideas directly transferable to the multilevel setting? [sent-4, score-0.659]
5 My reply: I think you should do partial pooling. [sent-5, score-0.131]
6 That said, if you do a group-level model, you need to consider including group-level averages of individual predictors (see here ). [sent-9, score-0.587]
7 And if the number of groups is small, there can be real gains from using an informative prior distribution on the hierarchical variance parameters. [sent-10, score-0.851]
8 This is something that Jennifer and I do not discuss in our book, unfortunately. [sent-11, score-0.087]
wordName wordTfidf (topN-words)
[('sectional', 0.429), ('cross', 0.31), ('multilevel', 0.284), ('transferable', 0.214), ('chow', 0.214), ('tests', 0.199), ('setting', 0.198), ('nelson', 0.167), ('gains', 0.161), ('supports', 0.155), ('panel', 0.144), ('dealing', 0.14), ('averages', 0.139), ('arises', 0.131), ('partial', 0.131), ('whether', 0.13), ('references', 0.123), ('jennifer', 0.121), ('need', 0.115), ('predictors', 0.107), ('wondering', 0.107), ('informative', 0.104), ('variance', 0.103), ('unfortunately', 0.101), ('groups', 0.098), ('special', 0.098), ('hierarchical', 0.098), ('series', 0.096), ('data', 0.093), ('discuss', 0.087), ('directly', 0.086), ('test', 0.084), ('individual', 0.084), ('distribution', 0.079), ('prior', 0.077), ('ideas', 0.075), ('time', 0.074), ('approach', 0.072), ('useful', 0.072), ('including', 0.072), ('standard', 0.071), ('consider', 0.07), ('working', 0.068), ('small', 0.068), ('number', 0.067), ('reply', 0.066), ('said', 0.065), ('real', 0.064), ('see', 0.06), ('book', 0.059)]
simIndex simValue blogId blogTitle
same-blog 1 0.99999994 2294 andrew gelman stats-2014-04-17-If you get to the point of asking, just do it. But some difficulties do arise . . .
Introduction: Nelson Villoria writes: I find the multilevel approach very useful for a problem I am dealing with, and I was wondering whether you could point me to some references about poolability tests for multilevel models. I am working with time series of cross sectional data and I want to test whether the data supports cross sectional and/or time pooling. In a standard panel data setting I do this with Chow tests and/or CUSUM. Are these ideas directly transferable to the multilevel setting? My reply: I think you should do partial pooling. Once the question arises, just do it. Other models are just special cases. I don’t see the need for any test. That said, if you do a group-level model, you need to consider including group-level averages of individual predictors (see here ). And if the number of groups is small, there can be real gains from using an informative prior distribution on the hierarchical variance parameters. This is something that Jennifer and I do not discuss in our
2 0.18718986 1248 andrew gelman stats-2012-04-06-17 groups, 6 group-level predictors: What to do?
Introduction: Yi-Chun Ou writes: I am using a multilevel model with three levels. I read that you wrote a book about multilevel models, and wonder if you can solve the following question. The data structure is like this: Level one: customer (8444 customers) Level two: companys (90 companies) Level three: industry (17 industries) I use 6 level-three variables (i.e. industry characteristics) to explain the variance of the level-one effect across industries. The question here is whether there is an over-fitting problem since there are only 17 industries. I understand that this must be a problem for non-multilevel models, but is it also a problem for multilevel models? My reply: Yes, this could be a problem. I’d suggest combining some of your variables into a common score, or using only some of the variables, or using strong priors to control the inferences. This is an interesting and important area of statistics research, to do this sort of thing systematically. There’s lots o
3 0.16760331 850 andrew gelman stats-2011-08-11-Understanding how estimates change when you move to a multilevel model
Introduction: Ramu Sudhagoni writes: I am working on combining three longitudinal studies using Bayesian hierarchical technique. In each study, I have at least 70 subjects follow up on 5 different visit months. My model consists of 10 different covariates including longitudinal and cross-sectional effects. Mixed models are used to fit the three studies individually using Bayesian approach and I noticed that few covariates were significant. When I combined using three level hierarchical approach, all the covariates became non-significant at the population level, and large estimates were found for variance parameters at the population level. I am struggling to understand why I am getting large variances at population level and wider credible intervals. I assumed non-informative normal priors for all my cross sectional and longitudinal effects, and non-informative inverse-gamma priors for variance parameters. I followed the approach explained by Inoue et al. (Title: Combining Longitudinal Studie
4 0.16409512 295 andrew gelman stats-2010-09-25-Clusters with very small numbers of observations
Introduction: James O’Brien writes: How would you explain, to a “classically-trained” hypothesis-tester, that “It’s OK to fit a multilevel model even if some groups have only one observation each”? I [O'Brien] think I understand the logic and the statistical principles at work in this, but I’ve having trouble being clear and persuasive. I also feel like I’m contending with some methodological conventional wisdom here. My reply: I’m so used to this idea that I find it difficult to defend it in some sort of general conceptual way. So let me retreat to a more functional defense, which is that multilevel modeling gives good estimates, especially when the number of observations per group is small. One way to see this in any particular example in through cross-validation. Another way is to consider the alternatives. If you try really hard you can come up with a “classical hypothesis testing” approach which will do as well as the multilevel model. It would just take a lot of work. I’d r
5 0.15423618 1737 andrew gelman stats-2013-02-25-Correlation of 1 . . . too good to be true?
Introduction: Alex Hoffman points me to this interview by Dylan Matthews of education researcher Thomas Kane, who at one point says, Once you corrected for measurement error, a teacher’s score on their chosen videos and on their unchosen videos were correlated at 1. They were perfectly correlated. Hoffman asks, “What do you think? Do you think that just maybe, perhaps, it’s possible we aught to consider, I’m just throwing out the possibility that it might be that the procedure for correcting measurement error might, you now, be a little too strong?” I don’t know exactly what’s happening here, but it might be something that I’ve seen on occasion when fitting multilevel models using a point estimate for the group-level variance. It goes like this: measurement-error models are multilevel models, they involve the estimation of a distribution of a latent variable. When fitting multilevel models, it is possible to estimate the group-level variance to be zero, even though the group-level varia
6 0.14831406 772 andrew gelman stats-2011-06-17-Graphical tools for understanding multilevel models
7 0.14228885 653 andrew gelman stats-2011-04-08-Multilevel regression with shrinkage for “fixed” effects
9 0.12933406 255 andrew gelman stats-2010-09-04-How does multilevel modeling affect the estimate of the grand mean?
10 0.12447056 2033 andrew gelman stats-2013-09-23-More on Bayesian methods and multilevel modeling
11 0.1238957 1966 andrew gelman stats-2013-08-03-Uncertainty in parameter estimates using multilevel models
12 0.12119388 779 andrew gelman stats-2011-06-25-Avoiding boundary estimates using a prior distribution as regularization
13 0.11851668 397 andrew gelman stats-2010-11-06-Multilevel quantile regression
14 0.11771498 1445 andrew gelman stats-2012-08-06-Slow progress
15 0.11618447 1425 andrew gelman stats-2012-07-23-Examples of the use of hierarchical modeling to generalize to new settings
17 0.11496706 704 andrew gelman stats-2011-05-10-Multiple imputation and multilevel analysis
18 0.11495634 1900 andrew gelman stats-2013-06-15-Exploratory multilevel analysis when group-level variables are of importance
19 0.11227782 77 andrew gelman stats-2010-06-09-Sof[t]
20 0.11141174 781 andrew gelman stats-2011-06-28-The holes in my philosophy of Bayesian data analysis
topicId topicWeight
[(0, 0.172), (1, 0.149), (2, 0.046), (3, 0.002), (4, 0.087), (5, 0.023), (6, 0.025), (7, -0.031), (8, 0.03), (9, 0.127), (10, 0.027), (11, 0.004), (12, 0.063), (13, -0.022), (14, 0.064), (15, -0.012), (16, -0.079), (17, -0.021), (18, 0.033), (19, -0.011), (20, 0.003), (21, 0.023), (22, -0.004), (23, 0.027), (24, -0.046), (25, -0.127), (26, -0.021), (27, -0.033), (28, -0.023), (29, 0.021), (30, -0.017), (31, -0.03), (32, 0.022), (33, 0.009), (34, 0.001), (35, 0.023), (36, -0.009), (37, 0.023), (38, 0.048), (39, 0.044), (40, -0.002), (41, -0.001), (42, 0.021), (43, -0.055), (44, -0.064), (45, 0.042), (46, 0.023), (47, -0.007), (48, -0.04), (49, -0.058)]
simIndex simValue blogId blogTitle
same-blog 1 0.95719367 2294 andrew gelman stats-2014-04-17-If you get to the point of asking, just do it. But some difficulties do arise . . .
Introduction: Nelson Villoria writes: I find the multilevel approach very useful for a problem I am dealing with, and I was wondering whether you could point me to some references about poolability tests for multilevel models. I am working with time series of cross sectional data and I want to test whether the data supports cross sectional and/or time pooling. In a standard panel data setting I do this with Chow tests and/or CUSUM. Are these ideas directly transferable to the multilevel setting? My reply: I think you should do partial pooling. Once the question arises, just do it. Other models are just special cases. I don’t see the need for any test. That said, if you do a group-level model, you need to consider including group-level averages of individual predictors (see here ). And if the number of groups is small, there can be real gains from using an informative prior distribution on the hierarchical variance parameters. This is something that Jennifer and I do not discuss in our
2 0.87213677 1248 andrew gelman stats-2012-04-06-17 groups, 6 group-level predictors: What to do?
Introduction: Yi-Chun Ou writes: I am using a multilevel model with three levels. I read that you wrote a book about multilevel models, and wonder if you can solve the following question. The data structure is like this: Level one: customer (8444 customers) Level two: companys (90 companies) Level three: industry (17 industries) I use 6 level-three variables (i.e. industry characteristics) to explain the variance of the level-one effect across industries. The question here is whether there is an over-fitting problem since there are only 17 industries. I understand that this must be a problem for non-multilevel models, but is it also a problem for multilevel models? My reply: Yes, this could be a problem. I’d suggest combining some of your variables into a common score, or using only some of the variables, or using strong priors to control the inferences. This is an interesting and important area of statistics research, to do this sort of thing systematically. There’s lots o
3 0.82624578 704 andrew gelman stats-2011-05-10-Multiple imputation and multilevel analysis
Introduction: Robert Birkelbach: I am writing my Bachelor Thesis in which I want to assess the reading competencies of German elementary school children using the PIRLS2006 data. My levels are classrooms and the individuals. However, my dependent variable is a multiple imputed (m=5) reading test. The problem I have is, that I do not know, whether I can just calculate 5 linear multilevel models and then average all the results (the coefficients, standard deviation, bic, intra class correlation, R2, t-statistics, p-values etc) or if I need different formulas for integrating the results of the five models into one because it is a multilevel analysis? Do you think there’s a better way in solving my problem? I would greatly appreciate if you could help me with a problem regarding my analysis — I am quite a newbie to multilevel modeling and especially to multiple imputation. Also: Is it okay to use frequentist models when the multiple imputation was done bayesian? Would the different philosophies of sc
4 0.80623788 653 andrew gelman stats-2011-04-08-Multilevel regression with shrinkage for “fixed” effects
Introduction: Dean Eckles writes: I remember reading on your blog that you were working on some tools to fit multilevel models that also include “fixed” effects — such as continuous predictors — that are also estimated with shrinkage (for example, an L1 or L2 penalty). Any new developments on this front? I often find myself wanting to fit a multilevel model to some data, but also needing to include a number of “fixed” effects, mainly continuous variables. This makes me wary of overfitting to these predictors, so then I’d want to use some kind of shrinkage. As far as I can tell, the main options for doing this now is by going fully Bayesian and using a Gibbs sampler. With MCMCglmm or BUGS/JAGS I could just specify a prior on the fixed effects that corresponds to a desired penalty. However, this is pretty slow, especially with a large data set and because I’d like to select the penalty parameter by cross-validation (which is where this isn’t very Bayesian I guess?). My reply: We allow info
5 0.7663362 948 andrew gelman stats-2011-10-10-Combining data from many sources
Introduction: Mark Grote writes: I’d like to request general feedback and references for a problem of combining disparate data sources in a regression model. We’d like to model log crop yield as a function of environmental predictors, but the observations come from many data sources and are peculiarly structured. Among the issues are: 1. Measurement precision in predictors and outcome varies widely with data sources. Some observations are in very coarse units of measurement, due to rounding or even observer guesswork. 2. There are obvious clusters of observations arising from studies in which crop yields were monitored over successive years in spatially proximate communities. Thus some variables may be constant within clusters–this is true even for log yield, probably due to rounding of similar yields. 3. Cluster size and intra-cluster association structure (temporal, spatial or both) vary widely across the dataset. My [Grote's] intuition is that we can learn about central tendency
6 0.76629251 295 andrew gelman stats-2010-09-25-Clusters with very small numbers of observations
7 0.7660858 753 andrew gelman stats-2011-06-09-Allowing interaction terms to vary
8 0.75884986 1194 andrew gelman stats-2012-03-04-Multilevel modeling even when you’re not interested in predictions for new groups
9 0.75301093 2296 andrew gelman stats-2014-04-19-Index or indicator variables
10 0.74477226 397 andrew gelman stats-2010-11-06-Multilevel quantile regression
12 0.7388258 1966 andrew gelman stats-2013-08-03-Uncertainty in parameter estimates using multilevel models
13 0.73304921 2086 andrew gelman stats-2013-11-03-How best to compare effects measured in two different time periods?
14 0.7290315 1737 andrew gelman stats-2013-02-25-Correlation of 1 . . . too good to be true?
15 0.72499001 772 andrew gelman stats-2011-06-17-Graphical tools for understanding multilevel models
17 0.7150386 383 andrew gelman stats-2010-10-31-Analyzing the entire population rather than a sample
18 0.71327364 2033 andrew gelman stats-2013-09-23-More on Bayesian methods and multilevel modeling
19 0.71311438 25 andrew gelman stats-2010-05-10-Two great tastes that taste great together
20 0.7119416 850 andrew gelman stats-2011-08-11-Understanding how estimates change when you move to a multilevel model
topicId topicWeight
[(2, 0.085), (3, 0.03), (21, 0.023), (24, 0.146), (63, 0.076), (89, 0.118), (99, 0.399)]
simIndex simValue blogId blogTitle
same-blog 1 0.97575796 2294 andrew gelman stats-2014-04-17-If you get to the point of asking, just do it. But some difficulties do arise . . .
Introduction: Nelson Villoria writes: I find the multilevel approach very useful for a problem I am dealing with, and I was wondering whether you could point me to some references about poolability tests for multilevel models. I am working with time series of cross sectional data and I want to test whether the data supports cross sectional and/or time pooling. In a standard panel data setting I do this with Chow tests and/or CUSUM. Are these ideas directly transferable to the multilevel setting? My reply: I think you should do partial pooling. Once the question arises, just do it. Other models are just special cases. I don’t see the need for any test. That said, if you do a group-level model, you need to consider including group-level averages of individual predictors (see here ). And if the number of groups is small, there can be real gains from using an informative prior distribution on the hierarchical variance parameters. This is something that Jennifer and I do not discuss in our
2 0.96308434 850 andrew gelman stats-2011-08-11-Understanding how estimates change when you move to a multilevel model
Introduction: Ramu Sudhagoni writes: I am working on combining three longitudinal studies using Bayesian hierarchical technique. In each study, I have at least 70 subjects follow up on 5 different visit months. My model consists of 10 different covariates including longitudinal and cross-sectional effects. Mixed models are used to fit the three studies individually using Bayesian approach and I noticed that few covariates were significant. When I combined using three level hierarchical approach, all the covariates became non-significant at the population level, and large estimates were found for variance parameters at the population level. I am struggling to understand why I am getting large variances at population level and wider credible intervals. I assumed non-informative normal priors for all my cross sectional and longitudinal effects, and non-informative inverse-gamma priors for variance parameters. I followed the approach explained by Inoue et al. (Title: Combining Longitudinal Studie
3 0.9607383 1628 andrew gelman stats-2012-12-17-Statistics in a world where nothing is random
Introduction: Rama Ganesan writes: I think I am having an existential crisis. I used to work with animals (rats, mice, gerbils etc.) Then I started to work in marketing research where we did have some kind of random sampling procedure. So up until a few years ago, I was sort of okay. Now I am teaching marketing research, and I feel like there is no real random sampling anymore. I take pains to get students to understand what random means, and then the whole lot of inferential statistics. Then almost anything they do – the sample is not random. They think I am contradicting myself. They use convenience samples at every turn – for their school work, and the enormous amount on online surveying that gets done. Do you have any suggestions for me? Other than say, something like this . My reply: Statistics does not require randomness. The three essential elements of statistics are measurement, comparison, and variation. Randomness is one way to supply variation, and it’s one way to model
4 0.95941222 566 andrew gelman stats-2011-02-09-The boxer, the wrestler, and the coin flip, again
Introduction: Mike Grosskopf writes: I came across your blog the other day and noticed your paper about “The Boxer, the Wrestler, and the Coin Flip” . . . I do not understand the objection to the robust Bayesian inference for conditioning on X=Y in the problem as you describe in the paper. The paper talks about how using Robust Bayes when conditioning on X=Y “degrades our inference about the coin flip” and “has led us to the claim that we can say nothing at all about the coin flip”. Does that have to be the case however, because while conditioning on X=Y does mean that p({X=1}|{X=Y}I) = p({Y=1}|{X=Y}I), I don’t see why it has to mean that both have the same π-distribution where Pr(Y = 1) = π. Which type of inference is being done about Y in the problem? If you are trying to make an inference on the results of the fight between the boxer and the wrestler that has already happened, in which your friend tells you that either the boxer won and he flipped heads with a coin or the boxer lost a
5 0.95190907 231 andrew gelman stats-2010-08-24-Yet another Bayesian job opportunity
Introduction: Steve Cohen writes: My [Cohen's] firm is looking for strong candidates to help us in developing software and analyzing data using Bayesian methods. We have been developing a suite of programs in C++ which allow us to do Bayesian hierarchical regression and logit/probit models on marketing data. These efforts have included the use of high performance computing tools like nVidia’s CUDA and the new OpenCL standard, which allow parallel processing of Bayesian models. Our software is very, very fast – even on databases that are ½ terabyte in size. The software still needs many additions and improvements and a person with the right skill set will have the chance to make a significant contribution. Here’s the job description he sent: Bayesian statistician and C++ programmer The company In4mation Insights is a marketing research, analytics, and consulting firm which operates on the leading-edge of our industry. Our clients are Fortune 500 companies and major management consul
6 0.95170784 1783 andrew gelman stats-2013-03-31-He’s getting ready to write a book
7 0.95005202 1196 andrew gelman stats-2012-03-04-Piss-poor monocausal social science
9 0.9467178 1173 andrew gelman stats-2012-02-17-Sports examples in class
10 0.94270861 1506 andrew gelman stats-2012-09-21-Building a regression model . . . with only 27 data points
11 0.94257635 1320 andrew gelman stats-2012-05-14-Question 4 of my final exam for Design and Analysis of Sample Surveys
12 0.94242442 27 andrew gelman stats-2010-05-11-Update on the spam email study
14 0.94033277 2110 andrew gelman stats-2013-11-22-A Bayesian model for an increasing function, in Stan!
15 0.94029003 623 andrew gelman stats-2011-03-21-Baseball’s greatest fielders
16 0.94027972 2088 andrew gelman stats-2013-11-04-Recently in the sister blog
17 0.93939221 1597 andrew gelman stats-2012-11-29-What is expected of a consultant
18 0.93929648 1855 andrew gelman stats-2013-05-13-Stan!
20 0.93841314 1690 andrew gelman stats-2013-01-23-When are complicated models helpful in psychology research and when are they overkill?