andrew_gelman_stats andrew_gelman_stats-2013 andrew_gelman_stats-2013-1934 knowledge-graph by maker-knowledge-mining
Source: html
Introduction: A sociologist writes in: Samuel Lucas has just published a paper in Quality and Quantity arguing that anything less than a full probability sample of higher levels in HLMs yields biased and unusable results. If I follow him correctly, he is arguing that not only are the SEs too small, but the parameter estimates themselves are biased and we cannot say in advance whether the bias is positive or negative. Lucas has thrown down a big gauntlet, advising us throw away our data unless the sample of macro units is right and ignore the published results that fail this standard. Extreme. Is there another conclusion to be drawn? Other advice to be given? A Bayesian path out of the valley? Heres’s the abstract to Lucas’s paper: The multilevel model has become a staple of social research. I textually and formally explicate sample design features that, I contend, are required for unbiased estimation of macro-level multilevel model parameters and the use of tools for statistical infe
sentIndex sentText sentNum sentScore
1 A sociologist writes in: Samuel Lucas has just published a paper in Quality and Quantity arguing that anything less than a full probability sample of higher levels in HLMs yields biased and unusable results. [sent-1, score-0.495]
2 Lucas has thrown down a big gauntlet, advising us throw away our data unless the sample of macro units is right and ignore the published results that fail this standard. [sent-3, score-0.453]
3 Heres’s the abstract to Lucas’s paper: The multilevel model has become a staple of social research. [sent-8, score-0.504]
4 I textually and formally explicate sample design features that, I contend, are required for unbiased estimation of macro-level multilevel model parameters and the use of tools for statistical inference, such as standard errors. [sent-9, score-0.822]
5 After detailing the limited and conflicting guidance on sample design in the multilevel model didactic literature, illustrative nationally-representative datasets and published examples that violate the posited requirements are identified. [sent-10, score-1.554]
6 Because the didactic literature is either silent on sample design requirements or in disagreement with the constraints posited here, two Monte Carlo simulations are conducted to clarify the issues. [sent-11, score-0.759]
7 The results indicate that bias follows use of samples that fail to satisfy the requirements outlined; notably, the bias is poorly-behaved, such that estimates provide neither upper nor lower bounds for the population parameter. [sent-12, score-1.008]
8 Thus, published multilevel model analyses using many workhorse datasets, including NELS, AdHealth, NLSY, GSS, PSID, and SIPP, often unwittingly convey substantive results and theoretical conclusions that lack foundation. [sent-14, score-0.706]
9 Future research using the multilevel model should be limited to cases that satisfy the sample requirements described. [sent-15, score-1.126]
10 Just as we can fit single-level regressions to data that are not random samples, we can fit multilevel models to data that are not two-stage random samples. [sent-18, score-0.696]
11 Ultimately we are interested in generalizing to a larger population, so if our data are not simple random samples, we need to account for this, a concern that I and others address using multilevel modeling and poststratification; see, for example, my recent paper with Yair. [sent-19, score-0.904]
12 But this is not a problem unique to multilevel models. [sent-20, score-0.352]
13 Hree’s what Lucas writes: I contend, some datasets on which the MLM has been estimated are non-probability samples for the MLM. [sent-22, score-0.357]
14 Further, this circumstance may not be rare; the processes transforming probability samples into problematic samples for the MLM may be inconspicuous but widespread. [sent-26, score-0.644]
15 So, yes, I think Lucas is correct to be concerned about generalizing from sample to population (what Lucas calls “bias”); it’s a huge issue in psychology and medical studies performed on volunteers or unrepresentative samples. [sent-29, score-0.627]
16 But I don’t wee anything specially problematic about multilevel models, especially if the researcher takes the next step and does poststratification (which is, essentially, regression adjustment) to correct for differences between sample and population. [sent-30, score-0.944]
17 If the data are crap , it’ll be hard to trust anything that comes out of your analysis, but multilevel modeling won’t be making things any worse. [sent-31, score-0.544]
18 On the contrary: multilevel analysis is a way to model bias and variation. [sent-32, score-0.582]
19 So in that sense I strongly disagree with Lucas, who writes, “Future research using the multilevel model should be limited to cases that satisfy the sample requirements described. [sent-34, score-1.126]
20 ” Random samples are great, and I admire Lucas’s thoughtful skepticism, but when we want to analyze data that are not random samples, I think it’s better to face up to the statistical difficulties and model them directly rather than running away from the problem. [sent-35, score-0.517]
wordName wordTfidf (topN-words)
[('lucas', 0.46), ('multilevel', 0.352), ('samples', 0.25), ('mlm', 0.242), ('requirements', 0.216), ('generalizing', 0.184), ('sample', 0.184), ('bias', 0.135), ('didactic', 0.133), ('satisfy', 0.132), ('contend', 0.121), ('random', 0.115), ('datasets', 0.107), ('biased', 0.107), ('posited', 0.097), ('limited', 0.097), ('model', 0.095), ('published', 0.087), ('problematic', 0.087), ('modeling', 0.084), ('poststratification', 0.079), ('design', 0.074), ('fail', 0.072), ('correct', 0.072), ('population', 0.068), ('concerned', 0.068), ('arguing', 0.066), ('concern', 0.062), ('workhorse', 0.061), ('unnecessarily', 0.061), ('unwittingly', 0.061), ('hlms', 0.061), ('specially', 0.061), ('tools', 0.06), ('regression', 0.058), ('data', 0.057), ('explicate', 0.057), ('staple', 0.057), ('circumstance', 0.057), ('illustrative', 0.057), ('outlined', 0.055), ('silent', 0.055), ('detailing', 0.055), ('samuel', 0.053), ('advising', 0.053), ('anything', 0.051), ('unrepresentative', 0.051), ('ses', 0.051), ('knowledge', 0.051), ('using', 0.05)]
simIndex simValue blogId blogTitle
Introduction: A sociologist writes in: Samuel Lucas has just published a paper in Quality and Quantity arguing that anything less than a full probability sample of higher levels in HLMs yields biased and unusable results. If I follow him correctly, he is arguing that not only are the SEs too small, but the parameter estimates themselves are biased and we cannot say in advance whether the bias is positive or negative. Lucas has thrown down a big gauntlet, advising us throw away our data unless the sample of macro units is right and ignore the published results that fail this standard. Extreme. Is there another conclusion to be drawn? Other advice to be given? A Bayesian path out of the valley? Heres’s the abstract to Lucas’s paper: The multilevel model has become a staple of social research. I textually and formally explicate sample design features that, I contend, are required for unbiased estimation of macro-level multilevel model parameters and the use of tools for statistical infe
2 0.21835633 383 andrew gelman stats-2010-10-31-Analyzing the entire population rather than a sample
Introduction: Lee Mobley writes: I recently read what you posted on your blog How does statistical analysis differ when analyzing the entire population rather than a sample? What you said in the blog accords with my training in econometrics. However I am concerned about a new wrinkle on this problem that derives from multilevel modeling. We are analyzing multilevel models of the probability of using cancer screening for the entire Medicare population. I argue that every state has different systems in place (politics, cancer control efforts, culture, insurance regulations, etc) so that essentially a different probability generating mechanism is in place for each state. Thus I estimate 50 separate regressions for the populations in each state, and then note and map the variability in the effect estimates (slope parameters) for each covariate. Reviewers argue that I should be using random slopes modeling, pooling all individuals in all states together. I am familiar with this approach
3 0.19979618 295 andrew gelman stats-2010-09-25-Clusters with very small numbers of observations
Introduction: James O’Brien writes: How would you explain, to a “classically-trained” hypothesis-tester, that “It’s OK to fit a multilevel model even if some groups have only one observation each”? I [O'Brien] think I understand the logic and the statistical principles at work in this, but I’ve having trouble being clear and persuasive. I also feel like I’m contending with some methodological conventional wisdom here. My reply: I’m so used to this idea that I find it difficult to defend it in some sort of general conceptual way. So let me retreat to a more functional defense, which is that multilevel modeling gives good estimates, especially when the number of observations per group is small. One way to see this in any particular example in through cross-validation. Another way is to consider the alternatives. If you try really hard you can come up with a “classical hypothesis testing” approach which will do as well as the multilevel model. It would just take a lot of work. I’d r
4 0.16309237 2033 andrew gelman stats-2013-09-23-More on Bayesian methods and multilevel modeling
Introduction: Ban Chuan Cheah writes: In a previous post, http://andrewgelman.com/2013/07/30/the-roy-causal-model/ you pointed to a paper on Bayesian methods by Heckman. At around the same time I came across another one of his papers, “The Effects of Cognitive and Noncognitive Abilities on Labor Market Outcomes and Social Behavior (2006)” (http://www.nber.org/papers/w12006 or published version http://www.jstor.org/stable/10.1086/504455). In this paper they implement their model as follows: We use Bayesian Markov chain Monte Carlo methods to compute the sample likelihood. Our use of Bayesian methods is only a computational convenience. Our identification analysis is strictly classical. Under our assumptions, the priors we use are asymptotically irrelevant. Some of the authors have also done something similar earlier in: Hansen, Karsten T. & Heckman, James J. & Mullen, K.J.Kathleen J., 2004. “The effect of schooling and ability on achievement test scores,” Journal of Econometrics, Elsevi
Introduction: Following up on our recent discussion of visually-weighted displays of uncertainty in regression curves, Lucas Leeman sent in the following two graphs: First, the basic spaghetti-style plot showing inferential uncertainty in the E(y|x) curve: Then, a version using even lighter intensities for the lines that go further from the center, to further de-emphasize the edges: P.S. More (including code!) here .
6 0.15054594 1248 andrew gelman stats-2012-04-06-17 groups, 6 group-level predictors: What to do?
7 0.14631937 352 andrew gelman stats-2010-10-19-Analysis of survey data: Design based models vs. hierarchical modeling?
8 0.14230445 772 andrew gelman stats-2011-06-17-Graphical tools for understanding multilevel models
9 0.13997047 1878 andrew gelman stats-2013-05-31-How to fix the tabloids? Toward replicable social science research
10 0.1390935 1289 andrew gelman stats-2012-04-29-We go to war with the data we have, not the data we want
12 0.13418399 77 andrew gelman stats-2010-06-09-Sof[t]
13 0.13203336 397 andrew gelman stats-2010-11-06-Multilevel quantile regression
14 0.1317693 1900 andrew gelman stats-2013-06-15-Exploratory multilevel analysis when group-level variables are of importance
15 0.13118535 704 andrew gelman stats-2011-05-10-Multiple imputation and multilevel analysis
16 0.13118072 1737 andrew gelman stats-2013-02-25-Correlation of 1 . . . too good to be true?
17 0.12996925 1144 andrew gelman stats-2012-01-29-How many parameters are in a multilevel model?
19 0.12412185 1691 andrew gelman stats-2013-01-25-Extreem p-values!
20 0.12249402 820 andrew gelman stats-2011-07-25-Design of nonrandomized cluster sample study
topicId topicWeight
[(0, 0.203), (1, 0.131), (2, 0.064), (3, -0.11), (4, 0.064), (5, 0.046), (6, -0.062), (7, -0.07), (8, 0.076), (9, 0.058), (10, 0.035), (11, -0.021), (12, -0.006), (13, 0.025), (14, 0.004), (15, -0.013), (16, -0.061), (17, -0.012), (18, 0.011), (19, 0.002), (20, -0.01), (21, -0.009), (22, -0.008), (23, 0.038), (24, -0.09), (25, -0.096), (26, -0.048), (27, 0.017), (28, 0.004), (29, 0.038), (30, -0.03), (31, -0.038), (32, -0.02), (33, 0.012), (34, -0.04), (35, 0.03), (36, -0.022), (37, 0.011), (38, 0.031), (39, 0.006), (40, 0.02), (41, 0.003), (42, 0.003), (43, -0.075), (44, 0.005), (45, -0.003), (46, 0.029), (47, 0.071), (48, -0.061), (49, -0.034)]
simIndex simValue blogId blogTitle
same-blog 1 0.96993983 1934 andrew gelman stats-2013-07-11-Yes, worry about generalizing from data to population. But multilevel modeling is the solution, not the problem
Introduction: A sociologist writes in: Samuel Lucas has just published a paper in Quality and Quantity arguing that anything less than a full probability sample of higher levels in HLMs yields biased and unusable results. If I follow him correctly, he is arguing that not only are the SEs too small, but the parameter estimates themselves are biased and we cannot say in advance whether the bias is positive or negative. Lucas has thrown down a big gauntlet, advising us throw away our data unless the sample of macro units is right and ignore the published results that fail this standard. Extreme. Is there another conclusion to be drawn? Other advice to be given? A Bayesian path out of the valley? Heres’s the abstract to Lucas’s paper: The multilevel model has become a staple of social research. I textually and formally explicate sample design features that, I contend, are required for unbiased estimation of macro-level multilevel model parameters and the use of tools for statistical infe
2 0.87398285 383 andrew gelman stats-2010-10-31-Analyzing the entire population rather than a sample
Introduction: Lee Mobley writes: I recently read what you posted on your blog How does statistical analysis differ when analyzing the entire population rather than a sample? What you said in the blog accords with my training in econometrics. However I am concerned about a new wrinkle on this problem that derives from multilevel modeling. We are analyzing multilevel models of the probability of using cancer screening for the entire Medicare population. I argue that every state has different systems in place (politics, cancer control efforts, culture, insurance regulations, etc) so that essentially a different probability generating mechanism is in place for each state. Thus I estimate 50 separate regressions for the populations in each state, and then note and map the variability in the effect estimates (slope parameters) for each covariate. Reviewers argue that I should be using random slopes modeling, pooling all individuals in all states together. I am familiar with this approach
3 0.80535209 948 andrew gelman stats-2011-10-10-Combining data from many sources
Introduction: Mark Grote writes: I’d like to request general feedback and references for a problem of combining disparate data sources in a regression model. We’d like to model log crop yield as a function of environmental predictors, but the observations come from many data sources and are peculiarly structured. Among the issues are: 1. Measurement precision in predictors and outcome varies widely with data sources. Some observations are in very coarse units of measurement, due to rounding or even observer guesswork. 2. There are obvious clusters of observations arising from studies in which crop yields were monitored over successive years in spatially proximate communities. Thus some variables may be constant within clusters–this is true even for log yield, probably due to rounding of similar yields. 3. Cluster size and intra-cluster association structure (temporal, spatial or both) vary widely across the dataset. My [Grote's] intuition is that we can learn about central tendency
4 0.79259312 1248 andrew gelman stats-2012-04-06-17 groups, 6 group-level predictors: What to do?
Introduction: Yi-Chun Ou writes: I am using a multilevel model with three levels. I read that you wrote a book about multilevel models, and wonder if you can solve the following question. The data structure is like this: Level one: customer (8444 customers) Level two: companys (90 companies) Level three: industry (17 industries) I use 6 level-three variables (i.e. industry characteristics) to explain the variance of the level-one effect across industries. The question here is whether there is an over-fitting problem since there are only 17 industries. I understand that this must be a problem for non-multilevel models, but is it also a problem for multilevel models? My reply: Yes, this could be a problem. I’d suggest combining some of your variables into a common score, or using only some of the variables, or using strong priors to control the inferences. This is an interesting and important area of statistics research, to do this sort of thing systematically. There’s lots o
5 0.77962363 1194 andrew gelman stats-2012-03-04-Multilevel modeling even when you’re not interested in predictions for new groups
Introduction: Fred Wu writes: I work at National Prescribing Services in Australia. I have a database representing say, antidiabetic drug utilisation for the entire Australia in the past few years. I planned to do a longitudinal analysis across GP Division Network (112 divisions in AUS) using mixed-effects models (or as you called in your book varying intercept and varying slope) on this data. The problem here is: as data actually represent the population who use antidiabetic drugs in AUS, should I use 112 fixed dummy variables to capture the random variations or use varying intercept and varying slope for the model ? Because some one may aruge, like divisions in AUS or states in USA can hardly be considered from a “superpopulation”, then fixed dummies should be used. What I think is the population are those who use the drugs, what will happen when the rest need to use them? In terms of exchangeability, using varying intercept and varying slopes can be justified. Also you provided in y
6 0.76634991 295 andrew gelman stats-2010-09-25-Clusters with very small numbers of observations
7 0.76014149 704 andrew gelman stats-2011-05-10-Multiple imputation and multilevel analysis
9 0.74788767 77 andrew gelman stats-2010-06-09-Sof[t]
10 0.7460922 1267 andrew gelman stats-2012-04-17-Hierarchical-multilevel modeling with “big data”
11 0.74424744 2086 andrew gelman stats-2013-11-03-How best to compare effects measured in two different time periods?
12 0.73888147 1468 andrew gelman stats-2012-08-24-Multilevel modeling and instrumental variables
13 0.73688847 352 andrew gelman stats-2010-10-19-Analysis of survey data: Design based models vs. hierarchical modeling?
14 0.73352778 1814 andrew gelman stats-2013-04-20-A mess with which I am comfortable
15 0.73087299 772 andrew gelman stats-2011-06-17-Graphical tools for understanding multilevel models
16 0.72644413 1900 andrew gelman stats-2013-06-15-Exploratory multilevel analysis when group-level variables are of importance
17 0.72606277 1383 andrew gelman stats-2012-06-18-Hierarchical modeling as a framework for extrapolation
18 0.72566903 1294 andrew gelman stats-2012-05-01-Modeling y = a + b + c
19 0.71848065 269 andrew gelman stats-2010-09-10-R vs. Stata, or, Different ways to estimate multilevel models
20 0.71098542 1425 andrew gelman stats-2012-07-23-Examples of the use of hierarchical modeling to generalize to new settings
topicId topicWeight
[(13, 0.012), (15, 0.019), (16, 0.071), (21, 0.021), (24, 0.137), (59, 0.015), (84, 0.036), (86, 0.016), (89, 0.036), (90, 0.128), (95, 0.014), (99, 0.344)]
simIndex simValue blogId blogTitle
1 0.98253703 1655 andrew gelman stats-2013-01-05-The statistics software signal
Introduction: Tyler Cowen links to a post by Sean Taylor, who writes the following about users of R: You are willing to invest in learning something difficult. You do not care about aesthetics, only availability of packages and getting results quickly. To me, R is easy and Sas is difficult. I once worked with some students who were running Sas and the output was unreadable! Pages and pages of numbers that made no sense. When it comes to ease or difficulty of use, I think it depends on what you’re used to! And I really don’t understand the bit about aesthetics. What about this ? One reason I use R is to make pretty graphs. That said, if I’d never learned R, I’d just be making pretty graphs in Fortran or whatever. My guess is, the way I program, R is actually hindering rather than helping my ability to make attractive graphs. Half the time I’m scrambling around, writing custom code to get around R’s defaults.
2 0.97864008 1417 andrew gelman stats-2012-07-15-Some decision analysis problems are pretty easy, no?
Introduction: Cassie Murdoch reports : A 47-year-old woman in Uxbridge, Massachusetts, got behind the wheel of her car after having a bit too much to drink, but instead of wreaking havoc on the road, she ended up lodged in a sand trap at a local golf course. Why? Because her GPS made her do it—obviously! She said the GPS told her to turn left, and she did, right into a cornfield. That didn’t faze her, and she just kept on going until she ended up on the golf course and got stuck in the sand. There were people on the course at the time, but thankfully nobody was injured. Police found a cup full of alcohol in her car and arrested her for driving drunk. Here’s the punchline: This is the fourth time she’s been arrested for a DUI. Assuming this story is accurate, I guess they don’t have one of those “three strikes” laws in Massachusetts? Personally, I’m a lot more afraid of a dangerous driver than of some drug dealer. I’d think a simple cost-benefit calculation would recommend taking away
3 0.97686082 1947 andrew gelman stats-2013-07-20-We are what we are studying
Introduction: Anthropologist Marshall Sahlins writes : When native Australians or New Guineans say that their totemic animals and plants are their kinsmen – that these species are persons like themselves, and that in offering them to others they are giving away part of their own substance – we have to take them seriously, which is to say empirically, if we want to understand the large consequences of these facts for how they organise their lives. The graveyard of ethnographic studies is strewn with the remains of reports which, thanks to anthropologists’ own presuppositions as to what constitutes empirical fact, were content to ignore or debunk the Amazonian peoples who said that the animals they hunted were their brothers-in-law, the Africans who described the way they systematically killed their kings when they became weak, or the Fijian chiefs who claimed they were gods. My first thought was . . . wait a minute! Whazzat with “presuppositions as to what constitutes empirical fact”? That a
4 0.97388095 1411 andrew gelman stats-2012-07-10-Defining ourselves arbitrarily
Introduction: Robin Hanson writes that he does’t use slang: I [Hanson] am not into slang. I want to talk to the widest possible audience, and to focus on timeless issues and insights, as opposed to the latest fashionable topics. I can see why people want to signal loyalty to their groups, especially in the military, but I have little confidence that this is good for the world as a whole. I don’t know anything about the military (I don’t think this really counts) so I can’t comment on that part, and I don’t see the opposition between slang and “timeless issues and insights, as opposed to the latest fashionable topics” (after all, Mark Twain used slang and he had some timeless insights), but I’d like to pick up on a slightly different angle here, which is the set of quasi-arbitrary choices we make in order to define ourselves. Robin Hanson happens not to use much slang and he uses this trait to define himself, not quite to stand out in the crowd but to put himself on one end of a scale. I
same-blog 5 0.97241092 1934 andrew gelman stats-2013-07-11-Yes, worry about generalizing from data to population. But multilevel modeling is the solution, not the problem
Introduction: A sociologist writes in: Samuel Lucas has just published a paper in Quality and Quantity arguing that anything less than a full probability sample of higher levels in HLMs yields biased and unusable results. If I follow him correctly, he is arguing that not only are the SEs too small, but the parameter estimates themselves are biased and we cannot say in advance whether the bias is positive or negative. Lucas has thrown down a big gauntlet, advising us throw away our data unless the sample of macro units is right and ignore the published results that fail this standard. Extreme. Is there another conclusion to be drawn? Other advice to be given? A Bayesian path out of the valley? Heres’s the abstract to Lucas’s paper: The multilevel model has become a staple of social research. I textually and formally explicate sample design features that, I contend, are required for unbiased estimation of macro-level multilevel model parameters and the use of tools for statistical infe
6 0.96851408 478 andrew gelman stats-2010-12-20-More on why “all politics is local” is an outdated slogan
7 0.96565044 15 andrew gelman stats-2010-05-03-Public Opinion on Health Care Reform
8 0.96389943 766 andrew gelman stats-2011-06-14-Last Wegman post (for now)
11 0.95834118 1163 andrew gelman stats-2012-02-12-Meta-analysis, game theory, and incentives to do replicable research
12 0.94987214 475 andrew gelman stats-2010-12-19-All politics are local — not
13 0.94853842 508 andrew gelman stats-2011-01-08-More evidence of growing nationalization of congressional elections
14 0.94733179 630 andrew gelman stats-2011-03-27-What is an economic “conspiracy theory”?
15 0.94689685 1732 andrew gelman stats-2013-02-22-Evaluating the impacts of welfare reform?
16 0.94680214 1960 andrew gelman stats-2013-07-28-More on that machine learning course
17 0.94417995 250 andrew gelman stats-2010-09-02-Blending results from two relatively independent multi-level models
18 0.94306535 131 andrew gelman stats-2010-07-07-A note to John
19 0.94199252 187 andrew gelman stats-2010-08-05-Update on state size and governors’ popularity
20 0.94196975 1597 andrew gelman stats-2012-11-29-What is expected of a consultant