andrew_gelman_stats andrew_gelman_stats-2012 andrew_gelman_stats-2012-1383 knowledge-graph by maker-knowledge-mining
Source: html
Introduction: Phil recently posted on the challenge of extrapolation of inferences to new data. After telling the story of a colleague who flat-out refused to make predictions from his model of buildings to new data, Phil wrote, “This is an interesting problem because it is sort of outside the realm of statistics, and into some sort of meta-statistical area. How can you judge whether your results can be extrapolated to the ‘real world,’ if you cant get a real-world sample to compare to?” In reply, I wrote: I agree that this is an important and general problem, but I don’t think it is outside the realm of statistics! I think that one useful statistical framework here is multilevel modeling. Suppose you are applying a procedure to J cases and want to predict case J+1 (in this case, the cases are buildings and J=52). Let the parameters be theta_1,…,theta_{J+1}, with data y_1,…,y_{J+1}, and case-level predictors X_1,…,X_{J+1}. The question is how to generalize from (theta_1,…,theta_J) to theta_{
sentIndex sentText sentNum sentScore
1 Phil recently posted on the challenge of extrapolation of inferences to new data. [sent-1, score-0.719]
2 After telling the story of a colleague who flat-out refused to make predictions from his model of buildings to new data, Phil wrote, “This is an interesting problem because it is sort of outside the realm of statistics, and into some sort of meta-statistical area. [sent-2, score-1.435]
3 How can you judge whether your results can be extrapolated to the ‘real world,’ if you cant get a real-world sample to compare to? [sent-3, score-0.466]
4 ” In reply, I wrote: I agree that this is an important and general problem, but I don’t think it is outside the realm of statistics! [sent-4, score-0.58]
5 I think that one useful statistical framework here is multilevel modeling. [sent-5, score-0.152]
6 Suppose you are applying a procedure to J cases and want to predict case J+1 (in this case, the cases are buildings and J=52). [sent-6, score-0.978]
7 Let the parameters be theta_1,…,theta_{J+1}, with data y_1,…,y_{J+1}, and case-level predictors X_1,…,X_{J+1}. [sent-7, score-0.07]
8 The question is how to generalize from (theta_1,…,theta_J) to theta_{J+1}. [sent-8, score-0.095]
9 This can be framed in a hierarchical model in which the J cases in your training set are a sample from population 1 and your new case is drawn from population 2. [sent-9, score-1.417]
10 Now you need to model how much the thetas can vary from one population to another, but this should be possible. [sent-10, score-0.514]
11 And, as with hierarchical models in general, the more information you have in the observed X’s, the less variation you would hope to have in the thetas. [sent-12, score-0.403]
12 Unfortunately, I posted this response in the comments and it seems to have gotten lost. [sent-13, score-0.224]
13 Or so I am guessing given that the long thread that followed included very little discussion of hierarchical modeling as a framework for extrapolation. [sent-14, score-0.725]
14 Instead, there was lots of general discussion of bias, extrapolation, randomization, and statistical foundations. [sent-15, score-0.263]
15 General principles are fine but I like my above suggestion to frame the extrapolation problem as a hierarchical model because it points a way forward, linking general concerns about out-of-sample predictions to information that could be available in a specific problem. [sent-16, score-1.703]
wordName wordTfidf (topN-words)
[('extrapolation', 0.359), ('buildings', 0.324), ('hierarchical', 0.256), ('realm', 0.251), ('general', 0.19), ('population', 0.168), ('cases', 0.164), ('phil', 0.161), ('framework', 0.152), ('thetas', 0.149), ('predictions', 0.144), ('extrapolated', 0.14), ('cant', 0.14), ('outside', 0.139), ('problem', 0.13), ('posted', 0.13), ('model', 0.117), ('randomization', 0.108), ('framed', 0.101), ('sample', 0.1), ('refused', 0.099), ('frame', 0.097), ('case', 0.096), ('generalize', 0.095), ('linking', 0.095), ('gotten', 0.094), ('thread', 0.093), ('judge', 0.086), ('drawn', 0.085), ('applying', 0.084), ('suggestion', 0.082), ('training', 0.082), ('vary', 0.08), ('guessing', 0.08), ('new', 0.08), ('concerns', 0.079), ('information', 0.078), ('challenge', 0.078), ('procedure', 0.077), ('telling', 0.076), ('principles', 0.076), ('colleague', 0.075), ('wrote', 0.074), ('discussion', 0.073), ('inferences', 0.072), ('included', 0.071), ('predictors', 0.07), ('observed', 0.069), ('predict', 0.069), ('forward', 0.069)]
simIndex simValue blogId blogTitle
same-blog 1 0.99999994 1383 andrew gelman stats-2012-06-18-Hierarchical modeling as a framework for extrapolation
Introduction: Phil recently posted on the challenge of extrapolation of inferences to new data. After telling the story of a colleague who flat-out refused to make predictions from his model of buildings to new data, Phil wrote, “This is an interesting problem because it is sort of outside the realm of statistics, and into some sort of meta-statistical area. How can you judge whether your results can be extrapolated to the ‘real world,’ if you cant get a real-world sample to compare to?” In reply, I wrote: I agree that this is an important and general problem, but I don’t think it is outside the realm of statistics! I think that one useful statistical framework here is multilevel modeling. Suppose you are applying a procedure to J cases and want to predict case J+1 (in this case, the cases are buildings and J=52). Let the parameters be theta_1,…,theta_{J+1}, with data y_1,…,y_{J+1}, and case-level predictors X_1,…,X_{J+1}. The question is how to generalize from (theta_1,…,theta_J) to theta_{
2 0.30224949 1425 andrew gelman stats-2012-07-23-Examples of the use of hierarchical modeling to generalize to new settings
Introduction: In a link to our back-and-forth on causal inference and the use of hierarchical models to bridge between different inferential settings, Elias Bareinboim (a computer scientist who is working with Judea Pearl) writes : In the past week, I have been engaged in a discussion with Andrew Gelman and his blog readers regarding causal inference, selection bias, confounding, and generalizability. I was trying to understand how his method which he calls “hierarchical modeling” would handle these issues and what guarantees it provides. . . . If anyone understands how “hierarchical modeling” can solve a simple toy problem (e.g., M-bias, control of confounding, mediation, generalizability), please share with us. In his post, Bareinboim raises a direct question about hierarchical modeling and also indirectly brings up larger questions about what is convincing evidence when evaluating a statistical method. As I wrote earlier, Bareinboim believes that “The only way investigators can decide w
3 0.21995254 1289 andrew gelman stats-2012-04-29-We go to war with the data we have, not the data we want
Introduction: This post is by Phil. Psychologists perform experiments on Canadian undergraduate psychology students and draws conclusions that (they believe) apply to humans in general; they publish in Science. A drug company decides to embark on additional trials that will cost tens of millions of dollars based on the results of a careful double-blind study….whose patients are all volunteers from two hospitals. A movie studio holds 9 screenings of a new movie for volunteer viewers and, based on their survey responses, decides to spend another $8 million to re-shoot the ending. A researcher interested in the effect of ventilation on worker performance conducts a months-long study in which ventilation levels are varied and worker performance is monitored…in a single building. In almost all fields of research, most studies are based on convenience samples, or on random samples from a larger population that is itself a convenience sample. The paragraph above gives just a few examples. The benefit
Introduction: Elias Bareinboim asked what I thought about his comment on selection bias in which he referred to a paper by himself and Judea Pearl, “Controlling Selection Bias in Causal Inference.” I replied that I have no problem with what he wrote, but that from my perspective I find it easier to conceptualize such problems in terms of multilevel models. I elaborated on that point in a recent post , “Hierarchical modeling as a framework for extrapolation,” which I think was read by only a few people (I say this because it received only two comments). I don’t think Bareinboim objected to anything I wrote, but like me he is comfortable working within his own framework. He wrote the following to me: In some sense, “not ad hoc” could mean logically consistent. In other words, if one agrees with the assumptions encoded in the model, one must also agree with the conclusions entailed by these assumptions. I am not aware of any other way of doing mathematics. As it turns out, to get causa
Introduction: David Radwin asks a question which comes up fairly often in one form or another: How should one respond to requests for statistical hypothesis tests for population (or universe) data? I [Radwin] first encountered this issue as an undergraduate when a professor suggested a statistical significance test for my paper comparing roll call votes between freshman and veteran members of Congress. Later I learned that such tests apply only to samples because their purpose is to tell you whether the difference in the observed sample is likely to exist in the population. If you have data for the whole population, like all members of the 103rd House of Representatives, you do not need a test to discern the true difference in the population. Sometimes researchers assume some sort of superpopulation like “all possible Congresses” or “Congresses across all time” and that the members of any given Congress constitute a sample. In my current work in education research, it is sometimes asserted t
6 0.13773227 906 andrew gelman stats-2011-09-14-Another day, another stats postdoc
7 0.13473137 2351 andrew gelman stats-2014-05-28-Bayesian nonparametric weighted sampling inference
8 0.12275224 383 andrew gelman stats-2010-10-31-Analyzing the entire population rather than a sample
9 0.12182393 1999 andrew gelman stats-2013-08-27-Bayesian model averaging or fitting a larger model
10 0.11656293 1270 andrew gelman stats-2012-04-19-Demystifying Blup
12 0.11128396 781 andrew gelman stats-2011-06-28-The holes in my philosophy of Bayesian data analysis
13 0.10954953 1695 andrew gelman stats-2013-01-28-Economists argue about Bayes
14 0.1092233 295 andrew gelman stats-2010-09-25-Clusters with very small numbers of observations
15 0.10843398 753 andrew gelman stats-2011-06-09-Allowing interaction terms to vary
16 0.10830998 1431 andrew gelman stats-2012-07-27-Overfitting
17 0.10772454 1248 andrew gelman stats-2012-04-06-17 groups, 6 group-level predictors: What to do?
18 0.10555506 244 andrew gelman stats-2010-08-30-Useful models, model checking, and external validation: a mini-discussion
20 0.10494372 2359 andrew gelman stats-2014-06-04-All the Assumptions That Are My Life
topicId topicWeight
[(0, 0.209), (1, 0.113), (2, 0.028), (3, -0.02), (4, 0.046), (5, 0.019), (6, -0.021), (7, -0.025), (8, 0.081), (9, 0.046), (10, 0.01), (11, 0.005), (12, 0.01), (13, 0.007), (14, -0.052), (15, 0.008), (16, -0.043), (17, -0.008), (18, 0.023), (19, 0.014), (20, -0.006), (21, -0.024), (22, -0.023), (23, 0.025), (24, -0.045), (25, 0.002), (26, -0.033), (27, 0.005), (28, 0.009), (29, 0.04), (30, 0.026), (31, -0.028), (32, 0.004), (33, 0.018), (34, -0.041), (35, 0.05), (36, -0.02), (37, -0.026), (38, -0.007), (39, 0.045), (40, 0.008), (41, -0.01), (42, 0.003), (43, -0.06), (44, -0.044), (45, -0.026), (46, -0.009), (47, -0.037), (48, -0.03), (49, -0.008)]
simIndex simValue blogId blogTitle
same-blog 1 0.97114319 1383 andrew gelman stats-2012-06-18-Hierarchical modeling as a framework for extrapolation
Introduction: Phil recently posted on the challenge of extrapolation of inferences to new data. After telling the story of a colleague who flat-out refused to make predictions from his model of buildings to new data, Phil wrote, “This is an interesting problem because it is sort of outside the realm of statistics, and into some sort of meta-statistical area. How can you judge whether your results can be extrapolated to the ‘real world,’ if you cant get a real-world sample to compare to?” In reply, I wrote: I agree that this is an important and general problem, but I don’t think it is outside the realm of statistics! I think that one useful statistical framework here is multilevel modeling. Suppose you are applying a procedure to J cases and want to predict case J+1 (in this case, the cases are buildings and J=52). Let the parameters be theta_1,…,theta_{J+1}, with data y_1,…,y_{J+1}, and case-level predictors X_1,…,X_{J+1}. The question is how to generalize from (theta_1,…,theta_J) to theta_{
2 0.90108961 1425 andrew gelman stats-2012-07-23-Examples of the use of hierarchical modeling to generalize to new settings
Introduction: In a link to our back-and-forth on causal inference and the use of hierarchical models to bridge between different inferential settings, Elias Bareinboim (a computer scientist who is working with Judea Pearl) writes : In the past week, I have been engaged in a discussion with Andrew Gelman and his blog readers regarding causal inference, selection bias, confounding, and generalizability. I was trying to understand how his method which he calls “hierarchical modeling” would handle these issues and what guarantees it provides. . . . If anyone understands how “hierarchical modeling” can solve a simple toy problem (e.g., M-bias, control of confounding, mediation, generalizability), please share with us. In his post, Bareinboim raises a direct question about hierarchical modeling and also indirectly brings up larger questions about what is convincing evidence when evaluating a statistical method. As I wrote earlier, Bareinboim believes that “The only way investigators can decide w
Introduction: A sociologist writes in: Samuel Lucas has just published a paper in Quality and Quantity arguing that anything less than a full probability sample of higher levels in HLMs yields biased and unusable results. If I follow him correctly, he is arguing that not only are the SEs too small, but the parameter estimates themselves are biased and we cannot say in advance whether the bias is positive or negative. Lucas has thrown down a big gauntlet, advising us throw away our data unless the sample of macro units is right and ignore the published results that fail this standard. Extreme. Is there another conclusion to be drawn? Other advice to be given? A Bayesian path out of the valley? Heres’s the abstract to Lucas’s paper: The multilevel model has become a staple of social research. I textually and formally explicate sample design features that, I contend, are required for unbiased estimation of macro-level multilevel model parameters and the use of tools for statistical infe
4 0.83031082 948 andrew gelman stats-2011-10-10-Combining data from many sources
Introduction: Mark Grote writes: I’d like to request general feedback and references for a problem of combining disparate data sources in a regression model. We’d like to model log crop yield as a function of environmental predictors, but the observations come from many data sources and are peculiarly structured. Among the issues are: 1. Measurement precision in predictors and outcome varies widely with data sources. Some observations are in very coarse units of measurement, due to rounding or even observer guesswork. 2. There are obvious clusters of observations arising from studies in which crop yields were monitored over successive years in spatially proximate communities. Thus some variables may be constant within clusters–this is true even for log yield, probably due to rounding of similar yields. 3. Cluster size and intra-cluster association structure (temporal, spatial or both) vary widely across the dataset. My [Grote's] intuition is that we can learn about central tendency
Introduction: Some things I respect When it comes to meta-models of statistics, here are two philosophies that I respect: 1. (My) Bayesian approach, which I associate with E. T. Jaynes, in which you construct models with strong assumptions, ride your models hard, check their fit to data, and then scrap them and improve them as necessary. 2. At the other extreme, model-free statistical procedures that are designed to work well under very weak assumptions—for example, instead of assuming a distribution is Gaussian, you would just want the procedure to work well under some conditions on the smoothness of the second derivative of the log density function. Both the above philosophies recognize that (almost) all important assumptions will be wrong, and they resolve this concern via aggressive model checking or via robustness. And of course there are intermediate positions, such as working with Bayesian models that have been shown to be robust, and then still checking them. Or, to flip it arou
6 0.80493873 295 andrew gelman stats-2010-09-25-Clusters with very small numbers of observations
7 0.80058128 1267 andrew gelman stats-2012-04-17-Hierarchical-multilevel modeling with “big data”
8 0.79269743 1395 andrew gelman stats-2012-06-27-Cross-validation (What is it good for?)
9 0.7809037 352 andrew gelman stats-2010-10-19-Analysis of survey data: Design based models vs. hierarchical modeling?
10 0.78045458 1289 andrew gelman stats-2012-04-29-We go to war with the data we have, not the data we want
11 0.77462393 2294 andrew gelman stats-2014-04-17-If you get to the point of asking, just do it. But some difficulties do arise . . .
12 0.77423245 524 andrew gelman stats-2011-01-19-Data exploration and multiple comparisons
13 0.77349991 946 andrew gelman stats-2011-10-07-Analysis of Power Law of Participation
14 0.77333736 972 andrew gelman stats-2011-10-25-How do you interpret standard errors from a regression fit to the entire population?
15 0.7703473 936 andrew gelman stats-2011-10-02-Covariate Adjustment in RCT - Model Overfitting in Multilevel Regression
16 0.76559377 1468 andrew gelman stats-2012-08-24-Multilevel modeling and instrumental variables
17 0.76265043 1248 andrew gelman stats-2012-04-06-17 groups, 6 group-level predictors: What to do?
18 0.76234353 383 andrew gelman stats-2010-10-31-Analyzing the entire population rather than a sample
19 0.76163316 250 andrew gelman stats-2010-09-02-Blending results from two relatively independent multi-level models
20 0.76071274 1047 andrew gelman stats-2011-12-08-I Am Too Absolutely Heteroskedastic for This Probit Model
topicId topicWeight
[(15, 0.022), (16, 0.042), (17, 0.141), (21, 0.025), (24, 0.131), (42, 0.012), (77, 0.01), (86, 0.017), (87, 0.012), (89, 0.042), (99, 0.449)]
simIndex simValue blogId blogTitle
1 0.98971581 309 andrew gelman stats-2010-10-01-Why Development Economics Needs Theory?
Introduction: Robert Neumann writes: in the JEP 24(3), page18, Daron Acemoglu states: Why Development Economics Needs Theory There is no general agreement on how much we should rely on economic theory in motivating empirical work and whether we should try to formulate and estimate “structural parameters.” I (Acemoglu) argue that the answer is largely “yes” because otherwise econometric estimates would lack external validity, in which case they can neither inform us about whether a particular model or theory is a useful approximation to reality, nor would they be useful in providing us guidance on what the effects of similar shocks and policies would be in different circumstances or if implemented in different scales. I therefore define “structural parameters” as those that provide external validity and would thus be useful in testing theories or in policy analysis beyond the specific environment and sample from which they are derived. External validity becomes a particularly challenging t
Introduction: Malka Gorfine writes: We noticed that the important topic of association measures and tests came up again in your blog, and we have few comments in this regard. It is useful to distinguish between the univariate and multivariate methods. A consistent multivariate method can recognise dependence between two vectors of random variables, while a univariate method can only loop over pairs of components and check for dependency between them. There are very few consistent multivariate methods. To the best of our knowledge there are three practical methods: 1) HSIC by Gretton et al. (http://www.gatsby.ucl.ac.uk/~gretton/papers/GreBouSmoSch05.pdf) 2) dcov by Szekely et al. (http://projecteuclid.org/euclid.aoas/1267453933) 3) A method we introduced in Heller et al (Biometrika, 2013, 503—510, http://biomet.oxfordjournals.org/content/early/2012/12/04/biomet.ass070.full.pdf+html, and an R package, HHG, is available as well http://cran.r-project.org/web/packages/HHG/index.html). A
3 0.98488963 397 andrew gelman stats-2010-11-06-Multilevel quantile regression
Introduction: Ryan Seals writes: I’m an epidemiologist at Emory University, and I’m working on a project of release patterns in jails (basically trying to model how long individuals are in jail before they’re release, for purposes of designing short-term health interventions, i.e. HIV testing, drug counseling, etc…). The question lends itself to quantile regression; we’re interested in the # of days it takes for 50% and 75% of inmates to be released. But being a clustered/nested data structure, it also obviously lends itself to multilevel modeling, with the group-level being individual jails. So: do you know of any work on multilevel quantile regression? My quick lit search didn’t yield much, and I don’t see any preprogrammed way to do it in SAS. My reply: To start with, I’m putting in the R keyword here, on the hope that some readers might be able to refer you to an R function that does what you want. Beyond this, I think it should be possible to program something in Bugs. In ARM we hav
4 0.98415875 1076 andrew gelman stats-2011-12-21-Derman, Rodrik and the nature of statistical models
Introduction: Interesting thoughts from Kaiser Fung. Derman seems to have a point in his criticisms of economic models—and things are just as bad in other social sciences. (I’ve criticized economists and political scientists for taking a crude, 80-year-old model of psychology as “foundational,” but even more sophisticated models in psychology and sociology have a lot of holes, if you go outside of certain clearly bounded areas such as psychometrics.) What can be done, then? One approach, which appeals to me as a statistician, is to more carefully define one’s range of inquiry. Even if we don’t have a great model of political bargaining, we can still use ideal-point models to capture a lot of the variation in legislative voting. And, in my blog post linked to above, I recommended that economists forget about coming up with the grand unified theory of human behavior (pretty impossible, given that they still don’t want to let go of much of their folk-psychology models) and put more effort i
5 0.98204309 1616 andrew gelman stats-2012-12-10-John McAfee is a Heinlein hero
Introduction: “A small group of mathematicians” Jenny Davidson points to this article by Krugman on Asimov’s Foundation Trilogy. Given the silliness of the topic, Krugman’s piece is disappointingly serious (“Maybe the first thing to say about Foundation is that it’s not exactly science fiction – not really. Yes, it’s set in the future, there’s interstellar travel, people shoot each other with blasters instead of pistols and so on. But these are superficial details . . . the story can sound arid and didactic. . . . you’ll also be disappointed if you’re looking for shoot-em-up action scenes, in which Han Solo and Luke Skywalker destroy the Death Star in the nick of time. . . .”). What really jumped out at me from Krugman’s piece, though, was this line: In Foundation, we learn that a small group of mathematicians have developed “psychohistory”, the aforementioned rigorous science of society. Like Davidson (and Krugman), I read the Foundation books as a child. I remember the “psychohisto
6 0.98065233 1362 andrew gelman stats-2012-06-03-Question 24 of my final exam for Design and Analysis of Sample Surveys
same-blog 7 0.98050475 1383 andrew gelman stats-2012-06-18-Hierarchical modeling as a framework for extrapolation
8 0.98039454 1136 andrew gelman stats-2012-01-23-Fight! (also a bit of reminiscence at the end)
9 0.96893412 1557 andrew gelman stats-2012-11-01-‘Researcher Degrees of Freedom’
10 0.96817738 705 andrew gelman stats-2011-05-10-Some interesting unpublished ideas on survey weighting
11 0.96796453 2359 andrew gelman stats-2014-06-04-All the Assumptions That Are My Life
12 0.96685386 1272 andrew gelman stats-2012-04-20-More proposals to reform the peer-review system
13 0.96311468 1230 andrew gelman stats-2012-03-26-Further thoughts on nonparametric correlation measures
14 0.96275675 1228 andrew gelman stats-2012-03-25-Continuous variables in Bayesian networks
15 0.95926249 1361 andrew gelman stats-2012-06-02-Question 23 of my final exam for Design and Analysis of Sample Surveys
16 0.95717293 1591 andrew gelman stats-2012-11-26-Politics as an escape hatch
17 0.9560883 1807 andrew gelman stats-2013-04-17-Data problems, coding errors…what can be done?
18 0.95486903 2258 andrew gelman stats-2014-03-21-Random matrices in the news
19 0.95481515 216 andrew gelman stats-2010-08-18-More forecasting competitions
20 0.95469004 524 andrew gelman stats-2011-01-19-Data exploration and multiple comparisons