andrew_gelman_stats andrew_gelman_stats-2012 andrew_gelman_stats-2012-1289 knowledge-graph by maker-knowledge-mining
Source: html
Introduction: This post is by Phil. Psychologists perform experiments on Canadian undergraduate psychology students and draws conclusions that (they believe) apply to humans in general; they publish in Science. A drug company decides to embark on additional trials that will cost tens of millions of dollars based on the results of a careful double-blind study….whose patients are all volunteers from two hospitals. A movie studio holds 9 screenings of a new movie for volunteer viewers and, based on their survey responses, decides to spend another $8 million to re-shoot the ending. A researcher interested in the effect of ventilation on worker performance conducts a months-long study in which ventilation levels are varied and worker performance is monitored…in a single building. In almost all fields of research, most studies are based on convenience samples, or on random samples from a larger population that is itself a convenience sample. The paragraph above gives just a few examples. The benefit
sentIndex sentText sentNum sentScore
1 A drug company decides to embark on additional trials that will cost tens of millions of dollars based on the results of a careful double-blind study…. [sent-3, score-0.495]
2 A movie studio holds 9 screenings of a new movie for volunteer viewers and, based on their survey responses, decides to spend another $8 million to re-shoot the ending. [sent-5, score-0.676]
3 A researcher interested in the effect of ventilation on worker performance conducts a months-long study in which ventilation levels are varied and worker performance is monitored…in a single building. [sent-6, score-0.648]
4 In almost all fields of research, most studies are based on convenience samples, or on random samples from a larger population that is itself a convenience sample. [sent-7, score-0.642]
5 This does not mean that it goes without thinking, of course: most researchers, and all good ones, think about the extent to which their results might or might not be applicable to a wider population and try to frame their conclusions accordingly. [sent-11, score-0.631]
6 But most or all researchers are willing to extrapolate their results to wider populations to some degree. [sent-12, score-0.305]
7 I’m currently working with time series data on building electricity consumption. [sent-16, score-0.366]
8 Everyone thinks their package is better than anyone else’s, or at least they say so, but there’s usually no evidence. [sent-21, score-0.258]
9 A few months ago a company approached us to ask if we will compare the accuracy of their tool to some standard methods and allow them to publicize the results if they want to. [sent-22, score-0.452]
10 We compiled electricity data from a few dozen large commercial buildings — the population of interest — blanked out big chunks of data, and gave the resulting dataset to the company. [sent-26, score-0.71]
11 Their task is to make predictions for the missing time periods and give them to us, and we will compare their predictions to reality. [sent-27, score-0.366]
12 There’s no practical way to get data from a random sample of buildings in the country or even from a single electric utility, in part because of privacy issues (the utilities can’t give out the data without permission of the building owners). [sent-30, score-0.806]
13 Our data are a grab bag from different sources, and certainly not representative of the broad population of commercial buildings in many ways. [sent-31, score-0.587]
14 He asserts that since we don’t have a sampling plan, just a haphazard collection of building data, we can’t say anything at all. [sent-35, score-0.393]
15 He says “sure, of course you can say which method performs better for these 52 buildings, but you cannot say a single thing about a 53rd building. [sent-36, score-0.435]
16 He thinks it’s wrong (incorrect, and borderline immoral) to say that our results are even suggestive. [sent-39, score-0.364]
17 I offered a wager: if we find that one method performs substantially better than the others on average, I will bet you dinner that it will perform better for a 53rd building. [sent-40, score-0.56]
18 He said sure, fine for wagering over dinner, but it is scientifically indefensible to make any statement at all about which of the methods performs better in general on the basis of anything we might find using our dataset. [sent-41, score-0.362]
19 How can you judge whether your results can be extrapolated to the “real world,” if you can’t get a real-world sample to compare to? [sent-45, score-0.44]
20 (And if you could get a sample to compare to, you would, and then this problem wouldn’t come up). [sent-46, score-0.257]
wordName wordTfidf (topN-words)
[('buildings', 0.205), ('wider', 0.188), ('compare', 0.162), ('performs', 0.146), ('commercial', 0.144), ('audiences', 0.143), ('convenience', 0.143), ('studio', 0.141), ('ventilation', 0.141), ('march', 0.139), ('building', 0.138), ('population', 0.133), ('datasets', 0.124), ('movie', 0.124), ('electricity', 0.123), ('sampling', 0.122), ('dinner', 0.119), ('energy', 0.118), ('results', 0.117), ('thinks', 0.114), ('worker', 0.111), ('decides', 0.109), ('viewers', 0.107), ('company', 0.106), ('data', 0.105), ('predictions', 0.102), ('conclusions', 0.101), ('sample', 0.095), ('frame', 0.092), ('world', 0.085), ('trials', 0.083), ('randomized', 0.08), ('random', 0.08), ('millions', 0.08), ('single', 0.078), ('better', 0.077), ('perform', 0.073), ('samples', 0.073), ('screenings', 0.071), ('disinterested', 0.071), ('wagering', 0.071), ('fields', 0.07), ('find', 0.068), ('say', 0.067), ('standard', 0.067), ('extrapolated', 0.066), ('borderline', 0.066), ('conducts', 0.066), ('haphazard', 0.066), ('careful', 0.066)]
simIndex simValue blogId blogTitle
same-blog 1 0.99999976 1289 andrew gelman stats-2012-04-29-We go to war with the data we have, not the data we want
Introduction: This post is by Phil. Psychologists perform experiments on Canadian undergraduate psychology students and draws conclusions that (they believe) apply to humans in general; they publish in Science. A drug company decides to embark on additional trials that will cost tens of millions of dollars based on the results of a careful double-blind study….whose patients are all volunteers from two hospitals. A movie studio holds 9 screenings of a new movie for volunteer viewers and, based on their survey responses, decides to spend another $8 million to re-shoot the ending. A researcher interested in the effect of ventilation on worker performance conducts a months-long study in which ventilation levels are varied and worker performance is monitored…in a single building. In almost all fields of research, most studies are based on convenience samples, or on random samples from a larger population that is itself a convenience sample. The paragraph above gives just a few examples. The benefit
2 0.21995254 1383 andrew gelman stats-2012-06-18-Hierarchical modeling as a framework for extrapolation
Introduction: Phil recently posted on the challenge of extrapolation of inferences to new data. After telling the story of a colleague who flat-out refused to make predictions from his model of buildings to new data, Phil wrote, “This is an interesting problem because it is sort of outside the realm of statistics, and into some sort of meta-statistical area. How can you judge whether your results can be extrapolated to the ‘real world,’ if you cant get a real-world sample to compare to?” In reply, I wrote: I agree that this is an important and general problem, but I don’t think it is outside the realm of statistics! I think that one useful statistical framework here is multilevel modeling. Suppose you are applying a procedure to J cases and want to predict case J+1 (in this case, the cases are buildings and J=52). Let the parameters be theta_1,…,theta_{J+1}, with data y_1,…,y_{J+1}, and case-level predictors X_1,…,X_{J+1}. The question is how to generalize from (theta_1,…,theta_J) to theta_{
3 0.18250459 1605 andrew gelman stats-2012-12-04-Write This Book
Introduction: This post is by Phil Price. I’ve been preparing a review of a new statistics textbook aimed at students and practitioners in the “physical sciences,” as distinct from the social sciences and also distinct from people who intend to take more statistics courses. I figured that since it’s been years since I looked at an intro stats textbook, I should look at a few others and see how they differ from this one, so in addition to the book I’m reviewing I’ve looked at some other textbooks aimed at similar audiences: Milton and Arnold; Hines, Montgomery, Goldsman, and Borror; and a few others. I also looked at the table of contents of several more. There is a lot of overlap in the coverage of these books — they all have discussions of common discrete and continuous distributions, joint distributions, descriptive statistics, parameter estimation, hypothesis testing, linear regression, ANOVA, factorial experimental design, and a few other topics. I can see how, from a statisti
4 0.15825619 1628 andrew gelman stats-2012-12-17-Statistics in a world where nothing is random
Introduction: Rama Ganesan writes: I think I am having an existential crisis. I used to work with animals (rats, mice, gerbils etc.) Then I started to work in marketing research where we did have some kind of random sampling procedure. So up until a few years ago, I was sort of okay. Now I am teaching marketing research, and I feel like there is no real random sampling anymore. I take pains to get students to understand what random means, and then the whole lot of inferential statistics. Then almost anything they do – the sample is not random. They think I am contradicting myself. They use convenience samples at every turn – for their school work, and the enormous amount on online surveying that gets done. Do you have any suggestions for me? Other than say, something like this . My reply: Statistics does not require randomness. The three essential elements of statistics are measurement, comparison, and variation. Randomness is one way to supply variation, and it’s one way to model
5 0.15635616 906 andrew gelman stats-2011-09-14-Another day, another stats postdoc
Introduction: This post is from Phil Price. I work in the Environmental Energy Technologies Division at Lawrence Berkeley National Laboratory, and I am looking for a postdoc who knows substantially more than I do about time-series modeling; in practice this probably means someone whose dissertation work involved that sort of thing. The work involves developing models to predict and/or forecast the time-dependent energy use in buildings, given historical data and some covariates such as outdoor temperature. Simple regression approaches (e.g. using time-of-week indicator variables, plus outdoor temperature) work fine for a lot of things, but we still have a variety of problems. To give one example, sometimes building behavior changes — due to retrofits, or a change in occupant behavior — so that a single model won’t fit well over a long time period. We want to recognize these changes automatically . We have many other issues besides: heteroskedasticity, need for good uncertainty estimates, abilit
6 0.14186493 2359 andrew gelman stats-2014-06-04-All the Assumptions That Are My Life
10 0.12516153 1807 andrew gelman stats-2013-04-17-Data problems, coding errors…what can be done?
11 0.12419067 1425 andrew gelman stats-2012-07-23-Examples of the use of hierarchical modeling to generalize to new settings
12 0.1233782 1551 andrew gelman stats-2012-10-28-A convenience sample and selected treatments
13 0.12335829 2018 andrew gelman stats-2013-09-12-Do you ever have that I-just-fit-a-model feeling?
14 0.12072215 1640 andrew gelman stats-2012-12-26-What do people do wrong? WSJ columnist is looking for examples!
15 0.11745898 405 andrew gelman stats-2010-11-10-Estimation from an out-of-date census
16 0.11657755 1455 andrew gelman stats-2012-08-12-Probabilistic screening to get an approximate self-weighted sample
18 0.11375719 352 andrew gelman stats-2010-10-19-Analysis of survey data: Design based models vs. hierarchical modeling?
19 0.11190556 2172 andrew gelman stats-2014-01-14-Advice on writing research articles
20 0.11142322 1527 andrew gelman stats-2012-10-10-Another reason why you can get good inferences from a bad model
topicId topicWeight
[(0, 0.273), (1, 0.018), (2, 0.026), (3, -0.082), (4, 0.063), (5, 0.065), (6, -0.036), (7, 0.006), (8, 0.043), (9, -0.02), (10, -0.029), (11, -0.05), (12, 0.005), (13, -0.016), (14, -0.079), (15, -0.013), (16, 0.021), (17, -0.019), (18, 0.04), (19, 0.004), (20, -0.027), (21, -0.041), (22, -0.031), (23, 0.08), (24, -0.049), (25, 0.002), (26, -0.025), (27, -0.003), (28, 0.092), (29, 0.082), (30, 0.003), (31, -0.038), (32, 0.03), (33, 0.013), (34, -0.015), (35, 0.035), (36, -0.026), (37, -0.014), (38, -0.04), (39, 0.022), (40, 0.056), (41, -0.021), (42, 0.006), (43, 0.011), (44, -0.049), (45, -0.048), (46, -0.022), (47, 0.014), (48, 0.028), (49, -0.004)]
simIndex simValue blogId blogTitle
same-blog 1 0.97116297 1289 andrew gelman stats-2012-04-29-We go to war with the data we have, not the data we want
Introduction: This post is by Phil. Psychologists perform experiments on Canadian undergraduate psychology students and draws conclusions that (they believe) apply to humans in general; they publish in Science. A drug company decides to embark on additional trials that will cost tens of millions of dollars based on the results of a careful double-blind study….whose patients are all volunteers from two hospitals. A movie studio holds 9 screenings of a new movie for volunteer viewers and, based on their survey responses, decides to spend another $8 million to re-shoot the ending. A researcher interested in the effect of ventilation on worker performance conducts a months-long study in which ventilation levels are varied and worker performance is monitored…in a single building. In almost all fields of research, most studies are based on convenience samples, or on random samples from a larger population that is itself a convenience sample. The paragraph above gives just a few examples. The benefit
2 0.83931386 1455 andrew gelman stats-2012-08-12-Probabilistic screening to get an approximate self-weighted sample
Introduction: Sharad had a survey sampling question: We’re trying to use mechanical turk to conduct some surveys, and have quickly discovered that turkers tend to be quite young. We’d really like a representative sample of the U.S., or at the least be able to recruit a diverse enough sample from turk that we can post-stratify to adjust the estimates. The approach we ended up taking is to pay turkers a small amount to answer a couple of screening questions (age & sex), and then probabilistically recruit individuals to complete the full survey (for more money) based on the estimated turk population parameters and our desired target distribution. We use rejection sampling, so the end result is that individuals who are invited to take the full survey look as if they came from a representative sample, at least in terms of age and sex. I’m wondering whether this sort of technique—a two step design in which participants are first screened and then probabilistically selected to mimic a target distributio
3 0.81266481 2359 andrew gelman stats-2014-06-04-All the Assumptions That Are My Life
Introduction: Statisticians take tours in other people’s data. All methods of statistical inference rest on statistical models. Experiments typically have problems with compliance, measurement error, generalizability to the real world, and representativeness of the sample. Surveys typically have problems of undercoverage, nonresponse, and measurement error. Real surveys are done to learn about the general population. But real surveys are not random samples. For another example, consider educational tests: what are they exactly measuring? Nobody knows. Medical research: even if it’s a randomized experiment, the participants in the study won’t be a random sample from the population for whom you’d recommend treatment. You don’t need random sampling to generalize the results of a medical experiment to the general population but you need some substantive theory to make the assumption that effects in your nonrepresentative sample of people will be similar to effects in the population of interest. Ve
4 0.80948639 1940 andrew gelman stats-2013-07-16-A poll that throws away data???
Introduction: Mark Blumenthal writes: What do you think about the “random rejection” method used by PPP that was attacked at some length today by a Republican pollster. Our just published post on the debate includes all the details as I know them. The Storify of Martino’s tweets has some additional data tables linked to toward the end. Also, more specifically, setting aside Martino’s suggestion of manipulation (which is also quite possible with post-stratification weights), would the PPP method introduce more potential random error than weighting? From Blumenthal’s blog: B.J. Martino, a senior vice president at the Republican polling firm The Tarrance Group, went on an 30-minute Twitter rant on Tuesday questioning the unorthodox method used by PPP [Public Policy Polling] to select samples and weight data: “Looking at @ppppolls new VA SW. Wondering how many interviews they discarded to get down to 601 completes? Because @ppppolls discards a LOT of interviews. Of 64,811 conducted
5 0.78890485 1628 andrew gelman stats-2012-12-17-Statistics in a world where nothing is random
Introduction: Rama Ganesan writes: I think I am having an existential crisis. I used to work with animals (rats, mice, gerbils etc.) Then I started to work in marketing research where we did have some kind of random sampling procedure. So up until a few years ago, I was sort of okay. Now I am teaching marketing research, and I feel like there is no real random sampling anymore. I take pains to get students to understand what random means, and then the whole lot of inferential statistics. Then almost anything they do – the sample is not random. They think I am contradicting myself. They use convenience samples at every turn – for their school work, and the enormous amount on online surveying that gets done. Do you have any suggestions for me? Other than say, something like this . My reply: Statistics does not require randomness. The three essential elements of statistics are measurement, comparison, and variation. Randomness is one way to supply variation, and it’s one way to model
6 0.78843117 405 andrew gelman stats-2010-11-10-Estimation from an out-of-date census
8 0.77339423 946 andrew gelman stats-2011-10-07-Analysis of Power Law of Participation
9 0.76978797 70 andrew gelman stats-2010-06-07-Mister P goes on a date
10 0.76412386 51 andrew gelman stats-2010-05-26-If statistics is so significantly great, why don’t statisticians use statistics?
12 0.75840294 1551 andrew gelman stats-2012-10-28-A convenience sample and selected treatments
14 0.74340558 107 andrew gelman stats-2010-06-24-PPS in Georgia
15 0.74090844 1191 andrew gelman stats-2012-03-01-Hoe noem je?
16 0.73709357 1383 andrew gelman stats-2012-06-18-Hierarchical modeling as a framework for extrapolation
17 0.73633718 603 andrew gelman stats-2011-03-07-Assumptions vs. conditions, part 2
18 0.73452604 242 andrew gelman stats-2010-08-29-The Subtle Micro-Effects of Peacekeeping
19 0.72930896 544 andrew gelman stats-2011-01-29-Splitting the data
topicId topicWeight
[(2, 0.018), (13, 0.02), (15, 0.022), (16, 0.092), (21, 0.044), (24, 0.119), (39, 0.012), (41, 0.013), (43, 0.011), (71, 0.061), (76, 0.026), (82, 0.018), (85, 0.014), (86, 0.045), (87, 0.01), (89, 0.014), (95, 0.019), (99, 0.3)]
simIndex simValue blogId blogTitle
same-blog 1 0.9722712 1289 andrew gelman stats-2012-04-29-We go to war with the data we have, not the data we want
Introduction: This post is by Phil. Psychologists perform experiments on Canadian undergraduate psychology students and draws conclusions that (they believe) apply to humans in general; they publish in Science. A drug company decides to embark on additional trials that will cost tens of millions of dollars based on the results of a careful double-blind study….whose patients are all volunteers from two hospitals. A movie studio holds 9 screenings of a new movie for volunteer viewers and, based on their survey responses, decides to spend another $8 million to re-shoot the ending. A researcher interested in the effect of ventilation on worker performance conducts a months-long study in which ventilation levels are varied and worker performance is monitored…in a single building. In almost all fields of research, most studies are based on convenience samples, or on random samples from a larger population that is itself a convenience sample. The paragraph above gives just a few examples. The benefit
Introduction: We had a discussion the other day of a paper, “The Economic Effects of Climate Change,” by economist Richard Tol. The paper came to my attention after I saw a notice from Adam Marcus that it was recently revised because of data errors. But after looking at the paper more carefully, I see a bunch of other problems that, to me, make the whole analysis close to useless as it stands. I think this is worth discussing because the paper has been somewhat influential (so far cited 328 times, according to Google Scholar) and has even been cited in the popular press as evidence that “Climate change has done more good than harm so far and is likely to continue doing so for most of this century . . . There are many likely effects of climate change: positive and negative, economic and ecological, humanitarian and financial. And if you aggregate them all, the overall effect is positive today — and likely to stay positive until around 2080. That was the conclusion of Professor Richard Tol
3 0.9678455 2281 andrew gelman stats-2014-04-04-The Notorious N.H.S.T. presents: Mo P-values Mo Problems
Introduction: A recent discussion between commenters Question and Fernando captured one of the recurrent themes here from the past year. Question: The problem is simple, the researchers are disproving always false null hypotheses and taking this disproof as near proof that their theory is correct. Fernando: Whereas it is probably true that researchers misuse NHT, the problem with tabloid science is broader and deeper. It is systemic. Question: I do not see how anything can be deeper than replacing careful description, prediction, falsification, and independent replication with dynamite plots, p-values, affirming the consequent, and peer review. From my own experience I am confident in saying that confusion caused by NHST is at the root of this problem. Fernando: Incentives? Impact factors? Publish or die? “Interesting” and “new” above quality and reliability, or actually answering a research question, and a silly and unbecoming obsession with being quoted in NYT, etc. . . . Giv
4 0.96630251 2137 andrew gelman stats-2013-12-17-Replication backlash
Introduction: Raghuveer Parthasarathy pointed me to an article in Nature by Mina Bissell, who writes , “The push to replicate findings could shelve promising research and unfairly damage the reputations of careful, meticulous scientists.” I can see where she’s coming from: if you work hard day after day in the lab, it’s gotta be a bit frustrating to find all your work questioned, for the frauds of the Dr. Anil Pottis and Diederik Stapels to be treated as a reason for everyone else’s work to be considered guilty until proven innocent. That said, I pretty much disagree with Bissell’s article, and really the best thing I can say about it is that I think it’s a good sign that the push for replication is so strong that now there’s a backlash against it. Traditionally, leading scientists have been able to simply ignore the push for replication. If they are feeling that the replication movement is strong enough that they need to fight it, that to me is good news. I’ll explain a bit in the conte
5 0.96441329 32 andrew gelman stats-2010-05-14-Causal inference in economics
Introduction: Aaron Edlin points me to this issue of the Journal of Economic Perspectives that focuses on statistical methods for causal inference in economics. (Michael Bishop’s page provides some links .) To quickly summarize my reactions to Angrist and Pischke’s book: I pretty much agree with them that the potential-outcomes or natural-experiment approach is the most useful way to think about causality in economics and related fields. My main amendments to Angrist and Pischke would be to recognize that: 1. Modeling is important, especially modeling of interactions . It’s unfortunate to see a debate between experimentalists and modelers. Some experimenters (not Angrist and Pischke) make the mistake of avoiding models: Once they have their experimental data, they check their brains at the door and do nothing but simple differences, not realizing how much more can be learned. Conversely, some modelers are unduly dismissive of experiments and formal observational studies, forgetting t
7 0.9639017 1910 andrew gelman stats-2013-06-22-Struggles over the criticism of the “cannabis users and IQ change” paper
8 0.96370727 1878 andrew gelman stats-2013-05-31-How to fix the tabloids? Toward replicable social science research
9 0.96367633 1404 andrew gelman stats-2012-07-03-Counting gays
10 0.96352637 110 andrew gelman stats-2010-06-26-Philosophy and the practice of Bayesian statistics
11 0.96287441 315 andrew gelman stats-2010-10-03-He doesn’t trust the fit . . . r=.999
12 0.96264106 711 andrew gelman stats-2011-05-14-Steven Rhoads’s book, “The Economist’s View of the World”
13 0.96249223 438 andrew gelman stats-2010-11-30-I just skyped in from Kentucky, and boy are my arms tired
14 0.96226448 1529 andrew gelman stats-2012-10-11-Bayesian brains?
15 0.96225584 670 andrew gelman stats-2011-04-20-Attractive but hard-to-read graph could be made much much better
16 0.96224761 288 andrew gelman stats-2010-09-21-Discussion of the paper by Girolami and Calderhead on Bayesian computation
17 0.96215296 2182 andrew gelman stats-2014-01-22-Spell-checking example demonstrates key aspects of Bayesian data analysis
18 0.96199101 2263 andrew gelman stats-2014-03-24-Empirical implications of Empirical Implications of Theoretical Models
19 0.9619813 1917 andrew gelman stats-2013-06-28-Econ coauthorship update
20 0.96174282 2337 andrew gelman stats-2014-05-18-Never back down: The culture of poverty and the culture of journalism