andrew_gelman_stats andrew_gelman_stats-2010 andrew_gelman_stats-2010-315 knowledge-graph by maker-knowledge-mining
Source: html
Introduction: I received the following question from an education researcher: I was wondering if I could ask you a question about an HLM model I’m working on. The basic design is that we have 5 years of 8th grade student achievement data (standardized test scores, this is the dependent variable), 4th grade test scores, demographics (e.g., gender and ethnicity) and status wrt special ed or ELL, etc.. In addition, we have some school- or second-level information such as school averages of the student information, type of school (grade configuration), enrollment and so. In total there are thousands of students and many schools over the 5 years of information. The model we’re using is quite parsimonious, using only 7 student-level effects and 4 school-level effects. What’s puzzling us is that the correlation between predicted and actual is unrealistically high…r=0.999. We’re using the HPMIXED procedure in SAS but that shouldn’t matter. By dropping variables, obviously we can get the corre
sentIndex sentText sentNum sentScore
1 I received the following question from an education researcher: I was wondering if I could ask you a question about an HLM model I’m working on. [sent-1, score-0.44]
2 The basic design is that we have 5 years of 8th grade student achievement data (standardized test scores, this is the dependent variable), 4th grade test scores, demographics (e. [sent-2, score-1.318]
3 , gender and ethnicity) and status wrt special ed or ELL, etc. [sent-4, score-0.26]
4 In addition, we have some school- or second-level information such as school averages of the student information, type of school (grade configuration), enrollment and so. [sent-6, score-0.701]
5 In total there are thousands of students and many schools over the 5 years of information. [sent-7, score-0.154]
6 The model we’re using is quite parsimonious, using only 7 student-level effects and 4 school-level effects. [sent-8, score-0.572]
7 What’s puzzling us is that the correlation between predicted and actual is unrealistically high…r=0. [sent-9, score-0.844]
8 We’re using the HPMIXED procedure in SAS but that shouldn’t matter. [sent-11, score-0.201]
9 By dropping variables, obviously we can get the correlation to go down, but we shouldn’t have to do this in my view. [sent-12, score-0.395]
10 It looks like we’re overfitting things but I don’t see how. [sent-13, score-0.23]
11 Is it important that the coefficient of variation for the dependent variable is about 10? [sent-14, score-0.523]
12 To me, this seems quite low and, coupled with a pretty narrow range of possible values (between 1 and 4. [sent-15, score-0.554]
13 We’re uncomfortable with an unrealistically good fit and are wondering how to fix it. [sent-18, score-0.611]
14 My reply: I’m not quite sure how this correlation is being predicted, but I wonder if what’s happening is that it’s using the estimates of the student effects–the unexplained student-level variation–in the predicted values. [sent-19, score-1.104]
15 One way to get a sense of this would be to use your model to predict each year’s data from the previous year. [sent-20, score-0.088]
16 I think you have to get a better sense of what these fitted values really mean. [sent-23, score-0.206]
wordName wordTfidf (topN-words)
[('unrealistically', 0.281), ('grade', 0.262), ('predicted', 0.239), ('overfitting', 0.23), ('correlation', 0.211), ('dependent', 0.188), ('student', 0.182), ('scores', 0.161), ('quite', 0.144), ('hlm', 0.14), ('wondering', 0.14), ('shouldn', 0.138), ('parsimonious', 0.134), ('variation', 0.134), ('configuration', 0.126), ('unexplained', 0.126), ('using', 0.124), ('values', 0.123), ('variable', 0.121), ('enrollment', 0.12), ('coupled', 0.12), ('school', 0.115), ('sas', 0.113), ('puzzling', 0.113), ('test', 0.11), ('standardized', 0.108), ('re', 0.106), ('dropping', 0.106), ('question', 0.106), ('demographics', 0.104), ('achievement', 0.1), ('uncomfortable', 0.099), ('narrow', 0.094), ('ethnicity', 0.093), ('effects', 0.092), ('ed', 0.092), ('averages', 0.091), ('fix', 0.091), ('gender', 0.088), ('model', 0.088), ('fitted', 0.083), ('coefficient', 0.08), ('status', 0.08), ('thousands', 0.079), ('obviously', 0.078), ('information', 0.078), ('happening', 0.078), ('procedure', 0.077), ('schools', 0.075), ('pretty', 0.073)]
simIndex simValue blogId blogTitle
same-blog 1 0.99999988 315 andrew gelman stats-2010-10-03-He doesn’t trust the fit . . . r=.999
Introduction: I received the following question from an education researcher: I was wondering if I could ask you a question about an HLM model I’m working on. The basic design is that we have 5 years of 8th grade student achievement data (standardized test scores, this is the dependent variable), 4th grade test scores, demographics (e.g., gender and ethnicity) and status wrt special ed or ELL, etc.. In addition, we have some school- or second-level information such as school averages of the student information, type of school (grade configuration), enrollment and so. In total there are thousands of students and many schools over the 5 years of information. The model we’re using is quite parsimonious, using only 7 student-level effects and 4 school-level effects. What’s puzzling us is that the correlation between predicted and actual is unrealistically high…r=0.999. We’re using the HPMIXED procedure in SAS but that shouldn’t matter. By dropping variables, obviously we can get the corre
2 0.16635111 1688 andrew gelman stats-2013-01-22-That claim that students whose parents pay for more of college get worse grades
Introduction: Theodore Vasiloudis writes: I came upon this article by Laura Hamilton, an assistant professor in the University of California at Merced, that claims that “The more money that parents provide for higher education, the lower the grades their children earn.” I can’t help but feel that there something wrong with the basis of the study or a confounding factor causing this apparent correlation, and since you often comment on studies on your blog I thought you might find this study interesting. My reply: I have to admit that the description above made me suspicious of the study before I even looked at it. On first thought, I’d expect the effect of parent’s financial contributions to be positive (as they free the student from the need to get a job during college), but not negative. Hamilton argues that “parental investments create a disincentive for student achievement,” which may be—but I’m generally suspicious of arguments in which the rebound is bigger than the main effect.
3 0.14824803 1900 andrew gelman stats-2013-06-15-Exploratory multilevel analysis when group-level variables are of importance
Introduction: Steve Miller writes: Much of what I do is cross-national analyses of survey data (largely World Values Survey). . . . My big question pertains to (what I would call) exploratory analysis of multilevel data, especially when the group-level predictors are of theoretical importance. A lot of what I do involves analyzing cross-national survey items of citizen attitudes, typically of political leadership. These survey items are usually yes/no responses, or four-part responses indicating a level of agreement (strongly agree, agree, disagree, strongly disagree) that can be condensed into a binary variable. I believe these can be explained by reference to country-level factors. Much of the group-level variables of interest are count variables with a modal value of 0, which can be quite messy. How would you recommend exploring the variation in the dependent variable as it could be explained by the group-level count variable of interest, before fitting the multilevel model itself? When
4 0.14598906 1803 andrew gelman stats-2013-04-14-Why girls do better in school
Introduction: Wayne Folta writes, “In light of your recent blog post on women in higher education, here’s one I just read about on a techie website regarding elementary education”: Why do girls get better grades in elementary school than boys—even when they perform worse on standardized tests? New research . . . suggests that it’s because of their classroom behavior, which may lead teachers to assign girls higher grades than their male counterparts. . . . The study, co-authored by [Christopher] Cornwell and David Mustard at UGA and Jessica Van Parys at Columbia, analyzed data on more than 5,800 students from kindergarten through fifth grade. It examined students’ performance on standardized tests in three categories—reading, math and science-linking test scores to teachers’ assessments of their students’ progress, both academically and more broadly. The data show, for the first time, that gender disparities in teacher grades start early and uniformly favor girls. In every subject area, bo
5 0.13632737 1980 andrew gelman stats-2013-08-13-Test scores and grades predict job performance (but maybe not at Google)
Introduction: Eric Loken writes : If you’re used to Google upending conventional wisdom, then yesterday’s interview with Laszlo Bock in the New York Times did not disappoint. Google has determined that test scores and transcripts are useless because they don’t predict performance among its employees. . . . I [Loken] am going to assume they’re well aware of the limits of their claim, and instead I’m going say that as readers of the interview we should not lose sight of a fundamental fact - Across a wide variety of employment settings, one of the most robust findings in organizational psychology is that tests of cognitive ability are strong predictors of job performance. If Google has found otherwise, what they have found is that grades and test scores are not predictive of performance at Google. In general, in the workplace tests are still highly predictive of success. If all the research says that test scores and grades predict performance, why would the people at Google want to igno
6 0.12834635 1353 andrew gelman stats-2012-05-30-Question 20 of my final exam for Design and Analysis of Sample Surveys
7 0.12814157 257 andrew gelman stats-2010-09-04-Question about standard range for social science correlations
8 0.12589236 226 andrew gelman stats-2010-08-23-More on those L.A. Times estimates of teacher effectiveness
10 0.12357817 1121 andrew gelman stats-2012-01-15-R-squared for multilevel models
11 0.12306032 2176 andrew gelman stats-2014-01-19-Transformations for non-normal data
12 0.12257455 529 andrew gelman stats-2011-01-21-“City Opens Inquiry on Grading Practices at a Top-Scoring Bronx School”
13 0.1209856 609 andrew gelman stats-2011-03-13-Coauthorship norms
14 0.12067655 95 andrew gelman stats-2010-06-17-“Rewarding Strivers: Helping Low-Income Students Succeed in College”
15 0.11956966 957 andrew gelman stats-2011-10-14-Questions about a study of charter schools
16 0.11906426 606 andrew gelman stats-2011-03-10-It’s no fun being graded on a curve
17 0.1159635 1218 andrew gelman stats-2012-03-18-Check your missing-data imputations using cross-validation
18 0.11503378 32 andrew gelman stats-2010-05-14-Causal inference in economics
19 0.10982664 1350 andrew gelman stats-2012-05-28-Value-added assessment: What went wrong?
20 0.10695697 1431 andrew gelman stats-2012-07-27-Overfitting
topicId topicWeight
[(0, 0.191), (1, 0.055), (2, 0.081), (3, -0.038), (4, 0.125), (5, 0.084), (6, 0.053), (7, 0.054), (8, 0.036), (9, 0.054), (10, 0.042), (11, 0.08), (12, -0.029), (13, -0.068), (14, 0.001), (15, 0.011), (16, 0.06), (17, 0.009), (18, -0.06), (19, 0.04), (20, -0.009), (21, 0.043), (22, 0.001), (23, -0.058), (24, 0.039), (25, -0.052), (26, -0.006), (27, -0.001), (28, -0.059), (29, 0.052), (30, 0.021), (31, 0.029), (32, 0.065), (33, 0.013), (34, -0.024), (35, 0.043), (36, 0.045), (37, 0.017), (38, 0.028), (39, 0.0), (40, 0.007), (41, -0.016), (42, 0.026), (43, 0.008), (44, -0.059), (45, 0.02), (46, 0.015), (47, -0.049), (48, 0.028), (49, 0.043)]
simIndex simValue blogId blogTitle
same-blog 1 0.96172839 315 andrew gelman stats-2010-10-03-He doesn’t trust the fit . . . r=.999
Introduction: I received the following question from an education researcher: I was wondering if I could ask you a question about an HLM model I’m working on. The basic design is that we have 5 years of 8th grade student achievement data (standardized test scores, this is the dependent variable), 4th grade test scores, demographics (e.g., gender and ethnicity) and status wrt special ed or ELL, etc.. In addition, we have some school- or second-level information such as school averages of the student information, type of school (grade configuration), enrollment and so. In total there are thousands of students and many schools over the 5 years of information. The model we’re using is quite parsimonious, using only 7 student-level effects and 4 school-level effects. What’s puzzling us is that the correlation between predicted and actual is unrealistically high…r=0.999. We’re using the HPMIXED procedure in SAS but that shouldn’t matter. By dropping variables, obviously we can get the corre
2 0.81511652 1350 andrew gelman stats-2012-05-28-Value-added assessment: What went wrong?
Introduction: Jacob Hartog writes the following in reaction to my post on the use of value-added modeling for teacher assessment: What I [Hartog] think has been inadequately discussed is the use of individual model specifications to assign these teacher ratings, rather than the zone of agreement across a broad swath of model specifications. For example, the model used by NYCDOE doesn’t just control for a student’s prior year test score (as I think everyone can agree is a good idea.) It also assumes that different demographic groups will learn different amounts in a given year, and assigns a school-level random effect. The result is that, as was much ballyhooed at the time of the release of the data,the average teacher rating for a given school is roughly the same, no matter whether the school is performing great or terribly. The headline from this was “excellent teachers spread evenly across the city’s schools,” rather than “the specification of these models assume that excellent teachers are
3 0.78446168 226 andrew gelman stats-2010-08-23-More on those L.A. Times estimates of teacher effectiveness
Introduction: In discussing the ongoing Los Angeles Times series on teacher effectiveness, Alex Tabarrok and I both were impressed that the newspaper was reporting results on individual teachers, moving beyond the general research findings (“teachers matter,” “KIPP really works, but it requires several extra hours in the school day,” and so forth) that we usually see from value-added analyses in education. My first reaction was that the L.A. Times could get away with this because, unlike academic researchers, they can do whatever they want as long as they don’t break the law. They don’t have to answer to an Institutional Review Board. (By referring to this study by its publication outlet rather than its authors, I’m violating my usual rule (see the last paragraph here ). In this case, I think it’s ok to refer to the “L.A. Times study” because what’s notable is not the analysis (thorough as it may be) but how it is being reported.) Here I’d like to highlight a few other things came up in our
4 0.76422894 1688 andrew gelman stats-2013-01-22-That claim that students whose parents pay for more of college get worse grades
Introduction: Theodore Vasiloudis writes: I came upon this article by Laura Hamilton, an assistant professor in the University of California at Merced, that claims that “The more money that parents provide for higher education, the lower the grades their children earn.” I can’t help but feel that there something wrong with the basis of the study or a confounding factor causing this apparent correlation, and since you often comment on studies on your blog I thought you might find this study interesting. My reply: I have to admit that the description above made me suspicious of the study before I even looked at it. On first thought, I’d expect the effect of parent’s financial contributions to be positive (as they free the student from the need to get a job during college), but not negative. Hamilton argues that “parental investments create a disincentive for student achievement,” which may be—but I’m generally suspicious of arguments in which the rebound is bigger than the main effect.
Introduction: Howard Wainer writes : When we focus only on the differences between groups, we too easily lose track of the big picture. Nowhere is this more obvious than in the current public discussions of the size of the gap in test scores that is observed between racial groups. It has been noted that in New Jersey the gap between the average scores of white and black students on the well-developed scale of the National Assessment of Educational Progress (NAEP) has shrunk by only about 25 percent over the past two decades. The conclusion drawn was that even though the change is in the right direction, it is far too slow. But focusing on the difference blinds us to what has been a remarkable success in education over the past 20 years. Although the direction and size of student improvements are considered across many subject areas and many age groups, I will describe just one — 4th grade mathematics. . . . there have been steep gains for both racial groups over this period (somewhat steeper g
6 0.74370444 95 andrew gelman stats-2010-06-17-“Rewarding Strivers: Helping Low-Income Students Succeed in College”
7 0.73240602 825 andrew gelman stats-2011-07-27-Grade inflation: why weren’t the instructors all giving all A’s already??
8 0.72880304 936 andrew gelman stats-2011-10-02-Covariate Adjustment in RCT - Model Overfitting in Multilevel Regression
9 0.72577244 1507 andrew gelman stats-2012-09-22-Grade inflation: why weren’t the instructors all giving all A’s already??
10 0.72212762 1218 andrew gelman stats-2012-03-18-Check your missing-data imputations using cross-validation
11 0.72076523 1803 andrew gelman stats-2013-04-14-Why girls do better in school
12 0.71720457 561 andrew gelman stats-2011-02-06-Poverty, educational performance – and can be done about it
13 0.71314907 213 andrew gelman stats-2010-08-17-Matching at two levels
14 0.71248746 1918 andrew gelman stats-2013-06-29-Going negative
15 0.70771348 606 andrew gelman stats-2011-03-10-It’s no fun being graded on a curve
16 0.70504147 257 andrew gelman stats-2010-09-04-Question about standard range for social science correlations
17 0.70072156 93 andrew gelman stats-2010-06-17-My proposal for making college admissions fairer
18 0.69857216 71 andrew gelman stats-2010-06-07-Pay for an A?
19 0.69624668 938 andrew gelman stats-2011-10-03-Comparing prediction errors
20 0.695234 542 andrew gelman stats-2011-01-28-Homework and treatment levels
topicId topicWeight
[(1, 0.052), (16, 0.112), (21, 0.032), (24, 0.132), (45, 0.016), (48, 0.012), (57, 0.014), (63, 0.032), (69, 0.012), (72, 0.037), (76, 0.039), (81, 0.015), (82, 0.013), (86, 0.066), (99, 0.303)]
simIndex simValue blogId blogTitle
same-blog 1 0.96851349 315 andrew gelman stats-2010-10-03-He doesn’t trust the fit . . . r=.999
Introduction: I received the following question from an education researcher: I was wondering if I could ask you a question about an HLM model I’m working on. The basic design is that we have 5 years of 8th grade student achievement data (standardized test scores, this is the dependent variable), 4th grade test scores, demographics (e.g., gender and ethnicity) and status wrt special ed or ELL, etc.. In addition, we have some school- or second-level information such as school averages of the student information, type of school (grade configuration), enrollment and so. In total there are thousands of students and many schools over the 5 years of information. The model we’re using is quite parsimonious, using only 7 student-level effects and 4 school-level effects. What’s puzzling us is that the correlation between predicted and actual is unrealistically high…r=0.999. We’re using the HPMIXED procedure in SAS but that shouldn’t matter. By dropping variables, obviously we can get the corre
2 0.96524376 2182 andrew gelman stats-2014-01-22-Spell-checking example demonstrates key aspects of Bayesian data analysis
Introduction: One of the new examples for the third edition of Bayesian Data Analysis is a spell-checking story. Here it is (just start at 2/3 down on the first page, with “Spelling correction”). I like this example—it demonstrates the Bayesian algebra, also gives a sense of the way that probability models (both “likelihood” and “prior”) are constructed from existing assumptions and data. The models aren’t just specified as a mathematical exercise, they represent some statement about reality. And the problem is close enough to our experience that we can consider ways in which the model can be criticized and improved, all in a simple example that has only three possibilities.
3 0.96481615 1266 andrew gelman stats-2012-04-16-Another day, another plagiarist
Introduction: This one isn’t actually new, but it’s new to me. It involves University of Michigan business school professor Karl Weick. Here’s the relevant paragraph of Weick’s Wikipedia entry (as of 13 Apr 2012): In several published articles, Weick related a story that originally appeared in a poem by Miroslav Holub that was published in the Times Literary Supplement. Weick plagiarized Holub in that he republished the poem (with some minor differences, including removing line breaks and making small changes in a few words) without quotation or attribution. Some of Weick’s articles included the material with no reference to Holub; others referred to Holub but without indicating that Weick had essentially done a direct copy of Holub’s writing. The plagiarism was detailed in an article by Thomas Basbøll and Henrik Graham. [5] In a response, Weick disputed the claim of plagiarism, writing, “By the time I began to see the Alps story as an example of cognition in the path of the action, I had lo
4 0.96036589 1019 andrew gelman stats-2011-11-19-Validation of Software for Bayesian Models Using Posterior Quantiles
Introduction: I love this stuff : This article presents a simulation-based method designed to establish the computational correctness of software developed to fit a specific Bayesian model, capitalizing on properties of Bayesian posterior distributions. We illustrate the validation technique with two examples. The validation method is shown to find errors in software when they exist and, moreover, the validation output can be informative about the nature and location of such errors. We also compare our method with that of an earlier approach. I hope we can put it into Stan.
5 0.95815361 1980 andrew gelman stats-2013-08-13-Test scores and grades predict job performance (but maybe not at Google)
Introduction: Eric Loken writes : If you’re used to Google upending conventional wisdom, then yesterday’s interview with Laszlo Bock in the New York Times did not disappoint. Google has determined that test scores and transcripts are useless because they don’t predict performance among its employees. . . . I [Loken] am going to assume they’re well aware of the limits of their claim, and instead I’m going say that as readers of the interview we should not lose sight of a fundamental fact - Across a wide variety of employment settings, one of the most robust findings in organizational psychology is that tests of cognitive ability are strong predictors of job performance. If Google has found otherwise, what they have found is that grades and test scores are not predictive of performance at Google. In general, in the workplace tests are still highly predictive of success. If all the research says that test scores and grades predict performance, why would the people at Google want to igno
6 0.95795679 2281 andrew gelman stats-2014-04-04-The Notorious N.H.S.T. presents: Mo P-values Mo Problems
7 0.95761168 697 andrew gelman stats-2011-05-05-A statistician rereads Bill James
8 0.95679665 1278 andrew gelman stats-2012-04-23-“Any old map will do” meets “God is in every leaf of every tree”
9 0.95669091 2058 andrew gelman stats-2013-10-11-Gladwell and Chabris, David and Goliath, and science writing as stone soup
10 0.95651799 257 andrew gelman stats-2010-09-04-Question about standard range for social science correlations
11 0.9564839 288 andrew gelman stats-2010-09-21-Discussion of the paper by Girolami and Calderhead on Bayesian computation
12 0.95598441 2137 andrew gelman stats-2013-12-17-Replication backlash
13 0.95593101 324 andrew gelman stats-2010-10-07-Contest for developing an R package recommendation system
14 0.95578122 187 andrew gelman stats-2010-08-05-Update on state size and governors’ popularity
15 0.95531106 1518 andrew gelman stats-2012-10-02-Fighting a losing battle
16 0.9551757 1016 andrew gelman stats-2011-11-17-I got 99 comparisons but multiplicity ain’t one
17 0.95512354 541 andrew gelman stats-2011-01-27-Why can’t I be more like Bill James, or, The use of default and default-like models
18 0.95493609 154 andrew gelman stats-2010-07-18-Predictive checks for hierarchical models
19 0.95489931 2184 andrew gelman stats-2014-01-24-Parables vs. stories
20 0.9547919 32 andrew gelman stats-2010-05-14-Causal inference in economics