andrew_gelman_stats andrew_gelman_stats-2011 andrew_gelman_stats-2011-544 knowledge-graph by maker-knowledge-mining
Source: html
Introduction: Antonio Rangel writes: I’m a neuroscientist at Caltech . . . I’m using the debate on the ESP paper , as I’m sure other labs around the world are, as an opportunity to discuss some basic statistical issues/ideas w/ my lab. Request: Is there any chance you would be willing to share your thoughts about the difference between exploratory “data mining” studies and confirmatory studies? What I have in mind is that one could use a dataset to explore/discover novel hypotheses and then conduct another experiment to test those hypotheses rigorously. It seems that a good combination of both approaches could be the best of both worlds, since the first would lead to novel hypothesis discovery, and the later to careful testing. . . it is a fundamental issue for neuroscience and psychology. My reply: I know that people talk about this sort of thing . . . but in any real setting, I think I’d want all my data right now to answer any questions I have. I like cross-validation and have used
sentIndex sentText sentNum sentScore
1 Antonio Rangel writes: I’m a neuroscientist at Caltech . [sent-1, score-0.155]
2 I’m using the debate on the ESP paper , as I’m sure other labs around the world are, as an opportunity to discuss some basic statistical issues/ideas w/ my lab. [sent-4, score-0.341]
3 Request: Is there any chance you would be willing to share your thoughts about the difference between exploratory “data mining” studies and confirmatory studies? [sent-5, score-0.488]
4 What I have in mind is that one could use a dataset to explore/discover novel hypotheses and then conduct another experiment to test those hypotheses rigorously. [sent-6, score-1.191]
5 It seems that a good combination of both approaches could be the best of both worlds, since the first would lead to novel hypothesis discovery, and the later to careful testing. [sent-7, score-0.519]
6 it is a fundamental issue for neuroscience and psychology. [sent-10, score-0.202]
7 but in any real setting, I think I’d want all my data right now to answer any questions I have. [sent-14, score-0.111]
8 I like cross-validation and have used it with success, but I don’t think I could bring myself to keep the split so rigorous as you describe. [sent-15, score-0.288]
9 Every once in awhile, the opportunity presents itself, though. [sent-17, score-0.243]
10 We analyzed the 2000 and 2004 elections using the Annenberg polls. [sent-18, score-0.181]
11 But when we were revising Red State Blue State to cover the 2008 election, the Annenberg data weren’t available, so we went with Pew Research polls instead. [sent-19, score-0.421]
12 (The Pew people are great–they post raw data on their website. [sent-20, score-0.201]
13 ) In the meantime, the 2008 Annenberg data have been released, so now we can check our results, once we get mrp all set up to do this. [sent-21, score-0.227]
wordName wordTfidf (topN-words)
[('annenberg', 0.428), ('hypotheses', 0.282), ('pew', 0.253), ('novel', 0.201), ('dataset', 0.17), ('neuroscientist', 0.155), ('antonio', 0.148), ('opportunity', 0.147), ('caltech', 0.143), ('revising', 0.129), ('confirmatory', 0.127), ('neuroscience', 0.123), ('mrp', 0.116), ('worlds', 0.113), ('studies', 0.112), ('mining', 0.112), ('data', 0.111), ('labs', 0.111), ('state', 0.11), ('esp', 0.11), ('split', 0.106), ('conduct', 0.106), ('request', 0.105), ('rigorous', 0.104), ('meantime', 0.101), ('released', 0.1), ('analyzed', 0.098), ('discovery', 0.096), ('presents', 0.096), ('exploratory', 0.094), ('polls', 0.091), ('weren', 0.091), ('raw', 0.09), ('cover', 0.09), ('combination', 0.085), ('elections', 0.083), ('debate', 0.083), ('willing', 0.081), ('approaches', 0.081), ('blue', 0.08), ('fundamental', 0.079), ('awhile', 0.079), ('bring', 0.078), ('success', 0.077), ('careful', 0.077), ('experiment', 0.075), ('lead', 0.075), ('mind', 0.075), ('share', 0.074), ('red', 0.074)]
simIndex simValue blogId blogTitle
same-blog 1 0.99999994 544 andrew gelman stats-2011-01-29-Splitting the data
Introduction: Antonio Rangel writes: I’m a neuroscientist at Caltech . . . I’m using the debate on the ESP paper , as I’m sure other labs around the world are, as an opportunity to discuss some basic statistical issues/ideas w/ my lab. Request: Is there any chance you would be willing to share your thoughts about the difference between exploratory “data mining” studies and confirmatory studies? What I have in mind is that one could use a dataset to explore/discover novel hypotheses and then conduct another experiment to test those hypotheses rigorously. It seems that a good combination of both approaches could be the best of both worlds, since the first would lead to novel hypothesis discovery, and the later to careful testing. . . it is a fundamental issue for neuroscience and psychology. My reply: I know that people talk about this sort of thing . . . but in any real setting, I think I’d want all my data right now to answer any questions I have. I like cross-validation and have used
2 0.17082252 2263 andrew gelman stats-2014-03-24-Empirical implications of Empirical Implications of Theoretical Models
Introduction: Robert Bloomfield writes: Most of the people in my field (accounting, which is basically applied economics and finance, leavened with psychology and organizational behavior) use ‘positive research methods’, which are typically described as coming to the data with a predefined theory, and using hypothesis testing to accept or reject the theory’s predictions. But a substantial minority use ‘interpretive research methods’ (sometimes called qualitative methods, for those that call positive research ‘quantitative’). No one seems entirely happy with the definition of this method, but I’ve found it useful to think of it as an attempt to see the world through the eyes of your subjects, much as Jane Goodall lived with gorillas and tried to see the world through their eyes.) Interpretive researchers often criticize positive researchers by noting that the latter don’t make the best use of their data, because they come to the data with a predetermined theory, and only test a narrow set of h
3 0.15700689 524 andrew gelman stats-2011-01-19-Data exploration and multiple comparisons
Introduction: Bill Harris writes: I’ve read your paper and presentation showing why you don’t usually worry about multiple comparisons. I see how that applies when you are comparing results across multiple settings (states, etc.). Does the same principle hold when you are exploring data to find interesting relationships? For example, you have some data, and you’re trying a series of models to see which gives you the most useful insight. Do you try your models on a subset of the data so you have another subset for confirmatory analysis later, or do you simply throw all the data against your models? My reply: I’d like to estimate all the relationships at once and use a multilevel model to do partial pooling to handle the mutiplicity issues. That said, in practice, in my applied work I’m always bouncing back and forth between different hypotheses and different datasets, and often I learn a lot when next year’s data come in and I can modify my hypotheses. The trouble with the classical
4 0.14207101 2056 andrew gelman stats-2013-10-09-Mister P: What’s its secret sauce?
Introduction: This is a long and technical post on an important topic: the use of multilevel regression and poststratification (MRP) to estimate state-level public opinion. MRP as a research method, and state-level opinion (or, more generally, attitudes in demographic and geographic subpopulation) as a subject, have both become increasingly important in political science—and soon, I expect, will become increasingly important in other social sciences as well. Being able to estimate state-level opinion from national surveys is just such a powerful thing, that if it can be done, people will do it. It’s taken 15 years or so for the method to really catch on, but the ready availability of survey data and of computing power—as well as our increasing comfort level, as a profession, with these techniques, has made MRP become more of a routine research tool. As a method becomes used more and more widely, there will be natural concerns about its domains of applicability. That is the subject of the pres
Introduction: The other day on the sister blog we discussed a recent Pew Research survey that seemed to show that Republicans are becoming more partisan about evolution (or, as Paul Krugman put it, “So what happened after 2009 that might be driving Republican views? . . . Republicans are being driven to identify in all ways with their tribe — and the tribal belief system is dominated by anti-science fundamentalists”). We presented some discussion and evidence from Dan Kahan suggesting that the evidence for such a change was not so clear at all. Kahan drew his conclusions from a more detailed analysis of the much-discussed Pew data, along with a comparison to a recent Gallup poll. Also following up on this is sociologist David Wealiem, who pulls some more data into the discussion: Although the Pew report mentions only the 2009 survey, the question has been asked a number of times since 2005. Here are the results—the numbers represent the percent saying “evolved” minus the percent sayin
6 0.13047092 114 andrew gelman stats-2010-06-28-More on Bayesian deduction-induction
7 0.11537049 2061 andrew gelman stats-2013-10-14-More on Mister P and how it does what it does
8 0.11429407 1570 andrew gelman stats-2012-11-08-Poll aggregation and election forecasting
9 0.10916648 682 andrew gelman stats-2011-04-27-“The ultimate left-wing novel”
10 0.10741259 506 andrew gelman stats-2011-01-06-That silly ESP paper and some silliness in a rebuttal as well
11 0.1073962 70 andrew gelman stats-2010-06-07-Mister P goes on a date
12 0.10504559 1635 andrew gelman stats-2012-12-22-More Pinker Pinker Pinker
13 0.10336392 1529 andrew gelman stats-2012-10-11-Bayesian brains?
14 0.1017383 678 andrew gelman stats-2011-04-25-Democrats do better among the most and least educated groups
15 0.098580636 1695 andrew gelman stats-2013-01-28-Economists argue about Bayes
16 0.098412544 1974 andrew gelman stats-2013-08-08-Statistical significance and the dangerous lure of certainty
17 0.097017385 511 andrew gelman stats-2011-01-11-One more time on that ESP study: The problem of overestimates and the shrinkage solution
18 0.096657336 1512 andrew gelman stats-2012-09-27-A Non-random Walk Down Campaign Street
19 0.096629515 1227 andrew gelman stats-2012-03-23-Voting patterns of America’s whites, from the masses to the elites
20 0.096498594 1605 andrew gelman stats-2012-12-04-Write This Book
topicId topicWeight
[(0, 0.174), (1, -0.019), (2, 0.029), (3, -0.015), (4, -0.018), (5, -0.01), (6, -0.077), (7, -0.015), (8, -0.008), (9, -0.026), (10, 0.026), (11, 0.01), (12, 0.032), (13, -0.078), (14, 0.016), (15, 0.001), (16, -0.008), (17, -0.033), (18, 0.039), (19, -0.008), (20, -0.009), (21, 0.003), (22, -0.029), (23, 0.005), (24, -0.063), (25, -0.037), (26, 0.02), (27, -0.004), (28, 0.074), (29, 0.043), (30, 0.063), (31, -0.077), (32, 0.052), (33, -0.003), (34, -0.013), (35, 0.057), (36, 0.068), (37, -0.011), (38, 0.04), (39, -0.04), (40, 0.012), (41, 0.012), (42, 0.019), (43, 0.035), (44, -0.029), (45, -0.011), (46, 0.006), (47, -0.022), (48, 0.031), (49, -0.01)]
simIndex simValue blogId blogTitle
same-blog 1 0.93864918 544 andrew gelman stats-2011-01-29-Splitting the data
Introduction: Antonio Rangel writes: I’m a neuroscientist at Caltech . . . I’m using the debate on the ESP paper , as I’m sure other labs around the world are, as an opportunity to discuss some basic statistical issues/ideas w/ my lab. Request: Is there any chance you would be willing to share your thoughts about the difference between exploratory “data mining” studies and confirmatory studies? What I have in mind is that one could use a dataset to explore/discover novel hypotheses and then conduct another experiment to test those hypotheses rigorously. It seems that a good combination of both approaches could be the best of both worlds, since the first would lead to novel hypothesis discovery, and the later to careful testing. . . it is a fundamental issue for neuroscience and psychology. My reply: I know that people talk about this sort of thing . . . but in any real setting, I think I’d want all my data right now to answer any questions I have. I like cross-validation and have used
2 0.71251529 2263 andrew gelman stats-2014-03-24-Empirical implications of Empirical Implications of Theoretical Models
Introduction: Robert Bloomfield writes: Most of the people in my field (accounting, which is basically applied economics and finance, leavened with psychology and organizational behavior) use ‘positive research methods’, which are typically described as coming to the data with a predefined theory, and using hypothesis testing to accept or reject the theory’s predictions. But a substantial minority use ‘interpretive research methods’ (sometimes called qualitative methods, for those that call positive research ‘quantitative’). No one seems entirely happy with the definition of this method, but I’ve found it useful to think of it as an attempt to see the world through the eyes of your subjects, much as Jane Goodall lived with gorillas and tried to see the world through their eyes.) Interpretive researchers often criticize positive researchers by noting that the latter don’t make the best use of their data, because they come to the data with a predetermined theory, and only test a narrow set of h
Introduction: Jeff Helzner writes: A friend of mine and I cited your open data article in our attempts to persuade a professor at another institution [Brian Leiter] into releasing the raw data from his influential rankings of philosophy departments. He is now claiming the national security response: . . . disclosing the reputational data would violate the terms on which the evaluators agreed to complete the surveys (did they even bother to read the description of the methodology, one wonders?). I [Helzner] do not find this to be a compelling reply in this case. In fact, I would say that when such data cannot be disclosed it reveals a flaw in the design of the survey. Experimental designs must be open so that others can run the experiment. Mathematical proofs must be open so that they can be reviewed by others. Likewise, it seems to me that the details of statistical argument should be open to inspection. Do you have any thoughts on this? Or do you know of any other leading statistici
4 0.69666672 2295 andrew gelman stats-2014-04-18-One-tailed or two-tailed?
Introduction: Someone writes: Suppose I have two groups of people, A and B, which differ on some characteristic of interest to me; and for each person I measure a single real-valued quantity X. I have a theory that group A has a higher mean value of X than group B. I test this theory by using a t-test. Am I entitled to use a *one-tailed* t-test? Or should I use a *two-tailed* one (thereby giving a p-value that is twice as large)? I know you will probably answer: Forget the t-test; you should use Bayesian methods instead. But what is the standard frequentist answer to this question? My reply: The quick answer here is that different people will do different things here. I would say the 2-tailed p-value is more standard but some people will insist on the one-tailed version, and it’s hard to make a big stand on this one, given all the other problems with p-values in practice: http://www.stat.columbia.edu/~gelman/research/unpublished/p_hacking.pdf http://www.stat.columbia.edu/~gelm
5 0.68670774 1823 andrew gelman stats-2013-04-24-The Tweets-Votes Curve
Introduction: Fabio Rojas points me to this excellently-titled working paper by Joseph DiGrazia, Karissa McKelvey, Johan Bollen, and himself: Is social media a valid indicator of political behavior? We answer this ques- tion using a random sample of 537,231,508 tweets from August 1 to November 1, 2010 and data from 406 competitive U.S. congressional elections provided by the Federal Election Commission. Our results show that the percentage of Republican-candidate name mentions correlates with the Republican vote margin in the subsequent election. This finding persists even when controlling for incumbency, district partisanship, media coverage of the race, time, and demographic variables such as the district’s racial and gender composi- tion. With over 500 million active users in 2012, Twitter now represents a new frontier for the study of human behavior. This research provides a framework for incorporating this emerging medium into the computational social science toolkit. One charming thing
6 0.68131691 1974 andrew gelman stats-2013-08-08-Statistical significance and the dangerous lure of certainty
7 0.67603904 1605 andrew gelman stats-2012-12-04-Write This Book
8 0.67221755 2309 andrew gelman stats-2014-04-28-Crowdstorming a dataset
9 0.66914028 2183 andrew gelman stats-2014-01-23-Discussion on preregistration of research studies
10 0.66747552 1805 andrew gelman stats-2013-04-16-Memo to Reinhart and Rogoff: I think it’s best to admit your errors and go on from there
11 0.66246796 799 andrew gelman stats-2011-07-13-Hypothesis testing with multiple imputations
12 0.66184425 1178 andrew gelman stats-2012-02-21-How many data points do you really have?
13 0.65885544 1289 andrew gelman stats-2012-04-29-We go to war with the data we have, not the data we want
14 0.6519165 242 andrew gelman stats-2010-08-29-The Subtle Micro-Effects of Peacekeeping
15 0.65044427 404 andrew gelman stats-2010-11-09-“Much of the recent reported drop in interstate migration is a statistical artifact”
16 0.64729363 70 andrew gelman stats-2010-06-07-Mister P goes on a date
17 0.64368635 946 andrew gelman stats-2011-10-07-Analysis of Power Law of Participation
18 0.64073229 1681 andrew gelman stats-2013-01-19-Participate in a short survey about the weight of evidence provided by statistics
19 0.63973987 2032 andrew gelman stats-2013-09-20-“Six red flags for suspect work”
20 0.63696325 162 andrew gelman stats-2010-07-25-Darn that Lindsey Graham! (or, “Mr. P Predicts the Kagan vote”)
topicId topicWeight
[(5, 0.013), (10, 0.022), (15, 0.013), (16, 0.086), (21, 0.031), (24, 0.106), (56, 0.012), (63, 0.226), (80, 0.017), (86, 0.033), (99, 0.348)]
simIndex simValue blogId blogTitle
1 0.9818942 313 andrew gelman stats-2010-10-03-A question for psychometricians
Introduction: Don Coffin writes: A colleague of mine and I are doing a presentation for new faculty on a number of topics related to teaching. Our charge is to identify interesting issues and to find research-based information for them about how to approach things. So, what I wondered is, do you know of any published research dealing with the sort of issues about structuring a course and final exam in the ways you talk about in this blog post ? Some poking around in the usual places hasn’t turned anything up yet. I don’t really know the psychometrics literature but I imagine that some good stuff has been written on principles of test design. There are probably some good papers from back in the 1920s. Can anyone supply some references?
Introduction: The official announcement: The Excellence in Statistical Reporting Award for 2010 is presented to Felix Salmon for his body of work, which exemplifies the highest standards of scientific reporting. His insightful use of statistics as a tool to understanding the world of business and economics, areas that are critical in today’s economy, sets a new standard in statistical investigative reporting. Here are some examples: Tiger Woods Nigerian spammers How the government fudges job statistics This one is important to me. The idea is that “statistical reporting” is not just traditional science reporting (journalist talks with scientists and tries to understand the consensus) or science popularization or silly feature stories about the lottery. Salmon is doing investigative reporting using statistical thinking. Also, from a political angle, Salmon’s smart and quantitatively sophisticated work (as well as that of others such as Nate Silver) is an important counterweigh
3 0.97504544 1621 andrew gelman stats-2012-12-13-Puzzles of criminal justice
Introduction: Four recent news stories about crime and punishment made me realize, yet again, how little I understand all this. 1. “HSBC to Pay $1.92 Billion to Settle Charges of Money Laundering” : State and federal authorities decided against indicting HSBC in a money-laundering case over concerns that criminal charges could jeopardize one of the world’s largest banks and ultimately destabilize the global financial system. Instead, HSBC announced on Tuesday that it had agreed to a record $1.92 billion settlement with authorities. . . . I don’t understand this idea of punishing the institution. I have the same problem when the NCAA punishes a college football program. These are individual people breaking the law (or the rules), right? So why not punish them directly? Giving 40 lashes to a bunch of HSBC executives and garnisheeing their salaries for life, say, that wouldn’t destabilize the global financial system would it? From the article: “A money-laundering indictment, or a guilt
4 0.96846485 782 andrew gelman stats-2011-06-29-Putting together multinomial discrete regressions by combining simple logits
Introduction: When predicting 0/1 data we can use logit (or probit or robit or some other robust model such as invlogit (0.01 + 0.98*X*beta)). Logit is simple enough and we can use bayesglm to regularize and avoid the problem of separation. What if there are more than 2 categories? If they’re ordered (1, 2, 3, etc), we can do ordered logit (and use bayespolr() to avoid separation). If the categories are unordered (vanilla, chocolate, strawberry), there are unordered multinomial logit and probit models out there. But it’s not so easy to fit these multinomial model in a multilevel setting (with coefficients that vary by group), especially if the computation is embedded in an iterative routine such as mi where you have real time constraints at each step. So this got me wondering whether we could kluge it with logits. Here’s the basic idea (in the ordered and unordered forms): - If you have a variable that goes 1, 2, 3, etc., set up a series of logits: 1 vs. 2,3,…; 2 vs. 3,…; and so forth
5 0.96754551 1480 andrew gelman stats-2012-09-02-“If our product is harmful . . . we’ll stop making it.”
Introduction: After our discussion of the sad case of Darrell Huff, the celebrated “How to Lie with Statistics” guy who had a lucrative side career disparaging the link between smoking and cancer, I was motivated to follow John Mashey’s recommendation and read the book, Golden Holocaust: Origins of the Cigarette Catastrophe and the Case for Abolition, by historian Robert Proctor. My first stop upon receiving the book was the index, in particular the entry for Rubin, Donald B. I followed the reference to pages 440-442 and found the description of Don’s activities to be accurate, neither diminished nor overstated, to the best of my knowledge. Rubin is the second-most-famous statistician to have been paid by the cigarette industry, but several other big and small names have been on the payroll at one time or another. Here’s a partial list . Just including the people I know or have heard of: Herbert Solomon, Stanford Richard Tweedie, Bond U Arnold Zellner, U of Chicago Paul Switzer, S
6 0.96453655 628 andrew gelman stats-2011-03-25-100-year floods
7 0.96348113 1484 andrew gelman stats-2012-09-05-Two exciting movie ideas: “Second Chance U” and “The New Dirty Dozen”
8 0.96198177 293 andrew gelman stats-2010-09-23-Lowess is great
9 0.94913602 1078 andrew gelman stats-2011-12-22-Tables as graphs: The Ramanujan principle
10 0.94441485 745 andrew gelman stats-2011-06-04-High-level intellectual discussions in the Columbia statistics department
11 0.94383955 739 andrew gelman stats-2011-05-31-When Did Girls Start Wearing Pink?
12 0.94347787 1201 andrew gelman stats-2012-03-07-Inference = data + model
13 0.94127941 126 andrew gelman stats-2010-07-03-Graphical presentation of risk ratios
14 0.94080865 286 andrew gelman stats-2010-09-20-Are the Democrats avoiding a national campaign?
same-blog 15 0.93991399 544 andrew gelman stats-2011-01-29-Splitting the data
16 0.93978554 102 andrew gelman stats-2010-06-21-Why modern art is all in the mind
17 0.93615222 2103 andrew gelman stats-2013-11-16-Objects of the class “Objects of the class”
18 0.91710818 1690 andrew gelman stats-2013-01-23-When are complicated models helpful in psychology research and when are they overkill?
19 0.91684663 460 andrew gelman stats-2010-12-09-Statistics gifts?
20 0.9127441 421 andrew gelman stats-2010-11-19-Just chaid