andrew_gelman_stats andrew_gelman_stats-2013 andrew_gelman_stats-2013-1690 knowledge-graph by maker-knowledge-mining
Source: html
Introduction: Nick Brown is bothered by this article , “An unscented Kalman filter approach to the estimation of nonlinear dynamical systems models,” by Sy-Miin Chow, Emilio Ferrer, and John Nesselroade. The introduction of the article cites a bunch of articles in serious psych/statistics journals. The question is, are such advanced statistical techniques really needed, or even legitimate, with the kind of very rough data that is usually available in psych applications? Or is it just fishing in the hope of discovering patterns that are not really there? I wrote: It seems like a pretty innocuous literature review. I agree that many of the applications are silly (for example, they cite the work of the notorious John Gottman in fitting a predator-prey model to spousal relations (!)), but overall they just seem to be presenting very standard ideas for the mathematical-psychology audience. It’s not clear whether advanced techniques are always appropriate here, but they come in through a natura
sentIndex sentText sentNum sentScore
1 Nick Brown is bothered by this article , “An unscented Kalman filter approach to the estimation of nonlinear dynamical systems models,” by Sy-Miin Chow, Emilio Ferrer, and John Nesselroade. [sent-1, score-0.077]
2 The question is, are such advanced statistical techniques really needed, or even legitimate, with the kind of very rough data that is usually available in psych applications? [sent-3, score-0.254]
3 Or is it just fishing in the hope of discovering patterns that are not really there? [sent-4, score-0.29]
4 I agree that many of the applications are silly (for example, they cite the work of the notorious John Gottman in fitting a predator-prey model to spousal relations (! [sent-6, score-0.083]
5 Ultimately you can get pretty big complicated models because the alternative is even worse. [sent-9, score-0.377]
6 Two areas in psychology where it does seem to make sense to use complicated models are: (1) personality types (we really are complicated multidimensional people) and (2) educational testing (where many different skills and abilities are tested at once). [sent-11, score-0.781]
7 Keep in mind that advances often represent bold and risky departures from current understanding. [sent-13, score-0.275]
8 Second, be open to capitalize on the rapid advances in measurement tools and mathematical and statistical models. [sent-17, score-0.28]
9 advances, while maintaining empirical and methodological rigor, emotion scientists working in positive psychology will be better equipped than ever before to find practical answers to age-old questions about what makes life good. [sent-21, score-0.431]
10 It seems to me that the authors are encouraging their readers to go on fishing expeditions, much like Bem (2000) (literally) did in a paragraph that Wagenmakers et al cited in a review of Bem’s appalling “psi is real” paper in 2011: Examine [the data] from every angle. [sent-22, score-0.416]
11 If a datum suggests a new hypothesis, try to find further evidencefor it elsewhere in the data. [sent-25, score-0.08]
12 If you see dim traces of interesting patterns, try to reorganize the data to bring them into bolder relief. [sent-26, score-0.077]
13 If there are participants you don’t like, or trials, observers, or interviewers who gave you anomalous results,place them aside temporarily and see if any coherent patterns emerge. [sent-27, score-0.39]
14 Chow et al’s use of some pre-existing empirical data to “validate” their model – the data coming from a previous study by Nesselroade – suggests to me that there is some cherrypicking going on. [sent-29, score-0.236]
15 Without confirmatory analysis on a fresh data set, this is just so much circular reasoning. [sent-30, score-0.08]
16 To which I replied: There’s a lot of misunderstanding here but I don’t think the paper you sent is particularly bad, it’s just part of a general attitude people have that there is a high-tech solution to any problem. [sent-31, score-0.087]
17 As I wrote a couple years ago, the problem, I think, is that they (like many economists) think of statistical methods not as a tool for learning but as a tool for rigor. [sent-37, score-0.259]
18 So they gravitate toward math-heavy methods based on testing, asymptotics, and abstract theories, rather than toward complex modeling. [sent-38, score-0.154]
19 The result is a disconnect between statistical methods and applied goals. [sent-39, score-0.077]
20 For the psychologists you’re looking at, the problem is somewhat different: they do want to use statistics to learn, they’re just willing to learn things that aren’t true. [sent-40, score-0.088]
wordName wordTfidf (topN-words)
[('chow', 0.25), ('complicated', 0.222), ('advances', 0.192), ('patterns', 0.16), ('advanced', 0.158), ('models', 0.155), ('economists', 0.151), ('fishing', 0.13), ('al', 0.123), ('nick', 0.122), ('bem', 0.113), ('view', 0.107), ('psychology', 0.102), ('techniques', 0.096), ('tool', 0.091), ('kalman', 0.088), ('appalling', 0.088), ('capitalize', 0.088), ('emotion', 0.088), ('expeditions', 0.088), ('recurrent', 0.088), ('willing', 0.088), ('attitude', 0.087), ('applications', 0.083), ('bannerjee', 0.083), ('departures', 0.083), ('equipped', 0.083), ('replied', 0.083), ('empirical', 0.081), ('testing', 0.08), ('suggests', 0.08), ('duflo', 0.08), ('objectionable', 0.08), ('anomalous', 0.08), ('circular', 0.08), ('progression', 0.077), ('traces', 0.077), ('asymptotics', 0.077), ('gravitate', 0.077), ('maintaining', 0.077), ('dynamical', 0.077), ('psi', 0.077), ('armed', 0.077), ('expedition', 0.077), ('methods', 0.077), ('et', 0.075), ('interviewers', 0.075), ('gottman', 0.075), ('temporarily', 0.075), ('observers', 0.075)]
simIndex simValue blogId blogTitle
same-blog 1 0.99999958 1690 andrew gelman stats-2013-01-23-When are complicated models helpful in psychology research and when are they overkill?
Introduction: Nick Brown is bothered by this article , “An unscented Kalman filter approach to the estimation of nonlinear dynamical systems models,” by Sy-Miin Chow, Emilio Ferrer, and John Nesselroade. The introduction of the article cites a bunch of articles in serious psych/statistics journals. The question is, are such advanced statistical techniques really needed, or even legitimate, with the kind of very rough data that is usually available in psych applications? Or is it just fishing in the hope of discovering patterns that are not really there? I wrote: It seems like a pretty innocuous literature review. I agree that many of the applications are silly (for example, they cite the work of the notorious John Gottman in fitting a predator-prey model to spousal relations (!)), but overall they just seem to be presenting very standard ideas for the mathematical-psychology audience. It’s not clear whether advanced techniques are always appropriate here, but they come in through a natura
2 0.21042864 241 andrew gelman stats-2010-08-29-Ethics and statistics in development research
Introduction: From Bannerjee and Duflo, “The Experimental Approach to Development Economics,” Annual Review of Economics (2009): One issue with the explicit acknowledgment of randomization as a fair way to allocate the program is that implementers may find that the easiest way to present it to the community is to say that an expansion of the program is planned for the control areas in the future (especially when such is indeed the case, as in phased-in design). I can’t quite figure out whether Bannerjee and Duflo are saying that they would lie and tell people that an expansion is planned when it isn’t, or whether they’re deploring that other people do it. I’m not bothered by a lot of the deception in experimental research–for example, I think the Milgram obedience experiment was just fine–but somehow the above deception bothers me. It just seems wrong to tell people that an expansion is planned if it’s not. P.S. Overall the article is pretty good. My only real problem with it is that
Introduction: Some things I respect When it comes to meta-models of statistics, here are two philosophies that I respect: 1. (My) Bayesian approach, which I associate with E. T. Jaynes, in which you construct models with strong assumptions, ride your models hard, check their fit to data, and then scrap them and improve them as necessary. 2. At the other extreme, model-free statistical procedures that are designed to work well under very weak assumptions—for example, instead of assuming a distribution is Gaussian, you would just want the procedure to work well under some conditions on the smoothness of the second derivative of the log density function. Both the above philosophies recognize that (almost) all important assumptions will be wrong, and they resolve this concern via aggressive model checking or via robustness. And of course there are intermediate positions, such as working with Bayesian models that have been shown to be robust, and then still checking them. Or, to flip it arou
Introduction: Chris Masse points me to this response by Daryl Bem and two statisticians (Jessica Utts and Wesley Johnson) to criticisms by Wagenmakers et.al. of Bem’s recent ESP study. I have nothing to add but would like to repeat a couple bits of my discussions of last month, of here : Classical statistical methods that work reasonably well when studying moderate or large effects (see the work of Fisher, Snedecor, Cochran, etc.) fall apart in the presence of small effects. I think it’s naive when people implicitly assume that the study’s claims are correct, or the study’s statistical methods are weak. Generally, the smaller the effects you’re studying, the better the statistics you need. ESP is a field of small effects and so ESP researchers use high-quality statistics. To put it another way: whatever methodological errors happen to be in the paper in question, probably occur in lots of researcher papers in “legitimate” psychology research. The difference is that when you’re studying a
5 0.13913466 1392 andrew gelman stats-2012-06-26-Occam
Introduction: Cosma Shalizi and Larry Wasserman discuss some papers from a conference on Ockham’s Razor. I don’t have anything new to add on this so let me link to past blog entries on the topic and repost the following from 2004 : A lot has been written in statistics about “parsimony”—that is, the desire to explain phenomena using fewer parameters–but I’ve never seen any good general justification for parsimony. (I don’t count “Occam’s Razor,” or “Ockham’s Razor,” or whatever, as a justification. You gotta do better than digging up a 700-year-old quote.) Maybe it’s because I work in social science, but my feeling is: if you can approximate reality with just a few parameters, fine. If you can use more parameters to fold in more information, that’s even better. In practice, I often use simple models—because they are less effort to fit and, especially, to understand. But I don’t kid myself that they’re better than more complicated efforts! My favorite quote on this comes from Rad
6 0.1352596 1695 andrew gelman stats-2013-01-28-Economists argue about Bayes
7 0.12689263 2177 andrew gelman stats-2014-01-19-“The British amateur who debunked the mathematics of happiness”
8 0.12583417 2254 andrew gelman stats-2014-03-18-Those wacky anti-Bayesians used to be intimidating, but now they’re just pathetic
9 0.12475857 2263 andrew gelman stats-2014-03-24-Empirical implications of Empirical Implications of Theoretical Models
10 0.1245743 1139 andrew gelman stats-2012-01-26-Suggested resolution of the Bem paradox
11 0.1242195 1482 andrew gelman stats-2012-09-04-Model checking and model understanding in machine learning
12 0.12399331 506 andrew gelman stats-2011-01-06-That silly ESP paper and some silliness in a rebuttal as well
13 0.11683498 291 andrew gelman stats-2010-09-22-Philosophy of Bayes and non-Bayes: A dialogue with Deborah Mayo
14 0.11348155 511 andrew gelman stats-2011-01-11-One more time on that ESP study: The problem of overestimates and the shrinkage solution
15 0.11295486 244 andrew gelman stats-2010-08-30-Useful models, model checking, and external validation: a mini-discussion
16 0.11220151 1719 andrew gelman stats-2013-02-11-Why waste time philosophizing?
17 0.1103368 2245 andrew gelman stats-2014-03-12-More on publishing in journals
18 0.10943525 2172 andrew gelman stats-2014-01-14-Advice on writing research articles
19 0.10883291 888 andrew gelman stats-2011-09-03-A psychology researcher asks: Is Anova dead?
20 0.10866974 1998 andrew gelman stats-2013-08-25-A new Bem theory
topicId topicWeight
[(0, 0.269), (1, 0.039), (2, -0.049), (3, -0.032), (4, -0.023), (5, 0.002), (6, -0.058), (7, -0.021), (8, 0.071), (9, 0.051), (10, -0.019), (11, 0.005), (12, -0.027), (13, -0.039), (14, 0.006), (15, -0.013), (16, -0.025), (17, -0.01), (18, -0.0), (19, -0.022), (20, 0.016), (21, -0.06), (22, -0.028), (23, 0.008), (24, -0.037), (25, -0.03), (26, 0.018), (27, -0.007), (28, 0.011), (29, -0.031), (30, -0.02), (31, 0.067), (32, 0.037), (33, -0.05), (34, 0.003), (35, -0.037), (36, -0.016), (37, 0.017), (38, -0.045), (39, -0.007), (40, -0.016), (41, 0.046), (42, 0.006), (43, 0.031), (44, 0.017), (45, 0.018), (46, -0.074), (47, -0.041), (48, 0.03), (49, -0.004)]
simIndex simValue blogId blogTitle
same-blog 1 0.97458965 1690 andrew gelman stats-2013-01-23-When are complicated models helpful in psychology research and when are they overkill?
Introduction: Nick Brown is bothered by this article , “An unscented Kalman filter approach to the estimation of nonlinear dynamical systems models,” by Sy-Miin Chow, Emilio Ferrer, and John Nesselroade. The introduction of the article cites a bunch of articles in serious psych/statistics journals. The question is, are such advanced statistical techniques really needed, or even legitimate, with the kind of very rough data that is usually available in psych applications? Or is it just fishing in the hope of discovering patterns that are not really there? I wrote: It seems like a pretty innocuous literature review. I agree that many of the applications are silly (for example, they cite the work of the notorious John Gottman in fitting a predator-prey model to spousal relations (!)), but overall they just seem to be presenting very standard ideas for the mathematical-psychology audience. It’s not clear whether advanced techniques are always appropriate here, but they come in through a natura
2 0.86417663 241 andrew gelman stats-2010-08-29-Ethics and statistics in development research
Introduction: From Bannerjee and Duflo, “The Experimental Approach to Development Economics,” Annual Review of Economics (2009): One issue with the explicit acknowledgment of randomization as a fair way to allocate the program is that implementers may find that the easiest way to present it to the community is to say that an expansion of the program is planned for the control areas in the future (especially when such is indeed the case, as in phased-in design). I can’t quite figure out whether Bannerjee and Duflo are saying that they would lie and tell people that an expansion is planned when it isn’t, or whether they’re deploring that other people do it. I’m not bothered by a lot of the deception in experimental research–for example, I think the Milgram obedience experiment was just fine–but somehow the above deception bothers me. It just seems wrong to tell people that an expansion is planned if it’s not. P.S. Overall the article is pretty good. My only real problem with it is that
3 0.85920352 421 andrew gelman stats-2010-11-19-Just chaid
Introduction: Reading somebody else’s statistics rant made me realize the inherent contradictions in much of my own statistical advice. Jeff Lax sent along this article by Philip Schrodt, along with the cryptic comment: Perhaps of interest to you. perhaps not. Not meant to be an excuse for you to rant against hypothesis testing again. In his article, Schrodt makes a reasonable and entertaining argument against the overfitting of data and the overuse of linear models. He states that his article is motivated by the quantitative papers he has been sent to review for journals or conferences, and he explicitly excludes “studies of United States voting behavior,” so at least I think Mister P is off the hook. I notice a bit of incoherence in Schrodt’s position–on one hand, he criticizes “kitchen-sink models” for overfitting and he criticizes “using complex methods without understanding the underlying assumptions” . . . but then later on he suggests that political scientists in this countr
4 0.85112 1861 andrew gelman stats-2013-05-17-Where do theories come from?
Introduction: Lee Sechrest sends along this article by Brian Haig and writes that it “presents what seems to me a useful perspective on much of what scientists/statisticians do and how science works, at least in the fields in which I work.” Here’s Haig’s abstract: A broad theory of scientific method is sketched that has particular relevance for the behavioral sciences. This theory of method assembles a complex of specific strategies and methods that are used in the detection of empirical phenomena and the subsequent construction of explanatory theories. A characterization of the nature of phenomena is given, and the process of their detection is briefly described in terms of a multistage model of data analysis. The construction of explanatory theories is shown to involve their generation through abductive, or explanatory, reasoning, their development through analogical modeling, and their fuller appraisal in terms of judgments of the best of competing explanations. The nature and limits of
Introduction: David Duvenaud writes: I’ve been following your recent discussions about how an AI could do statistics [see also here ]. I was especially excited about your suggestion for new statistical methods using “a language-like approach to recursively creating new models from a specified list of distributions and transformations, and an automatic approach to checking model fit.” Your discussion of these ideas was exciting to me and my colleagues because we recently did some work taking a step in this direction, automatically searching through a grammar over Gaussian process regression models. Roger Grosse previously did the same thing , but over matrix decomposition models using held-out predictive likelihood to check model fit. These are both examples of automatic Bayesian model-building by a search over more and more complex models, as you suggested. One nice thing is that both grammars include lots of standard models for free, and they seem to work pretty well, although the
7 0.80523056 2127 andrew gelman stats-2013-12-08-The never-ending (and often productive) race between theory and practice
8 0.80507821 1880 andrew gelman stats-2013-06-02-Flame bait
10 0.80406666 757 andrew gelman stats-2011-06-10-Controversy over the Christakis-Fowler findings on the contagion of obesity
11 0.7998274 789 andrew gelman stats-2011-07-07-Descriptive statistics, causal inference, and story time
12 0.79941332 496 andrew gelman stats-2011-01-01-Tukey’s philosophy
13 0.79815006 1482 andrew gelman stats-2012-09-04-Model checking and model understanding in machine learning
15 0.781416 738 andrew gelman stats-2011-05-30-Works well versus well understood
16 0.78061038 1740 andrew gelman stats-2013-02-26-“Is machine learning a subset of statistics?”
17 0.77570212 1200 andrew gelman stats-2012-03-06-Some economists are skeptical about microfoundations
18 0.77294528 1680 andrew gelman stats-2013-01-18-“If scientists wrote horoscopes, this is what yours would say”
19 0.76906234 744 andrew gelman stats-2011-06-03-Statistical methods for healthcare regulation: rating, screening and surveillance
20 0.76793063 1195 andrew gelman stats-2012-03-04-Multiple comparisons dispute in the tabloids
topicId topicWeight
[(0, 0.011), (15, 0.043), (16, 0.064), (21, 0.029), (24, 0.165), (45, 0.02), (59, 0.015), (63, 0.179), (77, 0.025), (99, 0.326)]
simIndex simValue blogId blogTitle
1 0.98679852 1621 andrew gelman stats-2012-12-13-Puzzles of criminal justice
Introduction: Four recent news stories about crime and punishment made me realize, yet again, how little I understand all this. 1. “HSBC to Pay $1.92 Billion to Settle Charges of Money Laundering” : State and federal authorities decided against indicting HSBC in a money-laundering case over concerns that criminal charges could jeopardize one of the world’s largest banks and ultimately destabilize the global financial system. Instead, HSBC announced on Tuesday that it had agreed to a record $1.92 billion settlement with authorities. . . . I don’t understand this idea of punishing the institution. I have the same problem when the NCAA punishes a college football program. These are individual people breaking the law (or the rules), right? So why not punish them directly? Giving 40 lashes to a bunch of HSBC executives and garnisheeing their salaries for life, say, that wouldn’t destabilize the global financial system would it? From the article: “A money-laundering indictment, or a guilt
2 0.98659742 782 andrew gelman stats-2011-06-29-Putting together multinomial discrete regressions by combining simple logits
Introduction: When predicting 0/1 data we can use logit (or probit or robit or some other robust model such as invlogit (0.01 + 0.98*X*beta)). Logit is simple enough and we can use bayesglm to regularize and avoid the problem of separation. What if there are more than 2 categories? If they’re ordered (1, 2, 3, etc), we can do ordered logit (and use bayespolr() to avoid separation). If the categories are unordered (vanilla, chocolate, strawberry), there are unordered multinomial logit and probit models out there. But it’s not so easy to fit these multinomial model in a multilevel setting (with coefficients that vary by group), especially if the computation is embedded in an iterative routine such as mi where you have real time constraints at each step. So this got me wondering whether we could kluge it with logits. Here’s the basic idea (in the ordered and unordered forms): - If you have a variable that goes 1, 2, 3, etc., set up a series of logits: 1 vs. 2,3,…; 2 vs. 3,…; and so forth
3 0.9864561 313 andrew gelman stats-2010-10-03-A question for psychometricians
Introduction: Don Coffin writes: A colleague of mine and I are doing a presentation for new faculty on a number of topics related to teaching. Our charge is to identify interesting issues and to find research-based information for them about how to approach things. So, what I wondered is, do you know of any published research dealing with the sort of issues about structuring a course and final exam in the ways you talk about in this blog post ? Some poking around in the usual places hasn’t turned anything up yet. I don’t really know the psychometrics literature but I imagine that some good stuff has been written on principles of test design. There are probably some good papers from back in the 1920s. Can anyone supply some references?
4 0.97972989 1484 andrew gelman stats-2012-09-05-Two exciting movie ideas: “Second Chance U” and “The New Dirty Dozen”
Introduction: I have a great idea for a movie. Actually two movies based on two variants of a similar idea. It all started when I saw this story: Dr. Anil Potti, the controversial cancer researcher whose work at Duke University led to lawsuits from patients, is now a medical oncologist at the Cancer Center of North Dakota in Grand Forks. When asked about Dr. Potti’s controversial appointment, his new boss said : If a guy can’t get a second chance here in North Dakota, where he trained, man, you can’t get a second chance anywhere. (Link from Retraction Watch , of course.) Potti’s boss is also quoted as saying, “Most, if not all, his patients have loved him.” On the other hand, the news article reports: “The North Carolina medical board’s website lists settlements against Potti of at least $75,000.” I guess there’s no reason you can’t love a guy and still want a juicy malpractice settlement. Second Chance U I don’t give two poops about Dr. Anil Potti. But seeing the above s
5 0.97865975 102 andrew gelman stats-2010-06-21-Why modern art is all in the mind
Introduction: This looks cool: Ten years ago researchers in America took two groups of three-year-olds and showed them a blob of paint on a canvas. Children who were told that the marks were the result of an accidental spillage showed little interest. The others, who had been told that the splodge of colour had been carefully created for them, started to refer to it as “a painting”. Now that experiment . . . has gone on to form part of the foundation of an influential new book that questions the way in which we respond to art. . . . The book, which is subtitled The New Science of Why We Like What We Like, is not an attack on modern or contemporary art and Bloom says fans of more traditional art are not capable of making purely aesthetic judgments either. “I don’t have a strong position about the art itself,” he said this weekend. “But I do have a strong position about why we actually like it.” This sounds fascinating. But I’m skeptical about this part: Humans are incapable of just getti
6 0.9779743 293 andrew gelman stats-2010-09-23-Lowess is great
7 0.97504342 1078 andrew gelman stats-2011-12-22-Tables as graphs: The Ramanujan principle
8 0.96993631 1480 andrew gelman stats-2012-09-02-“If our product is harmful . . . we’ll stop making it.”
9 0.95838338 745 andrew gelman stats-2011-06-04-High-level intellectual discussions in the Columbia statistics department
same-blog 11 0.94846749 1690 andrew gelman stats-2013-01-23-When are complicated models helpful in psychology research and when are they overkill?
12 0.94755077 286 andrew gelman stats-2010-09-20-Are the Democrats avoiding a national campaign?
13 0.94447547 2103 andrew gelman stats-2013-11-16-Objects of the class “Objects of the class”
14 0.94434917 544 andrew gelman stats-2011-01-29-Splitting the data
15 0.93833393 1506 andrew gelman stats-2012-09-21-Building a regression model . . . with only 27 data points
16 0.93674833 291 andrew gelman stats-2010-09-22-Philosophy of Bayes and non-Bayes: A dialogue with Deborah Mayo
17 0.93509722 421 andrew gelman stats-2010-11-19-Just chaid
18 0.93348038 1201 andrew gelman stats-2012-03-07-Inference = data + model
19 0.93165541 428 andrew gelman stats-2010-11-24-Flawed visualization of U.S. voting maybe has some good features
20 0.93041986 2148 andrew gelman stats-2013-12-25-Spam!