andrew_gelman_stats andrew_gelman_stats-2010 andrew_gelman_stats-2010-105 knowledge-graph by maker-knowledge-mining
Source: html
Introduction: A few months ago, I blogged on John Gottman, a psychologist whose headline-grabbing research on marriages (he got himself featured in Blink with a claim that he could predict with 83 percent accuracy whether a couple would be divorced—after meeting with them for 15 minutes!) was recently debunked in a book by Laurie Abraham. The question I raised was: how could someone who was evidently so intelligent and accomplished—Gottman, that is—get things so wrong? My brief conclusion was that once you have some success, I guess there’s not much of a motivation to change your ways. Also, I could well believe that, for all its flaws, Gottman’s work is better than much of the other research out there on marriages. There’s still the question of how this stuff gets published in scientific journals. I haven’t looked at Gottman’s articles in detail and so don’t really have thoughts on that one. Anyway, I recently corresponded with a mathematician who had heard of Gottman’s research and wrote
sentIndex sentText sentNum sentScore
1 A few months ago, I blogged on John Gottman, a psychologist whose headline-grabbing research on marriages (he got himself featured in Blink with a claim that he could predict with 83 percent accuracy whether a couple would be divorced—after meeting with them for 15 minutes! [sent-1, score-0.394]
2 I think she drops the ball, though, in the section you quote about false negatives and false positives. [sent-11, score-0.365]
3 ” But if he’s trying to predict early divorce, an event with 16% prevalence, I think it’s safe to say he does NOT mean “My method assigns a prediction of divorce to 20% of couples that didn’t actually divorce” — presumably the TOTAL number of positives his method gives is somewhere around 16%. [sent-13, score-1.103]
4 A much more natural interpretation of what he might mean by “80% accurate” is “the method gives the the right answer in 80% of cases” — of course, it would be richer information to report the false positive and false negative rate separately. [sent-14, score-0.397]
5 The mathematician then replied to me: A psychologist colleague convinced me my initial reaction was too hard on Gottman. [sent-18, score-0.211]
6 ) He finds the optimal linear combination of x_1 … x_n for predicting divorce, and similarly for y_1, … y_n. [sent-24, score-0.276]
7 He doesn’t report rates of false positives and false negatives separately, but he does report correlation between his linear combination of y_1, … y_n and (I think) the Bernoulli variable of divorce/not divorce, and gets something highly positive (with x_1, … x_n, it’s just slightly positive. [sent-27, score-0.836]
8 ) So that’s not consistent with, say, his model always predicting no divorce, or predicting that a randomly chosen 16% of the couples will divorce, which would give correlations of 0. [sent-28, score-0.321]
9 It IS consistent, I think, with numbers like Abraham’s, in which half of the couples he predicts will divorce actually stay together. [sent-29, score-0.839]
10 But I don’t think this really counts as remaking the model each time to fit the data; he has committed to this small set of variables, and replicates the finding that these variables explain more of the variation in some variable of interest than do competing sets of measurements. [sent-33, score-0.258]
11 I’m just saying that it doesn’t seem to be methodologically necessary that he commits himself after his first paper to a specific linear combination of y_1, … y_n which is to be his divorce prediction variable for all time. [sent-37, score-0.651]
12 A relevant question is: if you have a binary variable, and n other measurements, presumably you should expect to be able to find SOME linear combination of y_1 … y_n that has a healthy positive correlation with the binary variable. [sent-39, score-0.561]
13 But how positive a correlation does one expect to find, for a given n, under a null hypothesis that the y_i are actually independent from the binary variable? [sent-40, score-0.304]
14 Part of the problem I guess is that his y_i are presumably correlated with each other and I guess that affects the answer. [sent-42, score-0.199]
15 I guess my sense after looking at this stuff is that there’s really nothing wrong with Gottman’s published work, but that there is something wrong with Gladwell’s description of it, and to be honest, probably something wrong with Gottman’s own non-academic sales pitch for his own results. [sent-44, score-0.321]
16 I think the correct statistical critique is just the very basic one that the claim as written is empty: that if you want to predict whether a couple will divorce or stay together with 80% accuracy, you can just predict every couple will stay together. [sent-46, score-1.146]
17 To a statistician, it’s obvious that you can predict divorces to high accuracy over a 3-yr period by just saying that everyone will stay married, and those numbers such as 80% or 93% will raise suspicion. [sent-49, score-0.528]
18 Recall this quote from Gottman’s collabroator, James Murray: “The forecast of who would get divorced in his study of 700 couples over 12 years was 100 per cent correct, he said. [sent-51, score-0.29]
19 But “what reduced the accuracy of our predictions was those couples who we thought would stay married and unhappy actually ended up getting divorced”. [sent-52, score-0.503]
20 To put it another way: Gottman’s work could very well be useful, even if he doesn’t predict divorces at all, he could be developing good techniques for marriage counselors. [sent-54, score-0.33]
wordName wordTfidf (topN-words)
[('gottman', 0.591), ('divorce', 0.409), ('abraham', 0.251), ('couples', 0.191), ('predict', 0.172), ('stay', 0.123), ('mathematician', 0.119), ('false', 0.114), ('divorced', 0.099), ('accuracy', 0.088), ('combination', 0.085), ('variable', 0.084), ('positive', 0.081), ('divorces', 0.074), ('negatives', 0.074), ('linear', 0.073), ('numbers', 0.071), ('binary', 0.069), ('presumably', 0.067), ('positives', 0.066), ('guess', 0.066), ('predicting', 0.065), ('wrong', 0.064), ('stuff', 0.063), ('think', 0.063), ('variables', 0.06), ('correlation', 0.059), ('question', 0.058), ('married', 0.056), ('dots', 0.054), ('finds', 0.053), ('doesn', 0.052), ('competing', 0.051), ('looked', 0.051), ('honest', 0.05), ('psychologist', 0.05), ('anyway', 0.05), ('hypothesis', 0.05), ('kind', 0.046), ('method', 0.045), ('actually', 0.045), ('report', 0.043), ('detail', 0.043), ('couple', 0.042), ('papers', 0.042), ('could', 0.042), ('colleague', 0.042), ('coefficients', 0.042), ('scientific', 0.041), ('winging', 0.041)]
simIndex simValue blogId blogTitle
same-blog 1 1.0000002 105 andrew gelman stats-2010-06-23-More on those divorce prediction statistics, including a discussion of the innumeracy of (some) mathematicians
Introduction: A few months ago, I blogged on John Gottman, a psychologist whose headline-grabbing research on marriages (he got himself featured in Blink with a claim that he could predict with 83 percent accuracy whether a couple would be divorced—after meeting with them for 15 minutes!) was recently debunked in a book by Laurie Abraham. The question I raised was: how could someone who was evidently so intelligent and accomplished—Gottman, that is—get things so wrong? My brief conclusion was that once you have some success, I guess there’s not much of a motivation to change your ways. Also, I could well believe that, for all its flaws, Gottman’s work is better than much of the other research out there on marriages. There’s still the question of how this stuff gets published in scientific journals. I haven’t looked at Gottman’s articles in detail and so don’t really have thoughts on that one. Anyway, I recently corresponded with a mathematician who had heard of Gottman’s research and wrote
2 0.63874888 69 andrew gelman stats-2010-06-04-A Wikipedia whitewash
Introduction: After hearing a few times about the divorce predictions of researchers John Gottman and James Murray (work that was featured in Blink with a claim that they could predict with 83 percent accuracy whether a couple would be divorced–after meeting with them for 15 minutes) and feeling some skepticism , I decided to do the Lord’s work and amend Gottman’s wikipedia entry, which had a paragraph saying: Gottman found his methodology predicts with 90% accuracy which newlywed couples will remain married and which will divorce four to six years later. It is also 81% percent accurate in predicting which marriages will survive after seven to nine years. I added the following: Gottman’s claim of 81% or 90% accuracy is misleading, however, because the accuracy is measured only after fitting a model to his data. There is no evidence that he can predict the outcome of a marriage with high accuracy in advance. As Laurie Abraham writes, “For the 1998 study, which focused on videotapes of 57
Introduction: The Journal of the Royal Statistical Society publishes papers followed by discussions. Lots of discussions, each can be no more than 400 words. Here’s my most recent discussion: The authors are working on an important applied problem and I have no reason to doubt that their approach is a step forward beyond diagnostic criteria based on point estimation. An attempt at an accurate assessment of variation is important not just for statistical reasons but also because scientists have the duty to convey their uncertainty to the larger world. I am thinking, for example, of discredited claims such as that of the mathematician who claimed to predict divorces with 93% accuracy (Abraham, 2010). Regarding the paper at hand, I thought I would try an experiment in comment-writing. My usual practice is to read the graphs and then go back and clarify any questions through the text. So, very quickly: I would prefer Figure 1 to be displayed in terms of standard deviations, not variances. I
Introduction: I was given the opportunity to briefly comment on the paper , A Bayesian approach to complex clinical diagnoses: a case-study in child abuse, by Nicky Best, Deborah Ashby, Frank Dunstan, David Foreman, and Neil McIntosh, for the Journal of the Royal Statistical Society. Here is what I wrote: Best et al. are working on an important applied problem and I have no reason to doubt that their approach is a step forward beyond diagnostic criteria based on point estimation. An attempt at an accurate assessment of variation is important not just for statistical reasons but also because scientists have the duty to convey their uncertainty to the larger world. I am thinking, for example, of discredited claims such as that of the mathematician who claimed to predict divorces with 93% accuracy (Abraham, 2010). Regarding the paper at hand, I thought I would try an experiment in comment-writing. My usual practice is to read the graphs and then go back and clarify any questions through the t
Introduction: Nick Brown is bothered by this article , “An unscented Kalman filter approach to the estimation of nonlinear dynamical systems models,” by Sy-Miin Chow, Emilio Ferrer, and John Nesselroade. The introduction of the article cites a bunch of articles in serious psych/statistics journals. The question is, are such advanced statistical techniques really needed, or even legitimate, with the kind of very rough data that is usually available in psych applications? Or is it just fishing in the hope of discovering patterns that are not really there? I wrote: It seems like a pretty innocuous literature review. I agree that many of the applications are silly (for example, they cite the work of the notorious John Gottman in fitting a predator-prey model to spousal relations (!)), but overall they just seem to be presenting very standard ideas for the mathematical-psychology audience. It’s not clear whether advanced techniques are always appropriate here, but they come in through a natura
6 0.10082342 1883 andrew gelman stats-2013-06-04-Interrogating p-values
7 0.099697091 150 andrew gelman stats-2010-07-16-Gaydar update: Additional research on estimating small fractions of the population
8 0.094493009 1981 andrew gelman stats-2013-08-14-The robust beauty of improper linear models in decision making
9 0.09399797 2093 andrew gelman stats-2013-11-07-I’m negative on the expression “false positives”
10 0.091844119 688 andrew gelman stats-2011-04-30-Why it’s so relaxing to think about social issues
12 0.082633503 2177 andrew gelman stats-2014-01-19-“The British amateur who debunked the mathematics of happiness”
13 0.082078554 719 andrew gelman stats-2011-05-19-Everything is Obvious (once you know the answer)
14 0.081060648 888 andrew gelman stats-2011-09-03-A psychology researcher asks: Is Anova dead?
16 0.080909893 563 andrew gelman stats-2011-02-07-Evaluating predictions of political events
17 0.0799582 2236 andrew gelman stats-2014-03-07-Selection bias in the reporting of shaky research
18 0.078674264 2058 andrew gelman stats-2013-10-11-Gladwell and Chabris, David and Goliath, and science writing as stone soup
19 0.078292198 1955 andrew gelman stats-2013-07-25-Bayes-respecting experimental design and other things
20 0.077656366 2245 andrew gelman stats-2014-03-12-More on publishing in journals
topicId topicWeight
[(0, 0.196), (1, -0.022), (2, 0.022), (3, -0.026), (4, 0.011), (5, -0.038), (6, 0.053), (7, -0.012), (8, 0.057), (9, 0.003), (10, -0.005), (11, 0.022), (12, -0.03), (13, -0.049), (14, -0.015), (15, 0.043), (16, 0.011), (17, -0.0), (18, 0.021), (19, -0.006), (20, -0.019), (21, 0.033), (22, 0.001), (23, -0.001), (24, 0.001), (25, 0.008), (26, -0.019), (27, 0.0), (28, 0.005), (29, -0.008), (30, 0.051), (31, 0.05), (32, 0.027), (33, 0.007), (34, -0.028), (35, -0.043), (36, 0.037), (37, -0.021), (38, 0.002), (39, -0.068), (40, 0.003), (41, -0.033), (42, 0.022), (43, -0.026), (44, -0.016), (45, 0.011), (46, 0.001), (47, -0.027), (48, 0.07), (49, 0.01)]
simIndex simValue blogId blogTitle
same-blog 1 0.94292265 105 andrew gelman stats-2010-06-23-More on those divorce prediction statistics, including a discussion of the innumeracy of (some) mathematicians
Introduction: A few months ago, I blogged on John Gottman, a psychologist whose headline-grabbing research on marriages (he got himself featured in Blink with a claim that he could predict with 83 percent accuracy whether a couple would be divorced—after meeting with them for 15 minutes!) was recently debunked in a book by Laurie Abraham. The question I raised was: how could someone who was evidently so intelligent and accomplished—Gottman, that is—get things so wrong? My brief conclusion was that once you have some success, I guess there’s not much of a motivation to change your ways. Also, I could well believe that, for all its flaws, Gottman’s work is better than much of the other research out there on marriages. There’s still the question of how this stuff gets published in scientific journals. I haven’t looked at Gottman’s articles in detail and so don’t really have thoughts on that one. Anyway, I recently corresponded with a mathematician who had heard of Gottman’s research and wrote
2 0.85245675 69 andrew gelman stats-2010-06-04-A Wikipedia whitewash
Introduction: After hearing a few times about the divorce predictions of researchers John Gottman and James Murray (work that was featured in Blink with a claim that they could predict with 83 percent accuracy whether a couple would be divorced–after meeting with them for 15 minutes) and feeling some skepticism , I decided to do the Lord’s work and amend Gottman’s wikipedia entry, which had a paragraph saying: Gottman found his methodology predicts with 90% accuracy which newlywed couples will remain married and which will divorce four to six years later. It is also 81% percent accurate in predicting which marriages will survive after seven to nine years. I added the following: Gottman’s claim of 81% or 90% accuracy is misleading, however, because the accuracy is measured only after fitting a model to his data. There is no evidence that he can predict the outcome of a marriage with high accuracy in advance. As Laurie Abraham writes, “For the 1998 study, which focused on videotapes of 57
3 0.8154034 563 andrew gelman stats-2011-02-07-Evaluating predictions of political events
Introduction: Mike Cohen writes: The recent events in Egypt raise an interesting statistical question. It is of course common for news stations like CNN to interview various officials and policy experts to find out what is likely to happen next. The obvious response of people like us is why ask such people when they didn’t foresee a month ago that these dynamic events were about to happen. One would instead like to hear from those experts that did predict that something was about to happen in Tunisia, and Egypt, and Jordan, and maybe Yemen, etc. Well, are there such people? My friend Bob Burton says that of course one can find such people in the sense that they made such predictions, but that is like finding counties that have voted for the President in the last five elections, big deal, or psychics that predicted the last assassination, again big deal. There is a good deal of truth in that. However, it seems like we do a little better. There are two points to make. First, there is an i
Introduction: Hadley Wickham sent me this , by Keith Baggerly and Kevin Coombes: In this report we [Baggerly and Coombes] examine several related papers purporting to use microarray-based signatures of drug sensitivity derived from cell lines to predict patient response. Patients in clinical trials are currently being allocated to treatment arms on the basis of these results. However, we show in five case studies that the results incorporate several simple errors that may be putting patients at risk. One theme that emerges is that the most common errors are simple (e.g., row or column offsets); conversely, it is our experience that the most simple errors are common. This is horrible! But, in a way, it’s not surprising. I make big mistakes in my applied work all the time. I mean, all the time. Sometimes I scramble the order of the 50 states, or I’m plotting a pure noise variable, or whatever. But usually I don’t drift too far from reality because I have a lot of cross-checks and I (or my
5 0.75577837 1158 andrew gelman stats-2012-02-07-The more likely it is to be X, the more likely it is to be Not X?
Introduction: This post is by Phil Price. A paper by Wood, Douglas, and Sutton looks at “Beliefs in Contradictory Conspiracy Theories.” Unfortunately the subjects were 140 undergraduate psychology students, so one wonders how general the results are. I found this sort of arresting: In Study 1 (n=137), the more participants believed that Princess Diana faked her own death, the more they believed she was murdered. In Study 2 (n=102), the more participants believed that Osama Bin Laden was already dead when U.S. Special Forces raided his compound in Pakistan, the more they believed he is still alive. As the article says, “conspiracy advocates’ distrust of official narratives may be so strong that many alternative theories are simultaneously endorsed in spite of any contradictions between them.” But I think the authors overstate things when they say “One would think that there ought to be a negative correlation between beliefs in contradictory accounts of events — the more one believes in
6 0.75528461 1760 andrew gelman stats-2013-03-12-Misunderstanding the p-value
7 0.74154377 257 andrew gelman stats-2010-09-04-Question about standard range for social science correlations
8 0.73140502 1403 andrew gelman stats-2012-07-02-Moving beyond hopeless graphics
10 0.72765058 2204 andrew gelman stats-2014-02-09-Keli Liu and Xiao-Li Meng on Simpson’s paradox
11 0.7246362 189 andrew gelman stats-2010-08-06-Proposal for a moratorium on the use of the words “fashionable” and “trendy”
12 0.72101587 2053 andrew gelman stats-2013-10-06-Ideas that spread fast and slow
14 0.7197386 1187 andrew gelman stats-2012-02-27-“Apple confronts the law of large numbers” . . . huh?
15 0.71724057 923 andrew gelman stats-2011-09-24-What is the normal range of values in a medical test?
16 0.7144562 1603 andrew gelman stats-2012-12-03-Somebody listened to me!
17 0.71356237 1281 andrew gelman stats-2012-04-25-Dyson’s baffling love of crackpots
18 0.71231943 748 andrew gelman stats-2011-06-06-Why your Klout score is meaningless
19 0.70794278 2058 andrew gelman stats-2013-10-11-Gladwell and Chabris, David and Goliath, and science writing as stone soup
20 0.70781702 2352 andrew gelman stats-2014-05-29-When you believe in things that you don’t understand
topicId topicWeight
[(0, 0.011), (2, 0.011), (15, 0.037), (16, 0.066), (21, 0.026), (24, 0.119), (45, 0.119), (53, 0.011), (84, 0.011), (86, 0.08), (95, 0.069), (98, 0.014), (99, 0.286)]
simIndex simValue blogId blogTitle
1 0.97379804 69 andrew gelman stats-2010-06-04-A Wikipedia whitewash
Introduction: After hearing a few times about the divorce predictions of researchers John Gottman and James Murray (work that was featured in Blink with a claim that they could predict with 83 percent accuracy whether a couple would be divorced–after meeting with them for 15 minutes) and feeling some skepticism , I decided to do the Lord’s work and amend Gottman’s wikipedia entry, which had a paragraph saying: Gottman found his methodology predicts with 90% accuracy which newlywed couples will remain married and which will divorce four to six years later. It is also 81% percent accurate in predicting which marriages will survive after seven to nine years. I added the following: Gottman’s claim of 81% or 90% accuracy is misleading, however, because the accuracy is measured only after fitting a model to his data. There is no evidence that he can predict the outcome of a marriage with high accuracy in advance. As Laurie Abraham writes, “For the 1998 study, which focused on videotapes of 57
2 0.95831215 1325 andrew gelman stats-2012-05-17-More on the difficulty of “preaching what you practice”
Introduction: A couple months ago, in discussing Charles Murray’s argument that America’s social leaders should “preach what they practice” (Murray argues that they—we!—tend to lead good lives of hard work and moderation but are all too tolerant of antisocial and unproductive behavior among the lower classes), I wrote : Murray does not consider the case of Joe Paterno, but in many ways the Penn State football coach fits his story well. Paterno was said to live an exemplary personal and professional life, combining traditional morality with football success—but, by his actions, he showed little concern about the morality of his players and coaches. At a professional level, Paterno rose higher and higher, and in his personal life he was a responsible adult. But he had an increasing disconnect with the real world, to the extent that horrible crimes were occurring nearby (in the physical and social senses) but he was completely insulated from the consequences for many years. Paterno’s story is s
same-blog 3 0.95344412 105 andrew gelman stats-2010-06-23-More on those divorce prediction statistics, including a discussion of the innumeracy of (some) mathematicians
Introduction: A few months ago, I blogged on John Gottman, a psychologist whose headline-grabbing research on marriages (he got himself featured in Blink with a claim that he could predict with 83 percent accuracy whether a couple would be divorced—after meeting with them for 15 minutes!) was recently debunked in a book by Laurie Abraham. The question I raised was: how could someone who was evidently so intelligent and accomplished—Gottman, that is—get things so wrong? My brief conclusion was that once you have some success, I guess there’s not much of a motivation to change your ways. Also, I could well believe that, for all its flaws, Gottman’s work is better than much of the other research out there on marriages. There’s still the question of how this stuff gets published in scientific journals. I haven’t looked at Gottman’s articles in detail and so don’t really have thoughts on that one. Anyway, I recently corresponded with a mathematician who had heard of Gottman’s research and wrote
4 0.95329964 362 andrew gelman stats-2010-10-22-A redrawing of the Red-Blue map in November 2010?
Introduction: Here are my answers to the following questions asked by Pauline Peretz: 1. Many analysts have emphasized that there was a redrawing of the electoral map in 2008. To what extent will the November midterm elections affect this red-blue map? How long will the newly blue states remain blue? 2. Do you think the predictable loss of the Democrats in November definitely disqualifies the hypothesis that Obama’s election was the beginning of a realignment in American politics, that is a period of dominance for the Democratic party due to favourable demographics? 3. Some analysts consider that voting patterns are best explained by economic factors, others by values. How do you position yourself in the debate on culture wars vs. economic wars? 4. In your book Red State, Blue State, Rich State, Poor State, you renew the ongoing debate on the correlation between income and vote, showing it is much stronger in poor states. In light of this correlation, would you say that there currently is
5 0.94989133 1767 andrew gelman stats-2013-03-17-The disappearing or non-disappearing middle class
Introduction: Despite the title, this post is mostly not about economics or even politics but rather about the central role of comparisons in statistics and statistical graphics. It started when someone pointed me to this article in which Megan McArdle points out the misleadingness of a graph that seems to show a bimodal income distribution but only by combining cells in the tail: McArdle makes a good point: of course, if you spread the histogram along a uniform scale (or, for that matter, a log scale), you don’t see that bump at the high end. McArdle reproduces some Census charts showing income stability over the past few decades: Before I had a chance to chance to write about this, I noticed that Mark Palko did the job for me. Palko writes: To the extent that statistics includes data visualization, this is definitely bad statistics. When trying to depict trends and relationships, you generally want to get as much of the pertinent information as possible into the same grap
6 0.94940066 791 andrew gelman stats-2011-07-08-Censoring on one end, “outliers” on the other, what can we do with the middle?
7 0.9453665 1012 andrew gelman stats-2011-11-16-Blog bribes!
8 0.94508266 1031 andrew gelman stats-2011-11-27-Richard Stallman and John McCarthy
9 0.94399881 1658 andrew gelman stats-2013-01-07-Free advice from an academic writing coach!
10 0.93919736 543 andrew gelman stats-2011-01-28-NYT shills for personal DNA tests
11 0.93878239 673 andrew gelman stats-2011-04-20-Upper-income people still don’t realize they’re upper-income
12 0.93844599 1121 andrew gelman stats-2012-01-15-R-squared for multilevel models
13 0.93656075 192 andrew gelman stats-2010-08-08-Turning pages into data
14 0.93608773 1854 andrew gelman stats-2013-05-13-A Structural Comparison of Conspicuous Consumption in China and the United States
15 0.93578827 1504 andrew gelman stats-2012-09-20-Could someone please lock this guy and Niall Ferguson in a room together?
16 0.9346143 728 andrew gelman stats-2011-05-24-A (not quite) grand unified theory of plagiarism, as applied to the Wegman case
17 0.9339776 449 andrew gelman stats-2010-12-04-Generalized Method of Moments, whatever that is
18 0.93290025 1266 andrew gelman stats-2012-04-16-Another day, another plagiarist
19 0.9315601 777 andrew gelman stats-2011-06-23-Combining survey data obtained using different modes of sampling
20 0.93089211 811 andrew gelman stats-2011-07-20-Kind of Bayesian