andrew_gelman_stats andrew_gelman_stats-2010 andrew_gelman_stats-2010-11 knowledge-graph by maker-knowledge-mining
Source: html
Introduction: I just reviewed the book Bursts, by Albert-László Barabási, for Physics Today. But I had a lot more to say that couldn’t fit into the magazine’s 800-word limit. Here I’ll reproduce what I sent to Physics Today, followed by my additional thoughts. The back cover of Bursts book promises “a revolutionary new theory showing how we can predict human behavior.” I wasn’t fully convinced on that score, but the book does offer a well-written and thought-provoking window into author Albert-László Barabási’s research in power laws and network theory. Power laws–the mathematical pattern that little things are common and large things are rare–have been observed in many different domains, including incomes (as noted by economist Vilfredo Pareto in the nineteenth century), word frequencies (as noted by linguist George Zipf), city sizes, earthquakes, and virtually anything else that can be measured. In the mid-twentieth century, the mathematician Benoit Mandelbrot devoted an influential caree
sentIndex sentText sentNum sentScore
1 The back cover of Bursts book promises “a revolutionary new theory showing how we can predict human behavior. [sent-4, score-0.233]
2 ” I wasn’t fully convinced on that score, but the book does offer a well-written and thought-provoking window into author Albert-László Barabási’s research in power laws and network theory. [sent-5, score-0.294]
3 In the mid-twentieth century, the mathematician Benoit Mandelbrot devoted an influential career to the study of self-similarity, deriving power laws for phenomena ranging from taxonomies (the distribution of the lengths of index entries) to geographical measurements. [sent-7, score-0.176]
4 ) A similar distinction between regularity and fractality holds in the social world, with designed structures such as bus schedules having a smooth order, and actual distributions of bus waiting times (say) having a complex pattern of randomness. [sent-12, score-0.285]
5 Trained as a physicist, Albert-László Barabási has worked for several years on mathematical models for the emergence of power laws in complex systems such as the Internet. [sent-13, score-0.277]
6 In his latest book, Barabási describes many aspects of power laws, including a computer simulation of busy responses that went like this: a) I [Barbasi] selected the highest-priority task and removed it from the list, mimicking the real habit I have when I execute a task. [sent-14, score-0.177]
7 As with Albert Einstein’s theory of Brownian motion, such latent-variable models suggest new directions of research, in this case moving from a static analysis of waiting time distributions to a dynamic study of the decisions that underlie the stochastic process. [sent-18, score-0.173]
8 He distinguishes between traditional models of randomness–Poisson and Gaussian distributions–which are based on statistically independent events, and bursty processes, which arise from feedback processes that at times suppress and at other times amplify variation. [sent-22, score-0.316]
9 ) Barabási characterizes bursty processes as predictable; at one point he discusses the burstiness of people’s physical locations (we spend most of our time at home, school or work, or in between, but occasionally go on long trips). [sent-24, score-0.217]
10 From here, he takes a leap–which I couldn’t follow at all–to conjecture a more general order within human behavior and historical events, in his opinion calling into question Karl Popper’s argument that human history is inherently unpredictable. [sent-25, score-0.224]
11 The book also features a long excursion into Hungarian history, but the connection of this narrative to the scientific themes was unclear to me. [sent-26, score-0.264]
12 About half the book tells the story of a peasant rebellion in sixteenth-century Hungary (along with associated political maneuvering), but it’s broken up into a dozen chapters spread throughout the book, and I had to keep flipping back and forth to follow what was going on. [sent-36, score-0.177]
13 On one hand, I can hardly blame the author for trying to make his book accessible to general audiences; still, I kept wanting more detail, to fill in the gaps and understand exactly how the mathematical models and statistical analyses fit into the stories and the larger claims. [sent-46, score-0.317]
14 In some fundamental way, bursty processes can be thought of as predictable and not so random. [sent-50, score-0.25]
15 in particular, human behavior and even human history can perhaps be much more predictable than we thought? [sent-51, score-0.297]
16 I think the presentation would’ve been stronger had he discussed the variety of different mathematical models that researchers have developed to explain power laws. [sent-55, score-0.178]
17 It seems like a big leap to go from being able to predict people’s locations (mostly they’re at work from 9-5 and home at other times) and forecasting human history as one might forecast the weather. [sent-60, score-0.242]
18 Barabási quotes Popper as saying that social patterns don’t have the regularity of natural sciences–we can’t predict revolutions like we can predict eclipses because societies, unlike planets, don’t move in regular orbits. [sent-62, score-0.194]
19 Second, social progress depends on the progress of science, and scientific progress is inherently unpredictable. [sent-68, score-0.216]
20 - On page 199, he writes, “when it comes to the predicability of our actions, to our surprise power laws are replaced by Gaussians. [sent-87, score-0.216]
wordName wordTfidf (topN-words)
[('barab', 0.614), ('si', 0.487), ('bursts', 0.151), ('einstein', 0.132), ('popper', 0.125), ('book', 0.118), ('bursty', 0.117), ('laws', 0.099), ('szl', 0.088), ('power', 0.077), ('predictable', 0.073), ('waiting', 0.068), ('human', 0.065), ('albert', 0.064), ('letters', 0.063), ('narrative', 0.061), ('processes', 0.06), ('story', 0.059), ('zipf', 0.058), ('models', 0.053), ('mimicking', 0.053), ('dynamic', 0.052), ('larger', 0.051), ('predict', 0.05), ('mathematical', 0.048), ('earthquakes', 0.048), ('regularity', 0.048), ('history', 0.048), ('century', 0.048), ('task', 0.047), ('stories', 0.047), ('noted', 0.047), ('social', 0.046), ('behavior', 0.046), ('predictability', 0.045), ('visits', 0.045), ('scientific', 0.044), ('bright', 0.044), ('fractals', 0.044), ('times', 0.043), ('progress', 0.042), ('detail', 0.041), ('mandelbrot', 0.041), ('features', 0.041), ('bus', 0.04), ('locations', 0.04), ('persuasive', 0.04), ('page', 0.04), ('structure', 0.039), ('leap', 0.039)]
simIndex simValue blogId blogTitle
same-blog 1 1.0000001 11 andrew gelman stats-2010-04-29-Auto-Gladwell, or Can fractals be used to predict human history?
Introduction: I just reviewed the book Bursts, by Albert-László Barabási, for Physics Today. But I had a lot more to say that couldn’t fit into the magazine’s 800-word limit. Here I’ll reproduce what I sent to Physics Today, followed by my additional thoughts. The back cover of Bursts book promises “a revolutionary new theory showing how we can predict human behavior.” I wasn’t fully convinced on that score, but the book does offer a well-written and thought-provoking window into author Albert-László Barabási’s research in power laws and network theory. Power laws–the mathematical pattern that little things are common and large things are rare–have been observed in many different domains, including incomes (as noted by economist Vilfredo Pareto in the nineteenth century), word frequencies (as noted by linguist George Zipf), city sizes, earthquakes, and virtually anything else that can be measured. In the mid-twentieth century, the mathematician Benoit Mandelbrot devoted an influential caree
2 0.10328615 719 andrew gelman stats-2011-05-19-Everything is Obvious (once you know the answer)
Introduction: Duncan Watts gave his new book the above title, reflecting his irritation with those annoying people who, upon hearing of the latest social science research, reply with: Duh-I-knew-that. (I don’t know how to say Duh in Australian; maybe someone can translate that for me?) I, like Duncan, am easily irritated, and I looked forward to reading the book. I enjoyed it a lot, even though it has only one graph, and that graph has a problem with its y-axis. (OK, the book also has two diagrams and a graph of fake data, but that doesn’t count.) Before going on, let me say that I agree wholeheartedly with Duncan’s central point: social science research findings are often surprising, but the best results cause us to rethink our world in such a way that they seem completely obvious, in retrospect. (Don Rubin used to tell us that there’s no such thing as a “paradox”: once you fully understand a phenomenon, it should not seem paradoxical any more. When learning science, we sometimes speak
3 0.10048894 23 andrew gelman stats-2010-05-09-Popper’s great, but don’t bother with his theory of probability
Introduction: Adam Gurri writes: Any chance you could do a post explaining Popper’s propensity theory of probability? I have never understood it. My reply: I’m a big fan of Popper (search this blog for details), especially as interpreted by Lakatos, but as far as I can tell, Popper’s theory of probability is hopeless. We’ve made a lot of progress on probability in the past 75 years, and I don’t see any real need to go back to the bad old days.
4 0.092197686 2351 andrew gelman stats-2014-05-28-Bayesian nonparametric weighted sampling inference
Introduction: Yajuan Si, Natesh Pillai, and I write : It has historically been a challenge to perform Bayesian inference in a design-based survey context. The present paper develops a Bayesian model for sampling inference using inverse-probability weights. We use a hierarchical approach in which we model the distribution of the weights of the nonsampled units in the population and simultaneously include them as predictors in a nonparametric Gaussian process regression. We use simulation studies to evaluate the performance of our procedure and compare it to the classical design-based estimator. We apply our method to the Fragile Family Child Wellbeing Study. Our studies find the Bayesian nonparametric finite population estimator to be more robust than the classical design-based estimator without loss in efficiency. More work needs to be done for this to be a general practical tool—in particular, in the setup of this paper you only have survey weights and no direct poststratification variab
Introduction: As I wrote a couple years ago: Even though statistical analysis has demonstrated that presidential elections are predictable given economic conditions and previous votes in the states . . . it certainly doesn’t mean that every election can be accurately predicted ahead of time. Presidential general election campaigns have several distinct features that distinguish them from most other elections: 1. Two major candidates; 2. The candidates clearly differ in their political ideologies and in their positions on economic issues; 3. The two sides have roughly equal financial and organizational resources; 4. The current election is the latest in a long series of similar contests (every four years); 5. A long campaign, giving candidates a long time to present their case and giving voters a long time to make up their minds. Other elections look different. . . . Or, as I said in reference to the current NYC mayoral election: Et selon Andrew Gelman, expert de l’universi
6 0.087509908 1652 andrew gelman stats-2013-01-03-“The Case for Inductive Theory Building”
7 0.083089769 2245 andrew gelman stats-2014-03-12-More on publishing in journals
10 0.080122352 757 andrew gelman stats-2011-06-10-Controversy over the Christakis-Fowler findings on the contagion of obesity
11 0.078262858 2255 andrew gelman stats-2014-03-19-How Americans vote
12 0.077950291 2297 andrew gelman stats-2014-04-20-Fooled by randomness
13 0.077719525 2072 andrew gelman stats-2013-10-21-The future (and past) of statistical sciences
14 0.07729391 2292 andrew gelman stats-2014-04-15-When you believe in things that you don’t understand
15 0.075338803 1278 andrew gelman stats-2012-04-23-“Any old map will do” meets “God is in every leaf of every tree”
16 0.072513245 1634 andrew gelman stats-2012-12-21-Two reviews of Nate Silver’s new book, from Kaiser Fung and Cathy O’Neil
17 0.07215187 1397 andrew gelman stats-2012-06-27-Stand Your Ground laws and homicides
18 0.071236037 2263 andrew gelman stats-2014-03-24-Empirical implications of Empirical Implications of Theoretical Models
19 0.070423983 1605 andrew gelman stats-2012-12-04-Write This Book
20 0.070146151 2258 andrew gelman stats-2014-03-21-Random matrices in the news
topicId topicWeight
[(0, 0.178), (1, -0.033), (2, -0.027), (3, 0.013), (4, -0.023), (5, 0.018), (6, 0.003), (7, -0.011), (8, 0.041), (9, 0.049), (10, -0.016), (11, -0.013), (12, -0.027), (13, -0.016), (14, 0.012), (15, 0.002), (16, 0.016), (17, 0.005), (18, 0.058), (19, -0.057), (20, -0.017), (21, -0.022), (22, -0.016), (23, 0.01), (24, 0.006), (25, 0.025), (26, 0.034), (27, 0.024), (28, 0.02), (29, -0.025), (30, -0.069), (31, 0.014), (32, -0.01), (33, -0.009), (34, 0.019), (35, -0.0), (36, -0.015), (37, -0.01), (38, 0.029), (39, -0.023), (40, -0.016), (41, 0.014), (42, 0.031), (43, 0.004), (44, 0.001), (45, -0.031), (46, 0.005), (47, -0.001), (48, -0.007), (49, -0.002)]
simIndex simValue blogId blogTitle
same-blog 1 0.97956318 11 andrew gelman stats-2010-04-29-Auto-Gladwell, or Can fractals be used to predict human history?
Introduction: I just reviewed the book Bursts, by Albert-László Barabási, for Physics Today. But I had a lot more to say that couldn’t fit into the magazine’s 800-word limit. Here I’ll reproduce what I sent to Physics Today, followed by my additional thoughts. The back cover of Bursts book promises “a revolutionary new theory showing how we can predict human behavior.” I wasn’t fully convinced on that score, but the book does offer a well-written and thought-provoking window into author Albert-László Barabási’s research in power laws and network theory. Power laws–the mathematical pattern that little things are common and large things are rare–have been observed in many different domains, including incomes (as noted by economist Vilfredo Pareto in the nineteenth century), word frequencies (as noted by linguist George Zipf), city sizes, earthquakes, and virtually anything else that can be measured. In the mid-twentieth century, the mathematician Benoit Mandelbrot devoted an influential caree
2 0.88114125 2327 andrew gelman stats-2014-05-09-Nicholas Wade and the paradox of racism
Introduction: The paradox of racism is that at any given moment, the racism of the day seems reasonable and very possibly true, but the racism of the past always seems so ridiculous. I’ve been thinking about this for a few months ever since receiving in the mail a new book, “A Troublesome Inheritance: Genes, Race, and Human History,” by New York Times reporter Nicholas Wade. Here’s what I wrote in my review of this book for Slate : The word “inequality” does not appear in the book’s index, but what Wade is offering is essentially a theory of economic and social inequality, explaining systematic racial differences in prosperity based on a combination of innate traits (“the disinclination to save in tribal societies is linked to a strong propensity for immediate consumption”) and genetic adaptation to political and social institutions (arguing, for example, that generations of centralized rule have effected a selection pressure for Chinese to be accepting of authority). Wade is clearly in
3 0.82927454 1784 andrew gelman stats-2013-04-01-Wolfram on Mandelbrot
Introduction: The most perfect pairing of author and subject since Nicholson Baker and John Updike. Here’s Wolfram on the great researcher of fractals : In his way, Mandelbrot paid me some great compliments. When I was in my 20s, and he in his 60s, he would ask about my scientific work: “How can so many people take someone so young so seriously?” In 2002, my book “A New Kind of Science”—in which I argued that many phenomena across science are the complex results of relatively simple, program-like rules—appeared. Mandelbrot seemed to see it as a direct threat, once declaring that “Wolfram’s ‘science’ is not new except when it is clearly wrong; it deserves to be completely disregarded.” In private, though, several mutual friends told me, he fretted that in the long view of history it would overwhelm his work. In retrospect, I don’t think Mandelbrot had much to worry about on this account. The link from the above review came from Peter Woit, who also points to a review by Brian Hayes wit
4 0.80413932 2297 andrew gelman stats-2014-04-20-Fooled by randomness
Introduction: From 2006 : Naseem Taleb ‘s publisher sent me a copy of “Fooled by randomness: the hidden role of chance in life and the markets” to review. It’s an important topic, and the book is written in a charming style—I’ll try to respond in kind, with some miscellaneous comments. On the cover of the book is a blurb, “Named by Fortune one of the smartest books of all time.” But Taleb instructs us on page 161-162 to ignore book reviews because of selection bias (the mediocre reviews don’t make it to the book cover). Books vs. articles I prefer writing books to writing journal articles because books are written for the reader (and also, in the case of textbooks, for the teacher), whereas articles are written for referees. Taleb definitely seems to be writing to the reader, not the referee. There is risk in book-writing, since in some ways referees are the ideal audience of experts, but I enjoy the freedom in book-writing of being able to say what I really think. Variation and rando
5 0.79806346 1634 andrew gelman stats-2012-12-21-Two reviews of Nate Silver’s new book, from Kaiser Fung and Cathy O’Neil
Introduction: People keep asking me what I think of Nate’s book, and I keep replying that, as a blogger, I’m spoiled. I’m so used to getting books for free that I wouldn’t go out and buy a book just for the purpose of reviewing it. (That reminds me that I should post reviews of some of those books I’ve received in the mail over the past few months.) I have, however, encountered a couple of reviews of The Signal and the Noise so I thought I’d pass them on to you. Both these reviews are by statisticians / data scientists who work here in NYC in the non-academic “real world” so in that sense they are perhaps better situated than me to review the book (also, they have not collaborated with Nate so they have no conflict of interest). Kaiser Fung gives a positive review : It is in the subtitle—“why so many predictions fail – but some don’t”—that one learns the core philosophy of Silver: he is most concerned with the honest evaluation of the performance of predictive models. The failure to look
6 0.79672855 719 andrew gelman stats-2011-05-19-Everything is Obvious (once you know the answer)
7 0.78929251 1303 andrew gelman stats-2012-05-06-I’m skeptical about this skeptical article about left-handedness
8 0.78152579 973 andrew gelman stats-2011-10-26-Antman again courts controversy
9 0.77818644 483 andrew gelman stats-2010-12-23-Science, ideology, and human origins
10 0.77374011 116 andrew gelman stats-2010-06-29-How to grab power in a democracy – in 5 easy non-violent steps
12 0.77021796 285 andrew gelman stats-2010-09-18-Fiction is not for tirades? Tell that to Saul Bellow!
13 0.76715666 2189 andrew gelman stats-2014-01-28-History is too important to be left to the history professors
14 0.75760007 392 andrew gelman stats-2010-11-03-Taleb + 3.5 years
15 0.75727177 789 andrew gelman stats-2011-07-07-Descriptive statistics, causal inference, and story time
16 0.75668776 1414 andrew gelman stats-2012-07-12-Steven Pinker’s unconvincing debunking of group selection
17 0.75254887 893 andrew gelman stats-2011-09-06-Julian Symons on Frances Newman
18 0.75101286 1515 andrew gelman stats-2012-09-29-Jost Haidt
19 0.74820727 16 andrew gelman stats-2010-05-04-Burgess on Kipling
topicId topicWeight
[(5, 0.014), (9, 0.029), (15, 0.031), (16, 0.078), (18, 0.011), (21, 0.025), (24, 0.1), (36, 0.013), (41, 0.022), (53, 0.013), (64, 0.137), (65, 0.013), (72, 0.011), (77, 0.022), (86, 0.034), (99, 0.244)]
simIndex simValue blogId blogTitle
1 0.95438427 985 andrew gelman stats-2011-11-01-Doug Schoen has 2 poll reports
Introduction: According to Chris Wilson , there are two versions of the report of the Occupy Wall Street poll from so-called hack pollster Doug Schoen. Here’s the report that Azi Paybarah says that Schoen sent to him, and here’s the final question from the poll: And here’s what’s on Schoen’s own website: Very similar, except for that last phrase, “no matter what the cost.” I have no idea which was actually asked to the survey participants, but it’s a reminder of the difficulties of public opinion research—sometimes you don’t even know what question was asked! I’m not implying anything sinister on Schoen’s part, it’s just interesting to see these two documents floating around. P.S. More here from Kaiser Fung on fundamental flaws with Schoen’s poll.
2 0.95318902 1521 andrew gelman stats-2012-10-04-Columbo does posterior predictive checks
Introduction: I’m already on record as saying that Ronald Reagan was a statistician so I think this is ok too . . . Here’s what Columbo does. He hears the killer’s story and he takes it very seriously (it’s murder, and Columbo never jokes about murder), examines all its implications, and finds where it doesn’t fit the data. Then Columbo carefully examines the discrepancies, tries some model expansion, and eventually concludes that he’s proved there’s a problem. OK, now you’re saying: Yeah, yeah, sure, but how does that differ from any other fictional detective? The difference, I think, is that the tradition is for the detective to find clues and use these to come up with hypotheses, or to trap the killer via internal contradictions in his or her statement. I see Columbo is different—and more in keeping with chapter 6 of Bayesian Data Analysis—in that he is taking the killer’s story seriously and exploring all its implications. That’s the essence of predictive model checking: you t
3 0.95009834 595 andrew gelman stats-2011-02-28-What Zombies see in Scatterplots
Introduction: This video caught my interest – news video clip (from this post2 ) http://www.stat.columbia.edu/~cook/movabletype/archives/2011/02/on_summarizing.html The news commentator did seem to be trying to point out what a couple of states had to say about the claimed relationship – almost on their own. Some methods have been worked out for zombies to do just this! So I grabbed the data as close as I quickly could, modified the code slightly and here’s the zombie veiw of it. PoliticInt.pdf North Carolina is the bolded red curve, Idaho the bolded green curve. Missisipi and New York are the bolded blue. As ugly as it is this is the Bayasian marginal picture – exactly (given MCMC errror). K? p.s. you will get a very confusing picture if you forget to centre the x (i.e. see chapter 4 of Gelman and Hill book)
4 0.94782561 118 andrew gelman stats-2010-06-30-Question & Answer Communities
Introduction: StackOverflow has been a popular community where software developers would help one another. Recently they raised some VC funding , and to make profits they are selling job postings and expanding the model to other areas. Metaoptimize LLC has started a similar website, using the open-source OSQA framework for such as statistics and machine learning. Here’s a description: You and other data geeks can ask and answer questions on machine learning, natural language processing, artificial intelligence, text analysis, information retrieval, search, data mining, statistical modeling, and data visualization. Here you can ask and answer questions, comment and vote for the questions of others and their answers. Both questions and answers can be revised and improved. Questions can be tagged with the relevant keywords to simplify future access and organize the accumulated material. If you work very hard on your questions and answers, you will receive badges like “Guru”, “Studen
5 0.94410896 1058 andrew gelman stats-2011-12-14-Higgs bozos: Rosencrantz and Guildenstern are spinning in their graves
Introduction: David Hogg sends in this bizarre bit of news reporting by Robert Evans: Until now, in the four decades since it was first posited, no one has convincingly claimed to have glimpsed the Higgs Boson, let alone proved that it actually exists. At an eagerly awaited briefing on Tuesday at the CERN research centre near Geneva, two independent teams of “Higgs Hunters” – a term they themselves hate – were widely expected to suggest they were fairly confident they had spotted it. But not confident enough, in the physics world of ultra-precision where certainty has to be measured at nothing less than 100 percent, to announce “a discovery.” In the jargon, this level is described as 5 sigma . . . So far, so good. But then comes this doozy: As one scientist explained, that level of accuracy would equate to the 17th-century discoverer of gravity, Isaac Newton, sitting under his apple tree and a million apples one after another falling on his head without one missing. Huh? A free
same-blog 6 0.94298524 11 andrew gelman stats-2010-04-29-Auto-Gladwell, or Can fractals be used to predict human history?
7 0.93189478 724 andrew gelman stats-2011-05-21-New search engine for data & statistics
8 0.91503024 1637 andrew gelman stats-2012-12-24-Textbook for data visualization?
9 0.91359246 1653 andrew gelman stats-2013-01-04-Census dotmap
11 0.88600677 2137 andrew gelman stats-2013-12-17-Replication backlash
12 0.88213933 1878 andrew gelman stats-2013-05-31-How to fix the tabloids? Toward replicable social science research
13 0.88105106 2337 andrew gelman stats-2014-05-18-Never back down: The culture of poverty and the culture of journalism
14 0.88085246 110 andrew gelman stats-2010-06-26-Philosophy and the practice of Bayesian statistics
15 0.88007569 711 andrew gelman stats-2011-05-14-Steven Rhoads’s book, “The Economist’s View of the World”
16 0.87979573 675 andrew gelman stats-2011-04-22-Arrow’s other theorem
17 0.87952334 1435 andrew gelman stats-2012-07-30-Retracted articles and unethical behavior in economics journals?
18 0.87832785 1097 andrew gelman stats-2012-01-03-Libertarians in Space
19 0.87829363 1910 andrew gelman stats-2013-06-22-Struggles over the criticism of the “cannabis users and IQ change” paper
20 0.87789595 1980 andrew gelman stats-2013-08-13-Test scores and grades predict job performance (but maybe not at Google)