andrew_gelman_stats andrew_gelman_stats-2012 andrew_gelman_stats-2012-1512 knowledge-graph by maker-knowledge-mining
Source: html
Introduction: Political campaigns are commonly understood as random walks, during which, at any point in time, the level of support for any party or candidate is equally likely to go up or down. Each shift in the polls is then interpreted as the result of some combination of news and campaign strategies. A completely different story of campaigns is the mean reversion model in which the elections are determined by fundamental factors of the economy and partisanship; the role of the campaign is to give voters a chance to reach their predetermined positions. The popularity of the random walk model for polls may be partially explained via analogy to the widespread idea that stock prices reflect all available information, as popularized in Burton Malkiel’s book, A Random Walk Down Wall Street. Once the idea has sunk in that short-term changes in the stock market are inherently unpredictable, it is natural for journalists to think the same of polls. For example, political analyst Nate Silver wrote
sentIndex sentText sentNum sentScore
1 Political campaigns are commonly understood as random walks, during which, at any point in time, the level of support for any party or candidate is equally likely to go up or down. [sent-1, score-0.546]
2 Each shift in the polls is then interpreted as the result of some combination of news and campaign strategies. [sent-2, score-0.617]
3 A completely different story of campaigns is the mean reversion model in which the elections are determined by fundamental factors of the economy and partisanship; the role of the campaign is to give voters a chance to reach their predetermined positions. [sent-3, score-0.915]
4 The popularity of the random walk model for polls may be partially explained via analogy to the widespread idea that stock prices reflect all available information, as popularized in Burton Malkiel’s book, A Random Walk Down Wall Street. [sent-4, score-1.004]
5 Once the idea has sunk in that short-term changes in the stock market are inherently unpredictable, it is natural for journalists to think the same of polls. [sent-5, score-0.322]
6 For example, political analyst Nate Silver wrote in 2010: In races with lots of polling, instead, the most robust assumption is usually that polling is essentially a random walk, i. [sent-6, score-0.472]
7 , that the polls are about equally likely to move toward one or another candidate, regardless of which way they have moved in the past. [sent-8, score-0.462]
8 But, as I discussed then in the context of remarks by Silver and Justin Wolfers , polls are not stock markets: for many races, a forecast from the fundamentals gives a pretty good idea about where the polls are going to end up. [sent-9, score-0.992]
9 For example, in the 1988 presidential election campaign, even when Michael Dukakis was up 10 points in the polls, informed experts were pretty sure that George Bush was going to win. [sent-10, score-0.295]
10 Political scientists Erikson, Bafumi, and Wlezien have found predictable changes in the generic opinion polls in the year leading up to an election, with different patterns in presidential years and off years. [sent-12, score-0.69]
11 Individual polls are noisy, though, and predictability will generally only be detectable with a long enough series. [sent-13, score-0.561]
12 Noah Kaplan, David Park, and I have a paper on the topic (to appear in Presidential Studies Quarterly) reviewing the literature and analyzing data from polls during the 2000, 2004, and 2008 elections. [sent-14, score-0.378]
13 We show that, as the campaign progresses, vote preferences become more predictable based on fundamental variables such as political ideology and party identification. [sent-15, score-0.757]
14 This is consistent with a “mean reversion” model in which the campaign serves to solidify latent preferences, but it is not consistent with a random walk model in which a campaign is an accretion of unpredictable shocks. [sent-16, score-1.158]
15 As John Sides explained reasonably, you have to be careful when translating economic numbers into vote predictions. [sent-19, score-0.188]
16 For example, Michael Kinsley recently wrote : In 1988, Michael Dukakis, who was ahead in the polls just after the Democratic convention, declared in his acceptance speech: “This election isn’t about ideology. [sent-21, score-0.56]
17 ” Then he proceeded to blow a large lead and lose to George Bush the Elder, who turned out to be a tougher old bird than anyone suspected. [sent-23, score-0.379]
18 To say that Dukakis blew a lead is not quite to buy into a random-walk model, but I think it is close. [sent-26, score-0.184]
19 Given what we know about elections, I think it would be more accurate to say that the 1988 election was Bush’s to lose (and he didn’t). [sent-27, score-0.256]
20 I’m hoping that, by explicitly stating the random-walk and mean-reversion scenarios, I can make people more aware of the implicit models that underly their stories about campaigns and elections. [sent-29, score-0.202]
wordName wordTfidf (topN-words)
[('polls', 0.378), ('campaign', 0.239), ('dukakis', 0.229), ('campaigns', 0.202), ('kinsley', 0.183), ('election', 0.182), ('walk', 0.176), ('bush', 0.174), ('races', 0.147), ('predictable', 0.139), ('elections', 0.138), ('reversion', 0.136), ('stock', 0.129), ('random', 0.128), ('lead', 0.123), ('unpredictable', 0.118), ('bird', 0.118), ('predictability', 0.115), ('presidential', 0.113), ('political', 0.109), ('fundamentals', 0.107), ('michael', 0.091), ('silver', 0.088), ('polling', 0.088), ('equally', 0.084), ('preferences', 0.082), ('journalists', 0.076), ('lose', 0.074), ('candidate', 0.069), ('george', 0.068), ('burton', 0.068), ('detectable', 0.068), ('progresses', 0.068), ('economy', 0.068), ('explained', 0.067), ('model', 0.067), ('fundamental', 0.065), ('proceeded', 0.064), ('elder', 0.064), ('party', 0.063), ('consistent', 0.062), ('translating', 0.061), ('wlezien', 0.061), ('blew', 0.061), ('vote', 0.06), ('changes', 0.06), ('popularized', 0.059), ('gravity', 0.057), ('erikson', 0.057), ('sunk', 0.057)]
simIndex simValue blogId blogTitle
same-blog 1 0.9999997 1512 andrew gelman stats-2012-09-27-A Non-random Walk Down Campaign Street
Introduction: Political campaigns are commonly understood as random walks, during which, at any point in time, the level of support for any party or candidate is equally likely to go up or down. Each shift in the polls is then interpreted as the result of some combination of news and campaign strategies. A completely different story of campaigns is the mean reversion model in which the elections are determined by fundamental factors of the economy and partisanship; the role of the campaign is to give voters a chance to reach their predetermined positions. The popularity of the random walk model for polls may be partially explained via analogy to the widespread idea that stock prices reflect all available information, as popularized in Burton Malkiel’s book, A Random Walk Down Wall Street. Once the idea has sunk in that short-term changes in the stock market are inherently unpredictable, it is natural for journalists to think the same of polls. For example, political analyst Nate Silver wrote
2 0.43631455 364 andrew gelman stats-2010-10-22-Politics is not a random walk: Momentum and mean reversion in polling
Introduction: Nate Silver and Justin Wolfers are having a friendly blog-dispute about momentum in political polling. Nate and Justin each make good points but are also missing parts of the picture. These questions relate to my own research so I thought I’d discuss them here. There ain’t no mo’ Nate led off the discussion by writing that pundits are always talking about “momentum” in the polls: Turn on the news or read through much of the analysis put out by some of our friends, and you’re likely to hear a lot of talk about “momentum”: the term is used about 60 times per day by major media outlets in conjunction with articles about polling. When people say a particular candidate has momentum, what they are implying is that present trends are likely to perpetuate themselves into the future. Say, for instance, that a candidate trailed by 10 points in a poll three weeks ago — and now a new poll comes out showing the candidate down by just 5 points. It will frequently be said that this
3 0.2891672 210 andrew gelman stats-2010-08-16-What I learned from those tough 538 commenters
Introduction: Some recent blog comments made me rethink how to express disagreement when I write. I’ll first display my original blog entry, then some of the comments, then my reaction, and finally my new approach. As usual with negative comments, my first reaction was to be upset, but once the hurt feelings went away, I realized that these dudes had a point. Act 1 A few days I ago, I posted the following on 538: Self-described “ political operative ” Les Francis, with real-world experience as former executive director of the Democratic National Committee: I don’t need any polls to tell me that Republicans will do well in November. The “out” party almost always shows significant gains in the first midterm election of a new President. Political scientists Joe Bafumi, Bob Erikson, and Chris Wlezien, from elite, out-of-touch, ivory-tower institutions Dartmouth, Columbia, and Temple Universities: Game over. Act 2 After posting, I typically check the comments because so
4 0.26997912 1570 andrew gelman stats-2012-11-08-Poll aggregation and election forecasting
Introduction: At the sister blog, Henry writes about poll averaging and election forecasts. Henry writes that “These models need to crunch lots of polls, at the state and national level, if they’re going to provide good predictions.” Actually, you can get reasonable predictions from national-level forecasting models plus previous state-level election results, then when the election comes closer you can use national and state polls as needed. See my paper with Kari Lock, Bayesian combination of state polls and election forecasts . (That said, the method in that paper is fairly complicated, much more so than simply taking weighted averages of state polls, if such abundant data happen to be available. And I’m sure our approach would need to be altered if it were used for real-time forecasts.) Having a steady supply of polls of varying quality from various sources allows poll aggregators to produce news every day (in the sense of pushing their estimates around) but it doesn’t help much with a
5 0.26213753 270 andrew gelman stats-2010-09-12-Comparison of forecasts for the 2010 congressional elections
Introduction: Yesterday at the sister blog , Nate Silver forecast that the Republicans have a two-thirds chance of regaining the House of Representatives in the upcoming election, with an expected gain of 45 House seats. Last month, Bafumi, Erikson, and Wlezien released their forecast that gives the Republicans an 80% chance of takeover and an expected gain of 50 seats. As all the above writers emphasize, these forecasts are full of uncertainty, so I treat the two predictions–a 45-seat swing or a 50-seat swing–as essentially identical at the national level. And, as regular readers know, as far back as a year ago , the generic Congressional ballot (those questions of the form, “Which party do you plan to vote for in November?”) was also pointing to big Republican gains. As Bafumi et al. point out, early generic polls are strongly predictive of the election outcome, but they need to be interpreted carefully. The polls move in a generally predictable manner during the year leading up to an
7 0.21223372 2343 andrew gelman stats-2014-05-22-Big Data needs Big Model
9 0.16046505 131 andrew gelman stats-2010-07-07-A note to John
10 0.15652701 250 andrew gelman stats-2010-09-02-Blending results from two relatively independent multi-level models
11 0.15008719 2255 andrew gelman stats-2014-03-19-How Americans vote
12 0.14685428 2228 andrew gelman stats-2014-02-28-Combining two of my interests
13 0.14543027 391 andrew gelman stats-2010-11-03-Some thoughts on election forecasting
14 0.14436558 394 andrew gelman stats-2010-11-05-2010: What happened?
15 0.13987774 286 andrew gelman stats-2010-09-20-Are the Democrats avoiding a national campaign?
16 0.13941859 389 andrew gelman stats-2010-11-01-Why it can be rational to vote
17 0.13941859 1565 andrew gelman stats-2012-11-06-Why it can be rational to vote
18 0.13228968 1564 andrew gelman stats-2012-11-06-Choose your default, or your default will choose you (election forecasting edition)
19 0.12948573 292 andrew gelman stats-2010-09-23-Doug Hibbs on the fundamentals in 2010
topicId topicWeight
[(0, 0.179), (1, -0.055), (2, 0.182), (3, 0.108), (4, -0.111), (5, 0.013), (6, -0.087), (7, -0.066), (8, 0.021), (9, -0.018), (10, 0.101), (11, 0.049), (12, 0.034), (13, -0.1), (14, -0.096), (15, -0.003), (16, -0.042), (17, -0.001), (18, 0.012), (19, 0.033), (20, -0.059), (21, 0.066), (22, -0.009), (23, 0.073), (24, 0.014), (25, 0.027), (26, 0.024), (27, 0.023), (28, -0.009), (29, 0.095), (30, -0.053), (31, 0.051), (32, -0.017), (33, -0.059), (34, 0.025), (35, 0.071), (36, -0.02), (37, 0.006), (38, 0.042), (39, -0.034), (40, -0.029), (41, 0.059), (42, 0.052), (43, 0.002), (44, -0.037), (45, 0.024), (46, 0.035), (47, -0.029), (48, 0.066), (49, 0.005)]
simIndex simValue blogId blogTitle
same-blog 1 0.95955652 1512 andrew gelman stats-2012-09-27-A Non-random Walk Down Campaign Street
Introduction: Political campaigns are commonly understood as random walks, during which, at any point in time, the level of support for any party or candidate is equally likely to go up or down. Each shift in the polls is then interpreted as the result of some combination of news and campaign strategies. A completely different story of campaigns is the mean reversion model in which the elections are determined by fundamental factors of the economy and partisanship; the role of the campaign is to give voters a chance to reach their predetermined positions. The popularity of the random walk model for polls may be partially explained via analogy to the widespread idea that stock prices reflect all available information, as popularized in Burton Malkiel’s book, A Random Walk Down Wall Street. Once the idea has sunk in that short-term changes in the stock market are inherently unpredictable, it is natural for journalists to think the same of polls. For example, political analyst Nate Silver wrote
2 0.89126247 270 andrew gelman stats-2010-09-12-Comparison of forecasts for the 2010 congressional elections
Introduction: Yesterday at the sister blog , Nate Silver forecast that the Republicans have a two-thirds chance of regaining the House of Representatives in the upcoming election, with an expected gain of 45 House seats. Last month, Bafumi, Erikson, and Wlezien released their forecast that gives the Republicans an 80% chance of takeover and an expected gain of 50 seats. As all the above writers emphasize, these forecasts are full of uncertainty, so I treat the two predictions–a 45-seat swing or a 50-seat swing–as essentially identical at the national level. And, as regular readers know, as far back as a year ago , the generic Congressional ballot (those questions of the form, “Which party do you plan to vote for in November?”) was also pointing to big Republican gains. As Bafumi et al. point out, early generic polls are strongly predictive of the election outcome, but they need to be interpreted carefully. The polls move in a generally predictable manner during the year leading up to an
3 0.87191701 1570 andrew gelman stats-2012-11-08-Poll aggregation and election forecasting
Introduction: At the sister blog, Henry writes about poll averaging and election forecasts. Henry writes that “These models need to crunch lots of polls, at the state and national level, if they’re going to provide good predictions.” Actually, you can get reasonable predictions from national-level forecasting models plus previous state-level election results, then when the election comes closer you can use national and state polls as needed. See my paper with Kari Lock, Bayesian combination of state polls and election forecasts . (That said, the method in that paper is fairly complicated, much more so than simply taking weighted averages of state polls, if such abundant data happen to be available. And I’m sure our approach would need to be altered if it were used for real-time forecasts.) Having a steady supply of polls of varying quality from various sources allows poll aggregators to produce news every day (in the sense of pushing their estimates around) but it doesn’t help much with a
4 0.87113291 292 andrew gelman stats-2010-09-23-Doug Hibbs on the fundamentals in 2010
Introduction: Hibbs, one of the original economy-and-elections guys, writes : The number of House seats won by the presidents party at midterm elections is well explained by three pre-determined or exogenous variables: (1) the number of House seats won by the in-party at the previous on-year election, (2) the vote margin of the in-partys candidate at the previous presidential election, and (3) the average growth rate of per capita real disposable personal income during the congressional term. Given the partisan division of House seats following the 2008 on-year election, President Obamas margin of victory in 2008, and the weak growth of per capita real income during the rst 6 quarters of the 111th Congress, the Democrats chances of holding on to a House majority by winning at least 218 seats at the 2010 midterm election will depend on real income growth in the 3rd quarter of 2010. The data available at this writing indicate the that Democrats will win 211 seats, a loss of 45 from the 2008 o
5 0.85407877 364 andrew gelman stats-2010-10-22-Politics is not a random walk: Momentum and mean reversion in polling
Introduction: Nate Silver and Justin Wolfers are having a friendly blog-dispute about momentum in political polling. Nate and Justin each make good points but are also missing parts of the picture. These questions relate to my own research so I thought I’d discuss them here. There ain’t no mo’ Nate led off the discussion by writing that pundits are always talking about “momentum” in the polls: Turn on the news or read through much of the analysis put out by some of our friends, and you’re likely to hear a lot of talk about “momentum”: the term is used about 60 times per day by major media outlets in conjunction with articles about polling. When people say a particular candidate has momentum, what they are implying is that present trends are likely to perpetuate themselves into the future. Say, for instance, that a candidate trailed by 10 points in a poll three weeks ago — and now a new poll comes out showing the candidate down by just 5 points. It will frequently be said that this
6 0.84357601 43 andrew gelman stats-2010-05-19-What do Tuesday’s elections tell us about November?
7 0.84258944 237 andrew gelman stats-2010-08-27-Bafumi-Erikson-Wlezien predict a 50-seat loss for Democrats in November
8 0.80402553 300 andrew gelman stats-2010-09-28-A calibrated Cook gives Dems the edge in Nov, sez Sandy
9 0.80121964 1000 andrew gelman stats-2011-11-10-Forecasting 2012: How much does ideology matter?
10 0.79964566 2343 andrew gelman stats-2014-05-22-Big Data needs Big Model
11 0.79887819 210 andrew gelman stats-2010-08-16-What I learned from those tough 538 commenters
13 0.75166988 391 andrew gelman stats-2010-11-03-Some thoughts on election forecasting
14 0.74656439 934 andrew gelman stats-2011-09-30-Nooooooooooooooooooo!
16 0.73171967 654 andrew gelman stats-2011-04-09-There’s no evidence that voters choose presidential candidates based on their looks
17 0.72126412 1556 andrew gelman stats-2012-11-01-Recently in the sister blogs: special pre-election edition!
18 0.71564031 131 andrew gelman stats-2010-07-07-A note to John
19 0.71081394 286 andrew gelman stats-2010-09-20-Are the Democrats avoiding a national campaign?
20 0.7106812 649 andrew gelman stats-2011-04-05-Internal and external forecasting
topicId topicWeight
[(5, 0.26), (9, 0.036), (16, 0.031), (21, 0.02), (24, 0.095), (43, 0.012), (50, 0.023), (53, 0.02), (63, 0.031), (76, 0.015), (84, 0.014), (98, 0.028), (99, 0.26)]
simIndex simValue blogId blogTitle
1 0.97730315 422 andrew gelman stats-2010-11-20-A Gapminder-like data visualization package
Introduction: Ossama Hamed writes in with a new dynamic graphing software: I have the pleasure to brief you on our Data Visualization software “Trend Compass”. TC is a new concept in viewing statistics and trends in an animated way by displaying in one chart 5 axis (X, Y, Time, Bubble size & Bubble color) instead of just the traditional X and Y axis. . . .
2 0.96748114 1250 andrew gelman stats-2012-04-07-Hangman tips
Introduction: Jeff pointed me to this article by Nick Berry. It’s kind of fun but of course if you know your opponent will be following this strategy you can figure out how to outwit it. Also, Berry writes that ETAOIN SHRDLU CMFWYP VBGKQJ XZ is the “ordering of letter frequency in English language.” Indeed this is the conventional ordering but nobody thinks it’s right anymore. See here (with further discussion here ). I wonder what corpus he’s using. P.S. Klutz was my personal standby.
3 0.9510026 665 andrew gelman stats-2011-04-17-Yes, your wish shall be granted (in 25 years)
Introduction: This one was so beautiful I just had to repost it: From the New York Times, 9 Sept 1981: IF I COULD CHANGE PARK SLOPE If I could change Park Slope I would turn it into a palace with queens and kings and princesses to dance the night away at the ball. The trees would look like garden stalks. The lights would look like silver pearls and the dresses would look like soft silver silk. You should see the ball. It looks so luxurious to me. The Park Slope ball is great. Can you guess what street it’s on? “Yes. My street. That’s Carroll Street.” – Jennifer Chatmon, second grade, P.S. 321 This was a few years before my sister told me that she felt safer having a crack house down the block because the cops were surveilling it all the time.
Introduction: As I wrote a couple years ago: Even though statistical analysis has demonstrated that presidential elections are predictable given economic conditions and previous votes in the states . . . it certainly doesn’t mean that every election can be accurately predicted ahead of time. Presidential general election campaigns have several distinct features that distinguish them from most other elections: 1. Two major candidates; 2. The candidates clearly differ in their political ideologies and in their positions on economic issues; 3. The two sides have roughly equal financial and organizational resources; 4. The current election is the latest in a long series of similar contests (every four years); 5. A long campaign, giving candidates a long time to present their case and giving voters a long time to make up their minds. Other elections look different. . . . Or, as I said in reference to the current NYC mayoral election: Et selon Andrew Gelman, expert de l’universi
5 0.94370866 87 andrew gelman stats-2010-06-15-Statistical analysis and visualization of the drug war in Mexico
Introduction: Christian points me to this interesting (but sad) analysis by Diego Valle with an impressive series of graphs. There are a few things I’d change (notably the R default settings which result in ridiculously over-indexed y-axes, as well as axes for homicide rates which should (but do not) go town to zero (and sometimes, bizarrely, go negative), and a lack of coherent ordering of the 32 states (including D.F.), I’m no expert on Mexico (despite having coauthored a paper on Mexican politics) so I’ll leave it to others to evaluate the substantive claims in Valle’s blog. Just looking at what he’s done, though, it seems impressive to me. To put it another way, it’s like something Nate Silver might do.
6 0.94367522 224 andrew gelman stats-2010-08-22-Mister P gets married
7 0.94300854 513 andrew gelman stats-2011-01-12-“Tied for Warmest Year On Record”
same-blog 8 0.92625439 1512 andrew gelman stats-2012-09-27-A Non-random Walk Down Campaign Street
9 0.92585146 228 andrew gelman stats-2010-08-24-A new efficient lossless compression algorithm
10 0.91488492 1286 andrew gelman stats-2012-04-28-Agreement Groups in US Senate and Dynamic Clustering
12 0.88497138 1606 andrew gelman stats-2012-12-05-The Grinch Comes Back
13 0.88456172 364 andrew gelman stats-2010-10-22-Politics is not a random walk: Momentum and mean reversion in polling
14 0.87751567 123 andrew gelman stats-2010-07-01-Truth in headlines
15 0.87716043 1052 andrew gelman stats-2011-12-11-Rational Turbulence
16 0.86966598 2194 andrew gelman stats-2014-02-01-Recently in the sister blog
17 0.86011481 951 andrew gelman stats-2011-10-11-Data mining efforts for Obama’s campaign
18 0.85270631 1841 andrew gelman stats-2013-05-04-The Folk Theorem of Statistical Computing
20 0.83524024 1207 andrew gelman stats-2012-03-10-A quick suggestion