andrew_gelman_stats andrew_gelman_stats-2014 andrew_gelman_stats-2014-2332 knowledge-graph by maker-knowledge-mining
Source: html
Introduction: Pro tip: Don’t believe any claims about results not shown in a paper. Even if the paper has been published. Even if it’s been cited hundreds of times. If the results aren’t shown, they haven’t been checked. I learned this the hard way after receiving this note from Bin Liu, who wrote: Today I saw a paper [by Ziheng Yang and Carlos Rodríguez] titled “Searching for efficient Markov chain Monte Carlo proposal kernels.” The authors cited your work: “Gelman A, Roberts GO, Gilks WR (1996) Bayesian Statistics 5, eds Bernardo JM, et al. (Oxford Univ Press, Oxford), Vol 5, pp 599-607″, i.e. ref.6 in the paper. In the last sentence of pp.19310, the authors write that “… virtually no study has examined alternative kernels; this appears to be due to the influence of ref. 6, which claimed that different kernels had nearly identical performance. This conclusion is incorrect.” Here’s our paper, and here’s the offending quote, which appeared after we discussed results for the no
sentIndex sentText sentNum sentScore
1 Pro tip: Don’t believe any claims about results not shown in a paper. [sent-1, score-0.123]
2 If the results aren’t shown, they haven’t been checked. [sent-4, score-0.064]
3 ” The authors cited your work: “Gelman A, Roberts GO, Gilks WR (1996) Bayesian Statistics 5, eds Bernardo JM, et al. [sent-6, score-0.063]
4 6, which claimed that different kernels had nearly identical performance. [sent-13, score-0.115]
5 max=round(sqrt(n_iter)))$acf eff[j] <- 1/(2*sum(corrs)-1) p_mean[j] <- mean (p_save[,j]) esjd[j] <- mean (esjd_save[,j]) } return (cbind(scale,eff,p_mean,esjd)) } norm <- sims ("normal", seq(0. [sent-20, score-0.292]
6 ] Anyway, the simulations confirmed that Yang and Rodriguez were correct: we had been flat-out wrong in that passage from our influential 1996 paper. [sent-29, score-0.066]
7 The funny thing is, it was always my intuition that the uniform and, even more so, the bimodal jumping distributions would do better than the normal in that 1-d case. [sent-30, score-0.723]
8 Indeed, it is obvious that a normal jumping kernel is a poor choice in one dimension, and I’m embarrassed to have not rechecked our claims, back then! [sent-32, score-0.837]
9 That said, I doubt that these results will make much difference in higher dimensions where a normal kernel is close to a uniform draw from the sphere, so that you actually are moving some reasonable distance on each jump. [sent-33, score-0.905]
10 And in some applications, a unimodal kernel could have some advantages in that in some sense it could be considered as an adaptive solution, in that it occasionally makes small jumps and occasionally big jumps. [sent-34, score-0.63]
11 Indeed, perhaps a longer-tailed jumping distribution such as a t_4 could be even safer as a generic jumping rule in an algorithm that uses one-dimensional Metropolis jumps. [sent-35, score-0.455]
12 I haven’t been thinking too much about these things lately because now I’ve been fitting my models in Stan, which uses Hamiltonian Monte Carlo and works in multiple dimensions. [sent-36, score-0.055]
wordName wordTfidf (topN-words)
[('kernel', 0.45), ('yang', 0.23), ('jumping', 0.2), ('bimodal', 0.2), ('sims', 0.194), ('na', 0.192), ('normal', 0.187), ('rodriguez', 0.172), ('scale', 0.172), ('rnorm', 0.163), ('seq', 0.163), ('sqrt', 0.16), ('runif', 0.156), ('rep', 0.142), ('array', 0.139), ('uniform', 0.136), ('acf', 0.126), ('corrs', 0.126), ('eff', 0.126), ('kernels', 0.115), ('esjd', 0.108), ('oxford', 0.089), ('width', 0.086), ('else', 0.084), ('metropolis', 0.08), ('carlo', 0.072), ('distance', 0.068), ('monte', 0.068), ('shift', 0.067), ('simulations', 0.066), ('occasionally', 0.065), ('results', 0.064), ('cited', 0.063), ('shown', 0.059), ('pre', 0.057), ('summed', 0.057), ('cbind', 0.057), ('wr', 0.057), ('carlos', 0.057), ('uses', 0.055), ('bernardo', 0.054), ('vol', 0.054), ('sphere', 0.052), ('wordpress', 0.052), ('code', 0.05), ('autocorrelations', 0.05), ('unimodal', 0.05), ('stan', 0.05), ('expected', 0.049), ('mean', 0.049)]
simIndex simValue blogId blogTitle
same-blog 1 1.0000001 2332 andrew gelman stats-2014-05-12-“The results (not shown) . . .”
Introduction: Pro tip: Don’t believe any claims about results not shown in a paper. Even if the paper has been published. Even if it’s been cited hundreds of times. If the results aren’t shown, they haven’t been checked. I learned this the hard way after receiving this note from Bin Liu, who wrote: Today I saw a paper [by Ziheng Yang and Carlos Rodríguez] titled “Searching for efficient Markov chain Monte Carlo proposal kernels.” The authors cited your work: “Gelman A, Roberts GO, Gilks WR (1996) Bayesian Statistics 5, eds Bernardo JM, et al. (Oxford Univ Press, Oxford), Vol 5, pp 599-607″, i.e. ref.6 in the paper. In the last sentence of pp.19310, the authors write that “… virtually no study has examined alternative kernels; this appears to be due to the influence of ref. 6, which claimed that different kernels had nearly identical performance. This conclusion is incorrect.” Here’s our paper, and here’s the offending quote, which appeared after we discussed results for the no
Introduction: David Duvenaud writes: I’ve been following your recent discussions about how an AI could do statistics [see also here ]. I was especially excited about your suggestion for new statistical methods using “a language-like approach to recursively creating new models from a specified list of distributions and transformations, and an automatic approach to checking model fit.” Your discussion of these ideas was exciting to me and my colleagues because we recently did some work taking a step in this direction, automatically searching through a grammar over Gaussian process regression models. Roger Grosse previously did the same thing , but over matrix decomposition models using held-out predictive likelihood to check model fit. These are both examples of automatic Bayesian model-building by a search over more and more complex models, as you suggested. One nice thing is that both grammars include lots of standard models for free, and they seem to work pretty well, although the
3 0.18040852 2161 andrew gelman stats-2014-01-07-My recent debugging experience
Introduction: OK, so this sort of thing happens sometimes. I was working on a new idea (still working on it; if it ultimately works out—or if it doesn’t—I’ll let you know) and as part of it I was fitting little models in Stan, in a loop. I thought it would make sense to start with linear regression with normal priors and known data variance, because then the exact solution is Gaussian and I can also work with the problem analytically. So I programmed up the algorithm and, no surprise, it didn’t work. I went through my R code, put in print statements here and there, and cleared out bug after bug until at least it stopped crashing. But the algorithm still wasn’t doing what it was supposed to do. So I decided to do something simpler, and just check that the Stan linear regression gave the same answer as the analytic posterior distribution: I ran Stan for tons of iterations, then computed the sample mean and variance of the simulations. It was an example with two coefficients—I’d originally cho
Introduction: Marc Tanguay writes in with a specific question that has a very general answer. First, the question: I [Tanguay] am currently running a MCMC for which I have 3 parameters that are restricted to a specific space. 2 are bounded between 0 and 1 while the third is binary and updated by a Beta-Binomial. Since my priors are also bounded, I notice that, conditional on All the rest (which covers both data and other parameters), the density was not varying a lot within the space of the parameters. As a result, the acceptance rate is high, about 85%, and this despite the fact that all the parameter’s space is explore. Since in your book, the optimal acceptance rates prescribed are lower that 50% (in case of multiple parameters), do you think I should worry about getting 85%. Or is this normal given the restrictions on the parameters? First off: Yes, my guess is that you should be taking bigger jumps. 85% seems like too high an acceptance rate for Metropolis jumping. More generally, t
5 0.11371411 1478 andrew gelman stats-2012-08-31-Watercolor regression
Introduction: Solomon Hsiang writes: Two small follow-ups based on the discussion (the second/bigger one is to address your comment about the 95% CI edges). 1. I realized that if we plot the confidence intervals as a solid color that fades (eg. using the “fixed ink” scheme from before) we can make sure the regression line also has heightened visual weight where confidence is high by plotting the line white. This makes the contrast (and thus visual weight) between the regression line and the CI highest when the CI is narrow and dark. As the CI fade near the edges, so does the contrast with the regression line. This is a small adjustment, but I like it because it is so simple and it makes the graph much nicer. (see “visually_weighted_fill_reverse” attached). My posted code has been updated to do this automatically. 2. You and your readers didn’t like that the edges of the filled CI were so sharp and arbitrary. But I didn’t like that the contrast between the spaghetti lines and the background
6 0.11292163 965 andrew gelman stats-2011-10-19-Web-friendly visualizations in R
7 0.098237194 320 andrew gelman stats-2010-10-05-Does posterior predictive model checking fit with the operational subjective approach?
8 0.097497366 1361 andrew gelman stats-2012-06-02-Question 23 of my final exam for Design and Analysis of Sample Surveys
9 0.095205046 1333 andrew gelman stats-2012-05-20-Question 10 of my final exam for Design and Analysis of Sample Surveys
10 0.089038938 773 andrew gelman stats-2011-06-18-Should we always be using the t and robit instead of the normal and logit?
11 0.088754177 952 andrew gelman stats-2011-10-11-More reason to like Sims besides just his name
12 0.085869856 1309 andrew gelman stats-2012-05-09-The first version of my “inference from iterative simulation using parallel sequences” paper!
13 0.082018755 2176 andrew gelman stats-2014-01-19-Transformations for non-normal data
14 0.076977849 1941 andrew gelman stats-2013-07-16-Priors
15 0.074543118 1767 andrew gelman stats-2013-03-17-The disappearing or non-disappearing middle class
16 0.072549336 1474 andrew gelman stats-2012-08-29-More on scaled-inverse Wishart and prior independence
17 0.072118625 419 andrew gelman stats-2010-11-18-Derivative-based MCMC as a breakthrough technique for implementing Bayesian statistics
18 0.067362085 1156 andrew gelman stats-2012-02-06-Bayesian model-building by pure thought: Some principles and examples
20 0.066892989 669 andrew gelman stats-2011-04-19-The mysterious Gamma (1.4, 0.4)
topicId topicWeight
[(0, 0.118), (1, 0.041), (2, 0.008), (3, 0.011), (4, 0.029), (5, -0.008), (6, 0.041), (7, -0.064), (8, -0.046), (9, -0.045), (10, -0.013), (11, -0.012), (12, -0.046), (13, -0.007), (14, 0.006), (15, -0.021), (16, 0.002), (17, 0.022), (18, 0.006), (19, -0.037), (20, 0.02), (21, 0.006), (22, 0.005), (23, -0.007), (24, 0.047), (25, 0.028), (26, -0.015), (27, 0.025), (28, -0.014), (29, -0.004), (30, -0.002), (31, 0.026), (32, 0.003), (33, -0.003), (34, 0.03), (35, -0.022), (36, -0.02), (37, 0.007), (38, -0.037), (39, -0.024), (40, 0.009), (41, 0.015), (42, -0.007), (43, 0.013), (44, -0.035), (45, 0.007), (46, -0.025), (47, 0.008), (48, 0.045), (49, -0.015)]
simIndex simValue blogId blogTitle
same-blog 1 0.94590402 2332 andrew gelman stats-2014-05-12-“The results (not shown) . . .”
Introduction: Pro tip: Don’t believe any claims about results not shown in a paper. Even if the paper has been published. Even if it’s been cited hundreds of times. If the results aren’t shown, they haven’t been checked. I learned this the hard way after receiving this note from Bin Liu, who wrote: Today I saw a paper [by Ziheng Yang and Carlos Rodríguez] titled “Searching for efficient Markov chain Monte Carlo proposal kernels.” The authors cited your work: “Gelman A, Roberts GO, Gilks WR (1996) Bayesian Statistics 5, eds Bernardo JM, et al. (Oxford Univ Press, Oxford), Vol 5, pp 599-607″, i.e. ref.6 in the paper. In the last sentence of pp.19310, the authors write that “… virtually no study has examined alternative kernels; this appears to be due to the influence of ref. 6, which claimed that different kernels had nearly identical performance. This conclusion is incorrect.” Here’s our paper, and here’s the offending quote, which appeared after we discussed results for the no
2 0.75636226 2231 andrew gelman stats-2014-03-03-Running into a Stan Reference by Accident
Introduction: We were talking about parallelizing MCMC and I came up with what I thought was a neat idea for parallelizing MCMC (sample with fractional prior, average samples on a per-draw basis). But then I realized this approach could get the right posterior mean or right posterior variance, but not both, depending on how the prior was divided (for a beta-binomial example). Then Aki told me it had already been done in a more general form in a paper of Scott et al., Bayes and Big Data , which was then used as the baseline in: Willie Neiswanger, Chong Wang, and Eric Xing. 2013. Asymptotically Exact, Embarrassingly Parallel MCMC . arXiv 1311.4780. It’s a neat paper, which Xi’an already blogged about months ago. But what really struck me was the following quote: We use Stan, an automated Hamiltonian Monte Carlo (HMC) software package, to perform sampling for both the true posterior (for groundtruth and comparison methods) and for the subposteriors on each machine. One advantage o
3 0.75107867 535 andrew gelman stats-2011-01-24-Bleg: Automatic Differentiation for Log Prob Gradients?
Introduction: We need help picking out an automatic differentiation package for Hamiltonian Monte Carlo sampling from the posterior of a generalized linear model with deep interactions. Specifically, we need to compute gradients for log probability functions with thousands of parameters that involve matrix (determinants, eigenvalues, inverses), stats (distributions), and math (log gamma) functions. Any suggestions? The Application: Hybrid Monte Carlo for Posteriors We’re getting serious about implementing posterior sampling using Hamiltonian Monte Carlo. HMC speeds up mixing by including gradient information to help guide the Metropolis proposals toward areas high probability. In practice, the algorithm requires a handful or of gradient calculations per sample, but there are many dimensions and the functions are hairy enough we don’t want to compute derivaties by hand. Auto Diff: Perhaps not What you Think It may not have been clear to readers of this blog that automatic diffe
4 0.73626751 1753 andrew gelman stats-2013-03-06-Stan 1.2.0 and RStan 1.2.0
Introduction: Stan 1.2.0 and RStan 1.2.0 are now available for download. See: http://mc-stan.org/ Here are the highlights. Full Mass Matrix Estimation during Warmup Yuanjun Gao, a first-year grad student here at Columbia (!), built a regularized mass-matrix estimator. This helps for posteriors with high correlation among parameters and varying scales. We’re still testing this ourselves, so the estimation procedure may change in the future (don’t worry — it satisfies detailed balance as is, but we might be able to make it more computationally efficient in terms of time per effective sample). It’s not the default option. The major reason is the matrix operations required are expensive, raising the algorithm cost to , where is the average number of leapfrog steps, is the number of iterations, and is the number of parameters. Yuanjun did a great job with the Cholesky factorizations and implemented this about as efficiently as is possible. (His homework for Andrew’s class w
Introduction: From August 1990. It was in the form of a note sent to all the people in the statistics group of Bell Labs, where I’d worked that summer. To all: Here’s the abstract of the work I’ve done this summer. It’s stored in the file, /fs5/gelman/abstract.bell, and copies of the Figures 1-3 are on Trevor’s desk. Any comments are of course appreciated; I’m at gelman@stat.berkeley.edu. On the Routine Use of Markov Chains for Simulation Andrew Gelman and Donald Rubin, 6 August 1990 corrected version: 8 August 1990 1. Simulation In probability and statistics we can often specify multivariate distributions many of whose properties we do not fully understand–perhaps, as in the Ising model of statistical physics, we can write the joint density function, up to a multiplicative constant that cannot be expressed in closed form. For an example in statistics, consider the Normal random effects model in the analysis of variance, which can be easily placed in a Bayesian fram
6 0.72537309 2003 andrew gelman stats-2013-08-30-Stan Project: Continuous Relaxations for Discrete MRFs
7 0.72352993 2161 andrew gelman stats-2014-01-07-My recent debugging experience
8 0.71691442 931 andrew gelman stats-2011-09-29-Hamiltonian Monte Carlo stories
9 0.71568602 1799 andrew gelman stats-2013-04-12-Stan 1.3.0 and RStan 1.3.0 Ready for Action
11 0.70457006 555 andrew gelman stats-2011-02-04-Handy Matrix Cheat Sheet, with Gradients
12 0.68799198 1036 andrew gelman stats-2011-11-30-Stan uses Nuts!
13 0.68462449 1710 andrew gelman stats-2013-02-06-The new Stan 1.1.1, featuring Gaussian processes!
14 0.66479492 1339 andrew gelman stats-2012-05-23-Learning Differential Geometry for Hamiltonian Monte Carlo
15 0.64883292 2258 andrew gelman stats-2014-03-21-Random matrices in the news
16 0.64367545 2311 andrew gelman stats-2014-04-29-Bayesian Uncertainty Quantification for Differential Equations!
17 0.63580763 858 andrew gelman stats-2011-08-17-Jumping off the edge of the world
18 0.6333074 2020 andrew gelman stats-2013-09-12-Samplers for Big Science: emcee and BAT
19 0.63066018 1627 andrew gelman stats-2012-12-17-Stan and RStan 1.1.0
20 0.63037837 2201 andrew gelman stats-2014-02-06-Bootstrap averaging: Examples where it works and where it doesn’t work
topicId topicWeight
[(1, 0.022), (4, 0.03), (6, 0.101), (8, 0.036), (15, 0.013), (16, 0.045), (21, 0.037), (24, 0.079), (38, 0.027), (49, 0.018), (55, 0.021), (56, 0.017), (57, 0.012), (59, 0.032), (61, 0.068), (65, 0.019), (73, 0.056), (86, 0.013), (99, 0.242)]
simIndex simValue blogId blogTitle
same-blog 1 0.95260996 2332 andrew gelman stats-2014-05-12-“The results (not shown) . . .”
Introduction: Pro tip: Don’t believe any claims about results not shown in a paper. Even if the paper has been published. Even if it’s been cited hundreds of times. If the results aren’t shown, they haven’t been checked. I learned this the hard way after receiving this note from Bin Liu, who wrote: Today I saw a paper [by Ziheng Yang and Carlos Rodríguez] titled “Searching for efficient Markov chain Monte Carlo proposal kernels.” The authors cited your work: “Gelman A, Roberts GO, Gilks WR (1996) Bayesian Statistics 5, eds Bernardo JM, et al. (Oxford Univ Press, Oxford), Vol 5, pp 599-607″, i.e. ref.6 in the paper. In the last sentence of pp.19310, the authors write that “… virtually no study has examined alternative kernels; this appears to be due to the influence of ref. 6, which claimed that different kernels had nearly identical performance. This conclusion is incorrect.” Here’s our paper, and here’s the offending quote, which appeared after we discussed results for the no
2 0.9143756 1906 andrew gelman stats-2013-06-19-“Behind a cancer-treatment firm’s rosy survival claims”
Introduction: Brett Keller points to a recent news article by Sharon Begley and Robin Respaut: A lot of doctors, hospitals and other healthcare providers in the United States decline to treat people who can’t pay, or have inadequate insurance, among other reasons. What sets CTCA [Cancer Treatment Centers of America] apart is that rejecting certain patients and, even more, culling some of its patients from its survival data lets the company tout in ads and post on its website patient outcomes that look dramatically better than they would if the company treated all comers. These are the rosy survival numbers . . . Details: CTCA reports on its website that the percentage of its patients who are alive after six months, a year, 18 months and longer regularly tops national figures. For instance, 60 percent of its non-small-cell lung cancer patients are alive at six months, CTCA says, compared to 38 percent nationally. And 64 percent of its prostate cancer patients are alive at three years, vers
3 0.91065753 819 andrew gelman stats-2011-07-24-Don’t idealize “risk aversion”
Introduction: Richard Thaler writes (click here and search on Thaler): Both risk and risk aversion are concepts that were once well defined, but are now in danger of becoming Aetherized [this is Thaler's term for adding free parameters to a model to make it work, thus destroying the purity and much of the value of the original model]. Stocks that earn surprisingly high returns are labeled as risky, because in the theory, excess returns must be accompanied by higher risk. If, inconveniently, the traditional measures of risk such as variance or covariance with the market are not high, then the Aetherists tell us there must be some other risk; we just don’t know what it is. Similarly, traditionally the concept of risk aversion was taken to be a primitive; each person had a parameter, gamma, that measured her degree of risk aversion. Now risk aversion is allowed to be time varying, and Aetherists can say with a straight face that the market crashes of 2001 and 2008 were caused by sudden increases
4 0.9103722 221 andrew gelman stats-2010-08-21-Busted!
Introduction: I’m just glad that universities don’t sanction professors for publishing false theorems. If the guy really is nailed by the feds for fraud, I hope they don’t throw him in prison. In general, prison time seems like a brutal, expensive, and inefficient way to punish people. I’d prefer if the government just took 95% of his salary for several years, made him do community service (cleaning equipment at the local sewage treatment plant, perhaps; a lab scientist should be good at this sort of thing, no?), etc. If restriction of this dude’s personal freedom is judged be part of the sentence, he could be given some sort of electronic tag that would send a message to the police if he were ever more than 3 miles from his home. But no need to bill the taxpayers for the cost of keeping him in prison.
5 0.90454286 618 andrew gelman stats-2011-03-18-Prior information . . . about the likelihood
Introduction: I read this story by Adrian Chen on Gawker (yeah, yeah, so sue me): Why That ‘NASA Discovers Alien Life’ Story Is Bullshit Fox News has a super-exciting article today: “Exclusive: NASA Scientist claims Evidence of Alien Life on Meteorite.” OMG, aliens exist! Except this NASA scientist has been claiming to have evidence of alien life on meteorites for years. Chen continues with a quote from the Fox News item: [NASA scientist Richard B. Hoover] gave FoxNews.com early access to the out-of-this-world research, published late Friday evening in the March edition of the Journal of Cosmology. In it, Hoover describes the latest findings in his study of an extremely rare class of meteorites, called CI1 carbonaceous chondrites — only nine such meteorites are known to exist on Earth. . . . The bad news is that Hoover reported this same sort of finding in various low-rent venues for several years. Replication, huh? Chen also helpfully points us to the website of the Journal
6 0.90333986 1710 andrew gelman stats-2013-02-06-The new Stan 1.1.1, featuring Gaussian processes!
7 0.90026623 150 andrew gelman stats-2010-07-16-Gaydar update: Additional research on estimating small fractions of the population
8 0.89620775 1924 andrew gelman stats-2013-07-03-Kuhn, 1-f noise, and the fractal nature of scientific revolutions
9 0.8949858 1625 andrew gelman stats-2012-12-15-“I coach the jumpers here at Boise State . . .”
10 0.89175797 650 andrew gelman stats-2011-04-05-Monitor the efficiency of your Markov chain sampler using expected squared jumped distance!
13 0.88178599 21 andrew gelman stats-2010-05-07-Environmentally induced cancer “grossly underestimated”? Doubtful.
14 0.87978268 851 andrew gelman stats-2011-08-12-year + (1|year)
15 0.87940854 1489 andrew gelman stats-2012-09-09-Commercial Bayesian inference software is popping up all over
16 0.87896132 563 andrew gelman stats-2011-02-07-Evaluating predictions of political events
17 0.8765794 1714 andrew gelman stats-2013-02-09-Partial least squares path analysis
18 0.87607765 798 andrew gelman stats-2011-07-12-Sometimes a graph really is just ugly
19 0.87578493 2033 andrew gelman stats-2013-09-23-More on Bayesian methods and multilevel modeling
20 0.87569916 714 andrew gelman stats-2011-05-16-NYT Labs releases Openpaths, a utility for saving your iphone data