andrew_gelman_stats andrew_gelman_stats-2014 andrew_gelman_stats-2014-2231 knowledge-graph by maker-knowledge-mining
Source: html
Introduction: We were talking about parallelizing MCMC and I came up with what I thought was a neat idea for parallelizing MCMC (sample with fractional prior, average samples on a per-draw basis). But then I realized this approach could get the right posterior mean or right posterior variance, but not both, depending on how the prior was divided (for a beta-binomial example). Then Aki told me it had already been done in a more general form in a paper of Scott et al., Bayes and Big Data , which was then used as the baseline in: Willie Neiswanger, Chong Wang, and Eric Xing. 2013. Asymptotically Exact, Embarrassingly Parallel MCMC . arXiv 1311.4780. It’s a neat paper, which Xi’an already blogged about months ago. But what really struck me was the following quote: We use Stan, an automated Hamiltonian Monte Carlo (HMC) software package, to perform sampling for both the true posterior (for groundtruth and comparison methods) and for the subposteriors on each machine. One advantage o
sentIndex sentText sentNum sentScore
1 We were talking about parallelizing MCMC and I came up with what I thought was a neat idea for parallelizing MCMC (sample with fractional prior, average samples on a per-draw basis). [sent-1, score-1.216]
2 But then I realized this approach could get the right posterior mean or right posterior variance, but not both, depending on how the prior was divided (for a beta-binomial example). [sent-2, score-0.964]
3 Then Aki told me it had already been done in a more general form in a paper of Scott et al. [sent-3, score-0.104]
4 , Bayes and Big Data , which was then used as the baseline in: Willie Neiswanger, Chong Wang, and Eric Xing. [sent-4, score-0.109]
5 It’s a neat paper, which Xi’an already blogged about months ago. [sent-9, score-0.485]
6 But what really struck me was the following quote: We use Stan, an automated Hamiltonian Monte Carlo (HMC) software package, to perform sampling for both the true posterior (for groundtruth and comparison methods) and for the subposteriors on each machine. [sent-10, score-0.601]
7 One advantage of Stan is that it is implemented with C++ and uses the No-U-Turn sampler for HMC, which does not require any user-provided parameters. [sent-11, score-0.459]
8 It’s sort of like telling someone a story at a cocktail party and then having the story retold to you an hour later by a different person. [sent-12, score-0.756]
wordName wordTfidf (topN-words)
[('parallelizing', 0.362), ('mcmc', 0.297), ('neat', 0.271), ('hmc', 0.233), ('posterior', 0.203), ('willie', 0.165), ('retold', 0.165), ('xi', 0.155), ('embarrassingly', 0.155), ('cocktail', 0.155), ('stan', 0.143), ('fractional', 0.136), ('automated', 0.13), ('asymptotically', 0.127), ('wang', 0.121), ('aki', 0.115), ('sampler', 0.112), ('prior', 0.112), ('blogged', 0.11), ('baseline', 0.109), ('arxiv', 0.108), ('hamiltonian', 0.108), ('parallel', 0.106), ('already', 0.104), ('depending', 0.104), ('struck', 0.103), ('implemented', 0.103), ('carlo', 0.103), ('divided', 0.101), ('scott', 0.098), ('monte', 0.098), ('eric', 0.094), ('hour', 0.093), ('exact', 0.091), ('story', 0.091), ('realized', 0.091), ('package', 0.086), ('perform', 0.086), ('samples', 0.085), ('require', 0.084), ('telling', 0.084), ('basis', 0.083), ('advantage', 0.082), ('software', 0.079), ('uses', 0.078), ('party', 0.077), ('right', 0.075), ('variance', 0.074), ('quote', 0.073), ('bayes', 0.073)]
simIndex simValue blogId blogTitle
same-blog 1 0.99999988 2231 andrew gelman stats-2014-03-03-Running into a Stan Reference by Accident
Introduction: We were talking about parallelizing MCMC and I came up with what I thought was a neat idea for parallelizing MCMC (sample with fractional prior, average samples on a per-draw basis). But then I realized this approach could get the right posterior mean or right posterior variance, but not both, depending on how the prior was divided (for a beta-binomial example). Then Aki told me it had already been done in a more general form in a paper of Scott et al., Bayes and Big Data , which was then used as the baseline in: Willie Neiswanger, Chong Wang, and Eric Xing. 2013. Asymptotically Exact, Embarrassingly Parallel MCMC . arXiv 1311.4780. It’s a neat paper, which Xi’an already blogged about months ago. But what really struck me was the following quote: We use Stan, an automated Hamiltonian Monte Carlo (HMC) software package, to perform sampling for both the true posterior (for groundtruth and comparison methods) and for the subposteriors on each machine. One advantage o
2 0.20361349 1772 andrew gelman stats-2013-03-20-Stan at Google this Thurs and at Berkeley this Fri noon
Introduction: Michael Betancourt will be speaking at Google and at the University of California, Berkeley. The Google talk is closed to outsiders (but if you work at Google, you should go!); the Berkeley talk is open to all: Friday March 22, 12:10 pm, Evans Hall 1011. Title of talk: Stan : Practical Bayesian Inference with Hamiltonian Monte Carlo Abstract: Practical implementations of Bayesian inference are often limited to approximation methods that only slowly explore the posterior distribution. By taking advantage of the curvature of the posterior, however, Hamiltonian Monte Carlo (HMC) efficiently explores even the most highly contorted distributions. In this talk I will review the foundations of and recent developments within HMC, concluding with a discussion of Stan, a powerful inference engine that utilizes HMC, automatic differentiation, and adaptive methods to minimize user input. This is cool stuff. And he’ll be showing the whirlpool movie!
3 0.1893568 931 andrew gelman stats-2011-09-29-Hamiltonian Monte Carlo stories
Introduction: Tomas Iesmantas had asked me for advice on a regression problem with 50 parameters, and I’d recommended Hamiltonian Monte Carlo. A few weeks later he reported back: After trying several modifications (HMC for all parameters at once, HMC just for first level parameters and Riemman manifold Hamiltonian Monte Carlo method), I finally got it running with HMC just for first level parameters and for others using direct sampling, since conditional distributions turned out to have closed form. However, even in this case it is quite tricky, since I had to employ mass matrix and not just diagonal but at the beginning of algorithm generated it randomly (ensuring it is positive definite). Such random generation of mass matrix is quite blind step, but it proved to be quite helpful. Riemman manifold HMC is quite vagarious, or to be more specific, metric of manifold is very sensitive. In my model log-likelihood I had exponents and values of metrics matrix elements was very large and wh
4 0.18029761 1749 andrew gelman stats-2013-03-04-Stan in L.A. this Wed 3:30pm
Introduction: Michael Betancourt will be speaking at UCLA: The location for refreshment is in room 51-254 CHS at 3:00 PM. The place for the seminar is at CHS 33-105A at 3:30pm – 4:30pm, Wed 6 Mar. ["CHS" stands for Center for Health Sciences, the building of the UCLA schools of medicine and public health. Here's a map with directions .] Title of talk: Stan : Practical Bayesian Inference with Hamiltonian Monte Carlo Abstract: Practical implementations of Bayesian inference are often limited to approximation methods that only slowly explore the posterior distribution. By taking advantage of the curvature of the posterior, however, Hamiltonian Monte Carlo (HMC) efficiently explores even the most highly contorted distributions. In this talk I will review the foundations of and recent developments within HMC, concluding with a discussion of Stan, a powerful inference engine that utilizes HMC, automatic differentiation, and adaptive methods to minimize user input. This is cool stuff.
5 0.14713591 1339 andrew gelman stats-2012-05-23-Learning Differential Geometry for Hamiltonian Monte Carlo
Introduction: You can get a taste of Hamiltonian Monte Carlo (HMC) by reading the very gentle introduction in David MacKay’s general text on information theory: MacKay, D. 2003. Information Theory, Inference, and Learning Algorithms . Cambridge University Press. [see Chapter 31, which is relatively standalone and can be downloaded separately.] Follow this up with Radford Neal’s much more thorough introduction to HMC: Neal, R. 2011. MCMC Using Hamiltonian Dynamics . In Brooks, Gelman, Jones and Meng, eds., Handbook of Markov Chain Monte Carlo . Chapman and Hall/CRC Press. To understand why HMC works and set yourself on the path to understanding generalizations like Riemann manifold HMC , you’ll need to know a bit about differential geometry. I really liked the combination of these two books: Magnus, J. R. and H. Neudecker. 2007. Matrix Differential Calculus with Application in Statistics and Econometrics . 3rd Edition. Wiley? and Leimkuhler, B. and S.
6 0.14336115 1475 andrew gelman stats-2012-08-30-A Stan is Born
7 0.14202181 1735 andrew gelman stats-2013-02-24-F-f-f-fake data
8 0.14002292 1036 andrew gelman stats-2011-11-30-Stan uses Nuts!
9 0.13640858 1489 andrew gelman stats-2012-09-09-Commercial Bayesian inference software is popping up all over
11 0.12729925 184 andrew gelman stats-2010-08-04-That half-Cauchy prior
12 0.12626721 2003 andrew gelman stats-2013-08-30-Stan Project: Continuous Relaxations for Discrete MRFs
13 0.12405703 1443 andrew gelman stats-2012-08-04-Bayesian Learning via Stochastic Gradient Langevin Dynamics
14 0.11482875 1941 andrew gelman stats-2013-07-16-Priors
15 0.11096993 2291 andrew gelman stats-2014-04-14-Transitioning to Stan
16 0.10750656 778 andrew gelman stats-2011-06-24-New ideas on DIC from Martyn Plummer and Sumio Watanabe
17 0.10501287 419 andrew gelman stats-2010-11-18-Derivative-based MCMC as a breakthrough technique for implementing Bayesian statistics
18 0.10260285 1682 andrew gelman stats-2013-01-19-R package for Bayes factors
19 0.10197562 2161 andrew gelman stats-2014-01-07-My recent debugging experience
20 0.099425659 779 andrew gelman stats-2011-06-25-Avoiding boundary estimates using a prior distribution as regularization
topicId topicWeight
[(0, 0.134), (1, 0.1), (2, -0.002), (3, 0.041), (4, 0.026), (5, 0.031), (6, 0.048), (7, -0.135), (8, -0.113), (9, -0.092), (10, -0.052), (11, -0.024), (12, -0.043), (13, -0.006), (14, 0.033), (15, -0.056), (16, -0.01), (17, 0.039), (18, -0.008), (19, 0.003), (20, -0.027), (21, -0.016), (22, -0.014), (23, 0.012), (24, 0.036), (25, 0.029), (26, -0.054), (27, 0.054), (28, 0.017), (29, 0.02), (30, 0.034), (31, 0.027), (32, 0.032), (33, -0.009), (34, -0.019), (35, 0.018), (36, -0.006), (37, -0.032), (38, -0.034), (39, 0.019), (40, -0.037), (41, 0.052), (42, 0.005), (43, -0.012), (44, 0.022), (45, -0.041), (46, -0.042), (47, 0.002), (48, 0.031), (49, -0.05)]
simIndex simValue blogId blogTitle
same-blog 1 0.96243894 2231 andrew gelman stats-2014-03-03-Running into a Stan Reference by Accident
Introduction: We were talking about parallelizing MCMC and I came up with what I thought was a neat idea for parallelizing MCMC (sample with fractional prior, average samples on a per-draw basis). But then I realized this approach could get the right posterior mean or right posterior variance, but not both, depending on how the prior was divided (for a beta-binomial example). Then Aki told me it had already been done in a more general form in a paper of Scott et al., Bayes and Big Data , which was then used as the baseline in: Willie Neiswanger, Chong Wang, and Eric Xing. 2013. Asymptotically Exact, Embarrassingly Parallel MCMC . arXiv 1311.4780. It’s a neat paper, which Xi’an already blogged about months ago. But what really struck me was the following quote: We use Stan, an automated Hamiltonian Monte Carlo (HMC) software package, to perform sampling for both the true posterior (for groundtruth and comparison methods) and for the subposteriors on each machine. One advantage o
2 0.81499463 1036 andrew gelman stats-2011-11-30-Stan uses Nuts!
Introduction: We interrupt our usual program of Ed Wegman Gregg Easterbrook Niall Ferguson mockery to deliver a serious update on our statistical computing project. Stan (“Sampling Through Adaptive Neighborhoods”) is our new C++ program (written mostly by Bob Carpenter) that draws samples from Bayesian models. Stan can take different sorts of inputs: you can write the model in a Bugs-like syntax and it goes from there, or you can write the log-posterior directly as a C++ function. Most of the computation is done using Hamiltonian Monte Carlo. HMC requires some tuning, so Matt Hoffman up and wrote a new algorithm, Nuts (the “No-U-Turn Sampler”) which optimizes HMC adaptively. In many settings, Nuts is actually more computationally efficient than the optimal static HMC! When the the Nuts paper appeared on Arxiv, Christian Robert noticed it and had some reactions . In response to Xian’s comments, Matt writes: Christian writes: I wonder about the computing time (and the “una
3 0.77314764 931 andrew gelman stats-2011-09-29-Hamiltonian Monte Carlo stories
Introduction: Tomas Iesmantas had asked me for advice on a regression problem with 50 parameters, and I’d recommended Hamiltonian Monte Carlo. A few weeks later he reported back: After trying several modifications (HMC for all parameters at once, HMC just for first level parameters and Riemman manifold Hamiltonian Monte Carlo method), I finally got it running with HMC just for first level parameters and for others using direct sampling, since conditional distributions turned out to have closed form. However, even in this case it is quite tricky, since I had to employ mass matrix and not just diagonal but at the beginning of algorithm generated it randomly (ensuring it is positive definite). Such random generation of mass matrix is quite blind step, but it proved to be quite helpful. Riemman manifold HMC is quite vagarious, or to be more specific, metric of manifold is very sensitive. In my model log-likelihood I had exponents and values of metrics matrix elements was very large and wh
4 0.77165073 2003 andrew gelman stats-2013-08-30-Stan Project: Continuous Relaxations for Discrete MRFs
Introduction: Hamiltonian Monte Carlo (HMC), as used by Stan , is only defined for continuous parameters. We’d love to be able to do discrete sampling. So I was excited when I saw this: Yichuan Zhang, Charles Sutton, Amos J Storkey, and Zoubin Ghahramani. 2012. Continuous Relaxations for Discrete Hamiltonian Monte Carlo . NIPS 25. Abstract: Continuous relaxations play an important role in discrete optimization, but have not seen much use in approximate probabilistic inference. Here we show that a general form of the Gaussian Integral Trick makes it possible to transform a wide class of discrete variable undirected models into fully continuous systems. The continuous representation allows the use of gradient-based Hamiltonian Monte Carlo for inference, results in new ways of estimating normalization constants (partition functions), and in general opens up a number of new avenues for inference in difficult discrete systems. We demonstrate some of these continuous relaxation inference a
5 0.74261498 2150 andrew gelman stats-2013-12-27-(R-Py-Cmd)Stan 2.1.0
Introduction: We’re happy to announce the release of Stan C++, CmdStan, RStan, and PyStan 2.1.0. This is a minor feature release, but it is also an important bug fix release. As always, the place to start is the (all new) Stan web pages: http://mc-stan.org Major Bug in 2.0.0, 2.0.1 Stan 2.0.0 and Stan 2.0.1 introduced a bug in the implementation of the NUTS criterion that led to poor tail exploration and thus biased the posterior uncertainty downward. There was no bug in NUTS in Stan 1.3 or earlier, and 2.1 has been extensively tested and tests put in place so this problem will not recur. If you are using Stan 2.0.0 or 2.0.1, you should switch to 2.1.0 as soon as possible and rerun any models you care about. New Target Acceptance Rate Default for Stan 2.1.0 Another big change aimed at reducing posterior estimation bias was an increase in the target acceptance rate during adaptation from 0.65 to 0.80. The bad news is that iterations will take around 50% longer
6 0.74050033 555 andrew gelman stats-2011-02-04-Handy Matrix Cheat Sheet, with Gradients
7 0.73555255 2161 andrew gelman stats-2014-01-07-My recent debugging experience
8 0.71440655 1753 andrew gelman stats-2013-03-06-Stan 1.2.0 and RStan 1.2.0
9 0.71356308 1749 andrew gelman stats-2013-03-04-Stan in L.A. this Wed 3:30pm
10 0.69956207 1710 andrew gelman stats-2013-02-06-The new Stan 1.1.1, featuring Gaussian processes!
11 0.69909167 2020 andrew gelman stats-2013-09-12-Samplers for Big Science: emcee and BAT
12 0.68865532 2332 andrew gelman stats-2014-05-12-“The results (not shown) . . .”
13 0.68790388 1772 andrew gelman stats-2013-03-20-Stan at Google this Thurs and at Berkeley this Fri noon
14 0.68632984 1627 andrew gelman stats-2012-12-17-Stan and RStan 1.1.0
15 0.67997944 535 andrew gelman stats-2011-01-24-Bleg: Automatic Differentiation for Log Prob Gradients?
16 0.67452306 1991 andrew gelman stats-2013-08-21-BDA3 table of contents (also a new paper on visualization)
17 0.66648066 1799 andrew gelman stats-2013-04-12-Stan 1.3.0 and RStan 1.3.0 Ready for Action
18 0.65389544 1855 andrew gelman stats-2013-05-13-Stan!
19 0.65021831 2349 andrew gelman stats-2014-05-26-WAIC and cross-validation in Stan!
20 0.64750749 1580 andrew gelman stats-2012-11-16-Stantastic!
topicId topicWeight
[(5, 0.015), (16, 0.014), (23, 0.026), (24, 0.3), (46, 0.019), (59, 0.037), (67, 0.02), (73, 0.023), (77, 0.012), (82, 0.077), (84, 0.031), (86, 0.06), (89, 0.023), (92, 0.032), (95, 0.014), (99, 0.201)]
simIndex simValue blogId blogTitle
same-blog 1 0.98604119 2231 andrew gelman stats-2014-03-03-Running into a Stan Reference by Accident
Introduction: We were talking about parallelizing MCMC and I came up with what I thought was a neat idea for parallelizing MCMC (sample with fractional prior, average samples on a per-draw basis). But then I realized this approach could get the right posterior mean or right posterior variance, but not both, depending on how the prior was divided (for a beta-binomial example). Then Aki told me it had already been done in a more general form in a paper of Scott et al., Bayes and Big Data , which was then used as the baseline in: Willie Neiswanger, Chong Wang, and Eric Xing. 2013. Asymptotically Exact, Embarrassingly Parallel MCMC . arXiv 1311.4780. It’s a neat paper, which Xi’an already blogged about months ago. But what really struck me was the following quote: We use Stan, an automated Hamiltonian Monte Carlo (HMC) software package, to perform sampling for both the true posterior (for groundtruth and comparison methods) and for the subposteriors on each machine. One advantage o
2 0.95368373 1978 andrew gelman stats-2013-08-12-Fixing the race, ethnicity, and national origin questions on the U.S. Census
Introduction: In his new book, “What is Your Race? The Census and Our Flawed Efforts to Classify Americans,” former Census Bureau director Ken Prewitt recommends taking the race question off the decennial census: He recommends gradual changes, integrating the race and national origin questions while improving both. In particular, he would replace the main “race” question by a “race or origin” question, with the instruction to “Mark one or more” of the following boxes: “White,” “Black, African Am., or Negro,” “Hispanic, Latino, or Spanish origin,” “American Indian or Alaska Native,” “Asian”, “Native Hawaiian or Other Pacific Islander,” and “Some other race or origin.” Then the next question is to write in “specific race, origin, or enrolled or principal tribe.” Prewitt writes: His suggestion is to go with these questions in 2020 and 2030, then in 2040 “drop the race question and use only the national origin question.” He’s also relying on the American Community Survey to gather a lo
3 0.94977844 482 andrew gelman stats-2010-12-23-Capitalism as a form of voluntarism
Introduction: Interesting discussion by Alex Tabarrok (following up on an article by Rebecca Solnit) on the continuum between voluntarism (or, more generally, non-cash transactions) and markets with monetary exchange. I just have a few comments of my own: 1. Solnit writes of “the iceberg economy,” which she characterizes as “based on gift economies, barter, mutual aid, and giving without hope of return . . . the relations between friends, between family members, the activities of volunteers or those who have chosen their vocation on principle rather than for profit.” I just wonder whether “barter” completely fits in here. Maybe it depends on context. Sometimes barter is an informal way of keeping track (you help me and I help you), but in settings of low liquidity I could imagine barter being simply an inefficient way of performing an economic transaction. 2. I am no expert on capitalism but my impression is that it’s not just about “competition and selfishness” but also is related to the
4 0.94796336 2229 andrew gelman stats-2014-02-28-God-leaf-tree
Introduction: Govind Manian writes: I wanted to pass along a fragment from Lichtenberg’s Waste Books — which I am finding to be great stone soup — that reminded me of God is in Every Leaf : To the wise man nothing is great and nothing small…I believe he could write treatises on keyholes that sounded as weighty as a jus naturae and would be just as instructive. As the few adepts in such things well know, universal morality is to be found in little everyday penny-events just as much as in great ones. There is so much goodness and ingenuity in a raindrop that an apothecary wouldn’t let it go for less than half-a-crown… (Notebook B, 33)
5 0.94510579 2017 andrew gelman stats-2013-09-11-“Informative g-Priors for Logistic Regression”
Introduction: Tim Hanson sends along this paper (coauthored with Adam Branscum and Wesley Johnson): Eliciting information from experts for use in constructing prior distributions for logistic regression coefficients can be challenging. The task is especially difficult when the model contains many predictor variables, because the expert is asked to provide summary information about the probability of “success” for many subgroups of the population. Often, however, experts are confident only in their assessment of the population as a whole. This paper is about incorporating such overall, marginal or averaged, information easily into a logistic regression data analysis by using g-priors. We present a version of the g-prior such that the prior distribution on the probability of success can be set to closely match a beta dis- tribution, when averaged over the set of predictors in a logistic regression. A simple data augmentation formulation that can be implemented in standard statistical software pac
6 0.94317269 1092 andrew gelman stats-2011-12-29-More by Berger and me on weakly informative priors
7 0.94272304 938 andrew gelman stats-2011-10-03-Comparing prediction errors
8 0.94246459 1368 andrew gelman stats-2012-06-06-Question 27 of my final exam for Design and Analysis of Sample Surveys
11 0.94143844 241 andrew gelman stats-2010-08-29-Ethics and statistics in development research
12 0.94092906 743 andrew gelman stats-2011-06-03-An argument that can’t possibly make sense
13 0.9403981 1479 andrew gelman stats-2012-09-01-Mothers and Moms
14 0.94034332 38 andrew gelman stats-2010-05-18-Breastfeeding, infant hyperbilirubinemia, statistical graphics, and modern medicine
15 0.93846726 1376 andrew gelman stats-2012-06-12-Simple graph WIN: the example of birthday frequencies
16 0.93717897 1455 andrew gelman stats-2012-08-12-Probabilistic screening to get an approximate self-weighted sample
17 0.93699932 1999 andrew gelman stats-2013-08-27-Bayesian model averaging or fitting a larger model
18 0.93529475 278 andrew gelman stats-2010-09-15-Advice that might make sense for individuals but is negative-sum overall
19 0.93404388 953 andrew gelman stats-2011-10-11-Steve Jobs’s cancer and science-based medicine
20 0.93260849 643 andrew gelman stats-2011-04-02-So-called Bayesian hypothesis testing is just as bad as regular hypothesis testing