andrew_gelman_stats andrew_gelman_stats-2013 andrew_gelman_stats-2013-1706 knowledge-graph by maker-knowledge-mining
Source: html
Introduction: Justin Kinney writes: Since your blog has discussed the “maximal information coefficient” (MIC) of Reshef et al., I figured you might want to see the critique that Gurinder Atwal and I have posted. In short, Reshef et al.’s central claim that MIC is “equitable” is incorrect. We [Kinney and Atwal] offer mathematical proof that the definition of “equitability” Reshef et al. propose is unsatisfiable—no nontrivial dependence measure, including MIC, has this property. Replicating the simulations in their paper with modestly larger data sets validates this finding. The heuristic notion of equitability, however, can be formalized instead as a self-consistency condition closely related to the Data Processing Inequality. Mutual information satisfies this new definition of equitability but MIC does not. We therefore propose that simply estimating mutual information will, in many cases, provide the sort of dependence measure Reshef et al. seek. For background, here are my two p
sentIndex sentText sentNum sentScore
1 Justin Kinney writes: Since your blog has discussed the “maximal information coefficient” (MIC) of Reshef et al. [sent-1, score-0.371]
2 We [Kinney and Atwal] offer mathematical proof that the definition of “equitability” Reshef et al. [sent-5, score-0.303]
3 propose is unsatisfiable—no nontrivial dependence measure, including MIC, has this property. [sent-6, score-0.259]
4 Replicating the simulations in their paper with modestly larger data sets validates this finding. [sent-7, score-0.384]
5 The heuristic notion of equitability, however, can be formalized instead as a self-consistency condition closely related to the Data Processing Inequality. [sent-8, score-0.132]
6 Mutual information satisfies this new definition of equitability but MIC does not. [sent-9, score-0.443]
7 We therefore propose that simply estimating mutual information will, in many cases, provide the sort of dependence measure Reshef et al. [sent-10, score-1.034]
8 For background, here are my two posts ( Dec 2011 and Mar 2012 ) on this method for detecting novel associations in large data sets. [sent-12, score-0.215]
9 I never read the paper in detail but on quick skim it looked really cool to me. [sent-13, score-0.166]
10 As I saw it, the clever idea of the paper is that, instead of going for an absolute measure (which, as we’ve seen, will be scale-dependent), they focus on the problem of summarizing the grid of pairwise dependences in a large set of variables. [sent-14, score-0.694]
11 provide a relative rather than absolute measure of association, suitable for comparing pairs of variables within a single dataset even if the interpretation is not so clear between datasets. [sent-16, score-0.37]
12 What is the value of their association measure if applied to data that are on a circle? [sent-18, score-0.365]
13 For example, suppose you generate these 1000 points in R: n <- 1000 theta <- runif (n, 0, 2*pi) x <- cos (theta) y <- sin (theta) Simulated in this way, x and y have an R-squared of 0. [sent-19, score-0.276]
14 But, from the description of the method in the paper, it seems that their R-squared-like measure might be very close to 1. [sent-21, score-0.351]
15 No measure can be all things to all datasets, so let me emphasize that the above is not a criticism of the idea of Reshef et al. [sent-25, score-0.432]
16 ) A more general approach would be for their grid boxes to be adaptive. [sent-30, score-0.162]
17 In any case, all these methods (including the method discussed in the paper by Simon and Tibshirani) seem like a step forward from what we typically use in statistics. [sent-34, score-0.228]
18 Near the bottom of page 11 they suggest that inference about joint distributions (in their case, with the goal of estimating mutual information) is not a real concern now that we are in such a large-data world. [sent-40, score-0.242]
19 But, as we get more data, we also gain the ability and inclination to subdivide our data into smaller pieces. [sent-41, score-0.134]
20 For example, sure, “consumer research companies routinely analyze data sets containing information on ∼ 10^5 shoppers,” but it would be helpful to break up the data and learn about different people, times, and locations, rather than computing aggregate measures of association. [sent-42, score-0.383]
wordName wordTfidf (topN-words)
[('reshef', 0.467), ('atwal', 0.267), ('kinney', 0.267), ('mic', 0.25), ('et', 0.219), ('measure', 0.213), ('equitability', 0.2), ('mutual', 0.175), ('theta', 0.135), ('grid', 0.104), ('dependence', 0.103), ('propose', 0.1), ('information', 0.097), ('absolute', 0.097), ('paper', 0.092), ('definition', 0.084), ('method', 0.081), ('data', 0.08), ('skim', 0.074), ('sin', 0.074), ('shoppers', 0.074), ('sets', 0.072), ('association', 0.072), ('validates', 0.07), ('modestly', 0.07), ('equitable', 0.07), ('formalized', 0.07), ('estimating', 0.067), ('runif', 0.067), ('dependences', 0.067), ('central', 0.066), ('pairwise', 0.064), ('heuristic', 0.062), ('satisfies', 0.062), ('dec', 0.061), ('provide', 0.06), ('pi', 0.059), ('boxes', 0.058), ('replicating', 0.057), ('maximal', 0.057), ('close', 0.057), ('going', 0.057), ('nontrivial', 0.056), ('locations', 0.055), ('bivariate', 0.055), ('tibshirani', 0.055), ('discussed', 0.055), ('detecting', 0.054), ('containing', 0.054), ('inclination', 0.054)]
simIndex simValue blogId blogTitle
same-blog 1 0.99999994 1706 andrew gelman stats-2013-02-04-Too many MC’s not enough MIC’s, or What principles should govern attempts to summarize bivariate associations in large multivariate datasets?
Introduction: Justin Kinney writes: Since your blog has discussed the “maximal information coefficient” (MIC) of Reshef et al., I figured you might want to see the critique that Gurinder Atwal and I have posted. In short, Reshef et al.’s central claim that MIC is “equitable” is incorrect. We [Kinney and Atwal] offer mathematical proof that the definition of “equitability” Reshef et al. propose is unsatisfiable—no nontrivial dependence measure, including MIC, has this property. Replicating the simulations in their paper with modestly larger data sets validates this finding. The heuristic notion of equitability, however, can be formalized instead as a self-consistency condition closely related to the Data Processing Inequality. Mutual information satisfies this new definition of equitability but MIC does not. We therefore propose that simply estimating mutual information will, in many cases, provide the sort of dependence measure Reshef et al. seek. For background, here are my two p
2 0.5818783 2247 andrew gelman stats-2014-03-14-The maximal information coefficient
Introduction: Justin Kinney writes: I wanted to let you know that the critique Mickey Atwal and I wrote regarding equitability and the maximal information coefficient has just been published . We discussed this paper last year, under the heading, Too many MC’s not enough MIC’s, or What principles should govern attempts to summarize bivariate associations in large multivariate datasets? Kinney and Atwal’s paper is interesting, with my only criticism being that in some places they seem to aim for what might not be possible. For example, they write that “mutual information is already widely believed to quantify dependencies without bias for relationships of one type or another,” which seems a bit vague to me. And later they write, “How to compute such an estimate that does not bias the resulting mutual information value remains an open problem,” which seems to me to miss the point in that unbiased statistical estimates are not generally possible and indeed are often not desirable. Their
3 0.55675226 1062 andrew gelman stats-2011-12-16-Mr. Pearson, meet Mr. Mandelbrot: Detecting Novel Associations in Large Data Sets
Introduction: Jeremy Fox asks what I think about this paper by David N. Reshef, Yakir Reshef, Hilary Finucane, Sharon Grossman, Gilean McVean, Peter Turnbaugh, Eric Lander, Michael Mitzenmacher, and Pardis Sabeti which proposes a new nonlinear R-squared-like measure. My quick answer is that it looks really cool! From my quick reading of the paper, it appears that the method reduces on average to the usual R-squared when fit to data of the form y = a + bx + error, and that it also has a similar interpretation when “a + bx” is replaced by other continuous functions. Unlike R-squared, the method of Reshef et al. depends on a tuning parameter that controls the level of discretization, in a “How long is the coast of Britain” sort of way. The dependence on scale is inevitable for such a general method. Just consider: if you sample 1000 points from the unit bivariate normal distribution, (x,y) ~ N(0,I), you’ll be able to fit them perfectly by a 999-degree polynomial fit to the data. So the sca
4 0.47141621 1230 andrew gelman stats-2012-03-26-Further thoughts on nonparametric correlation measures
Introduction: Malka Gorfine, Ruth Heller, and Yair Heller write a comment on the paper of Reshef et al. that we discussed a few months ago. Just to remind you what’s going on here, here’s my quick summary from December: Reshef et al. propose a new nonlinear R-squared-like measure. Unlike R-squared, this new method depends on a tuning parameter that controls the level of discretization, in a “How long is the coast of Britain” sort of way. The dependence on scale is inevitable for such a general method. Just consider: if you sample 1000 points from the unit bivariate normal distribution, (x,y) ~ N(0,I), you’ll be able to fit them perfectly by a 999-degree polynomial fit to the data. So the scale of the fit matters. The clever idea of the paper is that, instead of going for an absolute measure (which, as we’ve seen, will be scale-dependent), they focus on the problem of summarizing the grid of pairwise dependences in a large set of variables. As they put it: “Imagine a data set with hundreds
5 0.36298341 2314 andrew gelman stats-2014-05-01-Heller, Heller, and Gorfine on univariate and multivariate information measures
Introduction: Malka Gorfine writes: We noticed that the important topic of association measures and tests came up again in your blog, and we have few comments in this regard. It is useful to distinguish between the univariate and multivariate methods. A consistent multivariate method can recognise dependence between two vectors of random variables, while a univariate method can only loop over pairs of components and check for dependency between them. There are very few consistent multivariate methods. To the best of our knowledge there are three practical methods: 1) HSIC by Gretton et al. (http://www.gatsby.ucl.ac.uk/~gretton/papers/GreBouSmoSch05.pdf) 2) dcov by Szekely et al. (http://projecteuclid.org/euclid.aoas/1267453933) 3) A method we introduced in Heller et al (Biometrika, 2013, 503—510, http://biomet.oxfordjournals.org/content/early/2012/12/04/biomet.ass070.full.pdf+html, and an R package, HHG, is available as well http://cran.r-project.org/web/packages/HHG/index.html). A
6 0.30710456 2324 andrew gelman stats-2014-05-07-Once more on nonparametric measures of mutual information
7 0.13997418 2315 andrew gelman stats-2014-05-02-Discovering general multidimensional associations
8 0.11141114 1941 andrew gelman stats-2013-07-16-Priors
10 0.099334918 1089 andrew gelman stats-2011-12-28-Path sampling for models of varying dimension
11 0.095650241 1695 andrew gelman stats-2013-01-28-Economists argue about Bayes
12 0.093856208 547 andrew gelman stats-2011-01-31-Using sample size in the prior distribution
13 0.087247223 899 andrew gelman stats-2011-09-10-The statistical significance filter
14 0.08667203 961 andrew gelman stats-2011-10-16-The “Washington read” and the algebra of conditional distributions
15 0.083952948 2004 andrew gelman stats-2013-09-01-Post-publication peer review: How it (sometimes) really works
16 0.082702637 1868 andrew gelman stats-2013-05-23-Validation of Software for Bayesian Models Using Posterior Quantiles
17 0.080352217 1247 andrew gelman stats-2012-04-05-More philosophy of Bayes
18 0.079407789 1996 andrew gelman stats-2013-08-24-All inference is about generalizing from sample to population
20 0.077561371 1981 andrew gelman stats-2013-08-14-The robust beauty of improper linear models in decision making
topicId topicWeight
[(0, 0.182), (1, 0.046), (2, 0.012), (3, -0.05), (4, 0.039), (5, -0.062), (6, -0.009), (7, -0.011), (8, -0.033), (9, 0.006), (10, 0.024), (11, 0.015), (12, -0.034), (13, -0.041), (14, -0.018), (15, 0.027), (16, 0.032), (17, 0.005), (18, -0.009), (19, -0.072), (20, 0.08), (21, 0.012), (22, 0.101), (23, -0.023), (24, 0.109), (25, 0.122), (26, 0.104), (27, 0.1), (28, 0.24), (29, 0.062), (30, 0.142), (31, 0.139), (32, 0.137), (33, -0.019), (34, 0.075), (35, 0.018), (36, 0.037), (37, -0.002), (38, -0.077), (39, -0.041), (40, 0.101), (41, 0.07), (42, 0.052), (43, -0.004), (44, -0.098), (45, -0.056), (46, 0.168), (47, -0.098), (48, -0.147), (49, 0.066)]
simIndex simValue blogId blogTitle
same-blog 1 0.94378382 1706 andrew gelman stats-2013-02-04-Too many MC’s not enough MIC’s, or What principles should govern attempts to summarize bivariate associations in large multivariate datasets?
Introduction: Justin Kinney writes: Since your blog has discussed the “maximal information coefficient” (MIC) of Reshef et al., I figured you might want to see the critique that Gurinder Atwal and I have posted. In short, Reshef et al.’s central claim that MIC is “equitable” is incorrect. We [Kinney and Atwal] offer mathematical proof that the definition of “equitability” Reshef et al. propose is unsatisfiable—no nontrivial dependence measure, including MIC, has this property. Replicating the simulations in their paper with modestly larger data sets validates this finding. The heuristic notion of equitability, however, can be formalized instead as a self-consistency condition closely related to the Data Processing Inequality. Mutual information satisfies this new definition of equitability but MIC does not. We therefore propose that simply estimating mutual information will, in many cases, provide the sort of dependence measure Reshef et al. seek. For background, here are my two p
2 0.93052727 2247 andrew gelman stats-2014-03-14-The maximal information coefficient
Introduction: Justin Kinney writes: I wanted to let you know that the critique Mickey Atwal and I wrote regarding equitability and the maximal information coefficient has just been published . We discussed this paper last year, under the heading, Too many MC’s not enough MIC’s, or What principles should govern attempts to summarize bivariate associations in large multivariate datasets? Kinney and Atwal’s paper is interesting, with my only criticism being that in some places they seem to aim for what might not be possible. For example, they write that “mutual information is already widely believed to quantify dependencies without bias for relationships of one type or another,” which seems a bit vague to me. And later they write, “How to compute such an estimate that does not bias the resulting mutual information value remains an open problem,” which seems to me to miss the point in that unbiased statistical estimates are not generally possible and indeed are often not desirable. Their
3 0.93032038 1230 andrew gelman stats-2012-03-26-Further thoughts on nonparametric correlation measures
Introduction: Malka Gorfine, Ruth Heller, and Yair Heller write a comment on the paper of Reshef et al. that we discussed a few months ago. Just to remind you what’s going on here, here’s my quick summary from December: Reshef et al. propose a new nonlinear R-squared-like measure. Unlike R-squared, this new method depends on a tuning parameter that controls the level of discretization, in a “How long is the coast of Britain” sort of way. The dependence on scale is inevitable for such a general method. Just consider: if you sample 1000 points from the unit bivariate normal distribution, (x,y) ~ N(0,I), you’ll be able to fit them perfectly by a 999-degree polynomial fit to the data. So the scale of the fit matters. The clever idea of the paper is that, instead of going for an absolute measure (which, as we’ve seen, will be scale-dependent), they focus on the problem of summarizing the grid of pairwise dependences in a large set of variables. As they put it: “Imagine a data set with hundreds
4 0.90972382 2324 andrew gelman stats-2014-05-07-Once more on nonparametric measures of mutual information
Introduction: Ben Murell writes: Our reply to Kinney and Atwal has come out (http://www.pnas.org/content/early/2014/04/29/1403623111.full.pdf) along with their response (http://www.pnas.org/content/early/2014/04/29/1404661111.full.pdf). I feel like they somewhat missed the point. If you’re still interested in this line of discussion, feel free to post, and maybe the Murrells and Kinney can bash it out in your comments! Background: Too many MC’s not enough MIC’s, or What principles should govern attempts to summarize bivariate associations in large multivariate datasets? Heller, Heller, and Gorfine on univariate and multivariate information measures Kinney and Atwal on the maximal information coefficient Mr. Pearson, meet Mr. Mandelbrot: Detecting Novel Associations in Large Data Sets Gorfine, Heller, Heller, Simon, and Tibshirani don’t like MIC The fun thing is that all these people are sending me their papers, and I’m enough of an outsider in this field that each of the
5 0.90246409 1062 andrew gelman stats-2011-12-16-Mr. Pearson, meet Mr. Mandelbrot: Detecting Novel Associations in Large Data Sets
Introduction: Jeremy Fox asks what I think about this paper by David N. Reshef, Yakir Reshef, Hilary Finucane, Sharon Grossman, Gilean McVean, Peter Turnbaugh, Eric Lander, Michael Mitzenmacher, and Pardis Sabeti which proposes a new nonlinear R-squared-like measure. My quick answer is that it looks really cool! From my quick reading of the paper, it appears that the method reduces on average to the usual R-squared when fit to data of the form y = a + bx + error, and that it also has a similar interpretation when “a + bx” is replaced by other continuous functions. Unlike R-squared, the method of Reshef et al. depends on a tuning parameter that controls the level of discretization, in a “How long is the coast of Britain” sort of way. The dependence on scale is inevitable for such a general method. Just consider: if you sample 1000 points from the unit bivariate normal distribution, (x,y) ~ N(0,I), you’ll be able to fit them perfectly by a 999-degree polynomial fit to the data. So the sca
6 0.90199012 2314 andrew gelman stats-2014-05-01-Heller, Heller, and Gorfine on univariate and multivariate information measures
7 0.67219919 1825 andrew gelman stats-2013-04-25-It’s binless! A program for computing normalizing functions
8 0.61632216 2315 andrew gelman stats-2014-05-02-Discovering general multidimensional associations
10 0.52956998 519 andrew gelman stats-2011-01-16-Update on the generalized method of moments
11 0.52341622 1424 andrew gelman stats-2012-07-22-Extreme events as evidence for differences in distributions
12 0.50476313 1346 andrew gelman stats-2012-05-27-Average predictive comparisons when changing a pair of variables
13 0.49842852 1828 andrew gelman stats-2013-04-27-Time-Sharing Experiments for the Social Sciences
14 0.4944126 301 andrew gelman stats-2010-09-28-Correlation, prediction, variation, etc.
15 0.48480237 1089 andrew gelman stats-2011-12-28-Path sampling for models of varying dimension
16 0.48250657 2311 andrew gelman stats-2014-04-29-Bayesian Uncertainty Quantification for Differential Equations!
17 0.46923029 778 andrew gelman stats-2011-06-24-New ideas on DIC from Martyn Plummer and Sumio Watanabe
18 0.46912757 939 andrew gelman stats-2011-10-03-DBQQ rounding for labeling charts and communicating tolerances
19 0.4566465 589 andrew gelman stats-2011-02-24-On summarizing a noisy scatterplot with a single comparison of two points
20 0.4551641 1975 andrew gelman stats-2013-08-09-Understanding predictive information criteria for Bayesian models
topicId topicWeight
[(15, 0.031), (16, 0.044), (17, 0.058), (21, 0.022), (24, 0.479), (41, 0.023), (64, 0.012), (68, 0.011), (73, 0.011), (96, 0.01), (99, 0.175)]
simIndex simValue blogId blogTitle
1 0.98537958 240 andrew gelman stats-2010-08-29-ARM solutions
Introduction: People sometimes email asking if a solution set is available for the exercises in ARM. The answer, unfortunately, is no. Many years ago, I wrote up 50 solutions for BDA and it was a lot of work–really, it was like writing a small book in itself. The trouble is that, once I started writing them up, I wanted to do it right, to set a good example. That’s a lot more effort than simply scrawling down some quick answers.
2 0.98510814 643 andrew gelman stats-2011-04-02-So-called Bayesian hypothesis testing is just as bad as regular hypothesis testing
Introduction: Steve Ziliak points me to this article by the always-excellent Carl Bialik, slamming hypothesis tests. I only wish Carl had talked with me before so hastily posting, though! I would’ve argued with some of the things in the article. In particular, he writes: Reese and Brad Carlin . . . suggest that Bayesian statistics are a better alternative, because they tackle the probability that the hypothesis is true head-on, and incorporate prior knowledge about the variables involved. Brad Carlin does great work in theory, methods, and applications, and I like the bit about the prior knowledge (although I might prefer the more general phrase “additional information”), but I hate that quote! My quick response is that the hypothesis of zero effect is almost never true! The problem with the significance testing framework–Bayesian or otherwise–is in the obsession with the possibility of an exact zero effect. The real concern is not with zero, it’s with claiming a positive effect whe
Introduction: Jouni Kerman did a cool bit of research justifying the Beta (1/3, 1/3) prior as noninformative for binomial data, and the Gamma (1/3, 0) prior for Poisson data. You probably thought that nothing new could be said about noninformative priors in such basic problems, but you were wrong! Here’s the story : The conjugate binomial and Poisson models are commonly used for estimating proportions or rates. However, it is not well known that the conventional noninformative conjugate priors tend to shrink the posterior quantiles toward the boundary or toward the middle of the parameter space, making them thus appear excessively informative. The shrinkage is always largest when the number of observed events is small. This behavior persists for all sample sizes and exposures. The effect of the prior is therefore most conspicuous and potentially controversial when analyzing rare events. As alternative default conjugate priors, I [Jouni] introduce Beta(1/3, 1/3) and Gamma(1/3, 0), which I cal
4 0.98231268 545 andrew gelman stats-2011-01-30-New innovations in spam
Introduction: I received the following (unsolicited) email today: Hello Andrew, I’m interested in whether you are accepting guest article submissions for your site Statistical Modeling, Causal Inference, and Social Science? I’m the owner of the recently created nonprofit site OnlineEngineeringDegree.org and am interested in writing / submitting an article for your consideration to be published on your site. Is that something you’d be willing to consider, and if so, what specs in terms of topics or length requirements would you be looking for? Thanks you for your time, and if you have any questions or are interested, I’d appreciate you letting me know. Sincerely, Samantha Rhodes Huh? P.S. My vote for most obnoxious spam remains this one , which does its best to dilute whatever remains of the reputation of Wolfram Research. Or maybe that particular bit of spam was written by a particularly awesome cellular automaton that Wolfram discovered? I guess in the world of big-time software
5 0.9809376 1437 andrew gelman stats-2012-07-31-Paying survey respondents
Introduction: I agree with Casey Mulligan that participants in government surveys should be paid, and I think it should be part of the code of ethics for commercial pollsters to compensate their respondents also. As Mulligan points out, if a survey is worth doing, it should be worth compensating the participants for their time and effort. P.S. Just to clarify, I do not recommend that Census surveys be made voluntary, I just think that respondents (who can be required to participate) should be paid a small amount. P.P.S. More rant here .
6 0.97817719 471 andrew gelman stats-2010-12-17-Attractive models (and data) wanted for statistical art show.
7 0.97461271 38 andrew gelman stats-2010-05-18-Breastfeeding, infant hyperbilirubinemia, statistical graphics, and modern medicine
8 0.97042412 241 andrew gelman stats-2010-08-29-Ethics and statistics in development research
9 0.96810818 938 andrew gelman stats-2011-10-03-Comparing prediction errors
10 0.96732116 1092 andrew gelman stats-2011-12-29-More by Berger and me on weakly informative priors
11 0.96721816 1978 andrew gelman stats-2013-08-12-Fixing the race, ethnicity, and national origin questions on the U.S. Census
12 0.96631694 1479 andrew gelman stats-2012-09-01-Mothers and Moms
same-blog 13 0.96311784 1706 andrew gelman stats-2013-02-04-Too many MC’s not enough MIC’s, or What principles should govern attempts to summarize bivariate associations in large multivariate datasets?
14 0.9605698 482 andrew gelman stats-2010-12-23-Capitalism as a form of voluntarism
15 0.96008867 1787 andrew gelman stats-2013-04-04-Wanna be the next Tyler Cowen? It’s not as easy as you might think!
16 0.95904255 2229 andrew gelman stats-2014-02-28-God-leaf-tree
17 0.94877839 743 andrew gelman stats-2011-06-03-An argument that can’t possibly make sense
19 0.94600976 59 andrew gelman stats-2010-05-30-Extended Binary Format Support for Mac OS X
20 0.93874109 373 andrew gelman stats-2010-10-27-It’s better than being forwarded the latest works of you-know-who