andrew_gelman_stats andrew_gelman_stats-2013 andrew_gelman_stats-2013-1842 knowledge-graph by maker-knowledge-mining

1842 andrew gelman stats-2013-05-05-Cleaning up science


meta infos for this blog

Source: html

Introduction: David Hogg pointed me to this post by Gary Marcus, reviewing this skeptics’ all-star issue of Perspectives on Psychological Science that features replication culture heroes Jelte Wicherts, Hal Pashler, Arina Bones, E. J. Wagenmakers, Gregory Francis, Hal Pashler, John Ioannidis, and Uri Simonsohn. I agree with pretty much everything Marcus has to say. In addition to Marcus’s suggestions, which might be called cultural or psychological, I also have various statistical ideas that might help move the field forward. Most notably I think we need to go beyond uniform priors and null-hypothesis testing to a more realistic set of models for effects and variation. I’ll discuss more at some other time, but in the meantime I thought I’d share these links. P.S. Marcus updates with a glass-is-half-full take.


Summary: the most important sentenses genereted by tfidf model

sentIndex sentText sentNum sentScore

1 David Hogg pointed me to this post by Gary Marcus, reviewing this skeptics’ all-star issue of Perspectives on Psychological Science that features replication culture heroes Jelte Wicherts, Hal Pashler, Arina Bones, E. [sent-1, score-0.666]

2 I agree with pretty much everything Marcus has to say. [sent-4, score-0.118]

3 In addition to Marcus’s suggestions, which might be called cultural or psychological, I also have various statistical ideas that might help move the field forward. [sent-5, score-0.681]

4 Most notably I think we need to go beyond uniform priors and null-hypothesis testing to a more realistic set of models for effects and variation. [sent-6, score-0.662]

5 I’ll discuss more at some other time, but in the meantime I thought I’d share these links. [sent-7, score-0.253]

6 Marcus updates with a glass-is-half-full take. [sent-10, score-0.131]


similar blogs computed by tfidf model

tfidf for this blog:

wordName wordTfidf (topN-words)

[('marcus', 0.552), ('pashler', 0.316), ('hal', 0.254), ('arina', 0.175), ('psychological', 0.172), ('jelte', 0.158), ('gregory', 0.153), ('bones', 0.153), ('wicherts', 0.148), ('heroes', 0.141), ('skeptics', 0.138), ('updates', 0.131), ('francis', 0.131), ('wagenmakers', 0.131), ('ioannidis', 0.129), ('hogg', 0.125), ('uri', 0.117), ('notably', 0.115), ('perspectives', 0.114), ('realistic', 0.113), ('reviewing', 0.111), ('meantime', 0.107), ('gary', 0.104), ('uniform', 0.104), ('cultural', 0.101), ('replication', 0.1), ('culture', 0.094), ('features', 0.09), ('suggestions', 0.087), ('priors', 0.082), ('addition', 0.081), ('testing', 0.08), ('share', 0.079), ('move', 0.073), ('pointed', 0.072), ('field', 0.069), ('discuss', 0.067), ('everything', 0.066), ('beyond', 0.064), ('david', 0.064), ('called', 0.063), ('might', 0.062), ('john', 0.062), ('issue', 0.058), ('ideas', 0.058), ('help', 0.057), ('various', 0.055), ('effects', 0.054), ('agree', 0.052), ('set', 0.05)]

similar blogs list:

simIndex simValue blogId blogTitle

same-blog 1 0.99999994 1842 andrew gelman stats-2013-05-05-Cleaning up science

Introduction: David Hogg pointed me to this post by Gary Marcus, reviewing this skeptics’ all-star issue of Perspectives on Psychological Science that features replication culture heroes Jelte Wicherts, Hal Pashler, Arina Bones, E. J. Wagenmakers, Gregory Francis, Hal Pashler, John Ioannidis, and Uri Simonsohn. I agree with pretty much everything Marcus has to say. In addition to Marcus’s suggestions, which might be called cultural or psychological, I also have various statistical ideas that might help move the field forward. Most notably I think we need to go beyond uniform priors and null-hypothesis testing to a more realistic set of models for effects and variation. I’ll discuss more at some other time, but in the meantime I thought I’d share these links. P.S. Marcus updates with a glass-is-half-full take.

2 0.18561548 1844 andrew gelman stats-2013-05-06-Against optimism about social science

Introduction: Social science research has been getting pretty bad press recently, what with the Excel buccaneers who didn’t know how to handle data with different numbers of observations per country, and the psychologist who published dozens of papers based on fabricated data, and the Evilicious guy who wouldn’t let people review his data tapes, etc etc. And that’s not even considering Dr. Anil Potti. On the other hand, the revelation of all these problems can be taken as evidence that things are getting better. Psychology researcher Gary Marcus writes : There is something positive that has come out of the crisis of replicability—something vitally important for all experimental sciences. For years, it was extremely difficult to publish a direct replication, or a failure to replicate an experiment, in a good journal. . . . Now, happily, the scientific culture has changed. . . . The Reproducibility Project, from the Center for Open Science is now underway . . . And sociologist Fabio Rojas

3 0.14826185 933 andrew gelman stats-2011-09-30-More bad news: The (mis)reporting of statistical results in psychology journals

Introduction: Another entry in the growing literature on systematic flaws in the scientific research literature. This time the bad tidings come from Marjan Bakker and Jelte Wicherts, who write : Around 18% of statistical results in the psychological literature are incorrectly reported. Inconsistencies were more common in low-impact journals than in high-impact journals. Moreover, around 15% of the articles contained at least one statistical conclusion that proved, upon recalculation, to be incorrect; that is, recalculation rendered the previously significant result insignificant, or vice versa. These errors were often in line with researchers’ expectations. Their research also had a qualitative component: To obtain a better understanding of the origins of the errors made in the reporting of statistics, we contacted the authors of the articles with errors in the second study and asked them to send us the raw data. Regrettably, only 24% of the authors shared their data, despite our request

4 0.12641908 1654 andrew gelman stats-2013-01-04-“Don’t think of it as duplication. Think of it as a single paper in a superposition of two quantum journals.”

Introduction: Adam Marcus at Retraction Watch reports on a physicist at the University of Toronto who had this unfortunate thing happen to him: This article has been retracted at the request of the Editor-in-Chief and first and corresponding author. The article was largely a duplication of a paper that had already appeared in ACS Nano, 4 (2010) 3374–3380, http://dx.doi.org/10.1021/nn100335g. The first and the corresponding authors (Kramer and Sargent) would like to apologize for this administrative error on their part . . . “Administrative error” . . . I love that! Is that what the robber says when he knocks over a liquor store and gets caught? As Marcus points out, the two papers have different titles and a different order of authors, which makes it less plausible that this was an administrative mistake (as could happen, for example, if a secretary was given a list of journals to submit the paper to, and accidentally submitted it to the second journal on the list without realizing it

5 0.11878934 897 andrew gelman stats-2011-09-09-The difference between significant and not significant…

Introduction: E. J. Wagenmakers writes: You may be interested in a recent article [by Nieuwenhuis, Forstmann, and Wagenmakers] showing how often researchers draw conclusions by comparing p-values. As you and Hal Stern have pointed out, this is potentially misleading because the difference between significant and not significant is not necessarily significant. We were really suprised to see how often researchers in the neurosciences make this mistake. In the paper we speculate a little bit on the cause of the error. From their paper: In theory, a comparison of two experimental effects requires a statistical test on their difference. In practice, this comparison is often based on an incorrect procedure involving two separate tests in which researchers conclude that effects differ when one effect is significant (P < 0.05) but the other is not (P > 0.05). We reviewed 513 behavioral, systems and cognitive neuroscience articles in five top-ranking journals (Science, Nature, Nature Neuroscien

6 0.10883132 1883 andrew gelman stats-2013-06-04-Interrogating p-values

7 0.097637683 1944 andrew gelman stats-2013-07-18-You’ll get a high Type S error rate if you use classical statistical methods to analyze data from underpowered studies

8 0.091673106 991 andrew gelman stats-2011-11-04-Insecure researchers aren’t sharing their data

9 0.090197548 2278 andrew gelman stats-2014-04-01-Association for Psychological Science announces a new journal

10 0.08880803 758 andrew gelman stats-2011-06-11-Hey, good news! Your p-value just passed the 0.05 threshold!

11 0.084697403 424 andrew gelman stats-2010-11-21-Data cleaning tool!

12 0.083493426 1878 andrew gelman stats-2013-05-31-How to fix the tabloids? Toward replicable social science research

13 0.082199015 1833 andrew gelman stats-2013-04-30-“Tragedy of the science-communication commons”

14 0.079792596 268 andrew gelman stats-2010-09-10-Fighting Migraine with Multilevel Modeling

15 0.0784721 450 andrew gelman stats-2010-12-04-The Joy of Stats

16 0.078057542 2343 andrew gelman stats-2014-05-22-Big Data needs Big Model

17 0.077994429 2090 andrew gelman stats-2013-11-05-How much do we trust a new claim that early childhood stimulation raised earnings by 42%?

18 0.071948305 1072 andrew gelman stats-2011-12-19-“The difference between . . .”: It’s not just p=.05 vs. p=.06

19 0.069684699 1329 andrew gelman stats-2012-05-18-Those mean psychologists, making fun of dodgy research!

20 0.067278221 576 andrew gelman stats-2011-02-15-With a bit of precognition, you’d have known I was going to post again on this topic, and with a lot of precognition, you’d have known I was going to post today


similar blogs computed by lsi model

lsi for this blog:

topicId topicWeight

[(0, 0.09), (1, 0.007), (2, -0.028), (3, -0.033), (4, -0.025), (5, -0.032), (6, -0.017), (7, -0.022), (8, -0.033), (9, 0.023), (10, 0.002), (11, 0.029), (12, 0.026), (13, -0.019), (14, 0.014), (15, -0.022), (16, -0.034), (17, -0.006), (18, -0.009), (19, -0.015), (20, -0.019), (21, -0.027), (22, -0.037), (23, 0.012), (24, -0.036), (25, -0.021), (26, 0.038), (27, 0.011), (28, 0.016), (29, -0.03), (30, -0.014), (31, -0.028), (32, -0.023), (33, -0.013), (34, 0.019), (35, -0.003), (36, -0.049), (37, -0.002), (38, 0.012), (39, 0.022), (40, -0.015), (41, 0.041), (42, 0.001), (43, 0.02), (44, -0.019), (45, 0.004), (46, -0.049), (47, -0.002), (48, 0.003), (49, 0.006)]

similar blogs list:

simIndex simValue blogId blogTitle

same-blog 1 0.95816267 1842 andrew gelman stats-2013-05-05-Cleaning up science

Introduction: David Hogg pointed me to this post by Gary Marcus, reviewing this skeptics’ all-star issue of Perspectives on Psychological Science that features replication culture heroes Jelte Wicherts, Hal Pashler, Arina Bones, E. J. Wagenmakers, Gregory Francis, Hal Pashler, John Ioannidis, and Uri Simonsohn. I agree with pretty much everything Marcus has to say. In addition to Marcus’s suggestions, which might be called cultural or psychological, I also have various statistical ideas that might help move the field forward. Most notably I think we need to go beyond uniform priors and null-hypothesis testing to a more realistic set of models for effects and variation. I’ll discuss more at some other time, but in the meantime I thought I’d share these links. P.S. Marcus updates with a glass-is-half-full take.

2 0.66408741 1833 andrew gelman stats-2013-04-30-“Tragedy of the science-communication commons”

Introduction: I’ve earlier written that science is science communication —that is, the act of communicating scientific ideas and findings to ourselves and others is itself a central part of science. My point was to push against a conventional separation between the act of science and the act of communication, the idea that science is done by scientists and communication is done by communicators. It’s a rare bit of science that does not include communication as part of it. As a scientist and science communicator myself, I’m particularly sensitive to devaluing of communication. (For example, Bayesian Data Analysis is full of original research that was done in order to communicate; or, to put it another way, we often think we understand a scientific idea, but once we try to communicate it, we recognize gaps in our understanding that motivate further research.) I once saw the following on one of those inspirational-sayings-for-every-day desk calendars: “To have ideas is to gather flowers. To thin

3 0.66182297 2326 andrew gelman stats-2014-05-08-Discussion with Steven Pinker on research that is attached to data that are so noisy as to be essentially uninformative

Introduction: I pointed Steven Pinker to my post, How much time (if any) should we spend criticizing research that’s fraudulent, crappy, or just plain pointless? , and he responded: Clearly it *is* important to call out publicized research whose conclusions are likely to be false. The only danger is that it’s so easy and fun to criticize, with all the perks of intellectual and moral superiority for so little cost, that there is a moral hazard to go overboard and become a professional slasher and snarker. (That’s a common phenomenon among literary critics, especially in the UK.) There’s also the risk of altering the incentive structure for innovative research, so that researchers stick to the safest kinds of paradigm-twiddling. I think these two considerations were what my late colleague Dan Wegner had in mind when he made the bumbler-pointer contrast — he himself was certainly a discerning critic of social science research. [Just to clarify: Wegner is the person who talked about bumblers and po

4 0.65858412 897 andrew gelman stats-2011-09-09-The difference between significant and not significant…

Introduction: E. J. Wagenmakers writes: You may be interested in a recent article [by Nieuwenhuis, Forstmann, and Wagenmakers] showing how often researchers draw conclusions by comparing p-values. As you and Hal Stern have pointed out, this is potentially misleading because the difference between significant and not significant is not necessarily significant. We were really suprised to see how often researchers in the neurosciences make this mistake. In the paper we speculate a little bit on the cause of the error. From their paper: In theory, a comparison of two experimental effects requires a statistical test on their difference. In practice, this comparison is often based on an incorrect procedure involving two separate tests in which researchers conclude that effects differ when one effect is significant (P < 0.05) but the other is not (P > 0.05). We reviewed 513 behavioral, systems and cognitive neuroscience articles in five top-ranking journals (Science, Nature, Nature Neuroscien

5 0.65374535 1959 andrew gelman stats-2013-07-28-50 shades of gray: A research story

Introduction: This is a killer story (from Brian Nosek, Jeffrey Spies, and Matt Motyl). Part 1: Two of the present authors, Motyl and Nosek, share interests in political ideology. We were inspired by the fast growing literature on embodiment that demonstrates surprising links between body and mind (Markman & Brendl, 2005; Proffitt, 2006) to investigate embodiment of political extremism. Participants from the political left, right and center (N = 1,979) completed a perceptual judgment task in which words were presented in different shades of gray. Participants had to click along a gradient representing grays from near black to near white to select a shade that matched the shade of the word. We calculated accuracy: How close to the actual shade did participants get? The results were stunning. Moderates perceived the shades of gray more accurately than extremists on the left and right (p = .01). Our conclusion: political extremists perceive the world in black-and-white, figuratively and literally

6 0.65195101 1555 andrew gelman stats-2012-10-31-Social scientists who use medical analogies to explain causal inference are, I think, implicitly trying to borrow some of the scientific and cultural authority of that field for our own purposes

7 0.63866055 2188 andrew gelman stats-2014-01-27-“Disappointed with your results? Boost your scientific paper”

8 0.63702881 2241 andrew gelman stats-2014-03-10-Preregistration: what’s in it for you?

9 0.6280269 1878 andrew gelman stats-2013-05-31-How to fix the tabloids? Toward replicable social science research

10 0.6270982 1671 andrew gelman stats-2013-01-13-Preregistration of Studies and Mock Reports

11 0.62060744 1880 andrew gelman stats-2013-06-02-Flame bait

12 0.6190412 2093 andrew gelman stats-2013-11-07-I’m negative on the expression “false positives”

13 0.6172117 1998 andrew gelman stats-2013-08-25-A new Bem theory

14 0.61435789 1883 andrew gelman stats-2013-06-04-Interrogating p-values

15 0.61168188 2227 andrew gelman stats-2014-02-27-“What Can we Learn from the Many Labs Replication Project?”

16 0.61088014 576 andrew gelman stats-2011-02-15-With a bit of precognition, you’d have known I was going to post again on this topic, and with a lot of precognition, you’d have known I was going to post today

17 0.60430777 1974 andrew gelman stats-2013-08-08-Statistical significance and the dangerous lure of certainty

18 0.60301852 1844 andrew gelman stats-2013-05-06-Against optimism about social science

19 0.60074127 1171 andrew gelman stats-2012-02-16-“False-positive psychology”

20 0.59745848 898 andrew gelman stats-2011-09-10-Fourteen magic words: an update


similar blogs computed by lda model

lda for this blog:

topicId topicWeight

[(4, 0.017), (6, 0.022), (15, 0.056), (16, 0.053), (21, 0.076), (24, 0.097), (34, 0.109), (53, 0.019), (56, 0.044), (85, 0.021), (86, 0.016), (90, 0.138), (99, 0.213)]

similar blogs list:

simIndex simValue blogId blogTitle

same-blog 1 0.92482793 1842 andrew gelman stats-2013-05-05-Cleaning up science

Introduction: David Hogg pointed me to this post by Gary Marcus, reviewing this skeptics’ all-star issue of Perspectives on Psychological Science that features replication culture heroes Jelte Wicherts, Hal Pashler, Arina Bones, E. J. Wagenmakers, Gregory Francis, Hal Pashler, John Ioannidis, and Uri Simonsohn. I agree with pretty much everything Marcus has to say. In addition to Marcus’s suggestions, which might be called cultural or psychological, I also have various statistical ideas that might help move the field forward. Most notably I think we need to go beyond uniform priors and null-hypothesis testing to a more realistic set of models for effects and variation. I’ll discuss more at some other time, but in the meantime I thought I’d share these links. P.S. Marcus updates with a glass-is-half-full take.

2 0.87709492 1522 andrew gelman stats-2012-10-05-High temperatures cause violent crime and implications for climate change, also some suggestions about how to better summarize these claims

Introduction: Solomon Hsiang writes : I [Hsiang] have posted about high temperature inducing individuals to exhibit more violent behavior when driving, playing baseball and prowling bars. These cases are neat anecdotes that let us see the “pure aggression” response in lab-like conditions. But they don’t affect most of us too much. But violent crime in the real world affects everyone. Earlier, I posted a paper by Jacob et al. that looked at assault in the USA for about a decade – they found that higher temperatures lead to more assault and that the rise in violent crimes rose more quickly than the analogous rise in non-violent property-crime, an indicator that there is a “pure aggression” component to the rise in violent crime. A new working paper “Crime, Weather, and Climate Change” by recent Harvard grad Matthew Ranson puts together an impressive data set of all types of crime in USA counties for 50 years. The results tell the aggression story using street-level data very clearly [click to

3 0.87407416 1932 andrew gelman stats-2013-07-10-Don’t trust the Turk

Introduction: Dan Kahan gives a bunch of reasons not to trust Mechanical Turk in psychology experiments, in particular when studying “hypotheses about cognition and political conflict over societal risks and other policy-relevant facts.”

4 0.86601889 475 andrew gelman stats-2010-12-19-All politics are local — not

Introduction: Mickey Kaus does a public service by trashing Tip O’Neill’s famous dictum that “all politics are local.” As Kaus point out, all the congressional elections in recent decades have been nationalized. I’d go one step further and say that, sure, all politics are local–if you’re Tip O’Neill and represent a ironclad Democratic seat in Congress. It’s easy to be smug about your political skills if you’re in a safe seat and have enough pull in state politics to avoid your district getting gerrymandered. Then you can sit there and sagely attribute your success to your continuing mastery of local politics rather than to whatever it took to get the seat in the first place.

5 0.86299717 1417 andrew gelman stats-2012-07-15-Some decision analysis problems are pretty easy, no?

Introduction: Cassie Murdoch reports : A 47-year-old woman in Uxbridge, Massachusetts, got behind the wheel of her car after having a bit too much to drink, but instead of wreaking havoc on the road, she ended up lodged in a sand trap at a local golf course. Why? Because her GPS made her do it—obviously! She said the GPS told her to turn left, and she did, right into a cornfield. That didn’t faze her, and she just kept on going until she ended up on the golf course and got stuck in the sand. There were people on the course at the time, but thankfully nobody was injured. Police found a cup full of alcohol in her car and arrested her for driving drunk. Here’s the punchline: This is the fourth time she’s been arrested for a DUI. Assuming this story is accurate, I guess they don’t have one of those “three strikes” laws in Massachusetts? Personally, I’m a lot more afraid of a dangerous driver than of some drug dealer. I’d think a simple cost-benefit calculation would recommend taking away

6 0.86231232 1911 andrew gelman stats-2013-06-23-AI Stats conference on Stan etc.

7 0.85749811 267 andrew gelman stats-2010-09-09-This Friday afternoon: Applied Statistics Center mini-conference on risk perception

8 0.85442674 15 andrew gelman stats-2010-05-03-Public Opinion on Health Care Reform

9 0.85383219 762 andrew gelman stats-2011-06-13-How should journals handle replication studies?

10 0.85166043 1411 andrew gelman stats-2012-07-10-Defining ourselves arbitrarily

11 0.84940398 1734 andrew gelman stats-2013-02-23-Life in the C-suite: A graph that is both ugly and bad, and an unrelated story

12 0.84198022 1947 andrew gelman stats-2013-07-20-We are what we are studying

13 0.84044302 292 andrew gelman stats-2010-09-23-Doug Hibbs on the fundamentals in 2010

14 0.84020311 1655 andrew gelman stats-2013-01-05-The statistics software signal

15 0.83777535 1501 andrew gelman stats-2012-09-18-More studies on the economic effects of climate change

16 0.83644819 956 andrew gelman stats-2011-10-13-Hey, you! Don’t take that class!

17 0.83221233 1111 andrew gelman stats-2012-01-10-The blog of the Cultural Cognition Project

18 0.82803679 1500 andrew gelman stats-2012-09-17-“2% per degree Celsius . . . the magic number for how worker productivity responds to warm-hot temperatures”

19 0.82618439 766 andrew gelman stats-2011-06-14-Last Wegman post (for now)

20 0.82599556 1144 andrew gelman stats-2012-01-29-How many parameters are in a multilevel model?