andrew_gelman_stats andrew_gelman_stats-2010 andrew_gelman_stats-2010-410 knowledge-graph by maker-knowledge-mining
Source: html
Introduction: Paul Nee sends in this amusing item: MELA Sciences claimed success in a clinical trial of its experimental skin cancer detection device only by altering the statistical method used to analyze the data in violation of an agreement with U.S. regulators, charges an independent healthcare analyst in a report issued last week. . . The BER report, however, relies on its own analysis to suggest that MELA struck out with FDA because the agency’s medical device reviewers discovered the MELAFind pivotal study failed to reach statistical significance despite the company’s claims to the contrary. And now here’s where it gets interesting: MELA claims that a phase III study of MELAFind met its primary endpoint by detecting accurately 112 of 114 eligible melanomas for a “sensitivity” rate of 98%. The lower confidence bound of the sensitivity analysis was 95.1%, which met the FDA’s standard for statistical significance in the study spelled out in a binding agreement with MELA, the compa
sentIndex sentText sentNum sentScore
1 Paul Nee sends in this amusing item: MELA Sciences claimed success in a clinical trial of its experimental skin cancer detection device only by altering the statistical method used to analyze the data in violation of an agreement with U. [sent-1, score-0.671]
2 regulators, charges an independent healthcare analyst in a report issued last week. [sent-3, score-0.242]
3 The BER report, however, relies on its own analysis to suggest that MELA struck out with FDA because the agency’s medical device reviewers discovered the MELAFind pivotal study failed to reach statistical significance despite the company’s claims to the contrary. [sent-6, score-0.551]
4 And now here’s where it gets interesting: MELA claims that a phase III study of MELAFind met its primary endpoint by detecting accurately 112 of 114 eligible melanomas for a “sensitivity” rate of 98%. [sent-7, score-0.495]
5 The lower confidence bound of the sensitivity analysis was 95. [sent-8, score-0.565]
6 1%, which met the FDA’s standard for statistical significance in the study spelled out in a binding agreement with MELA, the company says. [sent-9, score-0.712]
7 The binding agreement between MELA and FDA covering the conduct of the MELAFind study required the company to analyze the data using a statistical method known as the “exact mid-P” test. [sent-10, score-0.809]
8 Clearing the 95% hurdle for the lower confidence bound using the mid-P test is important because that means if the phase III study were repeated, there would be a 5% chance or less that MELAFind’s sensitivity to detect melanoma would be below 95%. [sent-11, score-0.89]
9 In its report this week, the research shop BER says it was unable to reproduce MELA’s positive results when it ran the MELAFind data independently using off-the-shelf statistics software. [sent-15, score-0.392]
10 “For the protocol-specified outcome group, the LCB [lower confidence bound] using the exact mid-P method is 92. [sent-16, score-0.409]
11 BER then ran the MELAFind data analysis again using a bunch of other statistics methods. [sent-19, score-0.142]
12 What BER discovered was that MELAFind’s publicly stated statistical results matched only when the “Wald normal approximation” method was used. [sent-20, score-0.287]
wordName wordTfidf (topN-words)
[('melafind', 0.517), ('mela', 0.443), ('ber', 0.369), ('bound', 0.181), ('fda', 0.162), ('sensitivity', 0.139), ('binding', 0.134), ('confidence', 0.129), ('agreement', 0.125), ('lower', 0.116), ('exact', 0.112), ('phase', 0.102), ('company', 0.101), ('iii', 0.1), ('method', 0.098), ('device', 0.098), ('report', 0.094), ('study', 0.094), ('discovered', 0.077), ('met', 0.074), ('analyze', 0.074), ('ran', 0.072), ('using', 0.07), ('statistical', 0.065), ('eligible', 0.063), ('clearing', 0.063), ('endpoint', 0.063), ('wald', 0.063), ('regulators', 0.061), ('cry', 0.061), ('significance', 0.06), ('pivotal', 0.059), ('hurdle', 0.059), ('spelled', 0.059), ('unable', 0.055), ('shop', 0.054), ('altering', 0.054), ('detection', 0.053), ('skin', 0.053), ('violation', 0.051), ('issued', 0.05), ('claims', 0.05), ('detecting', 0.049), ('charges', 0.049), ('agency', 0.049), ('healthcare', 0.049), ('relies', 0.048), ('covering', 0.048), ('matched', 0.047), ('independently', 0.047)]
simIndex simValue blogId blogTitle
same-blog 1 1.0000001 410 andrew gelman stats-2010-11-12-The Wald method has been the subject of extensive criticism by statisticians for exaggerating results”
Introduction: Paul Nee sends in this amusing item: MELA Sciences claimed success in a clinical trial of its experimental skin cancer detection device only by altering the statistical method used to analyze the data in violation of an agreement with U.S. regulators, charges an independent healthcare analyst in a report issued last week. . . The BER report, however, relies on its own analysis to suggest that MELA struck out with FDA because the agency’s medical device reviewers discovered the MELAFind pivotal study failed to reach statistical significance despite the company’s claims to the contrary. And now here’s where it gets interesting: MELA claims that a phase III study of MELAFind met its primary endpoint by detecting accurately 112 of 114 eligible melanomas for a “sensitivity” rate of 98%. The lower confidence bound of the sensitivity analysis was 95.1%, which met the FDA’s standard for statistical significance in the study spelled out in a binding agreement with MELA, the compa
2 0.14117794 322 andrew gelman stats-2010-10-06-More on the differences between drugs and medical devices
Introduction: Someone who works in statistics in the pharmaceutical industry (but prefers to remain anonymous) sent me this update to our discussion on the differences between approvals of drugs and medical devices: The ‘substantial equivalence’ threshold is a very outdated. Basically the FDA has to follow federal law and the law is antiquated and leads to two extraordinarily different paths for device approval. You could have a very simple but first-in-kind device with an easy to understand physiological mechanism of action (e.g. the FDA approved a simple tiny stent that would relieve pressure from a glaucoma patient’s eye this summer). This device would require a standard (likely controlled) trial at the one-sided 0.025 level. Even after the trial it would likely go to a panel where outside experts (e.g.practicing & academic MDs and statisticians) hear evidence from the company and FDA and vote on its safety and efficacy. FDA would then rule, consider the panel’s vote, on whether to appro
3 0.0986128 2121 andrew gelman stats-2013-12-02-Should personal genetic testing be regulated? Battle of the blogroll
Introduction: On the side of less regulation is Alex Tabarrok in “Our DNA, Our Selves”: At the same time that the NSA is secretly and illegally obtaining information about Americans the FDA is making it illegal for Americans to obtain information about themselves. In a warning letter the FDA has told Anne Wojcicki, The Most Daring CEO In America, that she “must immediately discontinue” selling 23andMe’s Personal Genome Service . . . Alex clarifies: I am not offended by all regulation of genetic tests. Indeed, genetic tests are already regulated. . . . the Clinical Laboratory Improvement Amendments (CLIA) . . . requires all labs, including the labs used by 23andMe, to be inspected for quality control, record keeping and the qualifications of their personnel. . . . What the FDA wants to do is categorically different. The FDA wants to regulate genetic tests as a high-risk medical device . . . the FDA wants to judge . . . the clinical validity, whether particular identified alleles are cau
4 0.07729423 314 andrew gelman stats-2010-10-03-Disconnect between drug and medical device approval
Introduction: Sanjay Kaul wrotes: By statute (“the least burdensome” pathway), the approval standard for devices by the US FDA is lower than for drugs. Before a new drug can be marketed, the sponsor must show “substantial evidence of effectiveness” as based on two or more well-controlled clinical studies (which literally means 2 trials, each with a p value of <0.05, or 1 large trial with a robust p value <0.00125). In contrast, the sponsor of a new device, especially those that are designated as high-risk (Class III) device, need only demonstrate "substantial equivalence" to an FDA-approved device via the 510(k) exemption or a "reasonable assurance of safety and effectiveness", evaluated through a pre-market approval and typically based on a single study. What does “reasonable assurance” or “substantial equivalence” imply to you as a Bayesian? These are obviously qualitative constructs, but if one were to quantify them, how would you go about addressing it? The regulatory definitions for
5 0.076673567 804 andrew gelman stats-2011-07-15-Static sensitivity analysis
Introduction: This is one of my favorite ideas. I used it in an application but have never formally studied it or written it up as a general method. Sensitivity analysis is when you check how inferences change when you vary fit several different models or when you vary inputs within a model. Sensitivity analysis is often recommended but is typically difficult to do, what with the hassle of carrying around all these different estimates. In Bayesian inference, sensitivity analysis is associated with varying the prior distribution, which irritates me: why not consider sensitivity to the likelihood, as that’s typically just as arbitrary as the prior while having a much larger effect on the inferences. So we came up with static sensitivity analysis , which is a way to assess sensitivity to assumptions while fitting only one model. The idea is that Bayesian posterior simulation gives you a range of parameter values, and from these you can learn about sensitivity directly. The published exampl
6 0.072201647 480 andrew gelman stats-2010-12-21-Instead of “confidence interval,” let’s say “uncertainty interval”
9 0.064053878 1527 andrew gelman stats-2012-10-10-Another reason why you can get good inferences from a bad model
10 0.057029642 1214 andrew gelman stats-2012-03-15-Of forecasts and graph theory and characterizing a statistical method by the information it uses
11 0.055832177 411 andrew gelman stats-2010-11-13-Ethical concerns in medical trials
12 0.054051675 2220 andrew gelman stats-2014-02-22-Quickies
13 0.053887527 1149 andrew gelman stats-2012-02-01-Philosophy of Bayesian statistics: my reactions to Cox and Mayo
14 0.053596336 695 andrew gelman stats-2011-05-04-Statistics ethics question
15 0.053008214 1766 andrew gelman stats-2013-03-16-“Nightshifts Linked to Increased Risk for Ovarian Cancer”
16 0.049964529 908 andrew gelman stats-2011-09-14-Type M errors in the lab
17 0.049769476 2223 andrew gelman stats-2014-02-24-“Edlin’s rule” for routinely scaling down published estimates
19 0.048984796 1317 andrew gelman stats-2012-05-13-Question 3 of my final exam for Design and Analysis of Sample Surveys
20 0.047447875 1878 andrew gelman stats-2013-05-31-How to fix the tabloids? Toward replicable social science research
topicId topicWeight
[(0, 0.082), (1, 0.011), (2, 0.007), (3, -0.065), (4, 0.002), (5, -0.011), (6, -0.021), (7, 0.012), (8, -0.017), (9, -0.027), (10, -0.033), (11, -0.018), (12, 0.022), (13, -0.019), (14, -0.019), (15, 0.01), (16, 0.01), (17, -0.018), (18, 0.028), (19, -0.034), (20, 0.026), (21, 0.022), (22, 0.015), (23, 0.004), (24, 0.016), (25, 0.002), (26, -0.034), (27, -0.034), (28, 0.008), (29, 0.016), (30, -0.015), (31, -0.015), (32, 0.028), (33, 0.012), (34, 0.031), (35, 0.046), (36, -0.004), (37, 0.032), (38, 0.008), (39, 0.012), (40, 0.014), (41, -0.025), (42, 0.026), (43, 0.024), (44, 0.027), (45, -0.014), (46, -0.031), (47, 0.025), (48, -0.01), (49, -0.012)]
simIndex simValue blogId blogTitle
same-blog 1 0.94006443 410 andrew gelman stats-2010-11-12-The Wald method has been the subject of extensive criticism by statisticians for exaggerating results”
Introduction: Paul Nee sends in this amusing item: MELA Sciences claimed success in a clinical trial of its experimental skin cancer detection device only by altering the statistical method used to analyze the data in violation of an agreement with U.S. regulators, charges an independent healthcare analyst in a report issued last week. . . The BER report, however, relies on its own analysis to suggest that MELA struck out with FDA because the agency’s medical device reviewers discovered the MELAFind pivotal study failed to reach statistical significance despite the company’s claims to the contrary. And now here’s where it gets interesting: MELA claims that a phase III study of MELAFind met its primary endpoint by detecting accurately 112 of 114 eligible melanomas for a “sensitivity” rate of 98%. The lower confidence bound of the sensitivity analysis was 95.1%, which met the FDA’s standard for statistical significance in the study spelled out in a binding agreement with MELA, the compa
Introduction: Commenter Rahul asked what I thought of this note by Scott Firestone ( link from Tyler Cowen) criticizing a recent discussion by Kevin Drum suggesting that lead exposure causes violent crime. Firestone writes: It turns out there was in fact a prospective study done—but its implications for Drum’s argument are mixed. The study was a cohort study done by researchers at the University of Cincinnati. Between 1979 and 1984, 376 infants were recruited. Their parents consented to have lead levels in their blood tested over time; this was matched with records over subsequent decades of the individuals’ arrest records, and specifically arrest for violent crime. Ultimately, some of these individuals were dropped from the study; by the end, 250 were selected for the results. The researchers found that for each increase of 5 micrograms of lead per deciliter of blood, there was a higher risk for being arrested for a violent crime, but a further look at the numbers shows a more mixe
3 0.73335427 1766 andrew gelman stats-2013-03-16-“Nightshifts Linked to Increased Risk for Ovarian Cancer”
Introduction: Zosia Chustecka writes : Much of the previous work on the link between cancer and nightshifts has focused on breast cancer . . . The latest report, focusing on ovarian cancer, was published in the April issue of Occupational and Environmental Medicine. This increase in the risk for ovarian cancer with nightshift work is consistent with, and of similar magnitude to, the risk for breast cancer, say lead author Parveen Bhatti, PhD, and colleagues from the epidemiology program at the Fred Hutchinson Cancer Research Center in Seattle, Washington. The researchers examined data from a local population-based cancer registry that is part of the Surveillance Epidemiology and End Results (SEER) Program. They identified 1101 women with advanced epithelial ovarian cancer, 389 with borderline disease, and 1832 without ovarian cancer (control group). The women, who were 35 to 74 years of age, were asked about the hours they worked, and specifically whether they had ever worked the nig
4 0.69721133 2248 andrew gelman stats-2014-03-15-Problematic interpretations of confidence intervals
Introduction: Rink Hoekstra writes: A couple of months ago, you were visiting the University of Groningen, and after the talk you gave there I spoke briefly with you about a study that I conducted with Richard Morey, Jeff Rouder and Eric-Jan Wagenmakers. In the study, we found that researchers’ knowledge of how to interpret a confidence interval (CI), was almost as limited as the knowledge of students who had had no inferential statistics course yet. Our manuscript was recently accepted for publication in Psychonomic Bulletin & Review , and it’s now available online (see e.g., here ). Maybe it’s interesting to discuss on your blog, especially since CIs are often promoted (for example in the new guidelines of Psychological Science ), but apparently researchers seem to have little idea how to interpret them. Given that the confidence percentage of a CI tells something about the procedure rather than about the data at hand, this might be understandable, but, according to us, it’s problematic neve
5 0.67457217 1672 andrew gelman stats-2013-01-14-How do you think about the values in a confidence interval?
Introduction: Philip Jones writes: As an interested reader of your blog, I wondered if you might consider a blog entry sometime on the following question I posed on CrossValidated (StackExchange). I originally posed the question based on my uncertainty about 95% CIs: “Are all values within the 95% CI equally likely (probable), or are the values at the “tails” of the 95% CI less likely than those in the middle of the CI closer to the point estimate?” I posed this question based on discordant information I found at a couple of different web sources (I posted these sources in the body of the question). I received some interesting replies, and the replies were not unanimous, in fact there is some serious disagreement there! After seeing this disagreement, I naturally thought of you, and whether you might be able to clear this up. Please note I am not referring to credible intervals, but rather to the common medical journal reporting standard of confidence intervals. My response: First
6 0.67232227 2049 andrew gelman stats-2013-10-03-On house arrest for p-hacking
7 0.65756375 2114 andrew gelman stats-2013-11-26-“Please make fun of this claim”
8 0.65335697 908 andrew gelman stats-2011-09-14-Type M errors in the lab
9 0.65239972 1557 andrew gelman stats-2012-11-01-‘Researcher Degrees of Freedom’
10 0.64843446 314 andrew gelman stats-2010-10-03-Disconnect between drug and medical device approval
11 0.6472916 2032 andrew gelman stats-2013-09-20-“Six red flags for suspect work”
12 0.63267583 1881 andrew gelman stats-2013-06-03-Boot
13 0.63085234 2220 andrew gelman stats-2014-02-22-Quickies
14 0.63084096 1533 andrew gelman stats-2012-10-14-If x is correlated with y, then y is correlated with x
16 0.62283671 1906 andrew gelman stats-2013-06-19-“Behind a cancer-treatment firm’s rosy survival claims”
17 0.61775535 918 andrew gelman stats-2011-09-21-Avoiding boundary estimates in linear mixed models
19 0.61037135 322 andrew gelman stats-2010-10-06-More on the differences between drugs and medical devices
20 0.61022413 480 andrew gelman stats-2010-12-21-Instead of “confidence interval,” let’s say “uncertainty interval”
topicId topicWeight
[(15, 0.055), (16, 0.073), (20, 0.017), (21, 0.015), (24, 0.105), (34, 0.014), (43, 0.014), (54, 0.034), (84, 0.014), (86, 0.013), (96, 0.266), (99, 0.191)]
simIndex simValue blogId blogTitle
1 0.95384145 169 andrew gelman stats-2010-07-29-Say again?
Introduction: “Ich glaube, dass die Wahrscheinlichkeitsrechnung das richtige Werkzeug zum Lösen solcher Probleme ist”, sagt Andrew Gelman , Statistikprofessor von der Columbia-Universität in New York. Wie oft aber derart knifflige Aufgaben im realen Leben auftauchen, könne er nicht sagen. Was fast schon beruhigend klingt. OK, fine.
2 0.94230479 1306 andrew gelman stats-2012-05-07-Lists of Note and Letters of Note
Introduction: These (from Shaun Usher) are surprisingly good, especially since he appears to come up with new lists and letters pretty regularly. I suppose a lot of them get sent in from readers, but still. Here’s my favorite recent item, a letter sent to the Seattle Bureau of Prohibition in 1931: Dear Sir: My husband is in the habit of buying a quart of wiskey every other day from a Chinese bootlegger named Chin Waugh living at 317-16th near Alder street. We need this money for household expenses. Will you please have his place raided? He keeps a supply planted in the garden and a smaller quantity under the back steps for quick delivery. If you make the raid at 9:30 any morning you will be sure to get the goods and Chin also as he leaves the house at 10 o’clock and may clean up before he goes. Thanking you in advance, I remain yours truly, Mrs. Hillyer
same-blog 3 0.88382661 410 andrew gelman stats-2010-11-12-The Wald method has been the subject of extensive criticism by statisticians for exaggerating results”
Introduction: Paul Nee sends in this amusing item: MELA Sciences claimed success in a clinical trial of its experimental skin cancer detection device only by altering the statistical method used to analyze the data in violation of an agreement with U.S. regulators, charges an independent healthcare analyst in a report issued last week. . . The BER report, however, relies on its own analysis to suggest that MELA struck out with FDA because the agency’s medical device reviewers discovered the MELAFind pivotal study failed to reach statistical significance despite the company’s claims to the contrary. And now here’s where it gets interesting: MELA claims that a phase III study of MELAFind met its primary endpoint by detecting accurately 112 of 114 eligible melanomas for a “sensitivity” rate of 98%. The lower confidence bound of the sensitivity analysis was 95.1%, which met the FDA’s standard for statistical significance in the study spelled out in a binding agreement with MELA, the compa
4 0.85187078 1731 andrew gelman stats-2013-02-21-If a lottery is encouraging addictive gambling, don’t expand it!
Introduction: This story from Vivian Yee seems just horrible to me. First the background: Pronto Lotto’s real business takes place in the carpeted, hushed area where its most devoted customers watch video screens from a scattering of tall silver tables, hour after hour, day after day. The players — mostly men, about a dozen at any given time — come on their lunch breaks or after work to study the screens, which are programmed with the Quick Draw lottery game, and flash a new set of winning numbers every four minutes. They have helped make Pronto Lotto the top Quick Draw vendor in the state, selling $3.3 million worth of tickets last year, more than $1 million more than the second busiest location, a World Books shop in Penn Station. Some stay for just a few minutes. Others play for the length of a workday, repeatedly traversing the few yards between their seats and the cash register as they hand the next wager to a clerk with a dollar bill or two, and return to wait. “It’s like my job, 24
5 0.84083784 2153 andrew gelman stats-2013-12-29-“Statistics Done Wrong”
Introduction: Govind Manian points me to this online textbook by Alex Reinhart. It’s hard for me to evaluate because I am so close to the material. But on first glance it looks pretty reasonable to me.
6 0.81472135 1118 andrew gelman stats-2012-01-14-A model rejection letter
7 0.81467199 327 andrew gelman stats-2010-10-07-There are never 70 distinct parameters
8 0.78634065 1023 andrew gelman stats-2011-11-22-Going Beyond the Book: Towards Critical Reading in Statistics Teaching
9 0.7678256 319 andrew gelman stats-2010-10-04-“Who owns Congress”
10 0.74296439 1338 andrew gelman stats-2012-05-23-Advice on writing research articles
11 0.73849642 2023 andrew gelman stats-2013-09-14-On blogging
12 0.7328313 302 andrew gelman stats-2010-09-28-This is a link to a news article about a scientific paper
13 0.73247933 934 andrew gelman stats-2011-09-30-Nooooooooooooooooooo!
14 0.73090279 205 andrew gelman stats-2010-08-13-Arnold Zellner
15 0.72671533 99 andrew gelman stats-2010-06-19-Paired comparisons
16 0.725155 787 andrew gelman stats-2011-07-05-Different goals, different looks: Infovis and the Chris Rock effect
17 0.71573019 405 andrew gelman stats-2010-11-10-Estimation from an out-of-date census
18 0.70725918 1887 andrew gelman stats-2013-06-07-“Happy Money: The Science of Smarter Spending”
19 0.70388234 1405 andrew gelman stats-2012-07-04-“Titanic Thompson: The Man Who Would Bet on Everything”
20 0.69919872 2065 andrew gelman stats-2013-10-17-Cool dynamic demographic maps provide beautiful illustration of Chris Rock effect