hunch_net hunch_net-2008 hunch_net-2008-314 knowledge-graph by maker-knowledge-mining
Source: html
Introduction: This post is about a technology which could develop in the future. Right now, a new drug might be tested by finding patients with some diagnosis and giving or not giving them a drug according to a secret randomization. The outcome is observed, and if the average outcome for those treated is measurably better than the average outcome for those not treated, the drug might become a standard treatment. Generalizing this, a filter F sorts people into two groups: those for treatment A and those not for treatment B based upon observations x . To measure the outcome, you randomize between treatment and nontreatment of group A and measure the relative performance of the treatment. A problem often arises: in many cases the treated group does not do better than the nontreated group. A basic question is: does this mean the treatment is bad? With respect to the filter F it may mean that, but with respect to another filter F’ , the treatment might be very effective. For exampl
sentIndex sentText sentNum sentScore
1 Right now, a new drug might be tested by finding patients with some diagnosis and giving or not giving them a drug according to a secret randomization. [sent-2, score-1.367]
2 The outcome is observed, and if the average outcome for those treated is measurably better than the average outcome for those not treated, the drug might become a standard treatment. [sent-3, score-1.395]
3 Generalizing this, a filter F sorts people into two groups: those for treatment A and those not for treatment B based upon observations x . [sent-4, score-1.201]
4 To measure the outcome, you randomize between treatment and nontreatment of group A and measure the relative performance of the treatment. [sent-5, score-0.843]
5 A problem often arises: in many cases the treated group does not do better than the nontreated group. [sent-6, score-0.297]
6 A basic question is: does this mean the treatment is bad? [sent-7, score-0.531]
7 With respect to the filter F it may mean that, but with respect to another filter F’ , the treatment might be very effective. [sent-8, score-1.067]
8 For example, a drug might work great for people which have one blood type, but not so well for others. [sent-9, score-0.407]
9 The basic import is that we can learn a rule F’ for filters which are more strict than the original F . [sent-12, score-0.178]
10 This can be done on past recorded data , and if done properly we can even statistically prove that F’ works, without another randomized trial. [sent-13, score-0.261]
11 Right now, the filters F are typically a diagnosis of one sort or another. [sent-16, score-0.427]
12 Instead, a doctor might record many observations, and have many learned filters F’ applied to suggest treatments. [sent-18, score-0.489]
13 The “not understanding the details” problem is sometimes severe, so we can expect a renewed push for understandable machine learning rules. [sent-19, score-0.28]
14 Some tradeoff between understandability and predictive power seems to exist creating a tension: do you want a good treatment or do you want an understandable treatment? [sent-20, score-0.698]
15 If we manage to reach a pointer in the future where Gattaca style near instantaneous genomic sequencing is available, feeding this into a learning algorithm is potentially very effective. [sent-22, score-0.141]
16 In general a constant pressure to measure more should be expected. [sent-23, score-0.293]
17 Given that we can learn from past data, going back and measuring additional characteristics of past patients may even be desirable. [sent-24, score-0.56]
18 Since many treatments are commercial in the US, there will be a great deal of pressure to find a filter F’ which appears good, and a company investing millions into the question is quite capable of overfitting so that F’ is better than it appears. [sent-25, score-0.825]
19 Safe and sane ways to deal with this exist, as showcased by various machine learning challenges, such as the Netflix challenge . [sent-26, score-0.126]
20 To gain trust in such approaches, a trustable and trusted third party capable of this sort of testing must exist. [sent-27, score-0.358]
wordName wordTfidf (topN-words)
[('treatment', 0.448), ('drug', 0.307), ('filter', 0.218), ('treated', 0.213), ('outcome', 0.207), ('filters', 0.178), ('diagnosis', 0.173), ('patients', 0.173), ('pressure', 0.173), ('understandable', 0.142), ('past', 0.123), ('measure', 0.12), ('exist', 0.108), ('might', 0.1), ('observations', 0.087), ('technology', 0.086), ('capable', 0.086), ('group', 0.084), ('mean', 0.083), ('giving', 0.083), ('average', 0.077), ('measuring', 0.077), ('feeding', 0.077), ('statistically', 0.077), ('commercial', 0.077), ('doctor', 0.077), ('generalizing', 0.077), ('renewed', 0.077), ('sort', 0.076), ('secret', 0.071), ('trusted', 0.071), ('randomize', 0.071), ('treatments', 0.071), ('investing', 0.071), ('finding', 0.07), ('learned', 0.069), ('trial', 0.067), ('tension', 0.067), ('fed', 0.067), ('millions', 0.067), ('safe', 0.067), ('applied', 0.065), ('pointer', 0.064), ('sane', 0.064), ('characteristics', 0.064), ('party', 0.064), ('deal', 0.062), ('push', 0.061), ('trust', 0.061), ('recorded', 0.061)]
simIndex simValue blogId blogTitle
same-blog 1 0.99999982 314 hunch net-2008-08-24-Mass Customized Medicine in the Future?
Introduction: This post is about a technology which could develop in the future. Right now, a new drug might be tested by finding patients with some diagnosis and giving or not giving them a drug according to a secret randomization. The outcome is observed, and if the average outcome for those treated is measurably better than the average outcome for those not treated, the drug might become a standard treatment. Generalizing this, a filter F sorts people into two groups: those for treatment A and those not for treatment B based upon observations x . To measure the outcome, you randomize between treatment and nontreatment of group A and measure the relative performance of the treatment. A problem often arises: in many cases the treated group does not do better than the nontreated group. A basic question is: does this mean the treatment is bad? With respect to the filter F it may mean that, but with respect to another filter F’ , the treatment might be very effective. For exampl
2 0.2090722 411 hunch net-2010-09-21-Regretting the dead
Introduction: Nikos pointed out this new york times article about poor clinical design killing people . For those of us who study learning from exploration information this is a reminder that low regret algorithms are particularly important, as regret in clinical trials is measured by patient deaths. Two obvious improvements on the experimental design are: With reasonable record keeping of existing outcomes for the standard treatments, there is no need to explicitly assign people to a control group with the standard treatment, as that approach is effectively explored with great certainty. Asserting otherwise would imply that the nature of effective treatments for cancer has changed between now and a year ago, which denies the value of any clinical trial. An optimal experimental design will smoothly phase between exploration and exploitation as evidence for a new treatment shows that it can be effective. This is old tech, for example in the EXP3.P algorithm (page 12 aka 59) although
3 0.1263624 115 hunch net-2005-09-26-Prediction Bounds as the Mathematics of Science
Introduction: “Science” has many meanings, but one common meaning is “the scientific method ” which is a principled method for investigating the world using the following steps: Form a hypothesis about the world. Use the hypothesis to make predictions. Run experiments to confirm or disprove the predictions. The ordering of these steps is very important to the scientific method. In particular, predictions must be made before experiments are run. Given that we all believe in the scientific method of investigation, it may be surprising to learn that cheating is very common. This happens for many reasons, some innocent and some not. Drug studies. Pharmaceutical companies make predictions about the effects of their drugs and then conduct blind clinical studies to determine their effect. Unfortunately, they have also been caught using some of the more advanced techniques for cheating here : including “reprobleming”, “data set selection”, and probably “overfitting by review”
4 0.11222901 217 hunch net-2006-11-06-Data Linkage Problems
Introduction: Data linkage is a problem which seems to come up in various applied machine learning problems. I have heard it mentioned in various data mining contexts, but it seems relatively less studied for systemic reasons. A very simple version of the data linkage problem is a cross hospital patient record merge. Suppose a patient (John Doe) is admitted to a hospital (General Health), treated, and released. Later, John Doe is admitted to a second hospital (Health General), treated, and released. Given a large number of records of this sort, it becomes very tempting to try and predict the outcomes of treatments. This is reasonably straightforward as a machine learning problem if there is a shared unique identifier for John Doe used by General Health and Health General along with time stamps. We can merge the records and create examples of the form “Given symptoms and treatment, did the patient come back to a hospital within the next year?” These examples could be fed into a learning algo
5 0.099050313 19 hunch net-2005-02-14-Clever Methods of Overfitting
Introduction: “Overfitting” is traditionally defined as training some flexible representation so that it memorizes the data but fails to predict well in the future. For this post, I will define overfitting more generally as over-representing the performance of systems. There are two styles of general overfitting: overrepresenting performance on particular datasets and (implicitly) overrepresenting performance of a method on future datasets. We should all be aware of these methods, avoid them where possible, and take them into account otherwise. I have used “reproblem” and “old datasets”, and may have participated in “overfitting by review”—some of these are very difficult to avoid. Name Method Explanation Remedy Traditional overfitting Train a complex predictor on too-few examples. Hold out pristine examples for testing. Use a simpler predictor. Get more training examples. Integrate over many predictors. Reject papers which do this. Parameter twe
6 0.096390948 125 hunch net-2005-10-20-Machine Learning in the News
7 0.089048773 56 hunch net-2005-04-14-Families of Learning Theory Statements
8 0.085286126 359 hunch net-2009-06-03-Functionally defined Nonlinear Dynamic Models
9 0.084789805 388 hunch net-2010-01-24-Specializations of the Master Problem
10 0.083724096 282 hunch net-2008-01-06-Research Political Issues
11 0.080214255 335 hunch net-2009-01-08-Predictive Analytics World
12 0.078334637 332 hunch net-2008-12-23-Use of Learning Theory
13 0.077438802 237 hunch net-2007-04-02-Contextual Scaling
14 0.075915836 454 hunch net-2012-01-30-ICML Posters and Scope
15 0.075135827 371 hunch net-2009-09-21-Netflix finishes (and starts)
16 0.073216848 260 hunch net-2007-08-25-The Privacy Problem
17 0.072593488 235 hunch net-2007-03-03-All Models of Learning have Flaws
18 0.071121693 362 hunch net-2009-06-26-Netflix nearly done
19 0.070906058 132 hunch net-2005-11-26-The Design of an Optimal Research Environment
20 0.06944827 347 hunch net-2009-03-26-Machine Learning is too easy
topicId topicWeight
[(0, 0.186), (1, 0.044), (2, -0.039), (3, 0.049), (4, -0.01), (5, -0.03), (6, 0.021), (7, 0.058), (8, 0.008), (9, -0.027), (10, -0.051), (11, 0.069), (12, 0.066), (13, -0.011), (14, -0.032), (15, -0.016), (16, 0.011), (17, -0.018), (18, -0.013), (19, 0.023), (20, -0.023), (21, -0.019), (22, -0.047), (23, 0.019), (24, -0.082), (25, -0.02), (26, 0.023), (27, 0.005), (28, 0.019), (29, -0.003), (30, -0.084), (31, 0.011), (32, 0.013), (33, -0.021), (34, -0.048), (35, -0.001), (36, -0.046), (37, 0.02), (38, -0.079), (39, -0.023), (40, 0.097), (41, 0.018), (42, 0.016), (43, 0.008), (44, -0.074), (45, -0.016), (46, 0.015), (47, -0.075), (48, 0.025), (49, -0.085)]
simIndex simValue blogId blogTitle
same-blog 1 0.92555517 314 hunch net-2008-08-24-Mass Customized Medicine in the Future?
Introduction: This post is about a technology which could develop in the future. Right now, a new drug might be tested by finding patients with some diagnosis and giving or not giving them a drug according to a secret randomization. The outcome is observed, and if the average outcome for those treated is measurably better than the average outcome for those not treated, the drug might become a standard treatment. Generalizing this, a filter F sorts people into two groups: those for treatment A and those not for treatment B based upon observations x . To measure the outcome, you randomize between treatment and nontreatment of group A and measure the relative performance of the treatment. A problem often arises: in many cases the treated group does not do better than the nontreated group. A basic question is: does this mean the treatment is bad? With respect to the filter F it may mean that, but with respect to another filter F’ , the treatment might be very effective. For exampl
2 0.73370832 411 hunch net-2010-09-21-Regretting the dead
Introduction: Nikos pointed out this new york times article about poor clinical design killing people . For those of us who study learning from exploration information this is a reminder that low regret algorithms are particularly important, as regret in clinical trials is measured by patient deaths. Two obvious improvements on the experimental design are: With reasonable record keeping of existing outcomes for the standard treatments, there is no need to explicitly assign people to a control group with the standard treatment, as that approach is effectively explored with great certainty. Asserting otherwise would imply that the nature of effective treatments for cancer has changed between now and a year ago, which denies the value of any clinical trial. An optimal experimental design will smoothly phase between exploration and exploitation as evidence for a new treatment shows that it can be effective. This is old tech, for example in the EXP3.P algorithm (page 12 aka 59) although
3 0.69429177 217 hunch net-2006-11-06-Data Linkage Problems
Introduction: Data linkage is a problem which seems to come up in various applied machine learning problems. I have heard it mentioned in various data mining contexts, but it seems relatively less studied for systemic reasons. A very simple version of the data linkage problem is a cross hospital patient record merge. Suppose a patient (John Doe) is admitted to a hospital (General Health), treated, and released. Later, John Doe is admitted to a second hospital (Health General), treated, and released. Given a large number of records of this sort, it becomes very tempting to try and predict the outcomes of treatments. This is reasonably straightforward as a machine learning problem if there is a shared unique identifier for John Doe used by General Health and Health General along with time stamps. We can merge the records and create examples of the form “Given symptoms and treatment, did the patient come back to a hospital within the next year?” These examples could be fed into a learning algo
4 0.61792409 348 hunch net-2009-04-02-Asymmophobia
Introduction: One striking feature of many machine learning algorithms is the gymnastics that designers go through to avoid symmetry breaking. In the most basic form of machine learning, there are labeled examples composed of features. Each of these can be treated symmetrically or asymmetrically by algorithms. feature symmetry Every feature is treated the same. In gradient update rules, the same update is applied whether the feature is first or last. In metric-based predictions, every feature is just as important in computing the distance. example symmetry Every example is treated the same. Batch learning algorithms are great exemplars of this approach. label symmetry Every label is treated the same. This is particularly noticeable in multiclass classification systems which predict according to arg max l w l x but it occurs in many other places as well. Empirically, breaking symmetry well seems to yield great algorithms. feature asymmetry For those who like t
5 0.61387914 115 hunch net-2005-09-26-Prediction Bounds as the Mathematics of Science
Introduction: “Science” has many meanings, but one common meaning is “the scientific method ” which is a principled method for investigating the world using the following steps: Form a hypothesis about the world. Use the hypothesis to make predictions. Run experiments to confirm or disprove the predictions. The ordering of these steps is very important to the scientific method. In particular, predictions must be made before experiments are run. Given that we all believe in the scientific method of investigation, it may be surprising to learn that cheating is very common. This happens for many reasons, some innocent and some not. Drug studies. Pharmaceutical companies make predictions about the effects of their drugs and then conduct blind clinical studies to determine their effect. Unfortunately, they have also been caught using some of the more advanced techniques for cheating here : including “reprobleming”, “data set selection”, and probably “overfitting by review”
6 0.60403013 262 hunch net-2007-09-16-Optimizing Machine Learning Programs
7 0.5976665 282 hunch net-2008-01-06-Research Political Issues
8 0.58660793 106 hunch net-2005-09-04-Science in the Government
9 0.58467299 373 hunch net-2009-10-03-Static vs. Dynamic multiclass prediction
10 0.57686251 125 hunch net-2005-10-20-Machine Learning in the News
11 0.57479429 171 hunch net-2006-04-09-Progress in Machine Translation
12 0.57115108 260 hunch net-2007-08-25-The Privacy Problem
13 0.57003057 153 hunch net-2006-02-02-Introspectionism as a Disease
14 0.56782204 120 hunch net-2005-10-10-Predictive Search is Coming
15 0.56031555 366 hunch net-2009-08-03-Carbon in Computer Science Research
16 0.5512929 272 hunch net-2007-11-14-BellKor wins Netflix
17 0.54892069 237 hunch net-2007-04-02-Contextual Scaling
18 0.54192036 55 hunch net-2005-04-10-Is the Goal Understanding or Prediction?
19 0.53924501 193 hunch net-2006-07-09-The Stock Prediction Machine Learning Problem
20 0.53720784 205 hunch net-2006-09-07-Objective and subjective interpretations of probability
topicId topicWeight
[(1, 0.356), (16, 0.016), (27, 0.195), (53, 0.069), (55, 0.081), (84, 0.039), (94, 0.111), (95, 0.04)]
simIndex simValue blogId blogTitle
1 0.93865573 205 hunch net-2006-09-07-Objective and subjective interpretations of probability
Introduction: An amusing tidbit (reproduced without permission) from Herman Chernoff’s delightful monograph, “Sequential analysis and optimal design”: The use of randomization raises a philosophical question which is articulated by the following probably apocryphal anecdote. The metallurgist told his friend the statistician how he planned to test the effect of heat on the strength of a metal bar by sawing the bar into six pieces. The first two would go into the hot oven, the next two into the medium oven, and the last two into the cool oven. The statistician, horrified, explained how he should randomize to avoid the effect of a possible gradient of strength in the metal bar. The method of randomization was applied, and it turned out that the randomized experiment called for putting the first two pieces into the hot oven, the next two into the medium oven, and the last two into the cool oven. “Obviously, we can’t do that,” said the metallurgist. “On the contrary, you have to do that,” said the st
2 0.86197728 39 hunch net-2005-03-10-Breaking Abstractions
Introduction: Sam Roweis ‘s comment reminds me of a more general issue that comes up in doing research: abstractions always break. Real number’s aren’t. Most real numbers can not be represented with any machine. One implication of this is that many real-number based algorithms have difficulties when implemented with floating point numbers. The box on your desk is not a turing machine. A turing machine can compute anything computable, given sufficient time. A typical computer fails terribly when the state required for the computation exceeds some limit. Nash equilibria aren’t equilibria. This comes up when trying to predict human behavior based on the result of the equilibria computation. Often, it doesn’t work. The probability isn’t. Probability is an abstraction expressing either our lack of knowledge (the Bayesian viewpoint) or fundamental randomization (the frequentist viewpoint). From the frequentist viewpoint the lack of knowledge typically precludes actually knowing the fu
3 0.84569371 76 hunch net-2005-05-29-Bad ideas
Introduction: I found these two essays on bad ideas interesting. Neither of these is written from the viewpoint of research, but they are both highly relevant. Why smart people have bad ideas by Paul Graham Why smart people defend bad ideas by Scott Berkun (which appeared on slashdot ) In my experience, bad ideas are common and over confidence in ideas is common. This overconfidence can take either the form of excessive condemnation or excessive praise. Some of this is necessary to the process of research. For example, some overconfidence in the value of your own research is expected and probably necessary to motivate your own investigation. Since research is a rather risky business, much of it does not pan out. Learning to accept when something does not pan out is a critical skill which is sometimes never acquired. Excessive condemnation can be a real ill when it’s encountered. This has two effects: When the penalty for being wrong is too large, it means people have a
same-blog 4 0.82611489 314 hunch net-2008-08-24-Mass Customized Medicine in the Future?
Introduction: This post is about a technology which could develop in the future. Right now, a new drug might be tested by finding patients with some diagnosis and giving or not giving them a drug according to a secret randomization. The outcome is observed, and if the average outcome for those treated is measurably better than the average outcome for those not treated, the drug might become a standard treatment. Generalizing this, a filter F sorts people into two groups: those for treatment A and those not for treatment B based upon observations x . To measure the outcome, you randomize between treatment and nontreatment of group A and measure the relative performance of the treatment. A problem often arises: in many cases the treated group does not do better than the nontreated group. A basic question is: does this mean the treatment is bad? With respect to the filter F it may mean that, but with respect to another filter F’ , the treatment might be very effective. For exampl
5 0.81278729 486 hunch net-2013-07-10-Thoughts on Artificial Intelligence
Introduction: David McAllester starts a blog .
6 0.66345423 8 hunch net-2005-02-01-NIPS: Online Bayes
7 0.59474182 140 hunch net-2005-12-14-More NIPS Papers II
8 0.58228225 5 hunch net-2005-01-26-Watchword: Probability
9 0.56706303 286 hunch net-2008-01-25-Turing’s Club for Machine Learning
10 0.56247962 337 hunch net-2009-01-21-Nearly all natural problems require nonlinearity
11 0.56181151 196 hunch net-2006-07-13-Regression vs. Classification as a Primitive
12 0.56003636 132 hunch net-2005-11-26-The Design of an Optimal Research Environment
13 0.55938631 40 hunch net-2005-03-13-Avoiding Bad Reviewing
14 0.55895364 435 hunch net-2011-05-16-Research Directions for Machine Learning and Algorithms
15 0.55528593 95 hunch net-2005-07-14-What Learning Theory might do
16 0.5540725 371 hunch net-2009-09-21-Netflix finishes (and starts)
17 0.55173683 96 hunch net-2005-07-21-Six Months
18 0.55145186 347 hunch net-2009-03-26-Machine Learning is too easy
19 0.55127722 359 hunch net-2009-06-03-Functionally defined Nonlinear Dynamic Models
20 0.55063093 351 hunch net-2009-05-02-Wielding a New Abstraction