hunch_net hunch_net-2011 hunch_net-2011-430 knowledge-graph by maker-knowledge-mining
Source: html
Introduction: The Heritage Health Prize is potentially the largest prediction prize yet at $3M, which is sure to get many people interested. Several elements of the competition may be worth discussing. The most straightforward way for HPN to deploy this predictor is in determining who to cover with insurance. This might easily cover the costs of running the contest itself, but the value to the health system of a whole is minimal, as people not covered still exist. While HPN itself is a provider network, they have active relationships with a number of insurance companies, and the right to resell any entrant. It’s worth keeping in mind that the research and development may nevertheless end up being useful in the longer term, especially as entrants also keep the right to their code. The judging metric is something I haven’t seen previously. If a patient has probability 0.5 of being in the hospital 0 days and probability 0.5 of being in the hospital ~53.6 days, the optimal prediction in e
sentIndex sentText sentNum sentScore
1 The Heritage Health Prize is potentially the largest prediction prize yet at $3M, which is sure to get many people interested. [sent-1, score-0.586]
2 Several elements of the competition may be worth discussing. [sent-2, score-0.111]
3 The most straightforward way for HPN to deploy this predictor is in determining who to cover with insurance. [sent-3, score-0.389]
4 This might easily cover the costs of running the contest itself, but the value to the health system of a whole is minimal, as people not covered still exist. [sent-4, score-0.898]
5 While HPN itself is a provider network, they have active relationships with a number of insurance companies, and the right to resell any entrant. [sent-5, score-0.291]
6 It’s worth keeping in mind that the research and development may nevertheless end up being useful in the longer term, especially as entrants also keep the right to their code. [sent-6, score-0.419]
7 The judging metric is something I haven’t seen previously. [sent-7, score-0.088]
8 5 of being in the hospital 0 days and probability 0. [sent-9, score-0.52]
9 This is evidence against point (1) above, since cost is probably closer to linear in the number of hospital days. [sent-13, score-0.506]
10 As a starting point, I suspect many people will simply optimize conditional squared loss and then back out an inferred prediction according to p=e x -1 , with clipping. [sent-14, score-0.389]
11 I’m not sure there is a good reason for it from a prediction point of view and 8 may be too hard a limit on team size, imposing bin packing problems on the entrants. [sent-17, score-0.621]
12 They anonymized the data, require entrants to protect the data, and admonish people to not try to break privacy. [sent-19, score-0.59]
13 Despite that, the data will be released to large numbers of people, so I wouldn’t be surprised if someone attempts a join attack of some sort. [sent-20, score-0.534]
14 Whether or not a join attack succeeds could make a huge difference in how this contest is viewed in the long term. [sent-21, score-0.801]
15 If they set it at an out-of-reach point (which they could easily do), the size of the prize becomes 0. [sent-23, score-0.517]
16 This part of the contest is supposed to be determined next month. [sent-25, score-0.471]
17 This contest is not a slam-dunk, but is has the potential to become one, and I’ll be interested to see how it turns out. [sent-26, score-0.291]
wordName wordTfidf (topN-words)
[('hospital', 0.293), ('contest', 0.291), ('entrants', 0.237), ('hpn', 0.237), ('prize', 0.209), ('team', 0.176), ('join', 0.163), ('attack', 0.149), ('health', 0.149), ('days', 0.136), ('prediction', 0.13), ('point', 0.125), ('cover', 0.123), ('worth', 0.111), ('huge', 0.106), ('relationships', 0.105), ('deploy', 0.105), ('size', 0.101), ('anonymized', 0.098), ('provider', 0.098), ('inferred', 0.098), ('bin', 0.098), ('heritage', 0.098), ('whole', 0.098), ('patient', 0.098), ('protect', 0.098), ('sure', 0.092), ('determined', 0.092), ('succeeds', 0.092), ('probability', 0.091), ('determining', 0.088), ('judging', 0.088), ('supposed', 0.088), ('closer', 0.088), ('insurance', 0.088), ('threshold', 0.084), ('easily', 0.082), ('people', 0.082), ('wouldn', 0.081), ('strange', 0.081), ('data', 0.081), ('suspect', 0.079), ('minimal', 0.075), ('break', 0.075), ('costs', 0.073), ('largest', 0.073), ('straightforward', 0.073), ('released', 0.071), ('keeping', 0.071), ('surprised', 0.07)]
simIndex simValue blogId blogTitle
same-blog 1 1.0 430 hunch net-2011-04-11-The Heritage Health Prize
Introduction: The Heritage Health Prize is potentially the largest prediction prize yet at $3M, which is sure to get many people interested. Several elements of the competition may be worth discussing. The most straightforward way for HPN to deploy this predictor is in determining who to cover with insurance. This might easily cover the costs of running the contest itself, but the value to the health system of a whole is minimal, as people not covered still exist. While HPN itself is a provider network, they have active relationships with a number of insurance companies, and the right to resell any entrant. It’s worth keeping in mind that the research and development may nevertheless end up being useful in the longer term, especially as entrants also keep the right to their code. The judging metric is something I haven’t seen previously. If a patient has probability 0.5 of being in the hospital 0 days and probability 0.5 of being in the hospital ~53.6 days, the optimal prediction in e
2 0.26074764 371 hunch net-2009-09-21-Netflix finishes (and starts)
Introduction: I attended the Netflix prize ceremony this morning. The press conference part is covered fine elsewhere , with the basic outcome being that BellKor’s Pragmatic Chaos won over The Ensemble by 15-20 minutes , because they were tied in performance on the ultimate holdout set. I’m sure the individual participants will have many chances to speak about the solution. One of these is Bell at the NYAS ML symposium on Nov. 6 . Several additional details may interest ML people. The degree of overfitting exhibited by the difference in performance on the leaderboard test set and the ultimate hold out set was small, but determining at .02 to .03%. A tie was possible, because the rules cut off measurements below the fourth digit based on significance concerns. In actuality, of course, the scores do differ before rounding, but everyone I spoke to claimed not to know how. The complete dataset has been released on UCI , so each team could compute their own score to whatever accu
3 0.20146739 211 hunch net-2006-10-02-$1M Netflix prediction contest
Introduction: Netflix is running a contest to improve recommender prediction systems. A 10% improvement over their current system yields a $1M prize. Failing that, the best smaller improvement yields a smaller $50K prize. This contest looks quite real, and the $50K prize money is almost certainly achievable with a bit of thought. The contest also comes with a dataset which is apparently 2 orders of magnitude larger than any other public recommendation system datasets.
4 0.18895254 217 hunch net-2006-11-06-Data Linkage Problems
Introduction: Data linkage is a problem which seems to come up in various applied machine learning problems. I have heard it mentioned in various data mining contexts, but it seems relatively less studied for systemic reasons. A very simple version of the data linkage problem is a cross hospital patient record merge. Suppose a patient (John Doe) is admitted to a hospital (General Health), treated, and released. Later, John Doe is admitted to a second hospital (Health General), treated, and released. Given a large number of records of this sort, it becomes very tempting to try and predict the outcomes of treatments. This is reasonably straightforward as a machine learning problem if there is a shared unique identifier for John Doe used by General Health and Health General along with time stamps. We can merge the records and create examples of the form “Given symptoms and treatment, did the patient come back to a hospital within the next year?” These examples could be fed into a learning algo
5 0.14518499 362 hunch net-2009-06-26-Netflix nearly done
Introduction: A $1M qualifying result was achieved on the public Netflix test set by a 3-way ensemble team . This is just in time for Yehuda ‘s presentation at KDD , which I’m sure will be one of the best attended ever. This isn’t quite over—there are a few days for another super-conglomerate team to come together and there is some small chance that the performance is nonrepresentative of the final test set, but I expect not. Regardless of the final outcome, the biggest lesson for ML from the Netflix contest has been the formidable performance edge of ensemble methods.
6 0.13937815 129 hunch net-2005-11-07-Prediction Competitions
7 0.13018453 390 hunch net-2010-03-12-Netflix Challenge 2 Canceled
8 0.1187503 369 hunch net-2009-08-27-New York Area Machine Learning Events
9 0.11011219 336 hunch net-2009-01-19-Netflix prize within epsilon
10 0.10255666 259 hunch net-2007-08-19-Choice of Metrics
11 0.10213138 406 hunch net-2010-08-22-KDD 2010
12 0.099091344 9 hunch net-2005-02-01-Watchword: Loss
13 0.095846951 400 hunch net-2010-06-13-The Good News on Exploration and Learning
14 0.094315149 270 hunch net-2007-11-02-The Machine Learning Award goes to …
15 0.092935473 426 hunch net-2011-03-19-The Ideal Large Scale Learning Class
16 0.092492864 341 hunch net-2009-02-04-Optimal Proxy Loss for Classification
17 0.089430764 427 hunch net-2011-03-20-KDD Cup 2011
18 0.086749904 260 hunch net-2007-08-25-The Privacy Problem
19 0.084404372 300 hunch net-2008-04-30-Concerns about the Large Scale Learning Challenge
20 0.084149346 132 hunch net-2005-11-26-The Design of an Optimal Research Environment
topicId topicWeight
[(0, 0.203), (1, 0.052), (2, -0.035), (3, -0.008), (4, -0.113), (5, 0.061), (6, -0.156), (7, 0.033), (8, 0.017), (9, -0.063), (10, -0.111), (11, 0.264), (12, -0.019), (13, 0.041), (14, -0.04), (15, -0.019), (16, 0.014), (17, 0.005), (18, 0.025), (19, 0.0), (20, -0.137), (21, -0.037), (22, 0.076), (23, -0.013), (24, -0.06), (25, -0.05), (26, -0.034), (27, -0.012), (28, 0.026), (29, 0.001), (30, 0.067), (31, -0.131), (32, -0.021), (33, 0.003), (34, 0.014), (35, -0.001), (36, 0.029), (37, 0.003), (38, -0.117), (39, 0.053), (40, -0.087), (41, 0.065), (42, 0.004), (43, 0.072), (44, -0.046), (45, 0.067), (46, -0.041), (47, -0.047), (48, 0.054), (49, -0.06)]
simIndex simValue blogId blogTitle
same-blog 1 0.96491724 430 hunch net-2011-04-11-The Heritage Health Prize
Introduction: The Heritage Health Prize is potentially the largest prediction prize yet at $3M, which is sure to get many people interested. Several elements of the competition may be worth discussing. The most straightforward way for HPN to deploy this predictor is in determining who to cover with insurance. This might easily cover the costs of running the contest itself, but the value to the health system of a whole is minimal, as people not covered still exist. While HPN itself is a provider network, they have active relationships with a number of insurance companies, and the right to resell any entrant. It’s worth keeping in mind that the research and development may nevertheless end up being useful in the longer term, especially as entrants also keep the right to their code. The judging metric is something I haven’t seen previously. If a patient has probability 0.5 of being in the hospital 0 days and probability 0.5 of being in the hospital ~53.6 days, the optimal prediction in e
2 0.81970125 371 hunch net-2009-09-21-Netflix finishes (and starts)
Introduction: I attended the Netflix prize ceremony this morning. The press conference part is covered fine elsewhere , with the basic outcome being that BellKor’s Pragmatic Chaos won over The Ensemble by 15-20 minutes , because they were tied in performance on the ultimate holdout set. I’m sure the individual participants will have many chances to speak about the solution. One of these is Bell at the NYAS ML symposium on Nov. 6 . Several additional details may interest ML people. The degree of overfitting exhibited by the difference in performance on the leaderboard test set and the ultimate hold out set was small, but determining at .02 to .03%. A tie was possible, because the rules cut off measurements below the fourth digit based on significance concerns. In actuality, of course, the scores do differ before rounding, but everyone I spoke to claimed not to know how. The complete dataset has been released on UCI , so each team could compute their own score to whatever accu
3 0.73528004 211 hunch net-2006-10-02-$1M Netflix prediction contest
Introduction: Netflix is running a contest to improve recommender prediction systems. A 10% improvement over their current system yields a $1M prize. Failing that, the best smaller improvement yields a smaller $50K prize. This contest looks quite real, and the $50K prize money is almost certainly achievable with a bit of thought. The contest also comes with a dataset which is apparently 2 orders of magnitude larger than any other public recommendation system datasets.
4 0.67213815 362 hunch net-2009-06-26-Netflix nearly done
Introduction: A $1M qualifying result was achieved on the public Netflix test set by a 3-way ensemble team . This is just in time for Yehuda ‘s presentation at KDD , which I’m sure will be one of the best attended ever. This isn’t quite over—there are a few days for another super-conglomerate team to come together and there is some small chance that the performance is nonrepresentative of the final test set, but I expect not. Regardless of the final outcome, the biggest lesson for ML from the Netflix contest has been the formidable performance edge of ensemble methods.
5 0.6589492 336 hunch net-2009-01-19-Netflix prize within epsilon
Introduction: The competitors for the Netflix Prize are tantalizingly close winning the million dollar prize. This year, BellKor and Commendo Research sent a combined solution that won the progress prize . Reading the writeups 2 is instructive. Several aspects of solutions are taken for granted including stochastic gradient descent, ensemble prediction, and targeting residuals (a form of boosting). Relatively to last year, it appears that many approaches have added parameterizations, especially for the purpose of modeling through time. The big question is: will they make the big prize? At this point, the level of complexity in entering the competition is prohibitive, so perhaps only the existing competitors will continue to try. (This equation might change drastically if the teams open source their existing solutions, including parameter settings.) One fear is that the progress is asymptoting on the wrong side of the 10% threshold. In the first year, the teams progressed through
6 0.57681614 275 hunch net-2007-11-29-The Netflix Crack
7 0.56946802 272 hunch net-2007-11-14-BellKor wins Netflix
8 0.54022688 217 hunch net-2006-11-06-Data Linkage Problems
9 0.52802193 427 hunch net-2011-03-20-KDD Cup 2011
10 0.50826961 129 hunch net-2005-11-07-Prediction Competitions
11 0.50316668 390 hunch net-2010-03-12-Netflix Challenge 2 Canceled
12 0.48382431 301 hunch net-2008-05-23-Three levels of addressing the Netflix Prize
13 0.45508379 259 hunch net-2007-08-19-Choice of Metrics
14 0.43273288 260 hunch net-2007-08-25-The Privacy Problem
15 0.43019888 270 hunch net-2007-11-02-The Machine Learning Award goes to …
16 0.42256707 119 hunch net-2005-10-08-We have a winner
17 0.39029974 165 hunch net-2006-03-23-The Approximation Argument
18 0.38817361 314 hunch net-2008-08-24-Mass Customized Medicine in the Future?
19 0.3762379 312 hunch net-2008-08-04-Electoralmarkets.com
20 0.37081102 160 hunch net-2006-03-02-Why do people count for learning?
topicId topicWeight
[(14, 0.32), (27, 0.263), (38, 0.046), (48, 0.023), (53, 0.041), (55, 0.058), (94, 0.078), (95, 0.086)]
simIndex simValue blogId blogTitle
1 0.97277027 94 hunch net-2005-07-13-Text Entailment at AAAI
Introduction: Rajat Raina presented a paper on the technique they used for the PASCAL Recognizing Textual Entailment challenge. “Text entailment” is the problem of deciding if one sentence implies another. For example the previous sentence entails: Text entailment is a decision problem. One sentence can imply another. The challenge was of the form: given an original sentence and another sentence predict whether there was an entailment. All current techniques for predicting correctness of an entailment are at the “flail” stage—accuracies of around 58% where humans could achieve near 100% accuracy, so there is much room to improve. Apparently, there may be another PASCAL challenge on this problem in the near future.
2 0.91860813 471 hunch net-2012-08-24-Patterns for research in machine learning
Introduction: There are a handful of basic code patterns that I wish I was more aware of when I started research in machine learning. Each on its own may seem pointless, but collectively they go a long way towards making the typical research workflow more efficient. Here they are: Separate code from data. Separate input data, working data and output data. Save everything to disk frequently. Separate options from parameters. Do not use global variables. Record the options used to generate each run of the algorithm. Make it easy to sweep options. Make it easy to execute only portions of the code. Use checkpointing. Write demos and tests. Click here for discussion and examples for each item. Also see Charles Sutton’s and HackerNews’ thoughts on the same topic. My guess is that these patterns will not only be useful for machine learning, but also any other computational work that involves either a) processing large amounts of data, or b) algorithms that take a signif
same-blog 3 0.87468642 430 hunch net-2011-04-11-The Heritage Health Prize
Introduction: The Heritage Health Prize is potentially the largest prediction prize yet at $3M, which is sure to get many people interested. Several elements of the competition may be worth discussing. The most straightforward way for HPN to deploy this predictor is in determining who to cover with insurance. This might easily cover the costs of running the contest itself, but the value to the health system of a whole is minimal, as people not covered still exist. While HPN itself is a provider network, they have active relationships with a number of insurance companies, and the right to resell any entrant. It’s worth keeping in mind that the research and development may nevertheless end up being useful in the longer term, especially as entrants also keep the right to their code. The judging metric is something I haven’t seen previously. If a patient has probability 0.5 of being in the hospital 0 days and probability 0.5 of being in the hospital ~53.6 days, the optimal prediction in e
4 0.87434345 380 hunch net-2009-11-29-AI Safety
Introduction: Dan Reeves introduced me to Michael Vassar who ran the Singularity Summit and educated me a bit on the subject of AI safety which the Singularity Institute has small grants for . I still believe that interstellar space travel is necessary for long term civilization survival, and the AI is necessary for interstellar space travel . On these grounds alone, we could judge that developing AI is much more safe than not. Nevertheless, there is a basic reasonable fear, as expressed by some commenters, that AI could go bad. A basic scenario starts with someone inventing an AI and telling it to make as much money as possible. The AI promptly starts trading in various markets to make money. To improve, it crafts a virus that takes over most of the world’s computers using it as a surveillance network so that it can always make the right decision. The AI also branches out into any form of distance work, taking over the entire outsourcing process for all jobs that are entirely di
5 0.84124386 407 hunch net-2010-08-23-Boosted Decision Trees for Deep Learning
Introduction: About 4 years ago, I speculated that decision trees qualify as a deep learning algorithm because they can make decisions which are substantially nonlinear in the input representation. Ping Li has proved this correct, empirically at UAI by showing that boosted decision trees can beat deep belief networks on versions of Mnist which are artificially hardened so as to make them solvable only by deep learning algorithms. This is an important point, because the ability to solve these sorts of problems is probably the best objective definition of a deep learning algorithm we have. I’m not that surprised. In my experience, if you can accept the computational drawbacks of a boosted decision tree, they can achieve pretty good performance. Geoff Hinton once told me that the great thing about deep belief networks is that they work. I understand that Ping had very substantial difficulty in getting this published, so I hope some reviewers step up to the standard of valuing wha
6 0.67660099 360 hunch net-2009-06-15-In Active Learning, the question changes
7 0.67091662 131 hunch net-2005-11-16-The Everything Ensemble Edge
8 0.66453385 12 hunch net-2005-02-03-Learning Theory, by assumption
9 0.66436708 235 hunch net-2007-03-03-All Models of Learning have Flaws
10 0.66089529 432 hunch net-2011-04-20-The End of the Beginning of Active Learning
11 0.66065264 351 hunch net-2009-05-02-Wielding a New Abstraction
12 0.65984577 132 hunch net-2005-11-26-The Design of an Optimal Research Environment
13 0.65981781 230 hunch net-2007-02-02-Thoughts regarding “Is machine learning different from statistics?”
14 0.6597048 370 hunch net-2009-09-18-Necessary and Sufficient Research
15 0.65946013 220 hunch net-2006-11-27-Continuizing Solutions
16 0.65927351 194 hunch net-2006-07-11-New Models
17 0.65918684 79 hunch net-2005-06-08-Question: “When is the right time to insert the loss function?”
18 0.65884757 371 hunch net-2009-09-21-Netflix finishes (and starts)
19 0.65874082 478 hunch net-2013-01-07-NYU Large Scale Machine Learning Class
20 0.65716779 196 hunch net-2006-07-13-Regression vs. Classification as a Primitive