hunch_net hunch_net-2009 hunch_net-2009-336 knowledge-graph by maker-knowledge-mining
Source: html
Introduction: The competitors for the Netflix Prize are tantalizingly close winning the million dollar prize. This year, BellKor and Commendo Research sent a combined solution that won the progress prize . Reading the writeups 2 is instructive. Several aspects of solutions are taken for granted including stochastic gradient descent, ensemble prediction, and targeting residuals (a form of boosting). Relatively to last year, it appears that many approaches have added parameterizations, especially for the purpose of modeling through time. The big question is: will they make the big prize? At this point, the level of complexity in entering the competition is prohibitive, so perhaps only the existing competitors will continue to try. (This equation might change drastically if the teams open source their existing solutions, including parameter settings.) One fear is that the progress is asymptoting on the wrong side of the 10% threshold. In the first year, the teams progressed through
sentIndex sentText sentNum sentScore
1 The competitors for the Netflix Prize are tantalizingly close winning the million dollar prize. [sent-1, score-0.778]
2 This year, BellKor and Commendo Research sent a combined solution that won the progress prize . [sent-2, score-0.816]
3 Several aspects of solutions are taken for granted including stochastic gradient descent, ensemble prediction, and targeting residuals (a form of boosting). [sent-4, score-0.726]
4 Relatively to last year, it appears that many approaches have added parameterizations, especially for the purpose of modeling through time. [sent-5, score-0.246]
5 The big question is: will they make the big prize? [sent-6, score-0.292]
6 At this point, the level of complexity in entering the competition is prohibitive, so perhaps only the existing competitors will continue to try. [sent-7, score-0.652]
7 (This equation might change drastically if the teams open source their existing solutions, including parameter settings. [sent-8, score-0.77]
8 ) One fear is that the progress is asymptoting on the wrong side of the 10% threshold. [sent-9, score-0.553]
9 In the first year, the teams progressed through 84. [sent-10, score-0.529]
10 3% of the 10% gap, and in the second year, they progressed through just 64. [sent-11, score-0.298]
11 While these numbers suggest an asymptote on the wrong side, in the month since the progress prize another 34. [sent-13, score-0.838]
12 It’s remarkable that it’s too close to call, with just a 0. [sent-15, score-0.26]
13 Clever people finding just the right parameterization might very well succeed. [sent-17, score-0.122]
wordName wordTfidf (topN-words)
[('prize', 0.349), ('progressed', 0.298), ('competitors', 0.264), ('teams', 0.231), ('progress', 0.199), ('gap', 0.187), ('year', 0.159), ('close', 0.154), ('big', 0.146), ('side', 0.141), ('solutions', 0.136), ('prohibitive', 0.132), ('granted', 0.122), ('dollar', 0.122), ('parameterization', 0.122), ('drastically', 0.122), ('bellkor', 0.122), ('million', 0.122), ('remainder', 0.122), ('wrong', 0.117), ('entering', 0.116), ('winning', 0.116), ('equation', 0.116), ('remaining', 0.11), ('existing', 0.11), ('including', 0.107), ('remarkable', 0.106), ('targeting', 0.106), ('sent', 0.102), ('clever', 0.099), ('month', 0.096), ('win', 0.096), ('fear', 0.096), ('modeling', 0.091), ('combined', 0.091), ('ensemble', 0.091), ('succeed', 0.087), ('netflix', 0.085), ('parameter', 0.084), ('continue', 0.084), ('stochastic', 0.082), ('aspects', 0.082), ('call', 0.079), ('boosting', 0.078), ('purpose', 0.078), ('competition', 0.078), ('suggest', 0.077), ('added', 0.077), ('won', 0.075), ('improvement', 0.073)]
simIndex simValue blogId blogTitle
same-blog 1 1.0 336 hunch net-2009-01-19-Netflix prize within epsilon
Introduction: The competitors for the Netflix Prize are tantalizingly close winning the million dollar prize. This year, BellKor and Commendo Research sent a combined solution that won the progress prize . Reading the writeups 2 is instructive. Several aspects of solutions are taken for granted including stochastic gradient descent, ensemble prediction, and targeting residuals (a form of boosting). Relatively to last year, it appears that many approaches have added parameterizations, especially for the purpose of modeling through time. The big question is: will they make the big prize? At this point, the level of complexity in entering the competition is prohibitive, so perhaps only the existing competitors will continue to try. (This equation might change drastically if the teams open source their existing solutions, including parameter settings.) One fear is that the progress is asymptoting on the wrong side of the 10% threshold. In the first year, the teams progressed through
2 0.15195422 371 hunch net-2009-09-21-Netflix finishes (and starts)
Introduction: I attended the Netflix prize ceremony this morning. The press conference part is covered fine elsewhere , with the basic outcome being that BellKor’s Pragmatic Chaos won over The Ensemble by 15-20 minutes , because they were tied in performance on the ultimate holdout set. I’m sure the individual participants will have many chances to speak about the solution. One of these is Bell at the NYAS ML symposium on Nov. 6 . Several additional details may interest ML people. The degree of overfitting exhibited by the difference in performance on the leaderboard test set and the ultimate hold out set was small, but determining at .02 to .03%. A tie was possible, because the rules cut off measurements below the fourth digit based on significance concerns. In actuality, of course, the scores do differ before rounding, but everyone I spoke to claimed not to know how. The complete dataset has been released on UCI , so each team could compute their own score to whatever accu
3 0.14966567 301 hunch net-2008-05-23-Three levels of addressing the Netflix Prize
Introduction: In October 2006, the online movie renter, Netflix, announced the Netflix Prize contest. They published a comprehensive dataset including more than 100 million movie ratings, which were performed by about 480,000 real customers on 17,770 movies.  Competitors in the challenge are required to estimate a few million ratings.  To win the “grand prize,” they need to deliver a 10% improvement in the prediction error compared with the results of Cinematch, Netflix’s proprietary recommender system. Best current results deliver 9.12% improvement , which is quite close to the 10% goal, yet painfully distant.  The Netflix Prize breathed new life and excitement into recommender systems research. The competition allowed the wide research community to access a large scale, real life dataset. Beyond this, the competition changed the rules of the game. Claiming that your nice idea could outperform some mediocre algorithms on some toy dataset is no longer acceptable. Now researcher
4 0.12321773 270 hunch net-2007-11-02-The Machine Learning Award goes to …
Introduction: Perhaps the biggest CS prize for research is the Turing Award , which has a $0.25M cash prize associated with it. It appears none of the prizes so far have been for anything like machine learning (the closest are perhaps database awards). In CS theory, there is the Gödel Prize which is smaller and newer, offering a $5K prize along and perhaps (more importantly) recognition. One such award has been given for Machine Learning, to Robert Schapire and Yoav Freund for Adaboost. In Machine Learning, there seems to be no equivalent of these sorts of prizes. There are several plausible reasons for this: There is no coherent community. People drift in and out of the central conferences all the time. Most of the author names from 10 years ago do not occur in the conferences of today. In addition, the entire subject area is fairly new. There are at least a core group of people who have stayed around. Machine Learning work doesn’t last Almost every paper is fo
5 0.11756071 86 hunch net-2005-06-28-The cross validation problem: cash reward
Introduction: I just presented the cross validation problem at COLT . The problem now has a cash prize (up to $500) associated with it—see the presentation for details. The write-up for colt .
6 0.115925 272 hunch net-2007-11-14-BellKor wins Netflix
7 0.11197491 377 hunch net-2009-11-09-NYAS ML Symposium this year.
8 0.11011219 430 hunch net-2011-04-11-The Heritage Health Prize
9 0.10754501 119 hunch net-2005-10-08-We have a winner
10 0.10263103 390 hunch net-2010-03-12-Netflix Challenge 2 Canceled
11 0.096719794 389 hunch net-2010-02-26-Yahoo! ML events
12 0.090444826 211 hunch net-2006-10-02-$1M Netflix prediction contest
13 0.082135141 111 hunch net-2005-09-12-Fast Gradient Descent
14 0.066806421 429 hunch net-2011-04-06-COLT open questions
15 0.06556274 96 hunch net-2005-07-21-Six Months
16 0.064277962 335 hunch net-2009-01-08-Predictive Analytics World
17 0.061399776 105 hunch net-2005-08-23-(Dis)similarities between academia and open source programmers
18 0.061098631 268 hunch net-2007-10-19-Second Annual Reinforcement Learning Competition
19 0.060241453 406 hunch net-2010-08-22-KDD 2010
20 0.060038034 477 hunch net-2013-01-01-Deep Learning 2012
topicId topicWeight
[(0, 0.139), (1, -0.006), (2, -0.057), (3, 0.024), (4, -0.035), (5, -0.014), (6, -0.096), (7, 0.005), (8, -0.035), (9, -0.038), (10, -0.073), (11, 0.178), (12, -0.062), (13, -0.023), (14, -0.028), (15, -0.008), (16, 0.054), (17, -0.029), (18, 0.024), (19, -0.055), (20, -0.181), (21, 0.041), (22, 0.06), (23, -0.047), (24, -0.005), (25, -0.058), (26, 0.002), (27, 0.051), (28, 0.045), (29, 0.03), (30, -0.028), (31, -0.037), (32, -0.121), (33, -0.064), (34, -0.111), (35, 0.036), (36, 0.009), (37, -0.072), (38, -0.062), (39, 0.018), (40, -0.118), (41, -0.039), (42, 0.006), (43, -0.034), (44, -0.041), (45, 0.027), (46, -0.055), (47, 0.04), (48, 0.142), (49, -0.018)]
simIndex simValue blogId blogTitle
same-blog 1 0.98282862 336 hunch net-2009-01-19-Netflix prize within epsilon
Introduction: The competitors for the Netflix Prize are tantalizingly close winning the million dollar prize. This year, BellKor and Commendo Research sent a combined solution that won the progress prize . Reading the writeups 2 is instructive. Several aspects of solutions are taken for granted including stochastic gradient descent, ensemble prediction, and targeting residuals (a form of boosting). Relatively to last year, it appears that many approaches have added parameterizations, especially for the purpose of modeling through time. The big question is: will they make the big prize? At this point, the level of complexity in entering the competition is prohibitive, so perhaps only the existing competitors will continue to try. (This equation might change drastically if the teams open source their existing solutions, including parameter settings.) One fear is that the progress is asymptoting on the wrong side of the 10% threshold. In the first year, the teams progressed through
2 0.6458174 430 hunch net-2011-04-11-The Heritage Health Prize
Introduction: The Heritage Health Prize is potentially the largest prediction prize yet at $3M, which is sure to get many people interested. Several elements of the competition may be worth discussing. The most straightforward way for HPN to deploy this predictor is in determining who to cover with insurance. This might easily cover the costs of running the contest itself, but the value to the health system of a whole is minimal, as people not covered still exist. While HPN itself is a provider network, they have active relationships with a number of insurance companies, and the right to resell any entrant. It’s worth keeping in mind that the research and development may nevertheless end up being useful in the longer term, especially as entrants also keep the right to their code. The judging metric is something I haven’t seen previously. If a patient has probability 0.5 of being in the hospital 0 days and probability 0.5 of being in the hospital ~53.6 days, the optimal prediction in e
3 0.5522539 301 hunch net-2008-05-23-Three levels of addressing the Netflix Prize
Introduction: In October 2006, the online movie renter, Netflix, announced the Netflix Prize contest. They published a comprehensive dataset including more than 100 million movie ratings, which were performed by about 480,000 real customers on 17,770 movies.  Competitors in the challenge are required to estimate a few million ratings.  To win the “grand prize,” they need to deliver a 10% improvement in the prediction error compared with the results of Cinematch, Netflix’s proprietary recommender system. Best current results deliver 9.12% improvement , which is quite close to the 10% goal, yet painfully distant.  The Netflix Prize breathed new life and excitement into recommender systems research. The competition allowed the wide research community to access a large scale, real life dataset. Beyond this, the competition changed the rules of the game. Claiming that your nice idea could outperform some mediocre algorithms on some toy dataset is no longer acceptable. Now researcher
4 0.54976195 272 hunch net-2007-11-14-BellKor wins Netflix
Introduction: … but only the little prize. The BellKor team focused on integrating predictions from many different methods. The base methods consist of: Nearest Neighbor Methods Matrix Factorization Methods (asymmetric and symmetric) Linear Regression on various feature spaces Restricted Boltzman Machines The final predictor was an ensemble (as was reasonable to expect), although it’s a little bit more complicated than just a weighted average—it’s essentially a customized learning algorithm. Base approaches (1)-(3) seem like relatively well-known approaches (although I haven’t seen the asymmetric factorization variant before). RBMs are the new approach. The writeup is pretty clear for more details. The contestants are close to reaching the big prize, but the last 1.5% is probably at least as hard as what’s been done. A few new structurally different methods for making predictions may need to be discovered and added into the mixture. In other words, research may be require
5 0.54463911 119 hunch net-2005-10-08-We have a winner
Introduction: The DARPA grandchallenge is a big contest for autonomous robot vehicle driving. It was run once in 2004 for the first time and all teams did badly. This year was notably different with the Stanford and CMU teams succesfully completing the course. A number of details are here and wikipedia has continuing coverage . A formal winner hasn’t been declared yet although Stanford completed the course quickest. The Stanford and CMU teams deserve a large round of applause as they have strongly demonstrated the feasibility of autonomous vehicles. The good news for machine learning is that the Stanford team (at least) is using some machine learning techniques.
6 0.53639489 270 hunch net-2007-11-02-The Machine Learning Award goes to …
7 0.53129089 371 hunch net-2009-09-21-Netflix finishes (and starts)
8 0.49655181 275 hunch net-2007-11-29-The Netflix Crack
9 0.47097152 362 hunch net-2009-06-26-Netflix nearly done
10 0.46055201 377 hunch net-2009-11-09-NYAS ML Symposium this year.
11 0.41839311 211 hunch net-2006-10-02-$1M Netflix prediction contest
12 0.41171998 389 hunch net-2010-02-26-Yahoo! ML events
13 0.41129732 268 hunch net-2007-10-19-Second Annual Reinforcement Learning Competition
14 0.38671559 390 hunch net-2010-03-12-Netflix Challenge 2 Canceled
15 0.38579711 86 hunch net-2005-06-28-The cross validation problem: cash reward
16 0.36630097 197 hunch net-2006-07-17-A Winner
17 0.35005447 111 hunch net-2005-09-12-Fast Gradient Descent
18 0.34994951 448 hunch net-2011-10-24-2011 ML symposium and the bears
19 0.34967315 167 hunch net-2006-03-27-Gradients everywhere
20 0.3482264 276 hunch net-2007-12-10-Learning Track of International Planning Competition
topicId topicWeight
[(27, 0.219), (54, 0.408), (55, 0.075), (94, 0.082), (95, 0.088), (98, 0.021)]
simIndex simValue blogId blogTitle
1 0.96286792 376 hunch net-2009-11-06-Yisong Yue on Self-improving Systems
Introduction: I’d like to point out Yisong Yue ‘s post on Self-improving systems , which is a nicely readable description of the necessity and potential of interactive learning to deal with the information overload problem that is endemic to the modern internet.
2 0.90778339 494 hunch net-2014-03-11-The New York ML Symposium, take 2
Introduction: The 20 13 14 is New York Machine Learning Symposium is finally happening on March 28th at the New York Academy of Science . Every invited speaker interests me personally. They are: Rayid Ghani (Chief Scientist at Obama 2012) Brian Kingsbury (Speech Recognition @ IBM) Jorge Nocedal (who did LBFGS) We’ve been somewhat disorganized in advertising this. As a consequence, anyone who has not submitted an abstract but would like to do so may send one directly to me (jl@hunch.net title NYASMLS) by Friday March 14. I will forward them to the rest of the committee for consideration.
same-blog 3 0.89038795 336 hunch net-2009-01-19-Netflix prize within epsilon
Introduction: The competitors for the Netflix Prize are tantalizingly close winning the million dollar prize. This year, BellKor and Commendo Research sent a combined solution that won the progress prize . Reading the writeups 2 is instructive. Several aspects of solutions are taken for granted including stochastic gradient descent, ensemble prediction, and targeting residuals (a form of boosting). Relatively to last year, it appears that many approaches have added parameterizations, especially for the purpose of modeling through time. The big question is: will they make the big prize? At this point, the level of complexity in entering the competition is prohibitive, so perhaps only the existing competitors will continue to try. (This equation might change drastically if the teams open source their existing solutions, including parameter settings.) One fear is that the progress is asymptoting on the wrong side of the 10% threshold. In the first year, the teams progressed through
4 0.8767485 335 hunch net-2009-01-08-Predictive Analytics World
Introduction: Carla Vicens and Eric Siegel contacted me about Predictive Analytics World in San Francisco February 18&19, which I wasn’t familiar with. A quick look at the agenda reveals several people I know working on applications of machine learning in businesses, covering deployed applications topics. It’s interesting to see a business-focused machine learning conference, as it says that we are succeeding as a field. If you are interested in deployed applications, you might attend. Eric and I did a quick interview by email. John > I’ve mostly published and participated in academic machine learning conferences like ICML, COLT, and NIPS. When I look at the set of speakers and subjects for your conference I think “machine learning for business”. Is that your understanding of things? What I’m trying to ask is: what do you view as the primary goal for this conference? Eric > You got it. This is the business event focused on the commercial deployment of technology developed at
5 0.7073648 33 hunch net-2005-02-28-Regularization
Introduction: Yaroslav Bulatov says that we should think about regularization a bit. It’s a complex topic which I only partially understand, so I’ll try to explain from a couple viewpoints. Functionally . Regularization is optimizing some representation to fit the data and minimize some notion of predictor complexity. This notion of complexity is often the l 1 or l 2 norm on a set of parameters, but the term can be used much more generally. Empirically, this often works much better than simply fitting the data. Statistical Learning Viewpoint Regularization is about the failiure of statistical learning to adequately predict generalization error. Let e(c,D) be the expected error rate with respect to D of classifier c and e(c,S) the observed error rate on a sample S . There are numerous bounds of the form: assuming i.i.d. samples, with high probability over the drawn samples S , e(c,D) less than e(c,S) + f(complexity) where complexity is some measure of the size of a s
6 0.70505989 458 hunch net-2012-03-06-COLT-ICML Open Questions and ICML Instructions
7 0.54696321 10 hunch net-2005-02-02-Kolmogorov Complexity and Googling
8 0.52088743 360 hunch net-2009-06-15-In Active Learning, the question changes
9 0.51511657 132 hunch net-2005-11-26-The Design of an Optimal Research Environment
10 0.51150084 220 hunch net-2006-11-27-Continuizing Solutions
11 0.51018816 369 hunch net-2009-08-27-New York Area Machine Learning Events
12 0.50824797 343 hunch net-2009-02-18-Decision by Vetocracy
13 0.50738561 378 hunch net-2009-11-15-The Other Online Learning
14 0.50736052 371 hunch net-2009-09-21-Netflix finishes (and starts)
15 0.50508285 51 hunch net-2005-04-01-The Producer-Consumer Model of Research
16 0.50405234 36 hunch net-2005-03-05-Funding Research
17 0.5026238 426 hunch net-2011-03-19-The Ideal Large Scale Learning Class
18 0.50254405 252 hunch net-2007-07-01-Watchword: Online Learning
19 0.50186408 345 hunch net-2009-03-08-Prediction Science
20 0.50176919 235 hunch net-2007-03-03-All Models of Learning have Flaws