hunch_net hunch_net-2006 hunch_net-2006-194 knowledge-graph by maker-knowledge-mining
Source: html
Introduction: How should we, as researchers in machine learning, organize ourselves? The most immediate measurable objective of computer science research is publishing a paper. The most difficult aspect of publishing a paper is having reviewers accept and recommend it for publication. The simplest mechanism for doing this is to show theoretical progress on some standard, well-known easily understood problem. In doing this, we often fall into a local minima of the research process. The basic problem in machine learning is that it is very unclear that the mathematical model is the right one for the (or some) real problem. A good mathematical model in machine learning should have one fundamental trait: it should aid the design of effective learning algorithms. To date, our ability to solve interesting learning problems (speech recognition, machine translation, object recognition, etc…) remains limited (although improving), so the “rightness” of our models is in doubt. If our mathematical mod
sentIndex sentText sentNum sentScore
1 The most immediate measurable objective of computer science research is publishing a paper. [sent-2, score-0.385]
2 The most difficult aspect of publishing a paper is having reviewers accept and recommend it for publication. [sent-3, score-0.207]
3 In doing this, we often fall into a local minima of the research process. [sent-5, score-0.262]
4 The basic problem in machine learning is that it is very unclear that the mathematical model is the right one for the (or some) real problem. [sent-6, score-0.876]
5 A good mathematical model in machine learning should have one fundamental trait: it should aid the design of effective learning algorithms. [sent-7, score-0.876]
6 To date, our ability to solve interesting learning problems (speech recognition, machine translation, object recognition, etc…) remains limited (although improving), so the “rightness” of our models is in doubt. [sent-8, score-0.38]
7 If our mathematical models are bad, the simple mechanism of research above can not yield the end goal. [sent-9, score-1.005]
8 (This should be agreed on even by people who disagree about what the end goal of machine learning is! [sent-10, score-0.361]
9 ) Instead, research which proposes and investigates new mathematical models for machine learning might yield the end goal. [sent-11, score-1.329]
10 Coming up with a new mathematical model is just plain not easy. [sent-13, score-1.017]
11 Some sources of inspiration include: Watching carefully: what happens succesfully in practice, can often be abstracted into a mathematical model. [sent-14, score-0.742]
12 Swapping fields: In other fields (for example crypto ), other methods of analysis have been developed. [sent-15, score-0.27]
13 Model repair: Existing mathematical models often have easily comprehendable failure modes. [sent-17, score-0.84]
14 By thinking about how to avoid such failure modes, we can sometimes produce a new mathematical model. [sent-18, score-0.71]
15 Often, when trying to converge on a new model, we think of it in terms of the difference with respect to an older model, leading to a tangled explanation. [sent-21, score-0.21]
16 For a reviewer with limited time, it is very tempting to assume that any particular paper is operating in some familiar model and fail out. [sent-23, score-0.501]
17 You can’t be too blunt about saying “this isn’t the model you are thinking about”. [sent-25, score-0.651]
18 When people don’t have a reference frame to understand the new model, they are unlikely to follow up, as is necessary for success in academia. [sent-27, score-0.427]
19 The good news here is that a succesful new model can be a big win. [sent-28, score-0.537]
20 I wish it was an easier win: the barriers to success are formidably high, and it seems we should do everything possible to lower the barriers to success for the sake of improving research. [sent-29, score-0.82]
wordName wordTfidf (topN-words)
[('model', 0.401), ('mathematical', 0.391), ('models', 0.196), ('barriers', 0.168), ('success', 0.139), ('new', 0.136), ('end', 0.127), ('recognition', 0.122), ('publishing', 0.118), ('fields', 0.118), ('improving', 0.11), ('research', 0.106), ('yield', 0.104), ('limited', 0.1), ('failure', 0.099), ('rightness', 0.096), ('abstracted', 0.096), ('blunt', 0.096), ('inspiration', 0.096), ('investigates', 0.096), ('sake', 0.096), ('super', 0.096), ('reviewers', 0.089), ('repair', 0.089), ('measurable', 0.089), ('plain', 0.089), ('proposes', 0.089), ('succeeding', 0.089), ('watching', 0.089), ('difficulty', 0.088), ('thinking', 0.084), ('swapping', 0.084), ('machine', 0.084), ('mechanism', 0.081), ('succesfully', 0.08), ('frame', 0.08), ('agreed', 0.08), ('often', 0.079), ('crypto', 0.077), ('minima', 0.077), ('modes', 0.077), ('methods', 0.075), ('easily', 0.075), ('translation', 0.074), ('converge', 0.074), ('continues', 0.074), ('unlikely', 0.072), ('objective', 0.072), ('disagree', 0.07), ('saying', 0.07)]
simIndex simValue blogId blogTitle
same-blog 1 1.0 194 hunch net-2006-07-11-New Models
Introduction: How should we, as researchers in machine learning, organize ourselves? The most immediate measurable objective of computer science research is publishing a paper. The most difficult aspect of publishing a paper is having reviewers accept and recommend it for publication. The simplest mechanism for doing this is to show theoretical progress on some standard, well-known easily understood problem. In doing this, we often fall into a local minima of the research process. The basic problem in machine learning is that it is very unclear that the mathematical model is the right one for the (or some) real problem. A good mathematical model in machine learning should have one fundamental trait: it should aid the design of effective learning algorithms. To date, our ability to solve interesting learning problems (speech recognition, machine translation, object recognition, etc…) remains limited (although improving), so the “rightness” of our models is in doubt. If our mathematical mod
2 0.26173189 95 hunch net-2005-07-14-What Learning Theory might do
Introduction: I wanted to expand on this post and some of the previous problems/research directions about where learning theory might make large strides. Why theory? The essential reason for theory is “intuition extension”. A very good applied learning person can master some particular application domain yielding the best computer algorithms for solving that problem. A very good theory can take the intuitions discovered by this and other applied learning people and extend them to new domains in a relatively automatic fashion. To do this, we take these basic intuitions and try to find a mathematical model that: Explains the basic intuitions. Makes new testable predictions about how to learn. Succeeds in so learning. This is “intuition extension”: taking what we have learned somewhere else and applying it in new domains. It is fundamentally useful to everyone because it increases the level of automation in solving problems. Where next for learning theory? I like the a
3 0.19835004 135 hunch net-2005-12-04-Watchword: model
Introduction: In everyday use a model is a system which explains the behavior of some system, hopefully at the level where some alteration of the model predicts some alteration of the real-world system. In machine learning “model” has several variant definitions. Everyday . The common definition is sometimes used. Parameterized . Sometimes model is a short-hand for “parameterized model”. Here, it refers to a model with unspecified free parameters. In the Bayesian learning approach, you typically have a prior over (everyday) models. Predictive . Even further from everyday use is the predictive model. Examples of this are “my model is a decision tree” or “my model is a support vector machine”. Here, there is no real sense in which an SVM explains the underlying process. For example, an SVM tells us nothing in particular about how alterations to the real-world system would create a change. Which definition is being used at any particular time is important information. For examp
4 0.18078065 301 hunch net-2008-05-23-Three levels of addressing the Netflix Prize
Introduction: In October 2006, the online movie renter, Netflix, announced the Netflix Prize contest. They published a comprehensive dataset including more than 100 million movie ratings, which were performed by about 480,000 real customers on 17,770 movies.  Competitors in the challenge are required to estimate a few million ratings.  To win the “grand prize,” they need to deliver a 10% improvement in the prediction error compared with the results of Cinematch, Netflix’s proprietary recommender system. Best current results deliver 9.12% improvement , which is quite close to the 10% goal, yet painfully distant.  The Netflix Prize breathed new life and excitement into recommender systems research. The competition allowed the wide research community to access a large scale, real life dataset. Beyond this, the competition changed the rules of the game. Claiming that your nice idea could outperform some mediocre algorithms on some toy dataset is no longer acceptable. Now researcher
5 0.16840719 97 hunch net-2005-07-23-Interesting papers at ACL
Introduction: A recent discussion indicated that one goal of this blog might be to allow people to post comments about recent papers that they liked. I think this could potentially be very useful, especially for those with diverse interests but only finite time to read through conference proceedings. ACL 2005 recently completed, and here are four papers from that conference that I thought were either good or perhaps of interest to a machine learning audience. David Chiang, A Hierarchical Phrase-Based Model for Statistical Machine Translation . (Best paper award.) This paper takes the standard phrase-based MT model that is popular in our field (basically, translate a sentence by individually translating phrases and reordering them according to a complicated statistical model) and extends it to take into account hierarchy in phrases, so that you can learn things like “X ‘s Y” -> “Y de X” in chinese, where X and Y are arbitrary phrases. This takes a step toward linguistic syntax for MT, whic
6 0.15896864 204 hunch net-2006-08-28-Learning Theory standards for NIPS 2006
7 0.15409045 51 hunch net-2005-04-01-The Producer-Consumer Model of Research
8 0.1225839 235 hunch net-2007-03-03-All Models of Learning have Flaws
9 0.12218384 454 hunch net-2012-01-30-ICML Posters and Scope
10 0.11521645 347 hunch net-2009-03-26-Machine Learning is too easy
11 0.11240302 343 hunch net-2009-02-18-Decision by Vetocracy
12 0.10961941 132 hunch net-2005-11-26-The Design of an Optimal Research Environment
13 0.10869491 89 hunch net-2005-07-04-The Health of COLT
14 0.10739357 162 hunch net-2006-03-09-Use of Notation
15 0.10358946 435 hunch net-2011-05-16-Research Directions for Machine Learning and Algorithms
16 0.10328845 22 hunch net-2005-02-18-What it means to do research.
17 0.1018286 484 hunch net-2013-06-16-Representative Reviewing
18 0.10168904 158 hunch net-2006-02-24-A Fundamentalist Organization of Machine Learning
19 0.10060982 3 hunch net-2005-01-24-The Humanloop Spectrum of Machine Learning
20 0.098267354 44 hunch net-2005-03-21-Research Styles in Machine Learning
topicId topicWeight
[(0, 0.245), (1, -0.012), (2, -0.016), (3, 0.114), (4, -0.002), (5, -0.043), (6, 0.036), (7, 0.069), (8, 0.1), (9, -0.083), (10, 0.061), (11, -0.069), (12, -0.1), (13, 0.007), (14, 0.052), (15, 0.034), (16, 0.121), (17, 0.069), (18, 0.135), (19, -0.182), (20, -0.077), (21, -0.087), (22, -0.096), (23, -0.058), (24, 0.011), (25, 0.007), (26, -0.074), (27, -0.063), (28, -0.094), (29, -0.104), (30, 0.022), (31, 0.117), (32, -0.028), (33, 0.06), (34, 0.002), (35, 0.044), (36, 0.068), (37, -0.14), (38, -0.079), (39, -0.151), (40, -0.064), (41, -0.001), (42, -0.138), (43, 0.012), (44, 0.039), (45, 0.004), (46, 0.035), (47, -0.047), (48, 0.063), (49, -0.05)]
simIndex simValue blogId blogTitle
same-blog 1 0.97928941 194 hunch net-2006-07-11-New Models
Introduction: How should we, as researchers in machine learning, organize ourselves? The most immediate measurable objective of computer science research is publishing a paper. The most difficult aspect of publishing a paper is having reviewers accept and recommend it for publication. The simplest mechanism for doing this is to show theoretical progress on some standard, well-known easily understood problem. In doing this, we often fall into a local minima of the research process. The basic problem in machine learning is that it is very unclear that the mathematical model is the right one for the (or some) real problem. A good mathematical model in machine learning should have one fundamental trait: it should aid the design of effective learning algorithms. To date, our ability to solve interesting learning problems (speech recognition, machine translation, object recognition, etc…) remains limited (although improving), so the “rightness” of our models is in doubt. If our mathematical mod
2 0.85481632 135 hunch net-2005-12-04-Watchword: model
Introduction: In everyday use a model is a system which explains the behavior of some system, hopefully at the level where some alteration of the model predicts some alteration of the real-world system. In machine learning “model” has several variant definitions. Everyday . The common definition is sometimes used. Parameterized . Sometimes model is a short-hand for “parameterized model”. Here, it refers to a model with unspecified free parameters. In the Bayesian learning approach, you typically have a prior over (everyday) models. Predictive . Even further from everyday use is the predictive model. Examples of this are “my model is a decision tree” or “my model is a support vector machine”. Here, there is no real sense in which an SVM explains the underlying process. For example, an SVM tells us nothing in particular about how alterations to the real-world system would create a change. Which definition is being used at any particular time is important information. For examp
3 0.75377822 95 hunch net-2005-07-14-What Learning Theory might do
Introduction: I wanted to expand on this post and some of the previous problems/research directions about where learning theory might make large strides. Why theory? The essential reason for theory is “intuition extension”. A very good applied learning person can master some particular application domain yielding the best computer algorithms for solving that problem. A very good theory can take the intuitions discovered by this and other applied learning people and extend them to new domains in a relatively automatic fashion. To do this, we take these basic intuitions and try to find a mathematical model that: Explains the basic intuitions. Makes new testable predictions about how to learn. Succeeds in so learning. This is “intuition extension”: taking what we have learned somewhere else and applying it in new domains. It is fundamentally useful to everyone because it increases the level of automation in solving problems. Where next for learning theory? I like the a
4 0.75051093 301 hunch net-2008-05-23-Three levels of addressing the Netflix Prize
Introduction: In October 2006, the online movie renter, Netflix, announced the Netflix Prize contest. They published a comprehensive dataset including more than 100 million movie ratings, which were performed by about 480,000 real customers on 17,770 movies.  Competitors in the challenge are required to estimate a few million ratings.  To win the “grand prize,” they need to deliver a 10% improvement in the prediction error compared with the results of Cinematch, Netflix’s proprietary recommender system. Best current results deliver 9.12% improvement , which is quite close to the 10% goal, yet painfully distant.  The Netflix Prize breathed new life and excitement into recommender systems research. The competition allowed the wide research community to access a large scale, real life dataset. Beyond this, the competition changed the rules of the game. Claiming that your nice idea could outperform some mediocre algorithms on some toy dataset is no longer acceptable. Now researcher
5 0.74304163 97 hunch net-2005-07-23-Interesting papers at ACL
Introduction: A recent discussion indicated that one goal of this blog might be to allow people to post comments about recent papers that they liked. I think this could potentially be very useful, especially for those with diverse interests but only finite time to read through conference proceedings. ACL 2005 recently completed, and here are four papers from that conference that I thought were either good or perhaps of interest to a machine learning audience. David Chiang, A Hierarchical Phrase-Based Model for Statistical Machine Translation . (Best paper award.) This paper takes the standard phrase-based MT model that is popular in our field (basically, translate a sentence by individually translating phrases and reordering them according to a complicated statistical model) and extends it to take into account hierarchy in phrases, so that you can learn things like “X ‘s Y” -> “Y de X” in chinese, where X and Y are arbitrary phrases. This takes a step toward linguistic syntax for MT, whic
6 0.65989816 204 hunch net-2006-08-28-Learning Theory standards for NIPS 2006
7 0.6506362 51 hunch net-2005-04-01-The Producer-Consumer Model of Research
8 0.56502116 440 hunch net-2011-08-06-Interesting thing at UAI 2011
9 0.51944435 172 hunch net-2006-04-14-JMLR is a success
10 0.50313157 162 hunch net-2006-03-09-Use of Notation
11 0.49214178 189 hunch net-2006-07-05-more icml papers
12 0.48351416 171 hunch net-2006-04-09-Progress in Machine Translation
13 0.48313379 280 hunch net-2007-12-20-Cool and Interesting things at NIPS, take three
14 0.47871608 208 hunch net-2006-09-18-What is missing for online collaborative research?
15 0.46688876 22 hunch net-2005-02-18-What it means to do research.
16 0.4636254 42 hunch net-2005-03-17-Going all the Way, Sometimes
17 0.4618223 44 hunch net-2005-03-21-Research Styles in Machine Learning
18 0.45939258 90 hunch net-2005-07-07-The Limits of Learning Theory
19 0.45886645 454 hunch net-2012-01-30-ICML Posters and Scope
20 0.45748755 142 hunch net-2005-12-22-Yes , I am applying
topicId topicWeight
[(3, 0.055), (21, 0.042), (27, 0.235), (37, 0.192), (38, 0.055), (53, 0.094), (55, 0.101), (64, 0.017), (94, 0.066), (95, 0.059)]
simIndex simValue blogId blogTitle
1 0.92801577 368 hunch net-2009-08-26-Another 10-year paper in Machine Learning
Introduction: When I was thinking about the best “10 year paper” for ICML , I also took a look at a few other conferences. Here is one from 10 years ago that interested me: David McAllester PAC-Bayesian Model Averaging , COLT 1999. 2001 Journal Draft . Prior to this paper, the only mechanism known for controlling or estimating the necessary sample complexity for learning over continuously parameterized predictors was VC theory and variants, all of which suffered from a basic problem: they were incredibly pessimistic in practice. This meant that only very gross guidance could be provided for learning algorithm design. The PAC-Bayes bound provided an alternative approach to sample complexity bounds which was radically tighter, quantitatively. It also imported and explained many of the motivations for Bayesian learning in a way that learning theory and perhaps optimization people might appreciate. Since this paper came out, there have been a number of moderately successful attempts t
2 0.92576337 138 hunch net-2005-12-09-Some NIPS papers
Introduction: Here is a set of papers that I found interesting (and why). A PAC-Bayes approach to the Set Covering Machine improves the set covering machine. The set covering machine approach is a new way to do classification characterized by a very close connection between theory and algorithm. At this point, the approach seems to be competing well with SVMs in about all dimensions: similar computational speed, similar accuracy, stronger learning theory guarantees, more general information source (a kernel has strictly more structure than a metric), and more sparsity. Developing a classification algorithm is not very easy, but the results so far are encouraging. Off-Road Obstacle Avoidance through End-to-End Learning and Learning Depth from Single Monocular Images both effectively showed that depth information can be predicted from camera images (using notably different techniques). This ability is strongly enabling because cameras are cheap, tiny, light, and potentially provider lo
3 0.906663 1 hunch net-2005-01-19-Why I decided to run a weblog.
Introduction: I have decided to run a weblog on machine learning and learning theory research. Here are some reasons: 1) Weblogs enable new functionality: Public comment on papers. No mechanism for this exists at conferences and most journals. I have encountered it once for a science paper. Some communities have mailing lists supporting this, but not machine learning or learning theory. I have often read papers and found myself wishing there was some method to consider other’s questions and read the replies. Conference shortlists. One of the most common conversations at a conference is “what did you find interesting?” There is no explicit mechanism for sharing this information at conferences, and it’s easy to imagine that it would be handy to do so. Evaluation and comment on research directions. Papers are almost exclusively about new research, rather than evaluation (and consideration) of research directions. This last role is satisfied by funding agencies to some extent, but
4 0.89935815 431 hunch net-2011-04-18-A paper not at Snowbird
Introduction: Unfortunately, a scheduling failure meant I missed all of AIStat and most of the learning workshop , otherwise known as Snowbird, when it’s at Snowbird . At snowbird, the talk on Sum-Product networks by Hoifung Poon stood out to me ( Pedro Domingos is a coauthor.). The basic point was that by appropriately constructing networks based on sums and products, the normalization problem in probabilistic models is eliminated, yielding a highly tractable yet flexible representation+learning algorithm. As an algorithm, this is noticeably cleaner than deep belief networks with a claim to being an order of magnitude faster and working better on an image completion task. Snowbird doesn’t have real papers—just the abstract above. I look forward to seeing the paper. (added: Rodrigo points out the deep learning workshop draft .)
same-blog 5 0.89133751 194 hunch net-2006-07-11-New Models
Introduction: How should we, as researchers in machine learning, organize ourselves? The most immediate measurable objective of computer science research is publishing a paper. The most difficult aspect of publishing a paper is having reviewers accept and recommend it for publication. The simplest mechanism for doing this is to show theoretical progress on some standard, well-known easily understood problem. In doing this, we often fall into a local minima of the research process. The basic problem in machine learning is that it is very unclear that the mathematical model is the right one for the (or some) real problem. A good mathematical model in machine learning should have one fundamental trait: it should aid the design of effective learning algorithms. To date, our ability to solve interesting learning problems (speech recognition, machine translation, object recognition, etc…) remains limited (although improving), so the “rightness” of our models is in doubt. If our mathematical mod
6 0.85968381 21 hunch net-2005-02-17-Learning Research Programs
7 0.81455451 63 hunch net-2005-04-27-DARPA project: LAGR
8 0.805049 478 hunch net-2013-01-07-NYU Large Scale Machine Learning Class
9 0.8046878 343 hunch net-2009-02-18-Decision by Vetocracy
10 0.80446959 370 hunch net-2009-09-18-Necessary and Sufficient Research
11 0.80422181 134 hunch net-2005-12-01-The Webscience Future
12 0.80315399 227 hunch net-2007-01-10-A Deep Belief Net Learning Problem
13 0.80293834 143 hunch net-2005-12-27-Automated Labeling
14 0.8019833 378 hunch net-2009-11-15-The Other Online Learning
15 0.80179423 12 hunch net-2005-02-03-Learning Theory, by assumption
16 0.80123484 183 hunch net-2006-06-14-Explorations of Exploration
17 0.80119979 351 hunch net-2009-05-02-Wielding a New Abstraction
18 0.80015737 337 hunch net-2009-01-21-Nearly all natural problems require nonlinearity
19 0.79920495 435 hunch net-2011-05-16-Research Directions for Machine Learning and Algorithms
20 0.79849452 158 hunch net-2006-02-24-A Fundamentalist Organization of Machine Learning