hunch_net hunch_net-2006 hunch_net-2006-204 knowledge-graph by maker-knowledge-mining
Source: html
Introduction: Bob Williamson and I are the learning theory PC members at NIPS this year. This is some attempt to state the standards and tests I applied to the papers. I think it is a good idea to talk about this for two reasons: Making community standards a matter of public record seems healthy. It give us a chance to debate what is and is not the right standard. It might even give us a bit more consistency across the years. It may save us all time. There are a number of papers submitted which just aren’t there yet. Avoiding submitting is the right decision in this case. There are several criteria for judging a paper. All of these were active this year. Some criteria are uncontroversial while others may be so. The paper must have a theorem establishing something new for which it is possible to derive high confidence in the correctness of the results. A surprising number of papers fail this test. This criteria seems essential to the definition of “theory”. Missing theo
sentIndex sentText sentNum sentScore
1 This is some attempt to state the standards and tests I applied to the papers. [sent-2, score-0.315]
2 It give us a chance to debate what is and is not the right standard. [sent-4, score-0.432]
3 It might even give us a bit more consistency across the years. [sent-5, score-0.274]
4 There are a number of papers submitted which just aren’t there yet. [sent-7, score-0.218]
5 There are several criteria for judging a paper. [sent-9, score-0.327]
6 Some criteria are uncontroversial while others may be so. [sent-11, score-0.254]
7 The paper must have a theorem establishing something new for which it is possible to derive high confidence in the correctness of the results. [sent-12, score-0.687]
8 This criteria seems essential to the definition of “theory”. [sent-14, score-0.254]
9 Missing theorem statement Missing proof This isn’t an automatic fail, because sometimes reviewers can be expected to fill in the proof from discussion. [sent-15, score-0.89]
10 Providing the right amount of detail to give confidence in the results is tricky, but general advice is: err on the side of being explicit. [sent-18, score-0.444]
11 Imprecise theorem statement A number of theorems are simply too imprecise to verify or imagine verifying. [sent-19, score-0.809]
12 Typos and thinkos Often a theorem statement or proof is “right” when expressed correctly, but it isn’t expressed correctly: typos and thinkos (little correctable bugs in how you were thinking) confuse the reader. [sent-21, score-1.617]
13 A theorem of the form “algorithm A can do B” is not new when we already know “algorithm C can do B”. [sent-23, score-0.318]
14 Sometimes a paper has a reasonable chance of passing evaluation as an algorithms paper (which has experimental requirements). [sent-26, score-0.365]
15 For a new model of learning, this test was applied only weakly. [sent-31, score-0.428]
16 I had a preference for papers presenting new mathematical models. [sent-35, score-0.332]
17 I liked Neil Lawrence ‘s comment: “If we started rejecting learning theory papers for having the wrong model, where would we stop? [sent-36, score-0.316]
18 ” There is a natural tendency to forget the drawbacks of the accepted models in machine learning when evaluating new models, so it seems appropriate to provide some encouragement towards exploration. [sent-37, score-0.26]
19 Sometimes experiments helped (especially when the theory was weak), and sometimes they had no effect. [sent-39, score-0.269]
20 It’s my hope that more such decisions can be made right in the future, so I’d like to invite comments on what the right criteria are and why. [sent-41, score-0.689]
wordName wordTfidf (topN-words)
[('criteria', 0.254), ('theorem', 0.21), ('imprecise', 0.197), ('thinkos', 0.197), ('proof', 0.189), ('sometimes', 0.164), ('typos', 0.162), ('right', 0.153), ('expressed', 0.14), ('statement', 0.138), ('papers', 0.135), ('correctly', 0.135), ('standards', 0.131), ('decisions', 0.129), ('give', 0.113), ('applied', 0.111), ('test', 0.11), ('new', 0.108), ('theory', 0.105), ('confidence', 0.105), ('paper', 0.104), ('fail', 0.102), ('theorems', 0.1), ('model', 0.099), ('missing', 0.097), ('mathematical', 0.089), ('establishing', 0.087), ('correctable', 0.087), ('agenda', 0.087), ('neil', 0.087), ('us', 0.085), ('number', 0.083), ('happens', 0.081), ('chance', 0.081), ('encouragement', 0.081), ('verify', 0.081), ('bugs', 0.081), ('stop', 0.081), ('english', 0.076), ('mixed', 0.076), ('confuse', 0.076), ('consistency', 0.076), ('passing', 0.076), ('rejecting', 0.076), ('williamson', 0.076), ('correctness', 0.073), ('judging', 0.073), ('tests', 0.073), ('err', 0.073), ('models', 0.071)]
simIndex simValue blogId blogTitle
same-blog 1 1.0000001 204 hunch net-2006-08-28-Learning Theory standards for NIPS 2006
Introduction: Bob Williamson and I are the learning theory PC members at NIPS this year. This is some attempt to state the standards and tests I applied to the papers. I think it is a good idea to talk about this for two reasons: Making community standards a matter of public record seems healthy. It give us a chance to debate what is and is not the right standard. It might even give us a bit more consistency across the years. It may save us all time. There are a number of papers submitted which just aren’t there yet. Avoiding submitting is the right decision in this case. There are several criteria for judging a paper. All of these were active this year. Some criteria are uncontroversial while others may be so. The paper must have a theorem establishing something new for which it is possible to derive high confidence in the correctness of the results. A surprising number of papers fail this test. This criteria seems essential to the definition of “theory”. Missing theo
2 0.21019347 454 hunch net-2012-01-30-ICML Posters and Scope
Introduction: Normally, I don’t indulge in posters for ICML , but this year is naturally an exception for me. If you want one, there are a small number left here , if you sign up before February. It also seems worthwhile to give some sense of the scope and reviewing criteria for ICML for authors considering submitting papers. At ICML, the (very large) program committee does the reviewing which informs final decisions by area chairs on most papers. Program chairs setup the process, deal with exceptions or disagreements, and provide advice for the reviewing process. Providing advice is tricky (and easily misleading) because a conference is a community, and in the end the aggregate interests of the community determine the conference. Nevertheless, as a program chair this year it seems worthwhile to state the overall philosophy I have and what I plan to encourage (and occasionally discourage). At the highest level, I believe ICML exists to further research into machine learning, which I gene
3 0.19260126 187 hunch net-2006-06-25-Presentation of Proofs is Hard.
Introduction: When presenting part of the Reinforcement Learning theory tutorial at ICML 2006 , I was forcibly reminded of this. There are several difficulties. When creating the presentation, the correct level of detail is tricky. With too much detail, the proof takes too much time and people may be lost to boredom. With too little detail, the steps of the proof involve too-great a jump. This is very difficult to judge. What may be an easy step in the careful thought of a quiet room is not so easy when you are occupied by the process of presentation. What may be easy after having gone over this (and other) proofs is not so easy to follow in the first pass by a viewer. These problems seem only correctable by process of repeated test-and-revise. When presenting the proof, simply speaking with sufficient precision is substantially harder than in normal conversation (where precision is not so critical). Practice can help here. When presenting the proof, going at the right p
4 0.17793934 318 hunch net-2008-09-26-The SODA Program Committee
Introduction: Claire asked me to be on the SODA program committee this year, which was quite a bit of work. I had a relatively light load—merely 49 theory papers. Many of these papers were not on subjects that I was expert about, so (as is common for theory conferences) I found various reviewers that I trusted to help review the papers. I ended up reviewing about 1/3 personally. There were a couple instances where I ended up overruling a subreviewer whose logic seemed off, but otherwise I generally let their reviews stand. There are some differences in standards for paper reviews between the machine learning and theory communities. In machine learning it is expected that a review be detailed, while in the theory community this is often not the case. Every paper given to me ended up with a review varying between somewhat and very detailed. I’m sure not every author was happy with the outcome. While we did our best to make good decisions, they were difficult decisions to make. For exam
5 0.17369002 95 hunch net-2005-07-14-What Learning Theory might do
Introduction: I wanted to expand on this post and some of the previous problems/research directions about where learning theory might make large strides. Why theory? The essential reason for theory is “intuition extension”. A very good applied learning person can master some particular application domain yielding the best computer algorithms for solving that problem. A very good theory can take the intuitions discovered by this and other applied learning people and extend them to new domains in a relatively automatic fashion. To do this, we take these basic intuitions and try to find a mathematical model that: Explains the basic intuitions. Makes new testable predictions about how to learn. Succeeds in so learning. This is “intuition extension”: taking what we have learned somewhere else and applying it in new domains. It is fundamentally useful to everyone because it increases the level of automation in solving problems. Where next for learning theory? I like the a
6 0.15896864 194 hunch net-2006-07-11-New Models
7 0.15785113 320 hunch net-2008-10-14-Who is Responsible for a Bad Review?
8 0.15721995 221 hunch net-2006-12-04-Structural Problems in NIPS Decision Making
9 0.15129197 325 hunch net-2008-11-10-ICML Reviewing Criteria
10 0.14084181 180 hunch net-2006-05-21-NIPS paper evaluation criteria
11 0.13931915 332 hunch net-2008-12-23-Use of Learning Theory
12 0.13274944 104 hunch net-2005-08-22-Do you believe in induction?
13 0.13192143 343 hunch net-2009-02-18-Decision by Vetocracy
14 0.12849163 304 hunch net-2008-06-27-Reviewing Horror Stories
15 0.12822783 235 hunch net-2007-03-03-All Models of Learning have Flaws
16 0.12373362 38 hunch net-2005-03-09-Bad Reviewing
17 0.11923906 233 hunch net-2007-02-16-The Forgetting
18 0.11919452 437 hunch net-2011-07-10-ICML 2011 and the future
19 0.11894008 213 hunch net-2006-10-08-Incompatibilities between classical confidence intervals and learning.
20 0.11474198 202 hunch net-2006-08-10-Precision is not accuracy
topicId topicWeight
[(0, 0.287), (1, -0.041), (2, 0.118), (3, 0.071), (4, 0.05), (5, 0.014), (6, 0.011), (7, 0.01), (8, 0.051), (9, -0.05), (10, 0.08), (11, 0.003), (12, -0.003), (13, 0.036), (14, 0.049), (15, -0.014), (16, 0.07), (17, 0.024), (18, 0.039), (19, -0.064), (20, -0.026), (21, 0.048), (22, -0.069), (23, -0.121), (24, -0.047), (25, -0.036), (26, -0.032), (27, -0.152), (28, -0.083), (29, -0.121), (30, -0.003), (31, 0.05), (32, -0.131), (33, 0.032), (34, 0.059), (35, -0.048), (36, -0.015), (37, 0.002), (38, 0.033), (39, -0.09), (40, 0.007), (41, 0.105), (42, -0.022), (43, 0.022), (44, -0.035), (45, 0.041), (46, 0.012), (47, 0.039), (48, 0.127), (49, -0.046)]
simIndex simValue blogId blogTitle
same-blog 1 0.97642273 204 hunch net-2006-08-28-Learning Theory standards for NIPS 2006
Introduction: Bob Williamson and I are the learning theory PC members at NIPS this year. This is some attempt to state the standards and tests I applied to the papers. I think it is a good idea to talk about this for two reasons: Making community standards a matter of public record seems healthy. It give us a chance to debate what is and is not the right standard. It might even give us a bit more consistency across the years. It may save us all time. There are a number of papers submitted which just aren’t there yet. Avoiding submitting is the right decision in this case. There are several criteria for judging a paper. All of these were active this year. Some criteria are uncontroversial while others may be so. The paper must have a theorem establishing something new for which it is possible to derive high confidence in the correctness of the results. A surprising number of papers fail this test. This criteria seems essential to the definition of “theory”. Missing theo
2 0.73991817 52 hunch net-2005-04-04-Grounds for Rejection
Introduction: It’s reviewing season right now, so I thought I would list (at a high level) the sorts of problems which I see in papers. Hopefully, this will help us all write better papers. The following flaws are fatal to any paper: Incorrect theorem or lemma statements A typo might be “ok”, if it can be understood. Any theorem or lemma which indicates an incorrect understanding of reality must be rejected. Not doing so would severely harm the integrity of the conference. A paper rejected for this reason must be fixed. Lack of Understanding If a paper is understood by none of the (typically 3) reviewers then it must be rejected for the same reason. This is more controversial than it sounds because there are some people who maximize paper complexity in the hope of impressing the reviewer. The tactic sometimes succeeds with some reviewers (but not with me). As a reviewer, I sometimes get lost for stupid reasons. This is why an anonymized communication channel with the author can
3 0.71779412 202 hunch net-2006-08-10-Precision is not accuracy
Introduction: In my experience, there are two different groups of people who believe the same thing: the mathematics encountered in typical machine learning conference papers is often of questionable value. The two groups who agree on this are applied machine learning people who have given up on math, and mature theoreticians who understand the limits of theory. Partly, this is just a statement about where we are with respect to machine learning. In particular, we have no mechanism capable of generating a prescription for how to solve all learning problems. In the absence of such certainty, people try to come up with formalisms that partially describe and motivate how and why they do things. This is natural and healthy—we might hope that it will eventually lead to just such a mechanism. But, part of this is simply an emphasis on complexity over clarity. A very natural and simple theoretical statement is often obscured by complexifications. Common sources of complexification include:
4 0.67634273 325 hunch net-2008-11-10-ICML Reviewing Criteria
Introduction: Michael Littman and Leon Bottou have decided to use a franchise program chair approach to reviewing at ICML this year. I’ll be one of the area chairs, so I wanted to mention a few things if you are thinking about naming me. I take reviewing seriously. That means papers to be reviewed are read, the implications are considered, and decisions are only made after that. I do my best to be fair, and there are zero subjects that I consider categorical rejects. I don’t consider several arguments for rejection-not-on-the-merits reasonable . I am generally interested in papers that (a) analyze new models of machine learning, (b) provide new algorithms, and (c) show that they work empirically on plausibly real problems. If a paper has the trifecta, I’m particularly interested. With 2 out of 3, I might be interested. I often find papers with only one element harder to accept, including papers with just (a). I’m a bit tough. I rarely jump-up-and-down about a paper, because I b
5 0.67536044 187 hunch net-2006-06-25-Presentation of Proofs is Hard.
Introduction: When presenting part of the Reinforcement Learning theory tutorial at ICML 2006 , I was forcibly reminded of this. There are several difficulties. When creating the presentation, the correct level of detail is tricky. With too much detail, the proof takes too much time and people may be lost to boredom. With too little detail, the steps of the proof involve too-great a jump. This is very difficult to judge. What may be an easy step in the careful thought of a quiet room is not so easy when you are occupied by the process of presentation. What may be easy after having gone over this (and other) proofs is not so easy to follow in the first pass by a viewer. These problems seem only correctable by process of repeated test-and-revise. When presenting the proof, simply speaking with sufficient precision is substantially harder than in normal conversation (where precision is not so critical). Practice can help here. When presenting the proof, going at the right p
6 0.65420818 454 hunch net-2012-01-30-ICML Posters and Scope
7 0.64640564 162 hunch net-2006-03-09-Use of Notation
8 0.6371156 194 hunch net-2006-07-11-New Models
9 0.63075662 233 hunch net-2007-02-16-The Forgetting
10 0.60749274 304 hunch net-2008-06-27-Reviewing Horror Stories
11 0.60521352 38 hunch net-2005-03-09-Bad Reviewing
12 0.59806359 256 hunch net-2007-07-20-Motivation should be the Responsibility of the Reviewer
13 0.5975818 98 hunch net-2005-07-27-Not goal metrics
14 0.596811 57 hunch net-2005-04-16-Which Assumptions are Reasonable?
15 0.59369755 30 hunch net-2005-02-25-Why Papers?
16 0.59230876 126 hunch net-2005-10-26-Fallback Analysis is a Secret to Useful Algorithms
17 0.5905633 320 hunch net-2008-10-14-Who is Responsible for a Bad Review?
18 0.57720178 221 hunch net-2006-12-04-Structural Problems in NIPS Decision Making
19 0.57518578 318 hunch net-2008-09-26-The SODA Program Committee
20 0.57170773 188 hunch net-2006-06-30-ICML papers
topicId topicWeight
[(4, 0.025), (27, 0.185), (38, 0.092), (51, 0.021), (53, 0.075), (55, 0.129), (56, 0.025), (79, 0.223), (89, 0.016), (94, 0.087), (95, 0.049)]
simIndex simValue blogId blogTitle
1 0.91456628 162 hunch net-2006-03-09-Use of Notation
Introduction: For most people, a mathematical notation is like a language: you learn it and stick with it. For people doing mathematical research, however, this is not enough: they must design new notations for new problems. The design of good notation is both hard and worthwhile since a bad initial notation can retard a line of research greatly. Before we had mathematical notation, equations were all written out in language. Since words have multiple meanings and variable precedences, long equations written out in language can be extraordinarily difficult and sometimes fundamentally ambiguous. A good representative example of this is the legalese in the tax code. Since we want greater precision and clarity, we adopt mathematical notation. One fundamental thing to understand about mathematical notation, is that humans as logic verifiers, are barely capable. This is the fundamental reason why one notation can be much better than another. This observation is easier to miss than you might
2 0.90380454 248 hunch net-2007-06-19-How is Compressed Sensing going to change Machine Learning ?
Introduction: Compressed Sensing (CS) is a new framework developed by Emmanuel Candes , Terry Tao and David Donoho . To summarize, if you acquire a signal in some basis that is incoherent with the basis in which you know the signal to be sparse in, it is very likely you will be able to reconstruct the signal from these incoherent projections. Terry Tao, the recent Fields medalist , does a very nice job at explaining the framework here . He goes further in the theory description in this post where he mentions the central issue of the Uniform Uncertainty Principle. It so happens that random projections are on average incoherent, within the UUP meaning, with most known basis (sines, polynomials, splines, wavelets, curvelets …) and are therefore an ideal basis for Compressed Sensing. [ For more in-depth information on the subject, the Rice group has done a very good job at providing a central library of papers relevant to the growing subject: http://www.dsp.ece.rice.edu/cs/ ] The Machine
same-blog 3 0.88149261 204 hunch net-2006-08-28-Learning Theory standards for NIPS 2006
Introduction: Bob Williamson and I are the learning theory PC members at NIPS this year. This is some attempt to state the standards and tests I applied to the papers. I think it is a good idea to talk about this for two reasons: Making community standards a matter of public record seems healthy. It give us a chance to debate what is and is not the right standard. It might even give us a bit more consistency across the years. It may save us all time. There are a number of papers submitted which just aren’t there yet. Avoiding submitting is the right decision in this case. There are several criteria for judging a paper. All of these were active this year. Some criteria are uncontroversial while others may be so. The paper must have a theorem establishing something new for which it is possible to derive high confidence in the correctness of the results. A surprising number of papers fail this test. This criteria seems essential to the definition of “theory”. Missing theo
4 0.8651554 423 hunch net-2011-02-02-User preferences for search engines
Introduction: I want to comment on the “Bing copies Google” discussion here , here , and here , because there are data-related issues which the general public may not understand, and some of the framing seems substantially misleading to me. As a not-distant-outsider, let me mention the sources of bias I may have. I work at Yahoo! , which has started using Bing . This might predispose me towards Bing, but on the other hand I’m still at Yahoo!, and have been using Linux exclusively as an OS for many years, including even a couple minor kernel patches. And, on the gripping hand , I’ve spent quite a bit of time thinking about the basic principles of incorporating user feedback in machine learning . Also note, this post is not related to official Yahoo! policy, it’s just my personal view. The issue Google engineers inserted synthetic responses to synthetic queries on google.com, then executed the synthetic searches on google.com using Internet Explorer with the Bing toolbar and later
5 0.84637856 354 hunch net-2009-05-17-Server Update
Introduction: The hunch.net server has been updated. I’ve taken the opportunity to upgrade the version of wordpress which caused cascading changes. Old threaded comments are now flattened. The system we used to use ( Brian’s threaded comments ) appears incompatible with the new threading system built into wordpress. I haven’t yet figured out a workaround. I setup a feedburner account . I added an RSS aggregator for both Machine Learning and other research blogs that I like to follow. This is something that I’ve wanted to do for awhile. Many other minor changes in font and format, with some help from Alina . If you have any suggestions for site tweaks, please speak up.
6 0.83374017 27 hunch net-2005-02-23-Problem: Reinforcement Learning with Classification
7 0.73922741 437 hunch net-2011-07-10-ICML 2011 and the future
8 0.7284711 403 hunch net-2010-07-18-ICML & COLT 2010
9 0.72631252 297 hunch net-2008-04-22-Taking the next step
10 0.72547412 202 hunch net-2006-08-10-Precision is not accuracy
11 0.72442418 256 hunch net-2007-07-20-Motivation should be the Responsibility of the Reviewer
12 0.72385532 95 hunch net-2005-07-14-What Learning Theory might do
13 0.72342563 343 hunch net-2009-02-18-Decision by Vetocracy
14 0.72254241 464 hunch net-2012-05-03-Microsoft Research, New York City
15 0.72216058 75 hunch net-2005-05-28-Running A Machine Learning Summer School
16 0.72206175 44 hunch net-2005-03-21-Research Styles in Machine Learning
17 0.71973038 19 hunch net-2005-02-14-Clever Methods of Overfitting
18 0.71950573 286 hunch net-2008-01-25-Turing’s Club for Machine Learning
19 0.71931136 454 hunch net-2012-01-30-ICML Posters and Scope
20 0.71857876 452 hunch net-2012-01-04-Why ICML? and the summer conferences