hunch_net hunch_net-2008 hunch_net-2008-304 knowledge-graph by maker-knowledge-mining
Source: html
Introduction: Essentially everyone who writes research papers suffers rejections. They always sting immediately, but upon further reflection many of these rejections come to seem reasonable. Maybe the equations had too many typos or maybe the topic just isn’t as important as was originally thought. A few rejections do not come to seem acceptable, and these form the basis of reviewing horror stories, a great material for conversations. I’ve decided to share three of mine, now all safely a bit distant in the past. Prediction Theory for Classification Tutorial . This is a tutorial about tight sample complexity bounds for classification that I submitted to JMLR . The first decision I heard was a reject which appeared quite unjust to me—for example one of the reviewers appeared to claim that all the content was in standard statistics books. Upon further inquiry, several citations were given, none of which actually covered the content. Later, I was shocked to hear the paper was accepted. App
sentIndex sentText sentNum sentScore
1 A few rejections do not come to seem acceptable, and these form the basis of reviewing horror stories, a great material for conversations. [sent-4, score-0.561]
2 The first decision I heard was a reject which appeared quite unjust to me—for example one of the reviewers appeared to claim that all the content was in standard statistics books. [sent-8, score-0.705]
3 This paper was the first one to give a datastructure for nearest neighbor search for an arbitrary metric which both (a) took logarithmic time under dimensionality constraint and (b) always required space competitive with brute force nearest neighbor search. [sent-13, score-0.556]
4 The cover tree paper suffered a triple rejection, the last one of which seems particularly poor to me. [sent-15, score-0.588]
5 We were rather confused, so we emailed the program chair asking if the decision was right and if so whether there was any more information we could get. [sent-21, score-0.495]
6 This paper shows that learning how to predict which of a pair of items is better strongly transfers to optimizing a ranking loss, in contrast to (for example) simply predicting a score and ordering according to predicted score. [sent-25, score-0.466]
7 Based upon what we could make out from a statement by the program committee, the logic of this decision is mostly kindly describable as badly flawed—somehow they confused the algorithm, the problem, and the analysis into a mess. [sent-28, score-0.648]
8 (A bit of disclosure: I was on the program committee at NIPS that year, although obviously not involved in the decision on this paper. [sent-30, score-0.647]
9 ) In all cases where a rejection occurs, the default presumption is that the correct decision was made because most of the time a good (or at least reasonable) decision was made. [sent-31, score-0.635]
10 The tutorial paper is fairly widely cited ( Google scholar places it 8th amongst my papers), and I continue to find it useful material for a lecture when teaching a class. [sent-33, score-0.726]
11 The cover tree is also fairly widely cited, and I know from various emails and download counts that it is used by several people. [sent-34, score-0.37]
12 One of the reasons you hear for why a paper was rejected and then accepted is that the paper improved in the meantime. [sent-38, score-0.606]
13 For example, at a conference, one reviewer by bidding preference, one reviewer by area chair, and one reviewer by another area chair or the program chair’s choice might reduce variance. [sent-46, score-0.919]
14 The standard at NIPS was to have author feedback when the ranking paper was submitted. [sent-48, score-0.756]
15 In effect, the standard was not followed for the ranking paper, and it’s easy to imagine this making a substantial difference given how badly flawed the basis of rejection was. [sent-49, score-0.974]
16 It is also easy to imagine that author feedback might have made a difference in the tutorial rejection, as the reviewer was wrong (author feedback was not the standard then). [sent-50, score-0.783]
17 It’s helpful to relate the basis of decision by the program committee, especially when it is not summarized in the reviews. [sent-52, score-0.517]
18 The cover tree case was one of the things which led me to add summaries to some of the NIPS papers when I was on the program committee, and I am committed to doing the same for SODA papers I’m reviewing this year. [sent-53, score-0.758]
19 Not having a summary saves the program committee the embarassment of accidentally admitting mistakes, but it is badly disrepectful of the authors and generally promotes misunderstanding. [sent-54, score-0.578]
20 I suspect that the time crunch of the NIPS program committee meeting was a contributing factor in the ranking paper case. [sent-57, score-0.847]
wordName wordTfidf (topN-words)
[('ranking', 0.253), ('paper', 0.213), ('committee', 0.209), ('tutorial', 0.192), ('decision', 0.191), ('rejection', 0.173), ('program', 0.172), ('basis', 0.154), ('reviewer', 0.142), ('stories', 0.134), ('chair', 0.132), ('cover', 0.13), ('horror', 0.126), ('tree', 0.115), ('nips', 0.113), ('cited', 0.112), ('submitted', 0.111), ('badly', 0.104), ('reviewing', 0.104), ('feedback', 0.102), ('accepted', 0.101), ('author', 0.1), ('rejections', 0.093), ('accidentally', 0.093), ('confused', 0.093), ('soda', 0.09), ('upon', 0.088), ('standard', 0.088), ('papers', 0.087), ('got', 0.086), ('flawed', 0.086), ('appeared', 0.084), ('material', 0.084), ('cases', 0.08), ('hear', 0.079), ('reviewers', 0.078), ('involved', 0.075), ('neighbor', 0.072), ('nearest', 0.068), ('widely', 0.068), ('poor', 0.067), ('later', 0.066), ('one', 0.063), ('maybe', 0.062), ('statistics', 0.06), ('substantial', 0.059), ('fairly', 0.057), ('claim', 0.057), ('difference', 0.057), ('back', 0.057)]
simIndex simValue blogId blogTitle
same-blog 1 0.99999988 304 hunch net-2008-06-27-Reviewing Horror Stories
Introduction: Essentially everyone who writes research papers suffers rejections. They always sting immediately, but upon further reflection many of these rejections come to seem reasonable. Maybe the equations had too many typos or maybe the topic just isn’t as important as was originally thought. A few rejections do not come to seem acceptable, and these form the basis of reviewing horror stories, a great material for conversations. I’ve decided to share three of mine, now all safely a bit distant in the past. Prediction Theory for Classification Tutorial . This is a tutorial about tight sample complexity bounds for classification that I submitted to JMLR . The first decision I heard was a reject which appeared quite unjust to me—for example one of the reviewers appeared to claim that all the content was in standard statistics books. Upon further inquiry, several citations were given, none of which actually covered the content. Later, I was shocked to hear the paper was accepted. App
2 0.27355349 484 hunch net-2013-06-16-Representative Reviewing
Introduction: When thinking about how best to review papers, it seems helpful to have some conception of what good reviewing is. As far as I can tell, this is almost always only discussed in the specific context of a paper (i.e. your rejected paper), or at most an area (i.e. what a “good paper” looks like for that area) rather than general principles. Neither individual papers or areas are sufficiently general for a large conference—every paper differs in the details, and what if you want to build a new area and/or cross areas? An unavoidable reason for reviewing is that the community of research is too large. In particular, it is not possible for a researcher to read every paper which someone thinks might be of interest. This reason for reviewing exists independent of constraints on rooms or scheduling formats of individual conferences. Indeed, history suggests that physical constraints are relatively meaningless over the long term — growing conferences simply use more rooms and/or change fo
3 0.2602939 343 hunch net-2009-02-18-Decision by Vetocracy
Introduction: Few would mistake the process of academic paper review for a fair process, but sometimes the unfairness seems particularly striking. This is most easily seen by comparison: Paper Banditron Offset Tree Notes Problem Scope Multiclass problems where only the loss of one choice can be probed. Strictly greater: Cost sensitive multiclass problems where only the loss of one choice can be probed. Often generalizations don’t matter. That’s not the case here, since every plausible application I’ve thought of involves loss functions substantially different from 0/1. What’s new Analysis and Experiments Algorithm, Analysis, and Experiments As far as I know, the essence of the more general problem was first stated and analyzed with the EXP4 algorithm (page 16) (1998). It’s also the time horizon 1 simplification of the Reinforcement Learning setting for the random trajectory method (page 15) (2002). The Banditron algorithm itself is functionally identi
4 0.25549945 320 hunch net-2008-10-14-Who is Responsible for a Bad Review?
Introduction: Although I’m greatly interested in machine learning, I think it must be admitted that there is a large amount of low quality logic being used in reviews. The problem is bad enough that sometimes I wonder if the Byzantine generals limit has been exceeded. For example, I’ve seen recent reviews where the given reasons for rejecting are: [ NIPS ] Theorem A is uninteresting because Theorem B is uninteresting. [ UAI ] When you learn by memorization, the problem addressed is trivial. [NIPS] The proof is in the appendix. [NIPS] This has been done before. (… but not giving any relevant citations) Just for the record I want to point out what’s wrong with these reviews. A future world in which such reasons never come up again would be great, but I’m sure these errors will be committed many times more in the future. This is nonsense. A theorem should be evaluated based on it’s merits, rather than the merits of another theorem. Learning by memorization requires an expon
5 0.24343754 318 hunch net-2008-09-26-The SODA Program Committee
Introduction: Claire asked me to be on the SODA program committee this year, which was quite a bit of work. I had a relatively light load—merely 49 theory papers. Many of these papers were not on subjects that I was expert about, so (as is common for theory conferences) I found various reviewers that I trusted to help review the papers. I ended up reviewing about 1/3 personally. There were a couple instances where I ended up overruling a subreviewer whose logic seemed off, but otherwise I generally let their reviews stand. There are some differences in standards for paper reviews between the machine learning and theory communities. In machine learning it is expected that a review be detailed, while in the theory community this is often not the case. Every paper given to me ended up with a review varying between somewhat and very detailed. I’m sure not every author was happy with the outcome. While we did our best to make good decisions, they were difficult decisions to make. For exam
6 0.23999655 453 hunch net-2012-01-28-Why COLT?
7 0.22247857 437 hunch net-2011-07-10-ICML 2011 and the future
8 0.22150099 38 hunch net-2005-03-09-Bad Reviewing
9 0.1952371 315 hunch net-2008-09-03-Bidding Problems
10 0.17745242 454 hunch net-2012-01-30-ICML Posters and Scope
11 0.17637828 221 hunch net-2006-12-04-Structural Problems in NIPS Decision Making
12 0.1726833 40 hunch net-2005-03-13-Avoiding Bad Reviewing
13 0.16981611 207 hunch net-2006-09-12-Incentive Compatible Reviewing
14 0.16894238 116 hunch net-2005-09-30-Research in conferences
15 0.16602641 461 hunch net-2012-04-09-ICML author feedback is open
16 0.15915306 466 hunch net-2012-06-05-ICML acceptance statistics
17 0.154569 333 hunch net-2008-12-27-Adversarial Academia
18 0.14869365 395 hunch net-2010-04-26-Compassionate Reviewing
19 0.14729328 288 hunch net-2008-02-10-Complexity Illness
20 0.14590794 403 hunch net-2010-07-18-ICML & COLT 2010
topicId topicWeight
[(0, 0.326), (1, -0.18), (2, 0.245), (3, 0.04), (4, 0.062), (5, 0.101), (6, -0.028), (7, 0.003), (8, -0.052), (9, -0.045), (10, 0.054), (11, 0.019), (12, 0.009), (13, -0.037), (14, -0.072), (15, -0.002), (16, 0.059), (17, 0.03), (18, 0.031), (19, -0.047), (20, 0.024), (21, 0.075), (22, 0.027), (23, -0.037), (24, -0.063), (25, 0.023), (26, 0.052), (27, -0.047), (28, 0.08), (29, -0.017), (30, -0.019), (31, -0.062), (32, -0.009), (33, 0.009), (34, -0.005), (35, 0.054), (36, 0.061), (37, 0.125), (38, 0.071), (39, 0.023), (40, 0.041), (41, 0.013), (42, -0.028), (43, 0.024), (44, -0.012), (45, 0.016), (46, 0.034), (47, -0.003), (48, 0.039), (49, 0.009)]
simIndex simValue blogId blogTitle
same-blog 1 0.98245406 304 hunch net-2008-06-27-Reviewing Horror Stories
Introduction: Essentially everyone who writes research papers suffers rejections. They always sting immediately, but upon further reflection many of these rejections come to seem reasonable. Maybe the equations had too many typos or maybe the topic just isn’t as important as was originally thought. A few rejections do not come to seem acceptable, and these form the basis of reviewing horror stories, a great material for conversations. I’ve decided to share three of mine, now all safely a bit distant in the past. Prediction Theory for Classification Tutorial . This is a tutorial about tight sample complexity bounds for classification that I submitted to JMLR . The first decision I heard was a reject which appeared quite unjust to me—for example one of the reviewers appeared to claim that all the content was in standard statistics books. Upon further inquiry, several citations were given, none of which actually covered the content. Later, I was shocked to hear the paper was accepted. App
2 0.88117385 320 hunch net-2008-10-14-Who is Responsible for a Bad Review?
Introduction: Although I’m greatly interested in machine learning, I think it must be admitted that there is a large amount of low quality logic being used in reviews. The problem is bad enough that sometimes I wonder if the Byzantine generals limit has been exceeded. For example, I’ve seen recent reviews where the given reasons for rejecting are: [ NIPS ] Theorem A is uninteresting because Theorem B is uninteresting. [ UAI ] When you learn by memorization, the problem addressed is trivial. [NIPS] The proof is in the appendix. [NIPS] This has been done before. (… but not giving any relevant citations) Just for the record I want to point out what’s wrong with these reviews. A future world in which such reasons never come up again would be great, but I’m sure these errors will be committed many times more in the future. This is nonsense. A theorem should be evaluated based on it’s merits, rather than the merits of another theorem. Learning by memorization requires an expon
3 0.87874615 221 hunch net-2006-12-04-Structural Problems in NIPS Decision Making
Introduction: This is a very difficult post to write, because it is about a perenially touchy subject. Nevertheless, it is an important one which needs to be thought about carefully. There are a few things which should be understood: The system is changing and responsive. We-the-authors are we-the-reviewers, we-the-PC, and even we-the-NIPS-board. NIPS has implemented ‘secondary program chairs’, ‘author response’, and ‘double blind reviewing’ in the last few years to help with the decision process, and more changes may happen in the future. Agreement creates a perception of correctness. When any PC meets and makes a group decision about a paper, there is a strong tendency for the reinforcement inherent in a group decision to create the perception of correctness. For the many people who have been on the NIPS PC it’s reasonable to entertain a healthy skepticism in the face of this reinforcing certainty. This post is about structural problems. What problems arise because of the structure
4 0.85304564 318 hunch net-2008-09-26-The SODA Program Committee
Introduction: Claire asked me to be on the SODA program committee this year, which was quite a bit of work. I had a relatively light load—merely 49 theory papers. Many of these papers were not on subjects that I was expert about, so (as is common for theory conferences) I found various reviewers that I trusted to help review the papers. I ended up reviewing about 1/3 personally. There were a couple instances where I ended up overruling a subreviewer whose logic seemed off, but otherwise I generally let their reviews stand. There are some differences in standards for paper reviews between the machine learning and theory communities. In machine learning it is expected that a review be detailed, while in the theory community this is often not the case. Every paper given to me ended up with a review varying between somewhat and very detailed. I’m sure not every author was happy with the outcome. While we did our best to make good decisions, they were difficult decisions to make. For exam
5 0.84646434 463 hunch net-2012-05-02-ICML: Behind the Scenes
Introduction: This is a rather long post, detailing the ICML 2012 review process. The goal is to make the process more transparent, help authors understand how we came to a decision, and discuss the strengths and weaknesses of this process for future conference organizers. Microsoft’s Conference Management Toolkit (CMT) We chose to use CMT over other conference management software mainly because of its rich toolkit. The interface is sub-optimal (to say the least!) but it has extensive capabilities (to handle bids, author response, resubmissions, etc.), good import/export mechanisms (to process the data elsewhere), excellent technical support (to answer late night emails, add new functionalities). Overall, it was the right choice, although we hope a designer will look at that interface sometime soon! Toronto Matching System (TMS) TMS is now being used by many major conferences in our field (including NIPS and UAI). It is an automated system (developed by Laurent Charlin and Rich Ze
6 0.82614368 484 hunch net-2013-06-16-Representative Reviewing
7 0.82023931 461 hunch net-2012-04-09-ICML author feedback is open
8 0.81633443 315 hunch net-2008-09-03-Bidding Problems
9 0.81232649 238 hunch net-2007-04-13-What to do with an unreasonable conditional accept
10 0.77052414 38 hunch net-2005-03-09-Bad Reviewing
11 0.75702596 343 hunch net-2009-02-18-Decision by Vetocracy
12 0.74786144 437 hunch net-2011-07-10-ICML 2011 and the future
13 0.74495345 180 hunch net-2006-05-21-NIPS paper evaluation criteria
14 0.73315966 52 hunch net-2005-04-04-Grounds for Rejection
15 0.69011474 325 hunch net-2008-11-10-ICML Reviewing Criteria
16 0.67523509 207 hunch net-2006-09-12-Incentive Compatible Reviewing
17 0.67059171 453 hunch net-2012-01-28-Why COLT?
18 0.66030186 333 hunch net-2008-12-27-Adversarial Academia
19 0.65447539 288 hunch net-2008-02-10-Complexity Illness
20 0.64879656 40 hunch net-2005-03-13-Avoiding Bad Reviewing
topicId topicWeight
[(3, 0.018), (9, 0.01), (10, 0.019), (27, 0.549), (38, 0.041), (51, 0.016), (53, 0.035), (55, 0.117), (90, 0.013), (94, 0.058), (95, 0.036)]
simIndex simValue blogId blogTitle
same-blog 1 0.99851543 304 hunch net-2008-06-27-Reviewing Horror Stories
Introduction: Essentially everyone who writes research papers suffers rejections. They always sting immediately, but upon further reflection many of these rejections come to seem reasonable. Maybe the equations had too many typos or maybe the topic just isn’t as important as was originally thought. A few rejections do not come to seem acceptable, and these form the basis of reviewing horror stories, a great material for conversations. I’ve decided to share three of mine, now all safely a bit distant in the past. Prediction Theory for Classification Tutorial . This is a tutorial about tight sample complexity bounds for classification that I submitted to JMLR . The first decision I heard was a reject which appeared quite unjust to me—for example one of the reviewers appeared to claim that all the content was in standard statistics books. Upon further inquiry, several citations were given, none of which actually covered the content. Later, I was shocked to hear the paper was accepted. App
2 0.99315268 352 hunch net-2009-05-06-Machine Learning to AI
Introduction: I recently had fun discussions with both Vikash Mansinghka and Thomas Breuel about approaching AI with machine learning. The general interest in taking a crack at AI with machine learning seems to be rising on many fronts including DARPA . As a matter of history, there was a great deal of interest in AI which died down before I began research. There remain many projects and conferences spawned in this earlier AI wave, as well as a good bit of experience about what did not work, or at least did not work yet. Here are a few examples of failure modes that people seem to run into: Supply/Product confusion . Sometimes we think “Intelligences use X, so I’ll create X and have an Intelligence.” An example of this is the Cyc Project which inspires some people as “intelligences use ontologies, so I’ll create an ontology and a system using it to have an Intelligence.” The flaw here is that Intelligences create ontologies, which they use, and without the ability to create ont
3 0.99036807 196 hunch net-2006-07-13-Regression vs. Classification as a Primitive
Introduction: For learning reductions we have been concentrating on reducing various complex learning problems to binary classification. This choice needs to be actively questioned, because it was not carefully considered. Binary clasification is learning a classifier c:X -> {0,1} so as to minimize the probability of being wrong, Pr x,y~D (c(x) y) . The primary alternative candidate seems to be squared error regression. In squared error regression, you learn a regressor s:X -> [0,1] so as to minimize squared error, E x,y~D (s(x)-y) 2 . It is difficult to judge one primitive against another. The judgement must at least partially be made on nontheoretical grounds because (essentially) we are evaluating a choice between two axioms/assumptions. These two primitives are significantly related. Classification can be reduced to regression in the obvious way: you use the regressor to predict D(y=1|x) , then threshold at 0.5 . For this simple reduction a squared error regret of r
4 0.98638815 45 hunch net-2005-03-22-Active learning
Introduction: Often, unlabeled data is easy to come by but labels are expensive. For instance, if you’re building a speech recognizer, it’s easy enough to get raw speech samples — just walk around with a microphone — but labeling even one of these samples is a tedious process in which a human must examine the speech signal and carefully segment it into phonemes. In the field of active learning, the goal is as usual to construct an accurate classifier, but the labels of the data points are initially hidden and there is a charge for each label you want revealed. The hope is that by intelligent adaptive querying, you can get away with significantly fewer labels than you would need in a regular supervised learning framework. Here’s an example. Suppose the data lie on the real line, and the classifiers are simple thresholding functions, H = {h w }: h w (x) = 1 if x > w, and 0 otherwise. VC theory tells us that if the underlying distribution P can be classified perfectly by some hypothesis in H (
5 0.986301 288 hunch net-2008-02-10-Complexity Illness
Introduction: One of the enduring stereotypes of academia is that people spend a great deal of intelligence, time, and effort finding complexity rather than simplicity. This is at least anecdotally true in my experience. Math++ Several people have found that adding useless math makes their paper more publishable as evidenced by a reject-add-accept sequence. 8 page minimum Who submitted a paper to ICML violating the 8 page minimum? Every author fears that the reviewers won’t take their work seriously unless the allowed length is fully used. The best minimum violation I know is Adam ‘s paper at SODA on generating random factored numbers , but this is deeply exceptional. It’s a fair bet that 90% of papers submitted are exactly at the page limit. We could imagine that this is because papers naturally take more space, but few people seem to be clamoring for more space. Journalong Has anyone been asked to review a 100 page journal paper? I have. Journal papers can be nice, becaus
6 0.98492527 172 hunch net-2006-04-14-JMLR is a success
7 0.98252887 341 hunch net-2009-02-04-Optimal Proxy Loss for Classification
8 0.98018277 9 hunch net-2005-02-01-Watchword: Loss
9 0.97950208 308 hunch net-2008-07-06-To Dual or Not
10 0.97921735 245 hunch net-2007-05-12-Loss Function Semantics
11 0.97859144 400 hunch net-2010-06-13-The Good News on Exploration and Learning
12 0.97845566 247 hunch net-2007-06-14-Interesting Papers at COLT 2007
13 0.97636771 293 hunch net-2008-03-23-Interactive Machine Learning
14 0.97346932 274 hunch net-2007-11-28-Computational Consequences of Classification
15 0.97052801 220 hunch net-2006-11-27-Continuizing Solutions
16 0.96600896 378 hunch net-2009-11-15-The Other Online Learning
17 0.96464002 166 hunch net-2006-03-24-NLPers
18 0.96464002 246 hunch net-2007-06-13-Not Posting
19 0.96464002 418 hunch net-2010-12-02-Traffic Prediction Problem
20 0.96250951 230 hunch net-2007-02-02-Thoughts regarding “Is machine learning different from statistics?”