hunch_net hunch_net-2008 hunch_net-2008-320 knowledge-graph by maker-knowledge-mining
Source: html
Introduction: Although I’m greatly interested in machine learning, I think it must be admitted that there is a large amount of low quality logic being used in reviews. The problem is bad enough that sometimes I wonder if the Byzantine generals limit has been exceeded. For example, I’ve seen recent reviews where the given reasons for rejecting are: [ NIPS ] Theorem A is uninteresting because Theorem B is uninteresting. [ UAI ] When you learn by memorization, the problem addressed is trivial. [NIPS] The proof is in the appendix. [NIPS] This has been done before. (… but not giving any relevant citations) Just for the record I want to point out what’s wrong with these reviews. A future world in which such reasons never come up again would be great, but I’m sure these errors will be committed many times more in the future. This is nonsense. A theorem should be evaluated based on it’s merits, rather than the merits of another theorem. Learning by memorization requires an expon
sentIndex sentText sentNum sentScore
1 For example, I’ve seen recent reviews where the given reasons for rejecting are: [ NIPS ] Theorem A is uninteresting because Theorem B is uninteresting. [sent-3, score-0.422]
2 A theorem should be evaluated based on it’s merits, rather than the merits of another theorem. [sent-10, score-0.27]
3 Learning by memorization requires an exponentially larger sample complexity than many other common approaches that often work well. [sent-11, score-0.401]
4 Consequently, what is possible under memorization does not have any substantial bearing on common practice or what might be useful in the future. [sent-12, score-0.3]
5 Every time I’ve seen a review (as an author or a fellow reviewer) where such claims are made without a concrete citation, they are false. [sent-17, score-0.39]
6 A softer version of (4) is when someone is cranky because their own paper wasn’t cited. [sent-19, score-0.277]
7 This avoids creating the extra work (for authors and reviewers) of yet another paper resubmission, and reasonable authors do take such suggestions into account. [sent-21, score-0.275]
8 While these are all instances in the last year, my experience after interacting with NIPS for almost a decade is that the average quality of reviews is particularly low there—in many instances reviewers clearly don’t read the papers before writing the review. [sent-23, score-0.954]
9 Furthermore, such low quality reviews are often the deciding factor for the paper decision. [sent-24, score-0.645]
10 Blaming the reviewer seems to be the easy solution for a bad review, but a bit more thought suggests other possibilities: Area Chair In some conferences an “area chair” or “senior PC” makes or effectively makes the decision on a paper. [sent-25, score-0.296]
11 In general, I’m not a fan of activist area chairs, but when a reviewer isn’t thinking well, I think it is appropriate to step in. [sent-26, score-0.478]
12 In my experience, many Area Chairs are eager to avoid any substantial controversy, and there is a general tendency to believe that something must be wrong with a paper that has a negative review, even if it isn’t what was actually pointed out. [sent-28, score-0.325]
13 For example, I know David McAllester understands that learning by memorization is a bogus reference point, and probably he was just too busy to really digest the reviews. [sent-31, score-0.35]
14 However, a Program Chair is responsible for finding appropriate reviewers for papers, and doing so (or not) has a huge impact on whether a paper is accepted. [sent-32, score-0.378]
15 Not surprisingly, if a paper about the sample complexity of learning is routed to people who have never seen a proof involving sample complexity before, the reviews tend to be spuriously negative (and the paper unread). [sent-33, score-1.187]
16 Author A reviewer might blame an author, if it turns out later that the reasons given in the review for rejection were bogus. [sent-34, score-0.4]
17 Culture A conference has a culture associated with it that is driven by the people who keep coming back. [sent-36, score-0.312]
18 If in this culture it is considered ok to do all the reviews on the last day, it’s unsurprising to see reviews lacking critical thought that could be written without reading the paper. [sent-37, score-1.008]
19 Similarly, it’s unsurprising to see little critical thought at the area chair level, or in the routing of papers to reviewers. [sent-38, score-0.664]
20 This answer is pretty convincing: it explains why low quality reviews keep happening year after year at a conference. [sent-39, score-0.741]
wordName wordTfidf (topN-words)
[('reviews', 0.256), ('chair', 0.237), ('culture', 0.233), ('memorization', 0.233), ('area', 0.164), ('paper', 0.143), ('reviewer', 0.136), ('nips', 0.135), ('low', 0.125), ('reviewers', 0.124), ('absurd', 0.124), ('quality', 0.121), ('review', 0.117), ('bogus', 0.117), ('negative', 0.115), ('chairs', 0.115), ('appropriate', 0.111), ('unsurprising', 0.111), ('proof', 0.108), ('merits', 0.107), ('concrete', 0.097), ('theorem', 0.096), ('citations', 0.094), ('sample', 0.091), ('author', 0.09), ('instances', 0.088), ('seen', 0.086), ('thought', 0.083), ('year', 0.08), ('reasons', 0.08), ('experience', 0.079), ('keep', 0.079), ('complexity', 0.077), ('program', 0.077), ('conferences', 0.077), ('writing', 0.073), ('isn', 0.071), ('matter', 0.07), ('critical', 0.069), ('fan', 0.067), ('evaluated', 0.067), ('corner', 0.067), ('eager', 0.067), ('softer', 0.067), ('bearing', 0.067), ('blame', 0.067), ('cranky', 0.067), ('effecting', 0.067), ('resubmission', 0.067), ('authors', 0.066)]
simIndex simValue blogId blogTitle
same-blog 1 0.9999997 320 hunch net-2008-10-14-Who is Responsible for a Bad Review?
Introduction: Although I’m greatly interested in machine learning, I think it must be admitted that there is a large amount of low quality logic being used in reviews. The problem is bad enough that sometimes I wonder if the Byzantine generals limit has been exceeded. For example, I’ve seen recent reviews where the given reasons for rejecting are: [ NIPS ] Theorem A is uninteresting because Theorem B is uninteresting. [ UAI ] When you learn by memorization, the problem addressed is trivial. [NIPS] The proof is in the appendix. [NIPS] This has been done before. (… but not giving any relevant citations) Just for the record I want to point out what’s wrong with these reviews. A future world in which such reasons never come up again would be great, but I’m sure these errors will be committed many times more in the future. This is nonsense. A theorem should be evaluated based on it’s merits, rather than the merits of another theorem. Learning by memorization requires an expon
2 0.39773861 484 hunch net-2013-06-16-Representative Reviewing
Introduction: When thinking about how best to review papers, it seems helpful to have some conception of what good reviewing is. As far as I can tell, this is almost always only discussed in the specific context of a paper (i.e. your rejected paper), or at most an area (i.e. what a “good paper” looks like for that area) rather than general principles. Neither individual papers or areas are sufficiently general for a large conference—every paper differs in the details, and what if you want to build a new area and/or cross areas? An unavoidable reason for reviewing is that the community of research is too large. In particular, it is not possible for a researcher to read every paper which someone thinks might be of interest. This reason for reviewing exists independent of constraints on rooms or scheduling formats of individual conferences. Indeed, history suggests that physical constraints are relatively meaningless over the long term — growing conferences simply use more rooms and/or change fo
3 0.3035692 343 hunch net-2009-02-18-Decision by Vetocracy
Introduction: Few would mistake the process of academic paper review for a fair process, but sometimes the unfairness seems particularly striking. This is most easily seen by comparison: Paper Banditron Offset Tree Notes Problem Scope Multiclass problems where only the loss of one choice can be probed. Strictly greater: Cost sensitive multiclass problems where only the loss of one choice can be probed. Often generalizations don’t matter. That’s not the case here, since every plausible application I’ve thought of involves loss functions substantially different from 0/1. What’s new Analysis and Experiments Algorithm, Analysis, and Experiments As far as I know, the essence of the more general problem was first stated and analyzed with the EXP4 algorithm (page 16) (1998). It’s also the time horizon 1 simplification of the Reinforcement Learning setting for the random trajectory method (page 15) (2002). The Banditron algorithm itself is functionally identi
4 0.29013142 461 hunch net-2012-04-09-ICML author feedback is open
Introduction: as of last night, late. When the reviewing deadline passed Wednesday night 15% of reviews were still missing, much higher than I expected. Between late reviews coming in, ACs working overtime through the weekend, and people willing to help in the pinch another ~390 reviews came in, reducing the missing mass to 0.2%. Nailing that last bit and a similar quantity of papers with uniformly low confidence reviews is what remains to be done in terms of basic reviews. We are trying to make all of those happen this week so authors have some chance to respond. I was surprised by the quantity of late reviews, and I think that’s an area where ICML needs to improve in future years. Good reviews are not done in a rush—they are done by setting aside time (like an afternoon), and carefully reading the paper while thinking about implications. Many reviewers do this well but a significant minority aren’t good at scheduling their personal time. In this situation there are several ways to fail:
5 0.25549945 304 hunch net-2008-06-27-Reviewing Horror Stories
Introduction: Essentially everyone who writes research papers suffers rejections. They always sting immediately, but upon further reflection many of these rejections come to seem reasonable. Maybe the equations had too many typos or maybe the topic just isn’t as important as was originally thought. A few rejections do not come to seem acceptable, and these form the basis of reviewing horror stories, a great material for conversations. I’ve decided to share three of mine, now all safely a bit distant in the past. Prediction Theory for Classification Tutorial . This is a tutorial about tight sample complexity bounds for classification that I submitted to JMLR . The first decision I heard was a reject which appeared quite unjust to me—for example one of the reviewers appeared to claim that all the content was in standard statistics books. Upon further inquiry, several citations were given, none of which actually covered the content. Later, I was shocked to hear the paper was accepted. App
6 0.24947953 437 hunch net-2011-07-10-ICML 2011 and the future
7 0.24801515 40 hunch net-2005-03-13-Avoiding Bad Reviewing
8 0.24312755 318 hunch net-2008-09-26-The SODA Program Committee
9 0.23742361 315 hunch net-2008-09-03-Bidding Problems
10 0.23255721 38 hunch net-2005-03-09-Bad Reviewing
11 0.20634365 116 hunch net-2005-09-30-Research in conferences
12 0.20198883 207 hunch net-2006-09-12-Incentive Compatible Reviewing
13 0.19305907 453 hunch net-2012-01-28-Why COLT?
14 0.19005296 333 hunch net-2008-12-27-Adversarial Academia
15 0.18893285 454 hunch net-2012-01-30-ICML Posters and Scope
16 0.18503121 98 hunch net-2005-07-27-Not goal metrics
17 0.18437251 221 hunch net-2006-12-04-Structural Problems in NIPS Decision Making
18 0.17430295 463 hunch net-2012-05-02-ICML: Behind the Scenes
19 0.1687243 468 hunch net-2012-06-29-ICML survey and comments
20 0.15815605 238 hunch net-2007-04-13-What to do with an unreasonable conditional accept
topicId topicWeight
[(0, 0.323), (1, -0.24), (2, 0.293), (3, 0.121), (4, 0.045), (5, 0.129), (6, 0.024), (7, 0.04), (8, -0.024), (9, -0.02), (10, 0.029), (11, -0.053), (12, 0.082), (13, -0.031), (14, -0.082), (15, -0.028), (16, 0.054), (17, 0.037), (18, 0.033), (19, -0.025), (20, -0.026), (21, 0.071), (22, 0.013), (23, -0.047), (24, -0.073), (25, 0.057), (26, 0.02), (27, -0.065), (28, 0.086), (29, -0.001), (30, 0.026), (31, -0.042), (32, -0.049), (33, 0.049), (34, -0.051), (35, 0.024), (36, 0.018), (37, 0.048), (38, -0.032), (39, 0.014), (40, 0.01), (41, -0.025), (42, 0.036), (43, -0.031), (44, -0.024), (45, -0.035), (46, -0.072), (47, -0.0), (48, 0.021), (49, -0.06)]
simIndex simValue blogId blogTitle
same-blog 1 0.98290718 320 hunch net-2008-10-14-Who is Responsible for a Bad Review?
Introduction: Although I’m greatly interested in machine learning, I think it must be admitted that there is a large amount of low quality logic being used in reviews. The problem is bad enough that sometimes I wonder if the Byzantine generals limit has been exceeded. For example, I’ve seen recent reviews where the given reasons for rejecting are: [ NIPS ] Theorem A is uninteresting because Theorem B is uninteresting. [ UAI ] When you learn by memorization, the problem addressed is trivial. [NIPS] The proof is in the appendix. [NIPS] This has been done before. (… but not giving any relevant citations) Just for the record I want to point out what’s wrong with these reviews. A future world in which such reasons never come up again would be great, but I’m sure these errors will be committed many times more in the future. This is nonsense. A theorem should be evaluated based on it’s merits, rather than the merits of another theorem. Learning by memorization requires an expon
2 0.93465728 484 hunch net-2013-06-16-Representative Reviewing
Introduction: When thinking about how best to review papers, it seems helpful to have some conception of what good reviewing is. As far as I can tell, this is almost always only discussed in the specific context of a paper (i.e. your rejected paper), or at most an area (i.e. what a “good paper” looks like for that area) rather than general principles. Neither individual papers or areas are sufficiently general for a large conference—every paper differs in the details, and what if you want to build a new area and/or cross areas? An unavoidable reason for reviewing is that the community of research is too large. In particular, it is not possible for a researcher to read every paper which someone thinks might be of interest. This reason for reviewing exists independent of constraints on rooms or scheduling formats of individual conferences. Indeed, history suggests that physical constraints are relatively meaningless over the long term — growing conferences simply use more rooms and/or change fo
3 0.90398222 38 hunch net-2005-03-09-Bad Reviewing
Introduction: This is a difficult subject to talk about for many reasons, but a discussion may be helpful. Bad reviewing is a problem in academia. The first step in understanding this is admitting to the problem, so here is a short list of examples of bad reviewing. Reviewer disbelieves theorem proof (ICML), or disbelieve theorem with a trivially false counterexample. (COLT) Reviewer internally swaps quantifiers in a theorem, concludes it has been done before and is trivial. (NIPS) Reviewer believes a technique will not work despite experimental validation. (COLT) Reviewers fail to notice flaw in theorem statement (CRYPTO). Reviewer erroneously claims that it has been done before (NIPS, SODA, JMLR)—(complete with references!) Reviewer inverts the message of a paper and concludes it says nothing important. (NIPS*2) Reviewer fails to distinguish between a DAG and a tree (SODA). Reviewer is enthusiastic about paper but clearly does not understand (ICML). Reviewer erroneously
4 0.90389198 315 hunch net-2008-09-03-Bidding Problems
Introduction: One way that many conferences in machine learning assign reviewers to papers is via bidding, which has steps something like: Invite people to review Accept papers Reviewers look at title and abstract and state the papers they are interested in reviewing. Some massaging happens, but reviewers often get approximately the papers they bid for. At the ICML business meeting, Andrew McCallum suggested getting rid of bidding for papers. A couple reasons were given: Privacy The title and abstract of the entire set of papers is visible to every participating reviewer. Some authors might be uncomfortable about this for submitted papers. I’m not sympathetic to this reason: the point of submitting a paper to review is to publish it, so the value (if any) of not publishing a part of it a little bit earlier seems limited. Cliques A bidding system is gameable. If you have 3 buddies and you inform each other of your submissions, you can each bid for your friend’s papers a
5 0.89963949 461 hunch net-2012-04-09-ICML author feedback is open
Introduction: as of last night, late. When the reviewing deadline passed Wednesday night 15% of reviews were still missing, much higher than I expected. Between late reviews coming in, ACs working overtime through the weekend, and people willing to help in the pinch another ~390 reviews came in, reducing the missing mass to 0.2%. Nailing that last bit and a similar quantity of papers with uniformly low confidence reviews is what remains to be done in terms of basic reviews. We are trying to make all of those happen this week so authors have some chance to respond. I was surprised by the quantity of late reviews, and I think that’s an area where ICML needs to improve in future years. Good reviews are not done in a rush—they are done by setting aside time (like an afternoon), and carefully reading the paper while thinking about implications. Many reviewers do this well but a significant minority aren’t good at scheduling their personal time. In this situation there are several ways to fail:
6 0.87148976 221 hunch net-2006-12-04-Structural Problems in NIPS Decision Making
7 0.86620754 463 hunch net-2012-05-02-ICML: Behind the Scenes
8 0.85623437 304 hunch net-2008-06-27-Reviewing Horror Stories
9 0.8142733 207 hunch net-2006-09-12-Incentive Compatible Reviewing
10 0.8104198 343 hunch net-2009-02-18-Decision by Vetocracy
11 0.80465466 318 hunch net-2008-09-26-The SODA Program Committee
12 0.79653227 238 hunch net-2007-04-13-What to do with an unreasonable conditional accept
13 0.78506696 40 hunch net-2005-03-13-Avoiding Bad Reviewing
14 0.75161546 52 hunch net-2005-04-04-Grounds for Rejection
15 0.72775638 437 hunch net-2011-07-10-ICML 2011 and the future
16 0.71240449 333 hunch net-2008-12-27-Adversarial Academia
17 0.70722419 485 hunch net-2013-06-29-The Benefits of Double-Blind Review
18 0.70261395 98 hunch net-2005-07-27-Not goal metrics
19 0.66122371 395 hunch net-2010-04-26-Compassionate Reviewing
20 0.64705884 288 hunch net-2008-02-10-Complexity Illness
topicId topicWeight
[(1, 0.012), (3, 0.049), (10, 0.038), (27, 0.24), (38, 0.053), (47, 0.19), (48, 0.014), (53, 0.043), (55, 0.138), (92, 0.015), (94, 0.092), (95, 0.035)]
simIndex simValue blogId blogTitle
1 0.93643236 329 hunch net-2008-11-28-A Bumper Crop of Machine Learning Graduates
Introduction: My impression is that this is a particularly strong year for machine learning graduates. Here’s my short list of the strong graduates I know. Analpha (for perversity’s sake) by last name: Jenn Wortmann . When Jenn visited us for the summer, she had one , two , three , four papers. That is typical—she’s smart, capable, and follows up many directions of research. I believe approximately all of her many papers are on different subjects. Ruslan Salakhutdinov . A Science paper on bijective dimensionality reduction , mastered and improved on deep belief nets which seems like an important flavor of nonlinear learning, and in my experience he’s very fast, capable and creative at problem solving. Marc’Aurelio Ranzato . I haven’t spoken with Marc very much, but he had a great visit at Yahoo! this summer, and has an impressive portfolio of applications and improvements on convolutional neural networks and other deep learning algorithms. Lihong Li . Lihong developed the
same-blog 2 0.92259431 320 hunch net-2008-10-14-Who is Responsible for a Bad Review?
Introduction: Although I’m greatly interested in machine learning, I think it must be admitted that there is a large amount of low quality logic being used in reviews. The problem is bad enough that sometimes I wonder if the Byzantine generals limit has been exceeded. For example, I’ve seen recent reviews where the given reasons for rejecting are: [ NIPS ] Theorem A is uninteresting because Theorem B is uninteresting. [ UAI ] When you learn by memorization, the problem addressed is trivial. [NIPS] The proof is in the appendix. [NIPS] This has been done before. (… but not giving any relevant citations) Just for the record I want to point out what’s wrong with these reviews. A future world in which such reasons never come up again would be great, but I’m sure these errors will be committed many times more in the future. This is nonsense. A theorem should be evaluated based on it’s merits, rather than the merits of another theorem. Learning by memorization requires an expon
3 0.89057463 36 hunch net-2005-03-05-Funding Research
Introduction: The funding of research (and machine learning research) is an issue which seems to have become more significant in the United States over the last decade. The word “research” is applied broadly here to science, mathematics, and engineering. There are two essential difficulties with funding research: Longshot Paying a researcher is often a big gamble. Most research projects don’t pan out, but a few big payoffs can make it all worthwhile. Information Only Much of research is about finding the right way to think about or do something. The Longshot difficulty means that there is high variance in payoffs. This can be compensated for by funding many different research projects, reducing variance. The Information-Only difficulty means that it’s hard to extract a profit directly from many types of research, so companies have difficulty justifying basic research. (Patents are a mechanism for doing this. They are often extraordinarily clumsy or simply not applicable.) T
4 0.82526469 343 hunch net-2009-02-18-Decision by Vetocracy
Introduction: Few would mistake the process of academic paper review for a fair process, but sometimes the unfairness seems particularly striking. This is most easily seen by comparison: Paper Banditron Offset Tree Notes Problem Scope Multiclass problems where only the loss of one choice can be probed. Strictly greater: Cost sensitive multiclass problems where only the loss of one choice can be probed. Often generalizations don’t matter. That’s not the case here, since every plausible application I’ve thought of involves loss functions substantially different from 0/1. What’s new Analysis and Experiments Algorithm, Analysis, and Experiments As far as I know, the essence of the more general problem was first stated and analyzed with the EXP4 algorithm (page 16) (1998). It’s also the time horizon 1 simplification of the Reinforcement Learning setting for the random trajectory method (page 15) (2002). The Banditron algorithm itself is functionally identi
5 0.8222822 437 hunch net-2011-07-10-ICML 2011 and the future
Introduction: Unfortunately, I ended up sick for much of this ICML. I did manage to catch one interesting paper: Richard Socher , Cliff Lin , Andrew Y. Ng , and Christopher D. Manning Parsing Natural Scenes and Natural Language with Recursive Neural Networks . I invited Richard to share his list of interesting papers, so hopefully we’ll hear from him soon. In the meantime, Paul and Hal have posted some lists. the future Joelle and I are program chairs for ICML 2012 in Edinburgh , which I previously enjoyed visiting in 2005 . This is a huge responsibility, that we hope to accomplish well. A part of this (perhaps the most fun part), is imagining how we can make ICML better. A key and critical constraint is choosing things that can be accomplished. So far we have: Colocation . The first thing we looked into was potential colocations. We quickly discovered that many other conferences precomitted their location. For the future, getting a colocation with ACL or SIGI
6 0.81717038 484 hunch net-2013-06-16-Representative Reviewing
7 0.81593406 40 hunch net-2005-03-13-Avoiding Bad Reviewing
8 0.81403375 51 hunch net-2005-04-01-The Producer-Consumer Model of Research
9 0.8092218 95 hunch net-2005-07-14-What Learning Theory might do
10 0.80916345 463 hunch net-2012-05-02-ICML: Behind the Scenes
11 0.80755359 204 hunch net-2006-08-28-Learning Theory standards for NIPS 2006
12 0.8058728 454 hunch net-2012-01-30-ICML Posters and Scope
13 0.80569774 325 hunch net-2008-11-10-ICML Reviewing Criteria
14 0.80406249 194 hunch net-2006-07-11-New Models
15 0.80297959 132 hunch net-2005-11-26-The Design of an Optimal Research Environment
16 0.80261821 98 hunch net-2005-07-27-Not goal metrics
17 0.80205047 225 hunch net-2007-01-02-Retrospective
18 0.8003909 44 hunch net-2005-03-21-Research Styles in Machine Learning
19 0.798163 403 hunch net-2010-07-18-ICML & COLT 2010
20 0.79784715 379 hunch net-2009-11-23-ICML 2009 Workshops (and Tutorials)