hunch_net hunch_net-2009 hunch_net-2009-343 knowledge-graph by maker-knowledge-mining
Source: html
Introduction: Few would mistake the process of academic paper review for a fair process, but sometimes the unfairness seems particularly striking. This is most easily seen by comparison: Paper Banditron Offset Tree Notes Problem Scope Multiclass problems where only the loss of one choice can be probed. Strictly greater: Cost sensitive multiclass problems where only the loss of one choice can be probed. Often generalizations don’t matter. That’s not the case here, since every plausible application I’ve thought of involves loss functions substantially different from 0/1. What’s new Analysis and Experiments Algorithm, Analysis, and Experiments As far as I know, the essence of the more general problem was first stated and analyzed with the EXP4 algorithm (page 16) (1998). It’s also the time horizon 1 simplification of the Reinforcement Learning setting for the random trajectory method (page 15) (2002). The Banditron algorithm itself is functionally identi
sentIndex sentText sentNum sentScore
1 Instead, the reviewer asserts that learning reductions are bogus because for an alternative notion of learning reduction, made up by the reviewer, an obviously useless approach yields a factor of 2 regret bound. [sent-29, score-0.469]
2 The first time we encountered this review, we assumed the reviewer was just cranky that day—maybe we weren’t quite clear enough in explaining everything as it’s always difficult to get every detail clear in new subject matter. [sent-31, score-0.681]
3 Sometimes when a reviewer is cranky, they change their mind after the authors respond, or perhaps later, or perhaps never but you get a new set of reviewers the next time. [sent-33, score-0.533]
4 If we are generous to the reviewer, and taking into account the fact that learning reduction analysis is a relatively new form of analysis, the fear that because an alternative notion of reduction is vacuous our notion of reduction might also be vacuous isn’t too outlandish. [sent-35, score-0.907]
5 This lower bound conclusively proves that our notion of learning reduction is not vacuous as is the reviewer’s notion of learning reduction. [sent-37, score-0.54]
6 Despite pointing out the lower bound quite explicitly, the reviewer simply ignored it. [sent-39, score-0.481]
7 Some reviewer is bidding for the paper with the intent to torpedo review it. [sent-41, score-0.77]
8 They may offend the reviewer they invited to review and personally know. [sent-55, score-0.492]
9 This reviewer has a demonstrated capability to sabotage the review process at ICML and NIPS and a demonstrated willingness to continue doing so indefinitely. [sent-76, score-0.693]
10 A clever abusive reviewer can sabotage perhaps 5 papers per conference (out of 8 reviewed), while maintaining a typical average score. [sent-80, score-0.551]
11 If, for example, 5% of reviewers are willing to abuse the process this way and there are 100 reviewers, every paper must survive 5 vetoes. [sent-82, score-0.698]
12 Similarly, the reviewing style in theory conferences seems better—the set of bidders for any paper is substantially smaller, implying papers must survive fewer vetos. [sent-91, score-0.445]
13 This decision making process can be modeled as a group of n decision makers, each of which has the opportunity to veto any action. [sent-92, score-0.534]
14 When n is relatively small, this decision making process might work ok, depending on the decision makers, but as n grows larger, it’s difficult to imagine a worse decision making process. [sent-93, score-0.506]
15 An essential force driving vetocracy creation is a desire to offload responsibility for decisions, so there is no clear decision maker. [sent-100, score-0.574]
16 A large organization not deciding by vetocracy must have a very different structure, with clearly dilineated responsibility. [sent-101, score-0.416]
17 There are one or two workshop chairs who are responsible for selecting amongst workshop proposals, after which the content of the workshop is entirely up to the workshop organizers. [sent-103, score-0.496]
18 Every agent except the reviewers is often known by the authors, and the reviewers don’t act as additional vetoers in nearly as strong a manner as reviewers with the opportunity to bid. [sent-112, score-0.729]
19 It only takes one to turn strong paper into a years-long odyssey, so public discussion of research directions and topics in a vetocracy is akin to voluntarily wearing a “kick me” sign. [sent-116, score-0.581]
20 Since a vetocracy creates a substantial disincentive to discuss research directions online, we can expect that communities sticking with decision by vetocracy to be at a substantial disadvantage. [sent-123, score-0.955]
wordName wordTfidf (topN-words)
[('reviewer', 0.309), ('vetocracy', 0.288), ('reviewers', 0.224), ('review', 0.183), ('offset', 0.154), ('analysis', 0.149), ('survive', 0.144), ('veto', 0.128), ('workshop', 0.124), ('responsibility', 0.12), ('sabotage', 0.108), ('reduction', 0.108), ('paper', 0.107), ('bidding', 0.099), ('tree', 0.099), ('lower', 0.099), ('decision', 0.099), ('banditron', 0.096), ('cranky', 0.096), ('vacuous', 0.096), ('process', 0.093), ('notion', 0.082), ('rejected', 0.081), ('substantial', 0.079), ('alternative', 0.078), ('papers', 0.074), ('bound', 0.073), ('inoffensive', 0.072), ('intent', 0.072), ('surviving', 0.072), ('enough', 0.072), ('algorithm', 0.072), ('maybe', 0.071), ('every', 0.07), ('organization', 0.068), ('clear', 0.067), ('nips', 0.065), ('discussion', 0.064), ('research', 0.063), ('must', 0.06), ('reviewing', 0.06), ('per', 0.06), ('editor', 0.059), ('directions', 0.059), ('making', 0.058), ('later', 0.057), ('opportunity', 0.057), ('shouldn', 0.056), ('organizations', 0.056), ('quantification', 0.056)]
simIndex simValue blogId blogTitle
same-blog 1 0.99999845 343 hunch net-2009-02-18-Decision by Vetocracy
Introduction: Few would mistake the process of academic paper review for a fair process, but sometimes the unfairness seems particularly striking. This is most easily seen by comparison: Paper Banditron Offset Tree Notes Problem Scope Multiclass problems where only the loss of one choice can be probed. Strictly greater: Cost sensitive multiclass problems where only the loss of one choice can be probed. Often generalizations don’t matter. That’s not the case here, since every plausible application I’ve thought of involves loss functions substantially different from 0/1. What’s new Analysis and Experiments Algorithm, Analysis, and Experiments As far as I know, the essence of the more general problem was first stated and analyzed with the EXP4 algorithm (page 16) (1998). It’s also the time horizon 1 simplification of the Reinforcement Learning setting for the random trajectory method (page 15) (2002). The Banditron algorithm itself is functionally identi
2 0.36079019 484 hunch net-2013-06-16-Representative Reviewing
Introduction: When thinking about how best to review papers, it seems helpful to have some conception of what good reviewing is. As far as I can tell, this is almost always only discussed in the specific context of a paper (i.e. your rejected paper), or at most an area (i.e. what a “good paper” looks like for that area) rather than general principles. Neither individual papers or areas are sufficiently general for a large conference—every paper differs in the details, and what if you want to build a new area and/or cross areas? An unavoidable reason for reviewing is that the community of research is too large. In particular, it is not possible for a researcher to read every paper which someone thinks might be of interest. This reason for reviewing exists independent of constraints on rooms or scheduling formats of individual conferences. Indeed, history suggests that physical constraints are relatively meaningless over the long term — growing conferences simply use more rooms and/or change fo
3 0.32725072 315 hunch net-2008-09-03-Bidding Problems
Introduction: One way that many conferences in machine learning assign reviewers to papers is via bidding, which has steps something like: Invite people to review Accept papers Reviewers look at title and abstract and state the papers they are interested in reviewing. Some massaging happens, but reviewers often get approximately the papers they bid for. At the ICML business meeting, Andrew McCallum suggested getting rid of bidding for papers. A couple reasons were given: Privacy The title and abstract of the entire set of papers is visible to every participating reviewer. Some authors might be uncomfortable about this for submitted papers. I’m not sympathetic to this reason: the point of submitting a paper to review is to publish it, so the value (if any) of not publishing a part of it a little bit earlier seems limited. Cliques A bidding system is gameable. If you have 3 buddies and you inform each other of your submissions, you can each bid for your friend’s papers a
4 0.3035692 320 hunch net-2008-10-14-Who is Responsible for a Bad Review?
Introduction: Although I’m greatly interested in machine learning, I think it must be admitted that there is a large amount of low quality logic being used in reviews. The problem is bad enough that sometimes I wonder if the Byzantine generals limit has been exceeded. For example, I’ve seen recent reviews where the given reasons for rejecting are: [ NIPS ] Theorem A is uninteresting because Theorem B is uninteresting. [ UAI ] When you learn by memorization, the problem addressed is trivial. [NIPS] The proof is in the appendix. [NIPS] This has been done before. (… but not giving any relevant citations) Just for the record I want to point out what’s wrong with these reviews. A future world in which such reasons never come up again would be great, but I’m sure these errors will be committed many times more in the future. This is nonsense. A theorem should be evaluated based on it’s merits, rather than the merits of another theorem. Learning by memorization requires an expon
5 0.28989938 38 hunch net-2005-03-09-Bad Reviewing
Introduction: This is a difficult subject to talk about for many reasons, but a discussion may be helpful. Bad reviewing is a problem in academia. The first step in understanding this is admitting to the problem, so here is a short list of examples of bad reviewing. Reviewer disbelieves theorem proof (ICML), or disbelieve theorem with a trivially false counterexample. (COLT) Reviewer internally swaps quantifiers in a theorem, concludes it has been done before and is trivial. (NIPS) Reviewer believes a technique will not work despite experimental validation. (COLT) Reviewers fail to notice flaw in theorem statement (CRYPTO). Reviewer erroneously claims that it has been done before (NIPS, SODA, JMLR)—(complete with references!) Reviewer inverts the message of a paper and concludes it says nothing important. (NIPS*2) Reviewer fails to distinguish between a DAG and a tree (SODA). Reviewer is enthusiastic about paper but clearly does not understand (ICML). Reviewer erroneously
6 0.2602939 304 hunch net-2008-06-27-Reviewing Horror Stories
7 0.25072443 311 hunch net-2008-07-26-Compositional Machine Learning Algorithm Design
8 0.23386025 40 hunch net-2005-03-13-Avoiding Bad Reviewing
9 0.23366518 437 hunch net-2011-07-10-ICML 2011 and the future
10 0.22904649 454 hunch net-2012-01-30-ICML Posters and Scope
11 0.22328602 333 hunch net-2008-12-27-Adversarial Academia
12 0.22062926 207 hunch net-2006-09-12-Incentive Compatible Reviewing
13 0.20960131 395 hunch net-2010-04-26-Compassionate Reviewing
14 0.20238566 116 hunch net-2005-09-30-Research in conferences
15 0.20183015 461 hunch net-2012-04-09-ICML author feedback is open
16 0.19851312 485 hunch net-2013-06-29-The Benefits of Double-Blind Review
17 0.19045663 98 hunch net-2005-07-27-Not goal metrics
18 0.18824072 318 hunch net-2008-09-26-The SODA Program Committee
19 0.18783528 463 hunch net-2012-05-02-ICML: Behind the Scenes
20 0.18490705 466 hunch net-2012-06-05-ICML acceptance statistics
topicId topicWeight
[(0, 0.441), (1, -0.156), (2, 0.251), (3, 0.058), (4, 0.033), (5, 0.133), (6, 0.104), (7, -0.002), (8, -0.033), (9, 0.083), (10, 0.039), (11, -0.018), (12, 0.094), (13, -0.014), (14, -0.105), (15, 0.033), (16, -0.009), (17, 0.044), (18, -0.04), (19, -0.001), (20, -0.046), (21, -0.042), (22, -0.076), (23, -0.023), (24, -0.04), (25, 0.055), (26, -0.026), (27, -0.035), (28, 0.072), (29, -0.003), (30, 0.019), (31, -0.025), (32, 0.007), (33, 0.047), (34, -0.052), (35, -0.017), (36, -0.017), (37, 0.003), (38, -0.031), (39, 0.014), (40, -0.037), (41, 0.03), (42, -0.027), (43, -0.043), (44, 0.068), (45, -0.051), (46, -0.001), (47, 0.01), (48, -0.028), (49, 0.001)]
simIndex simValue blogId blogTitle
same-blog 1 0.97308075 343 hunch net-2009-02-18-Decision by Vetocracy
Introduction: Few would mistake the process of academic paper review for a fair process, but sometimes the unfairness seems particularly striking. This is most easily seen by comparison: Paper Banditron Offset Tree Notes Problem Scope Multiclass problems where only the loss of one choice can be probed. Strictly greater: Cost sensitive multiclass problems where only the loss of one choice can be probed. Often generalizations don’t matter. That’s not the case here, since every plausible application I’ve thought of involves loss functions substantially different from 0/1. What’s new Analysis and Experiments Algorithm, Analysis, and Experiments As far as I know, the essence of the more general problem was first stated and analyzed with the EXP4 algorithm (page 16) (1998). It’s also the time horizon 1 simplification of the Reinforcement Learning setting for the random trajectory method (page 15) (2002). The Banditron algorithm itself is functionally identi
2 0.86834502 484 hunch net-2013-06-16-Representative Reviewing
Introduction: When thinking about how best to review papers, it seems helpful to have some conception of what good reviewing is. As far as I can tell, this is almost always only discussed in the specific context of a paper (i.e. your rejected paper), or at most an area (i.e. what a “good paper” looks like for that area) rather than general principles. Neither individual papers or areas are sufficiently general for a large conference—every paper differs in the details, and what if you want to build a new area and/or cross areas? An unavoidable reason for reviewing is that the community of research is too large. In particular, it is not possible for a researcher to read every paper which someone thinks might be of interest. This reason for reviewing exists independent of constraints on rooms or scheduling formats of individual conferences. Indeed, history suggests that physical constraints are relatively meaningless over the long term — growing conferences simply use more rooms and/or change fo
3 0.8502962 320 hunch net-2008-10-14-Who is Responsible for a Bad Review?
Introduction: Although I’m greatly interested in machine learning, I think it must be admitted that there is a large amount of low quality logic being used in reviews. The problem is bad enough that sometimes I wonder if the Byzantine generals limit has been exceeded. For example, I’ve seen recent reviews where the given reasons for rejecting are: [ NIPS ] Theorem A is uninteresting because Theorem B is uninteresting. [ UAI ] When you learn by memorization, the problem addressed is trivial. [NIPS] The proof is in the appendix. [NIPS] This has been done before. (… but not giving any relevant citations) Just for the record I want to point out what’s wrong with these reviews. A future world in which such reasons never come up again would be great, but I’m sure these errors will be committed many times more in the future. This is nonsense. A theorem should be evaluated based on it’s merits, rather than the merits of another theorem. Learning by memorization requires an expon
4 0.84162951 315 hunch net-2008-09-03-Bidding Problems
Introduction: One way that many conferences in machine learning assign reviewers to papers is via bidding, which has steps something like: Invite people to review Accept papers Reviewers look at title and abstract and state the papers they are interested in reviewing. Some massaging happens, but reviewers often get approximately the papers they bid for. At the ICML business meeting, Andrew McCallum suggested getting rid of bidding for papers. A couple reasons were given: Privacy The title and abstract of the entire set of papers is visible to every participating reviewer. Some authors might be uncomfortable about this for submitted papers. I’m not sympathetic to this reason: the point of submitting a paper to review is to publish it, so the value (if any) of not publishing a part of it a little bit earlier seems limited. Cliques A bidding system is gameable. If you have 3 buddies and you inform each other of your submissions, you can each bid for your friend’s papers a
5 0.82918668 38 hunch net-2005-03-09-Bad Reviewing
Introduction: This is a difficult subject to talk about for many reasons, but a discussion may be helpful. Bad reviewing is a problem in academia. The first step in understanding this is admitting to the problem, so here is a short list of examples of bad reviewing. Reviewer disbelieves theorem proof (ICML), or disbelieve theorem with a trivially false counterexample. (COLT) Reviewer internally swaps quantifiers in a theorem, concludes it has been done before and is trivial. (NIPS) Reviewer believes a technique will not work despite experimental validation. (COLT) Reviewers fail to notice flaw in theorem statement (CRYPTO). Reviewer erroneously claims that it has been done before (NIPS, SODA, JMLR)—(complete with references!) Reviewer inverts the message of a paper and concludes it says nothing important. (NIPS*2) Reviewer fails to distinguish between a DAG and a tree (SODA). Reviewer is enthusiastic about paper but clearly does not understand (ICML). Reviewer erroneously
6 0.81050044 461 hunch net-2012-04-09-ICML author feedback is open
7 0.80816364 333 hunch net-2008-12-27-Adversarial Academia
8 0.8070876 207 hunch net-2006-09-12-Incentive Compatible Reviewing
9 0.78866839 485 hunch net-2013-06-29-The Benefits of Double-Blind Review
10 0.7755425 221 hunch net-2006-12-04-Structural Problems in NIPS Decision Making
11 0.76963758 463 hunch net-2012-05-02-ICML: Behind the Scenes
12 0.76807344 304 hunch net-2008-06-27-Reviewing Horror Stories
13 0.75144082 437 hunch net-2011-07-10-ICML 2011 and the future
14 0.74601394 40 hunch net-2005-03-13-Avoiding Bad Reviewing
15 0.73926538 52 hunch net-2005-04-04-Grounds for Rejection
16 0.73170602 98 hunch net-2005-07-27-Not goal metrics
17 0.72221005 318 hunch net-2008-09-26-The SODA Program Committee
18 0.70913094 395 hunch net-2010-04-26-Compassionate Reviewing
19 0.67475164 116 hunch net-2005-09-30-Research in conferences
20 0.67249829 238 hunch net-2007-04-13-What to do with an unreasonable conditional accept
topicId topicWeight
[(3, 0.025), (10, 0.051), (12, 0.016), (16, 0.022), (27, 0.242), (38, 0.058), (48, 0.016), (53, 0.048), (55, 0.126), (64, 0.121), (92, 0.016), (94, 0.085), (95, 0.064)]
simIndex simValue blogId blogTitle
1 0.97352445 420 hunch net-2010-12-26-NIPS 2010
Introduction: I enjoyed attending NIPS this year, with several things interesting me. For the conference itself: Peter Welinder , Steve Branson , Serge Belongie , and Pietro Perona , The Multidimensional Wisdom of Crowds . This paper is about using mechanical turk to get label information, with results superior to a majority vote approach. David McAllester , Tamir Hazan , and Joseph Keshet Direct Loss Minimization for Structured Prediction . This is about another technique for directly optimizing the loss in structured prediction, with an application to speech recognition. Mohammad Saberian and Nuno Vasconcelos Boosting Classifier Cascades . This is about an algorithm for simultaneously optimizing loss and computation in a classifier cascade construction. There were several other papers on cascades which are worth looking at if interested. Alan Fern and Prasad Tadepalli , A Computational Decision Theory for Interactive Assistants . This paper carves out some
2 0.95688111 18 hunch net-2005-02-12-ROC vs. Accuracy vs. AROC
Introduction: Foster Provost and I discussed the merits of ROC curves vs. accuracy estimation. Here is a quick summary of our discussion. The “Receiver Operating Characteristic” (ROC) curve is an alternative to accuracy for the evaluation of learning algorithms on natural datasets. The ROC curve is a curve and not a single number statistic. In particular, this means that the comparison of two algorithms on a dataset does not always produce an obvious order. Accuracy (= 1 – error rate) is a standard method used to evaluate learning algorithms. It is a single-number summary of performance. AROC is the area under the ROC curve. It is a single number summary of performance. The comparison of these metrics is a subtle affair, because in machine learning, they are compared on different natural datasets. This makes some sense if we accept the hypothesis “Performance on past learning problems (roughly) predicts performance on future learning problems.” The ROC vs. accuracy discussion is o
same-blog 3 0.95416695 343 hunch net-2009-02-18-Decision by Vetocracy
Introduction: Few would mistake the process of academic paper review for a fair process, but sometimes the unfairness seems particularly striking. This is most easily seen by comparison: Paper Banditron Offset Tree Notes Problem Scope Multiclass problems where only the loss of one choice can be probed. Strictly greater: Cost sensitive multiclass problems where only the loss of one choice can be probed. Often generalizations don’t matter. That’s not the case here, since every plausible application I’ve thought of involves loss functions substantially different from 0/1. What’s new Analysis and Experiments Algorithm, Analysis, and Experiments As far as I know, the essence of the more general problem was first stated and analyzed with the EXP4 algorithm (page 16) (1998). It’s also the time horizon 1 simplification of the Reinforcement Learning setting for the random trajectory method (page 15) (2002). The Banditron algorithm itself is functionally identi
4 0.95370221 277 hunch net-2007-12-12-Workshop Summary—Principles of Learning Problem Design
Introduction: This is a summary of the workshop on Learning Problem Design which Alina and I ran at NIPS this year. The first question many people have is “What is learning problem design?” This workshop is about admitting that solving learning problems does not start with labeled data, but rather somewhere before. When humans are hired to produce labels, this is usually not a serious problem because you can tell them precisely what semantics you want the labels to have, and we can fix some set of features in advance. However, when other methods are used this becomes more problematic. This focus is important for Machine Learning because there are very large quantities of data which are not labeled by a hired human. The title of the workshop was a bit ambitious, because a workshop is not long enough to synthesize a diversity of approaches into a coherent set of principles. For me, the posters at the end of the workshop were quite helpful in getting approaches to gel. Here are some an
5 0.9406023 210 hunch net-2006-09-28-Programming Languages for Machine Learning Implementations
Introduction: Machine learning algorithms have a much better chance of being widely adopted if they are implemented in some easy-to-use code. There are several important concerns associated with machine learning which stress programming languages on the ease-of-use vs. speed frontier. Speed The rate at which data sources are growing seems to be outstripping the rate at which computational power is growing, so it is important that we be able to eak out every bit of computational power. Garbage collected languages ( java , ocaml , perl and python ) often have several issues here. Garbage collection often implies that floating point numbers are “boxed”: every float is represented by a pointer to a float. Boxing can cause an order of magnitude slowdown because an extra nonlocalized memory reference is made, and accesses to main memory can are many CPU cycles long. Garbage collection often implies that considerably more memory is used than is necessary. This has a variable effect. I
6 0.9063946 360 hunch net-2009-06-15-In Active Learning, the question changes
7 0.90505207 194 hunch net-2006-07-11-New Models
8 0.90289009 437 hunch net-2011-07-10-ICML 2011 and the future
9 0.89955497 424 hunch net-2011-02-17-What does Watson mean?
10 0.89936453 442 hunch net-2011-08-20-The Large Scale Learning Survey Tutorial
11 0.89924926 155 hunch net-2006-02-07-Pittsburgh Mind Reading Competition
12 0.89832008 320 hunch net-2008-10-14-Who is Responsible for a Bad Review?
13 0.89760292 51 hunch net-2005-04-01-The Producer-Consumer Model of Research
14 0.89683336 406 hunch net-2010-08-22-KDD 2010
15 0.89617711 371 hunch net-2009-09-21-Netflix finishes (and starts)
16 0.89555001 132 hunch net-2005-11-26-The Design of an Optimal Research Environment
17 0.89498633 95 hunch net-2005-07-14-What Learning Theory might do
18 0.8948586 351 hunch net-2009-05-02-Wielding a New Abstraction
19 0.89291823 454 hunch net-2012-01-30-ICML Posters and Scope
20 0.89226937 435 hunch net-2011-05-16-Research Directions for Machine Learning and Algorithms