hunch_net hunch_net-2009 hunch_net-2009-382 knowledge-graph by maker-knowledge-mining
Source: html
Introduction: Yesterday, there was a discussion about future publication models at NIPS . Yann and Zoubin have specific detailed proposals which I’ll add links to when I get them ( Yann’s proposal and Zoubin’s proposal ). What struck me about the discussion is that there are many simultaneous concerns as well as many simultaneous proposals, which makes it difficult to keep all the distinctions straight in a verbal conversation. It also seemed like people were serious enough about this that we may see some real movement. Certainly, my personal experience motivates that as I’ve posted many times about the substantial flaws in our review process, including some very poor personal experiences. Concerns include the following: (Several) Reviewers are overloaded, boosting the noise in decision making. ( Yann ) A new system should run with as little built-in delay and friction to the process of research as possible. ( Hanna Wallach (updated)) Double-blind review is particularly impor
sentIndex sentText sentNum sentScore
1 Yann and Zoubin have specific detailed proposals which I’ll add links to when I get them ( Yann’s proposal and Zoubin’s proposal ). [sent-2, score-0.714]
2 What struck me about the discussion is that there are many simultaneous concerns as well as many simultaneous proposals, which makes it difficult to keep all the distinctions straight in a verbal conversation. [sent-3, score-0.678]
3 Certainly, my personal experience motivates that as I’ve posted many times about the substantial flaws in our review process, including some very poor personal experiences. [sent-5, score-0.497]
4 ( Yann ) A new system should run with as little built-in delay and friction to the process of research as possible. [sent-7, score-0.192]
5 ( Hanna Wallach (updated)) Double-blind review is particularly important for people who are unknown or from an unknown institution. [sent-8, score-0.384]
6 (Several) But, it’s bad to take double blind so seriously as to disallow publishing on arxiv or personal webpages. [sent-9, score-0.724]
7 ( Yann ) And double-blind is bad when it prevents publishing for substantial periods of time. [sent-10, score-0.356]
8 ( Zoubin ) Any new system should appear to outsiders as if it’s the old system, or a journal, because it’s already hard enough to justify CS tenure cases to other disciplines. [sent-12, score-0.259]
9 There were other concerns as well, but these are the ones that I remember. [sent-14, score-0.203]
10 Elements of proposals include: ( Yann ) Everything should go to Arxiv or an arxiv-like system first, as per physics or mathematics. [sent-15, score-0.355]
11 This addresses (1), because it delinks dissemination from review, relieving some of the burden of reviewing. [sent-16, score-0.432]
12 It also addresses (2) since with good authors they can immediately begin building on each other’s work. [sent-17, score-0.422]
13 ( Fernando ) Create a conference coincident journal in which people can publish at any time. [sent-20, score-0.295]
14 It can be done smoothly by allowing submission in either conference deadline mode or journal mode. [sent-22, score-0.291]
15 This proposal addresses (1) by reducing peak demand on reviewing. [sent-23, score-0.669]
16 This addresses (1), by eliminating some concerns for the reviewer. [sent-26, score-0.556]
17 In biology, such a journal exists (pointer updated), because delays were becoming absurd and intolerable. [sent-28, score-0.295]
18 ( Yann ) There should be multiple publishing entities (people or groups of people) that can bless a paper as interesting. [sent-29, score-0.223]
19 There are many other proposal elements (too many for my memory), which hopefully we’ll see in particular proposals. [sent-31, score-0.333]
20 If other people have concrete proposals, now is probably the right time to formalize them. [sent-32, score-0.142]
wordName wordTfidf (topN-words)
[('addresses', 0.353), ('yann', 0.299), ('proposal', 0.243), ('proposals', 0.228), ('journal', 0.222), ('zoubin', 0.206), ('arxiv', 0.206), ('concerns', 0.203), ('publishing', 0.144), ('fernando', 0.131), ('personal', 0.129), ('system', 0.127), ('simultaneous', 0.126), ('updated', 0.118), ('apparently', 0.114), ('unknown', 0.104), ('review', 0.103), ('elements', 0.09), ('blind', 0.09), ('double', 0.085), ('struck', 0.079), ('orthogonal', 0.079), ('straight', 0.079), ('daphne', 0.079), ('entities', 0.079), ('burden', 0.079), ('yesterday', 0.079), ('include', 0.075), ('people', 0.073), ('periods', 0.073), ('conflicts', 0.073), ('biology', 0.073), ('absurd', 0.073), ('motivates', 0.073), ('peak', 0.073), ('cvpr', 0.073), ('bad', 0.07), ('shouldn', 0.069), ('begin', 0.069), ('justify', 0.069), ('prevents', 0.069), ('formalize', 0.069), ('smoothly', 0.069), ('pointer', 0.065), ('correctness', 0.065), ('delay', 0.065), ('distinctions', 0.065), ('overloaded', 0.065), ('tenure', 0.063), ('flaws', 0.063)]
simIndex simValue blogId blogTitle
same-blog 1 1.0000001 382 hunch net-2009-12-09-Future Publication Models @ NIPS
Introduction: Yesterday, there was a discussion about future publication models at NIPS . Yann and Zoubin have specific detailed proposals which I’ll add links to when I get them ( Yann’s proposal and Zoubin’s proposal ). What struck me about the discussion is that there are many simultaneous concerns as well as many simultaneous proposals, which makes it difficult to keep all the distinctions straight in a verbal conversation. It also seemed like people were serious enough about this that we may see some real movement. Certainly, my personal experience motivates that as I’ve posted many times about the substantial flaws in our review process, including some very poor personal experiences. Concerns include the following: (Several) Reviewers are overloaded, boosting the noise in decision making. ( Yann ) A new system should run with as little built-in delay and friction to the process of research as possible. ( Hanna Wallach (updated)) Double-blind review is particularly impor
2 0.15224043 40 hunch net-2005-03-13-Avoiding Bad Reviewing
Introduction: If we accept that bad reviewing often occurs and want to fix it, the question is “how”? Reviewing is done by paper writers just like yourself, so a good proxy for this question is asking “How can I be a better reviewer?” Here are a few things I’ve learned by trial (and error), as a paper writer, and as a reviewer. The secret ingredient is careful thought. There is no good substitution for a deep and careful understanding. Avoid reviewing papers that you feel competitive about. You almost certainly will be asked to review papers that feel competitive if you work on subjects of common interest. But, the feeling of competition can easily lead to bad judgement. If you feel biased for some other reason, then you should avoid reviewing. For example… Feeling angry or threatened by a paper is a form of bias. See above. Double blind yourself (avoid looking at the name even in a single-blind situation). The significant effect of a name you recognize is making you pay close a
3 0.14146233 333 hunch net-2008-12-27-Adversarial Academia
Introduction: One viewpoint on academia is that it is inherently adversarial: there are finite research dollars, positions, and students to work with, implying a zero-sum game between different participants. This is not a viewpoint that I want to promote, as I consider it flawed. However, I know several people believe strongly in this viewpoint, and I have found it to have substantial explanatory power. For example: It explains why your paper was rejected based on poor logic. The reviewer wasn’t concerned with research quality, but rather with rejecting a competitor. It explains why professors rarely work together. The goal of a non-tenured professor (at least) is to get tenure, and a case for tenure comes from a portfolio of work that is undisputably yours. It explains why new research programs are not quickly adopted. Adopting a competitor’s program is impossible, if your career is based on the competitor being wrong. Different academic groups subscribe to the adversarial viewp
4 0.1364153 65 hunch net-2005-05-02-Reviewing techniques for conferences
Introduction: The many reviews following the many paper deadlines are just about over. AAAI and ICML in particular were experimenting with several reviewing techniques. Double Blind: AAAI and ICML were both double blind this year. It seemed (overall) beneficial, but two problems arose. For theoretical papers, with a lot to say, authors often leave out the proofs. This is very hard to cope with under a double blind review because (1) you can not trust the authors got the proof right but (2) a blanket “reject” hits many probably-good papers. Perhaps authors should more strongly favor proof-complete papers sent to double blind conferences. On the author side, double blind reviewing is actually somewhat disruptive to research. In particular, it discourages the author from talking about the subject, which is one of the mechanisms of research. This is not a great drawback, but it is one not previously appreciated. Author feedback: AAAI and ICML did author feedback this year. It seem
5 0.12676543 437 hunch net-2011-07-10-ICML 2011 and the future
Introduction: Unfortunately, I ended up sick for much of this ICML. I did manage to catch one interesting paper: Richard Socher , Cliff Lin , Andrew Y. Ng , and Christopher D. Manning Parsing Natural Scenes and Natural Language with Recursive Neural Networks . I invited Richard to share his list of interesting papers, so hopefully we’ll hear from him soon. In the meantime, Paul and Hal have posted some lists. the future Joelle and I are program chairs for ICML 2012 in Edinburgh , which I previously enjoyed visiting in 2005 . This is a huge responsibility, that we hope to accomplish well. A part of this (perhaps the most fun part), is imagining how we can make ICML better. A key and critical constraint is choosing things that can be accomplished. So far we have: Colocation . The first thing we looked into was potential colocations. We quickly discovered that many other conferences precomitted their location. For the future, getting a colocation with ACL or SIGI
6 0.125531 48 hunch net-2005-03-29-Academic Mechanism Design
7 0.12143148 343 hunch net-2009-02-18-Decision by Vetocracy
8 0.1177882 16 hunch net-2005-02-09-Intuitions from applied learning
9 0.11334945 484 hunch net-2013-06-16-Representative Reviewing
10 0.11303617 395 hunch net-2010-04-26-Compassionate Reviewing
11 0.10755381 485 hunch net-2013-06-29-The Benefits of Double-Blind Review
12 0.10442385 116 hunch net-2005-09-30-Research in conferences
13 0.098745205 472 hunch net-2012-08-27-NYAS ML 2012 and ICML 2013
14 0.091679454 320 hunch net-2008-10-14-Who is Responsible for a Bad Review?
15 0.091382451 461 hunch net-2012-04-09-ICML author feedback is open
16 0.088353172 358 hunch net-2009-06-01-Multitask Poisoning
17 0.087507479 288 hunch net-2008-02-10-Complexity Illness
18 0.084646888 452 hunch net-2012-01-04-Why ICML? and the summer conferences
19 0.084570505 451 hunch net-2011-12-13-Vowpal Wabbit version 6.1 & the NIPS tutorial
20 0.081643514 207 hunch net-2006-09-12-Incentive Compatible Reviewing
topicId topicWeight
[(0, 0.177), (1, -0.119), (2, 0.061), (3, 0.075), (4, -0.017), (5, 0.025), (6, -0.031), (7, 0.01), (8, -0.001), (9, 0.02), (10, -0.084), (11, -0.037), (12, 0.033), (13, -0.003), (14, -0.01), (15, 0.033), (16, -0.027), (17, -0.012), (18, 0.017), (19, 0.013), (20, -0.021), (21, -0.069), (22, -0.022), (23, 0.005), (24, 0.057), (25, -0.015), (26, -0.032), (27, 0.027), (28, -0.053), (29, 0.047), (30, 0.051), (31, -0.015), (32, 0.049), (33, -0.059), (34, -0.033), (35, -0.026), (36, -0.052), (37, -0.045), (38, -0.042), (39, 0.037), (40, 0.023), (41, -0.055), (42, -0.07), (43, -0.044), (44, 0.097), (45, -0.068), (46, 0.086), (47, -0.146), (48, -0.021), (49, 0.075)]
simIndex simValue blogId blogTitle
same-blog 1 0.96255052 382 hunch net-2009-12-09-Future Publication Models @ NIPS
Introduction: Yesterday, there was a discussion about future publication models at NIPS . Yann and Zoubin have specific detailed proposals which I’ll add links to when I get them ( Yann’s proposal and Zoubin’s proposal ). What struck me about the discussion is that there are many simultaneous concerns as well as many simultaneous proposals, which makes it difficult to keep all the distinctions straight in a verbal conversation. It also seemed like people were serious enough about this that we may see some real movement. Certainly, my personal experience motivates that as I’ve posted many times about the substantial flaws in our review process, including some very poor personal experiences. Concerns include the following: (Several) Reviewers are overloaded, boosting the noise in decision making. ( Yann ) A new system should run with as little built-in delay and friction to the process of research as possible. ( Hanna Wallach (updated)) Double-blind review is particularly impor
2 0.66575855 395 hunch net-2010-04-26-Compassionate Reviewing
Introduction: Most long conversations between academics seem to converge on the topic of reviewing where almost no one is happy. A basic question is: Should most people be happy? The case against is straightforward. Anyone who watches the flow of papers realizes that most papers amount to little in the longer term. By it’s nature research is brutal, where the second-best method is worthless, and the second person to discover things typically gets no credit. If you think about this for a moment, it’s very different from most other human endeavors. The second best migrant laborer, construction worker, manager, conductor, quarterback, etc… all can manage quite well. If a reviewer has even a vaguely predictive sense of what’s important in the longer term, then most people submitting papers will be unhappy. But this argument unravels, in my experience. Perhaps half of reviews are thoughtless or simply wrong with a small part being simply malicious. And yet, I’m sure that most reviewers genuine
3 0.61264205 333 hunch net-2008-12-27-Adversarial Academia
Introduction: One viewpoint on academia is that it is inherently adversarial: there are finite research dollars, positions, and students to work with, implying a zero-sum game between different participants. This is not a viewpoint that I want to promote, as I consider it flawed. However, I know several people believe strongly in this viewpoint, and I have found it to have substantial explanatory power. For example: It explains why your paper was rejected based on poor logic. The reviewer wasn’t concerned with research quality, but rather with rejecting a competitor. It explains why professors rarely work together. The goal of a non-tenured professor (at least) is to get tenure, and a case for tenure comes from a portfolio of work that is undisputably yours. It explains why new research programs are not quickly adopted. Adopting a competitor’s program is impossible, if your career is based on the competitor being wrong. Different academic groups subscribe to the adversarial viewp
4 0.60300392 485 hunch net-2013-06-29-The Benefits of Double-Blind Review
Introduction: This post is a (near) transcript of a talk that I gave at the ICML 2013 Workshop on Peer Review and Publishing Models . Although there’s a PDF available on my website , I’ve chosen to post a slightly modified version here as well in order to better facilitate discussion. Disclaimers and Context I want to start with a couple of disclaimers and some context. First, I want to point out that although I’ve read a lot about double-blind review, this isn’t my research area and the research discussed in this post is not my own. As a result, I probably can’t answer super detailed questions about these studies. I also want to note that I’m not opposed to open peer review — I was a free and open source software developer for over ten years and I care a great deal about openness and transparency. Rather, my motivation in writing this post is simply to create awareness of and to initiate discussion about the benefits of double-blind review. Lastly, and most importantly, I think it’s e
5 0.59239131 40 hunch net-2005-03-13-Avoiding Bad Reviewing
Introduction: If we accept that bad reviewing often occurs and want to fix it, the question is “how”? Reviewing is done by paper writers just like yourself, so a good proxy for this question is asking “How can I be a better reviewer?” Here are a few things I’ve learned by trial (and error), as a paper writer, and as a reviewer. The secret ingredient is careful thought. There is no good substitution for a deep and careful understanding. Avoid reviewing papers that you feel competitive about. You almost certainly will be asked to review papers that feel competitive if you work on subjects of common interest. But, the feeling of competition can easily lead to bad judgement. If you feel biased for some other reason, then you should avoid reviewing. For example… Feeling angry or threatened by a paper is a form of bias. See above. Double blind yourself (avoid looking at the name even in a single-blind situation). The significant effect of a name you recognize is making you pay close a
6 0.55787224 116 hunch net-2005-09-30-Research in conferences
7 0.54170942 315 hunch net-2008-09-03-Bidding Problems
8 0.53723782 484 hunch net-2013-06-16-Representative Reviewing
9 0.53321153 65 hunch net-2005-05-02-Reviewing techniques for conferences
10 0.52884173 437 hunch net-2011-07-10-ICML 2011 and the future
11 0.52865082 461 hunch net-2012-04-09-ICML author feedback is open
12 0.52235121 207 hunch net-2006-09-12-Incentive Compatible Reviewing
13 0.52128243 343 hunch net-2009-02-18-Decision by Vetocracy
14 0.50641507 479 hunch net-2013-01-31-Remote large scale learning class participation
15 0.49322671 172 hunch net-2006-04-14-JMLR is a success
16 0.48304054 257 hunch net-2007-07-28-Asking questions
17 0.4815281 483 hunch net-2013-06-10-The Large Scale Learning class notes
18 0.47711062 454 hunch net-2012-01-30-ICML Posters and Scope
19 0.46564022 16 hunch net-2005-02-09-Intuitions from applied learning
20 0.45602986 468 hunch net-2012-06-29-ICML survey and comments
topicId topicWeight
[(3, 0.016), (10, 0.021), (27, 0.18), (38, 0.056), (53, 0.113), (55, 0.125), (68, 0.012), (85, 0.248), (94, 0.083), (95, 0.056)]
simIndex simValue blogId blogTitle
1 0.94511676 70 hunch net-2005-05-12-Math on the Web
Introduction: Andrej Bauer has setup a Mathematics and Computation Blog. As a first step he has tried to address the persistent and annoying problem of math on the web. As a basic tool for precisely stating and transfering understanding of technical subjects, mathematics is very necessary. Despite this necessity, every mechanism for expressing mathematics on the web seems unnaturally clumsy. Here are some of the methods and their drawbacks: MathML This was supposed to be the answer, but it has two severe drawbacks: “Internet Explorer” doesn’t read it and the language is an example of push-XML-to-the-limit which no one would ever consider writing in. (In contrast, html is easy to write in.) It’s also very annoying that math fonts must be installed independent of the browser, even for mozilla based browsers. Create inline images. This has several big drawbacks: font size is fixed for all viewers, you can’t cut & paste inside the images, and you can’t hyperlink from (say) symbol to de
same-blog 2 0.8948974 382 hunch net-2009-12-09-Future Publication Models @ NIPS
Introduction: Yesterday, there was a discussion about future publication models at NIPS . Yann and Zoubin have specific detailed proposals which I’ll add links to when I get them ( Yann’s proposal and Zoubin’s proposal ). What struck me about the discussion is that there are many simultaneous concerns as well as many simultaneous proposals, which makes it difficult to keep all the distinctions straight in a verbal conversation. It also seemed like people were serious enough about this that we may see some real movement. Certainly, my personal experience motivates that as I’ve posted many times about the substantial flaws in our review process, including some very poor personal experiences. Concerns include the following: (Several) Reviewers are overloaded, boosting the noise in decision making. ( Yann ) A new system should run with as little built-in delay and friction to the process of research as possible. ( Hanna Wallach (updated)) Double-blind review is particularly impor
3 0.82813269 481 hunch net-2013-04-15-NEML II
Introduction: Adam Kalai points out the New England Machine Learning Day May 1 at MSR New England. There is a poster session with abstracts due April 19. I understand last year’s NEML went well and it’s great to meet your neighbors at regional workshops like this.
4 0.80469429 466 hunch net-2012-06-05-ICML acceptance statistics
Introduction: People are naturally interested in slicing the ICML acceptance statistics in various ways. Here’s a rundown for the top categories. 18/66 = 0.27 in (0.18,0.36) Reinforcement Learning 10/52 = 0.19 in (0.17,0.37) Supervised Learning 9/51 = 0.18 not in (0.18, 0.37) Clustering 12/46 = 0.26 in (0.17, 0.37) Kernel Methods 11/40 = 0.28 in (0.15, 0.4) Optimization Algorithms 8/33 = 0.24 in (0.15, 0.39) Learning Theory 14/33 = 0.42 not in (0.15, 0.39) Graphical Models 10/32 = 0.31 in (0.15, 0.41) Applications (+5 invited) 8/29 = 0.28 in (0.14, 0.41]) Probabilistic Models 13/29 = 0.45 not in (0.14, 0.41) NN & Deep Learning 8/26 = 0.31 in (0.12, 0.42) Transfer and Multi-Task Learning 13/25 = 0.52 not in (0.12, 0.44) Online Learning 5/25 = 0.20 in (0.12, 0.44) Active Learning 6/22 = 0.27 in (0.14, 0.41) Semi-Superv
5 0.72797495 426 hunch net-2011-03-19-The Ideal Large Scale Learning Class
Introduction: At NIPS, Andrew Ng asked me what should be in a large scale learning class. After some discussion with him and Nando and mulling it over a bit, these are the topics that I think should be covered. There are many different kinds of scaling. Scaling in examples This is the most basic kind of scaling. Online Gradient Descent This is an old algorithm—I’m not sure if anyone can be credited with it in particular. Perhaps the Perceptron is a good precursor, but substantial improvements come from the notion of a loss function of which squared loss , logistic loss , Hinge Loss, and Quantile Loss are all worth covering. It’s important to cover the semantics of these loss functions as well. Vowpal Wabbit is a reasonably fast codebase implementing these. Second Order Gradient Descent methods For some problems, methods taking into account second derivative information can be more effective. I’ve seen preconditioned conjugate gradient work well, for which Jonath
6 0.71640176 437 hunch net-2011-07-10-ICML 2011 and the future
7 0.70635653 141 hunch net-2005-12-17-Workshops as Franchise Conferences
8 0.70474344 297 hunch net-2008-04-22-Taking the next step
9 0.70268768 207 hunch net-2006-09-12-Incentive Compatible Reviewing
10 0.69948274 204 hunch net-2006-08-28-Learning Theory standards for NIPS 2006
11 0.69879663 134 hunch net-2005-12-01-The Webscience Future
12 0.69810003 454 hunch net-2012-01-30-ICML Posters and Scope
13 0.69692713 256 hunch net-2007-07-20-Motivation should be the Responsibility of the Reviewer
14 0.69646251 423 hunch net-2011-02-02-User preferences for search engines
15 0.69484419 225 hunch net-2007-01-02-Retrospective
16 0.69484282 116 hunch net-2005-09-30-Research in conferences
17 0.69397235 403 hunch net-2010-07-18-ICML & COLT 2010
18 0.69209224 333 hunch net-2008-12-27-Adversarial Academia
19 0.69186467 286 hunch net-2008-01-25-Turing’s Club for Machine Learning
20 0.69181275 151 hunch net-2006-01-25-1 year