hunch_net hunch_net-2008 hunch_net-2008-318 knowledge-graph by maker-knowledge-mining
Source: html
Introduction: Claire asked me to be on the SODA program committee this year, which was quite a bit of work. I had a relatively light load—merely 49 theory papers. Many of these papers were not on subjects that I was expert about, so (as is common for theory conferences) I found various reviewers that I trusted to help review the papers. I ended up reviewing about 1/3 personally. There were a couple instances where I ended up overruling a subreviewer whose logic seemed off, but otherwise I generally let their reviews stand. There are some differences in standards for paper reviews between the machine learning and theory communities. In machine learning it is expected that a review be detailed, while in the theory community this is often not the case. Every paper given to me ended up with a review varying between somewhat and very detailed. I’m sure not every author was happy with the outcome. While we did our best to make good decisions, they were difficult decisions to make. For exam
sentIndex sentText sentNum sentScore
1 I had a relatively light load—merely 49 theory papers. [sent-2, score-0.27]
2 Many of these papers were not on subjects that I was expert about, so (as is common for theory conferences) I found various reviewers that I trusted to help review the papers. [sent-3, score-0.83]
3 There were a couple instances where I ended up overruling a subreviewer whose logic seemed off, but otherwise I generally let their reviews stand. [sent-5, score-0.307]
4 There are some differences in standards for paper reviews between the machine learning and theory communities. [sent-6, score-0.522]
5 In machine learning it is expected that a review be detailed, while in the theory community this is often not the case. [sent-7, score-0.279]
6 Every paper given to me ended up with a review varying between somewhat and very detailed. [sent-8, score-0.457]
7 For example, if there is a well written paper on an interesting topic which analyzes a flawed abstraction of the topic, should it get in? [sent-11, score-0.356]
8 The real time sink in reviewing a theory paper is reading it. [sent-15, score-0.383]
9 My impression of the last COLT was that COLT had entirely switched from minimal author feedback to substantial author feedback. [sent-18, score-0.483]
10 This year’s SODA was somewhere inbetween, depending on the PC member involved, which is a definite trend towards stronger comments for SODA. [sent-19, score-0.375]
11 Normalization There were very substantial differences amongst the PC members in what fraction of papers they wanted to accept, and this leaked into the final decisions. [sent-20, score-0.424]
12 Even with that help, further efforts at normalization in the future seem like they could help, for example in getting the decision on the paper above right. [sent-22, score-0.233]
13 Where the papers are sufficiently related, I think this is very helpful, and the act even changed my opinion on some papers a bit by putting them in better context. [sent-24, score-0.38]
14 (A traditional theory concern) or about the quality of the result? [sent-26, score-0.356]
15 Incidentally, there is substantial theoretical evidence that decisions by ordering are more robust than decisions by absolute score producing an ordering. [sent-29, score-0.684]
16 Writing quality I was surprised by the poor writing quality of some SODA papers—several were basically not readable without a thorough understanding of referenced papers, and a substantial ability to infer what was meant rather than what was said. [sent-30, score-0.717]
17 PC size The tradition in theory conferences is to have a relatively small program committee. [sent-32, score-0.458]
18 The program committe is small enough and SODA is broad enough that it seems dubious to claim that every PC member is an expert on the subject of all of their papers. [sent-34, score-0.512]
19 Also, (frankly) the highest quality reviews from my batch of papers weren’t written by me, but rather by reviewers that I picked who had the time to really grind through all the nitty-gritty of the paper. [sent-35, score-0.834]
20 It’s easy to imagine that a larger PC would improve reviewing quality by avoiding overload. [sent-36, score-0.298]
wordName wordTfidf (topN-words)
[('pc', 0.269), ('soda', 0.261), ('papers', 0.19), ('quality', 0.184), ('ordering', 0.178), ('ended', 0.173), ('theory', 0.172), ('decisions', 0.16), ('claire', 0.151), ('member', 0.143), ('normalization', 0.136), ('reviews', 0.134), ('differences', 0.119), ('substantial', 0.115), ('reviewing', 0.114), ('expert', 0.11), ('author', 0.109), ('written', 0.108), ('review', 0.107), ('helped', 0.1), ('help', 0.099), ('relatively', 0.098), ('paper', 0.097), ('program', 0.094), ('conferences', 0.094), ('writing', 0.09), ('every', 0.089), ('accept', 0.088), ('comments', 0.085), ('topic', 0.083), ('normalizing', 0.082), ('tastes', 0.082), ('somewhat', 0.08), ('reviewers', 0.076), ('switched', 0.076), ('trusted', 0.076), ('readable', 0.076), ('dubious', 0.076), ('definite', 0.076), ('feedback', 0.074), ('somewhere', 0.071), ('highest', 0.071), ('frankly', 0.071), ('weren', 0.071), ('absolute', 0.071), ('ratings', 0.071), ('picked', 0.071), ('report', 0.068), ('infer', 0.068), ('analyzes', 0.068)]
simIndex simValue blogId blogTitle
same-blog 1 0.99999982 318 hunch net-2008-09-26-The SODA Program Committee
Introduction: Claire asked me to be on the SODA program committee this year, which was quite a bit of work. I had a relatively light load—merely 49 theory papers. Many of these papers were not on subjects that I was expert about, so (as is common for theory conferences) I found various reviewers that I trusted to help review the papers. I ended up reviewing about 1/3 personally. There were a couple instances where I ended up overruling a subreviewer whose logic seemed off, but otherwise I generally let their reviews stand. There are some differences in standards for paper reviews between the machine learning and theory communities. In machine learning it is expected that a review be detailed, while in the theory community this is often not the case. Every paper given to me ended up with a review varying between somewhat and very detailed. I’m sure not every author was happy with the outcome. While we did our best to make good decisions, they were difficult decisions to make. For exam
2 0.33344388 221 hunch net-2006-12-04-Structural Problems in NIPS Decision Making
Introduction: This is a very difficult post to write, because it is about a perenially touchy subject. Nevertheless, it is an important one which needs to be thought about carefully. There are a few things which should be understood: The system is changing and responsive. We-the-authors are we-the-reviewers, we-the-PC, and even we-the-NIPS-board. NIPS has implemented ‘secondary program chairs’, ‘author response’, and ‘double blind reviewing’ in the last few years to help with the decision process, and more changes may happen in the future. Agreement creates a perception of correctness. When any PC meets and makes a group decision about a paper, there is a strong tendency for the reinforcement inherent in a group decision to create the perception of correctness. For the many people who have been on the NIPS PC it’s reasonable to entertain a healthy skepticism in the face of this reinforcing certainty. This post is about structural problems. What problems arise because of the structure
3 0.26443759 484 hunch net-2013-06-16-Representative Reviewing
Introduction: When thinking about how best to review papers, it seems helpful to have some conception of what good reviewing is. As far as I can tell, this is almost always only discussed in the specific context of a paper (i.e. your rejected paper), or at most an area (i.e. what a “good paper” looks like for that area) rather than general principles. Neither individual papers or areas are sufficiently general for a large conference—every paper differs in the details, and what if you want to build a new area and/or cross areas? An unavoidable reason for reviewing is that the community of research is too large. In particular, it is not possible for a researcher to read every paper which someone thinks might be of interest. This reason for reviewing exists independent of constraints on rooms or scheduling formats of individual conferences. Indeed, history suggests that physical constraints are relatively meaningless over the long term — growing conferences simply use more rooms and/or change fo
4 0.25733292 116 hunch net-2005-09-30-Research in conferences
Introduction: Conferences exist as part of the process of doing research. They provide many roles including “announcing research”, “meeting people”, and “point of reference”. Not all conferences are alike so a basic question is: “to what extent do individual conferences attempt to aid research?” This question is very difficult to answer in any satisfying way. What we can do is compare details of the process across multiple conferences. Comments The average quality of comments across conferences can vary dramatically. At one extreme, the tradition in CS theory conferences is to provide essentially zero feedback. At the other extreme, some conferences have a strong tradition of providing detailed constructive feedback. Detailed feedback can give authors significant guidance about how to improve research. This is the most subjective entry. Blind Virtually all conferences offer single blind review where authors do not know reviewers. Some also provide double blind review where rev
5 0.24343754 304 hunch net-2008-06-27-Reviewing Horror Stories
Introduction: Essentially everyone who writes research papers suffers rejections. They always sting immediately, but upon further reflection many of these rejections come to seem reasonable. Maybe the equations had too many typos or maybe the topic just isn’t as important as was originally thought. A few rejections do not come to seem acceptable, and these form the basis of reviewing horror stories, a great material for conversations. I’ve decided to share three of mine, now all safely a bit distant in the past. Prediction Theory for Classification Tutorial . This is a tutorial about tight sample complexity bounds for classification that I submitted to JMLR . The first decision I heard was a reject which appeared quite unjust to me—for example one of the reviewers appeared to claim that all the content was in standard statistics books. Upon further inquiry, several citations were given, none of which actually covered the content. Later, I was shocked to hear the paper was accepted. App
6 0.24312755 320 hunch net-2008-10-14-Who is Responsible for a Bad Review?
7 0.23497254 453 hunch net-2012-01-28-Why COLT?
8 0.2345365 437 hunch net-2011-07-10-ICML 2011 and the future
9 0.21040258 40 hunch net-2005-03-13-Avoiding Bad Reviewing
10 0.18990414 315 hunch net-2008-09-03-Bidding Problems
11 0.18824072 343 hunch net-2009-02-18-Decision by Vetocracy
12 0.18485835 468 hunch net-2012-06-29-ICML survey and comments
13 0.18220192 461 hunch net-2012-04-09-ICML author feedback is open
14 0.17905395 466 hunch net-2012-06-05-ICML acceptance statistics
15 0.17882884 454 hunch net-2012-01-30-ICML Posters and Scope
16 0.177967 395 hunch net-2010-04-26-Compassionate Reviewing
17 0.17793934 204 hunch net-2006-08-28-Learning Theory standards for NIPS 2006
18 0.16757071 207 hunch net-2006-09-12-Incentive Compatible Reviewing
19 0.16471609 452 hunch net-2012-01-04-Why ICML? and the summer conferences
20 0.15021996 65 hunch net-2005-05-02-Reviewing techniques for conferences
topicId topicWeight
[(0, 0.298), (1, -0.235), (2, 0.267), (3, 0.098), (4, 0.042), (5, 0.084), (6, -0.042), (7, -0.025), (8, -0.036), (9, -0.047), (10, 0.028), (11, 0.025), (12, -0.009), (13, -0.026), (14, -0.01), (15, -0.034), (16, 0.039), (17, 0.04), (18, 0.017), (19, 0.017), (20, 0.028), (21, 0.034), (22, 0.051), (23, -0.027), (24, -0.05), (25, 0.006), (26, 0.062), (27, -0.004), (28, 0.05), (29, 0.003), (30, 0.027), (31, -0.018), (32, -0.013), (33, -0.029), (34, 0.042), (35, 0.04), (36, 0.032), (37, -0.03), (38, 0.012), (39, -0.021), (40, -0.014), (41, 0.038), (42, 0.021), (43, 0.001), (44, -0.054), (45, 0.04), (46, -0.02), (47, -0.003), (48, -0.041), (49, 0.096)]
simIndex simValue blogId blogTitle
same-blog 1 0.97966963 318 hunch net-2008-09-26-The SODA Program Committee
Introduction: Claire asked me to be on the SODA program committee this year, which was quite a bit of work. I had a relatively light load—merely 49 theory papers. Many of these papers were not on subjects that I was expert about, so (as is common for theory conferences) I found various reviewers that I trusted to help review the papers. I ended up reviewing about 1/3 personally. There were a couple instances where I ended up overruling a subreviewer whose logic seemed off, but otherwise I generally let their reviews stand. There are some differences in standards for paper reviews between the machine learning and theory communities. In machine learning it is expected that a review be detailed, while in the theory community this is often not the case. Every paper given to me ended up with a review varying between somewhat and very detailed. I’m sure not every author was happy with the outcome. While we did our best to make good decisions, they were difficult decisions to make. For exam
2 0.91511506 221 hunch net-2006-12-04-Structural Problems in NIPS Decision Making
Introduction: This is a very difficult post to write, because it is about a perenially touchy subject. Nevertheless, it is an important one which needs to be thought about carefully. There are a few things which should be understood: The system is changing and responsive. We-the-authors are we-the-reviewers, we-the-PC, and even we-the-NIPS-board. NIPS has implemented ‘secondary program chairs’, ‘author response’, and ‘double blind reviewing’ in the last few years to help with the decision process, and more changes may happen in the future. Agreement creates a perception of correctness. When any PC meets and makes a group decision about a paper, there is a strong tendency for the reinforcement inherent in a group decision to create the perception of correctness. For the many people who have been on the NIPS PC it’s reasonable to entertain a healthy skepticism in the face of this reinforcing certainty. This post is about structural problems. What problems arise because of the structure
3 0.85502172 315 hunch net-2008-09-03-Bidding Problems
Introduction: One way that many conferences in machine learning assign reviewers to papers is via bidding, which has steps something like: Invite people to review Accept papers Reviewers look at title and abstract and state the papers they are interested in reviewing. Some massaging happens, but reviewers often get approximately the papers they bid for. At the ICML business meeting, Andrew McCallum suggested getting rid of bidding for papers. A couple reasons were given: Privacy The title and abstract of the entire set of papers is visible to every participating reviewer. Some authors might be uncomfortable about this for submitted papers. I’m not sympathetic to this reason: the point of submitting a paper to review is to publish it, so the value (if any) of not publishing a part of it a little bit earlier seems limited. Cliques A bidding system is gameable. If you have 3 buddies and you inform each other of your submissions, you can each bid for your friend’s papers a
4 0.85383767 484 hunch net-2013-06-16-Representative Reviewing
Introduction: When thinking about how best to review papers, it seems helpful to have some conception of what good reviewing is. As far as I can tell, this is almost always only discussed in the specific context of a paper (i.e. your rejected paper), or at most an area (i.e. what a “good paper” looks like for that area) rather than general principles. Neither individual papers or areas are sufficiently general for a large conference—every paper differs in the details, and what if you want to build a new area and/or cross areas? An unavoidable reason for reviewing is that the community of research is too large. In particular, it is not possible for a researcher to read every paper which someone thinks might be of interest. This reason for reviewing exists independent of constraints on rooms or scheduling formats of individual conferences. Indeed, history suggests that physical constraints are relatively meaningless over the long term — growing conferences simply use more rooms and/or change fo
5 0.84455639 304 hunch net-2008-06-27-Reviewing Horror Stories
Introduction: Essentially everyone who writes research papers suffers rejections. They always sting immediately, but upon further reflection many of these rejections come to seem reasonable. Maybe the equations had too many typos or maybe the topic just isn’t as important as was originally thought. A few rejections do not come to seem acceptable, and these form the basis of reviewing horror stories, a great material for conversations. I’ve decided to share three of mine, now all safely a bit distant in the past. Prediction Theory for Classification Tutorial . This is a tutorial about tight sample complexity bounds for classification that I submitted to JMLR . The first decision I heard was a reject which appeared quite unjust to me—for example one of the reviewers appeared to claim that all the content was in standard statistics books. Upon further inquiry, several citations were given, none of which actually covered the content. Later, I was shocked to hear the paper was accepted. App
6 0.83668619 320 hunch net-2008-10-14-Who is Responsible for a Bad Review?
7 0.83218277 463 hunch net-2012-05-02-ICML: Behind the Scenes
8 0.82887536 461 hunch net-2012-04-09-ICML author feedback is open
9 0.82171911 437 hunch net-2011-07-10-ICML 2011 and the future
10 0.7517516 453 hunch net-2012-01-28-Why COLT?
11 0.75139517 40 hunch net-2005-03-13-Avoiding Bad Reviewing
12 0.73416674 238 hunch net-2007-04-13-What to do with an unreasonable conditional accept
13 0.729725 207 hunch net-2006-09-12-Incentive Compatible Reviewing
14 0.71733028 468 hunch net-2012-06-29-ICML survey and comments
15 0.7104035 343 hunch net-2009-02-18-Decision by Vetocracy
16 0.71013737 116 hunch net-2005-09-30-Research in conferences
17 0.69617426 363 hunch net-2009-07-09-The Machine Learning Forum
18 0.69609708 466 hunch net-2012-06-05-ICML acceptance statistics
19 0.69131023 38 hunch net-2005-03-09-Bad Reviewing
20 0.68947107 395 hunch net-2010-04-26-Compassionate Reviewing
topicId topicWeight
[(3, 0.021), (10, 0.014), (27, 0.232), (38, 0.024), (48, 0.277), (53, 0.076), (55, 0.136), (94, 0.092), (95, 0.05)]
simIndex simValue blogId blogTitle
1 0.96943772 468 hunch net-2012-06-29-ICML survey and comments
Introduction: Just about nothing could keep me from attending ICML , except for Dora who arrived on Monday. Consequently, I have only secondhand reports that the conference is going well. For those who are remote (like me) or after the conference (like everyone), Mark Reid has setup the ICML discussion site where you can comment on any paper or subscribe to papers. Authors are automatically subscribed to their own papers, so it should be possible to have a discussion significantly after the fact, as people desire. We also conducted a survey before the conference and have the survey results now. This can be compared with the ICML 2010 survey results . Looking at the comparable questions, we can sometimes order the answers to have scores ranging from 0 to 3 or 0 to 4 with 3 or 4 being best and 0 worst, then compute the average difference between 2012 and 2010. Glancing through them, I see: Most people found the papers they reviewed a good fit for their expertise (-.037 w.r.t 20
same-blog 2 0.92666709 318 hunch net-2008-09-26-The SODA Program Committee
Introduction: Claire asked me to be on the SODA program committee this year, which was quite a bit of work. I had a relatively light load—merely 49 theory papers. Many of these papers were not on subjects that I was expert about, so (as is common for theory conferences) I found various reviewers that I trusted to help review the papers. I ended up reviewing about 1/3 personally. There were a couple instances where I ended up overruling a subreviewer whose logic seemed off, but otherwise I generally let their reviews stand. There are some differences in standards for paper reviews between the machine learning and theory communities. In machine learning it is expected that a review be detailed, while in the theory community this is often not the case. Every paper given to me ended up with a review varying between somewhat and very detailed. I’m sure not every author was happy with the outcome. While we did our best to make good decisions, they were difficult decisions to make. For exam
3 0.9234767 303 hunch net-2008-06-09-The Minimum Sample Complexity of Importance Weighting
Introduction: This post is about a trick that I learned from Dale Schuurmans which has been repeatedly useful for me over time. The basic trick has to do with importance weighting for monte carlo integration. Consider the problem of finding: N = E x ~ D f(x) given samples from D and knowledge of f . Often, we don’t have samples from D available. Instead, we must make do with samples from some other distribution Q . In that case, we can still often solve the problem, as long as Q(x) isn’t 0 when D(x) is nonzero, using the importance weighting formula: E x ~ Q f(x) D(x)/Q(x) A basic question is: How many samples from Q are required in order to estimate N to some precision? In general the convergence rate is not bounded, because f(x) D(x)/Q(x) is not bounded given the assumptions. Nevertheless, there is one special value Q(x) = f(x) D(x) / N where the sample complexity turns out to be 1 , which is typically substantially better than the sample complexity of the orig
4 0.906802 445 hunch net-2011-09-28-Somebody’s Eating Your Lunch
Introduction: Since we last discussed the other online learning , Stanford has very visibly started pushing mass teaching in AI , Machine Learning , and Databases . In retrospect, it’s not too surprising that the next step up in serious online teaching experiments are occurring at the computer science department of a university embedded in the land of startups. Numbers on the order of 100000 are quite significant—similar in scale to the number of computer science undergraduate students/year in the US. Although these populations surely differ, the fact that they could overlap is worth considering for the future. It’s too soon to say how successful these classes will be and there are many easy criticisms to make: Registration != Learning … but if only 1/10th complete these classes, the scale of teaching still surpasses the scale of any traditional process. 1st year excitement != nth year routine … but if only 1/10th take future classes, the scale of teaching still surpass
5 0.86474043 232 hunch net-2007-02-11-24
Introduction: To commemorate the Twenty Fourth Annual International Conference on Machine Learning (ICML-07), the FOX Network has decided to launch a new spin-off series in prime time. Through unofficial sources, I have obtained the story arc for the first season, which appears frighteningly realistic.
6 0.83534533 46 hunch net-2005-03-24-The Role of Workshops
7 0.83440179 327 hunch net-2008-11-16-Observations on Linearity for Reductions to Regression
8 0.78395402 466 hunch net-2012-06-05-ICML acceptance statistics
9 0.76639336 116 hunch net-2005-09-30-Research in conferences
10 0.75508064 403 hunch net-2010-07-18-ICML & COLT 2010
11 0.74399364 75 hunch net-2005-05-28-Running A Machine Learning Summer School
12 0.74131078 449 hunch net-2011-11-26-Giving Thanks
13 0.74027777 461 hunch net-2012-04-09-ICML author feedback is open
14 0.73875827 463 hunch net-2012-05-02-ICML: Behind the Scenes
15 0.73670244 343 hunch net-2009-02-18-Decision by Vetocracy
16 0.73544437 464 hunch net-2012-05-03-Microsoft Research, New York City
17 0.7326116 297 hunch net-2008-04-22-Taking the next step
18 0.73140693 320 hunch net-2008-10-14-Who is Responsible for a Bad Review?
19 0.72473693 452 hunch net-2012-01-04-Why ICML? and the summer conferences
20 0.72312713 333 hunch net-2008-12-27-Adversarial Academia