hunch_net hunch_net-2012 hunch_net-2012-466 knowledge-graph by maker-knowledge-mining
Source: html
Introduction: People are naturally interested in slicing the ICML acceptance statistics in various ways. Here’s a rundown for the top categories. 18/66 = 0.27 in (0.18,0.36) Reinforcement Learning 10/52 = 0.19 in (0.17,0.37) Supervised Learning 9/51 = 0.18 not in (0.18, 0.37) Clustering 12/46 = 0.26 in (0.17, 0.37) Kernel Methods 11/40 = 0.28 in (0.15, 0.4) Optimization Algorithms 8/33 = 0.24 in (0.15, 0.39) Learning Theory 14/33 = 0.42 not in (0.15, 0.39) Graphical Models 10/32 = 0.31 in (0.15, 0.41) Applications (+5 invited) 8/29 = 0.28 in (0.14, 0.41]) Probabilistic Models 13/29 = 0.45 not in (0.14, 0.41) NN & Deep Learning 8/26 = 0.31 in (0.12, 0.42) Transfer and Multi-Task Learning 13/25 = 0.52 not in (0.12, 0.44) Online Learning 5/25 = 0.20 in (0.12, 0.44) Active Learning 6/22 = 0.27 in (0.14, 0.41) Semi-Superv
sentIndex sentText sentNum sentScore
1 People are naturally interested in slicing the ICML acceptance statistics in various ways. [sent-1, score-0.124]
2 At a finer level, one way to add further interpretation is to pretend that the acceptance rate of all papers is 0. [sent-112, score-0.321]
3 27, then compute a 5% lower tail and a 5% upper tail. [sent-113, score-0.111]
4 Instead, we have 9, so there is some evidence that individual areas are particularly hot or cold. [sent-115, score-0.16]
5 In particular, the hot topics are Graphical models, Neural Networks and Deep Learning, Online Learning, Gaussian Processes, Ranking and Preference Learning, and Time Series Analysis. [sent-116, score-0.157]
6 We also experimented with AIStats resubmits (3/4 accepted) and NFP papers (4/7 accepted) but the numbers were to small to read anything significant. [sent-118, score-0.176]
7 One thing that surprised me was how uniform decisions were as a function of average score in reviews. [sent-119, score-0.324]
8 All reviews included a decision from {Strong Reject, Weak Reject, Weak Accept, Strong Accept}. [sent-120, score-0.236]
9 2 meant 0% chance of acceptance, and average review score > 3. [sent-123, score-0.394]
10 Due to discretization in the number of reviewers and review scores there were only 3 typical uncertain outcomes: 2. [sent-125, score-0.159]
11 In general, correlated assignment of reviewers can greatly increase the amount of variance, so one of our goals this year was doing as independent an assignment as possible. [sent-138, score-0.228]
12 If you accept that as independence, we essentially get 3 samples for each paper where the average standard deviation of reviewer scores before author feedback and discussion is 0. [sent-139, score-0.964]
13 After author feedback and discussion the standard deviation drops to 0. [sent-141, score-0.413]
14 If we pretend that papers have an intrinsic value between 1 and 4 then think of reviews as discretized gaussian measurements fed through the above decision criteria, we get the following: There are great caveats to this picture. [sent-143, score-0.545]
15 For example, treating the AC’s decision as random conditioned on the reviewer average is a worst-case analysis. [sent-144, score-0.531]
16 Similarly, treating the reviews observed after discussion as independent is clearly flawed. [sent-146, score-0.419]
17 A reasonable way to look at it is: author feedback and discussion get us about 1/3 or 1/4 of the way to the final decision from the initial reviews. [sent-147, score-0.424]
18 Conditioned on the papers, discussion, author feedback and reviews, AC’s are pretty uniform in their decisions with ~30 papers where ACs disagreed on the accept/reject decision. [sent-148, score-0.361]
19 We actually aimed higher: at least 3 people needed to make a wrong decision for the ICML 2012 reviewing process to kick out a wrong decision. [sent-151, score-0.114]
20 I expect this happened a few times given the overall level of quality disagreement and quantities involved, but hopefully we managed to reduce the noise appreciably. [sent-152, score-0.137]
wordName wordTfidf (topN-words)
[('weak', 0.39), ('reject', 0.271), ('accept', 0.27), ('strong', 0.241), ('accepts', 0.145), ('average', 0.139), ('acs', 0.138), ('methods', 0.129), ('treating', 0.125), ('acceptance', 0.124), ('reviews', 0.122), ('decision', 0.114), ('models', 0.113), ('gaussian', 0.112), ('tail', 0.111), ('pretend', 0.111), ('ac', 0.111), ('discussion', 0.11), ('score', 0.11), ('rarely', 0.103), ('deviation', 0.103), ('feedback', 0.101), ('author', 0.099), ('conditioned', 0.097), ('rejects', 0.092), ('hot', 0.092), ('numbers', 0.09), ('papers', 0.086), ('scores', 0.086), ('assignment', 0.083), ('clustering', 0.076), ('ensemble', 0.076), ('uniform', 0.075), ('variance', 0.075), ('review', 0.073), ('ranking', 0.072), ('meant', 0.072), ('preference', 0.072), ('level', 0.07), ('graphical', 0.068), ('areas', 0.068), ('processes', 0.067), ('noise', 0.067), ('accepted', 0.067), ('topics', 0.065), ('learning', 0.063), ('independent', 0.062), ('neural', 0.058), ('reviewer', 0.056), ('networks', 0.056)]
simIndex simValue blogId blogTitle
same-blog 1 1.0000002 466 hunch net-2012-06-05-ICML acceptance statistics
Introduction: People are naturally interested in slicing the ICML acceptance statistics in various ways. Here’s a rundown for the top categories. 18/66 = 0.27 in (0.18,0.36) Reinforcement Learning 10/52 = 0.19 in (0.17,0.37) Supervised Learning 9/51 = 0.18 not in (0.18, 0.37) Clustering 12/46 = 0.26 in (0.17, 0.37) Kernel Methods 11/40 = 0.28 in (0.15, 0.4) Optimization Algorithms 8/33 = 0.24 in (0.15, 0.39) Learning Theory 14/33 = 0.42 not in (0.15, 0.39) Graphical Models 10/32 = 0.31 in (0.15, 0.41) Applications (+5 invited) 8/29 = 0.28 in (0.14, 0.41]) Probabilistic Models 13/29 = 0.45 not in (0.14, 0.41) NN & Deep Learning 8/26 = 0.31 in (0.12, 0.42) Transfer and Multi-Task Learning 13/25 = 0.52 not in (0.12, 0.44) Online Learning 5/25 = 0.20 in (0.12, 0.44) Active Learning 6/22 = 0.27 in (0.14, 0.41) Semi-Superv
2 0.28223583 456 hunch net-2012-02-24-ICML+50%
Introduction: The ICML paper deadline has passed. Joelle and I were surprised to see the number of submissions jump from last year by about 50% to around 900 submissions. A tiny portion of these are immediate rejects(*), so this is a much larger set of papers than expected. The number of workshop submissions also doubled compared to last year, so ICML may grow significantly this year, if we can manage to handle the load well. The prospect of making 900 good decisions is fundamentally daunting, and success will rely heavily on the program committee and area chairs at this point. For those who want to rubberneck a bit more, here’s a breakdown of submissions by primary topic of submitted papers: 66 Reinforcement Learning 52 Supervised Learning 51 Clustering 46 Kernel Methods 40 Optimization Algorithms 39 Feature Selection and Dimensionality Reduction 33 Learning Theory 33 Graphical Models 33 Applications 29 Probabilistic Models 29 NN & Deep Learning 26 Transfer and Multi-Ta
3 0.20047779 484 hunch net-2013-06-16-Representative Reviewing
Introduction: When thinking about how best to review papers, it seems helpful to have some conception of what good reviewing is. As far as I can tell, this is almost always only discussed in the specific context of a paper (i.e. your rejected paper), or at most an area (i.e. what a “good paper” looks like for that area) rather than general principles. Neither individual papers or areas are sufficiently general for a large conference—every paper differs in the details, and what if you want to build a new area and/or cross areas? An unavoidable reason for reviewing is that the community of research is too large. In particular, it is not possible for a researcher to read every paper which someone thinks might be of interest. This reason for reviewing exists independent of constraints on rooms or scheduling formats of individual conferences. Indeed, history suggests that physical constraints are relatively meaningless over the long term — growing conferences simply use more rooms and/or change fo
4 0.18490705 343 hunch net-2009-02-18-Decision by Vetocracy
Introduction: Few would mistake the process of academic paper review for a fair process, but sometimes the unfairness seems particularly striking. This is most easily seen by comparison: Paper Banditron Offset Tree Notes Problem Scope Multiclass problems where only the loss of one choice can be probed. Strictly greater: Cost sensitive multiclass problems where only the loss of one choice can be probed. Often generalizations don’t matter. That’s not the case here, since every plausible application I’ve thought of involves loss functions substantially different from 0/1. What’s new Analysis and Experiments Algorithm, Analysis, and Experiments As far as I know, the essence of the more general problem was first stated and analyzed with the EXP4 algorithm (page 16) (1998). It’s also the time horizon 1 simplification of the Reinforcement Learning setting for the random trajectory method (page 15) (2002). The Banditron algorithm itself is functionally identi
5 0.18415692 116 hunch net-2005-09-30-Research in conferences
Introduction: Conferences exist as part of the process of doing research. They provide many roles including “announcing research”, “meeting people”, and “point of reference”. Not all conferences are alike so a basic question is: “to what extent do individual conferences attempt to aid research?” This question is very difficult to answer in any satisfying way. What we can do is compare details of the process across multiple conferences. Comments The average quality of comments across conferences can vary dramatically. At one extreme, the tradition in CS theory conferences is to provide essentially zero feedback. At the other extreme, some conferences have a strong tradition of providing detailed constructive feedback. Detailed feedback can give authors significant guidance about how to improve research. This is the most subjective entry. Blind Virtually all conferences offer single blind review where authors do not know reviewers. Some also provide double blind review where rev
6 0.17905395 318 hunch net-2008-09-26-The SODA Program Committee
7 0.1779197 468 hunch net-2012-06-29-ICML survey and comments
8 0.16290577 179 hunch net-2006-05-16-The value of the orthodox view of Boosting
9 0.15915306 304 hunch net-2008-06-27-Reviewing Horror Stories
10 0.15414342 461 hunch net-2012-04-09-ICML author feedback is open
11 0.15123796 98 hunch net-2005-07-27-Not goal metrics
12 0.14310969 315 hunch net-2008-09-03-Bidding Problems
13 0.14184074 40 hunch net-2005-03-13-Avoiding Bad Reviewing
14 0.14065877 65 hunch net-2005-05-02-Reviewing techniques for conferences
15 0.13390324 437 hunch net-2011-07-10-ICML 2011 and the future
16 0.12624353 320 hunch net-2008-10-14-Who is Responsible for a Bad Review?
17 0.12357719 454 hunch net-2012-01-30-ICML Posters and Scope
18 0.11216693 463 hunch net-2012-05-02-ICML: Behind the Scenes
19 0.11215939 238 hunch net-2007-04-13-What to do with an unreasonable conditional accept
20 0.10882462 201 hunch net-2006-08-07-The Call of the Deep
topicId topicWeight
[(0, 0.244), (1, -0.1), (2, 0.195), (3, 0.043), (4, 0.121), (5, 0.037), (6, -0.055), (7, 0.013), (8, 0.043), (9, -0.075), (10, -0.019), (11, 0.002), (12, -0.035), (13, -0.13), (14, -0.069), (15, 0.12), (16, -0.028), (17, 0.024), (18, -0.014), (19, -0.026), (20, 0.03), (21, -0.064), (22, -0.034), (23, -0.03), (24, -0.108), (25, -0.018), (26, -0.004), (27, -0.039), (28, 0.029), (29, 0.042), (30, 0.04), (31, -0.059), (32, 0.024), (33, -0.088), (34, 0.006), (35, 0.065), (36, -0.003), (37, -0.097), (38, -0.009), (39, -0.001), (40, -0.008), (41, -0.0), (42, -0.044), (43, 0.059), (44, -0.142), (45, 0.014), (46, -0.017), (47, 0.106), (48, 0.014), (49, 0.097)]
simIndex simValue blogId blogTitle
same-blog 1 0.96426731 466 hunch net-2012-06-05-ICML acceptance statistics
Introduction: People are naturally interested in slicing the ICML acceptance statistics in various ways. Here’s a rundown for the top categories. 18/66 = 0.27 in (0.18,0.36) Reinforcement Learning 10/52 = 0.19 in (0.17,0.37) Supervised Learning 9/51 = 0.18 not in (0.18, 0.37) Clustering 12/46 = 0.26 in (0.17, 0.37) Kernel Methods 11/40 = 0.28 in (0.15, 0.4) Optimization Algorithms 8/33 = 0.24 in (0.15, 0.39) Learning Theory 14/33 = 0.42 not in (0.15, 0.39) Graphical Models 10/32 = 0.31 in (0.15, 0.41) Applications (+5 invited) 8/29 = 0.28 in (0.14, 0.41]) Probabilistic Models 13/29 = 0.45 not in (0.14, 0.41) NN & Deep Learning 8/26 = 0.31 in (0.12, 0.42) Transfer and Multi-Task Learning 13/25 = 0.52 not in (0.12, 0.44) Online Learning 5/25 = 0.20 in (0.12, 0.44) Active Learning 6/22 = 0.27 in (0.14, 0.41) Semi-Superv
2 0.68358475 318 hunch net-2008-09-26-The SODA Program Committee
Introduction: Claire asked me to be on the SODA program committee this year, which was quite a bit of work. I had a relatively light load—merely 49 theory papers. Many of these papers were not on subjects that I was expert about, so (as is common for theory conferences) I found various reviewers that I trusted to help review the papers. I ended up reviewing about 1/3 personally. There were a couple instances where I ended up overruling a subreviewer whose logic seemed off, but otherwise I generally let their reviews stand. There are some differences in standards for paper reviews between the machine learning and theory communities. In machine learning it is expected that a review be detailed, while in the theory community this is often not the case. Every paper given to me ended up with a review varying between somewhat and very detailed. I’m sure not every author was happy with the outcome. While we did our best to make good decisions, they were difficult decisions to make. For exam
3 0.64748603 463 hunch net-2012-05-02-ICML: Behind the Scenes
Introduction: This is a rather long post, detailing the ICML 2012 review process. The goal is to make the process more transparent, help authors understand how we came to a decision, and discuss the strengths and weaknesses of this process for future conference organizers. Microsoft’s Conference Management Toolkit (CMT) We chose to use CMT over other conference management software mainly because of its rich toolkit. The interface is sub-optimal (to say the least!) but it has extensive capabilities (to handle bids, author response, resubmissions, etc.), good import/export mechanisms (to process the data elsewhere), excellent technical support (to answer late night emails, add new functionalities). Overall, it was the right choice, although we hope a designer will look at that interface sometime soon! Toronto Matching System (TMS) TMS is now being used by many major conferences in our field (including NIPS and UAI). It is an automated system (developed by Laurent Charlin and Rich Ze
4 0.64701569 456 hunch net-2012-02-24-ICML+50%
Introduction: The ICML paper deadline has passed. Joelle and I were surprised to see the number of submissions jump from last year by about 50% to around 900 submissions. A tiny portion of these are immediate rejects(*), so this is a much larger set of papers than expected. The number of workshop submissions also doubled compared to last year, so ICML may grow significantly this year, if we can manage to handle the load well. The prospect of making 900 good decisions is fundamentally daunting, and success will rely heavily on the program committee and area chairs at this point. For those who want to rubberneck a bit more, here’s a breakdown of submissions by primary topic of submitted papers: 66 Reinforcement Learning 52 Supervised Learning 51 Clustering 46 Kernel Methods 40 Optimization Algorithms 39 Feature Selection and Dimensionality Reduction 33 Learning Theory 33 Graphical Models 33 Applications 29 Probabilistic Models 29 NN & Deep Learning 26 Transfer and Multi-Ta
5 0.64201242 461 hunch net-2012-04-09-ICML author feedback is open
Introduction: as of last night, late. When the reviewing deadline passed Wednesday night 15% of reviews were still missing, much higher than I expected. Between late reviews coming in, ACs working overtime through the weekend, and people willing to help in the pinch another ~390 reviews came in, reducing the missing mass to 0.2%. Nailing that last bit and a similar quantity of papers with uniformly low confidence reviews is what remains to be done in terms of basic reviews. We are trying to make all of those happen this week so authors have some chance to respond. I was surprised by the quantity of late reviews, and I think that’s an area where ICML needs to improve in future years. Good reviews are not done in a rush—they are done by setting aside time (like an afternoon), and carefully reading the paper while thinking about implications. Many reviewers do this well but a significant minority aren’t good at scheduling their personal time. In this situation there are several ways to fail:
6 0.63004172 315 hunch net-2008-09-03-Bidding Problems
7 0.60959411 221 hunch net-2006-12-04-Structural Problems in NIPS Decision Making
8 0.60676455 304 hunch net-2008-06-27-Reviewing Horror Stories
9 0.59415638 484 hunch net-2013-06-16-Representative Reviewing
10 0.58910125 363 hunch net-2009-07-09-The Machine Learning Forum
11 0.56332266 320 hunch net-2008-10-14-Who is Responsible for a Bad Review?
12 0.55081642 207 hunch net-2006-09-12-Incentive Compatible Reviewing
13 0.54879707 343 hunch net-2009-02-18-Decision by Vetocracy
14 0.54834002 238 hunch net-2007-04-13-What to do with an unreasonable conditional accept
15 0.54146415 40 hunch net-2005-03-13-Avoiding Bad Reviewing
16 0.52659959 403 hunch net-2010-07-18-ICML & COLT 2010
17 0.52659535 468 hunch net-2012-06-29-ICML survey and comments
18 0.52574259 485 hunch net-2013-06-29-The Benefits of Double-Blind Review
19 0.52470177 438 hunch net-2011-07-11-Interesting Neural Network Papers at ICML 2011
20 0.51992971 395 hunch net-2010-04-26-Compassionate Reviewing
topicId topicWeight
[(3, 0.016), (10, 0.024), (27, 0.17), (30, 0.014), (38, 0.016), (48, 0.065), (49, 0.011), (53, 0.075), (55, 0.144), (77, 0.012), (85, 0.198), (94, 0.042), (95, 0.122)]
simIndex simValue blogId blogTitle
1 0.89700192 70 hunch net-2005-05-12-Math on the Web
Introduction: Andrej Bauer has setup a Mathematics and Computation Blog. As a first step he has tried to address the persistent and annoying problem of math on the web. As a basic tool for precisely stating and transfering understanding of technical subjects, mathematics is very necessary. Despite this necessity, every mechanism for expressing mathematics on the web seems unnaturally clumsy. Here are some of the methods and their drawbacks: MathML This was supposed to be the answer, but it has two severe drawbacks: “Internet Explorer” doesn’t read it and the language is an example of push-XML-to-the-limit which no one would ever consider writing in. (In contrast, html is easy to write in.) It’s also very annoying that math fonts must be installed independent of the browser, even for mozilla based browsers. Create inline images. This has several big drawbacks: font size is fixed for all viewers, you can’t cut & paste inside the images, and you can’t hyperlink from (say) symbol to de
2 0.87674886 382 hunch net-2009-12-09-Future Publication Models @ NIPS
Introduction: Yesterday, there was a discussion about future publication models at NIPS . Yann and Zoubin have specific detailed proposals which I’ll add links to when I get them ( Yann’s proposal and Zoubin’s proposal ). What struck me about the discussion is that there are many simultaneous concerns as well as many simultaneous proposals, which makes it difficult to keep all the distinctions straight in a verbal conversation. It also seemed like people were serious enough about this that we may see some real movement. Certainly, my personal experience motivates that as I’ve posted many times about the substantial flaws in our review process, including some very poor personal experiences. Concerns include the following: (Several) Reviewers are overloaded, boosting the noise in decision making. ( Yann ) A new system should run with as little built-in delay and friction to the process of research as possible. ( Hanna Wallach (updated)) Double-blind review is particularly impor
same-blog 3 0.86890531 466 hunch net-2012-06-05-ICML acceptance statistics
Introduction: People are naturally interested in slicing the ICML acceptance statistics in various ways. Here’s a rundown for the top categories. 18/66 = 0.27 in (0.18,0.36) Reinforcement Learning 10/52 = 0.19 in (0.17,0.37) Supervised Learning 9/51 = 0.18 not in (0.18, 0.37) Clustering 12/46 = 0.26 in (0.17, 0.37) Kernel Methods 11/40 = 0.28 in (0.15, 0.4) Optimization Algorithms 8/33 = 0.24 in (0.15, 0.39) Learning Theory 14/33 = 0.42 not in (0.15, 0.39) Graphical Models 10/32 = 0.31 in (0.15, 0.41) Applications (+5 invited) 8/29 = 0.28 in (0.14, 0.41]) Probabilistic Models 13/29 = 0.45 not in (0.14, 0.41) NN & Deep Learning 8/26 = 0.31 in (0.12, 0.42) Transfer and Multi-Task Learning 13/25 = 0.52 not in (0.12, 0.44) Online Learning 5/25 = 0.20 in (0.12, 0.44) Active Learning 6/22 = 0.27 in (0.14, 0.41) Semi-Superv
4 0.85633785 481 hunch net-2013-04-15-NEML II
Introduction: Adam Kalai points out the New England Machine Learning Day May 1 at MSR New England. There is a poster session with abstracts due April 19. I understand last year’s NEML went well and it’s great to meet your neighbors at regional workshops like this.
5 0.78930563 462 hunch net-2012-04-20-Both new: STOC workshops and NEML
Introduction: May 16 in Cambridge , is the New England Machine Learning Day , a first regional workshop/symposium on machine learning. To present a poster, submit an abstract by May 5 . May 19 in New York , STOC is coming to town and rather surprisingly having workshops which should be quite a bit of fun. I’ll be speaking at Algorithms for Distributed and Streaming Data .
6 0.7671591 105 hunch net-2005-08-23-(Dis)similarities between academia and open source programmers
7 0.75532007 464 hunch net-2012-05-03-Microsoft Research, New York City
8 0.75169665 468 hunch net-2012-06-29-ICML survey and comments
9 0.74612194 445 hunch net-2011-09-28-Somebody’s Eating Your Lunch
10 0.7424497 318 hunch net-2008-09-26-The SODA Program Committee
11 0.73649079 344 hunch net-2009-02-22-Effective Research Funding
12 0.73036748 452 hunch net-2012-01-04-Why ICML? and the summer conferences
13 0.72766674 225 hunch net-2007-01-02-Retrospective
14 0.72699988 437 hunch net-2011-07-10-ICML 2011 and the future
15 0.72387904 132 hunch net-2005-11-26-The Design of an Optimal Research Environment
16 0.72365516 456 hunch net-2012-02-24-ICML+50%
17 0.72341216 151 hunch net-2006-01-25-1 year
18 0.7224527 116 hunch net-2005-09-30-Research in conferences
19 0.72184986 89 hunch net-2005-07-04-The Health of COLT
20 0.72132742 403 hunch net-2010-07-18-ICML & COLT 2010