hunch_net hunch_net-2008 hunch_net-2008-306 knowledge-graph by maker-knowledge-mining
Source: html
Introduction: Should results of experiments on proprietary datasets be in the academic research literature? The arguments I can imagine in the “against” column are: Experiments are not repeatable. Repeatability in experiments is essential to science because it allows others to compare new methods with old and discover which is better. It’s unfair. Academics who don’t have insider access to proprietary data are at a substantial disadvantage when competing with others who do. I’m unsympathetic to argument (2). To me, it looks like their are simply some resource constraints, and these should not prevent research progress. For example, we wouldn’t prevent publishing about particle accelerator experiments by physicists at CERN because physicists at CMU couldn’t run their own experiments. Argument (1) seems like a real issue. The argument for is: Yes, they are another form of evidence that an algorithm is good. The degree to which they are evidence is less than for public
sentIndex sentText sentNum sentScore
1 Should results of experiments on proprietary datasets be in the academic research literature? [sent-1, score-1.029]
2 The arguments I can imagine in the “against” column are: Experiments are not repeatable. [sent-2, score-0.359]
3 Repeatability in experiments is essential to science because it allows others to compare new methods with old and discover which is better. [sent-3, score-0.802]
4 Academics who don’t have insider access to proprietary data are at a substantial disadvantage when competing with others who do. [sent-5, score-0.779]
5 To me, it looks like their are simply some resource constraints, and these should not prevent research progress. [sent-7, score-0.46]
6 For example, we wouldn’t prevent publishing about particle accelerator experiments by physicists at CERN because physicists at CMU couldn’t run their own experiments. [sent-8, score-1.171]
7 The argument for is: Yes, they are another form of evidence that an algorithm is good. [sent-10, score-0.307]
8 The degree to which they are evidence is less than for publicly repeatable experiments, but greater than nothing. [sent-11, score-0.331]
9 What if research can only be done in a proprietary setting? [sent-12, score-0.578]
10 It has to be good for society at large to know what works. [sent-13, score-0.089]
11 For example, suppose ICML decides to reject all papers with experiments on proprietary datasets. [sent-15, score-1.289]
12 And suppose KDD decides to consider them as weak evidence. [sent-16, score-0.545]
13 The long term result may be that beginning research on new topics which is only really doable in companies starts and then grows at KDD. [sent-17, score-0.508]
14 I consider the arguments for to be stronger than the arguments against, but I’m aware others have other beliefs. [sent-18, score-0.871]
15 I think it would be good to have a policy statement from machine learning conferences in their call for papers, as trends suggest this becoming a more serious problem in the mid-term future. [sent-19, score-0.376]
wordName wordTfidf (topN-words)
[('proprietary', 0.474), ('experiments', 0.385), ('arguments', 0.24), ('decides', 0.22), ('physicists', 0.207), ('argument', 0.201), ('prevent', 0.19), ('suppose', 0.138), ('others', 0.133), ('consider', 0.123), ('column', 0.119), ('couldn', 0.11), ('particle', 0.11), ('evidence', 0.106), ('research', 0.104), ('trends', 0.099), ('grows', 0.099), ('academics', 0.095), ('resource', 0.095), ('wouldn', 0.092), ('literature', 0.092), ('society', 0.089), ('publicly', 0.086), ('disadvantage', 0.086), ('competing', 0.086), ('cmu', 0.084), ('starts', 0.082), ('beginning', 0.078), ('compare', 0.077), ('companies', 0.075), ('discover', 0.074), ('game', 0.074), ('becoming', 0.074), ('publishing', 0.072), ('reject', 0.072), ('degree', 0.071), ('stronger', 0.071), ('looks', 0.071), ('call', 0.071), ('allows', 0.07), ('topics', 0.07), ('suggest', 0.069), ('greater', 0.068), ('constraints', 0.067), ('academic', 0.066), ('weak', 0.064), ('aware', 0.064), ('yes', 0.063), ('statement', 0.063), ('old', 0.063)]
simIndex simValue blogId blogTitle
same-blog 1 1.0000002 306 hunch net-2008-07-02-Proprietary Data in Academic Research?
Introduction: Should results of experiments on proprietary datasets be in the academic research literature? The arguments I can imagine in the “against” column are: Experiments are not repeatable. Repeatability in experiments is essential to science because it allows others to compare new methods with old and discover which is better. It’s unfair. Academics who don’t have insider access to proprietary data are at a substantial disadvantage when competing with others who do. I’m unsympathetic to argument (2). To me, it looks like their are simply some resource constraints, and these should not prevent research progress. For example, we wouldn’t prevent publishing about particle accelerator experiments by physicists at CERN because physicists at CMU couldn’t run their own experiments. Argument (1) seems like a real issue. The argument for is: Yes, they are another form of evidence that an algorithm is good. The degree to which they are evidence is less than for public
2 0.12121099 53 hunch net-2005-04-06-Structured Regret Minimization
Introduction: Geoff Gordon made an interesting presentation at the snowbird learning workshop discussing the use of no-regret algorithms for the use of several robot-related learning problems. There seems to be a draft here . This seems interesting in two ways: Drawback Removal One of the significant problems with these online algorithms is that they can’t cope with structure very easily. This drawback is addressed for certain structures. Experiments One criticism of such algorithms is that they are too “worst case”. Several experiments suggest that protecting yourself against this worst case does not necessarily incur a great loss.
3 0.11425486 115 hunch net-2005-09-26-Prediction Bounds as the Mathematics of Science
Introduction: “Science” has many meanings, but one common meaning is “the scientific method ” which is a principled method for investigating the world using the following steps: Form a hypothesis about the world. Use the hypothesis to make predictions. Run experiments to confirm or disprove the predictions. The ordering of these steps is very important to the scientific method. In particular, predictions must be made before experiments are run. Given that we all believe in the scientific method of investigation, it may be surprising to learn that cheating is very common. This happens for many reasons, some innocent and some not. Drug studies. Pharmaceutical companies make predictions about the effects of their drugs and then conduct blind clinical studies to determine their effect. Unfortunately, they have also been caught using some of the more advanced techniques for cheating here : including “reprobleming”, “data set selection”, and probably “overfitting by review”
4 0.10162358 325 hunch net-2008-11-10-ICML Reviewing Criteria
Introduction: Michael Littman and Leon Bottou have decided to use a franchise program chair approach to reviewing at ICML this year. I’ll be one of the area chairs, so I wanted to mention a few things if you are thinking about naming me. I take reviewing seriously. That means papers to be reviewed are read, the implications are considered, and decisions are only made after that. I do my best to be fair, and there are zero subjects that I consider categorical rejects. I don’t consider several arguments for rejection-not-on-the-merits reasonable . I am generally interested in papers that (a) analyze new models of machine learning, (b) provide new algorithms, and (c) show that they work empirically on plausibly real problems. If a paper has the trifecta, I’m particularly interested. With 2 out of 3, I might be interested. I often find papers with only one element harder to accept, including papers with just (a). I’m a bit tough. I rarely jump-up-and-down about a paper, because I b
5 0.095845148 370 hunch net-2009-09-18-Necessary and Sufficient Research
Introduction: Researchers are typically confronted with big problems that they have no idea how to solve. In trying to come up with a solution, a natural approach is to decompose the big problem into a set of subproblems whose solution yields a solution to the larger problem. This approach can go wrong in several ways. Decomposition failure . The solution to the decomposition does not in fact yield a solution to the overall problem. Artificial hardness . The subproblems created are sufficient if solved to solve the overall problem, but they are harder than necessary. As you can see, computational complexity forms a relatively new (in research-history) razor by which to judge an approach sufficient but not necessary. In my experience, the artificial hardness problem is very common. Many researchers abdicate the responsibility of choosing a problem to work on to other people. This process starts very naturally as a graduate student, when an incoming student might have relatively l
6 0.090131283 454 hunch net-2012-01-30-ICML Posters and Scope
7 0.083308324 343 hunch net-2009-02-18-Decision by Vetocracy
8 0.081044018 452 hunch net-2012-01-04-Why ICML? and the summer conferences
9 0.078617692 134 hunch net-2005-12-01-The Webscience Future
10 0.076276034 19 hunch net-2005-02-14-Clever Methods of Overfitting
11 0.07600411 204 hunch net-2006-08-28-Learning Theory standards for NIPS 2006
12 0.072749227 333 hunch net-2008-12-27-Adversarial Academia
13 0.072065942 388 hunch net-2010-01-24-Specializations of the Master Problem
14 0.069979936 315 hunch net-2008-09-03-Bidding Problems
15 0.069800451 220 hunch net-2006-11-27-Continuizing Solutions
16 0.068288185 132 hunch net-2005-11-26-The Design of an Optimal Research Environment
17 0.067729712 121 hunch net-2005-10-12-The unrealized potential of the research lab
18 0.067594767 177 hunch net-2006-05-05-An ICML reject
19 0.06633725 466 hunch net-2012-06-05-ICML acceptance statistics
20 0.065640733 395 hunch net-2010-04-26-Compassionate Reviewing
topicId topicWeight
[(0, 0.161), (1, -0.028), (2, -0.014), (3, 0.054), (4, -0.03), (5, -0.041), (6, 0.02), (7, -0.0), (8, 0.017), (9, 0.015), (10, 0.03), (11, 0.052), (12, 0.002), (13, -0.031), (14, -0.004), (15, 0.027), (16, -0.003), (17, -0.048), (18, -0.03), (19, -0.04), (20, 0.001), (21, -0.035), (22, -0.064), (23, 0.022), (24, 0.025), (25, -0.04), (26, 0.03), (27, -0.026), (28, -0.013), (29, -0.043), (30, -0.039), (31, 0.017), (32, 0.007), (33, -0.003), (34, 0.054), (35, -0.002), (36, -0.024), (37, 0.003), (38, 0.028), (39, 0.061), (40, 0.015), (41, 0.106), (42, 0.016), (43, 0.114), (44, -0.004), (45, -0.033), (46, 0.088), (47, -0.002), (48, -0.011), (49, 0.064)]
simIndex simValue blogId blogTitle
same-blog 1 0.95624447 306 hunch net-2008-07-02-Proprietary Data in Academic Research?
Introduction: Should results of experiments on proprietary datasets be in the academic research literature? The arguments I can imagine in the “against” column are: Experiments are not repeatable. Repeatability in experiments is essential to science because it allows others to compare new methods with old and discover which is better. It’s unfair. Academics who don’t have insider access to proprietary data are at a substantial disadvantage when competing with others who do. I’m unsympathetic to argument (2). To me, it looks like their are simply some resource constraints, and these should not prevent research progress. For example, we wouldn’t prevent publishing about particle accelerator experiments by physicists at CERN because physicists at CMU couldn’t run their own experiments. Argument (1) seems like a real issue. The argument for is: Yes, they are another form of evidence that an algorithm is good. The degree to which they are evidence is less than for public
2 0.5831666 454 hunch net-2012-01-30-ICML Posters and Scope
Introduction: Normally, I don’t indulge in posters for ICML , but this year is naturally an exception for me. If you want one, there are a small number left here , if you sign up before February. It also seems worthwhile to give some sense of the scope and reviewing criteria for ICML for authors considering submitting papers. At ICML, the (very large) program committee does the reviewing which informs final decisions by area chairs on most papers. Program chairs setup the process, deal with exceptions or disagreements, and provide advice for the reviewing process. Providing advice is tricky (and easily misleading) because a conference is a community, and in the end the aggregate interests of the community determine the conference. Nevertheless, as a program chair this year it seems worthwhile to state the overall philosophy I have and what I plan to encourage (and occasionally discourage). At the highest level, I believe ICML exists to further research into machine learning, which I gene
3 0.570494 325 hunch net-2008-11-10-ICML Reviewing Criteria
Introduction: Michael Littman and Leon Bottou have decided to use a franchise program chair approach to reviewing at ICML this year. I’ll be one of the area chairs, so I wanted to mention a few things if you are thinking about naming me. I take reviewing seriously. That means papers to be reviewed are read, the implications are considered, and decisions are only made after that. I do my best to be fair, and there are zero subjects that I consider categorical rejects. I don’t consider several arguments for rejection-not-on-the-merits reasonable . I am generally interested in papers that (a) analyze new models of machine learning, (b) provide new algorithms, and (c) show that they work empirically on plausibly real problems. If a paper has the trifecta, I’m particularly interested. With 2 out of 3, I might be interested. I often find papers with only one element harder to accept, including papers with just (a). I’m a bit tough. I rarely jump-up-and-down about a paper, because I b
4 0.56486005 108 hunch net-2005-09-06-A link
Introduction: I read through some of the essays of Michael Nielsen today, and recommend them. Principles of Effective Research and Extreme Thinking are both relevant to several discussions here.
5 0.56043559 134 hunch net-2005-12-01-The Webscience Future
Introduction: The internet has significantly effected the way we do research but it’s capabilities have not yet been fully realized. First, let’s acknowledge some known effects. Self-publishing By default, all researchers in machine learning (and more generally computer science and physics) place their papers online for anyone to download. The exact mechanism differs—physicists tend to use a central repository ( Arxiv ) while computer scientists tend to place the papers on their webpage. Arxiv has been slowly growing in subject breadth so it now sometimes used by computer scientists. Collaboration Email has enabled working remotely with coauthors. This has allowed collaborationis which would not otherwise have been possible and generally speeds research. Now, let’s look at attempts to go further. Blogs (like this one) allow public discussion about topics which are not easily categorized as “a new idea in machine learning” (like this topic). Organization of some subfield
6 0.55327392 296 hunch net-2008-04-21-The Science 2.0 article
7 0.54901546 393 hunch net-2010-04-14-MLcomp: a website for objectively comparing ML algorithms
8 0.5360918 414 hunch net-2010-10-17-Partha Niyogi has died
9 0.52107221 334 hunch net-2009-01-07-Interesting Papers at SODA 2009
10 0.51916271 1 hunch net-2005-01-19-Why I decided to run a weblog.
11 0.51573449 142 hunch net-2005-12-22-Yes , I am applying
12 0.5148527 106 hunch net-2005-09-04-Science in the Government
13 0.51285791 366 hunch net-2009-08-03-Carbon in Computer Science Research
14 0.51048309 333 hunch net-2008-12-27-Adversarial Academia
15 0.50866216 409 hunch net-2010-09-13-AIStats
16 0.50604242 399 hunch net-2010-05-20-Google Predict
17 0.5026558 282 hunch net-2008-01-06-Research Political Issues
18 0.5024482 255 hunch net-2007-07-13-The View From China
19 0.4955681 53 hunch net-2005-04-06-Structured Regret Minimization
20 0.49424294 397 hunch net-2010-05-02-What’s the difference between gambling and rewarding good prediction?
topicId topicWeight
[(3, 0.024), (19, 0.238), (27, 0.193), (38, 0.096), (53, 0.04), (55, 0.096), (79, 0.013), (94, 0.15), (95, 0.04)]
simIndex simValue blogId blogTitle
same-blog 1 0.91079062 306 hunch net-2008-07-02-Proprietary Data in Academic Research?
Introduction: Should results of experiments on proprietary datasets be in the academic research literature? The arguments I can imagine in the “against” column are: Experiments are not repeatable. Repeatability in experiments is essential to science because it allows others to compare new methods with old and discover which is better. It’s unfair. Academics who don’t have insider access to proprietary data are at a substantial disadvantage when competing with others who do. I’m unsympathetic to argument (2). To me, it looks like their are simply some resource constraints, and these should not prevent research progress. For example, we wouldn’t prevent publishing about particle accelerator experiments by physicists at CERN because physicists at CMU couldn’t run their own experiments. Argument (1) seems like a real issue. The argument for is: Yes, they are another form of evidence that an algorithm is good. The degree to which they are evidence is less than for public
2 0.88027149 419 hunch net-2010-12-04-Vowpal Wabbit, version 5.0, and the second heresy
Introduction: I’ve released version 5.0 of the Vowpal Wabbit online learning software. The major number has changed since the last release because I regard all earlier versions as obsolete—there are several new algorithms & features including substantial changes and upgrades to the default learning algorithm. The biggest changes are new algorithms: Nikos and I improved the default algorithm. The basic update rule still uses gradient descent, but the size of the update is carefully controlled so that it’s impossible to overrun the label. In addition, the normalization has changed. Computationally, these changes are virtually free and yield better results, sometimes much better. Less careful updates can be reenabled with –loss_function classic, although results are still not identical to previous due to normalization changes. Nikos also implemented the per-feature learning rates as per these two papers . Often, this works better than the default algorithm. It isn’t the defa
3 0.87084377 491 hunch net-2013-11-21-Ben Taskar is gone
Introduction: I was not as personally close to Ben as Sam , but the level of tragedy is similar and I can’t help but be greatly saddened by the loss. Various news stories have coverage, but the synopsis is that he had a heart attack on Sunday and is survived by his wife Anat and daughter Aviv. There is discussion of creating a memorial fund for them, which I hope comes to fruition, and plan to contribute to. I will remember Ben as someone who thought carefully and comprehensively about new ways to do things, then fought hard and successfully for what he believed in. It is an ideal we strive for, that Ben accomplished. Edit: donations go here , and more information is here .
4 0.73901755 221 hunch net-2006-12-04-Structural Problems in NIPS Decision Making
Introduction: This is a very difficult post to write, because it is about a perenially touchy subject. Nevertheless, it is an important one which needs to be thought about carefully. There are a few things which should be understood: The system is changing and responsive. We-the-authors are we-the-reviewers, we-the-PC, and even we-the-NIPS-board. NIPS has implemented ‘secondary program chairs’, ‘author response’, and ‘double blind reviewing’ in the last few years to help with the decision process, and more changes may happen in the future. Agreement creates a perception of correctness. When any PC meets and makes a group decision about a paper, there is a strong tendency for the reinforcement inherent in a group decision to create the perception of correctness. For the many people who have been on the NIPS PC it’s reasonable to entertain a healthy skepticism in the face of this reinforcing certainty. This post is about structural problems. What problems arise because of the structure
5 0.7336728 286 hunch net-2008-01-25-Turing’s Club for Machine Learning
Introduction: Many people in Machine Learning don’t fully understand the impact of computation, as demonstrated by a lack of big-O analysis of new learning algorithms. This is important—some current active research programs are fundamentally flawed w.r.t. computation, and other research programs are directly motivated by it. When considering a learning algorithm, I think about the following questions: How does the learning algorithm scale with the number of examples m ? Any algorithm using all of the data is at least O(m) , but in many cases this is O(m 2 ) (naive nearest neighbor for self-prediction) or unknown (k-means or many other optimization algorithms). The unknown case is very common, and it can mean (for example) that the algorithm isn’t convergent or simply that the amount of computation isn’t controlled. The above question can also be asked for test cases. In some applications, test-time performance is of great importance. How does the algorithm scale with the number of
6 0.72858828 423 hunch net-2011-02-02-User preferences for search engines
7 0.72691703 95 hunch net-2005-07-14-What Learning Theory might do
8 0.7242347 136 hunch net-2005-12-07-Is the Google way the way for machine learning?
9 0.71979153 204 hunch net-2006-08-28-Learning Theory standards for NIPS 2006
10 0.71873569 276 hunch net-2007-12-10-Learning Track of International Planning Competition
11 0.71658325 359 hunch net-2009-06-03-Functionally defined Nonlinear Dynamic Models
12 0.71512425 49 hunch net-2005-03-30-What can Type Theory teach us about Machine Learning?
13 0.71495181 229 hunch net-2007-01-26-Parallel Machine Learning Problems
14 0.71404231 237 hunch net-2007-04-02-Contextual Scaling
15 0.71057272 162 hunch net-2006-03-09-Use of Notation
16 0.71025252 98 hunch net-2005-07-27-Not goal metrics
17 0.70948267 371 hunch net-2009-09-21-Netflix finishes (and starts)
18 0.70813829 450 hunch net-2011-12-02-Hadoop AllReduce and Terascale Learning
19 0.70695859 351 hunch net-2009-05-02-Wielding a New Abstraction
20 0.70620286 132 hunch net-2005-11-26-The Design of an Optimal Research Environment