hunch_net hunch_net-2005 hunch_net-2005-51 knowledge-graph by maker-knowledge-mining
Source: html
Introduction: In the quest to understand what good reviewing is, perhaps it’s worthwhile to think about what good research is. One way to think about good research is in terms of a producer/consumer model. In the producer/consumer model of research, for any element of research there are producers (authors and coauthors of papers, for example) and consumers (people who use the papers to make new papers or code solving problems). An produced bit of research is judged as “good” if it is used by many consumers. There are two basic questions which immediately arise: Is this a good model of research? Are there alternatives? The producer/consumer model has some difficulties which can be (partially) addressed. Disconnect. A group of people doing research on some subject may become disconnected from the rest of the world. Each person uses the research of other people in the group so it appears good research is being done, but the group has no impact on the rest of the world. One way
sentIndex sentText sentNum sentScore
1 In the quest to understand what good reviewing is, perhaps it’s worthwhile to think about what good research is. [sent-1, score-0.689]
2 One way to think about good research is in terms of a producer/consumer model. [sent-2, score-0.548]
3 In the producer/consumer model of research, for any element of research there are producers (authors and coauthors of papers, for example) and consumers (people who use the papers to make new papers or code solving problems). [sent-3, score-1.309]
4 An produced bit of research is judged as “good” if it is used by many consumers. [sent-4, score-0.468]
5 There are two basic questions which immediately arise: Is this a good model of research? [sent-5, score-0.33]
6 The producer/consumer model has some difficulties which can be (partially) addressed. [sent-7, score-0.337]
7 A group of people doing research on some subject may become disconnected from the rest of the world. [sent-9, score-0.657]
8 Each person uses the research of other people in the group so it appears good research is being done, but the group has no impact on the rest of the world. [sent-10, score-1.272]
9 One way to detect this is by looking at the consumers 2 (the consumers of the consumers) and higher order powers. [sent-11, score-1.432]
10 If the set doesn’t expand much with higher order powers, then there is a disconnect. [sent-12, score-0.27]
11 It is extraordinarily difficult to determine in advance whether a piece of research will have many consumers. [sent-14, score-0.625]
12 A particular piece of work may be useful only after a very long period of time. [sent-15, score-0.31]
13 Self-fulfillment To some extent, interesting research by this definition is simply research presented to the largest possible audience. [sent-17, score-0.896]
14 The odds that someone will build on the research are simply larger when it is presented to a larger audience. [sent-18, score-0.768]
15 Some portion of this effect is “ok”—certainly attempting to educate other people is a good idea. [sent-19, score-0.394]
16 But in judging the value of a piece of research, discounting by the vigor with which it is presented may be healthy for the system. [sent-20, score-0.601]
17 Another important effect is that a reviewer who rejects a paper biases the number of citations a paper later recieves. [sent-25, score-0.591]
18 Another is that a rejected paper that has been resubmitted to another place may change so that it is simply a better paper. [sent-26, score-0.29]
19 Clearly, there are problems with this model for judging research (and at the second order, judgements of reviews of research). [sent-28, score-0.753]
20 However, I am not aware of any other abstract model for “good research” which is even this good. [sent-29, score-0.194]
wordName wordTfidf (topN-words)
[('consumers', 0.577), ('research', 0.338), ('model', 0.194), ('piece', 0.162), ('presented', 0.151), ('difficulties', 0.143), ('judging', 0.142), ('group', 0.141), ('good', 0.136), ('effect', 0.124), ('citations', 0.121), ('extent', 0.103), ('order', 0.101), ('higher', 0.098), ('rest', 0.094), ('reviewer', 0.087), ('blow', 0.085), ('consumer', 0.085), ('joe', 0.085), ('may', 0.084), ('impact', 0.084), ('reviewing', 0.079), ('detect', 0.079), ('alternatives', 0.079), ('judgements', 0.079), ('powers', 0.079), ('another', 0.076), ('terms', 0.074), ('larger', 0.073), ('proxy', 0.071), ('expand', 0.071), ('rejects', 0.071), ('simply', 0.069), ('judged', 0.068), ('attempting', 0.068), ('coauthors', 0.068), ('papers', 0.066), ('citation', 0.066), ('biases', 0.066), ('determine', 0.066), ('portion', 0.066), ('odds', 0.064), ('period', 0.064), ('produced', 0.062), ('act', 0.062), ('arise', 0.062), ('healthy', 0.062), ('paper', 0.061), ('isn', 0.061), ('extraordinarily', 0.059)]
simIndex simValue blogId blogTitle
same-blog 1 1.0000001 51 hunch net-2005-04-01-The Producer-Consumer Model of Research
Introduction: In the quest to understand what good reviewing is, perhaps it’s worthwhile to think about what good research is. One way to think about good research is in terms of a producer/consumer model. In the producer/consumer model of research, for any element of research there are producers (authors and coauthors of papers, for example) and consumers (people who use the papers to make new papers or code solving problems). An produced bit of research is judged as “good” if it is used by many consumers. There are two basic questions which immediately arise: Is this a good model of research? Are there alternatives? The producer/consumer model has some difficulties which can be (partially) addressed. Disconnect. A group of people doing research on some subject may become disconnected from the rest of the world. Each person uses the research of other people in the group so it appears good research is being done, but the group has no impact on the rest of the world. One way
2 0.17695837 36 hunch net-2005-03-05-Funding Research
Introduction: The funding of research (and machine learning research) is an issue which seems to have become more significant in the United States over the last decade. The word “research” is applied broadly here to science, mathematics, and engineering. There are two essential difficulties with funding research: Longshot Paying a researcher is often a big gamble. Most research projects don’t pan out, but a few big payoffs can make it all worthwhile. Information Only Much of research is about finding the right way to think about or do something. The Longshot difficulty means that there is high variance in payoffs. This can be compensated for by funding many different research projects, reducing variance. The Information-Only difficulty means that it’s hard to extract a profit directly from many types of research, so companies have difficulty justifying basic research. (Patents are a mechanism for doing this. They are often extraordinarily clumsy or simply not applicable.) T
3 0.17263129 132 hunch net-2005-11-26-The Design of an Optimal Research Environment
Introduction: How do you create an optimal environment for research? Here are some essential ingredients that I see. Stability . University-based research is relatively good at this. On any particular day, researchers face choices in what they will work on. A very common tradeoff is between: easy small difficult big For researchers without stability, the ‘easy small’ option wins. This is often “ok”—a series of incremental improvements on the state of the art can add up to something very beneficial. However, it misses one of the big potentials of research: finding entirely new and better ways of doing things. Stability comes in many forms. The prototypical example is tenure at a university—a tenured professor is almost imposssible to fire which means that the professor has the freedom to consider far horizon activities. An iron-clad guarantee of a paycheck is not necessary—industrial research labs have succeeded well with research positions of indefinite duration. Atnt rese
4 0.15409045 194 hunch net-2006-07-11-New Models
Introduction: How should we, as researchers in machine learning, organize ourselves? The most immediate measurable objective of computer science research is publishing a paper. The most difficult aspect of publishing a paper is having reviewers accept and recommend it for publication. The simplest mechanism for doing this is to show theoretical progress on some standard, well-known easily understood problem. In doing this, we often fall into a local minima of the research process. The basic problem in machine learning is that it is very unclear that the mathematical model is the right one for the (or some) real problem. A good mathematical model in machine learning should have one fundamental trait: it should aid the design of effective learning algorithms. To date, our ability to solve interesting learning problems (speech recognition, machine translation, object recognition, etc…) remains limited (although improving), so the “rightness” of our models is in doubt. If our mathematical mod
5 0.14681339 207 hunch net-2006-09-12-Incentive Compatible Reviewing
Introduction: Reviewing is a fairly formal process which is integral to the way academia is run. Given this integral nature, the quality of reviewing is often frustrating. I’ve seen plenty of examples of false statements, misbeliefs, reading what isn’t written, etc…, and I’m sure many other people have as well. Recently, mechanisms like double blind review and author feedback have been introduced to try to make the process more fair and accurate in many machine learning (and related) conferences. My personal experience is that these mechanisms help, especially the author feedback. Nevertheless, some problems remain. The game theory take on reviewing is that the incentive for truthful reviewing isn’t there. Since reviewers are also authors, there are sometimes perverse incentives created and acted upon. (Incidentially, these incentives can be both positive and negative.) Setting up a truthful reviewing system is tricky because their is no final reference truth available in any acce
6 0.14665613 98 hunch net-2005-07-27-Not goal metrics
7 0.14597762 454 hunch net-2012-01-30-ICML Posters and Scope
8 0.14256877 344 hunch net-2009-02-22-Effective Research Funding
9 0.13706714 116 hunch net-2005-09-30-Research in conferences
10 0.13330281 343 hunch net-2009-02-18-Decision by Vetocracy
11 0.13199192 484 hunch net-2013-06-16-Representative Reviewing
12 0.12773673 22 hunch net-2005-02-18-What it means to do research.
13 0.11812632 395 hunch net-2010-04-26-Compassionate Reviewing
14 0.11761057 315 hunch net-2008-09-03-Bidding Problems
15 0.1142187 110 hunch net-2005-09-10-“Failure” is an option
16 0.11395882 30 hunch net-2005-02-25-Why Papers?
17 0.11316142 333 hunch net-2008-12-27-Adversarial Academia
18 0.112422 121 hunch net-2005-10-12-The unrealized potential of the research lab
19 0.11089239 233 hunch net-2007-02-16-The Forgetting
20 0.11037668 208 hunch net-2006-09-18-What is missing for online collaborative research?
topicId topicWeight
[(0, 0.24), (1, -0.085), (2, 0.03), (3, 0.182), (4, -0.095), (5, -0.0), (6, 0.061), (7, 0.057), (8, 0.009), (9, 0.063), (10, 0.057), (11, -0.004), (12, -0.036), (13, -0.063), (14, 0.005), (15, 0.037), (16, -0.001), (17, 0.063), (18, -0.017), (19, -0.146), (20, -0.085), (21, -0.042), (22, 0.0), (23, -0.034), (24, 0.035), (25, 0.024), (26, -0.046), (27, -0.02), (28, -0.079), (29, -0.146), (30, 0.045), (31, 0.036), (32, -0.001), (33, 0.054), (34, -0.006), (35, 0.03), (36, 0.011), (37, -0.023), (38, -0.046), (39, -0.024), (40, -0.029), (41, -0.039), (42, -0.092), (43, 0.038), (44, 0.044), (45, -0.043), (46, -0.003), (47, 0.006), (48, 0.026), (49, -0.046)]
simIndex simValue blogId blogTitle
same-blog 1 0.97637802 51 hunch net-2005-04-01-The Producer-Consumer Model of Research
Introduction: In the quest to understand what good reviewing is, perhaps it’s worthwhile to think about what good research is. One way to think about good research is in terms of a producer/consumer model. In the producer/consumer model of research, for any element of research there are producers (authors and coauthors of papers, for example) and consumers (people who use the papers to make new papers or code solving problems). An produced bit of research is judged as “good” if it is used by many consumers. There are two basic questions which immediately arise: Is this a good model of research? Are there alternatives? The producer/consumer model has some difficulties which can be (partially) addressed. Disconnect. A group of people doing research on some subject may become disconnected from the rest of the world. Each person uses the research of other people in the group so it appears good research is being done, but the group has no impact on the rest of the world. One way
2 0.7700448 121 hunch net-2005-10-12-The unrealized potential of the research lab
Introduction: I attended the IBM research 60th anniversary . IBM research is, by any reasonable account, the industrial research lab which has managed to bring the most value to it’s parent company over the long term. This can be seen by simply counting the survivors: IBM research is the only older research lab which has not gone through a period of massive firing. (Note that there are also new research labs .) Despite this impressive record, IBM research has failed, by far, to achieve it’s potential. Examples which came up in this meeting include: It took about a decade to produce DRAM after it was invented in the lab. (In fact, Intel produced it first.) Relational databases and SQL were invented and then languished. It was only under external competition that IBM released it’s own relational database. Why didn’t IBM grow an Oracle division ? An early lead in IP networking hardware did not result in IBM growing a Cisco division . Why not? And remember … IBM research is a s
3 0.75044632 132 hunch net-2005-11-26-The Design of an Optimal Research Environment
Introduction: How do you create an optimal environment for research? Here are some essential ingredients that I see. Stability . University-based research is relatively good at this. On any particular day, researchers face choices in what they will work on. A very common tradeoff is between: easy small difficult big For researchers without stability, the ‘easy small’ option wins. This is often “ok”—a series of incremental improvements on the state of the art can add up to something very beneficial. However, it misses one of the big potentials of research: finding entirely new and better ways of doing things. Stability comes in many forms. The prototypical example is tenure at a university—a tenured professor is almost imposssible to fire which means that the professor has the freedom to consider far horizon activities. An iron-clad guarantee of a paycheck is not necessary—industrial research labs have succeeded well with research positions of indefinite duration. Atnt rese
4 0.74465066 344 hunch net-2009-02-22-Effective Research Funding
Introduction: With a worldwide recession on, my impression is that the carnage in research has not been as severe as might be feared, at least in the United States. I know of two notable negative impacts: It’s quite difficult to get a job this year, as many companies and universities simply aren’t hiring. This is particularly tough on graduating students. Perhaps 10% of IBM research was fired. In contrast, around the time of the dot com bust, ATnT Research and Lucent had one or several 50% size firings wiping out much of the remainder of Bell Labs , triggering a notable diaspora for the respected machine learning group there. As the recession progresses, we may easily see more firings as companies in particular reach a point where they can no longer support research. There are a couple positives to the recession as well. Both the implosion of Wall Street (which siphoned off smart people) and the general difficulty of getting a job coming out of an undergraduate education s
5 0.73544753 194 hunch net-2006-07-11-New Models
Introduction: How should we, as researchers in machine learning, organize ourselves? The most immediate measurable objective of computer science research is publishing a paper. The most difficult aspect of publishing a paper is having reviewers accept and recommend it for publication. The simplest mechanism for doing this is to show theoretical progress on some standard, well-known easily understood problem. In doing this, we often fall into a local minima of the research process. The basic problem in machine learning is that it is very unclear that the mathematical model is the right one for the (or some) real problem. A good mathematical model in machine learning should have one fundamental trait: it should aid the design of effective learning algorithms. To date, our ability to solve interesting learning problems (speech recognition, machine translation, object recognition, etc…) remains limited (although improving), so the “rightness” of our models is in doubt. If our mathematical mod
6 0.73198515 36 hunch net-2005-03-05-Funding Research
7 0.70098323 76 hunch net-2005-05-29-Bad ideas
8 0.68867731 208 hunch net-2006-09-18-What is missing for online collaborative research?
9 0.68155748 333 hunch net-2008-12-27-Adversarial Academia
10 0.67715454 98 hunch net-2005-07-27-Not goal metrics
11 0.66303211 22 hunch net-2005-02-18-What it means to do research.
12 0.64772809 485 hunch net-2013-06-29-The Benefits of Double-Blind Review
13 0.64299941 255 hunch net-2007-07-13-The View From China
14 0.63808721 110 hunch net-2005-09-10-“Failure” is an option
15 0.63295168 73 hunch net-2005-05-17-A Short Guide to PhD Graduate Study
16 0.62319672 172 hunch net-2006-04-14-JMLR is a success
17 0.61468059 358 hunch net-2009-06-01-Multitask Poisoning
18 0.61308682 48 hunch net-2005-03-29-Academic Mechanism Design
19 0.60961699 142 hunch net-2005-12-22-Yes , I am applying
20 0.60572255 449 hunch net-2011-11-26-Giving Thanks
topicId topicWeight
[(10, 0.061), (13, 0.226), (27, 0.213), (38, 0.047), (53, 0.017), (55, 0.161), (61, 0.017), (94, 0.097), (95, 0.066)]
simIndex simValue blogId blogTitle
1 0.93546277 117 hunch net-2005-10-03-Not ICML
Introduction: Alex Smola showed me this ICML 2006 webpage. This is NOT the ICML we know, but rather some people at “Enformatika”. Investigation shows that they registered with an anonymous yahoo email account from dotregistrar.com the “Home of the $6.79 wholesale domain!” and their nameservers are by Turkticaret , a Turkish internet company. It appears the website has since been altered to “ ICNL ” (the above link uses the google cache). They say that imitation is the sincerest form of flattery, so the organizers of the real ICML 2006 must feel quite flattered.
2 0.9260422 212 hunch net-2006-10-04-Health of Conferences Wiki
Introduction: Aaron Hertzmann points out the health of conferences wiki , which has a great deal of information about how many different conferences function.
3 0.89281178 386 hunch net-2010-01-13-Sam Roweis died
Introduction: and I can’t help but remember him. I first met Sam as an undergraduate at Caltech where he was TA for Hopfield ‘s class, and again when I visited Gatsby , when he invited me to visit Toronto , and at too many conferences to recount. His personality was a combination of enthusiastic and thoughtful, with a great ability to phrase a problem so it’s solution must be understood. With respect to my own work, Sam was the one who advised me to make my first tutorial , leading to others, and to other things, all of which I’m grateful to him for. In fact, my every interaction with Sam was positive, and that was his way. His death is being called a suicide which is so incompatible with my understanding of Sam that it strains my credibility. But we know that his many responsibilities were great, and it is well understood that basically all sane researchers have legions of inner doubts. Having been depressed now and then myself, it’s helpful to understand at least intellectually
4 0.890149 137 hunch net-2005-12-09-Machine Learning Thoughts
Introduction: I added a link to Olivier Bousquet’s machine learning thoughts blog. Several of the posts may be of interest.
same-blog 5 0.87056619 51 hunch net-2005-04-01-The Producer-Consumer Model of Research
Introduction: In the quest to understand what good reviewing is, perhaps it’s worthwhile to think about what good research is. One way to think about good research is in terms of a producer/consumer model. In the producer/consumer model of research, for any element of research there are producers (authors and coauthors of papers, for example) and consumers (people who use the papers to make new papers or code solving problems). An produced bit of research is judged as “good” if it is used by many consumers. There are two basic questions which immediately arise: Is this a good model of research? Are there alternatives? The producer/consumer model has some difficulties which can be (partially) addressed. Disconnect. A group of people doing research on some subject may become disconnected from the rest of the world. Each person uses the research of other people in the group so it appears good research is being done, but the group has no impact on the rest of the world. One way
6 0.85997319 492 hunch net-2013-12-01-NIPS tutorials and Vowpal Wabbit 7.4
7 0.85740119 406 hunch net-2010-08-22-KDD 2010
8 0.77907884 454 hunch net-2012-01-30-ICML Posters and Scope
9 0.77282268 379 hunch net-2009-11-23-ICML 2009 Workshops (and Tutorials)
10 0.76871318 343 hunch net-2009-02-18-Decision by Vetocracy
11 0.76608145 40 hunch net-2005-03-13-Avoiding Bad Reviewing
12 0.76354671 437 hunch net-2011-07-10-ICML 2011 and the future
13 0.7624346 464 hunch net-2012-05-03-Microsoft Research, New York City
14 0.75266671 204 hunch net-2006-08-28-Learning Theory standards for NIPS 2006
15 0.75055051 320 hunch net-2008-10-14-Who is Responsible for a Bad Review?
16 0.74895138 132 hunch net-2005-11-26-The Design of an Optimal Research Environment
17 0.74892026 423 hunch net-2011-02-02-User preferences for search engines
18 0.74848771 325 hunch net-2008-11-10-ICML Reviewing Criteria
19 0.747769 89 hunch net-2005-07-04-The Health of COLT
20 0.74731481 419 hunch net-2010-12-04-Vowpal Wabbit, version 5.0, and the second heresy