hunch_net hunch_net-2005 hunch_net-2005-48 knowledge-graph by maker-knowledge-mining
Source: html
Introduction: From game theory, there is a notion of “mechanism design”: setting up the structure of the world so that participants have some incentive to do sane things (rather than obviously counterproductive things). Application of this principle to academic research may be fruitful. What is misdesigned about academic research? The JMLG guides give many hints. The common nature of bad reviewing also suggests the system isn’t working optimally. There are many ways to experimentally “cheat” in machine learning . Funding Prisoner’s Delimma. Good researchers often write grant proposals for funding rather than doing research. Since the pool of grant money is finite, this means that grant proposals are often rejected, implying that more must be written. This is essentially a “prisoner’s delimma”: anyone not writing grant proposals loses, but the entire process of doing research is slowed by distraction. If everyone wrote 1/2 as many grant proposals, roughly the same distribution
sentIndex sentText sentNum sentScore
1 From game theory, there is a notion of “mechanism design”: setting up the structure of the world so that participants have some incentive to do sane things (rather than obviously counterproductive things). [sent-1, score-0.492]
2 Application of this principle to academic research may be fruitful. [sent-2, score-0.386]
3 The common nature of bad reviewing also suggests the system isn’t working optimally. [sent-5, score-0.219]
4 There are many ways to experimentally “cheat” in machine learning . [sent-6, score-0.194]
5 Good researchers often write grant proposals for funding rather than doing research. [sent-8, score-1.2]
6 Since the pool of grant money is finite, this means that grant proposals are often rejected, implying that more must be written. [sent-9, score-1.504]
7 This is essentially a “prisoner’s delimma”: anyone not writing grant proposals loses, but the entire process of doing research is slowed by distraction. [sent-10, score-1.264]
8 If everyone wrote 1/2 as many grant proposals, roughly the same distribution of funding would occur, and time would be freed for more research. [sent-11, score-0.948]
9 Mechanism design is not that easy—many counterintuitive effects can occur. [sent-12, score-0.359]
10 Academic mechanism design is particularly difficult problem because there are many details. [sent-13, score-0.39]
11 Nevertheless, it may be worthwhile because it’s hard to underestimate the value of an improvement in the rate of useful research. [sent-14, score-0.275]
12 The good news is that not everything needs to be solved at once. [sent-15, score-0.209]
13 For example, on the empirical side, if we setup an easy system allowing anyone to create challenges like KDDCup , we might achieve a better (i. [sent-16, score-0.491]
wordName wordTfidf (topN-words)
[('grant', 0.449), ('proposals', 0.369), ('prisoner', 0.285), ('funding', 0.237), ('academic', 0.212), ('mechanism', 0.161), ('design', 0.152), ('counterintuitive', 0.127), ('loses', 0.127), ('underestimate', 0.127), ('kddcup', 0.117), ('experimentally', 0.117), ('slowed', 0.111), ('wrote', 0.111), ('guides', 0.106), ('occur', 0.106), ('sane', 0.106), ('anyone', 0.105), ('incentive', 0.095), ('pool', 0.092), ('principle', 0.09), ('challenges', 0.088), ('research', 0.084), ('system', 0.082), ('rejected', 0.08), ('effects', 0.08), ('game', 0.079), ('worthwhile', 0.078), ('money', 0.078), ('many', 0.077), ('entire', 0.076), ('easy', 0.076), ('roughly', 0.074), ('finite', 0.074), ('participants', 0.074), ('news', 0.073), ('write', 0.073), ('rather', 0.072), ('obviously', 0.072), ('setup', 0.071), ('nature', 0.07), ('improvement', 0.07), ('writing', 0.07), ('allowing', 0.069), ('needs', 0.068), ('everything', 0.068), ('side', 0.068), ('implying', 0.067), ('suggests', 0.067), ('things', 0.066)]
simIndex simValue blogId blogTitle
same-blog 1 1.0 48 hunch net-2005-03-29-Academic Mechanism Design
Introduction: From game theory, there is a notion of “mechanism design”: setting up the structure of the world so that participants have some incentive to do sane things (rather than obviously counterproductive things). Application of this principle to academic research may be fruitful. What is misdesigned about academic research? The JMLG guides give many hints. The common nature of bad reviewing also suggests the system isn’t working optimally. There are many ways to experimentally “cheat” in machine learning . Funding Prisoner’s Delimma. Good researchers often write grant proposals for funding rather than doing research. Since the pool of grant money is finite, this means that grant proposals are often rejected, implying that more must be written. This is essentially a “prisoner’s delimma”: anyone not writing grant proposals loses, but the entire process of doing research is slowed by distraction. If everyone wrote 1/2 as many grant proposals, roughly the same distribution
2 0.24625966 425 hunch net-2011-02-25-Yahoo! Machine Learning grant due March 11
Introduction: Yahoo!’s Key Scientific Challenges for Machine Learning grant applications are due March 11. If you are a student working on relevant research, please consider applying. It’s for $5K of unrestricted funding.
3 0.16601463 132 hunch net-2005-11-26-The Design of an Optimal Research Environment
Introduction: How do you create an optimal environment for research? Here are some essential ingredients that I see. Stability . University-based research is relatively good at this. On any particular day, researchers face choices in what they will work on. A very common tradeoff is between: easy small difficult big For researchers without stability, the ‘easy small’ option wins. This is often “ok”—a series of incremental improvements on the state of the art can add up to something very beneficial. However, it misses one of the big potentials of research: finding entirely new and better ways of doing things. Stability comes in many forms. The prototypical example is tenure at a university—a tenured professor is almost imposssible to fire which means that the professor has the freedom to consider far horizon activities. An iron-clad guarantee of a paycheck is not necessary—industrial research labs have succeeded well with research positions of indefinite duration. Atnt rese
4 0.16483687 36 hunch net-2005-03-05-Funding Research
Introduction: The funding of research (and machine learning research) is an issue which seems to have become more significant in the United States over the last decade. The word “research” is applied broadly here to science, mathematics, and engineering. There are two essential difficulties with funding research: Longshot Paying a researcher is often a big gamble. Most research projects don’t pan out, but a few big payoffs can make it all worthwhile. Information Only Much of research is about finding the right way to think about or do something. The Longshot difficulty means that there is high variance in payoffs. This can be compensated for by funding many different research projects, reducing variance. The Information-Only difficulty means that it’s hard to extract a profit directly from many types of research, so companies have difficulty justifying basic research. (Patents are a mechanism for doing this. They are often extraordinarily clumsy or simply not applicable.) T
5 0.1432298 344 hunch net-2009-02-22-Effective Research Funding
Introduction: With a worldwide recession on, my impression is that the carnage in research has not been as severe as might be feared, at least in the United States. I know of two notable negative impacts: It’s quite difficult to get a job this year, as many companies and universities simply aren’t hiring. This is particularly tough on graduating students. Perhaps 10% of IBM research was fired. In contrast, around the time of the dot com bust, ATnT Research and Lucent had one or several 50% size firings wiping out much of the remainder of Bell Labs , triggering a notable diaspora for the respected machine learning group there. As the recession progresses, we may easily see more firings as companies in particular reach a point where they can no longer support research. There are a couple positives to the recession as well. Both the implosion of Wall Street (which siphoned off smart people) and the general difficulty of getting a job coming out of an undergraduate education s
6 0.13648254 345 hunch net-2009-03-08-Prediction Science
7 0.12850007 110 hunch net-2005-09-10-“Failure” is an option
8 0.125531 382 hunch net-2009-12-09-Future Publication Models @ NIPS
9 0.12492055 50 hunch net-2005-04-01-Basic computer science research takes a hit
10 0.11019739 154 hunch net-2006-02-04-Research Budget Changes
11 0.10526969 207 hunch net-2006-09-12-Incentive Compatible Reviewing
12 0.099509872 98 hunch net-2005-07-27-Not goal metrics
13 0.096164435 105 hunch net-2005-08-23-(Dis)similarities between academia and open source programmers
14 0.090000831 113 hunch net-2005-09-19-NIPS Workshops
15 0.08948452 225 hunch net-2007-01-02-Retrospective
16 0.089109793 475 hunch net-2012-10-26-ML Symposium and Strata-Hadoop World
17 0.084009402 395 hunch net-2010-04-26-Compassionate Reviewing
18 0.081339501 454 hunch net-2012-01-30-ICML Posters and Scope
19 0.079396784 343 hunch net-2009-02-18-Decision by Vetocracy
20 0.077044271 106 hunch net-2005-09-04-Science in the Government
topicId topicWeight
[(0, 0.184), (1, -0.042), (2, -0.067), (3, 0.132), (4, -0.131), (5, -0.01), (6, 0.06), (7, 0.074), (8, -0.035), (9, 0.05), (10, 0.06), (11, 0.007), (12, -0.003), (13, -0.041), (14, 0.027), (15, 0.044), (16, -0.093), (17, -0.007), (18, -0.068), (19, -0.012), (20, -0.062), (21, -0.038), (22, 0.025), (23, -0.005), (24, 0.004), (25, -0.011), (26, -0.044), (27, 0.024), (28, 0.048), (29, -0.041), (30, -0.044), (31, 0.044), (32, 0.099), (33, -0.036), (34, -0.016), (35, -0.102), (36, -0.135), (37, 0.02), (38, 0.029), (39, -0.069), (40, -0.092), (41, -0.067), (42, 0.035), (43, -0.022), (44, 0.008), (45, 0.041), (46, -0.032), (47, -0.044), (48, -0.045), (49, 0.038)]
simIndex simValue blogId blogTitle
same-blog 1 0.95598769 48 hunch net-2005-03-29-Academic Mechanism Design
Introduction: From game theory, there is a notion of “mechanism design”: setting up the structure of the world so that participants have some incentive to do sane things (rather than obviously counterproductive things). Application of this principle to academic research may be fruitful. What is misdesigned about academic research? The JMLG guides give many hints. The common nature of bad reviewing also suggests the system isn’t working optimally. There are many ways to experimentally “cheat” in machine learning . Funding Prisoner’s Delimma. Good researchers often write grant proposals for funding rather than doing research. Since the pool of grant money is finite, this means that grant proposals are often rejected, implying that more must be written. This is essentially a “prisoner’s delimma”: anyone not writing grant proposals loses, but the entire process of doing research is slowed by distraction. If everyone wrote 1/2 as many grant proposals, roughly the same distribution
2 0.70167148 154 hunch net-2006-02-04-Research Budget Changes
Introduction: The announcement of an increase in funding for basic research in the US is encouraging. There is some discussion of this at the Computing Research Policy blog. One part of this discussion has a graph of NSF funding over time, presumably in dollar budgets. I don’t believe that dollar budgets are the right way to judge the impact of funding changes on researchers. A better way to judge seems to be in terms of dollar budget divided by GDP which provides a measure of the relative emphasis on research. This graph was assembled by dividing the NSF budget by the US GDP . For 2005 GDP, I used the current estimate and for 2006 and 2007 assumed an increase by a factor of 1.04 per year. The 2007 number also uses the requested 2007 budget which is certain to change. This graph makes it clear why researchers were upset: research funding emphasis has fallen for 3 years in a row. The reality has been significantly more severe due to DARPA decreasing funding and industrial
3 0.68173784 36 hunch net-2005-03-05-Funding Research
Introduction: The funding of research (and machine learning research) is an issue which seems to have become more significant in the United States over the last decade. The word “research” is applied broadly here to science, mathematics, and engineering. There are two essential difficulties with funding research: Longshot Paying a researcher is often a big gamble. Most research projects don’t pan out, but a few big payoffs can make it all worthwhile. Information Only Much of research is about finding the right way to think about or do something. The Longshot difficulty means that there is high variance in payoffs. This can be compensated for by funding many different research projects, reducing variance. The Information-Only difficulty means that it’s hard to extract a profit directly from many types of research, so companies have difficulty justifying basic research. (Patents are a mechanism for doing this. They are often extraordinarily clumsy or simply not applicable.) T
4 0.66582996 344 hunch net-2009-02-22-Effective Research Funding
Introduction: With a worldwide recession on, my impression is that the carnage in research has not been as severe as might be feared, at least in the United States. I know of two notable negative impacts: It’s quite difficult to get a job this year, as many companies and universities simply aren’t hiring. This is particularly tough on graduating students. Perhaps 10% of IBM research was fired. In contrast, around the time of the dot com bust, ATnT Research and Lucent had one or several 50% size firings wiping out much of the remainder of Bell Labs , triggering a notable diaspora for the respected machine learning group there. As the recession progresses, we may easily see more firings as companies in particular reach a point where they can no longer support research. There are a couple positives to the recession as well. Both the implosion of Wall Street (which siphoned off smart people) and the general difficulty of getting a job coming out of an undergraduate education s
5 0.6618036 241 hunch net-2007-04-28-The Coming Patent Apocalypse
Introduction: Many people in computer science believe that patents are problematic. The truth is even worse—the patent system in the US is fundamentally broken in ways that will require much more significant reform than is being considered now . The myth of the patent is the following: Patents are a mechanism for inventors to be compensated according to the value of their inventions while making the invention available to all. This myth sounds pretty desirable, but the reality is a strange distortion slowly leading towards collapse. There are many problems associated with patents, but I would like to focus on just two of them: Patent Trolls The way that patents have generally worked over the last several decades is that they were a tool of large companies. Large companies would amass a large number of patents and then cross-license each other’s patents—in effect saying “we agree to owe each other nothing”. Smaller companies would sometimes lose in this game, essentially because they
6 0.64075637 50 hunch net-2005-04-01-Basic computer science research takes a hit
7 0.62525803 132 hunch net-2005-11-26-The Design of an Optimal Research Environment
8 0.62350827 106 hunch net-2005-09-04-Science in the Government
9 0.61998641 110 hunch net-2005-09-10-“Failure” is an option
10 0.55814558 73 hunch net-2005-05-17-A Short Guide to PhD Graduate Study
11 0.54909378 105 hunch net-2005-08-23-(Dis)similarities between academia and open source programmers
12 0.54439616 449 hunch net-2011-11-26-Giving Thanks
13 0.54431576 425 hunch net-2011-02-25-Yahoo! Machine Learning grant due March 11
14 0.53930295 51 hunch net-2005-04-01-The Producer-Consumer Model of Research
15 0.53096968 121 hunch net-2005-10-12-The unrealized potential of the research lab
16 0.52681476 485 hunch net-2013-06-29-The Benefits of Double-Blind Review
17 0.52643782 1 hunch net-2005-01-19-Why I decided to run a weblog.
18 0.51019132 282 hunch net-2008-01-06-Research Political Issues
19 0.50718635 335 hunch net-2009-01-08-Predictive Analytics World
20 0.48736295 142 hunch net-2005-12-22-Yes , I am applying
topicId topicWeight
[(10, 0.029), (27, 0.192), (38, 0.046), (53, 0.033), (55, 0.081), (68, 0.347), (94, 0.086), (95, 0.08)]
simIndex simValue blogId blogTitle
same-blog 1 0.90039486 48 hunch net-2005-03-29-Academic Mechanism Design
Introduction: From game theory, there is a notion of “mechanism design”: setting up the structure of the world so that participants have some incentive to do sane things (rather than obviously counterproductive things). Application of this principle to academic research may be fruitful. What is misdesigned about academic research? The JMLG guides give many hints. The common nature of bad reviewing also suggests the system isn’t working optimally. There are many ways to experimentally “cheat” in machine learning . Funding Prisoner’s Delimma. Good researchers often write grant proposals for funding rather than doing research. Since the pool of grant money is finite, this means that grant proposals are often rejected, implying that more must be written. This is essentially a “prisoner’s delimma”: anyone not writing grant proposals loses, but the entire process of doing research is slowed by distraction. If everyone wrote 1/2 as many grant proposals, roughly the same distribution
2 0.86116242 428 hunch net-2011-03-27-Vowpal Wabbit, v5.1
Introduction: I just created version 5.1 of vowpal wabbit . This almost entirely a bugfix release, so it’s an easy upgrade from v5.0. In addition: There is now a mailing list , which I and several other developers are subscribed to. The main website has shifted to the wiki on github. This means that anyone with a github account can now edit it. I’m planning to give a tutorial tomorrow on it at eHarmony / the LA machine learning meetup at 10am. Drop by if you’re interested. The status of VW amongst other open source projects has changed. When VW first came out, it was relatively unique amongst existing projects in terms of features. At this point, many other projects have started to appreciate the value of the design choices here. This includes: Mahout , which now has an SGD implementation. Shogun , where Soeren is keen on incorporating features . LibLinear , where they won the KDD best paper award for out-of-core learning . This is expected—any open sourc
3 0.84655905 110 hunch net-2005-09-10-“Failure” is an option
Introduction: This is about the hard choices that graduate students must make. The cultural definition of success in academic research is to: Produce good research which many other people appreciate. Produce many students who go on to do the same. There are fundamental reasons why this is success in the local culture. Good research appreciated by others means access to jobs. Many students succesful in the same way implies that there are a number of people who think in a similar way and appreciate your work. In order to graduate, a phd student must live in an academic culture for a period of several years. It is common to adopt the culture’s definition of success during this time. It’s also common for many phd students discover they are not suited to an academic research lifestyle. This collision of values and abilities naturally results in depression. The most fundamental advice when this happens is: change something. Pick a new advisor. Pick a new research topic. Or leave th
4 0.80888343 156 hunch net-2006-02-11-Yahoo’s Learning Problems.
Introduction: I just visited Yahoo Research which has several fundamental learning problems near to (or beyond) the set of problems we know how to solve well. Here are 3 of them. Ranking This is the canonical problem of all search engines. It is made extra difficult for several reasons. There is relatively little “good” supervised learning data and a great deal of data with some signal (such as click through rates). The learning must occur in a partially adversarial environment. Many people very actively attempt to place themselves at the top of rankings. It is not even quite clear whether the problem should be posed as ‘ranking’ or as ‘regression’ which is then used to produce a ranking. Collaborative filtering Yahoo has a large number of recommendation systems for music, movies, etc… In these sorts of systems, users specify how they liked a set of things, and then the system can (hopefully) find some more examples of things they might like by reasoning across multiple
5 0.71219873 345 hunch net-2009-03-08-Prediction Science
Introduction: One view of machine learning is that it’s about how to program computers to predict well. This suggests a broader research program centered around the more pervasive goal of simply predicting well. There are many distinct strands of this broader research program which are only partially unified. Here are the ones that I know of: Learning Theory . Learning theory focuses on several topics related to the dynamics and process of prediction. Convergence bounds like the VC bound give an intellectual foundation to many learning algorithms. Online learning algorithms like Weighted Majority provide an alternate purely game theoretic foundation for learning. Boosting algorithms yield algorithms for purifying prediction abiliity. Reduction algorithms provide means for changing esoteric problems into well known ones. Machine Learning . A great deal of experience has accumulated in practical algorithm design from a mixture of paradigms, including bayesian, biological, opt
6 0.69982141 65 hunch net-2005-05-02-Reviewing techniques for conferences
7 0.59784722 225 hunch net-2007-01-02-Retrospective
8 0.59113955 132 hunch net-2005-11-26-The Design of an Optimal Research Environment
9 0.58560425 464 hunch net-2012-05-03-Microsoft Research, New York City
10 0.57999247 411 hunch net-2010-09-21-Regretting the dead
11 0.57390982 207 hunch net-2006-09-12-Incentive Compatible Reviewing
12 0.57237899 382 hunch net-2009-12-09-Future Publication Models @ NIPS
13 0.56588423 358 hunch net-2009-06-01-Multitask Poisoning
14 0.56487238 360 hunch net-2009-06-15-In Active Learning, the question changes
15 0.56394231 366 hunch net-2009-08-03-Carbon in Computer Science Research
16 0.56297886 449 hunch net-2011-11-26-Giving Thanks
17 0.56232131 343 hunch net-2009-02-18-Decision by Vetocracy
18 0.55969554 105 hunch net-2005-08-23-(Dis)similarities between academia and open source programmers
19 0.5589335 36 hunch net-2005-03-05-Funding Research
20 0.55786341 95 hunch net-2005-07-14-What Learning Theory might do