hunch_net hunch_net-2009 hunch_net-2009-370 knowledge-graph by maker-knowledge-mining
Source: html
Introduction: Researchers are typically confronted with big problems that they have no idea how to solve. In trying to come up with a solution, a natural approach is to decompose the big problem into a set of subproblems whose solution yields a solution to the larger problem. This approach can go wrong in several ways. Decomposition failure . The solution to the decomposition does not in fact yield a solution to the overall problem. Artificial hardness . The subproblems created are sufficient if solved to solve the overall problem, but they are harder than necessary. As you can see, computational complexity forms a relatively new (in research-history) razor by which to judge an approach sufficient but not necessary. In my experience, the artificial hardness problem is very common. Many researchers abdicate the responsibility of choosing a problem to work on to other people. This process starts very naturally as a graduate student, when an incoming student might have relatively l
sentIndex sentText sentNum sentScore
1 In trying to come up with a solution, a natural approach is to decompose the big problem into a set of subproblems whose solution yields a solution to the larger problem. [sent-2, score-0.626]
2 The subproblems created are sufficient if solved to solve the overall problem, but they are harder than necessary. [sent-7, score-0.62]
3 As you can see, computational complexity forms a relatively new (in research-history) razor by which to judge an approach sufficient but not necessary. [sent-8, score-0.51]
4 This process starts very naturally as a graduate student, when an incoming student might have relatively little idea about how to do research, so they naturally abdicate the problem choice to an advisor. [sent-11, score-0.57]
5 In contrast to sufficient subgoals of a greater goal, there are also necessary subgoals. [sent-14, score-1.15]
6 A necessary subgoal is one which must be solved to solve the greater goal. [sent-15, score-1.114]
7 One of the reasons why the artificial hardness problem is so common is that the sufficient subgoals are commonly confused with necessary subgoals. [sent-16, score-1.137]
8 The essential test for a necessary subgoal is whether or not a solution to the global problem can be used as a solution to the subgoal. [sent-17, score-0.922]
9 My personal greater goal is creating a master machine learning algorithm that can solve any reasonable learning problem where “reasonable” includes at least the set that humans can solve. [sent-18, score-0.92]
10 Relative to this greater goal, many existing research programs do not appear necessary. [sent-19, score-0.523]
11 While we don’t stick children into a sensory deprivation tank to see how much it retards their ability to solve problems when grown, some experiments along these lines have been done with animals yielding obvious ability deficiency. [sent-26, score-0.479]
12 The ability to learn in an online environment with relatively little processing per bit of input is clearly a sufficient approach to solve many problems. [sent-28, score-0.831]
13 We can further argue the necessity by pointing out that interactive proofs appear much more powerful in computational complexity theory than noninteractive proofs. [sent-33, score-0.603]
14 The necessity of compositional design in machine learning is not entirely clear. [sent-36, score-0.488]
15 Nevertheless, since our basic goal in research is a much more efficient and faster design, it seems that the decision to take a research-based approach implies that compositional design is necessary. [sent-38, score-0.662]
16 It’s necessary that a learning algorithm be able to use a relatively large number of bits when making a decision. [sent-41, score-0.497]
17 If you have other arguments for what is or is not necessary for this greater goal, please speak up. [sent-46, score-0.746]
18 There are subgoals which are necessary and sufficient (in combination) to solve the greater goal. [sent-48, score-1.299]
19 The fourth category is subgoals which are neither necessary nor sufficient for a greater goal. [sent-50, score-1.15]
20 In general, decision making is pretty terrible because greater goals are rarely stated, perhaps as a form of strategic ambiguity. [sent-54, score-0.484]
wordName wordTfidf (topN-words)
[('greater', 0.367), ('necessary', 0.307), ('sufficient', 0.262), ('subgoal', 0.24), ('subgoals', 0.214), ('necessity', 0.2), ('compositional', 0.16), ('solve', 0.149), ('solution', 0.139), ('interactive', 0.138), ('artificial', 0.133), ('goal', 0.133), ('design', 0.128), ('hardness', 0.124), ('abdicate', 0.12), ('decomposition', 0.107), ('research', 0.106), ('animals', 0.099), ('problem', 0.097), ('decompose', 0.093), ('student', 0.092), ('ability', 0.088), ('relatively', 0.086), ('computational', 0.082), ('leaving', 0.08), ('approach', 0.08), ('overall', 0.08), ('subproblems', 0.078), ('argue', 0.078), ('arguments', 0.072), ('graduate', 0.069), ('nevertheless', 0.068), ('form', 0.067), ('humans', 0.064), ('personal', 0.059), ('clearly', 0.057), ('input', 0.056), ('much', 0.055), ('criticizable', 0.053), ('confronted', 0.053), ('periodically', 0.053), ('able', 0.053), ('naturally', 0.053), ('learn', 0.053), ('creating', 0.051), ('researchers', 0.051), ('solved', 0.051), ('large', 0.051), ('perhaps', 0.05), ('appear', 0.05)]
simIndex simValue blogId blogTitle
same-blog 1 0.99999994 370 hunch net-2009-09-18-Necessary and Sufficient Research
Introduction: Researchers are typically confronted with big problems that they have no idea how to solve. In trying to come up with a solution, a natural approach is to decompose the big problem into a set of subproblems whose solution yields a solution to the larger problem. This approach can go wrong in several ways. Decomposition failure . The solution to the decomposition does not in fact yield a solution to the overall problem. Artificial hardness . The subproblems created are sufficient if solved to solve the overall problem, but they are harder than necessary. As you can see, computational complexity forms a relatively new (in research-history) razor by which to judge an approach sufficient but not necessary. In my experience, the artificial hardness problem is very common. Many researchers abdicate the responsibility of choosing a problem to work on to other people. This process starts very naturally as a graduate student, when an incoming student might have relatively l
2 0.16115201 132 hunch net-2005-11-26-The Design of an Optimal Research Environment
Introduction: How do you create an optimal environment for research? Here are some essential ingredients that I see. Stability . University-based research is relatively good at this. On any particular day, researchers face choices in what they will work on. A very common tradeoff is between: easy small difficult big For researchers without stability, the ‘easy small’ option wins. This is often “ok”—a series of incremental improvements on the state of the art can add up to something very beneficial. However, it misses one of the big potentials of research: finding entirely new and better ways of doing things. Stability comes in many forms. The prototypical example is tenure at a university—a tenured professor is almost imposssible to fire which means that the professor has the freedom to consider far horizon activities. An iron-clad guarantee of a paycheck is not necessary—industrial research labs have succeeded well with research positions of indefinite duration. Atnt rese
3 0.14966488 22 hunch net-2005-02-18-What it means to do research.
Introduction: I want to try to describe what doing research means, especially from the point of view of an undergraduate. The shift from a class-taking mentality to a research mentality is very significant and not easy. Problem Posing Posing the right problem is often as important as solving them. Many people can get by in research by solving problems others have posed, but that’s not sufficient for really inspiring research. For learning in particular, there is a strong feeling that we just haven’t figured out which questions are the right ones to ask. You can see this, because the answers we have do not seem convincing. Gambling your life When you do research, you think very hard about new ways of solving problems, new problems, and new solutions. Many conversations are of the form “I wonder what would happen if…” These processes can be short (days or weeks) or years-long endeavours. The worst part is that you’ll only know if you were succesful at the end of the process (and some
4 0.14948905 293 hunch net-2008-03-23-Interactive Machine Learning
Introduction: A new direction of research seems to be arising in machine learning: Interactive Machine Learning. This isn’t a familiar term, although it does include some familiar subjects. What is Interactive Machine Learning? The fundamental requirement is (a) learning algorithms which interact with the world and (b) learn. For our purposes, let’s define learning as efficiently competing with a large set of possible predictors. Examples include: Online learning against an adversary ( Avrim’s Notes ). The interaction is almost trivial: the learning algorithm makes a prediction and then receives feedback. The learning is choosing based upon the advice of many experts. Active Learning . In active learning, the interaction is choosing which examples to label, and the learning is choosing from amongst a large set of hypotheses. Contextual Bandits . The interaction is choosing one of several actions and learning only the value of the chosen action (weaker than active learning
5 0.13429165 435 hunch net-2011-05-16-Research Directions for Machine Learning and Algorithms
Introduction: Muthu invited me to the workshop on algorithms in the field , with the goal of providing a sense of where near-term research should go. When the time came though, I bargained for a post instead, which provides a chance for many other people to comment. There are several things I didn’t fully understand when I went to Yahoo! about 5 years ago. I’d like to repeat them as people in academia may not yet understand them intuitively. Almost all the big impact algorithms operate in pseudo-linear or better time. Think about caching, hashing, sorting, filtering, etc… and you have a sense of what some of the most heavily used algorithms are. This matters quite a bit to Machine Learning research, because people often work with superlinear time algorithms and languages. Two very common examples of this are graphical models, where inference is often a superlinear operation—think about the n 2 dependence on the number of states in a Hidden Markov Model and Kernelized Support Vecto
6 0.13246672 454 hunch net-2012-01-30-ICML Posters and Scope
7 0.12898639 332 hunch net-2008-12-23-Use of Learning Theory
8 0.12544337 378 hunch net-2009-11-15-The Other Online Learning
9 0.12247958 412 hunch net-2010-09-28-Machined Learnings
10 0.11675611 343 hunch net-2009-02-18-Decision by Vetocracy
11 0.11260829 359 hunch net-2009-06-03-Functionally defined Nonlinear Dynamic Models
12 0.11240716 67 hunch net-2005-05-06-Don’t mix the solution into the problem
13 0.11072702 351 hunch net-2009-05-02-Wielding a New Abstraction
14 0.11017254 347 hunch net-2009-03-26-Machine Learning is too easy
15 0.10871885 68 hunch net-2005-05-10-Learning Reductions are Reductionist
16 0.10702597 345 hunch net-2009-03-08-Prediction Science
17 0.10516048 376 hunch net-2009-11-06-Yisong Yue on Self-improving Systems
18 0.10368761 358 hunch net-2009-06-01-Multitask Poisoning
19 0.10357618 388 hunch net-2010-01-24-Specializations of the Master Problem
20 0.10355983 352 hunch net-2009-05-06-Machine Learning to AI
topicId topicWeight
[(0, 0.284), (1, 0.047), (2, -0.07), (3, 0.118), (4, -0.036), (5, -0.047), (6, 0.067), (7, 0.044), (8, 0.017), (9, 0.061), (10, -0.011), (11, -0.038), (12, -0.008), (13, 0.11), (14, -0.005), (15, 0.056), (16, 0.038), (17, 0.002), (18, -0.033), (19, 0.057), (20, -0.008), (21, 0.01), (22, 0.004), (23, 0.043), (24, 0.036), (25, -0.019), (26, 0.043), (27, -0.057), (28, 0.037), (29, -0.037), (30, -0.02), (31, -0.002), (32, -0.011), (33, -0.043), (34, -0.03), (35, 0.008), (36, 0.016), (37, -0.006), (38, -0.032), (39, 0.081), (40, 0.08), (41, -0.005), (42, 0.041), (43, 0.048), (44, -0.006), (45, -0.048), (46, -0.128), (47, 0.046), (48, -0.01), (49, 0.003)]
simIndex simValue blogId blogTitle
same-blog 1 0.97741008 370 hunch net-2009-09-18-Necessary and Sufficient Research
Introduction: Researchers are typically confronted with big problems that they have no idea how to solve. In trying to come up with a solution, a natural approach is to decompose the big problem into a set of subproblems whose solution yields a solution to the larger problem. This approach can go wrong in several ways. Decomposition failure . The solution to the decomposition does not in fact yield a solution to the overall problem. Artificial hardness . The subproblems created are sufficient if solved to solve the overall problem, but they are harder than necessary. As you can see, computational complexity forms a relatively new (in research-history) razor by which to judge an approach sufficient but not necessary. In my experience, the artificial hardness problem is very common. Many researchers abdicate the responsibility of choosing a problem to work on to other people. This process starts very naturally as a graduate student, when an incoming student might have relatively l
2 0.79299903 358 hunch net-2009-06-01-Multitask Poisoning
Introduction: There are many ways that interesting research gets done. For example it’s common at a conference for someone to discuss a problem with a partial solution, and for someone else to know how to solve a piece of it, resulting in a paper. In some sense, these are the easiest results we can achieve, so we should ask: Can all research be this easy? The answer is certainly no for fields where research inherently requires experimentation to discover how the real world works. However, mathematics, including parts of physics, computer science, statistics, etc… which are effectively mathematics don’t require experimentation. In effect, a paper can be simply a pure expression of thinking. Can all mathematical-style research be this easy? What’s going on here is research-by-communication. Someone knows something, someone knows something else, and as soon as someone knows both things, a problem is solved. The interesting thing about research-by-communication is that it is becoming radic
3 0.73378378 153 hunch net-2006-02-02-Introspectionism as a Disease
Introduction: In the AI-related parts of machine learning, it is often tempting to examine how you do things in order to imagine how a machine should do things. This is introspection, and it can easily go awry. I will call introspection gone awry introspectionism. Introspectionism is almost unique to AI (and the AI-related parts of machine learning) and it can lead to huge wasted effort in research. It’s easiest to show how introspectionism arises by an example. Suppose we want to solve the problem of navigating a robot from point A to point B given a camera. Then, the following research action plan might seem natural when you examine your own capabilities: Build an edge detector for still images. Build an object recognition system given the edge detector. Build a system to predict distance and orientation to objects given the object recognition system. Build a system to plan a path through the scene you construct from {object identification, distance, orientation} predictions.
4 0.72502404 147 hunch net-2006-01-08-Debugging Your Brain
Introduction: One part of doing research is debugging your understanding of reality. This is hard work: How do you even discover where you misunderstand? If you discover a misunderstanding, how do you go about removing it? The process of debugging computer programs is quite analogous to debugging reality misunderstandings. This is natural—a bug in a computer program is a misunderstanding between you and the computer about what you said. Many of the familiar techniques from debugging have exact parallels. Details When programming, there are often signs that some bug exists like: “the graph my program output is shifted a little bit” = maybe you have an indexing error. In debugging yourself, we often have some impression that something is “not right”. These impressions should be addressed directly and immediately. (Some people have the habit of suppressing worries in favor of excess certainty. That’s not healthy for research.) Corner Cases A “corner case” is an input to a program wh
5 0.7235232 22 hunch net-2005-02-18-What it means to do research.
Introduction: I want to try to describe what doing research means, especially from the point of view of an undergraduate. The shift from a class-taking mentality to a research mentality is very significant and not easy. Problem Posing Posing the right problem is often as important as solving them. Many people can get by in research by solving problems others have posed, but that’s not sufficient for really inspiring research. For learning in particular, there is a strong feeling that we just haven’t figured out which questions are the right ones to ask. You can see this, because the answers we have do not seem convincing. Gambling your life When you do research, you think very hard about new ways of solving problems, new problems, and new solutions. Many conversations are of the form “I wonder what would happen if…” These processes can be short (days or weeks) or years-long endeavours. The worst part is that you’ll only know if you were succesful at the end of the process (and some
6 0.70154268 351 hunch net-2009-05-02-Wielding a New Abstraction
7 0.69466412 435 hunch net-2011-05-16-Research Directions for Machine Learning and Algorithms
8 0.68713045 256 hunch net-2007-07-20-Motivation should be the Responsibility of the Reviewer
9 0.67829686 347 hunch net-2009-03-26-Machine Learning is too easy
10 0.67598534 366 hunch net-2009-08-03-Carbon in Computer Science Research
11 0.67418396 215 hunch net-2006-10-22-Exemplar programming
12 0.6702292 68 hunch net-2005-05-10-Learning Reductions are Reductionist
13 0.66858745 73 hunch net-2005-05-17-A Short Guide to PhD Graduate Study
14 0.66814154 359 hunch net-2009-06-03-Functionally defined Nonlinear Dynamic Models
15 0.66653568 386 hunch net-2010-01-13-Sam Roweis died
16 0.65832055 67 hunch net-2005-05-06-Don’t mix the solution into the problem
17 0.65822554 132 hunch net-2005-11-26-The Design of an Optimal Research Environment
18 0.65602928 295 hunch net-2008-04-12-It Doesn’t Stop
19 0.64925098 352 hunch net-2009-05-06-Machine Learning to AI
20 0.64819413 44 hunch net-2005-03-21-Research Styles in Machine Learning
topicId topicWeight
[(3, 0.023), (27, 0.232), (37, 0.016), (38, 0.059), (48, 0.023), (53, 0.108), (55, 0.09), (64, 0.019), (74, 0.132), (88, 0.013), (92, 0.025), (94, 0.082), (95, 0.076)]
simIndex simValue blogId blogTitle
same-blog 1 0.95039594 370 hunch net-2009-09-18-Necessary and Sufficient Research
Introduction: Researchers are typically confronted with big problems that they have no idea how to solve. In trying to come up with a solution, a natural approach is to decompose the big problem into a set of subproblems whose solution yields a solution to the larger problem. This approach can go wrong in several ways. Decomposition failure . The solution to the decomposition does not in fact yield a solution to the overall problem. Artificial hardness . The subproblems created are sufficient if solved to solve the overall problem, but they are harder than necessary. As you can see, computational complexity forms a relatively new (in research-history) razor by which to judge an approach sufficient but not necessary. In my experience, the artificial hardness problem is very common. Many researchers abdicate the responsibility of choosing a problem to work on to other people. This process starts very naturally as a graduate student, when an incoming student might have relatively l
2 0.93714148 217 hunch net-2006-11-06-Data Linkage Problems
Introduction: Data linkage is a problem which seems to come up in various applied machine learning problems. I have heard it mentioned in various data mining contexts, but it seems relatively less studied for systemic reasons. A very simple version of the data linkage problem is a cross hospital patient record merge. Suppose a patient (John Doe) is admitted to a hospital (General Health), treated, and released. Later, John Doe is admitted to a second hospital (Health General), treated, and released. Given a large number of records of this sort, it becomes very tempting to try and predict the outcomes of treatments. This is reasonably straightforward as a machine learning problem if there is a shared unique identifier for John Doe used by General Health and Health General along with time stamps. We can merge the records and create examples of the form “Given symptoms and treatment, did the patient come back to a hospital within the next year?” These examples could be fed into a learning algo
3 0.89460319 67 hunch net-2005-05-06-Don’t mix the solution into the problem
Introduction: A common defect of many pieces of research is defining the problem in terms of the solution. Here are some examples in learning: “The learning problem is finding a good seperating hyperplane.” “The goal of learning is to minimize (y-p) 2 + C w 2 where y = the observation, p = the prediction and w = a parameter vector.” Defining the loss function to be the one that your algorithm optimizes rather than the one imposed by the world. The fundamental reason why this is a defect is that it creates artificial boundaries to problem solution. Artificial boundaries lead to the possibility of being blind-sided. For example, someone committing (1) or (2) above might find themselves themselves surprised to find a decision tree working well on a problem. Example (3) might result in someone else solving a learning problem better for real world purposes, even if it’s worse with respect to the algorithm optimization. This defect should be avoided so as to not artificially l
4 0.89091718 478 hunch net-2013-01-07-NYU Large Scale Machine Learning Class
Introduction: Yann LeCun and I are coteaching a class on Large Scale Machine Learning starting late January at NYU . This class will cover many tricks to get machine learning working well on datasets with many features, examples, and classes, along with several elements of deep learning and support systems enabling the previous. This is not a beginning class—you really need to have taken a basic machine learning class previously to follow along. Students will be able to run and experiment with large scale learning algorithms since Yahoo! has donated servers which are being configured into a small scale Hadoop cluster. We are planning to cover the frontier of research in scalable learning algorithms, so good class projects could easily lead to papers. For me, this is a chance to teach on many topics of past research. In general, it seems like researchers should engage in at least occasional teaching of research, both as a proof of teachability and to see their own research through th
5 0.88375849 194 hunch net-2006-07-11-New Models
Introduction: How should we, as researchers in machine learning, organize ourselves? The most immediate measurable objective of computer science research is publishing a paper. The most difficult aspect of publishing a paper is having reviewers accept and recommend it for publication. The simplest mechanism for doing this is to show theoretical progress on some standard, well-known easily understood problem. In doing this, we often fall into a local minima of the research process. The basic problem in machine learning is that it is very unclear that the mathematical model is the right one for the (or some) real problem. A good mathematical model in machine learning should have one fundamental trait: it should aid the design of effective learning algorithms. To date, our ability to solve interesting learning problems (speech recognition, machine translation, object recognition, etc…) remains limited (although improving), so the “rightness” of our models is in doubt. If our mathematical mod
6 0.88112539 12 hunch net-2005-02-03-Learning Theory, by assumption
7 0.87713885 371 hunch net-2009-09-21-Netflix finishes (and starts)
8 0.87665445 435 hunch net-2011-05-16-Research Directions for Machine Learning and Algorithms
9 0.87488759 131 hunch net-2005-11-16-The Everything Ensemble Edge
10 0.87415701 360 hunch net-2009-06-15-In Active Learning, the question changes
11 0.8737573 134 hunch net-2005-12-01-The Webscience Future
12 0.87361157 36 hunch net-2005-03-05-Funding Research
13 0.87336057 132 hunch net-2005-11-26-The Design of an Optimal Research Environment
14 0.87311995 351 hunch net-2009-05-02-Wielding a New Abstraction
15 0.87239528 343 hunch net-2009-02-18-Decision by Vetocracy
16 0.87182629 95 hunch net-2005-07-14-What Learning Theory might do
17 0.87125701 235 hunch net-2007-03-03-All Models of Learning have Flaws
18 0.8702901 359 hunch net-2009-06-03-Functionally defined Nonlinear Dynamic Models
19 0.87008041 286 hunch net-2008-01-25-Turing’s Club for Machine Learning
20 0.86982709 358 hunch net-2009-06-01-Multitask Poisoning