hunch_net hunch_net-2006 hunch_net-2006-147 knowledge-graph by maker-knowledge-mining
Source: html
Introduction: One part of doing research is debugging your understanding of reality. This is hard work: How do you even discover where you misunderstand? If you discover a misunderstanding, how do you go about removing it? The process of debugging computer programs is quite analogous to debugging reality misunderstandings. This is natural—a bug in a computer program is a misunderstanding between you and the computer about what you said. Many of the familiar techniques from debugging have exact parallels. Details When programming, there are often signs that some bug exists like: “the graph my program output is shifted a little bit” = maybe you have an indexing error. In debugging yourself, we often have some impression that something is “not right”. These impressions should be addressed directly and immediately. (Some people have the habit of suppressing worries in favor of excess certainty. That’s not healthy for research.) Corner Cases A “corner case” is an input to a program wh
sentIndex sentText sentNum sentScore
1 One part of doing research is debugging your understanding of reality. [sent-1, score-0.778]
2 The process of debugging computer programs is quite analogous to debugging reality misunderstandings. [sent-4, score-1.518]
3 This is natural—a bug in a computer program is a misunderstanding between you and the computer about what you said. [sent-5, score-0.843]
4 Many of the familiar techniques from debugging have exact parallels. [sent-6, score-0.653]
5 Details When programming, there are often signs that some bug exists like: “the graph my program output is shifted a little bit” = maybe you have an indexing error. [sent-7, score-0.619]
6 In debugging yourself, we often have some impression that something is “not right”. [sent-8, score-0.725]
7 We can often concoct our own corner cases and solve them. [sent-13, score-0.39]
8 If the solution doesn’t match our (mis)understanding, a bug has been found. [sent-14, score-0.269]
9 Warnings On The compiler “gcc” has the flag “-Wall” which means “turn all warnings about odd program forms on”. [sent-15, score-0.413]
10 You should always compile with “-Wall” as you immediately realize if you compare the time required to catch a bug that “-Wall” finds with the time required to debug the hard way. [sent-16, score-0.429]
11 The equivalent for debugging yourself is listening to others carefully. [sent-17, score-0.74]
12 Debugging by Design When programming, people often design the process of creating the program so that it is easy to debug. [sent-21, score-0.296]
13 Then take the next step, the next, etc… Isolation When a bug is discovered, the canonical early trouble shooting step is isolating the bug. [sent-23, score-0.434]
14 For a parse error, what is the smallest program exhibiting the error? [sent-24, score-0.394]
15 In these cases, it is often a good idea to rethink the problem the program is trying to solve. [sent-28, score-0.359]
16 The issue of how to represent the problem is perhaps even more important in research since human brains are not as adept as computers at shifting and using representations. [sent-30, score-0.274]
17 Some aspects of debugging a reality misunderstanding don’t have a good analogue for programming because debugging yourself often involves social interactions. [sent-33, score-1.839]
18 Everyone (including me) dislikes having others point out when they are wrong so there is a temptation to avoid admitting it (to others, or more harmfully to yourself). [sent-35, score-0.419]
19 With respect to others, admitting you are wrong allows a conversation to move on to other things. [sent-37, score-0.431]
20 With respect to yourself, admitting you are wrong allows you to move on to other things. [sent-38, score-0.431]
wordName wordTfidf (topN-words)
[('debugging', 0.653), ('bug', 0.269), ('program', 0.224), ('misunderstanding', 0.196), ('corner', 0.174), ('admitting', 0.139), ('warnings', 0.131), ('programming', 0.127), ('move', 0.12), ('habit', 0.116), ('wrong', 0.103), ('represent', 0.093), ('temptation', 0.09), ('others', 0.087), ('reality', 0.084), ('cases', 0.083), ('computer', 0.077), ('talking', 0.077), ('discover', 0.072), ('often', 0.072), ('simplest', 0.071), ('allows', 0.069), ('research', 0.064), ('problem', 0.063), ('understanding', 0.061), ('solve', 0.061), ('worries', 0.058), ('compiler', 0.058), ('parse', 0.058), ('smallest', 0.058), ('fought', 0.058), ('isolation', 0.058), ('mastery', 0.058), ('shooting', 0.058), ('inputs', 0.054), ('uninteresting', 0.054), ('brains', 0.054), ('stepwise', 0.054), ('analogue', 0.054), ('compiled', 0.054), ('exhibiting', 0.054), ('finds', 0.054), ('impressions', 0.054), ('indexing', 0.054), ('trouble', 0.054), ('step', 0.053), ('required', 0.053), ('analogy', 0.051), ('mis', 0.051), ('analogous', 0.051)]
simIndex simValue blogId blogTitle
same-blog 1 1.0000001 147 hunch net-2006-01-08-Debugging Your Brain
Introduction: One part of doing research is debugging your understanding of reality. This is hard work: How do you even discover where you misunderstand? If you discover a misunderstanding, how do you go about removing it? The process of debugging computer programs is quite analogous to debugging reality misunderstandings. This is natural—a bug in a computer program is a misunderstanding between you and the computer about what you said. Many of the familiar techniques from debugging have exact parallels. Details When programming, there are often signs that some bug exists like: “the graph my program output is shifted a little bit” = maybe you have an indexing error. In debugging yourself, we often have some impression that something is “not right”. These impressions should be addressed directly and immediately. (Some people have the habit of suppressing worries in favor of excess certainty. That’s not healthy for research.) Corner Cases A “corner case” is an input to a program wh
2 0.10742531 215 hunch net-2006-10-22-Exemplar programming
Introduction: There are many different abstractions for problem definition and solution. Here are a few examples: Functional programming: a set of functions are defined. The composed execution of these functions yields the solution. Linear programming: a set of constraints and a linear objective function are defined. An LP solver finds the constrained optimum. Quadratic programming: Like linear programming, but the language is a little more flexible (and the solution slower). Convex programming: like quadratic programming, but the language is more flexible (and the solutions even slower). Dynamic programming: a recursive definition of the problem is defined and then solved efficiently via caching tricks. SAT programming: A problem is specified as a satisfiability involving a conjunction of a disjunction of boolean variables. A general engine attempts to find a good satisfying assignment. For example Kautz’s blackbox planner. These abstractions have different tradeoffs betw
3 0.093665898 42 hunch net-2005-03-17-Going all the Way, Sometimes
Introduction: At many points in research, you face a choice: should I keep on improving some old piece of technology or should I do something new? For example: Should I refine bounds to make them tighter? Should I take some learning theory and turn it into a learning algorithm? Should I implement the learning algorithm? Should I test the learning algorithm widely? Should I release the algorithm as source code? Should I go see what problems people actually need to solve? The universal temptation of people attracted to research is doing something new. That is sometimes the right decision, but is also often not. I’d like to discuss some reasons why not. Expertise Once expertise are developed on some subject, you are the right person to refine them. What is the real problem? Continually improving a piece of technology is a mechanism forcing you to confront this question. In many cases, this confrontation is uncomfortable because you discover that your method has fundamen
4 0.087314844 22 hunch net-2005-02-18-What it means to do research.
Introduction: I want to try to describe what doing research means, especially from the point of view of an undergraduate. The shift from a class-taking mentality to a research mentality is very significant and not easy. Problem Posing Posing the right problem is often as important as solving them. Many people can get by in research by solving problems others have posed, but that’s not sufficient for really inspiring research. For learning in particular, there is a strong feeling that we just haven’t figured out which questions are the right ones to ask. You can see this, because the answers we have do not seem convincing. Gambling your life When you do research, you think very hard about new ways of solving problems, new problems, and new solutions. Many conversations are of the form “I wonder what would happen if…” These processes can be short (days or weeks) or years-long endeavours. The worst part is that you’ll only know if you were succesful at the end of the process (and some
5 0.082102433 262 hunch net-2007-09-16-Optimizing Machine Learning Programs
Introduction: Machine learning is often computationally bounded which implies that the ability to write fast code becomes important if you ever want to implement a machine learning algorithm. Basic tactical optimizations are covered well elsewhere , but I haven’t seen a reasonable guide to higher level optimizations, which are the most important in my experience. Here are some of the higher level optimizations I’ve often found useful. Algorithmic Improvement First . This is Hard, but it is the most important consideration, and typically yields the most benefits. Good optimizations here are publishable. In the context of machine learning, you should be familiar with the arguments for online vs. batch learning. Choice of Language . There are many arguments about the choice of language . Sometimes you don’t have a choice when interfacing with other people. Personally, I favor C/C++ when I want to write fast code. This (admittedly) makes me a slower programmer than when using higher lev
6 0.081729099 461 hunch net-2012-04-09-ICML author feedback is open
7 0.079516783 304 hunch net-2008-06-27-Reviewing Horror Stories
8 0.073880233 453 hunch net-2012-01-28-Why COLT?
9 0.073477648 454 hunch net-2012-01-30-ICML Posters and Scope
10 0.071575075 228 hunch net-2007-01-15-The Machine Learning Department
11 0.069246516 434 hunch net-2011-05-09-CI Fellows, again
12 0.068702199 358 hunch net-2009-06-01-Multitask Poisoning
13 0.067407571 320 hunch net-2008-10-14-Who is Responsible for a Bad Review?
14 0.064406604 132 hunch net-2005-11-26-The Design of an Optimal Research Environment
15 0.064104564 142 hunch net-2005-12-22-Yes , I am applying
16 0.062956132 14 hunch net-2005-02-07-The State of the Reduction
17 0.062675409 44 hunch net-2005-03-21-Research Styles in Machine Learning
18 0.062390279 210 hunch net-2006-09-28-Programming Languages for Machine Learning Implementations
19 0.061857529 25 hunch net-2005-02-20-At One Month
20 0.060871024 330 hunch net-2008-12-07-A NIPS paper
topicId topicWeight
[(0, 0.156), (1, -0.012), (2, -0.022), (3, 0.07), (4, -0.041), (5, 0.001), (6, 0.057), (7, 0.038), (8, -0.005), (9, 0.008), (10, -0.018), (11, -0.039), (12, -0.0), (13, -0.0), (14, 0.025), (15, -0.06), (16, 0.09), (17, -0.009), (18, -0.01), (19, 0.016), (20, 0.019), (21, 0.06), (22, 0.058), (23, 0.018), (24, -0.038), (25, 0.003), (26, 0.024), (27, -0.047), (28, 0.043), (29, 0.007), (30, -0.09), (31, 0.024), (32, 0.016), (33, 0.001), (34, -0.043), (35, 0.027), (36, 0.073), (37, -0.004), (38, -0.077), (39, 0.013), (40, 0.084), (41, -0.004), (42, -0.016), (43, 0.015), (44, 0.024), (45, -0.035), (46, -0.028), (47, -0.004), (48, 0.058), (49, 0.067)]
simIndex simValue blogId blogTitle
same-blog 1 0.9568131 147 hunch net-2006-01-08-Debugging Your Brain
Introduction: One part of doing research is debugging your understanding of reality. This is hard work: How do you even discover where you misunderstand? If you discover a misunderstanding, how do you go about removing it? The process of debugging computer programs is quite analogous to debugging reality misunderstandings. This is natural—a bug in a computer program is a misunderstanding between you and the computer about what you said. Many of the familiar techniques from debugging have exact parallels. Details When programming, there are often signs that some bug exists like: “the graph my program output is shifted a little bit” = maybe you have an indexing error. In debugging yourself, we often have some impression that something is “not right”. These impressions should be addressed directly and immediately. (Some people have the habit of suppressing worries in favor of excess certainty. That’s not healthy for research.) Corner Cases A “corner case” is an input to a program wh
2 0.62776101 22 hunch net-2005-02-18-What it means to do research.
Introduction: I want to try to describe what doing research means, especially from the point of view of an undergraduate. The shift from a class-taking mentality to a research mentality is very significant and not easy. Problem Posing Posing the right problem is often as important as solving them. Many people can get by in research by solving problems others have posed, but that’s not sufficient for really inspiring research. For learning in particular, there is a strong feeling that we just haven’t figured out which questions are the right ones to ask. You can see this, because the answers we have do not seem convincing. Gambling your life When you do research, you think very hard about new ways of solving problems, new problems, and new solutions. Many conversations are of the form “I wonder what would happen if…” These processes can be short (days or weeks) or years-long endeavours. The worst part is that you’ll only know if you were succesful at the end of the process (and some
3 0.62613279 358 hunch net-2009-06-01-Multitask Poisoning
Introduction: There are many ways that interesting research gets done. For example it’s common at a conference for someone to discuss a problem with a partial solution, and for someone else to know how to solve a piece of it, resulting in a paper. In some sense, these are the easiest results we can achieve, so we should ask: Can all research be this easy? The answer is certainly no for fields where research inherently requires experimentation to discover how the real world works. However, mathematics, including parts of physics, computer science, statistics, etc… which are effectively mathematics don’t require experimentation. In effect, a paper can be simply a pure expression of thinking. Can all mathematical-style research be this easy? What’s going on here is research-by-communication. Someone knows something, someone knows something else, and as soon as someone knows both things, a problem is solved. The interesting thing about research-by-communication is that it is becoming radic
4 0.62472957 370 hunch net-2009-09-18-Necessary and Sufficient Research
Introduction: Researchers are typically confronted with big problems that they have no idea how to solve. In trying to come up with a solution, a natural approach is to decompose the big problem into a set of subproblems whose solution yields a solution to the larger problem. This approach can go wrong in several ways. Decomposition failure . The solution to the decomposition does not in fact yield a solution to the overall problem. Artificial hardness . The subproblems created are sufficient if solved to solve the overall problem, but they are harder than necessary. As you can see, computational complexity forms a relatively new (in research-history) razor by which to judge an approach sufficient but not necessary. In my experience, the artificial hardness problem is very common. Many researchers abdicate the responsibility of choosing a problem to work on to other people. This process starts very naturally as a graduate student, when an incoming student might have relatively l
5 0.61940461 215 hunch net-2006-10-22-Exemplar programming
Introduction: There are many different abstractions for problem definition and solution. Here are a few examples: Functional programming: a set of functions are defined. The composed execution of these functions yields the solution. Linear programming: a set of constraints and a linear objective function are defined. An LP solver finds the constrained optimum. Quadratic programming: Like linear programming, but the language is a little more flexible (and the solution slower). Convex programming: like quadratic programming, but the language is more flexible (and the solutions even slower). Dynamic programming: a recursive definition of the problem is defined and then solved efficiently via caching tricks. SAT programming: A problem is specified as a satisfiability involving a conjunction of a disjunction of boolean variables. A general engine attempts to find a good satisfying assignment. For example Kautz’s blackbox planner. These abstractions have different tradeoffs betw
6 0.60691994 162 hunch net-2006-03-09-Use of Notation
7 0.59856522 262 hunch net-2007-09-16-Optimizing Machine Learning Programs
8 0.56970209 210 hunch net-2006-09-28-Programming Languages for Machine Learning Implementations
9 0.55551386 307 hunch net-2008-07-04-More Presentation Preparation
10 0.55237174 249 hunch net-2007-06-21-Presentation Preparation
11 0.53842729 411 hunch net-2010-09-21-Regretting the dead
12 0.5308131 231 hunch net-2007-02-10-Best Practices for Collaboration
13 0.52847183 67 hunch net-2005-05-06-Don’t mix the solution into the problem
14 0.52696812 314 hunch net-2008-08-24-Mass Customized Medicine in the Future?
15 0.52340436 257 hunch net-2007-07-28-Asking questions
16 0.52176064 458 hunch net-2012-03-06-COLT-ICML Open Questions and ICML Instructions
17 0.51740706 222 hunch net-2006-12-05-Recruitment Conferences
18 0.50985026 128 hunch net-2005-11-05-The design of a computing cluster
19 0.50929594 42 hunch net-2005-03-17-Going all the Way, Sometimes
20 0.49799481 366 hunch net-2009-08-03-Carbon in Computer Science Research
topicId topicWeight
[(1, 0.014), (3, 0.012), (10, 0.027), (27, 0.168), (38, 0.037), (53, 0.049), (55, 0.114), (56, 0.011), (71, 0.314), (77, 0.013), (83, 0.018), (89, 0.013), (94, 0.121)]
simIndex simValue blogId blogTitle
1 0.96908182 417 hunch net-2010-11-18-ICML 2011 – Call for Tutorials
Introduction: I would like to encourage people to consider giving a tutorial at next years ICML. The ideal tutorial attracts a wide audience, provides a gentle and easily taught introduction to the chosen research area, and also covers the most important contributions in depth. Submissions are due January 14 Â (about two weeks before paper deadline). http://www.icml-2011.org/tutorials.php Regards, Ulf
2 0.9161678 161 hunch net-2006-03-05-“Structural” Learning
Introduction: Fernando Pereira pointed out Ando and Zhang ‘s paper on “structural” learning. Structural learning is multitask learning on subproblems created from unlabeled data. The basic idea is to take a look at the unlabeled data and create many supervised problems. On text data, which they test on, these subproblems might be of the form “Given surrounding words predict the middle word”. The hope here is that successfully predicting on these subproblems is relevant to the prediction of your core problem. In the long run, the precise mechanism used (essentially, linear predictors with parameters tied by a common matrix) and the precise problems formed may not be critical. What seems critical is that the hope is realized: the technique provides a significant edge in practice. Some basic questions about this approach are: Are there effective automated mechanisms for creating the subproblems? Is it necessary to use a shared representation?
same-blog 3 0.87662172 147 hunch net-2006-01-08-Debugging Your Brain
Introduction: One part of doing research is debugging your understanding of reality. This is hard work: How do you even discover where you misunderstand? If you discover a misunderstanding, how do you go about removing it? The process of debugging computer programs is quite analogous to debugging reality misunderstandings. This is natural—a bug in a computer program is a misunderstanding between you and the computer about what you said. Many of the familiar techniques from debugging have exact parallels. Details When programming, there are often signs that some bug exists like: “the graph my program output is shifted a little bit” = maybe you have an indexing error. In debugging yourself, we often have some impression that something is “not right”. These impressions should be addressed directly and immediately. (Some people have the habit of suppressing worries in favor of excess certainty. That’s not healthy for research.) Corner Cases A “corner case” is an input to a program wh
4 0.75895196 450 hunch net-2011-12-02-Hadoop AllReduce and Terascale Learning
Introduction: Suppose you have a dataset with 2 terafeatures (we only count nonzero entries in a datamatrix), and want to learn a good linear predictor in a reasonable amount of time. How do you do it? As a learning theorist, the first thing you do is pray that this is too much data for the number of parameters—but that’s not the case, there are around 16 billion examples, 16 million parameters, and people really care about a high quality predictor, so subsampling is not a good strategy. Alekh visited us last summer, and we had a breakthrough (see here for details), coming up with the first learning algorithm I’ve seen that is provably faster than any future single machine learning algorithm. The proof of this is simple: We can output a optimal-up-to-precision linear predictor faster than the data can be streamed through the network interface of any single machine involved in the computation. It is necessary but not sufficient to have an effective communication infrastructure. It is ne
5 0.72657973 152 hunch net-2006-01-30-Should the Input Representation be a Vector?
Introduction: Let’s suppose that we are trying to create a general purpose machine learning box. The box is fed many examples of the function it is supposed to learn and (hopefully) succeeds. To date, most such attempts to produce a box of this form take a vector as input. The elements of the vector might be bits, real numbers, or ‘categorical’ data (a discrete set of values). On the other hand, there are a number of succesful applications of machine learning which do not seem to use a vector representation as input. For example, in vision, convolutional neural networks have been used to solve several vision problems. The input to the convolutional neural network is essentially the raw camera image as a matrix . In learning for natural languages, several people have had success on problems like parts-of-speech tagging using predictors restricted to a window surrounding the word to be predicted. A vector window and a matrix both imply a notion of locality which is being actively and
6 0.64752674 379 hunch net-2009-11-23-ICML 2009 Workshops (and Tutorials)
7 0.62161869 229 hunch net-2007-01-26-Parallel Machine Learning Problems
8 0.59980398 136 hunch net-2005-12-07-Is the Google way the way for machine learning?
9 0.59883606 221 hunch net-2006-12-04-Structural Problems in NIPS Decision Making
10 0.59134322 286 hunch net-2008-01-25-Turing’s Club for Machine Learning
11 0.58882296 423 hunch net-2011-02-02-User preferences for search engines
12 0.58751845 40 hunch net-2005-03-13-Avoiding Bad Reviewing
13 0.58681273 437 hunch net-2011-07-10-ICML 2011 and the future
14 0.5860588 96 hunch net-2005-07-21-Six Months
15 0.58568162 98 hunch net-2005-07-27-Not goal metrics
16 0.5854997 95 hunch net-2005-07-14-What Learning Theory might do
17 0.58494776 419 hunch net-2010-12-04-Vowpal Wabbit, version 5.0, and the second heresy
18 0.58164698 204 hunch net-2006-08-28-Learning Theory standards for NIPS 2006
19 0.57891649 297 hunch net-2008-04-22-Taking the next step
20 0.57866728 454 hunch net-2012-01-30-ICML Posters and Scope