hunch_net hunch_net-2005 hunch_net-2005-84 knowledge-graph by maker-knowledge-mining
Source: html
Introduction: A language is a set of primitives which can be combined to succesfully create complex objects. Languages arise in all sorts of situations: mechanical construction, martial arts, communication, etc… Languages appear to be the key to succesfully creating complex objects—it is difficult to come up with any convincing example of a complex object which is not built using some language. Since languages are so crucial to success, it is interesting to organize various machine learning research programs by language. The most common language in machine learning are languages for representing the solution to machine learning. This includes: Bayes Nets and Graphical Models A language for representing probability distributions. The key concept supporting modularity is conditional independence. Michael Kearns has been working on extending this to game theory. Kernelized Linear Classifiers A language for representing linear separators, possibly in a large space. The key form of
sentIndex sentText sentNum sentScore
1 A language is a set of primitives which can be combined to succesfully create complex objects. [sent-1, score-0.707]
2 Languages arise in all sorts of situations: mechanical construction, martial arts, communication, etc… Languages appear to be the key to succesfully creating complex objects—it is difficult to come up with any convincing example of a complex object which is not built using some language. [sent-2, score-0.679]
3 Since languages are so crucial to success, it is interesting to organize various machine learning research programs by language. [sent-3, score-0.784]
4 The most common language in machine learning are languages for representing the solution to machine learning. [sent-4, score-1.263]
5 This includes: Bayes Nets and Graphical Models A language for representing probability distributions. [sent-5, score-0.65]
6 The key concept supporting modularity is conditional independence. [sent-6, score-0.764]
7 Michael Kearns has been working on extending this to game theory. [sent-7, score-0.119]
8 Kernelized Linear Classifiers A language for representing linear separators, possibly in a large space. [sent-8, score-0.712]
9 The key form of modularity here is kernelization. [sent-9, score-0.402]
10 Neural Networks A language for representing and learning functions. [sent-10, score-0.65]
11 ) Decision Trees Another language for representing and learning functions. [sent-13, score-0.65]
12 The key concept supporting modularity is partitioning the input space. [sent-14, score-0.764]
13 In addition there are languages related to various aspects of learning. [sent-16, score-0.756]
14 Reductions A language for translating between varying real-world losses and core learning algorithm optimizations. [sent-17, score-0.489]
15 Feature Languages Exactly how features are specified varies from on learning algorithm to another. [sent-18, score-0.162]
16 Several people have been working on languages for features that cope with sparsity or the cross-product nature of databases. [sent-19, score-0.84]
17 Data interaction languages The statistical query model of learning algorithms provides a standardized interface between data and learning algorithm. [sent-20, score-0.776]
18 With respect to research “interesting” language-related questions include: For what aspects of learning is a language missing? [sent-22, score-0.463]
19 Are any of these languages fundamentally flawed or fundamentally advantageous with respect to another language? [sent-25, score-0.894]
20 What are the most easy to use and effective primitives for these languages? [sent-26, score-0.116]
wordName wordTfidf (topN-words)
[('languages', 0.613), ('representing', 0.334), ('language', 0.316), ('modularity', 0.216), ('key', 0.186), ('supporting', 0.181), ('concept', 0.181), ('succesfully', 0.12), ('primitives', 0.116), ('complex', 0.105), ('aspects', 0.09), ('fundamentally', 0.084), ('translating', 0.072), ('separators', 0.067), ('crucial', 0.067), ('extending', 0.067), ('standardized', 0.063), ('anytime', 0.063), ('kernelized', 0.063), ('linear', 0.062), ('features', 0.061), ('demos', 0.06), ('mechanical', 0.06), ('nets', 0.06), ('sparsity', 0.06), ('respect', 0.057), ('lists', 0.056), ('kearns', 0.056), ('flawed', 0.056), ('cope', 0.054), ('construction', 0.054), ('mlss', 0.054), ('various', 0.053), ('arise', 0.052), ('varying', 0.052), ('working', 0.052), ('miss', 0.051), ('specified', 0.051), ('organize', 0.051), ('convincing', 0.051), ('communication', 0.05), ('objects', 0.05), ('lecun', 0.05), ('varies', 0.05), ('query', 0.05), ('combined', 0.05), ('interaction', 0.05), ('encountered', 0.049), ('chicago', 0.049), ('losses', 0.049)]
simIndex simValue blogId blogTitle
same-blog 1 0.99999988 84 hunch net-2005-06-22-Languages of Learning
Introduction: A language is a set of primitives which can be combined to succesfully create complex objects. Languages arise in all sorts of situations: mechanical construction, martial arts, communication, etc… Languages appear to be the key to succesfully creating complex objects—it is difficult to come up with any convincing example of a complex object which is not built using some language. Since languages are so crucial to success, it is interesting to organize various machine learning research programs by language. The most common language in machine learning are languages for representing the solution to machine learning. This includes: Bayes Nets and Graphical Models A language for representing probability distributions. The key concept supporting modularity is conditional independence. Michael Kearns has been working on extending this to game theory. Kernelized Linear Classifiers A language for representing linear separators, possibly in a large space. The key form of
2 0.36095604 210 hunch net-2006-09-28-Programming Languages for Machine Learning Implementations
Introduction: Machine learning algorithms have a much better chance of being widely adopted if they are implemented in some easy-to-use code. There are several important concerns associated with machine learning which stress programming languages on the ease-of-use vs. speed frontier. Speed The rate at which data sources are growing seems to be outstripping the rate at which computational power is growing, so it is important that we be able to eak out every bit of computational power. Garbage collected languages ( java , ocaml , perl and python ) often have several issues here. Garbage collection often implies that floating point numbers are “boxed”: every float is represented by a pointer to a float. Boxing can cause an order of magnitude slowdown because an extra nonlocalized memory reference is made, and accesses to main memory can are many CPU cycles long. Garbage collection often implies that considerably more memory is used than is necessary. This has a variable effect. I
3 0.12448379 277 hunch net-2007-12-12-Workshop Summary—Principles of Learning Problem Design
Introduction: This is a summary of the workshop on Learning Problem Design which Alina and I ran at NIPS this year. The first question many people have is “What is learning problem design?” This workshop is about admitting that solving learning problems does not start with labeled data, but rather somewhere before. When humans are hired to produce labels, this is usually not a serious problem because you can tell them precisely what semantics you want the labels to have, and we can fix some set of features in advance. However, when other methods are used this becomes more problematic. This focus is important for Machine Learning because there are very large quantities of data which are not labeled by a hired human. The title of the workshop was a bit ambitious, because a workshop is not long enough to synthesize a diversity of approaches into a coherent set of principles. For me, the posters at the end of the workshop were quite helpful in getting approaches to gel. Here are some an
4 0.10680956 435 hunch net-2011-05-16-Research Directions for Machine Learning and Algorithms
Introduction: Muthu invited me to the workshop on algorithms in the field , with the goal of providing a sense of where near-term research should go. When the time came though, I bargained for a post instead, which provides a chance for many other people to comment. There are several things I didn’t fully understand when I went to Yahoo! about 5 years ago. I’d like to repeat them as people in academia may not yet understand them intuitively. Almost all the big impact algorithms operate in pseudo-linear or better time. Think about caching, hashing, sorting, filtering, etc… and you have a sense of what some of the most heavily used algorithms are. This matters quite a bit to Machine Learning research, because people often work with superlinear time algorithms and languages. Two very common examples of this are graphical models, where inference is often a superlinear operation—think about the n 2 dependence on the number of states in a Hidden Markov Model and Kernelized Support Vecto
5 0.10495805 166 hunch net-2006-03-24-NLPers
Introduction: Hal Daume has started the NLPers blog to discuss learning for language problems.
6 0.10450834 35 hunch net-2005-03-04-The Big O and Constants in Learning
7 0.093948826 74 hunch net-2005-05-21-What is the right form of modularity in structured prediction?
8 0.090440325 152 hunch net-2006-01-30-Should the Input Representation be a Vector?
9 0.086395703 60 hunch net-2005-04-23-Advantages and Disadvantages of Bayesian Learning
10 0.080651045 337 hunch net-2009-01-21-Nearly all natural problems require nonlinearity
11 0.07981649 6 hunch net-2005-01-27-Learning Complete Problems
12 0.079228297 235 hunch net-2007-03-03-All Models of Learning have Flaws
13 0.074380204 114 hunch net-2005-09-20-Workshop Proposal: Atomic Learning
14 0.072868966 158 hunch net-2006-02-24-A Fundamentalist Organization of Machine Learning
15 0.072152331 49 hunch net-2005-03-30-What can Type Theory teach us about Machine Learning?
16 0.071473129 253 hunch net-2007-07-06-Idempotent-capable Predictors
17 0.068426684 201 hunch net-2006-08-07-The Call of the Deep
18 0.064179517 136 hunch net-2005-12-07-Is the Google way the way for machine learning?
19 0.063032255 228 hunch net-2007-01-15-The Machine Learning Department
20 0.061394121 14 hunch net-2005-02-07-The State of the Reduction
topicId topicWeight
[(0, 0.138), (1, 0.055), (2, -0.062), (3, 0.013), (4, 0.022), (5, -0.027), (6, -0.013), (7, 0.011), (8, 0.05), (9, -0.05), (10, -0.102), (11, -0.107), (12, -0.029), (13, -0.023), (14, 0.012), (15, -0.012), (16, -0.003), (17, 0.044), (18, -0.028), (19, -0.095), (20, 0.083), (21, -0.029), (22, 0.035), (23, -0.053), (24, 0.006), (25, -0.028), (26, 0.012), (27, -0.021), (28, -0.031), (29, 0.084), (30, -0.087), (31, 0.055), (32, 0.182), (33, 0.203), (34, 0.002), (35, 0.077), (36, -0.012), (37, -0.033), (38, -0.062), (39, -0.087), (40, 0.08), (41, -0.027), (42, 0.082), (43, -0.141), (44, -0.095), (45, -0.059), (46, -0.003), (47, -0.056), (48, 0.064), (49, 0.109)]
simIndex simValue blogId blogTitle
same-blog 1 0.95394623 84 hunch net-2005-06-22-Languages of Learning
Introduction: A language is a set of primitives which can be combined to succesfully create complex objects. Languages arise in all sorts of situations: mechanical construction, martial arts, communication, etc… Languages appear to be the key to succesfully creating complex objects—it is difficult to come up with any convincing example of a complex object which is not built using some language. Since languages are so crucial to success, it is interesting to organize various machine learning research programs by language. The most common language in machine learning are languages for representing the solution to machine learning. This includes: Bayes Nets and Graphical Models A language for representing probability distributions. The key concept supporting modularity is conditional independence. Michael Kearns has been working on extending this to game theory. Kernelized Linear Classifiers A language for representing linear separators, possibly in a large space. The key form of
2 0.87278444 210 hunch net-2006-09-28-Programming Languages for Machine Learning Implementations
Introduction: Machine learning algorithms have a much better chance of being widely adopted if they are implemented in some easy-to-use code. There are several important concerns associated with machine learning which stress programming languages on the ease-of-use vs. speed frontier. Speed The rate at which data sources are growing seems to be outstripping the rate at which computational power is growing, so it is important that we be able to eak out every bit of computational power. Garbage collected languages ( java , ocaml , perl and python ) often have several issues here. Garbage collection often implies that floating point numbers are “boxed”: every float is represented by a pointer to a float. Boxing can cause an order of magnitude slowdown because an extra nonlocalized memory reference is made, and accesses to main memory can are many CPU cycles long. Garbage collection often implies that considerably more memory is used than is necessary. This has a variable effect. I
3 0.57764417 215 hunch net-2006-10-22-Exemplar programming
Introduction: There are many different abstractions for problem definition and solution. Here are a few examples: Functional programming: a set of functions are defined. The composed execution of these functions yields the solution. Linear programming: a set of constraints and a linear objective function are defined. An LP solver finds the constrained optimum. Quadratic programming: Like linear programming, but the language is a little more flexible (and the solution slower). Convex programming: like quadratic programming, but the language is more flexible (and the solutions even slower). Dynamic programming: a recursive definition of the problem is defined and then solved efficiently via caching tricks. SAT programming: A problem is specified as a satisfiability involving a conjunction of a disjunction of boolean variables. A general engine attempts to find a good satisfying assignment. For example Kautz’s blackbox planner. These abstractions have different tradeoffs betw
4 0.57166183 262 hunch net-2007-09-16-Optimizing Machine Learning Programs
Introduction: Machine learning is often computationally bounded which implies that the ability to write fast code becomes important if you ever want to implement a machine learning algorithm. Basic tactical optimizations are covered well elsewhere , but I haven’t seen a reasonable guide to higher level optimizations, which are the most important in my experience. Here are some of the higher level optimizations I’ve often found useful. Algorithmic Improvement First . This is Hard, but it is the most important consideration, and typically yields the most benefits. Good optimizations here are publishable. In the context of machine learning, you should be familiar with the arguments for online vs. batch learning. Choice of Language . There are many arguments about the choice of language . Sometimes you don’t have a choice when interfacing with other people. Personally, I favor C/C++ when I want to write fast code. This (admittedly) makes me a slower programmer than when using higher lev
5 0.52596116 171 hunch net-2006-04-09-Progress in Machine Translation
Introduction: I just visited ISI where Daniel Marcu and others are working on machine translation. Apparently, machine translation is rapidly improving. A particularly dramatic year was 2002->2003 when systems switched from word-based translation to phrase-based translation. From a (now famous) slide by Charles Wayne at DARPA (which funds much of the work on machine translation) here is some anecdotal evidence: 2002 2003 insistent Wednesday may recurred her trips to Libya tomorrow for flying. Cairo 6-4 ( AFP ) – An official announced today in the Egyptian lines company for flying Tuesday is a company “insistent for flying” may resumed a consideration of a day Wednesday tomorrow her trips to Libya of Security Council decision trace international the imposed ban comment. And said the official “the institution sent a speech to Ministry of Foreign Affairs of lifting on Libya air, a situation her recieving replying are so a trip will pull to Libya a morning Wednesday.” E
6 0.49770156 152 hunch net-2006-01-30-Should the Input Representation be a Vector?
7 0.4838765 277 hunch net-2007-12-12-Workshop Summary—Principles of Learning Problem Design
8 0.47387642 49 hunch net-2005-03-30-What can Type Theory teach us about Machine Learning?
9 0.4645167 337 hunch net-2009-01-21-Nearly all natural problems require nonlinearity
10 0.45701015 128 hunch net-2005-11-05-The design of a computing cluster
11 0.41709676 122 hunch net-2005-10-13-Site tweak
12 0.41597542 414 hunch net-2010-10-17-Partha Niyogi has died
13 0.40635425 37 hunch net-2005-03-08-Fast Physics for Learning
14 0.40342769 398 hunch net-2010-05-10-Aggregation of estimators, sparsity in high dimension and computational feasibility
15 0.40081891 435 hunch net-2011-05-16-Research Directions for Machine Learning and Algorithms
16 0.39833143 228 hunch net-2007-01-15-The Machine Learning Department
17 0.39658618 166 hunch net-2006-03-24-NLPers
18 0.39550602 444 hunch net-2011-09-07-KDD and MUCMD 2011
19 0.39022076 162 hunch net-2006-03-09-Use of Notation
20 0.384139 60 hunch net-2005-04-23-Advantages and Disadvantages of Bayesian Learning
topicId topicWeight
[(3, 0.018), (27, 0.203), (37, 0.028), (38, 0.074), (53, 0.08), (55, 0.08), (64, 0.017), (89, 0.305), (94, 0.028), (95, 0.044)]
simIndex simValue blogId blogTitle
1 0.93820822 50 hunch net-2005-04-01-Basic computer science research takes a hit
Introduction: The New York Times has an interesting article about how DARPA has dropped funding for computer science to universities by about a factor of 2 over the last 5 years and become less directed towards basic research. Partially in response, the number of grant submissions to NSF has grown by a factor of 3 (with the NSF budget staying approximately constant in the interim). This is the sort of policy decision which may make sense for the defense department, but which means a large hit for basic research on information technology development in the US. For example “darpa funded the invention of the internet” is reasonably correct. This policy decision is particularly painful in the context of NSF budget cuts and the end of extensive phone monopoly funded research at Bell labs. The good news from a learning perspective is that (based on anecdotal evidence) much of the remaining funding is aimed at learning and learning-related fields. Methods of making good automated predictions obv
2 0.92888254 328 hunch net-2008-11-26-Efficient Reinforcement Learning in MDPs
Introduction: Claude Sammut is attempting to put together an Encyclopedia of Machine Learning . I volunteered to write one article on Efficient RL in MDPs , which I would like to invite comment on. Is something critical missing?
same-blog 3 0.89699918 84 hunch net-2005-06-22-Languages of Learning
Introduction: A language is a set of primitives which can be combined to succesfully create complex objects. Languages arise in all sorts of situations: mechanical construction, martial arts, communication, etc… Languages appear to be the key to succesfully creating complex objects—it is difficult to come up with any convincing example of a complex object which is not built using some language. Since languages are so crucial to success, it is interesting to organize various machine learning research programs by language. The most common language in machine learning are languages for representing the solution to machine learning. This includes: Bayes Nets and Graphical Models A language for representing probability distributions. The key concept supporting modularity is conditional independence. Michael Kearns has been working on extending this to game theory. Kernelized Linear Classifiers A language for representing linear separators, possibly in a large space. The key form of
4 0.85489058 118 hunch net-2005-10-07-On-line learning of regular decision rules
Introduction: Many decision problems can be represented in the form FOR n =1,2,…: — Reality chooses a datum x n . — Decision Maker chooses his decision d n . — Reality chooses an observation y n . — Decision Maker suffers loss L ( y n , d n ). END FOR. The observation y n can be, for example, tomorrow’s stock price and the decision d n the number of shares Decision Maker chooses to buy. The datum x n ideally contains all information that might be relevant in making this decision. We do not want to assume anything about the way Reality generates the observations and data. Suppose there is a good and not too complex decision rule D mapping each datum x to a decision D ( x ). Can we perform as well, or almost as well, as D , without knowing it? This is essentially a special case of the problem of on-line learning . This is a simple result of this kind. Suppose the data x n are taken from [0,1] and L ( y , d )=| y – d |. A norm || h || of a function h on
5 0.8309654 480 hunch net-2013-03-22-I’m a bandit
Introduction: Sebastien Bubeck has a new ML blog focused on optimization and partial feedback which may interest people.
6 0.79263008 299 hunch net-2008-04-27-Watchword: Supervised Learning
7 0.6808784 36 hunch net-2005-03-05-Funding Research
8 0.62831503 21 hunch net-2005-02-17-Learning Research Programs
9 0.62392175 194 hunch net-2006-07-11-New Models
10 0.61564445 131 hunch net-2005-11-16-The Everything Ensemble Edge
11 0.61345685 227 hunch net-2007-01-10-A Deep Belief Net Learning Problem
12 0.61302 12 hunch net-2005-02-03-Learning Theory, by assumption
13 0.60806358 478 hunch net-2013-01-07-NYU Large Scale Machine Learning Class
14 0.60790503 230 hunch net-2007-02-02-Thoughts regarding “Is machine learning different from statistics?”
15 0.60608053 19 hunch net-2005-02-14-Clever Methods of Overfitting
16 0.60487807 370 hunch net-2009-09-18-Necessary and Sufficient Research
17 0.60349542 343 hunch net-2009-02-18-Decision by Vetocracy
18 0.60331708 464 hunch net-2012-05-03-Microsoft Research, New York City
19 0.60137117 225 hunch net-2007-01-02-Retrospective
20 0.60005015 309 hunch net-2008-07-10-Interesting papers, ICML 2008