hunch_net hunch_net-2011 hunch_net-2011-431 knowledge-graph by maker-knowledge-mining
Source: html
Introduction: Unfortunately, a scheduling failure meant I missed all of AIStat and most of the learning workshop , otherwise known as Snowbird, when it’s at Snowbird . At snowbird, the talk on Sum-Product networks by Hoifung Poon stood out to me ( Pedro Domingos is a coauthor.). The basic point was that by appropriately constructing networks based on sums and products, the normalization problem in probabilistic models is eliminated, yielding a highly tractable yet flexible representation+learning algorithm. As an algorithm, this is noticeably cleaner than deep belief networks with a claim to being an order of magnitude faster and working better on an image completion task. Snowbird doesn’t have real papers—just the abstract above. I look forward to seeing the paper. (added: Rodrigo points out the deep learning workshop draft .)
sentIndex sentText sentNum sentScore
1 Unfortunately, a scheduling failure meant I missed all of AIStat and most of the learning workshop , otherwise known as Snowbird, when it’s at Snowbird . [sent-1, score-0.713]
2 At snowbird, the talk on Sum-Product networks by Hoifung Poon stood out to me ( Pedro Domingos is a coauthor. [sent-2, score-0.467]
3 The basic point was that by appropriately constructing networks based on sums and products, the normalization problem in probabilistic models is eliminated, yielding a highly tractable yet flexible representation+learning algorithm. [sent-4, score-1.402]
4 As an algorithm, this is noticeably cleaner than deep belief networks with a claim to being an order of magnitude faster and working better on an image completion task. [sent-5, score-1.328]
5 Snowbird doesn’t have real papers—just the abstract above. [sent-6, score-0.098]
6 (added: Rodrigo points out the deep learning workshop draft . [sent-8, score-0.462]
wordName wordTfidf (topN-words)
[('snowbird', 0.553), ('networks', 0.25), ('cleaner', 0.166), ('products', 0.166), ('domingos', 0.166), ('hoifung', 0.166), ('poon', 0.166), ('completion', 0.154), ('pedro', 0.154), ('sums', 0.154), ('eliminated', 0.154), ('deep', 0.151), ('stood', 0.145), ('normalization', 0.138), ('aistat', 0.133), ('workshop', 0.129), ('scheduling', 0.128), ('flexible', 0.124), ('constructing', 0.124), ('forward', 0.121), ('seeing', 0.121), ('yielding', 0.115), ('draft', 0.115), ('missed', 0.115), ('image', 0.11), ('magnitude', 0.11), ('meant', 0.107), ('abstract', 0.098), ('added', 0.097), ('tractable', 0.095), ('faster', 0.09), ('highly', 0.09), ('probabilistic', 0.088), ('belief', 0.088), ('unfortunately', 0.086), ('representation', 0.085), ('failure', 0.085), ('claim', 0.084), ('otherwise', 0.081), ('look', 0.074), ('talk', 0.072), ('models', 0.068), ('known', 0.068), ('points', 0.067), ('order', 0.066), ('doesn', 0.066), ('working', 0.059), ('based', 0.054), ('yet', 0.053), ('point', 0.049)]
simIndex simValue blogId blogTitle
same-blog 1 0.99999994 431 hunch net-2011-04-18-A paper not at Snowbird
Introduction: Unfortunately, a scheduling failure meant I missed all of AIStat and most of the learning workshop , otherwise known as Snowbird, when it’s at Snowbird . At snowbird, the talk on Sum-Product networks by Hoifung Poon stood out to me ( Pedro Domingos is a coauthor.). The basic point was that by appropriately constructing networks based on sums and products, the normalization problem in probabilistic models is eliminated, yielding a highly tractable yet flexible representation+learning algorithm. As an algorithm, this is noticeably cleaner than deep belief networks with a claim to being an order of magnitude faster and working better on an image completion task. Snowbird doesn’t have real papers—just the abstract above. I look forward to seeing the paper. (added: Rodrigo points out the deep learning workshop draft .)
2 0.18611445 349 hunch net-2009-04-21-Interesting Presentations at Snowbird
Introduction: Here are a few of presentations interesting me at the snowbird learning workshop (which, amusingly, was in Florida with AIStat ). Thomas Breuel described machine learning problems within OCR and an open source OCR software/research platform with modular learning components as well has a 60Million size dataset derived from Google ‘s scanned books. Kristen Grauman and Fei-Fei Li discussed using active learning with different cost labels and large datasets for image ontology . Both of them used Mechanical Turk as a labeling system , which looks to become routine, at least for vision problems. Russ Tedrake discussed using machine learning for control, with a basic claim that it was the way to go for problems involving a medium Reynold’s number such as in bird flight, where simulation is extremely intense. Yann LeCun presented a poster on an FPGA for convolutional neural networks yielding a factor of 100 speedup in processing. In addition to the graphi
3 0.15417261 54 hunch net-2005-04-08-Fast SVMs
Introduction: There was a presentation at snowbird about parallelized support vector machines. In many cases, people parallelize by ignoring serial operations, but that is not what happened here—they parallelize with optimizations. Consequently, this seems to be the fastest SVM in existence. There is a related paper here .
4 0.15391311 53 hunch net-2005-04-06-Structured Regret Minimization
Introduction: Geoff Gordon made an interesting presentation at the snowbird learning workshop discussing the use of no-regret algorithms for the use of several robot-related learning problems. There seems to be a draft here . This seems interesting in two ways: Drawback Removal One of the significant problems with these online algorithms is that they can’t cope with structure very easily. This drawback is addressed for certain structures. Experiments One criticism of such algorithms is that they are too “worst case”. Several experiments suggest that protecting yourself against this worst case does not necessarily incur a great loss.
5 0.14196846 201 hunch net-2006-08-07-The Call of the Deep
Introduction: Many learning algorithms used in practice are fairly simple. Viewed representationally, many prediction algorithms either compute a linear separator of basic features (perceptron, winnow, weighted majority, SVM) or perhaps a linear separator of slightly more complex features (2-layer neural networks or kernelized SVMs). Should we go beyond this, and start using “deep” representations? What is deep learning? Intuitively, deep learning is about learning to predict in ways which can involve complex dependencies between the input (observed) features. Specifying this more rigorously turns out to be rather difficult. Consider the following cases: SVM with Gaussian Kernel. This is not considered deep learning, because an SVM with a gaussian kernel can’t succinctly represent certain decision surfaces. One of Yann LeCun ‘s examples is recognizing objects based on pixel values. An SVM will need a new support vector for each significantly different background. Since the number
6 0.13754176 440 hunch net-2011-08-06-Interesting thing at UAI 2011
7 0.13610454 80 hunch net-2005-06-10-Workshops are not Conferences
8 0.12628831 407 hunch net-2010-08-23-Boosted Decision Trees for Deep Learning
9 0.088676192 438 hunch net-2011-07-11-Interesting Neural Network Papers at ICML 2011
10 0.086953461 16 hunch net-2005-02-09-Intuitions from applied learning
11 0.083157435 227 hunch net-2007-01-10-A Deep Belief Net Learning Problem
12 0.079151958 330 hunch net-2008-12-07-A NIPS paper
13 0.078024574 477 hunch net-2013-01-01-Deep Learning 2012
14 0.066928372 224 hunch net-2006-12-12-Interesting Papers at NIPS 2006
15 0.062132038 234 hunch net-2007-02-22-Create Your Own ICML Workshop
16 0.060659114 23 hunch net-2005-02-19-Loss Functions for Discriminative Training of Energy-Based Models
17 0.060594164 359 hunch net-2009-06-03-Functionally defined Nonlinear Dynamic Models
18 0.060147755 435 hunch net-2011-05-16-Research Directions for Machine Learning and Algorithms
19 0.060048573 385 hunch net-2009-12-27-Interesting things at NIPS 2009
20 0.059379771 488 hunch net-2013-08-31-Extreme Classification workshop at NIPS
topicId topicWeight
[(0, 0.108), (1, -0.003), (2, -0.038), (3, -0.043), (4, 0.101), (5, 0.08), (6, -0.039), (7, -0.012), (8, 0.043), (9, -0.071), (10, -0.061), (11, -0.091), (12, -0.062), (13, -0.09), (14, -0.002), (15, 0.117), (16, -0.025), (17, 0.131), (18, -0.128), (19, 0.082), (20, -0.084), (21, 0.0), (22, -0.033), (23, 0.074), (24, 0.019), (25, 0.095), (26, 0.004), (27, -0.012), (28, -0.035), (29, 0.078), (30, -0.034), (31, 0.06), (32, -0.004), (33, 0.057), (34, 0.112), (35, -0.139), (36, 0.052), (37, 0.024), (38, -0.071), (39, -0.015), (40, -0.114), (41, 0.035), (42, -0.118), (43, 0.076), (44, 0.082), (45, -0.013), (46, -0.069), (47, 0.036), (48, 0.059), (49, 0.056)]
simIndex simValue blogId blogTitle
same-blog 1 0.97488868 431 hunch net-2011-04-18-A paper not at Snowbird
Introduction: Unfortunately, a scheduling failure meant I missed all of AIStat and most of the learning workshop , otherwise known as Snowbird, when it’s at Snowbird . At snowbird, the talk on Sum-Product networks by Hoifung Poon stood out to me ( Pedro Domingos is a coauthor.). The basic point was that by appropriately constructing networks based on sums and products, the normalization problem in probabilistic models is eliminated, yielding a highly tractable yet flexible representation+learning algorithm. As an algorithm, this is noticeably cleaner than deep belief networks with a claim to being an order of magnitude faster and working better on an image completion task. Snowbird doesn’t have real papers—just the abstract above. I look forward to seeing the paper. (added: Rodrigo points out the deep learning workshop draft .)
2 0.62101454 201 hunch net-2006-08-07-The Call of the Deep
Introduction: Many learning algorithms used in practice are fairly simple. Viewed representationally, many prediction algorithms either compute a linear separator of basic features (perceptron, winnow, weighted majority, SVM) or perhaps a linear separator of slightly more complex features (2-layer neural networks or kernelized SVMs). Should we go beyond this, and start using “deep” representations? What is deep learning? Intuitively, deep learning is about learning to predict in ways which can involve complex dependencies between the input (observed) features. Specifying this more rigorously turns out to be rather difficult. Consider the following cases: SVM with Gaussian Kernel. This is not considered deep learning, because an SVM with a gaussian kernel can’t succinctly represent certain decision surfaces. One of Yann LeCun ‘s examples is recognizing objects based on pixel values. An SVM will need a new support vector for each significantly different background. Since the number
3 0.61350125 407 hunch net-2010-08-23-Boosted Decision Trees for Deep Learning
Introduction: About 4 years ago, I speculated that decision trees qualify as a deep learning algorithm because they can make decisions which are substantially nonlinear in the input representation. Ping Li has proved this correct, empirically at UAI by showing that boosted decision trees can beat deep belief networks on versions of Mnist which are artificially hardened so as to make them solvable only by deep learning algorithms. This is an important point, because the ability to solve these sorts of problems is probably the best objective definition of a deep learning algorithm we have. I’m not that surprised. In my experience, if you can accept the computational drawbacks of a boosted decision tree, they can achieve pretty good performance. Geoff Hinton once told me that the great thing about deep belief networks is that they work. I understand that Ping had very substantial difficulty in getting this published, so I hope some reviewers step up to the standard of valuing wha
4 0.61268413 438 hunch net-2011-07-11-Interesting Neural Network Papers at ICML 2011
Introduction: Maybe it’s too early to call, but with four separate Neural Network sessions at this year’s ICML , it looks like Neural Networks are making a comeback. Here are my highlights of these sessions. In general, my feeling is that these papers both demystify deep learning and show its broader applicability. The first observation I made is that the once disreputable “Neural” nomenclature is being used again in lieu of “deep learning”. Maybe it’s because Adam Coates et al. showed that single layer networks can work surprisingly well. An Analysis of Single-Layer Networks in Unsupervised Feature Learning , Adam Coates , Honglak Lee , Andrew Y. Ng (AISTATS 2011) The Importance of Encoding Versus Training with Sparse Coding and Vector Quantization , Adam Coates , Andrew Y. Ng (ICML 2011) Another surprising result out of Andrew Ng’s group comes from Andrew Saxe et al. who show that certain convolutional pooling architectures can obtain close to state-of-the-art pe
5 0.61063391 54 hunch net-2005-04-08-Fast SVMs
Introduction: There was a presentation at snowbird about parallelized support vector machines. In many cases, people parallelize by ignoring serial operations, but that is not what happened here—they parallelize with optimizations. Consequently, this seems to be the fastest SVM in existence. There is a related paper here .
6 0.55881023 477 hunch net-2013-01-01-Deep Learning 2012
7 0.55025512 53 hunch net-2005-04-06-Structured Regret Minimization
8 0.55011904 349 hunch net-2009-04-21-Interesting Presentations at Snowbird
9 0.49665433 227 hunch net-2007-01-10-A Deep Belief Net Learning Problem
10 0.48129016 440 hunch net-2011-08-06-Interesting thing at UAI 2011
11 0.47263214 224 hunch net-2006-12-12-Interesting Papers at NIPS 2006
12 0.45546338 80 hunch net-2005-06-10-Workshops are not Conferences
13 0.45249474 16 hunch net-2005-02-09-Intuitions from applied learning
14 0.43183768 114 hunch net-2005-09-20-Workshop Proposal: Atomic Learning
15 0.39806706 234 hunch net-2007-02-22-Create Your Own ICML Workshop
16 0.39547479 329 hunch net-2008-11-28-A Bumper Crop of Machine Learning Graduates
17 0.38951361 265 hunch net-2007-10-14-NIPS workshp: Learning Problem Design
18 0.38625631 459 hunch net-2012-03-13-The Submodularity workshop and Lucca Professorship
19 0.34858879 404 hunch net-2010-08-20-The Workshop on Cores, Clusters, and Clouds
20 0.33747002 330 hunch net-2008-12-07-A NIPS paper
topicId topicWeight
[(27, 0.088), (37, 0.581), (53, 0.132), (55, 0.067)]
simIndex simValue blogId blogTitle
1 0.88743454 63 hunch net-2005-04-27-DARPA project: LAGR
Introduction: Larry Jackal has set up the LAGR (“Learning Applied to Ground Robotics”) project (and competition) which seems to be quite well designed. Features include: Many participants (8 going on 12?) Standardized hardware. In the DARPA grand challenge contestants entering with motorcycles are at a severe disadvantage to those entering with a Hummer. Similarly, contestants using more powerful sensors can gain huge advantages. Monthly contests, with full feedback (but since the hardware is standardized, only code is shipped). One of the premises of the program is that robust systems are desired. Monthly evaluations at different locations can help measure this and provide data. Attacks a known hard problem. (cross country driving)
same-blog 2 0.87921184 431 hunch net-2011-04-18-A paper not at Snowbird
Introduction: Unfortunately, a scheduling failure meant I missed all of AIStat and most of the learning workshop , otherwise known as Snowbird, when it’s at Snowbird . At snowbird, the talk on Sum-Product networks by Hoifung Poon stood out to me ( Pedro Domingos is a coauthor.). The basic point was that by appropriately constructing networks based on sums and products, the normalization problem in probabilistic models is eliminated, yielding a highly tractable yet flexible representation+learning algorithm. As an algorithm, this is noticeably cleaner than deep belief networks with a claim to being an order of magnitude faster and working better on an image completion task. Snowbird doesn’t have real papers—just the abstract above. I look forward to seeing the paper. (added: Rodrigo points out the deep learning workshop draft .)
3 0.61175334 368 hunch net-2009-08-26-Another 10-year paper in Machine Learning
Introduction: When I was thinking about the best “10 year paper” for ICML , I also took a look at a few other conferences. Here is one from 10 years ago that interested me: David McAllester PAC-Bayesian Model Averaging , COLT 1999. 2001 Journal Draft . Prior to this paper, the only mechanism known for controlling or estimating the necessary sample complexity for learning over continuously parameterized predictors was VC theory and variants, all of which suffered from a basic problem: they were incredibly pessimistic in practice. This meant that only very gross guidance could be provided for learning algorithm design. The PAC-Bayes bound provided an alternative approach to sample complexity bounds which was radically tighter, quantitatively. It also imported and explained many of the motivations for Bayesian learning in a way that learning theory and perhaps optimization people might appreciate. Since this paper came out, there have been a number of moderately successful attempts t
4 0.54128218 21 hunch net-2005-02-17-Learning Research Programs
Introduction: This is an attempt to organize the broad research programs related to machine learning currently underway. This isn’t easy—this map is partial, the categories often overlap, and there are many details left out. Nevertheless, it is (perhaps) helpful to have some map of what is happening where. The word ‘typical’ should not be construed narrowly here. Learning Theory Focuses on analyzing mathematical models of learning, essentially no experiments. Typical conference: COLT. Bayesian Learning Bayes law is always used. Focus on methods of speeding up or approximating integration, new probabilistic models, and practical applications. Typical conferences: NIPS,UAI Structured learning Predicting complex structured outputs, some applications. Typiical conferences: NIPS, UAI, others Reinforcement Learning Focused on ‘agent-in-the-world’ learning problems where the goal is optimizing reward. Typical conferences: ICML Unsupervised Learning/Clustering/Dimensionality Reduc
5 0.53295237 1 hunch net-2005-01-19-Why I decided to run a weblog.
Introduction: I have decided to run a weblog on machine learning and learning theory research. Here are some reasons: 1) Weblogs enable new functionality: Public comment on papers. No mechanism for this exists at conferences and most journals. I have encountered it once for a science paper. Some communities have mailing lists supporting this, but not machine learning or learning theory. I have often read papers and found myself wishing there was some method to consider other’s questions and read the replies. Conference shortlists. One of the most common conversations at a conference is “what did you find interesting?” There is no explicit mechanism for sharing this information at conferences, and it’s easy to imagine that it would be handy to do so. Evaluation and comment on research directions. Papers are almost exclusively about new research, rather than evaluation (and consideration) of research directions. This last role is satisfied by funding agencies to some extent, but
6 0.52369475 138 hunch net-2005-12-09-Some NIPS papers
7 0.33665809 194 hunch net-2006-07-11-New Models
8 0.33300143 144 hunch net-2005-12-28-Yet more nips thoughts
9 0.27105153 367 hunch net-2009-08-16-Centmail comments
10 0.26953921 2 hunch net-2005-01-24-Holy grails of machine learning?
11 0.2676205 91 hunch net-2005-07-10-Thinking the Unthought
12 0.26519865 6 hunch net-2005-01-27-Learning Complete Problems
13 0.26291949 416 hunch net-2010-10-29-To Vidoelecture or not
14 0.26131576 16 hunch net-2005-02-09-Intuitions from applied learning
15 0.25787118 145 hunch net-2005-12-29-Deadline Season
16 0.25469872 107 hunch net-2005-09-05-Site Update
17 0.25428957 25 hunch net-2005-02-20-At One Month
18 0.25270787 56 hunch net-2005-04-14-Families of Learning Theory Statements
19 0.24690625 151 hunch net-2006-01-25-1 year
20 0.24231802 438 hunch net-2011-07-11-Interesting Neural Network Papers at ICML 2011