hunch_net hunch_net-2013 hunch_net-2013-477 knowledge-graph by maker-knowledge-mining
Source: html
Introduction: 2012 was a tumultuous year for me, but it was undeniably a great year for deep learning efforts. Signs of this include: Winning a Kaggle competition . Wide adoption of deep learning for speech recognition . Significant industry support . Gains in image recognition . This is a rare event in research: a significant capability breakout. Congratulations are definitely in order for those who managed to achieve it. At this point, deep learning algorithms seem like a choice undeniably worth investigating for real applications with significant data.
sentIndex sentText sentNum sentScore
1 2012 was a tumultuous year for me, but it was undeniably a great year for deep learning efforts. [sent-1, score-1.128]
2 Signs of this include: Winning a Kaggle competition . [sent-2, score-0.136]
3 Wide adoption of deep learning for speech recognition . [sent-3, score-0.995]
4 This is a rare event in research: a significant capability breakout. [sent-6, score-0.702]
5 Congratulations are definitely in order for those who managed to achieve it. [sent-7, score-0.499]
6 At this point, deep learning algorithms seem like a choice undeniably worth investigating for real applications with significant data. [sent-8, score-1.751]
wordName wordTfidf (topN-words)
[('undeniably', 0.426), ('deep', 0.315), ('recognition', 0.292), ('gains', 0.23), ('significant', 0.226), ('investigating', 0.201), ('signs', 0.201), ('winning', 0.201), ('adoption', 0.201), ('industry', 0.192), ('capability', 0.172), ('rare', 0.172), ('definitely', 0.155), ('image', 0.152), ('wide', 0.149), ('managed', 0.146), ('speech', 0.146), ('year', 0.138), ('competition', 0.136), ('event', 0.132), ('worth', 0.121), ('include', 0.11), ('achieve', 0.107), ('support', 0.106), ('choice', 0.092), ('order', 0.091), ('applications', 0.09), ('seem', 0.075), ('great', 0.07), ('point', 0.068), ('real', 0.067), ('data', 0.059), ('algorithms', 0.052), ('research', 0.051), ('like', 0.045), ('learning', 0.041)]
simIndex simValue blogId blogTitle
same-blog 1 1.0 477 hunch net-2013-01-01-Deep Learning 2012
Introduction: 2012 was a tumultuous year for me, but it was undeniably a great year for deep learning efforts. Signs of this include: Winning a Kaggle competition . Wide adoption of deep learning for speech recognition . Significant industry support . Gains in image recognition . This is a rare event in research: a significant capability breakout. Congratulations are definitely in order for those who managed to achieve it. At this point, deep learning algorithms seem like a choice undeniably worth investigating for real applications with significant data.
2 0.22586417 201 hunch net-2006-08-07-The Call of the Deep
Introduction: Many learning algorithms used in practice are fairly simple. Viewed representationally, many prediction algorithms either compute a linear separator of basic features (perceptron, winnow, weighted majority, SVM) or perhaps a linear separator of slightly more complex features (2-layer neural networks or kernelized SVMs). Should we go beyond this, and start using “deep” representations? What is deep learning? Intuitively, deep learning is about learning to predict in ways which can involve complex dependencies between the input (observed) features. Specifying this more rigorously turns out to be rather difficult. Consider the following cases: SVM with Gaussian Kernel. This is not considered deep learning, because an SVM with a gaussian kernel can’t succinctly represent certain decision surfaces. One of Yann LeCun ‘s examples is recognizing objects based on pixel values. An SVM will need a new support vector for each significantly different background. Since the number
3 0.14211453 407 hunch net-2010-08-23-Boosted Decision Trees for Deep Learning
Introduction: About 4 years ago, I speculated that decision trees qualify as a deep learning algorithm because they can make decisions which are substantially nonlinear in the input representation. Ping Li has proved this correct, empirically at UAI by showing that boosted decision trees can beat deep belief networks on versions of Mnist which are artificially hardened so as to make them solvable only by deep learning algorithms. This is an important point, because the ability to solve these sorts of problems is probably the best objective definition of a deep learning algorithm we have. I’m not that surprised. In my experience, if you can accept the computational drawbacks of a boosted decision tree, they can achieve pretty good performance. Geoff Hinton once told me that the great thing about deep belief networks is that they work. I understand that Ping had very substantial difficulty in getting this published, so I hope some reviewers step up to the standard of valuing wha
4 0.11305062 268 hunch net-2007-10-19-Second Annual Reinforcement Learning Competition
Introduction: The Second Annual Reinforcement Learning Competition is about to get started. The aim of the competition is to facilitate direct comparisons between various learning methods on important and realistic domains. This year’s event will feature well-known benchmark domains as well as more challenging problems of real-world complexity, such as helicopter control and robot soccer keepaway. The competition begins on November 1st, 2007 when training software is released. Results must be submitted by July 1st, 2008. The competition will culminate in an event at ICML-08 in Helsinki, Finland, at which the winners will be announced. For more information, visit the competition website.
5 0.10757519 227 hunch net-2007-01-10-A Deep Belief Net Learning Problem
Introduction: “Deep learning” is used to describe learning architectures which have significant depth (as a circuit). One claim is that shallow architectures (one or two layers) can not concisely represent some functions while a circuit with more depth can concisely represent these same functions. Proving lower bounds on the size of a circuit is substantially harder than upper bounds (which are constructive), but some results are known. Luca Trevisan ‘s class notes detail how XOR is not concisely representable by “AC0″ (= constant depth unbounded fan-in AND, OR, NOT gates). This doesn’t quite prove that depth is necessary for the representations commonly used in learning (such as a thresholded weighted sum), but it is strongly suggestive that this is so. Examples like this are a bit disheartening because existing algorithms for deep learning (deep belief nets, gradient descent on deep neural networks, and a perhaps decision trees depending on who you ask) can’t learn XOR very easily.
6 0.099001467 424 hunch net-2011-02-17-What does Watson mean?
7 0.097044721 490 hunch net-2013-11-09-Graduates and Postdocs
8 0.095814563 475 hunch net-2012-10-26-ML Symposium and Strata-Hadoop World
9 0.087646164 291 hunch net-2008-03-07-Spock Challenge Winners
10 0.078086339 329 hunch net-2008-11-28-A Bumper Crop of Machine Learning Graduates
11 0.078024574 431 hunch net-2011-04-18-A paper not at Snowbird
12 0.07402432 352 hunch net-2009-05-06-Machine Learning to AI
13 0.069313347 158 hunch net-2006-02-24-A Fundamentalist Organization of Machine Learning
14 0.06865447 20 hunch net-2005-02-15-ESPgame and image labeling
15 0.066849001 159 hunch net-2006-02-27-The Peekaboom Dataset
16 0.065384254 276 hunch net-2007-12-10-Learning Track of International Planning Competition
17 0.064365178 494 hunch net-2014-03-11-The New York ML Symposium, take 2
18 0.064148426 396 hunch net-2010-04-28-CI Fellows program renewed
19 0.061416309 266 hunch net-2007-10-15-NIPS workshops extended to 3 days
20 0.06108835 132 hunch net-2005-11-26-The Design of an Optimal Research Environment
topicId topicWeight
[(0, 0.109), (1, -0.004), (2, -0.079), (3, 0.011), (4, 0.048), (5, -0.022), (6, -0.044), (7, 0.042), (8, 0.014), (9, -0.097), (10, -0.082), (11, -0.056), (12, -0.085), (13, -0.146), (14, -0.063), (15, 0.127), (16, 0.029), (17, 0.013), (18, -0.086), (19, 0.094), (20, -0.02), (21, 0.023), (22, 0.032), (23, 0.007), (24, 0.018), (25, -0.015), (26, -0.015), (27, 0.022), (28, 0.027), (29, 0.057), (30, 0.097), (31, 0.049), (32, -0.051), (33, 0.048), (34, 0.031), (35, -0.109), (36, 0.154), (37, -0.062), (38, -0.06), (39, 0.032), (40, -0.04), (41, 0.064), (42, -0.034), (43, -0.055), (44, -0.043), (45, 0.053), (46, 0.032), (47, 0.011), (48, 0.076), (49, -0.061)]
simIndex simValue blogId blogTitle
same-blog 1 0.97005951 477 hunch net-2013-01-01-Deep Learning 2012
Introduction: 2012 was a tumultuous year for me, but it was undeniably a great year for deep learning efforts. Signs of this include: Winning a Kaggle competition . Wide adoption of deep learning for speech recognition . Significant industry support . Gains in image recognition . This is a rare event in research: a significant capability breakout. Congratulations are definitely in order for those who managed to achieve it. At this point, deep learning algorithms seem like a choice undeniably worth investigating for real applications with significant data.
2 0.77885902 201 hunch net-2006-08-07-The Call of the Deep
Introduction: Many learning algorithms used in practice are fairly simple. Viewed representationally, many prediction algorithms either compute a linear separator of basic features (perceptron, winnow, weighted majority, SVM) or perhaps a linear separator of slightly more complex features (2-layer neural networks or kernelized SVMs). Should we go beyond this, and start using “deep” representations? What is deep learning? Intuitively, deep learning is about learning to predict in ways which can involve complex dependencies between the input (observed) features. Specifying this more rigorously turns out to be rather difficult. Consider the following cases: SVM with Gaussian Kernel. This is not considered deep learning, because an SVM with a gaussian kernel can’t succinctly represent certain decision surfaces. One of Yann LeCun ‘s examples is recognizing objects based on pixel values. An SVM will need a new support vector for each significantly different background. Since the number
3 0.7786268 407 hunch net-2010-08-23-Boosted Decision Trees for Deep Learning
Introduction: About 4 years ago, I speculated that decision trees qualify as a deep learning algorithm because they can make decisions which are substantially nonlinear in the input representation. Ping Li has proved this correct, empirically at UAI by showing that boosted decision trees can beat deep belief networks on versions of Mnist which are artificially hardened so as to make them solvable only by deep learning algorithms. This is an important point, because the ability to solve these sorts of problems is probably the best objective definition of a deep learning algorithm we have. I’m not that surprised. In my experience, if you can accept the computational drawbacks of a boosted decision tree, they can achieve pretty good performance. Geoff Hinton once told me that the great thing about deep belief networks is that they work. I understand that Ping had very substantial difficulty in getting this published, so I hope some reviewers step up to the standard of valuing wha
4 0.60621655 224 hunch net-2006-12-12-Interesting Papers at NIPS 2006
Introduction: Here are some papers that I found surprisingly interesting. Yoshua Bengio , Pascal Lamblin, Dan Popovici, Hugo Larochelle, Greedy Layer-wise Training of Deep Networks . Empirically investigates some of the design choices behind deep belief networks. Long Zhu , Yuanhao Chen, Alan Yuille Unsupervised Learning of a Probabilistic Grammar for Object Detection and Parsing. An unsupervised method for detecting objects using simple feature filters that works remarkably well on the (supervised) caltech-101 dataset . Shai Ben-David , John Blitzer , Koby Crammer , and Fernando Pereira , Analysis of Representations for Domain Adaptation . This is the first analysis I’ve seen of learning with respect to samples drawn differently from the evaluation distribution which depends on reasonable measurable quantities. All of these papers turn out to have a common theme—the power of unlabeled data to do generically useful things.
5 0.59716445 431 hunch net-2011-04-18-A paper not at Snowbird
Introduction: Unfortunately, a scheduling failure meant I missed all of AIStat and most of the learning workshop , otherwise known as Snowbird, when it’s at Snowbird . At snowbird, the talk on Sum-Product networks by Hoifung Poon stood out to me ( Pedro Domingos is a coauthor.). The basic point was that by appropriately constructing networks based on sums and products, the normalization problem in probabilistic models is eliminated, yielding a highly tractable yet flexible representation+learning algorithm. As an algorithm, this is noticeably cleaner than deep belief networks with a claim to being an order of magnitude faster and working better on an image completion task. Snowbird doesn’t have real papers—just the abstract above. I look forward to seeing the paper. (added: Rodrigo points out the deep learning workshop draft .)
6 0.56844479 438 hunch net-2011-07-11-Interesting Neural Network Papers at ICML 2011
7 0.55700767 227 hunch net-2007-01-10-A Deep Belief Net Learning Problem
8 0.51500839 16 hunch net-2005-02-09-Intuitions from applied learning
9 0.4960815 329 hunch net-2008-11-28-A Bumper Crop of Machine Learning Graduates
10 0.48861349 424 hunch net-2011-02-17-What does Watson mean?
11 0.43085638 490 hunch net-2013-11-09-Graduates and Postdocs
12 0.39568031 266 hunch net-2007-10-15-NIPS workshops extended to 3 days
13 0.38842684 3 hunch net-2005-01-24-The Humanloop Spectrum of Machine Learning
14 0.38641202 456 hunch net-2012-02-24-ICML+50%
15 0.36988875 349 hunch net-2009-04-21-Interesting Presentations at Snowbird
16 0.36406475 232 hunch net-2007-02-11-24
17 0.36307529 152 hunch net-2006-01-30-Should the Input Representation be a Vector?
18 0.35911646 276 hunch net-2007-12-10-Learning Track of International Planning Competition
19 0.3580195 475 hunch net-2012-10-26-ML Symposium and Strata-Hadoop World
20 0.35759079 119 hunch net-2005-10-08-We have a winner
topicId topicWeight
[(27, 0.123), (55, 0.112), (86, 0.461), (94, 0.151)]
simIndex simValue blogId blogTitle
same-blog 1 0.90849358 477 hunch net-2013-01-01-Deep Learning 2012
Introduction: 2012 was a tumultuous year for me, but it was undeniably a great year for deep learning efforts. Signs of this include: Winning a Kaggle competition . Wide adoption of deep learning for speech recognition . Significant industry support . Gains in image recognition . This is a rare event in research: a significant capability breakout. Congratulations are definitely in order for those who managed to achieve it. At this point, deep learning algorithms seem like a choice undeniably worth investigating for real applications with significant data.
2 0.86280352 54 hunch net-2005-04-08-Fast SVMs
Introduction: There was a presentation at snowbird about parallelized support vector machines. In many cases, people parallelize by ignoring serial operations, but that is not what happened here—they parallelize with optimizations. Consequently, this seems to be the fastest SVM in existence. There is a related paper here .
3 0.74543953 273 hunch net-2007-11-16-MLSS 2008
Introduction: … is in Kioloa, Australia from March 3 to March 14. It’s a great chance to learn something about Machine Learning and I’ve enjoyed several previous Machine Learning Summer Schools . The website has many more details , but registration is open now for the first 80 to sign up.
4 0.6253531 281 hunch net-2007-12-21-Vowpal Wabbit Code Release
Introduction: We are releasing the Vowpal Wabbit (Fast Online Learning) code as open source under a BSD (revised) license. This is a project at Yahoo! Research to build a useful large scale learning algorithm which Lihong Li , Alex Strehl , and I have been working on. To appreciate the meaning of “large”, it’s useful to define “small” and “medium”. A “small” supervised learning problem is one where a human could use a labeled dataset and come up with a reasonable predictor. A “medium” supervised learning problem dataset fits into the RAM of a modern desktop computer. A “large” supervised learning problem is one which does not fit into the RAM of a normal machine. VW tackles large scale learning problems by this definition of large. I’m not aware of any other open source Machine Learning tools which can handle this scale (although they may exist). A few close ones are: IBM’s Parallel Machine Learning Toolbox isn’t quite open source . The approach used by this toolbox is essenti
5 0.42042303 221 hunch net-2006-12-04-Structural Problems in NIPS Decision Making
Introduction: This is a very difficult post to write, because it is about a perenially touchy subject. Nevertheless, it is an important one which needs to be thought about carefully. There are a few things which should be understood: The system is changing and responsive. We-the-authors are we-the-reviewers, we-the-PC, and even we-the-NIPS-board. NIPS has implemented ‘secondary program chairs’, ‘author response’, and ‘double blind reviewing’ in the last few years to help with the decision process, and more changes may happen in the future. Agreement creates a perception of correctness. When any PC meets and makes a group decision about a paper, there is a strong tendency for the reinforcement inherent in a group decision to create the perception of correctness. For the many people who have been on the NIPS PC it’s reasonable to entertain a healthy skepticism in the face of this reinforcing certainty. This post is about structural problems. What problems arise because of the structure
6 0.41296855 229 hunch net-2007-01-26-Parallel Machine Learning Problems
7 0.40491873 120 hunch net-2005-10-10-Predictive Search is Coming
8 0.40138865 276 hunch net-2007-12-10-Learning Track of International Planning Competition
9 0.40115038 146 hunch net-2006-01-06-MLTV
10 0.39782119 81 hunch net-2005-06-13-Wikis for Summer Schools and Workshops
11 0.39453551 346 hunch net-2009-03-18-Parallel ML primitives
12 0.39421201 115 hunch net-2005-09-26-Prediction Bounds as the Mathematics of Science
13 0.39113009 379 hunch net-2009-11-23-ICML 2009 Workshops (and Tutorials)
14 0.38739869 423 hunch net-2011-02-02-User preferences for search engines
15 0.38562313 419 hunch net-2010-12-04-Vowpal Wabbit, version 5.0, and the second heresy
16 0.38463092 315 hunch net-2008-09-03-Bidding Problems
17 0.38441479 136 hunch net-2005-12-07-Is the Google way the way for machine learning?
18 0.38310528 40 hunch net-2005-03-13-Avoiding Bad Reviewing
19 0.38264897 96 hunch net-2005-07-21-Six Months
20 0.38102221 286 hunch net-2008-01-25-Turing’s Club for Machine Learning