fast_ml fast_ml-2013 fast_ml-2013-45 knowledge-graph by maker-knowledge-mining
Source: html
Introduction: Object recognition in images is where deep learning, and specifically convolutional neural networks, are often applied and benchmarked these days. To get a piece of the action, we’ll be using Alex Krizhevsky’s cuda-convnet , a shining diamond of machine learning software, in a Kaggle competition. Continuing to run things on a GPU, we turn to applying convolutional neural networks for object recognition. This kind of network was developed by Yann LeCun and it’s powerful, but a bit complicated: Image credit: EBLearn tutorial A typical convolutional network has two parts. The first is responsible for feature extraction and consists of one or more pairs of convolution and subsampling/max-pooling layers, as you can see above. The second part is just a classic fully-connected multilayer perceptron taking extracted features as input. For a detailed explanation of all this see unit 9 in Hugo LaRochelle’s neural networks course . Daniel Nouri has an interesting story about
sentIndex sentText sentNum sentScore
1 Object recognition in images is where deep learning, and specifically convolutional neural networks, are often applied and benchmarked these days. [sent-1, score-0.449]
2 Continuing to run things on a GPU, we turn to applying convolutional neural networks for object recognition. [sent-3, score-0.459]
3 Daniel Nouri has an interesting story about applying convnets to audio classification . [sent-8, score-0.247]
4 The set is a collection of 32x32 color images of cats, dogs, cars, trucks etc. [sent-13, score-0.341]
5 The training set is the same, the test set images are modified and buried among 290k other images to discourage cheating. [sent-16, score-0.639]
6 Worth mentioning is noccn , a collection of wrappers around cuda-convnet by Daniel Nouri. [sent-60, score-0.297]
7 Noccn can be used to make batches and produce predictions, although it’s a tiny bit complicated so for the purposes of the task at hand we chose to roll out some simpler code. [sent-62, score-0.367]
8 Images into batches As usual, we need to figure out how to get data in and predictions out. [sent-68, score-0.364]
9 Kaggle provides PNG images, while cuda-convnet expects data in form of batch files. [sent-69, score-0.254]
10 A batch file is a pickled Python dictionary containing at least image data and labels. [sent-70, score-0.511]
11 array( image ) # 32 x 32 x 3 image = np. [sent-76, score-0.558]
12 rollaxis( image, 2 ) # 3 x 32 x 32 image = image. [sent-77, score-0.279]
13 You can verify that it works by loading a train batch and printing the first row from the data array. [sent-80, score-0.224]
14 py test test_batches We end up with 30 new batches that we’d like predictions for. [sent-84, score-0.427]
15 The predictions How do you get predictions from cuda-convnet ? [sent-85, score-0.204]
16 They are saved in a batch file somewhat similiar to the training batches. [sent-89, score-0.285]
17 py -f--write-features=probs --feature-path=/path/to/predictions/dir --test-range=7-36 But wait… We followed the procedure for training the model with 13% test error. [sent-91, score-0.165]
18 This will produce better results than testing on the center patch alone. [sent-93, score-0.288]
19 It seems that we’re going to use multiview predictions from noccn . [sent-97, score-0.379]
20 Online image classification demos If you’d like your own image classified, try these online demos: http://horatio. [sent-108, score-0.73]
wordName wordTfidf (topN-words)
[('logprob', 0.297), ('sec', 0.297), ('image', 0.279), ('batches', 0.262), ('images', 0.222), ('noccn', 0.178), ('patches', 0.178), ('batch', 0.175), ('convnets', 0.148), ('object', 0.135), ('center', 0.119), ('collection', 0.119), ('demos', 0.119), ('patch', 0.119), ('png', 0.119), ('probs', 0.119), ('predictions', 0.102), ('applying', 0.099), ('connections', 0.099), ('daniel', 0.099), ('multiview', 0.099), ('convolutional', 0.094), ('array', 0.087), ('krizhevsky', 0.087), ('alex', 0.079), ('provides', 0.079), ('modified', 0.079), ('http', 0.073), ('recognition', 0.073), ('software', 0.071), ('networks', 0.071), ('axis', 0.063), ('test', 0.063), ('neural', 0.06), ('error', 0.059), ('file', 0.057), ('complicated', 0.056), ('training', 0.053), ('online', 0.053), ('names', 0.05), ('author', 0.05), ('testing', 0.05), ('followed', 0.049), ('roll', 0.049), ('verify', 0.049), ('afterwards', 0.049), ('averaged', 0.049), ('kill', 0.049), ('enumerate', 0.049), ('action', 0.049)]
simIndex simValue blogId blogTitle
same-blog 1 0.99999982 45 fast ml-2013-11-27-Object recognition in images with cuda-convnet
Introduction: Object recognition in images is where deep learning, and specifically convolutional neural networks, are often applied and benchmarked these days. To get a piece of the action, we’ll be using Alex Krizhevsky’s cuda-convnet , a shining diamond of machine learning software, in a Kaggle competition. Continuing to run things on a GPU, we turn to applying convolutional neural networks for object recognition. This kind of network was developed by Yann LeCun and it’s powerful, but a bit complicated: Image credit: EBLearn tutorial A typical convolutional network has two parts. The first is responsible for feature extraction and consists of one or more pairs of convolution and subsampling/max-pooling layers, as you can see above. The second part is just a classic fully-connected multilayer perceptron taking extracted features as input. For a detailed explanation of all this see unit 9 in Hugo LaRochelle’s neural networks course . Daniel Nouri has an interesting story about
2 0.16103074 49 fast ml-2014-01-10-Classifying images with a pre-trained deep network
Introduction: Recently at least two research teams made their pre-trained deep convolutional networks available, so you can classify your images right away. We’ll see how to go about it, with data from the Cats & Dogs competition at Kaggle as an example. We’ll be using OverFeat , a classifier and feature extractor from the New York guys lead by Yann LeCun and Rob Fergus. The principal author, Pierre Sermanet, is currently first on the Dogs vs. Cats leaderboard . The other available implementation we know of comes from Berkeley. It’s called Caffe and is a successor to decaf . Yangqing Jia , the main author of these, is also near the top of the leaderboard. Both networks were trained on ImageNet , which is an image database organized according to the WordNet hierarchy . It was the ImageNet Large Scale Visual Recognition Challenge 2012 in which Alex Krizhevsky crushed the competition with his network. His error was 16%, the second best - 26%. Data The Kaggle competition featur
3 0.14511053 52 fast ml-2014-02-02-Yesterday a kaggler, today a Kaggle master: a wrap-up of the cats and dogs competition
Introduction: Out of 215 contestants, we placed 8th in the Cats and Dogs competition at Kaggle. The top ten finish gave us the master badge. The competition was about discerning the animals in images and here’s how we did it. We extracted the features using pre-trained deep convolutional networks, specifically decaf and OverFeat . Then we trained some classifiers on these features. The whole thing was inspired by Kyle Kastner’s decaf + pylearn2 combo and we expanded this idea. The classifiers were linear models from scikit-learn and a neural network from Pylearn2 . At the end we created a voting ensemble of the individual models. OverFeat features We touched on OverFeat in Classifying images with a pre-trained deep network . A better way to use it in this competition’s context is to extract the features from the layer before the classifier, as Pierre Sermanet suggested in the comments. Concretely, in the larger OverFeat model ( -l ) layer 24 is the softmax, at least in the
4 0.13355611 50 fast ml-2014-01-20-How to get predictions from Pylearn2
Introduction: A while ago we’ve shown how to get predictions from a Pylearn2 model. It is a little tricky, partly because of splitting data into batches. If you’re able to fit your data in memory, you can strip the batch handling code and it becomes easier to see what’s going on. We exercise the concept to distinguish cats from dogs again, with superior results. Step by step You have a pickled model from Pylearn2. Let’s load it: from pylearn2.utils import serial model_path = 'model.pkl' model = serial.load( model_path ) Next, some Theano weirdness. Theano is a compiler for symbolic expressions and with these expressions we deal when predicting. We need to define expressions for X and Y: X = model.get_input_space().make_theano_batch() Y = model.fprop( X ) Mind you, these are not variables, but rather descriptions of how to get variables. Y is easy to understand: just feed the data to the model and forward-propagate. X is more of an idiom, the incantations above make sur
5 0.10008932 43 fast ml-2013-11-02-Maxing out the digits
Introduction: Recently we’ve been investigating the basics of Pylearn2 . Now it’s time for a more advanced example: a multilayer perceptron with dropout and maxout activation for the MNIST digits. Maxout explained If you’ve been following developments in deep learning, you know that Hinton’s most recent recommendation for supervised learning, after a few years of bashing backpropagation in favour of unsupervised pretraining, is to use classic multilayer perceptrons with dropout and rectified linear units. For us, this breath of simplicity is a welcome change. Rectified linear is f(x) = max( 0, x ) . This makes backpropagation trivial: for x > 0, the derivative is one, else zero. Note that ReLU consists of two linear functions. But why stop at two? Let’s take max. out of three, or four, or five linear functions… And so maxout is a generalization of ReLU. It can approximate any convex function. Now backpropagation is easy and dropout prevents overfitting, so we can train a deep
6 0.096142933 27 fast ml-2013-05-01-Deep learning made easy
7 0.093630932 40 fast ml-2013-10-06-Pylearn2 in practice
8 0.091367759 62 fast ml-2014-05-26-Yann LeCun's answers from the Reddit AMA
9 0.089510903 48 fast ml-2013-12-28-Regularizing neural networks with dropout and with DropConnect
10 0.079324201 34 fast ml-2013-07-14-Running things on a GPU
11 0.074428909 19 fast ml-2013-02-07-The secret of the big guys
12 0.074271381 12 fast ml-2012-12-21-Tuning hyperparams automatically with Spearmint
13 0.07425908 7 fast ml-2012-10-05-Predicting closed questions on Stack Overflow
14 0.073573276 46 fast ml-2013-12-07-13 NIPS papers that caught our eye
15 0.073227532 32 fast ml-2013-07-05-Processing large files, line by line
16 0.071663953 26 fast ml-2013-04-17-Regression as classification
17 0.071058683 54 fast ml-2014-03-06-PyBrain - a simple neural networks library in Python
18 0.069694147 3 fast ml-2012-09-01-Running Unix apps on Windows
19 0.068268977 20 fast ml-2013-02-18-Predicting advertised salaries
20 0.067301735 18 fast ml-2013-01-17-A very fast denoising autoencoder
topicId topicWeight
[(0, 0.286), (1, 0.069), (2, 0.127), (3, 0.03), (4, -0.258), (5, -0.219), (6, -0.189), (7, 0.028), (8, -0.001), (9, 0.026), (10, -0.046), (11, 0.129), (12, 0.069), (13, 0.058), (14, 0.022), (15, 0.002), (16, 0.057), (17, -0.102), (18, -0.054), (19, 0.008), (20, -0.056), (21, 0.048), (22, 0.019), (23, 0.123), (24, 0.231), (25, -0.008), (26, 0.082), (27, -0.049), (28, 0.15), (29, 0.062), (30, 0.158), (31, 0.031), (32, -0.089), (33, -0.138), (34, -0.211), (35, -0.136), (36, 0.077), (37, -0.233), (38, 0.053), (39, 0.162), (40, 0.092), (41, -0.03), (42, 0.252), (43, -0.265), (44, -0.277), (45, -0.225), (46, -0.29), (47, -0.093), (48, -0.008), (49, 0.022)]
simIndex simValue blogId blogTitle
same-blog 1 0.97270662 45 fast ml-2013-11-27-Object recognition in images with cuda-convnet
Introduction: Object recognition in images is where deep learning, and specifically convolutional neural networks, are often applied and benchmarked these days. To get a piece of the action, we’ll be using Alex Krizhevsky’s cuda-convnet , a shining diamond of machine learning software, in a Kaggle competition. Continuing to run things on a GPU, we turn to applying convolutional neural networks for object recognition. This kind of network was developed by Yann LeCun and it’s powerful, but a bit complicated: Image credit: EBLearn tutorial A typical convolutional network has two parts. The first is responsible for feature extraction and consists of one or more pairs of convolution and subsampling/max-pooling layers, as you can see above. The second part is just a classic fully-connected multilayer perceptron taking extracted features as input. For a detailed explanation of all this see unit 9 in Hugo LaRochelle’s neural networks course . Daniel Nouri has an interesting story about
2 0.33106869 49 fast ml-2014-01-10-Classifying images with a pre-trained deep network
Introduction: Recently at least two research teams made their pre-trained deep convolutional networks available, so you can classify your images right away. We’ll see how to go about it, with data from the Cats & Dogs competition at Kaggle as an example. We’ll be using OverFeat , a classifier and feature extractor from the New York guys lead by Yann LeCun and Rob Fergus. The principal author, Pierre Sermanet, is currently first on the Dogs vs. Cats leaderboard . The other available implementation we know of comes from Berkeley. It’s called Caffe and is a successor to decaf . Yangqing Jia , the main author of these, is also near the top of the leaderboard. Both networks were trained on ImageNet , which is an image database organized according to the WordNet hierarchy . It was the ImageNet Large Scale Visual Recognition Challenge 2012 in which Alex Krizhevsky crushed the competition with his network. His error was 16%, the second best - 26%. Data The Kaggle competition featur
3 0.23118678 52 fast ml-2014-02-02-Yesterday a kaggler, today a Kaggle master: a wrap-up of the cats and dogs competition
Introduction: Out of 215 contestants, we placed 8th in the Cats and Dogs competition at Kaggle. The top ten finish gave us the master badge. The competition was about discerning the animals in images and here’s how we did it. We extracted the features using pre-trained deep convolutional networks, specifically decaf and OverFeat . Then we trained some classifiers on these features. The whole thing was inspired by Kyle Kastner’s decaf + pylearn2 combo and we expanded this idea. The classifiers were linear models from scikit-learn and a neural network from Pylearn2 . At the end we created a voting ensemble of the individual models. OverFeat features We touched on OverFeat in Classifying images with a pre-trained deep network . A better way to use it in this competition’s context is to extract the features from the layer before the classifier, as Pierre Sermanet suggested in the comments. Concretely, in the larger OverFeat model ( -l ) layer 24 is the softmax, at least in the
4 0.2004535 62 fast ml-2014-05-26-Yann LeCun's answers from the Reddit AMA
Introduction: On May 15th Yann LeCun answered “ask me anything” questions on Reddit . We hand-picked some of his thoughts and grouped them by topic for your enjoyment. Toronto, Montreal and New York All three groups are strong and complementary. Geoff (who spends more time at Google than in Toronto now) and Russ Salakhutdinov like RBMs and deep Boltzmann machines. I like the idea of Boltzmann machines (it’s a beautifully simple concept) but it doesn’t scale well. Also, I totally hate sampling. Yoshua and his colleagues have focused a lot on various unsupervised learning, including denoising auto-encoders, contracting auto-encoders. They are not allergic to sampling like I am. On the application side, they have worked on text, not so much on images. In our lab at NYU (Rob Fergus, David Sontag, me and our students and postdocs), we have been focusing on sparse auto-encoders for unsupervised learning. They have the advantage of scaling well. We have also worked on applications, mostly to v
5 0.19435284 27 fast ml-2013-05-01-Deep learning made easy
Introduction: As usual, there’s an interesting competition at Kaggle: The Black Box. It’s connected to ICML 2013 Workshop on Challenges in Representation Learning, held by the deep learning guys from Montreal. There are a couple benchmarks for this competition and the best one is unusually hard to beat 1 - only less than a fourth of those taking part managed to do so. We’re among them. Here’s how. The key ingredient in our success is a recently developed secret Stanford technology for deep unsupervised learning: sparse filtering by Jiquan Ngiam et al. Actually, it’s not secret. It’s available at Github , and has one or two very appealling properties. Let us explain. The main idea of deep unsupervised learning, as we understand it, is feature extraction. One of the most common applications is in multimedia. The reason for that is that multimedia tasks, for example object recognition, are easy for humans, but difficult for computers 2 . Geoff Hinton from Toronto talks about two ends
6 0.18320204 50 fast ml-2014-01-20-How to get predictions from Pylearn2
7 0.18142211 43 fast ml-2013-11-02-Maxing out the digits
8 0.17304336 32 fast ml-2013-07-05-Processing large files, line by line
9 0.17202249 48 fast ml-2013-12-28-Regularizing neural networks with dropout and with DropConnect
10 0.15891428 40 fast ml-2013-10-06-Pylearn2 in practice
11 0.15630169 34 fast ml-2013-07-14-Running things on a GPU
12 0.15313369 7 fast ml-2012-10-05-Predicting closed questions on Stack Overflow
13 0.15191679 54 fast ml-2014-03-06-PyBrain - a simple neural networks library in Python
14 0.1471047 26 fast ml-2013-04-17-Regression as classification
15 0.14511605 19 fast ml-2013-02-07-The secret of the big guys
16 0.13845077 31 fast ml-2013-06-19-Go non-linear with Vowpal Wabbit
17 0.13621177 18 fast ml-2013-01-17-A very fast denoising autoencoder
18 0.13567618 12 fast ml-2012-12-21-Tuning hyperparams automatically with Spearmint
19 0.1349875 3 fast ml-2012-09-01-Running Unix apps on Windows
20 0.12834099 44 fast ml-2013-11-18-CUDA on a Linux laptop
topicId topicWeight
[(6, 0.019), (26, 0.055), (31, 0.068), (35, 0.094), (41, 0.011), (43, 0.013), (48, 0.014), (51, 0.045), (55, 0.029), (58, 0.011), (69, 0.102), (71, 0.022), (75, 0.368), (79, 0.023), (84, 0.015), (97, 0.021), (99, 0.026)]
simIndex simValue blogId blogTitle
same-blog 1 0.85988563 45 fast ml-2013-11-27-Object recognition in images with cuda-convnet
Introduction: Object recognition in images is where deep learning, and specifically convolutional neural networks, are often applied and benchmarked these days. To get a piece of the action, we’ll be using Alex Krizhevsky’s cuda-convnet , a shining diamond of machine learning software, in a Kaggle competition. Continuing to run things on a GPU, we turn to applying convolutional neural networks for object recognition. This kind of network was developed by Yann LeCun and it’s powerful, but a bit complicated: Image credit: EBLearn tutorial A typical convolutional network has two parts. The first is responsible for feature extraction and consists of one or more pairs of convolution and subsampling/max-pooling layers, as you can see above. The second part is just a classic fully-connected multilayer perceptron taking extracted features as input. For a detailed explanation of all this see unit 9 in Hugo LaRochelle’s neural networks course . Daniel Nouri has an interesting story about
2 0.40613064 49 fast ml-2014-01-10-Classifying images with a pre-trained deep network
Introduction: Recently at least two research teams made their pre-trained deep convolutional networks available, so you can classify your images right away. We’ll see how to go about it, with data from the Cats & Dogs competition at Kaggle as an example. We’ll be using OverFeat , a classifier and feature extractor from the New York guys lead by Yann LeCun and Rob Fergus. The principal author, Pierre Sermanet, is currently first on the Dogs vs. Cats leaderboard . The other available implementation we know of comes from Berkeley. It’s called Caffe and is a successor to decaf . Yangqing Jia , the main author of these, is also near the top of the leaderboard. Both networks were trained on ImageNet , which is an image database organized according to the WordNet hierarchy . It was the ImageNet Large Scale Visual Recognition Challenge 2012 in which Alex Krizhevsky crushed the competition with his network. His error was 16%, the second best - 26%. Data The Kaggle competition featur
3 0.37740311 52 fast ml-2014-02-02-Yesterday a kaggler, today a Kaggle master: a wrap-up of the cats and dogs competition
Introduction: Out of 215 contestants, we placed 8th in the Cats and Dogs competition at Kaggle. The top ten finish gave us the master badge. The competition was about discerning the animals in images and here’s how we did it. We extracted the features using pre-trained deep convolutional networks, specifically decaf and OverFeat . Then we trained some classifiers on these features. The whole thing was inspired by Kyle Kastner’s decaf + pylearn2 combo and we expanded this idea. The classifiers were linear models from scikit-learn and a neural network from Pylearn2 . At the end we created a voting ensemble of the individual models. OverFeat features We touched on OverFeat in Classifying images with a pre-trained deep network . A better way to use it in this competition’s context is to extract the features from the layer before the classifier, as Pierre Sermanet suggested in the comments. Concretely, in the larger OverFeat model ( -l ) layer 24 is the softmax, at least in the
4 0.34448677 9 fast ml-2012-10-25-So you want to work for Facebook
Introduction: Good news, everyone! There’s a new contest on Kaggle - Facebook is looking for talent . They won’t pay, but just might interview. This post is in a way a bonus for active readers because most visitors of fastml.com originally come from Kaggle forums. For this competition the forums are disabled to encourage own work . To honor this, we won’t publish any code. But own work doesn’t mean original work , and we wouldn’t want to reinvent the wheel, would we? The contest differs substantially from a Kaggle stereotype, if there is such a thing, in three major ways: there’s no money prizes, as mentioned above it’s not a real world problem, but rather an assignment to screen job candidates (this has important consequences, described below) it’s not a typical machine learning project, but rather a broader AI exercise You are given a graph of the internet, actually a snapshot of the graph for each of 15 time steps. You are also given a bunch of paths in this graph, which a
5 0.34004843 40 fast ml-2013-10-06-Pylearn2 in practice
Introduction: What do you get when you mix one part brilliant and one part daft? You get Pylearn2, a cutting edge neural networks library from Montreal that’s rather hard to use. Here we’ll show how to get through the daft part with your mental health relatively intact. Pylearn2 comes from the Lisa Lab in Montreal , led by Yoshua Bengio. Those are pretty smart guys and they concern themselves with deep learning. Recently they published a paper entitled Pylearn2: a machine learning research library [arxiv] . Here’s a quote: Pylearn2 is a machine learning research library - its users are researchers . This means (…) it is acceptable to assume that the user has some technical sophistication and knowledge of machine learning. The word research is possibly the most common word in the paper. There’s a reason for that: the library is certainly not production-ready. OK, it’s not that bad. There are only two difficult things: getting your data in getting predictions out What’
6 0.32321236 48 fast ml-2013-12-28-Regularizing neural networks with dropout and with DropConnect
7 0.31728762 12 fast ml-2012-12-21-Tuning hyperparams automatically with Spearmint
8 0.31542373 10 fast ml-2012-11-17-The Facebook challenge HOWTO
9 0.31076932 54 fast ml-2014-03-06-PyBrain - a simple neural networks library in Python
10 0.31062528 23 fast ml-2013-03-18-Large scale L1 feature selection with Vowpal Wabbit
11 0.30975544 7 fast ml-2012-10-05-Predicting closed questions on Stack Overflow
12 0.30761087 18 fast ml-2013-01-17-A very fast denoising autoencoder
13 0.30580691 26 fast ml-2013-04-17-Regression as classification
14 0.30481064 34 fast ml-2013-07-14-Running things on a GPU
15 0.29826823 17 fast ml-2013-01-14-Feature selection in practice
16 0.29819319 31 fast ml-2013-06-19-Go non-linear with Vowpal Wabbit
17 0.29735416 55 fast ml-2014-03-20-Good representations, distance, metric learning and supervised dimensionality reduction
18 0.29442784 43 fast ml-2013-11-02-Maxing out the digits
19 0.29153565 13 fast ml-2012-12-27-Spearmint with a random forest
20 0.29129165 27 fast ml-2013-05-01-Deep learning made easy