hunch_net hunch_net-2009 hunch_net-2009-365 knowledge-graph by maker-knowledge-mining
Source: html
Introduction: Today brings a new release of the Vowpal Wabbit fast online learning software. This time, unlike the previous release, the project itself is going open source, developing via github . For example, the lastest and greatest can be downloaded via: git clone git://github.com/JohnLangford/vowpal_wabbit.git If you aren’t familiar with git , it’s a distributed version control system which supports quick and easy branching, as well as reconciliation. This version of the code is confirmed to compile without complaint on at least some flavors of OSX as well as Linux boxes. As much of the point of this project is pushing the limits of fast and effective machine learning, let me mention a few datapoints from my experience. The program can effectively scale up to batch-style training on sparse terafeature (i.e. 10 12 sparse feature) size datasets. The limiting factor is typically i/o. I started using the the real datasets from the large-scale learning workshop as a conve
sentIndex sentText sentNum sentScore
1 Today brings a new release of the Vowpal Wabbit fast online learning software. [sent-1, score-0.375]
2 This time, unlike the previous release, the project itself is going open source, developing via github . [sent-2, score-0.188]
3 For example, the lastest and greatest can be downloaded via: git clone git://github. [sent-3, score-0.52]
4 git If you aren’t familiar with git , it’s a distributed version control system which supports quick and easy branching, as well as reconciliation. [sent-5, score-0.389]
5 This version of the code is confirmed to compile without complaint on at least some flavors of OSX as well as Linux boxes. [sent-6, score-0.622]
6 As much of the point of this project is pushing the limits of fast and effective machine learning, let me mention a few datapoints from my experience. [sent-7, score-0.495]
7 The program can effectively scale up to batch-style training on sparse terafeature (i. [sent-8, score-0.195]
8 I started using the the real datasets from the large-scale learning workshop as a convenient benchmark. [sent-12, score-0.236]
9 (This is using the native features that the organizers intended as a starting point, yet all contestants used. [sent-14, score-0.317]
10 In some cases, that admittedly gives you performance nowhere near to optimal. [sent-15, score-0.154]
11 This causes confusion when people brag about computational performance on tiny datasets with only 10 5 examples I would also like to emphasize that this is intended as an open source project rather than merely a code drop, as occurred last time. [sent-17, score-1.038]
12 What I think this project has to offer researchers is an infrastructure for implementing fast online algorithms. [sent-18, score-0.521]
13 For people applying machine learning, there is some obvious value in getting very fast feedback in a batch setting, as well as having an algorithm that actually works in a real online setting. [sent-20, score-0.408]
14 As one example of the ability to reuse the code for other purposes, an effective general purpose online implementation of the Offset Tree is included. [sent-21, score-0.654]
15 I haven’t seen any other implementation of an algorithm for learning in the agnostic partial label setting, so this code may be of substantial interest for people encountering these sorts of problems. [sent-22, score-0.718]
16 Some bigger changes are: We dropped SEG for now, because of code complexity reasons. [sent-24, score-0.514]
17 Multicore parallelization proceeds in a different fashion—parallelization over features instead of examples. [sent-25, score-0.208]
18 Note that all parallelization of the core algorithm is meaningless unless you use the -q flag, because otherwise you are i/o bound. [sent-27, score-0.366]
19 The code is more deeply threaded, with a separate thread for parsing. [sent-28, score-0.447]
20 I have substantial confidence that this code can do interesting and useful things, but improving it is a constant and ongoing process. [sent-31, score-0.529]
wordName wordTfidf (topN-words)
[('code', 0.364), ('git', 0.281), ('parallelization', 0.208), ('project', 0.188), ('intended', 0.167), ('fast', 0.154), ('surrounding', 0.146), ('file', 0.139), ('release', 0.115), ('sparse', 0.112), ('implementation', 0.108), ('version', 0.108), ('online', 0.106), ('substantial', 0.088), ('datasets', 0.086), ('thread', 0.083), ('gaining', 0.083), ('clone', 0.083), ('multicore', 0.083), ('causes', 0.083), ('buffering', 0.083), ('rewrite', 0.083), ('downloaded', 0.083), ('encountering', 0.083), ('meaningless', 0.083), ('reports', 0.083), ('terafeature', 0.083), ('contestants', 0.077), ('nowhere', 0.077), ('threaded', 0.077), ('tiny', 0.077), ('flavors', 0.077), ('ongoing', 0.077), ('convenient', 0.077), ('perception', 0.077), ('datapoints', 0.077), ('admittedly', 0.077), ('bigger', 0.077), ('effective', 0.076), ('algorithm', 0.075), ('real', 0.073), ('infrastructure', 0.073), ('dropped', 0.073), ('stuff', 0.073), ('complaint', 0.073), ('emphasize', 0.073), ('flag', 0.073), ('native', 0.073), ('greatest', 0.073), ('deals', 0.073)]
simIndex simValue blogId blogTitle
same-blog 1 1.0000006 365 hunch net-2009-07-31-Vowpal Wabbit Open Source Project
Introduction: Today brings a new release of the Vowpal Wabbit fast online learning software. This time, unlike the previous release, the project itself is going open source, developing via github . For example, the lastest and greatest can be downloaded via: git clone git://github.com/JohnLangford/vowpal_wabbit.git If you aren’t familiar with git , it’s a distributed version control system which supports quick and easy branching, as well as reconciliation. This version of the code is confirmed to compile without complaint on at least some flavors of OSX as well as Linux boxes. As much of the point of this project is pushing the limits of fast and effective machine learning, let me mention a few datapoints from my experience. The program can effectively scale up to batch-style training on sparse terafeature (i.e. 10 12 sparse feature) size datasets. The limiting factor is typically i/o. I started using the the real datasets from the large-scale learning workshop as a conve
2 0.22321345 281 hunch net-2007-12-21-Vowpal Wabbit Code Release
Introduction: We are releasing the Vowpal Wabbit (Fast Online Learning) code as open source under a BSD (revised) license. This is a project at Yahoo! Research to build a useful large scale learning algorithm which Lihong Li , Alex Strehl , and I have been working on. To appreciate the meaning of “large”, it’s useful to define “small” and “medium”. A “small” supervised learning problem is one where a human could use a labeled dataset and come up with a reasonable predictor. A “medium” supervised learning problem dataset fits into the RAM of a modern desktop computer. A “large” supervised learning problem is one which does not fit into the RAM of a normal machine. VW tackles large scale learning problems by this definition of large. I’m not aware of any other open source Machine Learning tools which can handle this scale (although they may exist). A few close ones are: IBM’s Parallel Machine Learning Toolbox isn’t quite open source . The approach used by this toolbox is essenti
3 0.20417486 441 hunch net-2011-08-15-Vowpal Wabbit 6.0
Introduction: I just released Vowpal Wabbit 6.0 . Since the last version: VW is now 2-3 orders of magnitude faster at linear learning, primarily thanks to Alekh . Given the baseline, this is loads of fun, allowing us to easily deal with terafeature datasets, and dwarfing the scale of any other open source projects. The core improvement here comes from effective parallelization over kilonode clusters (either Hadoop or not). This code is highly scalable, so it even helps with clusters of size 2 (and doesn’t hurt for clusters of size 1). The core allreduce technique appears widely and easily reused—we’ve already used it to parallelize Conjugate Gradient, LBFGS, and two variants of online learning. We’ll be documenting how to do this more thoroughly, but for now “README_cluster” and associated scripts should provide a good starting point. The new LBFGS code from Miro seems to commonly dominate the existing conjugate gradient code in time/quality tradeoffs. The new matrix factoriz
4 0.15571758 426 hunch net-2011-03-19-The Ideal Large Scale Learning Class
Introduction: At NIPS, Andrew Ng asked me what should be in a large scale learning class. After some discussion with him and Nando and mulling it over a bit, these are the topics that I think should be covered. There are many different kinds of scaling. Scaling in examples This is the most basic kind of scaling. Online Gradient Descent This is an old algorithm—I’m not sure if anyone can be credited with it in particular. Perhaps the Perceptron is a good precursor, but substantial improvements come from the notion of a loss function of which squared loss , logistic loss , Hinge Loss, and Quantile Loss are all worth covering. It’s important to cover the semantics of these loss functions as well. Vowpal Wabbit is a reasonably fast codebase implementing these. Second Order Gradient Descent methods For some problems, methods taking into account second derivative information can be more effective. I’ve seen preconditioned conjugate gradient work well, for which Jonath
5 0.15375407 262 hunch net-2007-09-16-Optimizing Machine Learning Programs
Introduction: Machine learning is often computationally bounded which implies that the ability to write fast code becomes important if you ever want to implement a machine learning algorithm. Basic tactical optimizations are covered well elsewhere , but I haven’t seen a reasonable guide to higher level optimizations, which are the most important in my experience. Here are some of the higher level optimizations I’ve often found useful. Algorithmic Improvement First . This is Hard, but it is the most important consideration, and typically yields the most benefits. Good optimizations here are publishable. In the context of machine learning, you should be familiar with the arguments for online vs. batch learning. Choice of Language . There are many arguments about the choice of language . Sometimes you don’t have a choice when interfacing with other people. Personally, I favor C/C++ when I want to write fast code. This (admittedly) makes me a slower programmer than when using higher lev
6 0.14475438 451 hunch net-2011-12-13-Vowpal Wabbit version 6.1 & the NIPS tutorial
7 0.13055797 419 hunch net-2010-12-04-Vowpal Wabbit, version 5.0, and the second heresy
8 0.12181456 393 hunch net-2010-04-14-MLcomp: a website for objectively comparing ML algorithms
9 0.11868 371 hunch net-2009-09-21-Netflix finishes (and starts)
10 0.11787633 311 hunch net-2008-07-26-Compositional Machine Learning Algorithm Design
11 0.11712749 346 hunch net-2009-03-18-Parallel ML primitives
12 0.11098382 492 hunch net-2013-12-01-NIPS tutorials and Vowpal Wabbit 7.4
13 0.10968354 267 hunch net-2007-10-17-Online as the new adjective
14 0.10673728 251 hunch net-2007-06-24-Interesting Papers at ICML 2007
15 0.10534634 471 hunch net-2012-08-24-Patterns for research in machine learning
16 0.10089592 435 hunch net-2011-05-16-Research Directions for Machine Learning and Algorithms
17 0.10031357 428 hunch net-2011-03-27-Vowpal Wabbit, v5.1
18 0.099609576 42 hunch net-2005-03-17-Going all the Way, Sometimes
19 0.099118911 286 hunch net-2008-01-25-Turing’s Club for Machine Learning
20 0.098848164 420 hunch net-2010-12-26-NIPS 2010
topicId topicWeight
[(0, 0.22), (1, 0.058), (2, -0.08), (3, -0.002), (4, 0.06), (5, 0.068), (6, -0.162), (7, -0.064), (8, -0.123), (9, 0.156), (10, -0.084), (11, -0.014), (12, 0.068), (13, -0.04), (14, -0.002), (15, -0.1), (16, -0.011), (17, -0.026), (18, 0.008), (19, -0.007), (20, 0.026), (21, 0.07), (22, 0.006), (23, -0.018), (24, -0.044), (25, -0.001), (26, -0.134), (27, -0.002), (28, -0.053), (29, 0.006), (30, 0.001), (31, 0.08), (32, 0.068), (33, 0.014), (34, -0.004), (35, -0.01), (36, -0.021), (37, -0.004), (38, 0.052), (39, -0.022), (40, 0.049), (41, 0.063), (42, 0.005), (43, -0.034), (44, 0.097), (45, 0.026), (46, 0.018), (47, 0.064), (48, -0.023), (49, -0.036)]
simIndex simValue blogId blogTitle
same-blog 1 0.96053052 365 hunch net-2009-07-31-Vowpal Wabbit Open Source Project
Introduction: Today brings a new release of the Vowpal Wabbit fast online learning software. This time, unlike the previous release, the project itself is going open source, developing via github . For example, the lastest and greatest can be downloaded via: git clone git://github.com/JohnLangford/vowpal_wabbit.git If you aren’t familiar with git , it’s a distributed version control system which supports quick and easy branching, as well as reconciliation. This version of the code is confirmed to compile without complaint on at least some flavors of OSX as well as Linux boxes. As much of the point of this project is pushing the limits of fast and effective machine learning, let me mention a few datapoints from my experience. The program can effectively scale up to batch-style training on sparse terafeature (i.e. 10 12 sparse feature) size datasets. The limiting factor is typically i/o. I started using the the real datasets from the large-scale learning workshop as a conve
2 0.91020131 441 hunch net-2011-08-15-Vowpal Wabbit 6.0
Introduction: I just released Vowpal Wabbit 6.0 . Since the last version: VW is now 2-3 orders of magnitude faster at linear learning, primarily thanks to Alekh . Given the baseline, this is loads of fun, allowing us to easily deal with terafeature datasets, and dwarfing the scale of any other open source projects. The core improvement here comes from effective parallelization over kilonode clusters (either Hadoop or not). This code is highly scalable, so it even helps with clusters of size 2 (and doesn’t hurt for clusters of size 1). The core allreduce technique appears widely and easily reused—we’ve already used it to parallelize Conjugate Gradient, LBFGS, and two variants of online learning. We’ll be documenting how to do this more thoroughly, but for now “README_cluster” and associated scripts should provide a good starting point. The new LBFGS code from Miro seems to commonly dominate the existing conjugate gradient code in time/quality tradeoffs. The new matrix factoriz
3 0.75310624 419 hunch net-2010-12-04-Vowpal Wabbit, version 5.0, and the second heresy
Introduction: I’ve released version 5.0 of the Vowpal Wabbit online learning software. The major number has changed since the last release because I regard all earlier versions as obsolete—there are several new algorithms & features including substantial changes and upgrades to the default learning algorithm. The biggest changes are new algorithms: Nikos and I improved the default algorithm. The basic update rule still uses gradient descent, but the size of the update is carefully controlled so that it’s impossible to overrun the label. In addition, the normalization has changed. Computationally, these changes are virtually free and yield better results, sometimes much better. Less careful updates can be reenabled with –loss_function classic, although results are still not identical to previous due to normalization changes. Nikos also implemented the per-feature learning rates as per these two papers . Often, this works better than the default algorithm. It isn’t the defa
4 0.7513895 436 hunch net-2011-06-22-Ultra LDA
Introduction: Shravan and Alex ‘s LDA code is released . On a single machine, I’m not sure how it currently compares to the online LDA in VW , but the ability to effectively scale across very many machines is surely interesting.
5 0.74470943 281 hunch net-2007-12-21-Vowpal Wabbit Code Release
Introduction: We are releasing the Vowpal Wabbit (Fast Online Learning) code as open source under a BSD (revised) license. This is a project at Yahoo! Research to build a useful large scale learning algorithm which Lihong Li , Alex Strehl , and I have been working on. To appreciate the meaning of “large”, it’s useful to define “small” and “medium”. A “small” supervised learning problem is one where a human could use a labeled dataset and come up with a reasonable predictor. A “medium” supervised learning problem dataset fits into the RAM of a modern desktop computer. A “large” supervised learning problem is one which does not fit into the RAM of a normal machine. VW tackles large scale learning problems by this definition of large. I’m not aware of any other open source Machine Learning tools which can handle this scale (although they may exist). A few close ones are: IBM’s Parallel Machine Learning Toolbox isn’t quite open source . The approach used by this toolbox is essenti
6 0.67305565 451 hunch net-2011-12-13-Vowpal Wabbit version 6.1 & the NIPS tutorial
7 0.6596064 450 hunch net-2011-12-02-Hadoop AllReduce and Terascale Learning
8 0.6428777 300 hunch net-2008-04-30-Concerns about the Large Scale Learning Challenge
9 0.64198089 262 hunch net-2007-09-16-Optimizing Machine Learning Programs
10 0.62855333 346 hunch net-2009-03-18-Parallel ML primitives
11 0.62212473 428 hunch net-2011-03-27-Vowpal Wabbit, v5.1
12 0.61338276 393 hunch net-2010-04-14-MLcomp: a website for objectively comparing ML algorithms
13 0.57728606 426 hunch net-2011-03-19-The Ideal Large Scale Learning Class
14 0.57416147 381 hunch net-2009-12-07-Vowpal Wabbit version 4.0, and a NIPS heresy
15 0.57146245 492 hunch net-2013-12-01-NIPS tutorials and Vowpal Wabbit 7.4
16 0.55405843 471 hunch net-2012-08-24-Patterns for research in machine learning
17 0.54480678 298 hunch net-2008-04-26-Eliminating the Birthday Paradox for Universal Features
18 0.536484 473 hunch net-2012-09-29-Vowpal Wabbit, version 7.0
19 0.52717781 286 hunch net-2008-01-25-Turing’s Club for Machine Learning
20 0.49429321 128 hunch net-2005-11-05-The design of a computing cluster
topicId topicWeight
[(27, 0.159), (38, 0.053), (49, 0.346), (51, 0.016), (53, 0.077), (55, 0.06), (64, 0.015), (94, 0.156), (95, 0.047)]
simIndex simValue blogId blogTitle
1 0.92774224 224 hunch net-2006-12-12-Interesting Papers at NIPS 2006
Introduction: Here are some papers that I found surprisingly interesting. Yoshua Bengio , Pascal Lamblin, Dan Popovici, Hugo Larochelle, Greedy Layer-wise Training of Deep Networks . Empirically investigates some of the design choices behind deep belief networks. Long Zhu , Yuanhao Chen, Alan Yuille Unsupervised Learning of a Probabilistic Grammar for Object Detection and Parsing. An unsupervised method for detecting objects using simple feature filters that works remarkably well on the (supervised) caltech-101 dataset . Shai Ben-David , John Blitzer , Koby Crammer , and Fernando Pereira , Analysis of Representations for Domain Adaptation . This is the first analysis I’ve seen of learning with respect to samples drawn differently from the evaluation distribution which depends on reasonable measurable quantities. All of these papers turn out to have a common theme—the power of unlabeled data to do generically useful things.
2 0.90538055 122 hunch net-2005-10-13-Site tweak
Introduction: Several people have had difficulty with comments which seem to have an allowed language significantly poorer than posts. The set of allowed html tags has been increased and the markdown filter has been put in place to try to make commenting easier. I’ll put some examples into the comments of this post.
same-blog 3 0.88386834 365 hunch net-2009-07-31-Vowpal Wabbit Open Source Project
Introduction: Today brings a new release of the Vowpal Wabbit fast online learning software. This time, unlike the previous release, the project itself is going open source, developing via github . For example, the lastest and greatest can be downloaded via: git clone git://github.com/JohnLangford/vowpal_wabbit.git If you aren’t familiar with git , it’s a distributed version control system which supports quick and easy branching, as well as reconciliation. This version of the code is confirmed to compile without complaint on at least some flavors of OSX as well as Linux boxes. As much of the point of this project is pushing the limits of fast and effective machine learning, let me mention a few datapoints from my experience. The program can effectively scale up to batch-style training on sparse terafeature (i.e. 10 12 sparse feature) size datasets. The limiting factor is typically i/o. I started using the the real datasets from the large-scale learning workshop as a conve
4 0.87534368 338 hunch net-2009-01-23-An Active Learning Survey
Introduction: Burr Settles wrote a fairly comprehensive survey of active learning . He intends to maintain and update the survey, so send him any suggestions you have.
5 0.83046776 37 hunch net-2005-03-08-Fast Physics for Learning
Introduction: While everyone is silently working on ICML submissions, I found this discussion about a fast physics simulator chip interesting from a learning viewpoint. In many cases, learning attempts to predict the outcome of physical processes. Access to a fast simulator for these processes might be quite helpful in predicting the outcome. Bayesian learning in particular may directly benefit while many other algorithms (like support vector machines) might have their speed greatly increased. The biggest drawback is that writing software for these odd architectures is always difficult and time consuming, but a several-orders-of-magnitude speedup might make that worthwhile.
6 0.77934223 348 hunch net-2009-04-02-Asymmophobia
7 0.7499131 23 hunch net-2005-02-19-Loss Functions for Discriminative Training of Energy-Based Models
8 0.68068266 359 hunch net-2009-06-03-Functionally defined Nonlinear Dynamic Models
9 0.60190761 438 hunch net-2011-07-11-Interesting Neural Network Papers at ICML 2011
10 0.58264297 426 hunch net-2011-03-19-The Ideal Large Scale Learning Class
11 0.57173997 286 hunch net-2008-01-25-Turing’s Club for Machine Learning
12 0.56925648 262 hunch net-2007-09-16-Optimizing Machine Learning Programs
13 0.56899714 221 hunch net-2006-12-04-Structural Problems in NIPS Decision Making
14 0.56885809 237 hunch net-2007-04-02-Contextual Scaling
15 0.56874782 276 hunch net-2007-12-10-Learning Track of International Planning Competition
16 0.56722856 366 hunch net-2009-08-03-Carbon in Computer Science Research
17 0.56671208 143 hunch net-2005-12-27-Automated Labeling
18 0.56290978 435 hunch net-2011-05-16-Research Directions for Machine Learning and Algorithms
19 0.56270128 136 hunch net-2005-12-07-Is the Google way the way for machine learning?
20 0.56151152 132 hunch net-2005-11-26-The Design of an Optimal Research Environment