hunch_net hunch_net-2011 hunch_net-2011-451 knowledge-graph by maker-knowledge-mining
Source: html
Introduction: I just made version 6.1 of Vowpal Wabbit . Relative to 6.0 , there are few new features, but many refinements. The cluster parallel learning code better supports multiple simultaneous runs, and other forms of parallelism have been mostly removed. This incidentally significantly simplifies the learning core. The online learning algorithms are more general, with support for l 1 (via a truncated gradient variant) and l 2 regularization, and a generalized form of variable metric learning. There is a solid persistent server mode which can train online, as well as serve answers to many simultaneous queries, either in text or binary. This should be a very good release if you are just getting started, as we’ve made it compile more automatically out of the box, have several new examples and updated documentation. As per tradition , we’re planning to do a tutorial at NIPS during the break at the parallel learning workshop at 2pm Spanish time Friday. I’ll cover the
sentIndex sentText sentNum sentScore
1 The cluster parallel learning code better supports multiple simultaneous runs, and other forms of parallelism have been mostly removed. [sent-5, score-1.081]
2 The online learning algorithms are more general, with support for l 1 (via a truncated gradient variant) and l 2 regularization, and a generalized form of variable metric learning. [sent-7, score-0.564]
3 There is a solid persistent server mode which can train online, as well as serve answers to many simultaneous queries, either in text or binary. [sent-8, score-1.112]
4 This should be a very good release if you are just getting started, as we’ve made it compile more automatically out of the box, have several new examples and updated documentation. [sent-9, score-0.426]
5 As per tradition , we’re planning to do a tutorial at NIPS during the break at the parallel learning workshop at 2pm Spanish time Friday. [sent-10, score-0.473]
6 I’ll cover the basics, leaving the fun stuff for others. [sent-11, score-0.571]
7 Miro will cover the L-BFGS implementation, which he created from scratch. [sent-12, score-0.25]
8 We have found this works quite well amongst batch learning algorithms. [sent-13, score-0.091]
9 Alekh will cover how to do cluster parallel learning . [sent-14, score-0.785]
10 If you have access to a large cluster, VW is orders of magnitude faster than any other public learning system accomplishing linear prediction. [sent-15, score-0.345]
11 And if you are as impatient as I am, it is a real pleasure when the computers can keep up with you. [sent-16, score-0.091]
12 This will be recorded, so it will hopefully be available for viewing online before too long. [sent-17, score-0.25]
wordName wordTfidf (topN-words)
[('cluster', 0.277), ('parallel', 0.258), ('cover', 0.25), ('simultaneous', 0.229), ('serve', 0.143), ('simplifies', 0.143), ('truncated', 0.143), ('online', 0.136), ('persistent', 0.132), ('accomplishing', 0.132), ('miro', 0.132), ('stuff', 0.125), ('box', 0.119), ('orders', 0.119), ('compile', 0.119), ('alekh', 0.119), ('tradition', 0.114), ('viewing', 0.114), ('supports', 0.114), ('server', 0.114), ('recorded', 0.114), ('parallelism', 0.114), ('solid', 0.114), ('queries', 0.11), ('incidentally', 0.107), ('updated', 0.107), ('runs', 0.107), ('leaving', 0.107), ('generalized', 0.104), ('regularization', 0.101), ('break', 0.101), ('made', 0.101), ('mode', 0.099), ('release', 0.099), ('variant', 0.096), ('train', 0.096), ('text', 0.096), ('vw', 0.096), ('re', 0.094), ('magnitude', 0.094), ('soon', 0.094), ('metric', 0.092), ('implementation', 0.092), ('computers', 0.091), ('batch', 0.091), ('forms', 0.089), ('answers', 0.089), ('variable', 0.089), ('fun', 0.089), ('vowpal', 0.087)]
simIndex simValue blogId blogTitle
same-blog 1 0.99999994 451 hunch net-2011-12-13-Vowpal Wabbit version 6.1 & the NIPS tutorial
Introduction: I just made version 6.1 of Vowpal Wabbit . Relative to 6.0 , there are few new features, but many refinements. The cluster parallel learning code better supports multiple simultaneous runs, and other forms of parallelism have been mostly removed. This incidentally significantly simplifies the learning core. The online learning algorithms are more general, with support for l 1 (via a truncated gradient variant) and l 2 regularization, and a generalized form of variable metric learning. There is a solid persistent server mode which can train online, as well as serve answers to many simultaneous queries, either in text or binary. This should be a very good release if you are just getting started, as we’ve made it compile more automatically out of the box, have several new examples and updated documentation. As per tradition , we’re planning to do a tutorial at NIPS during the break at the parallel learning workshop at 2pm Spanish time Friday. I’ll cover the
2 0.20057952 381 hunch net-2009-12-07-Vowpal Wabbit version 4.0, and a NIPS heresy
Introduction: I’m releasing version 4.0 ( tarball ) of Vowpal Wabbit . The biggest change (by far) in this release is experimental support for cluster parallelism, with notable help from Daniel Hsu . I also took advantage of the major version number to introduce some incompatible changes, including switching to murmurhash 2 , and other alterations to cachefiles. You’ll need to delete and regenerate them. In addition, the precise specification for a “tag” (i.e. string that can be used to identify an example) changed—you can’t have a space between the tag and the ‘|’ at the beginning of the feature namespace. And, of course, we made it faster. For the future, I put up my todo list outlining the major future improvements I want to see in the code. I’m planning to discuss the current mechanism and results of the cluster parallel implementation at the large scale machine learning workshop at NIPS later this week. Several people have asked me to do a tutorial/walkthrough of VW, wh
3 0.19241792 404 hunch net-2010-08-20-The Workshop on Cores, Clusters, and Clouds
Introduction: Alekh , John , Ofer , and I are organizing a workshop at NIPS this year on learning in parallel and distributed environments. The general interest level in parallel learning seems to be growing rapidly, so I expect quite a bit of attendance. Please join us if you are parallel-interested. And, if you are working in the area of parallel learning, please consider submitting an abstract due Oct. 17 for presentation at the workshop.
4 0.18618935 346 hunch net-2009-03-18-Parallel ML primitives
Introduction: Previously, we discussed parallel machine learning a bit. As parallel ML is rather difficult, I’d like to describe my thinking at the moment, and ask for advice from the rest of the world. This is particularly relevant right now, as I’m attending a workshop tomorrow on parallel ML. Parallelizing slow algorithms seems uncompelling. Parallelizing many algorithms also seems uncompelling, because the effort required to parallelize is substantial. This leaves the question: Which one fast algorithm is the best to parallelize? What is a substantially different second? One compellingly fast simple algorithm is online gradient descent on a linear representation. This is the core of Leon’s sgd code and Vowpal Wabbit . Antoine Bordes showed a variant was competitive in the large scale learning challenge . It’s also a decades old primitive which has been reused in many algorithms, and continues to be reused. It also applies to online learning rather than just online optimiz
5 0.16459532 441 hunch net-2011-08-15-Vowpal Wabbit 6.0
Introduction: I just released Vowpal Wabbit 6.0 . Since the last version: VW is now 2-3 orders of magnitude faster at linear learning, primarily thanks to Alekh . Given the baseline, this is loads of fun, allowing us to easily deal with terafeature datasets, and dwarfing the scale of any other open source projects. The core improvement here comes from effective parallelization over kilonode clusters (either Hadoop or not). This code is highly scalable, so it even helps with clusters of size 2 (and doesn’t hurt for clusters of size 1). The core allreduce technique appears widely and easily reused—we’ve already used it to parallelize Conjugate Gradient, LBFGS, and two variants of online learning. We’ll be documenting how to do this more thoroughly, but for now “README_cluster” and associated scripts should provide a good starting point. The new LBFGS code from Miro seems to commonly dominate the existing conjugate gradient code in time/quality tradeoffs. The new matrix factoriz
6 0.15217137 419 hunch net-2010-12-04-Vowpal Wabbit, version 5.0, and the second heresy
7 0.14475438 365 hunch net-2009-07-31-Vowpal Wabbit Open Source Project
8 0.14185317 229 hunch net-2007-01-26-Parallel Machine Learning Problems
9 0.1390802 267 hunch net-2007-10-17-Online as the new adjective
10 0.13184717 426 hunch net-2011-03-19-The Ideal Large Scale Learning Class
11 0.12913397 492 hunch net-2013-12-01-NIPS tutorials and Vowpal Wabbit 7.4
12 0.12516487 281 hunch net-2007-12-21-Vowpal Wabbit Code Release
13 0.12480675 450 hunch net-2011-12-02-Hadoop AllReduce and Terascale Learning
14 0.11947295 428 hunch net-2011-03-27-Vowpal Wabbit, v5.1
15 0.10248989 300 hunch net-2008-04-30-Concerns about the Large Scale Learning Challenge
16 0.102119 473 hunch net-2012-09-29-Vowpal Wabbit, version 7.0
17 0.10156055 490 hunch net-2013-11-09-Graduates and Postdocs
18 0.10123573 287 hunch net-2008-01-28-Sufficient Computation
19 0.10071308 442 hunch net-2011-08-20-The Large Scale Learning Survey Tutorial
20 0.099509269 366 hunch net-2009-08-03-Carbon in Computer Science Research
topicId topicWeight
[(0, 0.173), (1, 0.025), (2, -0.119), (3, -0.021), (4, 0.107), (5, 0.095), (6, -0.181), (7, -0.108), (8, -0.136), (9, 0.179), (10, -0.113), (11, -0.07), (12, 0.104), (13, -0.003), (14, 0.021), (15, -0.132), (16, -0.047), (17, 0.009), (18, -0.011), (19, -0.092), (20, -0.016), (21, 0.029), (22, -0.046), (23, 0.073), (24, 0.004), (25, 0.037), (26, -0.021), (27, -0.052), (28, 0.025), (29, 0.054), (30, 0.078), (31, -0.047), (32, -0.021), (33, 0.008), (34, 0.009), (35, 0.01), (36, 0.062), (37, 0.077), (38, -0.099), (39, 0.01), (40, -0.045), (41, 0.021), (42, -0.054), (43, 0.049), (44, 0.015), (45, 0.042), (46, 0.068), (47, 0.017), (48, 0.054), (49, 0.007)]
simIndex simValue blogId blogTitle
same-blog 1 0.94117558 451 hunch net-2011-12-13-Vowpal Wabbit version 6.1 & the NIPS tutorial
Introduction: I just made version 6.1 of Vowpal Wabbit . Relative to 6.0 , there are few new features, but many refinements. The cluster parallel learning code better supports multiple simultaneous runs, and other forms of parallelism have been mostly removed. This incidentally significantly simplifies the learning core. The online learning algorithms are more general, with support for l 1 (via a truncated gradient variant) and l 2 regularization, and a generalized form of variable metric learning. There is a solid persistent server mode which can train online, as well as serve answers to many simultaneous queries, either in text or binary. This should be a very good release if you are just getting started, as we’ve made it compile more automatically out of the box, have several new examples and updated documentation. As per tradition , we’re planning to do a tutorial at NIPS during the break at the parallel learning workshop at 2pm Spanish time Friday. I’ll cover the
2 0.80255067 381 hunch net-2009-12-07-Vowpal Wabbit version 4.0, and a NIPS heresy
Introduction: I’m releasing version 4.0 ( tarball ) of Vowpal Wabbit . The biggest change (by far) in this release is experimental support for cluster parallelism, with notable help from Daniel Hsu . I also took advantage of the major version number to introduce some incompatible changes, including switching to murmurhash 2 , and other alterations to cachefiles. You’ll need to delete and regenerate them. In addition, the precise specification for a “tag” (i.e. string that can be used to identify an example) changed—you can’t have a space between the tag and the ‘|’ at the beginning of the feature namespace. And, of course, we made it faster. For the future, I put up my todo list outlining the major future improvements I want to see in the code. I’m planning to discuss the current mechanism and results of the cluster parallel implementation at the large scale machine learning workshop at NIPS later this week. Several people have asked me to do a tutorial/walkthrough of VW, wh
3 0.73387796 419 hunch net-2010-12-04-Vowpal Wabbit, version 5.0, and the second heresy
Introduction: I’ve released version 5.0 of the Vowpal Wabbit online learning software. The major number has changed since the last release because I regard all earlier versions as obsolete—there are several new algorithms & features including substantial changes and upgrades to the default learning algorithm. The biggest changes are new algorithms: Nikos and I improved the default algorithm. The basic update rule still uses gradient descent, but the size of the update is carefully controlled so that it’s impossible to overrun the label. In addition, the normalization has changed. Computationally, these changes are virtually free and yield better results, sometimes much better. Less careful updates can be reenabled with –loss_function classic, although results are still not identical to previous due to normalization changes. Nikos also implemented the per-feature learning rates as per these two papers . Often, this works better than the default algorithm. It isn’t the defa
4 0.71525621 441 hunch net-2011-08-15-Vowpal Wabbit 6.0
Introduction: I just released Vowpal Wabbit 6.0 . Since the last version: VW is now 2-3 orders of magnitude faster at linear learning, primarily thanks to Alekh . Given the baseline, this is loads of fun, allowing us to easily deal with terafeature datasets, and dwarfing the scale of any other open source projects. The core improvement here comes from effective parallelization over kilonode clusters (either Hadoop or not). This code is highly scalable, so it even helps with clusters of size 2 (and doesn’t hurt for clusters of size 1). The core allreduce technique appears widely and easily reused—we’ve already used it to parallelize Conjugate Gradient, LBFGS, and two variants of online learning. We’ll be documenting how to do this more thoroughly, but for now “README_cluster” and associated scripts should provide a good starting point. The new LBFGS code from Miro seems to commonly dominate the existing conjugate gradient code in time/quality tradeoffs. The new matrix factoriz
5 0.70840377 346 hunch net-2009-03-18-Parallel ML primitives
Introduction: Previously, we discussed parallel machine learning a bit. As parallel ML is rather difficult, I’d like to describe my thinking at the moment, and ask for advice from the rest of the world. This is particularly relevant right now, as I’m attending a workshop tomorrow on parallel ML. Parallelizing slow algorithms seems uncompelling. Parallelizing many algorithms also seems uncompelling, because the effort required to parallelize is substantial. This leaves the question: Which one fast algorithm is the best to parallelize? What is a substantially different second? One compellingly fast simple algorithm is online gradient descent on a linear representation. This is the core of Leon’s sgd code and Vowpal Wabbit . Antoine Bordes showed a variant was competitive in the large scale learning challenge . It’s also a decades old primitive which has been reused in many algorithms, and continues to be reused. It also applies to online learning rather than just online optimiz
6 0.63032639 365 hunch net-2009-07-31-Vowpal Wabbit Open Source Project
7 0.62612927 450 hunch net-2011-12-02-Hadoop AllReduce and Terascale Learning
8 0.61709636 281 hunch net-2007-12-21-Vowpal Wabbit Code Release
9 0.6034354 492 hunch net-2013-12-01-NIPS tutorials and Vowpal Wabbit 7.4
10 0.58686709 436 hunch net-2011-06-22-Ultra LDA
11 0.57164234 442 hunch net-2011-08-20-The Large Scale Learning Survey Tutorial
12 0.57021308 404 hunch net-2010-08-20-The Workshop on Cores, Clusters, and Clouds
13 0.56654668 229 hunch net-2007-01-26-Parallel Machine Learning Problems
14 0.54010046 473 hunch net-2012-09-29-Vowpal Wabbit, version 7.0
15 0.53460926 300 hunch net-2008-04-30-Concerns about the Large Scale Learning Challenge
16 0.49607995 267 hunch net-2007-10-17-Online as the new adjective
17 0.48262531 136 hunch net-2005-12-07-Is the Google way the way for machine learning?
18 0.47047278 426 hunch net-2011-03-19-The Ideal Large Scale Learning Class
19 0.45419633 128 hunch net-2005-11-05-The design of a computing cluster
20 0.45366246 490 hunch net-2013-11-09-Graduates and Postdocs
topicId topicWeight
[(10, 0.035), (27, 0.215), (53, 0.047), (55, 0.069), (94, 0.155), (95, 0.042), (99, 0.343)]
simIndex simValue blogId blogTitle
1 0.92311817 64 hunch net-2005-04-28-Science Fiction and Research
Introduction: A big part of doing research is imagining how things could be different, and then trying to figure out how to get there. A big part of science fiction is imagining how things could be different, and then working through the implications. Because of the similarity here, reading science fiction can sometimes be helpful in understanding and doing research. (And, hey, it’s fun.) Here’s some list of science fiction books I enjoyed which seem particularly relevant to computer science and (sometimes) learning systems: Vernor Vinge, “True Names”, “A Fire Upon the Deep” Marc Stiegler, “David’s Sling”, “Earthweb” Charles Stross, “Singularity Sky” Greg Egan, “Diaspora” Joe Haldeman, “Forever Peace” (There are surely many others.) Incidentally, the nature of science fiction itself has changed. Decades ago, science fiction projected great increases in the power humans control (example: E.E. Smith Lensman series). That didn’t really happen in the last 50 years. Inste
same-blog 2 0.87951803 451 hunch net-2011-12-13-Vowpal Wabbit version 6.1 & the NIPS tutorial
Introduction: I just made version 6.1 of Vowpal Wabbit . Relative to 6.0 , there are few new features, but many refinements. The cluster parallel learning code better supports multiple simultaneous runs, and other forms of parallelism have been mostly removed. This incidentally significantly simplifies the learning core. The online learning algorithms are more general, with support for l 1 (via a truncated gradient variant) and l 2 regularization, and a generalized form of variable metric learning. There is a solid persistent server mode which can train online, as well as serve answers to many simultaneous queries, either in text or binary. This should be a very good release if you are just getting started, as we’ve made it compile more automatically out of the box, have several new examples and updated documentation. As per tradition , we’re planning to do a tutorial at NIPS during the break at the parallel learning workshop at 2pm Spanish time Friday. I’ll cover the
3 0.62036556 286 hunch net-2008-01-25-Turing’s Club for Machine Learning
Introduction: Many people in Machine Learning don’t fully understand the impact of computation, as demonstrated by a lack of big-O analysis of new learning algorithms. This is important—some current active research programs are fundamentally flawed w.r.t. computation, and other research programs are directly motivated by it. When considering a learning algorithm, I think about the following questions: How does the learning algorithm scale with the number of examples m ? Any algorithm using all of the data is at least O(m) , but in many cases this is O(m 2 ) (naive nearest neighbor for self-prediction) or unknown (k-means or many other optimization algorithms). The unknown case is very common, and it can mean (for example) that the algorithm isn’t convergent or simply that the amount of computation isn’t controlled. The above question can also be asked for test cases. In some applications, test-time performance is of great importance. How does the algorithm scale with the number of
4 0.61739564 221 hunch net-2006-12-04-Structural Problems in NIPS Decision Making
Introduction: This is a very difficult post to write, because it is about a perenially touchy subject. Nevertheless, it is an important one which needs to be thought about carefully. There are a few things which should be understood: The system is changing and responsive. We-the-authors are we-the-reviewers, we-the-PC, and even we-the-NIPS-board. NIPS has implemented ‘secondary program chairs’, ‘author response’, and ‘double blind reviewing’ in the last few years to help with the decision process, and more changes may happen in the future. Agreement creates a perception of correctness. When any PC meets and makes a group decision about a paper, there is a strong tendency for the reinforcement inherent in a group decision to create the perception of correctness. For the many people who have been on the NIPS PC it’s reasonable to entertain a healthy skepticism in the face of this reinforcing certainty. This post is about structural problems. What problems arise because of the structure
5 0.61594993 229 hunch net-2007-01-26-Parallel Machine Learning Problems
Introduction: Parallel machine learning is a subject rarely addressed at machine learning conferences. Nevertheless, it seems likely to increase in importance because: Data set sizes appear to be growing substantially faster than computation. Essentially, this happens because more and more sensors of various sorts are being hooked up to the internet. Serial speedups of processors seem are relatively stalled. The new trend is to make processors more powerful by making them multicore . Both AMD and Intel are making dual core designs standard, with plans for more parallelism in the future. IBM’s Cell processor has (essentially) 9 cores. Modern graphics chips can have an order of magnitude more separate execution units. The meaning of ‘core’ varies a bit from processor to processor, but the overall trend seems quite clear. So, how do we parallelize machine learning algorithms? The simplest and most common technique is to simply run the same learning algorithm with di
6 0.61067313 276 hunch net-2007-12-10-Learning Track of International Planning Competition
7 0.60981059 366 hunch net-2009-08-03-Carbon in Computer Science Research
8 0.60756916 95 hunch net-2005-07-14-What Learning Theory might do
9 0.6074385 450 hunch net-2011-12-02-Hadoop AllReduce and Terascale Learning
10 0.60696089 136 hunch net-2005-12-07-Is the Google way the way for machine learning?
11 0.60274369 371 hunch net-2009-09-21-Netflix finishes (and starts)
12 0.60272956 253 hunch net-2007-07-06-Idempotent-capable Predictors
13 0.6020453 359 hunch net-2009-06-03-Functionally defined Nonlinear Dynamic Models
14 0.60084355 132 hunch net-2005-11-26-The Design of an Optimal Research Environment
15 0.59903592 120 hunch net-2005-10-10-Predictive Search is Coming
16 0.5989995 237 hunch net-2007-04-02-Contextual Scaling
17 0.59883815 435 hunch net-2011-05-16-Research Directions for Machine Learning and Algorithms
18 0.59767944 43 hunch net-2005-03-18-Binomial Weighting
19 0.5970096 109 hunch net-2005-09-08-Online Learning as the Mathematics of Accountability
20 0.59681618 360 hunch net-2009-06-15-In Active Learning, the question changes