hunch_net hunch_net-2005 hunch_net-2005-128 knowledge-graph by maker-knowledge-mining
Source: html
Introduction: This is about the design of a computing cluster from the viewpoint of applied machine learning using current technology. We just built a small one at TTI so this is some evidence of what is feasible and thoughts about the design choices. Architecture There are several architectural choices. AMD Athlon64 based system. This seems to have the cheapest bang/buck. Maximum RAM is typically 2-3GB. AMD Opteron based system. Opterons provide the additional capability to buy an SMP motherboard with two chips, and the motherboards often support 16GB of RAM. The RAM is also the more expensive error correcting type. Intel PIV or Xeon based system. The PIV and Xeon based systems are the intel analog of the above 2. Due to architectural design reasons, these chips tend to run a bit hotter and be a bit more expensive. Dual core chips. Both Intel and AMD have chips that actually have 2 processors embedded in them. In the end, we decided to go with option (2). Roughly speaking,
sentIndex sentText sentNum sentScore
1 This is about the design of a computing cluster from the viewpoint of applied machine learning using current technology. [sent-1, score-0.186]
2 Architecture There are several architectural choices. [sent-3, score-0.178]
3 The PIV and Xeon based systems are the intel analog of the above 2. [sent-11, score-0.346]
4 Due to architectural design reasons, these chips tend to run a bit hotter and be a bit more expensive. [sent-12, score-0.526]
5 Both Intel and AMD have chips that actually have 2 processors embedded in them. [sent-14, score-0.264]
6 The opteron systems were desirable over the Athlon64 systems because halving the number of nodes aides system setup and maintenance while preserving about the same (or slightly more) cost/cpu. [sent-17, score-0.719]
7 In the last year or so, CPU speeds have not increased signficantly, instead dual core chips have come out. [sent-19, score-0.427]
8 Network Gigabit ethernet is cheap, easy, and even built into the motherboard. [sent-21, score-0.229]
9 The disadvantage is that using more than 4GB of RAM is awkward and you lose out on some minor architectural speedups of 64bit mode. [sent-24, score-0.248]
10 64bit linux We ended up choosing 32bit linux simply for stability and ease-of-setup reasons. [sent-25, score-0.987]
11 The exact variant of linux we used was a matter of some initial exploration determined by Don Coleman (TTIs master of machines). [sent-27, score-0.607]
12 It is very plausible that we will want to switch to 64bit linux at some point in the future. [sent-28, score-0.465]
13 Programming paradigm There are several paradigms for how to use a parallel machine. [sent-29, score-0.229]
14 Turn the cluster into a large virtual machine via openMosix and then simply launch several processes. [sent-37, score-0.212]
15 This is the worst option performance-wise and the best option convenience-wise. [sent-38, score-0.238]
16 To use it, you simply start processes and the system takes care of distributing them across the cluster. [sent-39, score-0.339]
17 Ideally, each of the nodes would be rackmounted (for ease of maintenance) and, except for the “master node”, use ethernet boot on startup. [sent-44, score-0.429]
18 The rackmounting was relatively easy, but the combination of ethernet boot, openmosix, and linux was frustrating. [sent-45, score-0.623]
19 Instead Don ordered some very small hard drives for each node and simply installed linux on them. [sent-46, score-0.66]
20 Another minor surprise is that the opteron motherboard required a video card in order to boot. [sent-47, score-0.48]
wordName wordTfidf (topN-words)
[('linux', 0.465), ('opteron', 0.238), ('chips', 0.211), ('openmosix', 0.211), ('ram', 0.204), ('amd', 0.196), ('architectural', 0.178), ('ethernet', 0.158), ('intel', 0.139), ('system', 0.12), ('boot', 0.119), ('io', 0.119), ('motherboard', 0.119), ('piv', 0.119), ('xeon', 0.119), ('option', 0.119), ('tti', 0.106), ('maintenance', 0.106), ('cluster', 0.102), ('paradigms', 0.098), ('processes', 0.095), ('cpu', 0.092), ('dual', 0.088), ('programming', 0.087), ('systems', 0.085), ('nodes', 0.085), ('node', 0.085), ('design', 0.084), ('core', 0.075), ('variant', 0.071), ('built', 0.071), ('master', 0.071), ('minor', 0.07), ('end', 0.07), ('based', 0.069), ('use', 0.067), ('maximum', 0.067), ('parallel', 0.064), ('simply', 0.057), ('additional', 0.055), ('installed', 0.053), ('analog', 0.053), ('calibrating', 0.053), ('card', 0.053), ('cheaper', 0.053), ('eak', 0.053), ('embedded', 0.053), ('hotter', 0.053), ('launch', 0.053), ('speeds', 0.053)]
simIndex simValue blogId blogTitle
same-blog 1 1.0000002 128 hunch net-2005-11-05-The design of a computing cluster
Introduction: This is about the design of a computing cluster from the viewpoint of applied machine learning using current technology. We just built a small one at TTI so this is some evidence of what is feasible and thoughts about the design choices. Architecture There are several architectural choices. AMD Athlon64 based system. This seems to have the cheapest bang/buck. Maximum RAM is typically 2-3GB. AMD Opteron based system. Opterons provide the additional capability to buy an SMP motherboard with two chips, and the motherboards often support 16GB of RAM. The RAM is also the more expensive error correcting type. Intel PIV or Xeon based system. The PIV and Xeon based systems are the intel analog of the above 2. Due to architectural design reasons, these chips tend to run a bit hotter and be a bit more expensive. Dual core chips. Both Intel and AMD have chips that actually have 2 processors embedded in them. In the end, we decided to go with option (2). Roughly speaking,
2 0.17916211 229 hunch net-2007-01-26-Parallel Machine Learning Problems
Introduction: Parallel machine learning is a subject rarely addressed at machine learning conferences. Nevertheless, it seems likely to increase in importance because: Data set sizes appear to be growing substantially faster than computation. Essentially, this happens because more and more sensors of various sorts are being hooked up to the internet. Serial speedups of processors seem are relatively stalled. The new trend is to make processors more powerful by making them multicore . Both AMD and Intel are making dual core designs standard, with plans for more parallelism in the future. IBM’s Cell processor has (essentially) 9 cores. Modern graphics chips can have an order of magnitude more separate execution units. The meaning of ‘core’ varies a bit from processor to processor, but the overall trend seems quite clear. So, how do we parallelize machine learning algorithms? The simplest and most common technique is to simply run the same learning algorithm with di
3 0.11377785 450 hunch net-2011-12-02-Hadoop AllReduce and Terascale Learning
Introduction: Suppose you have a dataset with 2 terafeatures (we only count nonzero entries in a datamatrix), and want to learn a good linear predictor in a reasonable amount of time. How do you do it? As a learning theorist, the first thing you do is pray that this is too much data for the number of parameters—but that’s not the case, there are around 16 billion examples, 16 million parameters, and people really care about a high quality predictor, so subsampling is not a good strategy. Alekh visited us last summer, and we had a breakthrough (see here for details), coming up with the first learning algorithm I’ve seen that is provably faster than any future single machine learning algorithm. The proof of this is simple: We can output a optimal-up-to-precision linear predictor faster than the data can be streamed through the network interface of any single machine involved in the computation. It is necessary but not sufficient to have an effective communication infrastructure. It is ne
4 0.096811995 132 hunch net-2005-11-26-The Design of an Optimal Research Environment
Introduction: How do you create an optimal environment for research? Here are some essential ingredients that I see. Stability . University-based research is relatively good at this. On any particular day, researchers face choices in what they will work on. A very common tradeoff is between: easy small difficult big For researchers without stability, the ‘easy small’ option wins. This is often “ok”—a series of incremental improvements on the state of the art can add up to something very beneficial. However, it misses one of the big potentials of research: finding entirely new and better ways of doing things. Stability comes in many forms. The prototypical example is tenure at a university—a tenured professor is almost imposssible to fire which means that the professor has the freedom to consider far horizon activities. An iron-clad guarantee of a paycheck is not necessary—industrial research labs have succeeded well with research positions of indefinite duration. Atnt rese
5 0.093249388 300 hunch net-2008-04-30-Concerns about the Large Scale Learning Challenge
Introduction: The large scale learning challenge for ICML interests me a great deal, although I have concerns about the way it is structured. From the instructions page , several issues come up: Large Definition My personal definition of dataset size is: small A dataset is small if a human could look at the dataset and plausibly find a good solution. medium A dataset is mediumsize if it fits in the RAM of a reasonably priced computer. large A large dataset does not fit in the RAM of a reasonably priced computer. By this definition, all of the datasets are medium sized. This might sound like a pissing match over dataset size, but I believe it is more than that. The fundamental reason for these definitions is that they correspond to transitions in the sorts of approaches which are feasible. From small to medium, the ability to use a human as the learning algorithm degrades. From medium to large, it becomes essential to have learning algorithms that don’t require ran
6 0.089617945 32 hunch net-2005-02-27-Antilearning: When proximity goes bad
7 0.080408536 281 hunch net-2007-12-21-Vowpal Wabbit Code Release
8 0.079165258 262 hunch net-2007-09-16-Optimizing Machine Learning Programs
9 0.078116529 210 hunch net-2006-09-28-Programming Languages for Machine Learning Implementations
10 0.074888349 297 hunch net-2008-04-22-Taking the next step
11 0.074684091 451 hunch net-2011-12-13-Vowpal Wabbit version 6.1 & the NIPS tutorial
12 0.07356061 215 hunch net-2006-10-22-Exemplar programming
13 0.072959304 308 hunch net-2008-07-06-To Dual or Not
14 0.072773173 208 hunch net-2006-09-18-What is missing for online collaborative research?
15 0.072520033 365 hunch net-2009-07-31-Vowpal Wabbit Open Source Project
16 0.067558497 35 hunch net-2005-03-04-The Big O and Constants in Learning
17 0.06603843 120 hunch net-2005-10-10-Predictive Search is Coming
18 0.065151066 347 hunch net-2009-03-26-Machine Learning is too easy
19 0.064757951 473 hunch net-2012-09-29-Vowpal Wabbit, version 7.0
20 0.064193927 454 hunch net-2012-01-30-ICML Posters and Scope
topicId topicWeight
[(0, 0.168), (1, 0.023), (2, -0.055), (3, 0.049), (4, 0.013), (5, 0.023), (6, -0.057), (7, 0.005), (8, -0.057), (9, 0.056), (10, -0.114), (11, -0.028), (12, 0.015), (13, -0.019), (14, 0.006), (15, -0.063), (16, -0.022), (17, 0.014), (18, -0.005), (19, -0.018), (20, 0.014), (21, 0.014), (22, -0.029), (23, 0.064), (24, -0.066), (25, 0.057), (26, 0.036), (27, -0.015), (28, 0.074), (29, 0.019), (30, -0.015), (31, 0.012), (32, 0.048), (33, 0.001), (34, 0.061), (35, -0.01), (36, 0.022), (37, -0.044), (38, 0.017), (39, -0.108), (40, 0.028), (41, -0.081), (42, 0.075), (43, 0.004), (44, -0.033), (45, -0.05), (46, 0.006), (47, 0.057), (48, 0.018), (49, 0.125)]
simIndex simValue blogId blogTitle
same-blog 1 0.95734364 128 hunch net-2005-11-05-The design of a computing cluster
Introduction: This is about the design of a computing cluster from the viewpoint of applied machine learning using current technology. We just built a small one at TTI so this is some evidence of what is feasible and thoughts about the design choices. Architecture There are several architectural choices. AMD Athlon64 based system. This seems to have the cheapest bang/buck. Maximum RAM is typically 2-3GB. AMD Opteron based system. Opterons provide the additional capability to buy an SMP motherboard with two chips, and the motherboards often support 16GB of RAM. The RAM is also the more expensive error correcting type. Intel PIV or Xeon based system. The PIV and Xeon based systems are the intel analog of the above 2. Due to architectural design reasons, these chips tend to run a bit hotter and be a bit more expensive. Dual core chips. Both Intel and AMD have chips that actually have 2 processors embedded in them. In the end, we decided to go with option (2). Roughly speaking,
2 0.73515594 229 hunch net-2007-01-26-Parallel Machine Learning Problems
Introduction: Parallel machine learning is a subject rarely addressed at machine learning conferences. Nevertheless, it seems likely to increase in importance because: Data set sizes appear to be growing substantially faster than computation. Essentially, this happens because more and more sensors of various sorts are being hooked up to the internet. Serial speedups of processors seem are relatively stalled. The new trend is to make processors more powerful by making them multicore . Both AMD and Intel are making dual core designs standard, with plans for more parallelism in the future. IBM’s Cell processor has (essentially) 9 cores. Modern graphics chips can have an order of magnitude more separate execution units. The meaning of ‘core’ varies a bit from processor to processor, but the overall trend seems quite clear. So, how do we parallelize machine learning algorithms? The simplest and most common technique is to simply run the same learning algorithm with di
3 0.66729361 210 hunch net-2006-09-28-Programming Languages for Machine Learning Implementations
Introduction: Machine learning algorithms have a much better chance of being widely adopted if they are implemented in some easy-to-use code. There are several important concerns associated with machine learning which stress programming languages on the ease-of-use vs. speed frontier. Speed The rate at which data sources are growing seems to be outstripping the rate at which computational power is growing, so it is important that we be able to eak out every bit of computational power. Garbage collected languages ( java , ocaml , perl and python ) often have several issues here. Garbage collection often implies that floating point numbers are “boxed”: every float is represented by a pointer to a float. Boxing can cause an order of magnitude slowdown because an extra nonlocalized memory reference is made, and accesses to main memory can are many CPU cycles long. Garbage collection often implies that considerably more memory is used than is necessary. This has a variable effect. I
4 0.63292152 450 hunch net-2011-12-02-Hadoop AllReduce and Terascale Learning
Introduction: Suppose you have a dataset with 2 terafeatures (we only count nonzero entries in a datamatrix), and want to learn a good linear predictor in a reasonable amount of time. How do you do it? As a learning theorist, the first thing you do is pray that this is too much data for the number of parameters—but that’s not the case, there are around 16 billion examples, 16 million parameters, and people really care about a high quality predictor, so subsampling is not a good strategy. Alekh visited us last summer, and we had a breakthrough (see here for details), coming up with the first learning algorithm I’ve seen that is provably faster than any future single machine learning algorithm. The proof of this is simple: We can output a optimal-up-to-precision linear predictor faster than the data can be streamed through the network interface of any single machine involved in the computation. It is necessary but not sufficient to have an effective communication infrastructure. It is ne
5 0.61735499 136 hunch net-2005-12-07-Is the Google way the way for machine learning?
Introduction: Urs Hoelzle from Google gave an invited presentation at NIPS . In the presentation, he strongly advocates interacting with data in a particular scalable manner which is something like the following: Make a cluster of machines. Build a unified filesystem. (Google uses GFS, but NFS or other approaches work reasonably well for smaller clusters.) Interact with data via MapReduce . Creating a cluster of machines is, by this point, relatively straightforward. Unified filesystems are a little bit tricky—GFS is capable by design of essentially unlimited speed throughput to disk. NFS can bottleneck because all of the data has to move through one machine. Nevertheless, this may not be a limiting factor for smaller clusters. MapReduce is a programming paradigm. Essentially, it is a combination of a data element transform (map) and an agreggator/selector (reduce). These operations are highly parallelizable and the claim is that they support the forms of data interacti
6 0.58430403 300 hunch net-2008-04-30-Concerns about the Large Scale Learning Challenge
7 0.58271986 262 hunch net-2007-09-16-Optimizing Machine Learning Programs
8 0.56366402 366 hunch net-2009-08-03-Carbon in Computer Science Research
9 0.54539412 346 hunch net-2009-03-18-Parallel ML primitives
10 0.53250986 120 hunch net-2005-10-10-Predictive Search is Coming
11 0.53001511 215 hunch net-2006-10-22-Exemplar programming
12 0.51446801 287 hunch net-2008-01-28-Sufficient Computation
13 0.50128877 250 hunch net-2007-06-23-Machine Learning Jobs are Growing on Trees
14 0.4848569 399 hunch net-2010-05-20-Google Predict
15 0.4786858 365 hunch net-2009-07-31-Vowpal Wabbit Open Source Project
16 0.47320792 424 hunch net-2011-02-17-What does Watson mean?
17 0.46423289 84 hunch net-2005-06-22-Languages of Learning
18 0.46256366 451 hunch net-2011-12-13-Vowpal Wabbit version 6.1 & the NIPS tutorial
19 0.44873288 281 hunch net-2007-12-21-Vowpal Wabbit Code Release
20 0.43968293 105 hunch net-2005-08-23-(Dis)similarities between academia and open source programmers
topicId topicWeight
[(0, 0.018), (3, 0.024), (27, 0.154), (38, 0.038), (48, 0.027), (51, 0.025), (53, 0.032), (55, 0.061), (62, 0.333), (64, 0.03), (78, 0.011), (94, 0.134), (95, 0.026)]
simIndex simValue blogId blogTitle
1 0.92294765 173 hunch net-2006-04-17-Rexa is live
Introduction: Rexa is now publicly available. Anyone can create an account and login. Rexa is similar to Citeseer and Google Scholar in functionality with more emphasis on the use of machine learning for intelligent information extraction. For example, Rexa can automatically display a picture on an author’s homepage when the author is searched for.
2 0.88525569 195 hunch net-2006-07-12-Who is having visa problems reaching US conferences?
Introduction: Many of the large machine learning conferences were in the US this summer. A common problem which students from abroad encounter is visa issues. Just getting a visa to visit can be pretty rough: you stand around in lines, sometimes for days. Even worse is the timing with respect to ticket buying. Airplane tickets typically need to be bought well in advance on nonrefundable terms to secure a reasonable rate for air travel. When a visa is denied, as happens reasonably often, a very expensive ticket is burnt. A serious effort is under way to raise this as in issue in need of fixing. Over the long term, effectively driving research conferences to locate outside of the US seems an unwise policy. Robert Schapire is planning to talk to a congressman. Sally Goldman suggested putting together a list of problem cases, and Phil Long setup an email address immigration.and.confs@gmail.com to collect them. If you (or someone you know) has had insurmountable difficulties reaching
same-blog 3 0.86628324 128 hunch net-2005-11-05-The design of a computing cluster
Introduction: This is about the design of a computing cluster from the viewpoint of applied machine learning using current technology. We just built a small one at TTI so this is some evidence of what is feasible and thoughts about the design choices. Architecture There are several architectural choices. AMD Athlon64 based system. This seems to have the cheapest bang/buck. Maximum RAM is typically 2-3GB. AMD Opteron based system. Opterons provide the additional capability to buy an SMP motherboard with two chips, and the motherboards often support 16GB of RAM. The RAM is also the more expensive error correcting type. Intel PIV or Xeon based system. The PIV and Xeon based systems are the intel analog of the above 2. Due to architectural design reasons, these chips tend to run a bit hotter and be a bit more expensive. Dual core chips. Both Intel and AMD have chips that actually have 2 processors embedded in them. In the end, we decided to go with option (2). Roughly speaking,
4 0.84136409 394 hunch net-2010-04-24-COLT Treasurer is now Phil Long
Introduction: For about 5 years, I’ve been the treasurer of the Association for Computational Learning, otherwise known as COLT, taking over from John Case before me. A transfer of duties to Phil Long is now about complete. This probably matters to almost no one, but I wanted to describe things a bit for those interested. The immediate impetus for this decision was unhappiness over reviewing decisions at COLT 2009 , one as an author and several as a member of the program committee. I seem to have disagreements fairly often about what is important work, partly because I’m focused on learning theory with practical implications, partly because I define learning theory more broadly than is typical amongst COLT members, and partly because COLT suffers a bit from insider-clique issues. The degree to which these issues come up varies substantially each year so last year is not predictive of this one. And, it’s important to understand that COLT remains healthy with these issues not nearly so bad
5 0.80659568 263 hunch net-2007-09-18-It’s MDL Jim, but not as we know it…(on Bayes, MDL and consistency)
Introduction: I have recently completed a 500+ page-book on MDL , the first comprehensive overview of the field (yes, this is a sneak advertisement ). Chapter 17 compares MDL to a menagerie of other methods and paradigms for learning and statistics. By far the most time (20 pages) is spent on the relation between MDL and Bayes. My two main points here are: In sharp contrast to Bayes, MDL is by definition based on designing universal codes for the data relative to some given (parametric or nonparametric) probabilistic model M. By some theorems due to Andrew Barron , MDL inference must therefore be statistically consistent, and it is immune to Bayesian inconsistency results such as those by Diaconis, Freedman and Barron (I explain what I mean by “inconsistency” further below). Hence, MDL must be different from Bayes! In contrast to what has sometimes been claimed, practical MDL algorithms do have a subjective component (which in many, but not all cases, may be implemented by somethin
6 0.5695194 229 hunch net-2007-01-26-Parallel Machine Learning Problems
7 0.54334092 221 hunch net-2006-12-04-Structural Problems in NIPS Decision Making
8 0.5385049 136 hunch net-2005-12-07-Is the Google way the way for machine learning?
9 0.53837603 276 hunch net-2007-12-10-Learning Track of International Planning Competition
10 0.53640753 286 hunch net-2008-01-25-Turing’s Club for Machine Learning
11 0.52964914 120 hunch net-2005-10-10-Predictive Search is Coming
12 0.5239526 300 hunch net-2008-04-30-Concerns about the Large Scale Learning Challenge
13 0.52370113 450 hunch net-2011-12-02-Hadoop AllReduce and Terascale Learning
14 0.52113706 359 hunch net-2009-06-03-Functionally defined Nonlinear Dynamic Models
15 0.52003103 95 hunch net-2005-07-14-What Learning Theory might do
16 0.5194428 210 hunch net-2006-09-28-Programming Languages for Machine Learning Implementations
17 0.51648182 237 hunch net-2007-04-02-Contextual Scaling
18 0.51600415 371 hunch net-2009-09-21-Netflix finishes (and starts)
19 0.51577246 306 hunch net-2008-07-02-Proprietary Data in Academic Research?
20 0.51381409 419 hunch net-2010-12-04-Vowpal Wabbit, version 5.0, and the second heresy