high_scalability high_scalability-2010 high_scalability-2010-882 knowledge-graph by maker-knowledge-mining
Source: html
Introduction: Misco: A MapReduce Framework for Mobile Systems is a very exciting paper to me because it's really one of the first explorations of some of the ideas in Building Super Scalable Systems: Blade Runner Meets Autonomic Computing in the Ambient Cloud . What they are trying to do is efficiently distribute work across a set cellphones using a now familiar MapReduce interface. Usually we think of MapReduce as working across large data center hosted clusters. Here, the cluster nodes are cellphones not contained in any data center, but compute nodes potentially distributed everywhere. I talked with Adam Dou , one of the paper's authors, and he said they don't see cellphone clusters replacing dedicated computer clusters, primarily because of the power required for both network communication and the map-reduce computations. Large multi-terabyte jobs aren't in the cards...yet. Adam estimates computationally that cellphones are performing similarly to desktops of ten years ago. Instead, they
sentIndex sentText sentNum sentScore
1 Here, the cluster nodes are cellphones not contained in any data center, but compute nodes potentially distributed everywhere. [sent-4, score-0.296]
2 Adam estimates computationally that cellphones are performing similarly to desktops of ten years ago. [sent-10, score-0.294]
3 It's interesting to contrast the economics of the ambient cloud to the economics of the data center cloud. [sent-14, score-0.49]
4 Since there are more phones than computers, using more of less capable devices will increase overall performance. [sent-20, score-0.266]
5 A quick introduction to Misco from the abstract: The proliferation of increasingly powerful, ubiquitous mobile devices has created a new and powerful sensing and computational environment. [sent-22, score-0.297]
6 We present a framework which provides a powerful software abstraction that hides many of such complexities from the application developer. [sent-24, score-0.228]
7 We design and implement a mobile MapReduce framework targeted at any device which supports Python and network connectivity. [sent-25, score-0.433]
8 An overview of the architecture is depicted by this excellent diagram: MasterServer - keeps track of user applications; maintains input, intermediary and results data associated with applications; tracks worker progress; assigns tasks to workers. [sent-27, score-0.569]
9 Communication between the server and worker is via HTTP. [sent-28, score-0.309]
10 A server consists of: Application Repository Scheduler component - keeps track of application input and output data. [sent-29, score-0.531]
11 WorkerNode - performs map and reduce operations and returns the results to the server. [sent-33, score-0.28]
12 A worker consists of: Requester component - interacts with the server to requests tasks; uploads and downloads data; triggers local task execution; performs upgrades. [sent-35, score-0.763]
13 Repository component - stores the input data, modules, and results for each task. [sent-36, score-0.359]
14 Logger component - maintains local process times and progress and uploads them to the server along with the results when a task completes. [sent-37, score-0.633]
15 A job consists of the deadline, input data, module, size of the map input pieces, and number of reduce partitions. [sent-41, score-0.693]
16 The server splits the input data into M inputs files. [sent-42, score-0.429]
17 The dominance of power usage of network transfers over local computation may dictate where optimization efforts go in the future. [sent-65, score-0.242]
18 Adam suggests a lot of energy may be saved by compressing the data so that there is much less data to send. [sent-67, score-0.244]
19 Say you are are performing and image recognition task on pictures that are already on the phone. [sent-69, score-0.251]
20 Batching data would use less network resources, but that must be balanced against responsiveness. [sent-71, score-0.223]
wordName wordTfidf (topN-words)
[('input', 0.24), ('misco', 0.229), ('worker', 0.208), ('cellphones', 0.208), ('mapreduce', 0.179), ('nokia', 0.155), ('cellphone', 0.138), ('adam', 0.136), ('ambient', 0.136), ('consists', 0.123), ('results', 0.119), ('mobile', 0.114), ('devices', 0.112), ('center', 0.106), ('server', 0.101), ('router', 0.094), ('smartphones', 0.093), ('map', 0.09), ('framework', 0.09), ('local', 0.088), ('data', 0.088), ('device', 0.087), ('uploads', 0.087), ('usage', 0.087), ('phones', 0.086), ('performing', 0.086), ('polling', 0.085), ('task', 0.085), ('workers', 0.083), ('mb', 0.082), ('failures', 0.081), ('communication', 0.08), ('pictures', 0.08), ('economics', 0.08), ('scheduler', 0.079), ('tasks', 0.078), ('progress', 0.077), ('maintains', 0.076), ('supports', 0.075), ('powerful', 0.071), ('performs', 0.071), ('cell', 0.071), ('explorations', 0.069), ('microphone', 0.069), ('testbed', 0.069), ('less', 0.068), ('network', 0.067), ('application', 0.067), ('exciting', 0.066), ('intern', 0.065)]
simIndex simValue blogId blogTitle
same-blog 1 1.0000001 882 high scalability-2010-08-18-Misco: A MapReduce Framework for Mobile Systems - Start of the Ambient Cloud?
Introduction: Misco: A MapReduce Framework for Mobile Systems is a very exciting paper to me because it's really one of the first explorations of some of the ideas in Building Super Scalable Systems: Blade Runner Meets Autonomic Computing in the Ambient Cloud . What they are trying to do is efficiently distribute work across a set cellphones using a now familiar MapReduce interface. Usually we think of MapReduce as working across large data center hosted clusters. Here, the cluster nodes are cellphones not contained in any data center, but compute nodes potentially distributed everywhere. I talked with Adam Dou , one of the paper's authors, and he said they don't see cellphone clusters replacing dedicated computer clusters, primarily because of the power required for both network communication and the map-reduce computations. Large multi-terabyte jobs aren't in the cards...yet. Adam estimates computationally that cellphones are performing similarly to desktops of ten years ago. Instead, they
Introduction: "But it is not complicated. [There's] just a lot of it." \--Richard Feynmanon how the immense variety of the world arises from simple rules.Contents:Have We Reached the End of Scaling?Applications Become Black Boxes Using Markets to Scale and Control CostsLet's Welcome our Neo-Feudal OverlordsThe Economic Argument for the Ambient CloudWhat Will Kill the Cloud?The Amazing Collective Compute Power of the Ambient CloudUsing the Ambient Cloud as an Application RuntimeApplications as Virtual StatesConclusionWe have not yet begun to scale. The world is still fundamentally disconnected and for all our wisdom we are still in the earliest days of learning how to build truly large planet-scaling applications.Today 350 million users on Facebook is a lot of users and five million followers on Twitter is a lot of followers. This may seem like a lot now, but consider we have no planet wide applications yet. None.Tomorrow the numbers foreshadow a newCambrian explosionof connectivity that will look as
Introduction: All in all this is still my favorite post and I still think it's an accurate vision of a future. Not everyone agrees, but I guess we'll see..."But it is not complicated. [There's] just a lot of it." \--Richard Feynmanon how the immense variety of the world arises from simple rules.Contents:Have We Reached the End of Scaling?Applications Become Black Boxes Using Markets to Scale and Control CostsLet's Welcome our Neo-Feudal OverlordsThe Economic Argument for the Ambient CloudWhat Will Kill the Cloud?The Amazing Collective Compute Power of the Ambient CloudUsing the Ambient Cloud as an Application RuntimeApplications as Virtual StatesConclusionWe have not yet begun to scale. The world is still fundamentally disconnected and for all our wisdom we are still in the earliest days of learning how to build truly large planet-scaling applications.Today 350 million users on Facebook is a lot of users and five million followers on Twitter is a lot of followers. This may seem like a lot now, but c
4 0.18568622 768 high scalability-2010-02-01-What Will Kill the Cloud?
Introduction: This is an excerpt from my article Building Super Scalable Systems: Blade Runner Meets Autonomic Computing in the Ambient Cloud. If datacenters are the new castles, then what will be the new gunpowder? As soon as gunpowder came on the scene, castles, which are defensive structures, quickly became the future's cold, drafty hotels. Gunpowder fueled cannon balls make short work of castle walls. There's a long history of "gunpowder" type inventions in the tech industry. PCs took out the timeshare model. The cloud is taking out the PC model. There must be something that will take out the cloud. Right now it's hard to believe the cloud will one day be no more. They seem so much the future, but something will transcend the cloud. We even have a law that says so: Bell's Law of Computer Classes which holds that roughly every decade a new, lower priced computer class forms based on a new programming platform, network, and interface resulting in new usage and the establishment of
5 0.18340461 778 high scalability-2010-02-15-The Amazing Collective Compute Power of the Ambient Cloud
Introduction: This is an excerpt from my article Building Super Scalable Systems: Blade Runner Meets Autonomic Computing in the Ambient Cloud. Earlier we talked about how a single botnet could harness more compute power than our largest super computers. Well, that's just the start of it. The amount of computer power available to the Ambient Cloud will be truly astounding. 2 Billion Personal Computers The number of personal computers is still growing. By 2014 one estimate is there will be 2 billion PCs . That's a giant reservoir of power to exploit, especially considering these new boxes are stuffed with multiple powerful processors and gigabytes of memory. 7 Billion Smartphones By now it's common wisdom smartphones are the computing platform of the future. It's plausible to assume the total number of mobile phones in use will roughly equal the number of people on earth. That's 7 billion smartphones. Smartphones aren't just tiny little wannabe computers anymore either. They are rea
6 0.18203363 448 high scalability-2008-11-22-Google Architecture
7 0.17347014 786 high scalability-2010-03-02-Using the Ambient Cloud as an Application Runtime
8 0.15843295 483 high scalability-2009-01-04-Paper: MapReduce: Simplified Data Processing on Large Clusters
9 0.15083545 761 high scalability-2010-01-17-Applications Become Black Boxes Using Markets to Scale and Control Costs
10 0.14456677 1000 high scalability-2011-03-08-Medialets Architecture - Defeating the Daunting Mobile Device Data Deluge
11 0.13417077 666 high scalability-2009-07-30-Learn How to Think at Scale
12 0.13233913 821 high scalability-2010-05-03-MocoSpace Architecture - 3 Billion Mobile Page Views a Month
13 0.12398146 1042 high scalability-2011-05-17-Facebook: An Example Canonical Architecture for Scaling Billions of Messages
14 0.1203239 1240 high scalability-2012-05-07-Startups are Creating a New System of the World for IT
15 0.1179098 1607 high scalability-2014-03-07-Stuff The Internet Says On Scalability For March 7th, 2014
16 0.1172497 823 high scalability-2010-05-05-How will memristors change everything?
17 0.11587621 951 high scalability-2010-12-01-8 Commonly Used Scalable System Design Patterns
18 0.11573999 954 high scalability-2010-12-06-What the heck are you actually using NoSQL for?
19 0.11562382 1600 high scalability-2014-02-21-Stuff The Internet Says On Scalability For February 21st, 2014
20 0.11548519 1382 high scalability-2013-01-07-Analyzing billions of credit card transactions and serving low-latency insights in the cloud
topicId topicWeight
[(0, 0.236), (1, 0.11), (2, 0.03), (3, 0.064), (4, -0.062), (5, 0.008), (6, 0.105), (7, 0.04), (8, -0.063), (9, 0.093), (10, 0.03), (11, 0.028), (12, 0.046), (13, -0.043), (14, 0.062), (15, 0.014), (16, -0.079), (17, -0.027), (18, 0.025), (19, 0.005), (20, -0.045), (21, 0.027), (22, -0.068), (23, 0.002), (24, 0.08), (25, 0.012), (26, -0.025), (27, 0.023), (28, -0.021), (29, 0.007), (30, -0.029), (31, 0.06), (32, -0.008), (33, 0.041), (34, 0.029), (35, -0.045), (36, 0.02), (37, -0.019), (38, -0.065), (39, 0.019), (40, 0.051), (41, -0.009), (42, -0.053), (43, 0.005), (44, 0.039), (45, -0.035), (46, -0.06), (47, 0.068), (48, -0.015), (49, -0.021)]
simIndex simValue blogId blogTitle
same-blog 1 0.93374443 882 high scalability-2010-08-18-Misco: A MapReduce Framework for Mobile Systems - Start of the Ambient Cloud?
Introduction: Misco: A MapReduce Framework for Mobile Systems is a very exciting paper to me because it's really one of the first explorations of some of the ideas in Building Super Scalable Systems: Blade Runner Meets Autonomic Computing in the Ambient Cloud . What they are trying to do is efficiently distribute work across a set cellphones using a now familiar MapReduce interface. Usually we think of MapReduce as working across large data center hosted clusters. Here, the cluster nodes are cellphones not contained in any data center, but compute nodes potentially distributed everywhere. I talked with Adam Dou , one of the paper's authors, and he said they don't see cellphone clusters replacing dedicated computer clusters, primarily because of the power required for both network communication and the map-reduce computations. Large multi-terabyte jobs aren't in the cards...yet. Adam estimates computationally that cellphones are performing similarly to desktops of ten years ago. Instead, they
2 0.75081551 826 high scalability-2010-05-12-The Rise of the Virtual Cellular Machines
Introduction: My apologies if you were looking for a post about cell phones. This post is about high density nanodevices. It's a follow up to How will memristors change everything? for those wishing to pursue these revolutionary ideas in more depth. This is one of those areas where if you are in the space then there's a lot of available information and if you are on the outside then it doesn't even seem to exist. Fortunately, Ben Chandler from The SyNAPSE Project , was kind enough to point me to a great set of presentations given at the 12th IEEE CNNA - International Workshop on Cellular Nanoscale Networks and their Applications - Towards Megaprocessor Computing. WARNING: these papers contain extreme technical content. If you are like me and you aren't an electrical engineer, much of it may make a sort of surface sense, but the deep and twisty details will fly over head. For the more software minded there are a couple more accessible presentations: Intelligent Machines built with Memristiv
3 0.73275518 839 high scalability-2010-06-09-Paper: Propagation Networks: A Flexible and Expressive Substrate for Computation
Introduction: Alexey Radul in his fascinating 174 page dissertation Propagation Networks: A Flexible and Expressive Substrate for Computation , offers to help us break free of the tyranny of linear time by arranging computation as a network of autonomous but interconnected machines . We can do this by organizing computation as a network of interconnected machines of some kind, each of which is free to run when it pleases, propagating information around the network as proves possible. The consequence of this freedom is that the structure of the aggregate does not impose an order of time. The abstract from his thesis is : In this dissertation I propose a shift in the foundations of computation. Modern programming systems are not expressive enough. The traditional image of a single computer that has global effects on a large memory is too restrictive. The propagation paradigm replaces this with computing by networks of local, independent, stateless machines interconnected with stateful storage
4 0.71813941 786 high scalability-2010-03-02-Using the Ambient Cloud as an Application Runtime
Introduction: This is an excerpt from my article Building Super Scalable Systems: Blade Runner Meets Autonomic Computing in the Ambient Cloud. The future looks many, big, complex, and adaptive: Many clouds. Many servers. Many operating systems. Many languages. Many storage services. Many database services. Many software services. Many adjunct human networks (like Mechanical Turk). Many fast interconnects. Many CDNs. Many cache memory pools. Many application profiles (simple request-response, live streaming, computationally complex, sensor driven, memory intensive, storage intensive, monolithic, decomposable, etc). Many legal jurisdictions. Don't want to perform a function on Patriot Act "protected" systems then move the function elsewhere. Many SLAs. Many data driven pricing policies that like airplane pricing algorithms will price "seats" to maximize profit using multi-variate time sensitive pricing models. Many competitive products. The need t
5 0.7109977 823 high scalability-2010-05-05-How will memristors change everything?
Introduction: A non-random sample of my tech friends shows that not many have heard of memristors (though I do suspect vote tampering). I'd read a little about memristors in 2008 when the initial hubbub about the existence of memristors was raised. I, however, immediately filed them into that comforting conceptual bucket of potentially revolutionary technologies I didn't have to worry about because like most wondertech, nothing would ever come of it. Wrong. After watching Finding the Missing Memristor by R. Stanley Williams I've had to change my mind. Memristors have gone from "maybe never" to holy cow this could happen soon and it could change everything. Let's assume for the sake of dreaming memristors do prove out. How will we design systems when we have access to a new material that is two orders of magnitude more efficient from a power perspective than traditional transistor technologies, contains multiple petabits (1 petabit = 128TB) of persistent storage, and can be reconfigured t
6 0.7001375 983 high scalability-2011-02-02-Piccolo - Building Distributed Programs that are 11x Faster than Hadoop
7 0.69578892 778 high scalability-2010-02-15-The Amazing Collective Compute Power of the Ambient Cloud
8 0.69478291 1509 high scalability-2013-08-30-Stuff The Internet Says On Scalability For August 30, 2013
9 0.69375336 953 high scalability-2010-12-03-GPU vs CPU Smackdown : The Rise of Throughput-Oriented Architectures
10 0.69365644 326 high scalability-2008-05-25-Product: Condor - Compute Intensive Workload Management
12 0.68732202 750 high scalability-2009-12-16-Building Super Scalable Systems: Blade Runner Meets Autonomic Computing in the Ambient Cloud
13 0.68446279 1316 high scalability-2012-09-04-Changing Architectures: New Datacenter Networks Will Set Your Code and Data Free
14 0.68214244 448 high scalability-2008-11-22-Google Architecture
15 0.68131828 761 high scalability-2010-01-17-Applications Become Black Boxes Using Markets to Scale and Control Costs
16 0.67865515 768 high scalability-2010-02-01-What Will Kill the Cloud?
17 0.67546117 1600 high scalability-2014-02-21-Stuff The Internet Says On Scalability For February 21st, 2014
18 0.67047805 1572 high scalability-2014-01-03-Stuff The Internet Says On Scalability For January 3rd, 2014
19 0.67009145 901 high scalability-2010-09-16-How Can the Large Hadron Collider Withstand One Petabyte of Data a Second?
20 0.66809636 1222 high scalability-2012-04-05-Big Data Counting: How to count a billion distinct objects using only 1.5KB of Memory
topicId topicWeight
[(1, 0.1), (2, 0.199), (5, 0.043), (10, 0.038), (30, 0.025), (47, 0.016), (52, 0.148), (61, 0.073), (77, 0.032), (79, 0.179), (85, 0.012), (94, 0.065)]
simIndex simValue blogId blogTitle
1 0.93235296 244 high scalability-2008-02-11-Yahoo Live's Scaling Problems Prove: Release Early and Often - Just Don't Screw Up
Introduction: Tech Crunch chomped down on some initial scaling problems with Yahoo's new live video streaming service Yahoo Live . After a bit of chewing on Yahoo's old bones, TC spat out: If Yahoo cant scale something like this (no matter how much they claim it’s an experiment, it’s still a live service), it shows how far the once brightest star of the online world has fallen . This kind of thinking kills innovation. When there's no room for a few hiccups or a little failure you have to cover your ass so completely nothing new will ever see the light of day. I thought we were supposed to be agile . We are supposed to release early and often. Not every 'i' has to be dotted and not every last router has to be installed before we take the first step of a grand new journey. Get it out there. Let users help you make it better. Listen to customers, make changes, push the new code out, listen some more, and fix problems as they come up. Following this process we'll make somethi
same-blog 2 0.93222034 882 high scalability-2010-08-18-Misco: A MapReduce Framework for Mobile Systems - Start of the Ambient Cloud?
Introduction: Misco: A MapReduce Framework for Mobile Systems is a very exciting paper to me because it's really one of the first explorations of some of the ideas in Building Super Scalable Systems: Blade Runner Meets Autonomic Computing in the Ambient Cloud . What they are trying to do is efficiently distribute work across a set cellphones using a now familiar MapReduce interface. Usually we think of MapReduce as working across large data center hosted clusters. Here, the cluster nodes are cellphones not contained in any data center, but compute nodes potentially distributed everywhere. I talked with Adam Dou , one of the paper's authors, and he said they don't see cellphone clusters replacing dedicated computer clusters, primarily because of the power required for both network communication and the map-reduce computations. Large multi-terabyte jobs aren't in the cards...yet. Adam estimates computationally that cellphones are performing similarly to desktops of ten years ago. Instead, they
Introduction: This is a guest post by Ivan de Prado and Pere Ferrera , founders of Datasalt , the company behind Pangool and Splout SQL Big Data open-source projects. The amount of payments performed using credit cards is huge. It is clear that there is inherent value in the data that can be derived from analyzing all the transactions. Client fidelity, demographics, heat maps of activity, shop recommendations, and many other statistics are useful to both clients and shops for improving their relationship with the market. At Datasalt we have developed a system in collaboration with the BBVA bank that is able to analyze years of data and serve insights and statistics to different low-latency web and mobile applications. The main challenge we faced besides processing Big Data input is that the output was also Big Data, and even bigger than the input . And this output needed to be served quickly, under high load. The solution we developed has an infrastructure cost of just a few tho
4 0.91490477 47 high scalability-2007-07-30-Product: Yslow to speed up your web pages
Introduction: Update : Speed up Apache - how I went from F to A in YSlow . Good example of using YSlow to speed up a website with solid code examples. Every layer in the multi-layer cake that is your website contributes to how long a page takes to display. YSlow , from Yahoo, is a cool tool for discovering how the ingredients of your site's top layer contribute to performance. YSlow analyzes web pages and tells you why they're slow based on the rules for high performance web sites. YSlow is a Firefox add-on integrated with the popular Firebug web development tool. YSlow gives you: Performance report card HTTP/HTML summary List of components in the page Tools including JSLint
Introduction: Scoble the Ubiquitous has a fascinating post on how Mogulus, a live video channel startup, uses S3/EC2 and doesn't own a single server. The trends that have been happening for a while now are going mainstream. To do great things you no longer need to start by creating a huge war chest. You can forage off the land, like any good mobile, light weight fighting unit. For a strategy hit he mentions the same needed change in perspective as Beau Lebens talked about when making FeedBlendr : One tip he gave us is that when using Amazon’s services you have to design your systems with the assumption that they will never be up and running. What he means by that is services are “volatile” and can go up and down without notice. So, he’s designed his systems to survive that. He told me that it meant his engineering teams had to be quite disciplined in designing their architecture.
6 0.89349341 1117 high scalability-2011-09-16-Stuff The Internet Says On Scalability For September 16, 2011
7 0.89285421 1451 high scalability-2013-05-03-Stuff The Internet Says On Scalability For May 3, 2013
8 0.88391817 872 high scalability-2010-08-05-Pairing NoSQL and Relational Data Storage: MySQL with MongoDB
9 0.88294196 601 high scalability-2009-05-17-Product: Hadoop
10 0.88241005 289 high scalability-2008-03-27-Amazon Announces Static IP Addresses and Multiple Datacenter Operation
11 0.88235515 1406 high scalability-2013-02-14-When all the Program's a Graph - Prismatic's Plumbing Library
13 0.8753038 1316 high scalability-2012-09-04-Changing Architectures: New Datacenter Networks Will Set Your Code and Data Free
14 0.87388796 409 high scalability-2008-10-13-Challenges from large scale computing at Google
15 0.87272191 687 high scalability-2009-08-24-How Google Serves Data from Multiple Datacenters
16 0.87200767 1048 high scalability-2011-05-27-Stuff The Internet Says On Scalability For May 27, 2011
17 0.87129521 517 high scalability-2009-02-21-Google AppEngine - A Second Look
18 0.87117803 786 high scalability-2010-03-02-Using the Ambient Cloud as an Application Runtime
20 0.87101817 763 high scalability-2010-01-22-How BuddyPoke Scales on Facebook Using Google App Engine