high_scalability high_scalability-2010 high_scalability-2010-786 knowledge-graph by maker-knowledge-mining
Source: html
Introduction: This is an excerpt from my article Building Super Scalable Systems: Blade Runner Meets Autonomic Computing in the Ambient Cloud. The future looks many, big, complex, and adaptive: Many clouds. Many servers. Many operating systems. Many languages. Many storage services. Many database services. Many software services. Many adjunct human networks (like Mechanical Turk). Many fast interconnects. Many CDNs. Many cache memory pools. Many application profiles (simple request-response, live streaming, computationally complex, sensor driven, memory intensive, storage intensive, monolithic, decomposable, etc). Many legal jurisdictions. Don't want to perform a function on Patriot Act "protected" systems then move the function elsewhere. Many SLAs. Many data driven pricing policies that like airplane pricing algorithms will price "seats" to maximize profit using multi-variate time sensitive pricing models. Many competitive products. The need t
sentIndex sentText sentNum sentScore
1 In a runtime you see whatever you built the program to see and whatever is out there to talk to in whatever language the things want to talk in. [sent-59, score-0.24]
2 You would have access to whatever is allowed on whatever you are running on. [sent-68, score-0.169]
3 The wide variety of different clouds and different compute resources will make it difficult to come up with a true standardization layer. [sent-88, score-0.249]
4 The inspiration for their approach is the human autonomic nervous system that unconsciously controls key bodily functions like respiration, heart rate, and blood pressure. [sent-106, score-0.208]
5 The question for applications is how far will cloud interoperability go? [sent-114, score-0.168]
6 For a different view on the potential power of all these computer resources take a look at the amazing FAWN (Fast Array of Wimpy Nodes) project out of Carnegie Mellon University. [sent-129, score-0.17]
7 Our prototype FAWN cluster links together a large number of tiny nodes built using embedded processors and small amounts (2-16GB) of flash memory into an ensemble capable of handling 1300 queries per second per node, while consuming fewer than 4 watts of power per node. [sent-132, score-0.326]
8 Given that data intensive applications are I/O sensitive it makes sense not to share. [sent-159, score-0.168]
9 Once flash densities increase even the per byte cost advantage that disks have now for large data sets and streaming media may be breached, the Ambient Cloud itself will be a formidable storage device. [sent-179, score-0.264]
10 One huge tension we have now when designing systems centers on the great disk vs RAM vs flash memory debate. [sent-185, score-0.182]
11 For large datasets RAM is used as a cache and disk as the storage of record. [sent-187, score-0.165]
12 As flash capacity and cost per unit is hitching a ride on Moore’s Law, we can speculate that in a few years that flash and flash/RAM hybrids will start replacing disk based architectures. [sent-197, score-0.302]
13 Using flash means the entire software stack needs to change for flash to be the best it can be . [sent-204, score-0.24]
14 Gordon is a system architecture for data-centric applications that combines low-power processors, flash memory, and datacentric programming systems to improve performance for data-centric applications while reducing power consumption . [sent-207, score-0.289]
15 To summarize, FAWN is a good template for a super scalable key-value data storage because: It operates efficiently on the hardware profile common in the Ambient Cloud: lower power and flash storage. [sent-208, score-0.273]
16 Cleversafe - Space and Bandwidth Efficient High Availability The Ambient Cloud differs from the traditional cloud in one very important way: nodes are unreliable. [sent-211, score-0.181]
17 It's possible using flash based storage might make a more parallelized design possible, especially if techniques like eventual consistency and client based read repair were employed. [sent-255, score-0.222]
18 GAE is built on a distributed file system so getting each entity is really retrieving a different disk block from a different cluster. [sent-297, score-0.162]
19 The reason why I think a market is so critical to the Ambient Cloud is because I think it's the only way to make the dynamic pool of exponentially growing compute resources available to applications. [sent-352, score-0.258]
20 Since the resources are dynamic there needs to be some sort of exchange set up to allow applications to know when new resources become available and search for existing resources. [sent-354, score-0.202]
wordName wordTfidf (topN-words)
[('fawn', 0.505), ('ambient', 0.476), ('flash', 0.12), ('autonomic', 0.117), ('cloud', 0.108), ('cleversafe', 0.095), ('oceanstore', 0.095), ('slices', 0.075), ('nodes', 0.073), ('resources', 0.071), ('market', 0.067), ('intensive', 0.067), ('whatever', 0.065), ('play', 0.063), ('storage', 0.063), ('disk', 0.062), ('applications', 0.06), ('erasure', 0.06), ('vision', 0.059), ('dispersal', 0.057), ('physicalization', 0.057), ('reassemble', 0.057), ('apis', 0.056), ('crazy', 0.053), ('gae', 0.053), ('computing', 0.052), ('different', 0.05), ('power', 0.049), ('maximize', 0.048), ('approach', 0.047), ('order', 0.045), ('runtime', 0.045), ('processors', 0.044), ('anywhere', 0.044), ('wimpy', 0.044), ('nervous', 0.044), ('reason', 0.042), ('replicas', 0.041), ('automatic', 0.041), ('data', 0.041), ('protected', 0.041), ('comparable', 0.041), ('large', 0.04), ('make', 0.039), ('access', 0.039), ('compute', 0.039), ('attempts', 0.039), ('store', 0.039), ('ram', 0.038), ('sensor', 0.038)]
simIndex simValue blogId blogTitle
same-blog 1 1.0000026 786 high scalability-2010-03-02-Using the Ambient Cloud as an Application Runtime
Introduction: This is an excerpt from my article Building Super Scalable Systems: Blade Runner Meets Autonomic Computing in the Ambient Cloud. The future looks many, big, complex, and adaptive: Many clouds. Many servers. Many operating systems. Many languages. Many storage services. Many database services. Many software services. Many adjunct human networks (like Mechanical Turk). Many fast interconnects. Many CDNs. Many cache memory pools. Many application profiles (simple request-response, live streaming, computationally complex, sensor driven, memory intensive, storage intensive, monolithic, decomposable, etc). Many legal jurisdictions. Don't want to perform a function on Patriot Act "protected" systems then move the function elsewhere. Many SLAs. Many data driven pricing policies that like airplane pricing algorithms will price "seats" to maximize profit using multi-variate time sensitive pricing models. Many competitive products. The need t
Introduction: All in all this is still my favorite post and I still think it's an accurate vision of a future. Not everyone agrees, but I guess we'll see..."But it is not complicated. [There's] just a lot of it." \--Richard Feynmanon how the immense variety of the world arises from simple rules.Contents:Have We Reached the End of Scaling?Applications Become Black Boxes Using Markets to Scale and Control CostsLet's Welcome our Neo-Feudal OverlordsThe Economic Argument for the Ambient CloudWhat Will Kill the Cloud?The Amazing Collective Compute Power of the Ambient CloudUsing the Ambient Cloud as an Application RuntimeApplications as Virtual StatesConclusionWe have not yet begun to scale. The world is still fundamentally disconnected and for all our wisdom we are still in the earliest days of learning how to build truly large planet-scaling applications.Today 350 million users on Facebook is a lot of users and five million followers on Twitter is a lot of followers. This may seem like a lot now, but c
Introduction: "But it is not complicated. [There's] just a lot of it." \--Richard Feynmanon how the immense variety of the world arises from simple rules.Contents:Have We Reached the End of Scaling?Applications Become Black Boxes Using Markets to Scale and Control CostsLet's Welcome our Neo-Feudal OverlordsThe Economic Argument for the Ambient CloudWhat Will Kill the Cloud?The Amazing Collective Compute Power of the Ambient CloudUsing the Ambient Cloud as an Application RuntimeApplications as Virtual StatesConclusionWe have not yet begun to scale. The world is still fundamentally disconnected and for all our wisdom we are still in the earliest days of learning how to build truly large planet-scaling applications.Today 350 million users on Facebook is a lot of users and five million followers on Twitter is a lot of followers. This may seem like a lot now, but consider we have no planet wide applications yet. None.Tomorrow the numbers foreshadow a newCambrian explosionof connectivity that will look as
4 0.32679015 761 high scalability-2010-01-17-Applications Become Black Boxes Using Markets to Scale and Control Costs
Introduction: This is an excerpt from my article Building Super Scalable Systems: Blade Runner Meets Autonomic Computing in the Ambient Cloud. We tend to think compute of resources as residing primarily in datacenters. Given the fast pace of innovation we will likely see compute resources become pervasive. Some will reside in datacenters, but compute resources can be anywhere, not just in the datacenter, we'll actually see the bulk of compute resources live outside of datacenters in the future. Given the diversity of compute resources it's reasonable to assume they won't be homogeneous or conform to a standard API. They will specialize by service. Programmers will have to use those specialized service interfaces to build applications that are adaptive enough to take advantage of whatever leverage they can find, whenever and wherever they can find it. Once found the application will have to reorganize on the fly to use whatever new resources it has found and let go of whatever resources it doe
5 0.27114633 768 high scalability-2010-02-01-What Will Kill the Cloud?
Introduction: This is an excerpt from my article Building Super Scalable Systems: Blade Runner Meets Autonomic Computing in the Ambient Cloud. If datacenters are the new castles, then what will be the new gunpowder? As soon as gunpowder came on the scene, castles, which are defensive structures, quickly became the future's cold, drafty hotels. Gunpowder fueled cannon balls make short work of castle walls. There's a long history of "gunpowder" type inventions in the tech industry. PCs took out the timeshare model. The cloud is taking out the PC model. There must be something that will take out the cloud. Right now it's hard to believe the cloud will one day be no more. They seem so much the future, but something will transcend the cloud. We even have a law that says so: Bell's Law of Computer Classes which holds that roughly every decade a new, lower priced computer class forms based on a new programming platform, network, and interface resulting in new usage and the establishment of
6 0.26918072 778 high scalability-2010-02-15-The Amazing Collective Compute Power of the Ambient Cloud
7 0.2099202 538 high scalability-2009-03-16-Are Cloud Based Memory Architectures the Next Big Thing?
8 0.1860812 1240 high scalability-2012-05-07-Startups are Creating a New System of the World for IT
9 0.17752379 790 high scalability-2010-03-09-Applications as Virtual States
10 0.17347014 882 high scalability-2010-08-18-Misco: A MapReduce Framework for Mobile Systems - Start of the Ambient Cloud?
11 0.17282833 753 high scalability-2009-12-21-Hot Holiday Scalability Links for 2009
12 0.16953817 823 high scalability-2010-05-05-How will memristors change everything?
13 0.16808036 758 high scalability-2010-01-11-Have We Reached the End of Scaling?
14 0.16693497 1112 high scalability-2011-09-07-What Google App Engine Price Changes Say About the Future of Web Architecture
15 0.16469687 920 high scalability-2010-10-15-Troubles with Sharding - What can we learn from the Foursquare Incident?
16 0.16104679 517 high scalability-2009-02-21-Google AppEngine - A Second Look
17 0.15190227 954 high scalability-2010-12-06-What the heck are you actually using NoSQL for?
18 0.15124042 765 high scalability-2010-01-25-Let's Welcome our Neo-Feudal Overlords
19 0.14646098 1369 high scalability-2012-12-10-Switch your databases to Flash storage. Now. Or you're doing it wrong.
20 0.14482893 448 high scalability-2008-11-22-Google Architecture
topicId topicWeight
[(0, 0.328), (1, 0.139), (2, 0.09), (3, 0.155), (4, -0.119), (5, -0.047), (6, 0.027), (7, 0.029), (8, -0.047), (9, 0.018), (10, -0.005), (11, 0.015), (12, -0.02), (13, 0.144), (14, 0.129), (15, 0.019), (16, -0.174), (17, -0.033), (18, -0.021), (19, 0.092), (20, -0.148), (21, 0.109), (22, -0.031), (23, 0.032), (24, 0.087), (25, 0.001), (26, 0.003), (27, 0.005), (28, 0.008), (29, -0.134), (30, -0.124), (31, 0.019), (32, 0.03), (33, -0.061), (34, -0.014), (35, -0.11), (36, -0.072), (37, -0.032), (38, -0.108), (39, -0.061), (40, 0.048), (41, -0.052), (42, -0.001), (43, 0.05), (44, 0.063), (45, 0.002), (46, -0.006), (47, 0.034), (48, 0.021), (49, 0.07)]
simIndex simValue blogId blogTitle
same-blog 1 0.92378837 786 high scalability-2010-03-02-Using the Ambient Cloud as an Application Runtime
Introduction: This is an excerpt from my article Building Super Scalable Systems: Blade Runner Meets Autonomic Computing in the Ambient Cloud. The future looks many, big, complex, and adaptive: Many clouds. Many servers. Many operating systems. Many languages. Many storage services. Many database services. Many software services. Many adjunct human networks (like Mechanical Turk). Many fast interconnects. Many CDNs. Many cache memory pools. Many application profiles (simple request-response, live streaming, computationally complex, sensor driven, memory intensive, storage intensive, monolithic, decomposable, etc). Many legal jurisdictions. Don't want to perform a function on Patriot Act "protected" systems then move the function elsewhere. Many SLAs. Many data driven pricing policies that like airplane pricing algorithms will price "seats" to maximize profit using multi-variate time sensitive pricing models. Many competitive products. The need t
Introduction: "But it is not complicated. [There's] just a lot of it." \--Richard Feynmanon how the immense variety of the world arises from simple rules.Contents:Have We Reached the End of Scaling?Applications Become Black Boxes Using Markets to Scale and Control CostsLet's Welcome our Neo-Feudal OverlordsThe Economic Argument for the Ambient CloudWhat Will Kill the Cloud?The Amazing Collective Compute Power of the Ambient CloudUsing the Ambient Cloud as an Application RuntimeApplications as Virtual StatesConclusionWe have not yet begun to scale. The world is still fundamentally disconnected and for all our wisdom we are still in the earliest days of learning how to build truly large planet-scaling applications.Today 350 million users on Facebook is a lot of users and five million followers on Twitter is a lot of followers. This may seem like a lot now, but consider we have no planet wide applications yet. None.Tomorrow the numbers foreshadow a newCambrian explosionof connectivity that will look as
Introduction: All in all this is still my favorite post and I still think it's an accurate vision of a future. Not everyone agrees, but I guess we'll see..."But it is not complicated. [There's] just a lot of it." \--Richard Feynmanon how the immense variety of the world arises from simple rules.Contents:Have We Reached the End of Scaling?Applications Become Black Boxes Using Markets to Scale and Control CostsLet's Welcome our Neo-Feudal OverlordsThe Economic Argument for the Ambient CloudWhat Will Kill the Cloud?The Amazing Collective Compute Power of the Ambient CloudUsing the Ambient Cloud as an Application RuntimeApplications as Virtual StatesConclusionWe have not yet begun to scale. The world is still fundamentally disconnected and for all our wisdom we are still in the earliest days of learning how to build truly large planet-scaling applications.Today 350 million users on Facebook is a lot of users and five million followers on Twitter is a lot of followers. This may seem like a lot now, but c
4 0.89318895 761 high scalability-2010-01-17-Applications Become Black Boxes Using Markets to Scale and Control Costs
Introduction: This is an excerpt from my article Building Super Scalable Systems: Blade Runner Meets Autonomic Computing in the Ambient Cloud. We tend to think compute of resources as residing primarily in datacenters. Given the fast pace of innovation we will likely see compute resources become pervasive. Some will reside in datacenters, but compute resources can be anywhere, not just in the datacenter, we'll actually see the bulk of compute resources live outside of datacenters in the future. Given the diversity of compute resources it's reasonable to assume they won't be homogeneous or conform to a standard API. They will specialize by service. Programmers will have to use those specialized service interfaces to build applications that are adaptive enough to take advantage of whatever leverage they can find, whenever and wherever they can find it. Once found the application will have to reorganize on the fly to use whatever new resources it has found and let go of whatever resources it doe
5 0.85906124 768 high scalability-2010-02-01-What Will Kill the Cloud?
Introduction: This is an excerpt from my article Building Super Scalable Systems: Blade Runner Meets Autonomic Computing in the Ambient Cloud. If datacenters are the new castles, then what will be the new gunpowder? As soon as gunpowder came on the scene, castles, which are defensive structures, quickly became the future's cold, drafty hotels. Gunpowder fueled cannon balls make short work of castle walls. There's a long history of "gunpowder" type inventions in the tech industry. PCs took out the timeshare model. The cloud is taking out the PC model. There must be something that will take out the cloud. Right now it's hard to believe the cloud will one day be no more. They seem so much the future, but something will transcend the cloud. We even have a law that says so: Bell's Law of Computer Classes which holds that roughly every decade a new, lower priced computer class forms based on a new programming platform, network, and interface resulting in new usage and the establishment of
6 0.85652977 790 high scalability-2010-03-09-Applications as Virtual States
7 0.84884357 778 high scalability-2010-02-15-The Amazing Collective Compute Power of the Ambient Cloud
8 0.77606982 765 high scalability-2010-01-25-Let's Welcome our Neo-Feudal Overlords
9 0.7544384 823 high scalability-2010-05-05-How will memristors change everything?
10 0.68090403 1584 high scalability-2014-01-22-How would you build the next Internet? Loons, Drones, Copters, Satellites, or Something Else?
11 0.67784005 826 high scalability-2010-05-12-The Rise of the Virtual Cellular Machines
12 0.65558028 1091 high scalability-2011-08-02-How Will DIDO Wireless Networking Change Everything?
13 0.65472215 1377 high scalability-2012-12-26-Ask HS: What will programming and architecture look like in 2020?
14 0.65056807 758 high scalability-2010-01-11-Have We Reached the End of Scaling?
15 0.63810706 753 high scalability-2009-12-21-Hot Holiday Scalability Links for 2009
16 0.6371479 1581 high scalability-2014-01-17-Stuff The Internet Says On Scalability For January 17th, 2014
17 0.63308382 1225 high scalability-2012-04-09-Why My Slime Mold is Better than Your Hadoop Cluster
18 0.63186693 839 high scalability-2010-06-09-Paper: Propagation Networks: A Flexible and Expressive Substrate for Computation
19 0.62756616 538 high scalability-2009-03-16-Are Cloud Based Memory Architectures the Next Big Thing?
20 0.62485307 1316 high scalability-2012-09-04-Changing Architectures: New Datacenter Networks Will Set Your Code and Data Free
topicId topicWeight
[(1, 0.112), (2, 0.154), (10, 0.047), (30, 0.019), (40, 0.021), (47, 0.014), (61, 0.07), (77, 0.022), (79, 0.363), (85, 0.026), (94, 0.042)]
simIndex simValue blogId blogTitle
1 0.98699927 871 high scalability-2010-08-04-Dremel: Interactive Analysis of Web-Scale Datasets - Data as a Programming Paradigm
Introduction: If Google was a boxer then MapReduce would be a probing right hand that sets up the massive left hook that is Dremel , Google's—scalable (thousands of CPUs, petabytes of data, trillions of rows), SQL based, columnar, interactive (results returned in seconds), ad-hoc—analytics system. If Google was a magician then MapReduce would be the shiny thing that distracts the mind while the trick goes unnoticed. I say that because even though Dremel has been around internally at Google since 2006, we have not heard a whisper about it. All we've heard about is MapReduce, clones of which have inspired entire new industries. Tricky . Dremel, according to Brian Bershad, Director of Engineering at Google, is targeted at solving BigData class problems : While we all know that systems are huge and will get even huger, the implications of this size on programmability, manageability, power, etc. is hard to comprehend. Alfred noted that the Internet is predicted to be carrying a zetta-byte (10 21
2 0.98479015 1403 high scalability-2013-02-08-Stuff The Internet Says On Scalability For February 8, 2013
Introduction: Hey, it's HighScalability time: 34TB : storage for GitHub search ; 2,880,000,000: log lines per day Quotable Quotes: @peakscale : The " IKEA effec t" << Contributes to NIH and why ppl still like IaaS over PaaS. :-\ @sheeshee : module named kafka.. creates weird & random processes, sends data from here to there & after 3 minutes noone knows what's happening anymore? @sometoomany : Ceased writing a talk about cloud computing infrastructure, and data centre power efficiency. Bored myself to death, but saved others. Larry Kass on aged bourbon : Where it spent those years is as important has how many years it spent. Lots of heat on Is MongoDB's fault tolerance broken? Yes it is . No it's not . YES it is . And the score: MongoDB Is Still Broken by Design 5-0 . Every insurgency must recruit from an existing population which is already affiliated elsewhere. For web properties the easiest group to recru
3 0.98341513 784 high scalability-2010-02-25-Paper: High Performance Scalable Data Stores
Introduction: The world of scalable databases is not a simple one. They come in every race, creed, and color. Rick Cattell has brought some harmony to that world by publishing High Performance Scalable Data Stores , a nicely detailed one stop shop paper comparing scalable databases soley on the content of their character. Ironically, the first step in that evaluation is dividing the world into four groups: Key-value stores: Redis, Scalaris, Voldmort, and Riak. Document stores: Couch DB, MongoDB, and SimpleDB. Record stores: BigTable, HBase, HyperTable, and Cassandra. Scalable RDBMSs: MySQL Cluster, ScaleDB, Drizzle, and VoltDB. The paper describes each system and then compares them on the dimensions of Concurrency Control, Data Storage Replication, Transaction Model, General Comments, Maturity, K-hits, License Language. And the winner is: there are no winners. Yet. Rick concludes by pointing to a great convergence: I believe that a few of these systems will gain critical mass an
4 0.98235762 1169 high scalability-2012-01-05-Shutterfly Saw a Speedup of 500% With Flashcache
Introduction: In the "should I or shouldn't I" debate around deploying SSD, it always helps to have real-world data. Fiesta! with a live-blog summary of a presentation by Kenny Gorman on Shutterfly on MongoDB Performance Tuning . What if you still need more performance after doing all of this tuning? One option is to use SSDs. Shutterfly uses Facebook’s flashcache : kernel module to cache data on SSD. Designed for MySQL/InnoDB. SSD in front of a disk, but exposed as a single mount point. This only makes sense when you have lots of physical I/O. Shutterfly saw a speedup of 500% w/ flashcache. A benefit is that you can delay sharding: less complexity. The whole series of posts has a lot of great information and is worth a longer look, especially if you are considering using MongoDB. Related Articles Slides for MongoSF 2011 slides: MongoDB Performance Tuning SSD+HDD sharding setup for large and permanently growing collections Imlementing MongoDB at Shutterfly by Kenny
5 0.97912097 1277 high scalability-2012-07-05-10 Golden Principles For Building Successful Mobile-Web Applications
Introduction: Wildly popular VC blogger Fred Wilson defines in an excellent 27 minute video the ten most important criteria he uses when deciding to give the gold, that is, fund a web application. Note, this video is from 2010 , so no doubt the ideas are still valid, but the importance of mobile vs web apps has probably shifted to mobile, as Mr. Wilson says in a recent post: mobile is growing like a weed . Speed - speed is more than a feature, it's a requirement. Mainstream users are unforgiving. If something is slow they won't use it. Pingdom is used to track speed across their portfolio. A trend they've noticed is that as an application slows down they don't grow as quickly. Instant Utility - a service must be instantly useful to users. Lengthy setup and configuration is a killer. Tricks like crawling the web to populate information you expect to get from your users later makes the service initially useful. YouTube won, for example, with instant availability of uploaded video.
6 0.97733188 1420 high scalability-2013-03-08-Stuff The Internet Says On Scalability For March 8, 2013
7 0.97586054 1100 high scalability-2011-08-18-Paper: The Akamai Network - 61,000 servers, 1,000 networks, 70 countries
8 0.97274011 1494 high scalability-2013-07-19-Stuff The Internet Says On Scalability For July 19, 2013
9 0.97243369 680 high scalability-2009-08-13-Reconnoiter - Large-Scale Trending and Fault-Detection
10 0.96595562 323 high scalability-2008-05-19-Twitter as a scalability case study
11 0.96587265 107 high scalability-2007-10-02-Some Real Financial Numbers for Your Startup
12 0.96325773 448 high scalability-2008-11-22-Google Architecture
13 0.95840698 650 high scalability-2009-07-02-Product: Hbase
same-blog 14 0.95637935 786 high scalability-2010-03-02-Using the Ambient Cloud as an Application Runtime
15 0.95265144 581 high scalability-2009-04-26-Map-Reduce for Machine Learning on Multicore
16 0.95037133 1181 high scalability-2012-01-25-Google Goes MoreSQL with Tenzing - SQL Over MapReduce
17 0.94797355 1048 high scalability-2011-05-27-Stuff The Internet Says On Scalability For May 27, 2011
18 0.94437718 443 high scalability-2008-11-14-Paper: Pig Latin: A Not-So-Foreign Language for Data Processing
19 0.94290221 1485 high scalability-2013-07-01-PRISM: The Amazingly Low Cost of Using BigData to Know More About You in Under a Minute
20 0.93917382 372 high scalability-2008-08-27-Updating distributed web applications