high_scalability high_scalability-2010 high_scalability-2010-761 knowledge-graph by maker-knowledge-mining
Source: html
Introduction: This is an excerpt from my article Building Super Scalable Systems: Blade Runner Meets Autonomic Computing in the Ambient Cloud. We tend to think compute of resources as residing primarily in datacenters. Given the fast pace of innovation we will likely see compute resources become pervasive. Some will reside in datacenters, but compute resources can be anywhere, not just in the datacenter, we'll actually see the bulk of compute resources live outside of datacenters in the future. Given the diversity of compute resources it's reasonable to assume they won't be homogeneous or conform to a standard API. They will specialize by service. Programmers will have to use those specialized service interfaces to build applications that are adaptive enough to take advantage of whatever leverage they can find, whenever and wherever they can find it. Once found the application will have to reorganize on the fly to use whatever new resources it has found and let go of whatever resources it doe
sentIndex sentText sentNum sentScore
1 We tend to think compute of resources as residing primarily in datacenters. [sent-2, score-0.339]
2 Given the fast pace of innovation we will likely see compute resources become pervasive. [sent-3, score-0.393]
3 Some will reside in datacenters, but compute resources can be anywhere, not just in the datacenter, we'll actually see the bulk of compute resources live outside of datacenters in the future. [sent-4, score-0.804]
4 Given the diversity of compute resources it's reasonable to assume they won't be homogeneous or conform to a standard API. [sent-5, score-0.339]
5 Once found the application will have to reorganize on the fly to use whatever new resources it has found and let go of whatever resources it doesn't have access to anymore. [sent-8, score-0.578]
6 In the past applications were designed to make use of fixed cost resources (racks, servers, SANs, switches, energy etc) that were incrementally increased in largish allotments acquired through an onerous management processes. [sent-28, score-0.722]
7 These free users cost because they use resources in the cloud. [sent-44, score-0.34]
8 If energy costs become a large part of algorithms in the future, for example, then you may be able to make a lot of money by creating a clever algorithm that uses less energy and exploits lower energy resources. [sent-55, score-0.488]
9 The Rise of Ambient Cloud Enabled Markets This image shows what a compute resource market might look like. [sent-61, score-0.358]
10 A rich and varied supply of compute resources on which to base a market is still in the future, but it is under construction. [sent-65, score-0.538]
11 Each of us will have a surprisingly large pool of resources to put into the market: smart phones, smart houses, smart appliances, smart cars, smart prosthetics, PCs, laptops, energy, and so on. [sent-76, score-1.5]
12 Step up a level and companies will have smart buildings, smart fleets, smart networks, smart sensors, datacenters, and so on to contribute into the pool. [sent-77, score-1.016]
13 Step up another level and organizations and governments will be able to contribute smart grids, smart buildings, datacenters, and so on. [sent-78, score-0.508]
14 Contributing X units of resources (CPU, memory, etc) to the Ambient Cloud would entitle you to Y units of resources on the Ambient Cloud. [sent-80, score-0.604]
15 We'll go into more detail on the amount of resources that could be available later, but consider in 5 years some estimate smart phones will have one petabyte of storage, which can store something like 3 billion photos. [sent-84, score-0.619]
16 In a typical market economy resources flow to the better deal. [sent-93, score-0.473]
17 We see markets at work today with algorithmic trading in electronic financial markets. [sent-95, score-0.429]
18 This idea actually revolutionizes how applications function in a way similar to how electronic trading revolutionized the trading floor. [sent-102, score-0.394]
19 Similarly, applications will need to be black-boxed, parameterized, made to process trading information, make decisions, and automatically make trades in the application's architecture. [sent-105, score-0.424]
20 A intermediary infrastructure service, for example, may aggregate memory from 100 million smart phones and make it available for object storage that will allow a 7 billion person friend list to be stored. [sent-115, score-0.395]
wordName wordTfidf (topN-words)
[('smart', 0.254), ('resources', 0.23), ('ambient', 0.223), ('market', 0.126), ('resource', 0.123), ('economy', 0.117), ('spanner', 0.113), ('cost', 0.11), ('compute', 0.109), ('algorithm', 0.106), ('trading', 0.104), ('applications', 0.099), ('trades', 0.097), ('markets', 0.097), ('black', 0.094), ('staggering', 0.092), ('electronic', 0.087), ('financial', 0.087), ('cloud', 0.083), ('money', 0.083), ('energy', 0.079), ('phones', 0.079), ('supply', 0.073), ('service', 0.073), ('meets', 0.073), ('datacenters', 0.072), ('units', 0.072), ('google', 0.072), ('fixed', 0.071), ('largish', 0.071), ('attack', 0.071), ('cpu', 0.068), ('budget', 0.068), ('make', 0.062), ('loss', 0.062), ('offers', 0.062), ('humans', 0.06), ('buildings', 0.059), ('whatever', 0.059), ('super', 0.059), ('specialized', 0.058), ('runner', 0.058), ('latency', 0.058), ('platform', 0.057), ('opens', 0.056), ('broker', 0.056), ('consider', 0.056), ('autonomic', 0.055), ('computation', 0.054), ('see', 0.054)]
simIndex simValue blogId blogTitle
same-blog 1 1.0000005 761 high scalability-2010-01-17-Applications Become Black Boxes Using Markets to Scale and Control Costs
Introduction: This is an excerpt from my article Building Super Scalable Systems: Blade Runner Meets Autonomic Computing in the Ambient Cloud. We tend to think compute of resources as residing primarily in datacenters. Given the fast pace of innovation we will likely see compute resources become pervasive. Some will reside in datacenters, but compute resources can be anywhere, not just in the datacenter, we'll actually see the bulk of compute resources live outside of datacenters in the future. Given the diversity of compute resources it's reasonable to assume they won't be homogeneous or conform to a standard API. They will specialize by service. Programmers will have to use those specialized service interfaces to build applications that are adaptive enough to take advantage of whatever leverage they can find, whenever and wherever they can find it. Once found the application will have to reorganize on the fly to use whatever new resources it has found and let go of whatever resources it doe
Introduction: "But it is not complicated. [There's] just a lot of it." \--Richard Feynmanon how the immense variety of the world arises from simple rules.Contents:Have We Reached the End of Scaling?Applications Become Black Boxes Using Markets to Scale and Control CostsLet's Welcome our Neo-Feudal OverlordsThe Economic Argument for the Ambient CloudWhat Will Kill the Cloud?The Amazing Collective Compute Power of the Ambient CloudUsing the Ambient Cloud as an Application RuntimeApplications as Virtual StatesConclusionWe have not yet begun to scale. The world is still fundamentally disconnected and for all our wisdom we are still in the earliest days of learning how to build truly large planet-scaling applications.Today 350 million users on Facebook is a lot of users and five million followers on Twitter is a lot of followers. This may seem like a lot now, but consider we have no planet wide applications yet. None.Tomorrow the numbers foreshadow a newCambrian explosionof connectivity that will look as
Introduction: All in all this is still my favorite post and I still think it's an accurate vision of a future. Not everyone agrees, but I guess we'll see..."But it is not complicated. [There's] just a lot of it." \--Richard Feynmanon how the immense variety of the world arises from simple rules.Contents:Have We Reached the End of Scaling?Applications Become Black Boxes Using Markets to Scale and Control CostsLet's Welcome our Neo-Feudal OverlordsThe Economic Argument for the Ambient CloudWhat Will Kill the Cloud?The Amazing Collective Compute Power of the Ambient CloudUsing the Ambient Cloud as an Application RuntimeApplications as Virtual StatesConclusionWe have not yet begun to scale. The world is still fundamentally disconnected and for all our wisdom we are still in the earliest days of learning how to build truly large planet-scaling applications.Today 350 million users on Facebook is a lot of users and five million followers on Twitter is a lot of followers. This may seem like a lot now, but c
4 0.32679015 786 high scalability-2010-03-02-Using the Ambient Cloud as an Application Runtime
Introduction: This is an excerpt from my article Building Super Scalable Systems: Blade Runner Meets Autonomic Computing in the Ambient Cloud. The future looks many, big, complex, and adaptive: Many clouds. Many servers. Many operating systems. Many languages. Many storage services. Many database services. Many software services. Many adjunct human networks (like Mechanical Turk). Many fast interconnects. Many CDNs. Many cache memory pools. Many application profiles (simple request-response, live streaming, computationally complex, sensor driven, memory intensive, storage intensive, monolithic, decomposable, etc). Many legal jurisdictions. Don't want to perform a function on Patriot Act "protected" systems then move the function elsewhere. Many SLAs. Many data driven pricing policies that like airplane pricing algorithms will price "seats" to maximize profit using multi-variate time sensitive pricing models. Many competitive products. The need t
5 0.27060553 768 high scalability-2010-02-01-What Will Kill the Cloud?
Introduction: This is an excerpt from my article Building Super Scalable Systems: Blade Runner Meets Autonomic Computing in the Ambient Cloud. If datacenters are the new castles, then what will be the new gunpowder? As soon as gunpowder came on the scene, castles, which are defensive structures, quickly became the future's cold, drafty hotels. Gunpowder fueled cannon balls make short work of castle walls. There's a long history of "gunpowder" type inventions in the tech industry. PCs took out the timeshare model. The cloud is taking out the PC model. There must be something that will take out the cloud. Right now it's hard to believe the cloud will one day be no more. They seem so much the future, but something will transcend the cloud. We even have a law that says so: Bell's Law of Computer Classes which holds that roughly every decade a new, lower priced computer class forms based on a new programming platform, network, and interface resulting in new usage and the establishment of
6 0.26157376 778 high scalability-2010-02-15-The Amazing Collective Compute Power of the Ambient Cloud
7 0.21269114 1240 high scalability-2012-05-07-Startups are Creating a New System of the World for IT
8 0.2090982 538 high scalability-2009-03-16-Are Cloud Based Memory Architectures the Next Big Thing?
9 0.20564218 790 high scalability-2010-03-09-Applications as Virtual States
10 0.20349437 765 high scalability-2010-01-25-Let's Welcome our Neo-Feudal Overlords
11 0.18483368 38 high scalability-2007-07-30-Build an Infinitely Scalable Infrastructure for $100 Using Amazon Services
12 0.18476203 1435 high scalability-2013-04-04-Paper: A Web of Things Application Architecture - Integrating the Real-World into the Web
13 0.18322884 758 high scalability-2010-01-11-Have We Reached the End of Scaling?
14 0.17660905 661 high scalability-2009-07-25-Latency is Everywhere and it Costs You Sales - How to Crush it
15 0.17549869 1112 high scalability-2011-09-07-What Google App Engine Price Changes Say About the Future of Web Architecture
16 0.17316037 853 high scalability-2010-07-08-Cloud AWS Infrastructure vs. Physical Infrastructure
17 0.17273958 920 high scalability-2010-10-15-Troubles with Sharding - What can we learn from the Foursquare Incident?
18 0.16460346 1413 high scalability-2013-02-27-42 Monster Problems that Attack as Loads Increase
19 0.16096015 96 high scalability-2007-09-18-Amazon Architecture
20 0.15636446 1328 high scalability-2012-09-24-Google Spanner's Most Surprising Revelation: NoSQL is Out and NewSQL is In
topicId topicWeight
[(0, 0.341), (1, 0.135), (2, 0.096), (3, 0.143), (4, -0.125), (5, -0.136), (6, 0.019), (7, 0.048), (8, -0.064), (9, -0.031), (10, -0.02), (11, 0.029), (12, -0.024), (13, 0.128), (14, 0.148), (15, -0.027), (16, -0.129), (17, -0.037), (18, 0.033), (19, 0.081), (20, -0.109), (21, 0.07), (22, -0.02), (23, -0.009), (24, 0.087), (25, 0.01), (26, -0.015), (27, 0.022), (28, 0.018), (29, -0.134), (30, -0.076), (31, 0.03), (32, 0.002), (33, -0.039), (34, -0.0), (35, -0.099), (36, -0.083), (37, -0.038), (38, -0.105), (39, -0.029), (40, 0.054), (41, -0.022), (42, 0.031), (43, 0.049), (44, 0.033), (45, 0.004), (46, -0.009), (47, 0.005), (48, -0.001), (49, 0.025)]
simIndex simValue blogId blogTitle
same-blog 1 0.96101725 761 high scalability-2010-01-17-Applications Become Black Boxes Using Markets to Scale and Control Costs
Introduction: This is an excerpt from my article Building Super Scalable Systems: Blade Runner Meets Autonomic Computing in the Ambient Cloud. We tend to think compute of resources as residing primarily in datacenters. Given the fast pace of innovation we will likely see compute resources become pervasive. Some will reside in datacenters, but compute resources can be anywhere, not just in the datacenter, we'll actually see the bulk of compute resources live outside of datacenters in the future. Given the diversity of compute resources it's reasonable to assume they won't be homogeneous or conform to a standard API. They will specialize by service. Programmers will have to use those specialized service interfaces to build applications that are adaptive enough to take advantage of whatever leverage they can find, whenever and wherever they can find it. Once found the application will have to reorganize on the fly to use whatever new resources it has found and let go of whatever resources it doe
Introduction: "But it is not complicated. [There's] just a lot of it." \--Richard Feynmanon how the immense variety of the world arises from simple rules.Contents:Have We Reached the End of Scaling?Applications Become Black Boxes Using Markets to Scale and Control CostsLet's Welcome our Neo-Feudal OverlordsThe Economic Argument for the Ambient CloudWhat Will Kill the Cloud?The Amazing Collective Compute Power of the Ambient CloudUsing the Ambient Cloud as an Application RuntimeApplications as Virtual StatesConclusionWe have not yet begun to scale. The world is still fundamentally disconnected and for all our wisdom we are still in the earliest days of learning how to build truly large planet-scaling applications.Today 350 million users on Facebook is a lot of users and five million followers on Twitter is a lot of followers. This may seem like a lot now, but consider we have no planet wide applications yet. None.Tomorrow the numbers foreshadow a newCambrian explosionof connectivity that will look as
Introduction: All in all this is still my favorite post and I still think it's an accurate vision of a future. Not everyone agrees, but I guess we'll see..."But it is not complicated. [There's] just a lot of it." \--Richard Feynmanon how the immense variety of the world arises from simple rules.Contents:Have We Reached the End of Scaling?Applications Become Black Boxes Using Markets to Scale and Control CostsLet's Welcome our Neo-Feudal OverlordsThe Economic Argument for the Ambient CloudWhat Will Kill the Cloud?The Amazing Collective Compute Power of the Ambient CloudUsing the Ambient Cloud as an Application RuntimeApplications as Virtual StatesConclusionWe have not yet begun to scale. The world is still fundamentally disconnected and for all our wisdom we are still in the earliest days of learning how to build truly large planet-scaling applications.Today 350 million users on Facebook is a lot of users and five million followers on Twitter is a lot of followers. This may seem like a lot now, but c
4 0.90143216 786 high scalability-2010-03-02-Using the Ambient Cloud as an Application Runtime
Introduction: This is an excerpt from my article Building Super Scalable Systems: Blade Runner Meets Autonomic Computing in the Ambient Cloud. The future looks many, big, complex, and adaptive: Many clouds. Many servers. Many operating systems. Many languages. Many storage services. Many database services. Many software services. Many adjunct human networks (like Mechanical Turk). Many fast interconnects. Many CDNs. Many cache memory pools. Many application profiles (simple request-response, live streaming, computationally complex, sensor driven, memory intensive, storage intensive, monolithic, decomposable, etc). Many legal jurisdictions. Don't want to perform a function on Patriot Act "protected" systems then move the function elsewhere. Many SLAs. Many data driven pricing policies that like airplane pricing algorithms will price "seats" to maximize profit using multi-variate time sensitive pricing models. Many competitive products. The need t
5 0.88796467 790 high scalability-2010-03-09-Applications as Virtual States
Introduction: This is an excerpt from my article Building Super Scalable Systems: Blade Runner Meets Autonomic Computing in the Ambient Cloud. As I was writing an article on the architecture of the Storm Botnet , I couldn't help but notice the deep similarity of how Storm works and changes we're seeing in the evolution of political systems. In particular, the rise of the virtual-state . As crazy as this may sound, I think this is also the direction applications will need follow to survive in a complex world of billions of compute devices. You may have already heard of virtual corporations . Virtual corporations are companies with limited office space, a distributed workforce, and production facilities located wherever it is profitable to locate them. The idea is to stay lean and compete using the rapid development and introduction of new products into high value-added markets . If you spot a market opportunity with a small time window, building your own factories and hiring and engine
6 0.86774522 768 high scalability-2010-02-01-What Will Kill the Cloud?
7 0.84782892 778 high scalability-2010-02-15-The Amazing Collective Compute Power of the Ambient Cloud
8 0.84253722 765 high scalability-2010-01-25-Let's Welcome our Neo-Feudal Overlords
10 0.74410915 823 high scalability-2010-05-05-How will memristors change everything?
11 0.72288632 1377 high scalability-2012-12-26-Ask HS: What will programming and architecture look like in 2020?
12 0.72220647 1410 high scalability-2013-02-20-Smart Companies Fail Because they Do Everything Right - Staying Alive to Scale
13 0.70687509 1326 high scalability-2012-09-20-How Vimeo Saves 50% on EC2 by Playing a Smarter Game
14 0.6960876 1091 high scalability-2011-08-02-How Will DIDO Wireless Networking Change Everything?
15 0.69596839 1012 high scalability-2011-03-28-Aztec Empire Strategy: Use Dual Pipes in Your Aqueduct for High Availability
16 0.69309396 758 high scalability-2010-01-11-Have We Reached the End of Scaling?
19 0.6798045 1225 high scalability-2012-04-09-Why My Slime Mold is Better than Your Hadoop Cluster
20 0.66874981 527 high scalability-2009-03-06-Cloud Programming Directly Feeds Cost Allocation Back into Software Design
topicId topicWeight
[(1, 0.132), (2, 0.176), (5, 0.01), (10, 0.048), (25, 0.011), (30, 0.03), (40, 0.015), (47, 0.029), (56, 0.013), (61, 0.059), (71, 0.126), (77, 0.017), (79, 0.156), (85, 0.038), (94, 0.059)]
simIndex simValue blogId blogTitle
1 0.96615392 1610 high scalability-2014-03-11-Douglas Adams - 3 Rules that Describe Our Reactions to Technologies
Introduction: Chris Dixon unearthed a great quote from Douglas Adams on the nature of technological adoption that unsurprisingly hits the mark in our ever changing and evolving world: Anything that is in the world when you’re born is normal and ordinary and is just a natural part of the way the world works. Anything that’s invented between when you’re fifteen and thirty-five is new and exciting and revolutionary and you can probably get a career in it. Anything invented after you’re thirty-five is against the natural order of things Some that come to mind: horse to car, index card to online search, PC to mobile, web to app, portal to messaging, Newton to Einstein, oil to electric, rock to rap, Aquinas to Bacon, buying to renting, files to streaming, network TV to cordkilling, broadcast to social, programming CPUs to programming biology, server to cloud, vm to container, wired to wireless, long read to TLDR, privacy to public to ephemeral, paper based news aggregation to digital a
same-blog 2 0.94230002 761 high scalability-2010-01-17-Applications Become Black Boxes Using Markets to Scale and Control Costs
Introduction: This is an excerpt from my article Building Super Scalable Systems: Blade Runner Meets Autonomic Computing in the Ambient Cloud. We tend to think compute of resources as residing primarily in datacenters. Given the fast pace of innovation we will likely see compute resources become pervasive. Some will reside in datacenters, but compute resources can be anywhere, not just in the datacenter, we'll actually see the bulk of compute resources live outside of datacenters in the future. Given the diversity of compute resources it's reasonable to assume they won't be homogeneous or conform to a standard API. They will specialize by service. Programmers will have to use those specialized service interfaces to build applications that are adaptive enough to take advantage of whatever leverage they can find, whenever and wherever they can find it. Once found the application will have to reorganize on the fly to use whatever new resources it has found and let go of whatever resources it doe
3 0.94196904 1219 high scalability-2012-03-30-Stuff The Internet Says On Scalability For March 30, 2012
Introduction: Choosy Mothers Choose HighScalability: Quotable quotes: @itarradellas : "Revolutions in science have often been preceded by revolutions in measurement" @jasongorman : Use dependency injection, not Spring. Use event-driven, asynchronous I/O, not Node.js. Use MVC, not http://ASP.NET MVC etc etc @bernardgolden : #netflix uses most aggressive #aws reservation system. Gets pricing down to ~ 33% of "list' pricing. @ikarzali : Hey, for all facebook's talk at scalability conferences, I have to say Timeline is super slow(!) Howz that memcache workin out for you now? Yahoo! : Amazon's Game-Changing Cloud Was Built By Some Guys In South Africa Foursquare : 1.5 billion check-ins from 15 million people at 30 million different places. How OMGPOP scaled to 36 million users in three weeks . Draw Something has been downloaded 35+ million times; 1 billion pictures created at 3,000 pictures per second; Couchbase is used as the database; SoftLayer is thei
4 0.90971798 1527 high scalability-2013-10-04-Stuff The Internet Says On Scalability For October 4th, 2013
Introduction: Hey, it's HighScalability time: ( Consumption as a Service : Amazon has 80 1.2 million square foot warehouses.) 100 megapixel cameras, taking 40 million pictures a second, creating of 1 petabyte of data every second: Large Hadron Collider in Higgs search Quotable Quotes: Tim Bell : You can't do transformation in transitional stages @lleung : Also, we could really use a “high scalability” like blog for Enterprise IT to see the interesting stuff xkcd : Functional programming combines the flexibility and power of abstract mathematics with the intuitive clarity of abstract mathematics. @pbailis : That said, just because you can now build distributed transactions doesn't mean they'll be fast or work well during failures. Ever wonder how you can get those shoes and t
5 0.90749478 1502 high scalability-2013-08-16-Stuff The Internet Says On Scalability For August 16, 2013
Introduction: Hey, it's HighScalability time: 1 trillion : edges in Facebook's search graph Quotable Quotes: Miguel de Icaza : Callbacks as our Generations' Go To Statement @kaleidic : "Haskell is a great language as a way of thinking, but I prefer programming in a language where I can cheat."--Meijer T.S. Eliot, who totally got the Internet: Distracted from distraction by distraction The argument eternal: Why Some Startups Say the Cloud Is a Waste of Money . The argument follows a getting back-to-nature pattern. Amazon is the glittering city of compromised values and colos are the familiar places of refuge and virtue. But be honest, don't we all really know the score by now? Fred Wilson: The Similarities Between Building and Scaling a Product and a Company : The system you and your team built will break if you don't keep tweaking it as demand grows. Greg Pass, who was VP Engineering at Twitter during the period where Tw
8 0.90262812 1147 high scalability-2011-11-25-Stuff The Internet Says On Scalability For November 25, 2011
9 0.89984035 1275 high scalability-2012-07-02-C is for Compute - Google Compute Engine (GCE)
10 0.89953446 289 high scalability-2008-03-27-Amazon Announces Static IP Addresses and Multiple Datacenter Operation
11 0.89807504 716 high scalability-2009-10-06-Building a Unique Data Warehouse
12 0.89683735 1630 high scalability-2014-04-11-Stuff The Internet Says On Scalability For April 11th, 2014
13 0.89663219 763 high scalability-2010-01-22-How BuddyPoke Scales on Facebook Using Google App Engine
14 0.896442 1654 high scalability-2014-06-05-Cloud Architecture Revolution
15 0.89481539 129 high scalability-2007-10-23-Hire Facebook, Ning, and Salesforce to Scale for You
16 0.89449906 1612 high scalability-2014-03-14-Stuff The Internet Says On Scalability For March 14th, 2014
17 0.8944931 517 high scalability-2009-02-21-Google AppEngine - A Second Look
18 0.8942852 851 high scalability-2010-07-02-Hot Scalability Links for July 2, 2010
19 0.89412093 1240 high scalability-2012-05-07-Startups are Creating a New System of the World for IT
20 0.8938812 888 high scalability-2010-08-27-OpenStack - The Answer to: How do We Compete with Amazon?