high_scalability high_scalability-2014 high_scalability-2014-1627 knowledge-graph by maker-knowledge-mining
Source: html
Introduction: For years a war has been fought in the software architecture trenches between the ideal of decentralized services and the power and practicality of centralized services. Centralized architectures, at least at the management and control plane level, are winning. And Google not only agrees, they are enthusiastic adopters of this model, even in places you don't think it should work. Here's an excerpt from Google Lifts Veil On “Andromeda” Virtual Networking , an excellent article by Timothy Morgan, that includes a money quote from Amin Vahdat , distinguished engineer and technical lead for networking at Google: Like many of the massive services that Google has created, the Andromeda network has centralized control. By the way, so did the Google File System and the MapReduce scheduler that gave rise to Hadoop when it was mimicked, so did the BigTable NoSQL data store that has spawned a number of quasi-clones, and even the B4 WAN and the Spanner distributed file system that have yet
sentIndex sentText sentNum sentScore
1 For years a war has been fought in the software architecture trenches between the ideal of decentralized services and the power and practicality of centralized services. [sent-1, score-0.973]
2 "What we have seen is that a logically centralized, hierarchical control plane with a peer-to-peer data plane beats full decentralization,” explained Vahdat in his keynote. [sent-6, score-0.368]
3 “All of these flew in the face of conventional wisdom,” he continued, referring to all of those projects above, and added that everyone was shocked back in 2002 that Google would, for instance, build a large-scale storage system like GFS with centralized control. [sent-7, score-0.482]
4 We can build a fundamentally more efficient system by prudently leveraging centralization rather than trying to manage things in a peer-to-peer, decentralized manner. [sent-9, score-0.525]
5 In a software-defined network, a centralized controller has a complete end-to-end view of the entire network, and knowledge of all network paths and device capabilities resides in a single application. [sent-12, score-0.82]
6 As a result, the controller can calculate paths based on both source and destination addresses; use different network paths for different traffic types; and react quickly to changing networking conditions. [sent-13, score-0.621]
7 So a centralized controller knows all and sees all and hardwires routes by directly programming routers. [sent-16, score-0.623]
8 With your own SDN on your hardware failure response times can be immediate, as the centralized controller will program routers with a possibly precalculated alternative route. [sent-18, score-0.682]
9 It's logically centralized , but may be split up among numerous machines as is typical in any service architecture. [sent-22, score-0.554]
10 With today's big iron, big memory, and fast networks the motivation for adopting a completely decentralized architecture for capacity reasons is not compelling except for all but the largest problems. [sent-24, score-0.408]
11 At Internet scale, the Autonomous System model of being logically and physically decentralized is still a win, it can scale wonderfully, but at the price of high coordination costs and slow reactions times. [sent-25, score-0.418]
12 We can see centralization winning in the services that people choose to use. [sent-30, score-0.305]
13 Email and NNTP, both fully decentralized services, while not dead by any means, have given way to centralized services like Twitter, Facebook, G+, WhatsApp, and push notifications. [sent-31, score-0.842]
14 While decentralization plays an important part in the back-end of most every software service, the services themselves are logically centralized. [sent-32, score-0.347]
15 So when you argue for a fully decentralized architecture it's hard to argue based on features or scalability, you have to look elsewhere. [sent-42, score-0.545]
16 Attempts to make a decentralized or federated Twitter service, for example, while technically feasible, have not busted out into general adoption. [sent-44, score-0.29]
17 Doc Searls in articles like Escaping the Black Holes of Centralization makes the case that decentralization is important for human rights and personal sovereignty reasons. [sent-50, score-0.33]
18 A fully distributed and encrypted P2P chat system is a lot harder to compromise than a centralized service run by a large faceless corporation. [sent-51, score-0.482]
19 If your system is smallish then a completely centralized architecture is still quite attractive. [sent-56, score-0.603]
20 For the vast middle ground Google has shown centralized management and control combined with distributed data is probably now the canonical architecture. [sent-57, score-0.512]
wordName wordTfidf (topN-words)
[('centralized', 0.426), ('decentralized', 0.29), ('andromeda', 0.261), ('centralization', 0.235), ('controller', 0.197), ('vahdat', 0.178), ('decentralization', 0.149), ('logically', 0.128), ('google', 0.125), ('networking', 0.125), ('lifts', 0.119), ('systemmodel', 0.119), ('sovereignty', 0.119), ('veil', 0.106), ('paths', 0.102), ('network', 0.095), ('qualities', 0.088), ('sdn', 0.088), ('control', 0.086), ('plane', 0.077), ('services', 0.07), ('war', 0.07), ('argue', 0.069), ('internet', 0.065), ('personal', 0.062), ('architecture', 0.061), ('offs', 0.059), ('nntp', 0.059), ('precalculated', 0.059), ('smallish', 0.059), ('stackgoogle', 0.059), ('explainedby', 0.059), ('timothy', 0.059), ('stashes', 0.059), ('completely', 0.057), ('secondary', 0.057), ('olden', 0.056), ('spawned', 0.056), ('holes', 0.056), ('practicality', 0.056), ('escaping', 0.056), ('flew', 0.056), ('fully', 0.056), ('autonomously', 0.053), ('mitigation', 0.053), ('purpose', 0.053), ('today', 0.052), ('inviting', 0.051), ('dish', 0.051), ('crawls', 0.05)]
simIndex simValue blogId blogTitle
same-blog 1 1.0000001 1627 high scalability-2014-04-07-Google Finds: Centralized Control, Distributed Data Architectures Work Better than Fully Decentralized Architectures
Introduction: For years a war has been fought in the software architecture trenches between the ideal of decentralized services and the power and practicality of centralized services. Centralized architectures, at least at the management and control plane level, are winning. And Google not only agrees, they are enthusiastic adopters of this model, even in places you don't think it should work. Here's an excerpt from Google Lifts Veil On “Andromeda” Virtual Networking , an excellent article by Timothy Morgan, that includes a money quote from Amin Vahdat , distinguished engineer and technical lead for networking at Google: Like many of the massive services that Google has created, the Andromeda network has centralized control. By the way, so did the Google File System and the MapReduce scheduler that gave rise to Hadoop when it was mimicked, so did the BigTable NoSQL data store that has spawned a number of quasi-clones, and even the B4 WAN and the Spanner distributed file system that have yet
2 0.1429984 1256 high scalability-2012-06-04-OpenFlow-SDN is Not a Silver Bullet for Network Scalability
Introduction: Ivan Pepelnjak (CCIE#1354 Emeritus) is Chief Technology Advisor at NIL Data Communications , author of numerous webinars and advanced networking books , and a prolific blogger . He’s focusing on data center and cloud networking, network virtualization, and scalable application design. OpenFlow is an interesting emerging networking technology appearing seemingly out of nowhere with much hype and fanfare in March 2011. More than a year later, there are two commercial products based on OpenFlow ( NEC’s Programmable Flow and Nicira’s Network Virtualization Platform ) and probably less than a dozen production-grade implementations (including Google’s G-Scale network and Indiana University’s campus network ). Is this an expected result for an emerging technology or another case of overhyped technology hitting limits imposed by reality? OpenFlow-based solutions have to overcome numerous problems every emerging technology is facing, in OpenFlow’s case ranging from compatibili
Introduction: All in all this is still my favorite post and I still think it's an accurate vision of a future. Not everyone agrees, but I guess we'll see..."But it is not complicated. [There's] just a lot of it." \--Richard Feynmanon how the immense variety of the world arises from simple rules.Contents:Have We Reached the End of Scaling?Applications Become Black Boxes Using Markets to Scale and Control CostsLet's Welcome our Neo-Feudal OverlordsThe Economic Argument for the Ambient CloudWhat Will Kill the Cloud?The Amazing Collective Compute Power of the Ambient CloudUsing the Ambient Cloud as an Application RuntimeApplications as Virtual StatesConclusionWe have not yet begun to scale. The world is still fundamentally disconnected and for all our wisdom we are still in the earliest days of learning how to build truly large planet-scaling applications.Today 350 million users on Facebook is a lot of users and five million followers on Twitter is a lot of followers. This may seem like a lot now, but c
Introduction: "But it is not complicated. [There's] just a lot of it." \--Richard Feynmanon how the immense variety of the world arises from simple rules.Contents:Have We Reached the End of Scaling?Applications Become Black Boxes Using Markets to Scale and Control CostsLet's Welcome our Neo-Feudal OverlordsThe Economic Argument for the Ambient CloudWhat Will Kill the Cloud?The Amazing Collective Compute Power of the Ambient CloudUsing the Ambient Cloud as an Application RuntimeApplications as Virtual StatesConclusionWe have not yet begun to scale. The world is still fundamentally disconnected and for all our wisdom we are still in the earliest days of learning how to build truly large planet-scaling applications.Today 350 million users on Facebook is a lot of users and five million followers on Twitter is a lot of followers. This may seem like a lot now, but consider we have no planet wide applications yet. None.Tomorrow the numbers foreshadow a newCambrian explosionof connectivity that will look as
5 0.12221688 832 high scalability-2010-05-31-Scalable federated security with Kerberos
Introduction: In my last post , I outlined considerations that need to be taken into account when choosing between a centralized and federated security model. So, how do we implement the chosen model? Based on a real-world case study, I will outline a Kerberos architecture that enables cutting-edge collaborative research through federated sharing of resources. Read more on BigDataMatters.com
6 0.12207472 1240 high scalability-2012-05-07-Startups are Creating a New System of the World for IT
7 0.11552875 1634 high scalability-2014-04-18-Stuff The Internet Says On Scalability For April 18th, 2014
8 0.11159234 448 high scalability-2008-11-22-Google Architecture
9 0.10910419 538 high scalability-2009-03-16-Are Cloud Based Memory Architectures the Next Big Thing?
10 0.10838252 96 high scalability-2007-09-18-Amazon Architecture
11 0.1079669 1140 high scalability-2011-11-10-Kill the Telcos Save the Internet - The Unsocial Network
12 0.10712632 910 high scalability-2010-09-30-Facebook and Site Failures Caused by Complex, Weakly Interacting, Layered Systems
13 0.10322931 1328 high scalability-2012-09-24-Google Spanner's Most Surprising Revelation: NoSQL is Out and NewSQL is In
14 0.10303231 517 high scalability-2009-02-21-Google AppEngine - A Second Look
15 0.10086551 1559 high scalability-2013-12-06-Stuff The Internet Says On Scalability For December 6th, 2013
16 0.096543424 1110 high scalability-2011-09-06-Big Data Application Platform
17 0.096039556 1316 high scalability-2012-09-04-Changing Architectures: New Datacenter Networks Will Set Your Code and Data Free
18 0.091732174 1177 high scalability-2012-01-19-Is it time to get rid of the Linux OS model in the cloud?
19 0.091044478 1502 high scalability-2013-08-16-Stuff The Internet Says On Scalability For August 16, 2013
20 0.090868078 761 high scalability-2010-01-17-Applications Become Black Boxes Using Markets to Scale and Control Costs
topicId topicWeight
[(0, 0.19), (1, 0.075), (2, 0.025), (3, 0.078), (4, -0.018), (5, -0.027), (6, 0.004), (7, 0.03), (8, -0.006), (9, 0.033), (10, -0.007), (11, 0.022), (12, -0.021), (13, -0.023), (14, 0.049), (15, 0.014), (16, -0.029), (17, -0.015), (18, 0.02), (19, -0.029), (20, 0.028), (21, 0.045), (22, 0.002), (23, -0.006), (24, 0.006), (25, 0.04), (26, -0.016), (27, 0.01), (28, -0.041), (29, 0.005), (30, -0.031), (31, -0.046), (32, 0.016), (33, -0.032), (34, -0.016), (35, 0.023), (36, 0.001), (37, 0.009), (38, 0.023), (39, 0.07), (40, 0.027), (41, -0.016), (42, 0.012), (43, -0.017), (44, -0.038), (45, 0.022), (46, -0.021), (47, -0.024), (48, -0.07), (49, 0.027)]
simIndex simValue blogId blogTitle
same-blog 1 0.97166938 1627 high scalability-2014-04-07-Google Finds: Centralized Control, Distributed Data Architectures Work Better than Fully Decentralized Architectures
Introduction: For years a war has been fought in the software architecture trenches between the ideal of decentralized services and the power and practicality of centralized services. Centralized architectures, at least at the management and control plane level, are winning. And Google not only agrees, they are enthusiastic adopters of this model, even in places you don't think it should work. Here's an excerpt from Google Lifts Veil On “Andromeda” Virtual Networking , an excellent article by Timothy Morgan, that includes a money quote from Amin Vahdat , distinguished engineer and technical lead for networking at Google: Like many of the massive services that Google has created, the Andromeda network has centralized control. By the way, so did the Google File System and the MapReduce scheduler that gave rise to Hadoop when it was mimicked, so did the BigTable NoSQL data store that has spawned a number of quasi-clones, and even the B4 WAN and the Spanner distributed file system that have yet
2 0.83830816 284 high scalability-2008-03-19-RAD Lab is Creating a Datacenter Operating System
Introduction: The RAD Lab (Reliable Adaptive Distributed Systems Laboratory) wants to leapfrog the Big Switch and create The Next Big Switch, skipping the cloud/utility evolutionary stage altogether. This hyper-evolutionary niche buster develops technology so advanced the cloud disperses and you can go back to building your own personal datacenters again. Where Google took years to create their datacenters, using a prefab Datacenter Operating System you might create your own in a long holiday weekend. Not St. Patrick's of course. Their vision: Enable one person to invent and run the next revolutionary IT service, operationally expressing a new business idea as a multi-million-user service over the course of a long weekend. By doing so we hope to enable an Internet "Fortune 1 million". How? By wizardry in the form of a “datacenter operating system” created from a pinch of "statistical machine learning (SML)" and a tincture of "recent insights from networking and distributed systems." Bu
Introduction: If you were going to design a next generation Internet at the physical layer that routes around the current Internet, what would it look like? What should it do? How should it work? Who should own it? How should it be paid for? How would you access it? It has long been said the Internet routes around obstacles. Snowden has revealed some major obstacles. The beauty of the current current app and web system is the physical network doesn't matter. We can just replace it with something else. Something that doesn't flow through choke points like backhaul networks , under sea cables , and cell towers . What might that something else look like? Google's Loon Project Project Loon was so named because the idea was thought to be loony. Maybe not. The idea is to float high-altitude balloons 20 miles in the air to create an aerial wireless network with up to 3G-like speeds. Signals travel through the balloon network from balloon to balloon, then to a ground-based station conne
4 0.78855282 1634 high scalability-2014-04-18-Stuff The Internet Says On Scalability For April 18th, 2014
Introduction: Hey, it's HighScalability time: Scaling to the top of "Bun Mountain" in Hong Kong 44 trillion gigabytes : size of the digital universe by 2020; 6 Times : we have six times more "stuff" than the generation before us. Quotable Quotes: Facebook : Our warehouse stores upwards of 300 PB of Hive data, with an incoming daily rate of about 600 TB. @windley : The problem with the Internet of Things is right now it’s more like the CompuServe of Things Chip Overclock : If you want to eventually generate revenue, you must first optimize for developer productivity; everything else is negotiable. @igrigorik : if you are gzipping your images.. you're doing it wrong: http://bit.ly/1pb8JhK - check your server configs! and your CDN... :) Seth Lloyd : The arrow of time is an arrow of increasing correlations. @kitmacgillivray : When will Google enable / require all android apps to have full deep search integration so all content is
5 0.78663671 1256 high scalability-2012-06-04-OpenFlow-SDN is Not a Silver Bullet for Network Scalability
Introduction: Ivan Pepelnjak (CCIE#1354 Emeritus) is Chief Technology Advisor at NIL Data Communications , author of numerous webinars and advanced networking books , and a prolific blogger . He’s focusing on data center and cloud networking, network virtualization, and scalable application design. OpenFlow is an interesting emerging networking technology appearing seemingly out of nowhere with much hype and fanfare in March 2011. More than a year later, there are two commercial products based on OpenFlow ( NEC’s Programmable Flow and Nicira’s Network Virtualization Platform ) and probably less than a dozen production-grade implementations (including Google’s G-Scale network and Indiana University’s campus network ). Is this an expected result for an emerging technology or another case of overhyped technology hitting limits imposed by reality? OpenFlow-based solutions have to overcome numerous problems every emerging technology is facing, in OpenFlow’s case ranging from compatibili
6 0.78522658 446 high scalability-2008-11-18-Scalability Perspectives #2: Van Jacobson – Content-Centric Networking
7 0.78043795 618 high scalability-2009-06-05-Google Wave Architecture
8 0.77402014 1651 high scalability-2014-05-20-It's Networking. In Space! Or How E.T. Will Phone Home.
9 0.76990545 1010 high scalability-2011-03-24-Strategy: Disk Backup for Speed, Tape Backup to Save Your Bacon, Just Ask Google
11 0.76128674 1140 high scalability-2011-11-10-Kill the Telcos Save the Internet - The Unsocial Network
12 0.76070404 1091 high scalability-2011-08-02-How Will DIDO Wireless Networking Change Everything?
13 0.75908792 1552 high scalability-2013-11-22-Stuff The Internet Says On Scalability For November 22th, 2013
14 0.75890207 1499 high scalability-2013-08-09-Stuff The Internet Says On Scalability For August 9, 2013
15 0.75817573 1315 high scalability-2012-08-30-Stuff The Internet Says On Scalability For August 31, 2012
16 0.75377548 1479 high scalability-2013-06-21-Stuff The Internet Says On Scalability For June 21, 2013
17 0.75348294 1630 high scalability-2014-04-11-Stuff The Internet Says On Scalability For April 11th, 2014
18 0.75188923 1502 high scalability-2013-08-16-Stuff The Internet Says On Scalability For August 16, 2013
19 0.74680507 1464 high scalability-2013-05-24-Stuff The Internet Says On Scalability For May 24, 2013
20 0.74399233 815 high scalability-2010-04-27-Paper: Dapper, Google's Large-Scale Distributed Systems Tracing Infrastructure
topicId topicWeight
[(1, 0.111), (2, 0.235), (7, 0.14), (10, 0.042), (26, 0.017), (30, 0.019), (40, 0.041), (47, 0.017), (51, 0.014), (61, 0.046), (77, 0.034), (79, 0.126), (85, 0.036), (94, 0.052)]
simIndex simValue blogId blogTitle
1 0.94292712 964 high scalability-2010-12-28-Netflix: Continually Test by Failing Servers with Chaos Monkey
Introduction: In 5 Lessons We’ve Learned Using AWS , Netflix's John Ciancutti says the best way to avoid failure is to fail constantly . In the cloud it's expected instances can fail at any time, so you always have to be prepared. In the real world we prepare by running drills. Remember all those exciting fire drills? It's not just fire drills of course. The military, football teams, fire fighters, beach rescue, virtually any entity that must react quickly and efficiently to disaster hones their responsiveness by running drills. Netflix aggressively moves this strategy into the cloud by randomly failing servers using a tool they built called Chaos Monkey. The idea is: If we aren’t constantly testing our ability to succeed despite failure, then it isn’t likely to work when it matters most – in the event of an unexpected outage. They respond to failures by degrading service , but they always respond: If the recommendations system is down they'll show popular titles instead. If the s
same-blog 2 0.9358176 1627 high scalability-2014-04-07-Google Finds: Centralized Control, Distributed Data Architectures Work Better than Fully Decentralized Architectures
Introduction: For years a war has been fought in the software architecture trenches between the ideal of decentralized services and the power and practicality of centralized services. Centralized architectures, at least at the management and control plane level, are winning. And Google not only agrees, they are enthusiastic adopters of this model, even in places you don't think it should work. Here's an excerpt from Google Lifts Veil On “Andromeda” Virtual Networking , an excellent article by Timothy Morgan, that includes a money quote from Amin Vahdat , distinguished engineer and technical lead for networking at Google: Like many of the massive services that Google has created, the Andromeda network has centralized control. By the way, so did the Google File System and the MapReduce scheduler that gave rise to Hadoop when it was mimicked, so did the BigTable NoSQL data store that has spawned a number of quasi-clones, and even the B4 WAN and the Spanner distributed file system that have yet
3 0.92842305 1397 high scalability-2013-02-01-Stuff The Internet Says On Scalability For February 1, 2013
Introduction: Hey, it's HighScalability time: 400 milliseconds : modern private wire for Natural Gas ; 620,000: size of a mobile phone botnet ; 1 million cores: jet noise modeling supercomputer ; 2.2 petabytes per gram: DNA with storage density ; 1 billion: Google's one quarter infrastructure spend Quotable Quotes: @StartupLJackson : "Good thing we spent 6mo on scalability pre-launch. The thing went hyper-viral day one, just as planned." - nobody ever. @dysinger : WELCOME TO DEVOPS ADVENTURE! YOU ARE STANDING INSIDE AWS. NEARBY IS AN ANGRY ELB. THERE ARE SOME SSH KEYS ON THE GROUND. ~@levie : Well cloud computing was fun while it lasted; now we can just store all our data in DNA strands. @mtnygard : Debates about “Tech X > tech T” aren’t religious wars. They’re also economic wars. @alfonsol : I do not believe your per load number of socks requires scalability. @mikeolson : Latency breeds contempt. @DrQz : GitHub looks to me like pow
4 0.92814124 68 high scalability-2007-08-20-TypePad Architecture
Introduction: TypePad is considered the largest paid blogging service in the world. After experience problems because of their meteoric growth, they eventually transitioned to an architecture patterned after their sister company, LiveJournal. Site: http://www.typepad.com/ The Platform MySQL Memcached Perl MogileFS Apache Linux The Stats As of 2005 TypePad sends 250mbps of traffic using multiple network pipes for 3TB of traffic a day. They were growing by 10-20% each month. I was unable to find more recent statistics. The Architecture Original Architecture: - Single server running Linux, Apache, Postgres, Perl, mod_perl - Storage was NFS on a filer. A Devastating Crash Caused a New Direction - A RAID controller failed and spewed data across all RAID disks. - The database was corrupted and the backups were corrupted. - Their redundant filers suffered from "split brain" syndrome. They move to LiveJournal Architecture type architecture which isn't surprising
5 0.90543872 518 high scalability-2009-02-22-Building and Scaling a Startup on Rails: 12 Things We Learned the Hard Way
Introduction: Garry Tan, cofounder of Posterous, lists 12 lessons for scaling that apply to more than just Rails. Use cloud storage for static files. Use HTTP Cache Control to tell the browser what it can cache. Use Sphinx for text search. Use InnoDB for more crash resistant and faster writes. Don't use textbook Rails ActiveRecord objects. Use New Relic to find exactly what is slow in your system. Use memcache later so you find your database bottlenecks now. Use mongrel proctitle to find your slow queries. You are only as fast as your slowest queries. Use asynchronous job queuing to do work in parallel. Use monitoring so you'll know when your site went down and why. Learn by reading the source code, fixing problems, and submitting them back to the community. Use new plugins. Old plugins can't be trusted. Use new information. Old information can't be trusted.
6 0.90238476 1267 high scalability-2012-06-18-The Clever Ways Chrome Hides Latency by Anticipating Your Every Need
7 0.89941567 1612 high scalability-2014-03-14-Stuff The Internet Says On Scalability For March 14th, 2014
8 0.8993876 325 high scalability-2008-05-25-How do you explain cloud computing to your grandma?
10 0.89905959 1044 high scalability-2011-05-19-Zynga's Z Cloud - Scale Fast or Fail Fast by Merging Private and Public Clouds
11 0.8983463 1065 high scalability-2011-06-21-Running TPC-C on MySQL-RDS
12 0.89540404 1602 high scalability-2014-02-26-The WhatsApp Architecture Facebook Bought For $19 Billion
13 0.89361906 1545 high scalability-2013-11-08-Stuff The Internet Says On Scalability For November 8th, 2013
14 0.89273328 636 high scalability-2009-06-23-Learn How to Exploit Multiple Cores for Better Performance and Scalability
15 0.89259273 789 high scalability-2010-03-05-Strategy: Planning for a Power Outage Google Style
16 0.89090377 1460 high scalability-2013-05-17-Stuff The Internet Says On Scalability For May 17, 2013
17 0.89081806 1436 high scalability-2013-04-05-Stuff The Internet Says On Scalability For April 5, 2013
18 0.89057606 1112 high scalability-2011-09-07-What Google App Engine Price Changes Say About the Future of Web Architecture
19 0.89034802 274 high scalability-2008-03-12-YouTube Architecture
20 0.89034212 1589 high scalability-2014-02-03-How Google Backs Up the Internet Along With Exabytes of Other Data