high_scalability high_scalability-2010 high_scalability-2010-789 knowledge-graph by maker-knowledge-mining
Source: html
Introduction: We can all learn from problems. The Google App Engine team has created a teachable moment through a remarkably honest and forthcoming post-mortem for February 24th, 2010 outage post, chronicling in elaborate detail a power outage that took down Google App Engine for a few hours. The world is ending! The cloud is unreliable! Jump ship! Not. This is not evidence that the cloud is a beautiful, powerful and unsinkable ship that goes down on its maiden voyage. Stuff happens, no matter how well you prepare. If you think private datacenters don't go down, well, then I have some rearangeable deck chairs to sell you. The goal is to keep improving and minimizing those failure windows. From that perspective there is a lot to learn from the problems the Google App Engine team encountered and how they plan to fix them. Please read the article for all the juicy details, but here's what struck me as key: Power fails. Plan for it . This seems to happen with unexpected frequency for su
sentIndex sentText sentNum sentScore
1 The Google App Engine team has created a teachable moment through a remarkably honest and forthcoming post-mortem for February 24th, 2010 outage post, chronicling in elaborate detail a power outage that took down Google App Engine for a few hours. [sent-2, score-0.772]
2 From that perspective there is a lot to learn from the problems the Google App Engine team encountered and how they plan to fix them. [sent-11, score-0.382]
3 This seems to happen with unexpected frequency for such expensive and well tended systems. [sent-14, score-0.311]
4 The point here is to consider how a complex distributed system responds to power failures, not dissecting the merits of various diesel powered backup systems. [sent-15, score-0.314]
5 It's quite common when disaster planning to assume single independent failures when creating scenarios of doom. [sent-18, score-0.319]
6 Developers can be responsible for dealing with problems or a separate op team can be responsible. [sent-28, score-0.439]
7 GAE apparently uses a separate ops team, which is nice because it means no beepers for developers. [sent-29, score-0.348]
8 When a separate ops team is involved it requires a very complicated set of procedures be put in place so people who know nothing about the software can manage it. [sent-31, score-0.689]
9 This process broke down and the ops team didn't know what to do when unexpected problems arose. [sent-33, score-0.693]
10 When you read it took 2 hours to reach "An engineer with familiarity with the unplanned failover procedure" , that's not good. [sent-34, score-0.225]
11 Software evolves like 10 times faster than ops interactions, so something was left undone. [sent-36, score-0.334]
12 Software releases need to become much more structured in their handoffs to the ops team so that these problems have less of a chance of happening. [sent-37, score-0.638]
13 Soldiers endlessly drill precisely so they can deal with emergency situations without thinking. [sent-40, score-0.21]
14 Weren't there higher level alarms indicating which systems were failing? [sent-45, score-0.231]
15 There's a line of thought that says you can assume partitioning can never happen in a datacenter. [sent-49, score-0.223]
16 The network infrastructure should just work and you can assume all or nothing reliability. [sent-50, score-0.215]
17 Not all of their machines failed which must have confused the failure detection and failover systems. [sent-52, score-0.265]
18 Failover is an all or nothing situation, which makes the failover systems a single point of failure. [sent-55, score-0.242]
19 Failing over smaller units of services can be rmore obust, but also has more failure scenarios involved. [sent-56, score-0.198]
20 GAE has chosen to take the high road and be honest and forthcoming with the problems they've had and how they plan to fix them. [sent-61, score-0.616]
wordName wordTfidf (topN-words)
[('gae', 0.374), ('ops', 0.266), ('honest', 0.239), ('team', 0.182), ('failover', 0.155), ('unexpected', 0.144), ('drill', 0.133), ('assume', 0.128), ('failure', 0.11), ('ship', 0.11), ('procedure', 0.108), ('failures', 0.103), ('failing', 0.102), ('problems', 0.101), ('plan', 0.099), ('black', 0.098), ('chosen', 0.097), ('happen', 0.095), ('outage', 0.094), ('chairs', 0.089), ('dissecting', 0.089), ('handoffs', 0.089), ('scenarios', 0.088), ('nothing', 0.087), ('mud', 0.083), ('alarms', 0.083), ('chronicling', 0.083), ('separate', 0.082), ('engine', 0.081), ('higher', 0.08), ('forthcoming', 0.08), ('merits', 0.08), ('mttf', 0.08), ('mttr', 0.08), ('swan', 0.08), ('diesel', 0.077), ('endlessly', 0.077), ('correlate', 0.077), ('op', 0.074), ('clue', 0.074), ('soldiers', 0.074), ('tended', 0.072), ('software', 0.072), ('unplanned', 0.07), ('balls', 0.07), ('backup', 0.068), ('app', 0.068), ('indicating', 0.068), ('evolves', 0.068), ('datacenter', 0.067)]
simIndex simValue blogId blogTitle
same-blog 1 0.99999988 789 high scalability-2010-03-05-Strategy: Planning for a Power Outage Google Style
Introduction: We can all learn from problems. The Google App Engine team has created a teachable moment through a remarkably honest and forthcoming post-mortem for February 24th, 2010 outage post, chronicling in elaborate detail a power outage that took down Google App Engine for a few hours. The world is ending! The cloud is unreliable! Jump ship! Not. This is not evidence that the cloud is a beautiful, powerful and unsinkable ship that goes down on its maiden voyage. Stuff happens, no matter how well you prepare. If you think private datacenters don't go down, well, then I have some rearangeable deck chairs to sell you. The goal is to keep improving and minimizing those failure windows. From that perspective there is a lot to learn from the problems the Google App Engine team encountered and how they plan to fix them. Please read the article for all the juicy details, but here's what struck me as key: Power fails. Plan for it . This seems to happen with unexpected frequency for su
2 0.31677741 1112 high scalability-2011-09-07-What Google App Engine Price Changes Say About the Future of Web Architecture
Introduction: When I was a child, I spake as a child, I understood as a child, I thought as a child: but when I became a man, I put away childish things . -- Corinthians With this new pricing, developments will be driven by the costs . I like to optimize my apps to make them better or faster, but to optimize them just to make them cheaper is a waste of time. -- Sylvain on Google Groups The dream is dead. Google App Engine's bold pay for what you use dream dies as it leaves childish things behind and becomes a real product . Pricing will change . Architectures will change. Customers will change. Hearts and minds will change. But Google App Engine will survive. Google is shutting down many of its projects . GAE is not among them. Do we have GAE's pricing change to thank for it surving the more wood behind more deadly arrows push? Without a radical and quick shift towards profitably GAE would no doubt be a historical footnote in the long scroll of good ideas. The urgency involve
3 0.26224872 517 high scalability-2009-02-21-Google AppEngine - A Second Look
Introduction: Update 6: : Back to the Future for Data Storage . We are in the middle of a renaissance in data storage with the application of many new ideas and techniques; there's huge potential for breaking out of thinking about data storage in just one way. Update 5 : Building Scalable Web Applications with Google App Engine by Brett Slatkin. Update 4 : Why Google App Engine is broken and what Google must do to fix it by Aral Balkan. We don't care that it can scale. We care that it does scale. And that it scales when you need it the most. Issues: 1MB limit on data structures; 1MB limit on data structures; the short-term high CPU quota; quotas in general; Admin? What's that? Update 3 : BigTable Blues . Catherine Devlin couldn't port an application to GAE because it can't do basic filtering and can't search 5,000 records without timing out: "Querying from 5000 records - too much for the mighty BigTable, apparently." Followup: not the future database . "90% of the work of this proje
4 0.17626688 1432 high scalability-2013-04-01-Khan Academy Checkbook Scaling to 6 Million Users a Month on GAE
Introduction: Khan Academy is a non profit company started by Salman Khan with the Big Hairy Audacious Goal of providing a free, world class education to anyone, anywhere, anytime. That’s a lot of knowledge. Having long been inspired and captivated by the Khan Academy, I was really curious to know how they plan to do it. Ben Kamens , lead developer at Khan Academy, gives the somewhat surprising answer in an interview: How to Scale your Startup to Millions of Users . The short answer : develop a strong team, focus on features, let Google App Engine do the heavy lifting. Some people seem to be turned off by all the GAE love in the interview. Part of it is that the interviewer is Fred Sauer, Developer Advocate for Google App Engine, so there’s a level of familiarity between the two. But the biggest part is simply that they really like GAE, for all the reasons your are supposed to like GAE. And that’s OK. In this day and age you are free to love whichever platform you choose. Bigge
5 0.15767416 1023 high scalability-2011-04-14-Strategy: Cache Application Start State to Reduce Spin-up Times
Introduction: Using this strategy, Valyala, a commenter on Are Long VM Instance Spin-Up Times In The Cloud Costing You Money? , was able to reduce their GAE application start-up times from 15 seconds down to to 1.5 seconds: Spin-up time for newly added Google AppEngine instances can be reduced using initial state caching. Usually the majority of spin-up time for the newly created GAE instance is spent in the pre-populating of the initial state, which is created from many data pieces loaded from slow data sources such as GAE's datastore. If the initial state is identical among GAE instances, then the entire state can be serialized and stored in a shared memory (either in the memcache or in the datastore) by the first created instance, so newly created instances could load and quickly unserialize the state from a single blob loaded from shared memory instead of spending a lot of time for creation of the state from multiple data pieces loaded from the datastore. I reduced spin-up time for new in
6 0.15577987 1055 high scalability-2011-06-08-Stuff to Watch from Google IO 2011
11 0.13087668 687 high scalability-2009-08-24-How Google Serves Data from Multiple Datacenters
12 0.12708688 1240 high scalability-2012-05-07-Startups are Creating a New System of the World for IT
13 0.12650314 763 high scalability-2010-01-22-How BuddyPoke Scales on Facebook Using Google App Engine
14 0.12467467 1098 high scalability-2011-08-15-Should any cloud be considered one availability zone? The Amazon experience says yes.
15 0.12354722 327 high scalability-2008-05-27-How I Learned to Stop Worrying and Love Using a Lot of Disk Space to Scale
16 0.12316065 1508 high scalability-2013-08-28-Sean Hull's 20 Biggest Bottlenecks that Reduce and Slow Down Scalability
17 0.12244576 1135 high scalability-2011-10-31-15 Ways to Make Your Application Feel More Responsive under Google App Engine
18 0.12087884 776 high scalability-2010-02-12-Hot Scalability Links for February 12, 2010
topicId topicWeight
[(0, 0.195), (1, 0.09), (2, -0.008), (3, 0.066), (4, 0.0), (5, -0.078), (6, 0.03), (7, -0.011), (8, 0.051), (9, -0.111), (10, -0.075), (11, -0.013), (12, -0.023), (13, -0.003), (14, 0.045), (15, -0.024), (16, -0.066), (17, -0.044), (18, 0.044), (19, 0.02), (20, 0.063), (21, -0.013), (22, 0.036), (23, -0.099), (24, -0.052), (25, 0.063), (26, 0.045), (27, -0.021), (28, -0.023), (29, -0.01), (30, -0.018), (31, -0.091), (32, 0.052), (33, -0.038), (34, -0.028), (35, 0.044), (36, -0.016), (37, 0.052), (38, 0.05), (39, -0.026), (40, -0.021), (41, -0.064), (42, -0.019), (43, 0.043), (44, 0.088), (45, -0.011), (46, -0.022), (47, 0.127), (48, 0.049), (49, -0.04)]
simIndex simValue blogId blogTitle
same-blog 1 0.96427155 789 high scalability-2010-03-05-Strategy: Planning for a Power Outage Google Style
Introduction: We can all learn from problems. The Google App Engine team has created a teachable moment through a remarkably honest and forthcoming post-mortem for February 24th, 2010 outage post, chronicling in elaborate detail a power outage that took down Google App Engine for a few hours. The world is ending! The cloud is unreliable! Jump ship! Not. This is not evidence that the cloud is a beautiful, powerful and unsinkable ship that goes down on its maiden voyage. Stuff happens, no matter how well you prepare. If you think private datacenters don't go down, well, then I have some rearangeable deck chairs to sell you. The goal is to keep improving and minimizing those failure windows. From that perspective there is a lot to learn from the problems the Google App Engine team encountered and how they plan to fix them. Please read the article for all the juicy details, but here's what struck me as key: Power fails. Plan for it . This seems to happen with unexpected frequency for su
2 0.76563227 1432 high scalability-2013-04-01-Khan Academy Checkbook Scaling to 6 Million Users a Month on GAE
Introduction: Khan Academy is a non profit company started by Salman Khan with the Big Hairy Audacious Goal of providing a free, world class education to anyone, anywhere, anytime. That’s a lot of knowledge. Having long been inspired and captivated by the Khan Academy, I was really curious to know how they plan to do it. Ben Kamens , lead developer at Khan Academy, gives the somewhat surprising answer in an interview: How to Scale your Startup to Millions of Users . The short answer : develop a strong team, focus on features, let Google App Engine do the heavy lifting. Some people seem to be turned off by all the GAE love in the interview. Part of it is that the interviewer is Fred Sauer, Developer Advocate for Google App Engine, so there’s a level of familiarity between the two. But the biggest part is simply that they really like GAE, for all the reasons your are supposed to like GAE. And that’s OK. In this day and age you are free to love whichever platform you choose. Bigge
3 0.76425254 517 high scalability-2009-02-21-Google AppEngine - A Second Look
Introduction: Update 6: : Back to the Future for Data Storage . We are in the middle of a renaissance in data storage with the application of many new ideas and techniques; there's huge potential for breaking out of thinking about data storage in just one way. Update 5 : Building Scalable Web Applications with Google App Engine by Brett Slatkin. Update 4 : Why Google App Engine is broken and what Google must do to fix it by Aral Balkan. We don't care that it can scale. We care that it does scale. And that it scales when you need it the most. Issues: 1MB limit on data structures; 1MB limit on data structures; the short-term high CPU quota; quotas in general; Admin? What's that? Update 3 : BigTable Blues . Catherine Devlin couldn't port an application to GAE because it can't do basic filtering and can't search 5,000 records without timing out: "Querying from 5000 records - too much for the mighty BigTable, apparently." Followup: not the future database . "90% of the work of this proje
4 0.74308807 1112 high scalability-2011-09-07-What Google App Engine Price Changes Say About the Future of Web Architecture
Introduction: When I was a child, I spake as a child, I understood as a child, I thought as a child: but when I became a man, I put away childish things . -- Corinthians With this new pricing, developments will be driven by the costs . I like to optimize my apps to make them better or faster, but to optimize them just to make them cheaper is a waste of time. -- Sylvain on Google Groups The dream is dead. Google App Engine's bold pay for what you use dream dies as it leaves childish things behind and becomes a real product . Pricing will change . Architectures will change. Customers will change. Hearts and minds will change. But Google App Engine will survive. Google is shutting down many of its projects . GAE is not among them. Do we have GAE's pricing change to thank for it surving the more wood behind more deadly arrows push? Without a radical and quick shift towards profitably GAE would no doubt be a historical footnote in the long scroll of good ideas. The urgency involve
5 0.74278063 1055 high scalability-2011-06-08-Stuff to Watch from Google IO 2011
Introduction: With the Google IO Developer Conference completed there are dozens and dozens of information packed videos now available. While you won't get any of the nifty free swag the attendees rake in, it's great of Google to make these videos available so quickly after the conference. Let's say you don't want to watch all the videos on the pretense you have a life, here are just a dozen scalability and architecture related videos you might find interesting: App Engine Backends by Justin Haugh, Greg Darke. One of this biggest complaints about GAE are the request deadlines. Those are now gone when you rent a new Backend node. Scaling App Engine Applications by Justin Haugh, Guido van Rossum. Two parts: how App Engine scales internally and how you as a programmer make it scale using the tools. Good discussion of scaling and why it's hard; 10,000 queries per second is a large app that needs good architecting; a good discussion of GAE's predictive scaling formula of how it decides w
6 0.69094062 763 high scalability-2010-01-22-How BuddyPoke Scales on Facebook Using Google App Engine
7 0.67555475 23 high scalability-2007-07-24-Major Websites Down: Or Why You Want to Run in Two or More Data Centers.
8 0.67143679 527 high scalability-2009-03-06-Cloud Programming Directly Feeds Cost Allocation Back into Software Design
9 0.66718817 1209 high scalability-2012-03-14-The Azure Outage: Time Is a SPOF, Leap Day Doubly So
10 0.66302752 301 high scalability-2008-04-08-Google AppEngine - A First Look
11 0.65120882 1084 high scalability-2011-07-22-Stuff The Internet Says On Scalability For July 22, 2011
12 0.64199442 1030 high scalability-2011-04-27-Heroku Emergency Strategy: Incident Command System and 8 Hour Ops Rotations for Fresh Minds
13 0.63481724 1010 high scalability-2011-03-24-Strategy: Disk Backup for Speed, Tape Backup to Save Your Bacon, Just Ask Google
14 0.63315135 1472 high scalability-2013-06-07-Stuff The Internet Says On Scalability For June 7, 2013
15 0.63032067 1626 high scalability-2014-04-04-Stuff The Internet Says On Scalability For April 4th, 2014
16 0.62057877 1027 high scalability-2011-04-20-Packet Pushers: How to Build a Low Cost Data Center
17 0.61425734 1464 high scalability-2013-05-24-Stuff The Internet Says On Scalability For May 24, 2013
18 0.60284829 1117 high scalability-2011-09-16-Stuff The Internet Says On Scalability For September 16, 2011
20 0.59800333 1624 high scalability-2014-04-01-The Mullet Cloud Selection Pattern
topicId topicWeight
[(1, 0.128), (2, 0.2), (10, 0.059), (47, 0.017), (60, 0.123), (61, 0.05), (77, 0.039), (79, 0.137), (85, 0.061), (94, 0.098)]
simIndex simValue blogId blogTitle
same-blog 1 0.9367696 789 high scalability-2010-03-05-Strategy: Planning for a Power Outage Google Style
Introduction: We can all learn from problems. The Google App Engine team has created a teachable moment through a remarkably honest and forthcoming post-mortem for February 24th, 2010 outage post, chronicling in elaborate detail a power outage that took down Google App Engine for a few hours. The world is ending! The cloud is unreliable! Jump ship! Not. This is not evidence that the cloud is a beautiful, powerful and unsinkable ship that goes down on its maiden voyage. Stuff happens, no matter how well you prepare. If you think private datacenters don't go down, well, then I have some rearangeable deck chairs to sell you. The goal is to keep improving and minimizing those failure windows. From that perspective there is a lot to learn from the problems the Google App Engine team encountered and how they plan to fix them. Please read the article for all the juicy details, but here's what struck me as key: Power fails. Plan for it . This seems to happen with unexpected frequency for su
2 0.9364925 1630 high scalability-2014-04-11-Stuff The Internet Says On Scalability For April 11th, 2014
Introduction: Hey, it's HighScalability time: DNA nanobots deliver drugs in living cockroaches which have as much compute power as a Commodore 64 40,000 : # of people it takes to colonize a star system; 600,000 : servers vulnerable to heartbleed Quotable Quotes: @laurencetratt : High frequency traders paid $300M to reduce New York <-> Chicago network transfers from 17 -> 13ms. @talios : People read http://highscalability.com for sexual arousal - jim webber #cm14 @viktorklang : LOL RT @giltene 2nd QOTD: @mjpt777 “Some people say ‘Thread affinity is for sissies.’Those people don’t make money.” @pbailis : Reminder: eventual consistency is orthogonal to durability and data loss as long as you correctly resolve update conflicts. @codinghorror : Converting a low-volume educational Discourse instance from Heroku at ~$152.50/month to Digital Ocean at $10/month. @FrancescoC : Scary post on kids who can't tell the diff. between at
3 0.93396479 1617 high scalability-2014-03-21-Stuff The Internet Says On Scalability For March 21st, 2014
Introduction: Hey, it's HighScalability time: Isaac Newton's College Notebook , such a slacker. Quotable Quotes: Chris Anderson : Petabytes allow us to say: ‘Correlation is enough.’ @adron : Back to writing the micro-services for my composite app with regionally distributed, highly available, key value crypto stores of unicorns! DevOps Cafe : when the canary dies you don't buy a stronger canary. Mark Page l: Creativity, like evolution, is merely a series of thefts. GitHub : Whoever has more capacity wins. The Master : We balance probabilities and choose the most likely. It is the scientific use of the imagination. Jonathan Ive : Steve, and I don’t recognize my friend in much of it. Yes, he had a surgically precise opinion. Yes, it could sting. Yes, he constantly questioned. ‘Is this good enough? Is this right?’ but he was so clever. His ideas were bold and magnificent. They could suck the air from the room. And when the ideas didn’t come, he d
4 0.9310081 249 high scalability-2008-02-16-S3 Failed Because of Authentication Overload
Introduction: Being an authentic human being is difficult and apparently authenticating all those S3 requests can be a bit overwhelming as well. Amazon fingered a lot of processor heavy authentication requests as the reason for their downtime: Early this morning, at 3:30am PST, we started seeing elevated levels of authenticated requests from multiple users in one of our locations. While we carefully monitor our overall request volumes and these remained within normal ranges, we had not been monitoring the proportion of authenticated requests. Importantly, these cryptographic requests consume more resources per call than other request types. Shortly before 4:00am PST, we began to see several other users significantly increase their volume of authenticated calls. The last of these pushed the authentication service over its maximum capacity before we could complete putting new capacity in place. In addition to processing authenticated requests, the authentication service also performs accou
5 0.92543894 1572 high scalability-2014-01-03-Stuff The Internet Says On Scalability For January 3rd, 2014
Introduction: Hey, it's HighScalability time, can you handle the truth? Should software architectures include parasites ? They increase diversity and complexity in the food web. 10 Million : classic hockey stick growth pattern for GitHub repositories Quotable Quotes: Seymour Cray : A supercomputer is a device for turning compute-bound problems into IO-bound problems. Robert Sapolsky : And why is self-organization so beautiful to my atheistic self? Because if complex, adaptive systems don’t require a blue print, they don’t require a blue print maker. If they don’t require lightning bolts, they don’t require Someone hurtling lightning bolts. @swardley : Asked for a history of PaaS? From memory, public launch - Zimki ('06), BungeeLabs ('06), Heroku ('07), GAE ('08), CloudFoundry ('11) ... @neil_conway : If you're designing scalable systems, you should understand backpressure and build mechanisms to support it. Scott Aaronson ...the
6 0.92034495 299 high scalability-2008-04-07-Rumors of Signs and Portents Concerning Freeish Google Cloud
7 0.91799486 1292 high scalability-2012-07-27-Stuff The Internet Says On Scalability For July 27, 2012
8 0.91607058 1260 high scalability-2012-06-07-Case Study on Scaling PaaS infrastructure
9 0.90719527 716 high scalability-2009-10-06-Building a Unique Data Warehouse
10 0.9032886 1516 high scalability-2013-09-13-Stuff The Internet Says On Scalability For September 13, 2013
11 0.90190619 1129 high scalability-2011-09-30-Stuff The Internet Says On Scalability For September 30, 2011
12 0.90099967 1612 high scalability-2014-03-14-Stuff The Internet Says On Scalability For March 14th, 2014
13 0.90063179 1389 high scalability-2013-01-18-Stuff The Internet Says On Scalability For January 18, 2013
14 0.89997089 289 high scalability-2008-03-27-Amazon Announces Static IP Addresses and Multiple Datacenter Operation
15 0.89764524 1174 high scalability-2012-01-13-Stuff The Internet Says On Scalability For January 13, 2012
16 0.89596266 517 high scalability-2009-02-21-Google AppEngine - A Second Look
17 0.89300543 1576 high scalability-2014-01-10-Stuff The Internet Says On Scalability For January 10th, 2014
18 0.89280355 1316 high scalability-2012-09-04-Changing Architectures: New Datacenter Networks Will Set Your Code and Data Free
19 0.89124203 976 high scalability-2011-01-20-75% Chance of Scale - Leveraging the New Scaleogenic Environment for Growth
20 0.89052969 1382 high scalability-2013-01-07-Analyzing billions of credit card transactions and serving low-latency insights in the cloud