high_scalability high_scalability-2008 high_scalability-2008-288 knowledge-graph by maker-knowledge-mining
Source: html
Introduction: Greg Linden links to a heavily lesson ladened LISA 2007 paper titled On Designing and Deploying Internet-Scale Services by James Hamilton of the Windows Live Services Platform group. I know people crave nitty-gritty details, but this isn't a how to configure a web server article. It hitches you to a rocket and zooms you up to 50,000 feet so you can take a look at best web operations practices from a broad, yet practical perspective. The author and his team of contributors obviously have a lot of in the trenches experience. Many non-obvious topics are covered. And there's a lot to learn from. The paper has too many details to cover here, but the big sections are: Recommendations Automatic Management and Provisioning Dependency Management Release Cycle and Testing Operations and Capacity Planning Graceful Degradation and Admission Control Customer Self-Provisioning and Self-Help Customer and Press Communication Plan In the recommendations we see some of our o
sentIndex sentText sentNum sentScore
1 Greg Linden links to a heavily lesson ladened LISA 2007 paper titled On Designing and Deploying Internet-Scale Services by James Hamilton of the Windows Live Services Platform group. [sent-1, score-0.163]
2 I know people crave nitty-gritty details, but this isn't a how to configure a web server article. [sent-2, score-0.118]
3 It hitches you to a rocket and zooms you up to 50,000 feet so you can take a look at best web operations practices from a broad, yet practical perspective. [sent-3, score-0.374]
4 The author and his team of contributors obviously have a lot of in the trenches experience. [sent-4, score-0.207]
5 Next are some good thoughts on how to design operations friendly software: Quick service health check. [sent-13, score-0.307]
6 This is the services version of a build verification test. [sent-14, score-0.098]
7 One pod or cluster should not affect another pod or cluster. [sent-18, score-0.499]
8 Treat operations utilities as part of the service. [sent-24, score-0.271]
9 Apparently a lot of software requires cloning hardware installations to support multiple customers. [sent-32, score-0.209]
10 Have your software work for multiple customers all on the same hardware. [sent-34, score-0.078]
11 And the paper continues along the same lines in each section. [sent-35, score-0.163]
12 You'll undoubtedly agree with some of the advice and disagree with some. [sent-37, score-0.446]
13 Greg wants faster release cycles, thinks having server affinity for some things is OK, and thinks the advice on allowing humans to throttle load won't work in a crisis. [sent-38, score-0.692]
14 Perfectly valid points, but what's fun is to consider them. [sent-39, score-0.079]
15 Some companies, for example, have a dead-man's switch that must be thrown before one master can failover to another in a multi-datacenter situation. [sent-40, score-0.091]
16 The advice to "document all conceivable component failures and modes and combinations" sounds good but is truly difficult to do in practice. [sent-43, score-0.452]
17 I went through this process once on a telco project and it took months just to cover all the failure scenarios on a few cards. [sent-44, score-0.413]
18 My favorite part of the whole paper is: We have long believed that 80% of operations issues originate in design and development, so this section on overall service design is the largest and most important. [sent-46, score-0.836]
19 When systems fail, there is a natural tendency to look first to operations since that is where the problem actually took place. [sent-47, score-0.272]
20 Most operations issues, however, either have their genesis in design and development are best solved there. [sent-48, score-0.307]
wordName wordTfidf (topN-words)
[('admission', 0.262), ('advice', 0.245), ('pod', 0.208), ('operations', 0.176), ('paper', 0.163), ('originate', 0.131), ('servicesby', 0.131), ('greg', 0.131), ('design', 0.131), ('thinks', 0.13), ('cover', 0.129), ('lisa', 0.118), ('conceivable', 0.118), ('cloning', 0.118), ('crave', 0.118), ('undoubtedly', 0.113), ('shadow', 0.107), ('telco', 0.107), ('trenches', 0.107), ('favorites', 0.104), ('believed', 0.104), ('feet', 0.1), ('contributors', 0.1), ('verification', 0.098), ('rocket', 0.098), ('throttle', 0.098), ('took', 0.096), ('utilities', 0.095), ('combinations', 0.093), ('installations', 0.091), ('thrown', 0.091), ('modes', 0.089), ('emergency', 0.089), ('affinity', 0.089), ('points', 0.089), ('disagree', 0.088), ('degradation', 0.088), ('press', 0.087), ('sections', 0.085), ('affect', 0.083), ('failure', 0.081), ('perfectly', 0.081), ('spirit', 0.08), ('valid', 0.079), ('details', 0.079), ('broad', 0.078), ('apparently', 0.078), ('customers', 0.078), ('naturally', 0.076), ('rare', 0.076)]
simIndex simValue blogId blogTitle
same-blog 1 0.99999994 288 high scalability-2008-03-25-Paper: On Designing and Deploying Internet-Scale Services
Introduction: Greg Linden links to a heavily lesson ladened LISA 2007 paper titled On Designing and Deploying Internet-Scale Services by James Hamilton of the Windows Live Services Platform group. I know people crave nitty-gritty details, but this isn't a how to configure a web server article. It hitches you to a rocket and zooms you up to 50,000 feet so you can take a look at best web operations practices from a broad, yet practical perspective. The author and his team of contributors obviously have a lot of in the trenches experience. Many non-obvious topics are covered. And there's a lot to learn from. The paper has too many details to cover here, but the big sections are: Recommendations Automatic Management and Provisioning Dependency Management Release Cycle and Testing Operations and Capacity Planning Graceful Degradation and Admission Control Customer Self-Provisioning and Self-Help Customer and Press Communication Plan In the recommendations we see some of our o
2 0.13113619 734 high scalability-2009-10-30-Hot Scalabilty Links for October 30 2009
Introduction: Life beyond Distributed Transactions: an Apostate’s Opinion by Pat Helland. In particular, we focus on the implications that fall out of assuming we cannot have large-scale distributed transactions. T ragedy of the Commons, and Cold Starts - Cold application starts on Google App Engine kill your application's responsiveness. Intel’s 1M IOPS desktop SSD setup by Kevin Burton. What do you get when you take 7 Intel SSDs and throw them in a desktop? 1M IOPS Videos from NoSQL Berlin sessions. Nicely done talks on CAP, MongoDB, Redis, 4th generation object databases, CouchDB, and Riak. Designs, Lessons and Advice from Building Large Distributed Systems by Jeff Dean of Google describing how they do their thing. Here are some glosses on the talk by Greg Linden and James Hamilton. You really can't do better than Greg and James. Advice from Google on Large Distributed Systems by Greg Linden. A nice summary of Jeff Dean's talk. A standard Google server
3 0.12624937 1240 high scalability-2012-05-07-Startups are Creating a New System of the World for IT
Introduction: It remains that, from the same principles, I now demonstrate the frame of the System of the World. -- Isaac Newton The practice of IT reminds me a lot of the practice of science before Isaac Newton. Aristotelianism was dead, but there was nothing to replace it. Then Newton came along, created a scientific revolution with his System of the World . And everything changed. That was New System of the World number one. New System of the World number two was written about by the incomparable Neal Stephenson in his incredible Baroque Cycle series. It explores the singular creation of a new way of organizing society grounded in new modes of thought in business, religion, politics, and science. Our modern world emerged Enlightened as it could from this roiling cauldron of forces. In IT we may have had a Leonardo da Vinci or even a Galileo, but we’ve never had our Newton. Maybe we don't need a towering genius to make everything clear? For years startups, like the frenetically inventive
Introduction: This is a guest post by Patrick Eaton , Software Engineer and Distributed Systems Architect at Stackdriver. Stackdriver provides intelligent monitoring-as-a-service for cloud hosted applications. Behind this easy-to-use service is a large distributed system for collecting and storing metrics and events, monitoring and alerting on them, analyzing them, and serving up all the results in a web UI. Because we ourselves run in the cloud (mostly on AWS), we spend a lot of time thinking about how to deal with faults in the cloud. We have developed a framework for thinking about fault mitigation for large, cloud-hosted systems. We endearingly call this framework the “Four Hamiltons” because it is inspired by an article from James Hamilton, the Vice President and Distinguished Engineer at Amazon Web Services. The article that led to this framework is called “ The Power Failure Seen Around the World ” . Hamilton analyzes the causes of the power outage that affected Super Bowl XL
5 0.10736992 96 high scalability-2007-09-18-Amazon Architecture
Introduction: This is a wonderfully informative Amazon update based on Joachim Rohde's discovery of an interview with Amazon's CTO. You'll learn about how Amazon organizes their teams around services, the CAP theorem of building scalable systems, how they deploy software, and a lot more. Many new additions from the ACM Queue article have also been included. Amazon grew from a tiny online bookstore to one of the largest stores on earth. They did it while pioneering new and interesting ways to rate, review, and recommend products. Greg Linden shared is version of Amazon's birth pangs in a series of blog articles Site: http://amazon.com Information Sources Early Amazon by Greg Linden How Linux saved Amazon millions Interview Werner Vogels - Amazon's CTO Asynchronous Architectures - a nice summary of Werner Vogels' talk by Chris Loosley Learning from the Amazon technology platform - A Conversation with Werner Vogels Werner Vogels' Weblog - building scalable and robus
7 0.097678706 1002 high scalability-2011-03-09-Productivity vs. Control tradeoffs in PaaS
8 0.097489521 1068 high scalability-2011-06-27-TripAdvisor Architecture - 40M Visitors, 200M Dynamic Page Views, 30TB Data
9 0.094752602 757 high scalability-2010-01-04-11 Strategies to Rock Your Startup’s Scalability in 2010
10 0.09372811 1508 high scalability-2013-08-28-Sean Hull's 20 Biggest Bottlenecks that Reduce and Slow Down Scalability
12 0.092932366 378 high scalability-2008-09-03-Some Facebook Secrets to Better Operations
13 0.092893541 750 high scalability-2009-12-16-Building Super Scalable Systems: Blade Runner Meets Autonomic Computing in the Ambient Cloud
14 0.092662193 313 high scalability-2008-05-02-Friends for Sale Architecture - A 300 Million Page View-Month Facebook RoR App
15 0.092061147 691 high scalability-2009-08-31-Squarespace Architecture - A Grid Handles Hundreds of Millions of Requests a Month
16 0.09201438 687 high scalability-2009-08-24-How Google Serves Data from Multiple Datacenters
17 0.091325507 1520 high scalability-2013-09-20-Stuff The Internet Says On Scalability For September 20, 2013
18 0.091263562 387 high scalability-2008-09-22-Paper: On Delivering Embarrassingly Distributed Cloud Services
19 0.09019766 1596 high scalability-2014-02-14-Stuff The Internet Says On Scalability For February 14th, 2014
topicId topicWeight
[(0, 0.182), (1, 0.067), (2, -0.003), (3, 0.054), (4, 0.027), (5, -0.034), (6, -0.02), (7, -0.007), (8, -0.048), (9, -0.016), (10, -0.061), (11, 0.01), (12, -0.013), (13, -0.036), (14, 0.043), (15, 0.017), (16, 0.048), (17, -0.006), (18, -0.014), (19, 0.017), (20, 0.064), (21, 0.016), (22, -0.053), (23, -0.035), (24, -0.056), (25, 0.006), (26, -0.003), (27, 0.035), (28, -0.014), (29, 0.049), (30, -0.002), (31, 0.015), (32, 0.033), (33, -0.002), (34, 0.009), (35, 0.027), (36, 0.015), (37, 0.015), (38, 0.021), (39, 0.05), (40, -0.033), (41, 0.018), (42, 0.013), (43, 0.03), (44, 0.043), (45, 0.009), (46, -0.004), (47, 0.024), (48, 0.021), (49, -0.072)]
simIndex simValue blogId blogTitle
same-blog 1 0.96957064 288 high scalability-2008-03-25-Paper: On Designing and Deploying Internet-Scale Services
Introduction: Greg Linden links to a heavily lesson ladened LISA 2007 paper titled On Designing and Deploying Internet-Scale Services by James Hamilton of the Windows Live Services Platform group. I know people crave nitty-gritty details, but this isn't a how to configure a web server article. It hitches you to a rocket and zooms you up to 50,000 feet so you can take a look at best web operations practices from a broad, yet practical perspective. The author and his team of contributors obviously have a lot of in the trenches experience. Many non-obvious topics are covered. And there's a lot to learn from. The paper has too many details to cover here, but the big sections are: Recommendations Automatic Management and Provisioning Dependency Management Release Cycle and Testing Operations and Capacity Planning Graceful Degradation and Admission Control Customer Self-Provisioning and Self-Help Customer and Press Communication Plan In the recommendations we see some of our o
2 0.76182061 1420 high scalability-2013-03-08-Stuff The Internet Says On Scalability For March 8, 2013
Introduction: Hey, it's HighScalability time: Quotable Quotes: @ibogost : Disabling features of SimCity due to ineffective central infrastructure is probably the most realistic simulation of the modern city. antirez : The point is simply to show how SSDs can't be considered, currently, as a bit slower version of memory. Their performance characteristics are a lot more about, simply, "faster disks". @jessenoller : I only use JavaScript so I can gain maximum scalability across multiple cores. Also unicorns. Paint thinner gingerbread @liammclennan : high-scalability ruby. Why bother? @scomma : Problem with BitCoin is not scalability, not even usability. It's whether someone will crack the algorithm and render BTC entirely useless. @webclimber : Amazing how often I find myself explaining that scalability is not magical @mvmsan : Flash as Primary Storage - Highest Cost, Lack of HA, scalability and management features #flas
3 0.74547929 1209 high scalability-2012-03-14-The Azure Outage: Time Is a SPOF, Leap Day Doubly So
Introduction: This is a guest post by Steve Newman, co-founder of Writely (Google Docs), tech lead on the Paxos-based synchronous replication in Megastore, and founder of cloud service provider Scalyr.com . Microsoft’s Azure service suffered a widely publicized outage on February 28th / 29th. Microsoft recently published an excellent postmortem . For anyone trying to run a high-availability service, this incident can teach several important lessons. The central lesson is that, no matter how much work you put into redundancy, problems will arise. Murphy is strong and, I might say, creative; things go wrong. So preventative measures are important, but how you react to problems is just as important. It’s interesting to review the Azure incident in this light. The postmortem is worth reading in its entirety, but here’s a quick summary: each time Azure launches a new VM, it creates a “transfer certificate” to secure communications with that VM. There was a bug in the code that determines the ce
4 0.73941594 1468 high scalability-2013-05-31-Stuff The Internet Says On Scalability For May 31, 2013
Introduction: Hey, it's HighScalability time: ( resource consumption scales sub-linearly with population; economic output scales super-linearly) 4PB : Ancestry.com's House of Us Quotable Quotes: @kellabyte : XBOX Live scaling up from 15,000 servers to 300,000 servers. That's some scale right there. @giuseppegurgone : [...] that could work for small projects. "why do we worry about scalability on day 1?" because tomorrow it could be too late :) Max Boot : American troops found their tactics and technology, still designed to defeat an opponent like the now defunct Red Army, woefully inadequate to deal with these new threats. In this sort of war, there were no flanks to turn, few bastions to storm, no capitals to seize. Steve Jobs on the power of having 120 million credit cards on file: You need us more than we need you. Netflix: Dystopia as a Service . Adrian Cockcroft describes an out of Eden world where hardware and softwa
5 0.73746461 23 high scalability-2007-07-24-Major Websites Down: Or Why You Want to Run in Two or More Data Centers.
Introduction: A lot of sites hosted in San Francisco are down because of at least 6 back-to-back power outages power outages. More details at laughingsquid . Sites like SecondLife, Craigstlist, Technorati, Yelp and all Six Apart properties, TypePad, LiveJournal and Vox are all down. The cause was an underground explosion in a transformer vault under a manhole at 560 Mission Street. Flames shot 6 feet out from the manhole cover. Over PG&E; 30,000 customers are without power. What's perplexing is the UPS backup and diesel generators didn't kick in to bring the datacenter back on line. I've never toured that datacenter, but they usually have massive backup systems. It's probably one of those multiple simultaneous failure situations that you hope never happen in real life, but too often do. Or maybe the infrastructure wasn't rolled out completely. Update: the cause was a cascade of failures in a tightly couples system that could never happen :-) Details at Failure Happens: A summary of the power
6 0.7263642 1431 high scalability-2013-03-29-Stuff The Internet Says On Scalability For March 29, 2013
7 0.72539908 139 high scalability-2007-10-30-Paper: Dynamo: Amazon’s Highly Available Key-value Store
8 0.71131945 1297 high scalability-2012-08-03-Stuff The Internet Says On Scalability For August 3, 2012
9 0.71127754 1506 high scalability-2013-08-23-Stuff The Internet Says On Scalability For August 23, 2013
11 0.70721751 474 high scalability-2008-12-21-The I.H.S.D.F. Theorem: A Proposed Theorem for the Trade-offs in Horizontally Scalable Systems
12 0.70510149 757 high scalability-2010-01-04-11 Strategies to Rock Your Startup’s Scalability in 2010
13 0.70441288 1379 high scalability-2012-12-31-Designing for Resiliency will be so 2013
14 0.70361662 1568 high scalability-2013-12-23-What Happens While Your Brain Sleeps is Surprisingly Like How Computers Stay Sane
15 0.701608 1252 high scalability-2012-05-25-Stuff The Internet Says On Scalability For May 25, 2012
16 0.69752157 1368 high scalability-2012-12-07-Stuff The Internet Says On Scalability For December 7, 2012
17 0.69663715 1520 high scalability-2013-09-20-Stuff The Internet Says On Scalability For September 20, 2013
18 0.69638902 1494 high scalability-2013-07-19-Stuff The Internet Says On Scalability For July 19, 2013
19 0.69371408 284 high scalability-2008-03-19-RAD Lab is Creating a Datacenter Operating System
20 0.69355142 1039 high scalability-2011-05-12-Paper: Mind the Gap: Reconnecting Architecture and OS Research
topicId topicWeight
[(1, 0.105), (2, 0.182), (10, 0.021), (19, 0.022), (30, 0.02), (40, 0.055), (46, 0.146), (61, 0.067), (77, 0.012), (79, 0.169), (85, 0.071), (94, 0.06)]
simIndex simValue blogId blogTitle
same-blog 1 0.9298386 288 high scalability-2008-03-25-Paper: On Designing and Deploying Internet-Scale Services
Introduction: Greg Linden links to a heavily lesson ladened LISA 2007 paper titled On Designing and Deploying Internet-Scale Services by James Hamilton of the Windows Live Services Platform group. I know people crave nitty-gritty details, but this isn't a how to configure a web server article. It hitches you to a rocket and zooms you up to 50,000 feet so you can take a look at best web operations practices from a broad, yet practical perspective. The author and his team of contributors obviously have a lot of in the trenches experience. Many non-obvious topics are covered. And there's a lot to learn from. The paper has too many details to cover here, but the big sections are: Recommendations Automatic Management and Provisioning Dependency Management Release Cycle and Testing Operations and Capacity Planning Graceful Degradation and Admission Control Customer Self-Provisioning and Self-Help Customer and Press Communication Plan In the recommendations we see some of our o
2 0.90898204 1464 high scalability-2013-05-24-Stuff The Internet Says On Scalability For May 24, 2013
Introduction: Hey, it's HighScalability time: ( Scaling the Mighty Redwood photograph by Michael Nichols for NatGEO ) ~20K : Netflix AWS instances; 100 million hours per minute : Youtube video upload; Quotable Quotes: @sw17ch : Computer Science is thinking about thinking. Software Engineering is thinking about how to avoid thinking. @neha : I am starting a distributed systems reading group at MIT. Suggestions on papers to read? Current list here . John Sheehan : Services are the new process @cheeseplus : Sharding isn’t a scalability strategy, it’s a failure mode in progress. @basharatw : @adrianco Features in days, not months; hw in mins not weeks; incident response in secs not hours … there's a trade off for utopia #gluecon @mgroeninger : @johnsheehan now telling a story about struggling against tools... quit to build a better hammer #gluecon < this is the heart of devops @mgroeninger : @adrianco Netflix had to get into the CDN space, bu
3 0.89571512 634 high scalability-2009-06-20-Building a data cycle at LinkedIn with Hadoop and Project Voldemort
Introduction: Update : Building Voldemort read-only stores with Hadoop . A write up on what LinkedIn is doing to integrate large offline Hadoop data processing jobs with a fast, distributed online key-value storage system, Project Voldemort .
4 0.8737722 1154 high scalability-2011-12-09-Stuff The Internet Says On Scalability For December 9, 2011
Introduction: It takes a licking and keeps on HighScalabilitying: Instagram: 60 photos per second ; Twitter: 8,868 Tweets per second MTV Awards ; Foursquare: 15 million users ; Facebook: 60 million qps ; Netflix: stream 1 billion hours in Q4 ; Evolution: 16,000 eyes ; Evernote: 750K Paid Users ; AOL: 60,000 Servers ; Netflix API: 20,000 requests per second ; Tumblr: 900 posts per second . Quotable quotes: @LusciousPear : This is my key-value store. There are many others like it, but this one is mine. @pavlobaron : @old_sound if this site isn't on highscalability.com , I won't open it @nzmrmn : Words I hate: moist, "sinfully decadent," facilitator, dopesauce, deliverables, accountability, coolio, "ping me," scalability. @anggriawan : Mark Callaghan (Facebook Engineer): "You can tune #MySQL to perform very fast per node if you know what you’re doing." Jesper Nordenberg : Java is good at hiding the real complexities of programm
Introduction: All in all this is still my favorite post and I still think it's an accurate vision of a future. Not everyone agrees, but I guess we'll see..."But it is not complicated. [There's] just a lot of it." \--Richard Feynmanon how the immense variety of the world arises from simple rules.Contents:Have We Reached the End of Scaling?Applications Become Black Boxes Using Markets to Scale and Control CostsLet's Welcome our Neo-Feudal OverlordsThe Economic Argument for the Ambient CloudWhat Will Kill the Cloud?The Amazing Collective Compute Power of the Ambient CloudUsing the Ambient Cloud as an Application RuntimeApplications as Virtual StatesConclusionWe have not yet begun to scale. The world is still fundamentally disconnected and for all our wisdom we are still in the earliest days of learning how to build truly large planet-scaling applications.Today 350 million users on Facebook is a lot of users and five million followers on Twitter is a lot of followers. This may seem like a lot now, but c
6 0.86706495 1316 high scalability-2012-09-04-Changing Architectures: New Datacenter Networks Will Set Your Code and Data Free
9 0.8627882 1612 high scalability-2014-03-14-Stuff The Internet Says On Scalability For March 14th, 2014
10 0.86153775 289 high scalability-2008-03-27-Amazon Announces Static IP Addresses and Multiple Datacenter Operation
11 0.86135912 1630 high scalability-2014-04-11-Stuff The Internet Says On Scalability For April 11th, 2014
12 0.86094433 1389 high scalability-2013-01-18-Stuff The Internet Says On Scalability For January 18, 2013
13 0.85823548 1170 high scalability-2012-01-06-Stuff The Internet Says On Scalability For January 6, 2012
14 0.85818499 1647 high scalability-2014-05-14-Google Says Cloud Prices Will Follow Moore’s Law: Are We All Renters Now?
15 0.85714793 517 high scalability-2009-02-21-Google AppEngine - A Second Look
16 0.85664278 768 high scalability-2010-02-01-What Will Kill the Cloud?
17 0.85657275 1382 high scalability-2013-01-07-Analyzing billions of credit card transactions and serving low-latency insights in the cloud
18 0.85631901 765 high scalability-2010-01-25-Let's Welcome our Neo-Feudal Overlords
19 0.85601789 763 high scalability-2010-01-22-How BuddyPoke Scales on Facebook Using Google App Engine
20 0.85549843 786 high scalability-2010-03-02-Using the Ambient Cloud as an Application Runtime