high_scalability high_scalability-2007 high_scalability-2007-189 knowledge-graph by maker-knowledge-mining
Source: html
Introduction: Release It! author Michael Nygard tells a tale of two web sites , both brought low by unexpectedly huge unbounded results sets that slowed down their sites to the speed of a Christmas checkout line. I've committed this error more than a few times. During testing the results sets are often small, so you don't see problems. Or when a product is new you don't have a lot of data so everything is fine, until some magic line is crossed and you get that dreaded 2AM fix it call. My most embarrassing bug of this type caused a rather spectacular failure at a customer site as the variance in response times was out of spec and this kicked in penalty clauses. What happened was the customer had a larger network than we could even test (customers always get the good stuff). I took a lock and went to get all the data. Because the result set was so much larger in their larger system I took the lock for many more milliseconds than I should have. Unknown to me a chunk of code on the criti
sentIndex sentText sentNum sentScore
1 author Michael Nygard tells a tale of two web sites , both brought low by unexpectedly huge unbounded results sets that slowed down their sites to the speed of a Christmas checkout line. [sent-2, score-0.806]
2 During testing the results sets are often small, so you don't see problems. [sent-4, score-0.316]
3 Or when a product is new you don't have a lot of data so everything is fine, until some magic line is crossed and you get that dreaded 2AM fix it call. [sent-5, score-0.33]
4 My most embarrassing bug of this type caused a rather spectacular failure at a customer site as the variance in response times was out of spec and this kicked in penalty clauses. [sent-6, score-0.813]
5 What happened was the customer had a larger network than we could even test (customers always get the good stuff). [sent-7, score-0.223]
6 Because the result set was so much larger in their larger system I took the lock for many more milliseconds than I should have. [sent-9, score-0.702]
7 Unknown to me a chunk of code on the critical path also was in the lock path and all hell broke loose. [sent-10, score-0.482]
8 I had to change the logic to process the result set in fixed size deterministic chunks, releasing locks as I went. [sent-11, score-0.439]
9 I then hunted down every other place I made the same mistake. [sent-14, score-0.126]
10 A common theme in many of the profiles on this site is protecting your system from requests that can bring down the system. [sent-17, score-0.097]
11 Mailinator has a lot resource exhaustion problems and does a good job solving them. [sent-18, score-0.12]
12 Ebay has an interesting strategy of doing as little work as possible in the database which leads them to do joins in application space. [sent-19, score-0.123]
13 With proper indexes performing selects in the database to minimize the result sets would seem to be a win as databases are good at this sort of thing. [sent-22, score-0.455]
14 Michael's strategy: You should make your apps be paranoid about their data. [sent-25, score-0.134]
15 If your app processes one record at a time, then looping through an entire result set might be OK---as long as you're not making a user wait while you do. [sent-26, score-0.41]
16 But if your app that turns rows into objects, then it had better be very selective about its SELECTs. [sent-27, score-0.104]
17 The data producer might have changed in a surprising way, particularly if it's not under your control. [sent-29, score-0.27]
18 Purging routines might not be in place, or might have gotten broken. [sent-30, score-0.496]
19 A diagnostic tactic is to benchmark all page response times and look for pages that take too long. [sent-32, score-0.326]
20 And because you logged everything you can figure out why the response was slow and then take measures to fix it. [sent-34, score-0.289]
wordName wordTfidf (topN-words)
[('sets', 0.201), ('lock', 0.176), ('result', 0.16), ('might', 0.144), ('paranoid', 0.134), ('yah', 0.134), ('larger', 0.134), ('hunted', 0.126), ('producer', 0.126), ('strategy', 0.123), ('dreaded', 0.12), ('crossed', 0.12), ('exhaustion', 0.12), ('purging', 0.12), ('nygard', 0.116), ('routines', 0.116), ('spectacular', 0.116), ('results', 0.115), ('path', 0.111), ('response', 0.111), ('diagnostic', 0.109), ('unexpectedly', 0.106), ('embarrassing', 0.106), ('kicked', 0.106), ('looping', 0.106), ('tactic', 0.106), ('spec', 0.104), ('selective', 0.104), ('checkout', 0.102), ('christmas', 0.1), ('slowed', 0.098), ('took', 0.098), ('protecting', 0.097), ('releasing', 0.095), ('unbounded', 0.094), ('selects', 0.094), ('logic', 0.093), ('cpu', 0.092), ('gotten', 0.092), ('deterministic', 0.091), ('variance', 0.091), ('penalty', 0.09), ('tale', 0.09), ('fix', 0.09), ('customer', 0.089), ('unknown', 0.088), ('measures', 0.088), ('emails', 0.087), ('opposite', 0.085), ('broke', 0.084)]
simIndex simValue blogId blogTitle
same-blog 1 1.0000001 189 high scalability-2007-12-21-Strategy: Limit Result Sets
Introduction: Release It! author Michael Nygard tells a tale of two web sites , both brought low by unexpectedly huge unbounded results sets that slowed down their sites to the speed of a Christmas checkout line. I've committed this error more than a few times. During testing the results sets are often small, so you don't see problems. Or when a product is new you don't have a lot of data so everything is fine, until some magic line is crossed and you get that dreaded 2AM fix it call. My most embarrassing bug of this type caused a rather spectacular failure at a customer site as the variance in response times was out of spec and this kicked in penalty clauses. What happened was the customer had a larger network than we could even test (customers always get the good stuff). I took a lock and went to get all the data. Because the result set was so much larger in their larger system I took the lock for many more milliseconds than I should have. Unknown to me a chunk of code on the criti
2 0.1393775 1413 high scalability-2013-02-27-42 Monster Problems that Attack as Loads Increase
Introduction: For solutions take a look at: 7 Life Saving Scalability Defenses Against Load Monster Attacks . This is a look at all the bad things that can happen to your carefully crafted program as loads increase: all hell breaks lose. Sure, you can scale out or scale up, but you can also choose to program better. Make your system handle larger loads. This saves money because fewer boxes are needed and it will make the entire application more reliable and have better response times. And it can be quite satisfying as a programmer. Large Number Of Objects We usually get into scaling problems when the number of objects gets larger. Clearly resource usage of all types is stressed as the number of objects grow. Continuous Failures Makes An Infinite Event Stream During large network failure scenarios there is never time for the system recover. We are in a continual state of stress. Lots of High Priority Work For example, rerouting is a high priority activity. If there is a large amount
3 0.13307719 920 high scalability-2010-10-15-Troubles with Sharding - What can we learn from the Foursquare Incident?
Introduction: For everything given something seems to be taken. Caching is a great scalability solution, but caching also comes with problems . Sharding is a great scalability solution, but as Foursquare recently revealed in a post-mortem about their 17 hours of downtime, sharding also has problems. MongoDB, the database Foursquare uses, also contributed their post-mortem of what went wrong too. Now that everyone has shared and resharded, what can we learn to help us skip these mistakes and quickly move on to a different set of mistakes? First, like for Facebook , huge props to Foursquare and MongoDB for being upfront and honest about their problems. This helps everyone get better and is a sign we work in a pretty cool industry. Second, overall, the fault didn't flow from evil hearts or gross negligence. As usual the cause was more mundane: a key system, that could be a little more robust, combined with a very popular application built by a small group of people, under immense pressure
4 0.12948835 15 high scalability-2007-07-16-Blog: MySQL Performance Blog - Everything about MySQL Performance.
Introduction: Follow this blog and you'll learn a lot about MySQL and how to make it sing. A Quick Hit of What's Inside Working with large data sets in MySQL, PHP Large result sets and summary tables, Implementing efficient counters with MySQL. Site: http://www.mysqlperformanceblog.com/
Introduction: In Don’t panic! Here’s how to quickly scale your mobile apps Mike Maelzer paints a wonderful picture of how Avocado , a mobile app for connecting couples, evolved to handle 30x traffic within a few weeks. If you are just getting started then this is a great example to learn from. What I liked: it's well written, packing a lot of useful information in a little space; it's failure driven, showing the process of incremental change driven by purposeful testing and production experience; it shows awareness of what's important, in their case, user signup; a replica setup was used for testing, a nice cloud benefit. Their Biggest lesson learned is a good one: It would have been great to start the scaling process much earlier. Due to time pressure we had to make compromises –like dropping four of our media resizer boxes. While throwing more hardware at some scaling problems does work, it’s less than ideal. Here's my gloss on the article: Evolution One - Make it Work When just
6 0.12300737 1646 high scalability-2014-05-12-4 Architecture Issues When Scaling Web Applications: Bottlenecks, Database, CPU, IO
7 0.12217157 1038 high scalability-2011-05-11-Troubleshooting response time problems – why you cannot trust your system metrics
8 0.12049066 464 high scalability-2008-12-13-Strategy: Facebook Tweaks to Handle 6 Time as Many Memcached Requests
9 0.11929125 1065 high scalability-2011-06-21-Running TPC-C on MySQL-RDS
10 0.11724228 1508 high scalability-2013-08-28-Sean Hull's 20 Biggest Bottlenecks that Reduce and Slow Down Scalability
11 0.10812119 934 high scalability-2010-11-04-Facebook at 13 Million Queries Per Second Recommends: Minimize Request Variance
12 0.10798206 1596 high scalability-2014-02-14-Stuff The Internet Says On Scalability For February 14th, 2014
13 0.10779382 306 high scalability-2008-04-21-The Search for the Source of Data - How SimpleDB Differs from a RDBMS
14 0.10527144 1204 high scalability-2012-03-06-Ask For Forgiveness Programming - Or How We'll Program 1000 Cores
15 0.10510872 954 high scalability-2010-12-06-What the heck are you actually using NoSQL for?
16 0.10499946 1633 high scalability-2014-04-16-Six Lessons Learned the Hard Way About Scaling a Million User System
17 0.10443272 910 high scalability-2010-09-30-Facebook and Site Failures Caused by Complex, Weakly Interacting, Layered Systems
18 0.10428992 96 high scalability-2007-09-18-Amazon Architecture
19 0.10317519 1425 high scalability-2013-03-18-Beyond Threads and Callbacks - Application Architecture Pros and Cons
20 0.10210092 327 high scalability-2008-05-27-How I Learned to Stop Worrying and Love Using a Lot of Disk Space to Scale
topicId topicWeight
[(0, 0.193), (1, 0.116), (2, -0.037), (3, -0.041), (4, 0.011), (5, -0.007), (6, 0.02), (7, 0.061), (8, -0.014), (9, -0.071), (10, -0.002), (11, 0.017), (12, -0.0), (13, 0.015), (14, 0.042), (15, -0.075), (16, 0.005), (17, -0.005), (18, 0.032), (19, 0.02), (20, 0.004), (21, -0.032), (22, 0.075), (23, -0.004), (24, -0.014), (25, 0.011), (26, -0.031), (27, 0.053), (28, 0.053), (29, -0.014), (30, 0.025), (31, 0.023), (32, 0.047), (33, 0.015), (34, 0.005), (35, 0.059), (36, -0.041), (37, 0.045), (38, -0.028), (39, -0.003), (40, -0.032), (41, -0.019), (42, 0.016), (43, -0.015), (44, 0.02), (45, 0.014), (46, 0.043), (47, 0.002), (48, 0.016), (49, -0.038)]
simIndex simValue blogId blogTitle
same-blog 1 0.97951436 189 high scalability-2007-12-21-Strategy: Limit Result Sets
Introduction: Release It! author Michael Nygard tells a tale of two web sites , both brought low by unexpectedly huge unbounded results sets that slowed down their sites to the speed of a Christmas checkout line. I've committed this error more than a few times. During testing the results sets are often small, so you don't see problems. Or when a product is new you don't have a lot of data so everything is fine, until some magic line is crossed and you get that dreaded 2AM fix it call. My most embarrassing bug of this type caused a rather spectacular failure at a customer site as the variance in response times was out of spec and this kicked in penalty clauses. What happened was the customer had a larger network than we could even test (customers always get the good stuff). I took a lock and went to get all the data. Because the result set was so much larger in their larger system I took the lock for many more milliseconds than I should have. Unknown to me a chunk of code on the criti
2 0.76223505 1413 high scalability-2013-02-27-42 Monster Problems that Attack as Loads Increase
Introduction: For solutions take a look at: 7 Life Saving Scalability Defenses Against Load Monster Attacks . This is a look at all the bad things that can happen to your carefully crafted program as loads increase: all hell breaks lose. Sure, you can scale out or scale up, but you can also choose to program better. Make your system handle larger loads. This saves money because fewer boxes are needed and it will make the entire application more reliable and have better response times. And it can be quite satisfying as a programmer. Large Number Of Objects We usually get into scaling problems when the number of objects gets larger. Clearly resource usage of all types is stressed as the number of objects grow. Continuous Failures Makes An Infinite Event Stream During large network failure scenarios there is never time for the system recover. We are in a continual state of stress. Lots of High Priority Work For example, rerouting is a high priority activity. If there is a large amount
Introduction: This is a guestrepostby Ron Pressler, the founder and CEO ofParallel Universe, a Y Combinator company building advanced middleware for real-time applications. Little's Law helps us determine the maximum request rate a server can handle. When we apply it, we find that the dominating factor limiting a server's capacity is not the hardware but theOS.Should we buy more hardware if software is the problem? If not, how can we remove that software limitation in a way that does not make the code much harder to write and understand?Many modern web applications are composed of multiple (often many)HTTPservices (this is often called a micro-service architecture). This architecture has many advantages in terms of code reuse and maintainability, scalability and fault tolerance. In this post I'd like to examine one particular bottleneck in the approach, which hinders scalability as well as fault tolerance, and various ways to deal with it (I am using the term "scalability" very loosely in this post
Introduction: Here's a common situation and question from the mechanical-sympathy Google group by Avinash Agrawal on the black art of capacity planning: How to get started with sizing and capacity planning, assuming we don't know the software behavior and its completely new product to deal with? Gil Tene , Vice President of Technology and CTO & Co-Founder, wrote a very understandable and useful answer that is worth highlighting: Start with requirements. I see way too many "capacity planning" exercises that go off spending weeks measuring some irrelevant metrics about a system (like how many widgets per hour can this thing do) without knowing what they actually need it to do. There are two key sets of metrics to state here: the "how much" set and the "how bad" set: In the "How Much" part, you need to establish, based on expected business needs, Numbers for things (like connections, users, streams, transactions or messages per second) that you expect to interact with at the peak t
5 0.75444609 1415 high scalability-2013-03-04-7 Life Saving Scalability Defenses Against Load Monster Attacks
Introduction: We talked about 42 Monster Problems That Attack As Loads Increase . Here are a few ways you can defend yourself, secrets revealed by scaling masters across the ages. Note that these are low level programming level moves, not large architecture type strategies. Use Resources Proportional To a Fixed Limit This is probably the most important rule for achieving scalability within an application. What it means: Find the resource that has a fixed limit that you know you can support. For example, a guarantee to handle a certain number of objects in memory. So if we always use resources proportional to the number of objects it is likely we can prevent resource exhaustion. Devise ways of tying what you need to do to the individual resources. Some examples: Keep a list of purchase orders with line items over $20 (or whatever). Do not keep a list of the line items because the number of items can be much larger than the number of purchase orders. You have kept the resource usage
6 0.74565583 1135 high scalability-2011-10-31-15 Ways to Make Your Application Feel More Responsive under Google App Engine
7 0.74218571 1250 high scalability-2012-05-23-Averages, web performance data, and how your analytics product is lying to you
8 0.73457927 1038 high scalability-2011-05-11-Troubleshooting response time problems – why you cannot trust your system metrics
9 0.73175925 1617 high scalability-2014-03-21-Stuff The Internet Says On Scalability For March 21st, 2014
10 0.72851086 1387 high scalability-2013-01-15-More Numbers Every Awesome Programmer Must Know
11 0.72628999 659 high scalability-2009-07-20-A Scalability Lament
12 0.72600853 344 high scalability-2008-06-09-FaceStat's Rousing Tale of Scaling Woe and Wisdom Won
13 0.72286475 1271 high scalability-2012-06-25-StubHub Architecture: The Surprising Complexity Behind the World’s Largest Ticket Marketplace
14 0.72048187 1209 high scalability-2012-03-14-The Azure Outage: Time Is a SPOF, Leap Day Doubly So
15 0.72008789 1336 high scalability-2012-10-09-Batoo JPA - The new JPA Implementation that runs over 15 times faster...
16 0.71636975 910 high scalability-2010-09-30-Facebook and Site Failures Caused by Complex, Weakly Interacting, Layered Systems
17 0.71610063 622 high scalability-2009-06-08-Distribution of queries per second
18 0.71456236 1204 high scalability-2012-03-06-Ask For Forgiveness Programming - Or How We'll Program 1000 Cores
19 0.71382529 249 high scalability-2008-02-16-S3 Failed Because of Authentication Overload
20 0.71293712 919 high scalability-2010-10-14-I, Cloud
topicId topicWeight
[(1, 0.111), (2, 0.276), (3, 0.199), (10, 0.043), (47, 0.02), (61, 0.109), (79, 0.129), (94, 0.036)]
simIndex simValue blogId blogTitle
Introduction: Awesome paper on how particular synchronization mechanisms scale on multi-core architectures: Everything You Always Wanted to Know About Synchronization but Were Afraid to Ask . The goal is to pick a locking approach that doesn't degrade as the number of cores increase. Like everything else in life, that doesn't appear to be generically possible: None of the nine locking schemes we consider consistently outperforms any other one, on all target architectures or workloads. Strictly speaking, to seek optimality, a lock algorithm should thus be selected based on the hardware platform and the expected workload . Abstract: This paper presents the most exhaustive study of synchronization to date. We span multiple layers, from hardware cache-coherence protocols up to high-level concurrent software. We do so on different types architectures, from single-socket – uniform and nonuniform – to multi-socket – directory and broadcastbased – many-cores. We draw a set of observations t
2 0.94290078 926 high scalability-2010-10-24-Hot Scalability Links For Oct 24, 2010
Introduction: On a cold and rainy Fall day, a day stolen from winter rather than our usual gorgeous Indian Summers , a day not even the SF Giants winning the pennant can help warm, here are some hot links to read by a digital flame: Using MySQL as a NoSQL - A story for exceeding 750,000 qps on a commodity server by Yoshinori Matsunobu. Wonderfully detailed post on how you can lookup a row by ID really fast if you bypass all the typical MySQL query parsing overhead. Minecraftwiki.net and minecraftforum.net now serve more traffic than Slashdot and Stackoverflow! 1 million pageviews and 100k uniques per day, per site; 10TB of bandwidth a month; 4+ machines running Varnish, HAProxy, PHP, MySQL, Nginx. Stuff the Internet Says: @ old_sound : Somebody make me a t-shirt that says "I've read the CAP theorem and I liked it" @dscape : How relevant do I think the CAP theorem is? Not at all. I honestly hate conversations where anyone talks about crap.. cap, sorry. @humidbei
same-blog 3 0.92899203 189 high scalability-2007-12-21-Strategy: Limit Result Sets
Introduction: Release It! author Michael Nygard tells a tale of two web sites , both brought low by unexpectedly huge unbounded results sets that slowed down their sites to the speed of a Christmas checkout line. I've committed this error more than a few times. During testing the results sets are often small, so you don't see problems. Or when a product is new you don't have a lot of data so everything is fine, until some magic line is crossed and you get that dreaded 2AM fix it call. My most embarrassing bug of this type caused a rather spectacular failure at a customer site as the variance in response times was out of spec and this kicked in penalty clauses. What happened was the customer had a larger network than we could even test (customers always get the good stuff). I took a lock and went to get all the data. Because the result set was so much larger in their larger system I took the lock for many more milliseconds than I should have. Unknown to me a chunk of code on the criti
4 0.91387743 1178 high scalability-2012-01-20-Stuff The Internet Says On Scalability For January 20, 2012
Introduction: If you’ve got the time, we’ve got the HighScalability: Google+: 90 million users ; Internet 2011 : 2.1 billion Internet users, 1 trillion YouTube views, 5.9 billion mobile subscriptions; Fusion-io: One Billion IOPS ; 12 atoms: size of IBM's new memory bit ; 32 Million: Stack monthly visitors ; Gmail: 350 Million Users ; TimTebow: 1.5 million Tweets Quotable Quotes: Similarity : There is no canonical schema anymore. Instead you should ask: What high-volume queries will I need to serve with my data? Then work backwards from there. @kvirjee : Dis/Agree? -- "there is no problem but scalability, and architecture is its solution" @robpegoraro : Eternal vigilance can be crowdsourced. Didn't Bill Gates say once that 48 bits would always be enough for an ID? Well, Oracle ran out of bits: Fundamental Oracle flaw revealed . 64 bits, that's the ticket, ipv6 went 128 bits. The day Kodak died : We developed the world's first consumer digital camera bu
5 0.90595561 365 high scalability-2008-08-16-Strategy: Serve Pre-generated Static Files Instead Of Dynamic Pages
Introduction: Pre-generating static files is an oldy but a goody, and as Thomas Brox Røst says, it's probably an underused strategy today. At one time this was the dominate technique for structuring a web site. Then the age of dynamic web sites arrived and we spent all our time worrying how to make the database faster and add more caching to recover the speed we had lost in the transition from static to dynamic. Static files have the advantage of being very fast to serve. Read from disk and display. Simple and fast. Especially when caching proxies are used. The issue is how do you bulk generate the initial files, how do you serve the files, and how do you keep the changed files up to date? This is the process Thomas covers in his excellent article Serving static files with Django and AWS - going fast on a budget" , where he explains how he converted 600K thousand previously dynamic pages to static pages for his site Eventseer.net , a service for tracking academic events. Eventseer.n
6 0.8874011 353 high scalability-2008-07-20-Strategy: Front S3 with a Caching Proxy
8 0.88170809 248 high scalability-2008-02-13-What's your scalability plan?
9 0.87765145 149 high scalability-2007-11-12-Scaling Using Cache Farms and Read Pooling
10 0.87315243 1194 high scalability-2012-02-16-A Super Short on the Youporn Stack - 300K QPS and 100 Million Page Views Per Day
11 0.87119853 1231 high scalability-2012-04-20-Stuff The Internet Says On Scalability For April 20, 2012
12 0.86361706 672 high scalability-2009-08-06-An Unorthodox Approach to Database Design : The Coming of the Shard
13 0.86332303 920 high scalability-2010-10-15-Troubles with Sharding - What can we learn from the Foursquare Incident?
14 0.86319035 628 high scalability-2009-06-13-Neo4j - a Graph Database that Kicks Buttox
15 0.86212254 661 high scalability-2009-07-25-Latency is Everywhere and it Costs You Sales - How to Crush it
16 0.86181033 714 high scalability-2009-10-02-HighScalability has Moved to Squarespace.com!
17 0.86178833 825 high scalability-2010-05-10-Sify.com Architecture - A Portal at 3900 Requests Per Second
18 0.86171305 1438 high scalability-2013-04-10-Check Yourself Before You Wreck Yourself - Avocado's 5 Early Stages of Architecture Evolution
19 0.86144519 589 high scalability-2009-05-05-Drop ACID and Think About Data
20 0.86109555 1065 high scalability-2011-06-21-Running TPC-C on MySQL-RDS