high_scalability high_scalability-2008 high_scalability-2008-232 knowledge-graph by maker-knowledge-mining
Source: html
Introduction: OK, I know this site is for scalable web site design. But as there aren't any sites I can find for graceful failure under "slashdotted" like pressure I'll ask here. Does anyone have a sensible way, once you have a "web application" that either won't scale, or can't scale, that you can give some users a good consistent experience and bounce other users to a busy site page. I have seen sites do this to varying degrees, some of which work better than others, but no explanations beyond simply bouncing requests to a "we're busy page server" when you have more than a given number of connections. This is obviously useless as a web page likely requires multiple connection (ignoring keep-alive, pipelining etc) multiple connection to completely render properly. The normal problem is users getting a page and not the "furniture" for that page like images or css. Other problems are having to wait ages to get the busy page or the site being slow even if you do "get in". And some site let
sentIndex sentText sentNum sentScore
1 OK, I know this site is for scalable web site design. [sent-1, score-0.723]
2 But as there aren't any sites I can find for graceful failure under "slashdotted" like pressure I'll ask here. [sent-2, score-0.364]
3 Does anyone have a sensible way, once you have a "web application" that either won't scale, or can't scale, that you can give some users a good consistent experience and bounce other users to a busy site page. [sent-3, score-1.031]
4 I have seen sites do this to varying degrees, some of which work better than others, but no explanations beyond simply bouncing requests to a "we're busy page server" when you have more than a given number of connections. [sent-4, score-0.908]
5 This is obviously useless as a web page likely requires multiple connection (ignoring keep-alive, pipelining etc) multiple connection to completely render properly. [sent-5, score-0.839]
6 The normal problem is users getting a page and not the "furniture" for that page like images or css. [sent-6, score-0.437]
7 Other problems are having to wait ages to get the busy page or the site being slow even if you do "get in". [sent-7, score-0.906]
8 And some site let a user "in" and then as they browse around they get bounced out suddenly to the busy page. [sent-8, score-0.939]
9 Obviously not being the developer for sites I deal with (I am an infrastructure bod) I can't solve the problem where it should have been pre-emptively solved. [sent-9, score-0.427]
10 That is to say I can't write the code to be scalable or re-write the code to do some simple session filtering or the like (and not being a developer I get dirty looks when I point developers at information like your site . [sent-10, score-0.883]
11 I can hear them thinking "how dare you suggest I don't know how to code a web site you lowly infrastructure cretin"). [sent-13, score-0.64]
12 Before developer on-line lynch me I should point out that sometimes the cause of not being able to scale a site is that I can't get in new hardware quick enough, but then who knows when you will get slashdotted right ? [sent-14, score-1.087]
13 So my question applies even when a developer of genius level brilliance has built a unsurpasibly scalable web site for me to run the infrastructure for. [sent-16, score-1.018]
14 My best guess so far is using something like HAProxy to load balance sessions, and then use it's more advanced total session count, and cookie issuing abilities to track users and bounce some at a given "heavy load" point. [sent-17, score-0.765]
15 This isn't ideal as the heavy load point would have to be based on connection counts not server load or server response times, but it's the best I can come up with so far. [sent-18, score-0.514]
16 Also, having mentioned brilliant developers writing great sites not always making my question redundant, could I ask, do people normally think about coping with overload when designing scalable solution - surely they should but I don't see much talk about it. [sent-19, score-0.737]
17 Couldn't a simple Java filter or the equivalent for other things be built into applications ? [sent-20, score-0.077]
18 It'd be nice to have a site that not only scales, but "is nice" when waiting for the infrastructure it runs on to be scaled, which could be several days when you have to purchase new hardware. [sent-21, score-0.466]
wordName wordTfidf (topN-words)
[('site', 0.274), ('busy', 0.271), ('slashdotted', 0.263), ('bounce', 0.208), ('page', 0.174), ('sites', 0.16), ('connection', 0.16), ('developer', 0.159), ('bounced', 0.123), ('bouncing', 0.123), ('brilliance', 0.123), ('lynch', 0.123), ('coping', 0.118), ('ca', 0.109), ('infrastructure', 0.108), ('issuing', 0.107), ('degrees', 0.104), ('pipelining', 0.104), ('ask', 0.104), ('heavy', 0.104), ('dare', 0.1), ('graceful', 0.1), ('ignoring', 0.1), ('sensible', 0.1), ('browse', 0.096), ('overload', 0.096), ('session', 0.095), ('scalable', 0.095), ('get', 0.094), ('surely', 0.093), ('ages', 0.093), ('abilities', 0.092), ('explanations', 0.091), ('genius', 0.091), ('cookie', 0.089), ('varying', 0.089), ('users', 0.089), ('question', 0.088), ('brilliant', 0.087), ('dirty', 0.086), ('alive', 0.085), ('load', 0.085), ('useless', 0.084), ('nice', 0.084), ('suddenly', 0.081), ('web', 0.08), ('point', 0.08), ('suggest', 0.078), ('filter', 0.077), ('render', 0.077)]
simIndex simValue blogId blogTitle
same-blog 1 1.0 232 high scalability-2008-01-29-When things aren't scalable
Introduction: OK, I know this site is for scalable web site design. But as there aren't any sites I can find for graceful failure under "slashdotted" like pressure I'll ask here. Does anyone have a sensible way, once you have a "web application" that either won't scale, or can't scale, that you can give some users a good consistent experience and bounce other users to a busy site page. I have seen sites do this to varying degrees, some of which work better than others, but no explanations beyond simply bouncing requests to a "we're busy page server" when you have more than a given number of connections. This is obviously useless as a web page likely requires multiple connection (ignoring keep-alive, pipelining etc) multiple connection to completely render properly. The normal problem is users getting a page and not the "furniture" for that page like images or css. Other problems are having to wait ages to get the busy page or the site being slow even if you do "get in". And some site let
2 0.17028362 1 high scalability-2007-07-06-Start Here
Introduction: This page is here to help you get started using High Scalability. Here are a few useful topics to get you going... Why does the High Scalability site exist? Good things to read. Participate by adding your own links to interesting sites and articles. Participate by signing up for the RSS feed. Consider the many benefits of registering as a user. How do I get notification of content and comment changes? Contact High Scalability. About. Why does the High Scalability site exist? To help you build successful scalable websites. This site tries to bring together all the lore, art, science, practice, and experience of building scalable websites into one place so you can learn how to build your website with confidence. When it becomes clear you must grow your website or die, most people have no idea where to start. It's not a skill you learn in school or pick up from a magazine article on a plane flight home. No, building scalable systems is a body o
3 0.14058894 638 high scalability-2009-06-26-PlentyOfFish Architecture
Introduction: Update 5 : PlentyOfFish Update - 6 Billion Pageviews And 32 Billion Images A Month Update 4 : Jeff Atwood costs out Markus' scale up approach against a scale out approach and finds scale up wanting. The discussion in the comments is as interesting as the article. My guess is Markus doesn't want to rewrite his software to work across a scale out cluster so even if it's more expensive scale up works better for his needs. Update 3 : POF now has 200 million images and serves 10,000 images served per second. They'll be moving to a 250,000 IOPS RamSan to handle the load. Also upgraded to a core database machine with 512 GB of RAM, 32 CPU’s, SQLServer 2008 and Windows 2008. Update 2 : This seems to be a POF Peer1 love fest infomercial . It's pretty content free, but the production values are high. Lots of quirky sounds and fish swimming on the screen. Update : by Facebook standards Read/WriteWeb says POF is worth a cool one billion dollars . It helps to talk like Dr. Evil whe
4 0.13970159 106 high scalability-2007-10-02-Secrets to Fotolog's Scaling Success
Introduction: Fotolog, a social blogging site centered around photos, grew from about 300 thousand users in 2004 to over 11 million users in 2007. Though they initially experienced the inevitable pains of rapid growth, they overcame their problems and now manage over 300 million photos and 800,000 new photos are added each day. Generating all that fabulous content are 20 million unique monthly visitors and a volunteer army of 30,000 new users each day. They did so well a very impressed suitor bought them out for a cool $90 million. That's scale meets success by anyone standards. How did they do it? Site: http://www.fotolog.com Information Sources Scaling the World's Largest Photo Blogging Community Congrats to Fotolog on $90mm sale to Hi-Media Fotolog overtaking Flickr? Fotolog Hits 11 Million Members and 300 Million Photos Posted Site of the Week: Fotolog.com by PC Magazine CEO John Borthwick's Blog . DBA Frank Mash's Blog Fotolog, lessons learnt by John B
5 0.13914903 1361 high scalability-2012-11-22-Gone Fishin': PlentyOfFish Architecture
Introduction: Other than StackOverflow , PlentyOfFish is perhaps the most spectacular example of scale-up architectures working for what your average sane person would consider a large system. It doesn't hurt that it's also a sexy story. Update 5 : PlentyOfFish Update - 6 Billion Pageviews And 32 Billion Images A Month Update 4 : Jeff Atwood costs out Markus' scale up approach against a scale out approach and finds scale up wanting. The discussion in the comments is as interesting as the article. My guess is Markus doesn't want to rewrite his software to work across a scale out cluster so even if it's more expensive scale up works better for his needs. Update 3 : POF now has 200 million images and serves 10,000 images served per second. They'll be moving to a 250,000 IOPS RamSan to handle the load. Also upgraded to a core database machine with 512 GB of RAM, 32 CPU’s, SQLServer 2008 and Windows 2008. Update 2 : This seems to be a POF Peer1 love fest infomercial . It's pretty cont
6 0.1380821 70 high scalability-2007-08-22-How many machines do you need to run your site?
7 0.13531461 691 high scalability-2009-08-31-Squarespace Architecture - A Grid Handles Hundreds of Millions of Requests a Month
8 0.13356555 1123 high scalability-2011-09-23-The Real News is Not that Facebook Serves Up 1 Trillion Pages a Month…
9 0.13298692 10 high scalability-2007-07-15-Book: Building Scalable Web Sites
10 0.12866759 721 high scalability-2009-10-13-Why are Facebook, Digg, and Twitter so hard to scale?
11 0.12673429 1102 high scalability-2011-08-22-Strategy: Run a Scalable, Available, and Cheap Static Site on S3 or GitHub
12 0.12473348 1438 high scalability-2013-04-10-Check Yourself Before You Wreck Yourself - Avocado's 5 Early Stages of Architecture Evolution
13 0.12235215 617 high scalability-2009-06-04-New Book: Even Faster Web Sites: Performance Best Practices for Web Developers
14 0.12187436 834 high scalability-2010-06-01-Web Speed Can Push You Off of Google Search Rankings! What Can You Do?
15 0.12049437 240 high scalability-2008-02-05-Handling of Session for a site running from more than 1 data center
16 0.1163455 517 high scalability-2009-02-21-Google AppEngine - A Second Look
17 0.11514954 276 high scalability-2008-03-15-New Website Design Considerations
18 0.11265522 202 high scalability-2008-01-06-Email Architecture
19 0.11244863 298 high scalability-2008-04-07-Lazy web sites run faster
20 0.11244836 965 high scalability-2010-12-29-Pinboard.in Architecture - Pay to Play to Keep a System Small
topicId topicWeight
[(0, 0.211), (1, 0.074), (2, -0.023), (3, -0.152), (4, 0.043), (5, -0.112), (6, -0.075), (7, 0.002), (8, -0.008), (9, 0.03), (10, -0.065), (11, -0.004), (12, -0.038), (13, -0.012), (14, 0.066), (15, -0.095), (16, 0.043), (17, -0.039), (18, 0.071), (19, 0.056), (20, 0.016), (21, -0.015), (22, -0.02), (23, -0.01), (24, -0.054), (25, -0.082), (26, -0.019), (27, 0.014), (28, 0.045), (29, -0.071), (30, 0.061), (31, 0.008), (32, 0.026), (33, -0.085), (34, 0.011), (35, 0.038), (36, 0.014), (37, 0.005), (38, -0.063), (39, 0.002), (40, -0.014), (41, 0.032), (42, -0.038), (43, 0.051), (44, 0.03), (45, -0.02), (46, 0.041), (47, -0.019), (48, 0.026), (49, 0.001)]
simIndex simValue blogId blogTitle
same-blog 1 0.99096054 232 high scalability-2008-01-29-When things aren't scalable
Introduction: OK, I know this site is for scalable web site design. But as there aren't any sites I can find for graceful failure under "slashdotted" like pressure I'll ask here. Does anyone have a sensible way, once you have a "web application" that either won't scale, or can't scale, that you can give some users a good consistent experience and bounce other users to a busy site page. I have seen sites do this to varying degrees, some of which work better than others, but no explanations beyond simply bouncing requests to a "we're busy page server" when you have more than a given number of connections. This is obviously useless as a web page likely requires multiple connection (ignoring keep-alive, pipelining etc) multiple connection to completely render properly. The normal problem is users getting a page and not the "furniture" for that page like images or css. Other problems are having to wait ages to get the busy page or the site being slow even if you do "get in". And some site let
2 0.85766059 8 high scalability-2007-07-12-Should I use LAMP or Windows?
Introduction: Hi, I stumb l ed on your s i te and I am th i nking about start i ng a website. I haven't rece i ved a good answer about what I shou l d use to bui l d i t, so I thought I wou l d give it a shot. I am a w i ndows guy. I know .Net and ASP and how to bu i ld web s i tes using that stack. But I not i ce most sites use LAMP and that's what most people ta l k about using. What's wrong w i th using Windows? .Net Programmer
3 0.85303438 632 high scalability-2009-06-15-starting small with growth in mind
Introduction: Hello all, I'm working on a web site that might totally flop or it might explode to be the next facebook/flickr/digg/etc. Since I really don't know how popular the site will be I don't want to spend a ton of money on the hardware/hosting right away but I want to be able to scale it easily if it does grow rapidly. With this in mind, what would be the best approach to launch the site? Thanks, Dan
4 0.82409066 965 high scalability-2010-12-29-Pinboard.in Architecture - Pay to Play to Keep a System Small
Introduction: How do you keep a system small enough, while still being successful, that a simple scale-up strategy becomes the preferred architecture? StackOverflow , for example, could stick with a tool chain they were comfortable with because they had a natural brake on how fast they could grow: there are only so many programmers in the world. If this doesn't work for you, here's another natural braking strategy to consider: charge for your service . Paul Houle summarized this nicely as: avoid scaling problems by building a service that's profitable at a small scale . This interesting point, one I hadn't properly considered before, was brought up by Maciej Ceglowski, co-founder of Pinboard.in , in an interview with Leo Laporte and Amber MacArthur on their their net@night show. Pinboard is a lean, mean, pay for bookmarking machine, a timely replacement for the nearly departed Delicious. And as a self professed anti-social bookmarking site, it emphasizes speed over socializing . Maciej
5 0.82160026 1 high scalability-2007-07-06-Start Here
Introduction: This page is here to help you get started using High Scalability. Here are a few useful topics to get you going... Why does the High Scalability site exist? Good things to read. Participate by adding your own links to interesting sites and articles. Participate by signing up for the RSS feed. Consider the many benefits of registering as a user. How do I get notification of content and comment changes? Contact High Scalability. About. Why does the High Scalability site exist? To help you build successful scalable websites. This site tries to bring together all the lore, art, science, practice, and experience of building scalable websites into one place so you can learn how to build your website with confidence. When it becomes clear you must grow your website or die, most people have no idea where to start. It's not a skill you learn in school or pick up from a magazine article on a plane flight home. No, building scalable systems is a body o
6 0.78580618 614 high scalability-2009-06-01-Guess How Many Users it Takes to Kill Your Site?
7 0.78264391 298 high scalability-2008-04-07-Lazy web sites run faster
8 0.7811923 10 high scalability-2007-07-15-Book: Building Scalable Web Sites
9 0.77935129 71 high scalability-2007-08-22-Profiling WEB applications
10 0.77930206 711 high scalability-2009-09-22-How Ravelry Scales to 10 Million Requests Using Rails
11 0.77658534 124 high scalability-2007-10-16-How Scalable are Single Page Ajax Apps?
12 0.77568316 344 high scalability-2008-06-09-FaceStat's Rousing Tale of Scaling Woe and Wisdom Won
13 0.77420533 106 high scalability-2007-10-02-Secrets to Fotolog's Scaling Success
14 0.76511598 1108 high scalability-2011-08-31-Pud is the Anti-Stack - Windows, CFML, Dropbox, Xeround, JungleDisk, ELB
15 0.75861603 611 high scalability-2009-05-31-Need help on Site loading & database optimization - URGENT
16 0.75479132 321 high scalability-2008-05-17-WebSphere Commerce High Availability and Performance Configurations
17 0.75130242 265 high scalability-2008-03-03-Two data streams for a happy website
18 0.75004667 158 high scalability-2007-11-17-Can How Bees Solve their Load Balancing Problems Help Build More Scalable Websites?
19 0.74799979 175 high scalability-2007-12-05-how to: Load Balancing with iis
20 0.74532819 1102 high scalability-2011-08-22-Strategy: Run a Scalable, Available, and Cheap Static Site on S3 or GitHub
topicId topicWeight
[(1, 0.155), (2, 0.235), (10, 0.03), (30, 0.031), (57, 0.178), (61, 0.124), (77, 0.016), (79, 0.088), (85, 0.013), (94, 0.052)]
simIndex simValue blogId blogTitle
1 0.98098296 159 high scalability-2007-11-18-Reverse Proxy
Introduction: Hi, I saw an year ago that Netapp sold netcache to blu-coat, my site is a heavy NetCache user and we cached 83% of our site. We tested with Blue-coat and F5 WA and we are not getting same performce as NetCache. Any of you guys have the same issue? or somebody knows another product can handle much traffic? Thanks Rodrigo
2 0.97398561 968 high scalability-2011-01-04-Map-Reduce With Ruby Using Hadoop
Introduction: A demonstration, with repeatable steps, of how to quickly fire-up a Hadoop cluster on Amazon EC2, load data onto the HDFS (Hadoop Distributed File-System), write map-reduce scripts in Ruby and use them to run a map-reduce job on your Hadoop cluster. You will not need to ssh into the cluster, as all tasks are run from your local machine. Below I am using my MacBook Pro as my local machine, but the steps I have provided should be reproducible on other platforms running bash and Java. Fire-Up Your Hadoop Cluster I choose the Cloudera distribution of Hadoop which is still 100% Apache licensed, but has some additional benefits. One of these benefits is that it is released by Doug Cutting , who started Hadoop and drove it’s development at Yahoo! He also started Lucene , which is another of my favourite Apache Projects, so I have good faith that he knows what he is doing. Another benefit, as you will see, is that it is simple to fire-up a Hadoop cluster. I am going to use C
3 0.97139442 433 high scalability-2008-10-29-CTL - Distributed Control Dispatching Framework
Introduction: CTL is a flexible distributed control dispatching framework that enables you to break management processes into reusable control modules and execute them in distributed fashion over the network . From their website: CTL is a flexible distributed control dispatching framework that enables you to break management processes into reusable control modules and execute them in distributed fashion over the network. What does CTL do? CTL helps you leverage your current scripts and tools to easily automate any kind of distributed systems management or application provisioning task. Its good for simplifiying large-scale scripting efforts or as another tool in your toolbox that helps you speed through your daily mix of ad-hoc administration tasks. What are CTL's features? CTL has many features, but the general highlights are: * Execute sophisticated procedures in distributed environments - Aren't you tired of writing and then endlessly modifying scripts that loop over nodes and invoke remot
4 0.97037941 1144 high scalability-2011-11-17-Five Misconceptions on Cloud Portability
Introduction: The term "cloud portability" is often considered a synonym for "Cloud API portability," which implies a series of misconceptions. If we break away from dogma, we can find that what we really looking for in cloud portability is Application portability between clouds which can be a vastly simpler requirement, as we can achieve application portability without settling on a common Cloud API. In this post i'll be covering five common misconceptions people have WRT to cloud portability. Cloud portability = Cloud API portability . API portability is easy; cloud API portability is not. The main incentive for Cloud Portability is - Avoiding Vendor lock-in .Cloud portability is more about business agility than it is about vendor lock-in. Cloud portability isn’t for startups . Every startup that is expecting rapid growth should re-examine their deployments and plan for cloud portability rather than wait to be forced to make the switch when you are least prepared to do so.
5 0.94888526 1211 high scalability-2012-03-19-LinkedIn: Creating a Low Latency Change Data Capture System with Databus
Introduction: This is a guest post by Siddharth Anand , a senior member of LinkedIn's Distributed Data Systems team. Over the past 3 years, I've had the good fortune to work with many emerging NoSQL products in the context of supporting the needs of a high-traffic, customer facing web site. In 2010, I helped Netflix to successfully transition its web scale use-cases from Oracle to SimpleDB , AWS' hosted database service. On completion of that migration, we started a second migration, this time from SimpleDB to Cassandra. The first transition was key to our move from our own data center to AWS' cloud. The second was key to our expansion from one AWS Region to multiple geographically-distributed Regions -- today Netflix serves traffic out of two AWS Regions, one in Virginia, the other in Ireland ( F1 ). Both of these transitions have been successful, but have involved integration pain points such as the creation of database replication technology. In December 2011, I moved to LinkedIn's D
6 0.94826347 807 high scalability-2010-04-09-Vagrant - Build and Deploy Virtualized Development Environments Using Ruby
7 0.94826275 731 high scalability-2009-10-28-Need for change in your IT infrastructure
8 0.94610578 1138 high scalability-2011-11-07-10 Core Architecture Pattern Variations for Achieving Scalability
9 0.93035692 6 high scalability-2007-07-11-Friendster Architecture
10 0.92936486 218 high scalability-2008-01-17-Moving old to new. Do not be afraid of the re-write -- but take some help
same-blog 11 0.92351592 232 high scalability-2008-01-29-When things aren't scalable
12 0.9184137 553 high scalability-2009-04-03-Collectl interface to Ganglia - any interest?
13 0.91650689 855 high scalability-2010-07-11-So, Why is Twitter Really Not Using Cassandra to Store Tweets?
14 0.91018271 351 high scalability-2008-07-16-The Mother of All Database Normalization Debates on Coding Horror
15 0.88729477 1385 high scalability-2013-01-11-Stuff The Internet Says On Scalability For January 11, 2013
16 0.88605183 972 high scalability-2011-01-11-Google Megastore - 3 Billion Writes and 20 Billion Read Transactions Daily
17 0.88379544 1507 high scalability-2013-08-26-Reddit: Lessons Learned from Mistakes Made Scaling to 1 Billion Pageviews a Month
18 0.88314402 1003 high scalability-2011-03-14-6 Lessons from Dropbox - One Million Files Saved Every 15 minutes
19 0.88193876 1087 high scalability-2011-07-26-Web 2.0 Killed the Middleware Star
20 0.87863868 857 high scalability-2010-07-13-DbShards Part Deux - The Internals