high_scalability high_scalability-2008 high_scalability-2008-302 knowledge-graph by maker-knowledge-mining
Source: html
Introduction: Hi, I am an owner of an large community website and currently we are having problems with our database architecture. We are using 2 database servers and spread tables across them to divide read/writes. We have about 90% reads and 10% writes. We use Memcached on all our webservers to cache as much as we can, traffic is load balanced between webservers. We have 2 extra servers ready to put to use! We have looked into a couple of solution so far: Continuent Uni/Cluster aka Sequoia -> Commercial version way too expensive and Java isn't as fast as it suppose to be. MySQL Proxy -> We couldn't find any good example on how to create a master - master with failover scenario. MySQL Clustering -> Seems to be not mature enough, had a lot of performance issues when we tried to go online with it. MySQL DRDB HA -> Only good for failover, cannot be scaled! MySQL Replication -> Well don't get me started ;) So now I turn to you guys to help me out, I am with my hands in my hair a
sentIndex sentText sentNum sentScore
1 Hi, I am an owner of an large community website and currently we are having problems with our database architecture. [sent-1, score-0.431]
2 We are using 2 database servers and spread tables across them to divide read/writes. [sent-2, score-0.523]
3 We use Memcached on all our webservers to cache as much as we can, traffic is load balanced between webservers. [sent-4, score-0.397]
4 We have looked into a couple of solution so far: Continuent Uni/Cluster aka Sequoia -> Commercial version way too expensive and Java isn't as fast as it suppose to be. [sent-6, score-0.858]
5 MySQL Proxy -> We couldn't find any good example on how to create a master - master with failover scenario. [sent-7, score-0.703]
6 MySQL Clustering -> Seems to be not mature enough, had a lot of performance issues when we tried to go online with it. [sent-8, score-0.453]
7 MySQL DRDB HA -> Only good for failover, cannot be scaled! [sent-9, score-0.076]
8 MySQL Replication -> Well don't get me started ;) So now I turn to you guys to help me out, I am with my hands in my hair and see the site still growning and performance slowly getting to its limit. [sent-10, score-1.104]
wordName wordTfidf (topN-words)
[('failover', 0.235), ('sequoia', 0.233), ('drdb', 0.226), ('hair', 0.226), ('suppose', 0.22), ('continuent', 0.205), ('master', 0.196), ('aka', 0.195), ('webservers', 0.192), ('mature', 0.178), ('help', 0.169), ('divide', 0.167), ('hands', 0.167), ('ha', 0.163), ('slowly', 0.153), ('looked', 0.149), ('commercial', 0.142), ('balanced', 0.139), ('clustering', 0.135), ('guys', 0.134), ('proxy', 0.131), ('scaled', 0.13), ('tried', 0.124), ('couple', 0.122), ('owner', 0.116), ('extra', 0.115), ('spread', 0.109), ('tables', 0.104), ('turn', 0.101), ('community', 0.094), ('ready', 0.091), ('reads', 0.089), ('far', 0.088), ('started', 0.087), ('expensive', 0.087), ('currently', 0.085), ('version', 0.085), ('online', 0.083), ('seems', 0.083), ('replication', 0.079), ('memcached', 0.077), ('servers', 0.077), ('good', 0.076), ('website', 0.07), ('issues', 0.068), ('java', 0.068), ('getting', 0.067), ('traffic', 0.066), ('database', 0.066), ('put', 0.066)]
simIndex simValue blogId blogTitle
same-blog 1 1.0 302 high scalability-2008-04-10-Mysql scalability and failover...
Introduction: Hi, I am an owner of an large community website and currently we are having problems with our database architecture. We are using 2 database servers and spread tables across them to divide read/writes. We have about 90% reads and 10% writes. We use Memcached on all our webservers to cache as much as we can, traffic is load balanced between webservers. We have 2 extra servers ready to put to use! We have looked into a couple of solution so far: Continuent Uni/Cluster aka Sequoia -> Commercial version way too expensive and Java isn't as fast as it suppose to be. MySQL Proxy -> We couldn't find any good example on how to create a master - master with failover scenario. MySQL Clustering -> Seems to be not mature enough, had a lot of performance issues when we tried to go online with it. MySQL DRDB HA -> Only good for failover, cannot be scaled! MySQL Replication -> Well don't get me started ;) So now I turn to you guys to help me out, I am with my hands in my hair a
2 0.21669894 102 high scalability-2007-09-27-Product: Sequoia Database Clustering Technology
Introduction: Sequoia is a transparent middleware solution offering clustering, load balancing and failover services for any database. Sequoia is the continuation of the C-JDBC project. The database is distributed and replicated among several nodes and Sequoia balances the queries among these nodes. Sequoia handles node and network failures with transparent failover. It also provides support for hot recovery, online maintenance operations and online upgrades. Features in a nutshell No modification of existing applications or databases. Operational with any database providing a JDBC driver. High availability provided by advanced RAIDb technology. Transparent failover and recovery capabilities. Performance scalability with unique load balancing and query result caching features. Integrated JMX-based administration and monitoring. 100% Java implementation allowing portability across platforms with a JRE 1.4 or greater. Open source licensed under Apache v2 license. Professi
3 0.13961914 63 high scalability-2007-08-09-Lots of questions for high scalability - high availability
Introduction: Hey, I do have a website that I would like to scale. Right now we have 10 servers but this does not scale well. I know how to deal with my apache web servers but have problems with sql servers. I would like to use the "scale out" system and add servers when we need. We have over 100Gb of data for mysql and we tried to have around 20G per server. It works well except that if a server goes down then 1/5 of the user can't access the website. We could use replication but we would need to at least double sql servers to replicate each server. And maybe in the future it's not gonna be enough we would need maybe 3 slaves per master ... well I don't really like this idea. I would prefer to have 8 servers that all deal with data from the 5 servers we have right now and then we could add new servers when we need. I looked at NFS but that does not seem to be a good idea for SQL servers ? Can you confirm?
4 0.13639234 1440 high scalability-2013-04-15-Scaling Pinterest - From 0 to 10s of Billions of Page Views a Month in Two Years
Introduction: Pinterest has been riding an exponential growth curve, doubling every month and half. They've gone from 0 to 10s of billions of page views a month in two years, from 2 founders and one engineer to over 40 engineers, from one little MySQL server to 180 Web Engines, 240 API Engines, 88 MySQL DBs (cc2.8xlarge) + 1 slave each, 110 Redis Instances, and 200 Memcache Instances.Stunning growth. So what's Pinterest's story? To tell their story we have our bards, Pinterest'sYashwanth NelapatiandMarty Weiner, who tell the dramatic story of Pinterest's architecture evolution in a talk titledScaling Pinterest. This is the talk they would have liked to hear a year and half ago when they were scaling fast and there were a lot of options to choose from. And they made a lot of incorrect choices.This is a great talk. It's full of amazing details. It's also very practical, down to earth, and it contains strategies adoptable by nearly anyone. Highly recommended.Two of my favorite lessons from the talk:Arc
5 0.1330044 151 high scalability-2007-11-12-a8cjdbc - Database Clustering via JDBC
Introduction: Practically any software project nowadays could not survive without a database (DBMS) backend storing all the business data that is vital to you and/or your customers. When projects grow larger, the amount of data usually grows larger exponentially. So you start moving the DBMS to a separate server to gain more speed and capacity. Which is all good and healthy but you do not gain any extra safety for this business data. You might be backing up your database once a day so in case the database server crashes you don't lose EVERYTHING, but how much can you really afford to lose? Well clearly this depends on what kind of data you are storing. In our case the users of our solutions use our software products to do their everyday (all day) work. They have "everything" they need for their business stored in the database we are providing. So is 24 hours of data loss acceptable? No, not really. One hour? Maybe. But what we really want is a second database running with the EXACT same data. We
6 0.13276215 86 high scalability-2007-09-09-Clustering Solution
7 0.12901451 884 high scalability-2010-08-23-6 Ways to Kill Your Servers - Learning How to Scale the Hard Way
8 0.12107085 4 high scalability-2007-07-10-Webcast: Advanced Database High Availability and Scalability Solutions
9 0.1206078 111 high scalability-2007-10-04-Number of load balanced servers
10 0.11972032 360 high scalability-2008-08-04-A Bunch of Great Strategies for Using Memcached and MySQL Better Together
11 0.11715622 117 high scalability-2007-10-08-Paper: Understanding and Building High Availability-Load Balanced Clusters
12 0.10943552 196 high scalability-2007-12-30-MySQL clustering strategies and comparisions
13 0.10691518 297 high scalability-2008-04-05-Skype Plans for PostgreSQL to Scale to 1 Billion Users
14 0.1024247 380 high scalability-2008-09-05-Product: Tungsten Replicator
15 0.10165533 538 high scalability-2009-03-16-Are Cloud Based Memory Architectures the Next Big Thing?
16 0.10146273 1501 high scalability-2013-08-13-In Memoriam: Lavabit Architecture - Creating a Scalable Email Service
17 0.098098807 1508 high scalability-2013-08-28-Sean Hull's 20 Biggest Bottlenecks that Reduce and Slow Down Scalability
18 0.093884826 687 high scalability-2009-08-24-How Google Serves Data from Multiple Datacenters
19 0.091566063 313 high scalability-2008-05-02-Friends for Sale Architecture - A 300 Million Page View-Month Facebook RoR App
20 0.090352625 672 high scalability-2009-08-06-An Unorthodox Approach to Database Design : The Coming of the Shard
topicId topicWeight
[(0, 0.158), (1, 0.046), (2, -0.051), (3, -0.119), (4, 0.022), (5, 0.024), (6, -0.028), (7, -0.079), (8, -0.007), (9, -0.037), (10, -0.047), (11, -0.025), (12, -0.004), (13, 0.035), (14, 0.051), (15, -0.025), (16, 0.014), (17, -0.023), (18, -0.031), (19, 0.033), (20, -0.008), (21, 0.01), (22, -0.064), (23, 0.037), (24, -0.035), (25, 0.063), (26, 0.063), (27, 0.002), (28, -0.015), (29, -0.015), (30, 0.006), (31, -0.003), (32, -0.106), (33, -0.027), (34, -0.019), (35, 0.026), (36, -0.009), (37, -0.024), (38, 0.019), (39, 0.038), (40, 0.02), (41, -0.016), (42, -0.049), (43, 0.027), (44, 0.0), (45, -0.025), (46, 0.06), (47, 0.032), (48, 0.026), (49, -0.008)]
simIndex simValue blogId blogTitle
same-blog 1 0.96417993 302 high scalability-2008-04-10-Mysql scalability and failover...
Introduction: Hi, I am an owner of an large community website and currently we are having problems with our database architecture. We are using 2 database servers and spread tables across them to divide read/writes. We have about 90% reads and 10% writes. We use Memcached on all our webservers to cache as much as we can, traffic is load balanced between webservers. We have 2 extra servers ready to put to use! We have looked into a couple of solution so far: Continuent Uni/Cluster aka Sequoia -> Commercial version way too expensive and Java isn't as fast as it suppose to be. MySQL Proxy -> We couldn't find any good example on how to create a master - master with failover scenario. MySQL Clustering -> Seems to be not mature enough, had a lot of performance issues when we tried to go online with it. MySQL DRDB HA -> Only good for failover, cannot be scaled! MySQL Replication -> Well don't get me started ;) So now I turn to you guys to help me out, I am with my hands in my hair a
2 0.77048868 884 high scalability-2010-08-23-6 Ways to Kill Your Servers - Learning How to Scale the Hard Way
Introduction: This is a guest post by Steffen Konerow, author of the High Performance Blog . Learning how to scale isn’t easy without any prior experience. Nowadays you have plenty of websites like highscalability.com to get some inspiration, but unfortunately there is no solution that fits all websites and needs. You still have to think on your own to find a concept that works for your requirements. So did I. A few years ago, my bosses came to me and said “We’ve got a new project for you. It’s the relaunch of a website that has already 1 million users a month. You have to build the website and make sure we’ll be able to grow afterwards”. I was already an experienced coder, but not in these dimensions, so I had to start learning how to scale – the hard way. The software behind the website was a PHP content management system, based on Smarty and MySQL. The first task was finding a proper hosting company who had the experience and would also manage the servers for us. After some researc
3 0.72284228 248 high scalability-2008-02-13-What's your scalability plan?
Introduction: How do you plan to scale your system as you reach predictable milestones? This topic came up in another venue and it reminded me about a great comment an Anonymous wrote a while ago and I wanted to make sure that comment didn't get lost. The Anonymous scaling plan was relatively simple and direct: My two cents on what I'm using to start a website from scratch using a single server for now. Later, I'll scale out horizontally when the need arises. Phase 1 Single Server, Dual Quad-Core 2.66, 8gb RAM, 500gb Disk Raid 10 OS: Fedora 8. You could go with pretty much any Linux though. I like Fedora 8 best for servers. Proxy Cache: Varnish - it is way faster than Squid per my own benchmarks. Squid chokes bigtime. Web Server: Lighttpd - faster than Apache 2 and easier to configure for me. Object Cache: Memcached. Very scalable. PHP Cache: APC. Easy to configure and seems to work fine. Language: PHP 5 - no bloated frameworks, waste of time for me. You spend too mu
4 0.71754402 86 high scalability-2007-09-09-Clustering Solution
Introduction: Hi, I'm i nterested in peop l es thoughts on the best choice for a database clustering so l ution. I have a database that is most l y varchars and numbers that doesn't store any b i nary data at a l l. It's used at about 70% read and 30% wr i tes - though we're using memcached at the moment so it's not rea l ly hit that hard. We're current l y using mysql w i th m/cluster, but are interested in a new so l ution. Possib l e candidate so far are unic l uster (which doesn't seem mature yet.) or DRBD. Had anyone had a simi l ar experience and can make any suggest i ons? Thanks
5 0.66900694 118 high scalability-2007-10-09-High Load on production Webservers after Sourcecode sync
Introduction: Hi everybody :) We have a bunch of webservers (about 14 at th i s time) running Apache. As app l ication framework we're using PHP with the APC Cache insta l led to improve performance. For load ba l ancing we're using a Big F5 system with dynamic ratio (SNMP driven) To sync new/updated sourcecode we're using subversion to "automaticly" update these servers w i th our latest software relases. After updating the new source to these production servers the load of the mashines is ra i sing to hell. While updating the servers, they are sti l l in "production", serving webpages to the users. Otherwise the process of updating wou l d take ages. Most of the time we're only updating in the morning hours while less users are online, because of the above issue. My guess is, that the load is raising that h i gh, because APC needs to recompile a bunch of new f i les each time. Before and while compi l ing the performance simply is "bad". My goal would be t
6 0.66497523 808 high scalability-2010-04-12-Poppen.de Architecture
7 0.65936637 602 high scalability-2009-05-17-Scaling Django Web Apps by Mike Malone
8 0.65916681 276 high scalability-2008-03-15-New Website Design Considerations
9 0.65532136 690 high scalability-2009-08-31-Scaling MySQL on Amazon Web Services
10 0.65523255 383 high scalability-2008-09-10-Shard servers -- go big or small?
11 0.65451634 68 high scalability-2007-08-20-TypePad Architecture
12 0.65290999 389 high scalability-2008-09-23-How to Scale with Ruby on Rails
13 0.65195996 102 high scalability-2007-09-27-Product: Sequoia Database Clustering Technology
14 0.64809519 151 high scalability-2007-11-12-a8cjdbc - Database Clustering via JDBC
15 0.64687663 1041 high scalability-2011-05-15-Building a Database remote availability site
16 0.63925105 554 high scalability-2009-04-04-Digg Architecture
17 0.63838583 283 high scalability-2008-03-18-Shared filesystem on EC2
18 0.63457996 675 high scalability-2009-08-08-1dbase vs. many and cloud hosting vs. dedicated server(s)?
19 0.63394713 63 high scalability-2007-08-09-Lots of questions for high scalability - high availability
20 0.63136435 711 high scalability-2009-09-22-How Ravelry Scales to 10 Million Requests Using Rails
topicId topicWeight
[(0, 0.097), (1, 0.142), (2, 0.241), (10, 0.167), (47, 0.086), (61, 0.054), (85, 0.105)]
simIndex simValue blogId blogTitle
same-blog 1 0.92619824 302 high scalability-2008-04-10-Mysql scalability and failover...
Introduction: Hi, I am an owner of an large community website and currently we are having problems with our database architecture. We are using 2 database servers and spread tables across them to divide read/writes. We have about 90% reads and 10% writes. We use Memcached on all our webservers to cache as much as we can, traffic is load balanced between webservers. We have 2 extra servers ready to put to use! We have looked into a couple of solution so far: Continuent Uni/Cluster aka Sequoia -> Commercial version way too expensive and Java isn't as fast as it suppose to be. MySQL Proxy -> We couldn't find any good example on how to create a master - master with failover scenario. MySQL Clustering -> Seems to be not mature enough, had a lot of performance issues when we tried to go online with it. MySQL DRDB HA -> Only good for failover, cannot be scaled! MySQL Replication -> Well don't get me started ;) So now I turn to you guys to help me out, I am with my hands in my hair a
Introduction: Toy solutions solving Twitter’s “problems” are a favorite scalability trope. Everybody has this idea that Twitter is easy. With a little architectural hand waving we have a scalable Twitter, just that simple. Well, it’s not that simple as Raffi Krikorian , VP of Engineering at Twitter, describes in his superb and very detailed presentation on Timelines at Scale . If you want to know how Twitter works - then start here. It happened gradually so you may have missed it, but Twitter has grown up. It started as a struggling three-tierish Ruby on Rails website to become a beautifully service driven core that we actually go to now to see if other services are down. Quite a change. Twitter now has 150M world wide active users, handles 300K QPS to generate timelines, and a firehose that churns out 22 MB/sec. 400 million tweets a day flow through the system and it can take up to 5 minutes for a tweet to flow from Lady Gaga’s fingers to her 31 million followers. A couple o
3 0.87582701 1331 high scalability-2012-10-02-An Epic TripAdvisor Update: Why Not Run on the Cloud? The Grand Experiment.
Introduction: This is a guest post by Shawn Hsiao , Luke Massa , and Victor Luu . Shawn runs TripAdvisor ’s Technical Operations team, Luke and Victor interned on his team this past summer. This post is introduced by Andy Gelfond , TripAdvisor’s head of engineering. It's been a little over a year since our last post about the TripAdvisor architecture . It has been an exciting year. Our business and team continues to grow, we are now an independent public company, and we have continued to keep/scale our development process and culture as we have grown - we still run dozens of independent teams, and each team continues to work across the entire stack. All that has changed are the numbers: 56M visitors per month 350M+ pages requests a day 120TB+ of warehouse data running on a large Hadoop cluster, and quickly growing We also had a very successful college intern program that brought on over 60 interns this past summer, all who were quickly on boarded and doing the same kind of work a
4 0.87266046 1631 high scalability-2014-04-14-How do you even do anything without using EBS?
Introduction: In a recent thread on Hacker News discussing recent AWS price changes , seldo mentioned they use AWS for business, they just never use EBS on AWS. A good question was asked: How do you even do anything without using EBS? Amazon certainly makes using EBS the easiest path. And EBS has a better reliability record as of late, but it's still often recommended to not use EBS. This avoids a single point of failure at the cost of a lot of complexity, though as AWS uses EBS internally, not using EBS may not save you if you use other AWS services like RDS or ELB. If you don't want to use EBS, it's hard to know where to even start. A dilemma to which Kevin Nuckolls gives a great answer : Well, you break your services out onto stateless and stateful machines. After that, you make sure that each of your stateful services is resilient to individual node failure. I prefer to believe that if you can't roll your entire infrastructure over to new nodes monthly then you're unprepared fo
5 0.86853439 792 high scalability-2010-03-10-How FarmVille Scales - The Follow-up
Introduction: Several readers had follow-up questions in response to How FarmVille Scales to Harvest 75 Million Players a Month . Here are Luke's response to those questions (and a few of mine). How does social networking makes things easier or harder? The primary interesting aspect of social networking games is how you wind up with a graph of connected users who need to be access each other's data on a frequent basis. This makes the overall dataset difficult if not impossible to partition. What are examples of the Facebook calls you try to avoid and how they impact game play? We can make a call for facebook friend data to retrieve information about your friends playing the game. Normally, we show a friend ladder at the bottom of the game that shows friend information, including name and facebook photo. Can you say where your cache is, what form it takes, and how much cached there is? Do you have a peering relationship with Facebook, as one might expect at that bandwidth?
6 0.86667353 271 high scalability-2008-03-08-Product: DRBD - Distributed Replicated Block Device
7 0.86545956 1004 high scalability-2011-03-14-Twitter by the Numbers - 460,000 New Accounts and 140 Million Tweets Per Day
8 0.85679018 1585 high scalability-2014-01-24-Stuff The Internet Says On Scalability For January 24th, 2014
9 0.85633361 1259 high scalability-2012-06-07-3 Secrets to Lightning Fast Mobile Design at Instagram
10 0.85438985 928 high scalability-2010-10-26-Scaling DISQUS to 75 Million Comments and 17,000 RPS
11 0.85368162 269 high scalability-2008-03-08-Audiogalaxy.com Architecture
12 0.85355926 689 high scalability-2009-08-28-Strategy: Solve Only 80 Percent of the Problem
14 0.8518759 1179 high scalability-2012-01-23-Facebook Timeline: Brought to You by the Power of Denormalization
15 0.85174483 1121 high scalability-2011-09-21-5 Scalability Poisons and 3 Cloud Scalability Antidotes
16 0.8485204 1380 high scalability-2013-01-02-Why Pinterest Uses the Cloud Instead of Going Solo - To Be Or Not To Be
17 0.84804779 1068 high scalability-2011-06-27-TripAdvisor Architecture - 40M Visitors, 200M Dynamic Page Views, 30TB Data
18 0.84801555 1046 high scalability-2011-05-23-Evernote Architecture - 9 Million Users and 150 Million Requests a Day
19 0.84552318 1133 high scalability-2011-10-27-Strategy: Survive a Comet Strike in the East With Reserved Instances in the West
20 0.84533769 257 high scalability-2008-02-22-Kevin's Great Adventures in SSDland