high_scalability high_scalability-2009 high_scalability-2009-690 knowledge-graph by maker-knowledge-mining
Source: html
Introduction: I've recently started working with a large company who is looking to take one of their heavily utilized applications and move it to Amazon Web Services. I'm not looking to start a debate on the merits of EC2, the decision to move to aws is already made (and is a much better decision than paying a vendor millions to host it). I've done my reasearch and I'm comfortable with creating this environment with one exception, scaling MySQL. I havent done much work with MySQL, i'm more of an Oracle guy up to now. I'm struggling to determine a way to scale MySQL on the fly in a way so that replication works, the server takes its proper place in line for master candidacy, and the apache servers become aware of it. So this is really three questions: 1. What are some proven methods of load balancing the read traffic going from apache to MySQL. 2. How do I let the load balancing mechanism know when I scale up / down a new Mysql Server? 3. How to alert the master of the new server and initiate
sentIndex sentText sentNum sentScore
1 I've recently started working with a large company who is looking to take one of their heavily utilized applications and move it to Amazon Web Services. [sent-1, score-0.445]
2 I'm not looking to start a debate on the merits of EC2, the decision to move to aws is already made (and is a much better decision than paying a vendor millions to host it). [sent-2, score-1.072]
3 I've done my reasearch and I'm comfortable with creating this environment with one exception, scaling MySQL. [sent-3, score-0.457]
4 I havent done much work with MySQL, i'm more of an Oracle guy up to now. [sent-4, score-0.418]
5 I'm struggling to determine a way to scale MySQL on the fly in a way so that replication works, the server takes its proper place in line for master candidacy, and the apache servers become aware of it. [sent-5, score-1.219]
6 What are some proven methods of load balancing the read traffic going from apache to MySQL. [sent-7, score-0.672]
7 How do I let the load balancing mechanism know when I scale up / down a new Mysql Server? [sent-9, score-0.224]
8 How to alert the master of the new server and initiate replication in an automated environment? [sent-11, score-0.618]
9 Personally, I dont like the idea of scaling the databases, but the traffic increases exponentially for three hours a day, and then plummets to almost nothing. [sent-12, score-0.943]
10 So this would provide a significant cost savings. [sent-13, score-0.084]
11 The only way I've read to manage this sort of scaling I read here on slides 18-25: http://assets. [sent-14, score-0.535]
12 com/1/event/21/Tricks%20and%20Tradeoffs%20of%20Deploying%20MySQL%20Clusters%20in%20the%20Cloud%20Presentation Has anyone tried this method and either had success or have scripts available to do this? [sent-17, score-0.375]
13 I try not to remake the wheel when I dont have to. [sent-18, score-0.639]
wordName wordTfidf (topN-words)
[('dont', 0.296), ('havent', 0.202), ('plummets', 0.202), ('remake', 0.202), ('initiate', 0.181), ('merits', 0.181), ('decision', 0.176), ('struggling', 0.153), ('master', 0.146), ('wheel', 0.141), ('utilized', 0.139), ('apache', 0.138), ('personally', 0.136), ('exception', 0.136), ('balancing', 0.134), ('mysql', 0.132), ('debate', 0.13), ('environment', 0.124), ('exponentially', 0.122), ('advance', 0.118), ('comfortable', 0.118), ('replication', 0.118), ('three', 0.113), ('guy', 0.112), ('slides', 0.112), ('read', 0.111), ('fly', 0.111), ('scaling', 0.111), ('move', 0.108), ('vendor', 0.104), ('done', 0.104), ('proper', 0.103), ('heavily', 0.103), ('aware', 0.102), ('paying', 0.102), ('methods', 0.101), ('traffic', 0.099), ('scripts', 0.097), ('method', 0.096), ('looking', 0.095), ('thanks', 0.094), ('tried', 0.092), ('alert', 0.092), ('mechanism', 0.09), ('way', 0.09), ('success', 0.09), ('proven', 0.089), ('determine', 0.087), ('significant', 0.084), ('server', 0.081)]
simIndex simValue blogId blogTitle
same-blog 1 0.99999982 690 high scalability-2009-08-31-Scaling MySQL on Amazon Web Services
Introduction: I've recently started working with a large company who is looking to take one of their heavily utilized applications and move it to Amazon Web Services. I'm not looking to start a debate on the merits of EC2, the decision to move to aws is already made (and is a much better decision than paying a vendor millions to host it). I've done my reasearch and I'm comfortable with creating this environment with one exception, scaling MySQL. I havent done much work with MySQL, i'm more of an Oracle guy up to now. I'm struggling to determine a way to scale MySQL on the fly in a way so that replication works, the server takes its proper place in line for master candidacy, and the apache servers become aware of it. So this is really three questions: 1. What are some proven methods of load balancing the read traffic going from apache to MySQL. 2. How do I let the load balancing mechanism know when I scale up / down a new Mysql Server? 3. How to alert the master of the new server and initiate
Introduction: I first heard an enthusiastic endorsement of Squarespace streaming from the ubiquitous Leo Laporte on one of his many Twit Live shows. Squarespace as a fully hosted, completely managed environment for creating and maintaining a website, blog or portfolio was of interest to me because they promise scalability and this site doesn't have enough of that. But sadly, since they don't offer a link preserving Drupal import our relationship was not meant to be. When a fine reader of High Scalability, Brian Egge, (and all my readers are thrifty, brave, and strong) asked me how Squarespace scaled I said I didn't know, but I would try and find out. I emailed Squarespace a few questions and founder Anthony Casalena and Director of Technical Operations Rolando Berrios were kind enough to reply in some detail. The questions were both from Brian and myself. Answers can be found below. Two things struck me most about Squarespace's approach: They based their system on a memory grid, in this
3 0.12073151 196 high scalability-2007-12-30-MySQL clustering strategies and comparisions
Introduction: Compare: 1. MySQL Clustering(ndb-cluster stogare) 2. MySQL / GFS-GNBD/ HA 3. MySQL / DRBD /HA 4. MySQL Write Master / Multiple MySQL Read Slaves 5. Standalone MySQL Servers(Functionally seperated)
4 0.11232627 227 high scalability-2008-01-28-Howto setup GFS-GNBD
Introduction: Before you proceed make sure you have physical volume(something like /dev/sda1, /dev/sda4, etc) with no data. This is going to be the gfs volume which you will export to other nodes. It should be on the node which is going to be your gnbd server. If you dont have such volume create one using fdisk. I used mounted gfs volume as a DOCUMENT ROOT for my Apache server nodes(Load Balanced). I tried it on FC4 64-bit. If you plan to try it on any other distribution or 32-bit arch.. still the procedure remains same. Since I built it from source but not RPMs, you may have to simply supply config options with a different CFLAGS. Full details at http://linuxsutra.chakravaka.com/redhat-cluster/2006/11/01/howto-gfs-gnbd
Introduction: I remain neutral, but time and again, when people talk Windows or SQL Server, they seem to consider them unreliable with limits around scalability, performance and availability. And then you start looking at some of the big boys you have listed here in the architectural section and most of them are on Linux, MySQL,Oracle platforms that we dont see Windows and SQL Server in there.. What are your thoughts ?
6 0.11033218 1508 high scalability-2013-08-28-Sean Hull's 20 Biggest Bottlenecks that Reduce and Slow Down Scalability
7 0.10321251 63 high scalability-2007-08-09-Lots of questions for high scalability - high availability
8 0.098103769 1557 high scalability-2013-12-02-Evolution of Bazaarvoice’s Architecture to 500M Unique Users Per Month
9 0.095843263 427 high scalability-2008-10-22-Server load balancing architectures, Part 2: Application-level load balancing
10 0.094164051 1440 high scalability-2013-04-15-Scaling Pinterest - From 0 to 10s of Billions of Page Views a Month in Two Years
13 0.093559325 313 high scalability-2008-05-02-Friends for Sale Architecture - A 300 Million Page View-Month Facebook RoR App
14 0.092167489 303 high scalability-2008-04-18-Scaling Mania at MySQL Conference 2008
topicId topicWeight
[(0, 0.175), (1, 0.035), (2, -0.011), (3, -0.079), (4, 0.007), (5, -0.022), (6, -0.052), (7, -0.095), (8, 0.004), (9, -0.049), (10, -0.026), (11, -0.033), (12, 0.033), (13, -0.012), (14, 0.064), (15, -0.026), (16, 0.063), (17, 0.007), (18, -0.035), (19, 0.031), (20, 0.011), (21, 0.047), (22, -0.072), (23, -0.013), (24, 0.021), (25, 0.062), (26, 0.055), (27, -0.062), (28, 0.004), (29, -0.0), (30, -0.024), (31, 0.003), (32, -0.032), (33, 0.045), (34, 0.004), (35, -0.009), (36, 0.004), (37, -0.036), (38, 0.04), (39, -0.083), (40, 0.076), (41, 0.013), (42, 0.013), (43, -0.057), (44, 0.054), (45, -0.002), (46, -0.026), (47, -0.004), (48, 0.066), (49, -0.001)]
simIndex simValue blogId blogTitle
same-blog 1 0.98229527 690 high scalability-2009-08-31-Scaling MySQL on Amazon Web Services
Introduction: I've recently started working with a large company who is looking to take one of their heavily utilized applications and move it to Amazon Web Services. I'm not looking to start a debate on the merits of EC2, the decision to move to aws is already made (and is a much better decision than paying a vendor millions to host it). I've done my reasearch and I'm comfortable with creating this environment with one exception, scaling MySQL. I havent done much work with MySQL, i'm more of an Oracle guy up to now. I'm struggling to determine a way to scale MySQL on the fly in a way so that replication works, the server takes its proper place in line for master candidacy, and the apache servers become aware of it. So this is really three questions: 1. What are some proven methods of load balancing the read traffic going from apache to MySQL. 2. How do I let the load balancing mechanism know when I scale up / down a new Mysql Server? 3. How to alert the master of the new server and initiate
2 0.71400529 884 high scalability-2010-08-23-6 Ways to Kill Your Servers - Learning How to Scale the Hard Way
Introduction: This is a guest post by Steffen Konerow, author of the High Performance Blog . Learning how to scale isn’t easy without any prior experience. Nowadays you have plenty of websites like highscalability.com to get some inspiration, but unfortunately there is no solution that fits all websites and needs. You still have to think on your own to find a concept that works for your requirements. So did I. A few years ago, my bosses came to me and said “We’ve got a new project for you. It’s the relaunch of a website that has already 1 million users a month. You have to build the website and make sure we’ll be able to grow afterwards”. I was already an experienced coder, but not in these dimensions, so I had to start learning how to scale – the hard way. The software behind the website was a PHP content management system, based on Smarty and MySQL. The first task was finding a proper hosting company who had the experience and would also manage the servers for us. After some researc
3 0.67393553 157 high scalability-2007-11-16-Product: lbpool - Load Balancing JDBC Pool
Introduction: From the website: The lbpool project provides a load balancing JDBC driver for use with DB connection pools. It wraps a normal JDBC driver providing reconnect semantics in the event of additional hardware availability, partial system failure, or uneven load distribution. It also evenly distributes all new connections among slave DB servers in a given pool. Each time connect() is called it will attempt to use the best server with the least system load. The biggest scalability issue with large applications that are mostly READ bound is the number of transactions per second that the disks in your cluster can handle. You can generally solve this in two ways. 1. Buy bigger and faster disks with expensive RAID controllers. 2. Buy CHEAP hardware on CHEAP disks but lots of machines. We prefer the cheap hardware approach and lbpool allows you to do this. Even if you *did* manage to use cheap hardware most load balancing hardware is expensive, requires a redundant balancer (if it
4 0.66810685 63 high scalability-2007-08-09-Lots of questions for high scalability - high availability
Introduction: Hey, I do have a website that I would like to scale. Right now we have 10 servers but this does not scale well. I know how to deal with my apache web servers but have problems with sql servers. I would like to use the "scale out" system and add servers when we need. We have over 100Gb of data for mysql and we tried to have around 20G per server. It works well except that if a server goes down then 1/5 of the user can't access the website. We could use replication but we would need to at least double sql servers to replicate each server. And maybe in the future it's not gonna be enough we would need maybe 3 slaves per master ... well I don't really like this idea. I would prefer to have 8 servers that all deal with data from the 5 servers we have right now and then we could add new servers when we need. I looked at NFS but that does not seem to be a good idea for SQL servers ? Can you confirm?
5 0.66622424 302 high scalability-2008-04-10-Mysql scalability and failover...
Introduction: Hi, I am an owner of an large community website and currently we are having problems with our database architecture. We are using 2 database servers and spread tables across them to divide read/writes. We have about 90% reads and 10% writes. We use Memcached on all our webservers to cache as much as we can, traffic is load balanced between webservers. We have 2 extra servers ready to put to use! We have looked into a couple of solution so far: Continuent Uni/Cluster aka Sequoia -> Commercial version way too expensive and Java isn't as fast as it suppose to be. MySQL Proxy -> We couldn't find any good example on how to create a master - master with failover scenario. MySQL Clustering -> Seems to be not mature enough, had a lot of performance issues when we tried to go online with it. MySQL DRDB HA -> Only good for failover, cannot be scaled! MySQL Replication -> Well don't get me started ;) So now I turn to you guys to help me out, I am with my hands in my hair a
6 0.66602254 86 high scalability-2007-09-09-Clustering Solution
7 0.664024 595 high scalability-2009-05-08-Publish-subscribe model does not scale?
8 0.66394693 140 high scalability-2007-11-02-How WordPress.com Tracks 300 Servers Handling 10 Million Pageviews
9 0.6604898 620 high scalability-2009-06-05-SSL RPC API Scalability
10 0.65529144 300 high scalability-2008-04-07-Scalr - Open Source Auto-scaling Hosting on Amazon EC2
11 0.64904743 222 high scalability-2008-01-25-Application Database and DAL Architecture
12 0.63729137 73 high scalability-2007-08-23-Postgresql on high availability websites?
13 0.63590801 227 high scalability-2008-01-28-Howto setup GFS-GNBD
14 0.63143778 303 high scalability-2008-04-18-Scaling Mania at MySQL Conference 2008
15 0.62985444 675 high scalability-2009-08-08-1dbase vs. many and cloud hosting vs. dedicated server(s)?
16 0.62871289 1557 high scalability-2013-12-02-Evolution of Bazaarvoice’s Architecture to 500M Unique Users Per Month
17 0.62753528 16 high scalability-2007-07-16-Book: High Performance MySQL
18 0.62575901 176 high scalability-2007-12-07-Synchronizing databases in different geographic locations
19 0.62391543 1470 high scalability-2013-06-05-A Simple 6 Step Transition Guide for Moving Away from X to AWS
20 0.62006855 948 high scalability-2010-11-24-Great Introductory Video on Scalability from Harvard Computer Science
topicId topicWeight
[(1, 0.136), (2, 0.192), (10, 0.056), (39, 0.174), (40, 0.029), (61, 0.175), (79, 0.122), (94, 0.023)]
simIndex simValue blogId blogTitle
same-blog 1 0.92052072 690 high scalability-2009-08-31-Scaling MySQL on Amazon Web Services
Introduction: I've recently started working with a large company who is looking to take one of their heavily utilized applications and move it to Amazon Web Services. I'm not looking to start a debate on the merits of EC2, the decision to move to aws is already made (and is a much better decision than paying a vendor millions to host it). I've done my reasearch and I'm comfortable with creating this environment with one exception, scaling MySQL. I havent done much work with MySQL, i'm more of an Oracle guy up to now. I'm struggling to determine a way to scale MySQL on the fly in a way so that replication works, the server takes its proper place in line for master candidacy, and the apache servers become aware of it. So this is really three questions: 1. What are some proven methods of load balancing the read traffic going from apache to MySQL. 2. How do I let the load balancing mechanism know when I scale up / down a new Mysql Server? 3. How to alert the master of the new server and initiate
2 0.90696967 842 high scalability-2010-06-16-Hot Scalability Links for June 16, 2010
Introduction: You're Doing it Wrong by Poul-Henning Kamp. Don't look so guilty, he's not talking about you know what, he's talking about writing high-performance server programs: Not just wrong as in not perfect, but wrong as in wasting half, or more, of your performance. What good is an O(log2(n)) algorithm if those operations cause page faults and slow disk operations? For most relevant datasets an O(n) or even an O(n^2) algorithm, which avoids page faults, will run circles around it. A Microsoft Windows Azure primer: the basics by Peter Bright. Nice article explaining the basics of Azure and how it compares to Google and Amazon. A call to change the name from NoSQL to Postmodern Databases . Interesting idea, but the problem is the same one I have for Postmodern Art, when is it? I always feel like I'm in the post-post modern period, yet for art it's really in the early 1900s. Let's save future developers from this existential time crisis. Constructions from Dots and Lines by M
3 0.89699656 901 high scalability-2010-09-16-How Can the Large Hadron Collider Withstand One Petabyte of Data a Second?
Introduction: Why is there something rather than nothing? That's the kind of question the Large Hadron Collider in CERN is hopefully poised to answer. And what is the output of this beautiful 17-mile long, 6 billion dollar wabi-sabish proton smashing machine? Data. Great heaping torrents of Grand Canyon sized data. 15 million gigabytes every year. That's 1000 times the information printed in books every year. It's so much data 10,000 scientists will use a grid of 80,000+ computers , in 300 computer centers , in 50 different countries just to help make sense of it all. How will all this data be collected, transported, stored, and analyzed? It turns out, using what amounts to sort of Internet of Particles instead of an Internet of Things. Two good articles have recently shed some electro-magnetic energy in the human visible spectrum on the IT aspects of the collider: LHC computing grid pushes petabytes of data, beats expectations by John Timmer on Ars Technica and an overview of the Br
Introduction: This is a guest post by Doug Judd , original creator of Hypertable and the CEO of Hypertable, Inc. Hypertable delivers 2X better throughput in most tests -- HBase fails 41 and 167 billion record insert tests, overwhelmed by garbage collection -- Both systems deliver similar results for random read uniform test We recently conducted a test comparing the performance of Hypertable ( @hypertable ) version 0.9.5.5 to that of HBase ( @HBase ) version 0.90.4 (CDH3u2) running Zookeeper 3.3.4. In this post, we summarize the results and offer explanations for the discrepancies. For the full test report, see Hypertable vs. HBase II . Introduction Hypertable and HBase are both open source, scalable databases modeled after Google's proprietary Bigtable database. The primary difference between the two systems is that Hypertable is written in C++, while HBase is written in Java. We modeled this test after the one described in section 7 of the Bigtable paper and tuned both systems fo
5 0.87615615 1148 high scalability-2011-11-29-DataSift Architecture: Realtime Datamining at 120,000 Tweets Per Second
Introduction: I remember the excitement of when Twitter first opened up their firehose. As an early adopter of the Twitter API I could easily imagine some of the cool things you could do with all that data. I also remember the disappointment of learning that in the land of BigData, data has a price, and that price would be too high for little fish like me. It was like learning for the first time there would be no BigData Santa Clause. For a while though I had the pleasure of pondering just how I would handle all that data. It's a fascinating problem. You have to be able to reliably consume it, normalize it, merge it with other data, apply functions on it, store it, query it, distribute it, and oh yah, monetize it. Most of that in realish-time. And if you are trying to create a platform for allowing the entire Internet do to the same thing to the firehose, the challenge is exponentially harder. DataSift is in the exciting position of creating just such a firehose eating, data chomping machine. Y
7 0.86756372 1498 high scalability-2013-08-07-RAFT - In Search of an Understandable Consensus Algorithm
8 0.86416924 1089 high scalability-2011-07-29-Stuff The Internet Says On Scalability For July 29, 2011
9 0.86288476 1461 high scalability-2013-05-20-The Tumblr Architecture Yahoo Bought for a Cool Billion Dollars
10 0.86255163 1191 high scalability-2012-02-13-Tumblr Architecture - 15 Billion Page Views a Month and Harder to Scale than Twitter
12 0.86100054 383 high scalability-2008-09-10-Shard servers -- go big or small?
13 0.86051065 931 high scalability-2010-10-28-Notes from A NOSQL Evening in Palo Alto
14 0.86020565 1153 high scalability-2011-12-08-Update on Scalable Causal Consistency For Wide-Area Storage With COPS
15 0.85759151 1031 high scalability-2011-04-28-PaaS on OpenStack - Run Applications on Any Cloud, Any Time Using Any Thing
16 0.85624492 1002 high scalability-2011-03-09-Productivity vs. Control tradeoffs in PaaS
17 0.85524386 856 high scalability-2010-07-12-Creating Scalable Digital Libraries
18 0.85392165 1289 high scalability-2012-07-23-State of the CDN: More Traffic, Stable Prices, More Products, Profits - Not So Much
19 0.8532356 571 high scalability-2009-04-15-Using HTTP cache headers effectively
20 0.85267913 1142 high scalability-2011-11-14-Using Gossip Protocols for Failure Detection, Monitoring, Messaging and Other Good Things