high_scalability high_scalability-2008 high_scalability-2008-367 knowledge-graph by maker-knowledge-mining
Source: html
Introduction: Update 2 : Michael Galpin in Cache Money and Cache Discussions likes memcached for it's expiry policy, complex graph data, process data, but says MySQL has many advantages: SQL, Uniform Data Access, Write-through, Read-through, Replication, Management, Cold starts, LRU eviction. Update: Dormando asks Should you use memcached? Should you just shard mysql more? . The idea of caching is the most important part of caching as it transports you beyond a simple CRUD worldview. Plan for caching and sharding by properly abstracting data access methods. Brace for change. Be ready to shard, be ready to cache. React and change to what you push out which is actually popular, vs over planning and wasting valuable time. Feedster's François Schiettecatte wonders if Fotolog's 21 memcached servers wouldn't be better used to further shard data by adding more MySQL servers? He mentions Feedster was able to drop memcached once they partitioned their data across more servers. The algorithm:
sentIndex sentText sentNum sentScore
1 Update 2 : Michael Galpin in Cache Money and Cache Discussions likes memcached for it's expiry policy, complex graph data, process data, but says MySQL has many advantages: SQL, Uniform Data Access, Write-through, Read-through, Replication, Management, Cold starts, LRU eviction. [sent-1, score-0.35]
2 The idea of caching is the most important part of caching as it transports you beyond a simple CRUD worldview. [sent-5, score-0.61]
3 Plan for caching and sharding by properly abstracting data access methods. [sent-6, score-0.568]
4 React and change to what you push out which is actually popular, vs over planning and wasting valuable time. [sent-9, score-0.168]
5 Feedster's François Schiettecatte wonders if Fotolog's 21 memcached servers wouldn't be better used to further shard data by adding more MySQL servers? [sent-10, score-0.656]
6 He mentions Feedster was able to drop memcached once they partitioned their data across more servers. [sent-11, score-0.557]
7 The algorithm: partition until all data resides in memory and then you may not need an additional memcached layer. [sent-12, score-0.625]
8 Partition the data that feeds the cache tier: "Keep your front end working through the cache. [sent-17, score-0.281]
9 " Read replication with MySQL by François Schiettecatte. [sent-19, score-0.258]
10 Read replication is dead and it should be used only for backup purposes. [sent-20, score-0.416]
11 Take the memory used for caching and give it to your database servers. [sent-21, score-0.408]
12 Caching overdone because it adds complexity, latency on a cache miss, and inefficiently uses cluster resources. [sent-27, score-0.306]
wordName wordTfidf (topN-words)
[('fran', 0.325), ('ois', 0.325), ('replication', 0.258), ('shard', 0.252), ('caching', 0.239), ('memcached', 0.211), ('partitioningby', 0.147), ('garg', 0.147), ('abstracting', 0.139), ('dormando', 0.139), ('expiry', 0.139), ('inefficiently', 0.139), ('read', 0.136), ('ronald', 0.132), ('transports', 0.132), ('mysql', 0.129), ('partition', 0.124), ('mysqlby', 0.123), ('crud', 0.117), ('data', 0.11), ('lru', 0.105), ('wasting', 0.102), ('cache', 0.101), ('mentions', 0.1), ('uniform', 0.1), ('ready', 0.1), ('react', 0.098), ('brian', 0.097), ('resides', 0.094), ('miss', 0.087), ('memory', 0.086), ('death', 0.085), ('used', 0.083), ('cold', 0.081), ('hitting', 0.081), ('properly', 0.08), ('policy', 0.08), ('replaced', 0.077), ('broken', 0.075), ('dead', 0.075), ('michael', 0.074), ('greg', 0.074), ('feeds', 0.07), ('partitioned', 0.068), ('layers', 0.068), ('drop', 0.068), ('adds', 0.066), ('valuable', 0.066), ('tier', 0.065), ('advantages', 0.064)]
simIndex simValue blogId blogTitle
same-blog 1 1.0000001 367 high scalability-2008-08-17-Strategy: Drop Memcached, Add More MySQL Servers
Introduction: Update 2 : Michael Galpin in Cache Money and Cache Discussions likes memcached for it's expiry policy, complex graph data, process data, but says MySQL has many advantages: SQL, Uniform Data Access, Write-through, Read-through, Replication, Management, Cold starts, LRU eviction. Update: Dormando asks Should you use memcached? Should you just shard mysql more? . The idea of caching is the most important part of caching as it transports you beyond a simple CRUD worldview. Plan for caching and sharding by properly abstracting data access methods. Brace for change. Be ready to shard, be ready to cache. React and change to what you push out which is actually popular, vs over planning and wasting valuable time. Feedster's François Schiettecatte wonders if Fotolog's 21 memcached servers wouldn't be better used to further shard data by adding more MySQL servers? He mentions Feedster was able to drop memcached once they partitioned their data across more servers. The algorithm:
2 0.24934022 360 high scalability-2008-08-04-A Bunch of Great Strategies for Using Memcached and MySQL Better Together
Introduction: The primero recommendation for speeding up a website is almost always to add cache and more cache. And after that add a little more cache just in case. Memcached is almost always given as the recommended cache to use. What we don't often hear is how to effectively use a cache in our own products. MySQL hosted two excellent webinars (referenced below) on the subject of how to deploy and use memcached. The star of the show, other than MySQL of course, is Farhan Mashraqi of Fotolog. You may recall we did an earlier article on Fotolog in Secrets to Fotolog's Scaling Success , which was one of my personal favorites. Fotolog, as they themselves point out, is probably the largest site nobody has ever heard of, pulling in more page views than even Flickr. Fotolog has 51 instances of memcached on 21 servers with 175G in use and 254G available. As a large successful photo-blogging site they have very demanding performance and scaling requirements. To meet those requirements they've developed a
3 0.18632235 672 high scalability-2009-08-06-An Unorthodox Approach to Database Design : The Coming of the Shard
Introduction: Update 4: Why you don’t want to shard. by Morgon on the MySQL Performance Blog. Optimize everything else first, and then if performance still isn’t good enough, it’s time to take a very bitter medicine. Update 3: Building Scalable Databases: Pros and Cons of Various Database Sharding Schemes by Dare Obasanjo. Excellent discussion of why and when you would choose a sharding architecture, how to shard, and problems with sharding. Update 2: Mr. Moore gets to punt on sharding by Alan Rimm-Kaufman of 37signals. Insightful article on design tradeoffs and the evils of premature optimization. With more memory, more CPU, and new tech like SSD, problems can be avoided before more exotic architectures like sharding are needed. Add features not infrastructure. Jeremy Zawodny says he's wrong wrong wrong. we're running multi-core CPUs at slower clock speeds. Moore won't save you. Update: Dan Pritchett shares some excellent Sharding Lessons : Size Your Shards, Use Math on Shard C
4 0.17848618 920 high scalability-2010-10-15-Troubles with Sharding - What can we learn from the Foursquare Incident?
Introduction: For everything given something seems to be taken. Caching is a great scalability solution, but caching also comes with problems . Sharding is a great scalability solution, but as Foursquare recently revealed in a post-mortem about their 17 hours of downtime, sharding also has problems. MongoDB, the database Foursquare uses, also contributed their post-mortem of what went wrong too. Now that everyone has shared and resharded, what can we learn to help us skip these mistakes and quickly move on to a different set of mistakes? First, like for Facebook , huge props to Foursquare and MongoDB for being upfront and honest about their problems. This helps everyone get better and is a sign we work in a pretty cool industry. Second, overall, the fault didn't flow from evil hearts or gross negligence. As usual the cause was more mundane: a key system, that could be a little more robust, combined with a very popular application built by a small group of people, under immense pressure
5 0.17459054 152 high scalability-2007-11-13-Flickr Architecture
Introduction: Update: Flickr hits 2 Billion photos served. That's a lot of hamburgers. Flickr is both my favorite bird and the web's leading photo sharing site. Flickr has an amazing challenge, they must handle a vast sea of ever expanding new content, ever increasing legions of users, and a constant stream of new features, all while providing excellent performance. How do they do it? Site: http://www.flickr.com Information Sources Flickr and PHP (an early document) Capacity Planning for LAMP Federation at Flickr: Doing Billions of Queries a Day by Dathan Pattishall. Building Scalable Web Sites by Cal Henderson from Flickr. Database War Stories #3: Flickr by Tim O'Reilly Cal Henderson's Talks . A lot of useful PowerPoint presentations. Platform PHP MySQL Shards Memcached for a caching layer. Squid in reverse-proxy for html and images. Linux (RedHat) Smarty for templating Perl PEAR for XML and Email parsing ImageMagick, for ima
6 0.16532423 729 high scalability-2009-10-28-And the winner is: MySQL or Memcached or Tokyo Tyrant?
8 0.13339092 1514 high scalability-2013-09-09-Need Help with Database Scalability? Understand I-O
9 0.13195775 436 high scalability-2008-11-02-Strategy: How to Manage Sessions Using Memcached
10 0.12916952 538 high scalability-2009-03-16-Are Cloud Based Memory Architectures the Next Big Thing?
11 0.12782179 495 high scalability-2009-01-17-Intro to Caching,Caching algorithms and caching frameworks part 1
12 0.1267772 359 high scalability-2008-07-29-Ehcache - A Java Distributed Cache
13 0.12393268 1041 high scalability-2011-05-15-Building a Database remote availability site
14 0.11914101 673 high scalability-2009-08-07-Strategy: Break Up the Memcache Dog Pile
15 0.11601724 149 high scalability-2007-11-12-Scaling Using Cache Farms and Read Pooling
16 0.11577337 358 high scalability-2008-07-26-Sharding the Hibernate Way
17 0.11240417 927 high scalability-2010-10-26-Marrying memcached and NoSQL
18 0.11135742 373 high scalability-2008-08-29-Product: ScaleOut StateServer is Memcached on Steroids
20 0.10905423 248 high scalability-2008-02-13-What's your scalability plan?
topicId topicWeight
[(0, 0.161), (1, 0.071), (2, -0.085), (3, -0.102), (4, 0.027), (5, 0.168), (6, -0.033), (7, -0.079), (8, -0.026), (9, -0.032), (10, 0.003), (11, -0.021), (12, -0.01), (13, 0.13), (14, -0.075), (15, 0.009), (16, -0.059), (17, -0.022), (18, 0.015), (19, 0.053), (20, -0.057), (21, 0.078), (22, 0.015), (23, 0.059), (24, -0.084), (25, 0.11), (26, 0.017), (27, 0.037), (28, -0.023), (29, 0.041), (30, 0.005), (31, -0.022), (32, 0.025), (33, -0.029), (34, 0.012), (35, 0.006), (36, 0.012), (37, 0.025), (38, 0.014), (39, 0.033), (40, 0.036), (41, 0.074), (42, 0.008), (43, -0.054), (44, -0.022), (45, -0.034), (46, -0.029), (47, 0.004), (48, -0.037), (49, -0.044)]
simIndex simValue blogId blogTitle
same-blog 1 0.95342606 367 high scalability-2008-08-17-Strategy: Drop Memcached, Add More MySQL Servers
Introduction: Update 2 : Michael Galpin in Cache Money and Cache Discussions likes memcached for it's expiry policy, complex graph data, process data, but says MySQL has many advantages: SQL, Uniform Data Access, Write-through, Read-through, Replication, Management, Cold starts, LRU eviction. Update: Dormando asks Should you use memcached? Should you just shard mysql more? . The idea of caching is the most important part of caching as it transports you beyond a simple CRUD worldview. Plan for caching and sharding by properly abstracting data access methods. Brace for change. Be ready to shard, be ready to cache. React and change to what you push out which is actually popular, vs over planning and wasting valuable time. Feedster's François Schiettecatte wonders if Fotolog's 21 memcached servers wouldn't be better used to further shard data by adding more MySQL servers? He mentions Feedster was able to drop memcached once they partitioned their data across more servers. The algorithm:
2 0.82353204 360 high scalability-2008-08-04-A Bunch of Great Strategies for Using Memcached and MySQL Better Together
Introduction: The primero recommendation for speeding up a website is almost always to add cache and more cache. And after that add a little more cache just in case. Memcached is almost always given as the recommended cache to use. What we don't often hear is how to effectively use a cache in our own products. MySQL hosted two excellent webinars (referenced below) on the subject of how to deploy and use memcached. The star of the show, other than MySQL of course, is Farhan Mashraqi of Fotolog. You may recall we did an earlier article on Fotolog in Secrets to Fotolog's Scaling Success , which was one of my personal favorites. Fotolog, as they themselves point out, is probably the largest site nobody has ever heard of, pulling in more page views than even Flickr. Fotolog has 51 instances of memcached on 21 servers with 175G in use and 254G available. As a large successful photo-blogging site they have very demanding performance and scaling requirements. To meet those requirements they've developed a
3 0.80410743 729 high scalability-2009-10-28-And the winner is: MySQL or Memcached or Tokyo Tyrant?
Introduction: Matt, from the ever excellent MySQL Performance Blog , decided to run a test using a simple scenario drawn from his client experience in the gaming space. The scenario: read a row based on a primary key, update the row, write it to disk, and use the row to lookup another row. Matt ran three different tests explained in a series of three different articles: MySQL and MySQL + Memcached, Memcached Only , and Tokyo Tyrant . The lovingly compiled details along with many cool graphs are in the articles, but in general the lessons learned are: When memcached has enough memory (so records being accessed are in RAM), memcached + MySQL can provide a 10x performance boost over MySQL alone. When the RAM is too small and hot records, those being accessed a lot, aren't in RAM, or perhaps applications are doing a sequential scan of a table and the entire table isn't cached, then memcached won't be the big bang improvement you may be hoping for. When the database is removed and memcac
4 0.74341106 911 high scalability-2010-09-30-More Troubles with Caching
Introduction: As a tasty pairing with Facebook And Site Failures Caused By Complex, Weakly Interacting, Layered Systems , is another excellent tale of caching gone wrong by Peter Zaitsev, in an exciting twin billing: Cache Miss Storm and More on dangers of the caches . This is fascinating case where the cause turned out to be software upgrade that ran long because it had to be rolled back. During the long recovery time many of the cache entries timed out. When the database came back, slam, all the clients queried the database to repopulate the cache and bad things happened to the database. The solution was equally interesting: So the immediate solution to bring the system up was surprisingly simple. We just had to get traffic on the system in stages allowing Memcache to be warmed up. There were no code which would allow to do it on application side so we did it on MySQL side instead. “SET GLOBAL max_connections=20” to limit number of connections to MySQL and so let application to err when i
5 0.73398769 174 high scalability-2007-12-05-Product: Tugela Cache
Introduction: Tugela Cache is a cache system like memecached, but instead of storing data just in RAM, it stores data in the file system using a b-tree. You trade latency in order to have a very large cache. It's useful for sites that have caching requirements that exceed their available memory. It uses the same wire protocol as memcached so it can be dropped in without a hassle. From the website: As large MediaWiki deployments may gain performance using Memcached, at some level cost of RAM to store all objects becomes too high. In order to balance resource usage and make more use of our Apache server disks, Tugela, the distributed cached on-disk hash database, has arrived. Tugela Cache is derived from Memcached. Much of the code remains the same, but notably, these changes: Internal slab allocator replaced by BerkeleyDB B-Tree database. Expiry policy management moved to external program tugela-expire Much statistics code made obsolete. An interesting point brought up in the comme
6 0.72694355 436 high scalability-2008-11-02-Strategy: How to Manage Sessions Using Memcached
7 0.71975505 1633 high scalability-2014-04-16-Six Lessons Learned the Hard Way About Scaling a Million User System
8 0.71839273 927 high scalability-2010-10-26-Marrying memcached and NoSQL
9 0.71407557 673 high scalability-2009-08-07-Strategy: Break Up the Memcache Dog Pile
10 0.71324199 359 high scalability-2008-07-29-Ehcache - A Java Distributed Cache
11 0.69678396 602 high scalability-2009-05-17-Scaling Django Web Apps by Mike Malone
12 0.68342066 149 high scalability-2007-11-12-Scaling Using Cache Farms and Read Pooling
13 0.67918527 467 high scalability-2008-12-16-[ANN] New Open Source Cache System
14 0.67016113 248 high scalability-2008-02-13-What's your scalability plan?
15 0.66610402 847 high scalability-2010-06-23-Product: dbShards - Share Nothing. Shard Everything.
16 0.66227591 1346 high scalability-2012-10-24-Saving Cash Using Less Cache - 90% Savings in the Caching Tier
17 0.65951324 152 high scalability-2007-11-13-Flickr Architecture
18 0.64358974 672 high scalability-2009-08-06-An Unorthodox Approach to Database Design : The Coming of the Shard
19 0.64326853 373 high scalability-2008-08-29-Product: ScaleOut StateServer is Memcached on Steroids
20 0.63870925 383 high scalability-2008-09-10-Shard servers -- go big or small?
topicId topicWeight
[(1, 0.16), (2, 0.276), (3, 0.012), (10, 0.045), (30, 0.037), (51, 0.01), (53, 0.25), (61, 0.028), (79, 0.052), (94, 0.032)]
simIndex simValue blogId blogTitle
same-blog 1 0.88467896 367 high scalability-2008-08-17-Strategy: Drop Memcached, Add More MySQL Servers
Introduction: Update 2 : Michael Galpin in Cache Money and Cache Discussions likes memcached for it's expiry policy, complex graph data, process data, but says MySQL has many advantages: SQL, Uniform Data Access, Write-through, Read-through, Replication, Management, Cold starts, LRU eviction. Update: Dormando asks Should you use memcached? Should you just shard mysql more? . The idea of caching is the most important part of caching as it transports you beyond a simple CRUD worldview. Plan for caching and sharding by properly abstracting data access methods. Brace for change. Be ready to shard, be ready to cache. React and change to what you push out which is actually popular, vs over planning and wasting valuable time. Feedster's François Schiettecatte wonders if Fotolog's 21 memcached servers wouldn't be better used to further shard data by adding more MySQL servers? He mentions Feedster was able to drop memcached once they partitioned their data across more servers. The algorithm:
Introduction: Instagram is a free photo sharing and social networking service for your iPhone that has been an instant success . Growing to 14 million users in just over a year, they reached 150 million photos in August while amassing several terabytes of photos, and they did this with just 3 Instaneers, all on the Amazon stack. The Instagram team has written up what can be considered the canonical description of an early stage startup in this era: What Powers Instagram: Hundreds of Instances, Dozens of Technologies . Instagram uses a pastiche of different technologies and strategies. The team is small yet has experience rapid growth riding the crest of a rising social and mobile wave, it uses a hybrid of SQL and NoSQL, it uses a ton of open source projects, they chose the cloud over colo, Amazon services are highly leveraged rather than building their own, reliability is through availability zones, async work scheduling links components together, the system is composed as much as possible
3 0.88071883 522 high scalability-2009-02-25-Learn how to manage change and complexity by Zachman Live.
Introduction: John Zachman (Father of enterprise architecture) Given this renascent interest, who better to explain the principles behind Enterprise Architecture than the man himself, John Zachman, the originator of the " Zachman Framework for Enterprise Architecture" Join this workshop in Johannesburg 25th Mar 09 and Cape town in 27th March 09 and Mr.Zachman will explain how and why Enterprise Architecture provides measure, such an implementation is a daunting task with opportunities to fail lurking in many places. For more details visit http://www.ITArchitectureSummit.com For registrations, group discounts or further details please contact Caroline.smith@icmgworld.com
4 0.87000424 1335 high scalability-2012-10-08-How UltraDNS Handles Hundreds of Thousands of Zones and Tens of Millions of Records
Introduction: This is a guest post by Jeffrey Damick , Principal Software Engineer for Neustar . Jeffrey has overseen the software architecture for UltraDNS for last two and half years as it went through substantial revitalization. UltraDNS is one the top the DNS providers, serving many top-level domains (TLDs) as well as second-level domains (SLDs) . This requires handling of several hundreds of thousands of zones with many containing millions of records each. Even with all of its success UltraDNS had fallen into a rut several years ago, its release schedule had become haphazard at best and the team was struggling to keep up with feature requests in a waterfall development style. Development Realizing that something had to be done the team came together and identified the most important areas to attack first. We began with the code base, stabilizing our flagship proprietary C++ DNS server, instituting common best practices and automation. Testing was very manual and not easily
5 0.8640371 1224 high scalability-2012-04-09-The Instagram Architecture Facebook Bought for a Cool Billion Dollars
Introduction: It's been a well kept secret, but you may have heard Facebook will Buy Photo-Sharing Service Instagram for $1 Billion . Just what is Facebook buying? Here's a quick gloss I did a little over a year ago on a presentation Instagram gave on their architecture. In that article I called Instagram's architecture the " canonical description of an early stage startup in this era." Little did we know how true that would turn out to be. If you want to learn how they did it then don't take a picture, just keep on reading... Instagram is a free photo sharing and social networking service for your iPhone that has been an instant success . Growing to 14 million users in just over a year (now 30 million users), they reached 150 million photos in August while amassing several terabytes of photos, and they did this with just 3 Instaneers, all on the Amazon stack. The Instagram team has written up what can be considered the canonical description of an early stage startup in this era: Wh
6 0.85386461 1522 high scalability-2013-09-25-Great Open Source Solution for Boring HA and Scalability Problems
7 0.82955682 345 high scalability-2008-06-11-Pyshards aspires to build sharding toolkit for Python
8 0.81587553 318 high scalability-2008-05-14-New Facebook Chat Feature Scales to 70 Million Users Using Erlang
9 0.80081958 674 high scalability-2009-08-07-The Canonical Cloud Architecture
10 0.79191017 77 high scalability-2007-08-30-Log Everything All the Time
11 0.78689146 5 high scalability-2007-07-10-mixi.jp Architecture
12 0.78402531 1346 high scalability-2012-10-24-Saving Cash Using Less Cache - 90% Savings in the Caching Tier
13 0.78368157 1646 high scalability-2014-05-12-4 Architecture Issues When Scaling Web Applications: Bottlenecks, Database, CPU, IO
14 0.78350973 360 high scalability-2008-08-04-A Bunch of Great Strategies for Using Memcached and MySQL Better Together
15 0.78345096 242 high scalability-2008-02-07-Looking for good business examples of compaines using Hadoop
16 0.78306413 1628 high scalability-2014-04-08-Microservices - Not a free lunch!
17 0.78304434 1124 high scalability-2011-09-26-17 Techniques Used to Scale Turntable.fm and Labmeeting to Millions of Users
18 0.78257132 1251 high scalability-2012-05-24-Build your own twitter like real time analytics - a step by step guide
19 0.78158069 1237 high scalability-2012-05-02-12 Ways to Increase Throughput by 32X and Reduce Latency by 20X
20 0.78146213 985 high scalability-2011-02-08-Mollom Architecture - Killing Over 373 Million Spams at 100 Requests Per Second