high_scalability high_scalability-2012 high_scalability-2012-1168 knowledge-graph by maker-knowledge-mining
Source: html
Introduction: How does Facebook handle the massive New Year's Eve traffic spike? Thanks to Mike Swift, in Facebook gets ready for New Year's Eve , we get a little insight as to their method for the madness, nothing really detailed, but still interesting. Problem Setup Facebook expects tha one billion+ photos will be shared on New Year's eve. Facebook's 800 million users are scattered around the world. Three quarters live outside the US. Each user is linked to an average of 130 friends. Photos and posts must appear in less than a second. Opening a homepage requires executing requests on a 100 different servers, and those requests have to be ranked, sorted, and privacy-checked, and then rendered. Different events put different stresses on different parts of Facebook. Photo and Video Uploads - Holidays require hundreds of terabytes of capacity News Feed - News events like big sports events and the death of Steve Jobs drive user status updates Coping Strategies Try
sentIndex sentText sentNum sentScore
1 Problem Setup Facebook expects tha one billion+ photos will be shared on New Year's eve. [sent-3, score-0.423]
2 Facebook's 800 million users are scattered around the world. [sent-4, score-0.113]
3 Each user is linked to an average of 130 friends. [sent-6, score-0.087]
4 Opening a homepage requires executing requests on a 100 different servers, and those requests have to be ranked, sorted, and privacy-checked, and then rendered. [sent-8, score-0.223]
5 Different events put different stresses on different parts of Facebook. [sent-9, score-0.362]
6 Photo and Video Uploads - Holidays require hundreds of terabytes of capacity News Feed - News events like big sports events and the death of Steve Jobs drive user status updates Coping Strategies Try to predict the surge in traffic. [sent-10, score-0.881]
7 Prepare to bring additional capacity online from data centers. [sent-13, score-0.26]
8 Implication is that your architecture can handle additional capacity and make meaningful use of it. [sent-14, score-0.349]
9 Overbuild a matter of culture so big events aren't big challenges. [sent-15, score-0.243]
10 These are knobs that can be turned to survive system failures or unanticipated surges in traffic. [sent-17, score-0.452]
11 For example, for a capacity problem they can serve smaller photos to reduce bandwidth usage. [sent-18, score-0.155]
12 The idea is to not go off-line completely when there's a problem, but to degrade gracefully. [sent-19, score-0.113]
13 Top Five Global Events for Facebook status updates in 2011 Death of Osama Bin Laden, May 2 Packers win the Super Bowl, Feb. [sent-20, score-0.13]
wordName wordTfidf (topN-words)
[('eve', 0.326), ('events', 0.243), ('steve', 0.157), ('photos', 0.155), ('capacity', 0.149), ('tha', 0.146), ('year', 0.145), ('bin', 0.137), ('articlesfacebook', 0.137), ('surges', 0.137), ('homepage', 0.137), ('swift', 0.137), ('madness', 0.131), ('verdict', 0.131), ('status', 0.13), ('holidays', 0.126), ('sankar', 0.126), ('quarters', 0.126), ('expects', 0.122), ('krishna', 0.122), ('stresses', 0.119), ('harvest', 0.119), ('unanticipated', 0.119), ('facebook', 0.118), ('anthony', 0.116), ('farmville', 0.116), ('scattered', 0.113), ('degrade', 0.113), ('additional', 0.111), ('knobs', 0.111), ('bowl', 0.111), ('jobs', 0.109), ('ranked', 0.104), ('news', 0.1), ('emergency', 0.099), ('july', 0.097), ('implication', 0.093), ('uploads', 0.092), ('sports', 0.092), ('opening', 0.09), ('surge', 0.089), ('meaningful', 0.089), ('linked', 0.087), ('photo', 0.086), ('sorted', 0.086), ('executing', 0.086), ('survive', 0.085), ('death', 0.084), ('mike', 0.084), ('spike', 0.082)]
simIndex simValue blogId blogTitle
same-blog 1 1.0000001 1168 high scalability-2012-01-04-How Facebook Handled the New Year's Eve Onslaught
Introduction: How does Facebook handle the massive New Year's Eve traffic spike? Thanks to Mike Swift, in Facebook gets ready for New Year's Eve , we get a little insight as to their method for the madness, nothing really detailed, but still interesting. Problem Setup Facebook expects tha one billion+ photos will be shared on New Year's eve. Facebook's 800 million users are scattered around the world. Three quarters live outside the US. Each user is linked to an average of 130 friends. Photos and posts must appear in less than a second. Opening a homepage requires executing requests on a 100 different servers, and those requests have to be ranked, sorted, and privacy-checked, and then rendered. Different events put different stresses on different parts of Facebook. Photo and Video Uploads - Holidays require hundreds of terabytes of capacity News Feed - News events like big sports events and the death of Steve Jobs drive user status updates Coping Strategies Try
2 0.24180888 424 high scalability-2008-10-22-EVE Online Architecture
Introduction: Sorry, the content for this post apparently did not make the transition from the old HighScalability website, it's all messed up, but there's still a some useful content... EVE Online is "The World's Largest Game Universe", a massively multiplayer online game ( MMO ) made by CCP. EVE Online's Architecture is unusual for a MMOG because it doesn't divide the player load among different servers or shards. Instead, the same cluster handles the entire EVE universe. It is an interesting to compare this with the Architecture of the Second Life Grid . How do they manage to scale? Information Sources EVE Insider Dev Blog EVE Online FAQ Massively - Eve Evolved: Eve Online's Server Model and its discussion on Slashdot EVE Online Forums Massively Multiplayer Game Development 2 Platform Stackless Python used for both server and client game logic. It allows programmers to reap the benefits of thread-based programming without the performance and complexity p
3 0.19574779 1405 high scalability-2013-02-13-7 Sensible and 1 Really Surprising Way EVE Online Scales to Play Huge Games
Introduction: "Everything in war is simple, but the simplest thing is difficult." -- Carl von Clausewitz Games are proving grounds for software architecture. They combine scale, high performance, challenging problems, a rabid user base, cost sensitivity, and the need for profit. And when games have in-game currency, like EVE Online has, there's money at play, so you can't just get away with a c'est la vie attitude. Engineering must be applied. In Planning for war: how the EVE Online servers deal with a 3,000 person battle , we learn some techniques EVE Online uses to handle large games: 7 Sensible... Do nothing. Most games are manageable or have spikes that quickly dissipate. Run it Hot . There's nothing to throttle as servers run at 100%. Why waste money? Use all your CPU. Shard it . Games are sharded by solar system and multiple solar systems run on a node. Move it . Games are moved when a machine becomes overloaded. Live Node Remap, where a live game is moved to
4 0.16378513 966 high scalability-2010-12-31-Facebook in 20 Minutes: 2.7M Photos, 10.2M Comments, 4.6M Messages
Introduction: To celebrate the new year Facebook has shared the results of a little end of the year introspection. It has been a fecund year for Facebook: 43,869,800 changed their status to single 3,025,791 changed their status to "it's complicated" 28,460,516 changed their status to in a relationship 5,974,574 changed their status to engaged 36,774,801 changes their status to married If these numbers are simply to large to grasp, it doesn't get any better when you look at happens in a mere 20 minutes: Shared links: 1,000,000 Tagged photos: 1,323,000 Event invites sent out: 1,484,000 Wall Posts: 1,587,000 Status updates: 1,851,000 Friend requests accepted: 1,972,000 Photos uploaded: 2,716,000 Comments: 10,208,000 Message: 4,632,000 If you want to see how Facebook supports these huge numbers take a look at a few posts . One wonders what the new year will bring? Related Articles What the World Eats from Time Magazine A Day in the Life of an An
5 0.1476744 840 high scalability-2010-06-10-The Four Meta Secrets of Scaling at Facebook
Introduction: Aditya Agarwal, Director of Engineering at Facebook, gave an excellent Scale at Facebook talk that covers their architecture, but the talk is really more about how to scale an organization by preserving the best parts of its culture. The key take home of the talk is: You can get the code right, you can get the products right, but you need to get the culture right first. If you don't get the culture right then your company won't scale. This leads into the four meta secrets of scaling at Facebook: Scaling takes Iteration Don't Over Design Choose the right tool for the job, but realize that your choice comes with overhead. Get the culture right. Move Fast - break things. Huge Impact - small teams. Be bold - innovate. Some Background Facebook is big : 400 million active users; users spend an average of 20 minutes a day; 5 billion pieces of content (status updates, comments, likes, photo uploads, video uploads, chat messages, inbox messages, group events, f
6 0.13897303 774 high scalability-2010-02-08-How FarmVille Scales to Harvest 75 Million Players a Month
7 0.11784588 760 high scalability-2010-01-13-10 Hot Scalability Links for January 13, 2010
8 0.11347423 1197 high scalability-2012-02-21-Pixable Architecture - Crawling, Analyzing, and Ranking 20 Million Photos a Day
9 0.11230461 152 high scalability-2007-11-13-Flickr Architecture
10 0.10566946 1123 high scalability-2011-09-23-The Real News is Not that Facebook Serves Up 1 Trillion Pages a Month…
11 0.10394613 106 high scalability-2007-10-02-Secrets to Fotolog's Scaling Success
13 0.098951206 1224 high scalability-2012-04-09-The Instagram Architecture Facebook Bought for a Cool Billion Dollars
14 0.097350046 513 high scalability-2009-02-16-Handle 1 Billion Events Per Day Using a Memory Grid
15 0.096336722 1008 high scalability-2011-03-22-Facebook's New Realtime Analytics System: HBase to Process 20 Billion Events Per Day
16 0.092720062 720 high scalability-2009-10-12-High Performance at Massive Scale – Lessons learned at Facebook
18 0.088883564 1622 high scalability-2014-03-31-How WhatsApp Grew to Nearly 500 Million Users, 11,000 cores, and 70 Million Messages a Second
19 0.085682981 129 high scalability-2007-10-23-Hire Facebook, Ning, and Salesforce to Scale for You
20 0.084474899 1359 high scalability-2012-11-15-Gone Fishin': Justin.Tv's Live Video Broadcasting Architecture
topicId topicWeight
[(0, 0.136), (1, 0.075), (2, 0.016), (3, -0.039), (4, 0.01), (5, -0.076), (6, -0.039), (7, 0.071), (8, 0.034), (9, -0.008), (10, 0.053), (11, 0.036), (12, 0.05), (13, 0.013), (14, -0.044), (15, 0.095), (16, 0.034), (17, -0.005), (18, -0.007), (19, 0.058), (20, 0.048), (21, 0.037), (22, 0.042), (23, -0.011), (24, -0.003), (25, -0.033), (26, 0.006), (27, 0.028), (28, 0.03), (29, -0.016), (30, 0.015), (31, -0.002), (32, -0.044), (33, -0.035), (34, 0.043), (35, -0.002), (36, 0.073), (37, -0.123), (38, 0.019), (39, -0.038), (40, 0.013), (41, -0.03), (42, -0.014), (43, 0.032), (44, -0.01), (45, 0.001), (46, 0.013), (47, -0.037), (48, 0.001), (49, -0.027)]
simIndex simValue blogId blogTitle
same-blog 1 0.95467281 1168 high scalability-2012-01-04-How Facebook Handled the New Year's Eve Onslaught
Introduction: How does Facebook handle the massive New Year's Eve traffic spike? Thanks to Mike Swift, in Facebook gets ready for New Year's Eve , we get a little insight as to their method for the madness, nothing really detailed, but still interesting. Problem Setup Facebook expects tha one billion+ photos will be shared on New Year's eve. Facebook's 800 million users are scattered around the world. Three quarters live outside the US. Each user is linked to an average of 130 friends. Photos and posts must appear in less than a second. Opening a homepage requires executing requests on a 100 different servers, and those requests have to be ranked, sorted, and privacy-checked, and then rendered. Different events put different stresses on different parts of Facebook. Photo and Video Uploads - Holidays require hundreds of terabytes of capacity News Feed - News events like big sports events and the death of Steve Jobs drive user status updates Coping Strategies Try
2 0.77270979 966 high scalability-2010-12-31-Facebook in 20 Minutes: 2.7M Photos, 10.2M Comments, 4.6M Messages
Introduction: To celebrate the new year Facebook has shared the results of a little end of the year introspection. It has been a fecund year for Facebook: 43,869,800 changed their status to single 3,025,791 changed their status to "it's complicated" 28,460,516 changed their status to in a relationship 5,974,574 changed their status to engaged 36,774,801 changes their status to married If these numbers are simply to large to grasp, it doesn't get any better when you look at happens in a mere 20 minutes: Shared links: 1,000,000 Tagged photos: 1,323,000 Event invites sent out: 1,484,000 Wall Posts: 1,587,000 Status updates: 1,851,000 Friend requests accepted: 1,972,000 Photos uploaded: 2,716,000 Comments: 10,208,000 Message: 4,632,000 If you want to see how Facebook supports these huge numbers take a look at a few posts . One wonders what the new year will bring? Related Articles What the World Eats from Time Magazine A Day in the Life of an An
3 0.76020074 792 high scalability-2010-03-10-How FarmVille Scales - The Follow-up
Introduction: Several readers had follow-up questions in response to How FarmVille Scales to Harvest 75 Million Players a Month . Here are Luke's response to those questions (and a few of mine). How does social networking makes things easier or harder? The primary interesting aspect of social networking games is how you wind up with a graph of connected users who need to be access each other's data on a frequent basis. This makes the overall dataset difficult if not impossible to partition. What are examples of the Facebook calls you try to avoid and how they impact game play? We can make a call for facebook friend data to retrieve information about your friends playing the game. Normally, we show a friend ladder at the bottom of the game that shows friend information, including name and facebook photo. Can you say where your cache is, what form it takes, and how much cached there is? Do you have a peering relationship with Facebook, as one might expect at that bandwidth?
4 0.7416029 870 high scalability-2010-08-02-7 Scaling Strategies Facebook Used to Grow to 500 Million Users
Introduction: Robert Johnson, a director of engineering at Facebook, celebrated Facebook's monumental achievement of reaching 500 million users by sharing the scaling principles that helped reach that milestone. In case you weren't suitably impressed by the 500 million user number, Robert ratchets up the numbers game with these impressive figures: 1 million users per engineer 500 million active users 100 billion hits per day 50 billion photos 2 trillion objects cached, with hundreds of millions of requests per second 130TB of logs every day How did Facebook get to this point? People Matter Most . It's people who build and run systems. The best tools for scaling are an engineering and operations teams that can handle anything. Scale Horizontally . Handling exponentially growing traffic requires spreading load arbitrarily across many machines. Using different databases for tables like accounts and profiles only doubles capacity. This approach hurts efficiency, but
5 0.70447832 1123 high scalability-2011-09-23-The Real News is Not that Facebook Serves Up 1 Trillion Pages a Month…
Introduction: There’s some amount of debate whether Facebook really crossed over the one trillion page view per month threshold. While one report says it did, another respected firm says it did not ; that its monthly page views are a mere 467 billion per month. In the big scheme of things, the discrepancy is somewhat irrelevant, as neither show the true load on Facebook’s infrastructure – which is far more impressive a set of numbers than its externally measured “page view” metric. Mashable reported in “ Facebook Surpasses 1 Trillion Pageviews per Month ” that the social networking giant saw “approximately 870 million unique visitors in June and 860 million in July” and followed up with some per visitor statistics, indicating “each visitor averaged approximately 1,160 page views in July and 40 per visit — enormous by any standard. Time spent on the site was around 25 minutes per user.” From an architectural standpoint it’s not just about the page views. It’s about requests and responses,
6 0.70141912 774 high scalability-2010-02-08-How FarmVille Scales to Harvest 75 Million Players a Month
7 0.69775653 840 high scalability-2010-06-10-The Four Meta Secrets of Scaling at Facebook
8 0.69100058 1602 high scalability-2014-02-26-The WhatsApp Architecture Facebook Bought For $19 Billion
10 0.66301364 720 high scalability-2009-10-12-High Performance at Massive Scale – Lessons learned at Facebook
11 0.65571684 1405 high scalability-2013-02-13-7 Sensible and 1 Really Surprising Way EVE Online Scales to Play Huge Games
12 0.61811304 424 high scalability-2008-10-22-EVE Online Architecture
13 0.61397797 904 high scalability-2010-09-21-Playfish's Social Gaming Architecture - 50 Million Monthly Users and Growing
14 0.60489213 1412 high scalability-2013-02-25-SongPop Scales to 1 Million Active Users on GAE, Showing PaaS is not Passé
15 0.59803069 758 high scalability-2010-01-11-Have We Reached the End of Scaling?
16 0.59715176 1595 high scalability-2014-02-13-Snabb Switch - Skip the OS and Get 40 million Requests Per Second in Lua
17 0.59547365 464 high scalability-2008-12-13-Strategy: Facebook Tweaks to Handle 6 Time as Many Memcached Requests
18 0.59481478 129 high scalability-2007-10-23-Hire Facebook, Ning, and Salesforce to Scale for You
19 0.59195995 1228 high scalability-2012-04-16-Instagram Architecture Update: What’s new with Instagram?
20 0.58015347 1444 high scalability-2013-04-23-Facebook Secrets of Web Performance
topicId topicWeight
[(1, 0.142), (2, 0.161), (40, 0.028), (51, 0.345), (61, 0.09), (79, 0.072), (85, 0.044), (94, 0.034)]
simIndex simValue blogId blogTitle
1 0.91037899 818 high scalability-2010-04-30-Behind the scenes of an online marketplace
Introduction: In a presentation originally held at the 4. O2 Hosting Event in Hamburg, I spoke about the technology at a large online marketplace in Germany called Hitmeister . Â Some of the topics discussed include: what makes up a marketplace? technically system principles development patterns tools philosophy data model hardware I am looking forward to comments and suggestions for both the presentation and our work.
same-blog 2 0.88257927 1168 high scalability-2012-01-04-How Facebook Handled the New Year's Eve Onslaught
Introduction: How does Facebook handle the massive New Year's Eve traffic spike? Thanks to Mike Swift, in Facebook gets ready for New Year's Eve , we get a little insight as to their method for the madness, nothing really detailed, but still interesting. Problem Setup Facebook expects tha one billion+ photos will be shared on New Year's eve. Facebook's 800 million users are scattered around the world. Three quarters live outside the US. Each user is linked to an average of 130 friends. Photos and posts must appear in less than a second. Opening a homepage requires executing requests on a 100 different servers, and those requests have to be ranked, sorted, and privacy-checked, and then rendered. Different events put different stresses on different parts of Facebook. Photo and Video Uploads - Holidays require hundreds of terabytes of capacity News Feed - News events like big sports events and the death of Steve Jobs drive user status updates Coping Strategies Try
3 0.78321666 98 high scalability-2007-09-18-Sync data on all servers
Introduction: I have a few apache servers ( arround 11 atm ) serving a small amount of data ( arround 44 gigs right now ). For some time I have been using rsync to keep all the content equal on all servers, but the amount of data has been growing, and rsync takes a few too much time to "compare" all data from source to destination, and create a lot of I/O. I have been taking a look at MogileFS, it seems a good and reliable option, but as the fuse module is not finished, we should have to rewrite all our apps, and its not an option atm. Any ideas? I just want a "real time, non resource-hungry" solution alternative for rsync. If I get more features on the way, then they are welcome :) Why I prefer to use a Distributed File System instead of using NAS + NFS? - I need 2 NAS, if I dont want a point of failure, and NAS hard is expensive. - Non-shared hardware, all server has their own local disks. - As files are replicated, I can save a lot of money, RAID is not a MUST. Thn
Introduction: Sponsored Jobs Get Your High Scalability Fix at Digg Huffington Post: Linux System Administrator Sponsored Events Velocity Web Performance and Operations Conference - Fast by Default Social Developer Summit (HighScalability.com readers save 15% with discount code SDSHS) Get Your High Scalability Fix at Digg Interested in working on cutting-edge high-scale infrastructure at Digg? We're making a big investment in scaling and have committed to the NoSQL (Not only SQL) path with Cassandra . We're using other open-source infrastructure to help us scale including Hadoop, RabbitMQ, Zookeeper, Thrift, HDFS and Lucene. We're rewriting Digg from the ground up and we need amazing developers to join our world-class team. If you think you are up for the challenge, or you know someone who might be, take a look at our jobs page for more information. Huffington Post: Linux System Administrator HuffingtonPost has an immediate need for a Lin
Introduction: We are not yet at the End of History for database theory as Peter Bailis and the Database Group at UC Berkeley continue to prove with a great companion blog post to their new paper: Scalable Atomic Visibility with RAMP Transactions . I like the approach of pairing a blog post with a paper. A paper almost by definition is formal, but a blog post can help put a paper in context and give it some heart. From the abstract: Databases can provide scalability by partitioning data across several servers. However, multi-partition, multi-operation transactional access is often expensive, employing coordination-intensive locking, validation, or scheduling mechanisms. Accordingly, many real-world systems avoid mechanisms that provide useful semantics for multi-partition operations . This leads to incorrect behavior for a large class of applications including secondary indexing, foreign key enforcement, and materialized view maintenance . In this work, we identify a new isolation mode
6 0.73961473 481 high scalability-2009-01-02-Strategy: Understanding Your Data Leads to the Best Scalability Solutions
7 0.73218089 741 high scalability-2009-11-16-Building Scalable Systems Using Data as a Composite Material
8 0.72979164 1134 high scalability-2011-10-28-Stuff The Internet Says On Scalability For October 28, 2011
10 0.70952153 1644 high scalability-2014-05-07-Update on Disqus: It's Still About Realtime, But Go Demolishes Python
11 0.70844907 298 high scalability-2008-04-07-Lazy web sites run faster
12 0.70334393 846 high scalability-2010-06-22-Sponsored Post: Jobs: Etsy, Digg, Huffington Post Event: Velocity Conference
13 0.70246458 510 high scalability-2009-02-09-Paper: Consensus Protocols: Two-Phase Commit
14 0.69818604 953 high scalability-2010-12-03-GPU vs CPU Smackdown : The Rise of Throughput-Oriented Architectures
15 0.68874359 138 high scalability-2007-10-30-Feedblendr Architecture - Using EC2 to Scale
16 0.66191417 626 high scalability-2009-06-10-Paper: Graph Databases and the Future of Large-Scale Knowledge Management
17 0.63604313 1638 high scalability-2014-04-28-How Disqus Went Realtime with 165K Messages Per Second and Less than .2 Seconds Latency
18 0.62259233 63 high scalability-2007-08-09-Lots of questions for high scalability - high availability
19 0.6135751 332 high scalability-2008-05-28-Job queue and search engine
20 0.612481 106 high scalability-2007-10-02-Secrets to Fotolog's Scaling Success