high_scalability high_scalability-2012 high_scalability-2012-1168 knowledge-graph by maker-knowledge-mining

1168 high scalability-2012-01-04-How Facebook Handled the New Year's Eve Onslaught


meta infos for this blog

Source: html

Introduction: How does Facebook handle the massive New Year's Eve traffic spike? Thanks to Mike Swift, in  Facebook gets ready for New Year's Eve , we get a little insight as to their method for the madness, nothing really detailed, but still interesting. Problem Setup Facebook expects tha one billion+ photos will be shared on New Year's eve. Facebook's 800 million users are scattered around the world. Three quarters live outside the US. Each user is linked to an average of 130 friends. Photos and posts must appear in less than a second. Opening a homepage requires executing requests on a 100 different servers, and those requests have to be ranked, sorted, and privacy-checked, and then rendered. Different events put different stresses on different parts of Facebook.  Photo and Video Uploads - Holidays require hundreds of terabytes of capacity  News Feed - News events like big sports events and the death of Steve Jobs drive user status updates Coping Strategies Try


Summary: the most important sentenses genereted by tfidf model

sentIndex sentText sentNum sentScore

1 Problem Setup Facebook expects tha one billion+ photos will be shared on New Year's eve. [sent-3, score-0.423]

2 Facebook's 800 million users are scattered around the world. [sent-4, score-0.113]

3 Each user is linked to an average of 130 friends. [sent-6, score-0.087]

4 Opening a homepage requires executing requests on a 100 different servers, and those requests have to be ranked, sorted, and privacy-checked, and then rendered. [sent-8, score-0.223]

5 Different events put different stresses on different parts of Facebook. [sent-9, score-0.362]

6 Photo and Video Uploads - Holidays require hundreds of terabytes of capacity  News Feed - News events like big sports events and the death of Steve Jobs drive user status updates Coping Strategies Try to predict the surge in traffic. [sent-10, score-0.881]

7 Prepare to bring additional capacity online from data centers. [sent-13, score-0.26]

8 Implication is that your architecture can handle additional capacity and make meaningful use of it. [sent-14, score-0.349]

9 Overbuild  a matter of culture so big events aren't big challenges. [sent-15, score-0.243]

10 These are knobs that can be turned to survive system failures or unanticipated surges in traffic. [sent-17, score-0.452]

11 For example, for a  capacity problem they can serve smaller photos to reduce bandwidth usage. [sent-18, score-0.155]

12 The idea is to not go off-line completely when there's a problem, but to degrade gracefully. [sent-19, score-0.113]

13 Top Five Global Events for Facebook status updates in 2011 Death of Osama Bin Laden, May 2 Packers win the Super Bowl, Feb. [sent-20, score-0.13]


similar blogs computed by tfidf model

tfidf for this blog:

wordName wordTfidf (topN-words)

[('eve', 0.326), ('events', 0.243), ('steve', 0.157), ('photos', 0.155), ('capacity', 0.149), ('tha', 0.146), ('year', 0.145), ('bin', 0.137), ('articlesfacebook', 0.137), ('surges', 0.137), ('homepage', 0.137), ('swift', 0.137), ('madness', 0.131), ('verdict', 0.131), ('status', 0.13), ('holidays', 0.126), ('sankar', 0.126), ('quarters', 0.126), ('expects', 0.122), ('krishna', 0.122), ('stresses', 0.119), ('harvest', 0.119), ('unanticipated', 0.119), ('facebook', 0.118), ('anthony', 0.116), ('farmville', 0.116), ('scattered', 0.113), ('degrade', 0.113), ('additional', 0.111), ('knobs', 0.111), ('bowl', 0.111), ('jobs', 0.109), ('ranked', 0.104), ('news', 0.1), ('emergency', 0.099), ('july', 0.097), ('implication', 0.093), ('uploads', 0.092), ('sports', 0.092), ('opening', 0.09), ('surge', 0.089), ('meaningful', 0.089), ('linked', 0.087), ('photo', 0.086), ('sorted', 0.086), ('executing', 0.086), ('survive', 0.085), ('death', 0.084), ('mike', 0.084), ('spike', 0.082)]

similar blogs list:

simIndex simValue blogId blogTitle

same-blog 1 1.0000001 1168 high scalability-2012-01-04-How Facebook Handled the New Year's Eve Onslaught

Introduction: How does Facebook handle the massive New Year's Eve traffic spike? Thanks to Mike Swift, in  Facebook gets ready for New Year's Eve , we get a little insight as to their method for the madness, nothing really detailed, but still interesting. Problem Setup Facebook expects tha one billion+ photos will be shared on New Year's eve. Facebook's 800 million users are scattered around the world. Three quarters live outside the US. Each user is linked to an average of 130 friends. Photos and posts must appear in less than a second. Opening a homepage requires executing requests on a 100 different servers, and those requests have to be ranked, sorted, and privacy-checked, and then rendered. Different events put different stresses on different parts of Facebook.  Photo and Video Uploads - Holidays require hundreds of terabytes of capacity  News Feed - News events like big sports events and the death of Steve Jobs drive user status updates Coping Strategies Try

2 0.24180888 424 high scalability-2008-10-22-EVE Online Architecture

Introduction: Sorry, the content for this post apparently did not make the transition from the old HighScalability website, it's all messed up, but there's still a some useful content... EVE Online is "The World's Largest Game Universe", a massively multiplayer online game ( MMO ) made by CCP. EVE Online's Architecture is unusual for a MMOG because it doesn't divide the player load among different servers or shards. Instead, the same cluster handles the entire EVE universe. It is an interesting to compare this with the Architecture of the Second Life Grid . How do they manage to scale? Information Sources EVE Insider Dev Blog EVE Online FAQ Massively - Eve Evolved: Eve Online's Server Model and its discussion on Slashdot EVE Online Forums Massively Multiplayer Game Development 2 Platform Stackless Python used for both server and client game logic. It allows programmers to reap the benefits of thread-based programming without the performance and complexity p

3 0.19574779 1405 high scalability-2013-02-13-7 Sensible and 1 Really Surprising Way EVE Online Scales to Play Huge Games

Introduction: "Everything in war is simple, but the simplest thing is difficult." -- Carl von Clausewitz Games are proving grounds for software architecture. They combine scale, high performance, challenging problems, a rabid user base, cost sensitivity, and the need for profit. And when games have in-game currency, like EVE Online has, there's money at play, so you can't just get away with a c'est la vie attitude. Engineering must be applied.  In Planning for war: how the EVE Online servers deal with a 3,000 person battle , we learn some techniques EVE Online uses to handle large games: 7 Sensible... Do nothing. Most games are manageable or have spikes that quickly dissipate.  Run it Hot . There's nothing to throttle as servers run at 100%. Why waste money? Use all your CPU. Shard it . Games are sharded by solar system and multiple solar systems run on a node. Move it . Games are moved when a machine becomes overloaded. Live Node Remap, where a live game is moved to

4 0.16378513 966 high scalability-2010-12-31-Facebook in 20 Minutes: 2.7M Photos, 10.2M Comments, 4.6M Messages

Introduction: To celebrate the new year Facebook has shared the results of a little end of the year introspection. It has been a fecund year for Facebook: 43,869,800 changed their status to single 3,025,791 changed their status to "it's complicated" 28,460,516 changed their status to in a relationship 5,974,574 changed their status to engaged 36,774,801 changes their status to married If these numbers are simply to large to grasp, it doesn't get any better when you look at happens in a mere 20 minutes: Shared links: 1,000,000  Tagged photos: 1,323,000 Event invites sent out: 1,484,000 Wall Posts: 1,587,000  Status updates: 1,851,000 Friend requests accepted: 1,972,000 Photos uploaded: 2,716,000 Comments: 10,208,000 Message: 4,632,000 If you want to see how Facebook supports these huge numbers take a look at a few  posts . One wonders what the new year will bring? Related Articles What the World Eats from Time Magazine  A Day in the Life of an An

5 0.1476744 840 high scalability-2010-06-10-The Four Meta Secrets of Scaling at Facebook

Introduction: Aditya Agarwal, Director of Engineering at Facebook, gave an excellent  Scale at Facebook talk that covers their architecture, but the talk is really more about how to scale an organization by preserving the best parts of its culture. The key take home of the talk is:  You can get the code right, you can get the products right, but you need to get the culture right first. If you don't get the culture right then your company won't scale. This leads into the four meta secrets of scaling at Facebook: Scaling takes Iteration Don't Over Design Choose the right tool for the job, but realize that your choice comes with overhead. Get the culture right. Move Fast - break things. Huge Impact - small teams. Be bold - innovate. Some Background  Facebook is big : 400 million active users; users spend an average of 20 minutes a day; 5 billion pieces of content (status updates, comments, likes, photo uploads, video uploads, chat messages, inbox messages, group events, f

6 0.13897303 774 high scalability-2010-02-08-How FarmVille Scales to Harvest 75 Million Players a Month

7 0.11784588 760 high scalability-2010-01-13-10 Hot Scalability Links for January 13, 2010

8 0.11347423 1197 high scalability-2012-02-21-Pixable Architecture - Crawling, Analyzing, and Ranking 20 Million Photos a Day

9 0.11230461 152 high scalability-2007-11-13-Flickr Architecture

10 0.10566946 1123 high scalability-2011-09-23-The Real News is Not that Facebook Serves Up 1 Trillion Pages a Month…

11 0.10394613 106 high scalability-2007-10-02-Secrets to Fotolog's Scaling Success

12 0.10054224 643 high scalability-2009-06-29-How to Succeed at Capacity Planning Without Really Trying : An Interview with Flickr's John Allspaw on His New Book

13 0.098951206 1224 high scalability-2012-04-09-The Instagram Architecture Facebook Bought for a Cool Billion Dollars

14 0.097350046 513 high scalability-2009-02-16-Handle 1 Billion Events Per Day Using a Memory Grid

15 0.096336722 1008 high scalability-2011-03-22-Facebook's New Realtime Analytics System: HBase to Process 20 Billion Events Per Day

16 0.092720062 720 high scalability-2009-10-12-High Performance at Massive Scale – Lessons learned at Facebook

17 0.09109246 1152 high scalability-2011-12-06-Instagram Architecture: 14 Million users, Terabytes of Photos, 100s of Instances, Dozens of Technologies

18 0.088883564 1622 high scalability-2014-03-31-How WhatsApp Grew to Nearly 500 Million Users, 11,000 cores, and 70 Million Messages a Second

19 0.085682981 129 high scalability-2007-10-23-Hire Facebook, Ning, and Salesforce to Scale for You

20 0.084474899 1359 high scalability-2012-11-15-Gone Fishin': Justin.Tv's Live Video Broadcasting Architecture


similar blogs computed by lsi model

lsi for this blog:

topicId topicWeight

[(0, 0.136), (1, 0.075), (2, 0.016), (3, -0.039), (4, 0.01), (5, -0.076), (6, -0.039), (7, 0.071), (8, 0.034), (9, -0.008), (10, 0.053), (11, 0.036), (12, 0.05), (13, 0.013), (14, -0.044), (15, 0.095), (16, 0.034), (17, -0.005), (18, -0.007), (19, 0.058), (20, 0.048), (21, 0.037), (22, 0.042), (23, -0.011), (24, -0.003), (25, -0.033), (26, 0.006), (27, 0.028), (28, 0.03), (29, -0.016), (30, 0.015), (31, -0.002), (32, -0.044), (33, -0.035), (34, 0.043), (35, -0.002), (36, 0.073), (37, -0.123), (38, 0.019), (39, -0.038), (40, 0.013), (41, -0.03), (42, -0.014), (43, 0.032), (44, -0.01), (45, 0.001), (46, 0.013), (47, -0.037), (48, 0.001), (49, -0.027)]

similar blogs list:

simIndex simValue blogId blogTitle

same-blog 1 0.95467281 1168 high scalability-2012-01-04-How Facebook Handled the New Year's Eve Onslaught

Introduction: How does Facebook handle the massive New Year's Eve traffic spike? Thanks to Mike Swift, in  Facebook gets ready for New Year's Eve , we get a little insight as to their method for the madness, nothing really detailed, but still interesting. Problem Setup Facebook expects tha one billion+ photos will be shared on New Year's eve. Facebook's 800 million users are scattered around the world. Three quarters live outside the US. Each user is linked to an average of 130 friends. Photos and posts must appear in less than a second. Opening a homepage requires executing requests on a 100 different servers, and those requests have to be ranked, sorted, and privacy-checked, and then rendered. Different events put different stresses on different parts of Facebook.  Photo and Video Uploads - Holidays require hundreds of terabytes of capacity  News Feed - News events like big sports events and the death of Steve Jobs drive user status updates Coping Strategies Try

2 0.77270979 966 high scalability-2010-12-31-Facebook in 20 Minutes: 2.7M Photos, 10.2M Comments, 4.6M Messages

Introduction: To celebrate the new year Facebook has shared the results of a little end of the year introspection. It has been a fecund year for Facebook: 43,869,800 changed their status to single 3,025,791 changed their status to "it's complicated" 28,460,516 changed their status to in a relationship 5,974,574 changed their status to engaged 36,774,801 changes their status to married If these numbers are simply to large to grasp, it doesn't get any better when you look at happens in a mere 20 minutes: Shared links: 1,000,000  Tagged photos: 1,323,000 Event invites sent out: 1,484,000 Wall Posts: 1,587,000  Status updates: 1,851,000 Friend requests accepted: 1,972,000 Photos uploaded: 2,716,000 Comments: 10,208,000 Message: 4,632,000 If you want to see how Facebook supports these huge numbers take a look at a few  posts . One wonders what the new year will bring? Related Articles What the World Eats from Time Magazine  A Day in the Life of an An

3 0.76020074 792 high scalability-2010-03-10-How FarmVille Scales - The Follow-up

Introduction: Several readers had follow-up questions in response to How FarmVille Scales to Harvest 75 Million Players a Month . Here are Luke's response to those questions (and a few of mine). How does social networking makes things easier or harder? The primary interesting aspect of social networking games is how you wind up with a graph of connected users who need to be access each other's data on a frequent basis. This makes the overall dataset difficult if not impossible to partition. What are examples of the Facebook calls you try to avoid and how they impact game play? We can make a call for facebook friend data to retrieve information about your friends playing the game. Normally, we show a friend ladder at the bottom of the game that shows friend information, including name and facebook photo.  Can you say where your cache is, what form it takes, and how much cached there is? Do you have a peering relationship with Facebook, as one might expect at that bandwidth?

4 0.7416029 870 high scalability-2010-08-02-7 Scaling Strategies Facebook Used to Grow to 500 Million Users

Introduction: Robert Johnson,   a director of engineering at Facebook, celebrated Facebook's monumental achievement of reaching 500 million users by sharing the scaling principles that helped  reach that milestone. In case you weren't suitably impressed by the 500 million user number, Robert ratchets up the numbers game with these impressive figures: 1 million users per engineer 500 million active users 100 billion hits per day 50 billion photos 2 trillion objects cached, with hundreds of millions of requests per second 130TB of logs every day How did Facebook get to this point? People Matter Most . It's people who build and run systems. The best tools for scaling are an engineering and operations teams that can handle anything. Scale Horizontally . Handling exponentially growing traffic requires spreading load arbitrarily across many machines. Using different databases for tables like accounts and profiles only doubles capacity. This approach hurts efficiency, but

5 0.70447832 1123 high scalability-2011-09-23-The Real News is Not that Facebook Serves Up 1 Trillion Pages a Month…

Introduction: There’s some amount of debate whether Facebook really crossed over the one trillion page view per month threshold. While one report says it did, another respected firm says it did not ; that its monthly page views are a mere 467 billion per month. In the big scheme of things, the discrepancy is somewhat irrelevant, as neither show the true load on Facebook’s infrastructure – which is far more impressive a set of numbers than its externally measured “page view” metric.  Mashable reported in “ Facebook Surpasses 1 Trillion Pageviews per Month ” that the social networking giant saw “approximately 870 million unique visitors in June and 860 million in July” and followed up with some per visitor statistics, indicating “each visitor averaged approximately 1,160 page views in July and 40 per visit — enormous by any standard. Time spent on the site was around 25 minutes per user.” From an architectural standpoint it’s not just about the page views. It’s about requests and responses,

6 0.70141912 774 high scalability-2010-02-08-How FarmVille Scales to Harvest 75 Million Players a Month

7 0.69775653 840 high scalability-2010-06-10-The Four Meta Secrets of Scaling at Facebook

8 0.69100058 1602 high scalability-2014-02-26-The WhatsApp Architecture Facebook Bought For $19 Billion

9 0.66994452 1619 high scalability-2014-03-26-Oculus Causes a Rift, but the Facebook Deal Will Avoid a Scaling Crisis for Virtual Reality

10 0.66301364 720 high scalability-2009-10-12-High Performance at Massive Scale – Lessons learned at Facebook

11 0.65571684 1405 high scalability-2013-02-13-7 Sensible and 1 Really Surprising Way EVE Online Scales to Play Huge Games

12 0.61811304 424 high scalability-2008-10-22-EVE Online Architecture

13 0.61397797 904 high scalability-2010-09-21-Playfish's Social Gaming Architecture - 50 Million Monthly Users and Growing

14 0.60489213 1412 high scalability-2013-02-25-SongPop Scales to 1 Million Active Users on GAE, Showing PaaS is not Passé

15 0.59803069 758 high scalability-2010-01-11-Have We Reached the End of Scaling?

16 0.59715176 1595 high scalability-2014-02-13-Snabb Switch - Skip the OS and Get 40 million Requests Per Second in Lua

17 0.59547365 464 high scalability-2008-12-13-Strategy: Facebook Tweaks to Handle 6 Time as Many Memcached Requests

18 0.59481478 129 high scalability-2007-10-23-Hire Facebook, Ning, and Salesforce to Scale for You

19 0.59195995 1228 high scalability-2012-04-16-Instagram Architecture Update: What’s new with Instagram?

20 0.58015347 1444 high scalability-2013-04-23-Facebook Secrets of Web Performance


similar blogs computed by lda model

lda for this blog:

topicId topicWeight

[(1, 0.142), (2, 0.161), (40, 0.028), (51, 0.345), (61, 0.09), (79, 0.072), (85, 0.044), (94, 0.034)]

similar blogs list:

simIndex simValue blogId blogTitle

1 0.91037899 818 high scalability-2010-04-30-Behind the scenes of an online marketplace

Introduction: In a presentation originally held at the 4. O2 Hosting Event in Hamburg, I spoke about the technology at a large online marketplace in Germany called Hitmeister .  Some of the topics discussed include: what makes up a marketplace? technically system principles development patterns tools philosophy data model hardware I am looking forward to comments and suggestions for both the presentation and our work.

same-blog 2 0.88257927 1168 high scalability-2012-01-04-How Facebook Handled the New Year's Eve Onslaught

Introduction: How does Facebook handle the massive New Year's Eve traffic spike? Thanks to Mike Swift, in  Facebook gets ready for New Year's Eve , we get a little insight as to their method for the madness, nothing really detailed, but still interesting. Problem Setup Facebook expects tha one billion+ photos will be shared on New Year's eve. Facebook's 800 million users are scattered around the world. Three quarters live outside the US. Each user is linked to an average of 130 friends. Photos and posts must appear in less than a second. Opening a homepage requires executing requests on a 100 different servers, and those requests have to be ranked, sorted, and privacy-checked, and then rendered. Different events put different stresses on different parts of Facebook.  Photo and Video Uploads - Holidays require hundreds of terabytes of capacity  News Feed - News events like big sports events and the death of Steve Jobs drive user status updates Coping Strategies Try

3 0.78321666 98 high scalability-2007-09-18-Sync data on all servers

Introduction: I have a few apache servers ( arround 11 atm ) serving a small amount of data ( arround 44 gigs right now ). For some time I have been using rsync to keep all the content equal on all servers, but the amount of data has been growing, and rsync takes a few too much time to "compare" all data from source to destination, and create a lot of I/O. I have been taking a look at MogileFS, it seems a good and reliable option, but as the fuse module is not finished, we should have to rewrite all our apps, and its not an option atm. Any ideas? I just want a "real time, non resource-hungry" solution alternative for rsync. If I get more features on the way, then they are welcome :) Why I prefer to use a Distributed File System instead of using NAS + NFS? - I need 2 NAS, if I dont want a point of failure, and NAS hard is expensive. - Non-shared hardware, all server has their own local disks. - As files are replicated, I can save a lot of money, RAID is not a MUST. Thn

4 0.77849531 838 high scalability-2010-06-08-Sponsored Post: Jobs: Digg, Huffington Post Events: Velocity Conference, Social Developer Summit

Introduction: Sponsored Jobs Get Your High Scalability Fix at Digg Huffington Post: Linux System Administrator Sponsored Events Velocity Web Performance and Operations Conference - Fast by Default Social Developer Summit  (HighScalability.com readers save 15% with discount code SDSHS)   Get Your High Scalability Fix at Digg   Interested in working on cutting-edge high-scale infrastructure at Digg? We're making a big investment in scaling and have committed to the NoSQL (Not only SQL) path with Cassandra . We're using other open-source infrastructure to help us scale including Hadoop, RabbitMQ, Zookeeper, Thrift, HDFS and Lucene. We're rewriting Digg from the ground up and we need amazing developers to join our world-class team. If you think you are up for the challenge, or you know someone who might be, take a look at our jobs page for more information. Huffington Post: Linux System Administrator HuffingtonPost has an immediate need for a  Lin

5 0.74629331 1629 high scalability-2014-04-10-Paper: Scalable Atomic Visibility with RAMP Transactions - Scale Linearly to 100 Servers

Introduction: We are not yet at the End of History for database theory as Peter Bailis and the Database Group at UC Berkeley continue to prove with a great companion blog post  to their new paper:  Scalable Atomic Visibility with RAMP Transactions . I like the approach of pairing a blog post with a paper. A paper almost by definition is formal, but a blog post can help put a paper in context and give it some heart. From the abstract: Databases can provide scalability by partitioning data across several servers. However, multi-partition, multi-operation transactional access is often expensive, employing coordination-intensive locking, validation, or scheduling mechanisms. Accordingly, many real-world systems avoid mechanisms that provide useful semantics for multi-partition operations . This leads to incorrect behavior for a large class of applications including secondary indexing, foreign key enforcement, and materialized view maintenance . In this work, we identify a new isolation mode

6 0.73961473 481 high scalability-2009-01-02-Strategy: Understanding Your Data Leads to the Best Scalability Solutions

7 0.73218089 741 high scalability-2009-11-16-Building Scalable Systems Using Data as a Composite Material

8 0.72979164 1134 high scalability-2011-10-28-Stuff The Internet Says On Scalability For October 28, 2011

9 0.72170353 1271 high scalability-2012-06-25-StubHub Architecture: The Surprising Complexity Behind the World’s Largest Ticket Marketplace

10 0.70952153 1644 high scalability-2014-05-07-Update on Disqus: It's Still About Realtime, But Go Demolishes Python

11 0.70844907 298 high scalability-2008-04-07-Lazy web sites run faster

12 0.70334393 846 high scalability-2010-06-22-Sponsored Post: Jobs: Etsy, Digg, Huffington Post Event: Velocity Conference

13 0.70246458 510 high scalability-2009-02-09-Paper: Consensus Protocols: Two-Phase Commit

14 0.69818604 953 high scalability-2010-12-03-GPU vs CPU Smackdown : The Rise of Throughput-Oriented Architectures

15 0.68874359 138 high scalability-2007-10-30-Feedblendr Architecture - Using EC2 to Scale

16 0.66191417 626 high scalability-2009-06-10-Paper: Graph Databases and the Future of Large-Scale Knowledge Management

17 0.63604313 1638 high scalability-2014-04-28-How Disqus Went Realtime with 165K Messages Per Second and Less than .2 Seconds Latency

18 0.62259233 63 high scalability-2007-08-09-Lots of questions for high scalability - high availability

19 0.6135751 332 high scalability-2008-05-28-Job queue and search engine

20 0.612481 106 high scalability-2007-10-02-Secrets to Fotolog's Scaling Success