high_scalability high_scalability-2010 high_scalability-2010-840 knowledge-graph by maker-knowledge-mining
Source: html
Introduction: Aditya Agarwal, Director of Engineering at Facebook, gave an excellent Scale at Facebook talk that covers their architecture, but the talk is really more about how to scale an organization by preserving the best parts of its culture. The key take home of the talk is: You can get the code right, you can get the products right, but you need to get the culture right first. If you don't get the culture right then your company won't scale. This leads into the four meta secrets of scaling at Facebook: Scaling takes Iteration Don't Over Design Choose the right tool for the job, but realize that your choice comes with overhead. Get the culture right. Move Fast - break things. Huge Impact - small teams. Be bold - innovate. Some Background Facebook is big : 400 million active users; users spend an average of 20 minutes a day; 5 billion pieces of content (status updates, comments, likes, photo uploads, video uploads, chat messages, inbox messages, group events, f
sentIndex sentText sentNum sentScore
1 The key take home of the talk is: You can get the code right, you can get the products right, but you need to get the culture right first. [sent-2, score-0.929]
2 If you don't get the culture right then your company won't scale. [sent-3, score-0.783]
3 This leads into the four meta secrets of scaling at Facebook: Scaling takes Iteration Don't Over Design Choose the right tool for the job, but realize that your choice comes with overhead. [sent-4, score-0.716]
4 Each type of content has its own access pattern which makes scaling difficult. [sent-11, score-0.186]
5 In Facebook users can't be partitioned because users can join any network. [sent-16, score-0.186]
6 Every new user can access any other user's data which means there's no way to partition users by geography, or any other metric. [sent-18, score-0.178]
7 Uploader stored the file in NFS and the meta-data was stored in MySQL. [sent-32, score-0.184]
8 It worked for the first 3 months and caused a lot of sleepless nights. [sent-33, score-0.157]
9 The third generation is an overlay system that creates a file that is a blob stored in the file system. [sent-42, score-0.189]
10 Images are stored in the blob and you know the offset of the photo in the blob. [sent-43, score-0.282]
11 Choose the right tool for the job, but realize that your choice comes with overhead . [sent-54, score-0.429]
12 Build an environment internally which promotes building the right thing first and then fixing as needed, not worrying about innovating, not worrying about breaking things, thinking big, thinking what is the next thing you need to build after the building the first thing. [sent-66, score-0.796]
13 You can get the code right, you can get the products right, but you need to get the culture right first. [sent-67, score-0.929]
14 If you don't get the culture right then your company won't scale. [sent-68, score-0.783]
15 This is a particular instance, but the most important thing is the culture, getting people to believe that the most important thing is how quickly they can move. [sent-75, score-0.281]
16 Get the right set of people, empower them, and let them work. [sent-79, score-0.281]
17 If you can't push responsibility down and you isolate the number of people who feel they are real owners, then the only people you'll be able to motivate are the people who think they are the real owners. [sent-89, score-0.567]
18 Isolate the part of the culture that you value and want to preserve. [sent-91, score-0.353]
19 Facebook organizes hackathons, the point of which is to show new engineers that if they come in at 8AM they can get a new feature up on the site in 12 hours. [sent-93, score-0.19]
20 Move fast isn't just a platitude, a company has to come up with ways to make people feel it's a reality. [sent-94, score-0.203]
wordName wordTfidf (topN-words)
[('culture', 0.353), ('right', 0.21), ('hiphop', 0.183), ('facebook', 0.17), ('payoff', 0.136), ('photos', 0.125), ('get', 0.122), ('realize', 0.112), ('break', 0.107), ('choice', 0.107), ('php', 0.106), ('owners', 0.106), ('people', 0.105), ('scaling', 0.101), ('isolate', 0.099), ('ownership', 0.099), ('uploads', 0.099), ('company', 0.098), ('blob', 0.097), ('meta', 0.096), ('nfs', 0.095), ('users', 0.093), ('photo', 0.093), ('worrying', 0.093), ('stored', 0.092), ('secrets', 0.09), ('thing', 0.088), ('month', 0.087), ('responsibility', 0.085), ('million', 0.085), ('access', 0.085), ('small', 0.084), ('chat', 0.082), ('ok', 0.08), ('friend', 0.08), ('sleepless', 0.079), ('uploader', 0.079), ('articlesshould', 0.079), ('hackathons', 0.079), ('first', 0.078), ('everyone', 0.076), ('transformer', 0.074), ('aditya', 0.074), ('empower', 0.071), ('worry', 0.068), ('memcache', 0.068), ('build', 0.068), ('organizes', 0.068), ('motivate', 0.068), ('teams', 0.067)]
simIndex simValue blogId blogTitle
same-blog 1 1.0000001 840 high scalability-2010-06-10-The Four Meta Secrets of Scaling at Facebook
Introduction: Aditya Agarwal, Director of Engineering at Facebook, gave an excellent Scale at Facebook talk that covers their architecture, but the talk is really more about how to scale an organization by preserving the best parts of its culture. The key take home of the talk is: You can get the code right, you can get the products right, but you need to get the culture right first. If you don't get the culture right then your company won't scale. This leads into the four meta secrets of scaling at Facebook: Scaling takes Iteration Don't Over Design Choose the right tool for the job, but realize that your choice comes with overhead. Get the culture right. Move Fast - break things. Huge Impact - small teams. Be bold - innovate. Some Background Facebook is big : 400 million active users; users spend an average of 20 minutes a day; 5 billion pieces of content (status updates, comments, likes, photo uploads, video uploads, chat messages, inbox messages, group events, f
2 0.41581663 1444 high scalability-2013-04-23-Facebook Secrets of Web Performance
Introduction: This is a repost of part 1 of an interview I did for the Boundary blog . Boundary: What is Facebook’s secret sauce for managing what’s got to be the biggest Big Data project, if you will, on the Web? Hoff: From several presentations we’ve learned what Facebook insiders like Aditya Agarwal and Robert Johnson , both former Directors of Engineering, consider their secret sauce: Scaling Takes Iteration . Solutions often work in the beginning, but you’ll have to modify them as you go. PHP, for example, is simple to use at first, but is not a good choice when you have tens of thousands of web servers. Scaling Takes Iteration . You can say that again. Don’t Over-Design . Just use what you need as you scale your system out. Figure out where you need to iterate on a solution, optimize something, or completely build a part of the stack yourself. Choose the Right Tool for the Job . Realize that any choice comes with overhead. If you really need to use P
3 0.22657001 870 high scalability-2010-08-02-7 Scaling Strategies Facebook Used to Grow to 500 Million Users
Introduction: Robert Johnson, a director of engineering at Facebook, celebrated Facebook's monumental achievement of reaching 500 million users by sharing the scaling principles that helped reach that milestone. In case you weren't suitably impressed by the 500 million user number, Robert ratchets up the numbers game with these impressive figures: 1 million users per engineer 500 million active users 100 billion hits per day 50 billion photos 2 trillion objects cached, with hundreds of millions of requests per second 130TB of logs every day How did Facebook get to this point? People Matter Most . It's people who build and run systems. The best tools for scaling are an engineering and operations teams that can handle anything. Scale Horizontally . Handling exponentially growing traffic requires spreading load arbitrarily across many machines. Using different databases for tables like accounts and profiles only doubles capacity. This approach hurts efficiency, but
4 0.19335109 1240 high scalability-2012-05-07-Startups are Creating a New System of the World for IT
Introduction: It remains that, from the same principles, I now demonstrate the frame of the System of the World. -- Isaac Newton The practice of IT reminds me a lot of the practice of science before Isaac Newton. Aristotelianism was dead, but there was nothing to replace it. Then Newton came along, created a scientific revolution with his System of the World . And everything changed. That was New System of the World number one. New System of the World number two was written about by the incomparable Neal Stephenson in his incredible Baroque Cycle series. It explores the singular creation of a new way of organizing society grounded in new modes of thought in business, religion, politics, and science. Our modern world emerged Enlightened as it could from this roiling cauldron of forces. In IT we may have had a Leonardo da Vinci or even a Galileo, but we’ve never had our Newton. Maybe we don't need a towering genius to make everything clear? For years startups, like the frenetically inventive
5 0.18640317 721 high scalability-2009-10-13-Why are Facebook, Digg, and Twitter so hard to scale?
Introduction: Real-time social graphs (connectivity between people, places, and things). That's why scaling Facebook is hard says Jeff Rothschild , Vice President of Technology at Facebook. Social networking sites like Facebook, Digg, and Twitter are simply harder than traditional websites to scale. Why is that? Why would social networking sites be any more difficult to scale than traditional web sites? Let's find out. Traditional websites are easier to scale than social networking sites for two reasons: They usually access only their own data and common cached data. Only 1-2% of users are active on the site at one time. Imagine a huge site like Yahoo. When you come to Yahoo they can get your profile record with one get and that's enough to build your view of the website for you. It's relatively straightforward to scale systems based around single records using distributed hashing schemes . And since only a few percent of the people are on the site at once it takes comparatively little
6 0.18022852 106 high scalability-2007-10-02-Secrets to Fotolog's Scaling Success
7 0.18004231 152 high scalability-2007-11-13-Flickr Architecture
8 0.17783178 96 high scalability-2007-09-18-Amazon Architecture
11 0.1710857 1068 high scalability-2011-06-27-TripAdvisor Architecture - 40M Visitors, 200M Dynamic Page Views, 30TB Data
12 0.16931383 1440 high scalability-2013-04-15-Scaling Pinterest - From 0 to 10s of Billions of Page Views a Month in Two Years
13 0.16886361 1008 high scalability-2011-03-22-Facebook's New Realtime Analytics System: HBase to Process 20 Billion Events Per Day
14 0.16763599 720 high scalability-2009-10-12-High Performance at Massive Scale – Lessons learned at Facebook
15 0.16668609 1461 high scalability-2013-05-20-The Tumblr Architecture Yahoo Bought for a Cool Billion Dollars
16 0.16644606 1191 high scalability-2012-02-13-Tumblr Architecture - 15 Billion Page Views a Month and Harder to Scale than Twitter
18 0.16488524 313 high scalability-2008-05-02-Friends for Sale Architecture - A 300 Million Page View-Month Facebook RoR App
19 0.1612488 1197 high scalability-2012-02-21-Pixable Architecture - Crawling, Analyzing, and Ranking 20 Million Photos a Day
20 0.16026887 954 high scalability-2010-12-06-What the heck are you actually using NoSQL for?
topicId topicWeight
[(0, 0.318), (1, 0.098), (2, -0.017), (3, -0.098), (4, 0.093), (5, -0.114), (6, -0.088), (7, 0.089), (8, 0.066), (9, 0.009), (10, 0.019), (11, 0.096), (12, 0.026), (13, 0.042), (14, -0.01), (15, 0.054), (16, -0.011), (17, -0.002), (18, -0.018), (19, 0.083), (20, 0.058), (21, 0.046), (22, 0.066), (23, 0.012), (24, 0.019), (25, -0.072), (26, 0.059), (27, -0.048), (28, 0.032), (29, 0.022), (30, -0.118), (31, 0.005), (32, -0.021), (33, -0.034), (34, 0.006), (35, 0.0), (36, 0.037), (37, -0.09), (38, -0.006), (39, 0.007), (40, -0.054), (41, -0.037), (42, 0.007), (43, -0.061), (44, -0.049), (45, -0.03), (46, 0.045), (47, -0.028), (48, 0.026), (49, -0.063)]
simIndex simValue blogId blogTitle
same-blog 1 0.97910649 840 high scalability-2010-06-10-The Four Meta Secrets of Scaling at Facebook
Introduction: Aditya Agarwal, Director of Engineering at Facebook, gave an excellent Scale at Facebook talk that covers their architecture, but the talk is really more about how to scale an organization by preserving the best parts of its culture. The key take home of the talk is: You can get the code right, you can get the products right, but you need to get the culture right first. If you don't get the culture right then your company won't scale. This leads into the four meta secrets of scaling at Facebook: Scaling takes Iteration Don't Over Design Choose the right tool for the job, but realize that your choice comes with overhead. Get the culture right. Move Fast - break things. Huge Impact - small teams. Be bold - innovate. Some Background Facebook is big : 400 million active users; users spend an average of 20 minutes a day; 5 billion pieces of content (status updates, comments, likes, photo uploads, video uploads, chat messages, inbox messages, group events, f
2 0.90426403 870 high scalability-2010-08-02-7 Scaling Strategies Facebook Used to Grow to 500 Million Users
Introduction: Robert Johnson, a director of engineering at Facebook, celebrated Facebook's monumental achievement of reaching 500 million users by sharing the scaling principles that helped reach that milestone. In case you weren't suitably impressed by the 500 million user number, Robert ratchets up the numbers game with these impressive figures: 1 million users per engineer 500 million active users 100 billion hits per day 50 billion photos 2 trillion objects cached, with hundreds of millions of requests per second 130TB of logs every day How did Facebook get to this point? People Matter Most . It's people who build and run systems. The best tools for scaling are an engineering and operations teams that can handle anything. Scale Horizontally . Handling exponentially growing traffic requires spreading load arbitrarily across many machines. Using different databases for tables like accounts and profiles only doubles capacity. This approach hurts efficiency, but
3 0.88558972 1444 high scalability-2013-04-23-Facebook Secrets of Web Performance
Introduction: This is a repost of part 1 of an interview I did for the Boundary blog . Boundary: What is Facebook’s secret sauce for managing what’s got to be the biggest Big Data project, if you will, on the Web? Hoff: From several presentations we’ve learned what Facebook insiders like Aditya Agarwal and Robert Johnson , both former Directors of Engineering, consider their secret sauce: Scaling Takes Iteration . Solutions often work in the beginning, but you’ll have to modify them as you go. PHP, for example, is simple to use at first, but is not a good choice when you have tens of thousands of web servers. Scaling Takes Iteration . You can say that again. Don’t Over-Design . Just use what you need as you scale your system out. Figure out where you need to iterate on a solution, optimize something, or completely build a part of the stack yourself. Choose the Right Tool for the Job . Realize that any choice comes with overhead. If you really need to use P
4 0.8416816 129 high scalability-2007-10-23-Hire Facebook, Ning, and Salesforce to Scale for You
Introduction: One of the premier scaling strategies is always: get someone else to do the work for you. But unlike Huckleberry Finn in Tom Sawyer , you won't have to trick anyone into whitewashing a fence for you. Times have changed. Companies like Ning , Facebook , and Salesforce are more than happy to help. Their price: lock-in. Previously you had few options when building a "real" website. You needed to do everything yourself. Infrastructure and application were all yours. Then companies stepped in by commoditizing parts of the infrastructure, but the application was still yours. The next step is full on Borg take no prisoners assimilation where the infrastructure and application are built as one collective. What you have to decide as someone faced with building a scalable website is if these new options are worth the price. Feeding this explosion of choice is one of the new strategy games on the intertubes: the Internet Platform Game. Ning's Marc Andreessen defines a platform as: a
Introduction: Facebook has been teasing us. While many of their recent acquisitions have been surprising, shocking is the only word adequately describing Facebook's 5 day whirlwind acquisition of Oculus , immersive virtual reality visionaries, for a now paltry sounding $2 billion. The backlash is a pandemic, jumping across social networks with the speed only a meme powered by the directly unaffected can generate. For more than 30 years VR has been the dream burning in the heart of every science fiction fan. Now that this future might finally be here, Facebook’s ownage makes it seem like a wonderful and hopeful timeline has been choked off, killing the Metaverse before it even had a chance to begin. For the many who voted for an open future with their Kickstarter dollars , there’s a deep and personal sense of betrayal, despite Facebook’s promise to leave Oculus alone. The intensity of the reaction is because Oculus matters to people. It's new, it's different, it create
6 0.83608067 966 high scalability-2010-12-31-Facebook in 20 Minutes: 2.7M Photos, 10.2M Comments, 4.6M Messages
7 0.83121461 1168 high scalability-2012-01-04-How Facebook Handled the New Year's Eve Onslaught
8 0.8190589 1602 high scalability-2014-02-26-The WhatsApp Architecture Facebook Bought For $19 Billion
9 0.8096258 264 high scalability-2008-03-03-Read This Site and Ace Your Next Interview!
10 0.80004877 378 high scalability-2008-09-03-Some Facebook Secrets to Better Operations
11 0.77364218 1011 high scalability-2011-03-25-Did the Microsoft Stack Kill MySpace?
12 0.76395768 721 high scalability-2009-10-13-Why are Facebook, Digg, and Twitter so hard to scale?
13 0.76235938 1323 high scalability-2012-09-15-4 Reasons Facebook Dumped HTML5 and Went Native
14 0.75560236 758 high scalability-2010-01-11-Have We Reached the End of Scaling?
15 0.75407255 1617 high scalability-2014-03-21-Stuff The Internet Says On Scalability For March 21st, 2014
16 0.74769264 1123 high scalability-2011-09-23-The Real News is Not that Facebook Serves Up 1 Trillion Pages a Month…
17 0.74723125 1223 high scalability-2012-04-06-Stuff The Internet Says On Scalability For April 6, 2012
18 0.74446654 1228 high scalability-2012-04-16-Instagram Architecture Update: What’s new with Instagram?
19 0.74333835 1008 high scalability-2011-03-22-Facebook's New Realtime Analytics System: HBase to Process 20 Billion Events Per Day
20 0.74003506 1171 high scalability-2012-01-09-The Etsy Saga: From Silos to Happy to Billions of Pageviews a Month
topicId topicWeight
[(1, 0.217), (2, 0.177), (10, 0.062), (28, 0.099), (40, 0.04), (61, 0.111), (79, 0.12), (85, 0.013), (94, 0.075), (96, 0.014)]
simIndex simValue blogId blogTitle
1 0.9669441 806 high scalability-2010-04-08-Hot Scalability Links for April 8, 2010
Introduction: Scalability porn (SFW). Real time meter for the number of ads being served by doubleclick. Amazing. A constant ~390,000 impressions a second are being served and 25 trillion since 1996. Thanks to Mike Rhoads for title idea. Scalability? Don't worry. Application complexity? Worry by Joe McKendrick. The next challenge on enterprise agendas: application complexity. This is something that lots of hardware — whether from the cloud or internal data center — cannot fix Leo Laporte and Steve Gibson talked about how the iPad was a denial of service attack on UPS delivery schedules. UPS trucks were filled with iPads. Cassandra: Fact vs fiction . Jonathan Ellies puts the beatdown on Cassandra misinformation. Don't you dare say Cassandra can't work across datacenters! JIT'd code calling conventions . Cliff Click Jr shows how Java’s calling convention can match compiled C code in speed, but allows for the flexibility of calling (code,slow) non-JIT'd code . Some assembly code re
2 0.96390343 1506 high scalability-2013-08-23-Stuff The Internet Says On Scalability For August 23, 2013
Introduction: Hey, it's HighScalability time: ( Parkour is to terrain as programming is to frameworks ) 5x : AWS vs combined size of other cloud vendors; Every Second on The Internet : Why we need so many servers. Quotable Quotes: @chaliy : Today I learned that I do not understand how #azure scaling works, instance scale does not affect requests/sec I can load. @Lariar : Note how crazy this is. An international launch would have been a huge deal. Now it's just another thing you do. smacktoward : The problem with relying on donations is that people don't make donations. @toddhoffious : Programming is a tool built by logical positivists to solve the problems of idealists and pragmatists. We have a fundamental mismatch here. @etherealmind : Me: "Weird, my phone data isn't working" Them: "They turned the 3G off at the tower because it interferes with the particle accelerator" John Carmack : In computer science, just about t
same-blog 3 0.9613198 840 high scalability-2010-06-10-The Four Meta Secrets of Scaling at Facebook
Introduction: Aditya Agarwal, Director of Engineering at Facebook, gave an excellent Scale at Facebook talk that covers their architecture, but the talk is really more about how to scale an organization by preserving the best parts of its culture. The key take home of the talk is: You can get the code right, you can get the products right, but you need to get the culture right first. If you don't get the culture right then your company won't scale. This leads into the four meta secrets of scaling at Facebook: Scaling takes Iteration Don't Over Design Choose the right tool for the job, but realize that your choice comes with overhead. Get the culture right. Move Fast - break things. Huge Impact - small teams. Be bold - innovate. Some Background Facebook is big : 400 million active users; users spend an average of 20 minutes a day; 5 billion pieces of content (status updates, comments, likes, photo uploads, video uploads, chat messages, inbox messages, group events, f
4 0.94272023 1294 high scalability-2012-08-01-Prismatic Update: Machine Learning on Documents and Users
Introduction: In update to Prismatic Architecture - Using Machine Learning on Social Networks to Figure Out What You Should Read on the Web , Jason Wolfe, even in the face of deadening fatigue from long nights spent getting their iPhone app out, has gallantly agreed to talk a little more about Primatic's approach to Machine Learning. Documents and users are two areas where Prismatic applies ML (machine learning): ML on Documents Given an HTML document:Â learn how to extract the main text of the page (rather than the sidebar, footer, comments, etc), its title, author, best images, etc determine features for relevance (e.g., what the article is about, topics, etc.) The setup for most of these tasks is pretty typical. Models are trained using big batch jobs on other machines that read data from s3, save the learned parameter files to s3, and then read (and periodically refresh) the models from s3 in the ingest pipeline. All of the data that flows out of the system can be
5 0.9390837 903 high scalability-2010-09-17-Hot Scalability Links For Sep 17, 2010
Introduction: Disqus - Scaling the Worlds Largest Django App. Interesting overview of a commenting system with 75 million comments and 250 million visitors. Lots of good details on how they partition their database, testing, continuous integration, feature switches, caching, delayed signals, and more. Things I learnt tracking a billion events in 24 hours : Know your host, Scaling isn't just servers, My servers need to talk to me more, Kill switches for users, What you don't know is the problem, Don't mix server roles, Know your most important users outside of your site. Tweets of Gold: georgebarnett : I read High Scalability for useful articles about large scaling. Sadly though, nothing useful ever shows up. #NoLongerBothering northscale : wow that is fast! :) RT @cgoldberg: was just running > 100k ops/sec against my 2-node #Membase cluster... zazooom #nosql turbofunctor : The root of many (horizontal) scalability problems is an application level access to a writab
6 0.93880212 888 high scalability-2010-08-27-OpenStack - The Answer to: How do We Compete with Amazon?
7 0.93807983 853 high scalability-2010-07-08-Cloud AWS Infrastructure vs. Physical Infrastructure
8 0.93801212 1216 high scalability-2012-03-27-Big Data In the Cloud Using Cloudify
10 0.93569225 389 high scalability-2008-09-23-How to Scale with Ruby on Rails
11 0.93554002 1557 high scalability-2013-12-02-Evolution of Bazaarvoice’s Architecture to 500M Unique Users Per Month
12 0.93552762 1189 high scalability-2012-02-07-Hypertable Routs HBase in Performance Test -- HBase Overwhelmed by Garbage Collection
13 0.93537658 450 high scalability-2008-11-24-Scalability Perspectives #3: Marc Andreessen – Internet Platforms
14 0.93452376 304 high scalability-2008-04-19-How to build a real-time analytics system?
15 0.93409503 1040 high scalability-2011-05-13-Stuff The Internet Says On Scalability For May 13, 2011
16 0.93355048 576 high scalability-2009-04-21-What CDN would you recommend?
17 0.93247479 1093 high scalability-2011-08-05-Stuff The Internet Says On Scalability For August 5, 2011
18 0.93224156 1137 high scalability-2011-11-04-Stuff The Internet Says On Scalability For November 4, 2011
19 0.93204647 458 high scalability-2008-12-01-Web Consolidation on the Sun Fire T1000 using Solaris Containers
20 0.93178713 1180 high scalability-2012-01-24-The State of NoSQL in 2012