high_scalability high_scalability-2008 high_scalability-2008-425 knowledge-graph by maker-knowledge-mining
Source: html
Introduction: At eBay, one of the primary architectural forces we contend with every day is scalability. It colors and drives every architectural and design decision we make. With hundreds of millions of users worldwide, over two billion page views a day, and petabytes of data in our systems, this is not a choice - it is a necessity. In a scalable architecture, resource usage should increase linearly (or better) with load, where load may be measured in user traffic, data volume, etc. Where performance is about the resource usage associated with a single unit of work, scalability is about how resource usage changes as units of work grow in number or size. Said another way, scalability is the shape of the price-performance curve, as opposed to its value at one point in that curve. There are many facets to scalability - transactional, operational, development effort. In this article, I will outline several of the key best practices we have learned over time to scale the transactional th
sentIndex sentText sentNum sentScore
1 At eBay, one of the primary architectural forces we contend with every day is scalability. [sent-1, score-0.803]
2 It colors and drives every architectural and design decision we make. [sent-2, score-0.655]
3 With hundreds of millions of users worldwide, over two billion page views a day, and petabytes of data in our systems, this is not a choice - it is a necessity. [sent-3, score-0.457]
4 In a scalable architecture, resource usage should increase linearly (or better) with load, where load may be measured in user traffic, data volume, etc. [sent-4, score-0.956]
5 Where performance is about the resource usage associated with a single unit of work, scalability is about how resource usage changes as units of work grow in number or size. [sent-5, score-1.519]
6 Said another way, scalability is the shape of the price-performance curve, as opposed to its value at one point in that curve. [sent-6, score-0.496]
7 There are many facets to scalability - transactional, operational, development effort. [sent-7, score-0.335]
8 In this article, I will outline several of the key best practices we have learned over time to scale the transactional throughput of a web-based system. [sent-8, score-0.864]
9 Most of these best practices will be familiar to you. [sent-9, score-0.422]
10 All come from the collective experience of the people who develop and operate the eBay site. [sent-11, score-0.372]
wordName wordTfidf (topN-words)
[('ebay', 0.244), ('usage', 0.23), ('facets', 0.22), ('architectural', 0.217), ('transactional', 0.217), ('resource', 0.212), ('practices', 0.21), ('infoq', 0.19), ('opposed', 0.177), ('contend', 0.177), ('outline', 0.177), ('collective', 0.174), ('measured', 0.147), ('colors', 0.142), ('units', 0.139), ('linearly', 0.138), ('curve', 0.136), ('article', 0.135), ('worldwide', 0.131), ('shape', 0.128), ('forces', 0.128), ('petabytes', 0.125), ('associated', 0.116), ('unit', 0.116), ('scalability', 0.115), ('operate', 0.113), ('familiar', 0.112), ('day', 0.11), ('drives', 0.109), ('volume', 0.108), ('decision', 0.107), ('operational', 0.102), ('best', 0.1), ('views', 0.098), ('said', 0.094), ('primary', 0.091), ('develop', 0.085), ('learned', 0.084), ('choice', 0.083), ('rest', 0.083), ('grow', 0.082), ('may', 0.081), ('hundreds', 0.08), ('every', 0.08), ('load', 0.08), ('throughput', 0.076), ('value', 0.076), ('millions', 0.071), ('increase', 0.068), ('work', 0.067)]
simIndex simValue blogId blogTitle
same-blog 1 0.99999994 425 high scalability-2008-10-22-Scalability Best Practices: Lessons from eBay
Introduction: At eBay, one of the primary architectural forces we contend with every day is scalability. It colors and drives every architectural and design decision we make. With hundreds of millions of users worldwide, over two billion page views a day, and petabytes of data in our systems, this is not a choice - it is a necessity. In a scalable architecture, resource usage should increase linearly (or better) with load, where load may be measured in user traffic, data volume, etc. Where performance is about the resource usage associated with a single unit of work, scalability is about how resource usage changes as units of work grow in number or size. Said another way, scalability is the shape of the price-performance curve, as opposed to its value at one point in that curve. There are many facets to scalability - transactional, operational, development effort. In this article, I will outline several of the key best practices we have learned over time to scale the transactional th
2 0.20616128 331 high scalability-2008-05-27-eBay Architecture
Introduction: Update 2: EBay's Randy Shoup spills the secrets of how to service hundreds of millions of users and over two billion page views a day in Scalability Best Practices: Lessons from eBay on InfoQ. The practices: Partition by Function, Split Horizontally, Avoid Distributed Transactions, Decouple Functions Asynchronously, Move Processing To Asynchronous Flows, Virtualize At All Levels, Cache Appropriately. Update: eBay Serves 5 Billion API Calls Each Month . Aren't we seeing more and more traffic driven by mashups composed on top of open APIs? APIs are no longer a bolt on, they are your application. Architecturally that argues for implementing your own application around the same APIs developers and users employ. Who hasn't wondered how eBay does their business? As one of the largest most loaded websites in the world, it can't be easy. And the subtitle of the presentation hints at how creating such a monster system requires true engineering: Striking a balance between site stabilit
3 0.11818583 1413 high scalability-2013-02-27-42 Monster Problems that Attack as Loads Increase
Introduction: For solutions take a look at: 7 Life Saving Scalability Defenses Against Load Monster Attacks . This is a look at all the bad things that can happen to your carefully crafted program as loads increase: all hell breaks lose. Sure, you can scale out or scale up, but you can also choose to program better. Make your system handle larger loads. This saves money because fewer boxes are needed and it will make the entire application more reliable and have better response times. And it can be quite satisfying as a programmer. Large Number Of Objects We usually get into scaling problems when the number of objects gets larger. Clearly resource usage of all types is stressed as the number of objects grow. Continuous Failures Makes An Infinite Event Stream During large network failure scenarios there is never time for the system recover. We are in a continual state of stress. Lots of High Priority Work For example, rerouting is a high priority activity. If there is a large amount
4 0.11281206 1123 high scalability-2011-09-23-The Real News is Not that Facebook Serves Up 1 Trillion Pages a Month…
Introduction: There’s some amount of debate whether Facebook really crossed over the one trillion page view per month threshold. While one report says it did, another respected firm says it did not ; that its monthly page views are a mere 467 billion per month. In the big scheme of things, the discrepancy is somewhat irrelevant, as neither show the true load on Facebook’s infrastructure – which is far more impressive a set of numbers than its externally measured “page view” metric. Mashable reported in “ Facebook Surpasses 1 Trillion Pageviews per Month ” that the social networking giant saw “approximately 870 million unique visitors in June and 860 million in July” and followed up with some per visitor statistics, indicating “each visitor averaged approximately 1,160 page views in July and 40 per visit — enormous by any standard. Time spent on the site was around 25 minutes per user.” From an architectural standpoint it’s not just about the page views. It’s about requests and responses,
5 0.11206616 742 high scalability-2009-11-17-10 eBay Secrets for Planet Wide Scaling
Introduction: You don't even have to make a bid, Randy Shoup, an eBay Distinguished Architect, gives this presentation on how eBay scales, for free. Randy has done a fabulous job in this presentation and in other talks listed at the end of this post getting at the heart of the principles behind scalability. It's more about ideas of how things work and fit together than a focusing on a particular technology stack. Impressive Stats In case you weren't sure, eBay is big, with lots of: users, data, features, and change... Over 89 million active users worldwide 190 million items for sale in 50,000 categories Over 8 billion URL requests per day Hundreds of new features per quarter Roughly 10% of items are listed or ended every day In 39 countries and 10 languages 24x7x365 70 billion read / write operations / day Processes 50TB of new, incremental data per day Analyzes 50PB of data per day 10 Lessons The presentation does a good job explaining each lesson, but the list is.
6 0.10380318 906 high scalability-2010-09-22-Applying Scalability Patterns to Infrastructure Architecture
9 0.10194401 273 high scalability-2008-03-09-Best Practices for Speeding Up Your Web Site
10 0.1018073 758 high scalability-2010-01-11-Have We Reached the End of Scaling?
11 0.096179843 761 high scalability-2010-01-17-Applications Become Black Boxes Using Markets to Scale and Control Costs
12 0.095714092 1158 high scalability-2011-12-16-Stuff The Internet Says On Scalability For December 16, 2011
13 0.091482565 1235 high scalability-2012-04-27-Stuff The Internet Says On Scalability For April 27, 2012
14 0.088665687 472 high scalability-2008-12-19-How to measure memory required for a user session
15 0.086369224 96 high scalability-2007-09-18-Amazon Architecture
16 0.085922845 1206 high scalability-2012-03-09-Stuff The Internet Says On Scalability For March 9, 2012
17 0.085681319 881 high scalability-2010-08-16-Scaling an AWS infrastructure - Tools and Patterns
18 0.084467873 1421 high scalability-2013-03-11-Low Level Scalability Solutions - The Conditioning Collection
19 0.083983354 550 high scalability-2009-03-30-Ebay history and architecture
20 0.083631054 1499 high scalability-2013-08-09-Stuff The Internet Says On Scalability For August 9, 2013
topicId topicWeight
[(0, 0.157), (1, 0.065), (2, 0.007), (3, -0.013), (4, 0.022), (5, -0.024), (6, -0.004), (7, 0.012), (8, -0.016), (9, 0.044), (10, -0.036), (11, 0.016), (12, -0.016), (13, 0.038), (14, -0.019), (15, -0.008), (16, 0.044), (17, -0.041), (18, 0.015), (19, 0.047), (20, 0.008), (21, 0.036), (22, 0.0), (23, -0.055), (24, -0.015), (25, -0.089), (26, -0.073), (27, 0.024), (28, 0.052), (29, 0.019), (30, 0.053), (31, 0.057), (32, 0.038), (33, 0.02), (34, -0.025), (35, -0.009), (36, 0.011), (37, 0.012), (38, -0.034), (39, -0.028), (40, 0.009), (41, -0.007), (42, -0.008), (43, 0.019), (44, 0.022), (45, 0.052), (46, 0.01), (47, 0.05), (48, -0.022), (49, 0.025)]
simIndex simValue blogId blogTitle
same-blog 1 0.96847516 425 high scalability-2008-10-22-Scalability Best Practices: Lessons from eBay
Introduction: At eBay, one of the primary architectural forces we contend with every day is scalability. It colors and drives every architectural and design decision we make. With hundreds of millions of users worldwide, over two billion page views a day, and petabytes of data in our systems, this is not a choice - it is a necessity. In a scalable architecture, resource usage should increase linearly (or better) with load, where load may be measured in user traffic, data volume, etc. Where performance is about the resource usage associated with a single unit of work, scalability is about how resource usage changes as units of work grow in number or size. Said another way, scalability is the shape of the price-performance curve, as opposed to its value at one point in that curve. There are many facets to scalability - transactional, operational, development effort. In this article, I will outline several of the key best practices we have learned over time to scale the transactional th
2 0.71774668 381 high scalability-2008-09-08-Guerrilla Capacity Planning and the Law of Universal Scalability
Introduction: In the era of Web 2.0 traditional approaches to capacity planning are often difficult to implement. Guerrilla Capacity Planning facilitates rapid forecasting of capacity requirements based on the opportunistic use of whatever performance data and tools are available. One unique Guerrilla tool is Virtual Load Testing, based on Dr. Gunther's "Universal Law of Computational Scaling", which provides a highly cost-effective method for assessing application scalability. Neil Gunther, M.Sc., Ph.D. is an internationally recognized computer system performance consultant who founded Performance Dynamics Company in 1994. Some reasons why you should understand this law: 1. A lot of people use the term "scalability" without clearly defining it, let alone defining it quantitatively. Computer system scalability must be quantified. If you can't quantify it, you can't guarantee it. The universal law of computational scaling provides that quantification. 2. One the greatest impediments to applying
3 0.70657307 404 high scalability-2008-10-06-Scalability for Startups: How to Grow Up without Blowing Up
Introduction: This is a useful post by Frank Mashraqi, Director of Business Operations & Technical Strategy for a top 50 website that delivers billions of page views per month. Since scalability is considered a non-functional requirement, it is often overlooked in the hopes of decreasing time to market. Adding scalability down the road can decrease the time to market but only after assuming significant technical debt. Balancing performance and scalability vs. fast iteration and cost efficiency can be a significant challenge for startups. The good news is that achieving this balance is not impossible. Read the rest of the article here and view a presentation here .
4 0.7040922 331 high scalability-2008-05-27-eBay Architecture
Introduction: Update 2: EBay's Randy Shoup spills the secrets of how to service hundreds of millions of users and over two billion page views a day in Scalability Best Practices: Lessons from eBay on InfoQ. The practices: Partition by Function, Split Horizontally, Avoid Distributed Transactions, Decouple Functions Asynchronously, Move Processing To Asynchronous Flows, Virtualize At All Levels, Cache Appropriately. Update: eBay Serves 5 Billion API Calls Each Month . Aren't we seeing more and more traffic driven by mashups composed on top of open APIs? APIs are no longer a bolt on, they are your application. Architecturally that argues for implementing your own application around the same APIs developers and users employ. Who hasn't wondered how eBay does their business? As one of the largest most loaded websites in the world, it can't be easy. And the subtitle of the presentation hints at how creating such a monster system requires true engineering: Striking a balance between site stabilit
5 0.6952396 903 high scalability-2010-09-17-Hot Scalability Links For Sep 17, 2010
Introduction: Disqus - Scaling the Worlds Largest Django App. Interesting overview of a commenting system with 75 million comments and 250 million visitors. Lots of good details on how they partition their database, testing, continuous integration, feature switches, caching, delayed signals, and more. Things I learnt tracking a billion events in 24 hours : Know your host, Scaling isn't just servers, My servers need to talk to me more, Kill switches for users, What you don't know is the problem, Don't mix server roles, Know your most important users outside of your site. Tweets of Gold: georgebarnett : I read High Scalability for useful articles about large scaling. Sadly though, nothing useful ever shows up. #NoLongerBothering northscale : wow that is fast! :) RT @cgoldberg: was just running > 100k ops/sec against my 2-node #Membase cluster... zazooom #nosql turbofunctor : The root of many (horizontal) scalability problems is an application level access to a writab
6 0.67013431 1047 high scalability-2011-05-25-Stuff to Watch from Surge 2010
7 0.65500998 742 high scalability-2009-11-17-10 eBay Secrets for Planet Wide Scaling
8 0.65417981 757 high scalability-2010-01-04-11 Strategies to Rock Your Startup’s Scalability in 2010
9 0.64389181 1123 high scalability-2011-09-23-The Real News is Not that Facebook Serves Up 1 Trillion Pages a Month…
10 0.64262468 600 high scalability-2009-05-15-Wolfram|Alpha Architecture
11 0.64084458 344 high scalability-2008-06-09-FaceStat's Rousing Tale of Scaling Woe and Wisdom Won
12 0.63678157 1596 high scalability-2014-02-14-Stuff The Internet Says On Scalability For February 14th, 2014
13 0.63587743 1411 high scalability-2013-02-22-Stuff The Internet Says On Scalability For February 22, 2013
14 0.63476068 1250 high scalability-2012-05-23-Averages, web performance data, and how your analytics product is lying to you
15 0.63348812 604 high scalability-2009-05-20-Paper: Flux: An Adaptive Partitioning Operator for Continuous Query Systems
16 0.63067228 1361 high scalability-2012-11-22-Gone Fishin': PlentyOfFish Architecture
17 0.62882 638 high scalability-2009-06-26-PlentyOfFish Architecture
18 0.62634659 1534 high scalability-2013-10-18-Stuff The Internet Says On Scalability For October 18th, 2013
19 0.62346542 906 high scalability-2010-09-22-Applying Scalability Patterns to Infrastructure Architecture
20 0.62275243 1330 high scalability-2012-09-28-Stuff The Internet Says On Scalability For September 28, 2012
topicId topicWeight
[(1, 0.192), (2, 0.166), (10, 0.083), (26, 0.21), (61, 0.075), (79, 0.118), (85, 0.051)]
simIndex simValue blogId blogTitle
Introduction: Why You Need To Attend THIS CONFERENCE • Understand the multi-dimensional view of business-technology alignment • A sense of urgency for aggressively pursuing Enterprise Architecture • A "language" (ie., a Framework) for improving enterprise communications about architecture issues • An understanding of the cultural changes implied by process evolution. How to effectively use the framework to anchor processes and procedures for delivering service and support for applications • An understanding of basic Enterprise physics • Recommendations for the Sr. Managers to understand the political realities and organizational resistance in realizing EA vision and some excellent advices for overcoming these barriers • Number of practical examples of how to work with people who affect decisions on EA implementation • How to create value for your organization by systematically recording assets, processes, connectivity, people, timing and motivation, through a simple framework
2 0.93687242 751 high scalability-2009-12-16-The most common flaw in software performance testing
Introduction: How many times have we all run across a situation where the performance tests on a piece of software pass with flying colors on the test systems only to see the software exhibit poor performance characteristics when the software is deployed in production? Read More Here...
3 0.9353103 715 high scalability-2009-10-06-10 Ways to Take your Site from One to One Million Users by Kevin Rose
Introduction: At the Future of Web Apps conference Kevin Rose (Digg, Pownce, Wefollow) gave a cool presentation on the top 10 down and dirty ways you can grow your web app . He took the questions he's most often asked and turned it into a very informative talk. This isn't the typical kind of scalability we cover on this site. There aren't any infrastructure and operations tips. But the reason we care about scalability is to support users and Kevin has a lot of good techniques to help your user base bloom. Here's a summary of the 10 ways to grow your consumer web application: 1. Ego. Ask does this feature increase the users self-worth or stroke the ego? What emotional and visible awards will a user receive for contributing to your site? Are they gaining reputation, badges, show case what they've done in the community? Sites that have done it well: Twitter.com followers . Followers turns every single celebrity as spokesperson for your service. Celebrities continually pimp your service i
4 0.9280358 697 high scalability-2009-09-09-GridwiseTech revolutionizes data management
Introduction: GridwiseTech has developed AdHoc , an advanced framework for sharing geographically distributed data and compute resources. It simplifies the resource management and makes cooperation secure and effective. The premise of AdHoc is to enable each member of the associated institution to control access to his or her resources without an IT administrator’s help, and with high security level of any exposed data or applications assured. It takes 3 easy steps to establish cooperation within AdHoc: create a virtual organization, add resources and share them. The application can be implemented within any organization to exchange data and resources or between institutions to join forces for more efficient results. AdHoc was initially created for a consortium of hospitals and institutions to share medical data sets. As a technical partner in that project, GridwiseTech implemented the Security Framework to provide access to that data and designed a graphical tool to facilitate the administration
5 0.92549717 1356 high scalability-2012-11-07-Gone Fishin': 10 Ways to Take your Site from One to One Million Users by Kevin Rose
Introduction: This is the post that got me kicked off my original shared hosting service and prompted the move to SquareSpace. I couldn't figure out why so many people were reading this article. But they kept on coming. The site went down and I was told to vamoose. It finally dawned on me nobody actually cared about the article, it was the name Kevin Rose that was magic. Learned a good lesson about publishing biz... At the Future of Web Apps conference Kevin Rose (Digg, Pownce, Wefollow) gave a cool presentation on the top 10 down and dirty ways you can grow your web app . He took the questions he's most often asked and turned it into a very informative talk. This isn't the typical kind of scalability we cover on this site. There aren't any infrastructure and operations tips. But the reason we care about scalability is to support users and Kevin has a lot of good techniques to help your user base bloom. Here's a summary of the 10 ways to grow your consumer web application: 1. Ego.
6 0.92485851 1410 high scalability-2013-02-20-Smart Companies Fail Because they Do Everything Right - Staying Alive to Scale
7 0.92450917 73 high scalability-2007-08-23-Postgresql on high availability websites?
8 0.90249521 1650 high scalability-2014-05-19-A Short On How the Wayback Machine Stores More Pages than Stars in the Milky Way
same-blog 9 0.87974542 425 high scalability-2008-10-22-Scalability Best Practices: Lessons from eBay
10 0.87236047 381 high scalability-2008-09-08-Guerrilla Capacity Planning and the Law of Universal Scalability
11 0.86885202 339 high scalability-2008-06-04-LinkedIn Architecture
12 0.84798402 148 high scalability-2007-11-11-Linkedin architecture
13 0.84617609 1570 high scalability-2014-01-01-Paper: Nanocubes: Nanocubes for Real-Time Exploration of Spatiotemporal Datasets
14 0.83520484 635 high scalability-2009-06-22-Improving performance and scalability with DDD
15 0.83054239 92 high scalability-2007-09-15-The Role of Memory within Web 2.0 Architectures and Deployments
16 0.82947439 340 high scalability-2008-06-06-Economies of Non-Scale
17 0.81065154 1044 high scalability-2011-05-19-Zynga's Z Cloud - Scale Fast or Fail Fast by Merging Private and Public Clouds
18 0.80978614 1235 high scalability-2012-04-27-Stuff The Internet Says On Scalability For April 27, 2012
19 0.80863088 442 high scalability-2008-11-13-Plenty of Fish Says Scaling for Free Doesn't Pay
20 0.8053658 853 high scalability-2010-07-08-Cloud AWS Infrastructure vs. Physical Infrastructure