high_scalability high_scalability-2009 high_scalability-2009-742 knowledge-graph by maker-knowledge-mining
Source: html
Introduction: You don't even have to make a bid, Randy Shoup, an eBay Distinguished Architect, gives this presentation on how eBay scales, for free. Randy has done a fabulous job in this presentation and in other talks listed at the end of this post getting at the heart of the principles behind scalability. It's more about ideas of how things work and fit together than a focusing on a particular technology stack. Impressive Stats In case you weren't sure, eBay is big, with lots of: users, data, features, and change... Over 89 million active users worldwide 190 million items for sale in 50,000 categories Over 8 billion URL requests per day Hundreds of new features per quarter Roughly 10% of items are listed or ended every day In 39 countries and 10 languages 24x7x365 70 billion read / write operations / day Processes 50TB of new, incremental data per day Analyzes 50PB of data per day 10 Lessons The presentation does a good job explaining each lesson, but the list is.
sentIndex sentText sentNum sentScore
1 You don't even have to make a bid, Randy Shoup, an eBay Distinguished Architect, gives this presentation on how eBay scales, for free. [sent-1, score-0.12]
2 Randy has done a fabulous job in this presentation and in other talks listed at the end of this post getting at the heart of the principles behind scalability. [sent-2, score-0.645]
3 It's more about ideas of how things work and fit together than a focusing on a particular technology stack. [sent-3, score-0.09]
4 Partition Everything - if you can't split it, you can't scale it. [sent-10, score-0.125]
5 Split everything into manageable chunks by function and data. [sent-11, score-0.293]
6 Asynchrony Everywhere - connect independent components through event queues Automate Everything - components should automatically adjust and the system should learn and improve itself. [sent-12, score-0.373]
7 Embrace Inconsistency - pick for each feature where you need to be on the CAP continuum, no distributed transactions, inconsistency can be minimized by careful operation ordering, become eventually consistent through async recovery and reconciliation. [sent-14, score-0.374]
8 Expect (R)evolution - change is constant, design for extensibility, incrementally deploy changes. [sent-15, score-0.084]
9 Dependencies Matter - minimize and control dependencies, use abstract interfaces and virtualization, components have an SLA, consumers responsible for recovering from SLA violations. [sent-16, score-0.477]
10 Be Authoritative - Know which data is authoritative, which data isn't, and treat it accordingly. [sent-17, score-0.094]
11 Custom Infrastructure - maximize the utilization of every resource. [sent-19, score-0.16]
wordName wordTfidf (topN-words)
[('randy', 0.474), ('ebay', 0.29), ('listed', 0.198), ('sla', 0.189), ('ebayby', 0.146), ('continuum', 0.137), ('shoup', 0.137), ('components', 0.134), ('split', 0.125), ('authoritative', 0.122), ('presentation', 0.12), ('items', 0.119), ('extensibility', 0.116), ('recovering', 0.116), ('fabulous', 0.113), ('manageable', 0.111), ('predictions', 0.109), ('bid', 0.107), ('minimized', 0.107), ('adjust', 0.105), ('sale', 0.103), ('inconsistency', 0.103), ('distinguished', 0.102), ('per', 0.099), ('everything', 0.099), ('treat', 0.094), ('countries', 0.093), ('focusing', 0.09), ('consumers', 0.089), ('maximize', 0.088), ('dependencies', 0.086), ('ordering', 0.086), ('incrementally', 0.084), ('chunks', 0.083), ('async', 0.083), ('explaining', 0.082), ('ended', 0.081), ('careful', 0.081), ('billion', 0.079), ('lesson', 0.078), ('heart', 0.075), ('incremental', 0.075), ('recommendations', 0.074), ('cap', 0.072), ('utilization', 0.072), ('abstract', 0.071), ('job', 0.07), ('principles', 0.069), ('interfaces', 0.067), ('url', 0.067)]
simIndex simValue blogId blogTitle
same-blog 1 0.99999994 742 high scalability-2009-11-17-10 eBay Secrets for Planet Wide Scaling
Introduction: You don't even have to make a bid, Randy Shoup, an eBay Distinguished Architect, gives this presentation on how eBay scales, for free. Randy has done a fabulous job in this presentation and in other talks listed at the end of this post getting at the heart of the principles behind scalability. It's more about ideas of how things work and fit together than a focusing on a particular technology stack. Impressive Stats In case you weren't sure, eBay is big, with lots of: users, data, features, and change... Over 89 million active users worldwide 190 million items for sale in 50,000 categories Over 8 billion URL requests per day Hundreds of new features per quarter Roughly 10% of items are listed or ended every day In 39 countries and 10 languages 24x7x365 70 billion read / write operations / day Processes 50TB of new, incremental data per day Analyzes 50PB of data per day 10 Lessons The presentation does a good job explaining each lesson, but the list is.
2 0.27293539 614 high scalability-2009-06-01-Guess How Many Users it Takes to Kill Your Site?
Introduction: Update: Here's the first result . Good response time until 400 users. At 1,340 users the response time was 6 seconds. And at 2000 users the site was effectively did. An interesting point was that errors that could harm a site's reputation started at 1000 users. Cheers to the company that had the guts to give this a try. That which doesn't kill your site makes it stronger. Or at least that's the capacity planning strategy John Allspaw recommends (not really, but I'm trying to make a point here) in The Art of Capacity Planning : Using production traffic to define your resources ceilings in a controlled setting allows you to see firsthand what would happen when you run out of capacity in a particular resource. Of course I'm not suggesting that you run your site into the ground, but better to know what your real (not simulated) loads are while you're watching, than find out the hard way. In addition, a lot of unexpected systemic things can happen when load increases in a particular
3 0.24436542 331 high scalability-2008-05-27-eBay Architecture
Introduction: Update 2: EBay's Randy Shoup spills the secrets of how to service hundreds of millions of users and over two billion page views a day in Scalability Best Practices: Lessons from eBay on InfoQ. The practices: Partition by Function, Split Horizontally, Avoid Distributed Transactions, Decouple Functions Asynchronously, Move Processing To Asynchronous Flows, Virtualize At All Levels, Cache Appropriately. Update: eBay Serves 5 Billion API Calls Each Month . Aren't we seeing more and more traffic driven by mashups composed on top of open APIs? APIs are no longer a bolt on, they are your application. Architecturally that argues for implementing your own application around the same APIs developers and users employ. Who hasn't wondered how eBay does their business? As one of the largest most loaded websites in the world, it can't be easy. And the subtitle of the presentation hints at how creating such a monster system requires true engineering: Striking a balance between site stabilit
4 0.11801325 726 high scalability-2009-10-22-Paper: The Case for RAMClouds: Scalable High-Performance Storage Entirely in DRAM
Introduction: Stanford Info Lab is taking pains to document a direction we've been moving for a while now, using RAM not just as a cache, but as the primary storage medium. Many quality products have built on this model. Even if the vision isn't radical, the paper does produce a lot of data backing up the transition, which is in itself helpful. From the The Abstract: Disk-oriented approaches to online storage are becoming increasingly problematic: they do not scale grace-fully to meet the needs of large-scale Web applications, and improvements in disk capacity have far out-stripped improvements in access latency and bandwidth. This paper argues for a new approach to datacenter storage called RAMCloud, where information is kept entirely in DRAM and large-scale systems are created by aggregating the main memories of thousands of commodity servers. We believe that RAMClouds can provide durable and available storage with 100-1000x the throughput of disk-based systems and 100-1000x lower access lat
5 0.11243584 550 high scalability-2009-03-30-Ebay history and architecture
Introduction: Ebay [1] Starts in 1995, initial name AuctionWeb (V1) : - very simple architecture - based on perl - no database, for data persistence they used plain files Because of rapid growth they needed to improve their architecture and so V2 (clever name) was born: - replaced perl with C/C++ - started using a database in a master-slave configuration - C++ back-end - XSLT front-end Any request will lead to an XML file being created in C++ and the XLST processor will transform that into html. *pretty sophisticated architecture for the 90s, XLST was cutting-edge back then* That hold out pretty well for a while but in the late 90s ebay experienced an exponential growth. They started having some trouble with outages and needed improvements, so V3 was developed: - based on java - search engine still used C++ - proof that relational databases can scale (aggressive caching) - developed a messaging layer for making a lot of asyncronious calls, they a
6 0.11206616 425 high scalability-2008-10-22-Scalability Best Practices: Lessons from eBay
7 0.094346821 720 high scalability-2009-10-12-High Performance at Massive Scale – Lessons learned at Facebook
8 0.093372926 1137 high scalability-2011-11-04-Stuff The Internet Says On Scalability For November 4, 2011
9 0.09198118 870 high scalability-2010-08-02-7 Scaling Strategies Facebook Used to Grow to 500 Million Users
10 0.085817106 94 high scalability-2007-09-17-Blog: Adding Simplicity by Dan Pritchett
11 0.084455788 828 high scalability-2010-05-17-7 Lessons Learned While Building Reddit to 270 Million Page Views a Month
12 0.084327668 1240 high scalability-2012-05-07-Startups are Creating a New System of the World for IT
13 0.083778687 589 high scalability-2009-05-05-Drop ACID and Think About Data
14 0.083691664 313 high scalability-2008-05-02-Friends for Sale Architecture - A 300 Million Page View-Month Facebook RoR App
15 0.082053177 920 high scalability-2010-10-15-Troubles with Sharding - What can we learn from the Foursquare Incident?
16 0.080264755 750 high scalability-2009-12-16-Building Super Scalable Systems: Blade Runner Meets Autonomic Computing in the Ambient Cloud
17 0.080254205 511 high scalability-2009-02-12-MySpace Architecture
19 0.079577878 241 high scalability-2008-02-05-SLA monitoring
20 0.07878539 1484 high scalability-2013-06-28-Stuff The Internet Says On Scalability For June 28, 2013
topicId topicWeight
[(0, 0.143), (1, 0.057), (2, -0.005), (3, 0.004), (4, 0.017), (5, -0.004), (6, -0.013), (7, 0.013), (8, 0.014), (9, 0.001), (10, -0.001), (11, 0.035), (12, -0.023), (13, 0.022), (14, -0.002), (15, 0.052), (16, 0.013), (17, -0.021), (18, -0.0), (19, 0.023), (20, 0.031), (21, 0.009), (22, -0.008), (23, -0.018), (24, -0.051), (25, -0.06), (26, -0.046), (27, 0.03), (28, 0.013), (29, 0.013), (30, 0.013), (31, 0.017), (32, 0.02), (33, -0.014), (34, 0.004), (35, 0.011), (36, -0.002), (37, 0.023), (38, 0.015), (39, 0.021), (40, -0.01), (41, -0.042), (42, -0.022), (43, 0.028), (44, -0.019), (45, 0.011), (46, 0.038), (47, -0.024), (48, -0.052), (49, 0.062)]
simIndex simValue blogId blogTitle
same-blog 1 0.92524409 742 high scalability-2009-11-17-10 eBay Secrets for Planet Wide Scaling
Introduction: You don't even have to make a bid, Randy Shoup, an eBay Distinguished Architect, gives this presentation on how eBay scales, for free. Randy has done a fabulous job in this presentation and in other talks listed at the end of this post getting at the heart of the principles behind scalability. It's more about ideas of how things work and fit together than a focusing on a particular technology stack. Impressive Stats In case you weren't sure, eBay is big, with lots of: users, data, features, and change... Over 89 million active users worldwide 190 million items for sale in 50,000 categories Over 8 billion URL requests per day Hundreds of new features per quarter Roughly 10% of items are listed or ended every day In 39 countries and 10 languages 24x7x365 70 billion read / write operations / day Processes 50TB of new, incremental data per day Analyzes 50PB of data per day 10 Lessons The presentation does a good job explaining each lesson, but the list is.
2 0.77992624 331 high scalability-2008-05-27-eBay Architecture
Introduction: Update 2: EBay's Randy Shoup spills the secrets of how to service hundreds of millions of users and over two billion page views a day in Scalability Best Practices: Lessons from eBay on InfoQ. The practices: Partition by Function, Split Horizontally, Avoid Distributed Transactions, Decouple Functions Asynchronously, Move Processing To Asynchronous Flows, Virtualize At All Levels, Cache Appropriately. Update: eBay Serves 5 Billion API Calls Each Month . Aren't we seeing more and more traffic driven by mashups composed on top of open APIs? APIs are no longer a bolt on, they are your application. Architecturally that argues for implementing your own application around the same APIs developers and users employ. Who hasn't wondered how eBay does their business? As one of the largest most loaded websites in the world, it can't be easy. And the subtitle of the presentation hints at how creating such a monster system requires true engineering: Striking a balance between site stabilit
3 0.73275995 1178 high scalability-2012-01-20-Stuff The Internet Says On Scalability For January 20, 2012
Introduction: If you’ve got the time, we’ve got the HighScalability: Google+: 90 million users ; Internet 2011 : 2.1 billion Internet users, 1 trillion YouTube views, 5.9 billion mobile subscriptions; Fusion-io: One Billion IOPS ; 12 atoms: size of IBM's new memory bit ; 32 Million: Stack monthly visitors ; Gmail: 350 Million Users ; TimTebow: 1.5 million Tweets Quotable Quotes: Similarity : There is no canonical schema anymore. Instead you should ask: What high-volume queries will I need to serve with my data? Then work backwards from there. @kvirjee : Dis/Agree? -- "there is no problem but scalability, and architecture is its solution" @robpegoraro : Eternal vigilance can be crowdsourced. Didn't Bill Gates say once that 48 bits would always be enough for an ID? Well, Oracle ran out of bits: Fundamental Oracle flaw revealed . 64 bits, that's the ticket, ipv6 went 128 bits. The day Kodak died : We developed the world's first consumer digital camera bu
4 0.72854298 1219 high scalability-2012-03-30-Stuff The Internet Says On Scalability For March 30, 2012
Introduction: Choosy Mothers Choose HighScalability: Quotable quotes: @itarradellas : "Revolutions in science have often been preceded by revolutions in measurement" @jasongorman : Use dependency injection, not Spring. Use event-driven, asynchronous I/O, not Node.js. Use MVC, not http://ASP.NET MVC etc etc @bernardgolden : #netflix uses most aggressive #aws reservation system. Gets pricing down to ~ 33% of "list' pricing. @ikarzali : Hey, for all facebook's talk at scalability conferences, I have to say Timeline is super slow(!) Howz that memcache workin out for you now? Yahoo! : Amazon's Game-Changing Cloud Was Built By Some Guys In South Africa Foursquare : 1.5 billion check-ins from 15 million people at 30 million different places. How OMGPOP scaled to 36 million users in three weeks . Draw Something has been downloaded 35+ million times; 1 billion pictures created at 3,000 pictures per second; Couchbase is used as the database; SoftLayer is thei
5 0.71661282 425 high scalability-2008-10-22-Scalability Best Practices: Lessons from eBay
Introduction: At eBay, one of the primary architectural forces we contend with every day is scalability. It colors and drives every architectural and design decision we make. With hundreds of millions of users worldwide, over two billion page views a day, and petabytes of data in our systems, this is not a choice - it is a necessity. In a scalable architecture, resource usage should increase linearly (or better) with load, where load may be measured in user traffic, data volume, etc. Where performance is about the resource usage associated with a single unit of work, scalability is about how resource usage changes as units of work grow in number or size. Said another way, scalability is the shape of the price-performance curve, as opposed to its value at one point in that curve. There are many facets to scalability - transactional, operational, development effort. In this article, I will outline several of the key best practices we have learned over time to scale the transactional th
6 0.71146119 903 high scalability-2010-09-17-Hot Scalability Links For Sep 17, 2010
7 0.7006411 1484 high scalability-2013-06-28-Stuff The Internet Says On Scalability For June 28, 2013
8 0.69667196 992 high scalability-2011-02-18-Stuff The Internet Says On Scalability For February 18, 2011
9 0.692541 511 high scalability-2009-02-12-MySpace Architecture
10 0.69156921 1368 high scalability-2012-12-07-Stuff The Internet Says On Scalability For December 7, 2012
12 0.68942219 1649 high scalability-2014-05-16-Stuff The Internet Says On Scalability For May 16th, 2014
13 0.68655002 1174 high scalability-2012-01-13-Stuff The Internet Says On Scalability For January 13, 2012
14 0.68616247 1507 high scalability-2013-08-26-Reddit: Lessons Learned from Mistakes Made Scaling to 1 Billion Pageviews a Month
15 0.68534392 1166 high scalability-2011-12-30-Stuff The Internet Says On Scalability For December 30, 2011
16 0.68221796 1403 high scalability-2013-02-08-Stuff The Internet Says On Scalability For February 8, 2013
17 0.68058085 999 high scalability-2011-03-04-Stuff The Internet Says On Scalability For March 4, 2011
18 0.67719513 106 high scalability-2007-10-02-Secrets to Fotolog's Scaling Success
19 0.67633736 1154 high scalability-2011-12-09-Stuff The Internet Says On Scalability For December 9, 2011
20 0.6741603 757 high scalability-2010-01-04-11 Strategies to Rock Your Startup’s Scalability in 2010
topicId topicWeight
[(1, 0.157), (2, 0.156), (40, 0.016), (47, 0.061), (61, 0.06), (79, 0.14), (85, 0.018), (91, 0.219), (94, 0.074)]
simIndex simValue blogId blogTitle
1 0.91411412 722 high scalability-2009-10-15-Hot Scalability Links for Oct 15 2009
Introduction: Update: Social networks in the database: using a graph database . Anders Nawroth puts graphs through their paces by representing, traversing, and performing other common social network operations using a graph database. Update: Deployment with Capistrano by Charles Max Wood. Simple step-by-step for using Capistrano for deployment. Log-structured file systems: There's one in every SSD by Valerie Aurora. SSDs have totally changed the performance characteristics of storage! Disks are dead! Long live flash! An Engineer's Guide to Bandwidth by DGentry. I t's a rough world out there, and we need to to a better job of thinking about and testing under realistic network conditions. Analyzing air traffic performance with InfoBright and MonetDB by Vadim of the MySQL Performance Blog. Scalable Delivery of Stream Query Result by Zhou, Y ; Salehi, A ; Aberer, K. In this paper, we leverage Distributed Publish/Subscribe System (DPSS), a scalable data dissemination infrastruct
2 0.88621926 712 high scalability-2009-10-01-Moving Beyond End-to-End Path Information to Optimize CDN Performance
Introduction: You go through the expense of installing CDNs all over the globe to make sure users always have a node close by and you notice something curious and furious: clients still experience poor latencies. What's up with that? What do you do to find the problem? If you are Google you build a tool (WhyHigh) to figure out what's up. This paper is about the tool and the unexpected problem of high latencies on CDNs. The main problems they found: inefficient routing to nearby nodes and packet queuing. But more useful is the architecture of WhyHigh and how it goes about identifying bottle necks. And even more useful is the general belief in creating sophisticated tools to understand and improve your service. That's what professionals do. From the abstract: Replicating content across a geographically distributed set of servers and redirecting clients to the closest server in terms of latency has emerged as a common paradigm for improving client performance. In this paper, we analyze latenc
same-blog 3 0.88253915 742 high scalability-2009-11-17-10 eBay Secrets for Planet Wide Scaling
Introduction: You don't even have to make a bid, Randy Shoup, an eBay Distinguished Architect, gives this presentation on how eBay scales, for free. Randy has done a fabulous job in this presentation and in other talks listed at the end of this post getting at the heart of the principles behind scalability. It's more about ideas of how things work and fit together than a focusing on a particular technology stack. Impressive Stats In case you weren't sure, eBay is big, with lots of: users, data, features, and change... Over 89 million active users worldwide 190 million items for sale in 50,000 categories Over 8 billion URL requests per day Hundreds of new features per quarter Roughly 10% of items are listed or ended every day In 39 countries and 10 languages 24x7x365 70 billion read / write operations / day Processes 50TB of new, incremental data per day Analyzes 50PB of data per day 10 Lessons The presentation does a good job explaining each lesson, but the list is.
4 0.87058324 453 high scalability-2008-12-01-Breakthrough Web-Tier Solutions with Record-Breaking Performance
Introduction: With the explosive growth of the Internet, increasing complexity of user requirements, and wide choice of hardware, operating systems, and middleware, IT executives are facing new challenges in their application infrastructures. Rapid expansion of the application tier has resulted in significant cost and complexity, and many organizations are simply running out of datacenter space, power, and cooling.
5 0.86799067 921 high scalability-2010-10-18-NoCAP
Introduction: In this post i wanted to spend sometime on the CAP theorem and clarify some of the confusion that i often see when people associate CAP with scalability without fully understanding the implications that comes with it and the alternative approaches You can read the full article here
6 0.85720199 826 high scalability-2010-05-12-The Rise of the Virtual Cellular Machines
7 0.85718149 1338 high scalability-2012-10-11-RAMCube: Exploiting Network Proximity for RAM-Based Key-Value Store
8 0.81786001 642 high scalability-2009-06-29-HighScalability Rated #3 Blog for Developers
9 0.80591083 1110 high scalability-2011-09-06-Big Data Application Platform
10 0.80465168 356 high scalability-2008-07-22-Scaling Bumper Sticker: A 1 Billion Page Per Month Facebook RoR App
11 0.79760605 167 high scalability-2007-11-27-Starting a website from scratch - what technologies should I use?
12 0.78888083 1053 high scalability-2011-06-06-Apple iCloud: Syncing and Distributed Storage Over Streaming and Centralized Storage
13 0.78540516 1093 high scalability-2011-08-05-Stuff The Internet Says On Scalability For August 5, 2011
14 0.77874005 1209 high scalability-2012-03-14-The Azure Outage: Time Is a SPOF, Leap Day Doubly So
15 0.77800095 1285 high scalability-2012-07-18-Disks Ain't Dead Yet: GraphChi - a disk-based large-scale graph computation
16 0.77226853 651 high scalability-2009-07-02-Product: Project Voldemort - A Distributed Database
17 0.76380116 197 high scalability-2007-12-31-Product: collectd
18 0.76310283 888 high scalability-2010-08-27-OpenStack - The Answer to: How do We Compete with Amazon?
19 0.76210368 380 high scalability-2008-09-05-Product: Tungsten Replicator
20 0.76160371 229 high scalability-2008-01-29-Building scalable storage into application - Instead of MogileFS OpenAFS etc.