high_scalability high_scalability-2007 high_scalability-2007-138 knowledge-graph by maker-knowledge-mining
Source: html
Introduction: A man had a dream. His dream was to blend a bunch of RSS/Atom/RDF feeds into a single feed. The man is Beau Lebens of Feedville and like most dreamers he was a little short on coin. So he took refuge in the home of a cheap hosting provider and Beau realized his dream, creating FEEDblendr . But FEEDblendr chewed up so much CPU creating blended feeds that the cheap hosting provider ordered Beau to find another home. Where was Beau to go? He eventually found a new home in the virtual machine room of Amazon's EC2. This is the story of how Beau was finally able to create his one feeds safe within the cradle of affordable CPU cycles. Site: http://feedblendr.com/ The Platform EC2 (Fedora Core 6 Lite distro) S3 Apache PHP MySQL DynDNS (for round robin DNS) The Stats Beau is a developer with some sysadmin skills, not a web server admin, so a lot of learning was involved in creating FEEDblendr. FEEDblendr uses 2 EC2 instances. The same Amazon Instance (AMI) is
sentIndex sentText sentNum sentScore
1 His dream was to blend a bunch of RSS/Atom/RDF feeds into a single feed. [sent-2, score-0.439]
2 So he took refuge in the home of a cheap hosting provider and Beau realized his dream, creating FEEDblendr . [sent-4, score-0.356]
3 But FEEDblendr chewed up so much CPU creating blended feeds that the cheap hosting provider ordered Beau to find another home. [sent-5, score-0.644]
4 This is the story of how Beau was finally able to create his one feeds safe within the cradle of affordable CPU cycles. [sent-8, score-0.361]
5 com/ The Platform EC2 (Fedora Core 6 Lite distro) S3 Apache PHP MySQL DynDNS (for round robin DNS) The Stats Beau is a developer with some sysadmin skills, not a web server admin, so a lot of learning was involved in creating FEEDblendr. [sent-10, score-0.228]
6 Processors on the 2 instances are actually pegged pretty high (load averages at ~ 10 - 20 most of the time). [sent-15, score-0.215]
7 The Architecture Round robin DNS is used to load balance between instances. [sent-16, score-0.286]
8 -The DNS is updated by hand as an instance is validited to work correctly before the DNS is updated. [sent-17, score-0.306]
9 It is a clean instance with some auto-deployment code to load the application off of S3. [sent-21, score-0.276]
10 - A makefile is used to get a revision, fix permissions etc, package and push to S3. [sent-24, score-0.28]
11 - When the AMI launches it runs a script to grab the software package from S3. [sent-25, score-0.225]
12 - The package is unpacked and a specific script inside is executed to continue the installation process. [sent-26, score-0.31]
13 This is to reduce the costly polling for feeds as much as possible. [sent-32, score-0.281]
14 Perhaps feeds could be written to each instance so they would be cached on each machine? [sent-34, score-0.507]
15 Round robin load balancing is slow and unreliable. [sent-37, score-0.286]
16 Many problems exist with RSS implementations that keep feeds from being effectively blended. [sent-39, score-0.353]
17 A lot of CPU is spent reading and blending feeds unecessarily because there's no reliable cross implementation way to tell when a feed has really changed or not. [sent-40, score-0.448]
18 It's really a big mindset change to consider that your instances can go away at any time. [sent-41, score-0.198]
19 Create an automated test system to validate an instance as it boots. [sent-47, score-0.219]
20 The last thing you want happening is your instance loading, and for some reason not being able to contact your SVN server, and thus failing to load properly. [sent-51, score-0.276]
wordName wordTfidf (topN-words)
[('beau', 0.424), ('feeds', 0.281), ('ami', 0.258), ('dns', 0.214), ('robin', 0.162), ('blends', 0.159), ('instance', 0.152), ('permissions', 0.152), ('instances', 0.135), ('package', 0.128), ('load', 0.124), ('ip', 0.109), ('script', 0.097), ('budget', 0.09), ('man', 0.09), ('etc', 0.089), ('dream', 0.089), ('chewed', 0.085), ('blending', 0.085), ('dreamers', 0.085), ('lebens', 0.085), ('unpacked', 0.085), ('feed', 0.082), ('provider', 0.081), ('cradle', 0.08), ('pegged', 0.08), ('symlinks', 0.08), ('updated', 0.078), ('fedora', 0.076), ('refuge', 0.076), ('hand', 0.076), ('cached', 0.074), ('distro', 0.073), ('articleswhat', 0.073), ('effectively', 0.072), ('home', 0.071), ('blend', 0.069), ('blended', 0.069), ('revision', 0.067), ('aggregator', 0.067), ('svn', 0.067), ('validate', 0.067), ('occurring', 0.066), ('sysadmin', 0.066), ('persisted', 0.064), ('hosting', 0.064), ('cheap', 0.064), ('mindset', 0.063), ('restarted', 0.063), ('ttl', 0.063)]
simIndex simValue blogId blogTitle
same-blog 1 0.99999982 138 high scalability-2007-10-30-Feedblendr Architecture - Using EC2 to Scale
Introduction: A man had a dream. His dream was to blend a bunch of RSS/Atom/RDF feeds into a single feed. The man is Beau Lebens of Feedville and like most dreamers he was a little short on coin. So he took refuge in the home of a cheap hosting provider and Beau realized his dream, creating FEEDblendr . But FEEDblendr chewed up so much CPU creating blended feeds that the cheap hosting provider ordered Beau to find another home. Where was Beau to go? He eventually found a new home in the virtual machine room of Amazon's EC2. This is the story of how Beau was finally able to create his one feeds safe within the cradle of affordable CPU cycles. Site: http://feedblendr.com/ The Platform EC2 (Fedora Core 6 Lite distro) S3 Apache PHP MySQL DynDNS (for round robin DNS) The Stats Beau is a developer with some sysadmin skills, not a web server admin, so a lot of learning was involved in creating FEEDblendr. FEEDblendr uses 2 EC2 instances. The same Amazon Instance (AMI) is
Introduction: Scoble the Ubiquitous has a fascinating post on how Mogulus, a live video channel startup, uses S3/EC2 and doesn't own a single server. The trends that have been happening for a while now are going mainstream. To do great things you no longer need to start by creating a huge war chest. You can forage off the land, like any good mobile, light weight fighting unit. For a strategy hit he mentions the same needed change in perspective as Beau Lebens talked about when making FeedBlendr : One tip he gave us is that when using Amazon’s services you have to design your systems with the assumption that they will never be up and running. What he means by that is services are “volatile” and can go up and down without notice. So, he’s designed his systems to survive that. He told me that it meant his engineering teams had to be quite disciplined in designing their architecture.
3 0.16917993 1517 high scalability-2013-09-16-The Hidden DNS Tax - Cascading Timeouts and Errors
Introduction: This is a guest post by Nick Burling , VP of Product Management of Bluestripe . Readers of High Scalability know are well versed in performance optimization techniques. Reverse proxies, Varnish, Redis — you hear about them daily. But what you may not realize is that one of the oldest technologies in your stack can be one of your biggest bottlenecks: DNS. People don't spend a lot of time thinking about DNS. It's not sexy. It's an infrastructure service, and it's just supposed to work. At BlueStripe, we work with many teams running applications that support millions of web requests a day. We keep seeing DNS delays and errors that the platform operations team never knows about. It's so common we've start calling it the Hidden DNS Tax . What is the Hidden DNS Tax? The Hidden DNS Tax is a hard-to-see performance hit your users take from DNS timeouts and errors in your back-end architecture. We've seen it bring down the main web application for a Fortune 10 company.
4 0.15727799 1331 high scalability-2012-10-02-An Epic TripAdvisor Update: Why Not Run on the Cloud? The Grand Experiment.
Introduction: This is a guest post by Shawn Hsiao , Luke Massa , and Victor Luu . Shawn runs TripAdvisor ’s Technical Operations team, Luke and Victor interned on his team this past summer. This post is introduced by Andy Gelfond , TripAdvisor’s head of engineering. It's been a little over a year since our last post about the TripAdvisor architecture . It has been an exciting year. Our business and team continues to grow, we are now an independent public company, and we have continued to keep/scale our development process and culture as we have grown - we still run dozens of independent teams, and each team continues to work across the entire stack. All that has changed are the numbers: 56M visitors per month 350M+ pages requests a day 120TB+ of warehouse data running on a large Hadoop cluster, and quickly growing We also had a very successful college intern program that brought on over 60 interns this past summer, all who were quickly on boarded and doing the same kind of work a
5 0.15020597 812 high scalability-2010-04-19-Strategy: Order Two Mediums Instead of Two Smalls and the EC2 Buffet
Introduction: Vaibhav Puranik in Web serving in the cloud – our experiences with nginx and instance sizes describes their experience trying to maximum traffic and minimum their web serving costs on EC2. Initially they tested with two m1.small instance types and then they the switched to two c1.mediums instance types. The m1s are the standard instance types and the c1s are the high CPU instance types. Obviously the mediums have greater capability, but the cost difference was interesting: In the long term they will save money using the larger instances and not autoscaling. With the small instances, traffic bursts caused autoscaling to kick in. New instances were started in response to load. The instances woud be up for a short period of time and then spin down again. This constant churn costs a lot of money. Selecting the larger instance sizes, which are capable of handling the load without autoscaling, turn out to save money even though they are more expensive. Starting new instances also tak
6 0.14492579 7 high scalability-2007-07-12-FeedBurner Architecture
7 0.14429194 1058 high scalability-2011-06-13-Automation on AWS with Ruby and Puppet
8 0.14197026 300 high scalability-2008-04-07-Scalr - Open Source Auto-scaling Hosting on Amazon EC2
9 0.13876507 289 high scalability-2008-03-27-Amazon Announces Static IP Addresses and Multiple Datacenter Operation
10 0.1337304 290 high scalability-2008-03-28-How to Get DNS Names of a Web Server
11 0.12937969 881 high scalability-2010-08-16-Scaling an AWS infrastructure - Tools and Patterns
12 0.1284631 759 high scalability-2010-01-11-Strategy: Don't Use Polling for Real-time Feeds
13 0.12820566 1112 high scalability-2011-09-07-What Google App Engine Price Changes Say About the Future of Web Architecture
14 0.12357461 195 high scalability-2007-12-28-Amazon's EC2: Pay as You Grow Could Cut Your Costs in Half
15 0.11660963 38 high scalability-2007-07-30-Build an Infinitely Scalable Infrastructure for $100 Using Amazon Services
16 0.11328844 109 high scalability-2007-10-03-Save on a Load Balancer By Using Client Side Load Balancing
17 0.11259492 1538 high scalability-2013-10-28-Design Decisions for Scaling Your High Traffic Feeds
18 0.11194108 1470 high scalability-2013-06-05-A Simple 6 Step Transition Guide for Moving Away from X to AWS
20 0.10836665 773 high scalability-2010-02-06-GEO-aware traffic load balancing and caching at CNBC.com
topicId topicWeight
[(0, 0.176), (1, 0.078), (2, -0.041), (3, -0.056), (4, -0.037), (5, -0.1), (6, 0.063), (7, -0.083), (8, 0.004), (9, -0.066), (10, -0.006), (11, -0.006), (12, 0.046), (13, -0.096), (14, -0.044), (15, -0.03), (16, 0.003), (17, 0.007), (18, 0.002), (19, 0.019), (20, -0.039), (21, 0.003), (22, 0.005), (23, -0.038), (24, 0.018), (25, 0.014), (26, -0.001), (27, 0.019), (28, -0.063), (29, -0.048), (30, -0.027), (31, 0.033), (32, -0.026), (33, 0.043), (34, -0.005), (35, -0.075), (36, -0.017), (37, -0.002), (38, -0.013), (39, 0.002), (40, -0.044), (41, 0.077), (42, 0.029), (43, 0.001), (44, 0.057), (45, 0.098), (46, -0.03), (47, 0.034), (48, 0.011), (49, -0.007)]
simIndex simValue blogId blogTitle
same-blog 1 0.97266948 138 high scalability-2007-10-30-Feedblendr Architecture - Using EC2 to Scale
Introduction: A man had a dream. His dream was to blend a bunch of RSS/Atom/RDF feeds into a single feed. The man is Beau Lebens of Feedville and like most dreamers he was a little short on coin. So he took refuge in the home of a cheap hosting provider and Beau realized his dream, creating FEEDblendr . But FEEDblendr chewed up so much CPU creating blended feeds that the cheap hosting provider ordered Beau to find another home. Where was Beau to go? He eventually found a new home in the virtual machine room of Amazon's EC2. This is the story of how Beau was finally able to create his one feeds safe within the cradle of affordable CPU cycles. Site: http://feedblendr.com/ The Platform EC2 (Fedora Core 6 Lite distro) S3 Apache PHP MySQL DynDNS (for round robin DNS) The Stats Beau is a developer with some sysadmin skills, not a web server admin, so a lot of learning was involved in creating FEEDblendr. FEEDblendr uses 2 EC2 instances. The same Amazon Instance (AMI) is
2 0.77819359 1331 high scalability-2012-10-02-An Epic TripAdvisor Update: Why Not Run on the Cloud? The Grand Experiment.
Introduction: This is a guest post by Shawn Hsiao , Luke Massa , and Victor Luu . Shawn runs TripAdvisor ’s Technical Operations team, Luke and Victor interned on his team this past summer. This post is introduced by Andy Gelfond , TripAdvisor’s head of engineering. It's been a little over a year since our last post about the TripAdvisor architecture . It has been an exciting year. Our business and team continues to grow, we are now an independent public company, and we have continued to keep/scale our development process and culture as we have grown - we still run dozens of independent teams, and each team continues to work across the entire stack. All that has changed are the numbers: 56M visitors per month 350M+ pages requests a day 120TB+ of warehouse data running on a large Hadoop cluster, and quickly growing We also had a very successful college intern program that brought on over 60 interns this past summer, all who were quickly on boarded and doing the same kind of work a
3 0.76687753 1058 high scalability-2011-06-13-Automation on AWS with Ruby and Puppet
Introduction: This is a guest post by Frédéric Faure (architect at Ysance ), you can follow him on twitter . Urbandive is an immersive view service launched by the French YellowPages which allows you to travel in cities in France thanks to a 360° view. Urbandive focuses on providing high definition pictures and accurate professional and social content. One of the biggest jobs was to enable a fast scalable architecture, because it was really difficult to forecast the traffic load at production time. Traffic load may be influenced if the service receives attention from users as a result of advertising. Below you will find a summary of the goals we achieve by using a Ruby scheduler built on top of Puppet on AWS to create a complete infrastructure. Workflow & XTR-Lucid Our scalability combo is : a home-made Ruby scheduler ( XTR-Lucid ) to deal with AWS APIs + the Puppet Master to install services and configure EC2 instances and keep them up-to-date during all the product
4 0.72441083 1543 high scalability-2013-11-05-10 Things You Should Know About AWS
Introduction: Authored by Chris Fregly : Former Netflix Streaming Platform Engineer, AWS Certified Solution Architect and Purveyor of fluxcapacitor.com. Ahead of the upcoming 2nd annual re:Invent conference, inspired by Simone Brunozzi’s recent presentation at an AWS Meetup in San Francisco, and collected from a few of my recent Fluxcapacitor.com consulting engagements, I’ve compiled a list of 10 useful time and clock-tick saving tips about AWS. 1) Query AWS resource metadata Can’t remember the EBS-Optimized IO throughput of your c1.xlarge cluster? How about the size limit of an S3 object on a single PUT? awsnow.info is the answer to all of your AWS-resource metadata questions. Interested in integrating awsnow.info with your application? You’re in luck. There’s now a REST API , as well! Note: These are default soft limits and will vary by account. 2) Tame your S3 buckets Delete an entire S3 bucket with a single CLI command:
5 0.72235721 289 high scalability-2008-03-27-Amazon Announces Static IP Addresses and Multiple Datacenter Operation
Introduction: Amazon is fixing two of their major problems: no static IP addresses and single datacenter operation. By adding these two new features developers can finally build a no apology system on Amazon. Before you always had to throw in an apology or two. No, we don't have low failover times because of the silly DNS games and unexceptionable DNS update and propagation times and no, we don't operate in more than one datacenter. No more. Now Amazon is adding Elastic IP Addresses and Availability Zones . Elastic IP addresses are far better than normal IP addresses because they are both in tight with Jessica Alba and they are: Static IP addresses designed for dynamic cloud computing. An Elastic IP address is associated with your account, not a particular instance, and you control that address until you choose to explicitly release it. Unlike traditional static IP addresses, however, Elastic IP addresses allow you to mask instance or availability zone failures by programmatica
6 0.7068339 812 high scalability-2010-04-19-Strategy: Order Two Mediums Instead of Two Smalls and the EC2 Buffet
7 0.70265049 1329 high scalability-2012-09-26-WordPress.com Serves 70,000 req-sec and over 15 Gbit-sec of Traffic using NGINX
9 0.69804794 1434 high scalability-2013-04-03-5 Steps to Benchmarking Managed NoSQL - DynamoDB vs Cassandra
10 0.69288599 1470 high scalability-2013-06-05-A Simple 6 Step Transition Guide for Moving Away from X to AWS
11 0.69150966 314 high scalability-2008-05-03-Product: nginx
12 0.68881482 300 high scalability-2008-04-07-Scalr - Open Source Auto-scaling Hosting on Amazon EC2
13 0.68026161 1517 high scalability-2013-09-16-The Hidden DNS Tax - Cascading Timeouts and Errors
14 0.68008268 798 high scalability-2010-03-22-7 Secrets to Successfully Scaling with Scalr (on Amazon) by Sebastian Stadil
15 0.67215729 773 high scalability-2010-02-06-GEO-aware traffic load balancing and caching at CNBC.com
16 0.6695165 228 high scalability-2008-01-28-Product: ISPMan Centralized ISP Management System
17 0.66577774 1296 high scalability-2012-08-02-Strategy: Use Spare Region Capacity to Survive Availability Zone Failures
18 0.66224146 1260 high scalability-2012-06-07-Case Study on Scaling PaaS infrastructure
19 0.65599614 881 high scalability-2010-08-16-Scaling an AWS infrastructure - Tools and Patterns
20 0.65537679 1188 high scalability-2012-02-06-The Design of 99designs - A Clean Tens of Millions Pageviews Architecture
topicId topicWeight
[(1, 0.133), (2, 0.196), (10, 0.068), (17, 0.022), (27, 0.011), (30, 0.039), (40, 0.051), (51, 0.175), (52, 0.013), (61, 0.106), (77, 0.018), (79, 0.051), (94, 0.041)]
simIndex simValue blogId blogTitle
1 0.93865061 1168 high scalability-2012-01-04-How Facebook Handled the New Year's Eve Onslaught
Introduction: How does Facebook handle the massive New Year's Eve traffic spike? Thanks to Mike Swift, in Facebook gets ready for New Year's Eve , we get a little insight as to their method for the madness, nothing really detailed, but still interesting. Problem Setup Facebook expects tha one billion+ photos will be shared on New Year's eve. Facebook's 800 million users are scattered around the world. Three quarters live outside the US. Each user is linked to an average of 130 friends. Photos and posts must appear in less than a second. Opening a homepage requires executing requests on a 100 different servers, and those requests have to be ranked, sorted, and privacy-checked, and then rendered. Different events put different stresses on different parts of Facebook. Photo and Video Uploads - Holidays require hundreds of terabytes of capacity News Feed - News events like big sports events and the death of Steve Jobs drive user status updates Coping Strategies Try
2 0.91036403 481 high scalability-2009-01-02-Strategy: Understanding Your Data Leads to the Best Scalability Solutions
Introduction: In article Building Super-Scalable Web Systems with REST Udi Dahan tells an interesting story of how they made a weather reporting system scale for over 10 million users. So many users hitting their weather database didn't scale. Caching in a straightforward way wouldn't work because weather is obviously local. Caching all local reports would bring the entire database into memory, which would work for some companies, but wasn't cost efficient for them. So in typical REST fashion they turned locations into URIs. For example: http://weather.myclient.com/UK/London. This allows the weather information to be cached by intermediaries instead of hitting their servers. Hopefully for each location their servers will be hit a few times and then the caches will be hit until expiry. In order to send users directly to the correct location an IP location check is performed on login and stored in a cookie. The lookup is done once and from then on out a GET is performed directly on the r
same-blog 3 0.90728652 138 high scalability-2007-10-30-Feedblendr Architecture - Using EC2 to Scale
Introduction: A man had a dream. His dream was to blend a bunch of RSS/Atom/RDF feeds into a single feed. The man is Beau Lebens of Feedville and like most dreamers he was a little short on coin. So he took refuge in the home of a cheap hosting provider and Beau realized his dream, creating FEEDblendr . But FEEDblendr chewed up so much CPU creating blended feeds that the cheap hosting provider ordered Beau to find another home. Where was Beau to go? He eventually found a new home in the virtual machine room of Amazon's EC2. This is the story of how Beau was finally able to create his one feeds safe within the cradle of affordable CPU cycles. Site: http://feedblendr.com/ The Platform EC2 (Fedora Core 6 Lite distro) S3 Apache PHP MySQL DynDNS (for round robin DNS) The Stats Beau is a developer with some sysadmin skills, not a web server admin, so a lot of learning was involved in creating FEEDblendr. FEEDblendr uses 2 EC2 instances. The same Amazon Instance (AMI) is
4 0.90531188 1134 high scalability-2011-10-28-Stuff The Internet Says On Scalability For October 28, 2011
Introduction: You deserve a HighScalability today : S3: 566 Billion Objects, 370K requests/sec ; Titan: 38,400-processor, 20-petaflop 1,000,000 daily users and no cache . Wooga flash game with 50K DB updates/second, Ruby backend. They hit an IO wall with MySQL at 1000 DB updates/sec. They needed more so they went with Redis. Not quite honest to say no cache was used as everything is RAM, but maybe that's the point. Use a lot of automation. Inactive users are archived. Moved away from EBS. Making dynamic sitesscale like static sites by Wim Godden. Use Varnish, Nginx, and memcached. The Lifecycle of a Web Page on StumbleUpon infographic. 2.2 mllion web pages are added to StumbleUpon each month. Nice discussion of bounce rate . James Hamilton with an excellent overview of the Storage Infrastructure Behind Facebook Messages . That's 6B+ messages a day. Scaling Twilio . Twilio has scaled traffic by more 100x over the past year, and expanded their server infrastructure from a
Introduction: StubHub is an interesting architecture to take a look at because, as market makers for tickets, they are in a different business than we normally get to consider. StubHub is surprisingly large, growing at 20% a year, serving 800K complex pages per hour, selling 5 million tickets per year, and handling 2 million API calls per hour. And the ticket space is surprisingly rich in complexity. StubHub's traffic is tricky. It's bursty, centering around unpredictable game outcomes, events, schedules, and seasons. There’s a lot of money involved. There are a lot of different actors involved. There are a lot of complex business processes involved. And StubHub has several complementary but very different parts of their business: they have an ad server component serving ads to sites like ESPN, a rich interactive UI, and a real-time ticket market component. Most interesting to me is how StubHub is bringing into the digital realm the once quintessentially high-touch physical world of tickets,
7 0.89247113 98 high scalability-2007-09-18-Sync data on all servers
8 0.88237047 1644 high scalability-2014-05-07-Update on Disqus: It's Still About Realtime, But Go Demolishes Python
9 0.88207531 818 high scalability-2010-04-30-Behind the scenes of an online marketplace
10 0.87494117 741 high scalability-2009-11-16-Building Scalable Systems Using Data as a Composite Material
11 0.86497992 510 high scalability-2009-02-09-Paper: Consensus Protocols: Two-Phase Commit
12 0.85825932 626 high scalability-2009-06-10-Paper: Graph Databases and the Future of Large-Scale Knowledge Management
13 0.85291761 953 high scalability-2010-12-03-GPU vs CPU Smackdown : The Rise of Throughput-Oriented Architectures
14 0.84982044 298 high scalability-2008-04-07-Lazy web sites run faster
15 0.84638166 1638 high scalability-2014-04-28-How Disqus Went Realtime with 165K Messages Per Second and Less than .2 Seconds Latency
17 0.83786923 106 high scalability-2007-10-02-Secrets to Fotolog's Scaling Success
18 0.8336305 609 high scalability-2009-05-28-Scaling PostgreSQL using CUDA
19 0.83013058 1372 high scalability-2012-12-14-Stuff The Internet Says On Scalability For December 14, 2012
20 0.82926035 1147 high scalability-2011-11-25-Stuff The Internet Says On Scalability For November 25, 2011