high_scalability high_scalability-2009 high_scalability-2009-663 knowledge-graph by maker-knowledge-mining
Source: html
Introduction: Update 7: Basecamp, now with more vroom . Basecamp application servers running Ruby code were upgraded and virtualization was removed. The result: A 66 % reduction in the response time while handling multiples of the traffic is beyond what I expected . They still use virtualization (Linux KVM), just less of it now. Update 6: Things We’ve Learned at 37Signals . Themes: less is more; don't worry be happy. Update 5: Nuts & Bolts: HAproxy . Nice explanation (post, screencast) by Mark Imbriaco of why HAProxy (load balancing proxy server) is their favorite (fast, efficient, graceful configuration, queues requests when Mongrels are busy) for spreading dynamic content between Apache web servers and Mongrel application servers. Update 4: O'Rielly's Tim O'Brien interviews David Hansson , Rails creator and 37signals partner. Says BaseCamp scales horizontally on the application and web tier. Scales up for the database, using one "big ass" 128GB machine. Says: As technology moves on,
sentIndex sentText sentNum sentScore
1 Update: highly useful information on creating a customer billing system. [sent-18, score-0.168]
2 Ask 37signals: How do you process credit cards? [sent-27, score-0.263]
3 Platform Ruby on Rails Memcached Xen MySQL S3 for image storage The Stats 30 servers ranging from single processor file servers to 8 CPU application servers for about 100 CPUs and 200GB of RAM. [sent-29, score-0.186]
4 Xen virtualization will be used to improve system management. [sent-31, score-0.165]
5 9 terabytes of customer-uploaded files * 888 GB files uploaded (900,000 requests) * 2 TB files downloaded (8,500,000 requests) The Architecture Memcached caching is used and they are looking to add more. [sent-33, score-0.494]
6 URL helper methods are used rather than building the URLs by hand. [sent-35, score-0.15]
7 It pays to be king :-) Amazon’s S3 is used for storage of files upload by users. [sent-38, score-0.2]
8 It makes credit card companies more comfortable because they won't be on the hook for a large chunk of change if your company goes out of business. [sent-41, score-0.669]
9 net is the gateway they use to process the credit card charge. [sent-49, score-0.766]
10 If the card is declined three times the account is frozen until a valid card number is provided. [sent-54, score-0.821]
11 Use a gateway service that makes it so you don't have to store credit card numbers on your site. [sent-59, score-0.905]
12 Some gateway services do have reoccurring billing so you don't have to do it yourself. [sent-61, score-0.326]
13 Customer Support Campfire is used for customer service. [sent-62, score-0.155]
14 Issues discussed are used to drive code changes and the subversion commit is shown in the conversation. [sent-64, score-0.189]
15 Seems to skip a bug tracking system, which would make it hard to manage bugs and features in any traditional sense, ie, you can't track subversion changes back to a bug and you can't report what features and bugs are in a release. [sent-65, score-0.637]
16 Support can solve problems by customers uploading images, sharing screens, sharing files, and chatting in real-time. [sent-66, score-0.496]
17 Developers and customers should interact in real-time on a public forum. [sent-72, score-0.174]
18 Developers learn what customers like and dislike which makes product development more agile. [sent-75, score-0.296]
19 Customers can see the responsiveness of the company to customers by reading the interactions. [sent-76, score-0.174]
20 This goes a long ways to give potential customers the confidence and the motivation to sign up. [sent-77, score-0.174]
wordName wordTfidf (topN-words)
[('card', 0.345), ('basecamp', 0.296), ('credit', 0.263), ('campfire', 0.241), ('customers', 0.174), ('highrise', 0.161), ('gateway', 0.158), ('chatting', 0.146), ('files', 0.118), ('subversion', 0.107), ('ror', 0.099), ('billing', 0.095), ('sharing', 0.088), ('virtualization', 0.083), ('used', 0.082), ('bugs', 0.082), ('bug', 0.081), ('numbers', 0.078), ('tracking', 0.076), ('customer', 0.073), ('backpack', 0.073), ('invoice', 0.073), ('positively', 0.073), ('reoccurring', 0.073), ('rails', 0.069), ('helper', 0.068), ('platformruby', 0.068), ('declined', 0.068), ('multiples', 0.068), ('storagethe', 0.068), ('ass', 0.065), ('mongrels', 0.065), ('bills', 0.065), ('freezing', 0.065), ('merchant', 0.065), ('nov', 0.065), ('normal', 0.064), ('features', 0.064), ('gb', 0.063), ('frozen', 0.063), ('screencast', 0.063), ('servers', 0.062), ('explanation', 0.062), ('makes', 0.061), ('dislike', 0.061), ('freeze', 0.061), ('resort', 0.061), ('sent', 0.06), ('items', 0.059), ('add', 0.058)]
simIndex simValue blogId blogTitle
same-blog 1 1.0000001 663 high scalability-2009-07-28-37signals Architecture
Introduction: Update 7: Basecamp, now with more vroom . Basecamp application servers running Ruby code were upgraded and virtualization was removed. The result: A 66 % reduction in the response time while handling multiples of the traffic is beyond what I expected . They still use virtualization (Linux KVM), just less of it now. Update 6: Things We’ve Learned at 37Signals . Themes: less is more; don't worry be happy. Update 5: Nuts & Bolts: HAproxy . Nice explanation (post, screencast) by Mark Imbriaco of why HAProxy (load balancing proxy server) is their favorite (fast, efficient, graceful configuration, queues requests when Mongrels are busy) for spreading dynamic content between Apache web servers and Mongrel application servers. Update 4: O'Rielly's Tim O'Brien interviews David Hansson , Rails creator and 37signals partner. Says BaseCamp scales horizontally on the application and web tier. Scales up for the database, using one "big ass" 128GB machine. Says: As technology moves on,
Introduction: This is a guest post by Ivan de Prado and Pere Ferrera , founders of Datasalt , the company behind Pangool and Splout SQL Big Data open-source projects. The amount of payments performed using credit cards is huge. It is clear that there is inherent value in the data that can be derived from analyzing all the transactions. Client fidelity, demographics, heat maps of activity, shop recommendations, and many other statistics are useful to both clients and shops for improving their relationship with the market. At Datasalt we have developed a system in collaboration with the BBVA bank that is able to analyze years of data and serve insights and statistics to different low-latency web and mobile applications. The main challenge we faced besides processing Big Data input is that the output was also Big Data, and even bigger than the input . And this output needed to be served quickly, under high load. The solution we developed has an infrastructure cost of just a few tho
3 0.15192144 96 high scalability-2007-09-18-Amazon Architecture
Introduction: This is a wonderfully informative Amazon update based on Joachim Rohde's discovery of an interview with Amazon's CTO. You'll learn about how Amazon organizes their teams around services, the CAP theorem of building scalable systems, how they deploy software, and a lot more. Many new additions from the ACM Queue article have also been included. Amazon grew from a tiny online bookstore to one of the largest stores on earth. They did it while pioneering new and interesting ways to rate, review, and recommend products. Greg Linden shared is version of Amazon's birth pangs in a series of blog articles Site: http://amazon.com Information Sources Early Amazon by Greg Linden How Linux saved Amazon millions Interview Werner Vogels - Amazon's CTO Asynchronous Architectures - a nice summary of Werner Vogels' talk by Chris Loosley Learning from the Amazon technology platform - A Conversation with Werner Vogels Werner Vogels' Weblog - building scalable and robus
4 0.14883965 313 high scalability-2008-05-02-Friends for Sale Architecture - A 300 Million Page View-Month Facebook RoR App
Introduction: Update: Jake in Does Django really scale better than Rails? thinks apps like FFS shouldn't need so much hardware to scale. In a short three months Friends for Sale (think Hot-or-Not with a market economy) grew to become a top 10 Facebook application handling 200 gorgeous requests per second and a stunning 300 million page views a month. They did all this using Ruby on Rails, two part time developers, a cluster of a dozen machines, and a fairly standard architecture. How did Friends for Sale scale to sell all those beautiful people? And how much do you think your friends are worth on the open market? Site: http://www.facebook.com/apps/application.php?id=7019261521 Information Sources Siqi Chen and Alexander Le, co-creators of Friends for Sale, answering my standard questionairre. Virality on Facebook The Platform Ruby on Rails CentOS 5 (64 bit) Capistrano - update and restart application servers. Memcached MySQL Nginx Starling - distrib
5 0.13557617 639 high scalability-2009-06-27-Scaling Twitter: Making Twitter 10000 Percent Faster
Introduction: Update 6: Some interesting changes from Twitter's Evan Weaver : everything in RAM now, database is a backup; peaks at 300 tweets/second; every tweet followed by average 126 people; vector cache of tweet IDs; row cache; fragment cache; page cache; keep separate caches; GC makes Ruby optimization resistant so went with Scala; Thrift and HTTP are used internally; 100s internal requests for every external request; rewrote MQ but kept interface the same; 3 queues are used to load balance requests; extensive A/B testing for backwards capability; switched to C memcached client for speed; optimize critical path; faster to get the cached results from the network memory than recompute them locally. Update 5: Twitter on Scala . A Conversation with Steve Jenson, Alex Payne, and Robey Pointer by Bill Venners. A fascinating discussion of why Twitter moved to the Java JVM for their server infrastructure (long lived processes) and why they moved to Scala to program against it (high level langu
6 0.12611496 38 high scalability-2007-07-30-Build an Infinitely Scalable Infrastructure for $100 Using Amazon Services
7 0.12111787 1420 high scalability-2013-03-08-Stuff The Internet Says On Scalability For March 8, 2013
8 0.12093404 1501 high scalability-2013-08-13-In Memoriam: Lavabit Architecture - Creating a Scalable Email Service
9 0.11845006 274 high scalability-2008-03-12-YouTube Architecture
10 0.11826557 241 high scalability-2008-02-05-SLA monitoring
11 0.11749635 1508 high scalability-2013-08-28-Sean Hull's 20 Biggest Bottlenecks that Reduce and Slow Down Scalability
12 0.11712596 664 high scalability-2009-07-29-Strategy: Devirtualize for More Vroom
13 0.11605326 1359 high scalability-2012-11-15-Gone Fishin': Justin.Tv's Live Video Broadcasting Architecture
14 0.11522555 796 high scalability-2010-03-16-Justin.tv's Live Video Broadcasting Architecture
16 0.10969743 750 high scalability-2009-12-16-Building Super Scalable Systems: Blade Runner Meets Autonomic Computing in the Ambient Cloud
17 0.1096874 954 high scalability-2010-12-06-What the heck are you actually using NoSQL for?
18 0.10918349 1455 high scalability-2013-05-10-Stuff The Internet Says On Scalability For May 10, 2013
20 0.10873041 195 high scalability-2007-12-28-Amazon's EC2: Pay as You Grow Could Cut Your Costs in Half
topicId topicWeight
[(0, 0.226), (1, 0.084), (2, -0.026), (3, -0.068), (4, 0.003), (5, -0.071), (6, 0.031), (7, -0.031), (8, -0.01), (9, 0.009), (10, -0.018), (11, -0.012), (12, 0.031), (13, -0.018), (14, 0.02), (15, -0.012), (16, 0.003), (17, 0.02), (18, -0.02), (19, -0.007), (20, 0.001), (21, 0.012), (22, 0.022), (23, 0.034), (24, 0.038), (25, 0.003), (26, -0.008), (27, -0.019), (28, 0.005), (29, 0.068), (30, 0.015), (31, -0.034), (32, 0.006), (33, -0.016), (34, -0.022), (35, -0.014), (36, -0.024), (37, 0.057), (38, 0.009), (39, -0.018), (40, -0.027), (41, -0.026), (42, -0.007), (43, 0.018), (44, -0.027), (45, 0.004), (46, -0.011), (47, -0.011), (48, -0.015), (49, -0.031)]
simIndex simValue blogId blogTitle
same-blog 1 0.97163785 663 high scalability-2009-07-28-37signals Architecture
Introduction: Update 7: Basecamp, now with more vroom . Basecamp application servers running Ruby code were upgraded and virtualization was removed. The result: A 66 % reduction in the response time while handling multiples of the traffic is beyond what I expected . They still use virtualization (Linux KVM), just less of it now. Update 6: Things We’ve Learned at 37Signals . Themes: less is more; don't worry be happy. Update 5: Nuts & Bolts: HAproxy . Nice explanation (post, screencast) by Mark Imbriaco of why HAProxy (load balancing proxy server) is their favorite (fast, efficient, graceful configuration, queues requests when Mongrels are busy) for spreading dynamic content between Apache web servers and Mongrel application servers. Update 4: O'Rielly's Tim O'Brien interviews David Hansson , Rails creator and 37signals partner. Says BaseCamp scales horizontally on the application and web tier. Scales up for the database, using one "big ass" 128GB machine. Says: As technology moves on,
Introduction: Update 4: Heroku versus GAE & GAE/J Update 3 : Heroku has gone live! . Congratulations to the team. It's difficult right now to get a feeling for the relative cost and reliability of Heroku, but it's an impressive accomplishment and a viable option for people looking for a delivery platform. Update 2 : Heroku Architecture . A great interactive presentation of the Heroku stack. Requests flow into Nginx used as a HTTP Reverse Proxy. Nginx routes requests into a Varnish based HTTP cache. Then requests are injected into an Erlang based routing mesh that balances requests across a grid of dynos. Dynos are your application "VMs" that implement application specific behaviors. Dynos themselves are a stack of: POSIX, Ruby VM, App Server, Rack, Middleware, Framework, Your App. Applications can access PostgreSQL. Memcached is used as an application caching layer. Update : Aaron Worsham Interview with James Lindenbaum, CEO of Heroku . Aaron nicely sums up their goal: Heroku is
3 0.80193835 313 high scalability-2008-05-02-Friends for Sale Architecture - A 300 Million Page View-Month Facebook RoR App
Introduction: Update: Jake in Does Django really scale better than Rails? thinks apps like FFS shouldn't need so much hardware to scale. In a short three months Friends for Sale (think Hot-or-Not with a market economy) grew to become a top 10 Facebook application handling 200 gorgeous requests per second and a stunning 300 million page views a month. They did all this using Ruby on Rails, two part time developers, a cluster of a dozen machines, and a fairly standard architecture. How did Friends for Sale scale to sell all those beautiful people? And how much do you think your friends are worth on the open market? Site: http://www.facebook.com/apps/application.php?id=7019261521 Information Sources Siqi Chen and Alexander Le, co-creators of Friends for Sale, answering my standard questionairre. Virality on Facebook The Platform Ruby on Rails CentOS 5 (64 bit) Capistrano - update and restart application servers. Memcached MySQL Nginx Starling - distrib
4 0.7999081 1508 high scalability-2013-08-28-Sean Hull's 20 Biggest Bottlenecks that Reduce and Slow Down Scalability
Introduction: This article is a lightly edited version of 20 Obstacles to Scalability by Sean Hull ( with permission) from the always excellent and thought provoking ACM Queue . 1. TWO-PHASE COMMIT Normally when data is changed in a database, it is written both to memory and to disk. When a commit happens, a relational database makes a commitment to freeze the data somewhere on real storage media. Remember, memory doesn't survive a crash or reboot. Even if the data is cached in memory, the database still has to write it to disk. MySQL binary logs or Oracle redo logs fit the bill. With a MySQL cluster or distributed file system such as DRBD (Distributed Replicated Block Device) or Amazon Multi-AZ (Multi-Availability Zone), a commit occurs not only locally, but also at the remote end. A two-phase commit means waiting for an acknowledgment from the far end. Because of network and other latency, those commits can be slowed down by milliseconds, as though all the cars on a highway were slowe
5 0.7910077 682 high scalability-2009-08-16-ThePort Network Architecture
Introduction: ThePort Network's Director of Engineering, TJ Muehleman was kind of enough to share some of the architectural details for their white label social media system. It currently runs about 50 social networks varying in size from less than 1000 members to more than 300,000 members, all on a Microsoft stack. In addition to their social networking platform, they offer Javascript APIs and web service APIs (both REST and SOAP) which account for a significant percentage of overall system usage. ThePort is an excellent example of a real world in-the-trenches product offering real value to customers. One of the most interesting problems they have to solve is multi-tenancy. How do you provide good performance, complete customization, support, develop new features, and provide individual search indexes for each customer? It's not an easy problem to solve. How did they solve their problems and build a successful system? Site: http://theport.com Platform Microsoft.NET 3.5 C# / VB.NET
6 0.7903741 356 high scalability-2008-07-22-Scaling Bumper Sticker: A 1 Billion Page Per Month Facebook RoR App
7 0.78931791 691 high scalability-2009-08-31-Squarespace Architecture - A Grid Handles Hundreds of Millions of Requests a Month
8 0.78852856 985 high scalability-2011-02-08-Mollom Architecture - Killing Over 373 Million Spams at 100 Requests Per Second
10 0.78531313 1501 high scalability-2013-08-13-In Memoriam: Lavabit Architecture - Creating a Scalable Email Service
11 0.78524095 1068 high scalability-2011-06-27-TripAdvisor Architecture - 40M Visitors, 200M Dynamic Page Views, 30TB Data
12 0.78498065 1573 high scalability-2014-01-06-How HipChat Stores and Indexes Billions of Messages Using ElasticSearch and Redis
13 0.78174609 150 high scalability-2007-11-12-Slashdot Architecture - How the Old Man of the Internet Learned to Scale
14 0.78154671 1131 high scalability-2011-10-24-StackExchange Architecture Updates - Running Smoothly, Amazon 4x More Expensive
15 0.7803638 1378 high scalability-2012-12-28-Stuff The Internet Says On Scalability For December 28, 2012
16 0.77988327 1494 high scalability-2013-07-19-Stuff The Internet Says On Scalability For July 19, 2013
17 0.77930641 1090 high scalability-2011-08-01-Peecho Architecture - scalability on a shoestring
18 0.77853858 1613 high scalability-2014-03-17-Intuitively Showing How To Scale a Web Application Using a Coffee Shop as an Example
19 0.77588552 1552 high scalability-2013-11-22-Stuff The Internet Says On Scalability For November 22th, 2013
20 0.77264905 76 high scalability-2007-08-29-Skype Failed the Boot Scalability Test: Is P2P fundamentally flawed?
topicId topicWeight
[(1, 0.145), (2, 0.24), (5, 0.011), (10, 0.071), (24, 0.143), (30, 0.022), (40, 0.011), (61, 0.052), (77, 0.013), (79, 0.119), (85, 0.066), (94, 0.019)]
simIndex simValue blogId blogTitle
1 0.95276797 1091 high scalability-2011-08-02-How Will DIDO Wireless Networking Change Everything?
Introduction: A conjunction of a few new technologies may trigger disruptive changes in the future. This observation was prompted by a talk Steve Perlman gave at the Columbia Engineering School: Benjamin Button, Cloud Everything and Why Shannon's Law Isn't . In it he covers a set of technologies that at first may seem unrelated, but turn out to be deeply related after all, culminating in a realization of the long talked about vision of an application utility, where all applications are hosted and run out of the cloud. First Perlman talks about the realistic human rendering technology developed at Rearden, his research incubator company. This technology was developed over many years and is the secret behind the wonderful effects found in movies like Benjamin Button . It is now being used in many other films, and promises to revolutionize film making, possibly even replacing actors with computers, in real-time. The next invention at Rearden is OnLive , a cloud based gaming technology for pla
2 0.94453192 1458 high scalability-2013-05-15-Lesson from Airbnb: Give Yourself Permission to Experiment with Non-scalable Changes
Introduction: If you are stuck drowning in too much data and too many options and are dazzled by all the possibilities of code, here's a helpful bit of advice from Airbnb's rags to riches origin story : it's okay to do things that don’t scale . A corollary is the idea of paying attention to and learning from what your users are actually doing and let that lead you without out that annoying voice in your head second guessing you, yelling but that will never scale! Worry about building something good, then worry about making it scale. In Airbnb's case they noticed people weren't booking rooms because the pictures sucked. So they flew to New York and shot some beautiful images. This is a very non-scalable and non-technical solution. Yet it was the turning point for Airbnb and sparked their climb out of the "trough of sorrow." Previously they had been limited by the Silicon Valley idea that every feature had to be scalable. Not every solution can be found behind a computer screen. For the full
3 0.94250232 385 high scalability-2008-09-16-Product: Func - Fedora Unified Network Controller
Introduction: Func is used to manage a large network using bash or Python scripts. It targets easy and simple remote scripting and one-off tasks over SSH by creating a secure (SSL certifications) XMLRPC API for communication. Any kind of application can be written on top of it. Other configuration management tools specialize in mass configuration. They say here's what the machine should look like and keep it that way. Func allows you to program your cluster. If you've ever tried to securely remote script a gang of machines using SSH keys you know what a total nightmare that can be. Some example commands: Using the command line: func "*.example.org" call yumcmd update Using the Pthon API: import func.overlord.client as fc client = fc.Client("*.example.org;*.example.com") client.yumcmd.update() client.service.start("acme-server") print client.hardware.info() Func may certainly overlap in functionality with other tools like Puppet and cfengine, but as programmers we always need more than one
4 0.93972164 958 high scalability-2010-12-16-7 Design Patterns for Almost-infinite Scalability
Introduction: Good article from manageability.com  summarizing design patterns from Pat Helland's amazing paper Life beyond Distributed Transactions: an Apostate's Opinion . Entities are uniquely identified - each entity which represents disjoint data (i.e. no overlap of data between entities) should have a unique key. Multiple disjoint scopes of transactional serializability - in other words there are these 'entities' and that you cannot perform atomic transactions across these entities. At-Least-Once messaging - that is an application must tolerate message retries and out-of-order arrival of messages. Messages are adressed to entities - that is one can't abstract away from the business logic the existence of the unique keys for addresing entities. Addressing however is independent of location. Entities manage conversational state per party - that is, to ensure idemptency an entity needs to remember that a message has been previously processed. Furthermore, in a world with
5 0.93820077 1015 high scalability-2011-04-01-Stuff The Internet Says On Scalability For April 1, 2011
Introduction: Submitted for your reading pleasure, no foolin'... Quotable Quotes: @zateriosystems : thinking about scalability?, are you OK to double your capacity in one week?, a startup should be ready...ready to jump. @sklacy : Maybe what I should have said is "Design for scalability, deploy without it." @MikeHale : Scalability is customer 2000 having the same experience as customer 1 #sqlsat67 @LusciousPear : The meaning of #NoSQL is shut up @deobrat : The biggest bottleneck to scalability are ignorant developers. Most don't even try saving extra CPU cycles or memory bytes :( @w_westendorp : . @ijansch : Cloud computing is like outsourcing your scalability problems @edyavno : Billy Newport essentially just affirmed the theme I've been propagating: "Distributed Caching is the enterprise NoSQL" #strangeloop #nosql @monkchips : HP CEO Leo Apotheker says "relational databases are becoming less and less relevant to the future stack" 12,000 Requests per secon
same-blog 6 0.93755555 663 high scalability-2009-07-28-37signals Architecture
7 0.93406516 1058 high scalability-2011-06-13-Automation on AWS with Ruby and Puppet
8 0.92890453 60 high scalability-2007-08-07-Can you profit from the coming Content Delivery Network wars?
9 0.92116314 738 high scalability-2009-11-06-Product: Resque - GitHub's Distrubuted Job Queue
10 0.91624105 1537 high scalability-2013-10-25-Stuff The Internet Says On Scalability For October 25th, 2013
11 0.91506213 1302 high scalability-2012-08-10-Stuff The Internet Says On Scalability For August 10, 2012
12 0.90822792 396 high scalability-2008-09-26-Lucasfilm: The Real Magic is in the Data Center
13 0.90407008 671 high scalability-2009-08-05-Stack Overflow Architecture
14 0.90380234 274 high scalability-2008-03-12-YouTube Architecture
15 0.90339595 1177 high scalability-2012-01-19-Is it time to get rid of the Linux OS model in the cloud?
16 0.90287912 674 high scalability-2009-08-07-The Canonical Cloud Architecture
17 0.90119052 1000 high scalability-2011-03-08-Medialets Architecture - Defeating the Daunting Mobile Device Data Deluge
18 0.90113699 1600 high scalability-2014-02-21-Stuff The Internet Says On Scalability For February 21st, 2014
19 0.90065145 1602 high scalability-2014-02-26-The WhatsApp Architecture Facebook Bought For $19 Billion