high_scalability high_scalability-2008 high_scalability-2008-461 knowledge-graph by maker-knowledge-mining
Source: html
Introduction: At 37 Signals Joshua Sierles describes how 37 Signals uses Sprinkle to configure their servers within EC2. Sprinkle defines a domain specific meta-language for describing and processing the installation of software . You can find an interesting discussion of Sprinkle's creation story by the creator himself, Marcus Crafter, in Sprinkle Some Powder! . Marcus divides provisioning tools into two categories: Task Based - the tool issues a list of commands to run on the remote system, either remotely via a network connection or smart client. Policy/state Based - the tool determines what needs to be run on the remote system by examining its current and final state. Sprinkle combines both models together in a chocolate-in-my-peanut-butter approach using normal Ruby code as the DSL (domain specific language) to declaratively describe remote system configurations. 37 Signals likes the use of Ruby as the DSL because it makes learning a separate syntax unnecessary. I've successfu
sentIndex sentText sentNum sentScore
1 Sprinkle defines a domain specific meta-language for describing and processing the installation of software . [sent-2, score-0.51]
2 You can find an interesting discussion of Sprinkle's creation story by the creator himself, Marcus Crafter, in Sprinkle Some Powder! [sent-3, score-0.091]
3 Marcus divides provisioning tools into two categories: Task Based - the tool issues a list of commands to run on the remote system, either remotely via a network connection or smart client. [sent-5, score-0.841]
4 Policy/state Based - the tool determines what needs to be run on the remote system by examining its current and final state. [sent-6, score-0.623]
5 Sprinkle combines both models together in a chocolate-in-my-peanut-butter approach using normal Ruby code as the DSL (domain specific language) to declaratively describe remote system configurations. [sent-7, score-0.657]
6 37 Signals likes the use of Ruby as the DSL because it makes learning a separate syntax unnecessary. [sent-8, score-0.092]
7 You already have a scripting language, why layer another one on top? [sent-10, score-0.077]
8 One reason not to is that you've now tied configuration and execution together so that only one tool can control the process, but the leverage is so high with this approach it's hard to ignore. [sent-11, score-0.442]
9 There's all the usual bits about defining packages, dependencies, installation logic, pre and post actions, etc. [sent-12, score-0.329]
10 The format is compact and clear because that's how Ruby is and the operations are task specific so there's no fluff. [sent-13, score-0.371]
11 Capistrano is used to communicate with remote systems though that is pluggable. [sent-14, score-0.224]
12 37 Signals uses the EC2 security group as way to specify the role an instance should take on when it boots. [sent-15, score-0.151]
13 A configuration script that can handle all roles is shipped with a near complete functional base image. [sent-16, score-0.368]
14 Sprinkle then configures the system the rest of the way based on the passed in role. [sent-17, score-0.271]
15 Joshua says they like this approach better than Puppet because it doesn't rely on a centralized configuration server or "pushing large sets of commands over SSH manually. [sent-18, score-0.367]
16 " There's always one more than one way to do "it" and Sprinkle carves out an interesting niche in the provisioning space. [sent-19, score-0.318]
17 The 37 Signal's approach doesn't scale to a large organization with many different flavor of servers, but for a specific set of tightly cooperating servers it's a very simple, clean, and robust way of doing business. [sent-20, score-0.684]
wordName wordTfidf (topN-words)
[('sprinkle', 0.374), ('signals', 0.232), ('remote', 0.224), ('dsl', 0.198), ('specific', 0.175), ('ruby', 0.164), ('provisioning', 0.146), ('installation', 0.143), ('puppet', 0.139), ('commands', 0.134), ('powder', 0.133), ('declaratively', 0.133), ('approach', 0.125), ('divides', 0.125), ('joshua', 0.119), ('configures', 0.119), ('examining', 0.119), ('tool', 0.118), ('domain', 0.116), ('flavor', 0.115), ('configuration', 0.108), ('task', 0.108), ('shipped', 0.105), ('pre', 0.101), ('cooperating', 0.097), ('remotely', 0.094), ('tightly', 0.093), ('niche', 0.093), ('capistrano', 0.093), ('syntax', 0.092), ('tied', 0.091), ('creator', 0.091), ('compact', 0.088), ('determines', 0.085), ('defining', 0.085), ('language', 0.084), ('categories', 0.084), ('signal', 0.082), ('packages', 0.081), ('ssh', 0.081), ('roles', 0.079), ('way', 0.079), ('dependencies', 0.078), ('final', 0.077), ('scripting', 0.077), ('script', 0.076), ('describing', 0.076), ('passed', 0.073), ('specify', 0.072), ('actions', 0.071)]
simIndex simValue blogId blogTitle
same-blog 1 1.0000001 461 high scalability-2008-12-05-Sprinkle - Provisioning Tool to Build Remote Servers
Introduction: At 37 Signals Joshua Sierles describes how 37 Signals uses Sprinkle to configure their servers within EC2. Sprinkle defines a domain specific meta-language for describing and processing the installation of software . You can find an interesting discussion of Sprinkle's creation story by the creator himself, Marcus Crafter, in Sprinkle Some Powder! . Marcus divides provisioning tools into two categories: Task Based - the tool issues a list of commands to run on the remote system, either remotely via a network connection or smart client. Policy/state Based - the tool determines what needs to be run on the remote system by examining its current and final state. Sprinkle combines both models together in a chocolate-in-my-peanut-butter approach using normal Ruby code as the DSL (domain specific language) to declaratively describe remote system configurations. 37 Signals likes the use of Ruby as the DSL because it makes learning a separate syntax unnecessary. I've successfu
2 0.2107003 255 high scalability-2008-02-21-Product: Capistrano - Automate Remote Tasks Via SSH
Introduction: Update: Deployment with Capistrano by Charles Max Wood. Nice simple step-by-step for using Capistrano for deployment. From their website: Simply put, Capistrano is a tool for automating tasks on one or more remote servers. It executes commands in parallel on all targeted machines, and provides a mechanism for rolling back changes across multiple machines. It is ideal for anyone doing any kind of system administration, either professionally or incidentally. * Great for automating tasks via SSH on remote servers, like software installation, application deployment, configuration management, ad hoc server monitoring, and more. * Ideal for system administrators, whether professional or incidental. * Easy to customize. Its configuration files use the Ruby programming language syntax, but you don't need to know Ruby to do most things with Capistrano. * Easy to extend. Capistrano is written in the Ruby programming language, and may be extended easily by writing additional Ruby mod
3 0.15085155 429 high scalability-2008-10-25-Product: Puppet the Automated Administration System
Introduction: Update: Digg on their choice and use of Puppet . They chose puppet over cfengine, and bcfg2 because they liked Puppet's resource abstraction layer (RAL), the ability to implement configuration management incrementally, support for bundles, and the overall design philosophy. Puppet implements a declarative (what not how) configuration language for automating common administration tasks. It's the system every large site writes for themselves and it's already made for you! Ilike was able to "easily" scale from 0 to hundreds of servers using Puppet. I can't believe I've never seen this before. It looks really cool. What is Puppet and how can it help you scale your website operations? From the Puppet website: Puppet has been developed to help the sysadmin community move to building and sharing mature tools that avoid the duplication of everyone solving the same problem. It does so in two ways: * It provides a powerful framework to simplify the majority of the technical tasks t
4 0.14056171 1058 high scalability-2011-06-13-Automation on AWS with Ruby and Puppet
Introduction: This is a guest post by Frédéric Faure (architect at Ysance ), you can follow him on twitter . Urbandive is an immersive view service launched by the French YellowPages which allows you to travel in cities in France thanks to a 360° view. Urbandive focuses on providing high definition pictures and accurate professional and social content. One of the biggest jobs was to enable a fast scalable architecture, because it was really difficult to forecast the traffic load at production time. Traffic load may be influenced if the service receives attention from users as a result of advertising. Below you will find a summary of the goals we achieve by using a Ruby scheduler built on top of Puppet on AWS to create a complete infrastructure. Workflow & XTR-Lucid Our scalability combo is : a home-made Ruby scheduler ( XTR-Lucid ) to deal with AWS APIs + the Puppet Master to install services and configure EC2 instances and keep them up-to-date during all the product
5 0.12029576 27 high scalability-2007-07-25-Product: 3 PAR REMOTE COPY
Introduction: 3PAR Remote Copy is a uniquely simple and efficient replication technology that allows customers to protect and share any application data affordably. Built upon 3PAR Thin Copy technology, Remote Copy lowers the total cost of storage by addressing the cost and complexity of remote replication. Common Uses of 3PAR Remote Copy: Affordable Disaster Recovery: Mirror data cost-effectively across town or across the world. Centralized Archive: Replicate data from multiple 3PAR InServs located in multiple data centers to a centralized data archive location. Resilient Pod Architecture: Mutually replicate tier 1 or 2 data to tier 3 capacity between two InServs (application pods). Remote Data Access: Replicate data to a remote location for sharing of data with remote users.
6 0.11384109 881 high scalability-2010-08-16-Scaling an AWS infrastructure - Tools and Patterns
7 0.11234075 1408 high scalability-2013-02-19-Puppet monitoring: how to monitor the success or failure of Puppet runs
8 0.11176378 568 high scalability-2009-04-14-Designing a Scalable Twitter
9 0.10963848 263 high scalability-2008-02-27-Product: System Imager - Automate Deployment and Installs
10 0.10811553 1230 high scalability-2012-04-18-Ansible - A Simple Model-Driven Configuration Management and Command Execution Framework
11 0.10472811 385 high scalability-2008-09-16-Product: Func - Fedora Unified Network Controller
12 0.1002984 369 high scalability-2008-08-18-Code deployment tools
13 0.09349668 272 high scalability-2008-03-08-Product: FAI - Fully Automatic Installation
14 0.092210725 329 high scalability-2008-05-27-Secure Remote Administration for Large-Scale Networks
15 0.089900196 161 high scalability-2007-11-20-Product: SmartFrog a Distributed Configuration and Deployment Framework
17 0.081288002 313 high scalability-2008-05-02-Friends for Sale Architecture - A 300 Million Page View-Month Facebook RoR App
18 0.081210911 1240 high scalability-2012-05-07-Startups are Creating a New System of the World for IT
19 0.079977438 906 high scalability-2010-09-22-Applying Scalability Patterns to Infrastructure Architecture
20 0.07995896 1469 high scalability-2013-06-03-GOV.UK - Not Your Father's Stack
topicId topicWeight
[(0, 0.138), (1, 0.064), (2, -0.01), (3, 0.003), (4, 0.008), (5, -0.008), (6, 0.047), (7, -0.026), (8, -0.028), (9, 0.013), (10, 0.002), (11, 0.078), (12, 0.033), (13, -0.077), (14, 0.01), (15, -0.033), (16, 0.01), (17, -0.011), (18, -0.013), (19, 0.016), (20, -0.009), (21, -0.051), (22, -0.042), (23, 0.011), (24, 0.018), (25, 0.049), (26, 0.004), (27, -0.019), (28, -0.063), (29, 0.013), (30, -0.049), (31, -0.005), (32, -0.036), (33, 0.069), (34, -0.03), (35, -0.04), (36, -0.028), (37, -0.005), (38, -0.042), (39, 0.08), (40, -0.089), (41, -0.042), (42, -0.025), (43, -0.004), (44, 0.005), (45, -0.014), (46, -0.06), (47, -0.049), (48, -0.005), (49, 0.045)]
simIndex simValue blogId blogTitle
same-blog 1 0.9534567 461 high scalability-2008-12-05-Sprinkle - Provisioning Tool to Build Remote Servers
Introduction: At 37 Signals Joshua Sierles describes how 37 Signals uses Sprinkle to configure their servers within EC2. Sprinkle defines a domain specific meta-language for describing and processing the installation of software . You can find an interesting discussion of Sprinkle's creation story by the creator himself, Marcus Crafter, in Sprinkle Some Powder! . Marcus divides provisioning tools into two categories: Task Based - the tool issues a list of commands to run on the remote system, either remotely via a network connection or smart client. Policy/state Based - the tool determines what needs to be run on the remote system by examining its current and final state. Sprinkle combines both models together in a chocolate-in-my-peanut-butter approach using normal Ruby code as the DSL (domain specific language) to declaratively describe remote system configurations. 37 Signals likes the use of Ruby as the DSL because it makes learning a separate syntax unnecessary. I've successfu
2 0.83854133 255 high scalability-2008-02-21-Product: Capistrano - Automate Remote Tasks Via SSH
Introduction: Update: Deployment with Capistrano by Charles Max Wood. Nice simple step-by-step for using Capistrano for deployment. From their website: Simply put, Capistrano is a tool for automating tasks on one or more remote servers. It executes commands in parallel on all targeted machines, and provides a mechanism for rolling back changes across multiple machines. It is ideal for anyone doing any kind of system administration, either professionally or incidentally. * Great for automating tasks via SSH on remote servers, like software installation, application deployment, configuration management, ad hoc server monitoring, and more. * Ideal for system administrators, whether professional or incidental. * Easy to customize. Its configuration files use the Ruby programming language syntax, but you don't need to know Ruby to do most things with Capistrano. * Easy to extend. Capistrano is written in the Ruby programming language, and may be extended easily by writing additional Ruby mod
3 0.82742327 385 high scalability-2008-09-16-Product: Func - Fedora Unified Network Controller
Introduction: Func is used to manage a large network using bash or Python scripts. It targets easy and simple remote scripting and one-off tasks over SSH by creating a secure (SSL certifications) XMLRPC API for communication. Any kind of application can be written on top of it. Other configuration management tools specialize in mass configuration. They say here's what the machine should look like and keep it that way. Func allows you to program your cluster. If you've ever tried to securely remote script a gang of machines using SSH keys you know what a total nightmare that can be. Some example commands: Using the command line: func "*.example.org" call yumcmd update Using the Pthon API: import func.overlord.client as fc client = fc.Client("*.example.org;*.example.com") client.yumcmd.update() client.service.start("acme-server") print client.hardware.info() Func may certainly overlap in functionality with other tools like Puppet and cfengine, but as programmers we always need more than one
Introduction: This is guest post by Michael DeHaan (@laserllama), a software developer and architect, on Ansible , a simple deployment, model-driven configuration management, and command execution framework. I owe High Scalability a great deal of credit for the idea behind my latest software project. I was reading about how an older tool I helped create, Func, was used at Tumblr , and it kicked some ideas into gear. This article is about what happened from that idea. My observation, which the article reinforced, was that many shops end up using a configuration management tool (Puppet, Chef, cfengine), a separate deployment tool (Capistrano, Fabric) and yet another separate ad-hoc task execution tool (Func, pssh, etc) because one class of tool historically hasn't been good at all three jobs. My other observation (not from the article) was that the whole "infrastructure as code" movement, while revolutionary, and definitely great for many, was probably secretly grating on a good number of
5 0.74971789 429 high scalability-2008-10-25-Product: Puppet the Automated Administration System
Introduction: Update: Digg on their choice and use of Puppet . They chose puppet over cfengine, and bcfg2 because they liked Puppet's resource abstraction layer (RAL), the ability to implement configuration management incrementally, support for bundles, and the overall design philosophy. Puppet implements a declarative (what not how) configuration language for automating common administration tasks. It's the system every large site writes for themselves and it's already made for you! Ilike was able to "easily" scale from 0 to hundreds of servers using Puppet. I can't believe I've never seen this before. It looks really cool. What is Puppet and how can it help you scale your website operations? From the Puppet website: Puppet has been developed to help the sysadmin community move to building and sharing mature tools that avoid the duplication of everyone solving the same problem. It does so in two ways: * It provides a powerful framework to simplify the majority of the technical tasks t
6 0.74954408 433 high scalability-2008-10-29-CTL - Distributed Control Dispatching Framework
7 0.74053615 1408 high scalability-2013-02-19-Puppet monitoring: how to monitor the success or failure of Puppet runs
8 0.73259526 807 high scalability-2010-04-09-Vagrant - Build and Deploy Virtualized Development Environments Using Ruby
9 0.71694911 161 high scalability-2007-11-20-Product: SmartFrog a Distributed Configuration and Deployment Framework
10 0.71200371 263 high scalability-2008-02-27-Product: System Imager - Automate Deployment and Installs
11 0.70533735 1531 high scalability-2013-10-13-AIDA: Badoo’s journey into Continuous Integration
12 0.68172091 484 high scalability-2009-01-05-Lessons Learned at 208K: Towards Debugging Millions of Cores
13 0.67313087 272 high scalability-2008-03-08-Product: FAI - Fully Automatic Installation
14 0.65765929 295 high scalability-2008-04-02-Product: Supervisor - Monitor and Control Your Processes
15 0.65031219 369 high scalability-2008-08-18-Code deployment tools
16 0.65027964 228 high scalability-2008-01-28-Product: ISPMan Centralized ISP Management System
17 0.64375609 1469 high scalability-2013-06-03-GOV.UK - Not Your Father's Stack
18 0.63908738 219 high scalability-2008-01-21-Product: Hyperic
19 0.63681984 697 high scalability-2009-09-09-GridwiseTech revolutionizes data management
20 0.63440186 245 high scalability-2008-02-12-Product: rPath - Creating and Managing Virtual Appliances
topicId topicWeight
[(1, 0.112), (2, 0.263), (10, 0.055), (12, 0.012), (30, 0.022), (37, 0.02), (47, 0.102), (56, 0.042), (61, 0.082), (64, 0.116), (79, 0.052), (85, 0.026)]
simIndex simValue blogId blogTitle
same-blog 1 0.95287502 461 high scalability-2008-12-05-Sprinkle - Provisioning Tool to Build Remote Servers
Introduction: At 37 Signals Joshua Sierles describes how 37 Signals uses Sprinkle to configure their servers within EC2. Sprinkle defines a domain specific meta-language for describing and processing the installation of software . You can find an interesting discussion of Sprinkle's creation story by the creator himself, Marcus Crafter, in Sprinkle Some Powder! . Marcus divides provisioning tools into two categories: Task Based - the tool issues a list of commands to run on the remote system, either remotely via a network connection or smart client. Policy/state Based - the tool determines what needs to be run on the remote system by examining its current and final state. Sprinkle combines both models together in a chocolate-in-my-peanut-butter approach using normal Ruby code as the DSL (domain specific language) to declaratively describe remote system configurations. 37 Signals likes the use of Ruby as the DSL because it makes learning a separate syntax unnecessary. I've successfu
2 0.92070919 550 high scalability-2009-03-30-Ebay history and architecture
Introduction: Ebay [1] Starts in 1995, initial name AuctionWeb (V1) : - very simple architecture - based on perl - no database, for data persistence they used plain files Because of rapid growth they needed to improve their architecture and so V2 (clever name) was born: - replaced perl with C/C++ - started using a database in a master-slave configuration - C++ back-end - XSLT front-end Any request will lead to an XML file being created in C++ and the XLST processor will transform that into html. *pretty sophisticated architecture for the 90s, XLST was cutting-edge back then* That hold out pretty well for a while but in the late 90s ebay experienced an exponential growth. They started having some trouble with outages and needed improvements, so V3 was developed: - based on java - search engine still used C++ - proof that relational databases can scale (aggressive caching) - developed a messaging layer for making a lot of asyncronious calls, they a
3 0.90345508 1076 high scalability-2011-07-08-Stuff The Internet Says On Scalability For July 8, 2011
Introduction: Submitted for your scaling pleasure: Facebook confirms 750 million users, sharing 4 billion items daily ; Yahoo: 42,000 Hadoop nodes storing 180-200 petabytes ; Formspring hits 25 million users . Zynga's Cadir Lee : It’s not the amount of hardware that matters. It’s the architecture of the application. You have to work at making your app architecture so that it takes advantage of Amazon. You have to have complete fluidity with the storage tier, the web tier. We are running our own data centers. We are looking more at doing our own data centers with more of a private cloud. Love the sensing making described by Hunch’s Infographic on their Taste Graph . 500 million people, 200 million items, 30 billion edges. 48 processors. 1 TB RAM. Is MongoDB is the New MySQL? Stephen O'Grady thinks so using a worse is better argument: wide adoption by applications, enterprise inroads, simple feature set, and the number complainers. Who plays PostgreSQL in this movie? Java is the
4 0.89422679 1440 high scalability-2013-04-15-Scaling Pinterest - From 0 to 10s of Billions of Page Views a Month in Two Years
Introduction: Pinterest has been riding an exponential growth curve, doubling every month and half. They've gone from 0 to 10s of billions of page views a month in two years, from 2 founders and one engineer to over 40 engineers, from one little MySQL server to 180 Web Engines, 240 API Engines, 88 MySQL DBs (cc2.8xlarge) + 1 slave each, 110 Redis Instances, and 200 Memcache Instances.Stunning growth. So what's Pinterest's story? To tell their story we have our bards, Pinterest'sYashwanth NelapatiandMarty Weiner, who tell the dramatic story of Pinterest's architecture evolution in a talk titledScaling Pinterest. This is the talk they would have liked to hear a year and half ago when they were scaling fast and there were a lot of options to choose from. And they made a lot of incorrect choices.This is a great talk. It's full of amazing details. It's also very practical, down to earth, and it contains strategies adoptable by nearly anyone. Highly recommended.Two of my favorite lessons from the talk:Arc
5 0.89250493 144 high scalability-2007-11-07-What CDN would you recommend?
Introduction: Hi all, a I run a site that after a complete redesign have gotten a lot more traffic. The site provides free flash games, so the biggest traffic share goes to serving flash files (from about 100K and up to several megabytes in size each.) I currently host the entire site on a hosting provider that have no traffic limits. But since they are very cheap (yet have served me very well all the time with at least 99,9% uptime), I don't trust them in allowing me to continue consuming more and more bandwidth. I just guess I'm going to reach some internal limit they have on day, so I'm looking into moving all the flash content over to a content delivery network of some sort. Some recent traffic stats: August: 12 GB September: 22 GB October: 55 GB November: Currently 2,3 GB pr day on average, but it's rising.. I've been looking into Amazon S3, but have not decided on anything yet. So therefor I'm asking if there are any other provides I should consider, that operates withi
6 0.8914687 1530 high scalability-2013-10-11-Stuff The Internet Says On Scalability For October 11th, 2013
7 0.89119405 358 high scalability-2008-07-26-Sharding the Hibernate Way
8 0.8892253 1166 high scalability-2011-12-30-Stuff The Internet Says On Scalability For December 30, 2011
9 0.88843507 248 high scalability-2008-02-13-What's your scalability plan?
10 0.88751811 302 high scalability-2008-04-10-Mysql scalability and failover...
11 0.88672256 925 high scalability-2010-10-22-Paper: Netflix’s Transition to High-Availability Storage Systems
12 0.88671863 676 high scalability-2009-08-08-Yahoo!'s PNUTS Database: Too Hot, Too Cold or Just Right?
13 0.8857972 602 high scalability-2009-05-17-Scaling Django Web Apps by Mike Malone
14 0.88461226 1215 high scalability-2012-03-26-7 Years of YouTube Scalability Lessons in 30 Minutes
15 0.88454241 1065 high scalability-2011-06-21-Running TPC-C on MySQL-RDS
16 0.88435322 756 high scalability-2009-12-30-Terrastore - Scalable, elastic, consistent document store.
17 0.8824842 406 high scalability-2008-10-08-Strategy: Flickr - Do the Essential Work Up-front and Queue the Rest
18 0.88197315 1565 high scalability-2013-12-16-22 Recommendations for Building Effective High Traffic Web Software
19 0.88164771 65 high scalability-2007-08-16-Scaling Secret #2: Denormalizing Your Way to Speed and Profit
20 0.88163167 821 high scalability-2010-05-03-MocoSpace Architecture - 3 Billion Mobile Page Views a Month