high_scalability high_scalability-2011 high_scalability-2011-1058 knowledge-graph by maker-knowledge-mining
Source: html
Introduction: This is a guest post by Frédéric Faure (architect at Ysance ), you can follow him on twitter . Urbandive is an immersive view service launched by the French YellowPages which allows you to travel in cities in France thanks to a 360° view. Urbandive focuses on providing high definition pictures and accurate professional and social content. One of the biggest jobs was to enable a fast scalable architecture, because it was really difficult to forecast the traffic load at production time. Traffic load may be influenced if the service receives attention from users as a result of advertising. Below you will find a summary of the goals we achieve by using a Ruby scheduler built on top of Puppet on AWS to create a complete infrastructure. Workflow & XTR-Lucid Our scalability combo is : a home-made Ruby scheduler ( XTR-Lucid ) to deal with AWS APIs + the Puppet Master to install services and configure EC2 instances and keep them up-to-date during all the product
sentIndex sentText sentNum sentScore
1 Below you will find a summary of the goals we achieve by using a Ruby scheduler built on top of Puppet on AWS to create a complete infrastructure. [sent-6, score-0.289]
2 Workflow & XTR-Lucid Our scalability combo is : a home-made Ruby scheduler ( XTR-Lucid ) to deal with AWS APIs + the Puppet Master to install services and configure EC2 instances and keep them up-to-date during all the production time. [sent-7, score-0.626]
3 The dashboard allows you to select a template (which contains the following informations : AMI id, instance type, availability zone, key, list of security groups, list of EBS – from snapshots or not -, …) and to set a name for the instance in the « create » workflow. [sent-10, score-0.575]
4 To be more accurate and to explain the workflow just a bit : The scheduler can be outside or inside of AWS. [sent-12, score-0.473]
5 First an action file is put into the TODO directory, by the web application (simple rhtml dashboard) or by the monitoring tool directly. [sent-13, score-0.303]
6 For posting files from the monitoring tool, we will have to define thresholds after some weeks of use of the application. [sent-14, score-0.269]
7 The action file is then processed by the scheduler : Connection to AWS and request to start an EC2 instance from the AMI. [sent-16, score-0.592]
8 Scheduler checks that instance is running, EBS (Elastic Block Store) are available , then in-use , and eventually that EC2 TCP stack is up and SSH is OK. [sent-17, score-0.226]
9 Puppet client is installed and started, so it sends a certificate request to the master. [sent-19, score-0.255]
10 Connection to the PuppetMaster : update the main files (nodes files of PuppetMaster, « /etc/hosts » file, roles file of Capistrano) and then accept the certificate request of the client. [sent-21, score-0.299]
11 Capistrano updates the « /etc/hosts » file on all instances of our infrastructure as soon as a new EC2 instance comes up (or down), without having to trigger ( puppetrun ) the simultaneous « pull configuration » of all the Puppet clients. [sent-22, score-0.908]
12 Puppet client connects to the master and authenticate. [sent-25, score-0.17]
13 The installation occurs on the new instance that will be soon available. [sent-27, score-0.372]
14 The scheduler doesn’t wait until the instance is fully configured to end the task : installation by Puppet is asynchronous. [sent-28, score-0.583]
15 The scheduler just ends by updating its repository and the monitoring tool. [sent-29, score-0.523]
16 AWS offer AMIs to « snapshot » an instance and clone this one as many as we want. [sent-41, score-0.226]
17 But when we come into production, we cannot build another AMI each time we do a change on a setting and deploy again EC2 instances from the new AMI. [sent-44, score-0.203]
18 Then our tool lets Puppet take over which ensures for each brand-new instance started from a standard Lucid Lynx AMI that all services, configurations, security rules are installed/applied. [sent-50, score-0.466]
19 So with the feedback of our monitoring tool, the infrastructure can grow and decrease alone (depending on the thresholds set) or we can deploy and stop instances in just one click on our web dashboard. [sent-51, score-0.511]
20 Once we have real-life feedback on traffic, some services will stay on AWS, others, maybe, will migrate into our physical datacenter. [sent-58, score-0.236]
wordName wordTfidf (topN-words)
[('puppet', 0.33), ('scheduler', 0.289), ('puppetmaster', 0.278), ('instance', 0.226), ('ami', 0.191), ('instances', 0.14), ('lucid', 0.139), ('lynx', 0.139), ('puppetrun', 0.139), ('thetododirectory', 0.139), ('aws', 0.136), ('repository', 0.12), ('trigger', 0.116), ('monitoring', 0.114), ('tool', 0.112), ('accurate', 0.108), ('ruby', 0.104), ('france', 0.102), ('pull', 0.101), ('le', 0.1), ('certificate', 0.1), ('feedback', 0.098), ('ops', 0.094), ('thresholds', 0.094), ('ebs', 0.093), ('client', 0.092), ('ubuntu', 0.083), ('soon', 0.078), ('connects', 0.078), ('file', 0.077), ('workflow', 0.076), ('configurations', 0.076), ('snapshot', 0.076), ('services', 0.075), ('installation', 0.068), ('simultaneous', 0.067), ('infrastructure', 0.065), ('security', 0.064), ('standard', 0.064), ('configure', 0.063), ('setting', 0.063), ('installed', 0.063), ('migrate', 0.063), ('ob', 0.063), ('traffic', 0.062), ('files', 0.061), ('combo', 0.059), ('informations', 0.059), ('ysance', 0.059), ('immersive', 0.059)]
simIndex simValue blogId blogTitle
same-blog 1 1.0000001 1058 high scalability-2011-06-13-Automation on AWS with Ruby and Puppet
Introduction: This is a guest post by Frédéric Faure (architect at Ysance ), you can follow him on twitter . Urbandive is an immersive view service launched by the French YellowPages which allows you to travel in cities in France thanks to a 360° view. Urbandive focuses on providing high definition pictures and accurate professional and social content. One of the biggest jobs was to enable a fast scalable architecture, because it was really difficult to forecast the traffic load at production time. Traffic load may be influenced if the service receives attention from users as a result of advertising. Below you will find a summary of the goals we achieve by using a Ruby scheduler built on top of Puppet on AWS to create a complete infrastructure. Workflow & XTR-Lucid Our scalability combo is : a home-made Ruby scheduler ( XTR-Lucid ) to deal with AWS APIs + the Puppet Master to install services and configure EC2 instances and keep them up-to-date during all the product
2 0.31349337 881 high scalability-2010-08-16-Scaling an AWS infrastructure - Tools and Patterns
Introduction: This is a guest post by Frédéric Faure (architect at Ysance ), you can follow him on twitter . How do you scale an AWS (Amazon Web Services) infrastructure? This article will give you a detailed reply in two parts: the tools you can use to make the most of Amazon’s dynamic approach, and the architectural model you should adopt for a scalable infrastructure. I base my report on my experience gained in several AWS production projects in casual gaming (Facebook), e-commerce infrastructures and within the mainstream GIS (Geographic Information System). It’s true that my experience in gaming ( IsCool, The Game ) is currently the most representative in terms of scalability, due to the number of users (over 800 thousand DAU – daily active users – at peak usage and over 20 million page views every day), however my experiences in e-commerce and GIS (currently underway) provide a different view of scalability, taking into account the various problems of availability and da
3 0.26893693 429 high scalability-2008-10-25-Product: Puppet the Automated Administration System
Introduction: Update: Digg on their choice and use of Puppet . They chose puppet over cfengine, and bcfg2 because they liked Puppet's resource abstraction layer (RAL), the ability to implement configuration management incrementally, support for bundles, and the overall design philosophy. Puppet implements a declarative (what not how) configuration language for automating common administration tasks. It's the system every large site writes for themselves and it's already made for you! Ilike was able to "easily" scale from 0 to hundreds of servers using Puppet. I can't believe I've never seen this before. It looks really cool. What is Puppet and how can it help you scale your website operations? From the Puppet website: Puppet has been developed to help the sysadmin community move to building and sharing mature tools that avoid the duplication of everyone solving the same problem. It does so in two ways: * It provides a powerful framework to simplify the majority of the technical tasks t
4 0.26344192 1408 high scalability-2013-02-19-Puppet monitoring: how to monitor the success or failure of Puppet runs
Introduction: This is a guest post by LogicMonitor's Director of Tech Ops, Jesse Aukeman , about the different ways they're monitoring the success or failure of Puppet runs. If you are like us, you are running some type of linux configuration management tool. The value of centralized configuration and deployment is well known and hard to overstate. Puppet is our tool of choice. It is powerful and works well for us, except when things don't go as planned. Failures of puppet can be innocuous and cosmetic, or they can cause production issues, for example when crucial updates do not get properly propagated. Why? In the most innocuous cases, the puppet agent craps out (we run puppet agent via cron). As nice as puppet is, we still need to goose it from time to time to get past some sort of network or host resource issue. A more dangerous case is when an administrator temporarily disables puppet runs on a host in order to perform some test or administrative task and then forgets to reenab
5 0.20951407 853 high scalability-2010-07-08-Cloud AWS Infrastructure vs. Physical Infrastructure
Introduction: This is a guest post by Frédéric Faure (architect at Ysance ) on the differences between using a cloud infrastructure and building your own. Frédéric was kind enough to translate the original French version of this article into English. I’ve been noticing many questions about the differences inherent in choosing between a Cloud infrastructure such as AWS (Amazon Web Services) and a traditional physical infrastructure. Firstly, there are a certain number of preconceived notions on this subject that I will attempt to decode for you. Then, it must be understood that each infrastructure has its advantages and disadvantages: a Cloud-type infrastructure does not necessarily fulfill your requirements in every case, however, it can satisfy some of them by optimizing or facilitating the features offered by a traditional physical infrastructure. I will therefore demonstrate the differences between the two that I have noticed, in order to help you make up your own mind. The Fram
6 0.20794383 1331 high scalability-2012-10-02-An Epic TripAdvisor Update: Why Not Run on the Cloud? The Grand Experiment.
7 0.19665818 1112 high scalability-2011-09-07-What Google App Engine Price Changes Say About the Future of Web Architecture
8 0.1595097 812 high scalability-2010-04-19-Strategy: Order Two Mediums Instead of Two Smalls and the EC2 Buffet
9 0.15300563 1543 high scalability-2013-11-05-10 Things You Should Know About AWS
10 0.1445637 1470 high scalability-2013-06-05-A Simple 6 Step Transition Guide for Moving Away from X to AWS
11 0.14429194 138 high scalability-2007-10-30-Feedblendr Architecture - Using EC2 to Scale
12 0.14056171 461 high scalability-2008-12-05-Sprinkle - Provisioning Tool to Build Remote Servers
14 0.13506489 1631 high scalability-2014-04-14-How do you even do anything without using EBS?
15 0.13385978 1240 high scalability-2012-05-07-Startups are Creating a New System of the World for IT
18 0.12298049 205 high scalability-2008-01-10-Letting Clients Know What's Changed: Push Me or Pull Me?
19 0.12057526 1291 high scalability-2012-07-25-Vertical Scaling Ascendant - How are SSDs Changing Architectures?
topicId topicWeight
[(0, 0.219), (1, 0.03), (2, -0.049), (3, 0.007), (4, -0.043), (5, -0.086), (6, 0.114), (7, -0.111), (8, 0.036), (9, -0.113), (10, -0.012), (11, 0.042), (12, 0.065), (13, -0.149), (14, -0.044), (15, -0.025), (16, 0.017), (17, 0.03), (18, 0.005), (19, 0.001), (20, 0.009), (21, -0.043), (22, 0.016), (23, 0.022), (24, -0.033), (25, 0.013), (26, -0.034), (27, -0.022), (28, -0.082), (29, -0.087), (30, -0.032), (31, 0.018), (32, 0.018), (33, 0.092), (34, -0.058), (35, -0.08), (36, -0.01), (37, -0.088), (38, -0.027), (39, 0.001), (40, -0.032), (41, -0.025), (42, -0.018), (43, -0.009), (44, -0.034), (45, 0.012), (46, -0.051), (47, 0.065), (48, 0.013), (49, 0.115)]
simIndex simValue blogId blogTitle
same-blog 1 0.97149897 1058 high scalability-2011-06-13-Automation on AWS with Ruby and Puppet
Introduction: This is a guest post by Frédéric Faure (architect at Ysance ), you can follow him on twitter . Urbandive is an immersive view service launched by the French YellowPages which allows you to travel in cities in France thanks to a 360° view. Urbandive focuses on providing high definition pictures and accurate professional and social content. One of the biggest jobs was to enable a fast scalable architecture, because it was really difficult to forecast the traffic load at production time. Traffic load may be influenced if the service receives attention from users as a result of advertising. Below you will find a summary of the goals we achieve by using a Ruby scheduler built on top of Puppet on AWS to create a complete infrastructure. Workflow & XTR-Lucid Our scalability combo is : a home-made Ruby scheduler ( XTR-Lucid ) to deal with AWS APIs + the Puppet Master to install services and configure EC2 instances and keep them up-to-date during all the product
2 0.81407988 1543 high scalability-2013-11-05-10 Things You Should Know About AWS
Introduction: Authored by Chris Fregly : Former Netflix Streaming Platform Engineer, AWS Certified Solution Architect and Purveyor of fluxcapacitor.com. Ahead of the upcoming 2nd annual re:Invent conference, inspired by Simone Brunozzi’s recent presentation at an AWS Meetup in San Francisco, and collected from a few of my recent Fluxcapacitor.com consulting engagements, I’ve compiled a list of 10 useful time and clock-tick saving tips about AWS. 1) Query AWS resource metadata Can’t remember the EBS-Optimized IO throughput of your c1.xlarge cluster? How about the size limit of an S3 object on a single PUT? awsnow.info is the answer to all of your AWS-resource metadata questions. Interested in integrating awsnow.info with your application? You’re in luck. There’s now a REST API , as well! Note: These are default soft limits and will vary by account. 2) Tame your S3 buckets Delete an entire S3 bucket with a single CLI command:
3 0.78147537 881 high scalability-2010-08-16-Scaling an AWS infrastructure - Tools and Patterns
Introduction: This is a guest post by Frédéric Faure (architect at Ysance ), you can follow him on twitter . How do you scale an AWS (Amazon Web Services) infrastructure? This article will give you a detailed reply in two parts: the tools you can use to make the most of Amazon’s dynamic approach, and the architectural model you should adopt for a scalable infrastructure. I base my report on my experience gained in several AWS production projects in casual gaming (Facebook), e-commerce infrastructures and within the mainstream GIS (Geographic Information System). It’s true that my experience in gaming ( IsCool, The Game ) is currently the most representative in terms of scalability, due to the number of users (over 800 thousand DAU – daily active users – at peak usage and over 20 million page views every day), however my experiences in e-commerce and GIS (currently underway) provide a different view of scalability, taking into account the various problems of availability and da
4 0.75474834 1331 high scalability-2012-10-02-An Epic TripAdvisor Update: Why Not Run on the Cloud? The Grand Experiment.
Introduction: This is a guest post by Shawn Hsiao , Luke Massa , and Victor Luu . Shawn runs TripAdvisor ’s Technical Operations team, Luke and Victor interned on his team this past summer. This post is introduced by Andy Gelfond , TripAdvisor’s head of engineering. It's been a little over a year since our last post about the TripAdvisor architecture . It has been an exciting year. Our business and team continues to grow, we are now an independent public company, and we have continued to keep/scale our development process and culture as we have grown - we still run dozens of independent teams, and each team continues to work across the entire stack. All that has changed are the numbers: 56M visitors per month 350M+ pages requests a day 120TB+ of warehouse data running on a large Hadoop cluster, and quickly growing We also had a very successful college intern program that brought on over 60 interns this past summer, all who were quickly on boarded and doing the same kind of work a
5 0.74715996 1408 high scalability-2013-02-19-Puppet monitoring: how to monitor the success or failure of Puppet runs
Introduction: This is a guest post by LogicMonitor's Director of Tech Ops, Jesse Aukeman , about the different ways they're monitoring the success or failure of Puppet runs. If you are like us, you are running some type of linux configuration management tool. The value of centralized configuration and deployment is well known and hard to overstate. Puppet is our tool of choice. It is powerful and works well for us, except when things don't go as planned. Failures of puppet can be innocuous and cosmetic, or they can cause production issues, for example when crucial updates do not get properly propagated. Why? In the most innocuous cases, the puppet agent craps out (we run puppet agent via cron). As nice as puppet is, we still need to goose it from time to time to get past some sort of network or host resource issue. A more dangerous case is when an administrator temporarily disables puppet runs on a host in order to perform some test or administrative task and then forgets to reenab
6 0.73518312 1347 high scalability-2012-10-25-Not All Regions are Created Equal - South America Es Bueno
7 0.73363656 812 high scalability-2010-04-19-Strategy: Order Two Mediums Instead of Two Smalls and the EC2 Buffet
8 0.72953558 1434 high scalability-2013-04-03-5 Steps to Benchmarking Managed NoSQL - DynamoDB vs Cassandra
9 0.72704804 1165 high scalability-2011-12-28-Strategy: Guaranteed Availability Requires Reserving Instances in Specific Zones
10 0.71207148 1286 high scalability-2012-07-18-Strategy: Kill Off Multi-tenant Instances with High CPU Stolen Time
11 0.70846742 138 high scalability-2007-10-30-Feedblendr Architecture - Using EC2 to Scale
12 0.67109156 853 high scalability-2010-07-08-Cloud AWS Infrastructure vs. Physical Infrastructure
13 0.66722172 1230 high scalability-2012-04-18-Ansible - A Simple Model-Driven Configuration Management and Command Execution Framework
15 0.66562849 1631 high scalability-2014-04-14-How do you even do anything without using EBS?
16 0.66190624 1121 high scalability-2011-09-21-5 Scalability Poisons and 3 Cloud Scalability Antidotes
17 0.6479823 1371 high scalability-2012-12-12-Pinterest Cut Costs from $54 to $20 Per Hour by Automatically Shutting Down Systems
18 0.64426583 1470 high scalability-2013-06-05-A Simple 6 Step Transition Guide for Moving Away from X to AWS
19 0.6437673 372 high scalability-2008-08-27-Updating distributed web applications
20 0.64049804 1531 high scalability-2013-10-13-AIDA: Badoo’s journey into Continuous Integration
topicId topicWeight
[(1, 0.155), (2, 0.176), (10, 0.091), (24, 0.199), (25, 0.022), (30, 0.03), (47, 0.024), (61, 0.066), (73, 0.011), (77, 0.01), (79, 0.1), (94, 0.045)]
simIndex simValue blogId blogTitle
1 0.92607594 385 high scalability-2008-09-16-Product: Func - Fedora Unified Network Controller
Introduction: Func is used to manage a large network using bash or Python scripts. It targets easy and simple remote scripting and one-off tasks over SSH by creating a secure (SSL certifications) XMLRPC API for communication. Any kind of application can be written on top of it. Other configuration management tools specialize in mass configuration. They say here's what the machine should look like and keep it that way. Func allows you to program your cluster. If you've ever tried to securely remote script a gang of machines using SSH keys you know what a total nightmare that can be. Some example commands: Using the command line: func "*.example.org" call yumcmd update Using the Pthon API: import func.overlord.client as fc client = fc.Client("*.example.org;*.example.com") client.yumcmd.update() client.service.start("acme-server") print client.hardware.info() Func may certainly overlap in functionality with other tools like Puppet and cfengine, but as programmers we always need more than one
2 0.91272235 60 high scalability-2007-08-07-Can you profit from the coming Content Delivery Network wars?
Introduction: Playing like the big boys may be getting cheaper. The big boys, like YouTube , farm the serving of their most popular videos to a third party CDN. A lot of people were surprised YouTube didn't serve all their content themselves, but it makes sense. It allows them to keep up with demand without a large hit for infrastructure build out, much like leasing computers instead of buying them. The problem has been CDNs are expensive. Om Malik reports in Akamai & the CDN Price Wars that may be changing. CDN service could be becoming affordable enough that you might consider using them as part of your scaling strategy. Akamai , once the clear leader in the CDN field, is facing strong competition from the likes of Limelight Networks, Level 3, Internap, CDNetworks, Panther Express and EdgeCast Networks. This commoditization may be bad for their stock prices, but it's good for website builders looking for new scaling strategies. EdgeCast, for example, passes on the cost savings when
same-blog 3 0.89577538 1058 high scalability-2011-06-13-Automation on AWS with Ruby and Puppet
Introduction: This is a guest post by Frédéric Faure (architect at Ysance ), you can follow him on twitter . Urbandive is an immersive view service launched by the French YellowPages which allows you to travel in cities in France thanks to a 360° view. Urbandive focuses on providing high definition pictures and accurate professional and social content. One of the biggest jobs was to enable a fast scalable architecture, because it was really difficult to forecast the traffic load at production time. Traffic load may be influenced if the service receives attention from users as a result of advertising. Below you will find a summary of the goals we achieve by using a Ruby scheduler built on top of Puppet on AWS to create a complete infrastructure. Workflow & XTR-Lucid Our scalability combo is : a home-made Ruby scheduler ( XTR-Lucid ) to deal with AWS APIs + the Puppet Master to install services and configure EC2 instances and keep them up-to-date during all the product
4 0.88665891 1458 high scalability-2013-05-15-Lesson from Airbnb: Give Yourself Permission to Experiment with Non-scalable Changes
Introduction: If you are stuck drowning in too much data and too many options and are dazzled by all the possibilities of code, here's a helpful bit of advice from Airbnb's rags to riches origin story : it's okay to do things that don’t scale . A corollary is the idea of paying attention to and learning from what your users are actually doing and let that lead you without out that annoying voice in your head second guessing you, yelling but that will never scale! Worry about building something good, then worry about making it scale. In Airbnb's case they noticed people weren't booking rooms because the pictures sucked. So they flew to New York and shot some beautiful images. This is a very non-scalable and non-technical solution. Yet it was the turning point for Airbnb and sparked their climb out of the "trough of sorrow." Previously they had been limited by the Silicon Valley idea that every feature had to be scalable. Not every solution can be found behind a computer screen. For the full
5 0.87682456 1091 high scalability-2011-08-02-How Will DIDO Wireless Networking Change Everything?
Introduction: A conjunction of a few new technologies may trigger disruptive changes in the future. This observation was prompted by a talk Steve Perlman gave at the Columbia Engineering School: Benjamin Button, Cloud Everything and Why Shannon's Law Isn't . In it he covers a set of technologies that at first may seem unrelated, but turn out to be deeply related after all, culminating in a realization of the long talked about vision of an application utility, where all applications are hosted and run out of the cloud. First Perlman talks about the realistic human rendering technology developed at Rearden, his research incubator company. This technology was developed over many years and is the secret behind the wonderful effects found in movies like Benjamin Button . It is now being used in many other films, and promises to revolutionize film making, possibly even replacing actors with computers, in real-time. The next invention at Rearden is OnLive , a cloud based gaming technology for pla
6 0.87077969 396 high scalability-2008-09-26-Lucasfilm: The Real Magic is in the Data Center
7 0.85238707 1015 high scalability-2011-04-01-Stuff The Internet Says On Scalability For April 1, 2011
8 0.84620839 958 high scalability-2010-12-16-7 Design Patterns for Almost-infinite Scalability
9 0.83029413 1302 high scalability-2012-08-10-Stuff The Internet Says On Scalability For August 10, 2012
10 0.82803655 663 high scalability-2009-07-28-37signals Architecture
11 0.81531453 1537 high scalability-2013-10-25-Stuff The Internet Says On Scalability For October 25th, 2013
12 0.80555874 738 high scalability-2009-11-06-Product: Resque - GitHub's Distrubuted Job Queue
13 0.80064517 1068 high scalability-2011-06-27-TripAdvisor Architecture - 40M Visitors, 200M Dynamic Page Views, 30TB Data
14 0.79958558 853 high scalability-2010-07-08-Cloud AWS Infrastructure vs. Physical Infrastructure
15 0.79872435 881 high scalability-2010-08-16-Scaling an AWS infrastructure - Tools and Patterns
16 0.79846984 1553 high scalability-2013-11-25-How To Make an Infinitely Scalable Relational Database Management System (RDBMS)
17 0.79611063 1369 high scalability-2012-12-10-Switch your databases to Flash storage. Now. Or you're doing it wrong.
18 0.79503602 1654 high scalability-2014-06-05-Cloud Architecture Revolution
19 0.79467607 1557 high scalability-2013-12-02-Evolution of Bazaarvoice’s Architecture to 500M Unique Users Per Month
20 0.79446977 1482 high scalability-2013-06-26-Leveraging Cloud Computing at Yelp - 102 Million Monthly Vistors and 39 Million Reviews