high_scalability high_scalability-2009 high_scalability-2009-680 knowledge-graph by maker-knowledge-mining
Source: html
Introduction: One of the top recommendations from the collective wisdom contained in Real Life Architectures is to add monitoring to your system. Now! Loud is the lament for not adding monitoring early and often. The reason is easy to understand. Without monitoring you don't know what your system is doing which means you can't fix it and you can't improve it. Feedback loops require data. Some popular monitor options are Munin, Nagios, Cacti and Hyperic. A relatively new entrant is a product called Reconnoiter from Theo Schlossnagle, President and CEO of OmniTI, leading consultants on solving problems of scalability, performance, architecture, infrastructure, and data management. Theo's name might sound familiar. He gives lots of talks and is the author of the very influential Scalable Internet Architectures book. So right away you know Reconnoiter has a good pedigree. As Theo says, their products are born of pain, from the fire of solving real-life problems and that's always a harbinger of
sentIndex sentText sentNum sentScore
1 One of the top recommendations from the collective wisdom contained in Real Life Architectures is to add monitoring to your system. [sent-1, score-0.232]
2 Loud is the lament for not adding monitoring early and often. [sent-3, score-0.173]
3 Without monitoring you don't know what your system is doing which means you can't fix it and you can't improve it. [sent-5, score-0.232]
4 A relatively new entrant is a product called Reconnoiter from Theo Schlossnagle, President and CEO of OmniTI, leading consultants on solving problems of scalability, performance, architecture, infrastructure, and data management. [sent-8, score-0.208]
5 As Theo says, their products are born of pain, from the fire of solving real-life problems and that's always a harbinger of good things to come. [sent-12, score-0.128]
6 The problem Reconnoiter is trying to solve is monitoring thousands of nodes across many datacenters where the nodes can vary widely in power, architecture, and software configuration. [sent-13, score-0.173]
7 Cheap checks that are made on the specified time interval and aren't late and don't cause a heavy load on the machine. [sent-15, score-0.158]
8 Separate data collection from visualization and fault-detection. [sent-18, score-0.066]
9 Analyze trends for long-term capacity planning and postmortem analysis. [sent-19, score-0.137]
10 Support trending: the intelligent data correlation, regression analysis/curve fitting and looking into the past to see how much you go where you are now so you can do better next time. [sent-21, score-0.133]
11 Create a monitoring system that doesn't require a separate powerful network and its own set of hosts on which to run. [sent-22, score-0.232]
12 If you've ever used or written a distributed stats collection system the architecture of Reconnoiter will look somewhat familiar: Some of the more interesting bits of the architecture are: PostgresSQL stores all the data. [sent-23, score-0.317]
13 It's not clear how well this approach will work but the hooks are there. [sent-26, score-0.063]
14 Data is not thrown away as it is with some systems so you can check against history. [sent-35, score-0.058]
15 There are a lot of separate steps and bits to configure. [sent-39, score-0.078]
16 Another potential problem is monitoring produces a lot of real-time data. [sent-40, score-0.173]
17 And I wasn't sure if graph data could be correlated with released features or other system changes. [sent-43, score-0.186]
18 In the video Theo mentions seeing in the graphs that using deflate improved performance, but I'm not sure just looking at the graph how you would be able correlate system data with system changes. [sent-44, score-0.478]
19 It's droolingly clear where Reconnoiter shines is on creating complex graphs, charts, and other visualizations. [sent-45, score-0.078]
20 The real time visualizations are spectacular and extremely are difficult to do in other systems. [sent-47, score-0.138]
wordName wordTfidf (topN-words)
[('reconnoiter', 0.645), ('theo', 0.263), ('monitoring', 0.173), ('postgressql', 0.167), ('nagios', 0.097), ('checks', 0.097), ('graphs', 0.094), ('pain', 0.089), ('loud', 0.083), ('stansberry', 0.083), ('toolsby', 0.083), ('analysisby', 0.078), ('deflate', 0.078), ('shines', 0.078), ('unveils', 0.078), ('bits', 0.078), ('funky', 0.075), ('grig', 0.075), ('whirlwind', 0.075), ('spectacular', 0.072), ('correlate', 0.072), ('entrant', 0.072), ('solving', 0.07), ('fitting', 0.07), ('planning', 0.069), ('correlated', 0.068), ('postmortem', 0.068), ('collection', 0.066), ('visualizations', 0.066), ('consultants', 0.066), ('sad', 0.065), ('granular', 0.063), ('hooks', 0.063), ('regression', 0.063), ('lua', 0.063), ('cacti', 0.062), ('munin', 0.061), ('interval', 0.061), ('system', 0.059), ('charts', 0.059), ('correlation', 0.059), ('collective', 0.059), ('graph', 0.059), ('allspaw', 0.058), ('securely', 0.058), ('thrown', 0.058), ('born', 0.058), ('architecture', 0.057), ('mentions', 0.057), ('transferred', 0.057)]
simIndex simValue blogId blogTitle
same-blog 1 0.99999994 680 high scalability-2009-08-13-Reconnoiter - Large-Scale Trending and Fault-Detection
Introduction: One of the top recommendations from the collective wisdom contained in Real Life Architectures is to add monitoring to your system. Now! Loud is the lament for not adding monitoring early and often. The reason is easy to understand. Without monitoring you don't know what your system is doing which means you can't fix it and you can't improve it. Feedback loops require data. Some popular monitor options are Munin, Nagios, Cacti and Hyperic. A relatively new entrant is a product called Reconnoiter from Theo Schlossnagle, President and CEO of OmniTI, leading consultants on solving problems of scalability, performance, architecture, infrastructure, and data management. Theo's name might sound familiar. He gives lots of talks and is the author of the very influential Scalable Internet Architectures book. So right away you know Reconnoiter has a good pedigree. As Theo says, their products are born of pain, from the fire of solving real-life problems and that's always a harbinger of
2 0.1452954 87 high scalability-2007-09-10-Blog: Esoteric Curio by Theo Schlossnagle
Introduction: Theo Schlossnagle is the author of Scalable Internet Architecture and the funder of OmniTI , a global leader in Internet technology services that power the World Wide Web and email. As you might imagine Theo frequently posts on interesting topics for the scalable website builder. A Quick Hit of What's Inside Partitioning vs. Federation vs. Sharding , PostgreSQL warm standby on ZFS crack , Scalability vs. Performance: it isn't a battle  Site: http://lethargy.org/~jesus
Introduction: Update 2: Velocity 09: John Allspaw, 10+ Deploys Per Day: Dev and Ops Cooperation at Flickr . Insightful talk. Some highlights: Change is good if you can build tools and culture to lower the risk of change. Operations and developers need to become of one mind and respect each other. An automated infrastructure is the one tool you need most. Common source control. One step build. One step deploy. Don't be a pussy, deploy. Always ship trunk. Feature flags - don't branch code, make features runtime configurable in code. Dark launch - release data paths early without UI component. Shared metrics. Adaptive feedback to prioritize important features. IRC for communication for human context. Best solutions occur when dev and op work together and trust each other. Trust is earned by helping each other solve their problems. Look at what new features imply for operations, what can go wrong, and how to recover. Provide knobs and levers to help operations. Devs should have access to production
4 0.11353626 89 high scalability-2007-09-10-Is there a difference between partitioning and federation and sharding?
Introduction: Unlike Theo Schlossnagle, author of Scalable Internet Architectures , I am not a stickler for semantics because I have an unswerving faith in the ultimate unknowability of the world as experienced by others. That's why it is Theo who bravely tackles the differences in his informative blog post Partitioning vs. Federation vs. Sharding . Royans Tharakan also talks about it on his blog . Is there a difference and does it really matter to all our intrepid scalable website builders? Generally whatever Theo says is probably close to the truth. Yet, in my mind I think of partitioning as a basic level category and federation and sharding as more specific (subordinate) instances of partitioning. And partitioning is a more specific instance of the more more general (superordinate) category divide-and-conquer. Which isn't a useful way to think about the topic at all. So, let's say federation is like Star Trek. The Vulcans, Klingons, and Humans live in very separate p
Introduction: Who's Hiring? Anybody interested in helping manage a 100+ Linux server deployment? Next Big Sound is a an analytics company for the music industry and is looking someone to help them scale. Red 5 Studios. Wanted: DBAs and Programmers interested in MySQL scalability and replication. If interested, please see us here . Fun and Informative Events Sign up for this free 30-minute webinar exploring how new technology can determine which ads have been seen by users and will discuss the C3 Metrics Labs analysis of over 2 billion impressions. Cool Products and Services aiCache creates a better user experience by increasing the speed scale and stability of your web-site. ElasticHosts award winning cloud server hosting launches across North America. Adding data centers in Los Angeles and Toronto. Free trial . LogicMonitor - Hosted monitoring of your entire technology stack. Dashboards, trending graphs, alerting. Try it free and be up and r
13 0.096602134 1047 high scalability-2011-05-25-Stuff to Watch from Surge 2010
topicId topicWeight
[(0, 0.176), (1, 0.012), (2, -0.006), (3, -0.002), (4, 0.009), (5, 0.003), (6, 0.008), (7, 0.028), (8, 0.023), (9, 0.033), (10, -0.026), (11, 0.036), (12, -0.004), (13, -0.016), (14, 0.023), (15, -0.017), (16, 0.041), (17, 0.063), (18, -0.007), (19, 0.025), (20, -0.03), (21, -0.017), (22, -0.022), (23, -0.002), (24, -0.049), (25, 0.047), (26, -0.001), (27, 0.056), (28, -0.024), (29, 0.011), (30, -0.012), (31, -0.013), (32, 0.039), (33, 0.01), (34, 0.005), (35, 0.02), (36, -0.046), (37, -0.031), (38, -0.032), (39, 0.027), (40, -0.054), (41, -0.005), (42, 0.033), (43, 0.017), (44, 0.041), (45, 0.025), (46, 0.008), (47, -0.037), (48, 0.002), (49, -0.002)]
simIndex simValue blogId blogTitle
same-blog 1 0.9447152 680 high scalability-2009-08-13-Reconnoiter - Large-Scale Trending and Fault-Detection
Introduction: One of the top recommendations from the collective wisdom contained in Real Life Architectures is to add monitoring to your system. Now! Loud is the lament for not adding monitoring early and often. The reason is easy to understand. Without monitoring you don't know what your system is doing which means you can't fix it and you can't improve it. Feedback loops require data. Some popular monitor options are Munin, Nagios, Cacti and Hyperic. A relatively new entrant is a product called Reconnoiter from Theo Schlossnagle, President and CEO of OmniTI, leading consultants on solving problems of scalability, performance, architecture, infrastructure, and data management. Theo's name might sound familiar. He gives lots of talks and is the author of the very influential Scalable Internet Architectures book. So right away you know Reconnoiter has a good pedigree. As Theo says, their products are born of pain, from the fire of solving real-life problems and that's always a harbinger of
2 0.71314424 1506 high scalability-2013-08-23-Stuff The Internet Says On Scalability For August 23, 2013
Introduction: Hey, it's HighScalability time: ( Parkour is to terrain as programming is to frameworks ) 5x : AWS vs combined size of other cloud vendors; Every Second on The Internet : Why we need so many servers. Quotable Quotes: @chaliy : Today I learned that I do not understand how #azure scaling works, instance scale does not affect requests/sec I can load. @Lariar : Note how crazy this is. An international launch would have been a huge deal. Now it's just another thing you do. smacktoward : The problem with relying on donations is that people don't make donations. @toddhoffious : Programming is a tool built by logical positivists to solve the problems of idealists and pragmatists. We have a fundamental mismatch here. @etherealmind : Me: "Weird, my phone data isn't working" Them: "They turned the 3G off at the tower because it interferes with the particle accelerator" John Carmack : In computer science, just about t
3 0.69517487 1283 high scalability-2012-07-13-Stuff The Internet Says On Scalability For July 13, 2012
Introduction: It's HighScalability Time (Good luck today): A Friday the 13th Postmorterama: James Hamilton with some high powered perspective on the report for the Fukushima Nuclear Accident. Apparently they haven't heard of the blameless post-mortem. Lots of interesting stuff, but this is a potentially disaster saving general lesson learned: operators can’t figure out what is happening or take appropriate action without detailed visibility into the state of the system. Evernote with a nicely detailed note on a recent outage . A kernel panic happened while upgrading two new “shard” servers with 3x as much RAM, SSDs instead of 15krpm disks, bonded networking, and an updated kernel. They had to revert and shite loves to happen when other shite happens. Heroku with their postmortem on what happened when AWS went down. They lost 30% of their instances across 3 AZs in the US-East region. Rich detail on the impact of the AWS, but not much on what they can do about it
4 0.69462061 719 high scalability-2009-10-09-Have you collectl'd yet? If not, maybe collectl-utils will make it easier to do so
Introduction: I'm not sure how many people who follow this have even tried collectl but I wanted to let you all know that I just released a set of utilities called strangely enough collectl-utils, which you can get at http://collectl-utils.sourceforge.net . One web-based utility called colplot gives you the ability to very easily plot data from multiple systems in a way that makes correlating them over time very easy. Another utility called colmux lets you look at multiple systems in real time. In fact if you go the page that describes it in more detail you'll see a photo which shows the CPU loads on 192 systems one a second, one set of data/line! in fact the display so wide it takes 3 large monitors side-by-side to see it all and even though you can't actually read the displays you can easily see which systems are loaded and which aren't. Anyhow give it a look and let me know what you think. -mark
5 0.69165045 219 high scalability-2008-01-21-Product: Hyperic
Introduction: From Wikipedia : Hyperic HQ is a popular open source IT Operations computer system and network monitoring application software. It auto-discovers all system resources and their metrics, including hardware, operating systems, virtualization, databases, middleware, applications, and services. It watches hosts and services that you specify, alerting you when things go bad and again when they get better. It also provides historical charting and event correlation for faster problem identification. The Hyperic HQ server is a distributed J2EE application that runs on top of the open source JBoss Application Server. It is written in Java and portable C code and runs on Linux, Windows, Solaris, HP-UX and Mac OS X. Hyperic HQ Portal is a Java and AJAX User Interface that includes: Inventory/Application Model & host hierarchy Monitoring of network services (SMTP, POP3, HTTP, NNTP, ICMP, SNMP) Monitoring of host resources (processor load, disk usage, system logs) Remote mon
6 0.69049078 558 high scalability-2009-04-06-How do you monitor the performance of your cluster?
7 0.6886726 1520 high scalability-2013-09-20-Stuff The Internet Says On Scalability For September 20, 2013
8 0.68347055 344 high scalability-2008-06-09-FaceStat's Rousing Tale of Scaling Woe and Wisdom Won
9 0.68153292 1225 high scalability-2012-04-09-Why My Slime Mold is Better than Your Hadoop Cluster
11 0.67950845 1468 high scalability-2013-05-31-Stuff The Internet Says On Scalability For May 31, 2013
12 0.67947662 1559 high scalability-2013-12-06-Stuff The Internet Says On Scalability For December 6th, 2013
13 0.67469329 237 high scalability-2008-02-03-Product: Collectl - Performance Data Collector
14 0.66976261 1263 high scalability-2012-06-13-Why My Soap Film is Better than Your Hadoop Cluster
15 0.66558194 994 high scalability-2011-02-23-This stuff isn't taught, you learn it bit by bit as you solve each problem.
16 0.66452891 1230 high scalability-2012-04-18-Ansible - A Simple Model-Driven Configuration Management and Command Execution Framework
17 0.66180778 1460 high scalability-2013-05-17-Stuff The Internet Says On Scalability For May 17, 2013
18 0.65768844 1607 high scalability-2014-03-07-Stuff The Internet Says On Scalability For March 7th, 2014
19 0.65730572 1484 high scalability-2013-06-28-Stuff The Internet Says On Scalability For June 28, 2013
20 0.65704656 1422 high scalability-2013-03-12-If Your System was a Symphony it Might Sound Like This...
topicId topicWeight
[(1, 0.139), (2, 0.139), (10, 0.022), (26, 0.015), (30, 0.026), (61, 0.043), (69, 0.01), (79, 0.496), (94, 0.031)]
simIndex simValue blogId blogTitle
Introduction: Update : as of the end of Q2 2011, Akamai had 95,811 servers deployed globally. Akamai is the CDN to the stars. It claims to deliver between 15 and 30 percent of all Web traffic, with major customers like Facebook, Twitter, Apple, and the US military. Traditionally quite secretive, we get a peek behind the curtain in this paper: The Akamai Network: A Platform for High-Performance Internet Applications by Erik Nygren, Ramesh Sitaraman, and Jennifer Sun. Abstract: Comprising more than 61,000 servers located across nearly 1,000 networks in 70 countries worldwide, the Akamai platform delivers hundreds of billions of Internet interactions daily, helping thousands of enterprises boost the performance and reliability of their Internet applications. In this paper, we give an overview of the components and capabilities of this large-scale distributed computing platform, and offer some insight into its architecture, design principles, operation, and management. Delivering a
2 0.99466747 1277 high scalability-2012-07-05-10 Golden Principles For Building Successful Mobile-Web Applications
Introduction: Wildly popular VC blogger Fred Wilson defines in an excellent 27 minute video the ten most important criteria he uses when deciding to give the gold, that is, fund a web application. Note, this video is from 2010 , so no doubt the ideas are still valid, but the importance of mobile vs web apps has probably shifted to mobile, as Mr. Wilson says in a recent post: mobile is growing like a weed . Speed - speed is more than a feature, it's a requirement. Mainstream users are unforgiving. If something is slow they won't use it. Pingdom is used to track speed across their portfolio. A trend they've noticed is that as an application slows down they don't grow as quickly. Instant Utility - a service must be instantly useful to users. Lengthy setup and configuration is a killer. Tricks like crawling the web to populate information you expect to get from your users later makes the service initially useful. YouTube won, for example, with instant availability of uploaded video.
3 0.9913429 107 high scalability-2007-10-02-Some Real Financial Numbers for Your Startup
Introduction: If you are a startup you may find useful Guy Kawasaki's post Financial Models for Underachievers: Two Years of the Real Numbers of a Startup . Part of any business plan are the projected guestimates. They are guestimates because everyone keeps these numbers hidden like a Swiss bank account. But not Redfin . They've bravely shared their initial cost projections, their actual numbers from real life, and the lessons they've learned from the discrepancy between the two... You can find their model estimates and actuals for Rent, Per Employee, Per Month (model: $250, actual: $336); Initial Per-Employee Equipment Cost; Monthly Benefits, Per-Employee; Annual Payroll Tax; Quarterly Bonus Payout, as a % of the Total Possible; Annual Payroll Increase for Existing Employees; All-Company Meeting Cost, Per-Meeting, Per-Employee; Annual Accounting Costs, and a few more. There is also a great lessons section: Focus on headcount; Plan slow, run fast; Run top-down sanity-checks; Fo
4 0.98885399 784 high scalability-2010-02-25-Paper: High Performance Scalable Data Stores
Introduction: The world of scalable databases is not a simple one. They come in every race, creed, and color. Rick Cattell has brought some harmony to that world by publishing High Performance Scalable Data Stores , a nicely detailed one stop shop paper comparing scalable databases soley on the content of their character. Ironically, the first step in that evaluation is dividing the world into four groups: Key-value stores: Redis, Scalaris, Voldmort, and Riak. Document stores: Couch DB, MongoDB, and SimpleDB. Record stores: BigTable, HBase, HyperTable, and Cassandra. Scalable RDBMSs: MySQL Cluster, ScaleDB, Drizzle, and VoltDB. The paper describes each system and then compares them on the dimensions of Concurrency Control, Data Storage Replication, Transaction Model, General Comments, Maturity, K-hits, License Language. And the winner is: there are no winners. Yet. Rick concludes by pointing to a great convergence: I believe that a few of these systems will gain critical mass an
5 0.98865372 1169 high scalability-2012-01-05-Shutterfly Saw a Speedup of 500% With Flashcache
Introduction: In the "should I or shouldn't I" debate around deploying SSD, it always helps to have real-world data. Fiesta! with a live-blog summary of a presentation by Kenny Gorman on Shutterfly on MongoDB Performance Tuning . What if you still need more performance after doing all of this tuning? One option is to use SSDs. Shutterfly uses Facebook’s flashcache : kernel module to cache data on SSD. Designed for MySQL/InnoDB. SSD in front of a disk, but exposed as a single mount point. This only makes sense when you have lots of physical I/O. Shutterfly saw a speedup of 500% w/ flashcache. A benefit is that you can delay sharding: less complexity. The whole series of posts has a lot of great information and is worth a longer look, especially if you are considering using MongoDB. Related Articles Slides for MongoSF 2011 slides: MongoDB Performance Tuning SSD+HDD sharding setup for large and permanently growing collections Imlementing MongoDB at Shutterfly by Kenny
6 0.98484135 372 high scalability-2008-08-27-Updating distributed web applications
7 0.97908574 443 high scalability-2008-11-14-Paper: Pig Latin: A Not-So-Foreign Language for Data Processing
8 0.97667992 323 high scalability-2008-05-19-Twitter as a scalability case study
9 0.97435892 782 high scalability-2010-02-23-When to migrate your database?
10 0.96602792 871 high scalability-2010-08-04-Dremel: Interactive Analysis of Web-Scale Datasets - Data as a Programming Paradigm
11 0.96454483 743 high scalability-2009-11-23-Big Data on Grids or on Clouds?
same-blog 12 0.95540196 680 high scalability-2009-08-13-Reconnoiter - Large-Scale Trending and Fault-Detection
13 0.95160961 1403 high scalability-2013-02-08-Stuff The Internet Says On Scalability For February 8, 2013
14 0.95004714 8 high scalability-2007-07-12-Should I use LAMP or Windows?
15 0.94886011 401 high scalability-2008-10-04-Is MapReduce going mainstream?
16 0.9413473 650 high scalability-2009-07-02-Product: Hbase
17 0.93729806 1420 high scalability-2013-03-08-Stuff The Internet Says On Scalability For March 8, 2013
18 0.93381339 1494 high scalability-2013-07-19-Stuff The Internet Says On Scalability For July 19, 2013
19 0.93118232 448 high scalability-2008-11-22-Google Architecture
20 0.92300248 1162 high scalability-2011-12-23-Funny: A Cautionary Tale About Storage and Backup