high_scalability high_scalability-2010 high_scalability-2010-903 knowledge-graph by maker-knowledge-mining
Source: html
Introduction: Disqus - Scaling the Worlds Largest Django App. Interesting overview of a commenting system with 75 million comments and 250 million visitors. Lots of good details on how they partition their database, testing, continuous integration, feature switches, caching, delayed signals, and more. Things I learnt tracking a billion events in 24 hours : Know your host, Scaling isn't just servers, My servers need to talk to me more, Kill switches for users, What you don't know is the problem, Don't mix server roles, Know your most important users outside of your site. Tweets of Gold: georgebarnett : I read High Scalability for useful articles about large scaling. Sadly though, nothing useful ever shows up. #NoLongerBothering northscale : wow that is fast! :) RT @cgoldberg: was just running > 100k ops/sec against my 2-node #Membase cluster... zazooom #nosql turbofunctor : The root of many (horizontal) scalability problems is an application level access to a writab
sentIndex sentText sentNum sentScore
1 Interesting overview of a commenting system with 75 million comments and 250 million visitors. [sent-2, score-0.329]
2 Lots of good details on how they partition their database, testing, continuous integration, feature switches, caching, delayed signals, and more. [sent-3, score-0.1]
3 Things I learnt tracking a billion events in 24 hours : Know your host, Scaling isn't just servers, My servers need to talk to me more, Kill switches for users, What you don't know is the problem, Don't mix server roles, Know your most important users outside of your site. [sent-4, score-0.262]
4 #NoLongerBothering northscale : wow that is fast! [sent-7, score-0.1]
5 zazooom #nosql turbofunctor : The root of many (horizontal) scalability problems is an application level access to a writable filesystem. [sent-11, score-0.243]
6 Distributed Systems: scalability and high availability . [sent-16, score-0.121]
7 A very nice slide deck by Renato Lucindo on Scalability, High Availability, Problems, and Tips & Tricks. [sent-17, score-0.171]
8 The site will have to support a massive amount of page views and users every day, on average 8 million a day for 1. [sent-21, score-0.29]
9 Some of their strategies: We proved our architecture before we built it, We cache a lot, We broke the page into personalised and standard components, We use loads of servers, We load tested the site before we launched. [sent-23, score-0.3]
10 Cool very safe for work cartoon on How Scalr Works . [sent-24, score-0.108]
11 That pesky network is a scarce, shared resource, and it often takes a network brownout to remind us that virtual machines are not all it takes to get everyone playing nice. [sent-27, score-0.324]
12 An existing approach for dealing with massive data sets is to stream over the input in few passes and perform computations with sublinear resources. [sent-29, score-0.388]
13 This method does not work for truly massive data where even making a single pass over the data with a processor is prohibitive. [sent-30, score-0.178]
14 One interesting pattern I've noticed popping up around my favorite new apps these days is that they follow what I'd call a "cloudtop" design Photoshop Scalability: Keeping It Simple . [sent-32, score-0.179]
15 Excellent interview on the issues around exploiting multiple processors in a real and complex application. [sent-33, score-0.085]
16 Photoshop's parallelism, born in the era of specialized expansion cards, has managed to scale well for the two- and four-core machines that have emerged over the past decade. [sent-34, score-0.371]
17 As Photoshop's engineers prepare for the eight- and 16-core machines that are coming, however, they have started to encounter more and more scaling problems, primarily a result of the effects of Amdahl's law and memory-bandwidth limitations. [sent-35, score-0.29]
wordName wordTfidf (topN-words)
[('photoshop', 0.286), ('massive', 0.178), ('switches', 0.133), ('batman', 0.129), ('learnt', 0.129), ('macvittie', 0.129), ('personalised', 0.129), ('unordered', 0.129), ('vogue', 0.129), ('iplayer', 0.122), ('sublinear', 0.122), ('anil', 0.122), ('lori', 0.122), ('writable', 0.122), ('scalability', 0.121), ('million', 0.112), ('sadly', 0.112), ('remind', 0.112), ('bbc', 0.108), ('cartoon', 0.108), ('disqus', 0.108), ('pesky', 0.108), ('commenting', 0.105), ('amdahl', 0.105), ('machines', 0.104), ('rt', 0.103), ('encounter', 0.103), ('wow', 0.1), ('popping', 0.1), ('emerged', 0.1), ('delayed', 0.1), ('scalr', 0.098), ('onhow', 0.092), ('simon', 0.092), ('reboot', 0.091), ('gfs', 0.091), ('proved', 0.089), ('born', 0.089), ('passes', 0.088), ('deck', 0.087), ('exploiting', 0.085), ('gold', 0.084), ('scarce', 0.084), ('slide', 0.084), ('prepare', 0.083), ('broke', 0.082), ('worlds', 0.08), ('noticed', 0.079), ('expansion', 0.078), ('django', 0.078)]
simIndex simValue blogId blogTitle
same-blog 1 0.99999994 903 high scalability-2010-09-17-Hot Scalability Links For Sep 17, 2010
Introduction: Disqus - Scaling the Worlds Largest Django App. Interesting overview of a commenting system with 75 million comments and 250 million visitors. Lots of good details on how they partition their database, testing, continuous integration, feature switches, caching, delayed signals, and more. Things I learnt tracking a billion events in 24 hours : Know your host, Scaling isn't just servers, My servers need to talk to me more, Kill switches for users, What you don't know is the problem, Don't mix server roles, Know your most important users outside of your site. Tweets of Gold: georgebarnett : I read High Scalability for useful articles about large scaling. Sadly though, nothing useful ever shows up. #NoLongerBothering northscale : wow that is fast! :) RT @cgoldberg: was just running > 100k ops/sec against my 2-node #Membase cluster... zazooom #nosql turbofunctor : The root of many (horizontal) scalability problems is an application level access to a writab
2 0.17406692 908 high scalability-2010-09-28-6 Strategies for Scaling BBC iPlayer
Introduction: The BBC's iPlayer site averages 8 million page views a day for 1.3 million users. Technical Architect Simon Frost describes how they scaled their site in Scaling the BBC iPlayer to handle demand : Use frameworks . Frameworks support component based development which makes it convenient for team development, but can introduce delays that have to be minimized. Zend/PHP is used because it supports components and is easy to recruit for. MySQL is used for program metadata. CouchDB is used for key-value access for fast read/write of user-focused data. Prove architecture before building it . Eliminate guesswork by coming up with alternate architectures and create prototypes to determine which option works best. Balance performance with factors like ease of development. Cache a lot . Data is cached in memcached for a few seconds to minutes. Short cache invalidation periods keep the data up to date for the users, but even these short periods make a huge difference in performance.
Introduction: Here's an Update On Disqus: It's Still About Realtime, But Go Demolishes Python . How do you add realtime functionality to a web scale application? That's what Adam Hitchcock , a Software Engineer at Disqus talks about in an excellent talk: Making DISQUS Realtime ( slides ). Disqus had to take their commenting system and add realtime capabilities to it. Not something that's easy to do when at the time of the talk (2013) they had had just hit a billion unique visitors a month. What Disqus developed is a realtime commenting system called “realertime” that was tested to handle 1.5 million concurrently connected users, 45,000 new connections per second, 165,000 messages/second, with less than .2 seconds latency end-to-end. The nature of a commenting system is that it is IO bound and has a high fanout, that is a comment comes in and must be sent out to a lot of readers. It's a problem very similar to what Twitter must solve . Disqus' solution was quite interesting as was th
4 0.12661257 928 high scalability-2010-10-26-Scaling DISQUS to 75 Million Comments and 17,000 RPS
Introduction: This presentation and video by Jason Yan and David Cramer discusses how they scaled DISQUS, a comments as a service service for easily adding comments to your site and connecting communities. The presentation is very good, so here are just a few highlights: Traffic : 17,000 requests/second peak; 450,000 websites; 15 million profiles; 75 million comments; 250 million visitors; 40 million monthly users / developer. Forces : unpredictable traffic patterns because of celebrity gossip and events like disasters; discussion never expire which means they can't fit in memory; must always be up. Machines : 100 servers; 30% web servers (Appache + mod_wsgi); 10% databases (PostgreSQL); 25% cache servers (memcached); 20% load balancing / high availability (HAProxy + heartbeat); 15% Utility servers (Python scripts). Architecture : Requests are load balanced across an Apache cluster. Apache talks to memcached, HAProxy/pgbouncer to handle connection pooling to the database, and a ce
5 0.1179364 448 high scalability-2008-11-22-Google Architecture
Introduction: Update 2: Sorting 1 PB with MapReduce . PB is not peanut-butter-and-jelly misspelled. It's 1 petabyte or 1000 terabytes or 1,000,000 gigabytes. It took six hours and two minutes to sort 1PB (10 trillion 100-byte records) on 4,000 computers and the results were replicated thrice on 48,000 disks. Update: Greg Linden points to a new Google article MapReduce: simplified data processing on large clusters . Some interesting stats: 100k MapReduce jobs are executed each day; more than 20 petabytes of data are processed per day; more than 10k MapReduce programs have been implemented; machines are dual processor with gigabit ethernet and 4-8 GB of memory. Google is the King of scalability. Everyone knows Google for their large, sophisticated, and fast searching, but they don't just shine in search. Their platform approach to building scalable applications allows them to roll out internet scale applications at an alarmingly high competition crushing rate. Their goal is always to build
6 0.11661697 906 high scalability-2010-09-22-Applying Scalability Patterns to Infrastructure Architecture
7 0.10910149 1644 high scalability-2014-05-07-Update on Disqus: It's Still About Realtime, But Go Demolishes Python
8 0.10785383 331 high scalability-2008-05-27-eBay Architecture
9 0.1058033 798 high scalability-2010-03-22-7 Secrets to Successfully Scaling with Scalr (on Amazon) by Sebastian Stadil
10 0.10577331 1228 high scalability-2012-04-16-Instagram Architecture Update: What’s new with Instagram?
11 0.10573119 638 high scalability-2009-06-26-PlentyOfFish Architecture
12 0.10557212 511 high scalability-2009-02-12-MySpace Architecture
13 0.10551485 1361 high scalability-2012-11-22-Gone Fishin': PlentyOfFish Architecture
14 0.10095193 1123 high scalability-2011-09-23-The Real News is Not that Facebook Serves Up 1 Trillion Pages a Month…
15 0.10073692 313 high scalability-2008-05-02-Friends for Sale Architecture - A 300 Million Page View-Month Facebook RoR App
17 0.09940201 750 high scalability-2009-12-16-Building Super Scalable Systems: Blade Runner Meets Autonomic Computing in the Ambient Cloud
18 0.098454878 954 high scalability-2010-12-06-What the heck are you actually using NoSQL for?
20 0.097452357 602 high scalability-2009-05-17-Scaling Django Web Apps by Mike Malone
topicId topicWeight
[(0, 0.199), (1, 0.087), (2, 0.01), (3, -0.032), (4, 0.012), (5, -0.019), (6, -0.046), (7, 0.02), (8, -0.018), (9, 0.036), (10, 0.012), (11, -0.034), (12, 0.002), (13, 0.019), (14, -0.033), (15, 0.003), (16, 0.035), (17, -0.048), (18, -0.004), (19, 0.049), (20, 0.048), (21, 0.002), (22, -0.05), (23, -0.054), (24, 0.019), (25, -0.015), (26, -0.019), (27, 0.081), (28, -0.007), (29, 0.039), (30, 0.053), (31, 0.015), (32, -0.02), (33, 0.005), (34, -0.034), (35, -0.008), (36, -0.031), (37, -0.029), (38, 0.005), (39, 0.057), (40, 0.015), (41, -0.043), (42, -0.077), (43, -0.006), (44, 0.035), (45, 0.015), (46, 0.024), (47, 0.012), (48, -0.055), (49, 0.011)]
simIndex simValue blogId blogTitle
same-blog 1 0.97019464 903 high scalability-2010-09-17-Hot Scalability Links For Sep 17, 2010
Introduction: Disqus - Scaling the Worlds Largest Django App. Interesting overview of a commenting system with 75 million comments and 250 million visitors. Lots of good details on how they partition their database, testing, continuous integration, feature switches, caching, delayed signals, and more. Things I learnt tracking a billion events in 24 hours : Know your host, Scaling isn't just servers, My servers need to talk to me more, Kill switches for users, What you don't know is the problem, Don't mix server roles, Know your most important users outside of your site. Tweets of Gold: georgebarnett : I read High Scalability for useful articles about large scaling. Sadly though, nothing useful ever shows up. #NoLongerBothering northscale : wow that is fast! :) RT @cgoldberg: was just running > 100k ops/sec against my 2-node #Membase cluster... zazooom #nosql turbofunctor : The root of many (horizontal) scalability problems is an application level access to a writab
2 0.76506621 1443 high scalability-2013-04-19-Stuff The Internet Says On Scalability For April 19, 2013
Introduction: Hey, it's HighScalability time: ( Ukrainian daredevil scaling buildings) Two Trillion Objects, 1.1 Million Requests / Second : S3; 1.4TB/s : Titan supercomputer has world’s fastest storage; four billion hours : Netflix streaming in last 3 months; $1.2B : Google's Q1 infrastructure spend Quotable Quotes: Google : We'll track EVERY task on EVERY data center server Stacey Higginbotham : All in all in the last five years the world has gained 54 Tbps of new capacity. @seveas : Scalability 103: Hardware sucks. Software sucks. Everything *will* break, prepare for failure of any component of your system. bloodredsun : The long and short of it is that Cassandra is a fantastic system for write heavy situations. What it is not good at are read heavy situations where deterministic low latency is required, which is pretty much what the pinterest guys were dealing with. @viktorklang : "The e-mail message could not be delivered because the user's
3 0.76185668 1178 high scalability-2012-01-20-Stuff The Internet Says On Scalability For January 20, 2012
Introduction: If you’ve got the time, we’ve got the HighScalability: Google+: 90 million users ; Internet 2011 : 2.1 billion Internet users, 1 trillion YouTube views, 5.9 billion mobile subscriptions; Fusion-io: One Billion IOPS ; 12 atoms: size of IBM's new memory bit ; 32 Million: Stack monthly visitors ; Gmail: 350 Million Users ; TimTebow: 1.5 million Tweets Quotable Quotes: Similarity : There is no canonical schema anymore. Instead you should ask: What high-volume queries will I need to serve with my data? Then work backwards from there. @kvirjee : Dis/Agree? -- "there is no problem but scalability, and architecture is its solution" @robpegoraro : Eternal vigilance can be crowdsourced. Didn't Bill Gates say once that 48 bits would always be enough for an ID? Well, Oracle ran out of bits: Fundamental Oracle flaw revealed . 64 bits, that's the ticket, ipv6 went 128 bits. The day Kodak died : We developed the world's first consumer digital camera bu
4 0.75990933 1057 high scalability-2011-06-10-Stuff The Internet Says On Scalability For June 10, 2011
Introduction: Submitted for your scaling pleasure: Achievements: Every day, Amazon Web Services adds enough new capacity to support all of Amazon.com’s global infrastructure through the company’s first 5 years, when it was $2.7 billion annual revenue. From Cloud Computing Is Driving Infrastructure Innovation by James Hamilton. Where's is all that money be spent? Facilities, servers, power, and popcorn. Evernote hits 10 million users . StackExchange hits 1 million users . No lawsuits expected in either case. Neural waves of brain . The brain's waves drive computation, sort of, in a 5 million core, 9 Hz computer. Scaling, Scaling, Scaled: textPlus Turns Two, Hits 10 Billion Messages Sent Milestone Quotes of a quotable essence: robinduckett : FACT: You are not a web developer if you need third party services which handle scalability so you can "focus on the programming". Twitter’s Bain : Facebook May Have More Scale, We Have More Engagement she
5 0.75030291 1315 high scalability-2012-08-30-Stuff The Internet Says On Scalability For August 31, 2012
Introduction: It's HighScalability Time: LHC compute jobs: use 1.5 CPU millennia every 3 days ; Obama helps load test Reddit: 4.3 million page views Quotable Quotes: @secastro : Want to see nearly a terabyte of memory? @DZone : Apache Projects are the Justice League of Scalability Apple And Google Might Be Negotiating Patents . Remember when empires and nation states would have a nice little summer war and then negotiate boundary lines and terms of trade? Google Faculty Summit 2012: The Online Revolution - Education at Scale . Someday we may have direct knowledge downloads and augmented wisdom packs, but until then these primitive attempts at learning process improvement are a good start. OnLive lost: how the paradise of streaming games was undone by one man's ego . The fascinating story behind a radical idea: applications hosted and rendered in the cloud while being displayed remotely on a device. You might have thought latency would be the killer
6 0.74150372 1484 high scalability-2013-06-28-Stuff The Internet Says On Scalability For June 28, 2013
7 0.73331255 425 high scalability-2008-10-22-Scalability Best Practices: Lessons from eBay
8 0.72603232 1378 high scalability-2012-12-28-Stuff The Internet Says On Scalability For December 28, 2012
9 0.72499561 1420 high scalability-2013-03-08-Stuff The Internet Says On Scalability For March 8, 2013
10 0.72363412 1024 high scalability-2011-04-15-Stuff The Internet Says On Scalability For April 15, 2011
11 0.72307086 1067 high scalability-2011-06-24-Stuff The Internet Says On Scalability For June 24, 2011
12 0.72306436 1166 high scalability-2011-12-30-Stuff The Internet Says On Scalability For December 30, 2011
13 0.71909904 331 high scalability-2008-05-27-eBay Architecture
15 0.71707553 142 high scalability-2007-11-05-Strategy: Diagonal Scaling - Don't Forget to Scale Out AND Up
16 0.71620154 1117 high scalability-2011-09-16-Stuff The Internet Says On Scalability For September 16, 2011
17 0.71450335 513 high scalability-2009-02-16-Handle 1 Billion Events Per Day Using a Memory Grid
18 0.71276176 992 high scalability-2011-02-18-Stuff The Internet Says On Scalability For February 18, 2011
19 0.71116322 1372 high scalability-2012-12-14-Stuff The Internet Says On Scalability For December 14, 2012
20 0.71073794 1645 high scalability-2014-05-09-Stuff The Internet Says On Scalability For May 9th, 2014
topicId topicWeight
[(1, 0.13), (2, 0.168), (11, 0.01), (28, 0.167), (30, 0.028), (40, 0.059), (52, 0.014), (61, 0.087), (77, 0.039), (79, 0.094), (85, 0.073), (94, 0.036)]
simIndex simValue blogId blogTitle
1 0.93276352 1294 high scalability-2012-08-01-Prismatic Update: Machine Learning on Documents and Users
Introduction: In update to Prismatic Architecture - Using Machine Learning on Social Networks to Figure Out What You Should Read on the Web , Jason Wolfe, even in the face of deadening fatigue from long nights spent getting their iPhone app out, has gallantly agreed to talk a little more about Primatic's approach to Machine Learning. Documents and users are two areas where Prismatic applies ML (machine learning): ML on Documents Given an HTML document:Â learn how to extract the main text of the page (rather than the sidebar, footer, comments, etc), its title, author, best images, etc determine features for relevance (e.g., what the article is about, topics, etc.) The setup for most of these tasks is pretty typical. Models are trained using big batch jobs on other machines that read data from s3, save the learned parameter files to s3, and then read (and periodically refresh) the models from s3 in the ingest pipeline. All of the data that flows out of the system can be
same-blog 2 0.91010863 903 high scalability-2010-09-17-Hot Scalability Links For Sep 17, 2010
Introduction: Disqus - Scaling the Worlds Largest Django App. Interesting overview of a commenting system with 75 million comments and 250 million visitors. Lots of good details on how they partition their database, testing, continuous integration, feature switches, caching, delayed signals, and more. Things I learnt tracking a billion events in 24 hours : Know your host, Scaling isn't just servers, My servers need to talk to me more, Kill switches for users, What you don't know is the problem, Don't mix server roles, Know your most important users outside of your site. Tweets of Gold: georgebarnett : I read High Scalability for useful articles about large scaling. Sadly though, nothing useful ever shows up. #NoLongerBothering northscale : wow that is fast! :) RT @cgoldberg: was just running > 100k ops/sec against my 2-node #Membase cluster... zazooom #nosql turbofunctor : The root of many (horizontal) scalability problems is an application level access to a writab
3 0.89718616 606 high scalability-2009-05-25-non-sequential, unique identifier, strategy question
Introduction: (Please bare with me, I'm a new, passionate, confident and terrified programmer :D ) Background: I'm pre-launch and 1 year into the development of my application. My target is to be able to eventually handle millions of registered users with 5-10% of them concurrent. Up to this point I've used auto-increment to assign unique identifiers to rows. I am now considering switching to a non-sequential strategy. Oh, I'm using the LAMP configuration. My reasons for avoiding auto-increment: 1. Complicates replication when scaling horizontally. Risk of collision is significant (when running multiple masters). Note: I've read the other entries in this forum that relate to ID generation and there have been some great suggestions -- including a strategy that uses auto-increment in a way that avoids this pitfall... That said, I'm still nervous about it. 2. Potential bottleneck when retrieving/assigning IDs -- IDs assigned at the database. My reasons for being nervous about
4 0.89442921 1506 high scalability-2013-08-23-Stuff The Internet Says On Scalability For August 23, 2013
Introduction: Hey, it's HighScalability time: ( Parkour is to terrain as programming is to frameworks ) 5x : AWS vs combined size of other cloud vendors; Every Second on The Internet : Why we need so many servers. Quotable Quotes: @chaliy : Today I learned that I do not understand how #azure scaling works, instance scale does not affect requests/sec I can load. @Lariar : Note how crazy this is. An international launch would have been a huge deal. Now it's just another thing you do. smacktoward : The problem with relying on donations is that people don't make donations. @toddhoffious : Programming is a tool built by logical positivists to solve the problems of idealists and pragmatists. We have a fundamental mismatch here. @etherealmind : Me: "Weird, my phone data isn't working" Them: "They turned the 3G off at the tower because it interferes with the particle accelerator" John Carmack : In computer science, just about t
Introduction: In Resolved: Widespread Application Outage , Heroku tells their story of how they dealt with the Amazon outage . While taking 100% responsibility for the downtime, they also shared a number of the strategies they used to bring their service back to full working order. One of Heroku's most interesting strategies wasn't a technical hack at all, but how they consciously went about deploying their Ops personnel in response to the emergency. An outline of their strategy is: Monitoring systems immediately alerted Ops to the problem. An on-call engineer applied triage logic to the problem and classified it as serious, which caused the on-call Incident Commander to be woken out of restful slumber. The IC contacted AWS . They were in constant contact with their AWS representative and worked closely with AWS to solve problems. The IC alerted Heroku engineers. A full crew: support, data, and other engineering teams worked around the clock to bring every
6 0.86822832 806 high scalability-2010-04-08-Hot Scalability Links for April 8, 2010
7 0.85141695 1395 high scalability-2013-01-28-DuckDuckGo Architecture - 1 Million Deep Searches a Day and Growing
8 0.84170294 1261 high scalability-2012-06-08-Stuff The Internet Says On Scalability For June 8, 2012
9 0.83385861 1611 high scalability-2014-03-12-Paper: Scalable Eventually Consistent Counters over Unreliable Networks
10 0.82531744 304 high scalability-2008-04-19-How to build a real-time analytics system?
12 0.82146394 1439 high scalability-2013-04-12-Stuff The Internet Says On Scalability For April 12, 2013
13 0.81935692 1020 high scalability-2011-04-12-Caching and Processing 2TB Mozilla Crash Reports in memory with Hazelcast
14 0.81788611 998 high scalability-2011-03-03-Stack Overflow Architecture Update - Now at 95 Million Page Views a Month
15 0.81699187 630 high scalability-2009-06-14-kngine 'Knowledge Engine' milestone 2
16 0.81648195 840 high scalability-2010-06-10-The Four Meta Secrets of Scaling at Facebook
17 0.81530648 106 high scalability-2007-10-02-Secrets to Fotolog's Scaling Success
18 0.81409597 1389 high scalability-2013-01-18-Stuff The Internet Says On Scalability For January 18, 2013
19 0.81358147 1559 high scalability-2013-12-06-Stuff The Internet Says On Scalability For December 6th, 2013
20 0.81214875 768 high scalability-2010-02-01-What Will Kill the Cloud?