high_scalability high_scalability-2009 high_scalability-2009-682 knowledge-graph by maker-knowledge-mining
Source: html
Introduction: ThePort Network's Director of Engineering, TJ Muehleman was kind of enough to share some of the architectural details for their white label social media system. It currently runs about 50 social networks varying in size from less than 1000 members to more than 300,000 members, all on a Microsoft stack. In addition to their social networking platform, they offer Javascript APIs and web service APIs (both REST and SOAP) which account for a significant percentage of overall system usage. ThePort is an excellent example of a real world in-the-trenches product offering real value to customers. One of the most interesting problems they have to solve is multi-tenancy. How do you provide good performance, complete customization, support, develop new features, and provide individual search indexes for each customer? It's not an easy problem to solve. How did they solve their problems and build a successful system? Site: http://theport.com Platform Microsoft.NET 3.5 C# / VB.NET
sentIndex sentText sentNum sentScore
1 In addition to their social networking platform, they offer Javascript APIs and web service APIs (both REST and SOAP) which account for a significant percentage of overall system usage. [sent-3, score-0.231]
2 NET SQL Server 2005 Visual Studio 2008 Pro Edition Prototype Subversion TortoiseSVN Trac (for internal defect tracking. [sent-13, score-0.159]
3 We initially looked at MemCacheD but our internal benchmarking indicated SharedCache had better performance – at least w/in a Microsoft environment. [sent-19, score-0.218]
4 NET's wildcard search was a little buggy in our initial beta testing. [sent-22, score-0.23]
5 Getting a new machine (2 quad cores with 16 GB of RAM) helped reduce the latency to less than a second. [sent-32, score-0.179]
6 Allowing CSS control isn't enough; we needed a templating system that allows total control over the site. [sent-38, score-0.394]
7 We built a proprietary templating system that unfortunately became too limiting and would one day lead to a drag on performance. [sent-43, score-0.453]
8 This made XSLT a natural choice from the templating angle. [sent-46, score-0.307]
9 XSLT as Custom Templating System Building a templating system in XSLT that actually allows the template author to make a web service call to our internal web service layer (or external web services) straight from the templating system. [sent-50, score-0.988]
10 What we've found in our internal testing is that these extension objects scale way better than our previous templating system (a homegrown proprietary system). [sent-53, score-0.841]
11 For now, we make the internal web services calls via HTTP, but we will soon be moving this to a TCP call to take advantage of the better connection pooling offered by TCP. [sent-56, score-0.219]
12 For instance, if we needed a collection of comments, previously we'd hit the database for the 5, 10, 100, etc comments we wanted, do the sorting / filtering in the DB, return a single dataset, cache that, and then display. [sent-60, score-0.17]
13 What we've started doing recently is caching the recent comment objects, and using our cache providers MultiGet ability to simultaneously retrieve all comments at the same time. [sent-62, score-0.235]
14 Given the growth of our templating system mentioned above, we realized it was best to truly separate the tiers into discrete areas. [sent-69, score-0.478]
15 Since our application is easily accessed via a set of REST APIs and our own internal skinning system (and who knows what in the future), dividing the application like this gives us a lot of leeway in being able to swap out components. [sent-70, score-0.246]
16 We think this system will work for the near to mid term but long term, we'll take advantage of a queuing system to keep the index in sync. [sent-78, score-0.373]
17 1 platform architect responsible for overall system architecture (selecting which systems to use, tuning them), 1 lead software architect, and 3 senior – mid level developers. [sent-81, score-0.176]
18 Since we're a start-up in a fast evolving market (social media) we find that we're constantly having to adjust to market demands and the latest in social functionality. [sent-82, score-0.238]
19 Internally we use Trac for bug tracking and devote roughly 20% of our week maintaining, supporting, and fixing issues. [sent-98, score-0.176]
20 Our next step will be to set up TCP and MSMQ bindings with WCF to handle our internal service requests and queuing, respectively. [sent-109, score-0.306]
wordName wordTfidf (topN-words)
[('templating', 0.307), ('xslt', 0.213), ('dell', 0.186), ('wcf', 0.176), ('lucene', 0.168), ('gb', 0.162), ('msmq', 0.16), ('internal', 0.159), ('sharedcache', 0.117), ('tj', 0.117), ('queuing', 0.11), ('comments', 0.109), ('quad', 0.101), ('buggy', 0.092), ('mid', 0.089), ('bug', 0.088), ('fixing', 0.088), ('ram', 0.087), ('system', 0.087), ('apis', 0.085), ('found', 0.085), ('discrete', 0.084), ('bindings', 0.083), ('social', 0.08), ('market', 0.079), ('helped', 0.078), ('sql', 0.076), ('objects', 0.076), ('additionally', 0.072), ('move', 0.071), ('search', 0.071), ('considerable', 0.071), ('processors', 0.07), ('tier', 0.07), ('tcp', 0.07), ('accomplish', 0.068), ('extension', 0.068), ('beta', 0.067), ('blade', 0.067), ('overcome', 0.065), ('retrieve', 0.065), ('solr', 0.064), ('service', 0.064), ('db', 0.063), ('configurable', 0.062), ('cache', 0.061), ('calls', 0.06), ('proprietary', 0.059), ('looked', 0.059), ('members', 0.058)]
simIndex simValue blogId blogTitle
same-blog 1 1.0000002 682 high scalability-2009-08-16-ThePort Network Architecture
Introduction: ThePort Network's Director of Engineering, TJ Muehleman was kind of enough to share some of the architectural details for their white label social media system. It currently runs about 50 social networks varying in size from less than 1000 members to more than 300,000 members, all on a Microsoft stack. In addition to their social networking platform, they offer Javascript APIs and web service APIs (both REST and SOAP) which account for a significant percentage of overall system usage. ThePort is an excellent example of a real world in-the-trenches product offering real value to customers. One of the most interesting problems they have to solve is multi-tenancy. How do you provide good performance, complete customization, support, develop new features, and provide individual search indexes for each customer? It's not an easy problem to solve. How did they solve their problems and build a successful system? Site: http://theport.com Platform Microsoft.NET 3.5 C# / VB.NET
2 0.16507751 195 high scalability-2007-12-28-Amazon's EC2: Pay as You Grow Could Cut Your Costs in Half
Introduction: Update 2: Summize Computes Computing Resources for a Startup . Lots of nice graphs showing Amazon is hard to beat for small machines and become less cost efficient for well used larger machines. Long term storage costs may eat your saving away. And out of cloud bandwidth costs are high. Update: via ProductionScale , a nice Digital Web article on how to setup S3 to store media files and how Blue Origin was able to handle 3.5 million requests and 758 GBs in bandwidth in a single day for very little $$$. Also a Right Scale article on Network performance within Amazon EC2 and to Amazon S3 . 75MB/s between EC2 instances, 10.2MB/s between EC2 and S3 for download, 6.9MB/s upload. Now that Amazon's S3 (storage service) is out of beta and EC2 (elastic compute cloud) has added new instance types (the class of machine you can rent) with more CPU and more RAM, I thought it would be interesting to take a look out how their pricing stacks up. The quick conclusion: the m
3 0.16442434 1068 high scalability-2011-06-27-TripAdvisor Architecture - 40M Visitors, 200M Dynamic Page Views, 30TB Data
Introduction: This is a guest post by Andy Gelfond , VP of Engineering for TripAdvisor. Andy has been with TripAdvisor for six and a half years, wrote a lot of code in the earlier days, and has been building and running a first class engineering and operations team that is responsible for the worlds largest travel site. There's an update for this article at An Epic TripAdvisor Update: Why Not Run On The Cloud? The Grand Experiment . For TripAdvisor , scalability is woven into our organization on many levels - data center, software architecture, development/deployment/operations, and, most importantly, within the culture and organization. It is not enough to have a scalable data center, or a scalable software architecture. The process of designing, coding, testing, and deploying code also needs to be scalable. All of this starts with hiring and a culture and an organization that values and supports a distributed, fast, and effective development and operation of a complex and highly scalable co
Introduction: All in all this is still my favorite post and I still think it's an accurate vision of a future. Not everyone agrees, but I guess we'll see..."But it is not complicated. [There's] just a lot of it." \--Richard Feynmanon how the immense variety of the world arises from simple rules.Contents:Have We Reached the End of Scaling?Applications Become Black Boxes Using Markets to Scale and Control CostsLet's Welcome our Neo-Feudal OverlordsThe Economic Argument for the Ambient CloudWhat Will Kill the Cloud?The Amazing Collective Compute Power of the Ambient CloudUsing the Ambient Cloud as an Application RuntimeApplications as Virtual StatesConclusionWe have not yet begun to scale. The world is still fundamentally disconnected and for all our wisdom we are still in the earliest days of learning how to build truly large planet-scaling applications.Today 350 million users on Facebook is a lot of users and five million followers on Twitter is a lot of followers. This may seem like a lot now, but c
Introduction: "But it is not complicated. [There's] just a lot of it." \--Richard Feynmanon how the immense variety of the world arises from simple rules.Contents:Have We Reached the End of Scaling?Applications Become Black Boxes Using Markets to Scale and Control CostsLet's Welcome our Neo-Feudal OverlordsThe Economic Argument for the Ambient CloudWhat Will Kill the Cloud?The Amazing Collective Compute Power of the Ambient CloudUsing the Ambient Cloud as an Application RuntimeApplications as Virtual StatesConclusionWe have not yet begun to scale. The world is still fundamentally disconnected and for all our wisdom we are still in the earliest days of learning how to build truly large planet-scaling applications.Today 350 million users on Facebook is a lot of users and five million followers on Twitter is a lot of followers. This may seem like a lot now, but consider we have no planet wide applications yet. None.Tomorrow the numbers foreshadow a newCambrian explosionof connectivity that will look as
6 0.15733154 1240 high scalability-2012-05-07-Startups are Creating a New System of the World for IT
7 0.15381216 1501 high scalability-2013-08-13-In Memoriam: Lavabit Architecture - Creating a Scalable Email Service
8 0.15276015 96 high scalability-2007-09-18-Amazon Architecture
9 0.14924307 775 high scalability-2010-02-10-ElasticSearch - Open Source, Distributed, RESTful Search Engine
10 0.14685546 313 high scalability-2008-05-02-Friends for Sale Architecture - A 300 Million Page View-Month Facebook RoR App
11 0.145418 920 high scalability-2010-10-15-Troubles with Sharding - What can we learn from the Foursquare Incident?
12 0.1452985 538 high scalability-2009-03-16-Are Cloud Based Memory Architectures the Next Big Thing?
13 0.14183822 821 high scalability-2010-05-03-MocoSpace Architecture - 3 Billion Mobile Page Views a Month
14 0.14070706 1508 high scalability-2013-08-28-Sean Hull's 20 Biggest Bottlenecks that Reduce and Slow Down Scalability
16 0.13685483 1521 high scalability-2013-09-23-Salesforce Architecture - How they Handle 1.3 Billion Transactions a Day
17 0.13570909 1440 high scalability-2013-04-15-Scaling Pinterest - From 0 to 10s of Billions of Page Views a Month in Two Years
18 0.13489364 671 high scalability-2009-08-05-Stack Overflow Architecture
19 0.13396084 1131 high scalability-2011-10-24-StackExchange Architecture Updates - Running Smoothly, Amazon 4x More Expensive
20 0.13133779 38 high scalability-2007-07-30-Build an Infinitely Scalable Infrastructure for $100 Using Amazon Services
topicId topicWeight
[(0, 0.293), (1, 0.105), (2, 0.003), (3, -0.073), (4, 0.035), (5, -0.028), (6, 0.009), (7, -0.024), (8, -0.005), (9, 0.018), (10, -0.015), (11, -0.007), (12, -0.033), (13, 0.033), (14, -0.008), (15, -0.008), (16, -0.056), (17, 0.011), (18, 0.015), (19, 0.013), (20, -0.026), (21, -0.044), (22, 0.043), (23, 0.019), (24, 0.001), (25, 0.002), (26, -0.015), (27, -0.077), (28, -0.009), (29, 0.006), (30, -0.006), (31, 0.039), (32, -0.018), (33, -0.032), (34, 0.092), (35, -0.019), (36, -0.025), (37, 0.049), (38, 0.021), (39, 0.05), (40, 0.043), (41, 0.016), (42, 0.037), (43, 0.026), (44, -0.025), (45, 0.055), (46, -0.017), (47, 0.008), (48, 0.008), (49, -0.024)]
simIndex simValue blogId blogTitle
same-blog 1 0.96810675 682 high scalability-2009-08-16-ThePort Network Architecture
Introduction: ThePort Network's Director of Engineering, TJ Muehleman was kind of enough to share some of the architectural details for their white label social media system. It currently runs about 50 social networks varying in size from less than 1000 members to more than 300,000 members, all on a Microsoft stack. In addition to their social networking platform, they offer Javascript APIs and web service APIs (both REST and SOAP) which account for a significant percentage of overall system usage. ThePort is an excellent example of a real world in-the-trenches product offering real value to customers. One of the most interesting problems they have to solve is multi-tenancy. How do you provide good performance, complete customization, support, develop new features, and provide individual search indexes for each customer? It's not an easy problem to solve. How did they solve their problems and build a successful system? Site: http://theport.com Platform Microsoft.NET 3.5 C# / VB.NET
Introduction: This is a guest post by Johann Schleier-Smith , CTO & co-founder, Tagged. Five snapshots on how Tagged scaled to more than 1,000 servers Since 2004, Tagged has grown from a tiny social experiment to one of the largest social networks, delivering five billion pages per month to many millions of members who visit to meet and socialize with new people. One step at a time, this evolution forced us to evolve our architecture, eventually arriving at an enormously capable platform . V1: PHP webapp, 100k users, 15 servers, 2004 Tagged was born in the rapid-prototyping culture of an incubator that usually launched two new concepts each year in search of the big winner. LAMP was the natural choice for this style of work, which emphasized flexibility and quick turnaround at a time when Java development was mostly oriented towards development at large enterprises, Python attracted too few programmers, and Perl brought the wrong sort. Also, we knew that Yahoo was
3 0.80030978 821 high scalability-2010-05-03-MocoSpace Architecture - 3 Billion Mobile Page Views a Month
Introduction: This is a guest post by Jamie Hall, Co-founder & CTO of MocoSpace , describing the architecture for their mobile social network. This is a timely architecture to learn from as it combines several hot trends: it is very large, mobile, and social. What they think is especially cool about their system is: how it optimizes for device/browser fragmentation on the mobile Web; their multi-tiered, read/write, local/distributed caching system; selecting PostgreSQL over MySQL as a relational DB that can scale. MocoSpace is a mobile social network, with 12 million members and 3 billion page views a month, which makes it one of the most highly trafficked mobile Websites in the US. Members access the site mainly from their mobile phone Web browser, ranging from high end smartphones to lower end devices, as well as the Web. Activities on the site include customizing profiles, chat, instant messaging, music, sharing photos & videos, games, eCards and blogs. The monetization strategy is focused on
4 0.79815698 1269 high scalability-2012-06-20-iDoneThis - Scaling an Email-based App from Scratch
Introduction: This is a guest post by Rodrigo Guzman, CTO of iDoneThis , which makes status reporting happen at your company with the lightest possible touch. iDoneThis is a simple management application that emails your team at the end of every day to ask, "What'd you get done today?" Just reply with a few lines of what you got done. The following morning everyone on your team gets a digest with what the team accomplished the previous day to keep everyone in the loop and kickstart another awesome day. Before we launched, we built iDoneThis over a weekend in the most rudimentary way possible. I kid you not, we sent the first few batches of daily emails using the BCC field of a Gmail inbox. The upshot is that we’ve had users on the site from Day 3 of its existence on. We’ve gone from launch in January 2011 when we sent hundreds of emails out per day by hand to sending out over 1 million emails and handling over 200,000 incoming emails per month. In total, customers have recorded over 1.
Introduction: This article is from an interview with Zuhaib Siddique , a production engineer at HipChat , makers of group chat and IM for teams. HipChat started in an unusual space, one you might not think would have much promise, enterprise group messaging, but as we are learning there is gold in them there enterprise hills . Which is why Atlassian, makers of well thought of tools like JIRA and Confluence, acquired HipChat in 2012 . And in a tale not often heard, the resources and connections of a larger parent have actually helped HipChat enter an exponential growth cycle . Having reached the 1.2 billion message storage mark they are now doubling the number of messages sent, stored, and indexed every few months. That kind of growth puts a lot of pressure on a once adequate infrastructure. HipChat exhibited a common scaling pattern. Start simple, experience traffic spikes, and then think what do we do now? Using bigger computers is usually the first and bes
6 0.79670393 1649 high scalability-2014-05-16-Stuff The Internet Says On Scalability For May 16th, 2014
7 0.79649371 1542 high scalability-2013-11-04-ESPN's Architecture at Scale - Operating at 100,000 Duh Nuh Nuhs Per Second
9 0.79241163 1508 high scalability-2013-08-28-Sean Hull's 20 Biggest Bottlenecks that Reduce and Slow Down Scalability
10 0.79176629 1395 high scalability-2013-01-28-DuckDuckGo Architecture - 1 Million Deep Searches a Day and Growing
11 0.79145533 1068 high scalability-2011-06-27-TripAdvisor Architecture - 40M Visitors, 200M Dynamic Page Views, 30TB Data
12 0.78512979 1302 high scalability-2012-08-10-Stuff The Internet Says On Scalability For August 10, 2012
13 0.78348547 985 high scalability-2011-02-08-Mollom Architecture - Killing Over 373 Million Spams at 100 Requests Per Second
14 0.77578199 835 high scalability-2010-06-03-Hot Scalability Links for June 3, 2010
15 0.77467698 1336 high scalability-2012-10-09-Batoo JPA - The new JPA Implementation that runs over 15 times faster...
16 0.77402174 1552 high scalability-2013-11-22-Stuff The Internet Says On Scalability For November 22th, 2013
17 0.77374512 663 high scalability-2009-07-28-37signals Architecture
18 0.77300757 1634 high scalability-2014-04-18-Stuff The Internet Says On Scalability For April 18th, 2014
19 0.77203405 1501 high scalability-2013-08-13-In Memoriam: Lavabit Architecture - Creating a Scalable Email Service
20 0.76919991 1158 high scalability-2011-12-16-Stuff The Internet Says On Scalability For December 16, 2011
topicId topicWeight
[(1, 0.152), (2, 0.182), (10, 0.047), (15, 0.117), (30, 0.058), (40, 0.019), (47, 0.028), (51, 0.01), (61, 0.113), (77, 0.01), (79, 0.101), (85, 0.038), (91, 0.014), (94, 0.029)]
simIndex simValue blogId blogTitle
same-blog 1 0.9433099 682 high scalability-2009-08-16-ThePort Network Architecture
Introduction: ThePort Network's Director of Engineering, TJ Muehleman was kind of enough to share some of the architectural details for their white label social media system. It currently runs about 50 social networks varying in size from less than 1000 members to more than 300,000 members, all on a Microsoft stack. In addition to their social networking platform, they offer Javascript APIs and web service APIs (both REST and SOAP) which account for a significant percentage of overall system usage. ThePort is an excellent example of a real world in-the-trenches product offering real value to customers. One of the most interesting problems they have to solve is multi-tenancy. How do you provide good performance, complete customization, support, develop new features, and provide individual search indexes for each customer? It's not an easy problem to solve. How did they solve their problems and build a successful system? Site: http://theport.com Platform Microsoft.NET 3.5 C# / VB.NET
2 0.92711222 1455 high scalability-2013-05-10-Stuff The Internet Says On Scalability For May 10, 2013
Introduction: Hey, it's HighScalability time: ( In Thailand, they figured out how to solve the age-old queuing problem! ) Nanoscale : Plants IM Using Nanoscale Sound Waves; 100 petabytes : CERN data storage Quotable Quotes: Geoff Arnold : Arguably all interesting advances in computer science and software engineering occur when a resource that was previously scarce or expensive becomes cheap and plentiful. @jamesurquhart : "Complexity is a characteristic of the system, not of the parts in it." -Dekker @louisnorthmore : Scaling down - now that's scalability! @peakscale : Where distributed systems people retire to forget the madness: http://en.wikipedia.org/wiki/Antipaxos @dozba : "The Linux Game Database" ... Well, at least they will never have scaling problems. Michael Widenius : There is no reason at all to use MySQL @steveloughran : Whenever someone says "unlimited scalability", ask if that exceeds the ber
3 0.92460561 85 high scalability-2007-09-08-Making the case for PHP at Yahoo! (Oct 2002)
Introduction: This presentation by Michael Radwin describes why Yahoo! had standardized on PHP going forward. It describes how after reviewing all the web technologies including their own internal ones, PHP was choosen. It shows that not only technical reasons , but also business and development processes were taken into account.
4 0.91276258 923 high scalability-2010-10-21-Machine VM + Cloud API - Rewriting the Cloud from Scratch
Introduction: Write a little "Hello World" program these days and it runs inside a bewildering Russian Doll of nested environments, each layer adding its own special performance and complexity tax. First, a language executes in its own environment of data structure libraries, memory management, and so on. That, more often than not, will run inside a language VM like the JVM, CLR, or V8. The language VM will in-turn run inside a process that runs inside an OS. An application will run in one or more threads inside a process. And the whole thing will run inside a machine sharing VM layer like Xen. And across all of that are frameworks for monitoring, elasticity, storage, and so on. That's a lot of overhead for a such a little program. What if we could remove all these taxes and run directly on the new bare metal, which some consider to be a combination of M achine VM + Cloud API ? That's exactly what a system called Mirage , described in the paper Turning down the LAMP: Software Specialisation for
5 0.91260421 1297 high scalability-2012-08-03-Stuff The Internet Says On Scalability For August 3, 2012
Introduction: It's HighScalability Time: Quotable Quotes: Ross Tur : the tricks you learned to make things big are not the same tricks you can apply to make things infinite. @gclaramunt : Son, I'm getting old, but let me tell you a secret: programming is hard, and high scalability and concurrent programming... frigging hard! @Carnage4Life : At Apple the iOS team didn't see iPhone hardware or hardware team see OS until it shipped @adrianco : #ebspio caps iops but latency variance is much lower than EBS @bernardgolden : RT @peakscale: A culture of automation is 10x more important than deployment/test/monkey thing you'd like to discuss < devops calling @JayCollier : 50 years ago, school standardization was needed for scale. Now, scalability and flexibility (variability) can coexist. #FOL2012 @adrianco : Compared to vanilla EBS many times better for random reads. Bandwidth limits both for sequential and writes. #ebspio @SQLPerfTips : More hardw
6 0.90839303 998 high scalability-2011-03-03-Stack Overflow Architecture Update - Now at 95 Million Page Views a Month
7 0.90838873 106 high scalability-2007-10-02-Secrets to Fotolog's Scaling Success
8 0.9077245 195 high scalability-2007-12-28-Amazon's EC2: Pay as You Grow Could Cut Your Costs in Half
9 0.90754241 1559 high scalability-2013-12-06-Stuff The Internet Says On Scalability For December 6th, 2013
10 0.90719968 152 high scalability-2007-11-13-Flickr Architecture
11 0.90657389 1109 high scalability-2011-09-02-Stuff The Internet Says On Scalability For September 2, 2011
13 0.90595746 1302 high scalability-2012-08-10-Stuff The Internet Says On Scalability For August 10, 2012
14 0.90570366 1180 high scalability-2012-01-24-The State of NoSQL in 2012
15 0.90566033 1093 high scalability-2011-08-05-Stuff The Internet Says On Scalability For August 5, 2011
16 0.90540564 1147 high scalability-2011-11-25-Stuff The Internet Says On Scalability For November 25, 2011
17 0.90539646 1533 high scalability-2013-10-16-Interview With Google's Ilya Grigorik On His New Book: High Performance Browser Networking
18 0.90492415 857 high scalability-2010-07-13-DbShards Part Deux - The Internals
19 0.90334487 853 high scalability-2010-07-08-Cloud AWS Infrastructure vs. Physical Infrastructure
20 0.90313184 1609 high scalability-2014-03-11-Building a Social Music Service Using AWS, Scala, Akka, Play, MongoDB, and Elasticsearch