high_scalability high_scalability-2009 high_scalability-2009-551 knowledge-graph by maker-knowledge-mining
Source: html
Introduction: Ladar Levison of Lavabit has written an incredible article on how they took a centralized off-the-shelf email server that could handle only few thousand users and built their own custom distributed infrastructure for handling hundreds of thousands of email users. Lavabit processes 70 gigabytes of data per day, is made up of 26 servers, hosts 260,000 email addresses, and processes 600,000 emails a day. That's a lot of email. Lavabit's mission has a little edge to it too: Lavabit was founded as a direct reaction to the larger free e-mail services available. We felt it was possible to create an e-mail service that was fast, reliable, feature rich and didn't achieve profitability by prostituting its user base to marketers. What I really like about this article is that Lavabit has some challenging elements in dealing with different email protocols while being able to scale to a lot of users. There's more going on than just trying to scale out a database. Many products contain com
sentIndex sentText sentNum sentScore
1 Ladar Levison of Lavabit has written an incredible article on how they took a centralized off-the-shelf email server that could handle only few thousand users and built their own custom distributed infrastructure for handling hundreds of thousands of email users. [sent-1, score-0.707]
2 Lavabit processes 70 gigabytes of data per day, is made up of 26 servers, hosts 260,000 email addresses, and processes 600,000 emails a day. [sent-2, score-0.559]
3 Lavabit's mission has a little edge to it too: Lavabit was founded as a direct reaction to the larger free e-mail services available. [sent-4, score-0.38]
4 We felt it was possible to create an e-mail service that was fast, reliable, feature rich and didn't achieve profitability by prostituting its user base to marketers. [sent-5, score-0.265]
5 What I really like about this article is that Lavabit has some challenging elements in dealing with different email protocols while being able to scale to a lot of users. [sent-6, score-0.53]
6 Many products contain complicated bits like this, so it's interesting to see how Ladar handled them. [sent-8, score-0.158]
7 Putting in this extra work in is what Ladar thinks makes Lavabit different: One of the ways to gain an advantage over your competition is to invest the time and money needed to build systems that are better than what is easily available to your competition. [sent-10, score-0.277]
8 It is the custom platform we developed that has allowed us to thrive while many other free email companies either stopped offering their service for free, or shut down altogether. [sent-11, score-0.699]
9 Since Ladar was so thorough I saved article as a separate html file. [sent-12, score-0.281]
10 Please select the visit link to read the entire article. [sent-13, score-0.147]
11 I'd like to thank Ladar again for taking the time and making the effort to document their architecture for the benefit of the community at large to learn from. [sent-14, score-0.209]
wordName wordTfidf (topN-words)
[('ladar', 0.626), ('lavabit', 0.567), ('email', 0.212), ('profitability', 0.098), ('article', 0.094), ('custom', 0.083), ('free', 0.082), ('reaction', 0.076), ('shut', 0.075), ('processes', 0.075), ('emails', 0.074), ('thrive', 0.074), ('thorough', 0.073), ('felt', 0.072), ('stopped', 0.068), ('invest', 0.065), ('gigabytes', 0.065), ('thank', 0.064), ('elements', 0.063), ('founded', 0.062), ('mission', 0.059), ('challenging', 0.059), ('hosts', 0.058), ('html', 0.058), ('contain', 0.058), ('thinks', 0.056), ('competition', 0.056), ('saved', 0.056), ('addresses', 0.055), ('centralized', 0.054), ('allowed', 0.054), ('bits', 0.053), ('direct', 0.052), ('dealing', 0.052), ('thousand', 0.052), ('gain', 0.052), ('offering', 0.051), ('select', 0.05), ('protocols', 0.05), ('benefit', 0.05), ('document', 0.049), ('visit', 0.049), ('edge', 0.049), ('rich', 0.048), ('extra', 0.048), ('link', 0.048), ('putting', 0.047), ('handled', 0.047), ('base', 0.047), ('effort', 0.046)]
simIndex simValue blogId blogTitle
same-blog 1 1.0 551 high scalability-2009-03-30-Lavabit Architecture - Creating a Scalable Email Service
Introduction: Ladar Levison of Lavabit has written an incredible article on how they took a centralized off-the-shelf email server that could handle only few thousand users and built their own custom distributed infrastructure for handling hundreds of thousands of email users. Lavabit processes 70 gigabytes of data per day, is made up of 26 servers, hosts 260,000 email addresses, and processes 600,000 emails a day. That's a lot of email. Lavabit's mission has a little edge to it too: Lavabit was founded as a direct reaction to the larger free e-mail services available. We felt it was possible to create an e-mail service that was fast, reliable, feature rich and didn't achieve profitability by prostituting its user base to marketers. What I really like about this article is that Lavabit has some challenging elements in dealing with different email protocols while being able to scale to a lot of users. There's more going on than just trying to scale out a database. Many products contain com
2 0.20740019 1501 high scalability-2013-08-13-In Memoriam: Lavabit Architecture - Creating a Scalable Email Service
Introduction: With Lavabit shutting down under murky circumstances , it seems fitting to repost an old (2009), yet still very good post by Ladar Levison on Lavabit's architecture. I don't know how much of this information is still current, but it should give you a general idea what Lavabit was all about. Getting to Know You What is the name of your system and where can we find out more about it? Note: these links are no longer valid... Lavabit http://lavabit.com http://lavabit.com/network.html http://lavabit.com/about.html What is your system for? Lavabit is a mid-sized email service provider. We currently have about 140,000 registered users with more than 260,000 email addresses. While most of our accounts belong to individual users, we also provide corporate email services to approximately 70 companies. Why did you decide to build this system? We built the system to compete against the other large free email providers, with an emphasis on serving the privacy c
3 0.13068821 221 high scalability-2008-01-24-Mailinator Architecture
Introduction: Update: A fun exploration of applied searching in How to search for the word "pen1s" in 185 emails every second . When indexOf doesn't cut it you just trie harder. Has a drunken friend ever inspired you to create a first of its kind internet service that is loved by millions, deemed subversive by thousands, all while handling over 1.2 billion emails a year on one rickity old server? That's how Paul Tyma came to build Mailinator. Mailinator is a free no-setup web service for thwarting evil spammers by creating throw-away registration email addresses. If you don't give web sites you real email address they can't spam you. They spam Mailinator instead :-) I love design with a point-of-view and Mailinator has a big giant harry one: performance first, second, and last. Why? Because Mailinator is free and that allows Paul to showcase his different perspective on design. While competitors buy big Iron to handle load, Paul uses a big idea instead: pick the right problem and create a
4 0.11422063 202 high scalability-2008-01-06-Email Architecture
Introduction: I would like to know email architecture used by large ISPs.. or even used by google. Can someone point me to some sites?? Thanks..
5 0.10702461 1269 high scalability-2012-06-20-iDoneThis - Scaling an Email-based App from Scratch
Introduction: This is a guest post by Rodrigo Guzman, CTO of iDoneThis , which makes status reporting happen at your company with the lightest possible touch. iDoneThis is a simple management application that emails your team at the end of every day to ask, "What'd you get done today?" Just reply with a few lines of what you got done. The following morning everyone on your team gets a digest with what the team accomplished the previous day to keep everyone in the loop and kickstart another awesome day. Before we launched, we built iDoneThis over a weekend in the most rudimentary way possible. I kid you not, we sent the first few batches of daily emails using the BCC field of a Gmail inbox. The upshot is that we’ve had users on the site from Day 3 of its existence on. We’ve gone from launch in January 2011 when we sent hundreds of emails out per day by hand to sending out over 1 million emails and handling over 200,000 incoming emails per month. In total, customers have recorded over 1.
6 0.088945575 253 high scalability-2008-02-19-Building a email communication system
7 0.070943251 765 high scalability-2010-01-25-Let's Welcome our Neo-Feudal Overlords
8 0.066373304 1477 high scalability-2013-06-18-Scaling Mailbox - From 0 to One Million Users in 6 Weeks and 100 Million Messages Per Day
9 0.065164261 750 high scalability-2009-12-16-Building Super Scalable Systems: Blade Runner Meets Autonomic Computing in the Ambient Cloud
11 0.060735669 96 high scalability-2007-09-18-Amazon Architecture
12 0.059206039 1151 high scalability-2011-12-05-Stuff The Internet Says On Scalability For December 5, 2011
13 0.05881691 1438 high scalability-2013-04-10-Check Yourself Before You Wreck Yourself - Avocado's 5 Early Stages of Architecture Evolution
20 0.055634264 80 high scalability-2007-09-06-Product: Perdition Mail Retrieval Proxy
topicId topicWeight
[(0, 0.109), (1, 0.017), (2, -0.004), (3, -0.019), (4, -0.001), (5, -0.034), (6, -0.0), (7, -0.001), (8, -0.002), (9, 0.012), (10, -0.015), (11, 0.027), (12, 0.005), (13, -0.011), (14, 0.038), (15, 0.006), (16, -0.016), (17, -0.0), (18, -0.011), (19, 0.006), (20, -0.013), (21, -0.025), (22, -0.014), (23, -0.014), (24, 0.011), (25, -0.013), (26, 0.029), (27, 0.006), (28, -0.029), (29, -0.001), (30, -0.02), (31, 0.004), (32, -0.024), (33, -0.013), (34, -0.024), (35, -0.026), (36, 0.021), (37, 0.019), (38, 0.025), (39, 0.003), (40, 0.018), (41, 0.058), (42, 0.038), (43, 0.002), (44, 0.003), (45, 0.026), (46, 0.034), (47, -0.025), (48, -0.023), (49, 0.014)]
simIndex simValue blogId blogTitle
same-blog 1 0.94641352 551 high scalability-2009-03-30-Lavabit Architecture - Creating a Scalable Email Service
Introduction: Ladar Levison of Lavabit has written an incredible article on how they took a centralized off-the-shelf email server that could handle only few thousand users and built their own custom distributed infrastructure for handling hundreds of thousands of email users. Lavabit processes 70 gigabytes of data per day, is made up of 26 servers, hosts 260,000 email addresses, and processes 600,000 emails a day. That's a lot of email. Lavabit's mission has a little edge to it too: Lavabit was founded as a direct reaction to the larger free e-mail services available. We felt it was possible to create an e-mail service that was fast, reliable, feature rich and didn't achieve profitability by prostituting its user base to marketers. What I really like about this article is that Lavabit has some challenging elements in dealing with different email protocols while being able to scale to a lot of users. There's more going on than just trying to scale out a database. Many products contain com
2 0.77567977 221 high scalability-2008-01-24-Mailinator Architecture
Introduction: Update: A fun exploration of applied searching in How to search for the word "pen1s" in 185 emails every second . When indexOf doesn't cut it you just trie harder. Has a drunken friend ever inspired you to create a first of its kind internet service that is loved by millions, deemed subversive by thousands, all while handling over 1.2 billion emails a year on one rickity old server? That's how Paul Tyma came to build Mailinator. Mailinator is a free no-setup web service for thwarting evil spammers by creating throw-away registration email addresses. If you don't give web sites you real email address they can't spam you. They spam Mailinator instead :-) I love design with a point-of-view and Mailinator has a big giant harry one: performance first, second, and last. Why? Because Mailinator is free and that allows Paul to showcase his different perspective on design. While competitors buy big Iron to handle load, Paul uses a big idea instead: pick the right problem and create a
3 0.76926625 253 high scalability-2008-02-19-Building a email communication system
Introduction: hi, the website i work for is looking to build a email system that can handle a fair few emails (up to a hundred thousand a day). These comprise emails like registration emails, newsletters, lots of user triggered emails and overnight emails. At present we queue them in SQL and feed them into an smtp server on one of our web servers when the queue drops below a certain level. this has caused our mail system to crash as well as hammer our DB server (shared!!!). We have got an architecture of what we want to build but thought there might be something we could buy off the shelf that allowed us to keep templated emails, lists of recipients, schedule sends etc and report on it. We can't find anything What do big websites like amazon etc use or people a little smaller but who still send loads of mail (flickr, ebuyer, or other ecommerce sites) Cheers tarqs
4 0.76740485 1269 high scalability-2012-06-20-iDoneThis - Scaling an Email-based App from Scratch
Introduction: This is a guest post by Rodrigo Guzman, CTO of iDoneThis , which makes status reporting happen at your company with the lightest possible touch. iDoneThis is a simple management application that emails your team at the end of every day to ask, "What'd you get done today?" Just reply with a few lines of what you got done. The following morning everyone on your team gets a digest with what the team accomplished the previous day to keep everyone in the loop and kickstart another awesome day. Before we launched, we built iDoneThis over a weekend in the most rudimentary way possible. I kid you not, we sent the first few batches of daily emails using the BCC field of a Gmail inbox. The upshot is that we’ve had users on the site from Day 3 of its existence on. We’ve gone from launch in January 2011 when we sent hundreds of emails out per day by hand to sending out over 1 million emails and handling over 200,000 incoming emails per month. In total, customers have recorded over 1.
Introduction: You know your product is doing well when most of your early blog posts deal with the status of the waiting list of hundreds of thousands of users eagerly waiting to download your product. That's the enviable position Mailbox , a free mobile email management app, found themselves early in their release cycle. Hasn't email been done already? Apparently not. Mailbox scaled to one million users in a paltry six weeks with a team of about 14 people . As of April they were delivering over 100 million messages per day . How did they do it? Mailbox engineering lead, Sean Beausoleil , gave an informative interview on readwrite.com on how Mailbox planned to scale... Gather signals early . A pre-release launch video helped generate interest, but it also allowed them to gauge early interest before even releasing. From the overwhelming response they knew they would need to have to scale and scale quickly. Have something unique . The average person might not think a mailbox app
6 0.71960938 1501 high scalability-2013-08-13-In Memoriam: Lavabit Architecture - Creating a Scalable Email Service
7 0.70897925 985 high scalability-2011-02-08-Mollom Architecture - Killing Over 373 Million Spams at 100 Requests Per Second
8 0.69269764 80 high scalability-2007-09-06-Product: Perdition Mail Retrieval Proxy
9 0.68857604 1635 high scalability-2014-04-21-This is why Microsoft won. And why they lost.
10 0.67868239 1638 high scalability-2014-04-28-How Disqus Went Realtime with 165K Messages Per Second and Less than .2 Seconds Latency
11 0.67343247 202 high scalability-2008-01-06-Email Architecture
12 0.66759175 67 high scalability-2007-08-17-What is the best hosting option?
13 0.66347998 228 high scalability-2008-01-28-Product: ISPMan Centralized ISP Management System
14 0.66241574 210 high scalability-2008-01-13-A Note on How to Create Teasers When Posting
15 0.649571 993 high scalability-2011-02-22-Is Node.js Becoming a Part of the Stack? SimpleGeo Says Yes.
16 0.64768374 1573 high scalability-2014-01-06-How HipChat Stores and Indexes Billions of Messages Using ElasticSearch and Redis
17 0.6473431 1068 high scalability-2011-06-27-TripAdvisor Architecture - 40M Visitors, 200M Dynamic Page Views, 30TB Data
18 0.64689302 1492 high scalability-2013-07-17-How do you create a 100th Monkey software development culture?
19 0.6449675 1012 high scalability-2011-03-28-Aztec Empire Strategy: Use Dual Pipes in Your Aqueduct for High Availability
20 0.64420736 1528 high scalability-2013-10-07-Ask HS: Is Microsoft the Right Technology for a Scalable Web-based System?
topicId topicWeight
[(1, 0.134), (2, 0.574), (40, 0.026), (61, 0.06), (79, 0.017), (94, 0.046)]
simIndex simValue blogId blogTitle
1 0.99741995 723 high scalability-2009-10-16-Paper: Scaling Online Social Networks without Pains
Introduction: We saw in Why are Facebook, Digg, and Twitter so hard to scale? scaling social networks is a lot harder than you might think. This paper, Scaling Online Social Networks without Pains , from a team at Telefonica Research in Spain hopes to meet the challenge of status distribution, user generated content distribution, and managing the social graph through a technique they call One-Hop Replication (OHR). OHR abstracts and delegates the complexity of scaling up from the social network application . The abstract: Online Social Networks (OSN) face serious scalability challenges due to their rapid growth and popularity. To address this issue we present a novel approach to scale up OSN called One Hop Replication (OHR). Our system combines partitioning and replication in a middleware to transparently scale up a centralized OSN design, and therefore, avoid the OSN application to undergo the costly transition to a fully distributed system to meet its scalability needs. OHR exploits some
2 0.9939881 205 high scalability-2008-01-10-Letting Clients Know What's Changed: Push Me or Pull Me?
Introduction: I had a false belief I thought I came here to stay We're all just visiting All just breaking like waves The oceans made me, but who came up with me? Push me, pull me, push me, or pull me out . So true Perl Jam (Push me Pull me lyrics) , so true. I too have wondered how web clients should be notified of model changes. Should servers push events to clients or should clients pull events from servers? A topic worthy of its own song if ever there was one. To pull events the client simply starts a timer and makes a request to the server. This is polling. You can either pull a complete set of fresh data or get a list of changes. The server "knows" if anything you are interested in has changed and makes those changes available to you. Knowing what has changed can be relatively simple with a publish-subscribe type backend or you can get very complex with fine grained bit maps of attributes and keeping per client state on what I client still needs to see. Polling is heavy man.
3 0.99356443 967 high scalability-2011-01-03-Stuff The Internet Says On Scalability For January 3, 2010
Introduction: Submitted for your reading pleasure... Quotable Quotes @hofmanndavid : Performance and scalability anxiety makes developers want to catch the flying butterflies @tivrfoa : "Scalability solutions aren't magic. They involve partitioning, indexing and replication." Twitter engineer Alan Perlis: Fools ignore complexity; pragmatists suffer it; experts avoid it; geniuses remove it. CIO update: Post-mortem on the Skype outage . Interesting tale of a cascading collapse in complex, distributed, interactive systems. For more background see the highly illuminating Explaining Supernodes by Dan York. RethinkDB and SSD Databases. SSD was not a revolution by Kevin Burton. What’s really shocking to me, is that while SSD and flash storage is very exciting, it wasn’t as revolutionary in 2010 as I would have liked to have seen. The case for Datastore-Side-Scripting . Russell Sullivan predicts real-time web applications are going in the direction of being enti
same-blog 4 0.99335712 551 high scalability-2009-03-30-Lavabit Architecture - Creating a Scalable Email Service
Introduction: Ladar Levison of Lavabit has written an incredible article on how they took a centralized off-the-shelf email server that could handle only few thousand users and built their own custom distributed infrastructure for handling hundreds of thousands of email users. Lavabit processes 70 gigabytes of data per day, is made up of 26 servers, hosts 260,000 email addresses, and processes 600,000 emails a day. That's a lot of email. Lavabit's mission has a little edge to it too: Lavabit was founded as a direct reaction to the larger free e-mail services available. We felt it was possible to create an e-mail service that was fast, reliable, feature rich and didn't achieve profitability by prostituting its user base to marketers. What I really like about this article is that Lavabit has some challenging elements in dealing with different email protocols while being able to scale to a lot of users. There's more going on than just trying to scale out a database. Many products contain com
5 0.99156898 1190 high scalability-2012-02-10-Stuff The Internet Says On Scalability For February 10, 2012
Introduction: HighScalability Tested, Mother Approved: 12,233TPS : Twitter @ Super Bowl; 11 Million Slices : Dominos @ Super Bowl; 500K requests per second : S3; The great mobile money drain . Mobile: high resource costs, low revenue. Mobile traffic on Plenty of Fish is growing at 3% a month , rising to 3 Billion pageviews a month, 40% of signups are mobile, and all traffic will soon be 60-70% mobile. The problem: how do you make money on mobile? Time to chuck microprocessors for a networks of cells? How Networks of Biological Cells Solve Distributed Computing Problems : Computer scientists prove that networks of cells can compute as efficiently as networks of computers linked via the internet. We believe that there is a need for a network model, where nodes are by design below the computation and communication capabilities of Turing machines. Unrelated? GDrive at last and S3 Drops Storage Pricing . If you are StackOverflow and your data is overflowing , what do you do? Mo
7 0.99022675 1199 high scalability-2012-02-27-Zen and the Art of Scaling - A Koan and Epigram Approach
8 0.99009943 594 high scalability-2009-05-08-Eight Best Practices for Building Scalable Systems
9 0.98850465 639 high scalability-2009-06-27-Scaling Twitter: Making Twitter 10000 Percent Faster
10 0.98844004 455 high scalability-2008-12-01-MySQL Database Scale-out and Replication for High Growth Businesses
11 0.98733962 1591 high scalability-2014-02-05-Little’s Law, Scalability and Fault Tolerance: The OS is your bottleneck. What you can do?
12 0.98416811 911 high scalability-2010-09-30-More Troubles with Caching
13 0.98389608 878 high scalability-2010-08-12-Strategy: Terminate SSL Connections in Hardware and Reduce Server Count by 40%
14 0.98379308 1006 high scalability-2011-03-17-Are long VM instance spin-up times in the cloud costing you money?
15 0.98271537 1283 high scalability-2012-07-13-Stuff The Internet Says On Scalability For July 13, 2012
16 0.98159027 836 high scalability-2010-06-04-Strategy: Cache Larger Chunks - Cache Hit Rate is a Bad Indicator
17 0.98133433 662 high scalability-2009-07-27-Handle 700 Percent More Requests Using Squid and APC Cache
18 0.97729975 50 high scalability-2007-07-31-BerkeleyDB & other distributed high performance key-value databases
19 0.97673845 861 high scalability-2010-07-20-Sponsored Post: ezRez, VoltDB and Digg are Hiring
20 0.97565532 844 high scalability-2010-06-18-Paper: The Declarative Imperative: Experiences and Conjectures in Distributed Logic