high_scalability high_scalability-2008 high_scalability-2008-253 knowledge-graph by maker-knowledge-mining
Source: html
Introduction: hi, the website i work for is looking to build a email system that can handle a fair few emails (up to a hundred thousand a day). These comprise emails like registration emails, newsletters, lots of user triggered emails and overnight emails. At present we queue them in SQL and feed them into an smtp server on one of our web servers when the queue drops below a certain level. this has caused our mail system to crash as well as hammer our DB server (shared!!!). We have got an architecture of what we want to build but thought there might be something we could buy off the shelf that allowed us to keep templated emails, lists of recipients, schedule sends etc and report on it. We can't find anything What do big websites like amazon etc use or people a little smaller but who still send loads of mail (flickr, ebuyer, or other ecommerce sites) Cheers tarqs
sentIndex sentText sentNum sentScore
1 hi, the website i work for is looking to build a email system that can handle a fair few emails (up to a hundred thousand a day). [sent-1, score-1.26]
2 These comprise emails like registration emails, newsletters, lots of user triggered emails and overnight emails. [sent-2, score-1.948]
3 At present we queue them in SQL and feed them into an smtp server on one of our web servers when the queue drops below a certain level. [sent-3, score-0.862]
4 this has caused our mail system to crash as well as hammer our DB server (shared! [sent-4, score-0.655]
5 We have got an architecture of what we want to build but thought there might be something we could buy off the shelf that allowed us to keep templated emails, lists of recipients, schedule sends etc and report on it. [sent-8, score-1.091]
wordName wordTfidf (topN-words)
[('emails', 0.661), ('mail', 0.245), ('recipients', 0.203), ('comprise', 0.176), ('hammer', 0.152), ('overnight', 0.149), ('queue', 0.146), ('etc', 0.143), ('smtp', 0.142), ('triggered', 0.14), ('shelf', 0.131), ('drops', 0.121), ('fair', 0.12), ('ecommerce', 0.119), ('crash', 0.112), ('sends', 0.112), ('flickr', 0.107), ('registration', 0.104), ('schedule', 0.099), ('feed', 0.098), ('lists', 0.097), ('allowed', 0.097), ('report', 0.095), ('hundred', 0.095), ('thousand', 0.093), ('caused', 0.092), ('loads', 0.087), ('present', 0.081), ('db', 0.08), ('smaller', 0.077), ('email', 0.076), ('certain', 0.074), ('buy', 0.074), ('build', 0.07), ('send', 0.069), ('got', 0.066), ('websites', 0.064), ('thought', 0.063), ('shared', 0.062), ('sites', 0.062), ('sql', 0.058), ('lots', 0.057), ('server', 0.054), ('website', 0.053), ('little', 0.049), ('looking', 0.048), ('day', 0.046), ('handle', 0.044), ('amazon', 0.044), ('might', 0.044)]
simIndex simValue blogId blogTitle
same-blog 1 0.99999994 253 high scalability-2008-02-19-Building a email communication system
Introduction: hi, the website i work for is looking to build a email system that can handle a fair few emails (up to a hundred thousand a day). These comprise emails like registration emails, newsletters, lots of user triggered emails and overnight emails. At present we queue them in SQL and feed them into an smtp server on one of our web servers when the queue drops below a certain level. this has caused our mail system to crash as well as hammer our DB server (shared!!!). We have got an architecture of what we want to build but thought there might be something we could buy off the shelf that allowed us to keep templated emails, lists of recipients, schedule sends etc and report on it. We can't find anything What do big websites like amazon etc use or people a little smaller but who still send loads of mail (flickr, ebuyer, or other ecommerce sites) Cheers tarqs
2 0.31646752 221 high scalability-2008-01-24-Mailinator Architecture
Introduction: Update: A fun exploration of applied searching in How to search for the word "pen1s" in 185 emails every second . When indexOf doesn't cut it you just trie harder. Has a drunken friend ever inspired you to create a first of its kind internet service that is loved by millions, deemed subversive by thousands, all while handling over 1.2 billion emails a year on one rickity old server? That's how Paul Tyma came to build Mailinator. Mailinator is a free no-setup web service for thwarting evil spammers by creating throw-away registration email addresses. If you don't give web sites you real email address they can't spam you. They spam Mailinator instead :-) I love design with a point-of-view and Mailinator has a big giant harry one: performance first, second, and last. Why? Because Mailinator is free and that allows Paul to showcase his different perspective on design. While competitors buy big Iron to handle load, Paul uses a big idea instead: pick the right problem and create a
3 0.29416898 1269 high scalability-2012-06-20-iDoneThis - Scaling an Email-based App from Scratch
Introduction: This is a guest post by Rodrigo Guzman, CTO of iDoneThis , which makes status reporting happen at your company with the lightest possible touch. iDoneThis is a simple management application that emails your team at the end of every day to ask, "What'd you get done today?" Just reply with a few lines of what you got done. The following morning everyone on your team gets a digest with what the team accomplished the previous day to keep everyone in the loop and kickstart another awesome day. Before we launched, we built iDoneThis over a weekend in the most rudimentary way possible. I kid you not, we sent the first few batches of daily emails using the BCC field of a Gmail inbox. The upshot is that we’ve had users on the site from Day 3 of its existence on. We’ve gone from launch in January 2011 when we sent hundreds of emails out per day by hand to sending out over 1 million emails and handling over 200,000 incoming emails per month. In total, customers have recorded over 1.
4 0.1854032 1501 high scalability-2013-08-13-In Memoriam: Lavabit Architecture - Creating a Scalable Email Service
Introduction: With Lavabit shutting down under murky circumstances , it seems fitting to repost an old (2009), yet still very good post by Ladar Levison on Lavabit's architecture. I don't know how much of this information is still current, but it should give you a general idea what Lavabit was all about. Getting to Know You What is the name of your system and where can we find out more about it? Note: these links are no longer valid... Lavabit http://lavabit.com http://lavabit.com/network.html http://lavabit.com/about.html What is your system for? Lavabit is a mid-sized email service provider. We currently have about 140,000 registered users with more than 260,000 email addresses. While most of our accounts belong to individual users, we also provide corporate email services to approximately 70 companies. Why did you decide to build this system? We built the system to compete against the other large free email providers, with an emphasis on serving the privacy c
5 0.12814701 233 high scalability-2008-01-30-How Rackspace Now Uses MapReduce and Hadoop to Query Terabytes of Data
Introduction: How do you query hundreds of gigabytes of new data each day streaming in from over 600 hyperactive servers? If you think this sounds like the perfect battle ground for a head-to-head skirmish in the great MapReduce Versus Database War , you would be correct. Bill Boebel, CTO of Mailtrust (Rackspace's mail division), has generously provided a fascinating account of how they evolved their log processing system from an early amoeba'ic text file stored on each machine approach, to a Neandertholic relational database solution that just couldn't compete, and finally to a Homo sapien'ic Hadoop based solution that works wisely for them and has virtually unlimited scalability potential. Rackspace faced a now familiar problem. Lots and lots of data streaming in. Where do you store all that data? How do you do anything useful with it? In the first version of their system logs were stored in flat text files and had to be manually searched by engineers logging into each individual machine. T
7 0.11811613 80 high scalability-2007-09-06-Product: Perdition Mail Retrieval Proxy
8 0.098073564 821 high scalability-2010-05-03-MocoSpace Architecture - 3 Billion Mobile Page Views a Month
9 0.088945575 551 high scalability-2009-03-30-Lavabit Architecture - Creating a Scalable Email Service
10 0.08817973 868 high scalability-2010-07-30-Basho Lives up to their Name With Consistent Smashing
11 0.085256875 189 high scalability-2007-12-21-Strategy: Limit Result Sets
12 0.073519394 49 high scalability-2007-07-30-allowed contributed
13 0.073332474 202 high scalability-2008-01-06-Email Architecture
14 0.071259119 81 high scalability-2007-09-06-Scaling IMAP and POP3
15 0.071185306 38 high scalability-2007-07-30-Build an Infinitely Scalable Infrastructure for $100 Using Amazon Services
16 0.071032241 30 high scalability-2007-07-26-Product: AWStats a Log Analyzer
17 0.069644101 1373 high scalability-2012-12-17-11 Uses For the Humble Presents Queue, er, Message Queue
18 0.069182843 57 high scalability-2007-08-03-Scaling IMAP and POP3
19 0.067276321 1010 high scalability-2011-03-24-Strategy: Disk Backup for Speed, Tape Backup to Save Your Bacon, Just Ask Google
20 0.065820366 674 high scalability-2009-08-07-The Canonical Cloud Architecture
topicId topicWeight
[(0, 0.104), (1, 0.04), (2, -0.013), (3, -0.07), (4, 0.019), (5, -0.052), (6, -0.01), (7, 0.004), (8, 0.002), (9, -0.009), (10, -0.019), (11, 0.026), (12, 0.024), (13, -0.028), (14, 0.054), (15, -0.007), (16, -0.008), (17, 0.029), (18, -0.011), (19, 0.024), (20, 0.001), (21, -0.056), (22, -0.02), (23, 0.003), (24, 0.037), (25, -0.043), (26, 0.024), (27, 0.033), (28, -0.031), (29, -0.008), (30, 0.002), (31, -0.011), (32, -0.034), (33, -0.069), (34, 0.024), (35, -0.038), (36, -0.008), (37, 0.052), (38, 0.038), (39, 0.03), (40, 0.061), (41, 0.056), (42, 0.035), (43, -0.008), (44, 0.003), (45, 0.021), (46, -0.019), (47, -0.049), (48, -0.057), (49, -0.004)]
simIndex simValue blogId blogTitle
same-blog 1 0.94422495 253 high scalability-2008-02-19-Building a email communication system
Introduction: hi, the website i work for is looking to build a email system that can handle a fair few emails (up to a hundred thousand a day). These comprise emails like registration emails, newsletters, lots of user triggered emails and overnight emails. At present we queue them in SQL and feed them into an smtp server on one of our web servers when the queue drops below a certain level. this has caused our mail system to crash as well as hammer our DB server (shared!!!). We have got an architecture of what we want to build but thought there might be something we could buy off the shelf that allowed us to keep templated emails, lists of recipients, schedule sends etc and report on it. We can't find anything What do big websites like amazon etc use or people a little smaller but who still send loads of mail (flickr, ebuyer, or other ecommerce sites) Cheers tarqs
2 0.72967768 551 high scalability-2009-03-30-Lavabit Architecture - Creating a Scalable Email Service
Introduction: Ladar Levison of Lavabit has written an incredible article on how they took a centralized off-the-shelf email server that could handle only few thousand users and built their own custom distributed infrastructure for handling hundreds of thousands of email users. Lavabit processes 70 gigabytes of data per day, is made up of 26 servers, hosts 260,000 email addresses, and processes 600,000 emails a day. That's a lot of email. Lavabit's mission has a little edge to it too: Lavabit was founded as a direct reaction to the larger free e-mail services available. We felt it was possible to create an e-mail service that was fast, reliable, feature rich and didn't achieve profitability by prostituting its user base to marketers. What I really like about this article is that Lavabit has some challenging elements in dealing with different email protocols while being able to scale to a lot of users. There's more going on than just trying to scale out a database. Many products contain com
3 0.72082502 202 high scalability-2008-01-06-Email Architecture
Introduction: I would like to know email architecture used by large ISPs.. or even used by google. Can someone point me to some sites?? Thanks..
4 0.71748024 221 high scalability-2008-01-24-Mailinator Architecture
Introduction: Update: A fun exploration of applied searching in How to search for the word "pen1s" in 185 emails every second . When indexOf doesn't cut it you just trie harder. Has a drunken friend ever inspired you to create a first of its kind internet service that is loved by millions, deemed subversive by thousands, all while handling over 1.2 billion emails a year on one rickity old server? That's how Paul Tyma came to build Mailinator. Mailinator is a free no-setup web service for thwarting evil spammers by creating throw-away registration email addresses. If you don't give web sites you real email address they can't spam you. They spam Mailinator instead :-) I love design with a point-of-view and Mailinator has a big giant harry one: performance first, second, and last. Why? Because Mailinator is free and that allows Paul to showcase his different perspective on design. While competitors buy big Iron to handle load, Paul uses a big idea instead: pick the right problem and create a
Introduction: Here's an Update On Disqus: It's Still About Realtime, But Go Demolishes Python . How do you add realtime functionality to a web scale application? That's what Adam Hitchcock , a Software Engineer at Disqus talks about in an excellent talk: Making DISQUS Realtime ( slides ). Disqus had to take their commenting system and add realtime capabilities to it. Not something that's easy to do when at the time of the talk (2013) they had had just hit a billion unique visitors a month. What Disqus developed is a realtime commenting system called “realertime” that was tested to handle 1.5 million concurrently connected users, 45,000 new connections per second, 165,000 messages/second, with less than .2 seconds latency end-to-end. The nature of a commenting system is that it is IO bound and has a high fanout, that is a comment comes in and must be sent out to a lot of readers. It's a problem very similar to what Twitter must solve . Disqus' solution was quite interesting as was th
7 0.6720562 1269 high scalability-2012-06-20-iDoneThis - Scaling an Email-based App from Scratch
8 0.65240192 80 high scalability-2007-09-06-Product: Perdition Mail Retrieval Proxy
9 0.64956093 1501 high scalability-2013-08-13-In Memoriam: Lavabit Architecture - Creating a Scalable Email Service
10 0.62556958 431 high scalability-2008-10-27-Notify.me Architecture - Synchronicity Kills
11 0.6205911 145 high scalability-2007-11-08-ID generator
12 0.61742264 1528 high scalability-2013-10-07-Ask HS: Is Microsoft the Right Technology for a Scalable Web-based System?
13 0.60649091 481 high scalability-2009-01-02-Strategy: Understanding Your Data Leads to the Best Scalability Solutions
14 0.60524571 1070 high scalability-2011-06-29-Second Hand Seizure : A New Cause of Site Death
15 0.60434878 985 high scalability-2011-02-08-Mollom Architecture - Killing Over 373 Million Spams at 100 Requests Per Second
16 0.60133988 1573 high scalability-2014-01-06-How HipChat Stores and Indexes Billions of Messages Using ElasticSearch and Redis
17 0.59966063 169 high scalability-2007-12-01-many website, one setup, many databases
18 0.59000778 290 high scalability-2008-03-28-How to Get DNS Names of a Web Server
19 0.58486539 682 high scalability-2009-08-16-ThePort Network Architecture
20 0.58044052 711 high scalability-2009-09-22-How Ravelry Scales to 10 Million Requests Using Rails
topicId topicWeight
[(1, 0.099), (2, 0.239), (25, 0.248), (61, 0.114), (77, 0.044), (79, 0.063), (94, 0.038), (96, 0.023)]
simIndex simValue blogId blogTitle
1 0.87905198 203 high scalability-2008-01-07-How Ruby on Rails Survived a 550k Pageview Digging
Introduction: Shanti Braford details how his Ruby on Rails based website survived a 24 hour 550,000+ pageview digg attack. His post cleanly lays out all the juicy setup details, so there's not much I can add. Hosting costs $370 a month for 1 web server, 1 database server, and sufficient bandwidth. The site is built on RoR, nginx, MySQL, and 7 mongrel servers. He thinks Rails 2.0 has improved performance and credits database avoidance and fragment caching for much of the performance boost. Keep in mind his system is relatively static, but it's a very interesting and useful experience report.
2 0.85705316 246 high scalability-2008-02-12-Search the tags across all post
Introduction: Let suppose i have table which stored tags .Now user can enter keywords and i have to search through all the records in table and find post which contain tags entered by user .user can enter more than 1 keywords. What strategy ,technique i use to search fast .There maybe more than millions records and many users are firing same query. Thanks
same-blog 3 0.85163063 253 high scalability-2008-02-19-Building a email communication system
Introduction: hi, the website i work for is looking to build a email system that can handle a fair few emails (up to a hundred thousand a day). These comprise emails like registration emails, newsletters, lots of user triggered emails and overnight emails. At present we queue them in SQL and feed them into an smtp server on one of our web servers when the queue drops below a certain level. this has caused our mail system to crash as well as hammer our DB server (shared!!!). We have got an architecture of what we want to build but thought there might be something we could buy off the shelf that allowed us to keep templated emails, lists of recipients, schedule sends etc and report on it. We can't find anything What do big websites like amazon etc use or people a little smaller but who still send loads of mail (flickr, ebuyer, or other ecommerce sites) Cheers tarqs
4 0.84276801 770 high scalability-2010-02-03-NoSQL Means Never Having to Store Blobs Again
Introduction: Morgan Tocker has an awesome article and comment thread in the MySQL Performance Blog about When should you store serialized objects in the database? Before the NoSQL age is was very common to simulate schemalessness by storing blobs in MySQL. Sharding was implemented by running multiple MySQL instances and spreading writes across them. While not ideal for the purpose, developers felt comfortable with MySQL. They knew how to install it, back it up, replicate it, in short:Â they knew how to make it work. Yet they also needed to store objects without the penalty of joins. Searches and aggregate queries were handled by indexes kept in separate tables, this offloaded the fast path to objects. This all made perfect sense. Usually we just want stuff to work and going with what you know is often the best path to that goal. And what we have known is MySQL. All the different pros and cons of this approach are covered wonderfully in the post. But the world has changed. With the same goal
5 0.8288126 992 high scalability-2011-02-18-Stuff The Internet Says On Scalability For February 18, 2011
Introduction: Submitted for your reading pleasure on this cold and rainy Friday... Quotable Quotes: CarryMillsap : You can't hardware yourself out of a performance problem you softwared yourself into. @juokaz : schema-less databases doesn't mean data should have no structure Scalability Porn: 3 Months To The First Million Users, Just 6 Weeks To The Second Million For Instagram S tudy by the USC Annenberg School for Communication & Journalism estimates: in 2007, humankind was able to store 2.9 × 1020 optimally compressed bytes, communicate almost 2 × 1021 bytes, and carry out 6.4 × 1018 instructions per second on general-purpose computers. Hadoop has hit a scalability limit at a whopping 4,000 machines and are looking to create the next generation architecture . Their target is clusters of 10,000 machines and 200,000 cores. The fundamental idea of the re-architecture is to divide the two major functions of the Job Tracker, resource management and job sc
6 0.82544643 1607 high scalability-2014-03-07-Stuff The Internet Says On Scalability For March 7th, 2014
7 0.82171142 408 high scalability-2008-10-10-Useful Corporate Blogs that Talk About Scalability
8 0.81433833 1214 high scalability-2012-03-23-Stuff The Internet Says On Scalability For March 23, 2012
9 0.81300461 346 high scalability-2008-06-28-ID generation schemes
10 0.80688447 870 high scalability-2010-08-02-7 Scaling Strategies Facebook Used to Grow to 500 Million Users
11 0.8026036 412 high scalability-2008-10-14-Sun N1 Grid Engine Software and the Tokyo Institute of Technology Super Computer Grid
12 0.79607362 805 high scalability-2010-04-06-Strategy: Make it Really Fast vs Do the Work Up Front
13 0.79064107 545 high scalability-2009-03-19-Product: Redis - Not Just Another Key-Value Store
14 0.78315282 419 high scalability-2008-10-15-The Tokyo Institute of Technology Supercomputer Grid: Architecture and Performance Overview
15 0.77427584 987 high scalability-2011-02-10-Dispelling the New SSL Myth
16 0.75185949 671 high scalability-2009-08-05-Stack Overflow Architecture
17 0.75026989 724 high scalability-2009-10-19-Drupal's Scalability Makeover - You give up some control and you get back scalability
18 0.74954027 933 high scalability-2010-11-01-Hot Trend: Move Behavior to Data for a New Interactive Application Architecture
19 0.74869072 379 high scalability-2008-09-04-Database question for upcoming project
20 0.74758607 1362 high scalability-2012-11-26-BigData using Erlang, C and Lisp to Fight the Tsunami of Mobile Data