high_scalability high_scalability-2007 high_scalability-2007-172 knowledge-graph by maker-knowledge-mining
Source: html
Introduction: nginx is a high performance smtp/pop/imap proxy that lets you do custom authorization and lookups and is very scalable. (just add nodes) Nginx by default is a reverse proxy and this is what it is doing here for pop/imap connections. It is also an excellelent reverse proxy for web servers. Advantage: You dont have to have a speacial database or ldap schema. Just an url to do auth and lookup with. A url that may be accessed by a unix or a tcp socket. Write your own auth handler - according to your own policy. For example: A user called atif tries to login with the pass testxyz. You pass this infomation to a URL such as socket:/var/tmp/xyz.sock or http://auth.corp.mailserver.net:someport/someurl The auth server replies with either a FAILURE such as Auth-Status: Invalid Login or password or with a success such as Auth-Status: OK Auth-Server: OneOfThe100Servers Auth-Port: optionalyAPort We have implemented it at our ISP and it has saves us a
sentIndex sentText sentNum sentScore
1 nginx is a high performance smtp/pop/imap proxy that lets you do custom authorization and lookups and is very scalable. [sent-1, score-0.949]
2 (just add nodes) Nginx by default is a reverse proxy and this is what it is doing here for pop/imap connections. [sent-2, score-0.644]
3 It is also an excellelent reverse proxy for web servers. [sent-3, score-0.541]
4 Advantage: You dont have to have a speacial database or ldap schema. [sent-4, score-0.323]
5 A url that may be accessed by a unix or a tcp socket. [sent-6, score-0.609]
6 Write your own auth handler - according to your own policy. [sent-7, score-0.733]
7 For example: A user called atif tries to login with the pass testxyz. [sent-8, score-0.878]
8 You pass this infomation to a URL such as socket:/var/tmp/xyz. [sent-9, score-0.218]
wordName wordTfidf (topN-words)
[('auth', 0.489), ('proxy', 0.291), ('url', 0.276), ('login', 0.248), ('pass', 0.218), ('reverse', 0.201), ('atif', 0.2), ('nginx', 0.189), ('imap', 0.173), ('invalid', 0.159), ('authorization', 0.155), ('ldap', 0.152), ('handler', 0.149), ('dont', 0.147), ('smtp', 0.14), ('replies', 0.138), ('isp', 0.133), ('unix', 0.123), ('lookups', 0.116), ('tries', 0.113), ('saves', 0.113), ('lookup', 0.105), ('default', 0.098), ('lets', 0.096), ('according', 0.095), ('tried', 0.092), ('accessed', 0.09), ('success', 0.089), ('tcp', 0.087), ('implemented', 0.078), ('custom', 0.074), ('either', 0.072), ('called', 0.062), ('failure', 0.062), ('yet', 0.058), ('nodes', 0.056), ('add', 0.054), ('write', 0.044), ('example', 0.042), ('us', 0.041), ('user', 0.037), ('may', 0.033), ('lot', 0.03), ('high', 0.028), ('work', 0.027), ('server', 0.027), ('would', 0.026), ('web', 0.025), ('database', 0.024), ('also', 0.024)]
simIndex simValue blogId blogTitle
same-blog 1 1.0 172 high scalability-2007-12-02-nginx: high performance smpt-pop-imap proxy
Introduction: nginx is a high performance smtp/pop/imap proxy that lets you do custom authorization and lookups and is very scalable. (just add nodes) Nginx by default is a reverse proxy and this is what it is doing here for pop/imap connections. It is also an excellelent reverse proxy for web servers. Advantage: You dont have to have a speacial database or ldap schema. Just an url to do auth and lookup with. A url that may be accessed by a unix or a tcp socket. Write your own auth handler - according to your own policy. For example: A user called atif tries to login with the pass testxyz. You pass this infomation to a URL such as socket:/var/tmp/xyz.sock or http://auth.corp.mailserver.net:someport/someurl The auth server replies with either a FAILURE such as Auth-Status: Invalid Login or password or with a success such as Auth-Status: OK Auth-Server: OneOfThe100Servers Auth-Port: optionalyAPort We have implemented it at our ISP and it has saves us a
2 0.21228775 314 high scalability-2008-05-03-Product: nginx
Introduction: Update 6 : nginx_http_push_module . Turn nginx into a long-polling message queuing HTTP push server. Update 5 : In Load Balancer Update Barry describes how WordPress.com moved from Pound to Nginx and are now "regularly serving about 8-9k requests/second and about 1.2Gbit/sec through a few Nginx instances and have plenty of room to grow!". Update 4 : Nginx better than Pound for load balancing. Pound spikes at 80% CPU, Nginx uses 3% and is easier to understand and better documented. Update 3 : igvita.com combines two cool tools together for better performance in Nginx and Memcached, a 400% boost! . Update 2 : Software Project on Installing Nginx Web Server w/ PHP and SSL . Breaking away from mother Apache can be a scary proposition and this kind of getting started article really helps easy the separation. Update: Slicehost has some nice tutorials on setting up Nginx . From their website: Nginx ("engine x") is a high-performance HTTP server and reverse proxy, as wel
3 0.14071955 353 high scalability-2008-07-20-Strategy: Front S3 with a Caching Proxy
Introduction: Given S3's recent failure ( Cloud Status tells the tale) Kevin Burton makes the excellent suggestion of fronting S3 with a caching proxy server . A caching proxy server can reply to service requests without contacting the specified server, by retrieving content saved from a previous request, made by the same client or even other clients. This is called caching. Caching proxies keep local copies of frequently requested resources. In normal operation when an asset (a user's avatar, for example) is requested the cache is tried first. If the asset is found in the cache then it's returned. If the asset is not in the cache it's retrieved from S3 (or wherever) and cached. So when S3 goes down it's likely you can ride out the down time by serving assets out of the cache. This strategy only works when using S3 as a CDN . If you are using S3 for its "real" purpose, as a storage service, then a caching proxy can't help you... Amazon doesn't used S3 as a CDN either Amazon Not
4 0.13114065 297 high scalability-2008-04-05-Skype Plans for PostgreSQL to Scale to 1 Billion Users
Introduction: Skype uses PostgreSQL as their backend database . PostgreSQL doesn't get enough run in the database world so I was excited to see how PostgreSQL is used "as the main DB for most of [Skype's] business needs." Their approach is to use a traditional stored procedure interface for accessing data and on top of that layer proxy servers which hash SQL requests to a set of database servers that actually carry out queries. The result is a horizontally partitioned system that they think will scale to handle 1 billion users. Skype's goal is an architecture that can handle 1 billion plus users. This level of scale isn't practically solvable with one really big computer, so our masked superhero horizontal scaling comes to the rescue. Hardware is dual or quad Opterons with SCSI RAID. Followed common database progression: Start with one DB. Add new databases partitioned by functionality. Replicate read-mostly data for better read access. Then horizontally partition data across multiple nod
5 0.12983303 456 high scalability-2008-12-01-Sun's High-Performance and Reliable Web Proxy Solution
Introduction: As individuals and businesses depend on the Web more than ever to conduct business, rapid and reliable content retrieval is critical. Reducing wait time improves productivity and increases user satisfaction. Web proxy technology has emerged as an effective solution to improve performance, help ensure content availability and enhance network security by caching and filtering Web content. The combination of Sun SPARC Enterprise servers with CoolThreads technology and the Sun Java System Web Proxy Server software provides a compelling foundation for a robust Web proxy solution. Sun SPARC Enterprise T1000 and T2000 servers include the UltraSPARC T1 processor with CoolThreads technology, offering six or eight cores with four threads per core. The Sun Java System Web Proxy Server software is highly threaded and takes advantage of the large number of threads supported by Sun UltraSPARC T1 processors with CoolThreads technology. Together, these products provide a highly scalable solution that
6 0.11728078 1501 high scalability-2013-08-13-In Memoriam: Lavabit Architecture - Creating a Scalable Email Service
7 0.11567764 808 high scalability-2010-04-12-Poppen.de Architecture
9 0.10444704 1640 high scalability-2014-04-30-10 Tips for Optimizing NGINX and PHP-fpm for High Traffic Sites
10 0.10259272 74 high scalability-2007-08-23-Product: Varnish
11 0.10125929 307 high scalability-2008-04-21-Using Google AppEngine for a Little Micro-Scalability
12 0.10007478 290 high scalability-2008-03-28-How to Get DNS Names of a Web Server
13 0.095413052 228 high scalability-2008-01-28-Product: ISPMan Centralized ISP Management System
14 0.086339869 623 high scalability-2009-06-10-Dealing with multi-partition transactions in a distributed KV solution
15 0.085838139 1443 high scalability-2013-04-19-Stuff The Internet Says On Scalability For April 19, 2013
16 0.085011542 1650 high scalability-2014-05-19-A Short On How the Wayback Machine Stores More Pages than Stars in the Milky Way
17 0.081365511 1565 high scalability-2013-12-16-22 Recommendations for Building Effective High Traffic Web Software
18 0.080649272 115 high scalability-2007-10-07-Using ThreadLocal to pass context information around in web applications
19 0.075693935 389 high scalability-2008-09-23-How to Scale with Ruby on Rails
20 0.07440564 800 high scalability-2010-03-26-Strategy: Caching 404s Saved the Onion 66% on Server Time
topicId topicWeight
[(0, 0.062), (1, 0.042), (2, -0.026), (3, -0.067), (4, 0.016), (5, -0.009), (6, 0.036), (7, -0.031), (8, -0.011), (9, 0.014), (10, -0.019), (11, 0.017), (12, 0.009), (13, -0.023), (14, 0.006), (15, -0.001), (16, -0.002), (17, 0.007), (18, 0.007), (19, -0.019), (20, -0.004), (21, -0.019), (22, -0.028), (23, -0.002), (24, 0.014), (25, 0.044), (26, 0.013), (27, 0.009), (28, -0.001), (29, 0.004), (30, -0.028), (31, -0.019), (32, -0.058), (33, 0.047), (34, 0.054), (35, -0.027), (36, -0.021), (37, 0.081), (38, 0.017), (39, 0.019), (40, -0.006), (41, 0.011), (42, -0.013), (43, -0.098), (44, -0.016), (45, 0.033), (46, -0.023), (47, -0.022), (48, -0.052), (49, 0.025)]
simIndex simValue blogId blogTitle
same-blog 1 0.95054501 172 high scalability-2007-12-02-nginx: high performance smpt-pop-imap proxy
Introduction: nginx is a high performance smtp/pop/imap proxy that lets you do custom authorization and lookups and is very scalable. (just add nodes) Nginx by default is a reverse proxy and this is what it is doing here for pop/imap connections. It is also an excellelent reverse proxy for web servers. Advantage: You dont have to have a speacial database or ldap schema. Just an url to do auth and lookup with. A url that may be accessed by a unix or a tcp socket. Write your own auth handler - according to your own policy. For example: A user called atif tries to login with the pass testxyz. You pass this infomation to a URL such as socket:/var/tmp/xyz.sock or http://auth.corp.mailserver.net:someport/someurl The auth server replies with either a FAILURE such as Auth-Status: Invalid Login or password or with a success such as Auth-Status: OK Auth-Server: OneOfThe100Servers Auth-Port: optionalyAPort We have implemented it at our ISP and it has saves us a
2 0.68370229 314 high scalability-2008-05-03-Product: nginx
Introduction: Update 6 : nginx_http_push_module . Turn nginx into a long-polling message queuing HTTP push server. Update 5 : In Load Balancer Update Barry describes how WordPress.com moved from Pound to Nginx and are now "regularly serving about 8-9k requests/second and about 1.2Gbit/sec through a few Nginx instances and have plenty of room to grow!". Update 4 : Nginx better than Pound for load balancing. Pound spikes at 80% CPU, Nginx uses 3% and is easier to understand and better documented. Update 3 : igvita.com combines two cool tools together for better performance in Nginx and Memcached, a 400% boost! . Update 2 : Software Project on Installing Nginx Web Server w/ PHP and SSL . Breaking away from mother Apache can be a scary proposition and this kind of getting started article really helps easy the separation. Update: Slicehost has some nice tutorials on setting up Nginx . From their website: Nginx ("engine x") is a high-performance HTTP server and reverse proxy, as wel
3 0.58791155 290 high scalability-2008-03-28-How to Get DNS Names of a Web Server
Introduction: For some special reason, I'm trying to make a web server able to get all the DNS names mapped to its IP. Let me explain more, I'm creating a website that will run in a web farm, every web server in the farm will have some subdomains mapped to its ip, what I want is that whenever my application starts on a web server is to be able to get all the subdomains mapped/assigned to that server, e.g. sub1.mydomain.com, sub2.mydomain.com. I understand that I have to use reverse dns lookup (i.e. give the IP get the domain name), but I also want to get all the subdomains not just the first one that maps to that IP. I've been reading about DNS on the internet but I don't seem to find any information on how to achieve what I want, normally you use dns to get the ip of a domain but I'm not sure that all servers enable reverse lookup. The problem is that I'm still not sure whether I'll host my own DNS server or use the services of some company (many companies offer DNS hosting services), so, my qu
4 0.56065696 74 high scalability-2007-08-23-Product: Varnish
Introduction: Varnish is a state-of-the-art, high-performance HTTP accelerator. Varnish is targeted primarily at the FreeBSD 6 and Linux 2.6 platforms, and will take full advantage of the virtual memory system and advanced I/O features offered by these operating systems. Varnish was written from the ground up to be a high performance caching reverse proxy. Squid is a forward proxy that can be configured as a reverse proxy. Besides - Squid is rather old and designed like computer programs where supposed to be designed in 1980. Varnish is reported to be 10x-20x faster than Squid on the same hardware.
5 0.55235642 228 high scalability-2008-01-28-Product: ISPMan Centralized ISP Management System
Introduction: From FRESH Ports and their website: ISPman is an ISP management software written in perl, using an LDAP backend to manage virtual hosts for an ISP. It can be used to manage, DNS, virtual hosts for apache config, postfix configuration, cyrus mail boxes, proftpd etc. ISPMan was written as a management tool for the network at 4unet where between 30 to 50 domains are hosted and the number is crazily growing. Managing these domains and their users was a little time consuming, and needed an Administrator who knows linux and these daemons fluently. Now the help-desk can easily manage the domains and users. LDAP data can be easily replicated site wide, and mail box server can be scaled from 1 to n as required. An LDAP entry called maildrop tells the SMTP server (postfix) where to deliver the mail. The SMTP servers can be loadbalanced with one of many load balancing techniques. The program is written with scalability and High availability in mind. This may not be the right s
6 0.53700221 996 high scalability-2011-02-28-A Practical Guide to Varnish - Why Varnish Matters
7 0.52902275 1579 high scalability-2014-01-14-SharePoint VPS solution
8 0.52860576 800 high scalability-2010-03-26-Strategy: Caching 404s Saved the Onion 66% on Server Time
10 0.51480979 297 high scalability-2008-04-05-Skype Plans for PostgreSQL to Scale to 1 Billion Users
11 0.51365352 80 high scalability-2007-09-06-Product: Perdition Mail Retrieval Proxy
12 0.50291288 1517 high scalability-2013-09-16-The Hidden DNS Tax - Cascading Timeouts and Errors
13 0.4985781 208 high scalability-2008-01-11-FTP Sanity: Redundancy, archiving, consolidation.
14 0.49772099 286 high scalability-2008-03-20-Paper: Asynchronous HTTP and Comet architectures
15 0.49270284 255 high scalability-2008-02-21-Product: Capistrano - Automate Remote Tasks Via SSH
16 0.4847618 461 high scalability-2008-12-05-Sprinkle - Provisioning Tool to Build Remote Servers
17 0.48242542 115 high scalability-2007-10-07-Using ThreadLocal to pass context information around in web applications
18 0.47113496 1501 high scalability-2013-08-13-In Memoriam: Lavabit Architecture - Creating a Scalable Email Service
19 0.46971115 253 high scalability-2008-02-19-Building a email communication system
20 0.46748054 481 high scalability-2009-01-02-Strategy: Understanding Your Data Leads to the Best Scalability Solutions
topicId topicWeight
[(2, 0.103), (10, 0.161), (76, 0.376), (79, 0.078), (94, 0.138)]
simIndex simValue blogId blogTitle
same-blog 1 0.7848016 172 high scalability-2007-12-02-nginx: high performance smpt-pop-imap proxy
Introduction: nginx is a high performance smtp/pop/imap proxy that lets you do custom authorization and lookups and is very scalable. (just add nodes) Nginx by default is a reverse proxy and this is what it is doing here for pop/imap connections. It is also an excellelent reverse proxy for web servers. Advantage: You dont have to have a speacial database or ldap schema. Just an url to do auth and lookup with. A url that may be accessed by a unix or a tcp socket. Write your own auth handler - according to your own policy. For example: A user called atif tries to login with the pass testxyz. You pass this infomation to a URL such as socket:/var/tmp/xyz.sock or http://auth.corp.mailserver.net:someport/someurl The auth server replies with either a FAILURE such as Auth-Status: Invalid Login or password or with a success such as Auth-Status: OK Auth-Server: OneOfThe100Servers Auth-Port: optionalyAPort We have implemented it at our ISP and it has saves us a
2 0.66020334 966 high scalability-2010-12-31-Facebook in 20 Minutes: 2.7M Photos, 10.2M Comments, 4.6M Messages
Introduction: To celebrate the new year Facebook has shared the results of a little end of the year introspection. It has been a fecund year for Facebook: 43,869,800 changed their status to single 3,025,791 changed their status to "it's complicated" 28,460,516 changed their status to in a relationship 5,974,574 changed their status to engaged 36,774,801 changes their status to married If these numbers are simply to large to grasp, it doesn't get any better when you look at happens in a mere 20 minutes: Shared links: 1,000,000 Tagged photos: 1,323,000 Event invites sent out: 1,484,000 Wall Posts: 1,587,000 Status updates: 1,851,000 Friend requests accepted: 1,972,000 Photos uploaded: 2,716,000 Comments: 10,208,000 Message: 4,632,000 If you want to see how Facebook supports these huge numbers take a look at a few posts . One wonders what the new year will bring? Related Articles What the World Eats from Time Magazine A Day in the Life of an An
3 0.56045264 483 high scalability-2009-01-04-Paper: MapReduce: Simplified Data Processing on Large Clusters
Introduction: Update: MapReduce and PageRank Notes from Remzi Arpaci-Dusseau's Fall 2008 class . Collects interesting facts about MapReduce and PageRank. For example, the history of the solution to searching for the term "flu" is traced through multiple generations of technology. With Google entering the cloud space with Google AppEngine and a maturing Hadoop product, the MapReduce scaling approach might finally become a standard programmer practice. This is the best paper on the subject and is an excellent primer on a content-addressable memory future. Some interesting stats from the paper: Google executes 100k MapReduce jobs each day; more than 20 petabytes of data are processed per day; more than 10k MapReduce programs have been implemented; machines are dual processor with gigabit ethernet and 4-8 GB of memory. One common criticism ex-Googlers have is that it takes months to get up and be productive in the Google environment. Hopefully a way will be found to lower the learning curve a
4 0.45405111 813 high scalability-2010-04-19-The cost of High Availability (HA) with Oracle
Introduction: What's the cost of downtime to your business? $100,000 per hour, $1,000,000 or more? The recent Volcanic ash that has grounded European flights is estimated to be costing the airlines $200M a day. In the IT world, High Availability (HA) architectures allow for disaster recovery as well as uninterrupted business continuity during system failure. This post focuses on a customer’s backend, comprised of a business application stack supported by a dozen Oracle databases. They wish to equip this infrastructure with HA features and ensure that outages do not cost business. How do we address the challenge of pricing the complete solution, with hardware, software, services and annual support? Read more on BigDataMatters.com
5 0.43882984 1161 high scalability-2011-12-22-Architecting Massively-Scalable Near-Real-Time Risk Analysis Solutions
Introduction: Constructing a scalable risk analysis solution is a fascinating architectural challenge. If you come from Financial Services you are sure to appreciate that. But even architects from other domains are bound to find the challenges fascinating, and the architectural patterns of my suggested solution highly useful in other domains. Recently I held an interesting webinar around architecting solutions for scalable and near-real-time risk analysis solutions based on experience gathered with Financial Services customers. Seeing the vast interest in the webinar, I would like to share the highlights with you here. From an architectural point of view, risk analysis is a data-intensive and a compute-intensive process, which also has an elaborate orchestration logic. volumes in this domain are massive and ever-increasing, together with an ever-increasing demand to reduce response time. These trends are aggravated by global financial regulatory reforms set following the late-2000s
6 0.4386473 1066 high scalability-2011-06-22-It's the Fraking IOPS - 1 SSD is 44,000 IOPS, Hard Drive is 180
7 0.43824348 371 high scalability-2008-08-24-A Scalable, Commodity Data Center Network Architecture
8 0.43545413 1179 high scalability-2012-01-23-Facebook Timeline: Brought to You by the Power of Denormalization
9 0.43506163 1305 high scalability-2012-08-16-Paper: A Provably Correct Scalable Concurrent Skip List
10 0.43323919 1299 high scalability-2012-08-06-Paper: High-Performance Concurrency Control Mechanisms for Main-Memory Databases
12 0.42469916 430 high scalability-2008-10-26-Should you use a SAN to scale your architecture?
13 0.42327744 1585 high scalability-2014-01-24-Stuff The Internet Says On Scalability For January 24th, 2014
14 0.42223534 767 high scalability-2010-01-27-Hot Scalability Links for January 28 2010
15 0.42120403 1635 high scalability-2014-04-21-This is why Microsoft won. And why they lost.
16 0.41985959 1222 high scalability-2012-04-05-Big Data Counting: How to count a billion distinct objects using only 1.5KB of Memory
17 0.41870502 889 high scalability-2010-08-30-Pomegranate - Storing Billions and Billions of Tiny Little Files
18 0.41818225 1631 high scalability-2014-04-14-How do you even do anything without using EBS?
19 0.41678223 1223 high scalability-2012-04-06-Stuff The Internet Says On Scalability For April 6, 2012
20 0.4151302 20 high scalability-2007-07-16-Paper: The Clustered Storage Revolution