high_scalability high_scalability-2009 high_scalability-2009-683 knowledge-graph by maker-knowledge-mining
Source: html
Introduction: I have put down my thoughts in the architecture discussed in the blog . Although I have done substantial research to understand how things should work before deciding this architecture but I will be requiring huge amount of inputs from everyone to come to an architecture decision. Hardware entities which were thought while designing the entities are: 1. Master Web Server which will map different users to web servers placed in different geographical locations. (will prefer storing a mapping table in RAM) 2. Web Servers 3. Application Servers 4. Master Database Servers (to implement entity wise look up sharding) 5. Slave Database Servers. Will really appreciate if some good inputs of using Cloud Computing are given and how to go about it against or in addition to the given architecture. Would like to in fact know people's view on when to decide using cloud computing techniques. Looking forward for inputs from the community.
sentIndex sentText sentNum sentScore
1 I have put down my thoughts in the architecture discussed in the blog . [sent-1, score-0.487]
2 Although I have done substantial research to understand how things should work before deciding this architecture but I will be requiring huge amount of inputs from everyone to come to an architecture decision. [sent-2, score-1.761]
3 Hardware entities which were thought while designing the entities are: 1. [sent-3, score-0.809]
4 Master Web Server which will map different users to web servers placed in different geographical locations. [sent-4, score-0.733]
5 Master Database Servers (to implement entity wise look up sharding) 5. [sent-8, score-0.418]
6 Will really appreciate if some good inputs of using Cloud Computing are given and how to go about it against or in addition to the given architecture. [sent-10, score-1.026]
7 Would like to in fact know people's view on when to decide using cloud computing techniques. [sent-11, score-0.502]
wordName wordTfidf (topN-words)
[('inputs', 0.477), ('entities', 0.313), ('master', 0.19), ('wise', 0.189), ('inthe', 0.174), ('deciding', 0.165), ('appreciate', 0.162), ('substantial', 0.156), ('geographical', 0.151), ('given', 0.151), ('discussed', 0.147), ('entity', 0.144), ('placed', 0.144), ('mapping', 0.143), ('prefer', 0.142), ('thoughts', 0.142), ('architecture', 0.134), ('requiring', 0.133), ('slave', 0.132), ('although', 0.128), ('computing', 0.123), ('decide', 0.122), ('forward', 0.119), ('map', 0.113), ('research', 0.102), ('sharding', 0.101), ('designing', 0.101), ('web', 0.096), ('storing', 0.092), ('fact', 0.092), ('community', 0.091), ('table', 0.09), ('cloud', 0.085), ('implement', 0.085), ('addition', 0.085), ('understand', 0.084), ('everyone', 0.084), ('thought', 0.082), ('view', 0.08), ('different', 0.077), ('huge', 0.077), ('amount', 0.077), ('servers', 0.075), ('ram', 0.071), ('come', 0.071), ('done', 0.067), ('database', 0.064), ('put', 0.064), ('looking', 0.062), ('hardware', 0.057)]
simIndex simValue blogId blogTitle
same-blog 1 1.0000001 683 high scalability-2009-08-18-Hardware Architecture Example (geographical level mapping of servers)
Introduction: I have put down my thoughts in the architecture discussed in the blog . Although I have done substantial research to understand how things should work before deciding this architecture but I will be requiring huge amount of inputs from everyone to come to an architecture decision. Hardware entities which were thought while designing the entities are: 1. Master Web Server which will map different users to web servers placed in different geographical locations. (will prefer storing a mapping table in RAM) 2. Web Servers 3. Application Servers 4. Master Database Servers (to implement entity wise look up sharding) 5. Slave Database Servers. Will really appreciate if some good inputs of using Cloud Computing are given and how to go about it against or in addition to the given architecture. Would like to in fact know people's view on when to decide using cloud computing techniques. Looking forward for inputs from the community.
2 0.17010161 327 high scalability-2008-05-27-How I Learned to Stop Worrying and Love Using a Lot of Disk Space to Scale
Introduction: Update 3 : ReadWriteWeb says Google App Engine Announces New Pricing Plans, APIs, Open Access . Pricing is specified but I'm not sure what to make of it yet. An image manipulation library is added (thus the need to pay for more CPU :-) and memcached support has been added. Memcached will help resolve the can't write for every read problem that pops up when keeping counters. Update 2 : onGWT.com threw a GAE load party and a lot of people came. The results at Load test : Google App Engine = 1, Community = 0 . GAE handled a peak of 35 requests/second and a sustained 10 requests/second. Some think performance was good, others not so good. My GMT watch broke and I was late to arrive. Maybe next time. Also added a few new design rules from the post. Update : Added a few new rules gleaned from the GAE Meetup : Design By Explicit Cost Model and Puts are Precious. How do you structure your database using a distributed hash table like BigTable ? The answer isn't what you might expect. If
3 0.13855159 279 high scalability-2008-03-17-Microsoft's New Database Cloud Ready to Rumble with Amazon
Introduction: Update: Zdnet says Ozzie signals Microsoft’s surrender to the cloud . CD ROMs are to the internet as the internet is to the cloud and Microsoft aims to scratch and claw its way into this paradigm shift as well. The gloves are off. The tag line for Microsoft's new SQL Server Data Service is Your Data, Any Place, Any Time . Thems fighten' words. Microsoft is itch'n for a fight! Who will be Amazon's second? The service description: SQL Server Data Services (SSDS) are highly scalable, on-demand data storage and query processing utility services. Built on robust SQL Server database and Windows Server technologies, these services provide high availability, security and support standards-based web interfaces for easy programming and quick provisioning. Sounds like a fast uppercut aimed squarely at SimpleDB's jaw. As a developer what do you need to know? Highly available and highly scalable. Targeted at applications that can tolerate high internet latencies. Pr
4 0.12751362 958 high scalability-2010-12-16-7 Design Patterns for Almost-infinite Scalability
Introduction: Good article from manageability.com  summarizing design patterns from Pat Helland's amazing paper Life beyond Distributed Transactions: an Apostate's Opinion . Entities are uniquely identified - each entity which represents disjoint data (i.e. no overlap of data between entities) should have a unique key. Multiple disjoint scopes of transactional serializability - in other words there are these 'entities' and that you cannot perform atomic transactions across these entities. At-Least-Once messaging - that is an application must tolerate message retries and out-of-order arrival of messages. Messages are adressed to entities - that is one can't abstract away from the business logic the existence of the unique keys for addresing entities. Addressing however is independent of location. Entities manage conversational state per party - that is, to ensure idemptency an entity needs to remember that a message has been previously processed. Furthermore, in a world with
5 0.12535483 516 high scalability-2009-02-19-Heavy upload server scalability
Introduction: Hi, We are running a backup solution that uploads every night the files our clients worked on during the day (Cabonite-like). We have currently about 10GB of data per night, via http PUT requests (1 per file), and the files are written as-is on a NAS. Our architecture is basically compound of a load balancer (hardware, sticky sessions), 5 servers (Tomcat under RHEL4/5, ) and a NAS (nfs 3). Since our number of clients is rising, (as is our system load) how would you recommend we could scale our infrastructure? hardware and software? Should we go towards NAS sharding, more servers, NIO on tomcat...? Thanks for your inputs!
6 0.11599361 155 high scalability-2007-11-15-Video: Dryad: A general-purpose distributed execution platform
7 0.11220644 672 high scalability-2009-08-06-An Unorthodox Approach to Database Design : The Coming of the Shard
8 0.10797106 1440 high scalability-2013-04-15-Scaling Pinterest - From 0 to 10s of Billions of Page Views a Month in Two Years
9 0.10738146 445 high scalability-2008-11-14-Useful Cloud Computing Blogs
10 0.10224775 538 high scalability-2009-03-16-Are Cloud Based Memory Architectures the Next Big Thing?
11 0.094054274 7 high scalability-2007-07-12-FeedBurner Architecture
12 0.093616977 63 high scalability-2007-08-09-Lots of questions for high scalability - high availability
13 0.091668978 301 high scalability-2008-04-08-Google AppEngine - A First Look
14 0.089794524 464 high scalability-2008-12-13-Strategy: Facebook Tweaks to Handle 6 Time as Many Memcached Requests
15 0.087589264 383 high scalability-2008-09-10-Shard servers -- go big or small?
16 0.086555108 325 high scalability-2008-05-25-How do you explain cloud computing to your grandma?
17 0.085066728 448 high scalability-2008-11-22-Google Architecture
18 0.085026972 1508 high scalability-2013-08-28-Sean Hull's 20 Biggest Bottlenecks that Reduce and Slow Down Scalability
20 0.082994118 750 high scalability-2009-12-16-Building Super Scalable Systems: Blade Runner Meets Autonomic Computing in the Ambient Cloud
topicId topicWeight
[(0, 0.144), (1, 0.045), (2, 0.028), (3, -0.016), (4, -0.0), (5, -0.009), (6, -0.023), (7, -0.077), (8, 0.005), (9, 0.023), (10, -0.018), (11, 0.01), (12, -0.039), (13, 0.032), (14, 0.046), (15, -0.019), (16, -0.034), (17, 0.003), (18, 0.003), (19, 0.016), (20, -0.005), (21, -0.013), (22, -0.043), (23, -0.006), (24, 0.05), (25, -0.043), (26, 0.071), (27, 0.006), (28, 0.021), (29, 0.051), (30, 0.004), (31, -0.05), (32, -0.027), (33, 0.024), (34, 0.016), (35, 0.024), (36, 0.08), (37, -0.046), (38, -0.014), (39, 0.006), (40, -0.023), (41, 0.091), (42, -0.071), (43, -0.017), (44, 0.057), (45, -0.018), (46, 0.024), (47, -0.02), (48, 0.045), (49, 0.04)]
simIndex simValue blogId blogTitle
same-blog 1 0.93569249 683 high scalability-2009-08-18-Hardware Architecture Example (geographical level mapping of servers)
Introduction: I have put down my thoughts in the architecture discussed in the blog . Although I have done substantial research to understand how things should work before deciding this architecture but I will be requiring huge amount of inputs from everyone to come to an architecture decision. Hardware entities which were thought while designing the entities are: 1. Master Web Server which will map different users to web servers placed in different geographical locations. (will prefer storing a mapping table in RAM) 2. Web Servers 3. Application Servers 4. Master Database Servers (to implement entity wise look up sharding) 5. Slave Database Servers. Will really appreciate if some good inputs of using Cloud Computing are given and how to go about it against or in addition to the given architecture. Would like to in fact know people's view on when to decide using cloud computing techniques. Looking forward for inputs from the community.
2 0.67640233 435 high scalability-2008-10-30-The case for functional decomposition
Introduction: Hi all, I'm a big fan of http://highscalability.com/ and have been looking in my current development to decompose my application along functional boundaries as a route to being able to scale out the server side, specifically the database layer. The problem comes when there are links between the data in different components, ie one component holds all the user data, but another component needs to reference a user as being an owner of some piece of data. I'm currently doing this by holding the primary key information for each side of the the link (as you would if they all lived in a single database), but this link table needs to exist in both components to allow lookups to be done in either direction, ie 'get the things a specific user owns' and 'get the owners of this specific thing' would each use different components. The alternative to this would be to store the link data in only one of the components, but then the reverse lookups would require 2 calls instead of just one. M
3 0.64284194 532 high scalability-2009-03-11-Sharding and Connection Pools
Introduction: Hi we are looking at sharding our existing Java/Oracle based application. We are looking to make the app servers able to process requests for multiple (any?) shard. The concern that has come up is the amount of memory that would be consumed by having so many connection pools on one app server. Additionally there is concern about having so many physical connections to the database server coming from all the various app servers that may talk to that particular shard. I was wondering if anyone else has dealt with this issue and how you resolved it? Thanks, Scott
4 0.63029969 178 high scalability-2007-12-10-1 Master, N Slaves
Introduction: Hello all, Reading the site you can note that "1 Master for writes, N Slaves for reads" scheme is used offen. How is this implemented? Who decides where writes and reads go? Something in application level or specific database proxies, like Slony-I? Thanks.
5 0.59475267 675 high scalability-2009-08-08-1dbase vs. many and cloud hosting vs. dedicated server(s)?
Introduction: Me and my partner are making a blueprint for an online webshop service. The purpose of this project is to make webshops available for small company's/ individuals automatically just by creating an account with us. Our webapp can be used to add products/pages/... to the store and we'll handle secure checkout by paypal. Our app should be scalable and manageable. Because we also want to offer free webshops, the amount of webshops could be +10.000 within a few years. We are building on the Zend framework and are using mysql for database. From the start we want to build our application for optimal and easy scalability in the future, to avoid a lot changes to our app/database in the future. Now our questions are: Should we use?: * one database for all shops (or limited to X shops ); * one database for each new shop (each having products, orders... tables); I think both approaches have PRO/CONS. What do you think ? Does anyone has experience with this kind of structure ? PRO: one datab
6 0.59387851 451 high scalability-2008-11-30-Creating a high-performing online database
7 0.58335733 383 high scalability-2008-09-10-Shard servers -- go big or small?
8 0.57901919 620 high scalability-2009-06-05-SSL RPC API Scalability
9 0.57828319 177 high scalability-2007-12-08-thesimsonstage.ea.com
10 0.57367563 798 high scalability-2010-03-22-7 Secrets to Successfully Scaling with Scalr (on Amazon) by Sebastian Stadil
11 0.57322001 369 high scalability-2008-08-18-Code deployment tools
12 0.57206511 266 high scalability-2008-03-04-Manage Downtime Risk by Connecting Multiple Data Centers into a Secure Virtual LAN
13 0.5704152 561 high scalability-2009-04-08-N+1+caching is ok?
14 0.56721163 880 high scalability-2010-08-13-Hot Scalability Links for Aug 13, 2010
15 0.56200635 953 high scalability-2010-12-03-GPU vs CPU Smackdown : The Rise of Throughput-Oriented Architectures
16 0.55979347 672 high scalability-2009-08-06-An Unorthodox Approach to Database Design : The Coming of the Shard
17 0.55912471 514 high scalability-2009-02-18-Numbers Everyone Should Know
18 0.55650121 1507 high scalability-2013-08-26-Reddit: Lessons Learned from Mistakes Made Scaling to 1 Billion Pageviews a Month
19 0.55619371 222 high scalability-2008-01-25-Application Database and DAL Architecture
20 0.55526012 671 high scalability-2009-08-05-Stack Overflow Architecture
topicId topicWeight
[(1, 0.143), (2, 0.174), (10, 0.072), (47, 0.107), (51, 0.04), (61, 0.08), (79, 0.13), (87, 0.111), (94, 0.026)]
simIndex simValue blogId blogTitle
same-blog 1 0.94028324 683 high scalability-2009-08-18-Hardware Architecture Example (geographical level mapping of servers)
Introduction: I have put down my thoughts in the architecture discussed in the blog . Although I have done substantial research to understand how things should work before deciding this architecture but I will be requiring huge amount of inputs from everyone to come to an architecture decision. Hardware entities which were thought while designing the entities are: 1. Master Web Server which will map different users to web servers placed in different geographical locations. (will prefer storing a mapping table in RAM) 2. Web Servers 3. Application Servers 4. Master Database Servers (to implement entity wise look up sharding) 5. Slave Database Servers. Will really appreciate if some good inputs of using Cloud Computing are given and how to go about it against or in addition to the given architecture. Would like to in fact know people's view on when to decide using cloud computing techniques. Looking forward for inputs from the community.
2 0.91945809 1054 high scalability-2011-06-06-NoSQL Pain? Learn How to Read-write Scale Without a Complete Re-write
Introduction: Lately I've been reading more cases were different people have started to realize the limitations of the NoSQL promise to database scalability. Note the references below: Why does Quora use MySQL as the data store instead of NoSQLs such as Cassandra, MongoDB, CouchDB etc? Why did Diaspora abandon MongoDB for MySQL? How scalable is CouchDB in practice, not just in theory? Take MongoDB for example. It's damn fast, but it doesn't really know how to save data reliably to disk. I've had it set up in a replica pair to mitigate that risk. Guess what - both servers in the pair failed and corrupted their data files at the same day. It appears that for many, the switch to NoSQL can be rather painful. IMO that doesn't necessarily mean that NoSQL is wrong in general, but it's a combination of 1) lack of maturity 2) not the right tool for the job. That brings the question of what's the alternative solution? In the following post I tried to summarize the lessons from
3 0.90581328 144 high scalability-2007-11-07-What CDN would you recommend?
Introduction: Hi all, a I run a site that after a complete redesign have gotten a lot more traffic. The site provides free flash games, so the biggest traffic share goes to serving flash files (from about 100K and up to several megabytes in size each.) I currently host the entire site on a hosting provider that have no traffic limits. But since they are very cheap (yet have served me very well all the time with at least 99,9% uptime), I don't trust them in allowing me to continue consuming more and more bandwidth. I just guess I'm going to reach some internal limit they have on day, so I'm looking into moving all the flash content over to a content delivery network of some sort. Some recent traffic stats: August: 12 GB September: 22 GB October: 55 GB November: Currently 2,3 GB pr day on average, but it's rising.. I've been looking into Amazon S3, but have not decided on anything yet. So therefor I'm asking if there are any other provides I should consider, that operates withi
4 0.90572882 760 high scalability-2010-01-13-10 Hot Scalability Links for January 13, 2010
Introduction: Has Amazon EC2 become over subscribed? by Alan Williamson. Systemic problems hit AWS as users experience problems across Amazon's infrastructure. It seems the strange attractor of a cloud may be the same as for a shared hosting service. Understanding Infrastructure 2.0 by James Urquhart. We need to take a systems view of our entire infrastructure, and build our automation around the end-to-end architecture of that system . Hey You, Get Off of My Cloud: Exploring Information Leakage in Third-Party Compute Clouds . We show that it is possible to map the internal cloud infrastructure . Hadoop World: Building Data Intensive Apps with Hadoop and EC2 by Pete Skomoroch. Dives into detail about how he built TrendingTopics.org using Hadoop and EC2 . A Crash Course in Modern Hardware by Cliff Click. Yes, your mind will hurt after watching this. And no, you probably don't know what your microprocessor is doing anymore. EVE Scalability Explained by James Harrison. This pos
5 0.9054659 676 high scalability-2009-08-08-Yahoo!'s PNUTS Database: Too Hot, Too Cold or Just Right?
Introduction: So far every massively scalable database is a bundle of compromises. For some the weak guarantees of Amazon's eventual consistency model are too cold. For many the strong guarantees of standard RDBMS distributed transactions are too hot. Google App Engine tries to get it just right with entity groups . Yahoo! is also trying to get is just right by offering per-record timeline consistency, which hopes to serve up a heaping bowl of rich database functionality and low latency at massive scale : We describe PNUTS [Platform for Nimble Universal Table Storage], a massively parallel and geographically distributed database system for Yahoo!’s web applications. PNUTS provides data storage organized as hashed or ordered tables, low latency for large numbers of con-current requests including updates and queries, and novel per-record consistency guarantees. It is a hosted, centrally managed, and geographically distributed service, and utilizes automated load-balancing and failover to redu
6 0.90280378 852 high scalability-2010-07-07-Strategy: Recompute Instead of Remember Big Data
7 0.89699185 1062 high scalability-2011-06-15-101 Questions to Ask When Considering a NoSQL Database
8 0.8935656 1326 high scalability-2012-09-20-How Vimeo Saves 50% on EC2 by Playing a Smarter Game
9 0.88557422 1526 high scalability-2013-10-02-RFC 1925 - The Twelve (Timeless) Networking Truths
10 0.88361216 1166 high scalability-2011-12-30-Stuff The Internet Says On Scalability For December 30, 2011
11 0.88345504 906 high scalability-2010-09-22-Applying Scalability Patterns to Infrastructure Architecture
12 0.8814835 1530 high scalability-2013-10-11-Stuff The Internet Says On Scalability For October 11th, 2013
13 0.87463212 1235 high scalability-2012-04-27-Stuff The Internet Says On Scalability For April 27, 2012
14 0.87410104 1440 high scalability-2013-04-15-Scaling Pinterest - From 0 to 10s of Billions of Page Views a Month in Two Years
15 0.87300318 590 high scalability-2009-05-06-Art of Distributed
16 0.87137002 1180 high scalability-2012-01-24-The State of NoSQL in 2012
17 0.87125713 1186 high scalability-2012-02-02-The Data-Scope Project - 6PB storage, 500GBytes-sec sequential IO, 20M IOPS, 130TFlops
18 0.86940747 1654 high scalability-2014-06-05-Cloud Architecture Revolution
19 0.86857605 327 high scalability-2008-05-27-How I Learned to Stop Worrying and Love Using a Lot of Disk Space to Scale
20 0.86769652 1344 high scalability-2012-10-19-Stuff The Internet Says On Scalability For October 19, 2012