high_scalability high_scalability-2008 high_scalability-2008-222 knowledge-graph by maker-knowledge-mining
Source: html
Introduction: Hi gurus, I'm totally new to this high scalability thing. I'm trying to create a website with scalability in mind (personal project). In my application I'll have forums for different groups of people (each group will have their own forums, members of groups can still post in other groups' forums but each group will mainly be using their forums most of the time). Now, I'm going to start with about 2000 groups with the potential of reaching up to 10000 groups (this is the maximum due to the nature of my application). I was thinking that having all posts in one table will be way too much for one table (esp. that some groups are expected to post hundreds or even thousands times per day, let's say about 500 of the groups, the rest of the groups won't be that active though) as I'll have to index the PostID, ParentPostID, GroupID and PostDate which can produce large indexes (consequentially causing slow inserts) if having everything in one table. So, I'm thinking of a way to divide the posts
sentIndex sentText sentNum sentScore
1 Hi gurus, I'm totally new to this high scalability thing. [sent-1, score-0.116]
2 I'm trying to create a website with scalability in mind (personal project). [sent-2, score-0.057]
3 In my application I'll have forums for different groups of people (each group will have their own forums, members of groups can still post in other groups' forums but each group will mainly be using their forums most of the time). [sent-3, score-2.564]
4 Now, I'm going to start with about 2000 groups with the potential of reaching up to 10000 groups (this is the maximum due to the nature of my application). [sent-4, score-1.145]
5 I was thinking that having all posts in one table will be way too much for one table (esp. [sent-5, score-0.522]
6 So, I'm thinking of a way to divide the posts in many tables, here are some of the things I thought of: 1. [sent-7, score-0.263]
7 ForumsPosts_x, where x is the GroupID (which has its own pros and cons, some of the pros that I can have small indexes and also use identity columns, I also assume it should be easy to move the tables to other databases should the application grow. [sent-10, score-1.023]
8 Well, I posted this idea on some other forums and most people told me it's a sign of bad design if I have thousands of tables in my database. [sent-11, score-0.81]
9 I was also concerned how to design my DAL if I do this. [sent-12, score-0.2]
10 Should I use sprocs with dynamic SQL or use SQL text directly in my DAL code and what about the query plan caching if having a large number of tables . [sent-13, score-0.24]
11 Put everything in one table and if the site grows move some of the groups to another database (I'm concerned though about having many databases on the same machine, will it affect performance? [sent-17, score-1.273]
12 of course I won't have hundreds of databases on the same machine but may be about 5 or even 10 databases on the same machine) I also have some other questions: I'm going to use ASP. [sent-18, score-0.504]
13 NET for this project, I was planning initially to use SQL Server as a database but I'm worried about the SQL Server part and the cost of growth, should I consider an alternative like MySQL? [sent-19, score-0.15]
wordName wordTfidf (topN-words)
[('groups', 0.484), ('forums', 0.408), ('dal', 0.268), ('groupid', 0.268), ('tables', 0.186), ('table', 0.167), ('pros', 0.161), ('concerned', 0.143), ('sql', 0.139), ('group', 0.117), ('databases', 0.113), ('though', 0.108), ('posts', 0.106), ('gurus', 0.102), ('indexes', 0.096), ('machine', 0.087), ('worried', 0.084), ('hi', 0.083), ('thinking', 0.082), ('cons', 0.081), ('hundreds', 0.079), ('affect', 0.076), ('project', 0.076), ('inserts', 0.076), ('columns', 0.076), ('thousands', 0.075), ('divide', 0.075), ('posted', 0.074), ('mainly', 0.072), ('wo', 0.072), ('suggestions', 0.071), ('causing', 0.071), ('identity', 0.069), ('told', 0.067), ('members', 0.066), ('initially', 0.066), ('reaching', 0.066), ('move', 0.065), ('personal', 0.064), ('grows', 0.062), ('scenario', 0.061), ('produce', 0.06), ('totally', 0.059), ('assume', 0.058), ('also', 0.057), ('scalability', 0.057), ('maximum', 0.056), ('going', 0.055), ('everything', 0.055), ('text', 0.054)]
simIndex simValue blogId blogTitle
same-blog 1 1.0 222 high scalability-2008-01-25-Application Database and DAL Architecture
Introduction: Hi gurus, I'm totally new to this high scalability thing. I'm trying to create a website with scalability in mind (personal project). In my application I'll have forums for different groups of people (each group will have their own forums, members of groups can still post in other groups' forums but each group will mainly be using their forums most of the time). Now, I'm going to start with about 2000 groups with the potential of reaching up to 10000 groups (this is the maximum due to the nature of my application). I was thinking that having all posts in one table will be way too much for one table (esp. that some groups are expected to post hundreds or even thousands times per day, let's say about 500 of the groups, the rest of the groups won't be that active though) as I'll have to index the PostID, ParentPostID, GroupID and PostDate which can produce large indexes (consequentially causing slow inserts) if having everything in one table. So, I'm thinking of a way to divide the posts
2 0.2553187 22 high scalability-2007-07-23-Weblink Template
Introduction: Information Sources Platform What's Inside? The Stats Lessons Learned To discuss this article please visit the forums at
3 0.15250207 894 high scalability-2010-09-03-Six guiding principles to Consolidate your IT
Introduction: The need for IT consolidation is most evident in two types of organizations. In the first group, IT grew organically with business over the decades, and survived changes of strategy, management, staff and vendor orientation. The second group of businesses capital groups are characterized by rapid growth through acquisitions (followed by attempts to integrate radically different IT environments). In both groups, their IT infrastructures have typically been pieced together over the past 20 (or more) years.ďťż Read more on BigDataMatters.com
4 0.14159076 239 high scalability-2008-02-04-Streaming Video on Amazon EC2?
Introduction: An Amazon EC2 Flash Video Streaming solution has been announced by Wowza Media. What do you think about the future of similar solutions? Is Amazon EC2 and S3 ready for video streaming? I have found threads on their forums related to the performance, scalability and high availability of the hosted streaming solution. How would you make it scalable? Is it really cheaper than traditional hosting? Looking forward to your thoughts!
5 0.14006634 972 high scalability-2011-01-11-Google Megastore - 3 Billion Writes and 20 Billion Read Transactions Daily
Introduction: A giant step into the fully distributed future has been taken by the Google App Engine team with the release of their High Replication Datastore . The HRD is targeted at mission critical applications that require data replicated to at least three datacenters, full ACID semantics for entity groups , and lower consistency guarantees across entity groups. This is a major accomplishment. Few organizations can implement a true multi-datacenter datastore. Other than SimpleDB, how many other publicly accessible database services can operate out of multiple datacenters? Now that capability can be had by anyone. But there is a price, literally and otherwise. Because the HRD uses three times the resources as Google App Engine's Master/Slave datastatore, it will cost three times as much. And because it is a distributed database, with all that implies in the CAP sense, developers will have to be very careful in how they architect their applications because as costs increased, reliability incre
6 0.12877464 531 high scalability-2009-03-11-Classifying XTP systems and how cloud changes which type startups will use
8 0.10768472 70 high scalability-2007-08-22-How many machines do you need to run your site?
9 0.10716639 829 high scalability-2010-05-20-Strategy: Scale Writes to 734 Million Records Per Day Using Time Partitioning
10 0.10679035 1281 high scalability-2012-07-11-FictionPress: Publishing 6 Million Works of Fiction on the Web
11 0.10654198 961 high scalability-2010-12-21-SQL + NoSQL = Yes !
12 0.10319018 342 high scalability-2008-06-08-Search fast in million rows
13 0.10171022 672 high scalability-2009-08-06-An Unorthodox Approach to Database Design : The Coming of the Shard
14 0.096317463 200 high scalability-2008-01-02-WEB hosting Select
15 0.095420413 315 high scalability-2008-05-05-HSCALE - Handling 200 Million Transactions Per Month Using Transparent Partitioning With MySQL Proxy
16 0.09462525 1163 high scalability-2011-12-23-Stuff The Internet Says On Scalability For December 23, 2011
17 0.094039075 351 high scalability-2008-07-16-The Mother of All Database Normalization Debates on Coding Horror
18 0.093741223 1529 high scalability-2013-10-08-F1 and Spanner Holistically Compared
19 0.092233211 281 high scalability-2008-03-18-Database Design 101
20 0.090195686 231 high scalability-2008-01-29-Too many databases
topicId topicWeight
[(0, 0.146), (1, 0.084), (2, -0.022), (3, -0.038), (4, 0.031), (5, 0.027), (6, -0.047), (7, -0.059), (8, 0.027), (9, -0.039), (10, -0.037), (11, -0.012), (12, -0.042), (13, 0.034), (14, 0.073), (15, -0.034), (16, -0.013), (17, -0.033), (18, -0.006), (19, 0.015), (20, -0.0), (21, -0.057), (22, -0.061), (23, -0.019), (24, 0.026), (25, -0.006), (26, -0.012), (27, -0.058), (28, 0.009), (29, -0.007), (30, -0.007), (31, 0.037), (32, 0.001), (33, 0.058), (34, -0.006), (35, 0.068), (36, 0.064), (37, -0.023), (38, 0.009), (39, 0.004), (40, 0.024), (41, -0.006), (42, -0.027), (43, -0.065), (44, -0.02), (45, -0.0), (46, 0.008), (47, 0.035), (48, 0.037), (49, 0.013)]
simIndex simValue blogId blogTitle
same-blog 1 0.9609983 222 high scalability-2008-01-25-Application Database and DAL Architecture
Introduction: Hi gurus, I'm totally new to this high scalability thing. I'm trying to create a website with scalability in mind (personal project). In my application I'll have forums for different groups of people (each group will have their own forums, members of groups can still post in other groups' forums but each group will mainly be using their forums most of the time). Now, I'm going to start with about 2000 groups with the potential of reaching up to 10000 groups (this is the maximum due to the nature of my application). I was thinking that having all posts in one table will be way too much for one table (esp. that some groups are expected to post hundreds or even thousands times per day, let's say about 500 of the groups, the rest of the groups won't be that active though) as I'll have to index the PostID, ParentPostID, GroupID and PostDate which can produce large indexes (consequentially causing slow inserts) if having everything in one table. So, I'm thinking of a way to divide the posts
2 0.7746098 675 high scalability-2009-08-08-1dbase vs. many and cloud hosting vs. dedicated server(s)?
Introduction: Me and my partner are making a blueprint for an online webshop service. The purpose of this project is to make webshops available for small company's/ individuals automatically just by creating an account with us. Our webapp can be used to add products/pages/... to the store and we'll handle secure checkout by paypal. Our app should be scalable and manageable. Because we also want to offer free webshops, the amount of webshops could be +10.000 within a few years. We are building on the Zend framework and are using mysql for database. From the start we want to build our application for optimal and easy scalability in the future, to avoid a lot changes to our app/database in the future. Now our questions are: Should we use?: * one database for all shops (or limited to X shops ); * one database for each new shop (each having products, orders... tables); I think both approaches have PRO/CONS. What do you think ? Does anyone has experience with this kind of structure ? PRO: one datab
3 0.74585819 451 high scalability-2008-11-30-Creating a high-performing online database
Introduction: Hi there, I have an idea for an online database that services a large number of people. I've been studying it for a while and it seems feasible to me to create it and get people to populate it. It will need time to grow but eventually it will get there. The model I'm looking at is IMDB, the depth of information is fascinating, yet it's fast, not so easy to use though, but it's pretty usable! What do you think I need to create a database an online database like IMDB. I know that IMDB power comes from it's information, not the design of the site. This is something I kind of figured out. But what I need to know is the best tools to publish database contents on the web, retrieve it in that fast way like IMDB. I'm sure that I will need to create data entry logs for my users to populate the database. What programming languages you suggest? development environment? approaches? your contribution is highly appreciated. Regards, Jalil
4 0.7403819 1065 high scalability-2011-06-21-Running TPC-C on MySQL-RDS
Introduction: I recently came across a TPC-C benchmark results held on MySQL based RDS databases. You can see it here . I think the results may bring light to many questions concerning MySQL scalability in general and RDS scalability in particular. (For disclosure, I'm working for ScaleBase where we run an internal scale out TPC-C benchmark these days, and will publish results soon). TPC-C TPC-C is a standard database benchmark, used to measure databases. The database vendors invest big bucks in running this test, and showing off which database is faster, and can scale better. It is a write intensive test, so it doesn’t necessarily reflect the behavior of the database in your application. But it does give some very important insights on what you can expect from your database under heavy load. The Benchmark Process First of all, I have some comments for the benchmark method itself. Generally - the benchmarks were held in an orderly fashion and in a rather methodological way – which i
5 0.73737854 379 high scalability-2008-09-04-Database question for upcoming project
Introduction: We will be developing an RIA that will have a lot of database access. Think something like a QuickBooks but with about 50 transactions entered per hour per user. Users will be in the system for 7 to 9 hours a day and there will be around 20,000 users, all logged in at the same time. Reporting will be done just like a QuickBooks style app plus a lot of extra things you don't do in QuickBooks. Our operations is familiar with W2003 Server and MS SQL Server so they are recommending we stick with that. I originally requested Linux and PostgreSQL. How far can a single database server get me? If we have a 4 processor, 8 core, 128gb server, how far am I going to get before I need to shard or do something else? I know there are a lot of factors involved but in general for this size of a site, what should the strategy be? I've read almost all articles on this website but most of the applications are not RIA type of apps with this type of usage or they are architectures for
6 0.72283721 231 high scalability-2008-01-29-Too many databases
7 0.71731365 1288 high scalability-2012-07-23-Ask HighScalability: How Do I Build My MegaUpload + Itunes + YouTube Startup?
8 0.71103287 654 high scalability-2009-07-09-No to SQL? Anti-database movement gains steam – My Take
9 0.70608318 277 high scalability-2008-03-16-Do you have any questions for the Elastra CEO?
10 0.7025317 435 high scalability-2008-10-30-The case for functional decomposition
11 0.68991882 671 high scalability-2009-08-05-Stack Overflow Architecture
12 0.67759109 361 high scalability-2008-08-08-Separation into read-write only databases
13 0.67494375 578 high scalability-2009-04-23-Which Key value pair database to be used
14 0.67255014 256 high scalability-2008-02-21-Tracking usage of public resources - throttling accesses per hour
15 0.66522729 236 high scalability-2008-02-03-Ideas on how to scale a shared inventory database???
16 0.66466349 151 high scalability-2007-11-12-a8cjdbc - Database Clustering via JDBC
17 0.66448641 276 high scalability-2008-03-15-New Website Design Considerations
18 0.66438681 281 high scalability-2008-03-18-Database Design 101
19 0.66437435 1025 high scalability-2011-04-16-The NewSQL Market Breakdown
20 0.66139483 995 high scalability-2011-02-24-Strategy: Eliminate Unnecessary SQL
topicId topicWeight
[(1, 0.143), (2, 0.095), (10, 0.028), (61, 0.23), (77, 0.245), (79, 0.143)]
simIndex simValue blogId blogTitle
same-blog 1 0.9156183 222 high scalability-2008-01-25-Application Database and DAL Architecture
Introduction: Hi gurus, I'm totally new to this high scalability thing. I'm trying to create a website with scalability in mind (personal project). In my application I'll have forums for different groups of people (each group will have their own forums, members of groups can still post in other groups' forums but each group will mainly be using their forums most of the time). Now, I'm going to start with about 2000 groups with the potential of reaching up to 10000 groups (this is the maximum due to the nature of my application). I was thinking that having all posts in one table will be way too much for one table (esp. that some groups are expected to post hundreds or even thousands times per day, let's say about 500 of the groups, the rest of the groups won't be that active though) as I'll have to index the PostID, ParentPostID, GroupID and PostDate which can produce large indexes (consequentially causing slow inserts) if having everything in one table. So, I'm thinking of a way to divide the posts
2 0.9049167 753 high scalability-2009-12-21-Hot Holiday Scalability Links for 2009
Introduction: Building Super Scalable Systems: Blade Runner Meets Autonomic Computing in the Ambient Cloud . The only independent platform most of us will have access to capable of hosting planet-scale applications is the Ambient Cloud. It forms a sort of digital potluck where everyone contributes memory, network, and other compute resources from whatever they happen to have available. Top 10 Internet Startup Scalability Killers . Strategies taken from The Art of Scalability . 1. Thinking Scalability Is Just About Technology; 2. Overuse of Synchronous Calls; 3. Failure to Weed or Seed Soon Enough; 4. Inappropriate Use of Databases; 5. Cesspools Instead of Swim Lanes; 6. Reliance on Vertical Scale; 7. Failure to Learn from History; 8. Changing Development Methodologies to Fix Problems; 9. Too Little Caching, Too Late; 10. Overreliance on Third Parties to Scale. The New Google: Internet Giant Opens Up About Real-Time and Local Search, Cloud Computing, and Data Liberation . In four separat
3 0.84731209 200 high scalability-2008-01-02-WEB hosting Select
Introduction: Hello, I am new to the back end side of things. Love this web site. Read all comments about Amazon hosting, actually I really like Amazon S3 but concerned that it may not be sufficient for my computing needs. And E3 just not too sure. What about hosting sites like host monster? Their prices seem amazing. Are they too good to be true? What are the cons and what are the things I should be considering? I am concerned about costs, but I want user experience to be world class. I am creating a media sharing site. Any help will be great. Thanks Fahad
4 0.84730732 1059 high scalability-2011-06-14-A TripAdvisor Short
Introduction: Sometimes I get article proposals and then there's no follow up. Though these TripAdvisor data points are from 2010, I thought them worth sharing: Our site serves in excess of 100M dynamically generated page view a day (all media and static content goes through CDN), and we do this with about 100 machines, no single point of failure, supported by distributed service architecture that that responds to over 2B requests a day, and a data warehouse of over 20TB that is used to drive email campaigns, SEM, and general reporting. We are a Linux/Java/Apache/Tomcat/Postgres/Lucene shop, and have built our own distributed computing architecture. We also maintain duplicate data centers (one active, one standby) for redundancy and maintenance purposes. Too bad, it sounds like it would have been a good article. Related Articles A new twist on "data-driven site" by Mac Slocum. How a billion points of app data shape TripAdvisor's website.
5 0.83595157 439 high scalability-2008-11-10-Scalability Perspectives #1: Nicholas Carr – The Big Switch
Introduction: Scalability Perspectives is a series of posts that highlights the ideas that will shape the next decade of IT architecture. Each post is dedicated to a thought leader of the information age and his vision of the future. Be warned though – the journey into the minds and perspectives of these people requires an open mind. Nicholas Carr A former executive editor of the Harvard Business Review, Nicholas Carr writes and speaks on technology, business, and culture. His provocative 2004 book Does IT Matter? set off a worldwide debate about the role of computers in business. The Big Switch – Rewiring the World, From Edison to Google Carr's core insight is that the development of the computer and the Internet remarkably parallels that of the last radically disruptive technology, electricity. He traces the rapid morphing of electrification from an in-house competitive advantage to a ubiquitous utility, and how the business advantage rapidly shifted from the innovators and early adop
7 0.8192935 258 high scalability-2008-02-24-Yandex Architecture
8 0.81604522 525 high scalability-2009-03-05-Product: Amazon Simple Storage Service
9 0.81347299 1377 high scalability-2012-12-26-Ask HS: What will programming and architecture look like in 2020?
10 0.80842799 1195 high scalability-2012-02-17-Stuff The Internet Says On Scalability For February 17, 2012
11 0.80745482 959 high scalability-2010-12-17-Stuff the Internet Says on Scalability For December 17th, 2010
12 0.79817432 1531 high scalability-2013-10-13-AIDA: Badoo’s journey into Continuous Integration
13 0.78809822 212 high scalability-2008-01-14-OpenSpaces.org community site launched - framework for building scale-out applications
14 0.78684431 211 high scalability-2008-01-13-Google Reveals New MapReduce Stats
15 0.77874881 1002 high scalability-2011-03-09-Productivity vs. Control tradeoffs in PaaS
16 0.77081007 474 high scalability-2008-12-21-The I.H.S.D.F. Theorem: A Proposed Theorem for the Trade-offs in Horizontally Scalable Systems
17 0.76740235 1107 high scalability-2011-08-29-The Three Ages of Google - Batch, Warehouse, Instant
18 0.76416034 766 high scalability-2010-01-26-Product: HyperGraphDB - A Graph Database
19 0.76405114 265 high scalability-2008-03-03-Two data streams for a happy website
20 0.76080048 739 high scalability-2009-11-09-10 NoSQL Systems Reviewed