high_scalability high_scalability-2008 high_scalability-2008-358 knowledge-graph by maker-knowledge-mining
Source: html
Introduction: Update : A very nice JavaWorld podcast interview with Google engineer Max Ross on Hibernate Shards . Max defines Hibernate Shards (horizontal partitioning), how it works (pretty well), virtual shards (don't ask), what they need to do in the future (query, replication, operational tools), and how it relates to Google AppEngine (not much). To scale you are supposed to partition your data. Sounds good, but how do you do it? When you actually sit down to work out all the details it’s not that easy. Hibernate Shards to the rescue! Hibernate shards is: an extension to the core Hibernate product that adds facilities for horizontal partitioning. If you know the core Hibernate API you know the shards API. No learning curve at all. Here is what a few members of the core group had to say about the Hibernate Shards open source project. Although there are some limitations, from the sound of it they are doing useful stuff in the right way and it’s very much worth looking at, especially if you us
sentIndex sentText sentNum sentScore
1 Max defines Hibernate Shards (horizontal partitioning), how it works (pretty well), virtual shards (don't ask), what they need to do in the future (query, replication, operational tools), and how it relates to Google AppEngine (not much). [sent-2, score-0.4]
2 Hibernate shards is: an extension to the core Hibernate product that adds facilities for horizontal partitioning. [sent-7, score-0.457]
3 If you know the core Hibernate API you know the shards API. [sent-8, score-0.413]
4 Most people don't want to shard because it makes everything complex. [sent-26, score-0.319]
5 But when you have too much data, when you fill your database up, you need another solution, which can be to shard the data across multiple relational databases. [sent-27, score-0.493]
6 Build simpler models that don't contain as many relationships because you don't have cross shard relationships. [sent-51, score-0.424]
7 Your objects graphs should be contained on one shard as much as possible. [sent-52, score-0.413]
8 Because the shards design doesn’t modify Hibernate core, you can design using shards from the start, even though you only have one database. [sent-54, score-0.816]
9 In this case the application would shard on the customer as a matter of policy, not simply scaling concerns. [sent-59, score-0.319]
10 Sharding should work across all databases Hibernate works on since shards is a layer on top of Hibernate core beneath the standard Hibernate interfaces. [sent-66, score-0.585]
11 What they are doing is figuring out how to do standard things like save objects, update, and query objects across multiple databases using standard Hibernate interfaces. [sent-68, score-0.353]
12 Can not manage cross shard foreign relationships (yet). [sent-71, score-0.436]
13 Do have runtime checks to detect when cross shard relations are used accidentally. [sent-72, score-0.361]
14 * Access Strategy – once you figure out which shard you are talking to, how do you want to access those shards (serially, 2 at a time, in parallel, etc)? [sent-80, score-0.69]
15 Configuration is set by creating a prototype configuration for all shards (remember, same schema). [sent-85, score-0.371]
16 Then you specify what's different from shard to shard like URL, user name and password, dialect (MySQL, Postgres, etc). [sent-86, score-0.638]
17 No clean way to manage read only data you want on every shard for performance and referential integrity reasons. [sent-94, score-0.375]
18 It makes sense to replicate that data on each shard so all queries using that data can stay on the shard. [sent-96, score-0.431]
19 It’s possible to shard across different databases as long as you keep the same schema in the same in each database. [sent-102, score-0.48]
20 The number of shards you can have is somewhat limited because each shard is backed by a connection pool which is a lot of databases connections. [sent-103, score-0.748]
wordName wordTfidf (topN-words)
[('hibernate', 0.766), ('shards', 0.371), ('shard', 0.319), ('ross', 0.084), ('max', 0.074), ('hql', 0.069), ('objects', 0.066), ('query', 0.062), ('across', 0.061), ('podcast', 0.061), ('sharding', 0.06), ('databases', 0.058), ('data', 0.056), ('entities', 0.056), ('standard', 0.053), ('strategies', 0.049), ('strategy', 0.048), ('factory', 0.046), ('sharded', 0.045), ('foreign', 0.044), ('horizontal', 0.044), ('cross', 0.042), ('schema', 0.042), ('core', 0.042), ('talk', 0.041), ('country', 0.04), ('orm', 0.04), ('partitioning', 0.037), ('design', 0.037), ('applying', 0.036), ('yet', 0.036), ('criteria', 0.036), ('api', 0.035), ('curve', 0.034), ('situations', 0.034), ('fit', 0.034), ('session', 0.034), ('splitting', 0.033), ('google', 0.033), ('etc', 0.033), ('employees', 0.032), ('contain', 0.032), ('persistence', 0.031), ('zip', 0.031), ('sourcesgoogle', 0.031), ('violated', 0.031), ('relationships', 0.031), ('rescue', 0.029), ('need', 0.029), ('much', 0.028)]
simIndex simValue blogId blogTitle
same-blog 1 1.0000001 358 high scalability-2008-07-26-Sharding the Hibernate Way
Introduction: Update : A very nice JavaWorld podcast interview with Google engineer Max Ross on Hibernate Shards . Max defines Hibernate Shards (horizontal partitioning), how it works (pretty well), virtual shards (don't ask), what they need to do in the future (query, replication, operational tools), and how it relates to Google AppEngine (not much). To scale you are supposed to partition your data. Sounds good, but how do you do it? When you actually sit down to work out all the details it’s not that easy. Hibernate Shards to the rescue! Hibernate shards is: an extension to the core Hibernate product that adds facilities for horizontal partitioning. If you know the core Hibernate API you know the shards API. No learning curve at all. Here is what a few members of the core group had to say about the Hibernate Shards open source project. Although there are some limitations, from the sound of it they are doing useful stuff in the right way and it’s very much worth looking at, especially if you us
2 0.78425211 24 high scalability-2007-07-24-Product: Hibernate Shards
Introduction: If you want to adopt a shard architecture, but don't want to start from scratch, you may want to consider Hibernate's sharding system. Hibernate Shards is a framework that is designed to encapsulate and minimize this complexity by adding support for horizontal partitioning to Hibernate Core. Hibernate Shards key features: Standard Hibernate programming model - Hibernate Shards allows you to continue using the Hibernate APIs you know and love: SessionFactory, Session, Criteria, Query. If you already know how to use Hibernate, you already know how to use Hibernate Shards. Flexible sharding strategies - Distribute data across your shards any way you want. Use one of the default strategies we provide or plug in your own application-specific logic. Support for virtual shards - Think your sharding strategy is never going to change? Think again. Adding new shards and redistributing your data is one of the toughest operational challenges you will face once you've deployed your
3 0.31086165 672 high scalability-2009-08-06-An Unorthodox Approach to Database Design : The Coming of the Shard
Introduction: Update 4: Why you don’t want to shard. by Morgon on the MySQL Performance Blog. Optimize everything else first, and then if performance still isn’t good enough, it’s time to take a very bitter medicine. Update 3: Building Scalable Databases: Pros and Cons of Various Database Sharding Schemes by Dare Obasanjo. Excellent discussion of why and when you would choose a sharding architecture, how to shard, and problems with sharding. Update 2: Mr. Moore gets to punt on sharding by Alan Rimm-Kaufman of 37signals. Insightful article on design tradeoffs and the evils of premature optimization. With more memory, more CPU, and new tech like SSD, problems can be avoided before more exotic architectures like sharding are needed. Add features not infrastructure. Jeremy Zawodny says he's wrong wrong wrong. we're running multi-core CPUs at slower clock speeds. Moore won't save you. Update: Dan Pritchett shares some excellent Sharding Lessons : Size Your Shards, Use Math on Shard C
4 0.27162045 920 high scalability-2010-10-15-Troubles with Sharding - What can we learn from the Foursquare Incident?
Introduction: For everything given something seems to be taken. Caching is a great scalability solution, but caching also comes with problems . Sharding is a great scalability solution, but as Foursquare recently revealed in a post-mortem about their 17 hours of downtime, sharding also has problems. MongoDB, the database Foursquare uses, also contributed their post-mortem of what went wrong too. Now that everyone has shared and resharded, what can we learn to help us skip these mistakes and quickly move on to a different set of mistakes? First, like for Facebook , huge props to Foursquare and MongoDB for being upfront and honest about their problems. This helps everyone get better and is a sign we work in a pretty cool industry. Second, overall, the fault didn't flow from evil hearts or gross negligence. As usual the cause was more mundane: a key system, that could be a little more robust, combined with a very popular application built by a small group of people, under immense pressure
5 0.21969399 152 high scalability-2007-11-13-Flickr Architecture
Introduction: Update: Flickr hits 2 Billion photos served. That's a lot of hamburgers. Flickr is both my favorite bird and the web's leading photo sharing site. Flickr has an amazing challenge, they must handle a vast sea of ever expanding new content, ever increasing legions of users, and a constant stream of new features, all while providing excellent performance. How do they do it? Site: http://www.flickr.com Information Sources Flickr and PHP (an early document) Capacity Planning for LAMP Federation at Flickr: Doing Billions of Queries a Day by Dathan Pattishall. Building Scalable Web Sites by Cal Henderson from Flickr. Database War Stories #3: Flickr by Tim O'Reilly Cal Henderson's Talks . A lot of useful PowerPoint presentations. Platform PHP MySQL Shards Memcached for a caching layer. Squid in reverse-proxy for html and images. Linux (RedHat) Smarty for templating Perl PEAR for XML and Email parsing ImageMagick, for ima
6 0.19269772 847 high scalability-2010-06-23-Product: dbShards - Share Nothing. Shard Everything.
7 0.18610901 1440 high scalability-2013-04-15-Scaling Pinterest - From 0 to 10s of Billions of Page Views a Month in Two Years
8 0.18001184 235 high scalability-2008-02-02-The case against ORM Frameworks in High Scalability Architectures
9 0.17108992 1514 high scalability-2013-09-09-Need Help with Database Scalability? Understand I-O
10 0.14944364 345 high scalability-2008-06-11-Pyshards aspires to build sharding toolkit for Python
11 0.14419644 546 high scalability-2009-03-20-Alternate strategy for database sharding
12 0.13820019 775 high scalability-2010-02-10-ElasticSearch - Open Source, Distributed, RESTful Search Engine
13 0.12636548 476 high scalability-2008-12-28-How to Organize a Database Table’s Keys for Scalability
14 0.12154493 561 high scalability-2009-04-08-N+1+caching is ok?
15 0.12118562 857 high scalability-2010-07-13-DbShards Part Deux - The Internals
16 0.11577337 367 high scalability-2008-08-17-Strategy: Drop Memcached, Add More MySQL Servers
17 0.11358235 383 high scalability-2008-09-10-Shard servers -- go big or small?
18 0.10412877 359 high scalability-2008-07-29-Ehcache - A Java Distributed Cache
19 0.10372037 1032 high scalability-2011-05-02-Stack Overflow Makes Slow Pages 100x Faster by Simple SQL Tuning
20 0.10051679 1336 high scalability-2012-10-09-Batoo JPA - The new JPA Implementation that runs over 15 times faster...
topicId topicWeight
[(0, 0.14), (1, 0.075), (2, -0.028), (3, -0.034), (4, 0.065), (5, 0.099), (6, -0.006), (7, -0.079), (8, 0.012), (9, -0.044), (10, -0.013), (11, 0.082), (12, -0.094), (13, 0.061), (14, 0.025), (15, -0.011), (16, -0.113), (17, 0.01), (18, -0.037), (19, 0.079), (20, -0.004), (21, -0.009), (22, -0.015), (23, -0.037), (24, -0.098), (25, 0.103), (26, -0.053), (27, -0.137), (28, -0.04), (29, 0.199), (30, 0.047), (31, -0.02), (32, 0.125), (33, 0.001), (34, 0.155), (35, -0.107), (36, 0.072), (37, -0.001), (38, -0.106), (39, -0.0), (40, -0.118), (41, 0.196), (42, 0.076), (43, -0.003), (44, 0.001), (45, -0.037), (46, -0.028), (47, 0.064), (48, -0.117), (49, 0.042)]
simIndex simValue blogId blogTitle
1 0.96986616 24 high scalability-2007-07-24-Product: Hibernate Shards
Introduction: If you want to adopt a shard architecture, but don't want to start from scratch, you may want to consider Hibernate's sharding system. Hibernate Shards is a framework that is designed to encapsulate and minimize this complexity by adding support for horizontal partitioning to Hibernate Core. Hibernate Shards key features: Standard Hibernate programming model - Hibernate Shards allows you to continue using the Hibernate APIs you know and love: SessionFactory, Session, Criteria, Query. If you already know how to use Hibernate, you already know how to use Hibernate Shards. Flexible sharding strategies - Distribute data across your shards any way you want. Use one of the default strategies we provide or plug in your own application-specific logic. Support for virtual shards - Think your sharding strategy is never going to change? Think again. Adding new shards and redistributing your data is one of the toughest operational challenges you will face once you've deployed your
same-blog 2 0.94011784 358 high scalability-2008-07-26-Sharding the Hibernate Way
Introduction: Update : A very nice JavaWorld podcast interview with Google engineer Max Ross on Hibernate Shards . Max defines Hibernate Shards (horizontal partitioning), how it works (pretty well), virtual shards (don't ask), what they need to do in the future (query, replication, operational tools), and how it relates to Google AppEngine (not much). To scale you are supposed to partition your data. Sounds good, but how do you do it? When you actually sit down to work out all the details it’s not that easy. Hibernate Shards to the rescue! Hibernate shards is: an extension to the core Hibernate product that adds facilities for horizontal partitioning. If you know the core Hibernate API you know the shards API. No learning curve at all. Here is what a few members of the core group had to say about the Hibernate Shards open source project. Although there are some limitations, from the sound of it they are doing useful stuff in the right way and it’s very much worth looking at, especially if you us
3 0.77310091 345 high scalability-2008-06-11-Pyshards aspires to build sharding toolkit for Python
Introduction: I've been interested in sharding concepts since first hearing the term "shard" a few years back. My interest had been piqued earlier, the first time I read about Google's original approach to distributed search. It was described as a hashtable-like system in which independent physical machines play the role of the buckets. More recently, I needed the capacity and performance of a Sharded system, but did not find helpful libraries or toolkits which would assist with the configuration for my language of preference these days, which is Python. And, since I had a few weeks on my hands, I decided I would begin the work of creating these tools. The result of my initial work the Pyshards project, a still-incomplete python and MySQL based horizontal partitioning and sharding toolkit. HighScalability.com readers will already know that horizontal partitioning is a data segmenting pattern in which distinct groups of physical row-based datasets are distributed across multiple partitions. Whe
4 0.75151211 476 high scalability-2008-12-28-How to Organize a Database Table’s Keys for Scalability
Introduction: The key (no pun intended) to understanding how to organize your dataset’s data is to think of each shard not as an individual database, but as one large singular database. Just as in a normal single server database setup where you have a unique key for each row within a table, each row key within each individual shard must be unique to the whole dataset partitioned across all shards. There are a few different ways we can accomplish uniqueness of row keys across a shard cluster. Each has its pro’s and con’s and the one chosen should be specific to the problems you’re trying to solve.
5 0.69618315 207 high scalability-2008-01-10-Sharding with Cookie-Based Session Storage
Introduction: In a recent project, I utilized RoR's cookie-based session storage to shard geographically distinct user groups. My technique for doing so was unique and, although it was a premature optimization, it is none-the-less an idea worth exploring.
6 0.67858624 857 high scalability-2010-07-13-DbShards Part Deux - The Internals
7 0.66135782 847 high scalability-2010-06-23-Product: dbShards - Share Nothing. Shard Everything.
8 0.65510201 546 high scalability-2009-03-20-Alternate strategy for database sharding
9 0.64672577 672 high scalability-2009-08-06-An Unorthodox Approach to Database Design : The Coming of the Shard
10 0.59206015 1514 high scalability-2013-09-09-Need Help with Database Scalability? Understand I-O
11 0.58868575 775 high scalability-2010-02-10-ElasticSearch - Open Source, Distributed, RESTful Search Engine
12 0.58377999 152 high scalability-2007-11-13-Flickr Architecture
13 0.57248425 561 high scalability-2009-04-08-N+1+caching is ok?
14 0.56377751 920 high scalability-2010-10-15-Troubles with Sharding - What can we learn from the Foursquare Incident?
15 0.50304145 1440 high scalability-2013-04-15-Scaling Pinterest - From 0 to 10s of Billions of Page Views a Month in Two Years
16 0.49519321 549 high scalability-2009-03-26-Performance - When do I start worrying?
17 0.47507185 89 high scalability-2007-09-10-Is there a difference between partitioning and federation and sharding?
18 0.46403128 383 high scalability-2008-09-10-Shard servers -- go big or small?
19 0.46309292 1606 high scalability-2014-03-05-10 Things You Should Know About Running MongoDB at Scale
20 0.45838371 933 high scalability-2010-11-01-Hot Trend: Move Behavior to Data for a New Interactive Application Architecture
topicId topicWeight
[(1, 0.124), (2, 0.28), (10, 0.024), (14, 0.034), (15, 0.023), (47, 0.084), (61, 0.11), (73, 0.011), (76, 0.013), (79, 0.094), (85, 0.017), (94, 0.055)]
simIndex simValue blogId blogTitle
same-blog 1 0.98331118 358 high scalability-2008-07-26-Sharding the Hibernate Way
Introduction: Update : A very nice JavaWorld podcast interview with Google engineer Max Ross on Hibernate Shards . Max defines Hibernate Shards (horizontal partitioning), how it works (pretty well), virtual shards (don't ask), what they need to do in the future (query, replication, operational tools), and how it relates to Google AppEngine (not much). To scale you are supposed to partition your data. Sounds good, but how do you do it? When you actually sit down to work out all the details it’s not that easy. Hibernate Shards to the rescue! Hibernate shards is: an extension to the core Hibernate product that adds facilities for horizontal partitioning. If you know the core Hibernate API you know the shards API. No learning curve at all. Here is what a few members of the core group had to say about the Hibernate Shards open source project. Although there are some limitations, from the sound of it they are doing useful stuff in the right way and it’s very much worth looking at, especially if you us
2 0.97598737 1440 high scalability-2013-04-15-Scaling Pinterest - From 0 to 10s of Billions of Page Views a Month in Two Years
Introduction: Pinterest has been riding an exponential growth curve, doubling every month and half. They've gone from 0 to 10s of billions of page views a month in two years, from 2 founders and one engineer to over 40 engineers, from one little MySQL server to 180 Web Engines, 240 API Engines, 88 MySQL DBs (cc2.8xlarge) + 1 slave each, 110 Redis Instances, and 200 Memcache Instances.Stunning growth. So what's Pinterest's story? To tell their story we have our bards, Pinterest'sYashwanth NelapatiandMarty Weiner, who tell the dramatic story of Pinterest's architecture evolution in a talk titledScaling Pinterest. This is the talk they would have liked to hear a year and half ago when they were scaling fast and there were a lot of options to choose from. And they made a lot of incorrect choices.This is a great talk. It's full of amazing details. It's also very practical, down to earth, and it contains strategies adoptable by nearly anyone. Highly recommended.Two of my favorite lessons from the talk:Arc
3 0.97287738 550 high scalability-2009-03-30-Ebay history and architecture
Introduction: Ebay [1] Starts in 1995, initial name AuctionWeb (V1) : - very simple architecture - based on perl - no database, for data persistence they used plain files Because of rapid growth they needed to improve their architecture and so V2 (clever name) was born: - replaced perl with C/C++ - started using a database in a master-slave configuration - C++ back-end - XSLT front-end Any request will lead to an XML file being created in C++ and the XLST processor will transform that into html. *pretty sophisticated architecture for the 90s, XLST was cutting-edge back then* That hold out pretty well for a while but in the late 90s ebay experienced an exponential growth. They started having some trouble with outages and needed improvements, so V3 was developed: - based on java - search engine still used C++ - proof that relational databases can scale (aggressive caching) - developed a messaging layer for making a lot of asyncronious calls, they a
4 0.97220981 1530 high scalability-2013-10-11-Stuff The Internet Says On Scalability For October 11th, 2013
Introduction: Hey, it's HighScalability time: In honor of Twitter's Cha-Ching moment, here's Twitter By the Numbers Quotable Quotes: @BrandonBloom : Mutable data structures are only faster b/c they have fewer features: ie no persistence. You must manually recover that feature w/ copying. @hayesdrumwright : Scale breaks hardware Speed breaks software Scale and speed breaks everything Adrian Cockcroft - Netflix #TechSummit Gladwell, Malcolm : Saul thinks of power in terms of physical might. He doesn’t appreciate that power can come in other forms as well— in breaking rules, in substituting speed and surprise for strength. Now here's an irony. The East India Company established a major trading post at Bantam in Java. They called their trading posts "factories." Get it? Java. Factories. That's stranger than fiction. Cost of Healthcare.gov: $634 Million — So Far . In a modern methodology you hardly ever just open a big site in
5 0.97116029 1166 high scalability-2011-12-30-Stuff The Internet Says On Scalability For December 30, 2011
Introduction: Pork. The Other HighScalability: PlentyOfFish: 6 Billion Page Views ; World: info doubling every 2 years ; 2015: 7,910 exabytes of global digital data ; Khan Academy: 4 million uniques ; G+: 62 million users ; Zynga: leased 9 megawatts of capacity ; Heroku: billions of page views a month Quoteable quotes: Udi Dhan : Scalability is not boolean. John Boyd: Look at the mission, not the technology. And if you do look at the mission, don't look at the most fashionable mission of the day. @BigDataClouds : I think the fear of change is the biggest challenge that companies are facing @cjzero : If you were wondering, the #Mythbusters scalability test of a Newton's Cradle using wrecking balls? Busted. @Xorlev : Scalability is really really hard. That's why it's fun. It pushes the limits of engineering talent. 100 Best Cloud & Data Stats of 2011 by Zenoss. Lots of fun facts about how mind bendingly huge the world of information is exponentially be
6 0.96697885 1065 high scalability-2011-06-21-Running TPC-C on MySQL-RDS
7 0.96557266 714 high scalability-2009-10-02-HighScalability has Moved to Squarespace.com!
8 0.9636721 144 high scalability-2007-11-07-What CDN would you recommend?
9 0.96227622 825 high scalability-2010-05-10-Sify.com Architecture - A Portal at 3900 Requests Per Second
10 0.96216661 240 high scalability-2008-02-05-Handling of Session for a site running from more than 1 data center
11 0.96205336 1154 high scalability-2011-12-09-Stuff The Internet Says On Scalability For December 9, 2011
12 0.96201509 672 high scalability-2009-08-06-An Unorthodox Approach to Database Design : The Coming of the Shard
13 0.96180266 628 high scalability-2009-06-13-Neo4j - a Graph Database that Kicks Buttox
14 0.9616636 661 high scalability-2009-07-25-Latency is Everywhere and it Costs You Sales - How to Crush it
15 0.96118146 1278 high scalability-2012-07-06-Stuff The Internet Says On Scalability For July 6, 2012
16 0.96105444 351 high scalability-2008-07-16-The Mother of All Database Normalization Debates on Coding Horror
17 0.9609735 248 high scalability-2008-02-13-What's your scalability plan?
18 0.9604224 1151 high scalability-2011-12-05-Stuff The Internet Says On Scalability For December 5, 2011
19 0.96019304 1135 high scalability-2011-10-31-15 Ways to Make Your Application Feel More Responsive under Google App Engine
20 0.9598093 1401 high scalability-2013-02-06-Super Bowl Advertisers Ready for the Traffic? Nope..It's Lights Out.