high_scalability high_scalability-2012 high_scalability-2012-1180 knowledge-graph by maker-knowledge-mining
Source: html
Introduction: This is a guest post by Siddharth Anand , a senior member of LinkedIn's Distributed Data Systems team. Preamble Ramble If you’ve been working in the online (e.g. internet) space over the past 3 years, you are no stranger to terms like “the cloud” and “NoSQL”. In 2007, Amazon published a paper on Dynamo . The paper detailed how Dynamo, employing a collection of techniques to solve several problems in fault-tolerance, provided a resilient solution to the on-line shopping cart problem. A few years go by while engineers at AWS toil in relative obscurity at standing up their public cloud. It’s December 2008 and I am a member of Netflix’s Software Infrastructure team. We’ve just been told that there is something called the “CAP theorem” and because of it, we are to abandon our datacenter in hopes of leveraging Cloud Computing. Huh? A month into the investigation, we start wondering about our Oracle database. How are we are going to move it into the cloud? That’s when we are
sentIndex sentText sentNum sentScore
1 A few years go by while engineers at AWS toil in relative obscurity at standing up their public cloud. [sent-7, score-0.181]
2 We’ve just been told that there is something called the “CAP theorem” and because of it, we are to abandon our datacenter in hopes of leveraging Cloud Computing. [sent-9, score-0.329]
3 That’s when we are told that we are to abandon our RDBMS too. [sent-13, score-0.21]
4 Fast forward to the end of 2011: the past 3 years have been an amazing ride. [sent-18, score-0.174]
5 Fast forward to today: I’ve completed my first month at LinkedIn. [sent-22, score-0.157]
6 I’ve spent this month getting familiar with the various NoSQL systems that LinkedIn has built in-house. [sent-23, score-0.186]
7 These include Voldemort (another Dynamo-based system), Krati (a single-machine data store), Espresso (a new system being actively developed), etc… LinkedIn is now facing similar challenges to the Netflix of 3 years ago. [sent-24, score-0.297]
8 public cloud question for the time-being, what would I pick today regarding a NoSQL alternative to Oracle? [sent-28, score-0.148]
9 LinkedIn’s support of Voldemort, Facebook & StumbleUpon’s support of HBase), and some are supported directly by companies (e. [sent-37, score-0.264]
10 In making a decision, I’ll consider the following learnings: Any system that you pick will require 24-7 operational support. [sent-40, score-0.167]
11 Many of the NoSQL vendors view the “battle of NoSQL” to be akin to the RDBMS battle of the 80s, a winner-take-all battle. [sent-56, score-0.246]
12 A distributed system picks specific design elements in order to perform well at certain operations. [sent-59, score-0.183]
13 Systems that shard data based on a primary key will do well when routed by that key. [sent-64, score-0.18]
14 When routed by a secondary key, the system will need to “spray” a query across all shards. [sent-65, score-0.375]
15 For this reason, it would make sense to store the secondary index on an unsharded (but replicated) system. [sent-69, score-0.261]
16 The secondary index will return rowids in the primary store. [sent-74, score-0.275]
17 In reviewing Voldemort code recently, I was impressed by the clarity and quality of the code. [sent-76, score-0.239]
18 In core systems, whether distributed or not, code quality and clarity goes a long way. [sent-77, score-0.131]
19 The implementors understood the DNA of the system and did not add functionality to the system that is ill-suited to the DNA. [sent-80, score-0.27]
20 Hence, the system we end up building might be composed of several specialty component systems that can be independently tuned, in some ways similar to HBase. [sent-83, score-0.294]
wordName wordTfidf (topN-words)
[('netflix', 0.214), ('nosql', 0.213), ('dna', 0.194), ('secondary', 0.187), ('voldemort', 0.156), ('linkedin', 0.155), ('hence', 0.14), ('clarity', 0.131), ('oracle', 0.129), ('abandon', 0.124), ('ap', 0.122), ('rebalancing', 0.119), ('datacenter', 0.119), ('impressed', 0.108), ('cassandra', 0.106), ('supported', 0.106), ('dynamo', 0.105), ('years', 0.103), ('systems', 0.1), ('similar', 0.098), ('battle', 0.097), ('aws', 0.097), ('system', 0.096), ('simpledb', 0.093), ('routed', 0.092), ('delete', 0.088), ('primary', 0.088), ('member', 0.087), ('elements', 0.087), ('month', 0.086), ('told', 0.086), ('etc', 0.083), ('support', 0.079), ('vendors', 0.079), ('toil', 0.078), ('bysiddharth', 0.078), ('cite', 0.078), ('implementors', 0.078), ('rdbms', 0.078), ('cloud', 0.077), ('excuses', 0.074), ('rescue', 0.074), ('indirectly', 0.074), ('manpower', 0.074), ('unsharded', 0.074), ('tuning', 0.072), ('forward', 0.071), ('pick', 0.071), ('akin', 0.07), ('mttr', 0.07)]
simIndex simValue blogId blogTitle
same-blog 1 0.9999997 1180 high scalability-2012-01-24-The State of NoSQL in 2012
Introduction: This is a guest post by Siddharth Anand , a senior member of LinkedIn's Distributed Data Systems team. Preamble Ramble If you’ve been working in the online (e.g. internet) space over the past 3 years, you are no stranger to terms like “the cloud” and “NoSQL”. In 2007, Amazon published a paper on Dynamo . The paper detailed how Dynamo, employing a collection of techniques to solve several problems in fault-tolerance, provided a resilient solution to the on-line shopping cart problem. A few years go by while engineers at AWS toil in relative obscurity at standing up their public cloud. It’s December 2008 and I am a member of Netflix’s Software Infrastructure team. We’ve just been told that there is something called the “CAP theorem” and because of it, we are to abandon our datacenter in hopes of leveraging Cloud Computing. Huh? A month into the investigation, we start wondering about our Oracle database. How are we are going to move it into the cloud? That’s when we are
2 0.18811131 925 high scalability-2010-10-22-Paper: Netflix’s Transition to High-Availability Storage Systems
Introduction: In an audacious move for such an established property, Netflix is moving their website out of the comfort of their own datacenter and into the wilds of the Amazon cloud. This paper by Netflix's Siddharth “Sid” Anand, Netflix’s Transition to High-Availability Storage Systems , gives a detailed look at this transition and does a deep dive on SimpleDB best practices, focussing especially on techniques useful to those who are making the move from a RDBMS. Sid is going to give a talk at QCon based on this paper and he would appreciate your feedback. So if you have any comments or thoughts please comment here or email Sid at r39132@hotmail.com or Twitter at @r39132 Here's the introduction from the paper: Circa late 2008, Netflix had a single data center. This single data center raised a few concerns. As a single-point-of-failure (a.k.a. SPOF), it represented a liability – data center outages meant interruptions to service and negative customer impact. Additionally, with growth in both
3 0.18713269 954 high scalability-2010-12-06-What the heck are you actually using NoSQL for?
Introduction: It's a truism that we should choose the right tool for the job . Everyone says that. And who can disagree? The problem is this is not helpful advice without being able to answer more specific questions like: What jobs are the tools good at? Will they work on jobs like mine? Is it worth the risk to try something new when all my people know something else and we have a deadline to meet? How can I make all the tools work together? In the NoSQL space this kind of real-world data is still a bit vague. When asked, vendors tend to give very general answers like NoSQL is good for BigData or key-value access. What does that mean for for the developer in the trenches faced with the task of solving a specific problem and there are a dozen confusing choices and no obvious winner? Not a lot. It's often hard to take that next step and imagine how their specific problems could be solved in a way that's worth taking the trouble and risk. Let's change that. What problems are you using NoSQL to sol
4 0.18214911 931 high scalability-2010-10-28-Notes from A NOSQL Evening in Palo Alto
Introduction: I along with 180 other people and veritable who's who of NoSQL vendors, attended the A NoSQL Evening in Palo Alto NoSQL Meetup on Tuesday. The format was a panel of 10 vendors--10gen, Basho, CouchOne, Cloudant, Cloudera, GoGrid, InfiniteGraph, Membase, Riptano, Scality--sitting in two rows of chairs in front of what seemed like a pretty diverse audience. Tim Anglade (founder, A NOSQL Summer) moderated. Tim kept things moving by asking a few leading questions and the panel chimed in with answers. Quite a few questions came from the audience, which was refreshing. Overall a genial evening with some good discussion. I was pleased that the panel members didn't just automatically slip into marketing speak. Most of the discussions were on point rather than just another excuse to hit the talking points. There were some complaints about the talk not being technical enough, but I don't think that was really the purpose of this kind of talk. The panel format is excellent at giving a wide ra
5 0.17314625 1211 high scalability-2012-03-19-LinkedIn: Creating a Low Latency Change Data Capture System with Databus
Introduction: This is a guest post by Siddharth Anand , a senior member of LinkedIn's Distributed Data Systems team. Over the past 3 years, I've had the good fortune to work with many emerging NoSQL products in the context of supporting the needs of a high-traffic, customer facing web site. In 2010, I helped Netflix to successfully transition its web scale use-cases from Oracle to SimpleDB , AWS' hosted database service. On completion of that migration, we started a second migration, this time from SimpleDB to Cassandra. The first transition was key to our move from our own data center to AWS' cloud. The second was key to our expansion from one AWS Region to multiple geographically-distributed Regions -- today Netflix serves traffic out of two AWS Regions, one in Virginia, the other in Ireland ( F1 ). Both of these transitions have been successful, but have involved integration pain points such as the creation of database replication technology. In December 2011, I moved to LinkedIn's D
6 0.17037508 1240 high scalability-2012-05-07-Startups are Creating a New System of the World for IT
7 0.16466767 739 high scalability-2009-11-09-10 NoSQL Systems Reviewed
8 0.16137968 920 high scalability-2010-10-15-Troubles with Sharding - What can we learn from the Foursquare Incident?
9 0.16114029 1017 high scalability-2011-04-06-Netflix: Run Consistency Checkers All the time to Fixup Transactions
10 0.13516256 634 high scalability-2009-06-20-Building a data cycle at LinkedIn with Hadoop and Project Voldemort
11 0.13504916 1022 high scalability-2011-04-13-Paper: NoSQL Databases - NoSQL Introduction and Overview
12 0.13428384 853 high scalability-2010-07-08-Cloud AWS Infrastructure vs. Physical Infrastructure
13 0.13234383 1565 high scalability-2013-12-16-22 Recommendations for Building Effective High Traffic Web Software
15 0.13148856 750 high scalability-2009-12-16-Building Super Scalable Systems: Blade Runner Meets Autonomic Computing in the Ambient Cloud
16 0.12750444 961 high scalability-2010-12-21-SQL + NoSQL = Yes !
17 0.12720609 339 high scalability-2008-06-04-LinkedIn Architecture
18 0.12710156 736 high scalability-2009-11-04-Damn, Which Database do I Use Now?
19 0.12577774 1064 high scalability-2011-06-20-35+ Use Cases for Choosing Your Next NoSQL Database
20 0.12550025 148 high scalability-2007-11-11-Linkedin architecture
topicId topicWeight
[(0, 0.251), (1, 0.094), (2, 0.011), (3, 0.127), (4, 0.035), (5, 0.073), (6, -0.025), (7, -0.096), (8, 0.061), (9, -0.051), (10, -0.024), (11, 0.015), (12, -0.074), (13, 0.002), (14, -0.024), (15, 0.04), (16, 0.059), (17, -0.006), (18, -0.056), (19, -0.083), (20, 0.012), (21, 0.008), (22, 0.03), (23, -0.01), (24, -0.001), (25, -0.037), (26, -0.021), (27, -0.002), (28, -0.045), (29, -0.003), (30, -0.017), (31, 0.042), (32, -0.022), (33, -0.013), (34, 0.081), (35, 0.029), (36, -0.031), (37, -0.049), (38, -0.046), (39, -0.011), (40, 0.031), (41, 0.053), (42, 0.04), (43, 0.01), (44, -0.004), (45, -0.029), (46, -0.076), (47, -0.058), (48, 0.043), (49, 0.024)]
simIndex simValue blogId blogTitle
same-blog 1 0.96603513 1180 high scalability-2012-01-24-The State of NoSQL in 2012
Introduction: This is a guest post by Siddharth Anand , a senior member of LinkedIn's Distributed Data Systems team. Preamble Ramble If you’ve been working in the online (e.g. internet) space over the past 3 years, you are no stranger to terms like “the cloud” and “NoSQL”. In 2007, Amazon published a paper on Dynamo . The paper detailed how Dynamo, employing a collection of techniques to solve several problems in fault-tolerance, provided a resilient solution to the on-line shopping cart problem. A few years go by while engineers at AWS toil in relative obscurity at standing up their public cloud. It’s December 2008 and I am a member of Netflix’s Software Infrastructure team. We’ve just been told that there is something called the “CAP theorem” and because of it, we are to abandon our datacenter in hopes of leveraging Cloud Computing. Huh? A month into the investigation, we start wondering about our Oracle database. How are we are going to move it into the cloud? That’s when we are
2 0.7564795 1022 high scalability-2011-04-13-Paper: NoSQL Databases - NoSQL Introduction and Overview
Introduction: Christof Strauch, from Stuttgart Media University, has written an incredible 120+ page paper titled NoSQL Databases as an introduction and overview to NoSQL databases . The paper was written between 2010-06 and 2011-02, so it may be a bit out of date, but if you are looking to take in the NoSQL world in one big gulp, this is your chance. I asked Christof to give us a short taste of what he was trying to accomplish in his paper: The paper aims at giving a systematic and thorough introduction and overview of the NoSQL field by assembling information dispersed among blogs, wikis and scientific papers. It firstly discusses reasons, rationales and motives for the development and usage of nonrelational database systems. These can be summarized by the need for high scalability, the processing of large amounts of data, the ability to distribute data among many (often commodity) servers, consequently a distribution-aware design of DBMSs. The paper then introduces fundamental concepts,
3 0.74283415 954 high scalability-2010-12-06-What the heck are you actually using NoSQL for?
Introduction: It's a truism that we should choose the right tool for the job . Everyone says that. And who can disagree? The problem is this is not helpful advice without being able to answer more specific questions like: What jobs are the tools good at? Will they work on jobs like mine? Is it worth the risk to try something new when all my people know something else and we have a deadline to meet? How can I make all the tools work together? In the NoSQL space this kind of real-world data is still a bit vague. When asked, vendors tend to give very general answers like NoSQL is good for BigData or key-value access. What does that mean for for the developer in the trenches faced with the task of solving a specific problem and there are a dozen confusing choices and no obvious winner? Not a lot. It's often hard to take that next step and imagine how their specific problems could be solved in a way that's worth taking the trouble and risk. Let's change that. What problems are you using NoSQL to sol
4 0.7409898 931 high scalability-2010-10-28-Notes from A NOSQL Evening in Palo Alto
Introduction: I along with 180 other people and veritable who's who of NoSQL vendors, attended the A NoSQL Evening in Palo Alto NoSQL Meetup on Tuesday. The format was a panel of 10 vendors--10gen, Basho, CouchOne, Cloudant, Cloudera, GoGrid, InfiniteGraph, Membase, Riptano, Scality--sitting in two rows of chairs in front of what seemed like a pretty diverse audience. Tim Anglade (founder, A NOSQL Summer) moderated. Tim kept things moving by asking a few leading questions and the panel chimed in with answers. Quite a few questions came from the audience, which was refreshing. Overall a genial evening with some good discussion. I was pleased that the panel members didn't just automatically slip into marketing speak. Most of the discussions were on point rather than just another excuse to hit the talking points. There were some complaints about the talk not being technical enough, but I don't think that was really the purpose of this kind of talk. The panel format is excellent at giving a wide ra
5 0.73944622 935 high scalability-2010-11-05-Hot Scalability Links For November 5th, 2010
Introduction: So much good stuff this week... Adrian Cockcroft Compares NoSQL Availability Models . Let's risk feeding the CAP trolls, and try to get some insight into the differences between the many NoSQL contenders . Adrian asks how each NoSQL product will add a movie to its favorites list, read it back, and how this works across availability zones. Much trickier than it sounds with multiple writers. Cassandra and MongoDB answer back. Stuff the Internet Says: @jerng : Reading up on scalability. WHY THE HELL FOR? Because I want to know the future. @freerangedata : The #nosql options are the micro brews/craft beers of data stores. So many good ones, so little time to try them all. @edward_ribeiro : Soon, Darwinism will start to play its role on #NoSQL systems. You know, only the fittest will survive. @connectionreq : I'm always wowed when I hear how Facebook abuses their MySQL databases in crazy ways @louismrose : This is the kind of scalability we should be wo
6 0.73815048 1137 high scalability-2011-11-04-Stuff The Internet Says On Scalability For November 4, 2011
7 0.73587394 732 high scalability-2009-10-29-Digg - Looking to the Future with Cassandra
8 0.73521012 940 high scalability-2010-11-12-Stuff the Internet Says on Scalability For November 12th, 2010
9 0.73442966 737 high scalability-2009-11-05-A Yes for a NoSQL Taxonomy
10 0.73072296 925 high scalability-2010-10-22-Paper: Netflix’s Transition to High-Availability Storage Systems
11 0.72905755 872 high scalability-2010-08-05-Pairing NoSQL and Relational Data Storage: MySQL with MongoDB
12 0.72838461 930 high scalability-2010-10-28-NoSQL Took Away the Relational Model and Gave Nothing Back
13 0.7221086 1327 high scalability-2012-09-21-Stuff The Internet Says On Scalability For September 21, 2012
14 0.72139579 1040 high scalability-2011-05-13-Stuff The Internet Says On Scalability For May 13, 2011
15 0.71840459 1141 high scalability-2011-11-11-Stuff The Internet Says On Scalability For November 11, 2011
16 0.71413404 1036 high scalability-2011-05-06-Stuff The Internet Says On Scalability For May 6th, 2011
17 0.71307486 1513 high scalability-2013-09-06-Stuff The Internet Says On Scalability For September 6, 2013
18 0.70856524 670 high scalability-2009-08-05-Anti-RDBMS: A list of distributed key-value stores
19 0.70794028 1211 high scalability-2012-03-19-LinkedIn: Creating a Low Latency Change Data Capture System with Databus
20 0.70560098 1017 high scalability-2011-04-06-Netflix: Run Consistency Checkers All the time to Fixup Transactions
topicId topicWeight
[(1, 0.133), (2, 0.149), (10, 0.071), (30, 0.019), (40, 0.011), (47, 0.035), (56, 0.04), (61, 0.128), (77, 0.016), (79, 0.12), (85, 0.037), (92, 0.082), (94, 0.075)]
simIndex simValue blogId blogTitle
same-blog 1 0.94831938 1180 high scalability-2012-01-24-The State of NoSQL in 2012
Introduction: This is a guest post by Siddharth Anand , a senior member of LinkedIn's Distributed Data Systems team. Preamble Ramble If you’ve been working in the online (e.g. internet) space over the past 3 years, you are no stranger to terms like “the cloud” and “NoSQL”. In 2007, Amazon published a paper on Dynamo . The paper detailed how Dynamo, employing a collection of techniques to solve several problems in fault-tolerance, provided a resilient solution to the on-line shopping cart problem. A few years go by while engineers at AWS toil in relative obscurity at standing up their public cloud. It’s December 2008 and I am a member of Netflix’s Software Infrastructure team. We’ve just been told that there is something called the “CAP theorem” and because of it, we are to abandon our datacenter in hopes of leveraging Cloud Computing. Huh? A month into the investigation, we start wondering about our Oracle database. How are we are going to move it into the cloud? That’s when we are
2 0.92865962 352 high scalability-2008-07-18-Robert Scoble's Rules for Successfully Scaling Startups
Introduction: Robert Scoble in an often poignant FriendFeed thread commiserating PodTech's unfortunate end, shared what he learned about creating a successful startup. Here's a summary of a Robert's rules and why Machiavelli just may agree with them: Have a story. Have everyone on board with that story. If anyone goes off of that story, make sure they get on board immediately or fire them. Make sure people are judged by the revenues they bring in. Those that bring in revenues should get to run the place. People who don't bring in revenues should get fewer and fewer responsibilities, not more and more. Work ONLY for a leader who will make the tough decisions. Build a place where excellence is expected, allowed, and is enabled. Fire idiots quickly. If your engineering team can't give a media team good measurements, the entire company is in trouble. Only things that are measured ever get improved. When your stars aren't listened to the company is in trouble. Getting rid of t
3 0.92389548 851 high scalability-2010-07-02-Hot Scalability Links for July 2, 2010
Introduction: What says 4th of July like Nathan's ultimate scalable hot dog eating contest? This totally requires a scale-up strategy. Facebook at 60,000 servers and counting. Deepak Singh has collected some impressive massive data stats on extreme Hadoop usage: Facebook : 36 PB of uncompressed data, 2250 machines, 23,000 cores, 32 GB of RAM per machine, processing 80-90TB/day; Yahoo : 70 PB of data in HDFS, 170 PB spread across the globe, 34000 servers, Processing 3 PB per day, 120 TB flow through Hadoop every day; Twitter : 7 TB/day into HDFS; LinkedIn: 120 Billion relationships; 82 Hadoop jobs daily (IIRC); 16 TB of intermedia data. Who knew DevOps could be so funny? Adam Jacob, CTO of Opscode, gave a hilarious talk at the Velocity conference on the true nature of DevOps. Warning: your neck may get sore from nodding in agreement so much and your belly may ache from laughing so much. Pig at LinkedIn . Not your average article: For me, understanding my work over the last year b
4 0.92325264 1636 high scalability-2014-04-23-Here's a 1300 Year Old Solution to Resilience - Rebuild, Rebuild, Rebuild
Introduction: How is it possible that a wooden Shinto shrine built in the 7th century is still standing? The answer depends on how you answer this philosophical head scratcher: With nearly every cell in your body continually being replaced, are you still the same person? The Ise Grand Shrine has been in continuous existence for over 1300 years because every twenty years an exact replica has been rebuilt on an adjacent footprint. The former temple is then dismantled. Now that's resilience. If you want something to last make it a living part of a culture. It's not so much the building that is remade, what is rebuilt and passed down from generation to generation is the meme that the shrine is important and worth preserving. The rest is an unfolding of that imperative. You can see echoes of this same process in Open Source projects like Linux and the libraries and frameworks that get themselves reconstructed in each new environment. The patterns of recurrence in software are the result of Darw
Introduction: This is a repost of part 2 ( part 1 ) of an interview I did for the Boundary blog . Boundary: There’s another battle coming down the pike between Amazon (AWS) and Google (GCE). How should the CTO decide which one’s best? Hoff: Given that GCE is still closed to public access we have very little common experience on which to judge. The best way to decide is as always, by running a few experiments. Pick a few representative projects, a representative team, implement the projects on both infrastructures, crunch some numbers, figure out the bigger picture and then select the one you wanted in the first place . Sebastian Stadil, founder of Scalr, recently wrote about his experiences on both platforms and found some interesting differences: AWS has a much richer set of services; GCE is on-demand only, so AWS can be cheaper; GCE has faster disk and faster network IO, especially between datacenters; GCE has faster boot times and can mount read-only partitions across multiple
6 0.92237091 853 high scalability-2010-07-08-Cloud AWS Infrastructure vs. Physical Infrastructure
8 0.91554093 1054 high scalability-2011-06-06-NoSQL Pain? Learn How to Read-write Scale Without a Complete Re-write
10 0.9126668 1559 high scalability-2013-12-06-Stuff The Internet Says On Scalability For December 6th, 2013
11 0.91110212 1028 high scalability-2011-04-22-Stuff The Internet Says On Scalability For April 22, 2011
12 0.91109556 576 high scalability-2009-04-21-What CDN would you recommend?
13 0.90973419 1040 high scalability-2011-05-13-Stuff The Internet Says On Scalability For May 13, 2011
14 0.90959883 1289 high scalability-2012-07-23-State of the CDN: More Traffic, Stable Prices, More Products, Profits - Not So Much
15 0.9087469 383 high scalability-2008-09-10-Shard servers -- go big or small?
16 0.90776533 790 high scalability-2010-03-09-Applications as Virtual States
17 0.90764207 1002 high scalability-2011-03-09-Productivity vs. Control tradeoffs in PaaS
18 0.90731126 1302 high scalability-2012-08-10-Stuff The Internet Says On Scalability For August 10, 2012
19 0.90722668 1344 high scalability-2012-10-19-Stuff The Internet Says On Scalability For October 19, 2012
20 0.90692723 1402 high scalability-2013-02-07-Ask HighScalability: Web asset server concept - 3rd party software available?