high_scalability high_scalability-2009 high_scalability-2009-529 knowledge-graph by maker-knowledge-mining
Source: html
Introduction: Update: Barbara Liskov’s Turing Award, and Byzantine Fault Tolerance . Henry Robinson has created an excellent series of articles on consensus protocols. We already covered his 2 Phase Commit article and he also has a 3 Phase Commit article showing how to handle 2PC under single node failures. But that is not enough! 3PC works well under node failures, but fails for network failures. So another consensus mechanism is needed that handles both network and node failures. And that's Paxos . Paxos correctly handles both types of failures, but it does this by becoming inaccessible if too many components fail. This is the "liveness" property of protocols. Paxos waits until the faults are fixed. Read queries can be handled, but updates will be blocked until the protocol thinks it can make forward progress. The liveness of Paxos is primarily dependent on network stability. In a distributed heterogeneous environment you are at risk of losing the ability to make updates. Users hate t
sentIndex sentText sentNum sentScore
1 Henry Robinson has created an excellent series of articles on consensus protocols. [sent-2, score-0.367]
2 We already covered his 2 Phase Commit article and he also has a 3 Phase Commit article showing how to handle 2PC under single node failures. [sent-3, score-0.296]
3 3PC works well under node failures, but fails for network failures. [sent-5, score-0.276]
4 So another consensus mechanism is needed that handles both network and node failures. [sent-6, score-0.745]
5 Paxos correctly handles both types of failures, but it does this by becoming inaccessible if too many components fail. [sent-8, score-0.193]
6 Read queries can be handled, but updates will be blocked until the protocol thinks it can make forward progress. [sent-11, score-0.239]
7 The liveness of Paxos is primarily dependent on network stability. [sent-12, score-0.481]
8 In a distributed heterogeneous environment you are at risk of losing the ability to make updates. [sent-13, score-0.309]
9 So when companies like Amazon do the seemingly insane thing of creating eventually consistent databases , it should be a little easier to understand now. [sent-15, score-0.201]
10 Not being able to write under partition failures is unacceptable. [sent-18, score-0.17]
11 Therefor create a system that can always write and work on consistency when all the downed partitions/networks are repaired. [sent-19, score-0.126]
wordName wordTfidf (topN-words)
[('consensus', 0.367), ('commitarticle', 0.322), ('paxos', 0.27), ('liveness', 0.261), ('failures', 0.17), ('liskov', 0.146), ('paxosby', 0.146), ('phase', 0.14), ('therefor', 0.137), ('lynch', 0.137), ('amir', 0.137), ('byzantine', 0.137), ('impossibility', 0.137), ('node', 0.133), ('robinson', 0.131), ('downed', 0.126), ('articlesgoogle', 0.119), ('turing', 0.119), ('barbara', 0.119), ('partitioning', 0.116), ('ken', 0.116), ('faulty', 0.116), ('waits', 0.111), ('handles', 0.11), ('nasty', 0.109), ('et', 0.102), ('seemingly', 0.102), ('blocked', 0.101), ('award', 0.099), ('insane', 0.099), ('hate', 0.096), ('jonathan', 0.096), ('heterogeneous', 0.093), ('covered', 0.092), ('faults', 0.089), ('correctly', 0.083), ('losing', 0.082), ('coordination', 0.082), ('dependent', 0.082), ('property', 0.075), ('fails', 0.073), ('thinks', 0.072), ('showing', 0.071), ('network', 0.07), ('brings', 0.068), ('primarily', 0.068), ('risk', 0.067), ('distributed', 0.067), ('forward', 0.066), ('mechanism', 0.065)]
simIndex simValue blogId blogTitle
same-blog 1 1.0000001 529 high scalability-2009-03-10-Paper: Consensus Protocols: Paxos
Introduction: Update: Barbara Liskov’s Turing Award, and Byzantine Fault Tolerance . Henry Robinson has created an excellent series of articles on consensus protocols. We already covered his 2 Phase Commit article and he also has a 3 Phase Commit article showing how to handle 2PC under single node failures. But that is not enough! 3PC works well under node failures, but fails for network failures. So another consensus mechanism is needed that handles both network and node failures. And that's Paxos . Paxos correctly handles both types of failures, but it does this by becoming inaccessible if too many components fail. This is the "liveness" property of protocols. Paxos waits until the faults are fixed. Read queries can be handled, but updates will be blocked until the protocol thinks it can make forward progress. The liveness of Paxos is primarily dependent on network stability. In a distributed heterogeneous environment you are at risk of losing the ability to make updates. Users hate t
2 0.28863993 510 high scalability-2009-02-09-Paper: Consensus Protocols: Two-Phase Commit
Introduction: Henry Robinson has created an excellent series of articles on consensus protocols. Henry starts with a very useful discussion of what all this talk about consensus really means: The consensus problem is the problem of getting a set of nodes in a distributed system to agree on something - it might be a value, a course of action or a decision. Achieving consensus allows a distributed system to act as a single entity, with every individual node aware of and in agreement with the actions of the whole of the network. In this article Henry tackles Two-Phase Commit, the protocol most databases use to arrive at a consensus for database writes. The article is very well written with lots of pretty and informative pictures. He did a really good job. In conclusion we learn 2PC is very efficient, a minimal number of messages are exchanged and latency is low. The problem is when a co-ordinator fails availability is dramatically reduced. This is why 2PC isn't generally used on highly distributed
3 0.21139561 357 high scalability-2008-07-26-Google's Paxos Made Live – An Engineering Perspective
Introduction: This is an unusually well written and useful paper . It talks in detail about experiences implementing a complex project, something we don't see very often. They shockingly even admit that creating a working implementation of Paxos was more difficult than just translating the pseudo code. Imagine that, programmers aren't merely typists! I particularly like the explanation of the Paxos algorithm and why anyone would care about it, working with disk corruption, using leases to support simultaneous reads, using epoch numbers to indicate a new master election, using snapshots to prevent unbounded logs, using MultiOp to implement database transactions, how they tested the system, and their openness with the various problems they had. A lot to learn here. From the paper: We describe our experience building a fault-tolerant data-base using the Paxos consensus algorithm. Despite the existing literature in the field, building such a database proved to be non-trivial. We describe selected alg
4 0.19635284 1243 high scalability-2012-05-10-Paper: Paxos Made Moderately Complex
Introduction: If you are a normal human being and find the Paxos protocol confusing, then this paper, Paxos Made Moderately Complex , is a great find. Robbert van Renesse from Cornell University has written a clear and well written paper with excellent explanations. The Abstract: For anybody who has ever tried to implement it, Paxos is by no means a simple protocol, even though it is based on relatively simple invariants. This paper provides imperative pseudo-code for the full Paxos (or Multi-Paxos) protocol without shying away from discussing various implementation details. The initial description avoids optimizations that complicate comprehension. Next we discuss liveness, and list various optimizations that make the protocol practical. Related Articles Paxos on HighScalability.com
5 0.14283493 1374 high scalability-2012-12-18-Georeplication: When Bad Things Happen to Good Systems
Introduction: Georeplication is one of the standard techniques for dealing when bad things--failure and latency--happen to good systems. The problem is always: how do you do that? Murat Demirbas , Associate Professor at SUNY Buffalo, has a couple of really good posts that can help: MDCC: Multi-Data Center Consistency and Making Geo-Replicated Systems Fast as Possible, Consistent when Necessary . In MDCC: Multi-Data Center Consistency Murat discusses a paper that says synchronous wide-area replication can be feasible. There's a quick and clear explanation of Paxos and various optimizations that is worth the price of admission. We find that strong consistency doesn't have to be lost across a WAN: The good thing about using Paxos over the WAN is you /almost/ get the full CAP (all three properties: consistency, availability, and partition-freedom). As we discussed earlier (Paxos taught), Paxos is CP, that is, in the presence of a partition, Paxos keeps consistency over availability. But, P
6 0.13794927 687 high scalability-2009-08-24-How Google Serves Data from Multiple Datacenters
8 0.10873055 1345 high scalability-2012-10-22-Spanner - It's About Programmers Building Apps Using SQL Semantics at NoSQL Scale
10 0.094537005 1451 high scalability-2013-05-03-Stuff The Internet Says On Scalability For May 3, 2013
11 0.088677421 1420 high scalability-2013-03-08-Stuff The Internet Says On Scalability For March 8, 2013
12 0.086119659 710 high scalability-2009-09-20-PaxosLease: Diskless Paxos for Leases
13 0.077046864 589 high scalability-2009-05-05-Drop ACID and Think About Data
14 0.072703473 1529 high scalability-2013-10-08-F1 and Spanner Holistically Compared
15 0.071013644 1527 high scalability-2013-10-04-Stuff The Internet Says On Scalability For October 4th, 2013
16 0.070672087 1459 high scalability-2013-05-16-Paper: Warp: Multi-Key Transactions for Key-Value Stores
17 0.069570892 920 high scalability-2010-10-15-Troubles with Sharding - What can we learn from the Foursquare Incident?
18 0.067503311 1498 high scalability-2013-08-07-RAFT - In Search of an Understandable Consensus Algorithm
19 0.062498122 1048 high scalability-2011-05-27-Stuff The Internet Says On Scalability For May 27, 2011
20 0.061393488 1166 high scalability-2011-12-30-Stuff The Internet Says On Scalability For December 30, 2011
topicId topicWeight
[(0, 0.092), (1, 0.061), (2, -0.004), (3, 0.038), (4, 0.011), (5, 0.055), (6, 0.01), (7, -0.02), (8, -0.052), (9, -0.032), (10, 0.009), (11, 0.031), (12, -0.061), (13, -0.038), (14, 0.053), (15, 0.036), (16, 0.036), (17, -0.022), (18, -0.0), (19, -0.023), (20, 0.071), (21, 0.053), (22, -0.029), (23, 0.024), (24, -0.092), (25, -0.007), (26, 0.069), (27, 0.028), (28, 0.003), (29, -0.007), (30, 0.003), (31, -0.024), (32, -0.065), (33, -0.01), (34, 0.013), (35, -0.061), (36, -0.019), (37, -0.005), (38, 0.01), (39, -0.008), (40, -0.035), (41, -0.028), (42, 0.008), (43, -0.012), (44, -0.034), (45, 0.019), (46, 0.039), (47, 0.048), (48, -0.045), (49, -0.039)]
simIndex simValue blogId blogTitle
same-blog 1 0.95958668 529 high scalability-2009-03-10-Paper: Consensus Protocols: Paxos
Introduction: Update: Barbara Liskov’s Turing Award, and Byzantine Fault Tolerance . Henry Robinson has created an excellent series of articles on consensus protocols. We already covered his 2 Phase Commit article and he also has a 3 Phase Commit article showing how to handle 2PC under single node failures. But that is not enough! 3PC works well under node failures, but fails for network failures. So another consensus mechanism is needed that handles both network and node failures. And that's Paxos . Paxos correctly handles both types of failures, but it does this by becoming inaccessible if too many components fail. This is the "liveness" property of protocols. Paxos waits until the faults are fixed. Read queries can be handled, but updates will be blocked until the protocol thinks it can make forward progress. The liveness of Paxos is primarily dependent on network stability. In a distributed heterogeneous environment you are at risk of losing the ability to make updates. Users hate t
2 0.82736075 510 high scalability-2009-02-09-Paper: Consensus Protocols: Two-Phase Commit
Introduction: Henry Robinson has created an excellent series of articles on consensus protocols. Henry starts with a very useful discussion of what all this talk about consensus really means: The consensus problem is the problem of getting a set of nodes in a distributed system to agree on something - it might be a value, a course of action or a decision. Achieving consensus allows a distributed system to act as a single entity, with every individual node aware of and in agreement with the actions of the whole of the network. In this article Henry tackles Two-Phase Commit, the protocol most databases use to arrive at a consensus for database writes. The article is very well written with lots of pretty and informative pictures. He did a really good job. In conclusion we learn 2PC is very efficient, a minimal number of messages are exchanged and latency is low. The problem is when a co-ordinator fails availability is dramatically reduced. This is why 2PC isn't generally used on highly distributed
3 0.78059775 890 high scalability-2010-09-01-Paper: The Case for Determinism in Database Systems
Introduction: Can you have your ACID cake and eat your distributed database too? Yes explains Daniel Abadi, Assistant Professor of Computer Science at Yale University, in an epic post, The problems with ACID, and how to fix them without going NoSQL , coauthored with Alexander Thomson , on their paper The Case for Determinism in Database Systems . We've already seen VoltDB offer the best of both worlds, this sounds like a completely different approach. The solution, they propose, is: ...an architecture and execution model that avoids deadlock, copes with failures without aborting transactions, and achieves high concurrency. The paper contains full details, but the basic idea is to use ordered locking coupled with optimistic lock location prediction, while exploiting deterministic systems' nice replication properties in the case of failures. The problem they are trying to solve is: In our opinion, the NoSQL decision to give up on ACID is the lazy solution to these scala
4 0.77364677 1374 high scalability-2012-12-18-Georeplication: When Bad Things Happen to Good Systems
Introduction: Georeplication is one of the standard techniques for dealing when bad things--failure and latency--happen to good systems. The problem is always: how do you do that? Murat Demirbas , Associate Professor at SUNY Buffalo, has a couple of really good posts that can help: MDCC: Multi-Data Center Consistency and Making Geo-Replicated Systems Fast as Possible, Consistent when Necessary . In MDCC: Multi-Data Center Consistency Murat discusses a paper that says synchronous wide-area replication can be feasible. There's a quick and clear explanation of Paxos and various optimizations that is worth the price of admission. We find that strong consistency doesn't have to be lost across a WAN: The good thing about using Paxos over the WAN is you /almost/ get the full CAP (all three properties: consistency, availability, and partition-freedom). As we discussed earlier (Paxos taught), Paxos is CP, that is, in the presence of a partition, Paxos keeps consistency over availability. But, P
Introduction: Teams from Princeton and CMU are working together to solve one of the most difficult problems in the repertoire: scalable geo-distributed data stores. Major companies like Google and Facebook have been working on multiple datacenter database functionality for some time, but there's still a general lack of available systems that work for complex data scenarios. The ideas in this paper-- Don’t Settle for Eventual: Scalable Causal Consistency for Wide-Area Storage with COPS --are different. It's not another eventually consistent system, or a traditional transaction oriented system, or a replication based system, or a system that punts on the issue. It's something new, a causally consistent system that achieves ALPS system properties. Move over CAP, NoSQL, etc, we have another acronym: ALPS - Available (operations always complete successfully), Low-latency (operations complete quickly (single digit milliseconds)), Partition-tolerant (operates with a partition), and Scalable (just a
6 0.74131292 1463 high scalability-2013-05-23-Paper: Calvin: Fast Distributed Transactions for Partitioned Database Systems
7 0.73994458 625 high scalability-2009-06-10-Managing cross partition transactions in a distributed KV system
8 0.73673189 1450 high scalability-2013-05-01-Myth: Eric Brewer on Why Banks are BASE Not ACID - Availability Is Revenue
9 0.73072743 705 high scalability-2009-09-16-Paper: A practical scalable distributed B-tree
10 0.72245979 1459 high scalability-2013-05-16-Paper: Warp: Multi-Key Transactions for Key-Value Stores
11 0.71465015 1142 high scalability-2011-11-14-Using Gossip Protocols for Failure Detection, Monitoring, Messaging and Other Good Things
12 0.71076691 676 high scalability-2009-08-08-Yahoo!'s PNUTS Database: Too Hot, Too Cold or Just Right?
13 0.69418585 1153 high scalability-2011-12-08-Update on Scalable Causal Consistency For Wide-Area Storage With COPS
14 0.68781883 1273 high scalability-2012-06-27-Paper: Logic and Lattices for Distributed Programming
15 0.68643302 1243 high scalability-2012-05-10-Paper: Paxos Made Moderately Complex
16 0.68439311 1629 high scalability-2014-04-10-Paper: Scalable Atomic Visibility with RAMP Transactions - Scale Linearly to 100 Servers
17 0.67017555 507 high scalability-2009-02-03-Paper: Optimistic Replication
18 0.66258413 963 high scalability-2010-12-23-Paper: CRDTs: Consistency without concurrency control
19 0.6600371 357 high scalability-2008-07-26-Google's Paxos Made Live – An Engineering Perspective
topicId topicWeight
[(1, 0.084), (2, 0.182), (6, 0.32), (10, 0.023), (47, 0.038), (51, 0.072), (61, 0.072), (79, 0.108)]
simIndex simValue blogId blogTitle
1 0.84672034 104 high scalability-2007-10-01-SmugMug Found their Perfect Storage Array
Introduction: SmugMug's CEO & Chief Geek Don MacAskill smugly (hard to resist) gushes over finally finding, after a long and arduous quest, their "best bang-for-the-buck storage array." It's the Dell MD300 . His in-depth explanation of why he prefers the MD3000 should help anyone with their own painful storage deliberations. His key points are: The price is right; DAS via SAS, 15 spindles at 15K rpm each, 512MB of mirrored battery-backed write cache; You can disable read caching; You can disable read-ahead prefetching; The stripe sizes are configurable up to 512KB; The controller ignores host-based flush commands by default; They support an ‘Enhanced JBOD’ mode. His reasoning for the desirability each option is astute and he even gives you the configuration options for carrying out the configuration. This is not your average CEO. Don also speculates that a three tier system using flash (system RAM + flash storage + RAID disks) is a possible future direction. Unfortunately, flash
same-blog 2 0.8397454 529 high scalability-2009-03-10-Paper: Consensus Protocols: Paxos
Introduction: Update: Barbara Liskov’s Turing Award, and Byzantine Fault Tolerance . Henry Robinson has created an excellent series of articles on consensus protocols. We already covered his 2 Phase Commit article and he also has a 3 Phase Commit article showing how to handle 2PC under single node failures. But that is not enough! 3PC works well under node failures, but fails for network failures. So another consensus mechanism is needed that handles both network and node failures. And that's Paxos . Paxos correctly handles both types of failures, but it does this by becoming inaccessible if too many components fail. This is the "liveness" property of protocols. Paxos waits until the faults are fixed. Read queries can be handled, but updates will be blocked until the protocol thinks it can make forward progress. The liveness of Paxos is primarily dependent on network stability. In a distributed heterogeneous environment you are at risk of losing the ability to make updates. Users hate t
3 0.82075554 832 high scalability-2010-05-31-Scalable federated security with Kerberos
Introduction: In my last post , I outlined considerations that need to be taken into account when choosing between a centralized and federated security model. So, how do we implement the chosen model? Based on a real-world case study, I will outline a Kerberos architecture that enables cutting-edge collaborative research through federated sharing of resources. Read more on BigDataMatters.com
4 0.79870445 710 high scalability-2009-09-20-PaxosLease: Diskless Paxos for Leases
Introduction: PaxosLease is a distributed algorithm for lease negotiation. It is based on Paxos, but does not require disk writes or clock synchrony. PaxosLease is used for master lease negotation in the open-source Keyspace replicated key-value store.
5 0.78434068 93 high scalability-2007-09-16-What software runs on this site?
Introduction: It's pretty slick! olla
6 0.76003146 794 high scalability-2010-03-11-What would you like to ask Justin.tv?
7 0.73024601 213 high scalability-2008-01-15-Does Sun Buying MySQL Change Your Scaling Strategy?
8 0.66277564 1423 high scalability-2013-03-13-Iron.io Moved From Ruby to Go: 28 Servers Cut and Colossal Clusterf**ks Prevented
9 0.65218747 1036 high scalability-2011-05-06-Stuff The Internet Says On Scalability For May 6th, 2011
10 0.63649344 243 high scalability-2008-02-07-clusteradmin.blogspot.com - blog about building and administering clusters
11 0.63575894 1389 high scalability-2013-01-18-Stuff The Internet Says On Scalability For January 18, 2013
12 0.6116398 1553 high scalability-2013-11-25-How To Make an Infinitely Scalable Relational Database Management System (RDBMS)
13 0.60278761 741 high scalability-2009-11-16-Building Scalable Systems Using Data as a Composite Material
14 0.59071332 687 high scalability-2009-08-24-How Google Serves Data from Multiple Datacenters
15 0.5900296 1134 high scalability-2011-10-28-Stuff The Internet Says On Scalability For October 28, 2011
16 0.58787447 138 high scalability-2007-10-30-Feedblendr Architecture - Using EC2 to Scale
17 0.58769548 1271 high scalability-2012-06-25-StubHub Architecture: The Surprising Complexity Behind the World’s Largest Ticket Marketplace
18 0.58472365 358 high scalability-2008-07-26-Sharding the Hibernate Way
19 0.58461696 736 high scalability-2009-11-04-Damn, Which Database do I Use Now?
20 0.58458364 672 high scalability-2009-08-06-An Unorthodox Approach to Database Design : The Coming of the Shard