high_scalability high_scalability-2014 high_scalability-2014-1648 knowledge-graph by maker-knowledge-mining
Source: html
Introduction: So how do you knit multiple datacenters and many thousands of phones and other clients into a single cooperating system? Usually you don't. It's too hard. We see nascent attempts in services like Firebase and Parse. SwiftCloud , as described in SwiftCloud: Fault-Tolerant Geo-Replication Integrated all the Way to the Client Machine , goes two steps further, by leveraging Conflict free Replicated Data Types (CRDTs), which means "data can be replicated at multiple sites and be updated independently with the guarantee that all replicas converge to the same value. In a cloud environment, this allows a user to access the data center closer to the user, thus optimizing the latency for all users." While we don't see these kind of systems just yet, they are a strong candidate for how things will work in the future, efficiently using resources at every level while supporting huge numbers of cooperating users. Abstract : Client-side logic and storage are increasingly used in web a
sentIndex sentText sentNum sentScore
1 So how do you knit multiple datacenters and many thousands of phones and other clients into a single cooperating system? [sent-1, score-0.431]
2 We see nascent attempts in services like Firebase and Parse. [sent-4, score-0.211]
3 In a cloud environment, this allows a user to access the data center closer to the user, thus optimizing the latency for all users. [sent-6, score-0.366]
4 " While we don't see these kind of systems just yet, they are a strong candidate for how things will work in the future, efficiently using resources at every level while supporting huge numbers of cooperating users. [sent-7, score-0.28]
5 Abstract : Client-side logic and storage are increasingly used in web and mobile applications to improve response time and availability. [sent-8, score-0.156]
6 Current approaches tend to be ad-hoc and poorly integrated with the server-side logic. [sent-9, score-0.285]
7 We present a principled approach to integrate client and server-side storage. [sent-10, score-0.48]
8 We support mergeable and strongly consistent transactions that target either client or server replicas and provide access to causally-consistent snapshots efficiently. [sent-11, score-0.847]
9 In the presence of infrastructure faults, a client-assisted failover solution allows client execution to resume immediately and seamlessly access consistent snapshots without waiting. [sent-12, score-0.914]
10 We implement this approach in SwiftCloud, the first transactional system to bring geo-replication all the way to the client machine. [sent-13, score-0.332]
11 Our experimental evaluation shows that SwiftCloud provides better fault tolerance and at the same time can improve both latency and throughput by up to an order of magnitude, compared to classical geo-replication techniques. [sent-15, score-0.473]
12 Related Articles SwiftComp is the twin project focusing on developing efficient techniques to process large ammount of data. [sent-16, score-0.211]
wordName wordTfidf (topN-words)
[('swiftcloud', 0.616), ('cooperating', 0.205), ('client', 0.204), ('snapshots', 0.159), ('replicas', 0.143), ('principled', 0.14), ('knit', 0.14), ('firebase', 0.14), ('twin', 0.125), ('nascent', 0.121), ('replicated', 0.116), ('crdts', 0.114), ('integrated', 0.111), ('converge', 0.106), ('classical', 0.106), ('poorly', 0.104), ('consistent', 0.099), ('access', 0.09), ('attempts', 0.09), ('conflict', 0.087), ('phones', 0.086), ('focusing', 0.086), ('experimental', 0.086), ('faults', 0.085), ('improve', 0.084), ('strongly', 0.082), ('seamlessly', 0.081), ('independently', 0.078), ('closer', 0.076), ('resume', 0.075), ('candidate', 0.075), ('presence', 0.074), ('increasingly', 0.072), ('guarantee', 0.071), ('leveraging', 0.071), ('allows', 0.071), ('integrate', 0.07), ('tend', 0.07), ('tolerance', 0.07), ('target', 0.07), ('steps', 0.067), ('optimizing', 0.066), ('approach', 0.066), ('described', 0.065), ('updated', 0.065), ('evaluation', 0.064), ('magnitude', 0.064), ('latency', 0.063), ('transactional', 0.062), ('failover', 0.061)]
simIndex simValue blogId blogTitle
same-blog 1 1.0 1648 high scalability-2014-05-15-Paper: SwiftCloud: Fault-Tolerant Geo-Replication Integrated all the Way to the Client Machine
Introduction: So how do you knit multiple datacenters and many thousands of phones and other clients into a single cooperating system? Usually you don't. It's too hard. We see nascent attempts in services like Firebase and Parse. SwiftCloud , as described in SwiftCloud: Fault-Tolerant Geo-Replication Integrated all the Way to the Client Machine , goes two steps further, by leveraging Conflict free Replicated Data Types (CRDTs), which means "data can be replicated at multiple sites and be updated independently with the guarantee that all replicas converge to the same value. In a cloud environment, this allows a user to access the data center closer to the user, thus optimizing the latency for all users." While we don't see these kind of systems just yet, they are a strong candidate for how things will work in the future, efficiently using resources at every level while supporting huge numbers of cooperating users. Abstract : Client-side logic and storage are increasingly used in web a
2 0.12629746 205 high scalability-2008-01-10-Letting Clients Know What's Changed: Push Me or Pull Me?
Introduction: I had a false belief I thought I came here to stay We're all just visiting All just breaking like waves The oceans made me, but who came up with me? Push me, pull me, push me, or pull me out . So true Perl Jam (Push me Pull me lyrics) , so true. I too have wondered how web clients should be notified of model changes. Should servers push events to clients or should clients pull events from servers? A topic worthy of its own song if ever there was one. To pull events the client simply starts a timer and makes a request to the server. This is polling. You can either pull a complete set of fresh data or get a list of changes. The server "knows" if anything you are interested in has changed and makes those changes available to you. Knowing what has changed can be relatively simple with a publish-subscribe type backend or you can get very complex with fine grained bit maps of attributes and keeping per client state on what I client still needs to see. Polling is heavy man.
3 0.10942273 687 high scalability-2009-08-24-How Google Serves Data from Multiple Datacenters
Introduction: Update: Streamy Explains CAP and HBase's Approach to CAP . We plan to employ inter-cluster replication, with each cluster located in a single DC. Remote replication will introduce some eventual consistency into the system, but each cluster will continue to be strongly consistent. Ryan Barrett, Google App Engine datastore lead, gave this talk Transactions Across Datacenters (and Other Weekend Projects) at the Google I/O 2009 conference. While the talk doesn't necessarily break new technical ground, Ryan does an excellent job explaining and evaluating the different options you have when architecting a system to work across multiple datacenters. This is called multihoming , operating from multiple datacenters simultaneously. As multihoming is one of the most challenging tasks in all computing, Ryan's clear and thoughtful style comfortably leads you through the various options. On the trip you learn: The different multi-homing options are: Backups, Master-Slave, Multi-M
4 0.10016266 406 high scalability-2008-10-08-Strategy: Flickr - Do the Essential Work Up-front and Queue the Rest
Introduction: This strategy is stated perfectly by Flickr's Myles Grant: The Flickr engineering team is obsessed with making pages load as quickly as possible. To that end, we’re refactoring large amounts of our code to do only the essential work up front, and rely on our queuing system to do the rest. Flickr uses a queuing system to process 11 million tasks a day. Leslie Michael Orchard also does a great job explaining the queuing meme in his excellent post Queue everything and delight everyone . Asynchronous work queues are how you scalably solve problems that are too big to handle in real-time. The process: Identify the minimum feedback the client (UI, API) needs to know an operation succeeded . It's enough, for example, to update a client's view when a posting a message to a microblogging service. The client probably isn't aware of all the other steps that happen when a message is added and doesn't really care when they happen as long as the obvious cases happen in an appropariate
5 0.094568767 963 high scalability-2010-12-23-Paper: CRDTs: Consistency without concurrency control
Introduction: For a great Christmas read forget The Night Before Christmas , a heart warming poem written by Clement Moore for his children, that created the modern idea of Santa Clause we all know and anticipate each Christmas eve. Instead, curl up with a some potent eggnog , nog being any drink made with rum, and read CRDTs: Consistency without concurrency control  by Mihai Letia, Nuno Preguiça, and Marc Shapiro, which talks about CRDTs (Commutative Replicated Data Type), a data type whose operations commute when they are concurrent . From the introduction, which also serves as a nice concise overview of distributed consistency issues: Shared read-only data is easy to scale by using well-understood replication techniques. However, sharing mutable data at a large scale is a difficult problem, because of the CAP impossibility result [5]. Two approaches dominate in practice. One ensures scalability by giving up consistency guarantees, for instance using the Last-Writer-Wins (LWW) approach [
6 0.093576647 750 high scalability-2009-12-16-Building Super Scalable Systems: Blade Runner Meets Autonomic Computing in the Ambient Cloud
8 0.09292113 318 high scalability-2008-05-14-New Facebook Chat Feature Scales to 70 Million Users Using Erlang
10 0.089410208 1487 high scalability-2013-07-05-Stuff The Internet Says On Scalability For July 5, 2013
11 0.084204026 1387 high scalability-2013-01-15-More Numbers Every Awesome Programmer Must Know
12 0.082408167 1276 high scalability-2012-07-04-Top Features of a Scalable Database
13 0.080325566 661 high scalability-2009-07-25-Latency is Everywhere and it Costs You Sales - How to Crush it
14 0.079227611 1529 high scalability-2013-10-08-F1 and Spanner Holistically Compared
15 0.079161085 881 high scalability-2010-08-16-Scaling an AWS infrastructure - Tools and Patterns
16 0.078635022 589 high scalability-2009-05-05-Drop ACID and Think About Data
17 0.0766102 507 high scalability-2009-02-03-Paper: Optimistic Replication
18 0.076516464 1637 high scalability-2014-04-25-Stuff The Internet Says On Scalability For April 25th, 2014
19 0.076360375 1565 high scalability-2013-12-16-22 Recommendations for Building Effective High Traffic Web Software
20 0.076283909 1001 high scalability-2011-03-09-Google and Netflix Strategy: Use Partial Responses to Reduce Request Sizes
topicId topicWeight
[(0, 0.144), (1, 0.046), (2, 0.009), (3, 0.023), (4, -0.025), (5, 0.036), (6, 0.056), (7, -0.003), (8, -0.073), (9, -0.006), (10, 0.016), (11, 0.07), (12, -0.042), (13, -0.034), (14, 0.036), (15, 0.027), (16, 0.01), (17, 0.016), (18, 0.044), (19, -0.044), (20, 0.018), (21, 0.057), (22, 0.013), (23, 0.012), (24, -0.014), (25, 0.002), (26, 0.009), (27, -0.019), (28, 0.017), (29, -0.076), (30, 0.009), (31, -0.006), (32, -0.021), (33, -0.025), (34, -0.017), (35, -0.016), (36, 0.034), (37, 0.012), (38, -0.031), (39, -0.01), (40, 0.004), (41, 0.011), (42, 0.024), (43, -0.002), (44, -0.033), (45, -0.057), (46, -0.037), (47, 0.004), (48, 0.033), (49, -0.026)]
simIndex simValue blogId blogTitle
same-blog 1 0.94491637 1648 high scalability-2014-05-15-Paper: SwiftCloud: Fault-Tolerant Geo-Replication Integrated all the Way to the Client Machine
Introduction: So how do you knit multiple datacenters and many thousands of phones and other clients into a single cooperating system? Usually you don't. It's too hard. We see nascent attempts in services like Firebase and Parse. SwiftCloud , as described in SwiftCloud: Fault-Tolerant Geo-Replication Integrated all the Way to the Client Machine , goes two steps further, by leveraging Conflict free Replicated Data Types (CRDTs), which means "data can be replicated at multiple sites and be updated independently with the guarantee that all replicas converge to the same value. In a cloud environment, this allows a user to access the data center closer to the user, thus optimizing the latency for all users." While we don't see these kind of systems just yet, they are a strong candidate for how things will work in the future, efficiently using resources at every level while supporting huge numbers of cooperating users. Abstract : Client-side logic and storage are increasingly used in web a
Introduction: Teams from Princeton and CMU are working together to solve one of the most difficult problems in the repertoire: scalable geo-distributed data stores. Major companies like Google and Facebook have been working on multiple datacenter database functionality for some time, but there's still a general lack of available systems that work for complex data scenarios. The ideas in this paper-- Don’t Settle for Eventual: Scalable Causal Consistency for Wide-Area Storage with COPS --are different. It's not another eventually consistent system, or a traditional transaction oriented system, or a replication based system, or a system that punts on the issue. It's something new, a causally consistent system that achieves ALPS system properties. Move over CAP, NoSQL, etc, we have another acronym: ALPS - Available (operations always complete successfully), Low-latency (operations complete quickly (single digit milliseconds)), Partition-tolerant (operates with a partition), and Scalable (just a
3 0.7522049 676 high scalability-2009-08-08-Yahoo!'s PNUTS Database: Too Hot, Too Cold or Just Right?
Introduction: So far every massively scalable database is a bundle of compromises. For some the weak guarantees of Amazon's eventual consistency model are too cold. For many the strong guarantees of standard RDBMS distributed transactions are too hot. Google App Engine tries to get it just right with entity groups . Yahoo! is also trying to get is just right by offering per-record timeline consistency, which hopes to serve up a heaping bowl of rich database functionality and low latency at massive scale : We describe PNUTS [Platform for Nimble Universal Table Storage], a massively parallel and geographically distributed database system for Yahoo!’s web applications. PNUTS provides data storage organized as hashed or ordered tables, low latency for large numbers of con-current requests including updates and queries, and novel per-record consistency guarantees. It is a hosted, centrally managed, and geographically distributed service, and utilizes automated load-balancing and failover to redu
4 0.74326408 879 high scalability-2010-08-12-Think of Latency as a Pseudo-permanent Network Partition
Introduction: The title of this post is a quote from Ilya Grigorik's post Weak Consistency and CAP Implications . Besides the article being excellent, I thought this idea had something to add to the great NoSQL versus RDBMS debate, where Mike Stonebraker makes the argument that network partitions are rare so designing eventually consistent systems for such rare occurrence is not worth losing ACID semantics over. Even if network partitions are rare, latency between datacenters is not rare, so the game is still on. The rare-partition argument seems to flow from a centralized-distributed view of systems. Such systems are scale-out in that they grow by adding distributed nodes, but the nodes generally do not cross datacenter boundaries. The assumption is the network is fast enough that distributed operations are roughly homogenous between nodes. In a fully-distributed system the nodes can be dispersed across datacenters, which gives operations a widely variable performance profile. Because everyt
5 0.74198556 1153 high scalability-2011-12-08-Update on Scalable Causal Consistency For Wide-Area Storage With COPS
Introduction: Here are a few updates on the article Paper: Don’t Settle For Eventual: Scalable Causal Consistency For Wide-Area Storage With COPS from Mike Freedman and Wyatt Lloyd. Q: How software architectures could change in response to casual+ consistency? A : I don't really think they would much. Somebody would still run a two-tier architecture in their datacenter: a front-tier of webservers running both (say) PHP and our client library, and a back tier of storage nodes running COPS. (I'm not sure if it was obvious given the discussion of our "thick" client -- you should think of the COPS client dropping in where a memcache client library does...albeit ours has per-session state.) Q: Why not just use vector clocks? A : The problem with vector clocks and scalability has always been that the size of vector clocks in O(N), where N is the number of nodes. So if we want to scale to a datacenter with 10K nodes, each piece of metadata must have size O(10K). And in fact, vector
6 0.74048662 507 high scalability-2009-02-03-Paper: Optimistic Replication
7 0.71941435 1544 high scalability-2013-11-07-Paper: Tempest: Scalable Time-Critical Web Services Platform
8 0.69730181 687 high scalability-2009-08-24-How Google Serves Data from Multiple Datacenters
9 0.68991923 205 high scalability-2008-01-10-Letting Clients Know What's Changed: Push Me or Pull Me?
10 0.68864357 890 high scalability-2010-09-01-Paper: The Case for Determinism in Database Systems
11 0.68297446 958 high scalability-2010-12-16-7 Design Patterns for Almost-infinite Scalability
12 0.68289548 1017 high scalability-2011-04-06-Netflix: Run Consistency Checkers All the time to Fixup Transactions
13 0.67877454 1142 high scalability-2011-11-14-Using Gossip Protocols for Failure Detection, Monitoring, Messaging and Other Good Things
14 0.67765689 705 high scalability-2009-09-16-Paper: A practical scalable distributed B-tree
15 0.67609608 1273 high scalability-2012-06-27-Paper: Logic and Lattices for Distributed Programming
16 0.67552316 661 high scalability-2009-07-25-Latency is Everywhere and it Costs You Sales - How to Crush it
18 0.67203259 1299 high scalability-2012-08-06-Paper: High-Performance Concurrency Control Mechanisms for Main-Memory Databases
19 0.6705206 963 high scalability-2010-12-23-Paper: CRDTs: Consistency without concurrency control
20 0.6659441 979 high scalability-2011-01-27-Comet - An Example of the New Key-Code Databases
topicId topicWeight
[(1, 0.12), (2, 0.25), (5, 0.194), (10, 0.06), (61, 0.085), (79, 0.103), (85, 0.091)]
simIndex simValue blogId blogTitle
1 0.94889295 534 high scalability-2009-03-12-Google TechTalk: Amdahl's Law in the Multicore Era
Introduction: Over the last several decades computer architects have been phenomenally successful turning the transistor bounty provided by Moore's Law into chips with ever increasing single-threaded performance. During many of these successful years, however, many researchers paid scant attention to multiprocessor work. Now as vendors turn to multicore chips, researchers are reacting with more papers on multi-threaded systems. While this is good, we are concerned that further work on single-thread performance will be squashed. To help understand future high-level trade-offs, we develop a corollary to Amdahl's Law for multicore chips [Hill & Marty, IEEE Computer 2008]. It models fixed chip resources for alternative designs that use symmetric cores, asymmetric cores, or dynamic techniques that allow cores to work together on sequential execution. Our results encourage multicore designers to view performance of the entire chip rather than focus on core efficiencies. Moreover, we observe that obtai
2 0.93310821 1113 high scalability-2011-09-09-Stuff The Internet Says On Scalability For September 9, 2011
Introduction: Scale the modern way / No brush / No lather / No rub-in / Big tube 35 cents - Drug stores / HighScalability : GAE Serves 1.5 Billion Pages a Day Potent quotables : @ kendallmiller : The code changes I'm most proud of are the ones few people will ever see - like I just tripled the scalability of our session analysis. @ Kellblog : Heard: "Cassandra is more a system on which you build a DBMS than a DBMS itself." @DDevine_au : Ah dammit . I'm thinking of using a # NoSQL database. Down the rabbit hole I go. A comprehensive guide to parallel video decoding . Emeric Grange with a sweet explanation of the decoding process. Node.js vs. Scala - "Scaling in the large" . tedsuo tldrs it: in node, there is only one concurrency model. A number of other platforms offer multiple concurrency models. If you want access to one of those other models down the line, you will have to carve off that part of your application and rewrite i
3 0.92265499 1273 high scalability-2012-06-27-Paper: Logic and Lattices for Distributed Programming
Introduction: Neil Conway from Berkeley CS is giving an advanced level talk at a meetup today in San Francisco on a new paper: Logic and Lattices for Distributed Programming - extending set logic to support CRDT-style lattices. The description of the meetup is probably the clearest introduction to the paper: Developers are increasingly choosing datastores that sacrifice strong consistency guarantees in exchange for improved performance and availability. Unfortunately, writing reliable distributed programs without the benefit of strong consistency can be very challenging. In this talk, I'll discuss work from our group at UC Berkeley that aims to make it easier to write distributed programs without relying on strong consistency. Bloom is a declarative programming language for distributed computing, while CALM is an analysis technique that identifies programs that are guaranteed to be eventually consistent. I'll then discuss our recent work on extending CALM to support a broader range of
4 0.91988939 153 high scalability-2007-11-13-Friendster Lost Lead Because of a Failure to Scale
Introduction: Hey, this scaling stuff might just be important. Jim Scheinman, former Bebo and Friendster exec, puts the blame squarely on Friendster's inability to scale as why they lost the social networking race: VB : Can you tell me a bit about what you learned in your time at Friendster? JS : For me, it basically came down to failed execution on the technology side — we had millions of Friendster members begging us to get the site working faster so they could log in and spend hours social networking with their friends. I remember coming in to the office for months reading thousands of customer service emails telling us that if we didn’t get our site working better soon, they’d be ‘forced to join’ a new social networking site that had just launched called MySpace…the rest is history. To be fair to Friendster’s technology team at the time, they were on the forefront of many new scaling and database issues that web sites simply hadn’t had to deal with prior to Friendster. As is often
5 0.91600466 899 high scalability-2010-09-09-How did Google Instant become Faster with 5-7X More Results Pages?
Introduction: We don't have a lot of details on how Google pulled off their technically very impressive Google Instant release, but in Google Instant behind the scenes , they did share some interesting facts: Google was serving more than a billion searches per day. With Google Instant they served 5-7X more results pages than previously. Typical search results were returned in less than a quarter of second. A team of 50+ worked on the project for an extended period of time. Although Google is associated with muscular data centers, they just didn't throw more server capacity at the problem, they worked smarter too. What were their general strategies? Increase backend server capacity. Add new caches to handle high request rates while keeping results fresh while the web is continuously crawled and re-indexed. Add User-state data to the back-ends to keep track of the results pages already shown to a given user, preventing the same results from being re-fetched repeatedly. Optim
6 0.91550237 1523 high scalability-2013-09-27-Stuff The Internet Says On Scalability For September 27, 2013
same-blog 7 0.91411531 1648 high scalability-2014-05-15-Paper: SwiftCloud: Fault-Tolerant Geo-Replication Integrated all the Way to the Client Machine
8 0.9114306 1487 high scalability-2013-07-05-Stuff The Internet Says On Scalability For July 5, 2013
9 0.90311831 341 high scalability-2008-06-06-GigaOm Structure 08 Conference on June 25th in San Francisco
10 0.89888984 1247 high scalability-2012-05-18-Stuff The Internet Says On Scalability For May 18, 2012
11 0.87931436 1424 high scalability-2013-03-15-Stuff The Internet Says On Scalability For March 15, 2013
12 0.872347 174 high scalability-2007-12-05-Product: Tugela Cache
13 0.86788964 485 high scalability-2009-01-05-Messaging is not just for investment banks
14 0.86675036 262 high scalability-2008-02-26-Architecture to Allow High Availability File Upload
15 0.85548711 1369 high scalability-2012-12-10-Switch your databases to Flash storage. Now. Or you're doing it wrong.
16 0.84856981 1460 high scalability-2013-05-17-Stuff The Internet Says On Scalability For May 17, 2013
17 0.84726697 1637 high scalability-2014-04-25-Stuff The Internet Says On Scalability For April 25th, 2014
18 0.84620488 595 high scalability-2009-05-08-Publish-subscribe model does not scale?
19 0.84471703 1080 high scalability-2011-07-15-Stuff The Internet Says On Scalability For July 15, 2011
20 0.84417474 1177 high scalability-2012-01-19-Is it time to get rid of the Linux OS model in the cloud?