high_scalability high_scalability-2010 high_scalability-2010-809 knowledge-graph by maker-knowledge-mining
Source: html
Introduction: Deferred Deletes is a technique where deleted items are marked as deleted but not garbage collected until some days or preferably weeks later . James Hamilton talks describes this strategy in his classic On Designing and Deploying Internet-Scale Services: Never delete anything. Just mark it deleted. When new data comes in, record the requests on the way. Keep a rolling two week (or more) history of all changes to help recover from software or administrative errors. If someone makes a mistake and forgets the where clause on a delete statement (it has happened before and it will again), all logical copies of the data are deleted. Neither RAID nor mirroring can protect against this form of error. The ability to recover the data can make the difference between a highly embarrassing issue or a minor, barely noticeable glitch. For those systems already doing off-line backups, this additional record of data coming into the service only needs to be since the last backup. But, bein
sentIndex sentText sentNum sentScore
1 Deferred Deletes is a technique where deleted items are marked as deleted but not garbage collected until some days or preferably weeks later . [sent-1, score-0.963]
2 James Hamilton talks describes this strategy in his classic On Designing and Deploying Internet-Scale Services: Never delete anything. [sent-2, score-0.354]
3 When new data comes in, record the requests on the way. [sent-4, score-0.139]
4 Keep a rolling two week (or more) history of all changes to help recover from software or administrative errors. [sent-5, score-0.55]
5 If someone makes a mistake and forgets the where clause on a delete statement (it has happened before and it will again), all logical copies of the data are deleted. [sent-6, score-0.998]
6 Neither RAID nor mirroring can protect against this form of error. [sent-7, score-0.212]
7 The ability to recover the data can make the difference between a highly embarrassing issue or a minor, barely noticeable glitch. [sent-8, score-0.53]
8 For those systems already doing off-line backups, this additional record of data coming into the service only needs to be since the last backup. [sent-9, score-0.139]
9 But, being cautious, we recommend going farther back anyway. [sent-10, score-0.252]
10 Mistakes happen and James says in Stonebraker on CAP Theorem and Databases that: Deferred delete is not full protection but it has saves my butt more than once and I’m a believer. [sent-11, score-0.619]
11 If you have an application error, administrative error, or database implementation bug that losses data, then it is simply gone unless you have an offline copy. [sent-12, score-0.699]
12 This, by the way, is why I’m a big fan of deferred delete. [sent-13, score-0.539]
wordName wordTfidf (topN-words)
[('deferred', 0.44), ('delete', 0.276), ('administrative', 0.215), ('deleted', 0.192), ('recover', 0.176), ('butt', 0.164), ('james', 0.16), ('farther', 0.154), ('cautious', 0.154), ('forgets', 0.147), ('losses', 0.147), ('record', 0.139), ('error', 0.137), ('embarrassing', 0.13), ('preferably', 0.13), ('clause', 0.13), ('mirroring', 0.124), ('marked', 0.118), ('noticeable', 0.118), ('minor', 0.11), ('barely', 0.106), ('theorem', 0.102), ('fan', 0.099), ('recommend', 0.098), ('neither', 0.097), ('mistake', 0.096), ('statement', 0.094), ('collected', 0.093), ('saves', 0.093), ('logical', 0.091), ('bug', 0.091), ('offline', 0.09), ('hamilton', 0.09), ('protect', 0.088), ('rolling', 0.088), ('mark', 0.087), ('protection', 0.086), ('backups', 0.083), ('copies', 0.083), ('technique', 0.083), ('happened', 0.081), ('cap', 0.081), ('unless', 0.08), ('garbage', 0.079), ('describes', 0.078), ('raid', 0.077), ('weeks', 0.076), ('gone', 0.076), ('deploying', 0.073), ('week', 0.071)]
simIndex simValue blogId blogTitle
same-blog 1 1.0 809 high scalability-2010-04-13-Strategy: Saving Your Butt With Deferred Deletes
Introduction: Deferred Deletes is a technique where deleted items are marked as deleted but not garbage collected until some days or preferably weeks later . James Hamilton talks describes this strategy in his classic On Designing and Deploying Internet-Scale Services: Never delete anything. Just mark it deleted. When new data comes in, record the requests on the way. Keep a rolling two week (or more) history of all changes to help recover from software or administrative errors. If someone makes a mistake and forgets the where clause on a delete statement (it has happened before and it will again), all logical copies of the data are deleted. Neither RAID nor mirroring can protect against this form of error. The ability to recover the data can make the difference between a highly embarrassing issue or a minor, barely noticeable glitch. For those systems already doing off-line backups, this additional record of data coming into the service only needs to be since the last backup. But, bein
2 0.08211495 1010 high scalability-2011-03-24-Strategy: Disk Backup for Speed, Tape Backup to Save Your Bacon, Just Ask Google
Introduction: In Stack Overflow Architecture Update - Now At 95 Million Page Views A Month , a commenter expressed surprise about Stack Overflow's backup strategy: Backup is to disk for fast retrieval and to tape for historical archiving. The comment was: Really? People still do this? I know some organizations invested a tremendous amount in automated, robotic tape backup, but seriously, a site founded in 2008 is backing up to tape? The Case of the Missing Gmail Accounts I admit that I was surprised at this strategy too. In this age of copying data to disk three times for safety, I also wondered if tape backups were still necessary? Then, like in a movie, an event happened that made sense of everything, Google suffered the quintessential #firstworldproblem , gmail accounts went missing! Queue emphatic music. And what's more they were taking a long time to come back. There was a palpable fear in the land that email accounts might never be restored. Think about that. They might ne
3 0.081064112 910 high scalability-2010-09-30-Facebook and Site Failures Caused by Complex, Weakly Interacting, Layered Systems
Introduction: Facebook has been so reliable that when a site outage does occur it's a definite learning opportunity. Fortunately for us we can learn something because in More Details on Today's Outage , Facebook's Robert Johnson gave a pretty candid explanation of what caused a rare 2.5 hour period of down time for Facebook. It wasn't a simple problem. The root causes were feedback loops and transient spikes caused ultimately by the complexity of weakly interacting layers in modern systems. You know, the kind everyone is building these days. Problems like this are notoriously hard to fix and finding a real solution may send Facebook back to the whiteboard. There's a technical debt that must be paid. The outline and my interpretation (reading between the lines) of what happened is: Remember that Facebook caches everything . They have 28 terabytes of memcached data on 800 servers. The database is the system of record, but memory is where the action is. So when a problem happens that i
4 0.080664419 1418 high scalability-2013-03-06-Low Level Scalability Solutions - The Aggregation Collection
Introduction: What good are problems without solutions? In 42 Monster Problems That Attack As Loads Increase we talked about problems. In this first post (OK, there was an earlier post, but I'm doing some reorganizing), we'll cover what I call aggregation strategies. Keep in mind these are low level architecture type suggestions of how to structure the components of your code and how they interact. We're not talking about massive scale-out clusters here, but of what your applications might like like internally, way below the service level interface level. There's a lot more to the world than evented architectures. Aggregation simply means we aren't using stupid queues. Our queues will be smart. We are deeply aware of queues as containers of work that eventually dictate how the entire system performs. As work containers we know intimately what requests and data sit in our queues and we can use that intelligence to our great advantage. Prioritize Work The key idea to it all is an almost mi
5 0.079774022 1589 high scalability-2014-02-03-How Google Backs Up the Internet Along With Exabytes of Other Data
Introduction: Raymond Blum leads a team of Site Reliability Engineers charged with keeping Google's data secret and keeping it safe. Of course Google would never say how much data this actually is, but from comments it seems that it is not yet a yottabyte , but is many exabytes in size. GMail alone is approaching low exabytes of data. Mr. Blum, in the video How Google Backs Up the Internet , explained common backup strategies don’t work for Google for a very googly sounding reason: typically they scale effort with capacity . If backing up twice as much data requires twice as much stuff to do it, where stuff is time, energy, space, etc., it won’t work, it doesn’t scale. You have to find efficiencies so that capacity can scale faster than the effort needed to support that capacity. A different plan is needed when making the jump from backing up one exabyte to backing up two exabytes. And the talk is largely about how Google makes that happen. Some major themes of the t
6 0.077042021 1415 high scalability-2013-03-04-7 Life Saving Scalability Defenses Against Load Monster Attacks
7 0.073508263 1279 high scalability-2012-07-09-Data Replication in NoSQL Databases
8 0.071644396 1209 high scalability-2012-03-14-The Azure Outage: Time Is a SPOF, Leap Day Doubly So
9 0.07064566 387 high scalability-2008-09-22-Paper: On Delivering Embarrassingly Distributed Cloud Services
10 0.069871441 1645 high scalability-2014-05-09-Stuff The Internet Says On Scalability For May 9th, 2014
11 0.067145683 921 high scalability-2010-10-18-NoCAP
12 0.062517248 926 high scalability-2010-10-24-Hot Scalability Links For Oct 24, 2010
13 0.061935514 1588 high scalability-2014-01-31-Stuff The Internet Says On Scalability For January 31st, 2014
14 0.061871208 806 high scalability-2010-04-08-Hot Scalability Links for April 8, 2010
15 0.061673291 378 high scalability-2008-09-03-Some Facebook Secrets to Better Operations
16 0.059722774 406 high scalability-2008-10-08-Strategy: Flickr - Do the Essential Work Up-front and Queue the Rest
17 0.058999512 950 high scalability-2010-11-30-NoCAP – Part III – GigaSpaces clustering explained..
18 0.058721948 1323 high scalability-2012-09-15-4 Reasons Facebook Dumped HTML5 and Went Native
19 0.058430504 1475 high scalability-2013-06-13-Busting 4 Modern Hardware Myths - Are Memory, HDDs, and SSDs Really Random Access?
20 0.057307012 1252 high scalability-2012-05-25-Stuff The Internet Says On Scalability For May 25, 2012
topicId topicWeight
[(0, 0.088), (1, 0.05), (2, -0.023), (3, 0.025), (4, 0.013), (5, 0.009), (6, 0.008), (7, -0.007), (8, -0.007), (9, -0.027), (10, -0.012), (11, 0.019), (12, 0.008), (13, -0.015), (14, 0.015), (15, 0.014), (16, 0.028), (17, 0.019), (18, 0.001), (19, 0.019), (20, -0.005), (21, -0.001), (22, 0.021), (23, 0.018), (24, -0.021), (25, -0.007), (26, 0.021), (27, 0.026), (28, -0.006), (29, 0.014), (30, -0.016), (31, -0.012), (32, 0.028), (33, 0.009), (34, -0.013), (35, 0.03), (36, 0.007), (37, 0.014), (38, 0.003), (39, 0.022), (40, 0.015), (41, -0.039), (42, -0.005), (43, 0.012), (44, 0.019), (45, -0.036), (46, 0.014), (47, 0.008), (48, -0.04), (49, -0.009)]
simIndex simValue blogId blogTitle
same-blog 1 0.90071607 809 high scalability-2010-04-13-Strategy: Saving Your Butt With Deferred Deletes
Introduction: Deferred Deletes is a technique where deleted items are marked as deleted but not garbage collected until some days or preferably weeks later . James Hamilton talks describes this strategy in his classic On Designing and Deploying Internet-Scale Services: Never delete anything. Just mark it deleted. When new data comes in, record the requests on the way. Keep a rolling two week (or more) history of all changes to help recover from software or administrative errors. If someone makes a mistake and forgets the where clause on a delete statement (it has happened before and it will again), all logical copies of the data are deleted. Neither RAID nor mirroring can protect against this form of error. The ability to recover the data can make the difference between a highly embarrassing issue or a minor, barely noticeable glitch. For those systems already doing off-line backups, this additional record of data coming into the service only needs to be since the last backup. But, bein
2 0.76123816 23 high scalability-2007-07-24-Major Websites Down: Or Why You Want to Run in Two or More Data Centers.
Introduction: A lot of sites hosted in San Francisco are down because of at least 6 back-to-back power outages power outages. More details at laughingsquid . Sites like SecondLife, Craigstlist, Technorati, Yelp and all Six Apart properties, TypePad, LiveJournal and Vox are all down. The cause was an underground explosion in a transformer vault under a manhole at 560 Mission Street. Flames shot 6 feet out from the manhole cover. Over PG&E; 30,000 customers are without power. What's perplexing is the UPS backup and diesel generators didn't kick in to bring the datacenter back on line. I've never toured that datacenter, but they usually have massive backup systems. It's probably one of those multiple simultaneous failure situations that you hope never happen in real life, but too often do. Or maybe the infrastructure wasn't rolled out completely. Update: the cause was a cascade of failures in a tightly couples system that could never happen :-) Details at Failure Happens: A summary of the power
3 0.72596705 1589 high scalability-2014-02-03-How Google Backs Up the Internet Along With Exabytes of Other Data
Introduction: Raymond Blum leads a team of Site Reliability Engineers charged with keeping Google's data secret and keeping it safe. Of course Google would never say how much data this actually is, but from comments it seems that it is not yet a yottabyte , but is many exabytes in size. GMail alone is approaching low exabytes of data. Mr. Blum, in the video How Google Backs Up the Internet , explained common backup strategies don’t work for Google for a very googly sounding reason: typically they scale effort with capacity . If backing up twice as much data requires twice as much stuff to do it, where stuff is time, energy, space, etc., it won’t work, it doesn’t scale. You have to find efficiencies so that capacity can scale faster than the effort needed to support that capacity. A different plan is needed when making the jump from backing up one exabyte to backing up two exabytes. And the talk is largely about how Google makes that happen. Some major themes of the t
4 0.72397411 1209 high scalability-2012-03-14-The Azure Outage: Time Is a SPOF, Leap Day Doubly So
Introduction: This is a guest post by Steve Newman, co-founder of Writely (Google Docs), tech lead on the Paxos-based synchronous replication in Megastore, and founder of cloud service provider Scalyr.com . Microsoft’s Azure service suffered a widely publicized outage on February 28th / 29th. Microsoft recently published an excellent postmortem . For anyone trying to run a high-availability service, this incident can teach several important lessons. The central lesson is that, no matter how much work you put into redundancy, problems will arise. Murphy is strong and, I might say, creative; things go wrong. So preventative measures are important, but how you react to problems is just as important. It’s interesting to review the Azure incident in this light. The postmortem is worth reading in its entirety, but here’s a quick summary: each time Azure launches a new VM, it creates a “transfer certificate” to secure communications with that VM. There was a bug in the code that determines the ce
Introduction: With a new Planet of the Apes coming out, this may be a touchy subject with our new overlords, but Netflix is using a whole lot more trouble injecting monkeys to test and iteratively harden their systems. We learned previously how Netflix used Chaos Monkey , a tool to test failover handling by continuously failing EC2 nodes. That was just a start. More monkeys have been added to the barrel. Node failure is just one problem in a system. Imagine a problem and you can imagine creating a monkey to test if your system is handling that problem properly. Yury Izrailevsky talks about just this approach in this very interesting post: The Netflix Simian Army . I know what you are thinking, if monkeys are so great then why has Netflix been down lately. Dmuino addressed this potential embarrassment, putting all fears of cloud inferiority to rest: Unfortunately we're not running 100% on the cloud today. We're working on it, and we could use more help. The latest outage was caused by a com
6 0.71620256 910 high scalability-2010-09-30-Facebook and Site Failures Caused by Complex, Weakly Interacting, Layered Systems
7 0.71246213 1283 high scalability-2012-07-13-Stuff The Internet Says On Scalability For July 13, 2012
9 0.70789242 1017 high scalability-2011-04-06-Netflix: Run Consistency Checkers All the time to Fixup Transactions
10 0.69523144 960 high scalability-2010-12-20-Netflix: Use Less Chatty Protocols in the Cloud - Plus 26 Fixes
11 0.69164199 1568 high scalability-2013-12-23-What Happens While Your Brain Sleeps is Surprisingly Like How Computers Stay Sane
12 0.68687868 1287 high scalability-2012-07-20-Stuff The Internet Says On Scalability For July 20, 2012
13 0.68666375 917 high scalability-2010-10-08-4 Scalability Themes from Surgecon
14 0.68504155 1365 high scalability-2012-11-30-Stuff The Internet Says On Scalability For November 30, 2012
15 0.68500453 350 high scalability-2008-07-15-ZooKeeper - A Reliable, Scalable Distributed Coordination System
16 0.68368614 1327 high scalability-2012-09-21-Stuff The Internet Says On Scalability For September 21, 2012
17 0.67933941 1372 high scalability-2012-12-14-Stuff The Internet Says On Scalability For December 14, 2012
20 0.67057657 1154 high scalability-2011-12-09-Stuff The Internet Says On Scalability For December 9, 2011
topicId topicWeight
[(1, 0.097), (2, 0.228), (10, 0.052), (30, 0.056), (50, 0.292), (61, 0.05), (79, 0.035), (85, 0.048), (94, 0.041)]
simIndex simValue blogId blogTitle
1 0.9122296 1357 high scalability-2012-11-12-Gone Fishin': Hilarious Video: Relational Database Vs NoSQL Fanbois
Introduction: This is an all time favorite post. Even though I've seen this video a hundred times I still can't help but laugh... This is so funny I laughed until I cried! Definitely NSFW. OMG it's hilarious, but it's also not a bad overview of the issues. Especially loved: You read the latest post on HighScalability.com and think you are a f*cking Google and architect and parrot slogans like Web Scale and Sharding but you have no idea what the f*ck you are talking about . There are so many more gems like that. Thanks to Alex Popescu for posting this on MongoDB is Web Scale . Whoever made this deserves a Webby.
2 0.88268542 895 high scalability-2010-09-05-Hilarious Video: Relational Database vs NoSQL Fanbois
Introduction: This is so funny I laughed until I cried! Definitely NSFW. OMG it's hilarious, but it's also not a bad overview of the issues. Especially loved: You read the latest post on HighScalability.com and think you are a f*cking Google and architect and parrot slogans like Web Scale and Sharding but you have no idea what the f*ck you are talking about . There are so many more gems like that. Thanks to Alex Popescu for posting this on MongoDB is Web Scale . Whoever made this deserves a Webby.
same-blog 3 0.87685329 809 high scalability-2010-04-13-Strategy: Saving Your Butt With Deferred Deletes
Introduction: Deferred Deletes is a technique where deleted items are marked as deleted but not garbage collected until some days or preferably weeks later . James Hamilton talks describes this strategy in his classic On Designing and Deploying Internet-Scale Services: Never delete anything. Just mark it deleted. When new data comes in, record the requests on the way. Keep a rolling two week (or more) history of all changes to help recover from software or administrative errors. If someone makes a mistake and forgets the where clause on a delete statement (it has happened before and it will again), all logical copies of the data are deleted. Neither RAID nor mirroring can protect against this form of error. The ability to recover the data can make the difference between a highly embarrassing issue or a minor, barely noticeable glitch. For those systems already doing off-line backups, this additional record of data coming into the service only needs to be since the last backup. But, bein
4 0.87597507 747 high scalability-2009-11-26-What I'm Thankful For on Thanksgiving
Introduction: I try to keep this blog targeted and on topic. So even though I may be thankful for the song of the tinniest sparrow at sunrise , I'll save you from all that. It's hard to tie scalability and the giving of thanks together, especially as it sometimes occurs to me that this blog may be a self-indulgent waste of time. But I think I found a sentiment in A New THEORY of AWESOMENESS and MIRACLES by James Bridle that manages to marry the topic of this blog and giving thanks meaningfully together: I distrust commercial definitions of innovation, and particularly of awesomeness. It’s an overused term. When I think of awesomeness, I want something awe-inspiring, vast and mind-expanding. So I started thinking about things that I think are awesome, or miraculous, and for me, it kept coming back to scale and complexity. We’re not actually very good about thinking about scale and complexity in real terms, so we have to use metaphors and examples. Douglas Adams writes s
5 0.83702523 586 high scalability-2009-04-29-Presentations: MySQL Conference & Expo 2009
Introduction: The Presentations of the MySQL Conference & Expo 2009 held April 20-23 in Santa Clara is available on the above link. They include: Beginner's Guide to Website Performance with MySQL and memcached by Adam Donnison Calpont: Open Source Columnar Storage Engine for Scalable MySQL DW by Jim Tommaney Creating Quick and Powerful Web Applications with MySQL, GlassFish, and NetBeans by Arun Gupta Deep-inspecting MySQL with DTrace by Domas Mituzas Distributed Innodb Caching with memcached by Matthew Yonkovit and Yves Trudeau Improving Performance by Running MySQL Multiple Times by MC Brown Introduction to Using DTrace with MySQL by Vince Carbone MySQL Cluster 7.0 - New Features by Johan Andersson Optimizing MySQL Performance with ZFS by Allan Packer SAN Performance on a Internal Disk Budget: The Coming Solid State Disk Revolution by Matthew Yonkovit This is Not a Web App: The Evolution of a MySQL Deployment at Google by Mark Callaghan
6 0.76525736 237 high scalability-2008-02-03-Product: Collectl - Performance Data Collector
8 0.75981164 1484 high scalability-2013-06-28-Stuff The Internet Says On Scalability For June 28, 2013
9 0.70281196 662 high scalability-2009-07-27-Handle 700 Percent More Requests Using Squid and APC Cache
10 0.68360865 1316 high scalability-2012-09-04-Changing Architectures: New Datacenter Networks Will Set Your Code and Data Free
11 0.67878109 661 high scalability-2009-07-25-Latency is Everywhere and it Costs You Sales - How to Crush it
12 0.67088807 1114 high scalability-2011-09-13-Must see: 5 Steps to Scaling MongoDB (Or Any DB) in 8 Minutes
13 0.66874605 1244 high scalability-2012-05-11-Stuff The Internet Says On Scalability For May 11, 2012
14 0.66387409 1064 high scalability-2011-06-20-35+ Use Cases for Choosing Your Next NoSQL Database
15 0.66298705 5 high scalability-2007-07-10-mixi.jp Architecture
16 0.66289318 1407 high scalability-2013-02-15-Stuff The Internet Says On Scalability For February 15, 2013
17 0.66232318 942 high scalability-2010-11-15-Strategy: Biggest Performance Impact is to Reduce the Number of HTTP Requests
18 0.66180348 256 high scalability-2008-02-21-Tracking usage of public resources - throttling accesses per hour
19 0.66013718 1346 high scalability-2012-10-24-Saving Cash Using Less Cache - 90% Savings in the Caching Tier
20 0.65991277 821 high scalability-2010-05-03-MocoSpace Architecture - 3 Billion Mobile Page Views a Month