high_scalability high_scalability-2009 high_scalability-2009-488 knowledge-graph by maker-knowledge-mining
Source: html
Introduction: I have two servers connected via Internet (NOT IN THE SAME LAN) serving the same website (http://www.ourexample.com).The problem is files uploaded on serverA and serverB cannot see each other immediately,thus rsync with certain intervals is not a good solution. Can anybody give me some advice on the following options? 1.NFS over Internet for file sharing 2.sshfs 3.inotify(our system's kernel does not support this and we donot want to risk upgrading our kernel as well) 4.drbd in active-active mode 5 or any other solutions Any suggestions will be welcomed. Thank you in advance.
sentIndex sentText sentNum sentScore
1 I have two servers connected via Internet (NOT IN THE SAME LAN) serving the same website (http://www. [sent-1, score-0.592]
2 The problem is files uploaded on serverA and serverB cannot see each other immediately,thus rsync with certain intervals is not a good solution. [sent-4, score-1.182]
3 Can anybody give me some advice on the following options? [sent-5, score-0.631]
4 inotify(our system's kernel does not support this and we donot want to risk upgrading our kernel as well) 4. [sent-9, score-1.235]
5 drbd in active-active mode 5 or any other solutions Any suggestions will be welcomed. [sent-10, score-0.209]
wordName wordTfidf (topN-words)
[('servera', 0.359), ('kernel', 0.356), ('intervals', 0.292), ('rsync', 0.273), ('lan', 0.263), ('anybody', 0.255), ('upgrading', 0.226), ('advance', 0.211), ('suggestions', 0.209), ('uploaded', 0.2), ('internet', 0.198), ('advice', 0.167), ('risk', 0.165), ('connected', 0.154), ('serving', 0.137), ('options', 0.137), ('certain', 0.13), ('following', 0.124), ('files', 0.116), ('via', 0.097), ('website', 0.093), ('file', 0.088), ('give', 0.085), ('http', 0.081), ('support', 0.073), ('problem', 0.064), ('well', 0.061), ('two', 0.06), ('want', 0.059), ('see', 0.057), ('servers', 0.051), ('good', 0.05), ('system', 0.037)]
simIndex simValue blogId blogTitle
same-blog 1 1.0 488 high scalability-2009-01-08-file synchronization solutions
Introduction: I have two servers connected via Internet (NOT IN THE SAME LAN) serving the same website (http://www.ourexample.com).The problem is files uploaded on serverA and serverB cannot see each other immediately,thus rsync with certain intervals is not a good solution. Can anybody give me some advice on the following options? 1.NFS over Internet for file sharing 2.sshfs 3.inotify(our system's kernel does not support this and we donot want to risk upgrading our kernel as well) 4.drbd in active-active mode 5 or any other solutions Any suggestions will be welcomed. Thank you in advance.
2 0.14049056 98 high scalability-2007-09-18-Sync data on all servers
Introduction: I have a few apache servers ( arround 11 atm ) serving a small amount of data ( arround 44 gigs right now ). For some time I have been using rsync to keep all the content equal on all servers, but the amount of data has been growing, and rsync takes a few too much time to "compare" all data from source to destination, and create a lot of I/O. I have been taking a look at MogileFS, it seems a good and reliable option, but as the fuse module is not finished, we should have to rewrite all our apps, and its not an option atm. Any ideas? I just want a "real time, non resource-hungry" solution alternative for rsync. If I get more features on the way, then they are welcome :) Why I prefer to use a Distributed File System instead of using NAS + NFS? - I need 2 NAS, if I dont want a point of failure, and NAS hard is expensive. - Non-shared hardware, all server has their own local disks. - As files are replicated, I can save a lot of money, RAID is not a MUST. Thn
3 0.12654716 1386 high scalability-2013-01-14-MongoDB and GridFS for Inter and Intra Datacenter Data Replication
Introduction: This is a guest post by Jeff Behl , VP Ops @ LogicMonitor. Jeff has been a bit herder for the last 20 years, architecting and overseeing the infrastructure for a number of SaaS based companies. Data Replication for Disaster Recovery An inevitable part of disaster recovery planning is making sure customer data exists in multiple locations. In the case of LogicMonitor, a SaaS-based monitoring solution for physical, virtual, and cloud environments, we wanted copies of customer data files both within a data center and outside of it. The former was to protect against the loss of individual servers within a facility, and the latter for recovery in the event of the complete loss of a data center. Where we were: Rsync Like most everyone who starts off in a Linux environment, we used our trusty friend rsync to copy data around. Rsync is tried, true and tested, and works well when the number of servers, the amount of data, and the number of files is not horrendous.
Introduction: Now that we have the C10K concurrent connection problem licked, how do we level up and support 10 million concurrent connections? Impossible you say. Nope, systems right now are delivering 10 million concurrent connections using techniques that are as radical as they may be unfamiliar. To learn how it’s done we turn to Robert Graham , CEO of Errata Security, and his absolutely fantastic talk at Shmoocon 2013 called C10M Defending The Internet At Scale . Robert has a brilliant way of framing the problem that I’ve never heard of before. He starts with a little bit of history, relating how Unix wasn’t originally designed to be a general server OS, it was designed to be a control system for a telephone network. It was the telephone network that actually transported the data so there was a clean separation between the control plane and the data plane. The problem is we now use Unix servers as part of the data plane , which we shouldn’t do at all. If we were des
5 0.10635562 199 high scalability-2008-01-01-S3 for image storing
Introduction: Hi all, Has anyone got any experience with using Amazon S3 as an uploaded photo store? I'm writing a website that I need to keep as low budget as possible, and I'm investigating solutions for storing uploaded photos from users - not too many, probably in the low thousands. The site is commercial so I'm straying away from the Flickrs of the world. S3 seems to offer a solution but I'd like to hear from those who have used it before. Thanks Andy
6 0.09414281 308 high scalability-2008-04-22-Simple NFS failover solution with symbolic link?
7 0.092048287 78 high scalability-2007-09-01-2 tier switch selection for colocation
8 0.083757393 1402 high scalability-2013-02-07-Ask HighScalability: Web asset server concept - 3rd party software available?
9 0.082378827 130 high scalability-2007-10-24-Scaling Operations Saves Money and Scales Faster
10 0.074507073 250 high scalability-2008-02-17-Web Accelerators - snake oil or miracle remedy?
11 0.071819276 889 high scalability-2010-08-30-Pomegranate - Storing Billions and Billions of Tiny Little Files
12 0.071596116 1460 high scalability-2013-05-17-Stuff The Internet Says On Scalability For May 17, 2013
13 0.071399838 1102 high scalability-2011-08-22-Strategy: Run a Scalable, Available, and Cheap Static Site on S3 or GitHub
14 0.070731893 229 high scalability-2008-01-29-Building scalable storage into application - Instead of MogileFS OpenAFS etc.
15 0.067522831 266 high scalability-2008-03-04-Manage Downtime Risk by Connecting Multiple Data Centers into a Secure Virtual LAN
16 0.063864723 1268 high scalability-2012-06-20-Ask HighScalability: How do I organize millions of images?
17 0.063846126 1161 high scalability-2011-12-22-Architecting Massively-Scalable Near-Real-Time Risk Analysis Solutions
18 0.061997555 1501 high scalability-2013-08-13-In Memoriam: Lavabit Architecture - Creating a Scalable Email Service
19 0.060167104 1174 high scalability-2012-01-13-Stuff The Internet Says On Scalability For January 13, 2012
20 0.059967943 283 high scalability-2008-03-18-Shared filesystem on EC2
topicId topicWeight
[(0, 0.076), (1, 0.022), (2, 0.006), (3, -0.049), (4, -0.012), (5, -0.013), (6, 0.014), (7, -0.0), (8, -0.001), (9, 0.032), (10, -0.024), (11, -0.037), (12, -0.003), (13, -0.032), (14, 0.046), (15, 0.018), (16, 0.048), (17, 0.028), (18, -0.027), (19, -0.027), (20, 0.013), (21, -0.019), (22, -0.033), (23, 0.049), (24, 0.021), (25, 0.003), (26, 0.054), (27, -0.013), (28, -0.052), (29, -0.023), (30, -0.01), (31, -0.002), (32, 0.016), (33, 0.009), (34, -0.008), (35, 0.053), (36, 0.032), (37, -0.012), (38, -0.02), (39, 0.007), (40, 0.007), (41, 0.014), (42, -0.033), (43, 0.001), (44, -0.011), (45, 0.056), (46, -0.035), (47, 0.025), (48, -0.037), (49, 0.013)]
simIndex simValue blogId blogTitle
same-blog 1 0.96396714 488 high scalability-2009-01-08-file synchronization solutions
Introduction: I have two servers connected via Internet (NOT IN THE SAME LAN) serving the same website (http://www.ourexample.com).The problem is files uploaded on serverA and serverB cannot see each other immediately,thus rsync with certain intervals is not a good solution. Can anybody give me some advice on the following options? 1.NFS over Internet for file sharing 2.sshfs 3.inotify(our system's kernel does not support this and we donot want to risk upgrading our kernel as well) 4.drbd in active-active mode 5 or any other solutions Any suggestions will be welcomed. Thank you in advance.
2 0.71122116 283 high scalability-2008-03-18-Shared filesystem on EC2
Introduction: Hi. I'm looking for a way to share files between EC2 nodes. Currently we are using glusterfs to do this. It has been reliable recently, but in the past it has crashed under high load and we've had trouble starting it up again. We've only been able to restart it by removing the files, restarting the cluster, and filing it up again with our files from backup. This takes ages, and will take even longer the more files we get. What worries me is that it seems to make each node a point of failure for the entire system. One node crashes and soon the entire cluster has crashed. The other problem is adding another node. It seems like you have to take down the whole thing, reconfigure to include the new node, and restart. This kind of defeats the horizontal scaling strategy. We are using 2 EC2 instances as web servers, 1 as a DB master, and 1 as a slave. GlusterFS is installed on the web server machines as well as the DB slave machine (we backup files to s3 from this machine). The files
3 0.63902241 143 high scalability-2007-11-06-Product: ChironFS
Introduction: If you are trying to create highly available file systems, especially across data centers, then ChironFS is one potential solution. It's relatively new, so there aren't lots of experience reports, but it looks worth considering. What is ChironFS and how does it work? Adapted from the ChironFS website: The Chiron Filesystem is a Fuse based filesystem that frees you from single points of failure. It's main purpose is to guarantee filesystem availability using replication. But it isn't a RAID implementation. RAID replicates DEVICES not FILESYSTEMS. Why not just use RAID over some network block device? Because it is a block device and if one server mounts that device in RW mode, no other server will be able to mount it in RW mode. Any real network may have many servers and offer a variety of services. Keeping everything running can become a real nightmare!
Introduction: We are serving dynamic (PHP) websites and their assets (JS/CSS/images/videos/binary downloads) via the same apache hosts. The static files are only being used as origins for CDN services used to distribute those files. Yet, in the current development-deploy pipeline, these files are checked into the same version control repositories as the code is. This is what we would like to change, for several reasons (decouple asset deployment from development & developers, lessen size of code repositories, etc.) My idea is to do the following: Set up a media server (cluster) which serves as an API (REST e.g.). You can PUT files to it, and get back the URL the file is available through from the public. In between input and output, the media service deals with everything that's necessary to serve the files: Upload them to the CDN, create the public URL, write the meta data to a (relational?) database, assign a version number... This API can be used by a) the application/website directly to provid
5 0.6078434 329 high scalability-2008-05-27-Secure Remote Administration for Large-Scale Networks
Introduction: This website has been a great resource for helping me to understand the successful (and failed) scalable network designs from organizations that have actually done it, but I haven't seen any explicite explanations about secure remote administration of these systems. I understand that the *nix people love to SSH, and the windows gang has their RDP, but how does one go about creating a network architecture that both allows one to manage their systems and does its best to avoid hacker interest? As I imagine, no big website will have the SSH/RDP/FTP ports open on the web server, so how is it that they go about remotely administering their geographically diverse groups of servers securely?
6 0.6068998 177 high scalability-2007-12-08-thesimsonstage.ea.com
7 0.60545146 262 high scalability-2008-02-26-Architecture to Allow High Availability File Upload
8 0.60505468 98 high scalability-2007-09-18-Sync data on all servers
9 0.60247046 78 high scalability-2007-09-01-2 tier switch selection for colocation
10 0.58671415 598 high scalability-2009-05-12-P2P server technology?
11 0.58438706 605 high scalability-2009-05-22-Distributed content system with bandwidth balancing
12 0.57983196 165 high scalability-2007-11-26-Scale to China
13 0.57520121 1288 high scalability-2012-07-23-Ask HighScalability: How Do I Build My MegaUpload + Itunes + YouTube Startup?
14 0.57438493 278 high scalability-2008-03-16-Product: GlusterFS
15 0.56333572 293 high scalability-2008-03-31-Read HighScalability on Your Mobile Phone Using WidSets Widgets
16 0.56268203 231 high scalability-2008-01-29-Too many databases
17 0.56152898 686 high scalability-2009-08-20-VMware to bridge a DMZ.
18 0.56144881 229 high scalability-2008-01-29-Building scalable storage into application - Instead of MogileFS OpenAFS etc.
19 0.55583775 308 high scalability-2008-04-22-Simple NFS failover solution with symbolic link?
20 0.54844707 251 high scalability-2008-02-18-How to deal with an I-O bottleneck to disk?
topicId topicWeight
[(1, 0.138), (2, 0.131), (10, 0.074), (69, 0.281), (79, 0.194), (94, 0.024)]
simIndex simValue blogId blogTitle
same-blog 1 0.90775955 488 high scalability-2009-01-08-file synchronization solutions
Introduction: I have two servers connected via Internet (NOT IN THE SAME LAN) serving the same website (http://www.ourexample.com).The problem is files uploaded on serverA and serverB cannot see each other immediately,thus rsync with certain intervals is not a good solution. Can anybody give me some advice on the following options? 1.NFS over Internet for file sharing 2.sshfs 3.inotify(our system's kernel does not support this and we donot want to risk upgrading our kernel as well) 4.drbd in active-active mode 5 or any other solutions Any suggestions will be welcomed. Thank you in advance.
2 0.85261631 87 high scalability-2007-09-10-Blog: Esoteric Curio by Theo Schlossnagle
Introduction: Theo Schlossnagle is the author of Scalable Internet Architecture and the funder of OmniTI , a global leader in Internet technology services that power the World Wide Web and email. As you might imagine Theo frequently posts on interesting topics for the scalable website builder. A Quick Hit of What's Inside Partitioning vs. Federation vs. Sharding , PostgreSQL warm standby on ZFS crack , Scalability vs. Performance: it isn't a battle  Site: http://lethargy.org/~jesus
3 0.82770896 1298 high scalability-2012-08-05-Ask MemSQL: Anything you want to know about MemSQL?
Introduction: A very shy team of ex-Facebookers have created MemSQL , what they claim is the world's fastest database, and I'll be interviewing them on Tuesday. Are there any questions you would like to ask the MemSQL team? If so, please contact me or make a comment on this thread.
Introduction: This is a repost of the blog entry written by NuoDB's Tommy Reilly . We at NuoDB were recently given the opportunity to kick the tires on the Google Compute Engine by our friends over at Google. You can watch the entire Google Developer Live Session by clicking here . In order to access the capabilities of GCE we decided to run the same YCSB based benchmark we ran at our General Availability Launch back in January. For those of you who missed it we demonstrated running the YCSB benchmark on a 24 machine cluster running on our private cloud in the NuoDB datacenter. The salient results were 1.7 million transactions per second with sub-millisecond latencies. Public cloud environments typically mean virtualization, inconsistent network performance and potentially slow or low bandwidth disk access. It just so happens that NuoDB was designed to work well in such harsh environments (we don’t call it a cloud database for nothing). Still, the faster the CPU, network and disk t
5 0.77101427 1047 high scalability-2011-05-25-Stuff to Watch from Surge 2010
Introduction: Surge is a conference put on by OmniTI targeting practical Scalability matters. OmniTI specializes in helping people solve their scalability problems, as is only natural, as it was founded by Theo Schlossnagle, author of the canonical Scalable Internet Architectures . Now that Surge 2011 is on the horizon, they've generously made available nearly all the videos from the Surge 2010 conference. A pattern hopefully every conference will follow (only don't wait a year please). We lose a lot of collective wisdom from events not being available online in a timely manner. In truth, nearly all the talks are on topic and are worth watching, but here are a few that seem especially relevant: Going 0 to 60: Scaling LinkedIn by Ruslan Belkin, Sr. Director of Engineering, LinkedIn. Have you ever wondered what architectures the site like LinkedIn may have used and what insights teams have learned while growing the system from serving just a handful to close to a hundred m
6 0.7690109 31 high scalability-2007-07-26-Product: Symfony a Web Framework
7 0.75159627 763 high scalability-2010-01-22-How BuddyPoke Scales on Facebook Using Google App Engine
8 0.70280695 1275 high scalability-2012-07-02-C is for Compute - Google Compute Engine (GCE)
9 0.69072884 515 high scalability-2009-02-19-GIS Application Hosting
10 0.68613774 89 high scalability-2007-09-10-Is there a difference between partitioning and federation and sharding?
11 0.68470824 448 high scalability-2008-11-22-Google Architecture
13 0.68042427 680 high scalability-2009-08-13-Reconnoiter - Large-Scale Trending and Fault-Detection
14 0.67881143 380 high scalability-2008-09-05-Product: Tungsten Replicator
15 0.67809683 421 high scalability-2008-10-17-A High Performance Memory Database for Web Application Caches
16 0.67708564 1343 high scalability-2012-10-18-Save up to 30% by Selecting Better Performing Amazon Instances
17 0.6760692 1136 high scalability-2011-11-03-Paper: G2 : A Graph Processing System for Diagnosing Distributed Systems
18 0.6754663 786 high scalability-2010-03-02-Using the Ambient Cloud as an Application Runtime
19 0.67227066 1328 high scalability-2012-09-24-Google Spanner's Most Surprising Revelation: NoSQL is Out and NewSQL is In
20 0.67211956 1494 high scalability-2013-07-19-Stuff The Internet Says On Scalability For July 19, 2013