high_scalability high_scalability-2008 high_scalability-2008-307 knowledge-graph by maker-knowledge-mining
Source: html
Introduction: Over the years I've accumulated quite a rag tag collection of personal systems scattered wide across a galaxy of different servers. For the past month I've been on a quest to rationalize this conglomeration by moving everything to a managed service of one kind or another. The goal: lift a load of worry from my mind. I like to do my own stuff my self so I learn something and have control. Control always comes with headaches and it was time for a little aspirin. As part of the process GAE came in handy as a host for a few Twitter related scripts I couldn't manage to run anywhere else. I recoded my simple little scripts into Python/GAE and learned a lot in the process. In the move I exported HighScalability from a VPS and imported it into a shared hosting service. I could never quite configure Apache and MySQL well enough that they wouldn't spike memory periodically and crash the VPS. And since a memory crash did not automatically restarted it was unacceptable. I also wrote a scrip
sentIndex sentText sentNum sentScore
1 I like to do my own stuff my self so I learn something and have control. [sent-4, score-0.176]
2 I recoded my simple little scripts into Python/GAE and learned a lot in the process. [sent-7, score-0.128]
3 I could never quite configure Apache and MySQL well enough that they wouldn't spike memory periodically and crash the VPS. [sent-9, score-0.148]
4 I also wrote a script to convert a few thousand pages of JSPWiki to MediaWiki format, moved from my own mail server, moved all my code to a hosted SVN server, and moved a few other blogs and static sites along the way. [sent-11, score-0.291]
5 In one script (Perl) I login to Twitter and download the most recent tweets for an account and display them on a web page. [sent-15, score-0.147]
6 But the warm fuzzy feeling you get from changing code and immediately running it never gets old. [sent-45, score-0.171]
7 My original Perl code looks like: use strict; use warnings; use CGI; use LWP; eval { my $query = new CGI; print $query->header; my $callback= $query->param("callback"); my $url= "http://twitter. [sent-47, score-0.166]
8 There’s not a practical “help get stuff done” line in it. [sent-56, score-0.182]
9 headers['Content-Type'] = 'text/plain' callback = self. [sent-61, score-0.163]
10 encodestring('%s:%s' % ("user", "password"))[:-1] headers = {'Authorization': "Basic %s" % base64string} url = "http://twitter. [sent-64, score-0.449]
11 content + ")") def main(): application = webapp. [sent-72, score-0.164]
12 WSGIApplication( [('/innertwitter', InnerTwitter)], debug=True) For me the Perl code was easier simply because there is example code everywhere. [sent-73, score-0.14]
13 My next task was to write a string to a twitter account. [sent-82, score-0.305]
14 com"; String password = "password"; try { String chime= getChimeForUser(username); String msg= "status=" + URLEncoder. [sent-84, score-0.181]
15 encode(chime); msg+= "&source;=innertwitter"; URL url = new URL("http://twitter. [sent-85, score-0.245]
16 headers['Content-Type'] = 'text/plain' username = self. [sent-128, score-0.196]
17 get("username") login = username password = "password" chime = self. [sent-130, score-0.861]
18 encodestring('%s:%s' % (login, password))[:-1] headers = {'Authorization': "Basic %s" % base64string} url = "http://twitter. [sent-133, score-0.449]
19 content) def get_chime(self): now = datetime. [sent-141, score-0.164]
20 ctime() return chime def main(): application = webapp. [sent-166, score-0.572]
wordName wordTfidf (topN-words)
[('chime', 0.408), ('url', 0.245), ('string', 0.241), ('innertwitter', 0.227), ('msg', 0.205), ('headers', 0.204), ('username', 0.196), ('password', 0.181), ('def', 0.164), ('payload', 0.163), ('callback', 0.163), ('didn', 0.123), ('python', 0.112), ('self', 0.11), ('gae', 0.108), ('cgi', 0.096), ('print', 0.096), ('bufferedreader', 0.091), ('getchimeforuser', 0.091), ('lwp', 0.091), ('outputstreamwriter', 0.091), ('sendchime', 0.091), ('date', 0.088), ('header', 0.084), ('req', 0.082), ('wasn', 0.082), ('scripts', 0.079), ('login', 0.076), ('perl', 0.076), ('http', 0.074), ('worry', 0.072), ('script', 0.071), ('code', 0.07), ('credentials', 0.067), ('line', 0.067), ('stuff', 0.066), ('basic', 0.064), ('authorization', 0.064), ('twitter', 0.064), ('hosting', 0.062), ('appengine', 0.056), ('vps', 0.053), ('figured', 0.053), ('never', 0.052), ('moved', 0.05), ('periodically', 0.049), ('agent', 0.049), ('little', 0.049), ('get', 0.049), ('quite', 0.047)]
simIndex simValue blogId blogTitle
same-blog 1 1.0 307 high scalability-2008-04-21-Using Google AppEngine for a Little Micro-Scalability
Introduction: Over the years I've accumulated quite a rag tag collection of personal systems scattered wide across a galaxy of different servers. For the past month I've been on a quest to rationalize this conglomeration by moving everything to a managed service of one kind or another. The goal: lift a load of worry from my mind. I like to do my own stuff my self so I learn something and have control. Control always comes with headaches and it was time for a little aspirin. As part of the process GAE came in handy as a host for a few Twitter related scripts I couldn't manage to run anywhere else. I recoded my simple little scripts into Python/GAE and learned a lot in the process. In the move I exported HighScalability from a VPS and imported it into a shared hosting service. I could never quite configure Apache and MySQL well enough that they wouldn't spike memory periodically and crash the VPS. And since a memory crash did not automatically restarted it was unacceptable. I also wrote a scrip
2 0.17438811 301 high scalability-2008-04-08-Google AppEngine - A First Look
Introduction: I haven't developed an AppEngine application yet, I'm just taking a look around their documentation and seeing what stands out for me. It's not the much speculated super cluster VM . AppEngine is solidly grounded in code and structure. It reminds me a little of the guy who ran a website out of S3 with a splash of Heroku thrown in as a chaser. The idea is clearly to take advantage of our massive multi-core future by creating a shared nothing infrastructure based firmly on a core set of infinitely scalable database, storage and CPU services. Don't forget Google also has a few other services to leverage: email, login, blogs, video, search, ads, metrics, and apps. A shared nothing request is a simple beast. By its very nature shared nothing architectures must be composed of services which are themselves already scalable and Google is signing up to supply that scalable infrastructure. Google has been busy creating a platform of out-of-the-box scalable services to build
3 0.14651273 1112 high scalability-2011-09-07-What Google App Engine Price Changes Say About the Future of Web Architecture
Introduction: When I was a child, I spake as a child, I understood as a child, I thought as a child: but when I became a man, I put away childish things . -- Corinthians With this new pricing, developments will be driven by the costs . I like to optimize my apps to make them better or faster, but to optimize them just to make them cheaper is a waste of time. -- Sylvain on Google Groups The dream is dead. Google App Engine's bold pay for what you use dream dies as it leaves childish things behind and becomes a real product . Pricing will change . Architectures will change. Customers will change. Hearts and minds will change. But Google App Engine will survive. Google is shutting down many of its projects . GAE is not among them. Do we have GAE's pricing change to thank for it surving the more wood behind more deadly arrows push? Without a radical and quick shift towards profitably GAE would no doubt be a historical footnote in the long scroll of good ideas. The urgency involve
4 0.1455597 517 high scalability-2009-02-21-Google AppEngine - A Second Look
Introduction: Update 6: : Back to the Future for Data Storage . We are in the middle of a renaissance in data storage with the application of many new ideas and techniques; there's huge potential for breaking out of thinking about data storage in just one way. Update 5 : Building Scalable Web Applications with Google App Engine by Brett Slatkin. Update 4 : Why Google App Engine is broken and what Google must do to fix it by Aral Balkan. We don't care that it can scale. We care that it does scale. And that it scales when you need it the most. Issues: 1MB limit on data structures; 1MB limit on data structures; the short-term high CPU quota; quotas in general; Admin? What's that? Update 3 : BigTable Blues . Catherine Devlin couldn't port an application to GAE because it can't do basic filtering and can't search 5,000 records without timing out: "Querying from 5000 records - too much for the mighty BigTable, apparently." Followup: not the future database . "90% of the work of this proje
5 0.1368957 571 high scalability-2009-04-15-Using HTTP cache headers effectively
Introduction: Hi, Some time ago , martin fowler bloged about how HTTP cache headers can be very effectively used in web site design. http://www.martinfowler.com/bliki/SegmentationByFreshness.html How actively HTTP cache headers are considered in web site design? I think it is a great tool to reduce lot of load on server and should be considered before designing any complex caching strategy. Thoughts? Thanks, Unmesh
6 0.12038045 115 high scalability-2007-10-07-Using ThreadLocal to pass context information around in web applications
7 0.11232122 62 high scalability-2007-08-08-Partial String Matching
8 0.11063869 1404 high scalability-2013-02-11-At Scale Even Little Wins Pay Off Big - Google and Facebook Examples
9 0.10405786 714 high scalability-2009-10-02-HighScalability has Moved to Squarespace.com!
10 0.10369707 1215 high scalability-2012-03-26-7 Years of YouTube Scalability Lessons in 30 Minutes
11 0.10369416 1501 high scalability-2013-08-13-In Memoriam: Lavabit Architecture - Creating a Scalable Email Service
12 0.10125929 172 high scalability-2007-12-02-nginx: high performance smpt-pop-imap proxy
13 0.099648654 67 high scalability-2007-08-17-What is the best hosting option?
14 0.098741099 623 high scalability-2009-06-10-Dealing with multi-partition transactions in a distributed KV solution
15 0.0950054 1102 high scalability-2011-08-22-Strategy: Run a Scalable, Available, and Cheap Static Site on S3 or GitHub
16 0.091804385 639 high scalability-2009-06-27-Scaling Twitter: Making Twitter 10000 Percent Faster
17 0.090986386 1068 high scalability-2011-06-27-TripAdvisor Architecture - 40M Visitors, 200M Dynamic Page Views, 30TB Data
18 0.090922594 1650 high scalability-2014-05-19-A Short On How the Wayback Machine Stores More Pages than Stars in the Milky Way
19 0.088501185 449 high scalability-2008-11-24-Product: Scribe - Facebook's Scalable Logging System
20 0.087923959 1048 high scalability-2011-05-27-Stuff The Internet Says On Scalability For May 27, 2011
topicId topicWeight
[(0, 0.163), (1, 0.074), (2, -0.022), (3, -0.025), (4, 0.023), (5, -0.066), (6, 0.008), (7, 0.014), (8, 0.033), (9, -0.028), (10, -0.035), (11, 0.009), (12, 0.017), (13, -0.027), (14, 0.003), (15, -0.086), (16, -0.029), (17, -0.036), (18, 0.034), (19, -0.062), (20, -0.031), (21, -0.072), (22, 0.001), (23, 0.006), (24, 0.029), (25, 0.039), (26, 0.053), (27, -0.07), (28, 0.013), (29, -0.004), (30, -0.03), (31, -0.072), (32, -0.056), (33, 0.034), (34, -0.018), (35, 0.008), (36, -0.02), (37, 0.033), (38, 0.005), (39, 0.008), (40, -0.026), (41, -0.022), (42, 0.008), (43, -0.002), (44, 0.03), (45, -0.027), (46, -0.008), (47, 0.014), (48, -0.016), (49, -0.007)]
simIndex simValue blogId blogTitle
same-blog 1 0.96112937 307 high scalability-2008-04-21-Using Google AppEngine for a Little Micro-Scalability
Introduction: Over the years I've accumulated quite a rag tag collection of personal systems scattered wide across a galaxy of different servers. For the past month I've been on a quest to rationalize this conglomeration by moving everything to a managed service of one kind or another. The goal: lift a load of worry from my mind. I like to do my own stuff my self so I learn something and have control. Control always comes with headaches and it was time for a little aspirin. As part of the process GAE came in handy as a host for a few Twitter related scripts I couldn't manage to run anywhere else. I recoded my simple little scripts into Python/GAE and learned a lot in the process. In the move I exported HighScalability from a VPS and imported it into a shared hosting service. I could never quite configure Apache and MySQL well enough that they wouldn't spike memory periodically and crash the VPS. And since a memory crash did not automatically restarted it was unacceptable. I also wrote a scrip
Introduction: Update 4: Heroku versus GAE & GAE/J Update 3 : Heroku has gone live! . Congratulations to the team. It's difficult right now to get a feeling for the relative cost and reliability of Heroku, but it's an impressive accomplishment and a viable option for people looking for a delivery platform. Update 2 : Heroku Architecture . A great interactive presentation of the Heroku stack. Requests flow into Nginx used as a HTTP Reverse Proxy. Nginx routes requests into a Varnish based HTTP cache. Then requests are injected into an Erlang based routing mesh that balances requests across a grid of dynos. Dynos are your application "VMs" that implement application specific behaviors. Dynos themselves are a stack of: POSIX, Ruby VM, App Server, Rack, Middleware, Framework, Your App. Applications can access PostgreSQL. Memcached is used as an application caching layer. Update : Aaron Worsham Interview with James Lindenbaum, CEO of Heroku . Aaron nicely sums up their goal: Heroku is
3 0.72574872 218 high scalability-2008-01-17-Moving old to new. Do not be afraid of the re-write -- but take some help
Introduction: Recently I had to help users on one of my opensource project ISPMan. http://ispman.net This project started in 2001 as I was too unwilling to take care of the DNS and VitualHosting stuff as it was a side-thing to the company I worked for (so i wrote a software that took care of all these little details) Summary: A large project that needs a rewrite can be done in a matter of day. I will not give you a full case study about a project that went through a re-write but a case study about how easy it is to re-write something. Details: My boss was cool enough to let me open-source the project and obviously, I got a lot of cool-cred out of it. Later on I also did some support and implementation and earned quiet some money with it. Eventually I had to let the project go out of my hand to the community as I only did it to facilitate a job that wasnt williing to do. (Setup DNS zones of multiple servers, find out which host should host the website and put VirtualHost
4 0.71828842 517 high scalability-2009-02-21-Google AppEngine - A Second Look
Introduction: Update 6: : Back to the Future for Data Storage . We are in the middle of a renaissance in data storage with the application of many new ideas and techniques; there's huge potential for breaking out of thinking about data storage in just one way. Update 5 : Building Scalable Web Applications with Google App Engine by Brett Slatkin. Update 4 : Why Google App Engine is broken and what Google must do to fix it by Aral Balkan. We don't care that it can scale. We care that it does scale. And that it scales when you need it the most. Issues: 1MB limit on data structures; 1MB limit on data structures; the short-term high CPU quota; quotas in general; Admin? What's that? Update 3 : BigTable Blues . Catherine Devlin couldn't port an application to GAE because it can't do basic filtering and can't search 5,000 records without timing out: "Querying from 5000 records - too much for the mighty BigTable, apparently." Followup: not the future database . "90% of the work of this proje
5 0.70842594 301 high scalability-2008-04-08-Google AppEngine - A First Look
Introduction: I haven't developed an AppEngine application yet, I'm just taking a look around their documentation and seeing what stands out for me. It's not the much speculated super cluster VM . AppEngine is solidly grounded in code and structure. It reminds me a little of the guy who ran a website out of S3 with a splash of Heroku thrown in as a chaser. The idea is clearly to take advantage of our massive multi-core future by creating a shared nothing infrastructure based firmly on a core set of infinitely scalable database, storage and CPU services. Don't forget Google also has a few other services to leverage: email, login, blogs, video, search, ads, metrics, and apps. A shared nothing request is a simple beast. By its very nature shared nothing architectures must be composed of services which are themselves already scalable and Google is signing up to supply that scalable infrastructure. Google has been busy creating a platform of out-of-the-box scalable services to build
6 0.70111549 993 high scalability-2011-02-22-Is Node.js Becoming a Part of the Stack? SimpleGeo Says Yes.
7 0.70079243 1269 high scalability-2012-06-20-iDoneThis - Scaling an Email-based App from Scratch
8 0.69971859 1432 high scalability-2013-04-01-Khan Academy Checkbook Scaling to 6 Million Users a Month on GAE
9 0.68896252 970 high scalability-2011-01-06-BankSimple Mini-Architecture - Using a Next Generation Toolchain
10 0.6876362 1112 high scalability-2011-09-07-What Google App Engine Price Changes Say About the Future of Web Architecture
11 0.6861611 1277 high scalability-2012-07-05-10 Golden Principles For Building Successful Mobile-Web Applications
12 0.68106723 369 high scalability-2008-08-18-Code deployment tools
13 0.67325032 556 high scalability-2009-04-05-At Some Point the Cost of Servers Outweighs the Cost of Programmers
14 0.67278093 1084 high scalability-2011-07-22-Stuff The Internet Says On Scalability For July 22, 2011
15 0.67225295 166 high scalability-2007-11-27-Solving the Client Side API Scalability Problem with a Little Game Theory
16 0.66928744 1469 high scalability-2013-06-03-GOV.UK - Not Your Father's Stack
17 0.66499871 639 high scalability-2009-06-27-Scaling Twitter: Making Twitter 10000 Percent Faster
18 0.65879494 1523 high scalability-2013-09-27-Stuff The Internet Says On Scalability For September 27, 2013
19 0.64843273 1470 high scalability-2013-06-05-A Simple 6 Step Transition Guide for Moving Away from X to AWS
20 0.64636046 1230 high scalability-2012-04-18-Ansible - A Simple Model-Driven Configuration Management and Command Execution Framework
topicId topicWeight
[(1, 0.083), (2, 0.171), (10, 0.032), (21, 0.153), (22, 0.018), (30, 0.011), (40, 0.017), (47, 0.023), (52, 0.02), (61, 0.066), (77, 0.049), (79, 0.089), (85, 0.053), (94, 0.096)]
simIndex simValue blogId blogTitle
1 0.90355653 951 high scalability-2010-12-01-8 Commonly Used Scalable System Design Patterns
Introduction: Ricky Ho in Scalable System Design Patterns has created a great list of scalability patterns along with very well done explanatory graphics. A summary of the patterns are: Load Balancer - a dispatcher determines which worker instance will handle a request based on different policies. Scatter and Gather - a dispatcher multicasts requests to all workers in a pool. Each worker will compute a local result and send it back to the dispatcher, who will consolidate them into a single response and then send back to the client. Result Cache - a dispatcher will first lookup if the request has been made before and try to find the previous result to return, in order to save the actual execution. Shared Space - all workers monitors information from the shared space and contributes partial knowledge back to the blackboard. The information is continuously enriched until a solution is reached. Pipe and Filter - all workers connected by pipes across which data flows. MapReduc
same-blog 2 0.89182401 307 high scalability-2008-04-21-Using Google AppEngine for a Little Micro-Scalability
Introduction: Over the years I've accumulated quite a rag tag collection of personal systems scattered wide across a galaxy of different servers. For the past month I've been on a quest to rationalize this conglomeration by moving everything to a managed service of one kind or another. The goal: lift a load of worry from my mind. I like to do my own stuff my self so I learn something and have control. Control always comes with headaches and it was time for a little aspirin. As part of the process GAE came in handy as a host for a few Twitter related scripts I couldn't manage to run anywhere else. I recoded my simple little scripts into Python/GAE and learned a lot in the process. In the move I exported HighScalability from a VPS and imported it into a shared hosting service. I could never quite configure Apache and MySQL well enough that they wouldn't spike memory periodically and crash the VPS. And since a memory crash did not automatically restarted it was unacceptable. I also wrote a scrip
3 0.86917025 1268 high scalability-2012-06-20-Ask HighScalability: How do I organize millions of images?
Introduction: Does anyone have any advice or suggestions on how to store millions of images? Currently images are stored in a MS SQL database which performance wise isn't ideal. We'd like to migrate the images over to a file system structure but I'd assume we don't just want to dump millions of images into a single directory. Besides having to contend with naming collisions, the windows filesystem might not perform optimally with that many files. I'm assuming one approach may be to assign each user a unique CSLID, create a folder based on the CSLID and then place one users files in that particular folder. Even so, this could result in hundreds of thousands of folders. Whats the best organizational scheme/heirachy for doing this?
4 0.84929663 1252 high scalability-2012-05-25-Stuff The Internet Says On Scalability For May 25, 2012
Introduction: It's HighScalability Time: 15 million 64 GB iPods every day : IBM on the Big Bang; Billions of msgs/day, 75Bn ops/day, 1.5M ops/sec peak. 250TB new data/mo and growing : Facebook HBase stats; 72 hours of video uploaded every minute : YouTube Quotable Quotes: @ammeep : Great discussion on data scalability and caching approaches at #yellowcamp "a cache is like your best mate you can never trust" @adrianco : Why do OpenStack talks I go to always focus on the story of how they did it rather than how it works and what it does for me? #Gluecon. Cliff Moon : Incuriosity killed the infrastructure. @seldo2 : "NoSQL is a land grab against the DBAs the prevent you from doing shit" - @monkchips #gluecon @mjasay : Netflix's @atseitlin: Build for fail, not for scale. Focus on service-level metrics, not individual servers, when monitoring system health @peakscale : I liked the part about "graph DBs are a niche data model... that only covers 95% of situation
5 0.84549117 1542 high scalability-2013-11-04-ESPN's Architecture at Scale - Operating at 100,000 Duh Nuh Nuhs Per Second
Introduction: ESPN went on airin 1978. In those 30+ years think of the wonders we've seen! When I think of ESPN I think of a world wide brand that is the very definition of prime time. And it shows in their stats. ESPN.com peaks at 100,000 requests per second. Their peak event is, not surprisingly, the World Cup. But would you be surprised to learn ESPN is powered by only a few hundred servers and a couple of dozen engineers? I was.And would you be surprised to learn ESPN is undergoing a fundamental transition from an Enterprise architecture to one capable of handling web scale loads driven by increasing mobile usage, personalization, and a service orientation? Again, thinking ESPN was just about watching sports on TV, I was surprised. ESPN is becoming much more than that. ESPN is becoming a sports platform. How does ESPN handle all of this complexity, responsibility, change, and load? Unlike most every other profile on HighScalability. The fascinating story of ESPN's architecture is told byManny Pe
6 0.84259152 277 high scalability-2008-03-16-Do you have any questions for the Elastra CEO?
7 0.82651979 1174 high scalability-2012-01-13-Stuff The Internet Says On Scalability For January 13, 2012
8 0.82366699 1516 high scalability-2013-09-13-Stuff The Internet Says On Scalability For September 13, 2013
9 0.82250166 716 high scalability-2009-10-06-Building a Unique Data Warehouse
10 0.82106572 1117 high scalability-2011-09-16-Stuff The Internet Says On Scalability For September 16, 2011
11 0.81942827 863 high scalability-2010-07-22-How can we spark the movement of research out of the Ivory Tower and into production?
12 0.81932288 1402 high scalability-2013-02-07-Ask HighScalability: Web asset server concept - 3rd party software available?
13 0.81902611 976 high scalability-2011-01-20-75% Chance of Scale - Leveraging the New Scaleogenic Environment for Growth
14 0.81868213 1362 high scalability-2012-11-26-BigData using Erlang, C and Lisp to Fight the Tsunami of Mobile Data
15 0.81472898 266 high scalability-2008-03-04-Manage Downtime Risk by Connecting Multiple Data Centers into a Secure Virtual LAN
16 0.81117922 1389 high scalability-2013-01-18-Stuff The Internet Says On Scalability For January 18, 2013
17 0.81090879 1382 high scalability-2013-01-07-Analyzing billions of credit card transactions and serving low-latency insights in the cloud
18 0.80922931 789 high scalability-2010-03-05-Strategy: Planning for a Power Outage Google Style
19 0.80836332 1561 high scalability-2013-12-09-Site Moves from PHP to Facebook's HipHop, Now Pages Load in .6 Seconds Instead of Five
20 0.80775219 517 high scalability-2009-02-21-Google AppEngine - A Second Look