high_scalability high_scalability-2012 high_scalability-2012-1359 knowledge-graph by maker-knowledge-mining
Source: html
Introduction: This is one of my favorite posts for a couple of reasons. I think it gives a lot of useful information in an interesting space. And Kyle Vogt was just a real pleasure to talk to. He was very helpful and forthcoming, which makes the whole experience better for everyone. The future is live. The future is real-time. The future is now. That's the hype anyway. And as it has a habit of doing, the hype is slowly becoming reality. We are seeing live searches, live tweets, live location, live reality augmentation, live crab (fresh and local), and live event publishing. One of the most challenging of all live technologies is that of live video broadcasting. Imagine a world in which everyone becomes a broadcaster and a consumer of video streams, all in real-time (< 250 msec latency), all so you can talk and interact directly without feeling like you are in the middle of a time shift war. The resources and the engineering needed to make this happened must be substantial. How do you do tha
sentIndex sentText sentNum sentScore
1 We are seeing live searches, live tweets, live location, live reality augmentation, live crab (fresh and local), and live event publishing. [sent-10, score-1.206]
2 One of the most challenging of all live technologies is that of live video broadcasting. [sent-11, score-0.832]
3 Imagine a world in which everyone becomes a broadcaster and a consumer of video streams, all in real-time (< 250 msec latency), all so you can talk and interact directly without feeling like you are in the middle of a time shift war. [sent-12, score-0.628]
4 Their 30 million unique monthly visitors even outshine YouTube in the video upload game, reportedly uploading nearly 30 hours per minute of video compared to YouTube's 23. [sent-19, score-0.86]
5 Justin talked about how live video was fundamentally different than YouTube's batch video approach, where all the video is stored on disk and replayed later on demand. [sent-22, score-1.491]
6 Live video can't be made by pushing video faster, it takes a completely differently architecture. [sent-23, score-0.86]
7 Since the YouTube Architecture article is the most popular article ever on this site, I thought people might also enjoy learning about live side of the video world. [sent-24, score-0.702]
8 tv makes all this live video magic happen, going way beyond the call, providing a tremendous number of juicy details. [sent-26, score-0.708]
9 Usher - custom business logic server for playing video streams. [sent-68, score-0.625]
10 The Live Video Architecture Why is live video difficult? [sent-81, score-0.631]
11 If you can't just do YouTube faster for live video, what makes live video such a challenge? [sent-85, score-0.832]
12 With live video if you exceed your network capacity even for a fraction of a second every single viewer will see buffering all at the same moment. [sent-90, score-0.832]
13 Every server can act as an edge server (where the video is streaming out of to a viewer) and an origin server (where the video is streaming into from a broadcaster). [sent-123, score-1.145]
14 The entire video stream stays in memory from the time it hits the origin server to when it's copied to other servers and when it's copied to viewers. [sent-131, score-0.882]
15 The video servers are fairly dumb, the overlay logic controlling the serving topology is managed by Usher. [sent-142, score-0.61]
16 They have a backbone network to get the video streams between datacenters. [sent-165, score-0.692]
17 While video streams are not streamed from disk, video is archived to disk. [sent-170, score-1.035]
18 The number of video servers may seem a little low for their traffic because with Usher they can run each video server to full capacity. [sent-185, score-1.138]
19 Building Usher as the core backbone of their video scalability on top of relatively dumb video servers is an excellent example of this strategy. [sent-299, score-0.933]
20 In live video this is often a big event and if you mess up a lot of people will spread the bad word about you. [sent-345, score-0.702]
wordName wordTfidf (topN-words)
[('video', 0.43), ('live', 0.201), ('usher', 0.198), ('streams', 0.175), ('peering', 0.171), ('transcoding', 0.133), ('kyle', 0.131), ('stream', 0.122), ('broadcaster', 0.121), ('viewers', 0.096), ('chat', 0.095), ('eyeball', 0.09), ('vogt', 0.09), ('youtube', 0.089), ('network', 0.087), ('crowds', 0.085), ('number', 0.077), ('everyone', 0.077), ('server', 0.076), ('load', 0.073), ('servers', 0.073), ('incoming', 0.071), ('people', 0.071), ('used', 0.068), ('custom', 0.067), ('gig', 0.065), ('copied', 0.062), ('capacity', 0.062), ('kan', 0.06), ('branch', 0.06), ('handle', 0.059), ('cost', 0.058), ('datacenter', 0.057), ('origin', 0.057), ('vp', 0.057), ('converse', 0.057), ('ca', 0.057), ('broadcast', 0.056), ('peer', 0.055), ('topology', 0.055), ('process', 0.055), ('feature', 0.054), ('interconnection', 0.054), ('served', 0.053), ('twice', 0.053), ('logic', 0.052), ('peak', 0.052), ('viewer', 0.052), ('traffic', 0.052), ('ip', 0.051)]
simIndex simValue blogId blogTitle
same-blog 1 0.99999851 1359 high scalability-2012-11-15-Gone Fishin': Justin.Tv's Live Video Broadcasting Architecture
Introduction: This is one of my favorite posts for a couple of reasons. I think it gives a lot of useful information in an interesting space. And Kyle Vogt was just a real pleasure to talk to. He was very helpful and forthcoming, which makes the whole experience better for everyone. The future is live. The future is real-time. The future is now. That's the hype anyway. And as it has a habit of doing, the hype is slowly becoming reality. We are seeing live searches, live tweets, live location, live reality augmentation, live crab (fresh and local), and live event publishing. One of the most challenging of all live technologies is that of live video broadcasting. Imagine a world in which everyone becomes a broadcaster and a consumer of video streams, all in real-time (< 250 msec latency), all so you can talk and interact directly without feeling like you are in the middle of a time shift war. The resources and the engineering needed to make this happened must be substantial. How do you do tha
2 0.99809456 796 high scalability-2010-03-16-Justin.tv's Live Video Broadcasting Architecture
Introduction: The future is live. The future is real-time. The future is now. That's the hype anyway. And as it has a habit of doing, the hype is slowly becoming reality. We are seeing live searches, live tweets, live location, live reality augmentation, live crab (fresh and local), and live event publishing. One of the most challenging of all live technologies is that of live video broadcasting. Imagine a world in which everyone becomes a broadcaster and a consumer of video streams, all in real-time (< 250 msec latency), all so you can talk and interact directly without feeling like you are in the middle of a time shift war. The resources and the engineering needed to make this happened must be substantial. How do you do that? To find out I talked to Kyle Vogt, Justin.tv Founder and VP of Engineering. Justin.tv certainly has the numbers. Their 30 million unique monthly visitors even outshine YouTube in the video upload game, reportedly uploading nearly 30 hours per minute of video compared to Y
3 0.32789889 274 high scalability-2008-03-12-YouTube Architecture
Introduction: Update 2: YouTube Reaches One Billion Views Per Day . That’s at least 11,574 views per second, 694,444 views per minute, and 41,666,667 views per hour. Update: YouTube: The Platform . YouTube adds a new rich set of APIs in order to become your video platform leader--all for free. Upload, edit, watch, search, and comment on video from your own site without visiting YouTube. Compose your site internally from APIs because you'll need to expose them later anyway. YouTube grew incredibly fast, to over 100 million video views per day, with only a handful of people responsible for scaling the site. How did they manage to deliver all that video to all those users? And how have they evolved since being acquired by Google? Information Sources Google Video Platform Apache Python Linux (SuSe) MySQL psyco, a dynamic python->C compiler lighttpd for video instead of Apache What's Inside? The Stats Supports the delivery of over 100 million vide
4 0.30849648 991 high scalability-2011-02-16-Paper: An Experimental Investigation of the Akamai Adaptive Video Streaming
Introduction: Video is hot on the Internet and people are really interested in knowing how to make it work. Dan Rayburn has a post pointing to a fascinating paper: An Experimental Investigation of the Akamai Adaptive Video Streaming , which talks in some detail about the protocols big players like YouTube, Skype and Akamai use to serve video over on an inherently video unfriendly medium like the Internet. For Akamai they found: Each video is encoded in five versions at different bit rates and stored in separate files. The client sends commands to the server with an average inter departure time of about 2 s, i.e. the control algorithm is executed on average each 2 seconds. Akamai uses only the video level to adapt the video source to the available bandwidth, whereas the frame rate of the video is kept constant. When a sudden drop in the available bandwidth occurs, short interruptions of the video playback can occur due to the a large actuation delay. For a sudden increase of the avai
5 0.28217837 1037 high scalability-2011-05-10-Viddler Architecture - 7 Million Embeds a Day and 1500 Req-Sec Peak
Introduction: Viddler is in the high quality Video as a Service business for a customer who wants to pay a fixed cost, be done with it, and just have it work. Similar to Blip and Ooyala, more focussed on business than YouTube. They serve thousands of business customers, including high traffic websites like FailBlog, Engadget, and Gawker. Viddler is a good case to learn from because they are a small company trying to provide a challenging service in a crowded field. We are catching them just as they transitioning from a startup that began in one direction, as a YouTube competitor, and pivoted into a slightly larger company focussed on paying business customers. Transition is the key word for Viddler: transitioning from a free YouTube clone to a high quality paid service. Transitioning from a few colo sites that didn't work well to a new higher quality datacenter. Transitioning from an architecture that was typical of a startup to one that features redundancy, high availability, and automation. Tr
6 0.26336929 1215 high scalability-2012-03-26-7 Years of YouTube Scalability Lessons in 30 Minutes
7 0.23470676 1201 high scalability-2012-02-29-Strategy: Put Mobile Video Into Cold Storage After 30 Days
8 0.20649372 576 high scalability-2009-04-21-What CDN would you recommend?
11 0.18973885 1501 high scalability-2013-08-13-In Memoriam: Lavabit Architecture - Creating a Scalable Email Service
12 0.18942486 313 high scalability-2008-05-02-Friends for Sale Architecture - A 300 Million Page View-Month Facebook RoR App
13 0.18589051 70 high scalability-2007-08-22-How many machines do you need to run your site?
14 0.18412684 1240 high scalability-2012-05-07-Startups are Creating a New System of the World for IT
15 0.18101285 661 high scalability-2009-07-25-Latency is Everywhere and it Costs You Sales - How to Crush it
16 0.17497595 239 high scalability-2008-02-04-Streaming Video on Amazon EC2?
17 0.17023931 538 high scalability-2009-03-16-Are Cloud Based Memory Architectures the Next Big Thing?
18 0.16862214 954 high scalability-2010-12-06-What the heck are you actually using NoSQL for?
19 0.16828035 920 high scalability-2010-10-15-Troubles with Sharding - What can we learn from the Foursquare Incident?
20 0.16811028 1008 high scalability-2011-03-22-Facebook's New Realtime Analytics System: HBase to Process 20 Billion Events Per Day
topicId topicWeight
[(0, 0.372), (1, 0.153), (2, -0.052), (3, -0.126), (4, -0.039), (5, -0.14), (6, -0.018), (7, 0.091), (8, -0.008), (9, 0.003), (10, -0.009), (11, -0.048), (12, -0.094), (13, -0.007), (14, 0.101), (15, 0.185), (16, 0.054), (17, 0.128), (18, -0.093), (19, -0.126), (20, -0.062), (21, 0.031), (22, 0.12), (23, 0.002), (24, 0.027), (25, 0.074), (26, -0.016), (27, 0.064), (28, 0.084), (29, 0.117), (30, 0.03), (31, -0.044), (32, -0.01), (33, -0.049), (34, 0.01), (35, -0.034), (36, -0.114), (37, -0.185), (38, 0.014), (39, 0.051), (40, 0.037), (41, 0.033), (42, 0.031), (43, -0.04), (44, 0.095), (45, -0.037), (46, 0.031), (47, 0.029), (48, 0.168), (49, 0.058)]
simIndex simValue blogId blogTitle
1 0.96882707 796 high scalability-2010-03-16-Justin.tv's Live Video Broadcasting Architecture
Introduction: The future is live. The future is real-time. The future is now. That's the hype anyway. And as it has a habit of doing, the hype is slowly becoming reality. We are seeing live searches, live tweets, live location, live reality augmentation, live crab (fresh and local), and live event publishing. One of the most challenging of all live technologies is that of live video broadcasting. Imagine a world in which everyone becomes a broadcaster and a consumer of video streams, all in real-time (< 250 msec latency), all so you can talk and interact directly without feeling like you are in the middle of a time shift war. The resources and the engineering needed to make this happened must be substantial. How do you do that? To find out I talked to Kyle Vogt, Justin.tv Founder and VP of Engineering. Justin.tv certainly has the numbers. Their 30 million unique monthly visitors even outshine YouTube in the video upload game, reportedly uploading nearly 30 hours per minute of video compared to Y
same-blog 2 0.96809721 1359 high scalability-2012-11-15-Gone Fishin': Justin.Tv's Live Video Broadcasting Architecture
Introduction: This is one of my favorite posts for a couple of reasons. I think it gives a lot of useful information in an interesting space. And Kyle Vogt was just a real pleasure to talk to. He was very helpful and forthcoming, which makes the whole experience better for everyone. The future is live. The future is real-time. The future is now. That's the hype anyway. And as it has a habit of doing, the hype is slowly becoming reality. We are seeing live searches, live tweets, live location, live reality augmentation, live crab (fresh and local), and live event publishing. One of the most challenging of all live technologies is that of live video broadcasting. Imagine a world in which everyone becomes a broadcaster and a consumer of video streams, all in real-time (< 250 msec latency), all so you can talk and interact directly without feeling like you are in the middle of a time shift war. The resources and the engineering needed to make this happened must be substantial. How do you do tha
3 0.87140065 1201 high scalability-2012-02-29-Strategy: Put Mobile Video Into Cold Storage After 30 Days
Introduction: Limelight says 95% of Mobile Video Views Take Place in First 90 Days and 88.8 percent of views take place in the first 30 days. Since a lot of people are working with video, which is expensive to store and serve, the implication: there's little need to keep your video close to the user or on a CDN after 30 days.
4 0.8601442 274 high scalability-2008-03-12-YouTube Architecture
Introduction: Update 2: YouTube Reaches One Billion Views Per Day . That’s at least 11,574 views per second, 694,444 views per minute, and 41,666,667 views per hour. Update: YouTube: The Platform . YouTube adds a new rich set of APIs in order to become your video platform leader--all for free. Upload, edit, watch, search, and comment on video from your own site without visiting YouTube. Compose your site internally from APIs because you'll need to expose them later anyway. YouTube grew incredibly fast, to over 100 million video views per day, with only a handful of people responsible for scaling the site. How did they manage to deliver all that video to all those users? And how have they evolved since being acquired by Google? Information Sources Google Video Platform Apache Python Linux (SuSe) MySQL psyco, a dynamic python->C compiler lighttpd for video instead of Apache What's Inside? The Stats Supports the delivery of over 100 million vide
5 0.85382789 1037 high scalability-2011-05-10-Viddler Architecture - 7 Million Embeds a Day and 1500 Req-Sec Peak
Introduction: Viddler is in the high quality Video as a Service business for a customer who wants to pay a fixed cost, be done with it, and just have it work. Similar to Blip and Ooyala, more focussed on business than YouTube. They serve thousands of business customers, including high traffic websites like FailBlog, Engadget, and Gawker. Viddler is a good case to learn from because they are a small company trying to provide a challenging service in a crowded field. We are catching them just as they transitioning from a startup that began in one direction, as a YouTube competitor, and pivoted into a slightly larger company focussed on paying business customers. Transition is the key word for Viddler: transitioning from a free YouTube clone to a high quality paid service. Transitioning from a few colo sites that didn't work well to a new higher quality datacenter. Transitioning from an architecture that was typical of a startup to one that features redundancy, high availability, and automation. Tr
6 0.84986842 991 high scalability-2011-02-16-Paper: An Experimental Investigation of the Akamai Adaptive Video Streaming
7 0.82178921 294 high scalability-2008-04-01-How to update video views count effectively?
8 0.78614205 1215 high scalability-2012-03-26-7 Years of YouTube Scalability Lessons in 30 Minutes
9 0.74931055 576 high scalability-2009-04-21-What CDN would you recommend?
10 0.73630095 198 high scalability-2008-01-01-HOW CDN works
11 0.71453184 447 high scalability-2008-11-19-High Definition Video Delivery on the Web?
12 0.69599402 267 high scalability-2008-03-05-Oprah is the Real Social Network
13 0.68058872 1289 high scalability-2012-07-23-State of the CDN: More Traffic, Stable Prices, More Products, Profits - Not So Much
14 0.67807239 60 high scalability-2007-08-07-Can you profit from the coming Content Delivery Network wars?
15 0.67268199 270 high scalability-2008-03-08-DNS-Record TTL on worst case scenarios
16 0.66401285 239 high scalability-2008-02-04-Streaming Video on Amazon EC2?
17 0.66249049 773 high scalability-2010-02-06-GEO-aware traffic load balancing and caching at CNBC.com
18 0.65099198 244 high scalability-2008-02-11-Yahoo Live's Scaling Problems Prove: Release Early and Often - Just Don't Screw Up
19 0.64781606 39 high scalability-2007-07-30-Product: Akamai
20 0.64258933 123 high scalability-2007-10-15-Olympic Site Architecture
topicId topicWeight
[(1, 0.142), (2, 0.218), (10, 0.03), (30, 0.032), (40, 0.015), (47, 0.028), (49, 0.127), (61, 0.086), (73, 0.013), (77, 0.021), (79, 0.136), (85, 0.035), (94, 0.042)]
simIndex simValue blogId blogTitle
Introduction: Michael Stonebraker sure knows how to stir up a storm. Unlike for others, that doesn't make him a troll in my mind, he's way too accomplished in the field to be that, but he does have a bit of Barnum & Bailey in him, which serves to get the discussion flowing, and that's a good thing. A lot of previously hidden wisdom and passion unlocks, which we'll try to capture here. This disturbance in the force is over OldSQL vs NoSQL vs NewSQL . Warning, these are not crisp categories, there's leakage all over the place, watch your step: OldSQL (Oracle, MySQL, etc) refers to what some want to term as legacy relational database like MySQL, that don't scale out horizontally with aplomb. NoSQL (CouchDB, Redis, Cassandra, HBase, MongoDB, Riak, Neo4j, etc) refers to, well, a collection of technologies that aren't OldSQL, these often are designed to scale out horizontally, aren't on ACID, and use schemaless non-relational datamodels. NewSQL (Xeround, Clustrix, NimbusDB, GenieDB, Sc
2 0.96404588 737 high scalability-2009-11-05-A Yes for a NoSQL Taxonomy
Introduction: NorthScale's Steven Yen in his highly entertaining NoSQL is a Horseless Carriage presentation has come up with a NoSQL taxonomy that thankfully focuses a little more on what NoSQL is, than what it isn't : key‐value‐cache memcached, repcached, coherence, infinispan, eXtreme scale, jboss cache, velocity, terracoqa key‐value‐store keyspace, flare, schema‐free, RAMCloud eventually‐consistent key‐value‐store dynamo, voldemort, Dynomite, SubRecord, Mo8onDb, Dovetaildb ordered‐key‐value‐store tokyo tyrant, lightcloud, NMDB, luxio, memcachedb, actord data‐structures server redis tuple‐store gigaspaces, coord, apache river object database ZopeDB, db4o, Shoal document store CouchDB, Mongo, Jackrabbit, XML Databases, ThruDB, CloudKit, Perservere, Riak Basho, Scalaris wide columnar store BigTable, Hbase, Cassandra, Hypertable, KAI, OpenNeptune, Qbase, KDI "Who will win?"
3 0.96066386 400 high scalability-2008-10-01-The Pattern Bible for Distributed Computing
Introduction: Software design patterns are an emerging tool for guiding and documenting system design. Patterns usually describe software abstractions used by advanced designers and programmers in their software. Patterns can provide guidance for designing highly scalable distributed systems. Let's see how! Patterns are in essence solutions to problems. Most of them are expressed in a format called Alexandrian form which draws on constructs used by Christopher Alexander. There are variants but most look like this: The pattern name The problem the pattern is trying to solve Context Solution Examples Design rationale: This tells where the pattern came from, why it works, and why experts use it Patterns rarely stand alone. Each pattern works on a context, and transforms the system in that context to produce a new system in a new context. New problems arise in the new system and context, and the next ‘‘layer’’ of patterns can be applied. A pattern language is a structured col
4 0.95231169 823 high scalability-2010-05-05-How will memristors change everything?
Introduction: A non-random sample of my tech friends shows that not many have heard of memristors (though I do suspect vote tampering). I'd read a little about memristors in 2008 when the initial hubbub about the existence of memristors was raised. I, however, immediately filed them into that comforting conceptual bucket of potentially revolutionary technologies I didn't have to worry about because like most wondertech, nothing would ever come of it. Wrong. After watching Finding the Missing Memristor by R. Stanley Williams I've had to change my mind. Memristors have gone from "maybe never" to holy cow this could happen soon and it could change everything. Let's assume for the sake of dreaming memristors do prove out. How will we design systems when we have access to a new material that is two orders of magnitude more efficient from a power perspective than traditional transistor technologies, contains multiple petabits (1 petabit = 128TB) of persistent storage, and can be reconfigured t
same-blog 5 0.95201784 1359 high scalability-2012-11-15-Gone Fishin': Justin.Tv's Live Video Broadcasting Architecture
Introduction: This is one of my favorite posts for a couple of reasons. I think it gives a lot of useful information in an interesting space. And Kyle Vogt was just a real pleasure to talk to. He was very helpful and forthcoming, which makes the whole experience better for everyone. The future is live. The future is real-time. The future is now. That's the hype anyway. And as it has a habit of doing, the hype is slowly becoming reality. We are seeing live searches, live tweets, live location, live reality augmentation, live crab (fresh and local), and live event publishing. One of the most challenging of all live technologies is that of live video broadcasting. Imagine a world in which everyone becomes a broadcaster and a consumer of video streams, all in real-time (< 250 msec latency), all so you can talk and interact directly without feeling like you are in the middle of a time shift war. The resources and the engineering needed to make this happened must be substantial. How do you do tha
6 0.95118493 796 high scalability-2010-03-16-Justin.tv's Live Video Broadcasting Architecture
7 0.94379318 1311 high scalability-2012-08-24-Stuff The Internet Says On Scalability For August 24, 2012
8 0.94227576 735 high scalability-2009-11-01-Squeeze more performance from Parallelism
9 0.9365896 183 high scalability-2007-12-12-Report from OpenSocial Meetup at Google
10 0.92496675 776 high scalability-2010-02-12-Hot Scalability Links for February 12, 2010
11 0.9234758 399 high scalability-2008-10-01-Joyent - Cloud Computing Built on Accelerators
12 0.92248881 119 high scalability-2007-10-10-WAN Accelerate Your Way to Lightening Fast Transfers Between Data Centers
13 0.92230839 927 high scalability-2010-10-26-Marrying memcached and NoSQL
14 0.92024231 1431 high scalability-2013-03-29-Stuff The Internet Says On Scalability For March 29, 2013
15 0.91930997 1519 high scalability-2013-09-18-If You're Programming a Cell Phone Like a Server You're Doing it Wrong
16 0.91866493 517 high scalability-2009-02-21-Google AppEngine - A Second Look
17 0.91796046 301 high scalability-2008-04-08-Google AppEngine - A First Look
18 0.91772085 1502 high scalability-2013-08-16-Stuff The Internet Says On Scalability For August 16, 2013
19 0.91652912 1051 high scalability-2011-06-01-Why is your network so slow? Your switch should tell you.
20 0.91632283 849 high scalability-2010-06-28-VoltDB Decapitates Six SQL Urban Myths and Delivers Internet Scale OLTP in the Process