high_scalability high_scalability-2010 high_scalability-2010-796 knowledge-graph by maker-knowledge-mining
Source: html
Introduction: The future is live. The future is real-time. The future is now. That's the hype anyway. And as it has a habit of doing, the hype is slowly becoming reality. We are seeing live searches, live tweets, live location, live reality augmentation, live crab (fresh and local), and live event publishing. One of the most challenging of all live technologies is that of live video broadcasting. Imagine a world in which everyone becomes a broadcaster and a consumer of video streams, all in real-time (< 250 msec latency), all so you can talk and interact directly without feeling like you are in the middle of a time shift war. The resources and the engineering needed to make this happened must be substantial. How do you do that? To find out I talked to Kyle Vogt, Justin.tv Founder and VP of Engineering. Justin.tv certainly has the numbers. Their 30 million unique monthly visitors even outshine YouTube in the video upload game, reportedly uploading nearly 30 hours per minute of video compared to Y
sentIndex sentText sentNum sentScore
1 We are seeing live searches, live tweets, live location, live reality augmentation, live crab (fresh and local), and live event publishing. [sent-6, score-1.218]
2 One of the most challenging of all live technologies is that of live video broadcasting. [sent-7, score-0.839]
3 Imagine a world in which everyone becomes a broadcaster and a consumer of video streams, all in real-time (< 250 msec latency), all so you can talk and interact directly without feeling like you are in the middle of a time shift war. [sent-8, score-0.632]
4 Their 30 million unique monthly visitors even outshine YouTube in the video upload game, reportedly uploading nearly 30 hours per minute of video compared to YouTube's 23. [sent-15, score-0.866]
5 Justin talked about how live video was fundamentally different than YouTube's batch video approach, where all the video is stored on disk and replayed later on demand. [sent-18, score-1.502]
6 Live video can't be made by pushing video faster, it takes a completely differently architecture. [sent-19, score-0.866]
7 Since the YouTube Architecture article is the most popular article ever on this site, I thought people might also enjoy learning about live side of the video world. [sent-20, score-0.707]
8 tv makes all this live video magic happen, going way beyond the call, providing a tremendous number of juicy details. [sent-22, score-0.714]
9 Usher - custom business logic server for playing video streams. [sent-64, score-0.63]
10 The Live Video Architecture Why is live video difficult? [sent-77, score-0.636]
11 If you can't just do YouTube faster for live video, what makes live video such a challenge? [sent-81, score-0.839]
12 With live video if you exceed your network capacity even for a fraction of a second every single viewer will see buffering all at the same moment. [sent-86, score-0.838]
13 Every server can act as an edge server (where the video is streaming out of to a viewer) and an origin server (where the video is streaming into from a broadcaster). [sent-119, score-1.154]
14 The entire video stream stays in memory from the time it hits the origin server to when it's copied to other servers and when it's copied to viewers. [sent-127, score-0.89]
15 The video servers are fairly dumb, the overlay logic controlling the serving topology is managed by Usher. [sent-138, score-0.616]
16 They have a backbone network to get the video streams between datacenters. [sent-161, score-0.697]
17 While video streams are not streamed from disk, video is archived to disk. [sent-166, score-1.042]
18 The number of video servers may seem a little low for their traffic because with Usher they can run each video server to full capacity. [sent-181, score-1.147]
19 Building Usher as the core backbone of their video scalability on top of relatively dumb video servers is an excellent example of this strategy. [sent-295, score-0.94]
20 In live video this is often a big event and if you mess up a lot of people will spread the bad word about you. [sent-341, score-0.707]
wordName wordTfidf (topN-words)
[('video', 0.433), ('live', 0.203), ('usher', 0.2), ('streams', 0.176), ('peering', 0.173), ('transcoding', 0.134), ('stream', 0.123), ('broadcaster', 0.121), ('kyle', 0.11), ('viewers', 0.096), ('chat', 0.095), ('eyeball', 0.091), ('youtube', 0.09), ('network', 0.088), ('crowds', 0.086), ('number', 0.078), ('everyone', 0.078), ('server', 0.077), ('load', 0.074), ('servers', 0.074), ('incoming', 0.071), ('people', 0.071), ('used', 0.069), ('custom', 0.067), ('gig', 0.066), ('copied', 0.063), ('capacity', 0.062), ('kan', 0.061), ('vogt', 0.061), ('branch', 0.061), ('handle', 0.059), ('cost', 0.059), ('datacenter', 0.057), ('origin', 0.057), ('vp', 0.057), ('converse', 0.057), ('ca', 0.057), ('broadcast', 0.057), ('peer', 0.056), ('topology', 0.056), ('process', 0.055), ('feature', 0.055), ('interconnection', 0.054), ('served', 0.053), ('twice', 0.053), ('logic', 0.053), ('peak', 0.053), ('viewer', 0.052), ('traffic', 0.052), ('ip', 0.052)]
simIndex simValue blogId blogTitle
same-blog 1 0.99999893 796 high scalability-2010-03-16-Justin.tv's Live Video Broadcasting Architecture
Introduction: The future is live. The future is real-time. The future is now. That's the hype anyway. And as it has a habit of doing, the hype is slowly becoming reality. We are seeing live searches, live tweets, live location, live reality augmentation, live crab (fresh and local), and live event publishing. One of the most challenging of all live technologies is that of live video broadcasting. Imagine a world in which everyone becomes a broadcaster and a consumer of video streams, all in real-time (< 250 msec latency), all so you can talk and interact directly without feeling like you are in the middle of a time shift war. The resources and the engineering needed to make this happened must be substantial. How do you do that? To find out I talked to Kyle Vogt, Justin.tv Founder and VP of Engineering. Justin.tv certainly has the numbers. Their 30 million unique monthly visitors even outshine YouTube in the video upload game, reportedly uploading nearly 30 hours per minute of video compared to Y
2 0.99809456 1359 high scalability-2012-11-15-Gone Fishin': Justin.Tv's Live Video Broadcasting Architecture
Introduction: This is one of my favorite posts for a couple of reasons. I think it gives a lot of useful information in an interesting space. And Kyle Vogt was just a real pleasure to talk to. He was very helpful and forthcoming, which makes the whole experience better for everyone. The future is live. The future is real-time. The future is now. That's the hype anyway. And as it has a habit of doing, the hype is slowly becoming reality. We are seeing live searches, live tweets, live location, live reality augmentation, live crab (fresh and local), and live event publishing. One of the most challenging of all live technologies is that of live video broadcasting. Imagine a world in which everyone becomes a broadcaster and a consumer of video streams, all in real-time (< 250 msec latency), all so you can talk and interact directly without feeling like you are in the middle of a time shift war. The resources and the engineering needed to make this happened must be substantial. How do you do tha
3 0.32939675 274 high scalability-2008-03-12-YouTube Architecture
Introduction: Update 2: YouTube Reaches One Billion Views Per Day . That’s at least 11,574 views per second, 694,444 views per minute, and 41,666,667 views per hour. Update: YouTube: The Platform . YouTube adds a new rich set of APIs in order to become your video platform leader--all for free. Upload, edit, watch, search, and comment on video from your own site without visiting YouTube. Compose your site internally from APIs because you'll need to expose them later anyway. YouTube grew incredibly fast, to over 100 million video views per day, with only a handful of people responsible for scaling the site. How did they manage to deliver all that video to all those users? And how have they evolved since being acquired by Google? Information Sources Google Video Platform Apache Python Linux (SuSe) MySQL psyco, a dynamic python->C compiler lighttpd for video instead of Apache What's Inside? The Stats Supports the delivery of over 100 million vide
4 0.3106814 991 high scalability-2011-02-16-Paper: An Experimental Investigation of the Akamai Adaptive Video Streaming
Introduction: Video is hot on the Internet and people are really interested in knowing how to make it work. Dan Rayburn has a post pointing to a fascinating paper: An Experimental Investigation of the Akamai Adaptive Video Streaming , which talks in some detail about the protocols big players like YouTube, Skype and Akamai use to serve video over on an inherently video unfriendly medium like the Internet. For Akamai they found: Each video is encoded in five versions at different bit rates and stored in separate files. The client sends commands to the server with an average inter departure time of about 2 s, i.e. the control algorithm is executed on average each 2 seconds. Akamai uses only the video level to adapt the video source to the available bandwidth, whereas the frame rate of the video is kept constant. When a sudden drop in the available bandwidth occurs, short interruptions of the video playback can occur due to the a large actuation delay. For a sudden increase of the avai
5 0.28334895 1037 high scalability-2011-05-10-Viddler Architecture - 7 Million Embeds a Day and 1500 Req-Sec Peak
Introduction: Viddler is in the high quality Video as a Service business for a customer who wants to pay a fixed cost, be done with it, and just have it work. Similar to Blip and Ooyala, more focussed on business than YouTube. They serve thousands of business customers, including high traffic websites like FailBlog, Engadget, and Gawker. Viddler is a good case to learn from because they are a small company trying to provide a challenging service in a crowded field. We are catching them just as they transitioning from a startup that began in one direction, as a YouTube competitor, and pivoted into a slightly larger company focussed on paying business customers. Transition is the key word for Viddler: transitioning from a free YouTube clone to a high quality paid service. Transitioning from a few colo sites that didn't work well to a new higher quality datacenter. Transitioning from an architecture that was typical of a startup to one that features redundancy, high availability, and automation. Tr
6 0.26311478 1215 high scalability-2012-03-26-7 Years of YouTube Scalability Lessons in 30 Minutes
7 0.23610234 1201 high scalability-2012-02-29-Strategy: Put Mobile Video Into Cold Storage After 30 Days
9 0.20649789 576 high scalability-2009-04-21-What CDN would you recommend?
10 0.20647633 750 high scalability-2009-12-16-Building Super Scalable Systems: Blade Runner Meets Autonomic Computing in the Ambient Cloud
11 0.19024107 1501 high scalability-2013-08-13-In Memoriam: Lavabit Architecture - Creating a Scalable Email Service
12 0.18937084 313 high scalability-2008-05-02-Friends for Sale Architecture - A 300 Million Page View-Month Facebook RoR App
13 0.18608478 70 high scalability-2007-08-22-How many machines do you need to run your site?
14 0.18327808 1240 high scalability-2012-05-07-Startups are Creating a New System of the World for IT
15 0.18087372 661 high scalability-2009-07-25-Latency is Everywhere and it Costs You Sales - How to Crush it
16 0.17578557 239 high scalability-2008-02-04-Streaming Video on Amazon EC2?
17 0.17041612 538 high scalability-2009-03-16-Are Cloud Based Memory Architectures the Next Big Thing?
18 0.16830303 1008 high scalability-2011-03-22-Facebook's New Realtime Analytics System: HBase to Process 20 Billion Events Per Day
19 0.16822396 920 high scalability-2010-10-15-Troubles with Sharding - What can we learn from the Foursquare Incident?
20 0.16809857 954 high scalability-2010-12-06-What the heck are you actually using NoSQL for?
topicId topicWeight
[(0, 0.371), (1, 0.152), (2, -0.053), (3, -0.127), (4, -0.04), (5, -0.14), (6, -0.017), (7, 0.09), (8, -0.009), (9, 0.003), (10, -0.008), (11, -0.049), (12, -0.094), (13, -0.007), (14, 0.101), (15, 0.186), (16, 0.053), (17, 0.13), (18, -0.093), (19, -0.127), (20, -0.063), (21, 0.032), (22, 0.122), (23, 0.002), (24, 0.028), (25, 0.074), (26, -0.016), (27, 0.063), (28, 0.085), (29, 0.117), (30, 0.03), (31, -0.044), (32, -0.011), (33, -0.051), (34, 0.01), (35, -0.036), (36, -0.115), (37, -0.186), (38, 0.014), (39, 0.052), (40, 0.037), (41, 0.033), (42, 0.031), (43, -0.04), (44, 0.095), (45, -0.039), (46, 0.031), (47, 0.029), (48, 0.168), (49, 0.058)]
simIndex simValue blogId blogTitle
same-blog 1 0.96825933 796 high scalability-2010-03-16-Justin.tv's Live Video Broadcasting Architecture
Introduction: The future is live. The future is real-time. The future is now. That's the hype anyway. And as it has a habit of doing, the hype is slowly becoming reality. We are seeing live searches, live tweets, live location, live reality augmentation, live crab (fresh and local), and live event publishing. One of the most challenging of all live technologies is that of live video broadcasting. Imagine a world in which everyone becomes a broadcaster and a consumer of video streams, all in real-time (< 250 msec latency), all so you can talk and interact directly without feeling like you are in the middle of a time shift war. The resources and the engineering needed to make this happened must be substantial. How do you do that? To find out I talked to Kyle Vogt, Justin.tv Founder and VP of Engineering. Justin.tv certainly has the numbers. Their 30 million unique monthly visitors even outshine YouTube in the video upload game, reportedly uploading nearly 30 hours per minute of video compared to Y
2 0.96745318 1359 high scalability-2012-11-15-Gone Fishin': Justin.Tv's Live Video Broadcasting Architecture
Introduction: This is one of my favorite posts for a couple of reasons. I think it gives a lot of useful information in an interesting space. And Kyle Vogt was just a real pleasure to talk to. He was very helpful and forthcoming, which makes the whole experience better for everyone. The future is live. The future is real-time. The future is now. That's the hype anyway. And as it has a habit of doing, the hype is slowly becoming reality. We are seeing live searches, live tweets, live location, live reality augmentation, live crab (fresh and local), and live event publishing. One of the most challenging of all live technologies is that of live video broadcasting. Imagine a world in which everyone becomes a broadcaster and a consumer of video streams, all in real-time (< 250 msec latency), all so you can talk and interact directly without feeling like you are in the middle of a time shift war. The resources and the engineering needed to make this happened must be substantial. How do you do tha
3 0.87121469 1201 high scalability-2012-02-29-Strategy: Put Mobile Video Into Cold Storage After 30 Days
Introduction: Limelight says 95% of Mobile Video Views Take Place in First 90 Days and 88.8 percent of views take place in the first 30 days. Since a lot of people are working with video, which is expensive to store and serve, the implication: there's little need to keep your video close to the user or on a CDN after 30 days.
4 0.85950094 274 high scalability-2008-03-12-YouTube Architecture
Introduction: Update 2: YouTube Reaches One Billion Views Per Day . That’s at least 11,574 views per second, 694,444 views per minute, and 41,666,667 views per hour. Update: YouTube: The Platform . YouTube adds a new rich set of APIs in order to become your video platform leader--all for free. Upload, edit, watch, search, and comment on video from your own site without visiting YouTube. Compose your site internally from APIs because you'll need to expose them later anyway. YouTube grew incredibly fast, to over 100 million video views per day, with only a handful of people responsible for scaling the site. How did they manage to deliver all that video to all those users? And how have they evolved since being acquired by Google? Information Sources Google Video Platform Apache Python Linux (SuSe) MySQL psyco, a dynamic python->C compiler lighttpd for video instead of Apache What's Inside? The Stats Supports the delivery of over 100 million vide
5 0.85277784 1037 high scalability-2011-05-10-Viddler Architecture - 7 Million Embeds a Day and 1500 Req-Sec Peak
Introduction: Viddler is in the high quality Video as a Service business for a customer who wants to pay a fixed cost, be done with it, and just have it work. Similar to Blip and Ooyala, more focussed on business than YouTube. They serve thousands of business customers, including high traffic websites like FailBlog, Engadget, and Gawker. Viddler is a good case to learn from because they are a small company trying to provide a challenging service in a crowded field. We are catching them just as they transitioning from a startup that began in one direction, as a YouTube competitor, and pivoted into a slightly larger company focussed on paying business customers. Transition is the key word for Viddler: transitioning from a free YouTube clone to a high quality paid service. Transitioning from a few colo sites that didn't work well to a new higher quality datacenter. Transitioning from an architecture that was typical of a startup to one that features redundancy, high availability, and automation. Tr
6 0.85107005 991 high scalability-2011-02-16-Paper: An Experimental Investigation of the Akamai Adaptive Video Streaming
7 0.8205654 294 high scalability-2008-04-01-How to update video views count effectively?
8 0.78290141 1215 high scalability-2012-03-26-7 Years of YouTube Scalability Lessons in 30 Minutes
9 0.74932444 576 high scalability-2009-04-21-What CDN would you recommend?
10 0.73744327 198 high scalability-2008-01-01-HOW CDN works
11 0.71407032 447 high scalability-2008-11-19-High Definition Video Delivery on the Web?
12 0.69373447 267 high scalability-2008-03-05-Oprah is the Real Social Network
13 0.68104059 1289 high scalability-2012-07-23-State of the CDN: More Traffic, Stable Prices, More Products, Profits - Not So Much
14 0.67781425 60 high scalability-2007-08-07-Can you profit from the coming Content Delivery Network wars?
15 0.67229337 270 high scalability-2008-03-08-DNS-Record TTL on worst case scenarios
16 0.66279328 239 high scalability-2008-02-04-Streaming Video on Amazon EC2?
17 0.66140765 773 high scalability-2010-02-06-GEO-aware traffic load balancing and caching at CNBC.com
18 0.65019768 39 high scalability-2007-07-30-Product: Akamai
19 0.64790899 244 high scalability-2008-02-11-Yahoo Live's Scaling Problems Prove: Release Early and Often - Just Don't Screw Up
20 0.63947624 1493 high scalability-2013-07-17-Steve Ballmer Says Microsoft has Over 1 Million Servers - What Does that Really Mean?
topicId topicWeight
[(1, 0.143), (2, 0.218), (10, 0.03), (30, 0.032), (40, 0.014), (47, 0.028), (49, 0.126), (61, 0.086), (73, 0.012), (77, 0.021), (79, 0.137), (85, 0.033), (94, 0.042)]
simIndex simValue blogId blogTitle
Introduction: Michael Stonebraker sure knows how to stir up a storm. Unlike for others, that doesn't make him a troll in my mind, he's way too accomplished in the field to be that, but he does have a bit of Barnum & Bailey in him, which serves to get the discussion flowing, and that's a good thing. A lot of previously hidden wisdom and passion unlocks, which we'll try to capture here. This disturbance in the force is over OldSQL vs NoSQL vs NewSQL . Warning, these are not crisp categories, there's leakage all over the place, watch your step: OldSQL (Oracle, MySQL, etc) refers to what some want to term as legacy relational database like MySQL, that don't scale out horizontally with aplomb. NoSQL (CouchDB, Redis, Cassandra, HBase, MongoDB, Riak, Neo4j, etc) refers to, well, a collection of technologies that aren't OldSQL, these often are designed to scale out horizontally, aren't on ACID, and use schemaless non-relational datamodels. NewSQL (Xeround, Clustrix, NimbusDB, GenieDB, Sc
2 0.96400625 737 high scalability-2009-11-05-A Yes for a NoSQL Taxonomy
Introduction: NorthScale's Steven Yen in his highly entertaining NoSQL is a Horseless Carriage presentation has come up with a NoSQL taxonomy that thankfully focuses a little more on what NoSQL is, than what it isn't : key‐value‐cache memcached, repcached, coherence, infinispan, eXtreme scale, jboss cache, velocity, terracoqa key‐value‐store keyspace, flare, schema‐free, RAMCloud eventually‐consistent key‐value‐store dynamo, voldemort, Dynomite, SubRecord, Mo8onDb, Dovetaildb ordered‐key‐value‐store tokyo tyrant, lightcloud, NMDB, luxio, memcachedb, actord data‐structures server redis tuple‐store gigaspaces, coord, apache river object database ZopeDB, db4o, Shoal document store CouchDB, Mongo, Jackrabbit, XML Databases, ThruDB, CloudKit, Perservere, Riak Basho, Scalaris wide columnar store BigTable, Hbase, Cassandra, Hypertable, KAI, OpenNeptune, Qbase, KDI "Who will win?"
3 0.96029341 400 high scalability-2008-10-01-The Pattern Bible for Distributed Computing
Introduction: Software design patterns are an emerging tool for guiding and documenting system design. Patterns usually describe software abstractions used by advanced designers and programmers in their software. Patterns can provide guidance for designing highly scalable distributed systems. Let's see how! Patterns are in essence solutions to problems. Most of them are expressed in a format called Alexandrian form which draws on constructs used by Christopher Alexander. There are variants but most look like this: The pattern name The problem the pattern is trying to solve Context Solution Examples Design rationale: This tells where the pattern came from, why it works, and why experts use it Patterns rarely stand alone. Each pattern works on a context, and transforms the system in that context to produce a new system in a new context. New problems arise in the new system and context, and the next ‘‘layer’’ of patterns can be applied. A pattern language is a structured col
4 0.95264024 1359 high scalability-2012-11-15-Gone Fishin': Justin.Tv's Live Video Broadcasting Architecture
Introduction: This is one of my favorite posts for a couple of reasons. I think it gives a lot of useful information in an interesting space. And Kyle Vogt was just a real pleasure to talk to. He was very helpful and forthcoming, which makes the whole experience better for everyone. The future is live. The future is real-time. The future is now. That's the hype anyway. And as it has a habit of doing, the hype is slowly becoming reality. We are seeing live searches, live tweets, live location, live reality augmentation, live crab (fresh and local), and live event publishing. One of the most challenging of all live technologies is that of live video broadcasting. Imagine a world in which everyone becomes a broadcaster and a consumer of video streams, all in real-time (< 250 msec latency), all so you can talk and interact directly without feeling like you are in the middle of a time shift war. The resources and the engineering needed to make this happened must be substantial. How do you do tha
5 0.95258242 823 high scalability-2010-05-05-How will memristors change everything?
Introduction: A non-random sample of my tech friends shows that not many have heard of memristors (though I do suspect vote tampering). I'd read a little about memristors in 2008 when the initial hubbub about the existence of memristors was raised. I, however, immediately filed them into that comforting conceptual bucket of potentially revolutionary technologies I didn't have to worry about because like most wondertech, nothing would ever come of it. Wrong. After watching Finding the Missing Memristor by R. Stanley Williams I've had to change my mind. Memristors have gone from "maybe never" to holy cow this could happen soon and it could change everything. Let's assume for the sake of dreaming memristors do prove out. How will we design systems when we have access to a new material that is two orders of magnitude more efficient from a power perspective than traditional transistor technologies, contains multiple petabits (1 petabit = 128TB) of persistent storage, and can be reconfigured t
same-blog 6 0.951882 796 high scalability-2010-03-16-Justin.tv's Live Video Broadcasting Architecture
7 0.94398814 1311 high scalability-2012-08-24-Stuff The Internet Says On Scalability For August 24, 2012
8 0.94109404 735 high scalability-2009-11-01-Squeeze more performance from Parallelism
9 0.9366743 183 high scalability-2007-12-12-Report from OpenSocial Meetup at Google
10 0.92617399 776 high scalability-2010-02-12-Hot Scalability Links for February 12, 2010
11 0.92443085 399 high scalability-2008-10-01-Joyent - Cloud Computing Built on Accelerators
12 0.92340076 119 high scalability-2007-10-10-WAN Accelerate Your Way to Lightening Fast Transfers Between Data Centers
13 0.92315239 927 high scalability-2010-10-26-Marrying memcached and NoSQL
14 0.92121696 1431 high scalability-2013-03-29-Stuff The Internet Says On Scalability For March 29, 2013
15 0.91984248 1519 high scalability-2013-09-18-If You're Programming a Cell Phone Like a Server You're Doing it Wrong
16 0.91983438 517 high scalability-2009-02-21-Google AppEngine - A Second Look
17 0.91931337 301 high scalability-2008-04-08-Google AppEngine - A First Look
18 0.9185406 1502 high scalability-2013-08-16-Stuff The Internet Says On Scalability For August 16, 2013
19 0.91741574 195 high scalability-2007-12-28-Amazon's EC2: Pay as You Grow Could Cut Your Costs in Half
20 0.91728896 1148 high scalability-2011-11-29-DataSift Architecture: Realtime Datamining at 120,000 Tweets Per Second