high_scalability high_scalability-2012 high_scalability-2012-1359 knowledge-graph by maker-knowledge-mining

1359 high scalability-2012-11-15-Gone Fishin': Justin.Tv's Live Video Broadcasting Architecture


meta infos for this blog

Source: html

Introduction: This is one of my favorite posts for a couple of reasons. I think it gives a lot of useful information in an interesting space. And Kyle Vogt was just a real pleasure to talk to. He was very helpful and forthcoming, which makes the whole experience better for everyone. The future is live. The future is real-time. The future is now. That's the hype anyway. And as it has a habit of doing, the hype is slowly becoming reality. We are seeing live searches, live tweets, live location, live reality augmentation, live crab (fresh and local), and live event publishing. One of the most challenging of all live technologies is that of live video broadcasting. Imagine a world in which everyone becomes a broadcaster and a consumer of video streams, all in real-time (< 250 msec latency), all so you can talk and interact directly without feeling like you are in the middle of a time shift war. The resources and the engineering needed to make this happened must be substantial. How do you do tha


Summary: the most important sentenses genereted by tfidf model

sentIndex sentText sentNum sentScore

1 We are seeing live searches, live tweets, live location, live reality augmentation, live crab (fresh and local), and live event publishing. [sent-10, score-1.206]

2 One of the most challenging of all live technologies is that of live video broadcasting. [sent-11, score-0.832]

3 Imagine a world in which everyone becomes a broadcaster and a consumer of video streams, all in real-time (< 250 msec latency), all so you can talk and interact directly without feeling like you are in the middle of a time shift war. [sent-12, score-0.628]

4 Their 30 million unique monthly visitors even outshine YouTube in the video upload game, reportedly uploading nearly 30 hours per minute of video compared to YouTube's 23. [sent-19, score-0.86]

5 Justin talked about how live video was fundamentally different than YouTube's batch video approach, where all the video is stored on disk and replayed later on demand. [sent-22, score-1.491]

6 Live video can't be made by pushing video faster, it takes a completely differently architecture. [sent-23, score-0.86]

7 Since the YouTube Architecture article is the most popular article ever on this site, I thought people might also enjoy learning about live side of the video world. [sent-24, score-0.702]

8 tv makes all this live video magic happen, going way beyond the call, providing a tremendous number of juicy details. [sent-26, score-0.708]

9 Usher - custom business logic server for playing video streams. [sent-68, score-0.625]

10 The Live Video Architecture Why is live video difficult? [sent-81, score-0.631]

11 If you can't just do YouTube faster for live video, what makes live video such a challenge? [sent-85, score-0.832]

12 With live video if you exceed your network capacity even for a fraction of a second every single viewer will see buffering all at the same moment. [sent-90, score-0.832]

13 Every server can act as an edge server (where the video is streaming out of to a viewer) and an origin server (where the video is streaming into from a broadcaster). [sent-123, score-1.145]

14 The entire video stream stays in memory from the time it hits the origin server to when it's copied to other servers and when it's copied to viewers. [sent-131, score-0.882]

15 The video servers are fairly dumb, the overlay logic controlling the serving topology is managed by Usher. [sent-142, score-0.61]

16 They have a backbone network to get the video streams between datacenters. [sent-165, score-0.692]

17 While video streams are not streamed from disk, video is archived to disk. [sent-170, score-1.035]

18 The number of video servers may seem a little low for their traffic because with Usher they can run each video server to full capacity. [sent-185, score-1.138]

19 Building Usher as the core backbone of their video scalability on top of relatively dumb video servers is an excellent example of this strategy. [sent-299, score-0.933]

20 In live video this is often a big event and if you mess up a lot of people will spread the bad word about you. [sent-345, score-0.702]


similar blogs computed by tfidf model

tfidf for this blog:

wordName wordTfidf (topN-words)

[('video', 0.43), ('live', 0.201), ('usher', 0.198), ('streams', 0.175), ('peering', 0.171), ('transcoding', 0.133), ('kyle', 0.131), ('stream', 0.122), ('broadcaster', 0.121), ('viewers', 0.096), ('chat', 0.095), ('eyeball', 0.09), ('vogt', 0.09), ('youtube', 0.089), ('network', 0.087), ('crowds', 0.085), ('number', 0.077), ('everyone', 0.077), ('server', 0.076), ('load', 0.073), ('servers', 0.073), ('incoming', 0.071), ('people', 0.071), ('used', 0.068), ('custom', 0.067), ('gig', 0.065), ('copied', 0.062), ('capacity', 0.062), ('kan', 0.06), ('branch', 0.06), ('handle', 0.059), ('cost', 0.058), ('datacenter', 0.057), ('origin', 0.057), ('vp', 0.057), ('converse', 0.057), ('ca', 0.057), ('broadcast', 0.056), ('peer', 0.055), ('topology', 0.055), ('process', 0.055), ('feature', 0.054), ('interconnection', 0.054), ('served', 0.053), ('twice', 0.053), ('logic', 0.052), ('peak', 0.052), ('viewer', 0.052), ('traffic', 0.052), ('ip', 0.051)]

similar blogs list:

simIndex simValue blogId blogTitle

same-blog 1 0.99999851 1359 high scalability-2012-11-15-Gone Fishin': Justin.Tv's Live Video Broadcasting Architecture

Introduction: This is one of my favorite posts for a couple of reasons. I think it gives a lot of useful information in an interesting space. And Kyle Vogt was just a real pleasure to talk to. He was very helpful and forthcoming, which makes the whole experience better for everyone. The future is live. The future is real-time. The future is now. That's the hype anyway. And as it has a habit of doing, the hype is slowly becoming reality. We are seeing live searches, live tweets, live location, live reality augmentation, live crab (fresh and local), and live event publishing. One of the most challenging of all live technologies is that of live video broadcasting. Imagine a world in which everyone becomes a broadcaster and a consumer of video streams, all in real-time (< 250 msec latency), all so you can talk and interact directly without feeling like you are in the middle of a time shift war. The resources and the engineering needed to make this happened must be substantial. How do you do tha

2 0.99809456 796 high scalability-2010-03-16-Justin.tv's Live Video Broadcasting Architecture

Introduction: The future is live. The future is real-time. The future is now. That's the hype anyway. And as it has a habit of doing, the hype is slowly becoming reality. We are seeing live searches, live tweets, live location, live reality augmentation, live crab (fresh and local), and live event publishing. One of the most challenging of all live technologies is that of live video broadcasting. Imagine a world in which everyone becomes a broadcaster and a consumer of video streams, all in real-time (< 250 msec latency), all so you can talk and interact directly without feeling like you are in the middle of a time shift war. The resources and the engineering needed to make this happened must be substantial. How do you do that? To find out I talked to Kyle Vogt, Justin.tv Founder and VP of Engineering. Justin.tv certainly has the numbers. Their 30 million unique monthly visitors even outshine YouTube in the video upload game, reportedly uploading nearly 30 hours per minute of video compared to Y

3 0.32789889 274 high scalability-2008-03-12-YouTube Architecture

Introduction: Update 2:   YouTube Reaches One Billion Views Per Day . That’s at least 11,574 views per second, 694,444 views per minute, and 41,666,667 views per hour. Update: YouTube: The Platform . YouTube adds a new rich set of APIs in order to become your video platform leader--all for free. Upload, edit, watch, search, and comment on video from your own site without visiting YouTube. Compose your site internally from APIs because you'll need to expose them later anyway. YouTube grew incredibly fast, to over 100 million video views per day, with only a handful of people responsible for scaling the site. How did they manage to deliver all that video to all those users? And how have they evolved since being acquired by Google? Information Sources Google Video Platform Apache Python Linux (SuSe) MySQL psyco, a dynamic python->C compiler lighttpd for video instead of Apache What's Inside? The Stats Supports the delivery of over 100 million vide

4 0.30849648 991 high scalability-2011-02-16-Paper: An Experimental Investigation of the Akamai Adaptive Video Streaming

Introduction: Video is hot on the Internet and people are really interested in knowing how to make it work. Dan Rayburn has a post  pointing to a fascinating paper:  An Experimental Investigation of the Akamai Adaptive Video Streaming , which talks in some detail about the protocols big players like YouTube, Skype and Akamai use to serve video over on an inherently video unfriendly medium like the Internet. For Akamai they found: Each video is encoded in five versions at different bit rates and stored in separate files. The client sends commands to the server with an average inter departure time of about 2 s, i.e. the control algorithm is executed on average each 2 seconds.  Akamai uses only the video level to adapt the video source to the available bandwidth, whereas the frame rate of the video is kept constant. When a sudden drop in the available bandwidth occurs, short interruptions of the video playback can occur due to the a large actuation delay. For a sudden increase of the avai

5 0.28217837 1037 high scalability-2011-05-10-Viddler Architecture - 7 Million Embeds a Day and 1500 Req-Sec Peak

Introduction: Viddler is in the high quality Video as a Service business for a customer who wants to pay a fixed cost, be done with it, and just have it work. Similar to Blip and Ooyala, more focussed on business than YouTube. They serve thousands of business customers, including high traffic websites like FailBlog, Engadget, and Gawker. Viddler is a good case to learn from because they are a small company trying to provide a challenging service in a crowded field. We are catching them just as they transitioning from a startup that began in one direction, as a YouTube competitor, and pivoted into a slightly larger company focussed on paying business customers. Transition is the key word for Viddler: transitioning from a free YouTube clone to a high quality paid service. Transitioning from a few colo sites that didn't work well to a new higher quality datacenter. Transitioning from an architecture that was typical of a startup to one that features redundancy, high availability, and automation. Tr

6 0.26336929 1215 high scalability-2012-03-26-7 Years of YouTube Scalability Lessons in 30 Minutes

7 0.23470676 1201 high scalability-2012-02-29-Strategy: Put Mobile Video Into Cold Storage After 30 Days

8 0.20649372 576 high scalability-2009-04-21-What CDN would you recommend?

9 0.20645942 1355 high scalability-2012-11-05-Gone Fishin': Building Super Scalable Systems: Blade Runner Meets Autonomic Computing In The Ambient Cloud

10 0.2062771 750 high scalability-2009-12-16-Building Super Scalable Systems: Blade Runner Meets Autonomic Computing in the Ambient Cloud

11 0.18973885 1501 high scalability-2013-08-13-In Memoriam: Lavabit Architecture - Creating a Scalable Email Service

12 0.18942486 313 high scalability-2008-05-02-Friends for Sale Architecture - A 300 Million Page View-Month Facebook RoR App

13 0.18589051 70 high scalability-2007-08-22-How many machines do you need to run your site?

14 0.18412684 1240 high scalability-2012-05-07-Startups are Creating a New System of the World for IT

15 0.18101285 661 high scalability-2009-07-25-Latency is Everywhere and it Costs You Sales - How to Crush it

16 0.17497595 239 high scalability-2008-02-04-Streaming Video on Amazon EC2?

17 0.17023931 538 high scalability-2009-03-16-Are Cloud Based Memory Architectures the Next Big Thing?

18 0.16862214 954 high scalability-2010-12-06-What the heck are you actually using NoSQL for?

19 0.16828035 920 high scalability-2010-10-15-Troubles with Sharding - What can we learn from the Foursquare Incident?

20 0.16811028 1008 high scalability-2011-03-22-Facebook's New Realtime Analytics System: HBase to Process 20 Billion Events Per Day


similar blogs computed by lsi model

lsi for this blog:

topicId topicWeight

[(0, 0.372), (1, 0.153), (2, -0.052), (3, -0.126), (4, -0.039), (5, -0.14), (6, -0.018), (7, 0.091), (8, -0.008), (9, 0.003), (10, -0.009), (11, -0.048), (12, -0.094), (13, -0.007), (14, 0.101), (15, 0.185), (16, 0.054), (17, 0.128), (18, -0.093), (19, -0.126), (20, -0.062), (21, 0.031), (22, 0.12), (23, 0.002), (24, 0.027), (25, 0.074), (26, -0.016), (27, 0.064), (28, 0.084), (29, 0.117), (30, 0.03), (31, -0.044), (32, -0.01), (33, -0.049), (34, 0.01), (35, -0.034), (36, -0.114), (37, -0.185), (38, 0.014), (39, 0.051), (40, 0.037), (41, 0.033), (42, 0.031), (43, -0.04), (44, 0.095), (45, -0.037), (46, 0.031), (47, 0.029), (48, 0.168), (49, 0.058)]

similar blogs list:

simIndex simValue blogId blogTitle

1 0.96882707 796 high scalability-2010-03-16-Justin.tv's Live Video Broadcasting Architecture

Introduction: The future is live. The future is real-time. The future is now. That's the hype anyway. And as it has a habit of doing, the hype is slowly becoming reality. We are seeing live searches, live tweets, live location, live reality augmentation, live crab (fresh and local), and live event publishing. One of the most challenging of all live technologies is that of live video broadcasting. Imagine a world in which everyone becomes a broadcaster and a consumer of video streams, all in real-time (< 250 msec latency), all so you can talk and interact directly without feeling like you are in the middle of a time shift war. The resources and the engineering needed to make this happened must be substantial. How do you do that? To find out I talked to Kyle Vogt, Justin.tv Founder and VP of Engineering. Justin.tv certainly has the numbers. Their 30 million unique monthly visitors even outshine YouTube in the video upload game, reportedly uploading nearly 30 hours per minute of video compared to Y

same-blog 2 0.96809721 1359 high scalability-2012-11-15-Gone Fishin': Justin.Tv's Live Video Broadcasting Architecture

Introduction: This is one of my favorite posts for a couple of reasons. I think it gives a lot of useful information in an interesting space. And Kyle Vogt was just a real pleasure to talk to. He was very helpful and forthcoming, which makes the whole experience better for everyone. The future is live. The future is real-time. The future is now. That's the hype anyway. And as it has a habit of doing, the hype is slowly becoming reality. We are seeing live searches, live tweets, live location, live reality augmentation, live crab (fresh and local), and live event publishing. One of the most challenging of all live technologies is that of live video broadcasting. Imagine a world in which everyone becomes a broadcaster and a consumer of video streams, all in real-time (< 250 msec latency), all so you can talk and interact directly without feeling like you are in the middle of a time shift war. The resources and the engineering needed to make this happened must be substantial. How do you do tha

3 0.87140065 1201 high scalability-2012-02-29-Strategy: Put Mobile Video Into Cold Storage After 30 Days

Introduction: Limelight says  95% of Mobile Video Views Take Place in First 90 Days  and 88.8 percent of views take place in the first 30 days. Since a lot of people are working with video, which is expensive to store and serve, the implication: there's little need to keep your video close to the user or on a CDN after 30 days.

4 0.8601442 274 high scalability-2008-03-12-YouTube Architecture

Introduction: Update 2:   YouTube Reaches One Billion Views Per Day . That’s at least 11,574 views per second, 694,444 views per minute, and 41,666,667 views per hour. Update: YouTube: The Platform . YouTube adds a new rich set of APIs in order to become your video platform leader--all for free. Upload, edit, watch, search, and comment on video from your own site without visiting YouTube. Compose your site internally from APIs because you'll need to expose them later anyway. YouTube grew incredibly fast, to over 100 million video views per day, with only a handful of people responsible for scaling the site. How did they manage to deliver all that video to all those users? And how have they evolved since being acquired by Google? Information Sources Google Video Platform Apache Python Linux (SuSe) MySQL psyco, a dynamic python->C compiler lighttpd for video instead of Apache What's Inside? The Stats Supports the delivery of over 100 million vide

5 0.85382789 1037 high scalability-2011-05-10-Viddler Architecture - 7 Million Embeds a Day and 1500 Req-Sec Peak

Introduction: Viddler is in the high quality Video as a Service business for a customer who wants to pay a fixed cost, be done with it, and just have it work. Similar to Blip and Ooyala, more focussed on business than YouTube. They serve thousands of business customers, including high traffic websites like FailBlog, Engadget, and Gawker. Viddler is a good case to learn from because they are a small company trying to provide a challenging service in a crowded field. We are catching them just as they transitioning from a startup that began in one direction, as a YouTube competitor, and pivoted into a slightly larger company focussed on paying business customers. Transition is the key word for Viddler: transitioning from a free YouTube clone to a high quality paid service. Transitioning from a few colo sites that didn't work well to a new higher quality datacenter. Transitioning from an architecture that was typical of a startup to one that features redundancy, high availability, and automation. Tr

6 0.84986842 991 high scalability-2011-02-16-Paper: An Experimental Investigation of the Akamai Adaptive Video Streaming

7 0.82178921 294 high scalability-2008-04-01-How to update video views count effectively?

8 0.78614205 1215 high scalability-2012-03-26-7 Years of YouTube Scalability Lessons in 30 Minutes

9 0.74931055 576 high scalability-2009-04-21-What CDN would you recommend?

10 0.73630095 198 high scalability-2008-01-01-HOW CDN works

11 0.71453184 447 high scalability-2008-11-19-High Definition Video Delivery on the Web?

12 0.69599402 267 high scalability-2008-03-05-Oprah is the Real Social Network

13 0.68058872 1289 high scalability-2012-07-23-State of the CDN: More Traffic, Stable Prices, More Products, Profits - Not So Much

14 0.67807239 60 high scalability-2007-08-07-Can you profit from the coming Content Delivery Network wars?

15 0.67268199 270 high scalability-2008-03-08-DNS-Record TTL on worst case scenarios

16 0.66401285 239 high scalability-2008-02-04-Streaming Video on Amazon EC2?

17 0.66249049 773 high scalability-2010-02-06-GEO-aware traffic load balancing and caching at CNBC.com

18 0.65099198 244 high scalability-2008-02-11-Yahoo Live's Scaling Problems Prove: Release Early and Often - Just Don't Screw Up

19 0.64781606 39 high scalability-2007-07-30-Product: Akamai

20 0.64258933 123 high scalability-2007-10-15-Olympic Site Architecture


similar blogs computed by lda model

lda for this blog:

topicId topicWeight

[(1, 0.142), (2, 0.218), (10, 0.03), (30, 0.032), (40, 0.015), (47, 0.028), (49, 0.127), (61, 0.086), (73, 0.013), (77, 0.021), (79, 0.136), (85, 0.035), (94, 0.042)]

similar blogs list:

simIndex simValue blogId blogTitle

1 0.9670099 1085 high scalability-2011-07-25-Is NoSQL a Premature Optimization that's Worse than Death? Or the Lady Gaga of the Database World?

Introduction: Michael Stonebraker sure knows how to stir up a storm. Unlike for others, that doesn't make him a troll in my mind, he's way too accomplished in the field to be that, but he does have a bit of Barnum & Bailey in him, which serves to get the discussion flowing, and that's a good thing. A lot of previously hidden wisdom and passion unlocks, which we'll try to capture here. This disturbance in the force is over OldSQL vs NoSQL vs NewSQL . Warning, these are not crisp categories, there's leakage all over the place, watch your step: OldSQL  (Oracle, MySQL, etc) refers to what some want to term as legacy relational database like MySQL, that don't scale out horizontally with aplomb. NoSQL  (CouchDB, Redis, Cassandra, HBase, MongoDB, Riak, Neo4j, etc) refers to, well, a collection of technologies that aren't OldSQL, these often are designed to scale out horizontally, aren't on ACID, and use schemaless non-relational datamodels. NewSQL   (Xeround, Clustrix, NimbusDB, GenieDB, Sc

2 0.96404588 737 high scalability-2009-11-05-A Yes for a NoSQL Taxonomy

Introduction: NorthScale's Steven Yen in his highly entertaining  NoSQL is a Horseless Carriage  presentation has come up with a NoSQL taxonomy that thankfully focuses a little more on what NoSQL is, than what it isn't : key‐value‐cache memcached, repcached, coherence, infinispan, eXtreme scale, jboss cache, velocity, terracoqa  key‐value‐store keyspace, flare, schema‐free, RAMCloud eventually‐consistent key‐value‐store dynamo, voldemort, Dynomite, SubRecord, Mo8onDb, Dovetaildb ordered‐key‐value‐store tokyo tyrant, lightcloud, NMDB, luxio, memcachedb, actord data‐structures server  redis tuple‐store gigaspaces, coord, apache river object database ZopeDB, db4o, Shoal document store  CouchDB, Mongo, Jackrabbit, XML Databases, ThruDB, CloudKit, Perservere, Riak Basho, Scalaris wide columnar store BigTable, Hbase, Cassandra, Hypertable, KAI, OpenNeptune, Qbase, KDI "Who will win?"

3 0.96066386 400 high scalability-2008-10-01-The Pattern Bible for Distributed Computing

Introduction: Software design patterns are an emerging tool for guiding and documenting system design. Patterns usually describe software abstractions used by advanced designers and programmers in their software. Patterns can provide guidance for designing highly scalable distributed systems. Let's see how! Patterns are in essence solutions to problems. Most of them are expressed in a format called Alexandrian form which draws on constructs used by Christopher Alexander. There are variants but most look like this: The pattern name The problem the pattern is trying to solve Context Solution Examples Design rationale: This tells where the pattern came from, why it works, and why experts use it Patterns rarely stand alone. Each pattern works on a context, and transforms the system in that context to produce a new system in a new context. New problems arise in the new system and context, and the next ‘‘layer’’ of patterns can be applied. A pattern language is a structured col

4 0.95231169 823 high scalability-2010-05-05-How will memristors change everything?

Introduction: A non-random sample of my tech friends shows that not many have heard of memristors (though I do suspect vote tampering). I'd read a little about memristors in 2008 when the initial hubbub about the existence of memristors was raised. I, however,  immediately filed them into that comforting conceptual bucket of potentially revolutionary technologies I didn't have to worry about because like most wondertech, nothing would ever come of it. Wrong. After watching Finding the Missing Memristor by R. Stanley Williams I've had to change my mind. Memristors have gone from "maybe never" to holy cow this could happen soon and it could change everything. Let's assume for the sake of dreaming memristors do prove out. How will we design systems when we have access to a new material that is two orders of magnitude more efficient from a power perspective than traditional transistor technologies, contains multiple petabits (1 petabit = 128TB) of persistent storage, and can be reconfigured t

same-blog 5 0.95201784 1359 high scalability-2012-11-15-Gone Fishin': Justin.Tv's Live Video Broadcasting Architecture

Introduction: This is one of my favorite posts for a couple of reasons. I think it gives a lot of useful information in an interesting space. And Kyle Vogt was just a real pleasure to talk to. He was very helpful and forthcoming, which makes the whole experience better for everyone. The future is live. The future is real-time. The future is now. That's the hype anyway. And as it has a habit of doing, the hype is slowly becoming reality. We are seeing live searches, live tweets, live location, live reality augmentation, live crab (fresh and local), and live event publishing. One of the most challenging of all live technologies is that of live video broadcasting. Imagine a world in which everyone becomes a broadcaster and a consumer of video streams, all in real-time (< 250 msec latency), all so you can talk and interact directly without feeling like you are in the middle of a time shift war. The resources and the engineering needed to make this happened must be substantial. How do you do tha

6 0.95118493 796 high scalability-2010-03-16-Justin.tv's Live Video Broadcasting Architecture

7 0.94379318 1311 high scalability-2012-08-24-Stuff The Internet Says On Scalability For August 24, 2012

8 0.94227576 735 high scalability-2009-11-01-Squeeze more performance from Parallelism

9 0.9365896 183 high scalability-2007-12-12-Report from OpenSocial Meetup at Google

10 0.92496675 776 high scalability-2010-02-12-Hot Scalability Links for February 12, 2010

11 0.9234758 399 high scalability-2008-10-01-Joyent - Cloud Computing Built on Accelerators

12 0.92248881 119 high scalability-2007-10-10-WAN Accelerate Your Way to Lightening Fast Transfers Between Data Centers

13 0.92230839 927 high scalability-2010-10-26-Marrying memcached and NoSQL

14 0.92024231 1431 high scalability-2013-03-29-Stuff The Internet Says On Scalability For March 29, 2013

15 0.91930997 1519 high scalability-2013-09-18-If You're Programming a Cell Phone Like a Server You're Doing it Wrong

16 0.91866493 517 high scalability-2009-02-21-Google AppEngine - A Second Look

17 0.91796046 301 high scalability-2008-04-08-Google AppEngine - A First Look

18 0.91772085 1502 high scalability-2013-08-16-Stuff The Internet Says On Scalability For August 16, 2013

19 0.91652912 1051 high scalability-2011-06-01-Why is your network so slow? Your switch should tell you.

20 0.91632283 849 high scalability-2010-06-28-VoltDB Decapitates Six SQL Urban Myths and Delivers Internet Scale OLTP in the Process