New Study From M-Lab Sheds Light On Widespread Harm Caused By Netflix Routing Decisions

On Tuesday, M-Lab released a new study on the impact of network interconnection on consumer Internet performance. The report entitled “ISP Interconnection and its Impact on Consumer Internet Performance“, details findings based on the speed test results collected by its test servers for various ISPs throughout the country over a roughly two-year period. For those not familiar with M-Lab, they provide the largest collection of open Internet performance data used by the FCC, amongst others, for the Measuring Broadband America program.

M-Lab data shows that around May 2013, suddenly and simultaneously throughout the country, speed test results for many ISPs (AT&T, Comcast, CenturyLink, Time Warner Cable, and Verizon) experienced a sudden and significant decline in performance to a specific set of transit providers (Cogent, Level 3 and XO). Just as suddenly around March 2014 the performance returns to normal for most of these same ISPs. Coincidentally, a few other ISPs who Netflix had negotiated direct Open Connect connections (Cablevision and Cox) did not experience similar decline in performance. The data presented in the study confirms what myself and others have surmised about Netflix being ultimately responsible for the dramatic, simultaneous decline in Netflix performance for all non-Open Connect ISPs.

If you look at the M-Lab measured history of the congestion, you will notice that these timelines line up very closely with Netflix’s migration from 3rd party CDNs onto their own Open Connect platform. The performance impact also matches closely with ISPs that did not agree to provide Netflix with Free Peering while other ISPs that agreed did not experience a performance impact.

96C2FD9F-3407-42DE-B9A0-C7C2971F9D40Looking at Figure 1 from the report (below), we can see that performance suddenly degrades for three of the four major broadband companies in the NY metro area according to an M-Lab test server housed on Cogent’s network in NYC around May 2013 and then performance suddenly improves for all three around March 2014. This tight coordination of impact for multiple ISPs simultaneously suggests that the cause was not something done by the ISPs, but rather by another entity. (Note: I added the heading and arrows to the chart)

36F74188-CE96-4A91-899F-AD8E14F61624What entity might be responsible? Well, figure 2 shows us that the fourth broadband ISP in the NY metro area testing on the M-Lab server on Cogent’s network, Cablevision (the only one of the four with a direct connection to Netflix’s Open Connect CDN) did not experience the same sudden drop/rise in performance over their link to Cogent.


Finally, M-Lab’s report also helpfully includes performance results for all four broadband ISPs in NY from a test server located on a different backbone connection (one that was not providing transit service to Netflix) showing no sudden performance changes for any ISP.


The report also shows that direct interconnection agreements between Comcast/Netflix increased performance for other ISPs. Unless there were performance issues further upstream of the interconnection, there should have been no impact on the interconnection agreement between Comcast/Netflix on other ISP networks. And according to M-Lab’s findings, performance issues on ISPs networks were not due to technical issues but rather the business deals between ISPs. They say, “we were able to conclude that in many cases degradation was not the result of major infrastructure failures at any specific point in a network, but rather connected with the business relationships between ISPs“.

While some may want to take this report as a smoking gun that ISPs are causing congestion, they may forget, not understand, or purposely leave out, the fact that large content providers control the delivery of their traffic and can AVOID congestion. A recent MIT study “Measuring Internet congestion: A preliminary report” pointed out the fact that the ISPs singled out in this report have multiple alternative paths to reach them. The report states that, “Congestion at interconnection points does not appear to be widespread. Apart from specific issues such as Netflix traffic, our measurements reveal only occasional points of congestion where ISPs interconnect. We typically see two or three links congested for a given ISP, perhaps for one or two hours a day, which is not surprising in even a well-engineered network, since traffic growth continues in general, and new capacity must be added from time to time as paths become overloaded.”

Most agree that when Netflix, again, moved their traffic off of these newly congested paths to direct connections, performance improved both for Netflix services as well as other services impacted by this new congestion. What is puzzling however is the timing of this improvement. If you look at the graph above you will notice that all ISPs improved simultaneously in Feb 2014. This is the exact same time that Netflix and Comcast migrated traffic to their direct connection. While it is understandable that Comcast would improve, no one has explained how a Comcast direct connection would improve AT&T, Verizon, and Time Warner unless there were additional problems between the Netflix server and their transit ISPs themselves. When Netflix moved this traffic their congestion within their transit ISPs improved other destinations.

What M-Labs is trying to do is good for the Internet, but they need to expose more of the end-to-end problem. If they truly want to understand Internet congestion and user experience, they need to not only focus on interconnect, but they also should expand their measurement to the quality of transit ISPs and acknowledge the choices content sources make when delivering traffic to their customers. For example, a measurement can identify if there are material differences between a variety of OTT sources such as Amazon Prime, Netflix, Hulu and YouTube on a given ISP. If Amazon Prime HD video quality was excellent, but another source was poor, it would be interesting to determine why that’s occurring, and what options the content provider has to improve their services.

While many were quick to blame ISPs for problems consumers were having with their Netflix streaming experience, we’ve now have a lot of data in the market showing that the choices Netflix made directly impacted the quality of their video and other services as well. Between this new M-Lab data, the interconnection findings published by David Clark at MIT/CAIDA, this data, and a recently published research report that says Netflix is using calls for greater net neutrality to drive down the prices they pay, it’s now clear just how much control Netflix really has over the quality of video they deliver.

Netflix Keynote Presentation To Detail Encoding Specs, 4K Video Encoding Pipeline

netflixWhen it comes to streaming video online, Netflix has been doing it longer than almost anyone and has some of the best quality video available. If you’ve ever wanted to know how Netflix gets such good quality and how they encode their videos, then you won’t want to miss their keynote on November 19th at the Streaming Media West Show. David Ronca, Director of Encoding Tools at Netflix will detail their encoding service and the key engineering decisions they made around video encoding. Some of the topics that David will be cover include scalability, progressive and parallel video encoding, and Netflix’s 4K video encoding pipeline. Register online using the code 200DR for a “Discovery Pass” and get free access to the keynotes, exhibit hall, discovery track sessions, and receptions. #smwest

Amazon Announces New Streaming Device, Fire TV Stick, Special Limited Pricing Of $19

Fire_TV_StickThis morning Amazon announced a new HDMI streaming device called the Fire TV Stick, which is already available for pre-order and will ship before black Friday. (Nov. 19th) Amazon is offering Prime members, or those who sign up for Prime, special pricing on the stick for only $19, for the next 48 hours, after which it will retail for $39. Many might be quick to compare this to Google’s Chromecast stick, but unlike Chromecast, Amazon’s Fire TV Stick won’t require another device to use it. Amazon’s streaming stick comes with a remote and most, but not all, of the functionality of their $99 Fire TV box.

The Fire TV Stick has a dual-core processor (Broadcom Capri 28155, dual-core 2xARM A9), 1GB of RAM, 8GB of storage, Bluetooth 3.0 and dual-band, dual-antenna Wi-Fi (MIMO). From a CPU standpoint, on paper it beats Roku’s Streaming Stick and Google Chromecast in terms of performance many times over. The Fire TV Stick comes with the same user interface as the Fire TV and has support for all of the current content channels on the Fire TV, with the exception of some gaming apps. Casual games can still be played on the stick, like Angry Birds, but some complex games, requiring more controls aren’t supported.

The remote that comes with the Fire TV Stick doesn’t have the same voice search functionality as the Fire TV remote, but it is compatible with the voice remote if bought separately for $29. Users can also use a free Android app on their phone to search for content with their voice and Amazon says an app for iOS is coming shortly. Non-gaming apps that are written for Fire TV will automatically work on the Fire TV Stick, which is also based on a flavor of Android. Amazon also confirmed that they are on track to have support for HBO Go on the Fire TV before the end of December, with the Fire TV Stick getting HBO GO in the new year.

Amazon told me they have “made a lot” of the Fire TV Sticks in preparation for the pre-orders, but in usual Amazon fashion, won’t disclose how many are available before they sell out or if they expect there to be a shortage during the holidays. Amazon has placed a buying limit of two devices per Prime member, at the special $19 price point.

While Amazon’s new stick is good for consumers, it’s bad news for Roku, which now has even more competition, at a lower price point. I love my Roku box and it still has more premium content channels than any other $99 or less device, but it’s only a matter of time before Amazon’s streaming stick, or Google’s new Nexus Player for that matter, catches up with Roku in terms of content choices. Roku simply can’t compete with Amazon or Google’s marketing dominance and their ability to package these products in with other services.

In the long run, the non-gaming streaming device space is going to be won by Apple, Google and Amazon. They all control multiple devices that tie into a larger ecosystem, make money from many avenues and at some point, Amazon or Google is going to get the price on these sticks down to where they are free. It’s not hard to imagine Amazon giving them away with a Prime membership or Google giving it away with a Nexus tablet or phone. At $19, the price point doesn’t have far to go before it reaches zero.

Just Announced: Cory Mummery, VP and GM of NFL Now To Keynote Streaming Media West Show

nflI’m pleased to announce that Cory Mummery, VP/GM of NFL Now will be the keynote speaker on the first day of the Streaming Media West Show, taking place November 17–19 at the Hyatt Regency Huntington Beach Resort & Spa in Huntington Beach CA. Come hear about their new cross-platform digital product, NFL Now, that packages all available NFL produced live and on-demand video from NFL Films, NFL Network, NFL games and all 32 NFL teams for viewing on multiple devices. Register online using the code 200DR for a “Discovery Pass” and get free access to the keynotes, exhibit hall, discovery track sessions, and receptions. #smwest

Google Launches New $99 Streaming Box With Android TV; Joins Apple & Amazon With Ecosystem Play

player-overview-1024The $99 streaming device market just got a little more crowded with Google’s announcement of their new streaming box, dubbed Nexus Player. Built in partnership with ASUS, it’s the first device to run Android TV and also allows you to play Android games on your TV, with a separately priced gamepad. Unlike Google’s Chromecast USB stick, the Nexus Player includes an app guide, content recommendations and voice search controls. It’s also Google Cast Ready so you can fling content from Chromebooks and Android/iOS devices to your TV. Inside the device is a 1.8GHz Quad Core, Intel Atom chip, 1GB of RAM, 8GB of storage, HDMI out, with WiFi support for 802.11ac 2×2 (MIMO). The lack of an ethernet port is a big downside as I’ll take the reliability of an ethernet cable over WiFi any day. The box is up for pre-order tomorrow and will be in stores on November 3rd.

The initial content options available at launch are limited on the box with the major ones being Google Play, Netflix, Hulu Plus, YouTube, Vevo, Pandora, iHeart Radio and support for Plex. Other apps from those like the Food Network and PBS Kids are also available, but content choices are pretty limited right now. Other boxes, including Amazon’s Fire TV were also limited in their content choices at launch, so we can expect to see a lot more content come to Google’s new box fairly quickly. I expect it won’t take more than a year before Google’s $99 Nexus Player will be very similar to Roku, Apple TV and Amazon’s Fire TV, with regards to content choices.

With so many streaming options in the market when it comes to Smart TV’s, game consoles, connected Blu-ray players and $99 streaming boxes, Google is entering a very crowded consumer market. But the advantage Google, Apple and Amazon have over everyone else is that they all operate and control an end-to-end video ecosystem. In the long run, they will all be the winners in the $99 streaming box market, and devices from Netgear, Sony, Western Digital and others stand no chance with their boxes. Just within the last year alone, devices from Vizio (Co-Star), D-Link (MovieNite Plus), Hisense (Pulse), Sony (SMP-N200) and Seagate (GoFlexTV) have all been discontinued in the market.

Long-term winners in the space will be Apple, Amazon, and Google for dedicated streaming boxes/USB sticks, and Microsoft and Sony with their more expensive gaming consoles. Where this leaves Roku in the long-term is unknown, as they still have the best $99 streamer in the market today when it comes to content choices, but that gap between them and others keeps shrinking. Roku doesn’t have the marketing power of Apple, Amazon or Google so it’s going to continue to be a tough fight for them in a very crowded market.