New Data Questions Netflix’s Assertion That ISPs Are At Fault For Poor Quality

In the Netflix versus ISPs peering dispute, there are a lot of opinions and debate around who’s at fault for letting some peering points degrade and who should be responsible for upgrading them. To date, many are having a hard time separating facts from opinions because Netflix and the ISPs haven’t released any concrete data to back up their claims. In most industries, if one company accused another of doing something wrong, it would be expected that the company making the claim would back up their position with detailed data that proves their point and leaves little doubt as to who’s responsible for the problem. Netflix has yet to do that.

Most seem to be giving Netflix a pass, with very few demanding real transparency into what’s taking place, or changed, that degraded Netflix performance back in September 2013. No company should try and force us to take their word for it, they should simply make the data public and let us decide on our own. Netflix says they are bringing transparency to the debate, but they are doing the opposite by using vague and high level terms with no definition. To date, Netflix has yet to set forth any details on how they want the current business models to change, how it should be regulated, what they consider “strong” net neutrality or even submitted a proposal to the FCC.

The best example of this is how Netflix’s player recently gave out messages saying that Verizon was at fault regarding quality issues, but then when challenged by Verizon to back up their claim, Netflix announced they would discontinue showing these messages on June 16th. Originally Netflix said these messages would be rolling out in a phased deployment on all networks, but in their blog post yesterday, they now say these messages were just a “test”. To me, it looks like Netflix simply created noise in the market, again with no data, and then when pressured by Verizon to prove their case, Netflix instead decided to stop sending the messages and now release any details. Why? If the problem lies within Verizon, Netflix should let us see the data that shows this and stand behind it. Why back down if they have the data to show where the problem is coming from?

This is just another example of many where all sides simply point the finger at each other and say it’s the other guys fault, but then provide zero details to back up their claims. However, that may change soon as Netflix will likely publish network and performance graphs around a peering event, taking place in DC on June 18th, to bolster their argument. At the same time, some ISPs are actively working to release some data to the market, like the internal chart below [Figure 2] that I received from a major U.S. broadband provider, which gives us some more visibility into what’s taking place inside an ISPs network.

Netflix’s accusation is that ISPs have purposely congested their peering points in order to specifically degrade the Netflix service. What Netflix has failed to be transparent about is that Netflix has always paid to deliver their traffic. CDNs like Akamai, Limelight and Level 3 successfully managed the majority of all of Netflix’s video and were responsible for Netflix customer performance. Each of these companies successfully delivered Netflix via all the same transit paths and business relationships equally available to Netflix today. When Netflix took over the routing controls for their video traffic with their own CDN Open Connect, customer performance began to suffer as highlighted in Netflix’s own data that they shared with the Washington Post. I added a red circle to the chart to show when the Netflix changes took place and the impact to customer performance by ISPs.

Figure 1.

chart1
While Netflix was able to convince smaller regional ISPs to voluntarily offer settlement free peering, most large ISPs maintain national/international infrastructures, which require peering policies and consistent business practices to ensure fair and equal treatment of traffic. For the providers Netflix did not qualify for peering, Netflix moved their traffic onto very specific Internet paths that were not capable of handling their massive load and caused the congestion that impacted customers (as highlighted in Figure 2 below). In other words, if Netflix receives free peering, ISP customers receive good performance and high rankings and blogging praise from Netflix. But if Netflix does not receive free peering, ISP customers do not receive good performance and get low rankings and shame from Netflix.

It was Netflix that specifically chose transit paths to those ISPs who refused to give it free peering that it knew (and measured) were not capable of handling an increase in load. In some cases I was told by ISPs that traffic levels increased by 500% in only a few months where normal Internet growth with these same peers was less than 20-30% across an entire year. These ISPs’ customers did not request traffic to be served from poorly performing paths. Netflix chose to create, and use, paths that they knew were congested, simply because they were cheaper than using paths that were less congested. While some may not like that decision, Netflix is running a business and like all businesses, cost is a factor in a lot of decisions. I’m fine with Netflix having to make tradeoffs between quality and cost, but it’s not true that 100% of every path going into Comcast was “congested”.

Some of the many other transit providers I have spoken with confirm this, saying that they could have handled incremental Netflix traffic into Comcast, but that it would have been more expensive than Cogent, which was Netflix’s primary transit provider at the time. Even Cogent would not deny they were the cheapest transit provider of all the ones Netflix was using, but as we’ve all learned, cheap does not guarantee quality. During this same time, Netflix was still using other third-party CDNs for some of their video delivery. These CDNs were delivering the same Netflix service at the same time, to the same locations and with good quality. That is why some customers said their Netflix video was working great, while others said it was buffering and it is also why if some customers used VPNs, their performance improved. Netflix had control over who to give good service to and who to degrade, as shown in this chart below, from a major U.S. broadband provider:

Figure 2: Major U.S. broadband provider

Red is Netflix Video Bitrate delivered via a 3rd Party CDN
Blue is Netflix Video Bitrate delivered via Netflix internal CDN

BCC6DD36-44AB-42A5-A192-0AC7D2BE1071

When Netflix delivered video through similar sized third Party CDNs, customers received a consistent HD video stream 24 hours a day and from the CDNs I have spoken to, none of them had any problems getting Netflix’s traffic to the ISPs. When Netflix sent the video stream through their own CDN using their used newly congested transit paths, prime time viewing based on these decisions had buffering and low quality video.

Amongst those in the industry, there is a widely used term “traffic manipulation” and is “one of the most clever and devious of all the [negotiation] tactics” as described in The Peering Playbook. As the playbook details, the problems this traffic manipulation strategy introduces are three-fold. It negatively impacts customers; the levels of Netflix traffic also impacts other services sharing the same path; and if the “traffic manipulation” tactic is successful by Netflix or any other large CDN provider, it will be repeated by others creating further instability on the Internet.

Netflix’s point of view is that the Internet has changed a lot since many of these paid interconnect deals were done ten years ago. Today only a few ISPs control the vast majority of the market in the U.S. and Netflix would like to see peering and interconnect business models change. While that’s a fine argument to make, Netflix has yet to deliver any proposal or suggest an alternative other than to say it should be free. Netflix only confuses the discussion by involving “net-neutrality” in the debate and opportunistically point fingers at specific ISPs like Verizon and Comcast. But, Netflix has always had all the same delivery options as every major CDN and video provider on the planet. Many of these CDNs and large content sources have large volumes of traffic as Netflix and able to deliver Netflix and other Internet services with high quality service for their customers. The transmission decisions that Netflix makes are just as suspect to what is impacting their customers’ performance. Putting all the fault on ISPs is not accurate as both sides share the blame in not being able to make this work.

I’ve also heard from some that Netflix has told them that they have self-limited their transit decisions to only ISPs without residential access customers. Most CDNs use ISPs like AT&T, Sprint, Centurylink, Verizon and Comcast as transit options to reach the entire Internet, including each other’s network. Akamai and Limelight for example are connected to every Tier 1 provider and most Tier 2’s to deliver their service. They do not impose these business limitations in order to ensure they deliver services with high quality, so why is Netflix? These are not “peering issues”, these are first mile network decisions that Netflix is 100% in control of.

By Netflix limiting themselves to only Cogent, Level 3 and smaller International ISPs, they could find themselves buying capacity that is just not available. “Just upgrade your peering links” is not always the answer due to Internet peering policies as well as concerns that after investing capital, Netflix will just move terabits of traffic again in a few months making the problem start all over. What if Comcast gave Cogent all the capacity it wanted and footed the bill for it, and then the next year, Netflix moved away from Cogent and used Tata instead? Now Comcast would be forced to have to give Tata all the capacity they wanted and this change could happen every year. In fact, over a 3-4 year period, Netflix moved some or all of their video traffic amongst multiple CDNs including Akamai, Limelight and Level 3 and in some cases, due to lower pricing, left one provider only to come back to them a year or two later.

Netflix dominates close to 70% of the long form streaming traffic which is 10x their nearest competitor.  This market dominance allows Netflix to use their massive traffic controls in ways to force or demand special free peering privileges from ISPs rather than continuing to include these market based delivery costs in their service as others do. These costs that Netflix is objecting to have always been a cost of doing business for large CDNs in the U.S. That’s why to date, no other large CDN like YouTube, Apple, Microsoft, Amazon, Akamai or Yahoo! has come out publicly and backed Netflix’s position. Some ISPs, like Google Fiber, have said they think interconnects should be free, but that’s not the CDN portion of Google’s business, which for years, has done some paid interconnect deals.

Netflix has always paid a portion of their traffic delivery cost just like all of Netflix competitors large and small. Netflix is unhappy that they are being held to the same Internet policies as every other player. With great power comes great responsibility, and if Netflix will be operating one of the largest CDNs in the world, they must start behaving like one of the largest CDNs of the world. If they think the current model is broken or feel that market dynamics have changed enough to where new models needs to be put in place, that’s a fair argument. But you can’t make that argument and expect any change when you don’t put forth any kind of proposal. Netflix says they want changes that are “fair” and rules that are “strong”, but those words don’t mean anything without details.

Netflix hasn’t been clear with their arguments. First it was really about transit, but now they are saying it’s about “terminating access monopolies,” broadband competition and such. Problem is that is a much different issue and they’re having a hard time making that case and tying it to why they must have this or that remedy to lower their own costs. If broadband competition is now their issue, then it clearly is unrelated to net neutrality or the Comcast and TWC merger since the companies don’t overlap. There are other ways to go about addressing broadband competition constructively without bad-mouthing ISPs and confusing the public and throwing out made-up regulatory remedies like Title II or strong net neutrality.

If you talk to the major transit providers, CDNs, ISPs and the companies tied up in this debate, many will detail what has been going on behind the scenes with traffic manipulation and only using the data that makes them look good. There is more to it than that, and it’s one of the main reasons why those involved won’t detail what’s really taking place, where the problem lies, what their business motivations are for some of the decisions they make and the impact these decisions have on their bottom line. Remember, Netflix locked in a fixed rate with Comcast, for more than five years, for nearly a third of their traffic, which gives them a clear advantage over any other OTT competitor. So lets not forget the positive impact these interconnect deals are having on Netflix’s bottom line. Cheaper costs, lower churn, better quality video and the longest length contract, which I have come across, in the industry.

If Netflix really believes broadband competition is an issue, why make enemies with the companies who you can partner with to address it? To me, it all seems like an irrational strategy, especially when some of the arguments Netflix makes, don’t align with what some transit providers, CDNs and data from within ISPs shows. If Netflix wants real transparency, then they need to come out and put their cards on the table and show us what they have. It’s easy for anyone to complain and blame the other guy, but just because you might not like your ISP, think your cable bill is too high or have some other reason not to like your ISP, that’s not a reason to give Netflix a free pass and not demand they give us the transparency that they themselves keep saying they want.

John Oliver’s Rant On Net Neutrality Hurts The Industry, Shows Lack Of Common Sense

While some want to suggest that TV personality John Oliver brought a lot of exposure to the topic of net neutrality by doing a segment about it on his show last Sunday, in reality, all he did was make the industry take two steps back. In addition to getting many of the details on net neutrality wrong [something the LA Times did a good piece on when they say, "if only he'd gotten the facts right"], asking people to focus their “indiscriminate rage” on the FCC simply lacks common sense.

All John did was get people to clog the FCC’s comment system, which means it’s now even harder for the FCC to find comments that are detailed or offer any kind of real feedback and suggestions. He’s done a disservice to the industry and to companies and consumers that want a change by asking “trolls” to bombard the FCC. I spent many painful hours going through a few thousand submissions and so many of them are completely off-topic, vulgar, racist and useless.

These are the types of comments that many, many people sent in and there are a LOT that are far worse than these, with language and comments I can’t even publish:

  • “the people running the fcc should just die”
  • “no one in the FCC is sexy”
  • “my cable should not be turned off because I didn’t pay my bill on time”
  • “smoke some drugs and maybe you’ll understand better”
  • “i hope that no more violence is ever shown on television again!”
  • “corporate America needs to die!”
  • “i don’t like big brother”
  • “this is how skynet started to take over the world”
  • “the U.S. has too much wealth”
  • “I’m an American born citizen”
  • “the President is great friends with the CEO of Comcast”
  • “torrent!”
  • “the US is on a downward trend to second-rate status”
  • “Estonia and the Czech Republic beat us at internets”
  • “i should not have to pay so much for my DVR”

Not surprisingly, there is a lot of confusion of what net neutrality is even about. So many comments talk to things that have nothing to do with net neutrality at all, and quite a few people say that net neutrally is not broken and the FCC does not need to fix anything. Some think that net neutrality already exists, doesn’t exist, already went away, is going to go away or that the FCC gets paid by cable companies.

John Oliver did nothing to educate consumers, didn’t get the facts right, and played on people’s emotions, rather than their intelligence. When you ask “trolls” to focus their “indiscriminate rage” on the FCC website, nothing good comes of that. The quality of comments the FCC gets should be the focus, the volume of comments received should come after that. Not the other way around, which is all John Oliver accomplished.

Netflix’s Network Congestion Message Rolling Out On All ISPs, Not Just Verizon

As some Netflix customers using Verizon as their ISP have recently found out, Netflix has been working on a way to notify consumers when they feel that many clients are experiencing congestion on a certain segment of a certain ISP’s network. Many in the media reported today that select Verizon customers have gotten a message in their Netflix video window saying that “The Verizon network is crowded right now” and that Netflix is “Adjusting video for smoother playback”. While some suggested this new Netflix message is specific to Verizon, it’s not.

These messages are already showing up on multiple networks and is rolling out in a phased deployment. They are not everywhere yet, but will be soon. Netflix has confirmed they will display these messages whether or not they have a direct connection (paid or free) in place with the ISP. Netflix said their goal in doing this is to help their subscribers understand when their experience is degrading based on their network provider (as opposed to their home WiFi, etc). When Netflix feels that many clients are experiencing congestion on a certain segment of a certain ISP’s network, they will display the message for clients who are experiencing degradation.

Of course we don’t how Netflix defines “degradation” or the methodology they are using to determine the threshold of when a message is displayed or not, but they aren’t only doing this to Verizon. Not surprisingly, Verizon isn’t happy with the message and other ISPs won’t be as well because Netflix’s message is very vague. Saying any network is “crowded”, really does not mean much, and it does not say where the congestion is in the network, or who’s responsible for it. I have asked Netflix to release their methodology on what they consider a poor user experience and how they define “degradation” and will update this post if they provide more details. Netflix replied and gave out some details on the methodology.

Updated 6/5: Netflix says their methodology is if you are streaming from an ISP/Designated Market Area pair where (1) the average bitrate is poor (SD), (2) there is high congestion (the ratio between peak and trough traffic is abnormally compressed), and (3) they see a high percentage of sessions with a rebuffer, then the player displays the warning during the initial buffering at play start. If those criteria are met AND the user is actually streaming at a low bitrate (SD or below) then Netflix also displays an indicator if the play control bar is activated during the playback. That’s how it works now, but Netflix may modify/tune as they continue to roll out and learn more.

Thursday Webinar: How To Get The Most Out Of Enterprise Video

Thursday at 2pm ET, I’ll be moderating another StreamingMedia.com webinar, this time on the topic of, “How to Get the Most Out of Enterprise Video.” Video has become an integral part of the way individuals and businesses communicate and connect. It’s no simple task, however, to implement a secure and robust video solution in your enterprise and encourage adoption so that video can thrive throughout your business. This webinar will detail how you can manage, organize and securely distribute live and on-demand video to every employee throughout your organization. Hear from Ramp, MediaPlatform, Kaltura and Qumu how to:

  • Integrate video into your social business environment (Jive, SharePoint, IBM Connections)
  • Implement enterprise security and governance
  • Deliver video for the mobile work force
  • Create a single portal for on-demand and live streaming
  • Use metadata to organize and find content
  • Solve the challenges of distributing enterprise video

REGISTER NOW to join us for this FREE Web event.

Microsoft Webinar: Benefits of Moving Media Workflows to the Cloud

Screen Shot 2014-05-29 at 1.07.15 PMOn Tuesday June 3rd at 2pm ET, I’ll be joining members of the Microsoft Azure Media Services team to talk about the “Benefits of Moving Media Workflows to the Cloud“. Building on the success of live streaming the 2014 Sochi Winter Olympics, Azure Media Services will soon be opening up the underlying technology that made it possible to all of their customers. Join us for a live, in-depth discussion about the exciting new features and key advantages of moving media processing to the cloud. Learn how five major broadcasters expanded their digital audience and reduced the cost and complexity of delivering a broadcast-caliber experience to millions of viewers in twenty-two countries on today’s most popular consumer and mobile devices.

Register to attend this free webinar on Tuesday, June 3rd from 2-3pm ET and bring your questions for a lively Q&A session at the end of the presentation.