Netflix & Level 3 Only Telling Half The Story, Won’t Detail What Changes They Want To Net Neutrality

Late yesterday, Netflix’s CEO published a blog post making the argument that “stronger” net neutrality is needed and that Netflix will “fight for the Internet”. This follows a post earlier in the week from Level 3 in which they said some ISPs are playing “chicken” with the Internet and purposely causing “congestion”. Unfortunately, both companies are only telling half the story when it comes to this subject and are leaving out some very important facts. They are also making some statements that are factually inaccurate, contradicting themselves and providing nothing in the way of clarity to the topic. [If you want to know how the interconnection deal between Comcast and Netflix deal works, from a technical level, with numbers on what it costs, read my other post: Here’s How The Comcast & Netflix Deal Is Structured, With Data & Numbers]

If anything, Netflix, Level 3 and Cogent are doing the opposite and muddying the conversation by using vague, generic and high-level terms, with no definition of what they mean or how they think they should be applied. Even worse, they aren’t detailing any business or technical alternatives on how they think the problem could be solved. They are all doing a lot of posturing and complaining in the media, yet to date, none have outlined any detailed proposal on how they would like to see interconnection relationships regulated or how they want net neutrality rules changed.

Netflix’s main argument in their post is that in this day and age, they feel “stronger” net neutrality rules are required. While that’s their main point and a valid one to make if they want to argue it, they don’t detail how it could be made stronger. They use the term “stronger” twelve times in their post, yet never once define it. Generic phrases aren’t what we need on a topic that is already very confusing. We need clear, concise and well articulated details on how things could change for the better. What exactly does Netflix want to see done and why can’t they come right out and say it? It bothers me that Netflix is telling us that they “will continue to fight for the Internet”, as if they are championing the fight for all of us, but won’t tell us how and what they want as the end result. Netflix does say that ISPs “must provide sufficient access to their network without charge”, but once again don’t define what “sufficient access” means or how it should be measured.

What also makes this complicated is that Netflix is only highlighting the things that benefit their argument and isn’t telling the whole story. In some cases, they are also making statements that aren’t accurate. Netflix likes to make it sound like they have no choice when it comes to sending their traffic into the ISPs networks, when in fact, they have many choices. The transit market is extremely competitive, with at least a dozen major providers who offer transit services at different price points and with different SLAs. Netflix could use multiple providers to connect to ISPs and could also use third party CDNs like Akamai, EdgeCast and Limelight, who are already connected to ISPs, to deliver their traffic. In fact, this is how Netflix delivered 100% of their traffic for many, many years, using third-party CDNs. Netflix likes to make it sound like there is only one way to deliver videos on the Internet when in fact, there are multiple ways. No one who understands how the Internet works would debate this.

Netflix’s whole argument is that ISPs are purposely letting their peering points get congested but what Netflix isn’t talking about is some of the stuff they have done behind the scenes to make matters worse. Saturating a peering point can easily be prevented if you buy transit from multiple providers, which Netflix does. But the reason Cogent is the one transit provider we always seem to hear about is because Netflix continued to push their traffic through Cogent even though they knew it was already congested. Even though Netflix was buying transit from multiple providers, it wasn’t routing around capacity issues, like all the other CDNs do. So why did Netflix continue to push their traffic through Cogent even though they knew the link was congested? That practice is abnormal for any CDN to do as it impacts the quality of the video being delivered. Remember, no ISP decides how the traffic comes into their network or which transit providers Netflix uses. When you use third party CDNs, they also buy transit from multiple providers so that the can route traffic in real time around places where there is congestion.

In reality, the blame could fall on Netflix for continuing to send traffic over a link they know is congested, when alternatives exist in the market. The blame could also fall on the transit provider who sold Netflix capacity that they know they can’t deliver based on their current peering arrangements. It could also be that the broadband service provider isn’t upgrading a link that is still under peering compliance, per their peering policy. Laying the blame on the ISP isn’t always the case or always accurate. It may be at times, but the company that should be blamed will be different depending on the business situation of the companies involved.

So as much as Netflix wants to make this into a net neutrally issue, it’s a business issue. Netflix has alternatives, they chose not to use them. One could also argue that, by Netflix not routing around the performance issues with Cogent, the end result is that it forces the ISP to take angry calls from consumers. And if the ISP gets enough of those calls, maybe the ISP would then agree to join Netflix’s Open Connect program and allow Netflix to come into the ISPs last mile to place their own servers. Netflix’s motives in this whole argument is to protect their business, which is fine, but then they should not portray their argument as one where they are “fighting for the Internet”.

In Netflix’s post they said that Cablevision is “already practicing strong net neutrality”, but neglects to remind us that Cablevision is in Netflix’s Open Connect program. So is that what Netflix defines as “strong” net neutrality, any ISP that agrees to Netflix’s terms? And those that don’t join Open Connect and don’t agree to Netflix’s terms, are those the ISP that have “weak” net neutrality principles? To me, that sounds more like Netflix defining who is “strong” or “weak” based on Netflix’s own business terms and not true net neutrality. Netflix also calls out Comcast as “supporting weak net neutrality”, but the fact is Comcast is the only ISP that legally even has to follow the rules, due to their purchase of NBC.

Netflix also mentions in their post that strong net neutrality prevents ISPs from charging a toll for interconnection to services like “Netflix, YouTube, or Skype” and intermediaries such as “Cogent, Akamai or Level 3”. So why aren’t Google, Akamai, Yahoo!, AOL, Facebook, Twitter, Microsoft, Apple, Limelight Networks, EdgeCast and other CDNs and content owners who have built their own CDNs also complaining as Netflix is? Netflix is the only major content owner whom we have heard from, in a public forum, that thinks interconnection should be covered under net neutrality rules. It’s interesting to note that Netflix mentions YouTube by name, but the vast majority of Google’s content is already delivered inside the last mile via GGC (Google Global Cache). Not every ISP, but the majority of them, have been happy to work with Google and place Google caches within their network. So one has to ask themselves why Google has success working with ISPs, but Netflix isn’t.

Moving on to Level 3’s blog post, they too use a lot of generic statements saying they want “reasonable terms” when it comes getting capacity from ISPs but don’t define what that means. Level 3 could outline technical alternatives, like bringing the content further into the last mile at their own cost, but to date, they haven’t proposed such a plan. Level 3’s post says that the way the Internet works, “providers must spend money and connect their networks together.” So on one hand they outline how it needs to be done, but then also argue no payment should be made.

If you thought that was confusing, just think about this. When Cogent wanted to send more traffic into Level 3’s network than Level 3 was sending in return, Level 3 told Cogent it had to pay. Level 3 said, “There are a number of factors that determine whether a peering relationship is mutually beneficial. For example, Cogent was sending far more traffic to the Level 3 network than Level 3 was sending to Cogent’s network. It is important to keep in mind that traffic received by Level 3 in a peering relationship must be moved across Level 3’s network at considerable expense. Simply put, this means that, without paying, Cogent was using far more of Level 3’s network, far more of the time, than the reverse. Following our review, we decided that it was unfair for us to be subsidizing Cogent’s business.” You read that right. Level 3 thinks Cogent should have to pay them, because the balance of traffic isn’t equal, but Level 3 doesn’t think it should have to pay Comcast, even though that traffic is also lopsided. Confused yet?

Level 3 did talk about wanting to implement a peering system based on “bit miles”, but that’s a new and unknown unit of measure that has no common definition in the industry, that I can find. Level 3 proposes that many of its CDN bits travel very few bit miles because they offer to place CDN servers in or near ISP networks so there is very little burden on the ISP. That makes sense on paper, but it ignores the whole balance of trade or investment value proposition on which peering is based. The CAPEX and OPEX differentials between building and growing a broadband network and building a CDN are immense. For the bit-miles approach to gain traction, it would likely require an organization like the IETF to define precisely what bit miles are, how they are measured and how an exchange of traffic could take place using such an approach. Defining a new measurement, basing new rules off of it and trying to enforce those rules without industry acceptance seems impossible.

My goal when writing about this subject is to try and bring some transparency to what is taking place, which is hard to do when the companies involved want to be vague and all have business motives behind their decisions. This whole debate is not going to move forward until these companies start detailing what they actually want, the real impact it will have on their business (with numbers) and give specifics on exactly what should be regulated how it should me measured, monitored and priced. Until that happens all they are doing is confusing the media, consumers, legislators and keeping this debate from moving forward.

Any company that want’s new rules, regulations and the government to get more involved in their business, without defining what they mean, better be ready to have to deal with some laws that might get passed in the end that they didn’t actually want. In my opinion, these companies that are asking for regulation, but not providing the details on what they really want are asking for potential trouble. There are a lot of smart people at these companies and all of them should have already proposed detailed alternatives on how this could be solved. I know that some of them plan to make additional filings today, but from what I have already seen and heard, none of these filings will detail or outline any real alternatives to the situation.

[My first post on the deal can be found here: Inside The Netflix/Comcast Deal and What The Media Is Getting Very Wrong]

Disclaimer: It’s nearly 2am ET and I need to get to bed, so apologies if my eyes missed some grammar errors. I will re-proof it in the morning.

  • Raindeer

    Bit miles are old, 90s stuff, every peering manager should know. What I remember is that J. Scott Marcus came up with it when he was CTO of Genuity. Dr. Peering writes about it on his blog.

    Do know that historically most peering managers did not buy into the balanced traffic argument. The simple economic argument is that the investments both parties need to make are equal whether the traffic is balanced or not. Both still have to buy new 10Gbit ports if traffic in one direction exceeds 10Gbit, regardless of traffic in the other direction. And historically companies like Level 3 only peered with networks that sent them at least a third of what they received from Level 3. But that was when the power was in the backbone. Now the power is in the last mile and the same ratio has been reversed. That is why you see such inconsistency with Level 3. It didn’t make sense before and now somebody changed the interpretation in favor of eyeball networks.

    With regards to Netflix, your argument seems to boil down to, that they should have paid Comcast directly or could do it indirectly via those that already paid up. Effectively you are arguing that Netflix should not develop its own CDN despite that it makes economic sense for it to do so. On top of it you argue that Netflix is not allowed to choose transit providers it wants to buy from and that have a peering agreement with Comcast, because Comcast is selective about who it upgrades. Or to say it differently: Everything would have been fine if Netflix had just used the Comcast approved CDN and transit provider.

    Historically the Internet worked on innovation. Companies rightly and wrongly thought they knew best how to route their traffic and build and distribute their servers. they had to buy transit. At first from the Tier 1s. These quickly turned into jerks. The reaction was Donut Peering, where particularly in Europe everybody peered at major exchangepoints, with up to 600 networks at an exchange. The remaining traffic needed was bought primarily from Tier 2 networks that had limited global coverage, but good regional coverage. This left the Tier 1 networks in the cold. The Peering decision was a financial decision, do I save more money by peering than if I buy transit and the answer is generally yes. This was made even easier by the use of Routeservers at major exchangepoints, which allow peerings to be set up automatically with every new peer that joins the exchange. Now sometimes some networks do not want to peer, but transit was a well functioning alternative. Both parties paid based on 95 percentile peak traffic to the transit provider of their choice. And then some eyeball networks realized if they did not upgrade their ports, they could make life hard for the other networks and squeeze them into buying paid peering. They could do that because in their market for eyeballs they faced no competition, whereas the peer in the market for CDNs and content did face competition. That is where we are today.

    It appears this innovation is what Netflix is appealing to. Today it can only buy from Comcast approved networks (those that already pay). It has no competitive alternatives.

    BTW this is not new. It happened in Korea between Samsung and KT, in France between Free and Google, in Norway between the public broadcaster and Telenor.

    • danrayburn

      “It appears this innovation is what Netflix is appealing to.”

      Sounds good, but what do they want exactly? Why won’t they say how they want it to work? Give out a framework? Provide alternative solutions? No company can go around saying they want change but then not suggest how the change should or could happen and expect to be taken seriously.

    • storm

      “So as much as Netflix wants to make this into a net neutrally issue,
      it’s a business issue. Netflix has alternatives, they chose not to use
      them.” He directly says Netflix should consider its alternatives, even if that means building its own CDN. Netflix doesn’t want to pursue that option so it instead is trying to use regulations to avoid the investment.

      • swollennode

        Actually, Netflix had its own CDN and welcomed any ISPs to use it for free. Comcast chose not to use it, therefore, Netflix had to scrap it.

  • Joe Smith

    Isn’t the reason Netflix chooses to route so much over Cogent because Cogent is significantly cheaper as a transport provider than competitors?

  • Greg Robbins

    Comcast is just exerting its monopoly power, limiting top quality service for high-traffic sites to cases where the consumers (with no alternative) and content providers (with no alternative) both pay Comcast directly.

    As Raindeer lays out, the canard of peering balance is a MacGuffin, since it’s meaningless in which direction the bits flow. For the monopolist, the goal is to ensure that paying for access to an open internet is replaced by a model of always paying the monopolist.

    • stom

      how on earth do people have no alternative? In every major market there are two broadband providers, and many have more than that.

      • bceagle04

        You would think so, but the city of Boston only has one broadband internet provider, Comcast. We are completely at their mercy.

      • Nope

        Only game in town in Atlanta. Try again.

      • GIRLFUCKER420

        not even remotely true lmao the american broadband market is abysmal

  • I agree this is not a net neutrality issue, this is a business issue. But it does illustrate a problem I’ve pointed out in other forums: no two people agree on exactly how to define “net neutrality”.

  • Guest

    There is also a response from Comcast in this re/code article that seems to echo your sentiments:

    “Here’s a statement from Comcast’s David Cohen, who oversees the company’s approach to regulators, among other duties:

    ‘There has been no company that has had a stronger commitment to openness of the Internet than Comcast. We supported the FCC’s Open Internet rules because they struck the appropriate balance between consumer protection and reasonable network management rights for ISPs. We are now the only ISP in the country that is bound by them.

    ‘The Open Internet rules never were designed to deal with peering and Internet interconnection, which have been an essential part of the growth of the Internet for two decades. Providers like Netflix have always paid for their interconnection to the Internet and have always had ample options to ensure that their customers receive an optimal performance through all ISPs. We are happy that Comcast and Netflix were able to reach an amicable, market-based solution to our interconnection issues and believe that our agreement demonstrates the effectiveness of the market as a mechanism to deal with these matters.'”

  • uniquename72

    I don’t know why the author here is being deliberately obtuse. The fact is, there is no such thing as “Net Neutrality” at this point in the U.S.

    Netflix isn’t asking for *changes* to Net Neutrality; they’re asking for the *codification of* Net Neutrality. Just like most customers and tax payers are (at least, the ones who are paying attention).

    No double-dipping from the ISP monopolies (who just happen to ALSO be major content providers and therefore Netflix competitors). This isn’t hard to understand, unless — like Comcast — there’s a monetary reason you’ve chosen to not understand it.

    • Uniquename12

      And your post is deliberately leaving out a few facts like: originators of traffic and receivers of traffic have always paid and therefore “double dipping” as you say is how the Internet has always been funded.

      Second missed fact by your comment is that Netflix has always paid someone and in this case they are just now allowing the networks that actually carry the traffic into the mix vs a middleman. That is not a bad thing if the price is right (which transit competition enforces)

      Netflix’s net-neutrality strategy is to get all users to pay for their internet costs vs keeping transit it in their customers cost of service. When they ship a DVD they include postage in the service. When they stream a movie their CDN transit costs are also included.

      Given all the graphs show simultaneous and strategic ISP degradation since October, it appears pretty clear that Netflix had their hand firmly on the performance wheel over the last 6 months to create this net neutrality campaign.

      • Uniquename22

        Reed Hastings = Frank Underwood. Manipulating the masses to do his bidding for him