New Study From M-Lab Sheds Light On Widespread Harm Caused By Netflix Routing Decisions

On Tuesday, M-Lab released a new study on the impact of network interconnection on consumer Internet performance. The report entitled “ISP Interconnection and its Impact on Consumer Internet Performance“, details findings based on the speed test results collected by its test servers for various ISPs throughout the country over a roughly two-year period. For those not familiar with M-Lab, they provide the largest collection of open Internet performance data used by the FCC, amongst others, for the Measuring Broadband America program.

M-Lab data shows that around May 2013, suddenly and simultaneously throughout the country, speed test results for many ISPs (AT&T, Comcast, CenturyLink, Time Warner Cable, and Verizon) experienced a sudden and significant decline in performance to a specific set of transit providers (Cogent, Level 3 and XO). Just as suddenly around March 2014 the performance returns to normal for most of these same ISPs. Coincidentally, a few other ISPs who Netflix had negotiated direct Open Connect connections (Cablevision and Cox) did not experience similar decline in performance. The data presented in the study confirms what myself and others have surmised about Netflix being ultimately responsible for the dramatic, simultaneous decline in Netflix performance for all non-Open Connect ISPs.

If you look at the M-Lab measured history of the congestion, you will notice that these timelines line up very closely with Netflix’s migration from 3rd party CDNs onto their own Open Connect platform. The performance impact also matches closely with ISPs that did not agree to provide Netflix with Free Peering while other ISPs that agreed did not experience a performance impact.

96C2FD9F-3407-42DE-B9A0-C7C2971F9D40Looking at Figure 1 from the report (below), we can see that performance suddenly degrades for three of the four major broadband companies in the NY metro area according to an M-Lab test server housed on Cogent’s network in NYC around May 2013 and then performance suddenly improves for all three around March 2014. This tight coordination of impact for multiple ISPs simultaneously suggests that the cause was not something done by the ISPs, but rather by another entity. (Note: I added the heading and arrows to the chart)

36F74188-CE96-4A91-899F-AD8E14F61624What entity might be responsible? Well, figure 2 shows us that the fourth broadband ISP in the NY metro area testing on the M-Lab server on Cogent’s network, Cablevision (the only one of the four with a direct connection to Netflix’s Open Connect CDN) did not experience the same sudden drop/rise in performance over their link to Cogent.


Finally, M-Lab’s report also helpfully includes performance results for all four broadband ISPs in NY from a test server located on a different backbone connection (one that was not providing transit service to Netflix) showing no sudden performance changes for any ISP.


The report also shows that direct interconnection agreements between Comcast/Netflix increased performance for other ISPs. Unless there were performance issues further upstream of the interconnection, there should have been no impact on the interconnection agreement between Comcast/Netflix on other ISP networks. And according to M-Lab’s findings, performance issues on ISPs networks were not due to technical issues but rather the business deals between ISPs. They say, “we were able to conclude that in many cases degradation was not the result of major infrastructure failures at any specific point in a network, but rather connected with the business relationships between ISPs“.

While some may want to take this report as a smoking gun that ISPs are causing congestion, they may forget, not understand, or purposely leave out, the fact that large content providers control the delivery of their traffic and can AVOID congestion. A recent MIT study “Measuring Internet congestion: A preliminary report” pointed out the fact that the ISPs singled out in this report have multiple alternative paths to reach them. The report states that, “Congestion at interconnection points does not appear to be widespread. Apart from specific issues such as Netflix traffic, our measurements reveal only occasional points of congestion where ISPs interconnect. We typically see two or three links congested for a given ISP, perhaps for one or two hours a day, which is not surprising in even a well-engineered network, since traffic growth continues in general, and new capacity must be added from time to time as paths become overloaded.”

Most agree that when Netflix, again, moved their traffic off of these newly congested paths to direct connections, performance improved both for Netflix services as well as other services impacted by this new congestion. What is puzzling however is the timing of this improvement. If you look at the graph above you will notice that all ISPs improved simultaneously in Feb 2014. This is the exact same time that Netflix and Comcast migrated traffic to their direct connection. While it is understandable that Comcast would improve, no one has explained how a Comcast direct connection would improve AT&T, Verizon, and Time Warner unless there were additional problems between the Netflix server and their transit ISPs themselves. When Netflix moved this traffic their congestion within their transit ISPs improved other destinations.

What M-Labs is trying to do is good for the Internet, but they need to expose more of the end-to-end problem. If they truly want to understand Internet congestion and user experience, they need to not only focus on interconnect, but they also should expand their measurement to the quality of transit ISPs and acknowledge the choices content sources make when delivering traffic to their customers. For example, a measurement can identify if there are material differences between a variety of OTT sources such as Amazon Prime, Netflix, Hulu and YouTube on a given ISP. If Amazon Prime HD video quality was excellent, but another source was poor, it would be interesting to determine why that’s occurring, and what options the content provider has to improve their services.

While many were quick to blame ISPs for problems consumers were having with their Netflix streaming experience, we’ve now have a lot of data in the market showing that the choices Netflix made directly impacted the quality of their video and other services as well. Between this new M-Lab data, the interconnection findings published by David Clark at MIT/CAIDA, this data, and a recently published research report that says Netflix is using calls for greater net neutrality to drive down the prices they pay, it’s now clear just how much control Netflix really has over the quality of video they deliver.

  • Tony Soprano

    Good ole Dan, still shilling for Comcast. How ya been?

    • danrayburn

      I have never worked for, been paid by, or compensated in any way by Comcast. Is that the best you got? Can’t you debate the real facts at hand?

      • Salvatore Bonpensi

        People only want to hear how ISPs are causing all the problems. Any data or analysis which says anything other than that needs to be squashed quickly and the best way is to personally attack the person that said it.

        In keeping with the mobster theme…. Sounds like OpenConnect was “I’m gonna make him an offer that he can’t refuse”

        • Jon_Irenicus

          People want to hear it explained why paid peering for endpoint monopoly/duopoly ISPs is some SACRED model that can’t be allowed to changed (read Dan’s view).

          I don’t mind paid peering in the commercial space because there are multiple tier 1 ISPs competing for those dollars, you have a natural market based check on prices and abuse. Where I live in Los Angeles, and in many other locations, there is only one real choice for high speed broadband. Cable internet. They have a captured audience, and so if they decided to increase the fees to interconnect with them to get content their own users are requesting, people are FORCED to pay those higher rates, and those rates will eventually scale to the customers using services like netflix in terms of higher prices.

          IF the home broadband market was more competitive, then paid peering would not be problematic, IF home broadband allowed third party ISPs to use the Cable infrastructure to get to customers like they do in the UK, it would be less of an issue, but none of those things apply for too many people. The attitude of just deal with it is not going to cut it. Saying netflix should just drop to it’s knees and accept the tributes demanded of them is sidestepping the argument. If some thug was holding a bridge and made me pay him to cross and get the cargo I was carrying to my customers then YES I could just pay him and avoid issues with the transit of that cargo. That is the argument Dan is making but so what. He does not deign to explain WHY it’s legitimate in the first place. He pretends that anti competitive concerns are the ravings of mad men, pointing to low rates today, ignoring the potential for abuse later on when the regulating agencies are not placing comcast/time warner under a microscope for their pending merger. I have every reason to expect that a publicly run company like comcast/time warner cable/verizon/at&t will do everything in it’s power to maximize shareholder value and profit. That is what they are SUPPOSED to do, and in monopoly ISP markets, there is no where else to go if they start exercising their market position.

          • Avatar Salvatore Bonpensi

            Your argument makes a lot of sense for Tier 1s and only large edge providers.

            Tier 1s are afraid of competition in the transit market and are very upset that an ISP can sell transit and compete their price points. Tier 1s want to sell others networks and don’t want others to be able to do the same.

            Edge providers want peering (when they otherwise would not qualify with Tier 1s or T1/2 w/ broadband) so it is best to inaccurately say that the transit market is not competitive and therefore Title II regulation is required.

            For the smaller guys that buy transit, they are left a bit more out of luck. They won’t qualify for the free peering due to size and Tier 1s can keep the price high because of less competition. Big guys win… small guys lose.

            Yes it can go both ways and ISPs can raise transit/PP rates, but then that has to compete with many others. Also ISPs can congest peering, but not really as that would negatively impact all services (not just Netflix) and all businesses (not just broadband).

            The negative ISP soundbites on this are easy, but the economics and implications are more complicated. I fear your position is far more anti-competitive for anyone other than the largest edge providers.

          • Jon_Irenicus

            Endpoint ISPs don’t have to compete with anyone in many markets for faster broadband. Who is the competition to stream netflix to their customers? dsl? How is dsl going to handle 4k streaming? Maybe the market will chance if vectored dsl is more scalable, but until then cable and fiber to the home is the only game in town. The entire point of the discussion is that THEY are the gatekeepers for transit to that customer base. They are not taking traffic and sending it to other networks, THEY are the endpoint for the traffic in question because once it’s on their network it goes directly to their customer base.

            The response of the customer base in these captured markets is not to switch to another broadband carrier, it’s to go suck lemon. That gives the large monopoly/duopoly ISPs far more leverage and potential areas for abuse than I am comfortable with.

            As for “free peering” to endpoint providers helping the big players more than the little guy, can we stop pretending that any of you care? “I” don’t even care about that. First, the peering is not no cost, if a netflix take the time and expense to build out their own capacity to send traffic all the way to comcasts front door, and pays for the upgraded routing capacity at the interconnects into comcasts network, what exactly is the great “expense” comcast is incurring? Having to deal with more traffic flowing on their network? THAT IS WHAT THEIR CUSTOMERS ARE PAYING FOR !!!!!!!!!!! Why is the onus to deal that that increased traffic not on comcast itself? Big players will have lower cost, because they almost always have lower costs by getting better rates based on volume, or by doing the heaving lifting of getting their traffic to the ISP in the first place. But while the “little guys” might have expenses that are not quite as low as a netflix, the fees needed to pay someone else like akamai to get their content into an ISP would still be lower if akamai did not have to pay comcast money for the privilege of doing what they should be doing in the first place. Handling the traffic requests of their customers on their network and the borders of THEIR network.

            And besides, reports are the content costs are far more material than these routing costs. So why does any of it matter? Because of the levers of power it places in the hands of ISPs. I don’t want comcast saying to netflix 5-7 years down the line, that they want radically higher rates to stream content into its network. And if they are still the only game in town, what is stopping them from doing that?

      • Paulie Walnuts

        Frost and Sullivan has, and Comcast sponsors your amateur-hour Streaming Media West. Be honest.

        • danrayburn

          Comcast has never sponsored the Streaming Media West show. Comcast has spoken at the show, for free, but so has Netflix, Level 3, Hulu, Disney and every other major media company and ISP. I have never worked on any project at Frost & Sullivan pertaining to Comcast, in any capacity

      • Jon_Irenicus

        You may not take payment, but you almost certainly have friends in the industry that you take the side of virtually every single time on this issue. You agree that the paradigm of paid peering for endpoint ISPs is a GOOD thing.

        Conveniently leaving out the levers of power that places in the hands of large ISPs to play hardball with interconnection fees down the line. I don’t want them to have those levers to pull, you do. You’ll engage in lies of omission and pretend you’ve never seen the tactic of starting out with small charges, then ratcheting those up ever higher over time that cable tv firms like comcast/time warner do all the time with television charges.

        • danrayburn

          So your argument is that cable TV firms raise rates all the time? While I agree with that statement, and have said on my blog many times that the rates are raised too often, that has nothing to do with the discussion.

          Read what I have said in multiple posts: “BOTH sides should bear the cost to improve video quality as it benefits both companies.” BOTH sides. Neither side should get it for free.

          Netflix should have to pay something, so should the ISP. Even Level 3 agrees that they should have to pay part of the cost of adding more capacity, they don’t argue that. They are arguing that the rate should be lower, but not that they shouldn’t pay. Only Netflix is arguing that it should be 100% free to them to have as much capacity as they want to any ISP. they want, at any time. That’s not rational. It’s not how the market works.

          Why is Netflix the ONLY content owner who thinks how the market works today is a bad idea? Why did Apple just pay for interconnects without any complaint when they built their new CDN? Why isn’t Google complaning? Or Akamai? Or Amazon? Or Twitch? Or Microsoft? Or any of the others that have built their own network?

          We have yet to see a single major content owner other than Netflix suggest that it should be completely free to them.

          • Jon_Irenicus

            I am perfectly fine with netflix paying for some of the costs associated with upgrading the ports and routers and capacity into an ISP to handle their increased traffic. But where does netflix cost burden end and comcasts begin?

            And you are deliberately ignoring the fact that many other ISPs in the US and around the world do not engage in the kinds of arrangements that comcast and verizon engage in. Why? If it’s such a raw deal for them, why don’t they charge companies like netflix and youtube to send traffic into their networks? Why does Cablevision choose not to charge netflix for all their traffic? Or Or google fiber?

            Are those companies just a bunch of suckers? Feeding netflix like some parasite? Perhaps they are just better companies, with a better attitude than you expect from an ISP. Companies that want to provide better service for their customer base for its own sake, without trying to milk every scrap they can and control as much as they can for competing content that gets sent into their networks. Perhaps they know that one of the main drivers of increased fees to higher tiers of broadband speed is the desire to stream video, and that netflix is not seen as a parasite, but as a sort of symbiotic service that drives an increased desire to use more and more of a core service. If the costs of sending that traffic into the network is a significant chunk of the profit/revenues of an ISP, then and ONLY then should the cost sharing should be a bigger focus. Until those numbers are provided though, I just expect more than you do from an ISP.

  • Matt Brown

    For every troll there are hundreds of people that are grateful for the information and service you provide.

  • Charles Finkelstein

    I worked with Dan for years. He’s Blunt, honest and does his best to tell the political truth without pissing off sponsors too much . I believe his report is unbiased and I thank him for his reporting style.

  • Thanks for the following informative post.
    lab report writing